linux-ext4 - Re: task hung in ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20231024013616.pq5wdgmfb55mnyze@awork3.anarazel.de>
Date:   Mon, 23 Oct 2023 18:36:16 -0700
From:   Andres Freund <andres@...razel.de>
To:     Dave Chinner <david@...morbit.com>
Cc:     Theodore Ts'o <tytso@....edu>,
        Thorsten Leemhuis <regressions@...mhuis.info>,
        Shreeya Patel <shreeya.patel@...labora.com>,
        linux-ext4@...r.kernel.org,
        Ricardo Cañuelo <ricardo.canuelo@...labora.com>,
        gustavo.padovan@...labora.com, zsm@...gle.com, garrick@...gle.com,
        Linux regressions mailing list <regressions@...ts.linux.dev>,
        Jens Axboe <axboe@...nel.dk>, io-uring@...r.kernel.org
Subject: Re: task hung in ext4_fallocate #2

Hi,

On 2023-10-24 12:12:23 +1100, Dave Chinner wrote:
> [cc Jens, io-uring]

Yes, good call, had been meaning to do that but forgot.

> On Tue, Oct 17, 2023 at 07:50:09PM -0700, Andres Freund wrote:
> > I had the idea to look at the stacks (via /proc/$pid/stack) for all postgres
> > processes and the associated io-uring threads, and then to deduplicate them.
> > 
> > 22x:
> > ext4_file_write_iter (./include/linux/fs.h:1073 fs/ext4/file.c:57 fs/ext4/file.c:564 fs/ext4/file.c:715)
> > io_write (./include/linux/fs.h:1956 io_uring/rw.c:926)
> > io_issue_sqe (io_uring/io_uring.c:1878)
> > io_wq_submit_work (io_uring/io_uring.c:1960)
> > io_worker_handle_work (io_uring/io-wq.c:596)
> > io_wq_worker (io_uring/io-wq.c:258 io_uring/io-wq.c:648)
> > ret_from_fork (arch/x86/kernel/process.c:147)
> > ret_from_fork_asm (arch/x86/entry/entry_64.S:312)
> 
> io_uring does some interesting stuff with IO completion and iomap
> now - IIRC IOCB_DIO_CALLER_COMP is new 6.6-rc1 functionality. This
> flag is set by io_write() to tell the iomap IO completion handler
> that the caller will the IO completion, avoiding a context switch
> to run the completion in a kworker thread.
> 
> It's not until the caller runs iocb->dio_complete() that
> inode_dio_end() is called to drop the i_dio_count. This happens when
> io_uring gets completion notification from iomap via
> iocb->ki_complete(iocb, 0);
> 
> It then requires the io_uring layer to process the completion
> and call iocb->dio_complete() before the inode->i_dio_count is
> dropped and inode_dio_wait() will complete.
> 
> So what I suspect here is that we have a situation where the worker
> that would run the completion is blocked on the inode rwsem because
> this isn't a IOCB_NOWAIT IO and the fallocate call holds the rwsem
> exclusive and is waiting on inode_dio_wait() to return.
> 
> Cc Jens, because I'm not sure this is actually an ext4 problem - I
> can't see why XFS won't have the same issue w.r.t.
> truncate/fallocate and IOCB_DIO_CALLER_COMP delaying the
> inode_dio_end() until whenever the io_uring code can get around to
> processing the delayed completion....

The absence of a reproducer obviously is not proof of an absence of
problems. With that said, I have run the workload on xfs for far longer on
both bigger and smaller machines, without triggering the same symptoms.

Greetings,

Andres Freund