Message-ID: <4f7e45b6-c237-404a-a4c0-4929fa3f1c4b@kernel.dk>
Date: Wed, 27 Nov 2024 14:16:24 -0700
From: Jens Axboe <axboe@...nel.dk>
To: Kent Overstreet <kent.overstreet@...ux.dev>, Jann Horn <jannh@...gle.com>
Cc: linux-bcachefs@...r.kernel.org, kernel list
 <linux-kernel@...r.kernel.org>, Pavel Begunkov <asml.silence@...il.com>,
 io-uring <io-uring@...r.kernel.org>
Subject: Re: bcachefs: suspicious mm pointer in struct dio_write

On 11/27/24 2:08 PM, Kent Overstreet wrote:
> On Wed, Nov 27, 2024 at 09:44:21PM +0100, Jann Horn wrote:
>> On Wed, Nov 27, 2024 at 9:25?PM Kent Overstreet
>> <kent.overstreet@...ux.dev> wrote:
>>> On Wed, Nov 27, 2024 at 11:09:14AM -0700, Jens Axboe wrote:
>>>> On 11/27/24 9:57 AM, Jann Horn wrote:
>>>>> Hi!
>>>>>
>>>>> In fs/bcachefs/fs-io-direct.c, "struct dio_write" contains a pointer
>>>>> to an mm_struct. This pointer is grabbed in bch2_direct_write()
>>>>> (without any kind of refcount increment), and used in
>>>>> bch2_dio_write_continue() for kthread_use_mm()/kthread_unuse_mm()
>>>>> which are used to enable userspace memory access from kthread context.
>>>>> I believe kthread_use_mm()/kthread_unuse_mm() require that the caller
>>>>> guarantees that the MM hasn't gone through exit_mmap() yet (normally
>>>>> by holding an mmget() reference).
>>>>>
>>>>> If we reach this codepath via io_uring, do we have a guarantee that
>>>>> the mm_struct that called bch2_direct_write() is still alive and
>>>>> hasn't yet gone through exit_mmap() when it is accessed from
>>>>> bch2_dio_write_continue()?
>>>>>
>>>>> I don't know the async direct I/O codepath particularly well, so I
>>>>> cc'ed the uring maintainers, who probably know this better than me.
>>>>
>>>> I _think_ this is fine as-is, even if it does look dubious and bcachefs
>>>> arguably should grab an mm ref for this just for safety to avoid future
>>>> problems. The reason is that bcachefs doesn't set FMODE_NOWAIT, which
>>>> means that on the io_uring side it cannot do non-blocking issue of
>>>> requests. This is slower as it always punts to an io-wq thread, which
>>>> shares the same mm. Hence if the request is alive, there's always a
>>>> thread with the same mm alive as well.
>>>>
>>>> Now if FMODE_NOWAIT was set, then the original task could exit. I'd need
>>>> to dig a bit deeper to verify that would always be safe and there's not
>>>> a of time today with a few days off in the US looming, so I'll defer
>>>> that to next week. It certainly would be fine with an mm ref grabbed.
>>>
>>> Wouldn't delivery of completions be tied to an address space (not a
>>> process) like it is for aio?
>>
>> An io_uring instance is primarily exposed to userspace as a file
>> descriptor, so AFAIK it is possible for the io_uring instance to live
>> beyond when the last mmput() happens. io_uring initially only holds an
>> mmgrab() reference on the MM (a comment above that explains: "This is
>> just grabbed for accounting purposes"), which I think is not enough to
>> make it stable enough for kthread_use_mm(); I think in io_uring, only
>> the worker threads actually keep the MM fully alive (and AFAIK the
>> uring worker threads can exit before the uring instance itself is torn
>> down).
>>
>> To receive io_uring completions, there are multiple ways, but they
>> don't use a pointer from the io_uring instance to the MM to access
>> userspace memory. Instead, you can have a VMA that points to the
>> io_uring instance, created by calling mmap() on the io_uring fd; or
>> alternatively, with IORING_SETUP_NO_MMAP, you can have io_uring grab
>> references to userspace-provided pages.
>>
>> On top of that, I think it might currently be possible to use the
>> io_uring file descriptor from another task to submit work. (That would
>> probably be fairly nonsensical, but I think the kernel does not
>> currently prevent it.)
> 
> Ok, that's a wrinkle.

I'd argue the fact that you are using an mm from a different process
without grabbing a reference is the wrinkle. I just don't think it's a
problem right now, but it could be... aio is tied to the mm because of
how it does completions, potentially, and hence needs this exit_aio()
hack because of that. aio also doesn't care, because it doesn't care
about blocking - it'll happily block during issue.

> Jens, is it really FMODE_NOWAIT that controls whether we can hit this? A
> very cursory glance leads me to suspect "no", it seems like this is a
> bug if io_uring is allowed on bcachefs at all.

I just looked at bcachefs dio writes, which look to be the only case of
this. And yes, for writes, if FMODE_NOWAIT isn't set, then io-wq is
always involved for the IO.

-- 
Jens Axboe

Powered by blists - more mailing lists