Message-ID: <20190510203922.GJ13038@ziepe.ca>
Date:   Fri, 10 May 2019 17:39:22 -0300
From:   Jason Gunthorpe <jgg@...pe.ca>
To:     Santosh Shilimkar <santosh.shilimkar@...cle.com>
Cc:     netdev@...r.kernel.org, davem@...emloft.net,
        Hans Westgaard Ry <hans.westgaard.ry@...cle.com>
Subject: Re: [net-next][PATCH v2 1/2] rds: handle unsupported rdma request to
 fs dax memory

On Fri, May 10, 2019 at 01:12:49PM -0700, Santosh Shilimkar wrote:
> On 5/10/2019 12:47 PM, Jason Gunthorpe wrote:
> > On Fri, May 10, 2019 at 12:38:31PM -0700, Santosh Shilimkar wrote:
> > > On 5/10/2019 12:20 PM, Jason Gunthorpe wrote:
> > > > On Fri, May 10, 2019 at 11:58:42AM -0700, santosh.shilimkar@...cle.com wrote:
> > > > > On 5/10/19 11:07 AM, Jason Gunthorpe wrote:
> > > > > > On Fri, May 10, 2019 at 11:02:35AM -0700, santosh.shilimkar@...cle.com wrote:
> > > 
> > > [...]
> > > 
> > > > > > Why would you need to detect FS DAX memory? GUP users are not supposed
> > > > > > to care.
> > > > > > 
> > > > > > GUP is supposed to work just 'fine' - other than the usual bugs we
> > > > > > have with GUP and any FS backed memory.
> > > > > > 
> > > > > Am not saying there is any issue with GUP. Let me try to explain the
> > > > > issue first. You are aware of various discussions about doing DMA
> > > > > or RDMA on FS DAX memory. e.g [1] [2] [3]
> > > > > 
> > > > > One of the proposal to do safely RDMA on FS DAX memory is/was ODP
> > > > 
> > > > It is not about safety. ODP is required in all places that would have
> > > > used gup_longterm because ODP avoids th gup_longterm entirely.
> > > > 
> > > > > Currently RDS doesn't have support for ODP MR registration
> > > > > and hence we don't want user application to do RDMA using
> > > > > fastreg/fmr on FS DAX memory which isn't safe.
> > > > 
> > > > No, it is safe.
> > > > 
> > > > The only issue is you need to determine if this use of GUP is longterm
> > > > or short term. Longterm means userspace is in control of how long the
> > > > GUP lasts, short term means the kernel is in control.
> > > > 
> > > > ie posting a fastreg, sending the data, then un-GUP'ing on completion
> > > > is a short term GUP and it is fine on any type of memory.
> > > > 
> > > > So if it is a long term pin then it needs to be corrected and the only
> > > > thing the comment needs to explain is that it is a long term pin.
> > > > 
> > > Thanks for clarification. At least the distinction is clear to me now. Yes
> > > the key can be valid for long term till the remote RDMA IO is issued and
> > > finished. After that user can issue an invalidate/free key or
> > > upfront specify a flag to free/invalidate the key on remote IO
> > > completion.
> > 
> > Again, the test is if *userspace* controls this. So if userspace is
> > the thing that does the invalidate/free then it is long term. Sounds
> > like if it specifies the free/invalidate flag then it short term.
> > 
> > At this point you'd probably be better to keep both options.
> > 
> Thats possible using the provided flag state but I am still not sure
> whether its guaranteed to be safe when marked as short term even with
> flag which tells kernel to invalidate/free the MR on remote IO
> completion. Till the remote server finishes the IO, 

This is fine.

> there is still a window where userspace on local server can modify
> the file mappings. Registered file handle say was ftuncated to zero
> by another process and the backing memory was allocated by some
> other process as part of fallocate.  

The FS is supposed to maintain sane semantics across GUP - fallocate
should block until GUP is done. This is normal.

> How do we avoid such an issue without GUP_longterm ?

You don't, there is no problem using GUP for short term - ie a time
frame entirely under control of the kernel.

Jason

Powered by blists - more mailing lists