Date:   Fri, 25 Oct 2019 02:29:04 +0300
From:   Boaz Harrosh <boaz@...xistor.com>
To:     Dave Chinner <david@...morbit.com>,
        Boaz Harrosh <boaz@...xistor.com>
Cc:     ira.weiny@...el.com, linux-kernel@...r.kernel.org,
        Alexander Viro <viro@...iv.linux.org.uk>,
        "Darrick J. Wong" <darrick.wong@...cle.com>,
        Dan Williams <dan.j.williams@...el.com>,
        Christoph Hellwig <hch@....de>,
        "Theodore Y. Ts'o" <tytso@....edu>, Jan Kara <jack@...e.cz>,
        linux-ext4@...r.kernel.org, linux-xfs@...r.kernel.org,
        linux-fsdevel@...r.kernel.org
Subject: Re: [PATCH 0/5] Enable per-file/directory DAX operations

On 25/10/2019 00:35, Dave Chinner wrote:
> On Thu, Oct 24, 2019 at 05:05:45PM +0300, Boaz Harrosh wrote:
<>
>> Yes I said that as well.
> 
> You said "it must be set between creation and first write",
> stating the requirement for an on-disk flag to work.

Sorry for not being clear. Its not new, I do not explain myself
very well.

The above quote is if you set/clear the flag directly.

> I'm describing how that requirement is actually implemented. i.e. what
> you are stating is something we actually implemented years ago...
> 

What you are talking about is when the flag is inherited from parent.
And I did mention that option as well.

[Which is my concern because my main point is that I want creation+write
 to be none-DAX. Then after writer is done. Switch to DAX-on so READs can
 be fast and not take any memory.
 And you talked about another case when I start DAX-on and then for
 say, use for RDMA turn it off later.
]

This flag is Unique because current RFC has an i_size == 0 check
that other flags do not have

>>> I also seem
>>> to recall that there was a need to take some vm level lock to really
>>> prevent page fault races, and that we can't safely take that in a
>>> safe combination with all the filesystem locks we need.
>>>
>>
>> We do not really care with page fault races in the Kernel as long
> 
> Oh yes we do. A write fault is a 2-part operation - a read fault to
> populate the pte and mapping, then a write fault (->page_mkwrite) to 
> do all the filesystem work needed to dirty the page and pte.
> 
> The read fault sets up the state for the write fault, and if we
> change the aops between these two operations, then the
> ->page_mkwrite implementation goes kaboom.
> 
> This isn't a theoretical problem - this is exactly the race
> condition that lead us to disabling the flag in the first place.
> There is no serialisation between the read and write parts of the
> page fault iand the filesystem changing the DAX flag and ops vector,
> and so fixing this problem requires hold yet more locks in the
> filesystem path to completely lock out page fault processing on the
> inode's mapping.
> 

Again sorry that I do not explain very good.

Already on the read fault we populate the xarray,
My point was that if I want to set the DAX mode I must enforce that
there are no other parallel users on my inode. The check that the
xarray is empty is my convoluted way to check that there are no other
users except me. If xarray is not empty I bail out with EBUISY

I agree this is stupid.

Which is the same stupid as i_size==0 check. Both have the same
intention, to make sure that nothing is going on in parallel to
me.

>> as I protect the xarray access and these are protected well if we
>> take truncate locking. But we have a bigger problem that you pointed
>> out with the change of the operations vector pointer.
>>
>> I was thinking about this last night. One way to do this is with
>> file-exclusive-lock. Correct me if I'm wrong:
>> file-exclusive-readwrite-lock means any other openers will fail and
>> if there are openers already the lock will fail. Which is what we want
>> no?
> 
> The filesystem ioctls and page faults have no visibility of file
> locks.  They don't know and can't find out in a sane manner that an
> inode has a single -user- reference.
> 

This is not true. The FS has full control. It is not too hard a work
to take a file-excl-lock from within the IOCTL implementation. then
unlock. that is option 1. Option 2 of the App taking the lock then
for the check we might need a new export from the lock-subsystem.

> And it introduces a new problem for any application using the
> fssetxattr() ioctl - accidentally not setting the S_DAX flag to be
> unmodified will now fail, and that means such a change breaks
> existing applications. Sure, you can say they are "buggy
> applications", but the fact is this user API change breaks them.
> 

I am not sure I completely understood. let me try ...

The app wants to set some foo flag. It can set the ignore mask to all
1(s) except the flag it wants to set/clear.
And/or get_current_flags(); modify foo_flag; set_flags().

Off course in both the ignore case or when the DAX bit does not change
we do not go on a locking rampage.

So I'm not sure I see the problem

> Hence I don't think we can change the user API for setting/clearing
> this flag like this.
> 

Yes we must not modify behavior of Apps that are modifing other flags.

> Cheers,
> Dave.
> 

Perhaps we always go by the directory. And then do an mv dir_DAX/foo dir_NODAX/foo
to have an effective change. In hard links the first one at iget time before populating
the inode cache takes affect. (And never change the flag on the fly)
(Just brain storming here)

Or perhaps another API?

Thanks Dave
Boaz

Powered by blists - more mailing lists