Message-ID: <03CEBF7E-649D-4BE3-B759-EA2E561B4110@oracle.com>
Date:   Thu, 8 Sep 2022 16:15:12 +0000
From:   Chuck Lever III <chuck.lever@...cle.com>
To:     Bruce Fields <bfields@...ldses.org>, Theodore Ts'o <tytso@....edu>
CC:     Jeff Layton <jlayton@...nel.org>, Jan Kara <jack@...e.cz>,
        Neil Brown <neilb@...e.de>,
        "adilger.kernel@...ger.ca" <adilger.kernel@...ger.ca>,
        "Darrick J. Wong" <djwong@...nel.org>,
        Dave Chinner <david@...morbit.com>,
        Trond Myklebust <trondmy@...merspace.com>,
        Al Viro <viro@...iv.linux.org.uk>,
        "zohar@...ux.ibm.com" <zohar@...ux.ibm.com>,
        "xiubli@...hat.com" <xiubli@...hat.com>,
        Lukas Czerner <lczerner@...hat.com>,
        Christian Brauner <brauner@...nel.org>,
        "fweimer@...hat.com" <fweimer@...hat.com>,
        "linux-man@...r.kernel.org" <linux-man@...r.kernel.org>,
        "linux-api@...r.kernel.org" <linux-api@...r.kernel.org>,
        "linux-btrfs@...r.kernel.org" <linux-btrfs@...r.kernel.org>,
        linux-fsdevel <linux-fsdevel@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        "ceph-devel@...r.kernel.org" <ceph-devel@...r.kernel.org>,
        "linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
        Linux NFS Mailing List <linux-nfs@...r.kernel.org>,
        Linux-XFS <linux-xfs@...r.kernel.org>
Subject: Re: [man-pages RFC PATCH v4] statx, inode: document the new
 STATX_INO_VERSION field



> On Sep 8, 2022, at 11:56 AM, J. Bruce Fields <bfields@...ldses.org> wrote:
> 
> On Thu, Sep 08, 2022 at 11:44:33AM -0400, Jeff Layton wrote:
>> On Thu, 2022-09-08 at 11:21 -0400, Theodore Ts'o wrote:
>>> On Thu, Sep 08, 2022 at 10:33:26AM +0200, Jan Kara wrote:
>>>> It boils down to the fact that we don't want to call mark_inode_dirty()
>>>> from IOCB_NOWAIT path because for lots of filesystems that means journal
>>>> operation and there are high chances that may block.
>>>> 
>>>> Presumably we could treat inode dirtying after i_version change similarly
>>>> to how we handle timestamp updates with lazytime mount option (i.e., not
>>>> dirty the inode immediately but only with a delay) but then the time window
>>>> for i_version inconsistencies due to a crash would be much larger.
>>> 
>>> Perhaps this is a radical suggestion, but there seems to be a lot of
>>> the problems which are due to the concern "what if the file system
>>> crashes" (and so we need to worry about making sure that any
>>> increments to i_version MUST be persisted after it is incremented).
>>> 
>>> Well, if we assume that unclean shutdowns are rare, then perhaps we
>>> shouldn't be optimizing for that case.  So.... what if a file system
>>> had a counter which got incremented each time its journal is replayed
>>> representing an unclean shutdown.  That shouldn't happen often, but if
>>> it does, there might be any number of i_version updates that may have
>>> gotten lost.  So in that case, the NFS client should invalidate all of
>>> its caches.
>>> 
>>> If the i_version field was large enough, we could just prefix the
>>> "unclean shutdown counter" with the existing i_version number when it
>>> is sent over the NFS protocol to the client.  But if that field is too
>>> small, and if (as I understand things) NFS just needs to know when
>>> i_version is different, we could just simply hash the "unclean
>>> shtudown counter" with the inode's "i_version counter", and let that
>>> be the version which is sent from the NFS client to the server.
>>> 
>>> If we could do that, then it doesn't become critical that every single
>>> i_version bump has to be persisted to disk, and we could treat it like
>>> a lazytime update; it's guaranteed to updated when we do an clean
>>> unmount of the file system (and when the file system is frozen), but
>>> on a crash, there is no guaranteee that all i_version bumps will be
>>> persisted, but we do have this "unclean shutdown" counter to deal with
>>> that case.
>>> 
>>> Would this make life easier for folks?
>>> 
>>> 						- Ted
>> 
>> Thanks for chiming in, Ted. That's part of the problem, but we're
>> actually not too worried about that case:
>> 
>> nfsd mixes the ctime in with i_version, so you'd have to crash+clock
>> jump backward by juuuust enough to allow you to get the i_version and
>> ctime into a state it was before the crash, but with different data.
>> We're assuming that that is difficult to achieve in practice.
> 
> But a change in the clock could still cause our returned change
> attribute to go backwards (even without a crash).  Not sure how to
> evaluate the risk, but it was enough that Trond hasn't been comfortable
> with nfsd advertising NFS4_CHANGE_TYPE_IS_MONOTONIC.
> 
> Ted's idea would be sufficient to allow us to turn that flag on, which I
> think allows some client-side optimizations.
> 
>> The issue with a reboot counter (or similar) is that on an unclean crash
>> the NFS client would end up invalidating every inode in the cache, as
>> all of the i_versions would change. That's probably excessive.
> 
> But if we use the crash counter on write instead of read, we don't
> invalidate caches unnecessarily.  And I think the monotonicity would
> still be close enough for our purposes?
> 
>> The bigger issue (at the moment) is atomicity: when we fetch an
>> i_version, the natural inclination is to associate that with the state
>> of the inode at some point in time, so we need this to be updated
>> atomically with certain other attributes of the inode. That's the part
>> I'm trying to sort through at the moment.
> 
> That may be, but I still suspect the crash counter would help.

Fwiw, I like the crash counter idea too.

--
Chuck Lever

Powered by blists - more mailing lists