linux-kernel - Re: fscache corruption in Linux 5.17?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <705278.1650462934@warthog.procyon.org.uk>
Date:   Wed, 20 Apr 2022 14:55:34 +0100
From:   David Howells <dhowells@...hat.com>
To:     Max Kellermann <mk@...all.com>
Cc:     dhowells@...hat.com, linux-cachefs@...hat.com,
        linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: fscache corruption in Linux 5.17?

Max Kellermann <mk@...all.com> wrote:

> > Do the NFS servers change the files that are being served - or is it
> > just WordPress pushing the changes to the NFS servers for the web
> > servers to then export?
> 
> I'm not sure if I understand this question correctly.  The NFS server
> (a NetApp, btw.) sees the new file contents correctly; all other web
> servers also see non-corrupt new files.  Only the one web server which
> performed the update saw broken files.

I was wondering if there was missing invalidation if the web clients were
modifying the same files in parallel, but it sounds like only one place is
doing the modification, and the problem is the lack of invalidation when a
file is opened for writing.

I have a tentative patch for this - see attached.

David
---
commit 9b00af0190dfee6073aab47ee88e15c31d3c357d
Author: David Howells <dhowells@...hat.com>
Date:   Wed Apr 20 14:27:17 2022 +0100

    fscache: Fix invalidation/lookup race
    
    If an NFS file is opened for writing and closed, fscache_invalidate() will
    be asked to invalidate the file - however, if the cookie is in the
    LOOKING_UP state (or the CREATING state), then request to invalidate
    doesn't get recorded for fscache_cookie_state_machine() to do something
    with.
    
    Fix this by making __fscache_invalidate() set a flag if it sees the cookie
    is in the LOOKING_UP state to indicate that we need to go to invalidation.
    Note that this requires a count on the n_accesses counter for the state
    machine, which that will release when it's done.
    
    fscache_cookie_state_machine() then shifts to the INVALIDATING state if it
    sees the flag.
    
    Without this, an nfs file can get corrupted if it gets modified locally and
    then read locally as the cache contents may not get updated.
    
    Fixes: d24af13e2e23 ("fscache: Implement cookie invalidation")
    Reported-by: Max Kellermann <mk@...all.com>
    Signed-off-by: David Howells <dhowells@...hat.com>
    Link: https://lore.kernel.org/r/YlWWbpW5Foynjllo@rabbit.intern.cm-ag [1]

diff --git a/fs/fscache/cookie.c b/fs/fscache/cookie.c
index 9d3cf0111709..3bb6deeb4279 100644
--- a/fs/fscache/cookie.c
+++ b/fs/fscache/cookie.c
@@ -705,7 +705,11 @@ static void fscache_cookie_state_machine(struct fscache_cookie *cookie)
 		spin_unlock(&cookie->lock);
 		fscache_init_access_gate(cookie);
 		fscache_perform_lookup(cookie);
-		goto again;
+		spin_lock(&cookie->lock);
+		if (test_and_clear_bit(FSCACHE_COOKIE_DO_INVALIDATE, &cookie->flags))
+			__fscache_set_cookie_state(cookie,
+						   FSCACHE_COOKIE_STATE_INVALIDATING);
+		goto again_locked;
 
 	case FSCACHE_COOKIE_STATE_INVALIDATING:
 		spin_unlock(&cookie->lock);
@@ -752,6 +756,9 @@ static void fscache_cookie_state_machine(struct fscache_cookie *cookie)
 			spin_lock(&cookie->lock);
 		}
 
+		if (test_and_clear_bit(FSCACHE_COOKIE_DO_INVALIDATE, &cookie->flags))
+			fscache_end_cookie_access(cookie, fscache_access_invalidate_cookie_end);
+
 		switch (state) {
 		case FSCACHE_COOKIE_STATE_RELINQUISHING:
 			fscache_see_cookie(cookie, fscache_cookie_see_relinquish);
@@ -1048,6 +1055,9 @@ void __fscache_invalidate(struct fscache_cookie *cookie,
 		return;
 
 	case FSCACHE_COOKIE_STATE_LOOKING_UP:
+		__fscache_begin_cookie_access(cookie, fscache_access_invalidate_cookie);
+		set_bit(FSCACHE_COOKIE_DO_INVALIDATE, &cookie->flags);
+		fallthrough;
 	case FSCACHE_COOKIE_STATE_CREATING:
 		spin_unlock(&cookie->lock);
 		_leave(" [look %x]", cookie->inval_counter);
diff --git a/include/linux/fscache.h b/include/linux/fscache.h
index e25539072463..a25804f141d3 100644
--- a/include/linux/fscache.h
+++ b/include/linux/fscache.h
@@ -129,6 +129,7 @@ struct fscache_cookie {
 #define FSCACHE_COOKIE_DO_PREP_TO_WRITE	12		/* T if cookie needs write preparation */
 #define FSCACHE_COOKIE_HAVE_DATA	13		/* T if this cookie has data stored */
 #define FSCACHE_COOKIE_IS_HASHED	14		/* T if this cookie is hashed */
+#define FSCACHE_COOKIE_DO_INVALIDATE	15		/* T if cookie needs invalidation */
 
 	enum fscache_cookie_state	state;
 	u8				advice;		/* FSCACHE_ADV_* */