linux-kernel - Re: Known and unfixed active data loss bug in MM + XFS with large folios since Dec 2021 (any kernel from 6.1 upwards)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <b40b2b1c-3ed5-4943-b8d0-316e04cb1dab@meta.com>
Date: Fri, 13 Sep 2024 12:33:49 -0400
From: Chris Mason <clm@...a.com>
To: Matthew Wilcox <willy@...radead.org>
Cc: Linus Torvalds <torvalds@...ux-foundation.org>,
        Jens Axboe <axboe@...nel.dk>, Christian Theune <ct@...ingcircus.io>,
        linux-mm@...ck.org,
        "linux-xfs@...r.kernel.org" <linux-xfs@...r.kernel.org>,
        linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
        Daniel Dao <dqminh@...udflare.com>, Dave Chinner <david@...morbit.com>,
        regressions@...ts.linux.dev, regressions@...mhuis.info
Subject: Re: Known and unfixed active data loss bug in MM + XFS with large
 folios since Dec 2021 (any kernel from 6.1 upwards)



On 9/13/24 11:51 AM, Matthew Wilcox wrote:
> On Fri, Sep 13, 2024 at 11:30:41AM -0400, Chris Mason wrote:
>> I've mentioned this in the past to both Willy and Dave Chinner, but so
>> far all of my attempts to reproduce it on purpose have failed.  It's
>> awkward because I don't like to send bug reports that I haven't
>> reproduced on a non-facebook kernel, but I'm pretty confident this bug
>> isn't specific to us.
> 
> I don't think the bug is specific to you either.  It's been hit by
> several people ... but it's really hard to hit ;-(  
> 
>> I'll double down on repros again during plumbers and hopefully come up
>> with a recipe for explosions.  On other important datapoint is that we
> 
> I appreciate the effort!
> 
>> The issue looked similar to Christian Theune's rcu stalls, but since it
>> was just one CPU spinning away, I was able to perf probe and drgn my way
>> to some details.  The xarray for the file had a series of large folios:
>>
>> [ index 0 large folio from the correct file ]
>> [ index 1: large folio from the correct file ]
>> ...
>> [ index N: large folio from a completely different file ]
>> [ index N+1: large folio from the correct file ]
>>
>> I'm being sloppy with index numbers, but the important part is that
>> we've got a large folio from the wrong file in the middle of the bunch.
> 
> If you could get the precise index numbers, that would be an important
> clue.  It would be interesting to know the index number in the xarray
> where the folio was found rather than folio->index (as I suspect that
> folio->index is completely bogus because folio->mapping is wrong).
> But gathering that info is going to be hard.

This particular debug session was late at night while we were urgently
trying to roll out some NFS features.  I didn't really save many of the
details because my plan was to reproduce it and make a full bug report.

Also, I was explaining the details to people in workplace chat, which is
wildly bad at rendering long lines of structured text, especially when
half the people in the chat are on a mobile device.

You're probably wondering why all of that is important...what I'm really
trying to say is that I've attached a screenshot of the debugging output.

It came from a older drgn script, where I'm still clinging to "radix",
and you probably can't trust the string representation of the page flags
because I wasn't yet using Omar's helpers and may have hard coded them
from an older kernel.

-chris
Download attachment "xarray-debug.png" of type "image/png" (933545 bytes)