linux-kernel - Re: Known and unfixed active data loss bug in MM + XFS with large folios since Dec 2021 (any kernel from 6.1 upwards)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <295BE120-8BF4-41AE-A506-3D6B10965F2B@flyingcircus.io>
Date: Mon, 30 Sep 2024 21:25:22 +0200
From: Christian Theune <ct@...ingcircus.io>
To: Linus Torvalds <torvalds@...ux-foundation.org>
Cc: Dave Chinner <david@...morbit.com>,
 Matthew Wilcox <willy@...radead.org>,
 Chris Mason <clm@...a.com>,
 Jens Axboe <axboe@...nel.dk>,
 linux-mm@...ck.org,
 "linux-xfs@...r.kernel.org" <linux-xfs@...r.kernel.org>,
 linux-fsdevel@...r.kernel.org,
 linux-kernel@...r.kernel.org,
 Daniel Dao <dqminh@...udflare.com>,
 regressions@...ts.linux.dev,
 regressions@...mhuis.info
Subject: Re: Known and unfixed active data loss bug in MM + XFS with large
 folios since Dec 2021 (any kernel from 6.1 upwards)


> On 30. Sep 2024, at 20:46, Linus Torvalds <torvalds@...ux-foundation.org> wrote:
> 
> On Mon, 30 Sept 2024 at 10:35, Christian Theune <ct@...ingcircus.io> wrote:
>> 
>> Sep 27 00:51:20 <redactedhostname>13 kernel:  folio_wait_bit_common+0x13f/0x340
>> Sep 27 00:51:20 <redactedhostname>13 kernel:  folio_wait_writeback+0x2b/0x80
> 
> Gaah. Every single case you point to is that folio_wait_writeback() case.
> 
> And this might be an old old annoyance.

I’m being told that I’m somewhat of a truffle pig for dirty code … how long ago does “old old” refer to, btw?

> […]
> IOW, this code is known-broken and might have extreme unfairness
> issues (although I had blissfully forgotten about it), because while
> the actual writeback *bit* itself is set and cleared atomically, the
> wakeup for the bit is asynchronous and can be delayed almost
> arbitrarily, so you can get basically spurious wakeups that were from
> a previous bit clear.

I wonder whether the extreme unfairness gets exacerbated when in a cgroup throttled context … It’s a limited number of workloads we 
have seen this with, some of which are parallelized and others aren’t. (and I guess non-parallelized code shouldn’t suffer much from this?)

Maybe I can reproduce this more easily and  ...

> So the code here is questionable, and might cause some issues, but the
> starvation of folio_wait_writeback() can't explain _all_ the cases you
> see.

… also get you more data and dig for maybe more cases more systematically.
Anything particular you’d like me to look for? Any specific additional data
points that would help?

We’re going to keep with 6.11 in staging and avoid rolling it out to the production machines for now.

Christian

-- 
Christian Theune · ct@...ingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick