linux-kernel - Re: [PATCH v8 00/27] mm/ksw: Introduce KStackWatch debugging tool

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aRTv0eHfX0j8vJOW@casper.infradead.org>
Date: Wed, 12 Nov 2025 20:36:33 +0000
From: Matthew Wilcox <willy@...radead.org>
To: Jinchao Wang <wangjinchao600@...il.com>
Cc: kasan-dev@...glegroups.com, linux-arm-kernel@...ts.infradead.org,
	linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
	linux-mm@...ck.org, linux-perf-users@...r.kernel.org,
	linux-trace-kernel@...r.kernel.org, llvm@...ts.linux.dev,
	workflows@...r.kernel.org, x86@...nel.org
Subject: Re: [PATCH v8 00/27] mm/ksw: Introduce KStackWatch debugging tool

[dropping all the individual email addresses; leaving only the
mailing lists]

On Wed, Nov 12, 2025 at 10:14:29AM +0800, Jinchao Wang wrote:
> On Mon, Nov 10, 2025 at 05:33:22PM +0000, Matthew Wilcox wrote:
> > On Tue, Nov 11, 2025 at 12:35:55AM +0800, Jinchao Wang wrote:
> > > Earlier this year, I debugged a stack corruption panic that revealed the
> > > limitations of existing debugging tools. The bug persisted for 739 days
> > > before being fixed (CVE-2025-22036), and my reproduction scenario
> > > differed from the CVE report—highlighting how unpredictably these bugs
> > > manifest.
> > 
> > Well, this demonstrates the dangers of keeping this problem siloed
> > within your own exfat group.  The fix made in 1bb7ff4204b6 is wrong!
> > It was fixed properly in 7375f22495e7 which lists its Fixes: as
> > Linux-2.6.12-rc2, but that's simply the beginning of git history.
> > It's actually been there since v2.4.6.4 where it's documented as simply:
> > 
> >       - some subtle fs/buffer.c race conditions (Andrew Morton, me)
> > 
> > As far as I can tell the changes made in 1bb7ff4204b6 should be
> > reverted.
> 
> Thank you for the correction and the detailed history. I wasn't aware this
> dated back to v2.4.6.4. I'm not part of the exfat group; I simply
> encountered a bug that 1bb7ff4204b6 happened to resolve in my scenario.
> The timeline actually illustrates the exact problem KStackWatch addresses:
> a bug introduced in 2001, partially addressed in 2025, then properly fixed
> months later. The 24-year gap suggests these silent stack corruptions are
> extremely difficult to locate.

I think that's a misdiagnosis caused by not understanding the limited
circumstances in which the problem occurs.  To hit this problem, you
have to have a buffer_head allocated on the stack.  That doesn't happen
in many places:

fs/buffer.c:    struct buffer_head tmp = {
fs/direct-io.c: struct buffer_head map_bh = { 0, };
fs/ext2/super.c:        struct buffer_head tmp_bh;
fs/ext2/super.c:        struct buffer_head tmp_bh;
fs/ext4/mballoc-test.c: struct buffer_head bitmap_bh;
fs/ext4/mballoc-test.c: struct buffer_head gd_bh;
fs/gfs2/bmap.c: struct buffer_head bh;
fs/gfs2/bmap.c: struct buffer_head bh;
fs/isofs/inode.c:       struct buffer_head dummy;
fs/jfs/super.c: struct buffer_head tmp_bh;
fs/jfs/super.c: struct buffer_head tmp_bh;
fs/mpage.c:     struct buffer_head map_bh;
fs/mpage.c:     struct buffer_head map_bh;

It's far more common for buffer_heads to be allocated from slab and
attached to folios.  The other necessary condition to hit this problem
is that get_block() has to actually read the data from disk.  That's
not normal either!  Most filesystems just fill in the metadata about
the block and defer the actual read to when the data is wanted.  That's
the high-performance way to do it.

So our opportunity to catch this bug was highly limited by the fact that
we just don't run the codepaths that would allow it to trigger.

> > > Initially, I enabled KASAN, but the bug did not reproduce. Reviewing the
> > > code in __blk_flush_plug(), I found it difficult to trace all logic
> > > paths due to indirect function calls through function pointers.
> > 
> > So why is the solution here not simply to fix KASAN instead of this
> > giant patch series?
> 
> KASAN caught 7375f22495e7 because put_bh() accessed bh->b_count after
> wait_on_buffer() of another thread returned—the stack was invalid.
> In 1bb7ff4204b6 and my case, corruption occurred before the victim
> function of another thread returned. The stack remained valid to KASAN,
> so no warning triggered. This is timing-dependent, not a KASAN deficiency.

I agree that it's a narrow race window, but nevertheless KASAN did catch
it with ntfs and not with exfat.  The KASAN documentation states that
it can catch this kind of bug:

Generic KASAN supports finding bugs in all of slab, page_alloc, vmap, vmalloc,
stack, and global memory.

Software Tag-Based KASAN supports slab, page_alloc, vmalloc, and stack memory.

Hardware Tag-Based KASAN supports slab, page_alloc, and non-executable vmalloc
memory.

(hm, were you using hwkasan instead of swkasan, and that's why you
couldn't see it?)