linux-kernel - Re: [REGRESSION] Null pointer dereference while shrinking zswap

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAKEwX=NM1y-K1-Yw=CH3cM-8odER1PZBVoWo-rs7_OdjFG_puw@mail.gmail.com>
Date: Tue, 16 Apr 2024 16:29:39 -0700
From: Nhat Pham <nphamcs@...il.com>
To: Christian Heusel <christian@...sel.eu>
Cc: Seth Jennings <sjenning@...hat.com>, Dan Streetman <ddstreet@...e.org>, 
	Vitaly Wool <vitaly.wool@...sulko.com>, Andrew Morton <akpm@...ux-foundation.org>, linux-mm@...ck.org, 
	linux-kernel@...r.kernel.org, David Runge <dave@...epmap.de>, 
	"Richard W.M. Jones" <rjones@...hat.com>, Mark W <instruform@...il.com>, regressions@...ts.linux.dev, 
	Johannes Weiner <hannes@...xchg.org>, Yosry Ahmed <yosryahmed@...gle.com>, 
	Chengming Zhou <chengming.zhou@...ux.dev>
Subject: Re: [REGRESSION] Null pointer dereference while shrinking zswap

On Tue, Apr 16, 2024 at 3:14 PM Nhat Pham <nphamcs@...il.com> wrote:
>
> On Tue, Apr 16, 2024 at 5:19 AM Christian Heusel <christian@...sel.eu> wrote:
> >
> > Hello everyone,
>
> Thanks for the report, Christian! Looking at it now.
>
> >
> > while rebuilding a few packages in Arch Linux we have recently come
> > across a regression in the linux kernel which was made visible by a test
> > failure in libguestfs[0], where the booted kernel showed a Call Trace
> > like the following one:
> >
> > [  218.738568] CPU: 0 PID: 167 Comm: guestfsd Not tainted 6.7.0-rc4-1-mainline-00158-gb5ba474f3f51 #1 bf39861cf50acae7a79c534e25532f28afe4e593^M
>
> Is this one of the kernel versions that was broken? That looks a bit
> odd, as zswap shrinker landed on 6.8...

Ah ignore this - I understand the versioning now...

>
> > [  218.739007] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Arch Linux 1.16.3-1-1 04/01/2014^M
> > [  218.739787] RIP: 0010:memcg_page_state+0x9/0x30^M
> > [  218.740299] Code: 0d b8 ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <48> 8b 87 00 06 00 00 48 63 f6 31 d2 48 8b 04 f0 48 85 c0 48 0f 48^M
> > [  218.740727] RSP: 0018:ffffb5fa808dfc10 EFLAGS: 00000202^M
> > [  218.740862] RAX: 0000000000000000 RBX: ffffb5fa808dfce0 RCX: 0000000000000002^M
> > [  218.741016] RDX: 0000000000000001 RSI: 0000000000000033 RDI: 0000000000000000^M
> > [  218.741168] RBP: 0000000000000000 R08: ffff976681ff8000 R09: 0000000000000000^M
> > [  218.741322] R10: 0000000000000001 R11: ffff9766833f9d00 R12: ffff9766ffffe780^M
> > [  218.742167] R13: 0000000000000000 R14: ffff976680cc1800 R15: ffff976682204d80^M
> > [  218.742376] FS:  00007f1479d9f540(0000) GS:ffff9766fbc00000(0000) knlGS:0000000000000000^M
> > [  218.742569] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033^M
> > [  218.743256] CR2: 0000000000000600 CR3: 0000000103606000 CR4: 0000000000750ef0^M
> > [  218.743494] PKRU: 55555554^M
> > [  218.743593] Call Trace:^M
> > [  218.743733]  <TASK>^M
> > [  218.743847]  ? __die+0x23/0x70^M
> > [  218.743957]  ? page_fault_oops+0x171/0x4e0^M
> > [  218.744056]  ? free_unref_page+0xf6/0x180^M
> > [  218.744458]  ? exc_page_fault+0x7f/0x180^M
> > [  218.744551]  ? asm_exc_page_fault+0x26/0x30^M
> > [  218.744684]  ? memcg_page_state+0x9/0x30^M
> > [  218.744779]  zswap_shrinker_count+0x9d/0x110^M
> > [  218.744896]  do_shrink_slab+0x3a/0x360^M
> > [  218.744990]  shrink_slab+0xc7/0x3c0^M
> > [  218.745609]  drop_slab+0x85/0x140^M
> > [  218.745691]  drop_caches_sysctl_handler+0x7e/0xd0^M
> > [  218.745799]  proc_sys_call_handler+0x1c0/0x2e0^M
> > [  218.745912]  vfs_write+0x23d/0x400^M
> > [  218.745998]  ksys_write+0x6f/0xf0^M
> > [  218.746080]  do_syscall_64+0x64/0xe0^M
> > [  218.746169]  ? exit_to_user_mode_prepare+0x132/0x1f0^M
> > [  218.746873]  entry_SYSCALL_64_after_hwframe+0x6e/0x76^M
> >
> > The regression is present in the mainline kernel and also was
> > independently reported to the redhat bugtracker[1].
> >
> > I have bisected (see log[2]) the regression between v6.9-rc4 and v6.6
> > and have landed on the following results (removed unrelated test commit)
> > as remainders since some of the commits were not buildable for me:
> > - 7108cc3f765c ("mm: memcg: add per-memcg zswap writeback stat")
> > - a65b0e7607cc ("zswap: make shrinking memcg-aware")
> > - b5ba474f3f51 ("zswap: shrink zswap pool based on memory pressure")

Hmmm, this smells like a memcg reference counting bug to me. The memcg
was successfully accessed just a couple of LoCs above, but failed
after the stats flushing.

My suspicion right now is the new memcg refcnt dance to select an
memcg for the global, capacity-based, shrinker, introduced in this
patch series. That dance looks solid to me tho - not sure where it
goes wrong. And, if it's a reference counting issue, that should pop
up in different places (i.e other mysterious memcg traces)...