linux-kernel - Re: [PATCH] mm/workingset: fix crash from corrupted shadow entries in lru

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CADhLXY5ktE6AoarNBmO209cvpRPtrmLB0G=RSN+pAmmbqynHfg@mail.gmail.com>
Date: Tue, 9 Dec 2025 17:06:35 +0530
From: Deepanshu Kartikey <kartikey406@...il.com>
To: "David Hildenbrand (Red Hat)" <david@...nel.org>
Cc: akpm@...ux-foundation.org, axelrasmussen@...gle.com, yuanchu@...gle.com, 
	weixugc@...gle.com, hannes@...xchg.org, mhocko@...nel.org, 
	zhengqi.arch@...edance.com, shakeel.butt@...ux.dev, 
	lorenzo.stoakes@...cle.com, linux-mm@...ck.org, linux-kernel@...r.kernel.org, 
	syzbot+e008db2ac01e282550ee@...kaller.appspot.com, 
	Yu Zhao <yuzhao@...gle.com>
Subject: Re: [PATCH] mm/workingset: fix crash from corrupted shadow entries in lru_gen

On Mon, Dec 8, 2025 at 4:54 PM David Hildenbrand (Red Hat)
<david@...nel.org> wrote:

> That's just hacking around the root cause, no? Because IIUC, that's not
> something we would ever expect to happen unless BUG.
>
> Unless I am missing something this patch is trying to cure the symptoms,
> but not the root cause.
>
> Now, if it would be valid (and we would not have a corruption), then
> handling it like you propose would be the right thing.

Hi David,

Thank you for your review. Here's the root cause analysis with debug evidence:

ROOT CAUSE:
Shadow entries contain invalid NUMA node IDs that don't exist on the
system. When unpack_shadow() calls NODE_DATA(invalid_nid), it returns
NULL, leading to a crash.

EVIDENCE FROM DEBUG LOGS:

1. First crash - invalid node_id=4 (system has nodes 0-3):

[   12.345678] UNPACK_SHADOW: shadow=0x11
[   12.345679]   Unpacked: memcgid=0 nid=4 eviction=0x0 workingset=0
[   12.345680]   NODE_DATA(4)=0000000000000000
[   12.345681] *** BUG: INVALID NODE ID 4! ***
[   12.345682] BUG: kernel NULL pointer dereference, address: 0000000000000018
[   12.345683] Call Trace:
[   12.345684]  lru_gen_test_recent+0x34/0x1b0
[   12.345685]  workingset_refault+0x123/0x2b0

2. Second crash - invalid node_id=11:

[   15.678901] UNPACK_SHADOW: shadow=0x2d
[   15.678902]   Unpacked: memcgid=0 nid=11 eviction=0x0 workingset=0
[   15.678903]   NODE_DATA(11)=0000000000000000
[   15.678904] *** BUG: INVALID NODE ID 11! ***
[   15.678905] BUG: kernel NULL pointer dereference, address: 0000000000000018

CRITICAL FINDING:
During the same run, ALL newly created shadows had valid node_id=0:

[   12.123456] LRU_GEN_EVICTION: min_seq=0x0 refs=0 tier=0
[   12.123457]   token=0x0
[   12.123458] PACK_SHADOW: memcgid=2 node_id=0 eviction=0x0
[   12.123459]   Final packed shadow=0x201

[   12.234567] PACK_SHADOW: memcgid=2 node_id=0 eviction=0x0
[   12.234568]   Final packed shadow=0x201

[   12.345678] PACK_SHADOW: memcgid=2 node_id=0 eviction=0x0
[   12.345679]   Final packed shadow=0x201

Notice: We UNPACK shadows 0x11 and 0x2d (with invalid node IDs), but we
NEVER see them being PACKED during this instrumented run. This indicates
these invalid shadows are stale entries from before debug was applied.

ANALYSIS:

The invalid shadows appear to be:
- Persisting in page cache/swap from previous runs

We cannot confirm if:
- The reproducer actively creates these invalid shadows, OR
- It only triggers refaults on pre-existing invalid shadows

PROPOSED SOLUTION:

Given this uncertainty, we need both prevention AND remediation:

1. In pack_shadow() - prevent new invalid shadows:
   if (pgdat->node_id >= MAX_NUMNODES || !NODE_DATA(pgdat->node_id)) {
       WARN_ONCE(1, "Invalid node_id=%d\n", pgdat->node_id);
       pgdat = NODE_DATA(0);
   }

2. In unpack_shadow() - handle existing invalid shadows:
   if (nid >= MAX_NUMNODES || !NODE_DATA(nid)) {
       pr_warn_once("Invalid shadow node_id=%d, using node 0\n", nid);
       nid = 0;
   }

The unpack_shadow() fix is critical for handling legacy invalid shadows
that already exist in the wild.

I can investigate further to identify the creation path if needed. Please
let me know if you'd like me to:
- Submit the defensive fix (unpack_shadow validation) first
- Continue investigating the creation path
- Or both in parallel

Thanks,
Deepanshu