Message-ID: <CAE4VaGAak=U-oLwXvPZsiNRnRvNL_ROKL8AJCSkCm+zPOjf2qQ@mail.gmail.com>
Date: Thu, 26 Jun 2025 23:46:20 +0200
From: Jirka Hladky <jhladky@...hat.com>
To: Abhigyan ghosh <zscript.team.zs@...il.com>, yu.c.chen@...el.com
Cc: linux-kernel@...r.kernel.org
Subject: Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL
 pointer dereference)

Hi Chen and all,

we have now verified that the following commit causes a kernel panic
discussed in this thread:

ad6b26b6a0a79 sched/numa: add statistics of numa balance task

Reverting this commit fixes the issue.

I'm happy to help debug this further or test a proposed fix.

Thank you!
Jirka

On Wed, Jun 18, 2025 at 1:34 PM Jirka Hladky <jhladky@...hat.com> wrote:
>
> Hi Abhigyan,
>
> The testing is done on bare metal. The kernel panics occur after
> several hours of benchmarking.
>
> Out of 20 servers, the problem has occurred on 6 of them:
> intel-sapphire-rapids-gold-6448y-2s
> intel-emerald-rapids-platinum-8558-2s
> amd-epyc5-turin-9655p-1s
> amd-epyc4-zen4c-bergamo-9754-1s
> amd-epyc3-milan-7713-2s
> intel-skylake-2s
>
> The number in the name is the CPU model. 1s: single socket, 2s: dual socket.
>
> We were not able to find a clear pattern. It appears to be a race
> condition of some kind.
>
> We run various performance benchmarks, including Linpack, Stream, NAS
> (https://www.nas.nasa.gov/software/npb.html), and Stress-ng. Testing
> is conducted with various thread counts and settings. All benchmarks
> together are running ~24 hours. One benchmark takes ~4 hours. Please
> also note that we repeat the benchmarks to collect performance
> statistics. In many cases, kernel panic has occurred when the
> benchmark was repeated.
>
> Crash occurred while running these tests:
> Stress_ng: Starting test 'fork' (#29 out of 41), number of threads 32,
> iteration 1 out of 5
> SPECjbb2005: Starting DEFAULT run with 4 SPECJBB2005 instances, each
> with 24 warehouses, iteration 2 out of 3
> Stress_ng: test 'sem' (#30 out of 41), number of threads 24, iteration
> 2 out of 5
> Stress_ng: test 'sem' (#30 out of 41), number of threads 64, iteration
> 4 out of 5
> SPECjbb2005: SINGLE run with 1 SPECJBB2005 instances, each with 128
> warehouses, iteration 2 out of 3
> Linpack: Benchmark-utils/linpackd, iteration 3, testType affinityRun,
> number of threads 128
> NAS: NPB_sources/bin/is.D.x
>
> There is no clear benchmark triggering the kernel panic. Looping
> Stress_ng's sem test looks, however, like it's worth trying.
>
> I hope this helps. Please let me know if there's anything I can help
> with to pinpoint the problem.
>
> Thanks
> Jirka
>
>
> On Wed, Jun 18, 2025 at 7:19 AM Abhigyan ghosh
> <zscript.team.zs@...il.com> wrote:
> >
> > Hi Jirka,
> >
> > Thanks for the detailed report.
> >
> > I'm curious about the specific setup in which this panic was triggered. Could you share more about the exact configuration or parameters you used for running `stress-ng` or Linpack? For instance:
> >
> > - How many threads/cores were used?
> > - Was it running inside a VM, container, or bare-metal?
> > - Was this under any thermal throttling or power-saving mode?
> >
> > I'd like to try reproducing it locally to study the failure further.
> >
> > Best regards,
> > Abhigyan Ghosh
> >
> > On 18 June 2025 1:35:30 am IST, Jirka Hladky <jhladky@...hat.com> wrote:
> > >Hi all,
> > >
> > >I’ve encountered a reproducible kernel panic on 6.16-rc1 and 6.16-rc2
> > >involving a NULL pointer dereference in `__migrate_swap_task()` during
> > >CPU migration. This occurred on various AMD and Intel systems while
> > >running a CPU-intensive workload (Linpack, Stress_ng - it's not
> > >specific to a benchmark).
> > >
> > >Full trace below:
> > >---
> > >BUG: kernel NULL pointer dereference, address: 00000000000004c8
> > >#PF: supervisor read access in kernel mode
> > >#PF: error_code(0x0000) - not-present page
> > >PGD 4078b99067 P4D 4078b99067 PUD 0
> > >Oops: Oops: 0000 [#1] SMP NOPTI
> > >CPU: 74 UID: 0 PID: 466 Comm: migration/74 Kdump: loaded Not tainted
> > >6.16.0-0.rc2.24.eln149.x86_64 #1 PREEMPT(lazy)
> > >Hardware name: GIGABYTE R182-Z91-00/MZ92-FS0-00, BIOS M07 09/03/2021
> > >Stopper: multi_cpu_stop+0x0/0x130 <- migrate_swap+0xa7/0x120
> > >RIP: 0010:__migrate_swap_task+0x2f/0x170
> > >Code: 41 55 4c 63 ee 41 54 55 53 48 89 fb 48 83 87 a0 04 00 00 01 65
> > >48 ff 05 e7 14 dd 02 48 8b af 50 0a 00 00 66 90 e8 61 93 07 00 <48> 8b
> > >bd c8 04 00 00 e8 85 11 35 00 48 85 c0 74 12 ba 01 00 00 00
> > >RSP: 0018:ffffce79cd90bdd0 EFLAGS: 00010002
> > >RAX: 0000000000000001 RBX: ffff8e9c7290d1c0 RCX: 0000000000000000
> > >RDX: ffff8e9c71e83680 RSI: 000000000000001b RDI: ffff8e9c7290d1c0
> > >RBP: 0000000000000000 R08: 00056e36392913e7 R09: 00000000002ab980
> > >R10: ffff8eac2fcb13c0 R11: ffff8e9c77997410 R12: ffff8e7c2fcf12c0
> > >R13: 000000000000001b R14: ffff8eac71eda944 R15: ffff8eac71eda944
> > >FS:  0000000000000000(0000) GS:ffff8eac9db4a000(0000) knlGS:0000000000000000
> > >CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > >CR2: 00000000000004c8 CR3: 0000003072388003 CR4: 0000000000f70ef0
> > >PKRU: 55555554
> > >Call Trace:
> > > <TASK>
> > > migrate_swap_stop+0xe8/0x190
> > > multi_cpu_stop+0xf3/0x130
> > > ? __pfx_multi_cpu_stop+0x10/0x10
> > > cpu_stopper_thread+0x97/0x140
> > > ? __pfx_smpboot_thread_fn+0x10/0x10
> > > smpboot_thread_fn+0xf3/0x220
> > > kthread+0xfc/0x240
> > > ? __pfx_kthread+0x10/0x10
> > > ? __pfx_kthread+0x10/0x10
> > > ret_from_fork+0xf0/0x110
> > > ? __pfx_kthread+0x10/0x10
> > > ret_from_fork_asm+0x1a/0x30
> > > </TASK>
> > >---
> > >
> > >**Kernel Version:**
> > >6.16.0-0.rc2.24.eln149.x86_64 (Fedora rawhide)
> > >https://koji.fedoraproject.org/koji/buildinfo?buildID=2732950
> > >
> > >**Reproducibility:**
> > >Happened multiple times during routine CPU-intensive operations. It
> > >happens with various benchmarks (Stress_ng, Linpack) after several
> > >hours of performance testing. `migration/*` kernel threads hit a NULL
> > >dereference in `__migrate_swap_task`.
> > >
> > >**System Info:**
> > >- Platform: GIGABYTE R182-Z91-00 (dual socket EPYC)
> > >- BIOS: M07 09/03/2021
> > >- Config: Based on Fedora’s debug kernel (`PREEMPT(lazy)`)
> > >
> > >**Crash Cause (tentative):**
> > >NULL dereference at offset `0x4c8` from a task struct pointer in
> > >`__migrate_swap_task`. Possibly an uninitialized or freed
> > >`task_struct` field.
> > >
> > >Please let me know if you’d like me to test a patch or if you need
> > >more details.
> > >
> > >Thanks,
> > >Jirka
> > >
> > >
> >
> > aghosh
> >
>
>
> --
> -Jirka



-- 
-Jirka

Powered by blists - more mailing lists