| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAE4VaGBcXDUn+i4GuZyod_g_uKs54641-D-O14_QCR5w9pddbA@mail.gmail.com>
Date: Fri, 4 Jul 2025 01:50:13 +0200
From: Jirka Hladky <jhladky@...hat.com>
To: "Aithal, Srikanth" <sraithal@....com>
Cc: "Chen, Yu C" <yu.c.chen@...el.com>, Abhigyan ghosh <zscript.team.zs@...il.com>,
linux-kernel@...r.kernel.org, Suneeth D <Suneeth.D@....com>
Subject: Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL
pointer dereference)
Let me close this loop.
All debug messages I have collected were connected with the task exit.
Thank you
Jirka
On Wed, Jul 2, 2025 at 3:24 PM Jirka Hladky <jhladky@...hat.com> wrote:
>
> Thank you, Aithal!
>
> Chenyu, here is another debug log - the flag is essentially the same
> as before, except that PF_NOFREEZE is also set, so it points to the
> same race condition (task exit).
>
> This happened on an AMD Bergamo 9754 single socket server.
>
> cat /sys/kernel/debug/tracing/trace
> # tracer: nop
> #
> # entries-in-buffer/entries-written: 1/1 #P:256
> #
> # _-----=> irqs-off/BH-disabled
> # / _----=> need-resched
> # | / _---=> hardirq/softirq
> # || / _--=> preempt-depth
> # ||| / _-=> migrate-disable
> # |||| / delay
> # TASK-PID CPU# ||||| TIMESTAMP FUNCTION
> # | | | ||||| | |
> migration/96-599 [096] dN.5. 84281.647548: __migrate_swap_task:
> !! (2886754 sysbench-binary) flags=40804c
>
> On Wed, Jul 2, 2025 at 2:19 PM Aithal, Srikanth <sraithal@....com> wrote:
> >
> > On 7/2/2025 5:23 PM, Jirka Hladky wrote:
> > > Hi Aithal,
> > >
> > > thank you for the reproducer.
> > >
> > > With the patched kernel, after running a reproducer, please do
> > >
> > > cat /sys/kernel/debug/tracing/trace
> > >
> > > If the counter is nonzero like this:
> > > entries-in-buffer/entries-written: 1/1
> > >
> > > please post the `cat /sys/kernel/debug/tracing/trace` here.
> > >
> > > It would be helpful to check if there are any other race conditions
> > > besides the task exit.
> > >
> > > Thank you!
> > > Jirka
> > >
> >
> >
> > None of the four machines running our workload [with the debug patch]
> > showed any entries.
> > Entries in Buffer/Entries Written: 0/0
> >
> >
> > > On Wed, Jul 2, 2025 at 1:17 PM Chen, Yu C <yu.c.chen@...el.com> wrote:
> > >>
> > >> On 7/2/2025 3:36 PM, Aithal, Srikanth wrote:
> > >>>
> > >>>
> > >>> On 7/2/2025 10:02 AM, Aithal, Srikanth wrote:
> > >>>> On 6/27/2025 1:03 PM, Chen, Yu C wrote:
> > >>>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> > >>>>>> Hi Jirka,
> > >>>>>>
> > >>>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> > >>>>>>> Hi Chen and all,
> > >>>>>>>
> > >>>>>>> we have now verified that the following commit causes a kernel panic
> > >>>>>>> discussed in this thread:
> > >>>>>>>
> > >>>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> > >>>>>>>
> > >>>>>>> Reverting this commit fixes the issue.
> > >>>>>>>
> > >>>>>>> I'm happy to help debug this further or test a proposed fix.
> > >>>>>>>
> > >>>>>>
> > >>>>>> Thanks very much for your report, it seems that there is a
> > >>>>>> race condition that when the swap task candidate was chosen,
> > >>>>>> but its mm_struct get released due to task exit, then later
> > >>>>>> when doing the task swaping, the p->mm is NULL which caused
> > >>>>>> the problem:
> > >>>>>>
> > >>>>>> CPU0 CPU1
> > >>>>>> :
> > >>>>>> ...
> > >>>>>> task_numa_migrate
> > >>>>>> task_numa_find_cpu
> > >>>>>> task_numa_compare
> > >>>>>> # a normal task p is chosen
> > >>>>>> env->best_task = p
> > >>>>>>
> > >>>>>> # p exit:
> > >>>>>> exit_signals(p);
> > >>>>>> p->flags |= PF_EXITING
> > >>>>>> exit_mm
> > >>>>>> p->mm = NULL;
> > >>>>>>
> > >>>>>> migrate_swap_stop
> > >>>>>> __migrate_swap_task((arg->src_task, arg->dst_cpu)
> > >>>>>> count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> > >>>>>>
> > >>>>>> Could you please help check if the following debug patch works,
> > >>>>>
> > >>>>> Attached the patch:
> > >>>>>
> > >>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > >>>>> index 8988d38d46a3..82fc966b390c 100644
> > >>>>> --- a/kernel/sched/core.c
> > >>>>> +++ b/kernel/sched/core.c
> > >>>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct
> > >>>>> task_struct *p, int cpu)
> > >>>>> {
> > >>>>> __schedstat_inc(p->stats.numa_task_swapped);
> > >>>>> count_vm_numa_event(NUMA_TASK_SWAP);
> > >>>>> - count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> > >>>>> + if (unlikely(!p->mm)) {
> > >>>>> + trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> > >>>>> + p->flags);
> > >>>>> + } else {
> > >>>>> + count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> > >>>>> + }
> > >>>>>
> > >>>>> if (task_on_rq_queued(p)) {
> > >>>>> struct rq *src_rq, *dst_rq;
> > >>>>
> > >>>> I was encountering the same issue as mentioned earlier in this thread,
> > >>>> which has been recurring in our daily linux-next CI builds within our
> > >>>> virtualization CI stream where we observed this BUG appearing randomly
> > >>>> during the runs.
> > >>>>
> > >>>> Additionally, we were able to reproduce this issue while running the
> > >>>> autonuma benchmark. As mentioned earlier, the BUG would occur randomly
> > >>>> across iterations, typically between the 5th and 10th iterations.
> > >>>>
> > >>>> We consistently encountered this issue up to the 6.16.0-rc4-
> > >>>> next-20250630 build [https://git.kernel.org/pub/scm/linux/kernel/git/
> > >>>> next/linux-next.git].
> > >>>>
> > >>>> After applying the aforementioned patch ontop of next-20250630 build,
> > >>>> I tested it in our virtualization CI and with the autonuma benchmark
> > >>>> reproducer described below, and the issue no longer occurred. The
> > >>>> patch appears to have resolved the reported problem.
> > >>>>
> > >>>>
> > >>>> git clone https://github.com/pholasek/autonuma-benchmark.git
> > >>>> cd autonuma-benchmark
> > >>>> for i in $(seq 1 80); do bash ./start_bench.sh -s -t; done
> > >>>> Note: The server running the autonuma-benchmark must have at least two
> > >>>> nodes.
> > >>>>
> > >>>> If the provided fix is final, please feel free to include the
> > >>>> following Tested-by tag:
> > >>>>
> > >>>> Tested-by: Srikanth Aithal <Srikanth.Aithal@....com>
> > >>>> Tested-by: Suneeth D <Suneeth.D@....com>
> > >>>>
> > >>>
> > >>>
> > >>> I apologize for overlooking that this was a debug patch. With this debug
> > >>> patch applied on top of 6.16.0-rc4-next-20250630, I am unable to
> > >>> reproduce the issue. However, when I revert this debug patch, I
> > >>> encounter the issue again on 6.16.0-rc4-next-20250630.
> > >>>
> > >>>
> > >>
> > >> Thank you Aithal, this is a debug + fix patch. And Jirka has confirmed
> > >> that this helps somehow. I'll post a formal one and Cc you.
> > >>
> > >> Thanks,
> > >> Chenyu
> > >>
> > >
> > >
> >
>
>
> --
> -Jirka
--
-Jirka
Powered by blists - more mailing lists