| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAE4VaGDBoqa-vj7MUuK12DV=XUNUByiNpzj4c=bse+zBSRBJTg@mail.gmail.com>
Date: Wed, 2 Jul 2025 13:43:52 +0200
From: Jirka Hladky <jhladky@...hat.com>
To: "Chen, Yu C" <yu.c.chen@...el.com>
Cc: Libo Chen <libo.chen@...cle.com>, Abhigyan ghosh <zscript.team.zs@...il.com>,
linux-kernel@...r.kernel.org, Chen Yu <yu.chen.surf@...mail.com>,
Philip Auld <pauld@...hat.com>, "jstancek@...hat.com" <jstancek@...hat.com>
Subject: Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL
pointer dereference)
> Thanks Jirka. So this race condition is related to task exit.
Yes! I'm glad we've found it! :-)
> The flag 0x40004c is
> PF_EXITING PF_POSTCOREDUMP PF_FORKNOEXEC PF_RANDOMIZE
> according to latest
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/sched.h?h=v6.16-rc4
Oh, my bad! Thank you for the correct pointer.
On Wed, Jul 2, 2025 at 1:14 PM Chen, Yu C <yu.c.chen@...el.com> wrote:
>
> On 7/2/2025 5:05 AM, Jirka Hladky wrote:
> > Hi Chenyu,
> >
> > I have the first
> >
> > if (unlikely(!p->mm)) {
> > trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm, p->flags);
> >
> > log from dual socket AMD Milanx 7573X server:
> >
> > [74615.841401] # entries-in-buffer/entries-written: 1/1 #P:128
> > [74615.841401] #
> > [74615.841401] # _-----=> irqs-off/BH-disabled
> > [74615.841401] # / _----=> need-resched
> > [74615.841401] # | / _---=> hardirq/softirq
> > [74615.841401] # || / _--=> preempt-depth
> > [74615.841401] # ||| / _-=> migrate-disable
> > [74615.841401] # |||| / delay
> > [74615.841401] # TASK-PID CPU# ||||| TIMESTAMP FUNCTION
> > [74615.841401] # | | | ||||| | |
> > [74615.841401] migration/6-54 [006] d..5. 58590.368364:
> > __migrate_swap_task: !! (2939199 stress-ng-sem) flags=40004c
> >
> > debug code caught a task with p->mm == NULL during __migrate_swap_task().
> >
> > Trace shows:
> > * It was a stress-ng-sem process (PID 2939199),
> > * That process was in the process of exiting (PF_EXITING | PF_EXITPIDONE),
>
> Thanks Jirka. So this race condition is related to task exit.
>
> > * And was a workqueue worker (PF_WQ_WORKER),
> > * It explains why p->mm had already been freed
> >
> > I'm curious as to why the command is stress-ng-sem, and the flag
> > indicates it is a workqueue worker.
>
> The flag 0x40004c is
> PF_EXITING PF_POSTCOREDUMP PF_FORKNOEXEC PF_RANDOMIZE
> according to latest
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/sched.h?h=v6.16-rc4
>
> thanks,
> Chenyu>
> > More jobs are still running. I will keep an eye on them and report the
> > other cases as they finish.
> >
> > Jirka
> >
> > On Tue, Jul 1, 2025 at 5:10 PM Jirka Hladky <jhladky@...hat.com> wrote:
> >>
> >> Hi Chenyu and Libo,
> >>
> >> I agree, let's wait for the results. Hopefully, we can fix the root
> >> cause based on the debug messages.
> >>
> >> There was a problem with the infrastructure over the weekend, so I had
> >> to restart the jobs yesterday. I should have results tomorrow.
> >>
> >> Stay tuned.
> >> Jirka
> >>
> >>
> >> On Tue, Jul 1, 2025 at 4:59 PM Chen, Yu C <yu.c.chen@...el.com> wrote:
> >>>
> >>> Hi Libo,
> >>>
> >>> On 7/1/2025 3:32 PM, Libo Chen wrote:
> >>>> Hi Chenyu,
> >>>>
> >>>> On 6/27/25 00:33, Chen, Yu C wrote:
> >>>>> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> >>>>>> Hi Jirka,
> >>>>>>
> >>>>>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> >>>>>>> Hi Chen and all,
> >>>>>>>
> >>>>>>> we have now verified that the following commit causes a kernel panic
> >>>>>>> discussed in this thread:
> >>>>>>>
> >>>>>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> >>>>>>>
> >>>>>>> Reverting this commit fixes the issue.
> >>>>>>>
> >>>>>>> I'm happy to help debug this further or test a proposed fix.
> >>>>>>>
> >>>>>>
> >>>>>> Thanks very much for your report, it seems that there is a
> >>>>>> race condition that when the swap task candidate was chosen,
> >>>>>> but its mm_struct get released due to task exit, then later
> >>>>>> when doing the task swaping, the p->mm is NULL which caused
> >>>>>> the problem:
> >>>>>>
> >>>>>> CPU0 CPU1
> >>>>>> :
> >>>>>> ...
> >>>>>> task_numa_migrate
> >>>>>> task_numa_find_cpu
> >>>>>> task_numa_compare
> >>>>>> # a normal task p is chosen
> >>>>>> env->best_task = p
> >>>>>>
> >>>>>> # p exit:
> >>>>>> exit_signals(p);
> >>>>>> p->flags |= PF_EXITING
> >>>>>> exit_mm
> >>>>>> p->mm = NULL;
> >>>>>>
> >>>>>> migrate_swap_stop
> >>>>>> __migrate_swap_task((arg->src_task, arg->dst_cpu)
> >>>>>> count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> >>>>>>
> >>>>>> Could you please help check if the following debug patch works,
> >>>>>
> >>>>> Attached the patch:
> >>>>>
> >>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>>>> index 8988d38d46a3..82fc966b390c 100644
> >>>>> --- a/kernel/sched/core.c
> >>>>> +++ b/kernel/sched/core.c
> >>>>> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
> >>>>> {
> >>>>> __schedstat_inc(p->stats.numa_task_swapped);
> >>>>> count_vm_numa_event(NUMA_TASK_SWAP);
> >>>>> - count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >>>>> + if (unlikely(!p->mm)) {
> >>>>
> >>>> I am starting to wonder if we should keep this check and add a big fat warning
> >>>> like "there is a bug here, please report it!" rather than brick the kernel.
> >>>> A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
> >>>> so it's a tradeoff I guess. What do you think?
> >>>>
> >>>
> >>> I agree with you that adding something like WARN_ON_ONCE()
> >>> could help, and it seems that the kernel panic is not expected
> >>> in this scenario because this feature is a statistic calculation
> >>> rather than the critical logic. Maybe waiting for Jirka's feedback
> >>> to decide the next step.
> >>>
> >>> thanks,
> >>> Chenyu
> >>>
> >>>
> >>>
> >>>>
> >>>> Thanks,
> >>>> Libo
> >>>>
> >>>>> + trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> >>>>> + p->flags);
> >>>>> + } else {
> >>>>> + count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> >>>>> + }
> >>>>>
> >>>>> if (task_on_rq_queued(p)) {
> >>>>> struct rq *src_rq, *dst_rq;
> >>>>
> >>>
> >>
> >>
> >> --
> >> -Jirka
> >
> >
> >
>
--
-Jirka
Powered by blists - more mailing lists