linux-kernel - Re: Debugging lost task in wait_task_inactive() when delivering signal (6.12)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAGis_TUp9_V-kBn9CF55f08NVR+Bx3iyP=O=+PH0QAf73eGY2Q@mail.gmail.com>
Date: Fri, 19 Sep 2025 16:16:32 +0100
From: Matt Fleming <mfleming@...udflare.com>
To: Oleg Nesterov <oleg@...hat.com>
Cc: Peter Zijlstra <peterz@...radead.org>, John Stultz <jstultz@...gle.com>, 
	kernel-team <kernel-team@...udflare.com>, LKML <linux-kernel@...r.kernel.org>, 
	Chris Arges <carges@...udflare.com>
Subject: Re: Debugging lost task in wait_task_inactive() when delivering
 signal (6.12)

On Fri, 19 Sept 2025 at 15:39, 'Oleg Nesterov' via kernel-team
<kernel-team@...udflare.com> wrote:
>
> Hi Matt,
>
> On 09/19, Matt Fleming wrote:
> >
> > Hi there,
> >
> > We're running into an intermittent issue where tasks end up in a state with
> > p->on_rq=1 and p->se.on_rq=0 when delivering a fatal signal to a thread
> > group. The thread handling the signal sits in wait_task_inactive() after
> > sending a SIGKILL to all other threads, most of which pass through
> > coredump_task_exit() just fine, but occasionally one thread calls into
> > coredump_task_exit()->schedule() and never comes back because of the above
> > state.
>
> I guess you mean coredump_wait() -> wait_task_inactive() ... Sorry I have no
> clue at least right now... And I don't see any problem in coredump_task_exit().
>
> Stupid question. Any chance you can reproduce, figure out the pid of that
> sub-thread which fools wait_task_inactive() and, say, do
> "cat /proc/pid-of-that-thread/stack" ? Or any other info, everything can
> help. Crash dump? Yes, you have already mentioned this is hard-to-reproduce :(

I do have some info. The callstack for the lost thread is:

Call Trace:
 <TASK>
 __schedule+0x4fb/0xbf0
 ? srso_return_thunk+0x5/0x5f
 schedule+0x27/0xf0
 do_exit+0xdd/0xaa0
 ? __pfx_futex_wake_mark+0x10/0x10
 do_group_exit+0x30/0x80
 get_signal+0x81e/0x860
 ? srso_return_thunk+0x5/0x5f
 ? futex_wake+0x177/0x1a0
 arch_do_signal_or_restart+0x2e/0x1f0
 ? srso_return_thunk+0x5/0x5f
 ? srso_return_thunk+0x5/0x5f
 ? __x64_sys_futex+0x10c/0x1d0
 syscall_exit_to_user_mode+0xa5/0x130
 do_syscall_64+0x57/0x110
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

do_exit+0xdd is here in coredump_task_wait():

                for (;;) {
                        set_current_state(TASK_IDLE|TASK_FREEZABLE);
                        if (!self.task) /* see coredump_finish() */
                                break;
                        schedule();
                }

i.e. the task calls schedule() and never comes back. The waiting task
sees p->on_rq=1 for this lost thread and spins in wait_task_inactive()
forever.

I have been able to use drgn to inspect the live system when the error
occurred so if you have specific things you want me to look at the
task_struct state I can try that. Or if you would like me to grab some
diagnostic info whenever the issue crops up again, just let me know.