linux-kernel - Re: workqueue lockup debug

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <9cfe9161-2ada-4fa0-bcdf-822401708fbf@oracle.com>
Date: Fri, 25 Oct 2024 10:28:24 +0100
From: John Garry <john.g.garry@...cle.com>
To: Tejun Heo <tj@...nel.org>
Cc: jiangshanlai@...il.com, mingo@...hat.com, peterz@...radead.org,
        juri.lelli@...hat.com, jack@...e.cz, david@...morbit.com,
        akpm@...ux-foundation.org, linux-kernel@...r.kernel.org
Subject: Re: workqueue lockup debug

On 24/10/2024 19:56, Tejun Heo wrote:

Hi Tejun,

> 
> On Thu, Oct 24, 2024 at 04:49:40PM +0100, John Garry wrote:
>> Hi workqueue and scheduler maintainers,
>>
>> As reported in https://urldefense.com/v3/__https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-ab6d-7ed9a4f1f9c0@oracle.com/T/*m506b9edb1340cdddd87c6d14d20222ca8d7e8796__;Iw!!ACWV5N9M2RV99hQ!Jn5gFi_KIzLb-7uCOvHRIcf9Zd4RyHCTnm15Lo8u0KsgaMmwEPZHH2m4l-w4Fr-b6aRICoRWb5DU$ ,
>> I am experiencing a workqueue lockup for v6.12-rcX.
>>
>> At the point it occurs, the system becomes unresponsive and I cannot bring
>> it back to life.
>>
>> Enabling /proc/sys/kernel/softlockup_all_cpu_backtrace does not give
>> anything extra in the way of debug. All I get is something like this:
>>
>> Message from syslogd@...rry-atomic-write-exp-e4-8-instance-20231214-1221 at
>> Oct 24 15:34:02 ...
>>   kernel:watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mysqld:14352]
>>
>> Message from syslogd@...rry-atomic-write-exp-e4-8-instance-20231214-1221 at
>> Oct 24 15:34:02 ...
>>   kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck
>> for 30s!
>>
>> Message from syslogd@...rry-atomic-write-exp-e4-8-instance-20231214-1221 at
>> Oct 24 15:34:02 ...
>>   kernel:BUG: workqueue lockup - pool cpus=31 node=0 flags=0x0 nice=0 stuck
>> for 49s!
>> ^C
>>
>> Can you advise on a robust method to get some debug from this system?
>>
>> Maybe this is a scheduler issue, as Dave mentioned in that same thread.
> 
> So, while the symptom *could* be a workqueue bug, wq lockups are usually
> more a side-effect of some other problem.

Sure

> All that wq is saying is that some
> work item hasn't made forward progress for too long. Given that there are
> soft lockup messages, which can't be caused by wq lockups but can definitely
> trigger wq lockups, I'd start with debugging what's happening with the CPU
> that triggered soft lockup. 

ok

> Soft lockups are nice in that they're
> synchronous failures, so probably start with the backtrace?
> 

I'm not getting any backtrace, which is not helping the issue.

But I'll continue to look.

Thanks,
John