linux-kernel - Re: BUG: Stall on adding/removing wokers into workqueue pool

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ZyLEYMZP2SiVGeSD@slm.duckdns.org>
Date: Wed, 30 Oct 2024 13:42:24 -1000
From: Tejun Heo <tj@...nel.org>
To: Tim Chen <tim.c.chen@...ux.intel.com>
Cc: Lai Jiangshan <jiangshanlai@...il.com>, linux-kernel@...r.kernel.org,
	Thomas Gleixner <tglx@...utronix.de>,
	Doug Nelson <doug.nelson@...el.com>, bp@...en8.de,
	dave.hansen@...ux.intel.com, hpa@...or.com, mingo@...hat.com,
	syzkaller-bugs@...glegroups.com, x86@...nel.org
Subject: Re: BUG: Stall on adding/removing wokers into workqueue pool

Hello, Tim.

On Tue, Oct 29, 2024 at 03:03:33PM -0700, Tim Chen wrote:
> Hi Tejun,
> 
> Forwarding this task hung seen by my colleague Doug Nelson. He tested
> the 6.12-rc4 kernel with an OLTP workload running on a 2 socket with
> Granite Rapids CPU that has 86 cores per socket. The traces 
> seem to indicate that the acquisition 
> of wq_pool_attach_mutex stalled in idle_cull_fn() when removing worker from
> the pool. Doug hit this problem occasionally in his tests.
> 
> Searching through the bug reports, there's a similar report by szybot on the
> 6.12-rc2 kernel. Szybot reported similar task hung when attaching workers to
> the pool: https://lore.kernel.org/all/6706c4ba.050a0220.1139e6.0008.GAE@google.com/T/
> So we suspect that the problem is not GNR CPU specific.
> 
> Wonder if this problem is a known one?

First time I see it. The trace doesn't show who's holding the mutex. There
doesn't seem to be any place where that mutex should leak at least on a
glance, so hopefully it shouldn't be too difficult to find who's holding it.
Can you trigger sysrq-d and sysrq-t and post the output?

Thanks.

-- 
tejun