linux-kernel - Re: v6.12-rc workqueue lockups

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20241023201314.hqfewgr6lej6f7df@quack3>
Date: Wed, 23 Oct 2024 22:13:14 +0200
From: Jan Kara <jack@...e.cz>
To: John Garry <john.g.garry@...cle.com>
Cc: linux-kernel@...r.kernel.org, viro@...iv.linux.org.uk,
	brauner@...nel.org, jack@...e.cz, linux-fsdevel@...r.kernel.org
Subject: Re: v6.12-rc workqueue lockups


Hi!

On Wed 23-10-24 11:19:24, John Garry wrote:
> I have been seeing lockups reliably occur on v6.12-rc1, 3, 4 and linus'
> master branch:
> 
> Message from syslogd@...rry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:07:15 ...
>  kernel:watchdog: BUG: soft lockup - CPU#12 stuck for 26s! [khugepaged:154]
> 
> Message from syslogd@...rry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
>  kernel:BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck
> for 44s!
> 
> Message from syslogd@...rry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
>  kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck
> for 35s!
> 
> Message from syslogd@...rry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
>  kernel:BUG: workqueue lockup - pool cpus=10 node=0 flags=0x0 nice=0 stuck
> for 33s!
> 
> This is while doing some server MySQL performance testing. v6.11 has no such
> issue.
> 
> I added some debug, and we seem to be spending a lot of time in FS
> writeback, specifically wb_workfn() -> wb_do_writeback() - ring any bells?

Thanks for report. This doesn't remind me of anything but checking the
writeback changes we have merged 532980cb1bf ("inode: port __I_SYNC to var
event") which could have odd consequences if we made mistake somewhere...
It probably won't be easy to revert but you could try whether the problem
reproduces before / after this commit.

								Honza
-- 
Jan Kara <jack@...e.com>
SUSE Labs, CR