linux-kernel - RE: Regression on linux-next (next-20240709)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <SJ1PR11MB612986837520A40CF484A1CCB9A62@SJ1PR11MB6129.namprd11.prod.outlook.com>
Date: Fri, 12 Jul 2024 05:52:07 +0000
From: "Borah, Chaitanya Kumar" <chaitanya.kumar.borah@...el.com>
To: "jiangshan.ljs@...group.com" <jiangshan.ljs@...group.com>
CC: "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>, "Nikula,
 Jani" <jani.nikula@...el.com>, "Saarinen, Jani" <jani.saarinen@...el.com>,
	"Kurmi, Suresh Kumar" <suresh.kumar.kurmi@...el.com>,
	"intel-gfx@...ts.freedesktop.org" <intel-gfx@...ts.freedesktop.org>
Subject: RE: Regression on linux-next (next-20240709)

[+intel-gfx]

Hello Jiangshan,

Hope you are doing well. I am Chaitanya from the linux graphics team in Intel.

This mail is regarding a regression we are seeing in our CI runs[1] on linux-next repository.

Since the version next-20240709 [2], we are seeing the following regression during boot up.

`````````````````````````````````````````````````````````````````````````````````
<4>[    0.813758] ============================================
<4>[    0.813758] WARNING: possible recursive locking detected
<4>[    0.813758] 6.10.0-rc7-next-20240709-next-20240709-g82d01fe6ee52+ #1 Not tainted
<4>[    0.813758] --------------------------------------------
<4>[    0.813758] swapper/0/1 is trying to acquire lock:
<4>[    0.813758] ffffffff8264adb0 (cpu_hotplug_lock){++++}-{0:0}, at: alloc_workqueue+0x31a/0x820
<4>[    0.813758] 
                  but task is already holding lock:
<4>[    0.813758] ffffffff8264adb0 (cpu_hotplug_lock){++++}-{0:0}, at: padata_alloc+0x56/0x140
<4>[    0.813758] 
                  other info that might help us debug this:
<4>[    0.813758]  Possible unsafe locking scenario:

<4>[    0.813758]        CPU0
<4>[    0.813758]        ----
<4>[    0.813758]   lock(cpu_hotplug_lock);
<4>[    0.813758]   lock(cpu_hotplug_lock);
<4>[    0.813758] 
                   *** DEADLOCK ***

<4>[    0.813758]  May be due to missing lock nesting notation

<4>[    0.813758] 1 lock held by swapper/0/1:
<4>[    0.813758]  #0: ffffffff8264adb0 (cpu_hotplug_lock){++++}-{0:0}, at: padata_alloc+0x56/0x140
<4>[    0.813758] 
                  stack backtrace:
<4>[    0.813758] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.10.0-rc7-next-20240709-next-20240709-g82d01fe6ee52+ #1
<4>[    0.813758] Hardware name: ASUS System Product Name/PRIME Z790-P WIFI, BIOS 0812 02/24/2023
<4>[    0.813758] Call Trace:
<4>[    0.813758]  <TASK>
<4>[    0.813758]  dump_stack_lvl+0x82/0xd0
<4>[    0.813758]  print_deadlock_bug+0x259/0x390
<4>[    0.813758]  __lock_acquire+0x733/0x1f60
<4>[    0.813758]  lock_acquire+0xd1/0x2f0
<4>[    0.813758]  ? alloc_workqueue+0x31a/0x820
<4>[    0.813758]  cpus_read_lock+0x3b/0xe0
<4>[    0.813758]  ? alloc_workqueue+0x31a/0x820
<4>[    0.813758]  alloc_workqueue+0x31a/0x820
<4>[    0.813758]  ? __pfx_pcrypt_init+0x10/0x10
<4>[    0.813758]  padata_alloc+0x6f/0x140
<4>[    0.813758]  pcrypt_init_padata+0x12/0x80
<4>[    0.813758]  ? __pfx_pcrypt_init+0x10/0x10
<4>[    0.813758]  pcrypt_init+0x3a/0xa0
<4>[    0.813758]  do_one_initcall+0x5c/0x2b0
<4>[    0.813758]  kernel_init_freeable+0x18e/0x340
<4>[    0.813758]  ? __pfx_kernel_init+0x10/0x10
<4>[    0.813758]  kernel_init+0x15/0x130
<4>[    0.813758]  ret_from_fork+0x2c/0x50
<4>[    0.813758]  ? __pfx_kernel_init+0x10/0x10
<4>[    0.813758]  ret_from_fork_asm+0x1a/0x30
<4>[    0.813758]  </TASK>
`````````````````````````````````````````````````````````````````````````````````
Details log can be found in [3].

After bisecting the tree, the following patch [4] seems to be the first "bad"
commit

`````````````````````````````````````````````````````````````````````````````````````````````````````````
commit 1726a17135905e2d2773f18d47bd4e17dd26e1ed
Author: Lai Jiangshan mailto:jiangshan.ljs@...group.com
Date:   Thu Jul 4 11:49:13 2024 +0800

    workqueue: Put PWQ allocation and WQ enlistment in the same lock C.S.
`````````````````````````````````````````````````````````````````````````````````````````````````````````

We could not revert the patch because of merge conflict but resetting to the parent of the commit seems to fix the issue

Could you please check why the patch causes this regression and provide a fix if necessary?

Thank you.

Regards

Chaitanya

[1] https://intel-gfx-ci.01.org/tree/linux-next/combined-alt.html?
[2] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20240709 
[3] https://gfx-ci.igk.intel.com/tree/linux-next/next-20240709/bat-rpls-4/boot0.txt
[4] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20240709&id=1726a17135905e2d2773f18d47bd4e17dd26e1ed