| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <4f2bd603-dbd9-434c-9dfe-f2d4f1becd82@linux.ibm.com>
Date: Thu, 21 Aug 2025 17:41:02 +0530
From: Nilay Shroff <nilay@...ux.ibm.com>
To: Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>, sunjunchao@...edance.com,
axboe@...nel.dk, yukuai3@...wei.com, Ming Lei <ming.lei@...hat.com>,
linux-block@...r.kernel.org,
Linux List Kernel Mailing <linux-kernel@...r.kernel.org>,
Linux regressions mailing list <regressions@...ts.linux.dev>
Subject: Re: [REGRESSION] 6.17-rc2: lockdep circular dependency at boot introduced by 8f5845e0743b (“block: restore default wbt enablement”)
On 8/21/25 12:26 PM, Mikhail Gavrilov wrote:
> Hi,
>
> After commit 8f5845e0743b (“block: restore default wbt enablement”)
> I started seeing a lockdep warning about a circular locking dependency
> on every boot.
>
> Bisect
> git bisect identifies 8f5845e0743bf3512b71b3cb8afe06c192d6acc4 as the
> first bad commit.
> Reverting this commit on top of 6.17.0-rc2-git-b19a97d57c15 makes the
> warning disappear completely.
>
> The warning looks like this:
> [ 12.595070] nvme nvme0: 32/0/0 default/read/poll queues
> [ 12.595566] nvme nvme1: 32/0/0 default/read/poll queues
>
> [ 12.610697] ======================================================
> [ 12.610705] WARNING: possible circular locking dependency detected
> [ 12.610714] 6.17.0-rc2-git-b19a97d57c15+ #158 Not tainted
> [ 12.610726] ------------------------------------------------------
> [ 12.610734] kworker/u129:3/911 is trying to acquire lock:
> [ 12.610743] ffffffff899ab700 (cpu_hotplug_lock){++++}-{0:0}, at:
> static_key_slow_inc+0x16/0x40
> [ 12.610760]
> but task is already holding lock:
> [ 12.610769] ffff8881d166d570
> (&q->q_usage_counter(io)#4){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x16/0x30
> [ 12.610787]
> which lock already depends on the new lock.
>
> [ 12.610798]
> the existing dependency chain (in reverse order) is:
> [ 12.610971]
> -> #2 (&q->q_usage_counter(io)#4){++++}-{0:0}:
> [ 12.611246] __lock_acquire+0x56a/0xbe0
> [ 12.611381] lock_acquire.part.0+0xc7/0x270
> [ 12.611518] blk_alloc_queue+0x5cd/0x720
> [ 12.611649] blk_mq_alloc_queue+0x143/0x250
> [ 12.611780] __blk_mq_alloc_disk+0x18/0xd0
> [ 12.611906] nvme_alloc_ns+0x240/0x1930 [nvme_core]
> [ 12.612042] nvme_scan_ns+0x320/0x3b0 [nvme_core]
> [ 12.612170] async_run_entry_fn+0x94/0x540
> [ 12.612289] process_one_work+0x87a/0x14e0
> [ 12.612406] worker_thread+0x5f2/0xfd0
> [ 12.612527] kthread+0x3b0/0x770
> [ 12.612641] ret_from_fork+0x3ef/0x510
> [ 12.612760] ret_from_fork_asm+0x1a/0x30
> [ 12.612875]
> -> #1 (fs_reclaim){+.+.}-{0:0}:
> [ 12.613102] __lock_acquire+0x56a/0xbe0
> [ 12.613215] lock_acquire.part.0+0xc7/0x270
> [ 12.613327] fs_reclaim_acquire+0xd9/0x130
> [ 12.613444] __kmalloc_cache_node_noprof+0x60/0x4e0
> [ 12.613560] amd_pmu_cpu_prepare+0x123/0x670
> [ 12.613674] cpuhp_invoke_callback+0x2c8/0x9c0
> [ 12.613791] __cpuhp_invoke_callback_range+0xbd/0x1f0
> [ 12.613904] _cpu_up+0x2f8/0x6c0
> [ 12.614015] cpu_up+0x11e/0x1c0
> [ 12.614124] cpuhp_bringup_mask+0xea/0x130
> [ 12.614231] bringup_nonboot_cpus+0xa9/0x170
> [ 12.614335] smp_init+0x2b/0xf0
> [ 12.614443] kernel_init_freeable+0x23f/0x2e0
> [ 12.614545] kernel_init+0x1c/0x150
> [ 12.614643] ret_from_fork+0x3ef/0x510
> [ 12.614744] ret_from_fork_asm+0x1a/0x30
> [ 12.614840]
> -> #0 (cpu_hotplug_lock){++++}-{0:0}:
> [ 12.615029] check_prev_add+0xe1/0xcf0
> [ 12.615126] validate_chain+0x4cf/0x740
> [ 12.615221] __lock_acquire+0x56a/0xbe0
> [ 12.615316] lock_acquire.part.0+0xc7/0x270
> [ 12.615414] cpus_read_lock+0x40/0xe0
> [ 12.615508] static_key_slow_inc+0x16/0x40
> [ 12.615602] rq_qos_add+0x264/0x440
> [ 12.615696] wbt_init+0x3b2/0x510
> [ 12.615793] blk_register_queue+0x334/0x470
> [ 12.615887] __add_disk+0x5fd/0xd50
> [ 12.615980] add_disk_fwnode+0x113/0x590
> [ 12.616073] nvme_alloc_ns+0x7be/0x1930 [nvme_core]
> [ 12.616173] nvme_scan_ns+0x320/0x3b0 [nvme_core]
> [ 12.616272] async_run_entry_fn+0x94/0x540
> [ 12.616366] process_one_work+0x87a/0x14e0
> [ 12.616464] worker_thread+0x5f2/0xfd0
> [ 12.616558] kthread+0x3b0/0x770
> [ 12.616651] ret_from_fork+0x3ef/0x510
> [ 12.616749] ret_from_fork_asm+0x1a/0x30
> [ 12.616841]
> other info that might help us debug this:
>
> [ 12.617108] Chain exists of:
> cpu_hotplug_lock --> fs_reclaim --> &q->q_usage_counter(io)#4
>
> [ 12.617385] Possible unsafe locking scenario:
>
> [ 12.617570] CPU0 CPU1
> [ 12.617662] ---- ----
> [ 12.617755] lock(&q->q_usage_counter(io)#4);
> [ 12.617847] lock(fs_reclaim);
> [ 12.617940] lock(&q->q_usage_counter(io)#4);
> [ 12.618035] rlock(cpu_hotplug_lock);
> [ 12.618129]
> *** DEADLOCK ***
>
This one is already being addressed here:
https://lore.kernel.org/all/20250814082612.500845-1-nilay@linux.ibm.com/
Thanks,
--Nilay
Powered by blists - more mailing lists