| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <8369f596d02735c49d618af2f0df4a4db4ae71f2.camel@linux.intel.com>
Date: Tue, 22 Jul 2025 09:39:46 +0200
From: Thomas Hellström <thomas.hellstrom@...ux.intel.com>
To: Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>, axboe@...nel.dk,
linux-block@...r.kernel.org, Linux List Kernel Mailing
<linux-kernel@...r.kernel.org>, Ming Lei <ming.lei@...hat.com>, Linux
regressions mailing list <regressions@...ts.linux.dev>
Subject: Re: 6.15/regression/bisected - lockdep warning: circular locking
dependency detected when plugging USB stick after ffa1e7ada456
Hi Mikhail,
On Tue, 2025-07-22 at 02:54 +0500, Mikhail Gavrilov wrote:
> Hi Thomas,
>
> After commit
> ffa1e7ada456 ("block: Make request_queue lockdep splats show up
> earlier"),
> I started seeing lockdep warnings about circular locking dependencies
> in the kernel log every time I plug in a USB flash drive.
This means that there is a real circular locking dependency problem.
However the commit you are pointing to only makes it show up early.
Without the commit a similar (but not identical) lockdep splat would
have shown up under memory pressure with swapping activated.
Since I'm not the right person to fix the underlying locking order
violation, this will likely get picked up by someone on the linux-block
list.
Thanks for reporting,
Thomas.
>
> The warning looks like this:
> [ 247.453773] sd 6:0:0:0: [sda] Attached SCSI removable disk
>
> [ 247.486193] ======================================================
> [ 247.486195] WARNING: possible circular locking dependency detected
> [ 247.486197] 6.16.0-rc7 #36 Tainted: G L
> [ 247.486199] ------------------------------------------------------
> [ 247.486200] (udev-worker)/4257 is trying to acquire lock:
> [ 247.486202] ffff88816b9c0650 (&q->elevator_lock){+.+.}-{4:4}, at:
> elevator_change+0xb6/0x380
> [ 247.486213]
> but task is already holding lock:
> [ 247.486214] ffff88816b9c00b0
> (&q->q_usage_counter(io)#6){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x16/0x30
> [ 247.486221]
> which lock already depends on the new lock.
>
> [ 247.486222]
> the existing dependency chain (in reverse order) is:
> [ 247.486224]
> -> #3 (&q->q_usage_counter(io)#6){++++}-{0:0}:
> [ 247.486228] __lock_acquire+0x56a/0xbe0
> [ 247.486233] lock_acquire.part.0+0xc8/0x270
> [ 247.486235] blk_alloc_queue+0x5cd/0x720
> [ 247.486237] blk_mq_alloc_queue+0x14d/0x260
> [ 247.486239] scsi_alloc_sdev+0x862/0xc90
> [ 247.486242] scsi_probe_and_add_lun+0x4be/0xc10
> [ 247.486245] __scsi_scan_target+0x18b/0x3b0
> [ 247.486247] scsi_scan_channel+0xee/0x180
> [ 247.486250] scsi_scan_host_selected+0x1fd/0x2c0
> [ 247.486252] do_scan_async+0x42/0x450
> [ 247.486254] async_run_entry_fn+0x94/0x540
> [ 247.486258] process_one_work+0x87a/0x14d0
> [ 247.486260] worker_thread+0x5f2/0xfd0
> [ 247.486262] kthread+0x3b0/0x770
> [ 247.486266] ret_from_fork+0x3ef/0x510
> [ 247.486269] ret_from_fork_asm+0x1a/0x30
> [ 247.486271]
> -> #2 (fs_reclaim){+.+.}-{0:0}:
> [ 247.486275] __lock_acquire+0x56a/0xbe0
> [ 247.486277] lock_acquire.part.0+0xc8/0x270
> [ 247.486279] fs_reclaim_acquire+0xd9/0x130
> [ 247.486282] prepare_alloc_pages+0x153/0x5a0
> [ 247.486284] __alloc_frozen_pages_noprof+0x142/0x490
> [ 247.486286] __alloc_pages_noprof+0x12/0x210
> [ 247.486288] pcpu_alloc_pages.isra.0+0xfa/0x4d0
> [ 247.486291] pcpu_populate_chunk+0x39/0x80
> [ 247.486293] pcpu_alloc_noprof+0x759/0xeb0
> [ 247.486296] iommu_dma_init_fq+0x19c/0x7c0
> [ 247.486299] iommu_dma_init_domain+0x53f/0x7f0
> [ 247.486301] iommu_setup_dma_ops+0xd3/0x200
> [ 247.486303] bus_iommu_probe+0x1f0/0x4b0
> [ 247.486306] iommu_device_register+0x186/0x280
> [ 247.486308] iommu_init_pci+0xc8c/0xd00
> [ 247.486312] amd_iommu_init_pci+0x83/0x4e0
> [ 247.486314] state_next+0x28f/0x5c0
> [ 247.486317] iommu_go_to_state+0x2b/0x60
> [ 247.486319] amd_iommu_init+0x21/0x60
> [ 247.486321] pci_iommu_init+0x38/0x60
> [ 247.486325] do_one_initcall+0xd2/0x450
> [ 247.486327] do_initcalls+0x216/0x240
> [ 247.486330] kernel_init_freeable+0x299/0x2d0
> [ 247.486332] kernel_init+0x1c/0x150
> [ 247.486335] ret_from_fork+0x3ef/0x510
> [ 247.486337] ret_from_fork_asm+0x1a/0x30
> [ 247.486338]
> -> #1 (pcpu_alloc_mutex){+.+.}-{4:4}:
> [ 247.486342] __lock_acquire+0x56a/0xbe0
> [ 247.486344] lock_acquire.part.0+0xc8/0x270
> [ 247.486346] __mutex_lock+0x1b2/0x1b70
> [ 247.486348] pcpu_alloc_noprof+0x884/0xeb0
> [ 247.486351] sbitmap_init_node+0x252/0x6a0
> [ 247.486354] sbitmap_queue_init_node+0x2d/0x420
> [ 247.486356] blk_mq_init_tags+0x154/0x2a0
> [ 247.486359] blk_mq_alloc_map_and_rqs+0xa6/0x310
> [ 247.486361] blk_mq_init_sched+0x2a4/0x580
> [ 247.486363] elevator_switch+0x18b/0x630
> [ 247.486365] elevator_change+0x209/0x380
> [ 247.486368] elevator_set_default+0x22d/0x2a0
> [ 247.486370] blk_register_queue+0x33a/0x490
> [ 247.486372] __add_disk+0x5fd/0xd50
> [ 247.486374] add_disk_fwnode+0x113/0x590
> [ 247.486377] sd_probe+0x873/0xe10
> [ 247.486380] really_probe+0x1de/0x890
> [ 247.486383] __driver_probe_device+0x18c/0x390
> [ 247.486385] driver_probe_device+0x4a/0x120
> [ 247.486388] __device_attach_driver+0x156/0x280
> [ 247.486389] bus_for_each_drv+0x111/0x1a0
> [ 247.486392] __device_attach_async_helper+0x19c/0x240
> [ 247.486394] async_run_entry_fn+0x94/0x540
> [ 247.486396] process_one_work+0x87a/0x14d0
> [ 247.486398] worker_thread+0x5f2/0xfd0
> [ 247.486400] kthread+0x3b0/0x770
> [ 247.486402] ret_from_fork+0x3ef/0x510
> [ 247.486404] ret_from_fork_asm+0x1a/0x30
> [ 247.486406]
> -> #0 (&q->elevator_lock){+.+.}-{4:4}:
> [ 247.486409] check_prev_add+0xe1/0xcf0
> [ 247.486411] validate_chain+0x4cf/0x740
> [ 247.486413] __lock_acquire+0x56a/0xbe0
> [ 247.486414] lock_acquire.part.0+0xc8/0x270
> [ 247.486416] __mutex_lock+0x1b2/0x1b70
> [ 247.486418] elevator_change+0xb6/0x380
> [ 247.486420] elv_iosched_store+0x24a/0x2c0
> [ 247.486422] queue_attr_store+0x238/0x340
> [ 247.486425] kernfs_fop_write_iter+0x39b/0x5a0
> [ 247.486428] vfs_write+0x524/0xe70
> [ 247.486430] ksys_write+0xff/0x200
> [ 247.486432] do_syscall_64+0x98/0x3c0
> [ 247.486435] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486438]
> other info that might help us debug this:
>
> [ 247.486439] Chain exists of:
> &q->elevator_lock --> fs_reclaim --> &q-
> >q_usage_counter(io)#6
>
> [ 247.486444] Possible unsafe locking scenario:
>
> [ 247.486446] CPU0 CPU1
> [ 247.486447] ---- ----
> [ 247.486448] lock(&q->q_usage_counter(io)#6);
> [ 247.486451] lock(fs_reclaim);
> [ 247.486453] lock(&q-
> >q_usage_counter(io)#6);
> [ 247.486456] lock(&q->elevator_lock);
> [ 247.486458]
> *** DEADLOCK ***
>
> [ 247.486459] 6 locks held by (udev-worker)/4257:
> [ 247.486461] #0: ffff88817b49c458 (sb_writers#4){.+.+}-{0:0}, at:
> ksys_write+0xff/0x200
> [ 247.486467] #1: ffff88819e767490 (&of->mutex#2){+.+.}-{4:4}, at:
> kernfs_fop_write_iter+0x25b/0x5a0
> [ 247.486473] #2: ffff8883b352c8c0 (kn->active#204){.+.+}-{0:0},
> at:
> kernfs_fop_write_iter+0x27e/0x5a0
> [ 247.486479] #3: ffff888169ecc3c0
> (&set->update_nr_hwq_lock){.+.+}-{4:4}, at:
> elv_iosched_store+0x1ba/0x2c0
> [ 247.486484] #4: ffff88816b9c00b0
> (&q->q_usage_counter(io)#6){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x16/0x30
> [ 247.486490] #5: ffff88816b9c00f0
> (&q->q_usage_counter(queue)#5){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x16/0x30
> [ 247.486495]
> stack backtrace:
> [ 247.486498] CPU: 3 UID: 0 PID: 4257 Comm: (udev-worker) Tainted: G
> L 6.16.0-rc7 #36 PREEMPT(lazy)
> [ 247.486501] Tainted: [L]=SOFTLOCKUP
> [ 247.486501] Hardware name: ASRock B650I Lightning WiFi/B650I
> Lightning WiFi, BIOS 3.30 06/16/2025
> [ 247.486503] Call Trace:
> [ 247.486503] <TASK>
> [ 247.486505] dump_stack_lvl+0x84/0xd0
> [ 247.486509] print_circular_bug.cold+0x38/0x46
> [ 247.486512] check_noncircular+0x14a/0x170
> [ 247.486515] check_prev_add+0xe1/0xcf0
> [ 247.486517] ? lock_acquire.part.0+0xc8/0x270
> [ 247.486518] validate_chain+0x4cf/0x740
> [ 247.486520] __lock_acquire+0x56a/0xbe0
> [ 247.486522] lock_acquire.part.0+0xc8/0x270
> [ 247.486524] ? elevator_change+0xb6/0x380
> [ 247.486526] ? __lock_release.isra.0+0x1cb/0x340
> [ 247.486527] ? rcu_is_watching+0x15/0xe0
> [ 247.486530] ? __pfx___might_resched+0x10/0x10
> [ 247.486532] ? elevator_change+0xb6/0x380
> [ 247.486534] ? lock_acquire+0xf7/0x140
> [ 247.486535] __mutex_lock+0x1b2/0x1b70
> [ 247.486537] ? elevator_change+0xb6/0x380
> [ 247.486539] ? elevator_change+0xb6/0x380
> [ 247.486541] ? __pfx_xa_find_after+0x10/0x10
> [ 247.486543] ? __pfx___mutex_lock+0x10/0x10
> [ 247.486545] ? __pfx___might_resched+0x10/0x10
> [ 247.486547] ? blk_mq_cancel_work_sync+0xc0/0x100
> [ 247.486549] ? __pfx_blk_mq_cancel_work_sync+0x10/0x10
> [ 247.486551] ? elevator_change+0xb6/0x380
> [ 247.486553] elevator_change+0xb6/0x380
> [ 247.486556] elv_iosched_store+0x24a/0x2c0
> [ 247.486558] ? __pfx_elv_iosched_store+0x10/0x10
> [ 247.486560] ? __pfx___might_resched+0x10/0x10
> [ 247.486562] ? __pfx_sysfs_kf_write+0x10/0x10
> [ 247.486564] queue_attr_store+0x238/0x340
> [ 247.486566] ? __pfx_queue_attr_store+0x10/0x10
> [ 247.486567] ? __lock_acquire+0x56a/0xbe0
> [ 247.486569] ? lock_acquire.part.0+0xc8/0x270
> [ 247.486570] ? find_held_lock+0x2b/0x80
> [ 247.486572] ? __lock_release.isra.0+0x1cb/0x340
> [ 247.486574] ? sysfs_file_kobj+0xb3/0x1c0
> [ 247.486576] ? sysfs_file_kobj+0xbd/0x1c0
> [ 247.486577] ? sysfs_kf_write+0x65/0x170
> [ 247.486579] ? __pfx_sysfs_kf_write+0x10/0x10
> [ 247.486580] kernfs_fop_write_iter+0x39b/0x5a0
> [ 247.486582] ? __pfx_kernfs_fop_write_iter+0x10/0x10
> [ 247.486584] vfs_write+0x524/0xe70
> [ 247.486586] ? __pfx_vfs_write+0x10/0x10
> [ 247.486588] ? __pfx___seccomp_filter+0x10/0x10
> [ 247.486591] ksys_write+0xff/0x200
> [ 247.486593] ? __pfx_ksys_write+0x10/0x10
> [ 247.486595] ? syscall_trace_enter+0x8e/0x2e0
> [ 247.486598] do_syscall_64+0x98/0x3c0
> [ 247.486600] ? __x64_sys_openat+0x10e/0x210
> [ 247.486602] ? do_syscall_64+0x161/0x3c0
> [ 247.486604] ? do_sys_openat2+0x109/0x180
> [ 247.486605] ? __pfx___x64_sys_openat+0x10/0x10
> [ 247.486607] ? __pfx_do_sys_openat2+0x10/0x10
> [ 247.486609] ? lockdep_hardirqs_on+0x8c/0x130
> [ 247.486611] ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486613] ? do_syscall_64+0x161/0x3c0
> [ 247.486615] ? __x64_sys_openat+0x10e/0x210
> [ 247.486616] ? lockdep_hardirqs_on+0x8c/0x130
> [ 247.486618] ? __pfx___x64_sys_openat+0x10/0x10
> [ 247.486621] ? lockdep_hardirqs_on+0x8c/0x130
> [ 247.486623] ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486624] ? do_syscall_64+0x161/0x3c0
> [ 247.486626] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486627] RIP: 0033:0x7f9afd67b5c6
> [ 247.486642] Code: 5d e8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 75
> 19 83 e2 39 83 fa 08 75 11 e8 26 ff ff ff 66 0f 1f 44 00 00 48 8b 45
> 10 0f 05 <48> 8b 5d f8 c9 c3 0f 1f 40 00 f3 0f 1e fa 55 48 89 e5 48
> 83
> ec 08
> [ 247.486643] RSP: 002b:00007fff244eb210 EFLAGS: 00000202 ORIG_RAX:
> 0000000000000001
> [ 247.486646] RAX: ffffffffffffffda RBX: 000055e41796e060 RCX:
> 00007f9afd67b5c6
> [ 247.486647] RDX: 0000000000000003 RSI: 00007fff244eb560 RDI:
> 0000000000000014
> [ 247.486647] RBP: 00007fff244eb230 R08: 0000000000000000 R09:
> 0000000000000000
> [ 247.486648] R10: 0000000000000000 R11: 0000000000000202 R12:
> 0000000000000003
> [ 247.486649] R13: 0000000000000003 R14: 00007fff244eb560 R15:
> 00007fff244eb560
> [ 247.486652] </TASK>
>
> I bisected the issue to this commit:
> ffa1e7ada456087c2402b37cd6b2863ced29aff0 is the first bad commit
> commit ffa1e7ada456087c2402b37cd6b2863ced29aff0
> Author: Thomas Hellström <thomas.hellstrom@...ux.intel.com>
> Date: Tue Mar 18 10:55:48 2025 +0100
>
> block: Make request_queue lockdep splats show up earlier
>
> In recent kernels, there are lockdep splats around the
> struct request_queue::io_lockdep_map, similar to [1], but they
> typically don't show up until reclaim with writeback happens.
>
> Having multiple kernel versions released with a known risc of
> kernel
> deadlock during reclaim writeback should IMHO be addressed and
> backported to -stable with the highest priority.
>
> In order to have these lockdep splats show up earlier,
> preferrably during system initialization, prime the
> struct request_queue::io_lockdep_map as GFP_KERNEL reclaim-
> tainted. This will instead lead to lockdep splats looking similar
> to [2], but without the need for reclaim + writeback
> happening.
>
> [1]:
> [ 189.762244]
> ======================================================
> [ 189.762432] WARNING: possible circular locking dependency
> detected
> [ 189.762441] 6.14.0-rc6-xe+ #6 Tainted: G U
> [ 189.762450] --------------------------------------------------
> ----
> [ 189.762459] kswapd0/119 is trying to acquire lock:
> [ 189.762467] ffff888110ceb710
> (&q->q_usage_counter(io)#26){++++}-{0:0}, at: __submit_bio+0x76/0x230
> [ 189.762485]
> but task is already holding lock:
> [ 189.762494] ffffffff834c97c0 (fs_reclaim){+.+.}-{0:0}, at:
> balance_pgdat+0xbe/0xb00
> [ 189.762507]
> which lock already depends on the new lock.
>
> [ 189.762519]
> the existing dependency chain (in reverse order)
> is:
> [ 189.762529]
> -> #2 (fs_reclaim){+.+.}-{0:0}:
> [ 189.762540] fs_reclaim_acquire+0xc5/0x100
> [ 189.762548] kmem_cache_alloc_lru_noprof+0x4a/0x480
> [ 189.762558] alloc_inode+0xaa/0xe0
> [ 189.762566] iget_locked+0x157/0x330
> [ 189.762573] kernfs_get_inode+0x1b/0x110
> [ 189.762582] kernfs_get_tree+0x1b0/0x2e0
> [ 189.762590] sysfs_get_tree+0x1f/0x60
> [ 189.762597] vfs_get_tree+0x2a/0xf0
> [ 189.762605] path_mount+0x4cd/0xc00
> [ 189.762613] __x64_sys_mount+0x119/0x150
> [ 189.762621] x64_sys_call+0x14f2/0x2310
> [ 189.762630] do_syscall_64+0x91/0x180
> [ 189.762637] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 189.762647]
> -> #1 (&root->kernfs_rwsem){++++}-{3:3}:
> [ 189.762659] down_write+0x3e/0xf0
> [ 189.762667] kernfs_remove+0x32/0x60
> [ 189.762676] sysfs_remove_dir+0x4f/0x60
> [ 189.762685] __kobject_del+0x33/0xa0
> [ 189.762709] kobject_del+0x13/0x30
> [ 189.762716] elv_unregister_queue+0x52/0x80
> [ 189.762725] elevator_switch+0x68/0x360
> [ 189.762733] elv_iosched_store+0x14b/0x1b0
> [ 189.762756] queue_attr_store+0x181/0x1e0
> [ 189.762765] sysfs_kf_write+0x49/0x80
> [ 189.762773] kernfs_fop_write_iter+0x17d/0x250
> [ 189.762781] vfs_write+0x281/0x540
> [ 189.762790] ksys_write+0x72/0xf0
> [ 189.762798] __x64_sys_write+0x19/0x30
> [ 189.762807] x64_sys_call+0x2a3/0x2310
> [ 189.762815] do_syscall_64+0x91/0x180
> [ 189.762823] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 189.762833]
> -> #0 (&q->q_usage_counter(io)#26){++++}-{0:0}:
> [ 189.762845] __lock_acquire+0x1525/0x2760
> [ 189.762854] lock_acquire+0xca/0x310
> [ 189.762861] blk_mq_submit_bio+0x8a2/0xba0
> [ 189.762870] __submit_bio+0x76/0x230
> [ 189.762878] submit_bio_noacct_nocheck+0x323/0x430
> [ 189.762888] submit_bio_noacct+0x2cc/0x620
> [ 189.762896] submit_bio+0x38/0x110
> [ 189.762904] __swap_writepage+0xf5/0x380
> [ 189.762912] swap_writepage+0x3c7/0x600
> [ 189.762920] shmem_writepage+0x3da/0x4f0
> [ 189.762929] pageout+0x13f/0x310
> [ 189.762937] shrink_folio_list+0x61c/0xf60
> [ 189.763261] evict_folios+0x378/0xcd0
> [ 189.763584] try_to_shrink_lruvec+0x1b0/0x360
> [ 189.763946] shrink_one+0x10e/0x200
> [ 189.764266] shrink_node+0xc02/0x1490
> [ 189.764586] balance_pgdat+0x563/0xb00
> [ 189.764934] kswapd+0x1e8/0x430
> [ 189.765249] kthread+0x10b/0x260
> [ 189.765559] ret_from_fork+0x44/0x70
> [ 189.765889] ret_from_fork_asm+0x1a/0x30
> [ 189.766198]
> other info that might help us debug this:
>
> [ 189.767089] Chain exists of:
> &q->q_usage_counter(io)#26 -->
> &root->kernfs_rwsem --> fs_reclaim
>
> [ 189.767971] Possible unsafe locking scenario:
>
> [ 189.768555] CPU0 CPU1
> [ 189.768849] ---- ----
> [ 189.769136] lock(fs_reclaim);
> [ 189.769421] lock(&root-
> >kernfs_rwsem);
> [ 189.769714] lock(fs_reclaim);
> [ 189.770016] rlock(&q->q_usage_counter(io)#26);
> [ 189.770305]
> *** DEADLOCK ***
>
> [ 189.771167] 1 lock held by kswapd0/119:
> [ 189.771453] #0: ffffffff834c97c0 (fs_reclaim){+.+.}-{0:0},
> at:
> balance_pgdat+0xbe/0xb00
> [ 189.771770]
> stack backtrace:
> [ 189.772351] CPU: 4 UID: 0 PID: 119 Comm: kswapd0 Tainted: G
> U 6.14.0-rc6-xe+ #6
> [ 189.772353] Tainted: [U]=USER
> [ 189.772354] Hardware name: ASUS System Product Name/PRIME
> B560M-A AC, BIOS 2001 02/01/2023
> [ 189.772354] Call Trace:
> [ 189.772355] <TASK>
> [ 189.772356] dump_stack_lvl+0x6e/0xa0
> [ 189.772359] dump_stack+0x10/0x18
> [ 189.772360] print_circular_bug.cold+0x17a/0x1b7
> [ 189.772363] check_noncircular+0x13a/0x150
> [ 189.772365] ? __pfx_stack_trace_consume_entry+0x10/0x10
> [ 189.772368] __lock_acquire+0x1525/0x2760
> [ 189.772368] ? ret_from_fork_asm+0x1a/0x30
> [ 189.772371] lock_acquire+0xca/0x310
> [ 189.772372] ? __submit_bio+0x76/0x230
> [ 189.772375] ? lock_release+0xd5/0x2c0
> [ 189.772376] blk_mq_submit_bio+0x8a2/0xba0
> [ 189.772378] ? __submit_bio+0x76/0x230
> [ 189.772380] __submit_bio+0x76/0x230
> [ 189.772382] ? trace_hardirqs_on+0x1e/0xe0
> [ 189.772384] submit_bio_noacct_nocheck+0x323/0x430
> [ 189.772386] ? submit_bio_noacct_nocheck+0x323/0x430
> [ 189.772387] ? __might_sleep+0x58/0xa0
> [ 189.772390] submit_bio_noacct+0x2cc/0x620
> [ 189.772391] ? count_memcg_events+0x68/0x90
> [ 189.772393] submit_bio+0x38/0x110
> [ 189.772395] __swap_writepage+0xf5/0x380
> [ 189.772396] swap_writepage+0x3c7/0x600
> [ 189.772397] shmem_writepage+0x3da/0x4f0
> [ 189.772401] pageout+0x13f/0x310
> [ 189.772406] shrink_folio_list+0x61c/0xf60
> [ 189.772409] ? isolate_folios+0xe80/0x16b0
> [ 189.772410] ? mark_held_locks+0x46/0x90
> [ 189.772412] evict_folios+0x378/0xcd0
> [ 189.772414] ? evict_folios+0x34a/0xcd0
> [ 189.772415] ? lock_is_held_type+0xa3/0x130
> [ 189.772417] try_to_shrink_lruvec+0x1b0/0x360
> [ 189.772420] shrink_one+0x10e/0x200
> [ 189.772421] shrink_node+0xc02/0x1490
> [ 189.772423] ? shrink_node+0xa08/0x1490
> [ 189.772424] ? shrink_node+0xbd8/0x1490
> [ 189.772425] ? mem_cgroup_iter+0x366/0x480
> [ 189.772427] balance_pgdat+0x563/0xb00
> [ 189.772428] ? balance_pgdat+0x563/0xb00
> [ 189.772430] ? trace_hardirqs_on+0x1e/0xe0
> [ 189.772431] ? finish_task_switch.isra.0+0xcb/0x330
> [ 189.772433] ? __switch_to_asm+0x33/0x70
> [ 189.772437] kswapd+0x1e8/0x430
> [ 189.772438] ? __pfx_autoremove_wake_function+0x10/0x10
> [ 189.772440] ? __pfx_kswapd+0x10/0x10
> [ 189.772441] kthread+0x10b/0x260
> [ 189.772443] ? __pfx_kthread+0x10/0x10
> [ 189.772444] ret_from_fork+0x44/0x70
> [ 189.772446] ? __pfx_kthread+0x10/0x10
> [ 189.772447] ret_from_fork_asm+0x1a/0x30
> [ 189.772450] </TASK>
>
> [2]:
> [ 8.760253]
> ======================================================
> [ 8.760254] WARNING: possible circular locking dependency
> detected
> [ 8.760255] 6.14.0-rc6-xe+ #7 Tainted: G U
> [ 8.760256] --------------------------------------------------
> ----
> [ 8.760257] (udev-worker)/674 is trying to acquire lock:
> [ 8.760259] ffff888100e39148 (&root->kernfs_rwsem){++++}-
> {3:3},
> at: kernfs_remove+0x32/0x60
> [ 8.760265]
> but task is already holding lock:
> [ 8.760266] ffff888110dc7680
> (&q->q_usage_counter(io)#27){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x12/0x30
> [ 8.760272]
> which lock already depends on the new lock.
>
> [ 8.760272]
> the existing dependency chain (in reverse order)
> is:
> [ 8.760273]
> -> #2 (&q->q_usage_counter(io)#27){++++}-{0:0}:
> [ 8.760276] blk_alloc_queue+0x30a/0x350
> [ 8.760279] blk_mq_alloc_queue+0x6b/0xe0
> [ 8.760281] scsi_alloc_sdev+0x276/0x3c0
> [ 8.760284] scsi_probe_and_add_lun+0x22a/0x440
> [ 8.760286] __scsi_scan_target+0x109/0x230
> [ 8.760288] scsi_scan_channel+0x65/0xc0
> [ 8.760290] scsi_scan_host_selected+0xff/0x140
> [ 8.760292] do_scsi_scan_host+0xa7/0xc0
> [ 8.760293] do_scan_async+0x1c/0x160
> [ 8.760295] async_run_entry_fn+0x32/0x150
> [ 8.760299] process_one_work+0x224/0x5f0
> [ 8.760302] worker_thread+0x1d4/0x3e0
> [ 8.760304] kthread+0x10b/0x260
> [ 8.760306] ret_from_fork+0x44/0x70
> [ 8.760309] ret_from_fork_asm+0x1a/0x30
> [ 8.760312]
> -> #1 (fs_reclaim){+.+.}-{0:0}:
> [ 8.760315] fs_reclaim_acquire+0xc5/0x100
> [ 8.760317] kmem_cache_alloc_lru_noprof+0x4a/0x480
> [ 8.760319] alloc_inode+0xaa/0xe0
> [ 8.760322] iget_locked+0x157/0x330
> [ 8.760323] kernfs_get_inode+0x1b/0x110
> [ 8.760325] kernfs_get_tree+0x1b0/0x2e0
> [ 8.760327] sysfs_get_tree+0x1f/0x60
> [ 8.760329] vfs_get_tree+0x2a/0xf0
> [ 8.760332] path_mount+0x4cd/0xc00
> [ 8.760334] __x64_sys_mount+0x119/0x150
> [ 8.760336] x64_sys_call+0x14f2/0x2310
> [ 8.760338] do_syscall_64+0x91/0x180
> [ 8.760340] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 8.760342]
> -> #0 (&root->kernfs_rwsem){++++}-{3:3}:
> [ 8.760345] __lock_acquire+0x1525/0x2760
> [ 8.760347] lock_acquire+0xca/0x310
> [ 8.760348] down_write+0x3e/0xf0
> [ 8.760350] kernfs_remove+0x32/0x60
> [ 8.760351] sysfs_remove_dir+0x4f/0x60
> [ 8.760353] __kobject_del+0x33/0xa0
> [ 8.760355] kobject_del+0x13/0x30
> [ 8.760356] elv_unregister_queue+0x52/0x80
> [ 8.760358] elevator_switch+0x68/0x360
> [ 8.760360] elv_iosched_store+0x14b/0x1b0
> [ 8.760362] queue_attr_store+0x181/0x1e0
> [ 8.760364] sysfs_kf_write+0x49/0x80
> [ 8.760366] kernfs_fop_write_iter+0x17d/0x250
> [ 8.760367] vfs_write+0x281/0x540
> [ 8.760370] ksys_write+0x72/0xf0
> [ 8.760372] __x64_sys_write+0x19/0x30
> [ 8.760374] x64_sys_call+0x2a3/0x2310
> [ 8.760376] do_syscall_64+0x91/0x180
> [ 8.760377] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 8.760380]
> other info that might help us debug this:
>
> [ 8.760380] Chain exists of:
> &root->kernfs_rwsem --> fs_reclaim -->
> &q->q_usage_counter(io)#27
>
> [ 8.760384] Possible unsafe locking scenario:
>
> [ 8.760384] CPU0 CPU1
> [ 8.760385] ---- ----
> [ 8.760385] lock(&q->q_usage_counter(io)#27);
> [ 8.760387] lock(fs_reclaim);
> [ 8.760388]
> lock(&q->q_usage_counter(io)#27);
> [ 8.760390] lock(&root->kernfs_rwsem);
> [ 8.760391]
> *** DEADLOCK ***
>
> [ 8.760391] 6 locks held by (udev-worker)/674:
> [ 8.760392] #0: ffff8881209ac420 (sb_writers#4){.+.+}-{0:0},
> at: ksys_write+0x72/0xf0
> [ 8.760398] #1: ffff88810c80f488 (&of->mutex#2){+.+.}-{3:3},
> at: kernfs_fop_write_iter+0x136/0x250
> [ 8.760402] #2: ffff888125d1d330 (kn->active#101){.+.+}-
> {0:0},
> at: kernfs_fop_write_iter+0x13f/0x250
> [ 8.760406] #3: ffff888110dc7bb0 (&q->sysfs_lock){+.+.}-
> {3:3},
> at: queue_attr_store+0x148/0x1e0
> [ 8.760411] #4: ffff888110dc7680
> (&q->q_usage_counter(io)#27){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x12/0x30
> [ 8.760416] #5: ffff888110dc76b8
> (&q->q_usage_counter(queue)#27){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x12/0x30
> [ 8.760421]
> stack backtrace:
> [ 8.760422] CPU: 7 UID: 0 PID: 674 Comm: (udev-worker)
> Tainted:
> G U 6.14.0-rc6-xe+ #7
> [ 8.760424] Tainted: [U]=USER
> [ 8.760425] Hardware name: ASUS System Product Name/PRIME
> B560M-A AC, BIOS 2001 02/01/2023
> [ 8.760426] Call Trace:
> [ 8.760427] <TASK>
> [ 8.760428] dump_stack_lvl+0x6e/0xa0
> [ 8.760431] dump_stack+0x10/0x18
> [ 8.760433] print_circular_bug.cold+0x17a/0x1b7
> [ 8.760437] check_noncircular+0x13a/0x150
> [ 8.760441] ? save_trace+0x54/0x360
> [ 8.760445] __lock_acquire+0x1525/0x2760
> [ 8.760446] ? irqentry_exit+0x3a/0xb0
> [ 8.760448] ? sysvec_apic_timer_interrupt+0x57/0xc0
> [ 8.760452] lock_acquire+0xca/0x310
> [ 8.760453] ? kernfs_remove+0x32/0x60
> [ 8.760457] down_write+0x3e/0xf0
> [ 8.760459] ? kernfs_remove+0x32/0x60
> [ 8.760460] kernfs_remove+0x32/0x60
> [ 8.760462] sysfs_remove_dir+0x4f/0x60
> [ 8.760464] __kobject_del+0x33/0xa0
> [ 8.760466] kobject_del+0x13/0x30
> [ 8.760467] elv_unregister_queue+0x52/0x80
> [ 8.760470] elevator_switch+0x68/0x360
> [ 8.760472] elv_iosched_store+0x14b/0x1b0
> [ 8.760475] queue_attr_store+0x181/0x1e0
> [ 8.760479] ? lock_acquire+0xca/0x310
> [ 8.760480] ? kernfs_fop_write_iter+0x13f/0x250
> [ 8.760482] ? lock_is_held_type+0xa3/0x130
> [ 8.760485] sysfs_kf_write+0x49/0x80
> [ 8.760487] kernfs_fop_write_iter+0x17d/0x250
> [ 8.760489] vfs_write+0x281/0x540
> [ 8.760494] ksys_write+0x72/0xf0
> [ 8.760497] __x64_sys_write+0x19/0x30
> [ 8.760499] x64_sys_call+0x2a3/0x2310
> [ 8.760502] do_syscall_64+0x91/0x180
> [ 8.760504] ? trace_hardirqs_off+0x5d/0xe0
> [ 8.760506] ? handle_softirqs+0x479/0x4d0
> [ 8.760508] ? hrtimer_interrupt+0x13f/0x280
> [ 8.760511] ? irqentry_exit_to_user_mode+0x8b/0x260
> [ 8.760513] ? clear_bhb_loop+0x15/0x70
> [ 8.760515] ? clear_bhb_loop+0x15/0x70
> [ 8.760516] ? clear_bhb_loop+0x15/0x70
> [ 8.760518] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 8.760520] RIP: 0033:0x7aa3bf2f5504
> [ 8.760522] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f
> 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d c5 8b 10 00 00 74 13 b8 01 00
> 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83
> ec
> 20 48 89
> [ 8.760523] RSP: 002b:00007ffc1e3697d8 EFLAGS: 00000202
> ORIG_RAX: 0000000000000001
> [ 8.760526] RAX: ffffffffffffffda RBX: 0000000000000003 RCX:
> 00007aa3bf2f5504
> [ 8.760527] RDX: 0000000000000003 RSI: 00007ffc1e369ae0 RDI:
> 000000000000001c
> [ 8.760528] RBP: 00007ffc1e369800 R08: 00007aa3bf3f51c8 R09:
> 00007ffc1e3698b0
> [ 8.760528] R10: 0000000000000000 R11: 0000000000000202 R12:
> 0000000000000003
> [ 8.760529] R13: 00007ffc1e369ae0 R14: 0000613ccf21f2f0 R15:
> 00007aa3bf3f4e80
> [ 8.760533] </TASK>
>
> v2:
> - Update a code comment to increase readability (Ming Lei).
>
> Cc: Jens Axboe <axboe@...nel.dk>
> Cc: linux-block@...r.kernel.org
> Cc: linux-kernel@...r.kernel.org
> Cc: Ming Lei <ming.lei@...hat.com>
> Signed-off-by: Thomas Hellström
> <thomas.hellstrom@...ux.intel.com>
> Reviewed-by: Ming Lei <ming.lei@...hat.com>
> Link:
> https://lore.kernel.org/r/20250318095548.5187-1-thomas.hellstrom@linux.intel.com
> Signed-off-by: Jens Axboe <axboe@...nel.dk>
>
> block/blk-core.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> Reverting commit ffa1e7ada456 on top of 6.16-rc7 completely
> eliminates
> the warning. No lockdep splats occur when plugging in USB devices
> after the revert.
>
> The full dmesg trace, kernel config, and hardware probe are attached
> below:
>
> Hardware: https://linux-hardware.org/?probe=347fa4c2c6
> Kernel config: (attached)
> Full kernel log: (attached)
>
> This looks like a false-positive introduced by priming io_lockdep_map
> earlier, as explained in the commit message. However, it creates
> noise
> for regular users and may obscure more critical lockdep reports.
>
> Thanks for looking into this.
>
Powered by blists - more mailing lists