| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <1335339396.16988.80.camel@lappy>
Date: Wed, 25 Apr 2012 09:36:36 +0200
From: Sasha Levin <sasha.levin@...cle.com>
To: tj@...nel.org, axboe@...nel.dk
Cc: "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
paulmck@...ux.vnet.ibm.com
Subject: blkcg: INFO: possible irq lock inversion dependency detected
Hi all,
I got the following warning when booting the latest -next kernel:
[ 14.019666] =========================================================
[ 14.020011] [ INFO: possible irq lock inversion dependency detected ]
[ 14.020011] 3.4.0-rc4-next-20120424-sasha #89 Tainted: G W
[ 14.020011] ---------------------------------------------------------
[ 14.020011] rcuc/0/7 just changed the state of lock:
[ 14.020011] (tg_stats_alloc_lock){+.....}, at: [<ffffffff818611ab>] throtl_pd_exit+0x3b/0x70
[ 14.020011] but this lock was taken by another, HARDIRQ-safe lock in the past:
[ 14.020011] (&(&q->__queue_lock)->rlock){-.-...}
[ 14.020011]
[ 14.020011] and interrupts could create inverse lock ordering between them.
[ 14.020011]
[ 14.020011]
[ 14.020011] other info that might help us debug this:
[ 14.020011] Chain exists of:
[ 14.020011] &(&q->__queue_lock)->rlock --> &(&blkcg->lock)->rlock --> tg_stats_alloc_lock
[ 14.020011]
[ 14.020011] Possible interrupt unsafe locking scenario:
[ 14.020011]
[ 14.020011] CPU0 CPU1
[ 14.020011] ---- ----
[ 14.020011] lock(tg_stats_alloc_lock);
[ 14.020011] local_irq_disable();
[ 14.020011] lock(&(&q->__queue_lock)->rlock);
[ 14.020011] lock(&(&blkcg->lock)->rlock);
[ 14.020011] <Interrupt>
[ 14.020011] lock(&(&q->__queue_lock)->rlock);
[ 14.020011]
[ 14.020011] *** DEADLOCK ***
[ 14.020011]
[ 14.020011] no locks held by rcuc/0/7.
[ 14.020011]
[ 14.020011] the shortest dependencies between 2nd lock and 1st lock:
[ 14.020011] -> (&(&q->__queue_lock)->rlock){-.-...} ops: 260 {
[ 14.020011] IN-HARDIRQ-W at:
[ 14.020011] [<ffffffff8111696b>] mark_irqflags+0x6b/0x170
[ 14.020011] [<ffffffff8111837b>] __lock_acquire+0x2bb/0x4c0
[ 14.020011] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.020011] [<ffffffff8288f791>] _raw_spin_lock_irqsave+0x81/0xc0
[ 14.020011] [<ffffffff81c43bea>] ata_qc_schedule_eh+0x6a/0xa0
[ 14.020011] [<ffffffff81c37f39>] ata_qc_complete+0xa9/0x210
[ 14.020011] [<ffffffff81c4a4d8>] ata_hsm_qc_complete+0xf8/0x130
[ 14.020011] [<ffffffff81c4ae57>] ata_sff_hsm_move+0x547/0x570
[ 14.020011] [<ffffffff81c4b0bc>] __ata_sff_port_intr+0xcc/0xe0
[ 14.020011] [<ffffffff81c4b190>] ata_bmdma_port_intr+0xc0/0x120
[ 14.020011] [<ffffffff81c4b56f>] ata_bmdma_interrupt+0xaf/0x210
[ 14.020011] [<ffffffff8114e189>] handle_irq_event_percpu+0x39/0x190
[ 14.020011] [<ffffffff8114e323>] handle_irq_event+0x43/0x70
[ 14.020011] [<ffffffff811519b8>] handle_edge_irq+0xe8/0x120
[ 14.020011] [<ffffffff81050444>] handle_irq+0x164/0x180
[ 14.020011] [<ffffffff8104f638>] do_IRQ+0x58/0xd0
[ 14.020011] [<ffffffff828906af>] ret_from_intr+0x0/0x1a
[ 14.020011] [<ffffffff81c4aba3>] ata_sff_hsm_move+0x293/0x570
[ 14.020011] [<ffffffff81c4afde>] ata_sff_pio_task+0x15e/0x170
[ 14.020011] [<ffffffff810ce4d1>] process_one_work+0x291/0x440
[ 14.020011] [<ffffffff810d19b1>] worker_thread+0x211/0x3a0
[ 14.020011] [<ffffffff810d6c32>] kthread+0xb2/0xc0
[ 14.020011] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.020011] IN-SOFTIRQ-W at:
[ 14.020011] [<ffffffff81116995>] mark_irqflags+0x95/0x170
[ 14.020011] [<ffffffff8111837b>] __lock_acquire+0x2bb/0x4c0
[ 14.020011] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.020011] [<ffffffff8288f5fb>] _raw_spin_lock+0x3b/0x70
[ 14.020011] [<ffffffff81bd5c3d>] scsi_device_unbusy+0x9d/0xd0
[ 14.020011] [<ffffffff81bcd929>] scsi_finish_command+0x39/0x120
[ 14.106037] [<ffffffff81bd645d>] scsi_softirq_done+0x13d/0x160
[ 14.106037] [<ffffffff8184f5d2>] blk_done_softirq+0xc2/0xe0
[ 14.106037] [<ffffffff810ba388>] __do_softirq+0xc8/0x1c0
[ 14.106037] [<ffffffff8289256c>] call_softirq+0x1c/0x30
[ 14.106037] [<ffffffff81050235>] do_softirq+0x75/0x120
[ 14.106037] [<ffffffff810ba04b>] irq_exit+0x5b/0xf0
[ 14.106037] [<ffffffff8104f68d>] do_IRQ+0xad/0xd0
[ 14.106037] [<ffffffff828906af>] ret_from_intr+0x0/0x1a
[ 14.106037] [<ffffffff81c4aba3>] ata_sff_hsm_move+0x293/0x570
[ 14.106037] [<ffffffff81c4afde>] ata_sff_pio_task+0x15e/0x170
[ 14.106037] [<ffffffff810ce4d1>] process_one_work+0x291/0x440
[ 14.106037] [<ffffffff810d19b1>] worker_thread+0x211/0x3a0
[ 14.106037] [<ffffffff810d6c32>] kthread+0xb2/0xc0
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037] INITIAL USE at:
[ 14.106037] [<ffffffff811183a5>] __lock_acquire+0x2e5/0x4c0
[ 14.106037] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.106037] [<ffffffff8288f831>] _raw_spin_lock_irq+0x61/0xa0
[ 14.106037] [<ffffffff81847f56>] blk_queue_bypass_start+0x16/0x90
[ 14.106037] [<ffffffff818603fb>] blkcg_activate_policy+0x3b/0x330
[ 14.106037] [<ffffffff818614dd>] blk_throtl_init+0xdd/0x100
[ 14.106037] [<ffffffff8185faa7>] blkcg_init_queue+0x27/0x30
[ 14.106037] [<ffffffff81847014>] blk_alloc_queue_node+0x254/0x290
[ 14.106037] [<ffffffff81848416>] blk_init_queue_node+0x26/0x70
[ 14.106037] [<ffffffff8184846e>] blk_init_queue+0xe/0x10
[ 14.106037] [<ffffffff83bbc267>] floppy_init+0x82/0x662
[ 14.106037] [<ffffffff83b77629>] do_one_initcall+0x7a/0x155
[ 14.106037] [<ffffffff83b777cb>] do_basic_setup+0x9c/0xba
[ 14.106037] [<ffffffff83b77a42>] kernel_init+0x1fe/0x280
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037] }
[ 14.106037] ... key at: [<ffffffff847e2ec0>] __key.43131+0x0/0x8
[ 14.106037] ... acquired at:
[ 14.106037] [<ffffffff81117fce>] validate_chain+0x69e/0x790
[ 14.106037] [<ffffffff811184e3>] __lock_acquire+0x423/0x4c0
[ 14.106037] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.106037] [<ffffffff8288f791>] _raw_spin_lock_irqsave+0x81/0xc0
[ 14.106037] [<ffffffff818931d0>] __debug_check_no_obj_freed+0x70/0x1f0
[ 14.106037] [<ffffffff81893365>] debug_check_no_obj_freed+0x15/0x20
[ 14.106037] [<ffffffff811c87cf>] kmem_cache_free+0x12f/0x1f0
[ 14.106037] [<ffffffff8186de96>] ida_get_new_above+0x1e6/0x220
[ 14.106037] [<ffffffff8186e65c>] ida_simple_get+0xac/0x120
[ 14.106037] [<ffffffff81846e09>] blk_alloc_queue_node+0x49/0x290
[ 14.106037] [<ffffffff81848416>] blk_init_queue_node+0x26/0x70
[ 14.106037] [<ffffffff8184846e>] blk_init_queue+0xe/0x10
[ 14.106037] [<ffffffff83bbc267>] floppy_init+0x82/0x662
[ 14.106037] [<ffffffff83b77629>] do_one_initcall+0x7a/0x155
[ 14.106037] [<ffffffff83b777cb>] do_basic_setup+0x9c/0xba
[ 14.106037] [<ffffffff83b77a42>] kernel_init+0x1fe/0x280
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037]
[ 14.106037] -> (&(&blkcg->lock)->rlock){......} ops: 107 {
[ 14.106037] INITIAL USE at:
[ 14.106037] [<ffffffff811183a5>] __lock_acquire+0x2e5/0x4c0
[ 14.106037] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.106037] [<ffffffff8288f5fb>] _raw_spin_lock+0x3b/0x70
[ 14.106037] [<ffffffff818602e2>] __blkg_lookup_create+0x132/0x210
[ 14.106037] [<ffffffff8186047f>] blkcg_activate_policy+0xbf/0x330
[ 14.106037] [<ffffffff818614dd>] blk_throtl_init+0xdd/0x100
[ 14.106037] [<ffffffff8185faa7>] blkcg_init_queue+0x27/0x30
[ 14.106037] [<ffffffff81847014>] blk_alloc_queue_node+0x254/0x290
[ 14.106037] [<ffffffff81848416>] blk_init_queue_node+0x26/0x70
[ 14.106037] [<ffffffff8184846e>] blk_init_queue+0xe/0x10
[ 14.106037] [<ffffffff83bbc267>] floppy_init+0x82/0x662
[ 14.106037] [<ffffffff83b77629>] do_one_initcall+0x7a/0x155
[ 14.106037] [<ffffffff83b777cb>] do_basic_setup+0x9c/0xba
[ 14.106037] [<ffffffff83b77a42>] kernel_init+0x1fe/0x280
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037] }
[ 14.106037] ... key at: [<ffffffff847e38d0>] __key.39904+0x0/0x8
[ 14.106037] ... acquired at:
[ 14.106037] [<ffffffff81117fce>] validate_chain+0x69e/0x790
[ 14.106037] [<ffffffff811184e3>] __lock_acquire+0x423/0x4c0
[ 14.106037] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.106037] [<ffffffff8288f5fb>] _raw_spin_lock+0x3b/0x70
[ 14.106037] [<ffffffff8186135e>] throtl_pd_init+0x8e/0xd0
[ 14.106037] [<ffffffff81860643>] blkcg_activate_policy+0x283/0x330
[ 14.106037] [<ffffffff818614dd>] blk_throtl_init+0xdd/0x100
[ 14.106037] [<ffffffff8185faa7>] blkcg_init_queue+0x27/0x30
[ 14.106037] [<ffffffff81847014>] blk_alloc_queue_node+0x254/0x290
[ 14.106037] [<ffffffff81848416>] blk_init_queue_node+0x26/0x70
[ 14.106037] [<ffffffff8184846e>] blk_init_queue+0xe/0x10
[ 14.106037] [<ffffffff83bbc267>] floppy_init+0x82/0x662
[ 14.106037] [<ffffffff83b77629>] do_one_initcall+0x7a/0x155
[ 14.106037] [<ffffffff83b777cb>] do_basic_setup+0x9c/0xba
[ 14.106037] [<ffffffff83b77a42>] kernel_init+0x1fe/0x280
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037]
[ 14.106037] -> (tg_stats_alloc_lock){+.....} ops: 107 {
[ 14.106037] HARDIRQ-ON-W at:
[ 14.106037] [<ffffffff811169f0>] mark_irqflags+0xf0/0x170
[ 14.106037] [<ffffffff8111837b>] __lock_acquire+0x2bb/0x4c0
[ 14.106037] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.106037] [<ffffffff8288f5fb>] _raw_spin_lock+0x3b/0x70
[ 14.106037] [<ffffffff818611ab>] throtl_pd_exit+0x3b/0x70
[ 14.106037] [<ffffffff8185f6da>] blkg_free+0x4a/0x90
[ 14.106037] [<ffffffff8185f72d>] blkg_rcu_free+0xd/0x10
[ 14.106037] [<ffffffff81157d75>] rcu_do_batch+0x285/0x4e0
[ 14.106037] [<ffffffff811584f0>] rcu_cpu_kthread+0x250/0x290
[ 14.106037] [<ffffffff810d6c32>] kthread+0xb2/0xc0
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037] INITIAL USE at:
[ 14.106037] [<ffffffff811183a5>] __lock_acquire+0x2e5/0x4c0
[ 14.106037] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.106037] [<ffffffff8288f5fb>] _raw_spin_lock+0x3b/0x70
[ 14.106037] [<ffffffff8186135e>] throtl_pd_init+0x8e/0xd0
[ 14.106037] [<ffffffff81860643>] blkcg_activate_policy+0x283/0x330
[ 14.106037] [<ffffffff818614dd>] blk_throtl_init+0xdd/0x100
[ 14.106037] [<ffffffff8185faa7>] blkcg_init_queue+0x27/0x30
[ 14.106037] [<ffffffff81847014>] blk_alloc_queue_node+0x254/0x290
[ 14.106037] [<ffffffff81848416>] blk_init_queue_node+0x26/0x70
[ 14.106037] [<ffffffff8184846e>] blk_init_queue+0xe/0x10
[ 14.106037] [<ffffffff83bbc267>] floppy_init+0x82/0x662
[ 14.106037] [<ffffffff83b77629>] do_one_initcall+0x7a/0x155
[ 14.106037] [<ffffffff83b777cb>] do_basic_setup+0x9c/0xba
[ 14.106037] [<ffffffff83b77a42>] kernel_init+0x1fe/0x280
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037] }
[ 14.106037] ... key at: [<ffffffff83793a78>] tg_stats_alloc_lock+0x18/0x50
[ 14.106037] ... acquired at:
[ 14.106037] [<ffffffff81115a75>] check_usage_backwards+0xd5/0xf0
[ 14.106037] [<ffffffff81116583>] mark_lock_irq+0xf3/0x270
[ 14.106037] [<ffffffff8111681d>] mark_lock+0x11d/0x200
[ 14.106037] [<ffffffff811169f0>] mark_irqflags+0xf0/0x170
[ 14.106037] [<ffffffff8111837b>] __lock_acquire+0x2bb/0x4c0
[ 14.106037] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.106037] [<ffffffff8288f5fb>] _raw_spin_lock+0x3b/0x70
[ 14.106037] [<ffffffff818611ab>] throtl_pd_exit+0x3b/0x70
[ 14.106037] [<ffffffff8185f6da>] blkg_free+0x4a/0x90
[ 14.106037] [<ffffffff8185f72d>] blkg_rcu_free+0xd/0x10
[ 14.106037] [<ffffffff81157d75>] rcu_do_batch+0x285/0x4e0
[ 14.106037] [<ffffffff811584f0>] rcu_cpu_kthread+0x250/0x290
[ 14.106037] [<ffffffff810d6c32>] kthread+0xb2/0xc0
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037]
[ 14.106037]
[ 14.106037] stack backtrace:
[ 14.106037] Pid: 7, comm: rcuc/0 Tainted: G W 3.4.0-rc4-next-20120424-sasha #89
[ 14.106037] Call Trace:
[ 14.106037] [<ffffffff81115583>] print_irq_inversion_bug+0x203/0x220
[ 14.106037] [<ffffffff81115a75>] check_usage_backwards+0xd5/0xf0
[ 14.106037] [<ffffffff8107e4a8>] ? pvclock_clocksource_read+0x58/0xd0
[ 14.106037] [<ffffffff811159a0>] ? check_usage_forwards+0xf0/0xf0
[ 14.106037] [<ffffffff81116583>] mark_lock_irq+0xf3/0x270
[ 14.106037] [<ffffffff8111681d>] mark_lock+0x11d/0x200
[ 14.106037] [<ffffffff811169f0>] mark_irqflags+0xf0/0x170
[ 14.106037] [<ffffffff8111837b>] __lock_acquire+0x2bb/0x4c0
[ 14.106037] [<ffffffff8111865c>] lock_acquire+0xdc/0x120
[ 14.106037] [<ffffffff818611ab>] ? throtl_pd_exit+0x3b/0x70
[ 14.106037] [<ffffffff8288f5fb>] _raw_spin_lock+0x3b/0x70
[ 14.106037] [<ffffffff818611ab>] ? throtl_pd_exit+0x3b/0x70
[ 14.106037] [<ffffffff81892e6f>] ? debug_object_deactivate+0xdf/0x140
[ 14.106037] [<ffffffff818611ab>] throtl_pd_exit+0x3b/0x70
[ 14.106037] [<ffffffff8185f6da>] blkg_free+0x4a/0x90
[ 14.106037] [<ffffffff8185f72d>] blkg_rcu_free+0xd/0x10
[ 14.106037] [<ffffffff81157d75>] rcu_do_batch+0x285/0x4e0
[ 14.106037] [<ffffffff811584f0>] rcu_cpu_kthread+0x250/0x290
[ 14.106037] [<ffffffff811582a0>] ? rcu_process_callbacks+0x80/0x80
[ 14.106037] [<ffffffff810d6c32>] kthread+0xb2/0xc0
[ 14.106037] [<ffffffff82892474>] kernel_thread_helper+0x4/0x10
[ 14.106037] [<ffffffff810e409d>] ? finish_task_switch+0x8d/0x110
[ 14.106037] [<ffffffff82890774>] ? retint_restore_args+0x13/0x13
[ 14.106037] [<ffffffff810d6b80>] ? __init_kthread_worker+0x70/0x70
[ 14.106037] [<ffffffff82892470>] ? gs_change+0x13/0x13
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists