| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20140308074243.GB30311@localhost>
Date: Sat, 8 Mar 2014 15:42:43 +0800
From: Fengguang Wu <fengguang.wu@...el.com>
To: Hong Zhiguo <zhiguohong@...cent.com>
Cc: Jens Axboe <axboe@...nel.dk>, Tejun Heo <tj@...nel.org>,
linux-kernel@...r.kernel.org
Subject: [blkg] WARNING: CPU: 0 PID: 298 at kernel/softirq.c:145
__local_bh_enable_ip()
Hi Hong Zhiguo,
The WARNINGs introduced by this commit are still in linux-next 20140307.
commit 2c575026fae6e63771bd2a4c1d407214a8096a89
Author: Hong Zhiguo <zhiguohong@...cent.com>
AuthorDate: Wed Nov 20 10:35:05 2013 -0700
Commit: Jens Axboe <axboe@...nel.dk>
CommitDate: Wed Nov 20 15:33:04 2013 -0700
Update of blkg_stat and blkg_rwstat may happen in bh context.
While u64_stats_fetch_retry is only preempt_disable on 32bit
UP system. This is not enough to avoid preemption by bh and
may read strange 64 bit value.
Signed-off-by: Hong Zhiguo <zhiguohong@...cent.com>
Acked-by: Tejun Heo <tj@...nel.org>
Cc: stable@...nel.org
Signed-off-by: Jens Axboe <axboe@...nel.dk>
+------------------------------------------------------------+------------+---------------+
| | 2c575026fa | next-20140306 |
+------------------------------------------------------------+------------+---------------+
| boot_successes | 0 | 0 |
| boot_failures | 20 | 19 |
| WARNING:CPU:PID:at_kernel/softirq.c:local_bh_enable() | 20 | |
| inconsistent_IN-SOFTIRQ-W-SOFTIRQ-ON-W_usage | 20 | |
| backtrace:vfs_read | 20 | 18 |
| backtrace:SyS_read | 20 | 18 |
| backtrace:smpboot_thread_fn | 14 | |
| page_allocation_failure:order:,mode | 1 | |
| backtrace:vmalloc | 1 | |
| backtrace:SyS_init_module | 1 | |
| WARNING:CPU:PID:at_kernel/softirq.c:__local_bh_enable_ip() | 0 | 16 |
| inconsistent_IN-HARDIRQ-W-HARDIRQ-ON-W_usage | 0 | 19 |
| Out_of_memory:Kill_process | 0 | 1 |
| backtrace:do_mount | 0 | 1 |
| backtrace:SyS_mount | 0 | 1 |
| BUG:kernel_boot_hang | 0 | 1 |
+------------------------------------------------------------+------------+---------------+
[ 8.215262] ------------[ cut here ]------------
[ 8.215318] WARNING: CPU: 0 PID: 287 at /kernel-tests/linux4/kernel/softirq.c:156 local_bh_enable+0xbf/0x20e()
[ 8.215331] Modules linked in:
[ 8.215338] CPU: 0 PID: 287 Comm: blkid Not tainted 3.12.0-10878-g2c57502 #109
[ 8.215340] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 8.215355] 00000000 c53a5bd0 c152b622 c53a5be8 c0259135 c025fdb2 c068e196 00000000
[ 8.215364] c56dac00 c53a5bf8 c02591a1 00000009 00000000 c53a5c04 c025fdb2 00000000
[ 8.215372] c53a5cac c068e196 00000001 00000000 c56d6000 c57ba840 c56dac00 c068002f
[ 8.215374] Call Trace:
[ 8.215403] [<c152b622>] dump_stack+0x40/0x5e
[ 8.215408] [<c0259135>] warn_slowpath_common+0xa9/0xea
[ 8.215413] [<c025fdb2>] ? local_bh_enable+0xbf/0x20e
[ 8.215435] [<c068e196>] ? __cfq_set_active_queue+0x110/0x27d
[ 8.215440] [<c02591a1>] warn_slowpath_null+0x2b/0x3d
[ 8.215444] [<c025fdb2>] local_bh_enable+0xbf/0x20e
[ 8.215449] [<c068e196>] __cfq_set_active_queue+0x110/0x27d
[ 8.215454] [<c068002f>] ? add_disk+0x12c/0x781
[ 8.215459] [<c06907a9>] ? cfq_resort_rr_list+0x49/0x5b
[ 8.215463] [<c0690899>] ? cfq_add_cfqq_rr+0xde/0xf3
[ 8.215468] [<c0690c67>] ? cfq_update_idle_window.isra.78+0xf2/0x1f4
[ 8.215480] [<c0294d19>] ? sched_clock_cpu+0x1c4/0x1f9
[ 8.215487] [<c029fb08>] ? trace_hardirqs_off+0x27/0x37
[ 8.215492] [<c0294e4d>] ? local_clock+0xff/0x176
[ 8.215497] [<c06951f7>] cfq_dispatch_requests+0xb92/0x11b0
[ 8.215504] [<c024812f>] ? kvm_clock_read+0x30/0x55
[ 8.215515] [<c020f559>] ? sched_clock+0x3e/0x50
[ 8.215520] [<c06931f4>] ? cfq_insert_request+0x3c9/0x869
[ 8.215525] [<c029fb08>] ? trace_hardirqs_off+0x27/0x37
[ 8.215530] [<c02a77fb>] ? __lock_acquire+0x15a4/0x16ca
[ 8.215538] [<c06aec9f>] ? list_sort+0x1fb/0x253
[ 8.215547] [<c066e10c>] blk_peek_request+0x9e/0x31a
[ 8.215566] [<c0b523bf>] scsi_request_fn+0x62/0x89e
[ 8.215571] [<c066e8fd>] ? blk_flush_plug_list+0x2a4/0x447
[ 8.215577] [<c0668eec>] __blk_run_queue+0x55/0x6c
[ 8.215582] [<c066c588>] queue_unplugged.isra.29+0x40/0x6a
[ 8.215586] [<c066e9bf>] blk_flush_plug_list+0x366/0x447
[ 8.215591] [<c066eabb>] blk_finish_plug+0x1b/0x51
[ 8.215607] [<c03275e3>] __do_page_cache_readahead+0x314/0x364
[ 8.215614] [<c032772f>] force_page_cache_readahead+0x9e/0xea
[ 8.215620] [<c0327d36>] page_cache_sync_readahead+0x5d/0x91
[ 8.215624] [<c0319608>] generic_file_aio_read+0x393/0xb7e
[ 8.215638] [<c03c7bb9>] blkdev_aio_read+0x95/0xb1
[ 8.215646] [<c0375ec0>] do_sync_read+0x5a/0xab
[ 8.215651] [<c0375e66>] ? do_sync_write+0xab/0xab
[ 8.215655] [<c0376b3b>] vfs_read+0xd3/0x1ee
[ 8.215659] [<c0376e5f>] SyS_read+0x73/0xe8
[ 8.215668] [<c154488a>] sysenter_do_call+0x12/0x3c
[ 8.215672] ---[ end trace 6f72697fe2d05017 ]---
[ 8.215727]
[ 8.215729] =================================
[ 8.215730] [ INFO: inconsistent lock state ]
[ 8.215733] 3.12.0-10878-g2c57502 #109 Tainted: G W
[ 8.215735] ---------------------------------
[ 8.215737] inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
[ 8.215740] blkid/287 [HC0[0]:SC0[0]:HE1:SE1] takes:
[ 8.215749] (&(&q->__queue_lock)->rlock){+.?...}, at: [<c066e8fd>] blk_flush_plug_list+0x2a4/0x447
[ 8.215750] {IN-SOFTIRQ-W} state was registered at:
[ 8.215756] [<c02a7102>] __lock_acquire+0xeab/0x16ca
[ 8.215760] [<c02a7a07>] lock_acquire+0xe6/0x147
[ 8.215767] [<c1542430>] _raw_spin_lock+0x4e/0xee
[ 8.215771] [<c0b52ce9>] scsi_device_unbusy+0xee/0x134
[ 8.215781] [<c0b47e22>] scsi_finish_command+0x2b/0x107
[ 8.215785] [<c0b5303f>] scsi_softirq_done+0x1b5/0x1cb
[ 8.215791] [<c067666b>] blk_done_softirq+0x10e/0x133
[ 8.215796] [<c025f5db>] __do_softirq+0x114/0x318
[ 8.215798] irq event stamp: 2152
[ 8.215803] hardirqs last enabled at (2149): [<c1542a61>] _raw_spin_unlock_irq+0x5a/0xba
[ 8.215808] hardirqs last disabled at (2150): [<c066e858>] blk_flush_plug_list+0x1ff/0x447
[ 8.215813] softirqs last enabled at (2152): [<c068e196>] __cfq_set_active_queue+0x110/0x27d
[ 8.215818] softirqs last disabled at (2151): [<c068e15f>] __cfq_set_active_queue+0xd9/0x27d
[ 8.215820]
[ 8.215820] other info that might help us debug this:
[ 8.215824] Possible unsafe locking scenario:
[ 8.215824]
[ 8.215825] CPU0
[ 8.215827] ----
[ 8.215830] lock(&(&q->__queue_lock)->rlock);
[ 8.215831] <Interrupt>
[ 8.215833] lock(&(&q->__queue_lock)->rlock);
[ 8.215834]
[ 8.215834] *** DEADLOCK ***
[ 8.215834]
[ 8.215837] 1 lock held by blkid/287:
[ 8.215844] #0: (&(&q->__queue_lock)->rlock){+.?...}, at: [<c066e8fd>] blk_flush_plug_list+0x2a4/0x447
[ 8.215846]
[ 8.215846] stack backtrace:
[ 8.215849] CPU: 0 PID: 287 Comm: blkid Tainted: G W 3.12.0-10878-g2c57502 #109
[ 8.215851] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 8.215859] 00000000 c53a5b78 c152b622 c53a5b98 c15263b3 c1a4e107 c1a4dedf c1a4ed62
[ 8.215867] 00000006 c535a8b0 00000040 c53a5bc0 c02a5d9e 00000006 00000000 c02a2e66
[ 8.215875] 00000004 c535a530 c535a8b0 c535a530 00000000 c53a5be0 c02a92e0 e2d05017
[ 8.215876] Call Trace:
[ 8.215881] [<c152b622>] dump_stack+0x40/0x5e
[ 8.215891] [<c15263b3>] print_usage_bug+0x493/0x4ad
[ 8.215896] [<c02a5d9e>] mark_lock+0x575/0xa2e
[ 8.215901] [<c02a2e66>] ? print_irq_inversion_bug+0x374/0x374
[ 8.215905] [<c02a92e0>] mark_held_locks+0x163/0x199
[ 8.215910] [<c025feb5>] ? local_bh_enable+0x1c2/0x20e
[ 8.215915] [<c02a9681>] trace_hardirqs_on_caller+0x36b/0x3b1
[ 8.215920] [<c068e196>] ? __cfq_set_active_queue+0x110/0x27d
[ 8.215924] [<c02a96ee>] trace_hardirqs_on+0x27/0x37
[ 8.215928] [<c025feb5>] local_bh_enable+0x1c2/0x20e
[ 8.215932] [<c068e196>] __cfq_set_active_queue+0x110/0x27d
[ 8.215937] [<c068002f>] ? add_disk+0x12c/0x781
[ 8.215941] [<c06907a9>] ? cfq_resort_rr_list+0x49/0x5b
[ 8.215945] [<c0690899>] ? cfq_add_cfqq_rr+0xde/0xf3
[ 8.215949] [<c0690c67>] ? cfq_update_idle_window.isra.78+0xf2/0x1f4
[ 8.215953] [<c0294d19>] ? sched_clock_cpu+0x1c4/0x1f9
[ 8.215958] [<c029fb08>] ? trace_hardirqs_off+0x27/0x37
[ 8.215962] [<c0294e4d>] ? local_clock+0xff/0x176
[ 8.215966] [<c06951f7>] cfq_dispatch_requests+0xb92/0x11b0
[ 8.215970] [<c024812f>] ? kvm_clock_read+0x30/0x55
[ 8.215975] [<c020f559>] ? sched_clock+0x3e/0x50
[ 8.215979] [<c06931f4>] ? cfq_insert_request+0x3c9/0x869
[ 8.215984] [<c029fb08>] ? trace_hardirqs_off+0x27/0x37
[ 8.215988] [<c02a77fb>] ? __lock_acquire+0x15a4/0x16ca
[ 8.215992] [<c06aec9f>] ? list_sort+0x1fb/0x253
[ 8.215997] [<c066e10c>] blk_peek_request+0x9e/0x31a
[ 8.216001] [<c0b523bf>] scsi_request_fn+0x62/0x89e
[ 8.216006] [<c066e8fd>] ? blk_flush_plug_list+0x2a4/0x447
[ 8.216010] [<c0668eec>] __blk_run_queue+0x55/0x6c
[ 8.216015] [<c066c588>] queue_unplugged.isra.29+0x40/0x6a
[ 8.216019] [<c066e9bf>] blk_flush_plug_list+0x366/0x447
[ 8.216024] [<c066eabb>] blk_finish_plug+0x1b/0x51
[ 8.216029] [<c03275e3>] __do_page_cache_readahead+0x314/0x364
[ 8.216035] [<c032772f>] force_page_cache_readahead+0x9e/0xea
[ 8.216040] [<c0327d36>] page_cache_sync_readahead+0x5d/0x91
[ 8.216044] [<c0319608>] generic_file_aio_read+0x393/0xb7e
git bisect start v3.13 v3.12 --
git bisect good 3bad8bb5cd3048a67df43ac6b1e2f191f19d9ff0 # 05:30 20+ 0 Merge branch 'for-next' of git://git.samba.org/sfrench/cifs-2.6
git bisect bad 754ac45745711a7ce0067087e5b55a9c277952ad # 05:52 0- 4 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good d644c026e6b21959af1f86b15206ae26c64c2e51 # 06:06 20+ 0 Merge git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
git bisect bad 002acf1fc16cf60e60345bd68e03734628505b83 # 06:19 0- 2 Merge tag 'pm-3.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect good da1965d353c7266fce4fe6c658f2096c88a8bebc # 06:31 20+ 0 Merge tag 'pinctrl-v3.13-2' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl
git bisect good ef1e4e32d595d3e6c9a6d3d2956f087d5886c5e5 # 06:45 20+ 0 Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect bad 5ee540613db504a10e15fafaf4c08cac96aa1823 # 06:58 0- 3 Merge branch 'for-linus' of git://git.kernel.dk/linux-block
git bisect good 29be6345bbaec8502a70c4e2204d5818b48c4e8f # 07:06 20+ 0 Merge tag 'nfs-for-3.13-3' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect bad e345d767f6530ec9cb0aabab7ea248072a9c6975 # 07:26 0- 5 Merge branch 'stable/for-jens-3.13-take-two' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into for-linus
git bisect bad c170bbb45febc03ac4d34ba2b8bb55e06104b7e7 # 07:35 0- 3 block: submit_bio_wait() conversions
git bisect bad 2c575026fae6e63771bd2a4c1d407214a8096a89 # 07:43 0- 4 Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value.
# first bad commit: [2c575026fae6e63771bd2a4c1d407214a8096a89] Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value.
git bisect good 82023bb7f75b0052f40d3e74169d191c3e4e6286 # 08:02 60+ 0 Merge tag 'pm+acpi-2-3.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect bad 0ffb2fe7b9c30082876fa3a17da018bf0632cf03 # 08:02 0- 19 Add linux-next specific files for 20140306
git bisect good 538959a390127125581a4fa2f2f7e035b82fa421 # 08:25 60+ 37 Revert "Update of blkg_stat and blkg_rwstat may happen in bh context."
git bisect bad b053940df41808f0f27568eb36820d10a8a987f8 # 08:34 0- 2 ARC: Use correct PTAG register for icache flush
git bisect bad 1b0a7e3263168a06d3858798e48c5a21d1c78d3c # 08:44 1- 16 Add linux-next specific files for 20140307
Thanks,
Fengguang
View attachment "dmesg-yocto-xian-9:20140307071216:i386-randconfig-x1-03070634:3.14.0-rc5-next-20140306-06952-g0ffb2fe:2" of type "text/plain" (60278 bytes)
Download attachment "i386-randconfig-x1-03070634-0ffb2fe7b9c30082876fa3a17da018bf0632cf03-inconsistent----usage--98979.log" of type "application/octet-stream" (64291 bytes)
View attachment "config-3.14.0-rc5-next-20140306-06952-g0ffb2fe" of type "text/plain" (107639 bytes)
View attachment "dmesg-quantal-snb-14:20140308074313:i386-randconfig-x1-03070634::" of type "text/plain" (60783 bytes)
Powered by blists - more mailing lists