| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Fri, 16 May 2014 10:13:44 +0800
From: Jet Chen <jet.chen@...el.com>
To: Hong Zhiguo <zhiguohong@...cent.com>
CC: Jens Axboe <axboe@...nel.dk>,
Fengguang Wu <fengguang.wu@...el.com>,
Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: [blk] WARNING: CPU: 0 PID: 305 at kernel/softirq.c:156 local_bh_enable()
Hi Zhiguo,
0day kernel testing robot got the below dmesg and the first bad commit is
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit 2c575026fae6e63771bd2a4c1d407214a8096a89
Author: Hong Zhiguo <zhiguohong@...cent.com>
AuthorDate: Wed Nov 20 10:35:05 2013 -0700
Commit: Jens Axboe <axboe@...nel.dk>
CommitDate: Wed Nov 20 15:33:04 2013 -0700
Update of blkg_stat and blkg_rwstat may happen in bh context.
While u64_stats_fetch_retry is only preempt_disable on 32bit
UP system. This is not enough to avoid preemption by bh and
may read strange 64 bit value.
Signed-off-by: Hong Zhiguo <zhiguohong@...cent.com>
Acked-by: Tejun Heo <tj@...nel.org>
Cc: stable@...nel.org
Signed-off-by: Jens Axboe <axboe@...nel.dk>
+-----------------------------------------------------------------------------+------------+------------+
| | 82023bb7f7 | 2c575026fa |
+-----------------------------------------------------------------------------+------------+------------+
| boot_successes | 56 | 19 |
| boot_failures | 73 | 24 |
| Out_of_memory:Kill_process | 2 | 2 |
| BUG:unable_to_handle_kernel_paging_request | 71 | 22 |
| Oops | 71 | 22 |
| EIP_is_at_process_one_work | 70 | 21 |
| BUG:sleeping_function_called_from_invalid_context_at_kernel/locking/rwsem.c | 70 | 22 |
| INFO:lockdep_is_turned_off | 70 | 22 |
| EIP_is_at_kthread_data | 68 | 22 |
| EIP_is_at_insert_work | 1 | |
| Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 1 | |
| WARNING:CPU:PID:at_kernel/softirq.c:local_bh_enable() | 0 | 22 |
| backtrace:do_mount | 0 | 19 |
| backtrace:SyS_mount | 0 | 19 |
| inconsistent_SOFTIRQ-ON-W-IN-SOFTIRQ-W_usage | 0 | 8 |
| backtrace:redo_fd_request | 0 | 8 |
| inconsistent_IN-SOFTIRQ-W-SOFTIRQ-ON-W_usage | 0 | 6 |
| EIP_is_at_worker_thread | 0 | 1 |
| backtrace:SyS_mprotect | 0 | 1 |
| backtrace:error_code | 0 | 1 |
| backtrace:smpboot_thread_fn | 0 | 1 |
+-----------------------------------------------------------------------------+------------+------------+
it seems like your commit introduces "WARNING:CPU:PID:at_kernel/softirq.c:local_bh_enable()" oops messages.
[ 8.933738] udev[102]: starting version 164
[ 10.273305] scsi_id (215) used greatest stack depth: 6328 bytes left
[ 11.860972] ------------[ cut here ]------------
[ 11.862089] WARNING: CPU: 0 PID: 305 at kernel/softirq.c:156 local_bh_enable+0xa1/0x1a7()
[ 11.864304] CPU: 0 PID: 305 Comm: mount Not tainted 3.12.0-10878-g2c57502 #1
[ 11.865769] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 11.866917] 00000000 caa03b74 c1db7b69 caa03b8c c104fec1 c10588c1 c15aaf27 cc1fa908
[ 11.870084] cd25dc00 caa03b9c c105003b 00000009 00000000 caa03ba8 c10588c1 cd25be00
[ 11.870084] caa03c50 c15aaf27 00000000 00000000 00000001 00000000 cc1f002f cd25be00
[ 11.870084] Call Trace:
[ 11.870084] [<c1db7b69>] dump_stack+0x40/0x5e
[ 11.870084] [<c104fec1>] warn_slowpath_common+0xae/0xef
[ 11.870084] [<c10588c1>] ? local_bh_enable+0xa1/0x1a7
[ 11.870084] [<c15aaf27>] ? __cfq_set_active_queue+0x1c9/0x359
[ 11.870084] [<c105003b>] warn_slowpath_null+0x30/0x42
[ 11.870084] [<c10588c1>] local_bh_enable+0xa1/0x1a7
[ 11.870084] [<c15aaf27>] __cfq_set_active_queue+0x1c9/0x359
[ 11.870084] [<c15adb25>] ? cfq_add_cfqq_rr+0x185/0x1ac
[ 11.870084] [<c15a9b70>] ? cfq_update_idle_window+0x105/0x2ac
[ 11.870084] [<c10ad526>] ? mark_held_locks+0x163/0x199
[ 11.870084] [<c1dc8b2b>] ? _raw_spin_unlock_irq+0x5f/0x98
[ 11.870084] [<c10ad964>] ? trace_hardirqs_on+0x27/0x37
[ 11.870084] [<c1dc8b2b>] ? _raw_spin_unlock_irq+0x5f/0x98
[ 11.870084] [<c15b24a6>] ? cfq_set_request+0x575/0x5e1
[ 11.870084] [<c15ab6aa>] ? cfq_get_next_cfqg+0xe/0x12b
[ 11.870084] [<c15b3255>] cfq_dispatch_requests+0xbdb/0x12ce
[ 11.870084] [<c15b0c14>] ? cfq_insert_request+0x3f7/0x87c
[ 11.870084] [<c10b1a67>] ? __lock_acquire+0x14f6/0x1606
[ 11.870084] [<c10af0ff>] ? __lock_is_held+0x46/0x7b
[ 11.870084] [<c1585f34>] blk_peek_request+0xa3/0x522
[ 11.870084] [<c15863cf>] blk_fetch_request+0x1c/0x56
[ 11.870084] [<c1760558>] do_nbd_request+0x23/0x1c9
[ 11.870084] [<c158178f>] __blk_run_queue+0x5a/0x71
[ 11.870084] [<c1587349>] blk_queue_bio+0x4cc/0x858
[ 11.870084] [<c15838cb>] generic_make_request+0xdb/0x158
[ 11.870084] [<c15839f5>] submit_bio+0xad/0x1d7
[ 11.870084] [<c120dc16>] ? bio_init+0x9/0x35
[ 11.870084] [<c120ddb1>] ? bio_alloc_bioset+0x16f/0x275
[ 11.870084] [<c1208cba>] _submit_bh+0x304/0x34a
[ 11.870084] [<c1208d1d>] submit_bh+0x1d/0x2d
[ 11.870084] [<c120cf56>] __bread+0x11d/0x1c1
[ 11.870084] [<c12a1d46>] ext4_fill_super+0x354/0x446c
[ 11.870084] [<c15c78a8>] ? strnlen+0x8/0x3a
[ 11.870084] [<c15c27b1>] ? string.isra.3+0x45/0x11d
[ 11.870084] [<c15c48e8>] ? snprintf+0x23/0x33
[ 11.870084] [<c1211d7d>] ? set_blocksize+0xe/0x15c
[ 11.870084] [<c159da6d>] ? disk_name+0x33/0xd7
[ 11.870084] [<c11b7957>] mount_bdev+0x220/0x2e9
[ 11.870084] [<c12a19f2>] ? ext4_calculate_overhead+0x512/0x512
[ 11.870084] [<c11b7957>] ? mount_bdev+0x220/0x2e9
[ 11.870084] [<c12a19f2>] ? ext4_calculate_overhead+0x512/0x512
[ 11.870084] [<c11a96c4>] ? __kmalloc_track_caller+0x260/0x4ec
[ 11.870084] [<c11e9180>] ? alloc_vfsmnt+0x19d/0x213
[ 11.870084] [<c129bb1d>] ext4_mount+0x23/0x33
[ 11.870084] [<c12a19f2>] ? ext4_calculate_overhead+0x512/0x512
[ 11.870084] [<c11b7d02>] mount_fs+0x21/0x149
[ 11.870084] [<c11ea5a3>] vfs_kern_mount+0x93/0x1db
[ 11.870084] [<c11ee3a6>] do_mount+0x1167/0x1372
[ 11.870084] [<c11ed03e>] ? copy_mount_options+0xe/0x1a6
[ 11.870084] [<c11ee9ad>] SyS_mount+0x101/0x184
[ 11.870084] [<c1dc9f47>] syscall_call+0x7/0xb
[ 11.870084] ---[ end trace 6f4587f6d881af3f ]---
[ 12.147799] block nbd15: Attempted send on closed socket
Attached dmesg for the parent commit, too, to help confirm whether it is a noise error.
git bisect start v3.13 v3.12 --
git bisect good 3bad8bb5cd3048a67df43ac6b1e2f191f19d9ff0 # 00:07 43+ 22 Merge branch 'for-next' of git://git.samba.org/sfrench/cifs-2.6
git bisect bad dd0508093b79141e0044ca02f0acb6319f69f546 # 00:29 1- 20 Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 3f02ff5c2c69753666787ed125708d283a823ffb # 00:40 0- 1 Merge tag 'tty-3.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
git bisect good 1ab231b274ba51a54acebec23c6aded0f3cdf54e # 00:50 43+ 27 Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 5ee540613db504a10e15fafaf4c08cac96aa1823 # 00:54 5- 14 Merge branch 'for-linus' of git://git.kernel.dk/linux-block
git bisect good 53c6de50262a8edd6932bb59a32db7b9d92f8d67 # 01:02 43+ 20 Merge branch 'x86/urgent' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 59fb2f0e9e30ad99a8bab0ff1efaf8f4a3b7105f # 01:14 43+ 13 Merge tag 'fbdev-fixes-3.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux
git bisect good ef1e4e32d595d3e6c9a6d3d2956f087d5886c5e5 # 01:31 43+ 26 Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good 29be6345bbaec8502a70c4e2204d5818b48c4e8f # 01:45 43+ 31 Merge tag 'nfs-for-3.13-3' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect bad e345d767f6530ec9cb0aabab7ea248072a9c6975 # 02:02 1- 25 Merge branch 'stable/for-jens-3.13-take-two' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into for-linus
git bisect bad c170bbb45febc03ac4d34ba2b8bb55e06104b7e7 # 02:23 6- 23 block: submit_bio_wait() conversions
git bisect bad 2c575026fae6e63771bd2a4c1d407214a8096a89 # 02:33 3- 10 Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value.
# first bad commit: [2c575026fae6e63771bd2a4c1d407214a8096a89] Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value.
git bisect good 82023bb7f75b0052f40d3e74169d191c3e4e6286 # 02:44 129+ 73 Merge tag 'pm+acpi-2-3.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect bad b2fef0d0972b3fa998a96a5eb07904575b196738 # 02:44 0- 13 Add linux-next specific files for 20140515
git bisect bad 14186fea0cb06bc43181ce239efe0df6f1af260a # 02:50 80- 104 Merge tag 'locks-v3.15-4' of git://git.samba.org/jlayton/linux
git bisect bad b2fef0d0972b3fa998a96a5eb07904575b196738 # 02:50 0- 13 Add linux-next specific files for 20140515
This script may reproduce the error.
-----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
initrd=yocto-minimal-i386.cgz
wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/blob/master/initrd/$initrd
kvm=(
qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel $kernel
-initrd $initrd
-smp 2
-m 256M
-net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
-net user,vlan=0
-net nic,vlan=1,model=e1000
-net user,vlan=1
-boot order=nc
-no-reboot
-watchdog i6300esb
-serial stdio
-display none
-monitor null
)
append=(
debug
sched_debug
apic=debug
ignore_loglevel
sysrq_always_enabled
panic=10
prompt_ramdisk=0
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
)
"${kvm[@]}" --append "${append[*]}"
-----------------------------------------------------------------------------
Thanks,
Jet
View attachment "dmesg-yocto-f2-48:20140516023254:i386-randconfig-tt0-05152256:3.12.0-10878-g2c57502:1" of type "text/plain" (51573 bytes)
View attachment "dmesg-quantal-f1-92:20140516023933:i386-randconfig-tt0-05152256:3.12.0-10312-g82023bb:1" of type "text/plain" (31902 bytes)
Download attachment "i386-randconfig-tt0-05152256-b2fef0d0972b3fa998a96a5eb07904575b196738-inconsistent----usage--120337.log" of type "application/octet-stream" (77213 bytes)
View attachment "config-3.12.0-10878-g2c57502" of type "text/plain" (71764 bytes)
View attachment "Attached Message Part" of type "text/plain" (87 bytes)
Powered by blists - more mailing lists