| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Sun, 5 Mar 2017 10:21:11 +0800
From: Fengguang Wu <fengguang.wu@...el.com>
To: Jan Kara <jack@...e.cz>
Cc: Jens Axboe <axboe@...nel.dk>, linux-block@...r.kernel.org,
linux-kernel@...r.kernel.org, LKP <lkp@...org>
Subject: [bdi_unregister] 165a5e22fa INFO: task swapper:1 blocked for more
than 120 seconds.
Hi Jan,
FYI next-20170303 is good while mainline is bad with this error.
The attached reproduce-* may help reproduce the issue.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit 165a5e22fafb127ecb5914e12e8c32a1f0d3f820
Author: Jan Kara <jack@...e.cz>
AuthorDate: Wed Feb 8 08:05:56 2017 +0100
Commit: Jens Axboe <axboe@...com>
CommitDate: Thu Mar 2 16:08:35 2017 -0700
block: Move bdi_unregister() to del_gendisk()
Commit 6cd18e711dd8 "block: destroy bdi before blockdev is
unregistered." moved bdi unregistration (at that time through
bdi_destroy()) from blk_release_queue() to blk_cleanup_queue() because
it needs to happen before blk_unregister_region() call in del_gendisk()
for MD. SCSI though will free up the device number from sd_remove()
called through a maze of callbacks from device_del() in
__scsi_remove_device() before blk_cleanup_queue() and thus similar races
as described in 6cd18e711dd8 can happen for SCSI as well as reported by
Omar [1].
Moving bdi_unregister() to del_gendisk() works for MD and fixes the
problem for SCSI since del_gendisk() gets called from sd_remove() before
freeing the device number.
This also makes device_add_disk() (calling bdi_register_owner()) more
symmetric with del_gendisk().
[1] http://marc.info/?l=linux-block&m=148554717109098&w=2
Tested-by: Lekshmi Pillai <lekshmicpillai@...ibm.com>
Acked-by: Tejun Heo <tj@...nel.org>
Signed-off-by: Jan Kara <jack@...e.cz>
Tested-by: Omar Sandoval <osandov@...com>
Signed-off-by: Jens Axboe <axboe@...com>
113285b473 blk-mq: ensure that bd->last is always set correctly
165a5e22fa block: Move bdi_unregister() to del_gendisk()
+--------------------------------------------------+------------+------------+
| | 113285b473 | 165a5e22fa |
+--------------------------------------------------+------------+------------+
| boot_successes | 35 | 0 |
| boot_failures | 0 | 15 |
| INFO:task_blocked_for_more_than#seconds | 0 | 15 |
| Kernel_panic-not_syncing:hung_task:blocked_tasks | 0 | 15 |
+--------------------------------------------------+------------+------------+
[ 187.365491] rcu_preempt: wait state: 1 ->state: 0x1
[ 187.366039] rcu_sched: wait state: 1 ->state: 0x1
[ 187.366039] rcu_sched: wait state: 1 ->state: 0x1
[ 187.366564] rcu_bh: wait state: 1 ->state: 0x1
[ 187.366564] rcu_bh: wait state: 1 ->state: 0x1
[ 243.680303] INFO: task swapper:1 blocked for more than 120 seconds.
[ 243.680303] INFO: task swapper:1 blocked for more than 120 seconds.
[ 243.682035] Not tainted 4.10.0-11089-g165a5e2 #2
[ 243.682035] Not tainted 4.10.0-11089-g165a5e2 #2
[ 243.683000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 243.683000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 243.684306] swapper D 0 1 0 0x00000000
[ 243.684306] swapper D 0 1 0 0x00000000
[ 243.685246] Call Trace:
[ 243.685246] Call Trace:
[ 243.685650] __schedule+0x31a/0xc00
[ 243.685650] __schedule+0x31a/0xc00
[ 243.686236] schedule+0x7b/0x170
[ 243.686236] schedule+0x7b/0x170
[ 243.686803] bdi_unregister+0x3a3/0x430
[ 243.686803] bdi_unregister+0x3a3/0x430
[ 243.687447] ? woken_wake_function+0x40/0x40
[ 243.687447] ? woken_wake_function+0x40/0x40
[ 243.688150] del_gendisk+0x232/0x370
[ 243.688150] del_gendisk+0x232/0x370
[ 243.688721] sd_remove+0x97/0x140
[ 243.688721] sd_remove+0x97/0x140
[ 243.689260] device_release_driver_internal+0x4b8/0x4d0
[ 243.689260] device_release_driver_internal+0x4b8/0x4d0
[ 243.690065] device_release_driver+0x1a/0x30
[ 243.690065] device_release_driver+0x1a/0x30
[ 243.690928] bus_remove_device+0x15a/0x1f0
[ 243.690928] bus_remove_device+0x15a/0x1f0
[ 243.691558] device_del+0x34d/0x5a0
[ 243.691558] device_del+0x34d/0x5a0
[ 243.692155] ? kobject_put+0xf6/0x1d0
[ 243.692155] ? kobject_put+0xf6/0x1d0
[ 243.692700] __scsi_remove_device+0x1a3/0x1c0
[ 243.692700] __scsi_remove_device+0x1a3/0x1c0
[ 243.693665] scsi_forget_host+0xb7/0x120
[ 243.693665] scsi_forget_host+0xb7/0x120
[ 243.694301] scsi_remove_host+0xf1/0x300
[ 243.694301] scsi_remove_host+0xf1/0x300
[ 243.694973] sdebug_driver_remove+0x34/0x120
[ 243.694973] sdebug_driver_remove+0x34/0x120
[ 243.695621] ? devices_kset_move_last+0x9d/0x100
[ 243.695621] ? devices_kset_move_last+0x9d/0x100
[ 243.696430] driver_probe_device+0x172/0x6b0
[ 243.696430] driver_probe_device+0x172/0x6b0
[ 243.697413] __device_attach_driver+0x15d/0x1d0
[ 243.697413] __device_attach_driver+0x15d/0x1d0
[ 243.698460] ? __driver_attach+0x1b0/0x1b0
[ 243.698460] ? __driver_attach+0x1b0/0x1b0
[ 243.699111] bus_for_each_drv+0xb1/0x110
[ 243.699111] bus_for_each_drv+0xb1/0x110
[ 243.699856] __device_attach+0x14d/0x210
[ 243.699856] __device_attach+0x14d/0x210
[ 243.700643] ? __driver_attach+0x1b0/0x1b0
[ 243.700643] ? __driver_attach+0x1b0/0x1b0
[ 243.701362] device_initial_probe+0x1b/0x30
[ 243.701362] device_initial_probe+0x1b/0x30
[ 243.701886] bus_probe_device+0xe5/0x100
[ 243.701886] bus_probe_device+0xe5/0x100
[ 243.702447] device_add+0x822/0xc20
[ 243.702447] device_add+0x822/0xc20
[ 243.702879] ? pm_runtime_init+0x125/0x140
[ 243.702879] ? pm_runtime_init+0x125/0x140
[ 243.703366] device_register+0x2e/0x40
[ 243.703366] device_register+0x2e/0x40
[ 243.703823] sdebug_add_adapter+0x144/0x2e0
[ 243.703823] sdebug_add_adapter+0x144/0x2e0
[ 243.704381] ? kobject_uevent+0x18/0x30
[ 243.704381] ? kobject_uevent+0x18/0x30
[ 243.704881] scsi_debug_init+0x955/0xb9d
[ 243.704881] scsi_debug_init+0x955/0xb9d
[ 243.705372] ? init_sg+0x1a9/0x212
[ 243.705372] ? init_sg+0x1a9/0x212
[ 243.705763] ? init_sg+0x212/0x212
[ 243.705763] ? init_sg+0x212/0x212
[ 243.706166] do_one_initcall+0x5d/0x2b0
[ 243.706166] do_one_initcall+0x5d/0x2b0
[ 243.706634] ? parse_args+0x44e/0x710
[ 243.706634] ? parse_args+0x44e/0x710
[ 243.707059] ? kernel_init_freeable+0x20b/0x3ab
[ 243.707059] ? kernel_init_freeable+0x20b/0x3ab
[ 243.707569] kernel_init_freeable+0x23c/0x3ab
[ 243.707569] kernel_init_freeable+0x23c/0x3ab
[ 243.708076] ? rest_init+0x270/0x270
[ 243.708076] ? rest_init+0x270/0x270
[ 243.708543] kernel_init+0x16/0x250
[ 243.708543] kernel_init+0x16/0x250
[ 243.708978] ret_from_fork+0x21/0x30
[ 243.708978] ret_from_fork+0x21/0x30
[ 243.709399]
[ 243.709399] Showing all locks held in the system:
[ 243.709399]
[ 243.709399] Showing all locks held in the system:
[ 243.710265] 3 locks held by swapper/1:
[ 243.710265] 3 locks held by swapper/1:
[ 243.710785] #0: (&dev->mutex){......}, at: [<987efeb8>] __device_attach+0x28/0x210
[ 243.710785] #0: (&dev->mutex){......}, at: [<987efeb8>] __device_attach+0x28/0x210
[ 243.711670] #1: (&shost->scan_mutex){+.+.+.}, at: [<9883cac1>] scsi_remove_host+0x21/0x300
[ 243.711670] #1: (&shost->scan_mutex){+.+.+.}, at: [<9883cac1>] scsi_remove_host+0x21/0x300
[ 243.712674] #2: (&dev->mutex){......}, at: [<987f0d94>] device_release_driver_internal+0x54/0x4d0
[ 243.712674] #2: (&dev->mutex){......}, at: [<987f0d94>] device_release_driver_internal+0x54/0x4d0
[ 243.713807] 2 locks held by khungtaskd/13:
[ 243.713807] 2 locks held by khungtaskd/13:
[ 243.714297] #0: (rcu_read_lock){......}, at: [<97f5f626>] watchdog+0x126/0x8c0
[ 243.714297] #0: (rcu_read_lock){......}, at: [<97f5f626>] watchdog+0x126/0x8c0
[ 243.715168] #1: (tasklist_lock){.+.+..}, at: [<97ee6d62>] debug_show_all_locks+0x52/0x320
[ 243.715168] #1: (tasklist_lock){.+.+..}, at: [<97ee6d62>] debug_show_all_locks+0x52/0x320
[ 243.716128]
[ 243.716128]
[ 243.716372] =============================================
[ 243.716372]
[ 243.716372] =============================================
[ 243.716372]
[ 243.717191] Kernel panic - not syncing: hung_task: blocked tasks
[ 243.717191] Kernel panic - not syncing: hung_task: blocked tasks
[ 243.717889] CPU: 0 PID: 13 Comm: khungtaskd Not tainted 4.10.0-11089-g165a5e2 #2
[ 243.717889] CPU: 0 PID: 13 Comm: khungtaskd Not tainted 4.10.0-11089-g165a5e2 #2
[ 243.718742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014
[ 243.718742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014
[ 243.719871] Call Trace:
[ 243.719871] Call Trace:
[ 243.720185] dump_stack+0x32/0x43
[ 243.720185] dump_stack+0x32/0x43
[ 243.720185] panic+0x135/0x408
[ 243.720185] panic+0x135/0x408
[ 243.720185] watchdog+0x6d3/0x8c0
[ 243.720185] watchdog+0x6d3/0x8c0
[ 243.720185] kthread+0x151/0x1d0
[ 243.720185] kthread+0x151/0x1d0
[ 243.720185] ? hung_task_panic+0x20/0x20
[ 243.720185] ? hung_task_panic+0x20/0x20
[ 243.720185] ? kthread_worker_fn+0x220/0x220
[ 243.720185] ? kthread_worker_fn+0x220/0x220
[ 243.720185] ret_from_fork+0x21/0x30
[ 243.720185] ret_from_fork+0x21/0x30
[ 243.720185] Kernel Offset: 0x16e00000 from 0x81000000 (relocation range: 0x80000000-0xa07dffff)
[ 243.720185] Kernel Offset: 0x16e00000 from 0x81000000 (relocation range: 0x80000000-0xa07dffff)
# HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD
git bisect start 0724f5d8ca3a89134a8c248aa4c311ce43d53512 c470abd4fde40ea6a0846a2beab642a578c0b8cd --
git bisect bad 304f9c7ff9aabc9a3cec12bf88f29d24139ac370 # 14:12 B 0 4 15 0 Merge 'linux-review/Roger-Quadros/Revert-usb-gadget-uvc-Add-missing-call-for-additional-setup-data/20170303-232925' into devel-spot-201703040603
git bisect bad a28ca518ffc36daf89550978066ad20094c19e51 # 14:33 B 0 8 22 3 Merge 'linux-review/Arushi-Singhal/staging-speakup-Alignment-match-open-parenthesis/20170304-030809' into devel-spot-201703040603
git bisect bad 93354ede964ded2791d3b573635d538b14590545 # 14:55 B 0 4 15 0 Merge 'linux-review/Georgios-Emmanouil/Staging-wilc1000-wilc_sdio-Modified-comment-style-to-preferred-kernel-comment-style/20170304-050826' into devel-spot-201703040603
git bisect bad 402decdec94a2e4e2635ef662e57f6217a305d81 # 15:36 B 0 11 22 0 Merge 'linux-review/Heiner-Kallweit/staging-fbtft-improve-a-syslog-message-in-fbtft_register_framebuffer/20170304-054107' into devel-spot-201703040603
git bisect bad 1735b51c4dda1ca7f2c239c4b790c3c638281f54 # 15:53 B 0 6 17 0 Merge 'block/for-linus' into devel-spot-201703040603
git bisect good cf5f16969d36dfa64e92cf19594b94de0c381519 # 16:05 G 11 0 0 0 0day base guard for 'devel-spot-201703040603'
git bisect good caa59428971d5ad81d19512365c9ba580d83268c # 16:23 G 11 0 0 0 Merge tag 'staging-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect good d5500a074741b78b7f778b4ab3415d5ecdcda0a7 # 16:48 G 11 0 0 0 Merge tag 'usercopy-v4.11-rc1.fix' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux
git bisect good 7067739df23ffd641ca99c967830e0ed2ba39eab # 17:13 G 11 0 0 0 Merge branch 'i2c/for-4.11' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
git bisect good b2deee2dc06db7cdf99b84346e69bdb9db9baa85 # 17:47 G 11 0 0 0 Merge tag 'ceph-for-4.11-rc1' of git://github.com/ceph/ceph-client
git bisect good 010426079ec1228a7f980d2eef766a84c0f9241a # 18:03 G 11 0 0 0 sched/headers: Prepare for new header dependencies before moving more code to <linux/sched/mm.h>
git bisect good 3f80dd67c367878aaad16e458eebc3c8980bb841 # 18:28 G 11 0 0 0 Merge tag 'acpi-extra-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect good e26512fea5bcd6602dbf02a551ed073cd4529449 # 18:42 G 10 0 0 0 sched/headers: Remove <linux/cred.h> inclusion from <linux/sched.h>
git bisect good 78769912f680fc0a79a67e798a0ae76f07e63a7b # 19:08 G 11 0 0 0 Merge tag 'linux-kselftest-4.11-rc1-urgent_fix' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest
git bisect good 1827adb11ad26b2290dc9fe2aaf54976b2439865 # 19:21 G 11 0 0 0 Merge branch 'WIP.sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 165a5e22fafb127ecb5914e12e8c32a1f0d3f820 # 19:39 B 0 11 22 0 block: Move bdi_unregister() to del_gendisk()
git bisect good 6a8a21546507a3ec88e81c2ec927a3fb63efa8ff # 19:55 G 11 0 0 0 nbd: stop leaking sockets
git bisect good e02898b423802b1f3a3aaa7f16e896da069ba8f7 # 20:36 G 11 0 0 0 loop: fix LO_FLAGS_PARTSCAN hang
git bisect good 7b36a7189fc320f0b783dd51bd1f541db56cfbdd # 20:53 G 11 0 0 0 block: don't call ioc_exit_icq() with the queue lock held for blk-mq
git bisect good 113285b473824922498d07d7f82459507b9792eb # 21:37 G 11 0 0 0 blk-mq: ensure that bd->last is always set correctly
# first bad commit: [165a5e22fafb127ecb5914e12e8c32a1f0d3f820] block: Move bdi_unregister() to del_gendisk()
git bisect good 113285b473824922498d07d7f82459507b9792eb # 21:41 G 31 0 0 0 blk-mq: ensure that bd->last is always set correctly
# extra tests with CONFIG_DEBUG_INFO_REDUCED
git bisect bad 165a5e22fafb127ecb5914e12e8c32a1f0d3f820 # 22:16 B 0 10 22 1 block: Move bdi_unregister() to del_gendisk()
# extra tests on HEAD of linux-devel/devel-spot-201703040603
git bisect bad 0724f5d8ca3a89134a8c248aa4c311ce43d53512 # 22:16 B 0 39 55 2 0day head guard for 'devel-spot-201703040603'
# extra tests on tree/branch linus/master
git bisect bad 0710f3ff91ecc4a715db6e4d0690472b13c4dac6 # 22:42 B 0 7 18 0 Merge branch 'work.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
# extra tests with first bad commit reverted
git bisect good 8b222187187cd9372d0d1ee036f1966e142bbd42 # 23:09 G 10 0 0 0 Revert "block: Move bdi_unregister() to del_gendisk()"
# extra tests on tree/branch linux-next/master
git bisect good c0b7b2b33bd17f7155956d0338ce92615da686c9 # 23:28 G 11 0 0 0 Add linux-next specific files for 20170303
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/lkp Intel Corporation
Download attachment "dmesg-quantal-lkp-hsw01-19:20170304193900:i386-randconfig-h0-03040625:4.10.0-11089-g165a5e2:2.gz" of type "application/gzip" (18882 bytes)
View attachment "reproduce-quantal-lkp-hsw01-19:20170304193900:i386-randconfig-h0-03040625:4.10.0-11089-g165a5e2:2" of type "text/plain" (886 bytes)
View attachment "config-4.10.0-11089-g165a5e2" of type "text/plain" (113430 bytes)
Powered by blists - more mailing lists