| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CACT4Y+Yk0hEDj2EcXTJ5BcTHE0m1Sn0e9RfY=6KC4nWhWOHPmw@mail.gmail.com>
Date: Tue, 3 Apr 2018 17:23:36 +0200
From: Dmitry Vyukov <dvyukov@...gle.com>
To: Paul McKenney <paulmck@...ux.vnet.ibm.com>
Cc: Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp>,
LKML <linux-kernel@...r.kernel.org>,
Andrew Morton <akpm@...ux-foundation.org>,
Ingo Molnar <mingo@...nel.org>,
Linus Torvalds <torvalds@...ux-foundation.org>,
Mandeep Singh Baines <msb@...omium.org>,
Peter Zijlstra <peterz@...radead.org>,
Thomas Gleixner <tglx@...utronix.de>,
Vegard Nossum <vegard.nossum@...cle.com>
Subject: Re: [PATCH] locking/hung_task: Show all hung tasks before panic
On Mon, Apr 2, 2018 at 5:35 PM, Paul E. McKenney
<paulmck@...ux.vnet.ibm.com> wrote:
> On Mon, Apr 02, 2018 at 11:12:04PM +0900, Tetsuo Handa wrote:
>> When we get a hung task it can often be valuable to see _all_ the hung
>> tasks on the system before calling panic().
>>
>> Quoting from https://syzkaller.appspot.com/text?tag=CrashReport&id=5412451675799552
>> ----------------------------------------
>> INFO: task syz-executor3:13421 blocked for more than 120 seconds.
>> Not tainted 4.16.0-rc7+ #9
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> syz-executor3 D24672 13421 4481 0x00000004
>> Call Trace:
>> context_switch kernel/sched/core.c:2862 [inline]
>> __schedule+0x8fb/0x1ec0 kernel/sched/core.c:3440
>> schedule+0xf5/0x430 kernel/sched/core.c:3499
>> __rwsem_down_read_failed_common kernel/locking/rwsem-xadd.c:269 [inline]
>> rwsem_down_read_failed+0x401/0x6e0 kernel/locking/rwsem-xadd.c:286
>> call_rwsem_down_read_failed+0x18/0x30 arch/x86/lib/rwsem.S:94
>> __down_read arch/x86/include/asm/rwsem.h:83 [inline]
>> down_read+0xa4/0x150 kernel/locking/rwsem.c:26
>> __get_super.part.9+0x1d3/0x280 fs/super.c:663
>> __get_super include/linux/spinlock.h:310 [inline]
>> get_super+0x2d/0x40 fs/super.c:692
>> fsync_bdev+0x19/0x80 fs/block_dev.c:468
>> invalidate_partition+0x35/0x60 block/genhd.c:1566
>> drop_partitions.isra.12+0xcd/0x1d0 block/partition-generic.c:440
>> rescan_partitions+0x72/0x900 block/partition-generic.c:513
>> __blkdev_reread_part+0x15f/0x1e0 block/ioctl.c:173
>> blkdev_reread_part+0x26/0x40 block/ioctl.c:193
>> loop_reread_partitions+0x12f/0x1a0 drivers/block/loop.c:619
>> loop_set_status+0x9bb/0xf60 drivers/block/loop.c:1161
>> loop_set_status64+0x9d/0x110 drivers/block/loop.c:1271
>> lo_ioctl+0xd86/0x1b70 drivers/block/loop.c:1381
>> (...snipped...)
>> Showing all locks held in the system:
>> (...snipped...)
>> 3 locks held by syz-executor3/13421:
>> #0: (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000834f78af>] lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
>> #1: (&bdev->bd_mutex){+.+.}, at: [<0000000003605603>] blkdev_reread_part+0x1e/0x40 block/ioctl.c:192
>> #2: (&type->s_umount_key#77){.+.+}, at: [<0000000077701649>] __get_super.part.9+0x1d3/0x280 fs/super.c:663 /* down_read(&sb->s_umount); */
>> (...snipped...)
>> 2 locks held by syz-executor0/13428:
>> #0: (&type->s_umount_key#76/1){+.+.}, at: [<00000000d25ba33a>] alloc_super fs/super.c:211 [inline]
>> #0: (&type->s_umount_key#76/1){+.+.}, at: [<00000000d25ba33a>] sget_userns+0x3a1/0xe40 fs/super.c:502 /* down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING); */
>> #1: (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000834f78af>] lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
>> ----------------------------------------
>>
>> In addition to showing hashed address of lock instances, it would be
>> nice if trace of 13428 is printed as well as 13421.
>>
>> Showing hung tasks up to /proc/sys/kernel/hung_task_warnings could delay
>> calling panic() but normally there should not be so many hung tasks.
>>
>> Signed-off-by: Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>
>> Cc: Vegard Nossum <vegard.nossum@...cle.com>
>> Cc: Andrew Morton <akpm@...ux-foundation.org>
>> Cc: Linus Torvalds <torvalds@...ux-foundation.org>
>> Cc: Mandeep Singh Baines <msb@...omium.org>
>> Cc: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
>
> I just know that I am going to regret this the first time this happens
> on a low-speed console port, but...
>
> Acked-by: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
Thanks! I think getting these last bits of debugging tools to be more
useful is very important in the context of syzbot. So:
Acked-by: Dmitry Vyukov <dvyukov@...gle.com>
>> Cc: Peter Zijlstra <peterz@...radead.org>
>> Cc: Thomas Gleixner <tglx@...utronix.de>
>> Cc: Ingo Molnar <mingo@...nel.org>
>> ---
>> kernel/hung_task.c | 11 +++++++----
>> 1 file changed, 7 insertions(+), 4 deletions(-)
>>
>> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
>> index 751593e..32b4794 100644
>> --- a/kernel/hung_task.c
>> +++ b/kernel/hung_task.c
>> @@ -44,6 +44,7 @@
>>
>> static int __read_mostly did_panic;
>> static bool hung_task_show_lock;
>> +static bool hung_task_call_panic;
>>
>> static struct task_struct *watchdog_task;
>>
>> @@ -127,10 +128,8 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
>> touch_nmi_watchdog();
>>
>> if (sysctl_hung_task_panic) {
>> - if (hung_task_show_lock)
>> - debug_show_all_locks();
>> - trigger_all_cpu_backtrace();
>> - panic("hung_task: blocked tasks");
>> + hung_task_show_lock = true;
>> + hung_task_call_panic = true;
>> }
>> }
>>
>> @@ -193,6 +192,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
>> rcu_read_unlock();
>> if (hung_task_show_lock)
>> debug_show_all_locks();
>> + if (hung_task_call_panic) {
>> + trigger_all_cpu_backtrace();
>> + panic("hung_task: blocked tasks");
>> + }
>> }
>>
>> static long hung_timeout_jiffies(unsigned long last_checked,
>> --
>> 1.8.3.1
>>
>
Powered by blists - more mailing lists