| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Fri, 30 Jan 2015 17:10:45 +0800
From: Ying Xue <ying.xue@...driver.com>
To: Thomas Graf <tgraf@...g.ch>, <davem@...emloft.net>
CC: <netdev@...r.kernel.org>
Subject: Re: [PATCH 0/6 net-next] rhashtable fixes
Hi Thomas,
I make sure that my local net-next tree is synchronized to the latest
version in which the commit fe6a043c535acfec8f8e554536c87923dcb45097
("rhashtable: rhashtable_remove() must unlink in both tbl and
future_tbl") is already contained, and then I manually applied the whole
series patches. But when I repeatedly run the test case I originally
posted, soft lockup happens. Please see its relevant log:
root@...alhost:/mnt# d[ 115.776178] ------------[ cut here ]------------
[ 115.776548] WARNING: CPU: 4 PID: 0 at net/sched/sch_generic.c:303
dev_watchdog+0x247/0x250()
[ 115.777106] NETDEV WATCHDOG: eth0 (e1000): transmit queue 0 timed out
[ 115.777533] Modules linked in: tipc
[ 115.777790] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 3.19.0-rc6+ #182
[ 115.778221] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 115.778602] 000000000000012f ffff880017d03d08 ffffffff8175cd25
0000000000001052
[ 115.779133] ffff880017d03d58 ffff880017d03d48 ffffffff81059717
ffffffff00000000
[ 115.779661] ffff880015c26000 ffff880015c263e0 ffff880015f07000
0000000000000001
[ 115.780165] Call Trace:
[ 115.780165] <IRQ> [<ffffffff8175cd25>] dump_stack+0x4c/0x65
[ 115.780165] [<ffffffff81059717>] warn_slowpath_common+0x97/0xe0
[ 115.780165] [<ffffffff81059816>] warn_slowpath_fmt+0x46/0x50
[ 115.780165] [<ffffffff81654537>] dev_watchdog+0x247/0x250
[ 115.780165] [<ffffffff816542f0>] ? pfifo_fast_dequeue+0xe0/0xe0
[ 115.780165] [<ffffffff816542f0>] ? pfifo_fast_dequeue+0xe0/0xe0
[ 115.780165] [<ffffffff810c5ebc>] call_timer_fn+0x8c/0x1e0
[ 115.780165] [<ffffffff810c5e35>] ? call_timer_fn+0x5/0x1e0
[ 115.780165] [<ffffffff817663b0>] ? _raw_spin_unlock_irq+0x30/0x40
[ 115.780165] [<ffffffff816542f0>] ? pfifo_fast_dequeue+0xe0/0xe0
[ 115.780165] [<ffffffff810c7994>] run_timer_softirq+0x2d4/0x320
[ 115.780165] [<ffffffff810d6214>] ? clockevents_program_event+0x74/0x100
[ 115.780165] [<ffffffff8105d653>] __do_softirq+0x123/0x360
[ 115.780165] [<ffffffff8105db2e>] irq_exit+0x8e/0xb0
[ 115.780165] [<ffffffff8176952a>] smp_apic_timer_interrupt+0x4a/0x60
[ 115.780165] [<ffffffff817678af>] apic_timer_interrupt+0x6f/0x80
[ 115.780165] <EOI> [<ffffffff8100d174>] ? default_idle+0x24/0x100
[ 115.780165] [<ffffffff8100d172>] ? default_idle+0x22/0x100
[ 115.780165] [<ffffffff8100daaf>] arch_cpu_idle+0xf/0x20
[ 115.780165] [<ffffffff8109a509>] cpu_startup_entry+0x2c9/0x3c0
[ 115.780165] [<ffffffff810d5ea2>] ?
clockevents_register_device+0xe2/0x140
[ 115.780165] [<ffffffff810333c1>] start_secondary+0x141/0x150
[ 115.780165] ---[ end trace 62da3388fe54379b ]---
[ 115.780165] e1000 0000:00:03.0 eth0: Reset adapter
[ 116.724005] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 116.724005] Modules linked in: tipc
[ 116.724005] irq event stamp: 331179
[ 116.724005] hardirqs last enabled at (331178): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 116.724005] hardirqs last disabled at (331179): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 116.724005] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 116.724005] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 116.724005] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
3.19.0-rc6+ #182
[ 116.724005] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 116.724005] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 116.724005] RIP: 0010:[<ffffffff8165d820>] [<ffffffff8165d820>]
netlink_compare+0x10/0x30
[ 116.724005] RSP: 0018:ffff8800105d3cf0 EFLAGS: 00000293
[ 116.724005] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 116.724005] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 116.724005] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 116.724005] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 116.724005] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 116.724005] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 116.724005] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 116.724005] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 116.724005] Stack:
[ 116.724005] ffffffff8139de4e ffffffff8139ddc0 000000028105d9c8
07ca27d400000000
[ 116.724005] ffff8800152d13c0 ffff8800105d3d48 ffff880016500000
ffff8800152d13c0
[ 116.724005] 0000000000000004 ffff8800152d13c0 ffff880010c07608
ffff8800105d3da8
[ 116.724005] Call Trace:
[ 116.724005] [<ffffffff8139de4e>] ? rhashtable_lookup_compare+0x8e/0x120
[ 116.724005] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 116.724005] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 116.724005] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 116.724005] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 116.724005] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 116.724005] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 116.724005] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 116.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 116.724005] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 116.724005] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 116.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 116.724005] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 116.724005] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 116.724005] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 116.724005] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 116.724005] Code: f0 ff 83 98 01 00 00 48 83 c4 08 5b 5d c3 66 66 66
66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 8b 56 08 39 97 68 04
00 00 <55> 48 89 e5 74 0a 5d c3 0f 1f 84 00 00 00 00 00 48 8b 47 30 48
[ 144.724006] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 144.724006] Modules linked in: tipc
[ 144.724006] irq event stamp: 345169
[ 144.724006] hardirqs last enabled at (345168): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 144.724006] hardirqs last disabled at (345169): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 144.724006] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 144.724006] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 144.724006] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 144.724006] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 144.724006] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 144.724006] RIP: 0010:[<ffffffff8165d826>] [<ffffffff8165d826>]
netlink_compare+0x16/0x30
[ 144.724006] RSP: 0018:ffff8800105d3ce8 EFLAGS: 00000293
[ 144.724006] RAX: 0000000000000000 RBX: ffff8800105d3c68 RCX:
00000000dbaee169
[ 144.724006] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 144.724006] RBP: ffff8800105d3ce8 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 144.724006] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d0000
[ 144.724006] R13: ffff880013970000 R14: 0000000000000000 R15:
0000000000000001
[ 144.724006] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 144.724006] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 144.724006] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 144.724006] Stack:
[ 144.724006] ffff8800105d3d48 ffffffff8139de4e ffffffff8139ddc0
000000028105d9c8
[ 144.724006] 07ca27d400000000 ffff8800152d13c0 ffff8800105d3d48
ffff880016500000
[ 144.724006] ffff8800152d13c0 0000000000000004 ffff8800152d13c0
ffff880010c07608
[ 144.724006] Call Trace:
[ 144.724006] [<ffffffff8139de4e>] rhashtable_lookup_compare+0x8e/0x120
[ 144.724006] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 144.724006] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 144.724006] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 144.724006] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 144.724006] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 144.724006] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 144.724006] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 144.724006] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 144.724006] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 144.724006] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 144.724006] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 144.724006] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 144.724006] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 144.724006] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 144.724006] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 144.724006] Code: 00 48 83 c4 08 5b 5d c3 66 66 66 66 2e 0f 1f 84 00
00 00 00 00 0f 1f 44 00 00 31 c0 8b 56 08 39 97 68 04 00 00 55 48 89 e5
74 0a <5d> c3 0f 1f 84 00 00 00 00 00 48 8b 47 30 48 39 06 5d 0f 94 c0
[ 150.732004] INFO: rcu_sched self-detected stall on CPU { 7} (t=15000
jiffies g=674 c=673 q=18)
[ 150.732004] Task dump for CPU 7:
[ 150.732004] bind_netlink R running task 0 631 561
0x20020008
[ 150.732004] 0000000000000231 ffff880017dc3d68 ffffffff81086c26
ffffffff81086b88
[ 150.732004] 00000000a5f9a5f8 0000000000000007 ffffffff81c53940
ffff880017dc3d88
[ 150.732004] ffffffff8108a57f ffffffff81c53940 ffffffff81c53940
ffff880017dc3db8
[ 150.732004] Call Trace:
[ 150.732004] <IRQ> [<ffffffff81086c26>] sched_show_task+0x106/0x170
[ 150.732004] [<ffffffff81086b88>] ? sched_show_task+0x68/0x170
[ 150.732004] [<ffffffff8108a57f>] dump_cpu_task+0x3f/0x50
[ 150.732004] [<ffffffff810bfa7b>] rcu_dump_cpu_stacks+0x8b/0xc0
[ 150.732004] [<ffffffff810c33d0>] rcu_check_callbacks+0x480/0x6d0
[ 150.732004] [<ffffffff810a133d>] ? trace_hardirqs_off+0xd/0x10
[ 150.732004] [<ffffffff810c8408>] update_process_times+0x38/0x70
[ 150.732004] [<ffffffff810d8623>] tick_sched_handle.isra.15+0x33/0x70
[ 150.732004] [<ffffffff810d88cb>] tick_sched_timer+0x4b/0x80
[ 150.732004] [<ffffffff810c8d8b>] __run_hrtimer+0x9b/0x290
[ 150.732004] [<ffffffff810d8880>] ? tick_sched_do_timer+0x40/0x40
[ 150.732004] [<ffffffff810c95e4>] ? hrtimer_interrupt+0x74/0x260
[ 150.732004] [<ffffffff810c9677>] hrtimer_interrupt+0x107/0x260
[ 150.732004] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 150.732004] [<ffffffff81034cb9>] local_apic_timer_interrupt+0x39/0x60
[ 150.732004] [<ffffffff81769525>] smp_apic_timer_interrupt+0x45/0x60
[ 150.732004] [<ffffffff817678af>] apic_timer_interrupt+0x6f/0x80
[ 150.732004] <EOI> [<ffffffff817675e0>] ? retint_restore_args+0xe/0xe
[ 150.732004] [<ffffffff8165d817>] ? netlink_compare+0x7/0x30
[ 150.732004] [<ffffffff8139de4e>] ? rhashtable_lookup_compare+0x8e/0x120
[ 150.732004] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 150.732004] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 150.732004] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 150.732004] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 150.732004] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 150.732004] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 150.732004] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 150.732004] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 150.732004] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 150.732004] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 150.732004] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 150.732004] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 150.732004] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 150.732004] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 150.732004] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 176.724005] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 176.724005] Modules linked in: tipc
[ 176.724005] irq event stamp: 361143
[ 176.724005] hardirqs last enabled at (361142): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 176.724005] hardirqs last disabled at (361143): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 176.724005] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 176.724005] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 176.724005] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 176.724005] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 176.724005] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 176.724005] RIP: 0010:[<ffffffff8139de55>] [<ffffffff8139de55>]
rhashtable_lookup_compare+0x95/0x120
[ 176.724005] RSP: 0018:ffff8800105d3cf8 EFLAGS: 00000246
[ 176.724005] RAX: 0000000000000000 RBX: 0000000000000003 RCX:
00000000dbaee169
[ 176.724005] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 176.724005] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 176.724005] R10: 0000000000000003 R11: 0000000000000001 R12:
ffffffff817675e0
[ 176.724005] R13: ffffffff810c2d18 R14: ffff8800105d3c58 R15:
0000000000000046
[ 176.724005] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 176.724005] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 176.724005] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 176.724005] Stack:
[ 176.724005] ffffffff8139ddc0 000000028105d9c8 07ca27d400000000
ffff8800152d13c0
[ 176.724005] ffff8800105d3d48 ffff880016500000 ffff8800152d13c0
0000000000000004
[ 176.724005] ffff8800152d13c0 ffff880010c07608 ffff8800105d3da8
ffffffff8139dfaf
[ 176.724005] Call Trace:
[ 176.724005] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 176.724005] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 176.724005] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 176.724005] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 176.724005] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 176.724005] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 176.724005] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 176.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 176.724005] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 176.724005] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 176.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 176.724005] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 176.724005] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 176.724005] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 176.724005] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 176.724005] Code: 8b 02 83 e8 01 23 45 c4 48 83 c0 02 4c 8b 74 c2 08
41 f6 c6 01 75 1a 4c 89 f7 48 2b 7b 30 4c 89 ee 41 ff d4 84 c0 75 46 4d
8b 36 <41> f6 c6 01 74 e6 4c 39 7d c8 75 6c 48 c7 c2 61 de 39 81 be 01
[ 204.724005] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 204.724005] Modules linked in: tipc
[ 204.724005] irq event stamp: 375131
[ 204.724005] hardirqs last enabled at (375130): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 204.724005] hardirqs last disabled at (375131): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 204.724005] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 204.724005] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 204.724005] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 204.724005] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 204.724005] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 204.724005] RIP: 0010:[<ffffffff8139de4b>] [<ffffffff8139de4b>]
rhashtable_lookup_compare+0x8b/0x120
[ 204.724005] RSP: 0018:ffff8800105d3cf8 EFLAGS: 00000286
[ 204.724005] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 204.724005] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 204.724005] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 204.724005] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 204.724005] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 204.724005] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 204.724005] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 204.724005] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 204.724005] Stack:
[ 204.724005] ffffffff8139ddc0 000000028105d9c8 07ca27d400000000
ffff8800152d13c0
[ 204.724005] ffff8800105d3d48 ffff880016500000 ffff8800152d13c0
0000000000000004
[ 204.724005] ffff8800152d13c0 ffff880010c07608 ffff8800105d3da8
ffffffff8139dfaf
[ 204.724005] Call Trace:
[ 204.724005] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 204.724005] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 204.724005] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 204.724005] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 204.724005] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 204.724005] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 204.724005] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 204.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 204.724005] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 204.724005] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 204.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 204.724005] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 204.724005] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 204.724005] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 204.724005] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 204.724005] Code: e8 05 89 45 c4 48 8b 55 c8 48 8b 02 83 e8 01 23 45
c4 48 83 c0 02 4c 8b 74 c2 08 41 f6 c6 01 75 1a 4c 89 f7 48 2b 7b 30 4c
89 ee <41> ff d4 84 c0 75 46 4d 8b 36 41 f6 c6 01 74 e6 4c 39 7d c8 75
[ 232.724005] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 232.724005] Modules linked in: tipc
[ 232.724005] irq event stamp: 389119
[ 232.724005] hardirqs last enabled at (389118): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 232.724005] hardirqs last disabled at (389119): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 232.724005] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 232.724005] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 232.724005] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 232.724005] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 232.724005] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 232.724005] RIP: 0010:[<ffffffff8165d820>] [<ffffffff8165d820>]
netlink_compare+0x10/0x30
[ 232.724005] RSP: 0018:ffff8800105d3cf0 EFLAGS: 00000293
[ 232.724005] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 232.724005] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 232.724005] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 232.724005] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 232.724005] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 232.724005] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 232.724005] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 232.724005] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 232.724005] Stack:
[ 232.724005] ffffffff8139de4e ffffffff8139ddc0 000000028105d9c8
07ca27d400000000
[ 232.724005] ffff8800152d13c0 ffff8800105d3d48 ffff880016500000
ffff8800152d13c0
[ 232.724005] 0000000000000004 ffff8800152d13c0 ffff880010c07608
ffff8800105d3da8
[ 232.724005] Call Trace:
[ 232.724005] [<ffffffff8139de4e>] ? rhashtable_lookup_compare+0x8e/0x120
[ 232.724005] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 232.724005] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 232.724005] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 232.724005] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 232.724005] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 232.724005] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 232.724005] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 232.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 232.724005] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 232.724005] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 232.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 232.724005] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 232.724005] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 232.724005] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 232.724005] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 232.724005] Code: f0 ff 83 98 01 00 00 48 83 c4 08 5b 5d c3 66 66 66
66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 8b 56 08 39 97 68 04
00 00 <55> 48 89 e5 74 0a 5d c3 0f 1f 84 00 00 00 00 00 48 8b 47 30 48
[ 240.772081] INFO: task kworker/7:1:70 blocked for more than 120 seconds.
[ 240.773035] Tainted: G W L 3.19.0-rc6+ #182
[ 240.773811] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 240.774897] kworker/7:1 D ffff8800167c3af8 0 70 2
0x00000000
[ 240.775940] Workqueue: events rht_deferred_worker
[ 240.776676] ffff8800167c3af8 0000000000001ffb ffff8800167b8000
00000000000139c0
[ 240.777763] ffff8800167c3fd8 0000000000000000 00000000000139c0
ffff880013970000
[ 240.778845] ffff8800167b8000 0000000000000000 ffff8800167c3c58
7fffffffffffffff
[ 240.779928] Call Trace:
[ 240.780318] [<ffffffff81760879>] schedule+0x29/0x70
[ 240.781014] [<ffffffff81764de5>] schedule_timeout+0x1d5/0x230
[ 240.781826] [<ffffffff810a68fa>] ? mark_held_locks+0x6a/0x90
[ 240.782622] [<ffffffff817663b0>] ? _raw_spin_unlock_irq+0x30/0x40
[ 240.783459] [<ffffffff810a6a25>] ? trace_hardirqs_on_caller+0x105/0x1d0
[ 240.784483] [<ffffffff81761d1b>] wait_for_completion+0xbb/0x120
[ 240.785322] [<ffffffff81087910>] ? try_to_wake_up+0x3c0/0x3c0
[ 240.786179] [<ffffffff810c12f0>] ? __call_rcu.constprop.61+0x260/0x260
[ 240.787122] [<ffffffff810bea9d>] wait_rcu_gp+0x4d/0x60
[ 240.787906] [<ffffffff810beab0>] ? wait_rcu_gp+0x60/0x60
[ 240.788721] [<ffffffff810c1cfd>] synchronize_sched+0x5d/0x70
[ 240.789538] [<ffffffff8139e653>] rhashtable_shrink+0x113/0x150
[ 240.790385] [<ffffffff8139ea30>] rht_deferred_worker+0x80/0xa0
[ 240.791235] [<ffffffff81073449>] process_one_work+0x1b9/0x530
[ 240.792100] [<ffffffff810733d2>] ? process_one_work+0x142/0x530
[ 240.792960] [<ffffffff81073c5f>] worker_thread+0x11f/0x480
[ 240.793740] [<ffffffff81073b40>] ? rescuer_thread+0x340/0x340
[ 240.794552] [<ffffffff81079b4f>] kthread+0xef/0x110
[ 240.795268] [<ffffffff81079a60>] ? flush_kthread_worker+0xf0/0xf0
[ 240.796185] [<ffffffff817668ec>] ret_from_fork+0x7c/0xb0
[ 240.796938] [<ffffffff81079a60>] ? flush_kthread_worker+0xf0/0xf0
[ 240.797785] 3 locks held by kworker/7:1/70:
[ 240.798411] #0: ("events"){.+.+.+}, at: [<ffffffff810733d2>]
process_one_work+0x142/0x530
[ 240.799699] #1: ((&ht->run_work)){+.+.+.}, at: [<ffffffff810733d2>]
process_one_work+0x142/0x530
[ 240.801082] #2: (&ht->mutex){+.+.+.}, at: [<ffffffff8139e9dd>]
rht_deferred_worker+0x2d/0xa0
[ 260.724007] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 260.724007] Modules linked in: tipc
[ 260.724007] irq event stamp: 403107
[ 260.724007] hardirqs last enabled at (403106): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 260.724007] hardirqs last disabled at (403107): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 260.724007] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 260.724007] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 260.724007] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 260.724007] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 260.724007] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 260.724007] RIP: 0010:[<ffffffff8165d826>] [<ffffffff8165d826>]
netlink_compare+0x16/0x30
[ 260.724007] RSP: 0018:ffff8800105d3ce8 EFLAGS: 00000293
[ 260.724007] RAX: 0000000000000000 RBX: ffff8800105d3c68 RCX:
00000000dbaee169
[ 260.724007] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 260.724007] RBP: ffff8800105d3ce8 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 260.724007] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d0000
[ 260.724007] R13: ffff880013970000 R14: 0000000000000000 R15:
0000000000000001
[ 260.724007] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 260.724007] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 260.724007] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 260.724007] Stack:
[ 260.724007] ffff8800105d3d48 ffffffff8139de4e ffffffff8139ddc0
000000028105d9c8
[ 260.724007] 07ca27d400000000 ffff8800152d13c0 ffff8800105d3d48
ffff880016500000
[ 260.724007] ffff8800152d13c0 0000000000000004 ffff8800152d13c0
ffff880010c07608
[ 260.724007] Call Trace:
[ 260.724007] [<ffffffff8139de4e>] rhashtable_lookup_compare+0x8e/0x120
[ 260.724007] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 260.724007] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 260.724007] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 260.724007] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 260.724007] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 260.724007] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 260.724007] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 260.724007] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 260.724007] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 260.724007] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 260.724007] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 260.724007] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 260.724007] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 260.724007] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 260.724007] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 260.724007] Code: 00 48 83 c4 08 5b 5d c3 66 66 66 66 2e 0f 1f 84 00
00 00 00 00 0f 1f 44 00 00 31 c0 8b 56 08 39 97 68 04 00 00 55 48 89 e5
74 0a <5d> c3 0f 1f 84 00 00 00 00 00 48 8b 47 30 48 39 06 5d 0f 94 c0
[ 288.724004] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 288.724004] Modules linked in: tipc
[ 288.724004] irq event stamp: 417095
[ 288.724004] hardirqs last enabled at (417094): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 288.724004] hardirqs last disabled at (417095): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 288.724004] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 288.724004] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 288.724004] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 288.724004] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 288.724004] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 288.724004] RIP: 0010:[<ffffffff8139de4b>] [<ffffffff8139de4b>]
rhashtable_lookup_compare+0x8b/0x120
[ 288.724004] RSP: 0018:ffff8800105d3cf8 EFLAGS: 00000286
[ 288.724004] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 288.724004] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 288.724004] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 288.724004] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 288.724004] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 288.724004] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 288.724004] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 288.724004] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 288.724004] Stack:
[ 288.724004] ffffffff8139ddc0 000000028105d9c8 07ca27d400000000
ffff8800152d13c0
[ 288.724004] ffff8800105d3d48 ffff880016500000 ffff8800152d13c0
0000000000000004
[ 288.724004] ffff8800152d13c0 ffff880010c07608 ffff8800105d3da8
ffffffff8139dfaf
[ 288.724004] Call Trace:
[ 288.724004] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 288.724004] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 288.724004] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 288.724004] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 288.724004] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 288.724004] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 288.724004] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 288.724004] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 288.724004] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 288.724004] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 288.724004] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 288.724004] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 288.724004] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 288.724004] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 288.724004] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 288.724004] Code: e8 05 89 45 c4 48 8b 55 c8 48 8b 02 83 e8 01 23 45
c4 48 83 c0 02 4c 8b 74 c2 08 41 f6 c6 01 75 1a 4c 89 f7 48 2b 7b 30 4c
89 ee <41> ff d4 84 c0 75 46 4d 8b 36 41 f6 c6 01 74 e6 4c 39 7d c8 75
[ 316.724005] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 316.724005] Modules linked in: tipc
[ 316.724005] irq event stamp: 431083
[ 316.724005] hardirqs last enabled at (431082): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 316.724005] hardirqs last disabled at (431083): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 316.724005] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 316.724005] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 316.724005] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 316.724005] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 316.724005] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 316.724005] RIP: 0010:[<ffffffff8139de4b>] [<ffffffff8139de4b>]
rhashtable_lookup_compare+0x8b/0x120
[ 316.724005] RSP: 0018:ffff8800105d3cf8 EFLAGS: 00000286
[ 316.724005] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 316.724005] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 316.724005] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 316.724005] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 316.724005] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 316.724005] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 316.724005] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 316.724005] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 316.724005] Stack:
[ 316.724005] ffffffff8139ddc0 000000028105d9c8 07ca27d400000000
ffff8800152d13c0
[ 316.724005] ffff8800105d3d48 ffff880016500000 ffff8800152d13c0
0000000000000004
[ 316.724005] ffff8800152d13c0 ffff880010c07608 ffff8800105d3da8
ffffffff8139dfaf
[ 316.724005] Call Trace:
[ 316.724005] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 316.724005] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 316.724005] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 316.724005] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 316.724005] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 316.724005] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 316.724005] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 316.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 316.724005] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 316.724005] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 316.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 316.724005] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 316.724005] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 316.724005] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 316.724005] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 316.724005] Code: e8 05 89 45 c4 48 8b 55 c8 48 8b 02 83 e8 01 23 45
c4 48 83 c0 02 4c 8b 74 c2 08 41 f6 c6 01 75 1a 4c 89 f7 48 2b 7b 30 4c
89 ee <41> ff d4 84 c0 75 46 4d 8b 36 41 f6 c6 01 74 e6 4c 39 7d c8 75
[ 330.748004] INFO: rcu_sched self-detected stall on CPU { 7} (t=60004
jiffies g=674 c=673 q=100)
[ 330.748004] Task dump for CPU 7:
[ 330.748004] bind_netlink R running task 0 631 561
0x20020008
[ 330.748004] 0000000000000231 ffff880017dc3d68 ffffffff81086c26
ffffffff81086b88
[ 330.748004] 00000000f75af759 0000000000000007 ffffffff81c53940
ffff880017dc3d88
[ 330.748004] ffffffff8108a57f ffffffff81c53940 ffffffff81c53940
ffff880017dc3db8
[ 330.748004] Call Trace:
[ 330.748004] <IRQ> [<ffffffff81086c26>] sched_show_task+0x106/0x170
[ 330.748004] [<ffffffff81086b88>] ? sched_show_task+0x68/0x170
[ 330.748004] [<ffffffff8108a57f>] dump_cpu_task+0x3f/0x50
[ 330.748004] [<ffffffff810bfa7b>] rcu_dump_cpu_stacks+0x8b/0xc0
[ 330.748004] [<ffffffff810c33d0>] rcu_check_callbacks+0x480/0x6d0
[ 330.748004] [<ffffffff810a133d>] ? trace_hardirqs_off+0xd/0x10
[ 330.748004] [<ffffffff810c8408>] update_process_times+0x38/0x70
[ 330.748004] [<ffffffff810d8623>] tick_sched_handle.isra.15+0x33/0x70
[ 330.748004] [<ffffffff810d88cb>] tick_sched_timer+0x4b/0x80
[ 330.748004] [<ffffffff810c8d8b>] __run_hrtimer+0x9b/0x290
[ 330.748004] [<ffffffff810d8880>] ? tick_sched_do_timer+0x40/0x40
[ 330.748004] [<ffffffff810c95e4>] ? hrtimer_interrupt+0x74/0x260
[ 330.748004] [<ffffffff810c9677>] hrtimer_interrupt+0x107/0x260
[ 330.748004] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 330.748004] [<ffffffff81034cb9>] local_apic_timer_interrupt+0x39/0x60
[ 330.748004] [<ffffffff81769525>] smp_apic_timer_interrupt+0x45/0x60
[ 330.748004] [<ffffffff817678af>] apic_timer_interrupt+0x6f/0x80
[ 330.748004] <EOI> [<ffffffff817675e0>] ? retint_restore_args+0xe/0xe
[ 330.748004] [<ffffffff8165d817>] ? netlink_compare+0x7/0x30
[ 330.748004] [<ffffffff8139de4e>] ? rhashtable_lookup_compare+0x8e/0x120
[ 330.748004] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 330.748004] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 330.748004] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 330.748004] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 330.748004] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 330.748004] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 330.748004] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 330.748004] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 330.748004] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 330.748004] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 330.748004] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 330.748004] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 330.748004] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 330.748004] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 330.748004] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 330.771015] INFO: rcu_sched detected stalls on CPUs/tasks: { 7}
(detected by 1, t=60009 jiffies, g=674, c=673, q=100)
[ 330.772011] Task dump for CPU 7:
[ 330.772011] bind_netlink R running task 0 631 561
0x20020008
[ 330.772011] 0000000000000000 0000000000000000 ffff8800105d3f28
0000000000000002
[ 330.772011] 00000000ffdfca20 000000000000000c ffff8800105d3f28
ffffffff8161701e
[ 330.772011] ffff8800105d3f78 ffffffff81652318 0000000000000001
ffdfca38000013f3
[ 330.772011] Call Trace:
[ 330.772011] [<ffffffff8161701e>] ? SyS_bind+0xe/0x10
[ 330.772011] [<ffffffff81652318>] ? compat_SyS_socketcall+0xa8/0x200
[ 330.772011] [<ffffffff81768df3>] ? sysenter_dispatch+0x7/0x1f
[ 330.772011] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 356.724013] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s!
[bind_netlink:631]
[ 356.724013] Modules linked in: tipc
[ 356.724013] irq event stamp: 451061
[ 356.724013] hardirqs last enabled at (451060): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 356.724013] hardirqs last disabled at (451061): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 356.724013] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 356.724013] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 356.724013] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 356.724013] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 356.724013] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 356.724013] RIP: 0010:[<ffffffff8139de55>] [<ffffffff8139de55>]
rhashtable_lookup_compare+0x95/0x120
[ 356.724013] RSP: 0018:ffff8800105d3cf8 EFLAGS: 00000246
[ 356.724013] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 356.724013] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 356.724013] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 356.724013] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 356.724013] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 356.724013] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 356.724013] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 356.724013] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 356.724013] Stack:
[ 356.724013] ffffffff8139ddc0 000000028105d9c8 07ca27d400000000
ffff8800152d13c0
[ 356.724013] ffff8800105d3d48 ffff880016500000 ffff8800152d13c0
0000000000000004
[ 356.724013] ffff8800152d13c0 ffff880010c07608 ffff8800105d3da8
ffffffff8139dfaf
[ 356.724013] Call Trace:
[ 356.724013] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 356.724013] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 356.724013] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 356.724013] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 356.724013] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 356.724013] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 356.724013] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 356.724013] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 356.724013] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 356.724013] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 356.724013] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 356.724013] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 356.724013] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 356.724013] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 356.724013] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 356.724013] Code: 8b 02 83 e8 01 23 45 c4 48 83 c0 02 4c 8b 74 c2 08
41 f6 c6 01 75 1a 4c 89 f7 48 2b 7b 30 4c 89 ee 41 ff d4 84 c0 75 46 4d
8b 36 <41> f6 c6 01 74 e6 4c 39 7d c8 75 6c 48 c7 c2 61 de 39 81 be 01
[ 360.800040] INFO: task kworker/7:1:70 blocked for more than 120 seconds.
[ 360.800687] Tainted: G W L 3.19.0-rc6+ #182
[ 360.801176] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 360.801858] kworker/7:1 D ffff8800167c3af8 0 70 2
0x00000000
[ 360.802495] Workqueue: events rht_deferred_worker
[ 360.802943] ffff8800167c3af8 0000000000001ffb ffff8800167b8000
00000000000139c0
[ 360.803624] ffff8800167c3fd8 0000000000000000 00000000000139c0
ffff880013970000
[ 360.804388] ffff8800167b8000 0000000000000000 ffff8800167c3c58
7fffffffffffffff
[ 360.805071] Call Trace:
[ 360.805295] [<ffffffff81760879>] schedule+0x29/0x70
[ 360.805751] [<ffffffff81764de5>] schedule_timeout+0x1d5/0x230
[ 360.806262] [<ffffffff810a68fa>] ? mark_held_locks+0x6a/0x90
[ 360.806764] [<ffffffff817663b0>] ? _raw_spin_unlock_irq+0x30/0x40
[ 360.807323] [<ffffffff810a6a25>] ? trace_hardirqs_on_caller+0x105/0x1d0
[ 360.808109] [<ffffffff81761d1b>] wait_for_completion+0xbb/0x120
[ 360.808653] [<ffffffff81087910>] ? try_to_wake_up+0x3c0/0x3c0
[ 360.809178] [<ffffffff810c12f0>] ? __call_rcu.constprop.61+0x260/0x260
[ 360.809772] [<ffffffff810bea9d>] wait_rcu_gp+0x4d/0x60
[ 360.810246] [<ffffffff810beab0>] ? wait_rcu_gp+0x60/0x60
[ 360.810733] [<ffffffff810c1cfd>] synchronize_sched+0x5d/0x70
[ 360.811358] [<ffffffff8139e653>] rhashtable_shrink+0x113/0x150
[ 360.811876] [<ffffffff8139ea30>] rht_deferred_worker+0x80/0xa0
[ 360.812424] [<ffffffff81073449>] process_one_work+0x1b9/0x530
[ 360.812957] [<ffffffff810733d2>] ? process_one_work+0x142/0x530
[ 360.813481] [<ffffffff81073c5f>] worker_thread+0x11f/0x480
[ 360.813969] [<ffffffff81073b40>] ? rescuer_thread+0x340/0x340
[ 360.814480] [<ffffffff81079b4f>] kthread+0xef/0x110
[ 360.814919] [<ffffffff81079a60>] ? flush_kthread_worker+0xf0/0xf0
[ 360.815459] [<ffffffff817668ec>] ret_from_fork+0x7c/0xb0
[ 360.815945] [<ffffffff81079a60>] ? flush_kthread_worker+0xf0/0xf0
[ 360.816523] 3 locks held by kworker/7:1/70:
[ 360.816893] #0: ("events"){.+.+.+}, at: [<ffffffff810733d2>]
process_one_work+0x142/0x530
[ 360.817696] #1: ((&ht->run_work)){+.+.+.}, at: [<ffffffff810733d2>]
process_one_work+0x142/0x530
[ 360.818527] #2: (&ht->mutex){+.+.+.}, at: [<ffffffff8139e9dd>]
rht_deferred_worker+0x2d/0xa0
[ 360.819349] INFO: task jbd2/sda-8:116 blocked for more than 120 seconds.
[ 360.819943] Tainted: G W L 3.19.0-rc6+ #182
[ 360.820472] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 360.821157] jbd2/sda-8 D ffff8800159a3af8 0 116 2
0x00000000
[ 360.821832] ffff8800159a3af8 ffff8800159a3aa8 ffff8800159c2110
00000000000139c0
[ 360.822515] ffff8800159a3fd8 ffff8800159a3ab8 00000000000139c0
ffff880015a64220
[ 360.823222] ffff8800159c2110 ffffffff81118113 ffff880017d14330
ffff8800159c2110
[ 360.823907] Call Trace:
[ 360.824162] [<ffffffff81118113>] ? __delayacct_blkio_start+0x23/0x30
[ 360.824817] [<ffffffff81761250>] ? bit_wait_timeout+0x80/0x80
[ 360.825397] [<ffffffff81760879>] schedule+0x29/0x70
[ 360.825863] [<ffffffff8176094e>] io_schedule+0x8e/0xd0
[ 360.826406] [<ffffffff8176127c>] bit_wait_io+0x2c/0x50
[ 360.826966] [<ffffffff81760fe5>] __wait_on_bit+0x65/0x90
[ 360.827458] [<ffffffff811f83bc>] ? _submit_bh+0x11c/0x150
[ 360.827952] [<ffffffff81761250>] ? bit_wait_timeout+0x80/0x80
[ 360.828488] [<ffffffff8176113c>] out_of_line_wait_on_bit+0x7c/0x90
[ 360.829056] [<ffffffff81099f30>] ? wake_atomic_t_function+0x40/0x40
[ 360.829628] [<ffffffff811f64a9>] __wait_on_buffer+0x49/0x50
[ 360.830147] [<ffffffff812bce65>]
jbd2_journal_commit_transaction+0x16c5/0x1be0
[ 360.830806] [<ffffffff810a6afd>] ? trace_hardirqs_on+0xd/0x10
[ 360.831319] [<ffffffff810c6335>] ? del_timer_sync+0x5/0xd0
[ 360.831811] [<ffffffff812c0a21>] kjournald2+0xc1/0x280
[ 360.832331] [<ffffffff81099eb0>] ? prepare_to_wait_event+0x120/0x120
[ 360.832897] [<ffffffff812c0960>] ? commit_timeout+0x10/0x10
[ 360.833416] [<ffffffff81079b4f>] kthread+0xef/0x110
[ 360.833854] [<ffffffff81079a60>] ? flush_kthread_worker+0xf0/0xf0
[ 360.834398] [<ffffffff817668ec>] ret_from_fork+0x7c/0xb0
[ 360.834873] [<ffffffff81079a60>] ? flush_kthread_worker+0xf0/0xf0
[ 360.835413] no locks held by jbd2/sda-8/116.
[ 384.724008] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s!
[bind_netlink:631]
[ 384.724008] Modules linked in: tipc
[ 384.724008] irq event stamp: 465049
[ 384.724008] hardirqs last enabled at (465048): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 384.724008] hardirqs last disabled at (465049): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 384.724008] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 384.724008] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 384.724008] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 384.724008] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 384.724008] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 384.724008] RIP: 0010:[<ffffffff8165d810>] [<ffffffff8165d810>]
netlink_compare+0x0/0x30
[ 384.724008] RSP: 0018:ffff8800105d3cf0 EFLAGS: 00000286
[ 384.724008] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 384.724008] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 384.724008] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 384.724008] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 384.724008] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 384.724008] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 384.724008] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 384.724008] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 384.724008] Stack:
[ 384.724008] ffffffff8139de4e ffffffff8139ddc0 000000028105d9c8
07ca27d400000000
[ 384.724008] ffff8800152d13c0 ffff8800105d3d48 ffff880016500000
ffff8800152d13c0
[ 384.724008] 0000000000000004 ffff8800152d13c0 ffff880010c07608
ffff8800105d3da8
[ 384.724008] Call Trace:
[ 384.724008] [<ffffffff8139de4e>] ? rhashtable_lookup_compare+0x8e/0x120
[ 384.724008] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 384.724008] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 384.724008] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 384.724008] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 384.724008] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 384.724008] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 384.724008] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 384.724008] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 384.724008] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 384.724008] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 384.724008] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 384.724008] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 384.724008] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 384.724008] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 384.724008] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 384.724008] Code: c7 87 30 03 00 00 69 00 00 00 ff 93 50 04 00 00 f0
ff 83 98 01 00 00 48 83 c4 08 5b 5d c3 66 66 66 66 2e 0f 1f 84 00 00 00
00 00 <0f> 1f 44 00 00 31 c0 8b 56 08 39 97 68 04 00 00 55 48 89 e5 74
[ 412.724006] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 412.724006] Modules linked in: tipc
[ 412.724006] irq event stamp: 479037
[ 412.724006] hardirqs last enabled at (479036): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 412.724006] hardirqs last disabled at (479037): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 412.724006] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 412.724006] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 412.724006] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 412.724006] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 412.724006] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 412.724006] RIP: 0010:[<ffffffff8165d820>] [<ffffffff8165d820>]
netlink_compare+0x10/0x30
[ 412.724006] RSP: 0018:ffff8800105d3cf0 EFLAGS: 00000293
[ 412.724006] RAX: 0000000000000000 RBX: 0000000000000003 RCX:
00000000dbaee169
[ 412.724006] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 412.724006] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 412.724006] R10: 0000000000000003 R11: 0000000000000001 R12:
ffffffff817675e0
[ 412.724006] R13: ffffffff810c2d18 R14: ffff8800105d3c58 R15:
0000000000000046
[ 412.724006] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 412.724006] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 412.724006] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 412.724006] Stack:
[ 412.724006] ffffffff8139de4e ffffffff8139ddc0 000000028105d9c8
07ca27d400000000
[ 412.724006] ffff8800152d13c0 ffff8800105d3d48 ffff880016500000
ffff8800152d13c0
[ 412.724006] 0000000000000004 ffff8800152d13c0 ffff880010c07608
ffff8800105d3da8
[ 412.724006] Call Trace:
[ 412.724006] [<ffffffff8139de4e>] ? rhashtable_lookup_compare+0x8e/0x120
[ 412.724006] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 412.724006] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 412.724006] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 412.724006] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 412.724006] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 412.724006] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 412.724006] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 412.724006] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 412.724006] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 412.724006] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 412.724006] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 412.724006] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 412.724006] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 412.724006] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 412.724006] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 412.724006] Code: f0 ff 83 98 01 00 00 48 83 c4 08 5b 5d c3 66 66 66
66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 8b 56 08 39 97 68 04
00 00 <55> 48 89 e5 74 0a 5d c3 0f 1f 84 00 00 00 00 00 48 8b 47 30 48
[ 440.724005] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 440.724005] Modules linked in: tipc
[ 440.724005] irq event stamp: 493025
[ 440.724005] hardirqs last enabled at (493024): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 440.724005] hardirqs last disabled at (493025): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 440.724005] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 440.724005] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 440.724005] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 440.724005] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 440.724005] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 440.724005] RIP: 0010:[<ffffffff8165d820>] [<ffffffff8165d820>]
netlink_compare+0x10/0x30
[ 440.724005] RSP: 0018:ffff8800105d3cf0 EFLAGS: 00000293
[ 440.724005] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 440.724005] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 440.724005] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 440.724005] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 440.724005] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 440.724005] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 440.724005] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 440.724005] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 440.724005] Stack:
[ 440.724005] ffffffff8139de4e ffffffff8139ddc0 000000028105d9c8
07ca27d400000000
[ 440.724005] ffff8800152d13c0 ffff8800105d3d48 ffff880016500000
ffff8800152d13c0
[ 440.724005] 0000000000000004 ffff8800152d13c0 ffff880010c07608
ffff8800105d3da8
[ 440.724005] Call Trace:
[ 440.724005] [<ffffffff8139de4e>] ? rhashtable_lookup_compare+0x8e/0x120
[ 440.724005] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 440.724005] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 440.724005] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 440.724005] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 440.724005] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 440.724005] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 440.724005] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 440.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 440.724005] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 440.724005] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 440.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 440.724005] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 440.724005] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 440.724005] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 440.724005] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 440.724005] Code: f0 ff 83 98 01 00 00 48 83 c4 08 5b 5d c3 66 66 66
66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 8b 56 08 39 97 68 04
00 00 <55> 48 89 e5 74 0a 5d c3 0f 1f 84 00 00 00 00 00 48 8b 47 30 48
[ 468.724005] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
[bind_netlink:631]
[ 468.724005] Modules linked in: tipc
[ 468.724005] irq event stamp: 507013
[ 468.724005] hardirqs last enabled at (507012): [<ffffffff817675e0>]
restore_args+0x0/0x30
[ 468.724005] hardirqs last disabled at (507013): [<ffffffff817678aa>]
apic_timer_interrupt+0x6a/0x80
[ 468.724005] softirqs last enabled at (318168): [<ffffffff8161897f>]
lock_sock_nested+0x4f/0xc0
[ 468.724005] softirqs last disabled at (318170): [<ffffffff8139d79a>]
lock_buckets+0x3a/0x80
[ 468.724005] CPU: 7 PID: 631 Comm: bind_netlink Tainted: G W
L 3.19.0-rc6+ #182
[ 468.724005] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 468.724005] task: ffff880013970000 ti: ffff8800105d0000 task.ti:
ffff8800105d0000
[ 468.724005] RIP: 0010:[<ffffffff8165d820>] [<ffffffff8165d820>]
netlink_compare+0x10/0x30
[ 468.724005] RSP: 0018:ffff8800105d3cf0 EFLAGS: 00000293
[ 468.724005] RAX: 0000000000000000 RBX: ffffffff817675e0 RCX:
00000000dbaee169
[ 468.724005] RDX: 0000000000001668 RSI: ffff8800105d3db8 RDI:
ffff88001651b800
[ 468.724005] RBP: ffff8800105d3d48 R08: 00000000df2b8827 R09:
ffff880010c07468
[ 468.724005] R10: 0000000000000003 R11: 0000000000000001 R12:
ffff8800105d3c68
[ 468.724005] R13: 0000000000000046 R14: ffff8800105d0000 R15:
ffff880013970000
[ 468.724005] FS: 0000000000000000(0000) GS:ffff880017dc0000(0063)
knlGS:00000000f75ae900
[ 468.724005] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 468.724005] CR2: 0000000008760000 CR3: 0000000015121000 CR4:
00000000000006e0
[ 468.724005] Stack:
[ 468.724005] ffffffff8139de4e ffffffff8139ddc0 000000028105d9c8
07ca27d400000000
[ 468.724005] ffff8800152d13c0 ffff8800105d3d48 ffff880016500000
ffff8800152d13c0
[ 468.724005] 0000000000000004 ffff8800152d13c0 ffff880010c07608
ffff8800105d3da8
[ 468.724005] Call Trace:
[ 468.724005] [<ffffffff8139de4e>] ? rhashtable_lookup_compare+0x8e/0x120
[ 468.724005] [<ffffffff8139ddc0>] ? rhashtable_remove+0x200/0x200
[ 468.724005] [<ffffffff8139dfaf>]
rhashtable_lookup_compare_insert+0x9f/0x110
[ 468.724005] [<ffffffff8139df45>] ?
rhashtable_lookup_compare_insert+0x35/0x110
[ 468.724005] [<ffffffff8165d810>] ? netlink_overrun+0x50/0x50
[ 468.724005] [<ffffffff8165e0f3>] ? netlink_insert+0x43/0xf0
[ 468.724005] [<ffffffff8165e141>] netlink_insert+0x91/0xf0
[ 468.724005] [<ffffffff81660470>] netlink_bind+0x210/0x260
[ 468.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 468.724005] [<ffffffff810a5b10>] ? lock_release_non_nested+0xa0/0x340
[ 468.724005] [<ffffffff81616d24>] SYSC_bind+0xa4/0xc0
[ 468.724005] [<ffffffff81184fc6>] ? might_fault+0x66/0xc0
[ 468.724005] [<ffffffff8161701e>] SyS_bind+0xe/0x10
[ 468.724005] [<ffffffff81652318>] compat_SyS_socketcall+0xa8/0x200
[ 468.724005] [<ffffffff81768df3>] sysenter_dispatch+0x7/0x1f
[ 468.724005] [<ffffffff8139660e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 468.724005] Code: f0 ff 83 98 01 00 00 48 83 c4 08 5b 5d c3 66 66 66
66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 8b 56 08 39 97 68 04
00 00 <55> 48 89 e5 74 0a 5d c3 0f 1f 84 00 00 00 00 00 48 8b 47 30 48
Regards,
Ying
On 01/30/2015 08:20 AM, Thomas Graf wrote:
> This is a series of fixes which have accumulated while tracking
> down the race condition reoprted by Ying Xue. The original
> DEBUG_PAGEALLOC splat is resolved.
>
> However, there is still a race (harder to trigger) remaining in
> which certain entries are unfindable when removing them from the
> table via netlink_remove() and thus they cause a use after free
> later on.
>
> Regardless, these fixes can go in now.
>
> Thomas Graf (6):
> rhashtable: key_hashfn() must return full hash value
> rhashtable: Use a single bucket lock for sibling buckets
> rhashtable: Wait for RCU readers after final unzip work
> rhashtable: Dump bucket tables on locking violation under
> PROVE_LOCKING
> rhashtable: Add more lock verification
> rhashtable: Avoid bucket cross reference after removal
>
> lib/rhashtable.c | 301 ++++++++++++++++++++++++++++++-------------------------
> 1 file changed, 166 insertions(+), 135 deletions(-)
>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists