lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Thu, 18 Dec 2014 00:13:27 -0500
From:	Dave Jones <davej@...hat.com>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	Chris Mason <clm@...com>,
	Mike Galbraith <umgwanakikbuti@...il.com>,
	Ingo Molnar <mingo@...nel.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Dâniel Fraga <fragabr@...il.com>,
	Sasha Levin <sasha.levin@...cle.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Suresh Siddha <sbsiddha@...il.com>,
	Oleg Nesterov <oleg@...hat.com>,
	Peter Anvin <hpa@...ux.intel.com>
Subject: Re: frequent lockups in 3.18rc4

On Mon, Dec 15, 2014 at 03:46:41PM -0800, Linus Torvalds wrote:
 > On Mon, Dec 15, 2014 at 10:21 AM, Linus Torvalds
 > <torvalds@...ux-foundation.org> wrote:
 > >
 > > So let's just fix it. Here's a completely untested patch.
 > 
 > So after looking at this more, I'm actually really convinced that this
 > was a pretty nasty bug.
 > 
 > I'm *not* convinced that it's necessarily *your* bug, but I still
 > think it could be.

Bah, I was getting all optimistic.
I came home this evening to a locked up machine.
Serial console had a *lot* more traces than usual though.
Full log below.  The 12xxx.xxxxxx traces we seemed to recover from,
followed by silence for a while, before the real fun begins at 157xx.xxxxxx


[12525.310743] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [trinity-c60:28574]
[12525.311429] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12525.315877] CPU: 2 PID: 28574 Comm: trinity-c60 Not tainted 3.18.0+ #106
[12525.318136] task: ffff880153a20000 ti: ffff8801c251c000 task.ti: ffff8801c251c000
[12525.318931] RIP: 0010:[<ffffffff81373805>]  [<ffffffff81373805>] copy_user_enhanced_fast_string+0x5/0x10
[12525.319758] RSP: 0018:ffff8801c251fbe0  EFLAGS: 00010206
[12525.320555] RAX: 00007f9c44b07f1d RBX: 0000000000000003 RCX: 00000000000009c0
[12525.321403] RDX: 0000000000001000 RSI: 00007f9c44b0855d RDI: ffff88007cb1f640
[12525.322227] RBP: ffff8801c251fc28 R08: 0000000000000000 R09: 0000000000000001
[12525.323046] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff817cf22b
[12525.323854] R13: ffff8801c251fb68 R14: ffff880061dccd40 R15: ffff880061dcce60
[12525.324676] FS:  00007f9c4714c740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12525.325468] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12525.326262] CR2: 00007f9c45b905b8 CR3: 000000022a30b000 CR4: 00000000001407e0
[12525.327065] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12525.327867] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12525.328666] Stack:
[12525.329459]  ffffffff8119fe56 ffff8801c251fbf8 140000010718f000 0000000000001000
[12525.330283]  140000010718f000 0000000000001000 ffff8801c251fd60 0000000000000000
[12525.331139]  ffff880061dcd070 ffff8801c251fcc8 ffffffff81175fe7 ffff8801c251fc88
[12525.331969] Call Trace:
[12525.332801]  [<ffffffff8119fe56>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[12525.333642]  [<ffffffff81175fe7>] generic_perform_write+0xf7/0x1f0
[12525.334475]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12525.335315]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12525.336149]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12525.336981]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12525.337809]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12525.338630]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12525.339435]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12525.340238]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12525.341068]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12525.341865]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12525.342664]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12525.343486]  [<ffffffff811e98d2>] SyS_pwritev+0xc2/0xf0
[12525.344292]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12525.345099] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[12525.346820] sending NMI to other CPUs:
[12525.347621] NMI backtrace for cpu 3
[12525.348409] CPU: 3 PID: 25560 Comm: trinity-c201 Not tainted 3.18.0+ #106
[12525.350733] task: ffff880095342da0 ti: ffff8801c048c000 task.ti: ffff8801c048c000
[12525.351503] RIP: 0010:[<ffffffff810fbc4e>]  [<ffffffff810fbc4e>] generic_exec_single+0xee/0x1b0
[12525.352270] RSP: 0018:ffff8801c048f9f8  EFLAGS: 00000202
[12525.353017] RAX: 0000000000000008 RBX: ffff8801c048fa10 RCX: 0000000000000038
[12525.353757] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[12525.354479] RBP: ffff8801c048fa58 R08: ffff8802437418f0 R09: 0000000000000000
[12525.355187] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[12525.355878] R13: 0000000000000001 R14: ffff880061e34fc0 R15: 0000000000000003
[12525.356558] FS:  00007f9c4714c740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12525.357236] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12525.357905] CR2: 0000000001ac9498 CR3: 00000002247cb000 CR4: 00000000001407e0
[12525.358577] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12525.359246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12525.359911] Stack:
[12525.360568]  0000000000000000 0000000000000000 ffff8801c048fa78 0000000000000000
[12525.361250]  ffffffff81048cd0 ffff8801c048fb08 0000000000000003 00000000639bd7ce
[12525.361928]  ffff880095342da0 00000000ffffffff 0000000000000002 ffffffff81048cd0
[12525.362601] Call Trace:
[12525.363259]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12525.363926]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12525.364589]  [<ffffffff810fbdb0>] smp_call_function_single+0x70/0xd0
[12525.365250]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12525.365904]  [<ffffffff810fc4a9>] smp_call_function_many+0x2b9/0x320
[12525.366556]  [<ffffffff811a396d>] ? unmap_single_vma+0x50d/0x900
[12525.367204]  [<ffffffff81049020>] flush_tlb_mm_range+0x90/0x1d0
[12525.367843]  [<ffffffff811a25b2>] tlb_flush_mmu_tlbonly+0x42/0x50
[12525.368476]  [<ffffffff811a2bac>] tlb_flush_mmu+0x1c/0x30
[12525.369102]  [<ffffffff811a2bd4>] tlb_finish_mmu+0x14/0x40
[12525.369726]  [<ffffffff811a3e78>] zap_page_range_single+0x118/0x160
[12525.370348]  [<ffffffff811a4044>] unmap_mapping_range+0x134/0x190
[12525.370971]  [<ffffffff81192a5d>] shmem_fallocate+0x4fd/0x520
[12525.371591]  [<ffffffff810bcb77>] ? prepare_to_wait+0x27/0x90
[12525.372210]  [<ffffffff811e6062>] do_fallocate+0x132/0x1d0
[12525.372825]  [<ffffffff811b8558>] SyS_madvise+0x398/0x870
[12525.373439]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12525.374061] Code: 48 89 de 48 03 14 c5 a0 f2 d1 81 48 89 df e8 fa e0 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[12525.375458] NMI backtrace for cpu 0
[12525.375565] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 27.836 msecs
[12525.376788] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.18.0+ #106
[12525.378827] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[12525.379514] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12525.380215] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[12525.380905] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[12525.381588] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[12525.382254] RBP: ffffffff81c03e68 R08: 000000008baf8a0b R09: 0000000000000000
[12525.382911] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[12525.383558] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[12525.384194] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:00000000000000[12525.398359] NMI backtrace for cpu 1
[12525.398368] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 50.720 msecs
[12525.399764] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.18.0+ #106
[12525.401932] task: ffff8802434b96d0 ti: ffff880243748000 task.ti: ffff880243748000
[12525.402687] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12525.403457] RSP: 0018:ffff88024374be08  EFLAGS: 00000046
[12525.404241] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[12525.405038] RDX: 0000000000000000 RSI: ffff88024374bfd8 RDI: 0000000000000001
[12525.405832] RBP: ffff88024374be38 R08: 000000008baf8a0b R09: 0000000000000000
[12525.406611] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[12525.407377] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243748000
[12525.408131] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12525.408881] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12525.409601] CR2: 00000000000000d2 CR3: 0000000001c11000 CR4: 00000000001407e0
[12525.410310] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12525.411011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12525.411694] Stack:
[12525.412363]  000000014374be38 b532cae23f43ed20 ffffe8ffff003200 0000000000000005
[12525.413051]  ffffffff81cb1bc0 0000000000000001 ffff88024374be88 ffffffff8165f7b5
[12525.413744]  00000b65f893e397 ffffffff81cb1d90 ffffffff81cb1bc0 ffffffff81d215f0
[12525.414439] Call Trace:
[12525.415125]  [<ffffffff8165f7b5>] cpuidle_enter_state+0x55/0x190
[12525.415824]  [<ffffffff8165f9a7>] cpuidle_enter+0x17/0x20
[12525.416523]  [<ffffffff810bd665>] cpu_startup_entry+0x355/0x410
[12525.417216]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[12525.417904] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[12525.419435] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 71.790 msecs
[12549.296847] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [trinity-c60:28574]
[12549.297585] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12549.302426] CPU: 2 PID: 28574 Comm: trinity-c60 Tainted: G             L 3.18.0+ #106
[12549.304874] task: ffff880153a20000 ti: ffff8801c251c000 task.ti: ffff8801c251c000
[12549.305730] RIP: 0010:[<ffffffff81373805>]  [<ffffffff81373805>] copy_user_enhanced_fast_string+0x5/0x10
[12549.306609] RSP: 0018:ffff8801c251fbe0  EFLAGS: 00010206
[12549.307502] RAX: 00007f9c44b07f1d RBX: 0000000000000003 RCX: 00000000000009c0
[12549.308400] RDX: 0000000000001000 RSI: 00007f9c44b0855d RDI: ffff88007cb1f640
[12549.309284] RBP: ffff8801c251fc28 R08: 0000000000000000 R09: 0000000000000001
[12549.310159] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff817cf22b
[12549.311018] R13: ffff8801c251fb68 R14: ffff880061dccd40 R15: ffff880061dcce60
[12549.311862] FS:  00007f9c4714c740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12549.312716] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12549.313581] CR2: 00007f9c45b905b8 CR3: 000000022a30b000 CR4: 00000000001407e0
[12549.314418] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12549.315239] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12549.316051] Stack:
[12549.316861]  ffffffff8119fe56 ffff8801c251fbf8 140000010718f000 0000000000001000
[12549.317679]  140000010718f000 0000000000001000 ffff8801c251fd60 0000000000000000
[12549.318514]  ffff880061dcd070 ffff8801c251fcc8 ffffffff81175fe7 ffff8801c251fc88
[12549.319338] Call Trace:
[12549.320153]  [<ffffffff8119fe56>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[12549.320988]  [<ffffffff81175fe7>] generic_perform_write+0xf7/0x1f0
[12549.321824]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12549.322658]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12549.323484]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12549.324306]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12549.325133]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12549.325961]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12549.326783]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12549.327646]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12549.328470]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12549.329279]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12549.330078]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12549.330874]  [<ffffffff811e98d2>] SyS_pwritev+0xc2/0xf0
[12549.331672]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12549.332471] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[12549.334222] sending NMI to other CPUs:
[12549.335023] NMI backtrace for cpu 3
[12549.335811] CPU: 3 PID: 25560 Comm: trinity-c201 Tainted: G             L 3.18.0+ #106
[12549.338137] task: ffff880095342da0 ti: ffff8801c048c000 task.ti: ffff8801c048c000
[12549.338904] RIP: 0010:[<ffffffff810fbc4e>]  [<ffffffff810fbc4e>] generic_exec_single+0xee/0x1b0
[12549.339670] RSP: 0018:ffff8801c048f9f8  EFLAGS: 00000202
[12549.340416] RAX: 0000000000000008 RBX: ffff8801c048fa10 RCX: 0000000000000038
[12549.341157] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[12549.341879] RBP: ffff8801c048fa58 R08: ffff8802437418f0 R09: 0000000000000000
[12549.342587] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[12549.343280] R13: 0000000000000001 R14: ffff880061e34fc0 R15: 0000000000000003
[12549.343961] FS:  00007f9c4714c740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12549.344639] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12549.345309] CR2: 0000000001ac9498 CR3: 00000002247cb000 CR4: 00000000001407e0
[12549.345981] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12549.346649] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12549.347313] Stack:
[12549.347969]  0000000000000000 0000000000000000 ffff8801c048fa78 0000000000000000
[12549.348648]  ffffffff81048cd0 ffff8801c048fb08 0000000000000003 00000000639bd7ce
[12549.349325]  ffff880095342da0 00000000ffffffff 0000000000000002 ffffffff81048cd0
[12549.349997] Call Trace:
[12549.350653]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12549.351319]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12549.351983]  [<ffffffff810fbdb0>] smp_call_function_single+0x70/0xd0
[12549.352644]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12549.353298]  [<ffffffff810fc4a9>] smp_call_function_many+0x2b9/0x320
[12549.353954]  [<ffffffff811a396d>] ? unmap_single_vma+0x50d/0x900
[12549.354604]  [<ffffffff81049020>] flush_tlb_mm_range+0x90/0x1d0
[12549.355245]  [<ffffffff811a25b2>] tlb_flush_mmu_tlbonly+0x42/0x50
[12549.355881]  [<ffffffff811a2bac>] tlb_flush_mmu+0x1c/0x30
[12549.356508]  [<ffffffff811a2bd4>] tlb_finish_mmu+0x14/0x40
[12549.357131]  [<ffffffff811a3e78>] zap_page_range_single+0x118/0x160
[12549.357756]  [<ffffffff811a4044>] unmap_mapping_range+0x134/0x190
[12549.358379]  [<ffffffff81192a5d>] shmem_fallocate+0x4fd/0x520
[12549.359001]  [<ffffffff810bcb77>] ? prepare_to_wait+0x27/0x90
[12549.359620]  [<ffffffff811e6062>] do_fallocate+0x132/0x1d0
[12549.360239]  [<ffffffff811b8558>] SyS_madvise+0x398/0x870
[12549.360855]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12549.361476] Code: 48 89 de 48 03 14 c5 a0 f2 d1 81 48 89 df e8 fa e0 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[12549.362876] NMI backtrace for cpu 1
[12549.363530] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L 3.18.0+ #106
[12549.365548] task: ffff8802434b96d0 ti: ffff880243748000 task.ti: ffff880243748000
[12549.366222] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12549.366902] RSP: 0018:ffff88024374be08  EFLAGS: 00000046
[12549.367578] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[12549.368258] RDX: 0000000000000000 RSI: ffff88024374bfd8 RDI: 0000000000000001
[12549.368920] RBP: ffff88024374be38 R08: 000000008baf89f8 R09: 0000000000000000
[12549.369565] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[12549.370201] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243748000
[12549.370828] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12549.371451] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12549.372069][12563.088868] INFO: rcu_sched self-detected stall on CPU
[12563.089596] 	2: (1 GPs behind) idle=d4b/140000000000001/0 softirq=781780/781781 
[12563.090304] 	 (t=6000 jiffies g=382738 c=382737 q=0)
[12563.090996] Task dump for CPU 2:
[12563.091676] trinity-c60     R  running task    12768 28574  19163 0x0000000c
[12563.092378]  ffff880153a20000 000000005c7b7872 ffff880245203d68 ffffffff810a743c
[12563.093085]  ffffffff810a73a2 0000000000000002 0000000000000004 0000000000000002
[12563.093810]  ffffffff81c52380 0000000000000092 ffff880245203d88 ffffffff810ab4ed
[12563.094507] Call Trace:
[12563.095197]  <IRQ>  [<ffffffff810a743c>] sched_show_task+0x11c/0x190
[12563.095910]  [<ffffffff810a73a2>] ? sched_show_task+0x82/0x190
[12563.096636]  [<ffffffff810ab4ed>] dump_cpu_task+0x3d/0x50
[12563.097336]  [<ffffffff810d9b90>] rcu_dump_cpu_stacks+0x90/0xd0
[12563.098019]  [<ffffffff810e0723>] rcu_check_callbacks+0x503/0x770
[12563.098712]  [<ffffffff8113058c>] ? acct_account_cputime+0x1c/0x20
[12563.099408]  [<ffffffff810abe27>] ? account_system_time+0x97/0x180
[12563.100022]  [<ffffffff810e63fb>] update_process_times+0x4b/0x80
[12563.100623]  [<ffffffff810f6db3>] ? tick_sched_timer+0x23/0x1b0
[12563.101214]  [<ffffffff810f6ddf>] tick_sched_timer+0x4f/0x1b0
[12563.101795]  [<ffffffff810e729f>] __run_hrtimer+0xaf/0x240
[12563.102370]  [<ffffffff810e75ab>] ? hrtimer_interrupt+0x8b/0x260
[12563.102941]  [<ffffffff810f6d90>] ? tick_init_highres+0x20/0x20
[12563.103514]  [<ffffffff810e7627>] hrtimer_interrupt+0x107/0x260
[12563.104086]  [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70
[12563.104660]  [<ffffffff817d29c5>] smp_apic_timer_interrupt+0x45/0x60
[12563.105234]  [<ffffffff817d0daf>] apic_timer_interrupt+0x6f/0x80
[12563.105805]  <EOI>  [<ffffffff81373805>] ? copy_user_enhanced_fast_string+0x5/0x10
[12563.106387]  [<ffffffff8119fe56>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[12563.106965]  [<ffffffff81175fe7>] generic_perform_write+0xf7/0x1f0
[12563.107545]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12563.108128]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12563.108705]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12563.109299]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12563.109869]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12563.110440]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12563.111003]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12563.111566]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12563.112125]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12563.112686]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12563.113248]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12563.113810]  [<ffffffff811e98d2>] SyS_pwritev+0xc2/0xf0
[12563.114373]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12577.290614] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [trinity-c201:25560]
[12577.291259] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12577.295639] CPU: 3 PID: 25560 Comm: trinity-c201 Tainted: G             L 3.18.0+ #106
[12577.297656] task: ffff880095342da0 ti: ffff8801c048c000 task.ti: ffff8801c048c000
[12577.298367] RIP: 0010:[<ffffffff810fbc4a>]  [<ffffffff810fbc4a>] generic_exec_single+0xea/0x1b0
[12577.299091] RSP: 0018:ffff8801c048f9f8  EFLAGS: 00000202
[12577.299825] RAX: 0000000000000008 RBX: ffffffff817d0ae0 RCX: 0000000000000038
[12577.300553] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[12577.301296] RBP: ffff8801c048fa58 R08: ffff8802437418f0 R09: 0000000000000000
[12577.302007] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c048f968
[12577.302726] R13: 0000000000001a4a R14: ffff8801c048c000 R15: ffff880095342da0
[12577.303444] FS:  00007f9c4714c740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12577.304169] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12577.304897] CR2: 0000000001ac9498 CR3: 00000002247cb000 CR4: 00000000001407e0
[12577.305636] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12577.306386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12577.307114] Stack:
[12577.307830]  0000000000000000 0000000000000000 ffff8801c048fa78 0000000000000000
[12577.308585]  ffffffff81048cd0 ffff8801c048fb08 0000000000000003 00000000639bd7ce
[12577.309355]  ffff880095342da0 00000000ffffffff 0000000000000002 ffffffff81048cd0
[12577.310106] Call Trace:
[12577.310898]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12577.311664]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12577.312440]  [<ffffffff810fbdb0>] smp_call_function_single+0x70/0xd0
[12577.313196]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12577.313935]  [<ffffffff810fc4a9>] smp_call_function_many+0x2b9/0x320
[12577.314684]  [<ffffffff811a396d>] ? unmap_single_vma+0x50d/0x900
[12577.315429]  [<ffffffff81049020>] flush_tlb_mm_range+0x90/0x1d0
[12577.316182]  [<ffffffff811a25b2>] tlb_flush_mmu_tlbonly+0x42/0x50
[12577.316913]  [<ffffffff811a2bac>] tlb_flush_mmu+0x1c/0x30
[12577.317664]  [<ffffffff811a2bd4>] tlb_finish_mmu+0x14/0x40
[12577.318399]  [<ffffffff811a3e78>] zap_page_range_single+0x118/0x160
[12577.319123]  [<ffffffff811a4044>] unmap_mapping_range+0x134/0x190
[12577.319871]  [<ffffffff81192a5d>] shmem_fallocate+0x4fd/0x520
[12577.320624]  [<ffffffff810bcb77>] ? prepare_to_wait+0x27/0x90
[12577.321411]  [<ffffffff811e6062>] do_fallocate+0x132/0x1d0
[12577.322149]  [<ffffffff811b8558>] SyS_madvise+0x398/0x870
[12577.322868]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12577.323617] Code: c0 3a 1d 00 48 89 de 48 03 14 c5 a0 f2 d1 81 48 89 df e8 fa e0 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[12577.325246] sending NMI to other CPUs:
[12577.325989] NMI backtrace for cpu 2
[12577.326717] CPU: 2 PID: 28574 Comm: trinity-c60 Tainted: G             L 3.18.0+ #106
[12577.328845] task: ffff880153a20000 ti: ffff8801c251c000 task.ti: ffff8801c251c000
[12577.329548] RIP: 0010:[<ffffffff810c523d>]  [<ffffffff810c523d>] __lock_acquire.isra.31+0x33d/0x9f0
[12577.330250] RSP: 0018:ffff880245203d88  EFLAGS: 00000097
[12577.330933] RAX: 0000000000000000 RBX: ffff880153a20000 RCX: ffff8802453cff98
[12577.331610] RDX: 0000000000000980 RSI: 0000000000000010 RDI: 0000000000000000
[12577.332272] RBP: ffff880245203df8 R08: 0000000000000001 R09: 0000000000000000
[12577.332924] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000097
[12577.333564] R13: 0000000000000002 R14: ffff8802453cc898 R15: ffff880153a207e0
[12577.334195] FS:  00007f9c4714c740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12577.334831] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12577.335469] CR2: 00007f9c45b905b8 CR3: 000000022a30b000 CR4: 00000000001407e0
[12577.336116] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12577.336759] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12577.337399] Stack:
[12577.338024]  ffff880153a20000 0000000000000046 ffff880245203dd8 0000000000000092
[12577.338671]  ffff8802453d2f18 ffff8802453d2f00 0000000000000002 0000000000000000
[12577.339313]  ffff880245203dc8 0000000000000046 0000000000000000 0000000000000000
[12577.339955] Call Trace:
[12577.340585]  <IRQ> 
[12577.340593]  [<ffffffff810c5fff>] lock_acquire+0x9f/0x120
[12577.341830]  [<ffffffff810e72c6>] ? __run_hrtimer+0xd6/0x240
[12577.342450]  [<ffffffff810f6db3>] ? tick_sched_timer+0x23/0x1b0
[12577.343065]  [<ffffffff817cee9e>] _raw_spin_lock+0x3e/0x80
[12577.343673]  [<ffffffff810e72c6>] ? __run_hrtimer+0xd6/0x240
[12577.344277]  [<ffffffff810e72c6>] __run_hrtimer+0xd6/0x240
[12577.344886]  [<ffffffff810e75ab>] ? hrtimer_interrupt+0x8b/0x260
[12577.345493]  [<ffffffff810f6d90>] ? tick_init_highres+0x20/0x20
[12577.346099]  [<ffffffff810e7627>] hrtimer_interrupt+0x107/0x260
[12577.346703]  [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70
[12577.347309]  [<ffffffff817d29c5>] smp_apic_timer_interrupt+0x45/0x60
[12577.347916]  [<ffffffff817d0daf>] apic_timer_interrupt+0x6f/0x80
[12577.348522]  <EOI> 
[12577.348529]  [<ffffffff81373805>] ? copy_user_enhanced_fast_string+0x5/0x10
[12577.349750]  [<ffffffff8119fe56>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[12577.350375]  [<ffffffff81175fe7>] generic_perform_write+0xf7/0x1f0
[12577.351001]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12577.351626]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12577.352253]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12577.352883]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12577.353515]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12577.354147]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12577.354763]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12577.355366]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12577.355956]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12577.356550]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12577.357141]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12577.357713]  [<ffffffff811e98d2>] SyS_pwritev+0xc2/0xf0
[12577.358267]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12577.358815] Code: 00 00 48 83 c4 48 44 89 e8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 2e 0f 1f 84 00 00 00 00 00 41 81 fc fe 1f 00 00 0f 87 34 04 00 00 <45> 85 ed 48 8b 83 60 07 00 00 75 09 48 85 c0 0f 85 df 03 00 00 
[12577.360038] NMI backtrace for cpu 0
[12577.360598] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #106
[12577.381697] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L 3.18.0+ #106
[12577.383862] task: ffff8802434b96d0 ti: ffff880243748000 task.ti: ffff880243748000
[12577.384585] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12577.385332] RSP: 0018:ffff88024374be08  EFLAGS: 00000046
[12577.386066] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[12577.386807] RDX: 0000000000000000 RSI: ffff88024374bfd8 RDI: 0000000000000001
[12577.387545] RBP: ffff88024374be38 R08: 000000008baf89e1 R09: 0000000000000000
[12577.388285] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[12577.389028] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243748000
[12577.389752] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12577.390463] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12577.391170] CR2: 00007f5271c07000 CR3: 0000000001c11000 CR4: 00000000001407e0
[12577.391888] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12577.392584] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12577.393258] Stack:
[12577.393915]  000000014374be38 b532cae23f43ed20 ffffe8ffff003200 0000000000000005
[12577.394595]  ffffffff81cb1bc0 0000000000000001 ffff88024374be88 ffffffff8165f7b5
[12577.395278]  00000b72149ea4e1 ffffffff81cb1d90 ffffffff81cb1bc0 ffffffff81d215f0
[12577.395972] Call Trace:
[12577.396643]  [<ffffffff8165f7b5>] cpuidle_enter_state+0x55/0x190
[12577.397327]  [<ffffffff8165f9a7>] cpuidle_enter+0x17/0x20
[12577.398010]  [<ffffffff810bd665>] cpu_startup_entry+0x355/0x410
[12577.398691]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[12577.399364] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[12577.400864] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 74.851 msecs
[12591.532403] sched: RT throttling activated
[12613.269774] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [trinity-c12:30629]
[12613.270528] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12613.275557] CPU: 3 PID: 30629 Comm: trinity-c12 Tainted: G             L 3.18.0+ #106
[12613.278212] task: ffff8802433b5b40 ti: ffff880180f34000 task.ti: ffff880180f34000
[12613.279152] RIP: 0010:[<ffffffff8117e2df>]  [<ffffffff8117e2df>] free_hot_cold_page+0x11f/0x1a0
[12613.280117] RSP: 0018:ffff880180f37b38  EFLAGS: 00000202
[12613.281030] RAX: 0000000000000003 RBX: 0000000000000002 RCX: 0000000000000240
[12613.281939] RDX: ffff88024540d380 RSI: 0000000000000000 RDI: ffff88024e5d3f40
[12613.282817] RBP: ffff880180f37b78 R08: 0000000000000000 R09: 000000000024e600
[12613.283701] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88024e5d3f40
[12613.284572] R13: 0000000180f37b08 R14: ffffffff8117df5b R15: ffff880180f37b28
[12613.285454] FS:  00007f688fc33740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12613.286408] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12613.287338] CR2: 00007f688baa9000 CR3: 00000002276f9000 CR4: 00000000001407e0
[12613.288228] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12613.289102] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12613.289957] Stack:
[12613.290804]  00000000001ddf58 ffff88024e5d3e80 ffff8802433b5b40 ffffea00076a8540
[12613.291675]  ffffea000777d600 0000000000000000 ffffea00062ec580 ffff880180f37c00
[12613.292574]  ffff880180f37bb8 ffffffff8117e3c5 000000000000000d ffffea00076a8540
[12613.293471] Call Trace:
[12613.294325]  [<ffffffff8117e3c5>] free_hot_cold_page_list+0x65/0xd0
[12613.295179]  [<ffffffff81184f2d>] release_pages+0x1bd/0x270
[12613.296027]  [<ffffffff81185f63>] __pagevec_release+0x43/0x60
[12613.296869]  [<ffffffff81191f80>] shmem_undo_range+0x460/0x710
[12613.297704]  [<ffffffff81192248>] shmem_truncate_range+0x18/0x40
[12613.298541]  [<ffffffff811924d6>] shmem_setattr+0x116/0x1a0
[12613.299383]  [<ffffffff812075d1>] notify_change+0x241/0x390
[12613.300250]  [<ffffffff811e5ad5>] do_truncate+0x75/0xc0
[12613.301103]  [<ffffffff811e5e4a>] ? do_sys_ftruncate.constprop.14+0xda/0x160
[12613.301962]  [<ffffffff811e5e7f>] do_sys_ftruncate.constprop.14+0x10f/0x160
[12613.302815]  [<ffffffff81374f6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[12613.303656]  [<ffffffff811e5f0e>] SyS_ftruncate+0xe/0x10
[12613.304497]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12613.305362] Code: 8d 7b 20 e8 64 3b 20 00 41 8b 04 24 83 c0 01 41 3b 44 24 04 41 89 04 24 7d 52 41 f7 c6 00 02 00 00 74 31 e8 94 bf fc ff 41 56 9d <48> 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 90 49 8d 34 04 
[12613.307230] sending NMI to other CPUs:
[12613.308088] NMI backtrace for cpu 2
[12613.308871] CPU: 2 PID: 30778 Comm: trinity-c161 Tainted: G             L 3.18.0+ #106
[12613.311213] task: ffff8800478eada0 ti: ffff8802251dc000 task.ti: ffff8802251dc000
[12613.311995] RIP: 0010:[<ffffffff810c96bd>]  [<ffffffff810c96bd>] do_raw_spin_trylock+0x2d/0x50
[12613.312776] RSP: 0018:ffff8802251dfd88  EFLAGS: 00000246
[12613.313536] RAX: 0000000000008f8f RBX: ffff880235610238 RCX: 0000000000000000
[12613.314287] RDX: 0000000000008f8f RSI: 000000000000908f RDI: ffff880235610238
[12613.315022] RBP: ffff8802251dfd88 R08: 0000000000000000 R09: 0000000000000000
[12613.315743] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880235610250
[12613.316446] R13: ffff88009f3cc640 R14: 0000000000000000 R15: ffff8802356103c0
[12613.317133] FS:  00007f688fc33740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12613.317813] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12613.318483] CR2: 000000288076098d CR3: 000000022a0d2000 CR4: 00000000001407e0
[12613.319155] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12613.319819] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12613.320475] Stack:
[12613.321122]  ffff8802251dfdb8 ffffffff817ceea6 ffffffff81214c6a ffffffff817cf22b
[12613.321796]  ffff8802356101b0 ffff880235610238 ffff8802251dff08 ffffffff81214c6a
[12613.322474]  00000000251dfde8 7fffffffffffffff ffff88009f3cc520 ffff88017b9a3c28
[12613.323153] Call Trace:
[12613.323815]  [<ffffffff817ceea6>] _raw_spin_lock+0x46/0x80
[12613.324479]  [<ffffffff81214c6a>] ? sync_inodes_sb+0x1ca/0x2b0
[12613.325138]  [<ffffffff817cf22b>] ? _raw_spin_unlock+0x2b/0x40
[12613.325798]  [<ffffffff81214c6a>] sync_inodes_sb+0x1ca/0x2b0
[12613.326458]  [<ffffffff817cb01f>] ? wait_for_completion+0xff/0x130
[12613.327114]  [<ffffffff8121c560>] ? vfs_fsync+0x40/0x40
[12613.327761]  [<ffffffff8121c579>] sync_inodes_one_sb+0x19/0x20
[12613.328404]  [<ffffffff811ec002>] iterate_supers+0xb2/0x110
[12613.329041]  [<ffffffff8121c824>] sys_sync+0x44/0xb0
[12613.329669]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12613.330292] Code: 44 00 00 0f b7 17 55 31 c9 48 89 e5 38 d6 74 0e 89 c8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 8d b2 00 01 00 00 89 d0 f0 66 0f b1 37 <66> 39 d0 75 e0 b1 01 5d 65 8b 04 25 2c a0 00 00 89 47 08 65 48 
[12613.331704] NMI backtrace for cpu 0
[12613.332380] CPU: 0 PID: 1460 Comm: trinity-c155 Tainted: G             L 3.18.0+ #106
[12613.334512] task: ffff880232a72da0 ti: ffff880232820000 task.ti: ffff880232820000
[12613.335239] RIP: 0033:[<000000336ee93d7d>]  [<000000336ee93d7d>] 0x336ee93d7d
[12613.335984] RSP: 002b:00007fff9d870a18  EFLAGS: 00000206
[12613.336708] RAX: ffffff68f0ef8610 RBX: 0000000000000200 RCX: 00007fff9d870f00
[12613.337440] RDX: 00007fff9d871a00 RSI: 00007f688e769020 RDI: 00007fff9d870a20
[12613.338173] RBP: 00007fff9d871a60 R08: ffffff68f0ef8600 R09: ffffff68f0ef85f0
[12613.338905] R10: ffffff68f0ef85e0 R11: 0000000000000246 R12: 0000000000000129
[12613.339627] R13: 0000000000001000 R14: 0000000000001000 R15: 00007f688e6a9000
[12613.340332] FS:  00007f688fc33740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[12613.341040] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12613.341753] CR2: 00007f688e702000 CR3: 00000001c9675000 CR4: 00000000001407f0
[12613.342468] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12613.343161] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12613.343836] 
[12613.344498] NMI backtrace for cpu 1
[12613.345187] CPU: 1 PID: 30783 Comm: trinity-c166 Tainted: G             L 3.18.0+ #106
[12613.347247] task: ffff8801c059ada0 ti: ffff880227058000 task.ti: ffff880227058000
[12613.347978] RIP: 0033:[<000000336ee39bc0>]  [<000000336ee39bc0>] 0x336ee39bc0
[12613.348715] RSP: 002b:00007fff9d8719d8  EFLAGS: 00000202
[12613.349440] RAX: e8ffff3573f747e9 RBX: 00007f688f22e000 RCX: 0000000000000000
[12613.350175] RDX: 8000000000000000 RSI: 00007fff9d8719cc RDI: 000000336f1b76e0
[12613.350903] RBP: 00007f688f22e068 R08: 000000336f1b7120 R09: 000000336f1b7140
[12613.351639] R10: ffffffffffff9f00 R11: 0000000000000202 R12: 00007f688fbb7528
[12613.352362] R13: 00007f688f22e068 R14: 0000000000000000 R15: 0000000000000000
[12613.353089] FS:  00007f688fc33740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12613.353814] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12613.354537] CR2: 0000000000000008 CR3: 0000000153ac1000 CR4: 00000000001407e0
[12613.355265] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12613.355989] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12613.356701] 
[12641.253566] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [trinity-main:30108]
[12641.254492] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12641.260655] CPU: 3 PID: 30108 Comm: trinity-main Tainted: G             L 3.18.0+ #106
[12641.263755] task: ffff8800a0d15b40 ti: ffff880234ec4000 task.ti: ffff880234ec4000
[12641.264753] RIP: 0010:[<ffffffff811729d7>]  [<ffffffff811729d7>] task_bp_pinned.isra.6.constprop.12+0x37/0x90
[12641.265768] RSP: 0018:ffff880234ec7a40  EFLAGS: 00000287
[12641.266763] RAX: ffff88023f199d18 RBX: 000000000000005c RCX: 00000000001ce490
[12641.267751] RDX: ffff8801c87c2fa8 RSI: ffff8801ad860000 RDI: 0000000000000007
[12641.268728] RBP: ffff880234ec7a70 R08: ffffffff81d215f8 R09: ffff880245dce490
[12641.269700] R10: 0000000000000000 R11: 0000000000000008 R12: ffff880234ec7a30
[12641.270657] R13: ffffffff810abaf5 R14: ffff880234ec79b0 R15: ffffea000252c000
[12641.271618] FS:  00007f688fc33740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12641.272585] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641.273547] CR2: 000000336f1b8490 CR3: 0000000229035000 CR4: 00000000001407e0
[12641.274502] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12641.275443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 000000000017060a
[12641.276381] Stack:
[12641.277317]  0000000000000002 ffff8800a0d15b40 ffff880234ec7aa0 0000000000000001
[12641.278267]  ffff880094b06618 0000000000000007 ffff880234ec7ad0 ffffffff81172af6
[12641.279214]  ffffffff81c64868 0000000100000246 00000000001ce490 ffffffff81d215f8
[12641.280161] Call Trace:
[12641.281101]  [<ffffffff81172af6>] toggle_bp_slot.constprop.7+0xc6/0x1b0
[12641.282055]  [<ffffffff81172dfb>] __reserve_bp_slot+0x1eb/0x230
[12641.283017]  [<ffffffff81172e77>] reserve_bp_slot+0x27/0x40
[12641.283972]  [<ffffffff81172f68>] register_perf_hw_breakpoint+0x18/0x60
[12641.284884]  [<ffffffff81172fdf>] hw_breakpoint_event_init+0x2f/0x50
[12641.285766]  [<ffffffff8116fcfb>] perf_init_event+0x17b/0x1e0
[12641.286650]  [<ffffffff8116fb80>] ? perf_bp_event+0xd0/0xd0
[12641.287512]  [<ffffffff811700f8>] perf_event_alloc+0x398/0x440
[12641.288373]  [<ffffffff810e3c1c>] ? do_init_timer+0x5c/0x60
[12641.289239]  [<ffffffff81170e3e>] inherit_event.isra.90+0x8e/0x260
[12641.290095]  [<ffffffff8117104e>] inherit_task_group.isra.92.part.93+0x3e/0xe0
[12641.290959]  [<ffffffff811716e3>] perf_event_init_task+0x163/0x2e0
[12641.291812]  [<ffffffff81075756>] copy_process.part.26+0x726/0x1a40
[12641.292664]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12641.293514]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12641.294347]  [<ffffffff810424b2>] ? __do_page_fault+0x232/0x610
[12641.295158]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[12641.295969]  [<ffffffff811ea0b1>] ? __fput+0x191/0x200
[12641.296791]  [<ffffffff81374fe4>] ? lockdep_sys_exit_thunk+0x35/0x67
[12641.297597]  [<ffffffff81077066>] SyS_clone+0x16/0x20
[12641.298395]  [<ffffffff817d0299>] stub_clone+0x69/0x90
[12641.299173]  [<ffffffff817cff12>] ? system_call_fastpath+0x12/0x17
[12641.299935] Code: 54 53 48 83 ec 18 48 8b 05 57 1f af 00 48 8d 98 c0 fe ff ff 48 3d 10 49 c6 81 74 4c 41 89 fd 45 31 e4 eb 17 48 8b 93 40 01 00 00 <48> 8d 9a c0 fe ff ff 48 81 fa 10 49 c6 81 74 39 48 3b b3 28 01 
[12641.301567] sending NMI to other CPUs:
[12641.302311] NMI backtrace for cpu 1
[12641.303001] CPU: 1 PID: 3545 Comm: trinity-c7 Tainted: G             L 3.18.0+ #106
[12641.305035] task: ffff8800963f96d0 ti: ffff880215b28000 task.ti: ffff880215b28000
[12641.305715] RIP: 0010:[<ffffffff810fc466>]  [<ffffffff810fc466>] smp_call_function_many+0x276/0x320
[12641.306405] RSP: 0000:ffff880215b2baa8  EFLAGS: 00000202
[12641.307074] RAX: 0000000000000003 RBX: ffff8802451d3b00 RCX: ffff8802455d9488
[12641.307748] RDX: 0000000000000003 RSI: 0000000000000008 RDI: 0000000000000000
[12641.308423] RBP: ffff880215b2baf8 R08: ffff880243741e30 R09: 0000000100180011
[12641.309097] R10: ffff880244804240 R11: 0000000000000000 R12: 0000000000000003
[12641.309770] R13: 0000000000000000 R14: 0000000000000008 R15: 0000000000000008
[12641.310436] FS:  00007f688fc33740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12641.311099] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641.311757] CR2: 0000000002258ff8 CR3: 0000000180fec000 CR4: 00000000001407e0
[12641.312420] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12641.313082] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12641.313744] Stack:
[12641.314390]  ffff88024e5d4d00 0000000000000001 0000000100000003 00000000001d3ac0
[12641.315059]  0000004000000000 ffffffff82bea298 0000000000000001 ffffffff8117e150
[12641.315726]  0000000000000000 0000000000000000 ffff880215b2bb28 ffffffff810fc652
[12641.316393] Call Trace:
[12641.317045]  [<ffffffff8117e150>] ? drain_pages+0xc0/0xc0
[12641.317698]  [<ffffffff810fc652>] on_each_cpu_mask+0x32/0xa0
[12641.318351]  [<ffffffff8117c371>] drain_all_pages+0x101/0x120
[12641.319003]  [<ffffffff8118024f>] __alloc_pages_nodemask+0x7af/0xb40
[12641.319657]  [<ffffffff811c8ebe>] alloc_pages_vma+0xee/0x1b0
[12641.320310]  [<ffffffff811a416a>] ? do_wp_page+0xca/0x770
[12641.320959]  [<ffffffff811a416a>] do_wp_page+0xca/0x770
[12641.321608]  [<ffffffff811a69bb>] handle_mm_fault+0x6cb/0xe90
[12641.322258]  [<ffffffff8104241d>] ? __do_page_fault+0x19d/0x610
[12641.322910]  [<ffffffff8104248c>] __do_page_fault+0x20c/0x610
[12641.323557]  [<ffffffff811aa2dc>] ? validate_mm+0x15c/0x2c0
[12641.324204]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12641.324847]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12641.325494]  [<ffffffff81374fad>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[12641.326142]  [<ffffffff8104289c>] do_page_fault+0xc/0x10
[12641.326785]  [<ffffffff817d1b32>] page_fault+0x22/0x30
[12641.327423] Code: 00 41 89 c4 39 f0 0f 8d 27 fe ff ff 48 63 d0 48 8b 0b 48 03 0c d5 a0 f2 d1 81 f6 41 18 01 74 14 0f 1f 44 00 00 f3 90 f6 41 18 01 <75> f8 48 63 35 05 63 c2 00 83 f8 ff 48 8b 7b 08 74 b0 39 c6 77 
[12641.328871] NMI backtrace for cpu 0
[12641.329555] CPU: 0 PID: 30823 Comm: trinity-c206 Tainted: G             L 3.18.0+ #106
[12641.331677] task: ffff8801ad862da0 ti: ffff88017d7d0000 task.ti: ffff88017d7d0000
[12641.332396] RIP: 0010:[<ffffffff810fc466>]  [<ffffffff810fc466>] smp_call_function_many+0x276/0x320
[12641.333117] RSP: 0000:ffff88017d7d3a18  EFLAGS: 00000202
[12641.333827] RAX: 0000000000000003 RBX: ffff880244fd3b00 RCX: ffff8802455d7228
[12641.334546] RDX: 0000000000000003 RSI: 0000000000000008 RDI: 0000000000000000
[12641.335264] RBP: ffff88017d7d3a68 R08: ffff88024483b650 R09: 0000000100180011
[12641.335968] R10: ffff880244804240 R11: 0000000000000000 R12: 0000000000000003
[12641.336653] R13: 0000000000000000 R14: 0000000000000008 R15: 0000000000000008
[12641.337327] FS:  00007f688fc33740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[12641.338000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641.338665] CR2: 0000000002258ff8 CR3: 0000000232b93000 CR4: 00000000001407f0
[126[12641.356633] NMI backtrace for cpu 2
[12641.357349] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #106
[12641.359555] task: ffff8802434bc470 ti: ffff880243758000 task.ti: ffff880243758000
[12641.360314] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12641.361070] RSP: 0000:ffff88024375be08  EFLAGS: 00000046
[12641.361812] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[12641.362561] RDX: 0000000000000000 RSI: ffff88024375bfd8 RDI: 0000000000000002
[12641.363289] RBP: ffff88024375be38 R08: 000000008baf89af R09: 0000000000000000
[12641.363998] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[12641.364680] R13: 0000000000000000 R14: 0000000000000001 R15: ffff880243758000
[12641.365350] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12641.366017] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641.366678] CR2: 0000000002258ff8 CR3: 0000000097dd1000 CR4: 00000000001407e0
[12641.367347] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12641.368016] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12641.368680] Stack:
[12641.369340]  000000024375be38 5dccb806cd27b931 ffffe8ffff203200 0000000000000001
[12641.370021]  ffffffff81cb1bc0 0000000000000002 ffff88024375be88 ffffffff8165f7b5
[12641.370710]  00000b80fc4e5dd1 ffffffff81cb1c30 ffffffff81cb1bc0 ffffffff81d215f0
[12641.371403] Call Trace:
[12641.372073]  [<ffffffff8165f7b5>] cpuidle_enter_state+0x55/0x190
[12641.372758]  [<ffffffff8165f9a7>] cpuidle_enter+0x17/0x20
[12641.373441]  [<ffffffff810bd665>] cpu_startup_entry+0x355/0x410
[12641.374108]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[12641.374769] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[12669.237353] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [trinity-c59:7816]
[12669.238086] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12669.242957] CPU: 3 PID: 7816 Comm: trinity-c59 Tainted: G             L 3.18.0+ #106
[12669.245468] task: ffff8802258796d0 ti: ffff88007df64000 task.ti: ffff88007df64000
[12669.246378] RIP: 0010:[<ffffffff810c6014>]  [<ffffffff810c6014>] lock_acquire+0xb4/0x120
[12669.247281] RSP: 0018:ffff88007df67da8  EFLAGS: 00000246
[12669.248181] RAX: ffff8802258796d0 RBX: ffffffff810c512c RCX: ffff8802455cff98
[12669.249074] RDX: 00000000000001e0 RSI: 0000000000000010 RDI: 0000000000000000
[12669.249977] RBP: ffff88007df67e08 R08: 0000000000000000 R09: 0000000000000000
[12669.250881] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff810abaf5
[12669.251790] R13: ffff88007df67d18 R14: ffffffff810abaf5 R15: ffff88007df67d08
[12669.252693] FS:  00007f688fc33740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12669.253585] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12669.254486] CR2: 000000336f1b7740 CR3: 00000000a0ef4000 CR4: 00000000001407e0
[12669.255431] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12669.256368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12669.257263] Stack:
[12669.258097]  ffffffff81086b05 0000000000000000 0000000000000000 0000000000000246
[12669.258933]  0000000000000002 ffffffff81c50e20 ffff88007df67e48 00000000000024ca
[12669.259768]  0000000000000000 ffff88007df67eb0 ffff8800953896d0 0000000000000000
[12669.260638] Call Trace:
[12669.261471]  [<ffffffff81086b05>] ? group_send_sig_info+0x5/0xc0
[12669.262318]  [<ffffffff81086b48>] group_send_sig_info+0x48/0xc0
[12669.263170]  [<ffffffff81086b05>] ? group_send_sig_info+0x5/0xc0
[12669.264028]  [<ffffffff81086d05>] kill_pid_info+0x65/0xb0
[12669.264945]  [<ffffffff81086ca5>] ? kill_pid_info+0x5/0xb0
[12669.265851]  [<ffffffff81086e3c>] SYSC_kill+0xcc/0x240
[12669.266674]  [<ffffffff81086df8>] ? SYSC_kill+0x88/0x240
[12669.267508]  [<ffffffff8107aebb>] ? SyS_wait4+0x8b/0x110
[12669.268328]  [<ffffffff81088cde>] SyS_kill+0xe/0x10
[12669.269145]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12669.269958] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[12669.271728] sending NMI to other CPUs:
[12669.272548] NMI backtrace for cpu 1
[12669.273314] CPU: 1 PID: 7878 Comm: trinity-c219 Tainted: G             L 3.18.0+ #106
[12669.275686] task: ffff880096860000 ti: ffff880097f2c000 task.ti: ffff880097f2c000
[12669.276489] RIP: 0010:[<ffffffff811037c2>]  [<ffffffff811037c2>] is_module_text_address+0x22/0x30
[12669.277296] RSP: 0018:ffff880097f2f9e8  EFLAGS: 00000046
[12669.278089] RAX: 0000000000000000 RBX: ffff880097f2fd70 RCX: 0000000000000004
[12669.278876] RDX: ffff880227e01b80 RSI: 0000000000000000 RDI: ffff880097f2fd70
[12669.279650] RBP: ffff880097f2fa00 R08: ffff880097f2fb40 R09: 0000000000000000
[12669.280408] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[12669.281147] R13: ffff880097f2fd80 R14: ffff880097f2fd50 R15: ffffffff81801b30
[12669.281874] FS:  00007f688fc33740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12669.282595] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12669.283299] CR2: 0000000001cb9a28 CR3: 00000000948d4000 CR4: 00000000001407e0
[12669.283997] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12669.284677] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12669.285348] Stack:
[12669.286003]  ffffffff810973c8 0000000000000000 ffff880097f2fd70 ffff880097f2fa70
[12669.286678]  ffffffff81006a7f ffff880097f2fa20 ffff880097f2fff8 ffff880097f2c000
[12669.287346]  ffff880097f2fd80 ffff880097f2fb40 ffffffffffffc000 0000000000000000
[12669.288011] Call Trace:
[12669.288672]  [<ffffffff810973c8>] ? __kernel_text_address+0x58/0x80
[12669.289343]  [<ffffffff81006a7f>] print_context_stack+0x8f/0x100
[12669.290009]  [<ffffffff8100559f>] dump_trace+0x16f/0x350
[12669.290668]  [<ffffffff811b22a9>] ? anon_vma_clone+0x49/0x140
[12669.291321]  [<ffffffff811b22a9>] ? anon_vma_clone+0x49/0x140
[12669.291963]  [<ffffffff810135bf>] save_stack_trace+0x2f/0x50
[12669.292603]  [<ffffffff811ce860>] set_track+0x70/0x140
[12669.293240]  [<ffffffff817c26de>] alloc_debug_processing+0x92/0x118
[12669.293872]  [<ffffffff817c33b2>] __slab_alloc+0x4da/0x58f
[12669.294498]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12669.295119]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[12669.295745]  [<ffffffff811b22a9>] ? anon_vma_clone+0x49/0x140
[12669.296362]  [<ffffffff811b22a9>] ? anon_vma_clone+0x49/0x140
[12669.296969]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[12669.297577]  [<ffffffff811b22a9>] anon_vma_clone+0x49/0x140
[12669.298185]  [<ffffffff811b23cd>] anon_vma_fork+0x2d/0x100
[12669.298790]  [<ffffffff81076392>] copy_process.part.26+0x1362/0x1a40
[12669.299398]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12669.300009]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[12669.300619]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12669.301232]  [<ffffffff81374fe4>] ? lockdep_sys_exit_thunk+0x35/0x67
[12669.301850]  [<ffffffff81077066>] SyS_clone+0x16/0x20
[12669.302466]  [<ffffffff817d0299>] stub_clone+0x69/0x90
[12669.303062]  [<ffffffff817cff12>] ? system_call_fastpath+0x12/0x17
[12669.303646] Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 65 ff 04 25 e0 a9 00 00 e8 4a b7 ff ff 65 ff 0c 25 e0 a9 00 00 48 85 c0 5d <0f> 95 c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 
[12669.304961] NMI backtrace for cpu 0
[12669.305579] CPU: 0 PID: 5863 Comm: trinity-c230 Tainted: G             L 3.18.0+ #106
[12669.307401] task: ffff88023f830000 ti: ffff880096b74000 task.ti: ffff880096b74000
[12669.308023] RIP: 0010:[<ffffffff81191f24>]  [<ffffffff81191f24>] shmem_undo_range+0x404/0x710
[12669.308649] RSP: 0000:ffff880096b77c78  EFLAGS: 00000246
[12669.309266] RAX: 002ffe000008003d RBX: 000000000000000c RCX: 0000000000000835
[12669.309888] RDX: ffffea0005e915c0 RSI: ffffea0005e91580 RDI:[12669.325471] NMI backtrace for cpu 2
[12669.326185] CPU: 2 PID: 7188 Comm: trinity-c210 Tainted: G             L 3.18.0+ #106
[12669.328331] task: ffff8800a0d14470 ti: ffff8801c96d4000 task.ti: ffff8801c96d4000
[12669.329085] RIP: 0010:[<ffffffff810c4f21>]  [<ffffffff810c4f21>] __lock_acquire.isra.31+0x21/0x9f0
[12669.329858] RSP: 0018:ffff8801c96d79b8  EFLAGS: 00000096
[12669.330627] RAX: ffffffff81185068 RBX: ffff8800a0d14470 RCX: 0000000000000000
[12669.331406] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88024e5d3c58
[12669.332191] RBP: ffff8801c96d7a28 R08: 0000000000000001 R09: 0000000000000000
[12669.332984] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[12669.333780] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[12669.334577] FS:  00007f688fc33740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12669.335370] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12669.336152] CR2: 00007f688d8a9000 CR3: 00000002271f7000 CR4: 00000000001407e0
[12669.336934] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12669.337707] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12669.338474] Stack:
[12669.339224]  ffff8801c96d79f8 0000000093c05f9b 00000000000000d0 ffffffff82c00cc0
[12669.340023]  ffff8800a0d14470 00000000000200da ffff8800a0d15780 0000000000000000
[12669.340803]  ffff8801c96d7a08 0000000000000046 0000000000000000 0000000000000000
[12669.341541] Call Trace:
[12669.342270]  [<ffffffff810c5fff>] lock_acquire+0x9f/0x120
[12669.342998]  [<ffffffff81185068>] ? pagevec_lru_move_fn+0x88/0x100
[12669.343728]  [<ffffffff817cf0a9>] _raw_spin_lock_irqsave+0x49/0x90
[12669.344455]  [<ffffffff81185068>] ? pagevec_lru_move_fn+0x88/0x100
[12669.345190]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12669.345916]  [<ffffffff81185068>] pagevec_lru_move_fn+0x88/0x100
[12669.346640]  [<ffffffff81185160>] ? __pagevec_lru_add+0x20/0x20
[12669.347374]  [<ffffffff81185811>] __lru_cache_add+0x71/0x90
[12669.348097]  [<ffffffff81185bf9>] lru_cache_add_anon+0x19/0x20
[12669.348811]  [<ffffffff81190b08>] shmem_getpage_gfp+0x528/0x7a0
[12669.349559]  [<ffffffff81190dc2>] shmem_write_begin+0x42/0x70
[12669.350260]  [<ffffffff81175fc4>] generic_perform_write+0xd4/0x1f0
[12669.350957]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12669.351655]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12669.352348]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12669.353037]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12669.353738]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12669.354434]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12669.355126]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12669.355828]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12669.356531]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12669.357228]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12669.357921]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12669.358596]  [<ffffffff811e967c>] SyS_writev+0x5c/0x100
[12669.359258]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12669.359905] Code: 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 48 44 8b 25 48 80 be 00 65 48 8b 1c 25 00 aa 00 00 <45> 85 e4 0f 84 ef 00 00 00 44 8b 1d e7 32 ab 01 49 89 fe 41 89 
[12669.361354] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 88.798 msecs
[15739.449422] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u16:3:14112]
[15739.450087] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15739.454475] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15739.456686] Workqueue: khelper __call_usermodehelper
[15739.457473] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15739.458231] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15739.459015] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15739.459794] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15739.460570] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15739.461343] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15739.462113] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15739.462885] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15739.463648] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15739.464431] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15739.465203] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15739.465985] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15739.466766] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15739.467542] Stack:
[15739.468308]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15739.469106]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15739.469912]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15739.470717] Call Trace:
[15739.471511]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15739.472325]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15739.473138]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15739.473947]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15739.474757]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15739.475571]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15739.476374]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15739.477174]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15739.477983]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15739.478781]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15739.479590]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15739.480387]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15739.481184]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15739.481983]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15739.482797]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15739.483598]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15739.484382]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15739.485160]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15739.485922]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15739.486669]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15739.487407]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15739.488126]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15739.488834]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15739.489529]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15739.490197]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15739.490845]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15739.491474]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15739.492092]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15739.492689]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15739.493288] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15739.494617] sending NMI to other CPUs:
[15739.495221] NMI backtrace for cpu 3
[15739.495787] CPU: 3 PID: 1650 Comm: trinity-c76 Tainted: G             L 3.18.0+ #106
[15739.497539] task: ffff8801adac4470 ti: ffff880180e04000 task.ti: ffff880180e04000
[15739.498153] RIP: 0010:[<ffffffff810c6014>]  [<ffffffff810c6014>] lock_acquire+0xb4/0x120
[15739.498772] RSP: 0018:ffff880180e07dd8  EFLAGS: 00000246
[15739.499382] RAX: ffff8801adac4470 RBX: 0000000000000246 RCX: ffff8802455cff98
[15739.499994] RDX: 00000000000006a0 RSI: 0000000000000000 RDI: 0000000000000000
[15739.500609] RBP: ffff880180e07e38 R08: 0000000000000000 R09: 0000000000000000
[15739.501224] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[15739.501829] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
[15739.502422] FS:  00007f79977d8740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15739.503021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15739.503621] CR2: 0000000000000001 CR3: 00000001c9593000 CR4: 00000000001407e0
[15739.504217] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15739.504812] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15739.505403] Stack:
[15739.505988]  ffffffff8107a8e9 0000000000000000 ffff8801adac4470 0000000000000246
[15739.506603]  0000000127cc4da0 ffffffff81c0a098 ffff8801adac4470 ffffffff81c0a080
[15739.507220]  ffffffff81c0a098 ffff8801adac4470 ffff8801adac4470 ffff8801adac4470
[15739.507839] Call Trace:
[15739.508448]  [<ffffffff8107a8e9>] ? do_wait+0xd9/0x280
[15739.509064]  [<ffffffff817cf3d1>] _raw_read_lock+0x41/0x80
[15739.509678]  [<ffffffff8107a8e9>] ? do_wait+0xd9/0x280
[15739.510293]  [<ffffffff8107a8e9>] do_wait+0xd9/0x280
[15739.510907]  [<ffffffff8107aeb0>] SyS_wait4+0x80/0x110
[15739.511518]  [<ffffffff81078990>] ? task_stopped_code+0x60/0x60
[15739.512130]  [<ffffffff817d0109>] tracesys_phase2+0xd4/0xd9
[15739.512739] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15739.514121] NMI backtrace for cpu 1
[15739.514769] CPU: 1 PID: 2849 Comm: trinity-c95 Tainted: G             L 3.18.0+ #106
[15739.516771] task: ffff880096b4c470 ti: ffff880153a4c000 task.ti: ffff880153a4c000
[15739.517467] RIP: 0010:[<ffffffff810961f9>]  [<ffffffff810961f9>] find_pid_ns+0x39/0x90
[15739.518177] RSP: 0018:ffff880153a4fe78  EFLAGS: 00000207
[15739.518884] RAX: ffff88024e517120 RBX: 0000000000000d7f RCX: 0000000000000034
[15739.519599] RDX: ffff880094be6500 RSI: ffffffff81c486c0 RDI: 0000000000000d7f
[15739.520316] RBP: ffff880153a4fe78 R08: 0000000000000000 R09: 0000000000000000
[15739.521036] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[15739.521752] R13: ffff880096b4c470 R14: 0000000000000000 R15: 0000000000000000
[15739.522464] FS:  00007f79977d8740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15739.523181] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15739.523882] CR2: 00007f7996f24220 CR3: 00000002251b4000 CR4: 00000000001407e0
[15739.524580] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15739.525269] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15739.525957] Stack:
[15739.526632]  ffff880153a4fe88 ffffffff8109627f ffff880153a4ff68 ffffffff81086e2a
[15739.527319]  ffffffff81086df8 0000000000000000 ffff880096b4c840 0000000000000000
[15739.527992]  ffff880100000000 000003e800000b21 ffff880094be64c0 00007fff123fb870
[15739.528656] Call Trace:
[15739.529301]  [<ffffffff8109627f>] find_vpid+0x2f/0x50
[15739.529947]  [<ffffffff81086e2a>] SYSC_kill+0xba/0x240
[15739.530587]  [<ffffffff81086df8>] ? SYSC_kill+0x88/0x240
[15739[15763.435537] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u16:3:14112]
[15763.436252] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15763.441025] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15763.443320] Workqueue: khelper __call_usermodehelper
[15763.444090] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15763.444861] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15763.445654] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15763.446422] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15763.447201] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15763.447985] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15763.448764] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15763.449545] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15763.450319] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15763.451100] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15763.451876] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15763.452662] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15763.453451] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15763.454230] Stack:
[15763.455003]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15763.455818]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15763.456608]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15763.457393] Call Trace:
[15763.458166]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15763.458960]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15763.459753]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15763.460544]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15763.461338]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15763.462127]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15763.462921]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15763.463707]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15763.464503]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15763.465294]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15763.466086]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15763.466875]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15763.467664]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15763.468458]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15763.469251]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15763.470041]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15763.470828]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15763.471603]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15763.472371]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15763.473120]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15763.473854]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15763.474578]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15763.475287]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15763.475975]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15763.476640]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15763.477285]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15763.477915]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15763.478532]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15763.479131]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15763.479741] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15763.481087] sending NMI to other CPUs:
[15763.481693] NMI backtrace for cpu 3
[15763.482255] CPU: 3 PID: 1650 Comm: trinity-c76 Tainted: G             L 3.18.0+ #106
[15763.484005] task: ffff8801adac4470 ti: ffff880180e04000 task.ti: ffff880180e04000
[15763.484618] RIP: 0010:[<ffffffff810c47f5>]  [<ffffffff810c47f5>] lock_acquired+0x45/0x370
[15763.485234] RSP: 0018:ffff880180e07db8  EFLAGS: 00000046
[15763.485841] RAX: 0000000000000001 RBX: ffff880227cc4da0 RCX: 0000000000000001
[15763.486453] RDX: 000000000000dbdb RSI: ffffffff810bcc8d RDI: ffff880227cc4db8
[15763.487068] RBP: ffff880180e07df8 R08: 0000000000000000 R09: 0000000000000001
[15763.487687] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801adac4470
[15763.488303] R13: ffff880227cc4db8 R14: 0000000000000046 R15: ffff8801adac4460
[15763.488909] FS:  00007f79977d8740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15763.489522] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15763.490133] CR2: 0000000000000001 CR3: 00000001c9593000 CR4: 00000000001407e0
[15763.490747] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15763.491356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15763.491964] Stack:
[15763.492564]  0000000100000002 ffffffff810bcc8d ffff880180e07dd8 ffff880227cc4da0
[15763.493194]  ffff880227cc4db8 0000000000000292 ffff8801adac4470 ffff8801adac4460
[15763.493828]  ffff880180e07e38 ffffffff817cf0d5 ffffffff810bcc8d 0000000000000296
[15763.494464] Call Trace:
[15763.495088]  [<ffffffff810bcc8d>] ? remove_wait_queue+0x1d/0x40
[15763.495726]  [<ffffffff817cf0d5>] _raw_spin_lock_irqsave+0x75/0x90
[15763.496364]  [<ffffffff810bcc8d>] ? remove_wait_queue+0x1d/0x40
[15763.497005]  [<ffffffff810bcc8d>] remove_wait_queue+0x1d/0x40
[15763.497645]  [<ffffffff8107a95b>] do_wait+0x14b/0x280
[15763.498283]  [<ffffffff8107aeb0>] SyS_wait4+0x80/0x110
[15763.498917]  [<ffffffff81078990>] ? task_stopped_code+0x60/0x60
[15763.499551]  [<ffffffff817d0109>] tracesys_phase2+0xd4/0xd9
[15763.500183] Code: b8 00 45 85 c9 0f 84 d8 00 00 00 65 4c 8b 24 25 00 aa 00 00 45 8b 84 24 6c 07 00 00 45 85 c0 0f 85 be 00 00 00 49 89 fd 9c 41 5e <fa> 8b 35 1c 3a ab 01 41 c7 84 24 6c 07 00 00 01 00 00 00 41 8b 
[15763.501611] NMI backtrace for cpu 1
[15763.502274] CPU: 1 PID: 3298 Comm: trinity-c183 Tainted: G             L 3.18.0+ #106
[15763.504345] task: ffff880227de16d0 ti: ffff880071060000 task.ti: ffff880071060000
[15763.505065] RIP: 0033:[<000000336eebc2fc>]  [<000000336eebc2fc>] 0x336eebc2fc
[15763.505796] RSP: 002b:00007fff123fb868  EFLAGS: 00000246
[15763.506522] RAX: 0000000000000000 RBX: 0000000000000d7b RCX: ffffffffffffffff
[15763.507257] RDX: 000000000000000b RSI: 00007fff123fb870 RDI: 0000000000000d7b
[15763.507994] RBP: 0000000000000000 R08: 00007f79977d8740 R09: 0000000000000000
[15763.508729] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f7996cf6000
[15763.509460] R13: 00007f7996cf6068 R14: 0000000000000000 R15: 0000000000000000
[15763.510188] FS:  00007f79977d8740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15763.510908] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15763.511612] CR2: 00007f7996f24220 CR3: 000000009a646000 CR4: 00000000001407e0
[15763.512319] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15763.513027] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15763.513727] 
[15763.514403] NMI backtrace for cpu 0
[15763.515067] CPU: 0 PID: 1876 Comm: trinity-c189 Tainted: G             L 3.18.0+ #106
[15763.517087] task: ffff880096b4ada0 ti: ffff8802253d0000 task.ti: ffff8802253d0000
[15763.517776] RIP: 0010:[<[15779.306349] INFO: rcu_sched detected stalls on CPUs/tasks:
[15779.307024] 	(detected by 0, t=6002 jiffies, g=481360, c=481359, q=0)
[15779.307662] INFO: Stall ended before state dump start
[15787.421647] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u16:3:14112]
[15787.422333] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15787.426946] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15787.429181] Workqueue: khelper __call_usermodehelper
[15787.429945] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15787.430719] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15787.431507] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15787.432297] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15787.433094] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15787.433886] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15787.434671] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15787.435452] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15787.436234] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15787.437028] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15787.437816] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15787.438605] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15787.439400] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15787.440183] Stack:
[15787.440959]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15787.441792]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15787.442615]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15787.443422] Call Trace:
[15787.444217]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15787.445027]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15787.445833]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15787.446647]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15787.447449]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15787.448258]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15787.449067]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15787.449879]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15787.450688]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15787.451497]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15787.452307]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15787.453122]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15787.453928]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15787.454733]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15787.455524]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15787.456299]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15787.457062]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15787.457826]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15787.458587]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15787.459336]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15787.460078]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15787.460794]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15787.461495]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15787.462182]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15787.462850]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15787.463496]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15787.464124]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15787.464742]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15787.465341]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15787.465947] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15787.467291] sending NMI to other CPUs:
[15787.467904] NMI backtrace for cpu 3
[15787.468463] CPU: 3 PID: 1650 Comm: trinity-c76 Tainted: G             L 3.18.0+ #106
[15787.470211] task: ffff8801adac4470 ti: ffff880180e04000 task.ti: ffff880180e04000
[15787.470824] RIP: 0033:[<000000336eebc2fc>]  [<000000336eebc2fc>] 0x336eebc2fc
[15787.471440] RSP: 002b:00007fff123fb868  EFLAGS: 00000246
[15787.472043] RAX: 0000000000000000 RBX: 0000000000000d7e RCX: ffffffffffffffff
[15787.472652] RDX: 000000000000000b RSI: 00007fff123fb870 RDI: 0000000000000d7e
[15787.473259] RBP: 0000000000000000 R08: 00007f79977d8740 R09: 0000000000000000
[15787.473868] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f7997265000
[15787.474468] R13: 00007f7997265068 R14: 0000000000000000 R15: 0000000000000000
[15787.475063] FS:  00007f79977d8740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15787.475660] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15787.476254] CR2: 0000000000000001 CR3: 00000001c9593000 CR4: 00000000001407e0
[15787.476851] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15787.477442] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15787.478032] 
[15787.478615] NMI backtrace for cpu 1
[15787.479205] CPU: 1 PID: 2849 Comm: trinity-c95 Tainted: G             L 3.18.0+ #106
[15787.481052] task: ffff880096b4c470 ti: ffff880153a4c000 task.ti: ffff880153a4c000
[15787.481696] RIP: 0010:[<ffffffff817cfea0>]  [<ffffffff817cfea0>] system_call+0x0/0x3
[15787.482351] RSP: 0018:00007fff123fb868  EFLAGS: 00000046
[15787.483007] RAX: 000000000000003d RBX: 0000000000000d7f RCX: 000000336eebc2fc
[15787.483674] RDX: 000000000000000b RSI: 00007fff123fb870 RDI: 0000000000000d7f
[15787.484338] RBP: 0000000000000000 R08: 00007f79977d8740 R09: 0000000000000000
[15787.485000] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f799716e000
[15787.485658] R13: 00007f799716e068 R14: 0000000000000000 R15: 0000000000000000
[15787.486314] FS:  00007f79977d8740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15787.486979] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15787.487641] CR2: 00007f7996f24220 CR3: 00000002251b4000 CR4: 00000000001407e0
[15787.488311] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15787.488982] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15787.489650] Stack:
[15787.490312]  000000000041401b 00007f79976944e8 00007f799716e000 0000000000000001
[15787.490999]  00007f799716e07c 00007f799716e000 0000000000000000 00007f799716e07c
[15787.491690]  0000000000416c03 00000000000066b2 0000000000000155 00000000cccccccd
[15787.492383] Call Trace:
[15787.493063]  <UNK> 
[15787.493070] Code: 8b 3c 24 4c 8b 74 24 08 4c 8b 6c 24 10 4c 8b 64 24 18 48 8b 6c 24 20 48 8b 5c 24 28 48 83 c4 30 e9 74 02 00 00 66 0f 1f 44 00 00 <0f> 01 f8 65 48 89 24 25 80 a0 00 00 65 48 8b 24 25 08 aa 00 00 
[15787.495270] NMI backtrace for cpu 0
[15787.495987] CPU: 0 PID: 1876 Comm: trinity-c189 Tainted: G             L 3.18.0+ #106
[15787.498136] task: ffff880096b4ada0 ti: ffff8802253d0000 task.ti: ffff8802253d0000
[15787.498870] RIP: 0010:[<ffffffff810c9698>]  [<ffffffff810c9698>] do_raw_spin_trylock+0x8/0x50
[15787.499618] RSP: 0018:ffff880244e03d00  EFLAGS: 00000092
[15787.500344] RAX: ffff880096b4ada0 RBX: ffff880240c51578 RCX: ffff880244fcff98
[15787.501064] RDX: 0000000000004a4a RSI: 0000000000000018 RDI: ffff880240c51578
[15787.501780] RBP: ffff880244e03d38 R08: 0000000000000001 R09: 0000000000000000
[15787.502487] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880240c51590
[15787.503190] R13: 0000000000000092 R14: ff[15811.407761] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u16:3:14112]
[15811.408334] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15811.412296] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15811.414324] Workqueue: khelper __call_usermodehelper
[15811.415025] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15811.415733] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15811.416455] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15811.417171] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15811.417913] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15811.418645] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15811.419374] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15811.420102] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15811.420831] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15811.421572] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15811.422317] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15811.423070] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15811.423828] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15811.424583] Stack:
[15811.425327]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15811.426099]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15811.426872]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15811.427657] Call Trace:
[15811.428437]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15811.429231]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15811.430025]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15811.430814]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15811.431600]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15811.432394]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15811.433180]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15811.433972]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15811.434768]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15811.435559]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15811.436349]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15811.437141]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15811.437937]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15811.438725]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15811.439519]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15811.440307]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15811.441094]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15811.441871]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15811.442637]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15811.443385]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15811.444121]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15811.444838]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15811.445538]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15811.446219]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15811.446883]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15811.447527]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15811.448159]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15811.448776]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15811.449371]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15811.449968] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15811.451292] sending NMI to other CPUs:
[15811.451901] NMI backtrace for cpu 3
[15811.452459] CPU: 3 PID: 1650 Comm: trinity-c76 Tainted: G             L 3.18.0+ #106
[15811.454206] task: ffff8801adac4470 ti: ffff880180e04000 task.ti: ffff880180e04000
[15811.454817] RIP: 0010:[<ffffffff810c511a>]  [<ffffffff810c511a>] __lock_acquire.isra.31+0x21a/0x9f0
[15811.455437] RSP: 0018:ffff880180e07d18  EFLAGS: 00000002
[15811.456045] RAX: 0000000000000008 RBX: ffff8801adac4470 RCX: 0000000000000000
[15811.456660] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[15811.457274] RBP: ffff880180e07d88 R08: 0000000000000001 R09: 0000000000000000
[15811.457889] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000014e
[15811.458493] R13: 0000000000000000 R14: ffff880227cc4db8 R15: ffff8801adac4be0
[15811.459092] FS:  00007f79977d8740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15811.459699] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15811.460303] CR2: 0000000000000001 CR3: 00000001c9593000 CR4: 00000000001407e0
[15811.460907] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15811.461507] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15811.462108] Stack:
[15811.462702]  ffff880180e07d98 ffffffff810c512c 0000000000000102 0000000000000000
[15811.463325]  ffff880180e07d48 ffffffff810abaf5 ffff880180e07dc8 0000000000000000
[15811.463945]  ffff880180e07dd8 0000000000000046 0000000000000000 0000000000000000
[15811.464564] Call Trace:
[15811.465173]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15811.465797]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15811.466421]  [<ffffffff810c5fff>] lock_acquire+0x9f/0x120
[15811.467047]  [<ffffffff810bcc8d>] ? remove_wait_queue+0x1d/0x40
[15811.467672]  [<ffffffff817cf0a9>] _raw_spin_lock_irqsave+0x49/0x90
[15811.468300]  [<ffffffff810bcc8d>] ? remove_wait_queue+0x1d/0x40
[15811.468924]  [<ffffffff810bcc8d>] remove_wait_queue+0x1d/0x40
[15811.469546]  [<ffffffff8107a95b>] do_wait+0x14b/0x280
[15811.470166]  [<ffffffff8107aeb0>] SyS_wait4+0x80/0x110
[15811.470782]  [<ffffffff81078990>] ? task_stopped_code+0x60/0x60
[15811.471398]  [<ffffffff817d0109>] tracesys_phase2+0xd4/0xd9
[15811.472011] Code: e0 7f 44 09 d0 41 88 47 31 41 0f b6 47 32 83 e0 f0 45 85 c0 0f 95 c2 09 c8 c1 e2 03 09 d0 41 88 47 32 0f b7 55 18 41 0f b7 47 32 <c1> e2 04 83 e0 0f 09 d0 66 41 89 47 32 e8 a4 69 fe ff 4c 8b 4d 
[15811.473409] NMI backtrace for cpu 1
[15811.474061] CPU: 1 PID: 2849 Comm: trinity-c95 Tainted: G             L 3.18.0+ #106
[15811.476098] task: ffff880096b4c470 ti: ffff880153a4c000 task.ti: ffff880153a4c000
[15811.476808] RIP: 0010:[<ffffffff810c63cf>]  [<ffffffff810c63cf>] lock_release+0x1f/0x240
[15811.477535] RSP: 0018:ffff880153a4fe40  EFLAGS: 00000246
[15811.478253] RAX: ffff880096b4c470 RBX: 0000000000000000 RCX: 00000000000003a0
[15811.478982] RDX: ffffffff81086d0c RSI: 0000000000000001 RDI: ffffffff81c50e20
[15811.479707] RBP: ffff880153a4fe48 R08: 0000000000000000 R09: 0000000000000000
[15811.480419] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880094be64c0
[15811.481115] R13: ffff880153a4feb0 R14: 0000000000000000 R15: 0000000000000000
[15811.481803] FS:  00007f79977d8740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15811.482504] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15811.483200] CR2: 00007f7996f24220 CR3: 00000002251b4000 CR4: 00000000001407e0
[15811.483890] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15811.484562] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15811.485221] Stack:
[15811.485862]  0000000000000000 ffff880153a4fe88 ffffffff81086d24 ffffffff81086ca5
[15811.486521]  0000000000000d7f 0000000000000d7f 0000000000000000 ffff880096[15835.393872] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u16:3:14112]
[15835.394432] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15835.398378] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15835.400379] Workqueue: khelper __call_usermodehelper
[15835.401080] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15835.401795] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15835.402534] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15835.403252] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15835.403995] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15835.404727] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15835.405456] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15835.406185] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15835.406916] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15835.407667] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15835.408410] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15835.409165] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15835.409924] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15835.410679] Stack:
[15835.411433]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15835.412226]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15835.413002]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15835.413784] Call Trace:
[15835.414563]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15835.415358]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15835.416162]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15835.416952]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15835.417745]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15835.418533]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15835.419328]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15835.420114]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15835.420904]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15835.421701]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15835.422491]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15835.423282]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15835.424077]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15835.424865]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15835.425658]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15835.426447]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15835.427239]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15835.428018]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15835.428776]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15835.429525]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15835.430259]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15835.430986]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15835.431694]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15835.432374]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15835.433049]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15835.433694]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15835.434328]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15835.434947]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15835.435543]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15835.436148] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15835.437494] sending NMI to other CPUs:


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists