| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <1507339518-16059-1-git-send-email-eswierk@skyportsystems.com>
Date: Fri, 6 Oct 2017 18:25:18 -0700
From: Ed Swierk <eswierk@...portsystems.com>
To: eswierk@...portsystems.com, linux-kernel@...r.kernel.org
Subject: Which came first, hard kernel lockup or SATA errors?
I'm trying to untangle a series of problems that suddenly occurred on
a dual-socket Xeon server system that had been running a bunch of KVM
workloads just fine for over 6 weeks (4.4.52-grsec kernel,
Debian-derived userspace).
Here are the highlights, with timestamps in seconds:
[3851435] NMI watchdog: Watchdog detected hard LOCKUP on cpu 13
[3851438] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0
[3851450] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [pocviexg:2136]
[3851450] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [CPU 0/KVM:9662]
[3851450] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [CPU 1/KVM:10302]
[3851478] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [pocviexg:2136]
[3851478] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [CPU 0/KVM:9662]
[3851478] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [CPU 1/KVM:10302]
[3851482] BUG: spinlock lockup suspected on CPU#0, migration/0/10
[3851482] Sending NMI to all CPUs:
[3851482] NMI backtrace for cpu 0
[3851483] NMI backtrace for cpu 1
[3851483] NMI backtrace for cpu 2
[3851483] NMI backtrace for cpu 3
[3851483] NMI backtrace for cpu 4
[3851484] ata8.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen **
[3851484] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen **
[3851484] NMI backtrace for cpu 5
[3851484] NMI backtrace for cpu 6
...
[3851489] NMI backtrace for cpu 24
[3851489] ata8.00: qc timeout (cmd 0xec) **
[3851489] ata7.00: qc timeout (cmd 0xec) **
[3851489] NMI backtrace for cpu 25
...
[3851490] NMI backtrace for cpu 31
[3851499] ata8.00: qc timeout (cmd 0xec) **
[3851499] ata7.00: qc timeout (cmd 0xec) **
[3851510] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [CPU 0/KVM:9662]
...and so on for 2 more hours, until the system was finally rebooted
First I want to know if I can blame the whole mess on flaky
hardware. The error messages marked with ** are from IO operations on
two SATA SSDs. On one hand, getting errors from two different drives
at the exact same time suggests that the drives themselves aren't the
problem. On the other hand, the fact that the system has two other
drives connected to the same controller that continued to work (even
though all were being accessed more or less simultaneously in a
software RAID-10 configuration) suggests that the controller isn't
totally broken. (The controller is embedded in the C610 PCH on the
motherboard.) No SATA errors have occurred on that system before or
since.
Is there any chance that those SATA errors could have been caused by
the kernel lockups--say due to a timing-sensitive interrupt not
getting serviced? Or by any other conceivable software problem?
If not, is it plausible that a hardware problem with the SATA
controller or drives caused the kernel lockups, even though the first
hard lockups were reported 48 seconds before the first SATA error
messages?
Following is an excerpt of the console log. Let me know if any other
details would be helpful.
[3851435.777762] NMI watchdog: Watchdog detected hard LOCKUP on cpu 13
[3851435.784582] Modules linked in: vhost_net tun vhost macvtap 8021q garp mrp drbg ansi_cprng dm_crypt algif_skcipher af_alg ip_gre ip_tunnel gre macvlan ebtable_filter ebtables openvswitch nf_defrag_ipv6 libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul raid10 crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac lpc_ich mfd_core mei_me mei i2c_i801 ioatdma ipmi_ssif ipmi_msghandler acpi_pad acpi_cpufreq squashfs lz4_decompress xhci_pci xhci_hcd ixgbe mdio vxlan ip6_udp_tunnel udp_tunnel ptp pps_core dca ehci_pci ehci_hcd usbcore ahci libahci usb_common tpm_tis
[3851435.874884] irq event stamp: 3828060294
[3851435.879398] hardirqs last enabled at (3828060293): [<ffffffff810db86c>] ffffffff810db86c
[3851435.888810] hardirqs last disabled at (3828060294): [<ffffffff8170ff74>] ffffffff8170ff74
[3851435.898254] softirqs last enabled at (3828060156): [<ffffffff8107f31e>] ffffffff8107f31e
[3851435.907666] softirqs last disabled at (3828060139): [<ffffffff8107f648>] ffffffff8107f648
[3851435.917094] CPU: 13 PID: 2146 Comm: pocviexg Not tainted 4.4.52-grsec #1
[3851435.925252] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851435.937473] task: ffff881c9ff35c00 ti: ffff881c9ff372b0 task.ti: ffff881c9ff372b0
[3851435.946104] RIP: 0010:[<ffffffff810ad854>] [<ffffffff810ad854>] try_to_wake_up+0xe4/0x500
[3851435.955614] RSP: 0000:ffffc90016beba40 EFLAGS: 00000002
[3851435.961780] RAX: 0000000000000000 RBX: ffff88102cc8e3d8 RCX: 0000000000000001
[3851435.970016] RDX: 0000000000000001 RSI: 00000000000001e4 RDI: 0000000000000000
[3851435.978259] RBP: ffffc90016beba80 R08: 0000000000000000 R09: 0000000000000001
[3851435.986493] R10: 00000000000000e0 R11: ffff881c9ff35c00 R12: 0000000000000082
[3851435.994727] R13: ffff88102cc8dc00 R14: 00000000001d3f00 R15: ffffffff81f25060
[3851436.002962] FS: 00006e79227fc700(0000) GS:ffff882034000000(0000) knlGS:0000000000000000
[3851436.012266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851436.018920] CR2: 00006e79227fba38 CR3: 000000201d2fe000 CR4: 00000000003626f0
[3851436.027155] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851436.035391] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851436.043626] Stack:
[3851436.046098] ffff8820341ce620 ffff8820341ce608 0000000000000000 ffff8820341ce600
[3851436.054695] ffffc90016bebb48 000000000000000d 0000000000000000 ffff881033fce600
[3851436.063259] ffffc90016beba90 ffffffff810adc80 ffffc90016bebbe8 ffffffff8112c830
[3851436.071868] Call Trace:
[3851436.074862] [<ffffffff810adc80>] wake_up_process+0x10/0x20
[3851436.081317] [<ffffffff8112c830>] stop_two_cpus+0x1b0/0x280
[3851436.087778] [<ffffffff8112c490>] ? cpu_stop_park+0x40/0x40
[3851436.094237] [<ffffffff8112c490>] ? cpu_stop_park+0x40/0x40
[3851436.100715] [<ffffffff810ad560>] ? __migrate_swap_task.part.94+0x70/0x70
[3851436.108539] [<ffffffff810ade01>] migrate_swap+0xf1/0x1f0
[3851436.114800] [<ffffffff810b9d55>] task_numa_migrate+0x345/0xb60
[3851436.121656] [<ffffffff811092e3>] ? futex_wait_queue_me+0xf3/0x160
[3851436.128801] [<ffffffff810ba5e3>] numa_migrate_preferred+0x73/0x80
[3851436.135936] [<ffffffff810bb944>] task_numa_fault+0x7a4/0xee0
[3851436.142604] [<ffffffff810bafe0>] ? should_numa_migrate_memory+0x50/0x130
[3851436.150437] [<ffffffff811b2db6>] handle_mm_fault+0xfd6/0x1e10
[3851436.157189] [<ffffffff811b1e28>] ? handle_mm_fault+0x48/0x1e10
[3851436.164051] [<ffffffff810a85aa>] ? ___might_sleep+0x1ea/0x220
[3851436.170804] [<ffffffff8106405f>] __do_page_fault+0x1bf/0x460
[3851436.177475] [<ffffffff8110ca44>] ? sys_futex+0x74/0x230
[3851436.183642] [<ffffffff8106432c>] do_page_fault+0x2c/0x40
[3851436.189909] [<ffffffff817128b8>] page_fault+0x28/0x30
[3851436.195892] Code: 0f 85 e0 02 00 00 41 8b 85 b8 16 00 00 89 45 d0 41 8b 7d 4c 49 c7 c6 00 3f 1d 00 85 ff 0f 85 68 01 00 00 41 8b 4d 30 85 c9 74 0a <f3> 90 41 8b 55 30 85 d2 75 f6 49 8b 45 00 a8 02 b8 00 00 00 00
[3851438.906435] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0
[3851438.913158] Modules linked in: vhost_net tun vhost macvtap 8021q garp mrp drbg ansi_cprng dm_crypt algif_skcipher af_alg ip_gre ip_tunnel gre macvlan ebtable_filter ebtables openvswitch nf_defrag_ipv6 libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul raid10 crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac lpc_ich mfd_core mei_me mei i2c_i801 ioatdma ipmi_ssif ipmi_msghandler acpi_pad acpi_cpufreq squashfs lz4_decompress xhci_pci xhci_hcd ixgbe mdio vxlan ip6_udp_tunnel udp_tunnel ptp pps_core dca ehci_pci ehci_hcd usbcore ahci libahci usb_common tpm_tis
[3851439.003452] irq event stamp: 25935634
[3851439.007769] hardirqs last enabled at (25935633): [<ffffffff81710341>] ffffffff81710341
[3851439.017010] hardirqs last disabled at (25935634): [<ffffffff8170ff74>] ffffffff8170ff74
[3851439.026226] softirqs last enabled at (25935520): [<ffffffff8107f31e>] ffffffff8107f31e
[3851439.035441] softirqs last disabled at (25935445): [<ffffffff8107f648>] ffffffff8107f648
[3851439.044657] CPU: 0 PID: 10 Comm: migration/0 Not tainted 4.4.52-grsec #1
[3851439.052892] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851439.065111] task: ffff88102cc8dc00 ti: ffff88102cc8f2b0 task.ti: ffff88102cc8f2b0
[3851439.073747] RIP: 0010:[<ffffffff813c4935>] [<ffffffff813c4935>] delay_tsc+0x25/0x70
[3851439.082679] RSP: 0000:ffffc900006ebb40 EFLAGS: 00000002
[3851439.088852] RAX: 000000002591f96b RBX: ffff8820341ce608 RCX: 0000000000000018
[3851439.101462] RDX: 001cab2e00000000 RSI: 001cab2e2591f96b RDI: 0000000000000001
[3851439.109695] RBP: ffffc900006ebb40 R08: 0000000000000000 R09: 0000000000000000
[3851439.117950] R10: 0000000000000038 R11: 0000000000000000 R12: 0000000021d37081
[3851439.126198] R13: 0000000000000001 R14: 000000007d234b28 R15: 0000000000000000
[3851439.134440] FS: 0000000000000000(0000) GS:ffff881033e00000(0000) knlGS:0000000000000000
[3851439.143741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851439.150393] CR2: 000000c420148008 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851439.158627] Stack:
[3851439.161096] ffffc900006ebb50 ffffffff813c487a ffffc900006ebb80 ffffffff810dba87
[3851439.169669] ffff8820341ce608 ffff8820341ce620 0000000000000086 00000000000006ae
[3851439.178228] ffffc900006ebbb0 ffffffff817101a2 ffffffff8112c1cb ffff8820341ce600
[3851439.186825] Call Trace:
[3851439.189784] [<ffffffff813c487a>] __delay+0xa/0x10
[3851439.195385] [<ffffffff810dba87>] do_raw_spin_lock+0x87/0x160
[3851439.202062] [<ffffffff817101a2>] _raw_spin_lock_irqsave+0xa2/0xb0
[3851439.209238] [<ffffffff8112c1cb>] ? cpu_stop_queue_work+0x2b/0x70
[3851439.216287] [<ffffffff8112c1cb>] cpu_stop_queue_work+0x2b/0x70
[3851439.223149] [<ffffffff8112c92b>] stop_one_cpu_nowait+0x2b/0x30
[3851439.229999] [<ffffffff810c1127>] load_balance+0x827/0x960
[3851439.236371] [<ffffffff810eae3d>] ? debug_lockdep_rcu_enabled+0x1d/0x20
[3851439.243995] [<ffffffff810c1608>] pick_next_task_fair+0x3a8/0x750
[3851439.251062] [<ffffffff810c14db>] ? pick_next_task_fair+0x27b/0x750
[3851439.258289] [<ffffffff81708fdd>] __schedule+0x1ad/0xbd0
[3851439.264456] [<ffffffff810d461d>] ? trace_hardirqs_on_caller+0x13d/0x1d0
[3851439.272204] [<ffffffff81709a37>] schedule+0x37/0x80
[3851439.277984] [<ffffffff810a51cb>] smpboot_thread_fn+0x16b/0x1e0
[3851439.284822] [<ffffffff810a5060>] ? sort_range+0x20/0x20
[3851439.291001] [<ffffffff810a124c>] kthread+0xfc/0x120
[3851439.296796] [<ffffffff8170adf1>] ? wait_for_completion+0xe1/0x110
[3851439.303936] [<ffffffff810a1150>] ? kthread_create_on_node+0x240/0x240
[3851439.311464] [<ffffffff81710aae>] ret_from_fork+0x3e/0x70
[3851439.317738] [<ffffffff810a1150>] ? kthread_create_on_node+0x240/0x240
[3851439.325281] Code: 00 5d c3 0f 1f 00 55 65 ff 05 b8 65 c4 7e 48 89 e5 65 44 8b 05 35 48 c4 7e 0f ae e8 0f 31 48 c1 e2 20 48 89 d6 48 09 c6 0f ae e8 <0f> 31 48 c1 e2 20 48 09 d0 48 89 c1 48 29 f1 48 39 cf 76 27 65
[3851450.239084] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [pocviexg:2136]
[3851450.247648] Modules linked in: vhost_net tun vhost macvtap 8021q garp mrp drbg ansi_cprng dm_crypt algif_skcipher af_alg ip_gre ip_tunnel gre macvlan ebtable_filter ebtables openvswitch nf_defrag_ipv6 libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul raid10 crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac lpc_ich mfd_core mei_me mei i2c_i801 ioatdma ipmi_ssif ipmi_msghandler acpi_pad acpi_cpufreq squashfs lz4_decompress xhci_pci xhci_hcd ixgbe mdio vxlan ip6_udp_tunnel udp_tunnel ptp pps_core dca ehci_pci ehci_hcd usbcore ahci libahci usb_common tpm_tis
[3851450.337782] irq event stamp: 2820288514
[3851450.342294] hardirqs last enabled at (2820288513): [<ffffffff81711261>] ffffffff81711261
[3851450.351088] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [CPU 0/KVM:9662]
[3851450.351137] Modules linked in: vhost_net tun vhost macvtap 8021q garp mrp drbg ansi_cprng dm_crypt algif_skcipher af_alg ip_gre ip_tunnel gre macvlan ebtable_filter ebtables openvswitch nf_defrag_ipv6 libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul raid10 crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac lpc_ich mfd_core mei_me mei i2c_i801 ioatdma ipmi_ssif ipmi_msghandler acpi_pad acpi_cpufreq squashfs lz4_decompress xhci_pci xhci_hcd ixgbe mdio vxlan ip6_udp_tunnel udp_tunnel ptp pps_core dca ehci_pci ehci_hcd usbcore ahci libahci usb_common tpm_tis
[3851450.351141] irq event stamp: 3489453974
[3851450.351143] hardirqs last enabled at (3489453973): [<ffffffff81711261>] ffffffff81711261
[3851450.351144] hardirqs last disabled at (3489453974): [<ffffffff81711489>] ffffffff81711489
[3851450.351145] softirqs last enabled at (3489453972): [<ffffffff8107f31e>] ffffffff8107f31e
[3851450.351146] softirqs last disabled at (3489453957): [<ffffffff8107f648>] ffffffff8107f648
[3851450.351149] CPU: 12 PID: 9662 Comm: CPU 0/KVM Not tainted 4.4.52-grsec #1
[3851450.351150] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851450.351151] task: ffff881c996d8000 ti: ffff881c996d96b0 task.ti: ffff881c996d96b0
[3851450.351158] RIP: 0010:[<ffffffff8110d717>] [<ffffffff8110d717>] smp_call_function_single+0xb7/0x120
[3851450.351159] RSP: 0018:ffffc90012543a70 EFLAGS: 00000202
[3851450.351160] RAX: 0000000000000000 RBX: 000000000000000d RCX: ffff8820341d9040
[3851450.351161] RDX: 0000000000000001 RSI: ffffc90012543a80 RDI: ffffc90012543a80
[3851450.351161] RBP: ffffc90012543ab0 R08: 0000000000000001 R09: 0000000000000000
[3851450.351162] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0bc6660
[3851450.351162] R13: 000000000000000c R14: 000000000000000c R15: 0000000000000000
[3851450.351164] FS: 00006b7b59ee9700(0000) GS:ffff882033e00000(0000) knlGS:0000000000000000
[3851450.351164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851450.351165] CR2: 00007f7b8743f000 CR3: 0000001c9bca5000 CR4: 00000000003626f0
[3851450.351166] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851450.351167] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851450.351167] Stack:
[3851450.351169] ffffc90012543b40 0000000000000001 ffff8820341d9040 ffffffffa0bc6660
[3851450.351170] ffff881bb6b1ae10 0000000000000003 ffff881bb6b18000 000000000000000c
[3851450.351172] ffffc90012543b90 ffffffffa0bcdaa4 ffff881c996d88a8 00000002996d8000
[3851450.351172] Call Trace:
[3851450.351182] [<ffffffffa0bc6660>] ? handle_pause+0x1c0/0x1c0 [kvm_intel]
[3851450.351185] [<ffffffffa0bcdaa4>] vmx_vcpu_load+0x1e4/0x2d0 [kvm_intel]
[3851450.351191] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851450.351193] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851450.351216] [<ffffffffa0b314e2>] kvm_arch_vcpu_load+0x32/0x2a0 [kvm]
[3851450.351230] [<ffffffffa0b18714>] kvm_sched_in+0x34/0x40 [kvm]
[3851450.351233] [<ffffffff810aad78>] finish_task_switch+0x188/0x250
[3851450.351235] [<ffffffff810aac57>] ? finish_task_switch+0x67/0x250
[3851450.351238] [<ffffffff81709253>] __schedule+0x423/0xbd0
[3851450.351239] [<ffffffff81709a37>] schedule+0x37/0x80
[3851450.351252] [<ffffffffa0b1c69e>] kvm_vcpu_block+0x6e/0x4b0 [kvm]
[3851450.351255] [<ffffffff810c8880>] ? wake_up_atomic_t+0x30/0x30
[3851450.351270] [<ffffffffa0b375d3>] kvm_arch_vcpu_ioctl_run+0x533/0x19c0 [kvm]
[3851450.351285] [<ffffffffa0b37a97>] ? kvm_arch_vcpu_ioctl_run+0x9f7/0x19c0 [kvm]
[3851450.351297] [<ffffffffa0b17c8d>] ? vcpu_load+0x1d/0x60 [kvm]
[3851450.351310] [<ffffffffa0b192b6>] kvm_vcpu_ioctl+0x2d6/0x620 [kvm]
[3851450.351312] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851450.351314] [<ffffffff810b4f4c>] ? local_clock+0x1c/0x20
[3851450.351317] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851450.351319] [<ffffffff81214aec>] do_vfs_ioctl+0x44c/0x820
[3851450.351321] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851450.351322] [<ffffffff81214f34>] sys_ioctl+0x74/0x80
[3851450.351324] [<ffffffff817106fa>] entry_SYSCALL_64_fastpath+0x16/0x7e
[3851450.351344] Code: 65 ff 0d dd d7 ef 7e 48 83 c4 30 5b 41 5c 5d c3 48 8d 75 d0 48 89 d1 89 df 4c 89 e2 e8 a3 fd ff ff 8b 55 e8 83 e2 01 74 d7 f3 90 <8b> 55 e8 83 e2 01 75 f6 eb cb 48 83 3d d7 18 e1 00 00 74 52 9c
[3851450.407090] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [CPU 1/KVM:10302]
[3851450.407140] Modules linked in: vhost_net tun vhost macvtap 8021q garp mrp drbg ansi_cprng dm_crypt algif_skcipher af_alg ip_gre ip_tunnel gre macvlan ebtable_filter ebtables openvswitch nf_defrag_ipv6 libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul raid10 crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac lpc_ich mfd_core mei_me mei i2c_i801 ioatdma ipmi_ssif ipmi_msghandler acpi_pad acpi_cpufreq squashfs lz4_decompress xhci_pci xhci_hcd ixgbe mdio vxlan ip6_udp_tunnel udp_tunnel ptp pps_core dca ehci_pci ehci_hcd usbcore ahci libahci usb_common tpm_tis
[3851450.407144] irq event stamp: 3177319932
[3851450.407146] hardirqs last enabled at (3177319931): [<ffffffff81711261>] ffffffff81711261
[3851450.407147] hardirqs last disabled at (3177319932): [<ffffffff81711489>] ffffffff81711489
[3851450.407147] softirqs last enabled at (3177319930): [<ffffffff8107f31e>] ffffffff8107f31e
[3851450.407148] softirqs last disabled at (3177319915): [<ffffffff8107f648>] ffffffff8107f648
[3851450.407150] CPU: 14 PID: 10302 Comm: CPU 1/KVM Tainted: G L 4.4.52-grsec #1
[3851450.407151] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851450.407152] task: ffff881ef2ae0000 ti: ffff881ef2ae16b0 task.ti: ffff881ef2ae16b0
[3851450.407156] RIP: 0010:[<ffffffff8110d717>] [<ffffffff8110d717>] smp_call_function_single+0xb7/0x120
[3851450.407156] RSP: 0018:ffffc90014b2ba70 EFLAGS: 00000202
[3851450.407157] RAX: 0000000000000000 RBX: 000000000000000d RCX: ffffc90012543a80
[3851450.407158] RDX: 0000000000000001 RSI: ffffc90014b2ba80 RDI: ffffc90014b2ba80
[3851450.407158] RBP: ffffc90014b2bab0 R08: 0000000000000001 R09: 0000000000000000
[3851450.407159] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0bc6660
[3851450.407159] R13: 000000000000000e R14: 000000000000000e R15: 0000000000000000
[3851450.407161] FS: 00007847e8e5d700(0000) GS:ffff882034200000(0000) knlGS:0000000000000000
[3851450.407161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851450.407162] CR2: 00007f96a99dc000 CR3: 00000018bb771000 CR4: 00000000003626f0
[3851450.407163] Stack:
[3851450.407165] ffffc90014b2bb40 0000000000000001 ffffc90012543a80 ffffffffa0bc6660
[3851450.407166] ffff8818b5ca2e10 0000000000000003 ffff8818b5ca0000 000000000000000e
[3851450.407168] ffffc90014b2bb90 ffffffffa0bcdaa4 ffff881ef2ae08a8 00000002f2ae0000
[3851450.407168] Call Trace:
[3851450.407174] [<ffffffffa0bc6660>] ? handle_pause+0x1c0/0x1c0 [kvm_intel]
[3851450.407177] [<ffffffffa0bcdaa4>] vmx_vcpu_load+0x1e4/0x2d0 [kvm_intel]
[3851450.407180] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851450.407182] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851450.407199] [<ffffffffa0b314e2>] kvm_arch_vcpu_load+0x32/0x2a0 [kvm]
[3851450.407213] [<ffffffffa0b18714>] kvm_sched_in+0x34/0x40 [kvm]
[3851450.407216] [<ffffffff810aad78>] finish_task_switch+0x188/0x250
[3851450.407217] [<ffffffff810aac57>] ? finish_task_switch+0x67/0x250
[3851450.407219] [<ffffffff81709253>] __schedule+0x423/0xbd0
[3851450.407220] [<ffffffff81709a37>] schedule+0x37/0x80
[3851450.407233] [<ffffffffa0b1c69e>] kvm_vcpu_block+0x6e/0x4b0 [kvm]
[3851450.407235] [<ffffffff810c8880>] ? wake_up_atomic_t+0x30/0x30
[3851450.407250] [<ffffffffa0b375d3>] kvm_arch_vcpu_ioctl_run+0x533/0x19c0 [kvm]
[3851450.407264] [<ffffffffa0b37a97>] ? kvm_arch_vcpu_ioctl_run+0x9f7/0x19c0 [kvm]
[3851450.407277] [<ffffffffa0b17c8d>] ? vcpu_load+0x1d/0x60 [kvm]
[3851450.407289] [<ffffffffa0b192b6>] kvm_vcpu_ioctl+0x2d6/0x620 [kvm]
[3851450.407291] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851450.407293] [<ffffffff810b4f4c>] ? local_clock+0x1c/0x20
[3851450.407295] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851450.407296] [<ffffffff81214aec>] do_vfs_ioctl+0x44c/0x820
[3851450.407298] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851450.407299] [<ffffffff81214f34>] sys_ioctl+0x74/0x80
[3851450.407301] [<ffffffff817106fa>] entry_SYSCALL_64_fastpath+0x16/0x7e
[3851450.407320] Code: 65 ff 0d dd d7 ef 7e 48 83 c4 30 5b 41 5c 5d c3 48 8d 75 d0 48 89 d1 89 df 4c 89 e2 e8 a3 fd ff ff 8b 55 e8 83 e2 01 74 d7 f3 90 <8b> 55 e8 83 e2 01 75 f6 eb cb 48 83 3d d7 18 e1 00 00 74 52 9c
[3851451.295949] hardirqs last disabled at (2820288514): [<ffffffff81711489>] ffffffff81711489
[3851451.305371] softirqs last enabled at (2820288512): [<ffffffff8107f31e>] ffffffff8107f31e
[3851451.314788] softirqs last disabled at (2820288503): [<ffffffff8107f648>] ffffffff8107f648
[3851451.324200] CPU: 8 PID: 2136 Comm: pocviexg Tainted: G L 4.4.52-grsec #1
[3851451.333602] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851451.345833] task: ffff881c9ff34500 ti: ffff881c9ff35bb0 task.ti: ffff881c9ff35bb0
[3851451.354454] RIP: 0010:[<ffffffff8110da9b>] [<ffffffff8110da9b>] smp_call_function_many+0x21b/0x280
[3851451.364885] RSP: 0000:ffffc90016b9bb10 EFLAGS: 00000202
[3851451.371053] RAX: 0000000000000003 RBX: 0000000000000100 RCX: 000000000000000d
[3851451.379279] RDX: ffff8820341d9040 RSI: 0000000000000100 RDI: ffff8820337d5048
[3851451.387513] RBP: ffffc90016b9bb48 R08: 0000000000000000 R09: 0000000020003044
[3851451.395759] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8820337d5048
[3851451.403994] R13: ffff8820337d5040 R14: ffffffff8106bb40 R15: ffffc90016b9bb60
[3851451.412241] FS: 00006e793effd700(0000) GS:ffff882033600000(0000) knlGS:0000000000000000
[3851451.421551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851451.428231] CR2: 00006e793effcbb0 CR3: 000000201d2fe000 CR4: 00000000003626f0
[3851451.436464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851451.444697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851451.452938] Stack:
[3851451.455407] 00000000001d5000 01ffc90000000001 ffff88201cf10a80 ffff88201cf10e10
[3851451.463978] ffff88201cf10e10 ffffffff81f25060 ffff881fe3241fe0 ffffc90016b9bb98
[3851451.472577] ffffffff8106be6b ffffffff811bd3c7 ffff88201cf10a80 00006e793effc000
[3851451.481170] Call Trace:
[3851451.484132] [<ffffffff8106be6b>] native_flush_tlb_others+0x7b/0x260
[3851451.491484] [<ffffffff811bd3c7>] ? __page_check_address+0xf7/0x200
[3851451.498720] [<ffffffff8106c51f>] flush_tlb_page+0x6f/0xb0
[3851451.505083] [<ffffffff811c370e>] ptep_clear_flush+0x3e/0x70
[3851451.511640] [<ffffffff811bdb9c>] try_to_unmap_one+0xec/0x4a0
[3851451.518313] [<ffffffff811bed40>] rmap_walk+0x220/0x330
[3851451.524383] [<ffffffff811bf047>] try_to_unmap+0x57/0x80
[3851451.530564] [<ffffffff811bdab0>] ? page_remove_rmap+0x110/0x110
[3851451.537510] [<ffffffff811bcb70>] ? invalid_migration_vma+0x30/0x30
[3851451.544746] [<ffffffff811be060>] ? put_anon_vma+0x20/0x20
[3851451.551108] [<ffffffff811bcb40>] ? invalid_mkclean_vma+0x20/0x20
[3851451.558160] [<ffffffff811e3ffd>] migrate_pages+0x44d/0x970
[3851451.564618] [<ffffffff811e1e40>] ? new_page_node+0xb0/0xb0
[3851451.571088] [<ffffffff813d7527>] ? __pax_list_add+0x17/0x40
[3851451.577662] [<ffffffff811e4d90>] migrate_misplaced_page+0xb0/0x110
[3851451.584917] [<ffffffff811b2d93>] handle_mm_fault+0xfb3/0x1e10
[3851451.591685] [<ffffffff811b1e28>] ? handle_mm_fault+0x48/0x1e10
[3851451.598535] [<ffffffff810a85aa>] ? ___might_sleep+0x1ea/0x220
[3851451.605287] [<ffffffff8106405f>] __do_page_fault+0x1bf/0x460
[3851451.611951] [<ffffffff8110ca44>] ? sys_futex+0x74/0x230
[3851451.618118] [<ffffffff8106432c>] do_page_fault+0x2c/0x40
[3851451.624375] [<ffffffff817128b8>] page_fault+0x28/0x30
[3851451.630357] Code: fe 2b 00 3b 05 2b a8 e1 00 89 c1 0f 8d 6c fe ff ff 48 98 49 8b 55 00 48 03 14 c5 60 de f1 81 8b 42 18 a8 01 74 ca f3 90 8b 42 18 <a8> 01 75 f7 eb bf 0f b6 4d d0 4c 89 fa 4c 89 f6 44 89 ef e8 ad
[3851478.240169] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [pocviexg:2136]
[3851478.248756] Modules linked in: vhost_net tun vhost macvtap 8021q garp mrp drbg ansi_cprng dm_crypt algif_skcipher af_alg ip_gre ip_tunnel gre macvlan ebtable_filter ebtables openvswitch nf_defrag_ipv6 libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul raid10 crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac lpc_ich mfd_core mei_me mei i2c_i801 ioatdma ipmi_ssif ipmi_msghandler acpi_pad acpi_cpufreq squashfs lz4_decompress xhci_pci xhci_hcd ixgbe mdio vxlan ip6_udp_tunnel udp_tunnel ptp pps_core dca ehci_pci ehci_hcd usbcore ahci libahci usb_common tpm_tis
[3851478.338846] irq event stamp: 2821077692
[3851478.343360] hardirqs last enabled at (2821077691): [<ffffffff81711261>] ffffffff81711261
[3851478.352170] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [CPU 0/KVM:9662]
[3851478.352216] Modules linked in: vhost_net tun vhost macvtap 8021q garp mrp drbg ansi_cprng dm_crypt algif_skcipher af_alg ip_gre ip_tunnel gre macvlan ebtable_filter ebtables openvswitch nf_defrag_ipv6 libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul raid10 crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac lpc_ich mfd_core mei_me mei i2c_i801 ioatdma ipmi_ssif ipmi_msghandler acpi_pad acpi_cpufreq squashfs lz4_decompress xhci_pci xhci_hcd ixgbe mdio vxlan ip6_udp_tunnel udp_tunnel ptp pps_core dca ehci_pci ehci_hcd usbcore ahci libahci usb_common tpm_tis
[3851478.352220] irq event stamp: 3489582042
[3851478.352221] hardirqs last enabled at (3489582041): [<ffffffff81711261>] ffffffff81711261
[3851478.352222] hardirqs last disabled at (3489582042): [<ffffffff81711489>] ffffffff81711489
[3851478.352223] softirqs last enabled at (3489582040): [<ffffffff8107f31e>] ffffffff8107f31e
[3851478.352224] softirqs last disabled at (3489582027): [<ffffffff8107f648>] ffffffff8107f648
[3851478.352227] CPU: 12 PID: 9662 Comm: CPU 0/KVM Tainted: G L 4.4.52-grsec #1
[3851478.352227] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851478.352228] task: ffff881c996d8000 ti: ffff881c996d96b0 task.ti: ffff881c996d96b0
[3851478.352236] RIP: 0010:[<ffffffff8110d717>] [<ffffffff8110d717>] smp_call_function_single+0xb7/0x120
[3851478.352236] RSP: 0018:ffffc90012543a70 EFLAGS: 00000202
[3851478.352237] RAX: 0000000000000000 RBX: 000000000000000d RCX: ffff8820341d9040
[3851478.352238] RDX: 0000000000000001 RSI: ffffc90012543a80 RDI: ffffc90012543a80
[3851478.352238] RBP: ffffc90012543ab0 R08: 0000000000000001 R09: 0000000000000000
[3851478.352239] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0bc6660
[3851478.352240] R13: 000000000000000c R14: 000000000000000c R15: 0000000000000000
[3851478.352241] FS: 00006b7b59ee9700(0000) GS:ffff882033e00000(0000) knlGS:0000000000000000
[3851478.352242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851478.352242] CR2: 00007f7b8743f000 CR3: 0000001c9bca5000 CR4: 00000000003626f0
[3851478.352243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851478.352244] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851478.352244] Stack:
[3851478.352246] ffffc90012543b40 0000000000000001 ffff8820341d9040 ffffffffa0bc6660
[3851478.352248] ffff881bb6b1ae10 0000000000000003 ffff881bb6b18000 000000000000000c
[3851478.352249] ffffc90012543b90 ffffffffa0bcdaa4 ffff881c996d88a8 00000002996d8000
[3851478.352250] Call Trace:
[3851478.352258] [<ffffffffa0bc6660>] ? handle_pause+0x1c0/0x1c0 [kvm_intel]
[3851478.352262] [<ffffffffa0bcdaa4>] vmx_vcpu_load+0x1e4/0x2d0 [kvm_intel]
[3851478.352268] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851478.352270] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851478.352292] [<ffffffffa0b314e2>] kvm_arch_vcpu_load+0x32/0x2a0 [kvm]
[3851478.352306] [<ffffffffa0b18714>] kvm_sched_in+0x34/0x40 [kvm]
[3851478.352310] [<ffffffff810aad78>] finish_task_switch+0x188/0x250
[3851478.352312] [<ffffffff810aac57>] ? finish_task_switch+0x67/0x250
[3851478.352314] [<ffffffff81709253>] __schedule+0x423/0xbd0
[3851478.352315] [<ffffffff81709a37>] schedule+0x37/0x80
[3851478.352328] [<ffffffffa0b1c69e>] kvm_vcpu_block+0x6e/0x4b0 [kvm]
[3851478.352331] [<ffffffff810c8880>] ? wake_up_atomic_t+0x30/0x30
[3851478.352346] [<ffffffffa0b375d3>] kvm_arch_vcpu_ioctl_run+0x533/0x19c0 [kvm]
[3851478.352360] [<ffffffffa0b37a97>] ? kvm_arch_vcpu_ioctl_run+0x9f7/0x19c0 [kvm]
[3851478.352373] [<ffffffffa0b17c8d>] ? vcpu_load+0x1d/0x60 [kvm]
[3851478.352385] [<ffffffffa0b192b6>] kvm_vcpu_ioctl+0x2d6/0x620 [kvm]
[3851478.352387] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851478.352389] [<ffffffff810b4f4c>] ? local_clock+0x1c/0x20
[3851478.352394] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851478.352396] [<ffffffff81214aec>] do_vfs_ioctl+0x44c/0x820
[3851478.352397] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851478.352399] [<ffffffff81214f34>] sys_ioctl+0x74/0x80
[3851478.352401] [<ffffffff817106fa>] entry_SYSCALL_64_fastpath+0x16/0x7e
[3851478.352420] Code: 65 ff 0d dd d7 ef 7e 48 83 c4 30 5b 41 5c 5d c3 48 8d 75 d0 48 89 d1 89 df 4c 89 e2 e8 a3 fd ff ff 8b 55 e8 83 e2 01 74 d7 f3 90 <8b> 55 e8 83 e2 01 75 f6 eb cb 48 83 3d d7 18 e1 00 00 74 52 9c
[3851478.408212] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [CPU 1/KVM:10302]
[3851478.408239] Modules linked in: vhost_net tun vhost macvtap 8021q garp mrp drbg ansi_cprng dm_crypt algif_skcipher af_alg ip_gre ip_tunnel gre macvlan ebtable_filter ebtables openvswitch nf_defrag_ipv6 libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul raid10 crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac lpc_ich mfd_core mei_me mei i2c_i801 ioatdma ipmi_ssif ipmi_msghandler acpi_pad acpi_cpufreq squashfs lz4_decompress xhci_pci xhci_hcd ixgbe mdio vxlan ip6_udp_tunnel udp_tunnel ptp pps_core dca ehci_pci ehci_hcd usbcore ahci libahci usb_common tpm_tis
[3851478.408243] irq event stamp: 3177446878
[3851478.408245] hardirqs last enabled at (3177446877): [<ffffffff81711261>] ffffffff81711261
[3851478.408246] hardirqs last disabled at (3177446878): [<ffffffff81711489>] ffffffff81711489
[3851478.408247] softirqs last enabled at (3177446876): [<ffffffff8107f31e>] ffffffff8107f31e
[3851478.408247] softirqs last disabled at (3177446863): [<ffffffff8107f648>] ffffffff8107f648
[3851478.408250] CPU: 14 PID: 10302 Comm: CPU 1/KVM Tainted: G L 4.4.52-grsec #1
[3851478.408250] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851478.408251] task: ffff881ef2ae0000 ti: ffff881ef2ae16b0 task.ti: ffff881ef2ae16b0
[3851478.408255] RIP: 0010:[<ffffffff8110d717>] [<ffffffff8110d717>] smp_call_function_single+0xb7/0x120
[3851478.408255] RSP: 0018:ffffc90014b2ba70 EFLAGS: 00000202
[3851478.408256] RAX: 0000000000000000 RBX: 000000000000000d RCX: ffffc90012543a80
[3851478.408257] RDX: 0000000000000001 RSI: ffffc90014b2ba80 RDI: ffffc90014b2ba80
[3851478.408257] RBP: ffffc90014b2bab0 R08: 0000000000000001 R09: 0000000000000000
[3851478.408258] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0bc6660
[3851478.408258] R13: 000000000000000e R14: 000000000000000e R15: 0000000000000000
[3851478.408260] FS: 00007847e8e5d700(0000) GS:ffff882034200000(0000) knlGS:0000000000000000
[3851478.408261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851478.408261] CR2: 00007f96a99dc000 CR3: 00000018bb771000 CR4: 00000000003626f0
[3851478.408262] Stack:
[3851478.408264] ffffc90014b2bb40 0000000000000001 ffffc90012543a80 ffffffffa0bc6660
[3851478.408265] ffff8818b5ca2e10 0000000000000003 ffff8818b5ca0000 000000000000000e
[3851478.408267] ffffc90014b2bb90 ffffffffa0bcdaa4 ffff881ef2ae08a8 00000002f2ae0000
[3851478.408267] Call Trace:
[3851478.408273] [<ffffffffa0bc6660>] ? handle_pause+0x1c0/0x1c0 [kvm_intel]
[3851478.408277] [<ffffffffa0bcdaa4>] vmx_vcpu_load+0x1e4/0x2d0 [kvm_intel]
[3851478.408279] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851478.408281] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851478.408299] [<ffffffffa0b314e2>] kvm_arch_vcpu_load+0x32/0x2a0 [kvm]
[3851478.408313] [<ffffffffa0b18714>] kvm_sched_in+0x34/0x40 [kvm]
[3851478.408315] [<ffffffff810aad78>] finish_task_switch+0x188/0x250
[3851478.408317] [<ffffffff810aac57>] ? finish_task_switch+0x67/0x250
[3851478.408318] [<ffffffff81709253>] __schedule+0x423/0xbd0
[3851478.408320] [<ffffffff81709a37>] schedule+0x37/0x80
[3851478.408333] [<ffffffffa0b1c69e>] kvm_vcpu_block+0x6e/0x4b0 [kvm]
[3851478.408334] [<ffffffff810c8880>] ? wake_up_atomic_t+0x30/0x30
[3851478.408349] [<ffffffffa0b375d3>] kvm_arch_vcpu_ioctl_run+0x533/0x19c0 [kvm]
[3851478.408364] [<ffffffffa0b37a97>] ? kvm_arch_vcpu_ioctl_run+0x9f7/0x19c0 [kvm]
[3851478.408377] [<ffffffffa0b17c8d>] ? vcpu_load+0x1d/0x60 [kvm]
[3851478.408389] [<ffffffffa0b192b6>] kvm_vcpu_ioctl+0x2d6/0x620 [kvm]
[3851478.408391] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851478.408392] [<ffffffff810b4f4c>] ? local_clock+0x1c/0x20
[3851478.408394] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851478.408395] [<ffffffff81214aec>] do_vfs_ioctl+0x44c/0x820
[3851478.408397] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851478.408398] [<ffffffff81214f34>] sys_ioctl+0x74/0x80
[3851478.408400] [<ffffffff817106fa>] entry_SYSCALL_64_fastpath+0x16/0x7e
[3851478.408420] Code: 65 ff 0d dd d7 ef 7e 48 83 c4 30 5b 41 5c 5d c3 48 8d 75 d0 48 89 d1 89 df 4c 89 e2 e8 a3 fd ff ff 8b 55 e8 83 e2 01 74 d7 f3 90 <8b> 55 e8 83 e2 01 75 f6 eb cb 48 83 3d d7 18 e1 00 00 74 52 9c
[3851479.298452] hardirqs last disabled at (2821077692): [<ffffffff81711489>] ffffffff81711489
[3851479.307864] softirqs last enabled at (2821077690): [<ffffffff8107f31e>] ffffffff8107f31e
[3851479.317274] softirqs last disabled at (2821077675): [<ffffffff8107f648>] ffffffff8107f648
[3851479.326687] CPU: 8 PID: 2136 Comm: pocviexg Tainted: G L 4.4.52-grsec #1
[3851479.336089] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851479.348327] task: ffff881c9ff34500 ti: ffff881c9ff35bb0 task.ti: ffff881c9ff35bb0
[3851479.356949] RIP: 0010:[<ffffffff8110da96>] [<ffffffff8110da96>] smp_call_function_many+0x216/0x280
[3851479.367334] RSP: 0000:ffffc90016b9bb10 EFLAGS: 00000202
[3851479.373500] RAX: 0000000000000003 RBX: 0000000000000100 RCX: 000000000000000d
[3851479.381735] RDX: ffff8820341d9040 RSI: 0000000000000100 RDI: ffff8820337d5048
[3851479.389978] RBP: ffffc90016b9bb48 R08: 0000000000000000 R09: 0000000020003044
[3851479.398238] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8820337d5048
[3851479.406484] R13: ffff8820337d5040 R14: ffffffff8106bb40 R15: ffffc90016b9bb60
[3851479.414718] FS: 00006e793effd700(0000) GS:ffff882033600000(0000) knlGS:0000000000000000
[3851479.424021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851479.430683] CR2: 00006e793effcbb0 CR3: 000000201d2fe000 CR4: 00000000003626f0
[3851479.438917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851479.447168] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851479.455403] Stack:
[3851479.457873] 00000000001d5000 01ffc90000000001 ffff88201cf10a80 ffff88201cf10e10
[3851479.466464] ffff88201cf10e10 ffffffff81f25060 ffff881fe3241fe0 ffffc90016b9bb98
[3851479.475026] ffffffff8106be6b ffffffff811bd3c7 ffff88201cf10a80 00006e793effc000
[3851479.483653] Call Trace:
[3851479.486613] [<ffffffff8106be6b>] native_flush_tlb_others+0x7b/0x260
[3851479.493969] [<ffffffff811bd3c7>] ? __page_check_address+0xf7/0x200
[3851479.501206] [<ffffffff8106c51f>] flush_tlb_page+0x6f/0xb0
[3851479.507567] [<ffffffff811c370e>] ptep_clear_flush+0x3e/0x70
[3851479.514123] [<ffffffff811bdb9c>] try_to_unmap_one+0xec/0x4a0
[3851479.520776] [<ffffffff811bed40>] rmap_walk+0x220/0x330
[3851479.526860] [<ffffffff811bf047>] try_to_unmap+0x57/0x80
[3851479.533027] [<ffffffff811bdab0>] ? page_remove_rmap+0x110/0x110
[3851479.539972] [<ffffffff811bcb70>] ? invalid_migration_vma+0x30/0x30
[3851479.547209] [<ffffffff811be060>] ? put_anon_vma+0x20/0x20
[3851479.553569] [<ffffffff811bcb40>] ? invalid_mkclean_vma+0x20/0x20
[3851479.560618] [<ffffffff811e3ffd>] migrate_pages+0x44d/0x970
[3851479.571475] [<ffffffff811e1e40>] ? new_page_node+0xb0/0xb0
[3851479.577944] [<ffffffff813d7527>] ? __pax_list_add+0x17/0x40
[3851479.584517] [<ffffffff811e4d90>] migrate_misplaced_page+0xb0/0x110
[3851479.591800] [<ffffffff811b2d93>] handle_mm_fault+0xfb3/0x1e10
[3851479.598551] [<ffffffff811b1e28>] ? handle_mm_fault+0x48/0x1e10
[3851479.605428] [<ffffffff810a85aa>] ? ___might_sleep+0x1ea/0x220
[3851479.612208] [<ffffffff8106405f>] __do_page_fault+0x1bf/0x460
[3851479.618862] [<ffffffff8110ca44>] ? sys_futex+0x74/0x230
[3851479.625029] [<ffffffff8106432c>] do_page_fault+0x2c/0x40
[3851479.631295] [<ffffffff817128b8>] page_fault+0x28/0x30
[3851479.637278] Code: 48 63 d2 e8 4d fe 2b 00 3b 05 2b a8 e1 00 89 c1 0f 8d 6c fe ff ff 48 98 49 8b 55 00 48 03 14 c5 60 de f1 81 8b 42 18 a8 01 74 ca <f3> 90 8b 42 18 a8 01 75 f7 eb bf 0f b6 4d d0 4c 89 fa 4c 89 f6
[3851482.648386] BUG: spinlock lockup suspected on CPU#0, migration/0/10
[3851482.655642] lock: ffff8820341ce608, .magic: dead4ead, .owner: pocviexg/2146, .owner_cpu: 13
[3851482.665239] CPU: 0 PID: 10 Comm: migration/0 Tainted: G L 4.4.52-grsec #1
[3851482.674858] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851482.687078] 0000000000000000 ffffc900006ebb30 ffffffff813b6d39 000000000000004f
[3851482.695652] ffff881c9ff35c00 ffff8820341ce608 ffffc900006ebb50 ffffffff810db973
[3851482.704219] ffff8820341ce608 000000007d234b28 ffffc900006ebb80 ffffffff810dba9b
[3851482.712790] Call Trace:
[3851482.715764] [<ffffffff813b6d39>] dump_stack+0x9a/0xe1
[3851482.721768] [<ffffffff810db973>] spin_dump+0x73/0xc0
[3851482.727664] [<ffffffff810dba9b>] do_raw_spin_lock+0x9b/0x160
[3851482.734337] [<ffffffff817101a2>] _raw_spin_lock_irqsave+0xa2/0xb0
[3851482.741476] [<ffffffff8112c1cb>] ? cpu_stop_queue_work+0x2b/0x70
[3851482.748518] [<ffffffff8112c1cb>] cpu_stop_queue_work+0x2b/0x70
[3851482.755371] [<ffffffff8112c92b>] stop_one_cpu_nowait+0x2b/0x30
[3851482.762242] [<ffffffff810c1127>] load_balance+0x827/0x960
[3851482.768605] [<ffffffff810eae3d>] ? debug_lockdep_rcu_enabled+0x1d/0x20
[3851482.776232] [<ffffffff810c1608>] pick_next_task_fair+0x3a8/0x750
[3851482.783272] [<ffffffff810c14db>] ? pick_next_task_fair+0x27b/0x750
[3851482.790498] [<ffffffff81708fdd>] __schedule+0x1ad/0xbd0
[3851482.796701] [<ffffffff810d461d>] ? trace_hardirqs_on_caller+0x13d/0x1d0
[3851482.804424] [<ffffffff81709a37>] schedule+0x37/0x80
[3851482.810211] [<ffffffff810a51cb>] smpboot_thread_fn+0x16b/0x1e0
[3851482.817071] [<ffffffff810a5060>] ? sort_range+0x20/0x20
[3851482.823238] [<ffffffff810a124c>] kthread+0xfc/0x120
[3851482.829006] [<ffffffff8170adf1>] ? wait_for_completion+0xe1/0x110
[3851482.836146] [<ffffffff810a1150>] ? kthread_create_on_node+0x240/0x240
[3851482.843673] [<ffffffff81710aae>] ret_from_fork+0x3e/0x70
[3851482.849936] [<ffffffff810a1150>] ? kthread_create_on_node+0x240/0x240
[3851482.857498] Sending NMI to all CPUs:
[3851482.862842] NMI backtrace for cpu 0
[3851482.866958] CPU: 0 PID: 10 Comm: migration/0 Tainted: G L 4.4.52-grsec #1
[3851482.876554] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851482.888796] task: ffff88102cc8dc00 ti: ffff88102cc8f2b0 task.ti: ffff88102cc8f2b0
[3851482.897416] RIP: 0010:[<ffffffff813cd878>] [<ffffffff813cd878>] _find_next_bit.part.0+0x28/0x70
[3851482.907505] RSP: 0000:ffffc900006eba48 EFLAGS: 00000046
[3851482.913667] RAX: ffffffffffffffff RBX: ffff881033e0c900 RCX: ffffffffffffffff
[3851482.921906] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffff881033e0c900
[3851482.930147] RBP: ffffc900006eba48 R08: 0000000000000000 R09: 00000000ff00ff00
[3851482.938381] R10: 0000000000000004 R11: 000000000000d59c R12: ffff88203360c920
[3851482.946613] R13: 000000000000c940 R14: ffff88203360c920 R15: 0000000000080000
[3851482.954846] FS: 0000000000000000(0000) GS:ffff881033e00000(0000) knlGS:0000000000000000
[3851482.964148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851482.970820] CR2: 000000c420148008 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851482.979059] Stack:
[3851482.981544] ffffc900006eba58 ffffffff813cd8d5 ffffc900006eba78 ffffffff813b6aef
[3851482.990075] 0000000000000000 ffff881033e0c900 ffffc900006ebad8 ffffffff8105a3f0
[3851482.998608] 0000000000000096 00000002810e1dfa 000000000000c920 0000000800000000
[3851483.007132] Call Trace:
[3851483.010096] [<ffffffff813cd8d5>] find_next_bit+0x15/0x20
[3851483.016393] [<ffffffff813b6aef>] cpumask_next_and+0x2f/0x40
[3851483.022962] [<ffffffff8105a3f0>] __x2apic_send_IPI_mask+0xe0/0x1e0
[3851483.030197] [<ffffffff810569f0>] ? irq_force_complete_move+0x120/0x120
[3851483.037829] [<ffffffff8105a54e>] x2apic_send_IPI_mask+0xe/0x10
[3851483.044676] [<ffffffff81056a06>] nmi_raise_cpu_backtrace+0x16/0x20
[3851483.051929] [<ffffffff813bb9bf>] nmi_trigger_all_cpu_backtrace+0x27f/0x290
[3851483.059942] [<ffffffff813b6d60>] ? dump_stack+0xc1/0xe1
[3851483.066108] [<ffffffff81056a64>] arch_trigger_all_cpu_backtrace+0x14/0x20
[3851483.074024] [<ffffffff810dbaa5>] do_raw_spin_lock+0xa5/0x160
[3851483.080675] [<ffffffff817101a2>] _raw_spin_lock_irqsave+0xa2/0xb0
[3851483.087814] [<ffffffff8112c1cb>] ? cpu_stop_queue_work+0x2b/0x70
[3851483.094857] [<ffffffff8112c1cb>] cpu_stop_queue_work+0x2b/0x70
[3851483.101702] [<ffffffff8112c92b>] stop_one_cpu_nowait+0x2b/0x30
[3851483.108549] [<ffffffff810c1127>] load_balance+0x827/0x960
[3851483.114906] [<ffffffff810eae3d>] ? debug_lockdep_rcu_enabled+0x1d/0x20
[3851483.122562] [<ffffffff810c1608>] pick_next_task_fair+0x3a8/0x750
[3851483.129601] [<ffffffff810c14db>] ? pick_next_task_fair+0x27b/0x750
[3851483.136847] [<ffffffff81708fdd>] __schedule+0x1ad/0xbd0
[3851483.143012] [<ffffffff810d461d>] ? trace_hardirqs_on_caller+0x13d/0x1d0
[3851483.150752] [<ffffffff81709a37>] schedule+0x37/0x80
[3851483.156529] [<ffffffff810a51cb>] smpboot_thread_fn+0x16b/0x1e0
[3851483.163389] [<ffffffff810a5060>] ? sort_range+0x20/0x20
[3851483.169561] [<ffffffff810a124c>] kthread+0xfc/0x120
[3851483.175357] [<ffffffff8170adf1>] ? wait_for_completion+0xe1/0x110
[3851483.182494] [<ffffffff810a1150>] ? kthread_create_on_node+0x240/0x240
[3851483.190037] [<ffffffff81710aae>] ret_from_fork+0x3e/0x70
[3851483.196289] [<ffffffff810a1150>] ? kthread_create_on_node+0x240/0x240
[3851483.203825] Code: 00 00 00 48 89 d0 49 89 c9 49 89 c8 48 c1 e8 06 89 d1 55 4c 33 0c c7 48 c7 c0 ff ff ff ff 48 83 e2 c0 48 d3 e0 48 89 e5 48 89 c1 <4c> 21 c9 75 27 48 83 c2 40 48 39 d6 77 0b eb 30 48 83 c2 40 48
[3851483.225227] NMI backtrace for cpu 1
[3851483.229349] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G L 4.4.52-grsec #1
[3851483.238680] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851483.250893] task: ffff88102cca9700 ti: ffff88102ccaadb0 task.ti: ffff88102ccaadb0
[3851483.259514] RIP: 0010:[<ffffffff8141c7d3>] [<ffffffff8141c7d3>] intel_idle+0xa3/0x100
[3851483.268623] RSP: 0000:ffffc900006fbe48 EFLAGS: 00000046
[3851483.274788] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[3851483.283022] RDX: 0000000000000000 RSI: ffffffff81ead9c0 RDI: 0000000001f36000
[3851483.291262] RBP: ffffc900006fbe68 R08: 001cab4395ddf367 R09: 0000000000000000
[3851483.299501] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000010
[3851483.307755] R13: 0000000000000002 R14: 0000000000000003 R15: 000daec481da188e
[3851483.315994] FS: 0000000000000000(0000) GS:ffff881034000000(0000) knlGS:0000000000000000
[3851483.325298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851483.331948] CR2: 000072ec71d43000 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851483.340195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851483.348438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851483.356671] Stack:
[3851483.359142] 0000000000000003 ffffffff81ead9c0 ffff8810341dd000 ffffffff81eadaf8
[3851483.367667] ffffc900006fbeb0 ffffffff81577352 0000000000000002 0000000000000000
[3851483.376191] ffff88102ccaadb0 ffff8810341dd000 ffffffff81f25060 ffff88102ccaadb0
[3851483.384725] Call Trace:
[3851483.387698] [<ffffffff81577352>] cpuidle_enter_state+0xe2/0x3a0
[3851483.394640] [<ffffffff81577632>] cpuidle_enter+0x12/0x20
[3851483.400902] [<ffffffff810c8e42>] cpu_startup_entry+0x2e2/0x470
[3851483.407748] [<ffffffff81051caf>] start_secondary+0x1bf/0x200
[3851483.414411] Code: 00 00 0f ae 38 0f ae f0 31 d2 65 48 8b 04 25 d8 ae 00 00 48 89 d1 0f 01 c8 48 8b 00 a8 08 75 0b b9 01 00 00 00 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 d8 ae 00 00 f0 80 60 02 df 0f ae f0 48 8b 00 a8
[3851483.435802] NMI backtrace for cpu 2
[3851483.439925] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 4.4.52-grsec #1
[3851483.449227] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851483.461448] task: ffff88102ccaae00 ti: ffff88102ccac4b0 task.ti: ffff88102ccac4b0
[3851483.470070] RIP: 0010:[<ffffffff8141c7d3>] [<ffffffff8141c7d3>] intel_idle+0xa3/0x100
[3851483.479213] RSP: 0000:ffffc9000cbf3e48 EFLAGS: 00000046
[3851483.485400] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[3851483.493647] RDX: 0000000000000000 RSI: ffffffff81ead9c0 RDI: 0000000001f36000
[3851483.501914] RBP: ffffc9000cbf3e68 R08: 001cab4395ddf367 R09: 0000000000000000
[3851483.510149] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000020
[3851483.518388] R13: 0000000000000003 R14: 0000000000000004 R15: 000daec481e1cbde
[3851483.526637] FS: 0000000000000000(0000) GS:ffff881034200000(0000) knlGS:0000000000000000
[3851483.535940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851483.546870] CR2: 0000646cd374a000 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851483.555111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851483.563351] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851483.571592] Stack:
[3851483.574062] 0000000000000004 ffffffff81ead9c0 ffff8810343dd000 ffffffff81eadb58
[3851483.582601] ffffc9000cbf3eb0 ffffffff81577352 0000000000000001 0000000000000000
[3851483.591126] ffff88102ccac4b0 ffff8810343dd000 ffffffff81f25060 ffff88102ccac4b0
[3851483.599658] Call Trace:
[3851483.602621] [<ffffffff81577352>] cpuidle_enter_state+0xe2/0x3a0
[3851483.609565] [<ffffffff81577632>] cpuidle_enter+0x12/0x20
[3851483.615828] [<ffffffff810c8e42>] cpu_startup_entry+0x2e2/0x470
[3851483.622672] [<ffffffff81051caf>] start_secondary+0x1bf/0x200
[3851483.629334] Code: 00 00 0f ae 38 0f ae f0 31 d2 65 48 8b 04 25 d8 ae 00 00 48 89 d1 0f 01 c8 48 8b 00 a8 08 75 0b b9 01 00 00 00 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 d8 ae 00 00 f0 80 60 02 df 0f ae f0 48 8b 00 a8
[3851483.650720] NMI backtrace for cpu 3
[3851483.654844] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 4.4.52-grsec #1
[3851483.664146] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851483.676365] task: ffff88102ccac500 ti: ffff88102ccadbb0 task.ti: ffff88102ccadbb0
[3851483.684986] RIP: 0010:[<ffffffff8141c7d3>] [<ffffffff8141c7d3>] intel_idle+0xa3/0x100
[3851483.694107] RSP: 0000:ffffc9000cbfbe48 EFLAGS: 00000046
[3851483.700273] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[3851483.708506] RDX: 0000000000000000 RSI: ffffffff81ead9c0 RDI: 0000000001f36000
[3851483.716770] RBP: ffffc9000cbfbe68 R08: 001cab4395ddf367 R09: 0000000000000000
[3851483.725010] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000020
[3851483.733260] R13: 0000000000000003 R14: 0000000000000004 R15: 000daec481e12ca5
[3851483.741509] FS: 0000000000000000(0000) GS:ffff881034400000(0000) knlGS:0000000000000000
[3851483.750811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851483.757470] CR2: 000072dd155dc000 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851483.765704] Stack:
[3851483.768175] 0000000000000004 ffffffff81ead9c0 ffff8810345dd000 ffffffff81eadb58
[3851483.776710] ffffc9000cbfbeb0 ffffffff81577352 0000000000000000 0000000000000000
[3851483.785234] ffff88102ccadbb0 ffff8810345dd000 ffffffff81f25060 ffff88102ccadbb0
[3851483.793774] Call Trace:
[3851483.796740] [<ffffffff81577352>] cpuidle_enter_state+0xe2/0x3a0
[3851483.803684] [<ffffffff81577632>] cpuidle_enter+0x12/0x20
[3851483.809955] [<ffffffff810c8e42>] cpu_startup_entry+0x2e2/0x470
[3851483.816812] [<ffffffff81051caf>] start_secondary+0x1bf/0x200
[3851483.823468] Code: 00 00 0f ae 38 0f ae f0 31 d2 65 48 8b 04 25 d8 ae 00 00 48 89 d1 0f 01 c8 48 8b 00 a8 08 75 0b b9 01 00 00 00 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 d8 ae 00 00 f0 80 60 02 df 0f ae f0 48 8b 00 a8
[3851483.844891] NMI backtrace for cpu 4
[3851483.849022] CPU: 4 PID: 10406 Comm: CPU 3/KVM Tainted: G L 4.4.52-grsec #1
[3851483.858715] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851483.870940] task: ffff881012931700 ti: ffff881012932db0 task.ti: ffff881012932db0
[3851483.879575] RIP: 0010:[<ffffffffa0bcd413>] [<ffffffffa0bcd413>] vmx_vcpu_run+0x3c3/0x670 [kvm_intel]
[3851483.890151] RSP: 0018:ffffc9001119bcb8 EFLAGS: 00000046
[3851483.896341] RAX: 0000000080000200 RBX: ffff88198e2e8000 RCX: ffff88198e2e8000
[3851483.904574] RDX: 0000000080000202 RSI: ffff88198e2e8000 RDI: ffff88198e2e8000
[3851483.912829] RBP: ffffc9001119bd10 R08: ffffffff8165e760 R09: ffff88023fd97520
[3851483.921063] R10: ffffffff811a2ac2 R11: ffffea0008c84880 R12: 0000000000000000
[3851483.929303] R13: 0000000000000001 R14: ffff88022f4c7d58 R15: 0000000000000000
[3851483.937543] FS: 00007c012ffff700(0000) GS:ffff881034600000(0000) knlGS:0000000000000000
[3851483.946846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851483.953498] CR2: 00007f58ed5fe000 CR3: 0000000d91425000 CR4: 00000000003626f0
[3851483.961723] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851483.969955] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851483.978189] Stack:
[3851483.980658] ffffc9001119bcd8 0000000000000000 ffff88198e2e8000 ffffffff81f25060
[3851483.989182] ffffc90000000000 ffffffff810eaf3f ffff88198e2e8000 ffff880daa89c100
[3851483.997707] 0000000000000000 0000000000000000 ffff880daa89c100 ffffc9001119bdc0
[3851484.006232] Call Trace:
[3851484.009217] [<ffffffff810eaf3f>] ? rcu_read_lock_sched_held+0x6f/0x80
[3851484.016775] [<ffffffffa0b379aa>] kvm_arch_vcpu_ioctl_run+0x90a/0x19c0 [kvm]
[3851484.024910] [<ffffffffa0b37a97>] ? kvm_arch_vcpu_ioctl_run+0x9f7/0x19c0 [kvm]
[3851484.032468] ata8.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[3851484.032470] ata8.00: failed command: FLUSH CACHE EXT
[3851484.032473] ata8.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 7
[3851484.032473] res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[3851484.032474] ata8.00: status: { DRDY }
[3851484.032477] ata8: hard resetting link
[3851484.032500] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[3851484.032501] ata7.00: failed command: FLUSH CACHE EXT
[3851484.032504] ata7.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 6
[3851484.032504] res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[3851484.032505] ata7.00: status: { DRDY }
[3851484.032508] ata7: hard resetting link
[3851484.109672] [<ffffffffa0b17c8d>] ? vcpu_load+0x1d/0x60 [kvm]
[3851484.116322] [<ffffffffa0b192b6>] kvm_vcpu_ioctl+0x2d6/0x620 [kvm]
[3851484.123459] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851484.129729] [<ffffffff810b4f4c>] ? local_clock+0x1c/0x20
[3851484.136000] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851484.142555] [<ffffffff81214aec>] do_vfs_ioctl+0x44c/0x820
[3851484.148929] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851484.155482] [<ffffffff81214f34>] sys_ioctl+0x74/0x80
[3851484.161357] [<ffffffff817106fa>] entry_SYSCALL_64_fastpath+0x16/0x7e
[3851484.168807] Code: 80 89 86 cc 2d 00 00 0f 84 b0 02 00 00 25 00 07 00 80 3d 00 02 00 80 0f 85 27 ff ff ff 48 8b 5d b8 48 89 df e8 ff 73 f5 ff cd 02 <48> 89 df e8 05 74 f5 ff e9 0c ff ff ff 48 8b 45 b8 8b 80 18 31
[3851484.190203] NMI backtrace for cpu 5
[3851484.194334] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G L 4.4.52-grsec #1
[3851484.203642] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851484.215863] task: ffff88202cda0000 ti: ffff88202cda16b0 task.ti: ffff88202cda16b0
[3851484.224502] RIP: 0010:[<ffffffff8141c7d3>] [<ffffffff8141c7d3>] intel_idle+0xa3/0x100
[3851484.233616] RSP: 0000:ffffc9000ccf3e48 EFLAGS: 00000046
[3851484.239794] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[3851484.248028] RDX: 0000000000000000 RSI: ffffffff81ead9c0 RDI: 0000000000000005
[3851484.256260] RBP: ffffc9000ccf3e68 R08: 001cab4395ddf367 R09: 0000000000000000
[3851484.264505] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000020
[3851484.272771] R13: 0000000000000003 R14: 0000000000000004 R15: 000daec481d1a46c
[3851484.281019] FS: 0000000000000000(0000) GS:ffff881034800000(0000) knlGS:0000000000000000
[3851484.290329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851484.296994] CR2: 000077f7564d8000 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851484.305241] Stack:
[3851484.307710] 0000000000000004 ffffffff81ead9c0 ffff8810349dd000 ffffffff81eadb58
[3851484.316234] ffffc9000ccf3eb0 ffffffff81577352 0000000000000001 0000000000000000
[3851484.324776] ffff88202cda16b0 ffff8810349dd000 ffffffff81f25060 ffff88202cda16b0
[3851484.333323] Call Trace:
[3851484.336280] [<ffffffff81577352>] cpuidle_enter_state+0xe2/0x3a0
[3851484.343222] [<ffffffff81577632>] cpuidle_enter+0x12/0x20
[3851484.349484] [<ffffffff810c8e42>] cpu_startup_entry+0x2e2/0x470
[3851484.352477] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[3851484.352516] ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[3851484.370634] [<ffffffff81051caf>] start_secondary+0x1bf/0x200
[3851484.377285] Code: 00 00 0f ae 38 0f ae f0 31 d2 65 48 8b 04 25 d8 ae 00 00 48 89 d1 0f 01 c8 48 8b 00 a8 08 75 0b b9 01 00 00 00 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 d8 ae 00 00 f0 80 60 02 df 0f ae f0 48 8b 00 a8
[3851484.398689] NMI backtrace for cpu 6
[3851484.402812] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G L 4.4.52-grsec #1
[3851484.412127] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851484.424367] task: ffff88202cda1700 ti: ffff88202cda2db0 task.ti: ffff88202cda2db0
[3851484.437361] RIP: 0010:[<ffffffff8141c7d3>] [<ffffffff8141c7d3>] intel_idle+0xa3/0x100
[3851484.446477] RSP: 0000:ffffc9000ccfbe48 EFLAGS: 00000046
[3851484.452642] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[3851484.460876] RDX: 0000000000000000 RSI: ffffffff81ead9c0 RDI: 0000000000000006
[3851484.469110] RBP: ffffc9000ccfbe68 R08: 001cab4395ddf367 R09: 0000000000000000
[3851484.477343] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000020
[3851484.485594] R13: 0000000000000003 R14: 0000000000000004 R15: 000daec481d19ed7
[3851484.493834] FS: 0000000000000000(0000) GS:ffff881034a00000(0000) knlGS:0000000000000000
[3851484.503176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851484.509827] CR2: 000072ec71d43000 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851484.518075] Stack:
[3851484.520545] 0000000000000004 ffffffff81ead9c0 ffff881034bdd000 ffffffff81eadb58
[3851484.529077] ffffc9000ccfbeb0 ffffffff81577352 0000000000000000 0000000000000000
[3851484.537607] ffff88202cda2db0 ffff881034bdd000 ffffffff81f25060 ffff88202cda2db0
[3851484.546163] Call Trace:
[3851484.549119] [<ffffffff81577352>] cpuidle_enter_state+0xe2/0x3a0
[3851484.556060] [<ffffffff81577632>] cpuidle_enter+0x12/0x20
[3851484.562328] [<ffffffff810c8e42>] cpu_startup_entry+0x2e2/0x470
[3851484.569173] [<ffffffff81051caf>] start_secondary+0x1bf/0x200
[3851484.575825] Code: 00 00 0f ae 38 0f ae f0 31 d2 65 48 8b 04 25 d8 ae 00 00 48 89 d1 0f 01 c8 48 8b 00 a8 08 75 0b b9 01 00 00 00 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 d8 ae 00 00 f0 80 60 02 df 0f ae f0 48 8b 00 a8
...
[3851489.162678] NMI backtrace for cpu 24
[3851489.166914] CPU: 24 PID: 0 Comm: swapper/24 Tainted: G L 4.4.52-grsec #1
[3851489.176426] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851489.188655] task: ffff88202cdcdc00 ti: ffff88202cdcf2b0 task.ti: ffff88202cdcf2b0
[3851489.197276] RIP: 0010:[<ffffffff8141c7d3>] [<ffffffff8141c7d3>] intel_idle+0xa3/0x100
[3851489.206384] RSP: 0000:ffffc9000cd8be48 EFLAGS: 00000046
[3851489.212550] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[3851489.220782] RDX: 0000000000000000 RSI: ffffffff81ead9c0 RDI: 0000000000000018
[3851489.229045] RBP: ffffc9000cd8be68 R08: 001cab4395ddf367 R09: 0000000000000000
[3851489.237277] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000020
[3851489.245524] R13: 0000000000000003 R14: 0000000000000004 R15: 000daec481d17b31
[3851489.253756] FS: 0000000000000000(0000) GS:ffff882034600000(0000) knlGS:0000000000000000
[3851489.263059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851489.269710] CR2: 000064bc226bc000 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851489.277950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851489.286183] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851489.294416] Stack:
[3851489.296886] 0000000000000004 ffffffff81ead9c0 ffff8820347dd000 ffffffff81eadb58
[3851489.305431] ffffc9000cd8beb0 ffffffff81577352 0000000000000001 0000000000000000
[3851489.313956] ffff88202cdcf2b0 ffff8820347dd000 ffffffff81f25060 ffff88202cdcf2b0
[3851489.322481] Call Trace:
[3851489.325437] [<ffffffff81577352>] cpuidle_enter_state+0xe2/0x3a0
[3851489.332380] [<ffffffff81577632>] cpuidle_enter+0x12/0x20
[3851489.338643] [<ffffffff810c8e42>] cpu_startup_entry+0x2e2/0x470
[3851489.345488] [<ffffffff81051caf>] start_secondary+0x1bf/0x200
[3851489.352149] Code: 00 00 0f ae 38 0f ae f0 31 d2 65 48 8b 04 25 d8 ae 00 00 48 89 d1 0f 01 c8 48 8b 00 a8 08 75 0b b9 01 00 00 00 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 d8 ae 00 00 f0 80 60 02 df 0f ae f0 48 8b 00 a8
[3851489.352686] ata8.00: qc timeout (cmd 0xec)
[3851489.352693] ata8.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[3851489.352694] ata8.00: revalidation failed (errno=-5)
[3851489.352697] ata8: hard resetting link
[3851489.352712] ata7.00: qc timeout (cmd 0xec)
[3851489.352719] ata7.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[3851489.352720] ata7.00: revalidation failed (errno=-5)
[3851489.352721] ata7: hard resetting link
[3851489.417293] NMI backtrace for cpu 25
[3851489.421527] CPU: 25 PID: 9363 Comm: CPU 1/KVM Tainted: G L 4.4.52-grsec #1
[3851489.431217] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851489.443451] task: ffff881fe3114500 ti: ffff881fe3115bb0 task.ti: ffff881fe3115bb0
[3851489.452089] RIP: 0010:[<ffffffffa0bcd413>] [<ffffffffa0bcd413>] vmx_vcpu_run+0x3c3/0x670 [kvm_intel]
[3851489.462655] RSP: 0018:ffffc900171b3cb8 EFLAGS: 00000046
[3851489.468821] RAX: 0000000080000200 RBX: ffff881bc03a8000 RCX: ffff881bc03a8000
[3851489.477054] RDX: 0000000080000202 RSI: ffff881bc03a8000 RDI: ffff881bc03a8000
[3851489.485288] RBP: ffffc900171b3d10 R08: 0000000000000040 R09: 0000000000000000
[3851489.493527] R10: ffff8804153a6dd0 R11: 0000000000000293 R12: ffff8803fcf3dd70
[3851489.501760] R13: ffff8803fcf3dd00 R14: ffff880411292080 R15: 0000000000000005
[3851489.510007] FS: 000069f8759ad700(0000) GS:ffff882034800000(0000) knlGS:0000000000000000
[3851489.519301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851489.525969] CR2: 00007f96b8e46000 CR3: 0000001bd1462000 CR4: 00000000003626f0
[3851489.534218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851489.542456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851489.550689] Stack:
[3851489.553159] ffffc900171b3cd8 0000000000000000 ffff881bc03a8000 ffffffff81f25060
[3851489.561683] ffffc90000000000 ffffffff810eaf3f ffff881bc03a8000 ffff881c9fdd8100
[3851489.570209] 0000000000000000 0000000000000000 ffff881c9fdd8100 ffffc900171b3dc0
[3851489.578733] Call Trace:
[3851489.581693] [<ffffffff810eaf3f>] ? rcu_read_lock_sched_held+0x6f/0x80
[3851489.589221] [<ffffffffa0b379aa>] kvm_arch_vcpu_ioctl_run+0x90a/0x19c0 [kvm]
[3851489.597364] [<ffffffffa0b37a97>] ? kvm_arch_vcpu_ioctl_run+0x9f7/0x19c0 [kvm]
[3851489.605695] [<ffffffffa0b17c8d>] ? vcpu_load+0x1d/0x60 [kvm]
[3851489.612347] [<ffffffffa0b192b6>] kvm_vcpu_ioctl+0x2d6/0x620 [kvm]
[3851489.619483] [<ffffffff81026153>] ? sched_clock+0x13/0x20
[3851489.625767] [<ffffffff810b4f4c>] ? local_clock+0x1c/0x20
[3851489.632038] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851489.638592] [<ffffffff81214aec>] do_vfs_ioctl+0x44c/0x820
[3851489.645013] [<ffffffff81221940>] ? expand_files+0x2c0/0x2c0
[3851489.651568] [<ffffffff81214f34>] sys_ioctl+0x74/0x80
[3851489.657442] [<ffffffff817106fa>] entry_SYSCALL_64_fastpath+0x16/0x7e
[3851489.664872] Code: 80 89 86 cc 2d 00 00 0f 84 b0 02 00 00 25 00 07 00 80 3d 00 02 00 80 0f 85 27 ff ff ff 48 8b 5d b8 48 89 df e8 ff 73 f5 ff cd 02 <48> 89 df e8 05 74 f5 ff e9 0c ff ff ff 48 8b 45 b8 8b 80 18 31
[3851489.672702] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[3851489.672736] ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[3851489.700546] NMI backtrace for cpu 26
[3851489.704759] CPU: 26 PID: 0 Comm: swapper/26 Tainted: G L 4.4.52-grsec #1
[3851489.714257] Hardware name: Intel S2600WTTR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
[3851489.726477] task: ffff88202cdd1700 ti: ffff88202cdd2db0 task.ti: ffff88202cdd2db0
[3851489.735099] RIP: 0010:[<ffffffff8141c7d3>] [<ffffffff8141c7d3>] intel_idle+0xa3/0x100
[3851489.744214] RSP: 0000:ffffc9000cd9be48 EFLAGS: 00000046
[3851489.750398] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[3851489.758630] RDX: 0000000000000000 RSI: ffffffff81ead9c0 RDI: 0000000001f36000
[3851489.766879] RBP: ffffc9000cd9be68 R08: 001cab4395ddf367 R09: 0000000000000000
[3851489.775119] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000020
[3851489.783358] R13: 0000000000000003 R14: 0000000000000004 R15: 000daec481e3287a
[3851489.791606] FS: 0000000000000000(0000) GS:ffff882034a00000(0000) knlGS:0000000000000000
[3851489.800910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3851489.807562] CR2: 00007fdd04300020 CR3: 0000000001f36000 CR4: 00000000003626f0
[3851489.815795] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3851489.824039] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3851489.832273] Stack:
[3851489.834743] 0000000000000004 ffffffff81ead9c0 ffff882034bdd000 ffffffff81eadb58
[3851489.843285] ffffc9000cd9beb0 ffffffff81577352 0000000000000000 0000000000000000
[3851489.851810] ffff88202cdd2db0 ffff882034bdd000 ffffffff81f25060 ffff88202cdd2db0
[3851489.860334] Call Trace:
[3851489.863299] [<ffffffff81577352>] cpuidle_enter_state+0xe2/0x3a0
[3851489.870243] [<ffffffff81577632>] cpuidle_enter+0x12/0x20
[3851489.876505] [<ffffffff810c8e42>] cpu_startup_entry+0x2e2/0x470
[3851489.883391] [<ffffffff81051caf>] start_secondary+0x1bf/0x200
[3851489.890055] Code: 00 00 0f ae 38 0f ae f0 31 d2 65 48 8b 04 25 d8 ae 00 00 48 89 d1 0f 01 c8 48 8b 00 a8 08 75 0b b9 01 00 00 00 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 d8 ae 00 00 f0 80 60 02 df 0f ae f0 48 8b 00 a8
...
Powered by blists - more mailing lists