| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 16 Feb 2016 19:43:34 +0100 From: Grzegorz Nosek <root@...aldomain.pl> To: linux-kernel@...r.kernel.org Subject: Panic in nf_ct_seq_adjust (be2net, synproxy, heavy load) Hi, I have finally managed to capture a stack trace from an intermittent crash we've been observing under heavy load. There are two identical machines w/dual-port Emulex 10 Gbps NICs running in an active-passive firewall cluster (though more like passive-passive in reality). As far as I know, at the time of the reboot there was a pretty large DoS attack (6 Gbps) on the same network segment, not directed at the firewalls themselves. The firewalls were being tested with a ~80Mbps stream of small HTTP requests forwarded with SYNPROXY, conntrackd and some traffic shaping enabled. Unfortunately, this trace comes from Debian distro kernel (3.16), as I haven't yet upgraded this machine and I don't have serial console on the other one at the moment. The other node is running 4.4.1 and rebooted at the same time +/- seconds, /proc/uptime says it's up 15 seconds longer. netconsole stays silent during reboots. > [436190.637310] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 > [436190.674698] IP: [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436190.709683] PGD 0 > [436190.719785] Oops: 0000 [#1] SMP > [436190.735431] Modules linked in: ipt_SYNPROXY nf_synproxy_core binfmt_misc xt_CLASSIFY xt_mac iptable_filter macvlan ip_set_hash_net cls_u32 sch_red sch_htb nf_conntrac > k_netlink xt_set ip_set nfnetlink iptable_mangle xt_tcpudp xt_CT xt_conntrack iptable_raw ip_tables arptable_filter arp_tables x_tables intel_powerclamp coretemp kvm_inte > l kvm crc32_pclmul aesni_intel aes_x86_64 joydev lrw gf128mul glue_helper ttm drm_kms_helper hpwdt drm i2c_algo_bit i2c_core hpilo psmouse evdev iTCO_wdt acpi_power_meter > pcspkr serio_raw iTCO_vendor_support lpc_ich ablk_helper cryptd shpchp i7core_edac nf_conntrack_ftp mfd_core button edac_core nf_conntrack_ipv4 nf_defrag_ipv4 processor > nf_conntrack ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct1 > 0dif_pclmul crct10dif_common crc32c_intel be2iscsi ehci_pci uhci_hcd iscsi_boot_sysfs libiscsi ehci_hcd usbcore scsi_transport_iscsi hpsa usb_common be2net scsi_mod therm > al vxlan thermal_sys > [436191.160517] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u2 > [436191.208587] Hardware name: HP ProLiant BL460c G7, BIOS I27 07/02/2013 > [436191.239665] task: ffff88020af6c110 ti: ffff88020af78000 task.ti: ffff88020af78000 > [436191.274842] RIP: 0010:[<ffffffffa014af73>] [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436191.321186] RSP: 0018:ffff88020ba43c40 EFLAGS: 00010246 > [436191.346218] RAX: 0000000000030003 RBX: ffff880202931300 RCX: 0000000000000003 > [436191.381252] RDX: 00000000ff93df09 RSI: 0000000000000028 RDI: ffff8802064bdb0c > [436191.414649] RBP: ffff8802064bdb08 R08: ffffffff81459ea0 R09: 0000000000000020 > [436191.449511] R10: ffff880203600000 R11: ffff8800e7b38000 R12: 0000000000000028 > [436191.482769] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801e21b3062 > [436191.516924] FS: 0000000000000000(0000) GS:ffff88020ba40000(0000) knlGS:0000000000000000 > [436191.554343] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [436191.581010] CR2: 0000000000000004 CR3: 0000000001813000 CR4: 00000000000007e0 > [436191.613999] Stack: > [436191.624137] ffffffff00000014 ffffc9000d8c1800 ffff8802064bdb0c ffffffff00000000 > [436191.660568] ffff8801f92b0af0 0000000000000000 ffff880202931300 ffff8802064bdb08 > [436191.696262] ffff880203600000 0000000000000000 ffff880202931300 ffffffff818e95a0 > [436191.730498] Call Trace: > [436191.742505] <IRQ> > [436191.752229] [<ffffffffa0206bd3>] ? ipv4_confirm+0xa3/0xf0 [nf_conntrack_ipv4] > [436191.786089] [<ffffffff81459ea0>] ? ip_fragment+0x880/0x880 > [436191.813367] [<ffffffff8144f575>] ? nf_iterate+0x65/0xa0 > [436191.838356] [<ffffffff81459ea0>] ? ip_fragment+0x880/0x880 > [436191.864832] [<ffffffff8144f626>] ? nf_hook_slow+0x76/0x130 > [436191.891480] [<ffffffff81459ea0>] ? ip_fragment+0x880/0x880 > [436191.917857] [<ffffffff8145ba42>] ? ip_output+0x82/0x90 > [436191.943454] [<ffffffff8141ebc3>] ? __netif_receive_skb_core+0x543/0x750 > [436191.974406] [<ffffffff8141ee4f>] ? netif_receive_skb_internal+0x1f/0x80 > [436192.006268] [<ffffffff8141f76a>] ? napi_gro_frags+0x1ba/0x2c0 > [436192.032907] [<ffffffffa00dc2f4>] ? be_process_rx+0x2c4/0x780 [be2net] > [436192.064273] [<ffffffffa00dc9a2>] ? be_poll+0x1f2/0x400 [be2net] > [436192.091625] [<ffffffff8141f1d0>] ? net_rx_action+0x140/0x240 > [436192.119082] [<ffffffff8106c681>] ? __do_softirq+0xf1/0x290 > [436192.144594] [<ffffffff8106ca55>] ? irq_exit+0x95/0xa0 > [436192.168465] [<ffffffff81516ae2>] ? do_IRQ+0x52/0xe0 > [436192.192353] [<ffffffff8151492d>] ? common_interrupt+0x6d/0x6d > [436192.219491] <EOI> > [436192.228773] [<ffffffff8108aded>] ? __hrtimer_start_range_ns+0x1cd/0x390 > [436192.261747] [<ffffffff813dfbbf>] ? cpuidle_enter_state+0x4f/0xc0 > [436192.290427] [<ffffffff813dfbb8>] ? cpuidle_enter_state+0x48/0xc0 > [436192.319838] [<ffffffff810a80d8>] ? cpu_startup_entry+0x2f8/0x400 > [436192.348103] [<ffffffff81042c5f>] ? start_secondary+0x20f/0x2d0 > [436192.376655] Code: 00 00 00 88 44 24 08 83 e0 01 48 8d 14 00 48 01 d0 4d 8d 74 85 00 48 8d 45 04 48 89 c7 48 89 44 24 10 e8 a1 88 3c e1 41 8b 57 04 <41> 8b 76 04 45 8b 46 08 41 8b 7f 08 89 d1 0f c9 41 39 0e 0f 88 > [436192.464643] RIP [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436192.499548] RSP <ffff88020ba43c40> > [436192.515631] CR2: 0000000000000004 > [436192.531367] ---[ end trace 84529a434dcea659 ]--- > [436192.531372] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 > [436192.531380] IP: [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436192.531381] PGD 0 > [436192.531382] Oops: 0000 [#2] SMP > [436192.531402] Modules linked in: ipt_SYNPROXY nf_synproxy_core binfmt_misc xt_CLASSIFY xt_mac iptable_filter macvlan ip_set_hash_net cls_u32 sch_red sch_htb nf_conntrack_netlink xt_set ip_set nfnetlink iptable_mangle xt_tcpudp xt_CT xt_conntrack iptable_raw ip_tables arptable_filter arp_tables x_tables intel_powerclamp coretemp kvm_intel kvm crc32_pclmul aesni_intel aes_x86_64 joydev lrw gf128mul glue_helper ttm drm_kms_helper hpwdt drm i2c_algo_bit i2c_core hpilo psmouse evdev iTCO_wdt acpi_power_meter pcspkr serio_raw iTCO_vendor_support lpc_ich ablk_helper cryptd shpchp i7core_edac nf_conntrack_ftp mfd_core button edac_core nf_conntrack_ipv4 nf_defrag_ipv4 processor nf_conntrack ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common crc32c_intel be2iscsi ehci_pci uhci_hcd iscsi_boot_sysfs libiscsi ehci_hcd usbcore scsi_transport_iscsi hpsa usb_commo n be2net scsi_mod thermal vxlan thermal_sys > [436192.531411] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u2 > [436192.531411] Hardware name: HP ProLiant BL460c G7, BIOS I27 07/02/2013 > [436192.531412] task: ffff88020af6d3b0 ti: ffff88020af70000 task.ti: ffff88020af70000 > [436192.531416] RIP: 0010:[<ffffffffa014af73>] [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436192.531416] RSP: 0018:ffff88010bc23ca0 EFLAGS: 00010246 > [436192.531417] RAX: 0000000000030003 RBX: ffff8800371f0800 RCX: 0000000000000003 > [436192.531418] RDX: 000000001a710bff RSI: 0000000000000028 RDI: ffff8801f913ac44 > [436192.531418] RBP: ffff8801f913ac40 R08: ffffffff81459ea0 R09: 0000000000000020 > [436192.531419] R10: ffff880203600000 R11: ffff8800e7b38000 R12: 0000000000000028 > [436192.531419] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88010b84b862 > [436192.531420] FS: 0000000000000000(0000) GS:ffff88010bc20000(0000) knlGS:000000 The kernel log ends here, even though it looks like we could be expecting another identical or similar trace on another CPU (#2 vs #4 earlier). The issue repeats intermittently but I cannot reliably reproduce it (except that DoS attacks tend to bring the whole thing down). I don't know the nature of the attack, but the machines have easily survived 9 Gbps of iperf tests. Any help is greatly appreciated while I dig into the code on my own. Best regards, Grzegorz Nosek
Powered by blists - more mailing lists