Message-ID: <20130520094127.546a2f3e@vostro>
Date:	Mon, 20 May 2013 09:41:27 +0300
From:	Timo Teras <timo.teras@....fi>
To:	Pravin B Shelar <pshelar@...ira.com>, netdev@...r.kernel.org
Subject: GRE+XFRM+GSO crashes

Since upgrade from 3.8 to 3.9 I've started getting the below mentioned
BUG crashes. One of the few relevant changes seems to be GSO support in
GRE driver.

Turning off SG (and GSO) seems to make these crashes disappear.

The basic setup is:
- gre1 is an NBMA tunnel (no explicit destination, nor bound
  target interface; opennhrp daemon creates neigh mappings)
- IPsec policy to encrypt all GRE traffic in transport mode
- VIA Padlock hardware for AES acceleration
- GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso off

Incidentally, when I tried exact same setup ran as virtualized, I was
unable to reproduce this crash. I suspect it depends on the target NIC
acceleration capabilities.

This is from vanilla 3.9.2 kernel:
 BUG: unable to handle kernel NULL pointer dereference at 00000010
 IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4
 *pde = 00000000 
 Oops: 0000 [#1] SMP 
 Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
 deflate zlib_deflate ctr twofish_i586 twofish_generic twofish_common
 camellia_generic serpent_sse2_i586 xts lrw gf128mul glue_helper
 ablk_helper cryptd serpent_generic blowfish_generic blowfish_common
 cast5_generic cast_common des_generic cbc xcbc rmd160 sha512_generic
 hmac crypto_null af_key xfrm_algo ip_gre ipt_REJECT iptable_filter
 ipt_MASQUERADE xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
 nf_nat_ipv4 nf_nat ip_tables ip6t_REJECT xt_LOG xt_limit xt_recent
 xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
 nf_conntrack xt_multiport ip6table_filter ip6_tables x_tables ipv6
 af_packet padlock_sha padlock_aes via_cputemp hwmon hwmon_vid
 serio_raw psmouse pcspkr shpchp pci_hotplug i2c_viapro i2c_core
 via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus
 snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd
 soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169
 firmware_class mii evdev fan parport_pc parport thermal button
 acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat
 sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi
 libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop
 Pid: 1794, comm: opennhrp Not tainted 3.9.2 #2-Alpine    /CN700-8237
 EIP: 0060:[<c123e5b8>] EFLAGS: 00010246 CPU: 0
 EIP is at xfrm_output_resume+0x63/0x2a4
 EAX: 00000000 EBX: f0572500 ECX: f5ffc8b8 EDX: 00000064
 ESI: f0496400 EDI: 00000000 EBP: f0499bf8 ESP: f0499be8
  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
 CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690
 DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
 DR6: ffff0ff0 DR7: 00000400
 Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0 task.ti=f0498000)
 Stack:
  f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14 c123e89b
  f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c c1236865 f0572500
  0000003c f0499c38 c12016e7 f0572500 f0499cb0 f84175fb f0572a00 f04bb400
 Call Trace:
  [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
  [<c123e806>] xfrm_output2+0xd/0xf
  [<c123e89b>] xfrm_output+0x93/0xa0
  [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
  [<c1236837>] xfrm4_output_finish+0x27/0x29
  [<c1236865>] xfrm4_output+0x2c/0x63
  [<c12016e7>] ip_local_out+0x1b/0x1e
  [<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre]
  [<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c
  [<c11de379>] dev_queue_xmit+0x289/0x31a
  [<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet]
  [<c1040200>] ? ttwu_do_wakeup+0xe/0xa8
  [<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre]
  [<c11cd5a9>] sock_sendmsg+0x79/0x94
  [<c10b33d4>] ? __pollwait+0xa4/0xa4
  [<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3
  [<c10367c5>] ? free_pid+0x99/0x9f
  [<c106b92a>] ? call_rcu_sched+0xf/0x12
  [<c1026e15>] ? release_task+0x36d/0x37d
  [<c10390e8>] ? remove_wait_queue+0x31/0x36
  [<c1027a1d>] ? do_wait+0x1a8/0x1b5
  [<c11cef10>] sys_sendmsg+0x2b/0x46
  [<c11cf394>] sys_socketcall+0x145/0x19f
  [<c1257888>] syscall_call+0x7/0xb
 Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94 c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0 fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53
 EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP 0068:f0499be8
 CR2: 0000000000000010
 ---[ end trace ac96c6b6b1a4992f ]---
 Kernel panic - not syncing: Fatal exception in interrupt


Based on the disassembly, it seems the crash happens in
net/xfrm/xfrm_output.c:

int xfrm_output_resume(struct sk_buff *skb, int err)
{
        while (likely((err = xfrm_output_one(skb, err)) == 0)) {
                nf_reset(skb);

                err = skb_dst(skb)->ops->local_out(skb);
		      ^^^^^^^^^^^^ is NULL
                if (unlikely(err != 1))
                        goto out;

                if (!skb_dst(skb)->xfrm)
                        return dst_output(skb);
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists