Message-ID: <19014.1415325508@famine>
Date:	Thu, 06 Nov 2014 17:58:28 -0800
From:	Jay Vosburgh <jay.vosburgh@...onical.com>
To:	netdev@...r.kernel.org, discuss@...nvswitch.org
Subject: kernel panic receiving flooded VXLAN traffic with OVS


	I am able to reproduce a kernel panic on an system using
openvswitch when receiving VXLAN traffic under a very specific set of
circumstances.  This occurs with a recent net-next as well as an Ubuntu
3.13 kernel.  I'm not sure if the error lies in OVS, GRO, or elsewhere.

	In summary, when the system receives multiple VXLAN encapsulated
TCP segments for a different system (not intended for local reception)
that are from the middle of an active connection (received due to a switch
flood), and are tagged to a VLAN not configured on the local host, then
the system panics in skb_segment when OVS calls __skb_gso_segment on the
GRO skb prior to performing an upcall to user space.

	The panic occurs in skbuff.c:skb_segment(), at the BUG_ON around
line 3036:

struct sk_buff *skb_segment(struct sk_buff *head_skb,
			    netdev_features_t features)
{
[...]
		skb_shinfo(nskb)->tx_flags = skb_shinfo(head_skb)->tx_flags &
			SKBTX_SHARED_FRAG;

		while (pos < offset + len) {
			if (i >= nfrags) {
				BUG_ON(skb_headlen(list_skb));

				i = 0;


	The BUG_ON triggers because the skbs that have been GRO
accumulated are partially or entirely linear, depending upon the receiving
network device (sky2 is partial, enic is entire).  The receive buffers end
up being linear evidently because the mtu is set to 9000, and
__netdev_alloc_skb calls __alloc_skb (and thus kmalloc) instead of
__netdev_alloc_frag followed by build_skb.

	The foreign-VLAN VXLAN TCP segments are not processed as normal
VXLAN traffic, as there is no listener on the VLAN in question, so once
GRO processes them, they are sent directly to ovs_vport_receive.  The
panic stack appears as follows:

[ 6558.812214] kernel BUG at net/core/skbuff.c:3025!
[ 6558.812214] invalid opcode: 0000 [#1] SMP 
[ 6558.812214] Modules linked in: veth 8021q garp mrp bonding xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables openvswitch vxlan ip6_udp_tunnel udp_tunnel gre libcrc32c i915 video drm_kms_helper coretemp drm kvm_intel kvm gpio_ich ppdev parport_pc lp lpc_ich serio_raw i2c_algo_bit parport mac_hid hid_generic usbhid hid psmouse r8169 mii sky2
[ 6558.812214] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted 3.17.0-rc7-testola+ #5
[ 6558.812214] Hardware name: LENOVO 0829F3U/To be filled by O.E.M., BIOS 90KT15AUS 07/21/2010
[ 6558.812214] task: ffff880139eb3200 ti: ffff880139ed0000 task.ti: ffff880139ed0000
[ 6558.812214] RIP: 0010:[<ffffffff81616bc2>]  [<ffffffff81616bc2>] skb_segment+0x9d2/0xa00
[ 6558.812214] RSP: 0018:ffff880139ed3610  EFLAGS: 00010216
[ 6558.812214] RAX: 00000000000002dc RBX: ffff8800a3be5e00 RCX: ffff8800b10a26f0
[ 6558.812214] RDX: 0000000000000074 RSI: ffff8800b10a2600 RDI: ffff8800b10a2000
[ 6558.812214] RBP: ffff880139ed36e0 R08: 0000000000000022 R09: 0000000000000000
[ 6558.812214] R10: ffff8800b11e6000 R11: 00000000000005ca R12: ffff8800b10a20f0
[ 6558.812214] R13: 0000000000000000 R14: ffff8800b116cb00 R15: 0000000000000074
[ 6558.812214] FS:  0000000000000000(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[ 6558.812214] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 6558.812214] CR2: 00007fa906f4f148 CR3: 00000000b2a46000 CR4: 00000000000407f0
[ 6558.812214] Stack:
[ 6558.812214]  00000000000016a0 ffff880031353800 ffffffffffffffde ffff8800000005ca
[ 6558.812214]  0000000000000022 0000000000000040 ffff8800b11e6000 00000001000016a0
[ 6558.812214]  0000000000000000 0000000000000022 00000000000005a8 ffff8800a3be5e00
[ 6558.812214] Call Trace:
[ 6558.812214]  [<ffffffff8168c97f>] udp4_ufo_fragment+0x10f/0x1a0
[ 6558.812214]  [<ffffffff81695c51>] inet_gso_segment+0x141/0x370
[ 6558.812214]  [<ffffffff810aa2c8>] ? __wake_up_common+0x58/0x90
[ 6558.812214]  [<ffffffff81624f4f>] skb_mac_gso_segment+0x9f/0x100
[ 6558.812214]  [<ffffffff81625016>] __skb_gso_segment+0x66/0xd0
[ 6558.812214]  [<ffffffffa01d4c91>] queue_gso_packets+0x41/0x130 [openvswitch]
[ 6558.812214]  [<ffffffff8121aa4d>] ? ep_poll_safewake+0x2d/0x30
[ 6558.812214]  [<ffffffff8121b03d>] ? ep_poll_callback+0xcd/0x170
[ 6558.812214]  [<ffffffff810aa2c8>] ? __wake_up_common+0x58/0x90
[ 6558.812214]  [<ffffffff810aa860>] ? __wake_up_sync_key+0x50/0x60
[ 6558.812214]  [<ffffffff8161c232>] ? __skb_flow_dissect+0x162/0x4c0
[ 6558.812214]  [<ffffffff8172001f>] ? __slab_free+0xfe/0x2c3
[ 6558.812214]  [<ffffffff816107af>] ? kfree_skbmem+0x3f/0xa0
[ 6558.812214]  [<ffffffff8161c5ba>] ? __skb_get_hash+0x2a/0x160
[ 6558.812214]  [<ffffffffa01d609e>] ovs_dp_upcall+0x2e/0x70 [openvswitch]
[ 6558.812214]  [<ffffffffa01d6193>] ovs_dp_process_packet+0xb3/0xd0 [openvswitch]
[ 6558.812214]  [<ffffffffa01dc860>] ovs_vport_receive+0x60/0x80 [openvswitch]
[ 6558.812214]  [<ffffffff811828f1>] ? zone_statistics+0x81/0xa0
[ 6558.812214]  [<ffffffff81617819>] ? skb_gro_receive+0x559/0x5f0
[ 6558.812214]  [<ffffffff81695ada>] ? inet_gro_receive+0x1da/0x210
[ 6558.812214]  [<ffffffffa01dd10a>] netdev_frame_hook+0xca/0x130 [openvswitch]
[ 6558.812214]  [<ffffffff816233aa>] __netif_receive_skb_core+0x1ba/0x7a0
[ 6558.812214]  [<ffffffff816239a8>] __netif_receive_skb+0x18/0x60
[ 6558.812214]  [<ffffffff81623a13>] netif_receive_skb_internal+0x23/0x90
[ 6558.812214]  [<ffffffff8168cefa>] ? udp4_gro_complete+0x6a/0x70
[ 6558.812214]  [<ffffffff81623b94>] napi_gro_complete+0xa4/0xe0
[ 6558.812214]  [<ffffffff81623c3d>] napi_gro_flush+0x6d/0x90
[ 6558.812214]  [<ffffffff81623c7e>] napi_complete+0x1e/0x50
[ 6558.812214]  [<ffffffffa0006538>] sky2_poll+0xa38/0xd80 [sky2]
[ 6558.812214]  [<ffffffff81623e02>] net_rx_action+0x152/0x250
[ 6558.812214]  [<ffffffff81070aa5>] __do_softirq+0xf5/0x2e0
[ 6558.812214]  [<ffffffff81070cc0>] run_ksoftirqd+0x30/0x50
[ 6558.812214]  [<ffffffff8108e0ff>] smpboot_thread_fn+0xff/0x1b0
[ 6558.812214]  [<ffffffff8108e000>] ? SyS_setgroups+0x1a0/0x1a0
[ 6558.812214]  [<ffffffff8108a5a2>] kthread+0xd2/0xf0
[ 6558.812214]  [<ffffffff8108a4d0>] ? kthread_create_on_node+0x180/0x180
[ 6558.812214]  [<ffffffff81729e3c>] ret_from_fork+0x7c/0xb0
[ 6558.812214]  [<ffffffff8108a4d0>] ? kthread_create_on_node+0x180/0x180
[ 6558.812214] Code: 8b 44 24 70 44 8b 4c 24 30 44 8b 5c 24 18 8b 54 24 08 48 8b 0c 24 0f 85 0f fd ff ff e9 06 fd ff ff 0f 1f 84 00 00 00 00 00 0f 0b <0f> 0b 0f 0b c6 44 24 3b 01 e9 28 f7 ff ff e8 76 db 10 00 0f 0b 
[ 6558.812214] RIP  [<ffffffff81616bc2>] skb_segment+0x9d2/0xa00
[ 6558.812214]  RSP <ffff880139ed3610>

	I'm not sure if this is an error on the part of the RX / GRO
processing in assembling the GRO skb, or in how OVS calls skb_segment.

	-J

---
	-Jay Vosburgh, jay.vosburgh@...onical.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists