netdev - Re: kernel panic receiving flooded VXLAN traffic with OVS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <30557.1415423596@famine>
Date:	Fri, 07 Nov 2014 21:13:16 -0800
From:	Jay Vosburgh <jay.vosburgh@...onical.com>
To:	Pravin Shelar <pshelar@...ira.com>
cc:	Jesse Gross <jesse@...ira.com>, netdev <netdev@...r.kernel.org>,
	"discuss@...nvswitch.org" <discuss@...nvswitch.org>
Subject: Re: kernel panic receiving flooded VXLAN traffic with OVS

Pravin Shelar <pshelar@...ira.com> wrote:

>On Fri, Nov 7, 2014 at 2:29 PM, Jesse Gross <jesse@...ira.com> wrote:
>> On Fri, Nov 7, 2014 at 1:13 PM, Jay Vosburgh <jay.vosburgh@...onical.com> wrote:
>>> Jesse Gross <jesse@...ira.com> wrote:
>>>
>>>>On Fri, Nov 7, 2014 at 10:34 AM, Jesse Gross <jesse@...ira.com> wrote:
>>>>> On Fri, Nov 7, 2014 at 9:40 AM, Pravin Shelar <pshelar@...ira.com> wrote:
>>>>>> On Thu, Nov 6, 2014 at 5:58 PM, Jay Vosburgh <jay.vosburgh@...onical.com> wrote:
>>> [...]
>>>>>>>         I'm not sure if this is an error on the part of the RX / GRO
>>>>>>> processing in assembling the GRO skb, or in how OVS calls skb_segment.
>>>>>>>
>>>>>>
>>>>>> I think this is related skb_segment() issue where it is not able to
>>>>>> handle this type of skb geometry. We need to fix skb-segmentation. I
>>>>>> will investigate it more.
>>>>>
>>>>> One problem that I see is that vxlan_gro_complete() doesn't add
>>>>> SKB_GSO_UDP_TUNNEL to gso_type. This causes us to attempt
>>>>> fragmentation as UDP rather than continuing down to do TCP
>>>>> segmentation. That probably screws up the skb geometry.
>>>>
>>>>I sent out a patch to fix this issue. I'm pretty sure that it is the
>>>>root cause of the originally reported case but I don't have a good way
>>>>to reproduce it so it would be great if you could test it Jay.
>>>
>>>         I'm having an issue there; when I set up my recreation on
>>> current net-next (3.18-rc2) without your new patch, I get the following
>>> oops when my ovs script does "ovs-vsctl --if-exists del-br br-ex":
>>
>> Hmm, that looks like a totally different problem. Pravin - any ideas?
>
>I am not able to reproduce with above command. Jay, Can you send me
>steps to reproduce this issue?

	Well, at the moment a 3.18.0-rc2 kernel panics in
ovs_flow_tbl_insert as soon as ovs-vswitchd starts up.  Booting to an
earlier kernel (3.17, for example) with no other changes doesn't panic.

	I moved /etc/openvswitch/conf.db away and rebooted the system
(which I think will eliminate the stored configuration), and the kernel
still hits this oops when ovs-vswitchd starts up.

	A bit of poking with crash 

[   22.180002] RIP: 0010:[<ffffffffa01a55ec>]  [<ffffffffa01a55ec>] ovs_flow_tbl_insert+0xdc/0x1f0 [openvswitch]
[   22.180002] RSP: 0018:ffff8801391a77a8  EFLAGS: 00010203
[   22.180002] RAX: 00000000076cc6f1 RBX: ffff8800b35c2020 RCX: 00000000fb994f19
[   22.180002] RDX: 000000009cc907e5 RSI: 00000000a490ff19 RDI: ffff8800b0c1c19c
[   22.180002] RBP: ffff8801391a77f8 R08: 000000006867223a R09: 00000000819d92a7
[   22.180002] R10: ffffffffa01a5696 R11: 0000000000000004 R12: ffff8800b35c2028
[   22.180002] R13: ffff8801391a7850 R14: ffff8800b35c2028 R15: ffff880134827800
[   22.180002] FS:  00007f0ef491a980(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000

0xffffffffa01a55e7 <ovs_flow_tbl_insert+0xd7>:	callq  0xffffffff813a75c0 <__jhash2>
0xffffffffa01a55ec <ovs_flow_tbl_insert+0xdc>:	mov    %eax,0x30(%r8)
0xffffffffa01a55f0 <ovs_flow_tbl_insert+0xe0>:	mov    (%rbx),%r13
0xffffffffa01a55f3 <ovs_flow_tbl_insert+0xe3>:	mov    %r8,%rsi
0xffffffffa01a55f6 <ovs_flow_tbl_insert+0xe6>:	mov    %r13,%rdi
0xffffffffa01a55f9 <ovs_flow_tbl_insert+0xe9>:	callq  0xffffffffa01a4ba0 <table_instance_insert>

	So it panics on return from __jhash2 because %r8 is invalid;
this is presumably

int ovs_flow_tbl_insert(struct flow_table *table, struct sw_flow *flow,
			struct sw_flow_mask *mask)
{
[...]
	flow->hash = flow_hash(&flow->key, flow->mask->range.start,
			flow->mask->range.end);
	ti = ovsl_dereference(table->ti);
	table_instance_insert(ti, flow);

	Looking at __jash2, it uses %r8 internally, but %r8 doesn't
appear to be saved and restored either around the call to, or within,
__jhash2.  In ovs_flow_tbl_insert %r8 appears to hold the "flow"
variable, so this panic might have nothing to do with ovs itself.

	I'll have to look at this further on Monday.

	-J

---
	-Jay Vosburgh, jay.vosburgh@...onical.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html