netdev - Re: [PATCH net-next 0/5] qed/qede: Tunnel hardware GRO support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <576B1AE8.6030309@hpe.com>
Date:	Wed, 22 Jun 2016 16:10:32 -0700
From:	Rick Jones <rick.jones2@....com>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	Yuval Mintz <Yuval.Mintz@...gic.com>,
	Alexander Duyck <alexander.duyck@...il.com>,
	Manish Chopra <manish.chopra@...gic.com>,
	David Miller <davem@...emloft.net>,
	netdev <netdev@...r.kernel.org>,
	Ariel Elior <Ariel.Elior@...gic.com>,
	Tom Herbert <tom@...bertland.com>,
	Hannes Frederic Sowa <hannes@...hat.com>
Subject: Re: [PATCH net-next 0/5] qed/qede: Tunnel hardware GRO support

On 06/22/2016 03:47 PM, Eric Dumazet wrote:
> On Wed, 2016-06-22 at 14:52 -0700, Rick Jones wrote:
>> On 06/22/2016 11:22 AM, Yuval Mintz wrote:
>>> But seriously, this isn't really anything new but rather a step forward in
>>> the direction we've already taken - bnx2x/qede are already performing
>>> the same for non-encapsulated TCP.
>>
>> Since you mention bnx2x...   I would argue that the NIC firmware on
>> those NICs driven by bnx2x is doing it badly.  Not so much from a
>> functional standpoint I suppose, but from a performance one.  The
>> NIC-firmware GRO done there has this rather unfortunate assumption about
>> "all MSSes will be directly driven by my own physical MTU" and when it
>> sees segments of a size other than would be suggested by the physical
>> MTU, will coalesce only two segments together.  They then do not get
>> further coalesced in the stack.
>>
>> Suffice it to say this does not do well from a performance standpoint.
>>
>> One can disable LRO via ethtool for these NICs, but what that does is
>> disable old-school LRO, not GRO-in-the-NIC.  To get that disabled, one
>> must also get the bnx2x module loaded with "disable-tpa=1" so the Linux
>> stack GRO gets used instead.
>>
>> Had the bnx2x-driven NICs' firmware not had that rather unfortunate
>> assumption about MSSes I probably would never have noticed.
>
> I do not see this behavior on my bnx2x nics ?
>
> ip ro add 10.246.11.52 via 10.246.11.254 dev eth0 mtu 1000
> lpk51:~# ./netperf -H 10.246.11.52 -l 1000
> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 10.246.11.52 () port 0 AF_INET

I first saw this with VMs which themselves had 1400 byte MTUs on their 
vNICs, speaking though bnx2x-driven NICs with a 1500 byte MTU, but I did 
later reproduce it by tweaking the MTU of my sending side NIC to 
something like 1400 bytes and running a "bare iron" netperf.  I believe 
you may be able to achieve the same thing by having netperf set a 
smaller MSS via the test-specific -G option.

My systems are presently in the midst of an install but I should be able 
to demonstrate it in the morning (US Pacific time, modulo the shuttle 
service of a car repair place)

> On receiver :

Paranoid question, but is LRO disabled on the receiver?  I don't know 
that LRO exhibits the behaviour, just GRO-in-the-NIC.

rick

>
> 15:46:08.296241 IP 10.246.11.52.46907 > 10.246.11.51.34131: Flags [.],
> ack 303360, win 8192, options [nop,nop,TS val 1245217243 ecr
> 1245306446], length 0
> 15:46:08.296430 IP 10.246.11.51.34131 > 10.246.11.52.46907: Flags [.],
> seq 303360:327060, ack 1, win 229, options [nop,nop,TS val 1245306446
> ecr 1245217242], length 23700
> 15:46:08.296441 IP 10.246.11.52.46907 > 10.246.11.51.34131: Flags [.],
> ack 327060, win 8192, options [nop,nop,TS val 1245217243 ecr
> 1245306446], length 0
> 15:46:08.296644 IP 10.246.11.51.34131 > 10.246.11.52.46907: Flags [.],
> seq 327060:350760, ack 1, win 229, options [nop,nop,TS val 1245306446
> ecr 1245217242], length 23700
> 15:46:08.296655 IP 10.246.11.52.46907 > 10.246.11.51.34131: Flags [.],
> ack 350760, win 8192, options [nop,nop,TS val 1245217244 ecr
> 1245306446], length 0
> 15:46:08.296854 IP 10.246.11.51.34131 > 10.246.11.52.46907: Flags [.],
> seq 350760:374460, ack 1, win 229, options [nop,nop,TS val 1245306446
> ecr 1245217242], length 23700
> 15:46:08.296897 IP 10.246.11.52.46907 > 10.246.11.51.34131: Flags [.],
> ack 374460, win 8192, options [nop,nop,TS val 1245217244 ecr
> 1245306446], length 0
> 15:46:08.297054 IP 10.246.11.51.34131 > 10.246.11.52.46907: Flags [.],
> seq 374460:398160, ack 1, win 229, options [nop,nop,TS val 1245306446
> ecr 1245217242], length 23700
> 15:46:08.297099 IP 10.246.11.52.46907 > 10.246.11.51.34131: Flags [.],
> ack 398160, win 8192, options [nop,nop,TS val 1245217244 ecr
> 1245306446], length 0
> 15:46:08.297258 IP 10.246.11.51.34131 > 10.246.11.52.46907: Flags [.],
> seq 398160:420912, ack 1, win 229, options [nop,nop,TS val 1245306446
> ecr 1245217242], length 22752
> 15:46:08.297301 IP 10.246.11.52.46907 > 10.246.11.51.34131: Flags [.],
> ack 420912, win 8192, options [nop,nop,TS val 1245217244 ecr
> 1245306446], length 0
>