netdev - Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1296114125.1783.139.camel@edumazet-laptop>
Date:	Thu, 27 Jan 2011 08:42:05 +0100
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Simon Kirby <sim@...tway.ca>
Cc:	Simon Horman <horms@...ge.net.au>, netdev@...r.kernel.org
Subject: Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

Le mercredi 26 janvier 2011 à 16:48 -0800, Simon Kirby a écrit :
> On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote:
> 
> > Hi Simon,
> > 
> > thanks for prodding me to respond to this post offline and sorry for not
> > responding earlier.
> > 
> > Firstly, I think that this is a receive-side problem so I don't believe
> > that GSO (generic segmentation offload) or other transmit-side options are
> > likely to have any affect.
> > 
> > My understanding is that on the receive-side there are two options which
> > when enabled can result in the behaviour that you describe.
> > 
> > * LRO (large receive offload)
> > 
> >   You have this disabled, and assuming it really is disabled it
> >   shouldn't be causing a problem.
> > 
> > * GRO (generic receive offload)
> > 
> >   This does not seem to be in the output of your ethtool commands at all.
> >   So I wonder if your ethtool is too old to support this option?
> 
> So, this was the case.  Our ethtool (lenny) was too old to see the GRO
> option, only GSO.  Disabling GRO on eth1.39 has no effect, but disabling
> it on eth1 caused it to stop receiving the merged frames, fixing the LVS
> packet loss (due to no sending GSO support from LVS/IPVS).
> 
> Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? 
> 
> >   In any case, I was able to reproduce the problem that you describe (or at
> >   least something very similar) using 2.6.36 with GRO enabled on eth1.1 and
> >   the problem did not manifest when I disabled GRO on eth1.1.
> 
> It worked for you to do ethtool -K eth1.1 gro off, then?  For me on
> 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even
> though packets arrive on eth1.39.
> 
> Also, strangely, 2.6.35.4's default state (with no received merged frames)
> has GRO on for eth1 but off for eth1.39:
> 
> # ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> ntuple-filters: off
> receive-hashing: off
> 
> # ethtool -k eth1.39
> Offload parameters for eth1.39:
> rx-checksumming: on
> tx-checksumming: off
> scatter-gather: off
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: off
> large-receive-offload: off
> ntuple-filters: off
> receive-hashing: off
> 
> If I set 2.6.37 to have all of the same options, I still see GRO frames
> on 2.6.37 (tg3), which is weird.
> 

Weird maybe, but GRO check/handling is done in dev_gro_receive(), on
eth1 receive path.

Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO
machine state) before being delivered to eth1.39

It would be useless/expensive to add another GRO layer on eth1.39

We might not report GRO state on vlan/bonding (or reflect real device
GRO state)



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html