Message-ID: <20110127144248.GE2277@verge.net.au>
Date:	Fri, 28 Jan 2011 00:42:48 +1000
From:	Simon Horman <horms@...ge.net.au>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	Simon Kirby <sim@...tway.ca>, netdev@...r.kernel.org
Subject: Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

On Thu, Jan 27, 2011 at 08:42:05AM +0100, Eric Dumazet wrote:
> Le mercredi 26 janvier 2011 à 16:48 -0800, Simon Kirby a écrit :
> > On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote:
> > 
> > > Hi Simon,
> > > 
> > > thanks for prodding me to respond to this post offline and sorry for not
> > > responding earlier.
> > > 
> > > Firstly, I think that this is a receive-side problem so I don't believe
> > > that GSO (generic segmentation offload) or other transmit-side options are
> > > likely to have any affect.
> > > 
> > > My understanding is that on the receive-side there are two options which
> > > when enabled can result in the behaviour that you describe.
> > > 
> > > * LRO (large receive offload)
> > > 
> > >   You have this disabled, and assuming it really is disabled it
> > >   shouldn't be causing a problem.
> > > 
> > > * GRO (generic receive offload)
> > > 
> > >   This does not seem to be in the output of your ethtool commands at all.
> > >   So I wonder if your ethtool is too old to support this option?
> > 
> > So, this was the case.  Our ethtool (lenny) was too old to see the GRO
> > option, only GSO.  Disabling GRO on eth1.39 has no effect, but disabling
> > it on eth1 caused it to stop receiving the merged frames, fixing the LVS
> > packet loss (due to no sending GSO support from LVS/IPVS).
> > 
> > Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? 
> > 
> > >   In any case, I was able to reproduce the problem that you describe (or at
> > >   least something very similar) using 2.6.36 with GRO enabled on eth1.1 and
> > >   the problem did not manifest when I disabled GRO on eth1.1.
> > 
> > It worked for you to do ethtool -K eth1.1 gro off, then?  For me on
> > 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even
> > though packets arrive on eth1.39.
> > 
> > Also, strangely, 2.6.35.4's default state (with no received merged frames)
> > has GRO on for eth1 but off for eth1.39:
> > 
> > # ethtool -k eth1
> > Offload parameters for eth1:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp-segmentation-offload: on
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off
> > ntuple-filters: off
> > receive-hashing: off
> > 
> > # ethtool -k eth1.39
> > Offload parameters for eth1.39:
> > rx-checksumming: on
> > tx-checksumming: off
> > scatter-gather: off
> > tcp-segmentation-offload: off
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: off
> > generic-receive-offload: off
> > large-receive-offload: off
> > ntuple-filters: off
> > receive-hashing: off
> > 
> > If I set 2.6.37 to have all of the same options, I still see GRO frames
> > on 2.6.37 (tg3), which is weird.
> > 
> 
> Weird maybe, but GRO check/handling is done in dev_gro_receive(), on
> eth1 receive path.
> 
> Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO
> machine state) before being delivered to eth1.39
> 
> It would be useless/expensive to add another GRO layer on eth1.39
> 
> We might not report GRO state on vlan/bonding (or reflect real device
> GRO state)

That makes sense. So to clarify, this is the expected behaviour?

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists