Message-ID: <20110113063422.GB14643@verge.net.au>
Date:	Thu, 13 Jan 2011 15:34:22 +0900
From:	Simon Horman <horms@...ge.net.au>
To:	Simon Kirby <sim@...tway.ca>
Cc:	Eric Dumazet <eric.dumazet@...il.com>, netdev@...r.kernel.org
Subject: Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

On Wed, Dec 22, 2010 at 03:39:48PM -0800, Simon Kirby wrote:
> On Thu, Dec 09, 2010 at 03:51:59AM +0100, Eric Dumazet wrote:
> 
> > Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit :
> > 
> > > Right.  So, on 2.6.35, ethtool shows:
> > > 
> > > # ethtool -k eth1
> > > Offload parameters for eth1:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > scatter-gather: on
> > > tcp segmentation offload: on
> > > udp fragmentation offload: off
> > > generic segmentation offload: on
> > > large receive offload: off
> > > 
> > > # ethtool -k eth1.39
> > > Offload parameters for eth1.39:
> > > rx-checksumming: on
> > > tx-checksumming: off
> > > scatter-gather: off
> > > tcp segmentation offload: off
> > > udp fragmentation offload: off
> > > generic segmentation offload: off
> > > large receive offload: off
> > > 
> > > On 2.6.36, ethtool shows:
> > > 
> > > # ethtool -k eth1
> > > Offload parameters for eth1:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > scatter-gather: on
> > > tcp segmentation offload: on
> > > udp fragmentation offload: off
> > > generic segmentation offload: on
> > > large receive offload: off
> > > 
> > > # ethtool -k eth1.39
> > > Offload parameters for eth1.39:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > scatter-gather: on
> > > tcp segmentation offload: on
> > > udp fragmentation offload: off
> > > generic segmentation offload: on
> > > large receive offload: off
> > > 
> > > And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off,
> > > it says "off", but I still see merged frames, as verified with an nc <
> > > /dev/zero and tcpdump -i any -n 'length > 1500'.
> > 
> > Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)
> 
> and turning it off makes no difference, either.  Even if I down/up the
> interface after.  I still see merged packets on receive, even if I try to
> use ethtool to turn off everything that is "on".  It's as if the new code
> to enable it on vlans works, but the disable from ethtool does not.
> 
> In other words, I cannot find a way to disable it (and thus make LVS
> work) on 2.6.36 or 2.6.37-rc6.

Hi Simon,

thanks for prodding me to respond to this post offline and sorry for not
responding earlier.

Firstly, I think that this is a receive-side problem so I don't believe
that GSO (generic segmentation offload) or other transmit-side options are
likely to have any affect.

My understanding is that on the receive-side there are two options which
when enabled can result in the behaviour that you describe.

* LRO (large receive offload)

  You have this disabled, and assuming it really is disabled it
  shouldn't be causing a problem.

* GRO (generic receive offload)

  This does not seem to be in the output of your ethtool commands at all.
  So I wonder if your ethtool is too old to support this option?

  In any case, I was able to reproduce the problem that you describe (or at
  least something very similar) using 2.6.36 with GRO enabled on eth1.1 and
  the problem did not manifest when I disabled GRO on eth1.1.

N.B: I'm not sure that I have any hardware whose driver supports LRO
     so my tests were made without LRO support.

> One interesting note is that if I destroy and recreate the VLAN interface,
> it comes back with gso enabled:

I guess that relate to how the default is set.
I don't imagine that it is the cause of the problem that you are seeing.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists