netdev - Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1291375743.2897.141.camel@edumazet-laptop>
Date:	Fri, 03 Dec 2010 12:29:03 +0100
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Simon Kirby <sim@...tway.ca>
Cc:	netdev@...r.kernel.org, Simon Horman <horms@...ge.net.au>
Subject: Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

Le vendredi 03 décembre 2010 à 02:34 -0800, Simon Kirby a écrit :
> Hello!
> 
> We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards
> (partno(BCM95721) rev 4201) with VLAN tags in use, to think that
> everything looked great, but in fact...
> 
> LVS was receiving magically-merged TCP packets which it tried to forward
> on to the real server, only to get annoyed at itself for trying to
> forward a packet bigger than the device MTU:
> 
> IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081>
> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098>
> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
> 

Hi Simon

This is a tcpdump on A ?
Could you take it also on B ?

tcpdump displays large buffers, but they should be split (of course)
when sent on wire.

> This caused packet loss for any merged frames, which caused abysmal
> performance for uploads via the LVS server.  Local performance to or
> from the box is still fine, because the stack doesn't care, only the
> forwarding part of LVS is running into the problem.
> 
> Furthermore, disabling _everything_ reported by ethtool -k doesn't seem
> to change the result, even if I down/up the interface after, and even if
> I try on every single interface including the VLANned ones.  This seems
> to be another bug.  Reverting to 2.6.35 makes it all work again.
> 
> Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653
> 
> So how should this be fixed?  Should LVS be taught to fragment, or must
> we disable the merging in this case?  It seems like it would work well if
> the sending side could do the same offload in reverse, but I'm not sure
> if that would be possible.
> 
> Simon-


I believe Simon Horman has some patches for GRO and LVS.

Please send the results of "ethtool -k eth0" on all your nics / vlans ?

For TSO, I am not sure why and where it could matter...



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html