netdev - TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20101203103447.GA29714@hostway.ca>
Date:	Fri, 3 Dec 2010 02:34:47 -0800
From:	Simon Kirby <sim@...tway.ca>
To:	Eric Dumazet <eric.dumazet@...il.com>, netdev@...r.kernel.org
Subject: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

Hello!

We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards
(partno(BCM95721) rev 4201) with VLAN tags in use, to think that
everything looked great, but in fact...

LVS was receiving magically-merged TCP packets which it tried to forward
on to the real server, only to get annoyed at itself for trying to
forward a packet bigger than the device MTU:

IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081>
IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098>
IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556

This caused packet loss for any merged frames, which caused abysmal
performance for uploads via the LVS server.  Local performance to or
from the box is still fine, because the stack doesn't care, only the
forwarding part of LVS is running into the problem.

Furthermore, disabling _everything_ reported by ethtool -k doesn't seem
to change the result, even if I down/up the interface after, and even if
I try on every single interface including the VLANned ones.  This seems
to be another bug.  Reverting to 2.6.35 makes it all work again.

Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653

So how should this be fixed?  Should LVS be taught to fragment, or must
we disable the merging in this case?  It seems like it would work well if
the sending side could do the same offload in reverse, but I'm not sure
if that would be possible.

Simon-
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html