netdev - Re: Weird TCP retransmit behaviour in recent kernels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Sun, 16 May 2010 01:45:11 +0300 (EEST)
From:	"Ilpo Järvinen" <ilpo.jarvinen@...sinki.fi>
To:	Michael Smith <michael@...ts.ca>
cc:	Netdev <netdev@...r.kernel.org>
Subject: Re: Weird TCP retransmit behaviour in recent kernels

On Fri, 14 May 2010, Michael Smith wrote:

> I'm struggling with TCP sessions stalling when Windows XP SP2 clients
> connect to a SUSE Linux Enterprise 11 server (kernel 2.6.27.x). The
> problem doesn't occur with kernel 2.6.18.8 on the server, and I'm
> wondering if something's changed since then in the retransmit logic.
> 
> It seems like when consecutive packets are lost, the SLES11
> server retransmits the first packet when the timeout fires. The client
> ACKs, but the server doesn't retransmit the next lost packet; instead,
> it sends a couple more new packets,

Which is expected and desired change, known as FRTO (RFC 5682).

> which don't get ACKed.

This is where your problem is, they should get acked in a _compliant_ 
network (with duplicate ACKs).

> The new packets don't show up in Wireshark - either something in the 
> network is dropping them,

There's some non-compliant middlebox in the network?

> or maybe Windows doesn't forward them to WinPcap because
> there's a hole in the sequence. The timeout fires again after double
> the time, and the second packet is retransmitted and ACKed, then
> more brand new packets are sent out. The transfer quickly grinds to a
> halt.
>
> There's a WAN and VPN between the clients and the server. HTTP downloads
> from the server stall at various points depending on the client. The
> point at which the connection stalls seems to be dependent on latency.
> For example, if the RTT to the client is 12 ms, the connection might
> usually stall after 120 KB; if it's 20 ms, it might stall at 1200 KB.
> 
> The problem doesn't occur when a Windows client talks to a Windows
> server.  When a Linux client talks to the SLES11 server, the connection
> doesn't stall completely but slows to a crawl (~3 KB/sec, as opposed to
> typical 50-200 KB/sec).
> 
> I was able to work around the problem for most clients by locking the
> TCP congestion window to a maximum of 6 on the SLES11 server. Some sites
> are pathologically bad and the connection stalls unless I lock the
> congestion window to 1 (!!).
> 
> I've put up a couple of sample traces from a pathological site where
> the problem shows up with cwnd locked to 3:
> 
> http://www.hurts.ca/sles11.router.pcap.gz - view from the server's firewall
> http://www.hurts.ca/sles11.windows.pcap.gz - view from a client PC
> 
> On the firewall, you can see the problem around packets 93-104. The server
> sends sequence 66781, 68041, 69301; retransmits 66781, gets an ACK, then
> sends 70561, 71821; retransmits 68041, gets an ACK, then sends 73081,
> 74341, and so on. On the client, the "future" sequence packets after
> the ACK never show up in Wireshark.
> 
> I've tried all of the obvious things:
> - disabling TCP segment/checksum offloading functions on client and server;
> - disabling SACK;
> - trying all available congestion control algorithms on SLES11
>   (cubic, reno, veno, illinois);
> - turning off anti-virus on the client.
> 
> The only 100% reliable workaround seems to be to proxy the connections
> through a kernel 2.6.18.8 machine on the same subnet. It seems like
> the problem exists with a vanilla 2.6.31 kernel, too.
> 
> Has anyone seen something like this before? Any ideas where to go next? I'm
> pretty sure there's nothing strange in the network - just plain old Cisco
> routers and site-to-site VPNs.

Some have seen similar phenomena, every time it has been fault in some 
middlebox/peer that does not do what it should. You can disable frto 
using tcp_frto sysctl if you like, however, I disagree with you as I'm 
pretty sure there is some broken middlebox in the network (which is trying 
to be too intelligent).

-- 
 i.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html