netdev - Re: TCP connection issues against Amazon S3

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <A58F8257-97F8-4662-98FC-7AC57F35A3A4@bengler.no>
Date:	Wed, 7 Jan 2015 20:37:19 +0000
From:	Erik Grinaker <erik@...gler.no>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	Yuchung Cheng <ycheng@...gle.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	netdev <netdev@...r.kernel.org>
Subject: Re: TCP connection issues against Amazon S3

On 07 Jan 2015, at 15:58, Eric Dumazet <eric.dumazet@...il.com> wrote:
> On Wed, 2015-01-07 at 13:31 +0000, Erik Grinaker wrote:
>> On 06 Jan 2015, at 22:00, Yuchung Cheng <ycheng@...gle.com> wrote:
>>> On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker <erik@...gler.no> wrote:
>>>> 
>>>>> On 06 Jan 2015, at 20:26, Erik Grinaker <erik@...gler.no> wrote:
>>>> This still doesn’t explain why it works with older kernels, but not newer ones. I’m thinking it’s
>>> probably some minor change, which gets amplified by the lack of SACKs
>>> on the loadbalancer. Anyway, I’ll bring it up with Amazon.
>>> can you post traces with the older kernels?
>> 
>> Here is a dump using 3.11.10 against a non-SACK-enabled loadbalancer:
>> 
>> http://abstrakt.bengler.no/tcp-issues-s3-nosack-3.11.10.pcap.bz2
>> 
>> The transfer shows lots of DUPACKs and retransmits, but this does not
>> seem to have as bad an effect as it did with the failing transfer we
>> saw on newer kernels:
>> 
>> http://abstrakt.bengler.no/tcp-issues-s3-failure.pcap.bz2
>> 
>> One big difference, which Rick touched on earlier, is that the newer
>> kernels keep sending TCP window updates as it’s going through the
>> retransmits. The older kernel does not do this.
> 
> The new kernel is the receiver : It does no retransmits.
> 
> Increasing window in ACK packets should not prevent sender into
> retransmitting missing packets.
> 
> Sender is not a linux host and is very buggy IMO : If receiver
> advertises a too big window, sender decides to not retransmit in some
> cases.

I agree. I have contacted Amazon about this, but am not too hopeful for a quick fix; they have been promising SACK-support on their loadbalancers since 2006, for example.

That said, since this change breaks a service as popular as S3, it might be worth reconsidering.

> You can play with /proc/sys/net/ipv4/tcp_rmem and adopt very low values
> to work around the sender bug.
> 
> ( Or use SO_RCVBUF in receiver application)

Thanks, setting SO_RCVBUF seems like a reasonable workaround.--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html