netdev - Re: TCP funny-ness when over-driving a 1Gbps link.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 19 May 2011 17:12:50 -0700
From:	Ben Greear <greearb@...delatech.com>
To:	rick.jones2@...com
CC:	Stephen Hemminger <shemminger@...tta.com>,
	netdev <netdev@...r.kernel.org>
Subject: Re: TCP funny-ness when over-driving a 1Gbps link.

On 05/19/2011 05:05 PM, Rick Jones wrote:
> On Thu, 2011-05-19 at 16:42 -0700, Ben Greear wrote:
>> On 05/19/2011 04:20 PM, Ben Greear wrote:
>>> On 05/19/2011 04:18 PM, Stephen Hemminger wrote:
>>
>>>> If you overdrive, TCP expects your network emulator to have
>>>> a some but limited queueing (like a real router).
>>>
>>> The emulator is fine, it's not being over-driven (and has limited
>>> queueing if it was
>>> being over-driven). The queues that are backing up are in the tcp
>>> sockets on the
>>> sending machine.
>>>
>>> But, just to make sure, I'll re-run the test with a looped back cable...
>>
>> Well, with looped back cable, it isn't so bad.  I still see a small drop
>> in aggregate throughput (around 900Mbps instead of 950Mbps), and
>> latency goes above 600ms, but it still performs better than when
>> going through the emulator.
>>
>> At 950+Mbps, the emulator is going to impart 1-2 ms of latency
>> even when configured for wide-open.
>>
>> If I use a bridge in place of the emulator, it seems to settle on
>> around 450Mbps in one direction and 945Mbps in the other (on the wire),
>> with round-trip latencies often over 5 seconds (user-space to user-space),
>> and a consistent large chunk of data in the socket send buffers:
>>
>> [root@...965-1 igb]# netstat -an|grep tcp|grep 8.1.1
>> tcp        0      0 8.1.1.1:33038               0.0.0.0:*                   LISTEN
>> tcp        0      0 8.1.1.1:33040               0.0.0.0:*                   LISTEN
>> tcp        0      0 8.1.1.1:33042               0.0.0.0:*                   LISTEN
>> tcp        0 9328612 8.1.1.2:33039               8.1.1.1:33040               ESTABLISHED
>> tcp        0 17083176 8.1.1.1:33038               8.1.1.2:33037               ESTABLISHED
>> tcp        0 9437340 8.1.1.2:33037               8.1.1.1:33038               ESTABLISHED
>> tcp        0 17024620 8.1.1.1:33040               8.1.1.2:33039               ESTABLISHED
>> tcp        0 19557040 8.1.1.1:33042               8.1.1.2:33041               ESTABLISHED
>> tcp        0 9416600 8.1.1.2:33041               8.1.1.1:33042               ESTABLISHED
>
> I take it your system has higher values for the tcp_wmem value:
>
> net.ipv4.tcp_wmem = 4096 16384 4194304

Yes:
[root@...965-1 igb]# cat /proc/sys/net/ipv4/tcp_wmem
4096	16384	50000000

> and whatever is creating the TCP connections is not making explicit
> setsockopt() calls to set SO_*BUF.

It is configured not to, but if you know of an independent way to verify
that, I'm interested.

Thanks,
Ben

>
> rick jones


-- 
Ben Greear <greearb@...delatech.com>
Candela Technologies Inc  http://www.candelatech.com

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html