netdev - Re: Interaction between GSO and TSQ for single TCP stream

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CA+mtBx_HhKGHLy_FfgpeepMJsV-JShZ0xKphT2j-3RDS5M+cMw@mail.gmail.com>
Date:	Tue, 27 Jan 2015 12:08:33 -0800
From:	Tom Herbert <therbert@...gle.com>
To:	Eric Dumazet <edumazet@...gle.com>
Cc:	Neal Cardwell <ncardwell@...gle.com>,
	Yuchung Cheng <ycheng@...gle.com>,
	Linux Netdev List <netdev@...r.kernel.org>
Subject: Re: Interaction between GSO and TSQ for single TCP stream

On Tue, Jan 27, 2015 at 12:02 PM, Eric Dumazet <edumazet@...gle.com> wrote:
> On Tue, Jan 27, 2015 at 11:57 AM, Tom Herbert <therbert@...gle.com> wrote:
>> Eric,
>>
>> I am looking at why we are unable to achieve line rate throughput with
>> a single connection using UDP encapsulation that supports GSO/GRO. I
>> discovered that even for a non-encpasulated TCP connection I'm not
>> able to get line rate using GSO/GRO (but do with TSO/LRO). There seems
>> to be some interaction between TSQ and GSO. Setting
>> tcp_limit_output_bytes to 1000000 gets single connection up to line
>> rate for both native and encapsulated TCP.
>>
>> Some data:
>>
>> - netperf TCP_STREAM for a single connection no encapsulation
>>   Using TSO/LRO
>>     9412.44 Gbps
>>   Using GSO/GRO (ethtool -K eth0 gso on gro on tso off lro off)
>>     9286.08 Gbps
>>   Using GSO/GRO with larger limit (echo 1000000 >
>> /proc/sys/net/ipv4/tcp_limit_output_bytes)
>>     9412.75
>>
>> - netperf TCP_STREAM for a single connection for GUE-IPIP encapsulation with RCO
>>   Using GSO/GRO
>>     8842.89 Gbps
>>   Using GSO/GRO with larger limit (echo 1000000 >
>> /proc/sys/net/ipv4/tcp_limit_output_bytes)
>>     9143.99 Gbps
>>
>> Any ideas on how to address this?
>>
>> Thanks,
>> Tom
>
> Hi Tom.
>
> Yes, Eyal Perry  reported the issue a while back, we plan to send
> upstream patches to fix the issue today.
>
Awesome, looking forward to seeing them!

> Make sure you tweaked coalescing parameters if you use mlx4 :
>
> ethtool -C eth0 tx-usecs 4 tx-frames 4
>
It's bnx2x.

> Otherwise, the NIC accumulates too many packets before sending an
> interrupt for TX completion.
>
> Then you can either wait the patches, or try the one I sent :
> http://permalink.gmane.org/gmane.linux.network/347023
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html