netdev - Re: Interaction between GSO and TSQ for single TCP stream

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iJCOuQfixdSp6TsPxpyvTxQSTLBJLgnsvbrbG2Am_p0-Q@mail.gmail.com>
Date:	Tue, 27 Jan 2015 12:02:41 -0800
From:	Eric Dumazet <edumazet@...gle.com>
To:	Tom Herbert <therbert@...gle.com>,
	Neal Cardwell <ncardwell@...gle.com>,
	Yuchung Cheng <ycheng@...gle.com>
Cc:	Linux Netdev List <netdev@...r.kernel.org>
Subject: Re: Interaction between GSO and TSQ for single TCP stream

On Tue, Jan 27, 2015 at 11:57 AM, Tom Herbert <therbert@...gle.com> wrote:
> Eric,
>
> I am looking at why we are unable to achieve line rate throughput with
> a single connection using UDP encapsulation that supports GSO/GRO. I
> discovered that even for a non-encpasulated TCP connection I'm not
> able to get line rate using GSO/GRO (but do with TSO/LRO). There seems
> to be some interaction between TSQ and GSO. Setting
> tcp_limit_output_bytes to 1000000 gets single connection up to line
> rate for both native and encapsulated TCP.
>
> Some data:
>
> - netperf TCP_STREAM for a single connection no encapsulation
>   Using TSO/LRO
>     9412.44 Gbps
>   Using GSO/GRO (ethtool -K eth0 gso on gro on tso off lro off)
>     9286.08 Gbps
>   Using GSO/GRO with larger limit (echo 1000000 >
> /proc/sys/net/ipv4/tcp_limit_output_bytes)
>     9412.75
>
> - netperf TCP_STREAM for a single connection for GUE-IPIP encapsulation with RCO
>   Using GSO/GRO
>     8842.89 Gbps
>   Using GSO/GRO with larger limit (echo 1000000 >
> /proc/sys/net/ipv4/tcp_limit_output_bytes)
>     9143.99 Gbps
>
> Any ideas on how to address this?
>
> Thanks,
> Tom

Hi Tom.

Yes, Eyal Perry  reported the issue a while back, we plan to send
upstream patches to fix the issue today.

Make sure you tweaked coalescing parameters if you use mlx4 :

ethtool -C eth0 tx-usecs 4 tx-frames 4

Otherwise, the NIC accumulates too many packets before sending an
interrupt for TX completion.

Then you can either wait the patches, or try the one I sent :
http://permalink.gmane.org/gmane.linux.network/347023
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html