linux-kernel - Re: [Xen-devel] "tcp: refine TSO autosizing" causes performance regression on Xen

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <552E9E8D.1080000@eu.citrix.com>
Date:	Wed, 15 Apr 2015 18:23:25 +0100
From:	George Dunlap <george.dunlap@...citrix.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
CC:	Jonathan Davies <Jonathan.Davies@...rix.com>,
	"xen-devel@...ts.xensource.com" <xen-devel@...ts.xensource.com>,
	Wei Liu <wei.liu2@...rix.com>,
	Ian Campbell <Ian.Campbell@...rix.com>,
	"Stefano Stabellini" <stefano.stabellini@...citrix.com>,
	netdev <netdev@...r.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Eric Dumazet <edumazet@...gle.com>,
	"Paul Durrant" <paul.durrant@...rix.com>,
	Christoffer Dall <christoffer.dall@...aro.org>,
	Felipe Franciosi <felipe.franciosi@...rix.com>,
	<linux-arm-kernel@...ts.infradead.org>,
	"David Vrabel" <david.vrabel@...rix.com>
Subject: Re: [Xen-devel] "tcp: refine TSO autosizing" causes performance regression
 on Xen

On 04/15/2015 05:38 PM, Eric Dumazet wrote:
> My thoughts that instead of these long talks you should guys read the
> code :
> 
>                 /* TCP Small Queues :
>                  * Control number of packets in qdisc/devices to two packets / or ~1 ms.
>                  * This allows for :
>                  *  - better RTT estimation and ACK scheduling
>                  *  - faster recovery
>                  *  - high rates
>                  * Alas, some drivers / subsystems require a fair amount
>                  * of queued bytes to ensure line rate.
>                  * One example is wifi aggregation (802.11 AMPDU)
>                  */
>                 limit = max(2 * skb->truesize, sk->sk_pacing_rate >> 10);
>                 limit = min_t(u32, limit, sysctl_tcp_limit_output_bytes);
> 
> 
> Then you'll see that most of your questions are already answered.
> 
> Feel free to try to improve the behavior, if it does not hurt critical workloads
> like TCP_RR, where we we send very small messages, millions times per second.

First of all, with regard to critical workloads, once this patch gets
into distros, *normal TCP streams* on every VM running on Amazon,
Rackspace, Linode, &c will get a 30% hit in performance *by default*.
Normal TCP streams on xennet *are* a critical workload, and deserve the
same kind of accommodation as TCP_RR (if not more).  The same goes for
virtio_net.

Secondly, according to Stefano's and Jonathan's tests,
tcp_limit_output_bytes completely fixes the problem for Xen.

Which means that max(2*skb->truesize, sk->sk_pacing_rate >>10) is
*already* larger for Xen; that calculation mentioned in the comment is
*already* doing the right thing.

As Jonathan pointed out, sysctl_tcp_limit_output_bytes is overriding an
automatic TSQ calculation which is actually choosing an effective value
for xennet.

It certainly makes sense for sysctl_tcp_limit_output_bytes to be an
actual maximum limit.  I went back and looked at the original patch
which introduced it (46d3ceabd), and it looks to me like it was designed
to be a rough, quick estimate of "two packets outstanding" (by choosing
the maximum size of the packet, 64k, and multiplying it by two).

Now that you have a better algorithm -- the size of 2 actual packets or
the amount transmitted in 1ms -- it seems like the default
sysctl_tcp_limit_output_bytes should be higher, and let the automatic
TSQ you have on the first line throttle things down when necessary.

 -George
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/