netdev - Re: "tcp: refine TSO autosizing" causes performance regression on Xen

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.DEB.2.02.1504091729160.7690@kaball.uk.xensource.com>
Date:	Thu, 9 Apr 2015 17:36:47 +0100
From:	Stefano Stabellini <stefano.stabellini@...citrix.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
CC:	Stefano Stabellini <stefano.stabellini@...citrix.com>,
	<xen-devel@...ts.xensource.com>,
	<linux-arm-kernel@...ts.infradead.org>,
	<linux-kernel@...r.kernel.org>, <christoffer.dall@...aro.org>,
	Ian Campbell <Ian.Campbell@...rix.com>,
	Wei Liu <wei.liu2@...rix.com>,
	David Vrabel <david.vrabel@...rix.com>, <edumazet@...gle.com>,
	Konrad Rzeszutek Wilk <konrad.wilk@...cle.com>,
	netdev <netdev@...r.kernel.org>
Subject: Re: "tcp: refine TSO autosizing" causes performance regression on
 Xen

On Thu, 9 Apr 2015, Eric Dumazet wrote:
> On Thu, 2015-04-09 at 16:46 +0100, Stefano Stabellini wrote:
> > Hi all,
> > 
> > I found a performance regression when running netperf -t TCP_MAERTS from
> > an external host to a Xen VM on ARM64: v3.19 and v4.0-rc4 running in the
> > virtual machine are 30% slower than v3.18.
> > 
> > Through bisection I found that the perf regression is caused by the
> > prensence of the following commit in the guest kernel:
> > 
> > 
> > commit 605ad7f184b60cfaacbc038aa6c55ee68dee3c89
> > Author: Eric Dumazet <edumazet@...gle.com>
> > Date:   Sun Dec 7 12:22:18 2014 -0800
> > 
> >     tcp: refine TSO autosizing
> > 
> > 
> > A simple revert would fix the issue.
> > 
> > Does anybody have any ideas on what could be the cause of the problem?
> > Suggestions on what to do to fix it?
> 
> You sent this to lkml while networking discussions are on netdev.
> 
> This topic had been discussed on netdev multiple times.

Sorry, and many thanks for the quick reply!


> This commit restored original TCP Small Queue behavior, which is the
> first step to fight bufferbloat.
> 
> Some network drivers are known to be problematic because of a delayed TX
> completion.
> 
> So far this commit did not impact max single flow throughput on 40Gb
> mlx4 NIC. (ie : line rate is possible)
> 
> Try to tweak /proc/sys/net/ipv4/tcp_limit_output_bytes to see if it
> makes a difference ?

A very big difference:

echo 262144 > /proc/sys/net/ipv4/tcp_limit_output_bytes
brings us much closer to the original performance, the slowdown is just
8%

echo 1048576 > /proc/sys/net/ipv4/tcp_limit_output_bytes
fills the gap entirely, same performance as before "refine TSO
autosizing"


What would be the next step for here?  Should I just document this as an
important performance tweaking step for Xen, or is there something else
we can do?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html