netdev - Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180427231149.119db14c@vmware.local.home>
Date:   Fri, 27 Apr 2018 23:11:49 -0400
From:   Steven Rostedt <rostedt@...dmis.org>
To:     Michael Wenig <mwenig@...are.com>
Cc:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "eric.dumazet@...il.com" <eric.dumazet@...il.com>,
        Shilpi Agarwal <sagarwal@...are.com>,
        Boon Ang <bang@...are.com>, Darren Hart <dvhart@...are.com>,
        Steven Rostedt <srostedt@...are.com>,
        Abdul Anshad Azeez <aazees@...are.com>
Subject: Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and
 later)


We'd like this email archived in netdev list, but since netdev is
notorious for blocking outlook email as spam, it didn't go through. So
I'm replying here to help get it into the archives.

Thanks!

-- Steve


On Fri, 27 Apr 2018 23:05:46 +0000
Michael Wenig <mwenig@...are.com> wrote:

> As part of VMware's performance testing with the Linux 4.15 kernel,
> we identified CPU cost and throughput regressions when comparing to
> the Linux 4.14 kernel. The impacted test cases are mostly TCP_STREAM
> send tests when using small message sizes. The regressions are
> significant (up 3x) and were tracked down to be a side effect of Eric
> Dumazat's RB tree changes that went into the Linux 4.15 kernel.
> Further investigation showed our use of the TCP_NODELAY flag in
> conjunction with Eric's change caused the regressions to show and
> simply disabling TCP_NODELAY brought performance back to normal.
> Eric's change also resulted into significant improvements in our
> TCP_RR test cases.
> 
> 
> 
> Based on these results, our theory is that Eric's change made the
> system overall faster (reduced latency) but as a side effect less
> aggregation is happening (with TCP_NODELAY) and that results in lower
> throughput. Previously even though TCP_NODELAY was set, system was
> slower and we still got some benefit of aggregation. Aggregation
> helps in better efficiency and higher throughput although it can
> increase the latency. If you are seeing a regression in your
> application throughput after this change, using TCP_NODELAY might
> help bring performance back however that might increase latency.
> 
> 
> 
> As such, we are not asking for a fix but simply want to document for
> others what we have found.
> 
> 
> 
> Michael Wenig
> 
> Performance Engineering
> 
> VMware, Inc.
>