netdev - Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180430123136.59caaf81@gandalf.local.home>
Date:   Mon, 30 Apr 2018 12:31:36 -0400
From:   Steven Rostedt <rostedt@...dmis.org>
To:     Ben Greear <greearb@...delatech.com>
Cc:     Michael Wenig <mwenig@...are.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "eric.dumazet@...il.com" <eric.dumazet@...il.com>,
        Shilpi Agarwal <sagarwal@...are.com>,
        Boon Ang <bang@...are.com>, Darren Hart <dvhart@...are.com>,
        Abdul Anshad Azeez <aazees@...are.com>
Subject: Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and
 later)

On Mon, 30 Apr 2018 09:14:04 -0700
Ben Greear <greearb@...delatech.com> wrote:

> >> As part of VMware's performance testing with the Linux 4.15 kernel,
> >> we identified CPU cost and throughput regressions when comparing to
> >> the Linux 4.14 kernel. The impacted test cases are mostly TCP_STREAM
> >> send tests when using small message sizes. The regressions are
> >> significant (up 3x) and were tracked down to be a side effect of Eric
> >> Dumazat's RB tree changes that went into the Linux 4.15 kernel.
> >> Further investigation showed our use of the TCP_NODELAY flag in
> >> conjunction with Eric's change caused the regressions to show and
> >> simply disabling TCP_NODELAY brought performance back to normal.
> >> Eric's change also resulted into significant improvements in our
> >> TCP_RR test cases.
> >>
> >>
> >>
> >> Based on these results, our theory is that Eric's change made the
> >> system overall faster (reduced latency) but as a side effect less
> >> aggregation is happening (with TCP_NODELAY) and that results in lower
> >> throughput. Previously even though TCP_NODELAY was set, system was
> >> slower and we still got some benefit of aggregation. Aggregation
> >> helps in better efficiency and higher throughput although it can
> >> increase the latency. If you are seeing a regression in your
> >> application throughput after this change, using TCP_NODELAY might
> >> help bring performance back however that might increase latency.  
> 
> I guess you mean _disabling_ TCP_NODELAY instead of _using_ TCP_NODELAY?

Yes, thank you for catching that.

-- Steve