netdev - Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <7e1f00ad-d859-0aab-c953-d6da2efe11f0@gmail.com>
Date:   Mon, 30 Apr 2018 09:36:00 -0700
From:   Eric Dumazet <eric.dumazet@...il.com>
To:     Ben Greear <greearb@...delatech.com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Michael Wenig <mwenig@...are.com>
Cc:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Shilpi Agarwal <sagarwal@...are.com>,
        Boon Ang <bang@...are.com>, Darren Hart <dvhart@...are.com>,
        Steven Rostedt <srostedt@...are.com>,
        Abdul Anshad Azeez <aazees@...are.com>
Subject: Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and
 later)



On 04/30/2018 09:14 AM, Ben Greear wrote:
> On 04/27/2018 08:11 PM, Steven Rostedt wrote:
>>
>> We'd like this email archived in netdev list, but since netdev is
>> notorious for blocking outlook email as spam, it didn't go through. So
>> I'm replying here to help get it into the archives.
>>
>> Thanks!
>>
>> -- Steve
>>
>>
>> On Fri, 27 Apr 2018 23:05:46 +0000
>> Michael Wenig <mwenig@...are.com> wrote:
>>
>>> As part of VMware's performance testing with the Linux 4.15 kernel,
>>> we identified CPU cost and throughput regressions when comparing to
>>> the Linux 4.14 kernel. The impacted test cases are mostly TCP_STREAM
>>> send tests when using small message sizes. The regressions are
>>> significant (up 3x) and were tracked down to be a side effect of Eric
>>> Dumazat's RB tree changes that went into the Linux 4.15 kernel.
>>> Further investigation showed our use of the TCP_NODELAY flag in
>>> conjunction with Eric's change caused the regressions to show and
>>> simply disabling TCP_NODELAY brought performance back to normal.
>>> Eric's change also resulted into significant improvements in our
>>> TCP_RR test cases.
>>>
>>>
>>>
>>> Based on these results, our theory is that Eric's change made the
>>> system overall faster (reduced latency) but as a side effect less
>>> aggregation is happening (with TCP_NODELAY) and that results in lower
>>> throughput. Previously even though TCP_NODELAY was set, system was
>>> slower and we still got some benefit of aggregation. Aggregation
>>> helps in better efficiency and higher throughput although it can
>>> increase the latency. If you are seeing a regression in your
>>> application throughput after this change, using TCP_NODELAY might
>>> help bring performance back however that might increase latency.
> 
> I guess you mean _disabling_ TCP_NODELAY instead of _using_ TCP_NODELAY?
>

Yeah, I guess auto-corking does not work as intended.