netdev - RE: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <BN3PR0501MB1425A937390695B738419E9CB0800@BN3PR0501MB1425.namprd05.prod.outlook.com>
Date:   Wed, 2 May 2018 21:47:14 +0000
From:   Michael Wenig <mwenig@...are.com>
To:     Eric Dumazet <eric.dumazet@...il.com>,
        Ben Greear <greearb@...delatech.com>,
        Steven Rostedt <rostedt@...dmis.org>
CC:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Shilpi Agarwal <sagarwal@...are.com>,
        Boon Ang <bang@...are.com>, Darren Hart <dvhart@...are.com>,
        Steven Rostedt <srostedt@...are.com>,
        Abdul Anshad Azeez <aazees@...are.com>,
        Rajender M <manir@...are.com>,
        Michael Wenig <mwenig@...are.com>
Subject: RE: Performance regressions in TCP_STREAM tests in Linux 4.15 (and
 later)

After applying Eric's proposed change (see below) to a 4.17 RC3 kernel, the regressions that we had observed in our TCP_STREAM small message tests with TCP_NODELAY enabled are now drastically reduced. Instead of the original 3x thruput and cpu cost regressions, the regression depth is now < 10% for thruput and between 10% - 20% for cpu cost. The improvements in the TCP_RR tests that we had observed after Eric's original commit are not impacted by the change. It would be great if this change could make it into a patch.

Michael Wenig
VMware Performance Engineering 

-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@...il.com] 
Sent: Monday, April 30, 2018 10:48 AM
To: Ben Greear <greearb@...delatech.com>; Steven Rostedt <rostedt@...dmis.org>; Michael Wenig <mwenig@...are.com>
Cc: netdev@...r.kernel.org; Shilpi Agarwal <sagarwal@...are.com>; Boon Ang <bang@...are.com>; Darren Hart <dvhart@...are.com>; Steven Rostedt <srostedt@...are.com>; Abdul Anshad Azeez <aazees@...are.com>
Subject: Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later)



On 04/30/2018 09:36 AM, Eric Dumazet wrote:
> 
> 
> On 04/30/2018 09:14 AM, Ben Greear wrote:
>> On 04/27/2018 08:11 PM, Steven Rostedt wrote:
>>>
>>> We'd like this email archived in netdev list, but since netdev is 
>>> notorious for blocking outlook email as spam, it didn't go through. 
>>> So I'm replying here to help get it into the archives.
>>>
>>> Thanks!
>>>
>>> -- Steve
>>>
>>>
>>> On Fri, 27 Apr 2018 23:05:46 +0000
>>> Michael Wenig <mwenig@...are.com> wrote:
>>>
>>>> As part of VMware's performance testing with the Linux 4.15 kernel, 
>>>> we identified CPU cost and throughput regressions when comparing to 
>>>> the Linux 4.14 kernel. The impacted test cases are mostly 
>>>> TCP_STREAM send tests when using small message sizes. The 
>>>> regressions are significant (up 3x) and were tracked down to be a 
>>>> side effect of Eric Dumazat's RB tree changes that went into the Linux 4.15 kernel.
>>>> Further investigation showed our use of the TCP_NODELAY flag in 
>>>> conjunction with Eric's change caused the regressions to show and 
>>>> simply disabling TCP_NODELAY brought performance back to normal.
>>>> Eric's change also resulted into significant improvements in our 
>>>> TCP_RR test cases.
>>>>
>>>>
>>>>
>>>> Based on these results, our theory is that Eric's change made the 
>>>> system overall faster (reduced latency) but as a side effect less 
>>>> aggregation is happening (with TCP_NODELAY) and that results in 
>>>> lower throughput. Previously even though TCP_NODELAY was set, 
>>>> system was slower and we still got some benefit of aggregation. 
>>>> Aggregation helps in better efficiency and higher throughput 
>>>> although it can increase the latency. If you are seeing a 
>>>> regression in your application throughput after this change, using 
>>>> TCP_NODELAY might help bring performance back however that might increase latency.
>>
>> I guess you mean _disabling_ TCP_NODELAY instead of _using_ TCP_NODELAY?
>>
> 
> Yeah, I guess auto-corking does not work as intended.

I would try the following patch :

diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 44be7f43455e4aefde8db61e2d941a69abcc642a..c9d00ef54deca15d5760bcbe154001a96fa1e2a7 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -697,7 +697,7 @@ static bool tcp_should_autocork(struct sock *sk, struct sk_buff *skb,  {
        return skb->len < size_goal &&
               sock_net(sk)->ipv4.sysctl_tcp_autocorking &&
-              skb != tcp_write_queue_head(sk) &&
+              !tcp_rtx_queue_empty(sk) &&
               refcount_read(&sk->sk_wmem_alloc) > skb->truesize;  }