netdev - Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <e97b2ebd-0b47-82e5-1f5f-d510db6eacd0@gmail.com>
Date:   Wed, 2 May 2018 15:41:17 -0700
From:   Eric Dumazet <eric.dumazet@...il.com>
To:     Michael Wenig <mwenig@...are.com>,
        Eric Dumazet <eric.dumazet@...il.com>,
        Ben Greear <greearb@...delatech.com>,
        Steven Rostedt <rostedt@...dmis.org>
Cc:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Shilpi Agarwal <sagarwal@...are.com>,
        Boon Ang <bang@...are.com>, Darren Hart <dvhart@...are.com>,
        Steven Rostedt <srostedt@...are.com>,
        Abdul Anshad Azeez <aazees@...are.com>,
        Rajender M <manir@...are.com>
Subject: Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and
 later)



On 05/02/2018 02:47 PM, Michael Wenig wrote:
> After applying Eric's proposed change (see below) to a 4.17 RC3 kernel, the regressions that we had observed in our TCP_STREAM small message tests with TCP_NODELAY enabled are now drastically reduced. Instead of the original 3x thruput and cpu cost regressions, the regression depth is now < 10% for thruput and between 10% - 20% for cpu cost. The improvements in the TCP_RR tests that we had observed after Eric's original commit are not impacted by the change. It would be great if this change could make it into a patch.
> 


Thanks for a lot testing, I will submit this patch after more tests from my side.



> Michael Wenig
> VMware Performance Engineering 
> 
> -----Original Message-----
> From: Eric Dumazet [mailto:eric.dumazet@...il.com] 
> Sent: Monday, April 30, 2018 10:48 AM
> To: Ben Greear <greearb@...delatech.com>; Steven Rostedt <rostedt@...dmis.org>; Michael Wenig <mwenig@...are.com>
> Cc: netdev@...r.kernel.org; Shilpi Agarwal <sagarwal@...are.com>; Boon Ang <bang@...are.com>; Darren Hart <dvhart@...are.com>; Steven Rostedt <srostedt@...are.com>; Abdul Anshad Azeez <aazees@...are.com>
> Subject: Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later)
> 
> 
> 
> On 04/30/2018 09:36 AM, Eric Dumazet wrote:
>>
>>
>> On 04/30/2018 09:14 AM, Ben Greear wrote:
>>> On 04/27/2018 08:11 PM, Steven Rostedt wrote:
>>>>
>>>> We'd like this email archived in netdev list, but since netdev is 
>>>> notorious for blocking outlook email as spam, it didn't go through. 
>>>> So I'm replying here to help get it into the archives.
>>>>
>>>> Thanks!
>>>>
>>>> -- Steve
>>>>
>>>>
>>>> On Fri, 27 Apr 2018 23:05:46 +0000
>>>> Michael Wenig <mwenig@...are.com> wrote:
>>>>
>>>>> As part of VMware's performance testing with the Linux 4.15 kernel, 
>>>>> we identified CPU cost and throughput regressions when comparing to 
>>>>> the Linux 4.14 kernel. The impacted test cases are mostly 
>>>>> TCP_STREAM send tests when using small message sizes. The 
>>>>> regressions are significant (up 3x) and were tracked down to be a 
>>>>> side effect of Eric Dumazat's RB tree changes that went into the Linux 4.15 kernel.
>>>>> Further investigation showed our use of the TCP_NODELAY flag in 
>>>>> conjunction with Eric's change caused the regressions to show and 
>>>>> simply disabling TCP_NODELAY brought performance back to normal.
>>>>> Eric's change also resulted into significant improvements in our 
>>>>> TCP_RR test cases.
>>>>>
>>>>>
>>>>>
>>>>> Based on these results, our theory is that Eric's change made the 
>>>>> system overall faster (reduced latency) but as a side effect less 
>>>>> aggregation is happening (with TCP_NODELAY) and that results in 
>>>>> lower throughput. Previously even though TCP_NODELAY was set, 
>>>>> system was slower and we still got some benefit of aggregation. 
>>>>> Aggregation helps in better efficiency and higher throughput 
>>>>> although it can increase the latency. If you are seeing a 
>>>>> regression in your application throughput after this change, using 
>>>>> TCP_NODELAY might help bring performance back however that might increase latency.
>>>
>>> I guess you mean _disabling_ TCP_NODELAY instead of _using_ TCP_NODELAY?
>>>
>>
>> Yeah, I guess auto-corking does not work as intended.
> 
> I would try the following patch :
> 
> diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 44be7f43455e4aefde8db61e2d941a69abcc642a..c9d00ef54deca15d5760bcbe154001a96fa1e2a7 100644
> --- a/net/ipv4/tcp.c
> +++ b/net/ipv4/tcp.c
> @@ -697,7 +697,7 @@ static bool tcp_should_autocork(struct sock *sk, struct sk_buff *skb,  {
>         return skb->len < size_goal &&
>                sock_net(sk)->ipv4.sysctl_tcp_autocorking &&
> -              skb != tcp_write_queue_head(sk) &&
> +              !tcp_rtx_queue_empty(sk) &&
>                refcount_read(&sk->sk_wmem_alloc) > skb->truesize;  }
>  
>