netdev - Re: Performance regression between 4.13 and 4.14

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <3db3e973-67be-781e-5550-33eb82b7ed1e@candelatech.com>
Date:   Wed, 9 May 2018 13:55:49 -0700
From:   Ben Greear <greearb@...delatech.com>
To:     Eric Dumazet <eric.dumazet@...il.com>,
        netdev <netdev@...r.kernel.org>
Subject: Re: Performance regression between 4.13 and 4.14

On 05/09/2018 12:02 PM, Ben Greear wrote:
> On 05/09/2018 11:48 AM, Eric Dumazet wrote:
>>
>>
>> On 05/09/2018 11:43 AM, Ben Greear wrote:
>>> On 05/08/2018 10:10 AM, Eric Dumazet wrote:
>>>>
>>>>
>>>> On 05/08/2018 09:44 AM, Ben Greear wrote:
>>>>> Hello,
>>>>>
>>>>> I am trying to track down a performance regression that appears to be between 4.13
>>>>> and 4.14.
>>>>>
>>>>> I first saw the problem with a hacked version of pktgen on some ixgbe NICs.  4.13 can do
>>>>> right at 10G bi-directional on two ports, and 4.14 and later can do only about 6Gbps.
>>>>>
>>>>> I also tried with user-space UDP traffic on a stock kernel, and I can get about 3.2Gbps combined tx+rx
>>>>> on 4.14 and about 4.4Gbps on 4.13.
>>>>>
>>>>> Attempting to bisect seems to be triggering a weirdness in git, and also lots of commits
>>>>> crash or do not bring up networking, which makes the bisect difficult.
>>>>>
>>>>> Looking at perf top, it would appear that some lock is probably to blame.
>>>>
>>>>
>>>> perf record -a -g -e cycles:pp sleep 5
>>>> perf report
>>>>
>>>> Then you'll be able to tell us which lock (or call graph) is killing your perf.
>>>>
>>>
>>> I seem to be chasing multiple issues.  For 4.13, at least part of my problem was that LOCKDEP was enabled,
>>> during my bisect, though it does NOT appear enabled in 4.16.  I think maybe CONFIG_LOCKDEP moved to CONFIG_PROVE_LOCKING
>>> in 4.16, or something like that?  My 4.16 .config does have CONFIG_LOCKDEP_SUPPORT enabled, and I see no option to disable it:
>>>
>>> [greearb@...-dt3 linux-4.16.x64]$ grep LOCKDEP .config
>>> CONFIG_LOCKDEP_SUPPORT=y
>>>
>>>
>>> For 4.16, I am disabling RETRAMPOLINE...are there any other such things I need
>>> to disable to keep from getting a performance hit from the spectre-related bug
>>> fixes?  At this point, I do not care about the security implications.
>>>
>>> greearb@...-dt3 linux-4.16.x64]$ grep RETPO .config
>>> # CONFIG_RETPOLINE is not set
>>>
>>>
>>> Thanks,
>>> Ben
>>>
>>
>> No idea really, you mention a 4.13 -> 4.14 regression and jump then to 4.16 :/
>
> I initially saw the problem in 4.16, then bisected, and 4.14 still showed the
> issue.

So, I guess I must have been enabling lockdep the whole time.  This __lock_acquire
is from lockdep as far as I can tell, not normal locking.  I re-built 4.16 after
verifying as best as I could that lockdep was not enabled, and now it performs
as expected.

I'm going to test a patch to change __lock_acquire to __lock_acquire_lockdep so
maybe someone else will not make the same mistake I made.

> +   17.78%    17.78%  kpktgend_1       [kernel.kallsyms]             [k] __lock_acquire.isra.3


Thanks,
Ben


-- 
Ben Greear <greearb@...delatech.com>
Candela Technologies Inc  http://www.candelatech.com