Message-ID: <45f90875-d4b0-4cea-b857-752bc93fd48b@redhat.com>
Date: Thu, 13 Feb 2025 15:06:16 -0500
From: Waiman Long <llong@...hat.com>
To: Marco Elver <elver@...gle.com>, Boqun Feng <boqun.feng@...il.com>
Cc: Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>,
 Will Deacon <will.deacon@....com>, linux-kernel@...r.kernel.org,
 Andrey Ryabinin <ryabinin.a.a@...il.com>,
 Alexander Potapenko <glider@...gle.com>,
 Andrey Konovalov <andreyknvl@...il.com>, Dmitry Vyukov <dvyukov@...gle.com>,
 Vincenzo Frascino <vincenzo.frascino@....com>, kasan-dev@...glegroups.com
Subject: Re: [PATCH v3 3/3] locking/lockdep: Disable KASAN instrumentation of
 lockdep.c

On 2/12/25 11:57 AM, Waiman Long wrote:
> On 2/12/25 6:30 AM, Marco Elver wrote:
>> On Wed, 12 Feb 2025 at 06:57, Boqun Feng <boqun.feng@...il.com> wrote:
>>> [Cc KASAN]
>>>
>>> A Reviewed-by or Acked-by from KASAN would be nice, thanks!
>>>
>>> Regards,
>>> Boqun
>>>
>>> On Sun, Feb 09, 2025 at 11:26:12PM -0500, Waiman Long wrote:
>>>> Both KASAN and LOCKDEP are commonly enabled in building a debug 
>>>> kernel.
>>>> Each of them can significantly slow down the speed of a debug kernel.
>>>> Enabling KASAN instrumentation of the LOCKDEP code will further slow
>>>> thing down.
>>>>
>>>> Since LOCKDEP is a high overhead debugging tool, it will never get
>>>> enabled in a production kernel. The LOCKDEP code is also pretty mature
>>>> and is unlikely to get major changes. There is also a possibility of
>>>> recursion similar to KCSAN.
>>>>
>>>> To evaluate the performance impact of disabling KASAN instrumentation
>>>> of lockdep.c, the time to do a parallel build of the Linux defconfig
>>>> kernel was used as the benchmark. Two x86-64 systems (Skylake & Zen 2)
>>>> and an arm64 system were used as test beds. Two sets of non-RT and RT
>>>> kernels with similar configurations except mainly CONFIG_PREEMPT_RT
>>>> were used for evaulation.
>>>>
>>>> For the Skylake system:
>>>>
>>>>    Kernel                      Run time            Sys time
>>>>    ------                      --------            --------
>>>>    Non-debug kernel (baseline) 0m47.642s 4m19.811s
>>>>    Debug kernel                        2m11.108s (x2.8) 38m20.467s 
>>>> (x8.9)
>>>>    Debug kernel (patched)      1m49.602s (x2.3) 31m28.501s (x7.3)
>>>>    Debug kernel
>>>>    (patched + mitigations=off)         1m30.988s (x1.9) 26m41.993s 
>>>> (x6.2)
>>>>
>>>>    RT kernel (baseline)                0m54.871s 7m15.340s
>>>>    RT debug kernel             6m07.151s (x6.7) 135m47.428s (x18.7)
>>>>    RT debug kernel (patched)   3m42.434s (x4.1) 74m51.636s (x10.3)
>>>>    RT debug kernel
>>>>    (patched + mitigations=off)         2m40.383s (x2.9) 57m54.369s 
>>>> (x8.0)
>>>>
>>>> For the Zen 2 system:
>>>>
>>>>    Kernel                      Run time            Sys time
>>>>    ------                      --------            --------
>>>>    Non-debug kernel (baseline) 1m42.806s 39m48.714s
>>>>    Debug kernel                        4m04.524s (x2.4) 125m35.904s 
>>>> (x3.2)
>>>>    Debug kernel (patched)      3m56.241s (x2.3) 127m22.378s (x3.2)
>>>>    Debug kernel
>>>>    (patched + mitigations=off)         2m38.157s (x1.5) 92m35.680s 
>>>> (x2.3)
>>>>
>>>>    RT kernel (baseline)                 1m51.500s 14m56.322s
>>>>    RT debug kernel             16m04.962s (x8.7) 244m36.463s (x16.4)
>>>>    RT debug kernel (patched)    9m09.073s (x4.9) 129m28.439s (x8.7)
>>>>    RT debug kernel
>>>>    (patched + mitigations=off)          3m31.662s (x1.9) 51m01.391s 
>>>> (x3.4)
>>>>
>>>> For the arm64 system:
>>>>
>>>>    Kernel                      Run time            Sys time
>>>>    ------                      --------            --------
>>>>    Non-debug kernel (baseline) 1m56.844s 8m47.150s
>>>>    Debug kernel                        3m54.774s (x2.0) 92m30.098s 
>>>> (x10.5)
>>>>    Debug kernel (patched)      3m32.429s (x1.8) 77m40.779s (x8.8)
>>>>
>>>>    RT kernel (baseline)                 4m01.641s 18m16.777s
>>>>    RT debug kernel             19m32.977s (x4.9) 304m23.965s (x16.7)
>>>>    RT debug kernel (patched)   16m28.354s (x4.1) 234m18.149s (x12.8)
>>>>
>>>> Turning the mitigations off doesn't seems to have any noticeable 
>>>> impact
>>>> on the performance of the arm64 system. So the mitigation=off entries
>>>> aren't included.
>>>>
>>>> For the x86 CPUs, cpu mitigations has a much bigger impact on
>>>> performance, especially the RT debug kernel. The SRSO mitigation in
>>>> Zen 2 has an especially big impact on the debug kernel. It is also the
>>>> majority of the slowdown with mitigations on. It is because the 
>>>> patched
>>>> ret instruction slows down function returns. A lot of helper functions
>>>> that are normally compiled out or inlined may become real function
>>>> calls in the debug kernel. The KASAN instrumentation inserts a lot
>>>> of __asan_loadX*() and __kasan_check_read() function calls to memory
>>>> access portion of the code. The lockdep's __lock_acquire() function,
>>>> for instance, has 66 __asan_loadX*() and 6 __kasan_check_read() calls
>>>> added with KASAN instrumentation. Of course, the actual numbers may 
>>>> vary
>>>> depending on the compiler used and the exact version of the lockdep 
>>>> code.
>> For completeness-sake, we'd also have to compare with
>> CONFIG_KASAN_INLINE=y, which gets rid of the __asan_ calls (not the
>> explicit __kasan_ checks). But I leave it up to you - I'm aware it
>> results in slow-downs, too. ;-)

That is not correct. Setting CONFIG_KASAN_INLINE=y does have an effect 
in lockdep.c to reduce the number of __asan_* calls. I have posted the 
v4 series with the updated test results. I have also added a new patch 
to KASAN checking in lock_acquire().

Cheers,
Longman

Powered by blists - more mailing lists