Message-ID: <4d31e1b4-2113-c557-b60a-3a45b2840f26@loongson.cn>
Date: Thu, 1 Aug 2024 16:00:19 +0800
From: maobibo <maobibo@...ngson.cn>
To: Chen Yu <yu.c.chen@...el.com>, Dave Hansen <dave.hansen@...ux.intel.com>,
 Thomas Gleixner <tglx@...utronix.de>, Ingo Molnar <mingo@...hat.com>,
 Borislav Petkov <bp@...en8.de>
Cc: "H. Peter Anvin" <hpa@...or.com>, Arnd Bergmann <arnd@...db.de>,
 virtualization@...ts.linux.dev, linux-kernel@...r.kernel.org,
 Juergen Gross <jgross@...e.com>, Nikolay Borisov <nik.borisov@...e.com>,
 Qiuxu Zhuo <qiuxu.zhuo@...el.com>, Prem Nath Dey <prem.nath.dey@...el.com>,
 Xiaoping Zhou <xiaoping.zhou@...el.com>
Subject: Re: [PATCH v4] x86/paravirt: Disable virt spinlock on bare metal

Chenyu,

I do not know much about x86, just give some comments(probably 
incorrected) from the code.

On 2024/7/29 下午2:52, Chen Yu wrote:
> The kernel can change spinlock behavior when running as a guest. But
> this guest-friendly behavior causes performance problems on bare metal.
> So there's a 'virt_spin_lock_key' static key to switch between the two
> modes.
> 
> In current code, the static key is always enabled by default when
> running in guest mode. The key is disabled for bare metal (and in
> some guests that want native behavior).
> 
> Large performance regression is reported when running encode/decode
> workload and BenchSEE cache sub-workload on the bare metal.
> Bisect points to commit ce0a1b608bfc ("x86/paravirt: Silence unused
> native_pv_lock_init() function warning"). When CONFIG_PARAVIRT_SPINLOCKS
> is disabled, the virt_spin_lock_key is incorrectly set to true on bare
> metal. The qspinlock degenerates to test-and-set spinlock, which
> decrease the performance on bare metal.
> 
> Set the default value of virt_spin_lock_key to false. If booting in
> a VM, enable this key. Later during the VM initialization, if other
> high-efficient spinlock is detected(paravirt-spinlock eg), the
> virt_spin_lock_key is disabled. According to the description above,
> the final effect will be as followed:
> 
> X86_FEATURE_HYPERVISOR         Y    Y    Y     N
> CONFIG_PARAVIRT_SPINLOCKS      Y    Y    N     Y/N
> PV spinlock                    Y    N    N     Y/N
> 
> virt_spin_lock_key             N    N    Y     N
> 
> To summarize, the virt_spin_lock_key is disabled on the bare metal
> no matter what other condidtion is. And the virt_spin_lock_key is
> also disabled when other spinlock mechanism is detected in the VM
> guest.
> 
> Fixes: ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning")
> Suggested-by: Dave Hansen <dave.hansen@...ux.intel.com>
> Suggested-by: Qiuxu Zhuo <qiuxu.zhuo@...el.com>
> Suggested-by: Nikolay Borisov <nik.borisov@...e.com>
> Reported-by: Prem Nath Dey <prem.nath.dey@...el.com>
> Reported-by: Xiaoping Zhou <xiaoping.zhou@...el.com>
> Reviewed-by: Nikolay Borisov <nik.borisov@...e.com>
> Signed-off-by: Chen Yu <yu.c.chen@...el.com>
> ---
> v3->v4:
>    Refine the commit log.
>    Added Reviewed-by tag from Nikolay.
> v2->v3:
>    Change the default value of virt_spin_lock_key from true to false.
>    Enable this key when it is in the VM, and disable it when needed.
>    This makes the code more readable. (Nikolay Borisov)
>    Dropped Reviewed-by because the code has been changed.
> v1->v2:
>    Refine the commit log per Dave's suggestion.
>    Simplify the fix by directly disabling the virt_spin_lock_key on bare metal.
>    Collect Reviewed-by from Juergen.
> ---
>   arch/x86/include/asm/qspinlock.h | 4 ++--
>   arch/x86/kernel/paravirt.c       | 7 +++----
>   2 files changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h
> index a053c1293975..a32bd2aabdf9 100644
> --- a/arch/x86/include/asm/qspinlock.h
> +++ b/arch/x86/include/asm/qspinlock.h
> @@ -66,13 +66,13 @@ static inline bool vcpu_is_preempted(long cpu)
>   
>   #ifdef CONFIG_PARAVIRT
>   /*
> - * virt_spin_lock_key - enables (by default) the virt_spin_lock() hijack.
> + * virt_spin_lock_key - disables (by default) the virt_spin_lock() hijack.
>    *
>    * Native (and PV wanting native due to vCPU pinning) should disable this key.
>    * It is done in this backwards fashion to only have a single direction change,
>    * which removes ordering between native_pv_spin_init() and HV setup.
>    */
> -DECLARE_STATIC_KEY_TRUE(virt_spin_lock_key);
> +DECLARE_STATIC_KEY_FALSE(virt_spin_lock_key);

@@ -87,7 +87,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock)
  {
         int val;

-       if (!static_branch_likely(&virt_spin_lock_key))
+       if (!static_branch_unlikely(&virt_spin_lock_key))
                 return false;

Do we need change it with static_branch_unlikely() if value of key is 
false by fault?
>   
>   /*
>    * Shortcut for the queued_spin_lock_slowpath() function that allows
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index 5358d43886ad..fec381533555 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text);
>   DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);
>   #endif
>   
> -DEFINE_STATIC_KEY_TRUE(virt_spin_lock_key);
> +DEFINE_STATIC_KEY_FALSE(virt_spin_lock_key);
>   
>   void __init native_pv_lock_init(void)
>   {
> -	if (IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) &&
> -	    !boot_cpu_has(X86_FEATURE_HYPERVISOR))
> -		static_branch_disable(&virt_spin_lock_key);
> +	if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
> +		static_branch_enable(&virt_spin_lock_key);
>   }

 From my point, the sentence static_branch_disable(&virt_spin_lock_key) 
can be removed in file arch/x86/xen/spinlock.c and 
arch/x86/kernel/kvm.c, since function native_smp_prepare_boot_cpu() is 
already called by xen_smp_prepare_boot_cpu() and kvm_smp_prepare_boot_cpu().

Regards
Bibo Mao

>   
>   static void native_tlb_remove_table(struct mmu_gather *tlb, void *table)
> 

Powered by blists - more mailing lists