linux-kernel - Re: [RFC PATCH] KVM: x86: hyper-v: Inhibit APICv with VP Assist on SPR/EMR

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <57F9FA8F-0B80-497E-932E-45510FF41559@nutanix.com>
Date: Tue, 6 Aug 2024 05:41:35 +0000
From: Eiichi Tsukata <eiichi.tsukata@...anix.com>
To: "chao.gao@...el.com" <chao.gao@...el.com>,
        "seanjc@...gle.com"
	<seanjc@...gle.com>,
        "pbonzini@...hat.com" <pbonzini@...hat.com>,
        "tglx@...utronix.de" <tglx@...utronix.de>,
        "mingo@...hat.com"
	<mingo@...hat.com>, "bp@...en8.de" <bp@...en8.de>,
        "dave.hansen@...ux.intel.com" <dave.hansen@...ux.intel.com>,
        "x86@...nel.org"
	<x86@...nel.org>, "hpa@...or.com" <hpa@...or.com>,
        Vitaly Kuznetsov
	<vkuznets@...hat.com>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
CC: Jon Kohler <jon@...anix.com>
Subject: Re: [RFC PATCH] KVM: x86: hyper-v: Inhibit APICv with VP Assist on
 SPR/EMR

Hi

This issue has been observed on SPR and EMR but may be reproducible on other
CPU models.  Appreciate if anyone know further detailed information about the  issue.

Minimized repro is:  
```
for i in `seq 1 20`  do
 /usr/libexec/qemu-kvm -accel kvm \
  -cpu kvm64,hv-vapic=on \
  -hda /path/to/win7-$i.qcow2 -m 4G \
  -smp 1 \    -vnc 127.0.0.1:$i > qemu.$i.log 2>&1 &
done
```

I'd like to note that this issue is observed only when the node runs
*multiple* Windows VMs at once.  i.e.: Running only one Windows VM doesn't
cause the issue.

Moreover, running VMs on a dedicated CPU doesn't cause the issue.
I mean, prepending "taskset" to qemu-kvm like this:

```
for i in `seq 1 20`  do
 taskset $i /usr/libexec/qemu-kvm -accel kvm \
  -cpu kvm64,hv-vapic=on \
  -hda /path/to/win7-$i.qcow2 -m 4G \
  -smp 1 \    -vnc 127.0.0.1:$i > qemu.$i.log 2>&1 &
done
```

Then the issue doesn't reproduce.

Best,
Eiichi

> On Aug 6, 2024, at 14:37, Eiichi Tsukata <eiichi.tsukata@...anix.com> wrote:
> 
> Running multiple Windows VMs with VP Assis and APICv causes KVM internal
> error on Spapphire Rpaids and Emerald Rapids as is reported in [1].
> Here Qemu outputs:
> 
>  KVM internal error. Suberror: 3
>  extra data[0]: 0x000000008000002f
>  extra data[1]: 0x0000000000000020
>  extra data[2]: 0x0000000000000582
>  extra data[3]: 0x0000000000000006
>  RAX=0000000000000000 RBX=0000000000000000 RCX=0000000040000070
>  RDX=0000000000000000
>  RSI=fffffa8001e3db60 RDI=fffffa8002bc8aa0 RBP=fffff88005a91670
>  RSP=fffff88005a915c8
>  R8 =0000000000000009 R9 =000000000000000b R10=fffff8000264b000
>  R11=fffff88005a91750
>  R12=fffff88002e40180 R13=fffffa8001e3dc68 R14=fffffa8001e3dc68
>  R15=0000000000000002
>  RIP=fffff8000271722c RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>  ES =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>  CS =0010 0000000000000000 00000000 00209b00 DPL=0 CS64 [-RA]
>  SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>  DS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>  FS =0053 00000000fff9a000 00007c00 0040f300 DPL=3 DS   [-WA]
>  GS =002b fffff88002e40000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>  LDT=0000 0000000000000000 ffffffff 00c00000
>  TR =0040 fffff88002e44ec0 00000067 00008b00 DPL=0 TSS64-busy
>  GDT=     fffff88002e4b4c0 0000007f
>  IDT=     fffff88002e4b540 00000fff
>  CR0=80050031 CR2=00000000002e408e CR3=000000001c6f5000 CR4=000406f8
>  DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>  DR3=0000000000000000
>  DR6=00000000fffe07f0 DR7=0000000000000400
>  EFER=0000000000000d01
>  Code=25 a8 4b 00 00 b9 70 00 00 40 0f ba 32 00 72 06 33 c0 8b d0 <0f> 30
>  5a 58 59 c3 90 90 90 90 90 90 90 90 90 90 90 90 90 90 cc cc cc cc cc cc
>  66 66 0f 1f
> 
> As is noted in [1], this issue is considered to be a microcode issue
> specific to SPR/EMR.
> 
> Disable APICv when guest tries to enable VP Assist page only when it's
> running on those problematic CPU models.
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=218267 [1]
> Signed-off-by: Eiichi Tsukata <eiichi.tsukata@...anix.com>
> ---
> arch/x86/include/asm/kvm_host.h |  8 ++++++++
> arch/x86/kvm/hyperv.c           | 13 +++++++++++++
> arch/x86/kvm/vmx/main.c         |  1 +
> 3 files changed, 22 insertions(+)
> 
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 950a03e0181e..9ff687c7326b 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -1213,6 +1213,13 @@ enum kvm_apicv_inhibit {
> */
> APICV_INHIBIT_REASON_HYPERV,
> 
> + /*
> + * Using VP Assist and APICv simultaneously on Sapphire Rapids
> + * or Emerald Rapids causes KVM internal error, which is
> + * considered to be a microcode issue.
> + */
> + APICV_INHIBIT_REASON_HYPERV_VP_ASSIST,
> +
> /*
> * APIC acceleration is inhibited because the userspace didn't yet
> * enable the kernel/split irqchip.
> @@ -1285,6 +1292,7 @@ enum kvm_apicv_inhibit {
> #define APICV_INHIBIT_REASONS \
> __APICV_INHIBIT_REASON(DISABLED), \
> __APICV_INHIBIT_REASON(HYPERV), \
> + __APICV_INHIBIT_REASON(HYPERV_VP_ASSIST), \
> __APICV_INHIBIT_REASON(ABSENT), \
> __APICV_INHIBIT_REASON(BLOCKIRQ), \
> __APICV_INHIBIT_REASON(PHYSICAL_ID_ALIASED), \
> diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
> index 4f0a94346d00..8d5a1f685191 100644
> --- a/arch/x86/kvm/hyperv.c
> +++ b/arch/x86/kvm/hyperv.c
> @@ -36,6 +36,7 @@
> 
> #include <asm/apicdef.h>
> #include <asm/mshyperv.h>
> +#include <asm/cpu_device_id.h>
> #include <trace/events/kvm.h>
> 
> #include "trace.h"
> @@ -1550,6 +1551,8 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host)
> case HV_X64_MSR_VP_ASSIST_PAGE: {
> u64 gfn;
> unsigned long addr;
> + struct kvm *kvm = vcpu->kvm;
> + struct cpuinfo_x86 *c = &boot_cpu_data;
> 
> if (!(data & HV_X64_MSR_VP_ASSIST_PAGE_ENABLE)) {
> hv_vcpu->hv_vapic = data;
> @@ -1571,6 +1574,16 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host)
> return 1;
> hv_vcpu->hv_vapic = data;
> kvm_vcpu_mark_page_dirty(vcpu, gfn);
> +
> + /*
> + * Using VP Assist and APICv simultaneously on Sapphire Rapids
> + * or Emerald Rapids causes KVM internal error, which is
> + * considered to be a microcode issue.
> + */
> + if (c->x86_vfm == INTEL_SAPPHIRERAPIDS_X ||
> +    c->x86_vfm == INTEL_EMERALDRAPIDS_X)
> + kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_HYPERV_VP_ASSIST);
> +
> if (kvm_lapic_set_pv_eoi(vcpu,
>    gfn_to_gpa(gfn) | KVM_MSR_ENABLED,
>    sizeof(struct hv_vp_assist_page)))
> diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c
> index 0bf35ebe8a1b..a1e7007133a1 100644
> --- a/arch/x86/kvm/vmx/main.c
> +++ b/arch/x86/kvm/vmx/main.c
> @@ -11,6 +11,7 @@
> (BIT(APICV_INHIBIT_REASON_DISABLED) | \
> BIT(APICV_INHIBIT_REASON_ABSENT) | \
> BIT(APICV_INHIBIT_REASON_HYPERV) | \
> + BIT(APICV_INHIBIT_REASON_HYPERV_VP_ASSIST) | \
> BIT(APICV_INHIBIT_REASON_BLOCKIRQ) | \
> BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \
> BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \
> -- 
> 2.45.2
>