Message-ID: <de0b5d53-095a-1f85-1ebf-a22be30f7d3b@amd.com>
Date:   Thu, 10 Dec 2020 16:57:56 -0600
From:   Babu Moger <babu.moger@....com>
To:     Jim Mattson <jmattson@...gle.com>
Cc:     Paolo Bonzini <pbonzini@...hat.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>,
        "Yu, Fenghua" <fenghua.yu@...el.com>,
        Tony Luck <tony.luck@...el.com>,
        Wanpeng Li <wanpengli@...cent.com>,
        kvm list <kvm@...r.kernel.org>,
        "Lendacky, Thomas" <Thomas.Lendacky@....com>,
        Peter Zijlstra <peterz@...radead.org>,
        Sean Christopherson <seanjc@...gle.com>,
        Joerg Roedel <joro@...tes.org>,
        the arch/x86 maintainers <x86@...nel.org>,
        "kyung.min.park@...el.com" <kyung.min.park@...el.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Krish Sadhukhan <krish.sadhukhan@...cle.com>,
        "H . Peter Anvin" <hpa@...or.com>,
        "mgross@...ux.intel.com" <mgross@...ux.intel.com>,
        Vitaly Kuznetsov <vkuznets@...hat.com>,
        "Phillips, Kim" <kim.phillips@....com>,
        "Huang2, Wei" <Wei.Huang2@....com>
Subject: Re: [PATCH 2/2] KVM: SVM: Add support for Virtual SPEC_CTRL



On 12/10/20 3:36 PM, Jim Mattson wrote:
> On Thu, Dec 10, 2020 at 1:26 PM Babu Moger <babu.moger@....com> wrote:
>>
>> Hi Jim,
>>
>>> -----Original Message-----
>>> From: Jim Mattson <jmattson@...gle.com>
>>> Sent: Monday, December 7, 2020 5:06 PM
>>> To: Moger, Babu <Babu.Moger@....com>
>>> Cc: Paolo Bonzini <pbonzini@...hat.com>; Thomas Gleixner
>>> <tglx@...utronix.de>; Ingo Molnar <mingo@...hat.com>; Borislav Petkov
>>> <bp@...en8.de>; Yu, Fenghua <fenghua.yu@...el.com>; Tony Luck
>>> <tony.luck@...el.com>; Wanpeng Li <wanpengli@...cent.com>; kvm list
>>> <kvm@...r.kernel.org>; Lendacky, Thomas <Thomas.Lendacky@....com>;
>>> Peter Zijlstra <peterz@...radead.org>; Sean Christopherson
>>> <seanjc@...gle.com>; Joerg Roedel <joro@...tes.org>; the arch/x86
>>> maintainers <x86@...nel.org>; kyung.min.park@...el.com; LKML <linux-
>>> kernel@...r.kernel.org>; Krish Sadhukhan <krish.sadhukhan@...cle.com>; H .
>>> Peter Anvin <hpa@...or.com>; mgross@...ux.intel.com; Vitaly Kuznetsov
>>> <vkuznets@...hat.com>; Phillips, Kim <kim.phillips@....com>; Huang2, Wei
>>> <Wei.Huang2@....com>
>>> Subject: Re: [PATCH 2/2] KVM: SVM: Add support for Virtual SPEC_CTRL
>>>
>>> On Mon, Dec 7, 2020 at 2:38 PM Babu Moger <babu.moger@....com> wrote:
>>>>
>>>> Newer AMD processors have a feature to virtualize the use of the
>>>> SPEC_CTRL MSR. When supported, the SPEC_CTRL MSR is automatically
>>>> virtualized and no longer requires hypervisor intervention.
>>>>
>>>> This feature is detected via CPUID function 0x8000000A_EDX[20]:
>>>> GuestSpecCtrl.
>>>>
>>>> Hypervisors are not required to enable this feature since it is
>>>> automatically enabled on processors that support it.
>>>>
>>>> When this feature is enabled, the hypervisor no longer has to
>>>> intercept the usage of the SPEC_CTRL MSR and no longer is required to
>>>> save and restore the guest SPEC_CTRL setting when switching
>>>> hypervisor/guest modes.  The effective SPEC_CTRL setting is the guest
>>>> SPEC_CTRL setting or'ed with the hypervisor SPEC_CTRL setting. This
>>>> allows the hypervisor to ensure a minimum SPEC_CTRL if desired.
>>>>
>>>> This support also fixes an issue where a guest may sometimes see an
>>>> inconsistent value for the SPEC_CTRL MSR on processors that support
>>>> this feature. With the current SPEC_CTRL support, the first write to
>>>> SPEC_CTRL is intercepted and the virtualized version of the SPEC_CTRL
>>>> MSR is not updated. When the guest reads back the SPEC_CTRL MSR, it
>>>> will be 0x0, instead of the actual expected value. There isn’t a
>>>> security concern here, because the host SPEC_CTRL value is or’ed with
>>>> the Guest SPEC_CTRL value to generate the effective SPEC_CTRL value.
>>>> KVM writes with the guest's virtualized SPEC_CTRL value to SPEC_CTRL
>>>> MSR just before the VMRUN, so it will always have the actual value
>>>> even though it doesn’t appear that way in the guest. The guest will
>>>> only see the proper value for the SPEC_CTRL register if the guest was
>>>> to write to the SPEC_CTRL register again. With Virtual SPEC_CTRL
>>>> support, the MSR interception of SPEC_CTRL is disabled during
>>>> vmcb_init, so this will no longer be an issue.
>>>>
>>>> Signed-off-by: Babu Moger <babu.moger@....com>
>>>> ---
>>>
>>> Shouldn't there be some code to initialize a new "guest SPEC_CTRL"
>>> value in the VMCB, both at vCPU creation, and at virtual processor reset?
>>
>> Yes, I think so. I will check on this.
>>
>>>
>>>>  arch/x86/kvm/svm/svm.c |   17 ++++++++++++++---
>>>>  1 file changed, 14 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index
>>>> 79b3a564f1c9..3d73ec0cdb87 100644
>>>> --- a/arch/x86/kvm/svm/svm.c
>>>> +++ b/arch/x86/kvm/svm/svm.c
>>>> @@ -1230,6 +1230,14 @@ static void init_vmcb(struct vcpu_svm *svm)
>>>>
>>>>         svm_check_invpcid(svm);
>>>>
>>>> +       /*
>>>> +        * If the host supports V_SPEC_CTRL then disable the interception
>>>> +        * of MSR_IA32_SPEC_CTRL.
>>>> +        */
>>>> +       if (boot_cpu_has(X86_FEATURE_V_SPEC_CTRL))
>>>> +               set_msr_interception(&svm->vcpu, svm->msrpm,
>>> MSR_IA32_SPEC_CTRL,
>>>> +                                    1, 1);
>>>> +
>>>>         if (kvm_vcpu_apicv_active(&svm->vcpu))
>>>>                 avic_init_vmcb(svm);
>>>>
>>>> @@ -3590,7 +3598,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct
>>> kvm_vcpu *vcpu)
>>>>          * is no need to worry about the conditional branch over the wrmsr
>>>>          * being speculatively taken.
>>>>          */
>>>> -       x86_spec_ctrl_set_guest(svm->spec_ctrl, svm->virt_spec_ctrl);
>>>> +       if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
>>>> +               x86_spec_ctrl_set_guest(svm->spec_ctrl,
>>>> + svm->virt_spec_ctrl);
>>>
>>> Is this correct for the nested case? Presumably, there is now a "guest
>>> SPEC_CTRL" value somewhere in the VMCB. If L1 does not intercept this MSR,
>>> then we need to transfer the "guest SPEC_CTRL" value from the
>>> vmcb01 to the vmcb02, don't we?
>>
>> Here is the text from to be published documentation.
>> "When in host mode, the host SPEC_CTRL value is in effect and writes
>> update only the host version of SPEC_CTRL. On a VMRUN, the processor loads
>> the guest version of SPEC_CTRL from the VMCB. For non- SNP enabled guests,
>> processor behavior is controlled by the logical OR of the two registers.
>> When the guest writes SPEC_CTRL, only the guest version is updated. On a
>> VMEXIT, the guest version is saved into the VMCB and the processor returns
>> to only using the host SPEC_CTRL for speculation control. The guest
>> SPEC_CTRL is located at offset 0x2E0 in the VMCB."  This offset is into
>> the save area of the VMCB (i.e. 0x400 + 0x2E0).
>>
>> The feature X86_FEATURE_V_SPEC_CTRL will not be advertised to guests.
>> So, the guest will use the same mechanism as today where it will save and
>> restore the value into/from svm->spec_ctrl. If the value saved in the VMSA
>> is left untouched, both an L1 and L2 guest will get the proper value.
>> Thing that matters is the initial setup of vmcb01 and vmcb02 when this
>> feature is available in host(bare metal). I am going to investigate that
>> part. Do you still think I am missing something here?
> 
> It doesn't matter whether X86_FEATURE_V_SPEC_CTRL is advertised to L1
> or not. If L1 doesn't virtualize MSR_SPEC_CTRL for L2, then L1 and L2
> share the same value for that MSR. With this change, the current value
> in vmcb01 is only in vmcb01, and doesn't get propagated anywhere else.
> Hence, if L1 changes the value of MSR_SPEC_CTRL, that change is not
> visible to L2.
> 
> Thinking about what Sean said about live migration, I think the
> correct solution here is that the authoritative value for this MSR
> should continue to live in svm->spec_ctrl. When the CPU supports
> X86_FEATURE_V_SPEC_CTRL, we should just transfer the value into the
> VMCB prior to VMRUN and out of the VMCB after #VMEXIT.

Ok. Got it. I will try this approach. Thanks for the suggestion.

> 
>>
>>>
>>>>         svm_vcpu_enter_exit(vcpu, svm);
>>>>
>>>> @@ -3609,12 +3618,14 @@ static __no_kcsan fastpath_t
>>> svm_vcpu_run(struct kvm_vcpu *vcpu)
>>>>          * If the L02 MSR bitmap does not intercept the MSR, then we need to
>>>>          * save it.
>>>>          */
>>>> -       if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)))
>>>> +       if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL) &&
>>>> +           unlikely(!msr_write_intercepted(vcpu,
>>>> + MSR_IA32_SPEC_CTRL)))
>>>>                 svm->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL);
>>>
>>> Is this correct for the nested case? If L1 does not intercept this MSR, then it
>>> might have changed while L2 is running. Presumably, the hardware has stored
>>> the new value somewhere in the vmcb02 at #VMEXIT, but now we need to move
>>> that value into the vmcb01, don't we?
>>>
>>>>         reload_tss(vcpu);
>>>>
>>>> -       x86_spec_ctrl_restore_host(svm->spec_ctrl, svm->virt_spec_ctrl);
>>>> +       if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
>>>> +               x86_spec_ctrl_restore_host(svm->spec_ctrl,
>>>> + svm->virt_spec_ctrl);
>>>>
>>>>         vcpu->arch.cr2 = svm->vmcb->save.cr2;
>>>>         vcpu->arch.regs[VCPU_REGS_RAX] = svm->vmcb->save.rax;
>>>>
>>>
>>> It would be great if you could add some tests to kvm-unit-tests.
>>
>> Yes. I will check on this part.
>>
>> Thanks
>> Babu

Powered by blists - more mailing lists