linux-kernel - Re: [PATCH 1/7] KVM: x86: Free vCPUs before freeing VM state

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <Z77EEGnHnyWsEt+h@yzhao56-desk.sh.intel.com>
Date: Wed, 26 Feb 2025 15:34:40 +0800
From: Yan Zhao <yan.y.zhao@...el.com>
To: Sean Christopherson <seanjc@...gle.com>
CC: Marc Zyngier <maz@...nel.org>, Oliver Upton <oliver.upton@...ux.dev>,
	Tianrui Zhao <zhaotianrui@...ngson.cn>, Bibo Mao <maobibo@...ngson.cn>,
	Huacai Chen <chenhuacai@...nel.org>, Madhavan Srinivasan
	<maddy@...ux.ibm.com>, Anup Patel <anup@...infault.org>, Paul Walmsley
	<paul.walmsley@...ive.com>, Palmer Dabbelt <palmer@...belt.com>, Albert Ou
	<aou@...s.berkeley.edu>, Christian Borntraeger <borntraeger@...ux.ibm.com>,
	Janosch Frank <frankja@...ux.ibm.com>, Claudio Imbrenda
	<imbrenda@...ux.ibm.com>, Paolo Bonzini <pbonzini@...hat.com>,
	<linux-arm-kernel@...ts.infradead.org>, <kvmarm@...ts.linux.dev>,
	<kvm@...r.kernel.org>, <loongarch@...ts.linux.dev>,
	<linux-mips@...r.kernel.org>, <linuxppc-dev@...ts.ozlabs.org>,
	<kvm-riscv@...ts.infradead.org>, <linux-riscv@...ts.infradead.org>,
	<linux-kernel@...r.kernel.org>, Aaron Lewis <aaronlewis@...gle.com>, "Jim
 Mattson" <jmattson@...gle.com>, Rick P Edgecombe
	<rick.p.edgecombe@...el.com>, Kai Huang <kai.huang@...el.com>, Isaku Yamahata
	<isaku.yamahata@...el.com>
Subject: Re: [PATCH 1/7] KVM: x86: Free vCPUs before freeing VM state

On Tue, Feb 25, 2025 at 07:04:55AM -0800, Sean Christopherson wrote:
> On Tue, Feb 25, 2025, Yan Zhao wrote:
> > > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > > index 58b82d6fd77c..045c61cc7e54 100644
> > > --- a/arch/x86/kvm/x86.c
> > > +++ b/arch/x86/kvm/x86.c
> > > @@ -12890,11 +12890,11 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
> > >  		mutex_unlock(&kvm->slots_lock);
> > >  	}
> > >  	kvm_unload_vcpu_mmus(kvm);
> > > +	kvm_destroy_vcpus(kvm);
> > >  	kvm_x86_call(vm_destroy)(kvm);
> > >  	kvm_free_msr_filter(srcu_dereference_check(kvm->arch.msr_filter, &kvm->srcu, 1));
> > >  	kvm_pic_destroy(kvm);
> > >  	kvm_ioapic_destroy(kvm);
> > > -	kvm_destroy_vcpus(kvm);
> > >  	kvfree(rcu_dereference_check(kvm->arch.apic_map, 1));
> > >  	kfree(srcu_dereference_check(kvm->arch.pmu_event_filter, &kvm->srcu, 1));
> > >  	kvm_mmu_uninit_vm(kvm);
> > After this change, now the sequence is that
> > 
> > 1. kvm_arch_pre_destroy_vm()
> > 2. kvm_arch_destroy_vm()
> >    2.1 kvm_destroy_vcpus()
> >    2.2 .vm_destroy hook
> >    2.3 kvm_mmu_uninit_vm() --> mirror root ref is 1 upon here. Zap the mirror
> >                                root and reclaim SETP page table pages.
> >    2.4 .vm_free hook
> > 
> > Since TDX needs to reclaim the TDR page after reclaiming all other pages, we
> > currently added a vm_free hook at 2.4, after 2.3.
> > 
> > Could we move kvm_mmu_uninit_vm() before the .vm_destroy hook and after
> > kvm_destroy_vcpus()?
> > 
> > Or move the .vm_destroy hook after kvm_mmu_uninit_vm(), e.g. after
> > kvm_page_track_cleanup()?
> 
> I would go for the first option.  I'll tack on a patch since I need to test all
> of these flows anyways, and I would much prefer to change course sooner rather
> than later if it doesn't work for whatever reason.
> 
> Is this comment accurate?
Yes.

And since tdx_mmu_release_hkid() is called before kvm_unload_vcpu_mmus(),
patch 4 in this series is required by TDX. Otherwise, the list_add in
tdx_vcpu_load will cause corruption during tearing down, since
tdx_mmu_release_hkid() has already invoked tdx_disassociate_vp() on all
vcpus.

kvm_unload_vcpu_mmu
  vcpu_load
    tdx_vcpu_load
      list_add(&tdx->cpu_list, &per_cpu(associated_tdvcpus, cpu))


So, maybe a change as below is also required by TDX in case vcpu_load() is
accidentally invoked after .vm_pre_destroy.

diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index b67df0af64f3..183192706ced 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -703,7 +704,7 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
        struct vcpu_tdx *tdx = to_tdx(vcpu);

        vmx_vcpu_pi_load(vcpu, cpu);
-       if (vcpu->cpu == cpu)
+       if (vcpu->cpu == cpu || !is_hkid_assigned(to_kvm_tdx(vcpu->kvm)))
                return;

        tdx_flush_vp_on_cpu(vcpu);



> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 1e5f6f820c0b..f5685f153e08 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -12874,13 +12874,19 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
>                 mutex_unlock(&kvm->slots_lock);
>         }
>         kvm_destroy_vcpus(kvm);
> +
> +       /*
> +        * Do final MMU teardown prior to calling into vendor code.  All pages
> +        * that were donated to the TDX module, e.g. for S-EPT tables, need to
> +        * be reclaimed before the VM metadata page can be freed.
> +        */
> +       kvm_mmu_uninit_vm(kvm);
>         kvm_x86_call(vm_destroy)(kvm);
>         kvm_free_msr_filter(srcu_dereference_check(kvm->arch.msr_filter, &kvm->srcu, 1));
>         kvm_pic_destroy(kvm);
>         kvm_ioapic_destroy(kvm);
>         kvfree(rcu_dereference_check(kvm->arch.apic_map, 1));
>         kfree(srcu_dereference_check(kvm->arch.pmu_event_filter, &kvm->srcu, 1));
> -       kvm_mmu_uninit_vm(kvm);
>         kvm_page_track_cleanup(kvm);
>         kvm_xen_destroy_vm(kvm);
>         kvm_hv_destroy_vm(kvm);