linux-kernel - Re: [PATCH v5 7/7] KVM: TDX: Explicitly do WBINVD when no more TDX SEAMCALLs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <44f6227778731f4e9b705320846879369ea27d96.camel@intel.com>
Date: Mon, 4 Aug 2025 12:48:42 +0000
From: "Huang, Kai" <kai.huang@...el.com>
To: "Gao, Chao" <chao.gao@...el.com>
CC: "kvm@...r.kernel.org" <kvm@...r.kernel.org>, "ashish.kalra@....com"
	<ashish.kalra@....com>, "Hansen, Dave" <dave.hansen@...el.com>,
	"thomas.lendacky@....com" <thomas.lendacky@....com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"mingo@...hat.com" <mingo@...hat.com>, "dwmw@...zon.co.uk"
	<dwmw@...zon.co.uk>, "pbonzini@...hat.com" <pbonzini@...hat.com>,
	"seanjc@...gle.com" <seanjc@...gle.com>, "tglx@...utronix.de"
	<tglx@...utronix.de>, "kas@...nel.org" <kas@...nel.org>, "Chatre, Reinette"
	<reinette.chatre@...el.com>, "Yamahata, Isaku" <isaku.yamahata@...el.com>,
	"nik.borisov@...e.com" <nik.borisov@...e.com>, "hpa@...or.com"
	<hpa@...or.com>, "peterz@...radead.org" <peterz@...radead.org>,
	"sagis@...gle.com" <sagis@...gle.com>, "Chen, Farrah"
	<farrah.chen@...el.com>, "Edgecombe, Rick P" <rick.p.edgecombe@...el.com>,
	"bp@...en8.de" <bp@...en8.de>, "binbin.wu@...ux.intel.com"
	<binbin.wu@...ux.intel.com>, "x86@...nel.org" <x86@...nel.org>, "Williams,
 Dan J" <dan.j.williams@...el.com>
Subject: Re: [PATCH v5 7/7] KVM: TDX: Explicitly do WBINVD when no more TDX
 SEAMCALLs

On Fri, 2025-08-01 at 16:30 +0800, Chao Gao wrote:
> On Tue, Jul 29, 2025 at 12:28:41AM +1200, Kai Huang wrote:
> > On TDX platforms, during kexec, the kernel needs to make sure there are
> > no dirty cachelines of TDX private memory before booting to the new
> > kernel to avoid silent memory corruption to the new kernel.
> > 
> > During kexec, the kexec-ing CPU firstly invokes native_stop_other_cpus()
> > to stop all remote CPUs before booting to the new kernel.  The remote
> > CPUs will then execute stop_this_cpu() to stop themselves.
> > 
> > The kernel has a percpu boolean to indicate whether the cache of a CPU
> > may be in incoherent state.  In stop_this_cpu(), the kernel does WBINVD
> > if that percpu boolean is true.
> > 
> > TDX turns on that percpu boolean on a CPU when the kernel does SEAMCALL.
> > This makes sure the caches will be flushed during kexec.
> > 
> > However, the native_stop_other_cpus() and stop_this_cpu() have a "race"
> > which is extremely rare to happen but could cause the system to hang.
> > 
> > Specifically, the native_stop_other_cpus() firstly sends normal reboot
> > IPI to remote CPUs and waits one second for them to stop.  If that times
> > out, native_stop_other_cpus() then sends NMIs to remote CPUs to stop
> > them.
> > 
> > The aforementioned race happens when NMIs are sent.  Doing WBINVD in
> > stop_this_cpu() makes each CPU take longer time to stop and increases
> > the chance of the race happening.
> > 
> > Explicitly flush cache in tdx_disable_virtualization_cpu() after which
> > no more TDX activity can happen on this cpu.  This moves the WBINVD to
> > an earlier stage than stop_this_cpus(), avoiding a possibly lengthy
> > operation at a time where it could cause this race.
> > 
> > Signed-off-by: Kai Huang <kai.huang@...el.com>
> > Acked-by: Paolo Bonzini <pbonzini@...hat.com>
> > Tested-by: Farrah Chen <farrah.chen@...el.com>
> > Reviewed-by: Binbin Wu <binbin.wu@...ux.intel.com>
> 
> Flushing cache after disabling virtualization looks clean. So,
> 
> Reviewed-by: Chao Gao <chao.gao@...el.com>

Thanks for the review (for this and patch 3)!