Message-ID: <81413f081fde380b07533a7839346334bb79d3cf.camel@intel.com>
Date: Thu, 15 May 2025 17:28:52 +0000
From: "Edgecombe, Rick P" <rick.p.edgecombe@...el.com>
To: "Zhao, Yan Y" <yan.y.zhao@...el.com>
CC: "kvm@...r.kernel.org" <kvm@...r.kernel.org>, "Li, Xiaoyao"
	<xiaoyao.li@...el.com>, "quic_eberman@...cinc.com"
	<quic_eberman@...cinc.com>, "Hansen, Dave" <dave.hansen@...el.com>,
	"david@...hat.com" <david@...hat.com>, "Li, Zhiquan1"
	<zhiquan1.li@...el.com>, "tabba@...gle.com" <tabba@...gle.com>,
	"vbabka@...e.cz" <vbabka@...e.cz>, "thomas.lendacky@....com"
	<thomas.lendacky@....com>, "michael.roth@....com" <michael.roth@....com>,
	"seanjc@...gle.com" <seanjc@...gle.com>, "Weiny, Ira" <ira.weiny@...el.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"pbonzini@...hat.com" <pbonzini@...hat.com>, "ackerleytng@...gle.com"
	<ackerleytng@...gle.com>, "Yamahata, Isaku" <isaku.yamahata@...el.com>,
	"binbin.wu@...ux.intel.com" <binbin.wu@...ux.intel.com>, "Peng, Chao P"
	<chao.p.peng@...el.com>, "Du, Fan" <fan.du@...el.com>, "Annapurve, Vishal"
	<vannapurve@...gle.com>, "jroedel@...e.de" <jroedel@...e.de>, "Miao, Jun"
	<jun.miao@...el.com>, "Shutemov, Kirill" <kirill.shutemov@...el.com>,
	"pgonda@...gle.com" <pgonda@...gle.com>, "x86@...nel.org" <x86@...nel.org>
Subject: Re: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper
 tdh_mem_page_demote()

On Thu, 2025-05-15 at 16:26 +0800, Yan Zhao wrote:
> On Wed, May 14, 2025 at 02:19:56AM +0800, Edgecombe, Rick P wrote:
> > On Thu, 2025-04-24 at 11:04 +0800, Yan Zhao wrote:
> > > From: Xiaoyao Li <xiaoyao.li@...el.com>
> > > 
> > > Add a wrapper tdh_mem_page_demote() to invoke SEAMCALL TDH_MEM_PAGE_DEMOTE
> > > to demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the
> > > TDX module only supports demotion of a 2M huge leaf entry. After a
> > > successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a
> > > non-leaf entry, linking to the newly-added page table page. The newly
> > > linked page table page then contains 512 leaf entries, pointing to the 2M
> > > guest private pages.
> > > 
> > > The "gpa" and "level" direct the TDX module to search and find the old
> > > huge leaf entry.
> > > 
> > > As the new non-leaf entry points to a page table page, callers need to
> > > pass in the page table page in parameter "page".
> > > 
> > > In case of S-EPT walk failure, the entry, level and state where the error
> > > was detected are returned in ext_err1 and ext_err2.
> > > 
> > > On interrupt pending, SEAMCALL TDH_MEM_PAGE_DEMOTE returns error
> > > TDX_INTERRUPTED_RESTARTABLE.
> > > 
> > > [Yan: Rebased and split patch, wrote changelog]
> > 
> > We should add the level of detail here like we did for the base series ones.
> I'll provide changelog details under "---" of each patch in the next version.

I mean the commit log (above the "---") needs the same tip style treatment as
the other SEAMCALL wrapper patches.

>  
> > > 
> > > Signed-off-by: Xiaoyao Li <xiaoyao.li@...el.com>
> > > Signed-off-by: Isaku Yamahata <isaku.yamahata@...el.com>
> > > Signed-off-by: Yan Zhao <yan.y.zhao@...el.com>
> > > ---
> > >  arch/x86/include/asm/tdx.h  |  2 ++
> > >  arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++
> > >  arch/x86/virt/vmx/tdx/tdx.h |  1 +
> > >  3 files changed, 23 insertions(+)
> > > 
> > > diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
> > > index 26ffc792e673..08eff4b2f5e7 100644
> > > --- a/arch/x86/include/asm/tdx.h
> > > +++ b/arch/x86/include/asm/tdx.h
> > > @@ -177,6 +177,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);
> > >  u64 tdh_mng_create(struct tdx_td *td, u16 hkid);
> > >  u64 tdh_vp_create(struct tdx_td *td, struct tdx_vp *vp);
> > >  u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data);
> > > +u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,
> > > +			u64 *ext_err1, u64 *ext_err2);
> > >  u64 tdh_mr_extend(struct tdx_td *td, u64 gpa, u64 *ext_err1, u64 *ext_err2);
> > >  u64 tdh_mr_finalize(struct tdx_td *td);
> > >  u64 tdh_vp_flush(struct tdx_vp *vp);
> > > diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
> > > index a66d501b5677..5699dfe500d9 100644
> > > --- a/arch/x86/virt/vmx/tdx/tdx.c
> > > +++ b/arch/x86/virt/vmx/tdx/tdx.c
> > > @@ -1684,6 +1684,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data)
> > >  }
> > >  EXPORT_SYMBOL_GPL(tdh_mng_rd);
> > >  
> > > +u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,
> > > +			u64 *ext_err1, u64 *ext_err2)
> > > +{
> > > +	struct tdx_module_args args = {
> > > +		.rcx = gpa | level,
> > 
> > This will only ever be level 2MB, how about dropping the arg?
> Do you mean hardcoding level to be 2MB in tdh_mem_page_demote()?

Yea, we don't support 1GB, so the level arg on the wrapper is superfluous.

> 
> The SEAMCALL TDH_MEM_PAGE_DEMOTE supports level of 1GB in current TDX module.
> 
> > > +		.rdx = tdx_tdr_pa(td),
> > > +		.r8 = page_to_phys(page),
> > > +	};
> > > +	u64 ret;
> > > +
> > > +	tdx_clflush_page(page);
> > > +	ret = seamcall_ret(TDH_MEM_PAGE_DEMOTE, &args);
> > > +
> > > +	*ext_err1 = args.rcx;
> > > +	*ext_err2 = args.rdx;
> > 
> > How about we just call these entry and level_state, like the caller.
> Not sure, but I feel that ext_err* might be better, because
> - according to the spec,
>   a) the args.rcx, args.rdx is unmodified (i.e. still hold the passed-in value)
>      in case of error TDX_INTERRUPTED_RESTARTABLE.
>   b) args.rcx, args.rdx can only be interpreted as entry and level_state in case
>      of EPT walk error.
>   c) in other cases, they are 0.
> - consistent with tdh_mem_page_aug(), tdh_mem_range_block()...

Yea, it's consistent. I'm ok leaving it as is.

> 
> 
> > > +
> > > +	return ret;
> > > +}
> > > +EXPORT_SYMBOL_GPL(tdh_mem_page_demote);
> > 
> > Looking in the docs, TDX module gives some somewhat constrained guidance:
> > 1. TDH.MEM.PAGE.DEMOTE should be invoked in a loop until it terminates
> > successfully.
> > 2. The host VMM should be designed to avoid cases where interrupt storms prevent
> > successful completion of TDH.MEM.PAGE.DEMOTE.
> > 
> > The caller looks like:
> > 	do {
> > 		err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
> > 					  &entry, &level_state);
> > 	} while (err == TDX_INTERRUPTED_RESTARTABLE);
> > 
> > 	if (unlikely(tdx_operand_busy(err))) {
> > 		tdx_no_vcpus_enter_start(kvm);
> > 		err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
> > 					  &entry, &level_state);
> > 		tdx_no_vcpus_enter_stop(kvm);
> > 	}
> > 
> > The brute force second case could also be subjected to a
> > TDX_INTERRUPTED_RESTARTABLE and is not handled. As for interrupt storms, I guess
> You are right.
> 
> > we could disable interrupts while we do the second brute force case. So the
> > TDX_INTERRUPTED_RESTARTABLE loop could have a max retries, and the brute force
> > case could also disable interrupts.
> Good idea.
> 
> > Hmm, how to pick the max retries count. It's a tradeoff between interrupt
> > latency and DOS/code complexity. Do we have any idea how long demote might take?
> I did a brief test on my SPR, where the host was not busy :
> tdh_mem_page_demote() was called 142 times, with each invocation taking around
> 10us.

10us doesn't seem too bad? Makes me think to not loop and instead just do a
single retry with interrupts disabled. We should definitely add the data based
reasoning to the log.

The counter point is that the SEAMCALL must be supporting
TDX_INTERRUPTED_RESTARTABLE for a reason. And the reason probably is that it
sometimes takes longer than someone that was reasonable. Maybe we should ask TDX
module folks if there is any history.

> 2 invocations were due to TDX_INTERRUPTED_RESTARTABLE.
> For each GFN, at most 1 retry was performed.
> 
> I will do more investigations.

Powered by blists - more mailing lists