Message-ID: <d2b0327fb2e26966162d435df2aaf281fe5c2737.camel@intel.com>
Date:   Thu, 24 Nov 2022 11:29:35 +0000
From:   "Huang, Kai" <kai.huang@...el.com>
To:     "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "Hansen, Dave" <dave.hansen@...el.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
CC:     "Luck, Tony" <tony.luck@...el.com>,
        "bagasdotme@...il.com" <bagasdotme@...il.com>,
        "ak@...ux.intel.com" <ak@...ux.intel.com>,
        "Wysocki, Rafael J" <rafael.j.wysocki@...el.com>,
        "kirill.shutemov@...ux.intel.com" <kirill.shutemov@...ux.intel.com>,
        "Christopherson,, Sean" <seanjc@...gle.com>,
        "Chatre, Reinette" <reinette.chatre@...el.com>,
        "pbonzini@...hat.com" <pbonzini@...hat.com>,
        "linux-mm@...ck.org" <linux-mm@...ck.org>,
        "Yamahata, Isaku" <isaku.yamahata@...el.com>,
        "peterz@...radead.org" <peterz@...radead.org>,
        "Shahar, Sagi" <sagis@...gle.com>,
        "imammedo@...hat.com" <imammedo@...hat.com>,
        "Gao, Chao" <chao.gao@...el.com>,
        "Brown, Len" <len.brown@...el.com>,
        "sathyanarayanan.kuppuswamy@...ux.intel.com" 
        <sathyanarayanan.kuppuswamy@...ux.intel.com>,
        "Huang, Ying" <ying.huang@...el.com>,
        "Williams, Dan J" <dan.j.williams@...el.com>
Subject: Re: [PATCH v7 12/20] x86/virt/tdx: Create TDMRs to cover all TDX
 memory regions


> > +static inline u64 tdmr_start(struct tdmr_info *tdmr)
> > +{
> > +	return tdmr->base;
> > +}
> 
> I'm always skeptical that it's a good idea to take this in code:
> 
> 	tdmr->base
> 
> and make it this:
> 
> 	tdmr_start(tdmr)
> 
> because the helper is *LESS* compact than the open-coded form!  I hope
> I'm proven wrong.

IIUC you prefer using tdmr->base directly. Will do.

> 
> > +static inline u64 tdmr_end(struct tdmr_info *tdmr)
> > +{
> > +	return tdmr->base + tdmr->size;
> > +}
> > +
> >  /* Calculate the actual TDMR_INFO size */
> >  static inline int cal_tdmr_size(void)
> >  {
> > @@ -492,14 +510,98 @@ static struct tdmr_info *alloc_tdmr_array(int *array_sz)
> >  	return alloc_pages_exact(*array_sz, GFP_KERNEL | __GFP_ZERO);
> >  }
> >  
> > +static struct tdmr_info *tdmr_array_entry(struct tdmr_info *tdmr_array,
> > +					  int idx)
> > +{
> > +	return (struct tdmr_info *)((unsigned long)tdmr_array +
> > +			cal_tdmr_size() * idx);
> > +}
> 
> FWIW, I think it's probably a bad idea to have 'struct tdmr_info *'
> types floating around since:
> 
> 	tmdr_info_array[0]
> 
> works, but:
> 
> 	tmdr_info_array[1]
> 
> will blow up in your face.  It would almost make sense to have
> 
> struct tdmr_info_list {
> 	struct tdmr_info *first_tdmr;
> }
> 
> and then pass around pointers to the 'struct tdmr_info_list'.  Maybe
> that's overkill, but it is kinda silly to call something an array if []
> doesn't work on it.

Then should I introduce 'struct tdmr_info_list' in the previous patch (which
allocates enough space for the tdmr_array), and add functions to allocate/free
this tdmr_info_list?

> 
> > +/*
> > + * Create TDMRs to cover all TDX memory regions.  The actual number
> > + * of TDMRs is set to @tdmr_num.
> > + */
> > +static int create_tdmrs(struct tdmr_info *tdmr_array, int *tdmr_num)
> > +{
> > +	struct tdx_memblock *tmb;
> > +	int tdmr_idx = 0;
> > +
> > +	/*
> > +	 * Loop over TDX memory regions and create TDMRs to cover them.
> > +	 * To keep it simple, always try to use one TDMR to cover
> > +	 * one memory region.
> > +	 */
> 
> This seems like it might tend to under-utilize TDMRs.  I'm sure this is
> done for simplicity, but is it OK?  Why is it OK?  How are you sure this
> won't bite us later?

In practice the maximum number of TDMRs is 64.  In reality we never met a
machine that could result in so many memory regions, and typically 20 TDMRs is
big enough to cover them.

But if user uses 'memmap' to deliberately create bunch of discrete memory
regions, then we can run out of TDMRs.  But I think we can blame user in this
case.

How about add a comment?

	/*
	 * In practice TDX1.0 supports 64 TDMRs, which should be big enough
	 * to cover all memory regions in reality if admin doesn't use 'memmap'
	 * to create bunch of discrete memory regions.
	 */

> 
> > +	list_for_each_entry(tmb, &tdx_memlist, list) {
> > +		struct tdmr_info *tdmr;
> > +		u64 start, end;
> > +
> > +		tdmr = tdmr_array_entry(tdmr_array, tdmr_idx);
> > +		start = TDMR_ALIGN_DOWN(tmb->start_pfn << PAGE_SHIFT);
> > +		end = TDMR_ALIGN_UP(tmb->end_pfn << PAGE_SHIFT);
> 
> Nit: a little vertical alignment can make this much more readable:
> 
> 		start = TDMR_ALIGN_DOWN(tmb->start_pfn << PAGE_SHIFT);
> 		end   = TDMR_ALIGN_UP  (tmb->end_pfn   << PAGE_SHIFT);

Sure. 

Btw Ying suggested we can use PHYS_PFN() for 
	
	<phys> >> PAGE_SHIFT

and PFN_PHYS() for

	<pfn> << PAGE_SHIFT

Should I apply them to this entire series?

> 
> > +
> > +		/*
> > +		 * If the current TDMR's size hasn't been initialized,
> > +		 * it is a new TDMR to cover the new memory region.
> > +		 * Otherwise, the current TDMR has already covered the
> > +		 * previous memory region.  In the latter case, check
> > +		 * whether the current memory region has been fully or
> > +		 * partially covered by the current TDMR, since TDMR is
> > +		 * 1G aligned.
> > +		 */
> 
> Again, we have a comment over a if() block that describes what the
> individual steps in the block do.  *Plus* each individual step is
> *ALREADY* commented.  What purpose does this comment serve?

I think the check of 'if (tdmr->size)' is still worth commenting.  The last
sentence can be removed -- as you said, it is kinda duplicated with the
individual comments within the if().

> 
> > +		if (tdmr->size) {
> > +			/*
> > +			 * Loop to the next memory region if the current
> > +			 * block has already been fully covered by the
> > +			 * current TDMR.
> > +			 */
> > +			if (end <= tdmr_end(tdmr))
> > +				continue;
> > +
> > +			/*
> > +			 * If part of the current memory region has
> > +			 * already been covered by the current TDMR,
> > +			 * skip the already covered part.
> > +			 */
> > +			if (start < tdmr_end(tdmr))
> > +				start = tdmr_end(tdmr);
> > +
> > +			/*
> > +			 * Create a new TDMR to cover the current memory
> > +			 * region, or the remaining part of it.
> > +			 */
> > +			tdmr_idx++;
> > +			if (tdmr_idx >= tdx_sysinfo.max_tdmrs)
> > +				return -E2BIG;
> > +
> > +			tdmr = tdmr_array_entry(tdmr_array, tdmr_idx);
> > +		}
> > +
> > +		tdmr->base = start;
> > +		tdmr->size = end - start;
> > +	}
> > +
> > +	/* @tdmr_idx is always the index of last valid TDMR. */
> > +	*tdmr_num = tdmr_idx + 1;
> > +
> > +	return 0;
> > +}
> 
> Seems like a positive return value could be the number of populated
> TDMRs.  That would get rid of the int* argument.

Yes we can.  I'll make the function return -E2BIG, or the actual number of
TDMRs.

Btw, I think it's better to print out some error message in case of -E2BIG so
user can easily tell the reason of failure?  Something like this:

	if (tdmr_idx >= tdx_sysinfo.max_tdmrs) {
		pr_info("no enough TDMRs to cover all TDX memory regions\n");
		return -E2BIG;
	}

> 
> >  /*
> >   * Construct an array of TDMRs to cover all TDX memory ranges.
> >   * The actual number of TDMRs is kept to @tdmr_num.
> >   */
> 
> OK, so something else allocated the 'tdmr_array' and it's being passed
> in here to fill it out.  "construct" and "create" are both near synonyms
> for "allocate", which isn't even being done here.
> 
> We want something here that will make it clear that this function is
> taking an already populated list of TDMRs and filling it out.
> "fill_tmdrs()" seems like it might be a better choice.
> 
> This is also a place where better words can help.  If the function is
> called "construct", then there's *ZERO* value in using the same word in
> the comment.  Using a word that is a close synonym but that can contrast
> it with something different would be really nice, say:

Thanks for the tip!
> 
> This is also a place where the calling convention can be used to add
> clarity.  If you implicitly use a global variable, you have to explain
> that.  But, if you pass *in* a variable, it's a lot more clear.
> 
> Take this, for instance:
> 
> /*
>  * Take the memory referenced in @tdx_memlist and populate the
>  * preallocated @tmdr_array, following all the special alignment
>  * and size rules for TDMR.
>  */
> static int fill_out_tdmrs(struct list_head *tdx_memlist,
> 			  struct tdmr_info *tdmr_array)
> {
> ...
> 
> That's 100% crystal clear about what's going on.  You know what the
> inputs are and the outputs.  You also know why this is even necessary.
> It's implied a bit, but it's because TDMRs have special rules about
> size/alignment and tdx_memlists do not.

Agreed.  Let me try this out.

Powered by blists - more mailing lists