Message-ID: <4aefad72-e8f3-4ad9-9f8f-fc32612358a0@amd.com>
Date: Wed, 4 Jun 2025 11:59:44 -0700
From: "Koralahalli Channabasappa, Smita"
 <Smita.KoralahalliChannabasappa@....com>
To: "Zhijian Li (Fujitsu)" <lizhijian@...itsu.com>,
 "linux-cxl@...r.kernel.org" <linux-cxl@...r.kernel.org>,
 "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
 "nvdimm@...ts.linux.dev" <nvdimm@...ts.linux.dev>,
 "linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
 "linux-pm@...r.kernel.org" <linux-pm@...r.kernel.org>
Cc: Davidlohr Bueso <dave@...olabs.net>,
 Jonathan Cameron <jonathan.cameron@...wei.com>,
 Dave Jiang <dave.jiang@...el.com>,
 Alison Schofield <alison.schofield@...el.com>,
 Vishal Verma <vishal.l.verma@...el.com>, Ira Weiny <ira.weiny@...el.com>,
 Dan Williams <dan.j.williams@...el.com>, Matthew Wilcox
 <willy@...radead.org>, Jan Kara <jack@...e.cz>,
 "Rafael J . Wysocki" <rafael@...nel.org>, Len Brown <len.brown@...el.com>,
 Pavel Machek <pavel@...nel.org>, Li Ming <ming.li@...omail.com>,
 Jeff Johnson <jeff.johnson@....qualcomm.com>,
 Ying Huang <huang.ying.caritas@...il.com>,
 "Xingtao Yao (Fujitsu)" <yaoxt.fnst@...itsu.com>,
 Peter Zijlstra <peterz@...radead.org>, Greg KH <gregkh@...uxfoundation.org>,
 Nathan Fontenot <nathan.fontenot@....com>,
 Terry Bowman <terry.bowman@....com>, Robert Richter <rrichter@....com>,
 Benjamin Cheatham <benjamin.cheatham@....com>,
 PradeepVineshReddy Kodamati <PradeepVineshReddy.Kodamati@....com>
Subject: Re: [PATCH v4 0/7] Add managed SOFT RESERVE resource handling

Hi Zhijian,

Thanks for testing my patches.

On 6/4/2025 1:43 AM, Zhijian Li (Fujitsu) wrote:
> Smita,
> 
> Thanks for your awesome work. I just tested the scenarios you listed, and they work as expected. Thanks again.
> (Minor comments inlined)
> 
> Tested-by: Li Zhijian <lizhijian@...itsu.com>
> 
> 
> To the CXL community,
> 
> The scenarios mentioned here essentially cover what a correct firmware may provide. However,
> I would like to discuss one more scenario that I can simulate with a modified QEMU:
> The E820 exposes a SOFT RESERVED region which is the same as a CFMW, but the HDM decoders are not committed. This means no region will be auto-created during boot.
> 
> As an example, after boot, the iomem tree is as follows:
> 1050000000-304fffffff : CXL Window 0
>     1050000000-304fffffff : Soft Reserved
>       <No region>
> 
> In this case, the SOFT RESERVED resource is not trimmed, so the end-user cannot create a new region.
> My question is: Is this scenario a problem? If it is, should we fix it in this patchset or create a new patch?
> 

I believe firmware should handle this correctly by ensuring that any 
exposed SOFT RESERVED ranges correspond to committed HDM decoders and 
result in region creation.

That said, I’d be interested in hearing what the rest of the community 
thinks.

> 
> 
> 
> On 04/06/2025 06:19, Smita Koralahalli wrote:
>> Add the ability to manage SOFT RESERVE iomem resources prior to them being
>> added to the iomem resource tree. This allows drivers, such as CXL, to
>> remove any pieces of the SOFT RESERVE resource that intersect with created
>> CXL regions.
>>
>> The current approach of leaving the SOFT RESERVE resources as is can cause
>> failures during hotplug of devices, such as CXL, because the resource is
>> not available for reuse after teardown of the device.
>>
>> The approach is to add SOFT RESERVE resources to a separate tree during
>> boot.
> 
> No special tree at all since V3

Will make changes. I overlooked the cover letter.

> 
> 
>> This allows any drivers to update the SOFT RESERVE resources before
>> they are merged into the iomem resource tree. In addition a notifier chain
>> is added so that drivers can be notified when these SOFT RESERVE resources
>> are added to the ioeme resource tree.
>>
>> The CXL driver is modified to use a worker thread that waits for the CXL
>> PCI and CXL mem drivers to be loaded and for their probe routine to
>> complete. Then the driver walks through any created CXL regions to trim any
>> intersections with SOFT RESERVE resources in the iomem tree.
>>
>> The dax driver uses the new soft reserve notifier chain so it can consume
>> any remaining SOFT RESERVES once they're added to the iomem tree.
>>
>> The following scenarios have been tested:
>>
>> Example 1: Exact alignment, soft reserved is a child of the region
>>
>> |---------- "Soft Reserved" -----------|
>> |-------------- "Region #" ------------|
>>
>> Before:
>>     1050000000-304fffffff : CXL Window 0
>>       1050000000-304fffffff : region0
>>         1050000000-304fffffff : Soft Reserved
>>           1080000000-2fffffffff : dax0.0
> 
> BTW, I'm curious how to set up a dax with an address range different from its corresponding region.

Hmm, this configuration was provided directly by our BIOS. The DAX 
device was mapped to a subset of the region's address space as part of 
the platform's firmware setup, so I did not explicitly configure it..

> 
> 
>>             1080000000-2fffffffff : System RAM (kmem)
>>
>> After:
>>     1050000000-304fffffff : CXL Window 0
>>       1050000000-304fffffff : region1
>>         1080000000-2fffffffff : dax0.0
>>           1080000000-2fffffffff : System RAM (kmem)
>>
>> Example 2: Start and/or end aligned and soft reserved spans multiple
>> regions
> 
> Tested
> 
>>
>> |----------- "Soft Reserved" -----------|
>> |-------- "Region #" -------|
>> or
>> |----------- "Soft Reserved" -----------|
>> |-------- "Region #" -------|
> 
> Typo? should be:
> |----------- "Soft Reserved" -----------|
>               |-------- "Region #" -------|

Yeah, Will fix.

> 
>>
>> Example 3: No alignment
>> |---------- "Soft Reserved" ----------|
>> 	|---- "Region #" ----|
> 
> Tested.
> 
> 
> Thanks
> Zhijian

Thanks
Smita

Powered by blists - more mailing lists