Message-ID: <6b999222-8a14-47f3-bac8-7af7efd79b12@amd.com>
Date: Mon, 7 Apr 2025 17:35:27 -0500
From: "Bowman, Terry" <terry.bowman@....com>
To: "Zhijian Li (Fujitsu)" <lizhijian@...itsu.com>,
 "dave@...olabs.net" <dave@...olabs.net>,
 "jonathan.cameron@...wei.com" <jonathan.cameron@...wei.com>,
 "dave.jiang@...el.com" <dave.jiang@...el.com>,
 "alison.schofield@...el.com" <alison.schofield@...el.com>,
 "vishal.l.verma@...el.com" <vishal.l.verma@...el.com>,
 "ira.weiny@...el.com" <ira.weiny@...el.com>,
 "dan.j.williams@...el.com" <dan.j.williams@...el.com>,
 "willy@...radead.org" <willy@...radead.org>, "jack@...e.cz" <jack@...e.cz>,
 "rafael@...nel.org" <rafael@...nel.org>,
 "len.brown@...el.com" <len.brown@...el.com>, "pavel@....cz" <pavel@....cz>,
 "ming.li@...omail.com" <ming.li@...omail.com>,
 "nathan.fontenot@....com" <nathan.fontenot@....com>,
 "Smita.KoralahalliChannabasappa@....com"
 <Smita.KoralahalliChannabasappa@....com>,
 "huang.ying.caritas@...il.com" <huang.ying.caritas@...il.com>,
 "Xingtao Yao (Fujitsu)" <yaoxt.fnst@...itsu.com>,
 "peterz@...radead.org" <peterz@...radead.org>,
 "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>,
 "quic_jjohnson@...cinc.com" <quic_jjohnson@...cinc.com>,
 "ilpo.jarvinen@...ux.intel.com" <ilpo.jarvinen@...ux.intel.com>,
 "bhelgaas@...gle.com" <bhelgaas@...gle.com>,
 "andriy.shevchenko@...ux.intel.com" <andriy.shevchenko@...ux.intel.com>,
 "mika.westerberg@...ux.intel.com" <mika.westerberg@...ux.intel.com>,
 "akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
 "gourry@...rry.net" <gourry@...rry.net>,
 "linux-cxl@...r.kernel.org" <linux-cxl@...r.kernel.org>,
 "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
 "nvdimm@...ts.linux.dev" <nvdimm@...ts.linux.dev>,
 "linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
 "linux-pm@...r.kernel.org" <linux-pm@...r.kernel.org>,
 "rrichter@....com" <rrichter@....com>,
 "benjamin.cheatham@....com" <benjamin.cheatham@....com>,
 "PradeepVineshReddy.Kodamati@....com" <PradeepVineshReddy.Kodamati@....com>
Cc: "Yasunori Gotou (Fujitsu)" <y-goto@...itsu.com>
Subject: Re: [PATCH v3 0/4] Add managed SOFT RESERVE resource handling


On 4/7/2025 2:31 AM, Zhijian Li (Fujitsu) wrote:
> Hi Terry,
>
> If I understand correctly, this patch set has only considered the situation where the
> soft reserved area and the region are exactly the same, as in pattern 1.

Hi Zhijian,

I'm working on example test case(s) for your questions. I'll respond here.

Regards,
Terry


> However, I believe we also need to consider situations where these two are not equal,
> which are outlined in pattern 2 and 3 below. Let me explain them:
>
> ===========================================
> Pattern 1:
> - region0 will be created during OS booting due to programed hdm decoder
> - After OS booted, region0 can be re-created again after destroy it
> ┌────────────────────┐
> │       CFMW         │
> └────────────────────┘
> ┌────────────────────┐
> │    reserved0       │
> └────────────────────┘
> ┌────────────────────┐
> │       mem0         │
> └────────────────────┘
> ┌────────────────────┐
> │      region0       │
> └────────────────────┘
>
>
> Pattern 2:
> The HDM decoder is not in a committed state, so during the kernel boot process,
> egion0 will not be created automatically. In this case, the soft reserved area will
> not be removed from the iomem tree. After the OS starts,
> users cannot create a region (cxl create-region) either, as there should
> be an intersection between the soft reserved area and the region.
>                               
> ┌────────────────────┐
> │       CFMW         │
> └────────────────────┘
> ┌────────────────────┐
> │    reserved0       │
> └────────────────────┘
> ┌────────────────────┐
> │       mem0*        │
> └────────────────────┘
> ┌────────────────────┐
> │      N/A           │ region0
> └────────────────────┘
> *HDM decoder in mem0 is not committed.
>                                        
>                
> Pattern 3:
> Region0 is a child of the soft reserved area. In this case, the soft reserved area will
> not be removed from the iomem tree, resulting in being unable to be recreated later after destroy.
> ┌────────────────────┐
> │       CFMW         │
> └────────────────────┘
> ┌────────────────────┐
> │   reserved         │
> └────────────────────┘
> ┌────────────────────┐
> │ mem0    | mem1*    │
> └────────────────────┘
> ┌────────────────────┐
> │region0  |  N/A     │ region1
> └────────────────────┘
> *HDM decoder in mem1 is not committed.
>
>
> Thanks
> Zhijian
>
>
>
> On 04/04/2025 02:33, Terry Bowman wrote:
>> Add the ability to manage SOFT RESERVE iomem resources prior to them being
>> added to the iomem resource tree. This allows drivers, such as CXL, to
>> remove any pieces of the SOFT RESERVE resource that intersect with created
>> CXL regions.
>>
>> The current approach of leaving the SOFT RESERVE resources as is can cause
>> failures during hotplug of devices, such as CXL, because the resource is
>> not available for reuse after teardown of the device.
>>
>> The approach is to add SOFT RESERVE resources to a separate tree during
>> boot. This allows any drivers to update the SOFT RESERVE resources before
>> they are merged into the iomem resource tree. In addition a notifier chain
>> is added so that drivers can be notified when these SOFT RESERVE resources
>> are added to the ioeme resource tree.
>>
>> The CXL driver is modified to use a worker thread that waits for the CXL
>> PCI and CXL mem drivers to be loaded and for their probe routine to
>> complete. Then the driver walks through any created CXL regions to trim any
>> intersections with SOFT RESERVE resources in the iomem tree.
>>
>> The dax driver uses the new soft reserve notifier chain so it can consume
>> any remaining SOFT RESERVES once they're added to the iomem tree.
>>
>> V3 updates:
>>   - Remove srmem resource tree from kernel/resource.c, this is no longer
>>     needed in the current implementation. All SOFT RESERVE resources now
>>     put on the iomem resource tree.
>>   - Remove the no longer needed SOFT_RESERVED_MANAGED kernel config option.
>>   - Add the 'nid' parameter back to hmem_register_resource();
>>   - Remove the no longer used soft reserve notification chain (introduced
>>     in v2). The dax driver is now notified of SOFT RESERVED resources by
>>     the CXL driver.
>>
>> v2 updates:
>>   - Add config option SOFT_RESERVE_MANAGED to control use of the
>>     separate srmem resource tree at boot.
>>   - Only add SOFT RESERVE resources to the soft reserve tree during
>>     boot, they go to the iomem resource tree after boot.
>>   - Remove the resource trimming code in the previous patch to re-use
>>     the existing code in kernel/resource.c
>>   - Add functionality for the cxl acpi driver to wait for the cxl PCI
>>     and me drivers to load.
>>
>> Nathan Fontenot (4):
>>    kernel/resource: Provide mem region release for SOFT RESERVES
>>    cxl: Update Soft Reserved resources upon region creation
>>    dax/mum: Save the dax mum platform device pointer
>>    cxl/dax: Delay consumption of SOFT RESERVE resources
>>
>>   drivers/cxl/Kconfig        |  4 ---
>>   drivers/cxl/acpi.c         | 28 +++++++++++++++++++
>>   drivers/cxl/core/Makefile  |  2 +-
>>   drivers/cxl/core/region.c  | 34 ++++++++++++++++++++++-
>>   drivers/cxl/core/suspend.c | 41 ++++++++++++++++++++++++++++
>>   drivers/cxl/cxl.h          |  3 +++
>>   drivers/cxl/cxlmem.h       |  9 -------
>>   drivers/cxl/cxlpci.h       |  1 +
>>   drivers/cxl/pci.c          |  2 ++
>>   drivers/dax/hmem/device.c  | 47 ++++++++++++++++----------------
>>   drivers/dax/hmem/hmem.c    | 10 ++++---
>>   include/linux/dax.h        | 11 +++++---
>>   include/linux/ioport.h     |  3 +++
>>   include/linux/pm.h         |  7 -----
>>   kernel/resource.c          | 55 +++++++++++++++++++++++++++++++++++---
>>   15 files changed, 202 insertions(+), 55 deletions(-)
>>
>>
>> base-commit: aae0594a7053c60b82621136257c8b648c67b512

Powered by blists - more mailing lists