Message-ID: <aV3CsPl8Zsp5JtGu@aschofie-mobl2.lan>
Date: Tue, 6 Jan 2026 18:19:28 -0800
From: Alison Schofield <alison.schofield@...el.com>
To: Gregory Price <gourry@...rry.net>
CC: <linux-cxl@...r.kernel.org>, <linux-doc@...r.kernel.org>,
	<linux-kernel@...r.kernel.org>, <kernel-team@...a.com>, <dave@...olabs.net>,
	<jonathan.cameron@...wei.com>, <dave.jiang@...el.com>,
	<vishal.l.verma@...el.com>, <ira.weiny@...el.com>,
	<dan.j.williams@...el.com>, <corbet@....net>, <rakuram.e96@...il.com>,
	<alucerop@....com>
Subject: Re: [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug
 section

On Fri, Dec 19, 2025 at 12:05:38PM -0500, Gregory Price wrote:
> Describe cxl memory device hotplug implications, in particular how the
> platform CEDT CFMWS must be described to support successful hot-add of
> memory devices.

Who is the intended audience for this doc?

Maybe it's already in another section not being edited, but I'd
expect to see CXL spec references.

> 
> Reviewed-by: Jonathan Cameron <jonathan.cameron@...wei.com>
> Signed-off-by: Gregory Price <gourry@...rry.net>
> ---
> v3: Jonathan updates, change some italics to bold, add some bits about
>     Linux's expectations for BIOS/EFI behavior at runtime.
> 
>  Documentation/driver-api/cxl/index.rst        |   1 +
>  .../driver-api/cxl/platform/bios-and-efi.rst  |   3 +
>  .../cxl/platform/device-hotplug.rst           | 130 ++++++++++++++++++
>  3 files changed, 134 insertions(+)
>  create mode 100644 Documentation/driver-api/cxl/platform/device-hotplug.rst
> 
> diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
> index c1106a68b67c..5a734988a5af 100644
> --- a/Documentation/driver-api/cxl/index.rst
> +++ b/Documentation/driver-api/cxl/index.rst
> @@ -30,6 +30,7 @@ that have impacts on each other.  The docs here break up configurations steps.
>     platform/acpi
>     platform/cdat
>     platform/example-configs
> +   platform/device-hotplug
>  
>  .. toctree::
>     :maxdepth: 2
> diff --git a/Documentation/driver-api/cxl/platform/bios-and-efi.rst b/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> index 9034c206cf8e..a4b44c018f09 100644
> --- a/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> +++ b/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> @@ -49,6 +49,9 @@ up without requiring CXL driver support.  These platform vendors should
>  test their configurations with the existing CXL driver and provide driver
>  support for their auto-configurations if features like RAS are required.
>  
> +Platforms requiring boot-time programming and/or locking of CXL fabric
> +components may prevent features, such as device hot-plug, from working.
> +
>  UEFI Settings
>  =============
>  If your platform supports it, the :code:`uefisettings` command can be used to
> diff --git a/Documentation/driver-api/cxl/platform/device-hotplug.rst b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> new file mode 100644
> index 000000000000..e4a065fdd3ec
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> @@ -0,0 +1,130 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +CXL Device Hotplug
> +==================
> +
> +Device hotplug refers to *physical* hotplug of a device (addition or removal
> +of a physical device from the machine).
> +
> +BIOS/EFI software is expected to configure sufficient resources **at boot
> +time** to allow hotplugged devices to be configured by software (such as
> +proximity domains, HPA regions, and host-bridge configurations).
> +
> +BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
> +devices at hotplug time (i.e. HDM decoders should be left unprogrammed).
> +
> +This document covers some examples of those resources, but should not
> +be considered exhaustive.

Disclaimer noted, but as soon as you start writing about it, questions
arise. 

> +
> +Hot-Remove
> +==========
> +Hot removal of a device typically requires careful removal of software
> +constructs (memory regions, associated drivers) which manage these devices.
> +
> +Hard-removing a CXL.mem device without carefully tearing down driver stacks
> +is likely to cause the system to machine-check (or at least SIGBUS if memory
> +access is limited to user space).
> +
> +Memory Device Hot-Add
> +=====================
> +A device present at boot may be associated with a CXL Fixed Memory Window
> +reported in :doc:`CEDT<acpi/cedt>`.  That CFMWS may match the size of the
> +device, but the construction of the CEDT CFMWS is platform-defined.
> +
> +Hot-adding a memory device requires this pre-defined, **static** CFMWS to
> +have sufficient HPA space to describe that device.
> +

Isn't it more like the usable capacity of the hot added device will be 
limited to the HPA space described in the CFMWS?

a bit similar comment below -


> +There are a few common scenarios to consider.
> +
> +Single-Endpoint Memory Device Present at Boot
> +---------------------------------------------
> +A device present at boot likely had its capacity reported in the
> +:doc:`CEDT<acpi/cedt>`.  If a device is removed and a new device hotplugged,
> +the capacity of the new device will be limited to the original CFMWS capacity.
> +
> +Adding capacity larger than the original device will cause memory region
> +creation to fail if the region size is greater than the CFMWS size.

While it’s true that the CFMWS bounds the maximum HPA space available,
hotplugging a device with larger physical capacity doesn’t necessarily
cause region creation to fail outright. It only fails if the requested
region size exceeds the CFMWS size. Users can still create smaller
regions that fit within the existing CFMWS window, even if the device
itself has additional unused capacity

> +
> +The CFMWS is **static** and cannot be adjusted.  Platforms which may expect
> +different sized devices to be hotplugged must allocate sufficient CFMWS space
> +**at boot time** to cover all future expected devices.

Yes. Above is crisp. Might be crisper by suggesting that the 'sufficient'
CFMWS space could be achieved with one monster CFMWS or multiple CFMWS of
all their imaginable capacities.

Like my first comment, I'm not clear on why we are dipping our toe in
here, when understanding requires CXL spec, platform provider, and more
guidance. Are we describing anything Linux specific here.

-- Alison


> +
> +Multi-Endpoint Memory Device Present at Boot
> +--------------------------------------------
> +Non-switch-based Multi-Endpoint devices are outside the scope of what the
> +CXL specification describes, but they are technically possible. We describe
> +them here for instructive reasons only - this does not imply Linux support.
> +
> +A hot-plug capable CXL memory device, such as one which presents multiple
> +expanders as a single large-capacity device, should report the **maximum
> +possible capacity** for the device at boot. ::
> +
> +                  HB0
> +                  RP0
> +                   |
> +     [Multi-Endpoint Memory Device]
> +              _____|_____
> +             |          |
> +        [Endpoint0]   [Empty]
> +
> +
> +Limiting the size to the capacity preset at boot will limit hot-add support
> +to replacing capacity that was present at boot.
> +
> +No CXL Device Present at Boot
> +-----------------------------
> +When no CXL memory device is present on boot, some platforms omit the CFMWS
> +in the :doc:`CEDT<acpi/cedt>`.  When this occurs, hot-add is not possible.
> +
> +This describes the base case for any given device not being present at boot.
> +If a future possible device is not described in the CEDT at boot, hot-add
> +of that device is either limited or not possible.
> +
> +For a platform to support hot-add of a full memory device, it must allocate
> +a CEDT CFMWS region with sufficient memory capacity to cover all future
> +potentially added capacity (along with any relevant CEDT CHBS entry).
> +
> +To support memory hotplug directly on the host bridge/root port, or on a switch
> +downstream of the host bridge, a platform must construct a CEDT CFMWS at boot
> +with sufficient resources to support the max possible (or expected) hotplug
> +memory capacity. ::
> +
> +         HB0                 HB1
> +      RP0    RP1             RP2
> +       |      |               |
> +     Empty  Empty            USP
> +                      ________|________
> +                      |    |    |     |
> +                     DSP  DSP  DSP   DSP
> +                      |    |    |    |
> +                         All  Empty
> +
> +For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
> +a pre-configured amount of memory capacity (per host bridge, or host bridge
> +interleave set), even if no device is attached to Root Ports or Downstream
> +Ports at boot (as depicted in the figure above).
> +
> +
> +Interleave Sets
> +===============
> +
> +Host Bridge Interleave
> +----------------------
> +Host-bridge interleaved memory regions are defined **statically** in the
> +:doc:`CEDT<acpi/cedt>`.  To apply cross-host-bridge interleave, a CFMWS entry
> +describing that interleave must have been provided **at boot**.  Hotplugged
> +devices cannot add host-bridge interleave capabilities at hotplug time.
> +
> +See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
> +example to see how a platform can provide this kind of flexibility regarding
> +hotplugged memory devices.  BIOS/EFI software should consider options to
> +present flexible CEDT configurations with hotplug support.
> +
> +HDM Interleave
> +--------------
> +Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
> +can be re-programmed after hotplug.
> +
> +To add or remove a device to/from an existing HDM-applied interleaved region,
> +that region must be torn down an re-created.
> -- 
> 2.52.0
> 
> 

Powered by blists - more mailing lists