Message-ID: <aV6BqOarpY0PUt7h@gourry-fedora-PF4VCD3F>
Date: Wed, 7 Jan 2026 10:54:16 -0500
From: Gregory Price <gourry@...rry.net>
To: Alison Schofield <alison.schofield@...el.com>
Cc: linux-cxl@...r.kernel.org, linux-doc@...r.kernel.org,
	linux-kernel@...r.kernel.org, kernel-team@...a.com,
	dave@...olabs.net, jonathan.cameron@...wei.com,
	dave.jiang@...el.com, vishal.l.verma@...el.com, ira.weiny@...el.com,
	dan.j.williams@...el.com, corbet@....net, rakuram.e96@...il.com,
	alucerop@....com
Subject: Re: [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug
 section

On Tue, Jan 06, 2026 at 06:19:28PM -0800, Alison Schofield wrote:
> On Fri, Dec 19, 2025 at 12:05:38PM -0500, Gregory Price wrote:
> > Describe cxl memory device hotplug implications, in particular how the
> > platform CEDT CFMWS must be described to support successful hot-add of
> > memory devices.
> 
> Who is the intended audience for this doc?
> 
> Maybe it's already in another section not being edited, but I'd
> expect to see CXL spec references.
>

The audience for this particular doc is platform folks who want to know
what limitations there are on cxl-device hotplug in linux.

This originated from a discussion at LPC where some device folks
understood the hotplug process, but did not understand the linux
limitations (in particular around the CFMWS and node creation).

Tl;dr: Linux consumes ACPI tables to generate NUMA nodes and constructs
memory regions.  This is linux "having an opinion" on "how ACPI tables
should reasonably be constructed" - and this tries to document that.

This is all obviously mutable, but it at least helps drive understanding
between device, bios, platform, and linux if we at least have this
document that says "this is what Linux wants/expects".

I'm not sure what if anything I would reference in the spec, since this
largely describes what linux wants, not what the spec says is possible.

Happy to improve.

> > +Memory Device Hot-Add
> > +=====================
> > +A device present at boot may be associated with a CXL Fixed Memory Window
> > +reported in :doc:`CEDT<acpi/cedt>`.  That CFMWS may match the size of the
> > +device, but the construction of the CEDT CFMWS is platform-defined.
> > +
> > +Hot-adding a memory device requires this pre-defined, **static** CFMWS to
> > +have sufficient HPA space to describe that device.
> > +
> 
> Isn't it more like the usable capacity of the hot added device will be 
> limited to the HPA space described in the CFMWS?
> 
> a bit similar comment below -
> 
> 

This is more precise and concise.  I can change that.

> > +Single-Endpoint Memory Device Present at Boot
> > +---------------------------------------------
> > +A device present at boot likely had its capacity reported in the
> > +:doc:`CEDT<acpi/cedt>`.  If a device is removed and a new device hotplugged,
> > +the capacity of the new device will be limited to the original CFMWS capacity.
> > +
> > +Adding capacity larger than the original device will cause memory region
> > +creation to fail if the region size is greater than the CFMWS size.
> 
> While it’s true that the CFMWS bounds the maximum HPA space available,
> hotplugging a device with larger physical capacity doesn’t necessarily
> cause region creation to fail outright. It only fails if the requested
> region size exceeds the CFMWS size. Users can still create smaller
> regions that fit within the existing CFMWS window, even if the device
> itself has additional unused capacity

This does say: "if the region size is greater than the CFMWS size"

But maybe it's more accurate to say "if the unused CFMWS HPA space is
smaller than a user-defined memory region capacity".

This is really all saying the same thing though - if your device
capacity is smaller than your CFMWS, you can't map the entire device.

> 
> > +
> > +The CFMWS is **static** and cannot be adjusted.  Platforms which may expect
> > +different sized devices to be hotplugged must allocate sufficient CFMWS space
> > +**at boot time** to cover all future expected devices.
> 
> Yes. Above is crisp. Might be crisper by suggesting that the 'sufficient'
> CFMWS space could be achieved with one monster CFMWS or multiple CFMWS of
> all their imaginable capacities.
> 
> Like my first comment, I'm not clear on why we are dipping our toe in
> here, when understanding requires CXL spec, platform provider, and more
> guidance. Are we describing anything Linux specific here.
> 

Linux specific piece here is the fact that once Linux proceeds past
__init, you cannot achieve hotplug for CXL memory not described (or
insufficiently described) in the ACPI tables.

In this case - memory is special.  We're (linux) basically telling
platform vendors "You can't add memory you haven't described in some way
since the beginning of time".

It's unclear that the same would be true on all operating systems, but
it is true on linux.

---

general note - i'm happy to drop this if folks think it's not the right
place to put it, but based on LPC discussions and feedback from Jonathan
and Dan, it seemed useful.

Maybe I should just be writing a book instead, lol. (Please no)

~Gregory

Powered by blists - more mailing lists