Message-ID: <4d66eac3-1a2b-4d9d-8a9a-529a19758439@kernel.org>
Date: Thu, 22 Jan 2026 23:14:15 +0100
From: "David Hildenbrand (Red Hat)" <david@...nel.org>
To: Gregory Price <gourry@...rry.net>, linux-cxl@...r.kernel.org
Cc: dan.j.williams@...el.com, dave.jiang@...el.com,
 jonathan.cameron@...wei.com, alison.schofield@...el.com,
 ira.weiny@...el.com, dave@...olabs.net, linux-kernel@...r.kernel.org,
 kernel-team@...a.com, vishal.l.verma@...el.com, benjamin.cheatham@....com,
 David Rientjes <rientjes@...gle.com>
Subject: Re: cxl/region.c improvements and DAX/Hotplug plumbing

This is a lot of stuff. In which meeting would you usually discuss these 
things?

Some of that (especially the interaction with core-mm) feels like it 
would be a good fit to discuss with he wider MM community in one of the 
bi-weekly mm meeting. (CCing David R.)

> My list of current discrete steps (some serial, some parallel):
> 
>     1) Internally formalize cxl_region.region_driver (no ABI exposure)
> 
>     3) Plumb additional information through to DAX based on driver
>        - dax-driver mode preference
>        - uuid for tagged capacity
> 
>     2) Create explicit sysram_driver
>        - Write in terms of DCD
>        - Tagged Extents:   use DAX glue to manage set of tagged extents
>        - Untagged Extents: Hotplug and manage directly
>        - new ABI: `region0/region_driver` - switch between [dax,sysram]
> 
>     4) Plumb additional hotplug policy from CXL into DAX and MHP
>        - dax0.0/hotplug  (atomic operation on all blocks)
>        - cxl region auto-online policy (region0/rctl/auto-online)
>        - block-protection policy? (memory_notifier controls)
>        - hiding memory blocks? (discussed in last meeting)

What is that about and what was the result of that discussion? :)

>        - ABI: `region0/rctl/*` controls
> 
>     5) Formalize DCD dax_region driver use
>        - each extent list = new dax device in devdax mode
>        - tags enforced to be globally unique
>        - dax_region.add_extents(tag, extent_list)
>            -> create new daxN.0
>            -> expose daxN.0/uuid
>        - dax_region.remove_extents(extent_list)
>        - dax_region.remove_tagged_extents(tag)
> 
>     6) Formalize DCD sysram_region driver use
>        - sysram_region.add_extents(tag, extent_list)
>            -> untagged capacity managed as individual memory blocks
>            -> tagged capacity managed with DAX glue
>        - sysram_region.remove_extents(extent_list) (untagged)
>        - sysram_region.remove_tagged_extents(tag)  (tagged)
> 
>     7) Add private_region infrastructure
>        - private_region driver design
>        - N_PRIVATE_MEMORY infrastructure
>        - derivative driver (in my case compressed memory)
>        - Probably wants memory_blocks hiding and/or retricted operations
> 

[...]

> ---------------------------------------------------------
> Problem: SysRAM Auto-Hotplug policy is too broadly scoped
> ---------------------------------------------------------
>    Hotplug SYSRAM indirection through DAX leads to complex auto-online
>    interactions and/or current policy options are too broad in scope.
>    (e.g. MHP_AUTO_ONLINE build option is bad cross-platform)
> 
>    Solution 1: Plumb auto-online policy from cxl_region into dax_kmem
> 
>      Build Options:
>         Default auto-online policy for auto-regions?
>         Moves scope from MHP-Global to CXL-local
> 
>      ABI: dax_region - regionN/rctl/auto-online
>         Gives the region creator a chance to define before probe()
> 
>    Solution 2:  Make a dedicated sysram_region with policy

What kind of region would that be?

> 
>    May want both solutions longer term (for tagged DCD capacity)
> 
>    ndctl extension:
>         cxl create-region --driver=sysram --auto-online=movable ?
> 


[...]

> ---------------------------------------------------------------
> Problem/Annoyance: DAX kmem per-block operation race conditions
> ---------------------------------------------------------------
>    DAX exposes SYSRAM regions as individual memory blocks, which
>    creates race conditions when trying to manage a set of blocks.
> 
>    Example: udev can have an auto-onlining policy that twiddles
>             memory_block bits while cxl driver is trying to unplug.
> 
>    Affects: DCD, SysRAM, potentially N_PRIVATE_MEMORY
> 
>    Solution 1: [unplug, online, online_movable] > dax0.0/hotplug
>                Does operation on all blocks under the hotplug lock.
> 
>    Solution 2: dedicated sysram_region driver w/ or w/o DAX.
>                Can support sparseness w/o DAX (see DCD problem)
> 	      Could use DAX for tagged DCD regions.
>                Tradeoff: May duplicate some DAX logic.

How would that look like?

> 
>    Solution 3: Hide nodeN/memory_block's w/ MHP Flag.
>                Issue: Possibly userland breaking.

Hacky. :)

> 
>    Solution 4: Prevent non-driver actions from changing state.
>                Also solves hotplug protection problem (see next)

The crucial part is solving what you spelled out in the description: 
"race conditions". Forbidding someone to re-configure system RAM sounds 
unnecessary.

For example, I use it a lot for testing issues with page migration while 
offlining memory from ZONE_MOVABLE.

> 
> Patch: Implements solution 1
> https://lore.kernel.org/linux-cxl/20260114235022.3437787-5-gourry@gourry.net/
> 
> --------------------------------------------------------------
> Problem: SYSRAM or N_PRIVATE want memory_block policy controls
> --------------------------------------------------------------
>    A SYSRAM or N_PRIVATE region may have an implied zone-policy to
>    protect - or N_PRIVATE blocks may want to restrict any operation.

Why is N_PRIVATE special here?

> 
>    Privileged userspace action could do this:
>      cat memoryN/state              => online_movable
>      cat memoryN/valid_zones        => movable
>      echo offline > memoryN/state   => offline
>      echo online > memoryN/state    => online
>      cat memoryN/valid_zones        => normal
> 
>    - A DCD driver wants to try to protect hotpluggability.
>    - userspace has no business twiddling private_region blocks.

Why?

> 
>    Solution: Prevent non-driver actions from changing state.

If you can handle race conditions properly, why disallow offline + 
re-online, for example? Sure, you could restrict the zone.

> 
>        Essentially, add memory_notifier to region_driver or DAX
>        that rejects operations according to driver-defined policy.
> 
>    May not require explicit, could be encoded in default region
>    driver policy (e.g. dcd implies protection).
> 
> Example Patch:
> https://lore.kernel.org/linux-cxl/20260114235022.3437787-6-gourry@gourry.net/


[...]

> ---------------------------------------------
> Problem: "Special" Device memory usage policy
> ---------------------------------------------
>     Memory devices may have special features that dictate use patterns.
>     They may also prefer using core mm/ services for basic operation.
>     (page_alloc, reclaim, migration, etc)
>     
>     But: This memory shouldn't be exposed as "Normal System RAM".
> 
>     Solution: N_PRIVATE_MEMORY node_state
> 
>     CXL Driver Piece: private_region driver
>         These drivers would know how to register N_PRIVATE_MEMORY
>         Would also allow device-specific usage behavior to be written.
>         Would likely be used by upper layer drivers rather than uapi.
> 
>     Example:  Compressed Memory
> 
>       general service can use page_alloc() for get_page_from_freelist()
>       region_driver registers memory on a compressed memory node
>       vmscan.c/memory-tiers.c calls back to driver to handle migration
> 
>     Example:  Accelerator Memory Region
> 
>        Accel library/drive does node-based allocs.
>        Driver callbacks might include write-faults (ZONE_DEVICE-esque
>          pattern that passes page ownership between CPU/GPU)
> 
>        Either way, driver applies mapping policy w/o accounting cargo
> 
>     Example:  Slow(er) memory
>        Some memory is "just memory", but might be particularly slow and
>        intended for use as a filesystem backend or as only a demotion
>        target.  Otherwise its allocated / mapped like any other memory,
>        but it still required isolation so isolated to the demotion path
>        and not a fallback allocation target

That doesn't quite fit the description of N_PRIVATE_MEMORY, though. Or 
what am I missing?


-- 
Cheers

David

Powered by blists - more mailing lists