linux-kernel - Re: [PATCH] cxl/region: Support multi-level interleaving with smaller granularities for lower levels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aQHkDFJsU0lg7NLT@rric.localdomain>
Date: Wed, 29 Oct 2025 10:53:16 +0100
From: Robert Richter <rrichter@....com>
To: Jonathan Cameron <jonathan.cameron@...wei.com>
Cc: Alison Schofield <alison.schofield@...el.com>,
	Vishal Verma <vishal.l.verma@...el.com>,
	Ira Weiny <ira.weiny@...el.com>,
	Dan Williams <dan.j.williams@...el.com>,
	Dave Jiang <dave.jiang@...el.com>,
	Davidlohr Bueso <dave@...olabs.net>, linux-cxl@...r.kernel.org,
	linux-kernel@...r.kernel.org, Gregory Price <gourry@...rry.net>,
	Terry Bowman <terry.bowman@....com>,
	Joshua Hahn <joshua.hahnjy@...il.com>
Subject: Re: [PATCH] cxl/region: Support multi-level interleaving with
 smaller granularities for lower levels

On 28.10.25 14:27:27, Jonathan Cameron wrote:
> On Tue, 28 Oct 2025 10:47:53 +0100
> Robert Richter <rrichter@....com> wrote:
> 
> > The CXL specification supports multi-level interleaving "as long as
> > all the levels use different, but consecutive, HPA bits to select the
> > target and no Interleave Set has more than 8 devices" (from 3.2).
> > 
> > Currently the kernel expects that a decoder's "interleave granularity
> > is a multiple of @parent_port granularity". That is, the granularity
> > of a lower level is bigger than those of the parent and uses the outer
> > HPA bits as selector. It works e.g. for the following 8-way config:
> > 
> >  * cross-link (cross-hostbridge config in CFMWS):
> >    * 4-way
> >    * 256 granularity
> >    * Selector: HPA[8:9]
> >  * sub-link (CXL Host bridge config of the HDM):
> >    * 2-way
> >    * 1024 granularity
> >    * Selector: HPA[10]
> > 
> > Now, if the outer HPA bits are used for the cross-hostbridge, an 8-way
> > config could look like this:
> > 
> >  * cross-link (cross-hostbridge config in CFMWS):
> >    * 4-way
> >    * 512 granularity
> >    * Selector: HPA[9:10]
> >  * sub-link (CXL Host bridge config of the HDM):
> >    * 2-way
> >    * 256 granularity
> >    * Selector: HPA[8]
> > 
> > The enumeration of decoders for this configuration fails then with
> > following error:
> > 
> >  cxl region0: pci0000:00:port1 cxl_port_setup_targets expected iw: 2 ig: 1024 [mem 0x10000000000-0x1ffffffffff flags 0x200]
> >  cxl region0: pci0000:00:port1 cxl_port_setup_targets got iw: 2 ig: 256 state: enabled 0x10000000000:0x1ffffffffff
> >  cxl_port endpoint12: failed to attach decoder12.0 to region0: -6
> > 
> > Note that this happens only if firmware is setting up the decoders
> > (CXL_REGION_F_AUTO). For userspace region assembly the granularities
> > are chosen to increase from root down to the lower levels. That is,
> > outer HPA bits are always used for lower interleaving levels.
> > 
> > Rework the implementation to also support multi-level interleaving
> > with smaller granularities for lower levels. Determine the interleave
> > set of autodetected decoders. Check that it is a subset of the root
> > interleave.
> > 
> > The HPA selector bits are extracted for all decoders of the set and
> > checked that there is no overlap and bits are consecutive. All
> > decoders can be programmed now to use any bit range within the
> > region's target selector.
> > 
> > Signed-off-by: Robert Richter <rrichter@....com>
> 
> We debated this back when the original support was added. At the time the
> discussion pretty much concluded there was no reason anyone would ever actually
> do this because it mostly doesn't provide much/any performance benefit and
> might result in significantly worse performance by not spreading out accesses as
> early as possible.
> 
> So do we have any idea why a bios is doing this? Something about the
> host interconnect?

There could be platform constraints that require the cross-hostbridge
logic to have bigger granularities. The sublinks have then a smaller
granularity. There are existing platforms with that config that would
be broken otherwise. It is not possible to handle this differently by
the bios and reconfigure that.

-Robert