Message-ID: <17128669.tgchFWduMW@fdefranc-mobl3>
Date: Thu, 17 Jul 2025 16:14:13 +0200
From: "Fabio M. De Francesco" <fabio.m.de.francesco@...ux.intel.com>
To: Gregory Price <gourry@...rry.net>
Cc: Dave Jiang <dave.jiang@...el.com>, linux-cxl@...r.kernel.org,
 Davidlohr Bueso <dave@...olabs.net>,
 Jonathan Cameron <jonathan.cameron@...wei.com>,
 Alison Schofield <alison.schofield@...el.com>,
 Vishal Verma <vishal.l.verma@...el.com>, Ira Weiny <ira.weiny@...el.com>,
 Dan Williams <dan.j.williams@...el.com>, Jonathan Corbet <corbet@....net>,
 linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org
Subject:
 Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS,
 LMH, ED

On Monday, July 7, 2025 9:55:03 PM Central European Summer Time Gregory Price wrote:
> On Fri, Jul 04, 2025 at 12:05:33PM +0200, Fabio M. De Francesco wrote:
> > On Thursday, July 3, 2025 9:40:21 PM Central European Summer Time Gregory Price wrote:
> > > On Tue, Jul 01, 2025 at 08:23:57AM -0700, Dave Jiang wrote:
> > > > 
> > > > 
> > > > On 6/23/25 12:19 PM, Gregory Price wrote:
> > > > > On Mon, Jun 23, 2025 at 05:29:02PM +0200, Fabio M. De Francesco wrote:
> > > > >> Add documentation on how to resolve conflicts between CXL Fixed Memory
> > > > >> Windows, Platform Memory Holes, and Endpoint Decoders.
> > > > >>
> > > > >> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@...ux.intel.com>
> > > > > 
> > > > > I won't block a doc update on a suggestion so
> > > > > 
> > > > > Reviewed-by: Gregory Price <gourry@...rry.net>
> > > > > 
> > > > >> +Platform Firmware (BIOS) might reserve part of physical addresses below
> > > > >> +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
> > > > >> +or a requirement for the greater than 8 way interleave CXL regions starting
> > > > >> +at address 0). In that case the Window Size value cannot be anymore
> > > > >> +constrained to the NIW * 256 MB above-mentioned rule.
> > > > > 
> > > > > It might be nice to have a diagram that explains this visually, as it's
> > > > > difficult for me to understand the implications through words alone...
> > > > 
> > > > +1 on request for diagram to explain. We should try to document this issue as clearly as possible. Thank you.
> > > >
> > > 
> > > At the very least, it would be nice to have an explicitly example that
> > > explains the expected cfmws/decoder configurations that are valid but
> > > "technically" violate the spec
> > > 
> > > I *think* this basically boils down to "CFMWS size is not aligned, but
> > > all the decoders it targets are"?  If I understand this correctly?
> > > 
> > Yes, this is the intended meaning. 
> > 
> > E.g, two windows, 384 GB total memory, LMH at 2 GB:
> > 
> > Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> >   0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
> >   1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
> 
> 
> In this example, does the root decoder have 0/2 and the host bridge
> decoder have 0/3?  Because root decoders objects are created by reading
> the CFMWS Base/Size - while host bridge decoder objects are created by
> reading the host bridge DVSECS.
>
> Or does the linux code read CFMWS Base/Size and create a root decoder
> with 0/3 because "that's what it should have"?  etc
>
No, sorry for any confusion. I also noticed that some unwanted additional
lines slipped in my last reply, probably for a copy/paste mistake.

Let me explain...

The table above shows a real configuration copied from an x86 platform 
where the Low Memory Hole (LMH) starts at 2GB. 

The"HDM Decoder Base/Size" refers specifically to the CXL Endpoint 
Decoders HPA range Base/Size. The first row of the table describes the 
first window (CFMWS[0]), whose HPA rage base/size is 0/2GB, and the 
Endpoint Decoder that the CXL driver should match with that CFMWS, 
whose HPA range base/size is 0/3GB.

The Endpoint Decoder follows the 256MB x 12 interleave ways rule 
mandated by the CXL specifications. Instead the CFMWS is truncated 
immediately before the beginning of the LMH, because it needs to publish
only those memory locations that the system can address.   

Currently the CXL driver fails during the Region construction and 
subsequent Endpoint Decoders attachment because it cannot match Root 
Decoders (that are created with HPA ranges base/size equal to CFMWS)
and Switch Decoders with Endpoint decoders due to this size discrepancy. 

The driver expects that the Endpoint Decoders HPA ranges to be contained 
into their corresponding Root Decoders. Furthermore, Linux fails to 
attach Endpoint decoders to already constructed CXL Regions because of 
the same size discrepancy issue. 
> 
> I think you need to describe what the expected behavior is for what linux
> will produce in terms of the decoder objects given the above.
>
The expected behavior is that Linux should be able to match the Endpoint 
Decoder with the Root Decoder range even if the CFMWS size is smaller 
than the Decoder's, as long as the latter adheres to the 256MB * interleave 
ways rule. Furthermore, Linux should be able to match the Endpoint decoders 
with already constructed CXL Regions and allow the attachment process to 
succeed. 

This would enable the utilization of the 2GB memory described by CFMWS[0].

I hope this time I explained the LMH interference with CXL Regions 
construction and Endpoint Decoders attachment with better clarity.

If this explanation suffices, I will incorporate it into the next version
of this patch and also explain that "HDM Decoder" stands for Endpoint Decoder 
and that the CFMWS HPA base/size describes the System Physical Address (SPA) 
which the CXL driver uses to make Root Decoders HPA range base/size. 

Thanks,

Fabio

Powered by blists - more mailing lists