Message-ID: <8047110.oPnVEEUbh3@fdefranc-mobl3>
Date: Tue, 22 Jul 2025 13:42:35 +0200
From: "Fabio M. De Francesco" <fabio.m.de.francesco@...ux.intel.com>
To: Gregory Price <gourry@...rry.net>
Cc: Dave Jiang <dave.jiang@...el.com>, linux-cxl@...r.kernel.org,
 Davidlohr Bueso <dave@...olabs.net>,
 Jonathan Cameron <jonathan.cameron@...wei.com>,
 Alison Schofield <alison.schofield@...el.com>,
 Vishal Verma <vishal.l.verma@...el.com>, Ira Weiny <ira.weiny@...el.com>,
 Dan Williams <dan.j.williams@...el.com>, Jonathan Corbet <corbet@....net>,
 linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org
Subject:
 Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS,
 LMH, ED

On Monday, July 21, 2025 2:51:14 AM Central European Summer Time Gregory Price wrote:
> On Thu, Jul 17, 2025 at 04:14:13PM +0200, Fabio M. De Francesco wrote:
> > The table above shows a real configuration copied from an x86 platform 
> > where the Low Memory Hole (LMH) starts at 2GB. 
> > 
> > The"HDM Decoder Base/Size" refers specifically to the CXL Endpoint 
> > Decoders HPA range Base/Size. The first row of the table describes the 
> > first window (CFMWS[0]), whose HPA rage base/size is 0/2GB, and the 
> > Endpoint Decoder that the CXL driver should match with that CFMWS, 
> > whose HPA range base/size is 0/3GB.
> 
> The only thing i ask is being more precise with decoder references.
> 
> HDM Decoder can refer to any of: root, switch, hb, or endpoint decoders.
> 
> Below you make this distinct in the explanation, but in the table it's
> simply general "HDM Decoder".  All I ask is for a bit more clarity on
> what decoder will contain what values to avoid further ambiguity.
> 
I agree, "HDM Decoder" was ambiguous. I will relabel it to "(intermediate) 
Switch and Endpoint Decoders".
>
> > The driver expects that the Endpoint Decoders HPA ranges to be contained 
> > into their corresponding Root Decoders. Furthermore, Linux fails to 
> > attach Endpoint decoders to already constructed CXL Regions because of 
> > the same size discrepancy issue. 
> > > 
> > > I think you need to describe what the expected behavior is for what linux
> > > will produce in terms of the decoder objects given the above.
> > >
> > The expected behavior is that Linux should be able to match the Endpoint 
> > Decoder with the Root Decoder range even if the CFMWS size is smaller 
> > than the Decoder's, as long as the latter adheres to the 256MB * interleave 
> > ways rule. Furthermore, Linux should be able to match the Endpoint decoders 
> > with already constructed CXL Regions and allow the attachment process to 
> > succeed. 
> > 
> 
> You may also need to describe more than just the contents of the
> endpoint decoder.  What would the content of any intermediate decoders
> be (matching the root or matching the endpoint?).
>
I think that the output of the simulation of a hole in the cxl_test 
mocked CXL topology, with a CFMWS HPA size of 3 * 256MiB, and Switch and 
Endpoint Decoders HPA sizes of 4 * 256MiB can help to clarify. The Region
Interleave ways is 2. 

In this example the CXL driver is patched and correctly deals with a 
simulated LMH that trimmed the CFMWS[0] range size and made it not aligned
to 256MiB * IW.

[root@...ora ndctl]# cxl list -RDu -r5
[
  {
    "root decoders":[
      {
        "decoder":"decoder9.0",
        "resource":"0x3ff010000000",
        "size":"768.00 MiB (805.31 MB)",
        "interleave_ways":1,
        "max_available_extent":0,
        "volatile_capable":true,
        "qos_class":42,
        "nr_targets":1,
        "regions:decoder9.0":[
          {
            "region":"region5",
            "resource":"0x3ff010000000",
            "size":"768.00 MiB (805.31 MB)",
            "type":"ram",
            "interleave_ways":2,
            "interleave_granularity":4096,
            "decode_state":"commit"
          }
        ]
      }
    ]
  },
  {
    "port decoders":[
      {
        "decoder":"decoder10.0",
        "resource":"0x3ff010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":1,
        "region":"region5",
        "nr_targets":1
      },
      {
        "decoder":"decoder14.0",
        "resource":"0x3ff010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region5",
        "nr_targets":2
      }
    ]
  },
  {
    "endpoint decoders":[
      {
        "decoder":"decoder28.0",
        "resource":"0x3ff010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region5",
        "dpa_resource":"0",
        "dpa_size":"384.00 MiB (402.65 MB)",
        "mode":"ram"
      },
      {
        "decoder":"decoder23.0",
        "resource":"0x3ff010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region5",
        "dpa_resource":"0",
        "dpa_size":"384.00 MiB (402.65 MB)",
        "mode":"ram"
      }
    ]
  }
]

The construction of Regions and subsequent attachment of Switch and 
Endpoint Decoders is based on matching Root Decoders and existing Regions
with Switch and Endpoint Decoders. If these objects can't be matched 
between them, the Regions can't be constructed and/or attached with 
the Switch and Endpoint Decoders.     

The CXL driver can always match Endpoint Decoders with Switch Decoders 
because their HPA range sizes are the same regardless of LMH's.  

But with LMH's that trim the CFMWS HPA range to smaller sizes, a 
non-patched CXL driver can't match the Root Decoders and Regions with the 
Switch and Endpoint decoders. Therefore, the CXL Region construction and 
Decoders attachment can't succeed.
      }
    ]
  }
]

The construction of Regions and subsequent attachment of Switch and 
Endpoint Decoders is based on matching Root Decoders and existing Regions
with Switch and Endpoint Decoders. If these objects can't be matched 
between them, the Regions can't be constructed and/or attached with 
the Switch and Endpoint Decoders.     

Anyway, the CXL driver can always match Endpoint Decoders with Switch
Decoders because the HPA range sizes are the same also when LMH's are  

But with an LMH that trim the CFMWS HPA range size to a smaller size, a 
non-patched CXL driver can't match the Root Decoders and Regions with the 
Switch and Endpoint decoders. Therefore, the CXL Region construction and 
Decoders attachment can't succeed.
>
> > If this explanation suffices, I will incorporate it into the next version
> > of this patch and also explain that "HDM Decoder" stands for Endpoint Decoder 
> > and that the CFMWS HPA base/size describes the System Physical Address (SPA) 
> > which the CXL driver uses to make Root Decoders HPA range base/size. 
> > 
> 
> This explanation is better, just need a few more bits of data and I
> think you're good to go.
> 
Thanks,

Fabio

Fabio
>
> ~Gregory
> 

Powered by blists - more mailing lists