Message-Id: <20230604-dcd-type2-upstream-v2-0-f740c47e7916@intel.com>
Date:   Mon, 28 Aug 2023 22:20:51 -0700
From:   Ira Weiny <ira.weiny@...el.com>
To:     Dan Williams <dan.j.williams@...el.com>
Cc:     Navneet Singh <navneet.singh@...el.com>,
        Fan Ni <fan.ni@...sung.com>,
        Jonathan Cameron <Jonathan.Cameron@...wei.com>,
        Davidlohr Bueso <dave@...olabs.net>,
        Dave Jiang <dave.jiang@...el.com>,
        Alison Schofield <alison.schofield@...el.com>,
        Vishal Verma <vishal.l.verma@...el.com>,
        Ira Weiny <ira.weiny@...el.com>, linux-cxl@...r.kernel.org,
        linux-kernel@...r.kernel.org
Subject: [PATCH RFC v2 00/18] DCD: Add support for Dynamic Capacity Devices
 (DCD)

A Dynamic Capacity Device (DCD) (CXL 3.0 spec 9.13.3) is a CXL memory
device that implements dynamic capacity.  Dynamic capacity feature
allows memory capacity to change dynamically, without the need for
resetting the device.

Even though this is marked v2 by b4, this is effectively a whole new
series for DCD support.  Quite a bit of the core support was completed
by Navneet in [4].  However, the architecture through the CXL region,
DAX region, and DAX Device layers is completely different.  Particular
attention was paid to:

	1) managing skip resources in the hardware device
	2) ensuring the host OS only sent a release memory mailbox
	   response when all DAX devices are done using an extent
	3) allowing dax devices to span extents
	4) allowing dax devices to use parts of extents

I could say all of the review comments from v1 are addressed but frankly
the series has changed so much that I can't guarantee anything.

The series continues to be based on the type-2 work posted from Dan.[2]
However, my branch with that work is a bit dated.  Therefore I have
posted this series on github here.[5]

Testing was sped up with cxl-test and ndctl dcd support.  A preview of
that work is on github.[6]  In addition Fan Ni's Qemu DCD series was
used part of the time.[3]

The major parts of this series are:

- Get the dynamic capacity (DC) region information from cxl device
- Configure device DC regions reported by hardware
- Enhance CXL and DAX regions for DC
	a. maintain separation between the hardware extents and the CXL
	   region extents to provide for the addition of interleaving in
	   the future.
- Get and maintain the hardware extent lists for each device via an
  initial extent list and DC event records
        a. Add capacity Events
	b. Add capacity response
	b. Release capacity events
	d. Release capacity response
- Notify region layers of extent changes
- Allow for DAX devices to be created on extents which are surfaced
- Maintain references on extents which are in use
	a. Send Release capacity Response only when DAX devices are not
	   using memory
- Allow DAX region extent labels to change to allow for flexibility in
  DAX device creation in the future (further enhancements are required
  to ndctl for this)
- Trace Dynamic Capacity events
- Add cxl-test infrastructure to allow for faster unit testing

To: Dan Williams <dan.j.williams@...el.com>
Cc: Navneet Singh <navneet.singh@...el.com>
Cc: Fan Ni <fan.ni@...sung.com>
Cc: Jonathan Cameron <Jonathan.Cameron@...wei.com>
Cc: Davidlohr Bueso <dave@...olabs.net>
Cc: Dave Jiang <dave.jiang@...el.com>
Cc: Alison Schofield <alison.schofield@...el.com>
Cc: Vishal Verma <vishal.l.verma@...el.com>
Cc: Ira Weiny <ira.weiny@...el.com>
Cc: linux-cxl@...r.kernel.org
Cc: linux-kernel@...r.kernel.org

[1] https://lore.kernel.org/all/64326437c1496_934b2949f@dwillia2-mobl3.amr.corp.intel.com.notmuch/
[2] https://lore.kernel.org/all/168592149709.1948938.8663425987110396027.stgit@dwillia2-xfh.jf.intel.com/
[3] https://lore.kernel.org/all/6483946e8152f_f1132294a2@iweiny-mobl.notmuch/
[4] https://lore.kernel.org/r/20230604-dcd-type2-upstream-v1-0-71b6341bae54@intel.com
[5] https://github.com/weiny2/linux-kernel/commits/dcd-v2-2023-08-28
[6] https://github.com/weiny2/ndctl/tree/dcd-region2

---
Changes in v2:
- iweiny: Complete rework of the entire series
- Link to v1: https://lore.kernel.org/r/20230604-dcd-type2-upstream-v1-0-71b6341bae54@intel.com

---
Ira Weiny (15):
      cxl/hdm: Debug, use decoder name function
      cxl/mbox: Flag support for Dynamic Capacity Devices (DCD)
      cxl/region: Add Dynamic Capacity decoder and region modes
      cxl/port: Add Dynamic Capacity mode support to endpoint decoders
      cxl/port: Add Dynamic Capacity size support to endpoint decoders
      cxl/region: Add Dynamic Capacity CXL region support
      cxl/mem: Read extents on memory device discovery
      cxl/mem: Handle DCD add and release capacity events.
      cxl/region: Expose DC extents on region driver load
      cxl/region: Notify regions of DC changes
      dax/bus: Factor out dev dax resize logic
      dax/region: Support DAX device creation on dynamic DAX regions
      tools/testing/cxl: Make event logs dynamic
      tools/testing/cxl: Add DC Regions to mock mem data
      tools/testing/cxl: Add Dynamic Capacity events

Navneet Singh (3):
      cxl/mem: Read Dynamic capacity configuration from the device
      cxl/mem: Expose device dynamic capacity configuration
      cxl/mem: Trace Dynamic capacity Event Record

 Documentation/ABI/testing/sysfs-bus-cxl |  56 ++-
 drivers/cxl/core/core.h                 |   1 +
 drivers/cxl/core/hdm.c                  | 215 ++++++++-
 drivers/cxl/core/mbox.c                 | 646 +++++++++++++++++++++++++-
 drivers/cxl/core/memdev.c               |  77 ++++
 drivers/cxl/core/port.c                 |  19 +
 drivers/cxl/core/region.c               | 418 +++++++++++++++--
 drivers/cxl/core/trace.h                |  65 +++
 drivers/cxl/cxl.h                       |  99 +++-
 drivers/cxl/cxlmem.h                    | 138 +++++-
 drivers/cxl/mem.c                       |  50 ++
 drivers/cxl/pci.c                       |   8 +
 drivers/dax/Makefile                    |   1 +
 drivers/dax/bus.c                       | 263 ++++++++---
 drivers/dax/bus.h                       |   1 +
 drivers/dax/cxl.c                       | 213 ++++++++-
 drivers/dax/dax-private.h               |  61 +++
 drivers/dax/extent.c                    | 133 ++++++
 tools/testing/cxl/test/mem.c            | 782 +++++++++++++++++++++++++++-----
 19 files changed, 3005 insertions(+), 241 deletions(-)
---
base-commit: c76cce37fb6f3796e8e146677ba98d3cca30a488
change-id: 20230604-dcd-type2-upstream-0cd15f6216fd

Best regards,
-- 
Ira Weiny <ira.weiny@...el.com>

Powered by blists - more mailing lists