Message-Id: <20250117151033.1517882-1-Dave.Martin@arm.com>
Date: Fri, 17 Jan 2025 15:10:10 +0000
From: Dave Martin <Dave.Martin@....com>
To: linux-kernel@...r.kernel.org
Cc: linux-arm-kernel@...ts.infradead.org,
	Zeng Heng <zengheng4@...wei.com>,
	Shaopeng Tan <tan.shaopeng@...itsu.com>,
	James Morse <james.morse@....com>
Subject: [RFC PATCH v2 00/11] Introduce PARTID grouping support

(Was: Introduce flexible CLOSID/RMID translation) [5]

This series introduces an ID remapping scheme for mapping multiple MPAM
PARTIDs to each resctrl control group, and programs the intPARTID
appropriately on MSCs that require and support PARTID Narrowing.
Rudimentary support is included for detecting whether this
configuration will work correctly on the detected hardware, but this
detection is currently inflexible and may disable this multiple PARTID
mapping behaviour unnecessarily on some platforms.

(Feedback would be appreciated from anyone who knows of hardware where
this approach is insufficient.)


On supported platforms, this allows the number of monitoring groups per
resctrl control group to be scaled up by the value of the (provisional)
mpam.partid_per_closid kernel command-line parameter, at the expense of
reducing the number of control groups that can be created.

See "Background" below, and the subsequent sections, for more details.	

This series is not a complete implementation, but should be sufficient
to provide additional resctrl monitoring groups beyond the number of
PMGs supported by the hardware (with some limitations; see
"Limitations", below).


Based on:
git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.12-rc1
4d17b37b670795954fd7a70c8ec83fb705a5e2ad ("arm64: mpam: Restore the expected MPAM sysregs on cpuhp")


Changes Since RFCv1: [5]

---
DROPPED:

 * arm_mpam: Clean up config update checks in mpam_apply_config()

      (present in the base branch)

CHANGED:

 * arm_mpam: Introduce flexible CLOSID/RMID translation

    * [bug fix] Add CDP index properly in mpam_resctrl_partid_range().

      (Also reported independently by Shaopeng Tan.)

    * [relaxation] Remove requirement that mpam_pmg_max + 1 is a power
      of two.

      (This constraint is redundant, due to the use of multiply/add
      transformations of MPAM IDs in place of shift/OR.)

NEW:

 * arm_mpam: Set INTERNAL as needed when setting MSC controls

      (resurrected from my local MPAM fixes branch; previously posted,
      but was not included in RFC v1 of this series)

 * fs/resctrl: Add arch hook to set up the default CLOSID
 * arm_mpam: resctrl: Implement arch hook to set up the default CLOSID

      Add and wire up a hook to update reprogram intPARTID everywhere
      when the resctrl<->MPAM ID mappings change.  This is needed for
      clean initialisation of the hardware, since the MPAM architecture
      doesn't specify the default mappings -- purging everything to a
      consistent state avoids the possibility of spurious aliasing.

 * arm_mpam: Add control partition identifier to mpam_config

      Add a mpam_config field to tell mpam_devices.c what to program
      for intPARTID.  (The idea is to keep the mapping logic in
      mpam_resctrl.c so that mpam_devices.c does not need to understand
      how this value is determined.)
      
 * arm_mpam: Skip inapplicable cases when reprogramming a resctrl domain

      To simplify programming of intPARTIDs, never try to program a
      PARTID with controls for the wrong resctrl_conf_type.  (If
      resctrl passed any control changes for the wrong
      resctrl_conf_type, that's a bug anyway...)

 * arm_mpam: Detect whether the system can alias PARTIDs

      Initial attempt to detect whether it is safe a map multiple
      PARTIDs to the same resctrl control group: if not, the
      user-specified mpam.partid_per_closid parameter is ignored, and
      the driver reverts back to the default behaviour of mapping a
      single PARTID to each CLOSID.
      
      This is currently simplistic, and only considers all MPAM MSCs
      and controls in the system (irrespective if they are going to be
      exposed through resctrl or not).
      

Background
==========

An MPAM Memory System Component (MSC) that does not implement the
PARTID Narrowing feature (Narrowing) provides independently
programmable resource controls for each PARTID up to the maximum PARTID
that the MSC supports.

An MSC that _does_ implement Narrowing performs a remapping step to
convert the PARTID from the incoming memory transaction ("request
PARTID", or "reqPARTID") to an internal ID ("internal PARTID", or
"intPARTID") before looking up and applying resource controls.  Such an
MSC does not necessarily need to provide the same number of
independently programmable resource control partitions (indexed by the
intPARTID) as that which the size of the supported PARTID ("reqPARTID")
space would otherwise require.  Instead, independently programmable
resource controls are provided for each intPARTID, accompanied by a
software programmable mapping that allows each reqPARTID to be mapped
independently to a selected intPARTID.

Typically, the number of intPARTIDs supported by a Narrowing MSC is
smaller than the number of reqPARTIDs.  Narrowing permits multiple
reqPARTIDs to be mapped to each intPARTID, so allowing the whole MPAM
PARTID space to be served.

A non-Narrowing MSC can be understood as implementing special case of
Narrowing, where the reqPARTID-to-intPARTID mapping is the identity
function and is not programmable.


The number of independently configurable control partitions at a given
MSC cannot exceed the number of intPARTIDs available, so the number of
resctrl CLOSIDs (which need to be independently controllable) must
typically be limited not to exceed the maximum number of intPARTIDs
available on any Narrowing MSC.

However, the presence of a programmable reqPARTID-to-intPARTID mapping
in each MSC creates an opportunity: because a group of multiple MPAM
PARTIDs can be mapped onto the same controls, they behave as a single
resource control partition.  Because the PARTID is used as a match
criterion for monitoring purposes along with the MPAM Performance
Monitoring Group (PMG) identifier assigned each memory system
transaction, the specific PARTID in the group can be used to
distinguish transactions for monitoring purposes, even within the same
resctrl control group and when the PMGs of the transactions are the
same.

This means that it becomes possible to provide a larger number of
resctrl monitoring groups than the number of PMGs supported by the MPAM
hardware implementation.


In order to take advantage of this, a more flexible mapping scheme is
required for mapping resctrl CLOSIDs and RMIDs to MPAM PARTIDs and PMGs
than the current MPAM driver provides.

Similar approaches have been suggested by other people
(see Acknowledgements).

However, since I was already working on this, I'm posting my approach
for comparison.


This Series
===========

This series introduces a remapping scheme as follows:

1) The resctrl partition identifiers are combined into a single
identifier (formally the Cartesian product of the CLOSID, Code/Data
partition where applicable, and RMID ID spaces).

2) The resulting identifier is then broken down into the MPAM PARTID
and PMG, in such a way that each possible <CLOSID, CDP index, RMID>
maps to a unique <PARTID, PMG> pair and vice versa, and so that any two
<PARTID, PMG> that differ only in the PMG value, map onto the same
<CLOSID, CDP index> pair.


This transformation could be done in intuitive way using bitwise
concatenation, shift and masking operations, but since the size of the
MPAM hardware ID spaces may limited, I have tried to generalise this
approach using multiplication in place of bit-shifts, to permit ID
spaces whose size is not a power of two.


Limitations
===========

This series aims to contain all translations within the mpam_resctrl.c
code.  This seems the most sensible place for it, since this is a
conversion scheme that will be used to interface the MPAM hardware with
the resctrl core code (but has nothing to do with the "pure" MPAM
architecture or the resctrl core code itself).

To simplify the interface between mpam_resctrl.c and mpam_devices.c,
the ID conversion scheme is arranged so that each resctrl CLOSID maps
to a contiguous sequence of MPAM PARTIDs.  This means that programming
controls for a CLOSID can be achieved by iterating over the appropriate
range of PARTIDs.

This series provides a basic configuration interface that may be too
inflexible for real hardware.


Acknowledgements
================

Thanks to Shaopeng Tan for pointing out this opportunity and suggesting
a related idea, and to Zeng Heng who has also implemented something
similar, albeit with a slightly different mapping scheme [2], [3], [4];
this has continued to develop. [1]


References
==========

Zeng Heng <zengheng4@...wei.com>:

[1] (v4) arm_mpam: Introduce the Narrow-PARTID feature for MPAM driver
https://lore.kernel.org/linux-arm-kernel/20250104101224.873926-1-zengheng4@huawei.com/

[2] (v3) arm_mpam: Introduce the Narrow-PARTID feature for MPAM driver
https://lore.kernel.org/linux-arm-kernel/20241207092136.2488426-1-zengheng4@huawei.com/

[3] (v2) arm_mpam: Introduce the Narrow-PARTID feature for MPAM driver
https://lore.kernel.org/linux-arm-kernel/20241119135104.595630-1-zengheng4@huawei.com/

[4] (v1) arm_mpam: Introduce the definitions of intPARTID and reqPARTID
https://lore.kernel.org/linux-arm-kernel/20241114135037.918470-1-zengheng4@huawei.com/

Dave Martin <Dave.Martin@....com>:

[5] (previous post) [RFC PATCH 0/6] Introduce flexible CLOSID/RMID translation
https://lore.kernel.org/lkml/20241212154000.330467-1-Dave.Martin@arm.com/


Dave Martin (11):
  arm_mpam: Fix read-back of cloned resource controls under CDP
    emulation
  arm_mpam: Delete unused function resctrl_arch_set_rmid()
  arm_mpam: Set INTERNAL as needed when setting MSC controls
  fs/resctrl: Add arch hook to set up the default CLOSID
  arm_mpam: resctrl: Implement arch hook to set up the default CLOSID
  arm_mpam: Introduce flexible CLOSID/RMID translation
  arm_mpam: [NFU] Rework ID remapping to use a kernel command-line
    argument
  arm_mpam: [NFU] Development diagnostics for MPAM ID assignments
  arm_mpam: Add control partition identifier to mpam_config
  arm_mpam: Skip inapplicable cases when reprogramming a resctrl domain
  arm_mpam: Detect whether the system can alias PARTIDs

 arch/arm64/include/asm/mpam.h               |  21 +-
 arch/x86/include/asm/resctrl.h              |  12 +
 drivers/platform/arm64/mpam/mpam_devices.c  |  88 +++++--
 drivers/platform/arm64/mpam/mpam_internal.h |   9 +
 drivers/platform/arm64/mpam/mpam_resctrl.c  | 260 +++++++++++++-------
 fs/resctrl/rdtgroup.c                       |  26 +-
 include/linux/arm_mpam.h                    |   5 +
 include/linux/resctrl.h                     |   2 +
 8 files changed, 299 insertions(+), 124 deletions(-)


base-commit: 4d17b37b670795954fd7a70c8ec83fb705a5e2ad
-- 
2.34.1

Powered by blists - more mailing lists