Message-ID: <DS7PR11MB6077FE180B11A9138D8E7ED7FC1DA@DS7PR11MB6077.namprd11.prod.outlook.com>
Date:   Thu, 24 Aug 2023 18:10:09 +0000
From:   "Luck, Tony" <tony.luck@...el.com>
To:     Amit Singh Tomar <amitsinght@...vell.com>,
        "Chatre, Reinette" <reinette.chatre@...el.com>,
        "Yu, Fenghua" <fenghua.yu@...el.com>,
        "james.morse@....com" <james.morse@....com>,
        George Cherian <gcherian@...vell.com>,
        "robh@...nel.org" <robh@...nel.org>,
        "peternewman@...gle.com" <peternewman@...gle.com>,
        Drew Fustini <dfustini@...libre.com>
CC:     "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-arm-kernel@...ts.infradead.org" 
        <linux-arm-kernel@...ts.infradead.org>
Subject: resctrl2 - status

I've posted before about my belief that it is time to re-invent the
Linux resctrl implementation to make it easier for CPU architectures
with different underlying resource control and monitoring capabilities to
implement those features without being unduly constrained by the quirks
of other architectural designs.

I'm super interested to hear from ARM folks on whether this can
be a suitable base for implementing MPAM.

I now have a reasonably complete example implementation that divides
the resctrl code into a an architecture neutral core layer that is
responsible for creating groups (directories) and associating tasks
with those groups. This layer also manages CPU hotplug events to
update domain lists for each control/monitor feature.

The architecture specific features are each implemented as loadable
modules that register with the core code to receive callbacks for
operations at the core level. E.g. modifying the schemata file for
a group results in callbacks to modules to parse changes to lines
that the module owns, and if all changes to all lines in the schemata
file are accepted by modules, there is a callback to instruct the
module to apply the changes (e.g. for X86 by updating MSRs).

Current snapshot of the code is here:

git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_v65rc7

The commits aren't yet broken into bite-sized pieces. Everything is
in four patches (first two of them are huge).

de8bb88b3285 resctrl2: Architecture independent code
        Although this is big. The code is broken into fifteen file
        that each handle a specific part of the code. The largest
        (directory.c that handles mkdir/rmdir/rename) is only 481
        lines. Most are under 200 lines each.

98831f3edd3b resctrl2: Arch x86 modules for most of the legacy control/monitor functions
        This is also divided into code for each loadable module
        to implement just one X86 RDT feature each. At some point it
        could be broken into separate patches for each module.

d0f1b0a0a555 resctrl2: AMD Slow Memory Bandwidth Allocation
        I did put all the code for this feature into one patch. Should
        give some idea on how easy it is to use the core features
        to implement a specific control feature.

2a1dcfebba2d resctrl2: AMD Bandwidth Monitoring Event Configuration
        Ditto for this which just has to provide writable "info"
        files to allow a user to provide configuration options for
        an architectural feature. Needs a new hook to reset the
        MBM counter files when the configuration is changed to
        completely replicate the legacy resctrl functionality.


I don't have AMD hardware, so the AMD pieces are untested.


To try it out on X86 you need to pick CONFIG_RESCTRL2_FS=y
and then pick which modules to build. Full list is:

CONFIG_AMD_MBEC=m
CONFIG_AMD_SMBA=m
CONFIG_X86_RDT_IORDT=m
CONFIG_X86_RDT_L3_CAT=m
CONFIG_X86_RDT_L2_CAT=m
CONFIG_X86_RDT_L3_CDP=m
CONFIG_X86_RDT_L2_CDP=m
CONFIG_X86_RDT_SHOW_IDS=m
CONFIG_X86_RDT_L3_MBA=m
CONFIG_X86_RDT_L3_MBA_MBPS=m
CONFIG_X86_RDT_LLC_OCCUPANCY=m
CONFIG_X86_RDT_MBM_TOTAL_BYTES=m
CONFIG_X86_RDT_MBM_LOCAL_BYTES=m
CONFIG_X86_RDT_MBM_TOTAL_RATE=m
CONFIG_X86_RDT_MBM_LOCAL_RATE=m
CONFIG_X86_RDT_L2_PSEUDOLOCK=m
CONFIG_X86_RDT_L3_PSEUDOLOCK=m

After booting load the modules you want for the features you want
before (or after) mounting /sys/fs/resctrl.

There are no mount options. Just pick the right modules. E.g.

# modprobe rdt_l3_cat

for basic L3 cache control

# modprobe rdt_l3_cdp

for L3 cache control with code/data prioritization

# modprobe rdt_l3_pseudolock

for L3 cache control with pseudo cache locking support

-Tony

Powered by blists - more mailing lists