Message-ID: <87lee8l9pb.fsf@rasp.mail-host-address-is-not-set>
Date:   Fri, 18 Aug 2023 07:13:52 -0700
From:   Carl Worth <carl@...amperecomputing.com>
To:     linux-kernel@...r.kernel.org
Cc:     James Morse <james.morse@....com>,
        "D. Scott Phillips" <scott@...amperecomputing.com>,
        Darren Hart <darren@...amperecomputing.com>,
        Amit Singh Tomar <amitsinght@...vell.com>
Subject: Initial testing of MPAM patches

Hi James,

I'm just getting the chance to start testing your MPAM patches on an
Ampere implementation. Specifically, I was working with your
mpam/snapshot/v6.3 branch [1], an earlier version of which you posted to
the list [2].

I have a few initial comments/questions. These are mostly from a
black-box point of view, (without yet having looked at the code
much). Later on, when I do dig into the code, I can follow up on some
of these issues within the context of the relevant patches.

1. Is there a way to query the MPAM PARTID for a particular resctrl group?

   I see a file named "id" in the directory for each resource group,
   but I get "Operation not supported" if I try to cat it. Meanwhile,
   in the code it looks like PARTIDs are being XORed with some random
   number. Is there a deliberate attempt to obscure the PARTID?

   I don't know how much an end user will care about PARTID values,
   (so it's nice that the driver manages these implicitly), but for
   me, while debugging this stuff, it would be nice to be able to
   query them.

2. Top-level resctrl resource group is not being made to take effect

   When I poke at the schemata of the top-level resource group, I can
   see that it is associated with PARTID 1. That is, if I do something like:

	echo L3:1=face > /sys/fs/resctrl/schemata

   I can see the value 0xface showing up in the cache portion bitmap
   registers associated with PARTID 1. So far, so good.

   But meanwhile, I am not seeing the MPAM0_EL1 system register being
   modified to associate the various cores in this resource group with
   PARTID 1.

   In contrast, for any lower-level resource group I create, I do see
   MPAM0_EL1 being set with PARTID values for the cores that I put
   into the 'cpus' node.

   So it appears that PARTID 1 and its top-level resource group will
   have no effect currently. Am I seeing that correctly?

3. Is there special treatment of cpu 0?

   When I put cpu 0 into a resource group I see both MPAM2_EL2 as well
   as MPAM0_EL1 for cpu 0 being set to the group's corresponding
   PARTID. But when I set any cpu other than 0, only MPAM0_EL1 is
   being set for that cpu. Is this the desired behavior?

   I know that PARTID 0 is treated as reserved by the code, but is cpu
   0 given any special treatment?

4. The current schemata allows for cache portion, but not cache capacity

   The schemata file allows me to specify a bitmask to control
   cache-portion partitioning. But I don't see any mechanism (nor
   backing code) to control cache capacity partitioning.

   Is this due to a limitation in mapping MPAM to the current resctrl
   interface? Or would it be easy to extend the schemata to include a
   cache capacity field as well?

   I see that Amit has proposed patches recentl for extending the
   schemata to add priority partitioning control. So I'd like to do
   something similar for capacity partitioning control as well.

5. Linked-list corruption with missing cache entries in PPTT

   At one point, I tried booting with the MPAM ACPI table populated
   for my L3 cache, but without the necessary entries in the PPTT ACPI
   table. The driver fell over with linked-list corruption, halting
   Linux boot. I'll follow up this report with more details.

6. No support for memory-side caches

   MPAM allows for controlling memory-side caches, and the ACPI
   specification allows them to be described. But the current code
   appears to ignore any such caches, and won't expose them via
   resctrl. I'm very interested to know what work would need to be
   done to allow a memory-side cache to be supported.

7. Cache portion configuration for incorrect PARTID is applied

   I've seen a case where, when trying to use a PARTID other than 1,
   (that is, a resource group other than the top-level), the
   configuration I've set in that resource group does not take
   effect. Instead, the configuration for PARTID 1 takes effect.

   Querying the controlling MPAM registers does not obviously explain
   the buggy behavior. Things look correct. I'll be examining this
   case more closely to see what's happening.

So, that's what I've encountered on an initial look. I didn't call out
all the things that are obviously working well, but there's a lot
there. So that hasn't gone unnoticed.

Thanks again for this series, and I'm looking forward to engaging on
it more going forward.

-Carl

[1] https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/log/?h=mpam/snapshot/v6.3
[2] https://lore.kernel.org/lkml/20210728170637.25610-1-james.morse@arm.com/

Powered by blists - more mailing lists