linux-kernel - RE: [PATCH v3 00/21] x86/resctrl: Make resctrl_arch_rmid

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <YjCCQpLkkMaJys2d@e120937-lin>
Date:   Tue, 15 Mar 2022 12:10:42 +0000
From:   Cristian Marussi <cristian.marussi@....com>
To:     'James Morse' <james.morse@....com>, x86@...nel.org,
        linux-kernel@...r.kernel.org
Cc:     Fenghua Yu <fenghua.yu@...el.com>,
        Reinette Chatre <reinette.chatre@...el.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>,
        H Peter Anvin <hpa@...or.com>,
        Babu Moger <Babu.Moger@....com>,
        shameerali.kolothum.thodi@...wei.com,
        Jamie Iles <jamie@...iainc.com>,
        D Scott Phillips OS <scott@...amperecomputing.com>,
        lcherian@...vell.com, bobo.shaobowang@...wei.com,
        cristian.marussi@....com
Subject: RE: [PATCH v3 00/21] x86/resctrl: Make resctrl_arch_rmid_read()
 return values in bytes

Hi James,

I tested this on an Intel(R) Xeon(R) Gold 5120T trying to compare gathered
resctrl monitor data with and without your series and see if results
were consistent.

I started from this paper [0] from Intel itself for my basic setup with
some minor variations: basically, using the attached test_monitors.sh
my test setup is as follows:

 - a cpuset shield is created upfront isolating all the cpus belonging
   to node1 (14-27,42-55)
 - 2 resctrl CoS are created:
    + 1 process (tar on a big file) act as a LC LatencyCritical actor
      and is run on one of the shielded CPUs with taskset (48)
    + other 3 processes instead runs stress-ng, supposedly acting as
      BE BestEffort noisy neighbours and are pinned to other 3 distinct
      cpus (49,50,51)

The script then triggers 4 different runs of the above crowd with different
cache allocation masks setup in lc/be CoS schemata for node1: ranging
basically from no dedicated allocation (7ff 7ff) to a cache allocation
highly unbalanced in favour of the LC task (7fe 001).

While doing that I collect in background (and out of node1 processors) all
the mon_data from the lc_cos group every 100ms and dump those in a file one
for each cache allocation mask. (mondata_LC_7f0_00f.txt etc)

I tested first against a v5.17-rc1 mainline without your series (named as
5.17.0-rc1-mainline in the results) and then again with your series on top
(named as 5.17.0-rc1-00021-g21c69a5706a5). Got your series from [1].

Then I used gnuplot to see what was the 'profile' of this data with and
without your series by plotting the LC process llc_occupancy data against
time for each one of the runs with the differerent cache allocated.
(each colored graphs represent a different run with a different
cache allocation as reported)

Note that during each run:

- at first the LC process is run without any noisy BEs
- then BEs neighbours are spawned and let to settle for 5s
- finally LC is run again while BEs are making a mess in bg

As a consequence in the plotted graphs, you can see a clear break between
the first part of the run and the last one with BEs.

Looking at the graphs it seems to me that the resctrl counters with and
without you series report a highly similar data profile, as expected
(and hoped :D).

I attach as references:

- a tarball of the raw data (test_mon_data.gz)
- the test_monitors.sh script (not nice but working)
- draw_resctrl.gp gnuplot script
- two PNG of the LC llc_occupancy graphs (all cachemasks runs)
  - with your series: LC_llc_occupancy_5.17.0-rc1-00021-g21c69a5706a5.png
  - without your series: LC_llc_occupancy_5.17.0-rc1-mainline.png

Gnuplot is run as:

 gnuplot -e "filedir='results/5.17.0-rc1-00021-g21c69a5706a5" draw_resctrl.gp

Hope this helps...

Thanks,
Cristian

[0] https://www.intel.com/content/www/us/en/developer/articles/technical/use-intel-resource-director-technology-to-allocate-last-level-cache-llc.html
[1] git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/resctrl_monitors_in_bytes/v3

View attachment "draw_resctrl.gp" of type "text/plain" (445 bytes)

Download attachment "test_mon_data.gz" of type "application/gzip" (38676 bytes)

Download attachment "test_monitors.sh" of type "application/x-sh" (3056 bytes)

Download attachment "LC_llc_occupancy_5.17.0-rc1-mainline.png" of type "image/png" (131515 bytes)

Download attachment "LC_llc_occupancy_5.17.0-rc1-00021-g21c69a5706a5.png" of type "image/png" (134354 bytes)