lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20121207215357.GA30130@gmail.com>
Date:	Fri, 7 Dec 2012 22:53:57 +0100
From:	Ingo Molnar <mingo@...nel.org>
To:	linux-kernel@...r.kernel.org, linux-mm@...ck.org
Cc:	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Paul Turner <pjt@...gle.com>,
	Lee Schermerhorn <Lee.Schermerhorn@...com>,
	Christoph Lameter <cl@...ux.com>,
	Rik van Riel <riel@...hat.com>, Mel Gorman <mgorman@...e.de>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Andrea Arcangeli <aarcange@...hat.com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Johannes Weiner <hannes@...xchg.org>,
	Hugh Dickins <hughd@...gle.com>,
	Arnaldo Carvalho de Melo <acme@...hat.com>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Mike Galbraith <efault@....de>
Subject: NUMA performance comparison between three NUMA kernels and mainline.
 [Mid-size NUMA system edition.]


Here's a (strongly NUMA-centric) performance comparison of the 
three NUMA kernels: the 'balancenuma-v10' tree from Mel, the 
AutoNUMA-v28 kernel from Andrea and the unified NUMA -v3 tree 
Peter and me are working on.

The goal of these measurements is to specifically quantify the 
NUMA optimization qualities of each of the three NUMA-optimizing 
kernels.

There are lots of numbers in this mail and lot of material to 
read - sorry about that! :-/

I used the latest available kernel versions everywhere: 
furthermore the AutoNUMA-v28 tree has been patched with Hugh 
Dickin's THP-migration support patch, to make it a fair 
apples-to-apples comparison.

I have used the 'perf bench numa' tool to do the measurements, 
which tool can be found at:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/bench

   # to build it install numactl-dev[el] and do "cd tools/perf; make -j install'

To get the raw numbers I ran "perf bench numa mem -a" multiple 
times on each kernel, on a 32-way, 64 GB RAM, 4-node Opteron 
test-system. Each kernel used the same base .config, copied from 
a Fedora RPM kernel, with the NUMA-balancing options enabled.

( Note that the testcases are tailored to my test-system: on
  a smaller system you'd want to run slightly smaller testcases,
  on a larger system you'd want to run a couple of larger 
  testcases as well. )

NUMA convergence latency measurements
-------------------------------------

'NUMA convergence' latency is the number of seconds a workload 
takes to reach 'perfectly NUMA balanced' state. This is measured 
on the CPU placement side: once it has converged then memory 
typically follows within a couple of seconds.

Because convergence is not guaranteed, a 100 seconds latency 
time-out is used in the benchmark. If you see a 100 seconds 
result in the table it means that that particular NUMA kernel 
did not manage to converge that workload unit test within 100 
seconds.

The NxM denotion means process/thread relationship: a 1x4 test 
is 1 process with 4 thread that share a workload - a 4x6 test 
are 4 processes with 6 threads in each process, the processes 
isolated from each other but the threads working on the same 
working set.

I used a wide set of test-cases I collected in the past:

                           [ Lower numbers are better. ]

 [test unit]            :   v3.7 |balancenuma-v10|  AutoNUMA-v28 |   numa-u-v3   |
------------------------------------------------------------------------------------------
 1x3-convergence        :  100.1 |         100.0 |           0.2 |           2.3 |  secs
 1x4-convergence        :  100.2 |         100.1 |         100.2 |           2.1 |  secs
 1x6-convergence        :  100.3 |         100.4 |         100.8 |           7.3 |  secs
 2x3-convergence        :  100.6 |         100.6 |         100.5 |           4.1 |  secs
 3x3-convergence        :  100.6 |         100.5 |         100.5 |           7.6 |  secs
 4x4-convergence        :  100.6 |         100.5 |           4.1 |           7.4 |  secs
 4x4-convergence-NOTHP  :  101.1 |         100.5 |          12.2 |           9.2 |  secs
 4x6-convergence        :    5.4 |         101.2 |          16.6 |          11.7 |  secs
 4x8-convergence        :  101.1 |         101.3 |           3.4 |           3.9 |  secs
 8x4-convergence        :  100.9 |         100.8 |          18.3 |           8.9 |  secs
 8x4-convergence-NOTHP  :  101.9 |         101.0 |          15.7 |          12.1 |  secs
 3x1-convergence        :    0.7 |           1.0 |           0.8 |           0.9 |  secs
 4x1-convergence        :    0.6 |           0.8 |           0.8 |           0.7 |  secs
 8x1-convergence        :    2.8 |           2.9 |           2.9 |           1.2 |  secs
 16x1-convergence       :    3.5 |           3.7 |           2.5 |           2.0 |  secs
 32x1-convergence       :    3.6 |           2.8 |           3.0 |           1.9 |  secs

As expected, mainline only manages to converge workloads where 
each worker process is isolated and the default 
spread-to-all-nodes scheduling policy creates an ideal layout, 
regardless of task ordering.

[ Note that the mainline kernel got a 'lucky strike' convergence 
  in the 4x6 workload: it's always possible for the workload
  to accidentally converge. On a repeat test this did not occur, 
  but I did not erase the outlier because luck is a valid and 
  existing phenomenon. ]

The 'balancenuma' kernel does not converge any of the workloads 
where worker threads or processes relate to each other.

AutoNUMA does pretty well, but it did not manage to converge for 
4 testcases of shared, under-loaded workloads.

The unified NUMA-v3 tree converged well in every testcase.


NUMA workload bandwidth measurements
------------------------------------

The other set of numbers I've collected are workload bandwidth 
measurements, run over 20 seconds. Using 20 seconds gives a 
healthy mix of pre-convergence and post-convergence bandwidth, 
giving the (non-trivial) expense of convergence and memory 
migraton a weight in the result as well. So these are not 
'ideal' results with long runtimes where migration cost gets 
averaged out.

[ The denotion of the workloads is similar to the latency 
  measurements: for example "2x3" means 2 processes, 3 threads 
  per process. See the 'perf bench' tool for details. ]

The 'numa02' and 'numa01-THREAD' tests are AutoNUMA-benchmark 
work-alike workloads, with a shorter runtime for numa01.

The results are:

                           [ Higher numbers are better. ]

 [test unit]            :   v3.7 |balancenuma-v10|  AutoNUMA-v28 | numa-u-v3     |
------------------------------------------------------------------------------------------
 2x1-bw-process         :   6.248|  6.136:  -1.8%|  8.073:  29.2%|  9.647:  54.4%|  GB/sec
 3x1-bw-process         :   7.292|  7.250:  -0.6%| 12.583:  72.6%| 14.528:  99.2%|  GB/sec
 4x1-bw-process         :   6.007|  6.867:  14.3%| 12.313: 105.0%| 18.903: 214.7%|  GB/sec
 8x1-bw-process         :   6.100|  7.974:  30.7%| 20.237: 231.8%| 26.829: 339.8%|  GB/sec
 8x1-bw-process-NOTHP   :   5.944|  5.937:  -0.1%| 17.831: 200.0%| 22.237: 274.1%|  GB/sec
 16x1-bw-process        :   5.607|  5.592:  -0.3%|  5.959:   6.3%| 29.294: 422.5%|  GB/sec
 4x1-bw-thread          :   6.035| 13.598: 125.3%| 17.443: 189.0%| 19.290: 219.6%|  GB/sec
 8x1-bw-thread          :   5.941| 16.356: 175.3%| 22.433: 277.6%| 26.391: 344.2%|  GB/sec
 16x1-bw-thread         :   5.648| 24.608: 335.7%| 20.204: 257.7%| 29.557: 423.3%|  GB/sec
 32x1-bw-thread         :   5.929| 25.477: 329.7%| 18.230: 207.5%| 30.232: 409.9%|  GB/sec
 2x3-bw-thread          :   5.756|  8.785:  52.6%| 14.652: 154.6%| 15.327: 166.3%|  GB/sec
 4x4-bw-thread          :   5.605|  6.366:  13.6%|  9.835:  75.5%| 27.957: 398.8%|  GB/sec
 4x6-bw-thread          :   5.771|  6.287:   8.9%| 15.372: 166.4%| 27.877: 383.1%|  GB/sec
 4x8-bw-thread          :   5.858|  5.860:   0.0%| 11.865: 102.5%| 28.439: 385.5%|  GB/sec
 4x8-bw-thread-NOTHP    :   5.645|  6.167:   9.2%|  9.224:  63.4%| 25.067: 344.1%|  GB/sec
 3x3-bw-thread          :   5.937|  8.235:  38.7%|  6.635:  11.8%| 21.560: 263.1%|  GB/sec
 5x5-bw-thread          :   5.771|  5.762:  -0.2%|  9.575:  65.9%| 26.081: 351.9%|  GB/sec
 2x16-bw-thread         :   5.953|  5.920:  -0.6%|  5.945:  -0.1%| 23.269: 290.9%|  GB/sec
 1x32-bw-thread         :   5.879|  5.828:  -0.9%|  5.848:  -0.5%| 18.985: 222.9%|  GB/sec
 numa02-bw              :   6.049| 29.054: 380.3%| 24.744: 309.1%| 31.431: 419.6%|  GB/sec
 numa02-bw-NOTHP        :   5.850| 27.064: 362.6%| 20.415: 249.0%| 29.104: 397.5%|  GB/sec
 numa01-bw-thread       :   5.834| 20.338: 248.6%| 15.169: 160.0%| 28.607: 390.3%|  GB/sec
 numa01-bw-thread-NOTHP :   5.581| 18.528: 232.0%| 12.108: 117.0%| 21.119: 278.4%|  GB/sec
------------------------------------------------------------------------------------------

The first column shows mainline kernel bandwidth in GB/sec, the 
following 3 colums show pairs of GB/sec bandwidth and percentage 
results, where percentage shows the speed difference to the 
mainline kernel.

Noise is 1-2% in these tests with these durations, so the good 
news is that none of the NUMA kernels regresses on these 
workloads against the mainline kernel. Perhaps balancenuma's 
"2x1-bw-process" and "3x1-bw-process" results might be worth a 
closer look.

No kernel shows particular vulnerability to the NOTHP tests that 
were mixed into the test stream.

As can be expected from the convergence latency results, the 
'balancenuma' tree does well with workloads where there's no 
relationship between threads - but even there it's outperformed 
by the AutoNUMA kernel, and outperformed by an even larger 
margin by the NUMA-v3 kernel. Workloads like the 4x JVM SPECjbb 
on the other hand pose a challenge to the balancenuma kernel, 
both the AutoNUMA and the NUMA-v3 kernels are several times 
faster in those tests.

The AutoNUMA kernel does well in most workloads - its weakness 
are system-wide shared workloads like 2x16-bw-thread and 
1x32-bw-thread, where it falls back to mainline performance.

The NUMA-v3 kernel outperforms every other NUMA kernel.

Here's a direct comparison between the two fastest kernels, the 
AutoNUMA and the NUMA-v3 kernels:


                        [ Higher numbers are better. ]

 [test unit]            :AutoNUMA| numa-u-v3     |
----------------------------------------------------------
 2x1-bw-process         :   8.073|  9.647:  19.5%|  GB/sec
 3x1-bw-process         :  12.583| 14.528:  15.5%|  GB/sec
 4x1-bw-process         :  12.313| 18.903:  53.5%|  GB/sec
 8x1-bw-process         :  20.237| 26.829:  32.6%|  GB/sec
 8x1-bw-process-NOTHP   :  17.831| 22.237:  24.7%|  GB/sec
 16x1-bw-process        :   5.959| 29.294: 391.6%|  GB/sec
 4x1-bw-thread          :  17.443| 19.290:  10.6%|  GB/sec
 8x1-bw-thread          :  22.433| 26.391:  17.6%|  GB/sec
 16x1-bw-thread         :  20.204| 29.557:  46.3%|  GB/sec
 32x1-bw-thread         :  18.230| 30.232:  65.8%|  GB/sec
 2x3-bw-thread          :  14.652| 15.327:   4.6%|  GB/sec
 4x4-bw-thread          :   9.835| 27.957: 184.3%|  GB/sec
 4x6-bw-thread          :  15.372| 27.877:  81.3%|  GB/sec
 4x8-bw-thread          :  11.865| 28.439: 139.7%|  GB/sec
 4x8-bw-thread-NOTHP    :   9.224| 25.067: 171.8%|  GB/sec
 3x3-bw-thread          :   6.635| 21.560: 224.9%|  GB/sec
 5x5-bw-thread          :   9.575| 26.081: 172.4%|  GB/sec
 2x16-bw-thread         :   5.945| 23.269: 291.4%|  GB/sec
 1x32-bw-thread         :   5.848| 18.985: 224.6%|  GB/sec
 numa02-bw              :  24.744| 31.431:  27.0%|  GB/sec
 numa02-bw-NOTHP        :  20.415| 29.104:  42.6%|  GB/sec
 numa01-bw-thread       :  15.169| 28.607:  88.6%|  GB/sec
 numa01-bw-thread-NOTHP :  12.108| 21.119:  74.4%|  GB/sec


NUMA workload "spread" measurements
-----------------------------------

A third, somewhat obscure category of measurements deals with 
the 'execution spread' between threads. Workloads that have to 
wait for the result of every thread before they can declare a 
result are directly limited by this spread.

The 'spread' is measured by the percentage difference between 
the slowest and fastest thread's execution time in a workload:

                           [ Lower numbers are better. ]

 [test unit]            :   v3.7  |balancenuma-v10|  AutoNUMA-v28 |   numa-u-v3   |
------------------------------------------------------------------------------------------
 RAM-bw-local           :    0.0% |          0.0% |          0.0% |          0.0% |  %
 RAM-bw-local-NOTHP     :    0.2% |          0.2% |          0.2% |          0.2% |  %
 RAM-bw-remote          :    0.0% |          0.0% |          0.0% |          0.0% |  %
 RAM-bw-local-2x        :    0.3% |          0.0% |          0.2% |          0.3% |  %
 RAM-bw-remote-2x       :    0.0% |          0.2% |          0.0% |          0.2% |  %
 RAM-bw-cross           :    0.4% |          0.2% |          0.0% |          0.1% |  %
 2x1-bw-process         :    0.5% |          0.2% |          0.2% |          0.2% |  %
 3x1-bw-process         :    0.6% |          0.2% |          0.2% |          0.1% |  %
 4x1-bw-process         :    0.4% |          0.8% |          0.2% |          0.3% |  %
 8x1-bw-process         :    0.8% |          0.1% |          0.2% |          0.2% |  %
 8x1-bw-process-NOTHP   :    0.9% |          0.7% |          0.4% |          0.5% |  %
 16x1-bw-process        :    1.0% |          0.9% |          0.6% |          0.1% |  %
 4x1-bw-thread          :    0.1% |          0.1% |          0.1% |          0.1% |  %
 8x1-bw-thread          :    0.2% |          0.1% |          0.1% |          0.2% |  %
 16x1-bw-thread         :    0.3% |          0.1% |          0.1% |          0.1% |  %
 32x1-bw-thread         :    0.3% |          0.1% |          0.1% |          0.1% |  %
 2x3-bw-thread          :    0.4% |          0.3% |          0.3% |          0.3% |  %
 4x4-bw-thread          :    2.3% |          1.4% |          0.8% |          0.4% |  %
 4x6-bw-thread          :    2.5% |          2.2% |          1.0% |          0.6% |  %
 4x8-bw-thread          :    3.9% |          3.7% |          1.3% |          0.9% |  %
 4x8-bw-thread-NOTHP    :    6.0% |          2.5% |          1.5% |          1.0% |  %
 3x3-bw-thread          :    0.5% |          0.4% |          0.5% |          0.3% |  %
 5x5-bw-thread          :    1.8% |          2.7% |          1.3% |          0.7% |  %
 2x16-bw-thread         :    3.7% |          4.1% |          3.6% |          1.1% |  %
 1x32-bw-thread         :    2.9% |          7.3% |          3.5% |          4.4% |  %
 numa02-bw              :    0.1% |          0.0% |          0.1% |          0.1% |  %
 numa02-bw-NOTHP        :    0.4% |          0.3% |          0.3% |          0.3% |  %
 numa01-bw-thread       :    1.3% |          0.4% |          0.3% |          0.3% |  %
 numa01-bw-thread-NOTHP :    1.8% |          0.8% |          0.8% |          0.9% |  %

The results are pretty good because the runs were relatively 
short with 20 seconds runtime.

Both mainline and balancenuma has trouble with the spread of 
shared workloads - possibly signalling memory allocation 
assymetries. Longer - 60 seconds or more - runs of the key 
workloads would certainly be informative there.

NOTHP (4K ptes) increases the spread and non-determinism of 
every NUMA kernel.

The AutoNUMA and NUMA-v3 kernels have the lowest spread, 
signalling stable NUMA convergence in most scenarios.

Finally, below is the (long!) dump of all the raw data, in case 
someone wants to double-check my results. The perf/bench tool 
can be used to double check the measurements on other systems.

Thanks,

	Ingo

-------------------->

Here are the exact kernel versions used:

 # kernel 1: {v3.7-rc8-18a2f371f5ed}
 # kernel 2: {balancenuma-v10}
 # kernel 3: {autonuma-v28-c4bba428cc5c}
 # kernel 4: {numa/base-v3}

-------------------->

 #
 # Running test on: Linux vega 3.7.0-rc8+ #3 SMP Fri Dec 7 18:29:16 CET 2012 x86_64 x86_64 x86_64 GNU/Linux
 #
# Running numa/mem benchmark...

 # Running main, "perf bench numa mem -a"

 # Running RAM-bw-local, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 0 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-local,                           20.111, secs,           runtime-max/thread
 RAM-bw-local,                           20.106, secs,           runtime-min/thread
 RAM-bw-local,                           20.106, secs,           runtime-avg/thread
 RAM-bw-local,                            0.013, %,              spread-runtime/thread
 RAM-bw-local,                          169.651, GB,             data/thread
 RAM-bw-local,                          169.651, GB,             data-total
 RAM-bw-local,                            0.119, nsecs,          runtime/byte/thread
 RAM-bw-local,                            8.436, GB/sec,         thread-speed
 RAM-bw-local,                            8.436, GB/sec,         total-speed

 # Running RAM-bw-local-NOTHP, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 0 -s 20 -zZq --thp  1 --no-data_rand_walk --thp -1"
 RAM-bw-local-NOTHP,                     20.125, secs,           runtime-max/thread
 RAM-bw-local-NOTHP,                     20.050, secs,           runtime-min/thread
 RAM-bw-local-NOTHP,                     20.050, secs,           runtime-avg/thread
 RAM-bw-local-NOTHP,                      0.187, %,              spread-runtime/thread
 RAM-bw-local-NOTHP,                    169.651, GB,             data/thread
 RAM-bw-local-NOTHP,                    169.651, GB,             data-total
 RAM-bw-local-NOTHP,                      0.119, nsecs,          runtime/byte/thread
 RAM-bw-local-NOTHP,                      8.430, GB/sec,         thread-speed
 RAM-bw-local-NOTHP,                      8.430, GB/sec,         total-speed

 # Running RAM-bw-remote, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 1 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-remote,                          20.141, secs,           runtime-max/thread
 RAM-bw-remote,                          20.134, secs,           runtime-min/thread
 RAM-bw-remote,                          20.134, secs,           runtime-avg/thread
 RAM-bw-remote,                           0.017, %,              spread-runtime/thread
 RAM-bw-remote,                         135.291, GB,             data/thread
 RAM-bw-remote,                         135.291, GB,             data-total
 RAM-bw-remote,                           0.149, nsecs,          runtime/byte/thread
 RAM-bw-remote,                           6.717, GB/sec,         thread-speed
 RAM-bw-remote,                           6.717, GB/sec,         total-speed

 # Running RAM-bw-local-2x, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,2 -M 0x2 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-local-2x,                        20.128, secs,           runtime-max/thread
 RAM-bw-local-2x,                        20.006, secs,           runtime-min/thread
 RAM-bw-local-2x,                        20.064, secs,           runtime-avg/thread
 RAM-bw-local-2x,                         0.302, %,              spread-runtime/thread
 RAM-bw-local-2x,                       132.607, GB,             data/thread
 RAM-bw-local-2x,                       265.214, GB,             data-total
 RAM-bw-local-2x,                         0.152, nsecs,          runtime/byte/thread
 RAM-bw-local-2x,                         6.588, GB/sec,         thread-speed
 RAM-bw-local-2x,                        13.177, GB/sec,         total-speed

 # Running RAM-bw-remote-2x, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,2 -M 1x2 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-remote-2x,                       20.102, secs,           runtime-max/thread
 RAM-bw-remote-2x,                       20.094, secs,           runtime-min/thread
 RAM-bw-remote-2x,                       20.094, secs,           runtime-avg/thread
 RAM-bw-remote-2x,                        0.021, %,              spread-runtime/thread
 RAM-bw-remote-2x,                       74.088, GB,             data/thread
 RAM-bw-remote-2x,                      148.176, GB,             data-total
 RAM-bw-remote-2x,                        0.271, nsecs,          runtime/byte/thread
 RAM-bw-remote-2x,                        3.686, GB/sec,         thread-speed
 RAM-bw-remote-2x,                        7.371, GB/sec,         total-speed

 # Running RAM-bw-cross, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,8 -M 1,0 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-cross,                           20.159, secs,           runtime-max/thread
 RAM-bw-cross,                           20.011, secs,           runtime-min/thread
 RAM-bw-cross,                           20.081, secs,           runtime-avg/thread
 RAM-bw-cross,                            0.369, %,              spread-runtime/thread
 RAM-bw-cross,                          122.407, GB,             data/thread
 RAM-bw-cross,                          244.813, GB,             data-total
 RAM-bw-cross,                            0.165, nsecs,          runtime/byte/thread
 RAM-bw-cross,                            6.072, GB/sec,         thread-speed
 RAM-bw-cross,                           12.144, GB/sec,         total-speed

 # Running  1x3-convergence, "perf bench numa mem -p 1 -t 3 -P 512 -s 100 -zZ0qcm --thp  1"
  1x3-convergence,                      100.103, secs,           NUMA-convergence-latency
  1x3-convergence,                      100.103, secs,           runtime-max/thread
  1x3-convergence,                      100.082, secs,           runtime-min/thread
  1x3-convergence,                      100.093, secs,           runtime-avg/thread
  1x3-convergence,                        0.010, %,              spread-runtime/thread
  1x3-convergence,                      278.636, GB,             data/thread
  1x3-convergence,                      835.908, GB,             data-total
  1x3-convergence,                        0.359, nsecs,          runtime/byte/thread
  1x3-convergence,                        2.784, GB/sec,         thread-speed
  1x3-convergence,                        8.351, GB/sec,         total-speed

 # Running  1x4-convergence, "perf bench numa mem -p 1 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  1x4-convergence,                      100.211, secs,           NUMA-convergence-latency
  1x4-convergence,                      100.211, secs,           runtime-max/thread
  1x4-convergence,                      100.070, secs,           runtime-min/thread
  1x4-convergence,                      100.140, secs,           runtime-avg/thread
  1x4-convergence,                        0.070, %,              spread-runtime/thread
  1x4-convergence,                      154.887, GB,             data/thread
  1x4-convergence,                      619.549, GB,             data-total
  1x4-convergence,                        0.647, nsecs,          runtime/byte/thread
  1x4-convergence,                        1.546, GB/sec,         thread-speed
  1x4-convergence,                        6.182, GB/sec,         total-speed

 # Running  1x6-convergence, "perf bench numa mem -p 1 -t 6 -P 1020 -s 100 -zZ0qcm --thp  1"
  1x6-convergence,                      100.343, secs,           NUMA-convergence-latency
  1x6-convergence,                      100.343, secs,           runtime-max/thread
  1x6-convergence,                      100.235, secs,           runtime-min/thread
  1x6-convergence,                      100.303, secs,           runtime-avg/thread
  1x6-convergence,                        0.054, %,              spread-runtime/thread
  1x6-convergence,                       95.725, GB,             data/thread
  1x6-convergence,                      574.347, GB,             data-total
  1x6-convergence,                        1.048, nsecs,          runtime/byte/thread
  1x6-convergence,                        0.954, GB/sec,         thread-speed
  1x6-convergence,                        5.724, GB/sec,         total-speed

 # Running  2x3-convergence, "perf bench numa mem -p 3 -t 3 -P 1020 -s 100 -zZ0qcm --thp  1"
  2x3-convergence,                      100.601, secs,           NUMA-convergence-latency
  2x3-convergence,                      100.601, secs,           runtime-max/thread
  2x3-convergence,                      100.054, secs,           runtime-min/thread
  2x3-convergence,                      100.307, secs,           runtime-avg/thread
  2x3-convergence,                        0.272, %,              spread-runtime/thread
  2x3-convergence,                       65.837, GB,             data/thread
  2x3-convergence,                      592.529, GB,             data-total
  2x3-convergence,                        1.528, nsecs,          runtime/byte/thread
  2x3-convergence,                        0.654, GB/sec,         thread-speed
  2x3-convergence,                        5.890, GB/sec,         total-speed

 # Running  3x3-convergence, "perf bench numa mem -p 3 -t 3 -P 1020 -s 100 -zZ0qcm --thp  1"
  3x3-convergence,                      100.572, secs,           NUMA-convergence-latency
  3x3-convergence,                      100.572, secs,           runtime-max/thread
  3x3-convergence,                      100.095, secs,           runtime-min/thread
  3x3-convergence,                      100.330, secs,           runtime-avg/thread
  3x3-convergence,                        0.238, %,              spread-runtime/thread
  3x3-convergence,                       65.837, GB,             data/thread
  3x3-convergence,                      592.529, GB,             data-total
  3x3-convergence,                        1.528, nsecs,          runtime/byte/thread
  3x3-convergence,                        0.655, GB/sec,         thread-speed
  3x3-convergence,                        5.892, GB/sec,         total-speed

 # Running  4x4-convergence, "perf bench numa mem -p 4 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  4x4-convergence,                      100.571, secs,           NUMA-convergence-latency
  4x4-convergence,                      100.571, secs,           runtime-max/thread
  4x4-convergence,                      100.122, secs,           runtime-min/thread
  4x4-convergence,                      100.386, secs,           runtime-avg/thread
  4x4-convergence,                        0.223, %,              spread-runtime/thread
  4x4-convergence,                       35.266, GB,             data/thread
  4x4-convergence,                      564.251, GB,             data-total
  4x4-convergence,                        2.852, nsecs,          runtime/byte/thread
  4x4-convergence,                        0.351, GB/sec,         thread-speed
  4x4-convergence,                        5.610, GB/sec,         total-speed

 # Running  4x4-convergence-NOTHP, "perf bench numa mem -p 4 -t 4 -P 512 -s 100 -zZ0qcm --thp  1 --thp -1"
  4x4-convergence-NOTHP,                101.051, secs,           NUMA-convergence-latency
  4x4-convergence-NOTHP,                101.051, secs,           runtime-max/thread
  4x4-convergence-NOTHP,                100.066, secs,           runtime-min/thread
  4x4-convergence-NOTHP,                100.683, secs,           runtime-avg/thread
  4x4-convergence-NOTHP,                  0.487, %,              spread-runtime/thread
  4x4-convergence-NOTHP,                 35.769, GB,             data/thread
  4x4-convergence-NOTHP,                572.304, GB,             data-total
  4x4-convergence-NOTHP,                  2.825, nsecs,          runtime/byte/thread
  4x4-convergence-NOTHP,                  0.354, GB/sec,         thread-speed
  4x4-convergence-NOTHP,                  5.664, GB/sec,         total-speed

 # Running  4x6-convergence, "perf bench numa mem -p 4 -t 6 -P 1020 -s 100 -zZ0qcm --thp  1"
  4x6-convergence,                        5.444, secs,           NUMA-convergence-latency
  4x6-convergence,                        5.444, secs,           runtime-max/thread
  4x6-convergence,                        2.853, secs,           runtime-min/thread
  4x6-convergence,                        4.531, secs,           runtime-avg/thread
  4x6-convergence,                       23.794, %,              spread-runtime/thread
  4x6-convergence,                        1.292, GB,             data/thread
  4x6-convergence,                       31.017, GB,             data-total
  4x6-convergence,                        4.212, nsecs,          runtime/byte/thread
  4x6-convergence,                        0.237, GB/sec,         thread-speed
  4x6-convergence,                        5.698, GB/sec,         total-speed

 # Running  4x8-convergence, "perf bench numa mem -p 4 -t 8 -P 512 -s 100 -zZ0qcm --thp  1"
  4x8-convergence,                      101.133, secs,           NUMA-convergence-latency
  4x8-convergence,                      101.133, secs,           runtime-max/thread
  4x8-convergence,                      100.455, secs,           runtime-min/thread
  4x8-convergence,                      100.803, secs,           runtime-avg/thread
  4x8-convergence,                        0.335, %,              spread-runtime/thread
  4x8-convergence,                       18.522, GB,             data/thread
  4x8-convergence,                      592.705, GB,             data-total
  4x8-convergence,                        5.460, nsecs,          runtime/byte/thread
  4x8-convergence,                        0.183, GB/sec,         thread-speed
  4x8-convergence,                        5.861, GB/sec,         total-speed

 # Running  8x4-convergence, "perf bench numa mem -p 8 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  8x4-convergence,                      100.878, secs,           NUMA-convergence-latency
  8x4-convergence,                      100.878, secs,           runtime-max/thread
  8x4-convergence,                      100.021, secs,           runtime-min/thread
  8x4-convergence,                      100.567, secs,           runtime-avg/thread
  8x4-convergence,                        0.425, %,              spread-runtime/thread
  8x4-convergence,                       18.388, GB,             data/thread
  8x4-convergence,                      588.411, GB,             data-total
  8x4-convergence,                        5.486, nsecs,          runtime/byte/thread
  8x4-convergence,                        0.182, GB/sec,         thread-speed
  8x4-convergence,                        5.833, GB/sec,         total-speed

 # Running  8x4-convergence-NOTHP, "perf bench numa mem -p 8 -t 4 -P 512 -s 100 -zZ0qcm --thp  1 --thp -1"
  8x4-convergence-NOTHP,                101.868, secs,           NUMA-convergence-latency
  8x4-convergence-NOTHP,                101.868, secs,           runtime-max/thread
  8x4-convergence-NOTHP,                100.499, secs,           runtime-min/thread
  8x4-convergence-NOTHP,                101.118, secs,           runtime-avg/thread
  8x4-convergence-NOTHP,                  0.672, %,              spread-runtime/thread
  8x4-convergence-NOTHP,                 17.851, GB,             data/thread
  8x4-convergence-NOTHP,                571.231, GB,             data-total
  8x4-convergence-NOTHP,                  5.707, nsecs,          runtime/byte/thread
  8x4-convergence-NOTHP,                  0.175, GB/sec,         thread-speed
  8x4-convergence-NOTHP,                  5.608, GB/sec,         total-speed

 # Running  3x1-convergence, "perf bench numa mem -p 3 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  3x1-convergence,                        0.652, secs,           NUMA-convergence-latency
  3x1-convergence,                        0.652, secs,           runtime-max/thread
  3x1-convergence,                        0.471, secs,           runtime-min/thread
  3x1-convergence,                        0.584, secs,           runtime-avg/thread
  3x1-convergence,                       13.878, %,              spread-runtime/thread
  3x1-convergence,                        1.432, GB,             data/thread
  3x1-convergence,                        4.295, GB,             data-total
  3x1-convergence,                        0.456, nsecs,          runtime/byte/thread
  3x1-convergence,                        2.195, GB/sec,         thread-speed
  3x1-convergence,                        6.584, GB/sec,         total-speed

 # Running  4x1-convergence, "perf bench numa mem -p 4 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  4x1-convergence,                        0.643, secs,           NUMA-convergence-latency
  4x1-convergence,                        0.643, secs,           runtime-max/thread
  4x1-convergence,                        0.479, secs,           runtime-min/thread
  4x1-convergence,                        0.562, secs,           runtime-avg/thread
  4x1-convergence,                       12.750, %,              spread-runtime/thread
  4x1-convergence,                        1.074, GB,             data/thread
  4x1-convergence,                        4.295, GB,             data-total
  4x1-convergence,                        0.599, nsecs,          runtime/byte/thread
  4x1-convergence,                        1.669, GB/sec,         thread-speed
  4x1-convergence,                        6.677, GB/sec,         total-speed

 # Running  8x1-convergence, "perf bench numa mem -p 8 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  8x1-convergence,                        2.803, secs,           NUMA-convergence-latency
  8x1-convergence,                        2.803, secs,           runtime-max/thread
  8x1-convergence,                        2.509, secs,           runtime-min/thread
  8x1-convergence,                        2.664, secs,           runtime-avg/thread
  8x1-convergence,                        5.250, %,              spread-runtime/thread
  8x1-convergence,                        2.147, GB,             data/thread
  8x1-convergence,                       17.180, GB,             data-total
  8x1-convergence,                        1.305, nsecs,          runtime/byte/thread
  8x1-convergence,                        0.766, GB/sec,         thread-speed
  8x1-convergence,                        6.129, GB/sec,         total-speed

 # Running 16x1-convergence, "perf bench numa mem -p 16 -t 1 -P 256 -s 100 -zZ0qcm --thp  1"
 16x1-convergence,                        3.482, secs,           NUMA-convergence-latency
 16x1-convergence,                        3.482, secs,           runtime-max/thread
 16x1-convergence,                        3.162, secs,           runtime-min/thread
 16x1-convergence,                        3.328, secs,           runtime-avg/thread
 16x1-convergence,                        4.603, %,              spread-runtime/thread
 16x1-convergence,                        1.242, GB,             data/thread
 16x1-convergence,                       19.864, GB,             data-total
 16x1-convergence,                        2.805, nsecs,          runtime/byte/thread
 16x1-convergence,                        0.357, GB/sec,         thread-speed
 16x1-convergence,                        5.704, GB/sec,         total-speed

 # Running 32x1-convergence, "perf bench numa mem -p 32 -t 1 -P 128 -s 100 -zZ0qcm --thp  1"
 32x1-convergence,                        3.612, secs,           NUMA-convergence-latency
 32x1-convergence,                        3.612, secs,           runtime-max/thread
 32x1-convergence,                        3.170, secs,           runtime-min/thread
 32x1-convergence,                        3.456, secs,           runtime-avg/thread
 32x1-convergence,                        6.118, %,              spread-runtime/thread
 32x1-convergence,                        0.671, GB,             data/thread
 32x1-convergence,                       21.475, GB,             data-total
 32x1-convergence,                        5.382, nsecs,          runtime/byte/thread
 32x1-convergence,                        0.186, GB/sec,         thread-speed
 32x1-convergence,                        5.945, GB/sec,         total-speed

 # Running  2x1-bw-process, "perf bench numa mem -p 2 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  2x1-bw-process,                        20.280, secs,           runtime-max/thread
  2x1-bw-process,                        20.059, secs,           runtime-min/thread
  2x1-bw-process,                        20.166, secs,           runtime-avg/thread
  2x1-bw-process,                         0.546, %,              spread-runtime/thread
  2x1-bw-process,                        63.351, GB,             data/thread
  2x1-bw-process,                       126.702, GB,             data-total
  2x1-bw-process,                         0.320, nsecs,          runtime/byte/thread
  2x1-bw-process,                         3.124, GB/sec,         thread-speed
  2x1-bw-process,                         6.248, GB/sec,         total-speed

 # Running  3x1-bw-process, "perf bench numa mem -p 3 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  3x1-bw-process,                        20.320, secs,           runtime-max/thread
  3x1-bw-process,                        20.078, secs,           runtime-min/thread
  3x1-bw-process,                        20.202, secs,           runtime-avg/thread
  3x1-bw-process,                         0.595, %,              spread-runtime/thread
  3x1-bw-process,                        49.392, GB,             data/thread
  3x1-bw-process,                       148.176, GB,             data-total
  3x1-bw-process,                         0.411, nsecs,          runtime/byte/thread
  3x1-bw-process,                         2.431, GB/sec,         thread-speed
  3x1-bw-process,                         7.292, GB/sec,         total-speed

 # Running  4x1-bw-process, "perf bench numa mem -p 4 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  4x1-bw-process,                        20.379, secs,           runtime-max/thread
  4x1-bw-process,                        20.210, secs,           runtime-min/thread
  4x1-bw-process,                        20.291, secs,           runtime-avg/thread
  4x1-bw-process,                         0.413, %,              spread-runtime/thread
  4x1-bw-process,                        30.602, GB,             data/thread
  4x1-bw-process,                       122.407, GB,             data-total
  4x1-bw-process,                         0.666, nsecs,          runtime/byte/thread
  4x1-bw-process,                         1.502, GB/sec,         thread-speed
  4x1-bw-process,                         6.007, GB/sec,         total-speed

 # Running  8x1-bw-process, "perf bench numa mem -p 8 -t 1 -P  512 -s 20 -zZ0q --thp  1"
  8x1-bw-process,                        20.419, secs,           runtime-max/thread
  8x1-bw-process,                        20.073, secs,           runtime-min/thread
  8x1-bw-process,                        20.328, secs,           runtime-avg/thread
  8x1-bw-process,                         0.848, %,              spread-runtime/thread
  8x1-bw-process,                        15.569, GB,             data/thread
  8x1-bw-process,                       124.554, GB,             data-total
  8x1-bw-process,                         1.311, nsecs,          runtime/byte/thread
  8x1-bw-process,                         0.762, GB/sec,         thread-speed
  8x1-bw-process,                         6.100, GB/sec,         total-speed

 # Running  8x1-bw-process-NOTHP, "perf bench numa mem -p 8 -t 1 -P  512 -s 20 -zZ0q --thp  1 --thp -1"
  8x1-bw-process-NOTHP,                  20.502, secs,           runtime-max/thread
  8x1-bw-process-NOTHP,                  20.113, secs,           runtime-min/thread
  8x1-bw-process-NOTHP,                  20.307, secs,           runtime-avg/thread
  8x1-bw-process-NOTHP,                   0.950, %,              spread-runtime/thread
  8x1-bw-process-NOTHP,                  15.234, GB,             data/thread
  8x1-bw-process-NOTHP,                 121.870, GB,             data-total
  8x1-bw-process-NOTHP,                   1.346, nsecs,          runtime/byte/thread
  8x1-bw-process-NOTHP,                   0.743, GB/sec,         thread-speed
  8x1-bw-process-NOTHP,                   5.944, GB/sec,         total-speed

 # Running 16x1-bw-process, "perf bench numa mem -p 16 -t 1 -P 256 -s 20 -zZ0q --thp  1"
 16x1-bw-process,                        20.539, secs,           runtime-max/thread
 16x1-bw-process,                        20.145, secs,           runtime-min/thread
 16x1-bw-process,                        20.407, secs,           runtime-avg/thread
 16x1-bw-process,                         0.959, %,              spread-runtime/thread
 16x1-bw-process,                         7.197, GB,             data/thread
 16x1-bw-process,                       115.159, GB,             data-total
 16x1-bw-process,                         2.854, nsecs,          runtime/byte/thread
 16x1-bw-process,                         0.350, GB/sec,         thread-speed
 16x1-bw-process,                         5.607, GB/sec,         total-speed

 # Running  4x1-bw-thread, "perf bench numa mem -p 1 -t 4 -T 256 -s 20 -zZ0q --thp  1"
  4x1-bw-thread,                         20.105, secs,           runtime-max/thread
  4x1-bw-thread,                         20.047, secs,           runtime-min/thread
  4x1-bw-thread,                         20.071, secs,           runtime-avg/thread
  4x1-bw-thread,                          0.144, %,              spread-runtime/thread
  4x1-bw-thread,                         30.333, GB,             data/thread
  4x1-bw-thread,                        121.333, GB,             data-total
  4x1-bw-thread,                          0.663, nsecs,          runtime/byte/thread
  4x1-bw-thread,                          1.509, GB/sec,         thread-speed
  4x1-bw-thread,                          6.035, GB/sec,         total-speed

 # Running  8x1-bw-thread, "perf bench numa mem -p 1 -t 8 -T 256 -s 20 -zZ0q --thp  1"
  8x1-bw-thread,                         20.106, secs,           runtime-max/thread
  8x1-bw-thread,                         20.021, secs,           runtime-min/thread
  8x1-bw-thread,                         20.062, secs,           runtime-avg/thread
  8x1-bw-thread,                          0.213, %,              spread-runtime/thread
  8x1-bw-thread,                         14.932, GB,             data/thread
  8x1-bw-thread,                        119.454, GB,             data-total
  8x1-bw-thread,                          1.347, nsecs,          runtime/byte/thread
  8x1-bw-thread,                          0.743, GB/sec,         thread-speed
  8x1-bw-thread,                          5.941, GB/sec,         total-speed

 # Running 16x1-bw-thread, "perf bench numa mem -p 1 -t 16 -T 128 -s 20 -zZ0q --thp  1"
 16x1-bw-thread,                         20.176, secs,           runtime-max/thread
 16x1-bw-thread,                         20.049, secs,           runtime-min/thread
 16x1-bw-thread,                         20.125, secs,           runtime-avg/thread
 16x1-bw-thread,                          0.314, %,              spread-runtime/thread
 16x1-bw-thread,                          7.122, GB,             data/thread
 16x1-bw-thread,                        113.951, GB,             data-total
 16x1-bw-thread,                          2.833, nsecs,          runtime/byte/thread
 16x1-bw-thread,                          0.353, GB/sec,         thread-speed
 16x1-bw-thread,                          5.648, GB/sec,         total-speed

 # Running 32x1-bw-thread, "perf bench numa mem -p 1 -t 32 -T 64 -s 20 -zZ0q --thp  1"
 32x1-bw-thread,                         20.159, secs,           runtime-max/thread
 32x1-bw-thread,                         20.034, secs,           runtime-min/thread
 32x1-bw-thread,                         20.120, secs,           runtime-avg/thread
 32x1-bw-thread,                          0.309, %,              spread-runtime/thread
 32x1-bw-thread,                          3.735, GB,             data/thread
 32x1-bw-thread,                        119.521, GB,             data-total
 32x1-bw-thread,                          5.397, nsecs,          runtime/byte/thread
 32x1-bw-thread,                          0.185, GB/sec,         thread-speed
 32x1-bw-thread,                          5.929, GB/sec,         total-speed

 # Running  2x3-bw-thread, "perf bench numa mem -p 2 -t 3 -P 512 -s 20 -zZ0q --thp  1"
  2x3-bw-thread,                         20.239, secs,           runtime-max/thread
  2x3-bw-thread,                         20.092, secs,           runtime-min/thread
  2x3-bw-thread,                         20.183, secs,           runtime-avg/thread
  2x3-bw-thread,                          0.363, %,              spread-runtime/thread
  2x3-bw-thread,                         19.417, GB,             data/thread
  2x3-bw-thread,                        116.501, GB,             data-total
  2x3-bw-thread,                          1.042, nsecs,          runtime/byte/thread
  2x3-bw-thread,                          0.959, GB/sec,         thread-speed
  2x3-bw-thread,                          5.756, GB/sec,         total-speed

 # Running  4x4-bw-thread, "perf bench numa mem -p 4 -t 4 -P 512 -s 20 -zZ0q --thp  1"
  4x4-bw-thread,                         20.978, secs,           runtime-max/thread
  4x4-bw-thread,                         20.005, secs,           runtime-min/thread
  4x4-bw-thread,                         20.576, secs,           runtime-avg/thread
  4x4-bw-thread,                          2.321, %,              spread-runtime/thread
  4x4-bw-thread,                          7.348, GB,             data/thread
  4x4-bw-thread,                        117.575, GB,             data-total
  4x4-bw-thread,                          2.855, nsecs,          runtime/byte/thread
  4x4-bw-thread,                          0.350, GB/sec,         thread-speed
  4x4-bw-thread,                          5.605, GB/sec,         total-speed

 # Running  4x6-bw-thread, "perf bench numa mem -p 4 -t 6 -P 512 -s 20 -zZ0q --thp  1"
  4x6-bw-thread,                         21.118, secs,           runtime-max/thread
  4x6-bw-thread,                         20.082, secs,           runtime-min/thread
  4x6-bw-thread,                         20.819, secs,           runtime-avg/thread
  4x6-bw-thread,                          2.451, %,              spread-runtime/thread
  4x6-bw-thread,                          5.078, GB,             data/thread
  4x6-bw-thread,                        121.870, GB,             data-total
  4x6-bw-thread,                          4.159, nsecs,          runtime/byte/thread
  4x6-bw-thread,                          0.240, GB/sec,         thread-speed
  4x6-bw-thread,                          5.771, GB/sec,         total-speed

 # Running  4x8-bw-thread, "perf bench numa mem -p 4 -t 8 -P 512 -s 20 -zZ0q --thp  1"
  4x8-bw-thread,                         21.994, secs,           runtime-max/thread
  4x8-bw-thread,                         20.290, secs,           runtime-min/thread
  4x8-bw-thread,                         21.387, secs,           runtime-avg/thread
  4x8-bw-thread,                          3.874, %,              spread-runtime/thread
  4x8-bw-thread,                          4.027, GB,             data/thread
  4x8-bw-thread,                        128.849, GB,             data-total
  4x8-bw-thread,                          5.462, nsecs,          runtime/byte/thread
  4x8-bw-thread,                          0.183, GB/sec,         thread-speed
  4x8-bw-thread,                          5.858, GB/sec,         total-speed

 # Running  4x8-bw-thread-NOTHP, "perf bench numa mem -p 4 -t 8 -P 512 -s 20 -zZ0q --thp  1 --thp -1"
  4x8-bw-thread-NOTHP,                   22.728, secs,           runtime-max/thread
  4x8-bw-thread-NOTHP,                   20.013, secs,           runtime-min/thread
  4x8-bw-thread-NOTHP,                   21.968, secs,           runtime-avg/thread
  4x8-bw-thread-NOTHP,                    5.975, %,              spread-runtime/thread
  4x8-bw-thread-NOTHP,                    4.010, GB,             data/thread
  4x8-bw-thread-NOTHP,                  128.312, GB,             data-total
  4x8-bw-thread-NOTHP,                    5.668, nsecs,          runtime/byte/thread
  4x8-bw-thread-NOTHP,                    0.176, GB/sec,         thread-speed
  4x8-bw-thread-NOTHP,                    5.645, GB/sec,         total-speed

 # Running  3x3-bw-thread, "perf bench numa mem -p 3 -t 3 -P 512 -s 20 -zZ0q --thp  1"
  3x3-bw-thread,                         20.526, secs,           runtime-max/thread
  3x3-bw-thread,                         20.317, secs,           runtime-min/thread
  3x3-bw-thread,                         20.467, secs,           runtime-avg/thread
  3x3-bw-thread,                          0.510, %,              spread-runtime/thread
  3x3-bw-thread,                         13.541, GB,             data/thread
  3x3-bw-thread,                        121.870, GB,             data-total
  3x3-bw-thread,                          1.516, nsecs,          runtime/byte/thread
  3x3-bw-thread,                          0.660, GB/sec,         thread-speed
  3x3-bw-thread,                          5.937, GB/sec,         total-speed

 # Running  5x5-bw-thread, "perf bench numa mem -p 5 -t 5 -P 512 -s 20 -zZ0q --thp  1"
  5x5-bw-thread,                         21.023, secs,           runtime-max/thread
  5x5-bw-thread,                         20.252, secs,           runtime-min/thread
  5x5-bw-thread,                         20.701, secs,           runtime-avg/thread
  5x5-bw-thread,                          1.833, %,              spread-runtime/thread
  5x5-bw-thread,                          4.853, GB,             data/thread
  5x5-bw-thread,                        121.333, GB,             data-total
  5x5-bw-thread,                          4.332, nsecs,          runtime/byte/thread
  5x5-bw-thread,                          0.231, GB/sec,         thread-speed
  5x5-bw-thread,                          5.771, GB/sec,         total-speed

 # Running 2x16-bw-thread, "perf bench numa mem -p 2 -t 16 -P 512 -s 20 -zZ0q --thp  1"
 2x16-bw-thread,                         21.646, secs,           runtime-max/thread
 2x16-bw-thread,                         20.065, secs,           runtime-min/thread
 2x16-bw-thread,                         21.026, secs,           runtime-avg/thread
 2x16-bw-thread,                          3.652, %,              spread-runtime/thread
 2x16-bw-thread,                          4.027, GB,             data/thread
 2x16-bw-thread,                        128.849, GB,             data-total
 2x16-bw-thread,                          5.376, nsecs,          runtime/byte/thread
 2x16-bw-thread,                          0.186, GB/sec,         thread-speed
 2x16-bw-thread,                          5.953, GB/sec,         total-speed

 # Running 1x32-bw-thread, "perf bench numa mem -p 1 -t 32 -P 2048 -s 20 -zZ0q --thp  1"
 1x32-bw-thread,                         23.377, secs,           runtime-max/thread
 1x32-bw-thread,                         22.030, secs,           runtime-min/thread
 1x32-bw-thread,                         22.936, secs,           runtime-avg/thread
 1x32-bw-thread,                          2.881, %,              spread-runtime/thread
 1x32-bw-thread,                          4.295, GB,             data/thread
 1x32-bw-thread,                        137.439, GB,             data-total
 1x32-bw-thread,                          5.443, nsecs,          runtime/byte/thread
 1x32-bw-thread,                          0.184, GB/sec,         thread-speed
 1x32-bw-thread,                          5.879, GB/sec,         total-speed

 # Running numa02-bw, "perf bench numa mem -p 1 -t 32 -T 32 -s 20 -zZ0q --thp  1"
 numa02-bw,                              20.065, secs,           runtime-max/thread
 numa02-bw,                              20.012, secs,           runtime-min/thread
 numa02-bw,                              20.050, secs,           runtime-avg/thread
 numa02-bw,                               0.132, %,              spread-runtime/thread
 numa02-bw,                               3.793, GB,             data/thread
 numa02-bw,                             121.366, GB,             data-total
 numa02-bw,                               5.290, nsecs,          runtime/byte/thread
 numa02-bw,                               0.189, GB/sec,         thread-speed
 numa02-bw,                               6.049, GB/sec,         total-speed

 # Running numa02-bw-NOTHP, "perf bench numa mem -p 1 -t 32 -T 32 -s 20 -zZ0q --thp  1 --thp -1"
 numa02-bw-NOTHP,                        20.132, secs,           runtime-max/thread
 numa02-bw-NOTHP,                        19.987, secs,           runtime-min/thread
 numa02-bw-NOTHP,                        20.049, secs,           runtime-avg/thread
 numa02-bw-NOTHP,                         0.360, %,              spread-runtime/thread
 numa02-bw-NOTHP,                         3.681, GB,             data/thread
 numa02-bw-NOTHP,                       117.776, GB,             data-total
 numa02-bw-NOTHP,                         5.470, nsecs,          runtime/byte/thread
 numa02-bw-NOTHP,                         0.183, GB/sec,         thread-speed
 numa02-bw-NOTHP,                         5.850, GB/sec,         total-speed

 # Running numa01-bw-thread, "perf bench numa mem -p 2 -t 16 -T 192 -s 20 -zZ0q --thp  1"
 numa01-bw-thread,                       20.704, secs,           runtime-max/thread
 numa01-bw-thread,                       20.185, secs,           runtime-min/thread
 numa01-bw-thread,                       20.571, secs,           runtime-avg/thread
 numa01-bw-thread,                        1.254, %,              spread-runtime/thread
 numa01-bw-thread,                        3.775, GB,             data/thread
 numa01-bw-thread,                      120.796, GB,             data-total
 numa01-bw-thread,                        5.485, nsecs,          runtime/byte/thread
 numa01-bw-thread,                        0.182, GB/sec,         thread-speed
 numa01-bw-thread,                        5.834, GB/sec,         total-speed

 # Running numa01-bw-thread-NOTHP, "perf bench numa mem -p 2 -t 16 -T 192 -s 20 -zZ0q --thp  1 --thp -1"
 numa01-bw-thread-NOTHP,                 20.780, secs,           runtime-max/thread
 numa01-bw-thread-NOTHP,                 20.023, secs,           runtime-min/thread
 numa01-bw-thread-NOTHP,                 20.418, secs,           runtime-avg/thread
 numa01-bw-thread-NOTHP,                  1.821, %,              spread-runtime/thread
 numa01-bw-thread-NOTHP,                  3.624, GB,             data/thread
 numa01-bw-thread-NOTHP,                115.964, GB,             data-total
 numa01-bw-thread-NOTHP,                  5.734, nsecs,          runtime/byte/thread
 numa01-bw-thread-NOTHP,                  0.174, GB/sec,         thread-speed
 numa01-bw-thread-NOTHP,                  5.581, GB/sec,         total-speed

 #
 # Running test on: Linux vega 3.7.0-rc6+ #2 SMP Fri Dec 7 17:59:13 CET 2012 x86_64 x86_64 x86_64 GNU/Linux
 #
# Running numa/mem benchmark...

 # Running main, "perf bench numa mem -a"

 # Running RAM-bw-local, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 0 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-local,                           20.049, secs,           runtime-max/thread
 RAM-bw-local,                           20.044, secs,           runtime-min/thread
 RAM-bw-local,                           20.044, secs,           runtime-avg/thread
 RAM-bw-local,                            0.014, %,              spread-runtime/thread
 RAM-bw-local,                          172.872, GB,             data/thread
 RAM-bw-local,                          172.872, GB,             data-total
 RAM-bw-local,                            0.116, nsecs,          runtime/byte/thread
 RAM-bw-local,                            8.622, GB/sec,         thread-speed
 RAM-bw-local,                            8.622, GB/sec,         total-speed

 # Running RAM-bw-local-NOTHP, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 0 -s 20 -zZq --thp  1 --no-data_rand_walk --thp -1"
 RAM-bw-local-NOTHP,                     20.135, secs,           runtime-max/thread
 RAM-bw-local-NOTHP,                     20.059, secs,           runtime-min/thread
 RAM-bw-local-NOTHP,                     20.059, secs,           runtime-avg/thread
 RAM-bw-local-NOTHP,                      0.189, %,              spread-runtime/thread
 RAM-bw-local-NOTHP,                    172.872, GB,             data/thread
 RAM-bw-local-NOTHP,                    172.872, GB,             data-total
 RAM-bw-local-NOTHP,                      0.116, nsecs,          runtime/byte/thread
 RAM-bw-local-NOTHP,                      8.586, GB/sec,         thread-speed
 RAM-bw-local-NOTHP,                      8.586, GB/sec,         total-speed

 # Running RAM-bw-remote, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 1 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-remote,                          20.080, secs,           runtime-max/thread
 RAM-bw-remote,                          20.073, secs,           runtime-min/thread
 RAM-bw-remote,                          20.073, secs,           runtime-avg/thread
 RAM-bw-remote,                           0.017, %,              spread-runtime/thread
 RAM-bw-remote,                         135.291, GB,             data/thread
 RAM-bw-remote,                         135.291, GB,             data-total
 RAM-bw-remote,                           0.148, nsecs,          runtime/byte/thread
 RAM-bw-remote,                           6.738, GB/sec,         thread-speed
 RAM-bw-remote,                           6.738, GB/sec,         total-speed

 # Running RAM-bw-local-2x, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,2 -M 0x2 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-local-2x,                        20.127, secs,           runtime-max/thread
 RAM-bw-local-2x,                        20.111, secs,           runtime-min/thread
 RAM-bw-local-2x,                        20.116, secs,           runtime-avg/thread
 RAM-bw-local-2x,                         0.038, %,              spread-runtime/thread
 RAM-bw-local-2x,                       130.997, GB,             data/thread
 RAM-bw-local-2x,                       261.993, GB,             data-total
 RAM-bw-local-2x,                         0.154, nsecs,          runtime/byte/thread
 RAM-bw-local-2x,                         6.509, GB/sec,         thread-speed
 RAM-bw-local-2x,                        13.017, GB/sec,         total-speed

 # Running RAM-bw-remote-2x, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,2 -M 1x2 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-remote-2x,                       20.183, secs,           runtime-max/thread
 RAM-bw-remote-2x,                       20.110, secs,           runtime-min/thread
 RAM-bw-remote-2x,                       20.143, secs,           runtime-avg/thread
 RAM-bw-remote-2x,                        0.180, %,              spread-runtime/thread
 RAM-bw-remote-2x,                       75.162, GB,             data/thread
 RAM-bw-remote-2x,                      150.324, GB,             data-total
 RAM-bw-remote-2x,                        0.269, nsecs,          runtime/byte/thread
 RAM-bw-remote-2x,                        3.724, GB/sec,         thread-speed
 RAM-bw-remote-2x,                        7.448, GB/sec,         total-speed

 # Running RAM-bw-cross, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,8 -M 1,0 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-cross,                           20.159, secs,           runtime-max/thread
 RAM-bw-cross,                           20.071, secs,           runtime-min/thread
 RAM-bw-cross,                           20.111, secs,           runtime-avg/thread
 RAM-bw-cross,                            0.220, %,              spread-runtime/thread
 RAM-bw-cross,                          124.017, GB,             data/thread
 RAM-bw-cross,                          248.034, GB,             data-total
 RAM-bw-cross,                            0.163, nsecs,          runtime/byte/thread
 RAM-bw-cross,                            6.152, GB/sec,         thread-speed
 RAM-bw-cross,                           12.304, GB/sec,         total-speed

 # Running  1x3-convergence, "perf bench numa mem -p 1 -t 3 -P 512 -s 100 -zZ0qcm --thp  1"
  1x3-convergence,                      100.038, secs,           NUMA-convergence-latency
  1x3-convergence,                      100.038, secs,           runtime-max/thread
  1x3-convergence,                      100.005, secs,           runtime-min/thread
  1x3-convergence,                      100.016, secs,           runtime-avg/thread
  1x3-convergence,                        0.016, %,              spread-runtime/thread
  1x3-convergence,                      379.210, GB,             data/thread
  1x3-convergence,                     1137.629, GB,             data-total
  1x3-convergence,                        0.264, nsecs,          runtime/byte/thread
  1x3-convergence,                        3.791, GB/sec,         thread-speed
  1x3-convergence,                       11.372, GB/sec,         total-speed

 # Running  1x4-convergence, "perf bench numa mem -p 1 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  1x4-convergence,                      100.091, secs,           NUMA-convergence-latency
  1x4-convergence,                      100.091, secs,           runtime-max/thread
  1x4-convergence,                      100.016, secs,           runtime-min/thread
  1x4-convergence,                      100.053, secs,           runtime-avg/thread
  1x4-convergence,                        0.037, %,              spread-runtime/thread
  1x4-convergence,                      162.672, GB,             data/thread
  1x4-convergence,                      650.688, GB,             data-total
  1x4-convergence,                        0.615, nsecs,          runtime/byte/thread
  1x4-convergence,                        1.625, GB/sec,         thread-speed
  1x4-convergence,                        6.501, GB/sec,         total-speed

 # Running  1x6-convergence, "perf bench numa mem -p 1 -t 6 -P 1020 -s 100 -zZ0qcm --thp  1"
  1x6-convergence,                      100.366, secs,           NUMA-convergence-latency
  1x6-convergence,                      100.366, secs,           runtime-max/thread
  1x6-convergence,                      100.005, secs,           runtime-min/thread
  1x6-convergence,                      100.144, secs,           runtime-avg/thread
  1x6-convergence,                        0.180, %,              spread-runtime/thread
  1x6-convergence,                      103.924, GB,             data/thread
  1x6-convergence,                      623.546, GB,             data-total
  1x6-convergence,                        0.966, nsecs,          runtime/byte/thread
  1x6-convergence,                        1.035, GB/sec,         thread-speed
  1x6-convergence,                        6.213, GB/sec,         total-speed

 # Running  2x3-convergence, "perf bench numa mem -p 3 -t 3 -P 1020 -s 100 -zZ0qcm --thp  1"
  2x3-convergence,                      100.632, secs,           NUMA-convergence-latency
  2x3-convergence,                      100.632, secs,           runtime-max/thread
  2x3-convergence,                      100.080, secs,           runtime-min/thread
  2x3-convergence,                      100.376, secs,           runtime-avg/thread
  2x3-convergence,                        0.274, %,              spread-runtime/thread
  2x3-convergence,                       87.941, GB,             data/thread
  2x3-convergence,                      791.465, GB,             data-total
  2x3-convergence,                        1.144, nsecs,          runtime/byte/thread
  2x3-convergence,                        0.874, GB/sec,         thread-speed
  2x3-convergence,                        7.865, GB/sec,         total-speed

 # Running  3x3-convergence, "perf bench numa mem -p 3 -t 3 -P 1020 -s 100 -zZ0qcm --thp  1"
  3x3-convergence,                      100.474, secs,           NUMA-convergence-latency
  3x3-convergence,                      100.474, secs,           runtime-max/thread
  3x3-convergence,                      100.070, secs,           runtime-min/thread
  3x3-convergence,                      100.338, secs,           runtime-avg/thread
  3x3-convergence,                        0.201, %,              spread-runtime/thread
  3x3-convergence,                      118.363, GB,             data/thread
  3x3-convergence,                     1065.269, GB,             data-total
  3x3-convergence,                        0.849, nsecs,          runtime/byte/thread
  3x3-convergence,                        1.178, GB/sec,         thread-speed
  3x3-convergence,                       10.602, GB/sec,         total-speed

 # Running  4x4-convergence, "perf bench numa mem -p 4 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  4x4-convergence,                      100.527, secs,           NUMA-convergence-latency
  4x4-convergence,                      100.527, secs,           runtime-max/thread
  4x4-convergence,                      100.179, secs,           runtime-min/thread
  4x4-convergence,                      100.353, secs,           runtime-avg/thread
  4x4-convergence,                        0.173, %,              spread-runtime/thread
  4x4-convergence,                       65.230, GB,             data/thread
  4x4-convergence,                     1043.677, GB,             data-total
  4x4-convergence,                        1.541, nsecs,          runtime/byte/thread
  4x4-convergence,                        0.649, GB/sec,         thread-speed
  4x4-convergence,                       10.382, GB/sec,         total-speed

 # Running  4x4-convergence-NOTHP, "perf bench numa mem -p 4 -t 4 -P 512 -s 100 -zZ0qcm --thp  1 --thp -1"
  4x4-convergence-NOTHP,                100.532, secs,           NUMA-convergence-latency
  4x4-convergence-NOTHP,                100.532, secs,           runtime-max/thread
  4x4-convergence-NOTHP,                100.095, secs,           runtime-min/thread
  4x4-convergence-NOTHP,                100.343, secs,           runtime-avg/thread
  4x4-convergence-NOTHP,                  0.217, %,              spread-runtime/thread
  4x4-convergence-NOTHP,                 57.311, GB,             data/thread
  4x4-convergence-NOTHP,                916.976, GB,             data-total
  4x4-convergence-NOTHP,                  1.754, nsecs,          runtime/byte/thread
  4x4-convergence-NOTHP,                  0.570, GB/sec,         thread-speed
  4x4-convergence-NOTHP,                  9.121, GB/sec,         total-speed

 # Running  4x6-convergence, "perf bench numa mem -p 4 -t 6 -P 1020 -s 100 -zZ0qcm --thp  1"
  4x6-convergence,                      101.230, secs,           NUMA-convergence-latency
  4x6-convergence,                      101.230, secs,           runtime-max/thread
  4x6-convergence,                      100.093, secs,           runtime-min/thread
  4x6-convergence,                      100.825, secs,           runtime-avg/thread
  4x6-convergence,                        0.562, %,              spread-runtime/thread
  4x6-convergence,                       28.076, GB,             data/thread
  4x6-convergence,                      673.815, GB,             data-total
  4x6-convergence,                        3.606, nsecs,          runtime/byte/thread
  4x6-convergence,                        0.277, GB/sec,         thread-speed
  4x6-convergence,                        6.656, GB/sec,         total-speed

 # Running  4x8-convergence, "perf bench numa mem -p 4 -t 8 -P 512 -s 100 -zZ0qcm --thp  1"
  4x8-convergence,                      101.310, secs,           NUMA-convergence-latency
  4x8-convergence,                      101.310, secs,           runtime-max/thread
  4x8-convergence,                      100.052, secs,           runtime-min/thread
  4x8-convergence,                      100.679, secs,           runtime-avg/thread
  4x8-convergence,                        0.621, %,              spread-runtime/thread
  4x8-convergence,                       18.740, GB,             data/thread
  4x8-convergence,                      599.685, GB,             data-total
  4x8-convergence,                        5.406, nsecs,          runtime/byte/thread
  4x8-convergence,                        0.185, GB/sec,         thread-speed
  4x8-convergence,                        5.919, GB/sec,         total-speed

 # Running  8x4-convergence, "perf bench numa mem -p 8 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  8x4-convergence,                      100.849, secs,           NUMA-convergence-latency
  8x4-convergence,                      100.849, secs,           runtime-max/thread
  8x4-convergence,                      100.020, secs,           runtime-min/thread
  8x4-convergence,                      100.570, secs,           runtime-avg/thread
  8x4-convergence,                        0.411, %,              spread-runtime/thread
  8x4-convergence,                       22.364, GB,             data/thread
  8x4-convergence,                      715.649, GB,             data-total
  8x4-convergence,                        4.509, nsecs,          runtime/byte/thread
  8x4-convergence,                        0.222, GB/sec,         thread-speed
  8x4-convergence,                        7.096, GB/sec,         total-speed

 # Running  8x4-convergence-NOTHP, "perf bench numa mem -p 8 -t 4 -P 512 -s 100 -zZ0qcm --thp  1 --thp -1"
  8x4-convergence-NOTHP,                100.976, secs,           NUMA-convergence-latency
  8x4-convergence-NOTHP,                100.976, secs,           runtime-max/thread
  8x4-convergence-NOTHP,                100.066, secs,           runtime-min/thread
  8x4-convergence-NOTHP,                100.580, secs,           runtime-avg/thread
  8x4-convergence-NOTHP,                  0.451, %,              spread-runtime/thread
  8x4-convergence-NOTHP,                 27.146, GB,             data/thread
  8x4-convergence-NOTHP,                868.657, GB,             data-total
  8x4-convergence-NOTHP,                  3.720, nsecs,          runtime/byte/thread
  8x4-convergence-NOTHP,                  0.269, GB/sec,         thread-speed
  8x4-convergence-NOTHP,                  8.603, GB/sec,         total-speed

 # Running  3x1-convergence, "perf bench numa mem -p 3 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  3x1-convergence,                        1.010, secs,           NUMA-convergence-latency
  3x1-convergence,                        1.010, secs,           runtime-max/thread
  3x1-convergence,                        0.869, secs,           runtime-min/thread
  3x1-convergence,                        0.958, secs,           runtime-avg/thread
  3x1-convergence,                        6.944, %,              spread-runtime/thread
  3x1-convergence,                        2.326, GB,             data/thread
  3x1-convergence,                        6.979, GB,             data-total
  3x1-convergence,                        0.434, nsecs,          runtime/byte/thread
  3x1-convergence,                        2.305, GB/sec,         thread-speed
  3x1-convergence,                        6.914, GB/sec,         total-speed

 # Running  4x1-convergence, "perf bench numa mem -p 4 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  4x1-convergence,                        0.782, secs,           NUMA-convergence-latency
  4x1-convergence,                        0.782, secs,           runtime-max/thread
  4x1-convergence,                        0.623, secs,           runtime-min/thread
  4x1-convergence,                        0.689, secs,           runtime-avg/thread
  4x1-convergence,                       10.122, %,              spread-runtime/thread
  4x1-convergence,                        1.208, GB,             data/thread
  4x1-convergence,                        4.832, GB,             data-total
  4x1-convergence,                        0.647, nsecs,          runtime/byte/thread
  4x1-convergence,                        1.545, GB/sec,         thread-speed
  4x1-convergence,                        6.181, GB/sec,         total-speed

 # Running  8x1-convergence, "perf bench numa mem -p 8 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  8x1-convergence,                        2.914, secs,           NUMA-convergence-latency
  8x1-convergence,                        2.914, secs,           runtime-max/thread
  8x1-convergence,                        2.533, secs,           runtime-min/thread
  8x1-convergence,                        2.750, secs,           runtime-avg/thread
  8x1-convergence,                        6.538, %,              spread-runtime/thread
  8x1-convergence,                        2.215, GB,             data/thread
  8x1-convergence,                       17.717, GB,             data-total
  8x1-convergence,                        1.316, nsecs,          runtime/byte/thread
  8x1-convergence,                        0.760, GB/sec,         thread-speed
  8x1-convergence,                        6.080, GB/sec,         total-speed

 # Running 16x1-convergence, "perf bench numa mem -p 16 -t 1 -P 256 -s 100 -zZ0qcm --thp  1"
 16x1-convergence,                        3.688, secs,           NUMA-convergence-latency
 16x1-convergence,                        3.688, secs,           runtime-max/thread
 16x1-convergence,                        3.358, secs,           runtime-min/thread
 16x1-convergence,                        3.533, secs,           runtime-avg/thread
 16x1-convergence,                        4.481, %,              spread-runtime/thread
 16x1-convergence,                        1.292, GB,             data/thread
 16x1-convergence,                       20.670, GB,             data-total
 16x1-convergence,                        2.855, nsecs,          runtime/byte/thread
 16x1-convergence,                        0.350, GB/sec,         thread-speed
 16x1-convergence,                        5.604, GB/sec,         total-speed

 # Running 32x1-convergence, "perf bench numa mem -p 32 -t 1 -P 128 -s 100 -zZ0qcm --thp  1"
 32x1-convergence,                        2.762, secs,           NUMA-convergence-latency
 32x1-convergence,                        2.762, secs,           runtime-max/thread
 32x1-convergence,                        2.552, secs,           runtime-min/thread
 32x1-convergence,                        2.735, secs,           runtime-avg/thread
 32x1-convergence,                        3.807, %,              spread-runtime/thread
 32x1-convergence,                        0.516, GB,             data/thread
 32x1-convergence,                       16.509, GB,             data-total
 32x1-convergence,                        5.354, nsecs,          runtime/byte/thread
 32x1-convergence,                        0.187, GB/sec,         thread-speed
 32x1-convergence,                        5.976, GB/sec,         total-speed

 # Running  2x1-bw-process, "perf bench numa mem -p 2 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  2x1-bw-process,                        20.123, secs,           runtime-max/thread
  2x1-bw-process,                        20.053, secs,           runtime-min/thread
  2x1-bw-process,                        20.085, secs,           runtime-avg/thread
  2x1-bw-process,                         0.173, %,              spread-runtime/thread
  2x1-bw-process,                        61.740, GB,             data/thread
  2x1-bw-process,                       123.480, GB,             data-total
  2x1-bw-process,                         0.326, nsecs,          runtime/byte/thread
  2x1-bw-process,                         3.068, GB/sec,         thread-speed
  2x1-bw-process,                         6.136, GB/sec,         total-speed

 # Running  3x1-bw-process, "perf bench numa mem -p 3 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  3x1-bw-process,                        20.143, secs,           runtime-max/thread
  3x1-bw-process,                        20.043, secs,           runtime-min/thread
  3x1-bw-process,                        20.091, secs,           runtime-avg/thread
  3x1-bw-process,                         0.249, %,              spread-runtime/thread
  3x1-bw-process,                        48.676, GB,             data/thread
  3x1-bw-process,                       146.029, GB,             data-total
  3x1-bw-process,                         0.414, nsecs,          runtime/byte/thread
  3x1-bw-process,                         2.417, GB/sec,         thread-speed
  3x1-bw-process,                         7.250, GB/sec,         total-speed

 # Running  4x1-bw-process, "perf bench numa mem -p 4 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  4x1-bw-process,                        20.327, secs,           runtime-max/thread
  4x1-bw-process,                        20.020, secs,           runtime-min/thread
  4x1-bw-process,                        20.168, secs,           runtime-avg/thread
  4x1-bw-process,                         0.754, %,              spread-runtime/thread
  4x1-bw-process,                        34.897, GB,             data/thread
  4x1-bw-process,                       139.586, GB,             data-total
  4x1-bw-process,                         0.582, nsecs,          runtime/byte/thread
  4x1-bw-process,                         1.717, GB/sec,         thread-speed
  4x1-bw-process,                         6.867, GB/sec,         total-speed

 # Running  8x1-bw-process, "perf bench numa mem -p 8 -t 1 -P  512 -s 20 -zZ0q --thp  1"
  8x1-bw-process,                        20.063, secs,           runtime-max/thread
  8x1-bw-process,                        20.004, secs,           runtime-min/thread
  8x1-bw-process,                        20.034, secs,           runtime-avg/thread
  8x1-bw-process,                         0.148, %,              spread-runtime/thread
  8x1-bw-process,                        19.998, GB,             data/thread
  8x1-bw-process,                       159.988, GB,             data-total
  8x1-bw-process,                         1.003, nsecs,          runtime/byte/thread
  8x1-bw-process,                         0.997, GB/sec,         thread-speed
  8x1-bw-process,                         7.974, GB/sec,         total-speed

 # Running  8x1-bw-process-NOTHP, "perf bench numa mem -p 8 -t 1 -P  512 -s 20 -zZ0q --thp  1 --thp -1"
  8x1-bw-process-NOTHP,                  20.435, secs,           runtime-max/thread
  8x1-bw-process-NOTHP,                  20.150, secs,           runtime-min/thread
  8x1-bw-process-NOTHP,                  20.255, secs,           runtime-avg/thread
  8x1-bw-process-NOTHP,                   0.699, %,              spread-runtime/thread
  8x1-bw-process-NOTHP,                  15.167, GB,             data/thread
  8x1-bw-process-NOTHP,                 121.333, GB,             data-total
  8x1-bw-process-NOTHP,                   1.347, nsecs,          runtime/byte/thread
  8x1-bw-process-NOTHP,                   0.742, GB/sec,         thread-speed
  8x1-bw-process-NOTHP,                   5.937, GB/sec,         total-speed

 # Running 16x1-bw-process, "perf bench numa mem -p 16 -t 1 -P 256 -s 20 -zZ0q --thp  1"
 16x1-bw-process,                        20.451, secs,           runtime-max/thread
 16x1-bw-process,                        20.078, secs,           runtime-min/thread
 16x1-bw-process,                        20.311, secs,           runtime-avg/thread
 16x1-bw-process,                         0.912, %,              spread-runtime/thread
 16x1-bw-process,                         7.147, GB,             data/thread
 16x1-bw-process,                       114.354, GB,             data-total
 16x1-bw-process,                         2.861, nsecs,          runtime/byte/thread
 16x1-bw-process,                         0.349, GB/sec,         thread-speed
 16x1-bw-process,                         5.592, GB/sec,         total-speed

 # Running  4x1-bw-thread, "perf bench numa mem -p 1 -t 4 -T 256 -s 20 -zZ0q --thp  1"
  4x1-bw-thread,                         20.038, secs,           runtime-max/thread
  4x1-bw-thread,                         20.006, secs,           runtime-min/thread
  4x1-bw-thread,                         20.023, secs,           runtime-avg/thread
  4x1-bw-thread,                          0.079, %,              spread-runtime/thread
  4x1-bw-thread,                         68.115, GB,             data/thread
  4x1-bw-thread,                        272.462, GB,             data-total
  4x1-bw-thread,                          0.294, nsecs,          runtime/byte/thread
  4x1-bw-thread,                          3.399, GB/sec,         thread-speed
  4x1-bw-thread,                         13.598, GB/sec,         total-speed

 # Running  8x1-bw-thread, "perf bench numa mem -p 1 -t 8 -T 256 -s 20 -zZ0q --thp  1"
  8x1-bw-thread,                         20.055, secs,           runtime-max/thread
  8x1-bw-thread,                         20.001, secs,           runtime-min/thread
  8x1-bw-thread,                         20.033, secs,           runtime-avg/thread
  8x1-bw-thread,                          0.136, %,              spread-runtime/thread
  8x1-bw-thread,                         41.004, GB,             data/thread
  8x1-bw-thread,                        328.028, GB,             data-total
  8x1-bw-thread,                          0.489, nsecs,          runtime/byte/thread
  8x1-bw-thread,                          2.045, GB/sec,         thread-speed
  8x1-bw-thread,                         16.356, GB/sec,         total-speed

 # Running 16x1-bw-thread, "perf bench numa mem -p 1 -t 16 -T 128 -s 20 -zZ0q --thp  1"
 16x1-bw-thread,                         20.044, secs,           runtime-max/thread
 16x1-bw-thread,                         19.994, secs,           runtime-min/thread
 16x1-bw-thread,                         20.021, secs,           runtime-avg/thread
 16x1-bw-thread,                          0.124, %,              spread-runtime/thread
 16x1-bw-thread,                         30.828, GB,             data/thread
 16x1-bw-thread,                        493.250, GB,             data-total
 16x1-bw-thread,                          0.650, nsecs,          runtime/byte/thread
 16x1-bw-thread,                          1.538, GB/sec,         thread-speed
 16x1-bw-thread,                         24.608, GB/sec,         total-speed

 # Running 32x1-bw-thread, "perf bench numa mem -p 1 -t 32 -T 64 -s 20 -zZ0q --thp  1"
 32x1-bw-thread,                         19.990, secs,           runtime-max/thread
 32x1-bw-thread,                         19.955, secs,           runtime-min/thread
 32x1-bw-thread,                         19.996, secs,           runtime-avg/thread
 32x1-bw-thread,                          0.087, %,              spread-runtime/thread
 32x1-bw-thread,                         15.915, GB,             data/thread
 32x1-bw-thread,                        509.289, GB,             data-total
 32x1-bw-thread,                          1.256, nsecs,          runtime/byte/thread
 32x1-bw-thread,                          0.796, GB/sec,         thread-speed
 32x1-bw-thread,                         25.477, GB/sec,         total-speed

 # Running  2x3-bw-thread, "perf bench numa mem -p 2 -t 3 -P 512 -s 20 -zZ0q --thp  1"
  2x3-bw-thread,                         20.168, secs,           runtime-max/thread
  2x3-bw-thread,                         20.028, secs,           runtime-min/thread
  2x3-bw-thread,                         20.103, secs,           runtime-avg/thread
  2x3-bw-thread,                          0.346, %,              spread-runtime/thread
  2x3-bw-thread,                         29.528, GB,             data/thread
  2x3-bw-thread,                        177.167, GB,             data-total
  2x3-bw-thread,                          0.683, nsecs,          runtime/byte/thread
  2x3-bw-thread,                          1.464, GB/sec,         thread-speed
  2x3-bw-thread,                          8.785, GB/sec,         total-speed

 # Running  4x4-bw-thread, "perf bench numa mem -p 4 -t 4 -P 512 -s 20 -zZ0q --thp  1"
  4x4-bw-thread,                         20.576, secs,           runtime-max/thread
  4x4-bw-thread,                         20.002, secs,           runtime-min/thread
  4x4-bw-thread,                         20.312, secs,           runtime-avg/thread
  4x4-bw-thread,                          1.394, %,              spread-runtime/thread
  4x4-bw-thread,                          8.187, GB,             data/thread
  4x4-bw-thread,                        130.997, GB,             data-total
  4x4-bw-thread,                          2.513, nsecs,          runtime/byte/thread
  4x4-bw-thread,                          0.398, GB/sec,         thread-speed
  4x4-bw-thread,                          6.366, GB/sec,         total-speed

 # Running  4x6-bw-thread, "perf bench numa mem -p 4 -t 6 -P 512 -s 20 -zZ0q --thp  1"
  4x6-bw-thread,                         21.007, secs,           runtime-max/thread
  4x6-bw-thread,                         20.075, secs,           runtime-min/thread
  4x6-bw-thread,                         20.573, secs,           runtime-avg/thread
  4x6-bw-thread,                          2.219, %,              spread-runtime/thread
  4x6-bw-thread,                          5.503, GB,             data/thread
  4x6-bw-thread,                        132.070, GB,             data-total
  4x6-bw-thread,                          3.817, nsecs,          runtime/byte/thread
  4x6-bw-thread,                          0.262, GB/sec,         thread-speed
  4x6-bw-thread,                          6.287, GB/sec,         total-speed

 # Running  4x8-bw-thread, "perf bench numa mem -p 4 -t 8 -P 512 -s 20 -zZ0q --thp  1"
  4x8-bw-thread,                         21.986, secs,           runtime-max/thread
  4x8-bw-thread,                         20.359, secs,           runtime-min/thread
  4x8-bw-thread,                         21.300, secs,           runtime-avg/thread
  4x8-bw-thread,                          3.701, %,              spread-runtime/thread
  4x8-bw-thread,                          4.027, GB,             data/thread
  4x8-bw-thread,                        128.849, GB,             data-total
  4x8-bw-thread,                          5.460, nsecs,          runtime/byte/thread
  4x8-bw-thread,                          0.183, GB/sec,         thread-speed
  4x8-bw-thread,                          5.860, GB/sec,         total-speed

 # Running  4x8-bw-thread-NOTHP, "perf bench numa mem -p 4 -t 8 -P 512 -s 20 -zZ0q --thp  1 --thp -1"
  4x8-bw-thread-NOTHP,                   21.155, secs,           runtime-max/thread
  4x8-bw-thread-NOTHP,                   20.115, secs,           runtime-min/thread
  4x8-bw-thread-NOTHP,                   20.705, secs,           runtime-avg/thread
  4x8-bw-thread-NOTHP,                    2.459, %,              spread-runtime/thread
  4x8-bw-thread-NOTHP,                    4.077, GB,             data/thread
  4x8-bw-thread-NOTHP,                  130.460, GB,             data-total
  4x8-bw-thread-NOTHP,                    5.189, nsecs,          runtime/byte/thread
  4x8-bw-thread-NOTHP,                    0.193, GB/sec,         thread-speed
  4x8-bw-thread-NOTHP,                    6.167, GB/sec,         total-speed

 # Running  3x3-bw-thread, "perf bench numa mem -p 3 -t 3 -P 512 -s 20 -zZ0q --thp  1"
  3x3-bw-thread,                         20.211, secs,           runtime-max/thread
  3x3-bw-thread,                         20.044, secs,           runtime-min/thread
  3x3-bw-thread,                         20.127, secs,           runtime-avg/thread
  3x3-bw-thread,                          0.413, %,              spread-runtime/thread
  3x3-bw-thread,                         18.492, GB,             data/thread
  3x3-bw-thread,                        166.430, GB,             data-total
  3x3-bw-thread,                          1.093, nsecs,          runtime/byte/thread
  3x3-bw-thread,                          0.915, GB/sec,         thread-speed
  3x3-bw-thread,                          8.235, GB/sec,         total-speed

 # Running  5x5-bw-thread, "perf bench numa mem -p 5 -t 5 -P 512 -s 20 -zZ0q --thp  1"
  5x5-bw-thread,                         21.244, secs,           runtime-max/thread
  5x5-bw-thread,                         20.115, secs,           runtime-min/thread
  5x5-bw-thread,                         20.873, secs,           runtime-avg/thread
  5x5-bw-thread,                          2.657, %,              spread-runtime/thread
  5x5-bw-thread,                          4.896, GB,             data/thread
  5x5-bw-thread,                        122.407, GB,             data-total
  5x5-bw-thread,                          4.339, nsecs,          runtime/byte/thread
  5x5-bw-thread,                          0.230, GB/sec,         thread-speed
  5x5-bw-thread,                          5.762, GB/sec,         total-speed

 # Running 2x16-bw-thread, "perf bench numa mem -p 2 -t 16 -P 512 -s 20 -zZ0q --thp  1"
 2x16-bw-thread,                         21.854, secs,           runtime-max/thread
 2x16-bw-thread,                         20.047, secs,           runtime-min/thread
 2x16-bw-thread,                         21.157, secs,           runtime-avg/thread
 2x16-bw-thread,                          4.135, %,              spread-runtime/thread
 2x16-bw-thread,                          4.043, GB,             data/thread
 2x16-bw-thread,                        129.386, GB,             data-total
 2x16-bw-thread,                          5.405, nsecs,          runtime/byte/thread
 2x16-bw-thread,                          0.185, GB/sec,         thread-speed
 2x16-bw-thread,                          5.920, GB/sec,         total-speed

 # Running 1x32-bw-thread, "perf bench numa mem -p 1 -t 32 -P 2048 -s 20 -zZ0q --thp  1"
 1x32-bw-thread,                         23.952, secs,           runtime-max/thread
 1x32-bw-thread,                         20.470, secs,           runtime-min/thread
 1x32-bw-thread,                         22.975, secs,           runtime-avg/thread
 1x32-bw-thread,                          7.268, %,              spread-runtime/thread
 1x32-bw-thread,                          4.362, GB,             data/thread
 1x32-bw-thread,                        139.586, GB,             data-total
 1x32-bw-thread,                          5.491, nsecs,          runtime/byte/thread
 1x32-bw-thread,                          0.182, GB/sec,         thread-speed
 1x32-bw-thread,                          5.828, GB/sec,         total-speed

 # Running numa02-bw, "perf bench numa mem -p 1 -t 32 -T 32 -s 20 -zZ0q --thp  1"
 numa02-bw,                              19.990, secs,           runtime-max/thread
 numa02-bw,                              19.975, secs,           runtime-min/thread
 numa02-bw,                              19.995, secs,           runtime-avg/thread
 numa02-bw,                               0.037, %,              spread-runtime/thread
 numa02-bw,                              18.150, GB,             data/thread
 numa02-bw,                             580.794, GB,             data-total
 numa02-bw,                               1.101, nsecs,          runtime/byte/thread
 numa02-bw,                               0.908, GB/sec,         thread-speed
 numa02-bw,                              29.054, GB/sec,         total-speed

 # Running numa02-bw-NOTHP, "perf bench numa mem -p 1 -t 32 -T 32 -s 20 -zZ0q --thp  1 --thp -1"
 numa02-bw-NOTHP,                        20.072, secs,           runtime-max/thread
 numa02-bw-NOTHP,                        19.965, secs,           runtime-min/thread
 numa02-bw-NOTHP,                        19.998, secs,           runtime-avg/thread
 numa02-bw-NOTHP,                         0.266, %,              spread-runtime/thread
 numa02-bw-NOTHP,                        16.975, GB,             data/thread
 numa02-bw-NOTHP,                       543.213, GB,             data-total
 numa02-bw-NOTHP,                         1.182, nsecs,          runtime/byte/thread
 numa02-bw-NOTHP,                         0.846, GB/sec,         thread-speed
 numa02-bw-NOTHP,                        27.064, GB/sec,         total-speed

 # Running numa01-bw-thread, "perf bench numa mem -p 2 -t 16 -T 192 -s 20 -zZ0q --thp  1"
 numa01-bw-thread,                       20.125, secs,           runtime-max/thread
 numa01-bw-thread,                       19.980, secs,           runtime-min/thread
 numa01-bw-thread,                       20.094, secs,           runtime-avg/thread
 numa01-bw-thread,                        0.361, %,              spread-runtime/thread
 numa01-bw-thread,                       12.791, GB,             data/thread
 numa01-bw-thread,                      409.297, GB,             data-total
 numa01-bw-thread,                        1.573, nsecs,          runtime/byte/thread
 numa01-bw-thread,                        0.636, GB/sec,         thread-speed
 numa01-bw-thread,                       20.338, GB/sec,         total-speed

 # Running numa01-bw-thread-NOTHP, "perf bench numa mem -p 2 -t 16 -T 192 -s 20 -zZ0q --thp  1 --thp -1"
 numa01-bw-thread-NOTHP,                 20.298, secs,           runtime-max/thread
 numa01-bw-thread-NOTHP,                 19.965, secs,           runtime-min/thread
 numa01-bw-thread-NOTHP,                 20.055, secs,           runtime-avg/thread
 numa01-bw-thread-NOTHP,                  0.820, %,              spread-runtime/thread
 numa01-bw-thread-NOTHP,                 11.752, GB,             data/thread
 numa01-bw-thread-NOTHP,                376.078, GB,             data-total
 numa01-bw-thread-NOTHP,                  1.727, nsecs,          runtime/byte/thread
 numa01-bw-thread-NOTHP,                  0.579, GB/sec,         thread-speed
 numa01-bw-thread-NOTHP,                 18.528, GB/sec,         total-speed

 #
 # Running test on: Linux vega 3.6.0+ #4 SMP Fri Dec 7 19:14:49 CET 2012 x86_64 x86_64 x86_64 GNU/Linux
 #
# Running numa/mem benchmark...

 # Running main, "perf bench numa mem -a"

 # Running RAM-bw-local, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 0 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-local,                           20.080, secs,           runtime-max/thread
 RAM-bw-local,                           20.073, secs,           runtime-min/thread
 RAM-bw-local,                           20.073, secs,           runtime-avg/thread
 RAM-bw-local,                            0.018, %,              spread-runtime/thread
 RAM-bw-local,                          170.725, GB,             data/thread
 RAM-bw-local,                          170.725, GB,             data-total
 RAM-bw-local,                            0.118, nsecs,          runtime/byte/thread
 RAM-bw-local,                            8.502, GB/sec,         thread-speed
 RAM-bw-local,                            8.502, GB/sec,         total-speed

 # Running RAM-bw-local-NOTHP, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 0 -s 20 -zZq --thp  1 --no-data_rand_walk --thp -1"
 RAM-bw-local-NOTHP,                     20.112, secs,           runtime-max/thread
 RAM-bw-local-NOTHP,                     20.028, secs,           runtime-min/thread
 RAM-bw-local-NOTHP,                     20.028, secs,           runtime-avg/thread
 RAM-bw-local-NOTHP,                      0.209, %,              spread-runtime/thread
 RAM-bw-local-NOTHP,                    169.651, GB,             data/thread
 RAM-bw-local-NOTHP,                    169.651, GB,             data-total
 RAM-bw-local-NOTHP,                      0.119, nsecs,          runtime/byte/thread
 RAM-bw-local-NOTHP,                      8.435, GB/sec,         thread-speed
 RAM-bw-local-NOTHP,                      8.435, GB/sec,         total-speed

 # Running RAM-bw-remote, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 1 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-remote,                          20.101, secs,           runtime-max/thread
 RAM-bw-remote,                          20.093, secs,           runtime-min/thread
 RAM-bw-remote,                          20.093, secs,           runtime-avg/thread
 RAM-bw-remote,                           0.021, %,              spread-runtime/thread
 RAM-bw-remote,                         134.218, GB,             data/thread
 RAM-bw-remote,                         134.218, GB,             data-total
 RAM-bw-remote,                           0.150, nsecs,          runtime/byte/thread
 RAM-bw-remote,                           6.677, GB/sec,         thread-speed
 RAM-bw-remote,                           6.677, GB/sec,         total-speed

 # Running RAM-bw-local-2x, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,2 -M 0x2 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-local-2x,                        20.109, secs,           runtime-max/thread
 RAM-bw-local-2x,                        20.011, secs,           runtime-min/thread
 RAM-bw-local-2x,                        20.056, secs,           runtime-avg/thread
 RAM-bw-local-2x,                         0.243, %,              spread-runtime/thread
 RAM-bw-local-2x,                       135.291, GB,             data/thread
 RAM-bw-local-2x,                       270.583, GB,             data-total
 RAM-bw-local-2x,                         0.149, nsecs,          runtime/byte/thread
 RAM-bw-local-2x,                         6.728, GB/sec,         thread-speed
 RAM-bw-local-2x,                        13.456, GB/sec,         total-speed

 # Running RAM-bw-remote-2x, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,2 -M 1x2 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-remote-2x,                       20.292, secs,           runtime-max/thread
 RAM-bw-remote-2x,                       20.279, secs,           runtime-min/thread
 RAM-bw-remote-2x,                       20.281, secs,           runtime-avg/thread
 RAM-bw-remote-2x,                        0.034, %,              spread-runtime/thread
 RAM-bw-remote-2x,                       74.625, GB,             data/thread
 RAM-bw-remote-2x,                      149.250, GB,             data-total
 RAM-bw-remote-2x,                        0.272, nsecs,          runtime/byte/thread
 RAM-bw-remote-2x,                        3.677, GB/sec,         thread-speed
 RAM-bw-remote-2x,                        7.355, GB/sec,         total-speed

 # Running RAM-bw-cross, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,8 -M 1,0 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-cross,                           20.177, secs,           runtime-max/thread
 RAM-bw-cross,                           20.158, secs,           runtime-min/thread
 RAM-bw-cross,                           20.163, secs,           runtime-avg/thread
 RAM-bw-cross,                            0.048, %,              spread-runtime/thread
 RAM-bw-cross,                          122.943, GB,             data/thread
 RAM-bw-cross,                          245.887, GB,             data-total
 RAM-bw-cross,                            0.164, nsecs,          runtime/byte/thread
 RAM-bw-cross,                            6.093, GB/sec,         thread-speed
 RAM-bw-cross,                           12.187, GB/sec,         total-speed

 # Running  1x3-convergence, "perf bench numa mem -p 1 -t 3 -P 512 -s 100 -zZ0qcm --thp  1"
  1x3-convergence,                        0.224, secs,           NUMA-convergence-latency
  1x3-convergence,                        0.224, secs,           runtime-max/thread
  1x3-convergence,                        0.205, secs,           runtime-min/thread
  1x3-convergence,                        0.214, secs,           runtime-avg/thread
  1x3-convergence,                        4.078, %,              spread-runtime/thread
  1x3-convergence,                        0.537, GB,             data/thread
  1x3-convergence,                        1.611, GB,             data-total
  1x3-convergence,                        0.417, nsecs,          runtime/byte/thread
  1x3-convergence,                        2.401, GB/sec,         thread-speed
  1x3-convergence,                        7.202, GB/sec,         total-speed

 # Running  1x4-convergence, "perf bench numa mem -p 1 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  1x4-convergence,                      100.173, secs,           NUMA-convergence-latency
  1x4-convergence,                      100.173, secs,           runtime-max/thread
  1x4-convergence,                      100.026, secs,           runtime-min/thread
  1x4-convergence,                      100.067, secs,           runtime-avg/thread
  1x4-convergence,                        0.073, %,              spread-runtime/thread
  1x4-convergence,                      162.672, GB,             data/thread
  1x4-convergence,                      650.688, GB,             data-total
  1x4-convergence,                        0.616, nsecs,          runtime/byte/thread
  1x4-convergence,                        1.624, GB/sec,         thread-speed
  1x4-convergence,                        6.496, GB/sec,         total-speed

 # Running  1x6-convergence, "perf bench numa mem -p 1 -t 6 -P 1020 -s 100 -zZ0qcm --thp  1"
  1x6-convergence,                      100.821, secs,           NUMA-convergence-latency
  1x6-convergence,                      100.821, secs,           runtime-max/thread
  1x6-convergence,                      100.428, secs,           runtime-min/thread
  1x6-convergence,                      100.706, secs,           runtime-avg/thread
  1x6-convergence,                        0.195, %,              spread-runtime/thread
  1x6-convergence,                       99.111, GB,             data/thread
  1x6-convergence,                      594.668, GB,             data-total
  1x6-convergence,                        1.017, nsecs,          runtime/byte/thread
  1x6-convergence,                        0.983, GB/sec,         thread-speed
  1x6-convergence,                        5.898, GB/sec,         total-speed

 # Running  2x3-convergence, "perf bench numa mem -p 3 -t 3 -P 1020 -s 100 -zZ0qcm --thp  1"
  2x3-convergence,                      100.539, secs,           NUMA-convergence-latency
  2x3-convergence,                      100.539, secs,           runtime-max/thread
  2x3-convergence,                      100.015, secs,           runtime-min/thread
  2x3-convergence,                      100.273, secs,           runtime-avg/thread
  2x3-convergence,                        0.260, %,              spread-runtime/thread
  2x3-convergence,                      147.954, GB,             data/thread
  2x3-convergence,                     1331.587, GB,             data-total
  2x3-convergence,                        0.680, nsecs,          runtime/byte/thread
  2x3-convergence,                        1.472, GB/sec,         thread-speed
  2x3-convergence,                       13.245, GB/sec,         total-speed

 # Running  3x3-convergence, "perf bench numa mem -p 3 -t 3 -P 1020 -s 100 -zZ0qcm --thp  1"
  3x3-convergence,                      100.463, secs,           NUMA-convergence-latency
  3x3-convergence,                      100.463, secs,           runtime-max/thread
  3x3-convergence,                      100.066, secs,           runtime-min/thread
  3x3-convergence,                      100.216, secs,           runtime-avg/thread
  3x3-convergence,                        0.198, %,              spread-runtime/thread
  3x3-convergence,                      132.624, GB,             data/thread
  3x3-convergence,                     1193.615, GB,             data-total
  3x3-convergence,                        0.758, nsecs,          runtime/byte/thread
  3x3-convergence,                        1.320, GB/sec,         thread-speed
  3x3-convergence,                       11.881, GB/sec,         total-speed

 # Running  4x4-convergence, "perf bench numa mem -p 4 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  4x4-convergence,                        4.119, secs,           NUMA-convergence-latency
  4x4-convergence,                        4.119, secs,           runtime-max/thread
  4x4-convergence,                        3.751, secs,           runtime-min/thread
  4x4-convergence,                        3.948, secs,           runtime-avg/thread
  4x4-convergence,                        4.462, %,              spread-runtime/thread
  4x4-convergence,                        1.980, GB,             data/thread
  4x4-convergence,                       31.675, GB,             data-total
  4x4-convergence,                        2.081, nsecs,          runtime/byte/thread
  4x4-convergence,                        0.481, GB/sec,         thread-speed
  4x4-convergence,                        7.690, GB/sec,         total-speed

 # Running  4x4-convergence-NOTHP, "perf bench numa mem -p 4 -t 4 -P 512 -s 100 -zZ0qcm --thp  1 --thp -1"
  4x4-convergence-NOTHP,                 12.166, secs,           NUMA-convergence-latency
  4x4-convergence-NOTHP,                 12.166, secs,           runtime-max/thread
  4x4-convergence-NOTHP,                 11.801, secs,           runtime-min/thread
  4x4-convergence-NOTHP,                 11.917, secs,           runtime-avg/thread
  4x4-convergence-NOTHP,                  1.502, %,              spread-runtime/thread
  4x4-convergence-NOTHP,                  5.234, GB,             data/thread
  4x4-convergence-NOTHP,                 83.752, GB,             data-total
  4x4-convergence-NOTHP,                  2.324, nsecs,          runtime/byte/thread
  4x4-convergence-NOTHP,                  0.430, GB/sec,         thread-speed
  4x4-convergence-NOTHP,                  6.884, GB/sec,         total-speed

 # Running  4x6-convergence, "perf bench numa mem -p 4 -t 6 -P 1020 -s 100 -zZ0qcm --thp  1"
  4x6-convergence,                       16.592, secs,           NUMA-convergence-latency
  4x6-convergence,                       16.592, secs,           runtime-max/thread
  4x6-convergence,                       15.407, secs,           runtime-min/thread
  4x6-convergence,                       16.109, secs,           runtime-avg/thread
  4x6-convergence,                        3.572, %,              spread-runtime/thread
  4x6-convergence,                        6.729, GB,             data/thread
  4x6-convergence,                      161.502, GB,             data-total
  4x6-convergence,                        2.466, nsecs,          runtime/byte/thread
  4x6-convergence,                        0.406, GB/sec,         thread-speed
  4x6-convergence,                        9.734, GB/sec,         total-speed

 # Running  4x8-convergence, "perf bench numa mem -p 4 -t 8 -P 512 -s 100 -zZ0qcm --thp  1"
  4x8-convergence,                        3.385, secs,           NUMA-convergence-latency
  4x8-convergence,                        3.385, secs,           runtime-max/thread
  4x8-convergence,                        1.465, secs,           runtime-min/thread
  4x8-convergence,                        2.846, secs,           runtime-avg/thread
  4x8-convergence,                       28.361, %,              spread-runtime/thread
  4x8-convergence,                        0.638, GB,             data/thread
  4x8-convergence,                       20.401, GB,             data-total
  4x8-convergence,                        5.309, nsecs,          runtime/byte/thread
  4x8-convergence,                        0.188, GB/sec,         thread-speed
  4x8-convergence,                        6.028, GB/sec,         total-speed

 # Running  8x4-convergence, "perf bench numa mem -p 8 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  8x4-convergence,                       18.295, secs,           NUMA-convergence-latency
  8x4-convergence,                       18.295, secs,           runtime-max/thread
  8x4-convergence,                       16.808, secs,           runtime-min/thread
  8x4-convergence,                       17.809, secs,           runtime-avg/thread
  8x4-convergence,                        4.064, %,              spread-runtime/thread
  8x4-convergence,                        3.406, GB,             data/thread
  8x4-convergence,                      108.985, GB,             data-total
  8x4-convergence,                        5.372, nsecs,          runtime/byte/thread
  8x4-convergence,                        0.186, GB/sec,         thread-speed
  8x4-convergence,                        5.957, GB/sec,         total-speed

 # Running  8x4-convergence-NOTHP, "perf bench numa mem -p 8 -t 4 -P 512 -s 100 -zZ0qcm --thp  1 --thp -1"
  8x4-convergence-NOTHP,                 15.675, secs,           NUMA-convergence-latency
  8x4-convergence-NOTHP,                 15.675, secs,           runtime-max/thread
  8x4-convergence-NOTHP,                 14.861, secs,           runtime-min/thread
  8x4-convergence-NOTHP,                 15.321, secs,           runtime-avg/thread
  8x4-convergence-NOTHP,                  2.596, %,              spread-runtime/thread
  8x4-convergence-NOTHP,                  5.302, GB,             data/thread
  8x4-convergence-NOTHP,                169.651, GB,             data-total
  8x4-convergence-NOTHP,                  2.957, nsecs,          runtime/byte/thread
  8x4-convergence-NOTHP,                  0.338, GB/sec,         thread-speed
  8x4-convergence-NOTHP,                 10.823, GB/sec,         total-speed

 # Running  3x1-convergence, "perf bench numa mem -p 3 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  3x1-convergence,                        0.811, secs,           NUMA-convergence-latency
  3x1-convergence,                        0.811, secs,           runtime-max/thread
  3x1-convergence,                        0.739, secs,           runtime-min/thread
  3x1-convergence,                        0.782, secs,           runtime-avg/thread
  3x1-convergence,                        4.431, %,              spread-runtime/thread
  3x1-convergence,                        1.969, GB,             data/thread
  3x1-convergence,                        5.906, GB,             data-total
  3x1-convergence,                        0.412, nsecs,          runtime/byte/thread
  3x1-convergence,                        2.428, GB/sec,         thread-speed
  3x1-convergence,                        7.284, GB/sec,         total-speed

 # Running  4x1-convergence, "perf bench numa mem -p 4 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  4x1-convergence,                        0.806, secs,           NUMA-convergence-latency
  4x1-convergence,                        0.806, secs,           runtime-max/thread
  4x1-convergence,                        0.728, secs,           runtime-min/thread
  4x1-convergence,                        0.780, secs,           runtime-avg/thread
  4x1-convergence,                        4.838, %,              spread-runtime/thread
  4x1-convergence,                        1.476, GB,             data/thread
  4x1-convergence,                        5.906, GB,             data-total
  4x1-convergence,                        0.546, nsecs,          runtime/byte/thread
  4x1-convergence,                        1.832, GB/sec,         thread-speed
  4x1-convergence,                        7.329, GB/sec,         total-speed

 # Running  8x1-convergence, "perf bench numa mem -p 8 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  8x1-convergence,                        2.879, secs,           NUMA-convergence-latency
  8x1-convergence,                        2.879, secs,           runtime-max/thread
  8x1-convergence,                        2.737, secs,           runtime-min/thread
  8x1-convergence,                        2.805, secs,           runtime-avg/thread
  8x1-convergence,                        2.475, %,              spread-runtime/thread
  8x1-convergence,                        3.288, GB,             data/thread
  8x1-convergence,                       26.307, GB,             data-total
  8x1-convergence,                        0.876, nsecs,          runtime/byte/thread
  8x1-convergence,                        1.142, GB/sec,         thread-speed
  8x1-convergence,                        9.137, GB/sec,         total-speed

 # Running 16x1-convergence, "perf bench numa mem -p 16 -t 1 -P 256 -s 100 -zZ0qcm --thp  1"
 16x1-convergence,                        2.484, secs,           NUMA-convergence-latency
 16x1-convergence,                        2.484, secs,           runtime-max/thread
 16x1-convergence,                        2.169, secs,           runtime-min/thread
 16x1-convergence,                        2.376, secs,           runtime-avg/thread
 16x1-convergence,                        6.353, %,              spread-runtime/thread
 16x1-convergence,                        0.906, GB,             data/thread
 16x1-convergence,                       14.496, GB,             data-total
 16x1-convergence,                        2.742, nsecs,          runtime/byte/thread
 16x1-convergence,                        0.365, GB/sec,         thread-speed
 16x1-convergence,                        5.835, GB/sec,         total-speed

 # Running 32x1-convergence, "perf bench numa mem -p 32 -t 1 -P 128 -s 100 -zZ0qcm --thp  1"
 32x1-convergence,                        3.039, secs,           NUMA-convergence-latency
 32x1-convergence,                        3.039, secs,           runtime-max/thread
 32x1-convergence,                        2.755, secs,           runtime-min/thread
 32x1-convergence,                        2.983, secs,           runtime-avg/thread
 32x1-convergence,                        4.672, %,              spread-runtime/thread
 32x1-convergence,                        0.579, GB,             data/thread
 32x1-convergence,                       18.522, GB,             data-total
 32x1-convergence,                        5.251, nsecs,          runtime/byte/thread
 32x1-convergence,                        0.190, GB/sec,         thread-speed
 32x1-convergence,                        6.094, GB/sec,         total-speed

 # Running  2x1-bw-process, "perf bench numa mem -p 2 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  2x1-bw-process,                        20.217, secs,           runtime-max/thread
  2x1-bw-process,                        20.126, secs,           runtime-min/thread
  2x1-bw-process,                        20.168, secs,           runtime-avg/thread
  2x1-bw-process,                         0.224, %,              spread-runtime/thread
  2x1-bw-process,                        81.604, GB,             data/thread
  2x1-bw-process,                       163.209, GB,             data-total
  2x1-bw-process,                         0.248, nsecs,          runtime/byte/thread
  2x1-bw-process,                         4.036, GB/sec,         thread-speed
  2x1-bw-process,                         8.073, GB/sec,         total-speed

 # Running  3x1-bw-process, "perf bench numa mem -p 3 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  3x1-bw-process,                        20.138, secs,           runtime-max/thread
  3x1-bw-process,                        20.075, secs,           runtime-min/thread
  3x1-bw-process,                        20.105, secs,           runtime-avg/thread
  3x1-bw-process,                         0.156, %,              spread-runtime/thread
  3x1-bw-process,                        84.468, GB,             data/thread
  3x1-bw-process,                       253.403, GB,             data-total
  3x1-bw-process,                         0.238, nsecs,          runtime/byte/thread
  3x1-bw-process,                         4.194, GB/sec,         thread-speed
  3x1-bw-process,                        12.583, GB/sec,         total-speed

 # Running  4x1-bw-process, "perf bench numa mem -p 4 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  4x1-bw-process,                        20.143, secs,           runtime-max/thread
  4x1-bw-process,                        20.052, secs,           runtime-min/thread
  4x1-bw-process,                        20.079, secs,           runtime-avg/thread
  4x1-bw-process,                         0.227, %,              spread-runtime/thread
  4x1-bw-process,                        62.009, GB,             data/thread
  4x1-bw-process,                       248.034, GB,             data-total
  4x1-bw-process,                         0.325, nsecs,          runtime/byte/thread
  4x1-bw-process,                         3.078, GB/sec,         thread-speed
  4x1-bw-process,                        12.313, GB/sec,         total-speed

 # Running  8x1-bw-process, "perf bench numa mem -p 8 -t 1 -P  512 -s 20 -zZ0q --thp  1"
  8x1-bw-process,                        20.109, secs,           runtime-max/thread
  8x1-bw-process,                        20.013, secs,           runtime-min/thread
  8x1-bw-process,                        20.072, secs,           runtime-avg/thread
  8x1-bw-process,                         0.238, %,              spread-runtime/thread
  8x1-bw-process,                        50.869, GB,             data/thread
  8x1-bw-process,                       406.948, GB,             data-total
  8x1-bw-process,                         0.395, nsecs,          runtime/byte/thread
  8x1-bw-process,                         2.530, GB/sec,         thread-speed
  8x1-bw-process,                        20.237, GB/sec,         total-speed

 # Running  8x1-bw-process-NOTHP, "perf bench numa mem -p 8 -t 1 -P  512 -s 20 -zZ0q --thp  1 --thp -1"
  8x1-bw-process-NOTHP,                  20.203, secs,           runtime-max/thread
  8x1-bw-process-NOTHP,                  20.033, secs,           runtime-min/thread
  8x1-bw-process-NOTHP,                  20.071, secs,           runtime-avg/thread
  8x1-bw-process-NOTHP,                   0.422, %,              spread-runtime/thread
  8x1-bw-process-NOTHP,                  45.030, GB,             data/thread
  8x1-bw-process-NOTHP,                 360.240, GB,             data-total
  8x1-bw-process-NOTHP,                   0.449, nsecs,          runtime/byte/thread
  8x1-bw-process-NOTHP,                   2.229, GB/sec,         thread-speed
  8x1-bw-process-NOTHP,                  17.831, GB/sec,         total-speed

 # Running 16x1-bw-process, "perf bench numa mem -p 16 -t 1 -P 256 -s 20 -zZ0q --thp  1"
 16x1-bw-process,                        20.271, secs,           runtime-max/thread
 16x1-bw-process,                        20.021, secs,           runtime-min/thread
 16x1-bw-process,                        20.175, secs,           runtime-avg/thread
 16x1-bw-process,                         0.615, %,              spread-runtime/thread
 16x1-bw-process,                         7.550, GB,             data/thread
 16x1-bw-process,                       120.796, GB,             data-total
 16x1-bw-process,                         2.685, nsecs,          runtime/byte/thread
 16x1-bw-process,                         0.372, GB/sec,         thread-speed
 16x1-bw-process,                         5.959, GB/sec,         total-speed

 # Running  4x1-bw-thread, "perf bench numa mem -p 1 -t 4 -T 256 -s 20 -zZ0q --thp  1"
  4x1-bw-thread,                         20.052, secs,           runtime-max/thread
  4x1-bw-thread,                         20.013, secs,           runtime-min/thread
  4x1-bw-thread,                         20.030, secs,           runtime-avg/thread
  4x1-bw-thread,                          0.097, %,              spread-runtime/thread
  4x1-bw-thread,                         87.443, GB,             data/thread
  4x1-bw-thread,                        349.771, GB,             data-total
  4x1-bw-thread,                          0.229, nsecs,          runtime/byte/thread
  4x1-bw-thread,                          4.361, GB/sec,         thread-speed
  4x1-bw-thread,                         17.443, GB/sec,         total-speed

 # Running  8x1-bw-thread, "perf bench numa mem -p 1 -t 8 -T 256 -s 20 -zZ0q --thp  1"
  8x1-bw-thread,                         20.067, secs,           runtime-max/thread
  8x1-bw-thread,                         20.011, secs,           runtime-min/thread
  8x1-bw-thread,                         20.038, secs,           runtime-avg/thread
  8x1-bw-thread,                          0.140, %,              spread-runtime/thread
  8x1-bw-thread,                         56.271, GB,             data/thread
  8x1-bw-thread,                        450.166, GB,             data-total
  8x1-bw-thread,                          0.357, nsecs,          runtime/byte/thread
  8x1-bw-thread,                          2.804, GB/sec,         thread-speed
  8x1-bw-thread,                         22.433, GB/sec,         total-speed

 # Running 16x1-bw-thread, "perf bench numa mem -p 1 -t 16 -T 128 -s 20 -zZ0q --thp  1"
 16x1-bw-thread,                         20.029, secs,           runtime-max/thread
 16x1-bw-thread,                         20.002, secs,           runtime-min/thread
 16x1-bw-thread,                         20.020, secs,           runtime-avg/thread
 16x1-bw-thread,                          0.067, %,              spread-runtime/thread
 16x1-bw-thread,                         25.292, GB,             data/thread
 16x1-bw-thread,                        404.666, GB,             data-total
 16x1-bw-thread,                          0.792, nsecs,          runtime/byte/thread
 16x1-bw-thread,                          1.263, GB/sec,         thread-speed
 16x1-bw-thread,                         20.204, GB/sec,         total-speed

 # Running 32x1-bw-thread, "perf bench numa mem -p 1 -t 32 -T 64 -s 20 -zZ0q --thp  1"
 32x1-bw-thread,                         19.989, secs,           runtime-max/thread
 32x1-bw-thread,                         19.962, secs,           runtime-min/thread
 32x1-bw-thread,                         20.004, secs,           runtime-avg/thread
 32x1-bw-thread,                          0.068, %,              spread-runtime/thread
 32x1-bw-thread,                         11.388, GB,             data/thread
 32x1-bw-thread,                        364.401, GB,             data-total
 32x1-bw-thread,                          1.755, nsecs,          runtime/byte/thread
 32x1-bw-thread,                          0.570, GB/sec,         thread-speed
 32x1-bw-thread,                         18.230, GB/sec,         total-speed

 # Running  2x3-bw-thread, "perf bench numa mem -p 2 -t 3 -P 512 -s 20 -zZ0q --thp  1"
  2x3-bw-thread,                         20.190, secs,           runtime-max/thread
  2x3-bw-thread,                         20.082, secs,           runtime-min/thread
  2x3-bw-thread,                         20.110, secs,           runtime-avg/thread
  2x3-bw-thread,                          0.268, %,              spread-runtime/thread
  2x3-bw-thread,                         49.303, GB,             data/thread
  2x3-bw-thread,                        295.816, GB,             data-total
  2x3-bw-thread,                          0.410, nsecs,          runtime/byte/thread
  2x3-bw-thread,                          2.442, GB/sec,         thread-speed
  2x3-bw-thread,                         14.652, GB/sec,         total-speed

 # Running  4x4-bw-thread, "perf bench numa mem -p 4 -t 4 -P 512 -s 20 -zZ0q --thp  1"
  4x4-bw-thread,                         20.307, secs,           runtime-max/thread
  4x4-bw-thread,                         20.002, secs,           runtime-min/thread
  4x4-bw-thread,                         20.202, secs,           runtime-avg/thread
  4x4-bw-thread,                          0.750, %,              spread-runtime/thread
  4x4-bw-thread,                         12.482, GB,             data/thread
  4x4-bw-thread,                        199.716, GB,             data-total
  4x4-bw-thread,                          1.627, nsecs,          runtime/byte/thread
  4x4-bw-thread,                          0.615, GB/sec,         thread-speed
  4x4-bw-thread,                          9.835, GB/sec,         total-speed

 # Running  4x6-bw-thread, "perf bench numa mem -p 4 -t 6 -P 512 -s 20 -zZ0q --thp  1"
  4x6-bw-thread,                         20.431, secs,           runtime-max/thread
  4x6-bw-thread,                         20.007, secs,           runtime-min/thread
  4x6-bw-thread,                         20.283, secs,           runtime-avg/thread
  4x6-bw-thread,                          1.036, %,              spread-runtime/thread
  4x6-bw-thread,                         13.086, GB,             data/thread
  4x6-bw-thread,                        314.069, GB,             data-total
  4x6-bw-thread,                          1.561, nsecs,          runtime/byte/thread
  4x6-bw-thread,                          0.641, GB/sec,         thread-speed
  4x6-bw-thread,                         15.372, GB/sec,         total-speed

 # Running  4x8-bw-thread, "perf bench numa mem -p 4 -t 8 -P 512 -s 20 -zZ0q --thp  1"
  4x8-bw-thread,                         20.543, secs,           runtime-max/thread
  4x8-bw-thread,                         20.015, secs,           runtime-min/thread
  4x8-bw-thread,                         20.324, secs,           runtime-avg/thread
  4x8-bw-thread,                          1.287, %,              spread-runtime/thread
  4x8-bw-thread,                          7.617, GB,             data/thread
  4x8-bw-thread,                        243.739, GB,             data-total
  4x8-bw-thread,                          2.697, nsecs,          runtime/byte/thread
  4x8-bw-thread,                          0.371, GB/sec,         thread-speed
  4x8-bw-thread,                         11.865, GB/sec,         total-speed

 # Running  4x8-bw-thread-NOTHP, "perf bench numa mem -p 4 -t 8 -P 512 -s 20 -zZ0q --thp  1 --thp -1"
  4x8-bw-thread-NOTHP,                   20.661, secs,           runtime-max/thread
  4x8-bw-thread-NOTHP,                   20.023, secs,           runtime-min/thread
  4x8-bw-thread-NOTHP,                   20.292, secs,           runtime-avg/thread
  4x8-bw-thread-NOTHP,                    1.546, %,              spread-runtime/thread
  4x8-bw-thread-NOTHP,                    5.956, GB,             data/thread
  4x8-bw-thread-NOTHP,                  190.589, GB,             data-total
  4x8-bw-thread-NOTHP,                    3.469, nsecs,          runtime/byte/thread
  4x8-bw-thread-NOTHP,                    0.288, GB/sec,         thread-speed
  4x8-bw-thread-NOTHP,                    9.224, GB/sec,         total-speed

 # Running  3x3-bw-thread, "perf bench numa mem -p 3 -t 3 -P 512 -s 20 -zZ0q --thp  1"
  3x3-bw-thread,                         20.310, secs,           runtime-max/thread
  3x3-bw-thread,                         20.116, secs,           runtime-min/thread
  3x3-bw-thread,                         20.202, secs,           runtime-avg/thread
  3x3-bw-thread,                          0.480, %,              spread-runtime/thread
  3x3-bw-thread,                         14.973, GB,             data/thread
  3x3-bw-thread,                        134.755, GB,             data-total
  3x3-bw-thread,                          1.356, nsecs,          runtime/byte/thread
  3x3-bw-thread,                          0.737, GB/sec,         thread-speed
  3x3-bw-thread,                          6.635, GB/sec,         total-speed

 # Running  5x5-bw-thread, "perf bench numa mem -p 5 -t 5 -P 512 -s 20 -zZ0q --thp  1"
  5x5-bw-thread,                         20.578, secs,           runtime-max/thread
  5x5-bw-thread,                         20.039, secs,           runtime-min/thread
  5x5-bw-thread,                         20.379, secs,           runtime-avg/thread
  5x5-bw-thread,                          1.309, %,              spread-runtime/thread
  5x5-bw-thread,                          7.881, GB,             data/thread
  5x5-bw-thread,                        197.032, GB,             data-total
  5x5-bw-thread,                          2.611, nsecs,          runtime/byte/thread
  5x5-bw-thread,                          0.383, GB/sec,         thread-speed
  5x5-bw-thread,                          9.575, GB/sec,         total-speed

 # Running 2x16-bw-thread, "perf bench numa mem -p 2 -t 16 -P 512 -s 20 -zZ0q --thp  1"
 2x16-bw-thread,                         21.581, secs,           runtime-max/thread
 2x16-bw-thread,                         20.043, secs,           runtime-min/thread
 2x16-bw-thread,                         20.958, secs,           runtime-avg/thread
 2x16-bw-thread,                          3.564, %,              spread-runtime/thread
 2x16-bw-thread,                          4.010, GB,             data/thread
 2x16-bw-thread,                        128.312, GB,             data-total
 2x16-bw-thread,                          5.382, nsecs,          runtime/byte/thread
 2x16-bw-thread,                          0.186, GB/sec,         thread-speed
 2x16-bw-thread,                          5.945, GB/sec,         total-speed

 # Running 1x32-bw-thread, "perf bench numa mem -p 1 -t 32 -P 2048 -s 20 -zZ0q --thp  1"
 1x32-bw-thread,                         23.503, secs,           runtime-max/thread
 1x32-bw-thread,                         21.850, secs,           runtime-min/thread
 1x32-bw-thread,                         22.953, secs,           runtime-avg/thread
 1x32-bw-thread,                          3.518, %,              spread-runtime/thread
 1x32-bw-thread,                          4.295, GB,             data/thread
 1x32-bw-thread,                        137.439, GB,             data-total
 1x32-bw-thread,                          5.472, nsecs,          runtime/byte/thread
 1x32-bw-thread,                          0.183, GB/sec,         thread-speed
 1x32-bw-thread,                          5.848, GB/sec,         total-speed

 # Running numa02-bw, "perf bench numa mem -p 1 -t 32 -T 32 -s 20 -zZ0q --thp  1"
 numa02-bw,                              19.948, secs,           runtime-max/thread
 numa02-bw,                              19.921, secs,           runtime-min/thread
 numa02-bw,                              19.983, secs,           runtime-avg/thread
 numa02-bw,                               0.068, %,              spread-runtime/thread
 numa02-bw,                              15.425, GB,             data/thread
 numa02-bw,                             493.586, GB,             data-total
 numa02-bw,                               1.293, nsecs,          runtime/byte/thread
 numa02-bw,                               0.773, GB/sec,         thread-speed
 numa02-bw,                              24.744, GB/sec,         total-speed

 # Running numa02-bw-NOTHP, "perf bench numa mem -p 1 -t 32 -T 32 -s 20 -zZ0q --thp  1 --thp -1"
 numa02-bw-NOTHP,                        20.055, secs,           runtime-max/thread
 numa02-bw-NOTHP,                        19.948, secs,           runtime-min/thread
 numa02-bw-NOTHP,                        19.991, secs,           runtime-avg/thread
 numa02-bw-NOTHP,                         0.267, %,              spread-runtime/thread
 numa02-bw-NOTHP,                        12.795, GB,             data/thread
 numa02-bw-NOTHP,                       409.431, GB,             data-total
 numa02-bw-NOTHP,                         1.567, nsecs,          runtime/byte/thread
 numa02-bw-NOTHP,                         0.638, GB/sec,         thread-speed
 numa02-bw-NOTHP,                        20.415, GB/sec,         total-speed

 # Running numa01-bw-thread, "perf bench numa mem -p 2 -t 16 -T 192 -s 20 -zZ0q --thp  1"
 numa01-bw-thread,                       20.107, secs,           runtime-max/thread
 numa01-bw-thread,                       19.978, secs,           runtime-min/thread
 numa01-bw-thread,                       20.067, secs,           runtime-avg/thread
 numa01-bw-thread,                        0.320, %,              spread-runtime/thread
 numa01-bw-thread,                        9.532, GB,             data/thread
 numa01-bw-thread,                      305.010, GB,             data-total
 numa01-bw-thread,                        2.110, nsecs,          runtime/byte/thread
 numa01-bw-thread,                        0.474, GB/sec,         thread-speed
 numa01-bw-thread,                       15.169, GB/sec,         total-speed

 # Running numa01-bw-thread-NOTHP, "perf bench numa mem -p 2 -t 16 -T 192 -s 20 -zZ0q --thp  1 --thp -1"
 numa01-bw-thread-NOTHP,                 20.319, secs,           runtime-max/thread
 numa01-bw-thread-NOTHP,                 19.978, secs,           runtime-min/thread
 numa01-bw-thread-NOTHP,                 20.076, secs,           runtime-avg/thread
 numa01-bw-thread-NOTHP,                  0.839, %,              spread-runtime/thread
 numa01-bw-thread-NOTHP,                  7.688, GB,             data/thread
 numa01-bw-thread-NOTHP,                246.021, GB,             data-total
 numa01-bw-thread-NOTHP,                  2.643, nsecs,          runtime/byte/thread
 numa01-bw-thread-NOTHP,                  0.378, GB/sec,         thread-speed
 numa01-bw-thread-NOTHP,                 12.108, GB/sec,         total-speed

 #
 # Running test on: Linux vega 3.7.0-rc8+ #2 SMP Fri Dec 7 02:46:02 CET 2012 x86_64 x86_64 x86_64 GNU/Linux
 #
# Running numa/mem benchmark...

 # Running main, "perf bench numa mem -a"

 # Running RAM-bw-local, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 0 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-local,                           20.132, secs,           runtime-max/thread
 RAM-bw-local,                           20.123, secs,           runtime-min/thread
 RAM-bw-local,                           20.123, secs,           runtime-avg/thread
 RAM-bw-local,                            0.024, %,              spread-runtime/thread
 RAM-bw-local,                          171.799, GB,             data/thread
 RAM-bw-local,                          171.799, GB,             data-total
 RAM-bw-local,                            0.117, nsecs,          runtime/byte/thread
 RAM-bw-local,                            8.534, GB/sec,         thread-speed
 RAM-bw-local,                            8.534, GB/sec,         total-speed

 # Running RAM-bw-local-NOTHP, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 0 -s 20 -zZq --thp  1 --no-data_rand_walk --thp -1"
 RAM-bw-local-NOTHP,                     20.133, secs,           runtime-max/thread
 RAM-bw-local-NOTHP,                     20.047, secs,           runtime-min/thread
 RAM-bw-local-NOTHP,                     20.047, secs,           runtime-avg/thread
 RAM-bw-local-NOTHP,                      0.214, %,              spread-runtime/thread
 RAM-bw-local-NOTHP,                    169.651, GB,             data/thread
 RAM-bw-local-NOTHP,                    169.651, GB,             data-total
 RAM-bw-local-NOTHP,                      0.119, nsecs,          runtime/byte/thread
 RAM-bw-local-NOTHP,                      8.427, GB/sec,         thread-speed
 RAM-bw-local-NOTHP,                      8.427, GB/sec,         total-speed

 # Running RAM-bw-remote, "perf bench numa mem -p 1 -t 1 -P 1024 -C 0 -M 1 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-remote,                          20.127, secs,           runtime-max/thread
 RAM-bw-remote,                          20.117, secs,           runtime-min/thread
 RAM-bw-remote,                          20.117, secs,           runtime-avg/thread
 RAM-bw-remote,                           0.025, %,              spread-runtime/thread
 RAM-bw-remote,                         134.218, GB,             data/thread
 RAM-bw-remote,                         134.218, GB,             data-total
 RAM-bw-remote,                           0.150, nsecs,          runtime/byte/thread
 RAM-bw-remote,                           6.669, GB/sec,         thread-speed
 RAM-bw-remote,                           6.669, GB/sec,         total-speed

 # Running RAM-bw-local-2x, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,2 -M 0x2 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-local-2x,                        20.139, secs,           runtime-max/thread
 RAM-bw-local-2x,                        20.011, secs,           runtime-min/thread
 RAM-bw-local-2x,                        20.070, secs,           runtime-avg/thread
 RAM-bw-local-2x,                         0.319, %,              spread-runtime/thread
 RAM-bw-local-2x,                       130.997, GB,             data/thread
 RAM-bw-local-2x,                       261.993, GB,             data-total
 RAM-bw-local-2x,                         0.154, nsecs,          runtime/byte/thread
 RAM-bw-local-2x,                         6.505, GB/sec,         thread-speed
 RAM-bw-local-2x,                        13.009, GB/sec,         total-speed

 # Running RAM-bw-remote-2x, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,2 -M 1x2 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-remote-2x,                       20.177, secs,           runtime-max/thread
 RAM-bw-remote-2x,                       20.083, secs,           runtime-min/thread
 RAM-bw-remote-2x,                       20.125, secs,           runtime-avg/thread
 RAM-bw-remote-2x,                        0.233, %,              spread-runtime/thread
 RAM-bw-remote-2x,                       74.088, GB,             data/thread
 RAM-bw-remote-2x,                      148.176, GB,             data-total
 RAM-bw-remote-2x,                        0.272, nsecs,          runtime/byte/thread
 RAM-bw-remote-2x,                        3.672, GB/sec,         thread-speed
 RAM-bw-remote-2x,                        7.344, GB/sec,         total-speed

 # Running RAM-bw-cross, "perf bench numa mem -p 2 -t 1 -P 1024 -C 0,8 -M 1,0 -s 20 -zZq --thp  1 --no-data_rand_walk"
 RAM-bw-cross,                           20.122, secs,           runtime-max/thread
 RAM-bw-cross,                           20.094, secs,           runtime-min/thread
 RAM-bw-cross,                           20.103, secs,           runtime-avg/thread
 RAM-bw-cross,                            0.070, %,              spread-runtime/thread
 RAM-bw-cross,                          121.870, GB,             data/thread
 RAM-bw-cross,                          243.739, GB,             data-total
 RAM-bw-cross,                            0.165, nsecs,          runtime/byte/thread
 RAM-bw-cross,                            6.057, GB/sec,         thread-speed
 RAM-bw-cross,                           12.113, GB/sec,         total-speed

 # Running  1x3-convergence, "perf bench numa mem -p 1 -t 3 -P 512 -s 100 -zZ0qcm --thp  1"
  1x3-convergence,                        2.333, secs,           NUMA-convergence-latency
  1x3-convergence,                        2.333, secs,           runtime-max/thread
  1x3-convergence,                        2.304, secs,           runtime-min/thread
  1x3-convergence,                        2.313, secs,           runtime-avg/thread
  1x3-convergence,                        0.620, %,              spread-runtime/thread
  1x3-convergence,                        7.516, GB,             data/thread
  1x3-convergence,                       22.549, GB,             data-total
  1x3-convergence,                        0.310, nsecs,          runtime/byte/thread
  1x3-convergence,                        3.222, GB/sec,         thread-speed
  1x3-convergence,                        9.665, GB/sec,         total-speed

 # Running  1x4-convergence, "perf bench numa mem -p 1 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  1x4-convergence,                        2.057, secs,           NUMA-convergence-latency
  1x4-convergence,                        2.057, secs,           runtime-max/thread
  1x4-convergence,                        1.958, secs,           runtime-min/thread
  1x4-convergence,                        1.998, secs,           runtime-avg/thread
  1x4-convergence,                        2.403, %,              spread-runtime/thread
  1x4-convergence,                        4.429, GB,             data/thread
  1x4-convergence,                       17.717, GB,             data-total
  1x4-convergence,                        0.464, nsecs,          runtime/byte/thread
  1x4-convergence,                        2.154, GB/sec,         thread-speed
  1x4-convergence,                        8.614, GB/sec,         total-speed

 # Running  1x6-convergence, "perf bench numa mem -p 1 -t 6 -P 1020 -s 100 -zZ0qcm --thp  1"
  1x6-convergence,                        7.327, secs,           NUMA-convergence-latency
  1x6-convergence,                        7.327, secs,           runtime-max/thread
  1x6-convergence,                        6.879, secs,           runtime-min/thread
  1x6-convergence,                        7.187, secs,           runtime-avg/thread
  1x6-convergence,                        3.063, %,              spread-runtime/thread
  1x6-convergence,                       11.052, GB,             data/thread
  1x6-convergence,                       66.312, GB,             data-total
  1x6-convergence,                        0.663, nsecs,          runtime/byte/thread
  1x6-convergence,                        1.508, GB/sec,         thread-speed
  1x6-convergence,                        9.050, GB/sec,         total-speed

 # Running  2x3-convergence, "perf bench numa mem -p 3 -t 3 -P 1020 -s 100 -zZ0qcm --thp  1"
  2x3-convergence,                        4.086, secs,           NUMA-convergence-latency
  2x3-convergence,                        4.086, secs,           runtime-max/thread
  2x3-convergence,                        3.779, secs,           runtime-min/thread
  2x3-convergence,                        3.960, secs,           runtime-avg/thread
  2x3-convergence,                        3.761, %,              spread-runtime/thread
  2x3-convergence,                        6.774, GB,             data/thread
  2x3-convergence,                       60.964, GB,             data-total
  2x3-convergence,                        0.603, nsecs,          runtime/byte/thread
  2x3-convergence,                        1.658, GB/sec,         thread-speed
  2x3-convergence,                       14.920, GB/sec,         total-speed

 # Running  3x3-convergence, "perf bench numa mem -p 3 -t 3 -P 1020 -s 100 -zZ0qcm --thp  1"
  3x3-convergence,                        7.627, secs,           NUMA-convergence-latency
  3x3-convergence,                        7.627, secs,           runtime-max/thread
  3x3-convergence,                        7.380, secs,           runtime-min/thread
  3x3-convergence,                        7.504, secs,           runtime-avg/thread
  3x3-convergence,                        1.624, %,              spread-runtime/thread
  3x3-convergence,                       15.093, GB,             data/thread
  3x3-convergence,                      135.833, GB,             data-total
  3x3-convergence,                        0.505, nsecs,          runtime/byte/thread
  3x3-convergence,                        1.979, GB/sec,         thread-speed
  3x3-convergence,                       17.809, GB/sec,         total-speed

 # Running  4x4-convergence, "perf bench numa mem -p 4 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  4x4-convergence,                        7.381, secs,           NUMA-convergence-latency
  4x4-convergence,                        7.381, secs,           runtime-max/thread
  4x4-convergence,                        7.149, secs,           runtime-min/thread
  4x4-convergence,                        7.277, secs,           runtime-avg/thread
  4x4-convergence,                        1.569, %,              spread-runtime/thread
  4x4-convergence,                        7.181, GB,             data/thread
  4x4-convergence,                      114.890, GB,             data-total
  4x4-convergence,                        1.028, nsecs,          runtime/byte/thread
  4x4-convergence,                        0.973, GB/sec,         thread-speed
  4x4-convergence,                       15.566, GB/sec,         total-speed

 # Running  4x4-convergence-NOTHP, "perf bench numa mem -p 4 -t 4 -P 512 -s 100 -zZ0qcm --thp  1 --thp -1"
  4x4-convergence-NOTHP,                  9.200, secs,           NUMA-convergence-latency
  4x4-convergence-NOTHP,                  9.200, secs,           runtime-max/thread
  4x4-convergence-NOTHP,                  8.944, secs,           runtime-min/thread
  4x4-convergence-NOTHP,                  9.047, secs,           runtime-avg/thread
  4x4-convergence-NOTHP,                  1.391, %,              spread-runtime/thread
  4x4-convergence-NOTHP,                 11.778, GB,             data/thread
  4x4-convergence-NOTHP,                188.442, GB,             data-total
  4x4-convergence-NOTHP,                  0.781, nsecs,          runtime/byte/thread
  4x4-convergence-NOTHP,                  1.280, GB/sec,         thread-speed
  4x4-convergence-NOTHP,                 20.483, GB/sec,         total-speed

 # Running  4x6-convergence, "perf bench numa mem -p 4 -t 6 -P 1020 -s 100 -zZ0qcm --thp  1"
  4x6-convergence,                       11.664, secs,           NUMA-convergence-latency
  4x6-convergence,                       11.664, secs,           runtime-max/thread
  4x6-convergence,                       11.155, secs,           runtime-min/thread
  4x6-convergence,                       11.420, secs,           runtime-avg/thread
  4x6-convergence,                        2.180, %,              spread-runtime/thread
  4x6-convergence,                       11.319, GB,             data/thread
  4x6-convergence,                      271.665, GB,             data-total
  4x6-convergence,                        1.030, nsecs,          runtime/byte/thread
  4x6-convergence,                        0.970, GB/sec,         thread-speed
  4x6-convergence,                       23.292, GB/sec,         total-speed

 # Running  4x8-convergence, "perf bench numa mem -p 4 -t 8 -P 512 -s 100 -zZ0qcm --thp  1"
  4x8-convergence,                        3.880, secs,           NUMA-convergence-latency
  4x8-convergence,                        3.880, secs,           runtime-max/thread
  4x8-convergence,                        3.613, secs,           runtime-min/thread
  4x8-convergence,                        3.784, secs,           runtime-avg/thread
  4x8-convergence,                        3.440, %,              spread-runtime/thread
  4x8-convergence,                        2.047, GB,             data/thread
  4x8-convergence,                       65.498, GB,             data-total
  4x8-convergence,                        1.896, nsecs,          runtime/byte/thread
  4x8-convergence,                        0.528, GB/sec,         thread-speed
  4x8-convergence,                       16.882, GB/sec,         total-speed

 # Running  8x4-convergence, "perf bench numa mem -p 8 -t 4 -P 512 -s 100 -zZ0qcm --thp  1"
  8x4-convergence,                        8.938, secs,           NUMA-convergence-latency
  8x4-convergence,                        8.938, secs,           runtime-max/thread
  8x4-convergence,                        8.556, secs,           runtime-min/thread
  8x4-convergence,                        8.744, secs,           runtime-avg/thread
  8x4-convergence,                        2.135, %,              spread-runtime/thread
  8x4-convergence,                        4.396, GB,             data/thread
  8x4-convergence,                      140.660, GB,             data-total
  8x4-convergence,                        2.033, nsecs,          runtime/byte/thread
  8x4-convergence,                        0.492, GB/sec,         thread-speed
  8x4-convergence,                       15.738, GB/sec,         total-speed

 # Running  8x4-convergence-NOTHP, "perf bench numa mem -p 8 -t 4 -P 512 -s 100 -zZ0qcm --thp  1 --thp -1"
  8x4-convergence-NOTHP,                 12.123, secs,           NUMA-convergence-latency
  8x4-convergence-NOTHP,                 12.123, secs,           runtime-max/thread
  8x4-convergence-NOTHP,                 11.749, secs,           runtime-min/thread
  8x4-convergence-NOTHP,                 11.936, secs,           runtime-avg/thread
  8x4-convergence-NOTHP,                  1.542, %,              spread-runtime/thread
  8x4-convergence-NOTHP,                  4.480, GB,             data/thread
  8x4-convergence-NOTHP,                143.345, GB,             data-total
  8x4-convergence-NOTHP,                  2.706, nsecs,          runtime/byte/thread
  8x4-convergence-NOTHP,                  0.370, GB/sec,         thread-speed
  8x4-convergence-NOTHP,                 11.824, GB/sec,         total-speed

 # Running  3x1-convergence, "perf bench numa mem -p 3 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  3x1-convergence,                        0.879, secs,           NUMA-convergence-latency
  3x1-convergence,                        0.879, secs,           runtime-max/thread
  3x1-convergence,                        0.810, secs,           runtime-min/thread
  3x1-convergence,                        0.839, secs,           runtime-avg/thread
  3x1-convergence,                        3.911, %,              spread-runtime/thread
  3x1-convergence,                        2.326, GB,             data/thread
  3x1-convergence,                        6.979, GB,             data-total
  3x1-convergence,                        0.378, nsecs,          runtime/byte/thread
  3x1-convergence,                        2.647, GB/sec,         thread-speed
  3x1-convergence,                        7.941, GB/sec,         total-speed

 # Running  4x1-convergence, "perf bench numa mem -p 4 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  4x1-convergence,                        0.685, secs,           NUMA-convergence-latency
  4x1-convergence,                        0.685, secs,           runtime-max/thread
  4x1-convergence,                        0.617, secs,           runtime-min/thread
  4x1-convergence,                        0.650, secs,           runtime-avg/thread
  4x1-convergence,                        4.967, %,              spread-runtime/thread
  4x1-convergence,                        1.476, GB,             data/thread
  4x1-convergence,                        5.906, GB,             data-total
  4x1-convergence,                        0.464, nsecs,          runtime/byte/thread
  4x1-convergence,                        2.154, GB/sec,         thread-speed
  4x1-convergence,                        8.616, GB/sec,         total-speed

 # Running  8x1-convergence, "perf bench numa mem -p 8 -t 1 -P 512 -s 100 -zZ0qcm --thp  1"
  8x1-convergence,                        1.158, secs,           NUMA-convergence-latency
  8x1-convergence,                        1.158, secs,           runtime-max/thread
  8x1-convergence,                        1.010, secs,           runtime-min/thread
  8x1-convergence,                        1.060, secs,           runtime-avg/thread
  8x1-convergence,                        6.396, %,              spread-runtime/thread
  8x1-convergence,                        1.745, GB,             data/thread
  8x1-convergence,                       13.959, GB,             data-total
  8x1-convergence,                        0.664, nsecs,          runtime/byte/thread
  8x1-convergence,                        1.507, GB/sec,         thread-speed
  8x1-convergence,                       12.054, GB/sec,         total-speed

 # Running 16x1-convergence, "perf bench numa mem -p 16 -t 1 -P 256 -s 100 -zZ0qcm --thp  1"
 16x1-convergence,                        2.010, secs,           NUMA-convergence-latency
 16x1-convergence,                        2.010, secs,           runtime-max/thread
 16x1-convergence,                        1.939, secs,           runtime-min/thread
 16x1-convergence,                        1.991, secs,           runtime-avg/thread
 16x1-convergence,                        1.760, %,              spread-runtime/thread
 16x1-convergence,                        2.668, GB,             data/thread
 16x1-convergence,                       42.681, GB,             data-total
 16x1-convergence,                        0.753, nsecs,          runtime/byte/thread
 16x1-convergence,                        1.327, GB/sec,         thread-speed
 16x1-convergence,                       21.237, GB/sec,         total-speed

 # Running 32x1-convergence, "perf bench numa mem -p 32 -t 1 -P 128 -s 100 -zZ0qcm --thp  1"
 32x1-convergence,                        1.946, secs,           NUMA-convergence-latency
 32x1-convergence,                        1.946, secs,           runtime-max/thread
 32x1-convergence,                        1.850, secs,           runtime-min/thread
 32x1-convergence,                        1.946, secs,           runtime-avg/thread
 32x1-convergence,                        2.479, %,              spread-runtime/thread
 32x1-convergence,                        1.242, GB,             data/thread
 32x1-convergence,                       39.728, GB,             data-total
 32x1-convergence,                        1.568, nsecs,          runtime/byte/thread
 32x1-convergence,                        0.638, GB/sec,         thread-speed
 32x1-convergence,                       20.410, GB/sec,         total-speed

 # Running  2x1-bw-process, "perf bench numa mem -p 2 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  2x1-bw-process,                        20.146, secs,           runtime-max/thread
  2x1-bw-process,                        20.068, secs,           runtime-min/thread
  2x1-bw-process,                        20.102, secs,           runtime-avg/thread
  2x1-bw-process,                         0.193, %,              spread-runtime/thread
  2x1-bw-process,                        97.174, GB,             data/thread
  2x1-bw-process,                       194.347, GB,             data-total
  2x1-bw-process,                         0.207, nsecs,          runtime/byte/thread
  2x1-bw-process,                         4.824, GB/sec,         thread-speed
  2x1-bw-process,                         9.647, GB/sec,         total-speed

 # Running  3x1-bw-process, "perf bench numa mem -p 3 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  3x1-bw-process,                        20.177, secs,           runtime-max/thread
  3x1-bw-process,                        20.127, secs,           runtime-min/thread
  3x1-bw-process,                        20.146, secs,           runtime-avg/thread
  3x1-bw-process,                         0.126, %,              spread-runtime/thread
  3x1-bw-process,                        97.711, GB,             data/thread
  3x1-bw-process,                       293.132, GB,             data-total
  3x1-bw-process,                         0.207, nsecs,          runtime/byte/thread
  3x1-bw-process,                         4.843, GB/sec,         thread-speed
  3x1-bw-process,                        14.528, GB/sec,         total-speed

 # Running  4x1-bw-process, "perf bench numa mem -p 4 -t 1 -P 1024 -s 20 -zZ0q --thp  1"
  4x1-bw-process,                        20.165, secs,           runtime-max/thread
  4x1-bw-process,                        20.025, secs,           runtime-min/thread
  4x1-bw-process,                        20.078, secs,           runtime-avg/thread
  4x1-bw-process,                         0.348, %,              spread-runtime/thread
  4x1-bw-process,                        95.295, GB,             data/thread
  4x1-bw-process,                       381.178, GB,             data-total
  4x1-bw-process,                         0.212, nsecs,          runtime/byte/thread
  4x1-bw-process,                         4.726, GB/sec,         thread-speed
  4x1-bw-process,                        18.903, GB/sec,         total-speed

 # Running  8x1-bw-process, "perf bench numa mem -p 8 -t 1 -P  512 -s 20 -zZ0q --thp  1"
  8x1-bw-process,                        20.131, secs,           runtime-max/thread
  8x1-bw-process,                        20.066, secs,           runtime-min/thread
  8x1-bw-process,                        20.090, secs,           runtime-avg/thread
  8x1-bw-process,                         0.161, %,              spread-runtime/thread
  8x1-bw-process,                        67.512, GB,             data/thread
  8x1-bw-process,                       540.092, GB,             data-total
  8x1-bw-process,                         0.298, nsecs,          runtime/byte/thread
  8x1-bw-process,                         3.354, GB/sec,         thread-speed
  8x1-bw-process,                        26.829, GB/sec,         total-speed

 # Running  8x1-bw-process-NOTHP, "perf bench numa mem -p 8 -t 1 -P  512 -s 20 -zZ0q --thp  1 --thp -1"
  8x1-bw-process-NOTHP,                  20.208, secs,           runtime-max/thread
  8x1-bw-process-NOTHP,                  20.002, secs,           runtime-min/thread
  8x1-bw-process-NOTHP,                  20.067, secs,           runtime-avg/thread
  8x1-bw-process-NOTHP,                   0.509, %,              spread-runtime/thread
  8x1-bw-process-NOTHP,                  56.170, GB,             data/thread
  8x1-bw-process-NOTHP,                 449.361, GB,             data-total
  8x1-bw-process-NOTHP,                   0.360, nsecs,          runtime/byte/thread
  8x1-bw-process-NOTHP,                   2.780, GB/sec,         thread-speed
  8x1-bw-process-NOTHP,                  22.237, GB/sec,         total-speed

 # Running 16x1-bw-process, "perf bench numa mem -p 16 -t 1 -P 256 -s 20 -zZ0q --thp  1"
 16x1-bw-process,                        20.068, secs,           runtime-max/thread
 16x1-bw-process,                        20.014, secs,           runtime-min/thread
 16x1-bw-process,                        20.042, secs,           runtime-avg/thread
 16x1-bw-process,                         0.136, %,              spread-runtime/thread
 16x1-bw-process,                        36.742, GB,             data/thread
 16x1-bw-process,                       587.874, GB,             data-total
 16x1-bw-process,                         0.546, nsecs,          runtime/byte/thread
 16x1-bw-process,                         1.831, GB/sec,         thread-speed
 16x1-bw-process,                        29.294, GB/sec,         total-speed

 # Running  4x1-bw-thread, "perf bench numa mem -p 1 -t 4 -T 256 -s 20 -zZ0q --thp  1"
  4x1-bw-thread,                         20.053, secs,           runtime-max/thread
  4x1-bw-thread,                         20.003, secs,           runtime-min/thread
  4x1-bw-thread,                         20.025, secs,           runtime-avg/thread
  4x1-bw-thread,                          0.123, %,              spread-runtime/thread
  4x1-bw-thread,                         96.704, GB,             data/thread
  4x1-bw-thread,                        386.815, GB,             data-total
  4x1-bw-thread,                          0.207, nsecs,          runtime/byte/thread
  4x1-bw-thread,                          4.822, GB/sec,         thread-speed
  4x1-bw-thread,                         19.290, GB/sec,         total-speed

 # Running  8x1-bw-thread, "perf bench numa mem -p 1 -t 8 -T 256 -s 20 -zZ0q --thp  1"
  8x1-bw-thread,                         20.068, secs,           runtime-max/thread
  8x1-bw-thread,                         20.004, secs,           runtime-min/thread
  8x1-bw-thread,                         20.031, secs,           runtime-avg/thread
  8x1-bw-thread,                          0.160, %,              spread-runtime/thread
  8x1-bw-thread,                         66.203, GB,             data/thread
  8x1-bw-thread,                        529.623, GB,             data-total
  8x1-bw-thread,                          0.303, nsecs,          runtime/byte/thread
  8x1-bw-thread,                          3.299, GB/sec,         thread-speed
  8x1-bw-thread,                         26.391, GB/sec,         total-speed

 # Running 16x1-bw-thread, "perf bench numa mem -p 1 -t 16 -T 128 -s 20 -zZ0q --thp  1"
 16x1-bw-thread,                         20.044, secs,           runtime-max/thread
 16x1-bw-thread,                         20.007, secs,           runtime-min/thread
 16x1-bw-thread,                         20.029, secs,           runtime-avg/thread
 16x1-bw-thread,                          0.092, %,              spread-runtime/thread
 16x1-bw-thread,                         37.027, GB,             data/thread
 16x1-bw-thread,                        592.437, GB,             data-total
 16x1-bw-thread,                          0.541, nsecs,          runtime/byte/thread
 16x1-bw-thread,                          1.847, GB/sec,         thread-speed
 16x1-bw-thread,                         29.557, GB/sec,         total-speed

 # Running 32x1-bw-thread, "perf bench numa mem -p 1 -t 32 -T 64 -s 20 -zZ0q --thp  1"
 32x1-bw-thread,                         20.029, secs,           runtime-max/thread
 32x1-bw-thread,                         19.975, secs,           runtime-min/thread
 32x1-bw-thread,                         20.015, secs,           runtime-avg/thread
 32x1-bw-thread,                          0.134, %,              spread-runtime/thread
 32x1-bw-thread,                         18.923, GB,             data/thread
 32x1-bw-thread,                        605.523, GB,             data-total
 32x1-bw-thread,                          1.058, nsecs,          runtime/byte/thread
 32x1-bw-thread,                          0.945, GB/sec,         thread-speed
 32x1-bw-thread,                         30.232, GB/sec,         total-speed

 # Running  2x3-bw-thread, "perf bench numa mem -p 2 -t 3 -P 512 -s 20 -zZ0q --thp  1"
  2x3-bw-thread,                         20.176, secs,           runtime-max/thread
  2x3-bw-thread,                         20.072, secs,           runtime-min/thread
  2x3-bw-thread,                         20.136, secs,           runtime-avg/thread
  2x3-bw-thread,                          0.257, %,              spread-runtime/thread
  2x3-bw-thread,                         51.540, GB,             data/thread
  2x3-bw-thread,                        309.238, GB,             data-total
  2x3-bw-thread,                          0.391, nsecs,          runtime/byte/thread
  2x3-bw-thread,                          2.555, GB/sec,         thread-speed
  2x3-bw-thread,                         15.327, GB/sec,         total-speed

 # Running  4x4-bw-thread, "perf bench numa mem -p 4 -t 4 -P 512 -s 20 -zZ0q --thp  1"
  4x4-bw-thread,                         20.183, secs,           runtime-max/thread
  4x4-bw-thread,                         20.013, secs,           runtime-min/thread
  4x4-bw-thread,                         20.086, secs,           runtime-avg/thread
  4x4-bw-thread,                          0.421, %,              spread-runtime/thread
  4x4-bw-thread,                         35.266, GB,             data/thread
  4x4-bw-thread,                        564.251, GB,             data-total
  4x4-bw-thread,                          0.572, nsecs,          runtime/byte/thread
  4x4-bw-thread,                          1.747, GB/sec,         thread-speed
  4x4-bw-thread,                         27.957, GB/sec,         total-speed

 # Running  4x6-bw-thread, "perf bench numa mem -p 4 -t 6 -P 512 -s 20 -zZ0q --thp  1"
  4x6-bw-thread,                         20.298, secs,           runtime-max/thread
  4x6-bw-thread,                         20.061, secs,           runtime-min/thread
  4x6-bw-thread,                         20.184, secs,           runtime-avg/thread
  4x6-bw-thread,                          0.584, %,              spread-runtime/thread
  4x6-bw-thread,                         23.578, GB,             data/thread
  4x6-bw-thread,                        565.862, GB,             data-total
  4x6-bw-thread,                          0.861, nsecs,          runtime/byte/thread
  4x6-bw-thread,                          1.162, GB/sec,         thread-speed
  4x6-bw-thread,                         27.877, GB/sec,         total-speed

 # Running  4x8-bw-thread, "perf bench numa mem -p 4 -t 8 -P 512 -s 20 -zZ0q --thp  1"
  4x8-bw-thread,                         20.350, secs,           runtime-max/thread
  4x8-bw-thread,                         20.004, secs,           runtime-min/thread
  4x8-bw-thread,                         20.190, secs,           runtime-avg/thread
  4x8-bw-thread,                          0.851, %,              spread-runtime/thread
  4x8-bw-thread,                         18.086, GB,             data/thread
  4x8-bw-thread,                        578.747, GB,             data-total
  4x8-bw-thread,                          1.125, nsecs,          runtime/byte/thread
  4x8-bw-thread,                          0.889, GB/sec,         thread-speed
  4x8-bw-thread,                         28.439, GB/sec,         total-speed

 # Running  4x8-bw-thread-NOTHP, "perf bench numa mem -p 4 -t 8 -P 512 -s 20 -zZ0q --thp  1 --thp -1"
  4x8-bw-thread-NOTHP,                   20.411, secs,           runtime-max/thread
  4x8-bw-thread-NOTHP,                   19.990, secs,           runtime-min/thread
  4x8-bw-thread-NOTHP,                   20.246, secs,           runtime-avg/thread
  4x8-bw-thread-NOTHP,                    1.032, %,              spread-runtime/thread
  4x8-bw-thread-NOTHP,                   15.989, GB,             data/thread
  4x8-bw-thread-NOTHP,                  511.638, GB,             data-total
  4x8-bw-thread-NOTHP,                    1.277, nsecs,          runtime/byte/thread
  4x8-bw-thread-NOTHP,                    0.783, GB/sec,         thread-speed
  4x8-bw-thread-NOTHP,                   25.067, GB/sec,         total-speed

 # Running  3x3-bw-thread, "perf bench numa mem -p 3 -t 3 -P 512 -s 20 -zZ0q --thp  1"
  3x3-bw-thread,                         20.170, secs,           runtime-max/thread
  3x3-bw-thread,                         20.050, secs,           runtime-min/thread
  3x3-bw-thread,                         20.109, secs,           runtime-avg/thread
  3x3-bw-thread,                          0.299, %,              spread-runtime/thread
  3x3-bw-thread,                         48.318, GB,             data/thread
  3x3-bw-thread,                        434.865, GB,             data-total
  3x3-bw-thread,                          0.417, nsecs,          runtime/byte/thread
  3x3-bw-thread,                          2.396, GB/sec,         thread-speed
  3x3-bw-thread,                         21.560, GB/sec,         total-speed

 # Running  5x5-bw-thread, "perf bench numa mem -p 5 -t 5 -P 512 -s 20 -zZ0q --thp  1"
  5x5-bw-thread,                         20.276, secs,           runtime-max/thread
  5x5-bw-thread,                         20.004, secs,           runtime-min/thread
  5x5-bw-thread,                         20.155, secs,           runtime-avg/thread
  5x5-bw-thread,                          0.671, %,              spread-runtime/thread
  5x5-bw-thread,                         21.153, GB,             data/thread
  5x5-bw-thread,                        528.818, GB,             data-total
  5x5-bw-thread,                          0.959, nsecs,          runtime/byte/thread
  5x5-bw-thread,                          1.043, GB/sec,         thread-speed
  5x5-bw-thread,                         26.081, GB/sec,         total-speed

 # Running 2x16-bw-thread, "perf bench numa mem -p 2 -t 16 -P 512 -s 20 -zZ0q --thp  1"
 2x16-bw-thread,                         20.465, secs,           runtime-max/thread
 2x16-bw-thread,                         20.004, secs,           runtime-min/thread
 2x16-bw-thread,                         20.284, secs,           runtime-avg/thread
 2x16-bw-thread,                          1.127, %,              spread-runtime/thread
 2x16-bw-thread,                         14.881, GB,             data/thread
 2x16-bw-thread,                        476.204, GB,             data-total
 2x16-bw-thread,                          1.375, nsecs,          runtime/byte/thread
 2x16-bw-thread,                          0.727, GB/sec,         thread-speed
 2x16-bw-thread,                         23.269, GB/sec,         total-speed

 # Running 1x32-bw-thread, "perf bench numa mem -p 1 -t 32 -P 2048 -s 20 -zZ0q --thp  1"
 1x32-bw-thread,                         21.944, secs,           runtime-max/thread
 1x32-bw-thread,                         20.031, secs,           runtime-min/thread
 1x32-bw-thread,                         20.878, secs,           runtime-avg/thread
 1x32-bw-thread,                          4.358, %,              spread-runtime/thread
 1x32-bw-thread,                         13.019, GB,             data/thread
 1x32-bw-thread,                        416.612, GB,             data-total
 1x32-bw-thread,                          1.686, nsecs,          runtime/byte/thread
 1x32-bw-thread,                          0.593, GB/sec,         thread-speed
 1x32-bw-thread,                         18.985, GB/sec,         total-speed

 # Running numa02-bw, "perf bench numa mem -p 1 -t 32 -T 32 -s 20 -zZ0q --thp  1"
 numa02-bw,                              20.000, secs,           runtime-max/thread
 numa02-bw,                              19.967, secs,           runtime-min/thread
 numa02-bw,                              19.994, secs,           runtime-avg/thread
 numa02-bw,                               0.081, %,              spread-runtime/thread
 numa02-bw,                              19.644, GB,             data/thread
 numa02-bw,                             628.609, GB,             data-total
 numa02-bw,                               1.018, nsecs,          runtime/byte/thread
 numa02-bw,                               0.982, GB/sec,         thread-speed
 numa02-bw,                              31.431, GB/sec,         total-speed

 # Running numa02-bw-NOTHP, "perf bench numa mem -p 1 -t 32 -T 32 -s 20 -zZ0q --thp  1 --thp -1"
 numa02-bw-NOTHP,                        20.062, secs,           runtime-max/thread
 numa02-bw-NOTHP,                        19.940, secs,           runtime-min/thread
 numa02-bw-NOTHP,                        19.988, secs,           runtime-avg/thread
 numa02-bw-NOTHP,                         0.304, %,              spread-runtime/thread
 numa02-bw-NOTHP,                        18.246, GB,             data/thread
 numa02-bw-NOTHP,                       583.881, GB,             data-total
 numa02-bw-NOTHP,                         1.100, nsecs,          runtime/byte/thread
 numa02-bw-NOTHP,                         0.909, GB/sec,         thread-speed
 numa02-bw-NOTHP,                        29.104, GB/sec,         total-speed

 # Running numa01-bw-thread, "perf bench numa mem -p 2 -t 16 -T 192 -s 20 -zZ0q --thp  1"
 numa01-bw-thread,                       20.106, secs,           runtime-max/thread
 numa01-bw-thread,                       19.989, secs,           runtime-min/thread
 numa01-bw-thread,                       20.052, secs,           runtime-avg/thread
 numa01-bw-thread,                        0.293, %,              spread-runtime/thread
 numa01-bw-thread,                       17.975, GB,             data/thread
 numa01-bw-thread,                      575.190, GB,             data-total
 numa01-bw-thread,                        1.119, nsecs,          runtime/byte/thread
 numa01-bw-thread,                        0.894, GB/sec,         thread-speed
 numa01-bw-thread,                       28.607, GB/sec,         total-speed

 # Running numa01-bw-thread-NOTHP, "perf bench numa mem -p 2 -t 16 -T 192 -s 20 -zZ0q --thp  1 --thp -1"
 numa01-bw-thread-NOTHP,                 20.391, secs,           runtime-max/thread
 numa01-bw-thread-NOTHP,                 20.010, secs,           runtime-min/thread
 numa01-bw-thread-NOTHP,                 20.085, secs,           runtime-avg/thread
 numa01-bw-thread-NOTHP,                  0.936, %,              spread-runtime/thread
 numa01-bw-thread-NOTHP,                 13.457, GB,             data/thread
 numa01-bw-thread-NOTHP,                430.638, GB,             data-total
 numa01-bw-thread-NOTHP,                  1.515, nsecs,          runtime/byte/thread
 numa01-bw-thread-NOTHP,                  0.660, GB/sec,         thread-speed
 numa01-bw-thread-NOTHP,                 21.119, GB/sec,         total-speed

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ