lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20210222082439.GB15880@xsang-OptiPlex-9020>
Date:   Mon, 22 Feb 2021 16:24:39 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Peter Puhov <peter.puhov@...aro.org>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com, aubrey.li@...ux.intel.com,
        yu.c.chen@...el.com
Subject: [sched/fair]  3edecfef02:  vm-scalability.throughput -9.8% regression


Greeting,

FYI, we noticed a -9.8% regression of vm-scalability.throughput due to commit:


commit: 3edecfef028536cb19a120ec8788bd8a11f93b9e ("sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: vm-scalability
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	runtime: 300s
	size: 8T
	test: anon-w-seq-hugetlb
	cpufreq_governor: performance
	ucode: 0x5003003

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------------+
| testcase: change | vm-scalability: vm-scalability.throughput -11.0% regression                     |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory            |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | runtime=300s                                                                    |
|                  | size=512G                                                                       |
|                  | test=anon-w-rand-hugetlb                                                        |
|                  | ucode=0x5003003                                                                 |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.build-apache.0.seconds 1.8% regression |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory            |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | test=build-apache-1.4.0                                                         |
|                  | ucode=0x5002f01                                                                 |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | vm-scalability: vm-scalability.throughput 7.3% improvement                      |
| test machine     | 104 threads Skylake with 192G memory                                            |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | runtime=300s                                                                    |
|                  | size=8T                                                                         |
|                  | test=anon-w-seq-mt                                                              |
|                  | ucode=0x2006906                                                                 |
+------------------+---------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/300s/8T/lkp-csl-2sp6/anon-w-seq-hugetlb/vm-scalability/0x5003003

commit: 
  46132e3ac5 ("sched: nohz: stop passing around unused "ticks" parameter.")
  3edecfef02 ("sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal")

46132e3ac58cb2ee 3edecfef028536cb19a120ec878 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:6            2%           0:6     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    458990            -9.7%     414275        vm-scalability.median
  44358717            -9.8%   40017738        vm-scalability.throughput
   5525037 ±  2%      -9.9%    4977835        vm-scalability.time.minor_page_faults
      8546            -0.8%       8478        vm-scalability.time.percent_of_cpu_this_job_got
     24407 ±  2%      -7.5%      22583        vm-scalability.time.voluntary_context_switches
 1.167e+10 ±  2%     -10.0%  1.051e+10        vm-scalability.workload
   1169502 ± 52%     +45.5%    1701161 ± 25%  cpuidle.C6.usage
  28118772 ±  6%      -9.9%   25339276 ±  6%  meminfo.DirectMap2M
      6.60 ±  3%      +1.0        7.62        mpstat.cpu.all.idle%
   3229132 ±  3%     -11.1%    2871949 ±  7%  numa-numastat.node1.numa_hit
    232.11 ± 21%     +69.2%     392.79 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    136950 ± 11%     +28.7%     176223 ±  7%  sched_debug.cpu.avg_idle.stddev
     35785 ± 17%     +49.4%      53474 ± 20%  sched_debug.cpu.max_idle_balance_cost.stddev
      2674 ±  7%     +23.0%       3288 ±  5%  slabinfo.PING.active_objs
      2674 ±  7%     +23.0%       3288 ±  5%  slabinfo.PING.num_objs
      1181 ± 15%     +25.4%       1481 ± 11%  slabinfo.khugepaged_mm_slot.active_objs
      1181 ± 15%     +25.4%       1481 ± 11%  slabinfo.khugepaged_mm_slot.num_objs
      1191 ± 10%     +20.7%       1438 ±  6%  slabinfo.pool_workqueue.active_objs
      4.23 ± 42%     -72.5%       1.17 ± 35%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.preempt_schedule_common._cond_resched.hugetlb_acct_memory
     16.19 ± 30%     -55.7%       7.18 ± 77%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.preempt_schedule_common._cond_resched.hugetlb_acct_memory
      1459 ±  5%     -12.1%       1283 ±  9%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      0.94 ± 78%    +164.2%       2.47 ± 14%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.io_schedule.__lock_page_killable.filemap_fault
      0.14 ±174%   +1655.4%       2.54 ± 91%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.preempt_schedule_common._cond_resched.kmem_cache_alloc
      0.57 ±165%   +2060.2%      12.23 ± 98%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.preempt_schedule_common._cond_resched.kmem_cache_alloc
   5067392 ±  2%     -10.0%    4562800        proc-vmstat.htlb_buddy_alloc_success
     26718            +1.7%      27175        proc-vmstat.nr_slab_reclaimable
   6370475 ±  2%      -8.6%    5823183        proc-vmstat.numa_hit
   6283794 ±  2%      -8.7%    5736041        proc-vmstat.numa_local
 2.596e+09 ±  2%     -10.0%  2.337e+09        proc-vmstat.pgalloc_normal
   6485971 ±  2%      -8.5%    5933932        proc-vmstat.pgfault
 2.595e+09 ±  2%     -10.0%  2.336e+09        proc-vmstat.pgfree
    901.00 ± 17%     -42.7%     516.67 ± 41%  interrupts.CPU14.RES:Rescheduling_interrupts
      6646 ±  3%     -34.4%       4361 ± 36%  interrupts.CPU29.NMI:Non-maskable_interrupts
      6646 ±  3%     -34.4%       4361 ± 36%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      6700 ±  4%     -36.2%       4272 ± 35%  interrupts.CPU31.NMI:Non-maskable_interrupts
      6700 ±  4%     -36.2%       4272 ± 35%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      6682 ±  3%     -21.1%       5270 ± 27%  interrupts.CPU36.NMI:Non-maskable_interrupts
      6682 ±  3%     -21.1%       5270 ± 27%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      6711 ±  3%     -27.5%       4865 ± 33%  interrupts.CPU38.NMI:Non-maskable_interrupts
      6711 ±  3%     -27.5%       4865 ± 33%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    600.50 ±  7%     +15.7%     694.50 ± 13%  interrupts.CPU56.CAL:Function_call_interrupts
    582.50 ±  7%     +80.9%       1054 ± 66%  interrupts.CPU69.CAL:Function_call_interrupts
      1242 ± 48%     -58.3%     518.67 ± 20%  interrupts.CPU7.RES:Rescheduling_interrupts
    148.33 ± 77%    +193.6%     435.50 ± 71%  interrupts.CPU79.RES:Rescheduling_interrupts
    171.33 ± 82%    +145.0%     419.83 ± 63%  interrupts.CPU83.RES:Rescheduling_interrupts
    162.67 ±100%    +151.6%     409.33 ± 48%  interrupts.CPU87.RES:Rescheduling_interrupts
 4.136e+10 ±  2%      -9.7%  3.736e+10        perf-stat.i.branch-instructions
      0.03 ±  2%      +0.0        0.03 ±  2%  perf-stat.i.branch-miss-rate%
   6151456            -6.0%    5779738 ±  4%  perf-stat.i.branch-misses
 4.562e+08 ±  2%      -9.9%  4.113e+08        perf-stat.i.cache-misses
 6.451e+08 ±  2%      -9.9%  5.811e+08        perf-stat.i.cache-references
      2.01 ±  4%     +15.2%       2.32        perf-stat.i.cpi
    232.45           -11.2%     206.42 ±  2%  perf-stat.i.cpu-migrations
    572.36 ±  4%     +14.8%     657.13        perf-stat.i.cycles-between-cache-misses
  3.25e+10 ±  2%      -9.8%  2.933e+10        perf-stat.i.dTLB-loads
    301519           -15.6%     254350 ± 15%  perf-stat.i.dTLB-store-misses
 9.324e+09 ±  2%      -9.6%  8.429e+09        perf-stat.i.dTLB-stores
   1.3e+11 ±  2%      -9.7%  1.174e+11        perf-stat.i.instructions
      0.55 ±  2%      -9.5%       0.50        perf-stat.i.ipc
    868.37 ±  2%      -9.6%     784.90        perf-stat.i.metric.M/sec
     21716            -8.6%      19839        perf-stat.i.minor-faults
     21.27 ± 22%      +8.0       29.24 ± 18%  perf-stat.i.node-load-miss-rate%
   1255035 ± 19%     +28.3%    1610117 ± 18%  perf-stat.i.node-load-misses
   6634872 ±  4%     -20.6%    5266347 ±  6%  perf-stat.i.node-loads
      8.84 ± 20%      +6.5       15.38 ±  5%  perf-stat.i.node-store-miss-rate%
   5327271 ± 18%     +67.2%    8907657 ±  6%  perf-stat.i.node-store-misses
  64286798 ±  3%     -11.1%   57150025 ±  2%  perf-stat.i.node-stores
     21715            -8.6%      19839        perf-stat.i.page-faults
      1.84 ±  2%     +10.7%       2.04        perf-stat.overall.cpi
    524.29 ±  2%     +10.8%     580.76        perf-stat.overall.cycles-between-cache-misses
      0.54 ±  2%      -9.7%       0.49        perf-stat.overall.ipc
     15.96 ± 20%      +7.1       23.10 ± 18%  perf-stat.overall.node-load-miss-rate%
      7.69 ± 19%      +5.7       13.43 ±  6%  perf-stat.overall.node-store-miss-rate%
 4.029e+10 ±  2%      -9.8%  3.633e+10        perf-stat.ps.branch-instructions
   5957957            -5.7%    5617140 ±  4%  perf-stat.ps.branch-misses
 4.449e+08 ±  2%      -9.9%  4.008e+08        perf-stat.ps.cache-misses
 6.291e+08 ±  2%      -9.9%  5.666e+08        perf-stat.ps.cache-references
    228.46           -11.1%     203.13 ±  2%  perf-stat.ps.cpu-migrations
 3.166e+10 ±  2%      -9.9%  2.853e+10        perf-stat.ps.dTLB-loads
    293736           -15.7%     247674 ± 15%  perf-stat.ps.dTLB-store-misses
 9.084e+09 ±  2%      -9.8%  8.198e+09        perf-stat.ps.dTLB-stores
 1.267e+11 ±  2%      -9.8%  1.142e+11        perf-stat.ps.instructions
     21187 ±  2%      -8.5%      19394        perf-stat.ps.minor-faults
   1239244 ± 19%     +30.6%    1618368 ± 18%  perf-stat.ps.node-load-misses
   6538174 ±  4%     -17.7%    5382104 ±  5%  perf-stat.ps.node-loads
   5198958 ± 18%     +65.4%    8600077 ±  5%  perf-stat.ps.node-store-misses
  62564031 ±  3%     -11.3%   55487648 ±  2%  perf-stat.ps.node-stores
     21187 ±  2%      -8.5%      19394        perf-stat.ps.page-faults
 3.844e+13 ±  2%      -9.9%  3.463e+13        perf-stat.total.instructions
     32299 ±  6%     -17.8%      26564 ±  4%  softirqs.CPU0.RCU
     27757 ±  4%     -12.6%      24249 ±  6%  softirqs.CPU10.RCU
     28495 ±  3%     -13.0%      24789 ±  7%  softirqs.CPU11.RCU
     27897 ±  3%     -11.8%      24618 ±  6%  softirqs.CPU12.RCU
     28373 ±  3%     -14.2%      24340 ±  6%  softirqs.CPU13.RCU
     27849 ±  3%     -12.6%      24349 ±  6%  softirqs.CPU14.RCU
     27734 ±  3%     -13.3%      24034 ±  6%  softirqs.CPU15.RCU
     27277 ±  4%     -13.2%      23673 ±  5%  softirqs.CPU17.RCU
     27160 ±  4%     -12.8%      23686 ±  6%  softirqs.CPU18.RCU
     27313 ±  5%     -14.0%      23495 ±  6%  softirqs.CPU19.RCU
     29155 ±  2%     -13.6%      25183 ±  7%  softirqs.CPU2.RCU
     27027 ±  5%     -12.9%      23528 ±  5%  softirqs.CPU20.RCU
     27046 ±  4%     -12.2%      23736 ±  7%  softirqs.CPU21.RCU
     27900 ±  5%     -13.0%      24261 ±  7%  softirqs.CPU23.RCU
     26879 ±  4%     -11.1%      23904 ±  6%  softirqs.CPU25.RCU
     25956 ±  5%     -10.4%      23255 ±  7%  softirqs.CPU27.RCU
     27445 ±  3%     -11.6%      24253 ±  8%  softirqs.CPU33.RCU
     27482 ±  4%     -11.6%      24286 ±  6%  softirqs.CPU34.RCU
     27108 ±  5%     -13.9%      23348 ±  6%  softirqs.CPU36.RCU
     27019 ±  4%     -10.9%      24085 ±  8%  softirqs.CPU37.RCU
     27504 ±  5%     -12.6%      24025 ±  7%  softirqs.CPU38.RCU
     27170 ±  4%     -11.2%      24138 ±  8%  softirqs.CPU39.RCU
     27287 ±  4%     -11.7%      24102 ±  7%  softirqs.CPU40.RCU
     27096 ±  4%     -11.2%      24056 ±  9%  softirqs.CPU43.RCU
     27009 ±  3%     -11.7%      23860 ±  8%  softirqs.CPU44.RCU
     27366 ±  4%     -13.9%      23569 ±  6%  softirqs.CPU45.RCU
     27323 ±  4%     -12.0%      24034 ±  7%  softirqs.CPU46.RCU
     27618 ±  3%     -12.8%      24087 ±  7%  softirqs.CPU47.RCU
     28356 ±  4%     -14.2%      24341 ±  7%  softirqs.CPU48.RCU
     26892 ±  4%      -9.4%      24354 ±  7%  softirqs.CPU49.RCU
     28403 ±  5%     -14.2%      24373 ±  7%  softirqs.CPU50.RCU
     28337 ±  5%     -15.6%      23910 ±  6%  softirqs.CPU53.RCU
     27952 ±  4%     -14.4%      23916 ±  6%  softirqs.CPU54.RCU
     28199 ±  4%     -15.4%      23857 ±  7%  softirqs.CPU55.RCU
      0.33 ±141%  +6.3e+05%       2093 ±223%  softirqs.CPU56.NET_RX
     28099 ±  4%     -13.8%      24224 ±  6%  softirqs.CPU56.RCU
     28113 ±  4%     -14.2%      24116 ±  7%  softirqs.CPU57.RCU
     27887 ±  4%     -13.5%      24124 ±  6%  softirqs.CPU58.RCU
     27956 ±  4%     -14.2%      23997 ±  7%  softirqs.CPU59.RCU
     28251 ±  4%     -13.4%      24471 ±  6%  softirqs.CPU6.RCU
     28020 ±  2%     -13.2%      24312 ±  7%  softirqs.CPU60.RCU
     27891 ±  3%     -12.9%      24280 ±  6%  softirqs.CPU61.RCU
     28095 ±  4%     -13.4%      24333 ±  8%  softirqs.CPU62.RCU
     28016 ±  4%     -14.1%      24058 ±  6%  softirqs.CPU63.RCU
     27348 ±  4%     -14.8%      23301 ±  6%  softirqs.CPU65.RCU
     27408 ±  4%     -14.5%      23427 ±  5%  softirqs.CPU66.RCU
     27249 ±  5%     -14.7%      23251 ±  5%  softirqs.CPU67.RCU
     27053 ±  4%     -13.8%      23315 ±  5%  softirqs.CPU68.RCU
     28186 ±  3%     -13.4%      24410 ±  6%  softirqs.CPU7.RCU
     27294 ±  3%     -14.7%      23281 ±  6%  softirqs.CPU71.RCU
     26266 ±  5%     -10.0%      23643 ±  5%  softirqs.CPU72.RCU
     26186 ±  5%     -12.1%      23027 ±  6%  softirqs.CPU73.RCU
     25744 ±  4%     -10.1%      23132 ±  5%  softirqs.CPU74.RCU
     25639 ±  5%     -12.3%      22473 ±  7%  softirqs.CPU75.RCU
     25588 ±  4%     -10.7%      22854 ±  7%  softirqs.CPU76.RCU
     25598 ±  4%      -9.4%      23183 ±  8%  softirqs.CPU77.RCU
     25639 ±  5%      -9.7%      23159 ±  8%  softirqs.CPU78.RCU
     25717 ±  5%     -11.0%      22881 ±  7%  softirqs.CPU79.RCU
     28004 ±  4%     -12.5%      24494 ±  6%  softirqs.CPU8.RCU
     26695 ±  4%     -11.7%      23561 ±  7%  softirqs.CPU80.RCU
     26830 ±  5%     -12.0%      23616 ±  6%  softirqs.CPU81.RCU
     26860 ±  4%     -11.9%      23673 ±  7%  softirqs.CPU82.RCU
     26629 ±  4%     -13.9%      22939 ±  8%  softirqs.CPU84.RCU
     26646 ±  4%     -11.7%      23524 ±  7%  softirqs.CPU85.RCU
     26604 ±  4%     -11.3%      23607 ±  7%  softirqs.CPU86.RCU
     26997 ±  3%     -12.8%      23554 ±  7%  softirqs.CPU87.RCU
     26914 ±  4%     -12.2%      23624 ±  7%  softirqs.CPU88.RCU
     27892 ±  3%     -13.0%      24257 ±  6%  softirqs.CPU9.RCU
     27074 ±  4%     -12.2%      23767 ±  6%  softirqs.CPU91.RCU
     26630 ±  4%     -11.4%      23605 ±  6%  softirqs.CPU92.RCU
     26610 ±  4%     -13.0%      23142 ±  4%  softirqs.CPU93.RCU
     26844 ±  4%     -12.0%      23615 ±  7%  softirqs.CPU94.RCU
     27037 ±  4%     -11.9%      23828 ±  7%  softirqs.CPU95.RCU
   2617482 ±  4%     -12.3%    2294936 ±  6%  softirqs.RCU


                                                                                
                                vm-scalability.median                           
                                                                                
  480000 +------------------------------------------------------------------+   
         |                                      +.                          |   
  470000 |-+ +.                                :  +.+   +         +.        |   
  460000 |-+ : +.                              :     + + +       +  +. .+.  |   
         |  :    +   +.       .+    .+. .+.+ .+       +   +. .+.+     +   + |   
  450000 |++:     + +  +    .+  + .+   +    +               +              +|   
         | +       +    + .+     +                                          |   
  440000 |-+             +                                                  |   
         |             O                                                    |   
  430000 |-+       O         O                                              |   
  420000 |-+   O     O         O       O   OO O O O O O                     |   
         |               O O     O O O   O                    O   O O       |   
  410000 |-+     O                                      O O O   O     O     |   
         | O O                                                              |   
  400000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp6: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/300s/512G/lkp-csl-2sp6/anon-w-rand-hugetlb/vm-scalability/0x5003003

commit: 
  46132e3ac5 ("sched: nohz: stop passing around unused "ticks" parameter.")
  3edecfef02 ("sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal")

46132e3ac58cb2ee 3edecfef028536cb19a120ec878 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          6:6          -56%           3:6     perf-profile.children.cycles-pp.error_entry
          0:6           -3%           0:6     perf-profile.self.cycles-pp.error_entry
           :6           83%           5:6     stderr.[perf_record:Captured_and_wrote#MB/tmp/lkp/perf-sched.data(#samples)]
           :6           83%           5:6     stderr.[perf_record:Woken_up#times_to_write_data]
         %stddev     %change         %stddev
             \          |                \  
     79915           -13.3%      69267 ±  7%  vm-scalability.median
   7788072 ±  2%     -11.0%    6929383 ±  7%  vm-scalability.throughput
    168.93 ±  4%     +29.5%     218.77 ± 11%  vm-scalability.time.elapsed_time
    168.93 ±  4%     +29.5%     218.77 ± 11%  vm-scalability.time.elapsed_time.max
      8533            -8.3%       7821 ±  2%  vm-scalability.time.percent_of_cpu_this_job_got
      1201 ±  5%     +32.5%       1591 ± 16%  vm-scalability.time.system_time
     13208 ±  3%     +17.3%      15490 ±  9%  vm-scalability.time.user_time
   2286392 ± 51%     +94.9%    4457153 ± 12%  cpuidle.C1E.usage
    143476           +29.9%     186313 ± 13%  meminfo.Shmem
    579371 ±  7%     +15.1%     666919 ±  7%  numa-numastat.node1.numa_hit
    443.79            -3.8%     426.76        pmeter.Average_Active_Power
     16396 ± 15%     +44.8%      23740 ± 16%  numa-meminfo.node0.Mapped
     53031 ±113%    +140.2%     127393 ± 48%  numa-meminfo.node1.Shmem
      4165 ± 15%     +45.2%       6048 ± 15%  numa-vmstat.node0.nr_mapped
     13269 ±113%    +140.0%      31850 ± 49%  numa-vmstat.node1.nr_shmem
    198.65 ±  3%     +25.1%     248.61 ±  9%  uptime.boot
      3885 ±  6%     +53.4%       5958 ± 12%  uptime.idle
      8.94 ± 14%      +7.7       16.63 ± 13%  mpstat.cpu.all.idle%
      2.33 ±  3%      -0.4        1.91 ±  5%  mpstat.cpu.all.irq%
     81.13            -7.4       73.71 ±  3%  mpstat.cpu.all.usr%
      9.50 ± 14%     +77.2%      16.83 ± 12%  vmstat.cpu.id
     79.33            -8.6%      72.50 ±  3%  vmstat.cpu.us
     86.83            -8.8%      79.17 ±  2%  vmstat.procs.r
      2683 ±  7%     +21.4%       3258 ±  3%  slabinfo.PING.active_objs
      2683 ±  7%     +21.4%       3258 ±  3%  slabinfo.PING.num_objs
      1151 ± 12%     +24.5%       1432 ±  5%  slabinfo.khugepaged_mm_slot.active_objs
      1151 ± 12%     +24.5%       1432 ±  5%  slabinfo.khugepaged_mm_slot.num_objs
      4425 ±  4%     +11.8%       4946 ±  4%  slabinfo.kmem_cache_node.active_objs
      4470 ±  4%     +12.1%       5009 ±  4%  slabinfo.kmem_cache_node.num_objs
      1239 ±  7%     +20.0%       1487 ±  6%  slabinfo.pool_workqueue.active_objs
      1262 ±  7%     +20.5%       1522 ±  5%  slabinfo.pool_workqueue.num_objs
      4554 ±  3%     +16.0%       5281 ±  3%  slabinfo.sock_inode_cache.active_objs
      4554 ±  3%     +16.0%       5281 ±  3%  slabinfo.sock_inode_cache.num_objs
      1049 ±  9%     +18.9%       1248 ±  5%  slabinfo.task_group.active_objs
      1049 ±  9%     +18.9%       1248 ±  5%  slabinfo.task_group.num_objs
     28.39 ± 12%     -16.7%      23.64 ±  8%  sched_debug.cfs_rq:/.load_avg.avg
    620.26 ± 28%     -51.9%     298.11 ± 26%  sched_debug.cfs_rq:/.load_avg.max
     94.54 ± 22%     -36.8%      59.75 ± 23%  sched_debug.cfs_rq:/.load_avg.stddev
    127570 ± 45%    +179.0%     355860 ± 23%  sched_debug.cfs_rq:/.min_vruntime.stddev
    407113 ± 42%    +130.6%     938842 ± 24%  sched_debug.cfs_rq:/.spread0.max
    127615 ± 45%    +178.8%     355784 ± 23%  sched_debug.cfs_rq:/.spread0.stddev
      0.31 ± 79%   +4490.9%      14.03 ± 71%  sched_debug.cfs_rq:/.util_est_enqueued.min
      7.92 ± 35%     +83.2%      14.50 ± 42%  sched_debug.cpu.clock.stddev
     87255 ± 19%     +34.3%     117144 ± 20%  sched_debug.cpu.clock_task.max
     82473 ± 20%     +36.1%     112246 ± 21%  sched_debug.cpu.clock_task.min
    850.75 ± 97%    +254.7%       3017 ± 34%  sched_debug.cpu.curr->pid.min
     53.57 ± 12%     -29.5%      37.75 ± 17%  sched_debug.cpu.nr_uninterruptible.max
    -36.50           -57.8%     -15.40        sched_debug.cpu.nr_uninterruptible.min
     11.35 ± 10%     -33.9%       7.50 ± 10%  sched_debug.cpu.nr_uninterruptible.stddev
     64866            -1.3%      64033        proc-vmstat.nr_anon_pages
   3522603            +7.3%    3781144        proc-vmstat.nr_dirty_background_threshold
   7053819            +7.3%    7571535        proc-vmstat.nr_dirty_threshold
    282108            +3.8%     292946 ±  2%  proc-vmstat.nr_file_pages
  35562144            +7.3%   38151646        proc-vmstat.nr_free_pages
     33958            +8.3%      36786 ±  4%  proc-vmstat.nr_inactive_anon
      8890 ±  2%     +44.1%      12812 ± 21%  proc-vmstat.nr_mapped
      1173            -3.7%       1130        proc-vmstat.nr_page_table_pages
     35769           +30.3%      46606 ± 13%  proc-vmstat.nr_shmem
     25463            +2.6%      26116        proc-vmstat.nr_slab_reclaimable
     33958            +8.3%      36786 ±  4%  proc-vmstat.nr_zone_inactive_anon
   1161650           +13.3%    1316444 ±  5%  proc-vmstat.numa_hit
   1075137           +14.4%    1229766 ±  5%  proc-vmstat.numa_local
     19245 ± 67%     +89.9%      36545 ± 35%  proc-vmstat.numa_pages_migrated
    154360 ± 19%    +107.5%     320246 ± 11%  proc-vmstat.numa_pte_updates
   1080095 ±  2%     +19.1%    1285944 ±  7%  proc-vmstat.pgfault
     19245 ± 67%     +89.9%      36545 ± 35%  proc-vmstat.pgmigrate_success
 1.725e+10 ±  4%     -21.8%  1.349e+10 ± 11%  perf-stat.i.branch-instructions
      0.07            +0.0        0.07 ±  5%  perf-stat.i.branch-miss-rate%
 8.169e+08 ±  4%     -22.1%  6.367e+08 ± 11%  perf-stat.i.cache-misses
 8.391e+08 ±  4%     -22.1%  6.537e+08 ± 11%  perf-stat.i.cache-references
 2.426e+11            -8.6%  2.217e+11 ±  2%  perf-stat.i.cpu-cycles
    172.12 ±  2%      -9.5%     155.81 ±  3%  perf-stat.i.cpu-migrations
    404.33 ±  5%     +18.6%     479.58 ± 12%  perf-stat.i.cycles-between-cache-misses
 2.052e+10 ±  4%     -21.8%  1.604e+10 ± 11%  perf-stat.i.dTLB-loads
 7.416e+09 ±  4%     -21.7%  5.804e+09 ± 11%  perf-stat.i.dTLB-stores
     86.96 ±  2%     -11.0       75.99 ±  4%  perf-stat.i.iTLB-load-miss-rate%
    826335 ±  2%     +15.1%     950959 ±  5%  perf-stat.i.iTLB-load-misses
    255821 ± 20%    +105.4%     525564 ± 15%  perf-stat.i.iTLB-loads
 7.322e+10 ±  4%     -21.8%  5.728e+10 ± 11%  perf-stat.i.instructions
    141223 ±  5%     -29.8%      99153 ± 15%  perf-stat.i.instructions-per-iTLB-miss
      0.31 ±  2%      -9.7%       0.28 ±  7%  perf-stat.i.ipc
      2.52            -8.6%       2.31 ±  2%  perf-stat.i.metric.GHz
    487.25 ±  4%     -21.7%     381.56 ± 10%  perf-stat.i.metric.M/sec
     38.29 ±  3%      +7.8       46.08 ±  6%  perf-stat.i.node-load-miss-rate%
     13.62 ± 26%     +18.9       32.56 ± 21%  perf-stat.i.node-store-miss-rate%
  92357446 ± 17%     +99.0%  1.838e+08 ± 13%  perf-stat.i.node-store-misses
 7.197e+08 ±  7%     -37.6%  4.488e+08 ± 20%  perf-stat.i.node-stores
      0.03 ±  2%      +0.0        0.04 ±  9%  perf-stat.overall.branch-miss-rate%
      3.33 ±  3%     +18.1%       3.94 ± 10%  perf-stat.overall.cpi
    298.83 ±  3%     +18.6%     354.39 ± 10%  perf-stat.overall.cycles-between-cache-misses
     76.28 ±  4%     -11.6       64.65 ±  4%  perf-stat.overall.iTLB-load-miss-rate%
     86734 ±  5%     -30.6%      60222 ± 16%  perf-stat.overall.instructions-per-iTLB-miss
      0.30 ±  3%     -14.6%       0.26 ±  9%  perf-stat.overall.ipc
     26.76 ±  6%      +7.1       33.91 ±  8%  perf-stat.overall.node-load-miss-rate%
     11.69 ± 22%     +18.5       30.19 ± 21%  perf-stat.overall.node-store-miss-rate%
   1.7e+10 ±  4%     -21.5%  1.335e+10 ± 11%  perf-stat.ps.branch-instructions
 8.054e+08 ±  4%     -21.8%    6.3e+08 ± 11%  perf-stat.ps.cache-misses
 8.274e+08 ±  4%     -21.8%  6.469e+08 ± 11%  perf-stat.ps.cache-references
 2.403e+11            -8.1%  2.208e+11 ±  2%  perf-stat.ps.cpu-cycles
    170.38 ±  3%     -10.0%     153.39 ±  3%  perf-stat.ps.cpu-migrations
 2.023e+10 ±  4%     -21.5%  1.587e+10 ± 11%  perf-stat.ps.dTLB-loads
 7.311e+09 ±  4%     -21.4%  5.744e+09 ± 11%  perf-stat.ps.dTLB-stores
    833122 ±  2%     +13.8%     948353 ±  4%  perf-stat.ps.iTLB-load-misses
    261616 ± 18%    +100.2%     523767 ± 15%  perf-stat.ps.iTLB-loads
 7.219e+10 ±  4%     -21.5%  5.669e+10 ± 11%  perf-stat.ps.instructions
   1245235 ±  3%     -18.4%    1015496 ± 14%  perf-stat.ps.node-loads
  92708873 ± 17%     +99.0%  1.845e+08 ± 13%  perf-stat.ps.node-store-misses
 7.078e+08 ±  7%     -37.6%  4.415e+08 ± 21%  perf-stat.ps.node-stores
      0.00 ±223%   +1263.2%       0.04 ± 56%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00 ±149%    +560.7%       0.03 ± 65%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
      0.14 ±218%    +456.5%       0.79 ± 64%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
      0.00 ±161%   +1627.3%       0.03 ± 54%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.do_wait.kernel_wait4.__do_sys_wait4
      0.00 ±223%  +4.3e+05%       2.85 ± 65%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.pipe_wait.wait_for_partner.fifo_open
      0.00 ±223%   +2015.4%       0.05 ± 51%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ±223%    +935.1%       0.06 ± 48%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.01 ±179%    +603.6%       0.07 ± 49%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
      0.84 ±219%   +1573.8%      14.11 ± 49%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
      0.01 ±190%  +24300.0%       1.87 ±200%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.do_wait.kernel_wait4.__do_sys_wait4
      0.42 ±127%   +1571.1%       7.00 ± 84%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
      0.00 ±223%  +7.9e+05%      15.89 ± 44%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.pipe_wait.wait_for_partner.fifo_open
      0.04 ±223%  +40511.0%      16.04 ± 45%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.preempt_schedule_common._cond_resched.stop_one_cpu
      0.01 ±223%   +1646.7%       0.09 ± 47%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.rcu_gp_kthread.kthread.ret_from_fork
      0.02 ± 80%  +16955.9%       3.87 ±215%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
      0.04 ± 54%    +199.1%       0.11 ± 40%  perf-sched.total_sch_delay.average.ms
      1214 ±158%   +1006.6%      13433 ± 44%  perf-sched.total_wait_and_delay.count.ms
      1066 ±198%    +455.3%       5923 ± 38%  perf-sched.total_wait_and_delay.max.ms
      1066 ±198%    +455.6%       5923 ± 38%  perf-sched.total_wait_time.max.ms
     28.84 ±220%    +748.6%     244.76 ± 18%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
      4.34 ±222%   +1319.9%      61.56 ± 45%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_wait.kernel_wait4.__do_sys_wait4
     10.60 ±223%   +1433.3%     162.49 ± 73%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
    103.54 ±223%    +536.5%     659.00 ± 14%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
     72.96 ±207%    +622.7%     527.29 ± 36%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
     35.83 ±221%    +562.3%     237.33 ± 35%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.do_wait.kernel_wait4.__do_sys_wait4
    235.33 ±133%   +1168.4%       2985 ± 61%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
    192.67 ±112%    +380.9%     926.50 ± 36%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.preempt_schedule_common._cond_resched.stop_one_cpu
    157.00 ±175%    +705.5%       1264 ± 42%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
    213.33 ±122%    +461.1%       1197 ± 38%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
    112.83 ±194%    +375.5%     536.50 ± 36%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
    167.94 ±221%   +1311.4%       2370 ± 57%  perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
     46.34 ±202%    +516.1%     285.50 ± 37%  perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
    174.52 ±223%   +2432.1%       4419 ± 43%  perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
      1061 ±199%    +428.4%       5608 ± 41%  perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      0.01 ±223%   +1716.3%       0.26 ± 40%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.__prepare_exit_to_usermode.prepare_exit_to_usermode.asm_sysvec_reschedule_ipi
     28.70 ±221%    +750.1%     243.98 ± 18%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
      4.33 ±222%   +1319.8%      61.53 ± 45%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_wait.kernel_wait4.__do_sys_wait4
      0.40 ±223%    +576.4%       2.70 ± 21%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.rcu_gp_kthread.kthread.ret_from_fork
     13.34 ±192%   +1372.4%     196.36 ± 91%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
     10.58 ±223%   +1434.6%     162.42 ± 73%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
    103.53 ±223%    +536.5%     658.98 ± 14%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
     72.51 ±208%    +627.0%     527.14 ± 36%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      0.04 ±223%  +31752.0%      12.16 ± 49%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.__prepare_exit_to_usermode.prepare_exit_to_usermode.asm_sysvec_reschedule_ipi
    167.75 ±221%   +1313.1%       2370 ± 57%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
      0.83 ±223%    +478.6%       4.80 ±  7%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.rcu_gp_kthread.kthread.ret_from_fork
    130.49 ±208%   +1704.9%       2355 ±110%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
     46.33 ±202%    +516.2%     285.46 ± 37%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
    174.50 ±223%   +2432.4%       4418 ± 43%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
      1059 ±200%    +429.5%       5608 ± 41%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
     65626 ±  2%     +28.5%      84315 ±  9%  softirqs.CPU1.TIMER
     19554 ±  6%     +20.5%      23569 ± 10%  softirqs.CPU10.RCU
     62870 ±  3%     +22.7%      77143 ±  9%  softirqs.CPU10.TIMER
     19436 ±  6%     +23.8%      24067 ± 10%  softirqs.CPU11.RCU
     19803 ±  7%     +20.9%      23944 ± 10%  softirqs.CPU12.RCU
     19779 ±  7%     +21.1%      23950 ± 11%  softirqs.CPU13.RCU
     20020 ±  5%     +18.0%      23618 ±  9%  softirqs.CPU14.RCU
     19605 ±  5%     +21.2%      23770 ± 12%  softirqs.CPU15.RCU
     62935 ±  3%     +23.3%      77620 ± 10%  softirqs.CPU15.TIMER
     21091 ±  9%     +18.5%      24997 ± 11%  softirqs.CPU17.RCU
     62600 ±  4%     +23.8%      77519 ±  8%  softirqs.CPU17.TIMER
     21645 ±  6%     +18.5%      25653 ±  8%  softirqs.CPU24.RCU
      4111 ± 18%     +48.6%       6108 ± 30%  softirqs.CPU24.SCHED
     20716 ±  9%     +19.7%      24799 ±  9%  softirqs.CPU25.RCU
     20482 ±  3%     +19.4%      24465 ±  9%  softirqs.CPU26.RCU
     20006 ±  6%     +23.5%      24709 ± 11%  softirqs.CPU27.RCU
     19976 ±  6%     +19.1%      23797 ± 10%  softirqs.CPU28.RCU
     20100 ±  6%     +17.6%      23644 ±  9%  softirqs.CPU29.RCU
     20524 ±  7%     +17.4%      24090 ± 10%  softirqs.CPU3.RCU
     19945 ±  6%     +18.1%      23564 ± 10%  softirqs.CPU30.RCU
     63771 ±  2%     +22.3%      77962 ± 10%  softirqs.CPU30.TIMER
     19917 ±  8%     +19.5%      23808 ±  9%  softirqs.CPU31.RCU
     19557 ±  6%     +20.0%      23479 ± 10%  softirqs.CPU32.RCU
      3852 ±  9%     +93.5%       7455 ± 21%  softirqs.CPU34.SCHED
     20202 ±  6%     +21.1%      24467 ± 10%  softirqs.CPU35.RCU
     19446 ±  5%     +21.8%      23679 ± 10%  softirqs.CPU36.RCU
     19717 ±  6%     +20.7%      23805 ± 11%  softirqs.CPU37.RCU
     19892 ±  6%     +20.9%      24041 ±  9%  softirqs.CPU4.RCU
     19835 ±  6%     +21.3%      24051 ± 10%  softirqs.CPU41.RCU
     63390 ±  2%     +23.3%      78189 ± 10%  softirqs.CPU41.TIMER
     19717 ±  7%     +21.5%      23948 ±  9%  softirqs.CPU42.RCU
     19858 ±  8%     +20.5%      23936 ± 10%  softirqs.CPU46.RCU
     62868 ±  2%     +24.8%      78447 ± 10%  softirqs.CPU46.TIMER
     19603 ±  7%     +22.3%      23970 ± 11%  softirqs.CPU47.RCU
     20557 ± 10%     +21.5%      24981 ±  9%  softirqs.CPU48.RCU
      4300 ± 21%     +74.5%       7504 ± 25%  softirqs.CPU48.SCHED
     63555 ±  3%     +23.9%      78714 ±  7%  softirqs.CPU49.TIMER
     20118 ±  6%     +18.5%      23848 ± 10%  softirqs.CPU5.RCU
     19767 ±  6%     +19.3%      23587 ± 10%  softirqs.CPU51.RCU
     19818 ±  6%     +21.6%      24095 ± 11%  softirqs.CPU52.RCU
     19648 ±  7%     +24.1%      24386 ± 11%  softirqs.CPU53.RCU
     19892 ±  9%     +22.8%      24436 ± 10%  softirqs.CPU54.RCU
     19783 ±  8%     +25.1%      24750 ± 10%  softirqs.CPU55.RCU
     63044 ±  3%     +22.0%      76910 ± 10%  softirqs.CPU55.TIMER
     20067 ± 10%     +22.9%      24656 ± 10%  softirqs.CPU56.RCU
     19618 ±  7%     +25.8%      24673 ± 10%  softirqs.CPU57.RCU
     19606 ±  8%     +24.6%      24435 ± 10%  softirqs.CPU58.RCU
     18847 ±  9%     +31.4%      24775 ± 11%  softirqs.CPU59.RCU
     20102 ±  6%     +19.9%      24098 ± 11%  softirqs.CPU6.RCU
     19547 ±  7%     +27.0%      24833 ± 11%  softirqs.CPU60.RCU
     19472 ±  8%     +28.8%      25083 ± 11%  softirqs.CPU61.RCU
     19562 ±  7%     +26.8%      24797 ± 11%  softirqs.CPU62.RCU
     19171 ±  7%     +28.4%      24623 ± 11%  softirqs.CPU63.RCU
     62091 ±  3%     +21.9%      75716 ±  9%  softirqs.CPU63.TIMER
     63260 ±  3%     +22.6%      77542 ±  9%  softirqs.CPU7.TIMER
     62533 ±  3%     +21.7%      76080 ±  7%  softirqs.CPU71.TIMER
      3900 ± 15%    +104.5%       7977 ± 20%  softirqs.CPU73.SCHED
     20119 ±  6%     +17.6%      23668 ± 10%  softirqs.CPU74.RCU
      4061 ± 18%    +103.7%       8272 ± 18%  softirqs.CPU74.SCHED
     19994 ±  6%     +15.1%      23018 ± 11%  softirqs.CPU75.RCU
      3830 ±  9%    +125.2%       8625 ± 19%  softirqs.CPU75.SCHED
      4206 ± 13%     +95.1%       8204 ± 27%  softirqs.CPU76.SCHED
     19954 ±  5%     +18.1%      23567 ± 10%  softirqs.CPU77.RCU
     19666 ±  9%     +19.9%      23589 ±  9%  softirqs.CPU79.RCU
     63515 ±  2%     +23.3%      78312 ± 10%  softirqs.CPU79.TIMER
     19985 ±  7%     +19.5%      23884 ± 11%  softirqs.CPU8.RCU
     18922 ±  6%     +23.1%      23291 ± 11%  softirqs.CPU80.RCU
      4015 ±  5%     +95.9%       7864 ± 24%  softirqs.CPU80.SCHED
     62536 ±  3%     +24.7%      77985 ± 10%  softirqs.CPU80.TIMER
     19059 ±  5%     +25.6%      23938 ±  9%  softirqs.CPU81.RCU
      4262 ± 23%     +83.0%       7801 ± 21%  softirqs.CPU81.SCHED
      4032 ± 10%     +86.2%       7507 ± 31%  softirqs.CPU82.SCHED
     19842 ±  6%     +20.8%      23968 ± 11%  softirqs.CPU83.RCU
      3667 ± 12%    +108.5%       7645 ± 26%  softirqs.CPU83.SCHED
     19061 ±  5%     +24.0%      23642 ± 11%  softirqs.CPU84.RCU
     19111 ±  7%     +23.6%      23617 ± 12%  softirqs.CPU85.RCU
      4056 ±  9%    +105.1%       8318 ± 23%  softirqs.CPU86.SCHED
     19320 ±  6%     +22.7%      23701 ± 12%  softirqs.CPU87.RCU
     19361 ±  5%     +23.0%      23820 ± 11%  softirqs.CPU89.RCU
     19486 ±  6%     +22.8%      23931 ± 11%  softirqs.CPU9.RCU
     62122 ±  3%     +24.4%      77267 ± 10%  softirqs.CPU9.TIMER
     63536 ±  4%     +22.8%      78029 ± 10%  softirqs.CPU90.TIMER
      3918 ± 15%     +88.7%       7392 ± 29%  softirqs.CPU92.SCHED
     62738 ±  2%     +25.1%      78506 ± 11%  softirqs.CPU92.TIMER
     19187 ±  5%     +21.4%      23301 ± 11%  softirqs.CPU93.RCU
      3853 ±  9%     +87.7%       7233 ± 27%  softirqs.CPU93.SCHED
     19058 ±  6%     +22.4%      23332 ± 10%  softirqs.CPU94.RCU
      4009 ± 13%     +93.6%       7762 ± 23%  softirqs.CPU94.SCHED
     62629 ±  3%     +24.5%      77993 ± 10%  softirqs.CPU94.TIMER
     19415 ±  5%     +22.4%      23757 ± 10%  softirqs.CPU95.RCU
   1928612 ±  6%     +20.3%    2320059 ± 10%  softirqs.RCU
    382010 ±  8%     +70.1%     649612 ± 14%  softirqs.SCHED
    341.67 ±  4%     +29.1%     441.00 ± 11%  interrupts.9:IO-APIC.9-fasteoi.acpi
     57582            +6.5%      61309 ±  3%  interrupts.CAL:Function_call_interrupts
    340131 ±  4%     +29.1%     439047 ± 11%  interrupts.CPU0.LOC:Local_timer_interrupts
    103.17 ± 76%     -73.7%      27.17 ± 83%  interrupts.CPU0.TLB:TLB_shootdowns
    341.67 ±  4%     +29.1%     441.00 ± 11%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    339991 ±  4%     +29.3%     439771 ± 11%  interrupts.CPU1.LOC:Local_timer_interrupts
    498.00 ±  5%     +16.4%     579.50 ± 14%  interrupts.CPU10.CAL:Function_call_interrupts
    340281 ±  4%     +29.2%     439731 ± 11%  interrupts.CPU10.LOC:Local_timer_interrupts
    182.00 ±  8%     +21.2%     220.50 ± 11%  interrupts.CPU10.RES:Rescheduling_interrupts
    340509 ±  4%     +29.2%     439804 ± 11%  interrupts.CPU11.LOC:Local_timer_interrupts
    340289 ±  4%     +29.2%     439621 ± 11%  interrupts.CPU12.LOC:Local_timer_interrupts
    340465 ±  4%     +29.2%     439824 ± 11%  interrupts.CPU13.LOC:Local_timer_interrupts
    339838 ±  4%     +29.4%     439785 ± 11%  interrupts.CPU14.LOC:Local_timer_interrupts
    340379 ±  4%     +29.2%     439826 ± 11%  interrupts.CPU15.LOC:Local_timer_interrupts
    340095 ±  4%     +29.3%     439679 ± 11%  interrupts.CPU16.LOC:Local_timer_interrupts
    182.50 ±  9%     +21.4%     221.50 ±  9%  interrupts.CPU16.RES:Rescheduling_interrupts
    340031 ±  4%     +29.3%     439782 ± 11%  interrupts.CPU17.LOC:Local_timer_interrupts
    340439 ±  4%     +29.2%     439759 ± 11%  interrupts.CPU18.LOC:Local_timer_interrupts
    340357 ±  4%     +29.2%     439748 ± 11%  interrupts.CPU19.LOC:Local_timer_interrupts
    340412 ±  4%     +29.1%     439364 ± 11%  interrupts.CPU2.LOC:Local_timer_interrupts
    340418 ±  4%     +29.2%     439903 ± 11%  interrupts.CPU20.LOC:Local_timer_interrupts
    340280 ±  4%     +29.2%     439783 ± 11%  interrupts.CPU21.LOC:Local_timer_interrupts
    340310 ±  4%     +29.3%     439888 ± 11%  interrupts.CPU22.LOC:Local_timer_interrupts
    340236 ±  4%     +29.2%     439693 ± 11%  interrupts.CPU23.LOC:Local_timer_interrupts
    339882 ±  4%     +29.4%     439890 ± 11%  interrupts.CPU24.LOC:Local_timer_interrupts
    339759 ±  4%     +29.5%     440027 ± 11%  interrupts.CPU25.LOC:Local_timer_interrupts
      7299 ±  8%     -18.1%       5977 ± 25%  interrupts.CPU25.NMI:Non-maskable_interrupts
      7299 ±  8%     -18.1%       5977 ± 25%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    339847 ±  4%     +29.5%     440016 ± 11%  interrupts.CPU26.LOC:Local_timer_interrupts
    616.83 ± 13%     +26.7%     781.33 ± 31%  interrupts.CPU27.CAL:Function_call_interrupts
    339261 ±  4%     +29.7%     439970 ± 11%  interrupts.CPU27.LOC:Local_timer_interrupts
    340015 ±  4%     +29.3%     439801 ± 11%  interrupts.CPU28.LOC:Local_timer_interrupts
    339855 ±  4%     +29.4%     439859 ± 11%  interrupts.CPU29.LOC:Local_timer_interrupts
    340361 ±  4%     +29.2%     439819 ± 11%  interrupts.CPU3.LOC:Local_timer_interrupts
    339860 ±  4%     +29.5%     439980 ± 11%  interrupts.CPU30.LOC:Local_timer_interrupts
    509.00 ±  6%     +15.5%     587.67 ±  8%  interrupts.CPU31.CAL:Function_call_interrupts
    339938 ±  4%     +29.4%     439861 ± 11%  interrupts.CPU31.LOC:Local_timer_interrupts
    339357 ±  4%     +29.6%     439716 ± 11%  interrupts.CPU32.LOC:Local_timer_interrupts
    499.00 ±  3%     +83.1%     913.83 ± 46%  interrupts.CPU33.CAL:Function_call_interrupts
    339849 ±  4%     +29.4%     439820 ± 11%  interrupts.CPU33.LOC:Local_timer_interrupts
    339868 ±  4%     +29.4%     439729 ± 11%  interrupts.CPU34.LOC:Local_timer_interrupts
    490.33 ±  3%     +29.3%     634.00 ± 20%  interrupts.CPU35.CAL:Function_call_interrupts
    339532 ±  4%     +29.5%     439758 ± 11%  interrupts.CPU35.LOC:Local_timer_interrupts
    496.17 ±  3%     +12.2%     556.50 ±  5%  interrupts.CPU36.CAL:Function_call_interrupts
    339718 ±  4%     +29.5%     439936 ± 11%  interrupts.CPU36.LOC:Local_timer_interrupts
    339869 ±  4%     +29.4%     439952 ± 11%  interrupts.CPU37.LOC:Local_timer_interrupts
    339888 ±  4%     +29.4%     439855 ± 11%  interrupts.CPU38.LOC:Local_timer_interrupts
    339940 ±  4%     +29.4%     439869 ± 11%  interrupts.CPU39.LOC:Local_timer_interrupts
    340135 ±  4%     +29.3%     439776 ± 11%  interrupts.CPU4.LOC:Local_timer_interrupts
    339932 ±  4%     +29.4%     439876 ± 11%  interrupts.CPU40.LOC:Local_timer_interrupts
    339713 ±  4%     +29.5%     439930 ± 11%  interrupts.CPU41.LOC:Local_timer_interrupts
    185.83 ± 15%     -21.5%     145.83 ± 14%  interrupts.CPU41.RES:Rescheduling_interrupts
    487.67           +10.2%     537.33 ±  4%  interrupts.CPU42.CAL:Function_call_interrupts
    339970 ±  4%     +29.4%     439827 ± 11%  interrupts.CPU42.LOC:Local_timer_interrupts
    458.17 ± 15%     +30.9%     599.67 ± 11%  interrupts.CPU43.CAL:Function_call_interrupts
    339686 ±  4%     +29.5%     439797 ± 11%  interrupts.CPU43.LOC:Local_timer_interrupts
    339957 ±  4%     +29.4%     439926 ± 11%  interrupts.CPU44.LOC:Local_timer_interrupts
    481.67 ±  2%     +73.1%     833.83 ± 69%  interrupts.CPU45.CAL:Function_call_interrupts
    339787 ±  4%     +29.5%     439868 ± 11%  interrupts.CPU45.LOC:Local_timer_interrupts
    493.83 ±  5%     +66.8%     823.67 ± 73%  interrupts.CPU46.CAL:Function_call_interrupts
    339874 ±  4%     +29.4%     439861 ± 11%  interrupts.CPU46.LOC:Local_timer_interrupts
    339758 ±  4%     +29.5%     439894 ± 11%  interrupts.CPU47.LOC:Local_timer_interrupts
    339804 ±  4%     +29.3%     439236 ± 11%  interrupts.CPU48.LOC:Local_timer_interrupts
    242.67 ± 15%     -17.0%     201.33 ±  7%  interrupts.CPU48.RES:Rescheduling_interrupts
    340200 ±  4%     +29.2%     439670 ± 11%  interrupts.CPU49.LOC:Local_timer_interrupts
    340322 ±  4%     +29.1%     439457 ± 11%  interrupts.CPU5.LOC:Local_timer_interrupts
    340041 ±  4%     +29.2%     439493 ± 11%  interrupts.CPU50.LOC:Local_timer_interrupts
    339801 ±  4%     +29.3%     439499 ± 11%  interrupts.CPU51.LOC:Local_timer_interrupts
    340202 ±  4%     +29.2%     439562 ± 11%  interrupts.CPU52.LOC:Local_timer_interrupts
    340102 ±  4%     +29.3%     439710 ± 11%  interrupts.CPU53.LOC:Local_timer_interrupts
    340363 ±  4%     +29.1%     439574 ± 11%  interrupts.CPU54.LOC:Local_timer_interrupts
    340203 ±  4%     +29.2%     439587 ± 11%  interrupts.CPU55.LOC:Local_timer_interrupts
    340396 ±  4%     +29.2%     439680 ± 11%  interrupts.CPU56.LOC:Local_timer_interrupts
    185.67 ± 26%     -29.2%     131.50 ± 15%  interrupts.CPU56.RES:Rescheduling_interrupts
    340301 ±  4%     +29.1%     439312 ± 11%  interrupts.CPU57.LOC:Local_timer_interrupts
    340195 ±  4%     +29.2%     439494 ± 11%  interrupts.CPU58.LOC:Local_timer_interrupts
    340372 ±  4%     +29.1%     439498 ± 11%  interrupts.CPU59.LOC:Local_timer_interrupts
    340433 ±  4%     +29.1%     439543 ± 11%  interrupts.CPU6.LOC:Local_timer_interrupts
    340431 ±  4%     +29.1%     439499 ± 11%  interrupts.CPU60.LOC:Local_timer_interrupts
    168.50 ± 16%     -26.0%     124.67 ± 17%  interrupts.CPU60.RES:Rescheduling_interrupts
    340384 ±  4%     +29.2%     439617 ± 11%  interrupts.CPU61.LOC:Local_timer_interrupts
    339850 ±  4%     +29.4%     439603 ± 11%  interrupts.CPU62.LOC:Local_timer_interrupts
    203.83 ± 37%     -40.6%     121.17 ± 20%  interrupts.CPU62.RES:Rescheduling_interrupts
    340481 ±  4%     +29.1%     439530 ± 11%  interrupts.CPU63.LOC:Local_timer_interrupts
    162.33 ± 11%     -34.0%     107.17 ± 24%  interrupts.CPU63.RES:Rescheduling_interrupts
    340265 ±  4%     +29.2%     439507 ± 11%  interrupts.CPU64.LOC:Local_timer_interrupts
    181.67 ± 15%     -37.2%     114.00 ± 25%  interrupts.CPU64.RES:Rescheduling_interrupts
    340241 ±  4%     +29.1%     439420 ± 11%  interrupts.CPU65.LOC:Local_timer_interrupts
    339987 ±  4%     +29.3%     439525 ± 11%  interrupts.CPU66.LOC:Local_timer_interrupts
    201.50 ± 29%     -40.3%     120.33 ± 20%  interrupts.CPU66.RES:Rescheduling_interrupts
    340318 ±  4%     +29.2%     439526 ± 11%  interrupts.CPU67.LOC:Local_timer_interrupts
    187.00 ± 27%     -38.0%     116.00 ± 23%  interrupts.CPU67.RES:Rescheduling_interrupts
    340390 ±  4%     +29.1%     439281 ± 11%  interrupts.CPU68.LOC:Local_timer_interrupts
    339729 ±  4%     +29.4%     439626 ± 11%  interrupts.CPU69.LOC:Local_timer_interrupts
    200.50 ± 20%     -46.7%     106.83 ± 25%  interrupts.CPU69.RES:Rescheduling_interrupts
    340337 ±  4%     +29.2%     439776 ± 11%  interrupts.CPU7.LOC:Local_timer_interrupts
    340020 ±  4%     +29.2%     439403 ± 11%  interrupts.CPU70.LOC:Local_timer_interrupts
    339815 ±  4%     +29.4%     439598 ± 11%  interrupts.CPU71.LOC:Local_timer_interrupts
    339751 ±  4%     +29.5%     440065 ± 11%  interrupts.CPU72.LOC:Local_timer_interrupts
      7152 ± 12%     -26.5%       5258 ± 36%  interrupts.CPU72.NMI:Non-maskable_interrupts
      7152 ± 12%     -26.5%       5258 ± 36%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
    340050 ±  4%     +29.3%     439820 ± 11%  interrupts.CPU73.LOC:Local_timer_interrupts
    339658 ±  4%     +29.5%     439755 ± 11%  interrupts.CPU74.LOC:Local_timer_interrupts
    186.17 ±  9%     -33.8%     123.33 ± 32%  interrupts.CPU74.RES:Rescheduling_interrupts
    339690 ±  4%     +29.4%     439718 ± 11%  interrupts.CPU75.LOC:Local_timer_interrupts
    339894 ±  4%     +29.4%     439684 ± 11%  interrupts.CPU76.LOC:Local_timer_interrupts
    339730 ±  4%     +29.5%     439909 ± 11%  interrupts.CPU77.LOC:Local_timer_interrupts
    196.33 ± 27%     -48.0%     102.00 ± 18%  interrupts.CPU77.RES:Rescheduling_interrupts
    339842 ±  4%     +29.4%     439770 ± 11%  interrupts.CPU78.LOC:Local_timer_interrupts
    170.00 ± 10%     -41.2%     100.00 ± 15%  interrupts.CPU78.RES:Rescheduling_interrupts
    339656 ±  4%     +29.5%     439774 ± 11%  interrupts.CPU79.LOC:Local_timer_interrupts
    182.67 ± 25%     -44.8%     100.83 ± 21%  interrupts.CPU79.RES:Rescheduling_interrupts
    340204 ±  4%     +29.2%     439604 ± 11%  interrupts.CPU8.LOC:Local_timer_interrupts
    512.17 ±  8%     +20.2%     615.67 ± 17%  interrupts.CPU80.CAL:Function_call_interrupts
    339513 ±  4%     +29.5%     439578 ± 11%  interrupts.CPU80.LOC:Local_timer_interrupts
    189.83 ±  9%     -38.6%     116.50 ± 13%  interrupts.CPU80.RES:Rescheduling_interrupts
    339817 ±  4%     +29.4%     439864 ± 11%  interrupts.CPU81.LOC:Local_timer_interrupts
    339810 ±  4%     +29.4%     439841 ± 11%  interrupts.CPU82.LOC:Local_timer_interrupts
    190.33 ± 17%     -41.0%     112.33 ± 26%  interrupts.CPU82.RES:Rescheduling_interrupts
    339918 ±  4%     +29.4%     439915 ± 11%  interrupts.CPU83.LOC:Local_timer_interrupts
    170.50 ± 25%     -40.2%     102.00 ± 24%  interrupts.CPU83.RES:Rescheduling_interrupts
    339936 ±  4%     +29.3%     439622 ± 11%  interrupts.CPU84.LOC:Local_timer_interrupts
    175.17 ± 18%     -33.9%     115.83 ± 25%  interrupts.CPU84.RES:Rescheduling_interrupts
    339819 ±  4%     +29.5%     440002 ± 11%  interrupts.CPU85.LOC:Local_timer_interrupts
    339972 ±  4%     +29.4%     439851 ± 11%  interrupts.CPU86.LOC:Local_timer_interrupts
    163.00 ± 10%     -44.9%      89.83 ± 21%  interrupts.CPU86.RES:Rescheduling_interrupts
    339821 ±  4%     +29.4%     439797 ± 11%  interrupts.CPU87.LOC:Local_timer_interrupts
    165.00 ± 21%     -42.6%      94.67 ± 28%  interrupts.CPU87.RES:Rescheduling_interrupts
    339442 ±  4%     +29.6%     439760 ± 11%  interrupts.CPU88.LOC:Local_timer_interrupts
    177.33 ± 28%     -43.4%     100.33 ± 20%  interrupts.CPU88.RES:Rescheduling_interrupts
    489.83 ±  4%      +9.7%     537.17 ±  4%  interrupts.CPU89.CAL:Function_call_interrupts
    339993 ±  4%     +29.4%     439881 ± 11%  interrupts.CPU89.LOC:Local_timer_interrupts
    142.17 ±  9%     -42.4%      81.83 ± 27%  interrupts.CPU89.RES:Rescheduling_interrupts
    340221 ±  4%     +29.3%     439787 ± 11%  interrupts.CPU9.LOC:Local_timer_interrupts
    340036 ±  4%     +29.3%     439661 ± 11%  interrupts.CPU90.LOC:Local_timer_interrupts
      6760 ± 20%     -39.7%       4074 ± 44%  interrupts.CPU90.NMI:Non-maskable_interrupts
      6760 ± 20%     -39.7%       4074 ± 44%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
    488.00 ±  3%     +11.0%     541.67 ±  5%  interrupts.CPU91.CAL:Function_call_interrupts
    339776 ±  4%     +29.4%     439757 ± 11%  interrupts.CPU91.LOC:Local_timer_interrupts
    146.83 ±  6%     -40.2%      87.83 ± 30%  interrupts.CPU91.RES:Rescheduling_interrupts
    485.83 ±  2%     +13.6%     551.83 ±  4%  interrupts.CPU92.CAL:Function_call_interrupts
    339958 ±  4%     +29.4%     439797 ± 11%  interrupts.CPU92.LOC:Local_timer_interrupts
    492.83 ±  3%     +15.7%     570.17 ±  8%  interrupts.CPU93.CAL:Function_call_interrupts
    339955 ±  4%     +29.3%     439696 ± 11%  interrupts.CPU93.LOC:Local_timer_interrupts
    339815 ±  4%     +29.4%     439689 ± 11%  interrupts.CPU94.LOC:Local_timer_interrupts
    156.33 ± 21%     -28.3%     112.17 ± 22%  interrupts.CPU94.RES:Rescheduling_interrupts
    339794 ±  4%     +29.4%     439829 ± 11%  interrupts.CPU95.LOC:Local_timer_interrupts
    183.17 ± 16%     -39.2%     111.33 ± 25%  interrupts.CPU95.RES:Rescheduling_interrupts
  32641163 ±  4%     +29.3%   42213313 ± 11%  interrupts.LOC:Local_timer_interrupts
     23461 ±  5%     -16.4%      19609 ±  7%  interrupts.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-csl-2sp8: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/debian-x86_64-phoronix/lkp-csl-2sp8/build-apache-1.4.0/phoronix-test-suite/0x5002f01

commit: 
  46132e3ac5 ("sched: nohz: stop passing around unused "ticks" parameter.")
  3edecfef02 ("sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal")

46132e3ac58cb2ee 3edecfef028536cb19a120ec878 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     14.63            +1.8%      14.89        phoronix-test-suite.build-apache.0.seconds



***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/300s/8T/lkp-skl-fpga01/anon-w-seq-mt/vm-scalability/0x2006906

commit: 
  46132e3ac5 ("sched: nohz: stop passing around unused "ticks" parameter.")
  3edecfef02 ("sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal")

46132e3ac58cb2ee 3edecfef028536cb19a120ec878 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4           -0%           0:4     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      0.01            +8.0%       0.01 ±  4%  vm-scalability.free_time
      3.38 ± 12%      +5.3        8.66 ± 12%  vm-scalability.median_stddev%
      3.26 ± 15%      +7.1       10.34 ±  7%  vm-scalability.stddev%
  56210450            +7.3%   60334907        vm-scalability.throughput
    265111 ±  8%     -60.1%     105700 ±  8%  vm-scalability.time.involuntary_context_switches
  13438401            -5.5%   12695434        vm-scalability.time.minor_page_faults
      7915           -17.9%       6496 ±  2%  vm-scalability.time.percent_of_cpu_this_job_got
     10648           -17.2%       8821        vm-scalability.time.system_time
     13250           -18.3%      10831 ±  2%  vm-scalability.time.user_time
    642704 ±  2%     -19.1%     519971 ±  3%  vm-scalability.time.voluntary_context_switches
  1.45e+10            -5.5%  1.369e+10        vm-scalability.workload
  10522087 ±  4%     -10.9%    9377010 ±  8%  numa-numastat.node0.local_node
  10538954 ±  3%     -10.8%    9398858 ±  8%  numa-numastat.node0.numa_hit
   4254139 ±  5%     -25.5%    3168793 ±  7%  cpuidle.C1.time
    130285 ± 18%     +63.3%     212818 ±  7%  cpuidle.C1.usage
   9419337 ± 14%     +93.4%   18214338 ± 19%  cpuidle.C1E.usage
     44700 ± 32%     +89.7%      84794 ± 15%  cpuidle.POLL.usage
     19.70 ±  4%     +13.9       33.63 ±  4%  mpstat.cpu.all.idle%
      2.96            -0.5        2.51 ±  2%  mpstat.cpu.all.irq%
      0.10 ±  4%      -0.0        0.09 ±  4%  mpstat.cpu.all.soft%
     34.80            -5.7       29.09        mpstat.cpu.all.sys%
     42.44            -7.8       34.68 ±  2%  mpstat.cpu.all.usr%
     19.75 ±  5%     +70.9%      33.75 ±  4%  vmstat.cpu.id
     41.75           -19.2%      33.75 ±  2%  vmstat.cpu.us
     85.75 ±  2%     -20.4%      68.25 ±  3%  vmstat.procs.r
     11715           -29.6%       8243 ±  2%  vmstat.system.cs
    218962            -1.6%     215402        vmstat.system.in
  43452071           -23.2%   33387811 ±  3%  meminfo.Active
  43451815           -23.2%   33387560 ±  3%  meminfo.Active(anon)
  35423963           -22.8%   27358835 ±  2%  meminfo.AnonHugePages
  43350866           -23.2%   33286318 ±  3%  meminfo.AnonPages
  85460025           -17.7%   70295644 ±  2%  meminfo.Committed_AS
  45916525           -22.0%   35824530 ±  2%  meminfo.Memused
     79322           -22.4%      61551        meminfo.PageTables
  22195752           -28.8%   15805718 ± 13%  numa-meminfo.node0.Active
  22195602           -28.8%   15805578 ± 13%  numa-meminfo.node0.Active(anon)
  18149414 ±  2%     -28.8%   12930701 ± 13%  numa-meminfo.node0.AnonHugePages
  22167733           -28.9%   15766874 ± 13%  numa-meminfo.node0.AnonPages
  23410954           -27.2%   17036704 ± 12%  numa-meminfo.node0.MemUsed
     40344 ±  3%     -28.2%      28973 ± 11%  numa-meminfo.node0.PageTables
     39297 ±  6%     -17.5%      32428 ±  9%  numa-meminfo.node1.PageTables
      2983 ±  7%     +19.4%       3563 ±  3%  slabinfo.PING.active_objs
      2983 ±  7%     +19.4%       3563 ±  3%  slabinfo.PING.num_objs
     68005            +9.9%      74707 ±  4%  slabinfo.filp.active_objs
      2138            +9.8%       2347 ±  3%  slabinfo.filp.active_slabs
     68454            +9.8%      75141 ±  3%  slabinfo.filp.num_objs
      2138            +9.8%       2347 ±  3%  slabinfo.filp.num_slabs
      2163 ±  2%     +15.4%       2495 ±  4%  slabinfo.khugepaged_mm_slot.active_objs
      2163 ±  2%     +15.4%       2495 ±  4%  slabinfo.khugepaged_mm_slot.num_objs
   5580820 ±  2%     -29.5%    3935820 ± 12%  numa-vmstat.node0.nr_active_anon
   5573129 ±  2%     -29.6%    3925922 ± 12%  numa-vmstat.node0.nr_anon_pages
      8850 ±  2%     -29.1%       6274 ± 12%  numa-vmstat.node0.nr_anon_transparent_hugepages
     10055 ±  3%     -28.5%       7192 ±  9%  numa-vmstat.node0.nr_page_table_pages
   5578984 ±  2%     -29.5%    3934948 ± 12%  numa-vmstat.node0.nr_zone_active_anon
   5904903 ±  3%     -11.1%    5251633 ±  5%  numa-vmstat.node0.numa_hit
   5854019 ±  3%     -11.7%    5167325 ±  6%  numa-vmstat.node0.numa_local
   5410137 ±  4%     -19.7%    4344194 ± 13%  numa-vmstat.node1.nr_active_anon
   5392926 ±  4%     -19.7%    4327994 ± 13%  numa-vmstat.node1.nr_anon_pages
      8511 ±  5%     -18.8%       6909 ± 13%  numa-vmstat.node1.nr_anon_transparent_hugepages
      9838 ±  6%     -19.1%       7959 ± 10%  numa-vmstat.node1.nr_page_table_pages
   5408962 ±  4%     -19.7%    4343650 ± 13%  numa-vmstat.node1.nr_zone_active_anon
  10810272 ±  2%     -22.9%    8338377 ±  3%  proc-vmstat.nr_active_anon
  10785903 ±  2%     -23.0%    8307576 ±  3%  proc-vmstat.nr_anon_pages
     17152           -22.3%      13324 ±  3%  proc-vmstat.nr_anon_transparent_hugepages
    416.25           +19.1%     495.75        proc-vmstat.nr_dirtied
   3746495            +6.6%    3995298        proc-vmstat.nr_dirty_background_threshold
   7502150            +6.6%    8000365        proc-vmstat.nr_dirty_threshold
  37755991            +6.6%   40234900        proc-vmstat.nr_free_pages
     30461            +1.5%      30909        proc-vmstat.nr_inactive_anon
    498.50            +8.6%     541.25        proc-vmstat.nr_inactive_file
     19383            -1.6%      19063        proc-vmstat.nr_kernel_stack
     10889            +3.6%      11278        proc-vmstat.nr_mapped
     19703 ±  2%     -22.3%      15308 ±  3%  proc-vmstat.nr_page_table_pages
     27773            +2.1%      28351        proc-vmstat.nr_slab_reclaimable
    384.00           +19.1%     457.25        proc-vmstat.nr_written
  10810258 ±  2%     -22.9%    8338369 ±  3%  proc-vmstat.nr_zone_active_anon
     30461            +1.5%      30909        proc-vmstat.nr_zone_inactive_anon
    498.50            +8.6%     541.25        proc-vmstat.nr_zone_inactive_file
      4107 ± 53%    +150.0%      10266 ± 44%  proc-vmstat.numa_hint_faults
      1942 ± 30%    +168.4%       5213 ± 31%  proc-vmstat.numa_hint_faults_local
  20506304            -5.3%   19420736        proc-vmstat.numa_hit
     17544           -14.3%      15040 ±  3%  proc-vmstat.numa_huge_pte_updates
  20472400            -5.3%   19386859        proc-vmstat.numa_local
   9028959           -14.1%    7754371 ±  3%  proc-vmstat.numa_pte_updates
     15121 ±  3%     -17.1%      12528 ±  7%  proc-vmstat.pgactivate
 3.229e+09            -5.5%   3.05e+09        proc-vmstat.pgalloc_normal
  14292546            -5.0%   13576339        proc-vmstat.pgfault
 3.229e+09            -5.6%  3.047e+09        proc-vmstat.pgfree
   6278434            -5.5%    5930869        proc-vmstat.thp_fault_alloc
 4.985e+10            -5.3%   4.72e+10        perf-stat.i.branch-instructions
     74.25            -6.3       67.92        perf-stat.i.cache-miss-rate%
 5.939e+08           -12.5%  5.197e+08 ±  2%  perf-stat.i.cache-misses
 7.946e+08            -5.5%  7.507e+08        perf-stat.i.cache-references
     11837           -30.6%       8214        perf-stat.i.context-switches
      1.41           -12.3%       1.24        perf-stat.i.cpi
 2.208e+11           -16.2%  1.851e+11 ±  2%  perf-stat.i.cpu-cycles
    322.82 ±  2%     -13.5%     279.12        perf-stat.i.cpu-migrations
    376.88            -3.5%     363.72        perf-stat.i.cycles-between-cache-misses
    489941 ±  3%      -8.6%     448035 ±  4%  perf-stat.i.dTLB-load-misses
 3.923e+10            -5.4%  3.711e+10        perf-stat.i.dTLB-loads
      0.00            -0.0        0.00        perf-stat.i.dTLB-store-miss-rate%
    373101           -11.4%     330388 ±  2%  perf-stat.i.dTLB-store-misses
 1.142e+10            -5.4%   1.08e+10        perf-stat.i.dTLB-stores
     32.70 ±  2%     -13.3       19.39 ±  4%  perf-stat.i.iTLB-load-miss-rate%
    309009           -20.7%     245173 ±  2%  perf-stat.i.iTLB-load-misses
    769782           +61.3%    1241809 ±  2%  perf-stat.i.iTLB-loads
 1.562e+11            -5.2%  1.481e+11        perf-stat.i.instructions
    524937 ±  2%     +19.2%     625485        perf-stat.i.instructions-per-iTLB-miss
      0.71           +15.5%       0.82        perf-stat.i.ipc
      2.12           -16.6%       1.77 ±  2%  perf-stat.i.metric.GHz
      1.03            -5.5%       0.97        perf-stat.i.metric.K/sec
    972.02            -5.7%     916.33        perf-stat.i.metric.M/sec
     47339            -5.4%      44776        perf-stat.i.minor-faults
  78879346            -8.5%   72186653 ±  3%  perf-stat.i.node-stores
     47339            -5.4%      44776        perf-stat.i.page-faults
      0.02 ±  2%      +0.0        0.02        perf-stat.overall.branch-miss-rate%
     74.75            -5.6       69.19        perf-stat.overall.cache-miss-rate%
      1.41           -11.6%       1.25        perf-stat.overall.cpi
    371.74            -4.2%     356.07        perf-stat.overall.cycles-between-cache-misses
      0.00            -0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     28.67           -12.2       16.49 ±  3%  perf-stat.overall.iTLB-load-miss-rate%
    506889 ±  2%     +19.0%     603033        perf-stat.overall.instructions-per-iTLB-miss
      0.71           +13.2%       0.80        perf-stat.overall.ipc
 4.952e+10            -5.4%  4.685e+10        perf-stat.ps.branch-instructions
 5.899e+08           -12.6%  5.157e+08 ±  2%  perf-stat.ps.cache-misses
 7.892e+08            -5.6%  7.453e+08        perf-stat.ps.cache-references
     11688           -29.9%       8197 ±  2%  perf-stat.ps.context-switches
 2.193e+11           -16.3%  1.836e+11 ±  2%  perf-stat.ps.cpu-cycles
    320.64 ±  2%     -13.7%     276.72        perf-stat.ps.cpu-migrations
    485051 ±  3%      -8.0%     446114 ±  4%  perf-stat.ps.dTLB-load-misses
 3.896e+10            -5.5%  3.683e+10        perf-stat.ps.dTLB-loads
    370445           -11.3%     328471 ±  2%  perf-stat.ps.dTLB-store-misses
 1.134e+10            -5.4%  1.072e+10        perf-stat.ps.dTLB-stores
    306250           -20.4%     243927 ±  2%  perf-stat.ps.iTLB-load-misses
    761977           +62.2%    1235784 ±  2%  perf-stat.ps.iTLB-loads
 1.552e+11            -5.3%   1.47e+11        perf-stat.ps.instructions
     47017            -5.3%      44532        perf-stat.ps.minor-faults
  78362383            -8.6%   71613722 ±  3%  perf-stat.ps.node-stores
     47017            -5.3%      44532        perf-stat.ps.page-faults
 4.689e+13            -4.9%  4.458e+13        perf-stat.total.instructions
     87937 ± 37%     -72.9%      23825 ± 65%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   8567806 ± 39%     -77.2%    1949852 ± 58%  sched_debug.cfs_rq:/.MIN_vruntime.max
    858434 ± 38%     -75.7%     209012 ± 58%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    116114           -21.5%      91139 ± 10%  sched_debug.cfs_rq:/.exec_clock.avg
    112872           -31.3%      77511 ± 11%  sched_debug.cfs_rq:/.exec_clock.min
      1639 ± 36%    +431.1%       8707 ± 22%  sched_debug.cfs_rq:/.exec_clock.stddev
      6473 ± 30%     -73.5%       1713 ± 68%  sched_debug.cfs_rq:/.load.min
      6.17 ± 16%     -71.1%       1.78 ± 76%  sched_debug.cfs_rq:/.load_avg.min
     87.37 ± 11%     +21.7%     106.37 ±  5%  sched_debug.cfs_rq:/.load_avg.stddev
     87938 ± 37%     -72.9%      23825 ± 65%  sched_debug.cfs_rq:/.max_vruntime.avg
   8567868 ± 39%     -77.2%    1949852 ± 58%  sched_debug.cfs_rq:/.max_vruntime.max
    858441 ± 38%     -75.7%     209012 ± 58%  sched_debug.cfs_rq:/.max_vruntime.stddev
  12384969           -39.1%    7540410 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
  12700396           -33.7%    8421155 ± 11%  sched_debug.cfs_rq:/.min_vruntime.max
  11776367           -44.9%    6487882 ± 13%  sched_debug.cfs_rq:/.min_vruntime.min
    192433 ± 19%    +224.6%     624659 ± 22%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.84 ±  3%     -39.5%       0.51 ±  4%  sched_debug.cfs_rq:/.nr_running.avg
      1.75 ± 10%     -27.1%       1.28 ± 14%  sched_debug.cfs_rq:/.nr_running.max
      0.67 ± 30%     -73.8%       0.17 ± 67%  sched_debug.cfs_rq:/.nr_running.min
     79.73 ±  3%     -76.0%      19.11 ± 28%  sched_debug.cfs_rq:/.nr_spread_over.avg
    461.38 ± 12%     -65.9%     157.42 ± 38%  sched_debug.cfs_rq:/.nr_spread_over.max
    109.97 ±  4%     -73.6%      29.00 ± 35%  sched_debug.cfs_rq:/.nr_spread_over.stddev
     80.00 ±  7%     +90.7%     152.53 ± 25%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     10.58 ± 21%     +73.0%      18.31 ± 26%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     80.00 ±  7%     +90.7%     152.53 ± 25%  sched_debug.cfs_rq:/.removed.util_avg.max
     10.58 ± 21%     +72.9%      18.30 ± 26%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    895.32           -45.1%     491.66 ±  9%  sched_debug.cfs_rq:/.runnable_avg.avg
      1932 ±  4%     -32.8%       1298 ± 13%  sched_debug.cfs_rq:/.runnable_avg.max
    692.46 ± 18%     -75.9%     166.95 ± 73%  sched_debug.cfs_rq:/.runnable_avg.min
    -98017          -464.3%     357113 ± 59%  sched_debug.cfs_rq:/.spread0.avg
    219601 ± 40%    +463.9%    1238325 ± 25%  sched_debug.cfs_rq:/.spread0.max
    192966 ± 19%    +223.7%     624595 ± 22%  sched_debug.cfs_rq:/.spread0.stddev
    870.55           -44.9%     479.80 ±  7%  sched_debug.cfs_rq:/.util_avg.avg
      1351 ±  6%     -18.0%       1108 ±  5%  sched_debug.cfs_rq:/.util_avg.max
    485.04 ± 23%     -75.7%     118.10 ± 80%  sched_debug.cfs_rq:/.util_avg.min
    117.61 ± 21%    +108.9%     245.68 ± 20%  sched_debug.cfs_rq:/.util_avg.stddev
    188.04 ± 32%     +42.6%     268.17 ± 16%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      1266 ±  4%     -26.0%     937.65 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.max
    194821 ±  6%     -23.5%     149100 ± 16%  sched_debug.cpu.avg_idle.stddev
     26.39 ± 10%     -63.4%       9.67 ± 36%  sched_debug.cpu.clock.stddev
    683.36 ±  2%     +13.8%     777.64 ±  7%  sched_debug.cpu.clock_task.stddev
     12549 ±  4%     -43.6%       7082 ±  6%  sched_debug.cpu.curr->pid.avg
    688.38 ± 32%     -77.3%     156.39 ± 62%  sched_debug.cpu.curr->pid.min
      0.00 ± 16%     -64.0%       0.00 ± 30%  sched_debug.cpu.next_balance.stddev
      0.86 ±  3%     -41.1%       0.50 ±  6%  sched_debug.cpu.nr_running.avg
      2.08 ±  3%     -30.4%       1.45 ± 16%  sched_debug.cpu.nr_running.max
      0.67 ± 30%     -73.8%       0.17 ± 67%  sched_debug.cpu.nr_running.min
     17246           -27.6%      12481 ±  9%  sched_debug.cpu.nr_switches.avg
      9381           -21.0%       7416 ± 12%  sched_debug.cpu.nr_switches.min
      8758 ±  7%     -20.8%       6938 ± 12%  sched_debug.cpu.nr_switches.stddev
    -66.08           -52.8%     -31.19        sched_debug.cpu.nr_uninterruptible.min
     27.79 ± 20%     -38.8%      17.00 ± 23%  sched_debug.cpu.nr_uninterruptible.stddev
     15920           -30.8%      11015 ± 10%  sched_debug.cpu.sched_count.avg
      8690 ±  2%     -22.4%       6743 ± 13%  sched_debug.cpu.sched_count.min
      8150 ±  7%     -20.4%       6489 ± 14%  sched_debug.cpu.sched_count.stddev
     10908 ± 12%     +78.3%      19451 ± 23%  sched_debug.cpu.sched_goidle.max
      1265 ±  8%    +115.7%       2730 ± 16%  sched_debug.cpu.sched_goidle.stddev
      7617           -31.0%       5258 ± 10%  sched_debug.cpu.ttwu_count.avg
      5158 ±  7%     -29.7%       3625 ±  7%  sched_debug.cpu.ttwu_count.stddev
      3145 ±  2%     -65.8%       1076 ± 15%  sched_debug.cpu.ttwu_local.avg
     20472 ±  7%     -32.5%      13814 ± 17%  sched_debug.cpu.ttwu_local.max
    511.29 ±  4%     -16.8%     425.39 ± 11%  sched_debug.cpu.ttwu_local.min
      3688 ±  7%     -54.8%       1666 ± 18%  sched_debug.cpu.ttwu_local.stddev
     79.71 ±  3%      -7.9       71.85        perf-profile.calltrace.cycles-pp.do_access
     44.86 ±  3%      -5.8       39.07 ±  2%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     44.74 ±  3%      -5.8       38.95 ±  2%  perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
     42.57 ±  3%      -5.5       37.03 ±  2%  perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
     37.92 ±  3%      -5.2       32.68 ±  2%  perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault
     40.26 ±  3%      -5.2       35.06        perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
     38.77 ±  3%      -2.9       35.84 ±  2%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.do_access
     39.00 ±  3%      -2.0       37.04 ±  2%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.do_access
     39.05 ±  3%      -2.0       37.09 ±  2%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault.do_access
     38.98 ±  3%      -2.0       37.02 ±  2%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.do_access
      0.97 ±  5%      -0.1        0.85 ±  2%  perf-profile.calltrace.cycles-pp.___might_sleep.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.69 ±  3%      +0.3        1.03 ±  6%  perf-profile.calltrace.cycles-pp.write
      0.68 ±  3%      +0.3        1.03 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      0.68 ±  3%      +0.3        1.03 ±  6%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.68 ±  3%      +0.3        1.03 ±  7%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.68 ±  3%      +0.3        1.03 ±  7%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.68 ±  3%      +0.3        1.03 ±  7%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.65 ±  3%      +0.3        1.00 ±  7%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.cold.new_sync_write
      0.66 ±  3%      +0.4        1.01 ±  7%  perf-profile.calltrace.cycles-pp.devkmsg_write.cold.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.65 ±  3%      +0.4        1.01 ±  7%  perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.cold.new_sync_write.vfs_write
      0.65 ±  3%      +0.4        1.01 ±  7%  perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.cold.new_sync_write.vfs_write.ksys_write
      0.76 ±  4%      +0.4        1.20 ±  7%  perf-profile.calltrace.cycles-pp.memcpy_erms.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread
      0.80 ±  4%      +0.4        1.25 ±  7%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.80 ±  4%      +0.4        1.25 ±  7%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.79 ±  5%      +0.4        1.23 ±  7%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.78 ±  4%      +0.4        1.23 ±  7%  perf-profile.calltrace.cycles-pp.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.79 ±  5%      +0.4        1.23 ±  7%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.25 ±100%      +0.6        0.81 ±  8%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.cold
      0.00            +0.6        0.59 ±  7%  perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
      0.00            +0.7        0.65 ±  7%  perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
      0.00            +0.7        0.65 ±  7%  perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
      0.00            +0.7        0.74 ±  8%  perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit
      1.03 ±  7%      +7.2        8.23 ± 10%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.15 ±  7%      +7.3        8.47 ±  9%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      1.16 ±  7%      +7.3        8.49 ±  9%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.21 ±  7%      +7.4        8.58 ±  9%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      1.21 ±  7%      +7.4        8.58 ±  9%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.21 ±  7%      +7.4        8.58 ±  9%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      1.22 ±  7%      +7.5        8.70 ±  9%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     44.91 ±  3%      -5.8       39.11 ±  2%  perf-profile.children.cycles-pp.handle_mm_fault
     44.87 ±  3%      -5.8       39.08 ±  2%  perf-profile.children.cycles-pp.__handle_mm_fault
     44.74 ±  3%      -5.8       38.95 ±  2%  perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page
     43.20 ±  3%      -5.6       37.56 ±  2%  perf-profile.children.cycles-pp.clear_huge_page
     67.98 ±  3%      -5.6       62.41        perf-profile.children.cycles-pp.do_access
     38.34 ±  3%      -5.3       33.06 ±  2%  perf-profile.children.cycles-pp.clear_page_erms
     40.49 ±  3%      -5.2       35.26 ±  2%  perf-profile.children.cycles-pp.clear_subpage
     45.23 ±  3%      -4.9       40.38 ±  3%  perf-profile.children.cycles-pp.asm_exc_page_fault
     45.17 ±  3%      -4.8       40.32 ±  3%  perf-profile.children.cycles-pp.exc_page_fault
     45.13 ±  3%      -4.8       40.30 ±  3%  perf-profile.children.cycles-pp.do_user_addr_fault
     35.89 ±  2%      -2.4       33.45 ±  2%  perf-profile.children.cycles-pp.do_rw_once
      1.36 ±  8%      -0.3        1.10 ± 11%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.84 ±  7%      -0.3        1.59 ±  8%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      1.61 ±  8%      -0.2        1.38 ±  9%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      1.58 ±  8%      -0.2        1.35 ±  9%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.31 ±  5%      -0.2        1.08 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      1.02 ± 10%      -0.2        0.80 ± 10%  perf-profile.children.cycles-pp.tick_sched_handle
      1.00 ± 10%      -0.2        0.79 ± 10%  perf-profile.children.cycles-pp.update_process_times
      1.08 ± 10%      -0.2        0.87 ± 11%  perf-profile.children.cycles-pp.tick_sched_timer
      0.66 ± 10%      -0.1        0.51 ±  9%  perf-profile.children.cycles-pp.scheduler_tick
      0.48 ± 11%      -0.1        0.37 ± 11%  perf-profile.children.cycles-pp.task_tick_fair
      0.95 ±  2%      -0.1        0.85 ±  3%  perf-profile.children.cycles-pp._cond_resched
      0.20 ± 10%      -0.0        0.16 ± 10%  perf-profile.children.cycles-pp.update_load_avg
      0.15 ±  4%      -0.0        0.11 ± 11%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.15 ±  2%      -0.0        0.11 ± 11%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.14            -0.0        1.10 ±  2%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.12 ±  3%      -0.0        0.09 ± 11%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.18 ±  4%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.pte_alloc_one
      0.14 ±  3%      -0.0        0.11        perf-profile.children.cycles-pp.mem_cgroup_charge
      0.12 ±  3%      -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.11 ±  4%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.lru_cache_add
      0.10            -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.08 ±  5%      -0.0        0.06        perf-profile.children.cycles-pp.__perf_sw_event
      0.08 ±  6%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.___perf_sw_event
      0.08 ±  5%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.page_add_new_anon_rmap
      0.07 ±  5%      -0.0        0.05 ±  9%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.07 ±  5%      -0.0        0.05 ±  9%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.09            -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.08 ±  5%      -0.0        0.07        perf-profile.children.cycles-pp.try_charge
      0.11 ± 10%      +0.0        0.15 ±  6%  perf-profile.children.cycles-pp.ktime_get
      0.10 ±  8%      +0.1        0.17 ± 12%  perf-profile.children.cycles-pp.__free_pages_ok
      0.14 ±  8%      +0.1        0.22 ±  9%  perf-profile.children.cycles-pp.release_pages
      0.01 ±173%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.io_serial_out
      0.13 ± 11%      +0.1        0.21 ± 10%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.13 ± 11%      +0.1        0.21 ± 10%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.12 ± 14%      +0.1        0.21 ± 15%  perf-profile.children.cycles-pp.__munmap
      0.15 ±  8%      +0.1        0.25 ± 11%  perf-profile.children.cycles-pp.unmap_region
      0.16 ±  7%      +0.1        0.26 ± 10%  perf-profile.children.cycles-pp.__vm_munmap
      0.16 ±  7%      +0.1        0.26 ± 10%  perf-profile.children.cycles-pp.__x64_sys_munmap
      0.16 ±  7%      +0.1        0.26 ± 10%  perf-profile.children.cycles-pp.__do_munmap
      0.27 ±  3%      +0.1        0.39 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.12 ± 39%  perf-profile.children.cycles-pp.start_kernel
      0.25 ±  4%      +0.1        0.37 ±  8%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.43 ±  3%      +0.2        0.67 ±  7%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.40 ±  3%      +0.3        0.67 ±  7%  perf-profile.children.cycles-pp.io_serial_in
      0.46 ±  4%      +0.3        0.73 ±  7%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.46 ±  3%      +0.3        0.76 ±  7%  perf-profile.children.cycles-pp.uart_console_write
      0.51 ±  4%      +0.3        0.82 ±  7%  perf-profile.children.cycles-pp.serial8250_console_write
      0.69 ±  3%      +0.3        1.03 ±  6%  perf-profile.children.cycles-pp.write
      0.68 ±  3%      +0.3        1.02 ±  7%  perf-profile.children.cycles-pp.console_unlock
      0.66 ±  3%      +0.4        1.01 ±  7%  perf-profile.children.cycles-pp.devkmsg_write.cold
      0.65 ±  3%      +0.4        1.01 ±  7%  perf-profile.children.cycles-pp.devkmsg_emit
      0.69 ±  3%      +0.4        1.05 ±  6%  perf-profile.children.cycles-pp.new_sync_write
      0.68 ±  3%      +0.4        1.04 ±  6%  perf-profile.children.cycles-pp.vprintk_emit
      0.69 ±  3%      +0.4        1.05 ±  6%  perf-profile.children.cycles-pp.ksys_write
      0.69 ±  3%      +0.4        1.05 ±  6%  perf-profile.children.cycles-pp.vfs_write
      1.01 ±  2%      +0.4        1.44 ±  5%  perf-profile.children.cycles-pp.do_syscall_64
      1.01 ±  2%      +0.4        1.44 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.80 ±  4%      +0.4        1.25 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
      0.80 ±  4%      +0.4        1.25 ±  7%  perf-profile.children.cycles-pp.kthread
      0.79 ±  5%      +0.4        1.23 ±  7%  perf-profile.children.cycles-pp.worker_thread
      0.78 ±  4%      +0.4        1.23 ±  7%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      0.79 ±  5%      +0.4        1.23 ±  7%  perf-profile.children.cycles-pp.process_one_work
      0.78 ±  4%      +0.4        1.23 ±  7%  perf-profile.children.cycles-pp.memcpy_erms
      0.06 ±173%      +1.0        1.02 ±116%  perf-profile.children.cycles-pp.osq_lock
      0.06 ±173%      +1.0        1.07 ±115%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.06 ±173%      +1.0        1.09 ±112%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      1.04 ±  8%      +7.3        8.34 ± 10%  perf-profile.children.cycles-pp.intel_idle
      1.21 ±  7%      +7.4        8.58 ±  9%  perf-profile.children.cycles-pp.start_secondary
      1.17 ±  7%      +7.4        8.61 ±  9%  perf-profile.children.cycles-pp.cpuidle_enter
      1.17 ±  7%      +7.4        8.61 ±  9%  perf-profile.children.cycles-pp.cpuidle_enter_state
      1.22 ±  7%      +7.5        8.70 ±  9%  perf-profile.children.cycles-pp.secondary_startup_64
      1.22 ±  7%      +7.5        8.70 ±  9%  perf-profile.children.cycles-pp.cpu_startup_entry
      1.22 ±  7%      +7.5        8.70 ±  9%  perf-profile.children.cycles-pp.do_idle
     37.48 ±  3%      -5.1       32.36 ±  2%  perf-profile.self.cycles-pp.clear_page_erms
     34.69 ±  2%      -2.2       32.45 ±  2%  perf-profile.self.cycles-pp.do_rw_once
     15.76 ±  4%      -1.0       14.78 ±  3%  perf-profile.self.cycles-pp.do_access
      1.07 ±  5%      -0.2        0.89 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      1.07 ±  3%      -0.2        0.91 ±  2%  perf-profile.self.cycles-pp.clear_huge_page
      0.56 ±  2%      -0.1        0.44 ±  6%  perf-profile.self.cycles-pp.rmqueue
      0.69 ±  3%      -0.1        0.60 ±  3%  perf-profile.self.cycles-pp._cond_resched
      0.12 ±  7%      -0.0        0.10 ±  9%  perf-profile.self.cycles-pp.update_load_avg
      0.10 ±  4%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.prepare_exit_to_usermode
      0.07 ±  6%      -0.0        0.05        perf-profile.self.cycles-pp.page_counter_try_charge
      0.09            -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.10 ± 11%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.ktime_get
      0.10 ±  5%      +0.1        0.15 ± 11%  perf-profile.self.cycles-pp.__free_pages_ok
      0.01 ±173%      +0.1        0.10 ± 17%  perf-profile.self.cycles-pp.io_serial_out
      0.25 ±  3%      +0.1        0.37 ±  8%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.40 ±  3%      +0.3        0.67 ±  7%  perf-profile.self.cycles-pp.io_serial_in
      0.77 ±  4%      +0.5        1.22 ±  7%  perf-profile.self.cycles-pp.memcpy_erms
      0.06 ±173%      +1.0        1.02 ±116%  perf-profile.self.cycles-pp.osq_lock
      1.04 ±  8%      +7.3        8.34 ± 10%  perf-profile.self.cycles-pp.intel_idle
     39227 ±  3%     -51.1%      19191 ±  2%  softirqs.CPU0.RCU
     13738 ±  5%     +43.4%      19703 ±  6%  softirqs.CPU0.SCHED
     36668 ±  4%     -50.2%      18261 ±  8%  softirqs.CPU1.RCU
     12281 ±  7%     +46.8%      18031 ±  9%  softirqs.CPU1.SCHED
     35606 ±  5%     -51.6%      17236 ±  9%  softirqs.CPU10.RCU
     12099 ±  4%     +36.4%      16498 ±  4%  softirqs.CPU10.SCHED
     34167 ±  3%     -50.7%      16828 ±  8%  softirqs.CPU100.RCU
     11294 ±  4%     +49.4%      16874 ±  9%  softirqs.CPU100.SCHED
     34521 ±  3%     -52.8%      16294 ±  9%  softirqs.CPU101.RCU
     10894 ±  7%     +56.9%      17090 ± 14%  softirqs.CPU101.SCHED
     34503 ±  4%     -51.5%      16744 ±  7%  softirqs.CPU102.RCU
     11268 ±  8%     +51.2%      17036 ± 13%  softirqs.CPU102.SCHED
     34224 ±  3%     -50.0%      17098 ±  9%  softirqs.CPU103.RCU
     11616 ±  3%     +49.1%      17324 ± 15%  softirqs.CPU103.SCHED
     35595 ±  5%     -50.7%      17535 ±  9%  softirqs.CPU11.RCU
     12003 ±  6%     +43.7%      17246 ±  7%  softirqs.CPU11.SCHED
     35592 ±  5%     -50.9%      17478 ±  9%  softirqs.CPU12.RCU
     11978 ±  8%     +44.4%      17291 ± 10%  softirqs.CPU12.SCHED
     35998 ±  4%     -50.3%      17902 ± 13%  softirqs.CPU13.RCU
     11929 ±  6%     +44.6%      17251 ± 15%  softirqs.CPU13.SCHED
     35709 ±  4%     -51.2%      17426 ±  8%  softirqs.CPU14.RCU
     11729 ±  8%     +47.2%      17263 ±  9%  softirqs.CPU14.SCHED
     36781 ±  3%     -50.5%      18219 ± 10%  softirqs.CPU15.RCU
     12002 ±  5%     +49.2%      17903 ± 13%  softirqs.CPU15.SCHED
     37007 ±  3%     -52.4%      17625 ±  8%  softirqs.CPU16.RCU
     11820 ±  7%     +40.9%      16656 ±  7%  softirqs.CPU16.SCHED
     37314 ±  3%     -52.1%      17862 ±  9%  softirqs.CPU17.RCU
     11778 ±  8%     +53.3%      18057 ± 10%  softirqs.CPU17.SCHED
     36941 ±  3%     -51.0%      18098 ±  7%  softirqs.CPU18.RCU
     11864 ± 10%     +49.8%      17773 ±  9%  softirqs.CPU18.SCHED
     36746 ±  4%     -50.6%      18158 ±  9%  softirqs.CPU19.RCU
     12175 ±  6%     +39.8%      17026 ± 15%  softirqs.CPU19.SCHED
     36794 ±  5%     -50.7%      18134 ± 10%  softirqs.CPU2.RCU
     12205 ±  5%     +42.0%      17329 ± 10%  softirqs.CPU2.SCHED
     37148 ±  4%     -51.9%      17873 ±  8%  softirqs.CPU20.RCU
     11958 ±  6%     +39.8%      16719 ± 10%  softirqs.CPU20.SCHED
     36866 ±  4%     -51.7%      17812 ±  9%  softirqs.CPU21.RCU
     11868 ±  7%     +47.2%      17467 ±  7%  softirqs.CPU21.SCHED
     37145 ±  3%     -52.0%      17815 ±  9%  softirqs.CPU22.RCU
     11998 ±  6%     +45.1%      17411 ± 12%  softirqs.CPU22.SCHED
     37098 ±  2%     -52.2%      17725 ±  9%  softirqs.CPU23.RCU
     11968 ±  5%     +46.2%      17492 ± 11%  softirqs.CPU23.SCHED
     37384 ±  4%     -53.0%      17552 ±  8%  softirqs.CPU24.RCU
     12034 ±  5%     +50.5%      18115 ± 10%  softirqs.CPU24.SCHED
     37017 ±  4%     -51.9%      17796 ±  9%  softirqs.CPU25.RCU
     12047 ±  5%     +39.3%      16782 ±  8%  softirqs.CPU25.SCHED
     33985 ±  8%     -53.9%      15655 ±  9%  softirqs.CPU26.RCU
     12177 ±  8%     +44.6%      17610 ±  5%  softirqs.CPU26.SCHED
     32818 ±  2%     -49.4%      16610 ± 15%  softirqs.CPU27.RCU
     11954 ±  5%     +41.4%      16901 ± 13%  softirqs.CPU27.SCHED
     32299 ±  3%     -51.5%      15655 ±  6%  softirqs.CPU28.RCU
     11815 ±  2%     +41.1%      16674 ±  6%  softirqs.CPU28.SCHED
     32897           -52.4%      15668 ±  7%  softirqs.CPU29.RCU
     11284           +32.9%      15002 ±  4%  softirqs.CPU29.SCHED
     36752 ±  5%     -51.4%      17857 ±  9%  softirqs.CPU3.RCU
     12000 ±  7%     +37.8%      16533 ±  9%  softirqs.CPU3.SCHED
     35961 ±  3%     -49.2%      18284 ± 10%  softirqs.CPU30.RCU
     12131 ± 10%     +39.8%      16961 ±  7%  softirqs.CPU30.SCHED
     35619 ±  3%     -48.1%      18482 ±  7%  softirqs.CPU31.RCU
     11580 ±  2%     +57.8%      18278 ± 20%  softirqs.CPU31.SCHED
    196.50 ± 21%   +4220.9%       8490 ±123%  softirqs.CPU32.NET_RX
     36027 ±  3%     -49.4%      18245 ±  5%  softirqs.CPU32.RCU
     11683           +44.3%      16855 ± 14%  softirqs.CPU32.SCHED
     35767 ±  3%     -50.6%      17651 ±  8%  softirqs.CPU33.RCU
     12103 ±  3%     +34.6%      16291 ±  6%  softirqs.CPU33.SCHED
     35444 ±  4%     -50.6%      17508 ±  8%  softirqs.CPU34.RCU
     11685           +41.4%      16527 ±  7%  softirqs.CPU34.SCHED
     35593 ±  3%     -50.0%      17794 ±  7%  softirqs.CPU35.RCU
     11408 ±  4%     +49.3%      17029 ± 12%  softirqs.CPU35.SCHED
     35583 ±  2%     -50.4%      17645 ±  8%  softirqs.CPU36.RCU
     11734 ±  5%     +47.8%      17348 ± 14%  softirqs.CPU36.SCHED
     35440 ±  3%     -49.9%      17744 ±  8%  softirqs.CPU37.RCU
     11223 ±  3%     +51.6%      17012 ± 11%  softirqs.CPU37.SCHED
     35564 ±  3%     -49.4%      18007 ± 10%  softirqs.CPU38.RCU
     11687 ±  3%     +47.0%      17178 ± 14%  softirqs.CPU38.SCHED
     34580 ±  6%     -49.6%      17437 ±  7%  softirqs.CPU39.RCU
     11539           +44.0%      16620 ±  8%  softirqs.CPU39.SCHED
     36070 ±  5%     -50.8%      17749 ±  9%  softirqs.CPU4.RCU
     11876 ±  6%     +52.9%      18159 ± 11%  softirqs.CPU4.SCHED
     35037 ±  4%     -50.0%      17520 ±  8%  softirqs.CPU40.RCU
     11258 ±  3%     +51.9%      17106 ± 12%  softirqs.CPU40.SCHED
     35834 ±  4%     -51.3%      17444 ±  9%  softirqs.CPU41.RCU
     11722 ±  2%     +36.2%      15968 ±  5%  softirqs.CPU41.SCHED
     35410 ±  3%     -50.0%      17722 ±  8%  softirqs.CPU42.RCU
     11372 ±  2%     +41.8%      16124 ±  4%  softirqs.CPU42.SCHED
     33792 ±  6%     -47.2%      17856 ±  9%  softirqs.CPU43.RCU
     12027           +42.3%      17112 ± 12%  softirqs.CPU43.SCHED
     36039 ±  4%     -50.4%      17888 ±  9%  softirqs.CPU44.RCU
     11549 ±  3%     +39.6%      16121 ±  7%  softirqs.CPU44.SCHED
     34809 ±  4%     -51.5%      16880 ±  9%  softirqs.CPU45.RCU
     11669 ±  2%     +42.8%      16669 ±  8%  softirqs.CPU45.SCHED
     34511 ±  4%     -51.8%      16626 ±  9%  softirqs.CPU46.RCU
     11311 ±  3%     +49.4%      16901 ± 14%  softirqs.CPU46.SCHED
     34270 ±  3%     -50.6%      16913 ±  7%  softirqs.CPU47.RCU
     10710 ±  9%     +55.8%      16684 ± 10%  softirqs.CPU47.SCHED
     34114 ±  3%     -51.3%      16615 ±  8%  softirqs.CPU48.RCU
     11141 ±  3%     +51.2%      16843 ±  9%  softirqs.CPU48.SCHED
     34470 ±  3%     -53.0%      16211 ±  8%  softirqs.CPU49.RCU
     11108 ±  3%     +51.6%      16844 ±  9%  softirqs.CPU49.SCHED
     35577 ±  5%     -50.0%      17775 ±  8%  softirqs.CPU5.RCU
     11904 ±  6%     +39.1%      16560 ±  5%  softirqs.CPU5.SCHED
     34402 ±  4%     -50.8%      16939 ±  9%  softirqs.CPU50.RCU
     11404           +49.2%      17011 ± 11%  softirqs.CPU50.SCHED
     34352 ±  3%     -50.9%      16850 ±  9%  softirqs.CPU51.RCU
     11032 ±  3%     +53.3%      16915 ±  9%  softirqs.CPU51.SCHED
     35684 ±  3%     -49.4%      18067 ± 16%  softirqs.CPU52.RCU
     11165 ±  9%     +57.3%      17566 ± 11%  softirqs.CPU52.SCHED
     36218 ±  3%     -51.7%      17505 ± 11%  softirqs.CPU53.RCU
     11749 ±  6%     +48.6%      17458 ± 11%  softirqs.CPU53.SCHED
     36922 ±  4%     -52.9%      17392 ± 10%  softirqs.CPU54.RCU
     12183 ±  7%     +38.3%      16852 ±  6%  softirqs.CPU54.SCHED
     36680 ±  4%     -51.8%      17694 ± 10%  softirqs.CPU55.RCU
     11480 ± 13%     +48.3%      17028 ± 14%  softirqs.CPU55.SCHED
     36393 ±  3%     -52.4%      17329 ±  9%  softirqs.CPU56.RCU
     11875 ±  9%     +45.6%      17295 ±  8%  softirqs.CPU56.SCHED
     36292 ±  3%     -51.5%      17608 ±  8%  softirqs.CPU57.RCU
     11865 ±  7%     +38.6%      16446 ±  9%  softirqs.CPU57.SCHED
     36323 ±  3%     -51.7%      17551 ±  8%  softirqs.CPU58.RCU
     12063 ±  5%     +43.3%      17283 ± 12%  softirqs.CPU58.SCHED
     35731 ±  3%     -51.9%      17195 ±  8%  softirqs.CPU59.RCU
     11533 ±  9%     +39.5%      16085 ±  9%  softirqs.CPU59.SCHED
     35813 ±  6%     -50.1%      17873 ±  9%  softirqs.CPU6.RCU
     12499 ±  7%     +32.6%      16576 ±  5%  softirqs.CPU6.SCHED
     36012 ±  5%     -51.2%      17570 ±  9%  softirqs.CPU60.RCU
     11872 ±  7%     +49.1%      17700 ± 11%  softirqs.CPU60.SCHED
     35774 ±  5%     -51.0%      17525 ±  8%  softirqs.CPU61.RCU
     11789 ± 10%     +46.1%      17221 ±  8%  softirqs.CPU61.SCHED
     37150 ±  9%     -52.2%      17752 ±  5%  softirqs.CPU62.RCU
     12245 ±  6%     +42.9%      17498 ±  9%  softirqs.CPU62.SCHED
     35690 ±  5%     -49.8%      17901 ± 10%  softirqs.CPU63.RCU
     11659 ±  6%     +45.9%      17007 ±  8%  softirqs.CPU63.SCHED
     35822 ±  5%     -49.9%      17952 ± 10%  softirqs.CPU64.RCU
     11675 ± 10%     +39.4%      16271 ±  8%  softirqs.CPU64.SCHED
     35664 ±  5%     -51.9%      17146 ± 13%  softirqs.CPU65.RCU
     11942 ±  4%     +46.0%      17441 ± 11%  softirqs.CPU65.SCHED
     35869 ±  5%     -50.8%      17631 ± 10%  softirqs.CPU66.RCU
     11788 ±  9%     +37.1%      16166 ± 11%  softirqs.CPU66.SCHED
     35752 ±  5%     -48.5%      18400 ±  6%  softirqs.CPU67.RCU
     11832 ±  5%     +30.7%      15465 ± 14%  softirqs.CPU67.SCHED
     35874 ±  5%     -51.3%      17482 ±  8%  softirqs.CPU68.RCU
     11849 ±  7%     +50.9%      17878 ± 12%  softirqs.CPU68.SCHED
     35797 ±  5%     -50.7%      17630 ± 10%  softirqs.CPU69.RCU
     12010 ±  7%     +45.1%      17429 ±  9%  softirqs.CPU69.SCHED
     35884 ±  7%     -50.5%      17756 ±  9%  softirqs.CPU7.RCU
     12198 ±  3%     +36.3%      16625 ±  8%  softirqs.CPU7.SCHED
     35478 ±  5%     -50.9%      17414 ± 10%  softirqs.CPU70.RCU
     11914 ±  6%     +45.7%      17361 ± 12%  softirqs.CPU70.SCHED
     35000 ±  5%     -50.7%      17241 ±  9%  softirqs.CPU71.RCU
     11654 ±  7%     +50.8%      17570 ± 10%  softirqs.CPU71.SCHED
     35540 ±  5%     -50.8%      17484 ± 10%  softirqs.CPU72.RCU
     11494 ± 12%     +52.2%      17495 ± 11%  softirqs.CPU72.SCHED
     35555 ±  6%     -51.3%      17302 ±  9%  softirqs.CPU73.RCU
     11638 ±  7%     +44.0%      16763 ±  7%  softirqs.CPU73.SCHED
     36066 ±  5%     -51.6%      17463 ± 10%  softirqs.CPU74.RCU
     11875 ±  6%     +51.8%      18030 ±  9%  softirqs.CPU74.SCHED
     32220           -51.9%      15495 ±  9%  softirqs.CPU75.RCU
     11872 ±  7%     +46.7%      17414 ±  8%  softirqs.CPU75.SCHED
     32127           -53.2%      15032 ±  9%  softirqs.CPU76.RCU
     11818 ±  6%     +48.7%      17568 ±  6%  softirqs.CPU76.SCHED
     32399           -53.4%      15091 ±  9%  softirqs.CPU77.RCU
     11919 ±  6%     +42.1%      16941 ± 10%  softirqs.CPU77.SCHED
     36650 ±  5%     -51.5%      17761 ±  7%  softirqs.CPU78.RCU
     11474 ±  3%     +46.0%      16747 ±  7%  softirqs.CPU78.SCHED
     36130 ±  5%     -50.7%      17798 ±  9%  softirqs.CPU79.RCU
     11307 ±  5%     +50.8%      17046 ± 12%  softirqs.CPU79.SCHED
     35939 ±  5%     -51.2%      17537 ±  9%  softirqs.CPU8.RCU
     11966 ±  5%     +44.2%      17258 ±  9%  softirqs.CPU8.SCHED
     36917 ±  4%     -51.1%      18050 ±  9%  softirqs.CPU80.RCU
     11329 ±  4%     +48.7%      16845 ±  9%  softirqs.CPU80.SCHED
     36402 ±  4%     -51.0%      17820 ±  8%  softirqs.CPU81.RCU
     10966 ±  3%     +54.3%      16923 ± 11%  softirqs.CPU81.SCHED
     36364 ±  4%     -51.5%      17622 ±  8%  softirqs.CPU82.RCU
     11781 ± 11%     +38.8%      16350 ±  5%  softirqs.CPU82.SCHED
     36145 ±  3%     -51.6%      17510 ±  8%  softirqs.CPU83.RCU
     11730           +41.6%      16605 ± 12%  softirqs.CPU83.SCHED
     36770 ±  3%     -53.3%      17168 ±  8%  softirqs.CPU84.RCU
     11123 ±  4%     +35.6%      15081 ±  9%  softirqs.CPU84.SCHED
     36253 ±  4%     -51.3%      17637 ±  8%  softirqs.CPU85.RCU
     11699 ±  2%     +42.3%      16642 ± 11%  softirqs.CPU85.SCHED
     36639 ±  5%     -52.2%      17529 ±  9%  softirqs.CPU86.RCU
     11708 ±  3%     +42.9%      16734 ± 10%  softirqs.CPU86.SCHED
     36173 ±  4%     -51.6%      17497 ±  8%  softirqs.CPU87.RCU
     11109 ±  3%     +52.3%      16916 ± 12%  softirqs.CPU87.SCHED
     36310 ±  6%     -48.9%      18547 ± 13%  softirqs.CPU88.RCU
     11397 ±  3%     +50.2%      17120 ± 13%  softirqs.CPU88.SCHED
     36144 ±  4%     -51.6%      17480 ±  8%  softirqs.CPU89.RCU
     10892 ±  4%     +52.1%      16570 ±  5%  softirqs.CPU89.SCHED
     35787 ±  5%     -51.3%      17438 ±  9%  softirqs.CPU9.RCU
     11873 ±  6%     +38.4%      16436 ±  6%  softirqs.CPU9.SCHED
     34586 ±  3%     -51.0%      16953 ±  8%  softirqs.CPU90.RCU
     11694 ±  3%     +44.4%      16886 ± 11%  softirqs.CPU90.SCHED
     33764 ±  7%     -49.8%      16954 ±  8%  softirqs.CPU91.RCU
     11530 ±  4%     +39.1%      16036 ±  9%  softirqs.CPU91.SCHED
     34534 ±  4%     -50.8%      16978 ±  8%  softirqs.CPU92.RCU
     11508 ±  2%     +46.2%      16828 ± 12%  softirqs.CPU92.SCHED
     34811 ±  4%     -51.4%      16925 ±  8%  softirqs.CPU93.RCU
     11510 ±  4%     +47.0%      16925 ± 11%  softirqs.CPU93.SCHED
     35079 ±  5%     -51.0%      17196 ± 10%  softirqs.CPU94.RCU
     11124 ±  6%     +55.7%      17316 ± 12%  softirqs.CPU94.SCHED
     33354 ±  5%     -48.1%      17313 ±  6%  softirqs.CPU95.RCU
     11697 ±  3%     +40.2%      16394 ±  9%  softirqs.CPU95.SCHED
     34858 ±  3%     -50.8%      17146 ±  8%  softirqs.CPU96.RCU
     11062 ±  4%     +47.6%      16333 ± 16%  softirqs.CPU96.SCHED
     34951 ±  5%     -51.1%      17098 ±  8%  softirqs.CPU97.RCU
     11490 ±  6%     +45.8%      16749 ±  9%  softirqs.CPU97.SCHED
     34851 ±  4%     -49.4%      17640 ±  8%  softirqs.CPU98.RCU
     11637 ±  2%     +47.3%      17146 ± 13%  softirqs.CPU98.SCHED
     34497 ±  3%     -50.2%      17183 ±  8%  softirqs.CPU99.RCU
     11354 ±  3%     +51.5%      17202 ± 12%  softirqs.CPU99.SCHED
   3698768 ±  4%     -51.0%    1812217 ±  8%  softirqs.RCU
   1217108 ±  3%     +45.2%    1767755 ±  2%  softirqs.SCHED
    306.25 ± 28%   +5353.0%      16699 ±125%  interrupts.40:PCI-MSI.67633155-edge.eth0-TxRx-2
      2375 ± 49%     -67.6%     770.75 ±111%  interrupts.41:PCI-MSI.67633156-edge.eth0-TxRx-3
   2596291           -29.7%    1824254 ±  3%  interrupts.CAL:Function_call_interrupts
     26901 ±  6%     -39.0%      16422 ±  8%  interrupts.CPU0.CAL:Function_call_interrupts
      5955 ±  3%     -30.2%       4158 ± 28%  interrupts.CPU0.NMI:Non-maskable_interrupts
      5955 ±  3%     -30.2%       4158 ± 28%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
     25934 ±  7%     -40.3%      15480 ±  8%  interrupts.CPU0.TLB:TLB_shootdowns
     26458 ±  8%     -36.4%      16839 ±  7%  interrupts.CPU1.CAL:Function_call_interrupts
      5972 ±  3%     -23.4%       4577 ± 16%  interrupts.CPU1.NMI:Non-maskable_interrupts
      5972 ±  3%     -23.4%       4577 ± 16%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
     26095 ± 10%     -36.2%      16659 ±  7%  interrupts.CPU1.TLB:TLB_shootdowns
     26356 ±  5%     -35.1%      17107 ±  3%  interrupts.CPU10.CAL:Function_call_interrupts
     25802 ±  7%     -35.2%      16726 ±  2%  interrupts.CPU10.TLB:TLB_shootdowns
     23846 ± 10%     -21.0%      18838 ±  6%  interrupts.CPU100.CAL:Function_call_interrupts
     22672 ± 14%     -20.4%      18044 ±  6%  interrupts.CPU100.TLB:TLB_shootdowns
     23478 ±  8%     -22.1%      18299 ±  7%  interrupts.CPU101.CAL:Function_call_interrupts
     22711 ± 10%     -20.8%      17988 ±  7%  interrupts.CPU101.TLB:TLB_shootdowns
     23877 ± 10%     -23.6%      18243 ±  6%  interrupts.CPU102.CAL:Function_call_interrupts
     22739 ± 12%     -21.0%      17973 ±  7%  interrupts.CPU102.TLB:TLB_shootdowns
     23793 ± 10%     -24.9%      17877 ± 13%  interrupts.CPU103.CAL:Function_call_interrupts
     22947 ± 13%     -22.7%      17727 ± 13%  interrupts.CPU103.TLB:TLB_shootdowns
     26231 ±  6%     -33.0%      17569 ±  4%  interrupts.CPU11.CAL:Function_call_interrupts
      6037 ±  2%     -26.5%       4434 ± 27%  interrupts.CPU11.NMI:Non-maskable_interrupts
      6037 ±  2%     -26.5%       4434 ± 27%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
     25881 ±  8%     -32.8%      17387 ±  3%  interrupts.CPU11.TLB:TLB_shootdowns
     25992 ±  7%     -33.1%      17400        interrupts.CPU12.CAL:Function_call_interrupts
      6043           -21.0%       4773 ± 10%  interrupts.CPU12.NMI:Non-maskable_interrupts
      6043           -21.0%       4773 ± 10%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
     25944 ±  8%     -34.3%      17046        interrupts.CPU12.TLB:TLB_shootdowns
     26058 ±  7%     -32.8%      17505 ±  2%  interrupts.CPU13.CAL:Function_call_interrupts
      5913 ±  2%     -14.6%       5048 ±  7%  interrupts.CPU13.NMI:Non-maskable_interrupts
      5913 ±  2%     -14.6%       5048 ±  7%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
     25920 ±  8%     -34.4%      17012        interrupts.CPU13.TLB:TLB_shootdowns
     26093 ±  6%     -35.7%      16769 ±  2%  interrupts.CPU14.CAL:Function_call_interrupts
     25667 ±  9%     -35.4%      16576 ±  2%  interrupts.CPU14.TLB:TLB_shootdowns
     25946 ±  7%     -36.3%      16538 ±  3%  interrupts.CPU15.CAL:Function_call_interrupts
     25661 ±  8%     -36.2%      16380 ±  3%  interrupts.CPU15.TLB:TLB_shootdowns
     26417 ±  7%     -31.4%      18115 ±  6%  interrupts.CPU16.CAL:Function_call_interrupts
      5917 ±  2%     -27.5%       4287 ± 31%  interrupts.CPU16.NMI:Non-maskable_interrupts
      5917 ±  2%     -27.5%       4287 ± 31%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
     26211 ±  8%     -33.4%      17457 ±  6%  interrupts.CPU16.TLB:TLB_shootdowns
     26115 ±  7%     -35.0%      16980 ±  2%  interrupts.CPU17.CAL:Function_call_interrupts
     25888 ±  8%     -34.7%      16892 ±  2%  interrupts.CPU17.TLB:TLB_shootdowns
     26065 ±  5%     -36.7%      16489 ±  4%  interrupts.CPU18.CAL:Function_call_interrupts
     25810 ±  8%     -37.0%      16253 ±  4%  interrupts.CPU18.TLB:TLB_shootdowns
     26518 ±  8%     -35.8%      17017 ±  3%  interrupts.CPU19.CAL:Function_call_interrupts
     25993 ±  9%     -34.8%      16946 ±  2%  interrupts.CPU19.TLB:TLB_shootdowns
     26301 ±  8%     -34.6%      17209 ±  3%  interrupts.CPU2.CAL:Function_call_interrupts
     26073 ±  9%     -36.0%      16688 ±  3%  interrupts.CPU2.TLB:TLB_shootdowns
     26826 ±  7%     -35.7%      17254 ±  4%  interrupts.CPU20.CAL:Function_call_interrupts
     26443 ±  8%     -35.6%      17031 ±  3%  interrupts.CPU20.TLB:TLB_shootdowns
     26404 ±  4%     -34.9%      17184 ±  2%  interrupts.CPU21.CAL:Function_call_interrupts
     26155 ±  5%     -36.5%      16621 ±  3%  interrupts.CPU21.TLB:TLB_shootdowns
     26577 ±  4%     -34.8%      17339 ±  4%  interrupts.CPU22.CAL:Function_call_interrupts
     26035 ±  7%     -34.6%      17018 ±  5%  interrupts.CPU22.TLB:TLB_shootdowns
     26559 ±  5%     -34.1%      17502 ±  3%  interrupts.CPU23.CAL:Function_call_interrupts
     26098 ±  7%     -34.8%      17004 ±  2%  interrupts.CPU23.TLB:TLB_shootdowns
     26327 ±  9%     -36.0%      16844 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
     26090 ±  9%     -36.6%      16540 ±  4%  interrupts.CPU24.TLB:TLB_shootdowns
     26421 ±  9%     -27.6%      19128 ± 18%  interrupts.CPU25.CAL:Function_call_interrupts
      5927 ±  3%     -14.3%       5080 ± 11%  interrupts.CPU25.NMI:Non-maskable_interrupts
      5927 ±  3%     -14.3%       5080 ± 11%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
     26211 ± 10%     -35.7%      16855 ±  4%  interrupts.CPU25.TLB:TLB_shootdowns
     23299 ± 11%     -27.4%      16911 ±  7%  interrupts.CPU26.CAL:Function_call_interrupts
      2697 ± 82%     -82.8%     464.75 ± 36%  interrupts.CPU26.RES:Rescheduling_interrupts
     22275 ± 14%     -24.9%      16730 ±  7%  interrupts.CPU26.TLB:TLB_shootdowns
     23123 ± 12%     -25.3%      17283 ±  7%  interrupts.CPU27.CAL:Function_call_interrupts
      6348 ±  5%     -20.8%       5026 ± 25%  interrupts.CPU27.NMI:Non-maskable_interrupts
      6348 ±  5%     -20.8%       5026 ± 25%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     22433 ± 15%     -24.2%      17010 ±  7%  interrupts.CPU27.TLB:TLB_shootdowns
     24043 ±  9%     -27.1%      17523 ±  3%  interrupts.CPU28.CAL:Function_call_interrupts
     22944 ± 11%     -26.1%      16960 ±  5%  interrupts.CPU28.TLB:TLB_shootdowns
     23364 ± 11%     -22.8%      18033 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
     22756 ± 13%     -24.0%      17295 ±  8%  interrupts.CPU29.TLB:TLB_shootdowns
     26600 ±  4%     -32.6%      17917 ±  2%  interrupts.CPU3.CAL:Function_call_interrupts
      5947           -14.3%       5096 ± 10%  interrupts.CPU3.NMI:Non-maskable_interrupts
      5947           -14.3%       5096 ± 10%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
     25848 ±  8%     -32.1%      17562 ±  2%  interrupts.CPU3.TLB:TLB_shootdowns
     23584 ± 10%     -23.4%      18065 ± 10%  interrupts.CPU30.CAL:Function_call_interrupts
      1233 ± 17%     -67.5%     400.25 ± 28%  interrupts.CPU30.RES:Rescheduling_interrupts
     22441 ± 12%     -21.1%      17703 ± 10%  interrupts.CPU30.TLB:TLB_shootdowns
     23389 ± 11%     -25.3%      17470 ±  9%  interrupts.CPU31.CAL:Function_call_interrupts
     22711 ± 12%     -25.4%      16942 ± 11%  interrupts.CPU31.TLB:TLB_shootdowns
    306.25 ± 28%   +5353.0%      16699 ±125%  interrupts.CPU32.40:PCI-MSI.67633155-edge.eth0-TxRx-2
     23232 ± 12%     -23.8%      17708 ±  9%  interrupts.CPU32.CAL:Function_call_interrupts
     22664 ± 14%     -23.1%      17424 ±  9%  interrupts.CPU32.TLB:TLB_shootdowns
      2375 ± 49%     -67.6%     770.75 ±111%  interrupts.CPU33.41:PCI-MSI.67633156-edge.eth0-TxRx-3
     23097 ± 10%     -23.5%      17661 ±  7%  interrupts.CPU33.CAL:Function_call_interrupts
    995.75 ± 41%     -62.5%     373.00 ± 19%  interrupts.CPU33.RES:Rescheduling_interrupts
     22364 ± 12%     -22.5%      17337 ±  7%  interrupts.CPU33.TLB:TLB_shootdowns
     23038 ± 10%     -23.5%      17632 ±  5%  interrupts.CPU34.CAL:Function_call_interrupts
    970.75 ± 45%     -65.4%     335.75 ±  6%  interrupts.CPU34.RES:Rescheduling_interrupts
     22573 ± 12%     -22.7%      17447 ±  7%  interrupts.CPU34.TLB:TLB_shootdowns
     23540 ±  9%     -24.7%      17735 ±  5%  interrupts.CPU35.CAL:Function_call_interrupts
     22564 ± 13%     -23.5%      17271 ±  7%  interrupts.CPU35.TLB:TLB_shootdowns
     24107 ± 11%     -28.4%      17267 ±  9%  interrupts.CPU36.CAL:Function_call_interrupts
     22590 ± 14%     -25.6%      16813 ± 11%  interrupts.CPU36.TLB:TLB_shootdowns
     23441 ± 10%     -25.3%      17502 ± 11%  interrupts.CPU37.CAL:Function_call_interrupts
      1031 ± 38%     -62.4%     387.75 ± 17%  interrupts.CPU37.RES:Rescheduling_interrupts
     22388 ± 12%     -25.4%      16705 ±  9%  interrupts.CPU37.TLB:TLB_shootdowns
     23398 ± 12%     -24.9%      17570 ± 12%  interrupts.CPU38.CAL:Function_call_interrupts
      6360 ±  5%     -30.4%       4426 ± 32%  interrupts.CPU38.NMI:Non-maskable_interrupts
      6360 ±  5%     -30.4%       4426 ± 32%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
     22726 ± 14%     -23.9%      17299 ± 12%  interrupts.CPU38.TLB:TLB_shootdowns
     23463 ± 10%     -24.1%      17817 ±  7%  interrupts.CPU39.CAL:Function_call_interrupts
     22600 ± 13%     -23.1%      17374 ±  7%  interrupts.CPU39.TLB:TLB_shootdowns
     27479 ± 13%     -36.8%      17366 ±  4%  interrupts.CPU4.CAL:Function_call_interrupts
      5893 ±  2%     -14.7%       5026 ± 10%  interrupts.CPU4.NMI:Non-maskable_interrupts
      5893 ±  2%     -14.7%       5026 ± 10%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
     25961 ±  7%     -34.9%      16888 ±  4%  interrupts.CPU4.TLB:TLB_shootdowns
     23574 ± 11%     -22.6%      18258 ±  3%  interrupts.CPU40.CAL:Function_call_interrupts
     22510 ± 13%     -21.9%      17588 ±  5%  interrupts.CPU40.TLB:TLB_shootdowns
     22991 ± 10%     -25.0%      17237 ±  7%  interrupts.CPU41.CAL:Function_call_interrupts
      1157 ± 36%     -66.5%     388.25 ± 16%  interrupts.CPU41.RES:Rescheduling_interrupts
     22090 ± 12%     -22.7%      17081 ±  6%  interrupts.CPU41.TLB:TLB_shootdowns
     22484 ± 11%     -22.9%      17338 ±  8%  interrupts.CPU42.CAL:Function_call_interrupts
     21831 ± 14%     -21.5%      17134 ±  9%  interrupts.CPU42.TLB:TLB_shootdowns
     23758 ± 11%     -26.4%      17475 ±  6%  interrupts.CPU43.CAL:Function_call_interrupts
     22690 ± 14%     -24.4%      17154 ±  7%  interrupts.CPU43.TLB:TLB_shootdowns
     23177 ± 11%     -24.9%      17402 ±  7%  interrupts.CPU44.CAL:Function_call_interrupts
     22680 ± 12%     -23.9%      17253 ±  6%  interrupts.CPU44.TLB:TLB_shootdowns
      6303 ±  4%     -28.5%       4506 ± 29%  interrupts.CPU45.NMI:Non-maskable_interrupts
      6303 ±  4%     -28.5%       4506 ± 29%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
     22021 ± 13%     -20.8%      17439 ±  6%  interrupts.CPU45.TLB:TLB_shootdowns
     22933 ±  9%     -17.5%      18928 ±  6%  interrupts.CPU46.CAL:Function_call_interrupts
     22048 ± 12%     -21.2%      17376 ±  6%  interrupts.CPU46.TLB:TLB_shootdowns
     23696 ± 11%     -24.2%      17972 ±  7%  interrupts.CPU47.CAL:Function_call_interrupts
    822.75 ± 35%     -55.9%     362.75 ± 13%  interrupts.CPU47.RES:Rescheduling_interrupts
     22394 ± 14%     -21.3%      17633 ±  6%  interrupts.CPU47.TLB:TLB_shootdowns
     24278 ± 10%     -29.7%      17073 ±  9%  interrupts.CPU48.CAL:Function_call_interrupts
     22670 ± 14%     -25.4%      16919 ±  8%  interrupts.CPU48.TLB:TLB_shootdowns
     23744 ±  8%     -23.5%      18171 ± 11%  interrupts.CPU49.CAL:Function_call_interrupts
     22246 ± 12%     -21.7%      17409 ±  8%  interrupts.CPU49.TLB:TLB_shootdowns
     26402 ±  4%     -35.5%      17034 ±  7%  interrupts.CPU5.CAL:Function_call_interrupts
     26131 ±  6%     -36.0%      16737 ±  6%  interrupts.CPU5.TLB:TLB_shootdowns
     22794 ± 10%     -21.8%      17815 ± 11%  interrupts.CPU50.CAL:Function_call_interrupts
    776.00 ± 37%     -56.5%     337.50 ± 15%  interrupts.CPU50.RES:Rescheduling_interrupts
     22030 ± 12%     -20.8%      17437 ± 10%  interrupts.CPU50.TLB:TLB_shootdowns
     24195 ±  9%     -24.2%      18332 ± 10%  interrupts.CPU51.CAL:Function_call_interrupts
      6343 ±  5%     -24.2%       4808 ± 24%  interrupts.CPU51.NMI:Non-maskable_interrupts
      6343 ±  5%     -24.2%       4808 ± 24%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
     22848 ± 13%     -23.3%      17513 ± 10%  interrupts.CPU51.TLB:TLB_shootdowns
     26296 ±  7%     -33.6%      17448        interrupts.CPU52.CAL:Function_call_interrupts
     25791 ± 10%     -35.0%      16771 ±  2%  interrupts.CPU52.TLB:TLB_shootdowns
     26438 ±  8%     -33.8%      17500 ±  4%  interrupts.CPU53.CAL:Function_call_interrupts
     26174 ± 10%     -33.5%      17415 ±  3%  interrupts.CPU53.TLB:TLB_shootdowns
     29497 ± 13%     -39.8%      17755 ±  2%  interrupts.CPU54.CAL:Function_call_interrupts
     26240 ±  9%     -34.1%      17300 ±  2%  interrupts.CPU54.TLB:TLB_shootdowns
     26667 ±  8%     -34.0%      17610 ±  2%  interrupts.CPU55.CAL:Function_call_interrupts
     26081 ±  8%     -34.1%      17184 ±  4%  interrupts.CPU55.TLB:TLB_shootdowns
     26006 ±  8%     -32.1%      17666 ±  2%  interrupts.CPU56.CAL:Function_call_interrupts
     25725 ±  9%     -33.6%      17070 ±  4%  interrupts.CPU56.TLB:TLB_shootdowns
     26790 ±  8%     -34.8%      17473 ±  4%  interrupts.CPU57.CAL:Function_call_interrupts
     26170 ±  7%     -34.6%      17123 ±  4%  interrupts.CPU57.TLB:TLB_shootdowns
     26756 ±  7%     -36.0%      17111 ±  4%  interrupts.CPU58.CAL:Function_call_interrupts
     26253 ±  8%     -35.4%      16972 ±  5%  interrupts.CPU58.TLB:TLB_shootdowns
     26404 ±  8%     -35.7%      16988 ±  4%  interrupts.CPU59.CAL:Function_call_interrupts
     26408 ±  7%     -36.4%      16783 ±  4%  interrupts.CPU59.TLB:TLB_shootdowns
     26344 ±  6%     -35.1%      17093 ±  3%  interrupts.CPU6.CAL:Function_call_interrupts
     26550 ±  7%     -36.1%      16952 ±  3%  interrupts.CPU6.TLB:TLB_shootdowns
     25912 ± 10%     -33.6%      17206        interrupts.CPU60.CAL:Function_call_interrupts
      5972 ±  2%     -28.8%       4254 ± 31%  interrupts.CPU60.NMI:Non-maskable_interrupts
      5972 ±  2%     -28.8%       4254 ± 31%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
     25837 ± 10%     -34.4%      16960 ±  2%  interrupts.CPU60.TLB:TLB_shootdowns
     26427 ±  9%     -34.7%      17251 ±  2%  interrupts.CPU61.CAL:Function_call_interrupts
     25174 ±  9%     -32.2%      17060 ±  3%  interrupts.CPU61.TLB:TLB_shootdowns
     25979 ±  8%     -32.1%      17646 ±  4%  interrupts.CPU62.CAL:Function_call_interrupts
      5874 ±  3%     -21.4%       4616 ± 29%  interrupts.CPU62.NMI:Non-maskable_interrupts
      5874 ±  3%     -21.4%       4616 ± 29%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
     25625 ±  9%     -32.0%      17420 ±  3%  interrupts.CPU62.TLB:TLB_shootdowns
     27245 ± 12%     -37.1%      17126 ±  6%  interrupts.CPU63.CAL:Function_call_interrupts
     26270 ± 10%     -36.9%      16572 ±  3%  interrupts.CPU63.TLB:TLB_shootdowns
     25949 ±  7%     -33.7%      17217 ±  4%  interrupts.CPU64.CAL:Function_call_interrupts
     25662 ±  9%     -33.8%      16979 ±  5%  interrupts.CPU64.TLB:TLB_shootdowns
     25744 ±  7%     -33.1%      17231 ±  3%  interrupts.CPU65.CAL:Function_call_interrupts
     25550 ±  8%     -33.3%      17030 ±  3%  interrupts.CPU65.TLB:TLB_shootdowns
     26475 ±  6%     -37.0%      16678 ±  2%  interrupts.CPU66.CAL:Function_call_interrupts
     26068 ±  7%     -36.9%      16460 ±  3%  interrupts.CPU66.TLB:TLB_shootdowns
     25969 ±  7%     -32.7%      17478 ±  3%  interrupts.CPU67.CAL:Function_call_interrupts
     25690 ±  9%     -32.7%      17291 ±  3%  interrupts.CPU67.TLB:TLB_shootdowns
     26047 ±  5%     -37.0%      16413 ±  4%  interrupts.CPU68.CAL:Function_call_interrupts
     25715 ±  6%     -36.9%      16215 ±  4%  interrupts.CPU68.TLB:TLB_shootdowns
     26268 ±  8%     -32.4%      17766 ±  3%  interrupts.CPU69.CAL:Function_call_interrupts
     25924 ±  8%     -32.8%      17414 ±  3%  interrupts.CPU69.TLB:TLB_shootdowns
     26458 ±  7%     -36.7%      16745 ±  3%  interrupts.CPU7.CAL:Function_call_interrupts
      5912 ±  2%     -24.9%       4440 ± 23%  interrupts.CPU7.NMI:Non-maskable_interrupts
      5912 ±  2%     -24.9%       4440 ± 23%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
     26221 ±  8%     -37.5%      16389 ±  3%  interrupts.CPU7.TLB:TLB_shootdowns
     26454 ±  8%     -33.9%      17491 ±  4%  interrupts.CPU70.CAL:Function_call_interrupts
      5936 ±  2%     -30.6%       4118 ± 26%  interrupts.CPU70.NMI:Non-maskable_interrupts
      5936 ±  2%     -30.6%       4118 ± 26%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
     25870 ±  8%     -34.5%      16941 ±  4%  interrupts.CPU70.TLB:TLB_shootdowns
     26897 ± 12%     -34.6%      17595 ±  2%  interrupts.CPU71.CAL:Function_call_interrupts
      5897 ±  2%     -27.3%       4289 ± 20%  interrupts.CPU71.NMI:Non-maskable_interrupts
      5897 ±  2%     -27.3%       4289 ± 20%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
     26199 ± 10%     -35.6%      16875 ±  2%  interrupts.CPU71.TLB:TLB_shootdowns
     28652 ± 17%     -41.6%      16739 ±  3%  interrupts.CPU72.CAL:Function_call_interrupts
      6007 ±  2%     -19.5%       4834 ± 13%  interrupts.CPU72.NMI:Non-maskable_interrupts
      6007 ±  2%     -19.5%       4834 ± 13%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
     26441 ± 10%     -37.5%      16538 ±  3%  interrupts.CPU72.TLB:TLB_shootdowns
     26463 ±  7%     -35.1%      17180        interrupts.CPU73.CAL:Function_call_interrupts
     26223 ±  8%     -35.7%      16857 ±  2%  interrupts.CPU73.TLB:TLB_shootdowns
     25734 ±  9%     -35.0%      16715        interrupts.CPU74.CAL:Function_call_interrupts
      5935 ±  3%     -17.2%       4914 ± 14%  interrupts.CPU74.NMI:Non-maskable_interrupts
      5935 ±  3%     -17.2%       4914 ± 14%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
     25506 ± 10%     -35.7%      16399        interrupts.CPU74.TLB:TLB_shootdowns
     26809 ± 10%     -37.0%      16878 ±  4%  interrupts.CPU75.CAL:Function_call_interrupts
     25727 ±  9%     -35.9%      16496 ±  3%  interrupts.CPU75.TLB:TLB_shootdowns
     26569 ±  6%     -32.8%      17849 ±  6%  interrupts.CPU76.CAL:Function_call_interrupts
     26155 ±  8%     -32.2%      17729 ±  5%  interrupts.CPU76.TLB:TLB_shootdowns
     26936 ± 12%     -34.4%      17679 ±  3%  interrupts.CPU77.CAL:Function_call_interrupts
     26474 ± 11%     -34.2%      17418 ±  3%  interrupts.CPU77.TLB:TLB_shootdowns
     23558 ± 10%     -26.4%      17350 ±  9%  interrupts.CPU78.CAL:Function_call_interrupts
      6292 ±  5%     -14.3%       5395 ± 13%  interrupts.CPU78.NMI:Non-maskable_interrupts
      6292 ±  5%     -14.3%       5395 ± 13%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
     22661 ± 12%     -25.1%      16965 ±  9%  interrupts.CPU78.TLB:TLB_shootdowns
     23900 ±  8%     -28.8%      17026 ± 10%  interrupts.CPU79.CAL:Function_call_interrupts
     22899 ± 11%     -28.3%      16408 ± 12%  interrupts.CPU79.TLB:TLB_shootdowns
     26589 ±  8%     -31.9%      18103 ±  6%  interrupts.CPU8.CAL:Function_call_interrupts
      5937           -15.9%       4996 ±  7%  interrupts.CPU8.NMI:Non-maskable_interrupts
      5937           -15.9%       4996 ±  7%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
     26109 ±  9%     -34.3%      17142 ±  3%  interrupts.CPU8.TLB:TLB_shootdowns
     23831 ± 10%     -24.0%      18112 ±  5%  interrupts.CPU80.CAL:Function_call_interrupts
      2139 ± 79%     -85.7%     305.00 ± 29%  interrupts.CPU80.RES:Rescheduling_interrupts
     22992 ± 11%     -23.4%      17621 ±  5%  interrupts.CPU80.TLB:TLB_shootdowns
     24073 ±  8%     -26.9%      17607 ±  3%  interrupts.CPU81.CAL:Function_call_interrupts
      6311 ±  5%     -21.5%       4957 ± 22%  interrupts.CPU81.NMI:Non-maskable_interrupts
      6311 ±  5%     -21.5%       4957 ± 22%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      4107 ± 74%     -92.4%     310.75 ± 49%  interrupts.CPU81.RES:Rescheduling_interrupts
     22165 ± 10%     -22.3%      17232 ±  3%  interrupts.CPU81.TLB:TLB_shootdowns
     23394 ±  9%     -25.1%      17527 ±  9%  interrupts.CPU82.CAL:Function_call_interrupts
     22402 ± 10%     -23.0%      17248 ±  9%  interrupts.CPU82.TLB:TLB_shootdowns
     24784 ± 18%     -25.2%      18548 ±  4%  interrupts.CPU83.CAL:Function_call_interrupts
      2295 ± 77%     -87.2%     293.25 ± 33%  interrupts.CPU83.RES:Rescheduling_interrupts
     22683 ± 12%     -19.2%      18323 ±  4%  interrupts.CPU83.TLB:TLB_shootdowns
     23284 ± 10%     -24.9%      17481 ±  7%  interrupts.CPU84.CAL:Function_call_interrupts
     22610 ± 13%     -25.5%      16848 ± 11%  interrupts.CPU84.TLB:TLB_shootdowns
     23078 ± 11%     -21.0%      18228 ±  7%  interrupts.CPU85.CAL:Function_call_interrupts
      1414 ± 48%     -81.0%     269.00 ± 28%  interrupts.CPU85.RES:Rescheduling_interrupts
     22355 ± 11%     -23.4%      17113 ±  6%  interrupts.CPU85.TLB:TLB_shootdowns
     23054 ±  9%     -24.5%      17402 ±  7%  interrupts.CPU86.CAL:Function_call_interrupts
     22287 ± 10%     -24.4%      16850 ±  8%  interrupts.CPU86.TLB:TLB_shootdowns
     22985 ±  9%     -23.7%      17533 ±  7%  interrupts.CPU87.CAL:Function_call_interrupts
      6227 ±  5%     -25.2%       4657 ± 25%  interrupts.CPU87.NMI:Non-maskable_interrupts
      6227 ±  5%     -25.2%       4657 ± 25%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
     22113 ± 11%     -22.5%      17145 ±  8%  interrupts.CPU87.TLB:TLB_shootdowns
     23566 ± 10%     -26.4%      17336 ±  7%  interrupts.CPU88.CAL:Function_call_interrupts
      6211 ±  5%     -30.4%       4322 ± 38%  interrupts.CPU88.NMI:Non-maskable_interrupts
      6211 ±  5%     -30.4%       4322 ± 38%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
     22944 ± 12%     -25.6%      17063 ±  7%  interrupts.CPU88.TLB:TLB_shootdowns
     22953 ±  8%     -23.0%      17664 ±  7%  interrupts.CPU89.CAL:Function_call_interrupts
      6257 ±  4%     -25.6%       4657 ± 24%  interrupts.CPU89.NMI:Non-maskable_interrupts
      6257 ±  4%     -25.6%       4657 ± 24%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
     22312 ± 11%     -22.3%      17338 ±  7%  interrupts.CPU89.TLB:TLB_shootdowns
     25895 ±  8%     -37.5%      16178 ±  4%  interrupts.CPU9.CAL:Function_call_interrupts
      5927 ±  2%     -16.9%       4924 ± 13%  interrupts.CPU9.NMI:Non-maskable_interrupts
      5927 ±  2%     -16.9%       4924 ± 13%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     25836 ±  9%     -38.2%      15954 ±  4%  interrupts.CPU9.TLB:TLB_shootdowns
     23307 ± 11%     -23.2%      17896 ±  4%  interrupts.CPU90.CAL:Function_call_interrupts
     22572 ± 13%     -23.0%      17390 ±  6%  interrupts.CPU90.TLB:TLB_shootdowns
     23119 ± 11%     -23.7%      17642 ±  8%  interrupts.CPU91.CAL:Function_call_interrupts
     22566 ± 12%     -23.1%      17359 ±  9%  interrupts.CPU91.TLB:TLB_shootdowns
     23060 ± 12%     -24.0%      17537 ± 10%  interrupts.CPU92.CAL:Function_call_interrupts
      1696 ± 33%     -84.7%     259.75 ± 37%  interrupts.CPU92.RES:Rescheduling_interrupts
     22200 ± 13%     -22.1%      17304 ± 11%  interrupts.CPU92.TLB:TLB_shootdowns
     22714 ± 12%     -25.0%      17040 ± 13%  interrupts.CPU93.CAL:Function_call_interrupts
    994.50 ± 14%     -72.0%     278.00 ± 48%  interrupts.CPU93.RES:Rescheduling_interrupts
     22186 ± 12%     -24.8%      16690 ± 13%  interrupts.CPU93.TLB:TLB_shootdowns
     23025 ± 11%     -22.0%      17960 ±  9%  interrupts.CPU94.CAL:Function_call_interrupts
     22298 ± 11%     -21.2%      17562 ±  9%  interrupts.CPU94.TLB:TLB_shootdowns
     23531 ± 10%     -19.8%      18871 ±  8%  interrupts.CPU95.CAL:Function_call_interrupts
     22568 ± 13%     -19.2%      18236 ±  5%  interrupts.CPU95.TLB:TLB_shootdowns
     23352 ± 10%     -23.9%      17771 ±  5%  interrupts.CPU96.CAL:Function_call_interrupts
     22819 ± 12%     -23.1%      17551 ±  4%  interrupts.CPU96.TLB:TLB_shootdowns
     23336 ± 16%     -25.0%      17498 ±  8%  interrupts.CPU97.CAL:Function_call_interrupts
      1713 ± 70%     -85.4%     250.00 ± 27%  interrupts.CPU97.RES:Rescheduling_interrupts
     22534 ± 14%     -24.0%      17136 ±  8%  interrupts.CPU97.TLB:TLB_shootdowns
     22961 ± 11%     -19.2%      18551 ±  7%  interrupts.CPU98.CAL:Function_call_interrupts
    770.25 ± 47%     -68.4%     243.25 ± 38%  interrupts.CPU98.RES:Rescheduling_interrupts
     22489 ± 13%     -18.7%      18274 ±  7%  interrupts.CPU98.TLB:TLB_shootdowns
     23336 ± 11%     -23.8%      17774 ±  8%  interrupts.CPU99.CAL:Function_call_interrupts
     22462 ± 13%     -23.1%      17272 ±  7%  interrupts.CPU99.TLB:TLB_shootdowns
    591378 ±  4%     -13.2%     513176 ±  8%  interrupts.NMI:Non-maskable_interrupts
    591378 ±  4%     -13.2%     513176 ±  8%  interrupts.PMI:Performance_monitoring_interrupts
    107989 ± 17%     -54.3%      49357 ±  5%  interrupts.RES:Rescheduling_interrupts
   2521833           -29.5%    1778413 ±  3%  interrupts.TLB:TLB_shootdowns





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.8.0-rc6-00041-g3edecfef02853" of type "text/plain" (169185 bytes)

View attachment "job-script" of type "text/plain" (8107 bytes)

View attachment "job.yaml" of type "text/plain" (5501 bytes)

View attachment "reproduce" of type "text/plain" (5962 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ