lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20191107083850.GS29418@shao2-debian>
Date:   Thu, 7 Nov 2019 16:38:50 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Hillf Danton <hdanton@...a.com>
Cc:     linux-mm <linux-mm@...ck.org>,
        fsdev <linux-fsdevel@...r.kernel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        Fengguang Wu <fengguang.wu@...el.com>,
        Tejun Heo <tj@...nel.org>, Jan Kara <jack@...e.com>,
        Johannes Weiner <hannes@...xchg.org>,
        Shakeel Butt <shakeelb@...gle.com>,
        Minchan Kim <minchan@...nel.org>, Mel Gorman <mgorman@...e.de>,
        Hillf Danton <hdanton@...a.com>, lkp@...ts.01.org
Subject: [writeback]  2b871886bb:  fio.write_bw_MBps 40.6% improvement

Greeting,

FYI, we noticed a 40.6% improvement of fio.write_bw_MBps due to commit:


commit: 2b871886bb64af404a6a6a4db832575a81ba3f7b ("[RFC v2] writeback: add elastic bdi in cgwb bdp")
url: https://github.com/0day-ci/linux/commits/Hillf-Danton/writeback-add-elastic-bdi-in-cgwb-bdp/20191029-135945


in testcase: fio-basic
on test machine: 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory
with following parameters:

	runtime: 300s
	disk: 1SSD
	fs: xfs
	nr_task: 100%
	test_size: 128G
	rw: write
	bs: 4k
	ioengine: sync
	cpufreq_governor: performance
	ucode: 0xb000038
	fs2: nfsv4

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio

In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------+
| testcase: change | fio-basic: fio.write_bw_MBps 9.8% improvement               |
| test machine     | 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory |
| test parameters  | bs=4M                                                       |
|                  | cpufreq_governor=performance                                |
|                  | disk=1SSD                                                   |
|                  | fs=btrfs                                                    |
|                  | ioengine=sync                                               |
|                  | nr_task=100%                                                |
|                  | runtime=300s                                                |
|                  | rw=randwrite                                                |
|                  | test_size=128G                                              |
|                  | ucode=0x4000021                                             |
+------------------+-------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs2/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
  4k/gcc-7/performance/1SSD/nfsv4/xfs/sync/x86_64-rhel-7.6/100%/debian-x86_64-2019-09-23.cgz/300s/write/lkp-bdw-ex2/128G/fio-basic/0xb000038

commit: 
  8005803a2c (" ARC fixes for 5.4-rc6")
  2b871886bb ("writeback: add elastic bdi in cgwb bdp")

8005803a2ca0af49 2b871886bb64af404a6a6a4db83 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.03 ± 10%      -0.0        0.00        fio.latency_100ms%
      1.32 ± 19%      +0.6        1.97 ± 10%  fio.latency_100us%
     45.00           +12.3       57.32 ±  5%  fio.latency_10us%
      0.02 ±  7%      -0.0        0.00        fio.latency_250ms%
      0.76 ± 92%      +1.8        2.59 ± 23%  fio.latency_250us%
      0.02 ± 18%      -0.0        0.01        fio.latency_2us%
     40.99 ± 11%     -16.3       24.69 ± 12%  fio.latency_4us%
      0.27 ±  3%      -0.3        0.01        fio.latency_50ms%
     56.22           -28.5%      40.21        fio.time.elapsed_time
     56.22           -28.5%      40.21        fio.time.elapsed_time.max
    795.75 ± 22%    +115.5%       1714 ±  9%  fio.time.percent_of_cpu_this_job_got
    408.25 ± 25%     +58.3%     646.29 ±  9%  fio.time.system_time
     39.45 ±  4%      +9.1%      43.06 ±  3%  fio.time.user_time
    120661 ±  5%     -92.9%       8584 ±  2%  fio.time.voluntary_context_switches
      2366           +40.6%       3328        fio.write_bw_MBps
     12192 ± 16%     +65.9%      20224 ±  9%  fio.write_clat_90%_us
     35008 ± 41%    +194.0%     102912 ± 21%  fio.write_clat_95%_us
    166788 ±  4%     -35.9%     106898 ±  5%  fio.write_clat_mean_us
   2893811 ±  2%    +227.5%    9476109 ±  3%  fio.write_clat_stddev
    605744           +40.6%     851968        fio.write_iops
  79715313            +9.2%   87032679 ±  2%  meminfo.Dirty
    360058 ±  3%     +49.9%     539639 ±  5%  meminfo.Writeback
   8086513 ± 11%     -20.1%    6459531 ± 16%  numa-numastat.node0.local_node
   8110044 ± 11%     -20.1%    6478491 ± 16%  numa-numastat.node0.numa_hit
  38832346 ± 38%     -65.2%   13495046 ±129%  cpuidle.C1.time
 2.639e+09 ± 34%     -97.9%   55443089 ± 35%  cpuidle.C1E.time
   6815395 ± 48%     -93.0%     478018 ± 46%  cpuidle.C1E.usage
     58.51 ±  8%     +58.8%      92.88        iostat.cpu.idle
     38.19 ± 13%     -99.2%       0.30 ±  8%  iostat.cpu.iowait
      3.00 ± 26%    +112.9%       6.39 ± 26%  iostat.cpu.system
     57.08 ±  9%     +35.6       92.63 ±  2%  mpstat.cpu.all.idle%
     39.52 ± 13%     -39.2        0.30 ±  8%  mpstat.cpu.all.iowait%
      3.09 ± 26%      +3.5        6.59 ± 28%  mpstat.cpu.all.sys%
     59073 ±  6%     -10.0%      53154 ± 12%  slabinfo.kmalloc-512.active_objs
      5519 ±  3%      -9.5%       4997 ±  5%  slabinfo.kmalloc-rcl-64.active_objs
      5519 ±  3%      -9.5%       4997 ±  5%  slabinfo.kmalloc-rcl-64.num_objs
     13783 ± 44%     +48.8%      20503 ± 17%  numa-meminfo.node0.Shmem
    121171 ±  9%     +88.0%     227855 ± 39%  numa-meminfo.node0.Writeback
  20293723 ±  4%     +17.7%   23891139 ±  2%  numa-meminfo.node2.Dirty
     85858 ± 58%     -74.7%      21763 ± 97%  numa-meminfo.node2.Writeback
  19927514            +9.2%   21757753 ±  2%  proc-vmstat.nr_dirty
     89137 ±  2%     +54.9%     138118 ±  2%  proc-vmstat.nr_writeback
  20019355            +9.4%   21897479 ±  2%  proc-vmstat.nr_zone_write_pending
    490616 ±  2%     -17.2%     406122 ± 20%  proc-vmstat.pgfree
     18.56 ± 22%     -34.1%      12.24 ± 50%  sched_debug.cfs_rq:/.exec_clock.max
      0.06 ± 13%     +35.7%       0.08 ± 25%  sched_debug.cfs_rq:/.nr_running.avg
      0.24 ±  6%     +17.2%       0.28 ± 14%  sched_debug.cfs_rq:/.nr_running.stddev
    130.14 ± 20%     +34.0%     174.35 ±  8%  sched_debug.cpu.curr->pid.avg
     58.25 ±  8%     +59.2%      92.75 ±  2%  vmstat.cpu.id
     37.75 ± 13%    -100.0%       0.00        vmstat.cpu.wa
     78.00 ± 13%    -100.0%       0.00        vmstat.procs.b
      7.50 ± 20%     +73.3%      13.00 ± 28%  vmstat.procs.r
      9174 ±  6%     -36.5%       5823 ±  4%  vmstat.system.cs
     22.50 ±  4%     -27.8%      16.25 ± 17%  nfsstat.Client.nfs.v3.access.percent
      4.00 ± 25%    +300.0%      16.00 ± 32%  nfsstat.Client.nfs.v3.getattr
      9.00 ± 17%    +213.9%      28.25 ± 32%  nfsstat.Client.nfs.v3.getattr.percent
     25.00 ±  4%     -27.0%      18.25 ± 19%  nfsstat.Client.nfs.v3.lookup.percent
     68.50 ±  3%     +24.1%      85.00 ± 12%  nfsstat.Client.rpc.authrefrsh
     68.50 ±  3%     +24.1%      85.00 ± 12%  nfsstat.Client.rpc.calls
      3445 ± 44%     +48.7%       5125 ± 17%  numa-vmstat.node0.nr_shmem
     30355 ±  8%     +84.7%      56063 ± 39%  numa-vmstat.node0.nr_writeback
   1573401 ± 30%     +58.1%    2486981 ± 14%  numa-vmstat.node0.numa_foreign
   5068995 ±  4%     +17.9%    5974323 ±  2%  numa-vmstat.node2.nr_dirty
     19499 ± 43%     -70.5%       5758 ±103%  numa-vmstat.node2.nr_writeback
   1097845 ± 49%     -61.6%     421721 ±109%  numa-vmstat.node2.nr_written
   5089002 ±  4%     +17.5%    5980287 ±  2%  numa-vmstat.node2.nr_zone_write_pending
    156.00 ± 22%     +67.0%     260.50 ± 13%  turbostat.Avg_MHz
      7.38 ± 19%      +4.5       11.92 ± 11%  turbostat.Busy%
      0.30 ± 39%      -0.2        0.10 ±114%  turbostat.C1%
   6807412 ± 48%     -93.0%     475690 ± 47%  turbostat.C1E
     21.27 ± 35%     -20.8        0.50 ± 12%  turbostat.C1E%
     11.07 ± 62%    +242.2%      37.88 ±  7%  turbostat.Pkg%pc2
     32.02 ±  5%     -18.0%      26.25 ±  9%  turbostat.RAMWatt
     73.75 ± 51%     -61.7%      28.25 ± 27%  interrupts.52:PCI-MSI.1572865-edge.eth0-TxRx-1
     73.75 ± 51%     -61.7%      28.25 ± 27%  interrupts.CPU1.52:PCI-MSI.1572865-edge.eth0-TxRx-1
    127528 ±  9%     -19.8%     102251 ± 14%  interrupts.CPU129.LOC:Local_timer_interrupts
      1466 ±  6%     -10.5%       1311 ±  7%  interrupts.CPU13.CAL:Function_call_interrupts
    127836 ±  9%     -20.3%     101859 ± 14%  interrupts.CPU130.LOC:Local_timer_interrupts
      1527 ±  7%     -18.0%       1252 ±  8%  interrupts.CPU149.CAL:Function_call_interrupts
      8.50 ± 74%    +802.9%      76.75 ± 45%  interrupts.CPU159.RES:Rescheduling_interrupts
      1517 ±  7%     -14.5%       1297 ±  5%  interrupts.CPU160.CAL:Function_call_interrupts
      1525 ±  7%      -9.0%       1388 ±  9%  interrupts.CPU163.CAL:Function_call_interrupts
    246.75 ± 36%     -58.2%     103.25 ± 74%  interrupts.CPU167.TLB:TLB_shootdowns
      1482 ±  4%      -9.9%       1335 ±  9%  interrupts.CPU27.CAL:Function_call_interrupts
    127435 ±  9%     -19.9%     102066 ± 14%  interrupts.CPU33.LOC:Local_timer_interrupts
    127785 ±  9%     -20.5%     101641 ± 13%  interrupts.CPU34.LOC:Local_timer_interrupts
      1488 ±  7%     -11.4%       1318 ±  7%  interrupts.CPU48.CAL:Function_call_interrupts
      1486 ±  7%     -11.1%       1321 ±  7%  interrupts.CPU49.CAL:Function_call_interrupts
      1488 ±  7%     -11.4%       1318 ±  7%  interrupts.CPU51.CAL:Function_call_interrupts
      1487 ±  7%     -11.6%       1315 ±  7%  interrupts.CPU52.CAL:Function_call_interrupts
     20.75 ± 31%    +336.1%      90.50 ± 84%  interrupts.CPU52.RES:Rescheduling_interrupts
    235.75 ± 44%     -77.2%      53.75 ± 70%  interrupts.CPU52.TLB:TLB_shootdowns
    261.75 ± 32%     -62.7%      97.75 ± 74%  interrupts.CPU53.TLB:TLB_shootdowns
      1478 ±  6%     -11.1%       1313 ±  7%  interrupts.CPU56.CAL:Function_call_interrupts
    221.75 ± 45%     -65.8%      75.75 ± 55%  interrupts.CPU59.TLB:TLB_shootdowns
    122835 ± 14%     -20.8%      97243 ± 11%  interrupts.CPU60.LOC:Local_timer_interrupts
      1485 ±  8%     -10.4%       1330 ±  7%  interrupts.CPU61.CAL:Function_call_interrupts
    239.75 ± 37%     -68.4%      75.75 ± 69%  interrupts.CPU61.TLB:TLB_shootdowns
    229.25 ± 40%     -76.6%      53.75 ± 76%  interrupts.CPU62.TLB:TLB_shootdowns
    230.75 ± 43%     -80.4%      45.25 ±116%  interrupts.CPU65.TLB:TLB_shootdowns
    226.00 ± 46%     -75.2%      56.00 ± 96%  interrupts.CPU66.TLB:TLB_shootdowns
    247.50 ± 38%     -71.3%      71.00 ± 80%  interrupts.CPU67.TLB:TLB_shootdowns
      1485 ±  7%     -20.6%       1180 ± 14%  interrupts.CPU68.CAL:Function_call_interrupts
      1512 ±  8%     -11.4%       1340 ± 10%  interrupts.CPU69.CAL:Function_call_interrupts
      1504 ±  7%     -17.7%       1237 ± 19%  interrupts.CPU70.CAL:Function_call_interrupts
    241.75 ± 35%     -54.5%     110.00 ± 51%  interrupts.CPU70.TLB:TLB_shootdowns
      1503 ±  7%     -11.2%       1335 ± 10%  interrupts.CPU71.CAL:Function_call_interrupts
    243.00 ± 36%     -78.7%      51.75 ±101%  interrupts.CPU71.TLB:TLB_shootdowns
 2.061e+09 ± 22%     +50.1%  3.093e+09 ± 13%  perf-stat.i.branch-instructions
   8999500 ±  7%     +41.2%   12704223 ± 17%  perf-stat.i.cache-misses
  1.35e+08 ± 13%     +56.9%  2.118e+08 ± 10%  perf-stat.i.cache-references
      9250 ±  6%     -39.4%       5604 ±  4%  perf-stat.i.context-switches
      3.16 ±  7%     +42.6%       4.51 ± 10%  perf-stat.i.cpi
 2.923e+10 ± 21%     +74.5%  5.101e+10 ± 13%  perf-stat.i.cpu-cycles
   4114141 ± 11%     +59.2%    6548758 ± 11%  perf-stat.i.dTLB-load-misses
 2.679e+09 ± 18%     +44.5%  3.871e+09 ± 14%  perf-stat.i.dTLB-loads
      0.14 ± 12%      +0.0        0.17 ±  3%  perf-stat.i.dTLB-store-miss-rate%
   1671837 ±  8%     +63.2%    2727790 ± 11%  perf-stat.i.dTLB-store-misses
 1.441e+09 ±  5%     +38.7%      2e+09 ±  9%  perf-stat.i.dTLB-stores
     53.65 ±  8%     +24.0       77.60 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   2095983 ±  9%     -48.7%    1074748 ± 11%  perf-stat.i.iTLB-loads
 9.371e+09 ± 18%     +45.9%  1.367e+10 ± 13%  perf-stat.i.instructions
      0.33 ±  6%     -30.2%       0.23 ±  9%  perf-stat.i.ipc
     84.85 ±  2%      +4.2       89.02 ±  2%  perf-stat.i.node-load-miss-rate%
   1289579 ±  8%     +48.7%    1917737 ± 17%  perf-stat.i.node-load-misses
    820818 ± 26%     +56.7%    1286257 ± 24%  perf-stat.i.node-store-misses
      3.12 ±  7%     +19.7%       3.74 ±  3%  perf-stat.overall.cpi
      0.12 ±  7%      +0.0        0.14 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     66.28 ±  2%     +16.7       82.93        perf-stat.overall.iTLB-load-miss-rate%
      0.32 ±  8%     -16.9%       0.27 ±  3%  perf-stat.overall.ipc
     84.57 ±  2%      +3.0       87.59        perf-stat.overall.node-load-miss-rate%
  2.08e+09 ± 22%     +49.5%  3.109e+09 ± 14%  perf-stat.ps.branch-instructions
   9040237 ±  7%     +40.0%   12660378 ± 17%  perf-stat.ps.cache-misses
  1.34e+08 ± 13%     +55.6%  2.084e+08 ± 10%  perf-stat.ps.cache-references
      9091 ±  6%     -39.6%       5487 ±  4%  perf-stat.ps.context-switches
 2.958e+10 ± 22%     +73.1%  5.122e+10 ± 14%  perf-stat.ps.cpu-cycles
   4060515 ± 10%     +58.0%    6416616 ± 11%  perf-stat.ps.dTLB-load-misses
 2.694e+09 ± 18%     +44.1%   3.88e+09 ± 14%  perf-stat.ps.dTLB-loads
   1654326 ±  8%     +61.8%    2676539 ± 11%  perf-stat.ps.dTLB-store-misses
 1.431e+09 ±  5%     +37.8%  1.971e+09 ±  9%  perf-stat.ps.dTLB-stores
   2064199 ±  9%     -48.6%    1060337 ± 11%  perf-stat.ps.iTLB-loads
 9.441e+09 ± 18%     +45.4%  1.373e+10 ± 14%  perf-stat.ps.instructions
   1299448 ±  9%     +47.9%    1922006 ± 18%  perf-stat.ps.node-load-misses
    831956 ± 26%     +55.6%    1294751 ± 24%  perf-stat.ps.node-store-misses
     17668 ± 10%     -25.2%      13217 ± 16%  softirqs.CPU10.RCU
     17346 ±  9%     -20.6%      13766 ± 16%  softirqs.CPU100.RCU
     17634 ± 11%     -28.1%      12681 ± 10%  softirqs.CPU101.RCU
     16931 ± 10%     -21.5%      13289 ± 10%  softirqs.CPU102.RCU
     17286 ± 12%     -22.3%      13425 ± 12%  softirqs.CPU103.RCU
     17593 ±  8%     -28.3%      12607 ±  7%  softirqs.CPU104.RCU
     17139 ±  9%     -29.2%      12134 ± 19%  softirqs.CPU105.RCU
     17272 ± 10%     -23.9%      13137 ± 15%  softirqs.CPU106.RCU
     16891 ±  9%     -26.3%      12452 ± 12%  softirqs.CPU107.RCU
     17028 ±  8%     -24.1%      12924 ± 11%  softirqs.CPU109.RCU
     17602 ± 10%     -16.8%      14650 ± 15%  softirqs.CPU11.RCU
     17316 ± 12%     -21.4%      13607 ± 11%  softirqs.CPU110.RCU
     17110 ±  9%     -20.8%      13545 ± 10%  softirqs.CPU111.RCU
     16315 ±  8%     -24.9%      12247 ± 10%  softirqs.CPU112.RCU
     15803 ± 10%     -23.3%      12123 ±  8%  softirqs.CPU113.RCU
     15994 ±  9%     -31.6%      10947 ±  8%  softirqs.CPU114.RCU
     15770 ±  7%     -25.3%      11775 ±  8%  softirqs.CPU115.RCU
     15796 ± 12%     -26.9%      11547 ± 10%  softirqs.CPU116.RCU
     15956 ± 11%     -20.2%      12739 ±  8%  softirqs.CPU117.RCU
     16294 ±  9%     -24.5%      12299 ± 10%  softirqs.CPU118.RCU
     16473 ± 10%     -29.3%      11644 ±  5%  softirqs.CPU119.RCU
     17745 ± 12%     -25.1%      13285 ± 10%  softirqs.CPU12.RCU
     15917 ± 10%     -32.5%      10745 ±  6%  softirqs.CPU121.RCU
     16455 ±  5%     -32.7%      11073 ±  8%  softirqs.CPU122.RCU
     16242 ±  7%     -35.9%      10412 ±  9%  softirqs.CPU123.RCU
     16082 ±  7%     -33.3%      10720 ±  7%  softirqs.CPU124.RCU
     16928 ±  7%     -36.4%      10765 ±  9%  softirqs.CPU125.RCU
     16714 ± 13%     -34.8%      10894 ±  8%  softirqs.CPU126.RCU
     16477 ±  7%     -34.0%      10872 ±  7%  softirqs.CPU127.RCU
     16450 ± 10%     -28.6%      11744 ±  5%  softirqs.CPU128.RCU
     16426 ± 13%     -28.4%      11758 ± 17%  softirqs.CPU129.RCU
     17469 ±  8%     -24.3%      13231 ± 10%  softirqs.CPU13.RCU
     17075 ± 10%     -29.4%      12054 ± 12%  softirqs.CPU130.RCU
     15298 ± 16%     -32.8%      10284 ± 16%  softirqs.CPU131.RCU
     16681 ± 12%     -37.2%      10473 ± 14%  softirqs.CPU133.RCU
     17512 ±  9%     -31.0%      12092 ± 13%  softirqs.CPU134.RCU
     16908 ± 12%     -23.7%      12896 ±  9%  softirqs.CPU135.RCU
     17693 ±  9%     -29.6%      12455 ± 10%  softirqs.CPU136.RCU
     18054 ±  5%     -27.4%      13103 ±  9%  softirqs.CPU137.RCU
     17674 ±  7%     -33.9%      11680 ±  8%  softirqs.CPU138.RCU
     17591 ±  7%     -33.8%      11651 ± 21%  softirqs.CPU139.RCU
     17345 ± 10%     -19.8%      13913 ± 10%  softirqs.CPU14.RCU
     16618 ±  8%     -24.4%      12567 ± 15%  softirqs.CPU140.RCU
     17626 ±  8%     -25.3%      13171 ± 16%  softirqs.CPU141.RCU
     17477 ±  9%     -27.9%      12604 ± 15%  softirqs.CPU142.RCU
     17161 ± 10%     -29.0%      12181 ± 12%  softirqs.CPU143.RCU
     17802 ±  5%     -33.2%      11894 ± 22%  softirqs.CPU144.RCU
      8698 ± 11%     -19.4%       7011 ± 14%  softirqs.CPU144.SCHED
     17350 ±  8%     -32.9%      11641 ± 25%  softirqs.CPU145.RCU
     17524 ±  6%     -33.3%      11692 ± 24%  softirqs.CPU146.RCU
     16418 ± 13%     -29.3%      11608 ± 30%  softirqs.CPU147.RCU
      8701 ± 14%     -24.2%       6599 ±  8%  softirqs.CPU147.SCHED
     26904 ± 14%     -20.3%      21455 ± 13%  softirqs.CPU147.TIMER
     17406 ±  7%     -34.0%      11491 ± 30%  softirqs.CPU148.RCU
      8636 ± 11%     -22.1%       6725 ±  9%  softirqs.CPU148.SCHED
     17361 ±  6%     -34.8%      11314 ± 28%  softirqs.CPU149.RCU
      8614 ± 11%     -24.1%       6539 ± 12%  softirqs.CPU149.SCHED
     26424 ± 10%     -19.2%      21356 ± 15%  softirqs.CPU149.TIMER
     17739 ±  9%     -19.7%      14236 ±  9%  softirqs.CPU15.RCU
     17169 ±  8%     -32.1%      11655 ± 19%  softirqs.CPU150.RCU
     17459 ±  7%     -38.6%      10728 ± 20%  softirqs.CPU151.RCU
     16426 ±  4%     -33.7%      10889 ± 19%  softirqs.CPU152.RCU
     17049 ±  6%     -33.1%      11408 ± 24%  softirqs.CPU153.RCU
     17222 ±  5%     -33.9%      11385 ± 30%  softirqs.CPU154.RCU
     17385 ±  6%     -37.2%      10917 ± 28%  softirqs.CPU155.RCU
      8545 ± 12%     -18.9%       6930 ± 15%  softirqs.CPU155.SCHED
     16934 ±  8%     -40.4%      10087 ± 34%  softirqs.CPU156.RCU
      8809 ± 11%     -26.8%       6450 ±  9%  softirqs.CPU156.SCHED
     25717 ± 11%     -19.3%      20748 ± 14%  softirqs.CPU156.TIMER
     17266 ±  8%     -36.1%      11029 ± 22%  softirqs.CPU157.RCU
      8561 ± 11%     -20.0%       6848 ± 11%  softirqs.CPU157.SCHED
     17651 ±  6%     -33.8%      11682 ± 19%  softirqs.CPU158.RCU
     17274 ±  7%     -30.7%      11964 ± 19%  softirqs.CPU159.RCU
     17058 ±  6%     -24.8%      12822 ± 10%  softirqs.CPU16.RCU
     15882 ±  7%     -33.7%      10525 ± 15%  softirqs.CPU160.RCU
     15895 ±  8%     -33.5%      10564 ± 21%  softirqs.CPU161.RCU
      8611 ± 10%     -19.3%       6946 ±  6%  softirqs.CPU161.SCHED
     15853 ±  5%     -38.5%       9756 ± 13%  softirqs.CPU162.RCU
     15993 ±  5%     -41.5%       9352 ± 22%  softirqs.CPU163.RCU
      8774 ± 12%     -24.4%       6636 ± 12%  softirqs.CPU163.SCHED
     15521 ±  5%     -36.8%       9814 ± 17%  softirqs.CPU164.RCU
     16132 ±  6%     -31.8%      10996 ± 23%  softirqs.CPU165.RCU
      8582 ± 12%     -20.1%       6853 ± 15%  softirqs.CPU165.SCHED
     15843 ±  7%     -34.2%      10427 ± 31%  softirqs.CPU166.RCU
      8637 ± 10%     -23.9%       6570 ± 17%  softirqs.CPU166.SCHED
     15791 ±  8%     -36.1%      10082 ± 19%  softirqs.CPU167.RCU
     15957 ±  6%     -33.9%      10554 ± 14%  softirqs.CPU168.RCU
     16040 ±  6%     -36.4%      10196 ± 11%  softirqs.CPU169.RCU
     16683 ± 10%     -25.4%      12452 ±  8%  softirqs.CPU17.RCU
     15717 ±  9%     -31.6%      10748 ± 12%  softirqs.CPU170.RCU
     16055 ±  6%     -33.4%      10695 ± 13%  softirqs.CPU171.RCU
     15680 ±  7%     -33.0%      10512 ± 13%  softirqs.CPU172.RCU
     14796 ±  4%     -32.1%      10041 ± 13%  softirqs.CPU173.RCU
     16681 ±  5%     -38.5%      10259 ± 10%  softirqs.CPU174.RCU
     16065 ±  6%     -33.4%      10691 ± 20%  softirqs.CPU175.RCU
     16644 ±  6%     -33.1%      11128 ±  8%  softirqs.CPU176.RCU
     15997 ± 10%     -34.4%      10492 ± 20%  softirqs.CPU177.RCU
     16663 ±  6%     -26.6%      12231 ±  8%  softirqs.CPU178.RCU
     16703 ±  8%     -31.6%      11422 ± 10%  softirqs.CPU179.RCU
     16760 ±  8%     -31.7%      11450 ±  7%  softirqs.CPU18.RCU
     16917 ±  6%     -29.5%      11921 ±  7%  softirqs.CPU180.RCU
     17130 ±  6%     -33.3%      11428 ±  8%  softirqs.CPU181.RCU
     16744 ±  5%     -24.2%      12691 ±  9%  softirqs.CPU182.RCU
     17388 ±  4%     -24.5%      13132 ± 10%  softirqs.CPU183.RCU
     17309 ±  5%     -26.0%      12803 ± 16%  softirqs.CPU184.RCU
     17498 ±  4%     -24.6%      13200 ± 10%  softirqs.CPU185.RCU
     17212 ±  5%     -32.6%      11599 ±  9%  softirqs.CPU186.RCU
     17234 ±  5%     -26.7%      12631 ±  8%  softirqs.CPU187.RCU
     17067 ±  8%     -26.5%      12551 ± 10%  softirqs.CPU188.RCU
     17363 ±  5%     -23.1%      13345 ± 10%  softirqs.CPU189.RCU
     16685 ±  9%     -26.8%      12219 ±  9%  softirqs.CPU19.RCU
     17353 ±  5%     -28.6%      12387 ± 15%  softirqs.CPU190.RCU
     17081 ±  5%     -30.9%      11799 ± 14%  softirqs.CPU191.RCU
     17866 ±  8%     -23.9%      13602 ± 13%  softirqs.CPU2.RCU
     16101 ±  9%     -25.4%      12014 ± 10%  softirqs.CPU20.RCU
     16771 ± 10%     -23.1%      12890 ± 10%  softirqs.CPU21.RCU
     16829 ±  9%     -24.4%      12716 ±  9%  softirqs.CPU22.RCU
     16730 ±  9%     -30.4%      11642 ±  9%  softirqs.CPU23.RCU
     17682 ±  6%     -29.7%      12438 ±  9%  softirqs.CPU24.RCU
     16563 ±  9%     -31.8%      11297 ±  6%  softirqs.CPU25.RCU
     16318 ±  6%     -29.0%      11584 ±  8%  softirqs.CPU26.RCU
     16593 ±  7%     -35.2%      10751 ±  8%  softirqs.CPU27.RCU
     16716 ±  8%     -32.3%      11319 ±  8%  softirqs.CPU28.RCU
     16692 ±  7%     -33.8%      11053 ± 10%  softirqs.CPU29.RCU
     16779 ± 10%     -32.6%      11308 ± 10%  softirqs.CPU30.RCU
     17070 ±  8%     -34.2%      11229 ±  8%  softirqs.CPU31.RCU
     17336 ± 10%     -29.0%      12308 ±  8%  softirqs.CPU32.RCU
     17183 ± 12%     -29.2%      12161 ± 14%  softirqs.CPU33.RCU
     17278 ± 12%     -27.6%      12508 ± 12%  softirqs.CPU34.RCU
     16499 ± 13%     -32.3%      11175 ± 15%  softirqs.CPU35.RCU
     17257 ±  8%     -34.3%      11335 ± 16%  softirqs.CPU36.RCU
     17281 ± 11%     -36.9%      10907 ± 14%  softirqs.CPU37.RCU
     17813 ±  8%     -29.8%      12513 ± 13%  softirqs.CPU38.RCU
     17630 ±  9%     -22.5%      13668 ± 13%  softirqs.CPU4.RCU
     18282 ±  8%     -25.0%      13719 ±  9%  softirqs.CPU40.RCU
     18757 ±  5%     -27.3%      13638 ±  7%  softirqs.CPU41.RCU
     18228 ±  8%     -32.1%      12372 ±  8%  softirqs.CPU42.RCU
     18059 ±  7%     -30.8%      12491 ± 18%  softirqs.CPU43.RCU
     17332 ±  9%     -26.2%      12793 ± 10%  softirqs.CPU44.RCU
     18087 ±  8%     -23.6%      13812 ± 10%  softirqs.CPU45.RCU
     17951 ±  9%     -26.6%      13179 ± 13%  softirqs.CPU46.RCU
     17931 ± 10%     -28.8%      12770 ± 10%  softirqs.CPU47.RCU
     18161 ±  5%     -33.5%      12079 ± 22%  softirqs.CPU48.RCU
     17888 ±  7%     -35.3%      11570 ± 25%  softirqs.CPU49.RCU
     17739 ± 10%     -36.5%      11258 ± 22%  softirqs.CPU5.RCU
     18310 ±  7%     -35.4%      11829 ± 22%  softirqs.CPU50.RCU
      9034 ± 12%     -23.8%       6885 ± 10%  softirqs.CPU50.SCHED
     17431 ± 10%     -29.5%      12285 ± 25%  softirqs.CPU51.RCU
      8741 ± 14%     -21.7%       6842 ±  6%  softirqs.CPU51.SCHED
     17937 ±  7%     -32.8%      12053 ± 27%  softirqs.CPU52.RCU
      8642 ± 11%     -23.0%       6657 ± 10%  softirqs.CPU52.SCHED
     17894 ±  6%     -32.7%      12048 ± 24%  softirqs.CPU53.RCU
     17514 ±  8%     -31.9%      11929 ± 19%  softirqs.CPU54.RCU
     18232 ±  7%     -36.8%      11529 ± 21%  softirqs.CPU55.RCU
      8666 ± 11%     -19.8%       6952 ± 12%  softirqs.CPU55.SCHED
     16922 ±  4%     -31.6%      11582 ± 12%  softirqs.CPU56.RCU
     17079 ±  4%     -32.1%      11597 ± 26%  softirqs.CPU57.RCU
      8602 ± 11%     -20.8%       6810 ± 19%  softirqs.CPU57.SCHED
     17954 ±  4%     -32.8%      12068 ± 32%  softirqs.CPU58.RCU
      8603 ± 10%     -19.0%       6968 ± 15%  softirqs.CPU58.SCHED
     18021 ±  6%     -36.7%      11401 ± 27%  softirqs.CPU59.RCU
     17363 ± 10%     -22.5%      13451 ± 10%  softirqs.CPU6.RCU
     18009 ±  9%     -42.3%      10386 ± 35%  softirqs.CPU60.RCU
      8868 ± 16%     -34.4%       5816 ± 23%  softirqs.CPU60.SCHED
     27562 ± 13%     -27.5%      19977 ± 20%  softirqs.CPU60.TIMER
      8601 ± 11%     -23.5%       6579 ± 13%  softirqs.CPU61.SCHED
     17526 ±  4%     -34.0%      11563 ± 23%  softirqs.CPU62.RCU
      8489 ± 12%     -23.2%       6523 ± 13%  softirqs.CPU62.SCHED
     17711 ±  6%     -26.5%      13023 ± 13%  softirqs.CPU63.RCU
     16525 ±  6%     -32.3%      11185 ± 14%  softirqs.CPU64.RCU
     16419 ±  8%     -31.5%      11251 ± 15%  softirqs.CPU65.RCU
     15997 ±  5%     -37.4%      10017 ± 12%  softirqs.CPU66.RCU
     16738 ±  4%     -39.4%      10147 ± 14%  softirqs.CPU67.RCU
      8722 ± 12%     -18.1%       7146 ± 18%  softirqs.CPU67.SCHED
     16105 ±  5%     -35.7%      10362 ± 15%  softirqs.CPU68.RCU
     16828 ±  5%     -35.3%      10881 ± 19%  softirqs.CPU69.RCU
      8596 ± 12%     -19.8%       6894 ± 10%  softirqs.CPU69.SCHED
     17992 ± 10%     -22.4%      13963 ± 12%  softirqs.CPU7.RCU
     16288 ±  7%     -39.6%       9831 ± 36%  softirqs.CPU70.RCU
      8635 ± 11%     -18.1%       7073 ± 13%  softirqs.CPU70.SCHED
     27396 ± 11%     -17.3%      22652 ± 14%  softirqs.CPU70.TIMER
     16328 ±  7%     -37.6%      10192 ± 22%  softirqs.CPU71.RCU
      8632 ± 11%     -22.8%       6663 ± 12%  softirqs.CPU71.SCHED
     16613 ±  5%     -33.1%      11113 ± 15%  softirqs.CPU72.RCU
     16568 ±  5%     -35.7%      10653 ± 10%  softirqs.CPU73.RCU
     16522 ±  8%     -30.1%      11543 ± 10%  softirqs.CPU74.RCU
     16523 ±  4%     -32.0%      11235 ± 11%  softirqs.CPU75.RCU
     16325 ±  8%     -31.4%      11204 ± 12%  softirqs.CPU76.RCU
     16551 ±  7%     -38.4%      10199 ± 13%  softirqs.CPU77.RCU
     16853 ±  5%     -37.0%      10624 ± 13%  softirqs.CPU78.RCU
     16340 ±  5%     -35.7%      10502 ± 16%  softirqs.CPU79.RCU
     17752 ± 10%     -27.8%      12811 ± 10%  softirqs.CPU8.RCU
     17211 ±  5%     -32.9%      11544 ± 10%  softirqs.CPU80.RCU
     16913 ±  8%     -33.0%      11325 ± 18%  softirqs.CPU81.RCU
     17273 ±  6%     -24.4%      13067 ±  9%  softirqs.CPU82.RCU
     17040 ±  7%     -31.1%      11741 ± 10%  softirqs.CPU83.RCU
     17521 ±  6%     -28.3%      12571 ±  7%  softirqs.CPU84.RCU
     18152 ±  6%     -32.6%      12241 ±  6%  softirqs.CPU85.RCU
     17159 ±  5%     -22.9%      13231 ± 10%  softirqs.CPU86.RCU
     17618 ±  4%     -22.1%      13721 ±  9%  softirqs.CPU87.RCU
     17834 ±  6%     -24.5%      13456 ± 13%  softirqs.CPU88.RCU
     18055 ±  3%     -24.7%      13594 ± 11%  softirqs.CPU89.RCU
     17707 ±  9%     -31.1%      12207 ± 20%  softirqs.CPU9.RCU
     17671 ±  5%     -32.0%      12017 ± 10%  softirqs.CPU90.RCU
     17425 ±  7%     -25.8%      12925 ±  8%  softirqs.CPU91.RCU
     17197 ±  6%     -23.9%      13086 ±  9%  softirqs.CPU92.RCU
     17975 ±  5%     -22.9%      13865 ±  8%  softirqs.CPU93.RCU
     18318 ±  5%     -29.6%      12889 ± 15%  softirqs.CPU94.RCU
     17833 ±  5%     -32.2%      12094 ± 16%  softirqs.CPU95.RCU
     18019 ±  7%     -40.1%      10798 ± 33%  softirqs.CPU96.RCU
     17300 ±  9%     -28.0%      12451 ± 13%  softirqs.CPU97.RCU
     17407 ± 10%     -22.1%      13558 ± 16%  softirqs.CPU98.RCU
     16855 ± 10%     -22.2%      13115 ± 12%  softirqs.CPU99.RCU
   3268149 ±  7%     -29.9%    2291114 ± 12%  softirqs.RCU


                                                                                
                                 fio.write_bw_MBps                              
                                                                                
  3500 +-+------------------------------------------------------------------+   
       O  O O  O  O O  O    O  O  O O  O  O O  O O  O  O O  O  O O  O  O O  O   
  3000 +-+                                                                  |   
       |                                                                    |   
  2500 +-+     +..+.+..+..+.          .+.. .+.. .+..+..+.+..+..             |   
       |       :            +..+..+.+.    +    +               +.+          |   
  2000 +-+    :                                                             |   
       |:     :                                                             |   
  1500 +-+    :                                                             |   
       | :    :                                                             |   
  1000 +-+   :                                                              |   
       | :   :                                                              |   
   500 +-+   :                                                              |   
       |  : :                                                               |   
     0 +-+----------------O-------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   fio.write_iops                               
                                                                                
  900000 +-+----------------------------------------------------------------+   
         O  O O  O O  O O     O O  O O  O  O O  O O  O O  O  O O  O O  O O  O   
  800000 +-+                                                                |   
  700000 +-+                                                                |   
         |       +.+..+.+..+..              .+.. .+.. .+..                  |   
  600000 +-+     :            +.+..+.+..+..+    +    +    +..+.+..+         |   
  500000 +-+    :                                                           |   
         |:     :                                                           |   
  400000 +-+    :                                                           |   
  300000 +-+    :                                                           |   
         | :   :                                                            |   
  200000 +-+   :                                                            |   
  100000 +-+   :                                                            |   
         |  : :                                                             |   
       0 +-+---------------O------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.write_clat_stddev                          
                                                                                
  1.2e+07 +-+---------------------------------------------------------------+   
          |                                                                 |   
    1e+07 +-+       O    O                 O                                |   
          O            O      O  O    O       O    O    O O  O O  O O  O O  O   
          |  O O  O                O    O       O    O                      |   
    8e+06 +-+                                                               |   
          |                                                                 |   
    6e+06 +-+                                                               |   
          |                                                                 |   
    4e+06 +-+                                                               |   
          |                                                                 |   
          |       +.+..+.+..+.+..+.+..+.+..+..+.+..+.+..+.+..+.+..+         |   
    2e+06 +-+    +                                                          |   
          | +   +                                                           |   
        0 +-+---------------O-----------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.latency_50ms_                              
                                                                                
  0.35 +-+------------------------------------------------------------------+   
       |                                                                    |   
   0.3 +-+            .+..    .+.. .+..   +.  .+.                           |   
       |         .+.+.     .+.    +     ..  +.   +..+..+.+..  .+.+          |   
  0.25 +-+     +.         +            +                    +.              |   
       |:      :                                                            |   
   0.2 +-+    :                                                             |   
       |:     :                                                             |   
  0.15 +-+    :                                                             |   
       | :    :                                                             |   
   0.1 +-+   :                                                              |   
       | :   :                                                              |   
  0.05 +-+   :                                                              |   
       |  : :                                                               |   
     0 O-+O-O--O--O-O--O--O-O--O--O-O--O--O-O--O-O--O--O-O--O--O-O--O--O-O--O   
                                                                                
                                                                                                                                                                
                              fio.time.elapsed_time                             
                                                                                
  60 +-+--------------------------------------------------------------------+   
     |       +..+..+.+..+..+..+.+..+..+..+.+..+..+.+..+..+..+.+..+          |   
  50 +-+     :                                                              |   
     |       :                                                              |   
     O:     :   O          O                       O                        |   
  40 +-+O O :O     O O        O O  O  O  O O  O  O    O  O  O O  O  O  O O  O   
     |:     :                                                               |   
  30 +-+    :                                                               |   
     | :   :                                                                |   
  20 +-+   :                                                                |   
     | :   :                                                                |   
     | :   :                                                                |   
  10 +-+: :                                                                 |   
     |  : :                                                                 |   
   0 +-+----------------O---------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            fio.time.elapsed_time.max                           
                                                                                
  60 +-+--------------------------------------------------------------------+   
     |       +..+..+.+..+..+..+.+..+..+..+.+..+..+.+..+..+..+.+..+          |   
  50 +-+     :                                                              |   
     |       :                                                              |   
     O:     :   O          O                       O                        |   
  40 +-+O O :O     O O        O O  O  O  O O  O  O    O  O  O O  O  O  O O  O   
     |:     :                                                               |   
  30 +-+    :                                                               |   
     | :   :                                                                |   
  20 +-+   :                                                                |   
     | :   :                                                                |   
     | :   :                                                                |   
  10 +-+: :                                                                 |   
     |  : :                                                                 |   
   0 +-+----------------O---------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         fio.time.voluntary_context_switches                    
                                                                                
  140000 +-+----------------------------------------------------------------+   
         |                     .+.. .+                           .+         |   
  120000 +-+       +.. .+..   +    +  +    +.+..+.+..    .+.. .+.           |   
         |        +   +     ..         + ..          +.+.    +              |   
  100000 +-+     +         +            +                                   |   
         |:      :                                                          |   
   80000 +-+    :                                                           |   
         |:     :                                                           |   
   60000 +-+    :                                                           |   
         | :    :                                                           |   
   40000 +-+   :                                                            |   
         | :   :                                                            |   
   20000 +-+   :                                                            |   
         O  O O  O O  O O     O O  O O  O  O O  O O  O O  O  O O  O O  O O  O   
       0 +-+---------------O------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2ap1: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
  4M/gcc-7/performance/1SSD/btrfs/sync/x86_64-rhel-7.6/100%/debian-x86_64-2019-09-23.cgz/300s/randwrite/lkp-csl-2ap1/128G/fio-basic/0x4000021

commit: 
  8005803a2c (" ARC fixes for 5.4-rc6")
  2b871886bb ("writeback: add elastic bdi in cgwb bdp")

8005803a2ca0af49 2b871886bb64af404a6a6a4db83 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          3:4          -75%            :4     kmsg.ACPI_Error
          1:4          -25%            :4     kmsg.ipmi_si_dmi-ipmi-si.#:IRQ_index#not_found
         %stddev     %change         %stddev
             \          |                \  
      7.28 ± 10%      +7.1       14.34 ± 12%  fio.latency_1000ms%
      0.55 ± 54%     +16.3       16.83 ± 25%  fio.latency_100ms%
      3.56 ± 43%      -3.1        0.45 ± 38%  fio.latency_10ms%
      1.17 ± 51%      +4.7        5.92 ±  5%  fio.latency_2000ms%
      5.44 ± 23%      -4.5        0.98 ± 25%  fio.latency_20ms%
      4.20 ± 10%      -2.4        1.81 ± 27%  fio.latency_250ms%
      2.09 ± 41%      -1.7        0.34 ± 55%  fio.latency_2ms%
      1.97 ± 70%      -1.7        0.28 ± 66%  fio.latency_4ms%
     11.03 ± 11%      -7.6        3.43 ± 13%  fio.latency_500ms%
     52.90 ±  2%     -10.6       42.33 ±  2%  fio.latency_750ms%
     14292 ±  3%    +138.0%      34013 ±  3%  fio.time.involuntary_context_switches
    244517 ±  2%     +26.4%     309176 ± 19%  fio.time.minor_page_faults
    365.00 ±  2%    +945.1%       3814        fio.time.percent_of_cpu_this_job_got
    344.09         +1394.5%       5142        fio.time.system_time
    159.16           -20.5%     126.55        fio.time.user_time
    211440 ±  4%  +12139.1%   25878346        fio.time.voluntary_context_switches
      1361 ±  2%      +9.8%       1495        fio.write_bw_MBps
 7.277e+08 ±  2%     +12.1%  8.158e+08        fio.write_clat_90%_us
 8.137e+08 ±  2%     +31.3%  1.068e+09        fio.write_clat_95%_us
  9.93e+08 ±  8%     +58.4%  1.573e+09        fio.write_clat_99%_us
 4.733e+08            +6.8%  5.053e+08        fio.write_clat_mean_us
 2.929e+08 ±  2%     +26.0%   3.69e+08        fio.write_clat_stddev
    340.39 ±  2%      +9.8%     373.86        fio.write_iops
    404578 ±  2%     +18.4%     478973        meminfo.SUnreclaim
    683332 ±  4%     +14.1%     779620 ±  2%  meminfo.Slab
     61.12 ±  7%     +32.7%      81.10 ±  2%  iostat.cpu.idle
     37.22 ± 11%     -98.9%       0.42 ± 10%  iostat.cpu.iowait
      1.20 ± 11%   +1401.5%      18.04 ± 10%  iostat.cpu.system
 7.766e+08 ± 63%     -92.6%   57276687 ± 16%  cpuidle.C1.time
  36109306 ± 38%     +89.6%   68466169 ± 13%  cpuidle.C1E.usage
 1.322e+10 ± 58%     -72.4%   3.65e+09 ±148%  cpuidle.C6.time
    309809 ± 70%   +3060.1%    9790140        cpuidle.POLL.usage
     60.65 ±  7%     +20.2       80.86 ±  2%  mpstat.cpu.all.idle%
     37.69 ± 11%     -37.3        0.43 ± 10%  mpstat.cpu.all.iowait%
      0.02 ± 52%      -0.0        0.00 ± 91%  mpstat.cpu.all.soft%
      1.18 ± 10%     +17.1       18.27 ± 11%  mpstat.cpu.all.sys%
     58.00 ± 19%    +434.5%     310.00 ± 10%  turbostat.Avg_MHz
      5.89 ±  7%     +15.8       21.69 ±  8%  turbostat.Busy%
      1927 ± 14%     +45.6%       2806 ±  2%  turbostat.Bzy_MHz
  18701743 ± 35%     +86.4%   34866559 ± 14%  turbostat.C1E
     84.37 ±  6%     -12.2%      74.12 ±  8%  turbostat.CPU%c1
    100.80 ± 11%     +38.2%     139.30 ±  3%  turbostat.PkgWatt
     60.75 ±  7%     +32.9%      80.75 ±  2%  vmstat.cpu.id
     36.75 ± 11%    -100.0%       0.00        vmstat.cpu.wa
     95.75 ± 12%     -99.5%       0.50 ±100%  vmstat.procs.b
      3.75 ± 22%    +773.3%      32.75 ±  8%  vmstat.procs.r
      9422 ± 11%   +5021.4%     482572 ± 10%  vmstat.system.cs
    371261 ±  3%      +5.5%     391573        vmstat.system.in
   3402172 ± 16%     +23.2%    4190048 ± 11%  numa-meminfo.node0.Active(anon)
   3316651 ± 16%     +20.9%    4010499 ± 10%  numa-meminfo.node0.AnonHugePages
   3401185 ± 16%     +23.2%    4189532 ± 11%  numa-meminfo.node0.AnonPages
   3039158 ± 17%     +30.4%    3964157 ± 10%  numa-meminfo.node0.Writeback
    530.50 ± 80%  +13622.7%      72799 ± 31%  numa-meminfo.node2.Active(file)
     59179 ±  9%     +22.5%      72467 ±  6%  numa-meminfo.node2.KReclaimable
     59179 ±  9%     +22.5%      72467 ±  6%  numa-meminfo.node2.SReclaimable
     89228 ±  3%     +31.0%     116907 ± 14%  numa-meminfo.node2.SUnreclaim
    148408 ±  3%     +27.6%     189375 ±  8%  numa-meminfo.node2.Slab
   3097550 ± 18%     +26.3%    3910836 ± 11%  numa-meminfo.node2.Writeback
      5811 ±  3%     +15.5%       6714 ±  5%  numa-meminfo.node3.KernelStack
   2853093 ± 15%     +34.0%    3821849 ± 10%  numa-meminfo.node3.Writeback
   8065593 ± 11%     -29.0%    5729949 ±  2%  numa-numastat.node0.local_node
   8087579 ± 11%     -28.8%    5761825 ±  2%  numa-numastat.node0.numa_hit
    241616 ± 48%    +804.9%    2186401 ±  3%  numa-numastat.node0.numa_miss
    263601 ± 44%    +741.5%    2218277 ±  3%  numa-numastat.node0.other_node
   7821081 ± 13%     -20.1%    6249152 ±  4%  numa-numastat.node1.local_node
   7846492 ± 13%     -20.0%    6277433 ±  4%  numa-numastat.node1.numa_hit
    329452 ±120%    +605.1%    2322911 ±  5%  numa-numastat.node2.numa_foreign
   7986054 ±  8%     -18.5%    6512107 ±  2%  numa-numastat.node3.local_node
    278221 ± 55%    +669.0%    2139580 ±  2%  numa-numastat.node3.numa_foreign
   8009821 ±  8%     -18.5%    6524937 ±  2%  numa-numastat.node3.numa_hit
    517534 ±118%    +384.9%    2509657 ±  7%  numa-numastat.node3.numa_miss
    541304 ±114%    +366.0%    2522489 ±  6%  numa-numastat.node3.other_node
     28451            +5.8%      30112        proc-vmstat.nr_kernel_stack
    101204 ±  2%     +18.3%     119705        proc-vmstat.nr_slab_unreclaimable
   3661968 ± 23%    +163.0%    9630675 ±  2%  proc-vmstat.numa_foreign
     14002 ± 37%    +462.7%      78790 ± 76%  proc-vmstat.numa_hint_faults
      7398 ± 70%    +649.8%      55472 ± 77%  proc-vmstat.numa_hint_faults_local
  30963779 ±  2%     -19.2%   25006064        proc-vmstat.numa_hit
     11764           +36.3%      16031        proc-vmstat.numa_huge_pte_updates
  30868842 ±  2%     -19.3%   24911053        proc-vmstat.numa_local
   3661968 ± 23%    +163.0%    9630675 ±  2%  proc-vmstat.numa_miss
   3756905 ± 23%    +158.9%    9725686 ±  2%  proc-vmstat.numa_other
     31769 ± 29%     +79.0%      56863 ± 39%  proc-vmstat.numa_pages_migrated
   6074115           +37.5%    8349461        proc-vmstat.numa_pte_updates
     38759 ±  3%     +46.7%      56870 ±  2%  proc-vmstat.pgactivate
     31769 ± 29%     +79.0%      56863 ± 39%  proc-vmstat.pgmigrate_success
      5.97 ± 93%      -6.0        0.00        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     11.01 ± 79%      -5.2        5.85 ±173%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     12.54 ± 82%      -3.6        8.93 ±173%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      6.92 ± 70%      -3.2        3.77 ±173%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      3.55 ±100%      -1.8        1.76 ±173%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      4.88 ±100%      -1.5        3.33 ±173%  perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.do_signal
      4.88 ±100%      -1.5        3.33 ±173%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal
      4.88 ±100%      -1.5        3.33 ±173%  perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit
      4.88 ±100%      -1.5        3.33 ±173%  perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit
      4.03 ±100%      +2.8        6.87 ±173%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.99 ± 93%      -5.9        0.06 ±173%  perf-profile.children.cycles-pp.poll_idle
      4.88 ±100%      -1.5        3.33 ±173%  perf-profile.children.cycles-pp.task_work_run
      4.88 ±100%      -1.5        3.33 ±173%  perf-profile.children.cycles-pp.__fput
      4.88 ±100%      -1.5        3.33 ±173%  perf-profile.children.cycles-pp.perf_release
      4.88 ±100%      -1.5        3.33 ±173%  perf-profile.children.cycles-pp.perf_event_release_kernel
      4.10 ±100%      +2.8        6.92 ±173%  perf-profile.children.cycles-pp.menu_select
      5.95 ± 94%      -5.9        0.06 ±173%  perf-profile.self.cycles-pp.poll_idle
    848974 ± 16%     +23.3%    1046745 ± 11%  numa-vmstat.node0.nr_active_anon
    848721 ± 16%     +23.3%    1046616 ± 11%  numa-vmstat.node0.nr_anon_pages
      1614 ± 16%     +21.3%       1958 ± 10%  numa-vmstat.node0.nr_anon_transparent_hugepages
    759174 ± 17%     +30.5%     990747 ± 10%  numa-vmstat.node0.nr_writeback
    848948 ± 16%     +23.3%    1046713 ± 11%  numa-vmstat.node0.nr_zone_active_anon
   6442701 ±  8%     -24.5%    4865030 ±  4%  numa-vmstat.node0.numa_hit
   6421546 ±  8%     -24.7%    4834337 ±  4%  numa-vmstat.node0.numa_local
    187129 ± 43%    +720.9%    1536098 ±  8%  numa-vmstat.node0.numa_miss
    208292 ± 39%    +652.2%    1566800 ±  8%  numa-vmstat.node0.numa_other
    133.25 ± 79%  +13519.5%      18148 ± 31%  numa-vmstat.node2.nr_active_file
     14813 ±  9%     +22.3%      18120 ±  6%  numa-vmstat.node2.nr_slab_reclaimable
     22315 ±  3%     +31.0%      29239 ± 14%  numa-vmstat.node2.nr_slab_unreclaimable
    773192 ± 18%     +26.5%     977871 ± 11%  numa-vmstat.node2.nr_writeback
    133.25 ± 79%  +13519.3%      18147 ± 31%  numa-vmstat.node2.nr_zone_active_file
    257200 ±118%    +538.8%    1642918 ±  6%  numa-vmstat.node2.numa_foreign
      5814 ±  3%     +15.4%       6711 ±  5%  numa-vmstat.node3.nr_kernel_stack
    711323 ± 15%     +34.2%     954878 ± 10%  numa-vmstat.node3.nr_writeback
    217267 ± 45%    +591.7%    1502900 ±  3%  numa-vmstat.node3.numa_foreign
   6335015 ±  8%     -15.6%    5345281 ±  6%  numa-vmstat.node3.numa_hit
   6224147 ±  9%     -15.7%    5245010 ±  6%  numa-vmstat.node3.numa_local
    387946 ±113%    +357.1%    1773190 ±  7%  numa-vmstat.node3.numa_miss
    498820 ± 89%    +275.6%    1873464 ±  7%  numa-vmstat.node3.numa_other
     38823 ± 10%     -17.2%      32153 ±  7%  softirqs.CPU1.RCU
     36924 ±  8%     -17.2%      30558 ±  2%  softirqs.CPU100.RCU
     36825 ± 10%     -17.0%      30577 ±  2%  softirqs.CPU101.RCU
     36788 ±  9%     -17.7%      30284 ±  2%  softirqs.CPU102.RCU
     36151 ±  8%     -15.9%      30411 ±  2%  softirqs.CPU103.RCU
     36135 ±  9%     -16.8%      30072 ±  2%  softirqs.CPU105.RCU
     36255 ±  9%     -16.1%      30409 ±  2%  softirqs.CPU107.RCU
     36656 ±  9%     -17.9%      30103 ±  2%  softirqs.CPU108.RCU
     37601 ±  9%     -17.3%      31079 ±  3%  softirqs.CPU11.RCU
     35348 ±  9%     -16.7%      29433 ±  3%  softirqs.CPU145.RCU
     73079 ± 19%     -18.1%      59824 ± 12%  softirqs.CPU165.TIMER
     37407 ± 10%     -17.3%      30954        softirqs.CPU3.RCU
     36441 ±  9%     -16.7%      30339 ±  4%  softirqs.CPU38.RCU
     36636 ± 10%     -19.1%      29652 ±  6%  softirqs.CPU44.RCU
     36899 ±  9%     -14.8%      31442 ±  4%  softirqs.CPU48.RCU
     35913 ±  9%     -14.7%      30646 ±  4%  softirqs.CPU49.RCU
     37747 ±  9%     -18.8%      30643 ±  3%  softirqs.CPU5.RCU
     35754 ±  9%     -14.2%      30681 ±  3%  softirqs.CPU55.RCU
     37656 ± 10%     -18.0%      30886 ±  2%  softirqs.CPU6.RCU
     36154 ±  9%     -14.6%      30876 ±  4%  softirqs.CPU60.RCU
     36509 ±  9%     -16.6%      30434 ±  2%  softirqs.CPU7.RCU
     33703 ±  8%     -11.2%      29936 ±  2%  softirqs.CPU97.RCU
     36870 ±  9%     -16.5%      30800        softirqs.CPU99.RCU
     30.31 ± 34%     -57.7%      12.83 ± 57%  perf-stat.i.MPKI
 8.307e+08 ± 15%    +308.5%  3.393e+09 ±  9%  perf-stat.i.branch-instructions
      2.70 ± 51%      -1.4        1.26 ± 68%  perf-stat.i.branch-miss-rate%
  39605408 ±  9%     +70.3%   67449576 ±  8%  perf-stat.i.cache-misses
      9463 ± 11%   +5275.8%     508737 ± 11%  perf-stat.i.context-switches
      3.62 ± 16%     +59.6%       5.77 ±  4%  perf-stat.i.cpi
 1.255e+10 ± 21%    +764.0%  1.084e+11 ±  9%  perf-stat.i.cpu-cycles
     16.65 ± 16%   +9658.8%       1624 ± 13%  perf-stat.i.cpu-migrations
    859.74 ± 31%     +89.5%       1629 ±  8%  perf-stat.i.cycles-between-cache-misses
      0.18 ± 71%      -0.1        0.06 ±150%  perf-stat.i.dTLB-load-miss-rate%
 1.179e+09 ± 13%    +226.5%   3.85e+09 ±  8%  perf-stat.i.dTLB-loads
 5.527e+08 ±  6%     +87.6%  1.037e+09 ±  8%  perf-stat.i.dTLB-stores
   2602703 ±  9%     +93.4%    5032916 ±  7%  perf-stat.i.iTLB-load-misses
   2390464 ±  9%     +89.8%    4536835 ±  6%  perf-stat.i.iTLB-loads
 4.143e+09 ± 12%    +256.5%  1.477e+10 ±  9%  perf-stat.i.instructions
      1609 ± 11%     +59.1%       2561 ±  8%  perf-stat.i.instructions-per-iTLB-miss
      0.33 ± 14%     -29.0%       0.24 ±  9%  perf-stat.i.ipc
      3373           +11.3%       3755 ±  8%  perf-stat.i.minor-faults
     65.65 ±  3%     +13.9       79.58 ±  2%  perf-stat.i.node-load-miss-rate%
   4059119 ± 15%    +211.1%   12626294 ±  9%  perf-stat.i.node-load-misses
   3031524 ± 12%     +54.3%    4676716 ±  6%  perf-stat.i.node-loads
     60.50 ± 10%     +15.0       75.50 ±  7%  perf-stat.i.node-store-miss-rate%
   2513313 ± 16%    +136.7%    5949610 ± 11%  perf-stat.i.node-store-misses
      3373           +11.3%       3756 ±  8%  perf-stat.i.page-faults
     24.85 ± 26%     -65.0%       8.70 ± 16%  perf-stat.overall.MPKI
      1.97 ± 44%      -1.4        0.56 ± 27%  perf-stat.overall.branch-miss-rate%
      3.02 ± 13%    +143.4%       7.34        perf-stat.overall.cpi
    317.52 ± 18%    +406.0%       1606 ±  3%  perf-stat.overall.cycles-between-cache-misses
      1595 ± 10%     +84.1%       2936 ±  5%  perf-stat.overall.instructions-per-iTLB-miss
      0.34 ± 15%     -59.8%       0.14        perf-stat.overall.ipc
     57.08 ±  5%     +15.8       72.88        perf-stat.overall.node-load-miss-rate%
     49.93 ± 12%     +15.8       65.75 ±  5%  perf-stat.overall.node-store-miss-rate%
  19363951 ±  5%    +238.2%   65489199 ±  2%  perf-stat.overall.path-length
 8.249e+08 ± 15%    +309.1%  3.375e+09 ±  9%  perf-stat.ps.branch-instructions
  39324992 ±  9%     +70.7%   67136107 ±  8%  perf-stat.ps.cache-misses
      9391 ± 11%   +5286.2%     505829 ± 11%  perf-stat.ps.context-switches
 1.247e+10 ± 21%    +765.3%  1.079e+11 ±  9%  perf-stat.ps.cpu-cycles
     16.52 ± 16%   +9676.7%       1615 ± 13%  perf-stat.ps.cpu-migrations
 1.171e+09 ± 13%    +227.1%   3.83e+09 ±  8%  perf-stat.ps.dTLB-loads
 5.489e+08 ±  6%     +87.8%  1.031e+09 ±  7%  perf-stat.ps.dTLB-stores
   2584257 ±  9%     +93.5%    5000746 ±  6%  perf-stat.ps.iTLB-load-misses
   2373291 ±  9%     +89.9%    4507558 ±  6%  perf-stat.ps.iTLB-loads
 4.114e+09 ± 12%    +257.1%  1.469e+10 ±  9%  perf-stat.ps.instructions
      3352           +11.4%       3732 ±  8%  perf-stat.ps.minor-faults
   4030058 ± 15%    +211.6%   12559382 ±  9%  perf-stat.ps.node-load-misses
   3009096 ± 12%     +54.8%    4659294 ±  6%  perf-stat.ps.node-loads
   2494693 ± 16%    +137.4%    5921547 ± 11%  perf-stat.ps.node-store-misses
      3352           +11.4%       3733 ±  8%  perf-stat.ps.page-faults
 6.358e+11 ±  5%    +238.2%   2.15e+12 ±  2%  perf-stat.total.instructions
     10129 ± 10%    +126.5%      22947 ±  9%  slabinfo.avc_xperms_data.active_objs
     79.00 ± 10%    +127.2%     179.50 ±  8%  slabinfo.avc_xperms_data.active_slabs
     10147 ± 10%    +126.9%      23022 ±  8%  slabinfo.avc_xperms_data.num_objs
     79.00 ± 10%    +127.2%     179.50 ±  8%  slabinfo.avc_xperms_data.num_slabs
      3100 ±  8%     +72.0%       5333 ±  5%  slabinfo.biovec-max.active_objs
    403.25 ±  8%     +69.2%     682.50 ±  5%  slabinfo.biovec-max.active_slabs
      3229 ±  8%     +69.3%       5465 ±  5%  slabinfo.biovec-max.num_objs
    403.25 ±  8%     +69.2%     682.50 ±  5%  slabinfo.biovec-max.num_slabs
      5653 ±  4%     +91.1%      10803 ± 14%  slabinfo.blkdev_ioc.active_objs
    146.50 ±  3%     +93.0%     282.75 ± 13%  slabinfo.blkdev_ioc.active_slabs
      5737 ±  3%     +92.7%      11054 ± 13%  slabinfo.blkdev_ioc.num_objs
    146.50 ±  3%     +93.0%     282.75 ± 13%  slabinfo.blkdev_ioc.num_slabs
     28861 ±  7%     +34.6%      38850 ±  8%  slabinfo.btrfs_extent_map.active_objs
    530.75 ±  6%     +34.2%     712.25 ±  7%  slabinfo.btrfs_extent_map.active_slabs
     29744 ±  6%     +34.2%      39910 ±  7%  slabinfo.btrfs_extent_map.num_objs
    530.75 ±  6%     +34.2%     712.25 ±  7%  slabinfo.btrfs_extent_map.num_slabs
      7526 ± 15%    +118.3%      16428 ±  2%  slabinfo.btrfs_ordered_extent.active_objs
    204.50 ± 14%    +108.8%     427.00 ±  2%  slabinfo.btrfs_ordered_extent.active_slabs
      7992 ± 14%    +108.6%      16674 ±  2%  slabinfo.btrfs_ordered_extent.num_objs
    204.50 ± 14%    +108.8%     427.00 ±  2%  slabinfo.btrfs_ordered_extent.num_slabs
      3514 ±  5%    +104.7%       7192 ±  6%  slabinfo.buffer_head.active_objs
      3516 ±  5%    +104.6%       7194 ±  6%  slabinfo.buffer_head.num_objs
     22906 ± 15%    +275.8%      86082 ±  7%  slabinfo.dmaengine-unmap-16.active_objs
    546.00 ± 15%    +276.7%       2057 ±  7%  slabinfo.dmaengine-unmap-16.active_slabs
     22949 ± 15%    +276.6%      86420 ±  7%  slabinfo.dmaengine-unmap-16.num_objs
    546.00 ± 15%    +276.7%       2057 ±  7%  slabinfo.dmaengine-unmap-16.num_slabs
      7227 ±  6%     +22.6%       8860        slabinfo.files_cache.active_objs
      7227 ±  6%     +22.6%       8860        slabinfo.files_cache.num_objs
     25836 ±  4%     +25.7%      32477 ±  2%  slabinfo.kmalloc-128.active_objs
     26365 ±  5%     +23.7%      32603 ±  2%  slabinfo.kmalloc-128.num_objs
      4244 ±  2%    +111.0%       8955 ±  5%  slabinfo.kmalloc-4k.active_objs
    558.75 ±  2%    +111.3%       1180 ±  5%  slabinfo.kmalloc-4k.active_slabs
      4475 ±  2%    +111.1%       9449 ±  5%  slabinfo.kmalloc-4k.num_objs
    558.75 ±  2%    +111.3%       1180 ±  5%  slabinfo.kmalloc-4k.num_slabs
      3746 ±  4%     +52.9%       5728 ±  4%  slabinfo.kmalloc-8k.active_objs
    961.50 ±  4%     +52.7%       1468 ±  4%  slabinfo.kmalloc-8k.active_slabs
      3847 ±  4%     +52.7%       5876 ±  4%  slabinfo.kmalloc-8k.num_objs
    961.50 ±  4%     +52.7%       1468 ±  4%  slabinfo.kmalloc-8k.num_slabs
      9920 ±  8%    +106.0%      20432 ±  2%  slabinfo.mnt_cache.active_objs
    205.50 ±  8%     +98.3%     407.50 ±  2%  slabinfo.mnt_cache.active_slabs
     10507 ±  8%     +98.1%      20811 ±  2%  slabinfo.mnt_cache.num_objs
    205.50 ±  8%     +98.3%     407.50 ±  2%  slabinfo.mnt_cache.num_slabs
      9584 ±  3%     +64.4%      15755 ±  6%  slabinfo.numa_policy.active_objs
    154.50 ±  3%     +64.6%     254.25 ±  6%  slabinfo.numa_policy.active_slabs
      9618 ±  3%     +64.3%      15806 ±  6%  slabinfo.numa_policy.num_objs
    154.50 ±  3%     +64.6%     254.25 ±  6%  slabinfo.numa_policy.num_slabs
     15440 ±  5%     +18.1%      18237 ±  2%  slabinfo.pde_opener.active_objs
     15440 ±  5%     +18.1%      18237 ±  2%  slabinfo.pde_opener.num_objs
      4084 ±  3%     +12.0%       4573        slabinfo.sighand_cache.active_objs
      4092 ±  3%     +11.8%       4574        slabinfo.sighand_cache.num_objs
      6489 ±  4%     +14.0%       7398        slabinfo.signal_cache.active_objs
      6489 ±  4%     +14.0%       7400        slabinfo.signal_cache.num_objs
     10176 ±  5%     +10.7%      11269        slabinfo.task_delay_info.active_objs
     10176 ±  5%     +10.7%      11269        slabinfo.task_delay_info.num_objs
     24.33 ±173%   +2103.7%     536.10 ± 71%  sched_debug.cfs_rq:/.MIN_vruntime.avg
      1903 ±  3%    +524.0%      11877 ± 34%  sched_debug.cfs_rq:/.exec_clock.avg
      4146 ±  5%    +225.7%      13504 ± 32%  sched_debug.cfs_rq:/.exec_clock.max
      1327 ±  6%    +756.0%      11365 ± 33%  sched_debug.cfs_rq:/.exec_clock.min
     17308 ± 43%    +869.7%     167844 ± 19%  sched_debug.cfs_rq:/.load.avg
     91268 ± 38%    +157.3%     234819 ± 12%  sched_debug.cfs_rq:/.load.stddev
     30.17 ± 13%    +543.3%     194.10 ± 23%  sched_debug.cfs_rq:/.load_avg.avg
    701.15 ±  3%    +291.8%       2747 ±115%  sched_debug.cfs_rq:/.load_avg.max
    106.48 ± 11%    +140.1%     255.64 ± 86%  sched_debug.cfs_rq:/.load_avg.stddev
     24.33 ±173%   +2103.7%     536.10 ± 71%  sched_debug.cfs_rq:/.max_vruntime.avg
      7854 ±  9%     -18.8%       6375 ± 13%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.04 ± 20%    +330.3%       0.19 ± 18%  sched_debug.cfs_rq:/.nr_running.avg
      0.20 ± 11%     +67.1%       0.34 ±  9%  sched_debug.cfs_rq:/.nr_running.stddev
      7.78 ± 39%    +632.1%      56.92 ± 15%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     47.64 ± 28%     +82.7%      87.03 ±  3%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     17292 ± 43%    +866.6%     167141 ± 19%  sched_debug.cfs_rq:/.runnable_weight.avg
     91207 ± 38%    +157.1%     234517 ± 12%  sched_debug.cfs_rq:/.runnable_weight.stddev
      7854 ±  9%     -18.8%       6375 ± 13%  sched_debug.cfs_rq:/.spread0.stddev
     74.72 ± 12%    +255.1%     265.31 ± 24%  sched_debug.cfs_rq:/.util_avg.avg
    730.90 ±  6%     +13.8%     831.69 ±  5%  sched_debug.cfs_rq:/.util_avg.max
    135.40 ±  3%     +20.6%     163.25 ±  7%  sched_debug.cfs_rq:/.util_avg.stddev
      7.01 ± 27%    +770.5%      61.02 ± 24%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     40.97 ± 21%    +166.2%     109.07 ± 24%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
   1011673 ±  4%     -35.6%     651113 ± 11%  sched_debug.cpu.avg_idle.avg
   2557230 ± 25%     -41.2%    1502483 ± 13%  sched_debug.cpu.avg_idle.max
     71256 ± 27%     -67.2%      23382 ± 89%  sched_debug.cpu.avg_idle.min
      6.64 ±  4%     +43.9%       9.55 ±  8%  sched_debug.cpu.clock.stddev
      6.64 ±  4%     +43.9%       9.55 ±  8%  sched_debug.cpu.clock_task.stddev
     87.70 ±  9%    +654.5%     661.69 ± 23%  sched_debug.cpu.curr->pid.avg
    618.95 ±  6%     +86.6%       1154 ± 10%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  7%     +44.2%       0.00 ± 12%  sched_debug.cpu.next_balance.stddev
      0.02 ±  8%    +660.7%       0.17 ± 24%  sched_debug.cpu.nr_running.avg
      0.14 ±  5%    +110.8%       0.30 ± 17%  sched_debug.cpu.nr_running.stddev
      6266 ±  6%   +2371.9%     154913 ± 36%  sched_debug.cpu.nr_switches.avg
     44815 ± 19%    +352.7%     202895 ± 39%  sched_debug.cpu.nr_switches.max
      1326 ± 30%   +9641.0%     129205 ± 38%  sched_debug.cpu.nr_switches.min
      5749 ±  7%    +105.2%      11799 ± 33%  sched_debug.cpu.nr_switches.stddev
      0.29 ± 10%     -95.2%       0.01 ± 22%  sched_debug.cpu.nr_uninterruptible.avg
    -34.85           -36.3%     -22.19        sched_debug.cpu.nr_uninterruptible.min
      4061 ± 10%   +3678.6%     153453 ± 36%  sched_debug.cpu.sched_count.avg
     40207 ± 21%    +392.5%     198017 ± 40%  sched_debug.cpu.sched_count.max
    137.35 ± 31%  +93278.4%     128259 ± 39%  sched_debug.cpu.sched_count.min
      5304 ± 10%    +112.6%      11279 ± 36%  sched_debug.cpu.sched_count.stddev
      1892 ± 10%   +3924.4%      76150 ± 36%  sched_debug.cpu.sched_goidle.avg
     17428 ± 18%    +464.9%      98457 ± 40%  sched_debug.cpu.sched_goidle.max
     26.56 ± 15%  +2.4e+05%      63596 ± 39%  sched_debug.cpu.sched_goidle.min
      2391 ±  8%    +135.1%       5622 ± 36%  sched_debug.cpu.sched_goidle.stddev
      2099 ± 10%   +3572.3%      77110 ± 36%  sched_debug.cpu.ttwu_count.avg
     28779 ± 34%    +346.2%     128426 ± 37%  sched_debug.cpu.ttwu_count.max
     38.58 ± 27%  +1.2e+05%      47152 ± 37%  sched_debug.cpu.ttwu_count.min
      3265 ± 16%    +363.1%      15124 ± 32%  sched_debug.cpu.ttwu_count.stddev
      1173 ± 10%     -79.7%     238.61 ± 58%  sched_debug.cpu.ttwu_local.avg
     18640 ± 37%     -84.1%       2956 ± 69%  sched_debug.cpu.ttwu_local.max
     22.75 ± 33%    +186.5%      65.19 ± 45%  sched_debug.cpu.ttwu_local.min
      2147 ± 18%     -83.5%     355.42 ± 79%  sched_debug.cpu.ttwu_local.stddev
      2310 ± 13%    +312.2%       9526 ±  6%  interrupts.CPU0.RES:Rescheduling_interrupts
    909.50 ± 48%    +820.9%       8376 ±  9%  interrupts.CPU1.RES:Rescheduling_interrupts
    147.25 ± 80%   +4270.3%       6435 ± 13%  interrupts.CPU10.RES:Rescheduling_interrupts
     59.25 ± 44%  +17462.9%      10406 ± 29%  interrupts.CPU100.RES:Rescheduling_interrupts
     95.25 ± 61%   +8394.0%       8090 ± 15%  interrupts.CPU101.RES:Rescheduling_interrupts
     65.75 ± 58%  +15541.8%      10284 ± 45%  interrupts.CPU102.RES:Rescheduling_interrupts
     75.25 ± 55%  +10011.3%       7608 ± 11%  interrupts.CPU103.RES:Rescheduling_interrupts
     41.75 ± 66%  +18538.9%       7781 ± 15%  interrupts.CPU104.RES:Rescheduling_interrupts
     64.75 ± 79%  +11194.6%       7313 ± 27%  interrupts.CPU105.RES:Rescheduling_interrupts
     63.25 ± 58%  +12224.9%       7795 ± 12%  interrupts.CPU106.RES:Rescheduling_interrupts
     64.75 ± 38%  +15997.3%      10423 ± 40%  interrupts.CPU107.RES:Rescheduling_interrupts
     48.25 ± 82%  +13846.6%       6729 ± 16%  interrupts.CPU108.RES:Rescheduling_interrupts
     45.00 ± 69%  +20118.3%       9098 ± 24%  interrupts.CPU109.RES:Rescheduling_interrupts
     80.50 ± 58%   +9189.8%       7478 ±  9%  interrupts.CPU11.RES:Rescheduling_interrupts
     54.00 ± 81%  +13330.1%       7252 ±  9%  interrupts.CPU110.RES:Rescheduling_interrupts
     32.50 ± 73%  +27363.8%       8925 ± 41%  interrupts.CPU111.RES:Rescheduling_interrupts
     50.75 ± 89%  +12812.3%       6553 ± 17%  interrupts.CPU112.RES:Rescheduling_interrupts
     73.00 ± 81%  +10557.2%       7779 ±  5%  interrupts.CPU113.RES:Rescheduling_interrupts
     46.25 ± 52%  +18256.2%       8489 ± 30%  interrupts.CPU114.RES:Rescheduling_interrupts
     40.25 ± 87%  +17741.6%       7181 ± 16%  interrupts.CPU115.RES:Rescheduling_interrupts
     28.75 ± 72%  +25192.2%       7271 ± 16%  interrupts.CPU116.RES:Rescheduling_interrupts
     60.75 ± 54%  +10877.4%       6668 ± 15%  interrupts.CPU117.RES:Rescheduling_interrupts
     25.75 ± 74%  +24464.1%       6325 ±  5%  interrupts.CPU118.RES:Rescheduling_interrupts
     57.75 ± 86%  +12988.7%       7558 ±  6%  interrupts.CPU119.RES:Rescheduling_interrupts
    147.25 ±135%   +4188.1%       6314 ± 18%  interrupts.CPU12.RES:Rescheduling_interrupts
     94.25 ± 67%   +9118.3%       8688 ±  3%  interrupts.CPU120.RES:Rescheduling_interrupts
     60.25 ± 54%  +11145.6%       6775 ± 13%  interrupts.CPU121.RES:Rescheduling_interrupts
     91.00 ± 83%   +9486.5%       8723 ± 30%  interrupts.CPU122.RES:Rescheduling_interrupts
     33.50 ± 50%  +21917.2%       7375 ±  8%  interrupts.CPU123.RES:Rescheduling_interrupts
     61.50 ± 48%  +14180.5%       8782 ± 16%  interrupts.CPU124.RES:Rescheduling_interrupts
     58.75 ± 87%  +13198.3%       7812 ± 33%  interrupts.CPU125.RES:Rescheduling_interrupts
     66.50 ± 55%  +10757.5%       7220 ± 14%  interrupts.CPU126.RES:Rescheduling_interrupts
     57.25 ± 67%  +14193.0%       8182 ± 27%  interrupts.CPU127.RES:Rescheduling_interrupts
     51.25 ± 60%  +14182.0%       7319 ± 11%  interrupts.CPU128.RES:Rescheduling_interrupts
     44.50 ± 41%  +17043.3%       7628 ± 22%  interrupts.CPU129.RES:Rescheduling_interrupts
     61.75 ± 56%  +11655.5%       7259 ± 20%  interrupts.CPU13.RES:Rescheduling_interrupts
     70.75 ± 78%  +12233.2%       8725 ± 13%  interrupts.CPU130.RES:Rescheduling_interrupts
     35.50 ± 48%  +21876.8%       7801 ± 17%  interrupts.CPU131.RES:Rescheduling_interrupts
     35.75 ± 76%  +28337.1%      10166 ± 25%  interrupts.CPU132.RES:Rescheduling_interrupts
     88.25 ± 67%  +10461.5%       9320 ± 19%  interrupts.CPU133.RES:Rescheduling_interrupts
     67.75 ± 29%  +12948.7%       8840 ± 17%  interrupts.CPU134.RES:Rescheduling_interrupts
     51.00 ± 60%  +14346.6%       7367 ±  9%  interrupts.CPU135.RES:Rescheduling_interrupts
     52.75 ± 50%  +15164.9%       8052 ± 13%  interrupts.CPU136.RES:Rescheduling_interrupts
     40.00 ± 69%  +21983.1%       8833 ± 19%  interrupts.CPU137.RES:Rescheduling_interrupts
     47.50 ± 62%  +16348.4%       7813 ± 13%  interrupts.CPU138.RES:Rescheduling_interrupts
     57.75 ± 29%  +13312.1%       7745 ± 22%  interrupts.CPU139.RES:Rescheduling_interrupts
     69.50 ± 83%  +11783.5%       8259 ± 14%  interrupts.CPU14.RES:Rescheduling_interrupts
     58.75 ± 26%  +11363.8%       6735 ± 21%  interrupts.CPU140.RES:Rescheduling_interrupts
    353.50 ±124%   +2133.7%       7896 ± 25%  interrupts.CPU141.RES:Rescheduling_interrupts
     75.75 ± 61%   +9415.2%       7207 ± 12%  interrupts.CPU142.RES:Rescheduling_interrupts
     38.25 ± 61%  +19568.0%       7523 ± 19%  interrupts.CPU143.RES:Rescheduling_interrupts
      3084 ± 28%     -28.2%       2213 ± 12%  interrupts.CPU144.CAL:Function_call_interrupts
     95.00 ± 35%   +7158.7%       6895 ± 17%  interrupts.CPU144.RES:Rescheduling_interrupts
      3104 ± 28%     -28.6%       2215 ± 10%  interrupts.CPU145.CAL:Function_call_interrupts
     76.25 ± 22%   +9220.0%       7106 ± 14%  interrupts.CPU145.RES:Rescheduling_interrupts
     97.50 ± 37%   +7470.3%       7381 ± 12%  interrupts.CPU146.RES:Rescheduling_interrupts
     93.75 ± 18%   +7962.4%       7558 ± 11%  interrupts.CPU147.RES:Rescheduling_interrupts
     90.50 ± 25%   +7503.6%       6881 ± 10%  interrupts.CPU148.RES:Rescheduling_interrupts
     90.75 ± 38%   +7659.2%       7041 ± 17%  interrupts.CPU149.RES:Rescheduling_interrupts
     58.75 ± 30%  +12590.6%       7455 ± 12%  interrupts.CPU15.RES:Rescheduling_interrupts
      3102 ± 28%     -27.5%       2250 ± 10%  interrupts.CPU150.CAL:Function_call_interrupts
     84.00 ± 40%   +8721.7%       7410 ± 14%  interrupts.CPU150.RES:Rescheduling_interrupts
     90.75 ± 30%   +8597.8%       7893 ±  7%  interrupts.CPU151.RES:Rescheduling_interrupts
     86.50 ± 26%   +8098.8%       7092 ± 13%  interrupts.CPU152.RES:Rescheduling_interrupts
     64.00 ± 13%  +12892.2%       8315 ± 15%  interrupts.CPU153.RES:Rescheduling_interrupts
    433.50 ±128%   +1729.0%       7928 ± 19%  interrupts.CPU154.RES:Rescheduling_interrupts
     62.00 ± 27%  +10373.0%       6493 ±  8%  interrupts.CPU155.RES:Rescheduling_interrupts
     66.75 ± 55%  +12414.2%       8353 ± 16%  interrupts.CPU156.RES:Rescheduling_interrupts
     98.75 ± 24%   +7876.2%       7876 ± 27%  interrupts.CPU157.RES:Rescheduling_interrupts
    138.00 ± 70%   +4954.3%       6975 ± 10%  interrupts.CPU158.RES:Rescheduling_interrupts
    964.75 ±161%    +714.8%       7861 ± 13%  interrupts.CPU159.RES:Rescheduling_interrupts
    136.00 ± 82%   +6568.9%       9069 ± 24%  interrupts.CPU16.RES:Rescheduling_interrupts
    289.50 ±135%   +2545.4%       7658 ± 14%  interrupts.CPU160.RES:Rescheduling_interrupts
     93.75 ± 52%   +8224.3%       7804 ± 16%  interrupts.CPU161.RES:Rescheduling_interrupts
     80.75 ± 34%  +11160.1%       9092 ± 13%  interrupts.CPU162.RES:Rescheduling_interrupts
    286.25 ±116%   +2894.7%       8572 ± 11%  interrupts.CPU163.RES:Rescheduling_interrupts
    156.50 ± 83%   +5149.5%       8215 ±  5%  interrupts.CPU164.RES:Rescheduling_interrupts
     68.75 ±  9%  +11591.3%       8037 ± 15%  interrupts.CPU165.RES:Rescheduling_interrupts
     65.00 ± 33%  +12680.4%       8307 ± 24%  interrupts.CPU166.RES:Rescheduling_interrupts
      3112 ± 28%     -28.5%       2226 ± 12%  interrupts.CPU167.CAL:Function_call_interrupts
     87.75 ± 22%   +7642.5%       6794 ± 12%  interrupts.CPU167.RES:Rescheduling_interrupts
     92.50 ± 20%   +9136.5%       8543 ± 18%  interrupts.CPU168.RES:Rescheduling_interrupts
    156.50 ± 67%   +4541.1%       7263 ±  4%  interrupts.CPU169.RES:Rescheduling_interrupts
     69.50 ± 49%  +10135.6%       7113 ± 16%  interrupts.CPU17.RES:Rescheduling_interrupts
    211.00 ±100%   +3690.6%       7998 ± 25%  interrupts.CPU170.RES:Rescheduling_interrupts
     99.00 ± 16%   +8928.3%       8938 ± 23%  interrupts.CPU171.RES:Rescheduling_interrupts
     99.00 ± 13%   +7508.1%       7532 ± 19%  interrupts.CPU172.RES:Rescheduling_interrupts
     93.50 ± 19%   +8237.7%       7795 ±  6%  interrupts.CPU173.RES:Rescheduling_interrupts
    160.75 ± 57%   +4735.0%       7772 ± 14%  interrupts.CPU174.RES:Rescheduling_interrupts
     82.50 ± 12%   +8944.8%       7462 ± 28%  interrupts.CPU175.RES:Rescheduling_interrupts
     79.75 ± 19%  +11865.2%       9542 ± 18%  interrupts.CPU176.RES:Rescheduling_interrupts
     93.25 ± 15%   +9092.8%       8572 ± 24%  interrupts.CPU177.RES:Rescheduling_interrupts
     59.25 ± 39%  +12435.4%       7427 ± 19%  interrupts.CPU178.RES:Rescheduling_interrupts
     72.25 ± 20%  +13548.1%       9860 ± 21%  interrupts.CPU179.RES:Rescheduling_interrupts
     58.75 ± 39%  +12754.9%       7552 ±  5%  interrupts.CPU18.RES:Rescheduling_interrupts
    198.25 ± 81%   +4894.1%       9900 ± 26%  interrupts.CPU180.RES:Rescheduling_interrupts
    130.50 ± 59%   +5984.7%       7940 ±  7%  interrupts.CPU181.RES:Rescheduling_interrupts
     66.50 ± 39%  +10878.9%       7301 ± 20%  interrupts.CPU182.RES:Rescheduling_interrupts
     92.25 ± 31%   +7939.8%       7416 ± 18%  interrupts.CPU183.RES:Rescheduling_interrupts
     95.25 ± 51%   +8743.6%       8423 ± 15%  interrupts.CPU184.RES:Rescheduling_interrupts
     61.50 ± 30%  +13371.1%       8284 ± 12%  interrupts.CPU185.RES:Rescheduling_interrupts
     64.00 ± 28%  +10939.1%       7065 ±  8%  interrupts.CPU186.RES:Rescheduling_interrupts
     63.00 ± 49%  +14207.9%       9014 ± 29%  interrupts.CPU187.RES:Rescheduling_interrupts
     80.50 ± 10%   +9292.2%       7560 ± 28%  interrupts.CPU188.RES:Rescheduling_interrupts
     56.75 ± 42%  +14584.1%       8333 ± 10%  interrupts.CPU189.RES:Rescheduling_interrupts
     70.25 ± 75%  +13413.5%       9493 ± 11%  interrupts.CPU19.RES:Rescheduling_interrupts
     74.50 ± 44%  +11989.3%       9006 ± 14%  interrupts.CPU190.RES:Rescheduling_interrupts
     72.75 ± 33%  +12254.6%       8988 ±  5%  interrupts.CPU191.RES:Rescheduling_interrupts
     50.75 ± 74%  +12573.4%       6431 ± 20%  interrupts.CPU20.RES:Rescheduling_interrupts
     73.00 ± 86%  +10112.3%       7455        interrupts.CPU21.RES:Rescheduling_interrupts
     52.25 ± 50%  +16711.0%       8783 ± 19%  interrupts.CPU22.RES:Rescheduling_interrupts
     52.25 ± 67%  +18581.8%       9761 ± 31%  interrupts.CPU23.RES:Rescheduling_interrupts
    229.75 ± 99%     -77.0%      52.75 ±173%  interrupts.CPU24.NMI:Non-maskable_interrupts
    229.75 ± 99%     -77.0%      52.75 ±173%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
    917.75 ±120%   +1045.7%      10514 ± 25%  interrupts.CPU24.RES:Rescheduling_interrupts
    544.25 ±151%   +1476.0%       8577 ± 19%  interrupts.CPU25.RES:Rescheduling_interrupts
    501.50 ±152%   +1561.5%       8332 ± 10%  interrupts.CPU26.RES:Rescheduling_interrupts
    660.75 ±162%   +1414.7%      10008 ± 26%  interrupts.CPU27.RES:Rescheduling_interrupts
     97.25 ± 61%  +10457.3%      10267 ±  7%  interrupts.CPU28.RES:Rescheduling_interrupts
     75.25 ± 50%   +9964.8%       7573 ± 13%  interrupts.CPU29.RES:Rescheduling_interrupts
     58.25 ± 43%  +16584.5%       9718 ± 26%  interrupts.CPU30.RES:Rescheduling_interrupts
     52.00 ± 45%  +15006.2%       7855 ± 27%  interrupts.CPU31.RES:Rescheduling_interrupts
      1307 ±161%    +550.1%       8500 ± 21%  interrupts.CPU32.RES:Rescheduling_interrupts
     76.00 ± 46%  +10088.8%       7743 ± 17%  interrupts.CPU33.RES:Rescheduling_interrupts
    146.75 ± 82%   +5481.3%       8190 ± 23%  interrupts.CPU34.RES:Rescheduling_interrupts
      1703 ±165%    +386.4%       8285 ±  5%  interrupts.CPU35.RES:Rescheduling_interrupts
    615.75 ±164%   +1123.1%       7531 ± 18%  interrupts.CPU36.RES:Rescheduling_interrupts
      1831 ±168%    +401.3%       9178 ± 12%  interrupts.CPU37.RES:Rescheduling_interrupts
     56.75 ± 74%  +12767.8%       7302 ± 13%  interrupts.CPU38.RES:Rescheduling_interrupts
      3116 ± 28%     -29.5%       2197 ±  8%  interrupts.CPU39.CAL:Function_call_interrupts
    279.50 ±148%   +2549.8%       7406 ± 15%  interrupts.CPU39.RES:Rescheduling_interrupts
     93.50 ± 57%   +8862.3%       8379 ±  8%  interrupts.CPU4.RES:Rescheduling_interrupts
     45.50 ± 66%  +23110.4%      10560 ± 23%  interrupts.CPU40.RES:Rescheduling_interrupts
     74.50 ± 51%  +18176.5%      13616 ± 49%  interrupts.CPU41.RES:Rescheduling_interrupts
    520.00 ±156%   +1369.4%       7641 ± 20%  interrupts.CPU42.RES:Rescheduling_interrupts
     87.25 ± 24%   +9102.9%       8029 ± 30%  interrupts.CPU43.RES:Rescheduling_interrupts
     59.00 ± 72%  +13051.7%       7759 ± 10%  interrupts.CPU45.RES:Rescheduling_interrupts
     31.00 ± 68%  +26443.5%       8228 ± 11%  interrupts.CPU46.RES:Rescheduling_interrupts
    131.00 ± 18%   +5715.6%       7618        interrupts.CPU48.RES:Rescheduling_interrupts
    125.00 ± 12%   +6334.8%       8043 ± 11%  interrupts.CPU49.RES:Rescheduling_interrupts
    176.75 ± 89%   +3955.6%       7168 ± 19%  interrupts.CPU5.RES:Rescheduling_interrupts
    133.50 ± 33%   +7606.2%      10287 ± 13%  interrupts.CPU50.RES:Rescheduling_interrupts
      3057 ± 29%     -33.0%       2048 ±  2%  interrupts.CPU51.CAL:Function_call_interrupts
     87.75 ± 23%   +8182.6%       7268 ± 10%  interrupts.CPU51.RES:Rescheduling_interrupts
      1076 ±161%    +673.4%       8323 ±  7%  interrupts.CPU52.RES:Rescheduling_interrupts
    239.25 ±114%   +3703.4%       9099 ±  9%  interrupts.CPU53.RES:Rescheduling_interrupts
    105.50 ±  8%   +8139.1%       8692 ± 30%  interrupts.CPU54.RES:Rescheduling_interrupts
    166.00 ± 95%   +4414.0%       7493 ± 17%  interrupts.CPU55.RES:Rescheduling_interrupts
      1464 ±156%    +552.9%       9560 ± 13%  interrupts.CPU56.RES:Rescheduling_interrupts
    139.25 ± 27%   +5096.1%       7235 ±  2%  interrupts.CPU57.RES:Rescheduling_interrupts
    491.75 ± 93%   +1433.8%       7542 ± 10%  interrupts.CPU58.RES:Rescheduling_interrupts
    113.50 ± 22%   +6189.6%       7138 ± 17%  interrupts.CPU59.RES:Rescheduling_interrupts
    791.00 ±157%    +826.3%       7327 ±  6%  interrupts.CPU6.RES:Rescheduling_interrupts
      3049 ± 29%     -34.0%       2013 ± 16%  interrupts.CPU60.CAL:Function_call_interrupts
    150.50 ± 47%   +5280.7%       8098 ±  4%  interrupts.CPU60.RES:Rescheduling_interrupts
     79.25 ± 34%   +8662.5%       6944 ± 12%  interrupts.CPU61.RES:Rescheduling_interrupts
     88.25 ± 47%   +7685.6%       6870 ± 20%  interrupts.CPU62.RES:Rescheduling_interrupts
    310.25 ± 77%   +2659.6%       8561 ± 16%  interrupts.CPU63.RES:Rescheduling_interrupts
    211.00 ± 79%   +4446.9%       9594 ± 25%  interrupts.CPU64.RES:Rescheduling_interrupts
    404.50 ±133%   +1852.0%       7896 ± 17%  interrupts.CPU65.RES:Rescheduling_interrupts
    115.75 ± 51%   +5336.7%       6293 ± 23%  interrupts.CPU66.RES:Rescheduling_interrupts
     75.50 ± 57%   +9611.9%       7332 ±  9%  interrupts.CPU67.RES:Rescheduling_interrupts
    232.75 ± 82%   +2983.6%       7177 ± 21%  interrupts.CPU68.RES:Rescheduling_interrupts
     74.50 ± 39%   +8348.7%       6294 ±  9%  interrupts.CPU69.RES:Rescheduling_interrupts
    326.25 ±132%   +2376.1%       8078 ± 19%  interrupts.CPU7.RES:Rescheduling_interrupts
     80.00 ± 23%   +9083.4%       7346 ± 11%  interrupts.CPU70.RES:Rescheduling_interrupts
      3077 ± 29%     -29.8%       2160 ±  7%  interrupts.CPU71.CAL:Function_call_interrupts
     72.50 ± 40%   +9001.7%       6598 ± 14%  interrupts.CPU71.RES:Rescheduling_interrupts
    211.00 ± 36%     -61.5%      81.25 ± 74%  interrupts.CPU71.TLB:TLB_shootdowns
    108.00 ± 29%   +7215.5%       7900 ± 12%  interrupts.CPU72.RES:Rescheduling_interrupts
     88.75 ± 15%   +9323.1%       8363 ± 17%  interrupts.CPU73.RES:Rescheduling_interrupts
    155.00 ± 65%   +5806.9%       9155 ± 10%  interrupts.CPU74.RES:Rescheduling_interrupts
     95.25 ± 14%  +11223.4%      10785 ± 19%  interrupts.CPU75.RES:Rescheduling_interrupts
     15.50 ± 95%    +466.1%      87.75 ± 47%  interrupts.CPU75.TLB:TLB_shootdowns
    126.50 ± 32%   +5975.7%       7685 ± 17%  interrupts.CPU76.RES:Rescheduling_interrupts
     11.25 ± 65%    +766.7%      97.50 ± 25%  interrupts.CPU76.TLB:TLB_shootdowns
    241.00 ±111%   +2907.7%       7248 ± 25%  interrupts.CPU77.RES:Rescheduling_interrupts
     33.25 ± 70%    +291.0%     130.00 ± 31%  interrupts.CPU77.TLB:TLB_shootdowns
     59.00 ± 35%  +11928.0%       7096 ± 13%  interrupts.CPU78.RES:Rescheduling_interrupts
    117.50 ± 12%   +5840.2%       6979 ± 14%  interrupts.CPU79.RES:Rescheduling_interrupts
    162.50 ± 25%   +4075.2%       6784 ± 22%  interrupts.CPU8.RES:Rescheduling_interrupts
     91.50 ± 20%   +9183.1%       8494 ± 20%  interrupts.CPU80.RES:Rescheduling_interrupts
     27.25 ± 72%    +372.5%     128.75 ± 34%  interrupts.CPU80.TLB:TLB_shootdowns
    111.50 ± 34%   +6877.6%       7780 ±  5%  interrupts.CPU81.RES:Rescheduling_interrupts
     60.50 ± 82%     +93.0%     116.75 ± 41%  interrupts.CPU81.TLB:TLB_shootdowns
    108.25 ± 18%   +7609.2%       8345 ± 23%  interrupts.CPU82.RES:Rescheduling_interrupts
     65.25 ± 50%    +101.9%     131.75 ± 30%  interrupts.CPU82.TLB:TLB_shootdowns
    110.75 ±  8%   +7285.6%       8179 ± 28%  interrupts.CPU83.RES:Rescheduling_interrupts
     95.75 ± 19%   +8309.1%       8051 ± 26%  interrupts.CPU84.RES:Rescheduling_interrupts
     88.00 ± 23%   +8124.1%       7237 ± 20%  interrupts.CPU85.RES:Rescheduling_interrupts
     52.50 ± 28%    +176.7%     145.25 ± 36%  interrupts.CPU85.TLB:TLB_shootdowns
     88.75 ± 28%   +8454.1%       7591 ± 12%  interrupts.CPU86.RES:Rescheduling_interrupts
     97.00 ± 26%   +7650.3%       7517 ± 18%  interrupts.CPU87.RES:Rescheduling_interrupts
    106.25 ±  7%   +8416.7%       9049 ± 25%  interrupts.CPU88.RES:Rescheduling_interrupts
     48.75 ± 87%    +140.5%     117.25 ± 44%  interrupts.CPU88.TLB:TLB_shootdowns
     91.50 ± 16%   +7517.5%       6970 ± 23%  interrupts.CPU89.RES:Rescheduling_interrupts
     91.00 ± 47%   +9440.4%       8681 ± 20%  interrupts.CPU9.RES:Rescheduling_interrupts
    193.25 ±113%   +4244.0%       8394 ± 28%  interrupts.CPU90.RES:Rescheduling_interrupts
     91.50 ± 11%   +7465.0%       6922 ± 17%  interrupts.CPU91.RES:Rescheduling_interrupts
     94.25 ± 36%   +8076.4%       7706 ± 31%  interrupts.CPU92.RES:Rescheduling_interrupts
     22.75 ± 68%    +379.1%     109.00 ± 46%  interrupts.CPU92.TLB:TLB_shootdowns
     93.50 ± 36%   +6881.6%       6527 ± 15%  interrupts.CPU93.RES:Rescheduling_interrupts
     71.75 ± 41%  +11132.8%       8059 ± 24%  interrupts.CPU94.RES:Rescheduling_interrupts
    160.75 ± 39%   +4284.9%       7048 ± 24%  interrupts.CPU95.RES:Rescheduling_interrupts
     58.75 ± 25%  +15752.3%       9313 ± 22%  interrupts.CPU96.RES:Rescheduling_interrupts
    104.50 ± 13%   +6269.9%       6656 ± 13%  interrupts.CPU97.RES:Rescheduling_interrupts
     70.75 ± 49%  +11488.7%       8199 ±  9%  interrupts.CPU98.RES:Rescheduling_interrupts
     69.50 ± 41%  +11445.3%       8024 ± 17%  interrupts.CPU99.RES:Rescheduling_interrupts
     52084 ± 14%   +2855.6%    1539388        interrupts.RES:Rescheduling_interrupts





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.4.0-rc5-00020-g2b871886bb64a" of type "text/plain" (200562 bytes)

View attachment "job-script" of type "text/plain" (8223 bytes)

View attachment "job.yaml" of type "text/plain" (5816 bytes)

View attachment "reproduce" of type "text/plain" (1178 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ