lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20200227022905.GH6548@shao2-debian>
Date:   Thu, 27 Feb 2020 10:29:05 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     js1304@...il.com
Cc:     Andrew Morton <akpm@...ux-foundation.org>, linux-mm@...ck.org,
        linux-kernel@...r.kernel.org, Johannes Weiner <hannes@...xchg.org>,
        Michal Hocko <mhocko@...nel.org>,
        Hugh Dickins <hughd@...gle.com>,
        Minchan Kim <minchan@...nel.org>,
        Vlastimil Babka <vbabka@...e.cz>,
        Mel Gorman <mgorman@...hsingularity.net>, kernel-team@....com,
        Joonsoo Kim <iamjoonsoo.kim@....com>, lkp@...ts.01.org
Subject: [mm/vmscan] dcf33bfdfb: vm-scalability.median 402.4% improvement

Greeting,

FYI, we noticed a 402.4% improvement of vm-scalability.median due to commit:


commit: dcf33bfdfbb73d0b956f6b375e4b5c1c67e5fe50 ("[PATCH 1/9] mm/vmscan: make active/inactive ratio as 1:1 for anon lru")
url: https://github.com/0day-ci/linux/commits/js1304-gmail-com/workingset-protection-detection-on-the-anonymous-LRU-list/20200213-111547


in testcase: vm-scalability
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	runtime: 300
	thp_enabled: never
	thp_defrag: always
	nr_task: 32
	nr_pmem: 1
	test: swap-w-rand-mt
	bp_memmap: 96G!18G
	cpufreq_governor: performance
	ucode: 0x500002c

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bp_memmap/compiler/cpufreq_governor/kconfig/nr_pmem/nr_task/rootfs/runtime/tbox_group/test/testcase/thp_defrag/thp_enabled/ucode:
  96G!18G/gcc-7/performance/x86_64-rhel-7.6/1/32/debian-x86_64-20191114.cgz/300/lkp-csl-2sp6/swap-w-rand-mt/vm-scalability/always/never/0x500002c

commit: 
  fdb6482244 ("perf/x86: Add Intel Tiger Lake uncore support")
  dcf33bfdfb ("mm/vmscan: make active/inactive ratio as 1:1 for anon lru")

fdb64822443ec9fb dcf33bfdfbb73d0b956f6b375e4 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           50%           2:4     dmesg.WARNING:at_ip___perf_sw_event/0x
           :4           50%           2:4     dmesg.WARNING:at_ip__slab_free/0x
          1:4          -25%            :4     dmesg.WARNING:at_ip_shrink_lruvec/0x
          0:4            2%           0:4     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      8466 ±114%    +402.4%      42539 ± 14%  vm-scalability.median
     31.15 ±115%     -67.7%      10.05 ±  4%  vm-scalability.stddev
    366811 ±110%    +328.5%    1571737 ± 13%  vm-scalability.throughput
    317.01 ±  3%     -38.9%     193.62 ± 14%  vm-scalability.time.elapsed_time
    317.01 ±  3%     -38.9%     193.62 ± 14%  vm-scalability.time.elapsed_time.max
   1770901 ± 14%     -64.2%     634783 ± 19%  vm-scalability.time.involuntary_context_switches
  50168072 ± 31%     -35.7%   32282722 ± 14%  vm-scalability.time.major_page_faults
      2762 ±  4%     -10.0%       2486 ±  2%  vm-scalability.time.percent_of_cpu_this_job_got
      8146 ± 18%     -66.3%       2746 ± 26%  vm-scalability.time.system_time
    625.00 ±135%    +232.3%       2077 ±  4%  vm-scalability.time.user_time
    164811 ± 12%     -79.2%      34341 ± 28%  vm-scalability.time.voluntary_context_switches
 1.938e+10           -34.5%   1.27e+10 ± 17%  cpuidle.C1E.time
  39846177           -41.1%   23465058 ± 35%  cpuidle.C1E.usage
     65.25 ±  3%     +10.7%      72.24        iostat.cpu.idle
     32.51 ± 17%     -49.7%      16.35 ±  9%  iostat.cpu.system
      2.23 ±135%    +411.0%      11.41 ± 10%  iostat.cpu.user
     65.03 ±  3%      +6.9       71.97        mpstat.cpu.all.idle%
     32.71 ± 17%     -16.2       16.50 ±  9%  mpstat.cpu.all.sys%
      2.24 ±136%      +9.3       11.53 ± 10%  mpstat.cpu.all.usr%
      2041 ±  5%    -100.0%       0.00        slabinfo.dmaengine-unmap-16.active_objs
      2041 ±  5%    -100.0%       0.00        slabinfo.dmaengine-unmap-16.num_objs
      1875 ±  8%     -19.5%       1510 ±  5%  slabinfo.kmalloc-rcl-96.active_objs
      1875 ±  8%     -19.5%       1510 ±  5%  slabinfo.kmalloc-rcl-96.num_objs
     64.50 ±  4%     +11.2%      71.75        vmstat.cpu.id
      1.75 ±173%    +514.3%      10.75 ± 10%  vmstat.cpu.us
  12537963 ± 28%    +128.4%   28638705 ± 10%  vmstat.memory.free
     35.50 ±  9%     -24.6%      26.75        vmstat.procs.r
     15622 ± 12%     -40.5%       9291 ± 19%  vmstat.system.cs
    996.50 ±  6%     -18.9%     808.25        turbostat.Avg_MHz
     35.75 ±  7%      -6.8       28.98        turbostat.Busy%
  39842096           -41.1%   23461830 ± 35%  turbostat.C1E
     64.25 ±  3%     +10.4%      70.91        turbostat.CPU%c1
 2.505e+09 ± 13%     -39.7%   1.51e+09 ± 18%  turbostat.IRQ
    123.71 ±  6%     +15.9%     143.33 ±  2%  turbostat.RAMWatt
 1.364e+08           -44.2%   76113715 ±  5%  meminfo.Active
 1.364e+08           -44.2%   76113426 ±  5%  meminfo.Active(anon)
     31073 ±  9%     +59.4%      49518 ± 18%  meminfo.CmaFree
   4626918         +1088.6%   54997790 ±  6%  meminfo.Inactive
   4626746         +1088.7%   54997590 ±  6%  meminfo.Inactive(anon)
  19420044 ± 11%     +50.8%   29281487 ± 14%  meminfo.MemAvailable
  19834275 ± 11%     +50.1%   29771645 ± 14%  meminfo.MemFree
  24708739 ±  2%     -45.4%   13487279 ±  3%  numa-meminfo.node0.Active
  24708607 ±  2%     -45.4%   13487147 ±  3%  numa-meminfo.node0.Active(anon)
   1492968          +700.4%   11949636        numa-meminfo.node0.Inactive
   1492852          +700.4%   11949531        numa-meminfo.node0.Inactive(anon)
   3664204 ± 13%     +22.1%    4472872 ± 12%  numa-meminfo.node0.MemFree
 1.105e+08           -43.5%   62382795 ±  5%  numa-meminfo.node1.Active
 1.105e+08           -43.5%   62382639 ±  5%  numa-meminfo.node1.Active(anon)
   3080818 ±  2%   +1287.9%   42758179 ±  8%  numa-meminfo.node1.Inactive
   3080752 ±  2%   +1287.9%   42758082 ±  8%  numa-meminfo.node1.Inactive(anon)
  17418249 ±  6%     +48.3%   25834001 ± 17%  numa-meminfo.node1.MemFree
   6121963           -45.0%    3368248 ±  2%  numa-vmstat.node0.nr_active_anon
    368430          +709.7%    2983079        numa-vmstat.node0.nr_inactive_anon
   6121687           -45.0%    3368112 ±  2%  numa-vmstat.node0.nr_zone_active_anon
    368520          +709.5%    2983175        numa-vmstat.node0.nr_zone_inactive_anon
  27337631           -43.0%   15571855 ±  5%  numa-vmstat.node1.nr_active_anon
      8825 ±  7%     +42.6%      12586 ± 17%  numa-vmstat.node1.nr_free_cma
   4651536 ±  7%     +39.6%    6493326 ± 17%  numa-vmstat.node1.nr_free_pages
    759791 ±  2%   +1305.5%   10678592 ±  8%  numa-vmstat.node1.nr_inactive_anon
    491.75 ± 28%     -62.4%     184.75 ± 58%  numa-vmstat.node1.nr_isolated_anon
  24766575 ± 18%     -44.0%   13859209 ± 50%  numa-vmstat.node1.nr_vmscan_write
  24766573 ± 18%     -44.0%   13859230 ± 50%  numa-vmstat.node1.nr_written
  27337629           -43.0%   15571850 ±  5%  numa-vmstat.node1.nr_zone_active_anon
    759789 ±  2%   +1305.5%   10678589 ±  8%  numa-vmstat.node1.nr_zone_inactive_anon
    549691 ± 35%     -67.5%     178751 ± 21%  proc-vmstat.allocstall_movable
     32524 ± 16%     -89.8%       3306 ± 29%  proc-vmstat.allocstall_normal
     31609 ± 17%     -90.9%       2875 ± 37%  proc-vmstat.compact_fail
     31612 ± 17%     -90.9%       2877 ± 37%  proc-vmstat.compact_stall
  33869380           -43.9%   19007185 ±  5%  proc-vmstat.nr_active_anon
  35011556            -6.4%   32756779 ±  3%  proc-vmstat.nr_anon_pages
    508004 ±  7%     +44.0%     731324 ± 13%  proc-vmstat.nr_dirty_background_threshold
   1017251 ±  7%     +44.0%    1464437 ± 13%  proc-vmstat.nr_dirty_threshold
      8160 ±  5%     +51.4%      12356 ± 17%  proc-vmstat.nr_free_cma
   5196946 ±  8%     +43.5%    7456562 ± 13%  proc-vmstat.nr_free_pages
   1149058         +1097.2%   13757046 ±  6%  proc-vmstat.nr_inactive_anon
    700.00 ± 24%     -53.9%     322.50 ± 19%  proc-vmstat.nr_isolated_anon
  34985843 ± 23%     -29.4%   24708434 ± 12%  proc-vmstat.nr_vmscan_write
  56426651 ± 21%     -42.6%   32385621 ± 14%  proc-vmstat.nr_written
  33869111           -43.9%   19007050 ±  5%  proc-vmstat.nr_zone_active_anon
   1149142         +1097.2%   13757150 ±  6%  proc-vmstat.nr_zone_inactive_anon
   1759803 ±160%    +258.2%    6303060 ± 43%  proc-vmstat.numa_pages_migrated
  83508236 ± 12%     -53.7%   38673221 ± 45%  proc-vmstat.numa_pte_updates
  50177452 ± 31%     -35.7%   32288386 ± 14%  proc-vmstat.pgmajfault
   1811523 ±158%    +250.4%    6348267 ± 43%  proc-vmstat.pgmigrate_success
  51090830 ± 19%     -43.5%   28849729 ± 14%  proc-vmstat.pgsteal_direct
  50190319 ± 31%     -35.7%   32291856 ± 14%  proc-vmstat.pswpin
  56429784 ± 21%     -42.6%   32386842 ± 14%  proc-vmstat.pswpout
     10369 ±  4%     -72.3%       2876 ± 94%  proc-vmstat.swap_ra
      7915 ±  5%     -71.0%       2292 ± 95%  proc-vmstat.swap_ra_hit
     15.21 ± 12%     +24.2%      18.89        perf-stat.i.MPKI
 3.399e+09 ± 21%     +45.3%  4.938e+09 ±  5%  perf-stat.i.branch-instructions
      0.72 ±  4%      -0.2        0.55 ±  2%  perf-stat.i.branch-miss-rate%
     42.66 ± 17%     +22.3       64.92 ±  3%  perf-stat.i.cache-miss-rate%
 1.205e+08 ± 59%    +150.7%   3.02e+08 ± 10%  perf-stat.i.cache-misses
 2.542e+08 ± 36%     +73.6%  4.413e+08 ±  6%  perf-stat.i.cache-references
     15855 ± 11%     -41.3%       9304 ± 19%  perf-stat.i.context-switches
      6.62 ± 22%     -48.1%       3.44 ±  7%  perf-stat.i.cpi
 9.463e+10 ±  7%     -18.7%   7.69e+10        perf-stat.i.cpu-cycles
    200.14 ± 12%     -38.2%     123.70 ±  3%  perf-stat.i.cpu-migrations
      1302 ± 35%     -73.1%     350.31 ±  9%  perf-stat.i.cycles-between-cache-misses
 4.039e+09 ± 25%     +50.8%  6.092e+09 ±  5%  perf-stat.i.dTLB-loads
      1.04 ±120%      +3.6        4.69 ±  5%  perf-stat.i.dTLB-store-miss-rate%
  25910902 ±139%    +425.0%   1.36e+08 ± 12%  perf-stat.i.dTLB-store-misses
 1.495e+09 ± 42%     +80.4%  2.697e+09 ±  6%  perf-stat.i.dTLB-stores
     77.52            -7.1       70.37 ±  3%  perf-stat.i.iTLB-load-miss-rate%
   1801278 ± 13%     +40.1%    2524387 ± 10%  perf-stat.i.iTLB-loads
 1.547e+10 ± 23%     +46.9%  2.273e+10 ±  5%  perf-stat.i.instructions
      2309 ± 15%    +108.6%       4818 ± 28%  perf-stat.i.instructions-per-iTLB-miss
      0.20 ± 27%     +68.4%       0.33 ±  5%  perf-stat.i.ipc
     78.32 ±  6%     -19.4       58.88 ±  4%  perf-stat.i.node-load-miss-rate%
  29707760 ± 53%    +107.1%   61538597 ± 12%  perf-stat.i.node-load-misses
  10577749 ± 98%    +389.1%   51733544 ± 13%  perf-stat.i.node-loads
  22026270 ± 78%    +238.8%   74620843 ± 15%  perf-stat.i.node-store-misses
  16362080 ±130%    +356.6%   74713148 ±  8%  perf-stat.i.node-stores
     15.93 ± 11%     +21.8%      19.40        perf-stat.overall.MPKI
      0.71 ±  6%      -0.2        0.49 ±  6%  perf-stat.overall.branch-miss-rate%
     44.06 ± 18%     +24.1       68.15 ±  4%  perf-stat.overall.cache-miss-rate%
      6.55 ± 23%     -48.0%       3.41 ±  6%  perf-stat.overall.cpi
      1042 ± 38%     -75.1%     259.32 ± 11%  perf-stat.overall.cycles-between-cache-misses
      1.15 ±107%      +3.6        4.76 ±  7%  perf-stat.overall.dTLB-store-miss-rate%
     83.19            -2.8       80.43        perf-stat.overall.iTLB-load-miss-rate%
      1708           +29.7%       2216 ±  9%  perf-stat.overall.instructions-per-iTLB-miss
      0.17 ± 32%     +78.0%       0.29 ±  6%  perf-stat.overall.ipc
     78.21 ±  8%     -23.8       54.46 ±  3%  perf-stat.overall.node-load-miss-rate%
    106084 ± 42%     -81.3%      19825 ±  8%  perf-stat.overall.path-length
  3.38e+09 ± 21%     +45.3%  4.913e+09 ±  5%  perf-stat.ps.branch-instructions
 1.183e+08 ± 59%    +153.5%  2.997e+08 ± 10%  perf-stat.ps.cache-misses
 2.511e+08 ± 37%     +74.8%  4.388e+08 ±  6%  perf-stat.ps.cache-references
     15666 ± 12%     -40.6%       9308 ± 20%  perf-stat.ps.context-switches
 9.496e+10 ±  7%     -19.1%  7.681e+10        perf-stat.ps.cpu-cycles
    200.19 ± 12%     -38.4%     123.32 ±  3%  perf-stat.ps.cpu-migrations
 4.007e+09 ± 25%     +51.2%   6.06e+09 ±  5%  perf-stat.ps.dTLB-loads
  25372732 ±140%    +430.9%  1.347e+08 ± 12%  perf-stat.ps.dTLB-store-misses
 1.468e+09 ± 43%     +82.7%  2.682e+09 ±  6%  perf-stat.ps.dTLB-stores
   1784409 ± 13%     +40.2%    2500926 ± 10%  perf-stat.ps.iTLB-loads
 1.535e+10 ± 23%     +47.3%  2.261e+10 ±  5%  perf-stat.ps.instructions
  29058792 ± 54%    +110.1%   61065800 ± 13%  perf-stat.ps.node-load-misses
  10281631 ± 99%    +397.6%   51163481 ± 13%  perf-stat.ps.node-loads
  21669221 ± 78%    +241.9%   74083941 ± 15%  perf-stat.ps.node-store-misses
  15995619 ±131%    +362.3%   73945284 ±  8%  perf-stat.ps.node-stores
     26558 ± 88%     -99.9%      38.22 ±100%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    975447 ± 74%     -99.6%       3631 ±100%  sched_debug.cfs_rq:/.MIN_vruntime.max
    144147 ± 78%     -99.7%     370.60 ±100%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     54372 ± 12%     -50.4%      26965 ± 26%  sched_debug.cfs_rq:/.exec_clock.avg
     90682 ± 12%     -37.9%      56334 ± 26%  sched_debug.cfs_rq:/.exec_clock.max
     15555 ± 30%     -71.3%       4470 ± 51%  sched_debug.cfs_rq:/.exec_clock.min
     19912 ± 14%     -40.2%      11913 ± 18%  sched_debug.cfs_rq:/.exec_clock.stddev
     47146 ± 23%     -54.7%      21343 ±  8%  sched_debug.cfs_rq:/.load.avg
    969122 ± 15%     -57.3%     414213 ± 16%  sched_debug.cfs_rq:/.load.max
    170010 ± 19%     -59.2%      69303 ± 12%  sched_debug.cfs_rq:/.load.stddev
     67.07 ±  9%     -44.4%      37.28 ±  9%  sched_debug.cfs_rq:/.load_avg.avg
    979.73 ±  7%     -28.2%     703.55 ±  3%  sched_debug.cfs_rq:/.load_avg.max
    164.28 ±  7%     -31.3%     112.79 ±  3%  sched_debug.cfs_rq:/.load_avg.stddev
     26558 ± 88%     -99.9%      38.22 ±100%  sched_debug.cfs_rq:/.max_vruntime.avg
    975447 ± 74%     -99.6%       3631 ±100%  sched_debug.cfs_rq:/.max_vruntime.max
    144147 ± 78%     -99.7%     370.60 ±100%  sched_debug.cfs_rq:/.max_vruntime.stddev
   1795885 ± 15%     -54.0%     826131 ± 24%  sched_debug.cfs_rq:/.min_vruntime.avg
   3028953 ± 15%     -41.9%    1761308 ± 24%  sched_debug.cfs_rq:/.min_vruntime.max
    475090 ± 40%     -66.9%     157363 ± 46%  sched_debug.cfs_rq:/.min_vruntime.min
    685787 ± 16%     -47.6%     359491 ± 15%  sched_debug.cfs_rq:/.min_vruntime.stddev
      1.69 ± 20%     -40.9%       1.00        sched_debug.cfs_rq:/.nr_running.max
    365.48 ± 42%     -96.4%      13.16 ± 27%  sched_debug.cfs_rq:/.nr_spread_over.avg
    611.97 ± 39%     -93.2%      41.82 ± 27%  sched_debug.cfs_rq:/.nr_spread_over.max
     96.61 ± 60%     -99.8%       0.20 ±173%  sched_debug.cfs_rq:/.nr_spread_over.min
    148.45 ± 36%     -93.6%       9.53 ± 21%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      6.57 ± 29%     -64.5%       2.34 ± 62%  sched_debug.cfs_rq:/.removed.load_avg.avg
    303.68 ± 30%     -64.5%     107.89 ± 62%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
     29.60 ±  6%     -32.8%      19.89 ±  8%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    741.80 ±  9%     -46.3%     398.72 ± 16%  sched_debug.cfs_rq:/.runnable_load_avg.max
    116.04 ±  8%     -42.7%      66.48 ± 11%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     46961 ± 23%     -55.0%      21133 ±  8%  sched_debug.cfs_rq:/.runnable_weight.avg
    966639 ± 16%     -57.7%     408956 ± 16%  sched_debug.cfs_rq:/.runnable_weight.max
    169671 ± 19%     -59.5%      68675 ± 12%  sched_debug.cfs_rq:/.runnable_weight.stddev
  -1585084           -54.1%    -727434        sched_debug.cfs_rq:/.spread0.min
    686090 ± 16%     -47.6%     359495 ± 15%  sched_debug.cfs_rq:/.spread0.stddev
    369169 ± 28%     -76.9%      85102 ±124%  sched_debug.cpu.avg_idle.min
    159830 ± 22%     +43.3%     229036 ± 16%  sched_debug.cpu.avg_idle.stddev
    169467 ±  9%     -37.7%     105562 ± 25%  sched_debug.cpu.clock.avg
    169556 ±  9%     -37.7%     105595 ± 25%  sched_debug.cpu.clock.max
    169374 ±  9%     -37.7%     105532 ± 25%  sched_debug.cpu.clock.min
     59.61 ± 27%     -61.0%      23.27 ± 93%  sched_debug.cpu.clock.stddev
    169467 ±  9%     -37.7%     105562 ± 25%  sched_debug.cpu.clock_task.avg
    169556 ±  9%     -37.7%     105595 ± 25%  sched_debug.cpu.clock_task.max
    169374 ±  9%     -37.7%     105532 ± 25%  sched_debug.cpu.clock_task.min
     59.61 ± 27%     -61.0%      23.27 ± 93%  sched_debug.cpu.clock_task.stddev
    763.75 ±  4%     -20.4%     607.71 ±  9%  sched_debug.cpu.curr->pid.avg
      0.00 ± 27%     -40.2%       0.00 ± 36%  sched_debug.cpu.next_balance.stddev
      0.34 ±  6%     -22.8%       0.26 ± 10%  sched_debug.cpu.nr_running.avg
      1.69 ± 20%     -37.9%       1.05 ±  8%  sched_debug.cpu.nr_running.max
     29805 ±  7%     -58.3%      12417 ± 19%  sched_debug.cpu.nr_switches.avg
     58194 ±  6%     -54.5%      26453 ± 13%  sched_debug.cpu.nr_switches.max
      7859 ± 14%     -68.5%       2475 ± 32%  sched_debug.cpu.nr_switches.min
     11385 ±  6%     -53.2%       5333 ± 19%  sched_debug.cpu.nr_switches.stddev
    -58.44           -51.9%     -28.08        sched_debug.cpu.nr_uninterruptible.min
     22.55 ± 23%     -50.9%      11.07 ±  9%  sched_debug.cpu.nr_uninterruptible.stddev
     28132 ±  7%     -61.8%      10745 ± 22%  sched_debug.cpu.sched_count.avg
     54386 ±  8%     -59.0%      22312 ± 15%  sched_debug.cpu.sched_count.max
      6691 ± 18%     -79.3%       1388 ± 49%  sched_debug.cpu.sched_count.min
     10944 ±  6%     -55.1%       4910 ± 21%  sched_debug.cpu.sched_count.stddev
      2111 ±  9%     -59.3%     859.70 ± 29%  sched_debug.cpu.sched_goidle.avg
      5200 ± 10%     -37.2%       3267 ± 13%  sched_debug.cpu.sched_goidle.max
    711.88 ± 20%     -68.7%     222.52 ± 49%  sched_debug.cpu.sched_goidle.min
    902.92 ±  6%     -37.9%     560.76 ± 20%  sched_debug.cpu.sched_goidle.stddev
     14109 ±  7%     -62.0%       5365 ± 22%  sched_debug.cpu.ttwu_count.avg
     28421 ±  8%     -59.0%      11649 ± 20%  sched_debug.cpu.ttwu_count.max
      2930 ± 28%     -78.3%     635.75 ± 54%  sched_debug.cpu.ttwu_count.min
      6027 ±  7%     -57.1%       2583 ± 21%  sched_debug.cpu.ttwu_count.stddev
      3334 ± 12%     -38.9%       2036 ± 22%  sched_debug.cpu.ttwu_local.avg
      6204 ± 14%     -29.9%       4349 ± 12%  sched_debug.cpu.ttwu_local.max
      1109 ± 22%     -75.3%     273.98 ± 57%  sched_debug.cpu.ttwu_local.min
      1173 ± 12%     -18.6%     955.02 ± 13%  sched_debug.cpu.ttwu_local.stddev
    169374 ±  9%     -37.7%     105533 ± 25%  sched_debug.cpu_clk
    168870 ±  9%     -37.8%     105029 ± 25%  sched_debug.ktime
    169728 ±  9%     -37.6%     105890 ± 25%  sched_debug.sched_clk
     34.81 ± 15%      -8.7       26.14 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     34.66 ± 16%      -8.6       26.09 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     34.66 ± 16%      -8.6       26.09 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     34.66 ± 16%      -8.6       26.09 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     34.19 ± 17%      -8.3       25.93 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     34.22 ± 17%      -8.3       25.96 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.73 ± 10%      -0.5        2.23 ± 10%  perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      0.80 ± 10%      -0.2        0.62 ±  9%  perf-profile.calltrace.cycles-pp.mem_cgroup_swapout.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec
      0.66 ±  9%      +0.1        0.78 ±  3%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.62 ± 10%      +0.1        0.76 ±  4%  perf-profile.calltrace.cycles-pp.drain_local_pages_wq.process_one_work.worker_thread.kthread.ret_from_fork
      0.62 ± 10%      +0.1        0.76 ±  4%  perf-profile.calltrace.cycles-pp.drain_pages.drain_local_pages_wq.process_one_work.worker_thread.kthread
      0.61 ±  9%      +0.1        0.75 ±  4%  perf-profile.calltrace.cycles-pp.drain_pages_zone.drain_pages.drain_local_pages_wq.process_one_work.worker_thread
      0.43 ± 58%      +0.3        0.72 ±  5%  perf-profile.calltrace.cycles-pp.free_pcppages_bulk.drain_pages_zone.drain_pages.drain_local_pages_wq.process_one_work
      1.13 ± 22%      +0.3        1.47 ±  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.mem_cgroup_commit_charge.do_swap_page.handle_pte_fault
      1.20 ± 22%      +0.4        1.58 ±  9%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.mem_cgroup_commit_charge.do_swap_page.handle_pte_fault.__handle_mm_fault
      1.48 ± 20%      +0.5        1.94 ±  8%  perf-profile.calltrace.cycles-pp.mem_cgroup_commit_charge.do_swap_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
      0.86 ± 12%      +0.7        1.52 ± 14%  perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list
      0.44 ± 59%      +0.7        1.12 ±  6%  perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced.shrink_active_list
      3.29 ± 15%      +0.8        4.10 ±  4%  perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat
      0.52 ± 67%      +0.8        1.36 ± 34%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.smp_call_function_interrupt.call_function_interrupt.do_access
      3.35 ± 15%      +0.8        4.19 ±  4%  perf-profile.calltrace.cycles-pp.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat.kswapd
      0.54 ± 68%      +0.9        1.39 ± 35%  perf-profile.calltrace.cycles-pp.smp_call_function_interrupt.call_function_interrupt.do_access
      0.67 ± 11%      +0.9        1.57 ± 22%  perf-profile.calltrace.cycles-pp.page_referenced_one.rmap_walk_anon.page_referenced.shrink_active_list.shrink_lruvec
      1.00 ±  9%      +1.0        2.02 ±  2%  perf-profile.calltrace.cycles-pp.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list
      0.63 ± 67%      +1.1        1.68 ± 34%  perf-profile.calltrace.cycles-pp.call_function_interrupt.do_access
      3.40 ± 15%      +1.1        4.46 ±  4%  perf-profile.calltrace.cycles-pp.shrink_node.balance_pgdat.kswapd.kthread.ret_from_fork
      3.40 ± 15%      +1.1        4.47 ±  4%  perf-profile.calltrace.cycles-pp.kswapd.kthread.ret_from_fork
      3.40 ± 15%      +1.1        4.47 ±  4%  perf-profile.calltrace.cycles-pp.balance_pgdat.kswapd.kthread.ret_from_fork
      3.39 ± 15%      +1.1        4.46 ±  4%  perf-profile.calltrace.cycles-pp.shrink_lruvec.shrink_node.balance_pgdat.kswapd.kthread
      1.41 ± 25%      +1.1        2.52        perf-profile.calltrace.cycles-pp.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec
      5.11 ±  9%      +1.1        6.25 ±  3%  perf-profile.calltrace.cycles-pp.ret_from_fork
      5.11 ±  9%      +1.1        6.25 ±  3%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      1.46 ± 25%      +1.1        2.60 ±  2%  perf-profile.calltrace.cycles-pp.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      0.77 ± 11%      +1.3        2.09 ±  6%  perf-profile.calltrace.cycles-pp.rmap_walk_anon.page_referenced.shrink_active_list.shrink_lruvec.shrink_node
      0.81 ± 11%      +1.4        2.18 ±  6%  perf-profile.calltrace.cycles-pp.page_referenced.shrink_active_list.shrink_lruvec.shrink_node.do_try_to_free_pages
      1.62 ± 24%      +1.9        3.52 ±  7%  perf-profile.calltrace.cycles-pp.shrink_active_list.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages
     34.82 ± 15%      -8.7       26.14 ±  3%  perf-profile.children.cycles-pp.do_idle
     34.81 ± 15%      -8.7       26.14 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64
     34.81 ± 15%      -8.7       26.14 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     34.66 ± 16%      -8.6       26.09 ±  3%  perf-profile.children.cycles-pp.start_secondary
     34.37 ± 16%      -8.4       26.00 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter
     34.37 ± 16%      -8.4       26.00 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter_state
      2.89 ±  9%      -0.5        2.41 ±  9%  perf-profile.children.cycles-pp.__remove_mapping
      0.58 ± 15%      -0.4        0.23 ± 20%  perf-profile.children.cycles-pp.shrink_slab
      0.56 ± 14%      -0.3        0.22 ± 23%  perf-profile.children.cycles-pp.do_shrink_slab
      0.41 ± 14%      -0.3        0.16 ± 23%  perf-profile.children.cycles-pp.count_shadow_nodes
      1.20 ±  9%      -0.2        0.96 ±  9%  perf-profile.children.cycles-pp.mem_cgroup_swapout
      0.95 ±  7%      -0.2        0.73 ±  9%  perf-profile.children.cycles-pp.swap_cgroup_record
      0.92 ± 14%      -0.2        0.73 ± 13%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.30 ± 15%      -0.1        0.20 ± 18%  perf-profile.children.cycles-pp.lru_add_drain
      0.72 ±  6%      -0.1        0.63 ±  5%  perf-profile.children.cycles-pp.cpumask_next
      0.11 ± 14%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.drain_all_pages
      0.13 ± 18%      -0.1        0.05 ± 59%  perf-profile.children.cycles-pp.zone_reclaimable_pages
      0.12 ± 20%      -0.1        0.05 ± 59%  perf-profile.children.cycles-pp.throttle_direct_reclaim
      0.12 ± 20%      -0.1        0.05 ± 59%  perf-profile.children.cycles-pp.allow_direct_reclaim
      0.10 ± 12%      -0.1        0.04 ±100%  perf-profile.children.cycles-pp.do_page_add_anon_rmap
      0.08 ± 15%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.queue_work_on
      0.13 ± 17%      -0.1        0.08 ± 17%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.15 ± 12%      -0.0        0.10 ± 11%  perf-profile.children.cycles-pp.__sched_text_start
      0.09 ± 13%      -0.0        0.05 ± 60%  perf-profile.children.cycles-pp.schedule
      0.08 ± 11%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.13 ±  6%      +0.0        0.15 ±  2%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.09 ±  4%      +0.0        0.11 ±  7%  perf-profile.children.cycles-pp.scheduler_tick
      0.04 ± 57%      +0.0        0.06        perf-profile.children.cycles-pp.__count_memcg_events
      0.11 ±  4%      +0.0        0.14 ±  6%  perf-profile.children.cycles-pp.sync_regs
      0.15 ± 10%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.flush_tlb_func_remote
      0.10 ±  8%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp.page_mapping
      0.15 ±  6%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.tick_sched_handle
      0.18 ±  8%      +0.0        0.22 ±  3%  perf-profile.children.cycles-pp.tick_sched_timer
      0.04 ± 57%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.page_evictable
      0.23 ±  8%      +0.0        0.28 ±  4%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.06 ± 26%      +0.0        0.11 ± 20%  perf-profile.children.cycles-pp.ptep_clear_flush_young
      0.03 ±100%      +0.1        0.08 ± 22%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.36 ±  9%      +0.1        0.42 ±  6%  perf-profile.children.cycles-pp.__lru_cache_add
      0.33 ± 10%      +0.1        0.39 ±  7%  perf-profile.children.cycles-pp.up_read
      0.13 ±  9%      +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.__isolate_lru_page
      0.65 ±  7%      +0.1        0.72 ±  3%  perf-profile.children.cycles-pp.xas_create_range
      0.65 ±  6%      +0.1        0.72 ±  3%  perf-profile.children.cycles-pp.xas_create
      0.01 ±173%      +0.1        0.09 ± 55%  perf-profile.children.cycles-pp.dup_mm
      0.04 ± 63%      +0.1        0.13 ± 45%  perf-profile.children.cycles-pp.__libc_fork
      0.02 ±173%      +0.1        0.12 ± 52%  perf-profile.children.cycles-pp.copy_process
      0.02 ±173%      +0.1        0.12 ± 49%  perf-profile.children.cycles-pp.__x64_sys_clone
      0.02 ±173%      +0.1        0.12 ± 49%  perf-profile.children.cycles-pp._do_fork
      0.62 ±  7%      +0.1        0.74 ±  6%  perf-profile.children.cycles-pp.down_read_trylock
      0.12 ± 36%      +0.1        0.24 ± 37%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.12 ± 36%      +0.1        0.24 ± 37%  perf-profile.children.cycles-pp.do_syscall_64
      0.66 ±  9%      +0.1        0.78 ±  3%  perf-profile.children.cycles-pp.process_one_work
      0.21 ±  8%      +0.1        0.34 ±  3%  perf-profile.children.cycles-pp.move_pages_to_lru
      0.62 ± 10%      +0.1        0.76 ±  4%  perf-profile.children.cycles-pp.drain_local_pages_wq
      0.62 ± 10%      +0.1        0.76 ±  4%  perf-profile.children.cycles-pp.drain_pages
      0.61 ±  9%      +0.1        0.75 ±  4%  perf-profile.children.cycles-pp.drain_pages_zone
      0.56 ±  9%      +0.2        0.73 ±  4%  perf-profile.children.cycles-pp.free_pcppages_bulk
      1.44 ±  7%      +0.2        1.63        perf-profile.children.cycles-pp.__default_send_IPI_dest_field
      0.30 ±  8%      +0.2        0.49 ±  9%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.44 ± 11%      +0.2        0.67 ±  3%  perf-profile.children.cycles-pp.page_lock_anon_vma_read
      0.35 ±  8%      +0.2        0.60 ±  5%  perf-profile.children.cycles-pp.isolate_lru_pages
      1.53 ± 20%      +0.5        2.00 ±  8%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      2.00 ± 20%      +0.7        2.72 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      3.40 ± 15%      +1.1        4.47 ±  4%  perf-profile.children.cycles-pp.kswapd
      3.40 ± 15%      +1.1        4.47 ±  4%  perf-profile.children.cycles-pp.balance_pgdat
      5.11 ±  9%      +1.1        6.25 ±  3%  perf-profile.children.cycles-pp.ret_from_fork
      5.11 ±  9%      +1.1        6.25 ±  3%  perf-profile.children.cycles-pp.kthread
      2.46 ± 12%      +1.7        4.12 ±  3%  perf-profile.children.cycles-pp.page_vma_mapped_walk
      2.48 ±  9%      +1.7        4.20 ±  3%  perf-profile.children.cycles-pp.page_referenced_one
      7.12 ± 10%      +1.9        9.01 ±  6%  perf-profile.children.cycles-pp.rmap_walk_anon
      1.92 ±  8%      +1.9        3.85 ±  6%  perf-profile.children.cycles-pp.shrink_active_list
      3.13 ±  9%      +2.1        5.23 ±  3%  perf-profile.children.cycles-pp.page_referenced
      0.21 ± 14%      -0.1        0.09 ± 20%  perf-profile.self.cycles-pp.count_shadow_nodes
      0.33 ± 29%      -0.1        0.23 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.08 ± 10%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.do_page_add_anon_rmap
      0.09 ± 14%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.find_next_bit
      0.08 ±  8%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.06 ± 11%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.mem_cgroup_commit_charge
      0.07 ±  7%      +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.___might_sleep
      0.13 ±  6%      +0.0        0.15 ±  8%  perf-profile.self.cycles-pp.clear_page_erms
      0.11 ±  4%      +0.0        0.14 ±  6%  perf-profile.self.cycles-pp.sync_regs
      0.13 ±  9%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.flush_tlb_func_remote
      0.08 ±  6%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.page_mapping
      0.03 ±100%      +0.0        0.06        perf-profile.self.cycles-pp.__count_memcg_events
      0.18 ±  7%      +0.0        0.22 ±  4%  perf-profile.self.cycles-pp.shrink_page_list
      0.08 ± 19%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.isolate_lru_pages
      0.26 ± 10%      +0.1        0.32 ±  8%  perf-profile.self.cycles-pp.up_read
      0.08 ± 13%      +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.rmap_walk_anon
      0.05 ±  9%      +0.1        0.11 ±  7%  perf-profile.self.cycles-pp.shrink_active_list
      0.02 ±173%      +0.1        0.08 ± 24%  perf-profile.self.cycles-pp.ptep_clear_flush_young
      0.13 ±  6%      +0.1        0.19 ±  7%  perf-profile.self.cycles-pp.__isolate_lru_page
      0.11 ±  7%      +0.1        0.19 ±  4%  perf-profile.self.cycles-pp.move_pages_to_lru
      0.49 ±  7%      +0.1        0.59 ±  4%  perf-profile.self.cycles-pp.down_read_trylock
      0.14 ± 18%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.page_lock_anon_vma_read
      0.84 ±  9%      +0.1        0.95 ±  3%  perf-profile.self.cycles-pp.__memcpy_mcsafe
      0.39 ± 10%      +0.1        0.54 ±  8%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.29 ±  7%      +0.2        0.48 ± 11%  perf-profile.self.cycles-pp.__list_del_entry_valid
      1.44 ±  7%      +0.2        1.63        perf-profile.self.cycles-pp.__default_send_IPI_dest_field
      1.66 ± 11%      +1.3        2.93 ±  2%  perf-profile.self.cycles-pp.page_vma_mapped_walk
      2.90 ± 29%      +4.0        6.88 ± 30%  perf-profile.self.cycles-pp.do_access
    212597 ±  3%     -36.7%     134609 ± 20%  softirqs.CPU0.RCU
     34458 ±  7%     -28.9%      24501 ± 15%  softirqs.CPU0.SCHED
    119264 ±  3%     -36.3%      75966 ± 16%  softirqs.CPU0.TIMER
    224311 ± 11%     -50.2%     111697 ± 24%  softirqs.CPU1.RCU
    108371 ±  4%     -33.3%      72274 ± 17%  softirqs.CPU1.TIMER
    205471 ± 10%     -42.2%     118805 ± 24%  softirqs.CPU10.RCU
     20700 ±  5%     -15.3%      17535 ± 10%  softirqs.CPU10.SCHED
    106098 ±  2%     -32.0%      72190 ± 14%  softirqs.CPU10.TIMER
    218193 ±  8%     -47.7%     114174 ± 16%  softirqs.CPU11.RCU
    106617 ±  3%     -33.8%      70591 ± 12%  softirqs.CPU11.TIMER
    187953 ± 13%     -29.7%     132077 ± 22%  softirqs.CPU12.RCU
     22696 ± 12%     -30.8%      15699 ± 15%  softirqs.CPU12.SCHED
    107273 ±  2%     -32.8%      72034 ± 11%  softirqs.CPU12.TIMER
    192397 ±  7%     -39.1%     117159 ±  8%  softirqs.CPU13.RCU
    106529 ±  2%     -32.2%      72195 ± 16%  softirqs.CPU13.TIMER
    197089 ± 13%     -43.5%     111326 ± 27%  softirqs.CPU14.RCU
     22178 ±  8%     -34.6%      14498 ± 22%  softirqs.CPU14.SCHED
    106991 ±  3%     -35.3%      69184 ± 14%  softirqs.CPU14.TIMER
    188761 ±  8%     -37.8%     117357 ± 31%  softirqs.CPU15.RCU
     22852 ± 11%     -33.1%      15277 ± 10%  softirqs.CPU15.SCHED
    107472 ±  2%     -34.8%      70115 ± 15%  softirqs.CPU15.TIMER
    191795 ±  9%     -38.9%     117230 ± 24%  softirqs.CPU16.RCU
    106959 ±  2%     -32.6%      72107 ± 15%  softirqs.CPU16.TIMER
    193369 ±  7%     -42.5%     111124 ± 19%  softirqs.CPU17.RCU
    107069 ±  2%     -32.9%      71827 ± 17%  softirqs.CPU17.TIMER
    201873 ±  8%     -48.7%     103571 ± 21%  softirqs.CPU18.RCU
    107085 ±  3%     -32.5%      72311 ± 16%  softirqs.CPU18.TIMER
    190240 ±  7%     -47.4%     100093 ± 13%  softirqs.CPU19.RCU
    106423 ±  3%     -32.9%      71413 ± 15%  softirqs.CPU19.TIMER
    195272 ±  6%     -40.5%     116251 ± 14%  softirqs.CPU2.RCU
    107597 ±  3%     -32.4%      72693 ± 17%  softirqs.CPU2.TIMER
    191574 ± 18%     -38.1%     118626 ± 19%  softirqs.CPU20.RCU
     23052 ±  6%     -32.6%      15537 ± 18%  softirqs.CPU20.SCHED
    107194 ±  3%     -35.3%      69389 ± 13%  softirqs.CPU20.TIMER
    193364 ±  8%     -47.5%     101574 ± 23%  softirqs.CPU21.RCU
    107251 ±  3%     -34.5%      70296 ± 17%  softirqs.CPU21.TIMER
    198429 ± 20%     -38.0%     123051 ±  9%  softirqs.CPU22.RCU
     23816 ±  9%     -32.3%      16131 ± 24%  softirqs.CPU22.SCHED
    107315 ±  3%     -34.0%      70811 ± 16%  softirqs.CPU22.TIMER
    227693 ± 22%     -47.7%     118996 ± 20%  softirqs.CPU23.RCU
    107443 ±  2%     -33.2%      71742 ± 16%  softirqs.CPU23.TIMER
    192478 ± 10%     -35.8%     123601 ±  9%  softirqs.CPU24.RCU
     25240           -42.0%      14641 ± 18%  softirqs.CPU24.SCHED
    110499 ±  3%     -36.7%      69986 ± 16%  softirqs.CPU24.TIMER
    192438 ± 10%     -36.0%     123249 ±  9%  softirqs.CPU25.RCU
     28029           -46.1%      15101 ± 19%  softirqs.CPU25.SCHED
    111552 ±  4%     -37.9%      69290 ± 16%  softirqs.CPU25.TIMER
    200642 ± 10%     -39.9%     120565 ± 17%  softirqs.CPU26.RCU
     27243 ±  3%     -42.0%      15812 ±  9%  softirqs.CPU26.SCHED
    111871 ±  4%     -30.6%      77670 ± 24%  softirqs.CPU26.TIMER
    224570 ± 14%     -48.1%     116451 ± 15%  softirqs.CPU27.RCU
     27137           -40.4%      16176 ± 17%  softirqs.CPU27.SCHED
    113075 ±  4%     -38.9%      69115 ± 14%  softirqs.CPU27.TIMER
    199983 ±  8%     -41.3%     117365 ± 28%  softirqs.CPU28.RCU
     28084 ±  3%     -42.6%      16129 ± 17%  softirqs.CPU28.SCHED
    112645 ±  4%     -37.0%      70917 ± 17%  softirqs.CPU28.TIMER
    214907 ± 18%     -40.8%     127251 ± 25%  softirqs.CPU29.RCU
     27439 ±  8%     -43.2%      15577 ± 30%  softirqs.CPU29.SCHED
    114488 ±  5%     -38.3%      70605 ± 18%  softirqs.CPU29.TIMER
    198933 ± 13%     -40.2%     118885 ± 19%  softirqs.CPU3.RCU
    107240 ±  3%     -32.8%      72061 ± 16%  softirqs.CPU3.TIMER
    208934 ± 30%     -41.6%     122075 ± 22%  softirqs.CPU30.RCU
     27966 ±  4%     -35.4%      18070 ± 28%  softirqs.CPU30.SCHED
    112092 ±  4%     -36.9%      70726 ± 18%  softirqs.CPU30.TIMER
    208817 ± 29%     -44.3%     116388 ± 11%  softirqs.CPU31.RCU
     28037 ±  6%     -32.1%      19038 ± 19%  softirqs.CPU31.SCHED
    112327 ±  5%     -36.2%      71639 ± 18%  softirqs.CPU31.TIMER
    216585 ± 13%     -49.9%     108477 ± 22%  softirqs.CPU32.RCU
     27324           -27.8%      19740 ± 25%  softirqs.CPU32.SCHED
    113318 ±  3%     -36.7%      71725 ± 17%  softirqs.CPU32.TIMER
    221135 ±  8%     -46.3%     118677        softirqs.CPU33.RCU
     27061 ±  3%     -31.9%      18441 ± 24%  softirqs.CPU33.SCHED
    111711 ±  4%     -37.1%      70255 ± 17%  softirqs.CPU33.TIMER
    210277 ± 32%     -46.4%     112744 ± 14%  softirqs.CPU34.RCU
     27717 ±  5%     -39.5%      16779 ± 34%  softirqs.CPU34.SCHED
    112366 ±  4%     -37.4%      70370 ± 17%  softirqs.CPU34.TIMER
    184212 ±  9%     -34.3%     120981 ± 15%  softirqs.CPU35.RCU
     28622 ±  5%     -39.6%      17290 ± 33%  softirqs.CPU35.SCHED
    113332 ±  5%     -37.6%      70771 ± 17%  softirqs.CPU35.TIMER
    204778 ± 25%     -36.0%     131152 ± 10%  softirqs.CPU36.RCU
     28349 ±  3%     -45.8%      15366 ± 27%  softirqs.CPU36.SCHED
    112186 ±  4%     -37.6%      70026 ± 17%  softirqs.CPU36.TIMER
    200541 ± 13%     -48.5%     103298 ± 28%  softirqs.CPU37.RCU
     28567 ±  3%     -37.1%      17964 ± 24%  softirqs.CPU37.SCHED
    112440 ±  5%     -37.3%      70535 ± 16%  softirqs.CPU37.TIMER
    212997 ±  8%     -41.8%     123943 ± 26%  softirqs.CPU38.RCU
     28241 ±  3%     -41.6%      16501 ± 31%  softirqs.CPU38.SCHED
    112497 ±  5%     -38.1%      69654 ± 18%  softirqs.CPU38.TIMER
    223866 ±  9%     -47.7%     116973 ± 29%  softirqs.CPU39.RCU
     28790 ±  4%     -39.6%      17397 ±  3%  softirqs.CPU39.SCHED
    112979 ±  5%     -36.9%      71312 ± 14%  softirqs.CPU39.TIMER
    202471 ±  4%     -54.9%      91230 ± 23%  softirqs.CPU4.RCU
    107747 ±  4%     -33.8%      71303 ± 17%  softirqs.CPU4.TIMER
    209287 ± 24%     -41.5%     122445 ± 21%  softirqs.CPU40.RCU
     28465 ±  7%     -31.0%      19646 ± 20%  softirqs.CPU40.SCHED
    114380 ±  6%     -37.7%      71248 ± 17%  softirqs.CPU40.TIMER
    198709 ± 17%     -35.7%     127859 ± 22%  softirqs.CPU41.RCU
     29645 ±  7%     -56.7%      12833 ± 14%  softirqs.CPU41.SCHED
    113409 ±  6%     -40.3%      67665 ± 14%  softirqs.CPU41.TIMER
    200259 ±  9%     -43.0%     114225 ± 32%  softirqs.CPU42.RCU
     29362 ±  4%     -40.3%      17534 ± 18%  softirqs.CPU42.SCHED
    112675 ±  4%     -38.0%      69855 ± 16%  softirqs.CPU42.TIMER
    184335 ± 26%     -32.2%     124922 ± 15%  softirqs.CPU43.RCU
     31523 ±  6%     -50.7%      15535 ± 21%  softirqs.CPU43.SCHED
    113482 ±  5%     -38.8%      69497 ± 14%  softirqs.CPU43.TIMER
    198234 ± 21%     -44.1%     110845 ± 11%  softirqs.CPU44.RCU
     31196 ±  8%     -39.2%      18976 ± 26%  softirqs.CPU44.SCHED
    128044 ± 14%     -44.4%      71240 ± 17%  softirqs.CPU44.TIMER
    170742 ±  7%     -33.8%     112977 ± 28%  softirqs.CPU45.RCU
     32034 ±  3%     -37.1%      20148 ±  9%  softirqs.CPU45.SCHED
    113150 ±  5%     -36.9%      71351 ± 16%  softirqs.CPU45.TIMER
    166500 ±  6%     -42.0%      96638 ± 20%  softirqs.CPU46.RCU
     33143 ±  5%     -32.6%      22329 ± 11%  softirqs.CPU46.SCHED
    114318 ±  5%     -36.2%      72945 ± 15%  softirqs.CPU46.TIMER
    189046 ± 20%     -45.3%     103440 ±  7%  softirqs.CPU47.RCU
     31361 ±  8%     -35.3%      20285 ± 19%  softirqs.CPU47.SCHED
    114516 ±  6%     -37.1%      72059 ± 17%  softirqs.CPU47.TIMER
    165928 ±  9%     -41.8%      96556 ± 15%  softirqs.CPU48.RCU
     25404 ±  3%     -25.7%      18888 ± 21%  softirqs.CPU48.SCHED
    107777 ±  3%     -34.5%      70617 ± 16%  softirqs.CPU48.TIMER
    161388 ±  9%     -34.4%     105851 ± 22%  softirqs.CPU49.RCU
     27065 ±  4%     -37.4%      16955 ± 30%  softirqs.CPU49.SCHED
    108064 ±  3%     -35.1%      70134 ± 18%  softirqs.CPU49.TIMER
    175421 ±  2%     -38.2%     108331 ± 15%  softirqs.CPU5.RCU
    107703 ±  3%     -33.1%      72078 ± 17%  softirqs.CPU5.TIMER
    159746 ±  3%     -26.0%     118265 ± 21%  softirqs.CPU50.RCU
     27883 ±  4%     -32.9%      18696 ± 16%  softirqs.CPU50.SCHED
    108779 ±  2%     -33.2%      72627 ± 15%  softirqs.CPU50.TIMER
    174614 ± 15%     -41.8%     101615 ± 26%  softirqs.CPU51.RCU
     27517 ±  9%     -27.8%      19862 ± 21%  softirqs.CPU51.SCHED
    109020 ±  3%     -34.0%      71899 ± 17%  softirqs.CPU51.TIMER
    162723 ±  5%     -27.7%     117628 ± 22%  softirqs.CPU52.RCU
     28545 ±  2%     -36.5%      18131 ± 29%  softirqs.CPU52.SCHED
    109328 ±  3%     -34.9%      71142 ± 18%  softirqs.CPU52.TIMER
    189188 ±  8%     -47.7%      98941 ± 29%  softirqs.CPU53.RCU
     27347 ±  4%     -23.6%      20882 ± 13%  softirqs.CPU53.SCHED
    109465 ±  3%     -34.3%      71930 ± 17%  softirqs.CPU53.TIMER
    171994 ± 15%     -36.4%     109466 ± 15%  softirqs.CPU54.RCU
     27908 ±  7%     -43.9%      15660 ± 33%  softirqs.CPU54.SCHED
    109320 ±  3%     -36.7%      69231 ± 18%  softirqs.CPU54.TIMER
     28857 ±  3%     -38.3%      17791 ± 23%  softirqs.CPU55.SCHED
    108633 ±  3%     -27.3%      78954 ± 16%  softirqs.CPU55.TIMER
    161091 ±  6%     -36.1%     102976 ± 21%  softirqs.CPU56.RCU
     28941 ±  3%     -39.6%      17491 ± 28%  softirqs.CPU56.SCHED
    108770 ±  3%     -35.3%      70414 ± 17%  softirqs.CPU56.TIMER
    168677 ± 11%     -47.4%      88780 ± 30%  softirqs.CPU57.RCU
     28579 ±  7%     -31.7%      19514 ± 22%  softirqs.CPU57.SCHED
    108879 ±  2%     -34.5%      71308 ± 18%  softirqs.CPU57.TIMER
    168121 ±  7%     -30.6%     116622 ± 17%  softirqs.CPU58.RCU
     28386 ±  4%     -26.8%      20764 ± 22%  softirqs.CPU58.SCHED
    108995 ±  3%     -33.8%      72152 ± 17%  softirqs.CPU58.TIMER
    166790 ±  8%     -37.8%     103725 ± 20%  softirqs.CPU59.RCU
     28874 ±  4%     -33.3%      19247 ± 24%  softirqs.CPU59.SCHED
    109477 ±  3%     -34.7%      71525 ± 16%  softirqs.CPU59.TIMER
    194300 ± 10%     -42.6%     111575 ± 28%  softirqs.CPU6.RCU
    107006 ±  2%     -32.4%      72323 ± 18%  softirqs.CPU6.TIMER
    167019 ±  8%     -38.7%     102450 ± 22%  softirqs.CPU60.RCU
     29256 ±  4%     -28.2%      21006 ± 14%  softirqs.CPU60.SCHED
    109445 ±  3%     -34.6%      71604 ± 16%  softirqs.CPU60.TIMER
    180219 ±  8%     -45.0%      99184 ± 17%  softirqs.CPU61.RCU
     28828 ±  3%     -43.0%      16439 ± 31%  softirqs.CPU61.SCHED
    109601 ±  3%     -36.5%      69576 ± 19%  softirqs.CPU61.TIMER
    176801 ± 14%     -46.5%      94524 ± 19%  softirqs.CPU62.RCU
     29519 ±  6%     -28.9%      20983 ± 13%  softirqs.CPU62.SCHED
    125577 ± 21%     -42.6%      72102 ± 16%  softirqs.CPU62.TIMER
    191800 ±  6%     -52.6%      90888 ± 13%  softirqs.CPU63.RCU
     28408 ±  2%     -27.9%      20493 ± 25%  softirqs.CPU63.SCHED
    110145 ±  3%     -35.9%      70639 ± 18%  softirqs.CPU63.TIMER
    173336 ± 10%     -46.2%      93292 ± 36%  softirqs.CPU64.RCU
     29389 ±  2%     -36.9%      18531 ±  9%  softirqs.CPU64.SCHED
    109917 ±  3%     -36.3%      69987 ± 12%  softirqs.CPU64.TIMER
    176959 ±  6%     -36.6%     112132 ± 18%  softirqs.CPU65.RCU
     28577 ±  4%     -39.7%      17226 ± 23%  softirqs.CPU65.SCHED
    109849 ±  3%     -36.5%      69733 ± 15%  softirqs.CPU65.TIMER
    167478 ±  5%     -33.7%     111034 ± 23%  softirqs.CPU66.RCU
     29741 ±  4%     -39.6%      17948 ± 13%  softirqs.CPU66.SCHED
    110509 ±  3%     -35.8%      70998 ± 13%  softirqs.CPU66.TIMER
    174165 ±  7%     -38.9%     106353 ± 31%  softirqs.CPU67.RCU
     29347 ±  6%     -32.8%      19713 ±  8%  softirqs.CPU67.SCHED
    110246 ±  3%     -35.9%      70668 ± 13%  softirqs.CPU67.TIMER
    182354 ± 14%     -52.2%      87224 ± 19%  softirqs.CPU68.RCU
     29972 ±  5%     -35.1%      19464 ± 28%  softirqs.CPU68.SCHED
    110529 ±  3%     -35.9%      70802 ± 17%  softirqs.CPU68.TIMER
    164911 ±  6%     -37.8%     102536 ± 18%  softirqs.CPU69.RCU
     30131 ±  3%     -36.4%      19173 ± 16%  softirqs.CPU69.SCHED
    110459 ±  3%     -36.3%      70392 ± 15%  softirqs.CPU69.TIMER
    199777 ±  4%     -49.6%     100649 ± 15%  softirqs.CPU7.RCU
    105988 ±  2%     -31.3%      72813 ± 15%  softirqs.CPU7.TIMER
    164423 ± 10%     -34.7%     107313 ± 24%  softirqs.CPU70.RCU
     29856 ±  5%     -28.1%      21458 ± 10%  softirqs.CPU70.SCHED
    110273 ±  3%     -34.3%      72435 ± 16%  softirqs.CPU70.TIMER
    180543 ± 15%     -44.2%     100691 ± 18%  softirqs.CPU71.RCU
     29996 ±  8%     -44.7%      16584 ± 15%  softirqs.CPU71.SCHED
    125872 ± 19%     -44.9%      69321 ± 15%  softirqs.CPU71.TIMER
    181740 ± 17%     -49.6%      91673 ± 28%  softirqs.CPU72.RCU
     33195 ±  6%     -32.0%      22586 ± 12%  softirqs.CPU72.SCHED
    115086 ±  4%     -37.1%      72401 ± 17%  softirqs.CPU72.TIMER
    189936 ± 20%     -50.9%      93328 ± 21%  softirqs.CPU73.RCU
     32580 ±  9%     -33.8%      21574 ± 21%  softirqs.CPU73.SCHED
    112581 ±  5%     -36.7%      71234 ± 16%  softirqs.CPU73.TIMER
    200182 ± 26%     -49.4%     101352 ± 11%  softirqs.CPU74.RCU
     33030 ±  8%     -37.4%      20688 ± 29%  softirqs.CPU74.SCHED
    113377 ±  4%     -37.5%      70823 ± 17%  softirqs.CPU74.TIMER
    169517 ± 19%     -40.2%     101311 ± 28%  softirqs.CPU75.RCU
     34987 ±  6%     -38.3%      21570 ± 13%  softirqs.CPU75.SCHED
    115544 ±  5%     -38.4%      71123 ± 17%  softirqs.CPU75.TIMER
    172364 ± 21%     -49.7%      86661 ± 25%  softirqs.CPU76.RCU
     34066 ± 10%     -36.5%      21638 ±  9%  softirqs.CPU76.SCHED
    114420 ±  6%     -38.0%      70971 ± 16%  softirqs.CPU76.TIMER
    164331 ± 17%     -49.3%      83344 ± 21%  softirqs.CPU77.RCU
     35590 ±  6%     -37.7%      22183 ± 19%  softirqs.CPU77.SCHED
    115677 ±  5%     -37.7%      72104 ± 18%  softirqs.CPU77.TIMER
    163560 ±  9%     -45.6%      88946 ± 19%  softirqs.CPU78.RCU
     34801 ±  3%     -34.5%      22778 ± 11%  softirqs.CPU78.SCHED
    115047 ±  5%     -38.0%      71276 ± 17%  softirqs.CPU78.TIMER
    159881 ± 12%     -43.7%      90013 ± 31%  softirqs.CPU79.RCU
     35304 ±  4%     -39.3%      21433 ±  3%  softirqs.CPU79.SCHED
    115204 ±  5%     -31.5%      78899 ± 16%  softirqs.CPU79.TIMER
    210242 ± 13%     -43.9%     117973 ± 11%  softirqs.CPU8.RCU
    106027 ±  2%     -32.4%      71712 ± 16%  softirqs.CPU8.TIMER
    180983 ± 15%     -45.7%      98246 ± 28%  softirqs.CPU80.RCU
     33559 ± 11%     -31.7%      22937 ± 12%  softirqs.CPU80.SCHED
    115467 ±  6%     -36.6%      73223 ± 15%  softirqs.CPU80.TIMER
    160786 ± 12%     -34.3%     105703 ± 25%  softirqs.CPU81.RCU
     35548 ±  4%     -38.7%      21781 ± 15%  softirqs.CPU81.SCHED
    114599 ±  4%     -37.9%      71160 ± 16%  softirqs.CPU81.TIMER
    179450 ±  4%     -42.0%     104152 ± 22%  softirqs.CPU82.RCU
     34590 ±  3%     -35.0%      22477 ± 11%  softirqs.CPU82.SCHED
    114826 ±  5%     -37.8%      71426 ± 16%  softirqs.CPU82.TIMER
    206850 ± 17%     -46.2%     111184 ± 21%  softirqs.CPU83.RCU
     33012 ± 10%     -36.2%      21078 ±  7%  softirqs.CPU83.SCHED
    114597 ±  6%     -38.1%      70946 ± 15%  softirqs.CPU83.TIMER
    183710 ±  6%     -52.5%      87228 ± 17%  softirqs.CPU84.RCU
     34383 ±  3%     -32.6%      23176 ± 14%  softirqs.CPU84.SCHED
    114928 ±  5%     -37.6%      71766 ± 16%  softirqs.CPU84.TIMER
    180742 ±  8%     -38.1%     111800 ± 24%  softirqs.CPU85.RCU
     32710 ±  8%     -34.2%      21531 ± 19%  softirqs.CPU85.SCHED
    113945 ±  5%     -37.0%      71777 ± 15%  softirqs.CPU85.TIMER
    178796 ± 20%     -42.9%     102078 ± 17%  softirqs.CPU86.RCU
     33621 ±  8%     -32.2%      22783 ± 15%  softirqs.CPU86.SCHED
    114472 ±  5%     -37.6%      71427 ± 15%  softirqs.CPU86.TIMER
    178562 ± 14%     -47.4%      94008 ± 19%  softirqs.CPU87.RCU
     32490 ±  9%     -33.9%      21477 ± 19%  softirqs.CPU87.SCHED
    114179 ±  5%     -37.7%      71175 ± 16%  softirqs.CPU87.TIMER
    168756 ± 12%     -31.8%     115059 ±  5%  softirqs.CPU88.RCU
     33314 ±  4%     -46.0%      17999 ± 20%  softirqs.CPU88.SCHED
    114296 ±  4%     -39.2%      69541 ± 16%  softirqs.CPU88.TIMER
    192885 ± 22%     -55.1%      86619 ± 15%  softirqs.CPU89.RCU
     31908 ±  4%     -30.3%      22242 ± 27%  softirqs.CPU89.SCHED
    114849 ±  5%     -37.6%      71640 ± 18%  softirqs.CPU89.TIMER
    204084 ± 10%     -41.6%     119170 ± 25%  softirqs.CPU9.RCU
    107007 ±  2%     -33.4%      71240 ± 17%  softirqs.CPU9.TIMER
    190289 ± 15%     -52.8%      89868 ± 25%  softirqs.CPU90.RCU
     32345 ±  2%     -39.0%      19718 ± 20%  softirqs.CPU90.SCHED
    115244 ±  4%     -39.0%      70320 ± 16%  softirqs.CPU90.TIMER
    196141 ± 10%     -42.6%     112613 ±  7%  softirqs.CPU91.RCU
     29384 ±  7%     -27.1%      21427 ±  9%  softirqs.CPU91.SCHED
    112670 ±  4%     -36.4%      71678 ± 16%  softirqs.CPU91.TIMER
    184292 ± 10%     -39.1%     112279 ± 35%  softirqs.CPU92.RCU
     31214 ±  3%     -34.1%      20567 ±  6%  softirqs.CPU92.SCHED
    113204 ±  4%     -37.0%      71267 ± 14%  softirqs.CPU92.TIMER
    210474 ± 21%     -45.4%     114953 ± 14%  softirqs.CPU93.RCU
     28455 ± 13%     -28.4%      20374 ± 10%  softirqs.CPU93.SCHED
    196417 ± 26%     -40.2%     117464 ± 22%  softirqs.CPU94.RCU
     28497 ±  9%     -39.9%      17130 ± 22%  softirqs.CPU94.SCHED
    112317 ±  5%     -38.0%      69687 ± 14%  softirqs.CPU94.TIMER
    186184 ± 15%     -42.5%     107091 ± 15%  softirqs.CPU95.RCU
     27770 ±  6%     -47.1%      14680 ± 12%  softirqs.CPU95.SCHED
    111714 ±  5%     -38.8%      68316 ± 14%  softirqs.CPU95.TIMER
  18112386 ±  9%     -42.3%   10453189 ± 14%  softirqs.RCU
   2711764           -33.1%    1814615 ± 14%  softirqs.SCHED
  10718372 ±  2%     -36.0%    6856989 ± 16%  softirqs.TIMER
    284.00 ±  4%     -36.4%     180.50 ± 13%  interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
      2.25 ±173%   +3066.7%      71.25 ±147%  interrupts.47:PCI-MSI.31981580-edge.i40e-eth0-TxRx-11
    638.50 ±  3%     -38.8%     390.50 ± 14%  interrupts.9:IO-APIC.9-fasteoi.acpi
 7.124e+08           -48.0%  3.701e+08 ± 26%  interrupts.CAL:Function_call_interrupts
   8169983 ±  9%     -53.8%    3770951 ± 28%  interrupts.CPU0.CAL:Function_call_interrupts
    637975 ±  3%     -38.9%     389699 ± 14%  interrupts.CPU0.LOC:Local_timer_interrupts
     21365 ±  9%     -67.7%       6911 ± 39%  interrupts.CPU0.RES:Rescheduling_interrupts
  24980480 ± 19%     -39.2%   15189222 ± 26%  interrupts.CPU0.TLB:TLB_shootdowns
    638.50 ±  3%     -38.8%     390.50 ± 14%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
   9447014 ±  5%     -63.9%    3415062 ± 18%  interrupts.CPU1.CAL:Function_call_interrupts
    637934 ±  3%     -38.9%     390077 ± 14%  interrupts.CPU1.LOC:Local_timer_interrupts
     23938 ± 10%     -78.8%       5067 ± 36%  interrupts.CPU1.RES:Rescheduling_interrupts
  23547639 ±  7%     -50.5%   11659613 ± 28%  interrupts.CPU1.TLB:TLB_shootdowns
  10647877 ±  8%     -56.0%    4683255 ± 44%  interrupts.CPU10.CAL:Function_call_interrupts
    637903 ±  3%     -38.9%     389709 ± 14%  interrupts.CPU10.LOC:Local_timer_interrupts
     24924 ± 17%     -75.0%       6241 ± 43%  interrupts.CPU10.RES:Rescheduling_interrupts
      2.00 ±173%   +3437.5%      70.75 ±148%  interrupts.CPU11.47:PCI-MSI.31981580-edge.i40e-eth0-TxRx-11
  11658580 ±  7%     -61.2%    4526780 ± 92%  interrupts.CPU11.CAL:Function_call_interrupts
    637805 ±  3%     -38.9%     389422 ± 14%  interrupts.CPU11.LOC:Local_timer_interrupts
     28774 ±  5%     -78.2%       6267 ± 40%  interrupts.CPU11.RES:Rescheduling_interrupts
  29106228 ± 11%     -59.0%   11939484 ± 70%  interrupts.CPU11.TLB:TLB_shootdowns
    637889 ±  3%     -38.9%     389956 ± 14%  interrupts.CPU12.LOC:Local_timer_interrupts
     23513 ± 23%     -69.8%       7089 ± 19%  interrupts.CPU12.RES:Rescheduling_interrupts
  10102074 ± 13%     -66.2%    3414811 ± 20%  interrupts.CPU13.CAL:Function_call_interrupts
    637853 ±  3%     -38.9%     389741 ± 14%  interrupts.CPU13.LOC:Local_timer_interrupts
     22508 ± 11%     -75.1%       5599 ± 25%  interrupts.CPU13.RES:Rescheduling_interrupts
  21384091 ± 12%     -47.4%   11240316 ± 22%  interrupts.CPU13.TLB:TLB_shootdowns
    637901 ±  3%     -38.9%     389846 ± 14%  interrupts.CPU14.LOC:Local_timer_interrupts
     24017 ± 16%     -78.7%       5126 ± 35%  interrupts.CPU14.RES:Rescheduling_interrupts
    637907 ±  3%     -38.9%     389832 ± 14%  interrupts.CPU15.LOC:Local_timer_interrupts
     21811 ±  5%     -77.4%       4922 ± 23%  interrupts.CPU15.RES:Rescheduling_interrupts
  10058392 ± 13%     -64.3%    3592843 ± 55%  interrupts.CPU16.CAL:Function_call_interrupts
    637878 ±  3%     -38.9%     389767 ± 14%  interrupts.CPU16.LOC:Local_timer_interrupts
     22763 ± 11%     -68.7%       7124 ± 42%  interrupts.CPU16.RES:Rescheduling_interrupts
  21967903 ± 17%     -47.8%   11468109 ± 50%  interrupts.CPU16.TLB:TLB_shootdowns
   9911585 ± 12%     -63.8%    3584350 ± 36%  interrupts.CPU17.CAL:Function_call_interrupts
    637852 ±  3%     -38.9%     389925 ± 14%  interrupts.CPU17.LOC:Local_timer_interrupts
     23595 ±  7%     -76.9%       5456 ± 39%  interrupts.CPU17.RES:Rescheduling_interrupts
  22652651 ± 13%     -53.6%   10511554 ± 25%  interrupts.CPU17.TLB:TLB_shootdowns
  10360609 ± 11%     -68.0%    3311352 ± 27%  interrupts.CPU18.CAL:Function_call_interrupts
    637915 ±  3%     -38.9%     389797 ± 14%  interrupts.CPU18.LOC:Local_timer_interrupts
     24849 ±  8%     -81.2%       4678 ± 23%  interrupts.CPU18.RES:Rescheduling_interrupts
  24608934 ± 11%     -61.6%    9449713 ± 33%  interrupts.CPU18.TLB:TLB_shootdowns
   9587439 ± 19%     -56.6%    4165607 ± 25%  interrupts.CPU19.CAL:Function_call_interrupts
    637934 ±  3%     -38.9%     389771 ± 14%  interrupts.CPU19.LOC:Local_timer_interrupts
     23349 ± 12%     -81.5%       4308 ± 14%  interrupts.CPU19.RES:Rescheduling_interrupts
  22028074 ± 14%     -49.2%   11184770 ± 37%  interrupts.CPU19.TLB:TLB_shootdowns
   9471372 ±  5%     -68.1%    3025687 ± 31%  interrupts.CPU2.CAL:Function_call_interrupts
    637900 ±  3%     -38.9%     390003 ± 14%  interrupts.CPU2.LOC:Local_timer_interrupts
     20597 ± 11%     -55.4%       9181 ± 75%  interrupts.CPU2.RES:Rescheduling_interrupts
  21364538 ± 18%     -49.7%   10754161 ± 23%  interrupts.CPU2.TLB:TLB_shootdowns
    637776 ±  3%     -38.9%     389792 ± 14%  interrupts.CPU20.LOC:Local_timer_interrupts
     21649 ± 17%     -72.4%       5968 ± 30%  interrupts.CPU20.RES:Rescheduling_interrupts
   9499313 ± 18%     -61.5%    3656157 ± 31%  interrupts.CPU21.CAL:Function_call_interrupts
    637883 ±  3%     -38.9%     389772 ± 14%  interrupts.CPU21.LOC:Local_timer_interrupts
     22499 ± 11%     -79.4%       4627 ± 38%  interrupts.CPU21.RES:Rescheduling_interrupts
  22344314 ± 20%     -53.8%   10322892 ± 37%  interrupts.CPU21.TLB:TLB_shootdowns
  10317764 ±  6%     -40.0%    6186455 ± 34%  interrupts.CPU22.CAL:Function_call_interrupts
    637854 ±  3%     -38.9%     389548 ± 14%  interrupts.CPU22.LOC:Local_timer_interrupts
     25775 ± 34%     -70.9%       7493 ± 37%  interrupts.CPU22.RES:Rescheduling_interrupts
   9374974 ± 18%     -46.1%    5057417 ± 19%  interrupts.CPU23.CAL:Function_call_interrupts
    637906 ±  3%     -39.0%     389404 ± 14%  interrupts.CPU23.LOC:Local_timer_interrupts
     27218 ± 17%     -77.6%       6102 ± 40%  interrupts.CPU23.RES:Rescheduling_interrupts
   8938604 ±  4%     -36.0%    5717014 ± 27%  interrupts.CPU24.CAL:Function_call_interrupts
    637962 ±  3%     -39.1%     388797 ± 15%  interrupts.CPU24.LOC:Local_timer_interrupts
     19242 ±  6%     -62.3%       7245 ± 14%  interrupts.CPU24.RES:Rescheduling_interrupts
  21160588 ± 20%     -31.3%   14547808 ± 31%  interrupts.CPU24.TLB:TLB_shootdowns
    637852 ±  3%     -39.0%     388782 ± 15%  interrupts.CPU25.LOC:Local_timer_interrupts
     17130 ±  8%     -65.3%       5948 ± 42%  interrupts.CPU25.RES:Rescheduling_interrupts
    637935 ±  3%     -39.0%     389021 ± 15%  interrupts.CPU26.LOC:Local_timer_interrupts
     17954 ± 10%     -68.1%       5735 ± 41%  interrupts.CPU26.RES:Rescheduling_interrupts
   8569522 ±  3%     -43.5%    4844421 ± 36%  interrupts.CPU27.CAL:Function_call_interrupts
    637877 ±  3%     -39.1%     388722 ± 15%  interrupts.CPU27.LOC:Local_timer_interrupts
      6475 ± 19%     -45.6%       3524 ± 22%  interrupts.CPU27.NMI:Non-maskable_interrupts
      6475 ± 19%     -45.6%       3524 ± 22%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     21452 ±  9%     -69.6%       6514 ± 44%  interrupts.CPU27.RES:Rescheduling_interrupts
  26296248 ± 20%     -45.1%   14447598 ± 28%  interrupts.CPU27.TLB:TLB_shootdowns
    637883 ±  3%     -39.0%     389126 ± 15%  interrupts.CPU28.LOC:Local_timer_interrupts
     17998 ± 10%     -69.2%       5545 ± 31%  interrupts.CPU28.RES:Rescheduling_interrupts
   8171100 ±  9%     -33.1%    5468641 ± 20%  interrupts.CPU29.CAL:Function_call_interrupts
    638053 ±  3%     -39.0%     388930 ± 15%  interrupts.CPU29.LOC:Local_timer_interrupts
     18752 ± 19%     -68.9%       5825 ± 48%  interrupts.CPU29.RES:Rescheduling_interrupts
  10749762 ± 10%     -67.0%    3547991 ± 39%  interrupts.CPU3.CAL:Function_call_interrupts
    637901 ±  3%     -38.9%     389896 ± 14%  interrupts.CPU3.LOC:Local_timer_interrupts
     22155 ± 13%     -78.4%       4796 ± 32%  interrupts.CPU3.RES:Rescheduling_interrupts
  24983653 ± 38%     -48.1%   12978346 ± 48%  interrupts.CPU3.TLB:TLB_shootdowns
   8366331 ± 22%     -47.6%    4386429 ± 10%  interrupts.CPU30.CAL:Function_call_interrupts
    637811 ±  3%     -39.0%     388946 ± 15%  interrupts.CPU30.LOC:Local_timer_interrupts
     21553 ± 56%     -69.5%       6582 ± 44%  interrupts.CPU30.RES:Rescheduling_interrupts
   7714889 ± 16%     -51.0%    3782993 ± 18%  interrupts.CPU31.CAL:Function_call_interrupts
    637849 ±  3%     -39.0%     388886 ± 15%  interrupts.CPU31.LOC:Local_timer_interrupts
     18277 ± 35%     -69.5%       5574 ± 50%  interrupts.CPU31.RES:Rescheduling_interrupts
   8416993 ±  5%     -51.8%    4059331 ± 55%  interrupts.CPU32.CAL:Function_call_interrupts
    637894 ±  3%     -39.0%     389194 ± 14%  interrupts.CPU32.LOC:Local_timer_interrupts
     20568 ± 13%     -77.1%       4700 ± 25%  interrupts.CPU32.RES:Rescheduling_interrupts
   8104306 ±  8%     -48.7%    4156841 ± 17%  interrupts.CPU33.CAL:Function_call_interrupts
    637906 ±  3%     -39.0%     389090 ± 15%  interrupts.CPU33.LOC:Local_timer_interrupts
     19702 ± 10%     -72.5%       5426 ± 61%  interrupts.CPU33.RES:Rescheduling_interrupts
  23321751 ± 15%     -40.3%   13911745 ± 11%  interrupts.CPU33.TLB:TLB_shootdowns
   8313250 ± 24%     -41.9%    4828735 ± 26%  interrupts.CPU34.CAL:Function_call_interrupts
    637817 ±  3%     -39.0%     388882 ± 15%  interrupts.CPU34.LOC:Local_timer_interrupts
     19633 ± 44%     -73.3%       5246 ± 53%  interrupts.CPU34.RES:Rescheduling_interrupts
    637914 ±  3%     -39.0%     388814 ± 15%  interrupts.CPU35.LOC:Local_timer_interrupts
     15387 ±  8%     -59.2%       6285 ± 48%  interrupts.CPU35.RES:Rescheduling_interrupts
    637816 ±  3%     -39.0%     388905 ± 15%  interrupts.CPU36.LOC:Local_timer_interrupts
   7760339 ± 10%     -45.1%    4259526 ± 51%  interrupts.CPU37.CAL:Function_call_interrupts
    637860 ±  3%     -39.0%     389247 ± 14%  interrupts.CPU37.LOC:Local_timer_interrupts
     16924 ± 17%     -75.5%       4150 ± 33%  interrupts.CPU37.RES:Rescheduling_interrupts
   7667373 ± 13%     -45.9%    4146802 ± 16%  interrupts.CPU38.CAL:Function_call_interrupts
    637887 ±  3%     -39.0%     389150 ± 15%  interrupts.CPU38.LOC:Local_timer_interrupts
     19031 ± 13%     -67.1%       6261 ± 35%  interrupts.CPU38.RES:Rescheduling_interrupts
    637835 ±  3%     -39.0%     389095 ± 15%  interrupts.CPU39.LOC:Local_timer_interrupts
     20281 ±  9%     -68.6%       6374 ± 49%  interrupts.CPU39.RES:Rescheduling_interrupts
  10703172 ±  6%     -58.8%    4410201 ± 37%  interrupts.CPU4.CAL:Function_call_interrupts
    637748 ±  3%     -38.8%     389989 ± 14%  interrupts.CPU4.LOC:Local_timer_interrupts
      5153 ± 26%     -47.9%       2683 ± 20%  interrupts.CPU4.NMI:Non-maskable_interrupts
      5153 ± 26%     -47.9%       2683 ± 20%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
     23312 ±  7%     -85.6%       3363 ± 40%  interrupts.CPU4.RES:Rescheduling_interrupts
  24501161 ±  5%     -47.4%   12882196 ± 50%  interrupts.CPU4.TLB:TLB_shootdowns
   7169437 ± 27%     -50.8%    3526172 ± 39%  interrupts.CPU40.CAL:Function_call_interrupts
    637793 ±  3%     -39.0%     388987 ± 15%  interrupts.CPU40.LOC:Local_timer_interrupts
     17800 ± 30%     -62.1%       6745 ± 43%  interrupts.CPU40.RES:Rescheduling_interrupts
    637814 ±  3%     -39.0%     388813 ± 15%  interrupts.CPU41.LOC:Local_timer_interrupts
     15700 ± 21%     -51.6%       7592 ± 50%  interrupts.CPU41.RES:Rescheduling_interrupts
    637928 ±  3%     -39.0%     389131 ± 14%  interrupts.CPU42.LOC:Local_timer_interrupts
     15916 ± 13%     -64.6%       5629 ± 52%  interrupts.CPU42.RES:Rescheduling_interrupts
    637927 ±  3%     -39.0%     388910 ± 15%  interrupts.CPU43.LOC:Local_timer_interrupts
   6158983 ± 27%     -33.0%    4129216 ± 37%  interrupts.CPU44.CAL:Function_call_interrupts
    637907 ±  3%     -39.0%     389086 ± 15%  interrupts.CPU44.LOC:Local_timer_interrupts
     14754 ± 39%     -64.2%       5278 ± 40%  interrupts.CPU44.RES:Rescheduling_interrupts
    637844 ±  3%     -39.0%     389142 ± 14%  interrupts.CPU45.LOC:Local_timer_interrupts
     10802 ± 28%     -51.2%       5268 ± 38%  interrupts.CPU45.RES:Rescheduling_interrupts
   4358901 ± 14%     -52.0%    2090950 ± 61%  interrupts.CPU46.CAL:Function_call_interrupts
    637921 ±  3%     -39.0%     389255 ± 14%  interrupts.CPU46.LOC:Local_timer_interrupts
     10504 ±  8%     -67.9%       3375 ± 61%  interrupts.CPU46.RES:Rescheduling_interrupts
  11462671 ± 22%     -38.2%    7078767 ± 57%  interrupts.CPU46.TLB:TLB_shootdowns
   5420818 ± 26%     -46.8%    2883334 ± 20%  interrupts.CPU47.CAL:Function_call_interrupts
    637892 ±  3%     -39.0%     389019 ± 15%  interrupts.CPU47.LOC:Local_timer_interrupts
      2490 ± 77%     +90.4%       4741 ± 29%  interrupts.CPU47.NMI:Non-maskable_interrupts
      2490 ± 77%     +90.4%       4741 ± 29%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
     12823 ± 38%     -68.4%       4047 ± 26%  interrupts.CPU47.RES:Rescheduling_interrupts
   8265453 ±  4%     -66.1%    2802212 ± 27%  interrupts.CPU48.CAL:Function_call_interrupts
    637836 ±  3%     -38.9%     389973 ± 14%  interrupts.CPU48.LOC:Local_timer_interrupts
      4508 ± 14%     -55.1%       2024 ± 32%  interrupts.CPU48.NMI:Non-maskable_interrupts
      4508 ± 14%     -55.1%       2024 ± 32%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
     17013 ± 12%     -72.1%       4741 ± 22%  interrupts.CPU48.RES:Rescheduling_interrupts
  15184273 ± 27%     -55.2%    6799418 ± 31%  interrupts.CPU48.TLB:TLB_shootdowns
    637909 ±  3%     -38.9%     390053 ± 14%  interrupts.CPU49.LOC:Local_timer_interrupts
     15613 ± 20%     -69.0%       4842 ± 52%  interrupts.CPU49.RES:Rescheduling_interrupts
  10497685 ± 23%     +46.2%   15346410 ± 24%  interrupts.CPU49.TLB:TLB_shootdowns
  10283470 ±  7%     -65.6%    3542279 ± 30%  interrupts.CPU5.CAL:Function_call_interrupts
    637961 ±  3%     -38.9%     389782 ± 14%  interrupts.CPU5.LOC:Local_timer_interrupts
     19979 ± 13%     -75.4%       4920 ± 17%  interrupts.CPU5.RES:Rescheduling_interrupts
  20176707 ± 12%     -50.9%    9916016 ± 28%  interrupts.CPU5.TLB:TLB_shootdowns
   6638558 ± 11%     -41.9%    3855726 ± 38%  interrupts.CPU50.CAL:Function_call_interrupts
    637938 ±  3%     -38.9%     389969 ± 14%  interrupts.CPU50.LOC:Local_timer_interrupts
     14686 ±  6%     -59.8%       5908 ± 27%  interrupts.CPU50.RES:Rescheduling_interrupts
   6836495 ± 16%     -56.7%    2957822 ± 53%  interrupts.CPU51.CAL:Function_call_interrupts
    637824 ±  3%     -38.9%     389881 ± 14%  interrupts.CPU51.LOC:Local_timer_interrupts
     16953 ± 32%     -71.1%       4895 ± 43%  interrupts.CPU51.RES:Rescheduling_interrupts
   6786596 ± 13%     -46.7%    3620362 ± 45%  interrupts.CPU52.CAL:Function_call_interrupts
    637930 ±  3%     -38.9%     389987 ± 14%  interrupts.CPU52.LOC:Local_timer_interrupts
     14219 ± 14%     -61.7%       5451 ± 28%  interrupts.CPU52.RES:Rescheduling_interrupts
   7752343 ± 12%     -64.5%    2748321 ± 48%  interrupts.CPU53.CAL:Function_call_interrupts
    637864 ±  3%     -38.9%     389936 ± 14%  interrupts.CPU53.LOC:Local_timer_interrupts
     19111 ± 17%     -78.8%       4047 ± 42%  interrupts.CPU53.RES:Rescheduling_interrupts
  17921732 ± 33%     -53.5%    8332043 ± 59%  interrupts.CPU53.TLB:TLB_shootdowns
    637879 ±  3%     -38.9%     390002 ± 14%  interrupts.CPU54.LOC:Local_timer_interrupts
     15877 ± 32%     -63.4%       5806 ± 15%  interrupts.CPU54.RES:Rescheduling_interrupts
    637909 ±  3%     -38.9%     389868 ± 14%  interrupts.CPU55.LOC:Local_timer_interrupts
     12799 ± 13%     -44.2%       7139 ± 21%  interrupts.CPU55.RES:Rescheduling_interrupts
  11058244 ± 20%     +43.3%   15849121 ± 16%  interrupts.CPU55.TLB:TLB_shootdowns
    637877 ±  3%     -38.9%     389762 ± 14%  interrupts.CPU56.LOC:Local_timer_interrupts
     13480 ± 12%     -66.9%       4466 ± 32%  interrupts.CPU56.RES:Rescheduling_interrupts
   6808721 ± 26%     -66.3%    2293654 ± 72%  interrupts.CPU57.CAL:Function_call_interrupts
    637896 ±  3%     -38.9%     389838 ± 14%  interrupts.CPU57.LOC:Local_timer_interrupts
     14380 ± 31%     -81.2%       2709 ± 58%  interrupts.CPU57.RES:Rescheduling_interrupts
   6662619 ±  8%     -66.4%    2239026 ± 65%  interrupts.CPU58.CAL:Function_call_interrupts
    637882 ±  3%     -38.9%     389842 ± 14%  interrupts.CPU58.LOC:Local_timer_interrupts
     14654 ± 11%     -65.1%       5108 ± 48%  interrupts.CPU58.RES:Rescheduling_interrupts
   6429290 ± 21%     -49.8%    3224896 ± 40%  interrupts.CPU59.CAL:Function_call_interrupts
    637923 ±  3%     -38.9%     390049 ± 14%  interrupts.CPU59.LOC:Local_timer_interrupts
     14619 ± 10%     -73.9%       3814 ± 53%  interrupts.CPU59.RES:Rescheduling_interrupts
  10625668 ±  5%     -66.7%    3537720 ± 48%  interrupts.CPU6.CAL:Function_call_interrupts
    637929 ±  3%     -38.9%     389757 ± 14%  interrupts.CPU6.LOC:Local_timer_interrupts
     21955 ± 12%     -76.4%       5181 ± 23%  interrupts.CPU6.RES:Rescheduling_interrupts
  22839599 ± 17%     -48.9%   11681589 ± 55%  interrupts.CPU6.TLB:TLB_shootdowns
   6267146 ± 13%     -56.6%    2716997 ± 52%  interrupts.CPU60.CAL:Function_call_interrupts
    637851 ±  3%     -38.9%     389731 ± 14%  interrupts.CPU60.LOC:Local_timer_interrupts
     14228 ± 26%     -69.0%       4405 ± 62%  interrupts.CPU60.RES:Rescheduling_interrupts
   7107324 ± 15%     -52.1%    3407103 ± 31%  interrupts.CPU61.CAL:Function_call_interrupts
    637925 ±  3%     -38.9%     389888 ± 14%  interrupts.CPU61.LOC:Local_timer_interrupts
     16367 ± 15%     -75.7%       3972 ± 26%  interrupts.CPU61.RES:Rescheduling_interrupts
  16738187 ± 25%     -57.6%    7096024 ± 45%  interrupts.CPU61.TLB:TLB_shootdowns
   7123646 ± 31%     -64.4%    2538337 ± 54%  interrupts.CPU62.CAL:Function_call_interrupts
    637922 ±  3%     -38.8%     390147 ± 14%  interrupts.CPU62.LOC:Local_timer_interrupts
     15310 ± 36%     -82.3%       2709 ± 41%  interrupts.CPU62.RES:Rescheduling_interrupts
   7391688 ±  9%     -73.7%    1942340 ± 16%  interrupts.CPU63.CAL:Function_call_interrupts
    637823 ±  3%     -38.9%     389938 ± 14%  interrupts.CPU63.LOC:Local_timer_interrupts
     17021 ±  6%     -86.2%       2345 ± 31%  interrupts.CPU63.RES:Rescheduling_interrupts
  18610246 ± 17%     -72.8%    5070168 ±  9%  interrupts.CPU63.TLB:TLB_shootdowns
    637997 ±  3%     -38.9%     390134 ± 14%  interrupts.CPU64.LOC:Local_timer_interrupts
     14437 ± 21%     -79.9%       2897 ± 41%  interrupts.CPU64.RES:Rescheduling_interrupts
    637939 ±  3%     -38.9%     389675 ± 14%  interrupts.CPU65.LOC:Local_timer_interrupts
     15830 ± 14%     -70.3%       4697 ± 13%  interrupts.CPU65.RES:Rescheduling_interrupts
    637910 ±  3%     -38.9%     389768 ± 14%  interrupts.CPU66.LOC:Local_timer_interrupts
     13657 ± 17%     -62.0%       5189 ± 54%  interrupts.CPU66.RES:Rescheduling_interrupts
    637907 ±  3%     -38.9%     389908 ± 14%  interrupts.CPU67.LOC:Local_timer_interrupts
     14687 ± 15%     -69.5%       4484 ± 32%  interrupts.CPU67.RES:Rescheduling_interrupts
   6914239 ± 30%     -65.6%    2376526 ± 61%  interrupts.CPU68.CAL:Function_call_interrupts
    637996 ±  3%     -38.9%     390018 ± 14%  interrupts.CPU68.LOC:Local_timer_interrupts
     15131 ± 32%     -89.4%       1610 ± 27%  interrupts.CPU68.RES:Rescheduling_interrupts
  16863308 ± 39%     -67.9%    5406426 ± 65%  interrupts.CPU68.TLB:TLB_shootdowns
    637891 ±  3%     -38.9%     389910 ± 14%  interrupts.CPU69.LOC:Local_timer_interrupts
     12418 ± 16%     -62.3%       4680 ± 65%  interrupts.CPU69.RES:Rescheduling_interrupts
    284.00 ±  4%     -36.4%     180.50 ± 13%  interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
  11353679 ±  8%     -79.6%    2314768 ± 32%  interrupts.CPU7.CAL:Function_call_interrupts
    637950 ±  3%     -38.8%     390167 ± 14%  interrupts.CPU7.LOC:Local_timer_interrupts
     23814 ± 13%     -83.7%       3892 ± 21%  interrupts.CPU7.RES:Rescheduling_interrupts
  24858110 ±  5%     -70.0%    7454065 ± 30%  interrupts.CPU7.TLB:TLB_shootdowns
   6419455 ± 27%     -51.9%    3087164 ± 47%  interrupts.CPU70.CAL:Function_call_interrupts
    637903 ±  3%     -38.9%     390063 ± 14%  interrupts.CPU70.LOC:Local_timer_interrupts
     13579 ± 22%     -69.8%       4102 ± 42%  interrupts.CPU70.RES:Rescheduling_interrupts
    638265 ±  3%     -38.9%     389698 ± 14%  interrupts.CPU71.LOC:Local_timer_interrupts
     14980 ± 21%     -66.2%       5061 ± 38%  interrupts.CPU71.RES:Rescheduling_interrupts
   4793961 ± 28%     -71.3%    1374682 ± 57%  interrupts.CPU72.CAL:Function_call_interrupts
    637769 ±  3%     -39.0%     389156 ± 15%  interrupts.CPU72.LOC:Local_timer_interrupts
     11207 ± 35%     -76.1%       2676 ± 56%  interrupts.CPU72.RES:Rescheduling_interrupts
    637943 ±  3%     -39.0%     389148 ± 14%  interrupts.CPU73.LOC:Local_timer_interrupts
     11031 ± 39%     -59.9%       4423 ± 82%  interrupts.CPU73.RES:Rescheduling_interrupts
    637919 ±  3%     -39.0%     389136 ± 15%  interrupts.CPU74.LOC:Local_timer_interrupts
    637875 ±  3%     -39.0%     389301 ± 14%  interrupts.CPU75.LOC:Local_timer_interrupts
    637940 ±  3%     -39.0%     389325 ± 14%  interrupts.CPU76.LOC:Local_timer_interrupts
      4483 ± 32%     -55.0%       2016 ± 32%  interrupts.CPU76.NMI:Non-maskable_interrupts
      4483 ± 32%     -55.0%       2016 ± 32%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
    637897 ±  3%     -39.0%     389241 ± 15%  interrupts.CPU77.LOC:Local_timer_interrupts
      4264 ± 33%     -43.1%       2426 ± 32%  interrupts.CPU77.NMI:Non-maskable_interrupts
      4264 ± 33%     -43.1%       2426 ± 32%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
   3630271 ± 15%     -53.4%    1691413 ± 55%  interrupts.CPU78.CAL:Function_call_interrupts
    637962 ±  3%     -39.0%     389230 ± 15%  interrupts.CPU78.LOC:Local_timer_interrupts
      8092 ± 33%     -72.5%       2228 ± 45%  interrupts.CPU78.RES:Rescheduling_interrupts
    637915 ±  3%     -39.0%     389338 ± 14%  interrupts.CPU79.LOC:Local_timer_interrupts
      6396 ± 42%     -72.6%       1750 ± 58%  interrupts.CPU79.RES:Rescheduling_interrupts
  11592613 ±  5%     -66.6%    3870682 ± 34%  interrupts.CPU8.CAL:Function_call_interrupts
    637894 ±  3%     -38.9%     390029 ± 14%  interrupts.CPU8.LOC:Local_timer_interrupts
     25352 ± 17%     -75.2%       6285 ± 42%  interrupts.CPU8.RES:Rescheduling_interrupts
  26649782 ± 24%     -54.0%   12266189 ± 14%  interrupts.CPU8.TLB:TLB_shootdowns
   4853990 ± 46%     -63.8%    1756269 ± 75%  interrupts.CPU80.CAL:Function_call_interrupts
    637901 ±  3%     -39.0%     389372 ± 14%  interrupts.CPU80.LOC:Local_timer_interrupts
      3173 ± 30%     -40.7%       1881 ± 40%  interrupts.CPU80.NMI:Non-maskable_interrupts
      3173 ± 30%     -40.7%       1881 ± 40%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      9229 ± 36%     -71.8%       2598 ± 74%  interrupts.CPU80.RES:Rescheduling_interrupts
    637905 ±  3%     -39.0%     388886 ± 15%  interrupts.CPU81.LOC:Local_timer_interrupts
   4222246 ± 10%     -42.1%    2444361 ± 34%  interrupts.CPU82.CAL:Function_call_interrupts
    637969 ±  3%     -39.0%     389233 ± 14%  interrupts.CPU82.LOC:Local_timer_interrupts
      9616 ± 17%     -64.8%       3388 ± 41%  interrupts.CPU82.RES:Rescheduling_interrupts
   6866891 ± 42%     -51.9%    3305419 ± 53%  interrupts.CPU83.CAL:Function_call_interrupts
    637953 ±  3%     -39.0%     389293 ± 14%  interrupts.CPU83.LOC:Local_timer_interrupts
     15587 ± 44%     -76.6%       3651 ± 40%  interrupts.CPU83.RES:Rescheduling_interrupts
  20441565 ± 45%     -49.3%   10372609 ± 39%  interrupts.CPU83.TLB:TLB_shootdowns
   4991019 ± 12%     -85.1%     743731 ± 94%  interrupts.CPU84.CAL:Function_call_interrupts
    637796 ±  3%     -39.0%     389309 ± 14%  interrupts.CPU84.LOC:Local_timer_interrupts
     10840 ± 18%     -85.6%       1557 ± 51%  interrupts.CPU84.RES:Rescheduling_interrupts
  14202654 ± 16%     -87.3%    1809946 ± 89%  interrupts.CPU84.TLB:TLB_shootdowns
   5508116 ± 23%     -60.7%    2166712 ± 76%  interrupts.CPU85.CAL:Function_call_interrupts
    637875 ±  3%     -39.0%     388910 ± 15%  interrupts.CPU85.LOC:Local_timer_interrupts
      9470 ± 15%     -57.2%       4052 ± 79%  interrupts.CPU85.RES:Rescheduling_interrupts
   4808316 ± 47%     -63.3%    1764313 ± 16%  interrupts.CPU86.CAL:Function_call_interrupts
    637782 ±  3%     -39.0%     389192 ± 15%  interrupts.CPU86.LOC:Local_timer_interrupts
      9035 ± 48%     -64.6%       3200 ± 42%  interrupts.CPU86.RES:Rescheduling_interrupts
   5448474 ± 34%     -59.9%    2186729 ± 14%  interrupts.CPU87.CAL:Function_call_interrupts
    637890 ±  3%     -39.0%     389198 ± 14%  interrupts.CPU87.LOC:Local_timer_interrupts
      9616 ± 24%     -79.6%       1966 ± 52%  interrupts.CPU87.RES:Rescheduling_interrupts
    637938 ±  3%     -39.0%     389053 ± 15%  interrupts.CPU88.LOC:Local_timer_interrupts
   5369050 ± 19%     -66.4%    1805316 ± 50%  interrupts.CPU89.CAL:Function_call_interrupts
    637823 ±  3%     -39.0%     389054 ± 15%  interrupts.CPU89.LOC:Local_timer_interrupts
     14252 ± 41%     -91.7%       1182 ± 68%  interrupts.CPU89.RES:Rescheduling_interrupts
  13176068 ± 20%     -65.3%    4569922 ± 45%  interrupts.CPU89.TLB:TLB_shootdowns
  11266939 ±  5%     -49.8%    5651301 ± 24%  interrupts.CPU9.CAL:Function_call_interrupts
    637921 ±  3%     -38.8%     390101 ± 14%  interrupts.CPU9.LOC:Local_timer_interrupts
     24283 ±  8%     -77.8%       5397 ± 44%  interrupts.CPU9.RES:Rescheduling_interrupts
   5939797 ±  8%     -54.4%    2707473 ± 41%  interrupts.CPU90.CAL:Function_call_interrupts
    637899 ±  3%     -39.0%     389159 ± 15%  interrupts.CPU90.LOC:Local_timer_interrupts
     11899 ± 21%     -83.9%       1911 ± 62%  interrupts.CPU90.RES:Rescheduling_interrupts
  15090475 ± 33%     -61.6%    5794511 ± 52%  interrupts.CPU90.TLB:TLB_shootdowns
   8155314 ± 17%     -60.1%    3251959 ± 21%  interrupts.CPU91.CAL:Function_call_interrupts
    637870 ±  3%     -39.0%     388875 ± 15%  interrupts.CPU91.LOC:Local_timer_interrupts
     14756 ± 22%     -62.8%       5483 ± 16%  interrupts.CPU91.RES:Rescheduling_interrupts
  20111546 ± 19%     -49.6%   10140305 ± 12%  interrupts.CPU91.TLB:TLB_shootdowns
    637927 ±  3%     -39.0%     389279 ± 14%  interrupts.CPU92.LOC:Local_timer_interrupts
     11729 ± 23%     -73.8%       3078 ± 69%  interrupts.CPU92.RES:Rescheduling_interrupts
   9630596 ± 25%     -56.4%    4203365 ± 65%  interrupts.CPU93.CAL:Function_call_interrupts
    637822 ±  3%     -39.0%     388919 ± 15%  interrupts.CPU93.LOC:Local_timer_interrupts
     15942 ± 15%     -74.5%       4072 ± 50%  interrupts.CPU93.RES:Rescheduling_interrupts
  23292920 ± 40%     -50.9%   11437229 ± 42%  interrupts.CPU93.TLB:TLB_shootdowns
    637909 ±  3%     -39.0%     388953 ± 15%  interrupts.CPU94.LOC:Local_timer_interrupts
     15403 ± 38%     -71.1%       4445 ± 54%  interrupts.CPU94.RES:Rescheduling_interrupts
    638242 ±  3%     -39.0%     389258 ± 15%  interrupts.CPU95.LOC:Local_timer_interrupts
     14879 ± 15%     -78.7%       3163 ± 58%  interrupts.CPU95.RES:Rescheduling_interrupts
  61238300 ±  3%     -38.9%   37389604 ± 14%  interrupts.LOC:Local_timer_interrupts
   1601898 ±  8%     -71.6%     455396 ± 22%  interrupts.RES:Rescheduling_interrupts
 1.729e+09 ± 19%     -36.3%  1.101e+09 ± 16%  interrupts.TLB:TLB_shootdowns


                                                                                
                               vm-scalability.throughput                        
                                                                                
    2e+06 +-----------------------------------------------------------------+   
  1.8e+06 |-O O  O O   O               O       O          O                 |   
          |              O                 O        O   O      O   O        |   
  1.6e+06 |-+               O   O O O        O        O          O          |   
  1.4e+06 |-+                                                               |   
          |                                                  O              |   
  1.2e+06 |-+                                                               |   
    1e+06 |-+                                                             + |   
   800000 |-+                                                             : |   
          |                                                              : :|   
   600000 |-+                                                            : :|   
   400000 |-+                                                            : :|   
          |       .+.                   .+.        .+.        .+.       :   |   
   200000 |.+.+..+   +.+.+..+.+.+. .+..+   +.+.+..+   +.+.+..+   +.+.+..+   |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                   vm-scalability.time.voluntary_context_switches               
                                                                                
  250000 +------------------------------------------------------------------+   
         |                                                                  |   
         |               +.                                                 |   
  200000 |-+     .+.+. ..  +.+.     +             .+..        .+.+          |   
         |.+.  .+     +        +    :+ .+..+.+. .+    +.+. .+.    + .+..    |   
         |   +.                :    : +        +          +        +    +  :|   
  150000 |-+                    :  :                                     + :|   
         |                      :  :                                      + |   
  100000 |-+                    :  :                                        |   
         |                       : :                                        |   
         |                       : :                                        |   
   50000 |-+                     : :                                        |   
         |                       ::                         O  O   O        |   
         |   O             O   O  : O        O        O O O      O          |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                   vm-scalability.time.involuntary_context_switches             
                                                                                
  2.5e+06 +-----------------------------------------------------------------+   
          |                                                                 |   
          |                                                               + |   
    2e+06 |-+                                                            : :|   
          |       .+.+.+.+..+.+.    +.. .+.        .+.        .+.+.      : :|   
          |.+.+..+              +   :  +   +.+.+..+   +.+.+..+     +.+..+   |   
  1.5e+06 |-+                   :   :                                       |   
          |                     :  :                                        |   
    1e+06 |-+                    : :                                        |   
          |                      : :                                        |   
          |                      : :                         O O   O        |   
   500000 |-+                    : :                                        |   
          |                       : O        O        O          O          |   
          | O O      O   O  O   O :    O   O   O    O   O O                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc1-00017-gdcf33bfdfbb73" of type "text/plain" (203610 bytes)

View attachment "job-script" of type "text/plain" (8385 bytes)

View attachment "job.yaml" of type "text/plain" (6014 bytes)

View attachment "reproduce" of type "text/plain" (968 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ