lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Wed, 26 Jun 2019 17:00:33 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Waiman Long <longman@...hat.com>
Cc:     Ingo Molnar <mingo@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Borislav Petkov <bp@...en8.de>,
        Davidlohr Bueso <dave@...olabs.net>,
        "H. Peter Anvin" <hpa@...or.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        Will Deacon <will.deacon@....com>,
        huang ying <huang.ying.caritas@...il.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Stephen Rothwell <sfr@...b.auug.org.au>, lkp@...org
Subject: [locking/rwsem] cf69482d62: will-it-scale.per_thread_ops 320.6%
 improvement

Greeting,

FYI, we noticed a 320.6% improvement of will-it-scale.per_thread_ops due to commit:


commit: cf69482d62d996d3ce840eeead8e160de281ac6c ("locking/rwsem: Enable readers spinning on writer")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

in testcase: will-it-scale
on test machine: 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory
with following parameters:

	nr_task: 16
	mode: thread
	test: malloc1
	cpufreq_governor: performance
	ucode: 0x21

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -30.0% regression         |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
| test parameters  | cpufreq_governor=performance                                          |
|                  | mode=thread                                                           |
|                  | nr_task=100%                                                          |
|                  | test=page_fault1                                                      |
|                  | ucode=0xb000036                                                       |
+------------------+-----------------------------------------------------------------------+


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/thread/16/debian-x86_64-2019-05-14.cgz/lkp-ivb-d02/malloc1/will-it-scale/0x21

commit: 
  02f1082b00 ("locking/rwsem: Clarify usage of owner's nonspinaable bit")
  cf69482d62 ("locking/rwsem: Enable readers spinning on writer")

02f1082b003a0cd4 cf69482d62d996d3ce840eeead8 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           21%           0:4     perf-profile.children.cycles-pp.error_entry
           :4           17%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    540.75 ± 14%    +320.6%       2274 ±  8%  will-it-scale.per_thread_ops
     77681 ±  9%     -67.0%      25634 ±  8%  will-it-scale.time.involuntary_context_switches
     34940 ±  6%     -25.0%      26222 ±  8%  will-it-scale.time.maximum_resident_set_size
   2662775 ± 14%    +323.2%   11270087 ±  7%  will-it-scale.time.minor_page_faults
    245.25 ±  5%     -57.4%     104.50        will-it-scale.time.percent_of_cpu_this_job_got
    731.58 ±  5%     -58.8%     301.07        will-it-scale.time.system_time
      8.34 ±  7%     +83.7%      15.32 ±  8%  will-it-scale.time.user_time
   7938402 ± 14%     -71.9%    2232334 ± 15%  will-it-scale.time.voluntary_context_switches
      8659 ± 14%    +320.3%      36396 ±  8%  will-it-scale.workload
   2113923           -13.6%    1825776        meminfo.Committed_AS
      3860 ±  3%     +16.7%       4506 ±  4%  slabinfo.anon_vma_chain.num_objs
     21.50 ±  4%      -5.5%      20.33        boot-time.boot
     57.75 ±  7%      -7.0%      53.71        boot-time.idle
     37.86 ±  9%     +34.5       72.32        mpstat.cpu.all.idle%
     60.62 ±  5%     -35.8       24.81        mpstat.cpu.all.sys%
      1.51 ±  4%      +1.4        2.87 ± 29%  mpstat.cpu.all.usr%
     37.75 ±  9%     +90.1%      71.75        vmstat.cpu.id
     59.75 ±  6%     -59.8%      24.00        vmstat.cpu.sy
      3.50 ± 14%     -71.4%       1.00        vmstat.procs.r
     50563 ± 13%     -58.1%      21165 ± 10%  vmstat.system.cs
     22262 ±  5%     -42.4%      12825 ±  2%  vmstat.system.in
 2.723e+08 ± 21%     -67.2%   89383372 ± 61%  cpuidle.C1.time
   6070652 ± 27%     -61.4%    2340407 ± 17%  cpuidle.C1.usage
 1.456e+08 ± 13%     -44.2%   81233642 ± 30%  cpuidle.C1E.time
   3354004 ±  9%     -57.1%    1439564 ± 25%  cpuidle.C1E.usage
   2308955 ± 19%  +23020.6%  5.338e+08 ± 18%  cpuidle.C6.time
      5316 ± 30%  +13259.2%     710208 ±  5%  cpuidle.C6.usage
    135976 ± 21%    +189.9%     394242 ± 29%  cpuidle.POLL.time
     35743 ± 30%    +988.4%     389044 ± 28%  cpuidle.POLL.usage
     17617 ±  3%     -27.2%      12819 ± 18%  softirqs.CPU0.RCU
     70344 ±  3%     -45.0%      38673 ±  5%  softirqs.CPU0.SCHED
     71957 ±  4%     -44.3%      40057 ±  4%  softirqs.CPU1.SCHED
     17017 ± 17%     -31.0%      11744 ± 18%  softirqs.CPU2.RCU
     68398 ±  2%     -42.2%      39557 ± 11%  softirqs.CPU2.SCHED
     66835 ±  2%     -41.4%      39184 ±  6%  softirqs.CPU3.SCHED
     69723 ± 10%     -30.0%      48807 ±  2%  softirqs.RCU
    277536 ±  2%     -43.3%     157472 ±  4%  softirqs.SCHED
      3054            -2.4%       2982        proc-vmstat.nr_inactive_anon
      3994            -3.2%       3865        proc-vmstat.nr_shmem
      3054            -2.4%       2982        proc-vmstat.nr_zone_inactive_anon
   6870896 ±  7%    +113.5%   14669432 ±  6%  proc-vmstat.numa_hit
   6870896 ±  7%    +113.5%   14669432 ±  6%  proc-vmstat.numa_local
 8.458e+08           -94.3%   48343337 ± 11%  proc-vmstat.pgalloc_normal
   3018558 ± 12%    +284.7%   11611836 ±  7%  proc-vmstat.pgfault
 8.458e+08           -94.3%   48328690 ± 11%  proc-vmstat.pgfree
   1641722           -96.0%      65789 ± 16%  proc-vmstat.thp_deferred_split_page
   1641722           -96.0%      65791 ± 16%  proc-vmstat.thp_fault_alloc
   1641714           -96.0%      65789 ± 16%  proc-vmstat.thp_split_pmd
      2105 ±  5%     -55.1%     945.25        turbostat.Avg_MHz
     63.96 ±  5%     -35.3       28.70        turbostat.Busy%
   6070653 ± 27%     -61.4%    2340407 ± 17%  turbostat.C1
     22.52 ± 21%     -15.1        7.38 ± 61%  turbostat.C1%
   3354004 ±  9%     -57.1%    1439564 ± 25%  turbostat.C1E
     12.04 ± 13%      -5.3        6.71 ± 30%  turbostat.C1E%
      5317 ± 30%  +13256.0%     710206 ±  5%  turbostat.C6
      0.19 ± 19%     +43.9       44.12 ± 18%  turbostat.C6%
      0.03 ±117%  +52575.0%      15.80 ± 76%  turbostat.CPU%c6
     14.97           -26.8%      10.96        turbostat.CorWatt
  13099132 ±  6%     -53.5%    6085596 ±  4%  turbostat.IRQ
     33.09           -15.8%      27.86        turbostat.PkgWatt
   3654470 ±  8%     -84.2%     577068 ± 17%  interrupts.CAL:Function_call_interrupts
    901116 ±  8%     -84.2%     142084 ± 16%  interrupts.CPU0.CAL:Function_call_interrupts
     29273 ±  9%     -74.8%       7379 ± 13%  interrupts.CPU0.RES:Rescheduling_interrupts
   1548017 ±  8%     -64.2%     553950 ± 10%  interrupts.CPU0.TLB:TLB_shootdowns
    911455 ±  9%     -84.0%     145392 ± 17%  interrupts.CPU1.CAL:Function_call_interrupts
    742262           +13.8%     844441        interrupts.CPU1.LOC:Local_timer_interrupts
     27853 ±  7%     -67.2%       9125 ± 12%  interrupts.CPU1.RES:Rescheduling_interrupts
   1582495 ±  7%     -66.8%     525320 ± 14%  interrupts.CPU1.TLB:TLB_shootdowns
    921310 ±  8%     -84.4%     144002 ± 17%  interrupts.CPU2.CAL:Function_call_interrupts
     31090 ± 12%     -73.1%       8368 ±  4%  interrupts.CPU2.RES:Rescheduling_interrupts
   1613141 ±  6%     -65.7%     553887 ± 10%  interrupts.CPU2.TLB:TLB_shootdowns
    920589 ±  8%     -84.2%     145590 ± 18%  interrupts.CPU3.CAL:Function_call_interrupts
     29934 ± 14%     -69.9%       9001 ± 14%  interrupts.CPU3.RES:Rescheduling_interrupts
   1588926 ±  6%     -65.1%     554506 ± 13%  interrupts.CPU3.TLB:TLB_shootdowns
   2967656           +10.2%    3270931        interrupts.LOC:Local_timer_interrupts
     12259 ± 21%     -38.1%       7591 ± 16%  interrupts.NMI:Non-maskable_interrupts
     12259 ± 21%     -38.1%       7591 ± 16%  interrupts.PMI:Performance_monitoring_interrupts
    118152 ± 10%     -71.3%      33874 ±  9%  interrupts.RES:Rescheduling_interrupts
   6332581 ±  7%     -65.5%    2187664 ± 11%  interrupts.TLB:TLB_shootdowns
     99.22 ±  2%     -94.5%       5.48 ± 12%  perf-stat.i.MPKI
 4.109e+08 ±  2%    +150.3%  1.029e+09 ±  3%  perf-stat.i.branch-instructions
      0.83 ±  6%      -0.4        0.42 ±  2%  perf-stat.i.branch-miss-rate%
   4819449 ± 12%     -28.8%    3431956 ± 17%  perf-stat.i.branch-misses
     85.96 ±  2%     -42.9       43.07 ±  8%  perf-stat.i.cache-miss-rate%
  1.71e+08 ±  2%     -96.4%    6108851 ± 15%  perf-stat.i.cache-misses
 1.985e+08           -93.0%   13851689 ± 10%  perf-stat.i.cache-references
     50880 ± 13%     -58.1%      21307 ± 10%  perf-stat.i.context-switches
      4.17 ±  7%     -72.3%       1.15        perf-stat.i.cpi
 8.402e+09 ±  5%     -53.8%  3.881e+09 ±  3%  perf-stat.i.cpu-cycles
      3873 ± 29%     -82.5%     677.79 ±  8%  perf-stat.i.cpu-migrations
     51.93 ±  3%   +1717.7%     943.97 ± 16%  perf-stat.i.cycles-between-cache-misses
   2575127 ± 10%     +14.5%    2947751 ±  6%  perf-stat.i.dTLB-load-misses
  5.65e+08 ±  2%     +49.1%  8.423e+08 ±  3%  perf-stat.i.dTLB-loads
      0.03 ± 10%      +0.1        0.18        perf-stat.i.dTLB-store-miss-rate%
    314295 ± 10%     +61.5%     507687 ±  5%  perf-stat.i.dTLB-store-misses
  9.93e+08           -70.5%  2.932e+08 ±  6%  perf-stat.i.dTLB-stores
     87.71            +5.2       92.94        perf-stat.i.iTLB-load-miss-rate%
    796301 ± 15%     +72.5%    1373983 ± 12%  perf-stat.i.iTLB-load-misses
 2.104e+09 ±  2%     +67.2%  3.518e+09 ±  3%  perf-stat.i.instructions
      2764 ± 12%     +25.5%       3468 ± 10%  perf-stat.i.instructions-per-iTLB-miss
      0.25 ±  7%    +250.9%       0.88        perf-stat.i.ipc
      9943 ± 12%    +286.5%      38435 ±  7%  perf-stat.i.minor-faults
      9943 ± 12%    +286.5%      38435 ±  7%  perf-stat.i.page-faults
     94.40 ±  2%     -95.8%       3.93 ±  7%  perf-stat.overall.MPKI
      1.17 ± 11%      -0.8        0.33 ± 15%  perf-stat.overall.branch-miss-rate%
     86.11 ±  2%     -42.1       43.99 ±  8%  perf-stat.overall.cache-miss-rate%
      4.00 ±  7%     -72.4%       1.10        perf-stat.overall.cpi
     49.11 ±  3%   +1221.2%     648.88 ± 14%  perf-stat.overall.cycles-between-cache-misses
      0.45 ±  8%      -0.1        0.35 ±  3%  perf-stat.overall.dTLB-load-miss-rate%
      0.03 ± 10%      +0.1        0.17        perf-stat.overall.dTLB-store-miss-rate%
     87.60            +5.7       93.26        perf-stat.overall.iTLB-load-miss-rate%
      0.25 ±  7%    +260.6%       0.91        perf-stat.overall.ipc
  74688511 ± 12%     -60.8%   29275233 ±  6%  perf-stat.overall.path-length
 4.105e+08 ±  2%    +149.8%  1.026e+09 ±  3%  perf-stat.ps.branch-instructions
   4814236 ± 12%     -28.9%    3421392 ± 17%  perf-stat.ps.branch-misses
 1.708e+08 ±  2%     -96.4%    6089754 ± 15%  perf-stat.ps.cache-misses
 1.983e+08           -93.0%   13808812 ± 10%  perf-stat.ps.cache-references
     50831 ± 13%     -58.2%      21241 ± 10%  perf-stat.ps.context-switches
 8.393e+09 ±  5%     -53.9%  3.869e+09 ±  3%  perf-stat.ps.cpu-cycles
      3870 ± 29%     -82.5%     675.70 ±  8%  perf-stat.ps.cpu-migrations
   2572531 ± 10%     +14.2%    2938686 ±  6%  perf-stat.ps.dTLB-load-misses
 5.644e+08 ±  2%     +48.8%  8.397e+08 ±  3%  perf-stat.ps.dTLB-loads
    313988 ± 10%     +61.2%     506119 ±  5%  perf-stat.ps.dTLB-store-misses
  9.92e+08           -70.5%  2.923e+08 ±  6%  perf-stat.ps.dTLB-stores
    795526 ± 15%     +72.2%    1369681 ± 12%  perf-stat.ps.iTLB-load-misses
 2.102e+09 ±  2%     +66.8%  3.507e+09 ±  3%  perf-stat.ps.instructions
      9933 ± 12%    +285.7%      38316 ±  7%  perf-stat.ps.minor-faults
      9933 ± 12%    +285.7%      38316 ±  7%  perf-stat.ps.page-faults
 6.352e+11 ±  2%     +66.9%   1.06e+12 ±  3%  perf-stat.total.instructions
     94012 ±  5%     -54.9%      42425        sched_debug.cfs_rq:/.exec_clock.avg
     95408 ±  5%     -47.0%      50575 ±  3%  sched_debug.cfs_rq:/.exec_clock.max
     92992 ±  4%     -68.8%      29026 ±  5%  sched_debug.cfs_rq:/.exec_clock.min
    921.18 ± 32%    +784.1%       8144 ± 13%  sched_debug.cfs_rq:/.exec_clock.stddev
    194225           +55.3%     301719 ± 11%  sched_debug.cfs_rq:/.load.avg
    407986 ±  5%     +21.8%     496836 ±  9%  sched_debug.cfs_rq:/.load.max
    149217 ±  7%     +28.0%     190986 ±  7%  sched_debug.cfs_rq:/.load.stddev
    134.08 ± 12%     -63.9%      48.38 ± 34%  sched_debug.cfs_rq:/.load_avg.min
    319132 ±  6%     -72.1%      89024        sched_debug.cfs_rq:/.min_vruntime.avg
    331241 ±  6%     -67.8%     106558 ±  5%  sched_debug.cfs_rq:/.min_vruntime.max
    308234 ±  6%     -77.8%      68383 ±  4%  sched_debug.cfs_rq:/.min_vruntime.min
      9447 ± 27%     +52.9%      14446 ± 21%  sched_debug.cfs_rq:/.min_vruntime.stddev
      8.49 ± 13%     -86.5%       1.15 ± 23%  sched_debug.cfs_rq:/.nr_spread_over.avg
     21.08 ± 24%     -85.8%       3.00 ± 41%  sched_debug.cfs_rq:/.nr_spread_over.max
      8.41 ± 31%     -85.4%       1.23 ± 41%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    125.09 ±  6%     +52.2%     190.42 ± 20%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    312.13 ±  4%     +30.9%     408.67 ± 17%  sched_debug.cfs_rq:/.runnable_load_avg.max
    121.18 ±  3%     +47.8%     179.13 ± 22%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
    144799 ±  6%     +84.4%     266949 ± 14%  sched_debug.cfs_rq:/.runnable_weight.avg
    324346 ± 10%     +39.0%     450911 ± 13%  sched_debug.cfs_rq:/.runnable_weight.max
    123682 ± 11%     +44.5%     178733 ± 11%  sched_debug.cfs_rq:/.runnable_weight.stddev
    928.46 ±135%   +1301.2%      13009 ± 51%  sched_debug.cfs_rq:/.spread0.max
      9447 ± 27%     +52.9%      14445 ± 21%  sched_debug.cfs_rq:/.spread0.stddev
    746.19 ±  4%     -33.9%     493.45 ±  4%  sched_debug.cfs_rq:/.util_avg.avg
      1149 ±  6%     -24.0%     873.79 ± 14%  sched_debug.cfs_rq:/.util_avg.max
    442.42 ± 10%     -56.7%     191.50 ± 19%  sched_debug.cfs_rq:/.util_avg.min
     76.07 ± 16%     -61.1%      29.59 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    249.17 ± 26%     -59.0%     102.25 ± 14%  sched_debug.cfs_rq:/.util_est_enqueued.max
    105.53 ± 28%     -59.3%      42.94 ± 14%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    188241 ±  4%    +208.9%     581518 ±  9%  sched_debug.cpu.avg_idle.avg
    258246 ±  7%    +201.8%     779409 ± 10%  sched_debug.cpu.avg_idle.max
    108040 ± 40%    +237.2%     364364 ± 12%  sched_debug.cpu.avg_idle.min
     58945 ± 38%    +186.9%     169095 ± 13%  sched_debug.cpu.avg_idle.stddev
      0.41 ± 19%     +38.6%       0.57 ± 25%  sched_debug.cpu.clock.stddev
      0.41 ± 19%     +38.6%       0.57 ± 25%  sched_debug.cpu.clock_task.stddev
    293.67 ± 23%     +40.9%     413.67 ± 29%  sched_debug.cpu.cpu_load[1].max
     58.75 ± 15%     -81.2%      11.04 ± 37%  sched_debug.cpu.cpu_load[1].min
     91.28 ± 28%     +80.3%     164.60 ± 31%  sched_debug.cpu.cpu_load[1].stddev
    245.21 ± 18%     +79.6%     440.46 ± 42%  sched_debug.cpu.cpu_load[2].max
     86.92 ± 12%     -83.3%      14.50 ± 28%  sched_debug.cpu.cpu_load[2].min
     61.67 ± 28%    +178.4%     171.66 ± 44%  sched_debug.cpu.cpu_load[2].stddev
    212.29 ± 14%    +125.4%     478.42 ± 45%  sched_debug.cpu.cpu_load[3].max
    106.38 ±  7%     -83.6%      17.42 ± 18%  sched_debug.cpu.cpu_load[3].min
     41.45 ± 29%    +357.3%     189.55 ± 48%  sched_debug.cpu.cpu_load[3].stddev
    190.08 ± 12%    +170.4%     513.92 ± 42%  sched_debug.cpu.cpu_load[4].max
    118.42 ±  5%     -83.2%      19.88 ± 16%  sched_debug.cpu.cpu_load[4].min
     28.45 ± 31%    +623.0%     205.68 ± 46%  sched_debug.cpu.cpu_load[4].stddev
    189.04 ± 12%     -63.1%      69.75 ± 34%  sched_debug.cpu.curr->pid.min
    208714 ±  9%     +53.5%     320473 ± 17%  sched_debug.cpu.load.avg
    437217 ±  8%     +27.7%     558205 ± 20%  sched_debug.cpu.load.max
     53276 ± 10%     -57.2%      22784 ± 14%  sched_debug.cpu.load.min
    152859 ±  6%     +39.3%     212858 ± 17%  sched_debug.cpu.load.stddev
      1.41 ±  5%     -20.0%       1.12 ± 10%  sched_debug.cpu.nr_running.avg
      3.08 ± 12%     -37.8%       1.92 ±  7%  sched_debug.cpu.nr_running.max
      1.12 ± 10%     -39.7%       0.68 ±  5%  sched_debug.cpu.nr_running.stddev
   2021948 ± 12%     -54.4%     922042 ±  7%  sched_debug.cpu.nr_switches.avg
   2089375 ± 11%     -51.4%    1014927 ±  8%  sched_debug.cpu.nr_switches.max
   1919423 ± 14%     -56.8%     830088 ±  7%  sched_debug.cpu.nr_switches.min
     23.54 ± 35%    +207.1%      72.29 ± 41%  sched_debug.cpu.nr_uninterruptible.max
    -17.04          +269.7%     -63.00        sched_debug.cpu.nr_uninterruptible.min
     15.66 ± 47%    +247.9%      54.49 ± 40%  sched_debug.cpu.nr_uninterruptible.stddev
   2014090 ± 12%     -54.7%     913057 ±  8%  sched_debug.cpu.sched_count.avg
   2082014 ± 12%     -51.8%    1004171 ±  8%  sched_debug.cpu.sched_count.max
   1911982 ± 14%     -56.9%     823980 ±  7%  sched_debug.cpu.sched_count.min
    873592 ± 13%     -48.8%     447315 ±  8%  sched_debug.cpu.sched_goidle.avg
    895195 ± 12%     -44.9%     492904 ±  9%  sched_debug.cpu.sched_goidle.max
    836533 ± 14%     -51.9%     402740 ±  8%  sched_debug.cpu.sched_goidle.min
     23025 ± 24%     +63.5%      37640 ± 28%  sched_debug.cpu.sched_goidle.stddev
   1198367 ± 12%     -59.4%     486768 ±  8%  sched_debug.cpu.ttwu_count.avg
   1238486 ± 11%     -57.8%     523203 ± 12%  sched_debug.cpu.ttwu_count.max
   1139425 ± 15%     -60.9%     445191 ±  4%  sched_debug.cpu.ttwu_count.min
    213496 ±  5%     +12.9%     240978 ±  5%  sched_debug.cpu.ttwu_local.max
     61.34 ±  5%     -59.0        2.39 ± 25%  perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
     61.62 ±  4%     -57.6        4.02 ± 17%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     61.66 ±  4%     -57.5        4.16 ± 16%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     59.68 ±  5%     -57.4        2.25 ± 25%  perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
     57.80 ±  5%     -55.7        2.12 ± 24%  perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
     56.75 ±  5%     -54.7        2.05 ± 25%  perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault
      5.41 ±  4%      -2.2        3.22 ± 20%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap
      5.51 ±  4%      -2.1        3.41 ± 20%  perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      1.57 ±  6%      -1.2        0.34 ±100%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      1.58 ±  6%      -1.2        0.43 ±100%  perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.63 ±  3%      -0.7        0.96 ± 23%  perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.__do_munmap.__vm_munmap.__x64_sys_munmap
      1.28 ±  9%      -0.6        0.69 ± 12%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      0.17 ±173%      +0.5        0.71 ± 25%  perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      3.15 ±  5%      +0.8        3.92 ± 17%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
      3.15 ±  5%      +0.8        3.93 ± 17%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.mmap64
      0.00            +0.8        0.78 ± 12%  perf-profile.calltrace.cycles-pp.___might_sleep.unmap_page_range.unmap_vmas.unmap_region.__do_munmap
      0.00            +1.0        0.99 ± 50%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.00            +1.1        1.06 ± 47%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      3.27 ±  4%      +1.2        4.44 ± 17%  perf-profile.calltrace.cycles-pp.mmap64
      0.00            +1.4        1.39 ± 10%  perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.00            +1.5        1.52 ± 16%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      0.31 ±101%      +1.7        2.03 ± 13%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.43 ±  8%      +5.4       24.80 ± 14%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     11.36 ± 60%     +12.8       24.21 ± 16%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     11.39 ± 60%     +12.9       24.30 ± 16%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     12.32 ± 60%     +13.3       25.57 ± 16%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     12.33 ± 60%     +13.3       25.58 ± 16%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     12.33 ± 60%     +13.3       25.58 ± 16%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      0.00           +33.2       33.18 ± 15%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault
      0.00           +39.0       38.98 ± 16%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault
      0.00           +49.9       49.95 ± 15%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault
     61.35 ±  5%     -59.0        2.39 ± 25%  perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page
     61.64 ±  4%     -57.6        4.06 ± 17%  perf-profile.children.cycles-pp.__handle_mm_fault
     59.77 ±  5%     -57.5        2.27 ± 25%  perf-profile.children.cycles-pp.clear_huge_page
     61.69 ±  4%     -57.5        4.19 ± 16%  perf-profile.children.cycles-pp.handle_mm_fault
     57.97 ±  5%     -55.9        2.12 ± 24%  perf-profile.children.cycles-pp.clear_subpage
     57.00 ±  5%     -54.6        2.39 ± 20%  perf-profile.children.cycles-pp.clear_page_erms
      5.51 ±  4%      -2.2        3.34 ± 20%  perf-profile.children.cycles-pp.unmap_page_range
      5.52 ±  4%      -2.1        3.43 ± 19%  perf-profile.children.cycles-pp.unmap_vmas
      1.49 ±  5%      -1.4        0.12 ± 42%  perf-profile.children.cycles-pp.__split_huge_pmd_locked
      1.51 ±  5%      -1.3        0.20 ± 35%  perf-profile.children.cycles-pp.__split_huge_pmd
      1.54 ±  5%      -1.2        0.31 ± 31%  perf-profile.children.cycles-pp.vma_adjust_trans_huge
      2.23 ±  2%      -1.1        1.08 ± 36%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      2.26 ±  2%      -1.0        1.25 ± 36%  perf-profile.children.cycles-pp.down_write_killable
      1.09 ± 10%      -1.0        0.10 ± 61%  perf-profile.children.cycles-pp.deferred_split_huge_page
      1.14 ± 11%      -1.0        0.18 ±  7%  perf-profile.children.cycles-pp.page_remove_rmap
      1.49 ±  8%      -0.7        0.78 ± 22%  perf-profile.children.cycles-pp.__schedule
      1.30 ±  2%      -0.7        0.64 ± 21%  perf-profile.children.cycles-pp.get_page_from_freelist
      1.36 ±  2%      -0.6        0.79 ± 21%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.13 ± 11%      -0.6        0.57 ± 28%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.99 ±  7%      -0.5        0.45 ± 16%  perf-profile.children.cycles-pp.schedule
      0.91 ± 11%      -0.5        0.42 ± 28%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      1.30 ±  9%      -0.5        0.81 ± 11%  perf-profile.children.cycles-pp.release_pages
      1.78 ± 13%      -0.5        1.30 ± 20%  perf-profile.children.cycles-pp.try_to_wake_up
      1.23 ± 16%      -0.5        0.76 ± 28%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.23 ± 17%      -0.5        0.76 ± 28%  perf-profile.children.cycles-pp.activate_task
      1.78 ± 13%      -0.5        1.31 ± 20%  perf-profile.children.cycles-pp.wake_up_q
      1.21 ± 17%      -0.5        0.74 ± 27%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.44 ± 19%      -0.4        0.03 ±105%  perf-profile.children.cycles-pp.__page_mapcount
      1.11 ± 16%      -0.4        0.72 ± 26%  perf-profile.children.cycles-pp.enqueue_entity
      0.58 ± 17%      -0.4        0.20 ± 31%  perf-profile.children.cycles-pp.rcu_all_qs
      0.74 ± 13%      -0.4        0.39 ± 33%  perf-profile.children.cycles-pp._cond_resched
      1.74            -0.3        1.44 ± 20%  perf-profile.children.cycles-pp.__vma_adjust
      0.30 ± 17%      -0.2        0.08 ± 63%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.24 ±  7%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.__tlb_remove_page_size
      0.43 ±  7%      -0.2        0.23 ± 38%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.53 ± 17%      -0.2        0.32 ± 28%  perf-profile.children.cycles-pp.schedule_idle
      0.34 ± 14%      -0.2        0.17 ± 39%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.21 ± 26%      -0.2        0.06 ± 74%  perf-profile.children.cycles-pp.call_function_interrupt
      0.23 ± 31%      -0.2        0.07 ±100%  perf-profile.children.cycles-pp.smp_call_function_many
      0.32 ± 15%      -0.1        0.20 ± 38%  perf-profile.children.cycles-pp.dequeue_entity
      0.24 ± 15%      -0.1        0.11 ± 52%  perf-profile.children.cycles-pp.update_load_avg
      0.20 ± 20%      -0.1        0.08 ± 60%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.41 ± 21%      -0.1        0.29 ± 17%  perf-profile.children.cycles-pp.unwind_next_frame
      0.15 ± 31%      -0.1        0.03 ±105%  perf-profile.children.cycles-pp.smp_call_function_interrupt
      0.20 ± 22%      -0.1        0.10 ± 71%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.13 ±  6%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.13 ± 10%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.13 ± 33%      -0.1        0.04 ±110%  perf-profile.children.cycles-pp.__switch_to
      0.13 ± 11%      -0.1        0.07 ± 17%  perf-profile.children.cycles-pp.set_next_entity
      0.11 ± 16%      -0.1        0.05 ± 60%  perf-profile.children.cycles-pp.select_idle_sibling
      0.08 ± 26%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.16 ± 20%      -0.0        0.11 ± 11%  perf-profile.children.cycles-pp.update_curr
      0.11 ± 33%      -0.0        0.07 ± 21%  perf-profile.children.cycles-pp.update_cfs_group
      0.03 ±100%      +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.irq_enter
      0.07 ± 49%      +0.1        0.13 ± 28%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.01 ±173%      +0.1        0.08 ± 30%  perf-profile.children.cycles-pp.free_unref_page_commit
      0.01 ±173%      +0.1        0.08 ± 37%  perf-profile.children.cycles-pp.__count_memcg_events
      0.02 ±173%      +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.04 ±101%      +0.1        0.12 ± 19%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.remove_vma
      0.02 ±173%      +0.1        0.10 ± 20%  perf-profile.children.cycles-pp.vmacache_find
      0.15 ± 24%      +0.1        0.23 ± 13%  perf-profile.children.cycles-pp.malloc
      0.01 ±173%      +0.1        0.10 ± 19%  perf-profile.children.cycles-pp.__vma_link_rb
      0.06 ± 60%      +0.1        0.14 ± 27%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.1        0.09 ± 39%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.00            +0.1        0.09 ± 23%  perf-profile.children.cycles-pp.avc_has_perm_noaudit
      0.00            +0.1        0.09 ± 28%  perf-profile.children.cycles-pp.strlcpy
      0.14 ± 19%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.find_vma
      0.00            +0.1        0.10 ± 25%  perf-profile.children.cycles-pp.mem_cgroup_try_charge
      0.01 ±173%      +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.__vma_rb_erase
      0.01 ±173%      +0.1        0.12 ± 53%  perf-profile.children.cycles-pp.kmem_cache_free
      0.00            +0.1        0.11 ± 29%  perf-profile.children.cycles-pp.up_write
      0.00            +0.1        0.11 ± 24%  perf-profile.children.cycles-pp.mem_cgroup_throttle_swaprate
      0.02 ±173%      +0.1        0.13 ± 33%  perf-profile.children.cycles-pp.cpumask_any_but
      0.07 ±  7%      +0.1        0.18 ± 22%  perf-profile.children.cycles-pp.__perf_sw_event
      0.00            +0.1        0.12 ± 47%  perf-profile.children.cycles-pp.__might_sleep
      0.00            +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.00            +0.1        0.12 ± 24%  perf-profile.children.cycles-pp.uncharge_batch
      0.03 ±100%      +0.1        0.16 ± 29%  perf-profile.children.cycles-pp.vma_compute_subtree_gap
      0.09 ± 30%      +0.1        0.21 ± 33%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.05 ± 61%      +0.1        0.18 ± 30%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.00            +0.1        0.13 ± 11%  perf-profile.children.cycles-pp.cred_has_capability
      0.07 ± 17%      +0.1        0.21 ± 16%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.06 ± 14%      +0.2        0.21 ± 40%  perf-profile.children.cycles-pp.unmapped_area_topdown
      0.00            +0.2        0.15 ± 28%  perf-profile.children.cycles-pp.__rb_insert_augmented
      0.01 ±173%      +0.2        0.17 ± 64%  perf-profile.children.cycles-pp.__pte_alloc
      0.01 ±173%      +0.2        0.18 ± 30%  perf-profile.children.cycles-pp.anon_vma_interval_tree_remove
      0.05 ± 61%      +0.2        0.21 ± 25%  perf-profile.children.cycles-pp.wake_q_add
      0.00            +0.2        0.16 ± 19%  perf-profile.children.cycles-pp.selinux_vm_enough_memory
      0.00            +0.2        0.17 ± 18%  perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.04 ± 59%      +0.2        0.22 ± 27%  perf-profile.children.cycles-pp.vm_area_dup
      0.14 ± 24%      +0.2        0.32 ± 19%  perf-profile.children.cycles-pp.tick_sched_handle
      0.12 ± 27%      +0.2        0.30 ± 18%  perf-profile.children.cycles-pp.update_process_times
      0.03 ±100%      +0.2        0.22 ±  9%  perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay
      0.03 ±100%      +0.2        0.22 ± 34%  perf-profile.children.cycles-pp.down_write
      0.10 ± 33%      +0.2        0.30 ± 28%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.10 ± 22%      +0.2        0.30 ± 11%  perf-profile.children.cycles-pp.flush_tlb_func_common
      0.00            +0.2        0.20 ± 47%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.12 ± 19%      +0.2        0.33 ± 34%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.02 ±173%      +0.2        0.23 ± 23%  perf-profile.children.cycles-pp.security_vm_enough_memory_mm
      0.03 ±102%      +0.2        0.26 ± 20%  perf-profile.children.cycles-pp.free_unref_page_list
      0.06 ± 13%      +0.2        0.29 ± 33%  perf-profile.children.cycles-pp.anon_vma_clone
      0.09 ± 36%      +0.2        0.34 ± 34%  perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.07 ± 71%      +0.3        0.33 ± 16%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.20 ±  7%      +0.3        0.47 ± 38%  perf-profile.children.cycles-pp.irq_exit
      0.12 ± 27%      +0.3        0.41 ± 17%  perf-profile.children.cycles-pp.perf_event_mmap
      0.12 ± 38%      +0.3        0.42 ± 29%  perf-profile.children.cycles-pp.get_unmapped_area
      0.11 ± 19%      +0.3        0.42 ± 24%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.05 ± 62%      +0.3        0.38 ± 27%  perf-profile.children.cycles-pp.free_p4d_range
      0.16 ± 18%      +0.3        0.49 ± 31%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.06 ± 67%      +0.4        0.42 ± 23%  perf-profile.children.cycles-pp.free_pgd_range
      0.09 ± 16%      +0.4        0.48 ± 12%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.06 ± 70%      +0.4        0.47 ± 22%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.03 ±173%      +0.4        0.43 ± 18%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.03 ±173%      +0.4        0.46 ± 14%  perf-profile.children.cycles-pp.lru_add_drain
      0.13 ± 24%      +0.5        0.63 ± 13%  perf-profile.children.cycles-pp.vma_merge
      0.09 ± 35%      +0.6        0.64 ± 25%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.75 ± 15%      +0.8        1.56 ± 35%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.61 ± 14%      +0.8        1.45 ± 38%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.38 ± 15%      +1.2        1.54 ± 15%  perf-profile.children.cycles-pp.mmap_region
      3.28 ±  4%      +1.2        4.47 ± 17%  perf-profile.children.cycles-pp.mmap64
      0.16 ± 39%      +1.2        1.41 ± 10%  perf-profile.children.cycles-pp.do_anonymous_page
      0.53 ± 20%      +1.5        2.05 ± 13%  perf-profile.children.cycles-pp.do_mmap
     19.43 ±  8%      +5.4       24.80 ± 14%  perf-profile.children.cycles-pp.intel_idle
     20.14 ±  8%      +6.3       26.46 ± 15%  perf-profile.children.cycles-pp.cpuidle_enter_state
     20.14 ±  8%      +6.3       26.46 ± 15%  perf-profile.children.cycles-pp.cpuidle_enter
     12.33 ± 60%     +13.3       25.58 ± 16%  perf-profile.children.cycles-pp.start_secondary
      1.33 ±  9%     +21.3       22.61 ± 15%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.39 ± 10%     +49.6       49.95 ± 15%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      0.00           +50.0       49.96 ± 15%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
     56.39 ±  5%     -54.0        2.34 ± 21%  perf-profile.self.cycles-pp.clear_page_erms
      1.24 ±  4%      -1.2        0.07 ± 73%  perf-profile.self.cycles-pp.__split_huge_pmd_locked
      1.20 ± 10%      -1.2        0.04 ±102%  perf-profile.self.cycles-pp.clear_subpage
      0.93 ±  8%      -0.8        0.13 ± 10%  perf-profile.self.cycles-pp.page_remove_rmap
      0.85 ± 11%      -0.6        0.20 ± 21%  perf-profile.self.cycles-pp.release_pages
      0.80 ±  8%      -0.6        0.18 ± 24%  perf-profile.self.cycles-pp.get_page_from_freelist
      1.03 ± 12%      -0.5        0.53 ± 31%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.88 ± 10%      -0.5        0.39 ± 30%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.40 ± 16%      -0.4        0.03 ±100%  perf-profile.self.cycles-pp.__page_mapcount
      0.43 ± 13%      -0.3        0.15 ± 36%  perf-profile.self.cycles-pp.rcu_all_qs
      0.29 ± 17%      -0.3        0.04 ±102%  perf-profile.self.cycles-pp.prep_new_page
      0.28 ± 15%      -0.2        0.04 ±108%  perf-profile.self.cycles-pp.free_pages_and_swap_cache
      0.17 ± 34%      -0.1        0.04 ±100%  perf-profile.self.cycles-pp.smp_call_function_many
      0.13 ±  6%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.23 ± 29%      -0.1        0.15 ±  3%  perf-profile.self.cycles-pp.unwind_next_frame
      0.12 ± 32%      -0.1        0.04 ±108%  perf-profile.self.cycles-pp.__switch_to
      0.20 ±  4%      -0.1        0.12 ± 26%  perf-profile.self.cycles-pp.__schedule
      0.01 ±173%      +0.1        0.07 ± 23%  perf-profile.self.cycles-pp.handle_mm_fault
      0.01 ±173%      +0.1        0.07 ± 30%  perf-profile.self.cycles-pp.arch_get_unmapped_area_topdown
      0.06 ± 34%      +0.1        0.13 ± 31%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.00            +0.1        0.07 ± 19%  perf-profile.self.cycles-pp.vm_area_dup
      0.00            +0.1        0.07 ± 21%  perf-profile.self.cycles-pp.__perf_sw_event
      0.01 ±173%      +0.1        0.08 ± 37%  perf-profile.self.cycles-pp.__count_memcg_events
      0.00            +0.1        0.07 ± 31%  perf-profile.self.cycles-pp.flush_tlb_func_common
      0.00            +0.1        0.07 ± 39%  perf-profile.self.cycles-pp.cpumask_any_but
      0.01 ±173%      +0.1        0.09 ± 23%  perf-profile.self.cycles-pp.rwsem_mark_wake
      0.02 ±173%      +0.1        0.10 ± 20%  perf-profile.self.cycles-pp.vmacache_find
      0.04 ± 57%      +0.1        0.12 ± 18%  perf-profile.self.cycles-pp.perf_iterate_sb
      0.00            +0.1        0.08 ± 28%  perf-profile.self.cycles-pp.__vma_rb_erase
      0.00            +0.1        0.09 ± 21%  perf-profile.self.cycles-pp.avc_has_perm_noaudit
      0.00            +0.1        0.09 ± 15%  perf-profile.self.cycles-pp.down_write_killable
      0.12 ± 25%      +0.1        0.22 ± 26%  perf-profile.self.cycles-pp._raw_spin_lock
      0.00            +0.1        0.09 ± 37%  perf-profile.self.cycles-pp.mem_cgroup_throttle_swaprate
      0.07 ± 16%      +0.1        0.16 ± 30%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.00            +0.1        0.10 ± 45%  perf-profile.self.cycles-pp.__might_sleep
      0.00            +0.1        0.10 ± 49%  perf-profile.self.cycles-pp.mmap_region
      0.00            +0.1        0.10 ± 32%  perf-profile.self.cycles-pp.vma_merge
      0.01 ±173%      +0.1        0.12 ± 53%  perf-profile.self.cycles-pp.kmem_cache_free
      0.05 ± 67%      +0.1        0.15 ± 23%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.00            +0.1        0.11 ± 29%  perf-profile.self.cycles-pp.up_write
      0.00            +0.1        0.11 ± 39%  perf-profile.self.cycles-pp.perf_event_mmap
      0.00            +0.1        0.11 ± 33%  perf-profile.self.cycles-pp.free_unref_page_list
      0.00            +0.1        0.12 ± 32%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.03 ±102%      +0.1        0.15 ± 30%  perf-profile.self.cycles-pp.vma_compute_subtree_gap
      0.00            +0.1        0.12 ± 24%  perf-profile.self.cycles-pp.__rb_insert_augmented
      0.03 ±100%      +0.1        0.15 ± 27%  perf-profile.self.cycles-pp.__do_munmap
      0.01 ±173%      +0.1        0.15 ± 18%  perf-profile.self.cycles-pp.munmap
      0.00            +0.1        0.14 ± 16%  perf-profile.self.cycles-pp.anon_vma_interval_tree_remove
      0.06 ± 14%      +0.2        0.21 ± 40%  perf-profile.self.cycles-pp.unmapped_area_topdown
      0.02 ±173%      +0.2        0.17 ± 27%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.05 ± 62%      +0.2        0.21 ± 26%  perf-profile.self.cycles-pp.wake_q_add
      0.00            +0.2        0.17 ± 18%  perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.06 ± 14%      +0.2        0.31 ± 36%  perf-profile.self.cycles-pp.prepare_exit_to_usermode
      0.07 ± 71%      +0.3        0.32 ± 14%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.07 ± 60%      +0.3        0.35 ± 23%  perf-profile.self.cycles-pp.__vma_adjust
      0.04 ± 60%      +0.3        0.35 ± 29%  perf-profile.self.cycles-pp.free_p4d_range
      0.08 ± 17%      +0.3        0.41 ± 12%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.06 ± 70%      +0.4        0.47 ± 22%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.12 ± 26%      +0.6        0.76 ± 11%  perf-profile.self.cycles-pp.do_syscall_64
      1.25 ±  9%      +2.7        3.99 ± 14%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
     19.43 ±  8%      +5.4       24.78 ± 14%  perf-profile.self.cycles-pp.intel_idle
      0.00           +45.7       45.68 ± 15%  perf-profile.self.cycles-pp.rwsem_optimistic_spin


                                                                                
                           will-it-scale.per_thread_ops                         
                                                                                
  4000 +-+------------------------------------------------------------------+   
       |                                                                    |   
  3500 O-+          O                                        O              |   
  3000 +-+ O O        O                                    O     O          |   
       | O     O                    O                                       |   
  2500 +-+              O O     O O            O O O O O O              O   |   
       |                    O O       O   O  O                    O   O     |   
  2000 +-+       O O                    O  O                        O       |   
       |                                                       O            |   
  1500 +-+                                                                  |   
  1000 +-+                                                                  |   
       |.+.+.+.+.+.++. .+.+.+.+.+.+.+.+.+.++.+.+.+.+. .+.+.+.+.+.+ .+.      |   
   500 +-+            +                              +            +   +.+.+.|   
       |                                                                    |   
     0 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               will-it-scale.workload                           
                                                                                
  60000 +-+-----------------------------------------------------------------+   
  55000 O-+          O                                       O              |   
        |   O          O                                   O     O          |   
  50000 +-+   O                                                             |   
  45000 +-O     O                 O O                                       |   
  40000 +-+              O O   O O    O         O OO O O O              O   |   
  35000 +-+                  O            O   O                    O  O     |   
        |        O O                    O   O                       O       |   
  30000 +-+                                                    O            |   
  25000 +-+                                                                 |   
  20000 +-+                                                                 |   
  15000 +-+         .+                                                      |   
        |.+.+.+.++.+  + .+.+.+.+.++.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.++.      |   
  10000 +-+            +                                              +.+.+.|   
   5000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          will-it-scale.time.system_time                        
                                                                                
  800 +-+-------------------------------------------------------------------+   
  750 +-+                                                              + :  |   
      |                                                               +  : +|   
  700 +-+                                                             :   + |   
  650 +-+                                                            :      |   
  600 +-+             +                                              :      |   
  550 +-+            : +                          .+.+. .+. +.+. .+.+       |   
      |.+.+.+.+.+.   :  ++.+.+.+.+.+.+.+.+.+.+.+.+     +   +    +           |   
  500 +-+         +.+                                                       |   
  450 O-+                                                  OO               |   
  400 +-O O O O     O O  O O O O O O O         O O O O O O                  |   
  350 +-+       O O     O              O O O O                              |   
      |                                                                     |   
  300 +-+                                                     O O O O O O   |   
  250 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         will-it-scale.time.minor_page_faults                   
                                                                                
  1.8e+07 +-+------------------------------------------------O--------------+   
          O            O                                                    |   
  1.6e+07 +-+ O          O                                  O    O          |   
  1.4e+07 +-O   OO                                                          |   
          |                       O O O          O                          |   
  1.2e+07 +-+              OO   O      O          O O O O O             O   |   
          |                   O          O O O O                   O   O    |   
    1e+07 +-+      O O                                               O      |   
          |                                                    O            |   
    8e+06 +-+                                                               |   
    6e+06 +-+                                                               |   
          |                                                                 |   
    4e+06 +-+.+.++.+.+.+.   +.+.+.+.+.++.+.+.+.+.++.   .+. .+   .+.         |   
          |              +.+                        +.+   +  +.+   +.+.+ .+.|   
    2e+06 +-+---------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                   will-it-scale.time.involuntary_context_switches              
                                                                                
  90000 +-+-----------------------------------------------------------------+   
        |                                                                 +.|   
  80000 +-+                                                     .+       +  |   
  70000 +-+     ++.+      .+        +         +.              .+  :   +.+   |   
        |.+.+  :    +   .+  + .+.+ : + .+.+  :  +.+ .+. .+. .+    :  +      |   
  60000 +-+  + :     +.+     +    ::  +    + :     +   +   +       ++       |   
        |     +                   +         +                               |   
  50000 +-+                                                                 |   
        |                                                                   |   
  40000 +-+                                                                 |   
  30000 +-+          O                                       O   O          |   
        O   O O O      O            O                      O       O  O O   |   
  20000 +-O              O O   O OO   O   O   O O OO O O O     O    O       |   
        |        O O         O          O   O                               |   
  10000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-2019-05-14.cgz/lkp-bdw-ep6/page_fault1/will-it-scale/0xb000036

commit: 
  02f1082b00 ("locking/rwsem: Clarify usage of owner's nonspinaable bit")
  cf69482d62 ("locking/rwsem: Enable readers spinning on writer")

02f1082b003a0cd4 cf69482d62d996d3ce840eeead8 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    204977           -30.0%     143502 ±  7%  will-it-scale.per_thread_ops
     32636           -35.0%      21207 ±  5%  will-it-scale.time.involuntary_context_switches
  11204844            -2.8%   10887521 ±  2%  will-it-scale.time.maximum_resident_set_size
  10685365           -30.5%    7425106 ±  7%  will-it-scale.time.minor_page_faults
      5916           -31.4%       4057 ±  6%  will-it-scale.time.percent_of_cpu_this_job_got
     17429           -31.5%      11939 ±  6%  will-it-scale.time.system_time
    501.94           -16.7%     418.29 ± 11%  will-it-scale.time.user_time
   9936329           -29.8%    6973727 ±  7%  will-it-scale.time.voluntary_context_switches
  18038066           -30.0%   12628201 ±  7%  will-it-scale.workload
     64388           -23.7%      49118 ±  6%  meminfo.Shmem
     39457           -16.8%      32829 ±  5%  meminfo.max_used_kB
     32.88           +21.7       54.59 ±  5%  mpstat.cpu.all.idle%
     65.20           -21.4       43.84 ±  7%  mpstat.cpu.all.sys%
      1.92            -0.4        1.57 ± 10%  mpstat.cpu.all.usr%
  11015081           -29.5%    7765645 ±  7%  numa-numastat.node0.local_node
  11043451           -29.6%    7778949 ±  6%  numa-numastat.node0.numa_hit
  11089634           -29.8%    7781258 ±  6%  numa-numastat.node1.local_node
  11089781           -29.7%    7796463 ±  6%  numa-numastat.node1.numa_hit
     33.00           +64.6%      54.33 ±  6%  vmstat.cpu.id
     64.00           -32.8%      43.00 ±  7%  vmstat.cpu.sy
     56.50           -25.7%      42.00 ±  3%  vmstat.procs.r
     65986           -29.8%      46301 ±  7%  vmstat.system.cs
    196835            -3.0%     191012        vmstat.system.in
   5430899 ±  3%     -28.2%    3899060 ± 11%  cpuidle.C1.time
    445580 ±  2%     -26.5%     327415 ±  9%  cpuidle.C1.usage
   8057932 ± 52%     -39.3%    4890175 ± 76%  cpuidle.C1E.usage
 1.187e+09 ± 30%    +452.7%   6.56e+09 ± 53%  cpuidle.C6.time
    850532 ±  3%     -26.8%     622902 ± 15%  cpuidle.POLL.time
     65271            -8.1%      59984 ±  5%  cpuidle.POLL.usage
      2680 ± 10%     +34.1%       3593 ±  3%  slabinfo.eventpoll_pwq.active_objs
      2680 ± 10%     +34.1%       3593 ±  3%  slabinfo.eventpoll_pwq.num_objs
    608.00 ± 10%     -14.0%     522.67 ±  7%  slabinfo.kmalloc-rcl-128.active_objs
    608.00 ± 10%     -14.0%     522.67 ±  7%  slabinfo.kmalloc-rcl-128.num_objs
      3270           +19.1%       3895 ±  4%  slabinfo.kmalloc-rcl-64.active_objs
      3270           +19.1%       3895 ±  4%  slabinfo.kmalloc-rcl-64.num_objs
      9012            -9.4%       8164 ±  6%  numa-meminfo.node0.KernelStack
     16564 ±  7%     -15.1%      14068 ±  8%  numa-meminfo.node0.Mapped
     11019           -17.6%       9074 ±  6%  numa-meminfo.node0.PageTables
     74420           -12.0%      65479 ±  6%  numa-meminfo.node0.SUnreclaim
      6038           +14.3%       6898 ±  7%  numa-meminfo.node1.KernelStack
     52253           +19.3%      62336 ±  7%  numa-meminfo.node1.SUnreclaim
     45838           -22.6%      35482 ± 10%  numa-meminfo.node1.Shmem
     85110           +14.7%      97597 ±  9%  numa-meminfo.node1.Slab
      1885           -30.0%       1319 ±  6%  turbostat.Avg_MHz
     67.98           -20.0       48.01 ±  6%  turbostat.Busy%
    440173 ±  2%     -27.1%     320801 ±  9%  turbostat.C1
   8057051 ± 52%     -39.3%    4889012 ± 76%  turbostat.C1E
      4.41 ± 30%     +19.9       24.33 ± 53%  turbostat.C6%
     20.95 ±  2%     +39.6%      29.24 ±  9%  turbostat.CPU%c1
      1.48 ± 35%    +717.8%      12.10 ± 67%  turbostat.CPU%c6
      1.59          +436.5%       8.53 ±  9%  turbostat.Pkg%pc2
    223.90           -16.9%     186.09 ±  3%  turbostat.PkgWatt
     35.98           -22.7%      27.82 ±  4%  turbostat.RAMWatt
    139.00 ± 66%     -89.9%      14.00 ±141%  numa-vmstat.node0.nr_active_file
    199.00 ± 97%     -93.5%      13.00 ±141%  numa-vmstat.node0.nr_dirtied
     81.50 ± 36%     -66.5%      27.33 ±141%  numa-vmstat.node0.nr_inactive_file
      9011            -9.4%       8164 ±  6%  numa-vmstat.node0.nr_kernel_stack
      4141 ±  7%     -15.1%       3516 ±  8%  numa-vmstat.node0.nr_mapped
      2746           -17.4%       2269 ±  6%  numa-vmstat.node0.nr_page_table_pages
     18604           -12.0%      16369 ±  6%  numa-vmstat.node0.nr_slab_unreclaimable
    195.50 ± 97%     -94.0%      11.67 ±141%  numa-vmstat.node0.nr_written
    139.00 ± 66%     -89.9%      14.00 ±141%  numa-vmstat.node0.nr_zone_active_file
     81.50 ± 36%     -66.5%      27.33 ±141%  numa-vmstat.node0.nr_zone_inactive_file
   6056519           -30.4%    4214673 ±  4%  numa-vmstat.node0.numa_hit
   6028296           -30.3%    4201155 ±  5%  numa-vmstat.node0.numa_local
      6036           +14.3%       6899 ±  7%  numa-vmstat.node1.nr_kernel_stack
     11460           -22.6%       8870 ± 10%  numa-vmstat.node1.nr_shmem
     13063           +19.3%      15584 ±  7%  numa-vmstat.node1.nr_slab_unreclaimable
   5840019           -27.5%    4234784 ±  5%  numa-vmstat.node1.numa_hit
   5687311           -28.5%    4067110 ±  4%  numa-vmstat.node1.numa_local
   1526898            -1.5%    1503772        proc-vmstat.nr_active_anon
    139.00 ± 66%     -69.8%      42.00        proc-vmstat.nr_active_file
   1507855            -1.1%    1491414        proc-vmstat.nr_anon_pages
    278993            -1.4%     275053        proc-vmstat.nr_file_pages
      4817            -6.3%       4512        proc-vmstat.nr_inactive_anon
    101.00 ±  9%     -17.8%      83.00        proc-vmstat.nr_inactive_file
      7287            -5.3%       6901        proc-vmstat.nr_mapped
      5113            -8.6%       4672 ±  3%  proc-vmstat.nr_page_table_pages
     16103           -23.7%      12279 ±  6%  proc-vmstat.nr_shmem
     17586            -0.6%      17481        proc-vmstat.nr_slab_reclaimable
   1526894            -1.5%    1503769        proc-vmstat.nr_zone_active_anon
    139.00 ± 66%     -69.8%      42.00        proc-vmstat.nr_zone_active_file
      4817            -6.3%       4512        proc-vmstat.nr_zone_inactive_anon
    101.00 ±  9%     -17.8%      83.00        proc-vmstat.nr_zone_inactive_file
     15389 ±  7%     -63.8%       5576 ± 18%  proc-vmstat.numa_hint_faults
      8640 ± 21%     -73.6%       2283 ± 25%  proc-vmstat.numa_hint_faults_local
  22134500           -29.6%   15591508 ±  6%  proc-vmstat.numa_hit
      1280 ± 34%     +67.6%       2145 ± 12%  proc-vmstat.numa_huge_pte_updates
  22105975           -29.6%   15562992 ±  6%  proc-vmstat.numa_local
     16741           -37.3%      10504 ± 10%  proc-vmstat.pgactivate
 5.468e+09           -30.5%  3.799e+09 ±  7%  proc-vmstat.pgalloc_normal
  11395702           -28.6%    8132778 ±  6%  proc-vmstat.pgfault
 5.467e+09           -30.5%  3.798e+09 ±  7%  proc-vmstat.pgfree
  10655696           -30.5%    7402060 ±  7%  proc-vmstat.thp_deferred_split_page
  10657019           -30.5%    7402961 ±  7%  proc-vmstat.thp_fault_alloc
    100455           -32.5%      67779 ±  5%  sched_debug.cfs_rq:/.exec_clock.avg
    101366           -31.2%      69767 ±  5%  sched_debug.cfs_rq:/.exec_clock.max
     98601           -32.0%      67062 ±  5%  sched_debug.cfs_rq:/.exec_clock.min
    291.53 ±  7%     +47.7%     430.62 ±  3%  sched_debug.cfs_rq:/.exec_clock.stddev
      7901 ±  3%     +28.6%      10161 ± 11%  sched_debug.cfs_rq:/.load.avg
      8467 ±  2%    +138.8%      20218 ± 68%  sched_debug.cfs_rq:/.load.stddev
     18.30 ± 10%     -25.9%      13.56 ± 11%  sched_debug.cfs_rq:/.load_avg.avg
   6201909           -44.9%    3415062 ±  7%  sched_debug.cfs_rq:/.min_vruntime.avg
   6248557           -44.8%    3447691 ±  7%  sched_debug.cfs_rq:/.min_vruntime.max
   6084722           -44.3%    3388823 ±  7%  sched_debug.cfs_rq:/.min_vruntime.min
     19981 ± 12%     -54.5%       9092 ±  3%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.31 ± 13%     -56.7%       0.13        sched_debug.cfs_rq:/.nr_spread_over.avg
      3.50           -41.3%       2.06 ±  7%  sched_debug.cfs_rq:/.nr_spread_over.max
      0.63 ±  3%     -41.9%       0.36 ±  3%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      4.82 ± 20%     -73.2%       1.29 ±141%  sched_debug.cfs_rq:/.removed.load_avg.avg
    221.46 ± 20%     -72.9%      59.98 ±141%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      1.82 ± 13%     -78.5%       0.39 ±141%  sched_debug.cfs_rq:/.removed.util_avg.avg
     11.19 ±  5%     -67.5%       3.64 ±141%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      7826 ±  3%     +29.5%      10132 ± 11%  sched_debug.cfs_rq:/.runnable_weight.avg
      8345          +141.9%      20192 ± 68%  sched_debug.cfs_rq:/.runnable_weight.stddev
   -118216           -82.5%     -20707        sched_debug.cfs_rq:/.spread0.min
     20021 ± 12%     -54.5%       9100 ±  3%  sched_debug.cfs_rq:/.spread0.stddev
    633.16           -23.8%     482.47 ± 12%  sched_debug.cfs_rq:/.util_avg.avg
      1261 ±  2%     -13.9%       1086 ±  5%  sched_debug.cfs_rq:/.util_avg.max
    140.27 ±  4%     -15.3%     118.80 ±  4%  sched_debug.cfs_rq:/.util_avg.stddev
    291.30 ±  4%     -33.4%     194.09 ± 24%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    936.92 ±  2%     -14.5%     800.89 ± 12%  sched_debug.cfs_rq:/.util_est_enqueued.max
    308.09           -24.1%     233.93 ± 12%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      5.34 ±  3%     -10.0%       4.81 ±  2%  sched_debug.cpu.cpu_load[0].avg
      6.01 ±  4%     -11.3%       5.33 ±  2%  sched_debug.cpu.cpu_load[1].avg
      6.66 ±  6%     -15.7%       5.62 ±  2%  sched_debug.cpu.cpu_load[2].avg
      7.55 ±  7%     -21.0%       5.96 ±  5%  sched_debug.cpu.cpu_load[3].avg
      1.75 ± 14%     -42.9%       1.00 ± 49%  sched_debug.cpu.cpu_load[3].min
      8.71 ±  4%     -24.0%       6.62 ± 10%  sched_debug.cpu.cpu_load[4].avg
      3.42 ± 17%     -74.0%       0.89 ±  8%  sched_debug.cpu.cpu_load[4].min
      7901 ±  3%     +20.2%       9496 ±  9%  sched_debug.cpu.load.avg
      8506           +68.8%      14357 ± 39%  sched_debug.cpu.load.stddev
      1312           -15.1%       1113 ±  4%  sched_debug.cpu.nr_load_updates.stddev
      0.45 ±  4%     -14.7%       0.38 ± 17%  sched_debug.cpu.nr_running.avg
    113042           -30.7%      78289 ±  5%  sched_debug.cpu.nr_switches.avg
    121999           -29.4%      86169 ±  6%  sched_debug.cpu.nr_switches.max
    110995           -31.2%      76401 ±  6%  sched_debug.cpu.nr_switches.min
      6.59 ±  4%     -13.9%       5.68 ±  9%  sched_debug.cpu.nr_uninterruptible.stddev
    112717           -30.8%      77983 ±  5%  sched_debug.cpu.sched_count.avg
    149308           -24.2%     113170 ±  2%  sched_debug.cpu.sched_count.max
    110675           -31.3%      76075 ±  6%  sched_debug.cpu.sched_count.min
     55809           -30.9%      38538 ±  5%  sched_debug.cpu.sched_goidle.avg
     57654           -29.2%      40795 ±  4%  sched_debug.cpu.sched_goidle.max
     55072           -31.2%      37870 ±  6%  sched_debug.cpu.sched_goidle.min
     56125           -31.0%      38708 ±  5%  sched_debug.cpu.ttwu_count.avg
     64581           -30.0%      45231 ±  4%  sched_debug.cpu.ttwu_count.max
     46030 ±  2%     -30.9%      31806 ±  5%  sched_debug.cpu.ttwu_count.min
      4346 ± 10%     -32.7%       2925 ±  6%  sched_debug.cpu.ttwu_count.stddev
    255.17           -17.5%     210.41 ±  3%  perf-stat.i.MPKI
 1.126e+09            -7.7%  1.039e+09        perf-stat.i.branch-instructions
      7.31            -0.3        6.99 ±  2%  perf-stat.i.cache-miss-rate%
  93265271           -31.3%   64081982 ±  7%  perf-stat.i.cache-misses
 1.275e+09           -28.5%  9.114e+08 ±  5%  perf-stat.i.cache-references
     66363           -29.7%      46640 ±  7%  perf-stat.i.context-switches
     32.66           -19.1%      26.43 ±  4%  perf-stat.i.cpi
 1.632e+11           -29.9%  1.145e+11 ±  6%  perf-stat.i.cpu-cycles
    124.92 ±  3%     -33.7%      82.83 ±  5%  perf-stat.i.cpu-migrations
      1750            +3.4%       1810        perf-stat.i.cycles-between-cache-misses
 1.369e+09           -14.4%  1.172e+09        perf-stat.i.dTLB-loads
      0.04 ±  4%      +0.0        0.05 ± 14%  perf-stat.i.dTLB-store-miss-rate%
   1334837 ±  4%      -8.1%    1227045 ± 10%  perf-stat.i.dTLB-store-misses
 3.118e+09           -25.0%  2.339e+09 ±  3%  perf-stat.i.dTLB-stores
     42.38           +10.4       52.76 ± 15%  perf-stat.i.iTLB-load-miss-rate%
    659514 ±  2%     +28.5%     847547 ± 16%  perf-stat.i.iTLB-load-misses
    897739           -14.7%     765495 ± 15%  perf-stat.i.iTLB-loads
 5.013e+09           -13.1%  4.358e+09 ±  2%  perf-stat.i.instructions
      7614 ±  2%     -30.2%       5317 ± 18%  perf-stat.i.instructions-per-iTLB-miss
      0.03           +34.1%       0.04 ±  4%  perf-stat.i.ipc
     37463           -29.1%      26569 ±  6%  perf-stat.i.minor-faults
   1425900           -32.3%     964726 ±  7%  perf-stat.i.node-load-misses
  14748538           -31.0%   10170816 ±  7%  perf-stat.i.node-loads
    462774           -32.0%     314783 ±  7%  perf-stat.i.node-store-misses
  57326846           -32.2%   38895914 ±  8%  perf-stat.i.node-stores
     37463           -29.1%      26569 ±  6%  perf-stat.i.page-faults
    254.34           -17.8%     208.98 ±  3%  perf-stat.overall.MPKI
      7.31            -0.3        7.02        perf-stat.overall.cache-miss-rate%
     32.56           -19.4%      26.25 ±  4%  perf-stat.overall.cpi
      1750            +2.2%       1788        perf-stat.overall.cycles-between-cache-misses
      0.04 ±  4%      +0.0        0.05 ± 13%  perf-stat.overall.dTLB-store-miss-rate%
     42.35           +10.1       52.44 ± 14%  perf-stat.overall.iTLB-load-miss-rate%
      7604 ±  2%     -30.3%       5299 ± 18%  perf-stat.overall.instructions-per-iTLB-miss
      0.03           +24.2%       0.04 ±  4%  perf-stat.overall.ipc
     84000           +25.4%     105327 ±  5%  perf-stat.overall.path-length
 1.122e+09            -7.7%  1.036e+09        perf-stat.ps.branch-instructions
  92950218           -31.3%   63867612 ±  7%  perf-stat.ps.cache-misses
 1.271e+09           -28.5%  9.084e+08 ±  5%  perf-stat.ps.cache-references
     66134           -29.7%      46484 ±  7%  perf-stat.ps.context-switches
 1.627e+11           -29.8%  1.141e+11 ±  6%  perf-stat.ps.cpu-cycles
    124.51 ±  3%     -33.7%      82.56 ±  5%  perf-stat.ps.cpu-migrations
 1.365e+09           -14.4%  1.168e+09        perf-stat.ps.dTLB-loads
   1330331 ±  4%      -8.1%    1222940 ± 10%  perf-stat.ps.dTLB-store-misses
 3.107e+09           -25.0%  2.331e+09 ±  3%  perf-stat.ps.dTLB-stores
    657352           +28.5%     844723 ± 16%  perf-stat.ps.iTLB-load-misses
    894671           -14.7%     762930 ± 15%  perf-stat.ps.iTLB-loads
 4.997e+09           -13.1%  4.344e+09 ±  2%  perf-stat.ps.instructions
     37338           -29.1%      26481 ±  6%  perf-stat.ps.minor-faults
   1421060           -32.3%     961504 ±  7%  perf-stat.ps.node-load-misses
  14698520           -31.0%   10136768 ±  7%  perf-stat.ps.node-loads
    461204           -32.0%     313732 ±  7%  perf-stat.ps.node-store-misses
  57132599           -32.1%   38765854 ±  8%  perf-stat.ps.node-stores
     37338           -29.1%      26481 ±  6%  perf-stat.ps.page-faults
 1.515e+12           -12.6%  1.324e+12        perf-stat.total.instructions
     88.47            -4.8       83.65        perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
     88.49            -4.8       83.67        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     88.53            -4.8       83.71        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.testcase
     92.22            -4.5       87.74        perf-profile.calltrace.cycles-pp.testcase
     82.44            -4.4       77.99        perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
     89.03            -4.3       84.73        perf-profile.calltrace.cycles-pp.page_fault.testcase
     88.89            -4.3       84.60        perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.testcase
     88.85            -4.3       84.55        perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault.testcase
     78.61            -4.2       74.43        perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
     76.14            -4.0       72.10        perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault
      4.60            -0.3        4.26 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
      4.58            -0.3        4.25 ±  4%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      3.82            -0.3        3.56 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page
      3.83            -0.3        3.57 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault
      1.45            -0.1        1.37 ±  2%  perf-profile.calltrace.cycles-pp.___might_sleep.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      1.39            -0.1        1.31 ±  2%  perf-profile.calltrace.cycles-pp.rcu_all_qs._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault
      0.84            -0.1        0.79        perf-profile.calltrace.cycles-pp._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.88 ±  2%      -0.0        0.84        perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.60 ±  2%      -0.0        0.57        perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      0.94            -0.0        0.91        perf-profile.calltrace.cycles-pp.apic_timer_interrupt.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page
      0.79 ±  3%      -0.0        0.76        perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      0.89 ±  2%      -0.0        0.85        perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      0.89 ±  2%      -0.0        0.85        perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      0.64 ±  3%      -0.0        0.60        perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.64 ±  3%      -0.0        0.60        perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap
      0.89 ±  2%      -0.0        0.86        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.munmap
      0.89 ±  2%      -0.0        0.86        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      0.89 ±  2%      -0.0        0.86        perf-profile.calltrace.cycles-pp.munmap
      0.56 ±  3%      -0.0        0.53        perf-profile.calltrace.cycles-pp.__free_pages_ok.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region
      0.00            +0.7        0.74 ± 22%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault.testcase
      0.00            +0.9        0.94 ± 14%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.00            +1.0        1.01 ± 15%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      5.44 ±  2%      +3.3        8.76 ± 13%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      5.86 ±  2%      +4.0        9.83 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      5.88 ±  2%      +4.0        9.91 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      6.42 ±  2%      +4.4       10.84 ± 13%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      6.42 ±  2%      +4.4       10.85 ± 13%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      6.42 ±  2%      +4.4       10.85 ± 13%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      6.50 ±  2%      +4.5       10.98 ± 13%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     88.51            -4.8       83.69        perf-profile.children.cycles-pp.__handle_mm_fault
     88.47            -4.8       83.65        perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page
     88.55            -4.8       83.73        perf-profile.children.cycles-pp.handle_mm_fault
     92.37            -4.5       87.88        perf-profile.children.cycles-pp.testcase
     83.08            -4.5       78.63        perf-profile.children.cycles-pp.clear_huge_page
     89.05            -4.3       84.75        perf-profile.children.cycles-pp.page_fault
     88.87            -4.3       84.57        perf-profile.children.cycles-pp.__do_page_fault
     88.91            -4.3       84.61        perf-profile.children.cycles-pp.do_page_fault
     78.63            -4.2       74.44        perf-profile.children.cycles-pp.clear_subpage
     76.80            -4.1       72.70        perf-profile.children.cycles-pp.clear_page_erms
      5.12            -0.4        4.76 ±  4%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      5.11            -0.4        4.74 ±  4%  perf-profile.children.cycles-pp.get_page_from_freelist
      4.33            -0.3        4.07 ±  3%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      3.90            -0.2        3.66 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.48            -0.1        1.39        perf-profile.children.cycles-pp._cond_resched
      1.54            -0.1        1.46 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      0.78            -0.0        0.73 ±  2%  perf-profile.children.cycles-pp.rcu_all_qs
      0.88 ±  2%      -0.0        0.84        perf-profile.children.cycles-pp.__do_munmap
      0.79 ±  3%      -0.0        0.76        perf-profile.children.cycles-pp.unmap_region
      0.89 ±  2%      -0.0        0.85        perf-profile.children.cycles-pp.__x64_sys_munmap
      0.89 ±  2%      -0.0        0.85        perf-profile.children.cycles-pp.__vm_munmap
      0.89 ±  2%      -0.0        0.86        perf-profile.children.cycles-pp.munmap
      0.64 ±  3%      -0.0        0.61        perf-profile.children.cycles-pp.tlb_finish_mmu
      0.64 ±  3%      -0.0        0.61        perf-profile.children.cycles-pp.tlb_flush_mmu
      0.53            -0.0        0.49 ±  3%  perf-profile.children.cycles-pp.pte_alloc_one
      0.61 ±  2%      -0.0        0.58        perf-profile.children.cycles-pp.release_pages
      0.56 ±  3%      -0.0        0.53        perf-profile.children.cycles-pp.__free_pages_ok
      0.55            -0.0        0.52        perf-profile.children.cycles-pp._raw_spin_lock
      0.11 ±  4%      -0.0        0.09 ±  9%  perf-profile.children.cycles-pp.call_function_interrupt
      0.21 ±  2%      -0.0        0.20 ±  4%  perf-profile.children.cycles-pp.prep_new_page
      0.08            -0.0        0.07        perf-profile.children.cycles-pp.downgrade_write
      0.07            +0.0        0.08        perf-profile.children.cycles-pp.rwsem_wake
      0.08            +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.06            +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.update_load_avg
      0.05            +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.05            +0.0        0.07        perf-profile.children.cycles-pp.update_cfs_group
      0.08 ±  6%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.arch_stack_walk
      0.05            +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.13            +0.0        0.16 ±  5%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.12 ±  4%      +0.0        0.16 ± 15%  perf-profile.children.cycles-pp.clockevents_program_event
      0.05            +0.0        0.08 ± 11%  perf-profile.children.cycles-pp.native_write_msr
      0.06            +0.0        0.09 ± 18%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.06            +0.0        0.09 ± 20%  perf-profile.children.cycles-pp.load_balance
      0.03 ±100%      +0.0        0.06        perf-profile.children.cycles-pp.serial8250_console_write
      0.03 ±100%      +0.0        0.06        perf-profile.children.cycles-pp.wait_for_xmitr
      0.03 ±100%      +0.0        0.06        perf-profile.children.cycles-pp.uart_console_write
      0.03 ±100%      +0.0        0.06        perf-profile.children.cycles-pp.serial8250_console_putchar
      0.12 ±  4%      +0.0        0.16 ± 16%  perf-profile.children.cycles-pp.ktime_get
      0.06 ±  9%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.update_blocked_averages
      0.03 ±100%      +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.console_unlock
      0.20 ±  2%      +0.0        0.24 ±  7%  perf-profile.children.cycles-pp.task_tick_fair
      0.06 ±  9%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.15 ±  3%      +0.0        0.20 ±  6%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.20            +0.0        0.24 ±  5%  perf-profile.children.cycles-pp.activate_task
      0.20            +0.0        0.24 ±  5%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.20            +0.0        0.24 ±  5%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.18            +0.0        0.22 ±  5%  perf-profile.children.cycles-pp.enqueue_entity
      0.03 ±100%      +0.0        0.07        perf-profile.children.cycles-pp.irq_work_run_list
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.printk
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.vprintk_emit
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.08 ±  6%      +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.start_kernel
      0.00            +0.1        0.06 ± 23%  perf-profile.children.cycles-pp.find_busiest_group
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.irq_work_run
      0.05            +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.21 ±  4%      +0.1        0.30 ±  8%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.27            +0.1        0.37 ±  7%  perf-profile.children.cycles-pp.scheduler_tick
      0.07 ±  7%      +0.1        0.17 ± 19%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.25            +0.1        0.38 ±  8%  perf-profile.children.cycles-pp.irq_exit
      0.08 ±  5%      +0.1        0.22 ± 20%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.36            +0.1        0.50 ±  8%  perf-profile.children.cycles-pp.update_process_times
      0.39            +0.1        0.54 ±  8%  perf-profile.children.cycles-pp.tick_sched_handle
      0.41            +0.2        0.58 ±  8%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.2        0.24 ± 30%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.18 ±  2%      +0.3        0.44 ± 20%  perf-profile.children.cycles-pp.menu_select
      0.54            +0.3        0.82 ± 11%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.76            +0.4        1.11 ±  9%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +0.5        0.49 ± 30%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.23            +0.5        0.74 ± 22%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      1.06            +0.6        1.63 ±  9%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.50            +0.6        2.09 ±  8%  perf-profile.children.cycles-pp.apic_timer_interrupt
      5.50 ±  2%      +3.4        8.87 ± 13%  perf-profile.children.cycles-pp.intel_idle
      5.96 ±  2%      +4.1       10.03 ± 13%  perf-profile.children.cycles-pp.cpuidle_enter
      5.96 ±  2%      +4.1       10.03 ± 13%  perf-profile.children.cycles-pp.cpuidle_enter_state
      6.42 ±  2%      +4.4       10.85 ± 13%  perf-profile.children.cycles-pp.start_secondary
      6.50 ±  2%      +4.5       10.98 ± 13%  perf-profile.children.cycles-pp.do_idle
      6.50 ±  2%      +4.5       10.98 ± 13%  perf-profile.children.cycles-pp.secondary_startup_64
      6.50 ±  2%      +4.5       10.98 ± 13%  perf-profile.children.cycles-pp.cpu_startup_entry
     76.12            -4.1       72.00        perf-profile.self.cycles-pp.clear_page_erms
      4.33            -0.3        4.07 ±  3%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      3.29            -0.2        3.11        perf-profile.self.cycles-pp.testcase
      1.85            -0.1        1.73 ±  2%  perf-profile.self.cycles-pp.clear_subpage
      2.04            -0.1        1.94        perf-profile.self.cycles-pp.clear_huge_page
      1.40            -0.1        1.34 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.58            -0.1        0.52 ±  7%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.85            -0.0        0.81        perf-profile.self.cycles-pp._cond_resched
      0.55 ±  2%      -0.0        0.51        perf-profile.self.cycles-pp.__free_pages_ok
      0.06            -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.prepare_exit_to_usermode
      0.05            +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.update_cfs_group
      0.05            +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.05            +0.0        0.08 ± 11%  perf-profile.self.cycles-pp.native_write_msr
      0.06            +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.task_tick_fair
      0.06            +0.1        0.16 ± 21%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.12 ± 24%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +0.4        0.39 ± 30%  perf-profile.self.cycles-pp.rwsem_optimistic_spin
      5.46 ±  2%      +3.4        8.83 ± 13%  perf-profile.self.cycles-pp.intel_idle
     19530 ± 14%     -21.6%      15312 ±  5%  softirqs.CPU0.RCU
     64270           -12.4%      56330 ±  3%  softirqs.CPU0.SCHED
     20015 ± 16%     -20.4%      15927 ±  3%  softirqs.CPU1.RCU
     60341           -11.1%      53658        softirqs.CPU1.SCHED
     19425 ± 15%     -22.3%      15095 ±  4%  softirqs.CPU10.RCU
     60073           -12.1%      52792 ±  2%  softirqs.CPU10.SCHED
     19624 ± 15%     -22.8%      15159 ±  5%  softirqs.CPU11.RCU
     60057           -12.1%      52787 ±  2%  softirqs.CPU11.SCHED
     20026 ± 15%     -23.9%      15236 ±  4%  softirqs.CPU12.RCU
     60183           -12.1%      52881 ±  2%  softirqs.CPU12.SCHED
     20460 ± 19%     -24.1%      15537 ±  3%  softirqs.CPU13.RCU
     59977           -11.5%      53056        softirqs.CPU13.SCHED
     20046 ± 15%     -22.0%      15627 ±  9%  softirqs.CPU14.RCU
     59956           -11.6%      53006        softirqs.CPU14.SCHED
     19120 ± 13%     -18.4%      15606 ±  5%  softirqs.CPU15.RCU
     59915           -11.3%      53117        softirqs.CPU15.SCHED
     19156 ± 14%     -23.4%      14680 ±  3%  softirqs.CPU16.RCU
     59967           -11.7%      52956        softirqs.CPU16.SCHED
     19217 ± 13%     -25.3%      14364 ±  4%  softirqs.CPU17.RCU
     60129           -12.1%      52872 ±  2%  softirqs.CPU17.SCHED
     19578 ± 12%     -25.3%      14629 ±  2%  softirqs.CPU18.RCU
     60176           -11.9%      53023        softirqs.CPU18.SCHED
     19518 ± 11%     -26.7%      14308 ±  5%  softirqs.CPU19.RCU
     59910           -11.6%      52934        softirqs.CPU19.SCHED
     24443 ± 11%     -32.1%      16606 ± 18%  softirqs.CPU2.RCU
     60404           -12.3%      52947        softirqs.CPU2.SCHED
     19632 ± 12%     -25.0%      14720 ±  4%  softirqs.CPU20.RCU
     60264           -12.1%      52961        softirqs.CPU20.SCHED
     19171 ± 18%     -22.8%      14801 ±  4%  softirqs.CPU21.RCU
     59852           -11.5%      52962        softirqs.CPU21.SCHED
     19805 ± 17%     -23.4%      15163 ±  3%  softirqs.CPU22.RCU
     60578           -10.9%      53965        softirqs.CPU22.SCHED
     19284 ± 14%     -22.3%      14990 ±  5%  softirqs.CPU23.RCU
     60019           -10.8%      53565        softirqs.CPU23.SCHED
     19721 ± 13%     -27.4%      14308        softirqs.CPU24.RCU
     60264           -11.8%      53145        softirqs.CPU24.SCHED
    106551           +30.4%     138925 ± 14%  softirqs.CPU24.TIMER
     19286 ± 14%     -25.7%      14332 ±  5%  softirqs.CPU25.RCU
     59984           -11.5%      53069        softirqs.CPU25.SCHED
     19481 ± 13%     -25.7%      14472 ±  4%  softirqs.CPU26.RCU
     60012           -11.7%      52976        softirqs.CPU26.SCHED
     19518 ± 13%     -25.8%      14484 ±  3%  softirqs.CPU27.RCU
     59977           -11.9%      52831        softirqs.CPU27.SCHED
     19214 ± 14%     -24.4%      14519 ±  4%  softirqs.CPU28.RCU
     59994           -11.7%      52959        softirqs.CPU28.SCHED
     18986 ± 14%     -23.6%      14510 ±  4%  softirqs.CPU29.RCU
     59937           -11.5%      53020        softirqs.CPU29.SCHED
     19988 ± 16%     -25.3%      14939 ±  4%  softirqs.CPU3.RCU
     60448           -11.4%      53567 ±  2%  softirqs.CPU3.SCHED
     19521 ± 13%     -24.3%      14777 ±  5%  softirqs.CPU30.RCU
     59856           -11.3%      53081        softirqs.CPU30.SCHED
     19464 ± 14%     -19.9%      15595 ±  6%  softirqs.CPU31.RCU
     59929           -11.2%      53196        softirqs.CPU31.SCHED
     20055 ± 16%     -24.2%      15211 ±  5%  softirqs.CPU32.RCU
     59943           -11.6%      52970        softirqs.CPU32.SCHED
     19578 ± 15%     -23.6%      14959 ±  4%  softirqs.CPU33.RCU
     59768           -11.2%      53083        softirqs.CPU33.SCHED
     19885 ± 16%     -23.1%      15296 ±  5%  softirqs.CPU34.RCU
     59587           -10.4%      53409        softirqs.CPU34.SCHED
     19735 ± 14%     -21.6%      15475 ±  2%  softirqs.CPU35.RCU
     59922           -11.2%      53230        softirqs.CPU35.SCHED
     19705 ± 16%     -19.3%      15903 ±  4%  softirqs.CPU36.RCU
     59931           -11.0%      53339        softirqs.CPU36.SCHED
     19754 ± 15%     -20.0%      15796 ±  3%  softirqs.CPU37.RCU
     59935           -11.4%      53078        softirqs.CPU37.SCHED
     19547 ± 15%     -20.4%      15565 ±  3%  softirqs.CPU38.RCU
     59731           -11.1%      53085        softirqs.CPU38.SCHED
     19656 ± 14%     -22.9%      15164 ±  3%  softirqs.CPU39.RCU
     59914           -11.5%      53041        softirqs.CPU39.SCHED
     19772 ± 16%     -24.2%      14984 ±  3%  softirqs.CPU4.RCU
     60336           -12.4%      52838 ±  2%  softirqs.CPU4.SCHED
     20611 ± 18%     -21.2%      16248 ±  5%  softirqs.CPU40.RCU
     59985           -11.8%      52919        softirqs.CPU40.SCHED
     19856 ± 13%     -23.3%      15225 ±  3%  softirqs.CPU41.RCU
     59898           -11.4%      53046        softirqs.CPU41.SCHED
     19637 ± 15%     -20.1%      15695 ±  3%  softirqs.CPU42.RCU
     59885           -11.3%      53089        softirqs.CPU42.SCHED
     19999 ± 14%     -18.2%      16356 ±  8%  softirqs.CPU43.RCU
     59603           -11.0%      53058        softirqs.CPU43.SCHED
     16120 ± 13%     -24.4%      12183 ±  4%  softirqs.CPU44.RCU
     60670           -12.9%      52873 ±  2%  softirqs.CPU44.SCHED
     19594 ± 14%     -14.2%      16818 ± 15%  softirqs.CPU45.RCU
     60408           -12.7%      52706 ±  2%  softirqs.CPU45.SCHED
     19825 ± 12%     -26.5%      14576 ±  6%  softirqs.CPU46.RCU
     60322           -12.5%      52794 ±  2%  softirqs.CPU46.SCHED
     19535 ± 13%     -23.7%      14900 ±  7%  softirqs.CPU47.RCU
     62149           -14.9%      52868 ±  2%  softirqs.CPU47.SCHED
     19858 ± 13%     -24.3%      15030 ±  5%  softirqs.CPU48.RCU
     61364           -12.5%      53686 ±  4%  softirqs.CPU48.SCHED
     20020 ± 12%     -25.7%      14875 ±  5%  softirqs.CPU49.RCU
     60360           -12.6%      52784 ±  2%  softirqs.CPU49.SCHED
     19484 ± 15%     -13.4%      16873 ± 11%  softirqs.CPU5.RCU
     60098           -12.0%      52900        softirqs.CPU5.SCHED
     17770 ±  5%     -15.6%      14996 ±  5%  softirqs.CPU50.RCU
     60397           -12.6%      52761 ±  2%  softirqs.CPU50.SCHED
     19304 ± 14%     -21.2%      15215 ±  7%  softirqs.CPU51.RCU
     60345           -12.5%      52796 ±  2%  softirqs.CPU51.SCHED
     18977 ± 16%     -22.9%      14623 ±  6%  softirqs.CPU52.RCU
     60278           -12.8%      52578 ±  2%  softirqs.CPU52.SCHED
     19505 ± 14%     -24.9%      14656 ±  6%  softirqs.CPU53.RCU
     60414           -13.1%      52529 ±  2%  softirqs.CPU53.SCHED
     20306 ±  7%     -26.1%      14996 ±  4%  softirqs.CPU54.RCU
     60265           -12.5%      52756 ±  2%  softirqs.CPU54.SCHED
     19701 ± 14%     -23.6%      15047 ±  5%  softirqs.CPU55.RCU
     60283           -12.2%      52944 ±  2%  softirqs.CPU55.SCHED
     19719 ± 14%     -23.8%      15033 ±  4%  softirqs.CPU56.RCU
     60325           -12.3%      52875 ±  2%  softirqs.CPU56.SCHED
     19793 ± 15%     -22.8%      15289 ±  8%  softirqs.CPU57.RCU
     60296           -12.2%      52960        softirqs.CPU57.SCHED
     19846 ± 13%     -24.9%      14903 ±  7%  softirqs.CPU58.RCU
     60379           -11.9%      53196        softirqs.CPU58.SCHED
     19768 ± 14%     -23.1%      15208 ±  5%  softirqs.CPU59.RCU
     60297           -11.9%      53101        softirqs.CPU59.SCHED
     18800 ±  5%     -13.3%      16302 ± 10%  softirqs.CPU6.RCU
     60513           -12.7%      52816 ±  2%  softirqs.CPU6.SCHED
     18575 ± 14%     -24.9%      13958 ±  4%  softirqs.CPU60.RCU
     60286           -12.0%      53039        softirqs.CPU60.SCHED
     18519 ± 14%     -24.1%      14055 ±  4%  softirqs.CPU61.RCU
     60252           -12.2%      52904        softirqs.CPU61.SCHED
     18589 ± 13%     -23.6%      14205 ±  4%  softirqs.CPU62.RCU
     60391           -12.1%      53106        softirqs.CPU62.SCHED
     18738 ± 15%     -25.8%      13897 ±  5%  softirqs.CPU63.RCU
     60394           -12.2%      53027        softirqs.CPU63.SCHED
     18818 ± 15%     -24.2%      14268 ±  5%  softirqs.CPU64.RCU
     60451           -12.0%      53185        softirqs.CPU64.SCHED
     18685 ± 14%     -24.9%      14026 ±  5%  softirqs.CPU65.RCU
     60245           -12.1%      52948 ±  2%  softirqs.CPU65.SCHED
     19470 ± 14%     -22.9%      15014 ±  5%  softirqs.CPU66.RCU
     60365           -12.1%      53085        softirqs.CPU66.SCHED
     19304 ± 14%     -22.1%      15041 ±  5%  softirqs.CPU67.RCU
     60116           -11.5%      53213        softirqs.CPU67.SCHED
     20880 ±  5%     -31.1%      14381        softirqs.CPU68.RCU
     60135           -12.2%      52811        softirqs.CPU68.SCHED
    105953           +12.6%     119354 ±  5%  softirqs.CPU68.TIMER
     19242 ± 14%     -24.5%      14523 ±  6%  softirqs.CPU69.RCU
     60233           -12.0%      52990        softirqs.CPU69.SCHED
     19350 ± 13%     -21.3%      15221 ±  5%  softirqs.CPU7.RCU
     60226           -12.3%      52815 ±  2%  softirqs.CPU7.SCHED
     19469 ± 15%     -24.3%      14736 ±  4%  softirqs.CPU70.RCU
     60193           -11.8%      53104        softirqs.CPU70.SCHED
     19362 ± 14%     -23.8%      14754 ±  4%  softirqs.CPU71.RCU
     60166           -11.7%      53109        softirqs.CPU71.SCHED
     19194 ± 14%     -23.1%      14763 ±  4%  softirqs.CPU72.RCU
     60282           -12.1%      53017        softirqs.CPU72.SCHED
     19144 ± 14%     -23.3%      14688 ±  5%  softirqs.CPU73.RCU
     60195           -11.9%      53025        softirqs.CPU73.SCHED
     19387 ± 15%     -25.3%      14478 ±  5%  softirqs.CPU74.RCU
     60132           -11.6%      53143        softirqs.CPU74.SCHED
     17915 ± 15%     -21.3%      14104 ±  8%  softirqs.CPU75.RCU
     60146           -11.5%      53254        softirqs.CPU75.SCHED
     18065 ± 14%     -22.4%      14019 ±  4%  softirqs.CPU76.RCU
     60068           -11.7%      53045        softirqs.CPU76.SCHED
     18264 ± 14%     -22.0%      14252 ±  4%  softirqs.CPU77.RCU
     60054           -11.9%      52913        softirqs.CPU77.SCHED
     18344 ± 14%     -21.7%      14368 ±  4%  softirqs.CPU78.RCU
     60044           -11.6%      53071        softirqs.CPU78.SCHED
     18579 ± 15%     -20.8%      14722 ±  5%  softirqs.CPU79.RCU
     60158           -11.5%      53226        softirqs.CPU79.SCHED
     18673 ± 19%     -21.3%      14702 ±  4%  softirqs.CPU8.RCU
     59969           -12.4%      52534 ±  2%  softirqs.CPU8.SCHED
     18594 ± 15%     -17.5%      15339 ±  2%  softirqs.CPU80.RCU
     60089           -11.4%      53257        softirqs.CPU80.SCHED
     18605 ± 14%     -19.2%      15038 ±  3%  softirqs.CPU81.RCU
     60055           -11.4%      53234        softirqs.CPU81.SCHED
     18588 ± 14%     -20.4%      14795 ±  3%  softirqs.CPU82.RCU
     60163           -11.7%      53119        softirqs.CPU82.SCHED
     18284 ± 14%     -20.5%      14531 ±  5%  softirqs.CPU83.RCU
     60105           -11.5%      53170        softirqs.CPU83.SCHED
     18546 ± 16%     -21.8%      14508 ±  3%  softirqs.CPU84.RCU
     60308           -12.3%      52899        softirqs.CPU84.SCHED
     18364 ± 15%     -20.6%      14574 ±  4%  softirqs.CPU85.RCU
     60163           -11.8%      53038        softirqs.CPU85.SCHED
     18810 ± 16%     -21.8%      14705 ±  3%  softirqs.CPU86.RCU
     60451           -12.4%      52947        softirqs.CPU86.SCHED
     18460 ± 14%     -20.2%      14734 ±  3%  softirqs.CPU87.RCU
     60133           -11.8%      53035        softirqs.CPU87.SCHED
     19636 ± 14%     -23.9%      14948 ±  6%  softirqs.CPU9.RCU
     60205           -12.8%      52512 ±  2%  softirqs.CPU9.SCHED
   1703438 ± 14%     -22.8%    1314494 ±  4%  softirqs.RCU
   5300291           -11.9%    4669588        softirqs.SCHED
    930.00 ±  5%     -52.8%     439.33 ± 68%  interrupts.36:IR-PCI-MSI.1572867-edge.eth0-TxRx-2
    249.50 ±  9%     +78.0%     444.00 ±  4%  interrupts.37:IR-PCI-MSI.1572868-edge.eth0-TxRx-3
    240.00 ± 21%     -22.2%     186.67 ±  7%  interrupts.39:IR-PCI-MSI.1572870-edge.eth0-TxRx-5
   5809962           -26.1%    4294506 ±  3%  interrupts.CAL:Function_call_interrupts
     65922           -26.2%      48647 ±  3%  interrupts.CPU0.CAL:Function_call_interrupts
      5489           -27.3%       3990 ±  4%  interrupts.CPU0.NMI:Non-maskable_interrupts
      5489           -27.3%       3990 ±  4%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      2707 ±  2%     -23.5%       2071 ±  7%  interrupts.CPU0.RES:Rescheduling_interrupts
     64172           -27.7%      46395 ±  3%  interrupts.CPU0.TLB:TLB_shootdowns
     66007           -26.3%      48638 ±  3%  interrupts.CPU1.CAL:Function_call_interrupts
      5492           -27.8%       3966 ±  4%  interrupts.CPU1.NMI:Non-maskable_interrupts
      5492           -27.8%       3966 ±  4%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
      2009 ±  6%     -28.9%       1429 ± 17%  interrupts.CPU1.RES:Rescheduling_interrupts
     64238           -27.9%      46323 ±  3%  interrupts.CPU1.TLB:TLB_shootdowns
     66187           -26.1%      48943 ±  3%  interrupts.CPU10.CAL:Function_call_interrupts
      5385           -26.7%       3946 ±  4%  interrupts.CPU10.NMI:Non-maskable_interrupts
      5385           -26.7%       3946 ±  4%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      1827 ±  2%     -29.3%       1292 ± 10%  interrupts.CPU10.RES:Rescheduling_interrupts
     64346           -27.5%      46653 ±  3%  interrupts.CPU10.TLB:TLB_shootdowns
     66190           -26.2%      48850 ±  3%  interrupts.CPU11.CAL:Function_call_interrupts
      5397           -26.8%       3948 ±  4%  interrupts.CPU11.NMI:Non-maskable_interrupts
      5397           -26.8%       3948 ±  4%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      1739 ±  5%     -25.0%       1304 ± 10%  interrupts.CPU11.RES:Rescheduling_interrupts
     64351           -27.6%      46580 ±  3%  interrupts.CPU11.TLB:TLB_shootdowns
     66290           -26.2%      48904 ±  3%  interrupts.CPU12.CAL:Function_call_interrupts
      5415           -27.3%       3938 ±  4%  interrupts.CPU12.NMI:Non-maskable_interrupts
      5415           -27.3%       3938 ±  4%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      1840 ±  2%     -32.5%       1242 ± 11%  interrupts.CPU12.RES:Rescheduling_interrupts
     64455           -27.6%      46681 ±  3%  interrupts.CPU12.TLB:TLB_shootdowns
     66279           -26.0%      49015 ±  3%  interrupts.CPU13.CAL:Function_call_interrupts
      5391           -26.9%       3943 ±  5%  interrupts.CPU13.NMI:Non-maskable_interrupts
      5391           -26.9%       3943 ±  5%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      1790           -26.5%       1315 ± 11%  interrupts.CPU13.RES:Rescheduling_interrupts
     64447           -27.4%      46806 ±  3%  interrupts.CPU13.TLB:TLB_shootdowns
     65793           -26.2%      48532 ±  3%  interrupts.CPU14.CAL:Function_call_interrupts
      5438           -40.7%       3224 ± 24%  interrupts.CPU14.NMI:Non-maskable_interrupts
      5438           -40.7%       3224 ± 24%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      1714           -26.4%       1261 ±  9%  interrupts.CPU14.RES:Rescheduling_interrupts
     64058           -27.8%      46278 ±  3%  interrupts.CPU14.TLB:TLB_shootdowns
    930.00 ±  5%     -52.8%     439.33 ± 68%  interrupts.CPU15.36:IR-PCI-MSI.1572867-edge.eth0-TxRx-2
     65685           -26.5%      48304 ±  3%  interrupts.CPU15.CAL:Function_call_interrupts
      5449           -40.2%       3257 ± 26%  interrupts.CPU15.NMI:Non-maskable_interrupts
      5449           -40.2%       3257 ± 26%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      1720           -26.6%       1261 ±  8%  interrupts.CPU15.RES:Rescheduling_interrupts
     63971           -27.8%      46199 ±  3%  interrupts.CPU15.TLB:TLB_shootdowns
    249.50 ±  9%     +78.0%     444.00 ±  4%  interrupts.CPU16.37:IR-PCI-MSI.1572868-edge.eth0-TxRx-3
     65809           -26.4%      48446 ±  3%  interrupts.CPU16.CAL:Function_call_interrupts
      5429           -40.2%       3246 ± 25%  interrupts.CPU16.NMI:Non-maskable_interrupts
      5429           -40.2%       3246 ± 25%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
      1663           -25.6%       1238 ± 11%  interrupts.CPU16.RES:Rescheduling_interrupts
     64115 ±  2%     -27.9%      46220 ±  3%  interrupts.CPU16.TLB:TLB_shootdowns
     66111           -26.2%      48774 ±  3%  interrupts.CPU17.CAL:Function_call_interrupts
      5463           -26.8%       3997 ±  4%  interrupts.CPU17.NMI:Non-maskable_interrupts
      5463           -26.8%       3997 ±  4%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      1701           -26.8%       1245 ± 12%  interrupts.CPU17.RES:Rescheduling_interrupts
     64433           -27.7%      46609 ±  3%  interrupts.CPU17.TLB:TLB_shootdowns
    240.00 ± 21%     -22.2%     186.67 ±  7%  interrupts.CPU18.39:IR-PCI-MSI.1572870-edge.eth0-TxRx-5
     66182           -26.1%      48885 ±  3%  interrupts.CPU18.CAL:Function_call_interrupts
      5409           -40.0%       3248 ± 24%  interrupts.CPU18.NMI:Non-maskable_interrupts
      5409           -40.0%       3248 ± 24%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
      1751           -23.4%       1340 ±  8%  interrupts.CPU18.RES:Rescheduling_interrupts
     64372           -27.5%      46649 ±  3%  interrupts.CPU18.TLB:TLB_shootdowns
     66316           -26.3%      48852 ±  3%  interrupts.CPU19.CAL:Function_call_interrupts
      5437           -51.3%       2648 ± 33%  interrupts.CPU19.NMI:Non-maskable_interrupts
      5437           -51.3%       2648 ± 33%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
      1724 ±  3%     -21.0%       1362 ± 10%  interrupts.CPU19.RES:Rescheduling_interrupts
     64485 ±  2%     -27.5%      46739 ±  3%  interrupts.CPU19.TLB:TLB_shootdowns
     65748           -26.3%      48476 ±  3%  interrupts.CPU2.CAL:Function_call_interrupts
      5446           -27.8%       3935 ±  4%  interrupts.CPU2.NMI:Non-maskable_interrupts
      5446           -27.8%       3935 ±  4%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1806 ±  2%     -28.8%       1286 ±  4%  interrupts.CPU2.RES:Rescheduling_interrupts
     63957           -27.7%      46229 ±  3%  interrupts.CPU2.TLB:TLB_shootdowns
     65862           -26.5%      48426 ±  3%  interrupts.CPU20.CAL:Function_call_interrupts
      5426           -51.8%       2614 ± 35%  interrupts.CPU20.NMI:Non-maskable_interrupts
      5426           -51.8%       2614 ± 35%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      1666 ±  3%     -23.3%       1278 ± 12%  interrupts.CPU20.RES:Rescheduling_interrupts
     64130 ±  2%     -27.9%      46226 ±  3%  interrupts.CPU20.TLB:TLB_shootdowns
     65814           -26.2%      48577 ±  3%  interrupts.CPU21.CAL:Function_call_interrupts
      5443           -52.4%       2592 ± 35%  interrupts.CPU21.NMI:Non-maskable_interrupts
      5443           -52.4%       2592 ± 35%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
      1674 ±  2%     -26.5%       1230 ± 12%  interrupts.CPU21.RES:Rescheduling_interrupts
     64044           -27.8%      46260 ±  3%  interrupts.CPU21.TLB:TLB_shootdowns
     65705           -26.2%      48476 ±  2%  interrupts.CPU22.CAL:Function_call_interrupts
      5399           -39.0%       3294 ± 30%  interrupts.CPU22.NMI:Non-maskable_interrupts
      5399           -39.0%       3294 ± 30%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      1891           -30.6%       1312 ± 11%  interrupts.CPU22.RES:Rescheduling_interrupts
     63991           -27.4%      46487 ±  3%  interrupts.CPU22.TLB:TLB_shootdowns
     65832           -26.2%      48553 ±  3%  interrupts.CPU23.CAL:Function_call_interrupts
      5466           -39.0%       3336 ± 31%  interrupts.CPU23.NMI:Non-maskable_interrupts
      5466           -39.0%       3336 ± 31%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      1867 ±  3%     -33.3%       1245 ±  6%  interrupts.CPU23.RES:Rescheduling_interrupts
     64115           -27.8%      46320 ±  3%  interrupts.CPU23.TLB:TLB_shootdowns
     65786           -26.1%      48602 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
      5403           -39.2%       3286 ± 31%  interrupts.CPU24.NMI:Non-maskable_interrupts
      5403           -39.2%       3286 ± 31%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      1774           -27.2%       1292 ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
     64059           -27.7%      46341 ±  3%  interrupts.CPU24.TLB:TLB_shootdowns
     66415           -26.2%      49007 ±  3%  interrupts.CPU25.CAL:Function_call_interrupts
      5378           -39.0%       3279 ± 32%  interrupts.CPU25.NMI:Non-maskable_interrupts
      5378           -39.0%       3279 ± 32%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      1842           -30.0%       1289 ± 11%  interrupts.CPU25.RES:Rescheduling_interrupts
     64581           -27.7%      46713 ±  3%  interrupts.CPU25.TLB:TLB_shootdowns
     66157           -26.3%      48761 ±  3%  interrupts.CPU26.CAL:Function_call_interrupts
      5370           -38.2%       3318 ± 30%  interrupts.CPU26.NMI:Non-maskable_interrupts
      5370           -38.2%       3318 ± 30%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      1867           -29.1%       1324 ±  9%  interrupts.CPU26.RES:Rescheduling_interrupts
     64437           -27.7%      46619 ±  3%  interrupts.CPU26.TLB:TLB_shootdowns
     66109           -25.9%      48990 ±  3%  interrupts.CPU27.CAL:Function_call_interrupts
      5391           -37.9%       3347 ± 32%  interrupts.CPU27.NMI:Non-maskable_interrupts
      5391           -37.9%       3347 ± 32%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      1840           -31.3%       1265 ± 10%  interrupts.CPU27.RES:Rescheduling_interrupts
     64357           -27.4%      46729 ±  3%  interrupts.CPU27.TLB:TLB_shootdowns
     65812           -26.1%      48632 ±  3%  interrupts.CPU28.CAL:Function_call_interrupts
      5400           -39.0%       3292 ± 32%  interrupts.CPU28.NMI:Non-maskable_interrupts
      5400           -39.0%       3292 ± 32%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
      1780           -28.5%       1272 ±  9%  interrupts.CPU28.RES:Rescheduling_interrupts
     64124           -27.7%      46373 ±  3%  interrupts.CPU28.TLB:TLB_shootdowns
     65715           -25.9%      48662 ±  3%  interrupts.CPU29.CAL:Function_call_interrupts
      5390           -38.9%       3295 ± 31%  interrupts.CPU29.NMI:Non-maskable_interrupts
      5390           -38.9%       3295 ± 31%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      1731 ±  2%     -27.3%       1259 ±  7%  interrupts.CPU29.RES:Rescheduling_interrupts
     64069 ±  2%     -27.6%      46404 ±  3%  interrupts.CPU29.TLB:TLB_shootdowns
     66127           -25.8%      49083 ±  3%  interrupts.CPU3.CAL:Function_call_interrupts
      5411           -26.7%       3968 ±  3%  interrupts.CPU3.NMI:Non-maskable_interrupts
      5411           -26.7%       3968 ±  3%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
      1761 ±  3%     -20.6%       1398 ±  4%  interrupts.CPU3.RES:Rescheduling_interrupts
     64308           -27.2%      46787 ±  3%  interrupts.CPU3.TLB:TLB_shootdowns
     65992           -25.6%      49077 ±  3%  interrupts.CPU30.CAL:Function_call_interrupts
      5399           -39.2%       3280 ± 31%  interrupts.CPU30.NMI:Non-maskable_interrupts
      5399           -39.2%       3280 ± 31%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      1876 ±  2%     -28.6%       1339 ± 10%  interrupts.CPU30.RES:Rescheduling_interrupts
     64568           -27.5%      46780 ±  3%  interrupts.CPU30.TLB:TLB_shootdowns
     66260           -26.3%      48814 ±  3%  interrupts.CPU31.CAL:Function_call_interrupts
      5380           -39.1%       3277 ± 32%  interrupts.CPU31.NMI:Non-maskable_interrupts
      5380           -39.1%       3277 ± 32%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      1814 ±  3%     -25.6%       1349 ± 13%  interrupts.CPU31.RES:Rescheduling_interrupts
     64518           -27.7%      46618 ±  3%  interrupts.CPU31.TLB:TLB_shootdowns
     65992           -26.1%      48797 ±  3%  interrupts.CPU32.CAL:Function_call_interrupts
      5351           -38.6%       3284 ± 32%  interrupts.CPU32.NMI:Non-maskable_interrupts
      5351           -38.6%       3284 ± 32%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      1809 ±  4%     -27.1%       1318 ±  9%  interrupts.CPU32.RES:Rescheduling_interrupts
     64365           -27.8%      46474 ±  3%  interrupts.CPU32.TLB:TLB_shootdowns
     66056           -25.9%      48946 ±  3%  interrupts.CPU33.CAL:Function_call_interrupts
      5371           -37.7%       3348 ± 33%  interrupts.CPU33.NMI:Non-maskable_interrupts
      5371           -37.7%       3348 ± 33%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
      1809 ±  3%     -26.9%       1322 ± 10%  interrupts.CPU33.RES:Rescheduling_interrupts
     64442           -27.5%      46689 ±  3%  interrupts.CPU33.TLB:TLB_shootdowns
     65648 ±  2%     -25.5%      48882 ±  3%  interrupts.CPU34.CAL:Function_call_interrupts
      5395           -38.7%       3306 ± 31%  interrupts.CPU34.NMI:Non-maskable_interrupts
      5395           -38.7%       3306 ± 31%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      1858 ±  3%     -29.9%       1303 ± 14%  interrupts.CPU34.RES:Rescheduling_interrupts
     64194 ±  2%     -27.3%      46657 ±  3%  interrupts.CPU34.TLB:TLB_shootdowns
     66264           -25.9%      49073 ±  3%  interrupts.CPU35.CAL:Function_call_interrupts
      5414           -39.9%       3252 ± 31%  interrupts.CPU35.NMI:Non-maskable_interrupts
      5414           -39.9%       3252 ± 31%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      1801 ±  2%     -26.3%       1328 ± 11%  interrupts.CPU35.RES:Rescheduling_interrupts
     64597           -27.4%      46883 ±  3%  interrupts.CPU35.TLB:TLB_shootdowns
     65591           -25.9%      48594 ±  3%  interrupts.CPU36.CAL:Function_call_interrupts
      5408           -39.2%       3286 ± 32%  interrupts.CPU36.NMI:Non-maskable_interrupts
      5408           -39.2%       3286 ± 32%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      1825 ±  4%     -31.1%       1258 ± 10%  interrupts.CPU36.RES:Rescheduling_interrupts
     64062           -27.6%      46353 ±  3%  interrupts.CPU36.TLB:TLB_shootdowns
     65671           -26.1%      48504 ±  3%  interrupts.CPU37.CAL:Function_call_interrupts
      5415           -38.7%       3320 ± 32%  interrupts.CPU37.NMI:Non-maskable_interrupts
      5415           -38.7%       3320 ± 32%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      1755           -25.1%       1314 ±  8%  interrupts.CPU37.RES:Rescheduling_interrupts
     63978           -27.7%      46268 ±  3%  interrupts.CPU37.TLB:TLB_shootdowns
     65648           -26.0%      48550 ±  3%  interrupts.CPU38.CAL:Function_call_interrupts
      5409           -38.4%       3330 ± 30%  interrupts.CPU38.NMI:Non-maskable_interrupts
      5409           -38.4%       3330 ± 30%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
      1780           -29.6%       1253 ±  8%  interrupts.CPU38.RES:Rescheduling_interrupts
     63982           -27.7%      46279 ±  3%  interrupts.CPU38.TLB:TLB_shootdowns
     66198           -26.1%      48905 ±  3%  interrupts.CPU39.CAL:Function_call_interrupts
      5398           -38.6%       3316 ± 30%  interrupts.CPU39.NMI:Non-maskable_interrupts
      5398           -38.6%       3316 ± 30%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      1810           -29.2%       1281 ± 11%  interrupts.CPU39.RES:Rescheduling_interrupts
     64427           -27.6%      46665 ±  3%  interrupts.CPU39.TLB:TLB_shootdowns
     66263           -26.5%      48708 ±  3%  interrupts.CPU4.CAL:Function_call_interrupts
      5396           -25.7%       4009 ±  6%  interrupts.CPU4.NMI:Non-maskable_interrupts
      5396           -25.7%       4009 ±  6%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
      1880 ±  2%     -26.3%       1385 ± 12%  interrupts.CPU4.RES:Rescheduling_interrupts
     64397           -27.7%      46541 ±  3%  interrupts.CPU4.TLB:TLB_shootdowns
     66121           -25.6%      49184 ±  3%  interrupts.CPU40.CAL:Function_call_interrupts
      5351           -38.5%       3289 ± 32%  interrupts.CPU40.NMI:Non-maskable_interrupts
      5351           -38.5%       3289 ± 32%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      1878 ±  2%     -29.8%       1317 ± 13%  interrupts.CPU40.RES:Rescheduling_interrupts
     64405           -27.2%      46867 ±  3%  interrupts.CPU40.TLB:TLB_shootdowns
     66296           -26.0%      49029 ±  3%  interrupts.CPU41.CAL:Function_call_interrupts
      5378           -38.0%       3333 ± 32%  interrupts.CPU41.NMI:Non-maskable_interrupts
      5378           -38.0%       3333 ± 32%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
      1838           -26.9%       1344 ± 10%  interrupts.CPU41.RES:Rescheduling_interrupts
     64481           -27.6%      46709 ±  3%  interrupts.CPU41.TLB:TLB_shootdowns
     65819           -26.1%      48611 ±  3%  interrupts.CPU42.CAL:Function_call_interrupts
      5369           -39.3%       3260 ± 31%  interrupts.CPU42.NMI:Non-maskable_interrupts
      5369           -39.3%       3260 ± 31%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      1873           -29.1%       1327 ± 11%  interrupts.CPU42.RES:Rescheduling_interrupts
     64104           -27.6%      46419 ±  3%  interrupts.CPU42.TLB:TLB_shootdowns
     65437           -25.8%      48560 ±  3%  interrupts.CPU43.CAL:Function_call_interrupts
      5420           -26.7%       3971 ±  4%  interrupts.CPU43.NMI:Non-maskable_interrupts
      5420           -26.7%       3971 ±  4%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      1924           -26.3%       1419 ± 13%  interrupts.CPU43.RES:Rescheduling_interrupts
     63826           -27.3%      46376 ±  3%  interrupts.CPU43.TLB:TLB_shootdowns
     65913           -26.1%      48678 ±  3%  interrupts.CPU44.CAL:Function_call_interrupts
      5463           -39.0%       3331 ± 26%  interrupts.CPU44.NMI:Non-maskable_interrupts
      5463           -39.0%       3331 ± 26%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      1757           -26.6%       1289 ± 11%  interrupts.CPU44.RES:Rescheduling_interrupts
     64194           -27.7%      46390 ±  3%  interrupts.CPU44.TLB:TLB_shootdowns
     65925           -26.1%      48735 ±  3%  interrupts.CPU45.CAL:Function_call_interrupts
      5378           -39.5%       3251 ± 25%  interrupts.CPU45.NMI:Non-maskable_interrupts
      5378           -39.5%       3251 ± 25%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
      1718           -27.3%       1249 ± 10%  interrupts.CPU45.RES:Rescheduling_interrupts
     64195           -27.8%      46364 ±  3%  interrupts.CPU45.TLB:TLB_shootdowns
     65864           -26.4%      48502 ±  3%  interrupts.CPU46.CAL:Function_call_interrupts
      5406           -40.5%       3214 ± 25%  interrupts.CPU46.NMI:Non-maskable_interrupts
      5406           -40.5%       3214 ± 25%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      1760           -29.9%       1234 ±  9%  interrupts.CPU46.RES:Rescheduling_interrupts
     64030           -27.9%      46193 ±  4%  interrupts.CPU46.TLB:TLB_shootdowns
     66129           -25.7%      49150 ±  3%  interrupts.CPU47.CAL:Function_call_interrupts
      5411           -39.7%       3264 ± 26%  interrupts.CPU47.NMI:Non-maskable_interrupts
      5411           -39.7%       3264 ± 26%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      1726           -25.4%       1288 ± 11%  interrupts.CPU47.RES:Rescheduling_interrupts
     64439           -27.4%      46783 ±  3%  interrupts.CPU47.TLB:TLB_shootdowns
     66312           -26.2%      48912 ±  3%  interrupts.CPU48.CAL:Function_call_interrupts
      5410           -40.5%       3216 ± 25%  interrupts.CPU48.NMI:Non-maskable_interrupts
      5410           -40.5%       3216 ± 25%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      1775           -27.6%       1285 ± 15%  interrupts.CPU48.RES:Rescheduling_interrupts
     64494           -27.8%      46539 ±  3%  interrupts.CPU48.TLB:TLB_shootdowns
     66039           -25.8%      49030 ±  3%  interrupts.CPU49.CAL:Function_call_interrupts
      5416           -27.5%       3926 ±  4%  interrupts.CPU49.NMI:Non-maskable_interrupts
      5416           -27.5%       3926 ±  4%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
      1716           -24.4%       1297 ±  9%  interrupts.CPU49.RES:Rescheduling_interrupts
     64385           -27.6%      46645 ±  3%  interrupts.CPU49.TLB:TLB_shootdowns
     66328           -26.4%      48843 ±  3%  interrupts.CPU5.CAL:Function_call_interrupts
      5416           -27.1%       3948 ±  5%  interrupts.CPU5.NMI:Non-maskable_interrupts
      5416           -27.1%       3948 ±  5%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      1864 ±  3%     -28.8%       1327 ±  8%  interrupts.CPU5.RES:Rescheduling_interrupts
     64464           -27.8%      46557 ±  4%  interrupts.CPU5.TLB:TLB_shootdowns
     65492           -25.4%      48828 ±  3%  interrupts.CPU50.CAL:Function_call_interrupts
      5414           -27.4%       3928 ±  3%  interrupts.CPU50.NMI:Non-maskable_interrupts
      5414           -27.4%       3928 ±  3%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      1714           -26.7%       1255 ±  7%  interrupts.CPU50.RES:Rescheduling_interrupts
     63887 ±  2%     -27.3%      46453 ±  3%  interrupts.CPU50.TLB:TLB_shootdowns
     65945           -26.2%      48685 ±  3%  interrupts.CPU51.CAL:Function_call_interrupts
      5444           -28.5%       3892 ±  4%  interrupts.CPU51.NMI:Non-maskable_interrupts
      5444           -28.5%       3892 ±  4%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      1683           -26.7%       1234 ± 10%  interrupts.CPU51.RES:Rescheduling_interrupts
     64156           -27.7%      46370 ±  3%  interrupts.CPU51.TLB:TLB_shootdowns
     66388           -26.0%      49103 ±  3%  interrupts.CPU52.CAL:Function_call_interrupts
      5405           -27.1%       3937 ±  3%  interrupts.CPU52.NMI:Non-maskable_interrupts
      5405           -27.1%       3937 ±  3%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      1768 ±  3%     -26.9%       1293 ± 11%  interrupts.CPU52.RES:Rescheduling_interrupts
     64534           -27.6%      46718 ±  3%  interrupts.CPU52.TLB:TLB_shootdowns
     66347           -26.1%      49050 ±  3%  interrupts.CPU53.CAL:Function_call_interrupts
      5405           -27.4%       3923 ±  5%  interrupts.CPU53.NMI:Non-maskable_interrupts
      5405           -27.4%       3923 ±  5%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      1720 ±  3%     -23.1%       1323 ± 13%  interrupts.CPU53.RES:Rescheduling_interrupts
     64539           -27.8%      46620 ±  3%  interrupts.CPU53.TLB:TLB_shootdowns
     66017           -25.8%      49015 ±  3%  interrupts.CPU54.CAL:Function_call_interrupts
      5372           -27.2%       3908 ±  4%  interrupts.CPU54.NMI:Non-maskable_interrupts
      5372           -27.2%       3908 ±  4%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      1837           -30.5%       1276 ± 14%  interrupts.CPU54.RES:Rescheduling_interrupts
     64317           -27.5%      46660 ±  3%  interrupts.CPU54.TLB:TLB_shootdowns
     66235           -26.1%      48940 ±  3%  interrupts.CPU55.CAL:Function_call_interrupts
      5406           -27.7%       3910 ±  4%  interrupts.CPU55.NMI:Non-maskable_interrupts
      5406           -27.7%       3910 ±  4%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
      1691           -23.9%       1287 ± 15%  interrupts.CPU55.RES:Rescheduling_interrupts
     64428           -27.7%      46574 ±  3%  interrupts.CPU55.TLB:TLB_shootdowns
     66226           -25.9%      49074 ±  3%  interrupts.CPU56.CAL:Function_call_interrupts
      5361           -26.4%       3948 ±  3%  interrupts.CPU56.NMI:Non-maskable_interrupts
      5361           -26.4%       3948 ±  3%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      1783           -30.8%       1234 ± 15%  interrupts.CPU56.RES:Rescheduling_interrupts
     64378           -27.5%      46675 ±  3%  interrupts.CPU56.TLB:TLB_shootdowns
     66301           -25.9%      49129 ±  3%  interrupts.CPU57.CAL:Function_call_interrupts
      5384           -25.6%       4007 ±  6%  interrupts.CPU57.NMI:Non-maskable_interrupts
      5384           -25.6%       4007 ±  6%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      1747           -25.9%       1294 ±  9%  interrupts.CPU57.RES:Rescheduling_interrupts
     64426           -27.4%      46745 ±  3%  interrupts.CPU57.TLB:TLB_shootdowns
     65759           -26.3%      48461 ±  3%  interrupts.CPU58.CAL:Function_call_interrupts
      5413           -28.0%       3896 ±  5%  interrupts.CPU58.NMI:Non-maskable_interrupts
      5413           -28.0%       3896 ±  5%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      1749 ±  4%     -28.6%       1249 ± 13%  interrupts.CPU58.RES:Rescheduling_interrupts
     64175           -27.9%      46281 ±  4%  interrupts.CPU58.TLB:TLB_shootdowns
     65645           -26.2%      48466 ±  3%  interrupts.CPU59.CAL:Function_call_interrupts
      5400           -27.4%       3921 ±  4%  interrupts.CPU59.NMI:Non-maskable_interrupts
      5400           -27.4%       3921 ±  4%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
      1714           -28.4%       1228 ± 11%  interrupts.CPU59.RES:Rescheduling_interrupts
     63936           -27.9%      46111 ±  4%  interrupts.CPU59.TLB:TLB_shootdowns
     65717           -25.9%      48685 ±  3%  interrupts.CPU6.CAL:Function_call_interrupts
      5387           -25.8%       3995 ±  3%  interrupts.CPU6.NMI:Non-maskable_interrupts
      5387           -25.8%       3995 ±  3%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      1701           -26.2%       1255 ±  7%  interrupts.CPU6.RES:Rescheduling_interrupts
     63963 ±  2%     -27.4%      46438 ±  3%  interrupts.CPU6.TLB:TLB_shootdowns
     65762           -26.2%      48519 ±  3%  interrupts.CPU60.CAL:Function_call_interrupts
      5391           -27.2%       3926 ±  4%  interrupts.CPU60.NMI:Non-maskable_interrupts
      5391           -27.2%       3926 ±  4%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      1707           -28.1%       1227 ± 10%  interrupts.CPU60.RES:Rescheduling_interrupts
     64018           -27.9%      46134 ±  3%  interrupts.CPU60.TLB:TLB_shootdowns
     66106           -25.9%      48953 ±  3%  interrupts.CPU61.CAL:Function_call_interrupts
      5396           -26.0%       3993 ±  3%  interrupts.CPU61.NMI:Non-maskable_interrupts
      5396           -26.0%       3993 ±  3%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      1698           -23.0%       1308 ± 10%  interrupts.CPU61.RES:Rescheduling_interrupts
     64308           -27.5%      46621 ±  3%  interrupts.CPU61.TLB:TLB_shootdowns
     66211           -26.1%      48955 ±  3%  interrupts.CPU62.CAL:Function_call_interrupts
      5329           -26.5%       3916 ±  4%  interrupts.CPU62.NMI:Non-maskable_interrupts
      5329           -26.5%       3916 ±  4%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      1843           -31.5%       1263 ± 13%  interrupts.CPU62.RES:Rescheduling_interrupts
     64359           -27.5%      46660 ±  3%  interrupts.CPU62.TLB:TLB_shootdowns
     66332           -26.0%      49104 ±  3%  interrupts.CPU63.CAL:Function_call_interrupts
      5374           -38.6%       3297 ± 30%  interrupts.CPU63.NMI:Non-maskable_interrupts
      5374           -38.6%       3297 ± 30%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
      1857 ±  7%     -26.6%       1363 ±  8%  interrupts.CPU63.RES:Rescheduling_interrupts
     64507           -27.6%      46697 ±  3%  interrupts.CPU63.TLB:TLB_shootdowns
     65889           -26.2%      48621 ±  3%  interrupts.CPU64.CAL:Function_call_interrupts
      5400           -39.5%       3269 ± 29%  interrupts.CPU64.NMI:Non-maskable_interrupts
      5400           -39.5%       3269 ± 29%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
      1793           -28.7%       1278 ± 12%  interrupts.CPU64.RES:Rescheduling_interrupts
     64030           -27.7%      46274 ±  3%  interrupts.CPU64.TLB:TLB_shootdowns
     65735           -26.0%      48634 ±  3%  interrupts.CPU65.CAL:Function_call_interrupts
      5423           -27.7%       3921 ±  4%  interrupts.CPU65.NMI:Non-maskable_interrupts
      5423           -27.7%       3921 ±  4%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
      1811           -28.8%       1289 ± 12%  interrupts.CPU65.RES:Rescheduling_interrupts
     63867           -27.6%      46233 ±  3%  interrupts.CPU65.TLB:TLB_shootdowns
     65912           -26.0%      48770 ±  3%  interrupts.CPU66.CAL:Function_call_interrupts
      5412           -27.4%       3931 ±  3%  interrupts.CPU66.NMI:Non-maskable_interrupts
      5412           -27.4%       3931 ±  3%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
      1842           -32.7%       1240 ± 10%  interrupts.CPU66.RES:Rescheduling_interrupts
     64131 ±  2%     -27.6%      46426 ±  3%  interrupts.CPU66.TLB:TLB_shootdowns
     65950           -25.9%      48866 ±  3%  interrupts.CPU67.CAL:Function_call_interrupts
      5438           -27.6%       3935 ±  4%  interrupts.CPU67.NMI:Non-maskable_interrupts
      5438           -27.6%       3935 ±  4%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      1771           -27.4%       1286 ±  4%  interrupts.CPU67.RES:Rescheduling_interrupts
     64164           -27.6%      46472 ±  3%  interrupts.CPU67.TLB:TLB_shootdowns
     65764           -26.0%      48659 ±  3%  interrupts.CPU68.CAL:Function_call_interrupts
      5403           -26.7%       3961 ±  5%  interrupts.CPU68.NMI:Non-maskable_interrupts
      5403           -26.7%       3961 ±  5%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
      1907 ±  3%     -37.0%       1201 ± 10%  interrupts.CPU68.RES:Rescheduling_interrupts
     63992           -27.5%      46365 ±  3%  interrupts.CPU68.TLB:TLB_shootdowns
     66484           -26.5%      48858 ±  3%  interrupts.CPU69.CAL:Function_call_interrupts
      5381           -26.5%       3956 ±  4%  interrupts.CPU69.NMI:Non-maskable_interrupts
      5381           -26.5%       3956 ±  4%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
      1790           -27.5%       1298 ± 12%  interrupts.CPU69.RES:Rescheduling_interrupts
     64630           -27.7%      46753 ±  3%  interrupts.CPU69.TLB:TLB_shootdowns
     65703           -26.2%      48519 ±  3%  interrupts.CPU7.CAL:Function_call_interrupts
      5441           -27.9%       3923 ±  4%  interrupts.CPU7.NMI:Non-maskable_interrupts
      5441           -27.9%       3923 ±  4%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      1734 ±  4%     -23.0%       1334 ±  5%  interrupts.CPU7.RES:Rescheduling_interrupts
     64011           -27.7%      46301 ±  3%  interrupts.CPU7.TLB:TLB_shootdowns
     66019 ±  2%     -25.8%      48978 ±  3%  interrupts.CPU70.CAL:Function_call_interrupts
      5363           -25.0%       4021 ±  4%  interrupts.CPU70.NMI:Non-maskable_interrupts
      5363           -25.0%       4021 ±  4%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
      1884 ±  4%     -32.5%       1271 ± 10%  interrupts.CPU70.RES:Rescheduling_interrupts
     64545           -27.8%      46613 ±  3%  interrupts.CPU70.TLB:TLB_shootdowns
     66233           -25.8%      49131 ±  3%  interrupts.CPU71.CAL:Function_call_interrupts
      5396           -26.8%       3950 ±  4%  interrupts.CPU71.NMI:Non-maskable_interrupts
      5396           -26.8%       3950 ±  4%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      1832 ±  2%     -29.9%       1284 ± 12%  interrupts.CPU71.RES:Rescheduling_interrupts
     64429           -27.5%      46740 ±  3%  interrupts.CPU71.TLB:TLB_shootdowns
     65787           -25.9%      48726 ±  3%  interrupts.CPU72.CAL:Function_call_interrupts
      5424           -27.7%       3923 ±  4%  interrupts.CPU72.NMI:Non-maskable_interrupts
      5424           -27.7%       3923 ±  4%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
      1819           -33.0%       1219 ±  8%  interrupts.CPU72.RES:Rescheduling_interrupts
     64134           -27.6%      46414 ±  3%  interrupts.CPU72.TLB:TLB_shootdowns
     65778           -25.9%      48724 ±  3%  interrupts.CPU73.CAL:Function_call_interrupts
      5390           -38.9%       3293 ± 30%  interrupts.CPU73.NMI:Non-maskable_interrupts
      5390           -38.9%       3293 ± 30%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
      1750           -29.6%       1231 ±  6%  interrupts.CPU73.RES:Rescheduling_interrupts
     64111           -27.4%      46542 ±  3%  interrupts.CPU73.TLB:TLB_shootdowns
     66248           -25.7%      49189 ±  3%  interrupts.CPU74.CAL:Function_call_interrupts
      5434           -38.2%       3357 ± 30%  interrupts.CPU74.NMI:Non-maskable_interrupts
      5434           -38.2%       3357 ± 30%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
      1867 ±  2%     -31.0%       1288 ± 12%  interrupts.CPU74.RES:Rescheduling_interrupts
     64530           -27.5%      46760 ±  3%  interrupts.CPU74.TLB:TLB_shootdowns
     66222           -26.0%      48973 ±  3%  interrupts.CPU75.CAL:Function_call_interrupts
      5429           -39.2%       3299 ± 31%  interrupts.CPU75.NMI:Non-maskable_interrupts
      5429           -39.2%       3299 ± 31%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
      1778 ±  2%     -27.0%       1297 ± 12%  interrupts.CPU75.RES:Rescheduling_interrupts
     64410           -27.6%      46609 ±  3%  interrupts.CPU75.TLB:TLB_shootdowns
     66150           -25.9%      49012 ±  3%  interrupts.CPU76.CAL:Function_call_interrupts
      5412           -38.7%       3319 ± 30%  interrupts.CPU76.NMI:Non-maskable_interrupts
      5412           -38.7%       3319 ± 30%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      1857           -26.1%       1372 ± 15%  interrupts.CPU76.RES:Rescheduling_interrupts
     64320           -27.5%      46624 ±  3%  interrupts.CPU76.TLB:TLB_shootdowns
     66204           -26.3%      48809 ±  3%  interrupts.CPU77.CAL:Function_call_interrupts
      5388           -38.0%       3341 ± 31%  interrupts.CPU77.NMI:Non-maskable_interrupts
      5388           -38.0%       3341 ± 31%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      1857           -30.8%       1284 ±  9%  interrupts.CPU77.RES:Rescheduling_interrupts
     64456           -27.5%      46731 ±  3%  interrupts.CPU77.TLB:TLB_shootdowns
     66314           -26.3%      48863 ±  3%  interrupts.CPU78.CAL:Function_call_interrupts
      5444           -37.1%       3426 ± 31%  interrupts.CPU78.NMI:Non-maskable_interrupts
      5444           -37.1%       3426 ± 31%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
      2201 ± 16%     -41.7%       1283 ± 13%  interrupts.CPU78.RES:Rescheduling_interrupts
     64578           -27.7%      46678 ±  3%  interrupts.CPU78.TLB:TLB_shootdowns
     66415           -26.2%      49019 ±  3%  interrupts.CPU79.CAL:Function_call_interrupts
      5418           -38.9%       3313 ± 31%  interrupts.CPU79.NMI:Non-maskable_interrupts
      5418           -38.9%       3313 ± 31%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      1812           -25.8%       1344 ± 10%  interrupts.CPU79.RES:Rescheduling_interrupts
     64616           -27.3%      46961 ±  3%  interrupts.CPU79.TLB:TLB_shootdowns
     66349           -26.2%      48969 ±  3%  interrupts.CPU8.CAL:Function_call_interrupts
      5396           -26.3%       3976 ±  3%  interrupts.CPU8.NMI:Non-maskable_interrupts
      5396           -26.3%       3976 ±  3%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      1785           -21.6%       1398 ±  6%  interrupts.CPU8.RES:Rescheduling_interrupts
     64528           -27.6%      46691 ±  3%  interrupts.CPU8.TLB:TLB_shootdowns
     65904           -26.1%      48700 ±  3%  interrupts.CPU80.CAL:Function_call_interrupts
      5440           -37.5%       3402 ± 31%  interrupts.CPU80.NMI:Non-maskable_interrupts
      5440           -37.5%       3402 ± 31%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      1785           -29.7%       1255 ± 12%  interrupts.CPU80.RES:Rescheduling_interrupts
     64103           -27.7%      46364 ±  3%  interrupts.CPU80.TLB:TLB_shootdowns
     65786           -26.4%      48419 ±  2%  interrupts.CPU81.CAL:Function_call_interrupts
      5449           -51.2%       2658 ± 34%  interrupts.CPU81.NMI:Non-maskable_interrupts
      5449           -51.2%       2658 ± 34%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      1736 ±  2%     -27.2%       1264 ±  7%  interrupts.CPU81.RES:Rescheduling_interrupts
     64037           -27.4%      46502 ±  3%  interrupts.CPU81.TLB:TLB_shootdowns
     65812           -26.2%      48562 ±  3%  interrupts.CPU82.CAL:Function_call_interrupts
      5457           -40.1%       3267 ± 24%  interrupts.CPU82.NMI:Non-maskable_interrupts
      5457           -40.1%       3267 ± 24%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
      1811           -31.7%       1237 ± 12%  interrupts.CPU82.RES:Rescheduling_interrupts
     64047           -27.7%      46291 ±  3%  interrupts.CPU82.TLB:TLB_shootdowns
     66346           -26.2%      48974 ±  3%  interrupts.CPU83.CAL:Function_call_interrupts
      5409           -39.8%       3254 ± 24%  interrupts.CPU83.NMI:Non-maskable_interrupts
      5409           -39.8%       3254 ± 24%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
      1809 ±  2%     -30.0%       1265 ±  8%  interrupts.CPU83.RES:Rescheduling_interrupts
     64452           -27.7%      46587 ±  3%  interrupts.CPU83.TLB:TLB_shootdowns
     66206           -25.7%      49188 ±  3%  interrupts.CPU84.CAL:Function_call_interrupts
      5412           -40.5%       3221 ± 24%  interrupts.CPU84.NMI:Non-maskable_interrupts
      5412           -40.5%       3221 ± 24%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
      1856 ±  2%     -30.4%       1292 ± 15%  interrupts.CPU84.RES:Rescheduling_interrupts
     64354           -27.3%      46811 ±  3%  interrupts.CPU84.TLB:TLB_shootdowns
     66350           -25.9%      49152 ±  3%  interrupts.CPU85.CAL:Function_call_interrupts
      5382           -39.8%       3240 ± 23%  interrupts.CPU85.NMI:Non-maskable_interrupts
      5382           -39.8%       3240 ± 23%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
      1836           -27.9%       1324 ± 12%  interrupts.CPU85.RES:Rescheduling_interrupts
     64547           -27.6%      46754 ±  3%  interrupts.CPU85.TLB:TLB_shootdowns
     65988           -26.3%      48657 ±  3%  interrupts.CPU86.CAL:Function_call_interrupts
      5399           -40.1%       3234 ± 24%  interrupts.CPU86.NMI:Non-maskable_interrupts
      5399           -40.1%       3234 ± 24%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
      1778 ±  3%     -29.2%       1258 ± 12%  interrupts.CPU86.RES:Rescheduling_interrupts
     64187           -27.8%      46370 ±  3%  interrupts.CPU86.TLB:TLB_shootdowns
     66011           -26.4%      48557 ±  3%  interrupts.CPU87.CAL:Function_call_interrupts
      5423           -39.9%       3257 ± 23%  interrupts.CPU87.NMI:Non-maskable_interrupts
      5423           -39.9%       3257 ± 23%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      2164 ± 16%     -41.3%       1270 ± 11%  interrupts.CPU87.RES:Rescheduling_interrupts
     64210           -27.9%      46323 ±  3%  interrupts.CPU87.TLB:TLB_shootdowns
     66283           -26.3%      48860 ±  3%  interrupts.CPU9.CAL:Function_call_interrupts
      5387           -24.7%       4054 ±  6%  interrupts.CPU9.NMI:Non-maskable_interrupts
      5387           -24.7%       4054 ±  6%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      1795 ±  4%     -26.9%       1312 ± 12%  interrupts.CPU9.RES:Rescheduling_interrupts
     64411           -27.6%      46655 ±  3%  interrupts.CPU9.TLB:TLB_shootdowns
    475926           -34.4%     312047 ± 10%  interrupts.NMI:Non-maskable_interrupts
    475926           -34.4%     312047 ± 10%  interrupts.PMI:Performance_monitoring_interrupts
    159423           -28.2%     114428 ± 10%  interrupts.RES:Rescheduling_interrupts
   5655945           -27.6%    4094298 ±  3%  interrupts.TLB:TLB_shootdowns





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.2.0-rc5-00071-gcf69482" of type "text/plain" (196393 bytes)

View attachment "job-script" of type "text/plain" (7514 bytes)

View attachment "job.yaml" of type "text/plain" (5130 bytes)

View attachment "reproduce" of type "text/plain" (310 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ