lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Sat, 31 Aug 2019 16:27:36 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Waiman Long <longman@...hat.com>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>,
        Will Deacon <will.deacon@....com>,
        huang ying <huang.ying.caritas@...il.com>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Borislav Petkov <bp@...en8.de>,
        Thomas Gleixner <tglx@...utronix.de>,
        Davidlohr Bueso <dave@...olabs.net>,
        "H. Peter Anvin" <hpa@...or.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Stephen Rothwell <sfr@...b.auug.org.au>, lkp@...org
Subject: [locking/rwsem]  91d2a812df:  will-it-scale.per_thread_ops 871.4%
 improvement

Greeting,

FYI, we noticed a 871.4% improvement of will-it-scale.per_thread_ops due to commit:


commit: 91d2a812dfb98b3b4dad661529c33bc38d303461 ("locking/rwsem: Make handoff writer optimistically spin on owner")
https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git master

in testcase: will-it-scale
on test machine: 104 threads Skylake with 192G memory
with following parameters:

	nr_task: 100%
	mode: thread
	test: malloc1
	cpufreq_governor: performance

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | reaim:                                                               |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | nr_task=900t                                                         |
|                  | runtime=300s                                                         |
|                  | test=disk_cp                                                         |
|                  | ucode=0x42e                                                          |
+------------------+----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-2019-05-14.cgz/lkp-skl-fpga01/malloc1/will-it-scale

commit: 
  8c779229d0 ("locking/lockdep: Report more stack trace statistics")
  91d2a812df ("locking/rwsem: Make handoff writer optimistically spin on owner")

8c779229d0f4fe83 91d2a812dfb98b3b4dad661529c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
          4:4         -100%            :4     kmsg.module:x86/modules:Skipping_invalid_relocation_target,existing_value_is_nonzero_for_type#,loc#,val
           :4            4%           0:4     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     15.75 ± 14%    +871.4%     153.00 ±  2%  will-it-scale.per_thread_ops
    131962 ±  8%     +14.4%     150940 ±  4%  will-it-scale.time.maximum_resident_set_size
    517607 ± 13%    +829.1%    4809315 ±  2%  will-it-scale.time.minor_page_faults
    126.75 ±  3%    +132.3%     294.50        will-it-scale.time.percent_of_cpu_this_job_got
    383.31 ±  3%    +129.4%     879.14        will-it-scale.time.system_time
      1.09 ± 15%    +790.6%       9.71 ±  9%  will-it-scale.time.user_time
    975972 ± 13%    +778.2%    8570839 ±  3%  will-it-scale.time.voluntary_context_switches
      1668 ± 14%    +857.2%      15969 ±  2%  will-it-scale.workload
      1.24 ±  2%      +1.5        2.74        mpstat.cpu.all.sys%
      0.03 ±  8%      +0.0        0.05 ±  6%  mpstat.cpu.all.usr%
    120616 ± 44%     +61.1%     194332 ± 11%  numa-meminfo.node0.Active
    120574 ± 44%     +61.1%     194218 ± 11%  numa-meminfo.node0.Active(anon)
    119210 ± 46%     +67.2%     199358 ± 11%  numa-meminfo.node0.AnonPages
    645009 ± 15%    +440.2%    3484401        numa-numastat.node0.local_node
    655058 ± 15%    +433.7%    3496130        numa-numastat.node0.numa_hit
    672506 ±  5%    +412.1%    3444198 ±  5%  numa-numastat.node1.local_node
    682632 ±  5%    +405.8%    3452614 ±  5%  numa-numastat.node1.numa_hit
     98.00            -1.0%      97.00        vmstat.cpu.id
      1.00          +125.0%       2.25 ± 19%  vmstat.procs.r
      7860 ± 10%    +632.9%      57610 ±  2%  vmstat.system.cs
    212249           +10.5%     234482 ±  3%  vmstat.system.in
  10481270 ± 14%  +15030.3%  1.586e+09 ±125%  cpuidle.C1.time
    351273 ± 13%   +3702.6%   13357426 ± 96%  cpuidle.C1.usage
 2.641e+10 ± 25%     -77.2%   6.03e+09 ± 40%  cpuidle.C1E.time
  57697113 ± 16%     -51.0%   28299437 ± 45%  cpuidle.C1E.usage
 4.261e+09 ±155%    +425.0%  2.237e+10 ±  4%  cpuidle.C6.time
    955286 ± 12%    +813.7%    8728274 ±  6%  cpuidle.POLL.time
    244692 ± 12%    +882.8%    2404855 ±  5%  cpuidle.POLL.usage
     30192 ± 44%     +60.6%      48483 ± 11%  numa-vmstat.node0.nr_active_anon
     29957 ± 46%     +66.5%      49880 ± 11%  numa-vmstat.node0.nr_anon_pages
     30192 ± 44%     +60.6%      48483 ± 11%  numa-vmstat.node0.nr_zone_active_anon
    939791 ±  9%    +142.6%    2280003 ±  4%  numa-vmstat.node0.numa_hit
    929623 ±  8%    +143.9%    2267785 ±  4%  numa-vmstat.node0.numa_local
    815710 ±  7%    +167.7%    2183596 ±  6%  numa-vmstat.node1.numa_hit
    637482 ±  8%    +214.9%    2007481 ±  6%  numa-vmstat.node1.numa_local
     84780            +2.2%      86660        proc-vmstat.nr_active_anon
     84230            +6.1%      89372        proc-vmstat.nr_anon_pages
    104.25            +2.4%     106.75        proc-vmstat.nr_anon_transparent_hugepages
      7928            +6.4%       8435        proc-vmstat.nr_shmem
     84780            +2.2%      86660        proc-vmstat.nr_zone_active_anon
   1363250 ±  6%    +411.1%    6968217 ±  3%  proc-vmstat.numa_hit
   1343062 ±  6%    +417.3%    6948062 ±  3%  proc-vmstat.numa_local
    959.25 ± 14%     +82.3%       1749 ±  2%  proc-vmstat.pgactivate
  12765687 ± 10%    +744.7%  1.078e+08 ±  7%  proc-vmstat.pgalloc_normal
   1303771 ±  5%    +328.5%    5586978 ±  2%  proc-vmstat.pgfault
  12737147 ± 10%    +746.4%  1.078e+08 ±  7%  proc-vmstat.pgfree
     22201 ± 11%    +788.5%     197266 ±  7%  proc-vmstat.thp_deferred_split_page
     22214 ± 11%    +788.1%     197279 ±  7%  proc-vmstat.thp_fault_alloc
     22179 ± 11%    +789.3%     197246 ±  7%  proc-vmstat.thp_split_pmd
     69.00           +76.4%     121.75        turbostat.Avg_MHz
      3.43 ±  3%      +1.8        5.19 ±  7%  turbostat.Busy%
      2026 ±  2%     +16.8%       2366 ±  7%  turbostat.Bzy_MHz
    349793 ± 13%   +3718.3%   13356194 ± 96%  turbostat.C1
      0.04 ± 14%      +5.0        5.04 ±125%  turbostat.C1%
  57693740 ± 16%     -51.0%   28298018 ± 45%  turbostat.C1E
     83.35 ± 25%     -64.2       19.15 ± 40%  turbostat.C1E%
     13.48 ±155%     +57.6       71.03 ±  4%  turbostat.C6%
      0.02 ± 79%  +30350.0%       6.09 ± 33%  turbostat.CPU%c6
  66247584           +28.1%   84848865 ±  2%  turbostat.IRQ
      0.00 ±173%    +2e+05%       4.91 ± 40%  turbostat.Pkg%pc2
     43.75            +5.7%      46.25 ±  2%  turbostat.PkgTmp
    190.53            +6.3%     202.55        turbostat.PkgWatt
     81.90           +15.6%      94.64 ±  3%  turbostat.RAMWatt
      1897 ±  2%    +125.2%       4274        sched_debug.cfs_rq:/.exec_clock.avg
    595.66 ± 19%    +346.3%       2658 ±  6%  sched_debug.cfs_rq:/.exec_clock.min
    531.92 ± 21%     -32.1%     361.25 ±  3%  sched_debug.cfs_rq:/.load_avg.max
     19490 ± 16%     +63.3%      31822 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
     12654 ± 16%     +74.8%      22118 ± 19%  sched_debug.cfs_rq:/.min_vruntime.min
      0.05 ± 16%     +26.1%       0.06 ±  6%  sched_debug.cfs_rq:/.nr_running.avg
    268.62 ±  3%     -29.8%     188.71 ±  3%  sched_debug.cfs_rq:/.runnable_load_avg.max
     33.10 ±  6%     -21.5%      25.98 ±  5%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
    135.82 ±  4%     +14.8%     155.98 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    129.99 ± 13%     +23.4%     160.37 ±  3%  sched_debug.cpu.curr->pid.avg
      0.05 ± 12%     +32.5%       0.06 ±  8%  sched_debug.cpu.nr_running.avg
      0.20 ±  7%     +16.6%       0.23 ±  6%  sched_debug.cpu.nr_running.stddev
     12840 ±  8%    +539.7%      82142 ±  3%  sched_debug.cpu.nr_switches.avg
     39710 ± 16%    +161.9%     104002 ±  3%  sched_debug.cpu.nr_switches.max
      9022 ±  9%    +709.7%      73053 ±  2%  sched_debug.cpu.nr_switches.min
      8.96 ± 12%     -24.4%       6.77 ± 17%  sched_debug.cpu.nr_uninterruptible.stddev
     10850 ± 10%    +649.5%      81326 ±  3%  sched_debug.cpu.sched_count.avg
     36450 ± 17%    +175.2%     100307 ±  4%  sched_debug.cpu.sched_count.max
      7737 ± 11%    +838.0%      72573 ±  2%  sched_debug.cpu.sched_count.min
      5384 ± 10%    +647.0%      40224 ±  3%  sched_debug.cpu.sched_goidle.avg
     18181 ± 17%    +173.5%      49720 ±  4%  sched_debug.cpu.sched_goidle.max
      3828 ± 11%    +837.0%      35873 ±  2%  sched_debug.cpu.sched_goidle.min
      5481 ± 10%    +696.7%      43672 ±  3%  sched_debug.cpu.ttwu_count.avg
     25509 ±  8%    +119.6%      56014 ±  6%  sched_debug.cpu.ttwu_count.max
      4060 ± 12%    +871.4%      39445 ±  2%  sched_debug.cpu.ttwu_count.min
    507.86           -23.2%     390.09        sched_debug.cpu.ttwu_local.avg
      2779 ± 21%     -57.5%       1182 ±  9%  sched_debug.cpu.ttwu_local.max
    458.45 ± 13%     -61.6%     175.87 ±  9%  sched_debug.cpu.ttwu_local.stddev
      3.69          +233.2%      12.29 ± 17%  perf-stat.i.MPKI
 4.462e+08 ±  2%     +39.5%  6.224e+08        perf-stat.i.branch-instructions
      1.15 ±  3%      +0.6        1.76 ± 12%  perf-stat.i.branch-miss-rate%
   5583034 ±  6%    +100.9%   11217217 ± 11%  perf-stat.i.branch-misses
   1012498 ± 20%    +370.8%    4766910 ±  2%  perf-stat.i.cache-misses
   6528605          +406.8%   33084508 ± 17%  perf-stat.i.cache-references
      7877 ± 10%    +635.9%      57976 ±  2%  perf-stat.i.context-switches
      3.58           +21.4%       4.34        perf-stat.i.cpi
  6.25e+09 ±  2%     +87.6%  1.172e+10        perf-stat.i.cpu-cycles
      3.32 ±  6%    +197.0%       9.85 ±  9%  perf-stat.i.cpu-migrations
      6920 ± 17%     -62.3%       2608 ±  4%  perf-stat.i.cycles-between-cache-misses
      0.10 ± 54%      +0.2        0.32 ±  8%  perf-stat.i.dTLB-load-miss-rate%
    380489 ± 52%    +487.4%    2235106 ±  9%  perf-stat.i.dTLB-load-misses
 3.969e+08           +75.5%  6.965e+08        perf-stat.i.dTLB-loads
      0.03 ± 56%      +0.1        0.09 ±  9%  perf-stat.i.dTLB-store-miss-rate%
     45271 ± 55%    +465.3%     255903 ± 10%  perf-stat.i.dTLB-store-misses
 1.752e+08           +58.1%  2.769e+08 ±  2%  perf-stat.i.dTLB-stores
      5.66 ± 62%     +13.6       19.26 ±  9%  perf-stat.i.iTLB-load-miss-rate%
     87525 ± 62%    +399.4%     437104 ± 10%  perf-stat.i.iTLB-load-misses
   1445519 ±  4%     +25.6%    1815215 ±  3%  perf-stat.i.iTLB-loads
 1.775e+09 ±  2%     +54.3%  2.738e+09        perf-stat.i.instructions
     29573 ± 38%     -77.6%       6632 ± 13%  perf-stat.i.instructions-per-iTLB-miss
      0.28           -17.7%       0.23        perf-stat.i.ipc
      4203 ±  5%    +338.0%      18409 ±  2%  perf-stat.i.minor-faults
    195759 ±  8%    +513.6%    1201122 ±  2%  perf-stat.i.node-load-misses
     24687 ± 12%    +445.8%     134738 ±  4%  perf-stat.i.node-loads
     73.77            +5.3       79.08 ±  2%  perf-stat.i.node-store-miss-rate%
     43851 ± 12%    +645.1%     326744 ±  3%  perf-stat.i.node-store-misses
     17114 ±  6%    +395.7%      84840 ±  9%  perf-stat.i.node-stores
      4203 ±  5%    +338.0%      18409 ±  2%  perf-stat.i.page-faults
      3.68          +228.3%      12.08 ± 17%  perf-stat.overall.MPKI
      1.25 ±  4%      +0.6        1.80 ± 11%  perf-stat.overall.branch-miss-rate%
      3.52           +21.6%       4.28        perf-stat.overall.cpi
      6361 ± 14%     -61.3%       2461 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.10 ± 52%      +0.2        0.32 ±  8%  perf-stat.overall.dTLB-load-miss-rate%
      0.03 ± 55%      +0.1        0.09 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
      5.69 ± 61%     +13.7       19.38 ±  9%  perf-stat.overall.iTLB-load-miss-rate%
     26147 ± 35%     -75.7%       6345 ± 11%  perf-stat.overall.instructions-per-iTLB-miss
      0.28           -17.8%       0.23        perf-stat.overall.ipc
     88.83            +1.1       89.91        perf-stat.overall.node-load-miss-rate%
     71.77 ±  2%      +7.6       79.38 ±  2%  perf-stat.overall.node-store-miss-rate%
 3.307e+08 ± 17%     -84.4%   51665467        perf-stat.overall.path-length
 4.448e+08 ±  2%     +39.5%  6.204e+08        perf-stat.ps.branch-instructions
   5569186 ±  6%    +100.8%   11184530 ± 11%  perf-stat.ps.branch-misses
   1009315 ± 20%    +370.7%    4751087 ±  2%  perf-stat.ps.cache-misses
   6508218          +406.7%   32975731 ± 17%  perf-stat.ps.cache-references
      7851 ± 10%    +635.9%      57779 ±  2%  perf-stat.ps.context-switches
 6.229e+09 ±  2%     +87.5%  1.168e+10        perf-stat.ps.cpu-cycles
      3.32 ±  6%    +196.4%       9.83 ±  9%  perf-stat.ps.cpu-migrations
    379246 ± 52%    +487.4%    2227615 ±  9%  perf-stat.ps.dTLB-load-misses
 3.957e+08           +75.5%  6.943e+08        perf-stat.ps.dTLB-loads
     45127 ± 55%    +465.2%     255052 ± 10%  perf-stat.ps.dTLB-store-misses
 1.747e+08           +58.0%   2.76e+08 ±  2%  perf-stat.ps.dTLB-stores
     87243 ± 62%    +399.3%     435646 ± 10%  perf-stat.ps.iTLB-load-misses
   1440695 ±  4%     +25.6%    1809091 ±  3%  perf-stat.ps.iTLB-loads
  1.77e+09 ±  2%     +54.3%   2.73e+09        perf-stat.ps.instructions
      4189 ±  5%    +337.9%      18348 ±  2%  perf-stat.ps.minor-faults
    195127 ±  8%    +513.5%    1197075 ±  2%  perf-stat.ps.node-load-misses
     24616 ± 12%    +445.5%     134297 ±  4%  perf-stat.ps.node-loads
     43710 ± 12%    +645.0%     325645 ±  3%  perf-stat.ps.node-store-misses
     17070 ±  6%    +395.4%      84569 ±  9%  perf-stat.ps.node-stores
      4190 ±  5%    +337.9%      18348 ±  2%  perf-stat.ps.page-faults
 5.377e+11 ±  2%     +53.4%  8.247e+11        perf-stat.total.instructions
      7973 ±  4%     +83.5%      14635 ± 17%  softirqs.CPU0.RCU
      8539 ± 11%     +66.4%      14210 ±  5%  softirqs.CPU1.RCU
      7894 ±  5%     +71.0%      13498 ±  4%  softirqs.CPU10.RCU
      8535 ±  5%     +52.1%      12984 ±  7%  softirqs.CPU100.RCU
      7662 ±  7%     +66.6%      12766 ±  8%  softirqs.CPU101.RCU
      7906 ±  6%     +66.3%      13149 ± 10%  softirqs.CPU102.RCU
      7931 ±  8%     +67.9%      13318 ±  9%  softirqs.CPU103.RCU
      8005 ±  6%     +75.8%      14076 ±  6%  softirqs.CPU11.RCU
      8099 ±  4%     +70.6%      13814 ±  5%  softirqs.CPU12.RCU
      8080 ±  7%     +68.2%      13592 ±  4%  softirqs.CPU13.RCU
      7832 ±  6%     +69.7%      13295 ±  3%  softirqs.CPU14.RCU
      7932 ±  7%     +72.7%      13696 ±  6%  softirqs.CPU15.RCU
      8192 ±  5%     +67.1%      13684 ±  4%  softirqs.CPU16.RCU
      8290 ± 12%     +67.3%      13868 ±  3%  softirqs.CPU17.RCU
      8150 ±  7%     +70.4%      13891 ±  4%  softirqs.CPU18.RCU
      8191 ±  8%     +70.0%      13922 ±  5%  softirqs.CPU19.RCU
      8149 ± 10%     +85.3%      15102 ± 16%  softirqs.CPU2.RCU
      8109 ± 10%     +65.4%      13410 ± 10%  softirqs.CPU20.RCU
      7951 ±  9%     +76.6%      14043        softirqs.CPU21.RCU
      7898 ±  8%     +74.6%      13791 ±  3%  softirqs.CPU22.RCU
      7963 ±  6%     +71.4%      13652 ±  4%  softirqs.CPU23.RCU
      7816 ±  5%     +76.8%      13821 ±  6%  softirqs.CPU24.RCU
      7880 ±  5%     +73.7%      13691 ±  4%  softirqs.CPU25.RCU
      7857 ±  9%     +73.6%      13639 ±  9%  softirqs.CPU26.RCU
      6680 ±  5%     +81.6%      12130 ±  6%  softirqs.CPU27.RCU
      7005 ±  6%     +75.3%      12277 ±  5%  softirqs.CPU28.RCU
      6965 ±  4%     +75.2%      12200 ±  5%  softirqs.CPU29.RCU
     12072 ± 18%     +31.5%      15877 ± 11%  softirqs.CPU3.RCU
      9192 ± 10%     +52.1%      13979 ±  8%  softirqs.CPU30.RCU
      8752 ±  3%     +64.1%      14361 ±  8%  softirqs.CPU31.RCU
      8694 ±  5%     +62.0%      14083 ±  9%  softirqs.CPU32.RCU
      8645 ±  7%     +59.6%      13797 ±  7%  softirqs.CPU33.RCU
      8309 ±  8%     +60.0%      13292 ± 11%  softirqs.CPU34.RCU
      8595 ±  7%     +64.2%      14111 ±  9%  softirqs.CPU35.RCU
      8413 ±  6%     +66.2%      13985 ±  9%  softirqs.CPU36.RCU
      8790 ±  9%     +59.7%      14037 ±  8%  softirqs.CPU37.RCU
      8954 ±  9%     +56.9%      14054 ±  8%  softirqs.CPU38.RCU
      8221 ±  7%     +63.6%      13448 ± 10%  softirqs.CPU39.RCU
      8165 ±  6%     +71.4%      13990 ±  6%  softirqs.CPU4.RCU
      8582 ±  8%     +57.2%      13491 ±  7%  softirqs.CPU40.RCU
      8326 ±  5%     +65.7%      13796 ±  7%  softirqs.CPU41.RCU
      8837 ±  5%     +58.1%      13971 ±  8%  softirqs.CPU42.RCU
      8709 ±  7%     +65.3%      14393 ±  6%  softirqs.CPU43.RCU
      8692 ±  6%     +63.4%      14207 ±  9%  softirqs.CPU44.RCU
      7741 ±  5%     +71.4%      13270 ±  6%  softirqs.CPU45.RCU
      7487 ±  5%     +72.5%      12912 ±  6%  softirqs.CPU46.RCU
      7558 ±  8%     +76.0%      13302 ±  7%  softirqs.CPU47.RCU
      7827 ±  9%     +68.5%      13187 ±  7%  softirqs.CPU48.RCU
      7491 ±  8%     +72.9%      12951 ±  5%  softirqs.CPU49.RCU
      8244 ±  8%     +67.3%      13790 ±  4%  softirqs.CPU5.RCU
      7917 ±  8%     +65.3%      13088 ±  5%  softirqs.CPU50.RCU
      7563 ±  7%     +74.4%      13194 ±  6%  softirqs.CPU51.RCU
      7242 ±  5%     +79.8%      13019 ± 11%  softirqs.CPU52.RCU
      7359 ±  7%     +78.7%      13148 ±  4%  softirqs.CPU53.RCU
      7996 ± 20%     +67.0%      13357 ±  5%  softirqs.CPU54.RCU
      7396 ±  8%     +79.2%      13257 ±  5%  softirqs.CPU55.RCU
      7559 ±  7%     +82.0%      13756 ±  4%  softirqs.CPU56.RCU
      7609 ±  5%     +74.9%      13310 ±  4%  softirqs.CPU57.RCU
      7512 ±  6%     +79.8%      13509 ±  5%  softirqs.CPU58.RCU
      7255 ±  5%     +80.5%      13093 ±  8%  softirqs.CPU59.RCU
      8639 ±  8%     +61.8%      13975 ±  2%  softirqs.CPU6.RCU
      7907 ±  7%     +66.3%      13152 ±  3%  softirqs.CPU60.RCU
      7747 ±  6%     +69.9%      13164 ±  2%  softirqs.CPU61.RCU
      7728 ±  4%     +68.8%      13047 ±  5%  softirqs.CPU62.RCU
      7721 ±  5%     +71.8%      13266 ±  4%  softirqs.CPU63.RCU
      7746 ±  3%     +69.4%      13120 ±  6%  softirqs.CPU64.RCU
      7714 ±  7%     +70.6%      13158 ±  4%  softirqs.CPU65.RCU
      7864 ±  9%     +63.5%      12855 ±  4%  softirqs.CPU66.RCU
      7854 ±  6%     +65.4%      12993 ±  8%  softirqs.CPU67.RCU
      7876 ±  6%     +67.3%      13173 ±  5%  softirqs.CPU68.RCU
      7771 ±  7%     +69.2%      13152 ±  4%  softirqs.CPU69.RCU
      8007 ±  6%     +68.6%      13503 ±  5%  softirqs.CPU7.RCU
      7955 ± 10%     +65.3%      13148 ±  4%  softirqs.CPU70.RCU
      7780 ±  7%     +78.7%      13907 ±  4%  softirqs.CPU71.RCU
      7709 ±  6%     +78.7%      13779 ±  7%  softirqs.CPU72.RCU
      7985 ±  6%     +64.0%      13093 ±  3%  softirqs.CPU73.RCU
      7579 ±  8%     +70.2%      12903 ±  4%  softirqs.CPU74.RCU
      6424 ±  8%     +81.4%      11654 ±  4%  softirqs.CPU75.RCU
      6321 ±  7%     +84.0%      11633 ±  5%  softirqs.CPU76.RCU
      6286 ±  6%     +84.1%      11576 ±  4%  softirqs.CPU77.RCU
      7791 ±  8%     +73.6%      13528 ±  5%  softirqs.CPU78.RCU
    151850 ±  2%     -27.1%     110672 ± 13%  softirqs.CPU78.TIMER
      7640 ±  4%     +72.5%      13182 ±  7%  softirqs.CPU79.RCU
      7857 ±  5%     +73.0%      13594 ±  6%  softirqs.CPU8.RCU
      8086 ±  5%     +70.2%      13766 ±  8%  softirqs.CPU80.RCU
      8144 ±  7%     +68.3%      13707 ±  7%  softirqs.CPU81.RCU
      8165 ±  7%     +68.2%      13737 ±  7%  softirqs.CPU82.RCU
      8352 ±  6%     +65.7%      13837 ±  7%  softirqs.CPU83.RCU
      8096 ±  6%     +68.6%      13649 ±  8%  softirqs.CPU84.RCU
      8090 ±  8%     +67.0%      13507 ±  7%  softirqs.CPU85.RCU
      7735 ±  8%     +74.1%      13464 ±  8%  softirqs.CPU86.RCU
      7982 ±  7%     +66.7%      13305 ± 13%  softirqs.CPU87.RCU
      8543 ± 16%     +59.8%      13651 ±  8%  softirqs.CPU88.RCU
      7962 ± 10%     +71.2%      13632 ±  7%  softirqs.CPU89.RCU
      8018 ±  5%     +76.0%      14114 ±  5%  softirqs.CPU9.RCU
      8007 ±  7%     +66.7%      13351 ±  8%  softirqs.CPU90.RCU
      7919 ±  7%     +58.4%      12546 ± 10%  softirqs.CPU91.RCU
      7896 ±  7%     +58.3%      12501 ±  6%  softirqs.CPU92.RCU
      7759 ±  5%     +64.3%      12746 ±  8%  softirqs.CPU93.RCU
      8152 ±  5%     +57.6%      12844 ±  8%  softirqs.CPU94.RCU
      7920 ±  4%     +67.0%      13225 ±  8%  softirqs.CPU95.RCU
      7985 ±  6%     +65.9%      13250 ±  9%  softirqs.CPU96.RCU
      8052 ±  7%     +64.1%      13211 ±  9%  softirqs.CPU97.RCU
      7958 ±  7%     +63.2%      12985 ±  9%  softirqs.CPU98.RCU
      7888 ±  7%     +64.6%      12984 ±  8%  softirqs.CPU99.RCU
    829865 ±  4%     +68.4%    1397799 ±  5%  softirqs.RCU
     26.29 ±  3%     -16.0       10.27 ±  9%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault
     25.38 ±  3%     -15.5        9.86 ±  9%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault
     26.77 ±  3%     -13.7       13.11 ±  5%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
     26.77 ±  3%     -13.7       13.12 ±  5%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
     26.77 ±  3%     -13.6       13.13 ±  5%  perf-profile.calltrace.cycles-pp.page_fault
     67.66            -7.0       60.61        perf-profile.calltrace.cycles-pp.secondary_startup_64
     67.05            -7.0       60.01        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     67.05            -7.0       60.01        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     67.01            -7.0       59.98        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     17.25            -6.1       11.13 ±  4%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     59.00            -5.5       53.53        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     58.46            -5.2       53.23        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     14.04            -4.7        9.38 ±  4%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      8.55 ±  4%      -2.7        5.81 ±  6%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      6.03 ± 11%      -1.9        4.09 ±  3%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.43 ±  4%      -1.9        0.58 ±  8%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault
      4.01 ±  6%      -1.5        2.47 ±  7%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      4.26 ±  6%      -1.5        2.79 ±  9%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      3.44 ±  8%      -1.4        2.06 ±  8%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      4.25 ±  4%      -1.2        3.01 ±  6%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      2.94 ±  2%      -0.9        2.05 ±  8%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      2.62 ±  5%      -0.7        1.89 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      2.34 ±  9%      -0.7        1.63 ±  8%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.51 ±  4%      -0.6        0.90 ± 17%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.67 ±  6%      -0.6        1.06 ±  2%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      1.48 ±  4%      -0.6        0.89 ± 16%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      1.21 ±  3%      -0.6        0.65 ±  7%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.95 ±  3%      -0.5        0.40 ± 58%  perf-profile.calltrace.cycles-pp.native_write_msr.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt
      1.12 ± 11%      -0.5        0.61 ± 14%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      1.15 ± 17%      -0.5        0.67 ± 22%  perf-profile.calltrace.cycles-pp.timekeeping_max_deferment.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.87 ± 12%      -0.4        0.43 ± 58%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.02 ±  8%      -0.4        0.64        perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
      1.61 ±  3%      -0.3        1.27 ±  5%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.91 ±  5%      -0.3        0.57 ±  7%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.53 ±  2%      -0.3        1.21 ±  6%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.62 ±  5%      -0.3        1.32 ± 12%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.65 ± 11%      -0.3        0.39 ± 57%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.6        0.56 ± 11%  perf-profile.calltrace.cycles-pp.__split_vma.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      0.00            +0.7        0.73 ±  2%  perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      0.00            +0.8        0.76 ± 12%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap
      0.00            +0.8        0.77 ± 12%  perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.00            +0.8        0.84 ± 31%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
      0.00            +0.9        0.90 ±  4%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap
      0.00            +0.9        0.92 ±  4%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.76 ± 61%      +1.0        1.74 ± 19%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
      0.82 ± 60%      +1.3        2.07 ± 16%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      0.83 ± 60%      +1.3        2.10 ± 15%  perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.7        1.68 ±  8%  perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault
      0.00            +1.9        1.90 ±  7%  perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      1.09 ± 29%      +1.9        3.00 ± 13%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.09 ± 30%      +1.9        3.00 ± 13%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.0        1.96 ±  8%  perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.00            +2.1        2.12 ±  8%  perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
      0.00            +2.2        2.25 ±  8%  perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.00            +2.5        2.48 ±  7%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.00            +2.5        2.50 ±  7%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.12 ±173%      +2.7        2.78 ±  8%  perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.58 ± 19%     +14.6       17.16 ±  6%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
      2.75 ± 15%     +15.4       18.17 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
      2.83 ± 15%     +15.9       18.74 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
      2.84 ± 15%     +15.9       18.79 ±  5%  perf-profile.calltrace.cycles-pp.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.32 ± 12%     +18.7       22.07 ±  4%  perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.32 ± 12%     +18.8       22.07 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.54 ± 16%     +20.7       25.23 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.54 ± 16%     +20.7       25.23 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     26.29 ±  3%     -16.0       10.27 ±  9%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
     26.80 ±  3%     -13.7       13.13 ±  5%  perf-profile.children.cycles-pp.__do_page_fault
     26.81 ±  3%     -13.7       13.14 ±  5%  perf-profile.children.cycles-pp.page_fault
     26.80 ±  3%     -13.7       13.14 ±  5%  perf-profile.children.cycles-pp.do_page_fault
     67.72            -7.1       60.65        perf-profile.children.cycles-pp.do_idle
     67.66            -7.0       60.61        perf-profile.children.cycles-pp.secondary_startup_64
     67.66            -7.0       60.61        perf-profile.children.cycles-pp.cpu_startup_entry
     67.05            -7.0       60.01        perf-profile.children.cycles-pp.start_secondary
     16.05            -5.5       10.59 ±  4%  perf-profile.children.cycles-pp.apic_timer_interrupt
     59.53            -5.5       54.07        perf-profile.children.cycles-pp.cpuidle_enter
     59.49            -5.4       54.06        perf-profile.children.cycles-pp.cpuidle_enter_state
     14.35            -4.7        9.64 ±  4%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      8.82 ±  3%      -2.8        6.02 ±  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      6.11 ± 11%      -2.0        4.15 ±  3%  perf-profile.children.cycles-pp.menu_select
      4.06 ±  6%      -1.6        2.51 ±  6%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      4.33 ±  6%      -1.5        2.86 ±  9%  perf-profile.children.cycles-pp.irq_exit
      4.40 ±  4%      -1.4        2.96 ±  9%  perf-profile.children.cycles-pp.ktime_get
      3.49 ±  8%      -1.4        2.10 ±  7%  perf-profile.children.cycles-pp.tick_nohz_next_event
      4.42 ±  3%      -1.3        3.14 ±  5%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      3.00 ±  2%      -0.9        2.12 ±  8%  perf-profile.children.cycles-pp.clockevents_program_event
      2.74 ±  3%      -0.8        1.98 ±  8%  perf-profile.children.cycles-pp.tick_sched_timer
      2.38 ±  8%      -0.7        1.67 ±  8%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.70 ±  6%      -0.6        1.08 ±  3%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      1.53 ±  4%      -0.6        0.92 ± 16%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      1.31 ±  4%      -0.6        0.75 ±  5%  perf-profile.children.cycles-pp.native_write_msr
      1.24 ±  4%      -0.6        0.68 ±  5%  perf-profile.children.cycles-pp.lapic_next_deadline
      1.12 ± 11%      -0.5        0.63 ± 14%  perf-profile.children.cycles-pp.rebalance_domains
      1.16 ± 17%      -0.5        0.67 ± 22%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      1.07 ±  5%      -0.5        0.61 ±  6%  perf-profile.children.cycles-pp.read_tsc
      1.18 ±  8%      -0.5        0.72        perf-profile.children.cycles-pp.__next_timer_interrupt
      1.15 ±  4%      -0.4        0.71 ±  4%  perf-profile.children.cycles-pp.native_irq_return_iret
      1.84 ±  2%      -0.4        1.43 ±  3%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.65 ± 15%      -0.4        0.29 ±  4%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      1.70 ±  4%      -0.4        1.34 ±  5%  perf-profile.children.cycles-pp.tick_sched_handle
      1.64 ±  4%      -0.3        1.29 ±  5%  perf-profile.children.cycles-pp.update_process_times
      0.94 ±  6%      -0.3        0.60 ±  7%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.89 ± 12%      -0.3        0.56 ± 15%  perf-profile.children.cycles-pp.irq_enter
      0.68 ±  7%      -0.3        0.39 ± 13%  perf-profile.children.cycles-pp.load_balance
      0.73 ± 13%      -0.3        0.47 ± 17%  perf-profile.children.cycles-pp.tick_irq_enter
      0.50 ± 11%      -0.2        0.26 ± 19%  perf-profile.children.cycles-pp.find_busiest_group
      0.69 ±  7%      -0.2        0.46 ±  2%  perf-profile.children.cycles-pp.find_next_bit
      0.46 ± 11%      -0.2        0.23 ± 18%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.64 ±  8%      -0.2        0.43 ±  6%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.36 ± 11%      -0.2        0.21 ±  9%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.50 ± 10%      -0.1        0.36 ±  8%  perf-profile.children.cycles-pp.sched_clock
      0.49 ±  8%      -0.1        0.35 ±  5%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.48 ± 10%      -0.1        0.34 ±  8%  perf-profile.children.cycles-pp.native_sched_clock
      0.69 ± 10%      -0.1        0.56 ±  5%  perf-profile.children.cycles-pp.scheduler_tick
      0.32 ±  3%      -0.1        0.20 ± 12%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.22 ± 47%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.__GI___libc_read
      0.22 ± 46%      -0.1        0.10 ±  7%  perf-profile.children.cycles-pp.__libc_start_main
      0.29 ± 14%      -0.1        0.18 ± 10%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.30 ±  7%      -0.1        0.18 ± 11%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.28 ± 10%      -0.1        0.16 ±  5%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.57 ±  9%      -0.1        0.46 ±  8%  perf-profile.children.cycles-pp.__lock_text_start
      0.28 ± 12%      -0.1        0.17 ± 11%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.37 ± 13%      -0.1        0.26 ±  2%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.21 ± 11%      -0.1        0.11 ±  9%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.35 ± 12%      -0.1        0.26 ± 18%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.23 ± 16%      -0.1        0.14 ±  8%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.16 ± 57%      -0.1        0.07 ± 17%  perf-profile.children.cycles-pp.perf_read
      0.34 ±  8%      -0.1        0.25 ±  4%  perf-profile.children.cycles-pp.run_timer_softirq
      0.21 ± 18%      -0.1        0.13 ± 11%  perf-profile.children.cycles-pp.menu_reflect
      0.15 ± 13%      -0.1        0.07 ± 22%  perf-profile.children.cycles-pp.cpumask_next_and
      0.11 ± 16%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.find_next_and_bit
      0.17 ± 17%      -0.1        0.10 ± 13%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.18 ± 10%      -0.1        0.11 ± 17%  perf-profile.children.cycles-pp.rcu_irq_exit
      0.20 ± 17%      -0.1        0.13 ±  8%  perf-profile.children.cycles-pp.interrupt_entry
      0.12 ± 63%      -0.1        0.06 ± 15%  perf-profile.children.cycles-pp.__perf_event_read_value
      0.29 ± 12%      -0.1        0.22 ± 17%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.10 ± 22%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.15 ±  7%      -0.1        0.10 ± 19%  perf-profile.children.cycles-pp.call_cpuidle
      0.12 ± 11%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.39 ±  5%      -0.0        0.34 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.15 ± 16%      -0.0        0.10 ± 17%  perf-profile.children.cycles-pp.get_cpu_device
      0.11 ± 17%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.11 ± 10%      -0.0        0.07 ± 22%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.20 ±  7%      -0.0        0.16 ±  9%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.12 ± 12%      -0.0        0.09 ± 16%  perf-profile.children.cycles-pp.trigger_load_balance
      0.08 ±  6%      -0.0        0.05 ±  9%  perf-profile.children.cycles-pp.rb_insert_color
      0.08 ± 15%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.process_one_work
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.free_pgd_range
      0.00            +0.1        0.06 ± 22%  perf-profile.children.cycles-pp.leave_mm
      0.00            +0.1        0.06 ± 20%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.update_cfs_group
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.osq_unlock
      0.00            +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.__default_send_IPI_dest_field
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.llist_add_batch
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.anon_vma_interval_tree_remove
      0.00            +0.1        0.07 ± 36%  perf-profile.children.cycles-pp.up_read
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.default_send_IPI_mask_sequence_phys
      0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.00            +0.1        0.07 ± 30%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.native_send_call_func_ipi
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.down_write
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.__switch_to
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.rcu_all_qs
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.deferred_split_huge_page
      0.00            +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.up_write
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.lru_add_drain
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.00            +0.1        0.09 ± 12%  perf-profile.children.cycles-pp.load_new_mm_cr3
      0.00            +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.wake_q_add
      0.00            +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.set_next_entity
      0.04 ± 58%      +0.1        0.14 ± 47%  perf-profile.children.cycles-pp.smp_call_function_single_interrupt
      0.14 ± 47%      +0.1        0.25 ± 14%  perf-profile.children.cycles-pp.smp_call_function_single
      0.00            +0.1        0.11 ± 12%  perf-profile.children.cycles-pp.page_remove_rmap
      0.00            +0.1        0.11 ± 10%  perf-profile.children.cycles-pp.update_curr
      0.07 ±  6%      +0.1        0.18 ± 37%  perf-profile.children.cycles-pp.call_function_single_interrupt
      0.00            +0.1        0.12 ±  9%  perf-profile.children.cycles-pp.downgrade_write
      0.00            +0.1        0.12 ±  3%  perf-profile.children.cycles-pp._cond_resched
      0.03 ±100%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.00            +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.unmapped_area_topdown
      0.00            +0.1        0.12 ±  8%  perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.54 ± 11%      +0.1        0.67 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.13 ± 17%  perf-profile.children.cycles-pp.__split_huge_pmd_locked
      0.00            +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.00            +0.1        0.14 ± 16%  perf-profile.children.cycles-pp.___might_sleep
      0.00            +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.finish_task_switch
      0.01 ±173%      +0.1        0.16 ± 11%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.00            +0.2        0.16 ± 15%  perf-profile.children.cycles-pp.get_unmapped_area
      0.01 ±173%      +0.2        0.17 ± 13%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.01 ±173%      +0.2        0.18 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.2        0.17 ± 14%  perf-profile.children.cycles-pp.do_anonymous_page
      0.01 ±173%      +0.2        0.18 ± 16%  perf-profile.children.cycles-pp.find_vma
      0.00            +0.2        0.17 ± 11%  perf-profile.children.cycles-pp.anon_vma_clone
      0.00            +0.2        0.17 ± 13%  perf-profile.children.cycles-pp.__split_huge_pmd
      0.06 ± 17%      +0.2        0.23 ±  8%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.2        0.17 ± 11%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.01 ±173%      +0.2        0.19 ±  5%  perf-profile.children.cycles-pp.flush_tlb_func_common
      0.00            +0.2        0.18 ± 15%  perf-profile.children.cycles-pp.release_pages
      0.00            +0.2        0.19 ±  6%  perf-profile.children.cycles-pp.dequeue_entity
      0.00            +0.2        0.20 ± 15%  perf-profile.children.cycles-pp.vma_merge
      0.00            +0.2        0.20 ± 13%  perf-profile.children.cycles-pp.vma_adjust_trans_huge
      0.01 ±173%      +0.2        0.21 ±  9%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.01 ±173%      +0.2        0.23 ± 13%  perf-profile.children.cycles-pp.unwind_next_frame
      0.03 ±100%      +0.3        0.29 ±  5%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.08 ±  8%      +0.3        0.36 ± 10%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.06 ± 58%      +0.3        0.34 ± 12%  perf-profile.children.cycles-pp.arch_stack_walk
      0.05 ± 71%      +0.3        0.34 ± 10%  perf-profile.children.cycles-pp.mmap_region
      0.01 ±173%      +0.3        0.32 ±  5%  perf-profile.children.cycles-pp.smp_call_function_many
      0.08 ± 19%      +0.3        0.39 ± 13%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.04 ± 57%      +0.3        0.38 ±  5%  perf-profile.children.cycles-pp.on_each_cpu_mask
      0.01 ±173%      +0.3        0.35 ± 15%  perf-profile.children.cycles-pp.smp_call_function_interrupt
      0.10 ± 22%      +0.4        0.48 ± 13%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.07 ± 27%      +0.4        0.47 ± 13%  perf-profile.children.cycles-pp.poll_idle
      0.07 ± 14%      +0.4        0.47 ±  4%  perf-profile.children.cycles-pp.on_each_cpu_cond_mask
      0.04 ±102%      +0.4        0.44 ± 15%  perf-profile.children.cycles-pp.call_function_interrupt
      0.08 ±  5%      +0.4        0.50        perf-profile.children.cycles-pp.schedule_idle
      0.08 ± 66%      +0.4        0.51 ± 11%  perf-profile.children.cycles-pp.do_mmap
      0.08 ± 60%      +0.5        0.53 ± 13%  perf-profile.children.cycles-pp.__vma_adjust
      0.11 ±  8%      +0.5        0.57 ±  7%  perf-profile.children.cycles-pp.schedule
      0.08 ± 59%      +0.5        0.56 ± 11%  perf-profile.children.cycles-pp.__split_vma
      0.08 ± 11%      +0.5        0.56 ± 10%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.14 ± 18%      +0.6        0.71 ± 10%  perf-profile.children.cycles-pp.enqueue_entity
      0.14 ± 19%      +0.6        0.76 ± 10%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.14 ± 19%      +0.6        0.77 ± 11%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.14 ± 19%      +0.6        0.77 ± 11%  perf-profile.children.cycles-pp.activate_task
      0.17 ± 22%      +0.6        0.80 ±  8%  perf-profile.children.cycles-pp.try_to_wake_up
      0.12 ± 20%      +0.6        0.76 ±  2%  perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.14 ± 25%      +0.6        0.78 ± 12%  perf-profile.children.cycles-pp.unmap_vmas
      0.14 ± 22%      +0.6        0.78 ± 12%  perf-profile.children.cycles-pp.unmap_page_range
      0.14 ± 19%      +0.7        0.82 ±  8%  perf-profile.children.cycles-pp.wake_up_q
      0.14 ± 16%      +0.8        0.90 ±  5%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.15 ± 18%      +0.8        0.92 ±  4%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.15 ± 18%      +0.8        0.96 ±  9%  perf-profile.children.cycles-pp.rwsem_wake
      0.18 ±  2%      +0.9        1.05 ±  3%  perf-profile.children.cycles-pp.__schedule
      0.27 ± 15%      +1.5        1.73 ±  8%  perf-profile.children.cycles-pp.clear_page_erms
      0.29 ± 22%      +1.6        1.90 ±  7%  perf-profile.children.cycles-pp.unmap_region
      0.30 ± 16%      +1.7        1.98 ±  8%  perf-profile.children.cycles-pp.clear_subpage
      0.33 ± 16%      +1.8        2.14 ±  8%  perf-profile.children.cycles-pp.clear_huge_page
      0.35 ± 15%      +1.9        2.25 ±  8%  perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page
      1.10 ± 30%      +1.9        3.00 ± 13%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      1.09 ± 29%      +1.9        3.00 ± 13%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.45 ± 12%      +2.1        2.50 ±  7%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.45 ± 12%      +2.1        2.52 ±  7%  perf-profile.children.cycles-pp.handle_mm_fault
      0.41 ± 22%      +2.4        2.79 ±  8%  perf-profile.children.cycles-pp.__do_munmap
      4.68 ±  7%     +14.8       19.45 ±  3%  perf-profile.children.cycles-pp.osq_lock
      3.78 ± 19%     +17.0       20.82 ±  5%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      3.79 ± 19%     +17.1       20.89 ±  5%  perf-profile.children.cycles-pp.down_write_killable
      3.32 ± 12%     +18.7       22.07 ±  4%  perf-profile.children.cycles-pp.__vm_munmap
      3.32 ± 12%     +18.8       22.07 ±  4%  perf-profile.children.cycles-pp.__x64_sys_munmap
      4.92 ± 16%     +20.5       25.43 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
      4.92 ± 16%     +20.5       25.44 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     24.21           -14.8        9.38 ±  9%  perf-profile.self.cycles-pp.rwsem_optimistic_spin
      3.47 ±  4%      -1.0        2.42 ±  9%  perf-profile.self.cycles-pp.ktime_get
      1.97 ±  3%      -0.8        1.18 ±  5%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.31 ±  3%      -0.6        0.75 ±  5%  perf-profile.self.cycles-pp.native_write_msr
      1.16 ± 17%      -0.5        0.67 ± 23%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      1.15 ±  4%      -0.4        0.71 ±  4%  perf-profile.self.cycles-pp.native_irq_return_iret
      1.02 ±  4%      -0.4        0.58 ±  7%  perf-profile.self.cycles-pp.read_tsc
      0.50 ± 17%      -0.3        0.20 ±  8%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.55 ± 14%      -0.3        0.27 ± 20%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.62 ± 10%      -0.3        0.36        perf-profile.self.cycles-pp.__next_timer_interrupt
      0.56 ±  5%      -0.2        0.39 ±  2%  perf-profile.self.cycles-pp.find_next_bit
      0.22 ± 24%      -0.1        0.08 ± 23%  perf-profile.self.cycles-pp.rebalance_domains
      0.46 ±  8%      -0.1        0.33 ±  8%  perf-profile.self.cycles-pp.native_sched_clock
      0.55 ± 10%      -0.1        0.43 ±  9%  perf-profile.self.cycles-pp.__lock_text_start
      0.27 ±  8%      -0.1        0.14 ± 20%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.27 ± 14%      -0.1        0.15 ± 12%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.33 ± 15%      -0.1        0.22 ±  5%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.33 ± 13%      -0.1        0.23 ± 21%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.25 ± 14%      -0.1        0.15 ±  5%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.20 ± 11%      -0.1        0.11 ±  6%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.22 ± 17%      -0.1        0.13 ± 11%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.20 ±  9%      -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.46 ±  8%      -0.1        0.38 ±  2%  perf-profile.self.cycles-pp.do_idle
      0.23 ± 15%      -0.1        0.14 ± 11%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.21 ± 10%      -0.1        0.13 ±  3%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.11 ± 19%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.find_next_and_bit
      0.19 ±  6%      -0.1        0.12 ± 15%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.18 ± 11%      -0.1        0.10 ± 17%  perf-profile.self.cycles-pp.rcu_irq_exit
      0.39 ±  5%      -0.1        0.32 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.20 ± 18%      -0.1        0.13 ±  5%  perf-profile.self.cycles-pp.interrupt_entry
      0.17 ± 21%      -0.1        0.10 ± 15%  perf-profile.self.cycles-pp.menu_reflect
      0.28 ± 14%      -0.1        0.22 ± 19%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.15 ±  7%      -0.1        0.10 ± 19%  perf-profile.self.cycles-pp.call_cpuidle
      0.10 ± 11%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.14 ± 13%      -0.0        0.09 ± 14%  perf-profile.self.cycles-pp.load_balance
      0.10 ± 12%      -0.0        0.07 ± 21%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.13 ± 13%      -0.0        0.10 ± 10%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.08 ±  6%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.up_read
      0.03 ±100%      +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.update_rq_clock
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp._cond_resched
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.set_next_entity
      0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.anon_vma_interval_tree_remove
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.osq_unlock
      0.00            +0.1        0.07 ± 13%  perf-profile.self.cycles-pp.__default_send_IPI_dest_field
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.llist_add_batch
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.down_write_killable
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.rwsem_mark_wake
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.__switch_to
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.release_pages
      0.00            +0.1        0.08 ± 23%  perf-profile.self.cycles-pp.up_write
      0.00            +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.__vma_adjust
      0.00            +0.1        0.09 ± 12%  perf-profile.self.cycles-pp.load_new_mm_cr3
      0.00            +0.1        0.09 ± 13%  perf-profile.self.cycles-pp.wake_q_add
      0.00            +0.1        0.09 ±  8%  perf-profile.self.cycles-pp.page_remove_rmap
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.update_load_avg
      0.03 ±105%      +0.1        0.13 ± 11%  perf-profile.self.cycles-pp.do_syscall_64
      0.00            +0.1        0.10 ± 18%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.00            +0.1        0.10 ± 24%  perf-profile.self.cycles-pp.__split_huge_pmd_locked
      0.00            +0.1        0.11 ± 11%  perf-profile.self.cycles-pp.finish_task_switch
      0.00            +0.1        0.12 ± 20%  perf-profile.self.cycles-pp.unmapped_area_topdown
      0.00            +0.1        0.12 ± 18%  perf-profile.self.cycles-pp.unwind_next_frame
      0.03 ±100%      +0.1        0.15 ± 14%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.00            +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.00            +0.1        0.13 ± 17%  perf-profile.self.cycles-pp.___might_sleep
      0.00            +0.1        0.13 ± 24%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.00            +0.1        0.15 ± 17%  perf-profile.self.cycles-pp.find_vma
      0.00            +0.2        0.17 ±  9%  perf-profile.self.cycles-pp.__schedule
      0.01 ±173%      +0.2        0.20 ±  5%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.01 ±173%      +0.2        0.20 ± 12%  perf-profile.self.cycles-pp.try_to_wake_up
      0.92            +0.2        1.11 ±  4%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.00            +0.2        0.20 ± 15%  perf-profile.self.cycles-pp.smp_call_function_many
      0.04 ± 58%      +0.3        0.30 ± 12%  perf-profile.self.cycles-pp.clear_subpage
      0.07 ± 23%      +0.3        0.33 ± 13%  perf-profile.self.cycles-pp.unmap_page_range
      0.07 ± 24%      +0.4        0.44 ± 11%  perf-profile.self.cycles-pp.poll_idle
      0.27 ± 15%      +1.4        1.70 ±  8%  perf-profile.self.cycles-pp.clear_page_erms
      4.66 ±  7%     +14.6       19.24 ±  3%  perf-profile.self.cycles-pp.osq_lock
   1437196 ± 12%    +597.8%   10028552        interrupts.CAL:Function_call_interrupts
     13139 ±  9%    +612.1%      93566        interrupts.CPU0.CAL:Function_call_interrupts
    303.25 ± 20%     +68.8%     511.75 ± 15%  interrupts.CPU0.NMI:Non-maskable_interrupts
    303.25 ± 20%     +68.8%     511.75 ± 15%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
    435.50 ± 24%    +736.5%       3643 ± 13%  interrupts.CPU0.RES:Rescheduling_interrupts
     13363 ± 14%    +844.7%     126243 ±  3%  interrupts.CPU0.TLB:TLB_shootdowns
     13574 ± 14%    +609.0%      96246        interrupts.CPU1.CAL:Function_call_interrupts
    295.50 ± 52%     +72.7%     510.25 ± 17%  interrupts.CPU1.NMI:Non-maskable_interrupts
    295.50 ± 52%     +72.7%     510.25 ± 17%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
    978.00 ± 26%    +201.9%       2952 ±  8%  interrupts.CPU1.RES:Rescheduling_interrupts
     14292 ± 16%    +804.5%     129282 ±  2%  interrupts.CPU1.TLB:TLB_shootdowns
     14032 ± 10%    +573.5%      94504        interrupts.CPU10.CAL:Function_call_interrupts
    215.00 ± 27%    +120.8%     474.75 ± 10%  interrupts.CPU10.NMI:Non-maskable_interrupts
    215.00 ± 27%    +120.8%     474.75 ± 10%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
    269.50 ± 21%    +735.1%       2250 ±  5%  interrupts.CPU10.RES:Rescheduling_interrupts
     14207 ± 14%    +796.6%     127391 ±  3%  interrupts.CPU10.TLB:TLB_shootdowns
     13877 ± 12%    +616.2%      99386 ±  4%  interrupts.CPU100.CAL:Function_call_interrupts
    254.25 ± 12%    +890.7%       2518 ± 14%  interrupts.CPU100.RES:Rescheduling_interrupts
     13899 ± 15%    +872.8%     135215 ±  5%  interrupts.CPU100.TLB:TLB_shootdowns
     14086 ± 12%    +585.6%      96577 ±  2%  interrupts.CPU101.CAL:Function_call_interrupts
    263.50 ± 28%     +78.0%     469.00 ± 24%  interrupts.CPU101.NMI:Non-maskable_interrupts
    263.50 ± 28%     +78.0%     469.00 ± 24%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
    286.00 ± 21%    +672.7%       2210 ±  9%  interrupts.CPU101.RES:Rescheduling_interrupts
     13741 ± 17%    +856.4%     131426 ±  4%  interrupts.CPU101.TLB:TLB_shootdowns
     14083 ± 12%    +595.2%      97910 ±  2%  interrupts.CPU102.CAL:Function_call_interrupts
    232.00 ± 31%     +85.7%     430.75 ± 12%  interrupts.CPU102.NMI:Non-maskable_interrupts
    232.00 ± 31%     +85.7%     430.75 ± 12%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
    259.00 ± 10%    +739.3%       2173 ±  8%  interrupts.CPU102.RES:Rescheduling_interrupts
     13850 ± 18%    +861.4%     133149 ±  3%  interrupts.CPU102.TLB:TLB_shootdowns
     14140 ± 14%    +596.3%      98468 ±  4%  interrupts.CPU103.CAL:Function_call_interrupts
    255.00 ± 50%    +102.2%     515.50 ± 22%  interrupts.CPU103.NMI:Non-maskable_interrupts
    255.00 ± 50%    +102.2%     515.50 ± 22%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
    385.75 ± 49%    +580.8%       2626 ± 19%  interrupts.CPU103.RES:Rescheduling_interrupts
     13814 ± 18%    +878.1%     135111 ±  6%  interrupts.CPU103.TLB:TLB_shootdowns
     13993 ± 12%    +567.9%      93458 ±  4%  interrupts.CPU11.CAL:Function_call_interrupts
    177.75 ± 25%    +216.3%     562.25 ± 20%  interrupts.CPU11.NMI:Non-maskable_interrupts
    177.75 ± 25%    +216.3%     562.25 ± 20%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    293.25 ± 12%    +727.2%       2425 ±  5%  interrupts.CPU11.RES:Rescheduling_interrupts
     14250 ± 16%    +784.6%     126066 ±  5%  interrupts.CPU11.TLB:TLB_shootdowns
     13737 ± 11%    +593.9%      95325 ±  2%  interrupts.CPU12.CAL:Function_call_interrupts
    327.50 ± 24%    +799.6%       2946 ± 23%  interrupts.CPU12.RES:Rescheduling_interrupts
     14073 ± 11%    +810.9%     128184 ±  3%  interrupts.CPU12.TLB:TLB_shootdowns
     14325 ±  9%    +560.7%      94649 ±  3%  interrupts.CPU13.CAL:Function_call_interrupts
    316.75 ± 19%    +601.1%       2220 ± 11%  interrupts.CPU13.RES:Rescheduling_interrupts
     14938 ± 14%    +746.9%     126516 ±  4%  interrupts.CPU13.TLB:TLB_shootdowns
     13369 ±  9%    +612.7%      95280 ±  2%  interrupts.CPU14.CAL:Function_call_interrupts
    272.75 ± 34%     +77.6%     484.50 ± 12%  interrupts.CPU14.NMI:Non-maskable_interrupts
    272.75 ± 34%     +77.6%     484.50 ± 12%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
    289.25 ± 22%    +778.1%       2540 ± 15%  interrupts.CPU14.RES:Rescheduling_interrupts
     13531 ± 15%    +842.8%     127574 ±  3%  interrupts.CPU14.TLB:TLB_shootdowns
     13161 ±  8%    +617.7%      94458 ±  3%  interrupts.CPU15.CAL:Function_call_interrupts
    193.75 ± 43%    +241.7%     662.00 ± 40%  interrupts.CPU15.NMI:Non-maskable_interrupts
    193.75 ± 43%    +241.7%     662.00 ± 40%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    266.25 ± 20%    +652.0%       2002 ±  9%  interrupts.CPU15.RES:Rescheduling_interrupts
     13905 ± 14%    +812.8%     126935 ±  4%  interrupts.CPU15.TLB:TLB_shootdowns
     13552 ±  9%    +612.5%      96563 ±  4%  interrupts.CPU16.CAL:Function_call_interrupts
    247.75 ± 18%    +129.3%     568.00 ± 23%  interrupts.CPU16.NMI:Non-maskable_interrupts
    247.75 ± 18%    +129.3%     568.00 ± 23%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    275.75 ± 15%    +677.9%       2145 ± 10%  interrupts.CPU16.RES:Rescheduling_interrupts
     14068 ± 15%    +821.3%     129604 ±  5%  interrupts.CPU16.TLB:TLB_shootdowns
     13268 ±  9%    +618.5%      95338 ±  2%  interrupts.CPU17.CAL:Function_call_interrupts
    213.25 ± 52%    +134.9%     501.00 ±  6%  interrupts.CPU17.NMI:Non-maskable_interrupts
    213.25 ± 52%    +134.9%     501.00 ±  6%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
    286.00 ± 27%    +755.9%       2447        interrupts.CPU17.RES:Rescheduling_interrupts
     13819 ± 14%    +823.2%     127580 ±  3%  interrupts.CPU17.TLB:TLB_shootdowns
     13886 ± 15%    +589.5%      95742 ±  3%  interrupts.CPU18.CAL:Function_call_interrupts
    193.00 ± 27%    +169.0%     519.25 ± 22%  interrupts.CPU18.NMI:Non-maskable_interrupts
    193.00 ± 27%    +169.0%     519.25 ± 22%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
    452.75 ± 75%    +419.5%       2352 ± 11%  interrupts.CPU18.RES:Rescheduling_interrupts
     13857 ± 21%    +822.4%     127819 ±  4%  interrupts.CPU18.TLB:TLB_shootdowns
     13704 ± 12%    +597.3%      95562 ±  2%  interrupts.CPU19.CAL:Function_call_interrupts
    226.00 ± 40%     +97.5%     446.25 ± 18%  interrupts.CPU19.NMI:Non-maskable_interrupts
    226.00 ± 40%     +97.5%     446.25 ± 18%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
    281.50 ± 12%    +777.5%       2470 ± 11%  interrupts.CPU19.RES:Rescheduling_interrupts
     13972 ± 18%    +816.2%     128021 ±  3%  interrupts.CPU19.TLB:TLB_shootdowns
     12947 ± 18%    +634.7%      95124        interrupts.CPU2.CAL:Function_call_interrupts
    299.25 ± 42%     +84.7%     552.75 ± 19%  interrupts.CPU2.NMI:Non-maskable_interrupts
    299.25 ± 42%     +84.7%     552.75 ± 19%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
    315.25 ±  5%    +680.2%       2459 ±  9%  interrupts.CPU2.RES:Rescheduling_interrupts
     14296 ± 16%    +791.0%     127390 ±  3%  interrupts.CPU2.TLB:TLB_shootdowns
     13731 ± 12%    +597.6%      95787 ±  2%  interrupts.CPU20.CAL:Function_call_interrupts
    189.00 ± 30%    +142.6%     458.50 ± 25%  interrupts.CPU20.NMI:Non-maskable_interrupts
    189.00 ± 30%    +142.6%     458.50 ± 25%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
    380.50 ± 24%    +541.7%       2441 ±  9%  interrupts.CPU20.RES:Rescheduling_interrupts
     13909 ± 17%    +823.5%     128454 ±  3%  interrupts.CPU20.TLB:TLB_shootdowns
     13077 ± 11%    +630.4%      95510 ±  2%  interrupts.CPU21.CAL:Function_call_interrupts
    243.00 ± 16%     +87.0%     454.50 ± 35%  interrupts.CPU21.NMI:Non-maskable_interrupts
    243.00 ± 16%     +87.0%     454.50 ± 35%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
    293.75 ± 45%    +575.0%       1982 ± 11%  interrupts.CPU21.RES:Rescheduling_interrupts
     13005 ± 17%    +883.5%     127906 ±  3%  interrupts.CPU21.TLB:TLB_shootdowns
     13763 ± 10%    +598.5%      96128 ±  4%  interrupts.CPU22.CAL:Function_call_interrupts
    256.50 ± 27%     +84.1%     472.25 ± 35%  interrupts.CPU22.NMI:Non-maskable_interrupts
    256.50 ± 27%     +84.1%     472.25 ± 35%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    419.00 ± 50%    +523.4%       2612 ± 24%  interrupts.CPU22.RES:Rescheduling_interrupts
     14236 ± 13%    +807.8%     129241 ±  5%  interrupts.CPU22.TLB:TLB_shootdowns
     13857 ± 16%    +588.9%      95459        interrupts.CPU23.CAL:Function_call_interrupts
    190.00 ± 22%    +189.9%     550.75 ± 10%  interrupts.CPU23.NMI:Non-maskable_interrupts
    190.00 ± 22%    +189.9%     550.75 ± 10%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    304.75 ± 35%    +677.7%       2370 ±  2%  interrupts.CPU23.RES:Rescheduling_interrupts
     14322 ± 23%    +792.0%     127754 ±  2%  interrupts.CPU23.TLB:TLB_shootdowns
     13568 ±  9%    +604.6%      95605 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
    169.25 ± 23%    +213.4%     530.50 ± 18%  interrupts.CPU24.NMI:Non-maskable_interrupts
    169.25 ± 23%    +213.4%     530.50 ± 18%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
    303.75 ±  6%    +743.9%       2563 ±  5%  interrupts.CPU24.RES:Rescheduling_interrupts
     13981 ± 14%    +819.4%     128539 ±  4%  interrupts.CPU24.TLB:TLB_shootdowns
     13543 ± 13%    +602.3%      95108        interrupts.CPU25.CAL:Function_call_interrupts
    240.75 ± 32%    +166.5%     641.50 ± 25%  interrupts.CPU25.NMI:Non-maskable_interrupts
    240.75 ± 32%    +166.5%     641.50 ± 25%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    298.00 ± 27%    +806.1%       2700 ±  3%  interrupts.CPU25.RES:Rescheduling_interrupts
     13812 ± 17%    +821.8%     127322 ±  2%  interrupts.CPU25.TLB:TLB_shootdowns
     13346 ± 13%    +619.3%      95999 ±  2%  interrupts.CPU26.CAL:Function_call_interrupts
    259.75 ± 18%    +625.7%       1885 ± 14%  interrupts.CPU26.RES:Rescheduling_interrupts
     13715 ± 19%    +846.2%     129777 ±  4%  interrupts.CPU26.TLB:TLB_shootdowns
     13779 ± 15%    +616.0%      98664        interrupts.CPU27.CAL:Function_call_interrupts
    255.75 ± 49%     +82.1%     465.75 ± 19%  interrupts.CPU27.NMI:Non-maskable_interrupts
    255.75 ± 49%     +82.1%     465.75 ± 19%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
    266.25 ± 19%    +805.5%       2411 ± 12%  interrupts.CPU27.RES:Rescheduling_interrupts
     14153 ± 21%    +841.6%     133269 ±  2%  interrupts.CPU27.TLB:TLB_shootdowns
     13694 ± 12%    +610.6%      97315        interrupts.CPU28.CAL:Function_call_interrupts
    241.50 ± 37%    +100.5%     484.25 ±  7%  interrupts.CPU28.NMI:Non-maskable_interrupts
    241.50 ± 37%    +100.5%     484.25 ±  7%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
    297.00 ± 28%    +712.4%       2412 ± 15%  interrupts.CPU28.RES:Rescheduling_interrupts
     13841 ± 18%    +853.9%     132033 ±  3%  interrupts.CPU28.TLB:TLB_shootdowns
     13648 ± 12%    +621.9%      98525 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
    257.75 ± 39%    +107.1%     533.75 ± 32%  interrupts.CPU29.NMI:Non-maskable_interrupts
    257.75 ± 39%    +107.1%     533.75 ± 32%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    282.75 ± 14%    +728.8%       2343 ± 11%  interrupts.CPU29.RES:Rescheduling_interrupts
     14041 ± 16%    +858.5%     134580 ±  6%  interrupts.CPU29.TLB:TLB_shootdowns
     13486 ± 15%    +604.9%      95066 ±  2%  interrupts.CPU3.CAL:Function_call_interrupts
    314.50 ± 26%    +738.1%       2635 ± 11%  interrupts.CPU3.RES:Rescheduling_interrupts
     14192 ± 16%    +804.9%     128426 ±  3%  interrupts.CPU3.TLB:TLB_shootdowns
     13600 ± 14%    +625.7%      98697 ±  2%  interrupts.CPU30.CAL:Function_call_interrupts
    200.25 ± 19%    +144.9%     490.50 ± 14%  interrupts.CPU30.NMI:Non-maskable_interrupts
    200.25 ± 19%    +144.9%     490.50 ± 14%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
    261.50 ±  6%    +883.4%       2571 ± 12%  interrupts.CPU30.RES:Rescheduling_interrupts
     14190 ± 18%    +850.6%     134883 ±  2%  interrupts.CPU30.TLB:TLB_shootdowns
     13292 ± 15%    +625.8%      96483 ±  2%  interrupts.CPU31.CAL:Function_call_interrupts
    374.00 ±  8%     +29.9%     486.00 ±  7%  interrupts.CPU31.NMI:Non-maskable_interrupts
    374.00 ±  8%     +29.9%     486.00 ±  7%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
    254.25 ± 21%    +806.2%       2304 ±  6%  interrupts.CPU31.RES:Rescheduling_interrupts
     13829 ± 22%    +844.0%     130550 ±  3%  interrupts.CPU31.TLB:TLB_shootdowns
     13874 ± 12%    +606.5%      98014 ±  2%  interrupts.CPU32.CAL:Function_call_interrupts
    260.50 ± 34%     +79.8%     468.25 ± 18%  interrupts.CPU32.NMI:Non-maskable_interrupts
    260.50 ± 34%     +79.8%     468.25 ± 18%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    726.50 ±111%    +207.8%       2236 ±  7%  interrupts.CPU32.RES:Rescheduling_interrupts
     13862 ± 16%    +861.4%     133276 ±  2%  interrupts.CPU32.TLB:TLB_shootdowns
     13885 ± 12%    +600.5%      97263 ±  3%  interrupts.CPU33.CAL:Function_call_interrupts
    314.25 ± 11%    +760.1%       2703 ± 27%  interrupts.CPU33.RES:Rescheduling_interrupts
     13836 ± 15%    +862.2%     133129 ±  3%  interrupts.CPU33.TLB:TLB_shootdowns
     13994 ± 13%    +624.2%     101346 ±  2%  interrupts.CPU34.CAL:Function_call_interrupts
    153.50 ±  9%    +235.7%     515.25 ± 14%  interrupts.CPU34.NMI:Non-maskable_interrupts
    153.50 ±  9%    +235.7%     515.25 ± 14%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
    266.00 ± 18%    +709.2%       2152 ±  7%  interrupts.CPU34.RES:Rescheduling_interrupts
     14077 ± 16%    +879.2%     137856 ±  3%  interrupts.CPU34.TLB:TLB_shootdowns
     13618 ± 16%    +635.1%     100110 ±  3%  interrupts.CPU35.CAL:Function_call_interrupts
    324.00 ± 17%     +60.2%     519.00 ± 25%  interrupts.CPU35.NMI:Non-maskable_interrupts
    324.00 ± 17%     +60.2%     519.00 ± 25%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
    269.75 ± 13%    +975.9%       2902 ± 16%  interrupts.CPU35.RES:Rescheduling_interrupts
     13524 ± 21%    +910.2%     136633 ±  4%  interrupts.CPU35.TLB:TLB_shootdowns
     13237 ± 13%    +641.9%      98205 ±  3%  interrupts.CPU36.CAL:Function_call_interrupts
    344.50 ± 25%     +61.2%     555.25 ± 16%  interrupts.CPU36.NMI:Non-maskable_interrupts
    344.50 ± 25%     +61.2%     555.25 ± 16%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    298.75 ± 23%    +843.0%       2817 ± 22%  interrupts.CPU36.RES:Rescheduling_interrupts
     12930 ± 17%    +933.8%     133676 ±  4%  interrupts.CPU36.TLB:TLB_shootdowns
     13332 ±  8%    +633.4%      97771 ±  2%  interrupts.CPU37.CAL:Function_call_interrupts
    308.25 ± 30%     +98.2%     611.00 ± 32%  interrupts.CPU37.NMI:Non-maskable_interrupts
    308.25 ± 30%     +98.2%     611.00 ± 32%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
    253.25 ± 13%    +931.7%       2612 ± 18%  interrupts.CPU37.RES:Rescheduling_interrupts
     13186 ±  9%    +909.8%     133150 ±  3%  interrupts.CPU37.TLB:TLB_shootdowns
     14096 ± 15%    +602.2%      98985 ±  3%  interrupts.CPU38.CAL:Function_call_interrupts
    278.00 ± 49%     +87.6%     521.50 ± 31%  interrupts.CPU38.NMI:Non-maskable_interrupts
    278.00 ± 49%     +87.6%     521.50 ± 31%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    340.25 ± 38%    +663.1%       2596 ± 11%  interrupts.CPU38.RES:Rescheduling_interrupts
     13883 ± 20%    +869.3%     134568 ±  4%  interrupts.CPU38.TLB:TLB_shootdowns
     13969 ± 12%    +604.9%      98471 ±  3%  interrupts.CPU39.CAL:Function_call_interrupts
    245.25 ± 38%     +95.0%     478.25 ± 16%  interrupts.CPU39.NMI:Non-maskable_interrupts
    245.25 ± 38%     +95.0%     478.25 ± 16%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
    259.75 ± 26%    +870.6%       2521 ±  8%  interrupts.CPU39.RES:Rescheduling_interrupts
     13539 ± 16%    +884.5%     133303 ±  3%  interrupts.CPU39.TLB:TLB_shootdowns
     13147 ± 15%    +619.1%      94547 ±  3%  interrupts.CPU4.CAL:Function_call_interrupts
    224.25 ± 27%    +117.9%     488.75 ± 16%  interrupts.CPU4.NMI:Non-maskable_interrupts
    224.25 ± 27%    +117.9%     488.75 ± 16%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    272.50 ± 25%    +693.0%       2161 ±  7%  interrupts.CPU4.RES:Rescheduling_interrupts
     13514 ± 18%    +840.9%     127151 ±  5%  interrupts.CPU4.TLB:TLB_shootdowns
     13667 ± 12%    +623.6%      98897 ±  3%  interrupts.CPU40.CAL:Function_call_interrupts
    229.25 ± 13%    +991.2%       2501 ± 11%  interrupts.CPU40.RES:Rescheduling_interrupts
     13321 ± 16%    +910.1%     134565 ±  4%  interrupts.CPU40.TLB:TLB_shootdowns
     13723 ± 11%    +610.9%      97566 ±  2%  interrupts.CPU41.CAL:Function_call_interrupts
    254.00 ± 14%    +878.6%       2485 ± 20%  interrupts.CPU41.RES:Rescheduling_interrupts
     13477 ± 14%    +890.0%     133428 ±  3%  interrupts.CPU41.TLB:TLB_shootdowns
     14131 ± 13%    +582.6%      96465 ±  2%  interrupts.CPU42.CAL:Function_call_interrupts
    246.00 ± 28%     +85.6%     456.50 ±  7%  interrupts.CPU42.NMI:Non-maskable_interrupts
    246.00 ± 28%     +85.6%     456.50 ±  7%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
    271.25 ±  9%    +790.8%       2416 ±  6%  interrupts.CPU42.RES:Rescheduling_interrupts
     13965 ± 18%    +836.3%     130756 ±  3%  interrupts.CPU42.TLB:TLB_shootdowns
     13715 ± 11%    +609.1%      97258 ±  3%  interrupts.CPU43.CAL:Function_call_interrupts
    253.00 ± 40%    +100.7%     507.75 ± 14%  interrupts.CPU43.NMI:Non-maskable_interrupts
    253.00 ± 40%    +100.7%     507.75 ± 14%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
    257.00 ± 21%    +808.9%       2335 ±  6%  interrupts.CPU43.RES:Rescheduling_interrupts
     13354 ± 14%    +888.1%     131955 ±  4%  interrupts.CPU43.TLB:TLB_shootdowns
     13945 ± 14%    +602.1%      97906 ±  3%  interrupts.CPU44.CAL:Function_call_interrupts
    212.00 ± 44%    +151.5%     533.25 ±  8%  interrupts.CPU44.NMI:Non-maskable_interrupts
    212.00 ± 44%    +151.5%     533.25 ±  8%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
    232.50 ± 19%   +1079.0%       2741 ± 20%  interrupts.CPU44.RES:Rescheduling_interrupts
     13714 ± 18%    +869.6%     132968 ±  4%  interrupts.CPU44.TLB:TLB_shootdowns
     14268 ± 16%    +580.8%      97139 ±  2%  interrupts.CPU45.CAL:Function_call_interrupts
    353.75 ± 54%    +607.1%       2501 ± 20%  interrupts.CPU45.RES:Rescheduling_interrupts
     14214 ± 21%    +827.3%     131801 ±  3%  interrupts.CPU45.TLB:TLB_shootdowns
     14210 ± 11%    +581.1%      96791        interrupts.CPU46.CAL:Function_call_interrupts
    247.00 ± 26%    +800.7%       2224 ±  6%  interrupts.CPU46.RES:Rescheduling_interrupts
     14067 ± 14%    +830.6%     130907 ±  2%  interrupts.CPU46.TLB:TLB_shootdowns
     14063 ± 11%    +603.5%      98928 ±  5%  interrupts.CPU47.CAL:Function_call_interrupts
    251.00 ± 50%     +94.1%     487.25 ± 28%  interrupts.CPU47.NMI:Non-maskable_interrupts
    251.00 ± 50%     +94.1%     487.25 ± 28%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
    291.25 ± 19%    +697.7%       2323 ± 11%  interrupts.CPU47.RES:Rescheduling_interrupts
     13700 ± 14%    +882.8%     134643 ±  5%  interrupts.CPU47.TLB:TLB_shootdowns
     14106 ± 13%    +592.8%      97735 ±  2%  interrupts.CPU48.CAL:Function_call_interrupts
    331.75 ± 40%    +738.7%       2782 ±  8%  interrupts.CPU48.RES:Rescheduling_interrupts
     13797 ± 15%    +864.5%     133075 ±  3%  interrupts.CPU48.TLB:TLB_shootdowns
     13616 ± 12%    +615.0%      97354 ±  4%  interrupts.CPU49.CAL:Function_call_interrupts
    281.00 ± 20%    +678.6%       2187 ± 13%  interrupts.CPU49.RES:Rescheduling_interrupts
     13215 ± 14%    +901.7%     132378 ±  5%  interrupts.CPU49.TLB:TLB_shootdowns
     13597 ± 12%    +596.6%      94714 ±  2%  interrupts.CPU5.CAL:Function_call_interrupts
    298.00 ± 30%     +93.2%     575.75 ± 17%  interrupts.CPU5.NMI:Non-maskable_interrupts
    298.00 ± 30%     +93.2%     575.75 ± 17%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
    325.25 ± 25%    +616.4%       2330 ±  2%  interrupts.CPU5.RES:Rescheduling_interrupts
     13942 ± 15%    +813.8%     127411 ±  3%  interrupts.CPU5.TLB:TLB_shootdowns
     14406 ± 12%    +578.5%      97746 ±  2%  interrupts.CPU50.CAL:Function_call_interrupts
    230.25 ± 42%     +89.6%     436.50 ± 17%  interrupts.CPU50.NMI:Non-maskable_interrupts
    230.25 ± 42%     +89.6%     436.50 ± 17%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
    229.75 ± 12%   +1000.4%       2528 ±  5%  interrupts.CPU50.RES:Rescheduling_interrupts
     14359 ± 15%    +827.0%     133114 ±  2%  interrupts.CPU50.TLB:TLB_shootdowns
     14308 ± 13%    +592.4%      99062 ±  3%  interrupts.CPU51.CAL:Function_call_interrupts
    271.50 ± 63%     +88.6%     512.00 ± 31%  interrupts.CPU51.NMI:Non-maskable_interrupts
    271.50 ± 63%     +88.6%     512.00 ± 31%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
    319.00 ± 16%    +767.3%       2766 ± 16%  interrupts.CPU51.RES:Rescheduling_interrupts
     14139 ± 16%    +853.6%     134839 ±  3%  interrupts.CPU51.TLB:TLB_shootdowns
     14062 ± 11%    +569.2%      94105 ±  2%  interrupts.CPU52.CAL:Function_call_interrupts
    237.50 ± 33%    +128.1%     541.75 ± 24%  interrupts.CPU52.NMI:Non-maskable_interrupts
    237.50 ± 33%    +128.1%     541.75 ± 24%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
    321.25 ± 45%    +734.7%       2681 ± 24%  interrupts.CPU52.RES:Rescheduling_interrupts
     14041 ± 15%    +799.2%     126262 ±  3%  interrupts.CPU52.TLB:TLB_shootdowns
     14247 ± 11%    +574.6%      96114        interrupts.CPU53.CAL:Function_call_interrupts
    260.75 ± 51%     +72.6%     450.00 ±  9%  interrupts.CPU53.NMI:Non-maskable_interrupts
    260.75 ± 51%     +72.6%     450.00 ±  9%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    279.75 ± 22%    +845.2%       2644 ±  5%  interrupts.CPU53.RES:Rescheduling_interrupts
     14435 ± 13%    +792.6%     128845 ±  3%  interrupts.CPU53.TLB:TLB_shootdowns
     14048 ± 12%    +571.2%      94285 ±  2%  interrupts.CPU54.CAL:Function_call_interrupts
    248.00 ± 27%    +120.2%     546.00 ±  7%  interrupts.CPU54.NMI:Non-maskable_interrupts
    248.00 ± 27%    +120.2%     546.00 ±  7%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
    290.25 ± 19%    +761.9%       2501 ± 11%  interrupts.CPU54.RES:Rescheduling_interrupts
     13942 ± 16%    +808.6%     126676 ±  3%  interrupts.CPU54.TLB:TLB_shootdowns
     13704 ±  9%    +589.1%      94438        interrupts.CPU55.CAL:Function_call_interrupts
    266.50 ± 16%    +764.2%       2303 ± 19%  interrupts.CPU55.RES:Rescheduling_interrupts
     14142 ± 12%    +798.3%     127034 ±  2%  interrupts.CPU55.TLB:TLB_shootdowns
     13849 ± 12%    +575.7%      93576        interrupts.CPU56.CAL:Function_call_interrupts
    199.75 ± 37%    +163.2%     525.75 ±  9%  interrupts.CPU56.NMI:Non-maskable_interrupts
    199.75 ± 37%    +163.2%     525.75 ±  9%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    250.25 ± 16%    +803.0%       2259 ± 11%  interrupts.CPU56.RES:Rescheduling_interrupts
     13829 ± 15%    +812.3%     126172 ±  3%  interrupts.CPU56.TLB:TLB_shootdowns
     13252 ± 11%    +612.7%      94453        interrupts.CPU57.CAL:Function_call_interrupts
    265.00 ± 46%    +123.6%     592.50 ± 15%  interrupts.CPU57.NMI:Non-maskable_interrupts
    265.00 ± 46%    +123.6%     592.50 ± 15%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
    258.75 ± 24%    +786.6%       2294 ±  5%  interrupts.CPU57.RES:Rescheduling_interrupts
     14114 ± 15%    +797.8%     126714 ±  2%  interrupts.CPU57.TLB:TLB_shootdowns
     13859 ± 12%    +583.5%      94723 ±  2%  interrupts.CPU58.CAL:Function_call_interrupts
    198.00 ± 15%    +143.1%     481.25 ± 10%  interrupts.CPU58.NMI:Non-maskable_interrupts
    198.00 ± 15%    +143.1%     481.25 ± 10%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
    236.50 ± 16%    +931.9%       2440 ±  9%  interrupts.CPU58.RES:Rescheduling_interrupts
     13631 ± 16%    +836.1%     127601 ±  4%  interrupts.CPU58.TLB:TLB_shootdowns
     14084 ± 12%    +575.3%      95116        interrupts.CPU59.CAL:Function_call_interrupts
    235.50 ± 45%    +136.3%     556.50 ±  8%  interrupts.CPU59.NMI:Non-maskable_interrupts
    235.50 ± 45%    +136.3%     556.50 ±  8%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
    266.50 ± 19%    +733.1%       2220 ±  9%  interrupts.CPU59.RES:Rescheduling_interrupts
     14034 ± 14%    +815.4%     128469 ±  3%  interrupts.CPU59.TLB:TLB_shootdowns
     13888 ± 12%    +582.3%      94759 ±  2%  interrupts.CPU6.CAL:Function_call_interrupts
    172.50 ± 19%    +172.5%     470.00 ± 16%  interrupts.CPU6.NMI:Non-maskable_interrupts
    172.50 ± 19%    +172.5%     470.00 ± 16%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    308.50 ± 13%    +693.1%       2446 ± 13%  interrupts.CPU6.RES:Rescheduling_interrupts
     14055 ± 18%    +806.0%     127341 ±  4%  interrupts.CPU6.TLB:TLB_shootdowns
     14010 ± 10%    +585.2%      95994        interrupts.CPU60.CAL:Function_call_interrupts
    224.00 ± 14%    +118.4%     489.25 ± 16%  interrupts.CPU60.NMI:Non-maskable_interrupts
    224.00 ± 14%    +118.4%     489.25 ± 16%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
    781.50 ± 92%    +338.4%       3426 ± 15%  interrupts.CPU60.RES:Rescheduling_interrupts
     13776 ± 13%    +838.6%     129306 ±  3%  interrupts.CPU60.TLB:TLB_shootdowns
     14519 ± 13%    +556.0%      95246        interrupts.CPU61.CAL:Function_call_interrupts
    177.75 ± 18%    +206.3%     544.50 ± 30%  interrupts.CPU61.NMI:Non-maskable_interrupts
    177.75 ± 18%    +206.3%     544.50 ± 30%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
    338.00 ± 15%    +866.3%       3266 ± 19%  interrupts.CPU61.RES:Rescheduling_interrupts
     14574 ± 17%    +783.6%     128776 ±  2%  interrupts.CPU61.TLB:TLB_shootdowns
     13695 ± 13%    +594.0%      95041 ±  2%  interrupts.CPU62.CAL:Function_call_interrupts
    264.00 ± 23%    +758.8%       2267 ±  9%  interrupts.CPU62.RES:Rescheduling_interrupts
     13407 ± 16%    +852.5%     127710 ±  4%  interrupts.CPU62.TLB:TLB_shootdowns
     14011 ± 11%    +564.8%      93151        interrupts.CPU63.CAL:Function_call_interrupts
    241.25 ± 36%    +114.5%     517.50 ± 15%  interrupts.CPU63.NMI:Non-maskable_interrupts
    241.25 ± 36%    +114.5%     517.50 ± 15%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
    298.25 ± 18%    +729.8%       2474 ±  6%  interrupts.CPU63.RES:Rescheduling_interrupts
     14293 ± 10%    +777.0%     125356 ±  3%  interrupts.CPU63.TLB:TLB_shootdowns
     13836 ± 14%    +577.3%      93714        interrupts.CPU64.CAL:Function_call_interrupts
    271.00 ± 33%    +109.2%     567.00 ± 26%  interrupts.CPU64.NMI:Non-maskable_interrupts
    271.00 ± 33%    +109.2%     567.00 ± 26%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
    276.75 ± 27%   +1038.4%       3150 ± 24%  interrupts.CPU64.RES:Rescheduling_interrupts
     13616 ± 17%    +826.3%     126134 ±  3%  interrupts.CPU64.TLB:TLB_shootdowns
     14166 ± 11%    +563.6%      94013 ±  3%  interrupts.CPU65.CAL:Function_call_interrupts
    216.00 ± 57%    +204.5%     657.75 ± 23%  interrupts.CPU65.NMI:Non-maskable_interrupts
    216.00 ± 57%    +204.5%     657.75 ± 23%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
    249.50 ± 14%    +892.8%       2477 ± 12%  interrupts.CPU65.RES:Rescheduling_interrupts
     14228 ± 13%    +789.3%     126530 ±  4%  interrupts.CPU65.TLB:TLB_shootdowns
     13977 ± 13%    +576.9%      94610 ±  2%  interrupts.CPU66.CAL:Function_call_interrupts
    283.00 ± 37%     +81.4%     513.25 ± 18%  interrupts.CPU66.NMI:Non-maskable_interrupts
    283.00 ± 37%     +81.4%     513.25 ± 18%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
    258.25 ± 12%    +960.9%       2739 ± 24%  interrupts.CPU66.RES:Rescheduling_interrupts
     13934 ± 16%    +810.9%     126928 ±  4%  interrupts.CPU66.TLB:TLB_shootdowns
     13920 ± 13%    +577.7%      94337        interrupts.CPU67.CAL:Function_call_interrupts
    181.75 ± 42%    +223.2%     587.50 ± 11%  interrupts.CPU67.NMI:Non-maskable_interrupts
    181.75 ± 42%    +223.2%     587.50 ± 11%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
    259.00 ± 30%    +790.6%       2306 ±  9%  interrupts.CPU67.RES:Rescheduling_interrupts
     13881 ± 17%    +816.0%     127161 ±  3%  interrupts.CPU67.TLB:TLB_shootdowns
     13474 ± 14%    +599.9%      94302 ±  2%  interrupts.CPU68.CAL:Function_call_interrupts
    263.00 ± 22%    +170.4%     711.25 ± 32%  interrupts.CPU68.NMI:Non-maskable_interrupts
    263.00 ± 22%    +170.4%     711.25 ± 32%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
    289.25 ± 29%    +693.8%       2296 ±  8%  interrupts.CPU68.RES:Rescheduling_interrupts
     13269 ± 18%    +856.5%     126915 ±  3%  interrupts.CPU68.TLB:TLB_shootdowns
     14328 ± 14%    +559.6%      94508 ±  3%  interrupts.CPU69.CAL:Function_call_interrupts
    227.25 ± 40%    +109.1%     475.25 ± 14%  interrupts.CPU69.NMI:Non-maskable_interrupts
    227.25 ± 40%    +109.1%     475.25 ± 14%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
    288.25 ± 27%    +812.2%       2629 ±  9%  interrupts.CPU69.RES:Rescheduling_interrupts
     14446 ± 17%    +773.8%     126233 ±  4%  interrupts.CPU69.TLB:TLB_shootdowns
     13991 ±  9%    +588.1%      96281        interrupts.CPU7.CAL:Function_call_interrupts
    238.00 ± 31%    +134.6%     558.25 ± 12%  interrupts.CPU7.NMI:Non-maskable_interrupts
    238.00 ± 31%    +134.6%     558.25 ± 12%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    336.75 ± 23%    +622.7%       2433 ±  7%  interrupts.CPU7.RES:Rescheduling_interrupts
     14321 ± 15%    +807.2%     129923 ±  3%  interrupts.CPU7.TLB:TLB_shootdowns
     13494 ±  9%    +601.0%      94588        interrupts.CPU70.CAL:Function_call_interrupts
    206.00 ± 32%    +184.2%     585.50 ± 37%  interrupts.CPU70.NMI:Non-maskable_interrupts
    206.00 ± 32%    +184.2%     585.50 ± 37%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
    234.25 ± 11%    +847.1%       2218 ±  6%  interrupts.CPU70.RES:Rescheduling_interrupts
     13965 ± 12%    +806.3%     126570 ±  2%  interrupts.CPU70.TLB:TLB_shootdowns
     14215 ± 10%    +554.9%      93092 ±  4%  interrupts.CPU71.CAL:Function_call_interrupts
    187.75 ± 33%    +215.4%     592.25 ± 31%  interrupts.CPU71.NMI:Non-maskable_interrupts
    187.75 ± 33%    +215.4%     592.25 ± 31%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
    341.25 ± 24%    +590.9%       2357 ± 14%  interrupts.CPU71.RES:Rescheduling_interrupts
     14087 ± 13%    +781.5%     124172 ±  6%  interrupts.CPU71.TLB:TLB_shootdowns
     13747 ± 10%    +576.0%      92934 ±  2%  interrupts.CPU72.CAL:Function_call_interrupts
    185.50 ± 31%    +181.3%     521.75 ± 20%  interrupts.CPU72.NMI:Non-maskable_interrupts
    185.50 ± 31%    +181.3%     521.75 ± 20%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
    281.00 ± 17%    +808.6%       2553 ±  8%  interrupts.CPU72.RES:Rescheduling_interrupts
     13452 ± 14%    +823.7%     124269 ±  3%  interrupts.CPU72.TLB:TLB_shootdowns
     14001 ± 12%    +582.7%      95592        interrupts.CPU73.CAL:Function_call_interrupts
    276.75 ± 16%     +62.8%     450.50 ± 15%  interrupts.CPU73.NMI:Non-maskable_interrupts
    276.75 ± 16%     +62.8%     450.50 ± 15%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
    303.75 ± 22%    +686.3%       2388 ±  6%  interrupts.CPU73.RES:Rescheduling_interrupts
     13804 ± 18%    +832.7%     128763 ±  2%  interrupts.CPU73.TLB:TLB_shootdowns
     13880 ± 11%    +575.8%      93806 ±  3%  interrupts.CPU74.CAL:Function_call_interrupts
    227.75 ± 34%     +94.0%     441.75 ± 14%  interrupts.CPU74.NMI:Non-maskable_interrupts
    227.75 ± 34%     +94.0%     441.75 ± 14%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    256.75 ± 19%    +771.3%       2237 ± 17%  interrupts.CPU74.RES:Rescheduling_interrupts
     13649 ± 16%    +821.6%     125794 ±  4%  interrupts.CPU74.TLB:TLB_shootdowns
     13445 ±  8%    +607.7%      95153 ±  2%  interrupts.CPU75.CAL:Function_call_interrupts
    177.25 ± 38%    +227.8%     581.00 ± 12%  interrupts.CPU75.NMI:Non-maskable_interrupts
    177.25 ± 38%    +227.8%     581.00 ± 12%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
    274.00 ± 16%    +724.4%       2258 ± 10%  interrupts.CPU75.RES:Rescheduling_interrupts
     12873 ± 11%    +896.8%     128324 ±  3%  interrupts.CPU75.TLB:TLB_shootdowns
     14056 ± 10%    +564.0%      93337 ±  2%  interrupts.CPU76.CAL:Function_call_interrupts
    185.00 ± 42%    +182.8%     523.25 ± 14%  interrupts.CPU76.NMI:Non-maskable_interrupts
    185.00 ± 42%    +182.8%     523.25 ± 14%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
    298.75 ± 22%    +736.7%       2499 ±  3%  interrupts.CPU76.RES:Rescheduling_interrupts
     13893 ± 14%    +803.5%     125523 ±  3%  interrupts.CPU76.TLB:TLB_shootdowns
     13709 ± 11%    +580.4%      93274        interrupts.CPU77.CAL:Function_call_interrupts
    193.75 ± 20%    +287.2%     750.25 ± 26%  interrupts.CPU77.NMI:Non-maskable_interrupts
    193.75 ± 20%    +287.2%     750.25 ± 26%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
    269.75 ± 16%    +889.0%       2667 ± 10%  interrupts.CPU77.RES:Rescheduling_interrupts
     13409 ± 16%    +829.6%     124647 ±  2%  interrupts.CPU77.TLB:TLB_shootdowns
     13919 ± 11%    +593.5%      96533 ±  3%  interrupts.CPU78.CAL:Function_call_interrupts
    329.75 ± 37%     +67.9%     553.75 ± 25%  interrupts.CPU78.NMI:Non-maskable_interrupts
    329.75 ± 37%     +67.9%     553.75 ± 25%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
    383.50 ± 23%    +468.1%       2178 ± 13%  interrupts.CPU78.RES:Rescheduling_interrupts
     13684 ± 17%    +857.3%     131008 ±  5%  interrupts.CPU78.TLB:TLB_shootdowns
     13814 ±  9%    +617.1%      99059 ±  3%  interrupts.CPU79.CAL:Function_call_interrupts
    316.25 ±  8%    +794.5%       2828 ± 16%  interrupts.CPU79.RES:Rescheduling_interrupts
     13341 ± 13%    +910.2%     134765 ±  4%  interrupts.CPU79.TLB:TLB_shootdowns
     14339 ± 10%    +567.2%      95668        interrupts.CPU8.CAL:Function_call_interrupts
    227.50 ± 23%    +119.7%     499.75 ± 25%  interrupts.CPU8.NMI:Non-maskable_interrupts
    227.50 ± 23%    +119.7%     499.75 ± 25%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    345.50 ± 27%    +892.0%       3427 ± 14%  interrupts.CPU8.RES:Rescheduling_interrupts
     14576 ± 13%    +783.7%     128813 ±  2%  interrupts.CPU8.TLB:TLB_shootdowns
     13929 ± 14%    +606.9%      98461 ±  3%  interrupts.CPU80.CAL:Function_call_interrupts
    210.25 ± 32%    +128.2%     479.75 ± 15%  interrupts.CPU80.NMI:Non-maskable_interrupts
    210.25 ± 32%    +128.2%     479.75 ± 15%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
    276.75 ± 30%    +737.2%       2317 ±  9%  interrupts.CPU80.RES:Rescheduling_interrupts
     13611 ± 19%    +881.5%     133599 ±  4%  interrupts.CPU80.TLB:TLB_shootdowns
     14023 ± 10%    +595.6%      97549 ±  4%  interrupts.CPU81.CAL:Function_call_interrupts
    255.50 ± 53%     +95.5%     499.50 ± 19%  interrupts.CPU81.NMI:Non-maskable_interrupts
    255.50 ± 53%     +95.5%     499.50 ± 19%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
    279.50 ± 15%    +909.9%       2822 ± 22%  interrupts.CPU81.RES:Rescheduling_interrupts
     13914 ± 13%    +850.6%     132266 ±  4%  interrupts.CPU81.TLB:TLB_shootdowns
     12964 ± 18%    +652.0%      97486 ±  3%  interrupts.CPU82.CAL:Function_call_interrupts
    179.50 ± 31%    +191.1%     522.50 ± 19%  interrupts.CPU82.NMI:Non-maskable_interrupts
    179.50 ± 31%    +191.1%     522.50 ± 19%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
    281.50 ± 27%    +746.5%       2383 ± 13%  interrupts.CPU82.RES:Rescheduling_interrupts
     12743 ± 17%    +942.8%     132883 ±  4%  interrupts.CPU82.TLB:TLB_shootdowns
     13822 ± 14%    +615.1%      98841 ±  3%  interrupts.CPU83.CAL:Function_call_interrupts
    286.50 ± 26%     +65.2%     473.25 ±  2%  interrupts.CPU83.NMI:Non-maskable_interrupts
    286.50 ± 26%     +65.2%     473.25 ±  2%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
    267.75 ± 27%    +698.6%       2138 ±  9%  interrupts.CPU83.RES:Rescheduling_interrupts
     13497 ± 17%    +894.4%     134223 ±  4%  interrupts.CPU83.TLB:TLB_shootdowns
     14051 ± 11%    +603.9%      98917        interrupts.CPU84.CAL:Function_call_interrupts
    758.00 ± 99%    +246.6%       2627 ±  7%  interrupts.CPU84.RES:Rescheduling_interrupts
     13763 ± 15%    +876.2%     134362 ±  2%  interrupts.CPU84.TLB:TLB_shootdowns
     14235 ± 13%    +593.1%      98672        interrupts.CPU85.CAL:Function_call_interrupts
    236.75 ± 52%     +74.9%     414.00 ± 30%  interrupts.CPU85.NMI:Non-maskable_interrupts
    236.75 ± 52%     +74.9%     414.00 ± 30%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
    286.75 ± 41%    +768.5%       2490 ± 13%  interrupts.CPU85.RES:Rescheduling_interrupts
     14124 ± 17%    +841.5%     132980 ±  3%  interrupts.CPU85.TLB:TLB_shootdowns
     13994 ± 12%    +604.6%      98611 ±  4%  interrupts.CPU86.CAL:Function_call_interrupts
    141.25 ± 26%    +226.0%     460.50 ± 39%  interrupts.CPU86.NMI:Non-maskable_interrupts
    141.25 ± 26%    +226.0%     460.50 ± 39%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
    248.50 ± 21%    +987.7%       2703 ± 20%  interrupts.CPU86.RES:Rescheduling_interrupts
     13731 ± 17%    +879.1%     134439 ±  4%  interrupts.CPU86.TLB:TLB_shootdowns
     14062 ± 12%    +607.3%      99463 ±  4%  interrupts.CPU87.CAL:Function_call_interrupts
    338.50 ± 39%    +686.0%       2660 ± 18%  interrupts.CPU87.RES:Rescheduling_interrupts
     13843 ± 18%    +884.5%     136282 ±  5%  interrupts.CPU87.TLB:TLB_shootdowns
     14059 ± 12%    +602.0%      98688 ±  5%  interrupts.CPU88.CAL:Function_call_interrupts
    339.75 ± 37%    +740.3%       2855 ± 15%  interrupts.CPU88.RES:Rescheduling_interrupts
     13787 ± 17%    +875.3%     134472 ±  5%  interrupts.CPU88.TLB:TLB_shootdowns
     14183 ± 12%    +584.9%      97142 ±  4%  interrupts.CPU89.CAL:Function_call_interrupts
    298.50 ± 18%    +798.2%       2681 ± 20%  interrupts.CPU89.RES:Rescheduling_interrupts
     13667 ± 16%    +864.4%     131807 ±  5%  interrupts.CPU89.TLB:TLB_shootdowns
     13844 ± 13%    +588.7%      95350 ±  2%  interrupts.CPU9.CAL:Function_call_interrupts
    185.00 ± 29%    +177.0%     512.50 ± 20%  interrupts.CPU9.NMI:Non-maskable_interrupts
    185.00 ± 29%    +177.0%     512.50 ± 20%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    354.00 ± 14%    +807.2%       3211 ± 22%  interrupts.CPU9.RES:Rescheduling_interrupts
     14011 ± 18%    +820.3%     128951 ±  2%  interrupts.CPU9.TLB:TLB_shootdowns
     14035 ± 12%    +584.3%      96040 ±  3%  interrupts.CPU90.CAL:Function_call_interrupts
    375.50 ± 37%    +521.8%       2335 ±  3%  interrupts.CPU90.RES:Rescheduling_interrupts
     13566 ± 15%    +859.0%     130102 ±  4%  interrupts.CPU90.TLB:TLB_shootdowns
     13724 ± 11%    +617.2%      98426 ±  2%  interrupts.CPU91.CAL:Function_call_interrupts
    239.00 ± 16%    +846.8%       2262 ± 15%  interrupts.CPU91.RES:Rescheduling_interrupts
     13060 ± 13%    +923.7%     133699 ±  2%  interrupts.CPU91.TLB:TLB_shootdowns
     13607 ± 12%    +622.1%      98261 ±  5%  interrupts.CPU92.CAL:Function_call_interrupts
    245.00 ±  7%    +893.9%       2435 ± 25%  interrupts.CPU92.RES:Rescheduling_interrupts
     13122 ± 16%    +917.5%     133515 ±  6%  interrupts.CPU92.TLB:TLB_shootdowns
     14072 ± 12%    +599.2%      98395 ±  2%  interrupts.CPU93.CAL:Function_call_interrupts
    306.75 ± 18%    +745.6%       2593 ± 13%  interrupts.CPU93.RES:Rescheduling_interrupts
     13776 ± 15%    +870.0%     133633 ±  4%  interrupts.CPU93.TLB:TLB_shootdowns
     13572 ± 12%    +621.3%      97901 ±  2%  interrupts.CPU94.CAL:Function_call_interrupts
    262.00 ± 12%    +908.3%       2641 ± 12%  interrupts.CPU94.RES:Rescheduling_interrupts
     12976 ± 16%    +926.4%     133193 ±  4%  interrupts.CPU94.TLB:TLB_shootdowns
     13769 ± 15%    +607.2%      97376 ±  2%  interrupts.CPU95.CAL:Function_call_interrupts
    364.75 ± 33%    +577.7%       2471 ± 15%  interrupts.CPU95.RES:Rescheduling_interrupts
     13294 ± 22%    +897.1%     132556 ±  3%  interrupts.CPU95.TLB:TLB_shootdowns
     14003 ± 11%    +588.1%      96354 ±  4%  interrupts.CPU96.CAL:Function_call_interrupts
    204.50 ± 37%    +146.0%     503.00 ± 42%  interrupts.CPU96.NMI:Non-maskable_interrupts
    204.50 ± 37%    +146.0%     503.00 ± 42%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
    333.25 ± 36%    +733.5%       2777 ± 19%  interrupts.CPU96.RES:Rescheduling_interrupts
     13738 ± 15%    +857.1%     131484 ±  4%  interrupts.CPU96.TLB:TLB_shootdowns
     13953 ± 13%    +603.0%      98087 ±  3%  interrupts.CPU97.CAL:Function_call_interrupts
    311.50 ± 31%    +757.5%       2671 ± 17%  interrupts.CPU97.RES:Rescheduling_interrupts
     13480 ± 16%    +888.8%     133293 ±  4%  interrupts.CPU97.TLB:TLB_shootdowns
     13710 ± 12%    +627.1%      99693 ±  4%  interrupts.CPU98.CAL:Function_call_interrupts
    347.50 ± 39%    +652.8%       2616 ±  9%  interrupts.CPU98.RES:Rescheduling_interrupts
     13400 ± 15%    +910.5%     135408 ±  5%  interrupts.CPU98.TLB:TLB_shootdowns
     13848 ± 13%    +609.3%      98225 ±  3%  interrupts.CPU99.CAL:Function_call_interrupts
    288.25 ± 25%    +830.2%       2681 ± 15%  interrupts.CPU99.RES:Rescheduling_interrupts
     13414 ± 17%    +890.8%     132913 ±  3%  interrupts.CPU99.TLB:TLB_shootdowns
     26764 ± 18%     +94.9%      52154 ±  2%  interrupts.NMI:Non-maskable_interrupts
     26764 ± 18%     +94.9%      52154 ±  2%  interrupts.PMI:Performance_monitoring_interrupts
     32757 ± 11%    +698.9%     261696        interrupts.RES:Rescheduling_interrupts
   1435977 ± 15%    +844.0%   13555654        interrupts.TLB:TLB_shootdowns


                                                                                
                           will-it-scale.per_thread_ops                         
                                                                                
  160 +-+----------O-----------------------------O----O---------------------+   
      O    O  O O       O    O      O              O    O                   |   
  140 +-+                         O                                         |   
  120 +-+            O                   O  O O                             |   
      |  O                O    O       O                                    |   
  100 +-+                                                                   |   
      |                                                                     |   
   80 +-+                                                                   |   
      |                                                                     |   
   60 +-+                                                                   |   
   40 +-+                                                                   |   
      |                                                                     |   
   20 +-+                                                 .+.  .+     .+..  |   
      |.. .+..+.+..+.+..+.+..+.+..+.+..      .+.. .+..+.+.   +.  +  .+    +.|   
    0 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               will-it-scale.workload                           
                                                                                
  18000 +-+-----------------------------------------------------------------+   
        |           O                            O    O                     |   
  16000 O-+  O O  O      O    O      O              O    O                  |   
  14000 +-+                        O        O                               |   
        |              O   O    O       O O    O                            |   
  12000 +-O                                                                 |   
  10000 +-+                                                                 |   
        |                                                                   |   
   8000 +-+                                                                 |   
   6000 +-+                                                                 |   
        |                                                                   |   
   4000 +-+                                                                 |   
   2000 +-+                                               .+.               |   
        |.  .+.  .+.+..+.+.+..+.+..+.+..      .+.  .+.+..+   +..+.  .+.+..+.|   
      0 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          will-it-scale.time.user_time                          
                                                                                
  12 +-+--------------------------------------------------------------------+   
     |                                                                      |   
  10 +-+  O       O                             O    O                      |   
     O       O O       O    O                                               |   
     |                           O O               O    O                   |   
   8 +-+            O    O    O       O  O O  O                             |   
     |  O                                                                   |   
   6 +-+                                                                    |   
     |                                                                      |   
   4 +-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
   2 +-+                                                 .+..               |   
     |..+.+..+.+..+.+..+.+..+.+..+.+..+..+.+..+.+..+.+..+    +.+..+.+..+.+..|   
   0 +-+--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          will-it-scale.time.system_time                        
                                                                                
  900 O-+--O-------O----O----O-------------------O----O---------------------+   
      |       O O    O    O    O  O O       O O    O    O                   |   
  800 +-+O                             O O                                  |   
      |                                                                     |   
      |                                                                     |   
  700 +-+                                                                   |   
      |                                                                     |   
  600 +-+                                                                   |   
      |                                                                     |   
  500 +-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
  400 +-+      .+.. .+..                             .+.  .+.+..+.    .+..+.|   
      |..+.+..+    +    +.+..+.+..+.+..+.+..+.+..+.+.   +.        +..+      |   
  300 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                  will-it-scale.time.percent_of_cpu_this_job_got                
                                                                                
  320 +-+-------------------------------------------------------------------+   
  300 +-+  O       O    O    O                   O    O                     |   
      O       O O    O    O    O  O O       O O    O    O                   |   
  280 +-+O                             O O                                  |   
  260 +-+                                                                   |   
  240 +-+                                                                   |   
  220 +-+                                                                   |   
      |                                                                     |   
  200 +-+                                                                   |   
  180 +-+                                                                   |   
  160 +-+                                                                   |   
  140 +-+                                                                   |   
      |                                                   .+.+..+.    .+..+.|   
  120 +-++.+..+.+..+.+..+.+..+.+..+.+..+.  .+.+..+.+..+.+.        +..+      |   
  100 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         will-it-scale.time.minor_page_faults                   
                                                                                
    5e+06 +-+---------O---------------------------O----O--------------------+   
  4.5e+06 O-+  O O O       O   O      O             O    O                  |   
          |                         O                                       |   
    4e+06 +-+           O    O    O        O O  O                           |   
  3.5e+06 +-O                            O                                  |   
          |                                                                 |   
    3e+06 +-+                                                               |   
  2.5e+06 +-+                                                               |   
    2e+06 +-+                                                               |   
          |                                                                 |   
  1.5e+06 +-+                                                               |   
    1e+06 +-+                                                               |   
          |                                                .+.+.+..   .+.. .|   
   500000 +-+..+.+.+..+.+..+.+.+..+.+.+..+. .+..+.+.+..+.+.        +.+    + |   
        0 +-+---------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    will-it-scale.time.voluntary_context_switches               
                                                                                
  1e+07 +-+-----------------------------------------------------------------+   
  9e+06 +-+         O                                                       |   
        O    O O  O      O    O      O           O  O O  O                  |   
  8e+06 +-+                        O                                        |   
  7e+06 +-+            O                    O  O                            |   
        | O                O    O       O O                                 |   
  6e+06 +-+                                                                 |   
  5e+06 +-+                                                                 |   
  4e+06 +-+                                                                 |   
        |                                                                   |   
  3e+06 +-+                                                                 |   
  2e+06 +-+                                                                 |   
        |                                                 .+.  .+     .+..  |   
  1e+06 +-+ .+.+..+.+..+.+.+..+.+..+.+..             .+..+   +.  +  .+    +.|   
      0 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/900t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/disk_cp/reaim/0x42e

commit: 
  8c779229d0 ("locking/lockdep: Report more stack trace statistics")
  91d2a812df ("locking/rwsem: Make handoff writer optimistically spin on owner")

8c779229d0f4fe83 91d2a812dfb98b3b4dad661529c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4            4%           0:4     perf-profile.children.cycles-pp.error_entry
          0:4            3%           0:4     perf-profile.self.cycles-pp.error_entry





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.3.0-rc1-00014-g91d2a812dfb98" of type "text/plain" (199532 bytes)

View attachment "job-script" of type "text/plain" (7178 bytes)

View attachment "job.yaml" of type "text/plain" (4810 bytes)

View attachment "reproduce" of type "text/plain" (311 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ