lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20190806062405.GD10969@shao2-debian>
Date:   Tue, 6 Aug 2019 14:24:05 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Waiman Long <longman@...hat.com>
Cc:     Ingo Molnar <mingo@...nel.org>,
        Luis Henriques <lhenriques@...e.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Borislav Petkov <bp@...en8.de>,
        Davidlohr Bueso <dave@...olabs.net>,
        "H. Peter Anvin" <hpa@...or.com>, Jeff Layton <jlayton@...nel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        Will Deacon <will.deacon@....com>,
        huang ying <huang.ying.caritas@...il.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...org
Subject: [locking/rwsem]  7813430057:  vm-scalability.median -42.5% regression

Greeting,

FYI, we noticed a -42.5% regression of vm-scalability.median due to commit:


commit: 78134300579a45f527ca173ec8fdb4701b69f16e ("locking/rwsem: Don't call owner_on_cpu() on read-owner")
https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git master

in testcase: vm-scalability
on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
with following parameters:

	runtime: 300s
	test: small-allocs-mt
	cpufreq_governor: performance
	ucode: 0xb000036

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/lkp-bdw-ep4/small-allocs-mt/vm-scalability/0xb000036

commit: 
  f9adc23ee9 ("futex: Cleanup generic SMP variant of arch_futex_atomic_op_inuser()")
  7813430057 ("locking/rwsem: Don't call owner_on_cpu() on read-owner")

f9adc23ee91e6f56 78134300579a45f527ca173ec8f 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -16%           0:4     perf-profile.children.cycles-pp.error_entry
          0:4          -12%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     10295 ± 10%     -42.5%       5918 ±  3%  vm-scalability.median
    904912 ± 10%     -42.5%     520174 ±  3%  vm-scalability.throughput
  60474168 ± 10%     -42.5%   34747417 ±  3%  vm-scalability.time.minor_page_faults
    551.25 ±  7%     +22.3%     674.25 ±  3%  vm-scalability.time.percent_of_cpu_this_job_got
      1523 ±  6%     +29.0%       1965 ±  3%  vm-scalability.time.system_time
    161.90 ± 13%     -51.5%      78.55 ±  4%  vm-scalability.time.user_time
  32285656 ± 13%     -74.5%    8235971 ±  3%  vm-scalability.time.voluntary_context_switches
 2.721e+08 ± 10%     -42.5%  1.563e+08 ±  3%  vm-scalability.workload
  37054490 ± 39%     -62.6%   13844254 ± 62%  cpuidle.C3.usage
      5.34 ±  6%      +2.0        7.38 ±  2%  mpstat.cpu.all.sys%
      0.59 ± 12%      -0.3        0.32 ±  4%  mpstat.cpu.all.usr%
     93.50            -1.6%      92.00        vmstat.cpu.id
    209869 ± 13%     -73.9%      54735 ±  3%  vmstat.system.cs
     29124 ±  9%     -42.5%      16745 ±  4%  numa-vmstat.node0.nr_page_table_pages
     91040 ±  8%     -35.0%      59192 ±  2%  numa-vmstat.node0.nr_slab_unreclaimable
     29038 ± 11%     -42.4%      16734 ±  4%  numa-vmstat.node1.nr_page_table_pages
     88159 ± 10%     -35.8%      56580 ±  4%  numa-vmstat.node1.nr_slab_unreclaimable
   2763837 ±  3%     -12.8%    2410986        meminfo.Memused
    232955 ± 10%     -42.6%     133795 ±  4%  meminfo.PageTables
    716735 ±  9%     -35.3%     463602 ±  3%  meminfo.SUnreclaim
    788566 ±  8%     -32.1%     535591 ±  3%  meminfo.Slab
     11681 ±  5%     -19.5%       9400        meminfo.max_used_kB
    240.75 ±  5%     +19.5%     287.75 ±  2%  turbostat.Avg_MHz
      8.68 ±  4%      +1.4       10.12 ±  3%  turbostat.Busy%
  37053983 ± 39%     -62.6%   13843816 ± 62%  turbostat.C3
     54.52 ±  6%     -25.0%      40.91 ± 14%  turbostat.CPU%c1
     12.86 ±  3%      -6.7%      12.00 ±  2%  turbostat.RAMWatt
    116649 ±  9%     -42.5%      67086 ±  4%  numa-meminfo.node0.PageTables
    364217 ±  8%     -34.9%     237199 ±  2%  numa-meminfo.node0.SUnreclaim
    401594 ±  5%     -30.7%     278271 ±  2%  numa-meminfo.node0.Slab
    116289 ± 11%     -42.3%      67047 ±  3%  numa-meminfo.node1.PageTables
    352683 ± 10%     -35.7%     226762 ±  4%  numa-meminfo.node1.SUnreclaim
    387136 ± 11%     -33.4%     257677 ±  4%  numa-meminfo.node1.Slab
    740.00 ±  3%     +10.8%     820.00 ±  3%  slabinfo.kmem_cache_node.active_objs
    783.25 ±  3%     +10.2%     863.00 ±  3%  slabinfo.kmem_cache_node.num_objs
   2947330 ± 11%     -43.3%    1670662 ±  4%  slabinfo.vm_area_struct.active_objs
     73683 ± 11%     -43.3%      41766 ±  4%  slabinfo.vm_area_struct.active_slabs
   2947343 ± 11%     -43.3%    1670680 ±  4%  slabinfo.vm_area_struct.num_objs
     73683 ± 11%     -43.3%      41766 ±  4%  slabinfo.vm_area_struct.num_slabs
     58082 ± 10%     -42.3%      33491 ±  4%  proc-vmstat.nr_page_table_pages
    179181 ±  9%     -35.3%     115969 ±  3%  proc-vmstat.nr_slab_unreclaimable
      8.75 ±133%   +8705.7%     770.50 ±149%  proc-vmstat.numa_hint_faults_local
   1030609 ±  2%     -10.7%     920710        proc-vmstat.numa_hit
   1014892 ±  2%     -11.0%     903417        proc-vmstat.numa_local
      8262 ±160%    +287.5%      32019 ± 65%  proc-vmstat.numa_pte_updates
      4248 ±  4%     +14.4%       4860 ±  5%  proc-vmstat.pgactivate
   1237129 ±  3%     -13.4%    1071267 ±  2%  proc-vmstat.pgalloc_normal
  61279755 ± 10%     -42.0%   35555364 ±  3%  proc-vmstat.pgfault
   1153261 ±  4%     -15.6%     972958 ± 12%  proc-vmstat.pgfree
     12276 ±  8%     +14.2%      14023 ±  4%  sched_debug.cfs_rq:/.exec_clock.max
    816.73 ±  7%     +29.3%       1056 ±  2%  sched_debug.cfs_rq:/.exec_clock.stddev
     90308 ±  8%     +33.2%     120279 ± 13%  sched_debug.cfs_rq:/.min_vruntime.avg
    122001 ±  6%     +30.7%     159494 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
      8108 ±  6%     +35.2%      10962 ±  2%  sched_debug.cfs_rq:/.min_vruntime.stddev
      3.80 ± 28%     +49.4%       5.67 ±  2%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     19.20 ± 24%     +33.7%      25.68 ± 11%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     50559 ± 12%     +32.5%      67006 ±  2%  sched_debug.cfs_rq:/.spread0.max
      8108 ±  6%     +35.2%      10962 ±  2%  sched_debug.cfs_rq:/.spread0.stddev
    715.29 ± 11%     +31.0%     937.06 ±  3%  sched_debug.cfs_rq:/.util_avg.max
      0.62 ± 96%    -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.min
    136.54 ± 16%     +49.4%     204.01 ± 11%  sched_debug.cfs_rq:/.util_avg.stddev
    248.08 ±  6%     +44.4%     358.25 ± 19%  sched_debug.cfs_rq:/.util_est_enqueued.max
    472136 ± 20%     +74.7%     824658 ±  3%  sched_debug.cpu.avg_idle.avg
     48568 ± 17%     -49.5%      24506 ± 29%  sched_debug.cpu.avg_idle.min
    161426 ± 19%     +27.8%     206264 ±  4%  sched_debug.cpu.avg_idle.stddev
      2.64 ±  3%      +7.0%       2.83 ±  3%  sched_debug.cpu.clock.stddev
      2.64 ±  3%      +7.0%       2.83 ±  3%  sched_debug.cpu.clock_task.stddev
    362539 ± 13%     -75.3%      89466 ± 12%  sched_debug.cpu.nr_switches.avg
    385763 ± 12%     -71.0%     111713 ± 13%  sched_debug.cpu.nr_switches.max
    339521 ± 14%     -78.2%      74098 ± 14%  sched_debug.cpu.nr_switches.min
      8.45 ±  6%     -17.0%       7.01 ±  9%  sched_debug.cpu.nr_uninterruptible.stddev
    363377 ± 13%     -75.2%      90290 ± 13%  sched_debug.cpu.sched_count.avg
    408058 ± 13%     -65.0%     142960 ± 15%  sched_debug.cpu.sched_count.max
    339524 ± 14%     -78.1%      74224 ± 14%  sched_debug.cpu.sched_count.min
    180453 ± 13%     -75.7%      43933 ± 13%  sched_debug.cpu.sched_goidle.avg
    191582 ± 12%     -71.6%      54363 ± 14%  sched_debug.cpu.sched_goidle.max
    169052 ± 14%     -78.5%      36425 ± 14%  sched_debug.cpu.sched_goidle.min
    183653 ± 13%     -75.1%      45783 ± 13%  sched_debug.cpu.ttwu_count.avg
    202354 ± 13%     -72.8%      55054 ± 15%  sched_debug.cpu.ttwu_count.max
    168679 ± 13%     -75.2%      41903 ± 13%  sched_debug.cpu.ttwu_count.min
      8022 ± 24%     -75.1%       2000 ± 12%  sched_debug.cpu.ttwu_count.stddev
      1218 ±  6%     +20.2%       1464 ±  6%  sched_debug.cpu.ttwu_local.max
    164.12 ± 17%     +41.9%     232.82 ±  3%  sched_debug.cpu.ttwu_local.stddev
      8.44 ± 11%     -36.3%       5.38 ± 11%  perf-stat.i.MPKI
 3.178e+09 ±  4%     +15.6%  3.674e+09        perf-stat.i.branch-instructions
   9232346 ±  8%     -38.0%    5722174 ±  9%  perf-stat.i.cache-misses
 1.006e+08 ± 15%     -26.2%   74257221 ±  6%  perf-stat.i.cache-references
    212153 ± 13%     -73.9%      55331 ±  3%  perf-stat.i.context-switches
 2.115e+10 ±  5%     +18.7%  2.511e+10 ±  2%  perf-stat.i.cpu-cycles
    482.97 ± 22%     -73.1%     129.87 ±  5%  perf-stat.i.cpu-migrations
      2713 ±  9%     +98.8%       5392 ± 16%  perf-stat.i.cycles-between-cache-misses
      0.20 ±  4%      -0.1        0.09 ±  2%  perf-stat.i.dTLB-load-miss-rate%
   6517636 ± 10%     -48.8%    3334950 ±  2%  perf-stat.i.dTLB-load-misses
 3.377e+09 ±  6%     +26.6%  4.274e+09 ±  2%  perf-stat.i.dTLB-loads
   1125113 ±  8%     -25.5%     838294 ±  4%  perf-stat.i.dTLB-store-misses
 7.672e+08 ±  5%     -21.6%  6.017e+08        perf-stat.i.dTLB-stores
     40.86 ± 11%     +10.2       51.09 ±  6%  perf-stat.i.iTLB-load-miss-rate%
   1489763 ± 10%     -34.9%     970146 ±  6%  perf-stat.i.iTLB-load-misses
   2248250 ± 14%     -58.7%     928739 ±  6%  perf-stat.i.iTLB-loads
 1.262e+10 ±  4%     +23.4%  1.558e+10        perf-stat.i.instructions
      8679 ±  6%     +87.6%      16280 ±  5%  perf-stat.i.instructions-per-iTLB-miss
      0.59            +4.0%       0.62        perf-stat.i.ipc
    200189 ± 10%     -41.7%     116680 ±  2%  perf-stat.i.minor-faults
   3344481 ± 10%     -52.0%    1606079 ±  3%  perf-stat.i.node-load-misses
     84375 ± 20%     -56.2%      36926 ± 37%  perf-stat.i.node-loads
   1565682 ± 10%     -47.0%     830068 ±  3%  perf-stat.i.node-store-misses
    869275 ± 12%     -46.0%     469789 ±  3%  perf-stat.i.node-stores
    200189 ± 10%     -41.7%     116680 ±  2%  perf-stat.i.page-faults
      7.94 ± 11%     -39.9%       4.77 ±  6%  perf-stat.overall.MPKI
      1.68            -3.8%       1.61        perf-stat.overall.cpi
      2299 ±  4%     +92.8%       4434 ± 11%  perf-stat.overall.cycles-between-cache-misses
      0.19 ±  4%      -0.1        0.08 ±  2%  perf-stat.overall.dTLB-load-miss-rate%
     40.08 ± 10%     +11.0       51.08 ±  5%  perf-stat.overall.iTLB-load-miss-rate%
      8525 ±  6%     +89.0%      16114 ±  5%  perf-stat.overall.instructions-per-iTLB-miss
      0.60            +4.0%       0.62        perf-stat.overall.ipc
     14280 ±  5%    +112.6%      30354 ±  2%  perf-stat.overall.path-length
 3.167e+09 ±  4%     +15.6%  3.662e+09        perf-stat.ps.branch-instructions
   9201666 ±  8%     -38.0%    5703837 ±  9%  perf-stat.ps.cache-misses
 1.002e+08 ± 15%     -26.2%   74012239 ±  6%  perf-stat.ps.cache-references
    211434 ± 13%     -73.9%      55142 ±  3%  perf-stat.ps.context-switches
 2.108e+10 ±  5%     +18.7%  2.502e+10 ±  2%  perf-stat.ps.cpu-cycles
    481.54 ± 22%     -73.1%     129.46 ±  5%  perf-stat.ps.cpu-migrations
   6495785 ± 10%     -48.8%    3323804 ±  2%  perf-stat.ps.dTLB-load-misses
 3.365e+09 ±  6%     +26.6%   4.26e+09 ±  2%  perf-stat.ps.dTLB-loads
   1121400 ±  8%     -25.5%     835514 ±  4%  perf-stat.ps.dTLB-store-misses
 7.647e+08 ±  5%     -21.6%  5.997e+08        perf-stat.ps.dTLB-stores
   1484812 ± 10%     -34.9%     966928 ±  6%  perf-stat.ps.iTLB-load-misses
   2240685 ± 14%     -58.7%     925639 ±  6%  perf-stat.ps.iTLB-loads
 1.258e+10 ±  4%     +23.4%  1.552e+10        perf-stat.ps.instructions
    199509 ± 10%     -41.7%     116282 ±  2%  perf-stat.ps.minor-faults
   3333192 ± 10%     -52.0%    1600811 ±  3%  perf-stat.ps.node-load-misses
     84222 ± 20%     -56.0%      37029 ± 37%  perf-stat.ps.node-loads
   1560349 ± 10%     -47.0%     827246 ±  3%  perf-stat.ps.node-store-misses
    866336 ± 12%     -46.0%     468214 ±  3%  perf-stat.ps.node-stores
    199509 ± 10%     -41.7%     116282 ±  2%  perf-stat.ps.page-faults
 3.862e+12 ±  4%     +22.8%  4.743e+12 ±  2%  perf-stat.total.instructions
     15090 ±  9%     -30.9%      10424 ±  6%  softirqs.CPU0.RCU
     14722 ±  5%     -28.2%      10570 ±  4%  softirqs.CPU1.RCU
     14675 ±  6%     -25.8%      10884 ±  4%  softirqs.CPU10.RCU
     14751 ±  4%     -27.5%      10698 ±  8%  softirqs.CPU11.RCU
     14902 ±  8%     -27.8%      10765 ±  7%  softirqs.CPU12.RCU
     14421 ±  3%     -26.2%      10637 ±  7%  softirqs.CPU13.RCU
     14747 ±  5%     -27.1%      10757 ±  6%  softirqs.CPU14.RCU
     14293           -27.0%      10439 ±  7%  softirqs.CPU15.RCU
     14692 ±  8%     -28.9%      10443 ±  6%  softirqs.CPU16.RCU
     13768 ±  5%     -25.4%      10275 ±  6%  softirqs.CPU17.RCU
     13952 ±  6%     -28.1%      10030 ±  6%  softirqs.CPU18.RCU
     13840 ±  3%     -27.2%      10079 ±  6%  softirqs.CPU19.RCU
     19253 ±  4%     -19.3%      15537 ±  4%  softirqs.CPU2.RCU
     14429 ±  5%     -27.2%      10501 ±  7%  softirqs.CPU20.RCU
     13775 ±  4%     -20.6%      10943 ± 12%  softirqs.CPU21.RCU
     15718 ±  7%     -28.4%      11254 ±  6%  softirqs.CPU22.RCU
     14218 ±  4%     -24.3%      10769 ± 10%  softirqs.CPU23.RCU
     13918 ±  4%     -27.7%      10065 ±  7%  softirqs.CPU24.RCU
     13805 ±  5%     -24.3%      10447 ± 16%  softirqs.CPU25.RCU
     14106 ±  5%     -27.2%      10269 ±  8%  softirqs.CPU26.RCU
     14291 ±  5%     -28.4%      10237 ±  7%  softirqs.CPU27.RCU
     14123 ±  4%     -23.4%      10821 ±  7%  softirqs.CPU28.RCU
     13957 ±  5%     -26.7%      10235 ±  7%  softirqs.CPU29.RCU
     14378 ±  3%     -22.0%      11219 ± 15%  softirqs.CPU3.RCU
     14266 ±  3%     -22.5%      11060 ± 12%  softirqs.CPU30.RCU
     13865 ±  4%     -30.4%       9645 ±  7%  softirqs.CPU31.RCU
     14048 ±  4%     -29.3%       9932 ±  5%  softirqs.CPU32.RCU
     14296 ±  4%     -29.6%      10069 ± 12%  softirqs.CPU33.RCU
     14998 ±  8%     -26.4%      11037 ± 11%  softirqs.CPU34.RCU
     14342 ±  5%     -27.1%      10453 ±  5%  softirqs.CPU35.RCU
     14865 ±  4%     -26.4%      10942 ±  3%  softirqs.CPU36.RCU
     14698 ±  4%     -26.3%      10834 ±  4%  softirqs.CPU37.RCU
     14639 ±  5%     -24.6%      11036 ±  6%  softirqs.CPU38.RCU
     13850 ±  8%     -23.8%      10559 ±  5%  softirqs.CPU39.RCU
     14451 ±  3%     -24.8%      10868 ±  6%  softirqs.CPU4.RCU
     14373 ±  5%     -23.8%      10958 ±  2%  softirqs.CPU40.RCU
     14794 ±  5%     -25.7%      10990 ±  2%  softirqs.CPU41.RCU
     14561 ±  5%     -25.4%      10866 ±  3%  softirqs.CPU42.RCU
     14823 ±  4%     -24.0%      11261 ± 10%  softirqs.CPU43.RCU
     11986 ±  3%     -24.8%       9019 ±  2%  softirqs.CPU44.RCU
     14143 ±  5%     -26.1%      10446 ±  6%  softirqs.CPU45.RCU
     14225 ±  4%     -26.3%      10487 ±  8%  softirqs.CPU46.RCU
     15111 ± 10%     -27.9%      10899 ±  7%  softirqs.CPU47.RCU
     14841 ± 10%     -28.5%      10611 ±  6%  softirqs.CPU48.RCU
     14323 ±  5%     -22.3%      11123 ±  3%  softirqs.CPU49.RCU
     14523 ±  3%     -26.5%      10680 ±  5%  softirqs.CPU5.RCU
     14354 ±  4%     -27.4%      10414 ±  6%  softirqs.CPU50.RCU
     14319 ±  4%     -22.5%      11099 ± 12%  softirqs.CPU51.RCU
     13868 ±  5%     -28.7%       9882 ±  9%  softirqs.CPU52.RCU
     14449 ±  4%     -26.9%      10559 ± 11%  softirqs.CPU53.RCU
     14865 ±  8%     -30.3%      10366 ±  7%  softirqs.CPU54.RCU
     14631 ±  2%     -23.3%      11227 ± 15%  softirqs.CPU55.RCU
     15058 ±  4%     -28.2%      10810 ±  7%  softirqs.CPU56.RCU
     14406 ±  4%     -25.8%      10685 ±  6%  softirqs.CPU57.RCU
     14768 ±  3%     -26.1%      10908 ±  6%  softirqs.CPU58.RCU
     14789 ±  6%     -26.6%      10857 ±  5%  softirqs.CPU59.RCU
     14731 ±  5%     -26.6%      10812 ±  6%  softirqs.CPU6.RCU
     13350 ±  2%     -24.3%      10108 ±  7%  softirqs.CPU60.RCU
     13291 ±  5%     -26.8%       9732 ±  8%  softirqs.CPU61.RCU
     13438 ±  6%     -27.1%       9803 ±  8%  softirqs.CPU62.RCU
     13352 ±  4%     -26.0%       9882 ±  6%  softirqs.CPU63.RCU
     13710 ±  5%     -25.2%      10260 ±  6%  softirqs.CPU64.RCU
     13618 ±  4%     -29.0%       9667 ±  8%  softirqs.CPU65.RCU
     14217 ±  6%     -25.3%      10615 ±  7%  softirqs.CPU66.RCU
     14102 ±  5%     -26.7%      10333 ±  8%  softirqs.CPU67.RCU
     14124 ±  4%     -27.1%      10295 ±  8%  softirqs.CPU68.RCU
     13718 ±  5%     -32.5%       9256 ±  9%  softirqs.CPU69.RCU
     14427 ±  4%     -25.3%      10776 ±  4%  softirqs.CPU7.RCU
     13928 ±  4%     -26.2%      10273 ±  9%  softirqs.CPU70.RCU
     13906 ±  6%     -26.0%      10295 ±  6%  softirqs.CPU71.RCU
     14128 ±  5%     -26.1%      10438 ±  8%  softirqs.CPU72.RCU
     14056 ±  4%     -27.3%      10212 ±  7%  softirqs.CPU73.RCU
     14121 ±  5%     -27.0%      10306 ± 10%  softirqs.CPU74.RCU
     13156 ±  4%     -24.5%       9933 ± 15%  softirqs.CPU75.RCU
     13735 ±  5%     -28.0%       9894 ± 15%  softirqs.CPU76.RCU
     13347 ±  4%     -22.9%      10286 ±  3%  softirqs.CPU77.RCU
     13722 ±  9%     -27.4%       9968 ±  7%  softirqs.CPU78.RCU
     13310 ±  5%     -27.4%       9660 ±  6%  softirqs.CPU79.RCU
     14992 ± 11%     -31.8%      10220 ±  9%  softirqs.CPU8.RCU
     13988 ±  5%     -27.7%      10118 ±  5%  softirqs.CPU80.RCU
     13918 ±  4%     -27.0%      10156 ±  5%  softirqs.CPU81.RCU
     13849 ±  5%     -27.0%      10110 ±  5%  softirqs.CPU82.RCU
     14076 ±  6%     -27.2%      10248        softirqs.CPU83.RCU
     13473 ±  5%     -24.3%      10205        softirqs.CPU84.RCU
     13814 ±  5%     -25.2%      10334 ±  5%  softirqs.CPU85.RCU
     13938 ±  4%     -27.7%      10077 ±  4%  softirqs.CPU86.RCU
     14634 ±  9%     -30.4%      10189 ±  5%  softirqs.CPU87.RCU
     13692 ± 11%     -25.4%      10219 ±  9%  softirqs.CPU9.RCU
   1255273 ±  4%     -26.4%     923618 ±  6%  softirqs.RCU
    308.00 ± 38%     -41.3%     180.75 ±  7%  interrupts.36:PCI-MSI.1572867-edge.eth0-TxRx-3
    296.75 ± 31%     -31.6%     203.00 ± 12%  interrupts.42:PCI-MSI.1572873-edge.eth0-TxRx-9
    196.50 ± 10%     -17.4%     162.25 ±  3%  interrupts.48:PCI-MSI.1572879-edge.eth0-TxRx-15
      5727 ±  7%     -38.4%       3530 ± 19%  interrupts.CPU0.RES:Rescheduling_interrupts
      5869 ± 14%     -43.1%       3342 ± 10%  interrupts.CPU1.RES:Rescheduling_interrupts
      5447 ±  9%     -40.8%       3226 ±  7%  interrupts.CPU10.RES:Rescheduling_interrupts
     40.50 ±101%    +126.5%      91.75 ± 38%  interrupts.CPU10.TLB:TLB_shootdowns
      5312 ± 11%     -40.9%       3137 ± 15%  interrupts.CPU11.RES:Rescheduling_interrupts
     49.00 ± 63%     +78.1%      87.25 ± 27%  interrupts.CPU11.TLB:TLB_shootdowns
      5528 ±  7%     -40.0%       3316 ± 15%  interrupts.CPU12.RES:Rescheduling_interrupts
      5491 ±  8%     -44.6%       3042 ±  6%  interrupts.CPU13.RES:Rescheduling_interrupts
      5147 ±  8%     -44.2%       2873 ± 14%  interrupts.CPU14.RES:Rescheduling_interrupts
    196.50 ± 10%     -17.4%     162.25 ±  3%  interrupts.CPU15.48:PCI-MSI.1572879-edge.eth0-TxRx-15
      5139 ±  7%     -50.1%       2566 ± 15%  interrupts.CPU15.RES:Rescheduling_interrupts
      5509 ± 11%     -36.7%       3489 ± 10%  interrupts.CPU16.RES:Rescheduling_interrupts
      5478 ±  5%     -44.1%       3065 ± 16%  interrupts.CPU17.RES:Rescheduling_interrupts
      5258 ± 12%     -36.9%       3319 ±  9%  interrupts.CPU18.RES:Rescheduling_interrupts
      5652 ± 11%     -35.5%       3643 ± 17%  interrupts.CPU19.RES:Rescheduling_interrupts
      5383 ± 14%     -45.1%       2954 ± 13%  interrupts.CPU2.RES:Rescheduling_interrupts
      5356 ±  9%     -46.0%       2894 ± 14%  interrupts.CPU20.RES:Rescheduling_interrupts
      5763 ± 19%     -50.8%       2836 ± 11%  interrupts.CPU21.RES:Rescheduling_interrupts
      5639 ±  7%     -39.4%       3418 ± 12%  interrupts.CPU22.RES:Rescheduling_interrupts
      5579 ±  6%     -33.4%       3713 ± 25%  interrupts.CPU23.RES:Rescheduling_interrupts
    912.75 ± 33%     +45.7%       1329 ±  6%  interrupts.CPU24.NMI:Non-maskable_interrupts
    912.75 ± 33%     +45.7%       1329 ±  6%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      5440 ±  6%     -39.2%       3305 ± 13%  interrupts.CPU24.RES:Rescheduling_interrupts
      5716 ±  7%     -36.5%       3628 ±  9%  interrupts.CPU25.RES:Rescheduling_interrupts
      5544 ±  8%     -37.3%       3474 ±  4%  interrupts.CPU26.RES:Rescheduling_interrupts
    948.00 ± 35%     +67.1%       1583 ± 17%  interrupts.CPU27.NMI:Non-maskable_interrupts
    948.00 ± 35%     +67.1%       1583 ± 17%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      5776 ±  7%     -31.0%       3985 ± 14%  interrupts.CPU27.RES:Rescheduling_interrupts
    972.75 ± 24%     +57.4%       1531 ± 14%  interrupts.CPU28.NMI:Non-maskable_interrupts
    972.75 ± 24%     +57.4%       1531 ± 14%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
      5638 ±  6%     -40.9%       3334 ± 15%  interrupts.CPU28.RES:Rescheduling_interrupts
      5537 ±  5%     -30.1%       3869 ± 22%  interrupts.CPU29.RES:Rescheduling_interrupts
    308.00 ± 38%     -41.3%     180.75 ±  7%  interrupts.CPU3.36:PCI-MSI.1572867-edge.eth0-TxRx-3
      5446 ± 14%     -39.7%       3282 ±  8%  interrupts.CPU3.RES:Rescheduling_interrupts
      1284 ±  3%     +21.6%       1561 ±  5%  interrupts.CPU30.NMI:Non-maskable_interrupts
      1284 ±  3%     +21.6%       1561 ±  5%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      5483 ± 12%     -43.1%       3119 ± 15%  interrupts.CPU30.RES:Rescheduling_interrupts
      5823 ±  7%     -42.4%       3356 ± 14%  interrupts.CPU31.RES:Rescheduling_interrupts
     28.25 ± 89%    +187.6%      81.25 ± 40%  interrupts.CPU31.TLB:TLB_shootdowns
      5776 ±  7%     -33.6%       3836 ± 13%  interrupts.CPU32.RES:Rescheduling_interrupts
      5675 ± 10%     -45.1%       3117 ± 12%  interrupts.CPU33.RES:Rescheduling_interrupts
      1170 ±  3%     +23.4%       1443 ± 11%  interrupts.CPU34.NMI:Non-maskable_interrupts
      1170 ±  3%     +23.4%       1443 ± 11%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      5754 ±  9%     -43.7%       3240 ± 14%  interrupts.CPU34.RES:Rescheduling_interrupts
      1252 ±  6%     +15.6%       1447 ±  8%  interrupts.CPU35.NMI:Non-maskable_interrupts
      1252 ±  6%     +15.6%       1447 ±  8%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      5266 ±  8%     -41.2%       3099 ± 11%  interrupts.CPU35.RES:Rescheduling_interrupts
      5239 ± 10%     -41.2%       3082 ± 23%  interrupts.CPU36.RES:Rescheduling_interrupts
      5576 ± 12%     -43.0%       3175 ± 11%  interrupts.CPU37.RES:Rescheduling_interrupts
      5176 ±  6%     -36.5%       3287 ± 16%  interrupts.CPU38.RES:Rescheduling_interrupts
      5559 ±  6%     -35.5%       3583 ± 19%  interrupts.CPU39.RES:Rescheduling_interrupts
      5568 ±  9%     -41.9%       3233 ± 15%  interrupts.CPU4.RES:Rescheduling_interrupts
      1096 ± 21%     +37.8%       1511 ± 11%  interrupts.CPU40.NMI:Non-maskable_interrupts
      1096 ± 21%     +37.8%       1511 ± 11%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      5151 ±  8%     -34.8%       3360 ± 10%  interrupts.CPU40.RES:Rescheduling_interrupts
      5181 ±  9%     -42.5%       2981 ± 16%  interrupts.CPU41.RES:Rescheduling_interrupts
      1139 ± 28%     +36.7%       1557 ±  6%  interrupts.CPU42.NMI:Non-maskable_interrupts
      1139 ± 28%     +36.7%       1557 ±  6%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      4909 ± 10%     -36.5%       3117 ± 18%  interrupts.CPU42.RES:Rescheduling_interrupts
      5249 ±  5%     -37.2%       3296 ±  9%  interrupts.CPU43.RES:Rescheduling_interrupts
      1146 ± 29%     +31.0%       1501 ±  8%  interrupts.CPU44.NMI:Non-maskable_interrupts
      1146 ± 29%     +31.0%       1501 ±  8%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      5653 ± 20%     -45.4%       3084 ±  7%  interrupts.CPU44.RES:Rescheduling_interrupts
      5853 ±  5%     -48.0%       3042 ±  3%  interrupts.CPU45.RES:Rescheduling_interrupts
     32.75 ± 47%    +221.4%     105.25 ± 45%  interrupts.CPU45.TLB:TLB_shootdowns
      5898 ± 10%     -43.1%       3358 ± 10%  interrupts.CPU46.RES:Rescheduling_interrupts
      5392 ±  9%     -39.0%       3290 ± 18%  interrupts.CPU47.RES:Rescheduling_interrupts
      5924 ±  6%     -46.5%       3170 ± 11%  interrupts.CPU48.RES:Rescheduling_interrupts
      5938 ±  5%     -41.4%       3480 ±  8%  interrupts.CPU49.RES:Rescheduling_interrupts
     31.00 ± 63%    +177.4%      86.00 ± 39%  interrupts.CPU49.TLB:TLB_shootdowns
      5543 ±  9%     -42.4%       3193 ±  7%  interrupts.CPU5.RES:Rescheduling_interrupts
      5277 ±  5%     -41.4%       3092 ±  3%  interrupts.CPU50.RES:Rescheduling_interrupts
      6101 ±  9%     -44.3%       3399 ± 10%  interrupts.CPU51.RES:Rescheduling_interrupts
      5834 ± 12%     -40.3%       3481 ±  6%  interrupts.CPU52.RES:Rescheduling_interrupts
      6214 ±  9%     -40.2%       3717 ± 12%  interrupts.CPU53.RES:Rescheduling_interrupts
     25.75 ± 95%    +198.1%      76.75 ± 46%  interrupts.CPU53.TLB:TLB_shootdowns
      6278 ± 11%     -52.7%       2967 ± 13%  interrupts.CPU54.RES:Rescheduling_interrupts
      5668 ± 11%     -43.4%       3205 ±  4%  interrupts.CPU55.RES:Rescheduling_interrupts
      5820 ±  8%     -39.9%       3500 ±  3%  interrupts.CPU56.RES:Rescheduling_interrupts
      5928 ±  6%     -43.5%       3350 ±  8%  interrupts.CPU57.RES:Rescheduling_interrupts
     35.00 ± 34%    +127.1%      79.50 ± 29%  interrupts.CPU57.TLB:TLB_shootdowns
      5632 ± 11%     -44.7%       3115 ± 16%  interrupts.CPU58.RES:Rescheduling_interrupts
      5440 ±  8%     -47.4%       2863 ±  8%  interrupts.CPU59.RES:Rescheduling_interrupts
      5485 ± 12%     -44.5%       3044 ±  8%  interrupts.CPU6.RES:Rescheduling_interrupts
      1153 ± 25%     +32.2%       1525 ±  7%  interrupts.CPU60.NMI:Non-maskable_interrupts
      1153 ± 25%     +32.2%       1525 ±  7%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      5514 ±  6%     -45.5%       3006 ± 11%  interrupts.CPU60.RES:Rescheduling_interrupts
      5600 ±  7%     -36.7%       3547 ± 14%  interrupts.CPU61.RES:Rescheduling_interrupts
      5612 ±  7%     -30.8%       3885 ± 27%  interrupts.CPU62.RES:Rescheduling_interrupts
      1214 ±  5%     +22.3%       1485 ±  5%  interrupts.CPU63.NMI:Non-maskable_interrupts
      1214 ±  5%     +22.3%       1485 ±  5%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
      6114 ± 11%     -45.5%       3335 ± 13%  interrupts.CPU63.RES:Rescheduling_interrupts
      5297 ± 14%     -45.4%       2891 ± 17%  interrupts.CPU64.RES:Rescheduling_interrupts
      1309 ± 10%     +23.4%       1616 ±  9%  interrupts.CPU65.NMI:Non-maskable_interrupts
      1309 ± 10%     +23.4%       1616 ±  9%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
      5366 ±  8%     -49.0%       2738 ± 17%  interrupts.CPU65.RES:Rescheduling_interrupts
      1267 ±  6%     +23.6%       1566 ± 13%  interrupts.CPU66.NMI:Non-maskable_interrupts
      1267 ±  6%     +23.6%       1566 ± 13%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
      5244 ± 12%     -37.8%       3261 ± 15%  interrupts.CPU66.RES:Rescheduling_interrupts
      5758 ± 11%     -41.1%       3390 ± 23%  interrupts.CPU67.RES:Rescheduling_interrupts
      5453 ±  7%     -35.8%       3501 ± 12%  interrupts.CPU68.RES:Rescheduling_interrupts
      5786 ± 11%     -41.0%       3413 ± 20%  interrupts.CPU69.RES:Rescheduling_interrupts
      5628 ± 10%     -45.4%       3072 ±  6%  interrupts.CPU7.RES:Rescheduling_interrupts
      1199 ±  3%     +23.0%       1474 ±  7%  interrupts.CPU70.NMI:Non-maskable_interrupts
      1199 ±  3%     +23.0%       1474 ±  7%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
      5714 ±  7%     -32.3%       3870 ± 13%  interrupts.CPU70.RES:Rescheduling_interrupts
      1242 ±  2%     +22.5%       1521 ±  8%  interrupts.CPU71.NMI:Non-maskable_interrupts
      1242 ±  2%     +22.5%       1521 ±  8%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      5650 ± 10%     -38.1%       3496 ± 13%  interrupts.CPU71.RES:Rescheduling_interrupts
     73.75 ± 33%     -55.3%      33.00 ± 40%  interrupts.CPU71.TLB:TLB_shootdowns
      1298 ± 14%     +16.8%       1515 ± 12%  interrupts.CPU72.NMI:Non-maskable_interrupts
      1298 ± 14%     +16.8%       1515 ± 12%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
      5255 ±  6%     -45.6%       2857 ± 15%  interrupts.CPU72.RES:Rescheduling_interrupts
      5522 ±  7%     -37.1%       3473 ± 19%  interrupts.CPU73.RES:Rescheduling_interrupts
      1255 ±  5%     +19.3%       1497 ± 10%  interrupts.CPU74.NMI:Non-maskable_interrupts
      1255 ±  5%     +19.3%       1497 ± 10%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
      5540 ±  8%     -39.1%       3371 ± 18%  interrupts.CPU74.RES:Rescheduling_interrupts
      5489 ±  3%     -45.6%       2986 ± 17%  interrupts.CPU75.RES:Rescheduling_interrupts
      5618 ±  7%     -37.9%       3489 ± 22%  interrupts.CPU76.RES:Rescheduling_interrupts
      1284 ± 13%     +30.2%       1673 ± 12%  interrupts.CPU77.NMI:Non-maskable_interrupts
      1284 ± 13%     +30.2%       1673 ± 12%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      5621 ±  6%     -29.8%       3943 ± 19%  interrupts.CPU77.RES:Rescheduling_interrupts
      1166 ±  4%     +24.9%       1456 ±  9%  interrupts.CPU78.NMI:Non-maskable_interrupts
      1166 ±  4%     +24.9%       1456 ±  9%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
      5624 ±  9%     -36.3%       3585 ± 19%  interrupts.CPU78.RES:Rescheduling_interrupts
     30.75 ± 81%    +166.7%      82.00 ± 47%  interrupts.CPU78.TLB:TLB_shootdowns
      5642 ± 10%     -33.2%       3772 ± 15%  interrupts.CPU79.RES:Rescheduling_interrupts
      5223 ±  8%     -35.4%       3374 ±  8%  interrupts.CPU8.RES:Rescheduling_interrupts
      1309 ± 10%     +23.4%       1615 ± 11%  interrupts.CPU80.NMI:Non-maskable_interrupts
      1309 ± 10%     +23.4%       1615 ± 11%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      5464 ± 12%     -35.6%       3521 ± 20%  interrupts.CPU80.RES:Rescheduling_interrupts
      1254 ±  5%     +28.0%       1606 ±  5%  interrupts.CPU81.NMI:Non-maskable_interrupts
      1254 ±  5%     +28.0%       1606 ±  5%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      5620 ±  7%     -40.9%       3323 ±  9%  interrupts.CPU81.RES:Rescheduling_interrupts
      1210 ±  2%     +20.2%       1454 ±  7%  interrupts.CPU82.NMI:Non-maskable_interrupts
      1210 ±  2%     +20.2%       1454 ±  7%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
      5422 ±  8%     -37.6%       3382 ± 20%  interrupts.CPU82.RES:Rescheduling_interrupts
      5565 ±  4%     -36.0%       3559 ± 14%  interrupts.CPU83.RES:Rescheduling_interrupts
      5603 ±  7%     -33.3%       3739 ± 16%  interrupts.CPU84.RES:Rescheduling_interrupts
      5346 ±  5%     -34.6%       3498 ± 15%  interrupts.CPU85.RES:Rescheduling_interrupts
      5122 ±  8%     -41.4%       3000 ± 11%  interrupts.CPU86.RES:Rescheduling_interrupts
      5468 ±  7%     -35.8%       3509 ± 12%  interrupts.CPU87.RES:Rescheduling_interrupts
    296.75 ± 31%     -31.6%     203.00 ± 12%  interrupts.CPU9.42:PCI-MSI.1572873-edge.eth0-TxRx-9
      5324 ± 10%     -42.6%       3053 ±  7%  interrupts.CPU9.RES:Rescheduling_interrupts
    102430 ±  6%     +16.8%     119675 ±  6%  interrupts.NMI:Non-maskable_interrupts
    102430 ±  6%     +16.8%     119675 ±  6%  interrupts.PMI:Performance_monitoring_interrupts
    488877 ±  8%     -40.4%     291343 ±  3%  interrupts.RES:Rescheduling_interrupts
     67.21 ±  2%     -13.1       54.11 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     66.51 ±  2%     -12.9       53.62 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     66.51 ±  2%     -12.9       53.62 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     66.47 ±  2%     -12.9       53.59 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     51.36            -8.5       42.83 ±  3%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     56.55            -7.7       48.81 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     56.16            -7.7       48.47 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      3.27 ± 17%      -2.3        0.98 ± 17%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.45 ± 16%      -2.1        0.31 ±100%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      2.72 ± 10%      -1.9        0.77 ± 12%  perf-profile.calltrace.cycles-pp.rwsem_wake.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.73 ± 17%      -1.9        0.83 ± 17%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      2.72 ± 17%      -1.9        0.83 ± 17%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      2.71 ± 17%      -1.9        0.83 ± 17%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
      2.52 ± 17%      -1.7        0.78 ± 16%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      2.30 ± 10%      -1.6        0.67 ± 12%  perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      2.08 ± 17%      -1.6        0.47 ± 59%  perf-profile.calltrace.cycles-pp.schedule.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault
      2.26 ± 10%      -1.6        0.66 ± 13%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.vm_mmap_pgoff.ksys_mmap_pgoff
      2.31 ± 18%      -1.6        0.71 ± 12%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.29 ± 18%      -1.6        0.71 ± 11%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      2.04 ± 17%      -1.6        0.47 ± 59%  perf-profile.calltrace.cycles-pp.__schedule.schedule.rwsem_down_read_slowpath.__do_page_fault.do_page_fault
     10.72 ±  4%      -1.4        9.31 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     10.71 ±  4%      -1.4        9.31 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.47 ±  4%      -1.3        9.16 ± 11%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.39 ±  4%      -1.3        9.12 ± 11%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.63 ± 13%      -1.0        0.67 ±  9%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.09 ± 14%      -0.8        0.29 ±100%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      1.28 ± 14%      -0.7        0.62 ± 11%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      2.83 ±  8%      -0.6        2.25 ±  8%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.80 ±  7%      -0.3        0.46 ± 59%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      1.42 ± 10%      -0.3        1.09 ±  9%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      1.10 ±  8%      -0.3        0.85 ± 10%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.63 ± 13%      +0.2        0.82 ± 14%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.70 ± 14%      +0.2        0.90 ± 14%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.88 ± 13%      +0.2        1.10 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.46 ±  9%      +0.3        1.79 ±  7%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.74 ± 20%      +1.6        2.31 ± 10%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault
      5.93 ±  2%      +1.7        7.62 ± 12%  perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.86 ±  2%      +1.7        7.58 ± 12%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      4.87 ±  4%      +2.2        7.11 ± 13%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
      3.71 ±  5%      +2.3        6.04 ± 14%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
     19.29 ± 10%     +16.1       35.34 ±  4%  perf-profile.calltrace.cycles-pp.page_fault
     19.24 ± 10%     +16.1       35.32 ±  4%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
     19.09 ± 10%     +16.2       35.26 ±  4%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
     15.78 ± 15%     +17.7       33.52 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault
      8.85 ± 29%     +18.5       27.35 ±  5%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault
     10.65 ± 28%     +20.5       31.17 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault
     67.21 ±  2%     -13.1       54.11 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64
     67.21 ±  2%     -13.1       54.11 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     67.23 ±  2%     -13.1       54.13 ±  3%  perf-profile.children.cycles-pp.do_idle
     66.51 ±  2%     -12.9       53.62 ±  3%  perf-profile.children.cycles-pp.start_secondary
     51.47            -8.5       42.96 ±  3%  perf-profile.children.cycles-pp.intel_idle
     57.16            -7.9       49.27 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter
     57.14            -7.9       49.26 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter_state
      4.84 ± 16%      -3.3        1.55 ± 13%  perf-profile.children.cycles-pp.__schedule
      4.04 ± 14%      -2.8        1.24 ± 14%  perf-profile.children.cycles-pp.ttwu_do_activate
      4.04 ± 14%      -2.8        1.24 ± 14%  perf-profile.children.cycles-pp.activate_task
      4.03 ± 14%      -2.8        1.23 ± 14%  perf-profile.children.cycles-pp.enqueue_task_fair
      3.78 ± 14%      -2.6        1.16 ± 13%  perf-profile.children.cycles-pp.enqueue_entity
      3.36 ± 16%      -2.4        1.00 ± 17%  perf-profile.children.cycles-pp.sched_ttwu_pending
      3.20 ±  8%      -2.1        1.08 ± 11%  perf-profile.children.cycles-pp.rwsem_wake
      2.59 ± 15%      -1.8        0.79 ± 14%  perf-profile.children.cycles-pp.__account_scheduler_latency
      2.72 ±  9%      -1.8        0.93 ± 11%  perf-profile.children.cycles-pp.wake_up_q
      2.67 ±  9%      -1.8        0.92 ± 11%  perf-profile.children.cycles-pp.try_to_wake_up
      2.56 ± 14%      -1.7        0.85 ± 13%  perf-profile.children.cycles-pp.schedule
      2.34 ± 18%      -1.6        0.72 ± 12%  perf-profile.children.cycles-pp.schedule_idle
      2.00 ± 14%      -1.5        0.48 ±  6%  perf-profile.children.cycles-pp.switch_mm_irqs_off
     10.72 ±  4%      -1.4        9.31 ± 10%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.90 ± 30%      -1.4        0.50 ± 14%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     10.71 ±  4%      -1.4        9.31 ± 10%  perf-profile.children.cycles-pp.do_syscall_64
      1.93 ± 28%      -1.4        0.54 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_irq
     10.47 ±  4%      -1.3        9.16 ± 11%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
     10.40 ±  4%      -1.3        9.12 ± 11%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      1.60 ± 15%      -1.1        0.51 ± 15%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      1.64 ± 13%      -1.0        0.67 ±  9%  perf-profile.children.cycles-pp.do_mmap
      1.37 ± 17%      -0.9        0.45 ± 17%  perf-profile.children.cycles-pp.arch_stack_walk
      1.02 ± 13%      -0.8        0.19 ±  6%  perf-profile.children.cycles-pp.switch_mm
      1.14 ± 14%      -0.8        0.34 ± 16%  perf-profile.children.cycles-pp.dequeue_task_fair
      1.06 ± 13%      -0.7        0.31 ± 14%  perf-profile.children.cycles-pp.dequeue_entity
      1.17 ±  9%      -0.7        0.50 ± 10%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      1.29 ± 14%      -0.7        0.62 ± 12%  perf-profile.children.cycles-pp.handle_mm_fault
      0.95 ± 16%      -0.7        0.29 ± 15%  perf-profile.children.cycles-pp.unwind_next_frame
      1.02 ± 12%      -0.6        0.40 ±  6%  perf-profile.children.cycles-pp.update_load_avg
      2.88 ±  8%      -0.6        2.29 ±  8%  perf-profile.children.cycles-pp.menu_select
      1.10 ± 14%      -0.6        0.53 ± 10%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.86 ± 15%      -0.6        0.30 ±  8%  perf-profile.children.cycles-pp.get_unmapped_area
      0.82 ± 15%      -0.5        0.29 ±  8%  perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.97 ± 13%      -0.5        0.45 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.90 ± 10%      -0.5        0.39 ±  8%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.79 ± 18%      -0.5        0.28 ± 22%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.78 ± 15%      -0.5        0.28 ±  9%  perf-profile.children.cycles-pp.unmapped_area_topdown
      0.57 ±  9%      -0.4        0.18 ± 20%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.72 ± 12%      -0.4        0.34 ± 10%  perf-profile.children.cycles-pp.mmap_region
      0.49 ± 10%      -0.3        0.14 ± 10%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.71 ± 11%      -0.3        0.38 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
      1.45 ± 10%      -0.3        1.11 ±  9%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.44 ± 15%      -0.3        0.14 ± 15%  perf-profile.children.cycles-pp.update_curr
      0.38 ± 15%      -0.3        0.08 ± 17%  perf-profile.children.cycles-pp.load_new_mm_cr3
      0.43 ± 17%      -0.3        0.14 ± 11%  perf-profile.children.cycles-pp.finish_task_switch
      0.43 ± 19%      -0.3        0.15 ± 10%  perf-profile.children.cycles-pp.___perf_sw_event
      0.41 ± 18%      -0.3        0.14 ± 23%  perf-profile.children.cycles-pp.set_next_entity
      0.51 ± 13%      -0.2        0.26 ±  8%  perf-profile.children.cycles-pp.do_anonymous_page
      1.12 ±  8%      -0.2        0.87 ± 10%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.31 ± 14%      -0.2        0.07 ± 30%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.82 ±  7%      -0.2        0.59 ± 14%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.40 ± 10%      -0.2        0.18 ±  9%  perf-profile.children.cycles-pp.vma_link
      0.38 ± 11%      -0.2        0.16 ±  5%  perf-profile.children.cycles-pp.update_rq_clock
      0.34 ± 13%      -0.2        0.12 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.29 ± 13%      -0.2        0.09 ±  9%  perf-profile.children.cycles-pp.__switch_to
      0.21 ±  5%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.27 ± 13%      -0.2        0.09 ± 14%  perf-profile.children.cycles-pp.update_cfs_group
      0.31 ±  9%      -0.2        0.13 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.25 ± 10%      -0.2        0.07 ± 26%  perf-profile.children.cycles-pp.update_cfs_rq_h_load
      0.26 ± 19%      -0.2        0.09 ± 17%  perf-profile.children.cycles-pp.__unwind_start
      0.54 ±  3%      -0.2        0.37 ±  2%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.22 ± 17%      -0.2        0.06 ± 16%  perf-profile.children.cycles-pp.__switch_to_asm
      0.26 ± 15%      -0.2        0.10 ± 10%  perf-profile.children.cycles-pp.__perf_sw_event
      0.34 ± 13%      -0.2        0.18 ± 10%  perf-profile.children.cycles-pp.down_read_trylock
      0.31 ± 11%      -0.1        0.17 ±  4%  perf-profile.children.cycles-pp.up_read
      0.25 ± 14%      -0.1        0.11 ± 22%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.31 ±  9%      -0.1        0.17 ± 14%  perf-profile.children.cycles-pp.find_vma
      0.26 ± 14%      -0.1        0.12 ± 14%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.46 ±  2%      -0.1        0.32 ±  5%  perf-profile.children.cycles-pp.native_write_msr
      0.18 ± 24%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.orc_find
      0.20 ± 21%      -0.1        0.07 ± 23%  perf-profile.children.cycles-pp.__orc_find
      0.34 ±  7%      -0.1        0.21 ±  5%  perf-profile.children.cycles-pp.sched_clock
      0.33 ±  6%      -0.1        0.20 ±  5%  perf-profile.children.cycles-pp.native_sched_clock
      0.30 ±  9%      -0.1        0.17 ± 16%  perf-profile.children.cycles-pp.down_read
      0.19 ± 18%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.34 ± 10%      -0.1        0.22 ±  7%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.51 ±  6%      -0.1        0.40 ± 14%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.20 ± 11%      -0.1        0.09 ±  9%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.13 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.27 ±  9%      -0.1        0.17 ±  8%  perf-profile.children.cycles-pp.__lock_text_start
      0.15 ±  9%      -0.1        0.06 ± 20%  perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
      0.18 ± 14%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.vmacache_find
      0.13 ± 28%      -0.1        0.05 ± 63%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.28 ± 15%      -0.1        0.20 ± 10%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.11 ± 33%      -0.1        0.04 ±102%  perf-profile.children.cycles-pp.__kernel_text_address
      0.35 ±  6%      -0.1        0.28 ±  9%  perf-profile.children.cycles-pp.read_tsc
      0.17 ± 13%      -0.1        0.10 ± 18%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.12 ± 18%      -0.1        0.06 ± 16%  perf-profile.children.cycles-pp.perf_event_mmap
      0.09 ± 34%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.kernel_text_address
      0.17 ± 16%      -0.1        0.11 ±  9%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.12 ± 10%      -0.0        0.08 ± 13%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.10 ± 18%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.10 ± 18%      -0.0        0.07 ± 15%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.22 ±  5%      +0.1        0.30 ± 15%  perf-profile.children.cycles-pp.tick_irq_enter
      0.13 ± 27%      +0.1        0.22 ± 11%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.28            +0.1        0.38 ± 15%  perf-profile.children.cycles-pp.irq_enter
      0.71 ± 13%      +0.2        0.95 ± 11%  perf-profile.children.cycles-pp.update_process_times
      0.78 ± 13%      +0.2        1.03 ± 11%  perf-profile.children.cycles-pp.tick_sched_handle
      0.96 ± 12%      +0.3        1.24 ± 10%  perf-profile.children.cycles-pp.tick_sched_timer
      1.59 ±  9%      +0.4        1.97 ±  7%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.57 ± 10%      +0.7        2.31 ±  3%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      5.93 ±  2%      +1.7        7.62 ± 12%  perf-profile.children.cycles-pp.down_write_killable
      5.86 ±  2%      +1.7        7.59 ± 12%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
     19.30 ± 10%     +16.0       35.35 ±  4%  perf-profile.children.cycles-pp.page_fault
     19.24 ± 10%     +16.1       35.33 ±  4%  perf-profile.children.cycles-pp.do_page_fault
     19.09 ± 10%     +16.2       35.27 ±  4%  perf-profile.children.cycles-pp.__do_page_fault
     15.78 ± 15%     +17.7       33.52 ±  4%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
     12.57 ± 22%     +20.8       33.40 ±  4%  perf-profile.children.cycles-pp.osq_lock
     15.96 ± 20%     +23.3       39.22 ±  3%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
     50.41            -7.7       42.72 ±  3%  perf-profile.self.cycles-pp.intel_idle
      1.90 ± 30%      -1.4        0.50 ± 14%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.62 ± 14%      -1.2        0.40 ±  8%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.79 ± 15%      -0.5        0.26 ± 12%  perf-profile.self.cycles-pp.__schedule
      0.78 ± 15%      -0.5        0.27 ±  8%  perf-profile.self.cycles-pp.unmapped_area_topdown
      0.81 ± 11%      -0.4        0.36 ±  7%  perf-profile.self.cycles-pp.prepare_exit_to_usermode
      0.85 ± 11%      -0.4        0.42 ±  8%  perf-profile.self.cycles-pp.rwsem_down_read_slowpath
      0.52 ± 11%      -0.4        0.14 ±  9%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.53 ± 13%      -0.4        0.15 ± 14%  perf-profile.self.cycles-pp.unwind_next_frame
      0.61 ±  8%      -0.3        0.27 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.47 ± 10%      -0.3        0.15 ± 10%  perf-profile.self.cycles-pp.try_to_wake_up
      0.66 ± 10%      -0.3        0.34 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock
      0.54 ± 15%      -0.3        0.24 ± 14%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.38 ± 15%      -0.3        0.08 ± 17%  perf-profile.self.cycles-pp.load_new_mm_cr3
      0.40 ± 16%      -0.3        0.13 ± 12%  perf-profile.self.cycles-pp.finish_task_switch
      0.37 ± 21%      -0.2        0.13 ± 10%  perf-profile.self.cycles-pp.___perf_sw_event
      0.40 ± 13%      -0.2        0.17 ±  5%  perf-profile.self.cycles-pp.update_load_avg
      1.16 ±  8%      -0.2        0.94 ±  7%  perf-profile.self.cycles-pp.menu_select
      0.43 ± 11%      -0.2        0.22 ±  8%  perf-profile.self.cycles-pp.do_idle
      0.28 ± 14%      -0.2        0.08 ± 11%  perf-profile.self.cycles-pp.enqueue_entity
      0.32 ± 13%      -0.2        0.12 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.28 ± 12%      -0.2        0.09 ±  9%  perf-profile.self.cycles-pp.__switch_to
      0.28 ±  9%      -0.2        0.09 ± 20%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.24 ± 14%      -0.2        0.06 ± 60%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.21 ±  5%      -0.2        0.03 ±100%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.41 ±  7%      -0.2        0.23 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.27 ± 12%      -0.2        0.09 ± 14%  perf-profile.self.cycles-pp.update_cfs_group
      0.25 ± 10%      -0.2        0.07 ± 26%  perf-profile.self.cycles-pp.update_cfs_rq_h_load
      0.30 ±  9%      -0.2        0.12 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.54 ±  3%      -0.2        0.37 ±  2%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.25 ± 10%      -0.2        0.08 ± 16%  perf-profile.self.cycles-pp.update_curr
      0.29 ± 11%      -0.2        0.13 ±  6%  perf-profile.self.cycles-pp.update_rq_clock
      0.24 ± 18%      -0.2        0.08 ± 22%  perf-profile.self.cycles-pp.set_next_entity
      0.28 ± 10%      -0.2        0.12 ± 16%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.21 ±  5%      -0.2        0.05        perf-profile.self.cycles-pp.stack_trace_save_tsk
      0.22 ± 17%      -0.2        0.06 ± 16%  perf-profile.self.cycles-pp.__switch_to_asm
      0.23 ± 19%      -0.2        0.07 ± 19%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.33 ± 13%      -0.2        0.18 ± 10%  perf-profile.self.cycles-pp.down_read_trylock
      0.31 ± 11%      -0.1        0.17 ±  3%  perf-profile.self.cycles-pp.up_read
      0.20 ± 11%      -0.1        0.06 ± 14%  perf-profile.self.cycles-pp.dequeue_entity
      0.24 ± 15%      -0.1        0.11 ± 19%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.46 ±  2%      -0.1        0.32 ±  5%  perf-profile.self.cycles-pp.native_write_msr
      0.20 ± 20%      -0.1        0.07 ± 23%  perf-profile.self.cycles-pp.__orc_find
      0.18 ± 23%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.orc_find
      0.31 ±  6%      -0.1        0.18 ±  6%  perf-profile.self.cycles-pp.native_sched_clock
      0.27 ± 10%      -0.1        0.15 ± 18%  perf-profile.self.cycles-pp.down_read
      0.20 ± 11%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.vma_interval_tree_insert
      0.14 ±  5%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.rwsem_mark_wake
      0.17 ±  8%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.18 ± 14%      -0.1        0.10 ± 15%  perf-profile.self.cycles-pp.vmacache_find
      0.15 ± 27%      -0.1        0.08 ± 20%  perf-profile.self.cycles-pp.__do_page_fault
      0.34 ±  5%      -0.1        0.27 ± 10%  perf-profile.self.cycles-pp.read_tsc
      0.22 ±  8%      -0.1        0.15 ± 11%  perf-profile.self.cycles-pp.__lock_text_start
      0.28 ±  7%      -0.1        0.21 ± 17%  perf-profile.self.cycles-pp.find_next_bit
      0.12 ±  6%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.find_vma
      0.09 ± 17%      -0.1        0.04 ± 60%  perf-profile.self.cycles-pp.handle_mm_fault
      0.10 ± 15%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.do_syscall_64
      0.23 ±  7%      -0.0        0.19 ± 11%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.07 ± 11%      -0.0        0.03 ±102%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.12 ± 12%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.10 ± 18%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.12 ± 37%      +0.0        0.15 ± 35%  perf-profile.self.cycles-pp.update_blocked_averages
      0.11 ± 26%      +0.1        0.18 ± 16%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      1.13 ±  4%      +0.2        1.36 ±  3%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      2.04 ± 23%      +2.2        4.23 ±  8%  perf-profile.self.cycles-pp.rwsem_optimistic_spin
     12.51 ± 22%     +20.7       33.24 ±  4%  perf-profile.self.cycles-pp.osq_lock


                                                                                
                                vm-scalability.median                           
                                                                                
  12000 +-+-----------------------------------------------------------------+   
        |.+    ::                           +..+   ::    +.+   ::    +      |   
  11000 +-++  : :   +.. .+                  :   :  : :  :   :  : :  : :   + |   
  10000 +-+ + :  :  :  +  :            .+  :    : :  :  :   : :  :  : : .. :|   
        |    +   : :      :  .+.+.. .+.  + :     +   : :     +   : :   +   :|   
   9000 +-+       ::       +.      +      +           ::          ::        |   
        |         +                                   +           +         |   
   8000 +-+                                                                 |   
        |                                                                   |   
   7000 +-+                O    O                                           |   
   6000 +-+            O                    O                               |   
        O    O O  O O    O    O      O  O O    O                            |   
   5000 +-+                        O                                        |   
        | O                                                                 |   
   4000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           vm-scalability.time.user_time                        
                                                                                
  220 +-+-------------------------------------------------------------------+   
      |                                                                     |   
  200 +-+     +                             +      +    +..     +           |   
  180 +-+    ::                            : +     ::   :      ::    +      |   
      |  +   : :       .+                  :  +.. : :  :   +   : :  : :   + |   
  160 +-+ + :  :    .+.  :                :       :  : :    + :  :  : :  + :|   
      |    +   :   +     :    .+..    .+. :      +   : :     +   : :   :+  :|   
  140 +-+       :..       : .+    +.+.   +            :           ::   +    |   
      |         +         +.                          +           +         |   
  120 +-+                                                                   |   
  100 +-+                                                                   |   
      |                   O    O                                            |   
   80 +-+            O                 O    O O                             |   
      O       O O  O    O    O      O    O                                  |   
   60 +-+O-O----------------------O-----------------------------------------+   
                                                                                
                                                                                                                                                                
                    vm-scalability.time.voluntary_context_switches              
                                                                                
    4e+07 +-+---------------------------------------------------------------+   
          |.+                                +..+        +..+        +    + |   
  3.5e+07 +-+:   +         +                 :   :  +    :   :  +    ::  : :|   
          |   : : :      .. :     +         :    : : +  :    : : +  : :  : :|   
    3e+07 +-+ : : :  .+.+   :   .. + .+..+. :     ::  + :     ::  + :  ::   |   
          |    +   +.        +.+    +      +      +    +      +    +   +    |   
  2.5e+07 +-+                                                               |   
          |                                                                 |   
    2e+07 +-+                                                               |   
          |                                                                 |   
  1.5e+07 +-+                                                               |   
          |                                                                 |   
    1e+07 +-+           O    O    O   O                                     |   
          O O  O O O  O    O   O    O    O O O  O                           |   
    5e+06 +-+---------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.3.0-rc1-00002-g78134300579a45" of type "text/plain" (199565 bytes)

View attachment "job-script" of type "text/plain" (7436 bytes)

View attachment "job.yaml" of type "text/plain" (5032 bytes)

View attachment "reproduce" of type "text/plain" (798 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ