lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Tue, 23 Feb 2021 10:30:04 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Valentin Schneider <valentin.schneider@....com>
Cc:     0day robot <lkp@...el.com>,
        Valentin Schneider <valentin.schneider@....com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com,
        Lingutla Chandrasekhar <clingutla@...eaurora.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...nel.org>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Morten Rasmussen <morten.rasmussen@....com>,
        Qais Yousef <qais.yousef@....com>,
        Quentin Perret <qperret@...gle.com>,
        Pavan Kondeti <pkondeti@...eaurora.org>,
        Rik van Riel <riel@...riel.com>, aubrey.li@...ux.intel.com,
        yu.c.chen@...el.com
Subject: [sched/fair]  b360fb5e59:  stress-ng.vm-segv.ops_per_sec -13.9%
 regression


Greeting,

FYI, we noticed a -13.9% regression of stress-ng.vm-segv.ops_per_sec due to commit:


commit: b360fb5e5954a8a440ef95bf11257e2e7ea90340 ("[PATCH v2 1/7] sched/fair: Ignore percpu threads for imbalance pulls")
url: https://github.com/0day-ci/linux/commits/Valentin-Schneider/sched-fair-misfit-task-load-balance-tweaks/20210219-211028
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git c5e6fc08feb2b88dc5dac2f3c817e1c2a4cafda4

in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	nr_threads: 10%
	disk: 1HDD
	testtime: 60s
	fs: ext4
	class: vm
	test: vm-segv
	cpufreq_governor: performance
	ucode: 0x5003003




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  vm/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/vm-segv/stress-ng/60s/0x5003003

commit: 
  c5e6fc08fe ("sched,x86: Allow !PREEMPT_DYNAMIC")
  b360fb5e59 ("sched/fair: Ignore percpu threads for imbalance pulls")

c5e6fc08feb2b88d b360fb5e5954a8a440ef95bf112 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:6           -3%           1:6     perf-profile.children.cycles-pp.error_entry
          0:6           -1%           0:6     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     11324 ±  3%     -28.1%       8140 ±  3%  stress-ng.time.involuntary_context_switches
      6818 ± 15%    +315.2%      28311 ± 12%  stress-ng.time.major_page_faults
  30952041           -12.8%   26988502        stress-ng.time.minor_page_faults
    378.82            +5.3%     398.75        stress-ng.time.system_time
    215.82           -10.0%     194.24        stress-ng.time.user_time
  62102177           -13.9%   53448474        stress-ng.time.voluntary_context_switches
    810348           -13.9%     698034        stress-ng.vm-segv.ops
     13505           -13.9%      11633        stress-ng.vm-segv.ops_per_sec
      9.76            -1.3%       9.63        iostat.cpu.system
    311.06            +3.9%     323.18        pmeter.Average_Active_Power
   1937515           -13.9%    1667746        vmstat.system.cs
  2.33e+08           +15.2%  2.685e+08        cpuidle.C1.time
  14107313            -9.8%   12722308 ±  2%  cpuidle.C1.usage
  51960611           -15.6%   43858596        cpuidle.POLL.usage
      1.11            -0.3        0.85        mpstat.cpu.all.irq%
      0.18            -0.0        0.16        mpstat.cpu.all.soft%
      0.40            -0.1        0.35        mpstat.cpu.all.usr%
     52529 ±  6%     -35.8%      33706 ± 44%  sched_debug.cfs_rq:/.min_vruntime.avg
     38574 ± 10%     -41.7%      22479 ± 41%  sched_debug.cfs_rq:/.min_vruntime.min
    624583           -56.9%     269004 ± 99%  sched_debug.cpu.nr_switches.avg
    421467 ±  6%     -65.0%     147633 ±102%  sched_debug.cpu.nr_switches.min
     33017 ± 41%     +71.9%      56752 ±  4%  numa-vmstat.node0.nr_anon_pages
     33861 ± 40%     +68.8%      57163 ±  4%  numa-vmstat.node0.nr_inactive_anon
     33861 ± 40%     +68.8%      57163 ±  4%  numa-vmstat.node0.nr_zone_inactive_anon
     31421 ± 44%     -75.4%       7730 ± 31%  numa-vmstat.node1.nr_anon_pages
     32588 ± 42%     -71.2%       9386 ± 25%  numa-vmstat.node1.nr_inactive_anon
      1949 ±  5%     -15.9%       1638 ±  9%  numa-vmstat.node1.nr_page_table_pages
     32588 ± 42%     -71.2%       9386 ± 25%  numa-vmstat.node1.nr_zone_inactive_anon
   7935635 ± 13%     -19.3%    6407159 ±  6%  numa-vmstat.node1.numa_hit
   7732506 ± 14%     -19.7%    6206498 ±  6%  numa-vmstat.node1.numa_local
    131518 ± 41%     +72.1%     226297 ±  4%  numa-meminfo.node0.AnonPages
    140036 ± 38%     +66.7%     233420 ±  4%  numa-meminfo.node0.AnonPages.max
    136203 ± 40%     +68.2%     229070 ±  4%  numa-meminfo.node0.Inactive
    135279 ± 40%     +68.9%     228531 ±  4%  numa-meminfo.node0.Inactive(anon)
      7950 ±  8%     +15.4%       9178 ±  2%  numa-meminfo.node0.PageTables
     40320 ± 68%     -79.8%       8138 ± 45%  numa-meminfo.node1.AnonHugePages
    125324 ± 44%     -75.9%      30215 ± 32%  numa-meminfo.node1.AnonPages
    208760 ± 26%     -48.1%     108370 ±  8%  numa-meminfo.node1.AnonPages.max
    129998 ± 42%     -71.1%      37570 ± 25%  numa-meminfo.node1.Inactive
    129996 ± 42%     -71.4%      37183 ± 25%  numa-meminfo.node1.Inactive(anon)
      7433 ±  6%     -18.1%       6091 ±  9%  numa-meminfo.node1.PageTables
      9921            +2.0%      10121        proc-vmstat.nr_mapped
     23344            -3.6%      22514        proc-vmstat.nr_slab_reclaimable
  30383698 ±  2%     -13.8%   26186115        proc-vmstat.numa_hit
  30297093 ±  2%     -13.9%   26099520        proc-vmstat.numa_local
      1600 ±149%    -100.0%       0.00        proc-vmstat.numa_pages_migrated
      9337 ±  2%     -12.5%       8174 ±  3%  proc-vmstat.pgactivate
  34255062 ±  2%     -14.7%   29225155        proc-vmstat.pgalloc_normal
  31199740           -12.6%   27277537        proc-vmstat.pgfault
  34113059 ±  2%     -14.7%   29083007        proc-vmstat.pgfree
      1600 ±149%    -100.0%       0.00        proc-vmstat.pgmigrate_success
    345591            -8.1%     317680        proc-vmstat.pgreuse
     11501 ±  5%      +9.6%      12610 ±  6%  softirqs.CPU12.RCU
     10678 ±  5%     +16.0%      12383 ±  2%  softirqs.CPU16.RCU
     10871 ±  4%     +13.1%      12294 ±  2%  softirqs.CPU17.RCU
     10724 ±  2%     +13.8%      12205 ±  3%  softirqs.CPU18.RCU
     10810 ±  4%     +16.2%      12560 ±  3%  softirqs.CPU19.RCU
     10647 ±  6%     +16.2%      12372 ±  6%  softirqs.CPU20.RCU
     10863 ±  3%     +14.7%      12461 ±  3%  softirqs.CPU21.RCU
     11231 ±  5%     +14.6%      12873 ±  6%  softirqs.CPU22.RCU
     11141 ±  6%     +21.0%      13480 ±  8%  softirqs.CPU64.RCU
     11209 ±  6%     +20.8%      13545 ±  2%  softirqs.CPU65.RCU
     11108 ±  3%     +20.0%      13334 ±  6%  softirqs.CPU66.RCU
     11414 ±  9%     +16.9%      13345 ±  6%  softirqs.CPU67.RCU
     11162 ±  4%     +16.2%      12968 ±  9%  softirqs.CPU68.RCU
     11035 ±  5%     +13.6%      12533 ±  4%  softirqs.CPU69.RCU
     11003 ±  5%     +18.9%      13078 ±  8%  softirqs.CPU70.RCU
     11097 ±  4%     +14.9%      12756 ±  5%  softirqs.CPU71.RCU
     76929           +14.7%      88227        slabinfo.anon_vma_chain.active_objs
      1205           +14.4%       1379        slabinfo.anon_vma_chain.active_slabs
     77192           +14.4%      88314        slabinfo.anon_vma_chain.num_objs
      1205           +14.4%       1379        slabinfo.anon_vma_chain.num_slabs
     23274           +12.2%      26121        slabinfo.kmalloc-512.active_objs
      9748 ±  2%     -10.0%       8776 ±  3%  slabinfo.pid.active_objs
      9748 ±  2%     -10.0%       8776 ±  3%  slabinfo.pid.num_objs
     28777           -11.3%      25539 ±  2%  slabinfo.proc_inode_cache.active_objs
     28919           -11.3%      25647 ±  2%  slabinfo.proc_inode_cache.num_objs
     11256 ±  5%     -23.8%       8581 ±  3%  slabinfo.task_delay_info.active_objs
     11256 ±  5%     -23.8%       8581 ±  3%  slabinfo.task_delay_info.num_objs
     53293           +12.8%      60111        slabinfo.vm_area_struct.active_objs
      1335           +12.6%       1503        slabinfo.vm_area_struct.active_slabs
     53426           +12.6%      60156        slabinfo.vm_area_struct.num_objs
      1335           +12.6%       1503        slabinfo.vm_area_struct.num_slabs
     20536 ±  2%     +25.3%      25732 ± 12%  slabinfo.vmap_area.active_objs
     24731           +23.5%      30551 ±  3%  slabinfo.vmap_area.num_objs
 4.799e+09           -12.2%  4.215e+09        perf-stat.i.branch-instructions
      1.09            +0.1        1.14        perf-stat.i.branch-miss-rate%
  50816825            -7.1%   47191966        perf-stat.i.branch-misses
      4.52 ±  3%      +5.6       10.16 ±  3%  perf-stat.i.cache-miss-rate%
  18266157 ±  2%    +125.2%   41128630 ±  3%  perf-stat.i.cache-misses
 4.817e+08           -11.6%  4.258e+08        perf-stat.i.cache-references
   1998506           -14.0%    1718841        perf-stat.i.context-switches
      1.89           +12.4%       2.12        perf-stat.i.cpi
 4.232e+10            -1.4%  4.175e+10        perf-stat.i.cpu-cycles
    836.21 ±  4%     +14.8%     959.82 ±  7%  perf-stat.i.cpu-migrations
      2507           -54.0%       1153 ±  5%  perf-stat.i.cycles-between-cache-misses
   4212845 ±  3%     -10.4%    3775244 ±  2%  perf-stat.i.dTLB-load-misses
 6.137e+09           -12.7%  5.359e+09        perf-stat.i.dTLB-loads
      0.03 ±  3%      +0.0        0.04 ±  2%  perf-stat.i.dTLB-store-miss-rate%
   1127230 ±  3%      -6.7%    1051240 ±  2%  perf-stat.i.dTLB-store-misses
 3.192e+09           -13.4%  2.766e+09        perf-stat.i.dTLB-stores
     67.76            -0.7       67.08        perf-stat.i.iTLB-load-miss-rate%
  17023777           -10.2%   15291627        perf-stat.i.iTLB-load-misses
   7885583            -7.2%    7319880        perf-stat.i.iTLB-loads
 2.237e+10           -12.5%  1.956e+10        perf-stat.i.instructions
      0.53           -10.8%       0.48        perf-stat.i.ipc
    111.04 ± 13%    +310.1%     455.43 ± 11%  perf-stat.i.major-faults
      0.44            -1.4%       0.43        perf-stat.i.metric.GHz
      0.06 ± 12%     +21.9%       0.07 ± 14%  perf-stat.i.metric.K/sec
    152.39           -12.5%     133.27        perf-stat.i.metric.M/sec
    494626           -12.8%     431405        perf-stat.i.minor-faults
     77.46           +12.6       90.07        perf-stat.i.node-load-miss-rate%
   3962536 ±  4%    +195.0%   11688629 ±  3%  perf-stat.i.node-load-misses
   1000403            +7.6%    1076834 ±  2%  perf-stat.i.node-loads
     41.34 ±  2%     +31.6       72.93        perf-stat.i.node-store-miss-rate%
   1494039 ±  5%    +198.5%    4459287 ±  3%  perf-stat.i.node-store-misses
   1975747 ±  3%     -25.8%    1466690 ±  2%  perf-stat.i.node-stores
    520459           -12.8%     453999        perf-stat.i.page-faults
      1.06            +0.1        1.12        perf-stat.overall.branch-miss-rate%
      3.78 ±  2%      +5.8        9.60 ±  3%  perf-stat.overall.cache-miss-rate%
      1.89           +12.7%       2.13        perf-stat.overall.cpi
      2324 ±  2%     -56.1%       1021 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.04 ±  3%      +0.0        0.04 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     68.35            -0.7       67.65        perf-stat.overall.iTLB-load-miss-rate%
      0.53           -11.2%       0.47        perf-stat.overall.ipc
     79.66           +11.8       91.48        perf-stat.overall.node-load-miss-rate%
     42.93 ±  3%     +32.1       75.07        perf-stat.overall.node-store-miss-rate%
 4.727e+09           -12.1%  4.154e+09        perf-stat.ps.branch-instructions
  50025626            -7.1%   46471888        perf-stat.ps.branch-misses
  17939323 ±  2%    +124.6%   40288410 ±  3%  perf-stat.ps.cache-misses
 4.743e+08           -11.5%  4.196e+08        perf-stat.ps.cache-references
   1968071           -13.9%    1694380        perf-stat.ps.context-switches
 4.167e+10            -1.4%   4.11e+10        perf-stat.ps.cpu-cycles
    821.15 ±  4%     +14.8%     942.91 ±  7%  perf-stat.ps.cpu-migrations
   4147982 ±  3%     -10.3%    3720585 ±  2%  perf-stat.ps.dTLB-load-misses
 6.043e+09           -12.6%  5.282e+09        perf-stat.ps.dTLB-loads
   1109704 ±  3%      -6.6%    1036060 ±  2%  perf-stat.ps.dTLB-store-misses
 3.144e+09           -13.3%  2.726e+09        perf-stat.ps.dTLB-stores
  16765267           -10.1%   15068821        perf-stat.ps.iTLB-load-misses
   7761893            -7.1%    7207514        perf-stat.ps.iTLB-loads
 2.203e+10           -12.5%  1.928e+10        perf-stat.ps.instructions
    108.63 ± 13%    +309.6%     444.94 ± 12%  perf-stat.ps.major-faults
    487068           -12.7%     425183        perf-stat.ps.minor-faults
   3881430 ±  4%    +194.6%   11435790 ±  3%  perf-stat.ps.node-load-misses
    989805            +7.5%    1064249 ±  2%  perf-stat.ps.node-loads
   1464554 ±  5%    +198.0%    4363838 ±  3%  perf-stat.ps.node-store-misses
   1945831 ±  3%     -25.6%    1448410 ±  2%  perf-stat.ps.node-stores
    512507           -12.7%     447452        perf-stat.ps.page-faults
 1.389e+12           -12.5%  1.215e+12        perf-stat.total.instructions
    239831           -33.6%     159201        interrupts.CAL:Function_call_interrupts
      2858 ± 21%     -36.1%       1825 ± 17%  interrupts.CPU0.CAL:Function_call_interrupts
     76.83 ± 14%     -73.1%      20.67 ± 24%  interrupts.CPU0.TLB:TLB_shootdowns
      3447 ± 10%     -38.8%       2108 ± 18%  interrupts.CPU1.CAL:Function_call_interrupts
      2457 ± 16%     -30.3%       1712 ± 15%  interrupts.CPU10.CAL:Function_call_interrupts
      2252 ± 16%     -29.6%       1585 ± 22%  interrupts.CPU11.CAL:Function_call_interrupts
      2156 ± 23%     -27.9%       1555 ± 13%  interrupts.CPU12.CAL:Function_call_interrupts
    378.33 ± 11%     +42.1%     537.50 ± 19%  interrupts.CPU13.RES:Rescheduling_interrupts
      2358 ± 15%     -40.2%       1409 ±  8%  interrupts.CPU14.CAL:Function_call_interrupts
      2216 ± 20%     -36.0%       1418 ± 10%  interrupts.CPU15.CAL:Function_call_interrupts
      2387 ± 15%     -24.2%       1811 ±  5%  interrupts.CPU16.CAL:Function_call_interrupts
      2329 ± 14%     -36.0%       1491 ± 14%  interrupts.CPU17.CAL:Function_call_interrupts
      2509 ± 13%     -39.2%       1525 ± 15%  interrupts.CPU18.CAL:Function_call_interrupts
      2333 ± 19%     -32.9%       1566 ± 18%  interrupts.CPU19.CAL:Function_call_interrupts
      3082 ± 15%     -36.0%       1972 ± 23%  interrupts.CPU2.CAL:Function_call_interrupts
      2019 ±  6%     -24.8%       1518 ± 17%  interrupts.CPU2.NMI:Non-maskable_interrupts
      2019 ±  6%     -24.8%       1518 ± 17%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      2456 ± 11%     -36.1%       1569 ± 11%  interrupts.CPU21.CAL:Function_call_interrupts
      2668 ± 21%     -37.7%       1662 ± 11%  interrupts.CPU22.CAL:Function_call_interrupts
      2361 ± 15%     -34.6%       1543 ± 15%  interrupts.CPU23.CAL:Function_call_interrupts
      2651 ±  8%     -22.4%       2057 ± 17%  interrupts.CPU24.CAL:Function_call_interrupts
      2936 ± 10%     -42.9%       1675 ± 20%  interrupts.CPU25.CAL:Function_call_interrupts
      2734 ± 22%     -37.4%       1712 ± 24%  interrupts.CPU26.CAL:Function_call_interrupts
      2614 ± 19%     -34.1%       1723 ± 18%  interrupts.CPU27.CAL:Function_call_interrupts
      2774 ± 13%     -46.3%       1490 ± 21%  interrupts.CPU28.CAL:Function_call_interrupts
      2454 ± 18%     -44.0%       1373 ± 18%  interrupts.CPU29.CAL:Function_call_interrupts
    433.83 ± 16%     -30.7%     300.67 ± 13%  interrupts.CPU29.RES:Rescheduling_interrupts
      2969 ± 19%     -40.4%       1769 ± 15%  interrupts.CPU3.CAL:Function_call_interrupts
      2566 ± 10%     -32.4%       1734 ± 13%  interrupts.CPU30.CAL:Function_call_interrupts
      2614 ± 16%     -43.6%       1475 ± 26%  interrupts.CPU31.CAL:Function_call_interrupts
      2212 ± 11%     -34.4%       1452 ± 11%  interrupts.CPU32.CAL:Function_call_interrupts
      2273 ± 16%     -39.6%       1372 ± 23%  interrupts.CPU33.CAL:Function_call_interrupts
      2313 ± 20%     -34.2%       1521 ± 18%  interrupts.CPU36.CAL:Function_call_interrupts
      2258 ± 19%     -31.3%       1550 ± 25%  interrupts.CPU38.CAL:Function_call_interrupts
      2482 ± 16%     -29.8%       1743 ± 14%  interrupts.CPU4.CAL:Function_call_interrupts
      2466 ± 16%     -40.9%       1458 ± 23%  interrupts.CPU40.CAL:Function_call_interrupts
      2488 ± 21%     -42.9%       1420 ± 21%  interrupts.CPU41.CAL:Function_call_interrupts
      2327 ± 16%     -39.7%       1403 ± 19%  interrupts.CPU42.CAL:Function_call_interrupts
      2352 ± 16%     -40.0%       1412 ± 20%  interrupts.CPU44.CAL:Function_call_interrupts
      2252 ± 15%     -41.0%       1329 ± 26%  interrupts.CPU46.CAL:Function_call_interrupts
      2490 ± 13%     -40.1%       1491 ± 23%  interrupts.CPU47.CAL:Function_call_interrupts
      2628 ± 13%     -32.4%       1777 ± 17%  interrupts.CPU48.CAL:Function_call_interrupts
      2752 ± 14%     -35.0%       1788 ± 12%  interrupts.CPU49.CAL:Function_call_interrupts
      2668 ± 15%     -39.8%       1605 ± 20%  interrupts.CPU5.CAL:Function_call_interrupts
      2693 ± 12%     -35.1%       1749 ± 15%  interrupts.CPU50.CAL:Function_call_interrupts
      2515 ± 17%     -35.9%       1613 ± 22%  interrupts.CPU51.CAL:Function_call_interrupts
      2809 ± 16%     -45.4%       1534 ± 14%  interrupts.CPU52.CAL:Function_call_interrupts
      2748 ± 11%     -35.2%       1780 ± 22%  interrupts.CPU53.CAL:Function_call_interrupts
      2634 ± 12%     -34.1%       1736 ±  9%  interrupts.CPU54.CAL:Function_call_interrupts
      2607 ± 11%     -38.0%       1616 ±  9%  interrupts.CPU55.CAL:Function_call_interrupts
      2527 ± 19%     -38.3%       1558 ± 13%  interrupts.CPU57.CAL:Function_call_interrupts
      2620 ± 12%     -22.3%       2037 ± 15%  interrupts.CPU58.CAL:Function_call_interrupts
      2569 ± 10%     -30.1%       1795 ± 16%  interrupts.CPU61.CAL:Function_call_interrupts
      2514 ±  8%     -36.3%       1601 ± 25%  interrupts.CPU62.CAL:Function_call_interrupts
      2525 ± 16%     -31.9%       1719 ± 25%  interrupts.CPU63.CAL:Function_call_interrupts
      2494 ±  8%     -24.6%       1880 ± 11%  interrupts.CPU64.CAL:Function_call_interrupts
    506.50 ± 15%     +26.9%     642.67 ±  9%  interrupts.CPU65.RES:Rescheduling_interrupts
    486.67 ±  9%     +29.5%     630.33 ± 15%  interrupts.CPU66.RES:Rescheduling_interrupts
    120.00 ±126%     -92.4%       9.17 ± 36%  interrupts.CPU66.TLB:TLB_shootdowns
      2444 ± 12%     -21.9%       1908 ± 18%  interrupts.CPU67.CAL:Function_call_interrupts
      2403 ± 16%     -32.6%       1620 ± 15%  interrupts.CPU69.CAL:Function_call_interrupts
      2673 ± 11%     -37.2%       1679 ± 24%  interrupts.CPU7.CAL:Function_call_interrupts
      2360 ± 12%     -27.0%       1723 ± 14%  interrupts.CPU70.CAL:Function_call_interrupts
      2408 ± 11%     -31.9%       1639 ± 16%  interrupts.CPU71.CAL:Function_call_interrupts
      2591 ± 13%     -36.9%       1635 ± 24%  interrupts.CPU75.CAL:Function_call_interrupts
      2793 ± 13%     -39.7%       1685 ± 17%  interrupts.CPU76.CAL:Function_call_interrupts
      2662 ± 14%     -37.3%       1668 ± 20%  interrupts.CPU77.CAL:Function_call_interrupts
      1899 ±  8%     -25.9%       1408 ± 33%  interrupts.CPU77.NMI:Non-maskable_interrupts
      1899 ±  8%     -25.9%       1408 ± 33%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      2624 ± 10%     -35.4%       1694 ± 27%  interrupts.CPU78.CAL:Function_call_interrupts
      2816 ± 12%     -35.0%       1829 ± 25%  interrupts.CPU79.CAL:Function_call_interrupts
      2393 ± 10%     -36.2%       1525 ± 11%  interrupts.CPU8.CAL:Function_call_interrupts
      2465 ± 16%     -33.6%       1637 ± 18%  interrupts.CPU80.CAL:Function_call_interrupts
      2430 ± 16%     -41.1%       1432 ±  9%  interrupts.CPU81.CAL:Function_call_interrupts
      2369 ±  7%     -25.5%       1765 ± 14%  interrupts.CPU83.CAL:Function_call_interrupts
      2509 ± 20%     -40.1%       1504 ± 19%  interrupts.CPU84.CAL:Function_call_interrupts
      2624 ± 20%     -34.7%       1713 ± 26%  interrupts.CPU85.CAL:Function_call_interrupts
      2329 ± 15%     -29.0%       1654 ± 21%  interrupts.CPU86.CAL:Function_call_interrupts
      2599 ± 14%     -39.2%       1580 ± 17%  interrupts.CPU88.CAL:Function_call_interrupts
      2413 ± 13%     -39.0%       1471 ± 12%  interrupts.CPU9.CAL:Function_call_interrupts
      2492 ± 19%     -36.1%       1592 ± 30%  interrupts.CPU90.CAL:Function_call_interrupts
      2325 ± 21%     -32.6%       1567 ± 19%  interrupts.CPU91.CAL:Function_call_interrupts
      2557 ± 13%     -42.3%       1474 ± 19%  interrupts.CPU92.CAL:Function_call_interrupts
      2576 ± 19%     -45.2%       1412 ± 19%  interrupts.CPU94.CAL:Function_call_interrupts
      2571 ± 10%     -46.4%       1377 ± 20%  interrupts.CPU95.CAL:Function_call_interrupts
    475.17 ± 15%     -22.4%     368.50 ± 13%  interrupts.CPU95.RES:Rescheduling_interrupts
      5869 ±  2%     -83.2%     988.50 ± 12%  interrupts.TLB:TLB_shootdowns
      2.74 ±  6%      -0.3        2.40 ±  3%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.66 ±  6%      -0.3        2.33 ±  3%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      1.88 ±  4%      -0.2        1.64 ±  4%  perf-profile.calltrace.cycles-pp.ptrace_request.arch_ptrace.__x64_sys_ptrace.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.89 ±  4%      -0.2        1.65 ±  4%  perf-profile.calltrace.cycles-pp.arch_ptrace.__x64_sys_ptrace.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.02 ±  5%      -0.2        1.79 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.do_notify_parent_cldstop.ptrace_stop.ptrace_do_notify.ptrace_notify
      1.87 ±  4%      -0.2        1.64 ±  4%  perf-profile.calltrace.cycles-pp.ptrace_resume.ptrace_request.arch_ptrace.__x64_sys_ptrace.do_syscall_64
      1.90 ±  6%      -0.2        1.67 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.do_notify_parent_cldstop.ptrace_stop.ptrace_do_notify
      1.73 ±  4%      -0.2        1.50 ±  4%  perf-profile.calltrace.cycles-pp.try_to_wake_up.ptrace_resume.ptrace_request.arch_ptrace.__x64_sys_ptrace
      1.46 ±  6%      -0.2        1.25 ±  4%  perf-profile.calltrace.cycles-pp.schedule.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      1.42 ±  6%      -0.2        1.22 ±  4%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.do_wait.kernel_wait4.__do_sys_wait4
      0.63 ±  5%      -0.2        0.46 ± 44%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.ptrace_resume
      1.25 ±  6%      -0.2        1.09 ±  3%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      1.19 ±  5%      -0.1        1.05 ±  4%  perf-profile.calltrace.cycles-pp.do_notify_parent_cldstop.ptrace_stop.ptrace_do_notify.ptrace_notify.syscall_trace_enter
      0.75 ±  7%      -0.1        0.64 ±  3%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.do_wait.kernel_wait4
      0.83 ±  4%      -0.1        0.73 ±  5%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.ptrace_resume.ptrace_request
      0.67 ±  7%      -0.1        0.57 ±  4%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.do_wait
      0.86 ±  4%      -0.1        0.76 ±  4%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.ptrace_resume.ptrace_request.arch_ptrace
      0.61 ±  6%      +0.1        0.69 ±  4%  perf-profile.calltrace.cycles-pp.kmem_cache_free.remove_vma.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.78 ±  6%      +0.1        0.88 ±  3%  perf-profile.calltrace.cycles-pp.remove_vma.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      0.67 ±  8%      +0.1        0.78 ±  4%  perf-profile.calltrace.cycles-pp.anon_vma_interval_tree_insert.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm
      0.58 ±  4%      +0.1        0.69 ±  6%  perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      1.09 ±  5%      +0.1        1.22 ±  4%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      1.35 ±  5%      +0.2        1.53 ±  4%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap
      1.36 ±  5%      +0.2        1.54 ±  4%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.79 ±  5%      +0.2        0.99 ±  4%  perf-profile.calltrace.cycles-pp.ptrace_check_attach.__x64_sys_ptrace.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.00 ±  6%      +0.2        2.24 ±  3%  perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm.copy_process
      0.68 ±  7%      +0.3        0.94 ±  6%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.67 ±  7%      +0.3        0.93 ±  6%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.69 ±  7%      +0.3        0.94 ±  6%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.31 ±  6%      +0.3        2.57 ±  3%  perf-profile.calltrace.cycles-pp.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      2.42 ±  6%      +0.3        2.68 ±  3%  perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      1.24 ±  6%      +0.3        1.58 ±  7%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.18 ±141%      +0.4        0.55 ±  6%  perf-profile.calltrace.cycles-pp.__vmalloc_node_range.dup_task_struct.copy_process.kernel_clone.__do_sys_clone
      1.36 ±  6%      +0.4        1.74 ±  6%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.94 ±  6%      +0.4        2.36 ±  4%  perf-profile.calltrace.cycles-pp.unlink_anon_vmas.free_pgtables.unmap_region.__do_munmap.__vm_munmap
      0.26 ±100%      +0.5        0.73 ±  9%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.unmap_region.__do_munmap.__vm_munmap
      0.00            +0.6        0.58 ±  4%  perf-profile.calltrace.cycles-pp.wait_task_inactive.ptrace_check_attach.__x64_sys_ptrace.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.69 ±  5%      +0.6        3.33 ±  3%  perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      5.68 ±  5%      -0.6        5.06 ±  3%  perf-profile.children.cycles-pp.__sched_text_start
      3.69 ±  5%      -0.5        3.22 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      2.79 ±  6%      -0.4        2.42 ±  3%  perf-profile.children.cycles-pp.schedule_idle
      2.49 ±  5%      -0.3        2.21 ±  4%  perf-profile.children.cycles-pp.do_notify_parent_cldstop
      2.23 ±  5%      -0.3        1.96 ±  3%  perf-profile.children.cycles-pp.__wake_up_common_lock
      2.08 ±  5%      -0.3        1.83 ±  3%  perf-profile.children.cycles-pp.__wake_up_common
      1.89 ±  4%      -0.2        1.65 ±  4%  perf-profile.children.cycles-pp.arch_ptrace
      1.89 ±  4%      -0.2        1.64 ±  4%  perf-profile.children.cycles-pp.ptrace_request
      1.87 ±  4%      -0.2        1.64 ±  4%  perf-profile.children.cycles-pp.ptrace_resume
      1.68 ±  5%      -0.2        1.47 ±  4%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.70 ±  5%      -0.2        1.50 ±  4%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.33 ±  5%      -0.2        1.14 ±  4%  perf-profile.children.cycles-pp.enqueue_entity
      1.45 ±  5%      -0.2        1.26 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.47 ±  5%      -0.2        1.30 ±  3%  perf-profile.children.cycles-pp.dequeue_task_fair
      1.17 ±  5%      -0.2        1.02 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      1.33 ±  5%      -0.2        1.18 ±  2%  perf-profile.children.cycles-pp.dequeue_entity
      0.84 ±  7%      -0.1        0.73 ±  4%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.52 ±  8%      -0.1        0.43 ±  3%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.75 ±  4%      -0.1        0.66 ±  3%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.65 ±  4%      -0.1        0.57 ±  3%  perf-profile.children.cycles-pp.update_curr
      0.52 ±  6%      -0.1        0.44 ±  6%  perf-profile.children.cycles-pp.___might_sleep
      0.39 ±  5%      -0.1        0.33 ±  5%  perf-profile.children.cycles-pp.__switch_to_asm
      0.50 ±  5%      -0.1        0.44 ±  3%  perf-profile.children.cycles-pp.wake_up_new_task
      0.46 ±  5%      -0.1        0.41 ±  5%  perf-profile.children.cycles-pp.wp_page_copy
      0.41 ±  4%      -0.0        0.36 ±  2%  perf-profile.children.cycles-pp.find_idlest_group
      0.38 ±  6%      -0.0        0.33 ±  3%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.39 ±  6%      -0.0        0.34 ±  5%  perf-profile.children.cycles-pp.sched_clock
      0.37 ±  6%      -0.0        0.33 ±  6%  perf-profile.children.cycles-pp.native_sched_clock
      0.28 ±  7%      -0.0        0.24 ±  6%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.31 ±  3%      -0.0        0.28 ±  3%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.23 ±  6%      -0.0        0.19 ±  6%  perf-profile.children.cycles-pp.rmqueue
      0.18 ±  5%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.16 ±  4%      -0.0        0.13 ±  5%  perf-profile.children.cycles-pp.__cond_resched
      0.23 ±  5%      -0.0        0.20 ±  6%  perf-profile.children.cycles-pp.reweight_entity
      0.07 ± 11%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.19 ±  5%      -0.0        0.16 ±  4%  perf-profile.children.cycles-pp.allocate_slab
      0.26 ±  5%      -0.0        0.23 ±  6%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.14 ±  7%      -0.0        0.11 ±  9%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.17 ±  4%      -0.0        0.14 ±  4%  perf-profile.children.cycles-pp.___perf_sw_event
      0.20 ± 11%      -0.0        0.17 ±  7%  perf-profile.children.cycles-pp.sync_regs
      0.15 ±  5%      -0.0        0.12 ±  7%  perf-profile.children.cycles-pp.free_unref_page_list
      0.19 ±  9%      -0.0        0.17 ±  6%  perf-profile.children.cycles-pp.__put_user_nocheck_4
      0.09 ±  7%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.memcpy_erms
      0.24 ±  4%      -0.0        0.21 ±  4%  perf-profile.children.cycles-pp.__task_pid_nr_ns
      0.07 ±  6%      -0.0        0.05        perf-profile.children.cycles-pp.rcu_eqs_enter
      0.15 ±  4%      -0.0        0.13 ±  5%  perf-profile.children.cycles-pp.place_entity
      0.07 ± 10%      -0.0        0.05 ± 45%  perf-profile.children.cycles-pp.perf_event_task
      0.12 ±  5%      -0.0        0.10 ±  6%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.08 ±  6%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.arch_dup_task_struct
      0.09 ±  8%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.call_cpuidle
      0.07 ±  7%      -0.0        0.05 ± 45%  perf-profile.children.cycles-pp.lru_add_drain
      0.14 ±  5%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.copy_page
      0.07 ±  6%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.rcu_all_qs
      0.07 ±  7%      -0.0        0.05 ±  7%  perf-profile.children.cycles-pp.unfreeze_partials
      0.09 ±  8%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.__calc_delta
      0.08 ± 14%      +0.0        0.11 ± 11%  perf-profile.children.cycles-pp.userfaultfd_unmap_prep
      0.06 ± 11%      +0.0        0.09 ± 10%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.14 ± 11%      +0.0        0.18 ±  9%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge
      0.33 ±  5%      +0.0        0.37 ±  6%  perf-profile.children.cycles-pp.__put_anon_vma
      0.25 ±  5%      +0.0        0.29 ±  6%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.18 ±  7%      +0.0        0.22 ± 11%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.12 ± 10%      +0.1        0.17 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.02 ±141%      +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.lock_page_lruvec_irqsave
      0.16 ±  9%      +0.1        0.21 ±  8%  perf-profile.children.cycles-pp.__rb_erase_color
      0.19 ± 11%      +0.1        0.24 ±  9%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.18 ±  9%      +0.1        0.24 ±  9%  perf-profile.children.cycles-pp.page_counter_cancel
      0.26 ± 10%      +0.1        0.32 ±  3%  perf-profile.children.cycles-pp.find_get_task_by_vpid
      0.49 ±  8%      +0.1        0.55 ±  6%  perf-profile.children.cycles-pp.__vmalloc_node_range
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.21 ±  6%      +0.1        0.28 ±  8%  perf-profile.children.cycles-pp.drain_obj_stock
      0.58 ±  4%      +0.1        0.65 ±  3%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.52 ±  5%      +0.1        0.59 ±  2%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.16 ±  7%      +0.1        0.23 ±  7%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.44 ±  5%      +0.1        0.51 ±  9%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.59 ±  5%      +0.1        0.67        perf-profile.children.cycles-pp.__softirqentry_text_start
      0.38 ±  5%      +0.1        0.46 ±  4%  perf-profile.children.cycles-pp._raw_read_lock
      0.21 ± 13%      +0.1        0.28 ± 12%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      0.28 ±  5%      +0.1        0.37 ± 12%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.34 ±  5%      +0.1        0.43 ±  4%  perf-profile.children.cycles-pp.anon_vma_interval_tree_remove
      0.79 ±  6%      +0.1        0.88 ±  3%  perf-profile.children.cycles-pp.remove_vma
      0.28 ± 13%      +0.1        0.38 ±  7%  perf-profile.children.cycles-pp.alloc_vmap_area
      0.51 ±  6%      +0.1        0.60 ±  6%  perf-profile.children.cycles-pp.refill_obj_stock
      0.29 ± 14%      +0.1        0.39 ±  6%  perf-profile.children.cycles-pp.__get_vm_area_node
      0.28 ±  7%      +0.1        0.38 ±  8%  perf-profile.children.cycles-pp.tick_sched_handle
      0.27 ±  7%      +0.1        0.37 ±  8%  perf-profile.children.cycles-pp.update_process_times
      0.33 ±  7%      +0.1        0.45 ±  7%  perf-profile.children.cycles-pp.tick_sched_timer
      0.67 ±  7%      +0.1        0.79 ±  5%  perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      0.62 ±  6%      +0.1        0.74 ±  4%  perf-profile.children.cycles-pp.down_write
      0.46 ±  6%      +0.1        0.58 ±  4%  perf-profile.children.cycles-pp.wait_task_inactive
      0.48 ±  5%      +0.1        0.61 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.28 ±  6%      +0.1        0.41 ±  7%  perf-profile.children.cycles-pp.clockevents_program_event
      1.11 ±  5%      +0.1        1.25 ±  4%  perf-profile.children.cycles-pp.release_pages
      0.62 ±  4%      +0.1        0.77 ±  4%  perf-profile.children.cycles-pp.ktime_get
      0.09 ± 24%      +0.2        0.26 ± 11%  perf-profile.children.cycles-pp.osq_lock
      1.37 ±  5%      +0.2        1.54 ±  4%  perf-profile.children.cycles-pp.tlb_finish_mmu
      1.36 ±  5%      +0.2        1.54 ±  4%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.79 ±  5%      +0.2        0.99 ±  4%  perf-profile.children.cycles-pp.ptrace_check_attach
      0.51 ±  4%      +0.2        0.73 ±  9%  perf-profile.children.cycles-pp.unlink_file_vma
      2.00 ±  6%      +0.2        2.24 ±  3%  perf-profile.children.cycles-pp.anon_vma_clone
      2.33 ±  6%      +0.3        2.59 ±  3%  perf-profile.children.cycles-pp.filemap_map_pages
      2.43 ±  6%      +0.3        2.69 ±  3%  perf-profile.children.cycles-pp.do_fault
      1.00 ±  5%      +0.3        1.27 ±  6%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      0.22 ± 12%      +0.3        0.49 ±  5%  perf-profile.children.cycles-pp.cgroup_enter_frozen
      0.90 ±  4%      +0.3        1.18 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.91 ±  4%      +0.3        1.19 ±  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.26 ±  5%      +0.3        0.57 ±  5%  perf-profile.children.cycles-pp.cgroup_leave_frozen
      0.19 ± 12%      +0.3        0.51 ±  7%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      1.47 ±  3%      +0.3        1.82 ±  3%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      1.73 ±  3%      +0.4        2.12 ±  4%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      1.95 ±  6%      +0.4        2.38 ±  3%  perf-profile.children.cycles-pp.unlink_anon_vmas
      1.88 ±  3%      +0.4        2.32 ±  3%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.83 ±  6%      +0.6        1.42 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.35 ± 13%      +0.6        0.96 ±  6%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      2.69 ±  5%      +0.6        3.34 ±  3%  perf-profile.children.cycles-pp.free_pgtables
      1.23 ±  8%      +0.8        2.06 ±  5%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.15 ±  5%      -0.1        1.03 ±  4%  perf-profile.self.cycles-pp.__sched_text_start
      0.48 ±  8%      -0.1        0.38 ±  4%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.55 ±  5%      -0.1        0.45 ±  5%  perf-profile.self.cycles-pp.update_load_avg
      0.71 ±  6%      -0.1        0.62 ±  4%  perf-profile.self.cycles-pp.update_rq_clock
      0.51 ±  6%      -0.1        0.43 ±  6%  perf-profile.self.cycles-pp.___might_sleep
      0.54 ±  3%      -0.1        0.48 ±  5%  perf-profile.self.cycles-pp.do_idle
      0.39 ±  6%      -0.1        0.33 ±  5%  perf-profile.self.cycles-pp.__switch_to_asm
      0.31 ±  5%      -0.1        0.26 ±  4%  perf-profile.self.cycles-pp.update_curr
      0.24 ±  7%      -0.0        0.20 ±  7%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.36 ±  4%      -0.0        0.32 ±  3%  perf-profile.self.cycles-pp.find_idlest_group
      0.23 ±  5%      -0.0        0.20 ±  6%  perf-profile.self.cycles-pp.reweight_entity
      0.22 ±  4%      -0.0        0.19 ±  4%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.20 ± 10%      -0.0        0.17 ±  7%  perf-profile.self.cycles-pp.sync_regs
      0.12 ± 11%      -0.0        0.09 ± 11%  perf-profile.self.cycles-pp.try_charge
      0.09 ±  7%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.memcpy_erms
      0.36 ±  2%      -0.0        0.34 ±  3%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.23 ±  3%      -0.0        0.20 ±  4%  perf-profile.self.cycles-pp.__task_pid_nr_ns
      0.12 ±  6%      -0.0        0.09 ± 11%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.13 ±  5%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.___perf_sw_event
      0.16 ±  7%      -0.0        0.14 ±  5%  perf-profile.self.cycles-pp.dequeue_entity
      0.14 ±  4%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.__wake_up_common
      0.14 ±  6%      -0.0        0.12 ±  3%  perf-profile.self.cycles-pp.copy_page
      0.08 ±  5%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.call_cpuidle
      0.09 ±  8%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.__calc_delta
      0.07 ±  7%      +0.0        0.08        perf-profile.self.cycles-pp.alloc_vmap_area
      0.06 ±  7%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.07 ± 14%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.userfaultfd_unmap_prep
      0.08 ±  9%      +0.0        0.11 ±  5%  perf-profile.self.cycles-pp.drain_obj_stock
      0.15 ±  8%      +0.0        0.19 ±  5%  perf-profile.self.cycles-pp.unlink_anon_vmas
      0.16 ±  9%      +0.0        0.21 ±  9%  perf-profile.self.cycles-pp.page_counter_cancel
      0.12 ± 11%      +0.1        0.18 ± 11%  perf-profile.self.cycles-pp.__rb_erase_color
      0.27 ±  5%      +0.1        0.33 ±  6%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.cgroup_enter_frozen
      0.15 ±  8%      +0.1        0.21 ±  6%  perf-profile.self.cycles-pp.find_get_task_by_vpid
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.00            +0.1        0.07 ± 18%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.38 ±  6%      +0.1        0.45 ±  3%  perf-profile.self.cycles-pp._raw_read_lock
      0.06 ± 11%      +0.1        0.15 ±  7%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.33 ±  5%      +0.1        0.43 ±  4%  perf-profile.self.cycles-pp.anon_vma_interval_tree_remove
      0.27 ±  4%      +0.1        0.37 ± 12%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.81 ±  4%      +0.1        0.92 ±  4%  perf-profile.self.cycles-pp.kmem_cache_free
      0.67 ±  8%      +0.1        0.78 ±  4%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      0.31 ±  8%      +0.1        0.43 ±  5%  perf-profile.self.cycles-pp.ptrace_stop
      0.85 ±  6%      +0.1        0.97 ±  4%  perf-profile.self.cycles-pp.release_pages
      0.49 ±  6%      +0.1        0.62 ±  5%  perf-profile.self.cycles-pp.down_write
      0.63 ±  6%      +0.1        0.76 ±  4%  perf-profile.self.cycles-pp.do_wait
      0.17 ±  9%      +0.2        0.32 ±  3%  perf-profile.self.cycles-pp.wait_task_inactive
      0.56 ±  6%      +0.2        0.72 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.43 ±  6%      +0.2        0.59 ±  5%  perf-profile.self.cycles-pp.ktime_get
      0.09 ± 23%      +0.2        0.26 ± 11%  perf-profile.self.cycles-pp.osq_lock
      0.18 ± 12%      +0.3        0.50 ±  7%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      1.33 ±  6%      +0.3        1.67 ±  3%  perf-profile.self.cycles-pp.filemap_map_pages
      1.22 ±  7%      +0.8        2.04 ±  5%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                            pmeter.Average_Active_Power                         
                                                                                
  328 +---------------------------------------------------------------------+   
  326 |-+                               O                         O         |   
      |  O   O   O O    O  O  O      O O  O  O  O      O  O      O     O    |   
  324 |-O  O  O      O       O  O  O           O    O   O    O       O   OO |   
  322 |-+       O     O   O       O               O         O       O       |   
  320 |-+                                                                   |   
  318 |-+                                                                   |   
      |                                                                     |   
  316 |-+            +                                                      |   
  314 |-+            ::                                          +          |   
  312 |-+  +.       : +.   +.      +.        +.        ++       + :     .+  |   
  310 |-+ +  + .++. :   +. : +     : +.     +  +   .+ +  :   +.+  +.+ .+    |   
      |.++    +    +      +   +.+.+    ++.++    +.+  +   : .+        +      |   
  308 |-+                                                 +                 |   
  306 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             stress-ng.time.user_time                           
                                                                                
  220 +---------------------------------------------------------------------+   
      |.+                 + .++.+.      +.+            +   .+    ++.        |   
  215 |-++   ++.+   .++. + +      +  +.+   +   ++.+.+ + :.+  :  +   ++.+.+  |   
      |   + +    +.+    +          :+       + +      +  +    +.+            |   
  210 |-+  +                       +         +                              |   
      |                                                                     |   
  205 |-+                                                                   |   
      |                                                                     |   
  200 |-+                                                                   |   
      |                                                                     |   
  195 |-+O         O OO O O  OO O O  O OO O  O  O   O  O  O  O    O OO    O |   
      | O  O  O  O         O       O       O   O  O  O  O   O  O O     O O  |   
  190 |-+    O  O                                                           |   
      |                                                                     |   
  185 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.time.system_time                          
                                                                                
  405 +---------------------------------------------------------------------+   
      | O  O OO OO                                                          |   
  400 |-+O                                     O        O      O            |   
      |                    O       O       O O  O O OO O  O OO   OO OO O OO |   
      |            O OO O O  O  O O  O OO O                                 |   
  395 |-+                     O                                             |   
      |                                                                     |   
  390 |-+                                                                   |   
      |                                                                     |   
  385 |-+                                                                   |   
      |                                                                     |   
      |   .+.                      +.       .+.         +.                  |   
  380 |-++   ++.++.+.++.+. +.    .+  +.+   +   ++.+.++.+  +.++.+. +.+ .+.+  |   
      |.+                 +  ++.+       +.+                      +   +      |   
  375 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          stress-ng.time.major_page_faults                      
                                                                                
  55000 +-------------------------------------------------------------------+   
  50000 |-+     O                                                           |   
        |                                                                   |   
  45000 |-O                                                                 |   
  40000 |-+O O       O                                                      |   
  35000 |-+        O    O           O O    O        O         O       O     |   
  30000 |-+      O    O    O       O        O         O        O   O        |   
        |     O          O   OO OO      O       O  O     O  O    O  O  O  O |   
  25000 |-+                                   O  O     O   O             O  |   
  20000 |-+                              O                                  |   
  15000 |-+                                                                 |   
  10000 |-+                                                                 |   
        |.+ .++.++.+.++.++.+.++.++. +.+.++.+ .+.++. +.+ .+. +. +.+.++.++.+  |   
   5000 |-++                       +        +      +   +   +  +             |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       stress-ng.time.voluntary_context_switches                
                                                                                
  6.4e+07 +-----------------------------------------------------------------+   
          |                    +.                       +   .+   +.+        |   
  6.2e+07 |.++   .+     ++. +.+  +.++   .++.++   .+ .+. :: +  :  :  +.+ .+  |   
          |   + +  +.  +   +         + +      + +  +   + : :  +.+      +    |   
    6e+07 |-+  +     ++               +        +          +                 |   
          |                                                                 |   
  5.8e+07 |-+                                                               |   
          |                                                                 |   
  5.6e+07 |-+                                                               |   
          |                                                                 |   
  5.4e+07 |-+            O OO  O O O   O O  O     O  O     O  O    OO O   O |   
          | OO       OO O     O     O O   O  O OO  O   OO O  O  OO     O O  |   
  5.2e+07 |-+  OO OO                                                        |   
          |                                                                 |   
    5e+07 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                stress-ng.vm-segv.ops                           
                                                                                
  820000 +------------------------------------------------------------------+   
         |.++   .+     +.+  ++  +.++    +.++.+   .+ .+ ::  + +   :    + .+  |   
  800000 |-+ + +  +.  +   :+        + .+      :.+  +  :: :+   +.:      +    |   
  780000 |-+  +     ++    +          +        +       +  +      +           |   
         |                                                                  |   
  760000 |-+                                                                |   
         |                                                                  |   
  740000 |-+                                                                |   
         |                                                                  |   
  720000 |-+                                                                |   
  700000 |-+             O  O  OO      O          O        O        O O     |   
         |  O       OO O  O  O    OO    O OO OO O  O OO OO  O O OO O   O  O |   
  680000 |-O  O  OO                  O                                   O  |   
         |     O                                                            |   
  660000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.vm-segv.ops_per_sec                       
                                                                                
  14000 +-------------------------------------------------------------------+   
        |                                                                   |   
  13500 |.++   .+      .+  +.++.++.+   .++.++    +  +  +   ++    +.++.+     |   
        |   : +  +.   +  :+        :  +      :  + + :+ :+ +  +  +      +.+  |   
        |   : :    +.+   +          :+       : +   +  +  +    ++            |   
  13000 |-+  +                      +         +                             |   
        |                                                                   |   
  12500 |-+                                                                 |   
        |                                                                   |   
  12000 |-+                                                                 |   
        |               O  O    O                                     O     |   
        |  O         OO  O    O  O O  O OO O  O OO  O  O O O  O    OO     O |   
  11500 |-O  O  OO O         O      O       O      O  O     O  O O     O O  |   
        |     O                                                             |   
  11000 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.11.0-00053-gb360fb5e5954" of type "text/plain" (172475 bytes)

View attachment "job-script" of type "text/plain" (8469 bytes)

View attachment "job.yaml" of type "text/plain" (5706 bytes)

View attachment "reproduce" of type "text/plain" (466 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ