lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20210414052151.GB21236@xsang-OptiPlex-9020>
Date:   Wed, 14 Apr 2021 13:21:51 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Valentin Schneider <valentin.schneider@....com>
Cc:     0day robot <lkp@...el.com>,
        Valentin Schneider <valentin.schneider@....com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com,
        Lingutla Chandrasekhar <clingutla@...eaurora.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...nel.org>,
        Morten Rasmussen <morten.rasmussen@....com>,
        Qais Yousef <qais.yousef@....com>,
        Quentin Perret <qperret@...gle.com>,
        Pavan Kondeti <pkondeti@...eaurora.org>,
        Rik van Riel <riel@...riel.com>, aubrey.li@...ux.intel.com,
        yu.c.chen@...el.com
Subject: [sched/fair]  38ac256d1c:  stress-ng.vm-segv.ops_per_sec -13.8%
 regression



Greeting,

FYI, we noticed a -13.8% regression of stress-ng.vm-segv.ops_per_sec due to commit:


commit: 38ac256d1c3e6b5155071ed7ba87db50a40a4b58 ("[PATCH v5 1/3] sched/fair: Ignore percpu threads for imbalance pulls")
url: https://github.com/0day-ci/linux/commits/Valentin-Schneider/sched-fair-load-balance-vs-capacity-margins/20210408-060830
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 0a2b65c03e9b47493e1442bf9c84badc60d9bffb

in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
with following parameters:

	nr_threads: 10%
	disk: 1HDD
	testtime: 60s
	fs: ext4
	class: os
	test: vm-segv
	cpufreq_governor: performance
	ucode: 0x5003006




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  os/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/vm-segv/stress-ng/60s/0x5003006

commit: 
  0a2b65c03e ("sched/topology: Remove redundant cpumask_and() in init_overlap_sched_group()")
  38ac256d1c ("sched/fair: Ignore percpu threads for imbalance pulls")

0a2b65c03e9b4749 38ac256d1c3e6b5155071ed7ba8 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     10556 ±  7%     -15.5%       8923 ±  6%  stress-ng.time.involuntary_context_switches
      8274 ± 19%    +271.7%      30754 ± 16%  stress-ng.time.major_page_faults
  31542886           -11.8%   27817867 ±  2%  stress-ng.time.minor_page_faults
    376.97            +5.3%     397.03        stress-ng.time.system_time
    220.33           -10.0%     198.19        stress-ng.time.user_time
  63365962           -13.9%   54528489        stress-ng.time.voluntary_context_switches
    825247           -13.8%     711542        stress-ng.vm-segv.ops
     13754           -13.8%      11859        stress-ng.vm-segv.ops_per_sec
      9.88            -3.2%       9.57        iostat.cpu.system
      9515 ±  5%     -16.2%       7971 ± 10%  numa-meminfo.node1.PageTables
      0.05 ± 38%     -40.6%       0.03 ± 46%  perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
   1976755           -13.9%    1701336        vmstat.system.cs
   7875426 ±  5%     -14.0%    6774658 ±  4%  numa-vmstat.node1.numa_hit
   7710987 ±  6%     -14.4%    6600351 ±  4%  numa-vmstat.node1.numa_local
      1.13 ± 10%      -0.4        0.75 ±  2%  mpstat.cpu.all.irq%
      0.16 ±  4%      -0.0        0.13        mpstat.cpu.all.soft%
      0.41            -0.0        0.36        mpstat.cpu.all.usr%
  16251159 ±  5%     -11.4%   14396125 ±  4%  numa-numastat.node0.local_node
  16294407 ±  5%     -11.3%   14449658 ±  4%  numa-numastat.node0.numa_hit
  14611342 ±  4%     -12.3%   12818759 ±  6%  numa-numastat.node1.local_node
  14654675 ±  4%     -12.3%   12851787 ±  6%  numa-numastat.node1.numa_hit
  14390097 ±  4%      -9.8%   12981975 ±  2%  cpuidle.C1.usage
 1.895e+09 ± 56%    +120.2%  4.172e+09 ± 31%  cpuidle.C1E.time
   9181457 ± 14%     +43.7%   13193802 ± 11%  cpuidle.C1E.usage
 2.981e+09 ± 35%     -74.9%  7.473e+08 ±171%  cpuidle.C6.time
   3933755 ± 16%     -76.8%     911993 ±161%  cpuidle.C6.usage
 2.532e+08           -11.5%  2.241e+08        cpuidle.POLL.time
  53648297           -16.4%   44835331        cpuidle.POLL.usage
     11505 ±  2%     +10.7%      12731 ±  5%  softirqs.CPU17.RCU
     11174 ±  3%     +12.9%      12612 ±  2%  softirqs.CPU19.RCU
     13689 ±  3%     -12.8%      11932 ±  4%  softirqs.CPU2.RCU
     12657 ±  8%     -13.6%      10936 ±  4%  softirqs.CPU41.RCU
     13120 ±  5%     -11.2%      11651 ±  5%  softirqs.CPU5.RCU
     13395 ±  3%     -11.1%      11912 ±  4%  softirqs.CPU59.RCU
     11401 ±  4%     +18.1%      13461 ±  3%  softirqs.CPU64.RCU
     11445 ±  4%     +13.2%      12957 ±  3%  softirqs.CPU70.RCU
     13007 ±  7%     -13.6%      11236 ±  4%  softirqs.CPU88.RCU
      3685 ± 22%     +83.8%       6775 ± 12%  sched_debug.cfs_rq:/.load_avg.max
    468.39 ± 25%     +69.7%     794.93 ± 14%  sched_debug.cfs_rq:/.load_avg.stddev
      1016 ±  7%     +35.1%       1372 ± 16%  sched_debug.cfs_rq:/.runnable_avg.max
      1015 ±  7%     +34.8%       1368 ± 16%  sched_debug.cfs_rq:/.util_avg.max
     40.25 ± 25%     -38.9%      24.60 ± 19%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    124.29 ±  9%     -20.8%      98.40 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    485296 ± 24%     +51.3%     734220 ±  2%  sched_debug.cpu.avg_idle.avg
     58068 ± 18%     -42.3%      33494        sched_debug.cpu.clock_task.min
     30525 ± 22%     +64.2%      50117 ± 15%  sched_debug.cpu.max_idle_balance_cost.stddev
    529606 ± 44%     -99.7%       1545        sched_debug.cpu.nr_switches.avg
     64010            +1.3%      64842        proc-vmstat.nr_anon_pages
     68096            +1.3%      68955        proc-vmstat.nr_inactive_anon
     20385            -3.0%      19773        proc-vmstat.nr_kernel_stack
     10555            +2.1%      10780        proc-vmstat.nr_mapped
      4795 ±  3%      -6.8%       4468 ±  3%  proc-vmstat.nr_page_table_pages
     23584            -4.0%      22641        proc-vmstat.nr_slab_reclaimable
     68096            +1.3%      68955        proc-vmstat.nr_zone_inactive_anon
  30681832           -11.9%   27040526 ±  2%  proc-vmstat.numa_hit
  30595238           -11.9%   26953947 ±  2%  proc-vmstat.numa_local
      9485 ±  2%     -11.6%       8382 ±  3%  proc-vmstat.pgactivate
  34797690           -12.6%   30406355        proc-vmstat.pgalloc_normal
  31794224           -11.6%   28112869 ±  2%  proc-vmstat.pgfault
  34656033           -12.7%   30264573        proc-vmstat.pgfree
    350358            -7.0%     325953        proc-vmstat.pgreuse
     77850           +13.6%      88414        slabinfo.anon_vma_chain.active_objs
      1219           +13.4%       1382        slabinfo.anon_vma_chain.active_slabs
     78081           +13.4%      88521        slabinfo.anon_vma_chain.num_objs
      1219           +13.4%       1382        slabinfo.anon_vma_chain.num_slabs
     21930           +12.7%      24710        slabinfo.kmalloc-512.active_objs
     23305           +10.9%      25855        slabinfo.kmalloc-512.num_objs
      9912 ±  2%      -9.8%       8937 ±  2%  slabinfo.pid.active_objs
      9912 ±  2%      -9.8%       8937 ±  2%  slabinfo.pid.num_objs
     28846           -11.5%      25518        slabinfo.proc_inode_cache.active_objs
     28964           -11.6%      25615        slabinfo.proc_inode_cache.num_objs
      8445 ±  3%      -9.0%       7684 ±  2%  slabinfo.signal_cache.active_objs
      8471 ±  3%      -8.6%       7741 ±  2%  slabinfo.signal_cache.num_objs
     10704 ±  5%     -15.1%       9084 ±  5%  slabinfo.task_delay_info.active_objs
     10704 ±  5%     -15.1%       9084 ±  5%  slabinfo.task_delay_info.num_objs
     53709           +11.3%      59765        slabinfo.vm_area_struct.active_objs
      1345           +11.1%       1494        slabinfo.vm_area_struct.active_slabs
     53830           +11.1%      59807        slabinfo.vm_area_struct.num_objs
      1345           +11.1%       1494        slabinfo.vm_area_struct.num_slabs
     19920           +32.1%      26323 ±  9%  slabinfo.vmap_area.active_objs
     24255           +28.6%      31193 ±  3%  slabinfo.vmap_area.num_objs
  4.91e+09           -12.3%  4.305e+09        perf-stat.i.branch-instructions
  54724882 ±  3%     -13.1%   47578232        perf-stat.i.branch-misses
      4.19 ±  7%      +6.1       10.28 ±  2%  perf-stat.i.cache-miss-rate%
  18351457 ±  4%    +131.0%   42389073 ±  3%  perf-stat.i.cache-misses
  4.88e+08           -12.4%  4.274e+08        perf-stat.i.cache-references
   2037763           -14.0%    1751993        perf-stat.i.context-switches
      1.88            +9.2%       2.05        perf-stat.i.cpi
  4.27e+10            -3.6%  4.117e+10        perf-stat.i.cpu-cycles
    889.49           +11.7%     993.90 ±  3%  perf-stat.i.cpu-migrations
      2504 ±  5%     -55.9%       1103 ±  4%  perf-stat.i.cycles-between-cache-misses
      0.07 ±  7%      -0.0        0.06 ±  7%  perf-stat.i.dTLB-load-miss-rate%
   4137770 ±  5%     -22.5%    3207849 ±  4%  perf-stat.i.dTLB-load-misses
 6.238e+09           -12.9%  5.431e+09        perf-stat.i.dTLB-loads
   1200459 ±  4%     -14.2%    1029904 ±  5%  perf-stat.i.dTLB-store-misses
 3.242e+09           -13.1%  2.818e+09        perf-stat.i.dTLB-stores
     63.73            -1.5       62.22        perf-stat.i.iTLB-load-miss-rate%
  16650309           -10.4%   14923703        perf-stat.i.iTLB-load-misses
 2.289e+10           -12.6%  2.001e+10        perf-stat.i.instructions
      0.54            -8.1%       0.49        perf-stat.i.ipc
    134.02 ± 19%    +268.2%     493.46 ± 16%  perf-stat.i.major-faults
      0.44            -3.6%       0.43        perf-stat.i.metric.GHz
      0.04 ± 17%     +57.5%       0.06 ± 23%  perf-stat.i.metric.K/sec
    155.20           -12.7%     135.54        perf-stat.i.metric.M/sec
    503703           -11.8%     444204 ±  2%  perf-stat.i.minor-faults
     77.50           +12.6       90.06        perf-stat.i.node-load-miss-rate%
   3996669 ±  3%    +201.3%   12040510 ±  2%  perf-stat.i.node-load-misses
    988076           +12.3%    1109737 ±  2%  perf-stat.i.node-loads
     40.07 ±  3%     +33.2       73.31        perf-stat.i.node-store-miss-rate%
   1484575 ±  6%    +217.6%    4714862 ±  2%  perf-stat.i.node-store-misses
   2054293           -25.9%    1521550 ±  2%  perf-stat.i.node-stores
    530016           -11.8%     467246 ±  2%  perf-stat.i.page-faults
      3.75 ±  4%      +6.1        9.86 ±  2%  perf-stat.overall.cache-miss-rate%
      1.87           +10.2%       2.06        perf-stat.overall.cpi
      2334 ±  4%     -58.2%     976.94 ±  2%  perf-stat.overall.cycles-between-cache-misses
     64.30            -1.3       62.95        perf-stat.overall.iTLB-load-miss-rate%
      0.54            -9.2%       0.49        perf-stat.overall.ipc
     79.96           +11.5       91.48        perf-stat.overall.node-load-miss-rate%
     41.83 ±  3%     +33.6       75.44        perf-stat.overall.node-store-miss-rate%
 4.834e+09           -12.2%  4.242e+09        perf-stat.ps.branch-instructions
  53845162 ±  3%     -13.0%   46852865        perf-stat.ps.branch-misses
  18040576 ±  4%    +130.2%   41533952 ±  3%  perf-stat.ps.cache-misses
 4.804e+08           -12.3%  4.212e+08        perf-stat.ps.cache-references
   2005942           -13.9%    1727004        perf-stat.ps.context-switches
 4.204e+10            -3.6%  4.054e+10        perf-stat.ps.cpu-cycles
    874.22           +11.7%     976.52 ±  3%  perf-stat.ps.cpu-migrations
   4070531 ±  5%     -22.3%    3161516 ±  4%  perf-stat.ps.dTLB-load-misses
 6.142e+09           -12.8%  5.353e+09        perf-stat.ps.dTLB-loads
   1180924 ±  4%     -14.1%    1014912 ±  5%  perf-stat.ps.dTLB-store-misses
 3.192e+09           -13.0%  2.777e+09        perf-stat.ps.dTLB-stores
  16392877           -10.3%   14705263        perf-stat.ps.iTLB-load-misses
 2.253e+10           -12.5%  1.972e+10        perf-stat.ps.instructions
    131.26 ± 19%    +267.7%     482.61 ± 16%  perf-stat.ps.major-faults
    495822           -11.7%     437777 ±  2%  perf-stat.ps.minor-faults
   3919395 ±  3%    +200.6%   11783130 ±  2%  perf-stat.ps.node-load-misses
    980975           +11.9%    1097908 ±  2%  perf-stat.ps.node-loads
   1457072 ±  6%    +216.8%    4615375 ±  2%  perf-stat.ps.node-store-misses
   2023350           -25.8%    1502038        perf-stat.ps.node-stores
    521723           -11.7%     460486 ±  2%  perf-stat.ps.page-faults
 1.418e+12           -12.4%  1.243e+12        perf-stat.total.instructions
    269464 ±  4%     -21.8%     210839        interrupts.CAL:Function_call_interrupts
      2722 ±  6%     -39.8%       1640 ± 24%  interrupts.CPU0.CAL:Function_call_interrupts
     87.50 ± 20%     -73.5%      23.17 ± 16%  interrupts.CPU0.TLB:TLB_shootdowns
      3915 ± 11%     -33.7%       2597 ± 22%  interrupts.CPU1.CAL:Function_call_interrupts
      2657 ± 17%     -39.2%       1614 ±  9%  interrupts.CPU10.CAL:Function_call_interrupts
      2798 ± 15%     -22.3%       2174 ±  8%  interrupts.CPU12.CAL:Function_call_interrupts
      2663 ±  7%     -36.2%       1700 ± 16%  interrupts.CPU13.CAL:Function_call_interrupts
    465.17 ± 12%     +34.7%     626.50 ± 15%  interrupts.CPU17.RES:Rescheduling_interrupts
    505.83 ± 12%     +30.9%     662.17 ±  7%  interrupts.CPU19.RES:Rescheduling_interrupts
      3886 ± 17%     -38.8%       2379 ± 24%  interrupts.CPU2.CAL:Function_call_interrupts
      2845 ± 10%     -27.6%       2059 ± 11%  interrupts.CPU23.CAL:Function_call_interrupts
      3309 ±  4%     -42.0%       1919 ± 16%  interrupts.CPU24.CAL:Function_call_interrupts
    920.17 ±  4%     +54.4%       1420 ± 12%  interrupts.CPU24.RES:Rescheduling_interrupts
    892.83 ±  8%    +105.1%       1831 ± 32%  interrupts.CPU25.RES:Rescheduling_interrupts
    756.50 ± 10%     +66.8%       1262 ± 30%  interrupts.CPU28.RES:Rescheduling_interrupts
    666.33 ± 12%     +83.5%       1222 ± 13%  interrupts.CPU29.RES:Rescheduling_interrupts
      3515 ±  7%     -34.1%       2315 ± 23%  interrupts.CPU3.CAL:Function_call_interrupts
     84.00 ± 12%     -82.1%      15.00 ± 56%  interrupts.CPU3.TLB:TLB_shootdowns
    597.83 ±  9%    +130.3%       1376 ± 32%  interrupts.CPU30.RES:Rescheduling_interrupts
    613.83 ± 24%    +150.7%       1538 ± 16%  interrupts.CPU31.RES:Rescheduling_interrupts
    673.17 ± 19%    +111.6%       1424 ± 24%  interrupts.CPU32.RES:Rescheduling_interrupts
      2680 ± 15%     -31.7%       1831 ± 15%  interrupts.CPU34.CAL:Function_call_interrupts
    699.67 ± 17%    +101.6%       1410 ± 29%  interrupts.CPU35.RES:Rescheduling_interrupts
    621.67 ± 10%    +127.1%       1411 ± 21%  interrupts.CPU36.RES:Rescheduling_interrupts
    682.33 ± 10%    +105.1%       1399 ± 36%  interrupts.CPU37.RES:Rescheduling_interrupts
      2765 ± 12%     -34.7%       1806 ±  3%  interrupts.CPU38.CAL:Function_call_interrupts
    681.67 ± 16%     +78.1%       1214 ± 18%  interrupts.CPU38.RES:Rescheduling_interrupts
    582.17 ± 20%    +192.1%       1700 ± 23%  interrupts.CPU39.RES:Rescheduling_interrupts
      3358 ±  7%     -31.5%       2299 ± 11%  interrupts.CPU4.CAL:Function_call_interrupts
      2485 ±  8%     -26.2%       1833 ± 21%  interrupts.CPU42.CAL:Function_call_interrupts
    652.17 ± 14%     +93.1%       1259 ± 28%  interrupts.CPU43.RES:Rescheduling_interrupts
    654.83 ±  8%    +160.4%       1705 ± 18%  interrupts.CPU44.RES:Rescheduling_interrupts
    670.67 ± 24%    +101.7%       1352 ± 22%  interrupts.CPU45.RES:Rescheduling_interrupts
    579.17 ± 19%    +140.3%       1391 ± 17%  interrupts.CPU47.RES:Rescheduling_interrupts
      3018 ± 16%     -20.9%       2385 ± 15%  interrupts.CPU49.CAL:Function_call_interrupts
      3292 ± 16%     -34.5%       2154 ± 20%  interrupts.CPU5.CAL:Function_call_interrupts
      3235 ± 13%     -32.4%       2185 ± 15%  interrupts.CPU50.CAL:Function_call_interrupts
      1645 ± 29%     +49.3%       2457 ±  7%  interrupts.CPU50.NMI:Non-maskable_interrupts
      1645 ± 29%     +49.3%       2457 ±  7%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      3248 ±  5%     -26.2%       2396 ± 19%  interrupts.CPU51.CAL:Function_call_interrupts
      3252 ± 18%     -31.0%       2245 ± 19%  interrupts.CPU52.CAL:Function_call_interrupts
    554.00 ± 14%     +31.3%     727.17 ± 17%  interrupts.CPU53.RES:Rescheduling_interrupts
      2787 ± 22%     -24.7%       2098 ± 11%  interrupts.CPU58.CAL:Function_call_interrupts
      2837 ±  9%     -25.8%       2105 ± 18%  interrupts.CPU59.CAL:Function_call_interrupts
    572.50 ± 14%     +45.6%     833.67 ±  9%  interrupts.CPU64.RES:Rescheduling_interrupts
      2850 ± 20%     -28.0%       2051 ± 19%  interrupts.CPU66.CAL:Function_call_interrupts
      2664 ± 15%     -26.7%       1954 ± 10%  interrupts.CPU67.CAL:Function_call_interrupts
      3049 ± 10%     -32.5%       2058 ± 19%  interrupts.CPU7.CAL:Function_call_interrupts
    547.67 ± 10%     +19.2%     653.00 ±  7%  interrupts.CPU70.RES:Rescheduling_interrupts
      2861 ± 17%     -32.0%       1946 ± 27%  interrupts.CPU71.CAL:Function_call_interrupts
    835.50 ± 17%    +143.1%       2031 ± 32%  interrupts.CPU72.RES:Rescheduling_interrupts
    740.17 ± 14%    +132.8%       1722 ± 17%  interrupts.CPU73.RES:Rescheduling_interrupts
      2531 ±  8%     -24.6%       1909 ± 11%  interrupts.CPU74.CAL:Function_call_interrupts
    712.00 ± 16%    +111.8%       1507 ± 22%  interrupts.CPU74.RES:Rescheduling_interrupts
      2902 ±  8%     -30.8%       2008 ± 15%  interrupts.CPU75.CAL:Function_call_interrupts
    761.67 ±  9%     +59.5%       1215 ± 21%  interrupts.CPU75.RES:Rescheduling_interrupts
    628.50 ± 12%    +125.2%       1415 ± 15%  interrupts.CPU77.RES:Rescheduling_interrupts
    604.67 ±  8%    +139.7%       1449 ± 21%  interrupts.CPU78.RES:Rescheduling_interrupts
      2793 ± 11%     -29.6%       1966 ± 32%  interrupts.CPU79.CAL:Function_call_interrupts
      3104 ± 19%     -38.3%       1916 ± 24%  interrupts.CPU8.CAL:Function_call_interrupts
    697.33 ± 16%    +110.5%       1467 ± 25%  interrupts.CPU80.RES:Rescheduling_interrupts
    714.83 ± 10%    +137.3%       1696 ± 19%  interrupts.CPU81.RES:Rescheduling_interrupts
    707.00 ± 13%    +125.0%       1591 ± 23%  interrupts.CPU82.RES:Rescheduling_interrupts
    732.00 ± 12%     +69.0%       1237 ± 24%  interrupts.CPU83.RES:Rescheduling_interrupts
    702.33 ± 10%    +142.2%       1700 ± 24%  interrupts.CPU84.RES:Rescheduling_interrupts
    729.50 ± 12%    +177.8%       2026 ± 16%  interrupts.CPU85.RES:Rescheduling_interrupts
      3080 ± 15%     -37.0%       1942 ± 16%  interrupts.CPU86.CAL:Function_call_interrupts
    793.50 ±  9%     +67.9%       1332 ± 23%  interrupts.CPU86.RES:Rescheduling_interrupts
      2618 ± 11%     -23.7%       1998 ± 18%  interrupts.CPU87.CAL:Function_call_interrupts
    715.50 ±  4%     +85.2%       1325 ± 15%  interrupts.CPU87.RES:Rescheduling_interrupts
    599.83 ±  9%    +126.3%       1357 ± 16%  interrupts.CPU89.RES:Rescheduling_interrupts
      2769 ± 17%     -28.2%       1988 ± 10%  interrupts.CPU9.CAL:Function_call_interrupts
    699.67 ±  3%    +150.1%       1749 ± 21%  interrupts.CPU90.RES:Rescheduling_interrupts
    699.00 ± 10%     +87.0%       1307 ± 20%  interrupts.CPU91.RES:Rescheduling_interrupts
      2623 ± 10%     -21.3%       2063 ± 12%  interrupts.CPU92.CAL:Function_call_interrupts
    717.17 ±  8%     +94.4%       1394 ± 23%  interrupts.CPU92.RES:Rescheduling_interrupts
    742.50 ±  9%     +98.3%       1472 ± 11%  interrupts.CPU93.RES:Rescheduling_interrupts
    653.67 ± 12%    +127.2%       1485 ± 16%  interrupts.CPU94.RES:Rescheduling_interrupts
      2667 ± 16%     -30.2%       1863 ± 15%  interrupts.CPU95.CAL:Function_call_interrupts
    578.67 ±  8%     +75.5%       1015 ± 31%  interrupts.CPU95.RES:Rescheduling_interrupts
     86.83 ± 26%     -67.0%      28.67 ± 20%  interrupts.CPU95.TLB:TLB_shootdowns
     59236           +64.3%      97341 ±  2%  interrupts.RES:Rescheduling_interrupts
      6261 ±  5%     -83.2%       1049 ± 17%  interrupts.TLB:TLB_shootdowns
     18.38 ±  2%      -2.7       15.65 ±  6%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      2.86 ±  2%      -0.5        2.34 ±  6%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.77 ±  2%      -0.5        2.27 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      2.03            -0.4        1.67 ±  6%  perf-profile.calltrace.cycles-pp.ptrace_request.arch_ptrace.__x64_sys_ptrace.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.04            -0.4        1.68 ±  6%  perf-profile.calltrace.cycles-pp.arch_ptrace.__x64_sys_ptrace.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.02            -0.4        1.66 ±  6%  perf-profile.calltrace.cycles-pp.ptrace_resume.ptrace_request.arch_ptrace.__x64_sys_ptrace.do_syscall_64
      2.28 ±  2%      -0.3        1.93 ±  7%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.do_notify_parent_cldstop.ptrace_stop.ptrace_do_notify.ptrace_notify
      2.16            -0.3        1.82 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.do_notify_parent_cldstop.ptrace_stop.ptrace_do_notify
      1.84            -0.3        1.50 ±  6%  perf-profile.calltrace.cycles-pp.try_to_wake_up.ptrace_resume.ptrace_request.arch_ptrace.__x64_sys_ptrace
      1.98            -0.3        1.67 ±  7%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.do_notify_parent_cldstop.ptrace_stop
      0.57 ±  4%      -0.3        0.26 ±100%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.64 ±  6%      -0.3        0.35 ± 71%  perf-profile.calltrace.cycles-pp.do_set_pte.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault
      1.26 ± 16%      -0.3        0.99 ±  5%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.50 ±  2%      -0.3        1.24 ±  6%  perf-profile.calltrace.cycles-pp.schedule.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      2.38 ±  6%      -0.3        2.12 ±  5%  perf-profile.calltrace.cycles-pp.copy_page_range.dup_mmap.dup_mm.copy_process.kernel_clone
      1.45 ±  3%      -0.2        1.20 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule.do_wait.kernel_wait4.__do_sys_wait4
      1.30 ±  2%      -0.2        1.08 ±  8%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.70 ±  2%      -0.2        0.48 ± 45%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.ptrace_resume
      1.32            -0.2        1.12 ±  6%  perf-profile.calltrace.cycles-pp.do_notify_parent_cldstop.ptrace_stop.ptrace_do_notify.ptrace_notify.syscall_trace_enter
      1.30 ±  3%      -0.2        1.11 ±  6%  perf-profile.calltrace.cycles-pp.do_notify_parent_cldstop.ptrace_stop.ptrace_do_notify.ptrace_notify.syscall_exit_to_user_mode_prepare
      1.47            -0.2        1.29 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule.ptrace_stop.ptrace_do_notify.ptrace_notify
      1.02 ±  4%      -0.2        0.85 ±  7%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle
      0.97            -0.2        0.80 ±  8%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.ptrace_resume.ptrace_request.arch_ptrace
      0.93            -0.2        0.76 ±  7%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.ptrace_resume.ptrace_request
      0.78 ±  2%      -0.2        0.62 ±  5%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.do_wait.kernel_wait4
      0.71 ±  2%      -0.1        0.56 ±  5%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.do_wait
      1.07 ±  3%      -0.1        0.94 ±  5%  perf-profile.calltrace.cycles-pp.vm_area_dup.dup_mmap.dup_mm.copy_process.kernel_clone
      0.78 ±  6%      -0.1        0.67 ±  5%  perf-profile.calltrace.cycles-pp.__pte_alloc.copy_pte_range.copy_page_range.dup_mmap.dup_mm
      0.72 ±  6%      -0.1        0.61 ±  5%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc.copy_pte_range.copy_page_range
      0.76 ±  6%      -0.1        0.66 ±  5%  perf-profile.calltrace.cycles-pp.pte_alloc_one.__pte_alloc.copy_pte_range.copy_page_range.dup_mmap
      0.76            -0.1        0.66 ±  7%  perf-profile.calltrace.cycles-pp.schedule.ptrace_stop.ptrace_do_notify.ptrace_notify.syscall_exit_to_user_mode_prepare
      0.75 ±  2%      -0.1        0.65 ±  5%  perf-profile.calltrace.cycles-pp.schedule.ptrace_stop.ptrace_do_notify.ptrace_notify.syscall_trace_enter
      0.68 ±  4%      -0.1        0.62 ±  5%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.67 ±  4%      -0.1        0.61 ±  5%  perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.67 ±  4%      -0.1        0.61 ±  5%  perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.68 ±  4%      -0.1        0.62 ±  5%  perf-profile.calltrace.cycles-pp.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.61 ±  3%      -0.1        0.55 ±  5%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.vm_area_dup.dup_mmap.dup_mm.copy_process
      1.17 ±  4%      +0.1        1.27 ±  5%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      0.60 ±  8%      +0.1        0.71 ±  7%  perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.68            +0.1        0.79 ±  4%  perf-profile.calltrace.cycles-pp.anon_vma_interval_tree_insert.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm
      0.79 ±  3%      +0.1        0.91 ±  6%  perf-profile.calltrace.cycles-pp.ptrace_check_attach.__x64_sys_ptrace.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.46 ±  4%      +0.1        1.58 ±  5%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap
      1.87            +0.2        2.06 ±  5%  perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm.copy_process
      0.46 ± 44%      +0.2        0.71 ±  2%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.unmap_region.__do_munmap.__vm_munmap
      2.66 ±  2%      +0.3        2.92 ±  5%  perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mmap.dup_mm.copy_process.kernel_clone
      1.87 ±  3%      +0.3        2.14 ±  5%  perf-profile.calltrace.cycles-pp.unlink_anon_vmas.free_pgtables.unmap_region.__do_munmap.__vm_munmap
      1.20 ±  3%      +0.3        1.52 ±  7%  perf-profile.calltrace.cycles-pp.next_uptodate_page.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault
      2.64 ±  3%      +0.4        3.05 ±  4%  perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
     32.82 ±  2%      +5.3       38.16 ±  7%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     18.60 ±  2%      -2.8       15.83 ±  5%  perf-profile.children.cycles-pp.poll_idle
      5.92            -1.0        4.95 ±  6%  perf-profile.children.cycles-pp.__schedule
      4.08            -0.7        3.39 ±  6%  perf-profile.children.cycles-pp.try_to_wake_up
      2.89 ±  2%      -0.5        2.37 ±  6%  perf-profile.children.cycles-pp.schedule_idle
      3.14            -0.5        2.66 ±  6%  perf-profile.children.cycles-pp.schedule
      2.78 ±  2%      -0.4        2.35 ±  6%  perf-profile.children.cycles-pp.do_notify_parent_cldstop
      2.49 ±  2%      -0.4        2.10 ±  6%  perf-profile.children.cycles-pp.__wake_up_common_lock
      2.35            -0.4        1.99 ±  6%  perf-profile.children.cycles-pp.__wake_up_common
      2.04            -0.4        1.68 ±  6%  perf-profile.children.cycles-pp.ptrace_request
      2.02            -0.4        1.67 ±  6%  perf-profile.children.cycles-pp.ptrace_resume
      2.04            -0.4        1.68 ±  6%  perf-profile.children.cycles-pp.arch_ptrace
      1.91            -0.3        1.57 ±  6%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.88            -0.3        1.54 ±  6%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.53            -0.3        1.25 ±  5%  perf-profile.children.cycles-pp.dequeue_task_fair
      1.29 ± 16%      -0.3        1.01 ±  4%  perf-profile.children.cycles-pp.menu_select
      1.45 ±  2%      -0.3        1.18 ±  6%  perf-profile.children.cycles-pp.enqueue_entity
      1.50            -0.3        1.24 ±  7%  perf-profile.children.cycles-pp.pick_next_task_fair
      2.39 ±  6%      -0.3        2.13 ±  5%  perf-profile.children.cycles-pp.copy_page_range
      1.39 ±  2%      -0.3        1.13 ±  5%  perf-profile.children.cycles-pp.dequeue_entity
      1.17 ±  2%      -0.2        0.95 ±  5%  perf-profile.children.cycles-pp.update_load_avg
      1.59 ±  3%      -0.2        1.38 ±  5%  perf-profile.children.cycles-pp.kmem_cache_alloc
      1.45 ±  4%      -0.2        1.26 ±  2%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.05 ±  4%      -0.2        0.86 ±  7%  perf-profile.children.cycles-pp.set_next_entity
      0.87 ±  7%      -0.2        0.71 ±  5%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.91 ±  2%      -0.1        0.77 ±  6%  perf-profile.children.cycles-pp.update_rq_clock
      1.07 ±  3%      -0.1        0.94 ±  6%  perf-profile.children.cycles-pp.vm_area_dup
      0.76 ±  3%      -0.1        0.64 ±  4%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.64 ±  6%      -0.1        0.53 ±  6%  perf-profile.children.cycles-pp.do_set_pte
      0.49 ±  6%      -0.1        0.38 ±  6%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.63 ±  7%      -0.1        0.52 ±  6%  perf-profile.children.cycles-pp.__slab_free
      0.78 ±  7%      -0.1        0.67 ±  5%  perf-profile.children.cycles-pp.__pte_alloc
      0.61 ±  6%      -0.1        0.51 ±  6%  perf-profile.children.cycles-pp.update_curr
      0.82 ±  5%      -0.1        0.72 ±  4%  perf-profile.children.cycles-pp.pte_alloc_one
      0.49 ±  4%      -0.1        0.40 ±  5%  perf-profile.children.cycles-pp.wp_page_copy
      0.47 ±  2%      -0.1        0.38 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.50 ±  3%      -0.1        0.41 ±  7%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.41 ±  2%      -0.1        0.33 ±  4%  perf-profile.children.cycles-pp.__switch_to_asm
      0.52 ±  3%      -0.1        0.44 ±  5%  perf-profile.children.cycles-pp.wake_up_new_task
      0.41 ±  3%      -0.1        0.33 ±  8%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.44 ±  3%      -0.1        0.37 ±  7%  perf-profile.children.cycles-pp.sched_clock
      0.30 ±  7%      -0.1        0.22 ± 11%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.44 ±  3%      -0.1        0.36 ±  4%  perf-profile.children.cycles-pp.find_idlest_group
      0.31 ±  6%      -0.1        0.24 ±  8%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.42 ±  3%      -0.1        0.35 ±  6%  perf-profile.children.cycles-pp.native_sched_clock
      0.45 ±  5%      -0.1        0.38 ±  3%  perf-profile.children.cycles-pp.__switch_to
      0.36 ±  3%      -0.1        0.29 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.57 ±  4%      -0.1        0.51 ±  5%  perf-profile.children.cycles-pp.do_sys_openat2
      0.19 ± 51%      -0.1        0.13 ± 57%  perf-profile.children.cycles-pp.vm_normal_page
      0.68 ±  4%      -0.1        0.62 ±  5%  perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
      0.27 ±  3%      -0.1        0.20 ± 11%  perf-profile.children.cycles-pp.reweight_entity
      0.32 ±  4%      -0.1        0.26 ±  6%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.57 ±  4%      -0.1        0.51 ±  6%  perf-profile.children.cycles-pp.do_sys_open
      0.60 ±  2%      -0.1        0.54 ±  5%  perf-profile.children.cycles-pp.up_write
      0.32 ±  4%      -0.1        0.26 ±  9%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.31 ±  4%      -0.1        0.26 ±  3%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.29 ±  7%      -0.1        0.24 ±  8%  perf-profile.children.cycles-pp.unlock_page
      0.22 ±  4%      -0.1        0.17 ±  5%  perf-profile.children.cycles-pp.rmqueue
      0.23 ±  5%      -0.0        0.18 ± 15%  perf-profile.children.cycles-pp.pick_next_entity
      0.30 ±  6%      -0.0        0.26 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.27 ±  7%      -0.0        0.22 ±  8%  perf-profile.children.cycles-pp.__send_signal
      0.19 ±  2%      -0.0        0.14 ± 11%  perf-profile.children.cycles-pp.lock_page_memcg
      0.07 ± 10%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.put_cpu_partial
      0.20 ±  6%      -0.0        0.16 ±  8%  perf-profile.children.cycles-pp.__put_user_nocheck_4
      0.23 ±  6%      -0.0        0.20 ± 10%  perf-profile.children.cycles-pp.perf_tp_event
      0.20 ±  7%      -0.0        0.17 ±  9%  perf-profile.children.cycles-pp.remove_wait_queue
      0.16 ±  6%      -0.0        0.12 ±  7%  perf-profile.children.cycles-pp.copy_page
      0.21 ±  3%      -0.0        0.18 ±  9%  perf-profile.children.cycles-pp.sync_regs
      0.17 ±  5%      -0.0        0.14 ±  7%  perf-profile.children.cycles-pp.___perf_sw_event
      0.08 ±  7%      -0.0        0.05 ± 46%  perf-profile.children.cycles-pp.arch_dup_task_struct
      0.15 ±  4%      -0.0        0.12 ±  9%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.24 ±  5%      -0.0        0.21 ±  5%  perf-profile.children.cycles-pp.__task_pid_nr_ns
      0.16 ±  4%      -0.0        0.13 ±  5%  perf-profile.children.cycles-pp.place_entity
      0.14 ±  6%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.09 ±  5%      -0.0        0.07 ± 17%  perf-profile.children.cycles-pp.memcpy_erms
      0.15 ±  5%      -0.0        0.12 ±  7%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.17 ±  5%      -0.0        0.14 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.30 ±  3%      -0.0        0.27 ±  5%  perf-profile.children.cycles-pp.update_cfs_group
      0.16 ±  6%      -0.0        0.13 ± 11%  perf-profile.children.cycles-pp.__x64_sys_kill
      0.16 ±  4%      -0.0        0.14 ± 10%  perf-profile.children.cycles-pp.__cond_resched
      0.15 ±  7%      -0.0        0.12 ±  7%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.09 ± 12%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.08 ±  9%      -0.0        0.05 ±  7%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.11 ±  8%      -0.0        0.09 ±  8%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.07 ±  7%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.child_wait_callback
      0.06 ±  7%      -0.0        0.04 ± 44%  perf-profile.children.cycles-pp.perf_event_task
      0.10 ±  7%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__calc_delta
      0.17 ±  4%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.27 ±  3%      -0.0        0.24 ±  3%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.14 ±  7%      -0.0        0.12 ±  9%  perf-profile.children.cycles-pp.__might_sleep
      0.08 ±  4%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.__vma_rb_erase
      0.17 ±  5%      +0.0        0.20 ±  6%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.12 ±  6%      +0.0        0.16 ±  6%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge
      0.16 ±  6%      +0.0        0.20 ±  6%  perf-profile.children.cycles-pp.page_counter_cancel
      0.23 ±  5%      +0.0        0.27 ±  5%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      0.17 ±  6%      +0.0        0.21 ± 13%  perf-profile.children.cycles-pp.scheduler_tick
      0.26 ±  2%      +0.1        0.31 ±  6%  perf-profile.children.cycles-pp.find_get_task_by_vpid
      0.42 ±  3%      +0.1        0.48        perf-profile.children.cycles-pp.page_counter_try_charge
      0.29 ±  6%      +0.1        0.36 ±  7%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.65 ±  3%      +0.1        0.73 ±  5%  perf-profile.children.cycles-pp.down_write
      0.18 ±  6%      +0.1        0.26 ±  3%  perf-profile.children.cycles-pp.drain_obj_stock
      1.19 ±  4%      +0.1        1.29 ±  5%  perf-profile.children.cycles-pp.release_pages
      0.32 ±  7%      +0.1        0.42 ±  7%  perf-profile.children.cycles-pp.anon_vma_interval_tree_remove
      0.34 ±  5%      +0.1        0.45 ±  3%  perf-profile.children.cycles-pp.refill_obj_stock
      0.68            +0.1        0.80 ±  4%  perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      0.79 ±  3%      +0.1        0.91 ±  6%  perf-profile.children.cycles-pp.ptrace_check_attach
      0.08 ± 14%      +0.1        0.22 ± 10%  perf-profile.children.cycles-pp.osq_lock
      0.54 ±  6%      +0.2        0.71 ±  2%  perf-profile.children.cycles-pp.unlink_file_vma
      1.88            +0.2        2.07 ±  5%  perf-profile.children.cycles-pp.anon_vma_clone
      0.21 ±  9%      +0.2        0.41 ±  6%  perf-profile.children.cycles-pp.cgroup_enter_frozen
      0.28 ±  9%      +0.2        0.50 ±  8%  perf-profile.children.cycles-pp.cgroup_leave_frozen
      1.02 ±  6%      +0.2        1.25 ±  7%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      0.18 ±  9%      +0.3        0.43 ±  6%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      2.66 ±  2%      +0.3        2.92 ±  5%  perf-profile.children.cycles-pp.anon_vma_fork
      1.88 ±  3%      +0.3        2.14 ±  5%  perf-profile.children.cycles-pp.unlink_anon_vmas
      1.20 ±  3%      +0.3        1.52 ±  6%  perf-profile.children.cycles-pp.next_uptodate_page
      2.65 ±  3%      +0.4        3.06 ±  4%  perf-profile.children.cycles-pp.free_pgtables
      0.85 ±  5%      +0.4        1.27 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.34 ± 10%      +0.5        0.83 ±  5%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      1.27 ±  8%      +0.6        1.86 ±  7%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     33.21 ±  2%      +5.1       38.34 ±  7%  perf-profile.children.cycles-pp.intel_idle
     18.34 ±  2%      -2.8       15.58 ±  5%  perf-profile.self.cycles-pp.poll_idle
      0.90 ± 15%      -0.2        0.68 ±  8%  perf-profile.self.cycles-pp.menu_select
      1.16 ±  2%      -0.1        1.02 ±  5%  perf-profile.self.cycles-pp.__schedule
      0.55 ±  7%      -0.1        0.42 ±  6%  perf-profile.self.cycles-pp.enqueue_entity
      0.63 ±  3%      -0.1        0.49 ±  6%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.73 ±  2%      -0.1        0.61 ±  6%  perf-profile.self.cycles-pp.update_rq_clock
      0.44 ±  7%      -0.1        0.33 ±  6%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.62 ±  7%      -0.1        0.51 ±  7%  perf-profile.self.cycles-pp.__slab_free
      0.53 ±  3%      -0.1        0.43 ±  6%  perf-profile.self.cycles-pp.update_load_avg
      0.54 ±  4%      -0.1        0.45 ±  6%  perf-profile.self.cycles-pp.do_idle
      0.46 ±  3%      -0.1        0.37 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.60 ±  5%      -0.1        0.51 ±  6%  perf-profile.self.cycles-pp.set_next_entity
      0.40 ±  2%      -0.1        0.33 ±  4%  perf-profile.self.cycles-pp.__switch_to_asm
      0.44 ±  4%      -0.1        0.37 ±  6%  perf-profile.self.cycles-pp.vm_area_dup
      0.43 ±  2%      -0.1        0.36 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.35 ±  2%      -0.1        0.28 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.40 ±  3%      -0.1        0.34 ±  7%  perf-profile.self.cycles-pp.native_sched_clock
      0.43 ±  5%      -0.1        0.36 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.28 ±  7%      -0.1        0.21 ± 12%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.38            -0.1        0.32 ±  5%  perf-profile.self.cycles-pp.find_idlest_group
      0.27 ±  3%      -0.1        0.20 ± 11%  perf-profile.self.cycles-pp.reweight_entity
      0.32 ±  4%      -0.1        0.26 ±  6%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.28 ±  8%      -0.0        0.23 ±  8%  perf-profile.self.cycles-pp.unlock_page
      0.29 ±  6%      -0.0        0.24 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.27 ± 10%      -0.0        0.23 ±  7%  perf-profile.self.cycles-pp.update_curr
      0.19 ±  8%      -0.0        0.15 ± 11%  perf-profile.self.cycles-pp.try_to_wake_up
      0.22 ±  5%      -0.0        0.18 ± 13%  perf-profile.self.cycles-pp.pick_next_entity
      0.18 ±  2%      -0.0        0.14 ±  9%  perf-profile.self.cycles-pp.lock_page_memcg
      0.18 ±  7%      -0.0        0.15 ±  7%  perf-profile.self.cycles-pp.dequeue_entity
      0.16 ±  6%      -0.0        0.12 ±  7%  perf-profile.self.cycles-pp.copy_page
      0.21 ±  4%      -0.0        0.17 ± 10%  perf-profile.self.cycles-pp.sync_regs
      0.14 ±  5%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.16 ±  4%      -0.0        0.13 ± 13%  perf-profile.self.cycles-pp.finish_task_switch
      0.09 ±  7%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.memcpy_erms
      0.15 ±  5%      -0.0        0.12 ±  5%  perf-profile.self.cycles-pp.place_entity
      0.12 ±  4%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.copy_page_range
      0.24 ±  4%      -0.0        0.21 ±  5%  perf-profile.self.cycles-pp.__task_pid_nr_ns
      0.11 ±  8%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.29 ±  3%      -0.0        0.26 ±  5%  perf-profile.self.cycles-pp.update_cfs_group
      0.15 ±  3%      -0.0        0.12 ± 10%  perf-profile.self.cycles-pp.unmap_page_range
      0.13 ±  5%      -0.0        0.10 ± 11%  perf-profile.self.cycles-pp.___perf_sw_event
      0.15 ±  7%      -0.0        0.13 ±  4%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.08 ± 12%      -0.0        0.05 ±  7%  perf-profile.self.cycles-pp.schedule
      0.13 ±  8%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.12 ±  6%      -0.0        0.10 ±  6%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.09 ±  7%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.__calc_delta
      0.07 ±  7%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.child_wait_callback
      0.13 ±  8%      -0.0        0.11 ±  6%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.12 ±  5%      -0.0        0.10 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.10 ± 10%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.08 ±  6%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.__vma_rb_erase
      0.09 ±  5%      -0.0        0.07 ±  9%  perf-profile.self.cycles-pp.__cond_resched
      0.10 ±  5%      +0.0        0.12 ±  7%  perf-profile.self.cycles-pp.ptrace_check_attach
      0.06 ±  6%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.15 ±  6%      +0.0        0.18 ±  3%  perf-profile.self.cycles-pp.refill_obj_stock
      0.07 ± 15%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.drain_obj_stock
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.propagate_protected_usage
      0.39 ±  2%      +0.1        0.44        perf-profile.self.cycles-pp.page_counter_try_charge
      0.15 ±  5%      +0.1        0.21 ±  6%  perf-profile.self.cycles-pp.find_get_task_by_vpid
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.cgroup_enter_frozen
      0.07 ± 14%      +0.1        0.13 ±  9%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.29 ±  7%      +0.1        0.36 ±  6%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.30 ±  5%      +0.1        0.38 ±  6%  perf-profile.self.cycles-pp.ptrace_stop
      0.16 ±  6%      +0.1        0.24 ±  5%  perf-profile.self.cycles-pp.wait_task_inactive
      0.68 ±  3%      +0.1        0.77 ±  5%  perf-profile.self.cycles-pp.do_wait
      0.32 ±  6%      +0.1        0.41 ±  7%  perf-profile.self.cycles-pp.anon_vma_interval_tree_remove
      0.52 ±  2%      +0.1        0.61 ±  6%  perf-profile.self.cycles-pp.down_write
      0.93 ±  5%      +0.1        1.03 ±  5%  perf-profile.self.cycles-pp.release_pages
      0.61 ±  5%      +0.1        0.72 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.67            +0.1        0.79 ±  4%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      0.08 ± 14%      +0.1        0.22 ±  9%  perf-profile.self.cycles-pp.osq_lock
      0.35 ± 19%      +0.2        0.53 ±  5%  perf-profile.self.cycles-pp.ktime_get
      0.17 ± 10%      +0.3        0.43 ±  6%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      1.19 ±  3%      +0.3        1.51 ±  6%  perf-profile.self.cycles-pp.next_uptodate_page
      1.26 ±  8%      +0.6        1.85 ±  7%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     33.21 ±  2%      +5.1       38.34 ±  7%  perf-profile.self.cycles-pp.intel_idle


                                                                                
                             stress-ng.time.user_time                           
                                                                                
  250 +---------------------------------------------------------------------+   
      |          +. .+      .++.   +.                    .+.         +.     |   
      |.++.+.++.+  +  +.+.++    +.+  +.++.++.+.++.+.++.++   ++.+.++.+  +.++.|   
  200 |-OO    O OO O OO O OO OO O OO O OO OO O OO O OO OO                   |   
      |                                                                     |   
      |                                                                     |   
  150 |-+                                                                   |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
   50 |-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.time.system_time                          
                                                                                
  400 +---------------------------------------------------------------------+   
      |.++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.|   
  350 |-+                                                                   |   
  300 |-+                                                                   |   
      |                                                                     |   
  250 |-+                                                                   |   
      |                                                                     |   
  200 |-+                                                                   |   
      |                                                                     |   
  150 |-+                                                                   |   
  100 |-+                                                                   |   
      |                                                                     |   
   50 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          stress-ng.time.major_page_faults                      
                                                                                
  45000 +-------------------------------------------------------------------+   
        |        O                                                          |   
  40000 |-+O         O                                 O                    |   
  35000 |-+                  O   O         O                                |   
        |             O    O    O     O  O      O     O  O                  |   
  30000 |-+        O          O     O   O     O     O                       |   
  25000 |-+     O       O          O        O    O O                        |   
        |                O                                                  |   
  20000 |-+                                                                 |   
  15000 |-+                                                                 |   
        |                                                                   |   
  10000 |.++. +. +.       .+.++.     .+.++.++. .++. +.+      .++.+.+   +.++.|   
   5000 |-+  +  +  +.++.++      ++.++         +    +   +.+.++       +.+     |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      stress-ng.time.voluntary_context_switches                 
                                                                                
  7e+07 +-------------------------------------------------------------------+   
        |.++.++.+ .+.++.++.+.++.++.++.+.++.++.+.++.++.++.+. +.++.+. +.++.+ .|   
  6e+07 |-+      +                                         +       +      + |   
        | OO    OO O OO OO O OO OO OO O OO OO O OO OO OO O                  |   
  5e+07 |-+                                                                 |   
        |                                                                   |   
  4e+07 |-+                                                                 |   
        |                                                                   |   
  3e+07 |-+                                                                 |   
        |                                                                   |   
  2e+07 |-+                                                                 |   
        |                                                                   |   
  1e+07 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                stress-ng.vm-segv.ops                           
                                                                                
  900000 +------------------------------------------------------------------+   
         |.++.++.++.++.+.++.++.++.++.+.++.++.++.+.++.++.++. +.+.++.++.++.++.|   
  800000 |-+                                               +                |   
  700000 |-OO    OO OO O OO OO OO OO O OO OO OO O OO OO OO                  |   
         |                                                                  |   
  600000 |-+                                                                |   
  500000 |-+                                                                |   
         |                                                                  |   
  400000 |-+                                                                |   
  300000 |-+                                                                |   
         |                                                                  |   
  200000 |-+                                                                |   
  100000 |-+                                                                |   
         |                                                                  |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.vm-segv.ops_per_sec                       
                                                                                
  16000 +-------------------------------------------------------------------+   
        |                                                                   |   
  14000 |.++.++.++.+.++.++.+.++.++.++.+.++.++.+.++.++.++.+.++.++.+.++.++.++.|   
  12000 |-+        O     O         OO O     O O O                           |   
        | OO    OO   OO O  O OO OO      OO O     O OO OO O                  |   
  10000 |-+                                                                 |   
        |                                                                   |   
   8000 |-+                                                                 |   
        |                                                                   |   
   6000 |-+                                                                 |   
   4000 |-+                                                                 |   
        |                                                                   |   
   2000 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.12.0-rc2-00040-g38ac256d1c3e" of type "text/plain" (172853 bytes)

View attachment "job-script" of type "text/plain" (8276 bytes)

View attachment "job.yaml" of type "text/plain" (5694 bytes)

View attachment "reproduce" of type "text/plain" (535 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ