lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20201002083311.GK393@shao2-debian>
Date:   Fri, 2 Oct 2020 16:33:11 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Boqun Feng <boqun.feng@...il.com>,
        linux-kernel@...r.kernel.org,
        Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
        Will Deacon <will@...nel.org>,
        "Paul E . McKenney" <paulmck@...nel.org>,
        Nicholas Piggin <npiggin@...il.com>,
        Andy Lutomirski <luto@...capital.net>,
        Andrew Morton <akpm@...ux-foundation.org>,
        0day robot <lkp@...el.com>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com,
        aubrey.li@...ux.intel.com, yu.c.chen@...el.com
Subject: [sched] bdfcae1140: will-it-scale.per_thread_ops -37.0% regression

Greeting,

FYI, we noticed a -37.0% regression of will-it-scale.per_thread_ops due to commit:


commit: bdfcae11403e5099769a7c8dc3262e3c4193edef ("[RFC PATCH 2/3] sched: membarrier: cover kthread_use_mm (v3)")
url: https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/Membarrier-updates/20200925-012549
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 848785df48835eefebe0c4eb5da7690690b0a8b7

in testcase: will-it-scale
on test machine: 104 threads Skylake with 192G memory
with following parameters:

	nr_task: 50%
	mode: thread
	test: context_switch1
	cpufreq_governor: performance
	ucode: 0x2006906

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/thread/50%/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/context_switch1/will-it-scale/0x2006906

commit: 
  c2daff748f ("sched: fix exit_mm vs membarrier (v3)")
  bdfcae1140 ("sched: membarrier: cover kthread_use_mm (v3)")

c2daff748f0ea954 bdfcae11403e5099769a7c8dc32 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    161714 ±  2%     -37.0%     101819 ±  2%  will-it-scale.per_thread_ops
   8409170 ±  2%     -37.0%    5294646 ±  2%  will-it-scale.workload
     56783 ± 89%     -81.2%      10674 ± 58%  numa-vmstat.node0.numa_other
    153233 ± 33%     +30.3%     199692 ±  3%  numa-vmstat.node1.numa_other
      1099 ±  6%      -5.4%       1039 ±  4%  slabinfo.file_lock_cache.active_objs
      1099 ±  6%      -5.4%       1039 ±  4%  slabinfo.file_lock_cache.num_objs
      7082            -8.4%       6490 ±  4%  slabinfo.kmalloc-rcl-64.active_objs
      7082            -8.4%       6490 ±  4%  slabinfo.kmalloc-rcl-64.num_objs
     20.00           +55.0%      31.00 ±  5%  vmstat.cpu.id
     69.75           -12.2%      61.25 ±  2%  vmstat.cpu.sy
  16629437 ±  2%     -37.1%   10466452 ±  2%  vmstat.system.cs
    335444           -39.7%     202300 ± 20%  vmstat.system.in
     20.06           +11.3       31.32 ±  4%  mpstat.cpu.all.idle%
      1.29            -0.2        1.07        mpstat.cpu.all.irq%
      0.13            -0.0        0.11 ±  2%  mpstat.cpu.all.soft%
     69.11            -8.4       60.67 ±  2%  mpstat.cpu.all.sys%
      9.40            -2.6        6.83 ±  3%  mpstat.cpu.all.usr%
   2556574 ± 18%  +1.6e+05%  4.162e+09 ± 18%  cpuidle.C1.time
    988160 ± 21%  +94377.8%  9.336e+08 ± 12%  cpuidle.C1.usage
 1.167e+08 ± 22%     -69.3%   35782884 ± 70%  cpuidle.C1E.time
  26016930 ± 90%    +318.9%   1.09e+08 ± 25%  cpuidle.C6.time
     73042 ± 62%    +140.7%     175848 ±  4%  cpuidle.C6.usage
 2.507e+09 ±  2%     -73.8%  6.571e+08 ± 23%  cpuidle.POLL.usage
     16809            -4.4%      16075        proc-vmstat.nr_active_anon
     20021            -3.9%      19247        proc-vmstat.nr_shmem
     16809            -4.4%      16075        proc-vmstat.nr_zone_active_anon
    908.50 ±104%    +311.8%       3741 ± 54%  proc-vmstat.numa_pages_migrated
     75555 ±  8%     +18.9%      89863 ± 11%  proc-vmstat.numa_pte_updates
     25016            -5.2%      23727        proc-vmstat.pgactivate
    908.50 ±104%    +311.8%       3741 ± 54%  proc-vmstat.pgmigrate_success
    436827 ±  9%     -42.9%     249596 ± 14%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   1339696 ±  4%     -26.6%     983616 ±  6%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     14427 ±  3%     -11.8%      12725 ±  4%  sched_debug.cfs_rq:/.load.avg
      7845 ±  6%     +16.9%       9167        sched_debug.cfs_rq:/.load.stddev
    436827 ±  9%     -42.9%     249596 ± 14%  sched_debug.cfs_rq:/.max_vruntime.avg
   1339696 ±  4%     -26.6%     983616 ±  6%  sched_debug.cfs_rq:/.max_vruntime.stddev
      0.74 ±  2%     -16.4%       0.62 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
      0.34 ±  6%     +25.6%       0.42 ±  2%  sched_debug.cfs_rq:/.nr_running.stddev
    315.71 ±  2%     -11.8%     278.53 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      3086 ±  5%    +101.4%       6218 ± 35%  sched_debug.cpu.avg_idle.min
      5.18 ±  2%     +10.9%       5.75 ±  3%  sched_debug.cpu.clock.stddev
  24097711 ±  2%     -37.0%   15171198 ±  2%  sched_debug.cpu.nr_switches.avg
  27008905 ±  4%     -33.8%   17870323 ±  4%  sched_debug.cpu.nr_switches.max
  20920055 ±  6%     -38.3%   12906743 ±  4%  sched_debug.cpu.nr_switches.min
 1.733e+10 ±  2%     -36.0%   1.11e+10 ±  2%  perf-stat.i.branch-instructions
 2.838e+08           -36.0%  1.817e+08 ±  2%  perf-stat.i.branch-misses
  16740394 ±  2%     -37.1%   10534103 ±  2%  perf-stat.i.context-switches
      3.47 ±  2%     +48.2%       5.15        perf-stat.i.cpi
 2.882e+11            -5.5%  2.724e+11        perf-stat.i.cpu-cycles
    659.32 ±  3%     +26.2%     832.24 ±  3%  perf-stat.i.cpu-migrations
  16889909 ±  2%     -36.8%   10671898 ±  2%  perf-stat.i.dTLB-load-misses
 2.424e+10 ±  2%     -36.5%  1.539e+10 ±  2%  perf-stat.i.dTLB-loads
      0.00 ±  3%      +0.0        0.00 ± 13%  perf-stat.i.dTLB-store-miss-rate%
     27601           -12.7%      24088 ± 10%  perf-stat.i.dTLB-store-misses
 1.533e+10 ±  2%     -36.9%  9.667e+09 ±  2%  perf-stat.i.dTLB-stores
     92.12           -75.8       16.35 ± 10%  perf-stat.i.iTLB-load-miss-rate%
  17765547           -26.0%   13139994 ±  2%  perf-stat.i.iTLB-load-misses
   1483996 ± 10%   +4486.1%   68058032 ± 10%  perf-stat.i.iTLB-loads
 8.286e+10 ±  2%     -36.2%  5.287e+10 ±  2%  perf-stat.i.instructions
      4739           -13.8%       4086        perf-stat.i.instructions-per-iTLB-miss
      0.29           -32.2%       0.20        perf-stat.i.ipc
      0.53           +37.7%       0.73 ± 27%  perf-stat.i.major-faults
      2.77            -5.5%       2.62        perf-stat.i.metric.GHz
    548.46 ±  2%     -36.2%     349.65 ±  2%  perf-stat.i.metric.M/sec
   1818026 ±  2%     -12.4%    1591936 ±  3%  perf-stat.i.node-store-misses
      7101 ±  5%     +21.4%       8618 ± 11%  perf-stat.i.node-stores
      3.48 ±  2%     +48.1%       5.15        perf-stat.overall.cpi
      0.00            +0.0        0.00 ± 10%  perf-stat.overall.dTLB-store-miss-rate%
     92.30           -76.0       16.34 ± 10%  perf-stat.overall.iTLB-load-miss-rate%
      4663           -13.7%       4024        perf-stat.overall.instructions-per-iTLB-miss
      0.29 ±  2%     -32.5%       0.19        perf-stat.overall.ipc
   2962542            +1.4%    3005468        perf-stat.overall.path-length
 1.727e+10 ±  2%     -35.9%  1.106e+10 ±  2%  perf-stat.ps.branch-instructions
 2.828e+08           -36.0%  1.811e+08 ±  2%  perf-stat.ps.branch-misses
  16681900 ±  2%     -37.1%   10499012 ±  2%  perf-stat.ps.context-switches
 2.872e+11            -5.5%  2.715e+11        perf-stat.ps.cpu-cycles
    657.84 ±  3%     +26.1%     829.45 ±  3%  perf-stat.ps.cpu-migrations
  16831485 ±  2%     -36.8%   10637252 ±  2%  perf-stat.ps.dTLB-load-misses
 2.416e+10 ±  2%     -36.5%  1.534e+10 ±  2%  perf-stat.ps.dTLB-loads
     27550           -12.7%      24040 ± 10%  perf-stat.ps.dTLB-store-misses
 1.527e+10 ±  2%     -36.9%  9.636e+09 ±  2%  perf-stat.ps.dTLB-stores
  17705408           -26.0%   13096731 ±  2%  perf-stat.ps.iTLB-load-misses
   1479549 ± 10%   +4483.7%   67818252 ± 10%  perf-stat.ps.iTLB-loads
 8.257e+10 ±  2%     -36.2%  5.269e+10 ±  2%  perf-stat.ps.instructions
      0.53           +37.4%       0.72 ± 27%  perf-stat.ps.major-faults
   1811676 ±  2%     -12.4%    1586474 ±  3%  perf-stat.ps.node-store-misses
      7102 ±  5%     +21.3%       8615 ± 11%  perf-stat.ps.node-stores
 2.491e+13 ±  2%     -36.1%  1.591e+13 ±  2%  perf-stat.total.instructions
     27105 ± 11%     +20.8%      32744 ±  8%  softirqs.CPU1.RCU
     26920 ± 12%     +19.1%      32067 ±  6%  softirqs.CPU10.RCU
     26751 ± 12%     +17.4%      31413 ±  6%  softirqs.CPU100.RCU
     26471 ± 12%     +17.3%      31045 ±  7%  softirqs.CPU101.RCU
     26980 ± 12%     +15.3%      31101 ±  7%  softirqs.CPU102.RCU
     27057 ± 13%     +17.1%      31695 ±  8%  softirqs.CPU103.RCU
     26830 ± 11%     +20.6%      32356 ±  7%  softirqs.CPU11.RCU
     26679 ± 11%     +27.6%      34039 ± 16%  softirqs.CPU12.RCU
     26651 ± 11%     +19.3%      31806 ±  7%  softirqs.CPU13.RCU
     26549 ± 11%     +20.7%      32044 ±  6%  softirqs.CPU14.RCU
     28907 ±  9%     +18.7%      34299 ±  6%  softirqs.CPU15.RCU
     28775 ± 11%     +18.6%      34116 ±  7%  softirqs.CPU16.RCU
     28749 ± 13%     +17.5%      33768 ±  7%  softirqs.CPU17.RCU
     28518 ± 10%     +17.8%      33600 ±  7%  softirqs.CPU18.RCU
     28382 ± 10%     +18.7%      33700 ±  7%  softirqs.CPU19.RCU
     28682 ± 10%     +17.7%      33759 ±  7%  softirqs.CPU20.RCU
     28510 ± 10%     +20.3%      34310 ±  7%  softirqs.CPU21.RCU
     28609 ± 10%     +18.4%      33874 ±  6%  softirqs.CPU22.RCU
     28591 ± 11%     +19.0%      34025 ±  7%  softirqs.CPU23.RCU
     27572 ±  7%     +21.3%      33449 ±  7%  softirqs.CPU24.RCU
     28439 ± 11%     +18.3%      33639 ±  8%  softirqs.CPU25.RCU
     27063 ± 10%     +18.6%      32089 ±  8%  softirqs.CPU27.RCU
     27056 ± 11%     +16.6%      31547 ±  7%  softirqs.CPU28.RCU
     27183 ± 10%     +18.0%      32078 ±  8%  softirqs.CPU3.RCU
     27508 ± 11%     +16.6%      32073 ±  7%  softirqs.CPU30.RCU
     27613 ± 12%     +15.1%      31776 ±  9%  softirqs.CPU32.RCU
     27570 ± 13%     +16.0%      31990 ±  7%  softirqs.CPU33.RCU
     27317 ± 11%     +17.6%      32121 ±  8%  softirqs.CPU34.RCU
     27119 ± 11%     +18.0%      32010 ±  8%  softirqs.CPU35.RCU
     27230 ± 11%     +18.6%      32306 ±  7%  softirqs.CPU37.RCU
     27458 ± 12%     +16.4%      31963 ±  8%  softirqs.CPU38.RCU
     27380 ± 11%     +18.4%      32431 ±  8%  softirqs.CPU39.RCU
     27095 ± 11%     +18.8%      32190 ±  7%  softirqs.CPU4.RCU
     27520 ± 12%     +19.8%      32962 ±  8%  softirqs.CPU40.RCU
     27596 ± 12%     +16.0%      32021 ±  8%  softirqs.CPU41.RCU
     27542 ± 11%     +17.0%      32218 ±  7%  softirqs.CPU42.RCU
     27511 ± 12%     +19.5%      32878 ±  6%  softirqs.CPU45.RCU
     27653 ± 13%     +15.7%      31998 ±  8%  softirqs.CPU46.RCU
     27508 ± 12%     +16.9%      32156 ±  7%  softirqs.CPU47.RCU
     27521 ± 12%     +17.9%      32444 ±  6%  softirqs.CPU48.RCU
     27251 ± 10%     +17.3%      31971 ±  7%  softirqs.CPU5.RCU
     27739 ± 12%     +15.9%      32140 ±  8%  softirqs.CPU51.RCU
     27818 ± 10%     +17.1%      32564 ±  7%  softirqs.CPU52.RCU
     28329 ± 11%     +19.7%      33910 ±  8%  softirqs.CPU53.RCU
     27982 ± 12%     +19.0%      33291 ±  6%  softirqs.CPU54.RCU
     28193 ± 11%     +18.6%      33433 ±  7%  softirqs.CPU55.RCU
     28119 ± 11%     +19.3%      33544 ±  7%  softirqs.CPU56.RCU
     28493 ± 10%     +16.5%      33182 ±  7%  softirqs.CPU57.RCU
     28030 ± 11%     +17.0%      32782 ±  6%  softirqs.CPU58.RCU
     27924 ± 11%     +19.8%      33461 ±  7%  softirqs.CPU59.RCU
     26752 ± 11%     +17.8%      31504 ±  6%  softirqs.CPU6.RCU
     26528 ± 13%     +19.8%      31789 ±  6%  softirqs.CPU60.RCU
     26420 ± 12%     +20.2%      31760 ±  8%  softirqs.CPU61.RCU
     27217 ± 12%     +16.3%      31656 ±  7%  softirqs.CPU62.RCU
     26954 ± 11%     +17.8%      31765 ±  7%  softirqs.CPU63.RCU
     26995 ± 11%     +18.8%      32066 ±  7%  softirqs.CPU64.RCU
     27085 ± 11%     +17.6%      31848 ±  7%  softirqs.CPU65.RCU
     26962 ± 10%     +18.9%      32056 ±  7%  softirqs.CPU66.RCU
     27161 ± 10%     +17.4%      31895 ±  7%  softirqs.CPU67.RCU
     27133 ± 11%     +17.9%      31994 ±  6%  softirqs.CPU68.RCU
     27356 ± 13%     +17.3%      32078 ±  7%  softirqs.CPU69.RCU
     27438 ± 10%     +16.7%      32025 ±  5%  softirqs.CPU7.RCU
     26932 ± 11%     +18.5%      31904 ±  6%  softirqs.CPU70.RCU
     26961 ± 11%     +17.9%      31784 ±  7%  softirqs.CPU71.RCU
     27060 ± 11%     +17.7%      31850 ±  6%  softirqs.CPU72.RCU
     26950 ± 11%     +18.6%      31968 ±  7%  softirqs.CPU73.RCU
     27074 ± 12%     +17.7%      31871 ±  7%  softirqs.CPU74.RCU
     26269 ± 10%     +18.6%      31163 ±  7%  softirqs.CPU75.RCU
     25904 ± 10%     +20.3%      31161 ±  7%  softirqs.CPU76.RCU
     26007 ± 11%     +28.0%      33287 ± 10%  softirqs.CPU77.RCU
     27643 ± 12%     +17.5%      32490 ±  9%  softirqs.CPU79.RCU
     26424 ± 12%     +20.5%      31852 ±  6%  softirqs.CPU8.RCU
     27861 ± 11%     +17.9%      32839 ±  8%  softirqs.CPU81.RCU
     28220 ± 10%     +14.7%      32355 ±  8%  softirqs.CPU82.RCU
     27666 ± 11%     +15.7%      32006 ±  9%  softirqs.CPU84.RCU
     27277 ± 11%     +17.6%      32086 ±  8%  softirqs.CPU85.RCU
     27530 ± 11%     +17.9%      32453 ±  8%  softirqs.CPU87.RCU
     27395 ± 11%     +17.9%      32286 ±  7%  softirqs.CPU89.RCU
     26430 ± 11%     +18.7%      31362 ±  9%  softirqs.CPU90.RCU
     26569 ± 12%     +17.4%      31202 ±  8%  softirqs.CPU91.RCU
     26618 ± 12%     +18.2%      31472 ±  8%  softirqs.CPU92.RCU
     26591 ± 10%     +17.3%      31180 ±  8%  softirqs.CPU93.RCU
     26376 ± 12%     +18.3%      31204 ±  8%  softirqs.CPU94.RCU
     26339 ± 12%     +17.1%      30837 ±  8%  softirqs.CPU96.RCU
     26506 ± 12%     +17.2%      31057 ±  7%  softirqs.CPU97.RCU
     26608 ± 12%     +16.6%      31034 ±  8%  softirqs.CPU98.RCU
   2853153 ± 11%     +17.8%    3360022 ±  7%  softirqs.RCU
     67.56           -19.1       48.43 ±  4%  perf-profile.calltrace.cycles-pp.__libc_read
     64.22           -17.9       46.31 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
     62.09           -17.3       44.74 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     61.78           -17.2       44.56 ±  4%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     61.17           -16.9       44.23 ±  4%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     60.19           -16.6       43.63 ±  4%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     59.84           -16.5       43.39 ±  4%  perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
     54.14           -12.8       41.30 ±  4%  perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
     54.07           -12.8       41.26 ±  4%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.pipe_read.new_sync_read.vfs_read
     25.92            -7.7       18.25 ±  2%  perf-profile.calltrace.cycles-pp.finish_task_switch.__sched_text_start.schedule.pipe_read.new_sync_read
     11.52 ±  2%      -3.0        8.49        perf-profile.calltrace.cycles-pp.__libc_write
      7.82 ±  2%      -2.7        5.14 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      7.58 ±  2%      -2.6        5.01 ±  2%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      6.98 ±  3%      -2.3        4.66 ±  2%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      9.47 ±  2%      -2.3        7.16        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
      6.13 ±  3%      -1.9        4.22 ±  2%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.88 ±  2%      -1.8        4.08 ±  2%  perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      1.49 ± 20%      -1.2        0.28 ±100%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read
      3.65            -1.2        2.46 ±  6%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      4.19 ±  3%      -1.2        3.04 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
      3.87 ±  3%      -1.0        2.83 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
      3.79 ±  3%      -1.0        2.78 ±  3%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
      1.29            -1.0        0.29 ±100%  perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      3.59 ±  3%      -0.9        2.67 ±  3%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      1.93            -0.5        1.41 ±  3%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read
      2.02 ±  3%      -0.5        1.50 ±  6%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.95 ±  4%      -0.5        1.47 ±  6%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      1.31 ±  3%      -0.4        0.88        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      0.98 ±  2%      -0.4        0.60 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__libc_read
      1.56 ±  4%      -0.4        1.20 ±  6%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.97 ±  2%      -0.3        0.63 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__libc_write
      0.85 ±  2%      -0.3        0.54 ±  3%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_read
      0.84            -0.3        0.55 ±  2%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_write
      2.24 ±  7%      -0.3        1.96 ±  7%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.pipe_read.new_sync_read
      0.82 ±  4%      -0.2        0.59 ±  3%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle
      0.72            +0.3        0.98 ±  8%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      1.57            +0.4        1.97 ±  3%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_write
      0.00            +0.8        0.82 ±  5%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.00            +3.8        3.84 ± 39%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      7.42 ±  3%     +12.2       19.66 ±  7%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      7.39 ±  3%     +12.2       19.63 ±  7%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      3.68 ±  6%     +12.4       16.11 ±  7%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      3.57 ±  6%     +12.5       16.04 ±  7%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.00           +13.1       13.14 ± 18%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     19.21           +22.5       41.73 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     19.18           +22.5       41.72 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     19.05           +22.6       41.65 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     19.39           +22.8       42.18 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     73.73           -20.2       53.51 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     69.92           -20.0       49.91 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
     67.70           -19.2       48.52 ±  4%  perf-profile.children.cycles-pp.__libc_read
     61.82           -17.2       44.59 ±  4%  perf-profile.children.cycles-pp.ksys_read
     61.20           -17.0       44.24 ±  4%  perf-profile.children.cycles-pp.vfs_read
     60.20           -16.6       43.63 ±  4%  perf-profile.children.cycles-pp.new_sync_read
     59.88           -16.5       43.41 ±  4%  perf-profile.children.cycles-pp.pipe_read
     54.14           -12.8       41.30 ±  4%  perf-profile.children.cycles-pp.schedule
     26.24            -7.8       18.48 ±  2%  perf-profile.children.cycles-pp.finish_task_switch
     11.66 ±  2%      -3.1        8.57        perf-profile.children.cycles-pp.__libc_write
      7.61 ±  2%      -2.6        5.03 ±  2%  perf-profile.children.cycles-pp.ksys_write
      7.00 ±  3%      -2.3        4.67 ±  2%  perf-profile.children.cycles-pp.vfs_write
      6.13 ±  2%      -1.9        4.23 ±  2%  perf-profile.children.cycles-pp.new_sync_write
      5.90 ±  2%      -1.8        4.09 ±  2%  perf-profile.children.cycles-pp.pipe_write
      3.70            -1.2        2.49 ±  6%  perf-profile.children.cycles-pp.menu_select
      2.03 ± 14%      -1.2        0.83 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      4.20 ±  3%      -1.2        3.04 ±  2%  perf-profile.children.cycles-pp.__wake_up_common_lock
      3.88 ±  3%      -1.0        2.84 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      3.80 ±  3%      -1.0        2.79 ±  3%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.17 ±  5%      -1.0        0.18 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      1.51 ± 20%      -1.0        0.53 ±  7%  perf-profile.children.cycles-pp.prepare_to_wait_event
      3.61 ±  3%      -0.9        2.69 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      1.33            -0.8        0.52 ± 13%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      1.96 ±  2%      -0.7        1.24 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      1.95            -0.7        1.25 ±  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.29            -0.6        0.67 ± 12%  perf-profile.children.cycles-pp.ktime_get
      1.60 ±  3%      -0.6        1.05 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.68 ±  3%      -0.5        0.13 ±  6%  perf-profile.children.cycles-pp.finish_wait
      2.04 ±  3%      -0.5        1.51 ±  6%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.97 ±  4%      -0.5        1.47 ±  6%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.93            -0.5        0.43 ± 12%  perf-profile.children.cycles-pp.read_tsc
      0.92 ±  2%      -0.4        0.48 ±  3%  perf-profile.children.cycles-pp.security_file_permission
      1.63 ±  4%      -0.4        1.23 ±  6%  perf-profile.children.cycles-pp.enqueue_entity
      0.72 ±  3%      -0.4        0.36 ±  5%  perf-profile.children.cycles-pp.mutex_lock
      0.91            -0.4        0.55 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.77 ±  2%      -0.4        0.41 ±  5%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.71 ±  2%      -0.3        0.38 ±  5%  perf-profile.children.cycles-pp.sched_clock
      0.67 ±  2%      -0.3        0.35 ±  4%  perf-profile.children.cycles-pp.native_sched_clock
      0.60 ±  3%      -0.3        0.29 ±  5%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.64            -0.3        0.33 ± 14%  perf-profile.children.cycles-pp.rcu_idle_exit
      1.50 ±  4%      -0.3        1.21 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.54 ± 14%      -0.3        0.26 ±  4%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      2.25 ±  7%      -0.3        1.96 ±  7%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.74 ±  2%      -0.3        0.46 ±  4%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.40 ±  3%      -0.3        0.15 ±  3%  perf-profile.children.cycles-pp.mutex_unlock
      0.86 ±  3%      -0.2        0.62 ±  3%  perf-profile.children.cycles-pp.set_next_entity
      0.65 ±  2%      -0.2        0.41 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.65 ±  5%      -0.2        0.41 ±  5%  perf-profile.children.cycles-pp.__fdget_pos
      0.33 ±  7%      -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.__pthread_enable_asynccancel
      0.40 ±  2%      -0.2        0.17 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      1.93 ±  7%      -0.2        1.70 ±  7%  perf-profile.children.cycles-pp.dequeue_entity
      0.61 ±  5%      -0.2        0.39 ±  6%  perf-profile.children.cycles-pp.__fget_light
      0.43 ±  2%      -0.2        0.22 ±  4%  perf-profile.children.cycles-pp.__x86_retpoline_rax
      0.50 ±  2%      -0.2        0.29 ±  2%  perf-profile.children.cycles-pp.copy_page_from_iter
      0.47 ±  2%      -0.2        0.27 ±  4%  perf-profile.children.cycles-pp.fsnotify
      0.40 ±  3%      -0.2        0.21 ±  5%  perf-profile.children.cycles-pp.___might_sleep
      0.43            -0.2        0.25 ±  4%  perf-profile.children.cycles-pp.common_file_perm
      0.28 ±  5%      -0.2        0.10 ±  5%  perf-profile.children.cycles-pp.fput_many
      0.46 ±  7%      -0.2        0.30 ±  6%  perf-profile.children.cycles-pp.__fget_files
      0.24 ±  5%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.switch_fpu_return
      0.29 ±  2%      -0.1        0.15 ±  3%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.17 ± 30%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.40 ±  3%      -0.1        0.25 ±  2%  perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
      0.21 ±  4%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.48            -0.1        0.34 ± 10%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.40 ±  2%      -0.1        0.26        perf-profile.children.cycles-pp.update_rq_clock
      0.46            -0.1        0.33 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.24 ±  4%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.copyout
      0.24 ±  2%      -0.1        0.13 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      0.34 ±  4%      -0.1        0.23 ±  3%  perf-profile.children.cycles-pp.reweight_entity
      0.19 ±  3%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.__might_fault
      0.26 ±  4%      -0.1        0.16 ±  4%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.27 ±  3%      -0.1        0.17 ± 14%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.44 ±  7%      -0.1        0.34 ±  5%  perf-profile.children.cycles-pp.update_cfs_group
      0.40 ±  3%      -0.1        0.30 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.33 ±  2%      -0.1        0.23 ±  4%  perf-profile.children.cycles-pp.___perf_sw_event
      0.15 ±  2%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.put_prev_entity
      0.18 ±  2%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.newidle_balance
      0.23 ±  2%      -0.1        0.15 ±  5%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.19 ±  2%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.24            -0.1        0.15 ±  5%  perf-profile.children.cycles-pp.menu_reflect
      0.25 ±  4%      -0.1        0.17 ±  8%  perf-profile.children.cycles-pp.current_time
      0.13 ±  3%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.can_stop_idle_tick
      0.10 ±  4%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__x64_sys_write
      0.24 ±  2%      -0.1        0.17 ±  7%  perf-profile.children.cycles-pp.touch_atime
      0.14 ±  5%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.20 ±  2%      -0.1        0.13 ±  5%  perf-profile.children.cycles-pp.copyin
      0.18 ±  2%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.select_idle_sibling
      0.27 ±  3%      -0.1        0.21 ±  3%  perf-profile.children.cycles-pp.file_update_time
      0.12 ±  7%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.17 ±  4%      -0.1        0.11        perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.18 ±  4%      -0.1        0.12 ±  6%  perf-profile.children.cycles-pp.atime_needs_update
      0.14 ±  3%      -0.1        0.08        perf-profile.children.cycles-pp._cond_resched
      0.16 ±  2%      -0.1        0.10        perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.20 ±  2%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__pthread_disable_asynccancel
      0.12 ±  4%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.put_prev_task_idle
      0.09            -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.rb_erase
      0.08 ± 14%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.aa_file_perm
      0.48 ±  5%      -0.0        0.44 ±  5%  perf-profile.children.cycles-pp.asm_call_on_stack
      0.10 ±  7%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.06 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.available_idle_cpu
      0.11 ±  4%      -0.0        0.07        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.12 ±  5%      -0.0        0.08 ± 13%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.08 ±  8%      -0.0        0.05        perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.12 ±  5%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.08 ±  5%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.timestamp_truncate
      0.10 ±  4%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.update_min_vruntime
      0.08 ± 10%      -0.0        0.06 ± 17%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.08 ±  5%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_got_tick
      0.15 ±  4%      -0.0        0.13        perf-profile.children.cycles-pp.pick_next_entity
      0.07 ±  7%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.place_entity
      0.60            +0.0        0.65 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock
      0.26 ±  4%      +0.1        0.35 ±  3%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.23 ±  4%      +0.1        0.33 ±  4%  perf-profile.children.cycles-pp.check_preempt_curr
      0.13 ±  6%      +0.1        0.26 ±  6%  perf-profile.children.cycles-pp.resched_curr
      0.21 ±  3%      +0.1        0.35 ± 12%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.14 ±  8%      +0.2        0.32 ± 14%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_enter
      0.74            +0.3        1.00 ±  7%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.18 ± 22%      +0.3        0.45 ±  4%  perf-profile.children.cycles-pp.start_kernel
      0.46            +0.4        0.83 ±  5%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.35 ± 32%      +3.5        3.89 ± 39%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      7.50 ±  3%     +12.4       19.87 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter
      7.48 ±  3%     +12.4       19.86 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter_state
      3.72 ±  5%     +12.6       16.29 ±  7%  perf-profile.children.cycles-pp.schedule_idle
      0.00           +13.3       13.30 ± 18%  perf-profile.children.cycles-pp.intel_idle
     19.21           +22.5       41.73 ±  4%  perf-profile.children.cycles-pp.start_secondary
     19.39           +22.8       42.18 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64
     19.39           +22.8       42.18 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
     19.31           +22.8       42.15 ±  4%  perf-profile.children.cycles-pp.do_idle
     25.88            -7.7       18.14 ±  2%  perf-profile.self.cycles-pp.finish_task_switch
      2.01 ± 14%      -1.2        0.82 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.16 ±  5%      -1.0        0.18 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      1.65 ±  2%      -0.9        0.75 ±  6%  perf-profile.self.cycles-pp.menu_select
      1.94            -0.7        1.24 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.70 ±  2%      -0.6        1.08 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.90 ±  2%      -0.5        0.41 ± 13%  perf-profile.self.cycles-pp.read_tsc
      0.91            -0.4        0.54 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      0.65 ±  2%      -0.3        0.34 ±  4%  perf-profile.self.cycles-pp.native_sched_clock
      0.69            -0.3        0.43 ±  3%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.39 ±  2%      -0.2        0.14 ±  3%  perf-profile.self.cycles-pp.mutex_unlock
      0.43 ±  6%      -0.2        0.19 ±  6%  perf-profile.self.cycles-pp.mutex_lock
      0.33 ±  7%      -0.2        0.10 ±  7%  perf-profile.self.cycles-pp.__pthread_enable_asynccancel
      0.38            -0.2        0.17 ± 13%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.36            -0.2        0.17 ±  3%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.45 ±  3%      -0.2        0.25 ±  5%  perf-profile.self.cycles-pp.fsnotify
      0.37 ±  2%      -0.2        0.18 ±  3%  perf-profile.self.cycles-pp.__x86_retpoline_rax
      0.39 ±  4%      -0.2        0.21 ±  6%  perf-profile.self.cycles-pp.___might_sleep
      0.27 ±  5%      -0.2        0.09 ±  8%  perf-profile.self.cycles-pp.fput_many
      0.45 ±  8%      -0.2        0.29 ±  6%  perf-profile.self.cycles-pp.__fget_files
      0.44 ±  3%      -0.2        0.28 ±  2%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.26 ± 11%      -0.2        0.11 ±  4%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.36 ±  5%      -0.2        0.21 ±  7%  perf-profile.self.cycles-pp.common_file_perm
      0.39 ±  3%      -0.2        0.24 ±  3%  perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
      0.23 ±  4%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.switch_fpu_return
      0.17 ± 29%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.44            -0.1        0.31 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.41 ±  2%      -0.1        0.29 ±  3%  perf-profile.self.cycles-pp.pipe_read
      0.30 ±  2%      -0.1        0.18 ±  4%  perf-profile.self.cycles-pp.vfs_read
      0.19 ±  2%      -0.1        0.07 ± 14%  perf-profile.self.cycles-pp.tick_nohz_idle_exit
      0.20 ± 14%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.security_file_permission
      0.30            -0.1        0.18 ±  3%  perf-profile.self.cycles-pp.pipe_write
      0.42 ±  3%      -0.1        0.31 ±  7%  perf-profile.self.cycles-pp.enqueue_entity
      0.14 ±  5%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.update_ts_time_stats
      0.34 ±  3%      -0.1        0.23 ±  4%  perf-profile.self.cycles-pp.reweight_entity
      0.29 ±  6%      -0.1        0.18 ±  3%  perf-profile.self.cycles-pp.vfs_write
      0.22 ±  3%      -0.1        0.11 ±  3%  perf-profile.self.cycles-pp.__might_sleep
      0.34 ±  4%      -0.1        0.23 ±  9%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.26 ±  4%      -0.1        0.16 ±  4%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.32 ±  2%      -0.1        0.22 ± 17%  perf-profile.self.cycles-pp.ktime_get
      0.38 ±  2%      -0.1        0.28 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.24 ±  3%      -0.1        0.15 ±  2%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.13 ±  3%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.put_prev_entity
      0.42 ±  8%      -0.1        0.33 ±  6%  perf-profile.self.cycles-pp.update_cfs_group
      0.23 ±  2%      -0.1        0.14 ±  6%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.17 ±  3%      -0.1        0.08        perf-profile.self.cycles-pp.newidle_balance
      0.28 ±  2%      -0.1        0.20 ±  6%  perf-profile.self.cycles-pp.___perf_sw_event
      0.27            -0.1        0.19 ±  3%  perf-profile.self.cycles-pp.new_sync_read
      0.29 ±  2%      -0.1        0.21 ±  7%  perf-profile.self.cycles-pp.dequeue_entity
      0.18 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.13 ±  5%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.can_stop_idle_tick
      0.19            -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.ksys_read
      0.19 ±  2%      -0.1        0.12        perf-profile.self.cycles-pp.__libc_write
      0.28 ±  4%      -0.1        0.21 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.19 ±  4%      -0.1        0.12 ±  4%  perf-profile.self.cycles-pp.try_to_wake_up
      0.11 ±  8%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.15 ±  4%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.autoremove_wake_function
      0.15 ±  3%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.15            -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.13            -0.1        0.07        perf-profile.self.cycles-pp.copy_page_from_iter
      0.14 ±  5%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.put_prev_task_fair
      0.16 ±  4%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.12 ±  4%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.13 ±  3%      -0.1        0.07 ± 17%  perf-profile.self.cycles-pp.rcu_eqs_exit
      0.08 ±  8%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.15 ±  2%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.menu_reflect
      0.18 ±  4%      -0.0        0.13 ±  3%  perf-profile.self.cycles-pp.__pthread_disable_asynccancel
      0.15 ±  4%      -0.0        0.10        perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.20            -0.0        0.15 ±  2%  perf-profile.self.cycles-pp.__libc_read
      0.14 ± 11%      -0.0        0.09 ±  9%  perf-profile.self.cycles-pp.__fget_light
      0.07 ±  5%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.__wake_up_common
      0.12 ±  3%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.do_syscall_64
      0.14 ±  9%      -0.0        0.10 ± 11%  perf-profile.self.cycles-pp.new_sync_write
      0.13 ±  3%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.update_rq_clock
      0.11 ±  4%      -0.0        0.06        perf-profile.self.cycles-pp.cpu_startup_entry
      0.12 ±  8%      -0.0        0.08 ± 15%  perf-profile.self.cycles-pp.ksys_write
      0.11 ± 12%      -0.0        0.07 ± 16%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.11 ±  4%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.27 ±  8%      -0.0        0.23 ±  8%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.14 ±  3%      -0.0        0.10 ±  4%  perf-profile.self.cycles-pp.file_update_time
      0.11 ±  4%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.put_prev_task_idle
      0.06 ±  6%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.available_idle_cpu
      0.09            -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.schedule_idle
      0.07 ±  5%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.select_idle_sibling
      0.21 ±  5%      -0.0        0.18 ±  4%  perf-profile.self.cycles-pp.set_next_entity
      0.12 ±  4%      -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.12 ±  7%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.11 ±  4%      -0.0        0.08 ± 13%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.10 ±  5%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.10 ±  4%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.check_preempt_curr
      0.08 ± 10%      -0.0        0.06 ± 15%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.08 ±  8%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.current_time
      0.08 ±  6%      -0.0        0.05        perf-profile.self.cycles-pp.timestamp_truncate
      0.11 ±  3%      -0.0        0.09        perf-profile.self.cycles-pp.pick_next_entity
      0.08 ±  5%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.tick_nohz_idle_got_tick
      0.10 ±  4%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.update_min_vruntime
      0.05 ±  8%      +0.0        0.07 ±  6%  perf-profile.self.cycles-pp.__enqueue_entity
      0.06 ±  7%      +0.0        0.08        perf-profile.self.cycles-pp.place_entity
      0.58 ±  2%      +0.1        0.64 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      0.12 ±  6%      +0.1        0.26 ±  6%  perf-profile.self.cycles-pp.resched_curr
      0.13 ±  6%      +0.2        0.32 ± 14%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_enter
      0.34 ± 32%      +3.5        3.89 ± 39%  perf-profile.self.cycles-pp.switch_mm_irqs_off
     24.89            +3.6       28.49 ±  2%  perf-profile.self.cycles-pp.__sched_text_start
      0.00           +13.3       13.30 ± 18%  perf-profile.self.cycles-pp.intel_idle
    659.50 ±108%    +688.7%       5201 ± 71%  interrupts.40:PCI-MSI.67633155-edge.eth0-TxRx-2
  23689902 ±  8%     -91.3%    2063004 ±  5%  interrupts.CAL:Function_call_interrupts
    195589 ± 29%     -92.1%      15544 ± 34%  interrupts.CPU0.CAL:Function_call_interrupts
     91255 ± 35%     -74.1%      23629 ± 11%  interrupts.CPU0.RES:Rescheduling_interrupts
    920.75 ±  5%     -17.2%     762.25 ±  6%  interrupts.CPU0.TLB:TLB_shootdowns
    211340 ± 33%     -91.7%      17475 ± 22%  interrupts.CPU1.CAL:Function_call_interrupts
     86057 ± 41%     -75.1%      21460 ± 21%  interrupts.CPU1.RES:Rescheduling_interrupts
    975.25 ±  6%     -23.7%     743.75 ±  7%  interrupts.CPU1.TLB:TLB_shootdowns
    254911 ± 55%     -90.7%      23715 ± 43%  interrupts.CPU10.CAL:Function_call_interrupts
    105373 ± 30%     -80.7%      20353 ± 14%  interrupts.CPU10.RES:Rescheduling_interrupts
    955.25 ±  4%     -22.0%     744.75 ±  4%  interrupts.CPU10.TLB:TLB_shootdowns
    249461 ± 11%     -93.8%      15439 ± 13%  interrupts.CPU100.CAL:Function_call_interrupts
    204149 ± 27%     -91.0%      18319 ± 13%  interrupts.CPU100.RES:Rescheduling_interrupts
    930.00 ±  4%     -18.6%     756.75 ±  9%  interrupts.CPU100.TLB:TLB_shootdowns
    259504 ±  9%     -93.4%      17229 ± 17%  interrupts.CPU101.CAL:Function_call_interrupts
    204261 ± 24%     -91.2%      18064 ±  6%  interrupts.CPU101.RES:Rescheduling_interrupts
    954.50 ±  5%     -23.8%     727.00 ± 10%  interrupts.CPU101.TLB:TLB_shootdowns
    258375 ± 24%     -94.2%      15030 ± 14%  interrupts.CPU102.CAL:Function_call_interrupts
    182795 ± 16%     -90.0%      18358 ± 15%  interrupts.CPU102.RES:Rescheduling_interrupts
    916.00 ±  3%     -20.4%     729.50 ±  6%  interrupts.CPU102.TLB:TLB_shootdowns
    291365 ± 33%     -94.3%      16647 ± 26%  interrupts.CPU103.CAL:Function_call_interrupts
    173485 ± 10%     -89.9%      17467 ± 19%  interrupts.CPU103.RES:Rescheduling_interrupts
    958.50           -26.6%     703.25 ±  7%  interrupts.CPU103.TLB:TLB_shootdowns
    179350 ± 33%     -90.4%      17303 ± 21%  interrupts.CPU11.CAL:Function_call_interrupts
    114563 ± 29%     -82.5%      20100 ± 23%  interrupts.CPU11.RES:Rescheduling_interrupts
    956.75 ±  2%     -17.4%     790.75 ±  6%  interrupts.CPU11.TLB:TLB_shootdowns
    262516 ± 28%     -92.6%      19306 ± 29%  interrupts.CPU12.CAL:Function_call_interrupts
     89238 ± 24%     -76.0%      21412 ± 23%  interrupts.CPU12.RES:Rescheduling_interrupts
    938.75 ±  2%     -19.8%     752.50 ±  3%  interrupts.CPU12.TLB:TLB_shootdowns
    243783 ± 17%     -89.7%      25212 ± 55%  interrupts.CPU13.CAL:Function_call_interrupts
     96811 ± 24%     -78.4%      20953 ± 14%  interrupts.CPU13.RES:Rescheduling_interrupts
    965.00 ±  3%     -22.7%     746.00 ±  5%  interrupts.CPU13.TLB:TLB_shootdowns
    223957 ± 47%     -90.6%      21081 ± 51%  interrupts.CPU14.CAL:Function_call_interrupts
     92346 ± 16%     -76.2%      22002 ± 17%  interrupts.CPU14.RES:Rescheduling_interrupts
    916.75 ±  7%     -17.8%     753.75 ±  5%  interrupts.CPU14.TLB:TLB_shootdowns
    238289 ± 23%     -93.8%      14785 ± 36%  interrupts.CPU15.CAL:Function_call_interrupts
     94092 ± 23%     -80.7%      18181 ± 11%  interrupts.CPU15.RES:Rescheduling_interrupts
    975.75 ±  2%     -25.0%     732.25 ±  5%  interrupts.CPU15.TLB:TLB_shootdowns
    232503 ± 46%     -92.2%      18043 ± 37%  interrupts.CPU16.CAL:Function_call_interrupts
     92258 ± 39%     -77.3%      20901 ± 14%  interrupts.CPU16.RES:Rescheduling_interrupts
    970.75           -21.4%     762.75 ±  2%  interrupts.CPU16.TLB:TLB_shootdowns
    212775 ± 51%     -93.4%      13990 ± 15%  interrupts.CPU17.CAL:Function_call_interrupts
     89631 ± 50%     -77.5%      20194 ± 17%  interrupts.CPU17.RES:Rescheduling_interrupts
    980.50 ±  3%     -22.4%     761.25 ±  5%  interrupts.CPU17.TLB:TLB_shootdowns
    402401 ± 33%     -96.3%      15049 ±  8%  interrupts.CPU18.CAL:Function_call_interrupts
     94841 ± 34%     -78.2%      20633 ± 14%  interrupts.CPU18.RES:Rescheduling_interrupts
      1008 ±  3%     -24.8%     758.00 ±  5%  interrupts.CPU18.TLB:TLB_shootdowns
    214995 ± 21%     -93.5%      14000 ± 15%  interrupts.CPU19.CAL:Function_call_interrupts
     98312 ± 27%     -79.7%      19918 ± 13%  interrupts.CPU19.RES:Rescheduling_interrupts
    971.75 ±  2%     -23.5%     743.75 ±  4%  interrupts.CPU19.TLB:TLB_shootdowns
    215367 ± 43%     -93.4%      14320 ± 17%  interrupts.CPU2.CAL:Function_call_interrupts
     86227 ± 36%     -79.8%      17380 ± 17%  interrupts.CPU2.RES:Rescheduling_interrupts
      1001 ±  4%     -26.6%     734.75 ±  5%  interrupts.CPU2.TLB:TLB_shootdowns
    201442 ± 23%     -92.4%      15238 ± 27%  interrupts.CPU20.CAL:Function_call_interrupts
    107574 ± 25%     -81.5%      19865 ± 18%  interrupts.CPU20.RES:Rescheduling_interrupts
    968.75 ±  2%     -23.8%     738.50        interrupts.CPU20.TLB:TLB_shootdowns
    229077 ± 36%     -93.9%      13909 ± 18%  interrupts.CPU21.CAL:Function_call_interrupts
     84901 ± 27%     -78.5%      18250 ±  7%  interrupts.CPU21.RES:Rescheduling_interrupts
    990.50 ±  4%     -25.6%     737.25 ±  5%  interrupts.CPU21.TLB:TLB_shootdowns
    290748 ± 28%     -94.6%      15580 ± 30%  interrupts.CPU22.CAL:Function_call_interrupts
     95230 ± 22%     -78.4%      20559 ± 14%  interrupts.CPU22.RES:Rescheduling_interrupts
    958.75 ±  3%     -22.1%     746.75 ±  3%  interrupts.CPU22.TLB:TLB_shootdowns
    237541 ± 25%     -92.2%      18579 ± 48%  interrupts.CPU23.CAL:Function_call_interrupts
    102695 ± 43%     -80.5%      20071 ± 24%  interrupts.CPU23.RES:Rescheduling_interrupts
    971.25 ±  3%     -22.7%     751.00 ±  3%  interrupts.CPU23.TLB:TLB_shootdowns
    297244 ± 50%     -94.1%      17649 ± 24%  interrupts.CPU24.CAL:Function_call_interrupts
     94620 ± 32%     -80.1%      18865 ± 12%  interrupts.CPU24.RES:Rescheduling_interrupts
    978.75 ±  3%     -26.0%     724.50 ±  4%  interrupts.CPU24.TLB:TLB_shootdowns
    253790 ± 22%     -94.1%      15076 ± 38%  interrupts.CPU25.CAL:Function_call_interrupts
     94250 ± 20%     -79.8%      19007 ± 16%  interrupts.CPU25.RES:Rescheduling_interrupts
    992.50 ±  2%     -22.8%     765.75 ±  4%  interrupts.CPU25.TLB:TLB_shootdowns
    205152 ±  5%     -90.1%      20339 ±  7%  interrupts.CPU26.CAL:Function_call_interrupts
    170083 ± 10%     -87.8%      20741 ± 18%  interrupts.CPU26.RES:Rescheduling_interrupts
      1002 ±  4%     -23.7%     765.25 ±  6%  interrupts.CPU26.TLB:TLB_shootdowns
    230776 ±  9%     -90.7%      21504 ± 22%  interrupts.CPU27.CAL:Function_call_interrupts
    173305 ± 10%     -87.7%      21271 ± 23%  interrupts.CPU27.RES:Rescheduling_interrupts
    978.00 ±  3%     -19.8%     784.00 ±  8%  interrupts.CPU27.TLB:TLB_shootdowns
    262539 ± 42%     -92.4%      19830 ± 19%  interrupts.CPU28.CAL:Function_call_interrupts
    147713 ± 21%     -86.3%      20248 ± 18%  interrupts.CPU28.RES:Rescheduling_interrupts
      1013           -24.6%     764.75 ±  8%  interrupts.CPU28.TLB:TLB_shootdowns
    286525 ± 46%     -93.3%      19228 ± 20%  interrupts.CPU29.CAL:Function_call_interrupts
    165484 ± 35%     -87.5%      20618 ± 18%  interrupts.CPU29.RES:Rescheduling_interrupts
    981.50 ±  3%     -19.7%     787.75 ± 10%  interrupts.CPU29.TLB:TLB_shootdowns
    180881 ± 33%     -90.6%      16928 ± 20%  interrupts.CPU3.CAL:Function_call_interrupts
    103722 ± 25%     -83.2%      17437 ± 14%  interrupts.CPU3.RES:Rescheduling_interrupts
    974.25 ±  3%     -19.8%     781.25 ±  6%  interrupts.CPU3.TLB:TLB_shootdowns
    283317 ± 38%     -94.2%      16329 ± 13%  interrupts.CPU30.CAL:Function_call_interrupts
    148002 ± 14%     -87.7%      18221 ± 13%  interrupts.CPU30.RES:Rescheduling_interrupts
    973.00 ±  2%     -22.0%     758.75 ±  5%  interrupts.CPU30.TLB:TLB_shootdowns
    241331 ± 11%     -91.3%      21053 ± 25%  interrupts.CPU31.CAL:Function_call_interrupts
    180308 ±  6%     -88.8%      20187 ± 10%  interrupts.CPU31.RES:Rescheduling_interrupts
    996.25 ±  3%     -22.1%     775.75 ±  9%  interrupts.CPU31.TLB:TLB_shootdowns
    659.50 ±108%    +688.7%       5201 ± 71%  interrupts.CPU32.40:PCI-MSI.67633155-edge.eth0-TxRx-2
    244508 ± 18%     -91.3%      21297 ± 26%  interrupts.CPU32.CAL:Function_call_interrupts
    161645 ± 15%     -88.4%      18762 ± 20%  interrupts.CPU32.RES:Rescheduling_interrupts
    972.25 ±  3%     -17.7%     800.25 ±  6%  interrupts.CPU32.TLB:TLB_shootdowns
    270008 ± 31%     -92.3%      20768 ± 25%  interrupts.CPU33.CAL:Function_call_interrupts
    154188 ± 12%     -86.1%      21419 ± 19%  interrupts.CPU33.RES:Rescheduling_interrupts
    987.50 ±  3%     -21.1%     779.00 ±  6%  interrupts.CPU33.TLB:TLB_shootdowns
    231024 ± 31%     -91.2%      20252 ± 33%  interrupts.CPU34.CAL:Function_call_interrupts
    168487 ± 16%     -88.7%      18989 ± 20%  interrupts.CPU34.RES:Rescheduling_interrupts
    997.50 ±  4%     -23.4%     764.50 ±  3%  interrupts.CPU34.TLB:TLB_shootdowns
    233104 ± 24%     -92.6%      17182 ± 18%  interrupts.CPU35.CAL:Function_call_interrupts
    183698 ± 18%     -89.8%      18677 ± 18%  interrupts.CPU35.RES:Rescheduling_interrupts
    999.50 ±  4%     -21.8%     781.50 ±  9%  interrupts.CPU35.TLB:TLB_shootdowns
    215073 ± 11%     -91.4%      18523 ± 16%  interrupts.CPU36.CAL:Function_call_interrupts
    153732 ± 18%     -89.2%      16659 ±  5%  interrupts.CPU36.RES:Rescheduling_interrupts
    979.50 ±  2%     -19.7%     786.50 ± 10%  interrupts.CPU36.TLB:TLB_shootdowns
    217389 ± 25%     -90.9%      19719 ± 30%  interrupts.CPU37.CAL:Function_call_interrupts
    174628 ± 20%     -89.0%      19148 ± 18%  interrupts.CPU37.RES:Rescheduling_interrupts
    990.75 ±  3%     -20.7%     786.00 ±  7%  interrupts.CPU37.TLB:TLB_shootdowns
    311522 ± 55%     -91.9%      25347 ± 45%  interrupts.CPU38.CAL:Function_call_interrupts
    161443 ±  7%     -87.5%      20216 ± 22%  interrupts.CPU38.RES:Rescheduling_interrupts
    975.25 ±  2%     -19.6%     784.50 ±  8%  interrupts.CPU38.TLB:TLB_shootdowns
    314470 ± 35%     -94.1%      18708 ± 20%  interrupts.CPU39.CAL:Function_call_interrupts
    139101 ± 26%     -87.0%      18130 ± 12%  interrupts.CPU39.RES:Rescheduling_interrupts
    991.25 ±  2%     -18.5%     808.25 ±  9%  interrupts.CPU39.TLB:TLB_shootdowns
    175190 ± 34%     -90.3%      16958 ± 19%  interrupts.CPU4.CAL:Function_call_interrupts
     89945 ± 33%     -81.8%      16380 ± 22%  interrupts.CPU4.RES:Rescheduling_interrupts
    964.75 ±  3%     -12.5%     844.50 ±  5%  interrupts.CPU4.TLB:TLB_shootdowns
    297916 ± 25%     -93.5%      19294 ± 29%  interrupts.CPU40.CAL:Function_call_interrupts
    179694 ±  7%     -89.3%      19231 ± 14%  interrupts.CPU40.RES:Rescheduling_interrupts
    964.75 ±  2%     -21.0%     762.50 ±  6%  interrupts.CPU40.TLB:TLB_shootdowns
    274482 ± 32%     -91.0%      24643 ± 12%  interrupts.CPU41.CAL:Function_call_interrupts
    196924 ±  3%     -88.7%      22200 ± 14%  interrupts.CPU41.RES:Rescheduling_interrupts
    983.50 ±  2%     -22.0%     767.50 ±  8%  interrupts.CPU41.TLB:TLB_shootdowns
    229609 ± 25%     -88.8%      25708 ± 33%  interrupts.CPU42.CAL:Function_call_interrupts
    154570 ± 11%     -86.5%      20869 ± 19%  interrupts.CPU42.RES:Rescheduling_interrupts
    979.00 ±  6%     -22.0%     763.50 ±  8%  interrupts.CPU42.TLB:TLB_shootdowns
    202307 ±  8%     -88.5%      23288 ± 32%  interrupts.CPU43.CAL:Function_call_interrupts
    175707 ± 19%     -87.8%      21465 ± 13%  interrupts.CPU43.RES:Rescheduling_interrupts
    995.75 ±  3%     -21.3%     783.25 ±  3%  interrupts.CPU43.TLB:TLB_shootdowns
    234634 ± 22%     -89.8%      23836 ± 30%  interrupts.CPU44.CAL:Function_call_interrupts
    167024 ±  7%     -87.8%      20416 ± 13%  interrupts.CPU44.RES:Rescheduling_interrupts
    975.00 ±  2%     -20.9%     771.50 ±  8%  interrupts.CPU44.TLB:TLB_shootdowns
    261681 ± 15%     -91.2%      23004 ± 16%  interrupts.CPU45.CAL:Function_call_interrupts
    160884 ± 14%     -87.1%      20679 ± 13%  interrupts.CPU45.RES:Rescheduling_interrupts
    963.50 ±  4%     -18.6%     784.25 ± 10%  interrupts.CPU45.TLB:TLB_shootdowns
    241381 ± 23%     -89.2%      25977 ± 36%  interrupts.CPU46.CAL:Function_call_interrupts
    187691 ± 24%     -87.6%      23222 ±  9%  interrupts.CPU46.RES:Rescheduling_interrupts
    998.25           -23.5%     763.25 ±  4%  interrupts.CPU46.TLB:TLB_shootdowns
    196129 ±  9%     -86.3%      26786 ± 44%  interrupts.CPU47.CAL:Function_call_interrupts
    171732 ± 18%     -88.1%      20448 ± 15%  interrupts.CPU47.RES:Rescheduling_interrupts
    952.75           -17.6%     785.25 ±  4%  interrupts.CPU47.TLB:TLB_shootdowns
    220500 ±  5%     -88.2%      25940 ± 44%  interrupts.CPU48.CAL:Function_call_interrupts
    184654 ± 22%     -88.1%      21930 ± 20%  interrupts.CPU48.RES:Rescheduling_interrupts
    992.25 ±  4%     -21.7%     777.00 ±  6%  interrupts.CPU48.TLB:TLB_shootdowns
    218929 ± 12%     -89.5%      23004 ± 17%  interrupts.CPU49.CAL:Function_call_interrupts
    177347 ± 22%     -88.1%      21118 ± 10%  interrupts.CPU49.RES:Rescheduling_interrupts
    951.75 ±  5%     -17.5%     785.00 ±  6%  interrupts.CPU49.TLB:TLB_shootdowns
    194819 ± 40%     -92.4%      14902 ± 18%  interrupts.CPU5.CAL:Function_call_interrupts
     79044 ± 32%     -75.2%      19640 ± 19%  interrupts.CPU5.RES:Rescheduling_interrupts
    980.75 ±  5%     -24.7%     738.50 ±  7%  interrupts.CPU5.TLB:TLB_shootdowns
    223002 ± 20%     -88.2%      26348 ± 45%  interrupts.CPU50.CAL:Function_call_interrupts
    160668 ± 23%     -86.2%      22177 ± 19%  interrupts.CPU50.RES:Rescheduling_interrupts
      1004 ±  5%     -23.9%     764.50 ±  6%  interrupts.CPU50.TLB:TLB_shootdowns
    309989 ± 32%     -91.0%      27839 ± 29%  interrupts.CPU51.CAL:Function_call_interrupts
    170499 ± 19%     -87.4%      21434 ±  8%  interrupts.CPU51.RES:Rescheduling_interrupts
    988.00 ±  4%     -21.2%     778.75 ±  6%  interrupts.CPU51.TLB:TLB_shootdowns
    203769 ± 18%     -91.2%      17893 ± 29%  interrupts.CPU52.CAL:Function_call_interrupts
    100629 ± 43%     -77.5%      22610 ± 52%  interrupts.CPU52.RES:Rescheduling_interrupts
    994.50 ±  5%     -24.5%     750.50 ±  3%  interrupts.CPU52.TLB:TLB_shootdowns
    268555 ± 46%     -93.6%      17244 ± 20%  interrupts.CPU53.CAL:Function_call_interrupts
     84390 ± 27%     -73.6%      22265 ± 25%  interrupts.CPU53.RES:Rescheduling_interrupts
    948.50 ±  2%     -22.7%     732.75 ±  4%  interrupts.CPU53.TLB:TLB_shootdowns
    243003 ± 32%     -92.3%      18730 ± 33%  interrupts.CPU54.CAL:Function_call_interrupts
     88915 ± 40%     -79.0%      18628 ± 14%  interrupts.CPU54.RES:Rescheduling_interrupts
    939.25 ±  3%     -17.2%     777.25 ±  6%  interrupts.CPU54.TLB:TLB_shootdowns
    202800 ± 23%     -90.7%      18836 ± 14%  interrupts.CPU55.CAL:Function_call_interrupts
    109024 ± 22%     -81.4%      20276 ± 16%  interrupts.CPU55.RES:Rescheduling_interrupts
    947.25 ±  3%     -19.7%     761.00 ±  4%  interrupts.CPU55.TLB:TLB_shootdowns
    224205 ± 41%     -87.8%      27356 ± 53%  interrupts.CPU56.CAL:Function_call_interrupts
    103035 ± 34%     -79.4%      21240 ± 13%  interrupts.CPU56.RES:Rescheduling_interrupts
    937.50 ±  3%     -19.7%     753.25 ±  6%  interrupts.CPU56.TLB:TLB_shootdowns
    239213 ± 26%     -92.9%      16972 ± 21%  interrupts.CPU57.CAL:Function_call_interrupts
     87777 ± 25%     -77.0%      20174 ± 21%  interrupts.CPU57.RES:Rescheduling_interrupts
    978.00 ±  2%     -22.9%     754.00 ±  2%  interrupts.CPU57.TLB:TLB_shootdowns
    297549 ± 46%     -93.1%      20527 ± 29%  interrupts.CPU58.CAL:Function_call_interrupts
     85880 ± 29%     -65.3%      29818 ± 42%  interrupts.CPU58.RES:Rescheduling_interrupts
    956.50 ±  4%     -24.0%     726.50 ±  5%  interrupts.CPU58.TLB:TLB_shootdowns
    174057 ± 18%     -85.9%      24531 ± 44%  interrupts.CPU59.CAL:Function_call_interrupts
    115811 ± 43%     -81.8%      21024 ± 23%  interrupts.CPU59.RES:Rescheduling_interrupts
    960.25 ±  2%     -21.3%     755.75 ±  9%  interrupts.CPU59.TLB:TLB_shootdowns
    224175 ± 41%     -94.2%      13090 ± 20%  interrupts.CPU6.CAL:Function_call_interrupts
     88182 ± 37%     -79.2%      18317 ± 34%  interrupts.CPU6.RES:Rescheduling_interrupts
    939.75 ±  5%     -18.6%     764.75 ±  6%  interrupts.CPU6.TLB:TLB_shootdowns
    181880 ± 33%     -87.2%      23212 ± 45%  interrupts.CPU60.CAL:Function_call_interrupts
     92656 ± 33%     -78.0%      20396 ± 13%  interrupts.CPU60.RES:Rescheduling_interrupts
    947.50 ±  3%     -19.2%     765.75 ±  8%  interrupts.CPU60.TLB:TLB_shootdowns
    204760 ± 19%     -91.4%      17653 ±  8%  interrupts.CPU61.CAL:Function_call_interrupts
     92688 ± 21%     -79.5%      19011 ± 11%  interrupts.CPU61.RES:Rescheduling_interrupts
    913.75 ±  3%     -16.4%     763.50        interrupts.CPU61.TLB:TLB_shootdowns
    220484 ± 44%     -91.7%      18307 ± 29%  interrupts.CPU62.CAL:Function_call_interrupts
    100766 ± 38%     -79.3%      20865 ± 15%  interrupts.CPU62.RES:Rescheduling_interrupts
    951.00 ±  4%     -19.7%     763.50 ±  4%  interrupts.CPU62.TLB:TLB_shootdowns
    157447 ± 21%     -89.1%      17150 ± 23%  interrupts.CPU63.CAL:Function_call_interrupts
    111514 ± 27%     -82.8%      19217 ± 24%  interrupts.CPU63.RES:Rescheduling_interrupts
    946.50 ±  4%     -24.4%     715.25 ±  4%  interrupts.CPU63.TLB:TLB_shootdowns
    211518 ±  9%     -92.6%      15749 ± 17%  interrupts.CPU64.CAL:Function_call_interrupts
     91890 ± 26%     -78.2%      19991 ± 20%  interrupts.CPU64.RES:Rescheduling_interrupts
    972.25           -23.8%     740.75 ±  4%  interrupts.CPU64.TLB:TLB_shootdowns
    191458 ± 36%     -89.0%      21148 ± 29%  interrupts.CPU65.CAL:Function_call_interrupts
     93824 ± 28%     -78.6%      20106 ± 18%  interrupts.CPU65.RES:Rescheduling_interrupts
    940.25 ±  5%     -21.7%     736.00 ±  6%  interrupts.CPU65.TLB:TLB_shootdowns
    175909 ± 28%     -88.4%      20457 ± 32%  interrupts.CPU66.CAL:Function_call_interrupts
     97276 ± 26%     -78.4%      21043 ± 29%  interrupts.CPU66.RES:Rescheduling_interrupts
    955.25 ±  4%     -20.5%     759.50 ±  4%  interrupts.CPU66.TLB:TLB_shootdowns
    147273 ± 16%     -85.2%      21858 ± 57%  interrupts.CPU67.CAL:Function_call_interrupts
     95512 ± 25%     -80.5%      18580 ± 14%  interrupts.CPU67.RES:Rescheduling_interrupts
    929.00 ±  3%     -17.6%     765.50 ±  5%  interrupts.CPU67.TLB:TLB_shootdowns
    145708 ± 41%     -87.0%      18998 ± 33%  interrupts.CPU68.CAL:Function_call_interrupts
     99859 ± 44%     -77.6%      22326 ± 25%  interrupts.CPU68.RES:Rescheduling_interrupts
    934.75 ±  3%     -19.7%     750.75 ±  2%  interrupts.CPU68.TLB:TLB_shootdowns
    199682 ± 39%     -91.4%      17253 ± 27%  interrupts.CPU69.CAL:Function_call_interrupts
     91410 ± 44%     -77.2%      20819 ± 23%  interrupts.CPU69.RES:Rescheduling_interrupts
    925.50 ±  2%     -20.4%     737.00 ±  5%  interrupts.CPU69.TLB:TLB_shootdowns
    222715 ± 50%     -92.1%      17636 ± 32%  interrupts.CPU7.CAL:Function_call_interrupts
     95265 ± 33%     -79.6%      19454 ± 19%  interrupts.CPU7.RES:Rescheduling_interrupts
    952.75 ±  4%     -19.7%     765.00 ±  3%  interrupts.CPU7.TLB:TLB_shootdowns
    142175 ± 20%     -87.9%      17144 ±  2%  interrupts.CPU70.CAL:Function_call_interrupts
     94122 ± 38%     -79.2%      19614 ± 14%  interrupts.CPU70.RES:Rescheduling_interrupts
    913.50 ±  5%     -18.2%     747.25 ±  6%  interrupts.CPU70.TLB:TLB_shootdowns
    183802 ± 17%     -87.6%      22799 ± 48%  interrupts.CPU71.CAL:Function_call_interrupts
     94544 ± 20%     -78.1%      20725 ± 22%  interrupts.CPU71.RES:Rescheduling_interrupts
    226205 ± 48%     -88.4%      26142 ± 52%  interrupts.CPU72.CAL:Function_call_interrupts
    103772 ± 24%     -80.4%      20331 ± 27%  interrupts.CPU72.RES:Rescheduling_interrupts
    926.00           -19.0%     750.00 ±  7%  interrupts.CPU72.TLB:TLB_shootdowns
    201882 ± 40%     -89.9%      20366 ± 38%  interrupts.CPU73.CAL:Function_call_interrupts
     88094 ± 30%     -77.4%      19940 ± 20%  interrupts.CPU73.RES:Rescheduling_interrupts
    915.50 ±  6%     -14.9%     779.00 ±  4%  interrupts.CPU73.TLB:TLB_shootdowns
    143309 ± 25%     -87.9%      17359 ± 36%  interrupts.CPU74.CAL:Function_call_interrupts
    109788 ± 26%     -82.2%      19581 ± 19%  interrupts.CPU74.RES:Rescheduling_interrupts
    935.00 ±  3%     -19.0%     757.00 ±  4%  interrupts.CPU74.TLB:TLB_shootdowns
    150353 ± 34%     -90.0%      15069 ± 31%  interrupts.CPU75.CAL:Function_call_interrupts
     92423 ± 32%     -78.9%      19478 ± 23%  interrupts.CPU75.RES:Rescheduling_interrupts
    924.50 ±  2%     -19.2%     746.75 ±  4%  interrupts.CPU75.TLB:TLB_shootdowns
    121045 ± 21%     -84.9%      18221 ± 36%  interrupts.CPU76.CAL:Function_call_interrupts
     94876 ± 25%     -79.3%      19631 ± 18%  interrupts.CPU76.RES:Rescheduling_interrupts
    954.50 ±  2%     -19.1%     771.75 ±  3%  interrupts.CPU76.TLB:TLB_shootdowns
    137345 ± 30%     -86.8%      18197 ± 63%  interrupts.CPU77.CAL:Function_call_interrupts
    104707 ± 35%     -81.9%      19003 ± 22%  interrupts.CPU77.RES:Rescheduling_interrupts
    917.00 ±  4%     -18.6%     746.75 ±  2%  interrupts.CPU77.TLB:TLB_shootdowns
    193947 ± 16%     -87.7%      23840 ± 22%  interrupts.CPU78.CAL:Function_call_interrupts
    173906 ± 14%     -88.4%      20248 ± 18%  interrupts.CPU78.RES:Rescheduling_interrupts
    961.00 ±  2%     -21.3%     756.00 ±  7%  interrupts.CPU78.TLB:TLB_shootdowns
    209991 ± 19%     -88.3%      24613 ± 29%  interrupts.CPU79.CAL:Function_call_interrupts
    168981 ± 12%     -88.1%      20098 ± 18%  interrupts.CPU79.RES:Rescheduling_interrupts
    966.75 ±  3%     -21.7%     757.25 ±  9%  interrupts.CPU79.TLB:TLB_shootdowns
    156899 ± 32%     -88.7%      17697 ± 37%  interrupts.CPU8.CAL:Function_call_interrupts
    102075 ± 31%     -81.5%      18874 ±  5%  interrupts.CPU8.RES:Rescheduling_interrupts
    968.00 ±  5%     -22.7%     748.25 ±  2%  interrupts.CPU8.TLB:TLB_shootdowns
    220471 ± 22%     -85.9%      31011 ± 45%  interrupts.CPU80.CAL:Function_call_interrupts
    144946 ± 16%     -85.6%      20909 ± 23%  interrupts.CPU80.RES:Rescheduling_interrupts
    928.50 ±  4%     -15.5%     784.25 ±  7%  interrupts.CPU80.TLB:TLB_shootdowns
    216800 ± 13%     -90.5%      20532 ± 10%  interrupts.CPU81.CAL:Function_call_interrupts
    143282 ± 20%     -86.1%      19864 ± 14%  interrupts.CPU81.RES:Rescheduling_interrupts
    961.50 ±  4%     -21.8%     751.50 ±  8%  interrupts.CPU81.TLB:TLB_shootdowns
    202116 ± 25%     -89.6%      20966 ± 27%  interrupts.CPU82.CAL:Function_call_interrupts
    146866 ± 12%     -87.6%      18262 ± 14%  interrupts.CPU82.RES:Rescheduling_interrupts
    941.00           -19.8%     754.25 ±  6%  interrupts.CPU82.TLB:TLB_shootdowns
    234326 ±  7%     -92.2%      18201 ± 16%  interrupts.CPU83.CAL:Function_call_interrupts
    182247 ±  9%     -89.8%      18610 ± 23%  interrupts.CPU83.RES:Rescheduling_interrupts
    944.50 ±  4%     -20.8%     748.25 ±  8%  interrupts.CPU83.TLB:TLB_shootdowns
    249155 ± 14%     -92.0%      19999 ± 23%  interrupts.CPU84.CAL:Function_call_interrupts
    170031 ± 12%     -83.5%      28060 ± 58%  interrupts.CPU84.RES:Rescheduling_interrupts
    939.00 ±  2%     -22.4%     728.25 ±  8%  interrupts.CPU84.TLB:TLB_shootdowns
    204078 ± 27%     -87.6%      25298 ± 20%  interrupts.CPU85.CAL:Function_call_interrupts
    155195 ± 21%     -87.1%      19997 ±  6%  interrupts.CPU85.RES:Rescheduling_interrupts
    926.25 ±  4%     -19.4%     746.25 ±  7%  interrupts.CPU85.TLB:TLB_shootdowns
    269649 ± 21%     -92.2%      21025 ± 24%  interrupts.CPU86.CAL:Function_call_interrupts
    172678 ± 20%     -87.9%      20899 ± 19%  interrupts.CPU86.RES:Rescheduling_interrupts
    945.50 ±  3%     -19.4%     762.25 ±  9%  interrupts.CPU86.TLB:TLB_shootdowns
    261448 ± 40%     -92.4%      19996 ± 22%  interrupts.CPU87.CAL:Function_call_interrupts
    186423 ± 22%     -90.2%      18237 ± 16%  interrupts.CPU87.RES:Rescheduling_interrupts
    933.50 ±  2%     -18.1%     764.25 ±  8%  interrupts.CPU87.TLB:TLB_shootdowns
    347606 ± 48%     -94.2%      20143 ± 23%  interrupts.CPU88.CAL:Function_call_interrupts
    170646 ± 11%     -86.0%      23921 ± 47%  interrupts.CPU88.RES:Rescheduling_interrupts
    969.50 ±  6%     -21.8%     757.75 ± 11%  interrupts.CPU88.TLB:TLB_shootdowns
    256935 ± 31%     -93.5%      16819 ± 22%  interrupts.CPU89.CAL:Function_call_interrupts
    177665 ± 17%     -89.0%      19552 ± 26%  interrupts.CPU89.RES:Rescheduling_interrupts
    963.00           -22.6%     745.00 ±  5%  interrupts.CPU89.TLB:TLB_shootdowns
    181023 ± 32%     -89.9%      18246 ± 28%  interrupts.CPU9.CAL:Function_call_interrupts
    109042 ± 36%     -82.2%      19398 ±  5%  interrupts.CPU9.RES:Rescheduling_interrupts
    968.50 ±  4%     -23.1%     744.50 ±  7%  interrupts.CPU9.TLB:TLB_shootdowns
    264579 ± 24%     -90.7%      24508 ± 24%  interrupts.CPU90.CAL:Function_call_interrupts
    183053 ± 12%     -88.8%      20547 ± 17%  interrupts.CPU90.RES:Rescheduling_interrupts
    977.50 ±  3%     -23.4%     749.25 ±  8%  interrupts.CPU90.TLB:TLB_shootdowns
    223839 ± 35%     -90.8%      20586 ± 17%  interrupts.CPU91.CAL:Function_call_interrupts
    144548 ± 14%     -87.1%      18586 ± 13%  interrupts.CPU91.RES:Rescheduling_interrupts
    961.00 ±  5%     -24.5%     726.00 ±  8%  interrupts.CPU91.TLB:TLB_shootdowns
    227207 ± 29%     -90.4%      21839 ± 21%  interrupts.CPU92.CAL:Function_call_interrupts
    175733 ± 14%     -88.6%      20048 ± 17%  interrupts.CPU92.RES:Rescheduling_interrupts
    944.00 ±  3%     -20.9%     746.25 ±  8%  interrupts.CPU92.TLB:TLB_shootdowns
    240500 ± 10%     -91.8%      19716 ± 23%  interrupts.CPU93.CAL:Function_call_interrupts
    190796 ± 12%     -89.9%      19239 ± 21%  interrupts.CPU93.RES:Rescheduling_interrupts
    944.75 ±  2%     -20.0%     755.75 ±  7%  interrupts.CPU93.TLB:TLB_shootdowns
    259259 ± 55%     -92.8%      18649 ± 42%  interrupts.CPU94.CAL:Function_call_interrupts
    154375 ± 15%     -87.4%      19425 ± 25%  interrupts.CPU94.RES:Rescheduling_interrupts
    944.25 ±  3%     -16.6%     787.25 ±  7%  interrupts.CPU94.TLB:TLB_shootdowns
    254216 ± 16%     -91.5%      21545 ± 24%  interrupts.CPU95.CAL:Function_call_interrupts
    186091 ± 17%     -89.2%      20135 ± 20%  interrupts.CPU95.RES:Rescheduling_interrupts
    938.75 ±  2%     -19.4%     756.75 ± 10%  interrupts.CPU95.TLB:TLB_shootdowns
    266996 ± 10%     -93.0%      18642 ± 19%  interrupts.CPU96.CAL:Function_call_interrupts
    172889 ± 16%     -89.7%      17801 ± 14%  interrupts.CPU96.RES:Rescheduling_interrupts
    951.00 ±  2%     -20.4%     757.25 ± 14%  interrupts.CPU96.TLB:TLB_shootdowns
    193323 ± 21%     -89.9%      19611 ± 27%  interrupts.CPU97.CAL:Function_call_interrupts
    160260 ± 15%     -86.7%      21327 ± 17%  interrupts.CPU97.RES:Rescheduling_interrupts
    947.00 ±  4%     -19.1%     766.50 ±  5%  interrupts.CPU97.TLB:TLB_shootdowns
    261735 ± 26%     -90.3%      25274 ± 31%  interrupts.CPU98.CAL:Function_call_interrupts
    196196 ± 25%     -89.8%      19959 ± 15%  interrupts.CPU98.RES:Rescheduling_interrupts
    958.50 ±  2%     -21.6%     751.75 ± 10%  interrupts.CPU98.TLB:TLB_shootdowns
    186979 ± 12%     -89.5%      19580 ± 22%  interrupts.CPU99.CAL:Function_call_interrupts
    171722 ± 21%     -88.3%      20089 ± 18%  interrupts.CPU99.RES:Rescheduling_interrupts
    961.75 ±  4%     -24.8%     723.25 ±  7%  interrupts.CPU99.TLB:TLB_shootdowns
  13865203 ±  5%     -84.9%    2096456 ±  6%  interrupts.RES:Rescheduling_interrupts
     99890 ±  2%     -20.9%      79026 ±  5%  interrupts.TLB:TLB_shootdowns


                                                                                
                            will-it-scale.per_thread_ops                        
                                                                                
  170000 +------------------------------------------------------------------+   
         |     .+.+. .+..+.+.+.+..   .+.    .+. .+.     +.+.+..+  +    .+.+.|   
  160000 |.+.+.     +             +.+   +..+   +   +.. +           +.+.     |   
  150000 |-+                                          +                     |   
         |                                                                  |   
  140000 |-+                                                                |   
         |                                                                  |   
  130000 |-+                                                                |   
         |                                                                  |   
  120000 |-+                                                                |   
  110000 |-+                                                                |   
         |                   O                          O                   |   
  100000 |-O O  O O O O    O   O  O O O O  O O O O O  O   O O  O            |   
         |               O                                                  |   
   90000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                will-it-scale.workload                          
                                                                                
    9e+06 +-----------------------------------------------------------------+   
          |                .+. .+.    .+    .+.  .+.     .+..+. .+.    .+.  |   
  8.5e+06 |.+   .+.+.+.+.+.   +   +.+.  + .+   +.   +. .+      +   +.+.   +.|   
    8e+06 |-+.+.                         +            +                     |   
          |                                                                 |   
  7.5e+06 |-+                                                               |   
    7e+06 |-+                                                               |   
          |                                                                 |   
  6.5e+06 |-+                                                               |   
    6e+06 |-+                                                               |   
          |                                                                 |   
  5.5e+06 |-+ O    O        O O O   O    O        O     O      O            |   
    5e+06 |-O    O   O O          O    O   O O O    O O   O  O              |   
          |              O                                                  |   
  4.5e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.9.0-rc1-00125-gbdfcae11403e50" of type "text/plain" (170215 bytes)

View attachment "job-script" of type "text/plain" (7743 bytes)

View attachment "job.yaml" of type "text/plain" (5198 bytes)

View attachment "reproduce" of type "text/plain" (346 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ