lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200811083059.GA7488@shao2-debian>
Date:   Tue, 11 Aug 2020 16:30:59 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Lorenzo Bianconi <lorenzo@...nel.org>
Cc:     Daniel Borkmann <daniel@...earbox.net>,
        Jesper Dangaard Brouer <brouer@...hat.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [bpf] 9216477449: will-it-scale.per_process_ops 196.3% improvement

Greeting,

FYI, we noticed a 196.3% improvement of will-it-scale.per_process_ops due to commit:


commit: 9216477449f33cdbc9c9a99d49f500b7fbb81702 ("bpf: cpumap: Add the possibility to attach an eBPF program to cpumap")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: will-it-scale
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	nr_task: 50%
	mode: process
	test: context_switch1
	cpufreq_governor: performance
	ucode: 0x5002f01

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/process/50%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2ap3/context_switch1/will-it-scale/0x5002f01

commit: 
  644bfe51fa ("cpumap: Formalize map value as a named struct")
  9216477449 ("bpf: cpumap: Add the possibility to attach an eBPF program to cpumap")

644bfe51fa49c222 9216477449f33cdbc9c9a99d49f 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    101816          +196.3%     301655 ±  6%  will-it-scale.per_process_ops
   9774396          +196.3%   28958917 ±  6%  will-it-scale.workload
      1019 ± 13%     -12.2%     895.00 ± 10%  slabinfo.file_lock_cache.active_objs
      1019 ± 13%     -12.2%     895.00 ± 10%  slabinfo.file_lock_cache.num_objs
 6.921e+08           -29.8%  4.856e+08 ± 18%  cpuidle.C1.time
 8.752e+08 ± 12%     -21.7%   6.85e+08 ± 14%  cpuidle.C1E.time
 1.603e+09          +241.6%  5.474e+09        cpuidle.POLL.time
 7.614e+08          +647.1%  5.689e+09 ±  7%  cpuidle.POLL.usage
     70.14           -22.4       47.77 ±  3%  mpstat.cpu.all.idle%
      0.89            +0.5        1.40 ±  2%  mpstat.cpu.all.irq%
     27.00           +20.5       47.52 ±  3%  mpstat.cpu.all.sys%
      1.73            +1.3        3.05 ±  6%  mpstat.cpu.all.usr%
     70.00           -31.8%      47.75 ±  3%  vmstat.cpu.id
     27.25           +77.1%      48.25 ±  3%  vmstat.cpu.sy
  19271255          +197.2%   57274788 ±  6%  vmstat.system.cs
   1394938           +62.0%    2259720 ±  2%  vmstat.system.in
    119901 ±  3%      -9.0%     109077 ±  5%  numa-meminfo.node0.SUnreclaim
     80409 ± 73%    +115.1%     172925 ± 26%  numa-meminfo.node1.Active
     79506 ± 75%    +117.5%     172918 ± 26%  numa-meminfo.node1.Active(anon)
     43624 ±105%    +170.5%     118004 ± 21%  numa-meminfo.node1.AnonHugePages
     78918 ± 76%    +118.4%     172320 ± 25%  numa-meminfo.node1.AnonPages
    116818           +11.5%     130293 ±  2%  numa-meminfo.node3.Shmem
      7496            -6.2%       7031        proc-vmstat.nr_inactive_anon
     12880            -6.3%      12074        proc-vmstat.nr_mapped
     33884 ±  3%      +8.0%      36587 ±  2%  proc-vmstat.nr_shmem
     36950            -1.3%      36459        proc-vmstat.nr_slab_reclaimable
      7496            -6.2%       7031        proc-vmstat.nr_zone_inactive_anon
   1090121            +2.1%    1113293        proc-vmstat.pgalloc_normal
     29975 ±  3%      -9.0%      27268 ±  5%  numa-vmstat.node0.nr_slab_unreclaimable
    676703 ±  8%     -22.6%     523904 ± 11%  numa-vmstat.node0.numa_hit
    651307 ±  9%     -28.3%     467226 ±  8%  numa-vmstat.node0.numa_local
     19868 ± 75%    +117.5%      43212 ± 26%  numa-vmstat.node1.nr_active_anon
     19721 ± 76%    +118.4%      43062 ± 25%  numa-vmstat.node1.nr_anon_pages
     19868 ± 75%    +117.5%      43212 ± 26%  numa-vmstat.node1.nr_zone_active_anon
     29254           +11.3%      32560 ±  2%  numa-vmstat.node3.nr_shmem
   2947447 ±  6%     -18.6%    2398308 ±  7%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   9344834           +18.6%   11087277        sched_debug.cfs_rq:/.MIN_vruntime.max
      1.25 ± 29%     -80.0%       0.25 ±110%  sched_debug.cfs_rq:/.load_avg.min
   2947447 ±  6%     -18.6%    2398308 ±  7%  sched_debug.cfs_rq:/.max_vruntime.avg
   9344834           +18.6%   11087277        sched_debug.cfs_rq:/.max_vruntime.max
   9251241           +18.7%   10977704        sched_debug.cfs_rq:/.min_vruntime.avg
   9391185           +19.1%   11182096        sched_debug.cfs_rq:/.min_vruntime.max
   8733630           +20.3%   10503192        sched_debug.cfs_rq:/.min_vruntime.min
     54678 ±  2%     +21.1%      66214 ±  5%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.52 ± 26%     -72.2%       0.14 ±173%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
     60.08 ± 26%     -88.6%       6.88 ±173%  sched_debug.cfs_rq:/.removed.runnable_avg.max
      5.30 ± 20%     -82.6%       0.92 ±173%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      0.52 ± 26%     -72.2%       0.14 ±173%  sched_debug.cfs_rq:/.removed.util_avg.avg
     60.08 ± 26%     -88.6%       6.88 ±173%  sched_debug.cfs_rq:/.removed.util_avg.max
      5.30 ± 20%     -82.6%       0.92 ±173%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     54902 ±  2%     +20.9%      66396 ±  5%  sched_debug.cfs_rq:/.spread0.stddev
      3978 ± 58%     -66.4%       1335 ±  4%  sched_debug.cpu.avg_idle.min
     14.55 ±  5%     -12.7%      12.71 ±  4%  sched_debug.cpu.clock.stddev
      1104 ±  2%     +47.6%       1629        sched_debug.cpu.curr->pid.avg
  15230386          +206.7%   46707837 ±  4%  sched_debug.cpu.nr_switches.avg
  17060788          +183.3%   48325371 ±  3%  sched_debug.cpu.nr_switches.max
  11757170          +261.6%   42519522 ±  5%  sched_debug.cpu.nr_switches.min
      0.74 ±  5%     -32.6%       0.50 ±  7%  sched_debug.rt_rq:/.rt_time.avg
    141.24 ±  5%     -32.6%      95.23 ±  7%  sched_debug.rt_rq:/.rt_time.max
     10.17 ±  5%     -32.6%       6.85 ±  7%  sched_debug.rt_rq:/.rt_time.stddev
     25629 ±  2%      -7.1%      23821 ±  4%  softirqs.CPU104.RCU
     25696 ±  2%      -9.9%      23140 ±  4%  softirqs.CPU11.RCU
     24547 ±  2%      -7.6%      22692 ±  4%  softirqs.CPU117.RCU
     24586 ±  2%      -8.0%      22613 ±  4%  softirqs.CPU118.RCU
     24588 ±  2%      -7.3%      22797 ±  5%  softirqs.CPU121.RCU
    123169 ±  5%     -12.1%     108286        softirqs.CPU123.TIMER
     25041            -9.6%      22646 ±  4%  softirqs.CPU124.RCU
     24739            -9.0%      22524 ±  4%  softirqs.CPU126.RCU
     24656 ±  3%      -7.4%      22833 ±  4%  softirqs.CPU133.RCU
     24545 ±  2%      -8.2%      22535 ±  4%  softirqs.CPU134.RCU
     25062 ±  2%      -9.7%      22636 ±  4%  softirqs.CPU138.RCU
    123045 ±  6%     -12.2%     108093        softirqs.CPU143.TIMER
     25562 ±  5%     -11.3%      22674 ±  3%  softirqs.CPU159.RCU
     24905 ±  2%      -8.2%      22874 ±  4%  softirqs.CPU160.RCU
     24158 ±  4%     -10.2%      21687 ±  4%  softirqs.CPU174.RCU
     23858 ±  2%      -9.0%      21710 ±  4%  softirqs.CPU179.RCU
     23821 ±  3%      -8.5%      21803 ±  4%  softirqs.CPU181.RCU
     23644 ±  2%      -8.5%      21627 ±  4%  softirqs.CPU184.RCU
     23914 ±  2%      -8.5%      21878 ±  4%  softirqs.CPU187.RCU
     24175 ±  4%     -10.1%      21725 ±  4%  softirqs.CPU189.RCU
     25336 ±  2%      -8.2%      23269 ±  4%  softirqs.CPU19.RCU
     25253 ±  2%      -7.9%      23246 ±  4%  softirqs.CPU21.RCU
     25493            -7.9%      23476 ±  4%  softirqs.CPU22.RCU
     25486 ±  2%      -7.9%      23471 ±  4%  softirqs.CPU25.RCU
     25388            -8.5%      23240 ±  4%  softirqs.CPU33.RCU
     25145 ±  2%      -7.9%      23167 ±  4%  softirqs.CPU38.RCU
     25187 ±  2%      -8.1%      23144 ±  4%  softirqs.CPU4.RCU
     25074 ±  2%      -8.2%      23017 ±  4%  softirqs.CPU46.RCU
     25289 ±  2%      -8.5%      23137 ±  3%  softirqs.CPU6.RCU
     25266 ±  3%      -9.6%      22852 ±  4%  softirqs.CPU7.RCU
     24232 ±  2%      -7.9%      22312 ±  4%  softirqs.CPU75.RCU
     24174 ±  2%      -7.8%      22277 ±  4%  softirqs.CPU80.RCU
     24158 ±  2%      -8.4%      22116 ±  4%  softirqs.CPU81.RCU
     24375 ±  2%      -8.8%      22228 ±  4%  softirqs.CPU84.RCU
     24577 ±  4%      -9.6%      22213 ±  4%  softirqs.CPU86.RCU
     24441 ±  3%      -8.8%      22278 ±  4%  softirqs.CPU89.RCU
     24513 ±  3%      -9.6%      22160 ±  4%  softirqs.CPU91.RCU
     24208 ±  2%      -9.1%      22010 ±  4%  softirqs.CPU94.RCU
      4.20 ±  4%     -73.6%       1.11 ± 17%  perf-stat.i.MPKI
 2.165e+10          +194.0%  6.366e+10 ±  5%  perf-stat.i.branch-instructions
      1.32            -0.1        1.25        perf-stat.i.branch-miss-rate%
 2.846e+08          +177.9%   7.91e+08 ±  5%  perf-stat.i.branch-misses
     33.51 ±  4%      -9.6       23.89 ±  6%  perf-stat.i.cache-miss-rate%
 1.449e+08           -47.2%   76585659 ±  6%  perf-stat.i.cache-misses
  4.35e+08 ±  4%     -25.0%  3.261e+08 ± 11%  perf-stat.i.cache-references
  19448498          +196.8%   57713587 ±  6%  perf-stat.i.context-switches
      5.67           -66.0%       1.93 ±  6%  perf-stat.i.cpi
 5.881e+11            -1.3%  5.807e+11        perf-stat.i.cpu-cycles
    857.28            -4.2%     821.59 ±  2%  perf-stat.i.cpu-migrations
      4051           +88.5%       7638 ±  5%  perf-stat.i.cycles-between-cache-misses
      0.00 ±  7%      -0.0        0.00 ±  8%  perf-stat.i.dTLB-load-miss-rate%
    930897 ±  7%     +41.2%    1314099 ±  3%  perf-stat.i.dTLB-load-misses
 2.845e+10          +196.0%  8.421e+10 ±  6%  perf-stat.i.dTLB-loads
      0.00 ±  3%      -0.0        0.00 ± 10%  perf-stat.i.dTLB-store-miss-rate%
 1.755e+10          +196.6%  5.206e+10 ±  6%  perf-stat.i.dTLB-stores
     95.54            +2.0       97.50        perf-stat.i.iTLB-load-miss-rate%
 1.692e+08          +160.9%  4.415e+08 ±  5%  perf-stat.i.iTLB-load-misses
   7737808           +35.5%   10486430 ± 18%  perf-stat.i.iTLB-loads
 1.036e+11          +193.7%  3.044e+11 ±  5%  perf-stat.i.instructions
    614.45           +12.5%     691.14        perf-stat.i.instructions-per-iTLB-miss
      0.18          +196.2%       0.52 ±  6%  perf-stat.i.ipc
      3.06            -1.2%       3.02        perf-stat.i.metric.GHz
      0.56 ±  9%     +30.9%       0.74 ±  6%  perf-stat.i.metric.K/sec
    354.75          +194.1%       1043 ±  5%  perf-stat.i.metric.M/sec
  14483626           -64.6%    5132353 ± 14%  perf-stat.i.node-load-misses
   2032599 ± 11%     -67.0%     670368 ± 19%  perf-stat.i.node-loads
    239537 ±  3%     +52.9%     366268 ±  2%  perf-stat.i.node-store-misses
     34399 ± 22%     +84.7%      63521 ± 18%  perf-stat.i.node-stores
      4.20 ±  4%     -74.3%       1.08 ± 16%  perf-stat.overall.MPKI
      1.32            -0.1        1.24        perf-stat.overall.branch-miss-rate%
     33.38 ±  4%      -9.7       23.65 ±  5%  perf-stat.overall.cache-miss-rate%
      5.68           -66.3%       1.91 ±  6%  perf-stat.overall.cpi
      4058           +87.8%       7620 ±  6%  perf-stat.overall.cycles-between-cache-misses
      0.00 ±  7%      -0.0        0.00 ±  8%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  5%      -0.0        0.00 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
     95.63            +2.0       97.66        perf-stat.overall.iTLB-load-miss-rate%
    612.03           +12.6%     689.29        perf-stat.overall.instructions-per-iTLB-miss
      0.18          +198.0%       0.52 ±  6%  perf-stat.overall.ipc
 2.157e+10          +194.4%  6.351e+10 ±  5%  perf-stat.ps.branch-instructions
 2.838e+08          +178.2%  7.895e+08 ±  5%  perf-stat.ps.branch-misses
 1.444e+08           -47.2%   76242818 ±  6%  perf-stat.ps.cache-misses
 4.337e+08 ±  4%     -25.2%  3.244e+08 ± 11%  perf-stat.ps.cache-references
  19388838          +197.1%   57611241 ±  6%  perf-stat.ps.context-switches
 5.861e+11            -1.3%  5.787e+11        perf-stat.ps.cpu-cycles
    850.02            -4.0%     816.00 ±  2%  perf-stat.ps.cpu-migrations
    916336 ±  7%     +42.1%    1302390 ±  3%  perf-stat.ps.dTLB-load-misses
 2.836e+10          +196.4%  8.405e+10 ±  5%  perf-stat.ps.dTLB-loads
 1.749e+10          +197.0%  5.195e+10 ±  6%  perf-stat.ps.dTLB-stores
 1.687e+08          +161.2%  4.407e+08 ±  5%  perf-stat.ps.iTLB-load-misses
   7718759           +35.1%   10430151 ± 18%  perf-stat.ps.iTLB-loads
 1.033e+11          +194.1%  3.037e+11 ±  5%  perf-stat.ps.instructions
  14438110           -64.7%    5100947 ± 14%  perf-stat.ps.node-load-misses
   2021614 ± 10%     -67.1%     664898 ± 19%  perf-stat.ps.node-loads
    238406 ±  3%     +53.3%     365532 ±  2%  perf-stat.ps.node-store-misses
     34159 ± 22%     +84.9%      63159 ± 18%  perf-stat.ps.node-stores
 3.131e+13          +194.0%  9.205e+13 ±  5%  perf-stat.total.instructions
  11549010 ± 16%    +302.8%   46516946 ± 12%  interrupts.CAL:Function_call_interrupts
     18852 ± 30%    +428.5%      99628 ± 15%  interrupts.CPU0.CAL:Function_call_interrupts
   1593568 ±  2%     +66.6%    2654692 ±  3%  interrupts.CPU0.RES:Rescheduling_interrupts
     20830 ± 22%   +1664.6%     367581 ±115%  interrupts.CPU1.CAL:Function_call_interrupts
      7623 ± 24%     -30.5%       5300 ± 34%  interrupts.CPU1.NMI:Non-maskable_interrupts
      7623 ± 24%     -30.5%       5300 ± 34%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
   1580171 ±  2%     +74.4%    2756599 ±  2%  interrupts.CPU1.RES:Rescheduling_interrupts
     25691 ± 60%    +320.7%     108091 ± 12%  interrupts.CPU10.CAL:Function_call_interrupts
   1571207 ±  3%     +69.2%    2659085 ±  3%  interrupts.CPU10.RES:Rescheduling_interrupts
     69672 ±105%    +225.8%     226990 ± 38%  interrupts.CPU100.CAL:Function_call_interrupts
   1568606 ±  2%     +76.8%    2773634 ±  5%  interrupts.CPU100.RES:Rescheduling_interrupts
     41727 ± 74%    +367.6%     195125 ± 74%  interrupts.CPU101.CAL:Function_call_interrupts
   1562854           +74.8%    2731299 ±  4%  interrupts.CPU101.RES:Rescheduling_interrupts
     44172 ± 62%    +157.8%     113882 ± 19%  interrupts.CPU102.CAL:Function_call_interrupts
   1576502           +74.7%    2753865 ±  6%  interrupts.CPU102.RES:Rescheduling_interrupts
   1566546           +71.8%    2690975 ±  4%  interrupts.CPU103.RES:Rescheduling_interrupts
     37536 ± 94%    +343.0%     166272 ± 34%  interrupts.CPU104.CAL:Function_call_interrupts
   1582475 ±  2%     +76.3%    2790438 ±  8%  interrupts.CPU104.RES:Rescheduling_interrupts
     28686 ± 44%    +229.8%      94614 ±  4%  interrupts.CPU105.CAL:Function_call_interrupts
   1583605 ±  2%     +71.3%    2713452 ±  4%  interrupts.CPU105.RES:Rescheduling_interrupts
   1559002           +76.4%    2750328 ±  4%  interrupts.CPU106.RES:Rescheduling_interrupts
     35817 ± 72%    +231.7%     118801 ± 22%  interrupts.CPU107.CAL:Function_call_interrupts
   1576980           +71.7%    2708199 ±  2%  interrupts.CPU107.RES:Rescheduling_interrupts
     30894 ± 58%    +388.6%     150946 ± 51%  interrupts.CPU108.CAL:Function_call_interrupts
   1586146           +76.1%    2793779 ±  4%  interrupts.CPU108.RES:Rescheduling_interrupts
     29986 ± 45%    +362.8%     138776 ± 54%  interrupts.CPU109.CAL:Function_call_interrupts
   1583817           +69.2%    2679361 ±  6%  interrupts.CPU109.RES:Rescheduling_interrupts
     25871 ± 33%   +1469.6%     406072 ± 92%  interrupts.CPU11.CAL:Function_call_interrupts
   1582938 ±  2%     +71.9%    2721399 ±  4%  interrupts.CPU11.RES:Rescheduling_interrupts
     29423 ± 47%    +242.0%     100630 ±  8%  interrupts.CPU110.CAL:Function_call_interrupts
   1583046           +73.5%    2746632 ±  4%  interrupts.CPU110.RES:Rescheduling_interrupts
     31433 ± 81%    +480.8%     182559 ± 79%  interrupts.CPU111.CAL:Function_call_interrupts
   1583320 ±  2%     +73.9%    2754136 ±  3%  interrupts.CPU111.RES:Rescheduling_interrupts
   1566623           +66.9%    2614437 ±  4%  interrupts.CPU112.RES:Rescheduling_interrupts
   1569427           +73.3%    2720296 ±  4%  interrupts.CPU113.RES:Rescheduling_interrupts
     36296 ± 58%    +152.1%      91502 ±  3%  interrupts.CPU114.CAL:Function_call_interrupts
   1580879           +72.2%    2722953 ±  3%  interrupts.CPU114.RES:Rescheduling_interrupts
     35065 ± 76%    +162.2%      91929 ±  3%  interrupts.CPU115.CAL:Function_call_interrupts
   1572597           +70.3%    2678064 ±  3%  interrupts.CPU115.RES:Rescheduling_interrupts
     16148 ± 16%    +892.1%     160213 ± 19%  interrupts.CPU116.CAL:Function_call_interrupts
   1608030 ±  2%     +81.5%    2918567 ±  7%  interrupts.CPU116.RES:Rescheduling_interrupts
     23014 ± 64%    +711.8%     186822 ± 88%  interrupts.CPU117.CAL:Function_call_interrupts
   1581689 ±  2%     +72.9%    2735014 ±  4%  interrupts.CPU117.RES:Rescheduling_interrupts
     17150 ± 22%    +475.9%      98772 ± 12%  interrupts.CPU118.CAL:Function_call_interrupts
   1596193 ±  2%     +66.2%    2653003 ±  2%  interrupts.CPU118.RES:Rescheduling_interrupts
     28345 ± 83%    +396.8%     140823 ± 61%  interrupts.CPU119.CAL:Function_call_interrupts
   1587317 ±  2%     +70.6%    2707261 ±  5%  interrupts.CPU119.RES:Rescheduling_interrupts
     54038 ± 22%     +90.4%     102862 ±  9%  interrupts.CPU12.CAL:Function_call_interrupts
   1571052 ±  2%     +74.6%    2743247 ±  4%  interrupts.CPU12.RES:Rescheduling_interrupts
     43268 ± 74%    +369.7%     203217 ± 77%  interrupts.CPU120.CAL:Function_call_interrupts
   1652051 ±  5%     +64.0%    2708741 ±  2%  interrupts.CPU120.RES:Rescheduling_interrupts
   1629277 ±  5%     +66.7%    2715936 ±  2%  interrupts.CPU121.RES:Rescheduling_interrupts
     58186 ± 18%    +122.7%     129600 ± 29%  interrupts.CPU122.CAL:Function_call_interrupts
   1625411 ±  6%     +70.0%    2763891 ±  3%  interrupts.CPU122.RES:Rescheduling_interrupts
     37387 ± 49%    +296.1%     148111 ± 49%  interrupts.CPU123.CAL:Function_call_interrupts
   1640951 ±  5%     +64.9%    2705180        interrupts.CPU123.RES:Rescheduling_interrupts
     40935 ± 67%    +248.0%     142474 ± 44%  interrupts.CPU124.CAL:Function_call_interrupts
   1636576 ±  5%     +68.2%    2753146 ±  4%  interrupts.CPU124.RES:Rescheduling_interrupts
     39535 ± 17%    +637.2%     291433 ±114%  interrupts.CPU125.CAL:Function_call_interrupts
   1632265 ±  5%     +65.3%    2697976 ±  3%  interrupts.CPU125.RES:Rescheduling_interrupts
     22954 ± 18%    +342.4%     101544 ± 11%  interrupts.CPU126.CAL:Function_call_interrupts
   1641343 ±  5%     +67.8%    2754166 ±  5%  interrupts.CPU126.RES:Rescheduling_interrupts
     37010 ± 59%    +365.6%     172319 ± 41%  interrupts.CPU127.CAL:Function_call_interrupts
   1647448 ±  5%     +61.4%    2658653 ±  5%  interrupts.CPU127.RES:Rescheduling_interrupts
   1641073 ±  4%     +68.2%    2760500 ±  3%  interrupts.CPU128.RES:Rescheduling_interrupts
     32411 ± 25%    +384.1%     156901 ± 72%  interrupts.CPU129.CAL:Function_call_interrupts
   1640572 ±  5%     +63.3%    2679303 ±  4%  interrupts.CPU129.RES:Rescheduling_interrupts
     48091 ± 35%    +679.9%     375081 ± 73%  interrupts.CPU13.CAL:Function_call_interrupts
   1570309           +69.4%    2660532 ±  3%  interrupts.CPU13.RES:Rescheduling_interrupts
     19791 ± 11%    +458.0%     110429 ± 26%  interrupts.CPU130.CAL:Function_call_interrupts
   1641751 ±  5%     +62.8%    2672944 ±  3%  interrupts.CPU130.RES:Rescheduling_interrupts
     25878 ± 42%    +267.1%      95000 ±  7%  interrupts.CPU131.CAL:Function_call_interrupts
   1645654 ±  4%     +67.5%    2755712 ±  3%  interrupts.CPU131.RES:Rescheduling_interrupts
     38150 ± 58%    +327.7%     163164 ± 47%  interrupts.CPU132.CAL:Function_call_interrupts
   1643436 ±  5%     +68.2%    2763660 ±  4%  interrupts.CPU132.RES:Rescheduling_interrupts
     44439 ± 39%    +122.7%      98980 ±  6%  interrupts.CPU133.CAL:Function_call_interrupts
   1644951 ±  5%     +66.2%    2733107 ±  3%  interrupts.CPU133.RES:Rescheduling_interrupts
     54428 ± 67%    +232.2%     180798 ± 48%  interrupts.CPU134.CAL:Function_call_interrupts
   1633910 ±  5%     +65.8%    2708280 ±  3%  interrupts.CPU134.RES:Rescheduling_interrupts
     35363 ± 32%    +258.8%     126901 ± 45%  interrupts.CPU135.CAL:Function_call_interrupts
   1645391 ±  5%     +63.9%    2696849 ±  3%  interrupts.CPU135.RES:Rescheduling_interrupts
     23893 ± 39%    +299.6%      95478 ±  8%  interrupts.CPU136.CAL:Function_call_interrupts
   1646400 ±  5%     +67.6%    2759382 ±  2%  interrupts.CPU136.RES:Rescheduling_interrupts
     32182 ± 41%    +467.5%     182649 ± 91%  interrupts.CPU137.CAL:Function_call_interrupts
   1640681 ±  5%     +68.9%    2771313 ±  4%  interrupts.CPU137.RES:Rescheduling_interrupts
     27299 ± 22%    +548.0%     176898 ± 49%  interrupts.CPU138.CAL:Function_call_interrupts
   1657546 ±  5%     +71.0%    2834867 ±  5%  interrupts.CPU138.RES:Rescheduling_interrupts
     27150 ± 45%    +256.6%      96827 ±  4%  interrupts.CPU139.CAL:Function_call_interrupts
   1648359 ±  5%     +67.0%    2752227 ±  4%  interrupts.CPU139.RES:Rescheduling_interrupts
     44514 ± 25%    +278.7%     168584 ± 22%  interrupts.CPU14.CAL:Function_call_interrupts
   1571267 ±  2%     +73.4%    2724031 ±  4%  interrupts.CPU14.RES:Rescheduling_interrupts
     44531 ± 74%    +118.3%      97218 ±  9%  interrupts.CPU140.CAL:Function_call_interrupts
   1642433 ±  4%     +67.7%    2755050 ±  3%  interrupts.CPU140.RES:Rescheduling_interrupts
     18209 ± 20%    +520.0%     112895 ±  9%  interrupts.CPU141.CAL:Function_call_interrupts
   1646542 ±  5%     +67.4%    2755857 ±  3%  interrupts.CPU141.RES:Rescheduling_interrupts
     27074 ± 25%    +248.7%      94399 ±  6%  interrupts.CPU142.CAL:Function_call_interrupts
   1636906 ±  6%     +70.5%    2790924 ±  4%  interrupts.CPU142.RES:Rescheduling_interrupts
     35996 ± 45%    +178.9%     100401 ± 18%  interrupts.CPU143.CAL:Function_call_interrupts
   1646893 ±  5%     +61.4%    2657945 ±  2%  interrupts.CPU143.RES:Rescheduling_interrupts
     65529 ± 42%    +559.7%     432327 ± 81%  interrupts.CPU144.CAL:Function_call_interrupts
   1642256 ±  6%     +74.6%    2867363        interrupts.CPU144.RES:Rescheduling_interrupts
   1620669 ±  6%     +72.6%    2797231 ±  5%  interrupts.CPU145.RES:Rescheduling_interrupts
   1628457 ±  7%     +67.8%    2731806 ±  4%  interrupts.CPU146.RES:Rescheduling_interrupts
     53098 ± 14%    +155.9%     135882 ± 35%  interrupts.CPU147.CAL:Function_call_interrupts
   1631018 ±  6%     +64.4%    2681548 ±  3%  interrupts.CPU147.RES:Rescheduling_interrupts
     51703 ± 43%    +284.0%     198542 ± 54%  interrupts.CPU148.CAL:Function_call_interrupts
   1632090 ±  7%     +71.4%    2797689 ±  5%  interrupts.CPU148.RES:Rescheduling_interrupts
     66088 ± 21%    +319.9%     277532 ± 56%  interrupts.CPU149.CAL:Function_call_interrupts
   1626229 ±  6%     +71.4%    2787534 ±  4%  interrupts.CPU149.RES:Rescheduling_interrupts
     65568 ± 73%    +208.4%     202203 ± 35%  interrupts.CPU15.CAL:Function_call_interrupts
   1567436 ±  2%     +74.0%    2727214 ±  3%  interrupts.CPU15.RES:Rescheduling_interrupts
     67724 ± 16%    +654.6%     511047 ± 84%  interrupts.CPU150.CAL:Function_call_interrupts
   1621540 ±  6%     +69.2%    2744442 ±  5%  interrupts.CPU150.RES:Rescheduling_interrupts
     54212 ± 21%    +261.1%     195770 ± 53%  interrupts.CPU151.CAL:Function_call_interrupts
   1635891 ±  6%     +66.9%    2729743 ±  5%  interrupts.CPU151.RES:Rescheduling_interrupts
     56923 ± 42%    +131.9%     132030 ± 30%  interrupts.CPU152.CAL:Function_call_interrupts
   1635561 ±  6%     +70.2%    2783394 ±  6%  interrupts.CPU152.RES:Rescheduling_interrupts
     40741 ± 55%    +816.4%     373366 ±118%  interrupts.CPU153.CAL:Function_call_interrupts
   1644945 ±  6%     +65.4%    2720699 ±  6%  interrupts.CPU153.RES:Rescheduling_interrupts
     25917 ± 35%    +748.3%     219860 ± 47%  interrupts.CPU154.CAL:Function_call_interrupts
   1643256 ±  6%     +67.6%    2754484 ±  3%  interrupts.CPU154.RES:Rescheduling_interrupts
     43656 ± 37%    +270.8%     161887 ± 51%  interrupts.CPU155.CAL:Function_call_interrupts
   1635704 ±  6%     +74.0%    2845520 ±  5%  interrupts.CPU155.RES:Rescheduling_interrupts
     42211 ± 12%    +304.6%     170804 ± 53%  interrupts.CPU156.CAL:Function_call_interrupts
   1637792 ±  7%     +78.3%    2919743 ±  9%  interrupts.CPU156.RES:Rescheduling_interrupts
     36474 ± 42%    +561.1%     241122 ± 48%  interrupts.CPU157.CAL:Function_call_interrupts
   1637206 ±  7%     +68.6%    2760749 ±  3%  interrupts.CPU157.RES:Rescheduling_interrupts
     35370 ± 40%    +219.6%     113045 ± 33%  interrupts.CPU158.CAL:Function_call_interrupts
   1637530 ±  6%     +65.9%    2717418 ±  3%  interrupts.CPU158.RES:Rescheduling_interrupts
     48121 ± 44%    +112.9%     102429 ± 11%  interrupts.CPU159.CAL:Function_call_interrupts
   1631427 ±  6%     +72.1%    2808239 ±  4%  interrupts.CPU159.RES:Rescheduling_interrupts
     32764 ± 32%   +1121.6%     400239 ±113%  interrupts.CPU16.CAL:Function_call_interrupts
   1558468 ±  2%     +71.0%    2664248 ±  2%  interrupts.CPU16.RES:Rescheduling_interrupts
     40380 ± 20%    +501.2%     242765 ± 49%  interrupts.CPU160.CAL:Function_call_interrupts
   1630714 ±  6%     +72.3%    2809922 ±  4%  interrupts.CPU160.RES:Rescheduling_interrupts
     34972 ± 46%    +186.5%     100206 ± 10%  interrupts.CPU161.CAL:Function_call_interrupts
   1631291 ±  6%     +71.2%    2792531 ±  5%  interrupts.CPU161.RES:Rescheduling_interrupts
     49790 ± 82%    +373.6%     235801 ± 58%  interrupts.CPU162.CAL:Function_call_interrupts
   1626980 ±  7%     +77.0%    2879835 ±  7%  interrupts.CPU162.RES:Rescheduling_interrupts
     37292 ± 41%    +143.6%      90840 ±  5%  interrupts.CPU163.CAL:Function_call_interrupts
   1640060 ±  6%     +62.9%    2672456 ±  5%  interrupts.CPU163.RES:Rescheduling_interrupts
     28255 ± 23%    +740.6%     237507 ±102%  interrupts.CPU164.CAL:Function_call_interrupts
   1639697 ±  6%     +67.6%    2747825 ±  3%  interrupts.CPU164.RES:Rescheduling_interrupts
     22536 ± 45%    +420.7%     117340 ± 19%  interrupts.CPU165.CAL:Function_call_interrupts
   1641676 ±  6%     +69.8%    2787394 ±  4%  interrupts.CPU165.RES:Rescheduling_interrupts
     17382 ± 25%    +734.1%     144994 ± 30%  interrupts.CPU166.CAL:Function_call_interrupts
   1643083 ±  6%     +66.4%    2733867 ±  4%  interrupts.CPU166.RES:Rescheduling_interrupts
     15869 ± 13%    +760.6%     136581 ± 47%  interrupts.CPU167.CAL:Function_call_interrupts
   1649069 ±  6%     +67.5%    2761766 ±  5%  interrupts.CPU167.RES:Rescheduling_interrupts
   1281843 ±  2%    +109.0%    2679156 ±  3%  interrupts.CPU168.RES:Rescheduling_interrupts
   1280224          +105.5%    2630742        interrupts.CPU169.RES:Rescheduling_interrupts
     51232 ± 48%    +207.8%     157679 ± 25%  interrupts.CPU17.CAL:Function_call_interrupts
   1559184           +74.4%    2719017 ±  3%  interrupts.CPU17.RES:Rescheduling_interrupts
    169671 ± 47%    +157.4%     436734 ± 41%  interrupts.CPU170.CAL:Function_call_interrupts
   1275612 ±  2%    +110.5%    2685203 ±  3%  interrupts.CPU170.RES:Rescheduling_interrupts
     97174 ± 26%    +486.8%     570205 ± 47%  interrupts.CPU171.CAL:Function_call_interrupts
   1290810          +108.2%    2687727 ±  3%  interrupts.CPU171.RES:Rescheduling_interrupts
    106669 ± 25%    +527.9%     669780 ± 41%  interrupts.CPU172.CAL:Function_call_interrupts
   1288447          +111.6%    2726608 ±  4%  interrupts.CPU172.RES:Rescheduling_interrupts
   1263014          +115.7%    2724821 ±  2%  interrupts.CPU173.RES:Rescheduling_interrupts
   1276082          +113.5%    2724465 ±  5%  interrupts.CPU174.RES:Rescheduling_interrupts
    138853 ± 39%    +339.0%     609594 ± 70%  interrupts.CPU175.CAL:Function_call_interrupts
   1277915 ±  2%    +110.5%    2690260 ±  2%  interrupts.CPU175.RES:Rescheduling_interrupts
     21539 ± 30%    +908.4%     217198 ± 43%  interrupts.CPU176.CAL:Function_call_interrupts
   1324601          +102.0%    2675209 ±  3%  interrupts.CPU176.RES:Rescheduling_interrupts
     29124 ± 52%    +580.3%     198148 ± 60%  interrupts.CPU177.CAL:Function_call_interrupts
   1317494          +104.9%    2699725 ±  2%  interrupts.CPU177.RES:Rescheduling_interrupts
     22946 ± 32%    +679.2%     178801 ± 86%  interrupts.CPU178.CAL:Function_call_interrupts
   1316930          +104.1%    2687575 ±  3%  interrupts.CPU178.RES:Rescheduling_interrupts
     25435 ± 51%   +2476.8%     655425 ± 53%  interrupts.CPU179.CAL:Function_call_interrupts
   1316656 ±  2%    +102.1%    2660994 ±  4%  interrupts.CPU179.RES:Rescheduling_interrupts
     35201 ± 31%    +581.2%     239793 ± 49%  interrupts.CPU18.CAL:Function_call_interrupts
   1576838 ±  2%     +72.7%    2722559 ±  4%  interrupts.CPU18.RES:Rescheduling_interrupts
     20621 ± 17%    +930.2%     212437 ± 46%  interrupts.CPU180.CAL:Function_call_interrupts
   1331612          +104.7%    2725346 ±  2%  interrupts.CPU180.RES:Rescheduling_interrupts
     27217 ± 58%    +815.5%     249174 ± 65%  interrupts.CPU181.CAL:Function_call_interrupts
   1322521          +102.5%    2677933 ±  2%  interrupts.CPU181.RES:Rescheduling_interrupts
     24084 ± 35%    +726.1%     198975 ± 83%  interrupts.CPU182.CAL:Function_call_interrupts
   1321894          +106.8%    2734059 ±  6%  interrupts.CPU182.RES:Rescheduling_interrupts
     20537 ± 51%   +1603.5%     349855 ± 74%  interrupts.CPU183.CAL:Function_call_interrupts
   1319355          +103.4%    2683610 ±  7%  interrupts.CPU183.RES:Rescheduling_interrupts
     50223 ± 88%    +343.6%     222783 ± 76%  interrupts.CPU184.CAL:Function_call_interrupts
   1297454          +108.6%    2707093 ±  5%  interrupts.CPU184.RES:Rescheduling_interrupts
     18052 ±  7%   +2377.9%     447321 ± 83%  interrupts.CPU185.CAL:Function_call_interrupts
   1316279           +94.4%    2558343 ±  6%  interrupts.CPU185.RES:Rescheduling_interrupts
     17291 ±  6%   +1937.7%     352356 ± 48%  interrupts.CPU186.CAL:Function_call_interrupts
   1322307           +98.2%    2620348 ±  5%  interrupts.CPU186.RES:Rescheduling_interrupts
     38595 ± 87%    +249.5%     134909 ± 19%  interrupts.CPU187.CAL:Function_call_interrupts
   1316431          +111.8%    2788380 ±  5%  interrupts.CPU187.RES:Rescheduling_interrupts
     57106 ± 43%    +295.3%     225760 ± 80%  interrupts.CPU188.CAL:Function_call_interrupts
   1307952          +103.1%    2655976 ±  3%  interrupts.CPU188.RES:Rescheduling_interrupts
     32244 ± 50%    +717.6%     263629 ± 73%  interrupts.CPU189.CAL:Function_call_interrupts
   1315554          +100.8%    2641922 ±  5%  interrupts.CPU189.RES:Rescheduling_interrupts
     67998 ±114%    +192.0%     198547 ± 47%  interrupts.CPU19.CAL:Function_call_interrupts
   1573581 ±  2%     +70.9%    2689524 ±  3%  interrupts.CPU19.RES:Rescheduling_interrupts
     37387 ± 62%    +713.4%     304093 ±121%  interrupts.CPU190.CAL:Function_call_interrupts
   1305475          +103.6%    2658062 ±  3%  interrupts.CPU190.RES:Rescheduling_interrupts
     53140 ± 38%    +262.1%     192419 ± 66%  interrupts.CPU191.CAL:Function_call_interrupts
   1247661          +111.7%    2641063 ±  4%  interrupts.CPU191.RES:Rescheduling_interrupts
     16795 ± 15%   +2485.6%     434256 ±131%  interrupts.CPU2.CAL:Function_call_interrupts
   1594055 ±  2%     +67.8%    2674488 ±  6%  interrupts.CPU2.RES:Rescheduling_interrupts
     37236 ± 43%    +274.4%     139408 ± 40%  interrupts.CPU20.CAL:Function_call_interrupts
   1579120 ±  3%     +71.5%    2707538 ±  2%  interrupts.CPU20.RES:Rescheduling_interrupts
     28176 ± 22%   +1194.0%     364597 ±127%  interrupts.CPU21.CAL:Function_call_interrupts
   1572352 ±  2%     +69.3%    2662650 ±  4%  interrupts.CPU21.RES:Rescheduling_interrupts
     39452 ± 47%    +602.9%     277294 ± 35%  interrupts.CPU22.CAL:Function_call_interrupts
   1554937 ±  2%     +74.3%    2709852 ±  3%  interrupts.CPU22.RES:Rescheduling_interrupts
     51612 ± 35%    +233.0%     171846 ± 23%  interrupts.CPU23.CAL:Function_call_interrupts
   1574750 ±  2%     +71.4%    2699019 ±  4%  interrupts.CPU23.RES:Rescheduling_interrupts
     32795 ± 66%    +214.7%     103209 ± 20%  interrupts.CPU24.CAL:Function_call_interrupts
   1650200 ±  6%     +62.5%    2681029 ±  3%  interrupts.CPU24.RES:Rescheduling_interrupts
     53359 ± 44%    +141.1%     128676 ± 10%  interrupts.CPU25.CAL:Function_call_interrupts
   1633278 ±  5%     +70.5%    2784665 ±  2%  interrupts.CPU25.RES:Rescheduling_interrupts
     34027 ± 22%    +310.6%     139731 ± 30%  interrupts.CPU26.CAL:Function_call_interrupts
   1635000 ±  6%     +67.0%    2730460 ±  3%  interrupts.CPU26.RES:Rescheduling_interrupts
   1617723 ±  6%     +63.7%    2647511 ±  2%  interrupts.CPU27.RES:Rescheduling_interrupts
     45272 ± 34%    +396.8%     224906 ± 99%  interrupts.CPU28.CAL:Function_call_interrupts
   1630635 ±  5%     +68.9%    2754025 ±  3%  interrupts.CPU28.RES:Rescheduling_interrupts
     29538 ± 17%    +490.9%     174539 ± 46%  interrupts.CPU29.CAL:Function_call_interrupts
   1633066 ±  5%     +63.2%    2665810 ±  3%  interrupts.CPU29.RES:Rescheduling_interrupts
     21892 ± 51%    +816.1%     200560 ± 61%  interrupts.CPU3.CAL:Function_call_interrupts
   1593146 ±  2%     +68.1%    2678612 ±  4%  interrupts.CPU3.RES:Rescheduling_interrupts
     70636 ± 46%     +86.5%     131752 ± 36%  interrupts.CPU30.CAL:Function_call_interrupts
   1625160 ±  5%     +77.4%    2882653 ±  9%  interrupts.CPU30.RES:Rescheduling_interrupts
     50126 ± 43%   +1844.3%     974582 ±134%  interrupts.CPU31.CAL:Function_call_interrupts
   1641072 ±  5%     +63.9%    2688933 ±  6%  interrupts.CPU31.RES:Rescheduling_interrupts
     39451 ± 25%    +254.6%     139903 ± 30%  interrupts.CPU32.CAL:Function_call_interrupts
   1636346 ±  6%     +68.3%    2754333 ±  3%  interrupts.CPU32.RES:Rescheduling_interrupts
     28515 ± 12%    +313.8%     117994 ± 26%  interrupts.CPU33.CAL:Function_call_interrupts
   1636100 ±  5%     +69.2%    2767497 ±  4%  interrupts.CPU33.RES:Rescheduling_interrupts
     49460 ± 35%    +749.4%     420127 ± 55%  interrupts.CPU34.CAL:Function_call_interrupts
   1617823 ±  6%     +73.9%    2813725 ±  7%  interrupts.CPU34.RES:Rescheduling_interrupts
     42970 ± 34%    +345.1%     191278 ± 74%  interrupts.CPU35.CAL:Function_call_interrupts
   1637737 ±  5%     +67.6%    2744100 ±  3%  interrupts.CPU35.RES:Rescheduling_interrupts
     36264 ± 48%    +277.6%     136920 ± 41%  interrupts.CPU36.CAL:Function_call_interrupts
   1644179 ±  5%     +66.9%    2744438 ±  4%  interrupts.CPU36.RES:Rescheduling_interrupts
     37013 ± 41%    +190.7%     107583 ± 18%  interrupts.CPU37.CAL:Function_call_interrupts
   1637646 ±  5%     +66.6%    2728878 ±  5%  interrupts.CPU37.RES:Rescheduling_interrupts
     30988 ± 37%    +287.7%     120146 ± 23%  interrupts.CPU38.CAL:Function_call_interrupts
   1633497 ±  5%     +64.0%    2678421 ±  4%  interrupts.CPU38.RES:Rescheduling_interrupts
     54684 ± 25%    +146.2%     134612 ± 30%  interrupts.CPU39.CAL:Function_call_interrupts
   1627521 ±  5%     +66.2%    2704610 ±  3%  interrupts.CPU39.RES:Rescheduling_interrupts
     24790 ± 40%    +705.4%     199668 ± 40%  interrupts.CPU4.CAL:Function_call_interrupts
   1576599 ±  2%     +78.6%    2815880 ± 10%  interrupts.CPU4.RES:Rescheduling_interrupts
   1628665 ±  5%     +75.8%    2862895 ±  8%  interrupts.CPU40.RES:Rescheduling_interrupts
     36650 ± 38%    +488.2%     215559 ±101%  interrupts.CPU41.CAL:Function_call_interrupts
   1637966 ±  5%     +64.7%    2698163 ±  3%  interrupts.CPU41.RES:Rescheduling_interrupts
     40300 ± 30%    +155.6%     102992 ± 11%  interrupts.CPU42.CAL:Function_call_interrupts
   1644487 ±  6%     +63.7%    2692491 ±  4%  interrupts.CPU42.RES:Rescheduling_interrupts
     28057 ± 32%    +561.2%     185519 ± 45%  interrupts.CPU43.CAL:Function_call_interrupts
   1641312 ±  6%     +66.6%    2733705 ±  3%  interrupts.CPU43.RES:Rescheduling_interrupts
     43418 ± 55%    +375.4%     206431 ± 86%  interrupts.CPU44.CAL:Function_call_interrupts
   1618052 ±  4%     +77.8%    2877115 ±  7%  interrupts.CPU44.RES:Rescheduling_interrupts
     38126 ± 48%    +172.4%     103839 ± 17%  interrupts.CPU45.CAL:Function_call_interrupts
   1635412 ±  5%     +65.8%    2711268 ±  4%  interrupts.CPU45.RES:Rescheduling_interrupts
     52804 ± 45%    +438.9%     284550 ±114%  interrupts.CPU46.CAL:Function_call_interrupts
   1627444 ±  6%     +71.5%    2791234 ±  4%  interrupts.CPU46.RES:Rescheduling_interrupts
     34251 ± 26%   +3047.6%    1078095 ±139%  interrupts.CPU47.CAL:Function_call_interrupts
   1634702 ±  6%     +64.2%    2683811 ±  2%  interrupts.CPU47.RES:Rescheduling_interrupts
     50563 ±107%    +242.9%     173392 ± 54%  interrupts.CPU48.CAL:Function_call_interrupts
   1646643 ±  6%     +64.3%    2705688 ±  4%  interrupts.CPU48.RES:Rescheduling_interrupts
      8712           -39.1%       5310 ± 35%  interrupts.CPU49.NMI:Non-maskable_interrupts
      8712           -39.1%       5310 ± 35%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
   1629863 ±  7%     +71.6%    2797619 ±  4%  interrupts.CPU49.RES:Rescheduling_interrupts
     15057 ± 12%   +1512.2%     242747 ± 46%  interrupts.CPU5.CAL:Function_call_interrupts
   1578515 ±  2%     +72.5%    2723222 ±  4%  interrupts.CPU5.RES:Rescheduling_interrupts
     34931 ± 57%    +572.4%     234893 ±104%  interrupts.CPU50.CAL:Function_call_interrupts
   1645373 ±  6%     +64.4%    2705771 ±  4%  interrupts.CPU50.RES:Rescheduling_interrupts
   1632822 ±  7%     +65.7%    2705838        interrupts.CPU51.RES:Rescheduling_interrupts
     46597 ± 75%    +665.1%     356539 ±126%  interrupts.CPU52.CAL:Function_call_interrupts
   1634317 ±  6%     +69.6%    2771193 ±  6%  interrupts.CPU52.RES:Rescheduling_interrupts
     29197 ± 32%    +744.0%     246437 ±110%  interrupts.CPU53.CAL:Function_call_interrupts
   1633076 ±  6%     +67.8%    2740665 ±  5%  interrupts.CPU53.RES:Rescheduling_interrupts
   1627028 ±  6%     +64.3%    2672670 ±  5%  interrupts.CPU54.RES:Rescheduling_interrupts
     48011 ± 71%    +440.4%     259465 ±100%  interrupts.CPU55.CAL:Function_call_interrupts
   1630706 ±  6%     +68.8%    2753329 ±  6%  interrupts.CPU55.RES:Rescheduling_interrupts
     47552 ± 69%    +603.9%     334709 ±109%  interrupts.CPU56.CAL:Function_call_interrupts
   1634017 ±  6%     +68.6%    2754559 ±  4%  interrupts.CPU56.RES:Rescheduling_interrupts
     45755 ± 46%    +368.5%     214385 ± 92%  interrupts.CPU57.CAL:Function_call_interrupts
   1645642 ±  5%     +65.1%    2716631 ±  3%  interrupts.CPU57.RES:Rescheduling_interrupts
   1626280 ±  7%     +65.3%    2687459 ±  3%  interrupts.CPU58.RES:Rescheduling_interrupts
   1612122 ±  6%     +74.5%    2812929 ±  4%  interrupts.CPU59.RES:Rescheduling_interrupts
     22594 ± 49%    +938.1%     234553 ± 97%  interrupts.CPU6.CAL:Function_call_interrupts
   1579837 ±  2%     +74.5%    2756807 ±  5%  interrupts.CPU6.RES:Rescheduling_interrupts
     38659 ± 61%    +288.1%     150029 ± 55%  interrupts.CPU60.CAL:Function_call_interrupts
   1636580 ±  6%     +70.5%    2790446 ±  5%  interrupts.CPU60.RES:Rescheduling_interrupts
     63587 ± 65%    +186.3%     182048 ± 42%  interrupts.CPU61.CAL:Function_call_interrupts
      7613 ± 24%     -30.3%       5306 ± 34%  interrupts.CPU61.NMI:Non-maskable_interrupts
      7613 ± 24%     -30.3%       5306 ± 34%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
   1633298 ±  6%     +73.1%    2826576 ±  6%  interrupts.CPU61.RES:Rescheduling_interrupts
     58419 ± 11%    +311.1%     240143 ± 88%  interrupts.CPU62.CAL:Function_call_interrupts
   1627358 ±  6%     +69.2%    2753650 ±  3%  interrupts.CPU62.RES:Rescheduling_interrupts
   1616150 ±  6%     +73.2%    2798751 ±  3%  interrupts.CPU63.RES:Rescheduling_interrupts
   1620897 ±  6%     +69.6%    2749634 ±  5%  interrupts.CPU64.RES:Rescheduling_interrupts
     72292 ± 16%    +288.7%     281002 ± 64%  interrupts.CPU65.CAL:Function_call_interrupts
   1625525 ±  6%     +72.0%    2796028 ±  4%  interrupts.CPU65.RES:Rescheduling_interrupts
     59368 ± 61%    +374.4%     281670 ± 84%  interrupts.CPU66.CAL:Function_call_interrupts
   1624883 ±  7%     +67.9%    2728039 ±  4%  interrupts.CPU66.RES:Rescheduling_interrupts
     56989 ± 18%    +489.4%     335874 ± 93%  interrupts.CPU67.CAL:Function_call_interrupts
   1635941 ±  6%     +63.3%    2671412 ±  4%  interrupts.CPU67.RES:Rescheduling_interrupts
     73479 ± 41%    +167.4%     196457 ± 37%  interrupts.CPU68.CAL:Function_call_interrupts
   1627694 ±  6%     +69.7%    2762708 ±  4%  interrupts.CPU68.RES:Rescheduling_interrupts
     65249 ± 17%    +299.9%     260907 ± 62%  interrupts.CPU69.CAL:Function_call_interrupts
   1619730 ±  6%     +69.8%    2750842 ±  3%  interrupts.CPU69.RES:Rescheduling_interrupts
     34584 ± 54%    +315.1%     143576 ± 27%  interrupts.CPU7.CAL:Function_call_interrupts
   1566878 ±  3%     +72.6%    2704987 ±  4%  interrupts.CPU7.RES:Rescheduling_interrupts
     86864 ± 43%    +376.6%     413988 ± 67%  interrupts.CPU70.CAL:Function_call_interrupts
   1611265 ±  7%     +69.7%    2734257 ±  4%  interrupts.CPU70.RES:Rescheduling_interrupts
   1622742 ±  6%     +70.0%    2758139 ±  4%  interrupts.CPU71.RES:Rescheduling_interrupts
   1293497          +102.8%    2623072 ±  3%  interrupts.CPU72.RES:Rescheduling_interrupts
   1269580 ±  2%    +104.1%    2591344        interrupts.CPU73.RES:Rescheduling_interrupts
   1286585          +109.8%    2699191 ±  7%  interrupts.CPU74.RES:Rescheduling_interrupts
   1275894          +105.7%    2624607 ±  4%  interrupts.CPU75.RES:Rescheduling_interrupts
   1288461          +105.4%    2645887 ±  5%  interrupts.CPU76.RES:Rescheduling_interrupts
   1268852          +109.2%    2654433 ±  3%  interrupts.CPU77.RES:Rescheduling_interrupts
   1270055          +111.0%    2679593 ±  5%  interrupts.CPU78.RES:Rescheduling_interrupts
    107430 ± 23%    +237.1%     362134 ± 61%  interrupts.CPU79.CAL:Function_call_interrupts
   1283084 ±  2%    +104.5%    2624210 ±  2%  interrupts.CPU79.RES:Rescheduling_interrupts
     33309 ± 62%    +366.2%     155279 ± 61%  interrupts.CPU8.CAL:Function_call_interrupts
   1581608 ±  3%     +74.8%    2764549 ±  9%  interrupts.CPU8.RES:Rescheduling_interrupts
   1269387          +108.3%    2644462 ±  4%  interrupts.CPU80.RES:Rescheduling_interrupts
   1280760          +116.8%    2776414 ±  3%  interrupts.CPU81.RES:Rescheduling_interrupts
   1258615          +117.9%    2742001 ±  4%  interrupts.CPU82.RES:Rescheduling_interrupts
   1269245          +108.8%    2649727 ±  5%  interrupts.CPU83.RES:Rescheduling_interrupts
   1273226          +112.2%    2701404 ±  2%  interrupts.CPU84.RES:Rescheduling_interrupts
   1270780          +109.3%    2659259 ±  3%  interrupts.CPU85.RES:Rescheduling_interrupts
   1247419          +114.7%    2678107 ±  5%  interrupts.CPU86.RES:Rescheduling_interrupts
   1274443          +110.5%    2682986 ±  8%  interrupts.CPU87.RES:Rescheduling_interrupts
   1272993 ±  2%    +109.3%    2664804 ±  5%  interrupts.CPU88.RES:Rescheduling_interrupts
   1256356          +112.3%    2667522 ±  9%  interrupts.CPU89.RES:Rescheduling_interrupts
     30589 ± 37%    +500.6%     183705 ± 58%  interrupts.CPU9.CAL:Function_call_interrupts
   1578766 ±  2%     +68.0%    2652415 ±  5%  interrupts.CPU9.RES:Rescheduling_interrupts
   1268068          +105.2%    2602518 ±  5%  interrupts.CPU90.RES:Rescheduling_interrupts
    127950 ± 27%    +285.9%     493790 ± 56%  interrupts.CPU91.CAL:Function_call_interrupts
   1288390          +110.3%    2709080 ±  4%  interrupts.CPU91.RES:Rescheduling_interrupts
    209436 ± 42%    +136.0%     494266 ± 35%  interrupts.CPU92.CAL:Function_call_interrupts
   1267400 ±  2%    +114.4%    2717227 ±  3%  interrupts.CPU92.RES:Rescheduling_interrupts
    135192 ± 31%    +496.4%     806290 ± 49%  interrupts.CPU93.CAL:Function_call_interrupts
   1282137          +112.0%    2718053 ±  5%  interrupts.CPU93.RES:Rescheduling_interrupts
    115622 ± 27%    +272.2%     430327 ± 51%  interrupts.CPU94.CAL:Function_call_interrupts
   1284890          +111.3%    2714683 ±  5%  interrupts.CPU94.RES:Rescheduling_interrupts
   1214298          +116.6%    2629677 ±  3%  interrupts.CPU95.RES:Rescheduling_interrupts
     65905 ± 50%    +195.8%     194917 ± 56%  interrupts.CPU96.CAL:Function_call_interrupts
   1569271           +78.7%    2804172 ±  3%  interrupts.CPU96.RES:Rescheduling_interrupts
   1566284           +74.8%    2737714 ±  5%  interrupts.CPU97.RES:Rescheduling_interrupts
     26112 ± 29%    +589.9%     180143 ± 62%  interrupts.CPU98.CAL:Function_call_interrupts
   1582854 ±  2%     +67.0%    2643026 ±  3%  interrupts.CPU98.RES:Rescheduling_interrupts
     41178 ± 45%    +304.3%     166501 ± 71%  interrupts.CPU99.CAL:Function_call_interrupts
   1577451           +69.0%    2665471 ±  4%  interrupts.CPU99.RES:Rescheduling_interrupts
    160.25 ±  8%    +125.1%     360.75 ±  2%  interrupts.IWI:IRQ_work_interrupts
 2.944e+08           +77.6%  5.229e+08 ±  3%  interrupts.RES:Rescheduling_interrupts
     48.38           -29.4       18.98 ±  3%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     66.86           -20.5       46.39        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     67.21           -20.3       46.89        perf-profile.calltrace.cycles-pp.secondary_startup_64
     66.91           -20.3       46.59        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     66.92           -20.3       46.63        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      1.64 ±  2%      -1.4        0.29 ±101%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     13.61            -0.6       13.00        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.72 ±  2%      -0.5        0.26 ±100%  perf-profile.calltrace.cycles-pp.file_update_time.pipe_write.new_sync_write.vfs_write.ksys_write
      1.05            -0.4        0.69        perf-profile.calltrace.cycles-pp.update_rq_clock.__sched_text_start.schedule.pipe_read.new_sync_read
      0.89            -0.3        0.59 ±  3%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.98            -0.1        0.84 ±  2%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.70            -0.1        0.56        perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.87            -0.0        0.83        perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.new_sync_read.vfs_read.ksys_read
      0.89 ±  2%      +0.1        0.98        perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
      0.92            +0.4        1.33        perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     12.34            +0.4       12.75        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.5        0.52        perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.5        0.52        perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.dequeue_entity.dequeue_task_fair.__sched_text_start
      0.00            +0.5        0.55 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      0.00            +0.6        0.55 ±  2%  perf-profile.calltrace.cycles-pp.put_prev_task_fair.__sched_text_start.schedule.pipe_read.new_sync_read
      0.00            +0.6        0.56        perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__sched_text_start.schedule.pipe_read
      0.00            +0.6        0.57 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read
      0.00            +0.6        0.59 ±  2%  perf-profile.calltrace.cycles-pp.finish_task_switch.__sched_text_start.schedule.pipe_read.new_sync_read
      0.00            +0.6        0.59        perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.59        perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.7        0.66 ±  3%  perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.70        perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +0.7        0.70 ±  5%  perf-profile.calltrace.cycles-pp.hrtimer_next_event_without.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.00            +0.7        0.73        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule.pipe_read.new_sync_read
      0.78            +0.7        1.52 ±  2%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read
      0.14 ±173%      +0.9        0.99 ±  2%  perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle
      0.00            +0.9        0.91 ±  2%  perf-profile.calltrace.cycles-pp.__switch_to_asm
      0.00            +0.9        0.92 ±  2%  perf-profile.calltrace.cycles-pp.__switch_to_asm.read
      0.13 ±173%      +1.0        1.09 ±  2%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.00            +1.0        0.96 ±  2%  perf-profile.calltrace.cycles-pp._find_next_bit.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length
      0.14 ±173%      +1.0        1.18 ±  3%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule
      0.71 ± 11%      +1.1        1.76 ±  2%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle
      0.00            +1.1        1.07 ±  2%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
      0.00            +1.1        1.07 ±  5%  perf-profile.calltrace.cycles-pp.timekeeping_max_deferment.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      6.07            +1.2        7.24 ±  2%  perf-profile.calltrace.cycles-pp.__switch_to
      0.90 ±  5%      +1.2        2.14 ±  2%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule
      0.00            +1.2        1.25 ±  2%  perf-profile.calltrace.cycles-pp.__switch_to.read
      0.62            +1.4        2.02 ±  2%  perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
      2.04 ±  4%      +1.4        3.47 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      0.57            +1.5        2.04 ±  4%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_idle_enter.do_idle.cpu_startup_entry.start_secondary
      2.60 ±  4%      +1.6        4.17        perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      2.73 ±  4%      +1.6        4.36        perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      2.76 ±  4%      +1.7        4.42        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.74            +1.7        2.48 ±  4%  perf-profile.calltrace.cycles-pp.tick_nohz_idle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.01 ±  8%      +1.8        2.85 ±  2%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      0.00            +2.0        1.99 ±  4%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary
      0.99            +2.1        3.07 ±  2%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.00            +2.4        2.40 ±  4%  perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.42            +2.9        8.35        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      1.86 ±  8%      +3.0        4.83        perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.pipe_read
      1.47            +3.1        4.55 ±  3%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      5.55            +3.1        8.63        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
      5.65            +3.1        8.78        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
      2.23 ±  7%      +3.2        5.43        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.pipe_read.new_sync_read
      5.92            +3.4        9.36        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
      2.03            +3.6        5.58 ±  3%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      1.95 ±  4%      +3.6        5.53 ±  2%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      2.02 ±  4%      +3.8        5.79 ±  2%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      8.70            +4.1       12.84        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.24            +4.2       12.44        perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     11.41            +4.4       15.82        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     11.64            +4.4       16.06        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     10.86            +4.4       15.30        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     10.21            +4.5       14.69        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     12.10            +5.1       17.21        perf-profile.calltrace.cycles-pp.write
      5.35 ±  2%      +5.8       11.15        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.pipe_read.new_sync_read.vfs_read
      5.59 ±  2%      +6.0       11.59        perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
      2.82            +6.3        9.09        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      9.58            +8.3       17.89        perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      9.88            +9.0       18.84        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.31           +10.1       21.46        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     11.73           +10.3       22.07        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     12.33           +10.9       23.27        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     12.59           +11.0       23.60        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     14.09           +13.3       27.35        perf-profile.calltrace.cycles-pp.read
     48.61           -29.5       19.12 ±  3%  perf-profile.children.cycles-pp.menu_select
     67.17           -20.4       46.77        perf-profile.children.cycles-pp.do_idle
     67.21           -20.3       46.89        perf-profile.children.cycles-pp.secondary_startup_64
     67.21           -20.3       46.89        perf-profile.children.cycles-pp.cpu_startup_entry
     66.92           -20.3       46.63        perf-profile.children.cycles-pp.start_secondary
      1.65 ±  2%      -1.1        0.54 ± 12%  perf-profile.children.cycles-pp.intel_idle
      1.41            -0.8        0.61 ±  3%  perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
     13.63            -0.6       13.02        perf-profile.children.cycles-pp.cpuidle_enter_state
     13.67            -0.6       13.08        perf-profile.children.cycles-pp.cpuidle_enter
      1.41            -0.5        0.90 ±  3%  perf-profile.children.cycles-pp.common_file_perm
      1.76            -0.5        1.29        perf-profile.children.cycles-pp.update_rq_clock
      1.88            -0.4        1.47 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      0.72 ±  2%      -0.2        0.51 ±  3%  perf-profile.children.cycles-pp.file_update_time
      0.17 ±  2%      -0.1        0.06 ± 20%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.31 ±  2%      -0.1        0.21 ±  3%  perf-profile.children.cycles-pp.aa_file_perm
      0.20 ±  3%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.can_stop_idle_tick
      0.23 ±  3%      -0.1        0.15 ±  2%  perf-profile.children.cycles-pp.__enqueue_entity
      0.40            -0.1        0.34 ±  3%  perf-profile.children.cycles-pp.iov_iter_init
      0.07 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.perf_swevent_event
      0.20 ±  4%      -0.0        0.16 ±  4%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.09            -0.0        0.06        perf-profile.children.cycles-pp.rw_verify_area
      0.10 ±  4%      -0.0        0.07        perf-profile.children.cycles-pp.irq_enter_rcu
      0.12            +0.0        0.13        perf-profile.children.cycles-pp._raw_spin_trylock
      0.06 ±  7%      +0.0        0.08        perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.16 ±  4%      +0.0        0.18 ±  8%  perf-profile.children.cycles-pp.child
      0.29            +0.0        0.31 ±  2%  perf-profile.children.cycles-pp.__syscall_return_slowpath
      0.09 ±  4%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.clockevents_program_event
      0.06            +0.0        0.09        perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.16 ±  4%      +0.0        0.19 ±  2%  perf-profile.children.cycles-pp.place_entity
      0.09            +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.__x64_sys_write
      0.12            +0.0        0.15 ±  5%  perf-profile.children.cycles-pp.tick_sched_handle
      0.12            +0.0        0.15 ±  5%  perf-profile.children.cycles-pp.update_process_times
      0.10 ±  5%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.41 ±  2%      +0.0        0.46 ±  4%  perf-profile.children.cycles-pp.current_time
      0.10 ±  4%      +0.0        0.15 ±  2%  perf-profile.children.cycles-pp.rb_insert_color
      0.23            +0.0        0.28 ±  2%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.14 ±  3%      +0.0        0.19 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.default_send_IPI_single_phys
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.tick_nohz_idle_got_tick
      0.43            +0.1        0.49 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.01 ±173%      +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.14 ±  3%      +0.1        0.20 ±  2%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.12 ±  6%      +0.1        0.18 ±  6%  perf-profile.children.cycles-pp.__sb_start_write
      0.05            +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.90            +0.1        0.96        perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.09            +0.1        0.15 ±  3%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_enter
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.cpuidle_reflect
      0.21 ±  2%      +0.1        0.28 ±  5%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.14            +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.__x64_sys_read
      0.06            +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.__sb_end_write
      0.10 ±  4%      +0.1        0.18 ±  3%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.36 ±  2%      +0.1        0.45        perf-profile.children.cycles-pp.call_cpuidle
      0.00            +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.perf_exclude_event
      0.06 ±  7%      +0.1        0.15 ±  2%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.21 ±  2%      +0.1        0.30 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.01 ±173%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.00            +0.1        0.10        perf-profile.children.cycles-pp.timer_clear_idle
      0.05 ±  9%      +0.1        0.16 ±  2%  perf-profile.children.cycles-pp.timestamp_truncate
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.rcu_needs_cpu
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.90            +0.1        1.01        perf-profile.children.cycles-pp.copy_page_from_iter
      0.00            +0.1        0.11        perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.init_wait_entry
      0.10 ±  5%      +0.1        0.21 ±  2%  perf-profile.children.cycles-pp.__update_idle_core
      0.07 ±  7%      +0.1        0.18 ±  5%  perf-profile.children.cycles-pp.testcase
      0.76            +0.1        0.88 ±  2%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.00            +0.1        0.12        perf-profile.children.cycles-pp.get_cpu_device
      0.05            +0.1        0.17        perf-profile.children.cycles-pp.rcu_all_qs
      0.00            +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.__list_add_valid
      0.10            +0.1        0.23 ±  3%  perf-profile.children.cycles-pp.available_idle_cpu
      0.10 ±  4%      +0.1        0.23 ±  3%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.07 ±  7%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.cpus_share_cache
      0.11            +0.1        0.25        perf-profile.children.cycles-pp.set_next_task_idle
      0.11 ±  3%      +0.1        0.25 ±  5%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.36            +0.1        0.51 ±  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.07 ±  5%      +0.1        0.22        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.35            +0.2        0.51 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.12 ±  3%      +0.2        0.27 ±  3%  perf-profile.children.cycles-pp.switch_fpu_return
      0.06 ±  9%      +0.2        0.21 ±  2%  perf-profile.children.cycles-pp.clear_buddies
      0.21 ±  3%      +0.2        0.36 ±  4%  perf-profile.children.cycles-pp.atime_needs_update
      0.15 ±  2%      +0.2        0.31        perf-profile.children.cycles-pp.pick_next_task_idle
      0.11 ±  4%      +0.2        0.26        perf-profile.children.cycles-pp.resched_curr
      0.00            +0.2        0.16 ±  2%  perf-profile.children.cycles-pp.put_prev_task_idle
      0.00            +0.2        0.16 ± 20%  perf-profile.children.cycles-pp.rb_next
      0.62            +0.2        0.79 ±  2%  perf-profile.children.cycles-pp.asm_call_on_stack
      0.00            +0.2        0.17 ±  4%  perf-profile.children.cycles-pp.menu_reflect
      0.44 ±  2%      +0.2        0.61        perf-profile.children.cycles-pp.select_idle_sibling
      0.09            +0.2        0.27        perf-profile.children.cycles-pp.rcu_eqs_exit
      0.03 ±100%      +0.2        0.20 ±  2%  perf-profile.children.cycles-pp.rb_erase
      0.28 ±  4%      +0.2        0.46        perf-profile.children.cycles-pp.account_entity_enqueue
      0.29 ±  2%      +0.2        0.48 ±  2%  perf-profile.children.cycles-pp.mutex_unlock
      0.09            +0.2        0.27        perf-profile.children.cycles-pp.finish_wait
      0.09            +0.2        0.28        perf-profile.children.cycles-pp._cond_resched
      0.17 ±  3%      +0.2        0.36        perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.20 ±  3%      +0.2        0.40        perf-profile.children.cycles-pp.copyout
      0.15 ±  2%      +0.2        0.35 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.25            +0.2        0.46 ±  2%  perf-profile.children.cycles-pp.touch_atime
      1.01            +0.2        1.21        perf-profile.children.cycles-pp.mutex_lock
      0.40 ±  2%      +0.2        0.61        perf-profile.children.cycles-pp.fsnotify_parent
      0.07 ±  7%      +0.2        0.28 ± 12%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.20 ±  2%      +0.2        0.42 ±  3%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.00            +0.2        0.23        perf-profile.children.cycles-pp.rcu_qs
      0.00            +0.2        0.24 ±  4%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.09 ±  4%      +0.2        0.33 ±  2%  perf-profile.children.cycles-pp.put_prev_entity
      0.13            +0.2        0.38 ±  2%  perf-profile.children.cycles-pp.copyin
      0.08            +0.2        0.33 ±  2%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.09            +0.2        0.34 ±  3%  perf-profile.children.cycles-pp.find_next_bit
      0.11 ±  8%      +0.3        0.36 ±  3%  perf-profile.children.cycles-pp.update_min_vruntime
      0.21 ±  3%      +0.3        0.47 ±  2%  perf-profile.children.cycles-pp.__prepare_exit_to_usermode
      0.00            +0.3        0.26 ±  3%  perf-profile.children.cycles-pp.deactivate_task
      0.38 ±  5%      +0.3        0.65 ±  2%  perf-profile.children.cycles-pp.reweight_entity
      0.58            +0.3        0.86        perf-profile.children.cycles-pp.___might_sleep
      0.38            +0.3        0.67        perf-profile.children.cycles-pp.__fdget_pos
      0.10            +0.3        0.40        perf-profile.children.cycles-pp.__might_fault
      0.16 ±  7%      +0.3        0.46 ±  3%  perf-profile.children.cycles-pp.cpuacct_charge
      0.17 ±  2%      +0.3        0.47 ±  2%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.16 ±  2%      +0.3        0.47 ±  2%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.14 ± 17%      +0.3        0.45 ±  2%  perf-profile.children.cycles-pp.account_entity_dequeue
      0.31            +0.3        0.64        perf-profile.children.cycles-pp.read_tsc
      0.17 ±  2%      +0.3        0.52        perf-profile.children.cycles-pp.__calc_delta
      0.24            +0.4        0.60        perf-profile.children.cycles-pp.check_preempt_curr
      0.20 ±  6%      +0.4        0.58 ±  2%  perf-profile.children.cycles-pp.pick_next_entity
      0.20 ±  3%      +0.4        0.58        perf-profile.children.cycles-pp.__fget_light
      0.18 ±  2%      +0.4        0.57        perf-profile.children.cycles-pp.put_prev_task_fair
      0.17 ±  2%      +0.4        0.57        perf-profile.children.cycles-pp.newidle_balance
      0.30            +0.4        0.71        perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.93            +0.4        1.33        perf-profile.children.cycles-pp.select_task_rq_fair
      0.46            +0.4        0.87        perf-profile.children.cycles-pp.___perf_sw_event
      0.00            +0.5        0.48        perf-profile.children.cycles-pp.rcu_note_context_switch
      0.21 ±  2%      +0.5        0.72 ±  5%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.61            +0.5        1.16 ±  2%  perf-profile.children.cycles-pp.native_sched_clock
      0.34 ±  8%      +0.5        0.89 ±  2%  perf-profile.children.cycles-pp.update_cfs_group
      0.63            +0.6        1.21 ±  2%  perf-profile.children.cycles-pp.sched_clock
      0.40            +0.6        0.98 ±  2%  perf-profile.children.cycles-pp.finish_task_switch
      0.27            +0.6        0.85        perf-profile.children.cycles-pp._raw_spin_lock
      0.50            +0.6        1.09 ±  2%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.40 ±  6%      +0.6        1.00 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.48            +0.6        1.12 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.66            +0.7        1.31 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.31 ±  2%      +0.7        0.98 ±  3%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.79            +0.8        1.57 ±  3%  perf-profile.children.cycles-pp.prepare_to_wait_event
      0.33 ±  6%      +0.8        1.11 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.40            +0.9        1.31 ±  3%  perf-profile.children.cycles-pp._find_next_bit
      0.14 ±  3%      +0.9        1.07 ±  5%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.23 ±  2%      +1.0        1.20 ±  2%  perf-profile.children.cycles-pp.fsnotify
      0.55            +1.0        1.52 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.46            +1.0        1.45 ±  3%  perf-profile.children.cycles-pp.__lock_text_start
      0.73 ± 11%      +1.1        1.85 ±  2%  perf-profile.children.cycles-pp.set_next_entity
      0.59 ±  2%      +1.3        1.92 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.98 ±  5%      +1.4        2.38 ±  2%  perf-profile.children.cycles-pp.update_curr
      2.07 ±  5%      +1.5        3.61 ±  2%  perf-profile.children.cycles-pp.enqueue_entity
      0.68            +1.6        2.25 ±  3%  perf-profile.children.cycles-pp.__next_timer_interrupt
      2.61 ±  4%      +1.6        4.20        perf-profile.children.cycles-pp.enqueue_task_fair
      2.74 ±  4%      +1.7        4.39        perf-profile.children.cycles-pp.activate_task
      2.78 ±  4%      +1.7        4.46        perf-profile.children.cycles-pp.ttwu_do_activate
      0.74            +1.8        2.52 ±  4%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      1.43 ± 12%      +2.0        3.44 ±  2%  perf-profile.children.cycles-pp.update_load_avg
      1.00            +2.1        3.10 ±  2%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.44            +2.2        2.64 ±  4%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      1.26 ±  6%      +2.5        3.72        perf-profile.children.cycles-pp.pick_next_task_fair
      6.64            +2.7        9.33        perf-profile.children.cycles-pp.__switch_to
      5.43            +3.0        8.39        perf-profile.children.cycles-pp.try_to_wake_up
      1.89 ±  8%      +3.0        4.91        perf-profile.children.cycles-pp.dequeue_entity
      5.55            +3.1        8.65        perf-profile.children.cycles-pp.autoremove_wake_function
      1.49            +3.1        4.61 ±  3%  perf-profile.children.cycles-pp.tick_nohz_next_event
      5.65            +3.1        8.79        perf-profile.children.cycles-pp.__wake_up_common
      2.23 ±  7%      +3.2        5.45        perf-profile.children.cycles-pp.dequeue_task_fair
      1.04            +3.3        4.37 ±  4%  perf-profile.children.cycles-pp.ktime_get
      5.92            +3.4        9.37        perf-profile.children.cycles-pp.__wake_up_common_lock
      2.04            +3.6        5.63 ±  3%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      2.03 ±  4%      +3.8        5.85 ±  2%  perf-profile.children.cycles-pp.schedule_idle
      8.71            +4.1       12.86        perf-profile.children.cycles-pp.new_sync_write
      8.25            +4.2       12.49        perf-profile.children.cycles-pp.pipe_write
     10.87            +4.4       15.31        perf-profile.children.cycles-pp.ksys_write
     10.23            +4.5       14.73        perf-profile.children.cycles-pp.vfs_write
     12.18            +5.3       17.50        perf-profile.children.cycles-pp.write
      5.60 ±  2%      +6.0       11.60        perf-profile.children.cycles-pp.schedule
      2.98            +6.3        9.24        perf-profile.children.cycles-pp.poll_idle
      9.63            +8.4       18.03        perf-profile.children.cycles-pp.pipe_read
      9.90            +9.0       18.88        perf-profile.children.cycles-pp.new_sync_read
      7.37 ±  2%      +9.6       16.92        perf-profile.children.cycles-pp.__sched_text_start
     11.32           +10.2       21.50        perf-profile.children.cycles-pp.vfs_read
     11.74           +10.3       22.09        perf-profile.children.cycles-pp.ksys_read
     14.18           +13.4       27.62        perf-profile.children.cycles-pp.read
     23.97           +15.2       39.20        perf-profile.children.cycles-pp.do_syscall_64
     24.27           +15.4       39.70        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     46.18           -33.2       12.96 ±  6%  perf-profile.self.cycles-pp.menu_select
      8.63            -5.8        2.82 ±  7%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.65 ±  2%      -1.1        0.54 ± 12%  perf-profile.self.cycles-pp.intel_idle
      1.25            -0.6        0.69        perf-profile.self.cycles-pp.update_rq_clock
      1.10            -0.4        0.69 ±  3%  perf-profile.self.cycles-pp.common_file_perm
      0.67 ±  2%      -0.3        0.32        perf-profile.self.cycles-pp.copy_page_from_iter
      0.37            -0.2        0.17 ±  4%  perf-profile.self.cycles-pp.ksys_write
      0.38 ±  2%      -0.2        0.22 ±  3%  perf-profile.self.cycles-pp.file_update_time
      0.17 ±  2%      -0.1        0.06 ± 20%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.30 ±  2%      -0.1        0.20 ±  2%  perf-profile.self.cycles-pp.aa_file_perm
      0.23 ±  3%      -0.1        0.13 ±  3%  perf-profile.self.cycles-pp.__enqueue_entity
      0.34 ±  3%      -0.1        0.24        perf-profile.self.cycles-pp.select_idle_sibling
      0.26            -0.1        0.16 ±  2%  perf-profile.self.cycles-pp.current_time
      0.20 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.can_stop_idle_tick
      0.18 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__fdget_pos
      0.40            -0.1        0.33 ±  3%  perf-profile.self.cycles-pp.iov_iter_init
      0.71            -0.1        0.65        perf-profile.self.cycles-pp.do_syscall_64
      0.20 ±  4%      -0.0        0.15 ±  2%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.09 ±  4%      -0.0        0.06        perf-profile.self.cycles-pp.rw_verify_area
      0.12            +0.0        0.13        perf-profile.self.cycles-pp._raw_spin_trylock
      0.15 ±  2%      +0.0        0.17 ±  4%  perf-profile.self.cycles-pp.place_entity
      0.21            +0.0        0.23        perf-profile.self.cycles-pp.tick_nohz_next_event
      0.09            +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.__x64_sys_write
      0.10            +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.__wake_up_common
      0.09 ±  4%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.22            +0.0        0.27 ±  3%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.10 ±  4%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.rb_insert_color
      0.15 ±  3%      +0.0        0.19        perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
      0.06            +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.ttwu_do_wakeup
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.cpuidle_enter
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.tick_nohz_idle_got_tick
      0.14 ±  6%      +0.1        0.19 ±  7%  perf-profile.self.cycles-pp.atime_needs_update
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.tick_nohz_idle_retain_tick
      0.06            +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__sb_end_write
      0.54 ±  4%      +0.1        0.60 ±  2%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.14 ±  3%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.12 ±  5%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp.__sb_start_write
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.cpuidle_reflect
      0.23            +0.1        0.29        perf-profile.self.cycles-pp.security_file_permission
      0.09            +0.1        0.15        perf-profile.self.cycles-pp.rcu_dynticks_eqs_enter
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.ttwu_do_activate
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.pick_next_task_idle
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.copyout
      0.14            +0.1        0.21 ±  3%  perf-profile.self.cycles-pp.__x64_sys_read
      0.16 ±  2%      +0.1        0.23 ±  8%  perf-profile.self.cycles-pp.__cgroup_account_cputime
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.update_ts_time_stats
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.copyin
      0.20 ±  2%      +0.1        0.27 ±  3%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.perf_exclude_event
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.touch_atime
      0.09 ±  8%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.06            +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.init_wait_entry
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.sched_clock
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.hrtimer_next_event_without
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.finish_wait
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.16 ±  2%      +0.1        0.24        perf-profile.self.cycles-pp.new_sync_write
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__might_fault
      0.05            +0.1        0.14        perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.06 ±  7%      +0.1        0.15 ±  4%  perf-profile.self.cycles-pp.autoremove_wake_function
      0.05            +0.1        0.15 ±  2%  perf-profile.self.cycles-pp.timestamp_truncate
      0.01 ±173%      +0.1        0.11        perf-profile.self.cycles-pp.rcu_eqs_exit
      0.00            +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.rcu_needs_cpu
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.10 ±  4%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.09 ±  4%      +0.1        0.19 ±  4%  perf-profile.self.cycles-pp.__update_idle_core
      0.00            +0.1        0.10        perf-profile.self.cycles-pp.sched_clock_cpu
      0.00            +0.1        0.10        perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.1        0.10        perf-profile.self.cycles-pp.timer_clear_idle
      0.06            +0.1        0.17 ±  7%  perf-profile.self.cycles-pp.testcase
      0.00            +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__list_add_valid
      0.00            +0.1        0.11 ±  3%  perf-profile.self.cycles-pp.get_cpu_device
      0.95            +0.1        1.07        perf-profile.self.cycles-pp.pipe_read
      0.10 ±  4%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.available_idle_cpu
      0.00            +0.1        0.11 ±  4%  perf-profile.self.cycles-pp._cond_resched
      0.20 ±  2%      +0.1        0.32 ±  3%  perf-profile.self.cycles-pp.schedule
      0.00            +0.1        0.12 ±  3%  perf-profile.self.cycles-pp.menu_reflect
      0.08 ±  5%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.__prepare_exit_to_usermode
      0.00            +0.1        0.12        perf-profile.self.cycles-pp.find_next_bit
      0.05            +0.1        0.17 ±  2%  perf-profile.self.cycles-pp.activate_task
      0.05 ±  8%      +0.1        0.18 ±  5%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.04 ± 57%      +0.1        0.17 ±  5%  perf-profile.self.cycles-pp.cpu_startup_entry
      0.08 ±  6%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.00            +0.1        0.13        perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.13        perf-profile.self.cycles-pp.put_prev_task_idle
      0.11 ±  6%      +0.1        0.24 ±  4%  perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.06 ±  9%      +0.1        0.19 ±  3%  perf-profile.self.cycles-pp.cpus_share_cache
      0.03 ±100%      +0.1        0.16        perf-profile.self.cycles-pp.clear_buddies
      0.05 ±  8%      +0.1        0.19 ±  3%  perf-profile.self.cycles-pp.schedule_idle
      0.33 ±  4%      +0.1        0.47        perf-profile.self.cycles-pp.dequeue_task_fair
      0.00            +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.07 ±  5%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.00            +0.1        0.15 ± 22%  perf-profile.self.cycles-pp.rb_next
      0.11 ±  4%      +0.1        0.26 ±  3%  perf-profile.self.cycles-pp.switch_fpu_return
      0.10 ±  8%      +0.2        0.26        perf-profile.self.cycles-pp.resched_curr
      0.30 ±  2%      +0.2        0.45 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.10 ±  4%      +0.2        0.26        perf-profile.self.cycles-pp.__syscall_return_slowpath
      0.26 ±  4%      +0.2        0.42 ±  2%  perf-profile.self.cycles-pp.account_entity_enqueue
      0.08            +0.2        0.24 ±  3%  perf-profile.self.cycles-pp.put_prev_task_fair
      0.39 ±  2%      +0.2        0.56        perf-profile.self.cycles-pp.fsnotify_parent
      0.28            +0.2        0.46 ±  2%  perf-profile.self.cycles-pp.mutex_unlock
      0.00            +0.2        0.18 ±  3%  perf-profile.self.cycles-pp.rb_erase
      0.10 ±  5%      +0.2        0.28        perf-profile.self.cycles-pp.check_preempt_curr
      0.16 ±  5%      +0.2        0.35 ±  2%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.15 ±  3%      +0.2        0.34 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.31 ±  2%      +0.2        0.51 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.08 ±  6%      +0.2        0.28 ±  2%  perf-profile.self.cycles-pp.put_prev_entity
      0.15 ±  4%      +0.2        0.36        perf-profile.self.cycles-pp.call_cpuidle
      0.20 ±  2%      +0.2        0.40 ±  3%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.00            +0.2        0.21 ±  5%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.14 ±  3%      +0.2        0.35 ±  3%  perf-profile.self.cycles-pp.write
      0.00            +0.2        0.22        perf-profile.self.cycles-pp.rcu_qs
      0.44            +0.2        0.66 ±  2%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.17 ±  3%      +0.2        0.40 ±  2%  perf-profile.self.cycles-pp.vfs_write
      0.00            +0.2        0.25        perf-profile.self.cycles-pp.rcu_note_context_switch
      0.11 ±  8%      +0.2        0.35 ±  3%  perf-profile.self.cycles-pp.update_min_vruntime
      0.34            +0.2        0.58 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.43            +0.3        0.68        perf-profile.self.cycles-pp.select_task_rq_fair
      0.16 ±  7%      +0.3        0.41 ±  3%  perf-profile.self.cycles-pp.pick_next_entity
      0.00            +0.3        0.25 ±  3%  perf-profile.self.cycles-pp.deactivate_task
      0.10 ±  5%      +0.3        0.35 ±  2%  perf-profile.self.cycles-pp.dequeue_entity
      0.15 ±  4%      +0.3        0.41 ±  2%  perf-profile.self.cycles-pp.read
      0.57            +0.3        0.84        perf-profile.self.cycles-pp.___might_sleep
      0.37 ±  7%      +0.3        0.65 ±  2%  perf-profile.self.cycles-pp.reweight_entity
      0.36 ±  3%      +0.3        0.64        perf-profile.self.cycles-pp.enqueue_entity
      0.13 ± 18%      +0.3        0.41 ±  2%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.15 ±  5%      +0.3        0.44 ±  4%  perf-profile.self.cycles-pp.cpuacct_charge
      0.06 ±  6%      +0.3        0.37 ±  3%  perf-profile.self.cycles-pp.tick_nohz_idle_exit
      0.18 ±  2%      +0.3        0.50 ±  2%  perf-profile.self.cycles-pp.mutex_lock
      0.30            +0.3        0.62 ±  2%  perf-profile.self.cycles-pp.read_tsc
      0.25            +0.3        0.58        perf-profile.self.cycles-pp.pipe_write
      0.18 ±  7%      +0.3        0.52        perf-profile.self.cycles-pp.set_next_entity
      0.17 ±  2%      +0.3        0.52        perf-profile.self.cycles-pp.__calc_delta
      0.25 ±  3%      +0.4        0.60 ±  2%  perf-profile.self.cycles-pp.finish_task_switch
      0.19 ±  3%      +0.4        0.55 ±  2%  perf-profile.self.cycles-pp.__fget_light
      0.37            +0.4        0.74        perf-profile.self.cycles-pp.___perf_sw_event
      0.17 ±  3%      +0.4        0.55        perf-profile.self.cycles-pp.newidle_balance
      0.12 ±  3%      +0.4        0.54 ±  4%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.16            +0.5        0.61 ±  2%  perf-profile.self.cycles-pp.new_sync_read
      0.36 ± 17%      +0.5        0.82 ±  2%  perf-profile.self.cycles-pp.update_curr
      0.59            +0.5        1.10 ±  3%  perf-profile.self.cycles-pp.native_sched_clock
      0.34 ±  7%      +0.5        0.86 ±  3%  perf-profile.self.cycles-pp.update_cfs_group
      0.26 ±  2%      +0.6        0.82        perf-profile.self.cycles-pp._raw_spin_lock
      0.24 ±  3%      +0.6        0.83 ±  3%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.36 ±  4%      +0.6        0.96 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.74 ± 19%      +0.6        1.35 ±  2%  perf-profile.self.cycles-pp.update_load_avg
      0.31 ±  2%      +0.7        0.97 ±  3%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.82            +0.7        1.51 ±  2%  perf-profile.self.cycles-pp.do_idle
      0.20 ±  2%      +0.7        0.92        perf-profile.self.cycles-pp.vfs_read
      0.31 ±  7%      +0.7        1.03 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.40            +0.9        1.29 ±  3%  perf-profile.self.cycles-pp._find_next_bit
      0.39            +0.9        1.30 ±  2%  perf-profile.self.cycles-pp.__lock_text_start
      0.14 ±  3%      +0.9        1.06 ±  5%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.53            +0.9        1.46 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.23            +0.9        1.18 ±  2%  perf-profile.self.cycles-pp.fsnotify
      0.79            +1.2        1.97 ±  2%  perf-profile.self.cycles-pp.__sched_text_start
      0.58            +1.3        1.90 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      6.62            +2.7        9.29        perf-profile.self.cycles-pp.__switch_to
      0.74            +3.0        3.72 ±  4%  perf-profile.self.cycles-pp.ktime_get
      2.74            +6.0        8.78        perf-profile.self.cycles-pp.poll_idle


                                                                                
                            will-it-scale.per_process_ops                       
                                                                                
  500000 +------------------------------------------------------------------+   
         | O O  O                                                           |   
  450000 |-+                                                                |   
  400000 |-+      O                                                         |   
         |               O O   O    O O                     O  O   O        |   
  350000 |-+                            O          O  O          O          |   
  300000 |-+                               O O          O O               O |   
         |          O O      O    O              O                   O  O   |   
  250000 |-+                                   O                            |   
  200000 |-+                                                                |   
         |                        +.+.+.                                    |   
  150000 |.+.+..+.+             ..      +..                                 |   
  100000 |-+       + .+..+.+.+.+           +.+.+.+                          |   
         |          +                                                       |   
   50000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.8.0-rc4-01449-g9216477449f33" of type "text/plain" (158454 bytes)

View attachment "job-script" of type "text/plain" (7530 bytes)

View attachment "job.yaml" of type "text/plain" (5151 bytes)

View attachment "reproduce" of type "text/plain" (347 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ