lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Mon, 23 Nov 2020 23:53:27 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Waiman Long <longman@...hat.com>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>, Will Deacon <will@...nel.org>,
        linux-kernel@...r.kernel.org, Davidlohr Bueso <dave@...olabs.net>,
        Phil Auld <pauld@...hat.com>, Waiman Long <longman@...hat.com>,
        0day robot <lkp@...el.com>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com
Subject: [locking/rwsem]  10a59003d2:  unixbench.score -25.5% regression



Greeting,

FYI, we noticed a -25.5% regression of unixbench.score due to commit:


commit: 10a59003d29fbfa855b2ef4f3534fee9bdf4e575 ("[PATCH v2 5/5] locking/rwsem: Remove reader optimistic spinning")
url: https://github.com/0day-ci/linux/commits/Waiman-Long/locking-rwsem-Rework-reader-optimistic-spinning/20201121-122118
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 932f8c64d38bb08f69c8c26a2216ba0c36c6daa8

in testcase: unixbench
on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
with following parameters:

	runtime: 300s
	nr_task: 30%
	test: shell8
	cpufreq_governor: performance
	ucode: 0xde

test-description: UnixBench is the original BYTE UNIX benchmark suite aims to test performance of Unix-like system.
test-url: https://github.com/kdlucas/byte-unixbench

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------+
| testcase: change | fio-basic: fio.write_iops -29.9% regression                               |
| test machine     | 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory               |
| test parameters  | bs=4k                                                                     |
|                  | cpufreq_governor=performance                                              |
|                  | disk=1SSD                                                                 |
|                  | fs=xfs                                                                    |
|                  | ioengine=sync                                                             |
|                  | nr_task=32                                                                |
|                  | runtime=300s                                                              |
|                  | rw=randwrite                                                              |
|                  | test_size=256g                                                            |
|                  | ucode=0x4003003                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min 952.6% improvement                                |
| test machine     | 96 threads Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz with 128G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | disk=4BRD_12G                                                             |
|                  | fs=f2fs                                                                   |
|                  | load=100                                                                  |
|                  | md=RAID0                                                                  |
|                  | test=sync_disk_rw                                                         |
|                  | ucode=0x4003003                                                           |
+------------------+---------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/30%/debian-10.4-x86_64-20200603.cgz/300s/lkp-cfl-e1/shell8/unixbench/0xde

commit: 
  c9847a7f94 ("locking/rwsem: Wake up all waiting readers if RWSEM_WAKE_READ_OWNED")
  10a59003d2 ("locking/rwsem: Remove reader optimistic spinning")

c9847a7f94679e74 10a59003d29fbfa855b2ef4f353 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     21939           -25.5%      16346        unixbench.score
   1287341           -46.8%     684642 ±  2%  unixbench.time.involuntary_context_switches
     37785 ±  2%      +7.6%      40661        unixbench.time.major_page_faults
 1.054e+08           -25.4%   78563350        unixbench.time.minor_page_faults
      1337           -30.1%     934.50        unixbench.time.percent_of_cpu_this_job_got
    363.18           -36.6%     230.42        unixbench.time.system_time
    481.37           -25.2%     360.16        unixbench.time.user_time
   3528615           +89.5%    6688263        unixbench.time.voluntary_context_switches
    829330           -25.5%     617908        unixbench.workload
     40455 ± 10%     +18.6%      47991 ±  3%  meminfo.AnonHugePages
    373.42 ±  3%     +73.7%     648.78        uptime.idle
      4.16 ± 86%      -4.2        0.00        perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
      6.25 ±110%      -6.2        0.00        perf-profile.children.cycles-pp.dput
      5.95 ± 97%      -5.9        0.00        perf-profile.children.cycles-pp.free_pgtables
     11058 ±  3%     -18.9%       8972 ±  6%  slabinfo.filp.active_objs
     12400 ±  2%     -16.6%      10346 ±  6%  slabinfo.filp.num_objs
      5364 ±  3%     -39.6%       3238 ± 16%  slabinfo.task_delay_info.active_objs
      5364 ±  3%     -39.6%       3238 ± 16%  slabinfo.task_delay_info.num_objs
     13.83 ±  4%     +26.6       40.40        mpstat.cpu.all.idle%
      1.06            -0.1        0.91        mpstat.cpu.all.irq%
      0.54            +0.1        0.69 ±  7%  mpstat.cpu.all.soft%
     50.14           -14.5       35.61        mpstat.cpu.all.sys%
     34.43           -12.0       22.39        mpstat.cpu.all.usr%
     15.50 ±  3%    +167.7%      41.50        vmstat.cpu.id
     49.75           -28.1%      35.75        vmstat.cpu.sy
     33.00           -35.6%      21.25 ±  2%  vmstat.cpu.us
     21.25 ±  5%     -35.3%      13.75 ± 12%  vmstat.procs.r
    113264           +82.8%     206990        vmstat.system.cs
     37223           -14.8%      31718 ± 20%  vmstat.system.in
  50992679 ±  8%    +329.6%   2.19e+08 ±  3%  cpuidle.C1.time
   1297530 ±  6%    +230.7%    4290676 ±  5%  cpuidle.C1.usage
    190270 ± 12%     -41.6%     111170 ± 18%  cpuidle.C10.time
  21241624 ±  6%    +186.1%   60774649 ± 10%  cpuidle.C1E.time
    458079 ±  6%    +119.6%    1005897 ±  9%  cpuidle.C1E.usage
   4577780 ± 17%    +595.5%   31836464 ± 12%  cpuidle.C3.time
     62068 ± 13%    +527.9%     389751 ± 11%  cpuidle.C3.usage
  31244183 ± 25%     +68.9%   52781560 ± 11%  cpuidle.C6.time
     50282 ± 17%    +325.2%     213780 ± 13%  cpuidle.C6.usage
    259071 ±  8%    +509.9%    1580003 ±  8%  cpuidle.POLL.time
     32585 ±  7%    +633.5%     239029 ±  7%  cpuidle.POLL.usage
    522.75            +9.7%     573.25 ±  3%  proc-vmstat.nr_active_anon
      5838            +3.4%       6034        proc-vmstat.nr_kernel_stack
      1416            +5.5%       1494        proc-vmstat.nr_page_table_pages
     13041            -1.0%      12911        proc-vmstat.nr_slab_reclaimable
    522.75            +9.7%     573.25 ±  3%  proc-vmstat.nr_zone_active_anon
  71020069           -24.7%   53452045        proc-vmstat.numa_hit
  71020069           -24.7%   53452045        proc-vmstat.numa_local
    115977           -39.0%      70793        proc-vmstat.pgactivate
  75313926           -24.8%   56608462        proc-vmstat.pgalloc_normal
 1.056e+08           -25.4%   78785247        proc-vmstat.pgfault
  75305695           -24.8%   56602456        proc-vmstat.pgfree
   5357159           -28.6%    3826576        proc-vmstat.pgreuse
      3475           -24.0%       2641        proc-vmstat.thp_fault_alloc
   1474994           -25.5%    1099150        proc-vmstat.unevictable_pgs_culled
    250788           -41.8%     146078 ±  3%  softirqs.CPU0.RCU
    248710           -39.9%     149519        softirqs.CPU1.RCU
    248300           -52.5%     118065 ± 41%  softirqs.CPU10.RCU
    246447 ±  2%     -39.2%     149847        softirqs.CPU11.RCU
    250321           -52.0%     120263 ± 43%  softirqs.CPU12.RCU
    247140 ±  2%     -39.3%     150089        softirqs.CPU13.RCU
     13849 ±  2%     +10.4%      15286        softirqs.CPU13.SCHED
    250080           -50.5%     123681 ± 36%  softirqs.CPU14.RCU
    248790           -38.6%     152862 ±  2%  softirqs.CPU15.RCU
    247174           -39.6%     149226        softirqs.CPU2.RCU
    248590           -39.3%     150877        softirqs.CPU3.RCU
    250818           -39.8%     150890 ±  2%  softirqs.CPU4.RCU
    246772 ±  2%     -39.1%     150263        softirqs.CPU5.RCU
    248637           -39.0%     151575        softirqs.CPU6.RCU
    249756           -40.1%     149525 ±  3%  softirqs.CPU7.RCU
    248263           -42.8%     142057 ±  2%  softirqs.CPU8.RCU
    242793 ±  2%     -38.8%     148663        softirqs.CPU9.RCU
   3973390           -42.0%    2303491 ±  5%  softirqs.RCU
     25441           -29.8%      17865        sched_debug.cfs_rq:/.exec_clock.avg
     26174           -29.0%      18583        sched_debug.cfs_rq:/.exec_clock.max
     25149           -30.4%      17493        sched_debug.cfs_rq:/.exec_clock.min
    293.59 ±  5%     +13.5%     333.29 ± 14%  sched_debug.cfs_rq:/.exec_clock.stddev
    188.59 ± 19%     -29.2%     133.54 ± 12%  sched_debug.cfs_rq:/.load_avg.avg
     26.12 ± 23%     -48.8%      13.38 ± 50%  sched_debug.cfs_rq:/.load_avg.min
    443268           -28.2%     318361        sched_debug.cfs_rq:/.min_vruntime.avg
    489789           -21.5%     384542 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
    431839           -29.6%     304066        sched_debug.cfs_rq:/.min_vruntime.min
      0.38 ± 22%     +31.7%       0.51 ±  6%  sched_debug.cfs_rq:/.nr_running.stddev
      1.92 ± 14%     -30.1%       1.34 ± 12%  sched_debug.cfs_rq:/.nr_spread_over.avg
    905.09 ±  3%      +9.9%     994.48 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
    870.48 ±  4%      +8.5%     944.88 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
     91.52 ± 17%     +58.8%     145.33 ± 23%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     30990 ± 17%     -34.1%      20422 ± 12%  sched_debug.cpu.avg_idle.min
     11558 ± 25%     -53.5%       5376 ± 38%  sched_debug.cpu.curr->pid.avg
      0.00 ±  4%     -26.3%       0.00 ± 19%  sched_debug.cpu.next_balance.stddev
      1.10 ±  7%     -27.0%       0.80 ± 19%  sched_debug.cpu.nr_running.avg
    225114           +80.9%     407331        sched_debug.cpu.nr_switches.avg
    239349           +79.5%     429598        sched_debug.cpu.nr_switches.max
    210368           +82.6%     384043        sched_debug.cpu.nr_switches.min
      6933 ±  8%     +47.7%      10240 ±  7%  sched_debug.cpu.nr_switches.stddev
      0.20 ±182%    +276.9%       0.77 ± 16%  sched_debug.cpu.nr_uninterruptible.avg
    105.88 ± 13%    +193.6%     310.88 ± 15%  sched_debug.cpu.nr_uninterruptible.max
   -188.38          +222.9%    -608.25        sched_debug.cpu.nr_uninterruptible.min
     72.96 ± 16%    +191.3%     212.52 ±  7%  sched_debug.cpu.nr_uninterruptible.stddev
    219447           +83.3%     402333        sched_debug.cpu.sched_count.avg
    223137           +83.2%     408852        sched_debug.cpu.sched_count.max
    208437           +82.8%     381062        sched_debug.cpu.sched_count.min
      3763 ± 10%     +75.0%       6584 ± 23%  sched_debug.cpu.sched_count.stddev
     56141 ±  3%    +199.1%     167917        sched_debug.cpu.sched_goidle.avg
     57397 ±  3%    +197.7%     170892        sched_debug.cpu.sched_goidle.max
     53349 ±  2%    +197.8%     158895        sched_debug.cpu.sched_goidle.min
      1064 ± 10%    +161.3%       2782 ± 22%  sched_debug.cpu.sched_goidle.stddev
     97665           +98.5%     193817        sched_debug.cpu.ttwu_count.avg
    100103           +97.6%     197827        sched_debug.cpu.ttwu_count.max
     91752 ±  2%    +100.6%     184037        sched_debug.cpu.ttwu_count.min
      2014 ± 18%     +66.4%       3351 ± 16%  sched_debug.cpu.ttwu_count.stddev
     42150           -25.4%      31431        sched_debug.cpu.ttwu_local.avg
     42994           -25.0%      32238        sched_debug.cpu.ttwu_local.max
     39998           -26.4%      29443 ±  3%  sched_debug.cpu.ttwu_local.min
    519.50 ±139%   +1032.1%       5881 ± 75%  interrupts.132:IR-PCI-MSI.2097153-edge.eth1-TxRx-0
      1078 ±159%     -95.8%      45.75 ± 10%  interrupts.134:IR-PCI-MSI.2097155-edge.eth1-TxRx-2
     21268 ±  3%    +143.8%      51854 ±  2%  interrupts.CAL:Function_call_interrupts
      1603 ± 15%     +93.9%       3109 ±  5%  interrupts.CPU0.CAL:Function_call_interrupts
     26260           -37.4%      16446 ±  3%  interrupts.CPU0.RES:Rescheduling_interrupts
    639.00 ± 11%    +240.8%       2177 ±  5%  interrupts.CPU0.TLB:TLB_shootdowns
    519.50 ±139%   +1032.1%       5881 ± 75%  interrupts.CPU1.132:IR-PCI-MSI.2097153-edge.eth1-TxRx-0
      1237 ±  6%    +155.9%       3166 ±  5%  interrupts.CPU1.CAL:Function_call_interrupts
     26527           -36.2%      16918        interrupts.CPU1.RES:Rescheduling_interrupts
    654.50 ±  2%    +241.6%       2235 ±  2%  interrupts.CPU1.TLB:TLB_shootdowns
      1267 ±  8%    +154.5%       3225 ±  5%  interrupts.CPU10.CAL:Function_call_interrupts
     26382           -35.7%      16965 ±  2%  interrupts.CPU10.RES:Rescheduling_interrupts
    663.75 ±  4%    +238.0%       2243 ±  2%  interrupts.CPU10.TLB:TLB_shootdowns
      1434 ± 13%    +128.8%       3280 ±  3%  interrupts.CPU11.CAL:Function_call_interrupts
     26307           -34.8%      17142        interrupts.CPU11.RES:Rescheduling_interrupts
    658.00 ±  4%    +236.8%       2216 ±  2%  interrupts.CPU11.TLB:TLB_shootdowns
      1320 ±  4%    +143.8%       3218 ±  2%  interrupts.CPU12.CAL:Function_call_interrupts
     26594           -35.4%      17171        interrupts.CPU12.RES:Rescheduling_interrupts
    697.75 ±  3%    +217.7%       2217        interrupts.CPU12.TLB:TLB_shootdowns
      1332 ±  3%    +153.4%       3375 ± 13%  interrupts.CPU13.CAL:Function_call_interrupts
     26096 ±  2%     -34.8%      17024        interrupts.CPU13.RES:Rescheduling_interrupts
    690.50 ±  6%    +220.1%       2210 ±  3%  interrupts.CPU13.TLB:TLB_shootdowns
      1335 ±  6%    +144.0%       3258 ±  3%  interrupts.CPU14.CAL:Function_call_interrupts
     26453           -34.7%      17268        interrupts.CPU14.RES:Rescheduling_interrupts
    667.50 ±  6%    +234.4%       2232        interrupts.CPU14.TLB:TLB_shootdowns
      1302 ±  3%    +146.2%       3205 ±  5%  interrupts.CPU15.CAL:Function_call_interrupts
     26159           -35.1%      16980        interrupts.CPU15.RES:Rescheduling_interrupts
    694.00 ±  8%    +226.7%       2267 ±  4%  interrupts.CPU15.TLB:TLB_shootdowns
      1283 ±  5%    +142.8%       3114 ±  3%  interrupts.CPU2.CAL:Function_call_interrupts
     25846           -35.0%      16792 ±  2%  interrupts.CPU2.RES:Rescheduling_interrupts
    662.75 ±  6%    +233.2%       2208        interrupts.CPU2.TLB:TLB_shootdowns
      1078 ±159%     -95.8%      45.75 ± 10%  interrupts.CPU3.134:IR-PCI-MSI.2097155-edge.eth1-TxRx-2
      1330 ±  6%    +178.9%       3710 ± 11%  interrupts.CPU3.CAL:Function_call_interrupts
     26346           -35.9%      16877        interrupts.CPU3.RES:Rescheduling_interrupts
    678.00 ±  8%    +224.9%       2202 ±  2%  interrupts.CPU3.TLB:TLB_shootdowns
      1300 ±  3%    +144.9%       3183        interrupts.CPU4.CAL:Function_call_interrupts
     26432           -37.2%      16586        interrupts.CPU4.RES:Rescheduling_interrupts
    675.75 ±  2%    +223.3%       2184 ±  2%  interrupts.CPU4.TLB:TLB_shootdowns
      1261          +150.0%       3153 ±  4%  interrupts.CPU5.CAL:Function_call_interrupts
     26321           -36.1%      16814        interrupts.CPU5.RES:Rescheduling_interrupts
    681.75 ±  6%    +223.0%       2201 ±  2%  interrupts.CPU5.TLB:TLB_shootdowns
      1274 ±  5%    +149.3%       3176 ±  4%  interrupts.CPU6.CAL:Function_call_interrupts
     26106           -34.9%      17005        interrupts.CPU6.RES:Rescheduling_interrupts
    659.50 ±  4%    +234.6%       2207        interrupts.CPU6.TLB:TLB_shootdowns
      1293 ±  2%    +150.7%       3241 ±  3%  interrupts.CPU7.CAL:Function_call_interrupts
     26066           -36.8%      16468 ±  2%  interrupts.CPU7.RES:Rescheduling_interrupts
    678.50 ±  7%    +228.1%       2226 ±  4%  interrupts.CPU7.TLB:TLB_shootdowns
      1354 ± 12%    +132.3%       3145 ±  5%  interrupts.CPU8.CAL:Function_call_interrupts
     26259           -37.1%      16509        interrupts.CPU8.RES:Rescheduling_interrupts
    679.75 ±  7%    +221.8%       2187 ±  2%  interrupts.CPU8.TLB:TLB_shootdowns
      1338 ± 12%    +145.8%       3289 ±  2%  interrupts.CPU9.CAL:Function_call_interrupts
     25685 ±  2%     -33.4%      17114        interrupts.CPU9.RES:Rescheduling_interrupts
    657.75 ±  2%    +244.5%       2266 ±  3%  interrupts.CPU9.TLB:TLB_shootdowns
    419844           -35.7%     270085        interrupts.RES:Rescheduling_interrupts
     10738 ±  4%    +230.4%      35483        interrupts.TLB:TLB_shootdowns
     48.03            +6.7%      51.27        perf-stat.i.MPKI
 1.008e+10           -23.2%  7.737e+09        perf-stat.i.branch-instructions
      2.27            +0.0        2.30        perf-stat.i.branch-miss-rate%
 2.205e+08           -21.7%  1.726e+08        perf-stat.i.branch-misses
      4.69            +0.2        4.89        perf-stat.i.cache-miss-rate%
 1.009e+08           -13.0%   87791224        perf-stat.i.cache-misses
 2.442e+09           -18.2%  1.997e+09        perf-stat.i.cache-references
    117262           +83.3%     214963        perf-stat.i.context-switches
      1.17            -3.4%       1.13        perf-stat.i.cpi
 5.529e+10           -26.1%  4.088e+10        perf-stat.i.cpu-cycles
     18271 ±  2%    +149.1%      45523        perf-stat.i.cpu-migrations
    593.84           -12.8%     517.54 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.06 ±  2%      +0.0        0.08 ±  3%  perf-stat.i.dTLB-load-miss-rate%
 1.253e+10           -23.5%  9.585e+09        perf-stat.i.dTLB-loads
      0.05            +0.0        0.05        perf-stat.i.dTLB-store-miss-rate%
   3638057           -19.1%    2941687        perf-stat.i.dTLB-store-misses
 7.236e+09           -23.7%   5.52e+09        perf-stat.i.dTLB-stores
     57.59            -3.4       54.19        perf-stat.i.iTLB-load-miss-rate%
  10438031           -19.4%    8408188        perf-stat.i.iTLB-load-misses
   7925579            -6.8%    7388686        perf-stat.i.iTLB-loads
 4.914e+10           -23.4%  3.763e+10        perf-stat.i.instructions
      5390            -4.2%       5162        perf-stat.i.instructions-per-iTLB-miss
      0.88            +3.5%       0.91        perf-stat.i.ipc
    591.27 ±  2%      +7.6%     636.05        perf-stat.i.major-faults
      3.46           -26.1%       2.55        perf-stat.i.metric.GHz
      2021           -23.1%       1555        perf-stat.i.metric.M/sec
   1619167           -25.4%    1207381        perf-stat.i.minor-faults
      0.00 ± 15%      +0.0        0.01 ±115%  perf-stat.i.node-load-miss-rate%
     47.02 ± 19%    +455.1%     260.98 ±117%  perf-stat.i.node-load-misses
   5399939            -9.5%    4887416        perf-stat.i.node-loads
     46.42 ± 20%    +472.0%     265.53 ±117%  perf-stat.i.node-store-misses
  32686027           -14.7%   27888853        perf-stat.i.node-stores
   1619758           -25.4%    1208017        perf-stat.i.page-faults
     49.70            +6.8%      53.06        perf-stat.overall.MPKI
      2.19            +0.0        2.23        perf-stat.overall.branch-miss-rate%
      4.13            +0.3        4.40        perf-stat.overall.cache-miss-rate%
      1.13            -3.4%       1.09        perf-stat.overall.cpi
    547.96           -15.0%     465.71        perf-stat.overall.cycles-between-cache-misses
      0.06 ±  2%      +0.0        0.08 ±  3%  perf-stat.overall.dTLB-load-miss-rate%
      0.05            +0.0        0.05        perf-stat.overall.dTLB-store-miss-rate%
     56.84            -3.6       53.23        perf-stat.overall.iTLB-load-miss-rate%
      4708            -4.9%       4475        perf-stat.overall.instructions-per-iTLB-miss
      0.89            +3.6%       0.92        perf-stat.overall.ipc
      0.00 ± 19%      +0.0        0.01 ±117%  perf-stat.overall.node-load-miss-rate%
      0.00 ± 19%      +0.0        0.00 ±117%  perf-stat.overall.node-store-miss-rate%
   3739839            +3.3%    3861837        perf-stat.overall.path-length
 9.918e+09           -23.3%  7.611e+09        perf-stat.ps.branch-instructions
  2.17e+08           -21.8%  1.698e+08        perf-stat.ps.branch-misses
  99277053           -13.0%   86352292        perf-stat.ps.cache-misses
 2.403e+09           -18.3%  1.964e+09        perf-stat.ps.cache-references
    115371           +83.3%     211429        perf-stat.ps.context-switches
  5.44e+10           -26.1%  4.021e+10        perf-stat.ps.cpu-cycles
     17977 ±  2%    +149.1%      44774        perf-stat.ps.cpu-migrations
 1.233e+10           -23.5%  9.428e+09        perf-stat.ps.dTLB-loads
   3579226           -19.2%    2893270        perf-stat.ps.dTLB-store-misses
 7.119e+09           -23.7%  5.429e+09        perf-stat.ps.dTLB-stores
  10269385           -19.5%    8270074        perf-stat.ps.iTLB-load-misses
   7797424            -6.8%    7267132        perf-stat.ps.iTLB-loads
 4.835e+10           -23.4%  3.702e+10        perf-stat.ps.instructions
    581.70 ±  2%      +7.5%     625.60        perf-stat.ps.major-faults
   1592966           -25.5%    1187499        perf-stat.ps.minor-faults
     46.26 ± 19%    +454.5%     256.51 ±117%  perf-stat.ps.node-load-misses
   5312864            -9.5%    4807129        perf-stat.ps.node-loads
     45.67 ± 20%    +471.5%     260.99 ±117%  perf-stat.ps.node-store-misses
  32157625           -14.7%   27429683        perf-stat.ps.node-stores
   1593547           -25.4%    1188124        perf-stat.ps.page-faults
 3.102e+12           -23.1%  2.386e+12        perf-stat.total.instructions


                                                                                
                             unixbench.time.user_time                           
                                                                                
  500 +---------------------------------------------------------------------+   
  480 |..+    +..+.+.    +.              +.   +..+..+.+..+..+..+.+..+       |   
      |                                                                     |   
  460 |-+                                                                   |   
  440 |-+                                                                   |   
      |                                                                     |   
  420 |-+                                                                   |   
  400 |-+                                                                   |   
  380 |-+                                                                   |   
      |                                                                     |   
  360 |-+          O  O  O  O O  O  O O  O  O O  O  O O  O  O  O O  O  O O  |   
  340 |-+                                                                   |   
      |    O                                                                |   
  320 |-+O    O  O                                                          |   
  300 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            unixbench.time.system_time                          
                                                                                
  380 +---------------------------------------------------------------------+   
      |       +.            +.+..+.   +.    +    +..+.+..+..  .+.+..+       |   
  360 |-+                                                   +.              |   
  340 |-+                                                                   |   
      |                                                                     |   
  320 |-+                                                                   |   
  300 |-+                                                                   |   
      |                                                                     |   
  280 |-+                                                                   |   
  260 |-+                                                                   |   
      |                                                                     |   
  240 |-+             O  O       O                                          |   
  220 |-+          O        O O     O O  O  O O  O  O O  O  O  O O  O  O O  |   
      |                                                                     |   
  200 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    unixbench.time.percent_of_cpu_this_job_got                  
                                                                                
  1400 +--------------------------------------------------------------------+   
       |..+.+..+..+.+..+..+.+..+..+.+..+..+.+..+.+..+..+.+..  .+.+..+       |   
  1300 |-+                                                  +.              |   
       |                                                                    |   
       |                                                                    |   
  1200 |-+                                                                  |   
       |                                                                    |   
  1100 |-+                                                                  |   
       |                                                                    |   
  1000 |-+                                                                  |   
       |                                                                    |   
       |            O  O  O O  O  O O  O  O O  O O  O  O O  O  O O  O  O O  |   
   900 |-+                                                                  |   
       |  O O  O  O                                                         |   
   800 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           unixbench.time.minor_page_faults                     
                                                                                
   1.1e+08 +----------------------------------------------------------------+   
           |..+.+..+.+..+    +    +.+..+.+..+.+..+.+..+.+..+.+..+.+..+      |   
  1.05e+08 |-+                                                              |   
     1e+08 |-+                                                              |   
           |                                                                |   
   9.5e+07 |-+                                                              |   
     9e+07 |-+                                                              |   
           |                                                                |   
   8.5e+07 |-+                                                              |   
     8e+07 |-+                                                              |   
           |            O O  O O  O O  O O  O O  O O  O O  O O  O O  O O  O |   
   7.5e+07 |-+                                                              |   
     7e+07 |-+O O  O                                                        |   
           |         O                                                      |   
   6.5e+07 +----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       unixbench.time.voluntary_context_switches                
                                                                                
    7e+06 +-----------------------------------------------------------------+   
          |            O O  O O  O O  O O  O  O O  O O  O O  O O  O O  O O  |   
  6.5e+06 |-+                                                               |   
    6e+06 |-+                                                               |   
          |  O O  O O                                                       |   
  5.5e+06 |-+                                                               |   
          |                                                                 |   
    5e+06 |-+                                                               |   
          |                                                                 |   
  4.5e+06 |-+                                                               |   
    4e+06 |-+                                                               |   
          |                                                                 |   
  3.5e+06 |-+                                               .+.+..+.+       |   
          |                     .+.    .+..  .+.  .+.+..+.+.                |   
    3e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      unixbench.time.involuntary_context_switches               
                                                                                
  1.4e+06 +-----------------------------------------------------------------+   
          |  +.+..+.+..+.+..+.+..+.+.   +..  .+.    .+..+.+..               |   
  1.3e+06 |..                              +.   +..+         +.+..+.+       |   
          |                                                                 |   
  1.2e+06 |-+                                                               |   
  1.1e+06 |-+                                                               |   
          |                                                                 |   
    1e+06 |-+                                                               |   
          |                                                                 |   
   900000 |-+                                                               |   
   800000 |-+                                                               |   
          |  O O  O                                                         |   
   700000 |-+       O  O    O O  O O  O    O         O  O O         O       |   
          |              O              O     O O  O         O O  O    O O  |   
   600000 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   unixbench.score                              
                                                                                
  23000 +-------------------------------------------------------------------+   
        |..+.+..+.+..+..+.+..+..+.+..+.+..+..+.+..+.+..+..+.                |   
  22000 |-+                                                 +..+..+.+       |   
  21000 |-+                                                                 |   
        |                                                                   |   
  20000 |-+                                                                 |   
  19000 |-+                                                                 |   
        |                                                                   |   
  18000 |-+                                                                 |   
  17000 |-+                                                                 |   
        |            O       O  O O  O    O  O O    O  O  O O  O  O O  O O  |   
  16000 |-+             O O            O          O                         |   
  15000 |-+                                                                 |   
        |  O O  O O                                                         |   
  14000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 unixbench.workload                             
                                                                                
  900000 +------------------------------------------------------------------+   
         |                        .+.                                       |   
  850000 |..+.+..+.+..+.+..+..+.+.   +..+..+.+..+.+..+.+..+.. .+..+.        |   
         |                                                   +      +       |   
  800000 |-+                                                                |   
  750000 |-+                                                                |   
         |                                                                  |   
  700000 |-+                                                                |   
         |                                                                  |   
  650000 |-+                                                                |   
  600000 |-+          O O  O  O O  O O  O  O O  O O  O O  O  O O  O O  O O  |   
         |                                                                  |   
  550000 |-+O O  O O                                                        |   
         |                                                                  |   
  500000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2ap1: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
  4k/gcc-9/performance/1SSD/xfs/sync/x86_64-rhel-8.3/32/debian-10.4-x86_64-20200603.cgz/300s/randwrite/lkp-csl-2ap1/256g/fio-basic/0x4003003

commit: 
  c9847a7f94 ("locking/rwsem: Wake up all waiting readers if RWSEM_WAKE_READ_OWNED")
  10a59003d2 ("locking/rwsem: Remove reader optimistic spinning")

c9847a7f94679e74 10a59003d29fbfa855b2ef4f353 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     kmsg.ACPI_Error
         %stddev     %change         %stddev
             \          |                \  
      0.04 ±  5%      +0.1        0.10 ±  6%  fio.latency_100us%
     33.62 ±  7%      +3.6       37.18 ±  3%  fio.latency_10us%
      0.01            +0.0        0.05 ±  6%  fio.latency_250us%
      0.14 ±  9%      -0.1        0.03 ± 61%  fio.latency_2us%
     59.95 ±  3%      -4.3       55.60 ±  2%  fio.latency_4us%
      0.34 ±  2%      +0.1        0.46        fio.latency_50ms%
    256.52           +17.5%     301.42        fio.time.elapsed_time
    256.52           +17.5%     301.42        fio.time.elapsed_time.max
 5.369e+08           -17.5%   4.43e+08        fio.time.file_system_outputs
      1384 ± 13%    +410.5%       7064 ± 21%  fio.time.involuntary_context_switches
    153.25           -23.5%     117.25 ±  3%  fio.time.percent_of_cpu_this_job_got
    316.95 ±  2%     -10.8%     282.66 ±  3%  fio.time.system_time
     77.63            -6.9%      72.27 ±  3%  fio.time.user_time
    631893 ±  7%    +225.6%    2057309 ±  4%  fio.time.voluntary_context_switches
  67108864           -17.5%   55373247        fio.workload
      1027           -29.9%     720.93        fio.write_bw_MBps
     20288 ±  5%     +19.2%      24192 ±  4%  fio.write_clat_99%_us
    117541           +46.9%     172624        fio.write_clat_mean_us
   1928136           +27.7%    2462685        fio.write_clat_stddev
    263161           -29.9%     184558        fio.write_iops
     14.11            +6.5%      15.03        iostat.cpu.iowait
      2.94           -16.2%       2.46        iostat.cpu.system
      1.51            -0.5        1.00        mpstat.cpu.all.sys%
      0.15            -0.0        0.12 ±  3%  mpstat.cpu.all.usr%
    297.78           +14.8%     342.00        uptime.boot
     47592           +14.2%      54341        uptime.idle
  13822886 ± 16%     -35.1%    8966774 ± 32%  numa-numastat.node1.local_node
  13838524 ± 16%     -35.0%    8997869 ± 32%  numa-numastat.node1.numa_hit
   2422719 ± 27%     -52.3%    1155068 ± 43%  numa-numastat.node2.numa_foreign
   6968721 ± 52%    +189.4%   20169562 ±  5%  cpuidle.C1.usage
 3.139e+10 ± 51%     +74.9%  5.488e+10        cpuidle.C1E.time
  76223737 ± 30%     +59.5%  1.216e+08        cpuidle.C1E.usage
  15993891 ± 63%    +148.7%   39780679 ±  4%  cpuidle.POLL.time
   2432485 ± 11%    +461.8%   13666817 ±  5%  cpuidle.POLL.usage
    929962           -32.4%     628917        vmstat.io.bo
 1.387e+08           -13.0%  1.206e+08        vmstat.memory.cache
  56328701           +32.2%   74444784        vmstat.memory.free
     11079 ±  2%   +2404.1%     277436        vmstat.system.cs
    526522            -3.1%     509959        vmstat.system.in
      8306            +9.6%       9105        meminfo.Active(anon)
 1.378e+08           -13.0%  1.198e+08        meminfo.Cached
  1.37e+08           -13.1%   1.19e+08        meminfo.Inactive
 1.356e+08           -13.3%  1.176e+08        meminfo.Inactive(file)
  56492602           +32.0%   74549292        meminfo.MemFree
 1.412e+08           -12.8%  1.232e+08        meminfo.Memused
     18226 ±  2%     -87.4%       2302 ± 24%  meminfo.Writeback
    763436           -16.4%     638161        meminfo.max_used_kB
  11511284 ±  6%     +48.5%   17092100 ± 28%  numa-meminfo.node0.MemFree
  37643579           -14.8%   32062762 ± 15%  numa-meminfo.node0.MemUsed
      5353 ±  7%     -87.6%     664.50 ± 30%  numa-meminfo.node0.Writeback
  37095792 ±  2%     -11.0%   33024787 ±  4%  numa-meminfo.node1.FilePages
  36877519 ±  2%     -10.9%   32843351 ±  4%  numa-meminfo.node1.Inactive
  36551402 ±  3%     -11.1%   32491409 ±  6%  numa-meminfo.node1.Inactive(file)
  12041430 ±  7%     +32.7%   15975564 ± 10%  numa-meminfo.node1.MemFree
  37493024 ±  2%     -10.5%   33558891 ±  4%  numa-meminfo.node1.MemUsed
      5475 ±  8%     -87.5%     686.25 ± 16%  numa-meminfo.node1.Writeback
  34723911 ±  5%     -17.0%   28828495 ± 11%  numa-meminfo.node2.FilePages
  34527637 ±  5%     -17.2%   28599325 ± 12%  numa-meminfo.node2.Inactive
  33909426 ±  4%     -16.6%   28295544 ± 11%  numa-meminfo.node2.Inactive(file)
  14433625 ± 12%     +36.9%   19759122 ± 18%  numa-meminfo.node2.MemFree
      2723 ± 67%     -49.6%       1373 ±118%  numa-meminfo.node2.PageTables
      4375 ± 11%     -85.0%     657.75 ± 11%  numa-meminfo.node2.Writeback
      3900 ± 29%     -84.2%     615.50 ± 37%  numa-meminfo.node3.Writeback
  10098273 ±  3%     -20.8%    7998683 ± 18%  numa-vmstat.node0.nr_dirtied
   2877523 ±  6%     +48.4%    4270350 ± 28%  numa-vmstat.node0.nr_free_pages
      1360 ± 11%     -86.5%     183.00 ± 23%  numa-vmstat.node0.nr_writeback
   8070468 ±  2%     -26.8%    5909623 ± 21%  numa-vmstat.node0.nr_written
  10724541 ±  5%     -21.4%    8433227 ±  7%  numa-vmstat.node1.nr_dirtied
   9274641 ±  2%     -11.0%    8258905 ±  4%  numa-vmstat.node1.nr_file_pages
   3009682 ±  7%     +32.6%    3991208 ± 10%  numa-vmstat.node1.nr_free_pages
   9138561 ±  3%     -11.1%    8125800 ±  6%  numa-vmstat.node1.nr_inactive_file
      1398 ±  8%     -85.1%     208.50 ± 22%  numa-vmstat.node1.nr_writeback
   8489928 ±  6%     -25.6%    6312338 ±  8%  numa-vmstat.node1.nr_written
   9138656 ±  3%     -11.1%    8125844 ±  6%  numa-vmstat.node1.nr_zone_inactive_file
   9486707 ±  6%     -24.2%    7193218 ± 12%  numa-vmstat.node2.nr_dirtied
   8682169 ±  5%     -17.0%    7210434 ± 11%  numa-vmstat.node2.nr_file_pages
   3607260 ± 12%     +36.9%    4936622 ± 18%  numa-vmstat.node2.nr_free_pages
   8478548 ±  4%     -16.5%    7077427 ± 11%  numa-vmstat.node2.nr_inactive_file
    677.50 ± 68%     -49.7%     341.00 ±119%  numa-vmstat.node2.nr_page_table_pages
      1037 ± 18%     -85.2%     153.00 ± 11%  numa-vmstat.node2.nr_writeback
   7392605 ±  7%     -30.2%    5158591 ± 15%  numa-vmstat.node2.nr_written
   8478605 ±  4%     -16.5%    7077446 ± 11%  numa-vmstat.node2.nr_zone_inactive_file
    951.75 ± 32%     -83.7%     154.75 ± 21%  numa-vmstat.node3.nr_writeback
      2423 ±  4%     -12.6%       2116 ±  3%  slabinfo.Acpi-Parse.active_objs
      2423 ±  4%     -12.6%       2116 ±  3%  slabinfo.Acpi-Parse.num_objs
      4105 ±  4%     -67.3%       1343 ±  3%  slabinfo.biovec-max.active_objs
    522.75 ±  4%     -67.0%     172.25 ±  4%  slabinfo.biovec-max.active_slabs
      4184 ±  4%     -67.0%       1380 ±  4%  slabinfo.biovec-max.num_objs
    522.75 ±  4%     -67.0%     172.25 ±  4%  slabinfo.biovec-max.num_slabs
      4329 ±  6%     +20.2%       5203 ±  5%  slabinfo.ip6-frags.active_objs
      4329 ±  6%     +20.2%       5203 ±  5%  slabinfo.ip6-frags.num_objs
     21021 ±  4%     -25.6%      15642        slabinfo.kmalloc-128.active_objs
     21433 ±  4%     -26.0%      15861        slabinfo.kmalloc-128.num_objs
     16475 ±  4%     -28.5%      11778        slabinfo.kmalloc-4k.active_objs
      2258 ±  4%     -28.8%       1608        slabinfo.kmalloc-4k.active_slabs
     18069 ±  4%     -28.8%      12871        slabinfo.kmalloc-4k.num_objs
      2258 ±  4%     -28.8%       1608        slabinfo.kmalloc-4k.num_slabs
      6140           -11.7%       5419        slabinfo.kmalloc-8k.active_slabs
     24564           -11.7%      21680        slabinfo.kmalloc-8k.num_objs
      6140           -11.7%       5419        slabinfo.kmalloc-8k.num_slabs
     27805 ±  4%     -16.3%      23275 ±  4%  slabinfo.numa_policy.active_objs
    470.00 ±  3%     -19.6%     377.75 ±  5%  slabinfo.numa_policy.active_slabs
     29169 ±  3%     -19.6%      23440 ±  4%  slabinfo.numa_policy.num_objs
    470.00 ±  3%     -19.6%     377.75 ±  5%  slabinfo.numa_policy.num_slabs
     20269 ±  4%     -27.1%      14776 ±  3%  slabinfo.pool_workqueue.active_objs
     20699 ±  4%     -27.9%      14916 ±  3%  slabinfo.pool_workqueue.num_objs
     16526           +13.9%      18824 ±  7%  slabinfo.proc_inode_cache.active_objs
     17590           +13.3%      19925 ±  6%  slabinfo.proc_inode_cache.num_objs
      3233 ±  9%     -82.7%     559.50 ± 10%  slabinfo.xfs_efd_item.active_objs
      3369 ±  9%     -83.2%     567.00 ± 10%  slabinfo.xfs_efd_item.num_objs
      3276 ±  8%     -82.7%     567.50 ± 10%  slabinfo.xfs_efi_item.active_objs
      3411 ±  8%     -83.1%     575.50 ± 10%  slabinfo.xfs_efi_item.num_objs
    183556 ± 15%     -40.0%     110053 ± 24%  proc-vmstat.compact_daemon_free_scanned
      1164 ± 17%     -56.0%     512.00 ± 38%  proc-vmstat.compact_daemon_wake
    178173 ± 14%     -36.5%     113212 ± 16%  proc-vmstat.compact_isolated
    768.25 ± 12%     -43.2%     436.25 ± 15%  proc-vmstat.kswapd_high_wmark_hit_quickly
      1242 ± 18%     -55.8%     548.75 ± 25%  proc-vmstat.kswapd_low_wmark_hit_quickly
      2073            +9.6%       2272        proc-vmstat.nr_active_anon
  67108825           -17.5%   55358412        proc-vmstat.nr_dirtied
  34429046           -13.1%   29922129        proc-vmstat.nr_file_pages
  14133405           +32.0%   18660812        proc-vmstat.nr_free_pages
  33887409           -13.3%   29379656        proc-vmstat.nr_inactive_file
    299680            -7.2%     277963        proc-vmstat.nr_slab_unreclaimable
      4828 ±  4%     -88.7%     546.75 ±  5%  proc-vmstat.nr_writeback
  59368449           -20.8%   47001378        proc-vmstat.nr_written
      2073            +9.6%       2272        proc-vmstat.nr_zone_active_anon
  33887718           -13.3%   29379826        proc-vmstat.nr_zone_inactive_file
  20453611 ±  9%     -18.6%   16639765 ± 10%  proc-vmstat.numa_foreign
  49075650 ±  3%     -16.2%   41121350 ±  5%  proc-vmstat.numa_hit
  48981929 ±  3%     -16.2%   41027625 ±  5%  proc-vmstat.numa_local
  20453611 ±  9%     -18.6%   16639765 ± 10%  proc-vmstat.numa_miss
  20547332 ±  9%     -18.6%   16733490 ± 10%  proc-vmstat.numa_other
      2371 ±  8%     -51.0%       1161 ±  8%  proc-vmstat.pageoutrun
  75679865           -15.3%   64118824        proc-vmstat.pgalloc_normal
   1556864           +10.8%    1724803        proc-vmstat.pgfault
  27733269           -38.9%   16947822 ±  3%  proc-vmstat.pgfree
     90755 ± 13%     -35.4%      58642 ± 15%  proc-vmstat.pgmigrate_success
 2.404e+08           -20.7%  1.906e+08        proc-vmstat.pgpgout
     56870           +16.6%      66312        proc-vmstat.pgreuse
  19170054           -56.4%    8359591        proc-vmstat.pgscan_file
  19170054           -56.4%    8359591        proc-vmstat.pgscan_kswapd
  19170022           -56.4%    8359567        proc-vmstat.pgsteal_file
  19170022           -56.4%    8359567        proc-vmstat.pgsteal_kswapd
    166406 ±  8%     -45.0%      91531 ± 33%  proc-vmstat.slabs_scanned
     45.86 ± 34%    +181.8%     129.26 ± 13%  sched_debug.cfs_rq:/.load_avg.avg
      2084 ± 61%    +101.1%       4192 ± 17%  sched_debug.cfs_rq:/.load_avg.max
    228.76 ± 54%    +113.4%     488.17 ± 15%  sched_debug.cfs_rq:/.load_avg.stddev
      0.05 ±  8%     -24.0%       0.04 ± 14%  sched_debug.cfs_rq:/.nr_running.avg
      0.22 ±  3%     -10.7%       0.19 ±  8%  sched_debug.cfs_rq:/.nr_running.stddev
     59.93 ±  3%     -26.4%      44.09 ±  9%  sched_debug.cfs_rq:/.runnable_avg.avg
    967.61 ±  9%     -27.3%     703.21 ±  3%  sched_debug.cfs_rq:/.runnable_avg.max
    151.63 ±  7%     -24.6%     114.37 ±  7%  sched_debug.cfs_rq:/.runnable_avg.stddev
     17842 ± 96%    +169.4%      48067 ± 34%  sched_debug.cfs_rq:/.spread0.max
     59.77 ±  3%     -28.2%      42.91 ±  9%  sched_debug.cfs_rq:/.util_avg.avg
    966.71 ±  9%     -28.5%     691.08 ±  2%  sched_debug.cfs_rq:/.util_avg.max
    151.34 ±  7%     -27.8%     109.22 ±  6%  sched_debug.cfs_rq:/.util_avg.stddev
     13.27 ± 13%     -58.4%       5.52 ± 27%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    768.17 ±  9%     -49.2%     389.92 ± 20%  sched_debug.cfs_rq:/.util_est_enqueued.max
     88.39 ±  9%     -54.0%      40.68 ± 22%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     39884 ± 51%     -86.8%       5276 ± 39%  sched_debug.cpu.avg_idle.min
    197646 ± 12%     +52.6%     301694 ±  5%  sched_debug.cpu.avg_idle.stddev
    145476 ± 10%     +30.6%     189963        sched_debug.cpu.clock.avg
    145491 ± 10%     +30.6%     189978        sched_debug.cpu.clock.max
    145461 ± 10%     +30.6%     189950        sched_debug.cpu.clock.min
    143795 ± 10%     +30.5%     187634        sched_debug.cpu.clock_task.avg
    144737 ± 10%     +30.4%     188805        sched_debug.cpu.clock_task.max
    136580 ± 10%     +32.0%     180288        sched_debug.cpu.clock_task.min
      7064 ±  7%     +20.3%       8499        sched_debug.cpu.curr->pid.max
      0.00 ± 49%     -43.7%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
      0.03 ±  2%     -15.5%       0.03 ± 11%  sched_debug.cpu.nr_running.avg
      9449 ±  7%   +2186.6%     216066        sched_debug.cpu.nr_switches.avg
     48468 ± 16%   +3290.8%    1643455 ±  3%  sched_debug.cpu.nr_switches.max
      1144 ± 14%     +28.6%       1471 ±  3%  sched_debug.cpu.nr_switches.min
      9828 ±  8%   +4156.6%     418353        sched_debug.cpu.nr_switches.stddev
    -28.69          +314.2%    -118.83        sched_debug.cpu.nr_uninterruptible.min
     16.51 ±  2%     +84.0%      30.38 ±  9%  sched_debug.cpu.nr_uninterruptible.stddev
      7861 ±  9%   +2639.8%     215396        sched_debug.cpu.sched_count.avg
     45309 ± 16%   +3537.5%    1648138 ±  3%  sched_debug.cpu.sched_count.max
    405.62 ± 14%     +39.8%     567.08        sched_debug.cpu.sched_count.min
      9538 ±  9%   +4305.9%     420274        sched_debug.cpu.sched_count.stddev
      3194 ±  9%   +2430.1%      80819        sched_debug.cpu.sched_goidle.avg
     22591 ± 16%   +2931.8%     684944 ±  2%  sched_debug.cpu.sched_goidle.max
    150.03 ± 13%     +38.5%     207.79        sched_debug.cpu.sched_goidle.min
      4250 ± 10%   +3787.0%     165209        sched_debug.cpu.sched_goidle.stddev
      4451 ±  8%   +2675.1%     123520        sched_debug.cpu.ttwu_count.avg
     52506 ±  8%   +2032.0%    1119423 ±  4%  sched_debug.cpu.ttwu_count.max
    141.88 ± 14%     +40.3%     199.08        sched_debug.cpu.ttwu_count.min
      9369 ±  5%   +2656.8%     258303        sched_debug.cpu.ttwu_count.stddev
      2233 ± 10%   +1279.5%      30807        sched_debug.cpu.ttwu_local.avg
     20049 ±  6%   +1200.9%     260826 ±  3%  sched_debug.cpu.ttwu_local.max
    139.15 ± 14%     +41.1%     196.33        sched_debug.cpu.ttwu_local.min
      3680 ±  5%   +1596.4%      62439        sched_debug.cpu.ttwu_local.stddev
    145462 ± 10%     +30.6%     189951        sched_debug.cpu_clk
    144588 ± 10%     +30.8%     189077        sched_debug.ktime
    145825 ± 10%     +30.5%     190314        sched_debug.sched_clk
 2.167e+09           -14.5%  1.852e+09        perf-stat.i.branch-instructions
     31.58 ± 12%      -7.0       24.62        perf-stat.i.cache-miss-rate%
  14614329 ±  2%     -13.8%   12595505        perf-stat.i.cache-misses
     11124 ±  2%   +2404.3%     278598        perf-stat.i.context-switches
      1.64            +8.0%       1.77 ±  2%  perf-stat.i.cpi
 1.802e+10 ±  2%      -9.8%  1.625e+10 ±  2%  perf-stat.i.cpu-cycles
    253.44         +2786.6%       7315 ±  7%  perf-stat.i.cpu-migrations
      1244 ±  2%      +3.5%       1288 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.07 ± 26%      -0.0        0.04 ± 28%  perf-stat.i.dTLB-load-miss-rate%
   2046807 ± 26%     -53.7%     948333 ± 28%  perf-stat.i.dTLB-load-misses
 2.887e+09           -16.4%  2.414e+09        perf-stat.i.dTLB-loads
      0.01 ± 37%      -0.0        0.00 ± 20%  perf-stat.i.dTLB-store-miss-rate%
    115412 ± 37%     -55.6%      51255 ± 20%  perf-stat.i.dTLB-store-misses
 1.449e+09           -11.7%  1.278e+09        perf-stat.i.dTLB-stores
     61.58           -11.5       50.07        perf-stat.i.iTLB-load-miss-rate%
   5301708 ±  4%     +65.5%    8775440        perf-stat.i.iTLB-loads
 1.076e+10           -16.4%  8.998e+09        perf-stat.i.instructions
      1262           -19.3%       1017        perf-stat.i.instructions-per-iTLB-miss
      0.62            -7.9%       0.57 ±  2%  perf-stat.i.ipc
      1094           -15.2%     928.13        perf-stat.i.major-faults
      0.09 ±  2%      -9.8%       0.08 ±  2%  perf-stat.i.metric.GHz
     34.19           -14.5%      29.22        perf-stat.i.metric.M/sec
     70.32 ±  4%      +8.1       78.39 ±  2%  perf-stat.i.node-load-miss-rate%
   1712035 ±  8%     -36.8%    1081998 ±  7%  perf-stat.i.node-loads
     59.44 ± 10%     +11.1       70.55 ±  2%  perf-stat.i.node-store-miss-rate%
    549390 ± 17%     -35.2%     355933 ±  5%  perf-stat.i.node-stores
      4832            -3.6%       4656        perf-stat.i.page-faults
      1.68            +7.8%       1.81 ±  2%  perf-stat.overall.cpi
      1234            +4.6%       1290 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.07 ± 26%      -0.0        0.04 ± 28%  perf-stat.overall.dTLB-load-miss-rate%
      0.01 ± 37%      -0.0        0.00 ± 20%  perf-stat.overall.dTLB-store-miss-rate%
     62.13           -11.7       50.40        perf-stat.overall.iTLB-load-miss-rate%
      1239           -18.5%       1009        perf-stat.overall.instructions-per-iTLB-miss
      0.60            -7.2%       0.55 ±  2%  perf-stat.overall.ipc
     68.97 ±  4%      +8.1       77.07 ±  2%  perf-stat.overall.node-load-miss-rate%
     41110           +19.2%      49006        perf-stat.overall.path-length
 2.159e+09           -14.5%  1.847e+09        perf-stat.ps.branch-instructions
  14564018 ±  2%     -13.7%   12562224        perf-stat.ps.cache-misses
     11076 ±  2%   +2406.2%     277608        perf-stat.ps.context-switches
 1.797e+10 ±  2%      -9.8%  1.621e+10 ±  2%  perf-stat.ps.cpu-cycles
    252.44         +2787.8%       7289 ±  7%  perf-stat.ps.cpu-migrations
   2038697 ± 26%     -53.6%     945188 ± 28%  perf-stat.ps.dTLB-load-misses
 2.877e+09           -16.3%  2.407e+09        perf-stat.ps.dTLB-loads
    115024 ± 37%     -55.5%      51145 ± 20%  perf-stat.ps.dTLB-store-misses
 1.443e+09           -11.7%  1.275e+09        perf-stat.ps.dTLB-stores
   5280755 ±  4%     +65.6%    8745507        perf-stat.ps.iTLB-loads
 1.072e+10           -16.3%  8.973e+09        perf-stat.ps.instructions
      1094           -14.9%     931.28        perf-stat.ps.major-faults
   1707854 ±  8%     -36.7%    1080972 ±  7%  perf-stat.ps.node-loads
    548040 ± 17%     -35.1%     355596 ±  5%  perf-stat.ps.node-stores
      4818            -3.5%       4647        perf-stat.ps.page-faults
 2.759e+12            -1.6%  2.713e+12        perf-stat.total.instructions
    516.00           +17.4%     606.00        interrupts.9:IO-APIC.9-fasteoi.acpi
    241214 ± 20%    +386.1%    1172555        interrupts.CAL:Function_call_interrupts
      1173 ± 16%   +1030.0%      13255 ± 81%  interrupts.CPU0.CAL:Function_call_interrupts
    514234           +17.7%     605443        interrupts.CPU0.LOC:Local_timer_interrupts
    516.00           +17.4%     606.00        interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    512723           +18.1%     605278        interrupts.CPU1.LOC:Local_timer_interrupts
      1416 ± 28%   +1078.1%      16682 ± 84%  interrupts.CPU10.CAL:Function_call_interrupts
    511935           +18.3%     605746        interrupts.CPU10.LOC:Local_timer_interrupts
      1143 ± 20%    +874.2%      11139 ± 83%  interrupts.CPU100.CAL:Function_call_interrupts
    510002           +18.8%     605836        interrupts.CPU100.LOC:Local_timer_interrupts
      1137 ± 20%    +863.0%      10952 ± 74%  interrupts.CPU101.CAL:Function_call_interrupts
    510758           +18.5%     605423        interrupts.CPU101.LOC:Local_timer_interrupts
      1112 ± 20%    +799.2%      10001 ± 82%  interrupts.CPU102.CAL:Function_call_interrupts
    513218           +18.0%     605498        interrupts.CPU102.LOC:Local_timer_interrupts
      1144 ± 20%    +760.0%       9840 ± 71%  interrupts.CPU103.CAL:Function_call_interrupts
    513340           +17.9%     605423        interrupts.CPU103.LOC:Local_timer_interrupts
      1139 ± 18%    +647.4%       8518 ± 75%  interrupts.CPU104.CAL:Function_call_interrupts
    511907           +18.3%     605394        interrupts.CPU104.LOC:Local_timer_interrupts
      1128 ± 20%    +638.4%       8330 ± 55%  interrupts.CPU105.CAL:Function_call_interrupts
    510862           +18.5%     605403        interrupts.CPU105.LOC:Local_timer_interrupts
      1136 ± 21%    +501.6%       6833 ± 54%  interrupts.CPU106.CAL:Function_call_interrupts
    510908           +18.5%     605427        interrupts.CPU106.LOC:Local_timer_interrupts
      1188 ± 21%    +566.4%       7919 ± 59%  interrupts.CPU107.CAL:Function_call_interrupts
    513539           +17.9%     605427        interrupts.CPU107.LOC:Local_timer_interrupts
      1233 ± 23%    +553.6%       8064 ± 63%  interrupts.CPU108.CAL:Function_call_interrupts
    514204           +17.7%     605443        interrupts.CPU108.LOC:Local_timer_interrupts
      1154 ± 18%    +470.9%       6592 ± 74%  interrupts.CPU109.CAL:Function_call_interrupts
    511674           +18.3%     605335        interrupts.CPU109.LOC:Local_timer_interrupts
      1318 ± 22%   +1264.3%      17984 ± 87%  interrupts.CPU11.CAL:Function_call_interrupts
    514818           +17.6%     605440        interrupts.CPU11.LOC:Local_timer_interrupts
      1130 ± 19%    +519.4%       7001 ± 59%  interrupts.CPU110.CAL:Function_call_interrupts
    510713           +18.6%     605450        interrupts.CPU110.LOC:Local_timer_interrupts
      1140 ± 17%    +493.6%       6767 ± 54%  interrupts.CPU111.CAL:Function_call_interrupts
    509610           +18.8%     605511        interrupts.CPU111.LOC:Local_timer_interrupts
      1192 ± 21%    +337.7%       5217 ± 51%  interrupts.CPU112.CAL:Function_call_interrupts
    511076           +18.6%     605947        interrupts.CPU112.LOC:Local_timer_interrupts
     37.00 ± 64%  +12832.4%       4785 ± 95%  interrupts.CPU112.RES:Rescheduling_interrupts
      1131 ± 16%    +322.9%       4784 ± 48%  interrupts.CPU113.CAL:Function_call_interrupts
    509857           +18.9%     606431        interrupts.CPU113.LOC:Local_timer_interrupts
     41.00 ± 57%  +11022.6%       4560 ± 96%  interrupts.CPU113.RES:Rescheduling_interrupts
      1174 ± 24%    +328.4%       5029 ± 45%  interrupts.CPU114.CAL:Function_call_interrupts
    509860           +18.8%     605586        interrupts.CPU114.LOC:Local_timer_interrupts
     46.75 ± 63%   +9096.8%       4299 ± 97%  interrupts.CPU114.RES:Rescheduling_interrupts
      1183 ± 21%    +290.9%       4626 ± 43%  interrupts.CPU115.CAL:Function_call_interrupts
    511295           +18.5%     605739        interrupts.CPU115.LOC:Local_timer_interrupts
     35.75 ± 63%  +10950.3%       3950 ± 94%  interrupts.CPU115.RES:Rescheduling_interrupts
      1195 ± 24%    +275.5%       4489 ± 59%  interrupts.CPU116.CAL:Function_call_interrupts
    509693           +19.1%     607228        interrupts.CPU116.LOC:Local_timer_interrupts
      1208 ± 20%    +290.3%       4716 ± 42%  interrupts.CPU117.CAL:Function_call_interrupts
    510905           +18.5%     605473        interrupts.CPU117.LOC:Local_timer_interrupts
      1136 ± 22%    +338.7%       4986 ± 48%  interrupts.CPU118.CAL:Function_call_interrupts
    509923           +18.7%     605494        interrupts.CPU118.LOC:Local_timer_interrupts
     33.75 ± 74%  +12737.0%       4332 ± 98%  interrupts.CPU118.RES:Rescheduling_interrupts
      1153 ± 20%    +365.1%       5366 ± 45%  interrupts.CPU119.CAL:Function_call_interrupts
    511178           +18.5%     605596        interrupts.CPU119.LOC:Local_timer_interrupts
      1274 ± 22%   +1407.0%      19210 ± 86%  interrupts.CPU12.CAL:Function_call_interrupts
    514753           +17.6%     605484        interrupts.CPU12.LOC:Local_timer_interrupts
    513843           +17.8%     605343        interrupts.CPU120.LOC:Local_timer_interrupts
    515081           +17.5%     605238        interrupts.CPU121.LOC:Local_timer_interrupts
    515027           +17.5%     605302        interrupts.CPU122.LOC:Local_timer_interrupts
    514802           +17.6%     605242        interrupts.CPU123.LOC:Local_timer_interrupts
    513728           +17.8%     605344        interrupts.CPU124.LOC:Local_timer_interrupts
    514733           +17.6%     605280        interrupts.CPU125.LOC:Local_timer_interrupts
    513881           +17.8%     605350        interrupts.CPU126.LOC:Local_timer_interrupts
    513886           +17.8%     605117        interrupts.CPU127.LOC:Local_timer_interrupts
    513632           +17.8%     605303        interrupts.CPU128.LOC:Local_timer_interrupts
    206.75 ± 34%     -45.5%     112.75 ± 34%  interrupts.CPU128.NMI:Non-maskable_interrupts
    206.75 ± 34%     -45.5%     112.75 ± 34%  interrupts.CPU128.PMI:Performance_monitoring_interrupts
    513642           +17.8%     605091        interrupts.CPU129.LOC:Local_timer_interrupts
      1196 ± 19%   +1408.7%      18047 ± 83%  interrupts.CPU13.CAL:Function_call_interrupts
    512136           +18.2%     605406        interrupts.CPU13.LOC:Local_timer_interrupts
    513792           +17.8%     605254        interrupts.CPU130.LOC:Local_timer_interrupts
    513659           +17.8%     605248        interrupts.CPU131.LOC:Local_timer_interrupts
    513651           +17.8%     605254        interrupts.CPU132.LOC:Local_timer_interrupts
    514585           +17.6%     605369        interrupts.CPU133.LOC:Local_timer_interrupts
    513797           +17.9%     605557        interrupts.CPU134.LOC:Local_timer_interrupts
    513767           +17.8%     605135        interrupts.CPU135.LOC:Local_timer_interrupts
    514328           +17.7%     605254        interrupts.CPU136.LOC:Local_timer_interrupts
      1177 ± 22%     +94.6%       2290 ± 30%  interrupts.CPU137.CAL:Function_call_interrupts
    513855           +17.9%     605641        interrupts.CPU137.LOC:Local_timer_interrupts
      3.00 ± 81%   +7733.3%     235.00 ±168%  interrupts.CPU137.RES:Rescheduling_interrupts
    513480           +17.9%     605256        interrupts.CPU138.LOC:Local_timer_interrupts
    513887           +17.8%     605283        interrupts.CPU139.LOC:Local_timer_interrupts
      1196 ± 18%   +1461.2%      18672 ± 88%  interrupts.CPU14.CAL:Function_call_interrupts
    511309           +18.4%     605432        interrupts.CPU14.LOC:Local_timer_interrupts
    513628           +17.8%     605234        interrupts.CPU140.LOC:Local_timer_interrupts
    513663           +17.8%     605266        interrupts.CPU141.LOC:Local_timer_interrupts
    513516           +17.9%     605500        interrupts.CPU142.LOC:Local_timer_interrupts
    514605           +17.7%     605624        interrupts.CPU143.LOC:Local_timer_interrupts
    514859           +17.6%     605583        interrupts.CPU144.LOC:Local_timer_interrupts
      1208 ± 23%    +802.6%      10905 ± 82%  interrupts.CPU145.CAL:Function_call_interrupts
    513118           +18.0%     605359        interrupts.CPU145.LOC:Local_timer_interrupts
    515103           +17.5%     605500        interrupts.CPU146.LOC:Local_timer_interrupts
    512407           +18.1%     605372        interrupts.CPU147.LOC:Local_timer_interrupts
    513393           +17.9%     605483        interrupts.CPU148.LOC:Local_timer_interrupts
    514296           +17.7%     605316        interrupts.CPU149.LOC:Local_timer_interrupts
      1216 ± 20%   +1147.0%      15166 ± 90%  interrupts.CPU15.CAL:Function_call_interrupts
    510385           +18.6%     605419        interrupts.CPU15.LOC:Local_timer_interrupts
    511128           +18.4%     605408        interrupts.CPU150.LOC:Local_timer_interrupts
    511504           +18.3%     605269        interrupts.CPU151.LOC:Local_timer_interrupts
    114.25 ± 11%    +339.4%     502.00 ± 77%  interrupts.CPU151.NMI:Non-maskable_interrupts
    114.25 ± 11%    +339.4%     502.00 ± 77%  interrupts.CPU151.PMI:Performance_monitoring_interrupts
    511130           +18.4%     605391        interrupts.CPU152.LOC:Local_timer_interrupts
    510699           +18.5%     605383        interrupts.CPU153.LOC:Local_timer_interrupts
    511113           +18.5%     605428        interrupts.CPU154.LOC:Local_timer_interrupts
     17.75 ±143%    +1e+06%     180613 ±101%  interrupts.CPU154.RES:Rescheduling_interrupts
    511706           +18.3%     605328        interrupts.CPU155.LOC:Local_timer_interrupts
    512493           +18.1%     605464        interrupts.CPU156.LOC:Local_timer_interrupts
    511511           +18.3%     605357        interrupts.CPU157.LOC:Local_timer_interrupts
      1182 ± 19%    +449.2%       6496 ± 81%  interrupts.CPU158.CAL:Function_call_interrupts
    512833           +18.0%     605345        interrupts.CPU158.LOC:Local_timer_interrupts
      1164 ± 20%    +406.7%       5899 ± 78%  interrupts.CPU159.CAL:Function_call_interrupts
    511132           +18.4%     605336        interrupts.CPU159.LOC:Local_timer_interrupts
    513120           +18.1%     605836        interrupts.CPU16.LOC:Local_timer_interrupts
      1186 ± 19%    +295.6%       4694 ± 68%  interrupts.CPU160.CAL:Function_call_interrupts
    511530           +18.3%     605359        interrupts.CPU160.LOC:Local_timer_interrupts
    510929           +18.6%     605714        interrupts.CPU161.LOC:Local_timer_interrupts
    510784           +18.6%     605596        interrupts.CPU162.LOC:Local_timer_interrupts
    511100           +18.5%     605424        interrupts.CPU163.LOC:Local_timer_interrupts
      1.25 ±103%  +3.4e+05%       4219 ±104%  interrupts.CPU163.RES:Rescheduling_interrupts
      1173 ± 18%    +279.7%       4453 ± 62%  interrupts.CPU164.CAL:Function_call_interrupts
    510706           +18.5%     605389        interrupts.CPU164.LOC:Local_timer_interrupts
      1.75 ± 24%  +2.7e+05%       4804 ± 99%  interrupts.CPU164.RES:Rescheduling_interrupts
      1214 ± 23%    +249.3%       4241 ± 67%  interrupts.CPU165.CAL:Function_call_interrupts
    510738           +18.6%     605932        interrupts.CPU165.LOC:Local_timer_interrupts
     91.00 ± 23%    +178.0%     253.00 ± 54%  interrupts.CPU165.NMI:Non-maskable_interrupts
     91.00 ± 23%    +178.0%     253.00 ± 54%  interrupts.CPU165.PMI:Performance_monitoring_interrupts
      6.75 ± 63%  +69411.1%       4692 ±102%  interrupts.CPU165.RES:Rescheduling_interrupts
      1209 ± 18%    +218.9%       3856 ± 62%  interrupts.CPU166.CAL:Function_call_interrupts
    510778           +18.6%     605622        interrupts.CPU166.LOC:Local_timer_interrupts
     90.00 ± 24%    +114.2%     192.75 ± 35%  interrupts.CPU166.NMI:Non-maskable_interrupts
     90.00 ± 24%    +114.2%     192.75 ± 35%  interrupts.CPU166.PMI:Performance_monitoring_interrupts
      1177 ± 18%    +309.5%       4820 ± 52%  interrupts.CPU167.CAL:Function_call_interrupts
    511077           +18.5%     605666        interrupts.CPU167.LOC:Local_timer_interrupts
     90.25 ± 24%    +216.3%     285.50 ± 19%  interrupts.CPU167.NMI:Non-maskable_interrupts
     90.25 ± 24%    +216.3%     285.50 ± 19%  interrupts.CPU167.PMI:Performance_monitoring_interrupts
      2.25 ± 72%    +2e+05%       4486 ± 99%  interrupts.CPU167.RES:Rescheduling_interrupts
      1262 ± 25%     +54.7%       1953 ± 27%  interrupts.CPU168.CAL:Function_call_interrupts
    516016           +17.3%     605276        interrupts.CPU168.LOC:Local_timer_interrupts
    515960           +17.3%     605469        interrupts.CPU169.LOC:Local_timer_interrupts
    434.75 ± 59%     -62.9%     161.50 ± 30%  interrupts.CPU169.NMI:Non-maskable_interrupts
    434.75 ± 59%     -62.9%     161.50 ± 30%  interrupts.CPU169.PMI:Performance_monitoring_interrupts
      1104 ± 22%    +528.0%       6934 ± 62%  interrupts.CPU17.CAL:Function_call_interrupts
    510328           +18.7%     605554        interrupts.CPU17.LOC:Local_timer_interrupts
      1246 ± 34%     +68.1%       2095 ± 39%  interrupts.CPU170.CAL:Function_call_interrupts
    516201           +17.3%     605483        interrupts.CPU170.LOC:Local_timer_interrupts
    334.75 ± 29%     -36.1%     213.75 ± 21%  interrupts.CPU170.NMI:Non-maskable_interrupts
    334.75 ± 29%     -36.1%     213.75 ± 21%  interrupts.CPU170.PMI:Performance_monitoring_interrupts
    516089           +17.3%     605414        interrupts.CPU171.LOC:Local_timer_interrupts
    340.00 ± 42%     -56.5%     148.00 ± 21%  interrupts.CPU171.NMI:Non-maskable_interrupts
    340.00 ± 42%     -56.5%     148.00 ± 21%  interrupts.CPU171.PMI:Performance_monitoring_interrupts
    515928           +17.4%     605456        interrupts.CPU172.LOC:Local_timer_interrupts
    515944           +17.3%     605313        interrupts.CPU173.LOC:Local_timer_interrupts
    515836           +17.4%     605365        interrupts.CPU174.LOC:Local_timer_interrupts
    304.25 ±100%     -62.9%     113.00 ±  8%  interrupts.CPU174.NMI:Non-maskable_interrupts
    304.25 ±100%     -62.9%     113.00 ±  8%  interrupts.CPU174.PMI:Performance_monitoring_interrupts
    515935           +17.3%     605235        interrupts.CPU175.LOC:Local_timer_interrupts
    515566           +17.4%     605490        interrupts.CPU176.LOC:Local_timer_interrupts
    515944           +17.3%     605365        interrupts.CPU177.LOC:Local_timer_interrupts
    515917           +17.4%     605465        interrupts.CPU178.LOC:Local_timer_interrupts
    515918           +17.3%     605298        interrupts.CPU179.LOC:Local_timer_interrupts
      1125 ± 21%    +455.4%       6248 ± 71%  interrupts.CPU18.CAL:Function_call_interrupts
    510435           +18.6%     605489        interrupts.CPU18.LOC:Local_timer_interrupts
    515832           +17.4%     605421        interrupts.CPU180.LOC:Local_timer_interrupts
    515886           +17.3%     605369        interrupts.CPU181.LOC:Local_timer_interrupts
    515913           +17.3%     605307        interrupts.CPU182.LOC:Local_timer_interrupts
    515907           +17.3%     605391        interrupts.CPU183.LOC:Local_timer_interrupts
    515873           +17.3%     605289        interrupts.CPU184.LOC:Local_timer_interrupts
    516177           +17.3%     605345        interrupts.CPU185.LOC:Local_timer_interrupts
      1321 ±  9%     +56.9%       2073 ± 33%  interrupts.CPU186.CAL:Function_call_interrupts
    515709           +17.4%     605398        interrupts.CPU186.LOC:Local_timer_interrupts
      1161 ± 19%     +25.2%       1453 ±  8%  interrupts.CPU187.CAL:Function_call_interrupts
    515995           +17.3%     605284        interrupts.CPU187.LOC:Local_timer_interrupts
    515996           +17.3%     605298        interrupts.CPU188.LOC:Local_timer_interrupts
    515899           +17.3%     605298        interrupts.CPU189.LOC:Local_timer_interrupts
    510586           +18.6%     605784        interrupts.CPU19.LOC:Local_timer_interrupts
    515909           +17.3%     605357        interrupts.CPU190.LOC:Local_timer_interrupts
      1248 ± 29%     +34.1%       1674 ±  7%  interrupts.CPU191.CAL:Function_call_interrupts
    515902           +17.3%     605378        interrupts.CPU191.LOC:Local_timer_interrupts
      1299 ± 21%   +1166.8%      16458 ± 80%  interrupts.CPU2.CAL:Function_call_interrupts
    511817           +18.3%     605535        interrupts.CPU2.LOC:Local_timer_interrupts
      1134 ± 20%    +419.2%       5891 ± 72%  interrupts.CPU20.CAL:Function_call_interrupts
    509831           +18.8%     605434        interrupts.CPU20.LOC:Local_timer_interrupts
      1134 ± 22%    +413.0%       5817 ± 69%  interrupts.CPU21.CAL:Function_call_interrupts
    510416           +18.7%     605698        interrupts.CPU21.LOC:Local_timer_interrupts
      1114 ± 23%    +413.3%       5720 ± 77%  interrupts.CPU22.CAL:Function_call_interrupts
    509905           +18.7%     605450        interrupts.CPU22.LOC:Local_timer_interrupts
      1176 ± 19%    +455.9%       6540 ± 77%  interrupts.CPU23.CAL:Function_call_interrupts
    510208           +18.8%     606161        interrupts.CPU23.LOC:Local_timer_interrupts
      1535 ± 10%     +78.2%       2735 ± 16%  interrupts.CPU24.CAL:Function_call_interrupts
    513859           +17.9%     605713        interrupts.CPU24.LOC:Local_timer_interrupts
    654.75 ± 28%     -71.4%     187.25 ±  7%  interrupts.CPU24.NMI:Non-maskable_interrupts
    654.75 ± 28%     -71.4%     187.25 ±  7%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      1368 ± 14%    +114.9%       2939 ± 32%  interrupts.CPU25.CAL:Function_call_interrupts
    515715           +17.5%     605936        interrupts.CPU25.LOC:Local_timer_interrupts
    619.50 ± 45%     -58.3%     258.50 ± 41%  interrupts.CPU25.NMI:Non-maskable_interrupts
    619.50 ± 45%     -58.3%     258.50 ± 41%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    515285           +17.5%     605484        interrupts.CPU26.LOC:Local_timer_interrupts
    514892           +17.6%     605366        interrupts.CPU27.LOC:Local_timer_interrupts
      1157 ± 23%     +66.9%       1931 ± 25%  interrupts.CPU28.CAL:Function_call_interrupts
    513790           +17.8%     605384        interrupts.CPU28.LOC:Local_timer_interrupts
    514833           +17.6%     605296        interrupts.CPU29.LOC:Local_timer_interrupts
      1215 ± 21%   +1322.8%      17287 ± 86%  interrupts.CPU3.CAL:Function_call_interrupts
    510059           +18.7%     605325        interrupts.CPU3.LOC:Local_timer_interrupts
    513997           +17.8%     605397        interrupts.CPU30.LOC:Local_timer_interrupts
    513960           +17.8%     605444        interrupts.CPU31.LOC:Local_timer_interrupts
    513717           +17.9%     605420        interrupts.CPU32.LOC:Local_timer_interrupts
      1289 ± 19%     +49.0%       1920 ± 13%  interrupts.CPU33.CAL:Function_call_interrupts
    513745           +17.9%     605929        interrupts.CPU33.LOC:Local_timer_interrupts
      1152 ± 17%     +91.7%       2208 ± 43%  interrupts.CPU34.CAL:Function_call_interrupts
    513887           +17.9%     605779        interrupts.CPU34.LOC:Local_timer_interrupts
      1135 ± 24%     +53.6%       1744 ± 28%  interrupts.CPU35.CAL:Function_call_interrupts
    513694           +17.8%     605333        interrupts.CPU35.LOC:Local_timer_interrupts
    513746           +18.0%     606055        interrupts.CPU36.LOC:Local_timer_interrupts
    514657           +17.6%     605387        interrupts.CPU37.LOC:Local_timer_interrupts
    513887           +17.8%     605533        interrupts.CPU38.LOC:Local_timer_interrupts
    513874           +17.8%     605393        interrupts.CPU39.LOC:Local_timer_interrupts
      1186 ± 19%   +1467.5%      18598 ± 87%  interrupts.CPU4.CAL:Function_call_interrupts
    510575           +18.6%     605409        interrupts.CPU4.LOC:Local_timer_interrupts
    514437           +17.7%     605399        interrupts.CPU40.LOC:Local_timer_interrupts
    513922           +17.8%     605488        interrupts.CPU41.LOC:Local_timer_interrupts
    513643           +17.9%     605362        interrupts.CPU42.LOC:Local_timer_interrupts
    514155           +17.8%     605708        interrupts.CPU43.LOC:Local_timer_interrupts
      1097 ± 24%     +91.0%       2096 ± 32%  interrupts.CPU44.CAL:Function_call_interrupts
    513654           +18.0%     606048        interrupts.CPU44.LOC:Local_timer_interrupts
    513702           +17.9%     605573        interrupts.CPU45.LOC:Local_timer_interrupts
    513557           +17.9%     605360        interrupts.CPU46.LOC:Local_timer_interrupts
    514556           +17.7%     605738        interrupts.CPU47.LOC:Local_timer_interrupts
      1699 ± 36%    +759.3%      14598 ± 79%  interrupts.CPU48.CAL:Function_call_interrupts
    514869           +17.6%     605464        interrupts.CPU48.LOC:Local_timer_interrupts
      1343 ± 14%   +1160.3%      16932 ± 86%  interrupts.CPU49.CAL:Function_call_interrupts
    513305           +18.0%     605464        interrupts.CPU49.LOC:Local_timer_interrupts
      1227 ± 19%   +1401.3%      18432 ± 89%  interrupts.CPU5.CAL:Function_call_interrupts
    510871           +18.5%     605627        interrupts.CPU5.LOC:Local_timer_interrupts
    515088           +17.5%     605376        interrupts.CPU50.LOC:Local_timer_interrupts
      1204 ± 17%   +1405.3%      18127 ± 92%  interrupts.CPU51.CAL:Function_call_interrupts
    512351           +18.2%     605549        interrupts.CPU51.LOC:Local_timer_interrupts
    513855           +17.8%     605391        interrupts.CPU52.LOC:Local_timer_interrupts
    514932           +17.6%     605560        interrupts.CPU53.LOC:Local_timer_interrupts
    511184           +18.5%     605528        interrupts.CPU54.LOC:Local_timer_interrupts
      1153 ± 16%   +1670.5%      20423 ± 93%  interrupts.CPU55.CAL:Function_call_interrupts
    511641           +18.3%     605473        interrupts.CPU55.LOC:Local_timer_interrupts
    511233           +18.5%     605581        interrupts.CPU56.LOC:Local_timer_interrupts
    107.75 ± 11%    +300.2%     431.25 ± 60%  interrupts.CPU56.NMI:Non-maskable_interrupts
    107.75 ± 11%    +300.2%     431.25 ± 60%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
     61.50 ±164%  +11461.4%       7110 ± 99%  interrupts.CPU56.RES:Rescheduling_interrupts
      1107 ± 23%   +1664.4%      19541 ± 91%  interrupts.CPU57.CAL:Function_call_interrupts
    510795           +18.6%     605563        interrupts.CPU57.LOC:Local_timer_interrupts
      3.25 ± 50%  +2.2e+05%       7281 ± 99%  interrupts.CPU57.RES:Rescheduling_interrupts
      1119 ± 23%   +1660.7%      19710 ± 91%  interrupts.CPU58.CAL:Function_call_interrupts
    511234           +18.5%     605882        interrupts.CPU58.LOC:Local_timer_interrupts
      5.25 ± 51%  +1.4e+05%       7342 ±100%  interrupts.CPU58.RES:Rescheduling_interrupts
      1117 ± 21%   +1541.4%      18342 ± 92%  interrupts.CPU59.CAL:Function_call_interrupts
    511755           +18.4%     605662        interrupts.CPU59.LOC:Local_timer_interrupts
      1283 ± 23%   +1341.6%      18502 ± 80%  interrupts.CPU6.CAL:Function_call_interrupts
    513323           +18.0%     605537        interrupts.CPU6.LOC:Local_timer_interrupts
    512545           +18.1%     605561        interrupts.CPU60.LOC:Local_timer_interrupts
      1135 ± 18%   +1505.6%      18228 ± 93%  interrupts.CPU61.CAL:Function_call_interrupts
    511576           +18.3%     605425        interrupts.CPU61.LOC:Local_timer_interrupts
      1414 ± 25%   +1110.5%      17119 ± 91%  interrupts.CPU62.CAL:Function_call_interrupts
    512827           +18.1%     605436        interrupts.CPU62.LOC:Local_timer_interrupts
      1126 ± 21%   +1188.2%      14514 ± 90%  interrupts.CPU63.CAL:Function_call_interrupts
    511174           +18.5%     605588        interrupts.CPU63.LOC:Local_timer_interrupts
      1130 ± 22%    +294.1%       4456 ± 71%  interrupts.CPU64.CAL:Function_call_interrupts
    511557           +18.4%     605728        interrupts.CPU64.LOC:Local_timer_interrupts
    510978           +18.6%     605824        interrupts.CPU65.LOC:Local_timer_interrupts
    510755           +18.6%     605917        interrupts.CPU66.LOC:Local_timer_interrupts
      2.25 ±101%  +2.8e+05%       6412 ±100%  interrupts.CPU66.RES:Rescheduling_interrupts
    511092           +18.5%     605580        interrupts.CPU67.LOC:Local_timer_interrupts
      1147 ± 18%    +389.3%       5613 ± 68%  interrupts.CPU68.CAL:Function_call_interrupts
    510789           +18.5%     605511        interrupts.CPU68.LOC:Local_timer_interrupts
      2.75 ±107%  +2.5e+05%       6913 ±100%  interrupts.CPU68.RES:Rescheduling_interrupts
    510770           +18.6%     605649        interrupts.CPU69.LOC:Local_timer_interrupts
      1271 ± 21%   +1246.2%      17116 ± 87%  interrupts.CPU7.CAL:Function_call_interrupts
    513454           +17.9%     605531        interrupts.CPU7.LOC:Local_timer_interrupts
    516.25 ± 44%     -55.2%     231.25 ± 42%  interrupts.CPU7.NMI:Non-maskable_interrupts
    516.25 ± 44%     -55.2%     231.25 ± 42%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    510835           +18.6%     606082        interrupts.CPU70.LOC:Local_timer_interrupts
      1120 ± 21%    +392.4%       5518 ± 61%  interrupts.CPU71.CAL:Function_call_interrupts
    511083           +18.5%     605658        interrupts.CPU71.LOC:Local_timer_interrupts
      2.25 ± 96%  +2.7e+05%       6041 ±100%  interrupts.CPU71.RES:Rescheduling_interrupts
      1325 ± 26%    +142.0%       3207 ± 11%  interrupts.CPU72.CAL:Function_call_interrupts
    515998           +17.3%     605229        interrupts.CPU72.LOC:Local_timer_interrupts
      1280 ± 24%     +69.5%       2171 ± 16%  interrupts.CPU73.CAL:Function_call_interrupts
    516149           +17.3%     605586        interrupts.CPU73.LOC:Local_timer_interrupts
    498.50 ± 49%     -60.9%     194.75 ± 40%  interrupts.CPU73.NMI:Non-maskable_interrupts
    498.50 ± 49%     -60.9%     194.75 ± 40%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
      1383 ± 39%     +63.0%       2255 ± 23%  interrupts.CPU74.CAL:Function_call_interrupts
    516376           +17.2%     605343        interrupts.CPU74.LOC:Local_timer_interrupts
    516271           +17.3%     605661        interrupts.CPU75.LOC:Local_timer_interrupts
    516310           +17.3%     605650        interrupts.CPU76.LOC:Local_timer_interrupts
    516172           +17.3%     605330        interrupts.CPU77.LOC:Local_timer_interrupts
    515955           +17.4%     605572        interrupts.CPU78.LOC:Local_timer_interrupts
    515907           +17.4%     605833        interrupts.CPU79.LOC:Local_timer_interrupts
      1220 ± 17%   +1376.1%      18019 ± 87%  interrupts.CPU8.CAL:Function_call_interrupts
    512148           +18.3%     605670        interrupts.CPU8.LOC:Local_timer_interrupts
    515903           +17.3%     605409        interrupts.CPU80.LOC:Local_timer_interrupts
    515934           +17.3%     605309        interrupts.CPU81.LOC:Local_timer_interrupts
    515929           +17.5%     606412        interrupts.CPU82.LOC:Local_timer_interrupts
    516037           +17.4%     606044        interrupts.CPU83.LOC:Local_timer_interrupts
    515996           +17.4%     605807        interrupts.CPU84.LOC:Local_timer_interrupts
    515950           +17.4%     605519        interrupts.CPU85.LOC:Local_timer_interrupts
    516261           +17.3%     605483        interrupts.CPU86.LOC:Local_timer_interrupts
    516222           +17.3%     605533        interrupts.CPU87.LOC:Local_timer_interrupts
    516245           +17.3%     605439        interrupts.CPU88.LOC:Local_timer_interrupts
    516333           +17.4%     606092        interrupts.CPU89.LOC:Local_timer_interrupts
    313.50 ± 39%     -69.5%      95.75 ± 25%  interrupts.CPU89.NMI:Non-maskable_interrupts
    313.50 ± 39%     -69.5%      95.75 ± 25%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
    511049           +18.5%     605658        interrupts.CPU9.LOC:Local_timer_interrupts
    516502           +17.2%     605354        interrupts.CPU90.LOC:Local_timer_interrupts
    516464           +17.3%     605747        interrupts.CPU91.LOC:Local_timer_interrupts
    516253           +17.4%     606137        interrupts.CPU92.LOC:Local_timer_interrupts
    516158           +17.3%     605298        interrupts.CPU93.LOC:Local_timer_interrupts
    515995           +17.4%     605674        interrupts.CPU94.LOC:Local_timer_interrupts
    516304           +17.2%     605311        interrupts.CPU95.LOC:Local_timer_interrupts
    513802           +17.8%     605397        interrupts.CPU96.LOC:Local_timer_interrupts
    512011           +18.2%     605447        interrupts.CPU97.LOC:Local_timer_interrupts
    511607           +18.3%     605448        interrupts.CPU98.LOC:Local_timer_interrupts
    509867           +18.7%     605442        interrupts.CPU99.LOC:Local_timer_interrupts
  98572095           +17.9%  1.163e+08        interrupts.LOC:Local_timer_interrupts
      0.00       +1.9e+104%     192.00        interrupts.MCP:Machine_check_polls
     18832 ±  4%  +28724.8%    5428424 ±  3%  interrupts.RES:Rescheduling_interrupts
     24593 ± 14%    +103.8%      50109 ±  2%  softirqs.CPU0.RCU
     20767 ±  7%    +119.9%      45671 ±  5%  softirqs.CPU1.RCU
     21282 ± 46%     +60.4%      34135 ± 18%  softirqs.CPU1.SCHED
     21727 ±  7%    +115.9%      46904 ±  7%  softirqs.CPU10.RCU
     19127 ± 15%    +133.1%      44585 ±  4%  softirqs.CPU100.RCU
     32966 ±  4%     +15.6%      38122 ±  6%  softirqs.CPU100.SCHED
     19577 ± 15%    +131.0%      45223 ±  2%  softirqs.CPU101.RCU
     27027 ± 31%     +48.0%      40011 ±  2%  softirqs.CPU101.SCHED
     20058 ± 13%    +121.1%      44343 ±  4%  softirqs.CPU102.RCU
     19259 ± 13%    +132.0%      44674 ±  3%  softirqs.CPU103.RCU
     19621 ± 14%    +130.5%      45232 ±  3%  softirqs.CPU104.RCU
     18948 ± 17%    +137.8%      45063 ±  3%  softirqs.CPU105.RCU
     33071 ±  4%     +16.4%      38506 ±  6%  softirqs.CPU105.SCHED
     19429 ± 16%    +127.5%      44201 ±  2%  softirqs.CPU106.RCU
     29852 ± 20%     +29.3%      38611 ±  5%  softirqs.CPU106.SCHED
     19791 ± 12%    +131.1%      45743        softirqs.CPU107.RCU
     32969 ±  4%     +18.2%      38982 ±  6%  softirqs.CPU107.SCHED
     19517 ± 16%    +132.4%      45355        softirqs.CPU108.RCU
     32768 ±  5%     +15.5%      37857 ±  8%  softirqs.CPU108.SCHED
     19232 ± 15%    +132.6%      44727 ±  3%  softirqs.CPU109.RCU
     32108 ±  7%     +24.0%      39802        softirqs.CPU109.SCHED
     21436 ±  9%    +124.3%      48089 ±  7%  softirqs.CPU11.RCU
     19168 ± 16%    +136.7%      45370 ±  4%  softirqs.CPU110.RCU
     32633 ±  5%     +22.0%      39828 ±  2%  softirqs.CPU110.SCHED
     19226 ± 16%    +131.4%      44486 ±  3%  softirqs.CPU111.RCU
     31876 ±  9%     +22.5%      39037 ±  7%  softirqs.CPU111.SCHED
     18752 ±  9%    +125.1%      42214 ±  5%  softirqs.CPU112.RCU
     33139 ±  4%     +19.3%      39537 ±  3%  softirqs.CPU112.SCHED
     18571 ± 11%    +126.2%      42004 ±  3%  softirqs.CPU113.RCU
     33130 ±  4%     +20.6%      39943 ±  2%  softirqs.CPU113.SCHED
     18861 ± 13%    +124.8%      42409 ±  5%  softirqs.CPU114.RCU
     32611 ±  6%     +21.7%      39691 ±  2%  softirqs.CPU114.SCHED
     18161 ± 14%    +125.1%      40883 ±  4%  softirqs.CPU115.RCU
     32644 ±  5%     +20.1%      39212 ±  2%  softirqs.CPU115.SCHED
     18222 ± 14%    +131.3%      42152 ±  4%  softirqs.CPU116.RCU
     33025 ±  4%     +20.9%      39938        softirqs.CPU116.SCHED
     18640 ± 10%    +125.3%      41999 ±  4%  softirqs.CPU117.RCU
     33222 ±  4%     +18.9%      39496 ±  3%  softirqs.CPU117.SCHED
     18294 ± 13%    +130.0%      42068 ±  3%  softirqs.CPU118.RCU
     33096 ±  4%     +19.3%      39468 ±  3%  softirqs.CPU118.SCHED
     18313 ± 11%    +134.4%      42927 ±  5%  softirqs.CPU119.RCU
     21114 ±  6%    +127.6%      48057 ±  9%  softirqs.CPU12.RCU
     19537 ±  8%    +128.5%      44642 ±  5%  softirqs.CPU120.RCU
     19009 ± 10%    +128.4%      43409 ±  6%  softirqs.CPU121.RCU
     30437 ± 20%     +33.8%      40736 ±  3%  softirqs.CPU121.SCHED
     18764 ± 10%    +127.0%      42605 ±  6%  softirqs.CPU122.RCU
     34261 ±  2%     +18.2%      40503 ±  3%  softirqs.CPU122.SCHED
     19388 ± 10%    +123.6%      43346 ±  5%  softirqs.CPU123.RCU
     34412 ±  2%     +18.0%      40608 ±  3%  softirqs.CPU123.SCHED
     18814 ±  9%    +131.7%      43588 ±  7%  softirqs.CPU124.RCU
     34324 ±  2%     +18.7%      40736 ±  3%  softirqs.CPU124.SCHED
     18457 ± 11%    +129.0%      42271 ±  5%  softirqs.CPU125.RCU
     34462 ±  2%     +18.4%      40809 ±  3%  softirqs.CPU125.SCHED
     18867 ±  9%    +126.5%      42733 ±  7%  softirqs.CPU126.RCU
     34378 ±  2%     +18.2%      40624 ±  3%  softirqs.CPU126.SCHED
     18279 ± 10%    +133.8%      42730 ±  6%  softirqs.CPU127.RCU
     34155 ±  3%     +18.4%      40435 ±  3%  softirqs.CPU127.SCHED
     21170 ± 12%    +120.8%      46755 ±  8%  softirqs.CPU128.RCU
     34108 ±  3%     +18.6%      40454 ±  3%  softirqs.CPU128.SCHED
     21281 ±  9%    +121.3%      47090 ±  8%  softirqs.CPU129.RCU
     34169 ±  2%     +18.4%      40450 ±  3%  softirqs.CPU129.SCHED
     21165 ±  7%    +130.3%      48734 ±  7%  softirqs.CPU13.RCU
     21353 ± 10%    +121.3%      47265 ±  7%  softirqs.CPU130.RCU
     34224 ±  2%     +17.9%      40342 ±  3%  softirqs.CPU130.SCHED
     20225 ± 20%    +138.3%      48201 ±  7%  softirqs.CPU131.RCU
     35035 ±  2%     +15.7%      40553 ±  3%  softirqs.CPU131.SCHED
     21975 ±  8%    +119.0%      48118 ±  7%  softirqs.CPU132.RCU
     34274 ±  2%     +18.4%      40576 ±  3%  softirqs.CPU132.SCHED
     21462 ± 10%    +123.6%      47991 ±  8%  softirqs.CPU133.RCU
     34187 ±  2%     +18.6%      40544 ±  3%  softirqs.CPU133.SCHED
     21054 ±  9%    +123.3%      47021 ±  9%  softirqs.CPU134.RCU
     34105 ±  2%     +18.4%      40375 ±  3%  softirqs.CPU134.SCHED
     21711 ±  9%    +117.8%      47297 ±  7%  softirqs.CPU135.RCU
     30667 ± 20%     +32.0%      40488 ±  3%  softirqs.CPU135.SCHED
     21637 ± 10%    +121.7%      47963 ±  7%  softirqs.CPU136.RCU
     34521 ±  3%     +18.0%      40725 ±  3%  softirqs.CPU136.SCHED
     21523 ± 12%    +124.0%      48201 ±  6%  softirqs.CPU137.RCU
     34201 ±  2%     +18.8%      40631 ±  3%  softirqs.CPU137.SCHED
     21772 ± 11%    +117.1%      47279 ±  7%  softirqs.CPU138.RCU
     34240 ±  2%     +18.0%      40419 ±  3%  softirqs.CPU138.SCHED
     21384 ± 11%    +120.6%      47169 ±  8%  softirqs.CPU139.RCU
     34207 ±  2%     +18.5%      40519 ±  3%  softirqs.CPU139.SCHED
     21089 ±  8%    +132.6%      49060 ±  7%  softirqs.CPU14.RCU
     21446 ± 12%    +117.5%      46641 ±  7%  softirqs.CPU140.RCU
     34283 ±  2%     +17.9%      40413 ±  3%  softirqs.CPU140.SCHED
     21394 ± 11%    +122.1%      47526 ±  7%  softirqs.CPU141.RCU
     34262 ±  2%     +18.2%      40484 ±  3%  softirqs.CPU141.SCHED
     21510 ± 12%    +122.5%      47856 ±  7%  softirqs.CPU142.RCU
     34272 ±  2%     +18.4%      40571 ±  3%  softirqs.CPU142.SCHED
     21794 ± 11%    +119.3%      47806 ±  7%  softirqs.CPU143.RCU
     34379 ±  2%     +17.7%      40464 ±  3%  softirqs.CPU143.SCHED
     22081 ±  8%    +107.0%      45701 ± 12%  softirqs.CPU144.RCU
     21204 ±  7%    +112.7%      45101 ± 11%  softirqs.CPU145.RCU
     21354 ±  8%    +111.7%      45201 ± 11%  softirqs.CPU146.RCU
     34494 ±  3%     +12.4%      38783 ±  3%  softirqs.CPU146.SCHED
     22014 ± 11%    +106.0%      45348 ± 11%  softirqs.CPU147.RCU
     21731 ±  8%    +110.1%      45650 ± 12%  softirqs.CPU148.RCU
     21498 ±  9%    +111.3%      45427 ± 11%  softirqs.CPU149.RCU
     22409 ±  5%    +116.3%      48480 ±  9%  softirqs.CPU15.RCU
     14902 ± 81%     +93.8%      28882 ± 39%  softirqs.CPU15.SCHED
     21817 ±  9%    +109.0%      45604 ± 11%  softirqs.CPU150.RCU
     21296 ± 10%    +111.0%      44935 ± 10%  softirqs.CPU151.RCU
     21265 ± 10%    +110.4%      44740 ± 10%  softirqs.CPU152.RCU
     21349 ± 10%    +110.1%      44847 ± 12%  softirqs.CPU153.RCU
     21748 ± 10%    +109.3%      45530 ± 11%  softirqs.CPU154.RCU
     21913 ±  9%    +106.8%      45320 ± 11%  softirqs.CPU155.RCU
     22080 ±  9%    +106.3%      45544 ± 11%  softirqs.CPU156.RCU
     21072 ±  9%    +111.5%      44572 ± 11%  softirqs.CPU157.RCU
     34333 ±  2%     +11.0%      38100 ±  5%  softirqs.CPU157.SCHED
     20975 ±  9%    +108.3%      43687 ± 11%  softirqs.CPU158.RCU
     34232 ±  2%     +11.5%      38183 ±  4%  softirqs.CPU158.SCHED
     20796 ± 13%    +114.4%      44584 ± 11%  softirqs.CPU159.RCU
     34381 ±  3%     +14.5%      39356 ±  2%  softirqs.CPU159.SCHED
     19673 ±  4%    +122.2%      43719 ±  4%  softirqs.CPU16.RCU
     19575 ±  5%    +109.4%      40996 ± 13%  softirqs.CPU160.RCU
     34469 ±  2%     +14.7%      39527 ±  2%  softirqs.CPU160.SCHED
     20128 ±  6%    +106.8%      41620 ± 12%  softirqs.CPU161.RCU
     34494 ±  2%     +14.2%      39401 ±  2%  softirqs.CPU161.SCHED
     20309 ±  6%    +106.9%      42026 ± 12%  softirqs.CPU162.RCU
     34472 ±  2%     +12.9%      38928 ±  3%  softirqs.CPU162.SCHED
     20127 ±  5%    +110.0%      42274 ± 14%  softirqs.CPU163.RCU
     34424 ±  2%     +12.9%      38880 ±  4%  softirqs.CPU163.SCHED
     20556 ±  8%    +105.5%      42253 ± 13%  softirqs.CPU164.RCU
     34389 ±  3%     +15.5%      39725 ±  2%  softirqs.CPU164.SCHED
     20009 ±  6%    +106.5%      41325 ± 13%  softirqs.CPU165.RCU
     34500 ±  2%     +14.1%      39380 ±  3%  softirqs.CPU165.SCHED
     20229 ±  6%    +108.4%      42151 ± 11%  softirqs.CPU166.RCU
     34465 ±  2%     +14.5%      39463 ±  2%  softirqs.CPU166.SCHED
     20114 ±  6%    +107.4%      41721 ± 13%  softirqs.CPU167.RCU
     34473 ±  2%     +14.3%      39391 ±  2%  softirqs.CPU167.SCHED
     19291 ±  5%    +118.3%      42117 ±  9%  softirqs.CPU168.RCU
     18811 ±  4%    +116.9%      40810 ±  9%  softirqs.CPU169.RCU
     33446 ±  3%     +21.1%      40487 ±  2%  softirqs.CPU169.SCHED
     19397 ±  9%    +126.7%      43977 ±  3%  softirqs.CPU17.RCU
     29414 ± 18%     +34.8%      39648 ±  2%  softirqs.CPU17.SCHED
     19066 ±  3%    +117.2%      41415 ±  8%  softirqs.CPU170.RCU
     19220 ±  3%    +115.5%      41424 ±  6%  softirqs.CPU171.RCU
     18493 ±  4%    +118.4%      40385 ±  8%  softirqs.CPU172.RCU
     33551 ±  2%     +20.4%      40412 ±  2%  softirqs.CPU172.SCHED
     18962 ±  6%    +117.2%      41185 ± 10%  softirqs.CPU173.RCU
     33688 ±  2%     +20.6%      40620 ±  2%  softirqs.CPU173.SCHED
     18847 ±  4%    +117.1%      40924 ±  8%  softirqs.CPU174.RCU
     32749 ±  7%     +24.3%      40700 ±  2%  softirqs.CPU174.SCHED
     18706 ±  5%    +119.4%      41049 ±  8%  softirqs.CPU175.RCU
     33451 ±  3%     +21.6%      40667 ±  2%  softirqs.CPU175.SCHED
     20138 ±  5%    +121.5%      44614 ±  5%  softirqs.CPU176.RCU
     33630 ±  2%     +21.1%      40714 ±  3%  softirqs.CPU176.SCHED
     20307 ±  6%    +124.4%      45565 ±  5%  softirqs.CPU177.RCU
     20210 ±  5%    +121.3%      44725 ±  5%  softirqs.CPU178.RCU
     33463 ±  3%     +21.1%      40524 ±  3%  softirqs.CPU178.SCHED
     20060 ±  5%    +123.6%      44863 ±  5%  softirqs.CPU179.RCU
     33449 ±  3%     +21.4%      40607 ±  2%  softirqs.CPU179.SCHED
     19239 ± 11%    +130.5%      44347 ±  4%  softirqs.CPU18.RCU
     32081 ±  6%     +23.7%      39687        softirqs.CPU18.SCHED
     19882 ±  5%    +122.4%      44224 ±  6%  softirqs.CPU180.RCU
     33050 ±  4%     +21.6%      40189 ±  3%  softirqs.CPU180.SCHED
     19705 ±  5%    +123.3%      44010 ±  6%  softirqs.CPU181.RCU
     32993 ±  4%     +21.8%      40173 ±  3%  softirqs.CPU181.SCHED
     20406 ±  6%    +118.9%      44661 ±  5%  softirqs.CPU182.RCU
     33538 ±  4%     +20.6%      40448 ±  2%  softirqs.CPU182.SCHED
     20334 ±  5%    +120.9%      44912 ±  5%  softirqs.CPU183.RCU
     33533 ±  3%     +20.7%      40474 ±  2%  softirqs.CPU183.SCHED
     20526 ±  6%    +119.7%      45086 ±  5%  softirqs.CPU184.RCU
     33453 ±  3%     +21.3%      40581 ±  2%  softirqs.CPU184.SCHED
     20277 ±  4%    +126.0%      45818 ±  8%  softirqs.CPU185.RCU
     33871 ±  2%     +19.6%      40513 ±  2%  softirqs.CPU185.SCHED
     20432 ±  3%    +108.4%      42581 ± 14%  softirqs.CPU186.RCU
     33469 ±  3%     +23.6%      41373        softirqs.CPU186.SCHED
     20139 ±  5%    +123.8%      45066 ±  5%  softirqs.CPU187.RCU
     33367 ±  3%     +21.1%      40417 ±  3%  softirqs.CPU187.SCHED
     20437 ±  4%    +117.8%      44508 ±  5%  softirqs.CPU188.RCU
     28880 ± 29%     +39.2%      40207 ±  4%  softirqs.CPU188.SCHED
     20088 ±  5%    +123.2%      44842 ±  7%  softirqs.CPU189.RCU
     33280 ±  3%     +21.7%      40511 ±  3%  softirqs.CPU189.SCHED
     18485 ± 12%    +125.9%      41754 ±  5%  softirqs.CPU19.RCU
     33600 ±  5%     +16.6%      39178 ±  2%  softirqs.CPU19.SCHED
     20316 ±  5%    +120.0%      44689 ±  6%  softirqs.CPU190.RCU
     33252 ±  4%     +21.5%      40406 ±  3%  softirqs.CPU190.SCHED
     20199 ±  4%    +122.6%      44967 ±  5%  softirqs.CPU191.RCU
     33567 ±  2%     +17.2%      39326 ±  5%  softirqs.CPU191.SCHED
     21749 ±  9%    +122.2%      48331 ±  6%  softirqs.CPU2.RCU
     20410 ± 47%     +64.2%      33512 ± 20%  softirqs.CPU2.SCHED
     19165 ± 10%    +126.1%      43335 ±  4%  softirqs.CPU20.RCU
     33501 ±  4%     +17.5%      39351 ±  2%  softirqs.CPU20.SCHED
     18974 ±  9%    +130.0%      43644 ±  3%  softirqs.CPU21.RCU
     33673 ±  3%     +17.3%      39510 ±  2%  softirqs.CPU21.SCHED
     19311 ± 11%    +123.9%      43243 ±  4%  softirqs.CPU22.RCU
     32454 ±  7%     +21.2%      39319 ±  2%  softirqs.CPU22.SCHED
     20205 ± 12%    +114.6%      43369 ±  4%  softirqs.CPU23.RCU
     32717 ±  5%     +21.1%      39623        softirqs.CPU23.SCHED
     20750 ±  7%    +119.6%      45563 ±  5%  softirqs.CPU24.RCU
     20136 ± 12%    +118.9%      44076 ±  6%  softirqs.CPU25.RCU
     30631 ± 16%     +31.3%      40231 ±  3%  softirqs.CPU25.SCHED
     19508 ± 10%    +123.3%      43554 ±  5%  softirqs.CPU26.RCU
     33343 ±  2%     +21.8%      40603 ±  3%  softirqs.CPU26.SCHED
     19461 ± 10%    +123.0%      43398 ±  4%  softirqs.CPU27.RCU
     34046 ±  3%     +18.4%      40309 ±  3%  softirqs.CPU27.SCHED
     19034 ± 11%    +128.3%      43462 ±  5%  softirqs.CPU28.RCU
     33842 ±  3%     +19.6%      40485 ±  2%  softirqs.CPU28.SCHED
     19568 ±  9%    +120.8%      43207 ±  5%  softirqs.CPU29.RCU
     34076 ±  3%     +17.9%      40180 ±  3%  softirqs.CPU29.SCHED
     22102 ±  7%    +121.1%      48876 ±  6%  softirqs.CPU3.RCU
     11352 ±  4%    +179.4%      31715 ± 28%  softirqs.CPU3.SCHED
     19397 ± 11%    +121.7%      43004 ±  5%  softirqs.CPU30.RCU
     34011 ±  3%     +18.1%      40155 ±  3%  softirqs.CPU30.SCHED
     19813 ± 10%    +116.7%      42936 ±  5%  softirqs.CPU31.RCU
     28877 ± 28%     +38.9%      40106 ±  3%  softirqs.CPU31.SCHED
     22985 ±  7%    +113.7%      49109 ±  7%  softirqs.CPU32.RCU
     33988 ±  3%     +17.8%      40039 ±  3%  softirqs.CPU32.SCHED
     22657 ±  9%    +112.3%      48097 ±  8%  softirqs.CPU33.RCU
     33971 ±  3%     +18.3%      40201 ±  3%  softirqs.CPU33.SCHED
     22744 ±  8%    +116.9%      49325 ±  6%  softirqs.CPU34.RCU
     33842 ±  3%     +18.6%      40153 ±  3%  softirqs.CPU34.SCHED
     22016 ± 12%    +127.9%      50172 ±  7%  softirqs.CPU35.RCU
     33824 ±  3%     +19.0%      40256 ±  3%  softirqs.CPU35.SCHED
     22803 ±  9%    +117.7%      49644 ±  7%  softirqs.CPU36.RCU
     33848 ±  3%     +18.7%      40192 ±  3%  softirqs.CPU36.SCHED
     22024 ±  7%    +123.5%      49219 ±  7%  softirqs.CPU37.RCU
     33798 ±  3%     +14.8%      38789 ±  5%  softirqs.CPU37.SCHED
     22554 ±  6%    +118.9%      49362 ±  8%  softirqs.CPU38.RCU
     33803 ±  3%     +18.6%      40075 ±  3%  softirqs.CPU38.SCHED
     22068 ± 10%    +121.3%      48832 ±  7%  softirqs.CPU39.RCU
     22089 ±  6%    +119.8%      48551 ±  7%  softirqs.CPU4.RCU
     22095 ±  6%    +123.8%      49458 ±  6%  softirqs.CPU40.RCU
     33886 ±  3%     +18.7%      40212 ±  3%  softirqs.CPU40.SCHED
     22898 ±  9%    +117.3%      49761 ±  6%  softirqs.CPU41.RCU
     33889 ±  3%     +18.4%      40126 ±  3%  softirqs.CPU41.SCHED
     22668 ±  8%    +118.9%      49630 ±  7%  softirqs.CPU42.RCU
     33918 ±  3%     +18.4%      40156 ±  3%  softirqs.CPU42.SCHED
     22911 ±  9%    +116.9%      49699 ±  8%  softirqs.CPU43.RCU
     28225 ± 32%     +41.9%      40046 ±  3%  softirqs.CPU43.SCHED
     21873 ±  9%    +124.1%      49008 ±  7%  softirqs.CPU44.RCU
     33829 ±  3%     +19.0%      40251 ±  3%  softirqs.CPU44.SCHED
     22436 ±  7%    +117.4%      48777 ±  8%  softirqs.CPU45.RCU
     33869 ±  3%     +18.3%      40071 ±  3%  softirqs.CPU45.SCHED
     22360 ± 10%    +119.4%      49066 ±  7%  softirqs.CPU46.RCU
     33855 ±  3%     +18.6%      40139 ±  3%  softirqs.CPU46.SCHED
     22573 ±  8%    +119.2%      49487 ±  8%  softirqs.CPU47.RCU
     33944 ±  3%     +18.7%      40308 ±  3%  softirqs.CPU47.SCHED
     23683 ±  8%    +107.2%      49070 ± 10%  softirqs.CPU48.RCU
     22691 ±  8%    +112.6%      48236 ±  9%  softirqs.CPU49.RCU
     21696 ±  7%    +123.7%      48539 ±  7%  softirqs.CPU5.RCU
     22538 ±  9%    +115.6%      48603 ±  8%  softirqs.CPU50.RCU
     22540 ±  9%    +117.4%      48995 ±  8%  softirqs.CPU51.RCU
     22994 ± 12%    +113.7%      49148 ±  8%  softirqs.CPU52.RCU
     22465 ±  8%    +121.5%      49768 ±  7%  softirqs.CPU53.RCU
     22513 ±  9%    +120.3%      49589 ±  8%  softirqs.CPU54.RCU
     22411 ± 10%    +120.3%      49378 ±  9%  softirqs.CPU55.RCU
     21931 ± 10%    +124.4%      49215 ±  7%  softirqs.CPU56.RCU
     22160 ± 10%    +120.3%      48823 ±  9%  softirqs.CPU57.RCU
     22106 ±  8%    +123.4%      49379 ±  7%  softirqs.CPU58.RCU
     21863 ±  9%    +126.7%      49571 ±  8%  softirqs.CPU59.RCU
     21951 ±  7%    +108.2%      45699 ± 10%  softirqs.CPU6.RCU
     22433 ± 10%    +121.4%      49674 ±  8%  softirqs.CPU60.RCU
     21930 ±  8%    +122.4%      48767 ±  7%  softirqs.CPU61.RCU
     22191 ±  9%    +118.4%      48457 ±  7%  softirqs.CPU62.RCU
     20572 ± 19%    +137.3%      48814 ±  7%  softirqs.CPU63.RCU
     20606 ±  3%    +106.0%      42440 ± 14%  softirqs.CPU64.RCU
     34281 ±  3%     +15.6%      39616 ±  4%  softirqs.CPU64.SCHED
     20639 ±  6%    +108.0%      42919 ± 11%  softirqs.CPU65.RCU
     34090 ±  3%     +13.8%      38778 ±  2%  softirqs.CPU65.SCHED
     20711 ±  5%    +107.9%      43068 ± 12%  softirqs.CPU66.RCU
     34059 ±  3%     +13.5%      38643 ±  3%  softirqs.CPU66.SCHED
     20714 ±  6%    +108.8%      43250 ± 12%  softirqs.CPU67.RCU
     33981 ±  3%     +14.3%      38855 ±  3%  softirqs.CPU67.SCHED
     21132 ±  6%    +104.5%      43216 ± 13%  softirqs.CPU68.RCU
     34152 ±  3%     +13.9%      38908 ±  3%  softirqs.CPU68.SCHED
     20870 ±  6%    +105.5%      42895 ± 12%  softirqs.CPU69.RCU
     34035 ±  3%     +13.6%      38649 ±  3%  softirqs.CPU69.SCHED
     22131 ± 11%    +118.0%      48247 ±  6%  softirqs.CPU7.RCU
     20514 ±  6%    +110.1%      43090 ± 11%  softirqs.CPU70.RCU
     34016 ±  3%     +14.5%      38957 ±  2%  softirqs.CPU70.SCHED
     20526 ±  5%    +102.7%      41599 ± 15%  softirqs.CPU71.RCU
     32989 ±  7%     +21.2%      39982 ±  4%  softirqs.CPU71.SCHED
     20770 ±  5%    +119.3%      45551 ±  8%  softirqs.CPU72.RCU
     19814 ±  6%    +118.6%      43309 ±  7%  softirqs.CPU73.RCU
     20012 ±  4%    +112.5%      42519 ±  7%  softirqs.CPU74.RCU
     29526 ± 24%     +36.3%      40247 ±  3%  softirqs.CPU74.SCHED
     19752 ±  6%    +113.7%      42217 ±  8%  softirqs.CPU75.RCU
     28565 ± 30%     +40.8%      40210 ±  3%  softirqs.CPU75.SCHED
     19667 ±  3%    +112.2%      41740 ±  8%  softirqs.CPU76.RCU
     27517 ± 38%     +44.8%      39842 ±  3%  softirqs.CPU76.SCHED
     19604 ±  4%    +112.2%      41597 ±  8%  softirqs.CPU77.RCU
     27223 ± 40%     +46.7%      39935 ±  3%  softirqs.CPU77.SCHED
     20148          +107.4%      41792 ±  8%  softirqs.CPU78.RCU
     28971 ± 27%     +37.9%      39960 ±  3%  softirqs.CPU78.SCHED
     19633 ±  5%    +113.1%      41846 ±  8%  softirqs.CPU79.RCU
     29302 ± 25%     +36.4%      39968 ±  3%  softirqs.CPU79.SCHED
     21966 ±  9%    +123.5%      49088 ±  7%  softirqs.CPU8.RCU
     20793 ±  6%    +116.5%      45026 ±  4%  softirqs.CPU80.RCU
     27598 ± 37%     +44.6%      39897 ±  3%  softirqs.CPU80.SCHED
     20668 ±  5%    +120.7%      45615 ±  5%  softirqs.CPU81.RCU
     29054 ± 26%     +37.3%      39903 ±  3%  softirqs.CPU81.SCHED
     20726 ±  5%    +119.1%      45421 ±  5%  softirqs.CPU82.RCU
     28499 ± 30%     +40.2%      39960 ±  3%  softirqs.CPU82.SCHED
     19390 ±  8%    +134.3%      45429 ±  4%  softirqs.CPU83.RCU
     30213 ± 19%     +32.5%      40043 ±  2%  softirqs.CPU83.SCHED
     20393 ±  4%    +122.6%      45388 ±  5%  softirqs.CPU84.RCU
     30765 ± 14%     +28.9%      39667 ±  3%  softirqs.CPU84.SCHED
     20725 ±  2%    +118.5%      45293 ±  5%  softirqs.CPU85.RCU
     27760 ± 35%     +43.3%      39776 ±  3%  softirqs.CPU85.SCHED
     20580 ±  5%    +121.4%      45554 ±  5%  softirqs.CPU86.RCU
     20718 ±  5%    +118.7%      45301 ±  5%  softirqs.CPU87.RCU
     26915 ± 42%     +48.6%      40003 ±  3%  softirqs.CPU87.SCHED
     20763 ±  4%    +120.2%      45728 ±  5%  softirqs.CPU88.RCU
     30810 ± 15%     +29.3%      39842 ±  3%  softirqs.CPU88.SCHED
     21289 ±  2%    +111.8%      45082 ±  4%  softirqs.CPU89.RCU
     33487 ±  2%     +19.0%      39835 ±  3%  softirqs.CPU89.SCHED
     21579 ±  9%    +123.8%      48300 ±  8%  softirqs.CPU9.RCU
     20400 ±  3%    +120.9%      45058 ±  5%  softirqs.CPU90.RCU
     20371 ±  4%    +126.7%      46184 ±  4%  softirqs.CPU91.RCU
     32988 ±  3%     +21.4%      40061 ±  3%  softirqs.CPU91.SCHED
     20263 ±  3%    +122.6%      45097 ±  4%  softirqs.CPU92.RCU
     33192 ±  3%     +20.0%      39828 ±  4%  softirqs.CPU92.SCHED
     20239 ±  5%    +123.1%      45150 ±  4%  softirqs.CPU93.RCU
     32978 ±  3%     +21.0%      39900 ±  3%  softirqs.CPU93.SCHED
     20446 ±  5%    +131.9%      47421 ±  8%  softirqs.CPU94.RCU
     33136 ±  4%     +20.8%      40016 ±  3%  softirqs.CPU94.SCHED
     20498 ±  5%    +126.6%      46446 ±  4%  softirqs.CPU95.RCU
     19594 ± 16%    +130.5%      45172 ±  3%  softirqs.CPU96.RCU
     18967 ± 11%    +130.5%      43724 ±  4%  softirqs.CPU97.RCU
     33298 ±  6%     +14.7%      38208 ±  5%  softirqs.CPU97.SCHED
     19122 ± 17%    +132.9%      44527 ±  3%  softirqs.CPU98.RCU
     19392 ± 17%    +131.6%      44907 ±  4%  softirqs.CPU99.RCU
     32791 ±  4%     +20.8%      39609 ±  3%  softirqs.CPU99.SCHED
     15568 ± 14%    +109.9%      32681 ± 45%  softirqs.NET_RX
   3963598 ±  7%    +119.8%    8713905 ±  6%  softirqs.RCU
   6125987 ±  2%     +17.3%    7186965 ±  2%  softirqs.SCHED
    167016 ±  2%     +12.5%     187925        softirqs.TIMER
     31.37 ±  5%      -7.9       23.48 ±  2%  perf-profile.calltrace.cycles-pp.ret_from_fork
     31.37 ±  5%      -7.9       23.48 ±  2%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      7.60 ±  6%      -7.6        0.00        perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.xfs_map_blocks.iomap_writepage_map.write_cache_pages
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
     14.19 ±  7%      -7.5        6.64 ±  3%  perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__writeback_single_inode
      7.23 ±  7%      -7.2        0.00        perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_read_slowpath.xfs_map_blocks.iomap_writepage_map
     12.80 ±  7%      -7.2        5.57 ±  3%  perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
     29.14 ±  4%      -6.6       22.56 ±  2%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
     29.16 ±  4%      -5.9       23.28 ±  2%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      8.69 ±  6%      -5.5        3.17 ±  3%  perf-profile.calltrace.cycles-pp.xfs_map_blocks.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
      7.74 ±  6%      -5.4        2.38 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_map_blocks.iomap_writepage_map.write_cache_pages.iomap_writepages
     13.59 ± 11%      -5.0        8.57 ±  8%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.47 ± 11%      -5.0        8.49 ±  8%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     12.95 ± 11%      -4.8        8.16 ±  8%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     12.93 ± 11%      -4.8        8.14 ±  8%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      9.88 ± 12%      -3.8        6.08 ±  8%  perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      4.27 ±  8%      -1.6        2.70 ± 12%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      4.14 ±  8%      -1.5        2.60 ± 12%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      4.12 ±  8%      -1.5        2.58 ± 12%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      7.25 ±  3%      -1.3        5.90        perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
      2.47 ± 23%      -1.2        1.30 ±  4%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.25 ±  7%      -1.1        1.10 ±  3%  perf-profile.calltrace.cycles-pp.iomap_submit_ioend.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
      2.22 ±  7%      -1.1        1.08 ±  3%  perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.iomap_writepage_map.write_cache_pages.iomap_writepages
      2.19 ±  7%      -1.1        1.05 ±  4%  perf-profile.calltrace.cycles-pp.submit_bio_noacct.submit_bio.iomap_submit_ioend.iomap_writepage_map.write_cache_pages
      2.38 ± 25%      -1.1        1.25 ±  4%  perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.63 ± 11%      -1.1        1.50 ± 10%  perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.35 ± 25%      -1.1        1.23 ±  4%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply
      1.83 ±  7%      -1.0        0.80 ±  4%  perf-profile.calltrace.cycles-pp.blk_mq_submit_bio.submit_bio_noacct.submit_bio.iomap_submit_ioend.iomap_writepage_map
      6.06 ±  2%      -1.0        5.07        perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
      2.90 ± 10%      -0.9        1.97 ±  9%  perf-profile.calltrace.cycles-pp.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      5.79 ±  2%      -0.9        4.87 ±  2%  perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend
      2.24 ± 11%      -0.9        1.33 ± 10%  perf-profile.calltrace.cycles-pp.xfs_iext_lookup_extent.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.13 ±  8%      -0.9        1.26 ±  6%  perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
      2.10 ±  8%      -0.9        1.24 ±  6%  perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
      3.60 ±  2%      -0.7        2.88        perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
      3.46 ±  2%      -0.7        2.75        perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
      1.74 ±  5%      -0.7        1.04 ± 12%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.02 ±  8%      -0.6        0.46 ± 58%  perf-profile.calltrace.cycles-pp.__set_page_dirty.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply
      1.28 ± 18%      -0.6        0.72 ± 12%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.71 ± 12%      -0.5        1.18 ± 14%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.26 ±  9%      -0.5        0.72 ± 10%  perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.69 ± 12%      -0.5        1.16 ± 14%  perf-profile.calltrace.cycles-pp.xas_load.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.24 ±  5%      -0.5        0.75 ± 13%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.80 ±  3%      -0.5        1.32 ±  3%  perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend
      1.22 ±  9%      -0.5        0.74 ±  8%  perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io
      2.74 ±  3%      -0.5        2.27 ±  2%  perf-profile.calltrace.cycles-pp.xfs_btree_lookup.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten
      1.19 ±  8%      -0.5        0.72 ±  8%  perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend
      1.55 ±  6%      -0.4        1.18 ±  2%  perf-profile.calltrace.cycles-pp.xfs_btree_lookup_get_block.xfs_btree_lookup.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write
      1.01 ± 10%      -0.4        0.66 ±  6%  perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
      0.72 ±  8%      -0.3        0.39 ± 58%  perf-profile.calltrace.cycles-pp.xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_btree_read_buf_block
      1.19 ±  4%      -0.3        0.86 ±  4%  perf-profile.calltrace.cycles-pp.xfs_buf_item_format.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten
      1.25 ±  8%      -0.3        0.92 ±  3%  perf-profile.calltrace.cycles-pp.xfs_btree_read_buf_block.xfs_btree_lookup_get_block.xfs_btree_lookup.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten
      1.06 ±  9%      -0.3        0.78 ±  4%  perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_btree_read_buf_block.xfs_btree_lookup_get_block.xfs_btree_lookup.xfs_bmap_add_extent_unwritten_real
      0.83 ±  9%      -0.2        0.59 ±  4%  perf-profile.calltrace.cycles-pp.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_btree_read_buf_block.xfs_btree_lookup_get_block.xfs_btree_lookup
      0.77 ±  8%      -0.2        0.55 ±  5%  perf-profile.calltrace.cycles-pp.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_btree_read_buf_block.xfs_btree_lookup_get_block
      0.72 ±  5%      -0.2        0.54 ±  4%  perf-profile.calltrace.cycles-pp.xfs_trans_alloc.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
      0.84 ±  3%      -0.2        0.68 ±  2%  perf-profile.calltrace.cycles-pp.xfs_buf_item_size.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend
      0.77 ±  4%      -0.1        0.62 ±  2%  perf-profile.calltrace.cycles-pp.xfs_buf_item_size_segment.xfs_buf_item_size.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten
      0.70 ±  9%      +0.1        0.79 ±  4%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu
      1.18 ±  7%      +0.3        1.51 ±  4%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.17 ±  7%      +0.3        1.51 ±  4%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.17 ±  7%      +0.3        1.51 ±  4%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt
      0.82 ±  4%      +0.4        1.17 ±  6%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.40 ±  6%      +0.5        1.91 ±  4%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +0.5        0.53 ±  4%  perf-profile.calltrace.cycles-pp.xfs_btree_make_block_unfull.xfs_btree_insrec.xfs_btree_insert.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten
      0.00            +0.5        0.53 ±  2%  perf-profile.calltrace.cycles-pp.__blk_mq_run_hw_queue.process_one_work.worker_thread.kthread.ret_from_fork
      1.55 ±  3%      +0.5        2.09 ±  6%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.50 ±  2%      +0.5        2.04 ±  6%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +0.6        0.59 ±  7%  perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork
      2.10 ±  6%      +0.6        2.71 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      0.00            +0.6        0.65 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.7        0.65 ±  4%  perf-profile.calltrace.cycles-pp.__schedule.schedule.rwsem_down_read_slowpath.xfs_map_blocks.iomap_writepage_map
      0.00            +0.7        0.67 ±  3%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.14 ±  5%      +0.7        3.85 ±  5%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
      0.00            +0.8        0.75 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__queue_work
      0.00            +0.8        0.80 ±  3%  perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
      0.00            +0.8        0.81 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__queue_work.mod_delayed_work_on
      0.00            +0.8        0.83 ±  3%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__queue_work.mod_delayed_work_on.kblockd_mod_delayed_work_on
      0.00            +1.0        0.96 ± 10%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.xfs_ilock.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
      0.00            +1.1        1.05 ±  4%  perf-profile.calltrace.cycles-pp.xfs_btree_insrec.xfs_btree_insert.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write
      0.00            +1.1        1.08 ±  3%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__queue_work.mod_delayed_work_on.kblockd_mod_delayed_work_on.blk_mq_sched_insert_requests
      0.00            +1.1        1.11 ±  4%  perf-profile.calltrace.cycles-pp.xfs_btree_insert.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten
      0.00            +1.1        1.11 ± 10%  perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
      0.00            +1.2        1.16 ±  4%  perf-profile.calltrace.cycles-pp.__queue_work.mod_delayed_work_on.kblockd_mod_delayed_work_on.blk_mq_sched_insert_requests.blk_mq_flush_plug_list
     12.23 ±  2%      +1.2       13.40        perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
      0.00            +1.2        1.18 ±  4%  perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      0.00            +1.3        1.26 ±  4%  perf-profile.calltrace.cycles-pp.mod_delayed_work_on.kblockd_mod_delayed_work_on.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list
      0.00            +1.3        1.27 ±  4%  perf-profile.calltrace.cycles-pp.kblockd_mod_delayed_work_on.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list.schedule
      0.00            +1.4        1.35 ±  4%  perf-profile.calltrace.cycles-pp.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list.schedule.rwsem_down_read_slowpath
      0.00            +1.4        1.37 ±  4%  perf-profile.calltrace.cycles-pp.blk_mq_flush_plug_list.blk_flush_plug_list.schedule.rwsem_down_read_slowpath.xfs_map_blocks
      0.00            +1.4        1.38 ±  4%  perf-profile.calltrace.cycles-pp.blk_flush_plug_list.schedule.rwsem_down_read_slowpath.xfs_map_blocks.iomap_writepage_map
      0.00            +1.4        1.43 ± 18%  perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
      0.00            +1.6        1.59 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.wake_up_q
      0.00            +1.6        1.64 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.wake_up_q.rwsem_wake
      0.00            +1.6        1.65 ±  4%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.wake_up_q.rwsem_wake.xfs_iunlock
      4.93 ± 10%      +1.7        6.67 ±  5%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +2.0        1.97 ±  4%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +2.1        2.08 ±  3%  perf-profile.calltrace.cycles-pp.schedule.rwsem_down_read_slowpath.xfs_map_blocks.iomap_writepage_map.write_cache_pages
      0.00            +2.1        2.12 ±  8%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +2.4        2.40 ±  2%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.xfs_iunlock.xfs_iomap_write_unwritten
      0.00            +2.4        2.44 ±  2%  perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.xfs_iunlock.xfs_iomap_write_unwritten.xfs_end_ioend
      0.00            +2.7        2.73 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_wake.xfs_iunlock.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
      0.00            +2.8        2.80 ±  3%  perf-profile.calltrace.cycles-pp.xfs_iunlock.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
     30.62 ±  6%      +4.1       34.76 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     13.75 ±  7%      +4.3       18.07 ±  7%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     44.69            +9.2       53.92        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     46.17 ±  2%     +10.1       56.24        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     51.80 ±  2%     +13.1       64.85        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     51.82 ±  2%     +13.1       64.89        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     51.82 ±  2%     +13.1       64.89        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     52.27 ±  2%     +13.2       65.45        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      8.01 ±  6%      -8.0        0.00        perf-profile.children.cycles-pp.rwsem_optimistic_spin
     31.37 ±  5%      -7.9       23.48 ±  2%  perf-profile.children.cycles-pp.kthread
     31.37 ±  5%      -7.9       23.48 ±  2%  perf-profile.children.cycles-pp.ret_from_fork
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.wb_workfn
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.wb_writeback
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.__writeback_inodes_wb
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.writeback_sb_inodes
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.__writeback_single_inode
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.do_writepages
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.xfs_vm_writepages
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.iomap_writepages
     14.20 ±  7%      -7.6        6.64 ±  3%  perf-profile.children.cycles-pp.write_cache_pages
     12.81 ±  7%      -7.2        5.58 ±  3%  perf-profile.children.cycles-pp.iomap_writepage_map
      7.42 ±  6%      -7.1        0.35 ± 10%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
     29.14 ±  4%      -6.6       22.57 ±  2%  perf-profile.children.cycles-pp.process_one_work
     29.16 ±  4%      -5.9       23.29 ±  2%  perf-profile.children.cycles-pp.worker_thread
      8.70 ±  6%      -5.5        3.17 ±  4%  perf-profile.children.cycles-pp.xfs_map_blocks
      7.75 ±  6%      -5.4        2.38 ±  3%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
     14.73 ± 10%      -5.2        9.48 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     14.62 ± 10%      -5.2        9.40 ±  7%  perf-profile.children.cycles-pp.do_syscall_64
     13.96 ± 11%      -5.2        8.81 ±  8%  perf-profile.children.cycles-pp.ksys_write
     13.88 ± 10%      -5.1        8.76 ±  8%  perf-profile.children.cycles-pp.vfs_write
     13.61 ± 11%      -5.0        8.59 ±  8%  perf-profile.children.cycles-pp.new_sync_write
     13.48 ± 11%      -5.0        8.49 ±  8%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     12.95 ± 11%      -4.8        8.16 ±  8%  perf-profile.children.cycles-pp.iomap_file_buffered_write
     12.93 ± 11%      -4.8        8.14 ±  8%  perf-profile.children.cycles-pp.iomap_apply
      9.88 ± 12%      -3.8        6.09 ±  8%  perf-profile.children.cycles-pp.iomap_write_actor
      4.28 ±  8%      -1.6        2.70 ± 12%  perf-profile.children.cycles-pp.iomap_write_begin
      4.15 ±  8%      -1.5        2.60 ± 12%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      4.12 ±  8%      -1.5        2.59 ± 12%  perf-profile.children.cycles-pp.pagecache_get_page
      7.27 ±  3%      -1.4        5.91        perf-profile.children.cycles-pp.xfs_bmapi_write
      1.38 ± 41%      -1.2        0.19 ± 22%  perf-profile.children.cycles-pp.xas_store
      2.47 ± 23%      -1.2        1.30 ±  4%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      2.25 ±  7%      -1.1        1.10 ±  3%  perf-profile.children.cycles-pp.iomap_submit_ioend
      2.23 ±  7%      -1.1        1.08 ±  3%  perf-profile.children.cycles-pp.submit_bio
      2.20 ±  7%      -1.1        1.06 ±  4%  perf-profile.children.cycles-pp.submit_bio_noacct
      2.64 ± 11%      -1.1        1.50 ± 10%  perf-profile.children.cycles-pp.iomap_write_end
      2.38 ± 25%      -1.1        1.25 ±  4%  perf-profile.children.cycles-pp.copyin
      2.37 ± 25%      -1.1        1.24 ±  4%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      2.90 ±  9%      -1.1        1.79 ±  7%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      1.84 ±  7%      -1.0        0.81 ±  4%  perf-profile.children.cycles-pp.blk_mq_submit_bio
      6.07 ±  2%      -1.0        5.07        perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
      2.90 ± 10%      -0.9        1.97 ±  9%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      5.80 ±  2%      -0.9        4.88 ±  2%  perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
      2.13 ±  8%      -0.9        1.26 ±  6%  perf-profile.children.cycles-pp.iomap_finish_ioends
      2.11 ±  8%      -0.9        1.25 ±  6%  perf-profile.children.cycles-pp.iomap_finish_ioend
      3.65 ±  2%      -0.8        2.90        perf-profile.children.cycles-pp.__xfs_trans_commit
      3.51 ±  2%      -0.7        2.78        perf-profile.children.cycles-pp.xfs_log_commit_cil
      2.14 ±  9%      -0.7        1.44 ± 14%  perf-profile.children.cycles-pp.xas_load
      1.74 ±  5%      -0.7        1.04 ± 12%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      0.85 ±  4%      -0.7        0.18 ± 10%  perf-profile.children.cycles-pp.xfs_btree_delete
      0.84 ±  4%      -0.7        0.17 ± 11%  perf-profile.children.cycles-pp.xfs_btree_delrec
      1.28 ± 19%      -0.6        0.72 ± 12%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      1.72 ± 12%      -0.5        1.18 ± 14%  perf-profile.children.cycles-pp.find_get_entry
      1.26 ±  9%      -0.5        0.72 ± 10%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      0.61 ±  8%      -0.5        0.10 ± 10%  perf-profile.children.cycles-pp.blk_mq_try_issue_list_directly
      1.83 ±  3%      -0.5        1.33 ±  3%  perf-profile.children.cycles-pp.xlog_cil_insert_items
      0.60 ±  8%      -0.5        0.10 ± 10%  perf-profile.children.cycles-pp.blk_mq_request_issue_directly
      1.37 ±  6%      -0.5        0.89 ±  5%  perf-profile.children.cycles-pp.xfs_buf_read_map
      1.24 ±  5%      -0.5        0.76 ± 14%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      0.58 ±  7%      -0.5        0.10 ±  8%  perf-profile.children.cycles-pp.__blk_mq_try_issue_directly
      1.22 ±  9%      -0.5        0.74 ±  8%  perf-profile.children.cycles-pp.end_page_writeback
      1.21 ±  9%      -0.5        0.74 ±  8%  perf-profile.children.cycles-pp.test_clear_page_writeback
      2.78 ±  3%      -0.4        2.33 ±  2%  perf-profile.children.cycles-pp.xfs_btree_lookup
      1.02 ±  8%      -0.4        0.58 ± 12%  perf-profile.children.cycles-pp.__set_page_dirty
      1.56 ±  5%      -0.4        1.20 ±  2%  perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
      1.14 ±  6%      -0.4        0.77 ±  5%  perf-profile.children.cycles-pp.xfs_buf_get_map
      1.07 ±  7%      -0.4        0.72 ±  5%  perf-profile.children.cycles-pp.xfs_buf_find
      1.03 ± 10%      -0.4        0.68 ±  6%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.44 ±  5%      -0.3        0.10 ± 19%  perf-profile.children.cycles-pp.blk_attempt_plug_merge
      1.24 ±  4%      -0.3        0.90 ±  4%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.57 ±  3%      -0.3        0.23 ± 11%  perf-profile.children.cycles-pp.xfs_btree_readahead_lblock
      0.56 ±  3%      -0.3        0.23 ± 11%  perf-profile.children.cycles-pp.xfs_btree_reada_bufl
      0.54 ±  4%      -0.3        0.22 ± 12%  perf-profile.children.cycles-pp.xfs_buf_readahead_map
      0.64 ±  8%      -0.3        0.35 ± 12%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.33 ± 40%      -0.3        0.04 ± 59%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.47            -0.3        0.19 ±  7%  perf-profile.children.cycles-pp.xfs_btree_decrement
      0.34 ± 10%      -0.3        0.08 ± 14%  perf-profile.children.cycles-pp.osq_lock
      0.57 ±  9%      -0.3        0.32 ± 12%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.57 ±  7%      -0.2        0.32 ± 17%  perf-profile.children.cycles-pp.__xa_set_mark
      0.52 ±  9%      -0.2        0.29 ± 19%  perf-profile.children.cycles-pp.xas_set_mark
      0.46 ±  9%      -0.2        0.25 ±  7%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.48 ±  7%      -0.2        0.27 ± 10%  perf-profile.children.cycles-pp.lru_cache_add
      0.74 ±  4%      -0.2        0.55 ±  4%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.66 ±  2%      -0.2        0.46 ±  6%  perf-profile.children.cycles-pp.xfs_next_bit
      1.32 ±  8%      -0.2        1.13 ±  3%  perf-profile.children.cycles-pp.xfs_btree_read_buf_block
      0.26 ±  7%      -0.2        0.08 ± 10%  perf-profile.children.cycles-pp.xfs_iext_remove
      0.55 ± 14%      -0.2        0.38 ±  6%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      1.14 ±  9%      -0.2        0.96 ±  4%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.41 ±  7%      -0.2        0.24 ±  7%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.83 ±  4%      -0.2        0.66 ±  2%  perf-profile.children.cycles-pp.xfs_buf_item_size_segment
      0.85 ±  4%      -0.2        0.68 ±  2%  perf-profile.children.cycles-pp.xfs_buf_item_size
      0.37 ±  8%      -0.2        0.21 ± 12%  perf-profile.children.cycles-pp.rmqueue
      0.50 ±  5%      -0.2        0.34 ±  5%  perf-profile.children.cycles-pp.kmem_cache_free
      0.59 ±  7%      -0.2        0.43 ±  8%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.41 ±  4%      -0.2        0.26 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_charge
      0.59 ±  7%      -0.2        0.44 ±  8%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.24 ±  5%      -0.1        0.09 ±  7%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.48 ±  8%      -0.1        0.34 ±  8%  perf-profile.children.cycles-pp.__irqentry_text_start
      0.52 ±  5%      -0.1        0.38 ±  5%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.33 ±  4%      -0.1        0.19 ±  5%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.68 ±  5%      -0.1        0.54 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.32 ± 18%      -0.1        0.18 ± 17%  perf-profile.children.cycles-pp.account_page_dirtied
      0.31 ±  8%      -0.1        0.18 ±  8%  perf-profile.children.cycles-pp.down_write
      0.43 ±  9%      -0.1        0.30 ±  9%  perf-profile.children.cycles-pp.mempool_alloc
      0.27 ±  8%      -0.1        0.15 ± 18%  perf-profile.children.cycles-pp.workingset_update_node
      0.31 ± 11%      -0.1        0.20 ±  7%  perf-profile.children.cycles-pp.xfs_file_aio_write_checks
      0.25 ± 17%      -0.1        0.15 ± 15%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.22 ±  7%      -0.1        0.11 ± 14%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.55 ±  6%      -0.1        0.45 ±  3%  perf-profile.children.cycles-pp.memcpy_erms
      0.41 ±  4%      -0.1        0.31 ±  6%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.30 ±  7%      -0.1        0.20 ±  6%  perf-profile.children.cycles-pp.bio_free
      0.42 ± 11%      -0.1        0.32 ±  6%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.33 ±  2%      -0.1        0.23 ±  6%  perf-profile.children.cycles-pp.xfs_inode_item_format
      0.25 ± 15%      -0.1        0.15 ±  4%  perf-profile.children.cycles-pp.xfs_btree_del_cursor
      0.25 ±  7%      -0.1        0.15 ± 19%  perf-profile.children.cycles-pp.xfs_buf_trylock
      0.31 ± 21%      -0.1        0.21 ±  9%  perf-profile.children.cycles-pp.submit_bio_checks
      0.22 ±  7%      -0.1        0.12 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.22 ± 13%      -0.1        0.12 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.20 ± 12%      -0.1        0.10 ± 12%  perf-profile.children.cycles-pp.list_sort
      0.24 ±  8%      -0.1        0.15 ± 21%  perf-profile.children.cycles-pp.down_trylock
      0.25 ± 15%      -0.1        0.17 ±  9%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.23 ± 10%      -0.1        0.15 ± 12%  perf-profile.children.cycles-pp.__slab_free
      0.29 ±  9%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.xfs_bmapi_finish
      0.25 ±  7%      -0.1        0.17 ± 16%  perf-profile.children.cycles-pp.xfs_buf_unlock
      0.25 ±  6%      -0.1        0.17 ± 11%  perf-profile.children.cycles-pp.xfs_buf_rele
      0.11 ±  4%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.unlock_page
      0.35 ±  8%      -0.1        0.28 ±  4%  perf-profile.children.cycles-pp.xfs_buf_offset
      0.19 ± 11%      -0.1        0.11 ± 20%  perf-profile.children.cycles-pp.__xa_clear_mark
      0.28 ± 10%      -0.1        0.21 ±  4%  perf-profile.children.cycles-pp.xfs_trans_log_inode
      0.12 ± 23%      -0.1        0.05 ± 64%  perf-profile.children.cycles-pp.node_dirty_ok
      0.20 ± 16%      -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.up_write
      0.17 ± 15%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.xfs_trans_brelse
      0.23 ± 14%      -0.1        0.16 ± 13%  perf-profile.children.cycles-pp.bvec_alloc
      0.22 ± 18%      -0.1        0.15 ± 14%  perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
      0.18 ±  5%      -0.1        0.12 ± 22%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.24 ±  8%      -0.1        0.17 ± 19%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.23 ±  7%      -0.1        0.17 ± 20%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.16 ± 28%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.___slab_alloc
      0.15 ±  5%      -0.1        0.08 ± 21%  perf-profile.children.cycles-pp.page_mapping
      0.20 ± 19%      -0.1        0.14 ± 10%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.18 ± 18%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.xfs_perag_get
      0.16 ±  5%      -0.1        0.10 ±  7%  perf-profile.children.cycles-pp.memset_erms
      0.17 ±  6%      -0.1        0.11 ± 14%  perf-profile.children.cycles-pp.nvme_pci_complete_rq
      0.14 ± 21%      -0.1        0.08 ± 10%  perf-profile.children.cycles-pp.xfs_trans_ijoin
      0.08 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_iext_update_node
      0.21 ± 15%      -0.1        0.15 ±  9%  perf-profile.children.cycles-pp.xfs_bmbt_init_cursor
      0.12 ±  6%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.21 ± 11%      -0.1        0.15 ±  4%  perf-profile.children.cycles-pp.xlog_ticket_alloc
      0.10 ± 15%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.xas_start
      0.08 ± 19%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_vn_update_time
      0.28 ±  6%      -0.1        0.23 ±  5%  perf-profile.children.cycles-pp.nvme_map_data
      0.16 ± 12%      -0.1        0.10 ± 25%  perf-profile.children.cycles-pp.try_charge
      0.08 ± 19%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_file_write_iter
      0.08 ± 14%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_break_layouts
      0.07 ± 15%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.xfs_inode_item_format_data_fork
      0.14 ± 10%      -0.0        0.10 ±  9%  perf-profile.children.cycles-pp.dec_zone_page_state
      0.14 ± 13%      -0.0        0.10 ± 11%  perf-profile.children.cycles-pp.__blk_queue_split
      0.13 ± 12%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.file_update_time
      0.12 ±  5%      -0.0        0.08 ± 16%  perf-profile.children.cycles-pp.up
      0.07 ±  7%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.kfree
      0.07 ±  7%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.xfs_isilocked
      0.10 ± 17%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_mod_fdblocks
      0.24 ±  5%      -0.0        0.20 ±  8%  perf-profile.children.cycles-pp.xfs_verify_agbno
      0.09 ±  7%      -0.0        0.06 ± 58%  perf-profile.children.cycles-pp.xfs_trans_del_item
      0.10 ± 11%      -0.0        0.06 ± 16%  perf-profile.children.cycles-pp.xa_get_order
      0.09 ±  9%      -0.0        0.06 ± 15%  perf-profile.children.cycles-pp._atomic_dec_and_lock
      0.10 ± 19%      -0.0        0.07 ± 29%  perf-profile.children.cycles-pp.blk_mq_rq_ctx_init
      0.19 ±  7%      -0.0        0.16 ±  8%  perf-profile.children.cycles-pp.xfs_trans_log_buf
      0.10 ±  8%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.__sb_start_write
      0.31            -0.0        0.28 ±  3%  perf-profile.children.cycles-pp.__xfs_btree_check_lblock
      0.09 ±  9%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
      0.15 ±  5%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.xfs_trans_dirty_buf
      0.09 ±  8%      -0.0        0.07 ± 13%  perf-profile.children.cycles-pp.blk_throtl_bio
      0.07 ±  6%      -0.0        0.05        perf-profile.children.cycles-pp.xfs_iext_update_extent
      0.06 ± 11%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__list_add_valid
      0.06 ± 16%      +0.0        0.08        perf-profile.children.cycles-pp.rb_insert_color
      0.11 ±  9%      +0.0        0.14 ±  8%  perf-profile.children.cycles-pp.xfs_end_bio
      0.09 ±  7%      +0.0        0.12 ±  9%  perf-profile.children.cycles-pp.blk_mq_start_request
      0.09 ±  7%      +0.0        0.12 ± 10%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.23 ±  6%      +0.0        0.27 ±  4%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.08 ± 19%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp._find_next_bit
      0.03 ±100%      +0.0        0.07 ±  6%  perf-profile.children.cycles-pp.xfs_btree_update_keys
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.rb_erase
      0.04 ± 58%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.11 ±  7%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.08 ± 17%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.call_cpuidle
      0.06 ± 60%      +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.cpumask_next_and
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.blk_mq_flush_busy_ctxs
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.update_cfs_rq_h_load
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.__xfs_btree_split
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
      0.16 ±  7%      +0.1        0.23 ± 15%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.00            +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.stack_access_ok
      0.00            +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.wq_worker_running
      0.00            +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.resched_curr
      0.09            +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.xfs_iext_insert
      0.12 ±  7%      +0.1        0.20 ± 10%  perf-profile.children.cycles-pp._cond_resched
      0.71 ±  9%      +0.1        0.80 ±  4%  perf-profile.children.cycles-pp.rebalance_domains
      0.11 ± 15%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.00            +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.in_sched_functions
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.xfs_btree_split_worker
      0.28 ± 11%      +0.1        0.38 ±  9%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.10 ± 12%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.00            +0.1        0.10 ± 21%  perf-profile.children.cycles-pp.__module_text_address
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.__unwind_start
      0.00            +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.preempt_schedule_common
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.00            +0.1        0.11 ± 19%  perf-profile.children.cycles-pp.is_module_text_address
      0.00            +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.21 ±  5%      +0.1        0.32 ±  4%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.26 ±  7%      +0.1        0.37 ±  7%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.1        0.12 ± 27%  perf-profile.children.cycles-pp.rcu_gp_kthread
      0.00            +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.00            +0.1        0.12 ±  8%  perf-profile.children.cycles-pp.___perf_sw_event
      0.00            +0.1        0.14 ± 15%  perf-profile.children.cycles-pp.__module_address
      0.00            +0.1        0.15 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.55 ±  5%      +0.2        0.70 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.47 ±  6%      +0.2        0.65 ±  4%  perf-profile.children.cycles-pp.native_sched_clock
      0.20 ± 18%      +0.2        0.37 ±  2%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.00            +0.2        0.17 ± 10%  perf-profile.children.cycles-pp.xfs_btree_rshift
      0.00            +0.2        0.18 ± 11%  perf-profile.children.cycles-pp.flush_smp_call_function_from_idle
      0.00            +0.2        0.18 ±  8%  perf-profile.children.cycles-pp.update_curr
      0.20 ± 11%      +0.2        0.38 ± 12%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.2        0.18 ± 10%  perf-profile.children.cycles-pp.kernel_text_address
      0.50 ±  6%      +0.2        0.68 ±  3%  perf-profile.children.cycles-pp.sched_clock
      0.05 ± 59%      +0.2        0.24 ± 11%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.2        0.19        perf-profile.children.cycles-pp.check_preempt_curr
      0.13 ±  9%      +0.2        0.33 ±  3%  perf-profile.children.cycles-pp.balance_dirty_pages
      0.00            +0.2        0.20 ± 10%  perf-profile.children.cycles-pp.note_gp_changes
      0.13 ± 14%      +0.2        0.33 ±  4%  perf-profile.children.cycles-pp.xfs_btree_lshift
      0.00            +0.2        0.21 ±  8%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.11 ± 13%      +0.2        0.32 ± 11%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.09 ± 20%      +0.2        0.30 ±  7%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.2        0.21 ±  2%  perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
      0.00            +0.2        0.21 ±  6%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.09 ± 21%      +0.2        0.30 ±  9%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.00            +0.2        0.22 ± 11%  perf-profile.children.cycles-pp.finish_task_switch
      0.00            +0.2        0.22 ± 10%  perf-profile.children.cycles-pp.__kernel_text_address
      0.00            +0.2        0.23 ±  9%  perf-profile.children.cycles-pp.__switch_to_asm
      0.00            +0.2        0.24 ±  5%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.00            +0.2        0.24 ±  5%  perf-profile.children.cycles-pp.set_next_entity
      0.03 ±102%      +0.2        0.28 ±  2%  perf-profile.children.cycles-pp.io_schedule_timeout
      0.00            +0.2        0.25 ± 11%  perf-profile.children.cycles-pp.__switch_to
      0.00            +0.3        0.26 ±  6%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.00            +0.3        0.27 ±  5%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.08 ± 20%      +0.3        0.36 ±  8%  perf-profile.children.cycles-pp.schedule_timeout
      0.13 ± 22%      +0.3        0.42 ±  7%  perf-profile.children.cycles-pp.idle_cpu
      0.00            +0.3        0.33        perf-profile.children.cycles-pp.dequeue_entity
      0.27 ±  7%      +0.3        0.61 ±  8%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.3        0.34 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      0.87 ±  4%      +0.3        1.21 ±  6%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.4        0.35 ± 10%  perf-profile.children.cycles-pp.orc_find
      1.20 ±  6%      +0.4        1.56 ±  3%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.00            +0.4        0.36 ± 11%  perf-profile.children.cycles-pp.__orc_find
      1.19 ±  6%      +0.4        1.56 ±  3%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.00            +0.4        0.38 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.00            +0.4        0.43 ±  5%  perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list
      0.00            +0.5        0.51 ±  2%  perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
      0.00            +0.5        0.52 ±  3%  perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
      1.63 ±  3%      +0.5        2.15 ±  6%  perf-profile.children.cycles-pp.tick_sched_handle
      0.31 ± 18%      +0.5        0.83 ±  2%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.32 ± 17%      +0.5        0.85 ±  3%  perf-profile.children.cycles-pp.find_busiest_group
      0.00            +0.5        0.53 ±  4%  perf-profile.children.cycles-pp.xfs_btree_make_block_unfull
      0.00            +0.5        0.53 ±  2%  perf-profile.children.cycles-pp.__blk_mq_run_hw_queue
      1.58 ±  2%      +0.5        2.12 ±  6%  perf-profile.children.cycles-pp.update_process_times
      1.50 ±  6%      +0.5        2.03 ±  4%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.42 ± 14%      +0.5        0.96 ±  2%  perf-profile.children.cycles-pp.load_balance
      0.10 ± 16%      +0.6        0.67 ±  2%  perf-profile.children.cycles-pp.newidle_balance
      2.19 ±  6%      +0.6        2.79 ±  8%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.7        0.68 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      3.26 ±  6%      +0.7        3.96 ±  5%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.29 ±  6%      +0.8        1.06 ±  3%  perf-profile.children.cycles-pp.xfs_btree_insrec
      0.31 ±  6%      +0.8        1.12 ±  4%  perf-profile.children.cycles-pp.xfs_btree_insert
      0.80 ± 12%      +0.8        1.62 ±  7%  perf-profile.children.cycles-pp.xfs_ilock
      0.61 ±  8%      +0.8        1.45 ±  3%  perf-profile.children.cycles-pp.blk_mq_sched_insert_requests
      0.62 ±  8%      +0.9        1.47 ±  4%  perf-profile.children.cycles-pp.blk_mq_flush_plug_list
      0.62 ±  9%      +0.9        1.48 ±  3%  perf-profile.children.cycles-pp.blk_flush_plug_list
      0.09 ± 23%      +0.9        1.04 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.44 ± 17%      +1.0        1.41 ±  7%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
     12.25 ±  2%      +1.2       13.41        perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
      0.00            +1.2        1.19 ±  5%  perf-profile.children.cycles-pp.unwind_next_frame
      0.00            +1.2        1.22 ±  3%  perf-profile.children.cycles-pp.__queue_work
      0.00            +1.3        1.27 ±  4%  perf-profile.children.cycles-pp.mod_delayed_work_on
      0.00            +1.3        1.27 ±  4%  perf-profile.children.cycles-pp.kblockd_mod_delayed_work_on
      0.42 ± 10%      +1.7        2.16 ±  9%  perf-profile.children.cycles-pp.poll_idle
      4.96 ± 11%      +1.8        6.72 ±  5%  perf-profile.children.cycles-pp.menu_select
      0.01 ±173%      +1.8        1.84 ±  3%  perf-profile.children.cycles-pp.arch_stack_walk
      0.01 ±173%      +2.0        2.00 ±  3%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.03 ±100%      +2.2        2.22 ±  4%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.15 ± 14%      +2.4        2.50 ±  2%  perf-profile.children.cycles-pp.__schedule
      0.00            +2.5        2.48 ±  2%  perf-profile.children.cycles-pp.wake_up_q
      0.06 ± 13%      +2.6        2.67 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      0.07 ± 14%      +2.7        2.78 ±  3%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.30 ±  8%      +2.7        3.03 ±  3%  perf-profile.children.cycles-pp.xfs_iunlock
      0.07 ± 14%      +2.7        2.81 ±  3%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +2.8        2.79 ±  3%  perf-profile.children.cycles-pp.rwsem_wake
      0.12 ± 13%      +3.1        3.20 ±  3%  perf-profile.children.cycles-pp.schedule
     12.03 ±  7%      +3.4       15.39 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.10 ± 13%      +3.6        3.67 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
     30.81 ±  6%      +4.0       34.84 ±  2%  perf-profile.children.cycles-pp.intel_idle
     46.58 ±  2%     +10.1       56.71        perf-profile.children.cycles-pp.cpuidle_enter_state
     46.61 ±  2%     +10.1       56.75        perf-profile.children.cycles-pp.cpuidle_enter
     51.82 ±  2%     +13.1       64.89        perf-profile.children.cycles-pp.start_secondary
     52.27 ±  2%     +13.2       65.44        perf-profile.children.cycles-pp.do_idle
     52.27 ±  2%     +13.2       65.45        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     52.27 ±  2%     +13.2       65.45        perf-profile.children.cycles-pp.cpu_startup_entry
      7.25 ±  7%      -7.0        0.28 ±  6%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      2.36 ± 25%      -1.1        1.23 ±  4%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      2.85 ±  9%      -1.1        1.74 ±  7%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      2.04 ±  9%      -0.6        1.40 ± 13%  perf-profile.self.cycles-pp.xas_load
      1.27 ± 19%      -0.6        0.72 ± 12%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      0.33 ± 40%      -0.3        0.04 ± 59%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.34 ± 11%      -0.3        0.08 ± 11%  perf-profile.self.cycles-pp.osq_lock
      0.45 ± 16%      -0.2        0.22 ± 14%  perf-profile.self.cycles-pp.iomap_finish_ioend
      0.52 ±  9%      -0.2        0.29 ± 19%  perf-profile.self.cycles-pp.xas_set_mark
      0.58 ±  3%      -0.2        0.42 ±  5%  perf-profile.self.cycles-pp.xfs_next_bit
      0.49 ±  6%      -0.1        0.34 ±  7%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.16 ±  5%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.xfs_iext_remove
      0.38 ±  8%      -0.1        0.24 ±  4%  perf-profile.self.cycles-pp.xfs_buf_find
      0.15 ± 10%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.blk_attempt_plug_merge
      0.26 ±  9%      -0.1        0.15 ± 18%  perf-profile.self.cycles-pp.workingset_update_node
      0.26 ± 15%      -0.1        0.15 ±  5%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.42 ±  4%      -0.1        0.32 ±  8%  perf-profile.self.cycles-pp.xfs_buf_item_format
      0.22            -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.24 ± 16%      -0.1        0.14 ± 15%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.22 ±  9%      -0.1        0.12 ± 13%  perf-profile.self.cycles-pp.down_write
      0.23 ± 16%      -0.1        0.13 ±  6%  perf-profile.self.cycles-pp.test_clear_page_writeback
      0.20 ± 12%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.31 ±  9%      -0.1        0.23 ± 17%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.51 ±  6%      -0.1        0.43 ±  4%  perf-profile.self.cycles-pp.memcpy_erms
      0.22 ± 12%      -0.1        0.14 ± 15%  perf-profile.self.cycles-pp.__slab_free
      0.10 ± 22%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.unlock_page
      0.19 ± 15%      -0.1        0.12 ± 15%  perf-profile.self.cycles-pp.up_write
      0.18 ±  5%      -0.1        0.11 ± 21%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.22 ±  9%      -0.1        0.15 ± 21%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.17 ± 11%      -0.1        0.10 ± 19%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.31 ±  6%      -0.1        0.24 ±  3%  perf-profile.self.cycles-pp.xfs_buf_item_size_segment
      0.16 ±  7%      -0.1        0.09 ± 11%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.20 ± 18%      -0.1        0.14 ± 11%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.16 ±  5%      -0.1        0.10 ± 13%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.23 ±  7%      -0.1        0.17 ±  8%  perf-profile.self.cycles-pp.kmem_cache_free
      0.28 ±  6%      -0.1        0.22 ±  5%  perf-profile.self.cycles-pp.xfs_buf_offset
      0.18 ± 16%      -0.1        0.12 ± 20%  perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
      0.16 ± 15%      -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.__might_sleep
      0.14 ± 10%      -0.1        0.08 ± 21%  perf-profile.self.cycles-pp.page_mapping
      0.15 ± 16%      -0.1        0.10 ± 15%  perf-profile.self.cycles-pp.down_read
      0.08 ± 26%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.account_page_dirtied
      0.28 ±  4%      -0.1        0.22 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.14 ±  5%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.memset_erms
      0.08 ± 19%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.xfs_file_write_iter
      0.09 ± 15%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.xas_start
      0.12 ±  3%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.21 ±  6%      -0.1        0.16 ±  7%  perf-profile.self.cycles-pp.xfs_inode_item_format
      0.11 ± 17%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.iomap_apply
      0.12 ±  9%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.list_sort
      0.07 ± 17%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.iomap_write_end
      0.10 ± 15%      -0.0        0.05 ± 58%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.07 ± 15%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.xfs_file_llseek
      0.07 ± 24%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      0.07 ± 17%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.__entry_text_start
      0.16 ± 12%      -0.0        0.11 ± 14%  perf-profile.self.cycles-pp.xfs_trans_log_inode
      0.07 ±  6%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.try_charge
      0.12 ± 15%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.__get_user_nocheck_1
      0.08 ±  5%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.xfs_log_ticket_ungrant
      0.18 ±  6%      -0.0        0.14 ± 15%  perf-profile.self.cycles-pp.xfs_bmap_add_extent_unwritten_real
      0.09 ± 11%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.rmqueue
      0.12 ± 14%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.__blk_queue_split
      0.14 ± 12%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.xfs_btree_lookup_get_block
      0.12 ± 13%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
      0.19 ±  5%      -0.0        0.16 ±  2%  perf-profile.self.cycles-pp.iomap_writepage_map
      0.10 ± 14%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.iomap_write_actor
      0.08 ± 13%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp._atomic_dec_and_lock
      0.10 ± 14%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.xfs_perag_get
      0.07 ±  7%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.xfs_buf_get_map
      0.11 ±  7%      -0.0        0.09 ± 10%  perf-profile.self.cycles-pp.dec_zone_page_state
      0.07 ±  7%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.xfs_perag_put
      0.08 ±  6%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.xfs_errortag_test
      0.16 ±  8%      +0.0        0.18 ± 11%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.04 ± 58%      +0.0        0.07 ± 10%  perf-profile.self.cycles-pp.__list_add_valid
      0.05 ± 60%      +0.0        0.08        perf-profile.self.cycles-pp.rb_insert_color
      0.08 ± 14%      +0.0        0.11 ± 14%  perf-profile.self.cycles-pp._find_next_bit
      0.07 ±  7%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.xfs_iext_insert
      0.14 ± 22%      +0.0        0.18 ±  7%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.14 ± 16%      +0.0        0.19 ±  6%  perf-profile.self.cycles-pp.rwsem_down_read_slowpath
      0.01 ±173%      +0.0        0.06 ± 14%  perf-profile.self.cycles-pp.scheduler_tick
      0.04 ± 58%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.08 ± 17%      +0.1        0.13 ±  9%  perf-profile.self.cycles-pp.call_cpuidle
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.in_sched_functions
      0.07 ± 13%      +0.1        0.12 ±  6%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.04 ± 57%      +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.rcu_eqs_exit
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.kernel_text_address
      0.00            +0.1        0.07 ± 13%  perf-profile.self.cycles-pp.schedule
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.xfs_btree_insrec
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.update_cfs_rq_h_load
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.finish_task_switch
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.arch_stack_walk
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.stack_access_ok
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.note_gp_changes
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.__unwind_start
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.dequeue_entity
      0.00            +0.1        0.08 ± 15%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.00            +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.resched_curr
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.00            +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.1        0.10 ± 13%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.28 ± 10%      +0.1        0.38 ±  9%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.___perf_sw_event
      0.00            +0.1        0.11 ± 17%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.00            +0.1        0.11 ±  3%  perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
      0.00            +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.__module_address
      0.00            +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.stack_trace_save_tsk
      0.00            +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.set_next_entity
      0.00            +0.1        0.14 ±  9%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.18 ±  8%      +0.2        0.33 ± 13%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.2        0.16 ± 13%  perf-profile.self.cycles-pp.try_to_wake_up
      0.00            +0.2        0.16 ±  4%  perf-profile.self.cycles-pp.orc_find
      0.46 ±  6%      +0.2        0.62 ±  4%  perf-profile.self.cycles-pp.native_sched_clock
      1.02 ±  6%      +0.2        1.20 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.20 ±  4%      +0.2        0.38 ±  5%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.2        0.19 ±  8%  perf-profile.self.cycles-pp.enqueue_entity
      0.00            +0.2        0.21 ±  5%  perf-profile.self.cycles-pp.rwsem_mark_wake
      0.09 ± 24%      +0.2        0.30 ±  6%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.10 ±  7%      +0.2        0.32 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.23 ± 10%  perf-profile.self.cycles-pp.__switch_to
      0.00            +0.2        0.23 ±  9%  perf-profile.self.cycles-pp.__switch_to_asm
      0.06 ± 17%      +0.3        0.32 ± 11%  perf-profile.self.cycles-pp.update_rq_clock
      0.12 ± 21%      +0.3        0.41 ±  6%  perf-profile.self.cycles-pp.idle_cpu
      0.00            +0.3        0.33 ±  7%  perf-profile.self.cycles-pp.__schedule
      0.21 ± 19%      +0.4        0.57 ±  2%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.4        0.36 ± 11%  perf-profile.self.cycles-pp.__orc_find
      0.00            +0.5        0.51 ±  4%  perf-profile.self.cycles-pp.unwind_next_frame
      0.00            +0.9        0.90 ± 10%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      2.50 ± 23%      +1.6        4.11 ±  6%  perf-profile.self.cycles-pp.menu_select
      0.34 ± 12%      +1.7        2.02 ±  9%  perf-profile.self.cycles-pp.poll_idle
      3.77 ± 21%      +1.8        5.52 ±  8%  perf-profile.self.cycles-pp.cpuidle_enter_state
     30.80 ±  6%      +4.0       34.84 ±  2%  perf-profile.self.cycles-pp.intel_idle



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.10.0-rc3-00006-g10a59003d29f" of type "text/plain" (170093 bytes)

View attachment "job-script" of type "text/plain" (7805 bytes)

View attachment "job.yaml" of type "text/plain" (5182 bytes)

View attachment "reproduce" of type "text/plain" (290 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ