lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Thu, 12 Sep 2019 08:58:27 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Darrick J. Wong" <darrick.wong@...cle.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...org
Subject: [xfs] 1aabb011bf: aim7.jobs-per-min -17.8% regression

Greeting,

FYI, we noticed a -17.8% regression of aim7.jobs-per-min due to commit:


commit: 1aabb011bfba5a8942d3e0d0c84a4ac7cad277b1 ("xfs: deferred inode inactivation")
https://kernel.googlesource.com/pub/scm/linux/kernel/git/djwong/xfs-linux.git rough-fixes

in testcase: aim7
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: xfs
	test: disk_rw
	load: 3000
	cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min -35.7% regression                             |
| test machine     | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory |
| test parameters  | cpufreq_governor=performance                                          |
|                  | disk=4BRD_12G                                                         |
|                  | fs=xfs                                                                |
|                  | load=3000                                                             |
|                  | md=RAID0                                                              |
|                  | test=disk_cp                                                          |
+------------------+-----------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID0/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_rw/aim7

commit: 
  d515962b00 ("xfs: pass around xfs_inode_ag_walk iget/irele helper functions")
  1aabb011bf ("xfs: deferred inode inactivation")

d515962b00d64454 1aabb011bfba5a8942d3e0d0c84 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    139763           -17.8%     114843        aim7.jobs-per-min
    129.25           +21.6%     157.23        aim7.time.elapsed_time
    129.25           +21.6%     157.23        aim7.time.elapsed_time.max
     31367 ±  3%     -39.2%      19066 ± 11%  aim7.time.involuntary_context_switches
      2862            -7.1%       2659        aim7.time.system_time
    798.18            -5.9%     751.02        aim7.time.user_time
   1049629           -21.6%     822793        aim7.time.voluntary_context_switches
 1.016e+09 ± 12%    +111.5%  2.149e+09 ± 11%  cpuidle.C6.time
   1873251 ±  7%     +86.0%    3483843 ± 18%  cpuidle.C6.usage
      2956 ± 10%     +25.2%       3701 ± 17%  cpuidle.POLL.usage
     29.00           +15.7       44.70 ±  2%  mpstat.cpu.all.idle%
     55.41           -12.2       43.21 ±  2%  mpstat.cpu.all.sys%
     15.55            -3.5       12.07        mpstat.cpu.all.usr%
     30.00           +51.2%      45.35 ±  2%  iostat.cpu.idle
     54.65           -21.8%      42.72 ±  2%  iostat.cpu.system
     15.34           -22.2%      11.93        iostat.cpu.user
     42.06 ± 11%    +139.1%     100.59 ±  7%  iostat.md0.w/s
    115553           +14.5%     132340 ±  3%  meminfo.AnonHugePages
     95312           +20.4%     114787        meminfo.KReclaimable
     95312           +20.4%     114787        meminfo.SReclaimable
    261148           +13.4%     296269        meminfo.Slab
     31512           -16.2%      26409 ±  2%  meminfo.max_used_kB
      2740 ± 44%    +108.3%       5707 ± 11%  numa-vmstat.node0.nr_shmem
     10048 ± 35%     +81.5%      18236 ± 11%  numa-vmstat.node0.nr_slab_reclaimable
     16555 ± 52%     +74.7%      28924 ± 28%  numa-vmstat.node0.nr_slab_unreclaimable
      3417 ± 16%     -20.1%       2730 ± 20%  numa-vmstat.node1.nr_mapped
      4555 ± 27%     -66.3%       1533 ± 53%  numa-vmstat.node1.nr_shmem
     29.50           +52.5%      45.00 ±  2%  vmstat.cpu.id
     54.25           -22.6%      42.00 ±  2%  vmstat.cpu.sy
     15.00           -26.7%      11.00        vmstat.cpu.us
     27.50           -22.7%      21.25 ±  2%  vmstat.procs.r
     17205            -9.9%      15509        vmstat.system.cs
     40194 ± 35%     +81.5%      72950 ± 11%  numa-meminfo.node0.KReclaimable
     40194 ± 35%     +81.5%      72950 ± 11%  numa-meminfo.node0.SReclaimable
     66241 ± 52%     +74.7%     115712 ± 28%  numa-meminfo.node0.SUnreclaim
     10965 ± 44%    +108.3%      22838 ± 11%  numa-meminfo.node0.Shmem
    106435 ± 44%     +77.3%     188663 ± 18%  numa-meminfo.node0.Slab
     13273 ± 15%     -18.7%      10793 ± 18%  numa-meminfo.node1.Mapped
     18226 ± 27%     -66.3%       6135 ± 53%  numa-meminfo.node1.Shmem
    346095            -2.0%     339002        proc-vmstat.nr_dirty
    346482            -2.0%     339409        proc-vmstat.nr_inactive_file
     23834           +20.4%      28691        proc-vmstat.nr_slab_reclaimable
     41519            +9.2%      45351        proc-vmstat.nr_slab_unreclaimable
    346482            -2.0%     339409        proc-vmstat.nr_zone_inactive_file
    346479            -2.0%     339510        proc-vmstat.nr_zone_write_pending
    509175           +14.6%     583653        proc-vmstat.pgfault
    871.00           -19.2%     703.50 ±  3%  turbostat.Avg_MHz
     71.90           -15.1       56.80        turbostat.Busy%
   1869906 ±  7%     +86.1%    3479825 ± 19%  turbostat.C6
     19.42 ± 12%     +14.4       33.80 ± 11%  turbostat.C6%
     22.63 ±  2%     +44.9%      32.79 ±  7%  turbostat.CPU%c1
      4.42 ± 34%    +114.8%       9.50 ± 30%  turbostat.CPU%c6
     47.42            -4.3%      45.40 ±  2%  turbostat.CorWatt
  10614698           +20.8%   12819724        turbostat.IRQ
     65.00 ±  3%      +3.8%      67.50        turbostat.PkgTmp
     73.99            -2.8%      71.95        turbostat.PkgWatt
     34.29            -4.1%      32.90        turbostat.RAMWatt
     10450           +21.5%      12696        turbostat.SMI
      2930 ±  5%    +309.5%      12001        slabinfo.dmaengine-unmap-16.active_objs
     73.00 ±  7%    +626.0%     530.00 ±  4%  slabinfo.dmaengine-unmap-16.active_slabs
      3088 ±  7%    +621.4%      22279 ±  4%  slabinfo.dmaengine-unmap-16.num_objs
     73.00 ±  7%    +626.0%     530.00 ±  4%  slabinfo.dmaengine-unmap-16.num_slabs
      2663 ±  3%     +28.7%       3427 ±  6%  slabinfo.kmalloc-128.active_objs
      2663 ±  3%     +28.7%       3427 ±  6%  slabinfo.kmalloc-128.num_objs
     19529          +129.5%      44820 ±  4%  slabinfo.kmalloc-16.active_objs
     75.75          +141.9%     183.25 ±  4%  slabinfo.kmalloc-16.active_slabs
     19529          +140.8%      47025 ±  4%  slabinfo.kmalloc-16.num_objs
     75.75          +141.9%     183.25 ±  4%  slabinfo.kmalloc-16.num_slabs
      2850 ±  3%     +19.0%       3391 ±  3%  slabinfo.kmalloc-2k.active_objs
      2905 ±  3%     +17.8%       3423 ±  3%  slabinfo.kmalloc-2k.num_objs
    769.75           +33.0%       1023 ±  3%  slabinfo.kmalloc-4k.active_objs
    782.75           +34.1%       1049 ±  3%  slabinfo.kmalloc-4k.num_objs
      6379          +151.9%      16067        slabinfo.kmalloc-512.active_objs
    209.25          +187.8%     602.25        slabinfo.kmalloc-512.active_slabs
      6712          +187.4%      19289        slabinfo.kmalloc-512.num_objs
    209.25          +187.8%     602.25        slabinfo.kmalloc-512.num_slabs
    453.75           +41.7%     643.00        slabinfo.kmalloc-8k.active_objs
    469.50           +49.6%     702.25        slabinfo.kmalloc-8k.num_objs
    923.25 ± 14%     +22.9%       1135 ±  7%  slabinfo.task_group.active_objs
    923.25 ± 14%     +22.9%       1135 ±  7%  slabinfo.task_group.num_objs
      1201           +79.2%       2152 ±  2%  slabinfo.xfs_buf_item.active_objs
      1201           +79.2%       2152 ±  2%  slabinfo.xfs_buf_item.num_objs
      1264           +51.1%       1909 ± 13%  slabinfo.xfs_efd_item.active_objs
      1264           +51.1%       1909 ± 13%  slabinfo.xfs_efd_item.num_objs
      2437 ±  2%    +375.6%      11590        slabinfo.xfs_inode.active_objs
     84.25 ±  3%    +688.7%     664.50 ±  4%  slabinfo.xfs_inode.active_slabs
      2710 ±  3%    +685.2%      21283 ±  4%  slabinfo.xfs_inode.num_objs
     84.25 ±  3%    +688.7%     664.50 ±  4%  slabinfo.xfs_inode.num_slabs
     40593           -23.2%      31163 ±  2%  sched_debug.cfs_rq:/.exec_clock.avg
     42413           -22.9%      32721 ±  2%  sched_debug.cfs_rq:/.exec_clock.max
     40001           -23.7%      30528 ±  2%  sched_debug.cfs_rq:/.exec_clock.min
     21776 ± 22%     +54.6%      33663 ± 23%  sched_debug.cfs_rq:/.load.avg
    131390 ±112%    +327.1%     561113 ± 52%  sched_debug.cfs_rq:/.load.max
     26684 ± 79%    +269.6%      98631 ± 45%  sched_debug.cfs_rq:/.load.stddev
    508.00 ± 37%    +133.3%       1185 ± 39%  sched_debug.cfs_rq:/.load_avg.max
      9.00 ± 17%     -44.4%       5.00 ± 33%  sched_debug.cfs_rq:/.load_avg.min
    120.95 ± 40%     +94.6%     235.39 ± 32%  sched_debug.cfs_rq:/.load_avg.stddev
   1315047           -37.2%     825488 ±  4%  sched_debug.cfs_rq:/.min_vruntime.avg
   1467409 ±  3%     -35.8%     941380 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
   1271354           -38.8%     777738 ±  4%  sched_debug.cfs_rq:/.min_vruntime.min
      0.62 ±  5%     -30.4%       0.43 ± 10%  sched_debug.cfs_rq:/.nr_running.avg
     35.08 ±  9%    +449.6%     192.83 ± 63%  sched_debug.cfs_rq:/.runnable_load_avg.max
     10.30          +221.3%      33.10 ± 54%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     20216 ± 22%     +48.6%      30046 ± 29%  sched_debug.cfs_rq:/.runnable_weight.avg
    120127 ±125%    +287.7%     465751 ± 60%  sched_debug.cfs_rq:/.runnable_weight.max
     25146 ± 87%    +234.5%      84116 ± 53%  sched_debug.cfs_rq:/.runnable_weight.stddev
     14472 ±135%    -101.5%    -211.35        sched_debug.cfs_rq:/.spread0.avg
    166817 ± 25%     -30.7%     115662 ± 20%  sched_debug.cfs_rq:/.spread0.max
    -28876           +66.0%     -47942        sched_debug.cfs_rq:/.spread0.min
    757.34 ±  2%     -27.9%     545.98 ±  4%  sched_debug.cfs_rq:/.util_avg.avg
      1299 ± 11%     -14.5%       1111        sched_debug.cfs_rq:/.util_avg.max
    397.33 ± 17%     -38.5%     244.17 ± 17%  sched_debug.cfs_rq:/.util_avg.min
    103.73 ± 13%     -38.5%      63.79 ± 13%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    204770 ±  4%     -10.9%     182535 ±  3%  sched_debug.cpu.avg_idle.stddev
      1795 ± 10%     -27.6%       1300 ± 17%  sched_debug.cpu.curr->pid.avg
      1336           +18.0%       1576 ±  5%  sched_debug.cpu.curr->pid.stddev
      0.64 ±  5%     -28.8%       0.45 ± 10%  sched_debug.cpu.nr_running.avg
     27275           -10.7%      24364        sched_debug.cpu.nr_switches.avg
     34560 ±  3%     +68.3%      58168 ±  4%  sched_debug.cpu.nr_switches.max
     24872           -31.7%      16990        sched_debug.cpu.nr_switches.min
      2128 ±  7%    +328.5%       9122 ±  2%  sched_debug.cpu.nr_switches.stddev
     25106           -11.7%      22178        sched_debug.cpu.sched_count.avg
     27859 ±  2%     +89.0%      52655 ±  3%  sched_debug.cpu.sched_count.max
     23769           -32.1%      16139        sched_debug.cpu.sched_count.min
    874.04 ± 15%    +856.0%       8355        sched_debug.cpu.sched_count.stddev
     11954           -10.8%      10660        sched_debug.cpu.sched_goidle.avg
     13338 ±  2%     +93.1%      25761 ±  3%  sched_debug.cpu.sched_goidle.max
     11339           -32.5%       7653        sched_debug.cpu.sched_goidle.min
    420.12 ± 13%    +887.8%       4149        sched_debug.cpu.sched_goidle.stddev
     12641           -12.2%      11095        sched_debug.cpu.ttwu_count.avg
     14696           +73.7%      25527        sched_debug.cpu.ttwu_count.max
     12080           -33.3%       8056        sched_debug.cpu.ttwu_count.min
    598.46 ±  9%    +580.7%       4073        sched_debug.cpu.ttwu_count.stddev
    470.25 ±  2%     +23.9%     582.47 ±  3%  sched_debug.cpu.ttwu_local.avg
    883.75 ± 10%     +23.9%       1094 ±  9%  sched_debug.cpu.ttwu_local.max
     91.75 ± 12%     +68.5%     154.59 ±  7%  sched_debug.cpu.ttwu_local.stddev
 3.867e+09           -14.4%  3.312e+09        perf-stat.i.branch-instructions
      2.08 ±  2%      -0.1        1.95 ±  3%  perf-stat.i.branch-miss-rate%
  55566907           -12.0%   48886699 ±  2%  perf-stat.i.branch-misses
     26.51            -1.7       24.84 ±  2%  perf-stat.i.cache-miss-rate%
  37534712           -23.4%   28761515        perf-stat.i.cache-misses
 1.407e+08           -16.4%  1.176e+08 ±  2%  perf-stat.i.cache-references
     17417            -9.9%      15689        perf-stat.i.context-switches
      2.05            -6.7%       1.91 ±  3%  perf-stat.i.cpi
 3.442e+10           -19.1%  2.784e+10 ±  4%  perf-stat.i.cpu-cycles
      2461           -48.3%       1271        perf-stat.i.cpu-migrations
    898.31            +4.4%     937.81 ±  4%  perf-stat.i.cycles-between-cache-misses
 1.176e+08 ±  9%     -22.4%   91285096 ±  2%  perf-stat.i.dTLB-load-misses
 6.032e+09           -14.2%  5.174e+09        perf-stat.i.dTLB-loads
  17987322 ±  3%     -17.7%   14796811 ±  3%  perf-stat.i.dTLB-store-misses
 4.184e+09           -15.1%  3.553e+09        perf-stat.i.dTLB-stores
   6915996 ±  2%      -5.9%    6510336        perf-stat.i.iTLB-load-misses
   2586772 ±  8%     -18.9%    2098325 ± 13%  perf-stat.i.iTLB-loads
 1.945e+10           -14.3%  1.667e+10        perf-stat.i.instructions
      2785 ±  3%     -10.5%       2492        perf-stat.i.instructions-per-iTLB-miss
      0.55            +7.4%       0.59 ±  2%  perf-stat.i.ipc
      3723            -5.1%       3534        perf-stat.i.minor-faults
   2160613           -24.3%    1634850 ±  2%  perf-stat.i.node-load-misses
   4136129           -22.3%    3213515        perf-stat.i.node-loads
     23.00            -1.2       21.79 ±  3%  perf-stat.i.node-store-miss-rate%
   8687009           -31.6%    5940686 ±  2%  perf-stat.i.node-store-misses
  29184014           -26.3%   21502342        perf-stat.i.node-stores
      3723            -5.1%       3534        perf-stat.i.page-faults
      7.23            -2.5%       7.05        perf-stat.overall.MPKI
     26.68            -2.2       24.47        perf-stat.overall.cache-miss-rate%
      1.77            -5.7%       1.67 ±  2%  perf-stat.overall.cpi
    917.20            +5.6%     968.21 ±  4%  perf-stat.overall.cycles-between-cache-misses
      2814 ±  2%      -9.0%       2560        perf-stat.overall.instructions-per-iTLB-miss
      0.57            +6.1%       0.60 ±  2%  perf-stat.overall.ipc
     34.31            -0.6       33.71        perf-stat.overall.node-load-miss-rate%
     22.94 ±  2%      -1.3       21.65        perf-stat.overall.node-store-miss-rate%
 3.835e+09           -14.2%  3.291e+09        perf-stat.ps.branch-instructions
  55111206           -11.9%   48573654 ±  2%  perf-stat.ps.branch-misses
  37224107           -23.2%   28577554        perf-stat.ps.cache-misses
 1.395e+08           -16.3%  1.168e+08 ±  2%  perf-stat.ps.cache-references
     17274            -9.8%      15588        perf-stat.ps.context-switches
 3.414e+10           -19.0%  2.766e+10 ±  4%  perf-stat.ps.cpu-cycles
      2441           -48.2%       1263        perf-stat.ps.cpu-migrations
 1.166e+08 ±  9%     -22.2%   90699316 ±  2%  perf-stat.ps.dTLB-load-misses
 5.983e+09           -14.1%  5.141e+09        perf-stat.ps.dTLB-loads
  17837008 ±  3%     -17.6%   14701867 ±  3%  perf-stat.ps.dTLB-store-misses
 4.149e+09           -14.9%   3.53e+09        perf-stat.ps.dTLB-stores
   6858348 ±  2%      -5.7%    6468604        perf-stat.ps.iTLB-load-misses
   2565117 ±  8%     -18.7%    2084894 ± 13%  perf-stat.ps.iTLB-loads
 1.929e+10           -14.1%  1.656e+10        perf-stat.ps.instructions
      3714            -5.4%       3513        perf-stat.ps.minor-faults
   2142679           -24.2%    1624370 ±  2%  perf-stat.ps.node-load-misses
   4101979           -22.2%    3192920        perf-stat.ps.node-loads
   8614251           -31.5%    5902590 ±  2%  perf-stat.ps.node-store-misses
  28939527           -26.2%   21364553        perf-stat.ps.node-stores
      3714            -5.4%       3513        perf-stat.ps.page-faults
 2.506e+12            +4.5%  2.619e+12        perf-stat.total.instructions
    145.25 ± 16%     +54.9%     225.00 ± 18%  interrupts.39:IR-PCI-MSI.524289-edge.eth0-TxRx-0
    103.00 ± 49%   +1794.7%       1951 ±161%  interrupts.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5
      1380 ± 11%     +21.4%       1676 ±  5%  interrupts.CPU0.CAL:Function_call_interrupts
    260369           +21.4%     316051        interrupts.CPU0.LOC:Local_timer_interrupts
      2090 ± 11%     -32.7%       1407 ±  6%  interrupts.CPU0.RES:Rescheduling_interrupts
    259956           +21.7%     316451        interrupts.CPU1.LOC:Local_timer_interrupts
    259976           +21.4%     315572        interrupts.CPU10.LOC:Local_timer_interrupts
      1869 ± 22%     -56.8%     808.50 ± 12%  interrupts.CPU10.RES:Rescheduling_interrupts
      1382 ±  9%     +15.6%       1598 ±  7%  interrupts.CPU11.CAL:Function_call_interrupts
    260390           +21.4%     316071        interrupts.CPU11.LOC:Local_timer_interrupts
      1514 ±  7%     -45.4%     827.00 ± 31%  interrupts.CPU11.RES:Rescheduling_interrupts
      1218 ±  6%     +36.8%       1665 ± 12%  interrupts.CPU12.CAL:Function_call_interrupts
    260198           +21.7%     316720        interrupts.CPU12.LOC:Local_timer_interrupts
      1484 ±  8%     -37.2%     932.50 ± 16%  interrupts.CPU12.RES:Rescheduling_interrupts
      1309 ± 10%     +25.9%       1648 ± 11%  interrupts.CPU13.CAL:Function_call_interrupts
    260491           +21.6%     316752        interrupts.CPU13.LOC:Local_timer_interrupts
      1435 ±  7%     -35.7%     923.00 ± 23%  interrupts.CPU13.RES:Rescheduling_interrupts
    260090           +21.6%     316365        interrupts.CPU14.LOC:Local_timer_interrupts
      2335 ± 33%     -60.6%     920.75 ± 57%  interrupts.CPU14.NMI:Non-maskable_interrupts
      2335 ± 33%     -60.6%     920.75 ± 57%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      1534 ±  7%     -43.3%     870.50 ± 27%  interrupts.CPU14.RES:Rescheduling_interrupts
    260275           +21.2%     315386        interrupts.CPU15.LOC:Local_timer_interrupts
      1533 ±  7%     -40.2%     916.50 ± 28%  interrupts.CPU15.RES:Rescheduling_interrupts
    259511           +21.3%     314731        interrupts.CPU16.LOC:Local_timer_interrupts
      1530 ±  6%     -47.6%     802.50 ± 13%  interrupts.CPU16.RES:Rescheduling_interrupts
    260066           +21.7%     316545        interrupts.CPU17.LOC:Local_timer_interrupts
    260096           +21.0%     314637        interrupts.CPU18.LOC:Local_timer_interrupts
      1502 ±  8%     -48.9%     767.75 ± 18%  interrupts.CPU18.RES:Rescheduling_interrupts
    260276           +21.4%     315940        interrupts.CPU19.LOC:Local_timer_interrupts
      1540 ±  7%     -33.4%       1025 ± 24%  interrupts.CPU19.RES:Rescheduling_interrupts
    103.00 ± 49%   +1794.7%       1951 ±161%  interrupts.CPU2.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5
    260077           +21.7%     316607        interrupts.CPU2.LOC:Local_timer_interrupts
      1696 ± 22%     -49.0%     865.00 ± 17%  interrupts.CPU2.RES:Rescheduling_interrupts
    260201           +21.8%     316819        interrupts.CPU20.LOC:Local_timer_interrupts
    446.50 ±141%    +283.6%       1712 ± 32%  interrupts.CPU20.NMI:Non-maskable_interrupts
    446.50 ±141%    +283.6%       1712 ± 32%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
    260321           +21.6%     316575        interrupts.CPU21.LOC:Local_timer_interrupts
    260161           +21.1%     315079        interrupts.CPU22.LOC:Local_timer_interrupts
      1533 ±  8%     -21.8%       1198 ± 23%  interrupts.CPU22.RES:Rescheduling_interrupts
      1356 ± 10%     +19.9%       1626 ±  4%  interrupts.CPU23.CAL:Function_call_interrupts
    259884           +21.6%     316144        interrupts.CPU23.LOC:Local_timer_interrupts
      1332 ±  5%     -43.3%     755.00        interrupts.CPU23.RES:Rescheduling_interrupts
    260225           +21.2%     315429        interrupts.CPU24.LOC:Local_timer_interrupts
      1388 ±  4%     -47.4%     730.25 ± 19%  interrupts.CPU24.RES:Rescheduling_interrupts
    259779           +21.8%     316330        interrupts.CPU25.LOC:Local_timer_interrupts
      1468 ±  8%     -29.7%       1033 ± 31%  interrupts.CPU25.RES:Rescheduling_interrupts
      1376 ± 11%     +22.5%       1687 ± 10%  interrupts.CPU26.CAL:Function_call_interrupts
    260128           +21.2%     315219        interrupts.CPU26.LOC:Local_timer_interrupts
    259346           +22.3%     317098        interrupts.CPU27.LOC:Local_timer_interrupts
      1465 ±  7%     -42.2%     846.25 ± 31%  interrupts.CPU27.RES:Rescheduling_interrupts
    260225           +21.0%     314795        interrupts.CPU28.LOC:Local_timer_interrupts
      1459 ±  7%     -42.3%     842.00 ± 17%  interrupts.CPU28.RES:Rescheduling_interrupts
    260085           +21.5%     316006        interrupts.CPU29.LOC:Local_timer_interrupts
      1379 ±  4%     -47.2%     727.75 ± 16%  interrupts.CPU29.RES:Rescheduling_interrupts
    260259           +21.7%     316731        interrupts.CPU3.LOC:Local_timer_interrupts
      1567 ±  9%     -39.8%     943.25 ± 30%  interrupts.CPU3.RES:Rescheduling_interrupts
    145.25 ± 16%     +54.9%     225.00 ± 18%  interrupts.CPU30.39:IR-PCI-MSI.524289-edge.eth0-TxRx-0
    260074           +21.1%     314955        interrupts.CPU30.LOC:Local_timer_interrupts
      1390 ±  4%     -50.9%     682.50 ± 14%  interrupts.CPU30.RES:Rescheduling_interrupts
      1358 ± 11%     +18.5%       1609 ± 14%  interrupts.CPU31.CAL:Function_call_interrupts
    260132           +21.7%     316628        interrupts.CPU31.LOC:Local_timer_interrupts
      1392 ± 10%     -31.4%     955.75 ± 17%  interrupts.CPU31.RES:Rescheduling_interrupts
      1360 ± 11%     +15.9%       1576 ± 11%  interrupts.CPU32.CAL:Function_call_interrupts
    260317           +20.9%     314826        interrupts.CPU32.LOC:Local_timer_interrupts
      1464 ± 11%     -45.5%     798.25 ± 18%  interrupts.CPU32.RES:Rescheduling_interrupts
    260012           +21.4%     315686        interrupts.CPU33.LOC:Local_timer_interrupts
      1486 ± 10%     -48.4%     767.00 ± 14%  interrupts.CPU33.RES:Rescheduling_interrupts
      1265 ± 18%     +25.0%       1582 ± 12%  interrupts.CPU34.CAL:Function_call_interrupts
    260359           +21.0%     315007        interrupts.CPU34.LOC:Local_timer_interrupts
      1603 ± 27%     -52.0%     770.00 ± 27%  interrupts.CPU34.RES:Rescheduling_interrupts
      1390 ±  8%     +16.0%       1612 ± 10%  interrupts.CPU35.CAL:Function_call_interrupts
    260282           +21.7%     316652        interrupts.CPU35.LOC:Local_timer_interrupts
      1434 ± 11%     -42.3%     827.50 ± 14%  interrupts.CPU35.RES:Rescheduling_interrupts
    260380           +20.8%     314555        interrupts.CPU36.LOC:Local_timer_interrupts
      1343 ± 10%     -38.3%     829.25 ±  9%  interrupts.CPU36.RES:Rescheduling_interrupts
    260230           +21.1%     315199        interrupts.CPU37.LOC:Local_timer_interrupts
      1346 ± 12%     +21.2%       1631 ± 13%  interrupts.CPU38.CAL:Function_call_interrupts
    259976           +21.8%     316532        interrupts.CPU38.LOC:Local_timer_interrupts
      1481 ±  6%     -51.1%     724.25 ±  9%  interrupts.CPU38.RES:Rescheduling_interrupts
    260061           +20.9%     314458        interrupts.CPU39.LOC:Local_timer_interrupts
      1436 ±  8%     -44.9%     791.00 ± 34%  interrupts.CPU39.RES:Rescheduling_interrupts
      1359 ± 13%     +22.9%       1671 ± 12%  interrupts.CPU4.CAL:Function_call_interrupts
    260201           +21.7%     316705        interrupts.CPU4.LOC:Local_timer_interrupts
      1668 ± 20%     -43.6%     941.00 ± 10%  interrupts.CPU4.RES:Rescheduling_interrupts
      1385 ±  9%     +12.1%       1552 ±  8%  interrupts.CPU5.CAL:Function_call_interrupts
    260181           +21.7%     316657        interrupts.CPU5.LOC:Local_timer_interrupts
      1164 ± 55%     -74.2%     300.25 ±173%  interrupts.CPU5.NMI:Non-maskable_interrupts
      1164 ± 55%     -74.2%     300.25 ±173%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      1464 ±  9%     -42.8%     838.00 ± 15%  interrupts.CPU5.RES:Rescheduling_interrupts
    260255           +21.7%     316804        interrupts.CPU6.LOC:Local_timer_interrupts
      1497 ±  6%     -50.5%     741.00 ± 19%  interrupts.CPU6.RES:Rescheduling_interrupts
      1372 ±  9%     +19.6%       1642 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
    259919           +21.5%     315765        interrupts.CPU7.LOC:Local_timer_interrupts
      1481 ± 10%     -26.4%       1089 ± 29%  interrupts.CPU7.RES:Rescheduling_interrupts
      1339 ± 13%     +17.6%       1575 ± 11%  interrupts.CPU8.CAL:Function_call_interrupts
    260401           +21.6%     316700        interrupts.CPU8.LOC:Local_timer_interrupts
      1861 ± 33%     -53.3%     869.50 ± 19%  interrupts.CPU8.RES:Rescheduling_interrupts
      1426 ±  6%     +18.8%       1695 ±  9%  interrupts.CPU9.CAL:Function_call_interrupts
    259391           +22.2%     316922        interrupts.CPU9.LOC:Local_timer_interrupts
      1551           -18.5%       1264 ±  3%  interrupts.CPU9.NMI:Non-maskable_interrupts
      1551           -18.5%       1264 ±  3%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      1646 ± 14%     -47.5%     864.75 ±  8%  interrupts.CPU9.RES:Rescheduling_interrupts
  10404571           +21.5%   12638118        interrupts.LOC:Local_timer_interrupts
     61340 ±  2%     -39.7%      37011 ±  2%  interrupts.RES:Rescheduling_interrupts
    265.00 ±  6%     +76.1%     466.75 ± 12%  interrupts.TLB:TLB_shootdowns
     21780 ±  3%     +34.1%      29206 ±  4%  softirqs.CPU0.RCU
     19352 ±  2%     +14.7%      22198        softirqs.CPU0.SCHED
     50306 ±  4%     +17.8%      59258 ±  8%  softirqs.CPU0.TIMER
     23163 ±  8%     +21.3%      28089 ±  6%  softirqs.CPU1.RCU
     16179           +20.8%      19547        softirqs.CPU1.SCHED
     48867 ±  5%     +16.5%      56934 ±  3%  softirqs.CPU1.TIMER
     21339 ±  2%     +31.5%      28064 ±  4%  softirqs.CPU10.RCU
     15492 ±  2%     +21.8%      18872        softirqs.CPU10.SCHED
     48447 ±  2%     +16.0%      56208 ±  2%  softirqs.CPU10.TIMER
     21278 ±  5%     +31.9%      28067 ±  5%  softirqs.CPU11.RCU
     15581           +20.5%      18770        softirqs.CPU11.SCHED
     48567           +17.7%      57140 ±  4%  softirqs.CPU11.TIMER
     22261 ±  3%     +36.5%      30385 ±  9%  softirqs.CPU12.RCU
     15644           +20.3%      18824 ±  2%  softirqs.CPU12.SCHED
     49536 ±  2%     +32.3%      65518 ± 21%  softirqs.CPU12.TIMER
     21724 ±  2%     +27.9%      27794 ±  4%  softirqs.CPU13.RCU
     15508           +21.1%      18776 ±  2%  softirqs.CPU13.SCHED
     48945           +19.2%      58366 ±  4%  softirqs.CPU13.TIMER
     21297 ±  3%     +35.0%      28761 ±  3%  softirqs.CPU14.RCU
     15583           +23.0%      19162 ±  5%  softirqs.CPU14.SCHED
     48772 ±  2%     +17.2%      57154 ±  4%  softirqs.CPU14.TIMER
     21231 ±  3%     +35.4%      28746 ±  8%  softirqs.CPU15.RCU
     15571           +21.9%      18985 ±  2%  softirqs.CPU15.SCHED
     48486           +17.6%      57020 ±  3%  softirqs.CPU15.TIMER
     21571 ±  7%     +31.7%      28415        softirqs.CPU16.RCU
     15314           +23.3%      18884 ±  2%  softirqs.CPU16.SCHED
     49988 ±  4%     +15.1%      57561 ±  3%  softirqs.CPU16.TIMER
     21326 ±  4%     +31.0%      27927 ±  4%  softirqs.CPU17.RCU
     15434           +22.8%      18960 ±  3%  softirqs.CPU17.SCHED
     48588 ±  2%     +17.1%      56905 ±  2%  softirqs.CPU17.TIMER
     21212 ±  2%     +31.8%      27968 ±  4%  softirqs.CPU18.RCU
     15571           +20.9%      18827 ±  3%  softirqs.CPU18.SCHED
     49151 ±  2%     +33.3%      65535 ± 15%  softirqs.CPU18.TIMER
     21490 ±  4%     +27.9%      27490 ±  5%  softirqs.CPU19.RCU
     15550           +20.6%      18747 ±  2%  softirqs.CPU19.SCHED
     50252 ±  8%     +13.4%      57009 ±  2%  softirqs.CPU19.TIMER
     23244 ± 11%     +32.3%      30747 ±  6%  softirqs.CPU2.RCU
     15982           +30.8%      20905        softirqs.CPU2.SCHED
     50703 ±  5%     +22.8%      62276 ±  4%  softirqs.CPU2.TIMER
     21304 ±  2%     +45.9%      31089 ±  7%  softirqs.CPU20.RCU
     15409 ±  2%     +23.2%      18988 ±  3%  softirqs.CPU20.SCHED
     22317 ±  4%     +28.2%      28608 ±  5%  softirqs.CPU21.RCU
     50492 ±  3%     +18.6%      59868 ±  8%  softirqs.CPU21.TIMER
     21099 ±  4%     +32.6%      27983 ±  3%  softirqs.CPU22.RCU
     15607           +21.3%      18935 ±  2%  softirqs.CPU22.SCHED
     20953 ±  2%     +36.0%      28492 ±  6%  softirqs.CPU23.RCU
     15463           +22.4%      18924 ±  6%  softirqs.CPU23.SCHED
     49790 ±  2%     +17.2%      58355 ±  4%  softirqs.CPU23.TIMER
     21424 ±  2%     +33.5%      28607 ±  4%  softirqs.CPU24.RCU
     15603           +19.8%      18692        softirqs.CPU24.SCHED
     49433 ±  4%     +18.7%      58682 ±  4%  softirqs.CPU24.TIMER
     21527 ±  2%     +34.8%      29025 ±  4%  softirqs.CPU25.RCU
     15413           +22.6%      18891 ±  2%  softirqs.CPU25.SCHED
     48601 ±  2%     +17.9%      57281 ±  3%  softirqs.CPU25.TIMER
     21284 ±  3%     +31.1%      27898 ±  2%  softirqs.CPU26.RCU
     15979 ±  4%     +18.2%      18888        softirqs.CPU26.SCHED
     50363 ±  3%     +12.6%      56721 ±  4%  softirqs.CPU26.TIMER
     21307 ±  2%     +34.4%      28646 ±  5%  softirqs.CPU27.RCU
     15342           +23.2%      18905 ±  3%  softirqs.CPU27.SCHED
     48693 ±  3%     +19.5%      58194 ±  2%  softirqs.CPU27.TIMER
     21278           +30.1%      27694 ±  3%  softirqs.CPU28.RCU
     15623           +20.2%      18773        softirqs.CPU28.SCHED
     49220 ±  2%     +14.4%      56308 ±  3%  softirqs.CPU28.TIMER
     21511 ±  4%     +30.1%      27979 ±  5%  softirqs.CPU29.RCU
     15816 ±  2%     +18.4%      18725        softirqs.CPU29.SCHED
     48698           +17.7%      57323 ±  3%  softirqs.CPU29.TIMER
     21335 ±  2%     +36.2%      29048 ±  6%  softirqs.CPU3.RCU
     15763 ±  2%     +20.0%      18917        softirqs.CPU3.SCHED
     21413 ±  2%     +32.4%      28350 ±  3%  softirqs.CPU30.RCU
     15489 ±  2%     +21.6%      18831 ±  2%  softirqs.CPU30.SCHED
     48627 ±  2%     +16.2%      56482 ±  2%  softirqs.CPU30.TIMER
     21314 ±  4%     +32.2%      28181 ±  5%  softirqs.CPU31.RCU
     15542           +21.2%      18844 ±  3%  softirqs.CPU31.SCHED
     49740 ±  6%     +14.4%      56918 ±  3%  softirqs.CPU31.TIMER
     19067 ± 11%     +19.3%      22739 ±  2%  softirqs.CPU32.RCU
     15786 ±  3%     +19.3%      18839 ±  2%  softirqs.CPU32.SCHED
     51277 ±  4%     +12.1%      57503 ±  4%  softirqs.CPU32.TIMER
     17283 ±  3%     +28.4%      22185 ±  4%  softirqs.CPU33.RCU
     15463 ±  2%     +20.5%      18635        softirqs.CPU33.SCHED
     48637 ±  2%     +32.4%      64412 ± 18%  softirqs.CPU33.TIMER
     17794 ±  3%     +29.8%      23103 ±  4%  softirqs.CPU34.RCU
     15838           +19.2%      18886        softirqs.CPU34.SCHED
     49140 ±  2%     +16.2%      57109 ±  4%  softirqs.CPU34.TIMER
     17452 ±  2%     +28.5%      22432 ±  3%  softirqs.CPU35.RCU
     15466           +19.9%      18543 ±  2%  softirqs.CPU35.SCHED
     48547           +16.8%      56684 ±  3%  softirqs.CPU35.TIMER
     17422 ±  4%     +36.2%      23721 ±  5%  softirqs.CPU36.RCU
     15474           +22.9%      19016 ±  3%  softirqs.CPU36.SCHED
     18024 ±  5%     +25.0%      22537 ±  3%  softirqs.CPU37.RCU
     15591 ±  2%     +21.2%      18892 ±  2%  softirqs.CPU37.SCHED
     49241 ±  2%     +15.7%      56982 ±  3%  softirqs.CPU37.TIMER
     18270 ±  6%     +23.0%      22468 ±  2%  softirqs.CPU38.RCU
     15698           +18.2%      18555 ±  2%  softirqs.CPU38.SCHED
     17354 ±  2%     +29.0%      22393 ±  3%  softirqs.CPU39.RCU
     15582           +18.3%      18430 ±  4%  softirqs.CPU39.SCHED
     48135 ±  2%     +35.5%      65242 ± 18%  softirqs.CPU39.TIMER
     21472           +30.6%      28049        softirqs.CPU4.RCU
     15684           +20.5%      18896 ±  4%  softirqs.CPU4.SCHED
     49567 ±  2%     +14.5%      56732 ±  4%  softirqs.CPU4.TIMER
     21886           +33.1%      29122 ±  8%  softirqs.CPU5.RCU
     15912 ±  4%     +19.3%      18983 ±  4%  softirqs.CPU5.SCHED
     49334 ±  4%     +17.3%      57889 ±  4%  softirqs.CPU5.TIMER
     22759 ± 12%     +28.1%      29150 ±  3%  softirqs.CPU6.RCU
     15561           +23.0%      19144        softirqs.CPU6.SCHED
     50896           +15.8%      58936 ±  3%  softirqs.CPU6.TIMER
     21754 ±  5%     +26.8%      27588 ±  4%  softirqs.CPU7.RCU
     15439           +22.1%      18851 ±  3%  softirqs.CPU7.SCHED
     51023 ±  3%     +12.0%      57169 ±  4%  softirqs.CPU7.TIMER
     21768 ±  3%     +28.9%      28061 ±  3%  softirqs.CPU8.RCU
     15680           +24.5%      19522 ±  5%  softirqs.CPU8.SCHED
     48977 ±  2%     +17.5%      57545 ±  3%  softirqs.CPU8.TIMER
     20951           +32.1%      27683 ±  4%  softirqs.CPU9.RCU
     15426           +20.8%      18638 ±  3%  softirqs.CPU9.SCHED
     48306 ±  3%     +18.3%      57152 ±  3%  softirqs.CPU9.TIMER
    834261 ±  2%     +31.2%    1094511 ±  3%  softirqs.RCU
    629022           +20.9%     760517 ±  2%  softirqs.SCHED
   2001302           +16.7%    2335824 ±  3%  softirqs.TIMER
     63.72           -11.9       51.82        perf-profile.calltrace.cycles-pp.write
     54.54           -10.4       44.19 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     54.05           -10.3       43.78 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     45.97            -9.1       36.91 ±  2%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     44.57            -8.8       35.77 ±  2%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     38.87            -7.6       31.22 ±  2%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     36.73            -7.1       29.68 ±  3%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     29.28            -5.2       24.09 ±  3%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     29.00            -5.1       23.86 ±  3%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     21.42            -4.1       17.29 ±  2%  perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      9.82            -2.0        7.84 ±  2%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
     10.19            -1.6        8.57 ±  4%  perf-profile.calltrace.cycles-pp.lseek64
      3.70            -1.5        2.15 ±  4%  perf-profile.calltrace.cycles-pp.close
      3.64            -1.5        2.13 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close
      3.64            -1.5        2.13 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      3.62            -1.5        2.11 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      3.60            -1.5        2.10 ±  3%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.60            -1.5        2.11 ±  3%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      3.48            -1.5        1.99 ±  4%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      3.46            -1.5        1.98 ±  3%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
      7.25 ±  2%      -1.5        5.77        perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      6.93            -1.4        5.53        perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      4.82 ±  2%      -1.2        3.59 ±  4%  perf-profile.calltrace.cycles-pp.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      4.07 ±  2%      -0.9        3.16        perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      5.73            -0.8        4.89 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.lseek64
      3.46 ±  2%      -0.8        2.65        perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      5.57 ±  2%      -0.8        4.76 ±  7%  perf-profile.calltrace.cycles-pp.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      4.54 ±  3%      -0.8        3.74 ±  3%  perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      3.36 ±  3%      -0.8        2.57 ±  2%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply
      4.53            -0.8        3.74 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      5.46            -0.8        4.69 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
      2.63            -0.7        1.91 ±  3%  perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.__dentry_kill.dput.__fput
      2.65            -0.7        1.93 ±  3%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      2.48 ±  4%      -0.7        1.79 ±  8%  perf-profile.calltrace.cycles-pp.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      3.72 ±  2%      -0.7        3.06 ±  2%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
      2.42 ±  4%      -0.6        1.83 ±  8%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.57 ±  3%      -0.6        3.98 ±  9%  perf-profile.calltrace.cycles-pp.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.68 ±  2%      -0.6        2.10        perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.69 ±  7%      -0.5        1.18 ± 10%  perf-profile.calltrace.cycles-pp.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write
      1.52 ± 17%      -0.5        1.03 ± 17%  perf-profile.calltrace.cycles-pp.xfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
      1.93            -0.4        1.49 ±  4%  perf-profile.calltrace.cycles-pp.iomap_read_page_sync.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.72 ±  6%      -0.4        1.30 ±  6%  perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.92 ±  2%      -0.4        1.51 ±  4%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.80 ±  5%      -0.4        1.40 ±  7%  perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
      1.19 ±  2%      -0.4        0.82 ± 12%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
      0.70 ± 24%      -0.4        0.33 ±104%  perf-profile.calltrace.cycles-pp.xfs_bmbt_to_iomap.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      1.13 ±  3%      -0.4        0.77 ± 13%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
      1.45            -0.4        1.10 ±  4%  perf-profile.calltrace.cycles-pp.memset_erms.iomap_read_page_sync.iomap_write_begin.iomap_write_actor.iomap_apply
      1.84            -0.3        1.49 ±  5%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.lseek64
      2.22 ±  2%      -0.3        1.88 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.lseek64
      1.12 ±  3%      -0.3        0.79 ±  3%  perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill
      1.14 ±  3%      -0.3        0.81 ±  3%  perf-profile.calltrace.cycles-pp.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill.dput
      0.71 ±  2%      -0.3        0.40 ± 57%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin
      1.11 ±  4%      -0.3        0.85 ±  7%  perf-profile.calltrace.cycles-pp.xas_load.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.34            -0.3        1.08 ±  3%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.06 ±  2%      -0.2        0.84 ±  3%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      0.66 ±  3%      -0.2        0.43 ± 57%  perf-profile.calltrace.cycles-pp.disk_rw
      1.08 ±  4%      -0.2        0.88 ±  5%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      0.76            -0.2        0.57 ±  5%  perf-profile.calltrace.cycles-pp.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.87 ±  3%      -0.2        1.68 ±  4%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.14 ±  6%      -0.2        0.95 ±  3%  perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      0.94 ±  6%      -0.2        0.76 ±  4%  perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      0.78 ±  9%      -0.2        0.61 ±  5%  perf-profile.calltrace.cycles-pp.xfs_iext_lookup_extent.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      0.72 ±  7%      -0.2        0.56 ±  3%  perf-profile.calltrace.cycles-pp.xfs_break_layouts.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      0.69 ±  6%      -0.2        0.53 ±  4%  perf-profile.calltrace.cycles-pp.__set_page_dirty.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply
      0.94 ±  5%      -0.2        0.79 ±  7%  perf-profile.calltrace.cycles-pp.down_write.xfs_ilock.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply
      0.78 ±  6%      -0.2        0.62 ±  4%  perf-profile.calltrace.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      0.86 ±  2%      +0.4        1.23 ±  3%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_open
      0.89 ±  2%      +0.4        1.25 ±  2%  perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
      1.59 ±  2%      +0.5        2.13 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
      1.58 ±  2%      +0.5        2.13 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.57            +0.5        2.12 ±  5%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.54 ±  2%      +0.6        2.09 ±  5%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.61            +0.6        2.17 ±  5%  perf-profile.calltrace.cycles-pp.creat
      1.54 ±  2%      +0.6        2.10 ±  5%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      0.00            +0.6        0.57 ± 10%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +0.6        0.62 ±  5%  perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc
      0.00            +0.7        0.69 ±  4%  perf-profile.calltrace.cycles-pp.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create
      0.00            +0.7        0.74 ±  3%  perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create
      0.00            +0.8        0.79 ±  2%  perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      0.00            +0.8        0.79 ±  2%  perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      0.00            +1.1        1.08 ±  7%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.00            +1.1        1.15 ±  7%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     16.68 ±  2%     +11.8       28.47 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     16.49 ±  4%     +12.5       29.02        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     16.51 ±  4%     +12.6       29.09        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     16.96 ±  4%     +12.9       29.88        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     16.96 ±  4%     +12.9       29.89        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     16.96 ±  4%     +12.9       29.89        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     17.71           +13.0       30.74 ±  2%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     67.26           -12.2       55.10 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     66.62           -12.1       54.57 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     64.32           -12.0       52.29        perf-profile.children.cycles-pp.write
     45.98            -9.1       36.93 ±  2%  perf-profile.children.cycles-pp.ksys_write
     44.63            -8.8       35.81 ±  2%  perf-profile.children.cycles-pp.vfs_write
     38.90            -7.7       31.25 ±  2%  perf-profile.children.cycles-pp.new_sync_write
     36.78            -7.1       29.72 ±  3%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     29.30            -5.2       24.10 ±  3%  perf-profile.children.cycles-pp.iomap_file_buffered_write
     29.06            -5.2       23.91 ±  4%  perf-profile.children.cycles-pp.iomap_apply
     21.54            -4.2       17.38 ±  2%  perf-profile.children.cycles-pp.iomap_write_actor
      9.86            -2.0        7.88 ±  2%  perf-profile.children.cycles-pp.iomap_write_begin
     10.49            -1.7        8.81 ±  4%  perf-profile.children.cycles-pp.lseek64
      3.70            -1.5        2.15 ±  4%  perf-profile.children.cycles-pp.close
      3.63            -1.5        2.12 ±  3%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      3.60            -1.5        2.10 ±  3%  perf-profile.children.cycles-pp.__fput
      3.60            -1.5        2.11 ±  3%  perf-profile.children.cycles-pp.task_work_run
      3.48            -1.5        2.00 ±  4%  perf-profile.children.cycles-pp.dput
      7.29 ±  2%      -1.5        5.80        perf-profile.children.cycles-pp.grab_cache_page_write_begin
      3.46            -1.5        1.98 ±  3%  perf-profile.children.cycles-pp.__dentry_kill
      6.98            -1.4        5.57        perf-profile.children.cycles-pp.pagecache_get_page
      4.87 ±  2%      -1.2        3.63 ±  5%  perf-profile.children.cycles-pp.xfs_file_aio_write_checks
      6.78            -1.1        5.67 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      6.23            -1.1        5.12 ±  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      4.10 ±  2%      -0.9        3.17        perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      3.49 ±  2%      -0.8        2.67        perf-profile.children.cycles-pp.copyin
      5.63 ±  2%      -0.8        4.82 ±  7%  perf-profile.children.cycles-pp.xfs_file_iomap_begin
      4.58 ±  3%      -0.8        3.77 ±  3%  perf-profile.children.cycles-pp.iomap_write_end
      3.38 ±  2%      -0.8        2.59 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      2.65            -0.7        1.93 ±  3%  perf-profile.children.cycles-pp.evict
      2.63            -0.7        1.92 ±  4%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      2.50 ±  4%      -0.7        1.81 ±  8%  perf-profile.children.cycles-pp.file_update_time
      4.65 ±  3%      -0.6        4.04 ± 10%  perf-profile.children.cycles-pp.xfs_file_iomap_begin_delay
      2.44 ±  4%      -0.6        1.84 ±  8%  perf-profile.children.cycles-pp.security_file_permission
      2.71 ±  2%      -0.6        2.12        perf-profile.children.cycles-pp.find_get_entry
      1.69 ±  7%      -0.5        1.18 ± 10%  perf-profile.children.cycles-pp.xfs_vn_update_time
      1.53 ± 17%      -0.5        1.04 ± 18%  perf-profile.children.cycles-pp.xfs_file_write_iter
      1.94            -0.4        1.50 ±  4%  perf-profile.children.cycles-pp.iomap_read_page_sync
      1.76 ±  6%      -0.4        1.33 ±  5%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      1.20 ±  2%      -0.4        0.77 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
      0.89            -0.4        0.46 ±  8%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.83 ±  5%      -0.4        1.42 ±  7%  perf-profile.children.cycles-pp.selinux_file_permission
      1.92 ±  2%      -0.4        1.52 ±  4%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      2.17 ±  5%      -0.4        1.81 ±  3%  perf-profile.children.cycles-pp.xfs_ilock
      1.84            -0.4        1.48 ± 14%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      1.19 ±  3%      -0.3        0.85 ±  3%  perf-profile.children.cycles-pp.release_pages
      1.47            -0.3        1.13 ±  4%  perf-profile.children.cycles-pp.memset_erms
      1.81 ±  5%      -0.3        1.47 ±  6%  perf-profile.children.cycles-pp.down_write
      1.14 ±  3%      -0.3        0.81 ±  3%  perf-profile.children.cycles-pp.__pagevec_release
      1.34 ±  4%      -0.3        1.05 ±  5%  perf-profile.children.cycles-pp.xas_load
      1.36 ±  2%      -0.3        1.11 ±  4%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      2.25 ±  4%      -0.3        2.00 ±  4%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      1.08 ±  2%      -0.2        0.86 ±  4%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.77            -0.2        0.57 ±  4%  perf-profile.children.cycles-pp.__lru_cache_add
      1.10 ±  4%      -0.2        0.90 ±  5%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      0.73            -0.2        0.53 ±  7%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      1.35 ±  3%      -0.2        1.18 ±  6%  perf-profile.children.cycles-pp.___might_sleep
      0.80 ±  8%      -0.2        0.62 ±  5%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      0.81 ±  4%      -0.2        0.63 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.75 ±  7%      -0.2        0.58 ±  3%  perf-profile.children.cycles-pp.xfs_break_layouts
      0.62 ±  5%      -0.2        0.45 ±  4%  perf-profile.children.cycles-pp.free_unref_page_list
      0.70 ±  6%      -0.2        0.53 ±  4%  perf-profile.children.cycles-pp.__set_page_dirty
      0.56 ±  3%      -0.2        0.39 ±  7%  perf-profile.children.cycles-pp.truncate_cleanup_page
      0.59 ±  7%      -0.2        0.43 ±  5%  perf-profile.children.cycles-pp.generic_write_checks
      1.16 ±  5%      -0.2        1.00 ±  5%  perf-profile.children.cycles-pp.xfs_iunlock
      0.46 ±  4%      -0.1        0.32 ±  9%  perf-profile.children.cycles-pp.__cancel_dirty_page
      0.53 ± 10%      -0.1        0.39 ± 11%  perf-profile.children.cycles-pp.xas_start
      0.60 ±  5%      -0.1        0.46 ±  6%  perf-profile.children.cycles-pp.delete_from_page_cache_batch
      0.50 ±  5%      -0.1        0.36 ±  8%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.72 ± 10%      -0.1        0.59 ±  6%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.44 ±  5%      -0.1        0.30 ±  7%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.78 ±  5%      -0.1        0.65 ±  3%  perf-profile.children.cycles-pp.up_write
      0.74 ±  3%      -0.1        0.61 ±  3%  perf-profile.children.cycles-pp._cond_resched
      0.39 ±  8%      -0.1        0.27 ± 15%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.47 ±  2%      -0.1        0.35 ±  6%  perf-profile.children.cycles-pp.current_time
      0.71 ±  3%      -0.1        0.58 ±  7%  perf-profile.children.cycles-pp.disk_rw
      0.57 ±  8%      -0.1        0.46 ±  6%  perf-profile.children.cycles-pp.xas_store
      0.42 ± 13%      -0.1        0.30 ±  9%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.61 ± 15%      -0.1        0.49 ±  6%  perf-profile.children.cycles-pp.__inode_security_revalidate
      0.44 ±  4%      -0.1        0.33 ±  3%  perf-profile.children.cycles-pp.account_page_dirtied
      0.43 ±  5%      -0.1        0.33 ± 13%  perf-profile.children.cycles-pp.iov_iter_advance
      0.42 ±  3%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.34 ±  6%      -0.1        0.24 ±  9%  perf-profile.children.cycles-pp.account_page_cleaned
      0.43 ±  9%      -0.1        0.35 ±  5%  perf-profile.children.cycles-pp.xfs_break_leased_layouts
      0.22 ± 11%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.xlog_grant_add_space
      0.35 ±  5%      -0.1        0.27        perf-profile.children.cycles-pp.file_remove_privs
      0.38 ±  9%      -0.1        0.30 ±  5%  perf-profile.children.cycles-pp.file_modified
      0.32 ± 15%      -0.1        0.24 ±  3%  perf-profile.children.cycles-pp.generic_write_check_limits
      0.35 ±  5%      -0.1        0.28 ±  9%  perf-profile.children.cycles-pp.unlock_page
      0.44 ±  6%      -0.1        0.36 ± 10%  perf-profile.children.cycles-pp.__x64_sys_write
      0.17 ±  4%      -0.1        0.11 ± 20%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.24 ±  8%      -0.1        0.19 ± 14%  perf-profile.children.cycles-pp.__x64_sys_lseek
      0.11 ± 22%      -0.1        0.05 ± 70%  perf-profile.children.cycles-pp.xfs_find_daxdev_for_inode
      0.36 ±  4%      -0.1        0.31        perf-profile.children.cycles-pp.rcu_all_qs
      0.24 ±  6%      -0.1        0.19 ± 15%  perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.12 ±  4%      -0.1        0.07 ± 17%  perf-profile.children.cycles-pp.xfs_isilocked
      0.23 ± 10%      -0.0        0.18 ± 12%  perf-profile.children.cycles-pp.node_dirty_ok
      0.16 ± 12%      -0.0        0.11 ±  7%  perf-profile.children.cycles-pp.timespec64_trunc
      0.18 ±  9%      -0.0        0.14 ±  6%  perf-profile.children.cycles-pp.find_get_entries
      0.18 ±  9%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.pagevec_lookup_entries
      0.22 ±  5%      -0.0        0.17 ± 16%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.16 ±  9%      -0.0        0.12 ± 15%  perf-profile.children.cycles-pp.__xa_set_mark
      0.30 ±  2%      -0.0        0.26 ±  9%  perf-profile.children.cycles-pp.page_mapping
      0.14 ± 15%      -0.0        0.10 ± 17%  perf-profile.children.cycles-pp.wake_up_q
      0.19 ± 12%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.18 ±  6%      -0.0        0.14 ± 10%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.12 ± 15%      -0.0        0.08 ± 19%  perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.22 ±  3%      -0.0        0.19 ± 10%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.11 ± 10%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_mod_fdblocks
      0.08 ±  6%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.xfs_bmap_add_extent_hole_delay
      0.07 ± 17%      -0.0        0.06 ± 15%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.09 ±  9%      +0.0        0.12 ± 14%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.03 ±100%      +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.run_timer_softirq
      0.09 ± 28%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.rcu_core
      0.06 ± 13%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.xfs_iunlink
      0.00            +0.1        0.06 ± 20%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.04 ± 59%      +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.update_blocked_averages
      0.05 ± 61%      +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.rebalance_domains
      0.05 ± 58%      +0.1        0.12 ± 18%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.00            +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.irq_enter
      0.00            +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.46 ±  6%      +0.1        0.58 ±  6%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.11 ± 17%      +0.1        0.23 ± 17%  perf-profile.children.cycles-pp.ktime_get
      0.12 ± 21%      +0.1        0.25 ± 23%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.1        0.15 ± 72%  perf-profile.children.cycles-pp.xfs_verify_agino
      0.04 ± 60%      +0.2        0.26 ± 71%  perf-profile.children.cycles-pp.xfs_inobt_get_maxrecs
      0.11 ±  4%      +0.2        0.34 ± 17%  perf-profile.children.cycles-pp.menu_select
      0.15 ± 42%      +0.3        0.41 ± 21%  perf-profile.children.cycles-pp.osq_lock
      0.26 ± 33%      +0.3        0.52 ± 11%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.64 ± 33%      +0.3        0.92 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.30 ± 29%      +0.3        0.59 ±  9%  perf-profile.children.cycles-pp.irq_exit
      0.76 ± 10%      +0.3        1.08 ± 12%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      0.63 ± 12%      +0.4        0.99 ± 11%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.86 ±  2%      +0.4        1.23 ±  3%  perf-profile.children.cycles-pp.xfs_create
      0.89 ±  2%      +0.4        1.25 ±  2%  perf-profile.children.cycles-pp.xfs_generic_create
      0.34 ±  6%      +0.4        0.74 ±  3%  perf-profile.children.cycles-pp.xfs_dialloc
      0.28 ±  7%      +0.4        0.69 ±  4%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      0.38 ±  6%      +0.4        0.79 ±  2%  perf-profile.children.cycles-pp.xfs_dir_ialloc
      0.38 ±  6%      +0.4        0.79 ±  2%  perf-profile.children.cycles-pp.xfs_ialloc
      0.12 ± 19%      +0.4        0.55 ± 68%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      0.12 ±  7%      +0.4        0.55 ± 70%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.10 ± 27%      +0.4        0.55 ± 70%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      0.88 ± 27%      +0.5        1.33 ±  8%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.67            +0.5        2.20 ±  5%  perf-profile.children.cycles-pp.do_sys_open
      1.61 ±  2%      +0.6        2.16 ±  5%  perf-profile.children.cycles-pp.do_filp_open
      1.61            +0.6        2.17 ±  5%  perf-profile.children.cycles-pp.creat
      1.61 ±  2%      +0.6        2.16 ±  5%  perf-profile.children.cycles-pp.path_openat
      0.15 ± 23%      +0.6        0.76 ± 69%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      0.20 ±  6%      +0.7        0.89 ± 70%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      1.25 ± 27%      +0.8        2.07 ±  8%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.36 ± 26%      +0.9        2.23 ±  8%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.41 ±  4%      +1.1        1.54 ± 68%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      0.06 ± 64%      +1.6        1.62 ±107%  perf-profile.children.cycles-pp.worker_thread
      0.06 ± 64%      +1.6        1.62 ±107%  perf-profile.children.cycles-pp.process_one_work
      0.07 ± 59%      +1.6        1.66 ±105%  perf-profile.children.cycles-pp.kthread
      0.07 ± 59%      +1.6        1.66 ±105%  perf-profile.children.cycles-pp.ret_from_fork
     16.80 ±  2%     +11.7       28.49 ±  2%  perf-profile.children.cycles-pp.intel_idle
     17.24           +12.7       29.90 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
     17.24           +12.7       29.91 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
     16.96 ±  4%     +12.9       29.89        perf-profile.children.cycles-pp.start_secondary
     17.71           +13.0       30.74 ±  2%  perf-profile.children.cycles-pp.do_idle
     17.71           +13.0       30.74 ±  2%  perf-profile.children.cycles-pp.secondary_startup_64
     17.71           +13.0       30.74 ±  2%  perf-profile.children.cycles-pp.cpu_startup_entry
     11.21            -1.7        9.51 ±  3%  perf-profile.self.cycles-pp.do_syscall_64
      6.22            -1.1        5.11 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      6.11            -1.0        5.14 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      3.33 ±  3%      -0.8        2.53 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.47 ± 18%      -0.5        1.00 ± 18%  perf-profile.self.cycles-pp.xfs_file_write_iter
      0.89            -0.4        0.46 ±  8%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.10 ±  8%      -0.4        0.75 ±  6%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      1.46            -0.3        1.11 ±  3%  perf-profile.self.cycles-pp.memset_erms
      1.53 ±  3%      -0.3        1.21 ±  2%  perf-profile.self.cycles-pp.find_get_entry
      1.18 ±  5%      -0.3        0.87 ±  8%  perf-profile.self.cycles-pp.selinux_file_permission
      2.19 ±  4%      -0.2        1.96 ±  3%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      0.83 ±  5%      -0.2        0.65 ±  5%  perf-profile.self.cycles-pp.iomap_apply
      1.31 ±  3%      -0.2        1.13 ±  7%  perf-profile.self.cycles-pp.___might_sleep
      0.60 ±  6%      -0.2        0.42 ± 10%  perf-profile.self.cycles-pp.security_file_permission
      0.77 ±  9%      -0.2        0.60 ±  5%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.81 ±  5%      -0.2        0.65 ±  6%  perf-profile.self.cycles-pp.xas_load
      0.84 ±  6%      -0.2        0.69 ±  9%  perf-profile.self.cycles-pp.xfs_file_iomap_begin
      0.72 ±  3%      -0.2        0.57 ±  4%  perf-profile.self.cycles-pp.write
      0.71 ±  4%      -0.1        0.56 ±  5%  perf-profile.self.cycles-pp.__might_sleep
      0.49 ± 11%      -0.1        0.36 ± 11%  perf-profile.self.cycles-pp.xas_start
      0.76 ±  3%      -0.1        0.63 ±  7%  perf-profile.self.cycles-pp.down_write
      0.66 ±  8%      -0.1        0.54 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.72 ±  6%      -0.1        0.60 ±  4%  perf-profile.self.cycles-pp.up_write
      0.65            -0.1        0.53 ±  4%  perf-profile.self.cycles-pp.iomap_write_end
      0.60 ±  8%      -0.1        0.48 ±  8%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.58 ±  4%      -0.1        0.48 ± 10%  perf-profile.self.cycles-pp.vfs_write
      0.65 ±  3%      -0.1        0.55 ±  5%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      0.52 ±  8%      -0.1        0.42 ±  5%  perf-profile.self.cycles-pp.pagecache_get_page
      0.41 ±  6%      -0.1        0.31 ± 11%  perf-profile.self.cycles-pp.iov_iter_advance
      0.28 ±  6%      -0.1        0.18 ± 11%  perf-profile.self.cycles-pp.generic_write_checks
      0.42 ±  3%      -0.1        0.32 ±  7%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.31 ±  9%      -0.1        0.22 ±  5%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.50 ±  4%      -0.1        0.41 ±  7%  perf-profile.self.cycles-pp.disk_rw
      0.29 ±  5%      -0.1        0.20 ±  9%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.22 ± 11%      -0.1        0.14 ± 11%  perf-profile.self.cycles-pp.xlog_grant_add_space
      0.34 ±  5%      -0.1        0.27 ±  8%  perf-profile.self.cycles-pp.unlock_page
      0.41 ±  8%      -0.1        0.33 ±  5%  perf-profile.self.cycles-pp.xfs_break_leased_layouts
      0.42 ± 13%      -0.1        0.34 ± 17%  perf-profile.self.cycles-pp.new_sync_write
      0.33 ±  7%      -0.1        0.25 ±  2%  perf-profile.self.cycles-pp.file_remove_privs
      0.36            -0.1        0.28 ±  7%  perf-profile.self.cycles-pp.lseek64
      0.30 ± 17%      -0.1        0.23 ±  4%  perf-profile.self.cycles-pp.generic_write_check_limits
      0.23 ±  9%      -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.xfs_break_layouts
      0.22 ±  8%      -0.1        0.15 ±  7%  perf-profile.self.cycles-pp.__fdget_pos
      0.33 ± 10%      -0.1        0.26 ±  5%  perf-profile.self.cycles-pp.file_update_time
      0.17 ± 13%      -0.1        0.10 ± 12%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.40 ±  3%      -0.1        0.34 ±  5%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.25 ± 10%      -0.1        0.19 ±  2%  perf-profile.self.cycles-pp.xas_store
      0.43 ±  2%      -0.1        0.37 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock
      0.22 ±  3%      -0.1        0.17 ± 12%  perf-profile.self.cycles-pp.current_time
      0.21 ±  8%      -0.1        0.16 ± 11%  perf-profile.self.cycles-pp.__x64_sys_lseek
      0.21 ± 10%      -0.1        0.16 ±  8%  perf-profile.self.cycles-pp.release_pages
      0.28 ±  9%      -0.1        0.22        perf-profile.self.cycles-pp.rcu_all_qs
      0.37 ±  7%      -0.1        0.32 ± 10%  perf-profile.self.cycles-pp.__x64_sys_write
      0.35 ±  6%      -0.1        0.30 ±  3%  perf-profile.self.cycles-pp._cond_resched
      0.16 ±  5%      -0.0        0.11 ± 20%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.14 ±  5%      -0.0        0.09 ±  8%  perf-profile.self.cycles-pp.account_page_cleaned
      0.07 ± 11%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.alloc_pages_current
      0.22 ±  3%      -0.0        0.17 ±  7%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.21 ± 13%      -0.0        0.16 ± 10%  perf-profile.self.cycles-pp.ksys_lseek
      0.37 ±  5%      -0.0        0.33 ±  6%  perf-profile.self.cycles-pp.ksys_write
      0.30 ±  4%      -0.0        0.26 ± 10%  perf-profile.self.cycles-pp.page_mapping
      0.18 ± 12%      -0.0        0.14 ±  8%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.14 ± 17%      -0.0        0.10 ± 15%  perf-profile.self.cycles-pp.account_page_dirtied
      0.15 ± 10%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.find_get_entries
      0.17 ±  6%      -0.0        0.13 ± 10%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.11 ±  7%      -0.0        0.07 ± 17%  perf-profile.self.cycles-pp.xfs_isilocked
      0.07 ± 14%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.09 ± 12%      -0.0        0.05 ± 60%  perf-profile.self.cycles-pp.__cancel_dirty_page
      0.09 ± 11%      -0.0        0.06 ± 58%  perf-profile.self.cycles-pp.__x86_indirect_thunk_r15
      0.17 ± 13%      -0.0        0.14 ±  8%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.13 ± 11%      -0.0        0.09 ± 20%  perf-profile.self.cycles-pp.workingset_update_node
      0.14 ±  7%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.timespec64_trunc
      0.11 ±  9%      -0.0        0.08 ± 19%  perf-profile.self.cycles-pp.truncate_inode_pages_range
      0.10 ± 15%      -0.0        0.07 ± 12%  perf-profile.self.cycles-pp.iomap_read_page_sync
      0.07 ± 12%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.mem_cgroup_commit_charge
      0.10 ±  8%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.09 ±  8%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.delete_from_page_cache_batch
      0.09 ±  9%      -0.0        0.07 ± 16%  perf-profile.self.cycles-pp.xfs_dir3_data_entsize
      0.11 ±  4%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.copyin
      0.00            +0.1        0.11 ± 14%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.06 ± 58%      +0.1        0.16 ± 22%  perf-profile.self.cycles-pp.ktime_get
      0.07 ±  5%      +0.1        0.19 ± 15%  perf-profile.self.cycles-pp.menu_select
      0.45 ±  5%      +0.1        0.57 ±  6%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.04 ± 60%      +0.2        0.22 ± 72%  perf-profile.self.cycles-pp.xfs_inobt_get_maxrecs
      0.04 ± 58%      +0.2        0.29 ± 68%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      0.14 ± 36%      +0.3        0.41 ± 22%  perf-profile.self.cycles-pp.osq_lock
     16.78 ±  2%     +11.7       28.46 ±  2%  perf-profile.self.cycles-pp.intel_idle


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  160000 +-+----------------------------------------------------------------+   
  155000 +-+                +                                               |   
         |                  :                                    +          |   
  150000 +-+               ::                                    ::         |   
  145000 +-+               : :                                  : :         |   
         |                :  :    .+.                           :  :     .+.|   
  140000 +-+.+.++.+.+.+.+.+  +.+.+   +.+.+.++.+.+. .+.+.++.+. .+   +.++.+   |   
  135000 +-+                                      +          +              |   
  130000 +-+                                                                |   
         |                                                                  |   
  125000 +-+                                                                |   
  120000 +-+                                                                |   
         |                               O                                  |   
  115000 O-O O OO O O O O O OO O O O O O   O                                |   
  110000 +-+----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  165 +-+-------------------------------------------------------------------+   
  160 O-O O O O O O O O OO               O                                  |   
      |                    O O O O O O                                      |   
  155 +-+                              O                                    |   
  150 +-+                                                                   |   
      |                                                                     |   
  145 +-+                                                                   |   
  140 +-+                                                                   |   
  135 +-+                                                                   |   
      |.+. .+.+. .+.+.+.                    .+. .+.+.+.+.+.++.              |   
  130 +-+ +     +       +  +.+.+.+.+.+.+.+.+   +              +   +.+.+.+.+.|   
  125 +-+               :  :                                   : :          |   
      |                  ::                                    : :          |   
  120 +-+                ::                                     +           |   
  115 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  165 +-+-------------------------------------------------------------------+   
  160 O-O O O O O O O O OO               O                                  |   
      |                    O O O O O O                                      |   
  155 +-+                              O                                    |   
  150 +-+                                                                   |   
      |                                                                     |   
  145 +-+                                                                   |   
  140 +-+                                                                   |   
  135 +-+                                                                   |   
      |.+. .+.+. .+.+.+.                    .+. .+.+.+.+.+.++.              |   
  130 +-+ +     +       +  +.+.+.+.+.+.+.+.+   +              +   +.+.+.+.+.|   
  125 +-+               :  :                                   : :          |   
      |                  ::                                    : :          |   
  120 +-+                ::                                     +           |   
  115 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         aim7.time.voluntary_context_switches                   
                                                                                
   1.1e+06 +-+--------------------------------------------------------------+   
           |                                                                |   
  1.05e+06 +-+.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.++.+.|   
           |                                                                |   
     1e+06 +-+                                                              |   
           |                                                                |   
    950000 +-+                                                              |   
           |                                                                |   
    900000 +-+                                                              |   
           |                                                                |   
    850000 O-+           O   O      O                                       |   
           | O OO O O O O  O   O OO   O O   O                               |   
    800000 +-+                            O                                 |   
           |                                                                |   
    750000 +-+--------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID0/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_cp/aim7

commit: 
  d515962b00 ("xfs: pass around xfs_inode_ag_walk iget/irele helper functions")
  1aabb011bf ("xfs: deferred inode inactivation")

d515962b00d64454 1aabb011bfba5a8942d3e0d0c84 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    204959           -35.7%     131770 ±  3%  aim7.jobs-per-min
     88.12           +55.7%     137.19 ±  3%  aim7.time.elapsed_time
     88.12           +55.7%     137.19 ±  3%  aim7.time.elapsed_time.max
     72330 ±  3%     +12.2%      81167 ±  4%  aim7.time.involuntary_context_switches
    151188            +6.4%     160846 ±  2%  aim7.time.minor_page_faults
      2437           +33.6%       3257 ±  3%  aim7.time.system_time
    305.87           +45.8%     446.05 ±  5%  aim7.time.user_time
    737855           -13.0%     642174 ±  3%  aim7.time.voluntary_context_switches
     22.63            +9.0       31.68        mpstat.cpu.all.idle%
     68.59            -8.5       60.06        mpstat.cpu.all.sys%
  61676265 ± 55%    +177.3%  1.711e+08 ± 18%  cpuidle.C1.time
    612540 ± 47%    +231.9%    2032764 ± 14%  cpuidle.C1.usage
 5.694e+08 ± 13%    +157.5%  1.466e+09 ±  6%  cpuidle.C6.time
    872994 ±  2%    +116.8%    1892610 ±  6%  cpuidle.C6.usage
     23.75 ±  3%     +34.7%      32.00 ±  2%  vmstat.cpu.id
     66.75           -11.6%      59.00        vmstat.cpu.sy
      1534 ± 26%     +87.2%       2873 ±  8%  vmstat.io.bo
     17277 ±  2%     -14.2%      14823        vmstat.system.cs
     24.17           +34.6%      32.54        iostat.cpu.idle
     67.23           -11.8%      59.31        iostat.cpu.system
      8.60            -5.2%       8.15        iostat.cpu.user
     46.02 ±  5%    +114.0%      98.46 ±  3%  iostat.md0.w/s
      1601 ± 25%     +86.8%       2991 ±  7%  iostat.md0.wkB/s
     81296 ±  2%     +47.8%     120146 ±  2%  meminfo.AnonHugePages
     16049 ±  7%     -14.2%      13766        meminfo.Dirty
     16741 ±  6%     -13.7%      14450 ±  2%  meminfo.Inactive(file)
     91250           +22.9%     112105        meminfo.KReclaimable
    109853 ±  5%      -7.0%     102116 ±  8%  meminfo.PageTables
     91250           +22.9%     112105        meminfo.SReclaimable
     36157 ±  2%     -12.9%      31489 ±  4%  meminfo.Shmem
    255575           +14.2%     291816        meminfo.Slab
     27539           -34.1%      18158 ±  4%  meminfo.max_used_kB
      2265           -45.2%       1241 ±  7%  turbostat.Avg_MHz
     77.95            -8.8       69.19        turbostat.Busy%
      2905           -38.3%       1793 ±  7%  turbostat.Bzy_MHz
    608401 ± 47%    +233.6%    2029542 ± 15%  turbostat.C1
      3.84 ± 73%      -3.2        0.68 ±  7%  turbostat.C3%
    868860 ±  2%    +117.3%    1888276 ±  6%  turbostat.C6
     15.87 ± 13%     +10.5       26.42 ±  2%  turbostat.C6%
     14.37 ±  8%     +46.5%      21.05        turbostat.CPU%c1
      6.78 ± 20%     +41.9%       9.62 ±  4%  turbostat.CPU%c6
    121.47           -44.3%      67.72 ±  8%  turbostat.CorWatt
   7419316           +52.0%   11279514 ±  4%  turbostat.IRQ
      3.43 ± 19%     -31.4%       2.35 ± 19%  turbostat.Pkg%pc2
     71.50            -4.0%      68.67        turbostat.PkgTmp
    148.87           -36.5%      94.53 ±  6%  turbostat.PkgWatt
     36.86            -5.5%      34.83        turbostat.RAMWatt
      7146           +55.8%      11133 ±  3%  turbostat.SMI
    623.50 ±105%    +220.2%       1996 ± 37%  numa-meminfo.node0.Active(file)
      8186 ±  3%     -17.1%       6789        numa-meminfo.node0.Dirty
     15459 ± 10%     -26.0%      11435 ±  5%  numa-meminfo.node0.Inactive
      7151 ± 19%     -41.4%       4190 ± 21%  numa-meminfo.node0.Inactive(anon)
      8307 ±  3%     -12.8%       7245 ±  4%  numa-meminfo.node0.Inactive(file)
   1057156 ± 15%     +30.0%    1374548 ±  4%  numa-meminfo.node0.MemUsed
     27710 ±145%    +258.8%      99419 ±  8%  numa-meminfo.node0.PageTables
     65500 ± 45%     +93.9%     126976        numa-meminfo.node0.SUnreclaim
    113557 ± 26%     +57.4%     178704 ±  6%  numa-meminfo.node0.Slab
      1849 ± 35%     -72.2%     513.33 ±141%  numa-meminfo.node1.Active(file)
      8087 ±  6%     -14.6%       6906        numa-meminfo.node1.Dirty
      8654 ±  9%     -17.9%       7105 ±  7%  numa-meminfo.node1.Inactive(file)
     43179 ± 16%     +39.7%      60339 ± 13%  numa-meminfo.node1.KReclaimable
     38009 ± 50%     -89.6%       3961 ±  9%  numa-meminfo.node1.KernelStack
   1284978 ± 12%     -21.8%    1004429 ±  4%  numa-meminfo.node1.MemUsed
     81947 ± 54%     -96.7%       2695        numa-meminfo.node1.PageTables
     43179 ± 16%     +39.7%      60339 ± 13%  numa-meminfo.node1.SReclaimable
    156.25 ±105%    +218.3%     497.33 ± 37%  numa-vmstat.node0.nr_active_file
  16205972 ±  2%     +14.6%   18577156        numa-vmstat.node0.nr_dirtied
      2062 ±  3%     -17.9%       1694 ±  3%  numa-vmstat.node0.nr_dirty
      1788 ± 19%     -41.2%       1051 ± 21%  numa-vmstat.node0.nr_inactive_anon
      2090 ±  7%     -13.2%       1814 ±  4%  numa-vmstat.node0.nr_inactive_file
      6975 ±145%    +255.1%      24766 ±  8%  numa-vmstat.node0.nr_page_table_pages
     16397 ± 45%     +93.3%      31699        numa-vmstat.node0.nr_slab_unreclaimable
    156.25 ±105%    +218.3%     497.33 ± 37%  numa-vmstat.node0.nr_zone_active_file
      1788 ± 19%     -41.2%       1051 ± 21%  numa-vmstat.node0.nr_zone_inactive_anon
      2093 ±  7%     -13.2%       1816 ±  4%  numa-vmstat.node0.nr_zone_inactive_file
  16642980 ±  3%     +14.9%   19130375        numa-vmstat.node0.numa_hit
  16605490 ±  2%     +14.9%   19079593        numa-vmstat.node0.numa_local
    461.25 ± 35%     -72.2%     128.33 ±141%  numa-vmstat.node1.nr_active_file
      2046           -15.2%       1735        numa-vmstat.node1.nr_dirty
      2184 ±  5%     -18.2%       1786 ±  6%  numa-vmstat.node1.nr_inactive_file
     37823 ± 50%     -89.5%       3960 ±  9%  numa-vmstat.node1.nr_kernel_stack
     20375 ± 54%     -96.7%     672.67        numa-vmstat.node1.nr_page_table_pages
     10793 ± 16%     +39.7%      15083 ± 13%  numa-vmstat.node1.nr_slab_reclaimable
    461.25 ± 35%     -72.2%     128.33 ±141%  numa-vmstat.node1.nr_zone_active_file
      2183 ±  5%     -18.3%       1785 ±  6%  numa-vmstat.node1.nr_zone_inactive_file
     91478            -2.1%      89547        proc-vmstat.nr_active_anon
    617.75            +1.5%     627.00        proc-vmstat.nr_active_file
      4045 ±  2%     -15.8%       3405        proc-vmstat.nr_dirty
      5069            -3.6%       4888        proc-vmstat.nr_inactive_anon
      4217 ±  2%     -15.4%       3567        proc-vmstat.nr_inactive_file
     53612            -1.9%      52592        proc-vmstat.nr_kernel_stack
      6369            -3.5%       6149        proc-vmstat.nr_mapped
      9055 ±  2%     -13.1%       7871 ±  4%  proc-vmstat.nr_shmem
     22808           +22.8%      28015        proc-vmstat.nr_slab_reclaimable
     41043            +9.4%      44903        proc-vmstat.nr_slab_unreclaimable
     91478            -2.1%      89547        proc-vmstat.nr_zone_active_anon
    617.75            +1.5%     627.00        proc-vmstat.nr_zone_active_file
      5069            -3.6%       4888        proc-vmstat.nr_zone_inactive_anon
      4217 ±  2%     -15.4%       3567        proc-vmstat.nr_zone_inactive_file
      3453 ±  2%      -8.4%       3163        proc-vmstat.nr_zone_write_pending
      1737 ±142%    +716.8%      14190 ±  3%  proc-vmstat.numa_hint_faults
      2731 ±113%    +389.5%      13370 ±  3%  proc-vmstat.numa_pages_migrated
      4460 ±  7%     -39.5%       2697 ± 14%  proc-vmstat.pgactivate
    402077           +31.0%     526544 ±  2%  proc-vmstat.pgfault
      2731 ±113%    +389.5%      13370 ±  3%  proc-vmstat.pgmigrate_success
    139264 ± 25%    +188.9%     402269 ± 12%  proc-vmstat.pgpgout
      2856          +360.5%      13152 ±  3%  slabinfo.dmaengine-unmap-16.active_objs
     77.75          +589.0%     535.67 ±  3%  slabinfo.dmaengine-unmap-16.active_slabs
      3292          +584.1%      22520 ±  3%  slabinfo.dmaengine-unmap-16.num_objs
     77.75          +589.0%     535.67 ±  3%  slabinfo.dmaengine-unmap-16.num_slabs
      2332 ±  7%     +39.7%       3257        slabinfo.kmalloc-128.active_objs
      2332 ±  7%     +39.7%       3257        slabinfo.kmalloc-128.num_objs
     19136          +126.1%      43271 ±  3%  slabinfo.kmalloc-16.active_objs
     74.75          +139.5%     179.00 ±  3%  slabinfo.kmalloc-16.active_slabs
     19136          +140.5%      46024 ±  3%  slabinfo.kmalloc-16.num_objs
     74.75          +139.5%     179.00 ±  3%  slabinfo.kmalloc-16.num_slabs
    741.00           +24.5%     922.67 ±  2%  slabinfo.kmalloc-4k.active_objs
    751.25           +24.4%     934.67 ±  2%  slabinfo.kmalloc-4k.num_objs
      6130          +175.2%      16869 ±  2%  slabinfo.kmalloc-512.active_objs
    202.00          +207.8%     621.67 ±  2%  slabinfo.kmalloc-512.active_slabs
      6484          +207.0%      19906 ±  2%  slabinfo.kmalloc-512.num_objs
    202.00          +207.8%     621.67 ±  2%  slabinfo.kmalloc-512.num_slabs
    461.50 ±  2%     +43.6%     662.67        slabinfo.kmalloc-8k.active_objs
    476.25 ±  2%     +54.5%     735.67        slabinfo.kmalloc-8k.num_objs
      1160           +72.2%       1998 ±  2%  slabinfo.xfs_buf_item.active_objs
      1160           +72.2%       1998 ±  2%  slabinfo.xfs_buf_item.num_objs
      1186           +64.4%       1950        slabinfo.xfs_efd_item.active_objs
      1186           +64.4%       1950        slabinfo.xfs_efd_item.num_objs
      2168          +487.1%      12729 ±  3%  slabinfo.xfs_inode.active_objs
     87.50          +676.8%     679.67 ±  4%  slabinfo.xfs_inode.active_slabs
      2819          +671.9%      21761 ±  4%  slabinfo.xfs_inode.num_objs
     87.50          +676.8%     679.67 ±  4%  slabinfo.xfs_inode.num_slabs
     13.63 ±  4%      -8.1%      12.53        perf-stat.i.MPKI
  4.74e+09           -32.0%  3.224e+09 ±  4%  perf-stat.i.branch-instructions
  59710263           -29.8%   41920989 ±  3%  perf-stat.i.branch-misses
  24239162 ±  2%     -29.6%   17056459 ±  6%  perf-stat.i.cache-misses
 2.448e+08           -32.9%  1.644e+08 ±  4%  perf-stat.i.cache-references
     17563 ±  2%     -14.8%      14962        perf-stat.i.context-switches
      3.90 ±  2%     -22.0%       3.04 ±  3%  perf-stat.i.cpi
 9.031e+10           -45.5%  4.926e+10 ±  8%  perf-stat.i.cpu-cycles
      3006           -59.5%       1218 ±  2%  perf-stat.i.cpu-migrations
      3432           -23.3%       2633        perf-stat.i.cycles-between-cache-misses
      1.64 ± 12%      -0.4        1.25 ±  2%  perf-stat.i.dTLB-load-miss-rate%
 1.328e+08 ± 13%     -50.8%   65422884 ±  5%  perf-stat.i.dTLB-load-misses
 7.404e+09           -31.8%  5.052e+09 ±  4%  perf-stat.i.dTLB-loads
  16794433 ±  5%     -37.4%   10521517 ±  3%  perf-stat.i.dTLB-store-misses
 4.914e+09           -32.9%    3.3e+09 ±  4%  perf-stat.i.dTLB-stores
     87.61            -3.8       83.84        perf-stat.i.iTLB-load-miss-rate%
   7267620           -33.6%    4828701 ±  2%  perf-stat.i.iTLB-load-misses
 2.391e+10           -31.8%   1.63e+10 ±  4%  perf-stat.i.instructions
      0.28 ±  2%     +28.8%       0.36 ±  2%  perf-stat.i.ipc
      4279           -14.3%       3669        perf-stat.i.minor-faults
     45.88            +0.6       46.48        perf-stat.i.node-load-miss-rate%
  13857418 ±  2%     -33.2%    9253829 ±  7%  perf-stat.i.node-load-misses
  15830808 ±  2%     -33.8%   10483641 ±  7%  perf-stat.i.node-loads
   5925929           -28.2%    4252744 ±  5%  perf-stat.i.node-store-misses
   7451625           -27.1%    5432276 ±  5%  perf-stat.i.node-stores
      4279           -14.3%       3669        perf-stat.i.page-faults
      1.26            +0.0        1.30        perf-stat.overall.branch-miss-rate%
      9.90            +0.5       10.37 ±  2%  perf-stat.overall.cache-miss-rate%
      3.78           -20.1%       3.02 ±  3%  perf-stat.overall.cpi
      3726           -22.6%       2884        perf-stat.overall.cycles-between-cache-misses
      1.76 ± 12%      -0.5        1.28 ±  2%  perf-stat.overall.dTLB-load-miss-rate%
      0.34 ±  5%      -0.0        0.32        perf-stat.overall.dTLB-store-miss-rate%
     87.44            -3.8       83.60        perf-stat.overall.iTLB-load-miss-rate%
      3290            +2.6%       3374        perf-stat.overall.instructions-per-iTLB-miss
      0.26           +25.4%       0.33 ±  3%  perf-stat.overall.ipc
 4.685e+09           -31.7%  3.199e+09 ±  4%  perf-stat.ps.branch-instructions
  59020206           -29.5%   41606551 ±  3%  perf-stat.ps.branch-misses
  23958401 ±  2%     -29.3%   16929391 ±  6%  perf-stat.ps.cache-misses
  2.42e+08           -32.6%  1.631e+08 ±  4%  perf-stat.ps.cache-references
     17359 ±  2%     -14.5%      14848        perf-stat.ps.context-switches
 8.926e+10           -45.2%   4.89e+10 ±  8%  perf-stat.ps.cpu-cycles
      2971           -59.3%       1208 ±  2%  perf-stat.ps.cpu-migrations
 1.313e+08 ± 13%     -50.5%   64933807 ±  5%  perf-stat.ps.dTLB-load-misses
 7.318e+09           -31.5%  5.014e+09 ±  4%  perf-stat.ps.dTLB-loads
  16598516 ±  5%     -37.1%   10442914 ±  3%  perf-stat.ps.dTLB-store-misses
 4.857e+09           -32.6%  3.275e+09 ±  4%  perf-stat.ps.dTLB-stores
   7182997           -33.3%    4792440 ±  2%  perf-stat.ps.iTLB-load-misses
 2.363e+10           -31.5%  1.618e+10 ±  4%  perf-stat.ps.instructions
      4233           -14.0%       3642        perf-stat.ps.minor-faults
  13695799 ±  2%     -32.9%    9184667 ±  7%  perf-stat.ps.node-load-misses
  15646175 ±  2%     -33.5%   10405160 ±  7%  perf-stat.ps.node-loads
   5856803           -27.9%    4221222 ±  5%  perf-stat.ps.node-store-misses
   7364898           -26.8%    5392052 ±  5%  perf-stat.ps.node-stores
      4233           -14.0%       3642        perf-stat.ps.page-faults
 2.091e+12            +6.2%  2.221e+12        perf-stat.total.instructions
     21578           +80.2%      38887        sched_debug.cfs_rq:/.exec_clock.avg
     23353           +75.3%      40945 ±  2%  sched_debug.cfs_rq:/.exec_clock.max
     21322           +79.5%      38269        sched_debug.cfs_rq:/.exec_clock.min
    304.37 ±  8%     +41.5%     430.55 ± 25%  sched_debug.cfs_rq:/.exec_clock.stddev
    632.50 ± 12%     +29.7%     820.11 ± 12%  sched_debug.cfs_rq:/.load_avg.max
    780200           +60.8%    1254863        sched_debug.cfs_rq:/.min_vruntime.avg
    872093           +58.1%    1378898 ±  3%  sched_debug.cfs_rq:/.min_vruntime.max
    762295           +57.2%    1198056        sched_debug.cfs_rq:/.min_vruntime.min
     17394 ±  5%     +57.7%      27431 ± 14%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.40 ±  5%     +15.0%       0.46 ±  8%  sched_debug.cfs_rq:/.nr_running.stddev
      0.13 ± 38%    +111.6%       0.28 ± 18%  sched_debug.cfs_rq:/.nr_spread_over.avg
      0.31 ± 29%     +63.8%       0.50 ± 13%  sched_debug.cfs_rq:/.nr_spread_over.stddev
     -1508          -590.1%       7393 ± 78%  sched_debug.cfs_rq:/.spread0.avg
    -19404          +154.1%     -49302        sched_debug.cfs_rq:/.spread0.min
     17422 ±  5%     +57.4%      27431 ± 14%  sched_debug.cfs_rq:/.spread0.stddev
    777.17           -15.3%     658.55 ± 11%  sched_debug.cfs_rq:/.util_avg.avg
      1497 ±  8%     -16.1%       1256 ± 15%  sched_debug.cfs_rq:/.util_avg.max
    123.49 ± 11%     -31.9%      84.04 ± 20%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    473039 ±  7%     +37.9%     652433 ±  2%  sched_debug.cpu.avg_idle.avg
    236058 ± 12%     -18.6%     192130 ±  4%  sched_debug.cpu.avg_idle.stddev
     71615           +43.2%     102524        sched_debug.cpu.clock.avg
     71626           +43.2%     102548        sched_debug.cpu.clock.max
     71600           +43.2%     102498        sched_debug.cpu.clock.min
      7.20 ± 22%    +102.5%      14.59 ± 12%  sched_debug.cpu.clock.stddev
     71615           +43.2%     102524        sched_debug.cpu.clock_task.avg
     71626           +43.2%     102548        sched_debug.cpu.clock_task.max
     71600           +43.2%     102498        sched_debug.cpu.clock_task.min
      7.20 ± 22%    +102.5%      14.59 ± 12%  sched_debug.cpu.clock_task.stddev
      1820 ± 11%     -18.6%       1482 ± 14%  sched_debug.cpu.curr->pid.avg
      3837 ±  8%     +37.7%       5283        sched_debug.cpu.curr->pid.max
      0.00 ±  3%     +32.7%       0.00 ± 15%  sched_debug.cpu.next_balance.stddev
      0.42 ±  8%     +17.3%       0.49 ±  4%  sched_debug.cpu.nr_running.stddev
     14410 ±  2%     +63.7%      23596        sched_debug.cpu.nr_switches.avg
     20732          +148.7%      51565 ±  9%  sched_debug.cpu.nr_switches.max
     12235           +29.2%      15807 ±  2%  sched_debug.cpu.nr_switches.min
      1916 ±  5%    +373.8%       9082 ±  8%  sched_debug.cpu.nr_switches.stddev
     37.10           +17.6%      43.62 ±  4%  sched_debug.cpu.nr_uninterruptible.avg
    -42.25           -35.6%     -27.22        sched_debug.cpu.nr_uninterruptible.min
     12279 ±  2%     +74.6%      21437        sched_debug.cpu.sched_count.avg
     14279 ±  2%    +231.0%      47258 ± 10%  sched_debug.cpu.sched_count.max
     11249 ±  2%     +31.0%      14739        sched_debug.cpu.sched_count.min
    621.89 ± 12%   +1268.2%       8508 ± 10%  sched_debug.cpu.sched_count.stddev
      5282 ±  3%     +80.1%       9514        sched_debug.cpu.sched_goidle.avg
      6253 ±  2%    +253.4%      22097 ± 10%  sched_debug.cpu.sched_goidle.max
      4834 ±  2%     +32.4%       6403        sched_debug.cpu.sched_goidle.min
    301.81 ± 13%   +1275.2%       4150 ± 10%  sched_debug.cpu.sched_goidle.stddev
      6298 ±  2%     +69.2%      10658        sched_debug.cpu.ttwu_count.avg
      7777          +200.5%      23368 ± 10%  sched_debug.cpu.ttwu_count.max
      5825 ±  3%     +23.6%       7199        sched_debug.cpu.ttwu_count.min
    379.64 ±  7%   +1010.1%       4214 ± 11%  sched_debug.cpu.ttwu_count.stddev
    526.01           +69.8%     893.13 ±  2%  sched_debug.cpu.ttwu_local.avg
    773.75 ±  8%    +125.6%       1745 ±  9%  sched_debug.cpu.ttwu_local.max
    424.12           +40.9%     597.78 ±  3%  sched_debug.cpu.ttwu_local.min
     68.72 ± 10%    +219.5%     219.59 ± 10%  sched_debug.cpu.ttwu_local.stddev
     71601           +43.1%     102497        sched_debug.cpu_clk
     68011           +45.4%      98899        sched_debug.ktime
     71961           +42.9%     102856        sched_debug.sched_clk
     13240 ±  4%     +36.2%      18033 ±  3%  softirqs.CPU0.RCU
     12004 ±  3%     +24.9%      14997 ±  3%  softirqs.CPU0.SCHED
     35478 ±  3%     +57.3%      55794        softirqs.CPU0.TIMER
      9817 ±  7%     +18.1%      11592 ±  7%  softirqs.CPU1.SCHED
     38872 ±  3%     +42.3%      55318        softirqs.CPU1.TIMER
     13094 ±  5%     +33.8%      17518 ±  5%  softirqs.CPU10.RCU
      8546 ±  8%     +35.4%      11568 ±  6%  softirqs.CPU10.SCHED
     33896 ±  3%     +59.6%      54082 ±  2%  softirqs.CPU10.TIMER
     12959 ±  4%     +33.4%      17288 ±  4%  softirqs.CPU11.RCU
      8569           +32.8%      11380 ±  4%  softirqs.CPU11.SCHED
     34467 ±  2%     +56.0%      53759        softirqs.CPU11.TIMER
     14180 ± 12%     +52.6%      21643 ±  4%  softirqs.CPU12.RCU
      8429 ±  3%     +36.6%      11515 ±  6%  softirqs.CPU12.SCHED
     36338 ±  7%     +59.4%      57933 ±  6%  softirqs.CPU12.TIMER
     13213 ±  5%     +33.5%      17641 ±  5%  softirqs.CPU13.RCU
      8630           +33.2%      11498 ±  4%  softirqs.CPU13.SCHED
     38830 ± 17%     +40.0%      54352        softirqs.CPU13.TIMER
     12844           +39.2%      17877 ±  3%  softirqs.CPU14.RCU
      8556           +37.5%      11765 ±  4%  softirqs.CPU14.SCHED
     39110 ± 16%     +38.8%      54296 ±  3%  softirqs.CPU14.TIMER
     13167 ±  4%     +35.0%      17771 ±  3%  softirqs.CPU15.RCU
      8742           +29.3%      11303 ±  4%  softirqs.CPU15.SCHED
     35002 ±  2%     +53.5%      53737        softirqs.CPU15.TIMER
     13022           +35.7%      17668 ±  5%  softirqs.CPU16.RCU
      8456 ±  2%     +37.4%      11616 ±  3%  softirqs.CPU16.SCHED
     35793 ±  3%     +52.5%      54602 ±  2%  softirqs.CPU16.TIMER
     13010 ±  2%     +36.1%      17702 ± 10%  softirqs.CPU17.RCU
      8813 ±  4%     +26.7%      11168 ±  3%  softirqs.CPU17.SCHED
     35279           +53.4%      54103        softirqs.CPU17.TIMER
      8652 ±  2%     +29.9%      11243        softirqs.CPU18.SCHED
     35157 ±  4%     +55.9%      54825 ±  5%  softirqs.CPU18.TIMER
     13004           +44.6%      18807 ± 12%  softirqs.CPU19.RCU
      8658           +28.9%      11159 ±  2%  softirqs.CPU19.SCHED
     34629           +55.5%      53864        softirqs.CPU19.TIMER
      8765 ±  7%     +35.4%      11871 ±  3%  softirqs.CPU2.SCHED
     35122 ±  6%     +53.9%      54041 ±  2%  softirqs.CPU2.TIMER
     12585           +54.3%      19421 ± 14%  softirqs.CPU20.RCU
      8327 ±  2%     +40.1%      11666        softirqs.CPU20.SCHED
     33760 ±  2%     +59.5%      53863 ±  2%  softirqs.CPU20.TIMER
     12746 ±  2%     +45.1%      18498 ± 10%  softirqs.CPU21.RCU
      8471 ±  4%     +41.3%      11968 ±  4%  softirqs.CPU21.SCHED
     35143 ±  2%     +73.9%      61115 ±  7%  softirqs.CPU21.TIMER
     13098           +34.9%      17671 ±  4%  softirqs.CPU22.RCU
      8566           +30.3%      11165 ±  6%  softirqs.CPU22.SCHED
     34415 ±  2%     +60.1%      55110        softirqs.CPU22.TIMER
     13351 ±  5%     +31.5%      17556 ±  5%  softirqs.CPU23.RCU
      9174 ±  8%     +26.7%      11622 ±  6%  softirqs.CPU23.SCHED
     37855 ± 13%     +54.2%      58371 ±  4%  softirqs.CPU23.TIMER
     13334 ±  4%     +35.3%      18035 ±  9%  softirqs.CPU24.RCU
      9011 ±  9%     +28.1%      11546 ± 11%  softirqs.CPU24.SCHED
     35153 ±  5%     +56.0%      54856        softirqs.CPU24.TIMER
     13056           +35.1%      17633 ±  5%  softirqs.CPU25.RCU
      8668           +31.9%      11435 ±  6%  softirqs.CPU25.SCHED
     34832 ±  2%     +56.5%      54499        softirqs.CPU25.TIMER
     13612 ±  6%     +31.1%      17846 ±  4%  softirqs.CPU26.RCU
      8564           +33.0%      11392 ±  5%  softirqs.CPU26.SCHED
     34752 ±  3%     +67.9%      58356 ±  9%  softirqs.CPU26.TIMER
     13063           +38.7%      18123 ±  5%  softirqs.CPU27.RCU
      8692           +24.6%      10831 ±  4%  softirqs.CPU27.SCHED
     34964 ±  2%     +57.0%      54889        softirqs.CPU27.TIMER
     12612           +37.8%      17376 ±  4%  softirqs.CPU28.RCU
      8625 ±  2%     +34.5%      11605 ±  7%  softirqs.CPU28.SCHED
     34376 ±  3%     +59.0%      54672        softirqs.CPU28.TIMER
     12943 ±  4%     +33.9%      17333 ±  6%  softirqs.CPU29.RCU
      8678 ±  2%     +28.6%      11158 ±  5%  softirqs.CPU29.SCHED
     36977 ± 14%     +47.1%      54377        softirqs.CPU29.TIMER
     12972           +42.8%      18526 ± 14%  softirqs.CPU3.RCU
      9119 ±  5%     +25.9%      11481 ±  6%  softirqs.CPU3.SCHED
     35879           +75.6%      63012 ± 17%  softirqs.CPU3.TIMER
     12581           +39.8%      17591 ±  5%  softirqs.CPU30.RCU
      8255           +31.4%      10844 ±  7%  softirqs.CPU30.SCHED
     33406 ±  3%     +58.7%      53003 ±  3%  softirqs.CPU30.TIMER
     13201 ±  5%     +32.6%      17503 ±  5%  softirqs.CPU31.RCU
      8582           +30.9%      11232 ±  5%  softirqs.CPU31.SCHED
     34045           +57.9%      53746        softirqs.CPU31.TIMER
     10821 ±  8%     +37.8%      14913        softirqs.CPU32.RCU
      8508 ±  3%     +37.7%      11712 ±  4%  softirqs.CPU32.SCHED
     34901 ±  3%     +59.6%      55704 ±  3%  softirqs.CPU32.TIMER
     10359           +34.3%      13912 ±  5%  softirqs.CPU33.RCU
      8624           +29.3%      11149 ±  3%  softirqs.CPU33.SCHED
     35940 ±  5%     +51.3%      54362 ±  2%  softirqs.CPU33.TIMER
     10512 ±  2%     +36.5%      14346 ±  6%  softirqs.CPU34.RCU
      8604           +28.7%      11077 ±  3%  softirqs.CPU34.SCHED
     34636 ±  2%     +53.5%      53154 ±  3%  softirqs.CPU34.TIMER
     10444 ±  2%     +35.0%      14100 ±  7%  softirqs.CPU35.RCU
      8547           +27.7%      10915 ±  5%  softirqs.CPU35.SCHED
     34476           +55.8%      53719        softirqs.CPU35.TIMER
     10196 ±  2%     +44.4%      14724 ±  4%  softirqs.CPU36.RCU
      8427 ±  2%     +34.3%      11321 ±  3%  softirqs.CPU36.SCHED
     34764 ±  2%     +56.2%      54288 ±  3%  softirqs.CPU36.TIMER
     10273 ±  2%     +33.6%      13728 ±  5%  softirqs.CPU37.RCU
      8527 ±  2%     +27.6%      10885 ±  3%  softirqs.CPU37.SCHED
     34727           +54.0%      53489        softirqs.CPU37.TIMER
     10103 ±  2%     +44.2%      14571 ±  2%  softirqs.CPU38.RCU
      8356 ±  2%     +37.1%      11454 ±  4%  softirqs.CPU38.SCHED
     35884 ±  9%     +68.3%      60394 ± 12%  softirqs.CPU38.TIMER
     10341 ±  3%     +32.3%      13681 ±  5%  softirqs.CPU39.RCU
      8556           +27.0%      10868 ±  6%  softirqs.CPU39.SCHED
     34267           +56.6%      53654        softirqs.CPU39.TIMER
     13012           +35.5%      17627 ±  5%  softirqs.CPU4.RCU
      8735 ±  2%     +27.5%      11142 ±  5%  softirqs.CPU4.SCHED
     38127 ± 18%     +44.2%      54976        softirqs.CPU4.TIMER
     12911           +35.8%      17529 ±  4%  softirqs.CPU5.RCU
      8699           +38.2%      12025 ± 13%  softirqs.CPU5.SCHED
     34819 ±  2%     +61.8%      56335 ±  2%  softirqs.CPU5.TIMER
     12740 ±  2%     +35.8%      17297 ±  5%  softirqs.CPU6.RCU
      8761 ±  2%     +27.0%      11124 ±  4%  softirqs.CPU6.SCHED
     35057 ±  3%     +74.8%      61271 ± 16%  softirqs.CPU6.TIMER
     12939           +36.1%      17608 ±  7%  softirqs.CPU7.RCU
      8556 ±  2%     +28.7%      11015 ±  4%  softirqs.CPU7.SCHED
     35488 ±  4%     +54.8%      54918        softirqs.CPU7.TIMER
     12719 ±  2%     +37.1%      17432 ±  2%  softirqs.CPU8.RCU
      8648           +32.3%      11438 ±  6%  softirqs.CPU8.SCHED
     34507 ±  2%     +58.6%      54712        softirqs.CPU8.TIMER
     12769 ±  3%     +35.1%      17249 ±  7%  softirqs.CPU9.RCU
      8597 ±  3%     +33.3%      11461 ±  2%  softirqs.CPU9.SCHED
     38857 ± 19%     +40.5%      54604        softirqs.CPU9.TIMER
    503590           +36.6%     688115 ±  4%  softirqs.RCU
    349541           +31.1%     458229 ±  4%  softirqs.SCHED
   1420004           +56.1%    2216175        softirqs.TIMER
     77.00 ± 51%   +3025.1%       2406 ±130%  interrupts.40:IR-PCI-MSI.524290-edge.eth0-TxRx-1
     77.00 ± 39%     +71.4%     132.00 ± 30%  interrupts.43:IR-PCI-MSI.524293-edge.eth0-TxRx-4
     54.75 ±  8%     +64.4%      90.00 ± 21%  interrupts.46:IR-PCI-MSI.524296-edge.eth0-TxRx-7
     38565           +40.0%      54002 ±  4%  interrupts.CAL:Function_call_interrupts
    983.50 ±  4%     +36.0%       1338 ±  8%  interrupts.CPU0.CAL:Function_call_interrupts
    177382           +55.8%     276400 ±  3%  interrupts.CPU0.LOC:Local_timer_interrupts
      4852 ± 66%     -87.6%     601.33 ±141%  interrupts.CPU0.NMI:Non-maskable_interrupts
      4852 ± 66%     -87.6%     601.33 ±141%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
    177458           +55.6%     276134 ±  3%  interrupts.CPU1.LOC:Local_timer_interrupts
      2399 ± 34%     -39.3%       1456 ±  8%  interrupts.CPU1.RES:Rescheduling_interrupts
      1007           +38.4%       1394 ±  7%  interrupts.CPU10.CAL:Function_call_interrupts
    175614 ±  2%     +57.4%     276332 ±  4%  interrupts.CPU10.LOC:Local_timer_interrupts
      4849 ± 33%     -54.8%       2193 ± 27%  interrupts.CPU10.NMI:Non-maskable_interrupts
      4849 ± 33%     -54.8%       2193 ± 27%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      2487 ± 31%     -33.4%       1655 ± 15%  interrupts.CPU10.RES:Rescheduling_interrupts
      1005 ±  2%     +33.0%       1337 ±  3%  interrupts.CPU11.CAL:Function_call_interrupts
    177053           +56.1%     276460 ±  3%  interrupts.CPU11.LOC:Local_timer_interrupts
      3695 ±  9%     -52.1%       1768 ±  4%  interrupts.CPU11.NMI:Non-maskable_interrupts
      3695 ±  9%     -52.1%       1768 ±  4%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    972.00 ±  5%     +41.8%       1378 ±  2%  interrupts.CPU12.CAL:Function_call_interrupts
    175821 ±  2%     +57.2%     276353 ±  4%  interrupts.CPU12.LOC:Local_timer_interrupts
      1947 ±  5%     -23.8%       1483 ± 16%  interrupts.CPU12.RES:Rescheduling_interrupts
    922.25 ±  6%     +42.0%       1309        interrupts.CPU13.CAL:Function_call_interrupts
    176892           +55.9%     275829 ±  3%  interrupts.CPU13.LOC:Local_timer_interrupts
    979.25 ±  5%     +31.0%       1282 ±  4%  interrupts.CPU14.CAL:Function_call_interrupts
    176168 ±  2%     +56.8%     276178 ±  3%  interrupts.CPU14.LOC:Local_timer_interrupts
    955.75 ±  3%     +40.3%       1341 ±  6%  interrupts.CPU15.CAL:Function_call_interrupts
    176793 ±  2%     +56.1%     276062 ±  3%  interrupts.CPU15.LOC:Local_timer_interrupts
      1858 ±  4%     -19.4%       1497 ± 13%  interrupts.CPU15.RES:Rescheduling_interrupts
    969.50 ±  3%     +35.7%       1315 ±  6%  interrupts.CPU16.CAL:Function_call_interrupts
    176054 ±  2%     +56.8%     276087 ±  3%  interrupts.CPU16.LOC:Local_timer_interrupts
      1927 ±  5%     -24.2%       1461 ±  8%  interrupts.CPU16.RES:Rescheduling_interrupts
    937.25 ±  2%     +39.8%       1310 ±  9%  interrupts.CPU17.CAL:Function_call_interrupts
    176586 ±  2%     +55.6%     274709 ±  3%  interrupts.CPU17.LOC:Local_timer_interrupts
    972.00 ±  4%     +41.1%       1371 ±  9%  interrupts.CPU18.CAL:Function_call_interrupts
    175927 ±  2%     +56.9%     275969 ±  3%  interrupts.CPU18.LOC:Local_timer_interrupts
      2285 ± 18%     -42.0%       1325 ±  7%  interrupts.CPU18.RES:Rescheduling_interrupts
    958.25 ±  4%     +37.9%       1321 ±  6%  interrupts.CPU19.CAL:Function_call_interrupts
    176354 ±  2%     +56.0%     275068 ±  4%  interrupts.CPU19.LOC:Local_timer_interrupts
      1007 ±  4%     +34.8%       1357 ± 11%  interrupts.CPU2.CAL:Function_call_interrupts
    176971           +56.1%     276231 ±  3%  interrupts.CPU2.LOC:Local_timer_interrupts
    948.00 ±  5%     +46.7%       1390 ±  6%  interrupts.CPU20.CAL:Function_call_interrupts
    176878           +56.1%     276171 ±  3%  interrupts.CPU20.LOC:Local_timer_interrupts
    970.25 ±  4%     +33.5%       1295 ±  3%  interrupts.CPU21.CAL:Function_call_interrupts
    177571           +55.5%     276093 ±  4%  interrupts.CPU21.LOC:Local_timer_interrupts
      1893           -34.6%       1237 ±  6%  interrupts.CPU21.RES:Rescheduling_interrupts
    963.50 ±  2%     +34.6%       1296 ±  2%  interrupts.CPU22.CAL:Function_call_interrupts
    176684           +56.3%     276201 ±  3%  interrupts.CPU22.LOC:Local_timer_interrupts
      1975 ±  2%     -35.7%       1270 ± 12%  interrupts.CPU22.RES:Rescheduling_interrupts
    949.50 ±  3%     +32.7%       1260 ±  2%  interrupts.CPU23.CAL:Function_call_interrupts
    177742           +55.1%     275624 ±  3%  interrupts.CPU23.LOC:Local_timer_interrupts
      1937 ±  4%     -34.9%       1261 ±  9%  interrupts.CPU23.RES:Rescheduling_interrupts
    968.75 ±  5%     +41.5%       1370 ±  7%  interrupts.CPU24.CAL:Function_call_interrupts
    176809           +56.1%     275962 ±  3%  interrupts.CPU24.LOC:Local_timer_interrupts
      1893           -30.5%       1316 ± 14%  interrupts.CPU24.RES:Rescheduling_interrupts
    949.00 ±  2%     +41.6%       1344 ±  4%  interrupts.CPU25.CAL:Function_call_interrupts
    177584           +55.4%     275974 ±  4%  interrupts.CPU25.LOC:Local_timer_interrupts
    922.50 ±  9%     +43.9%       1327 ±  4%  interrupts.CPU26.CAL:Function_call_interrupts
    177144           +55.9%     276093 ±  3%  interrupts.CPU26.LOC:Local_timer_interrupts
      4881 ± 35%     -76.1%       1165 ±141%  interrupts.CPU26.NMI:Non-maskable_interrupts
      4881 ± 35%     -76.1%       1165 ±141%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
    961.75 ±  5%     +32.1%       1270 ±  7%  interrupts.CPU27.CAL:Function_call_interrupts
    177543           +55.6%     276207 ±  4%  interrupts.CPU27.LOC:Local_timer_interrupts
      1920           -39.2%       1167 ±  7%  interrupts.CPU27.RES:Rescheduling_interrupts
    948.25 ±  3%     +43.4%       1359 ± 10%  interrupts.CPU28.CAL:Function_call_interrupts
    176358 ±  2%     +56.6%     276170 ±  3%  interrupts.CPU28.LOC:Local_timer_interrupts
    926.25 ±  2%     +52.0%       1408 ±  3%  interrupts.CPU29.CAL:Function_call_interrupts
    177193           +56.0%     276457 ±  3%  interrupts.CPU29.LOC:Local_timer_interrupts
      1014 ±112%    +163.8%       2677 ± 24%  interrupts.CPU29.NMI:Non-maskable_interrupts
      1014 ±112%    +163.8%       2677 ± 24%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      1927 ±  2%     -30.8%       1334 ±  9%  interrupts.CPU29.RES:Rescheduling_interrupts
    983.25 ±  4%     +43.7%       1413        interrupts.CPU3.CAL:Function_call_interrupts
    177682           +55.6%     276520 ±  3%  interrupts.CPU3.LOC:Local_timer_interrupts
      1974 ±  2%     -27.2%       1436 ±  9%  interrupts.CPU3.RES:Rescheduling_interrupts
    924.75 ±  2%     +54.0%       1424 ±  3%  interrupts.CPU30.CAL:Function_call_interrupts
    176131 ±  2%     +56.7%     276051 ±  3%  interrupts.CPU30.LOC:Local_timer_interrupts
      1888 ±  3%     -35.4%       1221        interrupts.CPU30.RES:Rescheduling_interrupts
    960.50 ±  4%     +42.4%       1367 ±  6%  interrupts.CPU31.CAL:Function_call_interrupts
    177002           +56.1%     276368 ±  3%  interrupts.CPU31.LOC:Local_timer_interrupts
      1939           -29.2%       1373 ± 14%  interrupts.CPU31.RES:Rescheduling_interrupts
     77.00 ± 51%   +3025.1%       2406 ±130%  interrupts.CPU32.40:IR-PCI-MSI.524290-edge.eth0-TxRx-1
    914.25 ± 14%     +51.8%       1388 ±  8%  interrupts.CPU32.CAL:Function_call_interrupts
    176087 ±  2%     +56.7%     276010 ±  3%  interrupts.CPU32.LOC:Local_timer_interrupts
      1947 ±  2%     -27.7%       1407 ± 11%  interrupts.CPU32.RES:Rescheduling_interrupts
    994.25 ±  3%     +39.9%       1390 ±  5%  interrupts.CPU33.CAL:Function_call_interrupts
    176835           +56.1%     276100 ±  3%  interrupts.CPU33.LOC:Local_timer_interrupts
      1923 ±  2%     -32.0%       1308 ± 15%  interrupts.CPU33.RES:Rescheduling_interrupts
    976.75 ±  3%     +43.2%       1398 ±  7%  interrupts.CPU34.CAL:Function_call_interrupts
    175901 ±  2%     +56.8%     275892 ±  3%  interrupts.CPU34.LOC:Local_timer_interrupts
      1920 ±  3%     -35.9%       1230 ±  6%  interrupts.CPU34.RES:Rescheduling_interrupts
    995.75 ±  3%     +35.5%       1349 ±  2%  interrupts.CPU35.CAL:Function_call_interrupts
    176626           +56.4%     276218 ±  3%  interrupts.CPU35.LOC:Local_timer_interrupts
      2002 ±  6%     -34.0%       1322 ±  4%  interrupts.CPU35.RES:Rescheduling_interrupts
    977.75 ±  3%     +41.3%       1381 ±  5%  interrupts.CPU36.CAL:Function_call_interrupts
    175681 ±  2%     +57.0%     275759 ±  3%  interrupts.CPU36.LOC:Local_timer_interrupts
      1944           -35.9%       1245 ±  3%  interrupts.CPU36.RES:Rescheduling_interrupts
    967.00 ±  5%     +42.5%       1378 ±  4%  interrupts.CPU37.CAL:Function_call_interrupts
    176570 ±  2%     +55.7%     274976 ±  3%  interrupts.CPU37.LOC:Local_timer_interrupts
      4719 ± 35%     -74.9%       1184 ± 67%  interrupts.CPU37.NMI:Non-maskable_interrupts
      4719 ± 35%     -74.9%       1184 ± 67%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      1897           -31.3%       1302 ± 13%  interrupts.CPU37.RES:Rescheduling_interrupts
     77.00 ± 39%     +71.4%     132.00 ± 30%  interrupts.CPU38.43:IR-PCI-MSI.524293-edge.eth0-TxRx-4
    953.50 ±  2%     +42.8%       1361        interrupts.CPU38.CAL:Function_call_interrupts
    175461 ±  3%     +57.2%     275862 ±  4%  interrupts.CPU38.LOC:Local_timer_interrupts
      1919 ±  3%     -32.3%       1300 ± 10%  interrupts.CPU38.RES:Rescheduling_interrupts
    977.75 ±  4%     +34.6%       1316 ±  6%  interrupts.CPU39.CAL:Function_call_interrupts
    176475 ±  2%     +56.5%     276208 ±  3%  interrupts.CPU39.LOC:Local_timer_interrupts
      1884           -33.0%       1263 ±  7%  interrupts.CPU39.RES:Rescheduling_interrupts
    897.00 ± 12%     +56.3%       1402 ±  8%  interrupts.CPU4.CAL:Function_call_interrupts
    176870           +56.2%     276260 ±  3%  interrupts.CPU4.LOC:Local_timer_interrupts
      2133 ± 10%     -36.2%       1361 ± 20%  interrupts.CPU4.RES:Rescheduling_interrupts
    995.00 ±  3%     +36.3%       1356 ±  5%  interrupts.CPU5.CAL:Function_call_interrupts
    177709           +55.3%     275932 ±  4%  interrupts.CPU5.LOC:Local_timer_interrupts
      5802 ± 33%     -62.7%       2163 ± 72%  interrupts.CPU5.NMI:Non-maskable_interrupts
      5802 ± 33%     -62.7%       2163 ± 72%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
     54.75 ±  8%     +64.4%      90.00 ± 21%  interrupts.CPU6.46:IR-PCI-MSI.524296-edge.eth0-TxRx-7
    931.75 ±  3%     +46.4%       1364 ±  7%  interrupts.CPU6.CAL:Function_call_interrupts
    176814           +56.1%     275995 ±  3%  interrupts.CPU6.LOC:Local_timer_interrupts
      1.25 ± 66%  +1.4e+05%       1764 ± 77%  interrupts.CPU6.NMI:Non-maskable_interrupts
      1.25 ± 66%  +1.4e+05%       1764 ± 77%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      2020 ±  4%     -26.1%       1493 ± 21%  interrupts.CPU6.RES:Rescheduling_interrupts
    951.50 ±  3%     +42.3%       1353 ±  5%  interrupts.CPU7.CAL:Function_call_interrupts
    177433           +55.7%     276226 ±  4%  interrupts.CPU7.LOC:Local_timer_interrupts
      1961           -39.5%       1187 ±  2%  interrupts.CPU7.RES:Rescheduling_interrupts
    958.00 ±  4%     +46.1%       1400 ±  3%  interrupts.CPU8.CAL:Function_call_interrupts
    176160 ±  2%     +56.6%     275946 ±  3%  interrupts.CPU8.LOC:Local_timer_interrupts
      2051 ±  3%     -33.9%       1356 ±  5%  interrupts.CPU8.RES:Rescheduling_interrupts
      1045 ±  2%     +26.9%       1326 ±  6%  interrupts.CPU9.CAL:Function_call_interrupts
    177126           +55.9%     276223 ±  3%  interrupts.CPU9.LOC:Local_timer_interrupts
      4878 ± 34%     -63.5%       1780 ± 80%  interrupts.CPU9.NMI:Non-maskable_interrupts
      4878 ± 34%     -63.5%       1780 ± 80%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
   7069159           +56.2%   11041322 ±  3%  interrupts.LOC:Local_timer_interrupts
    118316 ±  7%     -46.4%      63378 ± 17%  interrupts.NMI:Non-maskable_interrupts
    118316 ±  7%     -46.4%      63378 ± 17%  interrupts.PMI:Performance_monitoring_interrupts
     79699           -24.7%      60027 ±  2%  interrupts.RES:Rescheduling_interrupts
    176.75 ± 10%     +71.8%     303.67 ± 10%  interrupts.TLB:TLB_shootdowns
     58.95           -15.1       43.86 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     59.10           -15.1       44.02 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     54.72           -14.7       39.98 ±  4%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     54.25           -14.7       39.52 ±  4%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     61.93           -14.6       47.29 ±  3%  perf-profile.calltrace.cycles-pp.read
     45.30           -12.1       33.15 ±  2%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     38.88           -10.1       28.73 ±  2%  perf-profile.calltrace.cycles-pp.xfs_file_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
     36.45 ±  2%      -9.5       26.96        perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read.ksys_read
     20.84            -5.5       15.37 ±  3%  perf-profile.calltrace.cycles-pp.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read
      9.89 ±  4%      -3.2        6.69 ±  7%  perf-profile.calltrace.cycles-pp.touch_atime.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
      9.80 ±  4%      -3.2        6.62 ±  8%  perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter
      9.55 ±  6%      -2.6        6.97 ±  5%  perf-profile.calltrace.cycles-pp.down_read.xfs_ilock.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
      9.61 ±  7%      -2.6        7.04 ±  5%  perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read
      5.55 ± 10%      -1.7        3.83 ± 11%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.80 ±  5%      -1.5        4.32        perf-profile.calltrace.cycles-pp.up_read.xfs_iunlock.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
      5.86 ±  5%      -1.4        4.41        perf-profile.calltrace.cycles-pp.xfs_iunlock.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read
      4.33 ±  4%      -1.4        2.96 ±  3%  perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
      2.97 ±  4%      -1.0        2.01        perf-profile.calltrace.cycles-pp.__inode_security_revalidate.selinux_file_permission.security_file_permission.vfs_read.ksys_read
      2.46 ±  3%      -0.5        2.00 ±  3%  perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      1.56 ± 21%      -0.4        1.20 ±  9%  perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.08 ±  3%      -0.3        1.78 ±  6%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.14 ±  2%      -0.3        1.84 ±  6%  perf-profile.calltrace.cycles-pp.close
      2.09 ±  3%      -0.3        1.79 ±  6%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      2.11 ±  3%      -0.3        1.81 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close
      2.11 ±  2%      -0.3        1.81 ±  6%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      2.09 ±  3%      -0.3        1.79 ±  6%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      2.05 ±  3%      -0.3        1.75 ±  6%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
      2.06 ±  3%      -0.3        1.77 ±  6%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      1.39 ±  3%      -0.3        1.10 ±  5%  perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.97 ±  6%      -0.3        1.70 ±  5%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
      2.28 ±  6%      -0.2        2.07 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_open.do_syscall_64
      2.22 ±  6%      -0.2        2.03 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_open
      0.59 ±  2%      +0.1        0.69 ±  5%  perf-profile.calltrace.cycles-pp.memset_erms.iomap_read_page_sync.iomap_write_begin.iomap_write_actor.iomap_apply
      0.87 ±  2%      +0.1        0.97 ±  3%  perf-profile.calltrace.cycles-pp.iomap_read_page_sync.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.79 ±  5%      +0.1        0.89 ±  4%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.19            +0.1        1.32        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
      1.15 ±  5%      +0.1        1.28        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read
      1.56 ±  2%      +0.2        1.72 ±  6%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      1.55 ±  2%      +0.2        1.71 ±  6%  perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.__dentry_kill.dput.__fput
      1.40 ±  3%      +0.2        1.59        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read
      1.38 ±  2%      +0.2        1.61        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      3.72 ±  2%      +0.3        3.97        perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      3.86 ±  2%      +0.3        4.15        perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.72 ±  2%      +0.4        2.12        perf-profile.calltrace.cycles-pp.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.90 ±  5%      +0.4        3.31 ±  3%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.91 ±  5%      +0.4        3.31 ±  3%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      2.92 ±  5%      +0.4        3.33 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
      2.92 ±  5%      +0.4        3.32 ±  3%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      2.92 ±  5%      +0.4        3.33 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      2.94 ±  5%      +0.4        3.35 ±  2%  perf-profile.calltrace.cycles-pp.creat
      5.01 ±  2%      +0.5        5.47        perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      0.00            +0.6        0.58 ±  5%  perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create
      2.00 ±  2%      +0.6        2.59 ±  2%  perf-profile.calltrace.cycles-pp.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      0.00            +0.6        0.62 ±  3%  perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.62 ±  3%  perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64
      0.00            +0.6        0.63 ±  3%  perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      0.00            +0.6        0.63 ±  3%  perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      0.00            +0.6        0.65 ±  3%  perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      0.77 ±  4%      +0.7        1.45 ±  2%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      0.78 ±  5%      +0.7        1.46 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
      0.78 ±  5%      +0.7        1.46 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      0.00            +0.7        0.70 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.81 ±  5%      +0.7        1.51 ±  2%  perf-profile.calltrace.cycles-pp.unlink
      0.00            +0.7        0.74 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
     12.46 ±  2%      +0.8       13.22        perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     12.37 ±  2%      +0.8       13.13        perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      0.13 ±173%      +0.9        1.02        perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +1.0        1.00 ±  2%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_open
     15.73 ±  4%      +1.3       17.05        perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     22.72 ±  3%      +1.5       24.18        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     22.86 ±  3%      +1.5       24.35        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     16.41 ±  3%      +1.5       17.94        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.33 ±  3%      +1.7       19.99        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     18.75 ±  3%      +1.7       20.47        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     25.76 ±  3%      +1.9       27.68        perf-profile.calltrace.cycles-pp.write
      5.47 ± 15%     +10.7       16.19 ±  8%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      5.55 ± 15%     +11.2       16.79 ±  9%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      5.56 ± 15%     +11.3       16.82 ±  9%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.89 ± 12%     +11.5       17.42 ±  8%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      5.69 ± 14%     +11.6       17.25 ±  9%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.69 ± 14%     +11.6       17.26 ±  9%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      5.69 ± 14%     +11.6       17.26 ±  9%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     54.73           -14.7       40.00 ±  4%  perf-profile.children.cycles-pp.ksys_read
     54.28           -14.7       39.56 ±  4%  perf-profile.children.cycles-pp.vfs_read
     62.12           -14.6       47.49 ±  3%  perf-profile.children.cycles-pp.read
     87.70           -12.8       74.92 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     87.98           -12.8       75.22 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     45.31           -12.1       33.16 ±  2%  perf-profile.children.cycles-pp.new_sync_read
     38.88           -10.1       28.74 ±  2%  perf-profile.children.cycles-pp.xfs_file_read_iter
     36.47 ±  2%      -9.5       26.98        perf-profile.children.cycles-pp.xfs_file_buffered_aio_read
     20.86            -5.5       15.39 ±  3%  perf-profile.children.cycles-pp.generic_file_read_iter
      9.89 ±  4%      -3.2        6.70 ±  7%  perf-profile.children.cycles-pp.touch_atime
      9.80 ±  4%      -3.2        6.63 ±  8%  perf-profile.children.cycles-pp.atime_needs_update
      9.60 ±  6%      -2.6        7.02 ±  4%  perf-profile.children.cycles-pp.down_read
     10.44 ±  7%      -2.5        7.95 ±  5%  perf-profile.children.cycles-pp.xfs_ilock
      6.35 ±  9%      -1.7        4.67 ±  9%  perf-profile.children.cycles-pp.security_file_permission
      5.81 ±  5%      -1.5        4.34        perf-profile.children.cycles-pp.up_read
      6.41 ±  5%      -1.5        4.93        perf-profile.children.cycles-pp.xfs_iunlock
      4.94 ±  4%      -1.3        3.60 ±  3%  perf-profile.children.cycles-pp.selinux_file_permission
      3.17 ±  3%      -0.9        2.27        perf-profile.children.cycles-pp.__inode_security_revalidate
      2.47 ±  3%      -0.5        2.02 ±  3%  perf-profile.children.cycles-pp.iomap_write_end
      1.75 ± 18%      -0.3        1.41 ±  7%  perf-profile.children.cycles-pp.fsnotify
      2.14 ±  2%      -0.3        1.84 ±  6%  perf-profile.children.cycles-pp.close
      2.09 ±  3%      -0.3        1.79 ±  6%  perf-profile.children.cycles-pp.task_work_run
      2.08 ±  3%      -0.3        1.79 ±  6%  perf-profile.children.cycles-pp.__fput
      2.06 ±  3%      -0.3        1.77 ±  6%  perf-profile.children.cycles-pp.dput
      2.10 ±  3%      -0.3        1.81 ±  6%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      2.05 ±  3%      -0.3        1.76 ±  6%  perf-profile.children.cycles-pp.__dentry_kill
      1.40 ±  3%      -0.3        1.12 ±  5%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      0.74 ±  7%      -0.2        0.53 ±  8%  perf-profile.children.cycles-pp.rw_verify_area
      0.34 ±  3%      -0.2        0.17 ±  7%  perf-profile.children.cycles-pp.page_mapping
      0.43 ±  9%      -0.1        0.36 ±  3%  perf-profile.children.cycles-pp.up_write
      0.22 ±  7%      -0.0        0.20 ±  8%  perf-profile.children.cycles-pp.unlock_page
      0.07            -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.xfs_btree_lookup
      0.10 ±  4%      +0.0        0.12 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.09 ±  4%      +0.0        0.11        perf-profile.children.cycles-pp.workingset_update_node
      0.07 ± 10%      +0.0        0.09 ±  9%  perf-profile.children.cycles-pp.xfs_lookup
      0.07 ± 14%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.xfs_vn_lookup
      0.18 ±  7%      +0.0        0.21 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.13 ±  5%      +0.0        0.16        perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.16            +0.0        0.19 ±  7%  perf-profile.children.cycles-pp.__x64_sys_read
      0.07 ±  7%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.66 ±  2%      +0.0        0.69        perf-profile.children.cycles-pp.down_write
      0.17 ±  9%      +0.0        0.20 ±  6%  perf-profile.children.cycles-pp.disk_cp
      0.38 ±  2%      +0.0        0.41 ±  4%  perf-profile.children.cycles-pp.xas_store
      0.21 ±  3%      +0.0        0.25 ±  9%  perf-profile.children.cycles-pp.rcu_all_qs
      0.05 ±  8%      +0.0        0.09 ±  9%  perf-profile.children.cycles-pp.unwind_next_frame
      0.04 ± 57%      +0.0        0.08 ± 12%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.21 ±  3%      +0.0        0.25 ±  3%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.18 ±  4%      +0.0        0.22 ±  2%  perf-profile.children.cycles-pp.xfs_dir3_data_check
      0.18 ±  4%      +0.0        0.22 ±  2%  perf-profile.children.cycles-pp.__xfs_dir3_data_check
      0.35 ±  3%      +0.0        0.39 ±  3%  perf-profile.children.cycles-pp.delete_from_page_cache_batch
      0.44 ±  2%      +0.0        0.48 ±  5%  perf-profile.children.cycles-pp._cond_resched
      0.03 ±100%      +0.0        0.07 ±  6%  perf-profile.children.cycles-pp.kmem_zone_alloc
      0.05 ± 58%      +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.xfs_get_extsz_hint
      0.04 ± 57%      +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.read_tsc
      0.21 ±  6%      +0.1        0.27 ±  8%  perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.08 ± 10%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.11 ±  9%      +0.1        0.17 ± 20%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.xlog_space_left
      0.10 ±  9%      +0.1        0.15 ± 11%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.08 ±  5%      +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.arch_stack_walk
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.xfs_verify_ino
      0.27 ±  5%      +0.1        0.33        perf-profile.children.cycles-pp.xfs_dir2_node_removename
      0.11 ±  4%      +0.1        0.17 ± 19%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.06 ± 14%      +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.11 ±  7%      +0.1        0.17 ± 20%  perf-profile.children.cycles-pp.uart_console_write
      0.45 ±  4%      +0.1        0.52 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      0.28 ±  5%      +0.1        0.34 ±  2%  perf-profile.children.cycles-pp.xfs_dir_removename
      0.08 ± 58%      +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.xlog_ungrant_log_space
      0.01 ±173%      +0.1        0.08        perf-profile.children.cycles-pp.xfs_iunlink
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.12 ± 10%      +0.1        0.19 ±  8%  perf-profile.children.cycles-pp.enqueue_entity
      0.30 ±  4%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.12 ± 10%      +0.1        0.20 ± 10%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.10 ± 58%      +0.1        0.18 ±  9%  perf-profile.children.cycles-pp.xfs_log_done
      0.00            +0.1        0.08 ± 22%  perf-profile.children.cycles-pp.native_write_msr
      0.00            +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.xfs_agino_range
      0.12 ± 10%      +0.1        0.20 ± 12%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.12 ± 10%      +0.1        0.20 ± 12%  perf-profile.children.cycles-pp.activate_task
      0.12 ±  8%      +0.1        0.21 ± 20%  perf-profile.children.cycles-pp.console_unlock
      0.08 ±  8%      +0.1        0.17 ± 10%  perf-profile.children.cycles-pp.update_load_avg
      0.12 ±  8%      +0.1        0.21 ± 17%  perf-profile.children.cycles-pp.irq_work_run_list
      0.35 ±  8%      +0.1        0.44 ±  5%  perf-profile.children.cycles-pp.xfs_file_iomap_end
      0.00            +0.1        0.09 ± 50%  perf-profile.children.cycles-pp.xfs_inobt_get_maxrecs
      0.11 ± 11%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.11 ± 11%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.11 ± 11%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.irq_work_run
      0.11 ± 11%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.printk
      0.11 ± 11%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.vprintk_emit
      0.04 ± 57%      +0.1        0.13 ± 25%  perf-profile.children.cycles-pp.clockevents_program_event
      0.60            +0.1        0.69 ±  4%  perf-profile.children.cycles-pp.memset_erms
      0.88 ±  2%      +0.1        0.98 ±  3%  perf-profile.children.cycles-pp.iomap_read_page_sync
      0.14 ±  9%      +0.1        0.26 ±  6%  perf-profile.children.cycles-pp.task_tick_fair
      0.71 ±  2%      +0.1        0.82 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      0.00            +0.1        0.12 ± 26%  perf-profile.children.cycles-pp.ktime_get
      0.11 ± 61%      +0.1        0.24 ± 12%  perf-profile.children.cycles-pp.xlog_grant_add_space
      0.42 ±  8%      +0.1        0.55 ± 13%  perf-profile.children.cycles-pp.xfs_file_write_iter
      0.06 ± 11%      +0.1        0.20 ± 47%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.10 ± 15%      +0.1        0.24 ± 24%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.06 ±  6%      +0.2        0.22 ± 35%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      0.06 ±  7%      +0.2        0.22 ± 41%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      1.56 ±  2%      +0.2        1.72 ±  6%  perf-profile.children.cycles-pp.evict
      0.17 ±  8%      +0.2        0.34 ±  6%  perf-profile.children.cycles-pp.scheduler_tick
      1.55 ±  2%      +0.2        1.71 ±  6%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      0.11 ± 14%      +0.2        0.29 ± 25%  perf-profile.children.cycles-pp.irq_exit
      0.00            +0.2        0.18 ± 27%  perf-profile.children.cycles-pp.menu_select
      0.43 ±  3%      +0.2        0.62 ±  3%  perf-profile.children.cycles-pp.xfs_vn_unlink
      0.43 ±  3%      +0.2        0.62 ±  3%  perf-profile.children.cycles-pp.xfs_remove
      0.23 ± 41%      +0.2        0.42 ± 11%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.21 ± 46%      +0.2        0.40 ± 11%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.45 ±  2%      +0.2        0.65 ±  3%  perf-profile.children.cycles-pp.vfs_unlink
      0.34 ±  5%      +0.2        0.55 ±  4%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.08 ±  5%      +0.2        0.29 ± 42%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      0.28 ± 36%      +0.2        0.50 ±  9%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.26 ±  4%      +0.3        0.51        perf-profile.children.cycles-pp.update_process_times
      0.26 ±  3%      +0.3        0.52 ±  2%  perf-profile.children.cycles-pp.tick_sched_handle
      0.10 ±  8%      +0.3        0.36 ± 36%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      2.49 ±  6%      +0.3        2.75 ±  3%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      3.88 ±  2%      +0.3        4.17        perf-profile.children.cycles-pp.grab_cache_page_write_begin
      0.29 ±  3%      +0.3        0.59 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
      2.61 ±  2%      +0.3        2.94        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.12 ±  8%      +0.4        0.52 ±  4%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      3.00 ±  5%      +0.4        3.40 ±  2%  perf-profile.children.cycles-pp.do_sys_open
      2.97 ±  5%      +0.4        3.37 ±  2%  perf-profile.children.cycles-pp.path_openat
      2.97 ±  5%      +0.4        3.37 ±  3%  perf-profile.children.cycles-pp.do_filp_open
      0.20 ±  8%      +0.4        0.61 ± 37%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      1.75 ±  2%      +0.4        2.15        perf-profile.children.cycles-pp.xfs_file_iomap_begin_delay
      2.94 ±  5%      +0.4        3.35 ±  2%  perf-profile.children.cycles-pp.creat
      0.15 ± 10%      +0.4        0.58 ±  5%  perf-profile.children.cycles-pp.xfs_dialloc
      0.39 ±  3%      +0.4        0.83 ±  2%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      2.79 ±  2%      +0.4        3.23        perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.18 ±  7%      +0.4        0.63 ±  3%  perf-profile.children.cycles-pp.xfs_ialloc
      0.18 ±  7%      +0.5        0.63 ±  3%  perf-profile.children.cycles-pp.xfs_dir_ialloc
      5.03 ±  2%      +0.5        5.49        perf-profile.children.cycles-pp.iomap_write_begin
      0.45 ±  7%      +0.6        1.00        perf-profile.children.cycles-pp.xfs_create
      0.46 ±  7%      +0.6        1.02        perf-profile.children.cycles-pp.xfs_generic_create
      0.50 ±  4%      +0.6        1.08 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      2.02 ±  2%      +0.6        2.62        perf-profile.children.cycles-pp.xfs_file_iomap_begin
      0.77 ±  4%      +0.7        1.45 ±  2%  perf-profile.children.cycles-pp.do_unlinkat
      0.81 ±  5%      +0.7        1.51 ±  2%  perf-profile.children.cycles-pp.unlink
     12.39 ±  2%      +0.8       13.15        perf-profile.children.cycles-pp.iomap_apply
     12.46 ±  2%      +0.8       13.22        perf-profile.children.cycles-pp.iomap_file_buffered_write
      0.64 ±  5%      +0.8        1.46 ±  8%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.69 ±  4%      +0.9        1.56 ±  8%  perf-profile.children.cycles-pp.apic_timer_interrupt
     15.75 ±  4%      +1.3       17.07        perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     16.43 ±  3%      +1.5       17.95        perf-profile.children.cycles-pp.new_sync_write
     18.36 ±  3%      +1.7       20.02        perf-profile.children.cycles-pp.vfs_write
     18.77 ±  3%      +1.7       20.48        perf-profile.children.cycles-pp.ksys_write
     25.94 ±  3%      +2.0       27.92        perf-profile.children.cycles-pp.write
      5.65 ± 12%     +10.7       16.35 ±  7%  perf-profile.children.cycles-pp.intel_idle
      5.75 ± 13%     +11.2       16.98 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter_state
      5.75 ± 13%     +11.2       16.98 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter
      5.89 ± 12%     +11.5       17.42 ±  8%  perf-profile.children.cycles-pp.secondary_startup_64
      5.89 ± 12%     +11.5       17.42 ±  8%  perf-profile.children.cycles-pp.cpu_startup_entry
      5.89 ± 12%     +11.5       17.42 ±  8%  perf-profile.children.cycles-pp.do_idle
      5.69 ± 14%     +11.6       17.26 ±  9%  perf-profile.children.cycles-pp.start_secondary
      9.36 ±  3%      -3.2        6.20 ±  7%  perf-profile.self.cycles-pp.atime_needs_update
      9.38 ±  6%      -2.6        6.74 ±  5%  perf-profile.self.cycles-pp.down_read
      7.04 ±  2%      -2.2        4.82 ±  3%  perf-profile.self.cycles-pp.generic_file_read_iter
      6.34            -2.0        4.32 ±  6%  perf-profile.self.cycles-pp.new_sync_read
      5.76 ±  5%      -1.5        4.27        perf-profile.self.cycles-pp.up_read
      2.93 ±  4%      -1.0        1.96        perf-profile.self.cycles-pp.__inode_security_revalidate
      7.79            -0.6        7.14        perf-profile.self.cycles-pp.do_syscall_64
      1.73 ±  6%      -0.5        1.27 ±  6%  perf-profile.self.cycles-pp.selinux_file_permission
      1.73 ± 18%      -0.4        1.36 ±  8%  perf-profile.self.cycles-pp.fsnotify
      0.72 ±  7%      -0.2        0.51 ±  9%  perf-profile.self.cycles-pp.rw_verify_area
      0.34 ±  3%      -0.2        0.17 ± 10%  perf-profile.self.cycles-pp.page_mapping
      0.42 ±  4%      -0.1        0.30 ±  6%  perf-profile.self.cycles-pp.iomap_write_end
      0.24 ±  6%      -0.1        0.16 ± 10%  perf-profile.self.cycles-pp.iomap_set_page_dirty
      0.16 ± 11%      -0.1        0.08 ± 20%  perf-profile.self.cycles-pp.rwsem_optimistic_spin
      0.42 ±  8%      -0.1        0.34 ±  5%  perf-profile.self.cycles-pp.up_write
      0.33 ±  2%      -0.0        0.29 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      0.28 ±  5%      -0.0        0.24 ±  5%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.08            +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.workingset_update_node
      0.05 ±  8%      +0.0        0.07 ± 17%  perf-profile.self.cycles-pp.__lock_text_start
      0.14 ±  3%      +0.0        0.17 ±  2%  perf-profile.self.cycles-pp.__x64_sys_read
      0.22            +0.0        0.25 ±  6%  perf-profile.self.cycles-pp.read
      0.16            +0.0        0.19 ±  9%  perf-profile.self.cycles-pp.rcu_all_qs
      0.18 ±  2%      +0.0        0.21 ±  7%  perf-profile.self.cycles-pp._cond_resched
      0.11 ±  4%      +0.0        0.15 ±  9%  perf-profile.self.cycles-pp.xfs_file_aio_write_checks
      0.32 ±  5%      +0.0        0.35        perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      0.14 ±  6%      +0.0        0.17 ± 10%  perf-profile.self.cycles-pp.ksys_write
      0.04 ± 57%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.copyin
      0.41 ±  4%      +0.0        0.45 ±  3%  perf-profile.self.cycles-pp.__might_sleep
      0.20 ±  4%      +0.0        0.25 ±  3%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.20 ±  7%      +0.0        0.25 ±  6%  perf-profile.self.cycles-pp.iomap_write_begin
      0.03 ±100%      +0.1        0.08 ± 16%  perf-profile.self.cycles-pp.xfs_get_extsz_hint
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.read_tsc
      0.27 ±  6%      +0.1        0.32 ±  6%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.17 ± 13%      +0.1        0.23 ± 10%  perf-profile.self.cycles-pp.new_sync_write
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.memcpy_erms
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.xlog_space_left
      0.20 ±  4%      +0.1        0.26        perf-profile.self.cycles-pp.write
      0.10 ± 25%      +0.1        0.16 ± 18%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.08 ± 58%      +0.1        0.14 ± 10%  perf-profile.self.cycles-pp.xlog_ungrant_log_space
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.update_load_avg
      0.25 ±  4%      +0.1        0.32 ±  5%  perf-profile.self.cycles-pp.xfs_file_iomap_end
      0.23 ±  9%      +0.1        0.30 ±  8%  perf-profile.self.cycles-pp.pagecache_get_page
      0.00            +0.1        0.08 ± 30%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.1        0.08 ± 22%  perf-profile.self.cycles-pp.native_write_msr
      0.00            +0.1        0.08 ± 53%  perf-profile.self.cycles-pp.xfs_inobt_get_maxrecs
      0.59            +0.1        0.68 ±  4%  perf-profile.self.cycles-pp.memset_erms
      0.17 ±  9%      +0.1        0.26 ±  7%  perf-profile.self.cycles-pp.xfs_iunlock
      0.00            +0.1        0.09 ± 44%  perf-profile.self.cycles-pp.menu_select
      0.68 ±  2%      +0.1        0.79 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.40 ± 10%      +0.1        0.52 ± 13%  perf-profile.self.cycles-pp.xfs_file_write_iter
      0.00            +0.1        0.12 ± 38%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      0.11 ± 61%      +0.1        0.24 ± 12%  perf-profile.self.cycles-pp.xlog_grant_add_space
      0.24 ±  7%      +0.2        0.40        perf-profile.self.cycles-pp.xfs_file_iomap_begin
      0.31 ±  6%      +0.2        0.53 ±  4%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.32 ± 12%      +0.2        0.57        perf-profile.self.cycles-pp.xfs_file_iomap_begin_delay
      2.60 ±  2%      +0.3        2.94        perf-profile.self.cycles-pp.syscall_return_via_sysret
      2.52 ±  2%      +0.4        2.91        perf-profile.self.cycles-pp.entry_SYSCALL_64
      5.65 ± 12%     +10.7       16.34 ±  7%  perf-profile.self.cycles-pp.intel_idle





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.3.0-rc7-00093-g1aabb011bfba5" of type "text/plain" (199463 bytes)

View attachment "job-script" of type "text/plain" (8124 bytes)

View attachment "job.yaml" of type "text/plain" (5742 bytes)

View attachment "reproduce" of type "text/plain" (1044 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ