lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Mon, 22 Mar 2021 13:15:25 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     "Darrick J. Wong" <djwong@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com
Subject: [xfs]  301157ab53:  aim7.jobs-per-min -65.3% regression



Greeting,

FYI, we noticed a -65.3% regression of aim7.jobs-per-min due to commit:


commit: 301157ab53d2e20653f6ea744826a8ff7408c662 ("xfs: expose sysfs knob to control inode inactivation delay")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git deferred-inactivation-5.13


in testcase: aim7
on test machine: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID1
	fs: xfs
	test: disk_rw
	load: 3000
	cpufreq_governor: performance
	ucode: 0x5003006

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/4BRD_12G/xfs/x86_64-rhel-8.3/3000/RAID1/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/disk_rw/aim7/0x5003006

commit: 
  e8adf8f196 ("xfs: deferred inode inactivation")
  301157ab53 ("xfs: expose sysfs knob to control inode inactivation delay")

e8adf8f1967c217f 301157ab53d2e20653f6ea74482 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    504869           -65.3%     175186        aim7.jobs-per-min
     35.86          +187.4%     103.07        aim7.time.elapsed_time
     35.86          +187.4%     103.07        aim7.time.elapsed_time.max
 6.171e+08           +11.0%  6.849e+08        aim7.time.file_system_outputs
      2701         +3735.9%     103627 ±  5%  aim7.time.involuntary_context_switches
    169081 ±  2%      +6.5%     180145        aim7.time.minor_page_faults
    721.06          +476.9%       4159        aim7.time.system_time
     48.48           +27.4%      61.75        aim7.time.user_time
    775012          +825.0%    7168720        aim7.time.voluntary_context_switches
 1.556e+09 ± 39%    +147.3%  3.847e+09 ± 19%  cpuidle.C1E.time
     66.82          +100.0%     133.62        uptime.boot
      4642           +68.1%       7804 ±  2%  uptime.idle
     75.34           -23.1%      57.91        iostat.cpu.idle
      0.29 ±140%     -82.8%       0.05 ± 15%  iostat.cpu.iowait
     22.72           +81.9%      41.32 ±  2%  iostat.cpu.system
      1.66           -56.5%       0.72        iostat.cpu.user
     74.83           -23.4%      57.33 ±  2%  vmstat.cpu.id
      5194 ± 77%   +6308.3%     332877 ±  2%  vmstat.io.bo
   2011374           +22.0%    2454307        vmstat.memory.cache
     20.00 ±  4%    +243.3%      68.67 ±  6%  vmstat.procs.r
     65964           +43.4%      94595        vmstat.system.cs
     78.50           -17.1       61.42 ±  4%  mpstat.cpu.all.idle%
      0.24 ±145%      -0.2        0.04 ± 19%  mpstat.cpu.all.iowait%
      0.90 ±  7%      +0.1        1.04 ±  5%  mpstat.cpu.all.irq%
      0.14 ±  2%      +0.3        0.39 ±  2%  mpstat.cpu.all.soft%
     18.72           +17.7       36.39 ±  6%  mpstat.cpu.all.sys%
      1.50            -0.8        0.72        mpstat.cpu.all.usr%
     15441 ±  5%    +143.4%      37587 ±  6%  meminfo.Active
     15280 ±  5%    +144.9%      37427 ±  6%  meminfo.Active(anon)
     43977 ± 12%    +130.7%     101469 ±  2%  meminfo.AnonHugePages
   1910765           +19.9%    2291562        meminfo.Cached
    840958           -56.3%     367540 ±  6%  meminfo.Dirty
   1195011           +30.5%    1559420        meminfo.Inactive
    841771           +41.5%    1191405        meminfo.Inactive(file)
    120069           +40.3%     168476        meminfo.KReclaimable
   4409473 ±  7%    +239.9%   14986991        meminfo.Memused
    120069           +40.3%     168476        meminfo.SReclaimable
    282361           +17.9%     332992        meminfo.SUnreclaim
     27083 ±  4%    +116.6%      58663 ± 24%  meminfo.Shmem
    402431           +24.6%     501468        meminfo.Slab
    433.17 ±213%  +70447.7%     305589 ±  8%  meminfo.Writeback
   4526282 ±  7%    +236.6%   15237059        meminfo.max_used_kB
    419896           -55.9%     185046 ±  7%  numa-meminfo.node0.Dirty
    957879           +18.8%    1138079        numa-meminfo.node0.FilePages
    420302           +43.4%     602696        numa-meminfo.node0.Inactive(file)
  63446929            -9.8%   57254618        numa-meminfo.node0.MemFree
   2212777 ± 17%    +279.8%    8405088 ± 12%  numa-meminfo.node0.MemUsed
    205.67 ±211%  +75786.7%     156073 ±  9%  numa-meminfo.node0.Writeback
     12230 ± 12%    +173.4%      33433 ±  7%  numa-meminfo.node1.Active
     12192 ± 12%    +173.8%      33380 ±  7%  numa-meminfo.node1.Active(anon)
     13769 ± 96%    +322.1%      58119 ± 59%  numa-meminfo.node1.AnonHugePages
    420827           -57.0%     181115 ±  6%  numa-meminfo.node1.Dirty
    952703 ±  2%     +20.9%    1152149 ±  2%  numa-meminfo.node1.FilePages
    558993 ± 18%     +45.1%     810959 ±  8%  numa-meminfo.node1.Inactive
    421167           +39.4%     587047        numa-meminfo.node1.Inactive(file)
     55834 ± 20%     +57.1%      87731 ± 14%  numa-meminfo.node1.KReclaimable
   2197302 ± 28%    +199.6%    6582498 ± 15%  numa-meminfo.node1.MemUsed
     55834 ± 20%     +57.1%      87731 ± 14%  numa-meminfo.node1.SReclaimable
    128741 ± 36%     +66.2%     213929 ±  4%  numa-meminfo.node1.SUnreclaim
     15305 ± 10%    +186.1%      43786 ± 22%  numa-meminfo.node1.Shmem
    184576 ± 29%     +63.4%     301661        numa-meminfo.node1.Slab
    174.00 ±213%  +86435.8%     150572 ±  9%  numa-meminfo.node1.Writeback
  13602647 ±  3%     +42.2%   19339099        numa-vmstat.node0.nr_dirtied
    104485           -55.1%      46940 ±  3%  numa-vmstat.node0.nr_dirty
    238973           +19.1%     284599        numa-vmstat.node0.nr_file_pages
  15862224            -9.8%   14314037        numa-vmstat.node0.nr_free_pages
    104593           +44.1%     150724        numa-vmstat.node0.nr_inactive_file
     32.83 ±207%  +1.2e+05%      38758 ±  4%  numa-vmstat.node0.nr_writeback
     13676 ± 84%  +15994.3%    2201164        numa-vmstat.node0.nr_written
    104598           +44.1%     150718        numa-vmstat.node0.nr_zone_inactive_file
    104518           -18.1%      85641        numa-vmstat.node0.nr_zone_write_pending
  14490783 ±  3%     +35.4%   19617012 ±  2%  numa-vmstat.node0.numa_hit
  14433154 ±  3%     +35.0%   19481947 ±  2%  numa-vmstat.node0.numa_local
      3048 ± 12%    +173.6%       8341 ±  8%  numa-vmstat.node1.nr_active_anon
  14024725           +36.1%   19085657        numa-vmstat.node1.nr_dirtied
    105037           -56.4%      45841 ±  2%  numa-vmstat.node1.nr_dirty
    238060 ±  2%     +21.1%     288315 ±  2%  numa-vmstat.node1.nr_file_pages
    105112           +39.8%     146920        numa-vmstat.node1.nr_inactive_file
      3869 ±  9%    +185.5%      11045 ± 22%  numa-vmstat.node1.nr_shmem
     13959 ± 20%     +57.1%      21924 ± 14%  numa-vmstat.node1.nr_slab_reclaimable
     32176 ± 36%     +66.1%      53450 ±  4%  numa-vmstat.node1.nr_slab_unreclaimable
     32.83 ±211%  +1.1e+05%      37442 ±  4%  numa-vmstat.node1.nr_writeback
     12895 ± 79%  +16786.8%    2177582        numa-vmstat.node1.nr_written
      3048 ± 12%    +173.6%       8341 ±  8%  numa-vmstat.node1.nr_zone_active_anon
    105108           +39.8%     146919        numa-vmstat.node1.nr_zone_inactive_file
    105024           -20.7%      83304        numa-vmstat.node1.nr_zone_write_pending
  14841050 ±  2%     +28.3%   19037505        numa-vmstat.node1.numa_hit
  14669358 ±  2%     +29.1%   18941199        numa-vmstat.node1.numa_local
      3820 ±  5%    +146.0%       9397 ±  6%  proc-vmstat.nr_active_anon
     85450            +1.4%      86634        proc-vmstat.nr_anon_pages
  77132275           +11.0%   85615484        proc-vmstat.nr_dirtied
    210299           -56.4%      91747 ±  6%  proc-vmstat.nr_dirty
   3182511            -8.0%    2927108        proc-vmstat.nr_dirty_background_threshold
   6372805            -8.0%    5861373        proc-vmstat.nr_dirty_threshold
    477777           +19.8%     572528        proc-vmstat.nr_file_pages
  31815898            -8.3%   29171250        proc-vmstat.nr_free_pages
     88332            +4.1%      91964 ±  2%  proc-vmstat.nr_inactive_anon
    210491           +41.3%     297372        proc-vmstat.nr_inactive_file
      6785 ±  4%    +118.2%      14805 ± 25%  proc-vmstat.nr_shmem
     30015           +40.3%      42113        proc-vmstat.nr_slab_reclaimable
     70578           +17.8%      83165        proc-vmstat.nr_slab_unreclaimable
     91.33 ±212%  +83861.3%      76684 ±  8%  proc-vmstat.nr_writeback
     48019 ± 82%  +17971.3%    8677789        proc-vmstat.nr_written
      3820 ±  5%    +146.0%       9397 ±  6%  proc-vmstat.nr_zone_active_anon
     88332            +4.1%      91964 ±  2%  proc-vmstat.nr_zone_inactive_anon
    210492           +41.3%     297372        proc-vmstat.nr_zone_inactive_file
    210361           -19.9%     168453        proc-vmstat.nr_zone_write_pending
      3618 ± 66%    +691.8%      28653 ± 80%  proc-vmstat.numa_hint_faults
  76367720            +6.0%   80950711        proc-vmstat.numa_hit
  76288420            +6.0%   80870851        proc-vmstat.numa_local
      7057 ± 85%    +279.1%      26751 ± 10%  proc-vmstat.numa_pages_migrated
     24568 ± 50%    +202.9%      74423 ± 16%  proc-vmstat.numa_pte_updates
  77801452            +4.4%   81209870        proc-vmstat.pgalloc_normal
    327037 ±  2%     +81.1%     592360 ±  9%  proc-vmstat.pgfault
      7057 ± 85%    +279.1%      26751 ± 10%  proc-vmstat.pgmigrate_success
    202600 ± 77%  +17308.7%   35270099        proc-vmstat.pgpgout
     10311 ±  9%    +117.7%      22452        proc-vmstat.pgreuse
      1574            +1.3%       1594        proc-vmstat.unevictable_pgs_culled
      3312 ± 11%    +361.0%      15271 ± 32%  sched_debug.cfs_rq:/.load.avg
     36897 ± 29%   +1465.3%     577543 ± 68%  sched_debug.cfs_rq:/.load.max
      8219 ± 12%    +687.4%      64721 ± 62%  sched_debug.cfs_rq:/.load.stddev
    958.21 ± 15%     -46.0%     517.56 ±  3%  sched_debug.cfs_rq:/.load_avg.avg
     14713 ± 67%     -62.4%       5531 ±  6%  sched_debug.cfs_rq:/.load_avg.max
      2890 ± 28%     -50.5%       1429 ±  6%  sched_debug.cfs_rq:/.load_avg.stddev
     16688 ± 25%   +5337.2%     907374 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
     31398 ± 12%   +3050.3%     989168 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
     11467 ± 30%   +7259.4%     843958 ±  3%  sched_debug.cfs_rq:/.min_vruntime.min
      3312 ±  7%    +514.0%      20338 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
     28.02 ± 47%     -54.7%      12.69 ±  3%  sched_debug.cfs_rq:/.removed.load_avg.avg
      1001 ±  2%     -50.6%     494.67        sched_debug.cfs_rq:/.removed.load_avg.max
    159.28 ± 23%     -51.1%      77.86 ±  2%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    455.50 ± 20%     -54.4%     207.67 ± 18%  sched_debug.cfs_rq:/.removed.runnable_avg.max
    455.00 ± 20%     -54.4%     207.67 ± 18%  sched_debug.cfs_rq:/.removed.util_avg.max
     -8481          -810.6%      60273 ± 13%  sched_debug.cfs_rq:/.spread0.avg
      6237 ± 50%   +2196.0%     143217 ±  2%  sched_debug.cfs_rq:/.spread0.max
    -13668           -85.3%      -2009        sched_debug.cfs_rq:/.spread0.min
      3324 ±  7%    +516.5%      20497 ±  6%  sched_debug.cfs_rq:/.spread0.stddev
    328.99 ±  7%     -28.4%     235.70 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
     12645 ±190%   +2006.9%     266427 ± 31%  sched_debug.cpu.avg_idle.min
    308348 ±  3%     -34.9%     200842 ± 10%  sched_debug.cpu.avg_idle.stddev
     29873 ±  2%     +99.2%      59497        sched_debug.cpu.clock.avg
     29878 ±  2%     +99.2%      59518        sched_debug.cpu.clock.max
     29869 ±  2%     +99.1%      59473        sched_debug.cpu.clock.min
      2.52 ± 24%    +415.7%      13.00 ± 27%  sched_debug.cpu.clock.stddev
     29734 ±  2%     +98.4%      58989        sched_debug.cpu.clock_task.avg
     29862 ±  2%     +98.1%      59162        sched_debug.cpu.clock_task.max
     24927 ±  2%    +118.6%      54486        sched_debug.cpu.clock_task.min
    643.52 ±  3%     -10.9%     573.19 ±  6%  sched_debug.cpu.clock_task.stddev
    325.02 ± 21%    +227.5%       1064 ± 74%  sched_debug.cpu.curr->pid.avg
      2615           +80.0%       4708 ±  9%  sched_debug.cpu.curr->pid.max
    824.37 ±  8%     +40.3%       1156 ± 11%  sched_debug.cpu.curr->pid.stddev
    838487 ±  4%     -17.7%     690159 ±  3%  sched_debug.cpu.max_idle_balance_cost.max
     57354 ± 24%     -46.9%      30472 ± 20%  sched_debug.cpu.max_idle_balance_cost.stddev
      1.00          +383.3%       4.83 ± 97%  sched_debug.cpu.nr_running.max
      0.34 ±  8%    +145.2%       0.84 ± 68%  sched_debug.cpu.nr_running.stddev
      1606          +962.4%      17071 ±  2%  sched_debug.cpu.nr_switches.avg
      7889 ± 14%    +302.4%      31747 ±  4%  sched_debug.cpu.nr_switches.max
    425.33 ±  7%   +3063.8%      13456        sched_debug.cpu.nr_switches.min
      1202 ±  9%    +115.5%       2591        sched_debug.cpu.nr_switches.stddev
      0.01 ± 31%  +1.2e+05%      16.54 ±  3%  sched_debug.cpu.nr_uninterruptible.avg
     35.33 ± 18%    +498.1%     211.33 ± 24%  sched_debug.cpu.nr_uninterruptible.max
    -23.83          +471.3%    -136.17        sched_debug.cpu.nr_uninterruptible.min
      7.83 ± 14%    +516.4%      48.26 ±  4%  sched_debug.cpu.nr_uninterruptible.stddev
     29871 ±  2%     +99.1%      59474        sched_debug.cpu_clk
     29375 ±  2%    +100.8%      58980        sched_debug.ktime
     30228 ±  2%     +97.9%      59830        sched_debug.sched_clk
 1.318e+10           -27.4%  9.569e+09        perf-stat.i.branch-instructions
      1.53 ± 32%      -0.6        0.94 ±  3%  perf-stat.i.branch-miss-rate%
  94149427 ±  2%     -39.8%   56666991        perf-stat.i.branch-misses
  40013692 ±  2%    +193.1%  1.173e+08 ±  3%  perf-stat.i.cache-misses
 1.319e+08 ±  2%    +238.1%  4.458e+08 ±  4%  perf-stat.i.cache-references
     68158           +42.0%      96781        perf-stat.i.context-switches
      1.52 ±  9%     +74.0%       2.65        perf-stat.i.cpi
 6.189e+10           +86.6%  1.155e+11        perf-stat.i.cpu-cycles
      2485          +830.1%      23115 ±  4%  perf-stat.i.cpu-migrations
   1948007 ± 10%     -22.1%    1517784 ± 19%  perf-stat.i.dTLB-load-misses
 1.903e+10           -32.8%  1.279e+10        perf-stat.i.dTLB-loads
    149667 ± 15%     -29.6%     105332 ±  9%  perf-stat.i.dTLB-store-misses
 1.065e+10           -50.7%  5.252e+09 ±  2%  perf-stat.i.dTLB-stores
     78.42 ±  3%      -8.6       69.79        perf-stat.i.iTLB-load-miss-rate%
  68390878 ±  3%     -62.7%   25498455        perf-stat.i.iTLB-load-misses
   5910806 ±  2%     -24.7%    4449919        perf-stat.i.iTLB-loads
 6.629e+10           -29.1%    4.7e+10        perf-stat.i.instructions
      1372 ±  5%     +88.1%       2582        perf-stat.i.instructions-per-iTLB-miss
      0.92           -54.0%       0.42        perf-stat.i.ipc
      0.70           +87.0%       1.31        perf-stat.i.metric.GHz
      0.50 ±  6%     +91.1%       0.95 ±  6%  perf-stat.i.metric.K/sec
    488.76           -34.5%     320.01        perf-stat.i.metric.M/sec
      7500 ±  2%     -28.4%       5371 ±  9%  perf-stat.i.minor-faults
     65.15           +14.3       79.41        perf-stat.i.node-load-miss-rate%
   5923043 ±  2%    +207.7%   18226987 ±  4%  perf-stat.i.node-load-misses
   4157238           -24.8%    3125348 ±  2%  perf-stat.i.node-loads
     37.97 ±  6%     +33.3       71.28        perf-stat.i.node-store-miss-rate%
   3141141          +601.8%   22044155 ±  3%  perf-stat.i.node-store-misses
   9217355           -50.1%    4602041 ±  9%  perf-stat.i.node-stores
      7551 ±  2%     -28.6%       5392 ±  9%  perf-stat.i.page-faults
      1.99 ±  2%    +372.3%       9.39 ±  3%  perf-stat.overall.MPKI
      0.71            -0.1        0.59        perf-stat.overall.branch-miss-rate%
     30.39 ±  3%      -4.1       26.31        perf-stat.overall.cache-miss-rate%
      0.93          +163.0%       2.45        perf-stat.overall.cpi
      1546 ±  2%     -35.7%     994.40 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.00 ± 15%      +0.0        0.00 ±  7%  perf-stat.overall.dTLB-store-miss-rate%
     92.05            -6.8       85.27        perf-stat.overall.iTLB-load-miss-rate%
    970.29 ±  3%     +89.8%       1841        perf-stat.overall.instructions-per-iTLB-miss
      1.07           -62.0%       0.41        perf-stat.overall.ipc
     58.75           +26.6       85.37        perf-stat.overall.node-load-miss-rate%
     25.41           +57.3       82.72        perf-stat.overall.node-store-miss-rate%
  1.29e+10           -25.6%  9.589e+09        perf-stat.ps.branch-instructions
  92010202 ±  2%     -38.6%   56518243        perf-stat.ps.branch-misses
  39165790 ±  2%    +197.1%  1.164e+08 ±  3%  perf-stat.ps.cache-misses
  1.29e+08 ±  2%    +243.0%  4.424e+08 ±  4%  perf-stat.ps.cache-references
     66725           +43.7%      95872        perf-stat.ps.context-switches
     85562            +1.9%      87147        perf-stat.ps.cpu-clock
 6.054e+10           +91.0%  1.156e+11        perf-stat.ps.cpu-cycles
      2430          +844.5%      22955 ±  4%  perf-stat.ps.cpu-migrations
   1904387 ± 10%     -20.5%    1513883 ± 19%  perf-stat.ps.dTLB-load-misses
 1.862e+10           -31.2%  1.282e+10        perf-stat.ps.dTLB-loads
    145945 ± 15%     -28.1%     104924 ±  9%  perf-stat.ps.dTLB-store-misses
 1.042e+10           -49.6%  5.256e+09 ±  2%  perf-stat.ps.dTLB-stores
  66954377 ±  3%     -61.8%   25576955        perf-stat.ps.iTLB-load-misses
   5775113 ±  2%     -23.5%    4417748        perf-stat.ps.iTLB-loads
 6.488e+10           -27.4%  4.711e+10        perf-stat.ps.instructions
      7249 ±  2%     -26.9%       5295 ±  9%  perf-stat.ps.minor-faults
   5797198 ±  2%    +212.5%   18114079 ±  4%  perf-stat.ps.node-load-misses
   4068827           -23.8%    3101203 ±  2%  perf-stat.ps.node-loads
   3075305          +610.3%   21845234 ±  3%  perf-stat.ps.node-store-misses
   9024717           -49.4%    4568971 ±  9%  perf-stat.ps.node-stores
      7297 ±  2%     -27.1%       5316 ±  9%  perf-stat.ps.page-faults
     85562            +1.9%      87147        perf-stat.ps.task-clock
 2.392e+12          +105.4%  4.912e+12        perf-stat.total.instructions
      0.00 ±103%    +205.3%       0.01 ± 29%  perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.00 ± 58%   +3372.7%       0.06 ± 99%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 22%    +446.7%       0.04 ± 50%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ±  8%     -15.4%       0.01 ±  6%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ±142%    +409.1%       0.01 ± 36%  perf-sched.sch_delay.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
      0.40 ± 47%     -95.1%       0.02 ± 19%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.00 ±105%    +652.4%       0.03 ± 51%  perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.00 ± 49%  +81913.8%       3.96 ±128%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 22%  +19317.8%       1.46 ±112%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ±  8%     +50.0%       0.01 ±  6%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ±223%  +2.2e+07%     333.68 ±141%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.00 ±142%   +6100.0%       0.11 ± 74%  perf-sched.sch_delay.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
      2.84 ±  5%     +35.0%       3.84        perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.02 ± 34%    +771.3%       0.17 ± 82%  perf-sched.total_sch_delay.average.ms
      3.21 ± 14%  +12411.1%     401.69 ±107%  perf-sched.total_sch_delay.max.ms
      0.14 ± 16%   +4872.4%       7.04 ± 22%  perf-sched.total_wait_and_delay.average.ms
    302.33 ± 28%  +1.1e+05%     321148 ± 93%  perf-sched.total_wait_and_delay.count.ms
      8.84 ± 30%  +40484.4%       3588 ± 48%  perf-sched.total_wait_and_delay.max.ms
      0.12 ± 16%   +5507.3%       6.87 ± 24%  perf-sched.total_wait_time.average.ms
      8.84 ± 30%  +40515.6%       3588 ± 48%  perf-sched.total_wait_time.max.ms
      1.06 ± 45%  +29152.9%     308.81 ± 48%  perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.59 ±100%  +52125.1%     308.82 ± 48%  perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.02 ± 51%  +3.2e+05%      68.67 ±  7%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.07 ±113%  +2.2e+05%     163.93 ± 47%  perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ±223%  +1.1e+05%       9.00 ±  5%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      2.14 ± 38%  +11944.8%     257.70 ± 32%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 45%  +4.8e+06%     333.01 ± 48%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ±223%  +5.6e+06%      46.27 ±  4%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
      0.01 ± 13%  +6.8e+06%     407.41 ±  7%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.41 ± 43%  +46090.8%     188.07 ±  8%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
      1.67 ± 44%    +600.0%      11.67 ±  4%  perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1.00 ±100%   +1066.7%      11.67 ±  4%  perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
      5.00 ± 25%  +13106.7%     660.33 ± 60%  perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     91.33 ± 95%    +353.3%     414.00 ± 39%  perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
     88.17          +434.2%     471.00 ± 35%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      7.83 ± 21%    +278.7%      29.67 ± 16%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.83 ± 44%   +1220.0%      11.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.17 ±223%  +48900.0%      81.67 ± 43%  perf-sched.wait_and_delay.count.schedule_timeout.xfsaild.kthread.ret_from_fork
     88.50         +1382.1%       1311 ± 68%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     10.17 ± 69%  +15801.6%       1616 ± 52%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
      2.10 ± 45%  +1.7e+05%       3512 ± 51%  perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1.18 ±100%    +3e+05%       3512 ± 51%  perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.09 ± 45%  +1.1e+06%       1000        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.23 ± 97%  +1.6e+06%       3515 ± 51%  perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.70 ±223%  +1.4e+05%       1000        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      8.43 ± 39%  +38126.9%       3223 ± 55%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 45%  +4.9e+07%       3418 ± 53%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ±223%  +6.3e+06%      52.15        perf-sched.wait_and_delay.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
      0.03 ± 89%    +9e+06%       2916 ± 43%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      2.84 ±  5%  +51408.3%       1463 ± 28%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
      1.04 ± 45%  +29564.3%     308.81 ± 48%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.59 ±100%  +52389.9%     308.82 ± 48%  perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.02 ± 56%  +3.5e+05%      68.60 ±  7%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.07 ±125%  +2.4e+05%     163.92 ± 47%  perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ±223%  +1.1e+05%       9.00 ±  5%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      2.00 ± 33%  +12772.7%     257.69 ± 32%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      3.03 ±  8%     +42.9%       4.34 ±  5%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ±223%  +1.9e+06%     188.05 ±  8%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
      2.08 ± 45%  +1.7e+05%       3512 ± 51%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1.18 ±100%    +3e+05%       3512 ± 51%  perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.08 ± 48%  +1.2e+06%       1000        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.14 ±127%  +2.6e+06%       3515 ± 51%  perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.70 ±223%  +1.4e+05%       1000        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      8.43 ± 39%  +38159.4%       3223 ± 55%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      4.64 ±  9%   +2289.8%     110.99 ± 53%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.25 ±223%  +5.9e+05%       1463 ± 28%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
      4251           +27.4%       5414        slabinfo.Acpi-Parse.active_objs
      4251           +27.4%       5414        slabinfo.Acpi-Parse.num_objs
    181.67 ± 17%    +438.0%     977.33        slabinfo.biovec-max.active_objs
     23.17 ± 21%    +466.9%     131.33        slabinfo.biovec-max.active_slabs
    190.17 ± 21%    +454.8%       1055        slabinfo.biovec-max.num_objs
     23.17 ± 21%    +466.9%     131.33        slabinfo.biovec-max.num_slabs
      1824          +371.6%       8606 ±  3%  slabinfo.btrfs_ordered_extent.active_objs
     48.83          +382.6%     235.67 ±  3%  slabinfo.btrfs_ordered_extent.active_slabs
      1824          +379.2%       8744 ±  3%  slabinfo.btrfs_ordered_extent.num_objs
     48.83          +382.6%     235.67 ±  3%  slabinfo.btrfs_ordered_extent.num_slabs
      4963 ± 36%    +801.3%      44735 ±  6%  slabinfo.dmaengine-unmap-16.active_objs
    118.00 ± 36%    +806.5%       1069 ±  6%  slabinfo.dmaengine-unmap-16.active_slabs
      4973 ± 36%    +803.8%      44945 ±  6%  slabinfo.dmaengine-unmap-16.num_objs
    118.00 ± 36%    +806.5%       1069 ±  6%  slabinfo.dmaengine-unmap-16.num_slabs
      2709           +27.3%       3448        slabinfo.ip6-frags.active_objs
      2709           +27.3%       3448        slabinfo.ip6-frags.num_objs
     81516           +17.5%      95804        slabinfo.kernfs_node_cache.active_objs
      2546           +17.9%       3003        slabinfo.kernfs_node_cache.active_slabs
     81516           +17.9%      96123        slabinfo.kernfs_node_cache.num_objs
      2546           +17.9%       3003        slabinfo.kernfs_node_cache.num_slabs
      5143 ±  2%    +217.9%      16348 ±  3%  slabinfo.kmalloc-128.active_objs
    160.83 ±  2%    +220.6%     515.67 ±  3%  slabinfo.kmalloc-128.active_slabs
      5156 ±  2%    +220.1%      16507 ±  3%  slabinfo.kmalloc-128.num_objs
    160.83 ±  2%    +220.6%     515.67 ±  3%  slabinfo.kmalloc-128.num_slabs
     33261           +83.7%      61110 ±  7%  slabinfo.kmalloc-16.active_objs
    129.83           +97.2%     256.00 ±  6%  slabinfo.kmalloc-16.active_slabs
     33266           +97.5%      65706 ±  6%  slabinfo.kmalloc-16.num_objs
    129.83           +97.2%     256.00 ±  6%  slabinfo.kmalloc-16.num_slabs
      6235 ± 12%    +333.7%      27041 ±  5%  slabinfo.kmalloc-192.active_objs
    149.50 ± 12%    +334.8%     650.00 ±  5%  slabinfo.kmalloc-192.active_slabs
      6306 ± 12%    +333.4%      27330 ±  5%  slabinfo.kmalloc-192.num_objs
    149.50 ± 12%    +334.8%     650.00 ±  5%  slabinfo.kmalloc-192.num_slabs
      5762           +68.7%       9719        slabinfo.kmalloc-1k.active_objs
    182.50           +69.5%     309.33        slabinfo.kmalloc-1k.active_slabs
      5851           +69.2%       9904        slabinfo.kmalloc-1k.num_objs
    182.50           +69.5%     309.33        slabinfo.kmalloc-1k.num_slabs
      1855           +29.2%       2397        slabinfo.kmalloc-4k.active_objs
      1867           +31.3%       2451        slabinfo.kmalloc-4k.num_objs
     17504           +58.7%      27772        slabinfo.kmalloc-512.active_objs
    569.50           +68.0%     957.00        slabinfo.kmalloc-512.active_slabs
     18245           +67.9%      30639        slabinfo.kmalloc-512.num_objs
    569.50           +68.0%     957.00        slabinfo.kmalloc-512.num_slabs
     51165           +61.1%      82405 ±  2%  slabinfo.kmalloc-64.active_objs
    800.50           +61.7%       1294 ±  2%  slabinfo.kmalloc-64.active_slabs
     51252           +61.7%      82857 ±  2%  slabinfo.kmalloc-64.num_objs
    800.50           +61.7%       1294 ±  2%  slabinfo.kmalloc-64.num_slabs
    888.17           +25.8%       1117        slabinfo.kmalloc-8k.active_objs
    892.67           +32.9%       1186        slabinfo.kmalloc-8k.num_objs
    826.67 ± 14%     -26.5%     608.00 ± 14%  slabinfo.kmalloc-rcl-128.active_objs
    826.67 ± 14%     -26.5%     608.00 ± 14%  slabinfo.kmalloc-rcl-128.num_objs
      2000           +26.0%       2519        slabinfo.kmalloc-rcl-256.active_objs
      2000           +26.0%       2519        slabinfo.kmalloc-rcl-256.num_objs
      3722           +72.9%       6434        slabinfo.numa_policy.active_objs
      3722           +72.9%       6434        slabinfo.numa_policy.num_objs
      2552 ±  3%    +239.9%       8677 ±  9%  slabinfo.pool_workqueue.active_objs
     79.67 ±  4%    +248.5%     277.67 ±  9%  slabinfo.pool_workqueue.active_slabs
      2557 ±  4%    +247.8%       8896 ±  9%  slabinfo.pool_workqueue.num_objs
     79.67 ±  4%    +248.5%     277.67 ±  9%  slabinfo.pool_workqueue.num_slabs
     34443           +16.4%      40094        slabinfo.proc_inode_cache.active_objs
    723.33           +18.4%     856.33        slabinfo.proc_inode_cache.active_slabs
     34748           +18.4%      41125        slabinfo.proc_inode_cache.num_objs
    723.33           +18.4%     856.33        slabinfo.proc_inode_cache.num_slabs
     37280 ±  3%    +129.7%      85641        slabinfo.radix_tree_node.active_objs
    668.00 ±  3%    +129.9%       1535        slabinfo.radix_tree_node.active_slabs
     37431 ±  3%    +129.8%      86017        slabinfo.radix_tree_node.num_objs
    668.00 ±  3%    +129.9%       1535        slabinfo.radix_tree_node.num_slabs
      2220           +27.1%       2822        slabinfo.xfs_btree_cur.active_objs
      2220           +27.1%       2822        slabinfo.xfs_btree_cur.num_objs
      2261 ±  2%     +45.0%       3279        slabinfo.xfs_buf.active_objs
      2261 ±  2%     +45.0%       3279        slabinfo.xfs_buf.num_objs
      2047           +29.4%       2648        slabinfo.xfs_da_state.active_objs
      2047           +29.4%       2648        slabinfo.xfs_da_state.num_objs
      1874          +386.8%       9125 ±  6%  slabinfo.xfs_efi_item.active_objs
     48.83          +398.3%     243.33 ±  6%  slabinfo.xfs_efi_item.active_slabs
      1874          +394.1%       9262 ±  6%  slabinfo.xfs_efi_item.num_objs
     48.83          +398.3%     243.33 ±  6%  slabinfo.xfs_efi_item.num_slabs
      1614 ±  6%    +120.7%       3564        slabinfo.xfs_icr.active_objs
      1614 ±  6%    +120.7%       3564        slabinfo.xfs_icr.num_objs
      2609          +368.5%      12227        slabinfo.xfs_ili.active_objs
     61.67          +435.1%     330.00 ±  4%  slabinfo.xfs_ili.active_slabs
      2609          +431.4%      13868 ±  4%  slabinfo.xfs_ili.num_objs
     61.67          +435.1%     330.00 ±  4%  slabinfo.xfs_ili.num_slabs
      2182 ±  9%    +417.0%      11285        slabinfo.xfs_inode.active_objs
     68.50 ± 10%    +508.8%     417.00        slabinfo.xfs_inode.active_slabs
      2205 ± 11%    +506.0%      13365        slabinfo.xfs_inode.num_objs
     68.50 ± 10%    +508.8%     417.00        slabinfo.xfs_inode.num_slabs
      9505 ±  5%     +57.9%      15011 ±  3%  softirqs.CPU0.SCHED
      1106 ± 45%   +1133.7%      13646 ±  2%  softirqs.CPU0.TIMER
      8817 ±  8%     +46.5%      12913 ±  6%  softirqs.CPU1.SCHED
     98.00 ± 22%  +16851.0%      16612 ± 27%  softirqs.CPU1.TIMER
      7122 ±  2%     +73.6%      12362        softirqs.CPU10.SCHED
     73.17 ± 26%  +17354.2%      12770 ±  2%  softirqs.CPU10.TIMER
      7042 ±  6%     +79.5%      12640 ±  4%  softirqs.CPU11.SCHED
    263.67 ±166%   +4832.6%      13005        softirqs.CPU11.TIMER
      7127 ±  3%     +66.9%      11895        softirqs.CPU12.SCHED
     66.83 ±  8%  +18410.2%      12371 ±  2%  softirqs.CPU12.TIMER
      7096           +73.5%      12311 ±  2%  softirqs.CPU13.SCHED
     63.50 ± 16%  +20027.6%      12781 ±  3%  softirqs.CPU13.TIMER
      7234 ±  2%     +76.1%      12742 ±  5%  softirqs.CPU14.SCHED
    146.17 ± 90%   +8727.4%      12902 ±  4%  softirqs.CPU14.TIMER
      7017 ±  4%     +77.1%      12430        softirqs.CPU15.SCHED
     65.17 ± 16%  +20152.7%      13198        softirqs.CPU15.TIMER
      6872 ±  3%     +80.5%      12407 ±  3%  softirqs.CPU16.SCHED
     68.83 ± 24%  +18375.5%      12717        softirqs.CPU16.TIMER
      7059 ±  2%     +70.3%      12021        softirqs.CPU17.SCHED
     73.00 ± 41%  +16926.5%      12429 ±  6%  softirqs.CPU17.TIMER
      7165 ±  3%     +70.7%      12232        softirqs.CPU18.SCHED
     67.50 ± 30%  +19107.9%      12965 ±  6%  softirqs.CPU18.TIMER
      7064 ±  4%     +69.7%      11986 ±  5%  softirqs.CPU19.SCHED
    143.50 ± 97%   +8990.4%      13044 ±  4%  softirqs.CPU19.TIMER
      7667 ±  6%     +68.9%      12954 ±  6%  softirqs.CPU2.SCHED
    151.33 ± 87%   +8218.1%      12588 ±  5%  softirqs.CPU2.TIMER
      7361 ±  3%     +66.9%      12283 ±  4%  softirqs.CPU20.SCHED
    125.00 ± 93%  +10287.7%      12984 ±  4%  softirqs.CPU20.TIMER
      7113 ±  3%     +74.4%      12407 ±  2%  softirqs.CPU21.SCHED
     84.67 ± 34%  +15126.4%      12891 ±  8%  softirqs.CPU21.TIMER
      7512 ±  5%     +67.0%      12546 ±  2%  softirqs.CPU22.SCHED
    672.33 ± 89%   +2029.3%      14316 ±  5%  softirqs.CPU22.TIMER
      7326 ±  5%     +67.4%      12266        softirqs.CPU23.SCHED
    270.50 ±128%   +4778.6%      13196 ±  2%  softirqs.CPU23.TIMER
      7226 ±  3%     +73.7%      12551        softirqs.CPU24.SCHED
    139.50 ±102%   +9707.4%      13681 ±  2%  softirqs.CPU24.TIMER
      7190 ±  2%     +74.3%      12531        softirqs.CPU25.SCHED
     79.50 ± 44%  +17110.9%      13682 ±  3%  softirqs.CPU25.TIMER
      7220 ±  2%     +70.7%      12322 ±  2%  softirqs.CPU26.SCHED
    106.33 ± 75%  +12508.8%      13407 ±  2%  softirqs.CPU26.TIMER
      7120           +77.6%      12647        softirqs.CPU27.SCHED
     75.50 ± 23%  +18277.5%      13875 ±  4%  softirqs.CPU27.TIMER
      6985 ±  3%     +77.2%      12378 ±  3%  softirqs.CPU28.SCHED
     78.00 ± 60%  +17241.0%      13526        softirqs.CPU28.TIMER
      7135           +76.0%      12556 ±  2%  softirqs.CPU29.SCHED
     70.50 ± 25%  +19312.8%      13686 ±  2%  softirqs.CPU29.TIMER
      7127           +79.2%      12771 ±  3%  softirqs.CPU3.SCHED
     82.00 ± 29%  +15380.5%      12694 ±  4%  softirqs.CPU3.TIMER
      6959 ±  4%     +79.4%      12486 ±  2%  softirqs.CPU30.SCHED
     71.67 ± 27%  +18745.6%      13506        softirqs.CPU30.TIMER
      7032 ±  3%     +76.0%      12376 ±  3%  softirqs.CPU31.SCHED
    102.33 ± 35%  +12740.1%      13139 ±  2%  softirqs.CPU31.TIMER
      7115           +75.8%      12510 ±  2%  softirqs.CPU32.SCHED
    138.00 ± 90%   +9605.6%      13393 ±  2%  softirqs.CPU32.TIMER
      6875 ±  4%     +81.1%      12453        softirqs.CPU33.SCHED
     79.00 ± 27%  +17002.1%      13510        softirqs.CPU33.TIMER
      7113           +74.2%      12393        softirqs.CPU34.SCHED
     82.00 ± 14%  +15853.7%      13082        softirqs.CPU34.TIMER
      7108           +79.1%      12728 ±  2%  softirqs.CPU35.SCHED
     83.17 ± 20%  +16494.0%      13800 ±  3%  softirqs.CPU35.TIMER
      7068 ±  2%     +77.3%      12531        softirqs.CPU36.SCHED
     86.50 ± 56%  +15512.3%      13504        softirqs.CPU36.TIMER
      7263 ±  3%     +76.2%      12795 ±  2%  softirqs.CPU37.SCHED
     71.00 ± 23%  +19777.5%      14113        softirqs.CPU37.TIMER
      7108           +76.1%      12515        softirqs.CPU38.SCHED
     72.67 ± 14%  +18910.1%      13814 ±  3%  softirqs.CPU38.TIMER
      7069 ±  2%     +73.2%      12243 ±  3%  softirqs.CPU39.SCHED
     88.50 ± 58%  +15255.2%      13589 ±  2%  softirqs.CPU39.TIMER
      7150 ±  4%     +70.8%      12212        softirqs.CPU4.SCHED
    246.33 ±130%   +4968.2%      12484 ±  3%  softirqs.CPU4.TIMER
      5770           +33.0%       7672 ± 24%  softirqs.CPU40.RCU
      7122           +77.3%      12629 ±  3%  softirqs.CPU40.SCHED
     89.83 ± 61%  +15410.6%      13933 ±  3%  softirqs.CPU40.TIMER
      6948 ±  3%     +78.2%      12380 ±  3%  softirqs.CPU41.SCHED
     77.83 ± 18%  +17066.6%      13361 ±  3%  softirqs.CPU41.TIMER
      6902 ±  4%     +83.1%      12640        softirqs.CPU42.SCHED
     59.33 ± 14%  +22711.2%      13534 ±  2%  softirqs.CPU42.TIMER
      5941 ±  8%    +107.2%      12307 ±  3%  softirqs.CPU43.SCHED
    134.83 ±111%  +10145.2%      13814 ±  3%  softirqs.CPU43.TIMER
      6788 ±  4%     +75.5%      11912 ±  2%  softirqs.CPU44.SCHED
     92.50 ± 46%  +12616.8%      11763 ±  5%  softirqs.CPU44.TIMER
      7075 ±  3%     +70.2%      12045 ±  3%  softirqs.CPU45.SCHED
     71.50 ± 25%  +16571.3%      11920 ±  4%  softirqs.CPU45.TIMER
      6788 ±  5%     +75.5%      11912 ±  3%  softirqs.CPU46.SCHED
    104.67 ± 49%  +11238.9%      11868 ±  5%  softirqs.CPU46.TIMER
      6730 ±  5%     +81.1%      12187 ±  3%  softirqs.CPU47.SCHED
     82.50 ± 50%  +14319.4%      11896 ±  4%  softirqs.CPU47.TIMER
      7415 ±  4%     +67.6%      12424 ±  3%  softirqs.CPU48.SCHED
    567.50 ± 93%   +2084.8%      12399 ±  3%  softirqs.CPU48.TIMER
      7048           +70.9%      12048 ±  2%  softirqs.CPU49.SCHED
    115.50 ± 84%  +10269.7%      11977 ±  2%  softirqs.CPU49.TIMER
      7336 ±  4%     +65.7%      12157 ±  2%  softirqs.CPU5.SCHED
     75.33 ± 23%  +16814.6%      12742 ±  2%  softirqs.CPU5.TIMER
      7161 ±  4%     +67.5%      11996        softirqs.CPU50.SCHED
    262.83 ±129%   +4341.0%      11672 ±  3%  softirqs.CPU50.TIMER
      6962 ±  3%     +65.2%      11503 ±  4%  softirqs.CPU51.SCHED
     65.50 ± 18%  +17328.0%      11415 ±  4%  softirqs.CPU51.TIMER
      7141 ±  2%     +72.4%      12309        softirqs.CPU52.SCHED
    106.67 ± 86%  +11501.9%      12375 ±  5%  softirqs.CPU52.TIMER
      7157 ±  7%     +70.9%      12230 ±  2%  softirqs.CPU53.SCHED
    297.50 ±157%   +3808.7%      11628 ±  3%  softirqs.CPU53.TIMER
      7099 ±  2%     +70.4%      12096        softirqs.CPU54.SCHED
    100.17 ± 36%  +11642.4%      11762 ±  3%  softirqs.CPU54.TIMER
      7030 ±  2%     +73.1%      12172 ±  2%  softirqs.CPU55.SCHED
     58.67 ± 16%  +21051.1%      12408 ±  4%  softirqs.CPU55.TIMER
      7008 ±  2%     +73.8%      12178 ±  3%  softirqs.CPU56.SCHED
     54.50 ± 17%  +21824.2%      11948 ±  6%  softirqs.CPU56.TIMER
      7212 ±  3%     +65.8%      11958 ±  6%  softirqs.CPU57.SCHED
     82.67 ± 34%  +15168.5%      12622 ±  2%  softirqs.CPU57.TIMER
      7173 ±  3%     +66.9%      11970 ±  2%  softirqs.CPU58.SCHED
    189.50 ±133%   +6186.5%      11913 ±  3%  softirqs.CPU58.TIMER
      7231 ±  2%     +70.5%      12328        softirqs.CPU59.SCHED
    119.67 ± 83%   +9952.1%      12029 ±  2%  softirqs.CPU59.TIMER
      7045 ±  2%     +75.4%      12356 ±  2%  softirqs.CPU6.SCHED
    197.00 ± 71%   +6338.6%      12684        softirqs.CPU6.TIMER
      7321 ±  4%     +62.8%      11920 ±  4%  softirqs.CPU60.SCHED
     71.83 ± 11%  +16759.9%      12111 ±  3%  softirqs.CPU60.TIMER
      7193 ±  3%     +70.0%      12227        softirqs.CPU61.SCHED
     61.33 ± 16%  +18797.8%      11590 ±  3%  softirqs.CPU61.TIMER
      7180 ±  2%     +71.8%      12334        softirqs.CPU62.SCHED
     67.00 ± 24%  +18788.6%      12655 ±  3%  softirqs.CPU62.TIMER
      7061 ±  3%     +76.5%      12465 ±  3%  softirqs.CPU63.SCHED
     77.33 ± 14%  +20473.3%      15910 ± 32%  softirqs.CPU63.TIMER
      6996 ±  3%     +74.6%      12215 ±  2%  softirqs.CPU64.SCHED
     75.50 ± 39%  +15807.7%      12010 ±  3%  softirqs.CPU64.TIMER
      7077 ±  2%     +77.5%      12561 ±  4%  softirqs.CPU65.SCHED
    106.17 ± 67%  +11663.9%      12489 ±  8%  softirqs.CPU65.TIMER
      7164 ±  5%     +74.1%      12475        softirqs.CPU66.SCHED
    196.67 ±134%   +6430.3%      12843 ±  5%  softirqs.CPU66.TIMER
      7160           +73.1%      12394        softirqs.CPU67.SCHED
     84.50 ± 36%  +15319.3%      13029        softirqs.CPU67.TIMER
      7160           +72.6%      12361 ±  3%  softirqs.CPU68.SCHED
     78.67 ± 13%  +16497.9%      13057 ±  4%  softirqs.CPU68.TIMER
      7182           +71.2%      12296        softirqs.CPU69.SCHED
     78.83 ± 31%  +16352.0%      12969 ±  2%  softirqs.CPU69.TIMER
      7108 ±  2%     +74.0%      12370 ±  2%  softirqs.CPU7.SCHED
     69.33 ± 23%  +18549.5%      12930 ±  4%  softirqs.CPU7.TIMER
      7276 ±  3%     +67.5%      12187 ±  2%  softirqs.CPU70.SCHED
     76.33 ± 16%  +16856.3%      12943 ±  5%  softirqs.CPU70.TIMER
      7308 ±  3%     +71.8%      12553 ±  2%  softirqs.CPU71.SCHED
    202.67 ±142%   +6118.8%      12603        softirqs.CPU71.TIMER
      7190           +72.5%      12401        softirqs.CPU72.SCHED
     62.83 ± 16%  +19565.3%      12356        softirqs.CPU72.TIMER
      7140           +74.7%      12477        softirqs.CPU73.SCHED
    127.00 ±119%   +9902.1%      12702        softirqs.CPU73.TIMER
      7167           +74.4%      12500        softirqs.CPU74.SCHED
     73.83 ± 25%  +17287.8%      12838 ±  5%  softirqs.CPU74.TIMER
      6952 ±  6%     +76.1%      12244 ±  2%  softirqs.CPU75.SCHED
    105.00 ± 42%  +11935.2%      12637        softirqs.CPU75.TIMER
      7176           +70.8%      12259 ±  2%  softirqs.CPU76.SCHED
    154.00 ±123%   +7910.2%      12335 ±  2%  softirqs.CPU76.TIMER
      7119           +74.2%      12401 ±  2%  softirqs.CPU77.SCHED
     63.33 ± 17%  +20450.0%      13015 ±  3%  softirqs.CPU77.TIMER
      7228           +72.2%      12447 ±  3%  softirqs.CPU78.SCHED
     81.50 ± 52%  +15505.3%      12718 ±  3%  softirqs.CPU78.TIMER
      7239           +75.4%      12699 ±  2%  softirqs.CPU79.SCHED
    118.83 ± 57%  +10707.0%      12842 ±  3%  softirqs.CPU79.TIMER
      7269 ±  3%     +71.1%      12437        softirqs.CPU8.SCHED
    152.67 ±101%   +8360.7%      12916 ±  4%  softirqs.CPU8.TIMER
      7116           +79.6%      12784        softirqs.CPU80.SCHED
     59.50 ± 15%  +27879.8%      16648 ± 36%  softirqs.CPU80.TIMER
      7017 ±  3%     +79.3%      12580        softirqs.CPU81.SCHED
     59.00 ± 14%  +21762.7%      12899 ±  3%  softirqs.CPU81.TIMER
      7105 ±  3%     +74.5%      12400        softirqs.CPU82.SCHED
     87.50 ± 67%  +14481.7%      12759 ±  2%  softirqs.CPU82.TIMER
      6949 ±  3%     +76.7%      12279 ±  2%  softirqs.CPU83.SCHED
     57.83 ± 23%  +21837.2%      12687        softirqs.CPU83.TIMER
      7054 ±  3%     +77.5%      12518        softirqs.CPU84.SCHED
     59.17 ± 11%  +21385.6%      12712 ±  3%  softirqs.CPU84.TIMER
      7199 ±  2%     +74.5%      12561        softirqs.CPU85.SCHED
    104.67 ±105%  +12582.5%      13274 ±  2%  softirqs.CPU85.TIMER
      7221 ±  3%     +72.8%      12475 ±  2%  softirqs.CPU86.SCHED
    142.83 ±144%   +9249.8%      13354 ±  5%  softirqs.CPU86.TIMER
      7060 ±  2%     +74.2%      12298 ±  4%  softirqs.CPU87.SCHED
     64.83 ±  5%  +20128.3%      13114 ±  4%  softirqs.CPU87.TIMER
      7118 ±  4%     +80.6%      12852 ±  5%  softirqs.CPU9.SCHED
    121.50 ± 89%  +10547.7%      12937        softirqs.CPU9.TIMER
    629837           +73.2%    1091167        softirqs.SCHED
     11262        +10026.4%    1140490        softirqs.TIMER
     75.33          +177.0%     208.67        interrupts.9:IO-APIC.9-fasteoi.acpi
     69128 ±  2%      -4.4%      66113        interrupts.CAL:Function_call_interrupts
     73390 ±  3%    +184.1%     208526        interrupts.CPU0.LOC:Local_timer_interrupts
     86.33 ± 20%   +1276.8%       1188 ± 14%  interrupts.CPU0.RES:Rescheduling_interrupts
     75.33          +177.0%     208.67        interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
      1603 ± 54%     -48.1%     833.00 ± 22%  interrupts.CPU1.CAL:Function_call_interrupts
     73430 ±  3%    +183.5%     208179        interrupts.CPU1.LOC:Local_timer_interrupts
     65.50 ±  7%   +1542.2%       1075 ± 28%  interrupts.CPU1.RES:Rescheduling_interrupts
     73459 ±  3%    +184.1%     208711        interrupts.CPU10.LOC:Local_timer_interrupts
     67.67 ± 12%   +1181.8%     867.33 ± 16%  interrupts.CPU10.RES:Rescheduling_interrupts
     73450 ±  3%    +184.1%     208660        interrupts.CPU11.LOC:Local_timer_interrupts
     61.00 ± 16%   +1297.3%     852.33 ± 10%  interrupts.CPU11.RES:Rescheduling_interrupts
     73426 ±  3%    +184.2%     208679        interrupts.CPU12.LOC:Local_timer_interrupts
     59.83 ± 14%   +1296.7%     835.67 ± 16%  interrupts.CPU12.RES:Rescheduling_interrupts
    754.83 ±  4%     -12.0%     664.33        interrupts.CPU13.CAL:Function_call_interrupts
     73402 ±  3%    +184.1%     208539        interrupts.CPU13.LOC:Local_timer_interrupts
     57.33 ± 16%   +1377.9%     847.33 ± 15%  interrupts.CPU13.RES:Rescheduling_interrupts
     73478 ±  3%    +183.7%     208453        interrupts.CPU14.LOC:Local_timer_interrupts
     62.50 ± 13%   +1146.9%     779.33 ± 15%  interrupts.CPU14.RES:Rescheduling_interrupts
     73414 ±  3%    +184.1%     208584        interrupts.CPU15.LOC:Local_timer_interrupts
     63.67 ±  9%   +1228.8%     846.00 ± 12%  interrupts.CPU15.RES:Rescheduling_interrupts
     73359 ±  3%    +184.2%     208494        interrupts.CPU16.LOC:Local_timer_interrupts
     65.83 ± 15%   +1140.0%     816.33 ± 17%  interrupts.CPU16.RES:Rescheduling_interrupts
     73439 ±  3%    +184.1%     208619        interrupts.CPU17.LOC:Local_timer_interrupts
     63.00 ± 14%   +1212.2%     826.67 ± 13%  interrupts.CPU17.RES:Rescheduling_interrupts
     73432 ±  3%    +184.0%     208554        interrupts.CPU18.LOC:Local_timer_interrupts
     64.50 ±  7%   +1451.9%       1001 ± 26%  interrupts.CPU18.RES:Rescheduling_interrupts
    738.83 ±  4%      -8.8%     674.00 ±  2%  interrupts.CPU19.CAL:Function_call_interrupts
     73425 ±  3%    +184.1%     208617        interrupts.CPU19.LOC:Local_timer_interrupts
     70.00 ± 10%   +1003.3%     772.33 ± 11%  interrupts.CPU19.RES:Rescheduling_interrupts
     73406 ±  3%    +184.3%     208674        interrupts.CPU2.LOC:Local_timer_interrupts
     69.83 ± 10%   +1121.5%     853.00 ± 18%  interrupts.CPU2.RES:Rescheduling_interrupts
     73291 ±  3%    +184.7%     208646        interrupts.CPU20.LOC:Local_timer_interrupts
     63.00 ± 12%   +1332.3%     902.33 ± 19%  interrupts.CPU20.RES:Rescheduling_interrupts
     73415 ±  3%    +183.6%     208233        interrupts.CPU21.LOC:Local_timer_interrupts
     71.50 ± 22%   +1060.8%     830.00 ± 12%  interrupts.CPU21.RES:Rescheduling_interrupts
     73732 ±  2%    +181.2%     207362        interrupts.CPU22.LOC:Local_timer_interrupts
     75.33 ± 21%    +951.3%     792.00 ±  3%  interrupts.CPU22.RES:Rescheduling_interrupts
    765.50 ±  4%     -15.2%     649.00 ± 14%  interrupts.CPU23.CAL:Function_call_interrupts
     73827 ±  2%    +181.4%     207739        interrupts.CPU23.LOC:Local_timer_interrupts
     65.50 ± 28%   +1031.8%     741.33        interrupts.CPU23.RES:Rescheduling_interrupts
     73818 ±  2%    +181.4%     207759        interrupts.CPU24.LOC:Local_timer_interrupts
     77.17 ± 17%    +858.5%     739.67 ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
     73809 ±  2%    +180.6%     207107        interrupts.CPU25.LOC:Local_timer_interrupts
     76.33 ± 20%    +850.7%     725.67 ±  2%  interrupts.CPU25.RES:Rescheduling_interrupts
     73765 ±  2%    +182.0%     208017        interrupts.CPU26.LOC:Local_timer_interrupts
     74.33 ± 18%    +890.1%     736.00 ±  4%  interrupts.CPU26.RES:Rescheduling_interrupts
     73732 ±  2%    +182.1%     208000        interrupts.CPU27.LOC:Local_timer_interrupts
     71.67 ± 24%    +901.4%     717.67 ±  7%  interrupts.CPU27.RES:Rescheduling_interrupts
     73803 ±  2%    +181.9%     208035        interrupts.CPU28.LOC:Local_timer_interrupts
      3649 ±  3%      -6.5%       3411        interrupts.CPU28.NMI:Non-maskable_interrupts
      3649 ±  3%      -6.5%       3411        interrupts.CPU28.PMI:Performance_monitoring_interrupts
     72.83 ± 20%    +905.5%     732.33 ±  3%  interrupts.CPU28.RES:Rescheduling_interrupts
    814.50 ± 12%     -14.3%     698.33        interrupts.CPU29.CAL:Function_call_interrupts
     73815 ±  2%    +181.7%     207944        interrupts.CPU29.LOC:Local_timer_interrupts
     69.17 ± 16%    +959.3%     732.67 ±  3%  interrupts.CPU29.RES:Rescheduling_interrupts
     73547 ±  4%    +183.5%     208533        interrupts.CPU3.LOC:Local_timer_interrupts
     63.17 ±  5%   +1237.7%     845.00 ± 12%  interrupts.CPU3.RES:Rescheduling_interrupts
    765.83 ±  5%     -12.4%     671.00 ± 10%  interrupts.CPU30.CAL:Function_call_interrupts
     73943 ±  2%    +181.3%     208002        interrupts.CPU30.LOC:Local_timer_interrupts
     65.00 ± 13%   +1002.1%     716.33 ±  4%  interrupts.CPU30.RES:Rescheduling_interrupts
    752.00 ±  3%      -8.3%     689.33 ±  3%  interrupts.CPU31.CAL:Function_call_interrupts
     73933 ±  3%    +181.0%     207782        interrupts.CPU31.LOC:Local_timer_interrupts
     65.50 ±  7%    +972.8%     702.67 ±  2%  interrupts.CPU31.RES:Rescheduling_interrupts
     73807 ±  2%    +181.8%     208008        interrupts.CPU32.LOC:Local_timer_interrupts
     70.00 ± 13%    +863.3%     674.33 ±  4%  interrupts.CPU32.RES:Rescheduling_interrupts
    802.83 ±  6%     -11.3%     712.33 ±  4%  interrupts.CPU33.CAL:Function_call_interrupts
     73789 ±  2%    +181.6%     207813        interrupts.CPU33.LOC:Local_timer_interrupts
     66.17 ± 12%   +1004.8%     731.00 ±  4%  interrupts.CPU33.RES:Rescheduling_interrupts
     73772 ±  2%    +182.0%     208035        interrupts.CPU34.LOC:Local_timer_interrupts
     59.67 ± 17%   +1091.6%     711.00 ±  2%  interrupts.CPU34.RES:Rescheduling_interrupts
     73773 ±  2%    +182.0%     208056        interrupts.CPU35.LOC:Local_timer_interrupts
     66.17 ± 13%    +997.2%     726.00 ±  2%  interrupts.CPU35.RES:Rescheduling_interrupts
    739.67 ±  4%     -15.8%     623.00 ± 10%  interrupts.CPU36.CAL:Function_call_interrupts
     73768 ±  2%    +182.1%     208074        interrupts.CPU36.LOC:Local_timer_interrupts
     68.00 ± 12%    +893.1%     675.33 ±  7%  interrupts.CPU36.RES:Rescheduling_interrupts
     73787 ±  2%    +181.6%     207766        interrupts.CPU37.LOC:Local_timer_interrupts
     64.83 ± 13%    +983.3%     702.33 ±  3%  interrupts.CPU37.RES:Rescheduling_interrupts
     73784 ±  2%    +181.2%     207450        interrupts.CPU38.LOC:Local_timer_interrupts
     66.00 ± 11%    +938.9%     685.67        interrupts.CPU38.RES:Rescheduling_interrupts
    777.67 ±  5%     -10.6%     695.00 ±  3%  interrupts.CPU39.CAL:Function_call_interrupts
     73795 ±  2%    +181.7%     207844        interrupts.CPU39.LOC:Local_timer_interrupts
     66.67 ±  9%    +940.5%     693.67 ±  3%  interrupts.CPU39.RES:Rescheduling_interrupts
     73388 ±  3%    +183.7%     208191        interrupts.CPU4.LOC:Local_timer_interrupts
     62.00 ± 18%   +1291.9%     863.00 ± 15%  interrupts.CPU4.RES:Rescheduling_interrupts
     73798 ±  2%    +181.6%     207846        interrupts.CPU40.LOC:Local_timer_interrupts
     65.00 ± 14%    +978.5%     701.00 ±  2%  interrupts.CPU40.RES:Rescheduling_interrupts
     73771 ±  2%    +181.7%     207825        interrupts.CPU41.LOC:Local_timer_interrupts
     58.83 ± 11%   +1110.8%     712.33        interrupts.CPU41.RES:Rescheduling_interrupts
     73795 ±  2%    +181.5%     207735        interrupts.CPU42.LOC:Local_timer_interrupts
     62.50 ± 18%   +1063.7%     727.33 ±  3%  interrupts.CPU42.RES:Rescheduling_interrupts
     73704 ±  3%    +181.5%     207445        interrupts.CPU43.LOC:Local_timer_interrupts
     66.00 ±  6%    +971.2%     707.00 ±  5%  interrupts.CPU43.RES:Rescheduling_interrupts
    760.67 ±  3%      -9.2%     691.00 ±  5%  interrupts.CPU44.CAL:Function_call_interrupts
     73290 ±  3%    +184.4%     208461        interrupts.CPU44.LOC:Local_timer_interrupts
     39.00 ± 16%   +1653.8%     684.00 ± 14%  interrupts.CPU44.RES:Rescheduling_interrupts
     73393 ±  3%    +183.5%     208070        interrupts.CPU45.LOC:Local_timer_interrupts
     34.33 ± 14%   +1826.2%     661.33 ± 23%  interrupts.CPU45.RES:Rescheduling_interrupts
     73349 ±  3%    +184.4%     208609        interrupts.CPU46.LOC:Local_timer_interrupts
     37.83 ± 12%   +2113.2%     837.33 ± 23%  interrupts.CPU46.RES:Rescheduling_interrupts
     73320 ±  3%    +184.4%     208526        interrupts.CPU47.LOC:Local_timer_interrupts
     39.50 ±  9%   +2043.5%     846.67 ± 16%  interrupts.CPU47.RES:Rescheduling_interrupts
     73436 ±  3%    +183.4%     208153        interrupts.CPU48.LOC:Local_timer_interrupts
     43.50 ± 20%   +1739.1%     800.00 ± 18%  interrupts.CPU48.RES:Rescheduling_interrupts
     73456 ±  3%    +183.7%     208390        interrupts.CPU49.LOC:Local_timer_interrupts
     33.67 ± 23%   +2169.3%     764.00 ± 16%  interrupts.CPU49.RES:Rescheduling_interrupts
     73484 ±  3%    +183.6%     208388        interrupts.CPU5.LOC:Local_timer_interrupts
     62.83 ±  5%   +1256.0%     852.00 ± 14%  interrupts.CPU5.RES:Rescheduling_interrupts
     73391 ±  3%    +184.3%     208675        interrupts.CPU50.LOC:Local_timer_interrupts
     40.00 ± 26%   +1620.8%     688.33 ± 13%  interrupts.CPU50.RES:Rescheduling_interrupts
     73413 ±  3%    +184.1%     208602        interrupts.CPU51.LOC:Local_timer_interrupts
     39.33 ± 12%   +1889.0%     782.33 ± 14%  interrupts.CPU51.RES:Rescheduling_interrupts
    754.17 ±  6%     -10.1%     677.67        interrupts.CPU52.CAL:Function_call_interrupts
     73417 ±  3%    +184.2%     208660        interrupts.CPU52.LOC:Local_timer_interrupts
     38.67 ± 18%   +2088.8%     846.33 ± 22%  interrupts.CPU52.RES:Rescheduling_interrupts
     73377 ±  3%    +184.3%     208626        interrupts.CPU53.LOC:Local_timer_interrupts
     38.83 ± 18%   +1877.7%     768.00 ± 20%  interrupts.CPU53.RES:Rescheduling_interrupts
     73434 ±  3%    +184.3%     208740        interrupts.CPU54.LOC:Local_timer_interrupts
     44.67 ± 21%   +1759.0%     830.33 ± 16%  interrupts.CPU54.RES:Rescheduling_interrupts
     73427 ±  3%    +184.3%     208776        interrupts.CPU55.LOC:Local_timer_interrupts
     43.50 ± 38%   +1802.7%     827.67 ± 14%  interrupts.CPU55.RES:Rescheduling_interrupts
    746.50 ±  3%      -9.4%     676.67        interrupts.CPU56.CAL:Function_call_interrupts
     73338 ±  4%    +184.3%     208476        interrupts.CPU56.LOC:Local_timer_interrupts
     38.50 ± 25%   +1947.6%     788.33 ± 13%  interrupts.CPU56.RES:Rescheduling_interrupts
    784.67 ±  6%     -11.9%     691.67        interrupts.CPU57.CAL:Function_call_interrupts
     73452 ±  3%    +184.0%     208575        interrupts.CPU57.LOC:Local_timer_interrupts
     40.83 ± 14%   +1945.7%     835.33 ± 20%  interrupts.CPU57.RES:Rescheduling_interrupts
     73432 ±  3%    +184.1%     208643        interrupts.CPU58.LOC:Local_timer_interrupts
     41.83 ± 30%   +1687.3%     747.67 ±  9%  interrupts.CPU58.RES:Rescheduling_interrupts
    754.00 ±  3%      -8.5%     690.00 ±  4%  interrupts.CPU59.CAL:Function_call_interrupts
     73369 ±  3%    +184.3%     208599        interrupts.CPU59.LOC:Local_timer_interrupts
     46.83 ± 36%   +1675.1%     831.33 ± 16%  interrupts.CPU59.RES:Rescheduling_interrupts
     73432 ±  3%    +184.2%     208688        interrupts.CPU6.LOC:Local_timer_interrupts
     67.67 ± 25%   +1154.2%     848.67 ± 12%  interrupts.CPU6.RES:Rescheduling_interrupts
     73424 ±  3%    +184.0%     208506        interrupts.CPU60.LOC:Local_timer_interrupts
     41.67 ± 29%   +1631.2%     721.33 ± 14%  interrupts.CPU60.RES:Rescheduling_interrupts
    776.50 ±  6%      -9.8%     700.33        interrupts.CPU61.CAL:Function_call_interrupts
     73441 ±  3%    +184.2%     208706        interrupts.CPU61.LOC:Local_timer_interrupts
     49.17 ± 21%   +1419.3%     747.00 ± 12%  interrupts.CPU61.RES:Rescheduling_interrupts
     73469 ±  3%    +183.9%     208591        interrupts.CPU62.LOC:Local_timer_interrupts
     35.33 ± 11%   +2475.5%     910.00 ± 19%  interrupts.CPU62.RES:Rescheduling_interrupts
     73452 ±  3%    +184.1%     208685        interrupts.CPU63.LOC:Local_timer_interrupts
     35.50 ± 15%   +2874.6%       1056 ± 48%  interrupts.CPU63.RES:Rescheduling_interrupts
    775.50 ±  7%     -15.9%     652.00 ±  8%  interrupts.CPU64.CAL:Function_call_interrupts
     73300 ±  3%    +184.7%     208697        interrupts.CPU64.LOC:Local_timer_interrupts
      3615 ±  3%     -22.1%       2816 ± 26%  interrupts.CPU64.NMI:Non-maskable_interrupts
      3615 ±  3%     -22.1%       2816 ± 26%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
     32.17 ± 18%   +2197.4%     739.00 ± 12%  interrupts.CPU64.RES:Rescheduling_interrupts
    757.50 ±  6%     -10.5%     677.67 ±  2%  interrupts.CPU65.CAL:Function_call_interrupts
     73428 ±  3%    +183.7%     208309        interrupts.CPU65.LOC:Local_timer_interrupts
     36.67 ± 12%   +1975.5%     761.00 ± 14%  interrupts.CPU65.RES:Rescheduling_interrupts
    785.33 ±  2%      -8.7%     717.00 ±  5%  interrupts.CPU66.CAL:Function_call_interrupts
     73730 ±  2%    +181.2%     207350        interrupts.CPU66.LOC:Local_timer_interrupts
     43.50 ± 28%   +1331.4%     622.67        interrupts.CPU66.RES:Rescheduling_interrupts
     73809 ±  2%    +181.3%     207662        interrupts.CPU67.LOC:Local_timer_interrupts
     41.33 ±  9%   +1335.5%     593.33 ±  3%  interrupts.CPU67.RES:Rescheduling_interrupts
     73812 ±  2%    +181.0%     207425        interrupts.CPU68.LOC:Local_timer_interrupts
     49.67 ± 23%   +1169.8%     630.67 ±  2%  interrupts.CPU68.RES:Rescheduling_interrupts
     73800 ±  2%    +180.8%     207202        interrupts.CPU69.LOC:Local_timer_interrupts
     56.67 ± 17%   +1054.7%     654.33 ±  7%  interrupts.CPU69.RES:Rescheduling_interrupts
    804.83 ±  9%     -11.7%     710.33 ±  3%  interrupts.CPU7.CAL:Function_call_interrupts
     73462 ±  3%    +184.0%     208602        interrupts.CPU7.LOC:Local_timer_interrupts
     61.83 ± 15%   +1312.4%     873.33 ± 13%  interrupts.CPU7.RES:Rescheduling_interrupts
     73697 ±  2%    +182.2%     207976        interrupts.CPU70.LOC:Local_timer_interrupts
     53.33 ± 16%   +1107.5%     644.00 ±  6%  interrupts.CPU70.RES:Rescheduling_interrupts
     73800 ±  2%    +181.8%     207968        interrupts.CPU71.LOC:Local_timer_interrupts
     57.00 ± 17%   +1008.2%     631.67 ±  4%  interrupts.CPU71.RES:Rescheduling_interrupts
     73794 ±  2%    +181.9%     208043        interrupts.CPU72.LOC:Local_timer_interrupts
     53.67 ±  7%   +1087.0%     637.00        interrupts.CPU72.RES:Rescheduling_interrupts
    778.67 ±  4%      -8.5%     712.33        interrupts.CPU73.CAL:Function_call_interrupts
     73791 ±  2%    +181.9%     208002        interrupts.CPU73.LOC:Local_timer_interrupts
     47.33 ± 25%   +1217.6%     623.67 ±  3%  interrupts.CPU73.RES:Rescheduling_interrupts
     73801 ±  2%    +181.8%     207979        interrupts.CPU74.LOC:Local_timer_interrupts
     48.67 ± 25%   +1391.1%     725.67 ±  5%  interrupts.CPU74.RES:Rescheduling_interrupts
     73788 ±  2%    +181.9%     207998        interrupts.CPU75.LOC:Local_timer_interrupts
     50.00 ± 20%   +1206.0%     653.00 ±  4%  interrupts.CPU75.RES:Rescheduling_interrupts
     73802 ±  2%    +181.8%     207982        interrupts.CPU76.LOC:Local_timer_interrupts
     37.00 ± 19%   +1612.6%     633.67 ±  4%  interrupts.CPU76.RES:Rescheduling_interrupts
     73796 ±  2%    +181.6%     207787        interrupts.CPU77.LOC:Local_timer_interrupts
      3618 ±  4%     -21.2%       2851 ± 27%  interrupts.CPU77.NMI:Non-maskable_interrupts
      3618 ±  4%     -21.2%       2851 ± 27%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
     35.67 ± 13%   +1776.6%     669.33        interrupts.CPU77.RES:Rescheduling_interrupts
     73940 ±  2%    +181.4%     208067        interrupts.CPU78.LOC:Local_timer_interrupts
     32.50 ± 14%   +1878.5%     643.00 ±  4%  interrupts.CPU78.RES:Rescheduling_interrupts
     73833 ±  2%    +181.8%     208042        interrupts.CPU79.LOC:Local_timer_interrupts
     38.17 ± 14%   +1684.3%     681.00 ±  7%  interrupts.CPU79.RES:Rescheduling_interrupts
     73467 ±  3%    +184.1%     208701        interrupts.CPU8.LOC:Local_timer_interrupts
     62.00 ± 11%   +1272.0%     850.67 ± 16%  interrupts.CPU8.RES:Rescheduling_interrupts
     73749 ±  2%    +182.2%     208088        interrupts.CPU80.LOC:Local_timer_interrupts
     36.17 ± 17%   +1940.6%     738.00 ± 16%  interrupts.CPU80.RES:Rescheduling_interrupts
     73963 ±  2%    +181.3%     208078        interrupts.CPU81.LOC:Local_timer_interrupts
     40.17 ± 12%   +1511.6%     647.33 ±  5%  interrupts.CPU81.RES:Rescheduling_interrupts
     73804 ±  2%    +181.2%     207506        interrupts.CPU82.LOC:Local_timer_interrupts
     37.33 ±  6%   +1520.5%     605.00        interrupts.CPU82.RES:Rescheduling_interrupts
    813.00 ±  7%     -14.3%     697.00 ±  5%  interrupts.CPU83.CAL:Function_call_interrupts
     73808 ±  2%    +181.6%     207865        interrupts.CPU83.LOC:Local_timer_interrupts
     40.33 ± 11%   +1462.8%     630.33 ±  3%  interrupts.CPU83.RES:Rescheduling_interrupts
     73796 ±  2%    +181.7%     207859        interrupts.CPU84.LOC:Local_timer_interrupts
     50.00 ± 33%   +1256.7%     678.33 ±  3%  interrupts.CPU84.RES:Rescheduling_interrupts
     73809 ±  2%    +181.6%     207870        interrupts.CPU85.LOC:Local_timer_interrupts
     49.67 ± 27%   +1279.2%     685.00 ± 11%  interrupts.CPU85.RES:Rescheduling_interrupts
     73802 ±  2%    +181.5%     207718        interrupts.CPU86.LOC:Local_timer_interrupts
     43.50 ± 37%   +1347.5%     629.67 ±  7%  interrupts.CPU86.RES:Rescheduling_interrupts
     73830 ±  2%    +181.0%     207499        interrupts.CPU87.LOC:Local_timer_interrupts
     64.00 ± 34%    +945.3%     669.00 ±  4%  interrupts.CPU87.RES:Rescheduling_interrupts
     73622 ±  3%    +183.3%     208560        interrupts.CPU9.LOC:Local_timer_interrupts
     66.00 ± 11%   +1163.6%     834.00 ± 20%  interrupts.CPU9.RES:Rescheduling_interrupts
   6477736 ±  3%    +182.8%   18319545        interrupts.LOC:Local_timer_interrupts
      4806 ±  3%   +1290.6%      66843 ±  8%  interrupts.RES:Rescheduling_interrupts
     58.13 ±  3%     -37.9       20.24 ±  2%  perf-profile.calltrace.cycles-pp.write
     54.11 ±  3%     -34.8       19.27 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     53.10 ±  3%     -34.1       18.97 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     52.57 ±  3%     -33.8       18.80 ±  2%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     51.50 ±  3%     -33.0       18.46 ±  2%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     47.61 ±  3%     -30.5       17.14 ±  3%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     45.83 ±  3%     -29.2       16.61 ±  3%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     39.08 ±  3%     -26.2       12.91        perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write
     38.78 ±  3%     -26.0       12.81        perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write.vfs_write
     29.23 ±  8%     -24.3        4.89 ± 13%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     28.92 ±  8%     -24.1        4.82 ± 12%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     28.92 ±  8%     -24.1        4.82 ± 12%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     28.92 ±  8%     -24.1        4.82 ± 12%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     28.62 ±  8%     -24.1        4.55 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     28.66 ±  8%     -24.0        4.62 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     27.97 ±  8%     -24.0        3.96 ± 15%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     30.86 ±  3%     -21.7        9.16        perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write
     13.41 ±  3%      -9.5        3.94        perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_write
      9.65 ±  3%      -7.1        2.60        perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_write
      8.48 ±  3%      -6.1        2.39 ±  2%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      8.20 ±  3%      -5.9        2.30 ±  2%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      5.69 ±  3%      -4.3        1.43        perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      3.86 ±  4%      -2.8        1.10        perf-profile.calltrace.cycles-pp.__close
      3.85 ±  4%      -2.7        1.10        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close
      3.83 ±  4%      -2.7        1.10        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__close
      3.83 ±  4%      -2.7        1.10        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__close
      3.83 ±  4%      -2.7        1.09        perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__close
      3.82 ±  4%      -2.7        1.09        perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      3.70 ±  4%      -2.6        1.05        perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      3.67 ±  4%      -2.6        1.04        perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.exit_to_user_mode_prepare
      3.60 ±  4%      -2.6        1.03        perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      3.57 ±  4%      -2.5        1.02        perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.__dentry_kill.dput.__fput
      3.59 ±  3%      -2.5        1.10        perf-profile.calltrace.cycles-pp.llseek
      5.08 ±  3%      -2.4        2.70 ±  3%  perf-profile.calltrace.cycles-pp.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write
      3.63 ± 19%      -2.4        1.27 ±  2%  perf-profile.calltrace.cycles-pp.xfs_file_write_checks.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write
      3.65 ±  3%      -2.3        1.34        perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_write
      3.08 ±  3%      -2.2        0.88 ±  2%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      2.69 ±  3%      -2.0        0.70        perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.69 ±  4%      -1.9        0.79        perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.97 ±  3%      -1.8        1.15        perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.74 ±  3%      -1.7        1.08        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply
      1.94 ±  2%      -1.4        0.57 ±  3%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.94 ±  2%      -1.4        0.59        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.llseek
      0.73 ±  2%      -0.1        0.59 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.5        0.53 ±  2%  perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      0.00            +0.6        0.62        perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      0.00            +0.9        0.85 ±  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.__percpu_counter_compare.xfs_mod_fdblocks
      0.00            +0.9        0.86 ±  9%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.__percpu_counter_compare.xfs_mod_fdblocks.xfs_bmapi_reserve_delalloc
      0.00            +0.9        0.92 ±  8%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__percpu_counter_compare.xfs_mod_fdblocks.xfs_bmapi_reserve_delalloc.xfs_buffered_write_iomap_begin
      0.00            +0.9        0.93 ±  8%  perf-profile.calltrace.cycles-pp.__percpu_counter_compare.xfs_mod_fdblocks.xfs_bmapi_reserve_delalloc.xfs_buffered_write_iomap_begin.iomap_apply
      0.00            +1.0        0.95 ±  8%  perf-profile.calltrace.cycles-pp.xfs_mod_fdblocks.xfs_bmapi_reserve_delalloc.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write
      0.00            +1.0        1.01 ±  7%  perf-profile.calltrace.cycles-pp.xfs_bmapi_reserve_delalloc.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_write
      0.58 ±  7%      +1.2        1.82 ±  2%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.56 ±  7%      +1.3        1.82 ±  2%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.3        1.27 ±  4%  perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_submit_bio.submit_bio_noacct.flush_bio_list.raid1_unplug
      0.00            +1.3        1.29 ±  4%  perf-profile.calltrace.cycles-pp.submit_bio_noacct.flush_bio_list.raid1_unplug.blk_flush_plug_list.blk_finish_plug
      0.00            +1.3        1.29 ±  4%  perf-profile.calltrace.cycles-pp.brd_submit_bio.submit_bio_noacct.flush_bio_list.raid1_unplug.blk_flush_plug_list
      0.00            +1.3        1.30 ±  4%  perf-profile.calltrace.cycles-pp.flush_bio_list.raid1_unplug.blk_flush_plug_list.blk_finish_plug.wb_writeback
      0.00            +1.3        1.30 ±  4%  perf-profile.calltrace.cycles-pp.blk_finish_plug.wb_writeback.wb_workfn.process_one_work.worker_thread
      0.00            +1.3        1.30 ±  4%  perf-profile.calltrace.cycles-pp.blk_flush_plug_list.blk_finish_plug.wb_writeback.wb_workfn.process_one_work
      0.00            +1.3        1.30 ±  4%  perf-profile.calltrace.cycles-pp.raid1_unplug.blk_flush_plug_list.blk_finish_plug.wb_writeback.wb_workfn
      0.59 ±  8%      +1.3        1.91 ±  4%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.59 ±  8%      +1.3        1.91 ±  4%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.00            +1.4        1.38 ± 35%  perf-profile.calltrace.cycles-pp.xfs_inode_walk.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write
      0.00            +1.4        1.38 ± 35%  perf-profile.calltrace.cycles-pp.xfs_inode_walk_ag.xfs_inode_walk.xfs_file_buffered_write.new_sync_write.vfs_write
      0.00            +1.4        1.42 ±  3%  perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.4        1.42 ±  3%  perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
      1.99 ±  2%     +32.9       34.91        perf-profile.calltrace.cycles-pp.unlink
      1.97 ±  2%     +32.9       34.90        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.97 ±  2%     +32.9       34.90        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
      1.96 ±  2%     +32.9       34.90        perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.23 ±  3%     +33.4       34.59        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.85 ±  4%     +33.5       35.34        perf-profile.calltrace.cycles-pp.creat64
      1.84 ±  4%     +33.5       35.33        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat64
      1.83 ±  4%     +33.5       35.33        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      1.83 ±  4%     +33.5       35.33        perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      1.83 ±  4%     +33.5       35.33        perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      1.80 ±  4%     +33.5       35.33        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.80 ±  4%     +33.5       35.33        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      0.74 ±  4%     +33.8       34.51        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.00           +33.9       33.85        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      0.00           +34.0       33.97        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     58.18 ±  3%     -37.7       20.52 ±  2%  perf-profile.children.cycles-pp.write
     52.61 ±  3%     -33.8       18.82 ±  2%  perf-profile.children.cycles-pp.ksys_write
     51.60 ±  3%     -33.1       18.49 ±  2%  perf-profile.children.cycles-pp.vfs_write
     47.65 ±  3%     -30.5       17.16 ±  3%  perf-profile.children.cycles-pp.new_sync_write
     45.88 ±  3%     -29.3       16.62 ±  3%  perf-profile.children.cycles-pp.xfs_file_buffered_write
     39.09 ±  3%     -26.2       12.92        perf-profile.children.cycles-pp.iomap_file_buffered_write
     38.83 ±  3%     -26.0       12.82        perf-profile.children.cycles-pp.iomap_apply
     29.23 ±  8%     -24.3        4.89 ± 13%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     29.23 ±  8%     -24.3        4.89 ± 13%  perf-profile.children.cycles-pp.cpu_startup_entry
     29.23 ±  8%     -24.3        4.89 ± 13%  perf-profile.children.cycles-pp.do_idle
     28.96 ±  8%     -24.3        4.68 ± 13%  perf-profile.children.cycles-pp.cpuidle_enter_state
     28.96 ±  8%     -24.3        4.68 ± 13%  perf-profile.children.cycles-pp.cpuidle_enter
     28.26 ±  8%     -24.2        4.02 ± 15%  perf-profile.children.cycles-pp.intel_idle
     28.92 ±  8%     -24.1        4.82 ± 12%  perf-profile.children.cycles-pp.start_secondary
     30.98 ±  3%     -21.8        9.18        perf-profile.children.cycles-pp.iomap_write_actor
     13.43 ±  3%      -9.5        3.95        perf-profile.children.cycles-pp.iomap_write_begin
      9.73 ±  3%      -7.1        2.62        perf-profile.children.cycles-pp.iomap_write_end
      8.39 ±  3%      -6.3        2.13        perf-profile.children.cycles-pp.iomap_set_range_uptodate
      8.53 ±  3%      -6.1        2.41 ±  2%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      8.31 ±  3%      -6.0        2.33        perf-profile.children.cycles-pp.pagecache_get_page
      4.71 ±  3%      -3.4        1.36        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      4.29 ±  4%      -3.0        1.25        perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.04 ±  3%      -2.8        1.24        perf-profile.children.cycles-pp.llseek
      3.86 ±  4%      -2.8        1.10        perf-profile.children.cycles-pp.__close
      3.83 ±  4%      -2.7        1.10        perf-profile.children.cycles-pp.task_work_run
      3.82 ±  4%      -2.7        1.09        perf-profile.children.cycles-pp.__fput
      3.71 ±  4%      -2.7        1.06        perf-profile.children.cycles-pp.dput
      3.67 ±  4%      -2.6        1.04        perf-profile.children.cycles-pp.__dentry_kill
      3.60 ±  4%      -2.6        1.03        perf-profile.children.cycles-pp.evict
      3.57 ±  4%      -2.5        1.03        perf-profile.children.cycles-pp.truncate_inode_pages_range
      5.17 ±  3%      -2.4        2.72 ±  3%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      3.72 ± 19%      -2.4        1.29 ±  2%  perf-profile.children.cycles-pp.xfs_file_write_checks
      3.66 ±  3%      -2.3        1.35        perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      3.09 ±  3%      -2.2        0.88 ±  2%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      2.73 ±  4%      -1.9        0.80        perf-profile.children.cycles-pp.iomap_set_page_dirty
      3.01 ±  3%      -1.9        1.16        perf-profile.children.cycles-pp.copyin
      2.90 ±  3%      -1.8        1.13        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      2.32 ±  3%      -1.6        0.68 ±  2%  perf-profile.children.cycles-pp.__entry_text_start
      2.23 ±  2%      -1.5        0.69 ±  2%  perf-profile.children.cycles-pp.down_write
      1.96 ±  2%      -1.4        0.57 ±  4%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      1.99 ±  3%      -1.4        0.63 ±  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.72 ±  4%      -1.3        0.44 ±  3%  perf-profile.children.cycles-pp.release_pages
      1.67 ±  4%      -1.2        0.44 ±  2%  perf-profile.children.cycles-pp.__pagevec_release
      1.61 ± 13%      -1.1        0.47 ±  3%  perf-profile.children.cycles-pp.file_update_time
      1.50 ±  3%      -1.0        0.50 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      1.51 ±  4%      -1.0        0.53        perf-profile.children.cycles-pp.___might_sleep
      1.35 ±  2%      -0.9        0.41 ±  2%  perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      1.24 ±  3%      -0.9        0.37 ±  3%  perf-profile.children.cycles-pp.xas_load
      1.22 ±  3%      -0.8        0.37 ±  4%  perf-profile.children.cycles-pp.__fdget_pos
      1.17 ±  3%      -0.8        0.35        perf-profile.children.cycles-pp.ksys_lseek
      1.09 ±  4%      -0.8        0.29        perf-profile.children.cycles-pp.lru_cache_add
      1.16 ±  3%      -0.8        0.37 ±  3%  perf-profile.children.cycles-pp.common_file_perm
      1.14 ±  3%      -0.8        0.38 ±  2%  perf-profile.children.cycles-pp.__set_page_dirty
      1.16 ± 12%      -0.8        0.40 ±  6%  perf-profile.children.cycles-pp.__xfs_trans_commit
      1.14 ± 12%      -0.8        0.39 ±  6%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      1.02 ±  4%      -0.8        0.27        perf-profile.children.cycles-pp.__pagevec_lru_add
      1.01 ±  3%      -0.7        0.29 ±  4%  perf-profile.children.cycles-pp.__get_user_nocheck_1
      1.01 ± 19%      -0.7        0.30 ±  7%  perf-profile.children.cycles-pp.xfs_file_write_iter
      1.06 ±  3%      -0.7        0.35 ±  4%  perf-profile.children.cycles-pp.up_write
      1.00 ±  3%      -0.7        0.31 ±  5%  perf-profile.children.cycles-pp.__fget_light
      0.98 ±  3%      -0.7        0.30 ±  5%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.94 ±  4%      -0.7        0.26 ±  4%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      1.17 ±  3%      -0.7        0.49 ±  2%  perf-profile.children.cycles-pp.memset_erms
      0.88 ±  6%      -0.6        0.27 ± 14%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.77 ± 27%      -0.6        0.18 ± 10%  perf-profile.children.cycles-pp.xfs_vn_update_time
      0.80 ±  4%      -0.6        0.23 ±  3%  perf-profile.children.cycles-pp.disk_rw
      0.77 ±  3%      -0.6        0.22 ±  7%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.75 ±  3%      -0.6        0.20 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_charge
      0.76 ±  4%      -0.5        0.23 ±  2%  perf-profile.children.cycles-pp.truncate_cleanup_page
      0.80 ±  4%      -0.5        0.27 ±  3%  perf-profile.children.cycles-pp.delete_from_page_cache_batch
      0.75 ±  4%      -0.5        0.25 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      0.73 ±  3%      -0.5        0.25 ±  3%  perf-profile.children.cycles-pp.account_page_dirtied
      0.67 ±  4%      -0.5        0.19        perf-profile.children.cycles-pp.__cond_resched
      0.73 ± 28%      -0.5        0.26 ±  3%  perf-profile.children.cycles-pp.xfs_break_layouts
      0.65 ±  5%      -0.5        0.19        perf-profile.children.cycles-pp.__cancel_dirty_page
      0.68 ± 10%      -0.5        0.22 ±  7%  perf-profile.children.cycles-pp.xlog_cil_insert_items
      0.62 ±  4%      -0.4        0.22 ±  5%  perf-profile.children.cycles-pp.xas_store
      0.54 ±  4%      -0.4        0.13 ±  3%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.45 ±  9%      -0.4        0.07 ± 11%  perf-profile.children.cycles-pp.lock_page_memcg
      0.54 ±  4%      -0.4        0.16 ±  2%  perf-profile.children.cycles-pp.current_time
      0.45 ±  4%      -0.4        0.07 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.64 ±  4%      -0.4        0.27 ±  3%  perf-profile.children.cycles-pp.xfs_remove
      0.64 ±  3%      -0.4        0.28 ±  2%  perf-profile.children.cycles-pp.vfs_unlink
      0.47 ±  3%      -0.4        0.10 ± 12%  perf-profile.children.cycles-pp.rmqueue
      0.49 ±  5%      -0.4        0.14        perf-profile.children.cycles-pp.account_page_cleaned
      0.43 ±  6%      -0.3        0.08        perf-profile.children.cycles-pp.lock_page_lruvec_irqsave
      0.47 ±  4%      -0.3        0.12 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.50 ±  3%      -0.3        0.16        perf-profile.children.cycles-pp.unlock_page
      0.50 ±  3%      -0.3        0.17 ±  4%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.40 ±  6%      -0.3        0.08 ±  5%  perf-profile.children.cycles-pp.free_unref_page_list
      0.46 ±  5%      -0.3        0.15 ±  3%  perf-profile.children.cycles-pp.page_mapping
      0.47 ±  5%      -0.3        0.18 ±  2%  perf-profile.children.cycles-pp.generic_write_checks
      0.41 ±  6%      -0.3        0.12 ±  6%  perf-profile.children.cycles-pp.xas_start
      0.38 ±  4%      -0.3        0.10 ±  4%  perf-profile.children.cycles-pp.uncharge_batch
      0.49 ± 13%      -0.3        0.22 ±  3%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      0.37 ±  3%      -0.3        0.10 ±  8%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.31 ± 23%      -0.3        0.05 ± 81%  perf-profile.children.cycles-pp.start_kernel
      0.35 ±  4%      -0.3        0.10 ±  4%  perf-profile.children.cycles-pp.iov_iter_advance
      0.41 ±  3%      -0.3        0.16 ±  3%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.34 ±  5%      -0.2        0.09        perf-profile.children.cycles-pp.rcu_all_qs
      0.34 ±  4%      -0.2        0.09 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.37 ±  6%      -0.2        0.13 ±  3%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.35 ±  4%      -0.2        0.10 ±  9%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.32 ±  3%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.32 ±  3%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.page_counter_cancel
      0.51 ±  5%      -0.2        0.28 ± 13%  perf-profile.children.cycles-pp.xfs_inodegc_inactivate
      0.30 ±  4%      -0.2        0.09 ±  9%  perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.49 ±  8%      -0.2        0.28 ± 13%  perf-profile.children.cycles-pp.xfs_inactive
      0.32 ± 32%      -0.2        0.11 ± 11%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.30 ±  5%      -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.27 ±  3%      -0.2        0.08 ± 12%  perf-profile.children.cycles-pp.find_lock_entries
      0.26 ±  9%      -0.2        0.07 ±  6%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.23 ± 11%      -0.2        0.06 ±  8%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.25 ± 50%      -0.2        0.07 ± 11%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.24 ±  4%      -0.2        0.07 ± 14%  perf-profile.children.cycles-pp.try_charge
      0.23 ±  4%      -0.2        0.06 ± 16%  perf-profile.children.cycles-pp.file_modified
      0.25 ±  4%      -0.2        0.08        perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.23 ±  5%      -0.2        0.06 ± 14%  perf-profile.children.cycles-pp.xfs_buf_read_map
      0.24 ±  2%      -0.2        0.08 ±  6%  perf-profile.children.cycles-pp.aa_file_perm
      0.29 ± 17%      -0.2        0.13 ±  9%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      0.22 ±  4%      -0.2        0.06 ±  8%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.23 ±  5%      -0.2        0.07 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.21 ±  6%      -0.2        0.05 ±  8%  perf-profile.children.cycles-pp.xfs_buf_find
      0.36 ± 14%      -0.1        0.21 ±  7%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.34 ± 15%      -0.1        0.20 ± 10%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.21 ±  5%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.xfs_get_extsz_hint
      0.40 ± 11%      -0.1        0.26 ± 14%  perf-profile.children.cycles-pp.xfs_inactive_ifree
      0.20 ±  7%      -0.1        0.06 ± 13%  perf-profile.children.cycles-pp.node_dirty_ok
      0.22 ± 17%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.rcu_core
      0.23 ±  2%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.__x64_sys_write
      0.23 ±  5%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.generic_write_check_limits
      0.20 ±  4%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.timestamp_truncate
      0.20 ± 17%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.rcu_do_batch
      0.18 ±  3%      -0.1        0.06 ±  8%  perf-profile.children.cycles-pp.rw_verify_area
      0.17 ±  4%      -0.1        0.05        perf-profile.children.cycles-pp.page_counter_try_charge
      0.17 ±  3%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.iomap_page_create
      0.19 ± 15%      -0.1        0.08        perf-profile.children.cycles-pp.kmem_cache_free
      0.14 ±  5%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.xas_init_marks
      0.13 ±  7%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.17 ±  6%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.xfs_dir3_data_check
      0.17 ±  6%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.__xfs_dir3_data_check
      0.14 ±  8%      -0.1        0.05        perf-profile.children.cycles-pp.kmem_cache_alloc
      0.15 ±  7%      -0.1        0.06        perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.16 ±  4%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.14 ±  9%      -0.1        0.05 ± 70%  perf-profile.children.cycles-pp.try_to_wake_up
      0.13 ± 13%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.xfs_dir2_node_addname
      0.14 ±  7%      -0.1        0.06 ± 16%  perf-profile.children.cycles-pp.__x64_sys_lseek
      0.15 ±  8%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.12 ±  6%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__xa_set_mark
      0.12 ±  6%      -0.1        0.05        perf-profile.children.cycles-pp.xfs_dir2_leafn_remove
      0.30 ±  7%      -0.1        0.24 ± 14%  perf-profile.children.cycles-pp.xfs_ifree
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.iomap_writepage_map
      0.10 ± 10%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.menu_select
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.iomap_finish_ioends
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.iomap_finish_ioend
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.xfs_verify_agino
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.brd_lookup_page
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.xfs_end_io
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.xfs_end_ioend
      0.06 ±  6%      +0.1        0.16 ± 23%  perf-profile.children.cycles-pp.schedule_timeout
      0.00            +0.1        0.11 ± 26%  perf-profile.children.cycles-pp.down_write_trylock
      0.00            +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.iomap_writepages
      0.00            +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.write_cache_pages
      0.05 ± 46%      +0.1        0.17        perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      0.00            +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.writeback_sb_inodes
      0.00            +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.__writeback_single_inode
      0.00            +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.do_writepages
      0.00            +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.xfs_vm_writepages
      1.18 ±  2%      +0.1        1.30        perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.04 ± 44%      +0.1        0.16 ± 17%  perf-profile.children.cycles-pp.load_balance
      0.00            +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.find_busiest_group
      0.03 ± 99%      +0.1        0.15 ± 18%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.brd_insert_page
      0.00            +0.1        0.15 ± 17%  perf-profile.children.cycles-pp.newidle_balance
      0.00            +0.1        0.15 ± 39%  perf-profile.children.cycles-pp._atomic_dec_and_lock
      0.00            +0.2        0.15 ± 34%  perf-profile.children.cycles-pp.xfs_can_free_eofblocks
      0.09 ±  5%      +0.2        0.24 ±  3%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.00            +0.2        0.16 ± 41%  perf-profile.children.cycles-pp.iput
      0.07 ± 45%      +0.2        0.23 ±  2%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      0.34 ± 10%      +0.2        0.53 ±  2%  perf-profile.children.cycles-pp.xfs_dir_ialloc
      0.08 ± 44%      +0.2        0.27 ±  3%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      0.00            +0.2        0.25 ± 48%  perf-profile.children.cycles-pp.igrab
      0.15 ± 45%      +0.3        0.41 ±  3%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      0.21 ± 16%      +0.3        0.49 ±  3%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      0.35 ±  5%      +0.3        0.70        perf-profile.children.cycles-pp.xfs_check_agi_freecount
      0.67 ±  5%      +0.4        1.10 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.4        0.43 ± 34%  perf-profile.children.cycles-pp.xfs_inode_free_eofblocks
      0.00            +0.4        0.44 ± 40%  perf-profile.children.cycles-pp.xfs_inode_walk_ag_grab
      0.00            +0.6        0.57 ± 34%  perf-profile.children.cycles-pp.xfs_blockgc_scan_inode
      0.67 ±  9%      +0.6        1.28 ± 15%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.12 ±  7%      +0.9        1.01 ±  7%  perf-profile.children.cycles-pp.xfs_bmapi_reserve_delalloc
      0.07 ± 11%      +0.9        0.98 ±  8%  perf-profile.children.cycles-pp.__percpu_counter_compare
      0.05 ±  7%      +0.9        0.97 ±  8%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.07 ±  7%      +0.9        1.00 ±  7%  perf-profile.children.cycles-pp.xfs_mod_fdblocks
      0.52 ±  5%      +1.1        1.67 ± 31%  perf-profile.children.cycles-pp.xfs_inode_walk_ag
      0.44 ± 45%      +1.2        1.67 ± 31%  perf-profile.children.cycles-pp.xfs_inode_walk
      0.58 ±  7%      +1.2        1.82 ±  2%  perf-profile.children.cycles-pp.worker_thread
      0.56 ±  7%      +1.3        1.82 ±  2%  perf-profile.children.cycles-pp.process_one_work
      0.01 ±223%      +1.3        1.30 ±  4%  perf-profile.children.cycles-pp.blk_finish_plug
      0.01 ±223%      +1.3        1.30 ±  4%  perf-profile.children.cycles-pp.blk_flush_plug_list
      0.00            +1.3        1.30 ±  4%  perf-profile.children.cycles-pp.raid1_unplug
      0.01 ±223%      +1.3        1.32 ±  6%  perf-profile.children.cycles-pp.brd_do_bvec
      0.59 ±  8%      +1.3        1.91 ±  4%  perf-profile.children.cycles-pp.ret_from_fork
      0.59 ±  8%      +1.3        1.91 ±  4%  perf-profile.children.cycles-pp.kthread
      0.00            +1.3        1.34 ±  6%  perf-profile.children.cycles-pp.brd_submit_bio
      0.00            +1.3        1.35 ±  6%  perf-profile.children.cycles-pp.flush_bio_list
      0.01 ±223%      +1.3        1.36 ±  6%  perf-profile.children.cycles-pp.submit_bio_noacct
      0.01 ±223%      +1.4        1.42 ±  3%  perf-profile.children.cycles-pp.wb_workfn
      0.01 ±223%      +1.4        1.42 ±  3%  perf-profile.children.cycles-pp.wb_writeback
     63.92 ±  3%     +27.4       91.30        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     58.54 ±  3%     +31.2       89.74        perf-profile.children.cycles-pp.do_syscall_64
      1.99 ±  2%     +32.9       34.91        perf-profile.children.cycles-pp.unlink
      1.96 ±  2%     +32.9       34.90        perf-profile.children.cycles-pp.do_unlinkat
      1.91 ±  4%     +33.5       35.36        perf-profile.children.cycles-pp.do_sys_open
      1.91 ±  4%     +33.5       35.36        perf-profile.children.cycles-pp.do_sys_openat2
      1.86 ±  4%     +33.5       35.34        perf-profile.children.cycles-pp.creat64
      1.86 ±  4%     +33.5       35.35        perf-profile.children.cycles-pp.path_openat
      1.86 ±  4%     +33.5       35.35        perf-profile.children.cycles-pp.do_filp_open
      0.65 ±  6%     +67.2       67.84        perf-profile.children.cycles-pp.osq_lock
      1.97 ±  3%     +67.2       69.19        perf-profile.children.cycles-pp.rwsem_down_write_slowpath
     28.26 ±  8%     -24.2        4.02 ± 15%  perf-profile.self.cycles-pp.intel_idle
      8.29 ±  3%      -6.2        2.10        perf-profile.self.cycles-pp.iomap_set_range_uptodate
      2.54 ±  3%      -1.8        0.71        perf-profile.self.cycles-pp.pagecache_get_page
      2.84 ±  3%      -1.7        1.11        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.98 ±  3%      -1.3        0.63 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.71 ±  3%      -1.1        0.58        perf-profile.self.cycles-pp.iomap_apply
      1.54 ±  3%      -1.0        0.54 ±  5%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
      1.48 ±  4%      -1.0        0.52        perf-profile.self.cycles-pp.___might_sleep
      1.30 ±  3%      -0.9        0.40 ±  5%  perf-profile.self.cycles-pp.write
      1.15 ±  3%      -0.8        0.31        perf-profile.self.cycles-pp.down_write
      1.20 ±  3%      -0.8        0.41        perf-profile.self.cycles-pp.vfs_write
      1.09 ±  4%      -0.7        0.35 ±  3%  perf-profile.self.cycles-pp.iomap_write_actor
      1.00 ±  3%      -0.7        0.27        perf-profile.self.cycles-pp.__entry_text_start
      0.97 ±  4%      -0.7        0.28 ±  4%  perf-profile.self.cycles-pp.__get_user_nocheck_1
      1.01 ±  3%      -0.7        0.33 ±  4%  perf-profile.self.cycles-pp.up_write
      1.16 ±  3%      -0.7        0.49        perf-profile.self.cycles-pp.memset_erms
      0.96 ±  3%      -0.7        0.30 ±  4%  perf-profile.self.cycles-pp.__fget_light
      0.89 ±  3%      -0.6        0.26 ±  3%  perf-profile.self.cycles-pp.iomap_write_end
      0.92 ±  3%      -0.6        0.29        perf-profile.self.cycles-pp.common_file_perm
      0.85 ±  4%      -0.6        0.26 ±  5%  perf-profile.self.cycles-pp.xas_load
      0.80 ±  5%      -0.6        0.25 ± 14%  perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
      0.84 ± 14%      -0.5        0.29 ±  6%  perf-profile.self.cycles-pp.xfs_file_write_iter
      0.84 ±  3%      -0.5        0.30 ±  2%  perf-profile.self.cycles-pp.iomap_write_begin
      0.84 ± 14%      -0.5        0.31 ±  5%  perf-profile.self.cycles-pp.xfs_file_buffered_write
      0.71 ±  4%      -0.5        0.20 ±  2%  perf-profile.self.cycles-pp.disk_rw
      0.67 ±  4%      -0.5        0.19 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.63 ±  2%      -0.4        0.19 ±  2%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.65 ±  4%      -0.4        0.22 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      0.52 ±  4%      -0.4        0.13        perf-profile.self.cycles-pp.__mod_memcg_state
      0.58 ±  4%      -0.4        0.20 ±  2%  perf-profile.self.cycles-pp.llseek
      0.44 ±  8%      -0.4        0.07 ± 11%  perf-profile.self.cycles-pp.lock_page_memcg
      0.44 ±  4%      -0.4        0.07 ±  6%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.55 ±  3%      -0.4        0.19 ±  4%  perf-profile.self.cycles-pp.new_sync_write
      0.48 ±  3%      -0.3        0.14        perf-profile.self.cycles-pp.iomap_set_page_dirty
      0.51 ± 10%      -0.3        0.18 ±  2%  perf-profile.self.cycles-pp.xfs_iunlock
      0.48 ±  2%      -0.3        0.15 ±  3%  perf-profile.self.cycles-pp.unlock_page
      0.47 ±  3%      -0.3        0.16 ±  5%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.46 ±  4%      -0.3        0.15 ±  3%  perf-profile.self.cycles-pp.page_mapping
      0.47 ±  9%      -0.3        0.16 ±  5%  perf-profile.self.cycles-pp.xfs_ilock
      0.41 ±  5%      -0.3        0.13 ±  7%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.36 ±  3%      -0.3        0.10 ±  8%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.47 ± 15%      -0.3        0.21 ±  2%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.37 ±  4%      -0.3        0.11 ±  4%  perf-profile.self.cycles-pp.ksys_write
      0.37 ±  6%      -0.3        0.11 ±  7%  perf-profile.self.cycles-pp.xas_start
      0.35 ±  3%      -0.2        0.10 ±  8%  perf-profile.self.cycles-pp.__pagevec_lru_add
      0.34 ±  4%      -0.2        0.09 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.33 ±  4%      -0.2        0.09 ±  5%  perf-profile.self.cycles-pp.iov_iter_advance
      0.39 ± 10%      -0.2        0.15        perf-profile.self.cycles-pp.xfs_file_write_checks
      0.33 ±  3%      -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.__cond_resched
      0.34 ±  3%      -0.2        0.11 ±  4%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      0.33 ±  3%      -0.2        0.11 ±  7%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.32 ±  4%      -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.release_pages
      0.31 ±  4%      -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.26 ±  2%      -0.2        0.07        perf-profile.self.cycles-pp.page_counter_cancel
      0.29 ±  5%      -0.2        0.10 ±  8%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.32 ±  2%      -0.2        0.12 ±  3%  perf-profile.self.cycles-pp.file_update_time
      0.31 ±  5%      -0.2        0.12 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.24 ±  6%      -0.2        0.06 ±  7%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.26 ± 14%      -0.2        0.09 ±  5%  perf-profile.self.cycles-pp.xfs_break_layouts
      0.23 ±  7%      -0.2        0.06        perf-profile.self.cycles-pp.rcu_all_qs
      0.27 ±  3%      -0.2        0.10        perf-profile.self.cycles-pp.iomap_file_buffered_write
      0.23 ±  3%      -0.2        0.07 ±  7%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.25 ±  5%      -0.2        0.09        perf-profile.self.cycles-pp.xas_store
      0.21 ±  6%      -0.2        0.06 ±  8%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.24 ±  5%      -0.2        0.09 ±  5%  perf-profile.self.cycles-pp.generic_write_checks
      0.22 ±  4%      -0.2        0.07 ±  6%  perf-profile.self.cycles-pp.__fdget_pos
      0.21 ±  5%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.current_time
      0.18 ±  5%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.ksys_lseek
      0.20 ±  6%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.xfs_get_extsz_hint
      0.20 ±  4%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.20 ±  4%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.find_lock_entries
      0.20 ±  4%      -0.1        0.06        perf-profile.self.cycles-pp.aa_file_perm
      0.20 ±  4%      -0.1        0.06        perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.21 ±  3%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.__x64_sys_write
      0.21 ±  7%      -0.1        0.09        perf-profile.self.cycles-pp.generic_write_check_limits
      0.18 ±  7%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.security_file_permission
      0.17 ±  5%      -0.1        0.06 ±  8%  perf-profile.self.cycles-pp.timestamp_truncate
      0.16 ±  2%      -0.1        0.05        perf-profile.self.cycles-pp.iomap_page_create
      0.16 ±  7%      -0.1        0.05        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.14 ± 12%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.copyin
      0.15 ±  5%      -0.1        0.05        perf-profile.self.cycles-pp.rw_verify_area
      0.12 ± 16%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.kmem_cache_free
      0.13 ±  5%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.14 ±  5%      -0.1        0.06        perf-profile.self.cycles-pp.apparmor_file_permission
      0.12 ±  8%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__x64_sys_lseek
      0.07 ±  8%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.xfs_btree_check_sblock
      0.01 ±223%      +0.1        0.06        perf-profile.self.cycles-pp.__percpu_counter_sum
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.xfs_btree_get_rec
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.xfs_inobt_get_rec
      0.06 ± 13%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.xfs_btree_increment
      0.00            +0.1        0.08 ± 34%  perf-profile.self.cycles-pp.igrab
      0.00            +0.1        0.09 ± 22%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.10 ± 31%  perf-profile.self.cycles-pp.down_write_trylock
      0.04 ± 71%      +0.1        0.14 ±  3%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      1.16 ±  2%      +0.1        1.29        perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.00            +0.1        0.14 ± 38%  perf-profile.self.cycles-pp._atomic_dec_and_lock
      0.67 ± 10%      +0.6        1.27 ± 15%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.00            +1.1        1.10 ±  9%  perf-profile.self.cycles-pp.brd_do_bvec
      0.65 ±  7%     +66.8       67.41        perf-profile.self.cycles-pp.osq_lock


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  600000 +------------------------------------------------------------------+   
         |                                                                  |   
  500000 |-+                                                         .++++.+|   
         |+.++++.+++.++++.+++.++++.+++.+++.++++.+++.++++.+++.++++.+++       |   
         |                                                                  |   
  400000 |-+                                                                |   
         |                                                                  |   
  300000 |-+          O                O O    O  O                          |   
         |O OOOO OOO   OO OOO OOOO OOO  O  OOO  O O OO                      |   
  200000 |-+         O                                                      |   
         |                                               O O O              |   
         |                                                                  |   
  100000 |-+                                                                |   
         |                                                                  |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               aim7.time.system_time                            
                                                                                
  5000 +--------------------------------------------------------------------+   
       |                                               O O                  |   
  4500 |-+                                                                  |   
  4000 |-+                                           O  O  OO               |   
       |                                                                    |   
  3500 |-+                                                                  |   
  3000 |-+OO       O  O    O  O                    O                        |   
       |O   O OOO O O  OO O O  OO OOO OOO OOOO OOO  O                       |   
  2500 |-+                                                                  |   
  2000 |-+                                                                  |   
       |                                                                    |   
  1500 |-+                                                                  |   
  1000 |-+                                                                  |   
       |+.+++.+++.+++.+++.+++.+++.+++.+++.++++.+++.+++.+++.+++.+++.+++.+++.+|   
   500 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  110 +---------------------------------------------------------------------+   
      |                                              OO OO O                |   
  100 |-+                                                   O               |   
   90 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
      |  O        O              O       O                                  |   
   70 |O+ OO  O  O   OO OOO OO  O     O       O  OO                         |   
      |      O O   O          O   O OO  O O OO  O   O                       |   
   60 |-+                                                                   |   
   50 |-+                                                                   |   
      |                                                                     |   
   40 |-+                                                                   |   
      |+.+++.+++.+++.++.+++.+++.+++.+++.+++.+++.+++.+++.++.+++.+++.+++.+++.+|   
   30 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  110 +---------------------------------------------------------------------+   
      |                                              OO OO O                |   
  100 |-+                                                   O               |   
   90 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
      |  O        O              O       O                                  |   
   70 |O+ OO  O  O   OO OOO OO  O     O       O  OO                         |   
      |      O O   O          O   O OO  O O OO  O   O                       |   
   60 |-+                                                                   |   
   50 |-+                                                                   |   
      |                                                                     |   
   40 |-+                                                                   |   
      |+.+++.+++.+++.++.+++.+++.+++.+++.+++.+++.+++.+++.++.+++.+++.+++.+++.+|   
   30 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       aim7.time.involuntary_context_switches                   
                                                                                
  180000 +------------------------------------------------------------------+   
         |                                                O                 |   
  160000 |-+                                           O                    |   
  140000 |-+                                                                |   
         |                                                                  |   
  120000 |-+                                               O                |   
  100000 |-+                                             O   O              |   
         |                                            O                     |   
   80000 |-+                                                                |   
   60000 |-+                                                                |   
         |                                                                  |   
   40000 |-+                                                                |   
   20000 |-+                                                                |   
         |O OOOO OOO OOOO OOO OOOO OOO OOO OOOO OOO OO                      |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             aim7.time.file_system_outputs                      
                                                                                
  6.9e+08 +-----------------------------------------------------------------+   
          |          O                                O  O  OO              |   
  6.8e+08 |-+                                             O                 |   
  6.7e+08 |-+OO            O    O                   O                       |   
          |O   OO O O  OO    O   O O  O O  O  O O  O                        |   
  6.6e+08 |-+               O O   O    O       O     O  O                   |   
          |        O     O           O    O O    O                          |   
  6.5e+08 |-+                                                               |   
          |                                                                 |   
  6.4e+08 |-+                                                               |   
  6.3e+08 |-+                                                               |   
          |                                                                 |   
  6.2e+08 |-+                                                               |   
          |+.++++.++++.+++.++++.++++.++++.+++.++++.++++.+++.++++.++++.++++.+|   
  6.1e+08 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.12.0-rc2-00023-g301157ab53d2" of type "text/plain" (172899 bytes)

View attachment "job-script" of type "text/plain" (8094 bytes)

View attachment "job.yaml" of type "text/plain" (5568 bytes)

View attachment "reproduce" of type "text/plain" (1026 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ