lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Sun, 25 Jul 2021 23:38:35 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Dave Chinner <dchinner@...hat.com>
Cc:     "Darrick J. Wong" <djwong@...nel.org>,
        Chandan Babu R <chandanrlinux@...il.com>,
        Allison Henderson <allison.henderson@...cle.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com
Subject: [xfs]  eef983ffea:  stress-ng.fallocate.ops_per_sec -15.2% regression



Greeting,

FYI, we noticed a -15.2% regression of stress-ng.fallocate.ops_per_sec due to commit:


commit: eef983ffeae7a1cdde8c3338155ae2dd74b8621b ("xfs: journal IO cache flush reductions")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: stress-ng
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	nr_threads: 10%
	disk: 1HDD
	testtime: 60s
	fs: xfs
	class: filesystem
	test: fallocate
	cpufreq_governor: performance
	ucode: 0x5003006


In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.hdd_xfs_MWUM_72_bufferedio.works/sec 27.5% improvement           |
| test machine     | 272 threads 2 sockets Intel(R) Xeon Phi(TM) CPU 7255 @ 1.10GHz with 112G memory |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | directio=bufferedio                                                             |
|                  | disk=1HDD                                                                       |
|                  | fstype=xfs                                                                      |
|                  | media=hdd                                                                       |
|                  | test=MWUM                                                                       |
|                  | ucode=0x11                                                                      |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.hdd_xfs_MWUL_72_bufferedio.works/sec 17.4% improvement           |
| test machine     | 272 threads 2 sockets Intel(R) Xeon Phi(TM) CPU 7255 @ 1.10GHz with 112G memory |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | directio=bufferedio                                                             |
|                  | disk=1HDD                                                                       |
|                  | fstype=xfs                                                                      |
|                  | media=hdd                                                                       |
|                  | test=MWUL                                                                       |
|                  | ucode=0x11                                                                      |
+------------------+---------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  filesystem/gcc-9/performance/1HDD/xfs/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/fallocate/stress-ng/60s/0x5003006

commit: 
  3468bb1ca6 ("xfs: remove need_start_rec parameter from xlog_write()")
  eef983ffea ("xfs: journal IO cache flush reductions")

3468bb1ca6e88407 eef983ffeae7a1cdde8c3338155 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    538.00           -15.3%     455.83        stress-ng.fallocate.ops
      8.96           -15.2%       7.60        stress-ng.fallocate.ops_per_sec
      5262           +63.9%       8626 ±  4%  stress-ng.time.involuntary_context_switches
    102363           -19.9%      82031        stress-ng.time.voluntary_context_switches
    870.67 ± 32%     -31.1%     599.83 ± 24%  interrupts.CPU16.CAL:Function_call_interrupts
      1.59 ±  3%     +44.8%       2.30        iostat.cpu.iowait
      1.63 ±  3%      +0.7        2.36        mpstat.cpu.all.iowait%
     32539 ±  3%     -13.4%      28168        proc-vmstat.pgpgout
     15408           -20.8%      12198        softirqs.BLOCK
    177637 ± 18%     -40.9%     104995 ± 34%  numa-numastat.node0.local_node
     66118 ± 22%     -56.2%      28966 ± 78%  numa-numastat.node1.other_node
     23431 ± 70%    +164.3%      61929 ± 37%  numa-vmstat.node0.numa_other
     75268 ± 22%     -51.1%      36837 ± 63%  numa-vmstat.node1.numa_other
      2688 ±  6%     -10.2%       2415 ±  3%  slabinfo.kmalloc-rcl-96.active_objs
      2688 ±  6%     -10.2%       2415 ±  3%  slabinfo.kmalloc-rcl-96.num_objs
    498.50 ±  3%     -12.9%     434.00        vmstat.io.bo
      1.00          +100.0%       2.00        vmstat.procs.b
      7210           -11.7%       6366        vmstat.system.cs
     18.41 ± 15%     -26.2%      13.60 ± 19%  perf-sched.total_wait_and_delay.max.ms
     17.47 ± 11%     -24.8%      13.14 ± 17%  perf-sched.total_wait_time.max.ms
     17.88 ± 18%     -37.2%      11.23 ± 37%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.59 ±183%     -98.0%       0.01 ±223%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.perf_event_release_kernel.perf_release
      1.14 ±191%     -97.9%       0.02 ±223%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.perf_event_release_kernel.perf_release
     17.17 ± 12%     -34.6%      11.22 ± 37%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     14.84 ± 54%      -7.5        7.34 ± 20%  perf-stat.i.cache-miss-rate%
      6993           -12.6%       6113        perf-stat.i.context-switches
 1.583e+08 ±  2%      -2.4%  1.544e+08        perf-stat.i.dTLB-stores
    918685 ± 15%     -13.8%     792034        perf-stat.i.iTLB-load-misses
     14.90 ± 48%      -7.0        7.92 ± 22%  perf-stat.overall.cache-miss-rate%
      6881           -12.6%       6015        perf-stat.ps.context-switches
 1.558e+08 ±  2%      -2.4%   1.52e+08        perf-stat.ps.dTLB-stores
    904016 ± 15%     -13.8%     779391        perf-stat.ps.iTLB-load-misses
      0.17 ± 21%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.__libc_start_main
      0.17 ± 21%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.main
      0.17 ± 21%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.run_builtin
      0.17 ± 21%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.cmd_stat
      0.17 ± 21%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.dispatch_events
      0.17 ± 21%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.process_interval
      0.16 ± 22%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.read_counters
      0.17 ± 41%      -0.1        0.10 ± 31%  perf-profile.children.cycles-pp.do_sys_ftruncate
      0.17 ± 40%      -0.1        0.10 ± 31%  perf-profile.children.cycles-pp.do_truncate
      0.16 ± 43%      -0.1        0.10 ± 28%  perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
      0.17 ± 41%      -0.1        0.10 ± 31%  perf-profile.children.cycles-pp.notify_change
      0.17 ± 42%      -0.1        0.10 ± 31%  perf-profile.children.cycles-pp.xfs_vn_setattr
      0.04 ± 72%      +0.1        0.10 ± 20%  perf-profile.children.cycles-pp.can_stop_idle_tick
      0.06 ±  7%      +0.1        0.13 ± 66%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.56 ±  8%      +0.2        0.72 ±  9%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.04 ± 71%      +0.1        0.09 ± 22%  perf-profile.self.cycles-pp.can_stop_idle_tick
      0.50 ±  8%      +0.1        0.61 ± 12%  perf-profile.self.cycles-pp.__hrtimer_next_event_base


                                                                                
                         stress-ng.fallocate.ops_per_sec                        
                                                                                
  12 +----------------------------------------------------------------------+   
     |                                                                      |   
  10 |.+.+.+.+.+.++.+.+   +.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.++           |   
     |                :   :                                      +   .+.+. .|   
     |           O    :   :                                       +.+     + |   
   8 |-O O O O O  O O O:O : O O O O O O OO O O O O O O O O O                |   
     |                 : :                                                  |   
   6 |-+               : :                                                  |   
     |                 : :                                                  |   
   4 |-+               : :                                                  |   
     |                 : :                                                  |   
     |                  ::                                                  |   
   2 |-+                :                                                   |   
     |                  :                                                   |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-knm02: 272 threads 2 sockets Intel(R) Xeon Phi(TM) CPU 7255 @ 1.10GHz with 112G memory
=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/bufferedio/1HDD/xfs/x86_64-rhel-8.3/hdd/debian-10.4-x86_64-20200603.cgz/lkp-knm02/MWUM/fxmark/0x11

commit: 
  3468bb1ca6 ("xfs: remove need_start_rec parameter from xlog_write()")
  eef983ffea ("xfs: journal IO cache flush reductions")

3468bb1ca6e88407 eef983ffeae7a1cdde8c3338155 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    264.33 ±  8%     -17.8%     217.23 ±  6%  fxmark.hdd_xfs_MWUM_18_bufferedio.idle_sec
      6.79 ±  7%     -14.0%       5.83 ±  5%  fxmark.hdd_xfs_MWUM_18_bufferedio.irq_sec
     16.63 ±  7%     -16.3%      13.92 ±  5%  fxmark.hdd_xfs_MWUM_18_bufferedio.real_sec
     16.22 ±  3%     -15.2%      13.75 ±  5%  fxmark.hdd_xfs_MWUM_18_bufferedio.secs
      0.36            +8.9%       0.40 ±  4%  fxmark.hdd_xfs_MWUM_18_bufferedio.softirq_util
      7.23 ±  4%     +17.3%       8.48 ±  2%  fxmark.hdd_xfs_MWUM_18_bufferedio.sys_util
      0.63 ±  3%     +17.2%       0.74 ±  4%  fxmark.hdd_xfs_MWUM_18_bufferedio.user_util
      3206 ±  2%     +17.9%       3780 ±  5%  fxmark.hdd_xfs_MWUM_18_bufferedio.works/sec
      0.11 ± 34%     -74.8%       0.03 ±141%  fxmark.hdd_xfs_MWUM_1_bufferedio.idle_util
      0.04 ± 10%     -76.9%       0.01        fxmark.hdd_xfs_MWUM_2_bufferedio.iowait_sec
      0.19 ± 11%     -75.8%       0.05 ±  2%  fxmark.hdd_xfs_MWUM_2_bufferedio.iowait_util
      0.56 ± 10%     +26.0%       0.71 ±  9%  fxmark.hdd_xfs_MWUM_36_bufferedio.iowait_util
     22.42 ±  4%     -20.7%      17.78        fxmark.hdd_xfs_MWUM_36_bufferedio.secs
      0.26           +11.3%       0.29 ±  2%  fxmark.hdd_xfs_MWUM_36_bufferedio.softirq_util
      2.56           +18.4%       3.03 ±  3%  fxmark.hdd_xfs_MWUM_36_bufferedio.sys_util
      2357 ±  4%     +25.5%       2958 ±  2%  fxmark.hdd_xfs_MWUM_36_bufferedio.works/sec
      0.36 ± 25%     -35.5%       0.23 ±  9%  fxmark.hdd_xfs_MWUM_4_bufferedio.iowait_sec
      0.81 ± 20%     -29.4%       0.57 ± 10%  fxmark.hdd_xfs_MWUM_4_bufferedio.iowait_util
      1.53 ±  3%     +43.7%       2.20 ± 18%  fxmark.hdd_xfs_MWUM_54_bufferedio.sys_util
      3.25 ±  3%    +116.4%       7.04 ± 28%  fxmark.hdd_xfs_MWUM_54_bufferedio.user_sec
      0.29 ±  6%    +462.4%       1.64 ± 85%  fxmark.hdd_xfs_MWUM_54_bufferedio.user_util
      0.37 ±  4%     +18.8%       0.44        fxmark.hdd_xfs_MWUM_72_bufferedio.iowait_util
     29.50 ±  2%     -21.4%      23.20 ±  2%  fxmark.hdd_xfs_MWUM_72_bufferedio.secs
      1.01 ±  5%     +34.8%       1.37 ±  8%  fxmark.hdd_xfs_MWUM_72_bufferedio.sys_util
      1833 ±  2%     +27.5%       2337        fxmark.hdd_xfs_MWUM_72_bufferedio.works/sec
    355.61            -5.1%     337.48        fxmark.time.elapsed_time
    355.61            -5.1%     337.48        fxmark.time.elapsed_time.max
     22015            -4.8%      20965        fxmark.time.involuntary_context_switches
     81060            +2.8%      83310        fxmark.time.voluntary_context_switches
    137.35            -1.2%     135.70        boot-time.boot
      2.18            -2.8%       2.12        boot-time.smp_boot
     53.58            -4.1%      51.36        iostat.cpu.idle
     28.35            +4.5%      29.62        iostat.cpu.system
     43.67 ± 25%     -56.5%      19.00 ± 34%  numa-numastat.node1.numa_hit
     43.67 ± 25%     -56.5%      19.00 ± 34%  numa-numastat.node1.other_node
     10882           +26.9%      13814        vmstat.io.bo
      7623            +4.0%       7927        vmstat.system.cs
     12126 ±  6%     -13.5%      10484 ±  5%  meminfo.Active
     12094 ±  6%     -13.6%      10449 ±  5%  meminfo.Active(anon)
    697.67          +227.7%       2286        meminfo.Mlocked
     12212 ±  6%     -16.3%      10221 ±  5%  numa-meminfo.node0.Active
     12180 ±  6%     -16.4%      10185 ±  5%  numa-meminfo.node0.Active(anon)
    694.67          +229.4%       2288        numa-meminfo.node0.Mlocked
      1294           +13.1%       1463 ±  6%  slabinfo.file_lock_cache.num_objs
      1122 ±  4%     -10.0%       1010 ±  9%  slabinfo.kmalloc-rcl-96.active_objs
    564.33 ±  5%     +17.8%     664.67 ±  7%  slabinfo.skbuff_fclone_cache.active_objs
    764.33 ±  3%     +11.3%     850.67 ±  6%  slabinfo.skbuff_fclone_cache.num_objs
      3047 ±  6%     -16.4%       2549 ±  5%  numa-vmstat.node0.nr_active_anon
    271442           +22.2%     331698        numa-vmstat.node0.nr_dirtied
    142.00 ± 10%     +39.2%     197.67 ±  5%  numa-vmstat.node0.nr_dirty
    173.67          +229.2%     571.67        numa-vmstat.node0.nr_mlock
    271273           +22.2%     331494        numa-vmstat.node0.nr_written
      3047 ±  6%     -16.4%       2549 ±  5%  numa-vmstat.node0.nr_zone_active_anon
      3089 ±  6%     -15.5%       2611 ±  4%  proc-vmstat.nr_active_anon
     68420            +0.6%      68849        proc-vmstat.nr_anon_pages
    465673           +19.9%     558162        proc-vmstat.nr_dirtied
    134.00 ±  7%     +54.7%     207.33 ±  3%  proc-vmstat.nr_dirty
     10656            +2.8%      10952        proc-vmstat.nr_mapped
    173.00          +230.4%     571.67        proc-vmstat.nr_mlock
    465673           +19.9%     558162        proc-vmstat.nr_written
      3089 ±  6%     -15.5%       2611 ±  4%  proc-vmstat.nr_zone_active_anon
    169.33 ±  6%     +45.9%     247.00 ±  3%  proc-vmstat.nr_zone_write_pending
   1511406            -2.5%    1474355        proc-vmstat.numa_hit
   1511362            -2.4%    1474336        proc-vmstat.numa_local
     53181 ± 17%     -25.7%      39487 ± 34%  proc-vmstat.numa_pte_updates
   1768573            -1.6%    1740840        proc-vmstat.pgalloc_normal
   1103721            -4.6%    1052829        proc-vmstat.pgfault
   1704090            -1.8%    1673299        proc-vmstat.pgfree
   3714826           +20.1%    4462188        proc-vmstat.pgpgout
    107504            -3.0%     104277        proc-vmstat.pgreuse
      0.56 ±  7%     +16.8%       0.66        perf-stat.i.MPKI
      0.20 ±  5%      +0.0        0.23 ±  3%  perf-stat.i.branch-miss-rate%
      0.42 ±  4%      +0.0        0.46 ±  5%  perf-stat.i.cache-miss-rate%
   2763955 ±  4%      +8.9%    3008571 ±  5%  perf-stat.i.cache-references
      6783            +3.2%       6999        perf-stat.i.context-switches
      0.12 ±  8%     +13.0%       0.14 ±  6%  perf-stat.i.cpi
     29.24 ±  7%     +14.5%      33.49 ±  2%  perf-stat.i.cycles-between-cache-misses
 1.197e+08 ±  3%      +6.8%  1.278e+08 ±  5%  perf-stat.i.iTLB-loads
      1.90 ±  3%     +20.5%       2.29 ±  9%  perf-stat.i.instructions-per-iTLB-miss
      0.00 ±  3%     +16.6%       0.01 ±  7%  perf-stat.i.ipc
      1.12 ±  7%     +10.2%       1.24 ±  6%  perf-stat.i.major-faults
     11.53 ±  4%      +8.5%      12.50 ±  5%  perf-stat.i.metric.K/sec
      0.52 ±  3%      +7.0%       0.56 ±  5%  perf-stat.i.metric.M/sec
     23.11            +2.1%      23.59        perf-stat.overall.MPKI
   2811536 ±  4%      +8.5%    3051404 ±  6%  perf-stat.ps.cache-references
      6725            +3.2%       6940        perf-stat.ps.context-switches
 1.216e+08 ±  3%      +6.5%  1.294e+08 ±  5%  perf-stat.ps.iTLB-loads
      1.12 ±  7%     +10.1%       1.23 ±  6%  perf-stat.ps.major-faults
     24319           -10.4%      21779 ±  2%  softirqs.CPU10.SCHED
      5160 ± 77%     -59.3%       2098 ±  2%  softirqs.CPU106.RCU
     24458 ±  3%      -9.4%      22160        softirqs.CPU14.SCHED
     25258 ±  2%     -11.6%      22339        softirqs.CPU15.SCHED
      1034 ± 54%    +350.0%       4656 ± 82%  softirqs.CPU16.NET_RX
     24844           -11.5%      21992 ±  6%  softirqs.CPU17.SCHED
     21245           -12.6%      18578 ±  5%  softirqs.CPU19.SCHED
     31656 ±  5%     -12.1%      27818        softirqs.CPU2.SCHED
     21064            -9.2%      19127        softirqs.CPU20.SCHED
     21553           -15.0%      18319 ±  4%  softirqs.CPU22.SCHED
      6980 ± 59%     -56.9%       3007 ±  3%  softirqs.CPU232.RCU
     20768 ±  4%      -9.5%      18791 ±  3%  softirqs.CPU24.SCHED
     21076           -11.9%      18566 ±  2%  softirqs.CPU26.SCHED
     20452 ±  5%     -13.3%      17735 ±  7%  softirqs.CPU28.SCHED
     21113 ±  3%     -10.0%      19011        softirqs.CPU29.SCHED
     21302           -13.6%      18416 ±  8%  softirqs.CPU31.SCHED
     21184           -11.0%      18849 ±  2%  softirqs.CPU32.SCHED
     19825 ±  4%     -12.6%      17321 ±  3%  softirqs.CPU34.SCHED
     20150 ±  4%      -8.6%      18412 ±  4%  softirqs.CPU35.SCHED
     16534           -12.4%      14477 ±  8%  softirqs.CPU38.SCHED
     16494 ±  2%      -8.4%      15107 ±  2%  softirqs.CPU39.SCHED
     16517           -14.7%      14087 ±  8%  softirqs.CPU42.SCHED
     16473           -13.0%      14329 ±  5%  softirqs.CPU43.SCHED
     16370 ±  4%      -9.0%      14896        softirqs.CPU44.SCHED
     25366 ±  2%     -10.3%      22741 ±  3%  softirqs.CPU5.SCHED
     15927 ±  4%     -11.1%      14165 ±  8%  softirqs.CPU50.SCHED
     13790 ± 14%     -14.9%      11737 ±  8%  softirqs.CPU52.SCHED
     11412            -9.9%      10278        softirqs.CPU56.SCHED
     11935 ±  3%     -20.4%       9506 ± 13%  softirqs.CPU57.SCHED
     11474            -8.9%      10450 ±  2%  softirqs.CPU58.SCHED
     11480           -13.1%       9972 ±  5%  softirqs.CPU59.SCHED
     24830           -10.8%      22142        softirqs.CPU6.SCHED
     11474            -9.9%      10339 ±  3%  softirqs.CPU65.SCHED
     11310 ±  2%     -17.5%       9334 ±  9%  softirqs.CPU66.SCHED
     11451 ±  2%     -17.0%       9508 ±  8%  softirqs.CPU67.SCHED
     13860 ± 10%     -18.1%      11352 ±  2%  softirqs.CPU69.SCHED
     25199            -9.7%      22746 ±  4%  softirqs.CPU7.SCHED
     24591 ±  3%     -11.5%      21752 ±  3%  softirqs.CPU8.SCHED
     24328 ±  2%      -8.0%      22387 ±  2%  softirqs.CPU9.SCHED
   1035077            +2.0%    1056227        interrupts.CAL:Function_call_interrupts
      3097           -49.3%       1570        interrupts.CPU0.171:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
    432.33 ± 66%   +1399.3%       6482 ± 67%  interrupts.CPU0.36:IR-PCI-MSI.2621442-edge.eth1-TxRx-1
      2944 ± 28%     -55.7%       1303 ±  7%  interrupts.CPU0.NMI:Non-maskable_interrupts
      2944 ± 28%     -55.7%       1303 ±  7%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      6993 ± 10%     -14.9%       5954 ±  9%  interrupts.CPU0.RES:Rescheduling_interrupts
      9085 ±  3%     -53.5%       4220 ± 40%  interrupts.CPU1.171:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
      3570 ±  2%     -62.7%       1331 ±  7%  interrupts.CPU1.NMI:Non-maskable_interrupts
      3570 ±  2%     -62.7%       1331 ±  7%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
      9354 ± 12%     -26.8%       6845 ±  9%  interrupts.CPU10.CAL:Function_call_interrupts
      3480 ±  3%     -49.8%       1747 ± 39%  interrupts.CPU12.NMI:Non-maskable_interrupts
      3480 ±  3%     -49.8%       1747 ± 39%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      1748 ±  3%     -27.1%       1274 ±  5%  interrupts.CPU15.NMI:Non-maskable_interrupts
      1748 ±  3%     -27.1%       1274 ±  5%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      2294 ± 31%     -43.9%       1287 ±  5%  interrupts.CPU17.NMI:Non-maskable_interrupts
      2294 ± 31%     -43.9%       1287 ±  5%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      2917 ± 25%     -55.3%       1305 ±  6%  interrupts.CPU18.NMI:Non-maskable_interrupts
      2917 ± 25%     -55.3%       1305 ±  6%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
     12689 ±  7%     +16.1%      14727 ±  2%  interrupts.CPU2.CAL:Function_call_interrupts
      2372 ± 32%     -42.7%       1358 ±  6%  interrupts.CPU2.NMI:Non-maskable_interrupts
      2372 ± 32%     -42.7%       1358 ±  6%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      2757 ± 28%     -52.5%       1310 ±  8%  interrupts.CPU20.NMI:Non-maskable_interrupts
      2757 ± 28%     -52.5%       1310 ±  8%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      2884 ± 31%     -54.8%       1302 ±  4%  interrupts.CPU28.NMI:Non-maskable_interrupts
      2884 ± 31%     -54.8%       1302 ±  4%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
      5032 ± 10%     +22.0%       6137 ± 15%  interrupts.CPU29.CAL:Function_call_interrupts
      2830 ± 29%     -54.9%       1276 ±  6%  interrupts.CPU29.NMI:Non-maskable_interrupts
      2830 ± 29%     -54.9%       1276 ±  6%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      1175 ±  7%     -25.9%     870.67 ± 11%  interrupts.CPU3.RES:Rescheduling_interrupts
      2867 ± 29%     -55.2%       1285 ±  6%  interrupts.CPU33.NMI:Non-maskable_interrupts
      2867 ± 29%     -55.2%       1285 ±  6%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
      2876 ± 29%     -55.8%       1271 ±  4%  interrupts.CPU34.NMI:Non-maskable_interrupts
      2876 ± 29%     -55.8%       1271 ±  4%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      5063 ± 14%     +33.3%       6750 ± 21%  interrupts.CPU35.CAL:Function_call_interrupts
      2907 ± 30%     -56.9%       1254 ±  5%  interrupts.CPU35.NMI:Non-maskable_interrupts
      2907 ± 30%     -56.9%       1254 ±  5%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      3814           +58.2%       6033 ± 13%  interrupts.CPU36.CAL:Function_call_interrupts
      3756 ±  3%     +48.6%       5583 ± 12%  interrupts.CPU37.CAL:Function_call_interrupts
      2285 ± 33%     -43.6%       1290 ±  6%  interrupts.CPU37.NMI:Non-maskable_interrupts
      2285 ± 33%     -43.6%       1290 ±  6%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      2259 ± 33%     -43.1%       1285 ±  7%  interrupts.CPU38.NMI:Non-maskable_interrupts
      2259 ± 33%     -43.1%       1285 ±  7%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
      3398           -62.3%       1280 ±  9%  interrupts.CPU39.NMI:Non-maskable_interrupts
      3398           -62.3%       1280 ±  9%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      1747 ±  2%     +49.9%       2619 ±  6%  interrupts.CPU4.NMI:Non-maskable_interrupts
      1747 ±  2%     +49.9%       2619 ±  6%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
      5027 ± 14%     -16.6%       4194 ± 14%  interrupts.CPU40.CAL:Function_call_interrupts
      2850 ± 29%     -41.9%       1656 ± 25%  interrupts.CPU40.NMI:Non-maskable_interrupts
      2850 ± 29%     -41.9%       1656 ± 25%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      3913 ±  8%     +13.9%       4458 ±  9%  interrupts.CPU43.CAL:Function_call_interrupts
      2855 ± 25%     -55.8%       1262 ±  7%  interrupts.CPU43.NMI:Non-maskable_interrupts
      2855 ± 25%     -55.8%       1262 ±  7%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      2250 ± 34%     -37.8%       1400 ± 14%  interrupts.CPU47.NMI:Non-maskable_interrupts
      2250 ± 34%     -37.8%       1400 ± 14%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      2915 ± 32%     -40.8%       1725 ± 39%  interrupts.CPU50.NMI:Non-maskable_interrupts
      2915 ± 32%     -40.8%       1725 ± 39%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      3451 ±  4%     -37.0%       2173 ± 26%  interrupts.CPU51.NMI:Non-maskable_interrupts
      3451 ±  4%     -37.0%       2173 ± 26%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      2376 ± 40%     -44.4%       1321 ±  5%  interrupts.CPU53.NMI:Non-maskable_interrupts
      2376 ± 40%     -44.4%       1321 ±  5%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      3284           +20.4%       3954 ± 16%  interrupts.CPU65.CAL:Function_call_interrupts
    217388           -11.9%     191509 ±  6%  interrupts.CPU68.LOC:Local_timer_interrupts
      4542 ±  9%     +24.8%       5670 ±  9%  interrupts.CPU69.CAL:Function_call_interrupts
      6208 ±  3%     +27.5%       7914 ± 17%  interrupts.CPU9.CAL:Function_call_interrupts
    133580 ±  4%     -31.2%      91950 ±  5%  interrupts.NMI:Non-maskable_interrupts
    133580 ±  4%     -31.2%      91950 ±  5%  interrupts.PMI:Performance_monitoring_interrupts
     63.06 ± 70%     -63.1        0.00        perf-profile.calltrace.cycles-pp.open64
     63.04 ± 70%     -63.0        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64
     63.03 ± 70%     -63.0        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
     63.02 ± 70%     -63.0        0.00        perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
     63.02 ± 70%     -63.0        0.00        perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
     89.31            -4.1       85.16        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
     92.25            -3.6       88.67        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2.do_sys_open
     94.89            -3.3       91.57        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     94.88            -3.3       91.57        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      0.89 ±  2%      +0.1        0.95 ±  5%  perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      0.57 ± 10%      +0.1        0.70 ± 10%  perf-profile.calltrace.cycles-pp.__libc_start_main
      0.57 ± 10%      +0.1        0.70 ± 10%  perf-profile.calltrace.cycles-pp.main.__libc_start_main
      0.57 ± 10%      +0.1        0.70 ± 10%  perf-profile.calltrace.cycles-pp.run_builtin.main.__libc_start_main
      1.93 ±  2%      +0.2        2.14 ±  5%  perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      1.89            +0.2        2.10 ±  5%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2
      0.19 ±141%      +0.4        0.60 ± 11%  perf-profile.calltrace.cycles-pp.perf_session__process_events.record__finish_output.cmd_record.cmd_sched.run_builtin
      0.19 ±141%      +0.4        0.60 ± 12%  perf-profile.calltrace.cycles-pp.record__finish_output.cmd_record.cmd_sched.run_builtin.main
      0.19 ±141%      +0.4        0.63 ± 11%  perf-profile.calltrace.cycles-pp.cmd_record.cmd_sched.run_builtin.main.__libc_start_main
      0.19 ±141%      +0.4        0.63 ± 11%  perf-profile.calltrace.cycles-pp.cmd_sched.run_builtin.main.__libc_start_main
      2.66            +0.5        3.13 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      1.99 ± 11%      +0.6        2.60 ±  2%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.7        0.65 ± 13%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.66 ± 15%      +0.7        1.33 ± 12%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.46 ± 70%      +0.8        1.25 ± 12%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.07 ± 10%      +1.0        2.11 ± 12%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      1.13 ± 10%      +1.1        2.25 ± 12%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.46 ± 10%      +1.5        2.95 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      1.51 ± 10%      +1.5        3.05 ± 11%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.10 ±  9%      +2.1        4.23 ± 11%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.10 ±  9%      +2.1        4.23 ± 11%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
      2.09 ±  9%      +2.1        4.22 ± 11%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.16 ±  9%      +2.1        4.29 ± 11%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     63.06 ± 70%     -63.1        0.00        perf-profile.children.cycles-pp.open64
     89.36            -4.1       85.22        perf-profile.children.cycles-pp.osq_lock
     92.26            -3.3       88.92        perf-profile.children.cycles-pp.rwsem_down_write_slowpath
     95.00            -3.3       91.68        perf-profile.children.cycles-pp.do_sys_open
     95.00            -3.3       91.68        perf-profile.children.cycles-pp.do_sys_openat2
     94.94            -3.3       91.62        perf-profile.children.cycles-pp.do_filp_open
     94.93            -3.3       91.62        perf-profile.children.cycles-pp.path_openat
     96.10            -2.5       93.63        perf-profile.children.cycles-pp.do_syscall_64
     96.11            -2.5       93.64        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.05            +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.iomap_apply
      0.05            +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.iomap_file_buffered_write
      0.07 ± 11%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.06 ±  7%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.do_fault
      0.05 ±  8%      +0.0        0.07        perf-profile.children.cycles-pp.xfs_buf_item_format_segment
      0.06 ±  7%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.do_iter_readv_writev
      0.06 ±  7%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_file_buffered_write
      0.10 ±  4%      +0.0        0.12 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock
      0.09 ±  5%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.walk_component
      0.07 ±  7%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.16 ±  3%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.xfs_da3_node_read
      0.06 ±  7%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.do_iter_write
      0.05 ±  8%      +0.0        0.08 ± 12%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.05 ±  8%      +0.0        0.08 ± 12%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.11 ±  8%      +0.0        0.13 ± 12%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.08 ± 10%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.06 ±  7%      +0.0        0.09 ±  9%  perf-profile.children.cycles-pp.lo_write_bvec
      0.12 ± 10%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.11 ±  8%      +0.0        0.14 ± 13%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.05            +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.21 ±  3%      +0.0        0.24 ±  3%  perf-profile.children.cycles-pp.xfs_da_read_buf
      0.09 ±  5%      +0.0        0.12 ± 15%  perf-profile.children.cycles-pp.link_path_walk
      0.06 ±  8%      +0.0        0.09 ± 19%  perf-profile.children.cycles-pp.xfs_next_bit
      0.05            +0.0        0.08 ± 14%  perf-profile.children.cycles-pp.flush_smp_call_function_from_idle
      0.17 ±  2%      +0.0        0.21 ± 11%  perf-profile.children.cycles-pp.__libc_write
      0.17 ±  4%      +0.0        0.20 ± 12%  perf-profile.children.cycles-pp.__generic_file_write_iter
      0.16 ±  7%      +0.0        0.20 ±  8%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.13 ±  9%      +0.0        0.16 ±  5%  perf-profile.children.cycles-pp.build_id__mark_dso_hit
      0.05 ±  8%      +0.0        0.09 ± 10%  perf-profile.children.cycles-pp.timerqueue_del
      0.03 ± 70%      +0.0        0.07 ± 14%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.08 ± 10%      +0.0        0.11 ± 16%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.08 ± 16%      +0.0        0.11 ± 19%  perf-profile.children.cycles-pp.execve
      0.12            +0.0        0.15 ±  6%  perf-profile.children.cycles-pp.wake_up_q
      0.19 ±  4%      +0.0        0.23 ± 11%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.16 ±  5%      +0.0        0.20 ± 14%  perf-profile.children.cycles-pp.generic_perform_write
      0.17 ±  4%      +0.0        0.21 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.13 ±  3%      +0.0        0.16 ±  5%  perf-profile.children.cycles-pp.rwsem_wake
      0.08 ±  6%      +0.0        0.11 ± 18%  perf-profile.children.cycles-pp.xfs_inobt_irec_to_allocmask
      0.07 ±  7%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.read_tsc
      0.07 ± 25%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.xfs_file_fsync
      0.04 ± 71%      +0.0        0.07 ± 17%  perf-profile.children.cycles-pp.xfsaild
      0.06 ±  8%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.rcu_core
      0.03 ± 70%      +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.xfsaild_push
      0.13 ±  9%      +0.0        0.17 ±  4%  perf-profile.children.cycles-pp.handle_mm_fault
      0.17 ±  4%      +0.0        0.21 ± 10%  perf-profile.children.cycles-pp.exc_page_fault
      0.16 ±  8%      +0.0        0.20 ±  7%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      0.13 ± 13%      +0.0        0.17 ±  7%  perf-profile.children.cycles-pp.xfs_dir2_node_addname_int
      0.06 ± 16%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.__poll
      0.06 ± 16%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.__x64_sys_poll
      0.06 ± 16%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.do_sys_poll
      0.02 ±141%      +0.0        0.06 ±  8%  perf-profile.children.cycles-pp.rb_insert_color
      0.02 ±141%      +0.0        0.06 ±  8%  perf-profile.children.cycles-pp.schedule_timeout
      0.03 ± 70%      +0.0        0.07 ± 17%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.17 ±  7%      +0.0        0.21 ± 11%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.13 ±  3%      +0.0        0.17 ±  7%  perf-profile.children.cycles-pp.xfs_btree_lookup
      0.07 ± 20%      +0.0        0.11 ± 12%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.08 ±  6%      +0.0        0.12 ± 11%  perf-profile.children.cycles-pp.uart_console_write
      0.31 ±  6%      +0.0        0.35 ±  3%  perf-profile.children.cycles-pp.xfs_dir_createname
      0.14 ±  5%      +0.0        0.18 ± 10%  perf-profile.children.cycles-pp.xlog_cil_insert_items
      0.12 ±  6%      +0.0        0.16 ± 16%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.04 ± 71%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.11 ±  7%      +0.0        0.15 ±  8%  perf-profile.children.cycles-pp.xfs_buf_find
      0.22 ±  7%      +0.0        0.26 ±  8%  perf-profile.children.cycles-pp.machines__deliver_event
      0.07 ±  6%      +0.0        0.12 ± 11%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.17 ±  2%      +0.0        0.22 ± 12%  perf-profile.children.cycles-pp.xfs_trans_alloc_icreate
      0.08 ±  6%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.12 ±  3%      +0.0        0.17 ± 12%  perf-profile.children.cycles-pp.__percpu_counter_compare
      0.07 ± 11%      +0.0        0.12 ± 14%  perf-profile.children.cycles-pp.tick_irq_enter
      0.23 ± 12%      +0.0        0.28 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      0.27 ±  8%      +0.0        0.32 ±  2%  perf-profile.children.cycles-pp.xfs_dir2_node_addname
      0.15 ±  3%      +0.0        0.20 ±  6%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.08 ± 14%      +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.08 ±  6%      +0.1        0.13 ± 14%  perf-profile.children.cycles-pp.serial8250_console_write
      0.02 ±141%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.enqueue_entity
      0.02 ±141%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.error_entry
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.do_dentry_open
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.xlog_cil_committed
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.iomap_write_actor
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.xlog_cil_process_committed
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.kblockd_mod_delayed_work_on
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.mod_delayed_work_on
      0.20 ±  4%      +0.1        0.25 ±  8%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      0.15 ± 12%      +0.1        0.20 ± 10%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.13 ±  6%      +0.1        0.18 ± 12%  perf-profile.children.cycles-pp.xfs_mod_fdblocks
      0.15 ±  3%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.xfs_buf_read_map
      0.11 ±  7%      +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.04 ± 70%      +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__kmalloc
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.13 ±  7%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.02 ±141%      +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.idle_cpu
      0.19 ±  4%      +0.1        0.25 ± 10%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.10            +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
      0.06 ±  7%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.10 ±  9%      +0.1        0.16 ±  5%  perf-profile.children.cycles-pp.dequeue_entity
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.__cond_resched
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.02 ±141%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.blk_complete_reqs
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.xfs_read_agi
      0.04 ± 70%      +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.io_serial_in
      0.14 ± 15%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.loop_queue_work
      0.11 ± 16%      +0.1        0.17 ± 15%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.xfs_bmapi_allocate
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.xlog_ioend_work
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.xlog_state_do_callback
      0.11 ±  7%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      0.24 ±  5%      +0.1        0.30 ±  4%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.24 ±  5%      +0.1        0.30 ±  9%  perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
      0.23 ± 15%      +0.1        0.29 ±  9%  perf-profile.children.cycles-pp.perf_session__process_user_event
      0.23 ± 15%      +0.1        0.29 ±  9%  perf-profile.children.cycles-pp.__ordered_events__flush
      0.09 ± 10%      +0.1        0.15 ±  5%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.09 ±  9%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.08 ±  5%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.printk
      0.08 ± 14%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.update_blocked_averages
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.finish_task_switch
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.perf_poll
      0.11 ± 12%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.13 ±  3%      +0.1        0.19 ±  4%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.17 ± 24%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.14 ± 21%      +0.1        0.21 ±  6%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.08 ± 12%      +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.00            +0.1        0.07 ± 25%  perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.24 ±  7%      +0.1        0.30 ±  9%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      0.09 ±  5%      +0.1        0.15 ± 15%  perf-profile.children.cycles-pp.vprintk_emit
      0.09 ±  5%      +0.1        0.15 ± 15%  perf-profile.children.cycles-pp.console_unlock
      0.89 ±  2%      +0.1        0.95 ±  5%  perf-profile.children.cycles-pp.xfs_dialloc
      0.28 ±  9%      +0.1        0.35 ±  8%  perf-profile.children.cycles-pp.perf_session__deliver_event
      0.15 ±  8%      +0.1        0.22 ±  9%  perf-profile.children.cycles-pp.newidle_balance
      0.15 ± 14%      +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.kthread_worker_fn
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.blk_mq_submit_bio
      0.12 ±  8%      +0.1        0.19 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.__queue_work
      0.16 ±  6%      +0.1        0.23 ±  7%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      0.16 ±  5%      +0.1        0.24 ± 14%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.11 ±  7%      +0.1        0.19 ± 11%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.xfs_dir_removename
      0.18 ±  4%      +0.1        0.26 ± 15%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.07 ± 23%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.irq_work_run_list
      0.15 ±  6%      +0.1        0.25 ± 10%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.07 ± 23%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.05 ± 72%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.05 ± 72%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.05 ± 72%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.05 ± 72%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.irq_work_run
      0.05 ± 72%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.irq_work_single
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.submit_bio
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.submit_bio_noacct
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.xlog_state_release_iclog
      0.48 ±  9%      +0.1        0.57 ± 12%  perf-profile.children.cycles-pp.process_simple
      0.21 ±  2%      +0.1        0.31 ±  4%  perf-profile.children.cycles-pp.try_to_wake_up
      0.16 ±  8%      +0.1        0.26 ±  8%  perf-profile.children.cycles-pp.find_busiest_group
      0.53 ±  9%      +0.1        0.63 ± 11%  perf-profile.children.cycles-pp.cmd_record
      0.30 ±  4%      +0.1        0.40 ±  8%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      0.50 ±  9%      +0.1        0.60 ± 12%  perf-profile.children.cycles-pp.record__finish_output
      0.50 ±  9%      +0.1        0.60 ± 12%  perf-profile.children.cycles-pp.perf_session__process_events
      0.18 ± 11%      +0.1        0.28        perf-profile.children.cycles-pp.clockevents_program_event
      0.16 ± 13%      +0.1        0.27 ±  6%  perf-profile.children.cycles-pp.ktime_get
      0.37 ±  3%      +0.1        0.48 ± 10%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      0.51 ±  9%      +0.1        0.63 ± 11%  perf-profile.children.cycles-pp.cmd_sched
      0.32 ±  4%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp.__xfs_trans_commit
      0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.xlog_write
      0.04 ± 71%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.xlog_cil_push_work
      0.57 ± 10%      +0.1        0.70 ± 10%  perf-profile.children.cycles-pp.__libc_start_main
      0.57 ± 10%      +0.1        0.70 ± 10%  perf-profile.children.cycles-pp.main
      0.57 ± 10%      +0.1        0.70 ± 10%  perf-profile.children.cycles-pp.run_builtin
      0.23 ±  9%      +0.1        0.36 ±  6%  perf-profile.children.cycles-pp.load_balance
      0.00            +0.1        0.13 ± 33%  perf-profile.children.cycles-pp.xfs_difree_finobt
      0.10 ± 19%      +0.1        0.24 ± 13%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.2        0.16 ± 31%  perf-profile.children.cycles-pp.xfs_difree
      0.64 ±  3%      +0.2        0.81 ±  9%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      0.38 ±  7%      +0.2        0.56 ±  9%  perf-profile.children.cycles-pp.schedule
      0.00            +0.2        0.18 ± 34%  perf-profile.children.cycles-pp.xfs_ifree
      1.94 ±  2%      +0.2        2.14 ±  5%  perf-profile.children.cycles-pp.xfs_generic_create
      1.89            +0.2        2.10 ±  5%  perf-profile.children.cycles-pp.xfs_create
      0.24 ± 12%      +0.2        0.47 ± 14%  perf-profile.children.cycles-pp.intel_idle
      0.00            +0.2        0.22 ± 20%  perf-profile.children.cycles-pp.xfs_vn_unlink
      0.00            +0.2        0.22 ± 20%  perf-profile.children.cycles-pp.xfs_remove
      0.00            +0.2        0.23 ± 33%  perf-profile.children.cycles-pp.xfs_inactive
      0.00            +0.2        0.23 ± 33%  perf-profile.children.cycles-pp.xfs_inactive_ifree
      0.00            +0.2        0.23 ± 33%  perf-profile.children.cycles-pp.xfs_fs_destroy_inode
      0.00            +0.2        0.23 ± 19%  perf-profile.children.cycles-pp.vfs_unlink
      0.00            +0.2        0.23 ± 34%  perf-profile.children.cycles-pp.destroy_inode
      0.33 ±  5%      +0.2        0.57 ±  9%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.51 ±  7%      +0.2        0.76 ±  7%  perf-profile.children.cycles-pp.__schedule
      0.43 ±  6%      +0.3        0.72 ±  8%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.31 ±  9%      +0.4        0.67 ± 13%  perf-profile.children.cycles-pp.menu_select
      2.62 ± 10%      +0.4        3.05 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      2.67            +0.7        3.33 ±  4%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      3.17 ±  9%      +0.7        3.90 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +0.7        0.74 ± 26%  perf-profile.children.cycles-pp.do_unlinkat
      3.24 ±  9%      +0.8        4.01 ±  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      3.85 ±  7%      +1.2        5.03 ±  5%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      4.08 ±  8%      +1.2        5.31 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      1.55 ± 10%      +1.5        3.07 ± 11%  perf-profile.children.cycles-pp.cpuidle_enter_state
      1.55 ± 11%      +1.5        3.09 ± 11%  perf-profile.children.cycles-pp.cpuidle_enter
      2.10 ±  9%      +2.1        4.23 ± 11%  perf-profile.children.cycles-pp.start_secondary
      2.16 ±  9%      +2.1        4.29 ± 11%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
      2.16 ±  9%      +2.1        4.29 ± 11%  perf-profile.children.cycles-pp.cpu_startup_entry
      2.16 ±  9%      +2.1        4.29 ± 11%  perf-profile.children.cycles-pp.do_idle
     86.82            -4.0       82.82        perf-profile.self.cycles-pp.osq_lock
      0.06 ±  7%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.08 ± 10%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.update_load_avg
      0.06 ± 13%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.09 ±  9%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.09 ± 10%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      0.06 ± 16%      +0.0        0.08 ± 17%  perf-profile.self.cycles-pp.update_cfs_group
      0.06 ±  8%      +0.0        0.08 ± 17%  perf-profile.self.cycles-pp.xfs_next_bit
      0.07 ±  6%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.10 ±  4%      +0.0        0.13 ± 16%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.07 ± 17%      +0.0        0.11 ± 15%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.06 ± 13%      +0.0        0.10 ± 31%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.06 ± 14%      +0.0        0.10 ± 12%  perf-profile.self.cycles-pp.read_tsc
      0.06 ±  7%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.02 ±141%      +0.0        0.06 ±  8%  perf-profile.self.cycles-pp.update_curr
      0.07 ± 20%      +0.0        0.11 ± 12%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.07 ±  6%      +0.0        0.12 ± 14%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.04 ± 71%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.02 ±141%      +0.0        0.07 ± 18%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.10 ±  4%      +0.0        0.15 ±  8%  perf-profile.self.cycles-pp.native_sched_clock
      0.02 ±141%      +0.1        0.07 ± 25%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.02 ±141%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.irq_exit_rcu
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.xfs_btree_get_rec
      0.03 ± 70%      +0.1        0.09 ± 14%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.02 ±141%      +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.idle_cpu
      0.08 ± 14%      +0.1        0.14 ± 17%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.error_entry
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.scheduler_tick
      0.04 ± 70%      +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.io_serial_in
      0.16 ± 21%      +0.1        0.21 ±  9%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.03 ± 70%      +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.do_idle
      0.10            +0.1        0.17 ± 23%  perf-profile.self.cycles-pp.update_process_times
      0.10 ±  8%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.08 ± 10%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.10 ± 16%      +0.1        0.18 ± 12%  perf-profile.self.cycles-pp.ktime_get
      0.18 ± 10%      +0.2        0.35 ± 12%  perf-profile.self.cycles-pp.menu_select
      0.24 ± 11%      +0.2        0.47 ± 14%  perf-profile.self.cycles-pp.intel_idle
      2.59            +0.6        3.21 ±  4%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.03 ±  3%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.blk_mq_freeze_queue_wait.lo_release.__blkdev_put.blkdev_put
      0.06 ± 18%   +2809.6%       1.62 ± 77%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.03 ±  9%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.devtmpfs_work_loop.devtmpfsd.kthread.ret_from_fork
      0.02 ± 22%    +100.0%       0.05 ± 16%  perf-sched.sch_delay.avg.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
      0.04 ±  5%     -27.2%       0.03 ±  5%  perf-sched.sch_delay.avg.ms.kthread_worker_fn.kthread.ret_from_fork
      0.20 ±138%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.percpu_down_write._cpu_down.cpu_down.device_offline
      0.25 ±  7%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
      6.73 ±140%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_shepherd
      0.27 ±124%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_start
      0.00 ± 10%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.pte_alloc_one.__pte_alloc
      0.48 ± 77%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__flush_work.lru_add_drain_all.khugepaged
      0.02 ± 63%   +1336.7%       0.23 ±125%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_lookup
      0.04 ± 35%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xlog_alloc_log
      0.03 ± 53%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.simple_recursive_removal.debugfs_remove
      0.01 ± 43%     -85.0%       0.00 ±141%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma
      0.51 ±132%     -97.1%       0.02 ± 23%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      0.02 ± 38%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.crypto_exit_scomp_ops_async.crypto_destroy_tfm
      0.02 ± 36%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.device_del.device_unregister
      0.03 ±  2%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.flush_workqueue.xfs_buftarg_drain
      0.03 ±  4%     -63.9%       0.01 ± 14%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.synchronize_rcu_expedited.namespace_unlock.path_umount
      0.00 ± 28%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.vfree.free_vm_stack_cache.cpuhp_invoke_callback
      0.01 ±107%     -88.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      0.03 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion_io_timeout.submit_bio_wait.xfs_rw_bdev
      0.04 ± 28%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.write_inode_now.iput.xfs_rtunmount_inodes
      0.04 ± 19%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rescuer_thread.kthread.ret_from_fork
      0.12 ± 15%     -52.7%       0.06 ± 24%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.10 ± 80%     -81.7%       0.02 ±  7%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.04          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.isra.0.blkdev_get_by_dev
      0.06 ± 37%     -51.1%       0.03 ±  7%  perf-sched.sch_delay.avg.ms.schedule_timeout.__down.down.xfs_buf_lock
      0.02 ±  4%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__blkdev_direct_IO_simple.generic_file_direct_write
      0.03          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__blkdev_direct_IO_simple.generic_file_read_iter
      0.03 ±  4%     +25.6%       0.03 ±  4%  perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
      0.05 ±  3%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.sync_rcu_exp_select_cpus
      0.03 ± 12%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
      0.14 ±  9%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__wait_rcu_gp.synchronize_rcu
      0.05 ± 60%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.cpuhp_kick_ap.cpuhp_kick_ap_work
      0.03          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.devtmpfs_submit_req.devtmpfs_delete_node
      0.01 ± 38%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.kthread_stop.destroy_workqueue
      0.05 ± 20%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.kthread_stop.xfs_trans_ail_destroy
      0.04 ± 20%    +150.8%       0.10 ± 41%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
      0.09 ± 26%     -48.2%       0.05 ± 22%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.02           -19.0%       0.02 ±  8%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.08 ±  3%     +18.0%       0.09 ±  2%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.07 ± 22%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.xfs_ail_push_all_sync.xfs_log_quiesce.xfs_log_unmount.xfs_unmountfs
      0.05 ± 55%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.xfsaild.kthread.ret_from_fork
      0.05 ± 12%     -37.6%       0.03 ±  3%  perf-sched.sch_delay.avg.ms.xlog_state_get_iclog_space.xlog_write.xlog_cil_push_work.process_one_work
      0.06 ± 79%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.__xfs_trans_commit
      0.02 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.xfs_log_unmount_write.xfs_log_unmount.xfs_unmountfs
      0.06 ±  3%     -94.5%       0.00 ± 47%  perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
      0.03 ±  5%    -100.0%       0.00        perf-sched.sch_delay.max.ms.blk_mq_freeze_queue_wait.lo_release.__blkdev_put.blkdev_put
      0.48 ± 43%   +2728.2%      13.53 ± 71%  perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.05 ± 23%    -100.0%       0.00        perf-sched.sch_delay.max.ms.devtmpfs_work_loop.devtmpfsd.kthread.ret_from_fork
      0.06 ± 30%     -40.7%       0.04 ± 47%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.40 ± 48%     -56.5%       0.17 ± 25%  perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.17 ± 43%     -67.6%       0.05 ± 15%  perf-sched.sch_delay.max.ms.io_schedule.put_and_wait_on_page_locked.filemap_get_pages.filemap_read
      1.02 ± 54%     -77.0%       0.23 ± 50%  perf-sched.sch_delay.max.ms.kthread_worker_fn.kthread.ret_from_fork
      0.12 ± 60%     -63.6%       0.04 ± 26%  perf-sched.sch_delay.max.ms.kthreadd.ret_from_fork
      3.46 ±139%    -100.0%       0.00        perf-sched.sch_delay.max.ms.percpu_down_write._cpu_down.cpu_down.device_offline
      0.43 ±  4%    -100.0%       0.00        perf-sched.sch_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
     13.41 ±140%    -100.0%       0.00        perf-sched.sch_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_shepherd
      0.77 ±131%    -100.0%       0.00        perf-sched.sch_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_start
    351.09 ±134%     -96.6%      11.92 ± 14%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.02 ± 20%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.pte_alloc_one.__pte_alloc
      0.48 ± 77%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__flush_work.lru_add_drain_all.khugepaged
      0.03         +4333.7%       1.32 ±118%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_lookup
      0.04 ± 35%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xlog_alloc_log
      0.13 ± 86%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.simple_recursive_removal.debugfs_remove
      0.02 ± 73%     +91.1%       0.03 ± 21%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file
      0.02 ±  9%     -73.1%       0.01 ±141%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma
      2.53 ±133%     -98.6%       0.04 ± 20%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      0.04 ± 25%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.crypto_exit_scomp_ops_async.crypto_destroy_tfm
      0.03 ± 28%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.device_del.device_unregister
      0.03 ±  2%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.flush_workqueue.xfs_buftarg_drain
      0.22 ± 45%    +874.8%       2.19 ± 34%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.synchronize_rcu_expedited.namespace_unlock.path_umount
      0.03 ±  7%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.vfree.free_vm_stack_cache.cpuhp_invoke_callback
      4.56 ±138%     -98.3%       0.08 ± 27%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      0.12 ± 40%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion_io_timeout.submit_bio_wait.xfs_rw_bdev
      0.13 ± 55%     -62.4%       0.05 ± 14%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.write_cache_pages.iomap_writepages.xfs_vm_writepages
      0.04 ± 28%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.write_inode_now.iput.xfs_rtunmount_inodes
      0.09 ± 45%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rescuer_thread.kthread.ret_from_fork
      0.04 ± 21%    +336.4%       0.19 ± 57%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_create
     15.73 ± 34%     -98.8%       0.19 ± 80%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      4.15 ±112%     -97.0%       0.12 ± 17%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.04          -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.blkdev_get_by_dev
      0.06 ± 23%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__blkdev_direct_IO_simple.generic_file_direct_write
      0.04 ±  3%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__blkdev_direct_IO_simple.generic_file_read_iter
      0.05 ±  3%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.sync_rcu_exp_select_cpus
      0.78 ±  4%     -33.4%       0.52 ± 37%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
      0.03 ±  7%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
      0.61 ± 39%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__wait_rcu_gp.synchronize_rcu
      0.33 ±121%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.cpuhp_kick_ap.cpuhp_kick_ap_work
      0.05 ±  8%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.devtmpfs_submit_req.devtmpfs_delete_node
      0.02 ±  3%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.kthread_stop.destroy_workqueue
      0.05 ± 20%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.kthread_stop.xfs_trans_ail_destroy
      0.55 ±121%     -92.2%       0.04 ± 23%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.xfs_buf_iowait.__xfs_buf_submit
      0.05 ± 70%    +113.4%       0.12 ±  6%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.xlog_cil_push_work.process_one_work
     27.32 ± 21%     -65.1%       9.52 ± 91%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.17 ± 35%     -56.5%       0.08 ± 34%  perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
     23.62 ± 24%     +45.6%      34.39 ± 12%  perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.09 ± 34%    -100.0%       0.00        perf-sched.sch_delay.max.ms.xfs_ail_push_all_sync.xfs_log_quiesce.xfs_log_unmount.xfs_unmountfs
      0.05 ± 51%    -100.0%       0.00        perf-sched.sch_delay.max.ms.xfsaild.kthread.ret_from_fork
      0.09 ± 99%    -100.0%       0.00        perf-sched.sch_delay.max.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.__xfs_trans_commit
      0.02 ± 11%    -100.0%       0.00        perf-sched.sch_delay.max.ms.xlog_wait_on_iclog.xfs_log_unmount_write.xfs_log_unmount.xfs_unmountfs
      0.06 ± 20%    +323.2%       0.27 ± 51%  perf-sched.sch_delay.max.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
     36572 ±  2%     +17.3%      42889 ±  3%  perf-sched.total_wait_and_delay.count.ms
    798.63 ±  5%      +8.4%     865.36 ±  5%  perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    148.76 ± 20%    +252.3%     524.14 ± 15%  perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    152.01 ± 21%    +262.1%     550.43 ± 21%  perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
     40.46 ± 30%     -29.3%      28.62 ± 18%  perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      5.82 ± 71%     -84.4%       0.91 ± 41%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     78.40 ± 39%    +339.2%     344.36 ± 55%  perf-sched.wait_and_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      3.92           -33.2%       2.62        perf-sched.wait_and_delay.avg.ms.kthread_worker_fn.kthread.ret_from_fork
    229.81 ± 68%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
     95.84 ± 31%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_shepherd
     42.95 ±  4%     +45.0%      62.29 ± 12%  perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     11.98 ±  3%     +19.5%      14.31        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
     60.36 ± 22%     -71.3%      17.32        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.xlog_cil_push_work.process_one_work
      0.35 ± 52%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.write_cache_pages.iomap_writepages.xfs_vm_writepages
      6.15 ± 12%    +111.7%      13.01 ± 11%  perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    129.71 ± 14%    +100.3%     259.79 ± 15%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    230.79           +44.1%     332.67 ±  9%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    240.28          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.isra.0.blkdev_get_by_dev
      6.59 ± 38%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
      4.36 ± 72%    +160.4%      11.35 ±  4%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
     70.14          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
      0.08 ± 28%   +2103.6%       1.84 ±  7%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_for_completion.xlog_cil_push_work.process_one_work
     60.94 ±  5%     -25.2%      45.61        perf-sched.wait_and_delay.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
    293.56 ±  4%     +83.5%     538.81        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    134.37 ±  7%     -26.7%      98.46 ±  2%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
    102.90 ± 14%     -85.2%      15.25 ±141%  perf-sched.wait_and_delay.avg.ms.xlog_cil_force_lsn.xfs_log_force.xfsaild_push.xfsaild
      4.26 ±141%    +270.7%      15.78 ±  2%  perf-sched.wait_and_delay.avg.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
     95.02 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
     34.33 ±  3%     -72.8%       9.33 ± 13%  perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     33.67 ±  3%     -73.3%       9.00 ± 15%  perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
    301.67 ±  7%     -18.3%     246.33        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    997.33 ± 15%     -34.6%     652.00 ± 26%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     57.33 ±  7%     -89.5%       6.00 ± 23%  perf-sched.wait_and_delay.count.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
    380.33 ± 14%    +187.2%       1092 ±  4%  perf-sched.wait_and_delay.count.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range
    547.33 ±  6%    +126.9%       1242 ±  3%  perf-sched.wait_and_delay.count.kthread_worker_fn.kthread.ret_from_fork
     13.67 ±  9%     -65.9%       4.67 ± 53%  perf-sched.wait_and_delay.count.kthreadd.ret_from_fork
     13.00 ± 10%    -100.0%       0.00        perf-sched.wait_and_delay.count.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
      1.67 ± 28%    -100.0%       0.00        perf-sched.wait_and_delay.count.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_shepherd
      3699 ±  5%     -28.4%       2647 ± 11%  perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
    205.33 ± 10%     +84.4%     378.67        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.wait_for_completion.xlog_cil_push_work.process_one_work
    378.67 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.write_cache_pages.iomap_writepages.xfs_vm_writepages
     14154 ± 18%     +47.8%      20918 ±  4%  perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    284.67 ±  5%     -92.9%      20.33 ±  6%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     90.00 ±  3%     -36.7%      57.00 ± 10%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      1.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_preempt_disabled.__mutex_lock.isra.0.blkdev_get_by_dev
    440.67 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
    137.33 ± 71%    +184.7%     391.00        perf-sched.wait_and_delay.count.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
      2.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
    206.33 ± 10%     +98.2%     409.00        perf-sched.wait_and_delay.count.schedule_timeout.wait_for_completion.xlog_cil_push_work.process_one_work
      2237 ±  6%     -47.9%       1165        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
      4070 ±  4%     +59.4%       6488 ±  2%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
     66.33 ±141%    +319.6%     278.33        perf-sched.wait_and_delay.count.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
     10.33 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.count.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
      3711 ±  3%     -57.2%       1590 ± 26%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      2889 ± 92%     -90.0%     289.85 ± 84%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
    232.01           -64.4%      82.63 ± 34%  perf-sched.wait_and_delay.max.ms.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range
     74.01 ± 25%     -42.0%      42.95 ± 16%  perf-sched.wait_and_delay.max.ms.kthread_worker_fn.kthread.ret_from_fork
      1110 ± 55%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
    138.50 ± 42%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_shepherd
      1681 ± 24%     -89.9%     170.60 ± 18%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.xlog_cil_push_work.process_one_work
     29.22 ± 63%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.write_cache_pages.iomap_writepages.xfs_vm_writepages
      5403 ± 25%     -54.9%       2435 ±  2%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      1544 ± 24%     -35.2%       1000        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    240.28          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.blkdev_get_by_dev
    822.36 ±103%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
    139.05          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
      5.80 ± 77%   +1110.3%      70.22 ±  8%  perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_for_completion.xlog_cil_push_work.process_one_work
      2017           -96.0%      81.57 ± 13%  perf-sched.wait_and_delay.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
    137.05 ±  8%     -82.3%      24.28 ±141%  perf-sched.wait_and_delay.max.ms.xlog_cil_force_lsn.xfs_log_force.xfsaild_push.xfsaild
     12.48 ±141%    +279.5%      47.38 ± 13%  perf-sched.wait_and_delay.max.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
    148.05 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
    798.58 ±  5%      +8.4%     865.32 ±  5%  perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     13.03 ±  7%    -100.0%       0.00        perf-sched.wait_time.avg.ms.blk_mq_freeze_queue_wait.lo_release.__blkdev_put.blkdev_put
    148.70 ± 20%    +251.4%     522.52 ± 16%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     43.36 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.devtmpfs_work_loop.devtmpfsd.kthread.ret_from_fork
    151.92 ± 21%    +260.9%     548.23 ± 21%  perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
     69.92 ±105%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.kthread.ret_from_fork
     40.39 ± 30%     -29.3%      28.57 ± 18%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     23.00 ± 89%     -87.8%       2.80 ± 61%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      5.81 ± 71%     -84.5%       0.90 ± 40%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     78.32 ± 39%    +339.5%     344.26 ± 55%  perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
     10.20 ±  7%     -56.6%       4.42 ± 69%  perf-sched.wait_time.avg.ms.io_schedule.put_and_wait_on_page_locked.filemap_get_pages.filemap_read
      2.95 ± 88%    +607.1%      20.89 ± 10%  perf-sched.wait_time.avg.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
      3.88           -33.3%       2.59        perf-sched.wait_time.avg.ms.kthread_worker_fn.kthread.ret_from_fork
      6.66 ± 56%    -100.0%       0.00        perf-sched.wait_time.avg.ms.percpu_down_write._cpu_down.cpu_down.device_offline
    229.56 ± 68%    -100.0%       0.00        perf-sched.wait_time.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
     89.11 ± 37%    -100.0%       0.00        perf-sched.wait_time.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_shepherd
     42.81 ±  4%     +45.4%      62.23 ± 12%  perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.36 ±129%     -93.0%       0.03 ± 13%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__get_user_pages.__get_user_pages_remote.get_arg_page
      5.45 ± 37%    +113.1%      11.61 ± 36%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
      0.10 ± 14%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xlog_alloc_log
      0.01 ±141%    +437.5%       0.06 ± 51%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.apparmor_file_alloc_security.security_file_alloc.__alloc_file
      0.01 ±141%  +11751.4%       1.38 ±126%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_from_iter.pipe_write.new_sync_write
      0.01 ±141%    +988.2%       0.06 ± 79%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.filemap_read.new_sync_read
      0.27 ± 62%     -89.9%       0.03 ± 26%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_strings.isra.0
     21.74 ± 85%     -97.3%       0.59 ± 36%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.02 ± 71%   +2686.0%       0.53 ±133%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_read.path_openat.do_filp_open
     17.05 ± 72%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.simple_recursive_removal.debugfs_remove
      0.02 ± 71%    +146.9%       0.05 ± 47%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.map_vdso.load_elf_binary
      0.03 ± 15%     -31.0%       0.02 ± 29%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.nd_jump_root.pick_link
     63.11 ±141%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
     10.85 ± 25%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.simple_recursive_removal.debugfs_remove
      0.03 ± 10%     -17.6%       0.02 ± 14%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.exit_mmap.mmput.begin_new_exec
      0.86 ±137%     -99.0%       0.01 ±141%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.getname_flags.user_path_at_empty
      6.61 ±133%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.prepare_creds.copy_creds
     17.45 ± 42%    +109.1%      36.50 ± 39%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
      7.07 ± 20%     +95.7%      13.84 ± 35%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      0.52 ± 64%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.crypto_exit_scomp_ops_async.crypto_destroy_tfm
      0.04 ± 20%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.device_del.device_unregister
    230.88 ± 65%     -76.6%      53.99 ±141%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.drm_gem_shmem_vunmap.drm_gem_vunmap
      0.09 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.flush_workqueue.xfs_buftarg_drain
      0.36 ± 28%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.perf_event_exit_cpu_context.perf_event_exit_cpu
      0.90 ±  8%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.slub_cpu_dead.cpuhp_invoke_callback
      9.12 ± 12%     +72.4%      15.72 ± 11%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.process_one_work.worker_thread.kthread
      0.04 ± 49%     -75.0%       0.01 ±141%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.mmap_region
      0.06 ± 69%     +88.5%       0.12 ± 42%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.exit_mmap.mmput
      0.62 ± 56%    +131.7%       1.43 ± 45%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      8.01 ±  6%     -48.8%       4.10 ±  3%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      1.04 ± 10%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.vfree.free_vm_stack_cache.cpuhp_invoke_callback
      1.26 ± 87%     -98.7%       0.02 ±141%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.vfs_write.ksys_write.do_syscall_64
     11.97 ±  3%     +19.5%      14.30        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      0.03 ±  3%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.stop_machine_cpuslocked.takedown_cpu
      1.90 ±141%   +2221.3%      44.14 ± 36%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.xfs_btree_split.xfs_btree_make_block_unfull
     60.31 ± 22%     -71.3%      17.28        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.xlog_cil_push_work.process_one_work
      0.18 ± 16%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion_io_timeout.submit_bio_wait.xfs_rw_bdev
      0.42          -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.write_inode_now.iput.xfs_rtunmount_inodes
      0.07 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rescuer_thread.kthread.ret_from_fork
     21.26 ± 15%     +30.6%      27.76 ±  8%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_create
      0.10 ± 48%     -86.2%       0.01 ±141%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.xfs_map_blocks.iomap_writepage_map.write_cache_pages
      6.08 ± 13%    +112.8%      12.94 ± 11%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    129.60 ± 14%    +100.4%     259.73 ± 15%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    230.69           +44.2%     332.65 ±  9%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    240.24          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.isra.0.blkdev_get_by_dev
      5.01 ± 80%    +648.5%      37.53 ±  6%  perf-sched.wait_time.avg.ms.schedule_timeout.__down.down.xfs_buf_lock
      0.46          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.__blkdev_direct_IO_simple.generic_file_direct_write
      6.74 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.__blkdev_direct_IO_simple.generic_file_read_iter
      6.56 ± 38%     -81.2%       1.24 ± 26%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
      0.04 ± 15%     -45.2%       0.02 ± 14%  perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      6.46 ±  9%     +74.6%      11.28 ±  4%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
     70.12          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
    148.48 ± 98%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.__wait_rcu_gp.synchronize_rcu
     45.56 ± 10%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.cpuhp_kick_ap.cpuhp_kick_ap_work
      8.06 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.devtmpfs_submit_req.devtmpfs_delete_node
     17.37 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.kthread_park.takedown_cpu
      0.03 ±  7%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.kthread_stop.destroy_workqueue
     35.73 ± 11%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.kthread_stop.xfs_trans_ail_destroy
      1.69 ±  6%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.stop_machine_cpuslocked.takedown_cpu
      2.04          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.takedown_cpu.cpuhp_invoke_callback
      5.50 ±  3%    +149.9%      13.75 ± 32%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.xfs_buf_iowait.__xfs_buf_submit
      0.08 ± 28%   +2098.8%       1.84 ±  7%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.xlog_cil_push_work.process_one_work
      9.28 ± 17%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
     60.87 ±  5%     -25.2%      45.56        perf-sched.wait_time.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
    293.47 ±  4%     +83.6%     538.76        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.14 ±  3%    -100.0%       0.00        perf-sched.wait_time.avg.ms.synchronize_rcu_expedited.namespace_unlock.path_umount.ksys_umount
    134.29 ±  7%     -26.8%      98.37 ±  2%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
     24.70 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.xfs_ail_push_all_sync.xfs_log_quiesce.xfs_log_unmount.xfs_unmountfs
     20.17 ± 72%    -100.0%       0.00        perf-sched.wait_time.avg.ms.xfsaild.kthread.ret_from_fork
    102.86 ± 14%     -59.0%      42.19 ±  9%  perf-sched.wait_time.avg.ms.xlog_cil_force_lsn.xfs_log_force.xfsaild_push.xfsaild
     14.35 ± 22%     +69.9%      24.39 ± 11%  perf-sched.wait_time.avg.ms.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve
     13.80 ±  5%     -41.7%       8.05        perf-sched.wait_time.avg.ms.xlog_state_get_iclog_space.xlog_write.xlog_cil_push_work.process_one_work
     11.47 ± 32%    -100.0%       0.00        perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.__xfs_trans_commit
     12.51           +25.8%      15.74 ±  2%  perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      0.18 ± 18%    -100.0%       0.00        perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xfs_log_unmount_write.xfs_log_unmount.xfs_unmountfs
     94.98 ±  8%     -56.4%      41.38 ±  4%  perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
     23.03 ±  5%     -96.4%       0.83 ± 21%  perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
     45.72 ±  8%    -100.0%       0.00        perf-sched.wait_time.max.ms.blk_mq_freeze_queue_wait.lo_release.__blkdev_put.blkdev_put
    179.00 ± 27%    -100.0%       0.00        perf-sched.wait_time.max.ms.devtmpfs_work_loop.devtmpfsd.kthread.ret_from_fork
      3710 ±  3%     -57.1%       1590 ± 26%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    486.28 ±106%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.kthread.ret_from_fork
    396.63 ±108%     -95.7%      16.96 ± 65%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      2889 ± 92%     -90.0%     289.85 ± 84%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
    372.65           -97.4%       9.69 ± 39%  perf-sched.wait_time.max.ms.io_schedule.put_and_wait_on_page_locked.filemap_get_pages.filemap_read
    231.98           -64.4%      82.59 ± 34%  perf-sched.wait_time.max.ms.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range
     73.96 ± 25%     -42.0%      42.89 ± 15%  perf-sched.wait_time.max.ms.kthread_worker_fn.kthread.ret_from_fork
     57.20 ± 51%    -100.0%       0.00        perf-sched.wait_time.max.ms.percpu_down_write._cpu_down.cpu_down.device_offline
      1110 ± 55%    -100.0%       0.00        perf-sched.wait_time.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
    125.09 ± 48%    -100.0%       0.00        perf-sched.wait_time.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.vmstat_shepherd
    191.52 ± 77%    +662.1%       1459 ± 91%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.shmem_alloc_page
      0.04 ± 13%    +166.4%       0.10 ± 54%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.wp_page_copy
      1.71 ±137%     -98.1%       0.03 ±  3%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__get_user_pages.__get_user_pages_remote.get_arg_page
     18.74 ± 42%    +370.6%      88.21 ± 52%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
     16.66 ± 53%    +181.4%      46.90 ± 42%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_lookup
      0.10 ± 14%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xlog_alloc_log
      0.01 ±141%    +437.5%       0.06 ± 51%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.apparmor_file_alloc_security.security_file_alloc.__alloc_file
      0.01 ±141%  +19957.1%       2.34 ±114%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_page_from_iter.pipe_write.new_sync_write
      3.28 ± 75%     -97.5%       0.08 ± 77%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_strings.isra.0
    533.33 ± 66%     -98.3%       8.86 ± 45%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.03 ± 12%    +498.9%       0.19 ±103%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.__vma_adjust.shift_arg_pages
     19.17 ± 70%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.simple_recursive_removal.debugfs_remove
      0.07 ± 12%     -54.1%       0.03 ±  5%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.unlink_anon_vmas.free_pgtables
      0.02 ± 71%    +503.1%       0.13 ±102%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.map_vdso.load_elf_binary
     13.50 ±  5%   +2558.0%     358.82 ±126%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
    126.20 ±141%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
     28.66 ± 15%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.simple_recursive_removal.debugfs_remove
      1.70 ±139%     -99.5%       0.01 ±141%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.getname_flags.user_path_at_empty
      6.61 ±133%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.prepare_creds.copy_creds
      2.64 ±137%    +137.1%       6.27 ± 72%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.security_file_alloc.__alloc_file
     21.30 ± 12%    +544.1%     137.19 ± 41%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      0.03 ±  5%     -67.4%       0.01 ±141%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mmput.begin_new_exec.load_elf_binary
      1.14 ± 73%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.crypto_exit_scomp_ops_async.crypto_destroy_tfm
      0.04 ± 25%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.device_del.device_unregister
    362.97 ± 56%     -85.1%      53.99 ±141%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.drm_gem_shmem_vunmap.drm_gem_vunmap
      0.09 ±  4%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.flush_workqueue.xfs_buftarg_drain
      0.55 ± 47%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.perf_event_exit_cpu_context.perf_event_exit_cpu
      1.81 ±  8%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.slub_cpu_dead.cpuhp_invoke_callback
    382.39 ± 20%    +114.1%     818.63        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.process_one_work.worker_thread.kthread
      0.04 ± 36%     -77.4%       0.01 ±141%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.mmap_region
      1.93 ±  9%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.vfree.free_vm_stack_cache.cpuhp_invoke_callback
      1.27 ± 86%     -98.7%       0.02 ±141%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.vfs_write.ksys_write.do_syscall_64
      0.05 ± 13%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.stop_machine_cpuslocked.takedown_cpu
      1.90 ±141%   +3242.9%      63.56 ± 37%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.xfs_btree_split.xfs_btree_make_block_unfull
      1681 ± 24%     -89.9%     170.56 ± 18%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.xlog_cil_push_work.process_one_work
      0.56 ± 26%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion_io_timeout.submit_bio_wait.xfs_rw_bdev
      0.42          -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.write_inode_now.iput.xfs_rtunmount_inodes
      0.12 ± 39%    -100.0%       0.00        perf-sched.wait_time.max.ms.rescuer_thread.kthread.ret_from_fork
      5.23 ±137%    +408.0%      26.59 ± 79%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
      0.48 ± 63%     -93.6%       0.03 ±141%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_map_blocks.iomap_writepage_map.write_cache_pages
    135.13 ± 44%     +50.3%     203.13 ± 17%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      5403 ± 25%     -54.9%       2435 ±  2%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      1541 ± 24%     -35.1%       1000        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    240.24          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.blkdev_get_by_dev
     60.21 ± 56%    +157.4%     154.98 ± 39%  perf-sched.wait_time.max.ms.schedule_timeout.__down.down.xfs_buf_lock
      1.51 ±  3%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.__blkdev_direct_IO_simple.generic_file_direct_write
     19.99 ±  9%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.__blkdev_direct_IO_simple.generic_file_read_iter
    822.32 ±103%     -97.5%      20.19 ± 22%  perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
      0.04 ± 15%     -45.2%       0.02 ± 14%  perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
    169.84 ± 12%     -48.0%      88.33 ±  9%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
    139.02          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
      1305 ± 72%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.__wait_rcu_gp.synchronize_rcu
    105.74 ± 23%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.cpuhp_kick_ap.cpuhp_kick_ap_work
     30.33 ±  3%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.devtmpfs_submit_req.devtmpfs_delete_node
     31.52 ±  3%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.kthread_park.takedown_cpu
      0.05 ± 29%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.kthread_stop.destroy_workqueue
     35.73 ± 11%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.kthread_stop.xfs_trans_ail_destroy
      2.03          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.stop_machine_cpuslocked.takedown_cpu
      2.04          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.takedown_cpu.cpuhp_invoke_callback
    142.32 ±  2%     -71.3%      40.86 ± 54%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.xfs_buf_iowait.__xfs_buf_submit
      5.75 ± 77%   +1121.0%      70.17 ±  8%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.xlog_cil_push_work.process_one_work
     45.42 ±  2%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
      2017           -96.0%      81.50 ± 13%  perf-sched.wait_time.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
      0.14 ±  3%    -100.0%       0.00        perf-sched.wait_time.max.ms.synchronize_rcu_expedited.namespace_unlock.path_umount.ksys_umount
     26.83 ± 10%    -100.0%       0.00        perf-sched.wait_time.max.ms.xfs_ail_push_all_sync.xfs_log_quiesce.xfs_log_unmount.xfs_unmountfs
     34.55 ± 70%    -100.0%       0.00        perf-sched.wait_time.max.ms.xfsaild.kthread.ret_from_fork
    137.01 ±  8%     -33.2%      91.58 ± 22%  perf-sched.wait_time.max.ms.xlog_cil_force_lsn.xfs_log_force.xfsaild_push.xfsaild
     56.24 ± 36%    +130.5%     129.66 ± 27%  perf-sched.wait_time.max.ms.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve
     21.56 ± 35%    -100.0%       0.00        perf-sched.wait_time.max.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.__xfs_trans_commit
     36.56 ±  8%     +29.5%      47.34 ± 13%  perf-sched.wait_time.max.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      0.18 ± 18%    -100.0%       0.00        perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xfs_log_unmount_write.xfs_log_unmount.xfs_unmountfs
    148.02 ±  6%     -38.1%      91.67 ±  3%  perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
     68.26 ± 14%     -63.8%      24.72        perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread



***************************************************************************************************
lkp-knm02: 272 threads 2 sockets Intel(R) Xeon Phi(TM) CPU 7255 @ 1.10GHz with 112G memory
=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/bufferedio/1HDD/xfs/x86_64-rhel-8.3/hdd/debian-10.4-x86_64-20200603.cgz/lkp-knm02/MWUL/fxmark/0x11

commit: 
  3468bb1ca6 ("xfs: remove need_start_rec parameter from xlog_write()")
  eef983ffea ("xfs: journal IO cache flush reductions")

3468bb1ca6e88407 eef983ffeae7a1cdde8c3338155 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     85.12           -14.8%      72.56 ±  2%  fxmark.hdd_xfs_MWUL_18_bufferedio.idle_sec
      0.20 ±  4%     -52.5%       0.09 ±  5%  fxmark.hdd_xfs_MWUL_18_bufferedio.iowait_sec
      0.19 ±  4%     -45.6%       0.10 ±  7%  fxmark.hdd_xfs_MWUL_18_bufferedio.iowait_util
      2.89           -10.2%       2.59 ±  2%  fxmark.hdd_xfs_MWUL_18_bufferedio.irq_sec
      5.98           -12.4%       5.23 ±  2%  fxmark.hdd_xfs_MWUL_18_bufferedio.real_sec
      5.66           -13.0%       4.93 ±  2%  fxmark.hdd_xfs_MWUL_18_bufferedio.secs
      0.46 ±  2%     -24.1%       0.35 ±  2%  fxmark.hdd_xfs_MWUL_18_bufferedio.softirq_sec
      0.43           -13.1%       0.37        fxmark.hdd_xfs_MWUL_18_bufferedio.softirq_util
     14.83           +13.0%      16.75 ±  2%  fxmark.hdd_xfs_MWUL_18_bufferedio.sys_util
      9185           +15.0%      10566 ±  2%  fxmark.hdd_xfs_MWUL_18_bufferedio.works/sec
      0.19 ± 10%     -15.5%       0.16 ±  7%  fxmark.hdd_xfs_MWUL_1_bufferedio.softirq_sec
      1.63 ± 10%     -14.4%       1.39 ±  7%  fxmark.hdd_xfs_MWUL_1_bufferedio.softirq_util
      0.65 ± 36%     -56.9%       0.28 ± 26%  fxmark.hdd_xfs_MWUL_2_bufferedio.idle_sec
      4.81 ± 35%     -54.5%       2.19 ± 26%  fxmark.hdd_xfs_MWUL_2_bufferedio.idle_util
      0.17            -9.8%       0.15 ±  3%  fxmark.hdd_xfs_MWUL_2_bufferedio.softirq_sec
      0.40 ± 15%     -38.7%       0.24        fxmark.hdd_xfs_MWUL_36_bufferedio.iowait_sec
      0.11 ±  9%     -45.0%       0.06 ±  2%  fxmark.hdd_xfs_MWUL_36_bufferedio.iowait_util
      5.29 ± 19%     -64.7%       1.87 ± 22%  fxmark.hdd_xfs_MWUL_4_bufferedio.idle_sec
     25.40 ± 14%     -55.1%      11.39 ± 19%  fxmark.hdd_xfs_MWUL_4_bufferedio.idle_util
      0.07 ± 44%    -100.0%       0.00        fxmark.hdd_xfs_MWUL_4_bufferedio.iowait_sec
      0.35 ± 42%    -100.0%       0.00        fxmark.hdd_xfs_MWUL_4_bufferedio.iowait_util
      0.96 ±  2%     -16.3%       0.81 ±  2%  fxmark.hdd_xfs_MWUL_4_bufferedio.irq_sec
      5.28 ±  5%     -22.0%       4.12 ±  3%  fxmark.hdd_xfs_MWUL_4_bufferedio.real_sec
      4.96 ±  5%     -23.7%       3.78 ±  2%  fxmark.hdd_xfs_MWUL_4_bufferedio.secs
      0.20 ± 14%     -48.3%       0.10 ±  9%  fxmark.hdd_xfs_MWUL_4_bufferedio.softirq_sec
      0.96 ± 10%     -34.1%       0.63 ±  8%  fxmark.hdd_xfs_MWUL_4_bufferedio.softirq_util
     64.31 ±  5%     +21.5%      78.13 ±  2%  fxmark.hdd_xfs_MWUL_4_bufferedio.sys_util
      4.31 ± 10%     +13.5%       4.89 ±  3%  fxmark.hdd_xfs_MWUL_4_bufferedio.user_util
     10394 ±  6%     +30.5%      13568 ±  2%  fxmark.hdd_xfs_MWUL_4_bufferedio.works/sec
    848.97 ±  7%     -16.1%     712.05 ±  6%  fxmark.hdd_xfs_MWUL_54_bufferedio.idle_sec
      0.75 ±  6%     -22.3%       0.58 ±  6%  fxmark.hdd_xfs_MWUL_54_bufferedio.iowait_sec
     21.07 ±  5%     -18.1%      17.25 ±  7%  fxmark.hdd_xfs_MWUL_54_bufferedio.irq_sec
     16.91 ±  6%     -16.3%      14.16 ±  6%  fxmark.hdd_xfs_MWUL_54_bufferedio.real_sec
     16.85 ±  5%     -17.0%      13.98 ±  7%  fxmark.hdd_xfs_MWUL_54_bufferedio.secs
      2.59 ±  5%     -22.0%       2.02 ±  7%  fxmark.hdd_xfs_MWUL_54_bufferedio.softirq_sec
      7.75           -58.0%       3.26 ±  2%  fxmark.hdd_xfs_MWUL_54_bufferedio.user_sec
      0.86 ±  8%     -49.9%       0.43 ±  5%  fxmark.hdd_xfs_MWUL_54_bufferedio.user_util
      3185 ±  6%     +20.3%       3832 ±  7%  fxmark.hdd_xfs_MWUL_54_bufferedio.works/sec
      1309 ±  3%     -14.0%       1126 ±  2%  fxmark.hdd_xfs_MWUL_72_bufferedio.idle_sec
      0.09           +12.8%       0.10 ±  3%  fxmark.hdd_xfs_MWUL_72_bufferedio.iowait_util
     30.04 ±  2%     -11.9%      26.47 ±  2%  fxmark.hdd_xfs_MWUL_72_bufferedio.irq_sec
     19.10 ±  3%     -13.6%      16.49 ±  2%  fxmark.hdd_xfs_MWUL_72_bufferedio.real_sec
     19.61 ±  4%     -15.2%      16.64 ±  2%  fxmark.hdd_xfs_MWUL_72_bufferedio.secs
      3.10 ±  3%      -9.7%       2.80        fxmark.hdd_xfs_MWUL_72_bufferedio.softirq_sec
      1.60 ±  4%     +16.2%       1.86        fxmark.hdd_xfs_MWUL_72_bufferedio.sys_util
      0.33 ±  2%     +13.9%       0.37        fxmark.hdd_xfs_MWUL_72_bufferedio.user_util
      2774 ±  4%     +17.4%       3257 ±  2%  fxmark.hdd_xfs_MWUL_72_bufferedio.works/sec
    273.82            -5.5%     258.66        fxmark.time.elapsed_time
    273.82            -5.5%     258.66        fxmark.time.elapsed_time.max
     22.00            +4.5%      23.00        fxmark.time.percent_of_cpu_this_job_got
     47980 ±  2%     -22.5%      37196 ±  2%  fxmark.time.voluntary_context_switches
      0.11 ±  3%      -0.0        0.09 ±  2%  mpstat.cpu.all.iowait%
      9432 ±  9%     +28.9%      12160 ± 22%  numa-meminfo.node1.KReclaimable
      9432 ±  9%     +28.9%      12160 ± 22%  numa-meminfo.node1.SReclaimable
     56.60            -3.1%      54.84        iostat.cpu.idle
     20.21            +3.7%      20.96        iostat.cpu.system
      2.76            -5.3%       2.61        iostat.cpu.user
    138197           +32.4%     182972        numa-vmstat.node0.nr_dirtied
    138039           +32.4%     182774        numa-vmstat.node0.nr_written
      2358 ±  9%     +28.9%       3040 ± 22%  numa-vmstat.node1.nr_slab_reclaimable
      7134           +39.6%       9960        vmstat.io.bo
     10348            +3.4%      10700        vmstat.system.cs
    131002            +3.7%     135808        vmstat.system.in
      1471 ±  2%     +12.4%       1653        slabinfo.PING.num_objs
      6396           +16.0%       7418        slabinfo.proc_inode_cache.active_objs
      6863           +17.8%       8084        slabinfo.proc_inode_cache.num_objs
    715.67 ±  5%     -19.4%     576.67 ±  8%  slabinfo.skbuff_fclone_cache.active_objs
    923.67 ±  3%     -16.0%     776.00 ±  5%  slabinfo.skbuff_fclone_cache.num_objs
      9068            +2.7%       9312        perf-stat.i.context-switches
     58455            +2.9%      60128        perf-stat.i.cpu-clock
    110.25            +1.5%     111.90        perf-stat.i.cpu-migrations
      2.32 ±  4%      +4.5%       2.42        perf-stat.i.major-faults
      2771            +3.3%       2862        perf-stat.i.minor-faults
      2774            +3.3%       2864        perf-stat.i.page-faults
     58455            +2.9%      60128        perf-stat.i.task-clock
      9024            +2.5%       9249        perf-stat.ps.context-switches
     58710            +2.6%      60231        perf-stat.ps.cpu-clock
    110.31            +1.5%     111.98        perf-stat.ps.cpu-migrations
      2757            +3.1%       2843        perf-stat.ps.minor-faults
      2759            +3.1%       2845        perf-stat.ps.page-faults
     58709            +2.6%      60231        perf-stat.ps.task-clock
     70145            +0.9%      70790        proc-vmstat.nr_anon_pages
    231942           +30.2%     301999        proc-vmstat.nr_dirtied
      8977            +3.1%       9253        proc-vmstat.nr_inactive_file
     61051            +1.3%      61816        proc-vmstat.nr_slab_unreclaimable
    231942           +30.2%     301999        proc-vmstat.nr_written
      8977            +3.1%       9253        proc-vmstat.nr_zone_inactive_file
    187.33 ± 12%     +22.1%     228.67 ± 11%  proc-vmstat.nr_zone_write_pending
   1372660            -1.9%    1346336        proc-vmstat.numa_hit
   1372618            -1.9%    1346286        proc-vmstat.numa_local
     24895 ±  7%     -59.3%      10129 ±104%  proc-vmstat.pgactivate
   1631337            -1.4%    1608916        proc-vmstat.pgalloc_normal
    930890            -2.1%     911419        proc-vmstat.pgfault
   1575367            -1.7%    1548619        proc-vmstat.pgfree
   1850163           +30.6%    2417156        proc-vmstat.pgpgout
     93731            -2.5%      91424        proc-vmstat.pgreuse
     28009 ± 13%     -15.9%      23545        softirqs.CPU1.RCU
     28414 ±  3%     -11.3%      25216        softirqs.CPU1.SCHED
     20483           -15.4%      17328 ±  7%  softirqs.CPU11.SCHED
     25094           -11.3%      22262        softirqs.CPU2.SCHED
     23633 ±  2%     -10.6%      21131 ±  4%  softirqs.CPU3.SCHED
     16271           -13.9%      14017 ±  3%  softirqs.CPU34.SCHED
     14823 ±  2%     -10.7%      13235 ±  2%  softirqs.CPU36.SCHED
     15230 ±  2%     -11.1%      13533 ±  4%  softirqs.CPU37.SCHED
     15258 ±  3%     -13.7%      13163 ±  3%  softirqs.CPU41.SCHED
     14601            -9.6%      13193        softirqs.CPU42.SCHED
     11341 ±  2%      -9.6%      10251 ±  3%  softirqs.CPU48.RCU
     21521            -9.3%      19514 ±  2%  softirqs.CPU5.SCHED
     14550           -11.1%      12936 ±  2%  softirqs.CPU50.SCHED
     11269 ±  4%      -9.8%      10163 ±  3%  softirqs.CPU51.RCU
     12578 ± 11%     -17.3%      10407 ±  4%  softirqs.CPU53.RCU
     14481 ±  2%      -9.0%      13182 ±  2%  softirqs.CPU53.SCHED
     10095           -11.6%       8922 ±  7%  softirqs.CPU55.SCHED
      7828 ±  2%     +38.5%      10841 ± 19%  softirqs.CPU56.RCU
     37535           -25.6%      27919 ±  2%  interrupts.171:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
    968498            +1.0%     978318        interrupts.CAL:Function_call_interrupts
      3150           -48.5%       1621        interrupts.CPU0.171:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
    660.33 ± 12%     -34.9%     430.00 ± 37%  interrupts.CPU0.36:IR-PCI-MSI.2621442-edge.eth1-TxRx-1
     10125 ±  6%      -7.8%       9335 ±  7%  interrupts.CPU0.CAL:Function_call_interrupts
    270.33 ± 39%    +230.0%     892.00 ± 21%  interrupts.CPU0.NMI:Non-maskable_interrupts
    270.33 ± 39%    +230.0%     892.00 ± 21%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
    237.67 ±  6%    +123.8%     532.00 ± 32%  interrupts.CPU1.NMI:Non-maskable_interrupts
    237.67 ±  6%    +123.8%     532.00 ± 32%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
      3607 ± 23%     -44.2%       2012 ± 33%  interrupts.CPU1.RES:Rescheduling_interrupts
      4360 ±  4%     +31.3%       5725 ±  5%  interrupts.CPU11.CAL:Function_call_interrupts
      4817 ±  9%     +24.2%       5984 ±  2%  interrupts.CPU12.CAL:Function_call_interrupts
    233.67 ± 36%     +68.6%     394.00 ± 41%  interrupts.CPU12.NMI:Non-maskable_interrupts
    233.67 ± 36%     +68.6%     394.00 ± 41%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      4950 ± 11%     +15.4%       5713 ±  2%  interrupts.CPU13.CAL:Function_call_interrupts
      1298 ± 75%     -74.9%     325.67 ± 22%  interrupts.CPU17.37:IR-PCI-MSI.2621443-edge.eth1-TxRx-2
      5720 ±  9%     -12.9%       4981 ±  4%  interrupts.CPU17.CAL:Function_call_interrupts
    264.67 ± 24%     +67.1%     442.33 ± 38%  interrupts.CPU19.NMI:Non-maskable_interrupts
    264.67 ± 24%     +67.1%     442.33 ± 38%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
      8948 ±  8%     +21.3%      10852 ±  8%  interrupts.CPU2.CAL:Function_call_interrupts
    268.67 ± 35%     +41.8%     381.00 ± 20%  interrupts.CPU2.NMI:Non-maskable_interrupts
    268.67 ± 35%     +41.8%     381.00 ± 20%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
    726.33 ± 14%     -45.3%     397.33 ±  8%  interrupts.CPU2.RES:Rescheduling_interrupts
    440.67 ± 29%     -35.9%     282.67 ± 24%  interrupts.CPU25.NMI:Non-maskable_interrupts
    440.67 ± 29%     -35.9%     282.67 ± 24%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      3885 ±  3%     +16.6%       4529 ±  9%  interrupts.CPU27.CAL:Function_call_interrupts
    236.00 ± 29%     +54.5%     364.67 ± 14%  interrupts.CPU28.NMI:Non-maskable_interrupts
    236.00 ± 29%     +54.5%     364.67 ± 14%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
    859.67 ± 21%     -37.5%     537.00 ± 10%  interrupts.CPU3.RES:Rescheduling_interrupts
      3875           +12.8%       4371 ±  5%  interrupts.CPU32.CAL:Function_call_interrupts
    230.00 ± 33%     +33.8%     307.67 ± 23%  interrupts.CPU32.NMI:Non-maskable_interrupts
    230.00 ± 33%     +33.8%     307.67 ± 23%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      4005 ±  2%     +12.5%       4506 ±  9%  interrupts.CPU33.CAL:Function_call_interrupts
    313.00 ± 23%     -30.9%     216.33 ± 26%  interrupts.CPU34.NMI:Non-maskable_interrupts
    313.00 ± 23%     -30.9%     216.33 ± 26%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
    184.67 ±  9%     -39.2%     112.33 ±  9%  interrupts.CPU36.NMI:Non-maskable_interrupts
    184.67 ±  9%     -39.2%     112.33 ±  9%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    239.67 ± 32%     -53.5%     111.33 ± 36%  interrupts.CPU37.NMI:Non-maskable_interrupts
    239.67 ± 32%     -53.5%     111.33 ± 36%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
    398.00 ± 14%     -60.0%     159.33 ± 50%  interrupts.CPU39.NMI:Non-maskable_interrupts
    398.00 ± 14%     -60.0%     159.33 ± 50%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      4435 ± 10%     +54.2%       6840 ± 20%  interrupts.CPU4.CAL:Function_call_interrupts
    262.00 ± 23%    +100.5%     525.33 ± 33%  interrupts.CPU4.NMI:Non-maskable_interrupts
    262.00 ± 23%    +100.5%     525.33 ± 33%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
     87.00 ± 14%     +78.2%     155.00 ± 24%  interrupts.CPU4.RES:Rescheduling_interrupts
    239.00 ± 33%     -53.1%     112.00 ± 34%  interrupts.CPU40.NMI:Non-maskable_interrupts
    239.00 ± 33%     -53.1%     112.00 ± 34%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      3861 ±  9%     +23.5%       4768 ± 12%  interrupts.CPU43.CAL:Function_call_interrupts
    180.33 ±  5%     -26.6%     132.33 ±  5%  interrupts.CPU43.NMI:Non-maskable_interrupts
    180.33 ±  5%     -26.6%     132.33 ±  5%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
    342.00 ± 21%     -53.2%     160.00 ± 19%  interrupts.CPU46.NMI:Non-maskable_interrupts
    342.00 ± 21%     -53.2%     160.00 ± 19%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    278.00 ± 35%     -54.3%     127.00 ± 37%  interrupts.CPU48.NMI:Non-maskable_interrupts
    278.00 ± 35%     -54.3%     127.00 ± 37%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
     91.33 ± 28%     +33.6%     122.00 ± 22%  interrupts.CPU5.RES:Rescheduling_interrupts
      5156 ±  7%     +25.9%       6491 ± 14%  interrupts.CPU6.CAL:Function_call_interrupts
    248.00 ± 28%     +31.3%     325.67 ± 22%  interrupts.CPU6.NMI:Non-maskable_interrupts
    248.00 ± 28%     +31.3%     325.67 ± 22%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      3327           +10.6%       3681 ±  4%  interrupts.CPU63.CAL:Function_call_interrupts
      4531 ±  8%     -18.0%       3716 ± 14%  interrupts.CPU68.CAL:Function_call_interrupts
      4740 ±  4%     +21.7%       5769 ± 12%  interrupts.CPU7.CAL:Function_call_interrupts
     11894 ±  7%     -18.3%       9712 ±  7%  interrupts.RES:Rescheduling_interrupts
      0.04 ±  6%     +25.0%       0.05 ± 17%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 44%     -85.3%       0.00 ±141%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.04 ±  4%     -20.8%       0.03 ±  9%  perf-sched.sch_delay.avg.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
      0.03 ±  3%     +13.0%       0.03        perf-sched.sch_delay.avg.ms.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range
      0.04           -16.0%       0.03 ±  7%  perf-sched.sch_delay.avg.ms.kthread_worker_fn.kthread.ret_from_fork
      0.11 ± 95%    +399.4%       0.57 ± 23%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
      0.00 ±141%    +258.3%       0.01 ± 40%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_read.walk_component.link_path_walk
      0.02 ±  7%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.__vma_adjust.__split_vma
      0.14 ± 46%     -56.2%       0.06 ± 41%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
      0.10 ± 98%    +248.0%       0.35 ± 19%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma
      0.20 ± 54%     +96.0%       0.39 ±  7%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
      0.03 ± 16%     -43.5%       0.02 ± 40%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      0.03           -77.1%       0.01 ± 12%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      0.01 ± 37%     -70.3%       0.00 ± 89%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
      0.04 ± 11%     -35.1%       0.02 ±  8%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
      0.02 ± 19%     +59.3%       0.03 ± 31%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.04 ± 26%     -36.3%       0.03 ±  4%  perf-sched.sch_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      0.03           +56.6%       0.04 ± 43%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.05           +16.0%       0.06 ±  6%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
      0.05 ± 28%     +96.0%       0.10 ± 33%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.07 ±  3%     +36.5%       0.10 ±  2%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.04 ±  4%     -36.4%       0.03 ± 15%  perf-sched.sch_delay.avg.ms.xlog_state_get_iclog_space.xlog_write.xlog_cil_push_work.process_one_work
      0.04 ±  4%     -94.3%       0.00 ± 20%  perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
      0.05 ± 78%    +176.6%       0.13 ± 33%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      3.07 ±130%     -95.9%       0.13 ± 74%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.43 ± 37%  +78350.5%     335.51 ±140%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.54 ± 97%    +448.3%       2.98 ± 24%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
      3.75 ± 71%     -61.0%       1.46 ±141%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_removename
      0.02 ±  5%     +80.4%       0.03 ± 41%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.01 ±141%    +305.3%       0.03 ± 34%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_read.walk_component.link_path_walk
      0.02 ± 23%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.__vma_adjust.__split_vma
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma
      0.28 ±108%    +256.7%       0.99 ± 79%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
      0.05 ± 31%     -51.3%       0.02 ± 16%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      0.08 ± 18%     -61.6%       0.03 ± 87%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.process_one_work.worker_thread.kthread
      0.03 ±  5%      +9.0%       0.03 ±  5%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion_io_timeout.submit_bio_wait.blkdev_issue_flush
      0.04 ± 21%     +31.5%       0.05 ± 19%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.write_cache_pages.iomap_writepages.xfs_vm_writepages
      0.04 ±  6%    +617.0%       0.32 ±111%  perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.10 ± 39%    +126.3%       0.22 ± 44%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
      0.04 ± 17%   +1158.8%       0.50 ±108%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
      0.08 ± 25%    +409.4%       0.40 ±107%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.04 ± 26%     -36.3%       0.03 ±  4%  perf-sched.sch_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      0.09 ± 12%  +13536.6%      11.68 ±123%  perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.04 ± 42%     -58.7%       0.02 ± 70%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.07 ± 34%     -49.3%       0.04 ± 41%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.xlog_cil_push_work.process_one_work
      0.19 ±  6%     +54.2%       0.29 ± 22%  perf-sched.sch_delay.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
     45.44 ± 18%     -54.4%      20.70 ± 52%  perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.14 ± 14%     -25.6%       0.11 ± 18%  perf-sched.sch_delay.max.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
      0.03 ±  2%     +20.2%       0.04 ± 11%  perf-sched.total_sch_delay.average.ms
    273.48           -32.5%     184.68 ± 12%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    795.10 ± 12%     -67.6%     257.37 ± 47%  perf-sched.wait_and_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      4.72 ±  2%     -46.0%       2.55 ±  3%  perf-sched.wait_and_delay.avg.ms.kthread_worker_fn.kthread.ret_from_fork
     33.82 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
    549.04 ± 38%     -52.6%     260.33 ± 49%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    446.94 ±  2%     -31.2%     307.28 ± 16%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     14.28 ±  2%     -34.4%       9.37 ± 18%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    129.29 ±  3%      -9.4%     117.19 ±  7%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
    135.30           -74.7%      34.23 ±  7%  perf-sched.wait_and_delay.avg.ms.xlog_cil_force_lsn.xfs_log_force.xfsaild_push.xfsaild
      6.69 ± 10%     -26.5%       4.91 ± 15%  perf-sched.wait_and_delay.avg.ms.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve
    140.19 ± 10%     -68.1%      44.66 ± 12%  perf-sched.wait_and_delay.avg.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
      9.67 ± 12%    +110.3%      20.33 ± 57%  perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      9.33 ± 13%    +117.9%      20.33 ± 57%  perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
    239.67           +43.3%     343.33 ± 14%  perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      7.67 ± 22%    +365.2%      35.67 ± 62%  perf-sched.wait_and_delay.count.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
    877.33 ±  3%     +40.5%       1232 ± 12%  perf-sched.wait_and_delay.count.kthread_worker_fn.kthread.ret_from_fork
     15.00 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
     28.33 ± 26%    +342.4%     125.33 ± 94%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     40.33 ±  2%     +66.1%      67.00 ± 24%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    662.67 ±  2%     +58.5%       1050 ± 22%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    252.00           +17.9%     297.00 ±  5%  perf-sched.wait_and_delay.count.schedule_timeout.xfsaild.kthread.ret_from_fork
      4591           +10.9%       5090 ±  6%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
     33.67 ±  6%     +54.5%      52.00 ± 19%  perf-sched.wait_and_delay.count.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
     63.43 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
    380.57 ± 15%     -33.8%     251.78 ±  9%  perf-sched.wait_and_delay.max.ms.schedule_timeout.__down.down.xfs_buf_lock
     52.05         +1141.3%     646.11 ±126%  perf-sched.wait_and_delay.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
    184.41 ±  7%     -64.9%      64.75 ± 10%  perf-sched.wait_and_delay.max.ms.xlog_cil_force_lsn.xfs_log_force.xfsaild_push.xfsaild
    382.80 ± 14%     -38.8%     234.21 ±  2%  perf-sched.wait_and_delay.max.ms.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve
    405.68 ±  4%     -65.7%     139.00 ±  7%  perf-sched.wait_and_delay.max.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
    273.46           -32.5%     184.66 ± 12%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      4.91           +63.1%       8.01 ±  7%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     53.43 ± 86%     -96.0%       2.16 ±128%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
    795.03 ± 12%     -67.6%     257.32 ± 47%  perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      9.00 ± 22%    +297.0%      35.75 ± 19%  perf-sched.wait_time.avg.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
      0.67 ±  3%    +125.1%       1.50 ± 24%  perf-sched.wait_time.avg.ms.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range
      4.69 ±  2%     -46.3%       2.52 ±  3%  perf-sched.wait_time.avg.ms.kthread_worker_fn.kthread.ret_from_fork
     16.50 ±109%     -82.0%       2.97 ± 18%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_inactive_ifree
      0.07 ±  6%    +135.9%       0.16 ± 52%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.terminate_walk.path_openat
     14.16 ±  5%     -51.0%       6.94 ± 67%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
      4.04 ± 46%     -94.6%       0.22 ±100%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      6.94 ±135%     -98.1%       0.13 ± 74%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.__fdget_pos.ksys_write
      0.18 ± 24%   +3086.3%       5.87 ±132%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.exit_mmap.mmput
     14.48           -71.2%       4.17        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      0.34 ± 20%   +2192.7%       7.75 ± 65%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion_io_timeout.submit_bio_wait.blkdev_issue_flush
     15.09 ±  8%     -60.5%       5.97 ± 18%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
     33.78 ±  5%     -65.8%      11.55 ± 49%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
    548.66 ± 38%     -52.6%     260.28 ± 49%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    446.92 ±  2%     -31.3%     307.25 ± 16%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      1.02 ± 38%    +389.0%       5.01 ±  3%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
     14.26 ±  2%     -34.6%       9.33 ± 18%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      7.28 ±  3%     +31.4%       9.57 ± 12%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
    129.22 ±  3%      -9.4%     117.09 ±  7%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    135.28           -74.7%      34.21 ±  7%  perf-sched.wait_time.avg.ms.xlog_cil_force_lsn.xfs_log_force.xfsaild_push.xfsaild
      6.66 ± 10%     -26.7%       4.88 ± 15%  perf-sched.wait_time.avg.ms.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve
     13.42 ±  2%     -37.5%       8.39 ± 13%  perf-sched.wait_time.avg.ms.xlog_state_get_iclog_space.xlog_write.xlog_cil_push_work.process_one_work
     13.36           +11.5%      14.89        perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
    140.17 ± 10%     -68.2%      44.64 ± 12%  perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
     13.54 ±  5%     -92.2%       1.05 ± 34%  perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
    209.61          +174.9%     576.20 ± 52%  perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    241.09 ± 71%     -99.6%       1.07 ± 68%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      1840 ± 64%     -96.6%      62.86 ±136%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.29 ± 26%   +1823.1%     563.33 ± 18%  perf-sched.wait_time.max.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
      5.45 ± 26%    +569.0%      36.44 ±112%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
      0.19 ± 44%     +89.9%       0.37 ± 20%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.terminate_walk.path_openat
    238.89 ±  6%     -64.5%      84.71 ± 90%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
      7.49 ± 32%     -89.1%       0.82 ±130%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
    199.71 ± 70%     -93.0%      13.95 ±138%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.perf_poll.do_sys_poll
    104.91 ±  8%    +534.3%     665.44 ±108%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.xlog_cil_push_work.process_one_work
      0.84 ± 80%   +4305.6%      36.80 ± 68%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion_io_timeout.submit_bio_wait.blkdev_issue_flush
    233.55 ±  6%     -63.8%      84.49 ± 30%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
      3.31 ± 71%    +859.1%      31.72 ±110%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
     63.38 ± 16%    +271.8%     235.66 ± 89%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
    379.32 ± 15%     -38.8%     232.15 ±  2%  perf-sched.wait_time.max.ms.schedule_timeout.__down.down.xfs_buf_lock
     29.29 ± 73%   +2771.8%     841.27 ± 81%  perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
    187.12 ± 11%     -51.6%      90.49 ±  4%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
     49.67           -45.8%      26.91 ± 53%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.xlog_cil_push_work.process_one_work
     52.01         +1142.2%     646.05 ±126%  perf-sched.wait_time.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
    184.39 ±  7%     -64.9%      64.73 ± 10%  perf-sched.wait_time.max.ms.xlog_cil_force_lsn.xfs_log_force.xfsaild_push.xfsaild
    380.22 ± 15%     -38.6%     233.31 ±  2%  perf-sched.wait_time.max.ms.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve
    405.66 ±  4%     -65.7%     138.98 ±  7%  perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
     39.60 ±  2%     -19.0       20.61 ± 38%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     40.73 ±  2%     -17.9       22.78 ± 36%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     40.70 ±  2%     -17.9       22.76 ± 36%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.97           -12.8       11.15 ± 39%  perf-profile.calltrace.cycles-pp.destroy_inode.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.72           -12.7       11.01 ± 39%  perf-profile.calltrace.cycles-pp.xfs_fs_destroy_inode.destroy_inode.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.65           -12.7       10.98 ± 39%  perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.do_unlinkat.do_syscall_64
     23.58           -12.6       10.95 ± 39%  perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.do_unlinkat
     16.15           -10.6        5.56 ± 41%  perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode
     14.15 ±  2%      -9.9        4.28 ± 43%  perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
     12.58 ±  4%      -9.2        3.34 ± 45%  perf-profile.calltrace.cycles-pp.xfs_difree_inobt.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive
     12.02 ±  4%      -9.0        3.06 ± 46%  perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree.xfs_ifree.xfs_inactive_ifree
     43.11 ±  2%      -7.3       35.78 ± 10%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     43.10 ±  2%      -7.3       35.78 ± 10%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     43.00 ±  2%      -7.3       35.69 ± 10%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     43.95 ±  2%      -7.2       36.76 ± 10%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      7.09 ±  4%      -5.5        1.56 ± 70%  perf-profile.calltrace.cycles-pp.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree.xfs_ifree
     14.21 ±  5%      -5.5        8.71 ± 38%  perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.93 ±  5%      -5.4        8.55 ± 38%  perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.88 ±  5%      -5.4        8.53 ± 38%  perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64
     29.28 ±  3%      -4.8       24.43 ± 11%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     28.30 ±  3%      -4.8       23.55 ± 11%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     22.56 ±  4%      -4.0       18.61 ± 12%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     20.51 ±  4%      -3.5       17.01 ± 10%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      4.13 ±  8%      -3.3        0.83 ± 71%  perf-profile.calltrace.cycles-pp.xfs_btree_increment.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree.xfs_ifree
      3.92 ±  4%      -3.1        0.85 ± 70%  perf-profile.calltrace.cycles-pp.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree
     13.08 ±  4%      -2.4       10.65 ± 11%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     12.41 ±  5%      -2.3       10.12 ± 10%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      2.57 ±  8%      -2.0        0.52 ± 71%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree
      2.42 ±  8%      -1.9        0.52 ± 70%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_difree_inobt
      3.16 ±  4%      -1.8        1.32 ± 43%  perf-profile.calltrace.cycles-pp.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      7.96 ±  5%      -1.6        6.38 ± 10%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      5.02 ±  9%      -1.4        3.57 ± 33%  perf-profile.calltrace.cycles-pp.xfs_trans_alloc.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      1.79 ±  9%      -1.4        0.36 ± 70%  perf-profile.calltrace.cycles-pp.__xfs_btree_check_sblock.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_difree_inobt
      4.94 ±  9%      -1.4        3.51 ± 33%  perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_remove.xfs_vn_unlink.vfs_unlink
      6.77 ±  6%      -1.3        5.46 ± 11%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.72 ±  7%      -1.3        2.43 ± 40%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      3.37 ±  4%      -1.2        2.17 ± 35%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode
      3.49 ±  7%      -1.2        2.31 ± 41%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink
      1.60 ± 15%      -1.2        0.43 ± 72%  perf-profile.calltrace.cycles-pp.__xfs_btree_check_sblock.xfs_btree_check_sblock.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount
      4.83 ±  4%      -1.1        3.70 ± 31%  perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow
      3.12 ±  4%      -1.1        2.07 ± 36%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      4.92 ±  5%      -1.0        3.91 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      4.71 ±  4%      -1.0        3.74 ±  6%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      4.39 ±  6%      -0.9        3.51 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      4.15 ±  6%      -0.8        3.34 ± 11%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.50 ± 11%      -0.8        0.73 ± 70%  perf-profile.calltrace.cycles-pp.xfs_difree_finobt.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive
      3.80 ±  3%      -0.8        3.03 ±  6%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.21 ± 10%      -0.6        0.56 ± 71%  perf-profile.calltrace.cycles-pp.xfs_mod_fdblocks.xfs_trans_reserve.xfs_trans_alloc.xfs_remove.xfs_vn_unlink
      1.19 ± 11%      -0.6        0.54 ± 71%  perf-profile.calltrace.cycles-pp.__percpu_counter_compare.xfs_mod_fdblocks.xfs_trans_reserve.xfs_trans_alloc.xfs_remove
      4.68 ±  2%      -0.6        4.04 ± 17%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      2.93 ±  4%      -0.6        2.31 ± 23%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      1.71 ±  3%      -0.5        1.17 ± 36%  perf-profile.calltrace.cycles-pp.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      0.90 ±  9%      -0.5        0.40 ± 71%  perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove.xfs_vn_unlink
      1.37 ±  6%      -0.5        0.87 ± 13%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.15 ±  4%      -0.5        0.66 ± 72%  perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive
      1.29 ±  5%      -0.5        0.81 ± 11%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt
      0.96 ± 13%      -0.5        0.49 ± 70%  perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_difree_finobt.xfs_difree.xfs_ifree.xfs_inactive_ifree
      1.09 ±  6%      -0.5        0.63 ± 71%  perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree
      1.94 ±  5%      -0.4        1.54 ± 24%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.98 ±  8%      -0.4        0.58 ± 71%  perf-profile.calltrace.cycles-pp.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove.xfs_ifree
      0.95 ± 10%      -0.4        0.57 ± 71%  perf-profile.calltrace.cycles-pp.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove
      1.44 ±  6%      -0.4        1.08 ±  8%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_from_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.89 ±  5%      -0.4        0.53 ± 71%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      1.41 ±  5%      -0.3        1.06 ± 34%  perf-profile.calltrace.cycles-pp.xfs_log_ticket_ungrant.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive
      1.36 ±  5%      -0.3        1.03 ± 33%  perf-profile.calltrace.cycles-pp.xfs_log_space_wake.xfs_log_ticket_ungrant.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
      0.67 ±  9%      -0.3        0.38 ± 70%  perf-profile.calltrace.cycles-pp.hrtimer_next_event_without.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      1.16 ± 10%      -0.3        0.87 ± 11%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__percpu_counter_compare.xfs_mod_fdblocks.xfs_trans_reserve.xfs_trans_alloc
      1.54 ±  6%      -0.3        1.27 ± 11%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.62 ±  6%      -0.3        1.36 ± 12%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.74 ±  3%      -0.3        1.48 ±  8%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.99 ±  6%      -0.2        0.74 ± 13%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_from_idle.do_idle.cpu_startup_entry.start_secondary
      0.60 ±  5%      -0.2        0.36 ± 70%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.10 ± 11%      -0.2        0.89 ±  8%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.29 ±  4%      -0.2        1.10 ± 13%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt
      0.69 ± 16%      -0.1        0.55        perf-profile.calltrace.cycles-pp.hrtimer_update_next_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.88            -0.1        0.76 ± 13%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu
      0.82 ±  3%      -0.1        0.70 ± 12%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start
      0.76            -0.1        0.65 ±  6%  perf-profile.calltrace.cycles-pp.update_rq_clock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.75 ±  3%      +0.2        0.93 ± 20%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      0.58 ±  4%      +0.2        0.80 ±  9%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.58 ±  4%      +0.2        0.80 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      0.58 ±  4%      +0.2        0.80 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.58 ±  4%      +0.2        0.80 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.58 ±  4%      +0.2        0.80 ±  9%  perf-profile.calltrace.cycles-pp.execve
      0.80 ± 17%      +0.3        1.09 ± 20%  perf-profile.calltrace.cycles-pp.loop_queue_work.kthread_worker_fn.kthread.ret_from_fork
      0.54            +0.4        0.90 ± 14%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      0.52 ±  3%      +0.4        0.87 ± 13%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      0.87 ± 21%      +0.4        1.25 ± 16%  perf-profile.calltrace.cycles-pp.kthread_worker_fn.kthread.ret_from_fork
      0.19 ±141%      +0.5        0.66 ±  5%  perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.35 ± 70%      +0.5        0.85 ± 12%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +0.6        0.62 ±  4%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.67 ±  9%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +0.8        0.76 ± 14%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +1.1        1.07 ± 13%  perf-profile.calltrace.cycles-pp.__invalidate_mapping_pages.drop_pagecache_sb.iterate_supers.drop_caches_sysctl_handler.proc_sys_call_handler
      0.00            +1.1        1.08 ± 34%  perf-profile.calltrace.cycles-pp.mnt_get_count.mntput_no_expire.simple_recursive_removal.debugfs_remove.update_sched_domain_debugfs
      0.00            +1.1        1.10 ± 34%  perf-profile.calltrace.cycles-pp.mntput_no_expire.simple_recursive_removal.debugfs_remove.update_sched_domain_debugfs.rebuild_sched_domains_locked
      0.00            +1.3        1.31 ± 33%  perf-profile.calltrace.cycles-pp.simple_recursive_removal.debugfs_remove.update_sched_domain_debugfs.rebuild_sched_domains_locked.rebuild_sched_domains
      0.00            +1.3        1.32 ± 33%  perf-profile.calltrace.cycles-pp.debugfs_remove.update_sched_domain_debugfs.rebuild_sched_domains_locked.rebuild_sched_domains.cpuset_hotplug_workfn
      0.61 ± 15%      +1.3        1.95 ±  9%  perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.4        1.41 ± 13%  perf-profile.calltrace.cycles-pp.drop_pagecache_sb.iterate_supers.drop_caches_sysctl_handler.proc_sys_call_handler.new_sync_write
      0.00            +1.4        1.41 ± 13%  perf-profile.calltrace.cycles-pp.iterate_supers.drop_caches_sysctl_handler.proc_sys_call_handler.new_sync_write.vfs_write
      0.43 ± 72%      +1.4        1.87 ±  9%  perf-profile.calltrace.cycles-pp.drm_fb_memcpy_dstclip.mgag200_simple_display_pipe_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail.commit_tail
      0.43 ± 72%      +1.4        1.88 ±  9%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_helper_dirtyfb
      0.43 ± 72%      +1.4        1.88 ±  9%  perf-profile.calltrace.cycles-pp.mgag200_simple_display_pipe_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail.commit_tail.drm_atomic_helper_commit
      0.43 ± 72%      +1.4        1.88 ±  9%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_helper_dirtyfb.drm_fb_helper_damage_work
      0.43 ± 72%      +1.5        1.89 ±  9%  perf-profile.calltrace.cycles-pp.commit_tail.drm_atomic_helper_commit.drm_atomic_helper_dirtyfb.drm_fb_helper_damage_work.process_one_work
      0.43 ± 71%      +1.5        1.89 ±  9%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit.drm_atomic_helper_dirtyfb.drm_fb_helper_damage_work.process_one_work.worker_thread
      0.43 ± 71%      +1.5        1.90 ±  9%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_dirtyfb.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread
      0.00            +1.5        1.54 ± 33%  perf-profile.calltrace.cycles-pp.update_sched_domain_debugfs.rebuild_sched_domains_locked.rebuild_sched_domains.cpuset_hotplug_workfn.process_one_work
      0.00            +1.8        1.76 ± 14%  perf-profile.calltrace.cycles-pp.drop_caches_sysctl_handler.proc_sys_call_handler.new_sync_write.vfs_write.ksys_write
      0.00            +1.8        1.78 ± 13%  perf-profile.calltrace.cycles-pp.proc_sys_call_handler.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.00            +2.0        1.96 ± 31%  perf-profile.calltrace.cycles-pp.cpuset_hotplug_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +2.0        1.96 ± 31%  perf-profile.calltrace.cycles-pp.rebuild_sched_domains.cpuset_hotplug_workfn.process_one_work.worker_thread.kthread
      0.00            +2.0        1.96 ± 31%  perf-profile.calltrace.cycles-pp.rebuild_sched_domains_locked.rebuild_sched_domains.cpuset_hotplug_workfn.process_one_work.worker_thread
      0.39 ±141%      +2.3        2.73 ± 23%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.memcpy_toio.drm_fb_memcpy_dstclip.mgag200_simple_display_pipe_update.drm_atomic_helper_commit_planes
      0.00            +2.5        2.48 ±  5%  perf-profile.calltrace.cycles-pp.rcu_momentary_dyntick_idle.multi_cpu_stop.cpu_stopper_thread.smpboot_thread_fn.kthread
      0.00            +2.5        2.52 ±  9%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +2.5        2.52 ±  9%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +2.5        2.52 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      0.00            +2.5        2.52 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +2.5        2.53 ±  9%  perf-profile.calltrace.cycles-pp.write
      0.73 ±  3%      +2.6        3.36 ± 11%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.7        2.73 ±  7%  perf-profile.calltrace.cycles-pp.multi_cpu_stop.cpu_stopper_thread.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +2.8        2.79 ±  8%  perf-profile.calltrace.cycles-pp.cpu_stopper_thread.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +3.3        3.28 ±  9%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
      2.33 ±  5%      +3.5        5.81 ± 13%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      1.99 ±  4%      +3.5        5.47 ± 13%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      4.11 ±  8%      +6.8       10.89 ±  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      4.11 ±  8%      +6.8       10.90 ±  5%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +9.8        9.79 ±113%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2
      0.00           +10.0        9.96 ±112%  perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.00           +11.8       11.83 ±108%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      0.00           +11.8       11.85 ±108%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +12.1       12.08 ±107%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      0.00           +12.1       12.09 ±107%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      0.00           +12.1       12.14 ±107%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      0.00           +12.1       12.15 ±107%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64
      0.00           +12.3       12.29 ±107%  perf-profile.calltrace.cycles-pp.open64
     39.60 ±  2%     -19.0       20.63 ± 38%  perf-profile.children.cycles-pp.do_unlinkat
     23.98           -12.8       11.18 ± 39%  perf-profile.children.cycles-pp.destroy_inode
     23.72           -12.7       11.01 ± 39%  perf-profile.children.cycles-pp.xfs_fs_destroy_inode
     23.66           -12.7       10.98 ± 39%  perf-profile.children.cycles-pp.xfs_inactive
     23.58           -12.6       10.95 ± 39%  perf-profile.children.cycles-pp.xfs_inactive_ifree
     16.15           -10.6        5.57 ± 41%  perf-profile.children.cycles-pp.xfs_ifree
     14.15 ±  2%      -9.9        4.28 ± 43%  perf-profile.children.cycles-pp.xfs_difree
     12.58 ±  4%      -9.2        3.34 ± 45%  perf-profile.children.cycles-pp.xfs_difree_inobt
     43.11 ±  2%      -7.3       35.78 ± 10%  perf-profile.children.cycles-pp.start_secondary
     43.95 ±  2%      -7.2       36.75 ± 10%  perf-profile.children.cycles-pp.do_idle
     43.95 ±  2%      -7.2       36.76 ± 10%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     43.95 ±  2%      -7.2       36.76 ± 10%  perf-profile.children.cycles-pp.cpu_startup_entry
     13.03 ±  3%      -6.6        6.47 ± 30%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
     14.21 ±  5%      -5.5        8.72 ± 38%  perf-profile.children.cycles-pp.vfs_unlink
     13.93 ±  5%      -5.4        8.55 ± 38%  perf-profile.children.cycles-pp.xfs_vn_unlink
     13.89 ±  5%      -5.4        8.54 ± 38%  perf-profile.children.cycles-pp.xfs_remove
     29.75 ±  3%      -4.7       25.03 ± 11%  perf-profile.children.cycles-pp.cpuidle_enter_state
     29.87 ±  3%      -4.7       25.15 ± 11%  perf-profile.children.cycles-pp.cpuidle_enter
      7.80 ±  2%      -4.2        3.62 ± 25%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
     22.93 ±  4%      -3.2       19.71 ±  7%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      5.43 ±  3%      -3.0        2.43 ± 26%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      4.45 ±  7%      -2.4        2.04 ± 36%  perf-profile.children.cycles-pp.xfs_btree_increment
     24.74 ±  4%      -2.4       22.35 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      4.30 ±  3%      -2.4        1.95 ± 23%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      2.31 ± 12%      -2.2        0.14 ± 85%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      3.68 ±  7%      -2.1        1.56 ± 22%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
     15.19 ±  4%      -2.1       13.13 ±  9%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
     14.47 ±  4%      -1.9       12.57 ±  8%  perf-profile.children.cycles-pp.hrtimer_interrupt
      3.17 ±  4%      -1.8        1.32 ± 43%  perf-profile.children.cycles-pp.xfs_dir_removename
      6.95 ±  6%      -1.4        5.59 ± 12%  perf-profile.children.cycles-pp.menu_select
      9.68 ±  4%      -1.2        8.52 ±  7%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.39 ± 16%      -1.0        0.41 ± 98%  perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
      5.28 ±  4%      -0.9        4.34 ±  4%  perf-profile.children.cycles-pp.irq_exit_rcu
      4.79 ±  4%      -0.8        4.00        perf-profile.children.cycles-pp.__softirqentry_text_start
      1.34 ± 12%      -0.8        0.56 ± 21%  perf-profile.children.cycles-pp.xfs_inobt_get_maxrecs
      0.81 ± 15%      -0.7        0.08 ± 16%  perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      1.34 ±  5%      -0.7        0.66 ± 19%  perf-profile.children.cycles-pp.xfs_inobt_irec_to_allocmask
      3.03 ±  5%      -0.7        2.37 ± 23%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.50 ± 11%      -0.6        0.88 ± 34%  perf-profile.children.cycles-pp.xfs_difree_finobt
      1.71 ±  3%      -0.5        1.17 ± 36%  perf-profile.children.cycles-pp.xfs_iunlink_remove
      1.12 ± 12%      -0.5        0.63 ± 30%  perf-profile.children.cycles-pp.xfs_errortag_test
      1.42 ±  6%      -0.5        0.95 ± 11%  perf-profile.children.cycles-pp.run_rebalance_domains
      1.44 ±  7%      -0.5        0.97 ±  9%  perf-profile.children.cycles-pp.update_blocked_averages
      0.93 ±  9%      -0.4        0.49 ± 20%  perf-profile.children.cycles-pp.xfs_next_bit
      1.46 ±  6%      -0.4        1.11 ±  7%  perf-profile.children.cycles-pp.flush_smp_call_function_from_idle
      0.73 ± 14%      -0.3        0.39 ± 49%  perf-profile.children.cycles-pp.xfs_droplink
      0.67 ±  6%      -0.3        0.35 ± 24%  perf-profile.children.cycles-pp.filename_parentat
      1.76 ±  6%      -0.3        1.46 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      0.66 ±  9%      -0.3        0.36 ± 25%  perf-profile.children.cycles-pp.xfs_verify_agino
      0.63 ±  9%      -0.3        0.34 ± 26%  perf-profile.children.cycles-pp.path_parentat
      1.63 ±  7%      -0.3        1.34 ± 11%  perf-profile.children.cycles-pp.rebalance_domains
      0.46 ± 14%      -0.3        0.20 ± 34%  perf-profile.children.cycles-pp.xfs_btree_get_block
      0.73 ±  9%      -0.2        0.48 ± 32%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      1.84 ±  3%      -0.2        1.60 ± 12%  perf-profile.children.cycles-pp.ktime_get
      1.07 ±  7%      -0.2        0.84 ± 13%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.51 ± 11%      -0.2        0.28 ± 41%  perf-profile.children.cycles-pp.xfs_btree_check_block
      1.08 ± 13%      -0.2        0.85 ±  8%  perf-profile.children.cycles-pp.update_load_avg
      1.41 ±  6%      -0.2        1.20 ± 12%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.90 ±  7%      -0.2        0.71 ±  5%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.24 ±  6%      -0.2        1.05 ±  8%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.75 ±  7%      -0.2        0.57 ±  7%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.36 ± 20%      -0.2        0.19 ± 24%  perf-profile.children.cycles-pp.xfs_inobt_btrec_to_irec
      0.76 ±  5%      -0.2        0.60 ±  4%  perf-profile.children.cycles-pp.enqueue_entity
      0.69 ±  8%      -0.2        0.53 ± 16%  perf-profile.children.cycles-pp.xfs_imap_to_bp
      1.13 ±  7%      -0.2        0.97 ± 10%  perf-profile.children.cycles-pp.native_sched_clock
      1.12            -0.2        0.97 ±  5%  perf-profile.children.cycles-pp.update_rq_clock
      0.91 ±  4%      -0.2        0.76 ±  7%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.86 ±  5%      -0.1        0.71 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.44 ±  7%      -0.1        0.30 ± 33%  perf-profile.children.cycles-pp.xfs_iunlink_update_inode
      0.63 ±  7%      -0.1        0.48 ±  4%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.36 ±  8%      -0.1        0.23 ± 29%  perf-profile.children.cycles-pp.xfs_btree_rec_offset
      0.43 ±  7%      -0.1        0.31 ± 14%  perf-profile.children.cycles-pp.blk_complete_reqs
      1.11 ±  8%      -0.1        0.99 ±  5%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.46 ±  9%      -0.1        0.34 ±  2%  perf-profile.children.cycles-pp.timerqueue_del
      0.17 ± 22%      -0.1        0.06 ± 73%  perf-profile.children.cycles-pp.xfs_inode_mark_reclaimable
      0.30 ± 15%      -0.1        0.19 ± 19%  perf-profile.children.cycles-pp.do_softirq
      0.22 ±  5%      -0.1        0.11 ± 18%  perf-profile.children.cycles-pp.update_dl_rq_load_avg
      0.29 ± 10%      -0.1        0.18 ±  7%  perf-profile.children.cycles-pp.scsi_io_completion
      0.41 ± 23%      -0.1        0.31 ±  5%  perf-profile.children.cycles-pp.xfs_file_fsync
      0.28 ± 11%      -0.1        0.18 ±  7%  perf-profile.children.cycles-pp.scsi_end_request
      0.44 ±  9%      -0.1        0.34 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.20 ± 20%      -0.1        0.10 ± 40%  perf-profile.children.cycles-pp.xfs_btree_readahead
      0.32 ± 10%      -0.1        0.22 ± 11%  perf-profile.children.cycles-pp.xfs_buf_item_init
      0.68 ±  3%      -0.1        0.58 ±  4%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.21 ± 12%      -0.1        0.11 ± 29%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.22 ± 16%      -0.1        0.13 ± 30%  perf-profile.children.cycles-pp.__x64_sys_unlink
      0.16 ±  6%      -0.1        0.07 ± 25%  perf-profile.children.cycles-pp.flush_end_io
      0.23 ±  4%      -0.1        0.15 ± 17%  perf-profile.children.cycles-pp.xfs_perag_get
      0.14 ± 16%      -0.1        0.06 ± 70%  perf-profile.children.cycles-pp.xfs_lock_two_inodes
      0.16 ± 20%      -0.1        0.08 ± 71%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.40            -0.1        0.32 ± 19%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.12 ± 10%      -0.1        0.04 ± 71%  perf-profile.children.cycles-pp.blk_mq_requeue_work
      0.63 ±  5%      -0.1        0.55 ±  2%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.57 ±  7%      -0.1        0.50 ±  6%  perf-profile.children.cycles-pp.perf_output_copy
      0.16 ±  5%      -0.1        0.08 ± 20%  perf-profile.children.cycles-pp.blk_flush_complete_seq
      0.17 ± 17%      -0.1        0.10 ± 31%  perf-profile.children.cycles-pp.string
      0.13 ±  3%      -0.1        0.05 ± 84%  perf-profile.children.cycles-pp.xfs_dir2_data_make_free
      0.14 ± 17%      -0.1        0.07 ± 78%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.45 ±  8%      -0.1        0.38 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.21 ±  9%      -0.1        0.15 ± 12%  perf-profile.children.cycles-pp.xfs_buf_item_put
      0.13 ±  3%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.__blk_mq_delay_run_hw_queue
      0.10 ± 21%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.xfs_perag_set_inode_tag
      0.23 ± 16%      -0.1        0.16 ± 12%  perf-profile.children.cycles-pp.get_cpu_device
      0.16 ± 16%      -0.1        0.10 ± 21%  perf-profile.children.cycles-pp.perf_trace_buf_alloc
      0.36 ±  8%      -0.1        0.30 ±  4%  perf-profile.children.cycles-pp.__wake_up_common
      0.10 ±  9%      -0.1        0.04 ± 76%  perf-profile.children.cycles-pp.__lookup_hash
      0.13 ± 37%      -0.1        0.08 ± 30%  perf-profile.children.cycles-pp.asm_common_interrupt
      0.13 ± 37%      -0.1        0.08 ± 30%  perf-profile.children.cycles-pp.common_interrupt
      0.15 ±  8%      -0.1        0.09 ± 15%  perf-profile.children.cycles-pp.__switch_to_asm
      0.31 ±  3%      -0.1        0.26 ±  6%  perf-profile.children.cycles-pp.rb_erase
      0.31 ±  8%      -0.1        0.26 ±  5%  perf-profile.children.cycles-pp.pipe_read
      0.25 ±  6%      -0.1        0.20 ± 16%  perf-profile.children.cycles-pp.xfs_trans_brelse
      0.09 ± 18%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.scsi_queue_rq
      0.14 ±  8%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.send_call_function_single_ipi
      0.09 ±  5%      -0.1        0.04 ± 71%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.17 ± 22%      -0.0        0.12 ± 10%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.27 ±  9%      -0.0        0.22 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.22 ±  7%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.rb_next
      0.18 ±  6%      -0.0        0.14 ± 20%  perf-profile.children.cycles-pp.rcu_nmi_exit
      0.16 ± 10%      -0.0        0.11 ± 12%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.13 ± 25%      -0.0        0.09 ± 10%  perf-profile.children.cycles-pp.wait_for_completion
      0.13 ± 25%      -0.0        0.09 ± 10%  perf-profile.children.cycles-pp.xfs_buf_inode_iodone
      0.14 ± 10%      -0.0        0.10 ± 24%  perf-profile.children.cycles-pp.xfs_trans_buf_item_match
      0.13 ±  9%      -0.0        0.09 ± 21%  perf-profile.children.cycles-pp.is_bpf_text_address
      0.08 ± 14%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp.xfs_dir2_data_entry_tag_p
      0.11 ± 12%      -0.0        0.07 ± 20%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.19 ±  9%      -0.0        0.15 ±  8%  perf-profile.children.cycles-pp.xfs_lookup_get_search_key
      0.07 ± 17%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp.check_tsc_unstable
      0.07 ± 11%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp.devkmsg_read
      0.10 ± 21%      -0.0        0.07 ± 25%  perf-profile.children.cycles-pp.xfs_log_force_lsn
      0.09 ±  5%      -0.0        0.06 ± 19%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.05 ±  8%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.up_write
      0.06 ± 29%      +0.0        0.09 ± 10%  perf-profile.children.cycles-pp.__note_gp_changes
      0.06 ±  7%      +0.0        0.09 ± 28%  perf-profile.children.cycles-pp.current_time
      0.06 ± 13%      +0.0        0.09 ± 18%  perf-profile.children.cycles-pp.__fxstat64
      0.07 ± 23%      +0.0        0.10 ± 19%  perf-profile.children.cycles-pp.perf_tp_event_match
      0.06 ± 14%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.elf_map
      0.07 ± 25%      +0.0        0.10        perf-profile.children.cycles-pp.__slab_free
      0.07 ± 11%      +0.0        0.11 ± 14%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.03 ± 70%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.clear_page_erms
      0.05 ±  8%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.__do_sys_wait4
      0.05 ±  8%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.kernel_wait4
      0.06 ± 13%      +0.0        0.11 ± 23%  perf-profile.children.cycles-pp.map_id_range_down
      0.02 ±141%      +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.perf_output_put_handle
      0.03 ± 70%      +0.0        0.08 ± 11%  perf-profile.children.cycles-pp.page_remove_rmap
      0.03 ± 70%      +0.0        0.08 ± 20%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.04 ± 71%      +0.1        0.09 ± 21%  perf-profile.children.cycles-pp.__mem_cgroup_charge
      0.06 ± 14%      +0.1        0.11 ± 29%  perf-profile.children.cycles-pp.balance_fair
      0.02 ±141%      +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.free_unref_page_list
      0.19 ±  5%      +0.1        0.24 ± 15%  perf-profile.children.cycles-pp.xfs_trans_committed_bulk
      0.02 ±141%      +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.create_elf_tables
      0.14 ±  6%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.__might_sleep
      0.14 ± 29%      +0.1        0.20 ± 17%  perf-profile.children.cycles-pp.__open64_nocancel
      0.09 ± 10%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.07 ± 11%      +0.1        0.13 ± 26%  perf-profile.children.cycles-pp.mem_cgroup_charge
      0.08 ± 40%      +0.1        0.13 ± 21%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.08 ± 40%      +0.1        0.13 ± 15%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.04 ± 71%      +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.do_wait
      0.03 ± 70%      +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.prep_new_page
      0.08 ± 31%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.alloc_inode
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.__kmalloc_node
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.copy_pte_range
      0.09 ±  5%      +0.1        0.16 ± 25%  perf-profile.children.cycles-pp.__perf_sw_event
      0.04 ± 71%      +0.1        0.11 ± 19%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.32 ±  6%      +0.1        0.39 ± 11%  perf-profile.children.cycles-pp.__module_text_address
      0.37 ±  3%      +0.1        0.44 ±  6%  perf-profile.children.cycles-pp.__libc_fork
      0.28 ± 16%      +0.1        0.34 ± 15%  perf-profile.children.cycles-pp.step_into
      0.02 ±141%      +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.unlink_file_vma
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.do_mkdirat
      0.07 ± 17%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.__close_nocancel
      0.05 ± 71%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.blk_mq_sched_insert_request
      0.00            +0.1        0.07 ± 30%  perf-profile.children.cycles-pp.unlock_page
      0.02 ±141%      +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.proc_pid_make_inode
      0.07 ± 30%      +0.1        0.14 ± 23%  perf-profile.children.cycles-pp.xfs_trans_ijoin
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.mkdir
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.do_faccessat
      0.02 ±141%      +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.call_rcu
      0.30 ±  8%      +0.1        0.37 ±  5%  perf-profile.children.cycles-pp.submit_bio_noacct
      0.30 ±  8%      +0.1        0.37 ±  6%  perf-profile.children.cycles-pp.submit_bio
      0.00            +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.___d_drop
      0.00            +0.1        0.08 ± 40%  perf-profile.children.cycles-pp.epoll_wait
      0.09 ± 18%      +0.1        0.17 ± 20%  perf-profile.children.cycles-pp.perf_event_mmap
      0.02 ±141%      +0.1        0.09 ± 44%  perf-profile.children.cycles-pp.obj_cgroup_charge_pages
      0.23 ± 11%      +0.1        0.31 ± 13%  perf-profile.children.cycles-pp.__cond_resched
      0.10 ± 25%      +0.1        0.18 ± 24%  perf-profile.children.cycles-pp.free_pgtables
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.__d_drop
      0.00            +0.1        0.08 ± 44%  perf-profile.children.cycles-pp.lru_add_drain
      0.02 ±141%      +0.1        0.10 ± 24%  perf-profile.children.cycles-pp.submit_bio_checks
      0.02 ±141%      +0.1        0.10 ± 31%  perf-profile.children.cycles-pp.finish_fault
      0.02 ±141%      +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.__do_fault
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.update_tasks_root_domain
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.copy_page_range
      0.00            +0.1        0.08 ± 31%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      0.10 ± 21%      +0.1        0.18 ± 18%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.29 ± 14%      +0.1        0.37 ± 12%  perf-profile.children.cycles-pp.getname_flags
      0.17 ± 12%      +0.1        0.26 ± 20%  perf-profile.children.cycles-pp.evlist__parse_sample
      0.00            +0.1        0.09 ± 38%  perf-profile.children.cycles-pp.dentry_lru_isolate
      0.00            +0.1        0.09 ±  5%  perf-profile.children.cycles-pp.try_charge
      0.02 ±141%      +0.1        0.10 ± 25%  perf-profile.children.cycles-pp.__vma_adjust
      0.00            +0.1        0.09 ± 47%  perf-profile.children.cycles-pp.inode_init_always
      0.00            +0.1        0.09 ± 33%  perf-profile.children.cycles-pp._PyEval_EvalFrameDefault
      0.00            +0.1        0.09 ± 33%  perf-profile.children.cycles-pp.lookup_one_len
      0.00            +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.page_mapping
      0.00            +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.openat64
      0.14 ± 28%      +0.1        0.24 ± 15%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.new_inode
      0.18 ± 19%      +0.1        0.28 ± 17%  perf-profile.children.cycles-pp.do_anonymous_page
      0.42 ± 11%      +0.1        0.52 ±  5%  perf-profile.children.cycles-pp.__module_address
      0.10 ± 12%      +0.1        0.20 ± 37%  perf-profile.children.cycles-pp.next_uptodate_page
      0.00            +0.1        0.10 ± 16%  perf-profile.children.cycles-pp.vm_area_dup
      0.16 ± 11%      +0.1        0.26 ± 19%  perf-profile.children.cycles-pp.zap_pte_range
      0.02 ±141%      +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.mprotect_fixup
      0.20 ±  4%      +0.1        0.31 ± 28%  perf-profile.children.cycles-pp.iomap_write_actor
      0.13 ± 37%      +0.1        0.24 ± 15%  perf-profile.children.cycles-pp.__do_munmap
      0.00            +0.1        0.11 ± 37%  perf-profile.children.cycles-pp.__debugfs_create_file
      0.04 ± 76%      +0.1        0.15 ± 39%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.03 ±141%      +0.1        0.14 ± 51%  perf-profile.children.cycles-pp.xfs_inode_free_callback
      0.00            +0.1        0.11 ± 41%  perf-profile.children.cycles-pp.__list_lru_walk_one
      0.00            +0.1        0.11 ± 41%  perf-profile.children.cycles-pp.list_lru_walk_one
      0.00            +0.1        0.11 ± 32%  perf-profile.children.cycles-pp.start_creating
      0.02 ±141%      +0.1        0.13 ± 65%  perf-profile.children.cycles-pp.__x64_sys_close
      0.20 ± 16%      +0.1        0.31 ± 27%  perf-profile.children.cycles-pp.delay_tsc
      0.00            +0.1        0.11 ± 23%  perf-profile.children.cycles-pp.xas_load
      0.18 ± 10%      +0.1        0.30 ± 13%  perf-profile.children.cycles-pp.begin_new_exec
      0.10 ± 25%      +0.1        0.22 ±  5%  perf-profile.children.cycles-pp.wp_page_copy
      0.00            +0.1        0.11 ± 33%  perf-profile.children.cycles-pp.invalidate_inode_page
      0.02 ±141%      +0.1        0.13 ± 18%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      0.02 ±141%      +0.1        0.13 ± 18%  perf-profile.children.cycles-pp.do_mprotect_pkey
      0.18 ± 13%      +0.1        0.30 ± 16%  perf-profile.children.cycles-pp.unmap_page_range
      0.18 ± 31%      +0.1        0.30 ± 11%  perf-profile.children.cycles-pp.cpumask_next
      0.17 ± 11%      +0.1        0.28 ± 13%  perf-profile.children.cycles-pp.setlocale
      0.04 ± 71%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.pagecache_get_page
      0.11 ±  4%      +0.1        0.23 ±  3%  perf-profile.children.cycles-pp.dup_mmap
      0.12 ± 11%      +0.1        0.24 ± 10%  perf-profile.children.cycles-pp.dup_mm
      0.28 ± 18%      +0.1        0.41 ± 16%  perf-profile.children.cycles-pp.___might_sleep
      0.20 ± 10%      +0.1        0.33 ± 16%  perf-profile.children.cycles-pp.unmap_vmas
      0.52 ± 10%      +0.1        0.66 ±  9%  perf-profile.children.cycles-pp.__xstat64
      0.00            +0.1        0.13 ± 33%  perf-profile.children.cycles-pp.xfs_file_fallocate
      0.00            +0.1        0.13 ± 33%  perf-profile.children.cycles-pp.xfs_free_file_space
      0.05 ± 81%      +0.1        0.18 ± 31%  perf-profile.children.cycles-pp.d_alloc
      0.20 ± 20%      +0.1        0.34 ±  9%  perf-profile.children.cycles-pp.xlog_state_release_iclog
      0.10 ±  9%      +0.1        0.24 ±  3%  perf-profile.children.cycles-pp.path_init
      0.50 ±  9%      +0.1        0.65 ±  4%  perf-profile.children.cycles-pp.vfs_statx
      0.02 ±141%      +0.1        0.17 ± 22%  perf-profile.children.cycles-pp.__split_vma
      0.48 ± 15%      +0.2        0.63 ± 10%  perf-profile.children.cycles-pp.io_serial_in
      0.05 ± 84%      +0.2        0.20 ± 59%  perf-profile.children.cycles-pp.do_dentry_open
      0.20 ± 16%      +0.2        0.35 ± 12%  perf-profile.children.cycles-pp.__alloc_pages
      0.51 ± 11%      +0.2        0.66 ±  4%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.06 ± 71%      +0.2        0.21 ± 45%  perf-profile.children.cycles-pp.__alloc_file
      0.07 ± 23%      +0.2        0.23 ± 39%  perf-profile.children.cycles-pp.alloc_empty_file
      0.50 ± 11%      +0.2        0.66 ±  5%  perf-profile.children.cycles-pp.xlog_cil_push_work
      0.25 ±  5%      +0.2        0.41 ± 26%  perf-profile.children.cycles-pp.filemap_map_pages
      0.33 ±  9%      +0.2        0.49 ±  8%  perf-profile.children.cycles-pp.xlog_write
      0.00            +0.2        0.17 ± 38%  perf-profile.children.cycles-pp.xfs_dir2_block_lookup
      0.00            +0.2        0.17 ± 38%  perf-profile.children.cycles-pp.xfs_dir2_block_lookup_int
      0.30 ±  6%      +0.2        0.46 ±  7%  perf-profile.children.cycles-pp.__do_sys_clone
      0.00            +0.2        0.17 ± 17%  perf-profile.children.cycles-pp.shrink_dentry_list
      0.30 ±  8%      +0.2        0.47 ±  6%  perf-profile.children.cycles-pp.kernel_clone
      0.24 ±  5%      +0.2        0.42 ± 28%  perf-profile.children.cycles-pp.iomap_apply
      0.98 ±  5%      +0.2        1.15 ±  7%  perf-profile.children.cycles-pp.task_tick_fair
      0.16 ± 42%      +0.2        0.33 ± 24%  perf-profile.children.cycles-pp.perf_mmap__read_head
      0.24 ±  5%      +0.2        0.42 ± 28%  perf-profile.children.cycles-pp.iomap_file_buffered_write
      0.43 ± 18%      +0.2        0.61 ± 10%  perf-profile.children.cycles-pp.kmem_cache_free
      0.08 ± 32%      +0.2        0.26 ± 29%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.43 ±  8%      +0.2        0.62        perf-profile.children.cycles-pp.filename_lookup
      1.15 ± 12%      +0.2        1.34 ±  2%  perf-profile.children.cycles-pp.xfs_buf_lock
      0.21 ±  3%      +0.2        0.40 ± 11%  perf-profile.children.cycles-pp.copy_process
      1.14 ± 12%      +0.2        1.33 ±  2%  perf-profile.children.cycles-pp.__down
      1.35 ± 11%      +0.2        1.54 ±  7%  perf-profile.children.cycles-pp.newidle_balance
      1.28 ± 11%      +0.2        1.47 ±  2%  perf-profile.children.cycles-pp.schedule_timeout
      0.13 ± 16%      +0.2        0.33 ± 15%  perf-profile.children.cycles-pp.__lookup_slow
      0.00            +0.2        0.20 ± 20%  perf-profile.children.cycles-pp.__pagevec_release
      1.14 ± 12%      +0.2        1.34 ±  2%  perf-profile.children.cycles-pp.down
      0.25 ± 19%      +0.2        0.46 ±  4%  perf-profile.children.cycles-pp.mmap_region
      0.71 ±  8%      +0.2        0.92 ± 11%  perf-profile.children.cycles-pp.walk_component
      0.23 ±  9%      +0.2        0.44 ± 10%  perf-profile.children.cycles-pp.dput
      0.38 ±  6%      +0.2        0.59 ±  9%  perf-profile.children.cycles-pp.load_elf_binary
      0.07 ± 89%      +0.2        0.29 ±  8%  perf-profile.children.cycles-pp.release_pages
      0.40 ±  5%      +0.2        0.62 ±  8%  perf-profile.children.cycles-pp.exec_binprm
      0.43 ±  8%      +0.2        0.65 ±  3%  perf-profile.children.cycles-pp.path_lookupat
      0.58 ±  4%      +0.2        0.80 ±  9%  perf-profile.children.cycles-pp.execve
      0.27 ± 21%      +0.2        0.49 ±  9%  perf-profile.children.cycles-pp.do_mmap
      0.06 ± 19%      +0.2        0.29 ± 14%  perf-profile.children.cycles-pp.__dentry_kill
      0.33 ±  8%      +0.2        0.56 ± 29%  perf-profile.children.cycles-pp.xfs_file_buffered_write
      0.37 ±  9%      +0.2        0.60 ± 28%  perf-profile.children.cycles-pp.do_iter_write
      0.13 ± 64%      +0.2        0.36 ± 59%  perf-profile.children.cycles-pp.__fput
      0.09 ± 70%      +0.2        0.32 ± 18%  perf-profile.children.cycles-pp.xfs_dir3_leaf_read
      0.10 ± 39%      +0.2        0.33 ± 61%  perf-profile.children.cycles-pp.do_open
      0.47            +0.2        0.71 ±  7%  perf-profile.children.cycles-pp.bprm_execve
      0.00            +0.2        0.24 ± 12%  perf-profile.children.cycles-pp.xas_find
      0.26 ± 16%      +0.2        0.50 ±  9%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.37 ±  9%      +0.2        0.61 ± 29%  perf-profile.children.cycles-pp.lo_write_bvec
      0.30 ± 17%      +0.2        0.55 ±  7%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.30 ± 14%      +0.2        0.55 ±  7%  perf-profile.children.cycles-pp.do_exit
      0.30 ± 14%      +0.3        0.55 ±  6%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.30 ± 14%      +0.3        0.55 ±  6%  perf-profile.children.cycles-pp.do_group_exit
      0.33 ±  8%      +0.3        0.59 ± 26%  perf-profile.children.cycles-pp.do_iter_readv_writev
      0.27 ± 37%      +0.3        0.54 ± 25%  perf-profile.children.cycles-pp.rcu_do_batch
      0.21 ± 45%      +0.3        0.48 ± 36%  perf-profile.children.cycles-pp.perf_mmap__push
      0.00            +0.3        0.27 ± 30%  perf-profile.children.cycles-pp.common_cpu_die.cold
      0.00            +0.3        0.27 ± 23%  perf-profile.children.cycles-pp.prune_dcache_sb
      0.35 ±  9%      +0.3        0.63 ±  8%  perf-profile.children.cycles-pp.exit_mmap
      0.23 ± 46%      +0.3        0.51 ± 36%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.35 ±  9%      +0.3        0.64 ±  9%  perf-profile.children.cycles-pp.mmput
      0.16 ± 52%      +0.3        0.45 ± 51%  perf-profile.children.cycles-pp.task_work_run
      0.00            +0.3        0.30 ± 28%  perf-profile.children.cycles-pp.takedown_cpu
      0.80 ± 17%      +0.3        1.10 ± 21%  perf-profile.children.cycles-pp.loop_queue_work
      0.63 ± 23%      +0.3        0.93 ± 16%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.3        0.31 ± 35%  perf-profile.children.cycles-pp.build_sched_domains
      0.17 ± 70%      +0.3        0.48 ±  8%  perf-profile.children.cycles-pp.xfs_dir2_leaf_lookup_int
      0.58 ±  4%      +0.3        0.90 ± 11%  perf-profile.children.cycles-pp.do_execveat_common
      0.58 ±  4%      +0.3        0.91 ± 12%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.00            +0.3        0.33 ± 33%  perf-profile.children.cycles-pp.partition_sched_domains_locked
      0.37 ±  9%      +0.3        0.70 ± 21%  perf-profile.children.cycles-pp.do_fault
      0.75 ±  9%      +0.3        1.09 ± 26%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.77 ±  5%      +0.3        1.12 ± 10%  perf-profile.children.cycles-pp.vprintk_emit
      0.77 ±  5%      +0.3        1.12 ±  9%  perf-profile.children.cycles-pp.console_unlock
      0.00            +0.4        0.35 ± 29%  perf-profile.children.cycles-pp.super_cache_scan
      0.00            +0.4        0.35 ± 30%  perf-profile.children.cycles-pp.drop_slab
      0.00            +0.4        0.35 ± 30%  perf-profile.children.cycles-pp.drop_slab_node
      0.00            +0.4        0.35 ± 30%  perf-profile.children.cycles-pp.shrink_slab
      0.00            +0.4        0.35 ± 30%  perf-profile.children.cycles-pp.do_shrink_slab
      0.74 ±  2%      +0.4        1.10 ± 11%  perf-profile.children.cycles-pp.printk
      0.57 ±  7%      +0.4        0.93 ± 12%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.58 ±  7%      +0.4        0.95 ± 13%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.28 ± 31%      +0.4        0.64 ± 43%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.58 ±  8%      +0.4        0.95 ± 13%  perf-profile.children.cycles-pp.uart_console_write
      0.87 ± 21%      +0.4        1.25 ± 16%  perf-profile.children.cycles-pp.kthread_worker_fn
      0.30 ± 28%      +0.4        0.68 ± 41%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.59 ±  8%      +0.4        0.98 ± 14%  perf-profile.children.cycles-pp.serial8250_console_write
      0.00            +0.4        0.43 ± 12%  perf-profile.children.cycles-pp.cpuhp_invoke_callback_range
      0.00            +0.4        0.44 ± 14%  perf-profile.children.cycles-pp.cpu_down
      0.00            +0.4        0.44 ± 14%  perf-profile.children.cycles-pp._cpu_down
      0.00            +0.4        0.45 ± 14%  perf-profile.children.cycles-pp.kernfs_fop_write_iter
      0.00            +0.4        0.45 ± 14%  perf-profile.children.cycles-pp.online_store
      0.00            +0.4        0.45 ± 14%  perf-profile.children.cycles-pp.device_offline
      0.00            +0.5        0.46 ± 13%  perf-profile.children.cycles-pp.cpuhp_invoke_callback
      0.00            +0.6        0.56 ± 10%  perf-profile.children.cycles-pp.find_lock_entries
      0.05 ±  8%      +0.6        0.65 ± 84%  perf-profile.children.cycles-pp.xfs_ialloc_read_agi
      1.35 ± 25%      +0.6        1.95 ± 14%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.80 ±  8%      +0.6        1.40 ± 16%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.00            +0.6        0.61 ± 98%  perf-profile.children.cycles-pp.xfs_init_new_inode
      0.00            +0.6        0.63 ±102%  perf-profile.children.cycles-pp.xfs_dir_lookup
      0.00            +0.6        0.64 ±100%  perf-profile.children.cycles-pp.xfs_lookup
      2.15 ± 16%      +0.7        2.84 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock
      0.89 ±  5%      +0.7        1.59 ± 14%  perf-profile.children.cycles-pp.handle_mm_fault
      0.00            +0.7        0.70 ±101%  perf-profile.children.cycles-pp.xfs_vn_lookup
      1.11 ±  4%      +0.7        1.83 ± 14%  perf-profile.children.cycles-pp.do_user_addr_fault
      1.18 ±  3%      +0.7        1.90 ± 14%  perf-profile.children.cycles-pp.exc_page_fault
      1.26            +0.8        2.06 ± 15%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.00            +1.0        1.00 ±102%  perf-profile.children.cycles-pp.xfs_dir_createname
      0.00            +1.1        1.09 ± 12%  perf-profile.children.cycles-pp.__invalidate_mapping_pages
      0.00            +1.1        1.14 ± 34%  perf-profile.children.cycles-pp.mnt_get_count
      0.00            +1.2        1.19 ± 32%  perf-profile.children.cycles-pp.mntput_no_expire
      0.58 ± 18%      +1.3        1.87 ±  9%  perf-profile.children.cycles-pp.drm_fb_memcpy_dstclip
      0.59 ± 17%      +1.3        1.88 ±  9%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_planes
      0.59 ± 17%      +1.3        1.88 ±  9%  perf-profile.children.cycles-pp.mgag200_simple_display_pipe_update
      0.58 ± 19%      +1.3        1.87 ±  9%  perf-profile.children.cycles-pp.memcpy_toio
      0.59 ± 17%      +1.3        1.88 ±  9%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_tail
      0.59 ± 17%      +1.3        1.89 ±  9%  perf-profile.children.cycles-pp.commit_tail
      0.60 ± 15%      +1.3        1.90 ±  9%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
      0.59 ± 17%      +1.3        1.89 ±  9%  perf-profile.children.cycles-pp.drm_atomic_helper_commit
      0.00            +1.3        1.31 ± 33%  perf-profile.children.cycles-pp.simple_recursive_removal
      0.00            +1.3        1.32 ± 33%  perf-profile.children.cycles-pp.debugfs_remove
      0.61 ± 15%      +1.3        1.95 ±  9%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.00            +1.4        1.41 ± 13%  perf-profile.children.cycles-pp.iterate_supers
      0.00            +1.4        1.41 ± 13%  perf-profile.children.cycles-pp.drop_pagecache_sb
      0.00            +1.5        1.54 ± 33%  perf-profile.children.cycles-pp.update_sched_domain_debugfs
      0.00            +1.8        1.76 ± 14%  perf-profile.children.cycles-pp.drop_caches_sysctl_handler
      0.00            +1.8        1.78 ± 13%  perf-profile.children.cycles-pp.proc_sys_call_handler
      0.00            +2.0        1.96 ± 31%  perf-profile.children.cycles-pp.cpuset_hotplug_workfn
      0.00            +2.0        1.96 ± 31%  perf-profile.children.cycles-pp.rebuild_sched_domains
      0.00            +2.0        1.96 ± 31%  perf-profile.children.cycles-pp.rebuild_sched_domains_locked
      0.40 ± 10%      +2.1        2.53 ±  9%  perf-profile.children.cycles-pp.write
      0.00            +2.2        2.17 ±106%  perf-profile.children.cycles-pp.xfs_trans_alloc_icreate
      1.06 ±  2%      +2.3        3.36 ± 11%  perf-profile.children.cycles-pp.new_sync_write
      1.10 ±  2%      +2.3        3.42 ± 11%  perf-profile.children.cycles-pp.vfs_write
      1.11 ±  3%      +2.3        3.45 ± 12%  perf-profile.children.cycles-pp.ksys_write
      0.00            +2.5        2.49 ±  5%  perf-profile.children.cycles-pp.rcu_momentary_dyntick_idle
      0.00            +2.8        2.76 ±  8%  perf-profile.children.cycles-pp.multi_cpu_stop
      0.02 ±141%      +2.8        2.79 ±  8%  perf-profile.children.cycles-pp.cpu_stopper_thread
      0.33 ± 38%      +3.0        3.28 ±  9%  perf-profile.children.cycles-pp.smpboot_thread_fn
      2.33 ±  5%      +3.5        5.81 ± 13%  perf-profile.children.cycles-pp.worker_thread
      1.99 ±  4%      +3.5        5.47 ± 13%  perf-profile.children.cycles-pp.process_one_work
      0.00            +4.0        4.02 ±120%  perf-profile.children.cycles-pp.xfs_dialloc
      4.11 ±  8%      +6.8       10.89 ±  5%  perf-profile.children.cycles-pp.kthread
      4.14 ±  8%      +6.8       10.93 ±  5%  perf-profile.children.cycles-pp.ret_from_fork
      0.00            +9.8        9.84 ±112%  perf-profile.children.cycles-pp.xfs_create
      0.00           +10.0       10.00 ±112%  perf-profile.children.cycles-pp.xfs_generic_create
      0.47 ±  4%     +11.9       12.40 ±103%  perf-profile.children.cycles-pp.path_openat
      0.47 ±  4%     +12.0       12.42 ±103%  perf-profile.children.cycles-pp.do_filp_open
      0.22 ± 14%     +12.1       12.30 ±107%  perf-profile.children.cycles-pp.open64
      0.54           +12.2       12.77 ±101%  perf-profile.children.cycles-pp.do_sys_openat2
      0.55 ±  2%     +12.2       12.79 ±101%  perf-profile.children.cycles-pp.do_sys_open
      2.21 ±  8%      -1.3        0.94 ± 26%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      1.05 ±  8%      -0.6        0.45 ± 42%  perf-profile.self.cycles-pp.xfs_inobt_get_rec
      3.21 ± 12%      -0.6        2.63 ± 11%  perf-profile.self.cycles-pp.menu_select
      0.84 ±  7%      -0.5        0.32 ± 18%  perf-profile.self.cycles-pp.xfs_btree_get_rec
      0.93 ±  5%      -0.5        0.42 ± 19%  perf-profile.self.cycles-pp.xfs_inobt_get_maxrecs
      0.96 ± 11%      -0.5        0.46 ± 47%  perf-profile.self.cycles-pp.xfs_btree_increment
      1.02 ± 10%      -0.5        0.56 ± 25%  perf-profile.self.cycles-pp.xfs_errortag_test
      0.86 ± 10%      -0.4        0.47 ± 20%  perf-profile.self.cycles-pp.xfs_next_bit
      0.70 ± 12%      -0.4        0.32 ± 42%  perf-profile.self.cycles-pp.xfs_btree_check_sblock
      0.62 ± 11%      -0.3        0.33 ± 25%  perf-profile.self.cycles-pp.xfs_verify_agino
      0.41 ± 10%      -0.2        0.18 ± 33%  perf-profile.self.cycles-pp.xfs_btree_get_block
      0.43 ± 20%      -0.2        0.22 ± 22%  perf-profile.self.cycles-pp.xfs_inobt_irec_to_allocmask
      0.48 ± 10%      -0.2        0.27 ± 44%  perf-profile.self.cycles-pp.xfs_btree_check_block
      0.33 ± 11%      -0.2        0.15 ± 19%  perf-profile.self.cycles-pp.xfs_check_agi_freecount
      0.36 ± 20%      -0.2        0.19 ± 26%  perf-profile.self.cycles-pp.xfs_inobt_btrec_to_irec
      0.39 ± 18%      -0.2        0.24 ±  8%  perf-profile.self.cycles-pp.update_blocked_averages
      0.67 ±  6%      -0.1        0.52 ±  4%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.33 ± 11%      -0.1        0.19 ± 29%  perf-profile.self.cycles-pp.clockevents_program_event
      0.33 ±  6%      -0.1        0.20 ± 34%  perf-profile.self.cycles-pp.xfs_btree_rec_offset
      0.42 ± 11%      -0.1        0.30 ± 19%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.78 ±  3%      -0.1        0.67 ± 11%  perf-profile.self.cycles-pp.read_tsc
      0.22 ±  5%      -0.1        0.11 ± 18%  perf-profile.self.cycles-pp.update_dl_rq_load_avg
      0.26 ± 12%      -0.1        0.15 ± 12%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.20 ± 18%      -0.1        0.10 ± 37%  perf-profile.self.cycles-pp.xfs_btree_readahead
      0.29 ± 10%      -0.1        0.20 ±  4%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.41 ±  7%      -0.1        0.32 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.20 ± 10%      -0.1        0.11 ± 28%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.66 ±  3%      -0.1        0.58 ±  5%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.35 ±  6%      -0.1        0.27 ±  9%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.31 ± 10%      -0.1        0.23 ± 17%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.40 ±  2%      -0.1        0.31 ± 18%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.19 ±  4%      -0.1        0.12 ± 10%  perf-profile.self.cycles-pp.tick_sched_timer
      0.22 ±  2%      -0.1        0.14 ±  8%  perf-profile.self.cycles-pp.enqueue_entity
      0.31 ±  3%      -0.1        0.23 ± 19%  perf-profile.self.cycles-pp.irq_exit_rcu
      0.62 ±  5%      -0.1        0.55 ±  2%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.44 ±  8%      -0.1        0.37 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.21 ± 14%      -0.1        0.14 ± 25%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.16 ± 15%      -0.1        0.09 ± 26%  perf-profile.self.cycles-pp.hrtimer_next_event_without
      0.26 ± 15%      -0.1        0.19 ± 10%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.22 ± 14%      -0.1        0.16 ± 13%  perf-profile.self.cycles-pp.get_cpu_device
      0.17 ±  8%      -0.1        0.12 ± 22%  perf-profile.self.cycles-pp.get_perf_callchain
      0.14 ± 11%      -0.1        0.09 ± 15%  perf-profile.self.cycles-pp.__switch_to_asm
      0.32 ±  5%      -0.1        0.26 ± 11%  perf-profile.self.cycles-pp.scheduler_tick
      0.31 ±  4%      -0.1        0.25 ±  4%  perf-profile.self.cycles-pp.rb_erase
      0.14 ±  8%      -0.1        0.09 ±  5%  perf-profile.self.cycles-pp.send_call_function_single_ipi
      0.17 ± 22%      -0.0        0.12 ± 10%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.14 ± 20%      -0.0        0.09 ± 27%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.16 ± 10%      -0.0        0.11 ± 12%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.12 ± 27%      -0.0        0.07 ± 28%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.08 ± 14%      -0.0        0.04 ± 70%  perf-profile.self.cycles-pp.xfs_trans_reserve
      0.14 ± 13%      -0.0        0.09 ± 20%  perf-profile.self.cycles-pp.xfs_trans_buf_item_match
      0.20 ±  6%      -0.0        0.16 ± 10%  perf-profile.self.cycles-pp.rb_next
      0.08 ± 14%      -0.0        0.04 ± 71%  perf-profile.self.cycles-pp.xfs_buf_item_size_segment
      0.24 ± 11%      -0.0        0.20 ±  2%  perf-profile.self.cycles-pp.perf_output_copy
      0.18 ±  2%      -0.0        0.14 ± 10%  perf-profile.self.cycles-pp.__remove_hrtimer
      0.16 ±  2%      -0.0        0.13 ±  6%  perf-profile.self.cycles-pp.update_curr
      0.10 ± 21%      -0.0        0.07 ± 20%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.17 ± 14%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.09 ± 14%      -0.0        0.07 ± 14%  perf-profile.self.cycles-pp.account_process_tick
      0.11 ±  7%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.dequeue_entity
      0.07 ± 28%      +0.0        0.10 ± 14%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.07 ± 18%      +0.0        0.10 ± 12%  perf-profile.self.cycles-pp.perf_tp_event_match
      0.07 ± 23%      +0.0        0.10 ± 12%  perf-profile.self.cycles-pp.symbol__is_idle
      0.06 ±  8%      +0.0        0.09        perf-profile.self.cycles-pp.__cond_resched
      0.07 ± 25%      +0.0        0.10        perf-profile.self.cycles-pp.__slab_free
      0.11 ± 17%      +0.0        0.15 ± 11%  perf-profile.self.cycles-pp.memset_erms
      0.10 ±  9%      +0.0        0.14 ± 28%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.06 ± 13%      +0.0        0.10 ± 29%  perf-profile.self.cycles-pp.map_id_range_down
      0.05 ± 70%      +0.0        0.09 ± 15%  perf-profile.self.cycles-pp.down_write
      0.12 ±  7%      +0.1        0.18 ± 10%  perf-profile.self.cycles-pp.__might_sleep
      0.09 ± 10%      +0.1        0.15 ± 17%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.02 ±141%      +0.1        0.07 ± 23%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.06 ± 23%      +0.1        0.12 ± 27%  perf-profile.self.cycles-pp.zap_pte_range
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.vma_interval_tree_insert
      0.00            +0.1        0.06 ± 23%  perf-profile.self.cycles-pp.call_rcu
      0.04 ± 71%      +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.04 ± 73%      +0.1        0.10 ± 32%  perf-profile.self.cycles-pp.sys_imageblit
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.perf_output_put_handle
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.try_charge
      0.00            +0.1        0.07 ± 25%  perf-profile.self.cycles-pp.pagecache_get_page
      0.04 ± 71%      +0.1        0.11 ± 23%  perf-profile.self.cycles-pp.cpumask_next
      0.17 ± 14%      +0.1        0.24 ± 23%  perf-profile.self.cycles-pp.evlist__parse_sample
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.___d_drop
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.clear_page_erms
      0.00            +0.1        0.07 ± 28%  perf-profile.self.cycles-pp.xas_load
      0.00            +0.1        0.08 ± 16%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.03 ±141%      +0.1        0.11 ± 34%  perf-profile.self.cycles-pp.perf_session__process_events
      0.00            +0.1        0.09 ± 30%  perf-profile.self.cycles-pp._PyEval_EvalFrameDefault
      0.33 ± 15%      +0.1        0.42 ±  8%  perf-profile.self.cycles-pp.kmem_cache_free
      0.42 ± 11%      +0.1        0.51 ±  6%  perf-profile.self.cycles-pp.__module_address
      0.00            +0.1        0.09 ± 18%  perf-profile.self.cycles-pp.page_mapping
      0.20 ± 16%      +0.1        0.31 ± 26%  perf-profile.self.cycles-pp.delay_tsc
      0.00            +0.1        0.12 ± 30%  perf-profile.self.cycles-pp.multi_cpu_stop
      0.02 ±141%      +0.1        0.15 ± 16%  perf-profile.self.cycles-pp.path_init
      0.48 ± 15%      +0.2        0.63 ± 10%  perf-profile.self.cycles-pp.io_serial_in
      0.00            +0.2        0.15 ± 17%  perf-profile.self.cycles-pp.release_pages
      0.14 ± 45%      +0.2        0.31 ± 27%  perf-profile.self.cycles-pp.perf_mmap__read_head
      0.00            +0.2        0.18 ±  9%  perf-profile.self.cycles-pp.xas_find
      0.00            +0.2        0.20 ± 12%  perf-profile.self.cycles-pp.drop_pagecache_sb
      0.00            +0.3        0.34 ± 10%  perf-profile.self.cycles-pp.find_lock_entries
      1.32 ± 26%      +0.5        1.83 ± 13%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.00            +1.0        0.98 ± 37%  perf-profile.self.cycles-pp.mnt_get_count
      0.51 ± 13%      +1.3        1.81 ±  7%  perf-profile.self.cycles-pp.memcpy_toio
      0.00            +2.5        2.49 ±  5%  perf-profile.self.cycles-pp.rcu_momentary_dyntick_idle





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.13.0-rc4-00092-geef983ffeae7" of type "text/plain" (174208 bytes)

View attachment "job-script" of type "text/plain" (8459 bytes)

View attachment "job.yaml" of type "text/plain" (5791 bytes)

View attachment "reproduce" of type "text/plain" (556 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ