lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Mon, 8 Mar 2021 23:03:13 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     "Darrick J. Wong" <djwong@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com
Subject: [xfs]  571519716f:  aim7.jobs-per-min -8.5% regression


Greeting,

FYI, we noticed a -8.5% regression of aim7.jobs-per-min due to commit:


commit: 571519716f116bfc5c2370a15dea5afb0f04179b ("xfs: parallelize inode inactivation")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git vectorized-scrub


in testcase: aim7
on test machine: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: xfs
	test: disk_src
	load: 3000
	cpufreq_governor: performance
	ucode: 0x5003006

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/4BRD_12G/xfs/x86_64-rhel-8.3/3000/RAID0/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/disk_src/aim7/0x5003006

commit: 
  5990126595 ("xfs: force inactivation before fallocate when space is low")
  571519716f ("xfs: parallelize inode inactivation")

59901265956d20cb 571519716f116bfc5c2370a15de 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    108149            -8.5%      98947        aim7.jobs-per-min
    236.54           -22.5%     183.30        aim7.time.elapsed_time
    236.54           -22.5%     183.30        aim7.time.elapsed_time.max
      4566          +358.7%      20944 ±  4%  aim7.time.involuntary_context_switches
    177434           +22.4%     217099        aim7.time.minor_page_faults
      2279          +147.7%       5646 ±  3%  aim7.time.system_time
     31.61           +12.3%      35.49        aim7.time.user_time
   6704390          +141.0%   16158662        aim7.time.voluntary_context_switches
     86.90           -27.9%      62.66        iostat.cpu.idle
     12.88          +187.3%      37.01 ±  3%  iostat.cpu.system
    267.24           -19.9%     214.14        uptime.boot
     20352           -39.5%      12308        uptime.idle
 1.027e+08          +752.5%  8.751e+08 ± 53%  cpuidle.C1.time
   3604271          +271.3%   13383057 ± 33%  cpuidle.C1.usage
    891624 ±  2%    +138.6%    2126996 ±  2%  cpuidle.POLL.time
     87879 ± 16%     +68.3%     147905 ± 13%  cpuidle.POLL.usage
     87.39           -21.6       65.77        mpstat.cpu.all.idle%
      0.96 ±  7%      -0.1        0.83 ±  2%  mpstat.cpu.all.irq%
     11.30           +21.6       32.91 ±  2%  mpstat.cpu.all.sys%
      0.23            +0.1        0.36        mpstat.cpu.all.usr%
   2211040 ±  4%     -47.3%    1164747 ±  3%  numa-numastat.node0.local_node
   2246947 ±  4%     -46.2%    1209142 ±  3%  numa-numastat.node0.numa_hit
   2402618 ±  3%     -46.2%    1293556 ±  4%  numa-numastat.node1.local_node
   2446199 ±  3%     -45.7%    1328623 ±  3%  numa-numastat.node1.numa_hit
     86.33           -28.0%      62.17        vmstat.cpu.id
     66546           +35.0%      89821        vmstat.io.bo
   8312207           -69.0%    2576633        vmstat.memory.cache
 1.142e+08           +10.8%  1.266e+08        vmstat.memory.free
     10.50 ±  4%    +209.5%      32.50 ±  3%  vmstat.procs.r
     92757          +194.4%     273041        vmstat.system.cs
    178826            +3.1%     184349        vmstat.system.in
  14290706           +10.6%   15810805        numa-vmstat.node0.nr_free_pages
      5692 ± 17%     +79.7%      10231 ± 13%  numa-vmstat.node0.nr_mapped
    839062 ±  7%     -87.5%     104521 ±  3%  numa-vmstat.node0.nr_slab_reclaimable
    324884 ±  2%     -80.0%      64947 ± 14%  numa-vmstat.node0.nr_slab_unreclaimable
   2058119 ±  5%     -35.8%    1322045 ±  5%  numa-vmstat.node0.numa_hit
   1997075 ±  7%     -36.2%    1273530 ±  6%  numa-vmstat.node0.numa_local
  14247635           +11.1%   15827627        numa-vmstat.node1.nr_free_pages
      7389 ± 12%     +94.4%      14364 ± 14%  numa-vmstat.node1.nr_mapped
    887148 ±  7%     -87.8%     108115 ±  3%  numa-vmstat.node1.nr_slab_reclaimable
    328412 ±  2%     -79.7%      66532 ± 13%  numa-vmstat.node1.nr_slab_unreclaimable
   2167284 ±  5%     -35.0%    1407907 ±  4%  numa-vmstat.node1.numa_hit
   1996184 ±  7%     -38.7%    1224459 ±  6%  numa-vmstat.node1.numa_local
   3357217 ±  7%     -87.5%     418102 ±  3%  numa-meminfo.node0.KReclaimable
     23024 ± 16%     +79.8%      41388 ± 14%  numa-meminfo.node0.Mapped
  57161095           +10.6%   63242385        numa-meminfo.node0.MemFree
   8498611 ±  3%     -71.6%    2417321 ±  6%  numa-meminfo.node0.MemUsed
   3357217 ±  7%     -87.5%     418102 ±  3%  numa-meminfo.node0.SReclaimable
   1298826 ±  2%     -80.0%     259793 ± 14%  numa-meminfo.node0.SUnreclaim
   4656044 ±  5%     -85.4%     677897 ±  4%  numa-meminfo.node0.Slab
   3549237 ±  6%     -87.8%     432411 ±  3%  numa-meminfo.node1.KReclaimable
     29245 ± 11%     +95.8%      57264 ± 15%  numa-meminfo.node1.Mapped
  56989796           +11.1%   63309357        numa-meminfo.node1.MemFree
   9024618 ±  3%     -70.0%    2705057 ±  6%  numa-meminfo.node1.MemUsed
   3549237 ±  6%     -87.8%     432411 ±  3%  numa-meminfo.node1.SReclaimable
   1313398 ±  2%     -79.7%     266262 ± 13%  numa-meminfo.node1.SUnreclaim
   4862636 ±  5%     -85.6%     698674 ±  4%  numa-meminfo.node1.Slab
    359748           +62.7%     585292 ±  4%  meminfo.Active
    359708           +62.7%     585249 ±  4%  meminfo.Active(anon)
    388216           +17.1%     454459        meminfo.AnonPages
   1436350           +20.0%    1722983        meminfo.Cached
   2376805           +40.2%    3333117        meminfo.Committed_AS
    418804           +30.7%     547365        meminfo.Inactive
    417939           +30.7%     546210        meminfo.Inactive(anon)
    864.33           +33.6%       1155        meminfo.Inactive(file)
   6903307           -87.7%     850432        meminfo.KReclaimable
     49305           +25.7%      61981        meminfo.KernelStack
     51829 ±  3%     +88.3%      97613 ±  6%  meminfo.Mapped
 1.142e+08           +10.9%  1.266e+08        meminfo.MemFree
  17519745           -70.8%    5121789        meminfo.Memused
     78147 ±  6%     +36.8%     106920 ±  6%  meminfo.PageTables
   6903307           -87.7%     850432        meminfo.SReclaimable
   2610613           -79.9%     525729        meminfo.SUnreclaim
    390244           +73.6%     677602 ±  4%  meminfo.Shmem
   9513922           -85.5%    1376163        meminfo.Slab
    265300           -19.6%     213393        meminfo.VmallocUsed
  18622214           -71.4%    5323447        meminfo.max_used_kB
    803.18 ±143%   +1241.2%      10772 ± 67%  sched_debug.cfs_rq:/.MIN_vruntime.avg
     70627 ±143%    +822.7%     651662 ± 44%  sched_debug.cfs_rq:/.MIN_vruntime.max
      7488 ±143%    +989.7%      81606 ± 54%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    803.18 ±143%   +1241.2%      10772 ± 67%  sched_debug.cfs_rq:/.max_vruntime.avg
     70627 ±143%    +822.7%     651662 ± 44%  sched_debug.cfs_rq:/.max_vruntime.max
      7488 ±143%    +989.7%      81606 ± 54%  sched_debug.cfs_rq:/.max_vruntime.stddev
    559726 ± 10%    +257.3%    1999799 ±  5%  sched_debug.cfs_rq:/.min_vruntime.avg
    655367 ± 11%    +220.0%    2097164 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
    530979 ±  9%    +260.6%    1914846 ±  5%  sched_debug.cfs_rq:/.min_vruntime.min
     16452 ± 23%     +95.8%      32210 ±  4%  sched_debug.cfs_rq:/.min_vruntime.stddev
     16452 ± 23%     +96.2%      32276 ±  4%  sched_debug.cfs_rq:/.spread0.stddev
     13.60 ±  9%    +122.6%      30.28 ± 58%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    748950 ±  4%     -23.4%     573996 ±  9%  sched_debug.cpu.avg_idle.avg
    194819 ± 10%     +24.5%     242630 ±  3%  sched_debug.cpu.avg_idle.stddev
     99657 ± 14%     +20.1%     119713        sched_debug.cpu.clock.max
     93959 ± 14%     +21.2%     113859        sched_debug.cpu.clock_task.min
      6510 ±  7%     +10.3%       7178        sched_debug.cpu.curr->pid.max
     95951 ± 20%    +183.5%     272062        sched_debug.cpu.nr_switches.avg
    242598 ± 17%     +60.8%     390165 ±  2%  sched_debug.cpu.nr_switches.max
     65092 ± 22%    +209.7%     201612        sched_debug.cpu.nr_switches.min
     37119 ± 14%     +72.8%      64159 ±  2%  sched_debug.cpu.nr_switches.stddev
     88.33 ±  6%    +103.7%     179.96 ± 16%  sched_debug.cpu.nr_uninterruptible.max
    -23.43          +404.3%    -118.17        sched_debug.cpu.nr_uninterruptible.min
     20.00 ±  9%    +257.2%      71.44 ±  8%  sched_debug.cpu.nr_uninterruptible.stddev
     89887           +62.6%     146172 ±  4%  proc-vmstat.nr_active_anon
     97013           +17.2%     113668        proc-vmstat.nr_anon_pages
   2834284           +10.9%    3143758        proc-vmstat.nr_dirty_background_threshold
   5675499           +10.9%    6295204        proc-vmstat.nr_dirty_threshold
    359131           +20.0%     430872        proc-vmstat.nr_file_pages
  28538834           +10.9%   31638038        proc-vmstat.nr_free_pages
    104526           +30.9%     136876        proc-vmstat.nr_inactive_anon
    215.67           +33.5%     288.00        proc-vmstat.nr_inactive_file
     49271           +25.9%      62014        proc-vmstat.nr_kernel_stack
     13238 ±  3%     +88.3%      24922 ±  6%  proc-vmstat.nr_mapped
     19507 ±  6%     +37.1%      26751 ±  6%  proc-vmstat.nr_page_table_pages
     97603           +73.7%     169525 ±  4%  proc-vmstat.nr_shmem
   1725915           -87.7%     212524        proc-vmstat.nr_slab_reclaimable
    652898           -79.9%     131472        proc-vmstat.nr_slab_unreclaimable
     89887           +62.6%     146172 ±  4%  proc-vmstat.nr_zone_active_anon
    104526           +30.9%     136876        proc-vmstat.nr_zone_inactive_anon
    215.67           +33.5%     288.00        proc-vmstat.nr_zone_inactive_file
     13328 ± 27%    +827.4%     123607 ±  7%  proc-vmstat.numa_hint_faults
      5863 ± 25%    +798.8%      52700 ±  8%  proc-vmstat.numa_hint_faults_local
   4726940           -45.5%    2576177        proc-vmstat.numa_hit
   4647432           -46.3%    2496652        proc-vmstat.numa_local
     15254 ± 84%    +209.5%      47215 ± 21%  proc-vmstat.numa_pages_migrated
    139617 ± 16%     +89.6%     264772 ±  8%  proc-vmstat.numa_pte_updates
    146307          +108.2%     304588 ±  5%  proc-vmstat.pgactivate
   8634293           -56.0%    3796525        proc-vmstat.pgalloc_normal
   6513591 ±  8%     -52.5%    3091884 ±  2%  proc-vmstat.pgfree
     15254 ± 84%    +209.5%      47215 ± 21%  proc-vmstat.pgmigrate_success
  15960004            +5.0%   16750452        proc-vmstat.pgpgout
     46486           -19.8%      37267        proc-vmstat.pgreuse
      5987 ±  3%     -19.4%       4827 ±  4%  slabinfo.btrfs_ordered_extent.active_objs
      5998 ±  3%     -19.5%       4827 ±  4%  slabinfo.btrfs_ordered_extent.num_objs
     29400            -9.4%      26637        slabinfo.filp.active_objs
     25981           -18.6%      21161 ±  2%  slabinfo.kmalloc-128.active_objs
    812.33           -18.5%     661.83 ±  2%  slabinfo.kmalloc-128.active_slabs
     26006           -18.5%      21192 ±  2%  slabinfo.kmalloc-128.num_objs
    812.33           -18.5%     661.83 ±  2%  slabinfo.kmalloc-128.num_slabs
   4320451           -90.6%     405269        slabinfo.kmalloc-16.active_objs
     21613           -91.2%       1900        slabinfo.kmalloc-16.active_slabs
   5533157           -91.2%     486533        slabinfo.kmalloc-16.num_objs
     21613           -91.2%       1900        slabinfo.kmalloc-16.num_slabs
    216577           -47.7%     113370 ±  2%  slabinfo.kmalloc-32.active_objs
      1890           -52.9%     891.00        slabinfo.kmalloc-32.active_slabs
    242048           -52.8%     114126        slabinfo.kmalloc-32.num_objs
      1890           -52.9%     891.00        slabinfo.kmalloc-32.num_slabs
   3506764           -92.1%     277452        slabinfo.kmalloc-512.active_objs
    138094           -90.7%      12806        slabinfo.kmalloc-512.active_slabs
   4419029           -90.7%     409833        slabinfo.kmalloc-512.num_objs
    138094           -90.7%      12806        slabinfo.kmalloc-512.num_slabs
     58302           -12.0%      51304        slabinfo.kmalloc-64.active_objs
    915.33           -11.8%     807.00        slabinfo.kmalloc-64.active_slabs
     58616           -11.8%      51675        slabinfo.kmalloc-64.num_objs
    915.33           -11.8%     807.00        slabinfo.kmalloc-64.num_slabs
      1922           -12.6%       1681        slabinfo.kmalloc-8k.active_objs
      2131           -13.0%       1854        slabinfo.kmalloc-8k.num_objs
    173129           -88.6%      19713        slabinfo.kmalloc-rcl-256.active_objs
      5459           -88.7%     618.17        slabinfo.kmalloc-rcl-256.active_slabs
    174709           -88.7%      19796        slabinfo.kmalloc-rcl-256.num_objs
      5459           -88.7%     618.17        slabinfo.kmalloc-rcl-256.num_slabs
     31847 ±  2%     -26.3%      23461 ±  4%  slabinfo.numa_policy.active_objs
    513.83 ±  2%     -26.4%     378.17 ±  4%  slabinfo.numa_policy.active_slabs
     31891 ±  2%     -26.4%      23482 ±  4%  slabinfo.numa_policy.num_objs
    513.83 ±  2%     -26.4%     378.17 ±  4%  slabinfo.numa_policy.num_slabs
     37272           +13.1%      42137        slabinfo.proc_inode_cache.active_objs
     99554           -73.3%      26579        slabinfo.radix_tree_node.active_objs
      2098           -76.3%     497.50        slabinfo.radix_tree_node.active_slabs
    117570           -76.3%      27891        slabinfo.radix_tree_node.num_objs
      2098           -76.3%     497.50        slabinfo.radix_tree_node.num_slabs
      3356           +22.7%       4117        slabinfo.task_struct.active_objs
      3363           +22.6%       4124        slabinfo.task_struct.active_slabs
      3363           +22.6%       4124        slabinfo.task_struct.num_objs
      3363           +22.6%       4124        slabinfo.task_struct.num_slabs
    125930           +18.9%     149685        slabinfo.vm_area_struct.active_objs
      3168           +18.6%       3757        slabinfo.vm_area_struct.active_slabs
    126739           +18.6%     150308        slabinfo.vm_area_struct.num_objs
      3168           +18.6%       3757        slabinfo.vm_area_struct.num_slabs
    138388           -87.6%      17135        slabinfo.xfs_buf.active_objs
      4064           -88.4%     470.50 ±  2%  slabinfo.xfs_buf.active_slabs
    170729           -88.4%      19786 ±  2%  slabinfo.xfs_buf.num_objs
      4064           -88.4%     470.50 ±  2%  slabinfo.xfs_buf.num_slabs
      6055 ±  3%     -20.2%       4830 ±  4%  slabinfo.xfs_efi_item.active_objs
      6065 ±  3%     -20.4%       4830 ±  4%  slabinfo.xfs_efi_item.num_objs
   4420533           -88.8%     495791        slabinfo.xfs_ili.active_objs
    131095           -89.3%      14022        slabinfo.xfs_ili.active_slabs
   5506031           -89.3%     588952        slabinfo.xfs_ili.num_objs
    131095           -89.3%      14022        slabinfo.xfs_ili.num_slabs
   4419485           -88.8%     495393        slabinfo.xfs_inode.active_objs
    171799           -89.3%      18353        slabinfo.xfs_inode.active_slabs
   5497596           -89.3%     587310        slabinfo.xfs_inode.num_objs
    171799           -89.3%      18353        slabinfo.xfs_inode.num_slabs
 3.792e+09          +103.9%   7.73e+09 ±  2%  perf-stat.i.branch-instructions
  26669879 ±  4%     +42.7%   38045435        perf-stat.i.branch-misses
  32443895           +54.5%   50136348        perf-stat.i.cache-misses
  92675768 ±  4%     +52.0%  1.408e+08        perf-stat.i.cache-references
     94130          +198.8%     281225        perf-stat.i.context-switches
      1.49 ±  3%     +40.9%       2.10        perf-stat.i.cpi
 3.231e+10          +185.3%  9.217e+10 ±  2%  perf-stat.i.cpu-cycles
    914.25 ±  3%    +588.9%       6298 ±  2%  perf-stat.i.cpu-migrations
      1002          +111.7%       2121 ±  3%  perf-stat.i.cycles-between-cache-misses
   2147985 ±  8%     +61.7%    3473598 ± 16%  perf-stat.i.dTLB-load-misses
 4.571e+09          +112.8%  9.726e+09 ±  2%  perf-stat.i.dTLB-loads
    168896 ± 11%     +31.1%     221504 ±  3%  perf-stat.i.dTLB-store-misses
 1.528e+09           +46.7%  2.241e+09        perf-stat.i.dTLB-stores
     43.03            -8.2       34.86        perf-stat.i.iTLB-load-miss-rate%
   5299797           +41.1%    7479940        perf-stat.i.iTLB-load-misses
   6807413          +115.7%   14682398        perf-stat.i.iTLB-loads
 1.879e+10          +104.3%   3.84e+10 ±  2%  perf-stat.i.instructions
      3804           +49.2%       5676 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.74 ±  2%     -24.4%       0.56        perf-stat.i.ipc
      2.02 ± 13%     +63.7%       3.30 ± 28%  perf-stat.i.major-faults
      0.37          +184.9%       1.05 ±  2%  perf-stat.i.metric.GHz
      0.24 ± 23%     +37.7%       0.33 ±  5%  perf-stat.i.metric.K/sec
    113.74           +98.4%     225.67        perf-stat.i.metric.M/sec
      4076           +28.8%       5251        perf-stat.i.minor-faults
     74.21            +7.2       81.46        perf-stat.i.node-load-miss-rate%
  10181763           +59.4%   16227400        perf-stat.i.node-load-misses
   2901599           +26.3%    3665420        perf-stat.i.node-loads
     64.72           +12.6       77.35        perf-stat.i.node-store-miss-rate%
   4109077           +71.7%    7053317        perf-stat.i.node-store-misses
   1691346           +13.9%    1925971        perf-stat.i.node-stores
      4078           +28.8%       5255        perf-stat.i.page-faults
      4.89 ±  4%     -26.2%       3.61 ±  2%  perf-stat.overall.MPKI
      0.70 ±  3%      -0.2        0.49 ±  2%  perf-stat.overall.branch-miss-rate%
      1.73           +39.5%       2.42        perf-stat.overall.cpi
      1011           +85.8%       1879 ±  2%  perf-stat.overall.cycles-between-cache-misses
     43.75           -10.0       33.78        perf-stat.overall.iTLB-load-miss-rate%
      3573           +45.9%       5215 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.58           -28.3%       0.41        perf-stat.overall.ipc
     77.85            +3.7       81.58        perf-stat.overall.node-load-miss-rate%
     70.86            +7.7       78.54        perf-stat.overall.node-store-miss-rate%
 3.807e+09          +104.5%  7.787e+09 ±  2%  perf-stat.ps.branch-instructions
  26589990 ±  4%     +42.1%   37794427        perf-stat.ps.branch-misses
  32336766           +53.9%   49779157        perf-stat.ps.cache-misses
  92359248 ±  4%     +51.3%  1.397e+08        perf-stat.ps.cache-references
     93602          +194.9%     276022        perf-stat.ps.context-switches
 3.272e+10          +186.0%  9.358e+10 ±  3%  perf-stat.ps.cpu-cycles
    917.78 ±  3%    +579.6%       6237 ±  2%  perf-stat.ps.cpu-migrations
   2138997 ±  8%     +61.3%    3450024 ± 17%  perf-stat.ps.dTLB-load-misses
 4.592e+09          +113.4%    9.8e+09 ±  2%  perf-stat.ps.dTLB-loads
    168107 ± 11%     +30.4%     219268 ±  3%  perf-stat.ps.dTLB-store-misses
 1.524e+09           +46.0%  2.224e+09        perf-stat.ps.dTLB-stores
   5280039           +40.5%    7416135        perf-stat.ps.iTLB-load-misses
   6789662          +114.1%   14538495        perf-stat.ps.iTLB-loads
 1.887e+10          +105.0%  3.868e+10 ±  2%  perf-stat.ps.instructions
      2.02 ± 12%     +60.8%       3.25 ± 28%  perf-stat.ps.major-faults
      4063           +28.0%       5201        perf-stat.ps.minor-faults
  10148506           +58.7%   16101826        perf-stat.ps.node-load-misses
   2887866           +25.9%    3636671        perf-stat.ps.node-loads
   4097260           +70.9%    7003091        perf-stat.ps.node-store-misses
   1684554           +13.6%    1913918        perf-stat.ps.node-stores
      4065           +28.0%       5204        perf-stat.ps.page-faults
 4.482e+12           +59.1%   7.13e+12 ±  2%  perf-stat.total.instructions
     32257 ±  7%     -25.0%      24200 ±  2%  softirqs.CPU0.SCHED
     31089 ±  4%     -27.8%      22458 ±  2%  softirqs.CPU1.SCHED
     29531 ±  7%     -27.2%      21499 ±  5%  softirqs.CPU10.SCHED
     30577 ±  4%     -30.7%      21184 ±  2%  softirqs.CPU11.SCHED
     29076 ±  8%     -26.9%      21241 ±  3%  softirqs.CPU12.SCHED
     30643 ±  5%     -30.1%      21425 ±  2%  softirqs.CPU13.SCHED
     29796 ±  6%     -27.6%      21558 ±  2%  softirqs.CPU16.SCHED
     30077 ±  4%     -28.8%      21411 ±  3%  softirqs.CPU17.SCHED
     31036 ±  5%     -31.7%      21196 ±  4%  softirqs.CPU18.SCHED
     30336 ±  5%     -28.9%      21576        softirqs.CPU19.SCHED
     30159 ±  6%     -28.2%      21665 ±  2%  softirqs.CPU2.SCHED
     30306 ± 10%     -29.9%      21255 ±  3%  softirqs.CPU20.SCHED
     30154 ±  6%     -29.2%      21360        softirqs.CPU21.SCHED
     30337 ±  6%     -26.8%      22195 ±  2%  softirqs.CPU22.SCHED
     30520 ±  5%     -29.7%      21458 ±  3%  softirqs.CPU23.SCHED
     30980 ±  5%     -30.8%      21429 ±  4%  softirqs.CPU24.SCHED
     31106 ±  5%     -30.7%      21565 ±  4%  softirqs.CPU25.SCHED
     31004 ±  6%     -29.9%      21749 ±  2%  softirqs.CPU26.SCHED
     31567 ±  3%     -31.4%      21643 ±  3%  softirqs.CPU27.SCHED
     31377 ±  2%     -32.7%      21110 ±  3%  softirqs.CPU28.SCHED
     30984 ±  5%     -31.5%      21211 ±  3%  softirqs.CPU29.SCHED
     30096 ±  7%     -28.8%      21427 ±  3%  softirqs.CPU3.SCHED
     33370            -9.5%      30210        softirqs.CPU30.RCU
     30860 ±  8%     -31.8%      21041        softirqs.CPU30.SCHED
     31341 ±  4%     -33.4%      20885 ±  2%  softirqs.CPU31.SCHED
     32988 ±  2%      -9.0%      30012        softirqs.CPU32.RCU
     30660 ±  6%     -30.2%      21394 ±  3%  softirqs.CPU32.SCHED
     32983 ±  2%      -8.7%      30103        softirqs.CPU33.RCU
     30899 ±  4%     -31.8%      21075        softirqs.CPU33.SCHED
     33000 ±  2%      -8.7%      30143        softirqs.CPU34.RCU
     30705 ±  6%     -31.1%      21168        softirqs.CPU34.SCHED
     33272 ±  2%      -9.5%      30096        softirqs.CPU35.RCU
     32786 ±  3%      -8.8%      29914        softirqs.CPU36.RCU
     29637 ±  8%     -27.6%      21444 ±  2%  softirqs.CPU36.SCHED
     33128            -9.3%      30041        softirqs.CPU37.RCU
     30495 ±  6%     -29.1%      21635 ±  4%  softirqs.CPU37.SCHED
     33546 ±  5%     -10.9%      29901        softirqs.CPU38.RCU
     32487 ±  3%      -8.0%      29883        softirqs.CPU39.RCU
     30262 ±  7%     -29.7%      21273 ±  2%  softirqs.CPU39.SCHED
     30376 ±  6%     -28.0%      21860 ±  4%  softirqs.CPU4.SCHED
     32849 ±  2%      -8.8%      29961        softirqs.CPU40.RCU
     30683 ±  7%     -30.2%      21428 ±  3%  softirqs.CPU40.SCHED
     32715            -9.1%      29739        softirqs.CPU41.RCU
     31132 ±  4%     -31.4%      21350 ±  3%  softirqs.CPU41.SCHED
     31235 ±  6%     -31.8%      21307 ±  3%  softirqs.CPU42.SCHED
     33200 ±  2%      -9.7%      29975        softirqs.CPU43.RCU
     29816 ±  7%     -29.0%      21167 ±  3%  softirqs.CPU43.SCHED
     28088 ±  9%     -26.7%      20593 ±  2%  softirqs.CPU44.SCHED
     29448 ±  5%     -29.5%      20755 ±  2%  softirqs.CPU45.SCHED
     29935 ±  4%     -30.7%      20735 ±  2%  softirqs.CPU47.SCHED
     30298 ±  7%     -29.9%      21244 ±  2%  softirqs.CPU48.SCHED
     30101 ±  7%     -29.3%      21272 ±  2%  softirqs.CPU49.SCHED
     30905 ±  4%     -30.4%      21509 ±  2%  softirqs.CPU5.SCHED
     28795 ±  5%     -27.2%      20973 ±  2%  softirqs.CPU50.SCHED
     29020 ±  7%     -28.0%      20885 ±  2%  softirqs.CPU51.SCHED
     29807 ±  6%     -29.4%      21039 ±  2%  softirqs.CPU52.SCHED
     29641 ±  4%     -28.2%      21282 ±  2%  softirqs.CPU53.SCHED
     29488 ±  6%     -28.6%      21066 ±  2%  softirqs.CPU54.SCHED
     30602 ±  3%     -30.8%      21171 ±  2%  softirqs.CPU55.SCHED
     30095 ±  6%     -30.5%      20910 ±  4%  softirqs.CPU56.SCHED
     30033 ±  5%     -29.3%      21238 ±  3%  softirqs.CPU57.SCHED
     29260 ±  7%     -28.3%      20975 ±  3%  softirqs.CPU58.SCHED
     28387 ±  8%     -24.9%      21331        softirqs.CPU59.SCHED
     30548 ±  4%     -30.1%      21342 ±  3%  softirqs.CPU6.SCHED
     29871 ±  5%     -29.3%      21120 ±  3%  softirqs.CPU60.SCHED
     29237 ±  6%     -28.6%      20879 ±  3%  softirqs.CPU61.SCHED
     29240 ±  7%     -27.5%      21185 ±  3%  softirqs.CPU62.SCHED
     29563 ±  7%     -28.5%      21141 ±  3%  softirqs.CPU63.SCHED
     29578 ±  7%     -28.0%      21304 ±  2%  softirqs.CPU64.SCHED
     29657 ±  6%     -28.4%      21220 ±  3%  softirqs.CPU65.SCHED
     31249 ±  5%     -31.6%      21362 ±  3%  softirqs.CPU66.SCHED
     29401 ±  8%     -28.2%      21105 ±  3%  softirqs.CPU67.SCHED
     30709 ±  8%     -30.1%      21467 ±  3%  softirqs.CPU68.SCHED
     30552 ±  7%     -31.3%      20979 ±  3%  softirqs.CPU69.SCHED
     29593 ±  6%     -28.3%      21221 ±  4%  softirqs.CPU7.SCHED
     30623 ±  5%     -29.8%      21489 ±  3%  softirqs.CPU70.SCHED
     30580 ±  6%     -29.8%      21458 ±  3%  softirqs.CPU71.SCHED
     29326 ±  9%     -28.7%      20897 ±  3%  softirqs.CPU72.SCHED
     30369 ±  8%     -31.4%      20830 ±  3%  softirqs.CPU73.SCHED
     30481 ±  6%     -30.2%      21280 ±  3%  softirqs.CPU74.SCHED
     30655 ±  5%     -31.5%      21001 ±  2%  softirqs.CPU75.SCHED
     30996 ±  2%      -9.3%      28123        softirqs.CPU76.RCU
     30060 ±  9%     -29.8%      21090 ±  4%  softirqs.CPU76.SCHED
     30671 ±  6%     -30.7%      21241 ±  3%  softirqs.CPU77.SCHED
     30675 ±  5%     -31.1%      21128 ±  3%  softirqs.CPU78.SCHED
     30741 ±  6%     -30.3%      21421 ±  3%  softirqs.CPU79.SCHED
     29887 ±  6%     -28.5%      21362 ±  5%  softirqs.CPU8.SCHED
     30991 ±  2%      -8.4%      28376        softirqs.CPU80.RCU
     29958 ±  9%     -29.8%      21030        softirqs.CPU80.SCHED
     31177 ±  2%     -10.2%      28000 ±  4%  softirqs.CPU81.RCU
     31046 ±  7%     -30.6%      21547 ±  2%  softirqs.CPU81.SCHED
     30746 ±  5%     -31.1%      21175 ±  3%  softirqs.CPU82.SCHED
     30702 ±  6%     -30.5%      21346 ±  3%  softirqs.CPU83.SCHED
     30884 ±  4%     -31.3%      21218 ±  3%  softirqs.CPU84.SCHED
     30868            -8.2%      28334 ±  3%  softirqs.CPU85.RCU
     29412 ±  9%     -28.0%      21177 ±  3%  softirqs.CPU85.SCHED
     30337 ±  7%     -29.8%      21290 ±  5%  softirqs.CPU86.SCHED
     30525 ±  6%     -29.9%      21406 ±  3%  softirqs.CPU87.SCHED
     30317 ±  4%     -29.9%      21254 ±  2%  softirqs.CPU9.SCHED
   2649123           -29.2%    1874677 ±  2%  softirqs.SCHED
     43467           -11.9%      38299        softirqs.TIMER
      0.00 ± 14%   +1217.2%       0.06 ± 77%  perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.00 ±103%  +31150.0%       0.42 ±138%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.00          +141.7%       0.00 ± 32%  perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.00 ± 27%  +11266.7%       0.34 ± 52%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
      0.00 ±223%  +91425.0%       0.61 ± 95%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_createname
      0.00 ±141%  +20187.5%       0.27 ±140%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_removename
      0.00 ± 89%   +2996.0%       0.13 ±145%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_read.xfs_log_commit_cil.__xfs_trans_commit
      0.00 ± 39%   +6771.4%       0.24 ± 85%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.vfs_unlink.do_unlinkat
      0.00 ±223%  +17240.0%       0.14 ±210%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_create
      0.00 ±141%    +870.0%       0.02 ± 87%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_iget
      0.00 ±146%  +32676.9%       0.71 ± 99%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
      0.00 ± 21%     -68.4%       0.00 ± 57%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_reclaim_inodes_ag
      0.00 ± 69%   +7960.9%       0.31 ± 47%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
      0.04 ±131%   +1051.0%       0.50 ± 21%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
      0.00 ± 88%  +24285.7%       0.57 ± 86%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
      0.00 ±142%   +8377.8%       0.13 ±173%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file
      0.00 ± 47%   +6723.5%       0.19 ±147%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
      0.00 ± 58%   +4442.1%       0.14 ± 97%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
      0.01 ± 90%   +8454.3%       0.50 ± 90%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_alloc.xfs_iget
      0.01 ±131%   +5169.8%       0.55 ± 74%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      0.00 ±223%  +31900.0%       0.27 ±212%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      0.00 ±147%    +412.5%       0.01 ± 49%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.mnt_want_write.path_openat.do_filp_open
      0.01 ± 25%     -70.5%       0.00 ± 67%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ±116%   +5442.9%       0.32 ± 53%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_create
      0.01 ±172%   +3109.5%       0.34 ± 48%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
      0.00 ± 28%   +9663.2%       0.31 ±144%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
      0.01 ± 11%     +37.8%       0.01 ±  5%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 23%    +115.6%       0.01 ± 24%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.00 ±  8%     +68.0%       0.01 ± 27%  perf-sched.sch_delay.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
      0.01 ± 44%     +73.9%       0.01 ± 28%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 40%    +977.3%       0.04 ±153%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.01 ±  3%     +28.8%       0.01 ±  5%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.00 ± 11%     +52.6%       0.00 ± 14%  perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
      0.04 ±157%  +17335.3%       7.24 ± 70%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.00 ±102%  +1.2e+05%       3.09 ±141%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.01 ± 68%  +65418.1%       7.86 ± 50%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 15%    +118.2%       0.03 ± 26%  perf-sched.sch_delay.max.ms.md_flush_request.raid0_make_request.md_handle_request.md_submit_bio
      0.04 ±100%   +4478.8%       1.91 ±138%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ± 94%  +69869.7%      10.38 ± 43%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
      0.00 ±223%  +5.5e+05%       3.66 ± 96%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_createname
      0.00 ±145%  +2.5e+05%       4.09 ±141%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_removename
      0.01 ± 62%  +42319.6%       3.25 ±146%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_read.xfs_log_commit_cil.__xfs_trans_commit
      0.01 ± 39%  +65216.2%       4.03 ± 83%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.vfs_unlink.do_unlinkat
      0.00 ±223%  +1.3e+05%       1.08 ±221%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_create
      0.00 ±141%  +11920.0%       0.20 ±128%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_iget
      0.00 ±146%  +2.1e+05%       4.49 ±103%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
      0.02 ±106%  +50756.3%      11.44 ± 26%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
      1.16 ±141%   +1506.4%      18.64 ± 51%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
      0.00 ± 76%  +1.7e+05%       4.97 ± 85%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
      0.00 ±101%  +92057.1%       2.15 ±172%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.terminate_walk.path_openat
      0.00 ±142%  +36855.6%       0.55 ±199%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file
      0.00 ± 14%  +59448.3%       2.88 ±161%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
      0.01 ± 50%  +96897.8%       7.27 ±106%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
      0.01 ±133%  +50795.3%       5.43 ± 82%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_alloc.xfs_iget
      0.00 ±105%  +61522.2%       2.77 ±157%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_item_init.xfs_trans_ijoin
      0.04 ±187%  +18313.4%       7.80 ± 70%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      0.00 ±223%  +1.5e+05%       1.29 ±220%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      0.01 ± 16%     +60.8%       0.02 ± 14%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.mempool_alloc.bio_alloc_bioset._xfs_buf_ioapply
      0.00 ±142%    +518.2%       0.01 ± 24%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.mnt_want_write.path_openat.do_filp_open
      0.06 ±  9%    +120.0%       0.14 ± 40%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.process_one_work.worker_thread.kthread
      0.00 ± 11%  +48352.6%       1.53 ±220%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      4.39 ±  4%     +36.8%       6.01 ± 15%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      0.80 ±213%   +1526.4%      12.98 ± 19%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_create
      0.07 ±192%    +568.4%       0.46 ± 53%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
      1.48 ±221%    +828.4%      13.75 ± 17%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
      0.01 ± 27%  +75997.6%       5.33 ±140%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
      0.01 ± 12%    +153.7%       0.02 ± 55%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
      0.01 ± 19%     +91.1%       0.03 ± 29%  perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.05 ±180%   +2739.9%       1.48 ±158%  perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      3.14 ± 14%     +87.3%       5.88 ± 10%  perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
     50.73           -51.7%      24.48 ±  6%  perf-sched.total_wait_and_delay.average.ms
    624152          +111.0%    1316952 ±  6%  perf-sched.total_wait_and_delay.count.ms
     50.71           -51.8%      24.46 ±  6%  perf-sched.total_wait_time.average.ms
     30.98 ±102%   +1091.8%     369.21 ± 20%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
     54.93 ±152%    +644.4%     408.86 ± 45%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.vfs_unlink.do_unlinkat
     73.54 ±141%    +554.9%     481.61 ± 25%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
    184.26 ± 37%     -73.5%      48.86 ± 49%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_reclaim_inodes_ag
     60.96 ±106%    +310.8%     250.39 ± 21%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
     56.59 ± 74%    +318.1%     236.60 ± 22%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
     48.88 ±165%    +405.0%     246.85 ± 35%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
     13.35 ±223%    +798.3%     119.96 ± 39%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
     72.64 ±142%    +294.2%     286.32 ± 34%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_alloc.xfs_iget
     91.56 ± 68%    +282.2%     349.96 ± 18%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      0.34 ± 25%    +130.6%       0.79 ± 34%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.process_one_work.worker_thread.kthread
      1693 ± 21%     -94.3%      95.94 ± 98%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork
     73.64 ±  2%     -50.7%      36.31 ± 18%  perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     81.86 ±  2%     -35.3%      52.97 ± 10%  perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
     63.89 ±  2%     -50.2%      31.81 ±  8%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     40.92 ±  3%     -66.4%      13.76 ±  6%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.__down.down.xfs_buf_lock
      5703           +35.4%       7721 ±  5%  perf-sched.wait_and_delay.count.md_flush_request.raid0_make_request.md_handle_request.md_submit_bio
     10.83 ±113%   +1038.5%     123.33 ± 35%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
      1.33 ± 70%   +1525.0%      21.67 ± 46%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_iunlink
      0.50 ±152%   +5800.0%      29.50 ± 32%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.down_write.vfs_unlink.do_unlinkat
      0.33 ±141%   +1650.0%       5.83 ± 53%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
     11.17 ±100%   +2282.1%     266.00 ± 30%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
     15.83 ± 83%   +1971.6%     328.00 ± 22%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
      0.67 ±165%   +1300.0%       9.33 ± 51%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
      0.67 ±223%  +12175.0%      81.83 ± 18%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
      0.83 ±175%   +1700.0%      15.00 ± 37%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_alloc.xfs_iget
      1.00 ±100%    +383.3%       4.83 ± 36%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_item_init.xfs_trans_ijoin
      3.00 ± 69%    +811.1%      27.33 ± 27%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      5586 ±  3%     +47.6%       8242 ±  7%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.process_one_work.worker_thread.kthread
      5763 ±  3%     +46.1%       8420 ±  7%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      3.00 ±120%    +583.3%      20.50 ± 34%  perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
    114066           +96.6%     224275 ±  9%  perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     79541          +120.0%     174998 ±  8%  perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
     23.83 ± 73%    +785.3%     211.00 ± 12%  perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
    277.83 ±  2%     +94.0%     539.00 ±  6%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    350693          +130.8%     809570 ±  7%  perf-sched.wait_and_delay.count.schedule_timeout.__down.down.xfs_buf_lock
      1328 ±  3%     +34.4%       1785 ±  8%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     36430           +14.5%      41695 ±  5%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    125.97 ±100%   +1046.5%       1444 ± 30%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
    175.07 ± 72%    +597.0%       1220 ± 47%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_iunlink
     73.18 ±141%   +1792.9%       1385 ± 32%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.vfs_unlink.do_unlinkat
     73.54 ±141%   +1599.9%       1250 ± 38%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
    152.32 ±100%    +852.2%       1450 ± 31%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
    187.67 ± 72%    +675.3%       1454 ± 30%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
     73.37 ±141%   +1253.4%     993.03 ± 37%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
     50.23 ±223%   +2596.4%       1354 ± 37%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
     92.92 ±143%    +836.1%     869.82 ± 22%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_alloc.xfs_iget
    196.84 ± 45%    +586.9%       1352 ± 28%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      4730 ± 14%     -77.8%       1049 ± 80%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork
    116.73 ±100%    +812.9%       1065 ± 32%  perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
    269.27 ± 16%    +455.7%       1496 ± 28%  perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
    272.68 ± 17%    +450.1%       1500 ± 28%  perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    312.66 ± 97%   +1205.5%       4081 ± 65%  perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
    270.57 ± 16%   +1554.1%       4475 ± 48%  perf-sched.wait_and_delay.max.ms.schedule_timeout.__down.down.xfs_buf_lock
      1.45 ±103%   +1055.5%      16.78 ± 33%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.12 ±223%  +2.1e+05%     265.14 ±101%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     43.51 ± 47%    +747.7%     368.87 ± 20%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
      0.01 ±145%  +5.7e+06%     335.30 ± 51%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_createname
      0.05 ±201%  +5.1e+05%     256.34 ± 35%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_removename
     55.31 ±151%    +638.8%     408.62 ± 45%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.vfs_unlink.do_unlinkat
     35.76 ±223%    +706.0%     288.24 ±104%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_create
      0.00 ±150%    +2e+06%      57.23 ±104%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_iget
     73.53 ±141%    +554.0%     480.90 ± 25%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
    184.25 ± 37%     -71.7%      52.07 ± 34%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_reclaim_inodes_ag
     71.60 ± 77%    +249.3%     250.08 ± 21%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
     56.55 ± 74%    +317.5%     236.09 ± 22%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
     48.88 ±165%    +403.9%     246.28 ± 35%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
      0.00 ±105%  +5.5e+06%     147.15 ±114%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file
      0.19 ±215%  +1.8e+05%     355.78 ± 48%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
     18.05 ±163%    +563.7%     119.82 ± 39%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
     72.64 ±142%    +293.4%     285.82 ± 34%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_alloc.xfs_iget
      0.02 ± 74%    +295.3%       0.07 ± 67%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_ifree
     91.55 ± 68%    +281.7%     349.40 ± 18%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      0.00 ±174%  +4.1e+06%     199.11 ± 66%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      0.01 ±143%  +3.7e+06%     263.71 ± 61%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mnt_want_write.do_unlinkat.do_syscall_64
     17.72 ±223%   +2215.6%     410.38 ± 67%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mnt_want_write.path_openat.do_filp_open
      0.31 ± 28%    +142.3%       0.75 ± 35%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.process_one_work.worker_thread.kthread
      1693 ± 21%     -94.2%      97.73 ± 94%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork
     33.33 ± 11%    +329.8%     143.27 ± 32%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_create
     35.37 ± 26%    +391.3%     173.78 ± 29%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
     73.63 ±  2%     -50.7%      36.28 ± 18%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     81.84 ±  2%     -35.3%      52.92 ± 10%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
     63.89 ±  2%     -50.2%      31.80 ±  8%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     40.91 ±  3%     -66.4%      13.75 ±  6%  perf-sched.wait_time.avg.ms.schedule_timeout.__down.down.xfs_buf_lock
     28.15           -42.6%      16.15 ± 15%  perf-sched.wait_time.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
    177.70 ±207%    +463.0%       1000        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     53.27 ±223%    +743.9%     449.56 ±101%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.12 ±223%  +4.4e+05%     541.66 ±101%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      3.46 ±135%  +29325.9%       1019 ± 47%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    235.77 ±  9%    +512.6%       1444 ± 30%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.kmem_alloc_large
      0.01 ±181%  +9.4e+06%       1066 ± 46%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_createname
      0.10 ±209%  +1.3e+06%       1251 ± 33%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_dir_removename
    174.25 ± 72%    +599.4%       1218 ± 47%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.kmem_alloc.xfs_iunlink
    149.37 ± 71%    +526.9%     936.48 ± 54%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read.xfs_log_commit_cil.__xfs_trans_commit
     74.70 ±137%   +1754.5%       1385 ± 32%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.vfs_unlink.do_unlinkat
     35.76 ±223%   +1906.4%     717.51 ±100%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_create
      0.00 ±150%  +1.8e+07%     496.79 ± 94%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_iget
     73.53 ±141%   +1598.8%       1249 ± 38%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes
      0.00 ±152%    +5e+06%     150.62 ±168%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
    225.55 ± 48%    +543.1%       1450 ± 31%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
    187.67 ± 72%    +674.1%       1452 ± 29%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
     73.38 ±141%   +1252.0%     992.04 ± 37%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.path_put.vfs_statx
     72.44 ±141%    +383.3%     350.13 ±101%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.terminate_walk.path_openat
      0.00 ±105%  +1.3e+07%     356.89 ± 91%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file
      0.94 ±221%  +1.3e+05%       1196 ± 44%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
    101.80 ±141%   +1228.4%       1352 ± 37%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor
     92.92 ±143%    +835.9%     869.67 ± 22%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_inode_alloc.xfs_iget
      0.02 ± 84%   +1780.0%       0.39 ± 98%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_ifree
    196.84 ± 45%    +586.9%       1352 ± 28%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate
      0.01 ±190%  +6.5e+06%     499.56 ± 84%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve
      0.01 ±143%  +1.4e+07%       1009 ± 40%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mnt_want_write.do_unlinkat.do_syscall_64
     35.44 ±223%   +2219.6%     822.09 ± 59%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mnt_want_write.path_openat.do_filp_open
      4730 ± 14%     -77.8%       1052 ± 80%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork
    265.12 ± 16%    +449.0%       1455 ± 30%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_create
      0.21 ±149%   +1313.5%       2.93 ±165%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
    265.50 ± 16%    +428.2%       1402 ± 30%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
    190.00 ± 45%    +460.1%       1064 ± 32%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
    268.97 ± 16%    +455.1%       1492 ± 28%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
    270.59 ± 16%    +451.8%       1493 ± 28%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    399.37 ± 58%    +922.1%       4081 ± 65%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
    270.56 ± 16%   +1554.1%       4475 ± 48%  perf-sched.wait_time.max.ms.schedule_timeout.__down.down.xfs_buf_lock
     11.14 ±  5%     +27.6%      14.22 ±  6%  perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xfsaild_push.xfsaild.kthread
     29.23 ±  4%     -21.4        7.87 ± 19%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     28.91 ±  4%     -21.1        7.78 ± 18%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     28.91 ±  4%     -21.1        7.78 ± 19%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     28.90 ±  4%     -21.1        7.78 ± 18%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     28.20 ±  4%     -20.8        7.43 ± 19%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     28.09 ±  4%     -20.7        7.41 ± 19%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     26.48 ±  5%     -19.4        7.06 ± 19%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      6.54 ±  3%      -3.4        3.10 ±  4%  perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      4.03 ±  2%      -1.6        2.42 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.36 ±  3%      -1.1        2.26 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      3.05 ±  4%      -0.6        2.47 ± 11%  perf-profile.calltrace.cycles-pp.ret_from_fork
      3.05 ±  4%      -0.6        2.47 ± 11%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.00            +0.6        0.63 ±  6%  perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode
     37.10 ±  3%      +8.4       45.46 ±  2%  perf-profile.calltrace.cycles-pp.creat64
     37.02 ±  3%      +8.4       45.43 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat64
     36.99 ±  3%      +8.4       45.41 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
     36.98 ±  3%      +8.4       45.41 ±  2%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
     36.97 ±  3%      +8.4       45.41 ±  2%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
     36.82 ±  3%      +8.5       45.34 ±  2%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     36.82 ±  3%      +8.5       45.34 ±  2%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
     27.43 ±  4%     +13.4       40.80 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2.do_sys_open
     29.07           +14.4       43.52        perf-profile.calltrace.cycles-pp.unlink
     28.92           +14.5       43.46        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
     28.85           +14.6       43.42        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
     28.78           +14.6       43.40        perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
     23.75 ±  4%     +14.6       38.40 ±  3%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
     21.48 ±  2%     +18.4       39.90 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
     17.19 ±  3%     +20.1       37.34 ±  2%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.23 ±  4%     -21.4        7.87 ± 19%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     29.23 ±  4%     -21.4        7.87 ± 19%  perf-profile.children.cycles-pp.cpu_startup_entry
     29.23 ±  4%     -21.4        7.87 ± 19%  perf-profile.children.cycles-pp.do_idle
     28.91 ±  4%     -21.1        7.78 ± 19%  perf-profile.children.cycles-pp.start_secondary
     28.53 ±  4%     -21.0        7.51 ± 19%  perf-profile.children.cycles-pp.cpuidle_enter
     28.52 ±  4%     -21.0        7.51 ± 19%  perf-profile.children.cycles-pp.cpuidle_enter_state
     26.78 ±  5%     -19.6        7.15 ± 19%  perf-profile.children.cycles-pp.intel_idle
      7.86 ± 15%      -4.3        3.57 ± 16%  perf-profile.children.cycles-pp.xfs_create
      6.54 ±  3%      -3.4        3.10 ±  4%  perf-profile.children.cycles-pp.vfs_unlink
      6.39 ±  3%      -3.4        3.04 ±  4%  perf-profile.children.cycles-pp.xfs_remove
      5.92 ±  2%      -2.7        3.18 ±  7%  perf-profile.children.cycles-pp.__xfs_trans_commit
      5.89 ±  2%      -2.7        3.18 ±  9%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      7.40 ±  2%      -2.7        4.68 ±  2%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      3.71 ±  4%      -2.3        1.45 ±  4%  perf-profile.children.cycles-pp.xfs_dir3_data_check
      3.68 ±  4%      -2.2        1.43 ±  4%  perf-profile.children.cycles-pp.__xfs_dir3_data_check
      2.37 ± 19%      -1.8        0.62 ± 28%  perf-profile.children.cycles-pp.xfs_dir2_block_removename
      2.95 ±  3%      -1.3        1.61 ±  6%  perf-profile.children.cycles-pp.xlog_cil_insert_items
      2.48 ±  2%      -1.2        1.29 ±  8%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      1.60 ±  2%      -1.2        0.45 ± 10%  perf-profile.children.cycles-pp.xfs_dir2_block_addname
      1.99 ±  4%      -1.1        0.88 ±  9%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      1.83 ±  5%      -1.0        0.82 ±  9%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      1.37 ±  7%      -1.0        0.40 ± 11%  perf-profile.children.cycles-pp.xfs_dir2_block_lookup_int
      1.78 ±  2%      -0.8        0.96 ±  6%  perf-profile.children.cycles-pp.xfs_buf_item_format
      1.49 ±  2%      -0.7        0.80 ±  9%  perf-profile.children.cycles-pp.xfs_buf_read_map
      1.55 ±  7%      -0.7        0.87 ± 20%  perf-profile.children.cycles-pp.xfs_buf_get_map
      1.27 ±  2%      -0.6        0.62 ±  6%  perf-profile.children.cycles-pp.memcpy_erms
      1.34            -0.6        0.72 ± 10%  perf-profile.children.cycles-pp.xfs_buf_find
      1.19 ±  4%      -0.6        0.60 ±  9%  perf-profile.children.cycles-pp.xfs_btree_lookup
      1.22 ±  5%      -0.6        0.62 ± 10%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      1.20 ±  5%      -0.6        0.62 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
      3.05 ±  4%      -0.6        2.47 ± 11%  perf-profile.children.cycles-pp.ret_from_fork
      3.05 ±  4%      -0.6        2.47 ± 11%  perf-profile.children.cycles-pp.kthread
      1.13 ±  3%      -0.5        0.60 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
      0.84 ±  3%      -0.5        0.31 ±  4%  perf-profile.children.cycles-pp.xfs_init_new_inode
      1.40 ±  8%      -0.5        0.88 ± 13%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      0.64 ±  5%      -0.4        0.25 ±  4%  perf-profile.children.cycles-pp.xfs_dialloc_ag_update_inobt
      0.54 ±  9%      -0.4        0.16 ± 13%  perf-profile.children.cycles-pp.ktime_get
      0.64 ±  5%      -0.4        0.28 ±  5%  perf-profile.children.cycles-pp.xfs_dialloc_select_ag
      0.68 ±  9%      -0.3        0.33 ±  8%  perf-profile.children.cycles-pp.xfs_iunlink
      0.57 ±  5%      -0.3        0.25 ±  3%  perf-profile.children.cycles-pp.__xstat64
      0.47 ±  2%      -0.3        0.17 ±  8%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.41 ±  9%      -0.3        0.11 ±  8%  perf-profile.children.cycles-pp.clockevents_program_event
      0.51 ±  4%      -0.3        0.22 ±  2%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.43            -0.3        0.15 ± 10%  perf-profile.children.cycles-pp.xfs_iget
      0.40 ±  8%      -0.3        0.14 ± 12%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.41 ±  6%      -0.3        0.15 ± 11%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.54 ± 19%      -0.3        0.27 ± 11%  perf-profile.children.cycles-pp.xfs_verify_ino
      0.47 ±  4%      -0.3        0.21 ±  4%  perf-profile.children.cycles-pp.vfs_statx
      0.45 ± 11%      -0.2        0.20 ±  5%  perf-profile.children.cycles-pp.xfs_trans_log_inode
      0.43 ±  3%      -0.2        0.19 ±  3%  perf-profile.children.cycles-pp.link_path_walk
      0.33 ± 36%      -0.2        0.09 ± 53%  perf-profile.children.cycles-pp.start_kernel
      0.48 ±  5%      -0.2        0.25 ±  5%  perf-profile.children.cycles-pp.__percpu_counter_compare
      0.57 ±  3%      -0.2        0.34 ± 15%  perf-profile.children.cycles-pp.__schedule
      0.46 ±  5%      -0.2        0.23 ±  5%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.63            -0.2        0.40 ± 14%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.46 ±  4%      -0.2        0.23 ±  6%  perf-profile.children.cycles-pp.xfs_imap_to_bp
      0.51 ±  2%      -0.2        0.29 ±  7%  perf-profile.children.cycles-pp.xfs_buf_item_size
      0.40 ±  6%      -0.2        0.19 ±  8%  perf-profile.children.cycles-pp.xfs_buf_item_init
      0.49 ± 14%      -0.2        0.28 ± 21%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      0.44 ±  4%      -0.2        0.25 ± 15%  perf-profile.children.cycles-pp.schedule
      0.34 ±  5%      -0.2        0.15 ±  6%  perf-profile.children.cycles-pp.xfsaild
      0.34 ±  5%      -0.2        0.15 ±  6%  perf-profile.children.cycles-pp.xfsaild_push
      0.42 ±  6%      -0.2        0.24 ±  9%  perf-profile.children.cycles-pp.xfs_inode_item_format
      0.30 ±  6%      -0.2        0.13 ±  2%  perf-profile.children.cycles-pp.filename_lookup
      0.36 ±  4%      -0.2        0.19 ±  8%  perf-profile.children.cycles-pp.xfs_buf_rele
      0.29 ±  7%      -0.2        0.12 ±  4%  perf-profile.children.cycles-pp.path_lookupat
      0.25 ±  7%      -0.2        0.08 ± 17%  perf-profile.children.cycles-pp.menu_select
      0.35 ±  3%      -0.2        0.19 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.35 ±  5%      -0.2        0.19 ± 20%  perf-profile.children.cycles-pp.up
      0.22 ±  3%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.__kmalloc
      0.25 ±  5%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.getname_flags
      0.49            -0.1        0.34 ± 16%  perf-profile.children.cycles-pp.tick_sched_timer
      0.33 ±  8%      -0.1        0.19 ± 19%  perf-profile.children.cycles-pp.xfs_dir2_node_addname
      0.28 ± 12%      -0.1        0.14 ± 10%  perf-profile.children.cycles-pp.xlog_cil_push_work
      0.23 ±  7%      -0.1        0.10 ±  6%  perf-profile.children.cycles-pp.__close
      0.23 ±  5%      -0.1        0.10 ±  6%  perf-profile.children.cycles-pp.inode_permission
      0.24 ±  5%      -0.1        0.11 ±  6%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.20 ±  6%      -0.1        0.08 ± 12%  perf-profile.children.cycles-pp.strncpy_from_user
      0.42 ±  3%      -0.1        0.29 ± 19%  perf-profile.children.cycles-pp.tick_sched_handle
      0.41 ±  4%      -0.1        0.28 ± 13%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.23 ±  7%      -0.1        0.10 ± 10%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.21 ±  9%      -0.1        0.09 ± 11%  perf-profile.children.cycles-pp.rcu_core
      0.27 ± 12%      -0.1        0.15 ± 18%  perf-profile.children.cycles-pp.xlog_cil_committed
      0.21 ±  9%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.xfs_trans_log_buf
      0.23 ± 14%      -0.1        0.11 ± 10%  perf-profile.children.cycles-pp.xlog_write
      0.22 ±  7%      -0.1        0.11 ± 10%  perf-profile.children.cycles-pp.xfs_inode_item_push
      0.19 ±  3%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.walk_component
      0.27 ±  3%      -0.1        0.16 ± 20%  perf-profile.children.cycles-pp.__down
      0.27 ±  3%      -0.1        0.17 ± 20%  perf-profile.children.cycles-pp.down
      0.19 ±  8%      -0.1        0.08 ± 13%  perf-profile.children.cycles-pp.load_balance
      0.24 ±  3%      -0.1        0.14 ± 21%  perf-profile.children.cycles-pp.schedule_timeout
      0.18 ±  4%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.destroy_inode
      0.15 ±  4%      -0.1        0.06 ±  8%  perf-profile.children.cycles-pp.kmem_alloc_large
      0.22 ±  5%      -0.1        0.12 ±  4%  perf-profile.children.cycles-pp.xfs_buf_offset
      0.18 ±  5%      -0.1        0.08 ±  4%  perf-profile.children.cycles-pp.filename_parentat
      0.23 ±  5%      -0.1        0.13 ±  8%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.17 ±  7%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.path_parentat
      0.21 ±  9%      -0.1        0.12 ± 10%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.16 ±  5%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.alloc_empty_file
      0.19 ±  8%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.kmem_cache_free
      0.11 ±  6%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.dsearch
      0.15 ±  7%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.__alloc_file
      0.12 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.user_path_at_empty
      0.16 ± 10%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.rcu_do_batch
      0.11 ± 17%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.xfs_buf_delwri_submit_buffers
      0.15 ±  8%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.do_open
      0.15 ±  8%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.find_busiest_group
      0.11 ±  6%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.kfree
      0.13 ±  8%      -0.1        0.05 ± 45%  perf-profile.children.cycles-pp.try_to_unlazy
      0.15 ±  9%      -0.1        0.07 ± 13%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.11 ± 15%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.submit_bio
      0.14 ±  6%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.dput
      0.32 ±  5%      -0.1        0.24 ± 18%  perf-profile.children.cycles-pp.try_to_wake_up
      0.15 ±  5%      -0.1        0.07 ±  9%  perf-profile.children.cycles-pp.down_write
      0.14 ± 10%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.xfs_dialloc_ag_finobt_near
      0.22 ±  3%      -0.1        0.14 ± 21%  perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.10 ± 10%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.xfs_iunlink_insert_backref
      0.19 ±  6%      -0.1        0.11 ±  8%  perf-profile.children.cycles-pp.xfs_trans_alloc_icreate
      0.14 ± 11%      -0.1        0.07 ± 14%  perf-profile.children.cycles-pp.submit_bio_noacct
      0.15 ±  9%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.newidle_balance
      0.13 ±  7%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.lookup_fast
      0.13 ±  5%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.d_lookup
      0.20 ±  8%      -0.1        0.13 ± 24%  perf-profile.children.cycles-pp.xfs_dir2_leafn_remove
      0.12 ±  6%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__d_lookup
      0.12 ± 15%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.task_work_run
      0.10 ±  7%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_trans_brelse
      0.16 ±  3%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.xfs_dir2_node_addname_int
      0.11 ±  8%      -0.1        0.04 ± 44%  perf-profile.children.cycles-pp.do_dentry_open
      0.12 ± 12%      -0.1        0.06 ± 13%  perf-profile.children.cycles-pp.brd_do_bvec
      0.10 ± 12%      -0.1        0.04 ± 71%  perf-profile.children.cycles-pp.__slab_free
      0.16 ±  7%      -0.1        0.10 ± 18%  perf-profile.children.cycles-pp.schedule_idle
      0.09 ± 11%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.enqueue_entity
      0.11 ±  6%      -0.1        0.05 ±  7%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.09 ± 11%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.rhashtable_rehash_table
      0.12 ±  9%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.10 ± 11%      -0.1        0.04 ± 45%  perf-profile.children.cycles-pp.rht_deferred_worker
      0.10 ± 12%      -0.1        0.04 ± 45%  perf-profile.children.cycles-pp.down_read
      0.12 ±  6%      -0.1        0.07 ± 20%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.12 ±  6%      -0.1        0.07 ± 20%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.08 ± 11%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.update_rq_clock
      0.11 ±  8%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.xfs_lookup_get_search_key
      0.11 ±  9%      -0.0        0.07 ± 14%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.11 ±  7%      -0.0        0.07 ± 16%  perf-profile.children.cycles-pp.dequeue_entity
      0.10 ±  8%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.xfs_dabuf_map
      0.09 ±  8%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.__list_add_valid
      0.13 ±  5%      -0.0        0.09 ± 15%  perf-profile.children.cycles-pp.update_load_avg
      0.09 ± 10%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.xfs_buf_inode_iodone
      0.09 ±  6%      -0.0        0.06 ± 17%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.08 ±  6%      -0.0        0.05 ±  9%  perf-profile.children.cycles-pp.update_curr
      0.07 ± 17%      +0.0        0.10 ± 34%  perf-profile.children.cycles-pp.update_cfs_group
      0.06 ±  7%      +0.2        0.27 ±  8%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.00            +0.2        0.22 ± 21%  perf-profile.children.cycles-pp.xfs_dir2_leaf_addname
      0.23 ±  7%      +0.2        0.48 ±  5%  perf-profile.children.cycles-pp.xfs_difree_inobt
      0.00            +0.3        0.31 ± 20%  perf-profile.children.cycles-pp.xfs_dir2_leaf_removename
     37.12 ±  3%      +8.4       45.47 ±  2%  perf-profile.children.cycles-pp.creat64
     37.00 ±  3%      +8.4       45.42 ±  2%  perf-profile.children.cycles-pp.do_sys_open
     36.99 ±  3%      +8.4       45.42 ±  2%  perf-profile.children.cycles-pp.do_sys_openat2
     36.84 ±  3%      +8.5       45.36 ±  2%  perf-profile.children.cycles-pp.do_filp_open
     36.83 ±  3%      +8.5       45.35 ±  2%  perf-profile.children.cycles-pp.path_openat
     29.10           +14.4       43.53        perf-profile.children.cycles-pp.unlink
     28.78           +14.6       43.40        perf-profile.children.cycles-pp.do_unlinkat
     66.83 ±  2%     +22.4       89.28 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     66.57 ±  2%     +22.6       89.16 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     48.94 ±  2%     +31.8       80.71 ±  2%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
     40.96 ±  3%     +34.8       75.76 ±  2%  perf-profile.children.cycles-pp.osq_lock
     26.78 ±  5%     -19.6        7.15 ± 19%  perf-profile.self.cycles-pp.intel_idle
      7.31 ±  2%      -2.7        4.64 ±  2%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      2.60 ±  5%      -1.7        0.92 ±  7%  perf-profile.self.cycles-pp.__xfs_dir3_data_check
      1.25 ±  2%      -0.6        0.62 ±  6%  perf-profile.self.cycles-pp.memcpy_erms
      1.06 ±  3%      -0.5        0.51 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock
      0.49 ± 10%      -0.3        0.14 ± 13%  perf-profile.self.cycles-pp.ktime_get
      0.48 ±  3%      -0.2        0.25 ±  8%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.36 ±  8%      -0.2        0.14 ± 17%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.38 ±  5%      -0.2        0.18 ±  8%  perf-profile.self.cycles-pp.xfs_buf_item_init
      0.34 ±  4%      -0.2        0.18 ±  9%  perf-profile.self.cycles-pp.xfs_buf_find
      0.30 ±  5%      -0.2        0.14 ±  7%  perf-profile.self.cycles-pp.xfs_inode_item_format
      0.32 ±  4%      -0.1        0.17 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.30 ±  4%      -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.38 ±  3%      -0.1        0.25 ±  6%  perf-profile.self.cycles-pp.xfs_buf_item_format
      0.31 ±  2%      -0.1        0.18 ±  7%  perf-profile.self.cycles-pp.xlog_cil_insert_items
      0.27            -0.1        0.15 ± 16%  perf-profile.self.cycles-pp.xfs_verify_ino
      0.17 ±  7%      -0.1        0.05 ± 46%  perf-profile.self.cycles-pp.menu_select
      0.13 ±  7%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.strncpy_from_user
      0.22 ±  4%      -0.1        0.13 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.14 ± 13%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.xfs_log_ticket_ungrant
      0.11 ±  3%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.kfree
      0.14 ±  4%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.inode_permission
      0.14 ±  7%      -0.1        0.07 ± 16%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.17 ±  5%      -0.1        0.10 ±  7%  perf-profile.self.cycles-pp.xfs_buf_offset
      0.12 ±  4%      -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.down_write
      0.11 ±  8%      -0.1        0.04 ± 44%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.11 ±  7%      -0.1        0.05 ± 45%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.10 ± 15%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.__slab_free
      0.14 ±  9%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.kmem_cache_free
      0.12 ±  4%      -0.1        0.06 ±  9%  perf-profile.self.cycles-pp.xfs_btree_lookup
      0.08 ± 16%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.brd_do_bvec
      0.09 ±  7%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.down_read
      0.11 ±  6%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.xfs_buf_get_map
      0.11 ± 12%      -0.0        0.06        perf-profile.self.cycles-pp.__might_sleep
      0.09 ±  7%      -0.0        0.05 ± 45%  perf-profile.self.cycles-pp.__schedule
      0.09 ±  9%      -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.__list_add_valid
      0.07 ± 17%      +0.0        0.10 ± 34%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.xfs_btree_check_sblock
      0.00            +0.1        0.08 ±  9%  perf-profile.self.cycles-pp.xfs_btree_increment
      0.03 ±100%      +0.1        0.13 ± 28%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
     40.70 ±  3%     +34.6       75.29 ±  2%  perf-profile.self.cycles-pp.osq_lock
    477.00           -22.4%     370.00        interrupts.9:IO-APIC.9-fasteoi.acpi
    617161           +45.0%     894630 ±  2%  interrupts.CAL:Function_call_interrupts
      7114 ± 12%     +49.0%      10601 ±  8%  interrupts.CPU0.CAL:Function_call_interrupts
    476300           -22.3%     369901        interrupts.CPU0.LOC:Local_timer_interrupts
      1624 ± 31%    +168.8%       4366 ± 27%  interrupts.CPU0.NMI:Non-maskable_interrupts
      1624 ± 31%    +168.8%       4366 ± 27%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
    315.00 ± 14%    +166.6%     839.67 ± 18%  interrupts.CPU0.RES:Rescheduling_interrupts
      8.50 ± 40%   +3241.2%     284.00 ± 39%  interrupts.CPU0.TLB:TLB_shootdowns
    477.00           -22.4%     370.00        interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
      6799 ±  5%     +51.0%      10267 ±  8%  interrupts.CPU1.CAL:Function_call_interrupts
    476126           -22.3%     369920        interrupts.CPU1.LOC:Local_timer_interrupts
      1303 ± 37%    +266.5%       4778 ± 20%  interrupts.CPU1.NMI:Non-maskable_interrupts
      1303 ± 37%    +266.5%       4778 ± 20%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
    302.50 ± 26%    +144.6%     739.83 ± 21%  interrupts.CPU1.RES:Rescheduling_interrupts
      7077 ±  8%     +40.8%       9961 ±  4%  interrupts.CPU10.CAL:Function_call_interrupts
    476230           -22.3%     369915        interrupts.CPU10.LOC:Local_timer_interrupts
      1718 ± 21%    +183.3%       4867 ± 22%  interrupts.CPU10.NMI:Non-maskable_interrupts
      1718 ± 21%    +183.3%       4867 ± 22%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
    262.00 ± 21%    +165.3%     695.00 ± 23%  interrupts.CPU10.RES:Rescheduling_interrupts
      7006 ±  4%     +45.8%      10217 ±  4%  interrupts.CPU11.CAL:Function_call_interrupts
    476235           -22.3%     369967        interrupts.CPU11.LOC:Local_timer_interrupts
      1977 ±  5%    +165.4%       5247 ±  4%  interrupts.CPU11.NMI:Non-maskable_interrupts
      1977 ±  5%    +165.4%       5247 ±  4%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    305.67 ± 40%    +130.0%     703.17 ± 25%  interrupts.CPU11.RES:Rescheduling_interrupts
      7027 ±  2%     +45.8%      10243 ±  6%  interrupts.CPU12.CAL:Function_call_interrupts
    476271           -22.3%     369848        interrupts.CPU12.LOC:Local_timer_interrupts
      1731 ± 18%    +202.3%       5234 ±  4%  interrupts.CPU12.NMI:Non-maskable_interrupts
      1731 ± 18%    +202.3%       5234 ±  4%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
    254.67 ± 19%    +181.4%     716.67 ± 26%  interrupts.CPU12.RES:Rescheduling_interrupts
      7284 ±  9%     +39.0%      10125 ±  6%  interrupts.CPU13.CAL:Function_call_interrupts
    475403           -22.2%     369898        interrupts.CPU13.LOC:Local_timer_interrupts
      1490 ± 37%    +223.3%       4819 ± 22%  interrupts.CPU13.NMI:Non-maskable_interrupts
      1490 ± 37%    +223.3%       4819 ± 22%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    274.50 ± 25%    +160.0%     713.67 ± 23%  interrupts.CPU13.RES:Rescheduling_interrupts
      7113 ±  3%     +44.0%      10245 ±  4%  interrupts.CPU14.CAL:Function_call_interrupts
    476211           -22.3%     369906        interrupts.CPU14.LOC:Local_timer_interrupts
      1463 ± 32%    +228.5%       4807 ± 22%  interrupts.CPU14.NMI:Non-maskable_interrupts
      1463 ± 32%    +228.5%       4807 ± 22%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
    250.67 ± 13%    +184.3%     712.67 ± 23%  interrupts.CPU14.RES:Rescheduling_interrupts
      7353 ±  4%     +45.3%      10682 ±  6%  interrupts.CPU15.CAL:Function_call_interrupts
    476191           -22.4%     369761        interrupts.CPU15.LOC:Local_timer_interrupts
      1699 ± 21%    +157.0%       4366 ± 30%  interrupts.CPU15.NMI:Non-maskable_interrupts
      1699 ± 21%    +157.0%       4366 ± 30%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    252.50 ± 16%    +198.0%     752.50 ± 24%  interrupts.CPU15.RES:Rescheduling_interrupts
      7181 ±  4%     +42.0%      10200 ±  4%  interrupts.CPU16.CAL:Function_call_interrupts
    476219           -22.3%     369892        interrupts.CPU16.LOC:Local_timer_interrupts
      1752 ± 21%    +175.0%       4820 ± 21%  interrupts.CPU16.NMI:Non-maskable_interrupts
      1752 ± 21%    +175.0%       4820 ± 21%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    294.00 ± 16%    +146.3%     724.00 ± 21%  interrupts.CPU16.RES:Rescheduling_interrupts
      6692 ±  6%     +49.9%      10034 ±  5%  interrupts.CPU17.CAL:Function_call_interrupts
    476243           -22.3%     369868        interrupts.CPU17.LOC:Local_timer_interrupts
      1725 ± 19%    +180.7%       4842 ± 22%  interrupts.CPU17.NMI:Non-maskable_interrupts
      1725 ± 19%    +180.7%       4842 ± 22%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
    269.17 ± 16%    +167.4%     719.83 ± 26%  interrupts.CPU17.RES:Rescheduling_interrupts
    476217           -22.3%     369895        interrupts.CPU18.LOC:Local_timer_interrupts
      1884 ±  6%    +133.1%       4391 ± 30%  interrupts.CPU18.NMI:Non-maskable_interrupts
      1884 ±  6%    +133.1%       4391 ± 30%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
    274.00 ± 22%    +163.1%     721.00 ± 24%  interrupts.CPU18.RES:Rescheduling_interrupts
      6692 ±  2%     +55.5%      10406 ±  8%  interrupts.CPU19.CAL:Function_call_interrupts
    476418           -22.4%     369919        interrupts.CPU19.LOC:Local_timer_interrupts
      1756 ± 23%    +197.6%       5227 ±  4%  interrupts.CPU19.NMI:Non-maskable_interrupts
      1756 ± 23%    +197.6%       5227 ±  4%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
    256.67 ± 22%    +166.9%     685.00 ± 21%  interrupts.CPU19.RES:Rescheduling_interrupts
      6525 ± 18%     +56.8%      10233 ±  7%  interrupts.CPU2.CAL:Function_call_interrupts
    476011           -22.3%     369910        interrupts.CPU2.LOC:Local_timer_interrupts
      1419 ± 34%    +238.4%       4804 ± 20%  interrupts.CPU2.NMI:Non-maskable_interrupts
      1419 ± 34%    +238.4%       4804 ± 20%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
    255.67 ± 21%    +179.1%     713.50 ± 23%  interrupts.CPU2.RES:Rescheduling_interrupts
    476226           -22.3%     369856        interrupts.CPU20.LOC:Local_timer_interrupts
      1747 ± 21%    +174.3%       4794 ± 21%  interrupts.CPU20.NMI:Non-maskable_interrupts
      1747 ± 21%    +174.3%       4794 ± 21%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
    247.33 ± 14%    +172.1%     673.00 ± 24%  interrupts.CPU20.RES:Rescheduling_interrupts
      7261 ±  4%     +44.1%      10462 ±  8%  interrupts.CPU21.CAL:Function_call_interrupts
    476173           -22.3%     369863        interrupts.CPU21.LOC:Local_timer_interrupts
      1764 ± 21%    +172.6%       4810 ± 21%  interrupts.CPU21.NMI:Non-maskable_interrupts
      1764 ± 21%    +172.6%       4810 ± 21%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
    260.67 ±  8%    +186.2%     746.00 ± 25%  interrupts.CPU21.RES:Rescheduling_interrupts
      7017 ±  8%     +60.1%      11233 ±  3%  interrupts.CPU22.CAL:Function_call_interrupts
    476194           -22.3%     369930        interrupts.CPU22.LOC:Local_timer_interrupts
      1715 ± 21%    +179.3%       4791 ± 21%  interrupts.CPU22.NMI:Non-maskable_interrupts
      1715 ± 21%    +179.3%       4791 ± 21%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    216.33 ± 14%    +235.1%     725.00 ± 23%  interrupts.CPU22.RES:Rescheduling_interrupts
      8.33 ± 79%   +3968.0%     339.00 ± 74%  interrupts.CPU22.TLB:TLB_shootdowns
      7077 ±  5%     +40.9%       9973 ±  3%  interrupts.CPU23.CAL:Function_call_interrupts
    476237           -22.3%     369927        interrupts.CPU23.LOC:Local_timer_interrupts
      1715 ± 21%    +153.7%       4353 ± 29%  interrupts.CPU23.NMI:Non-maskable_interrupts
      1715 ± 21%    +153.7%       4353 ± 29%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    259.83 ± 45%    +155.5%     664.00 ± 23%  interrupts.CPU23.RES:Rescheduling_interrupts
      6713 ±  4%     +51.3%      10155 ±  7%  interrupts.CPU24.CAL:Function_call_interrupts
    476259           -22.3%     369838        interrupts.CPU24.LOC:Local_timer_interrupts
      1707 ± 20%    +180.9%       4795 ± 21%  interrupts.CPU24.NMI:Non-maskable_interrupts
      1707 ± 20%    +180.9%       4795 ± 21%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
    204.50 ± 21%    +230.3%     675.50 ± 22%  interrupts.CPU24.RES:Rescheduling_interrupts
      6695 ±  4%     +46.2%       9786 ±  4%  interrupts.CPU25.CAL:Function_call_interrupts
    476210           -22.3%     369882        interrupts.CPU25.LOC:Local_timer_interrupts
      1731 ± 21%    +150.9%       4343 ± 29%  interrupts.CPU25.NMI:Non-maskable_interrupts
      1731 ± 21%    +150.9%       4343 ± 29%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    213.67 ± 23%    +228.5%     701.83 ± 25%  interrupts.CPU25.RES:Rescheduling_interrupts
      6949 ±  8%     +47.9%      10279 ±  9%  interrupts.CPU26.CAL:Function_call_interrupts
    476106           -22.3%     369885        interrupts.CPU26.LOC:Local_timer_interrupts
      1713 ± 19%    +180.1%       4800 ± 21%  interrupts.CPU26.NMI:Non-maskable_interrupts
      1713 ± 19%    +180.1%       4800 ± 21%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
    208.83 ± 11%    +237.4%     704.67 ± 22%  interrupts.CPU26.RES:Rescheduling_interrupts
      7025 ±  9%     +42.6%      10014 ±  4%  interrupts.CPU27.CAL:Function_call_interrupts
    476186           -22.3%     369906        interrupts.CPU27.LOC:Local_timer_interrupts
      1566 ± 28%    +206.7%       4803 ± 21%  interrupts.CPU27.NMI:Non-maskable_interrupts
      1566 ± 28%    +206.7%       4803 ± 21%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
    215.50 ± 15%    +242.9%     739.00 ± 26%  interrupts.CPU27.RES:Rescheduling_interrupts
      6850 ±  9%     +44.4%       9890 ±  3%  interrupts.CPU28.CAL:Function_call_interrupts
    476202           -22.3%     369883        interrupts.CPU28.LOC:Local_timer_interrupts
      1564 ± 27%    +179.8%       4378 ± 30%  interrupts.CPU28.NMI:Non-maskable_interrupts
      1564 ± 27%    +179.8%       4378 ± 30%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
    204.67 ± 22%    +253.2%     722.83 ± 22%  interrupts.CPU28.RES:Rescheduling_interrupts
      6740 ±  8%     +50.0%      10114 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
    476154           -22.3%     369890        interrupts.CPU29.LOC:Local_timer_interrupts
      1789 ± 21%    +141.6%       4324 ± 26%  interrupts.CPU29.NMI:Non-maskable_interrupts
      1789 ± 21%    +141.6%       4324 ± 26%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    206.00 ± 25%    +229.4%     678.50 ± 22%  interrupts.CPU29.RES:Rescheduling_interrupts
      7201 ± 12%     +42.3%      10250 ±  6%  interrupts.CPU3.CAL:Function_call_interrupts
    476049           -22.3%     369842        interrupts.CPU3.LOC:Local_timer_interrupts
      1811 ± 23%    +164.7%       4795 ± 20%  interrupts.CPU3.NMI:Non-maskable_interrupts
      1811 ± 23%    +164.7%       4795 ± 20%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    256.50 ±  7%    +196.0%     759.33 ± 21%  interrupts.CPU3.RES:Rescheduling_interrupts
      7044 ±  3%     +43.9%      10137 ±  2%  interrupts.CPU30.CAL:Function_call_interrupts
    476240           -22.3%     369906        interrupts.CPU30.LOC:Local_timer_interrupts
      1879          +131.5%       4352 ± 26%  interrupts.CPU30.NMI:Non-maskable_interrupts
      1879          +131.5%       4352 ± 26%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
    198.50 ± 15%    +241.3%     677.50 ± 24%  interrupts.CPU30.RES:Rescheduling_interrupts
      6967 ±  3%     +47.2%      10253 ±  4%  interrupts.CPU31.CAL:Function_call_interrupts
    476045           -22.3%     369913        interrupts.CPU31.LOC:Local_timer_interrupts
      1779 ± 22%    +143.1%       4325 ± 26%  interrupts.CPU31.NMI:Non-maskable_interrupts
      1779 ± 22%    +143.1%       4325 ± 26%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
    197.17 ± 15%    +247.4%     685.00 ± 21%  interrupts.CPU31.RES:Rescheduling_interrupts
      6976 ±  5%     +45.4%      10142 ±  4%  interrupts.CPU32.CAL:Function_call_interrupts
    476104           -22.3%     369898        interrupts.CPU32.LOC:Local_timer_interrupts
      1780 ± 22%    +143.8%       4340 ± 26%  interrupts.CPU32.NMI:Non-maskable_interrupts
      1780 ± 22%    +143.8%       4340 ± 26%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    232.17 ± 20%    +201.1%     699.17 ± 23%  interrupts.CPU32.RES:Rescheduling_interrupts
      6852 ±  5%     +46.4%      10030 ±  2%  interrupts.CPU33.CAL:Function_call_interrupts
    475383           -22.2%     370035        interrupts.CPU33.LOC:Local_timer_interrupts
      1893 ±  3%    +129.9%       4354 ± 27%  interrupts.CPU33.NMI:Non-maskable_interrupts
      1893 ±  3%    +129.9%       4354 ± 27%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
    228.33 ± 23%    +195.3%     674.33 ± 18%  interrupts.CPU33.RES:Rescheduling_interrupts
      6907 ±  9%     +45.9%      10077 ±  3%  interrupts.CPU34.CAL:Function_call_interrupts
    475355           -22.2%     369890        interrupts.CPU34.LOC:Local_timer_interrupts
      1927 ±  4%    +125.5%       4346 ± 26%  interrupts.CPU34.NMI:Non-maskable_interrupts
      1927 ±  4%    +125.5%       4346 ± 26%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
    220.50 ± 36%    +198.4%     658.00 ± 23%  interrupts.CPU34.RES:Rescheduling_interrupts
      6849 ±  5%     +42.9%       9786 ±  3%  interrupts.CPU35.CAL:Function_call_interrupts
    476198           -22.3%     369902        interrupts.CPU35.LOC:Local_timer_interrupts
      1892 ±  2%    +152.1%       4771 ± 17%  interrupts.CPU35.NMI:Non-maskable_interrupts
      1892 ±  2%    +152.1%       4771 ± 17%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
    206.50 ± 17%    +227.7%     676.67 ± 24%  interrupts.CPU35.RES:Rescheduling_interrupts
      6737 ±  5%     +47.6%       9945 ±  3%  interrupts.CPU36.CAL:Function_call_interrupts
    476284           -22.3%     369840        interrupts.CPU36.LOC:Local_timer_interrupts
      1915 ±  4%    +172.7%       5221 ±  5%  interrupts.CPU36.NMI:Non-maskable_interrupts
      1915 ±  4%    +172.7%       5221 ±  5%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    194.00 ± 13%    +241.4%     662.33 ± 24%  interrupts.CPU36.RES:Rescheduling_interrupts
      6742 ±  4%     +49.3%      10064 ±  4%  interrupts.CPU37.CAL:Function_call_interrupts
    476219           -22.3%     369905        interrupts.CPU37.LOC:Local_timer_interrupts
      1866 ±  3%    +132.3%       4335 ± 26%  interrupts.CPU37.NMI:Non-maskable_interrupts
      1866 ±  3%    +132.3%       4335 ± 26%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
    216.50 ± 22%    +190.8%     629.67 ± 18%  interrupts.CPU37.RES:Rescheduling_interrupts
      6707           +50.5%      10092 ±  3%  interrupts.CPU38.CAL:Function_call_interrupts
    475789           -22.3%     369896        interrupts.CPU38.LOC:Local_timer_interrupts
      1684 ± 19%    +180.6%       4727 ± 17%  interrupts.CPU38.NMI:Non-maskable_interrupts
      1684 ± 19%    +180.6%       4727 ± 17%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    208.67 ± 13%    +230.8%     690.33 ± 22%  interrupts.CPU38.RES:Rescheduling_interrupts
      6640 ±  3%     +48.4%       9856 ±  3%  interrupts.CPU39.CAL:Function_call_interrupts
    475869           -22.3%     369860        interrupts.CPU39.LOC:Local_timer_interrupts
      1652 ± 17%    +218.7%       5266 ±  5%  interrupts.CPU39.NMI:Non-maskable_interrupts
      1652 ± 17%    +218.7%       5266 ±  5%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
    201.17 ± 22%    +236.0%     675.83 ± 21%  interrupts.CPU39.RES:Rescheduling_interrupts
      7533 ± 11%     +44.2%      10861 ±  7%  interrupts.CPU4.CAL:Function_call_interrupts
    476016           -22.3%     369708        interrupts.CPU4.LOC:Local_timer_interrupts
      1757 ± 21%    +197.3%       5225 ±  4%  interrupts.CPU4.NMI:Non-maskable_interrupts
      1757 ± 21%    +197.3%       5225 ±  4%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    269.00 ± 14%    +185.2%     767.17 ± 24%  interrupts.CPU4.RES:Rescheduling_interrupts
      7026 ±  4%     +38.0%       9699 ±  3%  interrupts.CPU40.CAL:Function_call_interrupts
    476211           -22.3%     370073        interrupts.CPU40.LOC:Local_timer_interrupts
      1745 ± 21%    +199.6%       5229 ±  5%  interrupts.CPU40.NMI:Non-maskable_interrupts
      1745 ± 21%    +199.6%       5229 ±  5%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    197.33 ± 23%    +225.4%     642.17 ± 19%  interrupts.CPU40.RES:Rescheduling_interrupts
      6634 ±  2%     +48.0%       9822 ±  4%  interrupts.CPU41.CAL:Function_call_interrupts
    476106           -22.3%     369880        interrupts.CPU41.LOC:Local_timer_interrupts
      1558 ± 28%    +236.0%       5236 ±  4%  interrupts.CPU41.NMI:Non-maskable_interrupts
      1558 ± 28%    +236.0%       5236 ±  4%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
    201.50 ± 19%    +237.5%     680.00 ± 24%  interrupts.CPU41.RES:Rescheduling_interrupts
      7273 ± 17%     +35.2%       9835 ±  4%  interrupts.CPU42.CAL:Function_call_interrupts
    476213           -22.3%     369888        interrupts.CPU42.LOC:Local_timer_interrupts
      1887 ±  2%    +153.7%       4789 ± 21%  interrupts.CPU42.NMI:Non-maskable_interrupts
      1887 ±  2%    +153.7%       4789 ± 21%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
    194.33 ± 18%    +225.0%     631.50 ± 22%  interrupts.CPU42.RES:Rescheduling_interrupts
      7241 ±  9%     +36.7%       9898 ±  5%  interrupts.CPU43.CAL:Function_call_interrupts
    476178           -22.3%     369870        interrupts.CPU43.LOC:Local_timer_interrupts
      1893 ±  3%    +175.8%       5221 ±  5%  interrupts.CPU43.NMI:Non-maskable_interrupts
      1893 ±  3%    +175.8%       5221 ±  5%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
    211.50 ± 13%    +236.1%     710.83 ± 20%  interrupts.CPU43.RES:Rescheduling_interrupts
      7120 ±  3%     +43.1%      10186 ±  5%  interrupts.CPU44.CAL:Function_call_interrupts
    476290           -22.4%     369789        interrupts.CPU44.LOC:Local_timer_interrupts
      1767 ± 19%    +200.6%       5312 ±  4%  interrupts.CPU44.NMI:Non-maskable_interrupts
      1767 ± 19%    +200.6%       5312 ±  4%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
    260.33 ± 15%    +159.3%     675.00 ± 21%  interrupts.CPU44.RES:Rescheduling_interrupts
      7135 ±  3%     +40.4%      10018 ±  6%  interrupts.CPU45.CAL:Function_call_interrupts
    476181           -22.3%     369846        interrupts.CPU45.LOC:Local_timer_interrupts
      1742 ± 20%    +175.9%       4809 ± 21%  interrupts.CPU45.NMI:Non-maskable_interrupts
      1742 ± 20%    +175.9%       4809 ± 21%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    270.33 ± 31%    +152.3%     682.00 ± 25%  interrupts.CPU45.RES:Rescheduling_interrupts
      6998 ±  7%     +41.9%       9930 ±  7%  interrupts.CPU46.CAL:Function_call_interrupts
    475915           -22.3%     369876        interrupts.CPU46.LOC:Local_timer_interrupts
      1579 ± 27%    +232.5%       5251 ±  4%  interrupts.CPU46.NMI:Non-maskable_interrupts
      1579 ± 27%    +232.5%       5251 ±  4%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    246.50 ± 17%    +175.2%     678.33 ± 26%  interrupts.CPU46.RES:Rescheduling_interrupts
      6908 ±  5%     +45.5%      10054 ±  5%  interrupts.CPU47.CAL:Function_call_interrupts
    476115           -22.3%     369855        interrupts.CPU47.LOC:Local_timer_interrupts
      1799 ± 18%    +168.8%       4837 ± 21%  interrupts.CPU47.NMI:Non-maskable_interrupts
      1799 ± 18%    +168.8%       4837 ± 21%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
    264.33 ± 19%    +165.1%     700.67 ± 21%  interrupts.CPU47.RES:Rescheduling_interrupts
      7021 ±  4%     +42.4%       9996 ±  7%  interrupts.CPU48.CAL:Function_call_interrupts
    476142           -22.3%     369893        interrupts.CPU48.LOC:Local_timer_interrupts
      1769 ± 19%    +194.9%       5218 ±  4%  interrupts.CPU48.NMI:Non-maskable_interrupts
      1769 ± 19%    +194.9%       5218 ±  4%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    263.67 ± 20%    +153.5%     668.50 ± 23%  interrupts.CPU48.RES:Rescheduling_interrupts
      7073 ±  5%     +43.0%      10116 ±  7%  interrupts.CPU49.CAL:Function_call_interrupts
    476187           -22.3%     369850        interrupts.CPU49.LOC:Local_timer_interrupts
      1608 ± 28%    +223.4%       5200 ±  4%  interrupts.CPU49.NMI:Non-maskable_interrupts
      1608 ± 28%    +223.4%       5200 ±  4%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
    249.83 ± 19%    +181.3%     702.67 ± 26%  interrupts.CPU49.RES:Rescheduling_interrupts
      7164 ±  4%     +42.3%      10195 ±  5%  interrupts.CPU5.CAL:Function_call_interrupts
    476239           -22.3%     369864        interrupts.CPU5.LOC:Local_timer_interrupts
      1777 ± 20%    +167.0%       4746 ± 18%  interrupts.CPU5.NMI:Non-maskable_interrupts
      1777 ± 20%    +167.0%       4746 ± 18%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
    254.33 ± 19%    +191.7%     742.00 ± 24%  interrupts.CPU5.RES:Rescheduling_interrupts
      7246 ±  3%     +39.7%      10120 ±  8%  interrupts.CPU50.CAL:Function_call_interrupts
    476219           -22.3%     370027        interrupts.CPU50.LOC:Local_timer_interrupts
      1724 ± 19%    +203.1%       5228 ±  4%  interrupts.CPU50.NMI:Non-maskable_interrupts
      1724 ± 19%    +203.1%       5228 ±  4%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
    256.33 ± 20%    +180.2%     718.17 ± 28%  interrupts.CPU50.RES:Rescheduling_interrupts
      7235 ±  5%     +38.8%      10040 ±  7%  interrupts.CPU51.CAL:Function_call_interrupts
    476182           -22.3%     369876        interrupts.CPU51.LOC:Local_timer_interrupts
      1753 ± 18%    +174.2%       4806 ± 22%  interrupts.CPU51.NMI:Non-maskable_interrupts
      1753 ± 18%    +174.2%       4806 ± 22%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
    267.67 ± 26%    +155.7%     684.33 ± 24%  interrupts.CPU51.RES:Rescheduling_interrupts
      7067 ±  3%     +45.3%      10266 ±  5%  interrupts.CPU52.CAL:Function_call_interrupts
    476238           -22.3%     369857        interrupts.CPU52.LOC:Local_timer_interrupts
      1736 ± 19%    +202.1%       5246 ±  4%  interrupts.CPU52.NMI:Non-maskable_interrupts
      1736 ± 19%    +202.1%       5246 ±  4%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
    239.83 ± 11%    +193.1%     702.83 ± 22%  interrupts.CPU52.RES:Rescheduling_interrupts
      6906 ±  4%     +49.1%      10301 ±  4%  interrupts.CPU53.CAL:Function_call_interrupts
    476255           -22.3%     369874        interrupts.CPU53.LOC:Local_timer_interrupts
      1729 ± 19%    +202.7%       5235 ±  4%  interrupts.CPU53.NMI:Non-maskable_interrupts
      1729 ± 19%    +202.7%       5235 ±  4%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    259.67 ± 12%    +163.6%     684.50 ± 24%  interrupts.CPU53.RES:Rescheduling_interrupts
      7148 ±  4%     +45.7%      10411 ±  3%  interrupts.CPU54.CAL:Function_call_interrupts
    476244           -22.3%     369899        interrupts.CPU54.LOC:Local_timer_interrupts
      1721 ± 19%    +205.5%       5260 ±  5%  interrupts.CPU54.NMI:Non-maskable_interrupts
      1721 ± 19%    +205.5%       5260 ±  5%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
    272.17 ± 16%    +168.6%     731.00 ± 24%  interrupts.CPU54.RES:Rescheduling_interrupts
      7283 ±  4%     +42.6%      10388 ±  4%  interrupts.CPU55.CAL:Function_call_interrupts
    476440           -22.4%     369878        interrupts.CPU55.LOC:Local_timer_interrupts
      1785 ± 19%    +194.0%       5247 ±  4%  interrupts.CPU55.NMI:Non-maskable_interrupts
      1785 ± 19%    +194.0%       5247 ±  4%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
    262.83 ± 21%    +164.2%     694.33 ± 24%  interrupts.CPU55.RES:Rescheduling_interrupts
      7291 ±  2%     +41.7%      10330 ±  4%  interrupts.CPU56.CAL:Function_call_interrupts
    476375           -22.4%     369871        interrupts.CPU56.LOC:Local_timer_interrupts
      1727 ± 19%    +179.4%       4826 ± 20%  interrupts.CPU56.NMI:Non-maskable_interrupts
      1727 ± 19%    +179.4%       4826 ± 20%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    279.17 ± 32%    +143.6%     680.00 ± 23%  interrupts.CPU56.RES:Rescheduling_interrupts
      7120 ±  2%     +44.8%      10310 ±  4%  interrupts.CPU57.CAL:Function_call_interrupts
    475816           -22.3%     369892        interrupts.CPU57.LOC:Local_timer_interrupts
      1586 ± 24%    +232.1%       5270 ±  4%  interrupts.CPU57.NMI:Non-maskable_interrupts
      1586 ± 24%    +232.1%       5270 ±  4%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
    264.00 ± 21%    +158.3%     682.00 ± 22%  interrupts.CPU57.RES:Rescheduling_interrupts
      6780 ±  7%     +50.1%      10175 ±  6%  interrupts.CPU58.CAL:Function_call_interrupts
    476224           -22.3%     369850        interrupts.CPU58.LOC:Local_timer_interrupts
      1787 ± 20%    +192.6%       5228 ±  4%  interrupts.CPU58.NMI:Non-maskable_interrupts
      1787 ± 20%    +192.6%       5228 ±  4%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
    244.67 ± 16%    +185.8%     699.17 ± 27%  interrupts.CPU58.RES:Rescheduling_interrupts
      6554 ± 10%     +57.3%      10307 ±  3%  interrupts.CPU59.CAL:Function_call_interrupts
    476246           -22.3%     369881        interrupts.CPU59.LOC:Local_timer_interrupts
      1739 ± 18%    +175.2%       4786 ± 21%  interrupts.CPU59.NMI:Non-maskable_interrupts
      1739 ± 18%    +175.2%       4786 ± 21%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
    237.83 ± 11%    +186.5%     681.33 ± 24%  interrupts.CPU59.RES:Rescheduling_interrupts
      7072 ±  3%     +46.1%      10331 ± 11%  interrupts.CPU6.CAL:Function_call_interrupts
    476225           -22.3%     369913        interrupts.CPU6.LOC:Local_timer_interrupts
      1917 ±  4%    +148.5%       4764 ± 18%  interrupts.CPU6.NMI:Non-maskable_interrupts
      1917 ±  4%    +148.5%       4764 ± 18%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    276.83 ± 19%    +155.1%     706.33 ± 18%  interrupts.CPU6.RES:Rescheduling_interrupts
      6616 ±  7%     +58.3%      10474 ±  3%  interrupts.CPU60.CAL:Function_call_interrupts
    476292           -22.3%     369897        interrupts.CPU60.LOC:Local_timer_interrupts
      1755 ± 20%    +198.6%       5241 ±  4%  interrupts.CPU60.NMI:Non-maskable_interrupts
      1755 ± 20%    +198.6%       5241 ±  4%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
    257.67 ± 17%    +158.2%     665.17 ± 24%  interrupts.CPU60.RES:Rescheduling_interrupts
      6714 ±  5%     +53.8%      10327 ±  4%  interrupts.CPU61.CAL:Function_call_interrupts
    476274           -22.3%     369917        interrupts.CPU61.LOC:Local_timer_interrupts
      1730 ± 22%    +202.4%       5233 ±  4%  interrupts.CPU61.NMI:Non-maskable_interrupts
      1730 ± 22%    +202.4%       5233 ±  4%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
    310.17 ± 27%    +123.1%     691.83 ± 24%  interrupts.CPU61.RES:Rescheduling_interrupts
      6888 ±  7%     +46.7%      10105 ±  3%  interrupts.CPU62.CAL:Function_call_interrupts
    476163           -22.3%     369895        interrupts.CPU62.LOC:Local_timer_interrupts
      1712 ± 19%    +203.9%       5205 ±  4%  interrupts.CPU62.NMI:Non-maskable_interrupts
      1712 ± 19%    +203.9%       5205 ±  4%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
    265.83 ± 18%    +158.6%     687.33 ± 21%  interrupts.CPU62.RES:Rescheduling_interrupts
      6847           +53.9%      10535 ±  6%  interrupts.CPU63.CAL:Function_call_interrupts
    476208           -22.3%     369874        interrupts.CPU63.LOC:Local_timer_interrupts
      1906 ±  4%    +173.8%       5219 ±  4%  interrupts.CPU63.NMI:Non-maskable_interrupts
      1906 ±  4%    +173.8%       5219 ±  4%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
    269.83 ± 18%    +167.0%     720.50 ± 19%  interrupts.CPU63.RES:Rescheduling_interrupts
      7023 ±  3%     +50.2%      10549 ±  8%  interrupts.CPU64.CAL:Function_call_interrupts
    476247           -22.3%     369882        interrupts.CPU64.LOC:Local_timer_interrupts
      1911          +151.2%       4801 ± 21%  interrupts.CPU64.NMI:Non-maskable_interrupts
      1911          +151.2%       4801 ± 21%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
    276.50 ± 20%    +147.7%     685.00 ± 21%  interrupts.CPU64.RES:Rescheduling_interrupts
      7128 ±  2%     +49.7%      10672 ±  7%  interrupts.CPU65.CAL:Function_call_interrupts
    476244           -22.3%     369903        interrupts.CPU65.LOC:Local_timer_interrupts
      1756 ± 22%    +174.2%       4815 ± 21%  interrupts.CPU65.NMI:Non-maskable_interrupts
      1756 ± 22%    +174.2%       4815 ± 21%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
    257.33 ± 11%    +177.7%     714.67 ± 25%  interrupts.CPU65.RES:Rescheduling_interrupts
      7488 ± 12%     +51.8%      11368 ±  5%  interrupts.CPU66.CAL:Function_call_interrupts
    476308           -22.3%     369891        interrupts.CPU66.LOC:Local_timer_interrupts
      1878 ±  2%    +155.3%       4794 ± 20%  interrupts.CPU66.NMI:Non-maskable_interrupts
      1878 ±  2%    +155.3%       4794 ± 20%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
    207.67 ± 22%    +245.3%     717.17 ± 25%  interrupts.CPU66.RES:Rescheduling_interrupts
      7053 ±  3%     +43.7%      10137 ±  3%  interrupts.CPU67.CAL:Function_call_interrupts
    476207           -22.3%     369874        interrupts.CPU67.LOC:Local_timer_interrupts
      1881 ±  4%    +154.2%       4783 ± 21%  interrupts.CPU67.NMI:Non-maskable_interrupts
      1881 ±  4%    +154.2%       4783 ± 21%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
    206.83 ± 22%    +222.9%     667.83 ± 26%  interrupts.CPU67.RES:Rescheduling_interrupts
      7025 ±  5%     +40.8%       9893 ±  2%  interrupts.CPU68.CAL:Function_call_interrupts
    476208           -22.3%     369900        interrupts.CPU68.LOC:Local_timer_interrupts
      1912 ±  4%    +150.5%       4789 ± 21%  interrupts.CPU68.NMI:Non-maskable_interrupts
      1912 ±  4%    +150.5%       4789 ± 21%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
    220.17 ± 21%    +234.7%     736.83 ± 33%  interrupts.CPU68.RES:Rescheduling_interrupts
      6670 ±  6%     +49.0%       9937 ±  3%  interrupts.CPU69.CAL:Function_call_interrupts
    476223           -22.3%     369911        interrupts.CPU69.LOC:Local_timer_interrupts
      1723 ± 18%    +178.7%       4802 ± 21%  interrupts.CPU69.NMI:Non-maskable_interrupts
      1723 ± 18%    +178.7%       4802 ± 21%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
    221.67 ± 27%    +212.7%     693.17 ± 30%  interrupts.CPU69.RES:Rescheduling_interrupts
      6765 ±  5%     +48.0%      10010 ±  6%  interrupts.CPU7.CAL:Function_call_interrupts
    476263           -22.3%     369871        interrupts.CPU7.LOC:Local_timer_interrupts
      1931 ±  6%    +124.3%       4330 ± 27%  interrupts.CPU7.NMI:Non-maskable_interrupts
      1931 ±  6%    +124.3%       4330 ± 27%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    265.17 ± 21%    +164.8%     702.17 ± 19%  interrupts.CPU7.RES:Rescheduling_interrupts
      6791 ±  3%     +46.9%       9976 ±  2%  interrupts.CPU70.CAL:Function_call_interrupts
    476236           -22.3%     369910        interrupts.CPU70.LOC:Local_timer_interrupts
      1610 ± 31%    +166.5%       4291 ± 25%  interrupts.CPU70.NMI:Non-maskable_interrupts
      1610 ± 31%    +166.5%       4291 ± 25%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
    243.33 ± 26%    +226.4%     794.33 ± 31%  interrupts.CPU70.RES:Rescheduling_interrupts
      7258 ±  6%     +40.3%      10183 ±  4%  interrupts.CPU71.CAL:Function_call_interrupts
    476253           -22.3%     369844        interrupts.CPU71.LOC:Local_timer_interrupts
      1549 ± 27%    +179.0%       4323 ± 25%  interrupts.CPU71.NMI:Non-maskable_interrupts
      1549 ± 27%    +179.0%       4323 ± 25%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
    229.67 ± 19%    +227.4%     751.83 ± 30%  interrupts.CPU71.RES:Rescheduling_interrupts
      7140 ±  2%     +42.3%      10161 ±  3%  interrupts.CPU72.CAL:Function_call_interrupts
    476206           -22.3%     369937        interrupts.CPU72.LOC:Local_timer_interrupts
      1899 ±  2%    +153.3%       4811 ± 21%  interrupts.CPU72.NMI:Non-maskable_interrupts
      1899 ±  2%    +153.3%       4811 ± 21%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
    226.83 ± 21%    +232.3%     753.83 ± 32%  interrupts.CPU72.RES:Rescheduling_interrupts
      6933 ±  4%     +48.0%      10263 ±  3%  interrupts.CPU73.CAL:Function_call_interrupts
    476265           -22.3%     369914        interrupts.CPU73.LOC:Local_timer_interrupts
      1627 ± 28%    +163.6%       4290 ± 25%  interrupts.CPU73.NMI:Non-maskable_interrupts
      1627 ± 28%    +163.6%       4290 ± 25%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
    213.50 ± 23%    +229.8%     704.17 ± 31%  interrupts.CPU73.RES:Rescheduling_interrupts
      7038 ±  4%     +45.4%      10233 ±  4%  interrupts.CPU74.CAL:Function_call_interrupts
    476235           -22.3%     369905        interrupts.CPU74.LOC:Local_timer_interrupts
      1592 ± 27%    +170.4%       4306 ± 24%  interrupts.CPU74.NMI:Non-maskable_interrupts
      1592 ± 27%    +170.4%       4306 ± 24%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    222.67 ± 23%    +238.8%     754.50 ± 35%  interrupts.CPU74.RES:Rescheduling_interrupts
      7283 ±  4%     +39.9%      10186 ±  6%  interrupts.CPU75.CAL:Function_call_interrupts
    475989           -22.3%     369895        interrupts.CPU75.LOC:Local_timer_interrupts
      1767 ± 20%    +168.8%       4750 ± 17%  interrupts.CPU75.NMI:Non-maskable_interrupts
      1767 ± 20%    +168.8%       4750 ± 17%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
    212.17 ± 20%    +242.2%     726.00 ± 29%  interrupts.CPU75.RES:Rescheduling_interrupts
      7129 ±  3%     +41.3%      10073 ±  3%  interrupts.CPU76.CAL:Function_call_interrupts
    476200           -22.3%     369927        interrupts.CPU76.LOC:Local_timer_interrupts
      1607 ± 29%    +169.1%       4326 ± 26%  interrupts.CPU76.NMI:Non-maskable_interrupts
      1607 ± 29%    +169.1%       4326 ± 26%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
    218.83 ± 15%    +218.4%     696.83 ± 17%  interrupts.CPU76.RES:Rescheduling_interrupts
      7283 ±  6%     +37.0%       9976 ±  4%  interrupts.CPU77.CAL:Function_call_interrupts
    475988           -22.3%     369928        interrupts.CPU77.LOC:Local_timer_interrupts
      1716 ± 20%    +178.0%       4772 ± 18%  interrupts.CPU77.NMI:Non-maskable_interrupts
      1716 ± 20%    +178.0%       4772 ± 18%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
    210.83 ± 15%    +235.3%     707.00 ± 14%  interrupts.CPU77.RES:Rescheduling_interrupts
      7201 ±  5%     +38.4%       9964 ±  4%  interrupts.CPU78.CAL:Function_call_interrupts
    475939           -22.3%     369923        interrupts.CPU78.LOC:Local_timer_interrupts
      1753 ± 20%    +146.3%       4317 ± 25%  interrupts.CPU78.NMI:Non-maskable_interrupts
      1753 ± 20%    +146.3%       4317 ± 25%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
    223.00 ± 17%    +212.1%     696.00 ± 18%  interrupts.CPU78.RES:Rescheduling_interrupts
      6960 ±  6%     +41.6%       9852 ±  4%  interrupts.CPU79.CAL:Function_call_interrupts
    476136           -22.3%     369924        interrupts.CPU79.LOC:Local_timer_interrupts
      1725 ± 20%    +149.6%       4307 ± 26%  interrupts.CPU79.NMI:Non-maskable_interrupts
      1725 ± 20%    +149.6%       4307 ± 26%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    215.50 ± 16%    +224.2%     698.67 ± 19%  interrupts.CPU79.RES:Rescheduling_interrupts
      6978 ±  3%     +42.5%       9947 ±  6%  interrupts.CPU8.CAL:Function_call_interrupts
    476186           -22.3%     369891        interrupts.CPU8.LOC:Local_timer_interrupts
      1934 ±  6%    +146.8%       4773 ± 18%  interrupts.CPU8.NMI:Non-maskable_interrupts
      1934 ±  6%    +146.8%       4773 ± 18%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    286.67 ± 16%    +163.7%     755.83 ± 23%  interrupts.CPU8.RES:Rescheduling_interrupts
      6862 ±  5%     +44.5%       9913 ±  4%  interrupts.CPU80.CAL:Function_call_interrupts
    476242           -22.3%     369922        interrupts.CPU80.LOC:Local_timer_interrupts
      1747 ± 19%    +145.5%       4289 ± 26%  interrupts.CPU80.NMI:Non-maskable_interrupts
      1747 ± 19%    +145.5%       4289 ± 26%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
    247.83 ± 25%    +198.9%     740.83 ± 17%  interrupts.CPU80.RES:Rescheduling_interrupts
      6936 ±  5%     +45.9%      10121 ±  3%  interrupts.CPU81.CAL:Function_call_interrupts
    476220           -22.3%     369909        interrupts.CPU81.LOC:Local_timer_interrupts
      1892 ±  3%    +126.8%       4291 ± 25%  interrupts.CPU81.NMI:Non-maskable_interrupts
      1892 ±  3%    +126.8%       4291 ± 25%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
    212.67 ± 14%    +226.9%     695.17 ± 16%  interrupts.CPU81.RES:Rescheduling_interrupts
      6772 ± 10%     +49.1%      10095 ±  4%  interrupts.CPU82.CAL:Function_call_interrupts
    475272           -22.2%     369914        interrupts.CPU82.LOC:Local_timer_interrupts
      1712 ± 20%    +150.6%       4290 ± 25%  interrupts.CPU82.NMI:Non-maskable_interrupts
      1712 ± 20%    +150.6%       4290 ± 25%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
    242.17 ± 28%    +192.3%     707.83 ± 17%  interrupts.CPU82.RES:Rescheduling_interrupts
      7143 ±  4%     +39.7%       9977 ±  3%  interrupts.CPU83.CAL:Function_call_interrupts
    476161           -22.3%     370056        interrupts.CPU83.LOC:Local_timer_interrupts
      1525 ± 29%    +244.2%       5252 ±  5%  interrupts.CPU83.NMI:Non-maskable_interrupts
      1525 ± 29%    +244.2%       5252 ±  5%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
    217.67 ± 18%    +215.8%     687.33 ± 21%  interrupts.CPU83.RES:Rescheduling_interrupts
      7063           +39.4%       9847 ±  4%  interrupts.CPU84.CAL:Function_call_interrupts
    476240           -22.3%     369919        interrupts.CPU84.LOC:Local_timer_interrupts
      1753 ± 21%    +197.5%       5218 ±  5%  interrupts.CPU84.NMI:Non-maskable_interrupts
      1753 ± 21%    +197.5%       5218 ±  5%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
    235.67 ± 24%    +225.4%     766.83 ± 25%  interrupts.CPU84.RES:Rescheduling_interrupts
      7012 ±  4%     +41.3%       9905 ±  4%  interrupts.CPU85.CAL:Function_call_interrupts
    476251           -22.3%     369914        interrupts.CPU85.LOC:Local_timer_interrupts
      1747 ± 20%    +122.9%       3895 ± 31%  interrupts.CPU85.NMI:Non-maskable_interrupts
      1747 ± 20%    +122.9%       3895 ± 31%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
    268.83 ± 20%    +219.6%     859.17 ± 29%  interrupts.CPU85.RES:Rescheduling_interrupts
      7167 ±  3%     +41.2%      10122 ±  2%  interrupts.CPU86.CAL:Function_call_interrupts
    476267           -22.3%     369901        interrupts.CPU86.LOC:Local_timer_interrupts
      1764 ± 21%    +143.3%       4292 ± 25%  interrupts.CPU86.NMI:Non-maskable_interrupts
      1764 ± 21%    +143.3%       4292 ± 25%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
    277.50 ± 36%    +181.6%     781.33 ± 27%  interrupts.CPU86.RES:Rescheduling_interrupts
      6977 ±  9%     +40.2%       9779 ±  5%  interrupts.CPU87.CAL:Function_call_interrupts
    476240           -22.3%     369945        interrupts.CPU87.LOC:Local_timer_interrupts
      1757 ± 22%    +171.1%       4765 ± 21%  interrupts.CPU87.NMI:Non-maskable_interrupts
      1757 ± 22%    +171.1%       4765 ± 21%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
    241.33 ± 19%    +239.8%     820.17 ± 29%  interrupts.CPU87.RES:Rescheduling_interrupts
      7026 ±  5%     +44.8%      10171 ±  6%  interrupts.CPU9.CAL:Function_call_interrupts
    476181           -22.3%     369914        interrupts.CPU9.LOC:Local_timer_interrupts
      1882 ±  2%    +178.2%       5237 ±  4%  interrupts.CPU9.NMI:Non-maskable_interrupts
      1882 ±  2%    +178.2%       5237 ±  4%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    262.83 ± 19%    +162.4%     689.67 ± 22%  interrupts.CPU9.RES:Rescheduling_interrupts
    269.83          +170.7%     730.33 ±  4%  interrupts.IWI:IRQ_work_interrupts
  41901193           -22.3%   32550806        interrupts.LOC:Local_timer_interrupts
    153407 ±  2%    +174.7%     421404 ±  5%  interrupts.NMI:Non-maskable_interrupts
    153407 ±  2%    +174.7%     421404 ±  5%  interrupts.PMI:Performance_monitoring_interrupts
     21376 ±  3%    +191.8%      62368        interrupts.RES:Rescheduling_interrupts
    763.17 ± 32%    +141.5%       1842 ± 29%  interrupts.TLB:TLB_shootdowns


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  115000 +------------------------------------------------------------------+   
         |   ++  +  +.++++.+++ .++    ++    +.+++ + ++.++++.+:              |   
         |        ++          +                    +          :             |   
  110000 |-+                                                  :             |   
         |                                                    ++.++++.++++.+|   
         |                                                                  |   
  105000 |-+                                                                |   
         |                                                                  |   
  100000 |-+                                                                |   
         |                                       O OOO OO                   |   
         |  OOOO OOO  OOOO OOOO OO  OOOO O OO  O                            |   
   95000 |O+        O             O       O     O                           |   
         |                                    O                             |   
         |                                                                  |   
   90000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               aim7.time.system_time                            
                                                                                
  7000 +--------------------------------------------------------------------+   
       |                                     O                              |   
  6000 |-+ O       O O       O    O OO    OO  OO                            |   
       |O O   OOOO  O  OOO OO O OO    O OO       O O OO                     |   
       |    O                                     O    O                    |   
  5000 |-+                                                                  |   
       |                                                                    |   
  4000 |-+                                                                  |   
       |                                                                    |   
  3000 |-+                                                                  |   
       |                                                                    |   
       |                                                      +++.++++.+++.+|   
  2000 |-+                                                   :              |   
       |                                                     :              |   
  1000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             aim7.time.minor_page_faults                        
                                                                                
  250000 +------------------------------------------------------------------+   
         |                                    O                             |   
  240000 |-+                                                                |   
  230000 |-+                  O       O                                     |   
         |       O  O                O      O   O                           |   
  220000 |-+ O O            O   O   O  O   O     O  O  O                    |   
         |O O O   OO  OO   O O   OO      OO    O   O O                      |   
  210000 |-+            O                               O                   |   
         |               O                                                  |   
  200000 |-+                                                                |   
  190000 |-+                                                                |   
         |             +          +     .+                                  |   
  180000 |:.+ ++.+ +   ::   ++    :+ + +  :+ .+ + .++    ++  +++.  + .+  +. |   
         |+  +    + +.+ ++.+  +.++  + +   + +  + +   :  +  + :   ++ +  ++  +|   
  170000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       aim7.time.involuntary_context_switches                   
                                                                                
  30000 +-------------------------------------------------------------------+   
        |                                                                   |   
  25000 |-+                                  O  O                           |   
        |O O    O O O O       O    OO O    OO O                             |   
        |   O O  O   O O   O O OO O    OOO       O  OOOO                    |   
  20000 |-+  O           OO                       O                         |   
        |                                                                   |   
  15000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
   5000 |-+                                                   +++.+++.++++.+|   
        |+.++++.+++.++++.+++.++++.+++.++++.++++.+++.++++.+++.+              |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
      perf-sched.wait_time.avg.ms.schedule_timeout.__down.down.xfs_buf_lock     
                                                                                
  45 +----------------------------------------------------------------------+   
     |                                                              ++. ++.+|   
  40 |-+                                                     +.+++.+   +    |   
  35 |-+ +   ++.+++.+++.+++.++ .+  .+ +.  .+ +.+ +.++ .+++.++               |   
     |+.+ +.+                 +  ++  +  ++  +   +    +                      |   
  30 |-+                                                                    |   
  25 |-+                                                                    |   
     |                                                                      |   
  20 |-+                                                                    |   
  15 |-+             O    O  O       O              O                       |   
     |       OO OOO O O OO  O O OOO O O OO OOO OOO O O O                    |   
  10 |-+    O                                                               |   
   5 |-+                                                                    |   
     |                                                                      |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
   perf-sched.wait_and_delay.avg.ms.schedule_timeout.__down.down.xfs_buf_lock   
                                                                                
  45 +----------------------------------------------------------------------+   
     |                                                              ++. ++.+|   
  40 |-+                                                     +.+++.+   +    |   
  35 |-+ +   ++.+++.+++.+++.++ .+  .+ +.  .+ +.+ +.++ .+++.++               |   
     |+.+ +.+                 +  ++  +  ++  +   +    +                      |   
  30 |-+                                                                    |   
  25 |-+                                                                    |   
     |                                                                      |   
  20 |-+                                                                    |   
  15 |-+             O    O  O       O              O                       |   
     |       OO OOO O O OO  O O OOO O O OO OOO OOO O O O                    |   
  10 |-+    O                                                               |   
   5 |-+                                                                    |   
     |                                                                      |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.12.0-rc1-00068-g571519716f11" of type "text/plain" (172871 bytes)

View attachment "job-script" of type "text/plain" (8052 bytes)

View attachment "job.yaml" of type "text/plain" (5574 bytes)

View attachment "reproduce" of type "text/plain" (1028 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ