lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200817093640.GB18179@shao2-debian>
Date:   Mon, 17 Aug 2020 17:36:40 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Darrick J. Wong" <darrick.wong@...cle.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...ts.01.org
Subject: [xfs] 086c2d4ccc: aim7.jobs-per-min -59.6% regression

Greeting,

FYI, we noticed a -59.6% regression of aim7.jobs-per-min due to commit:


commit: 086c2d4ccc61ec0aad9a2ec299a368af78f5db25 ("xfs: deferred inode inactivation")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git realtime-rmap


in testcase: aim7
on test machine: 96 threads Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID1
	fs: xfs
	test: disk_rw
	load: 3000
	cpufreq_governor: performance
	ucode: 0x4002f01

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/4BRD_12G/xfs/x86_64-rhel-8.3/3000/RAID1/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp2/disk_rw/aim7/0x4002f01

commit: 
2
  9b8a1c2f9c ("xfs: track unlinked inactive inode quota counters")
  086c2d4ccc ("xfs: deferred inode inactivation")

9b8a1c2f9c788abe 086c2d4ccc61ec0aad9a2ec299a 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4          100%           4:4     kmsg.XFS(md#):xlog_verify_grant_tail:space>BBTOB(tail_blocks)
         %stddev     %change         %stddev
             \          |                \  
    466695 ±  3%     -59.6%     188321 ±  4%  aim7.jobs-per-min
     38.80 ±  3%    +147.4%      95.98 ±  4%  aim7.time.elapsed_time
     38.80 ±  3%    +147.4%      95.98 ±  4%  aim7.time.elapsed_time.max
  6.17e+08            +9.3%  6.743e+08        aim7.time.file_system_outputs
      2636 ± 16%    +273.9%       9856 ± 10%  aim7.time.involuntary_context_switches
    714.37 ±  3%    +254.9%       2534 ±  4%  aim7.time.system_time
    951157           -55.5%     423711 ± 10%  aim7.time.voluntary_context_switches
     44.76 ±  2%      -2.7%      43.53 ±  2%  boot-time.boot
  11065418 ± 30%    +372.0%   52233041 ± 44%  cpuidle.C1.time
    159454 ± 38%   +2827.1%    4667422 ±140%  cpuidle.POLL.time
     77.36 ±  2%     -11.0%      68.82 ±  2%  iostat.cpu.idle
     19.24           +48.1%      28.51        iostat.cpu.system
     76.59 ±  2%      -8.4       68.22 ±  2%  mpstat.cpu.all.idle%
      1.09 ±  8%      +0.2        1.25 ± 13%  mpstat.cpu.all.irq%
      0.15 ±  4%      -0.0        0.11 ±  7%  mpstat.cpu.all.soft%
     18.66            +9.0       27.70 ±  2%  mpstat.cpu.all.sys%
    384.00 ± 24%     -53.4%     178.88 ± 61%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     45.88 ± 29%     -50.0%      22.96 ± 68%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
    384.00 ± 24%     -53.4%     178.88 ± 61%  sched_debug.cfs_rq:/.removed.util_avg.max
     45.88 ± 29%     -50.0%      22.96 ± 68%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     77.00 ±  2%     -11.4%      68.25 ±  2%  vmstat.cpu.id
      7558 ± 24%   +3878.9%     300753        vmstat.io.bo
     20.00 ± 11%     +42.5%      28.50 ±  3%  vmstat.procs.r
     48440 ±  3%     -74.5%      12350 ± 11%  vmstat.system.cs
    202698            -6.8%     188966 ±  5%  vmstat.system.in
     92966 ±  5%     +49.1%     138648 ±  3%  meminfo.AnonHugePages
   1202645           -66.1%     407469 ± 12%  meminfo.Dirty
    140229 ±  2%     +31.8%     184860 ±  2%  meminfo.KReclaimable
   4375874          +230.3%   14451378 ±  2%  meminfo.Memused
    140229 ±  2%     +31.8%     184860 ±  2%  meminfo.SReclaimable
    338942           +12.7%     381985        meminfo.SUnreclaim
     29939 ±  5%     +58.4%      47423 ±  6%  meminfo.Shmem
    479172           +18.3%     566847        meminfo.Slab
      9.25 ±102%  +3.1e+06%     290779 ±  3%  meminfo.Writeback
    133253 ±  2%     +34.6%     179348 ±  4%  meminfo.max_used_kB
    277464 ± 18%     -51.6%     134428 ± 56%  numa-meminfo.node0.Active
    274799 ± 18%     -52.5%     130440 ± 56%  numa-meminfo.node0.Active(anon)
    272892 ± 18%     -52.3%     130115 ± 57%  numa-meminfo.node0.AnonPages
    601760           -66.5%     201510 ± 14%  numa-meminfo.node0.Dirty
     35584 ± 98%     -97.4%     941.75 ± 23%  numa-meminfo.node0.PageTables
      6.50 ±164%  +2.2e+06%     143107 ±  2%  numa-meminfo.node0.Writeback
    126638 ± 40%    +129.0%     290015 ± 27%  numa-meminfo.node1.Active
    123972 ± 41%    +132.6%     288307 ± 27%  numa-meminfo.node1.Active(anon)
    108798 ± 46%    +133.0%     253535 ± 30%  numa-meminfo.node1.AnonPages
    598918 ±  2%     -64.9%     210091 ± 10%  numa-meminfo.node1.Dirty
     69664 ± 18%     +64.7%     114729 ±  4%  numa-meminfo.node1.KReclaimable
   2079550 ± 14%    +303.8%    8398063 ± 30%  numa-meminfo.node1.MemUsed
     69664 ± 18%     +64.7%     114729 ±  4%  numa-meminfo.node1.SReclaimable
    163185 ± 29%     +46.6%     239205 ±  8%  numa-meminfo.node1.SUnreclaim
     23227 ± 15%     +90.9%      44335 ± 15%  numa-meminfo.node1.Shmem
    232851 ± 25%     +52.0%     353934 ±  6%  numa-meminfo.node1.Slab
      3.25 ±139%  +4.5e+06%     147197 ±  3%  numa-meminfo.node1.Writeback
     99650 ±  2%      +5.0%     104660        proc-vmstat.nr_active_anon
  76519853            +9.4%   83723026        proc-vmstat.nr_dirtied
    299699           -65.3%     104084 ± 12%  proc-vmstat.nr_dirty
   3193035            -7.8%    2942441        proc-vmstat.nr_dirty_background_threshold
   6393878            -7.8%    5892078        proc-vmstat.nr_dirty_threshold
    554120            +2.5%     567705        proc-vmstat.nr_file_pages
  31830662            -7.9%   29311932        proc-vmstat.nr_free_pages
    299875            +3.0%     308897        proc-vmstat.nr_inactive_file
      7484 ±  5%     +58.3%      11849 ±  6%  proc-vmstat.nr_shmem
     35058 ±  2%     +31.8%      46202 ±  2%  proc-vmstat.nr_slab_reclaimable
     84711           +12.8%      95514        proc-vmstat.nr_slab_unreclaimable
      2.25 ±101%  +3.1e+06%      70854 ±  2%  proc-vmstat.nr_writeback
     40861 ± 48%  +17830.4%    7326601 ±  4%  proc-vmstat.nr_written
     99650 ±  2%      +5.0%     104660        proc-vmstat.nr_zone_active_anon
    299875            +3.0%     308897        proc-vmstat.nr_zone_inactive_file
    300041           -41.7%     174864 ±  7%  proc-vmstat.nr_zone_write_pending
  75741698            +4.9%   79467299        proc-vmstat.numa_hit
  75710426            +4.9%   79435937        proc-vmstat.numa_local
  77043687            +4.4%   80444945        proc-vmstat.pgalloc_normal
    313886 ±  3%     +52.6%     478867 ±  2%  proc-vmstat.pgfault
    314729 ± 27%   +9383.7%   29848024 ±  4%  proc-vmstat.pgpgout
      1557            +6.2%       1654        proc-vmstat.unevictable_pgs_culled
     68647 ± 18%     -52.5%      32595 ± 57%  numa-vmstat.node0.nr_active_anon
     68170 ± 18%     -52.3%      32514 ± 57%  numa-vmstat.node0.nr_anon_pages
  13913253           +28.4%   17862796 ±  3%  numa-vmstat.node0.nr_dirtied
    149185 ±  2%     -65.4%      51577 ± 14%  numa-vmstat.node0.nr_dirty
      8868 ± 98%     -97.3%     236.25 ± 24%  numa-vmstat.node0.nr_page_table_pages
     26.25 ±160%  +1.3e+05%      34370 ±  5%  numa-vmstat.node0.nr_writeback
      9900 ± 20%  +16049.8%    1598826        numa-vmstat.node0.nr_written
     68647 ± 18%     -52.5%      32595 ± 57%  numa-vmstat.node0.nr_zone_active_anon
    149492 ±  2%     -42.5%      85884 ±  8%  numa-vmstat.node0.nr_zone_write_pending
  14722161           +22.9%   18090223 ±  3%  numa-vmstat.node0.numa_hit
  14702878           +22.6%   18024478 ±  3%  numa-vmstat.node0.numa_local
     19354 ± 70%    +239.8%      65769 ± 80%  numa-vmstat.node0.numa_other
     30995 ± 41%    +132.4%      72029 ± 27%  numa-vmstat.node1.nr_active_anon
     27202 ± 46%    +132.8%      63335 ± 30%  numa-vmstat.node1.nr_anon_pages
  14041982 ±  2%     +35.1%   18970019 ±  4%  numa-vmstat.node1.nr_dirtied
    149252 ±  3%     -63.9%      53823 ± 11%  numa-vmstat.node1.nr_dirty
      5807 ± 15%     +91.1%      11097 ± 15%  numa-vmstat.node1.nr_shmem
     17421 ± 18%     +64.7%      28687 ±  4%  numa-vmstat.node1.nr_slab_reclaimable
     40779 ± 29%     +46.5%      59736 ±  8%  numa-vmstat.node1.nr_slab_unreclaimable
     37.25 ±170%  +95245.0%      35516 ±  4%  numa-vmstat.node1.nr_writeback
      9230 ± 20%  +18278.3%    1696408 ±  8%  numa-vmstat.node1.nr_written
     30994 ± 41%    +132.4%      72029 ± 27%  numa-vmstat.node1.nr_zone_active_anon
    149470 ±  3%     -40.2%      89333 ±  6%  numa-vmstat.node1.nr_zone_write_pending
  14776117 ±  2%     +34.0%   19797121        numa-vmstat.node1.numa_hit
  14620592 ±  2%     +34.7%   19687191        numa-vmstat.node1.numa_local
      7.14 ± 56%     +88.4%      13.45 ± 44%  perf-stat.i.MPKI
 1.209e+10 ±  3%     -32.5%  8.162e+09        perf-stat.i.branch-instructions
  95610974 ±  2%     -55.5%   42548125 ±  2%  perf-stat.i.branch-misses
     26.79 ±  7%     +10.0       36.79 ± 18%  perf-stat.i.cache-miss-rate%
  46117805 ±  2%     -37.4%   28872456 ±  3%  perf-stat.i.cache-misses
 1.509e+08 ±  6%     -45.0%   82935217 ± 12%  perf-stat.i.cache-references
     49213 ±  3%     -74.8%      12400 ± 12%  perf-stat.i.context-switches
      1.74 ±  7%     +55.2%       2.70 ±  4%  perf-stat.i.cpi
 6.858e+10 ±  8%     +29.5%  8.878e+10 ±  5%  perf-stat.i.cpu-cycles
      2049 ±  9%     -66.1%     695.64 ±  8%  perf-stat.i.cpu-migrations
      1722 ±  5%     +61.4%       2780 ±  5%  perf-stat.i.cycles-between-cache-misses
 1.811e+10 ±  3%     -35.1%  1.176e+10 ±  2%  perf-stat.i.dTLB-loads
 1.028e+10 ±  3%     -56.1%   4.51e+09 ±  3%  perf-stat.i.dTLB-stores
     81.91            -6.3       75.66 ±  2%  perf-stat.i.iTLB-load-miss-rate%
  71279393 ±  3%     -58.1%   29896322 ±  7%  perf-stat.i.iTLB-load-misses
   5293782           -54.6%    2402490 ±  8%  perf-stat.i.iTLB-loads
 6.186e+10 ±  3%     -32.9%  4.149e+10        perf-stat.i.instructions
      1109 ±  8%     +28.3%       1424 ±  8%  perf-stat.i.instructions-per-iTLB-miss
      0.79 ±  9%     -44.6%       0.44 ±  2%  perf-stat.i.ipc
      0.71 ±  8%     +29.4%       0.92 ±  5%  perf-stat.i.metric.GHz
      0.96 ±  2%     -43.3%       0.54 ± 16%  perf-stat.i.metric.K/sec
    423.81 ±  3%     -39.7%     255.45 ±  2%  perf-stat.i.metric.M/sec
      7124 ±  2%     -33.8%       4714        perf-stat.i.minor-faults
   6465680 ±  3%     -42.4%    3721175 ±  3%  perf-stat.i.node-load-misses
   4074433 ±  3%     -47.5%    2138406 ±  4%  perf-stat.i.node-loads
     33.09 ± 15%     +12.5       45.59 ±  3%  perf-stat.i.node-store-miss-rate%
   3102997           +18.6%    3679228        perf-stat.i.node-store-misses
  10007996 ±  7%     -51.7%    4829381 ±  4%  perf-stat.i.node-stores
      7124 ±  2%     -33.8%       4714        perf-stat.i.page-faults
      2.43 ±  3%     -18.2%       1.99 ± 12%  perf-stat.overall.MPKI
      0.79 ±  4%      -0.3        0.52 ±  2%  perf-stat.overall.branch-miss-rate%
      1.11 ± 11%     +92.5%       2.14 ±  5%  perf-stat.overall.cpi
      1490 ± 11%    +107.5%       3092 ±  5%  perf-stat.overall.cycles-between-cache-misses
    869.73 ±  6%     +60.5%       1396 ±  7%  perf-stat.overall.instructions-per-iTLB-miss
      0.91 ± 11%     -48.6%       0.47 ±  5%  perf-stat.overall.ipc
     61.35            +2.2       63.54        perf-stat.overall.node-load-miss-rate%
     23.76 ±  7%     +19.4       43.15 ±  2%  perf-stat.overall.node-store-miss-rate%
 1.184e+10 ±  3%     -31.3%  8.142e+09        perf-stat.ps.branch-instructions
  93534611 ±  2%     -54.7%   42331825 ±  2%  perf-stat.ps.branch-misses
  45161124 ±  2%     -36.5%   28668448 ±  3%  perf-stat.ps.cache-misses
 1.476e+08 ±  6%     -44.2%   82354790 ± 11%  perf-stat.ps.cache-references
     48196 ±  3%     -74.6%      12248 ± 12%  perf-stat.ps.context-switches
     93537            +1.5%      94985        perf-stat.ps.cpu-clock
 6.716e+10 ±  9%     +31.9%  8.858e+10 ±  5%  perf-stat.ps.cpu-cycles
      2005 ±  9%     -65.5%     692.22 ±  8%  perf-stat.ps.cpu-migrations
 1.774e+10 ±  3%     -33.9%  1.173e+10 ±  2%  perf-stat.ps.dTLB-loads
 1.007e+10 ±  3%     -55.4%  4.497e+09 ±  3%  perf-stat.ps.dTLB-stores
  69823910 ±  3%     -57.3%   29816510 ±  8%  perf-stat.ps.iTLB-load-misses
   5176433           -54.1%    2377281 ±  8%  perf-stat.ps.iTLB-loads
 6.058e+10 ±  2%     -31.7%  4.139e+10        perf-stat.ps.instructions
      6904 ±  2%     -32.4%       4664        perf-stat.ps.minor-faults
   6331819 ±  3%     -41.6%    3696139 ±  3%  perf-stat.ps.node-load-misses
   3989503 ±  3%     -46.8%    2121229 ±  4%  perf-stat.ps.node-loads
   3039544           +19.8%    3640553        perf-stat.ps.node-store-misses
   9806434 ±  7%     -51.0%    4801444 ±  4%  perf-stat.ps.node-stores
      6904 ±  2%     -32.4%       4664        perf-stat.ps.page-faults
     93537            +1.5%      94985        perf-stat.ps.task-clock
 2.397e+12           +67.0%  4.003e+12 ±  2%  perf-stat.total.instructions
    245.75 ±  4%    +264.6%     896.00 ±  8%  slabinfo.biovec-max.active_objs
    262.25 ±  4%    +271.7%     974.75 ±  8%  slabinfo.biovec-max.num_objs
      2394           +22.3%       2927 ±  3%  slabinfo.blkdev_ioc.active_objs
      2394           +22.3%       2927 ±  3%  slabinfo.blkdev_ioc.num_objs
      7778 ± 14%    +404.2%      39220 ±  6%  slabinfo.dmaengine-unmap-16.active_objs
    185.00 ± 14%    +407.0%     938.00 ±  6%  slabinfo.dmaengine-unmap-16.active_slabs
      7785 ± 14%    +406.2%      39415 ±  6%  slabinfo.dmaengine-unmap-16.num_objs
    185.00 ± 14%    +407.0%     938.00 ±  6%  slabinfo.dmaengine-unmap-16.num_slabs
      3031           +23.0%       3727        slabinfo.ip6-frags.active_objs
      3031           +23.0%       3727        slabinfo.ip6-frags.num_objs
      5362          +185.6%      15313 ±  6%  slabinfo.kmalloc-128.active_objs
    167.00          +188.8%     482.25 ±  6%  slabinfo.kmalloc-128.active_slabs
      5362          +187.8%      15435 ±  6%  slabinfo.kmalloc-128.num_objs
    167.00          +188.8%     482.25 ±  6%  slabinfo.kmalloc-128.num_slabs
     36840           +68.8%      62196 ±  4%  slabinfo.kmalloc-16.active_objs
     37529           +66.2%      62359 ±  4%  slabinfo.kmalloc-16.num_objs
     11694 ±  5%    +124.3%      26230 ± 10%  slabinfo.kmalloc-192.active_objs
    279.25 ±  5%    +126.3%     632.00 ± 10%  slabinfo.kmalloc-192.active_slabs
     11750 ±  5%    +126.1%      26569 ± 10%  slabinfo.kmalloc-192.num_objs
    279.25 ±  5%    +126.3%     632.00 ± 10%  slabinfo.kmalloc-192.num_slabs
     10537 ±  2%     +50.1%      15818 ±  3%  slabinfo.kmalloc-1k.active_objs
    329.75 ±  2%     +51.8%     500.50 ±  3%  slabinfo.kmalloc-1k.active_slabs
     10570 ±  2%     +51.7%      16032 ±  3%  slabinfo.kmalloc-1k.num_objs
    329.75 ±  2%     +51.8%     500.50 ±  3%  slabinfo.kmalloc-1k.num_slabs
      3919 ±  3%     +17.1%       4589 ±  3%  slabinfo.kmalloc-4k.active_objs
    491.00 ±  3%     +17.7%     578.00 ±  3%  slabinfo.kmalloc-4k.active_slabs
      3930 ±  3%     +17.7%       4627 ±  3%  slabinfo.kmalloc-4k.num_objs
    491.00 ±  3%     +17.7%     578.00 ±  3%  slabinfo.kmalloc-4k.num_slabs
     17209           +42.2%      24479        slabinfo.kmalloc-512.active_objs
    550.75           +50.2%     827.25        slabinfo.kmalloc-512.active_slabs
     17639           +50.2%      26490        slabinfo.kmalloc-512.num_objs
    550.75           +50.2%     827.25        slabinfo.kmalloc-512.num_slabs
     62142           +12.2%      69723        slabinfo.kmalloc-64.active_objs
    972.75           +12.1%       1090        slabinfo.kmalloc-64.active_slabs
     62303           +12.0%      69808        slabinfo.kmalloc-64.num_objs
    972.75           +12.1%       1090        slabinfo.kmalloc-64.num_slabs
    975.25           +28.0%       1248        slabinfo.kmalloc-8k.active_objs
    987.25           +30.7%       1290        slabinfo.kmalloc-8k.num_objs
      3780 ±  3%     -10.1%       3398 ±  6%  slabinfo.kmalloc-rcl-96.active_objs
      2810           +10.8%       3113        slabinfo.names_cache.active_objs
      2834           +10.9%       3143        slabinfo.names_cache.num_objs
      4673           +37.1%       6407        slabinfo.numa_policy.active_objs
      4673           +37.1%       6407        slabinfo.numa_policy.num_objs
      2729          +128.8%       6244 ±  8%  slabinfo.pool_workqueue.active_objs
     85.00          +138.2%     202.50 ±  8%  slabinfo.pool_workqueue.active_slabs
      2733          +137.8%       6500 ±  8%  slabinfo.pool_workqueue.num_objs
     85.00          +138.2%     202.50 ±  8%  slabinfo.pool_workqueue.num_slabs
     45407          +102.9%      92111 ±  2%  slabinfo.radix_tree_node.active_objs
    812.00          +102.8%       1647 ±  2%  slabinfo.radix_tree_node.active_slabs
     45516          +102.7%      92258 ±  2%  slabinfo.radix_tree_node.num_objs
    812.00          +102.8%       1647 ±  2%  slabinfo.radix_tree_node.num_slabs
      2481           +22.8%       3047        slabinfo.xfs_btree_cur.active_objs
      2481           +22.8%       3047        slabinfo.xfs_btree_cur.num_objs
      2526           +39.9%       3533        slabinfo.xfs_buf.active_objs
      2526           +39.9%       3533        slabinfo.xfs_buf.num_objs
      2295           +24.7%       2861        slabinfo.xfs_da_state.active_objs
      2295           +24.7%       2861        slabinfo.xfs_da_state.num_objs
      2177          +251.8%       7658 ±  5%  slabinfo.xfs_efd_item.active_objs
     58.25          +259.2%     209.25 ±  5%  slabinfo.xfs_efd_item.active_slabs
      2177          +256.8%       7767 ±  5%  slabinfo.xfs_efd_item.num_objs
     58.25          +259.2%     209.25 ±  5%  slabinfo.xfs_efd_item.num_slabs
      2235          +248.0%       7781 ±  4%  slabinfo.xfs_efi_item.active_objs
     58.25          +255.8%     207.25 ±  4%  slabinfo.xfs_efi_item.active_slabs
      2235          +253.2%       7897 ±  4%  slabinfo.xfs_efi_item.num_objs
     58.25          +255.8%     207.25 ±  4%  slabinfo.xfs_efi_item.num_slabs
      1949 ±  7%     +97.2%       3843        slabinfo.xfs_icr.active_objs
      1949 ±  7%     +97.2%       3843        slabinfo.xfs_icr.num_objs
      2893          +327.3%      12362        slabinfo.xfs_ili.active_objs
     68.25          +355.3%     310.75        slabinfo.xfs_ili.active_slabs
      2893          +352.1%      13080        slabinfo.xfs_ili.num_objs
     68.25          +355.3%     310.75        slabinfo.xfs_ili.num_slabs
      2350 ±  3%    +380.8%      11301        slabinfo.xfs_inode.active_objs
     73.50 ±  4%    +420.1%     382.25        slabinfo.xfs_inode.active_slabs
      2370 ±  4%    +416.9%      12249        slabinfo.xfs_inode.num_objs
     73.50 ±  4%    +420.1%     382.25        slabinfo.xfs_inode.num_slabs
     11582 ±  3%     +37.5%      15930 ±  9%  softirqs.CPU0.SCHED
     28207 ±  7%     +77.2%      49992 ± 11%  softirqs.CPU0.TIMER
      9700 ±  8%     +36.8%      13273 ±  5%  softirqs.CPU1.SCHED
     23666 ±  4%     +84.9%      43769 ±  5%  softirqs.CPU1.TIMER
      8200 ±  3%     +53.5%      12585 ±  3%  softirqs.CPU10.SCHED
     21183 ±  7%    +113.4%      45197 ±  7%  softirqs.CPU10.TIMER
      8085 ±  5%     +53.9%      12440        softirqs.CPU11.SCHED
     20224 ±  4%    +111.7%      42817 ±  6%  softirqs.CPU11.TIMER
      8177 ±  5%     +52.5%      12469 ±  2%  softirqs.CPU12.SCHED
     20581 ±  6%    +120.7%      45420 ±  4%  softirqs.CPU12.TIMER
      8247 ±  4%     +50.3%      12396 ±  2%  softirqs.CPU13.SCHED
     20701 ±  5%    +109.8%      43425 ±  6%  softirqs.CPU13.TIMER
      8340 ±  2%     +21.9%      10164 ± 17%  softirqs.CPU14.RCU
      8397 ±  7%     +49.4%      12545        softirqs.CPU14.SCHED
     23761 ± 20%    +106.2%      49004 ± 13%  softirqs.CPU14.TIMER
      8093           +51.2%      12239 ±  3%  softirqs.CPU15.SCHED
     22218 ± 16%     +90.8%      42383 ±  6%  softirqs.CPU15.TIMER
      8006 ±  4%     +57.9%      12641 ±  4%  softirqs.CPU16.SCHED
     21472 ±  4%    +104.9%      43998 ±  6%  softirqs.CPU16.TIMER
      8047 ±  6%     +55.7%      12532 ±  2%  softirqs.CPU17.SCHED
     20627 ±  4%    +110.7%      43457 ±  2%  softirqs.CPU17.TIMER
      8209           +46.8%      12048 ±  2%  softirqs.CPU18.SCHED
     21558 ±  6%     +94.7%      41975 ±  6%  softirqs.CPU18.TIMER
      8263 ±  2%     +50.4%      12425        softirqs.CPU19.SCHED
     20683 ±  2%    +107.5%      42921 ±  6%  softirqs.CPU19.TIMER
      8637 ±  7%     +48.7%      12845 ±  5%  softirqs.CPU2.SCHED
     21793 ±  9%    +101.6%      43943 ±  4%  softirqs.CPU2.TIMER
      8185 ±  4%     +48.7%      12169 ±  3%  softirqs.CPU20.SCHED
     20336 ±  5%    +110.6%      42832 ±  2%  softirqs.CPU20.TIMER
      8192 ±  6%     +47.9%      12119 ±  4%  softirqs.CPU21.SCHED
     22062 ±  5%     +92.3%      42430 ±  4%  softirqs.CPU21.TIMER
      8108 ±  2%     +45.4%      11790 ±  5%  softirqs.CPU22.SCHED
     22000 ± 13%     +94.5%      42798 ±  4%  softirqs.CPU22.TIMER
      8202 ±  2%     +49.7%      12279 ±  2%  softirqs.CPU23.SCHED
     21431 ±  8%     +97.6%      42343 ±  6%  softirqs.CPU23.TIMER
      8190 ±  4%     +49.4%      12234 ± 12%  softirqs.CPU24.SCHED
     22902 ± 11%     +84.1%      42172 ± 14%  softirqs.CPU24.TIMER
      8201 ±  3%     +50.2%      12321 ±  4%  softirqs.CPU25.SCHED
     22214 ± 13%    +123.8%      49709 ± 20%  softirqs.CPU25.TIMER
      7901 ±  5%     +56.6%      12376 ±  6%  softirqs.CPU26.SCHED
     19527 ±  5%    +110.2%      41047 ± 11%  softirqs.CPU26.TIMER
      8531 ± 11%     +40.6%      11994 ±  7%  softirqs.CPU27.SCHED
     21020 ±  9%     +90.9%      40134 ± 13%  softirqs.CPU27.TIMER
      9415 ± 11%     +27.7%      12027 ±  4%  softirqs.CPU28.SCHED
     22888 ± 10%     +75.0%      40046 ± 11%  softirqs.CPU28.TIMER
      8290 ±  4%     +50.4%      12472 ±  5%  softirqs.CPU29.SCHED
     21969 ± 12%     +85.9%      40836 ± 11%  softirqs.CPU29.TIMER
      8493 ±  7%     +43.2%      12166 ±  4%  softirqs.CPU3.SCHED
     22469 ±  8%     +97.0%      44258 ±  3%  softirqs.CPU3.TIMER
      8250 ±  5%     +53.6%      12674 ±  7%  softirqs.CPU30.SCHED
     20543 ±  4%     +99.5%      40978 ± 12%  softirqs.CPU30.TIMER
      7881 ±  5%     +56.9%      12369 ± 12%  softirqs.CPU31.SCHED
     20034 ±  2%     +98.8%      39830 ± 16%  softirqs.CPU31.TIMER
      7606 ±  6%     +55.4%      11819 ±  4%  softirqs.CPU32.SCHED
     19884 ±  3%     +95.0%      38777 ± 13%  softirqs.CPU32.TIMER
      8133 ±  4%     +45.1%      11799 ±  5%  softirqs.CPU33.SCHED
     20245 ±  4%     +97.4%      39954 ± 11%  softirqs.CPU33.TIMER
      8113 ±  3%     +22.7%       9955 ± 16%  softirqs.CPU34.RCU
      8314 ±  6%     +44.3%      11999 ±  5%  softirqs.CPU34.SCHED
     20855 ±  9%     +94.3%      40531 ± 12%  softirqs.CPU34.TIMER
      8256 ±  4%     +42.6%      11773 ±  5%  softirqs.CPU35.SCHED
     20666 ±  5%     +92.7%      39833 ± 11%  softirqs.CPU35.TIMER
      8086 ±  3%     +24.5%      10067 ± 19%  softirqs.CPU36.RCU
      7906 ±  4%     +47.8%      11685 ±  6%  softirqs.CPU36.SCHED
     20112 ±  3%     +96.6%      39544 ± 12%  softirqs.CPU36.TIMER
      8251 ±  6%     +41.6%      11684 ±  6%  softirqs.CPU37.SCHED
     21526 ± 15%     +81.2%      39010 ± 12%  softirqs.CPU37.TIMER
      8518 ±  8%     +36.0%      11586 ±  3%  softirqs.CPU38.SCHED
     20729 ±  8%     +95.5%      40521 ± 12%  softirqs.CPU38.TIMER
      8113 ±  4%     +46.5%      11885 ±  3%  softirqs.CPU39.SCHED
     21158 ±  7%     +87.0%      39564 ± 11%  softirqs.CPU39.TIMER
      9224 ± 10%     +35.9%      12535 ±  3%  softirqs.CPU4.SCHED
     21374 ±  6%    +108.3%      44518        softirqs.CPU4.TIMER
      8081 ±  2%     +46.2%      11816 ±  3%  softirqs.CPU40.SCHED
     21415 ± 15%     +86.1%      39849 ± 12%  softirqs.CPU40.TIMER
      8124 ±  3%     +48.5%      12067 ±  3%  softirqs.CPU41.SCHED
     21063 ± 10%     +89.0%      39800 ± 12%  softirqs.CPU41.TIMER
      8299 ±  6%     +43.8%      11936 ±  5%  softirqs.CPU42.SCHED
     20874 ±  4%     +96.5%      41022 ± 11%  softirqs.CPU42.TIMER
      8125 ±  2%     +42.6%      11587 ±  5%  softirqs.CPU43.SCHED
     19990 ±  3%     +95.2%      39013 ± 13%  softirqs.CPU43.TIMER
      7980 ±  3%     +47.8%      11794 ±  2%  softirqs.CPU44.SCHED
     19649 ±  2%    +109.9%      41241 ± 10%  softirqs.CPU44.TIMER
      8087 ±  2%     +46.1%      11812 ±  6%  softirqs.CPU45.SCHED
     19744 ±  2%     +97.3%      38947 ± 14%  softirqs.CPU45.TIMER
      8212 ±  2%     +46.0%      11989 ±  4%  softirqs.CPU46.SCHED
     20123 ±  2%     +95.8%      39403 ± 12%  softirqs.CPU46.TIMER
      7452 ±  2%     +60.5%      11963 ±  3%  softirqs.CPU47.SCHED
     21314 ±  3%     +86.0%      39652 ± 13%  softirqs.CPU47.TIMER
      8150 ±  2%     +44.2%      11755        softirqs.CPU48.SCHED
     21125 ±  8%     +98.3%      41889 ±  6%  softirqs.CPU48.TIMER
      8209 ±  4%     +46.6%      12032        softirqs.CPU49.SCHED
     20667 ±  6%     +99.9%      41310 ±  7%  softirqs.CPU49.TIMER
      8403 ±  3%     +48.7%      12492        softirqs.CPU5.SCHED
     21700 ±  5%     +99.0%      43194 ±  4%  softirqs.CPU5.TIMER
      8404 ±  2%     +35.5%      11390 ±  8%  softirqs.CPU50.SCHED
     21892 ±  5%     +89.6%      41518 ±  6%  softirqs.CPU50.TIMER
      8323 ±  6%     +49.4%      12435 ±  2%  softirqs.CPU51.SCHED
     20156 ±  5%    +112.1%      42749 ±  2%  softirqs.CPU51.TIMER
      8042 ±  3%     +48.5%      11944        softirqs.CPU52.SCHED
     20056          +114.9%      43101        softirqs.CPU52.TIMER
      8380 ±  2%     +47.4%      12351 ±  2%  softirqs.CPU53.SCHED
     20236 ±  2%    +109.2%      42341 ±  3%  softirqs.CPU53.TIMER
      8333           +47.0%      12252 ±  4%  softirqs.CPU54.SCHED
     20002 ±  3%    +109.9%      41988 ±  6%  softirqs.CPU54.TIMER
      8372 ±  8%     +22.1%      10224 ± 15%  softirqs.CPU55.RCU
      7962 ±  3%     +51.3%      12046 ±  5%  softirqs.CPU55.SCHED
     20810 ±  8%    +119.6%      45697 ± 12%  softirqs.CPU55.TIMER
      8379 ±  5%     +50.7%      12624        softirqs.CPU56.SCHED
     20785 ±  2%    +110.3%      43720 ±  4%  softirqs.CPU56.TIMER
      8297 ±  3%     +46.5%      12155 ±  3%  softirqs.CPU57.SCHED
     20375 ±  4%    +108.6%      42507 ±  7%  softirqs.CPU57.TIMER
      8124 ±  4%     +22.6%       9957 ± 16%  softirqs.CPU58.RCU
      8507 ±  4%     +50.3%      12785 ±  3%  softirqs.CPU58.SCHED
     20398 ±  6%    +112.2%      43286 ±  6%  softirqs.CPU58.TIMER
      9276 ±  9%     +34.6%      12488        softirqs.CPU59.SCHED
     23703 ± 15%     +80.2%      42708 ±  7%  softirqs.CPU59.TIMER
      8148 ±  3%     +55.1%      12642 ±  3%  softirqs.CPU6.SCHED
     20641 ±  4%    +110.6%      43469 ±  7%  softirqs.CPU6.TIMER
      8504 ±  4%     +47.2%      12522 ±  2%  softirqs.CPU60.SCHED
     21149 ± 11%    +100.3%      42360 ±  7%  softirqs.CPU60.TIMER
      8607 ±  6%     +41.7%      12200        softirqs.CPU61.SCHED
     21138 ± 10%     +96.5%      41545 ±  6%  softirqs.CPU61.TIMER
      8649 ±  8%     +39.3%      12045        softirqs.CPU62.SCHED
     20907 ± 10%    +101.7%      42170 ±  4%  softirqs.CPU62.TIMER
      8762 ±  7%     +35.5%      11871 ±  6%  softirqs.CPU63.SCHED
     21089 ±  4%     +94.3%      40969 ±  7%  softirqs.CPU63.TIMER
      8377 ±  3%     +47.3%      12336 ±  4%  softirqs.CPU64.SCHED
     23172 ± 22%     +87.5%      43441 ±  7%  softirqs.CPU64.TIMER
      8649 ±  4%     +43.7%      12427 ±  3%  softirqs.CPU65.SCHED
     20967 ±  7%    +103.4%      42644 ±  4%  softirqs.CPU65.TIMER
      8414 ±  2%     +47.1%      12377 ±  3%  softirqs.CPU66.SCHED
     20248 ±  4%    +106.2%      41746 ±  6%  softirqs.CPU66.TIMER
      8620 ±  3%     +40.1%      12080 ± 12%  softirqs.CPU67.SCHED
     22315 ± 13%     +89.3%      42245 ±  7%  softirqs.CPU67.TIMER
      8339 ±  3%     +46.6%      12226 ±  2%  softirqs.CPU68.SCHED
     22152 ± 13%     +90.3%      42163 ±  2%  softirqs.CPU68.TIMER
      8371 ±  2%     +47.6%      12358 ±  3%  softirqs.CPU69.SCHED
     20011 ±  4%    +110.5%      42125 ±  5%  softirqs.CPU69.TIMER
      7969 ±  3%     +54.5%      12310 ±  5%  softirqs.CPU7.SCHED
     20424 ±  4%    +108.5%      42588 ±  5%  softirqs.CPU7.TIMER
      8230           +47.9%      12173 ±  2%  softirqs.CPU70.SCHED
     21733 ±  6%     +93.7%      42106 ±  5%  softirqs.CPU70.TIMER
      8626           +42.1%      12260 ±  2%  softirqs.CPU71.SCHED
     21308 ±  2%     +99.9%      42603 ±  8%  softirqs.CPU71.TIMER
      8329           +50.2%      12514 ±  6%  softirqs.CPU72.SCHED
     19760          +111.9%      41877 ± 16%  softirqs.CPU72.TIMER
      8301 ±  5%     +39.3%      11566 ±  3%  softirqs.CPU73.SCHED
     19630 ±  6%     +93.6%      38003 ± 12%  softirqs.CPU73.TIMER
      8218 ±  6%     +52.3%      12514 ±  9%  softirqs.CPU74.SCHED
     19504 ±  7%    +119.3%      42774 ± 18%  softirqs.CPU74.TIMER
      8389 ±  4%     +44.7%      12135 ±  8%  softirqs.CPU75.SCHED
     20008 ±  4%    +105.9%      41190 ± 12%  softirqs.CPU75.TIMER
      8407 ±  3%     +45.9%      12270 ±  6%  softirqs.CPU76.SCHED
     20019 ±  4%    +104.0%      40838 ± 11%  softirqs.CPU76.TIMER
      8189 ±  2%     +46.8%      12024 ±  4%  softirqs.CPU77.SCHED
     19972 ±  3%    +100.1%      39965 ± 11%  softirqs.CPU77.TIMER
      8420 ±  4%     +54.7%      13024 ± 14%  softirqs.CPU78.SCHED
     21232 ± 15%    +100.8%      42630 ± 22%  softirqs.CPU78.TIMER
      7766 ±  3%     +51.5%      11766 ±  2%  softirqs.CPU79.SCHED
     19702 ±  3%     +98.8%      39161 ± 13%  softirqs.CPU79.TIMER
      8063 ±  4%     +55.3%      12524        softirqs.CPU8.SCHED
     20660 ±  5%    +112.0%      43790 ±  4%  softirqs.CPU8.TIMER
      8035 ±  3%     +48.3%      11914 ±  4%  softirqs.CPU80.SCHED
     19495 ±  2%    +100.6%      39100 ± 13%  softirqs.CPU80.TIMER
      8092 ±  3%     +54.6%      12508 ±  7%  softirqs.CPU81.SCHED
     19282          +114.3%      41318 ± 15%  softirqs.CPU81.TIMER
      8285 ±  3%     +43.5%      11890 ±  2%  softirqs.CPU82.SCHED
     19583 ±  3%     +99.1%      38989 ± 13%  softirqs.CPU82.TIMER
      7950 ±  3%     +58.1%      12569 ±  4%  softirqs.CPU83.SCHED
     19265          +112.0%      40833 ±  9%  softirqs.CPU83.TIMER
      8199 ±  2%     +50.7%      12355 ±  3%  softirqs.CPU84.SCHED
     19486          +106.3%      40203 ± 11%  softirqs.CPU84.TIMER
      8322 ±  4%     +44.3%      12010 ±  4%  softirqs.CPU85.SCHED
     19593 ±  5%    +100.1%      39201 ± 12%  softirqs.CPU85.TIMER
      8346 ±  3%     +45.9%      12180 ±  3%  softirqs.CPU86.SCHED
     19652 ±  3%    +104.6%      40204 ± 12%  softirqs.CPU86.TIMER
      8521 ±  2%     +42.3%      12125 ±  2%  softirqs.CPU87.SCHED
     20622 ±  4%     +92.9%      39776 ± 11%  softirqs.CPU87.TIMER
      8120 ±  2%     +52.5%      12384 ± 10%  softirqs.CPU88.SCHED
     19132          +111.0%      40366 ± 18%  softirqs.CPU88.TIMER
      8228 ±  3%     +50.5%      12380 ±  3%  softirqs.CPU89.SCHED
     19433          +111.1%      41022 ± 10%  softirqs.CPU89.TIMER
      8308 ±  4%     +21.2%      10071 ± 18%  softirqs.CPU9.RCU
      8562 ±  4%     +43.4%      12276 ±  3%  softirqs.CPU9.SCHED
     21944 ±  7%     +99.0%      43659 ±  6%  softirqs.CPU9.TIMER
      7960 ±  2%     +50.2%      11958 ±  6%  softirqs.CPU90.SCHED
     19365 ±  3%    +105.6%      39806 ± 12%  softirqs.CPU90.TIMER
      8003 ±  2%     +51.5%      12124 ±  5%  softirqs.CPU91.SCHED
     19058 ±  3%    +111.5%      40302 ± 16%  softirqs.CPU91.TIMER
      8404 ±  2%     +41.9%      11924 ±  4%  softirqs.CPU92.SCHED
     19885          +105.0%      40761 ± 11%  softirqs.CPU92.TIMER
      8261 ±  2%     +45.4%      12014 ±  3%  softirqs.CPU93.SCHED
     19429          +122.1%      43144 ± 15%  softirqs.CPU93.TIMER
      8249 ±  2%     +46.5%      12087 ±  6%  softirqs.CPU94.SCHED
     19385          +110.4%      40790 ± 13%  softirqs.CPU94.TIMER
      8204 ±  3%     +52.5%      12514 ±  6%  softirqs.CPU95.SCHED
     19992 ±  2%    +101.2%      40217 ± 14%  softirqs.CPU95.TIMER
      1418 ± 44%    +271.5%       5269 ± 67%  softirqs.NET_RX
    798980 ±  2%     +47.1%    1175269 ±  2%  softirqs.SCHED
   2001097 ±  2%    +100.7%    4017052 ±  3%  softirqs.TIMER
     38.00 ± 52%   +1536.2%     621.75 ± 94%  interrupts.66:PCI-MSI.70258690-edge.eth2-TxRx-1
     43.75 ± 52%   +3383.4%       1524 ±156%  interrupts.70:PCI-MSI.70258694-edge.eth2-TxRx-5
     80.00 ±  5%    +144.4%     195.50 ±  4%  interrupts.9:IO-APIC.9-fasteoi.acpi
     80170 ±  3%    +136.1%     189307 ±  9%  interrupts.CPU0.LOC:Local_timer_interrupts
     94.75 ± 15%     +81.3%     171.75 ±  4%  interrupts.CPU0.RES:Rescheduling_interrupts
     80.00 ±  5%    +144.4%     195.50 ±  4%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
     80085 ±  3%    +136.6%     189442 ±  9%  interrupts.CPU1.LOC:Local_timer_interrupts
     67.50 ± 10%    +113.0%     143.75 ± 19%  interrupts.CPU1.RES:Rescheduling_interrupts
     80204 ±  3%    +135.9%     189194 ±  9%  interrupts.CPU10.LOC:Local_timer_interrupts
      3490 ±  8%     -33.4%       2323 ± 27%  interrupts.CPU10.NMI:Non-maskable_interrupts
      3490 ±  8%     -33.4%       2323 ± 27%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
     66.25 ± 20%    +112.5%     140.75 ± 15%  interrupts.CPU10.RES:Rescheduling_interrupts
     80179 ±  3%    +136.0%     189229 ±  9%  interrupts.CPU11.LOC:Local_timer_interrupts
      3444 ±  7%     -43.0%       1964 ± 34%  interrupts.CPU11.NMI:Non-maskable_interrupts
      3444 ±  7%     -43.0%       1964 ± 34%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
     69.75 ± 23%    +105.7%     143.50 ± 11%  interrupts.CPU11.RES:Rescheduling_interrupts
     80186 ±  3%    +136.0%     189270 ±  9%  interrupts.CPU12.LOC:Local_timer_interrupts
      3533 ±  5%     -44.9%       1948 ± 32%  interrupts.CPU12.NMI:Non-maskable_interrupts
      3533 ±  5%     -44.9%       1948 ± 32%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
     64.25 ± 14%    +103.9%     131.00 ± 14%  interrupts.CPU12.RES:Rescheduling_interrupts
     80015 ±  3%    +136.7%     189408 ±  9%  interrupts.CPU13.LOC:Local_timer_interrupts
      3535 ± 12%     -44.9%       1946 ± 31%  interrupts.CPU13.NMI:Non-maskable_interrupts
      3535 ± 12%     -44.9%       1946 ± 31%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
     73.50 ±  8%     +81.0%     133.00 ± 15%  interrupts.CPU13.RES:Rescheduling_interrupts
     80179 ±  3%    +136.1%     189269 ±  9%  interrupts.CPU14.LOC:Local_timer_interrupts
      3493 ±  5%     -34.1%       2303 ± 27%  interrupts.CPU14.NMI:Non-maskable_interrupts
      3493 ±  5%     -34.1%       2303 ± 27%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
     70.00 ±  9%     +73.6%     121.50 ± 11%  interrupts.CPU14.RES:Rescheduling_interrupts
     80177 ±  3%    +136.0%     189210 ±  9%  interrupts.CPU15.LOC:Local_timer_interrupts
      3534 ±  8%     -35.6%       2275 ± 27%  interrupts.CPU15.NMI:Non-maskable_interrupts
      3534 ±  8%     -35.6%       2275 ± 27%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
     64.00 ±  8%    +119.1%     140.25 ± 20%  interrupts.CPU15.RES:Rescheduling_interrupts
     80182 ±  3%    +135.6%     188929 ±  9%  interrupts.CPU16.LOC:Local_timer_interrupts
      3601 ±  4%     -35.2%       2334 ± 28%  interrupts.CPU16.NMI:Non-maskable_interrupts
      3601 ±  4%     -35.2%       2334 ± 28%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
     79.50 ± 24%     +78.6%     142.00 ± 21%  interrupts.CPU16.RES:Rescheduling_interrupts
     80174 ±  3%    +136.0%     189214 ±  9%  interrupts.CPU17.LOC:Local_timer_interrupts
      3471 ± 10%     -22.0%       2708 ± 14%  interrupts.CPU17.NMI:Non-maskable_interrupts
      3471 ± 10%     -22.0%       2708 ± 14%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
     65.00 ± 12%    +102.7%     131.75 ± 13%  interrupts.CPU17.RES:Rescheduling_interrupts
     80173 ±  3%    +136.1%     189266 ±  9%  interrupts.CPU18.LOC:Local_timer_interrupts
     63.50 ± 19%    +122.4%     141.25 ± 26%  interrupts.CPU18.RES:Rescheduling_interrupts
     80181 ±  3%    +135.9%     189145 ±  9%  interrupts.CPU19.LOC:Local_timer_interrupts
     69.25 ± 16%     +99.6%     138.25 ± 15%  interrupts.CPU19.RES:Rescheduling_interrupts
     79914 ±  3%    +136.8%     189252 ±  9%  interrupts.CPU2.LOC:Local_timer_interrupts
     73.25 ± 17%     +90.4%     139.50 ±  5%  interrupts.CPU2.RES:Rescheduling_interrupts
    626.00 ±  9%     -11.2%     555.75 ±  3%  interrupts.CPU20.CAL:Function_call_interrupts
     80176 ±  3%    +135.9%     189157 ±  9%  interrupts.CPU20.LOC:Local_timer_interrupts
      3403 ± 10%     -20.7%       2699 ± 15%  interrupts.CPU20.NMI:Non-maskable_interrupts
      3403 ± 10%     -20.7%       2699 ± 15%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
     70.50 ± 18%     +72.3%     121.50 ± 15%  interrupts.CPU20.RES:Rescheduling_interrupts
     80187 ±  3%    +135.8%     189119 ±  9%  interrupts.CPU21.LOC:Local_timer_interrupts
      3519 ± 10%     -24.0%       2675 ± 15%  interrupts.CPU21.NMI:Non-maskable_interrupts
      3519 ± 10%     -24.0%       2675 ± 15%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
     75.75 ±  9%     +88.4%     142.75 ± 15%  interrupts.CPU21.RES:Rescheduling_interrupts
     80187 ±  3%    +136.1%     189307 ±  9%  interrupts.CPU22.LOC:Local_timer_interrupts
     58.75 ± 16%    +160.0%     152.75 ± 17%  interrupts.CPU22.RES:Rescheduling_interrupts
    612.75 ±  6%     -11.3%     543.50 ±  4%  interrupts.CPU23.CAL:Function_call_interrupts
     80181 ±  3%    +136.0%     189253 ±  9%  interrupts.CPU23.LOC:Local_timer_interrupts
     72.25 ± 11%     +69.9%     122.75 ± 11%  interrupts.CPU23.RES:Rescheduling_interrupts
     80144 ±  3%    +134.4%     187839 ± 10%  interrupts.CPU24.LOC:Local_timer_interrupts
     78.75 ± 16%     +81.9%     143.25 ±  8%  interrupts.CPU24.RES:Rescheduling_interrupts
     80147 ±  3%    +134.5%     187973 ± 10%  interrupts.CPU25.LOC:Local_timer_interrupts
     75.50 ± 13%     +97.7%     149.25 ± 11%  interrupts.CPU25.RES:Rescheduling_interrupts
     80190 ±  3%    +134.3%     187926 ± 10%  interrupts.CPU26.LOC:Local_timer_interrupts
     73.00 ± 24%    +127.1%     165.75 ± 10%  interrupts.CPU26.RES:Rescheduling_interrupts
     80207 ±  3%    +134.4%     187967 ± 10%  interrupts.CPU27.LOC:Local_timer_interrupts
     79.25 ± 16%    +132.5%     184.25 ±  5%  interrupts.CPU27.RES:Rescheduling_interrupts
     80167 ±  3%    +134.4%     187921 ± 10%  interrupts.CPU28.LOC:Local_timer_interrupts
     89.50 ± 28%     +56.1%     139.75 ± 12%  interrupts.CPU28.RES:Rescheduling_interrupts
     80472 ±  3%    +133.6%     187951 ± 10%  interrupts.CPU29.LOC:Local_timer_interrupts
     70.00 ± 22%    +122.1%     155.50 ± 17%  interrupts.CPU29.RES:Rescheduling_interrupts
     80163 ±  3%    +136.1%     189268 ±  9%  interrupts.CPU3.LOC:Local_timer_interrupts
     80.25 ± 28%     +71.3%     137.50 ±  4%  interrupts.CPU3.RES:Rescheduling_interrupts
     80362 ±  3%    +133.9%     187984 ± 10%  interrupts.CPU30.LOC:Local_timer_interrupts
     75.75 ± 19%    +161.1%     197.75 ± 15%  interrupts.CPU30.RES:Rescheduling_interrupts
     80155 ±  3%    +134.6%     188037 ± 10%  interrupts.CPU31.LOC:Local_timer_interrupts
     68.00 ± 31%    +105.5%     139.75 ±  8%  interrupts.CPU31.RES:Rescheduling_interrupts
     80125 ±  3%    +134.7%     188049 ± 10%  interrupts.CPU32.LOC:Local_timer_interrupts
     72.50 ± 11%    +123.8%     162.25 ± 16%  interrupts.CPU32.RES:Rescheduling_interrupts
     80183 ±  3%    +134.5%     187996 ± 10%  interrupts.CPU33.LOC:Local_timer_interrupts
     61.25 ± 17%    +185.3%     174.75 ±  8%  interrupts.CPU33.RES:Rescheduling_interrupts
     80174 ±  3%    +134.4%     187915 ± 10%  interrupts.CPU34.LOC:Local_timer_interrupts
     73.75 ± 25%    +110.5%     155.25 ± 12%  interrupts.CPU34.RES:Rescheduling_interrupts
     80174 ±  3%    +134.5%     187978 ± 10%  interrupts.CPU35.LOC:Local_timer_interrupts
     78.25 ± 21%     +92.0%     150.25 ± 15%  interrupts.CPU35.RES:Rescheduling_interrupts
     80170 ±  3%    +134.5%     188038 ± 10%  interrupts.CPU36.LOC:Local_timer_interrupts
     63.00 ± 16%    +154.0%     160.00 ±  5%  interrupts.CPU36.RES:Rescheduling_interrupts
     80170 ±  3%    +134.5%     187964 ± 10%  interrupts.CPU37.LOC:Local_timer_interrupts
     60.75 ± 11%    +152.3%     153.25 ± 10%  interrupts.CPU37.RES:Rescheduling_interrupts
     80175 ±  3%    +134.4%     187906 ± 10%  interrupts.CPU38.LOC:Local_timer_interrupts
     73.50 ±  6%     +98.6%     146.00 ± 16%  interrupts.CPU38.RES:Rescheduling_interrupts
     80175 ±  3%    +134.4%     187950 ± 10%  interrupts.CPU39.LOC:Local_timer_interrupts
     68.25 ± 16%    +164.8%     180.75 ± 16%  interrupts.CPU39.RES:Rescheduling_interrupts
     80173 ±  3%    +136.1%     189270 ±  9%  interrupts.CPU4.LOC:Local_timer_interrupts
     71.25 ± 16%     +87.0%     133.25 ± 22%  interrupts.CPU4.RES:Rescheduling_interrupts
     80172 ±  3%    +134.4%     187890 ± 10%  interrupts.CPU40.LOC:Local_timer_interrupts
     74.75 ± 11%     +81.6%     135.75 ± 14%  interrupts.CPU40.RES:Rescheduling_interrupts
     80169 ±  3%    +134.4%     187895 ± 10%  interrupts.CPU41.LOC:Local_timer_interrupts
     82.75 ± 12%     +88.2%     155.75 ± 12%  interrupts.CPU41.RES:Rescheduling_interrupts
     80152 ±  3%    +134.4%     187871 ± 10%  interrupts.CPU42.LOC:Local_timer_interrupts
     74.25 ± 23%     +95.6%     145.25 ± 12%  interrupts.CPU42.RES:Rescheduling_interrupts
     80164 ±  3%    +134.3%     187825 ± 10%  interrupts.CPU43.LOC:Local_timer_interrupts
     75.75 ± 12%    +102.0%     153.00 ± 12%  interrupts.CPU43.RES:Rescheduling_interrupts
     80179 ±  3%    +134.3%     187874 ± 10%  interrupts.CPU44.LOC:Local_timer_interrupts
     69.25 ± 12%    +136.8%     164.00 ± 13%  interrupts.CPU44.RES:Rescheduling_interrupts
     80167 ±  3%    +134.4%     187906 ± 10%  interrupts.CPU45.LOC:Local_timer_interrupts
     74.00 ± 19%     +99.3%     147.50 ± 10%  interrupts.CPU45.RES:Rescheduling_interrupts
     80179 ±  3%    +134.5%     188035 ± 10%  interrupts.CPU46.LOC:Local_timer_interrupts
     66.75 ± 17%    +147.9%     165.50 ± 34%  interrupts.CPU46.RES:Rescheduling_interrupts
     80178 ±  3%    +134.4%     187902 ± 10%  interrupts.CPU47.LOC:Local_timer_interrupts
     74.75 ± 12%    +106.7%     154.50 ± 19%  interrupts.CPU47.RES:Rescheduling_interrupts
    604.50 ±  4%     -10.3%     542.25        interrupts.CPU48.CAL:Function_call_interrupts
     80175 ±  3%    +136.1%     189256 ±  9%  interrupts.CPU48.LOC:Local_timer_interrupts
     48.75 ± 19%    +134.4%     114.25 ±  8%  interrupts.CPU48.RES:Rescheduling_interrupts
     80159 ±  3%    +136.3%     189404 ±  9%  interrupts.CPU49.LOC:Local_timer_interrupts
     50.25 ± 17%    +118.9%     110.00 ±  9%  interrupts.CPU49.RES:Rescheduling_interrupts
     80040 ±  3%    +136.3%     189132 ±  9%  interrupts.CPU5.LOC:Local_timer_interrupts
     69.50 ±  5%     +88.5%     131.00 ± 15%  interrupts.CPU5.RES:Rescheduling_interrupts
    609.75 ±  5%     -12.4%     534.00        interrupts.CPU50.CAL:Function_call_interrupts
     80168 ±  3%    +136.1%     189308 ±  9%  interrupts.CPU50.LOC:Local_timer_interrupts
     67.75 ± 37%     +86.7%     126.50 ± 11%  interrupts.CPU50.RES:Rescheduling_interrupts
     80159 ±  3%    +136.1%     189287 ±  9%  interrupts.CPU51.LOC:Local_timer_interrupts
     56.75 ± 18%    +123.3%     126.75 ±  9%  interrupts.CPU51.RES:Rescheduling_interrupts
     80176 ±  3%    +136.1%     189273 ±  9%  interrupts.CPU52.LOC:Local_timer_interrupts
     57.25 ± 27%    +127.1%     130.00 ± 13%  interrupts.CPU52.RES:Rescheduling_interrupts
     80182 ±  3%    +136.0%     189230 ±  9%  interrupts.CPU53.LOC:Local_timer_interrupts
     64.75 ± 36%     +94.6%     126.00 ± 10%  interrupts.CPU53.RES:Rescheduling_interrupts
    582.50 ±  3%      -6.5%     544.50 ±  3%  interrupts.CPU54.CAL:Function_call_interrupts
     80176 ±  3%    +136.0%     189229 ±  9%  interrupts.CPU54.LOC:Local_timer_interrupts
     45.00 ± 18%    +163.9%     118.75 ±  4%  interrupts.CPU54.RES:Rescheduling_interrupts
     80144 ±  3%    +136.2%     189307 ±  9%  interrupts.CPU55.LOC:Local_timer_interrupts
     48.75 ± 20%    +160.0%     126.75 ±  4%  interrupts.CPU55.RES:Rescheduling_interrupts
     80165 ±  3%    +136.0%     189197 ±  9%  interrupts.CPU56.LOC:Local_timer_interrupts
     50.75 ±  9%    +165.5%     134.75 ±  8%  interrupts.CPU56.RES:Rescheduling_interrupts
    633.25 ±  4%     -13.8%     546.00 ±  2%  interrupts.CPU57.CAL:Function_call_interrupts
     80156 ±  3%    +135.9%     189101 ±  9%  interrupts.CPU57.LOC:Local_timer_interrupts
     51.00 ±  7%    +141.7%     123.25 ± 12%  interrupts.CPU57.RES:Rescheduling_interrupts
     80175 ±  3%    +136.1%     189305 ±  9%  interrupts.CPU58.LOC:Local_timer_interrupts
     48.50 ± 18%    +158.2%     125.25 ±  5%  interrupts.CPU58.RES:Rescheduling_interrupts
    617.75 ±  7%     -10.4%     553.25 ±  4%  interrupts.CPU59.CAL:Function_call_interrupts
     80180 ±  3%    +136.0%     189231 ±  9%  interrupts.CPU59.LOC:Local_timer_interrupts
     49.00 ±  6%    +174.0%     134.25 ±  9%  interrupts.CPU59.RES:Rescheduling_interrupts
     80171 ±  3%    +136.0%     189170 ±  9%  interrupts.CPU6.LOC:Local_timer_interrupts
     65.00 ±  5%    +107.3%     134.75 ±  6%  interrupts.CPU6.RES:Rescheduling_interrupts
     80175 ±  3%    +136.0%     189241 ±  9%  interrupts.CPU60.LOC:Local_timer_interrupts
     55.00 ± 13%    +142.7%     133.50 ± 16%  interrupts.CPU60.RES:Rescheduling_interrupts
     80180 ±  3%    +136.2%     189393 ±  9%  interrupts.CPU61.LOC:Local_timer_interrupts
     60.25 ±  8%     +99.2%     120.00 ± 12%  interrupts.CPU61.RES:Rescheduling_interrupts
     80177 ±  3%    +136.2%     189391 ±  9%  interrupts.CPU62.LOC:Local_timer_interrupts
     49.50 ± 16%    +142.4%     120.00 ± 11%  interrupts.CPU62.RES:Rescheduling_interrupts
     80161 ±  3%    +136.1%     189225 ±  9%  interrupts.CPU63.LOC:Local_timer_interrupts
     50.75 ±  9%    +166.0%     135.00 ± 10%  interrupts.CPU63.RES:Rescheduling_interrupts
     80063 ±  3%    +136.2%     189086 ±  9%  interrupts.CPU64.LOC:Local_timer_interrupts
     56.25 ± 15%    +158.2%     145.25 ± 13%  interrupts.CPU64.RES:Rescheduling_interrupts
     80174 ±  3%    +136.0%     189209 ±  9%  interrupts.CPU65.LOC:Local_timer_interrupts
     63.25 ± 24%    +113.8%     135.25 ±  9%  interrupts.CPU65.RES:Rescheduling_interrupts
     80174 ±  3%    +136.1%     189270 ±  9%  interrupts.CPU66.LOC:Local_timer_interrupts
     52.00 ± 15%    +190.4%     151.00 ± 13%  interrupts.CPU66.RES:Rescheduling_interrupts
     80170 ±  3%    +135.9%     189145 ±  9%  interrupts.CPU67.LOC:Local_timer_interrupts
     53.50 ± 15%    +156.1%     137.00 ±  9%  interrupts.CPU67.RES:Rescheduling_interrupts
     80175 ±  3%    +135.9%     189147 ±  9%  interrupts.CPU68.LOC:Local_timer_interrupts
     52.75 ±  9%    +169.7%     142.25 ±  7%  interrupts.CPU68.RES:Rescheduling_interrupts
     80176 ±  3%    +135.9%     189121 ±  9%  interrupts.CPU69.LOC:Local_timer_interrupts
     54.00 ± 11%    +134.7%     126.75 ± 15%  interrupts.CPU69.RES:Rescheduling_interrupts
     80070 ±  3%    +136.4%     189314 ±  9%  interrupts.CPU7.LOC:Local_timer_interrupts
     71.00 ±  9%     +94.4%     138.00 ± 12%  interrupts.CPU7.RES:Rescheduling_interrupts
     80184 ±  3%    +136.1%     189285 ±  9%  interrupts.CPU70.LOC:Local_timer_interrupts
     47.50 ± 14%    +212.6%     148.50 ± 11%  interrupts.CPU70.RES:Rescheduling_interrupts
     80183 ±  3%    +136.0%     189248 ±  9%  interrupts.CPU71.LOC:Local_timer_interrupts
     53.25 ±  9%    +151.2%     133.75 ± 12%  interrupts.CPU71.RES:Rescheduling_interrupts
     38.00 ± 52%   +1536.2%     621.75 ± 94%  interrupts.CPU72.66:PCI-MSI.70258690-edge.eth2-TxRx-1
     80179 ±  3%    +134.5%     187983 ± 10%  interrupts.CPU72.LOC:Local_timer_interrupts
     48.75 ±  7%    +255.4%     173.25 ± 19%  interrupts.CPU72.RES:Rescheduling_interrupts
     80190 ±  3%    +134.4%     187965 ± 10%  interrupts.CPU73.LOC:Local_timer_interrupts
     53.75 ± 12%    +192.1%     157.00 ± 11%  interrupts.CPU73.RES:Rescheduling_interrupts
     80183 ±  3%    +134.4%     187947 ± 10%  interrupts.CPU74.LOC:Local_timer_interrupts
     56.75 ± 15%    +151.1%     142.50 ± 11%  interrupts.CPU74.RES:Rescheduling_interrupts
     80190 ±  3%    +134.4%     187954 ± 10%  interrupts.CPU75.LOC:Local_timer_interrupts
     54.50 ±  4%    +176.6%     150.75 ±  4%  interrupts.CPU75.RES:Rescheduling_interrupts
     43.75 ± 52%   +3383.4%       1524 ±156%  interrupts.CPU76.70:PCI-MSI.70258694-edge.eth2-TxRx-5
     80190 ±  3%    +134.5%     188035 ± 10%  interrupts.CPU76.LOC:Local_timer_interrupts
     57.00 ± 26%    +167.1%     152.25 ± 10%  interrupts.CPU76.RES:Rescheduling_interrupts
     80197 ±  3%    +134.4%     187946 ± 10%  interrupts.CPU77.LOC:Local_timer_interrupts
     51.75 ± 12%    +182.1%     146.00 ± 11%  interrupts.CPU77.RES:Rescheduling_interrupts
     80179 ±  3%    +134.5%     188001 ± 10%  interrupts.CPU78.LOC:Local_timer_interrupts
     50.50 ± 11%    +186.1%     144.50 ± 14%  interrupts.CPU78.RES:Rescheduling_interrupts
     80174 ±  3%    +134.6%     188105 ± 10%  interrupts.CPU79.LOC:Local_timer_interrupts
     50.00 ± 19%    +240.5%     170.25 ± 10%  interrupts.CPU79.RES:Rescheduling_interrupts
    704.75 ± 12%     -21.3%     554.50 ±  4%  interrupts.CPU8.CAL:Function_call_interrupts
     80153 ±  3%    +136.1%     189206 ±  9%  interrupts.CPU8.LOC:Local_timer_interrupts
     66.50 ± 18%    +112.0%     141.00 ± 24%  interrupts.CPU8.RES:Rescheduling_interrupts
     80140 ±  3%    +134.7%     188053 ± 10%  interrupts.CPU80.LOC:Local_timer_interrupts
     49.25 ±  8%    +204.1%     149.75 ± 14%  interrupts.CPU80.RES:Rescheduling_interrupts
     80003 ±  3%    +135.0%     187993 ± 10%  interrupts.CPU81.LOC:Local_timer_interrupts
      3680 ±  5%     -22.7%       2843 ± 18%  interrupts.CPU81.NMI:Non-maskable_interrupts
      3680 ±  5%     -22.7%       2843 ± 18%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
     50.00 ±  6%    +225.5%     162.75 ±  7%  interrupts.CPU81.RES:Rescheduling_interrupts
     80135 ±  3%    +134.5%     187932 ± 10%  interrupts.CPU82.LOC:Local_timer_interrupts
      3610 ±  4%     -20.9%       2856 ± 17%  interrupts.CPU82.NMI:Non-maskable_interrupts
      3610 ±  4%     -20.9%       2856 ± 17%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
     59.00 ± 36%    +172.0%     160.50 ± 17%  interrupts.CPU82.RES:Rescheduling_interrupts
     80158 ±  3%    +134.5%     187959 ± 10%  interrupts.CPU83.LOC:Local_timer_interrupts
      3491 ±  9%     -18.5%       2845 ± 15%  interrupts.CPU83.NMI:Non-maskable_interrupts
      3491 ±  9%     -18.5%       2845 ± 15%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
     50.50 ± 15%    +224.3%     163.75 ± 17%  interrupts.CPU83.RES:Rescheduling_interrupts
     80171 ±  3%    +134.5%     188017 ± 10%  interrupts.CPU84.LOC:Local_timer_interrupts
     51.50 ±  9%    +183.5%     146.00 ± 10%  interrupts.CPU84.RES:Rescheduling_interrupts
     80165 ±  3%    +134.5%     187991 ± 10%  interrupts.CPU85.LOC:Local_timer_interrupts
      3494 ±  6%     -18.9%       2834 ± 17%  interrupts.CPU85.NMI:Non-maskable_interrupts
      3494 ±  6%     -18.9%       2834 ± 17%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
     47.75 ± 13%    +214.1%     150.00 ±  7%  interrupts.CPU85.RES:Rescheduling_interrupts
     80170 ±  3%    +134.4%     187922 ± 10%  interrupts.CPU86.LOC:Local_timer_interrupts
     49.50 ± 20%    +187.4%     142.25 ± 16%  interrupts.CPU86.RES:Rescheduling_interrupts
     80167 ±  3%    +134.3%     187809 ± 10%  interrupts.CPU87.LOC:Local_timer_interrupts
     65.50 ± 51%    +164.1%     173.00 ± 13%  interrupts.CPU87.RES:Rescheduling_interrupts
     80174 ±  3%    +134.4%     187903 ± 10%  interrupts.CPU88.LOC:Local_timer_interrupts
     50.75 ± 22%    +188.2%     146.25 ± 13%  interrupts.CPU88.RES:Rescheduling_interrupts
     80177 ±  3%    +134.3%     187874 ± 10%  interrupts.CPU89.LOC:Local_timer_interrupts
     57.00 ± 17%    +154.8%     145.25 ± 10%  interrupts.CPU89.RES:Rescheduling_interrupts
    626.50 ±  7%     -22.6%     484.75 ± 16%  interrupts.CPU9.CAL:Function_call_interrupts
     80172 ±  3%    +135.9%     189123 ±  9%  interrupts.CPU9.LOC:Local_timer_interrupts
      3530 ±  6%     -43.2%       2004 ± 33%  interrupts.CPU9.NMI:Non-maskable_interrupts
      3530 ±  6%     -43.2%       2004 ± 33%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     60.75 ±  6%    +100.4%     121.75 ±  8%  interrupts.CPU9.RES:Rescheduling_interrupts
     80155 ±  3%    +134.4%     187897 ± 10%  interrupts.CPU90.LOC:Local_timer_interrupts
     60.50 ±  7%    +147.5%     149.75 ±  6%  interrupts.CPU90.RES:Rescheduling_interrupts
     80163 ±  3%    +134.3%     187801 ± 10%  interrupts.CPU91.LOC:Local_timer_interrupts
     52.00 ± 15%    +167.3%     139.00 ± 15%  interrupts.CPU91.RES:Rescheduling_interrupts
     80162 ±  3%    +134.5%     187971 ± 10%  interrupts.CPU92.LOC:Local_timer_interrupts
     65.50 ± 32%    +195.0%     193.25 ± 38%  interrupts.CPU92.RES:Rescheduling_interrupts
     80164 ±  3%    +134.4%     187921 ± 10%  interrupts.CPU93.LOC:Local_timer_interrupts
     44.75 ± 21%    +211.2%     139.25 ±  9%  interrupts.CPU93.RES:Rescheduling_interrupts
     80162 ±  3%    +134.3%     187859 ± 10%  interrupts.CPU94.LOC:Local_timer_interrupts
     55.00 ± 23%    +174.1%     150.75 ± 15%  interrupts.CPU94.RES:Rescheduling_interrupts
     80213 ±  3%    +134.2%     187833 ± 10%  interrupts.CPU95.LOC:Local_timer_interrupts
     59.50 ± 23%    +165.5%     158.00 ± 17%  interrupts.CPU95.RES:Rescheduling_interrupts
   7696045 ±  3%    +135.2%   18104643 ±  9%  interrupts.LOC:Local_timer_interrupts
      6012 ±  8%    +131.3%      13905 ±  6%  interrupts.RES:Rescheduling_interrupts
    972.50 ± 35%     -59.5%     393.50 ± 75%  interrupts.TLB:TLB_shootdowns
     57.05 ±  4%     -33.9       23.16 ±  7%  perf-profile.calltrace.cycles-pp.write
     53.49 ±  4%     -31.5       22.02 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     52.95 ±  4%     -31.1       21.80 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     47.90 ±  7%     -28.2       19.73 ±  5%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     46.89 ±  7%     -27.6       19.30 ±  5%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     43.25 ±  7%     -25.6       17.66 ±  5%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     41.81 ±  7%     -24.8       17.02 ±  5%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     35.82 ±  8%     -21.6       14.21 ±  6%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     35.56 ±  8%     -21.5       14.09 ±  6%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     27.88 ± 15%     -19.8        8.09 ± 20%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     26.50 ± 15%     -19.7        6.79 ± 23%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     27.67 ± 15%     -19.7        8.00 ± 20%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     27.67 ± 15%     -19.7        8.00 ± 20%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     27.67 ± 15%     -19.7        8.00 ± 20%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     27.22 ± 15%     -19.6        7.58 ± 21%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     27.26 ± 15%     -19.6        7.67 ± 21%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     28.96 ±  8%     -17.9       11.06 ±  6%  perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
     11.60 ±  9%      -7.1        4.50 ±  6%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
     10.51 ±  9%      -6.9        3.59 ±  6%  perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      7.44 ±  8%      -4.7        2.75 ±  6%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      7.23 ±  8%      -4.6        2.65 ±  6%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      5.92 ±  8%      -4.1        1.85 ±  8%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      5.16 ± 37%      -3.1        2.08 ± 39%  perf-profile.calltrace.cycles-pp.llseek
      4.29 ±  7%      -2.9        1.34 ±  6%  perf-profile.calltrace.cycles-pp.__close
      4.27 ±  7%      -2.9        1.34 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close
      4.27 ±  7%      -2.9        1.34 ±  6%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
      4.25 ±  7%      -2.9        1.33 ±  6%  perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
      4.25 ±  7%      -2.9        1.33 ±  6%  perf-profile.calltrace.cycles-pp.task_work_run.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
      4.24 ±  7%      -2.9        1.33 ±  6%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.11 ±  7%      -2.8        1.28 ±  6%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.__prepare_exit_to_usermode.do_syscall_64
      4.09 ±  7%      -2.8        1.27 ±  6%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.__prepare_exit_to_usermode
      4.26 ±  7%      -2.3        2.00 ±  5%  perf-profile.calltrace.cycles-pp.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      3.30 ±  6%      -2.1        1.25 ±  6%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      3.27 ±  6%      -2.0        1.24 ±  6%  perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.__dentry_kill.dput.__fput
      3.19 ± 10%      -2.0        1.18 ±  6%  perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      3.45 ±  5%      -2.0        1.48 ±  4%  perf-profile.calltrace.cycles-pp.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      3.00 ± 10%      -1.9        1.15 ±  6%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      3.37 ±  3%      -1.8        1.56 ±  4%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.32 ±  7%      -1.5        0.78 ±  6%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      2.82 ±  3%      -1.5        1.33 ±  4%  perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.26 ±  6%      -1.4        0.84 ±  7%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.55 ±  3%      -1.3        1.22 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply
      1.50 ±  6%      -1.2        0.27 ±100%  perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill
      1.88 ± 10%      -1.1        0.73 ±  6%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.53 ±  6%      -1.1        0.40 ± 58%  perf-profile.calltrace.cycles-pp.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill.dput
      1.45 ± 11%      -1.0        0.48 ± 58%  perf-profile.calltrace.cycles-pp.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      1.50 ±  6%      -0.9        0.60 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      1.22 ±  8%      -0.8        0.39 ± 57%  perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      1.12 ±  8%      -0.7        0.39 ± 57%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.00 ± 33%      -0.7        0.30 ±100%  perf-profile.calltrace.cycles-pp.memset_erms.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.67 ±  8%      -0.3        0.40 ± 57%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
      0.87 ±  9%      -0.2        0.70 ±  6%  perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.85 ±  8%      -0.2        0.69 ±  6%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2
      0.62 ±  7%      -0.1        0.52 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
      0.00            +1.0        1.01 ± 26%  perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.flush_bio_list.raid1_unplug
      0.00            +1.0        1.02 ± 25%  perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.flush_bio_list.raid1_unplug.blk_flush_plug_list
      0.00            +1.0        1.02 ± 26%  perf-profile.calltrace.cycles-pp.generic_make_request.flush_bio_list.raid1_unplug.blk_flush_plug_list.blk_finish_plug
      0.00            +1.0        1.03 ± 25%  perf-profile.calltrace.cycles-pp.flush_bio_list.raid1_unplug.blk_flush_plug_list.blk_finish_plug.wb_writeback
      0.00            +1.0        1.03 ± 25%  perf-profile.calltrace.cycles-pp.blk_finish_plug.wb_writeback.wb_workfn.process_one_work.worker_thread
      0.00            +1.0        1.03 ± 25%  perf-profile.calltrace.cycles-pp.blk_flush_plug_list.blk_finish_plug.wb_writeback.wb_workfn.process_one_work
      0.00            +1.0        1.03 ± 25%  perf-profile.calltrace.cycles-pp.raid1_unplug.blk_flush_plug_list.blk_finish_plug.wb_writeback.wb_workfn
      0.00            +1.1        1.12 ± 25%  perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.1        1.12 ± 25%  perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
      0.00            +1.3        1.32 ± 20%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.3        1.33 ± 20%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.00            +1.4        1.38 ± 19%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +1.4        1.38 ± 19%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      2.53 ±  5%     +28.9       31.41 ±  5%  perf-profile.calltrace.cycles-pp.unlink
      2.52 ±  5%     +28.9       31.41 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
      2.51 ±  5%     +28.9       31.41 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      2.50 ±  5%     +28.9       31.40 ±  5%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.68 ±  5%     +29.4       31.06 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.36 ±  8%     +29.6       31.00 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.22 ±  4%     +29.7       31.88 ±  6%  perf-profile.calltrace.cycles-pp.creat64
      2.21 ±  4%     +29.7       31.87 ±  6%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      2.21 ±  4%     +29.7       31.88 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat64
      2.21 ±  4%     +29.7       31.87 ±  6%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      2.21 ±  4%     +29.7       31.87 ±  6%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      2.18 ±  4%     +29.7       31.86 ±  6%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.18 ±  4%     +29.7       31.86 ±  6%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      0.68 ± 10%     +29.8       30.48 ±  5%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
      1.00 ±  2%     +30.1       31.05 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.78 ±  5%     +30.2       30.99 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      0.00           +30.5       30.46 ±  6%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
     57.13 ±  4%     -33.6       23.48 ±  6%  perf-profile.children.cycles-pp.write
     47.93 ±  7%     -28.2       19.74 ±  5%  perf-profile.children.cycles-pp.ksys_write
     47.10 ±  7%     -27.7       19.38 ±  5%  perf-profile.children.cycles-pp.vfs_write
     43.30 ±  7%     -25.6       17.68 ±  5%  perf-profile.children.cycles-pp.new_sync_write
     41.86 ±  7%     -24.8       17.04 ±  5%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     35.84 ±  8%     -21.6       14.22 ±  6%  perf-profile.children.cycles-pp.iomap_file_buffered_write
     35.61 ±  8%     -21.5       14.11 ±  6%  perf-profile.children.cycles-pp.iomap_apply
     26.70 ± 15%     -19.8        6.86 ± 23%  perf-profile.children.cycles-pp.intel_idle
     27.88 ± 15%     -19.8        8.09 ± 20%  perf-profile.children.cycles-pp.secondary_startup_64
     27.88 ± 15%     -19.8        8.09 ± 20%  perf-profile.children.cycles-pp.cpu_startup_entry
     27.88 ± 15%     -19.8        8.09 ± 20%  perf-profile.children.cycles-pp.do_idle
     27.47 ± 15%     -19.7        7.75 ± 21%  perf-profile.children.cycles-pp.cpuidle_enter_state
     27.47 ± 15%     -19.7        7.75 ± 21%  perf-profile.children.cycles-pp.cpuidle_enter
     27.67 ± 15%     -19.7        8.00 ± 20%  perf-profile.children.cycles-pp.start_secondary
     29.05 ±  8%     -18.0       11.10 ±  6%  perf-profile.children.cycles-pp.iomap_write_actor
     11.64 ±  9%      -7.1        4.52 ±  6%  perf-profile.children.cycles-pp.iomap_write_begin
     10.55 ±  9%      -6.9        3.61 ±  6%  perf-profile.children.cycles-pp.iomap_write_end
      8.19 ±  8%      -5.5        2.70 ±  8%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      7.48 ±  8%      -4.7        2.77 ±  6%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      7.29 ±  8%      -4.6        2.67 ±  6%  perf-profile.children.cycles-pp.pagecache_get_page
      5.55 ± 34%      -3.3        2.24 ± 36%  perf-profile.children.cycles-pp.llseek
      4.52 ±  7%      -3.1        1.46 ±  6%  perf-profile.children.cycles-pp.__prepare_exit_to_usermode
      4.29 ±  7%      -2.9        1.34 ±  6%  perf-profile.children.cycles-pp.__close
      4.25 ±  7%      -2.9        1.33 ±  6%  perf-profile.children.cycles-pp.task_work_run
      4.24 ±  7%      -2.9        1.33 ±  6%  perf-profile.children.cycles-pp.__fput
      4.13 ±  7%      -2.8        1.29 ±  6%  perf-profile.children.cycles-pp.dput
      4.09 ±  7%      -2.8        1.27 ±  6%  perf-profile.children.cycles-pp.__dentry_kill
      4.29 ±  7%      -2.3        2.02 ±  6%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      3.30 ±  6%      -2.1        1.25 ±  6%  perf-profile.children.cycles-pp.evict
      3.27 ±  6%      -2.0        1.24 ±  6%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      3.22 ± 10%      -2.0        1.19 ±  6%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      3.53 ±  5%      -2.0        1.51 ±  4%  perf-profile.children.cycles-pp.xfs_file_aio_write_checks
      3.01 ± 10%      -1.9        1.15 ±  6%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      3.40 ±  3%      -1.8        1.57 ±  4%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      2.36 ±  7%      -1.6        0.80 ±  6%  perf-profile.children.cycles-pp.find_get_entry
      2.83 ±  3%      -1.5        1.34 ±  4%  perf-profile.children.cycles-pp.copyin
      2.71 ±  3%      -1.4        1.28 ±  4%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      2.31 ±  6%      -1.4        0.91 ±  6%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      2.36 ±  6%      -1.4        0.97 ±  7%  perf-profile.children.cycles-pp.xfs_ilock
      1.90 ± 10%      -1.2        0.74 ±  6%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      1.95 ±  7%      -1.2        0.80 ±  7%  perf-profile.children.cycles-pp.down_write
      1.81 ±  9%      -1.1        0.74 ±  9%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.58 ±  6%      -1.0        0.55 ±  6%  perf-profile.children.cycles-pp.release_pages
      1.53 ±  6%      -1.0        0.52 ±  6%  perf-profile.children.cycles-pp.__pagevec_release
      1.48 ± 11%      -0.9        0.62 ± 13%  perf-profile.children.cycles-pp.file_update_time
      1.40 ±  9%      -0.8        0.57 ±  8%  perf-profile.children.cycles-pp.xfs_iunlock
      1.23 ± 10%      -0.8        0.47 ±  8%  perf-profile.children.cycles-pp.__mod_lruvec_state
      1.31 ± 10%      -0.7        0.56 ± 10%  perf-profile.children.cycles-pp.__xfs_trans_commit
      1.04 ± 15%      -0.7        0.30 ± 11%  perf-profile.children.cycles-pp.lock_page_memcg
      1.29 ±  9%      -0.7        0.55 ± 11%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      1.15 ±  8%      -0.7        0.47 ±  7%  perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      1.07 ±  9%      -0.7        0.39 ±  6%  perf-profile.children.cycles-pp.lru_cache_add
      1.02 ±  9%      -0.6        0.38 ±  7%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      1.12 ±  9%      -0.6        0.48 ±  6%  perf-profile.children.cycles-pp.__set_page_dirty
      1.18 ±  6%      -0.6        0.55 ±  5%  perf-profile.children.cycles-pp.___might_sleep
      1.14 ±  8%      -0.6        0.52 ±  4%  perf-profile.children.cycles-pp.security_file_permission
      1.72 ±  4%      -0.6        1.10 ±  4%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.94 ± 11%      -0.6        0.34 ±  8%  perf-profile.children.cycles-pp.mem_cgroup_charge
      1.00 ±  6%      -0.6        0.40 ±  5%  perf-profile.children.cycles-pp.xas_load
      0.93 ±  9%      -0.5        0.38 ±  7%  perf-profile.children.cycles-pp.up_write
      0.89 ±  6%      -0.5        0.36 ±  7%  perf-profile.children.cycles-pp.ksys_lseek
      0.86 ±  5%      -0.5        0.37 ±  7%  perf-profile.children.cycles-pp.__fdget_pos
      0.80 ±  8%      -0.5        0.30 ±  6%  perf-profile.children.cycles-pp.truncate_cleanup_page
      0.88 ±  6%      -0.5        0.39 ±  9%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.03 ± 31%      -0.5        0.55 ± 11%  perf-profile.children.cycles-pp.memset_erms
      0.85 ±  6%      -0.5        0.38 ±  3%  perf-profile.children.cycles-pp.common_file_perm
      0.82 ±  8%      -0.5        0.35 ±  7%  perf-profile.children.cycles-pp.xfs_file_write_iter
      0.79 ±  7%      -0.5        0.33 ±  8%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_end
      0.76 ± 20%      -0.5        0.31 ± 28%  perf-profile.children.cycles-pp.xfs_vn_update_time
      1.01 ±  8%      -0.5        0.56 ± 20%  perf-profile.children.cycles-pp._raw_spin_lock
      0.70 ± 16%      -0.4        0.25 ±  7%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.74 ±  7%      -0.4        0.30 ±  6%  perf-profile.children.cycles-pp.disk_rw
      0.69 ± 10%      -0.4        0.26 ±  6%  perf-profile.children.cycles-pp.__cancel_dirty_page
      0.67 ± 11%      -0.4        0.25 ±  6%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.69 ±  9%      -0.4        0.28 ± 13%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.76 ±  6%      -0.4        0.34 ±  7%  perf-profile.children.cycles-pp.__sb_start_write
      0.71 ±  8%      -0.4        0.30 ±  7%  perf-profile.children.cycles-pp.xfs_break_layouts
      0.71 ±  6%      -0.4        0.31 ± 12%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.62 ± 12%      -0.4        0.23 ± 11%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.68 ± 12%      -0.4        0.28 ±  7%  perf-profile.children.cycles-pp.account_page_dirtied
      0.71 ± 13%      -0.4        0.32 ± 12%  perf-profile.children.cycles-pp.xlog_cil_insert_items
      0.69 ±  5%      -0.4        0.30 ±  6%  perf-profile.children.cycles-pp.__fget_light
      0.69 ± 13%      -0.4        0.31 ±  5%  perf-profile.children.cycles-pp.vfs_unlink
      0.67 ±  4%      -0.4        0.29 ±  5%  perf-profile.children.cycles-pp.__might_sleep
      0.68 ± 13%      -0.4        0.30 ±  4%  perf-profile.children.cycles-pp.xfs_vn_unlink
      0.68 ± 13%      -0.4        0.30 ±  4%  perf-profile.children.cycles-pp.xfs_remove
      0.67 ±  6%      -0.4        0.29 ±  7%  perf-profile.children.cycles-pp.delete_from_page_cache_batch
      0.74 ± 10%      -0.4        0.39        perf-profile.children.cycles-pp.xfs_inactive
      0.45 ±  2%      -0.4        0.10 ±  8%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.55 ± 10%      -0.3        0.20 ±  6%  perf-profile.children.cycles-pp.account_page_cleaned
      0.63 ±  7%      -0.3        0.30 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.42 ±  5%      -0.3        0.11 ±  4%  perf-profile.children.cycles-pp.free_unref_page_list
      0.45 ±  6%      -0.3        0.15 ±  8%  perf-profile.children.cycles-pp.rmqueue
      0.53 ±  7%      -0.3        0.23 ±  8%  perf-profile.children.cycles-pp._cond_resched
      0.43 ±  6%      -0.3        0.14 ±  5%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.62 ± 10%      -0.3        0.35 ±  3%  perf-profile.children.cycles-pp.xfs_inactive_ifree
      0.47 ± 10%      -0.3        0.20 ±  7%  perf-profile.children.cycles-pp.fsnotify
      0.44 ±  4%      -0.3        0.18 ± 12%  perf-profile.children.cycles-pp.current_time
      0.43 ±  8%      -0.3        0.18 ±  6%  perf-profile.children.cycles-pp.unlock_page
      0.39 ±  7%      -0.2        0.15 ±  8%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.41 ±  8%      -0.2        0.17 ±  6%  perf-profile.children.cycles-pp.page_mapping
      0.46 ±  3%      -0.2        0.22 ±  3%  perf-profile.children.cycles-pp.xas_store
      0.41 ±  9%      -0.2        0.17 ±  7%  perf-profile.children.cycles-pp.xfs_break_leased_layouts
      0.28 ±  4%      -0.2        0.05 ± 58%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.45 ± 10%      -0.2        0.22 ±  8%  perf-profile.children.cycles-pp.xfs_errortag_test
      0.28 ±  5%      -0.2        0.05 ± 58%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.27 ±  5%      -0.2        0.04 ± 58%  perf-profile.children.cycles-pp.enqueue_entity
      0.39 ± 14%      -0.2        0.17 ±  4%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.34 ±  8%      -0.2        0.12 ± 11%  perf-profile.children.cycles-pp.uncharge_batch
      0.36 ± 13%      -0.2        0.14 ±  5%  perf-profile.children.cycles-pp.xfs_dir_removename
      0.38 ±  7%      -0.2        0.17 ±  4%  perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
      0.34 ± 14%      -0.2        0.14 ±  7%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      0.41 ±  4%      -0.2        0.21 ±  5%  perf-profile.children.cycles-pp.generic_write_checks
      0.29 ±  8%      -0.2        0.09 ±  7%  perf-profile.children.cycles-pp.xfs_buf_read_map
      0.26 ±  7%      -0.2        0.06 ± 13%  perf-profile.children.cycles-pp.try_to_wake_up
      0.28 ±  6%      -0.2        0.09 ±  4%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.32 ± 11%      -0.2        0.12 ±  4%  perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
      0.32 ± 10%      -0.2        0.13 ±  3%  perf-profile.children.cycles-pp.iov_iter_advance
      0.21 ±  6%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.26 ±  6%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_buf_find
      0.30 ± 10%      -0.2        0.12 ±  7%  perf-profile.children.cycles-pp.try_charge
      0.31 ±  7%      -0.2        0.13 ±  7%  perf-profile.children.cycles-pp.fsnotify_parent
      0.29 ±  7%      -0.2        0.12 ± 11%  perf-profile.children.cycles-pp.xfs_file_llseek
      0.28 ± 11%      -0.2        0.10 ± 10%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.25 ± 15%      -0.2        0.08 ± 10%  perf-profile.children.cycles-pp.__count_memcg_events
      0.87 ±  9%      -0.2        0.70 ±  6%  perf-profile.children.cycles-pp.xfs_generic_create
      0.43 ± 10%      -0.2        0.27 ±  5%  perf-profile.children.cycles-pp.xfs_ifree
      0.85 ±  8%      -0.2        0.69 ±  6%  perf-profile.children.cycles-pp.xfs_create
      0.29 ± 10%      -0.2        0.13 ±  3%  perf-profile.children.cycles-pp.__sb_end_write
      0.43 ±  7%      -0.2        0.27 ±  6%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      0.29 ± 20%      -0.2        0.13 ± 31%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.22 ± 18%      -0.2        0.07 ± 12%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.25 ±  9%      -0.2        0.09 ± 11%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.24 ±  8%      -0.2        0.09 ± 11%  perf-profile.children.cycles-pp.page_counter_cancel
      0.26 ±  9%      -0.1        0.11 ±  7%  perf-profile.children.cycles-pp.rcu_all_qs
      0.25 ± 20%      -0.1        0.10 ± 35%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.26 ± 19%      -0.1        0.11 ± 31%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.26 ±  5%      -0.1        0.12 ±  7%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.24 ± 11%      -0.1        0.10 ±  7%  perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.20 ± 10%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.file_modified
      0.21 ±  6%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.__sched_text_start
      0.21 ± 35%      -0.1        0.08 ± 23%  perf-profile.children.cycles-pp.start_kernel
      0.23 ±  5%      -0.1        0.10 ±  8%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.22 ±  7%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.__syscall_return_slowpath
      0.24 ±  7%      -0.1        0.11 ±  9%  perf-profile.children.cycles-pp.xas_start
      0.21 ±  8%      -0.1        0.09 ± 15%  perf-profile.children.cycles-pp.mem_cgroup_track_foreign_dirty_slowpath
      0.19 ± 11%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.xfs_da_read_buf
      0.18 ±  8%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__unlock_page_memcg
      0.15 ± 22%      -0.1        0.04 ±101%  perf-profile.children.cycles-pp.xlog_grant_add_space
      0.18 ± 12%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.aa_file_perm
      0.18 ±  8%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.node_dirty_ok
      0.18 ± 12%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_dir3_data_check
      0.18 ± 12%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.__xfs_dir3_data_check
      0.17 ± 13%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.14 ± 12%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.iomap_page_create
      0.28 ± 12%      -0.1        0.18 ± 17%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.28 ± 13%      -0.1        0.18 ± 18%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.16 ± 14%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.xfs_dir_createname
      0.16 ±  2%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.timestamp_truncate
      0.16 ±  6%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.schedule
      0.14 ± 12%      -0.1        0.05        perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
      0.15 ± 14%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_dir2_node_addname
      0.14 ± 13%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.20 ±  2%      -0.1        0.11 ±  6%  perf-profile.children.cycles-pp.generic_write_check_limits
      0.30 ± 13%      -0.1        0.21 ± 18%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.11 ± 11%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_dir_ino_validate
      0.14 ± 11%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.pagevec_lookup_entries
      0.13 ±  8%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.find_get_entries
      0.12 ± 10%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.rw_verify_area
      0.10 ± 10%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_free_eofblocks
      0.11 ±  9%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.13 ± 13%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_dir2_leafn_remove
      0.12 ±  6%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.xfs_btree_lookup
      0.10 ± 15%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.rcu_core
      0.11 ±  7%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.xfs_release
      0.11 ±  4%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.alloc_pages_current
      0.14 ± 12%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.12 ± 13%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.xfs_get_extsz_hint
      0.21 ± 19%      -0.1        0.15 ± 22%  perf-profile.children.cycles-pp.clockevents_program_event
      0.12 ± 13%      -0.1        0.07 ± 12%  perf-profile.children.cycles-pp.wait_for_stable_page
      0.11 ±  7%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.11            -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.xas_init_marks
      0.13 ±  9%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.__lock_text_start
      0.11 ±  4%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__xa_set_mark
      0.09 ±  7%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.iov_iter_init
      0.10 ± 15%      -0.1        0.04 ± 63%  perf-profile.children.cycles-pp.rebalance_domains
      0.28 ± 10%      -0.0        0.23 ±  4%  perf-profile.children.cycles-pp.xfs_difree
      0.14 ± 15%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.13 ±  6%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.__x64_sys_write
      0.10 ± 12%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.memcpy_erms
      0.09 ± 15%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.update_load_avg
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.update_cfs_group
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.07            +0.0        0.11 ± 15%  perf-profile.children.cycles-pp.__percpu_counter_compare
      0.06            +0.0        0.11 ± 13%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.29 ± 11%      +0.1        0.34 ±  2%  perf-profile.children.cycles-pp.tick_sched_handle
      0.11 ± 21%      +0.1        0.17 ± 10%  perf-profile.children.cycles-pp.task_tick_fair
      0.28 ± 12%      +0.1        0.34 ±  2%  perf-profile.children.cycles-pp.update_process_times
      0.06 ± 15%      +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.xfs_mod_fdblocks
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.33 ± 11%      +0.1        0.39 ±  2%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.xfs_end_io
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.xfs_end_ioend
      0.18 ± 18%      +0.1        0.24 ±  2%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.xfs_verify_agino
      0.07 ±  6%      +0.1        0.14 ± 15%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.iomap_writepages
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.write_cache_pages
      0.00            +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.writeback_sb_inodes
      0.00            +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.__writeback_single_inode
      0.00            +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.do_writepages
      0.00            +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.xfs_vm_writepages
      0.42 ± 10%      +0.1        0.51 ±  7%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.11 ±  4%      +0.1        0.20 ± 11%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      0.11 ± 11%      +0.1        0.21 ± 10%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.xfs_inactive_worker
      0.00            +0.1        0.10 ± 28%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.12 ±  3%      +0.1        0.23 ± 10%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      0.00            +0.1        0.12 ± 27%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.22 ±  7%      +0.1        0.34 ± 10%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      0.11 ±  7%      +0.1        0.24 ± 31%  perf-profile.children.cycles-pp.menu_select
      0.26 ± 10%      +0.1        0.40 ± 13%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      0.43 ±  6%      +0.2        0.60 ± 10%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      0.00            +0.3        0.29 ±  2%  perf-profile.children.cycles-pp.xfs_inode_free_blocks
      0.00            +0.4        0.39 ±  2%  perf-profile.children.cycles-pp.xfs_inactive_inode
      0.00            +0.4        0.40        perf-profile.children.cycles-pp.__xfs_inode_walk
      0.00            +0.4        0.40        perf-profile.children.cycles-pp.xfs_inode_walk_ag
      0.00            +1.0        1.04 ± 24%  perf-profile.children.cycles-pp.brd_do_bvec
      0.00            +1.0        1.04 ± 25%  perf-profile.children.cycles-pp.blk_finish_plug
      0.00            +1.0        1.04 ± 25%  perf-profile.children.cycles-pp.blk_flush_plug_list
      0.00            +1.0        1.04 ± 25%  perf-profile.children.cycles-pp.raid1_unplug
      0.00            +1.0        1.05 ± 24%  perf-profile.children.cycles-pp.brd_make_request
      0.00            +1.1        1.05 ± 24%  perf-profile.children.cycles-pp.flush_bio_list
      0.00            +1.1        1.06 ± 24%  perf-profile.children.cycles-pp.generic_make_request
      0.00            +1.1        1.12 ± 25%  perf-profile.children.cycles-pp.wb_workfn
      0.00            +1.1        1.12 ± 25%  perf-profile.children.cycles-pp.wb_writeback
      0.08 ±  5%      +1.3        1.38 ± 19%  perf-profile.children.cycles-pp.kthread
      0.08 ±  5%      +1.3        1.38 ± 19%  perf-profile.children.cycles-pp.ret_from_fork
      0.01 ±173%      +1.3        1.33 ± 20%  perf-profile.children.cycles-pp.worker_thread
      0.00            +1.3        1.32 ± 20%  perf-profile.children.cycles-pp.process_one_work
     66.34 ±  6%     +21.9       88.23        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     65.62 ±  6%     +22.3       87.95        perf-profile.children.cycles-pp.do_syscall_64
      2.54 ±  5%     +28.9       31.41 ±  5%  perf-profile.children.cycles-pp.unlink
      2.50 ±  5%     +28.9       31.40 ±  5%  perf-profile.children.cycles-pp.do_unlinkat
      2.28 ±  4%     +29.6       31.90 ±  6%  perf-profile.children.cycles-pp.do_sys_open
      2.28 ±  4%     +29.6       31.90 ±  6%  perf-profile.children.cycles-pp.do_sys_openat2
      2.23 ±  4%     +29.6       31.88 ±  6%  perf-profile.children.cycles-pp.creat64
      2.23 ±  4%     +29.6       31.88 ±  6%  perf-profile.children.cycles-pp.do_filp_open
      2.23 ±  4%     +29.6       31.88 ±  6%  perf-profile.children.cycles-pp.path_openat
      2.69 ±  4%     +59.4       62.12 ±  6%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      2.15 ±  7%     +59.9       62.00 ±  6%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.80 ±  9%     +60.2       60.97 ±  6%  perf-profile.children.cycles-pp.osq_lock
     26.70 ± 15%     -19.8        6.86 ± 23%  perf-profile.self.cycles-pp.intel_idle
      8.10 ±  8%      -5.4        2.67 ±  8%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      2.69 ±  3%      -1.4        1.27 ±  4%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.79 ±  9%      -1.1        0.73 ± 10%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.48 ±  6%      -1.0        0.45 ±  8%  perf-profile.self.cycles-pp.find_get_entry
      1.45 ±  5%      -0.8        0.65 ±  3%  perf-profile.self.cycles-pp.iomap_apply
      1.18 ±  5%      -0.7        0.44 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.03 ± 15%      -0.7        0.29 ± 11%  perf-profile.self.cycles-pp.lock_page_memcg
      1.17 ±  5%      -0.7        0.48 ±  5%  perf-profile.self.cycles-pp.write
      1.12 ±  7%      -0.7        0.46 ±  7%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      1.02 ±  7%      -0.7        0.37 ± 10%  perf-profile.self.cycles-pp.down_write
      1.17 ±  6%      -0.6        0.54 ±  5%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
      1.05 ±  8%      -0.6        0.42 ±  5%  perf-profile.self.cycles-pp.iomap_write_end
      1.16 ±  6%      -0.6        0.55 ±  5%  perf-profile.self.cycles-pp.___might_sleep
      1.71 ±  4%      -0.6        1.10 ±  3%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.93 ±  6%      -0.5        0.40 ±  7%  perf-profile.self.cycles-pp.iomap_write_actor
      0.88 ±  8%      -0.5        0.36 ±  6%  perf-profile.self.cycles-pp.up_write
      1.02 ± 31%      -0.5        0.54 ± 11%  perf-profile.self.cycles-pp.memset_erms
      0.75 ±  6%      -0.5        0.29 ±  4%  perf-profile.self.cycles-pp.xas_load
      0.80 ±  8%      -0.5        0.34 ±  7%  perf-profile.self.cycles-pp.xfs_file_write_iter
      0.81 ±  5%      -0.5        0.35 ±  5%  perf-profile.self.cycles-pp.vfs_write
      0.73 ± 17%      -0.4        0.28 ± 15%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.65 ± 16%      -0.4        0.23 ±  8%  perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
      0.69 ±  9%      -0.4        0.28 ± 13%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.71 ±  6%      -0.4        0.30 ±  5%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      0.71 ±  8%      -0.4        0.30 ±  8%  perf-profile.self.cycles-pp.iomap_write_begin
      0.61 ± 12%      -0.4        0.22 ± 13%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.64 ±  8%      -0.4        0.27 ±  6%  perf-profile.self.cycles-pp.disk_rw
      0.65 ±  5%      -0.4        0.29 ±  5%  perf-profile.self.cycles-pp.__fget_light
      0.66 ±  5%      -0.4        0.30 ±  3%  perf-profile.self.cycles-pp.common_file_perm
      0.45 ±  3%      -0.3        0.10 ±  8%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.58 ±  8%      -0.3        0.23 ±  5%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.58 ±  4%      -0.3        0.25 ±  4%  perf-profile.self.cycles-pp.__might_sleep
      0.57 ±  6%      -0.3        0.23 ±  8%  perf-profile.self.cycles-pp.pagecache_get_page
      0.57 ±  5%      -0.3        0.25 ±  7%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_end
      0.52 ±  4%      -0.3        0.22 ±  5%  perf-profile.self.cycles-pp.llseek
      0.49 ± 10%      -0.3        0.20 ±  9%  perf-profile.self.cycles-pp.xfs_iunlock
      0.52 ±  5%      -0.3        0.23 ± 11%  perf-profile.self.cycles-pp.new_sync_write
      0.44 ±  4%      -0.3        0.18 ±  6%  perf-profile.self.cycles-pp.xfs_ilock
      0.46 ± 10%      -0.3        0.20 ±  6%  perf-profile.self.cycles-pp.fsnotify
      0.42 ±  7%      -0.3        0.16 ±  5%  perf-profile.self.cycles-pp.iomap_set_page_dirty
      0.42 ±  8%      -0.2        0.17 ±  4%  perf-profile.self.cycles-pp.unlock_page
      0.39 ±  9%      -0.2        0.15 ±  5%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.40 ±  8%      -0.2        0.17 ±  6%  perf-profile.self.cycles-pp.page_mapping
      0.39 ±  9%      -0.2        0.16 ±  7%  perf-profile.self.cycles-pp.xfs_break_leased_layouts
      0.64 ±  6%      -0.2        0.43 ± 29%  perf-profile.self.cycles-pp._raw_spin_lock
      0.37 ±  7%      -0.2        0.16 ±  6%  perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
      0.36 ± 10%      -0.2        0.15 ±  9%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.39 ±  9%      -0.2        0.19 ±  7%  perf-profile.self.cycles-pp.xfs_errortag_test
      0.31 ± 10%      -0.2        0.11 ±  7%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.33 ±  5%      -0.2        0.15 ±  7%  perf-profile.self.cycles-pp.xfs_file_aio_write_checks
      0.31 ± 11%      -0.2        0.13 ±  5%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.30 ± 11%      -0.2        0.12 ±  3%  perf-profile.self.cycles-pp.iov_iter_advance
      0.27 ± 10%      -0.2        0.10 ± 10%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.25 ± 15%      -0.2        0.08 ± 10%  perf-profile.self.cycles-pp.__count_memcg_events
      0.28 ±  8%      -0.2        0.11 ±  9%  perf-profile.self.cycles-pp.xfs_file_llseek
      0.29 ±  5%      -0.2        0.13 ±  6%  perf-profile.self.cycles-pp.ksys_write
      0.42 ±  7%      -0.2        0.26 ±  7%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.28 ±  6%      -0.2        0.12 ±  8%  perf-profile.self.cycles-pp.fsnotify_parent
      0.26 ±  6%      -0.2        0.10 ± 10%  perf-profile.self.cycles-pp.release_pages
      0.29 ±  8%      -0.2        0.14 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.27 ± 10%      -0.2        0.12 ±  3%  perf-profile.self.cycles-pp.__sb_end_write
      0.28 ±  5%      -0.1        0.14 ±  3%  perf-profile.self.cycles-pp.file_update_time
      0.26 ±  5%      -0.1        0.11 ±  7%  perf-profile.self.cycles-pp._cond_resched
      0.26 ±  4%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__prepare_exit_to_usermode
      0.21 ± 10%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.page_counter_cancel
      0.23 ±  7%      -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.xfs_break_layouts
      0.23 ±  6%      -0.1        0.11 ±  6%  perf-profile.self.cycles-pp.iomap_file_buffered_write
      0.21 ±  8%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.account_page_cleaned
      0.22 ±  5%      -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.23 ±  6%      -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.__sb_start_write
      0.21 ±  7%      -0.1        0.08 ± 13%  perf-profile.self.cycles-pp.mem_cgroup_track_foreign_dirty_slowpath
      0.20 ±  8%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.__syscall_return_slowpath
      0.15 ± 22%      -0.1        0.04 ±103%  perf-profile.self.cycles-pp.xlog_grant_add_space
      0.18 ±  8%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.__fdget_pos
      0.17 ± 10%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__unlock_page_memcg
      0.21 ±  7%      -0.1        0.10 ±  7%  perf-profile.self.cycles-pp.generic_write_checks
      0.21 ±  8%      -0.1        0.10 ± 10%  perf-profile.self.cycles-pp.xas_start
      0.17 ± 11%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.aa_file_perm
      0.17 ±  7%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.current_time
      0.20 ±  7%      -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.xas_store
      0.18 ±  8%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.rcu_all_qs
      0.17 ± 14%      -0.1        0.07 ± 11%  perf-profile.self.cycles-pp.account_page_dirtied
      0.15 ± 14%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.security_file_permission
      0.13 ±  9%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.node_dirty_ok
      0.15 ±  8%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.try_charge
      0.14 ±  7%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.ksys_lseek
      0.19 ±  2%      -0.1        0.10 ±  8%  perf-profile.self.cycles-pp.generic_write_check_limits
      0.12 ± 13%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.mem_cgroup_charge
      0.11 ±  7%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.14 ±  5%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.timestamp_truncate
      0.13 ± 14%      -0.1        0.05        perf-profile.self.cycles-pp.page_counter_try_charge
      0.11 ±  7%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.rw_verify_area
      0.14 ± 10%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.13 ±  9%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.rmqueue
      0.10 ± 12%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.xfs_get_extsz_hint
      0.11 ±  4%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.12 ± 11%      -0.1        0.07        perf-profile.self.cycles-pp.apparmor_file_permission
      0.10 ±  4%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.copyin
      0.11 ± 11%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.wait_for_stable_page
      0.10 ± 10%      -0.0        0.06        perf-profile.self.cycles-pp.__lock_text_start
      0.10 ± 12%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.memcpy_erms
      0.11 ±  4%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.__x64_sys_write
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.update_cfs_group
      0.06 ±  7%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.xfs_btree_check_sblock
      0.03 ±100%      +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.xfs_btree_increment
      0.06 ± 17%      +0.1        0.14 ± 37%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +0.7        0.72 ± 28%  perf-profile.self.cycles-pp.brd_do_bvec
      0.80 ±  9%     +59.8       60.60 ±  6%  perf-profile.self.cycles-pp.osq_lock


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  500000 +------------------------------------------------------------------+   
  450000 |.+  +.+  +.+.++.+   ++.+.++.+   ++   +.+  +.+  +.+. +.+.++.+.+  + |   
         | :  : :  :                  :   ::   : :  : :  :   +              |   
  400000 |-:  : :  :                  :   ::   : :  : :  :                  |   
  350000 |-:  : :  :   OO     O  O O  :   ::   : :  : :  :                  |   
         | :O:   :O:      O O  O    O O:O:  : :  : :  : :                   |   
  300000 |-: :  O: : O                 : :  : :  : :  : :                   |   
  250000 |-: :   : :                   : :  : :  : :  : :                   |   
  200000 |-+::   ::                    : :  : :O  ::   ::                   |   
         |  ::   ::                    : :OO:O:   ::   ::                   |   
  150000 |-+::   ::                    : :  : :   ::   ::                   |   
  100000 |-+:     :                     :    :    :    :                    |   
         |  :     :                     :    :    :    :                    |   
   50000 |-+:     :                     :    :    :    :                    |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               aim7.time.system_time                            
                                                                                
  3000 +--------------------------------------------------------------------+   
       |                                                                    |   
  2500 |-+                               OO O                               |   
       |                                      O                             |   
       | O                                                                  |   
  2000 |-+ O  O    O      O   O    O                                        |   
       |        O O  O O O  O   O O  O O                                    |   
  1500 |-+                                                                  |   
       |                                                                    |   
  1000 |-+                                                                  |   
       |                                                          .+.       |   
       |.+  +.+   ++.+.+.++.+.+.+.++.+   ++   +.+  +.+   ++.+.+.++   +.++.+.|   
   500 |-+: :  : :                    : :  : :  : :   : :                   |   
       |  ::   : :                    : :  : :   ::   : :                   |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  120 +---------------------------------------------------------------------+   
      |                                                                     |   
  100 |-+                                 O                                 |   
      |                                 O   O                               |   
      |                                      O                              |   
   80 |-+                                                                   |   
      | O    O                                                              |   
   60 |-+ O        O    O O  O     O  O                                     |   
      |        O O  O O     O  O O   O                                      |   
   40 |-+  +.    +. +. .+. .+ .+.       +.         +.   +. .+. +. .+. .+ .+.|   
      |.+  : +   : +  +   +  +   +.+.+  : +  +.+   : +  : +   +  +   +  +   |   
      |  : :  : :                    : :   : :  : :  : :                    |   
   20 |-+::   : :                     ::   : :  : :   ::                    |   
      |   :    :                      :     :    :    :                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  120 +---------------------------------------------------------------------+   
      |                                                                     |   
  100 |-+                                 O                                 |   
      |                                 O   O                               |   
      |                                      O                              |   
   80 |-+                                                                   |   
      | O    O                                                              |   
   60 |-+ O        O    O O  O     O  O                                     |   
      |        O O  O O     O  O O   O                                      |   
   40 |-+  +.    +. +. .+. .+ .+.       +.         +.   +. .+. +. .+. .+ .+.|   
      |.+  : +   : +  +   +  +   +.+.+  : +  +.+   : +  : +   +  +   +  +   |   
      |  : :  : :                    : :   : :  : :  : :                    |   
   20 |-+::   : :                     ::   : :  : :   ::                    |   
      |   :    :                      :     :    :    :                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        aim7.time.voluntary_context_switches                    
                                                                                
   1e+06 +------------------------------------------------------------------+   
  900000 |.+  +.+  +.+.++.+.+.++.+.++.+   ++   +.+  +.+  +.+.++.+.++.+.+.++.|   
         | :  : :  :                  :   ::   : :  : :  :                  |   
  800000 |-:  : :  :                  :   ::   : :  : :  :                  |   
  700000 |-:  : :  :                  :   ::   : :  : :  :                  |   
         | : :   :O:    O     O  O O   : :  : :  : :  : :                   |   
  600000 |-: :   : : O O  O O  O    O O: :  : :  : :  : :                   |   
  500000 |-:O:  O: :                   :O:  : :O : :  : :                   |   
  400000 |-O::   ::                    : :O :O:   ::   ::                   |   
         |  ::   ::                    : : O: :   ::   ::                   |   
  300000 |-+::   ::                    : :  : :   ::   ::                   |   
  200000 |-+:     :                     :    :    :    :                    |   
         |  :     :                     :    :    :    :                    |   
  100000 |-+:     :                     :    :    :    :                    |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       aim7.time.involuntary_context_switches                   
                                                                                
  12000 +-------------------------------------------------------------------+   
        |                                O                                  |   
  10000 |-+                                  O                              |   
        |                                  O                                |   
        | O                                   O                             |   
   8000 |-+    O    O                                                       |   
        |        O         O   O    O                                       |   
   6000 |-+O      O   O OO   O  O O   OO                                    |   
        |                                                                   |   
   4000 |-+                                                                 |   
        |    +.      .+.    .+.          +.         +    +. +.     +.     +.|   
        |.+  : +  +.+   ++.+   ++. .+.+  : +  +.+   :+   : +  +.+.+  +.+.+  |   
   2000 |-: :   : :               +   : :   : :  : :  : :                   |   
        |  ::   ::                     ::   ::   : :  : :                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.8.0-00182-g086c2d4ccc61ec" of type "text/plain" (158411 bytes)

View attachment "job-script" of type "text/plain" (8006 bytes)

View attachment "job.yaml" of type "text/plain" (5579 bytes)

View attachment "reproduce" of type "text/plain" (1026 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ