lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20191112094848.GC30590@shao2-debian>
Date:   Tue, 12 Nov 2019 17:48:48 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Darrick J. Wong" <darrick.wong@...cle.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...ts.01.org
Subject: [xfs] 646a9fb7fb: aim7.jobs-per-min -33.0% regression

Greeting,

FYI, we noticed a -33.0% regression of aim7.jobs-per-min due to commit:


commit: 646a9fb7fb983a7d53da1892c7f1225929de5bd1 ("xfs: deferred inode inactivation")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git repair-hard-problems

in testcase: aim7
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: xfs
	test: disk_wrt
	load: 3000
	cpufreq_governor: performance
	ucode: 0x2000064

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID0/debian-x86_64-2019-09-23.cgz/lkp-skl-2sp7/disk_wrt/aim7/0x2000064

commit: 
  109ace4e35 ("xfs: pass around xfs_inode_ag_walk iget/irele helper functions")
  646a9fb7fb ("xfs: deferred inode inactivation")

109ace4e35577294 646a9fb7fb983a7d53da1892c7f 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :4          100%           4:4     kmsg.XFS(md#):xlog_verify_grant_tail:space>BBTOB(tail_blocks)
         %stddev     %change         %stddev
             \          |                \  
    438138           -33.0%     293397 ±  2%  aim7.jobs-per-min
     41.26           +49.2%      61.56 ±  2%  aim7.time.elapsed_time
     41.26           +49.2%      61.56 ±  2%  aim7.time.elapsed_time.max
      1229 ±  5%    +677.3%       9553 ±  4%  aim7.time.involuntary_context_switches
    429.70          +363.3%       1990 ±  2%  aim7.time.system_time
     84.68            +6.5%      90.16        aim7.time.user_time
    924587           -52.2%     442059 ±  4%  aim7.time.voluntary_context_switches
      8217 ±114%     +99.3%      16377 ± 42%  numa-numastat.node1.other_node
  26110006 ± 88%     -95.6%    1144783 ± 20%  cpuidle.C1.time
    160209 ± 34%     -76.5%      37711 ± 18%  cpuidle.C1.usage
   4453545 ±119%     -77.6%     995813 ± 15%  cpuidle.POLL.time
     82.85           -27.5       55.31 ±  5%  mpstat.cpu.all.idle%
      0.00 ± 55%      -0.0        0.00        mpstat.cpu.all.iowait%
     14.24 ±  2%     +28.4       42.68 ±  7%  mpstat.cpu.all.sys%
      2.89            -0.9        2.01 ±  4%  mpstat.cpu.all.usr%
     83.25           -32.7%      56.00 ±  5%  vmstat.cpu.id
      3086 ±  5%    +127.4%       7020 ±  7%  vmstat.io.bo
     11.25 ±  3%    +173.3%      30.75 ±  9%  vmstat.procs.r
     43674           -62.1%      16550 ±  2%  vmstat.system.cs
     83.45           -32.6%      56.26 ±  5%  iostat.cpu.idle
     13.74 ±  2%    +203.9%      41.76 ±  7%  iostat.cpu.system
      2.80           -29.4%       1.98 ±  4%  iostat.cpu.user
     92.59          +205.5%     282.91 ±  7%  iostat.md0.w/s
      3231 ±  5%    +124.5%       7252 ±  7%  iostat.md0.wkB/s
    566.50 ±  2%    +145.4%       1390 ±  7%  turbostat.Avg_MHz
     19.01 ±  4%     +26.9       45.89 ±  6%  turbostat.Busy%
    156911 ± 34%     -77.3%      35696 ± 14%  turbostat.C1
      0.84 ± 87%      -0.8        0.03 ± 19%  turbostat.C1%
     77.66 ±  5%     -34.0%      51.24 ± 10%  turbostat.CPU%c1
      1.33 ±173%      +3.9        5.20 ± 23%  turbostat.PKG_%
      3347 ± 14%     +21.3%       4060 ± 10%  numa-meminfo.node0.Dirty
      3514 ± 27%     +59.8%       5617 ± 10%  numa-meminfo.node0.Inactive(file)
     51128 ± 18%     +44.4%      73837 ± 20%  numa-meminfo.node0.KReclaimable
   1273839 ± 17%     +16.7%    1486631 ± 15%  numa-meminfo.node0.MemUsed
     51128 ± 18%     +44.4%      73837 ± 20%  numa-meminfo.node0.SReclaimable
    110431 ± 36%     +46.5%     161821 ± 25%  numa-meminfo.node0.SUnreclaim
    161560 ± 30%     +45.9%     235659 ± 22%  numa-meminfo.node0.Slab
      5501 ± 39%    +115.0%      11829 ± 54%  numa-meminfo.node1.Inactive
      1514 ± 66%    +346.2%       6755 ±103%  numa-meminfo.node1.Inactive(anon)
    844.75 ± 14%     +24.8%       1054 ± 11%  numa-vmstat.node0.nr_dirty
    882.00 ± 26%     +64.5%       1451 ± 10%  numa-vmstat.node0.nr_inactive_file
     12779 ± 18%     +44.4%      18455 ± 20%  numa-vmstat.node0.nr_slab_reclaimable
     27602 ± 36%     +46.5%      40444 ± 25%  numa-vmstat.node0.nr_slab_unreclaimable
    889.25 ± 27%     +62.7%       1446 ± 10%  numa-vmstat.node0.nr_zone_inactive_file
    779.00 ± 21%     +33.5%       1039 ± 12%  numa-vmstat.node1.nr_dirty
    378.00 ± 66%    +346.7%       1688 ±103%  numa-vmstat.node1.nr_inactive_anon
      1577 ± 40%     +92.4%       3034 ± 15%  numa-vmstat.node1.nr_written
    378.00 ± 66%    +346.7%       1688 ±103%  numa-vmstat.node1.nr_zone_inactive_anon
     47945 ±  3%     +32.1%      63332 ±  9%  meminfo.AnonHugePages
     25623 ±  2%     +17.5%      30095 ±  2%  meminfo.Inactive
     18131            +8.8%      19719        meminfo.Inactive(anon)
      7491 ±  8%     +38.5%      10375 ±  8%  meminfo.Inactive(file)
     98549           +37.2%     135204        meminfo.KReclaimable
     98549           +37.2%     135204        meminfo.SReclaimable
    244175            +9.9%     268384        meminfo.SUnreclaim
     31605 ±  2%     +57.8%      49875 ± 11%  meminfo.Shmem
    342725           +17.8%     403589        meminfo.Slab
     63159           -30.5%      43885 ±  5%  meminfo.max_used_kB
     16.00          +346.1%      71.38 ±116%  sched_debug.cfs_rq:/.runnable_load_avg.max
    613880 ±  4%     +13.9%     698933 ±  8%  sched_debug.cpu.avg_idle.avg
      2219 ± 51%   +6020.3%     135810 ± 70%  sched_debug.cpu.avg_idle.min
    258921 ±  3%     -22.1%     201779 ± 12%  sched_debug.cpu.avg_idle.stddev
     30262 ±  2%     +75.0%      52944 ± 24%  sched_debug.cpu.clock.avg
     30265 ±  2%     +75.0%      52950 ± 24%  sched_debug.cpu.clock.max
     30258 ±  2%     +74.9%      52936 ± 24%  sched_debug.cpu.clock.min
      1.73 ±  7%    +104.3%       3.54 ± 64%  sched_debug.cpu.clock.stddev
     30262 ±  2%     +75.0%      52944 ± 24%  sched_debug.cpu.clock_task.avg
     30265 ±  2%     +75.0%      52950 ± 24%  sched_debug.cpu.clock_task.max
     30258 ±  2%     +74.9%      52936 ± 24%  sched_debug.cpu.clock_task.min
      1.73 ±  7%    +104.3%       3.54 ± 64%  sched_debug.cpu.clock_task.stddev
      0.01 ±173%  +47900.0%       3.33 ±109%  sched_debug.cpu.nr_uninterruptible.avg
    -18.25           +98.6%     -36.25        sched_debug.cpu.nr_uninterruptible.min
     30259 ±  2%     +74.9%      52938 ± 24%  sched_debug.cpu_clk
     27548 ±  2%     +82.3%      50227 ± 25%  sched_debug.ktime
     30597 ±  2%     +74.1%      53282 ± 24%  sched_debug.sched_clk
     90738            +2.7%      93156        proc-vmstat.nr_active_anon
      1651 ±  9%     +24.4%       2054 ± 10%  proc-vmstat.nr_dirty
    281450            +1.7%     286223        proc-vmstat.nr_file_pages
      4532            +8.8%       4929        proc-vmstat.nr_inactive_anon
      1886 ±  8%     +43.0%       2698 ±  9%  proc-vmstat.nr_inactive_file
     57113            -6.6%      53362 ±  5%  proc-vmstat.nr_kernel_stack
      7081            +6.1%       7517 ±  2%  proc-vmstat.nr_mapped
      7901 ±  2%     +57.8%      12469 ± 11%  proc-vmstat.nr_shmem
     24642           +37.1%      33789        proc-vmstat.nr_slab_reclaimable
     61028            +9.9%      67087        proc-vmstat.nr_slab_unreclaimable
      8606 ± 25%     +70.9%      14710 ± 15%  proc-vmstat.nr_written
     90738            +2.7%      93156        proc-vmstat.nr_zone_active_anon
      4532            +8.8%       4929        proc-vmstat.nr_zone_inactive_anon
      1886 ±  8%     +43.0%       2698 ±  9%  proc-vmstat.nr_zone_inactive_file
    615.25 ±139%    +561.8%       4071 ± 62%  proc-vmstat.numa_pages_migrated
    289105 ±  2%     +23.6%     357466 ±  2%  proc-vmstat.pgfault
    615.25 ±139%    +561.8%       4071 ± 62%  proc-vmstat.pgmigrate_success
    136493 ±  5%    +242.5%     467494 ±  3%  proc-vmstat.pgpgout
      1970 ±  5%     +18.1%       2326 ±  3%  slabinfo.blkdev_ioc.active_objs
      1970 ±  5%     +18.1%       2326 ±  3%  slabinfo.blkdev_ioc.num_objs
      3708 ±  3%    +639.3%      27420 ±  5%  slabinfo.dmaengine-unmap-16.active_objs
     93.50 ±  2%    +803.7%     845.00        slabinfo.dmaengine-unmap-16.active_slabs
      3952 ±  2%    +798.3%      35504        slabinfo.dmaengine-unmap-16.num_objs
     93.50 ±  2%    +803.7%     845.00        slabinfo.dmaengine-unmap-16.num_slabs
      2324           +12.8%       2622        slabinfo.ip6-frags.active_objs
      2324           +12.8%       2622        slabinfo.ip6-frags.num_objs
      3633           +36.6%       4962 ±  2%  slabinfo.kmalloc-128.active_objs
      3633           +36.6%       4962 ±  2%  slabinfo.kmalloc-128.num_objs
     27732          +330.8%     119465 ±  5%  slabinfo.kmalloc-16.active_objs
    108.75          +331.3%     469.00 ±  5%  slabinfo.kmalloc-16.active_slabs
     27840          +331.9%     120245 ±  5%  slabinfo.kmalloc-16.num_objs
    108.75          +331.3%     469.00 ±  5%  slabinfo.kmalloc-16.num_slabs
      1210           +19.1%       1441 ±  3%  slabinfo.kmalloc-4k.active_objs
      1225           +18.5%       1452 ±  3%  slabinfo.kmalloc-4k.num_objs
      9390          +250.5%      32916 ±  4%  slabinfo.kmalloc-512.active_objs
    305.50          +296.2%       1210 ±  2%  slabinfo.kmalloc-512.active_slabs
      9791          +295.7%      38742 ±  2%  slabinfo.kmalloc-512.num_objs
    305.50          +296.2%       1210 ±  2%  slabinfo.kmalloc-512.num_slabs
      1316           +23.8%       1629        slabinfo.kmalloc-8k.active_objs
      1322           +28.1%       1694        slabinfo.kmalloc-8k.num_objs
     26510 ±  2%     +22.0%      32331 ±  9%  slabinfo.radix_tree_node.active_objs
    473.50 ±  2%     +23.0%     582.50 ±  9%  slabinfo.radix_tree_node.active_slabs
     26551 ±  2%     +23.0%      32654 ±  9%  slabinfo.radix_tree_node.num_objs
    473.50 ±  2%     +23.0%     582.50 ±  9%  slabinfo.radix_tree_node.num_slabs
      1902           +12.8%       2145        slabinfo.xfs_btree_cur.active_objs
      1902           +12.8%       2145        slabinfo.xfs_btree_cur.num_objs
      2088 ±  6%     +25.3%       2616        slabinfo.xfs_buf.active_objs
      2088 ±  6%     +25.3%       2616        slabinfo.xfs_buf.num_objs
      3141 ±  4%     +22.4%       3844        slabinfo.xfs_buf_item.active_objs
      3141 ±  4%     +22.4%       3844        slabinfo.xfs_buf_item.num_objs
      1760           +12.6%       1983        slabinfo.xfs_da_state.active_objs
      1760           +12.6%       1983        slabinfo.xfs_da_state.num_objs
      1636 ±  3%     +68.9%       2765 ±  2%  slabinfo.xfs_efd_item.active_objs
      1636 ±  3%     +68.9%       2765 ±  2%  slabinfo.xfs_efd_item.num_objs
      2704 ±  3%    +876.7%      26413 ±  6%  slabinfo.xfs_inode.active_objs
     97.75 ±  4%    +985.4%       1061        slabinfo.xfs_inode.active_slabs
      3142 ±  4%    +981.0%      33967        slabinfo.xfs_inode.num_objs
     97.75 ±  4%    +985.4%       1061        slabinfo.xfs_inode.num_slabs
 6.732e+09           -11.8%  5.938e+09 ±  5%  perf-stat.i.branch-instructions
  79001438 ±  3%     -31.3%   54255443 ±  4%  perf-stat.i.branch-misses
  21293936 ±  7%     -33.2%   14225969 ±  3%  perf-stat.i.cache-misses
  94432275 ± 12%     -40.5%   56195402 ± 12%  perf-stat.i.cache-references
     45374           -62.9%      16826 ±  2%  perf-stat.i.context-switches
      1.72 ± 17%     +77.6%       3.05 ±  8%  perf-stat.i.cpi
 4.018e+10 ±  2%    +149.8%  1.003e+11 ±  6%  perf-stat.i.cpu-cycles
      1252 ± 31%     -45.3%     685.42 ±  8%  perf-stat.i.cpu-migrations
      2504 ±  9%    +136.5%       5924 ±  5%  perf-stat.i.cycles-between-cache-misses
  10043892 ±  3%     -41.3%    5896357 ±  7%  perf-stat.i.dTLB-load-misses
 9.848e+09            -8.8%  8.977e+09 ±  5%  perf-stat.i.dTLB-loads
    181346 ± 13%     -49.7%      91182 ± 31%  perf-stat.i.dTLB-store-misses
 5.606e+09           -31.8%  3.826e+09 ±  4%  perf-stat.i.dTLB-stores
   8571284           -39.5%    5185323 ±  4%  perf-stat.i.iTLB-load-misses
   9289772 ±  5%     -38.9%    5675605 ± 13%  perf-stat.i.iTLB-loads
  3.39e+10           -10.5%  3.034e+10 ±  5%  perf-stat.i.instructions
      0.74 ±  2%     -46.7%       0.40 ±  3%  perf-stat.i.ipc
      6021           -14.5%       5149 ±  2%  perf-stat.i.minor-faults
   3332988 ± 14%     -31.9%    2270988 ±  4%  perf-stat.i.node-load-misses
   1658009 ±  6%     -40.3%     989211 ±  5%  perf-stat.i.node-loads
     26.86 ±  9%     +10.2       37.10 ±  2%  perf-stat.i.node-store-miss-rate%
   4189562 ±  2%     -39.5%    2532889 ±  4%  perf-stat.i.node-stores
      6021           -14.5%       5149 ±  2%  perf-stat.i.page-faults
      2.78 ± 11%     -33.4%       1.85 ± 12%  perf-stat.overall.MPKI
      1.17 ±  2%      -0.3        0.91        perf-stat.overall.branch-miss-rate%
      1.18          +178.8%       3.30        perf-stat.overall.cpi
      1900 ±  9%    +270.7%       7045 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.10 ±  2%      -0.0        0.07 ±  5%  perf-stat.overall.dTLB-load-miss-rate%
      3955           +47.9%       5852        perf-stat.overall.instructions-per-iTLB-miss
      0.84           -64.1%       0.30        perf-stat.overall.ipc
     17.47 ± 16%      +6.6       24.11        perf-stat.overall.node-store-miss-rate%
 6.565e+09           -11.0%  5.843e+09 ±  5%  perf-stat.ps.branch-instructions
  77051640 ±  3%     -30.7%   53394071 ±  4%  perf-stat.ps.branch-misses
  20767108 ±  7%     -32.6%   13999643 ±  3%  perf-stat.ps.cache-misses
  92101166 ± 12%     -40.0%   55304912 ± 12%  perf-stat.ps.cache-references
     44251           -62.6%      16558 ±  2%  perf-stat.ps.context-switches
 3.918e+10 ±  2%    +152.0%  9.874e+10 ±  6%  perf-stat.ps.cpu-cycles
      1221 ± 31%     -44.8%     674.55 ±  8%  perf-stat.ps.cpu-migrations
   9795401 ±  3%     -40.8%    5802426 ±  7%  perf-stat.ps.dTLB-load-misses
 9.604e+09            -8.0%  8.834e+09 ±  5%  perf-stat.ps.dTLB-loads
    176901 ± 13%     -49.3%      89755 ± 31%  perf-stat.ps.dTLB-store-misses
 5.467e+09           -31.1%  3.765e+09 ±  4%  perf-stat.ps.dTLB-stores
   8359064           -39.0%    5102649 ±  4%  perf-stat.ps.iTLB-load-misses
   9060035 ±  5%     -38.3%    5585619 ± 13%  perf-stat.ps.iTLB-loads
 3.306e+10            -9.7%  2.986e+10 ±  5%  perf-stat.ps.instructions
      5878           -13.7%       5071 ±  2%  perf-stat.ps.minor-faults
   3250486 ± 14%     -31.2%    2234779 ±  4%  perf-stat.ps.node-load-misses
   1616982 ±  6%     -39.8%     973433 ±  4%  perf-stat.ps.node-loads
   4085833 ±  2%     -39.0%    2492523 ±  4%  perf-stat.ps.node-stores
      5878           -13.7%       5071 ±  2%  perf-stat.ps.page-faults
 1.369e+12           +40.8%  1.928e+12        perf-stat.total.instructions
     10189 ±  4%     -21.0%       8053 ±  9%  softirqs.CPU0.RCU
     20822 ±  7%     +39.9%      29137 ±  2%  softirqs.CPU0.TIMER
      9974 ±  4%     -23.7%       7606 ±  8%  softirqs.CPU1.RCU
     19331 ±  2%     +43.8%      27788        softirqs.CPU1.TIMER
     20495 ±  6%     +49.8%      30696 ± 20%  softirqs.CPU10.TIMER
      9678 ±  4%     -21.8%       7567 ±  8%  softirqs.CPU11.RCU
     19183 ±  4%     +44.0%      27627 ±  2%  softirqs.CPU11.TIMER
      9717 ±  2%     -20.3%       7744 ±  8%  softirqs.CPU12.RCU
     19610 ±  2%     +41.8%      27816 ±  3%  softirqs.CPU12.TIMER
      9975 ±  4%     -23.7%       7615 ±  9%  softirqs.CPU13.RCU
     19641 ±  3%     +43.9%      28262 ±  3%  softirqs.CPU13.TIMER
      9934 ±  3%     -18.9%       8061 ± 16%  softirqs.CPU14.RCU
     19792 ±  4%     +37.8%      27277 ±  3%  softirqs.CPU14.TIMER
     19248           +45.0%      27911 ±  2%  softirqs.CPU15.TIMER
     18812 ±  4%     +46.4%      27541        softirqs.CPU16.TIMER
      9643 ±  7%     -21.7%       7550 ± 15%  softirqs.CPU17.RCU
     21982 ± 14%     +38.7%      30500 ± 16%  softirqs.CPU17.TIMER
     11100 ±  3%     -23.5%       8488 ± 12%  softirqs.CPU18.RCU
     21432 ± 10%     +40.8%      30174 ± 12%  softirqs.CPU18.TIMER
     10254 ±  2%     -24.1%       7786 ± 12%  softirqs.CPU19.RCU
     20127 ±  4%     +37.2%      27610 ±  7%  softirqs.CPU19.TIMER
     19067 ±  4%     +55.2%      29600 ±  2%  softirqs.CPU2.TIMER
     10238 ±  2%     -24.3%       7749 ± 19%  softirqs.CPU20.RCU
     20680 ±  7%     +44.2%      29819 ± 12%  softirqs.CPU20.TIMER
     11414 ± 11%     -34.9%       7429 ± 12%  softirqs.CPU21.RCU
     20495 ±  9%     +33.5%      27370 ±  5%  softirqs.CPU21.TIMER
     10724 ±  9%     -29.1%       7602 ± 10%  softirqs.CPU22.RCU
     22190 ± 16%     +26.3%      28035 ±  6%  softirqs.CPU22.TIMER
     20468 ±  7%     +50.4%      30776 ± 15%  softirqs.CPU23.TIMER
      9837 ±  2%     -22.4%       7630 ± 12%  softirqs.CPU24.RCU
     19337 ±  5%     +46.7%      28366 ±  4%  softirqs.CPU24.TIMER
     10048           -24.3%       7606 ± 14%  softirqs.CPU25.RCU
     19879 ±  4%     +39.0%      27629 ±  4%  softirqs.CPU25.TIMER
     10113           -25.6%       7523 ± 12%  softirqs.CPU26.RCU
     19752 ±  4%     +41.4%      27929 ±  5%  softirqs.CPU26.TIMER
      9885 ±  2%     -25.2%       7393 ± 12%  softirqs.CPU27.RCU
     19471 ±  4%     +42.8%      27802 ±  4%  softirqs.CPU27.TIMER
     10655 ±  9%     -24.8%       8016 ± 13%  softirqs.CPU28.RCU
     19580 ±  6%     +47.0%      28775 ±  6%  softirqs.CPU28.TIMER
      9955 ±  3%     -23.0%       7664 ± 14%  softirqs.CPU29.RCU
     19909 ±  3%     +40.5%      27979 ±  4%  softirqs.CPU29.TIMER
     11278 ± 17%     -22.8%       8705 ± 12%  softirqs.CPU3.RCU
     20144 ±  4%     +41.1%      28431 ±  2%  softirqs.CPU3.TIMER
     19379 ±  5%     +43.7%      27839 ±  5%  softirqs.CPU30.TIMER
     10566 ± 10%     -33.6%       7018 ± 13%  softirqs.CPU31.RCU
     19802 ±  4%     +39.2%      27560 ±  5%  softirqs.CPU31.TIMER
      9621 ±  6%     -28.3%       6897 ± 11%  softirqs.CPU32.RCU
      9602 ±  4%     -27.3%       6978 ± 13%  softirqs.CPU33.RCU
     19288 ±  5%     +45.5%      28056 ±  7%  softirqs.CPU33.TIMER
      9711 ±  4%     -26.2%       7163 ± 11%  softirqs.CPU34.RCU
     19495 ±  4%     +46.3%      28528 ±  3%  softirqs.CPU34.TIMER
     19688 ±  5%     +47.7%      29082 ±  3%  softirqs.CPU35.TIMER
     10457 ±  2%     -25.5%       7791 ±  8%  softirqs.CPU36.RCU
     19860 ±  8%     +37.6%      27321        softirqs.CPU36.TIMER
     10690 ±  6%     -28.5%       7647 ±  9%  softirqs.CPU37.RCU
     20270 ± 11%     +36.2%      27614 ±  2%  softirqs.CPU37.TIMER
     10147           -25.1%       7597 ± 10%  softirqs.CPU38.RCU
     18995 ±  2%     +42.4%      27043        softirqs.CPU38.TIMER
     10330           -23.5%       7903 ±  8%  softirqs.CPU39.RCU
     19115           +44.2%      27561        softirqs.CPU39.TIMER
     12080 ± 17%     -34.5%       7913 ±  6%  softirqs.CPU4.RCU
     19952 ±  5%     +40.6%      28061        softirqs.CPU4.TIMER
     10233 ±  2%     -23.5%       7824 ±  9%  softirqs.CPU40.RCU
     18201 ±  9%     +50.6%      27413        softirqs.CPU40.TIMER
     10309 ±  4%     -28.4%       7381 ± 10%  softirqs.CPU41.RCU
     19269 ±  4%     +41.4%      27252        softirqs.CPU41.TIMER
     18614 ±  2%     +43.9%      26794 ±  2%  softirqs.CPU42.TIMER
     10415 ±  7%     -27.1%       7598 ± 10%  softirqs.CPU43.RCU
     18598 ±  2%     +46.2%      27199 ±  2%  softirqs.CPU43.TIMER
     10246 ±  2%     -27.9%       7392 ± 12%  softirqs.CPU44.RCU
     18630 ±  4%     +44.4%      26904        softirqs.CPU44.TIMER
     10857 ± 14%     -31.0%       7487 ± 10%  softirqs.CPU45.RCU
     10197 ±  2%     -29.0%       7242 ±  5%  softirqs.CPU46.RCU
     19247 ±  3%     +44.0%      27723 ±  3%  softirqs.CPU46.TIMER
     10174 ±  3%     -26.4%       7488 ± 11%  softirqs.CPU47.RCU
     18969 ±  6%     +43.5%      27222        softirqs.CPU47.TIMER
     10377 ±  5%     -28.6%       7411 ± 11%  softirqs.CPU48.RCU
     19583 ±  7%     +37.4%      26902 ±  3%  softirqs.CPU48.TIMER
     10084           -24.1%       7653 ± 11%  softirqs.CPU49.RCU
     18508 ±  4%     +47.1%      27224        softirqs.CPU49.TIMER
     10078 ±  3%     -22.2%       7842 ±  7%  softirqs.CPU5.RCU
     19160 ±  3%     +46.4%      28054        softirqs.CPU5.TIMER
     10208 ±  3%     -26.8%       7473 ±  8%  softirqs.CPU50.RCU
     18450 ±  2%     +47.9%      27292        softirqs.CPU50.TIMER
     10212 ±  2%     -27.9%       7367 ± 12%  softirqs.CPU51.RCU
     18613 ±  3%     +43.5%      26706 ±  3%  softirqs.CPU51.TIMER
     10012           -25.3%       7478 ± 10%  softirqs.CPU52.RCU
     18440 ±  3%     +45.8%      26888        softirqs.CPU52.TIMER
      9782 ±  6%     -23.7%       7467 ± 13%  softirqs.CPU53.RCU
     19168 ±  3%     +43.9%      27588 ±  2%  softirqs.CPU53.TIMER
      9896 ±  5%     -27.4%       7183 ± 10%  softirqs.CPU54.RCU
     20511 ± 11%     +39.6%      28632 ± 10%  softirqs.CPU54.TIMER
      9689 ±  5%     -30.5%       6732 ± 15%  softirqs.CPU55.RCU
     19413 ±  4%     +42.4%      27642 ±  8%  softirqs.CPU55.TIMER
      9719 ±  3%     -24.9%       7303 ± 19%  softirqs.CPU56.RCU
     19312 ±  4%     +43.8%      27773 ±  8%  softirqs.CPU56.TIMER
     20063 ± 15%     +36.1%      27306 ±  6%  softirqs.CPU57.TIMER
      9789 ± 14%     -28.6%       6990 ± 11%  softirqs.CPU58.RCU
     18743 ±  7%     +43.4%      26870 ±  6%  softirqs.CPU58.TIMER
      9592 ±  4%     -28.5%       6857 ± 18%  softirqs.CPU59.RCU
     19654 ±  5%     +40.1%      27534 ±  4%  softirqs.CPU59.TIMER
     11114 ± 23%     -29.7%       7818 ± 10%  softirqs.CPU6.RCU
     19544 ±  8%     +40.4%      27442 ±  5%  softirqs.CPU6.TIMER
     19208 ±  5%     +46.0%      28048 ±  8%  softirqs.CPU60.TIMER
     19390 ±  5%     +39.5%      27051 ±  5%  softirqs.CPU61.TIMER
      9604 ±  9%     -29.5%       6767 ± 14%  softirqs.CPU62.RCU
     19032 ±  5%     +40.1%      26660 ±  5%  softirqs.CPU62.TIMER
     19398 ±  5%     +39.1%      26985 ±  7%  softirqs.CPU63.TIMER
     18919 ±  5%     +48.7%      28127 ±  6%  softirqs.CPU64.TIMER
     19285 ±  7%     +41.8%      27352 ±  5%  softirqs.CPU65.TIMER
     18880 ±  6%     +46.7%      27689 ±  4%  softirqs.CPU66.TIMER
     18997 ±  6%     +39.9%      26579 ±  5%  softirqs.CPU67.TIMER
     18619 ±  6%     +46.3%      27247 ±  6%  softirqs.CPU68.TIMER
     18851 ±  5%     +46.1%      27546 ±  5%  softirqs.CPU69.TIMER
     19457 ±  2%     +43.3%      27888        softirqs.CPU7.TIMER
     18964 ±  5%     +43.5%      27211 ±  7%  softirqs.CPU70.TIMER
     18996 ±  5%     +58.8%      30160 ± 12%  softirqs.CPU71.TIMER
     19593 ±  3%     +54.6%      30290 ±  8%  softirqs.CPU8.TIMER
     18531 ±  2%     +46.4%      27131 ±  2%  softirqs.CPU9.TIMER
    715216           -24.4%     540757 ± 10%  softirqs.RCU
    557516           -11.4%     494067 ±  7%  softirqs.SCHED
   1405899 ±  3%     +42.9%    2008869 ±  3%  softirqs.TIMER
     86.00           +53.5%     132.00 ±  5%  interrupts.9:IO-APIC.9-fasteoi.acpi
     52948 ±  2%     +28.4%      67994 ±  4%  interrupts.CAL:Function_call_interrupts
    749.25           +28.5%     962.75 ±  3%  interrupts.CPU0.CAL:Function_call_interrupts
     86.00           +53.5%     132.00 ±  5%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    677.00 ± 19%     +43.3%     970.00        interrupts.CPU1.CAL:Function_call_interrupts
     97.50 ± 14%    +253.1%     344.25 ± 28%  interrupts.CPU10.RES:Rescheduling_interrupts
    746.75 ±  2%     +24.9%     932.75        interrupts.CPU11.CAL:Function_call_interrupts
      3026 ± 17%     +66.9%       5052 ±  2%  interrupts.CPU11.NMI:Non-maskable_interrupts
      3026 ± 17%     +66.9%       5052 ±  2%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    101.00 ± 11%    +220.8%     324.00 ± 14%  interrupts.CPU11.RES:Rescheduling_interrupts
    746.75 ±  2%     +25.1%     934.50 ±  6%  interrupts.CPU12.CAL:Function_call_interrupts
    101.75 ± 17%    +309.6%     416.75 ± 49%  interrupts.CPU12.RES:Rescheduling_interrupts
    744.50 ±  2%     +27.7%     951.00 ±  3%  interrupts.CPU13.CAL:Function_call_interrupts
      2787 ± 33%     +80.9%       5044 ±  4%  interrupts.CPU13.NMI:Non-maskable_interrupts
      2787 ± 33%     +80.9%       5044 ±  4%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    109.75 ± 30%    +194.1%     322.75 ± 22%  interrupts.CPU13.RES:Rescheduling_interrupts
    731.00 ±  4%     +31.3%     959.50 ±  4%  interrupts.CPU14.CAL:Function_call_interrupts
     98.25 ± 20%    +201.3%     296.00 ±  5%  interrupts.CPU14.RES:Rescheduling_interrupts
    748.25           +27.0%     950.50 ±  4%  interrupts.CPU15.CAL:Function_call_interrupts
    113.50 ± 16%    +191.0%     330.25 ± 12%  interrupts.CPU15.RES:Rescheduling_interrupts
      3052 ± 17%     +62.5%       4959 ±  4%  interrupts.CPU16.NMI:Non-maskable_interrupts
      3052 ± 17%     +62.5%       4959 ±  4%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
     96.25 ± 26%    +193.2%     282.25 ±  6%  interrupts.CPU16.RES:Rescheduling_interrupts
    748.00           +27.3%     952.25 ±  5%  interrupts.CPU17.CAL:Function_call_interrupts
    107.75 ± 20%    +539.0%     688.50 ± 96%  interrupts.CPU17.RES:Rescheduling_interrupts
    668.75 ± 19%     +42.1%     950.50 ±  4%  interrupts.CPU18.CAL:Function_call_interrupts
     82.75 ± 24%    +456.2%     460.25 ± 63%  interrupts.CPU18.RES:Rescheduling_interrupts
    740.25 ±  2%     +25.3%     927.75 ±  8%  interrupts.CPU19.CAL:Function_call_interrupts
    163.00 ± 77%    +210.6%     506.25 ± 44%  interrupts.CPU19.RES:Rescheduling_interrupts
    743.75 ±  2%     +24.6%     926.50 ±  9%  interrupts.CPU2.CAL:Function_call_interrupts
      2641 ± 16%     +90.4%       5027 ±  4%  interrupts.CPU2.NMI:Non-maskable_interrupts
      2641 ± 16%     +90.4%       5027 ±  4%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
    157.25 ± 41%     +96.2%     308.50 ±  7%  interrupts.CPU2.RES:Rescheduling_interrupts
    734.50 ±  3%     +30.2%     956.25 ±  4%  interrupts.CPU20.CAL:Function_call_interrupts
     84.75 ± 19%    +231.0%     280.50 ± 11%  interrupts.CPU20.RES:Rescheduling_interrupts
    701.25 ±  6%     +36.5%     957.50 ±  4%  interrupts.CPU21.CAL:Function_call_interrupts
     85.50 ± 18%    +235.1%     286.50 ± 19%  interrupts.CPU21.RES:Rescheduling_interrupts
    737.50 ±  2%     +29.8%     957.25 ±  4%  interrupts.CPU22.CAL:Function_call_interrupts
     84.25 ±  7%    +322.0%     355.50 ± 13%  interrupts.CPU22.RES:Rescheduling_interrupts
     82.00 ± 19%    +265.2%     299.50 ±  8%  interrupts.CPU23.RES:Rescheduling_interrupts
    742.00 ±  2%     +24.3%     922.25 ±  9%  interrupts.CPU24.CAL:Function_call_interrupts
     99.50 ± 18%    +169.8%     268.50 ±  9%  interrupts.CPU24.RES:Rescheduling_interrupts
    735.00 ±  3%     +30.4%     958.25 ±  4%  interrupts.CPU25.CAL:Function_call_interrupts
     89.00 ±  6%    +241.6%     304.00 ±  6%  interrupts.CPU25.RES:Rescheduling_interrupts
    739.75           +28.8%     953.00 ±  4%  interrupts.CPU26.CAL:Function_call_interrupts
     85.25 ±  8%    +225.5%     277.50 ± 13%  interrupts.CPU26.RES:Rescheduling_interrupts
     80.50 ± 19%    +244.7%     277.50 ± 12%  interrupts.CPU27.RES:Rescheduling_interrupts
    743.00           +28.7%     956.25 ±  4%  interrupts.CPU28.CAL:Function_call_interrupts
     97.50 ± 11%    +179.5%     272.50 ±  3%  interrupts.CPU28.RES:Rescheduling_interrupts
    718.00 ±  7%     +33.2%     956.50 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
    134.25 ± 46%     +87.5%     251.75 ± 12%  interrupts.CPU29.RES:Rescheduling_interrupts
     95.75 ± 17%    +227.2%     313.25 ±  6%  interrupts.CPU3.RES:Rescheduling_interrupts
    740.75 ±  2%     +29.2%     957.25 ±  4%  interrupts.CPU30.CAL:Function_call_interrupts
     87.75 ± 10%    +204.3%     267.00 ± 12%  interrupts.CPU30.RES:Rescheduling_interrupts
    659.75 ± 22%     +44.6%     953.75 ±  4%  interrupts.CPU31.CAL:Function_call_interrupts
     86.75 ± 18%    +229.1%     285.50 ± 16%  interrupts.CPU31.RES:Rescheduling_interrupts
    718.75 ±  6%     +33.7%     960.75 ±  5%  interrupts.CPU32.CAL:Function_call_interrupts
     90.75 ± 21%    +251.0%     318.50 ±  8%  interrupts.CPU32.RES:Rescheduling_interrupts
    764.00 ±  6%     +25.4%     957.75 ±  4%  interrupts.CPU33.CAL:Function_call_interrupts
     78.00 ± 15%    +264.1%     284.00 ± 12%  interrupts.CPU33.RES:Rescheduling_interrupts
    670.00 ± 16%     +41.6%     949.00 ±  4%  interrupts.CPU34.CAL:Function_call_interrupts
     82.50 ± 12%    +245.2%     284.75 ± 11%  interrupts.CPU34.RES:Rescheduling_interrupts
    692.00 ±  9%     +35.2%     935.25 ±  4%  interrupts.CPU35.CAL:Function_call_interrupts
     95.75 ± 18%    +268.9%     353.25 ± 33%  interrupts.CPU35.RES:Rescheduling_interrupts
    762.75 ±  3%     +28.8%     982.50 ±  4%  interrupts.CPU36.CAL:Function_call_interrupts
     90.25 ± 14%    +265.1%     329.50 ± 12%  interrupts.CPU36.RES:Rescheduling_interrupts
    743.25 ±  2%     +28.7%     956.25 ±  4%  interrupts.CPU37.CAL:Function_call_interrupts
     96.25 ± 22%    +228.3%     316.00 ±  8%  interrupts.CPU37.RES:Rescheduling_interrupts
    743.50 ±  2%     +28.1%     952.50 ±  4%  interrupts.CPU38.CAL:Function_call_interrupts
    108.50 ± 26%    +166.8%     289.50 ±  5%  interrupts.CPU38.RES:Rescheduling_interrupts
    744.75 ±  2%     +28.3%     955.50 ±  4%  interrupts.CPU39.CAL:Function_call_interrupts
    102.75 ± 15%    +180.5%     288.25 ± 12%  interrupts.CPU39.RES:Rescheduling_interrupts
    754.75           +27.0%     958.50 ±  4%  interrupts.CPU4.CAL:Function_call_interrupts
    744.00 ±  2%     +20.0%     893.00 ±  9%  interrupts.CPU40.CAL:Function_call_interrupts
    114.00 ±  8%    +160.3%     296.75 ±  6%  interrupts.CPU40.RES:Rescheduling_interrupts
    743.00 ±  2%     +27.9%     950.50 ±  4%  interrupts.CPU41.CAL:Function_call_interrupts
    110.00 ± 19%    +192.0%     321.25 ± 15%  interrupts.CPU41.RES:Rescheduling_interrupts
    742.50 ±  2%     +28.4%     953.25 ±  4%  interrupts.CPU42.CAL:Function_call_interrupts
    102.50 ± 12%    +193.4%     300.75 ±  7%  interrupts.CPU42.RES:Rescheduling_interrupts
    741.50           +28.6%     953.50 ±  4%  interrupts.CPU43.CAL:Function_call_interrupts
     96.75 ± 21%    +196.4%     286.75 ±  7%  interrupts.CPU43.RES:Rescheduling_interrupts
    742.75 ±  2%     +22.9%     912.50 ±  4%  interrupts.CPU44.CAL:Function_call_interrupts
     94.00 ± 27%    +221.0%     301.75 ± 13%  interrupts.CPU44.RES:Rescheduling_interrupts
    740.75 ±  2%     +22.3%     905.75 ±  5%  interrupts.CPU45.CAL:Function_call_interrupts
     95.50 ± 23%    +215.4%     301.25 ±  6%  interrupts.CPU45.RES:Rescheduling_interrupts
    757.75 ±  3%     +25.8%     953.00 ±  4%  interrupts.CPU46.CAL:Function_call_interrupts
    106.75 ± 14%    +154.1%     271.25 ± 13%  interrupts.CPU46.RES:Rescheduling_interrupts
    744.00 ±  2%     +28.2%     953.50 ±  4%  interrupts.CPU47.CAL:Function_call_interrupts
    106.25 ± 17%    +174.1%     291.25 ± 12%  interrupts.CPU47.RES:Rescheduling_interrupts
    742.50           +21.7%     903.75 ± 11%  interrupts.CPU48.CAL:Function_call_interrupts
    108.75 ± 14%    +163.9%     287.00 ±  8%  interrupts.CPU48.RES:Rescheduling_interrupts
    746.50 ±  2%     +27.4%     951.00 ±  4%  interrupts.CPU49.CAL:Function_call_interrupts
     97.50 ± 20%    +223.6%     315.50 ±  7%  interrupts.CPU49.RES:Rescheduling_interrupts
    733.00 ±  4%     +31.1%     961.00 ±  4%  interrupts.CPU5.CAL:Function_call_interrupts
    102.50 ± 17%    +197.3%     304.75 ±  7%  interrupts.CPU5.RES:Rescheduling_interrupts
    744.25 ±  2%     +27.3%     947.75 ±  3%  interrupts.CPU50.CAL:Function_call_interrupts
    102.50 ± 17%    +241.0%     349.50 ± 26%  interrupts.CPU50.RES:Rescheduling_interrupts
    743.25 ±  2%     +28.1%     951.75 ±  4%  interrupts.CPU51.CAL:Function_call_interrupts
    100.00 ± 23%    +203.8%     303.75 ±  6%  interrupts.CPU51.RES:Rescheduling_interrupts
    746.00 ±  2%     +28.0%     955.25 ±  4%  interrupts.CPU52.CAL:Function_call_interrupts
      3028 ± 18%     +64.1%       4969 ±  4%  interrupts.CPU52.NMI:Non-maskable_interrupts
      3028 ± 18%     +64.1%       4969 ±  4%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
     89.50 ± 20%    +231.3%     296.50 ±  9%  interrupts.CPU52.RES:Rescheduling_interrupts
      3105 ± 18%     +61.7%       5022 ±  3%  interrupts.CPU53.NMI:Non-maskable_interrupts
      3105 ± 18%     +61.7%       5022 ±  3%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
     97.50 ± 15%    +194.1%     286.75 ±  5%  interrupts.CPU53.RES:Rescheduling_interrupts
    743.00 ±  2%     +28.7%     956.25 ±  4%  interrupts.CPU54.CAL:Function_call_interrupts
      2719 ± 24%     +80.0%       4895 ±  3%  interrupts.CPU54.NMI:Non-maskable_interrupts
      2719 ± 24%     +80.0%       4895 ±  3%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
     88.50 ± 16%    +208.5%     273.00 ±  7%  interrupts.CPU54.RES:Rescheduling_interrupts
    741.00 ±  2%     +29.0%     955.75 ±  3%  interrupts.CPU55.CAL:Function_call_interrupts
     83.75 ± 20%    +253.4%     296.00 ± 14%  interrupts.CPU55.RES:Rescheduling_interrupts
    743.25 ±  2%     +28.3%     953.50 ±  4%  interrupts.CPU56.CAL:Function_call_interrupts
      2693 ± 23%     +84.6%       4972 ±  3%  interrupts.CPU56.NMI:Non-maskable_interrupts
      2693 ± 23%     +84.6%       4972 ±  3%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
     80.75 ± 24%    +227.2%     264.25 ± 12%  interrupts.CPU56.RES:Rescheduling_interrupts
    740.50 ±  2%     +28.0%     947.75 ±  3%  interrupts.CPU57.CAL:Function_call_interrupts
      2697 ± 25%     +84.8%       4986 ±  4%  interrupts.CPU57.NMI:Non-maskable_interrupts
      2697 ± 25%     +84.8%       4986 ±  4%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
     77.25 ± 20%    +284.8%     297.25 ± 15%  interrupts.CPU57.RES:Rescheduling_interrupts
    741.75           +28.4%     952.50 ±  4%  interrupts.CPU58.CAL:Function_call_interrupts
      2653 ± 24%     +88.1%       4991 ±  2%  interrupts.CPU58.NMI:Non-maskable_interrupts
      2653 ± 24%     +88.1%       4991 ±  2%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
     81.50 ± 27%    +229.4%     268.50 ± 12%  interrupts.CPU58.RES:Rescheduling_interrupts
    729.50 ±  4%     +30.7%     953.25 ±  4%  interrupts.CPU59.CAL:Function_call_interrupts
      2732 ± 27%     +84.6%       5044        interrupts.CPU59.NMI:Non-maskable_interrupts
      2732 ± 27%     +84.6%       5044        interrupts.CPU59.PMI:Performance_monitoring_interrupts
     75.75 ± 16%    +281.2%     288.75 ± 10%  interrupts.CPU59.RES:Rescheduling_interrupts
    729.25 ±  5%     +31.2%     956.75 ±  4%  interrupts.CPU6.CAL:Function_call_interrupts
    113.25 ± 38%    +165.3%     300.50 ± 10%  interrupts.CPU6.RES:Rescheduling_interrupts
    740.25           +28.7%     952.75 ±  4%  interrupts.CPU60.CAL:Function_call_interrupts
      2729 ± 24%     +82.2%       4972 ±  4%  interrupts.CPU60.NMI:Non-maskable_interrupts
      2729 ± 24%     +82.2%       4972 ±  4%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
     74.25 ± 11%    +338.0%     325.25 ± 29%  interrupts.CPU60.RES:Rescheduling_interrupts
    728.25 ±  3%     +30.9%     953.00 ±  3%  interrupts.CPU61.CAL:Function_call_interrupts
      2606 ± 25%     +93.6%       5045 ±  3%  interrupts.CPU61.NMI:Non-maskable_interrupts
      2606 ± 25%     +93.6%       5045 ±  3%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
     82.25 ± 16%    +228.0%     269.75 ±  9%  interrupts.CPU61.RES:Rescheduling_interrupts
    735.75 ±  2%     +30.2%     958.00 ±  3%  interrupts.CPU62.CAL:Function_call_interrupts
     81.25 ± 23%    +262.2%     294.25 ± 15%  interrupts.CPU62.RES:Rescheduling_interrupts
    743.50 ±  2%     +28.9%     958.00 ±  3%  interrupts.CPU63.CAL:Function_call_interrupts
    743.00 ±  2%     +28.9%     958.00 ±  3%  interrupts.CPU64.CAL:Function_call_interrupts
     77.75 ± 28%    +266.2%     284.75 ± 10%  interrupts.CPU64.RES:Rescheduling_interrupts
    725.25 ±  2%     +32.0%     957.50 ±  3%  interrupts.CPU65.CAL:Function_call_interrupts
     76.25 ± 19%    +283.3%     292.25 ± 18%  interrupts.CPU65.RES:Rescheduling_interrupts
    705.75 ± 10%     +36.5%     963.00 ±  4%  interrupts.CPU66.CAL:Function_call_interrupts
     71.50 ± 17%    +282.5%     273.50 ± 14%  interrupts.CPU66.RES:Rescheduling_interrupts
    728.75           +31.8%     960.50 ±  4%  interrupts.CPU67.CAL:Function_call_interrupts
     81.50 ± 16%    +231.3%     270.00 ± 12%  interrupts.CPU67.RES:Rescheduling_interrupts
    729.00           +31.4%     958.00 ±  4%  interrupts.CPU68.CAL:Function_call_interrupts
     76.75 ± 12%    +272.6%     286.00 ± 15%  interrupts.CPU68.RES:Rescheduling_interrupts
    732.50           +31.0%     959.75 ±  4%  interrupts.CPU69.CAL:Function_call_interrupts
     86.75 ± 11%    +238.9%     294.00 ± 13%  interrupts.CPU69.RES:Rescheduling_interrupts
    752.75 ±  3%     +29.7%     976.50 ±  7%  interrupts.CPU7.CAL:Function_call_interrupts
     95.00 ± 13%    +204.7%     289.50 ±  9%  interrupts.CPU7.RES:Rescheduling_interrupts
    739.75 ±  2%     +29.5%     958.25 ±  4%  interrupts.CPU70.CAL:Function_call_interrupts
     69.00 ± 17%    +290.2%     269.25 ± 11%  interrupts.CPU70.RES:Rescheduling_interrupts
    742.75 ±  2%     +29.2%     959.75 ±  4%  interrupts.CPU71.CAL:Function_call_interrupts
     80.25 ± 27%    +291.6%     314.25 ± 11%  interrupts.CPU71.RES:Rescheduling_interrupts
    747.00           +27.8%     955.00 ±  3%  interrupts.CPU8.CAL:Function_call_interrupts
    701.75 ±  9%     +31.6%     923.50 ±  3%  interrupts.CPU9.CAL:Function_call_interrupts
    112.00 ± 17%    +574.8%     755.75 ± 63%  interrupts.CPU9.RES:Rescheduling_interrupts
      9539 ± 13%    +147.7%      23630 ±  6%  interrupts.RES:Rescheduling_interrupts
     52.54 ±  6%     -31.5       20.99 ±  4%  perf-profile.calltrace.cycles-pp.write
     34.34 ± 11%     -28.3        6.00 ± 10%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     33.91 ± 11%     -28.0        5.94 ± 10%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     33.91 ± 11%     -28.0        5.94 ± 10%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     33.90 ± 11%     -28.0        5.94 ± 10%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     33.03 ± 11%     -27.3        5.68 ± 11%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     32.93 ± 11%     -27.3        5.64 ± 11%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     44.64 ±  6%     -26.4       18.27 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     31.31 ± 13%     -26.3        5.04 ± 13%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     44.31 ±  6%     -26.2       18.16 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     37.67 ±  6%     -22.1       15.52 ±  3%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     36.53 ±  6%     -21.4       15.10 ±  3%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     32.77 ±  6%     -19.1       13.66 ±  3%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     31.25 ±  6%     -18.2       13.06 ±  3%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     26.95 ±  6%     -16.1       10.89 ±  4%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     26.73 ±  6%     -15.9       10.82 ±  4%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     20.45 ±  6%     -12.1        8.38 ±  4%  perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
     10.73 ±  7%      -6.0        4.70 ±  4%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      6.12 ±  7%      -3.6        2.50 ±  2%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      5.95 ±  7%      -3.5        2.44 ±  2%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      4.77 ±  7%      -3.4        1.35 ±  3%  perf-profile.calltrace.cycles-pp.close
      4.69 ±  7%      -3.4        1.32 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.69 ±  7%      -3.4        1.33 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close
      4.67 ±  7%      -3.4        1.32 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.67 ±  7%      -3.4        1.32 ±  3%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.67 ±  7%      -3.4        1.31 ±  3%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.65 ±  7%      -3.3        1.31 ±  3%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      4.63 ±  7%      -3.3        1.30 ±  3%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
      4.88 ±  5%      -3.0        1.85 ±  4%  perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      4.20 ± 12%      -2.8        1.44 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      4.42 ±  5%      -2.7        1.75 ±  5%  perf-profile.calltrace.cycles-pp.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      3.77 ±  4%      -2.3        1.47 ±  6%  perf-profile.calltrace.cycles-pp.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      3.44 ±  7%      -2.2        1.27 ±  3%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      3.42 ±  7%      -2.2        1.26 ±  3%  perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.__dentry_kill.dput.__fput
      3.30 ±  5%      -2.1        1.16 ±  5%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
      4.06 ±  6%      -2.1        1.97 ±  5%  perf-profile.calltrace.cycles-pp.iomap_read_page_sync.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      3.16 ±  5%      -2.0        1.17 ±  6%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_read_page_sync.iomap_write_begin.iomap_write_actor.iomap_apply
      2.53 ±  6%      -1.6        0.95 ±  4%  perf-profile.calltrace.cycles-pp.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      2.65 ±  8%      -1.5        1.18 ±  2%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      2.04 ±  6%      -1.3        0.76 ±  4%  perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.07 ±  5%      -1.3        0.79 ±  4%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.15 ±  6%      -1.3        0.88 ±  4%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      1.53 ± 15%      -1.2        0.30 ±100%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.43 ± 16%      -1.1        0.28 ±100%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      1.26 ±  4%      -1.0        0.26 ±100%  perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
      1.69 ±  7%      -1.0        0.73 ±  4%  perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.37 ±  4%      -1.0        0.42 ± 57%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.53 ±  7%      -0.9        0.67 ±  4%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply
      1.37 ±  4%      -0.8        0.58 ±  6%  perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.33 ±  4%      -0.8        0.57 ±  6%  perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64
      1.33 ±  4%      -0.8        0.58 ±  6%  perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.30 ±  7%      -0.6        0.69 ±  2%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.34 ±  2%      -0.6        0.76 ±  2%  perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
      1.31 ±  2%      -0.6        0.76 ±  3%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_open
      1.06 ±  4%      -0.4        0.67 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
      1.08 ±  4%      -0.4        0.70 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
      4.07 ±  3%     +31.3       35.37 ±  2%  perf-profile.calltrace.cycles-pp.unlink
      3.98 ±  3%     +31.4       35.33 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
      3.98 ±  3%     +31.4       35.33 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      3.94 ±  3%     +31.4       35.32 ±  2%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      3.42 ±  3%     +32.2       35.67 ±  2%  perf-profile.calltrace.cycles-pp.creat
      3.38 ±  3%     +32.3       35.65 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      3.38 ±  3%     +32.3       35.65 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
      3.36 ±  3%     +32.3       35.65 ±  2%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      2.41 ±  3%     +32.3       34.69 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      3.33 ±  3%     +32.3       35.64 ±  2%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.33 ±  3%     +32.3       35.64 ±  2%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.85 ±  4%     +32.8       34.64 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.76 ±  3%     +33.2       33.92 ±  2%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
      1.56 ±  3%     +33.2       34.72 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_open.do_syscall_64
      1.21 ±  4%     +33.5       34.67 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_open
      0.00           +34.0       33.98 ±  2%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
     52.91 ±  6%     -31.7       21.24 ±  4%  perf-profile.children.cycles-pp.write
     34.34 ± 11%     -28.3        6.00 ± 10%  perf-profile.children.cycles-pp.do_idle
     34.34 ± 11%     -28.3        6.00 ± 10%  perf-profile.children.cycles-pp.secondary_startup_64
     34.34 ± 11%     -28.3        6.00 ± 10%  perf-profile.children.cycles-pp.cpu_startup_entry
     33.91 ± 11%     -28.0        5.94 ± 10%  perf-profile.children.cycles-pp.start_secondary
     33.45 ± 11%     -27.7        5.73 ± 11%  perf-profile.children.cycles-pp.cpuidle_enter
     33.45 ± 11%     -27.7        5.73 ± 11%  perf-profile.children.cycles-pp.cpuidle_enter_state
     31.56 ± 12%     -26.5        5.08 ± 13%  perf-profile.children.cycles-pp.intel_idle
     37.68 ±  6%     -22.1       15.53 ±  3%  perf-profile.children.cycles-pp.ksys_write
     36.57 ±  6%     -21.4       15.12 ±  3%  perf-profile.children.cycles-pp.vfs_write
     32.78 ±  6%     -19.1       13.66 ±  3%  perf-profile.children.cycles-pp.new_sync_write
     31.30 ±  6%     -18.2       13.08 ±  3%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     26.95 ±  6%     -16.1       10.90 ±  4%  perf-profile.children.cycles-pp.iomap_file_buffered_write
     26.78 ±  6%     -15.9       10.84 ±  4%  perf-profile.children.cycles-pp.iomap_apply
     20.52 ±  6%     -12.1        8.41 ±  4%  perf-profile.children.cycles-pp.iomap_write_actor
     10.74 ±  7%      -6.0        4.70 ±  3%  perf-profile.children.cycles-pp.iomap_write_begin
      6.17 ±  7%      -3.6        2.52 ±  2%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      5.99 ±  7%      -3.5        2.46 ±  3%  perf-profile.children.cycles-pp.pagecache_get_page
      4.77 ±  7%      -3.4        1.35 ±  3%  perf-profile.children.cycles-pp.close
      4.67 ±  7%      -3.4        1.31 ±  3%  perf-profile.children.cycles-pp.dput
      4.67 ±  7%      -3.4        1.31 ±  3%  perf-profile.children.cycles-pp.__fput
      4.67 ±  7%      -3.4        1.32 ±  3%  perf-profile.children.cycles-pp.task_work_run
      4.67 ±  7%      -3.3        1.32 ±  3%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      4.63 ±  7%      -3.3        1.30 ±  3%  perf-profile.children.cycles-pp.__dentry_kill
      5.23 ±  5%      -3.3        1.96 ±  5%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      4.90 ±  5%      -3.0        1.85 ±  4%  perf-profile.children.cycles-pp.iomap_write_end
      4.48 ±  5%      -2.7        1.77 ±  5%  perf-profile.children.cycles-pp.xfs_file_iomap_begin
      3.96 ±  6%      -2.5        1.45 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      3.84 ±  5%      -2.5        1.34 ±  5%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      3.82 ±  4%      -2.3        1.48 ±  6%  perf-profile.children.cycles-pp.xfs_file_iomap_begin_delay
      3.44 ±  7%      -2.2        1.27 ±  3%  perf-profile.children.cycles-pp.evict
      3.42 ±  7%      -2.2        1.27 ±  3%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      4.07 ±  6%      -2.1        1.99 ±  5%  perf-profile.children.cycles-pp.iomap_read_page_sync
      2.58 ±  6%      -1.6        0.96 ±  3%  perf-profile.children.cycles-pp.xfs_file_aio_write_checks
      2.92 ±  3%      -1.5        1.46 ±  2%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      2.65 ±  8%      -1.5        1.19 ±  2%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      2.08 ±  6%      -1.3        0.78 ±  4%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      2.15 ±  6%      -1.3        0.88 ±  5%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      1.64 ±  6%      -1.0        0.62 ±  6%  perf-profile.children.cycles-pp.xfs_ilock
      1.71 ±  6%      -1.0        0.74 ±  4%  perf-profile.children.cycles-pp.copyin
      1.63 ±  7%      -0.9        0.70 ±  4%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      1.97 ± 15%      -0.9        1.04 ±  8%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.83 ± 15%      -0.9        0.97 ± 10%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.37 ±  6%      -0.8        0.54 ±  5%  perf-profile.children.cycles-pp.down_write
      1.38 ±  4%      -0.8        0.55 ±  5%  perf-profile.children.cycles-pp.security_file_permission
      1.30 ±  8%      -0.8        0.48 ±  5%  perf-profile.children.cycles-pp.find_get_entry
      1.27 ± 10%      -0.8        0.46 ±  6%  perf-profile.children.cycles-pp.__lru_cache_add
      1.37 ±  4%      -0.8        0.58 ±  6%  perf-profile.children.cycles-pp.vfs_unlink
      1.26 ±  3%      -0.8        0.50 ±  5%  perf-profile.children.cycles-pp.selinux_file_permission
      1.21 ± 12%      -0.8        0.45 ±  3%  perf-profile.children.cycles-pp.release_pages
      1.33 ±  4%      -0.8        0.57 ±  6%  perf-profile.children.cycles-pp.xfs_remove
      1.33 ±  4%      -0.8        0.58 ±  6%  perf-profile.children.cycles-pp.xfs_vn_unlink
      1.18 ± 10%      -0.7        0.44 ±  7%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      1.19 ±  4%      -0.7        0.45 ±  7%  perf-profile.children.cycles-pp.___might_sleep
      1.15 ± 13%      -0.7        0.43 ±  4%  perf-profile.children.cycles-pp.__pagevec_release
      1.11 ±  9%      -0.7        0.40 ±  8%  perf-profile.children.cycles-pp.xfs_file_iomap_end
      1.30 ±  5%      -0.7        0.59 ±  3%  perf-profile.children.cycles-pp.__xfs_trans_commit
      1.16 ±  8%      -0.7        0.46 ±  5%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.26 ±  6%      -0.7        0.57 ±  2%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      1.09 ±  9%      -0.7        0.41 ±  3%  perf-profile.children.cycles-pp.__set_page_dirty
      1.08 ±  6%      -0.7        0.42 ±  8%  perf-profile.children.cycles-pp.xfs_file_write_iter
      1.01 ±  6%      -0.6        0.36 ±  5%  perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      1.33 ±  7%      -0.6        0.69 ±  2%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      0.99 ±  6%      -0.6        0.35 ±  5%  perf-profile.children.cycles-pp.xfs_iunlock
      0.97 ±  5%      -0.6        0.37 ±  5%  perf-profile.children.cycles-pp.__sb_start_write
      0.89 ±  4%      -0.6        0.30 ±  5%  perf-profile.children.cycles-pp.truncate_cleanup_page
      1.34 ±  2%      -0.6        0.76 ±  2%  perf-profile.children.cycles-pp.xfs_generic_create
      0.92 ±  4%      -0.6        0.36 ±  7%  perf-profile.children.cycles-pp.xfs_dir_removename
      1.31 ±  2%      -0.6        0.76 ±  3%  perf-profile.children.cycles-pp.xfs_create
      0.90 ±  6%      -0.6        0.35 ±  2%  perf-profile.children.cycles-pp.delete_from_page_cache_batch
      0.92 ±  8%      -0.5        0.37 ±  4%  perf-profile.children.cycles-pp.file_update_time
      0.90 ±  7%      -0.5        0.35 ±  7%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.90 ±  5%      -0.5        0.35 ±  6%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      0.85 ±  8%      -0.5        0.32 ±  4%  perf-profile.children.cycles-pp.xas_load
      0.80 ± 12%      -0.5        0.29 ±  3%  perf-profile.children.cycles-pp.account_page_dirtied
      0.75 ±  3%      -0.5        0.26 ±  5%  perf-profile.children.cycles-pp.__cancel_dirty_page
      0.79 ± 11%      -0.5        0.29 ±  6%  perf-profile.children.cycles-pp.__fdget_pos
      0.77 ± 19%      -0.5        0.29 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.79 ±  5%      -0.5        0.30 ±  9%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      1.12 ±  6%      -0.5        0.65 ± 12%  perf-profile.children.cycles-pp.xfs_inactive
      0.71 ±  5%      -0.5        0.25 ±  7%  perf-profile.children.cycles-pp.up_write
      0.66 ±  5%      -0.4        0.22 ±  3%  perf-profile.children.cycles-pp.xfs_break_layouts
      0.68 ±  6%      -0.4        0.25 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      1.12 ± 13%      -0.4        0.70 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.68 ±  4%      -0.4        0.27 ±  3%  perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
      0.68 ±  7%      -0.4        0.27 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.61 ±  3%      -0.4        0.20 ±  5%  perf-profile.children.cycles-pp.account_page_cleaned
      0.66 ±  6%      -0.4        0.26 ±  7%  perf-profile.children.cycles-pp.xfs_dir3_data_check
      0.66 ±  6%      -0.4        0.26 ±  7%  perf-profile.children.cycles-pp.__xfs_dir3_data_check
      0.64 ±  6%      -0.4        0.24 ±  4%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.62 ± 12%      -0.4        0.24 ±  7%  perf-profile.children.cycles-pp.__fget_light
      0.98 ±  6%      -0.4        0.61 ± 14%  perf-profile.children.cycles-pp.xfs_inactive_ifree
      0.43 ± 27%      -0.4        0.06 ± 20%  perf-profile.children.cycles-pp.start_kernel
      0.78 ±  8%      -0.4        0.42 ±  3%  perf-profile.children.cycles-pp.xas_store
      0.53 ± 26%      -0.4        0.16 ±  7%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.55 ± 20%      -0.3        0.22 ± 14%  perf-profile.children.cycles-pp.irq_exit
      0.52 ±  3%      -0.3        0.20 ±  4%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.40 ±  3%      -0.3        0.08 ±  8%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.40 ±  3%      -0.3        0.08 ±  8%  perf-profile.children.cycles-pp.activate_task
      0.40 ±  3%      -0.3        0.08 ±  8%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.51 ±  6%      -0.3        0.19 ±  6%  perf-profile.children.cycles-pp._cond_resched
      0.49 ±  4%      -0.3        0.18 ±  6%  perf-profile.children.cycles-pp.fsnotify
      0.51 ± 26%      -0.3        0.19 ± 11%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.49 ±  6%      -0.3        0.18 ±  7%  perf-profile.children.cycles-pp.xfs_errortag_test
      0.47 ±  2%      -0.3        0.17 ±  6%  perf-profile.children.cycles-pp.__inode_security_revalidate
      0.38 ±  4%      -0.3        0.07 ±  5%  perf-profile.children.cycles-pp.enqueue_entity
      0.48 ± 13%      -0.3        0.18 ±  2%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.46 ±  5%      -0.3        0.18 ±  6%  perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.35 ± 14%      -0.3        0.08 ±  5%  perf-profile.children.cycles-pp.try_to_wake_up
      0.41 ±  4%      -0.3        0.14 ±  3%  perf-profile.children.cycles-pp.xfs_break_leased_layouts
      0.51 ±  5%      -0.3        0.24 ± 13%  perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
      0.41 ±  6%      -0.3        0.15 ±  7%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      0.43 ± 22%      -0.3        0.17 ± 20%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.39 ±  4%      -0.3        0.13 ±  6%  perf-profile.children.cycles-pp.__x64_sys_write
      0.31 ±  3%      -0.2        0.06 ± 11%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.45 ± 18%      -0.2        0.20 ± 21%  perf-profile.children.cycles-pp.ktime_get
      0.41 ± 16%      -0.2        0.16 ±  9%  perf-profile.children.cycles-pp.xfs_vn_update_time
      0.37 ±  2%      -0.2        0.13 ±  8%  perf-profile.children.cycles-pp.xfs_buf_read_map
      0.43 ±  4%      -0.2        0.20 ±  4%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.36 ±  2%      -0.2        0.13 ±  5%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.35 ±  6%      -0.2        0.12 ±  4%  perf-profile.children.cycles-pp.unlock_page
      0.30 ±  4%      -0.2        0.08 ± 10%  perf-profile.children.cycles-pp.__schedule
      0.37 ± 16%      -0.2        0.16 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      0.34 ±  6%      -0.2        0.12 ±  8%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.32 ±  7%      -0.2        0.11 ±  3%  perf-profile.children.cycles-pp.xfs_dir_createname
      0.33 ±  3%      -0.2        0.11 ±  7%  perf-profile.children.cycles-pp.xfs_buf_find
      0.65 ±  5%      -0.2        0.45 ± 17%  perf-profile.children.cycles-pp.xfs_ifree
      0.26 ±  5%      -0.2        0.05 ±  8%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.24 ±  5%      -0.2        0.04 ± 57%  perf-profile.children.cycles-pp.arch_stack_walk
      0.31 ±  7%      -0.2        0.11 ±  4%  perf-profile.children.cycles-pp.xfs_dir2_node_addname
      0.32 ±  3%      -0.2        0.13 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.35 ± 18%      -0.2        0.15 ± 27%  perf-profile.children.cycles-pp.menu_select
      0.31 ±  4%      -0.2        0.12 ±  5%  perf-profile.children.cycles-pp.page_mapping
      0.32 ±  7%      -0.2        0.13 ± 10%  perf-profile.children.cycles-pp.current_time
      0.29 ± 12%      -0.2        0.11 ± 13%  perf-profile.children.cycles-pp.xas_start
      0.28 ±  4%      -0.2        0.10 ±  7%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.24 ±  4%      -0.2        0.06 ±  6%  perf-profile.children.cycles-pp.xfs_read_agi
      0.29 ±  3%      -0.2        0.11 ±  3%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.33 ±  8%      -0.2        0.15 ±  5%  perf-profile.children.cycles-pp.generic_write_checks
      0.62 ± 14%      -0.2        0.44 ± 13%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.29 ± 14%      -0.2        0.11 ±  3%  perf-profile.children.cycles-pp.node_dirty_ok
      0.28 ±  7%      -0.2        0.10 ±  7%  perf-profile.children.cycles-pp.rcu_all_qs
      0.24 ± 15%      -0.2        0.06 ±  6%  perf-profile.children.cycles-pp.xfs_buf_item_release
      0.29 ±  5%      -0.2        0.11 ±  9%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.22 ± 16%      -0.2        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_buf_unlock
      0.21 ± 17%      -0.2        0.05        perf-profile.children.cycles-pp.up
      0.26 ± 11%      -0.2        0.10 ± 12%  perf-profile.children.cycles-pp.pagevec_lookup_entries
      0.26 ± 11%      -0.2        0.10 ± 12%  perf-profile.children.cycles-pp.find_get_entries
      0.57 ±  3%      -0.2        0.42 ±  5%  perf-profile.children.cycles-pp.xfs_dir_ialloc
      0.57 ±  4%      -0.2        0.41 ±  6%  perf-profile.children.cycles-pp.xfs_ialloc
      0.21 ±  5%      -0.1        0.07        perf-profile.children.cycles-pp.iov_iter_advance
      0.23 ±  3%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.xfs_da_read_buf
      0.23 ±  9%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.free_unref_page_list
      0.18 ±  5%      -0.1        0.06 ± 14%  perf-profile.children.cycles-pp.schedule
      0.19 ± 11%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.xfs_dir3_data_entsize
      0.18 ± 24%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.rcu_core
      0.14 ±  9%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.file_modified
      0.20 ± 10%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.xfs_btree_lookup
      0.50 ±  3%      -0.1        0.39 ±  6%  perf-profile.children.cycles-pp.xfs_dialloc
      0.17 ±  7%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.lock_page_memcg
      0.19 ±  3%      -0.1        0.08 ±  8%  perf-profile.children.cycles-pp.xfs_dir_ino_validate
      0.18 ±  3%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.17 ±  4%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_vn_lookup
      0.20 ±  6%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.17 ±  6%      -0.1        0.06        perf-profile.children.cycles-pp.xfs_lookup
      0.17 ±  9%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.xfs_dir3_leaf_check_int
      0.25 ± 13%      -0.1        0.15 ± 24%  perf-profile.children.cycles-pp.mem_cgroup_try_charge
      0.43 ± 20%      -0.1        0.33 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
      0.20 ±  8%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.workingset_update_node
      0.18 ± 10%      -0.1        0.08 ± 12%  perf-profile.children.cycles-pp.xfs_iunlink_remove
      0.22 ±  9%      -0.1        0.12 ± 20%  perf-profile.children.cycles-pp.xfs_find_bdev_for_inode
      0.15 ±  6%      -0.1        0.05        perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.17 ±  9%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.xas_init_marks
      0.16 ±  4%      -0.1        0.06        perf-profile.children.cycles-pp.xfs_dir_lookup
      0.12 ±  4%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.18 ±  6%      -0.1        0.09 ± 17%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.15 ±  5%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.46 ±  4%      -0.1        0.36 ± 18%  perf-profile.children.cycles-pp.xfs_difree
      0.15 ±  8%      -0.1        0.06        perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.13 ± 12%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.xas_clear_mark
      0.15 ±  5%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_da3_node_read
      0.17 ± 13%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.generic_write_check_limits
      0.14 ±  6%      -0.1        0.05        perf-profile.children.cycles-pp.xfs_dir2_node_lookup
      0.12 ±  6%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.xfs_trans_unreserve_and_mod_sb
      0.15 ± 10%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__xa_set_mark
      0.15 ±  5%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__sb_end_write
      0.37 ±  5%      -0.1        0.29 ± 18%  perf-profile.children.cycles-pp.xfs_difree_inobt
      0.13 ±  3%      -0.1        0.06 ± 15%  perf-profile.children.cycles-pp.timestamp_truncate
      0.36 ± 22%      -0.1        0.28 ±  4%  perf-profile.children.cycles-pp.tick_sched_handle
      0.13 ±  6%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.alloc_pages_current
      0.11 ±  7%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.native_write_msr
      0.13 ±  5%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_verify_ino
      0.10 ±  5%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_btree_read_buf_block
      0.43 ±  4%      -0.1        0.36 ±  6%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      0.70 ±  4%      -0.1        0.64 ±  8%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      0.12 ±  5%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp._xfs_trans_bjoin
      0.13 ±  6%      -0.1        0.07 ± 14%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.12 ±  6%      -0.1        0.06 ± 16%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.12 ± 12%      -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
      0.09 ±  9%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.14 ±  6%      -0.1        0.08 ± 12%  perf-profile.children.cycles-pp.xfs_agino_range
      0.10 ±  4%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.20 ±  5%      -0.1        0.14 ± 10%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.10 ± 19%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.update_load_avg
      0.15 ±  2%      -0.0        0.11 ±  6%  perf-profile.children.cycles-pp.memcpy_erms
      0.16 ±  9%      -0.0        0.12 ± 21%  perf-profile.children.cycles-pp.xfs_bmapi_reserve_delalloc
      0.08 ±  5%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.xfs_iunlink
      0.09 ± 24%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.task_tick_fair
      0.00            +0.1        0.10 ± 51%  perf-profile.children.cycles-pp.xfs_inactive_worker
      0.06 ± 17%      +0.1        0.17 ± 29%  perf-profile.children.cycles-pp.worker_thread
      0.06 ±  9%      +0.1        0.17 ± 29%  perf-profile.children.cycles-pp.process_one_work
      0.08 ± 19%      +0.1        0.21 ± 22%  perf-profile.children.cycles-pp.ret_from_fork
      0.07 ± 14%      +0.1        0.21 ± 22%  perf-profile.children.cycles-pp.kthread
      0.00            +0.6        0.58 ± 24%  perf-profile.children.cycles-pp.xfs_inode_free_blocks
      0.00            +0.7        0.65 ± 13%  perf-profile.children.cycles-pp.xfs_inactive_inode
      0.00            +0.7        0.67 ± 14%  perf-profile.children.cycles-pp.xfs_ici_walk_fns
      0.00            +0.7        0.67 ± 14%  perf-profile.children.cycles-pp.xfs_ici_walk_ag
      4.07 ±  3%     +31.3       35.37 ±  2%  perf-profile.children.cycles-pp.unlink
      3.94 ±  3%     +31.4       35.32 ±  2%  perf-profile.children.cycles-pp.do_unlinkat
      3.43 ±  3%     +32.2       35.67 ±  2%  perf-profile.children.cycles-pp.creat
      3.37 ±  3%     +32.3       35.65 ±  2%  perf-profile.children.cycles-pp.do_sys_open
      3.33 ±  3%     +32.3       35.64 ±  2%  perf-profile.children.cycles-pp.do_filp_open
      3.33 ±  3%     +32.3       35.64 ±  2%  perf-profile.children.cycles-pp.path_openat
     56.86 ±  5%     +33.8       90.66        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     56.54 ±  5%     +34.0       90.56        perf-profile.children.cycles-pp.do_syscall_64
      3.97 ±  3%     +65.5       69.42 ±  2%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      3.07 ±  4%     +66.2       69.31 ±  2%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.84 ±  4%     +67.1       67.93 ±  2%  perf-profile.children.cycles-pp.osq_lock
     31.55 ± 12%     -26.5        5.08 ± 13%  perf-profile.self.cycles-pp.intel_idle
      6.23 ±  7%      -3.7        2.48 ±  5%  perf-profile.self.cycles-pp.do_syscall_64
      5.17 ±  5%      -3.2        1.93 ±  5%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      3.83 ±  5%      -2.5        1.33 ±  6%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      3.44 ±  6%      -2.2        1.25 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      2.90 ±  3%      -1.4        1.45 ±  2%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      1.60 ±  7%      -0.9        0.70 ±  4%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.13 ±  6%      -0.7        0.39 ±  7%  perf-profile.self.cycles-pp.xfs_file_iomap_begin_delay
      1.15 ±  4%      -0.7        0.43 ±  7%  perf-profile.self.cycles-pp.___might_sleep
      1.05 ±  6%      -0.6        0.40 ±  8%  perf-profile.self.cycles-pp.xfs_file_write_iter
      0.99 ±  6%      -0.6        0.35 ±  5%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      0.84 ±  8%      -0.5        0.32 ± 10%  perf-profile.self.cycles-pp.xfs_file_iomap_end
      0.68 ±  6%      -0.4        0.24 ±  7%  perf-profile.self.cycles-pp.up_write
      0.77 ±  7%      -0.4        0.33 ±  5%  perf-profile.self.cycles-pp.selinux_file_permission
      0.69 ±  8%      -0.4        0.25 ±  5%  perf-profile.self.cycles-pp.down_write
      0.70 ±  8%      -0.4        0.28 ±  3%  perf-profile.self.cycles-pp.write
      0.66 ±  7%      -0.4        0.26 ±  7%  perf-profile.self.cycles-pp.iomap_apply
      0.66 ±  7%      -0.4        0.26 ± 10%  perf-profile.self.cycles-pp.iomap_write_actor
      0.64 ±  7%      -0.4        0.24 ±  8%  perf-profile.self.cycles-pp.find_get_entry
      0.62 ±  5%      -0.4        0.23 ±  5%  perf-profile.self.cycles-pp.__might_sleep
      0.60 ± 12%      -0.4        0.22 ±  6%  perf-profile.self.cycles-pp.__fget_light
      0.58 ± 10%      -0.4        0.23 ±  8%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      0.56 ±  8%      -0.3        0.21 ±  2%  perf-profile.self.cycles-pp.xas_load
      0.48 ± 29%      -0.3        0.14 ±  6%  perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
      0.58 ±  6%      -0.3        0.25 ±  3%  perf-profile.self.cycles-pp.xfs_file_iomap_begin
      0.50 ±  6%      -0.3        0.18 ± 10%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.51 ± 26%      -0.3        0.19 ± 11%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.48 ±  4%      -0.3        0.18 ±  6%  perf-profile.self.cycles-pp.fsnotify
      0.52 ±  4%      -0.3        0.21 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.47 ±  6%      -0.3        0.18 ±  6%  perf-profile.self.cycles-pp.iomap_write_end
      0.46 ±  6%      -0.3        0.16 ±  9%  perf-profile.self.cycles-pp.xfs_errortag_test
      0.45 ±  4%      -0.3        0.16 ±  2%  perf-profile.self.cycles-pp.pagecache_get_page
      0.43 ±  4%      -0.3        0.15 ±  7%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.47 ±  6%      -0.3        0.19 ±  5%  perf-profile.self.cycles-pp.__sb_start_write
      0.45 ±  8%      -0.3        0.17 ±  2%  perf-profile.self.cycles-pp.vfs_write
      0.41 ±  6%      -0.3        0.15 ±  5%  perf-profile.self.cycles-pp.iomap_write_begin
      0.41 ±  3%      -0.3        0.14 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.41 ±  5%      -0.3        0.15 ±  7%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.39 ±  4%      -0.3        0.14 ±  3%  perf-profile.self.cycles-pp.xfs_break_leased_layouts
      0.40 ± 13%      -0.2        0.15 ±  5%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.36 ±  4%      -0.2        0.12 ±  6%  perf-profile.self.cycles-pp.__x64_sys_write
      0.35 ±  6%      -0.2        0.12 ±  4%  perf-profile.self.cycles-pp.unlock_page
      0.33 ±  4%      -0.2        0.11 ± 15%  perf-profile.self.cycles-pp.ksys_write
      0.39 ± 20%      -0.2        0.17 ± 23%  perf-profile.self.cycles-pp.ktime_get
      0.32 ±  7%      -0.2        0.10 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.35 ±  7%      -0.2        0.14 ±  6%  perf-profile.self.cycles-pp.release_pages
      0.33 ±  7%      -0.2        0.12 ± 10%  perf-profile.self.cycles-pp.iomap_set_page_dirty
      0.34 ±  6%      -0.2        0.14 ± 10%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.32 ±  6%      -0.2        0.12 ±  7%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.38 ±  5%      -0.2        0.18 ±  5%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.32 ±  3%      -0.2        0.13 ±  6%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.30 ±  4%      -0.2        0.11 ±  7%  perf-profile.self.cycles-pp.new_sync_write
      0.29 ±  7%      -0.2        0.11 ±  4%  perf-profile.self.cycles-pp.xfs_iunlock
      0.28 ±  6%      -0.2        0.10 ±  8%  perf-profile.self.cycles-pp.xfs_ilock
      0.29 ±  4%      -0.2        0.11 ±  7%  perf-profile.self.cycles-pp.page_mapping
      0.27 ±  4%      -0.2        0.10 ±  9%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.26 ±  9%      -0.2        0.08 ± 10%  perf-profile.self.cycles-pp.xfs_file_aio_write_checks
      0.28 ±  3%      -0.2        0.11 ±  3%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.24 ± 20%      -0.2        0.07 ±  5%  perf-profile.self.cycles-pp.account_page_dirtied
      0.27 ± 12%      -0.2        0.10 ± 14%  perf-profile.self.cycles-pp.xas_start
      0.23 ±  3%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.account_page_cleaned
      0.24 ±  7%      -0.2        0.09 ±  7%  perf-profile.self.cycles-pp._cond_resched
      0.20 ± 15%      -0.2        0.05 ± 61%  perf-profile.self.cycles-pp.menu_select
      0.25 ±  3%      -0.1        0.11 ± 12%  perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
      0.20 ±  6%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.iov_iter_advance
      0.22 ± 10%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.find_get_entries
      0.19 ±  6%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.xfs_break_layouts
      0.20 ±  9%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.31 ±  7%      -0.1        0.19 ±  5%  perf-profile.self.cycles-pp.xas_store
      0.24 ±  6%      -0.1        0.12 ± 10%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.19 ± 13%      -0.1        0.07 ±  5%  perf-profile.self.cycles-pp.node_dirty_ok
      0.14 ±  9%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.19 ±  6%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.rcu_all_qs
      0.20 ±  4%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.file_update_time
      0.16 ±  6%      -0.1        0.05 ±  9%  perf-profile.self.cycles-pp.iomap_file_buffered_write
      0.13 ±  9%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.16 ±  5%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.lock_page_memcg
      0.16 ±  7%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.xfs_dir3_leaf_check_int
      0.21 ± 10%      -0.1        0.11 ± 21%  perf-profile.self.cycles-pp.xfs_find_bdev_for_inode
      0.17 ±  9%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.__fdget_pos
      0.16 ±  9%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.xfs_dir3_data_entsize
      0.15 ±  8%      -0.1        0.06 ±  9%  perf-profile.self.cycles-pp.truncate_inode_pages_range
      0.14 ±  3%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.__inode_security_revalidate
      0.17 ±  8%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.15 ± 10%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.17 ± 15%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.generic_write_check_limits
      0.12 ±  7%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.timestamp_truncate
      0.11 ±  9%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.security_file_permission
      0.17 ± 11%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.workingset_update_node
      0.12 ± 10%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.xas_clear_mark
      0.15 ±  3%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.generic_write_checks
      0.15 ±  8%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.12 ±  5%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.__xfs_dir3_data_check
      0.14 ±  5%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__sb_end_write
      0.11 ±  7%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.native_write_msr
      0.12 ±  5%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.current_time
      0.09 ±  7%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.09 ±  9%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.11 ±  7%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.xfs_agino_range
      0.15 ±  4%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.memcpy_erms
      0.84 ±  4%     +66.7       67.58 ±  2%  perf-profile.self.cycles-pp.osq_lock


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  460000 +-+----------------------------------------------------------------+   
         |  .+.. .+. .+..        .+. .+.    .+. .+.+..+.+.               .+.|   
  440000 +-+    +   +    +.+.+.+.   +   +..+   +          +.+..+.+.+.+..+   |   
  420000 +-+                                                                |   
         |                                                                  |   
  400000 +-+                                       O    O                   |   
  380000 +-+ O        O                                                     |   
         |        O                     O  O   O O             O            |   
  360000 O-+    O        O O O O    O O      O        O   O O               |   
  340000 +-O        O             O                                         |   
         |                                                                  |   
  320000 +-+                                                                |   
  300000 +-+                                                       O        |   
         |                                                           O  O   |   
  280000 +-+-----------------------------------------------------O----------+   
                                                                                
                                                                                                                                                                
                               aim7.time.system_time                            
                                                                                
  2200 +-+------------------------------------------------------------------+   
       |                                                             O O    |   
  2000 +-+                                                       O O        |   
  1800 +-+                                                                  |   
       | O                                                                  |   
  1600 O-+    O    O          O             O        O    O                 |   
  1400 +-+      O      O O  O   O  O O        O O           O O             |   
       |    O        O                 O  O        O   O                    |   
  1200 +-+                                                                  |   
  1000 +-+                                                                  |   
       |                                                                    |   
   800 +-+                                                                  |   
   600 +-+                                                                  |   
       |                                                                    |   
   400 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             aim7.time.elapsed_time                             
                                                                                
  65 +-+--------------------------------------------------------------------+   
     |                                                          O    O      |   
     |                                                                 O    |   
  60 +-+                                                           O        |   
     |                                                                      |   
     |                                                                      |   
  55 +-+                                                                    |   
     | O         O             O                                            |   
  50 O-+    O         O O O  O   O  O      O        O    O O                |   
     |        O                       O  O   O  O             O             |   
     |    O        O                              O    O                    |   
  45 +-+                                                                    |   
     |                                                                      |   
     |.+.. .+.  .+.  .+.+.+..+. .+.. .+..+. .+..      .+.+.+..+.+..+.+.+.. .|   
  40 +-+--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           aim7.time.elapsed_time.max                           
                                                                                
  65 +-+--------------------------------------------------------------------+   
     |                                                          O    O      |   
     |                                                                 O    |   
  60 +-+                                                           O        |   
     |                                                                      |   
     |                                                                      |   
  55 +-+                                                                    |   
     | O         O             O                                            |   
  50 O-+    O         O O O  O   O  O      O        O    O O                |   
     |        O                       O  O   O  O             O             |   
     |    O        O                              O    O                    |   
  45 +-+                                                                    |   
     |                                                                      |   
     |.+.. .+.  .+.  .+.+.+..+. .+.. .+..+. .+..      .+.+.+..+.+..+.+.+.. .|   
  40 +-+--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        aim7.time.voluntary_context_switches                    
                                                                                
   1e+06 +-+----------------------------------------------------------------+   
         |                                                                  |   
  900000 +-+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+.+.+..+.+.+.+..+.+.+.+..+.+.|   
         |                                                                  |   
         |                                                                  |   
  800000 +-+                                                                |   
         |                                                                  |   
  700000 +-+                                                                |   
         |                                                                  |   
  600000 +-+                                                                |   
         |                                                                  |   
         |            O      O        O O  O                   O            |   
  500000 O-+ O  O O      O O   O  O O        O O O O  O O O O      O        |   
         | O        O                                            O      O   |   
  400000 +-+---------------------------------------------------------O------+   
                                                                                
                                                                                                                                                                
                       aim7.time.involuntary_context_switches                   
                                                                                
  10000 +-+------------------------------------------------------O-----O----+   
        |                                                            O      |   
   9000 +-+                                                        O        |   
   8000 +-O        O                                        O               |   
        O                 O    O   O                 O    O                 |   
   7000 +-+  O O            O               O  O O            O             |   
   6000 +-+      O      O        O   O             O    O                   |   
        |             O                 O O                                 |   
   5000 +-+                                                                 |   
   4000 +-+                                                                 |   
        |                                                                   |   
   3000 +-+                                                                 |   
   2000 +-+                                                                 |   
        |.+..     .+.. .+. .+..+.      .+.+.               .+.  .+.         |   
   1000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.4.0-rc7-00092-g646a9fb7fb983" of type "text/plain" (200562 bytes)

View attachment "job-script" of type "text/plain" (7894 bytes)

View attachment "job.yaml" of type "text/plain" (5518 bytes)

View attachment "reproduce" of type "text/plain" (1015 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ