lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20190916010021.GX15734@shao2-debian>
Date:   Mon, 16 Sep 2019 09:00:22 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Darrick J. Wong" <darrick.wong@...cle.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...org
Subject: [xfs] 73a16a2894: aim7.jobs-per-min -31.6% regression

Greeting,

FYI, we noticed a -31.6% regression of aim7.jobs-per-min due to commit:


commit: 73a16a2894549c9e544ef61aad3c30a60019a2cb ("xfs: parallelize inode inactivation")
https://kernel.googlesource.com/pub/scm/linux/kernel/git/djwong/xfs-linux.git rough-fixes

in testcase: aim7
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: xfs
	test: disk_src
	load: 3000
	cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID0/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_src/aim7

commit: 
  6298ae61d2 ("xfs: force inactivation before fallocate when space is low")
  73a16a2894 ("xfs: parallelize inode inactivation")

6298ae61d2ecd22d 73a16a2894549c9e544ef61aad3 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          3:4          -75%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :4           50%           2:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
         %stddev     %change         %stddev
             \          |                \  
     28682           -31.6%      19623        aim7.jobs-per-min
    628.00           +46.1%     917.77        aim7.time.elapsed_time
    628.00           +46.1%     917.77        aim7.time.elapsed_time.max
     32590 ±  6%     -35.9%      20896 ±  6%  aim7.time.file_system_inputs
     21342           +51.1%      32253        aim7.time.involuntary_context_switches
      3491           +58.6%       5538        aim7.time.system_time
    122.28            +8.1%     132.16        aim7.time.user_time
   7211143           +64.6%   11870679 ±  2%  aim7.time.voluntary_context_switches
      1285            -1.2%       1269        boot-time.idle
      6590 ± 70%     +98.0%      13046        numa-numastat.node1.other_node
 1.549e+10 ± 10%     +43.6%  2.224e+10 ±  9%  cpuidle.C6.time
  26014805 ± 17%     +61.0%   41883750 ± 19%  cpuidle.C6.usage
    365519           +66.2%     607360        cpuidle.POLL.time
     23959 ±  3%     +60.6%      38475        cpuidle.POLL.usage
     84.05            -3.7%      80.90        iostat.cpu.idle
     15.38           +21.3%      18.66        iostat.cpu.system
      1296           -24.8%     975.49        iostat.md0.w/s
     28242           -22.7%      21842        iostat.md0.wkB/s
      2265 ± 41%     +92.7%       4364 ± 28%  numa-vmstat.node0.nr_shmem
    479835           -94.1%      28115 ±  2%  numa-vmstat.node0.nr_slab_reclaimable
    323938 ±  2%     -92.3%      24986 ± 32%  numa-vmstat.node0.nr_slab_unreclaimable
    488494           -95.4%      22470 ±  3%  numa-vmstat.node1.nr_slab_reclaimable
    315962 ±  2%     -89.7%      32671 ± 24%  numa-vmstat.node1.nr_slab_unreclaimable
    255.50            +8.4%     277.00 ±  2%  turbostat.Avg_MHz
     20.01            +3.0       23.04 ±  2%  turbostat.Busy%
  26010933 ± 17%     +61.0%   41879491 ± 19%  turbostat.C6
  51358007           +44.5%   74196457        turbostat.IRQ
      1.18 ± 18%     -36.8%       0.74 ± 11%  turbostat.Pkg%pc2
     50370           +46.0%      73540        turbostat.SMI
     83.50            -3.9%      80.25        vmstat.cpu.id
     27381           -22.8%      21144        vmstat.io.bo
   4984658           -73.7%    1313263        vmstat.memory.cache
      6.00           +16.7%       7.00        vmstat.procs.r
     38725            +5.5%      40838        vmstat.system.cs
     81494            -1.0%      80660        vmstat.system.in
   3872245           -94.8%     202334        meminfo.KReclaimable
  13158326           -61.0%    5130076        meminfo.Memused
    101553 ±  5%     +14.5%     116292 ±  8%  meminfo.PageTables
   3872245           -94.8%     202334        meminfo.SReclaimable
   2558865           -91.0%     230631        meminfo.SUnreclaim
     31509           +11.1%      35010        meminfo.Shmem
   6431111           -93.3%     432966        meminfo.Slab
    338337           -16.4%     282859        meminfo.VmallocUsed
     37848           -78.1%       8278        meminfo.max_used_kB
   1918514           -94.1%     112464 ±  2%  numa-meminfo.node0.KReclaimable
   6329865 ±  6%     -58.8%    2610379 ±  6%  numa-meminfo.node0.MemUsed
   1918514           -94.1%     112464 ±  2%  numa-meminfo.node0.SReclaimable
   1295259 ±  2%     -92.3%      99946 ± 32%  numa-meminfo.node0.SUnreclaim
      9059 ± 41%     +92.7%      17459 ± 28%  numa-meminfo.node0.Shmem
   3213774           -93.4%     212411 ± 15%  numa-meminfo.node0.Slab
   1953128           -95.4%      89882 ±  3%  numa-meminfo.node1.KReclaimable
   6827591 ±  5%     -63.1%    2519244 ±  5%  numa-meminfo.node1.MemUsed
   1953128           -95.4%      89882 ±  3%  numa-meminfo.node1.SReclaimable
   1263297 ±  2%     -89.7%     130693 ± 24%  numa-meminfo.node1.SUnreclaim
   3216425           -93.1%     220576 ± 14%  numa-meminfo.node1.Slab
   9520889            +2.1%    9721313        proc-vmstat.nr_dirty_background_threshold
  19065060            +2.1%   19466397        proc-vmstat.nr_dirty_threshold
  95762281            +2.1%   97769457        proc-vmstat.nr_free_pages
      5062            +2.5%       5191        proc-vmstat.nr_inactive_anon
     25386 ±  5%     +14.5%      29072 ±  8%  proc-vmstat.nr_page_table_pages
      7877           +11.1%       8752        proc-vmstat.nr_shmem
    968099           -94.8%      50584        proc-vmstat.nr_slab_reclaimable
    639746           -91.0%      57656        proc-vmstat.nr_slab_unreclaimable
      5062            +2.5%       5191        proc-vmstat.nr_zone_inactive_anon
   5050595            -3.7%    4861550        proc-vmstat.numa_hit
   5037460            -3.8%    4848364        proc-vmstat.numa_local
      1324 ± 11%     +32.0%       1748 ±  9%  proc-vmstat.pgactivate
   7847951           -22.1%    6117074        proc-vmstat.pgalloc_normal
   1764564           +39.9%    2469500        proc-vmstat.pgfault
   2388604           +99.2%    4758851        proc-vmstat.pgfree
     16475 ±  5%     -35.5%      10624 ±  8%  proc-vmstat.pgpgin
  17259303           +12.8%   19464058        proc-vmstat.pgpgout
      1879            +1.5%       1907        proc-vmstat.unevictable_pgs_culled
     48666           +76.8%      86051        sched_debug.cfs_rq:/.exec_clock.avg
    135393           -33.2%      90412        sched_debug.cfs_rq:/.exec_clock.max
     26618          +215.6%      84021        sched_debug.cfs_rq:/.exec_clock.min
     14312           -91.0%       1289 ±  9%  sched_debug.cfs_rq:/.exec_clock.stddev
    393911 ±  4%     +96.7%     774700 ±  4%  sched_debug.cfs_rq:/.min_vruntime.avg
    579123 ±  2%     +57.0%     909216 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
    159226 ± 10%    +355.1%     724696 ±  4%  sched_debug.cfs_rq:/.min_vruntime.min
     65888 ± 12%     -40.6%      39155 ± 30%  sched_debug.cfs_rq:/.min_vruntime.stddev
      1.12 ± 12%     +49.9%       1.67 ± 32%  sched_debug.cfs_rq:/.nr_spread_over.avg
      0.45           +75.3%       0.80 ± 12%  sched_debug.cfs_rq:/.nr_spread_over.min
     18.27 ± 15%     +72.7%      31.55 ±  8%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    241590 ± 29%     -43.5%     136534 ± 14%  sched_debug.cfs_rq:/.spread0.max
   -178307           -73.1%     -48028        sched_debug.cfs_rq:/.spread0.min
     65886 ± 12%     -40.6%      39156 ± 30%  sched_debug.cfs_rq:/.spread0.stddev
    157.45           +17.3%     184.66 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
     23.29 ±  9%     +71.3%      39.90 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     78.72 ±  7%     +46.0%     114.96 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     93948 ±  9%    +146.0%     231107 ±  6%  sched_debug.cpu.avg_idle.min
    343542           +44.8%     497596        sched_debug.cpu.clock.avg
    343569           +44.8%     497627        sched_debug.cpu.clock.max
    343513           +44.8%     497564        sched_debug.cpu.clock.min
    343542           +44.8%     497596        sched_debug.cpu.clock_task.avg
    343569           +44.8%     497627        sched_debug.cpu.clock_task.max
    343513           +44.8%     497564        sched_debug.cpu.clock_task.min
     11973           +31.8%      15786        sched_debug.cpu.curr->pid.max
      2293           +20.8%       2771 ±  2%  sched_debug.cpu.curr->pid.stddev
    291435           +59.6%     465252        sched_debug.cpu.nr_switches.avg
   3322828           -84.8%     506717        sched_debug.cpu.nr_switches.max
    123943 ±  6%    +243.8%     426081 ±  2%  sched_debug.cpu.nr_switches.min
    485747           -96.4%      17504 ±  4%  sched_debug.cpu.nr_switches.stddev
    154.41 ±  6%     +25.5%     193.75 ±  9%  sched_debug.cpu.nr_uninterruptible.max
     45.66 ± 15%     +44.5%      65.97 ±  9%  sched_debug.cpu.nr_uninterruptible.stddev
    289211           +60.1%     462992        sched_debug.cpu.sched_count.avg
   3319749           -84.8%     503177        sched_debug.cpu.sched_count.max
    121121 ±  7%    +249.9%     423747 ±  2%  sched_debug.cpu.sched_count.min
    485550           -96.5%      17224 ±  4%  sched_debug.cpu.sched_count.stddev
    129527           +68.6%     218367 ±  2%  sched_debug.cpu.sched_goidle.avg
   1495010           -84.2%     236513 ±  2%  sched_debug.cpu.sched_goidle.max
     56307 ±  7%    +256.6%     200766 ±  2%  sched_debug.cpu.sched_goidle.min
    218759           -96.4%       7832 ±  5%  sched_debug.cpu.sched_goidle.stddev
    146549           +59.7%     233968        sched_debug.cpu.ttwu_count.avg
   1376890           -81.6%     253020 ±  2%  sched_debug.cpu.ttwu_count.max
     66441 ±  5%    +221.1%     213322        sched_debug.cpu.ttwu_count.min
    197195           -95.6%       8696 ±  3%  sched_debug.cpu.ttwu_count.stddev
     14892           +23.4%      18370        sched_debug.cpu.ttwu_local.avg
      3035 ± 15%    +355.4%      13824 ±  2%  sched_debug.cpu.ttwu_local.min
      4226 ±  3%     -39.2%       2569 ±  6%  sched_debug.cpu.ttwu_local.stddev
    343511           +44.8%     497561        sched_debug.cpu_clk
    339915           +45.3%     493967        sched_debug.ktime
    343869           +44.8%     497922        sched_debug.sched_clk
   3715816           -98.4%      61244        slabinfo.dmaengine-unmap-16.active_objs
     88703           -97.7%       2027        slabinfo.dmaengine-unmap-16.active_slabs
   3725580           -97.7%      85200        slabinfo.dmaengine-unmap-16.num_objs
     88703           -97.7%       2027        slabinfo.dmaengine-unmap-16.num_slabs
    933.75 ±  6%      -8.5%     854.75 ±  7%  slabinfo.file_lock_cache.active_objs
    933.75 ±  6%      -8.5%     854.75 ±  7%  slabinfo.file_lock_cache.num_objs
     11739           -10.7%      10482        slabinfo.filp.num_objs
      3550 ±  3%    +187.1%      10192        slabinfo.kmalloc-128.active_objs
    110.25 ±  3%    +189.8%     319.50        slabinfo.kmalloc-128.active_slabs
      3550 ±  3%    +188.6%      10244        slabinfo.kmalloc-128.num_objs
    110.25 ±  3%    +189.8%     319.50        slabinfo.kmalloc-128.num_slabs
   3699063           -97.6%      89992        slabinfo.kmalloc-16.active_objs
     14449           -97.3%     384.75        slabinfo.kmalloc-16.active_slabs
   3699207           -97.3%      98606        slabinfo.kmalloc-16.num_objs
     14449           -97.3%     384.75        slabinfo.kmalloc-16.num_slabs
      4338 ±  2%     +33.2%       5779 ±  2%  slabinfo.kmalloc-1k.active_objs
      4357 ±  2%     +32.7%       5780 ±  2%  slabinfo.kmalloc-1k.num_objs
    129558           -85.0%      19424 ±  6%  slabinfo.kmalloc-32.active_objs
      1011           -84.9%     152.75 ±  6%  slabinfo.kmalloc-32.active_slabs
    129579           -84.8%      19638 ±  6%  slabinfo.kmalloc-32.num_objs
      1011           -84.9%     152.75 ±  6%  slabinfo.kmalloc-32.num_slabs
   3138133           -98.7%      41306        slabinfo.kmalloc-512.active_objs
     98102           -98.1%       1906        slabinfo.kmalloc-512.active_slabs
   3139296           -98.1%      61014        slabinfo.kmalloc-512.num_objs
     98102           -98.1%       1906        slabinfo.kmalloc-512.num_slabs
    288.50 ±  9%     +46.0%     421.25        slabinfo.nfs_read_data.active_objs
    288.50 ±  9%     +46.0%     421.25        slabinfo.nfs_read_data.num_objs
     83301           -70.8%      24296        slabinfo.radix_tree_node.active_objs
      2981           -70.5%     878.50        slabinfo.radix_tree_node.active_slabs
     83509           -70.5%      24607        slabinfo.radix_tree_node.num_objs
      2981           -70.5%     878.50        slabinfo.radix_tree_node.num_slabs
     15967           +19.7%      19115        slabinfo.vmap_area.active_objs
     16696           +17.5%      19610        slabinfo.vmap_area.num_objs
    324.00 ± 19%     +49.7%     485.00 ±  3%  slabinfo.xfrm_state.active_objs
    324.00 ± 19%     +49.7%     485.00 ±  3%  slabinfo.xfrm_state.num_objs
    116536           -93.7%       7313 ±  3%  slabinfo.xfs_buf.active_objs
      2774           -93.5%     180.25 ±  3%  slabinfo.xfs_buf.active_slabs
    116547           -93.5%       7597 ±  3%  slabinfo.xfs_buf.num_objs
      2774           -93.5%     180.25 ±  3%  slabinfo.xfs_buf.num_slabs
      9169           -35.4%       5923        slabinfo.xfs_buf_item.active_objs
    310.25           -35.4%     200.50        slabinfo.xfs_buf_item.active_slabs
      9315           -35.3%       6028        slabinfo.xfs_buf_item.num_objs
    310.25           -35.4%     200.50        slabinfo.xfs_buf_item.num_slabs
      1966 ±  4%    +226.1%       6411 ±  5%  slabinfo.xfs_efd_item.active_objs
     52.75 ±  4%    +230.3%     174.25 ±  5%  slabinfo.xfs_efd_item.active_slabs
      1966 ±  4%    +228.8%       6464 ±  5%  slabinfo.xfs_efd_item.num_objs
     52.75 ±  4%    +230.3%     174.25 ±  5%  slabinfo.xfs_efd_item.num_slabs
   3715508           -98.4%      60982        slabinfo.xfs_inode.active_objs
    116415           -97.7%       2626        slabinfo.xfs_inode.active_slabs
   3725291           -97.7%      84066        slabinfo.xfs_inode.num_objs
    116415           -97.7%       2626        slabinfo.xfs_inode.num_slabs
     16.65 ±  2%     -36.6%      10.55 ±  5%  perf-stat.i.MPKI
 1.507e+09           +28.4%  1.935e+09        perf-stat.i.branch-instructions
      3.14            -0.6        2.54        perf-stat.i.branch-miss-rate%
     15.46 ±  2%      -2.1       13.36 ±  4%  perf-stat.i.cache-miss-rate%
  17898559 ±  3%     -34.4%   11746315 ±  7%  perf-stat.i.cache-misses
 1.131e+08 ±  2%     -21.2%   89131641 ±  5%  perf-stat.i.cache-references
     38840            +5.3%      40916        perf-stat.i.context-switches
      1.60 ±  2%     -20.5%       1.27 ±  3%  perf-stat.i.cpi
    561.37           +65.4%     928.27 ±  7%  perf-stat.i.cpu-migrations
    669.68 ±  2%     +43.5%     961.21 ±  5%  perf-stat.i.cycles-between-cache-misses
      0.62 ±  5%      -0.3        0.28 ±  3%  perf-stat.i.dTLB-load-miss-rate%
  11040116 ±  5%     -40.3%    6594857 ±  4%  perf-stat.i.dTLB-load-misses
 1.823e+09           +42.1%   2.59e+09        perf-stat.i.dTLB-loads
      0.15 ±  4%      -0.1        0.07 ±  7%  perf-stat.i.dTLB-store-miss-rate%
   1176632 ±  3%     -34.2%     773684 ±  9%  perf-stat.i.dTLB-store-misses
 7.961e+08 ±  2%     +48.9%  1.185e+09 ±  5%  perf-stat.i.dTLB-stores
     41.74 ±  2%      +1.8       43.58        perf-stat.i.iTLB-load-miss-rate%
   1680621 ±  2%      -9.0%    1529900 ±  3%  perf-stat.i.iTLB-load-misses
   2421937 ±  4%     -14.1%    2079312 ±  2%  perf-stat.i.iTLB-loads
 7.145e+09           +33.7%  9.555e+09        perf-stat.i.instructions
      4232 ±  2%     +46.3%       6193 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.65           +27.9%       0.83 ±  3%  perf-stat.i.ipc
      2724            -3.7%       2622        perf-stat.i.minor-faults
     42.83            +1.5       44.34        perf-stat.i.node-load-miss-rate%
   6361451           -47.5%    3336831        perf-stat.i.node-load-misses
   9312635           -55.1%    4185756 ±  2%  perf-stat.i.node-loads
   2772972           -25.5%    2064781        perf-stat.i.node-store-misses
   4138412           -27.1%    3017312        perf-stat.i.node-stores
      2724            -3.7%       2622        perf-stat.i.page-faults
     15.83 ±  2%     -41.1%       9.33 ±  4%  perf-stat.overall.MPKI
      2.96            -0.6        2.33        perf-stat.overall.branch-miss-rate%
     15.82 ±  2%      -2.7       13.17 ±  4%  perf-stat.overall.cache-miss-rate%
      1.52           -22.3%       1.18 ±  2%  perf-stat.overall.cpi
    608.08 ±  2%     +58.9%     966.12 ±  5%  perf-stat.overall.cycles-between-cache-misses
      0.60 ±  6%      -0.3        0.25 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
      0.15 ±  4%      -0.1        0.07 ±  7%  perf-stat.overall.dTLB-store-miss-rate%
      4253 ±  2%     +47.0%       6251 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      0.66           +28.7%       0.85 ±  2%  perf-stat.overall.ipc
     40.59            +3.8       44.36        perf-stat.overall.node-load-miss-rate%
     40.12            +0.5       40.63        perf-stat.overall.node-store-miss-rate%
 1.505e+09           +28.5%  1.933e+09        perf-stat.ps.branch-instructions
  17869714 ±  3%     -34.3%   11733432 ±  7%  perf-stat.ps.cache-misses
 1.129e+08 ±  2%     -21.1%   89033383 ±  5%  perf-stat.ps.cache-references
     38777            +5.4%      40870        perf-stat.ps.context-switches
    560.48           +65.4%     927.25 ±  7%  perf-stat.ps.cpu-migrations
  11022157 ±  5%     -40.2%    6587600 ±  4%  perf-stat.ps.dTLB-load-misses
  1.82e+09           +42.1%  2.587e+09        perf-stat.ps.dTLB-loads
   1174725 ±  3%     -34.2%     772840 ±  9%  perf-stat.ps.dTLB-store-misses
 7.948e+08 ±  2%     +48.9%  1.184e+09 ±  5%  perf-stat.ps.dTLB-stores
   1677919 ±  2%      -8.9%    1528226 ±  3%  perf-stat.ps.iTLB-load-misses
   2418014 ±  4%     -14.1%    2077000 ±  2%  perf-stat.ps.iTLB-loads
 7.134e+09           +33.8%  9.544e+09        perf-stat.ps.instructions
      2720            -3.7%       2620        perf-stat.ps.minor-faults
   6351137           -47.5%    3333134        perf-stat.ps.node-load-misses
   9297533           -55.0%    4181124 ±  2%  perf-stat.ps.node-loads
   2768483           -25.5%    2062498        perf-stat.ps.node-store-misses
   4131727           -27.1%    3013998        perf-stat.ps.node-stores
      2720            -3.7%       2620        perf-stat.ps.page-faults
 4.486e+12           +95.4%  8.766e+12        perf-stat.total.instructions
      1494           +15.6%       1727 ±  2%  interrupts.10:IR-IO-APIC.10-edge.ipmi_si
    792.50 ± 13%     +36.3%       1080 ± 14%  interrupts.39:IR-PCI-MSI.524289-edge.eth0-TxRx-0
    366.50 ±  3%   +4427.7%      16594 ±166%  interrupts.40:IR-PCI-MSI.524290-edge.eth0-TxRx-1
    523.25 ± 26%   +3155.0%      17032 ± 57%  interrupts.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6
    204268           +43.9%     293848        interrupts.CAL:Function_call_interrupts
      4733 ±  2%     +46.0%       6909        interrupts.CPU0.CAL:Function_call_interrupts
   1258857           +46.1%    1838720        interrupts.CPU0.LOC:Local_timer_interrupts
      5356 ±  6%     +38.4%       7416 ±  4%  interrupts.CPU1.CAL:Function_call_interrupts
   1258156           +46.1%    1838261        interrupts.CPU1.LOC:Local_timer_interrupts
      4926 ±  8%     +60.7%       7918        interrupts.CPU10.CAL:Function_call_interrupts
   1258344           +46.1%    1838592        interrupts.CPU10.LOC:Local_timer_interrupts
    164.50 ±151%    +376.6%     784.00 ±  2%  interrupts.CPU10.NMI:Non-maskable_interrupts
    164.50 ±151%    +376.6%     784.00 ±  2%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      4305 ± 32%    +103.6%       8764 ±  4%  interrupts.CPU10.RES:Rescheduling_interrupts
      4914 ±  6%     +53.7%       7556 ±  7%  interrupts.CPU11.CAL:Function_call_interrupts
   1258282           +46.1%    1838248        interrupts.CPU11.LOC:Local_timer_interrupts
      5243 ±  8%     +45.5%       7629 ±  7%  interrupts.CPU12.CAL:Function_call_interrupts
   1258363           +46.1%    1838546        interrupts.CPU12.LOC:Local_timer_interrupts
      4884 ±  6%     +56.6%       7646 ±  6%  interrupts.CPU13.CAL:Function_call_interrupts
   1258209           +46.1%    1838551        interrupts.CPU13.LOC:Local_timer_interrupts
      4984 ±  7%     +42.6%       7109 ±  8%  interrupts.CPU14.CAL:Function_call_interrupts
   1258516           +46.1%    1838593        interrupts.CPU14.LOC:Local_timer_interrupts
      5286 ±  5%     +33.5%       7057 ±  5%  interrupts.CPU15.CAL:Function_call_interrupts
   1258314           +46.1%    1838531        interrupts.CPU15.LOC:Local_timer_interrupts
      4737 ±  3%     +56.5%       7413 ±  7%  interrupts.CPU16.CAL:Function_call_interrupts
   1258679           +46.1%    1838447        interrupts.CPU16.LOC:Local_timer_interrupts
      4237 ± 36%     +73.6%       7356 ± 23%  interrupts.CPU16.RES:Rescheduling_interrupts
      4806           +57.5%       7572 ±  5%  interrupts.CPU17.CAL:Function_call_interrupts
   1257486           +46.2%    1838428        interrupts.CPU17.LOC:Local_timer_interrupts
      4968 ±  7%     +57.9%       7847 ±  3%  interrupts.CPU18.CAL:Function_call_interrupts
   1258427           +46.1%    1838038        interrupts.CPU18.LOC:Local_timer_interrupts
      4864 ± 39%     +87.5%       9119 ±  8%  interrupts.CPU18.RES:Rescheduling_interrupts
      4956 ±  4%     +43.7%       7120 ±  4%  interrupts.CPU19.CAL:Function_call_interrupts
   1257616           +46.2%    1838487        interrupts.CPU19.LOC:Local_timer_interrupts
      4973 ±  6%     +57.3%       7820 ±  2%  interrupts.CPU2.CAL:Function_call_interrupts
   1258621           +46.1%    1838386        interrupts.CPU2.LOC:Local_timer_interrupts
    165.75 ±138%    +487.3%     973.50 ± 32%  interrupts.CPU2.NMI:Non-maskable_interrupts
    165.75 ±138%    +487.3%     973.50 ± 32%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      4318 ± 35%     +96.7%       8494 ±  2%  interrupts.CPU2.RES:Rescheduling_interrupts
      5505           +44.5%       7952        interrupts.CPU20.CAL:Function_call_interrupts
   1258218           +46.1%    1838462        interrupts.CPU20.LOC:Local_timer_interrupts
      4940 ±  5%     +42.3%       7028 ±  8%  interrupts.CPU21.CAL:Function_call_interrupts
   1257717           +46.2%    1838448        interrupts.CPU21.LOC:Local_timer_interrupts
      5320 ±  7%     +28.8%       6853        interrupts.CPU22.CAL:Function_call_interrupts
   1257738           +46.2%    1838357        interrupts.CPU22.LOC:Local_timer_interrupts
    792.75 ± 32%     -88.4%      91.75 ±106%  interrupts.CPU22.NMI:Non-maskable_interrupts
    792.75 ± 32%     -88.4%      91.75 ±106%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      5453 ±  3%     +33.8%       7297 ±  5%  interrupts.CPU23.CAL:Function_call_interrupts
   1257399           +46.2%    1838017        interrupts.CPU23.LOC:Local_timer_interrupts
      5133 ±  7%     +43.9%       7386 ±  7%  interrupts.CPU24.CAL:Function_call_interrupts
   1257560           +46.2%    1838381        interrupts.CPU24.LOC:Local_timer_interrupts
      4539 ± 33%     +50.1%       6813 ± 18%  interrupts.CPU24.RES:Rescheduling_interrupts
      5308 ± 10%     +35.9%       7214 ±  9%  interrupts.CPU25.CAL:Function_call_interrupts
   1258418           +46.1%    1838405        interrupts.CPU25.LOC:Local_timer_interrupts
      5124 ±  7%     +40.3%       7188 ±  6%  interrupts.CPU26.CAL:Function_call_interrupts
   1258415           +46.1%    1838409        interrupts.CPU26.LOC:Local_timer_interrupts
      5248 ±  8%     +43.1%       7512 ±  6%  interrupts.CPU27.CAL:Function_call_interrupts
   1258222           +46.1%    1838381        interrupts.CPU27.LOC:Local_timer_interrupts
      5114 ±  8%     +42.8%       7305 ±  5%  interrupts.CPU28.CAL:Function_call_interrupts
   1258628           +46.0%    1838215        interrupts.CPU28.LOC:Local_timer_interrupts
      4114 ± 47%     +64.9%       6786 ± 22%  interrupts.CPU28.RES:Rescheduling_interrupts
      4910 ±  7%     +52.7%       7501 ±  5%  interrupts.CPU29.CAL:Function_call_interrupts
   1258603           +46.1%    1838647        interrupts.CPU29.LOC:Local_timer_interrupts
      4737 ±  3%     +53.9%       7289 ±  7%  interrupts.CPU3.CAL:Function_call_interrupts
   1258257           +46.1%    1838143        interrupts.CPU3.LOC:Local_timer_interrupts
    792.50 ± 13%     +36.3%       1080 ± 14%  interrupts.CPU30.39:IR-PCI-MSI.524289-edge.eth0-TxRx-0
      5271 ±  5%     +40.0%       7381 ±  6%  interrupts.CPU30.CAL:Function_call_interrupts
   1258798           +46.1%    1838480        interrupts.CPU30.LOC:Local_timer_interrupts
      5050 ± 37%     +38.1%       6973 ± 25%  interrupts.CPU30.RES:Rescheduling_interrupts
      5266 ±  6%     +33.7%       7041 ±  7%  interrupts.CPU31.CAL:Function_call_interrupts
   1258115           +46.1%    1838307        interrupts.CPU31.LOC:Local_timer_interrupts
    366.50 ±  3%   +4427.7%      16594 ±166%  interrupts.CPU32.40:IR-PCI-MSI.524290-edge.eth0-TxRx-1
      4706 ±  3%     +50.6%       7088 ±  5%  interrupts.CPU32.CAL:Function_call_interrupts
   1258290           +46.0%    1837663        interrupts.CPU32.LOC:Local_timer_interrupts
      3612 ± 57%     +79.6%       6488 ± 20%  interrupts.CPU32.RES:Rescheduling_interrupts
      5301 ±  7%     +31.2%       6957 ±  7%  interrupts.CPU33.CAL:Function_call_interrupts
   1258353           +46.1%    1838459        interrupts.CPU33.LOC:Local_timer_interrupts
      5273 ±  5%     +46.0%       7697 ±  5%  interrupts.CPU34.CAL:Function_call_interrupts
   1258565           +46.0%    1838076        interrupts.CPU34.LOC:Local_timer_interrupts
      5048 ±  5%     +49.0%       7520 ±  7%  interrupts.CPU35.CAL:Function_call_interrupts
   1258233           +46.1%    1838319        interrupts.CPU35.LOC:Local_timer_interrupts
      5447 ±  3%     +36.0%       7409 ±  6%  interrupts.CPU36.CAL:Function_call_interrupts
   1258484           +46.0%    1837980        interrupts.CPU36.LOC:Local_timer_interrupts
      5440 ±  3%     +29.5%       7047 ±  8%  interrupts.CPU37.CAL:Function_call_interrupts
   1258540           +46.1%    1838267        interrupts.CPU37.LOC:Local_timer_interrupts
      5299 ±  5%     +30.1%       6896        interrupts.CPU38.CAL:Function_call_interrupts
   1258420           +46.1%    1837968        interrupts.CPU38.LOC:Local_timer_interrupts
      5023 ±  9%     +44.8%       7273 ± 10%  interrupts.CPU39.CAL:Function_call_interrupts
   1258243           +46.1%    1838310        interrupts.CPU39.LOC:Local_timer_interrupts
    143058 ±163%     -95.5%       6406 ± 16%  interrupts.CPU39.RES:Rescheduling_interrupts
    523.25 ± 26%   +3155.0%      17032 ± 57%  interrupts.CPU4.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6
      5110 ±  8%     +43.8%       7346 ±  7%  interrupts.CPU4.CAL:Function_call_interrupts
   1258092           +46.1%    1838428        interrupts.CPU4.LOC:Local_timer_interrupts
      5054 ±  7%     +45.3%       7341 ±  6%  interrupts.CPU5.CAL:Function_call_interrupts
   1258423           +46.1%    1838142        interrupts.CPU5.LOC:Local_timer_interrupts
      5153 ±  7%     +47.1%       7580 ±  8%  interrupts.CPU6.CAL:Function_call_interrupts
   1258463           +46.1%    1838103        interrupts.CPU6.LOC:Local_timer_interrupts
      4679 ± 44%     +65.2%       7732 ± 21%  interrupts.CPU6.RES:Rescheduling_interrupts
      4876 ±  5%     +47.2%       7178 ±  6%  interrupts.CPU7.CAL:Function_call_interrupts
   1258508           +46.1%    1838410        interrupts.CPU7.LOC:Local_timer_interrupts
      5121 ±  5%     +48.9%       7623 ±  8%  interrupts.CPU8.CAL:Function_call_interrupts
   1258614           +46.1%    1838880        interrupts.CPU8.LOC:Local_timer_interrupts
      5307 ±  5%     +30.4%       6922 ±  8%  interrupts.CPU9.CAL:Function_call_interrupts
   1258499           +46.1%    1838174        interrupts.CPU9.LOC:Local_timer_interrupts
  50331699           +46.1%   73533674        interrupts.LOC:Local_timer_interrupts
    768133           -62.9%     285162 ±  4%  interrupts.RES:Rescheduling_interrupts
      1532 ±  6%     +45.9%       2234 ± 10%  interrupts.TLB:TLB_shootdowns
    141788 ±  6%     +41.9%     201238 ±  4%  softirqs.CPU0.RCU
     89538 ±  7%     +50.5%     134779        softirqs.CPU0.SCHED
    237443 ±  6%     +37.1%     325641 ± 11%  softirqs.CPU0.TIMER
    146251 ±  3%     +41.1%     206410 ±  5%  softirqs.CPU1.RCU
     92026 ±  3%     +45.0%     133455 ±  3%  softirqs.CPU1.SCHED
    224949 ± 10%     +38.3%     310998 ±  8%  softirqs.CPU1.TIMER
    154233           +33.6%     206060 ±  3%  softirqs.CPU10.RCU
     90504 ±  2%     +46.3%     132452 ±  2%  softirqs.CPU10.SCHED
    242657 ±  6%     +33.8%     324585 ± 11%  softirqs.CPU10.TIMER
    148976 ±  4%     +39.3%     207517 ±  4%  softirqs.CPU11.RCU
     90739           +48.1%     134427        softirqs.CPU11.SCHED
    249720 ±  4%     +26.7%     316341 ± 10%  softirqs.CPU11.TIMER
     93965 ±  3%     +41.6%     133035        softirqs.CPU12.SCHED
    248423 ±  6%     +32.4%     328906 ± 11%  softirqs.CPU12.TIMER
    149906 ±  4%     +40.2%     210201 ±  6%  softirqs.CPU13.RCU
     90989           +49.6%     136105        softirqs.CPU13.SCHED
    251792 ±  4%     +42.1%     357900 ± 19%  softirqs.CPU13.TIMER
    152226 ±  2%     +35.9%     206915 ±  3%  softirqs.CPU14.RCU
     92046           +44.6%     133104        softirqs.CPU14.SCHED
    244799 ±  7%     +34.3%     328652 ± 11%  softirqs.CPU14.TIMER
    153485 ±  2%     +37.2%     210539 ±  4%  softirqs.CPU15.RCU
     91285           +47.8%     134965        softirqs.CPU15.SCHED
    252238 ±  4%     +32.8%     335030 ± 12%  softirqs.CPU15.TIMER
    147988 ±  2%     +40.2%     207451 ±  5%  softirqs.CPU16.RCU
     93079 ±  2%     +43.5%     133600        softirqs.CPU16.SCHED
    146479 ±  4%     +42.1%     208082 ±  7%  softirqs.CPU17.RCU
     91037           +48.3%     134972        softirqs.CPU17.SCHED
    254044 ±  4%     +33.6%     339310 ± 12%  softirqs.CPU17.TIMER
    150731 ±  2%     +35.3%     203892 ±  4%  softirqs.CPU18.RCU
     92163           +45.1%     133721 ±  2%  softirqs.CPU18.SCHED
    139984 ±  9%     +48.0%     207167 ±  6%  softirqs.CPU19.RCU
     87051 ±  8%     +55.4%     135288        softirqs.CPU19.SCHED
    256404 ±  3%     +31.0%     335908 ± 12%  softirqs.CPU19.TIMER
    153581           +33.5%     205065 ±  3%  softirqs.CPU2.RCU
     91389 ±  4%     +44.5%     132091 ±  2%  softirqs.CPU2.SCHED
    252327 ± 16%     +28.5%     324205 ± 11%  softirqs.CPU2.TIMER
     91204 ±  5%     +44.1%     131392 ±  2%  softirqs.CPU20.SCHED
    233549 ±  5%     +38.4%     323275 ± 11%  softirqs.CPU20.TIMER
    147747 ±  2%     +42.2%     210110 ±  6%  softirqs.CPU21.RCU
     91321 ±  3%     +45.1%     132530 ±  3%  softirqs.CPU21.SCHED
    261236 ± 13%     +19.5%     312199 ±  9%  softirqs.CPU21.TIMER
    152947           +36.4%     208566 ±  4%  softirqs.CPU22.RCU
     90046 ±  4%     +46.4%     131871        softirqs.CPU22.SCHED
    217967 ±  9%     +50.4%     327721 ± 11%  softirqs.CPU22.TIMER
    151811 ±  3%     +39.3%     211522 ±  6%  softirqs.CPU23.RCU
     91204           +48.4%     135364 ±  3%  softirqs.CPU23.SCHED
    250031 ±  3%     +19.2%     297969 ±  9%  softirqs.CPU23.TIMER
    158314           +33.2%     210939 ±  3%  softirqs.CPU24.RCU
     89110 ±  2%     +48.1%     131999 ±  2%  softirqs.CPU24.SCHED
    228849 ±  3%     +42.3%     325682 ± 11%  softirqs.CPU24.TIMER
    157157           +36.5%     214446 ±  5%  softirqs.CPU25.RCU
     90115           +47.5%     132882 ±  3%  softirqs.CPU25.SCHED
    246845 ±  4%     +27.7%     315332 ± 10%  softirqs.CPU25.TIMER
    157705           +33.4%     210455 ±  3%  softirqs.CPU26.RCU
     89991 ±  2%     +46.7%     132047 ±  2%  softirqs.CPU26.SCHED
    229330 ±  5%     +42.3%     326286 ± 11%  softirqs.CPU26.TIMER
    154583 ±  2%     +33.7%     206619 ±  2%  softirqs.CPU27.RCU
     90397           +47.6%     133442 ±  3%  softirqs.CPU27.SCHED
    247821 ±  4%     +27.8%     316809 ± 10%  softirqs.CPU27.TIMER
    150252 ±  9%     +39.8%     209980 ±  2%  softirqs.CPU28.RCU
     88703 ±  8%     +48.2%     131470        softirqs.CPU28.SCHED
    229069 ± 12%     +43.5%     328754 ± 11%  softirqs.CPU28.TIMER
    155740           +37.8%     214536 ±  5%  softirqs.CPU29.RCU
     90612           +48.1%     134168 ±  3%  softirqs.CPU29.SCHED
    248694 ±  4%     +41.4%     351652 ± 20%  softirqs.CPU29.TIMER
    149921 ±  3%     +40.5%     210592 ±  4%  softirqs.CPU3.RCU
     91270 ±  2%     +48.0%     135072 ±  3%  softirqs.CPU3.SCHED
    247492 ±  4%     +40.2%     347080 ± 21%  softirqs.CPU3.TIMER
    153901           +35.8%     209011 ±  3%  softirqs.CPU30.RCU
     90334 ±  2%     +46.3%     132136 ±  2%  softirqs.CPU30.SCHED
    243300 ±  6%     +34.8%     327959 ± 11%  softirqs.CPU30.TIMER
    149985 ±  3%     +41.2%     211733 ±  6%  softirqs.CPU31.RCU
     90682           +47.7%     133945        softirqs.CPU31.SCHED
    249462 ±  4%     +27.5%     318038 ±  9%  softirqs.CPU31.TIMER
    215.00 ±  4%   +4426.2%       9731 ±167%  softirqs.CPU32.NET_RX
    130036 ±  4%     +36.9%     177992 ±  3%  softirqs.CPU32.RCU
     87420 ±  9%     +52.8%     133544 ±  3%  softirqs.CPU32.SCHED
    248459 ±  7%     +34.4%     333827 ±  9%  softirqs.CPU32.TIMER
    131167 ±  2%     +36.7%     179367 ±  5%  softirqs.CPU33.RCU
     90830           +49.0%     135292        softirqs.CPU33.SCHED
    253786 ±  5%     +22.5%     310907 ± 13%  softirqs.CPU33.TIMER
    134361           +31.8%     177135        softirqs.CPU34.RCU
     92004           +43.8%     132280        softirqs.CPU34.SCHED
    244369 ±  7%     +33.7%     326841 ± 11%  softirqs.CPU34.TIMER
    137605 ±  2%     +30.7%     179816 ±  3%  softirqs.CPU35.RCU
     91052           +47.2%     134057        softirqs.CPU35.SCHED
    252092 ±  4%     +32.2%     333240 ± 12%  softirqs.CPU35.TIMER
    137204           +31.4%     180221 ±  2%  softirqs.CPU36.RCU
     93058 ±  2%     +42.6%     132667        softirqs.CPU36.SCHED
    236948 ± 13%     +39.1%     329695 ± 12%  softirqs.CPU36.TIMER
    138811 ±  2%     +31.6%     182739 ±  4%  softirqs.CPU37.RCU
     90951           +46.7%     133448 ±  2%  softirqs.CPU37.SCHED
    253893 ±  4%     +33.7%     339581 ± 13%  softirqs.CPU37.TIMER
    138263           +28.9%     178202        softirqs.CPU38.RCU
     92003           +43.7%     132213 ±  2%  softirqs.CPU38.SCHED
    257946 ±  3%     +28.0%     330201 ± 11%  softirqs.CPU38.TIMER
     93241 ±  3%     +43.2%     133556        softirqs.CPU39.SCHED
    254720 ±  4%     +32.1%     336502 ± 13%  softirqs.CPU39.TIMER
    292.50 ± 24%   +3105.5%       9376 ± 56%  softirqs.CPU4.NET_RX
    156349           +34.9%     210934 ±  3%  softirqs.CPU4.RCU
     89535 ±  2%     +48.3%     132816 ±  2%  softirqs.CPU4.SCHED
    229314 ±  3%     +43.5%     329036 ± 10%  softirqs.CPU4.TIMER
    155921 ±  2%     +35.7%     211534 ±  4%  softirqs.CPU5.RCU
     90416           +47.4%     133232 ±  2%  softirqs.CPU5.SCHED
    247319 ±  4%     +27.7%     315787 ± 10%  softirqs.CPU5.TIMER
    156332           +32.8%     207536 ±  3%  softirqs.CPU6.RCU
     90409 ±  2%     +46.2%     132144 ±  2%  softirqs.CPU6.SCHED
    229623 ±  4%     +41.8%     325539 ± 10%  softirqs.CPU6.TIMER
    154113 ±  2%     +34.4%     207084 ±  2%  softirqs.CPU7.RCU
     90560           +47.7%     133733 ±  2%  softirqs.CPU7.SCHED
    248302 ±  4%     +47.4%     365894 ± 16%  softirqs.CPU7.TIMER
     94614 ±  3%     +39.0%     131538        softirqs.CPU8.SCHED
    261763 ± 13%     +24.2%     325131 ± 11%  softirqs.CPU8.TIMER
    154317           +34.8%     208035 ±  4%  softirqs.CPU9.RCU
     90493           +48.3%     134160 ±  2%  softirqs.CPU9.SCHED
    248684 ±  4%     +22.7%     305256 ± 11%  softirqs.CPU9.TIMER
   6073358           +33.6%    8111475 ±  4%  softirqs.RCU
   3637411           +46.7%    5335018 ±  2%  softirqs.SCHED
   9847530 ±  3%     +33.2%   13112728 ± 10%  softirqs.TIMER
     61.94 ±  2%      -6.5       55.44        perf-profile.calltrace.cycles-pp.secondary_startup_64
      6.08 ± 32%      -6.1        0.00        perf-profile.calltrace.cycles-pp.xfs_inactive_inodes.xfs_inactive_worker.process_one_work.worker_thread.kthread
      6.08 ± 32%      -6.1        0.00        perf-profile.calltrace.cycles-pp.xfs_ici_walk_fns.xfs_inactive_inodes.xfs_inactive_worker.process_one_work.worker_thread
      6.08 ± 32%      -6.1        0.00        perf-profile.calltrace.cycles-pp.xfs_ici_walk_ag.xfs_ici_walk_fns.xfs_inactive_inodes.xfs_inactive_worker.process_one_work
      6.06 ± 32%      -6.1        0.00        perf-profile.calltrace.cycles-pp.xfs_inactive_inode.xfs_ici_walk_ag.xfs_ici_walk_fns.xfs_inactive_inodes.xfs_inactive_worker
      6.02 ± 33%      -6.0        0.00        perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_inactive_inode.xfs_ici_walk_ag.xfs_ici_walk_fns.xfs_inactive_inodes
      6.00 ± 33%      -6.0        0.00        perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode.xfs_ici_walk_ag.xfs_ici_walk_fns
     60.89 ±  2%      -5.5       55.40        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     60.89 ±  2%      -5.5       55.40        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     60.84 ±  2%      -5.5       55.37        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     55.86 ±  2%      -4.3       51.52 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     55.48 ±  2%      -4.2       51.25 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      7.39 ±  4%      -2.8        4.58 ±  9%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      6.97 ±  4%      -2.7        4.29 ± 10%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      3.90 ±  9%      -1.6        2.30 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.49 ±  6%      -1.3        1.20 ±  8%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.60 ± 11%      -1.1        1.51 ±  7%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      2.75 ±  4%      -0.7        2.02 ± 13%  perf-profile.calltrace.cycles-pp.xfs_dir2_block_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink
      2.21 ± 12%      -0.7        1.51 ± 19%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.39 ±  6%      -0.6        1.79 ±  4%  perf-profile.calltrace.cycles-pp.xfs_dir_createname.xfs_create.xfs_generic_create.path_openat.do_filp_open
      1.41 ± 13%      -0.5        0.87 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.59 ±  5%      -0.5        1.07 ±  9%  perf-profile.calltrace.cycles-pp.xfs_dir2_block_addname.xfs_dir_createname.xfs_create.xfs_generic_create.path_openat
      0.97 ± 12%      -0.5        0.47 ± 59%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.77 ± 15%      -0.5        1.28 ± 21%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      7.51 ±  4%      -0.5        7.04 ±  2%  perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.13 ± 12%      -0.5        0.68 ± 14%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      1.23 ±  4%      -0.4        0.82 ±  8%  perf-profile.calltrace.cycles-pp.xfs_dir3_data_check.xfs_dir2_block_addname.xfs_dir_createname.xfs_create.xfs_generic_create
      1.21 ±  4%      -0.4        0.80 ±  8%  perf-profile.calltrace.cycles-pp.__xfs_dir3_data_check.xfs_dir3_data_check.xfs_dir2_block_addname.xfs_dir_createname.xfs_create
      4.24 ±  3%      -0.4        3.87 ±  4%  perf-profile.calltrace.cycles-pp.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      1.38 ±  2%      -0.4        1.03 ± 16%  perf-profile.calltrace.cycles-pp.xfs_dir2_block_lookup_int.xfs_dir2_block_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink
      1.20 ±  8%      -0.3        0.85 ± 10%  perf-profile.calltrace.cycles-pp.xfs_dir3_data_check.xfs_dir2_block_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink
      1.18 ±  9%      -0.3        0.84 ± 10%  perf-profile.calltrace.cycles-pp.__xfs_dir3_data_check.xfs_dir3_data_check.xfs_dir2_block_removename.xfs_dir_removename.xfs_remove
      1.20 ±  2%      -0.3        0.90 ± 17%  perf-profile.calltrace.cycles-pp.__xfs_dir3_data_check.xfs_dir3_data_check.xfs_dir2_block_lookup_int.xfs_dir2_block_removename.xfs_dir_removename
      1.21 ±  2%      -0.3        0.92 ± 16%  perf-profile.calltrace.cycles-pp.xfs_dir3_data_check.xfs_dir2_block_lookup_int.xfs_dir2_block_removename.xfs_dir_removename.xfs_remove
      2.18 ±  6%      -0.3        1.90 ±  3%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_create.xfs_generic_create.path_openat.do_filp_open
      2.09 ±  5%      -0.3        1.81 ±  3%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_create.xfs_generic_create.path_openat
      0.87 ±  2%      -0.1        0.76 ±  2%  perf-profile.calltrace.cycles-pp.__xstat64
      0.77 ±  4%      -0.1        0.67 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__xstat64
      0.77 ±  4%      -0.1        0.67 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64
      0.67 ±  3%      -0.1        0.58 ±  4%  perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64
      0.59 ±  9%      +0.3        0.88 ±  8%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.26 ±100%      +0.3        0.59 ±  9%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.27 ±100%      +0.3        0.60 ± 10%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.46 ± 58%      +0.3        0.80 ±  5%  perf-profile.calltrace.cycles-pp.xfs_da3_node_lookup_int.xfs_dir2_node_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink
      0.74 ±  8%      +0.3        1.08 ±  9%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      0.78 ±  8%      +0.3        1.12 ± 10%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      0.78 ±  8%      +0.3        1.12 ± 10%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      0.77 ±  8%      +0.3        1.11 ± 10%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
      0.40 ± 58%      +0.3        0.75 ± 10%  perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      0.90 ±  8%      +0.4        1.25 ± 10%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.42 ± 58%      +0.4        0.78 ±  9%  perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
      1.92 ±  6%      +0.4        2.36 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
      1.02 ±  7%      +0.4        1.47 ±  6%  perf-profile.calltrace.cycles-pp.xfs_dir2_node_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink
      0.12 ±173%      +0.5        0.64 ±  4%  perf-profile.calltrace.cycles-pp.xfs_dir2_leafn_lookup_for_entry.xfs_da3_node_lookup_int.xfs_dir2_node_removename.xfs_dir_removename.xfs_remove
      0.95 ± 32%      +0.6        1.57 ± 18%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_difree_inobt
      1.08 ± 35%      +0.6        1.72 ± 14%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode
      0.59 ±  5%      +0.7        1.27 ±  3%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_dialloc_ag
      0.99 ± 36%      +0.7        1.67 ± 17%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree
     12.23 ±  2%      +0.7       12.95 ±  3%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      0.61 ±  6%      +0.7        1.33 ±  3%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc
      0.46 ±104%      +0.8        1.21 ± 14%  perf-profile.calltrace.cycles-pp.__xfs_btree_check_sblock.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_difree_inobt
      1.01 ± 27%      +0.8        1.81 ±  4%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
      1.32 ± 29%      +0.8        2.15 ± 17%  perf-profile.calltrace.cycles-pp.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree
      1.33 ± 35%      +0.9        2.26 ± 16%  perf-profile.calltrace.cycles-pp.xfs_btree_increment.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree.xfs_ifree
      0.00            +0.9        0.92 ±  2%  perf-profile.calltrace.cycles-pp.__xfs_btree_check_sblock.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_dialloc_ag
      0.81 ±  5%      +1.0        1.78 ±  3%  perf-profile.calltrace.cycles-pp.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc
      0.86 ±  6%      +1.0        1.87 ±  2%  perf-profile.calltrace.cycles-pp.xfs_btree_increment.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc
      0.76 ±  8%      +1.0        1.79 ±  6%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
      2.95 ± 12%      +1.2        4.19 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_open
      3.28 ±  2%      +1.2        4.52 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.18 ± 12%      +1.3        4.45 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_open.do_syscall_64
      3.54            +1.3        4.83 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
     10.09 ±  3%      +1.4       11.45        perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
      9.80 ±  3%      +1.5       11.25        perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_open
      0.55 ± 64%      +1.5        2.01 ±  9%  perf-profile.calltrace.cycles-pp.__xfs_btree_check_sblock.xfs_btree_check_sblock.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount
      1.32 ±  3%      +1.6        2.92        perf-profile.calltrace.cycles-pp.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc
      4.81 ±  3%      +2.4        7.18        perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      4.76 ±  3%      +2.4        7.16        perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      3.62 ±  4%      +2.4        6.03        perf-profile.calltrace.cycles-pp.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create
      4.14 ±  3%      +2.5        6.65        perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create
      3.67 ± 35%      +2.5        6.19 ± 15%  perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree.xfs_ifree.xfs_inactive_ifree
     15.05 ±  4%      +2.6       17.61 ±  2%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
     15.15 ±  4%      +2.6       17.71        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
     15.14 ±  4%      +2.6       17.70        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      3.91 ± 34%      +2.6        6.48 ± 15%  perf-profile.calltrace.cycles-pp.xfs_difree_inobt.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive
     14.85 ±  5%      +2.6       17.43 ±  2%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
     14.82 ±  5%      +2.6       17.41        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     15.41 ±  4%      +2.6       18.03 ±  2%  perf-profile.calltrace.cycles-pp.creat
      4.11 ± 34%      +2.7        6.83 ± 16%  perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode
      2.67 ±  4%      +2.8        5.46        perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc
      4.75 ± 32%      +3.0        7.71 ± 16%  perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode.xfs_ici_walk_ag
      6.08 ± 32%      +3.7        9.80 ± 16%  perf-profile.calltrace.cycles-pp.xfs_inactive_worker.process_one_work.worker_thread.kthread.ret_from_fork
      6.85 ± 29%      +3.8       10.61 ± 15%  perf-profile.calltrace.cycles-pp.ret_from_fork
      6.85 ± 29%      +3.8       10.61 ± 15%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      6.58 ± 29%      +3.8       10.34 ± 15%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      6.56 ± 29%      +3.8       10.32 ± 15%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +9.7        9.67 ± 16%  perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode.xfs_ici_walk_ag.xfs_inactive_worker
      0.00            +9.7        9.70 ± 16%  perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_inactive_inode.xfs_ici_walk_ag.xfs_inactive_worker.process_one_work
      0.00            +9.8        9.77 ± 16%  perf-profile.calltrace.cycles-pp.xfs_inactive_inode.xfs_ici_walk_ag.xfs_inactive_worker.process_one_work.worker_thread
      0.00            +9.8        9.80 ± 16%  perf-profile.calltrace.cycles-pp.xfs_ici_walk_ag.xfs_inactive_worker.process_one_work.worker_thread.kthread
     61.96 ±  2%      -6.5       55.45        perf-profile.children.cycles-pp.do_idle
     61.94 ±  2%      -6.5       55.44        perf-profile.children.cycles-pp.secondary_startup_64
     61.94 ±  2%      -6.5       55.44        perf-profile.children.cycles-pp.cpu_startup_entry
      6.08 ± 32%      -6.1        0.00        perf-profile.children.cycles-pp.xfs_inactive_inodes
      6.08 ± 32%      -6.1        0.00        perf-profile.children.cycles-pp.xfs_ici_walk_fns
     60.89 ±  2%      -5.5       55.40        perf-profile.children.cycles-pp.start_secondary
     56.85 ±  2%      -5.3       51.56 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
     56.81 ±  2%      -5.3       51.55 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
      8.15 ±  4%      -2.7        5.42 ± 11%  perf-profile.children.cycles-pp.apic_timer_interrupt
      7.68 ±  4%      -2.6        5.07 ± 11%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      4.43 ±  8%      -1.5        2.89 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
      2.56 ±  4%      -1.3        1.22 ±  7%  perf-profile.children.cycles-pp.menu_select
      3.00 ±  9%      -1.0        1.97 ±  9%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.76 ±  3%      -0.8        3.97 ±  4%  perf-profile.children.cycles-pp.__xfs_dir3_data_check
      4.77 ±  3%      -0.8        3.99 ±  4%  perf-profile.children.cycles-pp.xfs_dir3_data_check
      2.75 ±  4%      -0.7        2.02 ± 13%  perf-profile.children.cycles-pp.xfs_dir2_block_removename
      2.35 ± 12%      -0.7        1.68 ± 19%  perf-profile.children.cycles-pp.irq_exit
      2.39 ±  6%      -0.6        1.79 ±  4%  perf-profile.children.cycles-pp.xfs_dir_createname
      0.88 ±  5%      -0.6        0.29 ± 17%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.59 ±  5%      -0.5        1.07 ±  9%  perf-profile.children.cycles-pp.xfs_dir2_block_addname
      7.51 ±  4%      -0.5        7.05 ±  2%  perf-profile.children.cycles-pp.vfs_unlink
      1.65 ± 11%      -0.5        1.19 ± 15%  perf-profile.children.cycles-pp.tick_sched_timer
      0.66 ±  8%      -0.5        0.21 ± 21%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.89 ± 16%      -0.4        1.46 ± 20%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.35 ± 10%      -0.4        0.97 ± 18%  perf-profile.children.cycles-pp.tick_sched_handle
      4.24 ±  3%      -0.4        3.87 ±  4%  perf-profile.children.cycles-pp.xfs_dir_removename
      1.58 ±  3%      -0.4        1.23 ± 22%  perf-profile.children.cycles-pp.xfs_dir2_block_lookup_int
      0.67 ±  8%      -0.3        0.35 ±  9%  perf-profile.children.cycles-pp.irq_enter
      1.20 ±  9%      -0.3        0.89 ± 19%  perf-profile.children.cycles-pp.update_process_times
      0.36 ± 11%      -0.3        0.06 ± 65%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.40 ±  7%      -0.3        0.14 ±  6%  perf-profile.children.cycles-pp.xfs_dialloc_ag_update_inobt
      0.48 ±  8%      -0.2        0.24 ± 13%  perf-profile.children.cycles-pp.tick_irq_enter
      0.24 ±  2%      -0.2        0.03 ±102%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.48 ± 11%      -0.2        0.29 ±  3%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.37 ± 12%      -0.2        0.19 ± 24%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.54 ± 19%      -0.2        0.37 ± 22%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.30 ± 11%      -0.2        0.14 ±  5%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.24 ±  8%      -0.2        0.08 ± 16%  perf-profile.children.cycles-pp.__slab_alloc
      0.38 ±  7%      -0.1        0.23 ± 12%  perf-profile.children.cycles-pp.read_tsc
      0.32 ±  7%      -0.1        0.17 ± 22%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.40 ± 10%      -0.1        0.26 ± 21%  perf-profile.children.cycles-pp.native_write_msr
      0.22 ±  3%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.___slab_alloc
      0.41 ± 11%      -0.1        0.27 ± 19%  perf-profile.children.cycles-pp.kmem_zone_alloc
      0.21 ± 17%      -0.1        0.07 ± 19%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.62 ± 13%      -0.1        0.49 ±  8%  perf-profile.children.cycles-pp.console_unlock
      0.26 ±  7%      -0.1        0.13 ± 27%  perf-profile.children.cycles-pp.timerqueue_del
      0.31 ±  4%      -0.1        0.18 ± 33%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.49 ± 11%      -0.1        0.36 ± 11%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.32 ± 20%      -0.1        0.20 ± 13%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.88 ±  3%      -0.1        0.76 ±  2%  perf-profile.children.cycles-pp.__xstat64
      0.50 ±  2%      -0.1        0.38 ± 11%  perf-profile.children.cycles-pp.close
      0.33 ± 16%      -0.1        0.22 ± 18%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.32 ±  6%      -0.1        0.21 ± 19%  perf-profile.children.cycles-pp.__lock_text_start
      0.18 ±  7%      -0.1        0.07 ± 12%  perf-profile.children.cycles-pp.find_next_bit
      0.71 ±  7%      -0.1        0.60 ± 11%  perf-profile.children.cycles-pp.xfs_da_read_buf
      0.82 ±  3%      -0.1        0.72 ±  3%  perf-profile.children.cycles-pp.xfs_dir_ino_validate
      0.32 ± 18%      -0.1        0.22 ± 17%  perf-profile.children.cycles-pp.xfs_dir3_block_read
      0.23 ±  2%      -0.1        0.14 ± 19%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.65 ±  5%      -0.1        0.56 ±  9%  perf-profile.children.cycles-pp.link_path_walk
      0.34 ± 15%      -0.1        0.25 ±  7%  perf-profile.children.cycles-pp.destroy_inode
      0.19 ± 13%      -0.1        0.10 ±  9%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.59            -0.1        0.49 ±  5%  perf-profile.children.cycles-pp.xfs_verify_ino
      0.67 ±  3%      -0.1        0.58 ±  4%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.30 ±  7%      -0.1        0.22 ± 19%  perf-profile.children.cycles-pp.xfs_dabuf_map
      0.11 ± 20%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.pm_qos_request
      0.28 ±  8%      -0.1        0.20 ±  7%  perf-profile.children.cycles-pp.__kmalloc
      0.25 ±  4%      -0.1        0.17 ± 11%  perf-profile.children.cycles-pp.run_timer_softirq
      0.14 ± 16%      -0.1        0.06 ± 63%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.19 ±  9%      -0.1        0.11 ± 20%  perf-profile.children.cycles-pp.timerqueue_add
      0.23 ± 16%      -0.1        0.15 ± 12%  perf-profile.children.cycles-pp.delay_tsc
      0.17 ±  4%      -0.1        0.10 ± 18%  perf-profile.children.cycles-pp.xfs_inode_alloc
      0.10 ± 11%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_btree_del_cursor
      0.19 ± 14%      -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.kmem_alloc_large
      0.17 ± 14%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.note_gp_changes
      0.30 ±  7%      -0.1        0.23 ±  6%  perf-profile.children.cycles-pp.kmem_alloc
      0.28 ±  9%      -0.1        0.21 ±  5%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.10 ±  4%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.get_cpu_device
      0.18 ± 20%      -0.1        0.11 ±  6%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.22 ± 11%      -0.1        0.16 ± 17%  perf-profile.children.cycles-pp.xfs_bmapi_read
      0.18 ± 17%      -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.xfs_default_hashname
      0.15 ± 13%      -0.1        0.10 ± 22%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.24 ± 14%      -0.1        0.18 ±  7%  perf-profile.children.cycles-pp.sched_clock
      0.23 ± 12%      -0.1        0.18 ±  8%  perf-profile.children.cycles-pp.native_sched_clock
      0.11 ± 25%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.rb_next
      0.10 ± 14%      -0.1        0.05        perf-profile.children.cycles-pp.__x64_sys_close
      0.08 ± 10%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_trans_brelse
      0.35 ±  9%      -0.1        0.30 ±  2%  perf-profile.children.cycles-pp.xfs_dir3_data_get_ftype
      0.09 ±  8%      -0.0        0.04 ± 63%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.10 ± 14%      -0.0        0.05 ± 59%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.14 ± 13%      -0.0        0.09 ± 20%  perf-profile.children.cycles-pp.__destroy_inode
      0.14 ± 25%      -0.0        0.09 ± 23%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.11 ± 24%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.account_process_tick
      0.12 ±  4%      -0.0        0.08 ± 15%  perf-profile.children.cycles-pp.rb_erase
      0.09 ± 17%      -0.0        0.04 ± 59%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.10 ± 15%      -0.0        0.06 ± 20%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.07 ±  7%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp._xfs_buf_ioapply
      0.08 ±  5%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.xfs_buf_item_put
      0.11 ± 10%      -0.0        0.07 ± 26%  perf-profile.children.cycles-pp.d_instantiate
      0.10 ±  4%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.brd_make_request
      0.11 ±  7%      -0.0        0.09 ± 12%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.10 ±  8%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.dput
      0.08 ±  5%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.brd_do_bvec
      0.07 ± 12%      +0.0        0.10 ± 11%  perf-profile.children.cycles-pp.___perf_sw_event
      0.20 ±  9%      +0.0        0.24 ±  6%  perf-profile.children.cycles-pp.xfs_inobt_update
      0.11 ± 17%      +0.0        0.15 ± 12%  perf-profile.children.cycles-pp.xfs_btree_check_block
      0.03 ±100%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.11 ± 17%      +0.0        0.16 ± 11%  perf-profile.children.cycles-pp.xfs_log_done
      0.11 ± 10%      +0.1        0.16 ± 15%  perf-profile.children.cycles-pp.xfs_iextents_copy
      0.07 ± 14%      +0.1        0.13 ± 12%  perf-profile.children.cycles-pp.xfs_btree_ptr_to_daddr
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.check_preempt_curr
      0.03 ±100%      +0.1        0.09 ± 26%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.07 ± 17%      +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.xfs_iunlink_map_prev
      0.03 ±100%      +0.1        0.09 ± 44%  perf-profile.children.cycles-pp.xfs_iunlink_map_ino
      0.15 ± 19%      +0.1        0.21 ± 12%  perf-profile.children.cycles-pp.xfs_dir3_leaf_check_int
      0.14 ± 22%      +0.1        0.21 ± 22%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.08 ± 60%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
      0.11 ± 30%      +0.1        0.19 ± 20%  perf-profile.children.cycles-pp.kernel_text_address
      0.27 ±  7%      +0.1        0.35 ± 10%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.08 ± 27%      +0.1        0.16 ± 16%  perf-profile.children.cycles-pp.xfs_inobt_rec_freecount
      0.04 ±100%      +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.__module_text_address
      0.04 ±100%      +0.1        0.13 ± 23%  perf-profile.children.cycles-pp.is_module_text_address
      0.40 ± 11%      +0.1        0.49 ±  7%  perf-profile.children.cycles-pp.wake_up_q
      0.13 ± 23%      +0.1        0.22 ± 19%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.04 ±100%      +0.1        0.13 ± 17%  perf-profile.children.cycles-pp.xfs_inobt_key_diff
      0.23 ±  6%      +0.1        0.32 ± 18%  perf-profile.children.cycles-pp.xfs_imap_to_bp
      0.11 ± 33%      +0.1        0.21 ± 20%  perf-profile.children.cycles-pp.__kernel_text_address
      0.23 ±  7%      +0.1        0.33 ± 11%  perf-profile.children.cycles-pp.__orc_find
      0.20 ±  9%      +0.1        0.30 ± 10%  perf-profile.children.cycles-pp.xfs_btree_rec_offset
      0.07 ± 14%      +0.1        0.18 ± 17%  perf-profile.children.cycles-pp.xfs_btree_readahead
      0.33 ± 12%      +0.1        0.43 ± 12%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.29 ± 12%      +0.1        0.39 ±  6%  perf-profile.children.cycles-pp.orc_find
      0.49 ± 10%      +0.1        0.60 ± 10%  perf-profile.children.cycles-pp.schedule_idle
      0.38 ±  7%      +0.1        0.50 ±  5%  perf-profile.children.cycles-pp.xfs_ialloc_read_agi
      0.16 ± 19%      +0.1        0.29 ±  7%  perf-profile.children.cycles-pp.xfs_btree_get_block
      0.18 ± 25%      +0.1        0.32 ± 23%  perf-profile.children.cycles-pp.xfs_difree_finobt
      0.65 ± 21%      +0.2        0.84 ±  8%  perf-profile.children.cycles-pp.xfs_buf_unlock
      0.74 ± 20%      +0.2        0.94 ±  7%  perf-profile.children.cycles-pp.xfs_buf_item_release
      0.63 ±  7%      +0.2        0.84 ±  4%  perf-profile.children.cycles-pp.unwind_next_frame
      0.44 ± 13%      +0.2        0.65 ±  3%  perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.55 ± 23%      +0.2        0.77 ±  9%  perf-profile.children.cycles-pp.up
      0.28 ± 31%      +0.2        0.51 ±  7%  perf-profile.children.cycles-pp.xfs_inobt_btrec_to_irec
      0.79 ± 11%      +0.3        1.04 ±  7%  perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
      1.25 ± 12%      +0.3        1.54 ± 10%  perf-profile.children.cycles-pp.__sched_text_start
      0.97 ± 16%      +0.3        1.27 ±  6%  perf-profile.children.cycles-pp.try_to_wake_up
      0.93 ±  7%      +0.4        1.29 ±  9%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.92 ± 11%      +0.4        1.29 ±  6%  perf-profile.children.cycles-pp.arch_stack_walk
      0.96 ± 11%      +0.4        1.36 ±  6%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      1.07 ± 10%      +0.4        1.51 ±  5%  perf-profile.children.cycles-pp.__account_scheduler_latency
      1.02 ±  7%      +0.4        1.47 ±  6%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      1.36 ±  9%      +0.5        1.86 ±  7%  perf-profile.children.cycles-pp.enqueue_entity
      1.41 ± 10%      +0.5        1.92 ±  7%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.43 ± 10%      +0.5        1.94 ±  7%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.43 ± 10%      +0.5        1.94 ±  7%  perf-profile.children.cycles-pp.activate_task
      0.62 ± 41%      +0.6        1.17 ±  9%  perf-profile.children.cycles-pp.xfs_verify_agino
      0.80 ± 15%      +0.6        1.36 ± 10%  perf-profile.children.cycles-pp.xfs_errortag_test
      4.44 ±  3%      +0.7        5.09 ±  6%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
     12.23 ±  2%      +0.7       12.95 ±  3%  perf-profile.children.cycles-pp.do_unlinkat
      1.14 ± 18%      +0.9        2.05 ± 12%  perf-profile.children.cycles-pp.xfs_inobt_get_maxrecs
     10.09 ±  3%      +1.4       11.45        perf-profile.children.cycles-pp.xfs_generic_create
      9.80 ±  3%      +1.4       11.25        perf-profile.children.cycles-pp.xfs_create
      2.21 ± 16%      +1.8        4.05 ±  9%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      1.79 ± 16%      +1.9        3.65 ±  5%  perf-profile.children.cycles-pp.osq_lock
      2.27 ± 20%      +2.0        4.24 ±  8%  perf-profile.children.cycles-pp.xfs_btree_increment
      2.40 ± 19%      +2.0        4.43 ±  8%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      4.81 ±  3%      +2.4        7.18        perf-profile.children.cycles-pp.xfs_dir_ialloc
      4.76 ±  3%      +2.4        7.16        perf-profile.children.cycles-pp.xfs_ialloc
      3.62 ±  3%      +2.4        6.03        perf-profile.children.cycles-pp.xfs_dialloc_ag
      4.14 ±  3%      +2.5        6.65        perf-profile.children.cycles-pp.xfs_dialloc
      6.28 ±  6%      +2.5        8.79 ±  5%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
     15.09 ±  4%      +2.6       17.64 ±  2%  perf-profile.children.cycles-pp.do_sys_open
      6.72 ±  6%      +2.6        9.28 ±  4%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      3.91 ± 34%      +2.6        6.48 ± 15%  perf-profile.children.cycles-pp.xfs_difree_inobt
     14.88 ±  5%      +2.6       17.45        perf-profile.children.cycles-pp.do_filp_open
     14.85 ±  5%      +2.6       17.43        perf-profile.children.cycles-pp.path_openat
     15.43 ±  4%      +2.6       18.05 ±  2%  perf-profile.children.cycles-pp.creat
      4.11 ± 34%      +2.7        6.83 ± 16%  perf-profile.children.cycles-pp.xfs_difree
      3.28 ± 19%      +2.8        6.09 ±  9%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
     29.22 ±  3%      +2.9       32.14 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     29.19 ±  3%      +2.9       32.12 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
      4.75 ± 32%      +3.0        7.71 ± 16%  perf-profile.children.cycles-pp.xfs_ifree
      3.55 ± 22%      +3.0        6.56 ±  8%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      6.00 ± 33%      +3.7        9.67 ± 16%  perf-profile.children.cycles-pp.xfs_inactive_ifree
      6.02 ± 33%      +3.7        9.70 ± 16%  perf-profile.children.cycles-pp.xfs_inactive
      6.06 ± 32%      +3.7        9.77 ± 16%  perf-profile.children.cycles-pp.xfs_inactive_inode
      6.08 ± 32%      +3.7        9.80 ± 16%  perf-profile.children.cycles-pp.xfs_inactive_worker
      6.08 ± 32%      +3.7        9.80 ± 16%  perf-profile.children.cycles-pp.xfs_ici_walk_ag
      6.86 ± 29%      +3.8       10.61 ± 15%  perf-profile.children.cycles-pp.ret_from_fork
      6.85 ± 29%      +3.8       10.61 ± 15%  perf-profile.children.cycles-pp.kthread
      6.58 ± 29%      +3.8       10.34 ± 15%  perf-profile.children.cycles-pp.worker_thread
      6.56 ± 29%      +3.8       10.32 ± 15%  perf-profile.children.cycles-pp.process_one_work
      6.42 ± 20%      +5.4       11.80 ±  8%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      1.24 ±  4%      -0.6        0.69 ±  8%  perf-profile.self.cycles-pp.menu_select
      2.15 ±  4%      -0.4        1.70 ±  6%  perf-profile.self.cycles-pp.__xfs_dir3_data_check
      0.71 ±  8%      -0.3        0.43 ± 17%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.30 ± 11%      -0.2        0.14 ±  5%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.36 ±  6%      -0.1        0.21 ± 15%  perf-profile.self.cycles-pp.read_tsc
      0.40 ±  9%      -0.1        0.26 ± 21%  perf-profile.self.cycles-pp.native_write_msr
      0.19 ± 15%      -0.1        0.07 ± 17%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.16 ±  9%      -0.1        0.06 ± 20%  perf-profile.self.cycles-pp.find_next_bit
      0.22 ± 14%      -0.1        0.12 ± 13%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.20 ± 21%      -0.1        0.10 ± 25%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.12 ± 20%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.clockevents_program_event
      0.29 ± 13%      -0.1        0.20 ±  7%  perf-profile.self.cycles-pp.do_idle
      0.11 ± 20%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.pm_qos_request
      0.15 ± 17%      -0.1        0.08 ± 16%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.36 ± 17%      -0.1        0.29 ± 18%  perf-profile.self.cycles-pp.io_serial_in
      0.14 ± 15%      -0.1        0.07 ± 61%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.10 ± 13%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.irq_exit
      0.10 ±  5%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.get_cpu_device
      0.25 ± 10%      -0.1        0.19 ±  8%  perf-profile.self.cycles-pp.xfs_verify_ino
      0.49 ±  5%      -0.1        0.43 ±  5%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.16 ± 15%      -0.1        0.10 ± 12%  perf-profile.self.cycles-pp.xfs_default_hashname
      0.22 ± 15%      -0.1        0.16 ± 13%  perf-profile.self.cycles-pp.native_sched_clock
      0.19 ±  7%      -0.1        0.13 ±  8%  perf-profile.self.cycles-pp.run_timer_softirq
      0.33 ±  8%      -0.1        0.28 ±  2%  perf-profile.self.cycles-pp.xfs_dir3_data_get_ftype
      0.09 ±  8%      -0.0        0.04 ± 63%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.12 ±  4%      -0.0        0.08 ± 19%  perf-profile.self.cycles-pp.rb_erase
      0.11 ± 24%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.account_process_tick
      0.20 ±  9%      -0.0        0.15 ± 12%  perf-profile.self.cycles-pp.delay_tsc
      0.13 ± 23%      -0.0        0.09 ± 23%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.12 ± 12%      -0.0        0.08 ± 13%  perf-profile.self.cycles-pp.timerqueue_add
      0.08 ±  5%      -0.0        0.05 ± 61%  perf-profile.self.cycles-pp.native_apic_mem_write
      0.10 ±  7%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.07 ± 10%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.09 ± 23%      -0.0        0.06 ± 17%  perf-profile.self.cycles-pp.rcu_core
      0.07 ±  7%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.xfs_buf_item_put
      0.11 ±  4%      -0.0        0.09 ± 12%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.08 ± 19%      +0.0        0.11 ± 17%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.13 ± 12%      +0.0        0.17 ±  5%  perf-profile.self.cycles-pp.update_load_avg
      0.04 ± 59%      +0.0        0.09 ± 15%  perf-profile.self.cycles-pp.xfs_btree_check_block
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.xfs_btree_ptr_to_daddr
      0.14 ± 18%      +0.1        0.20 ± 10%  perf-profile.self.cycles-pp.xfs_dir3_leaf_check_int
      0.03 ±105%      +0.1        0.11 ± 20%  perf-profile.self.cycles-pp.xfs_inobt_key_diff
      0.05 ± 62%      +0.1        0.14 ± 19%  perf-profile.self.cycles-pp.xfs_inobt_rec_freecount
      0.14 ±  8%      +0.1        0.22 ±  8%  perf-profile.self.cycles-pp.xfs_btree_get_block
      0.07 ± 20%      +0.1        0.16 ± 18%  perf-profile.self.cycles-pp.xfs_btree_readahead
      0.23 ±  7%      +0.1        0.33 ± 10%  perf-profile.self.cycles-pp.__orc_find
      0.15 ± 19%      +0.1        0.25 ±  7%  perf-profile.self.cycles-pp.xfs_check_agi_freecount
      0.19 ±  8%      +0.1        0.29 ± 13%  perf-profile.self.cycles-pp.xfs_btree_rec_offset
      0.15 ± 28%      +0.1        0.27 ± 13%  perf-profile.self.cycles-pp.xfs_ag_block_count
      0.26 ± 29%      +0.2        0.46 ±  6%  perf-profile.self.cycles-pp.xfs_inobt_btrec_to_irec
      0.21 ± 29%      +0.2        0.42 ±  8%  perf-profile.self.cycles-pp.xfs_verify_agino
      0.50 ± 17%      +0.2        0.74 ±  7%  perf-profile.self.cycles-pp.xfs_agino_range
      0.26 ± 18%      +0.3        0.55 ± 13%  perf-profile.self.cycles-pp.xfs_btree_check_sblock
      0.34 ± 32%      +0.3        0.67 ±  9%  perf-profile.self.cycles-pp.xfs_btree_get_rec
      0.40 ± 17%      +0.3        0.74 ± 11%  perf-profile.self.cycles-pp.xfs_inobt_get_rec
      0.42 ± 13%      +0.4        0.80 ±  9%  perf-profile.self.cycles-pp.xfs_btree_increment
      0.73 ± 19%      +0.5        1.25 ± 11%  perf-profile.self.cycles-pp.xfs_errortag_test
      4.36 ±  3%      +0.6        4.97 ±  5%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.95 ± 14%      +0.8        1.77 ± 13%  perf-profile.self.cycles-pp.xfs_inobt_get_maxrecs
      1.20 ± 22%      +1.0        2.20 ±  7%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      1.75 ± 16%      +1.8        3.59 ±  5%  perf-profile.self.cycles-pp.osq_lock


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  32000 +-+-----------------------------------------------------------------+   
        |.+. +.+.+. .++.+.+.++.+.+.+.++.+. .+.  .+. .+.        +.+          |   
  30000 +-+ +      +                      +   ++   +   ++.+.+.+   :         |   
        |                                                         : .+.++.+.|   
  28000 +-+                                                        +        |   
        |                                                                   |   
  26000 +-+                                                                 |   
        |                                                                   |   
  24000 +-+                                                                 |   
        |                                                                   |   
  22000 +-+                                                                 |   
        |                                                                   |   
  20000 +-+        O        OO O O O O                                      |   
        O O OO O O   OO O O                                                 |   
  18000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               aim7.time.system_time                            
                                                                                
  6000 +-+------------------------------------------------------------------+   
       O                                                                    |   
  5500 +-O OO O O O O OO O O   OO O                                         |   
       |                     O      O                                       |   
       |                                                                    |   
  5000 +-+                                                                  |   
       |                                                                    |   
  4500 +-+                                                                  |   
       |                                                                    |   
  4000 +-+                                                                  |   
       |                                                                    |   
       |                                                                +.  |   
  3500 +-+.++.+.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.+.+  +.|   
       |                                                                    |   
  3000 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  950 +-+-------------------------------------------------------------------+   
      O O O OO O O O O OO O O O O OO                                        |   
  900 +-+                                                                   |   
  850 +-+                                                                   |   
      |                                                                     |   
  800 +-+                                                                   |   
      |                                                                     |   
  750 +-+                                                                   |   
      |                                                                     |   
  700 +-+                                                                   |   
  650 +-+                                                                   |   
      |                                                            +.++.+.+.|   
  600 +-+       .+.                     .+. .+.+ .+. .+.+.++.     +         |   
      |.+.+.++.+   +.+.++.+.+.+.+.++.+.+   +    +   +        +.+.+          |   
  550 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  950 +-+-------------------------------------------------------------------+   
      O O O OO O O O O OO O O O O OO                                        |   
  900 +-+                                                                   |   
  850 +-+                                                                   |   
      |                                                                     |   
  800 +-+                                                                   |   
      |                                                                     |   
  750 +-+                                                                   |   
      |                                                                     |   
  700 +-+                                                                   |   
  650 +-+                                                                   |   
      |                                                            +.++.+.+.|   
  600 +-+       .+.                     .+. .+.+ .+. .+.+.++.     +         |   
      |.+.+.++.+   +.+.++.+.+.+.+.++.+.+   +    +   +        +.+.+          |   
  550 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.3.0-rc7-00096-g73a16a2894549" of type "text/plain" (199463 bytes)

View attachment "job-script" of type "text/plain" (8042 bytes)

View attachment "job.yaml" of type "text/plain" (5739 bytes)

View attachment "reproduce" of type "text/plain" (1046 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ