lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200304085116.GA5972@shao2-debian>
Date:   Wed, 4 Mar 2020 16:51:16 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Darrick J. Wong" <darrick.wong@...cle.com>
Cc:     Christoph Hellwig <hch@....de>,
        LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...ts.01.org
Subject: [xfs] 41d44159a3: aim7.jobs-per-min -23.2% regression

Greeting,

FYI, we noticed a -23.2% regression of aim7.jobs-per-min due to commit:


commit: 41d44159a357586656a11e5624c6b4ee61896960 ("xfs: force writes to delalloc regions to unwritten")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git verifier-fixes

in testcase: aim7
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	disk: 1BRD_48G
	fs: xfs
	test: sync_disk_rw
	load: 600
	cpufreq_governor: performance
	ucode: 0x2000065

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/1BRD_48G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-skl-2sp7/sync_disk_rw/aim7/0x2000065

commit: 
  v5.6-rc3
  41d44159a3 ("xfs: force writes to delalloc regions to unwritten")

        v5.6-rc3 41d44159a357586656a11e5624c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     29394           -23.2%      22585        aim7.jobs-per-min
    122.54           +30.1%     159.44        aim7.time.elapsed_time
    122.54           +30.1%     159.44        aim7.time.elapsed_time.max
    839395           +22.5%    1027864        aim7.time.involuntary_context_switches
     52609            +8.0%      56817        aim7.time.minor_page_faults
      4976           +23.7%       6158        aim7.time.system_time
     50.32            +4.9%      52.78        aim7.time.user_time
  62497154           +33.5%   83423990        aim7.time.voluntary_context_switches
      9352 ±121%    +151.6%      23531        numa-numastat.node0.other_node
     40.06            +3.8%      41.57        iostat.cpu.idle
     59.24            -2.6%      57.70        iostat.cpu.system
      0.01 ± 13%      +0.2        0.18 ± 16%  mpstat.cpu.all.iowait%
      0.68            -0.1        0.55        mpstat.cpu.all.usr%
     39.60            +3.5%      41.00        vmstat.cpu.id
     58.80            -3.1%      57.00        vmstat.cpu.sy
   1068131           -16.0%     897006        vmstat.io.bo
    873514            +8.0%     943352        vmstat.system.cs
    103012 ±  3%     +30.2%     134107 ±  3%  meminfo.AnonHugePages
     82859            +9.3%      90604        meminfo.Dirty
     23122           +40.2%      32422 ±  2%  meminfo.KernelStack
    168083           +13.1%     190065        meminfo.SUnreclaim
     24634           -20.6%      19556        meminfo.max_used_kB
 6.407e+08           +31.7%  8.437e+08        cpuidle.C1.time
  17320738           +23.0%   21301068        cpuidle.C1.usage
 2.516e+09 ± 12%     +45.6%  3.662e+09 ±  4%  cpuidle.C1E.time
  24679196 ±  2%     +43.1%   35318973        cpuidle.C1E.usage
  12523597           +34.9%   16892668        cpuidle.POLL.time
   2600319           +24.6%    3241112        cpuidle.POLL.usage
     10341           +10.8%      11462 ±  2%  numa-vmstat.node0.nr_dirty
      4647 ± 38%     -58.6%       1925 ± 99%  numa-vmstat.node0.nr_page_table_pages
     10567 ± 10%     -22.1%       8230 ±  7%  numa-vmstat.node0.nr_slab_reclaimable
      8116 ± 45%    +125.0%      18265 ± 26%  numa-vmstat.node1.nr_kernel_stack
      1846 ± 93%    +140.6%       4443 ± 43%  numa-vmstat.node1.nr_page_table_pages
     10293 ± 11%     +27.7%      13149 ±  4%  numa-vmstat.node1.nr_slab_reclaimable
     19223 ± 10%     +24.8%      23982 ±  9%  numa-vmstat.node1.nr_slab_unreclaimable
     40847           +12.4%      45929        numa-meminfo.node0.Dirty
     42270 ± 10%     -22.1%      32923 ±  7%  numa-meminfo.node0.KReclaimable
     18587 ± 38%     -58.5%       7722 ± 99%  numa-meminfo.node0.PageTables
     42270 ± 10%     -22.1%      32923 ±  7%  numa-meminfo.node0.SReclaimable
     41180 ± 11%     +27.7%      52597 ±  4%  numa-meminfo.node1.KReclaimable
      8103 ± 45%    +125.3%      18258 ± 26%  numa-meminfo.node1.KernelStack
   1436886 ±  5%     +12.7%    1619084 ±  5%  numa-meminfo.node1.MemUsed
      7367 ± 93%    +141.2%      17772 ± 43%  numa-meminfo.node1.PageTables
     41180 ± 11%     +27.7%      52597 ±  4%  numa-meminfo.node1.SReclaimable
     76895 ± 10%     +24.8%      95934 ±  9%  numa-meminfo.node1.SUnreclaim
    118075 ±  8%     +25.8%     148533 ±  5%  numa-meminfo.node1.Slab
     20698           +10.8%      22929        proc-vmstat.nr_dirty
      4910            -2.8%       4774        proc-vmstat.nr_inactive_anon
     70258            +3.0%      72338        proc-vmstat.nr_inactive_file
     23112           +40.3%      32434 ±  3%  proc-vmstat.nr_kernel_stack
     20863            +2.5%      21380        proc-vmstat.nr_slab_reclaimable
     42017           +13.1%      47514        proc-vmstat.nr_slab_unreclaimable
      4910            -2.8%       4774        proc-vmstat.nr_zone_inactive_anon
     70258            +3.0%      72338        proc-vmstat.nr_zone_inactive_file
     19943            +8.4%      21627        proc-vmstat.nr_zone_write_pending
   8277577            +1.7%    8418473        proc-vmstat.numa_hit
   8254156            +1.7%    8394365        proc-vmstat.numa_local
     23421            +2.9%      24108        proc-vmstat.numa_other
   8362865            +1.9%    8517751        proc-vmstat.pgalloc_normal
    413855           +21.7%     503821        proc-vmstat.pgfault
 1.337e+08            +8.6%  1.451e+08        proc-vmstat.pgpgout
   2435788           -11.9%    2146497        sched_debug.cfs_rq:/.MIN_vruntime.max
      4072 ±  9%     +24.3%       5060 ±  4%  sched_debug.cfs_rq:/.load_avg.max
    902.68 ±  3%     +15.6%       1043 ±  3%  sched_debug.cfs_rq:/.load_avg.stddev
   2435788           -11.9%    2146497        sched_debug.cfs_rq:/.max_vruntime.max
   2428275           -12.1%    2133587        sched_debug.cfs_rq:/.min_vruntime.avg
   2478382           -12.1%    2179499        sched_debug.cfs_rq:/.min_vruntime.max
   2384628           -11.9%    2101188        sched_debug.cfs_rq:/.min_vruntime.min
      6.57 ± 14%    +100.9%      13.20 ± 20%  sched_debug.cfs_rq:/.nr_spread_over.avg
     14.33 ± 11%     +70.9%      24.50 ± 26%  sched_debug.cfs_rq:/.nr_spread_over.max
      2.82 ± 10%     +43.7%       4.05 ± 13%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    -66658           -42.9%     -38086        sched_debug.cfs_rq:/.spread0.min
      2151 ± 15%     +15.4%       2483 ± 14%  sched_debug.cfs_rq:/.util_avg.max
      9773 ± 18%     -37.4%       6116 ± 15%  sched_debug.cpu.avg_idle.min
      3.57 ±  8%     +48.0%       5.29 ±  3%  sched_debug.cpu.clock.stddev
      3.57 ±  8%     +48.0%       5.29 ±  3%  sched_debug.cpu.clock_task.stddev
      3966           +10.8%       4392        sched_debug.cpu.curr->pid.max
     -1567           -27.5%      -1136        sched_debug.cpu.nr_uninterruptible.min
     23858          +239.6%      81025        sched_debug.cpu.ttwu_local.avg
     25153          +231.8%      83456        sched_debug.cpu.ttwu_local.max
     22454          +251.1%      78831        sched_debug.cpu.ttwu_local.min
    630.46 ±  5%     +79.9%       1134 ± 25%  sched_debug.cpu.ttwu_local.stddev
    567.80           +98.9%       1129 ±  8%  slabinfo.biovec-max.active_objs
    567.80           +98.9%       1129 ±  8%  slabinfo.biovec-max.num_objs
      2930          -100.0%       0.00        slabinfo.dmaengine-unmap-16.active_objs
      2930          -100.0%       0.00        slabinfo.dmaengine-unmap-16.num_objs
      3791           +27.4%       4828        slabinfo.kmalloc-128.active_objs
      3791           +27.4%       4828        slabinfo.kmalloc-128.num_objs
      4761          +202.2%      14387        slabinfo.kmalloc-256.active_objs
    148.20          +205.2%     452.25        slabinfo.kmalloc-256.active_slabs
      4761          +204.1%      14479        slabinfo.kmalloc-256.num_objs
    148.20          +205.2%     452.25        slabinfo.kmalloc-256.num_slabs
      4974 ±  2%     +22.3%       6081        slabinfo.kmalloc-2k.active_objs
      5023 ±  2%     +22.2%       6139        slabinfo.kmalloc-2k.num_objs
      7800 ±  3%     +94.4%      15162        slabinfo.kmalloc-512.active_objs
    243.20 ±  4%     +96.0%     476.75        slabinfo.kmalloc-512.active_slabs
      7804 ±  3%     +95.7%      15273        slabinfo.kmalloc-512.num_objs
    243.20 ±  4%     +96.0%     476.75        slabinfo.kmalloc-512.num_slabs
      5098           +10.9%       5652 ±  3%  slabinfo.kmalloc-96.active_objs
      5213            +9.0%       5684 ±  3%  slabinfo.kmalloc-96.num_objs
      2007 ±  7%      +9.8%       2205 ±  4%  slabinfo.kmalloc-rcl-96.active_objs
      2007 ±  7%      +9.8%       2205 ±  4%  slabinfo.kmalloc-rcl-96.num_objs
      3741            +9.9%       4111        slabinfo.pid.active_objs
      3741            +9.9%       4111        slabinfo.pid.num_objs
     13974           +10.6%      15458        slabinfo.proc_inode_cache.active_objs
     13974           +10.6%      15458        slabinfo.proc_inode_cache.num_objs
      2335           +19.3%       2787        slabinfo.sighand_cache.active_objs
      2335           +19.3%       2787        slabinfo.sighand_cache.num_objs
      3532           +10.0%       3884        slabinfo.signal_cache.active_objs
      3532           +10.0%       3884        slabinfo.signal_cache.num_objs
      1635           +35.6%       2216 ±  2%  slabinfo.task_struct.active_objs
      1639           +35.4%       2220 ±  2%  slabinfo.task_struct.active_slabs
      1639           +35.4%       2220 ±  2%  slabinfo.task_struct.num_objs
      1639           +35.4%       2220 ±  2%  slabinfo.task_struct.num_slabs
      8503           +15.9%       9854        slabinfo.vmap_area.active_objs
      8560           +15.5%       9889        slabinfo.vmap_area.num_objs
    722.40 ±  7%    +293.6%       2843        slabinfo.xfs_buf.active_objs
    722.40 ±  7%    +293.6%       2843        slabinfo.xfs_buf.num_objs
 3.756e+09            +1.0%  3.794e+09        perf-stat.i.branch-instructions
  55679849            +6.7%   59387832        perf-stat.i.branch-misses
     24.50            -0.7       23.77        perf-stat.i.cache-miss-rate%
  70311533            -3.0%   68194124        perf-stat.i.cache-misses
 2.772e+08            +2.4%  2.839e+08        perf-stat.i.cache-references
    886579            +7.6%     954024        perf-stat.i.context-switches
      7.65            -2.7%       7.44        perf-stat.i.cpi
 1.371e+11            -2.5%  1.338e+11        perf-stat.i.cpu-cycles
    174499            -2.2%     170613        perf-stat.i.cpu-migrations
  13855802            -5.3%   13121077        perf-stat.i.iTLB-load-misses
 1.715e+10            +1.1%  1.734e+10        perf-stat.i.instructions
      3164            -4.8%       3013        perf-stat.i.minor-faults
  22061892            -4.4%   21088063        perf-stat.i.node-load-misses
   3566947            -5.1%    3384010        perf-stat.i.node-loads
   8568626            -6.7%    7990367        perf-stat.i.node-store-misses
   3055898            -9.4%    2769195        perf-stat.i.node-stores
      3164            -4.8%       3013        perf-stat.i.page-faults
      1.48            +0.1        1.57        perf-stat.overall.branch-miss-rate%
     25.37            -1.3       24.02        perf-stat.overall.cache-miss-rate%
      7.99            -3.5%       7.71        perf-stat.overall.cpi
     62.51            -1.5       60.97        perf-stat.overall.iTLB-load-miss-rate%
      1237            +6.8%       1321        perf-stat.overall.instructions-per-iTLB-miss
      0.13            +3.6%       0.13        perf-stat.overall.ipc
 3.726e+09            +1.2%   3.77e+09        perf-stat.ps.branch-instructions
  55220993            +6.9%   59009261        perf-stat.ps.branch-misses
  2.75e+08            +2.6%  2.821e+08        perf-stat.ps.cache-references
    879527            +7.8%     948179        perf-stat.ps.context-switches
  1.36e+11            -2.3%  1.329e+11        perf-stat.ps.cpu-cycles
    173111            -2.0%     169568        perf-stat.ps.cpu-migrations
  13745541            -5.1%   13040659        perf-stat.ps.iTLB-load-misses
 1.702e+10            +1.3%  1.723e+10        perf-stat.ps.instructions
      3139            -4.6%       2993        perf-stat.ps.minor-faults
  21886378            -4.2%   20958825        perf-stat.ps.node-load-misses
   3538560            -5.0%    3363274        perf-stat.ps.node-loads
   8500464            -6.6%    7941361        perf-stat.ps.node-store-misses
   3031592            -9.2%    2752190        perf-stat.ps.node-stores
      3139            -4.6%       2993        perf-stat.ps.page-faults
 2.097e+12           +31.8%  2.764e+12        perf-stat.total.instructions
    120.00 ±  3%     +31.7%     158.00 ±  6%  interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
    247.60           +30.0%     322.00        interrupts.9:IO-APIC.9-fasteoi.acpi
    244759 ±  2%     +30.9%     320449        interrupts.CPU0.LOC:Local_timer_interrupts
    139460           +20.6%     168220 ±  2%  interrupts.CPU0.RES:Rescheduling_interrupts
    247.60           +30.0%     322.00        interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    244739 ±  2%     +30.9%     320463        interrupts.CPU1.LOC:Local_timer_interrupts
    138651 ±  2%     +20.5%     167118 ±  2%  interrupts.CPU1.RES:Rescheduling_interrupts
    244755 ±  2%     +30.9%     320463        interrupts.CPU10.LOC:Local_timer_interrupts
    138444 ±  2%     +20.4%     166732 ±  2%  interrupts.CPU10.RES:Rescheduling_interrupts
    244620 ±  2%     +31.0%     320449        interrupts.CPU11.LOC:Local_timer_interrupts
    138712           +19.8%     166173 ±  3%  interrupts.CPU11.RES:Rescheduling_interrupts
    244780 ±  2%     +30.9%     320481        interrupts.CPU12.LOC:Local_timer_interrupts
    138384 ±  2%     +20.4%     166613 ±  3%  interrupts.CPU12.RES:Rescheduling_interrupts
    244790 ±  2%     +30.9%     320423        interrupts.CPU13.LOC:Local_timer_interrupts
    138020 ±  2%     +19.8%     165289 ±  2%  interrupts.CPU13.RES:Rescheduling_interrupts
    244772 ±  2%     +30.9%     320510        interrupts.CPU14.LOC:Local_timer_interrupts
    138772 ±  2%     +19.8%     166232 ±  2%  interrupts.CPU14.RES:Rescheduling_interrupts
    244741 ±  2%     +31.0%     320492        interrupts.CPU15.LOC:Local_timer_interrupts
    138567 ±  2%     +20.5%     167036 ±  2%  interrupts.CPU15.RES:Rescheduling_interrupts
    244776 ±  2%     +30.9%     320473        interrupts.CPU16.LOC:Local_timer_interrupts
    139350           +20.0%     167226 ±  3%  interrupts.CPU16.RES:Rescheduling_interrupts
    244665 ±  2%     +31.0%     320462        interrupts.CPU17.LOC:Local_timer_interrupts
    138413 ±  2%     +19.8%     165868 ±  2%  interrupts.CPU17.RES:Rescheduling_interrupts
    244811 ±  2%     +30.9%     320477        interrupts.CPU18.LOC:Local_timer_interrupts
    143605 ±  3%     +16.2%     166822 ±  3%  interrupts.CPU18.RES:Rescheduling_interrupts
    244710 ±  2%     +31.0%     320484        interrupts.CPU19.LOC:Local_timer_interrupts
    144915 ±  3%     +16.1%     168195 ±  3%  interrupts.CPU19.RES:Rescheduling_interrupts
    244767 ±  2%     +30.9%     320466        interrupts.CPU2.LOC:Local_timer_interrupts
    138747 ±  2%     +20.0%     166484 ±  3%  interrupts.CPU2.RES:Rescheduling_interrupts
    244812 ±  2%     +30.9%     320453        interrupts.CPU20.LOC:Local_timer_interrupts
    144745 ±  3%     +16.3%     168271 ±  2%  interrupts.CPU20.RES:Rescheduling_interrupts
    244771 ±  2%     +30.9%     320455        interrupts.CPU21.LOC:Local_timer_interrupts
    144259 ±  3%     +16.9%     168623 ±  2%  interrupts.CPU21.RES:Rescheduling_interrupts
    244717 ±  2%     +31.0%     320466        interrupts.CPU22.LOC:Local_timer_interrupts
    144190 ±  3%     +16.7%     168260 ±  2%  interrupts.CPU22.RES:Rescheduling_interrupts
    244740 ±  2%     +30.9%     320460        interrupts.CPU23.LOC:Local_timer_interrupts
    144810 ±  3%     +16.6%     168794 ±  2%  interrupts.CPU23.RES:Rescheduling_interrupts
    244801 ±  2%     +30.9%     320478        interrupts.CPU24.LOC:Local_timer_interrupts
    145115 ±  3%     +16.5%     169114 ±  2%  interrupts.CPU24.RES:Rescheduling_interrupts
    244783 ±  2%     +30.9%     320456        interrupts.CPU25.LOC:Local_timer_interrupts
    144745 ±  3%     +17.2%     169628 ±  3%  interrupts.CPU25.RES:Rescheduling_interrupts
    244772 ±  2%     +30.9%     320377        interrupts.CPU26.LOC:Local_timer_interrupts
    143932 ±  3%     +16.9%     168187 ±  2%  interrupts.CPU26.RES:Rescheduling_interrupts
    244713 ±  2%     +30.9%     320439        interrupts.CPU27.LOC:Local_timer_interrupts
    144956 ±  3%     +16.7%     169179 ±  2%  interrupts.CPU27.RES:Rescheduling_interrupts
    244773 ±  2%     +30.9%     320444        interrupts.CPU28.LOC:Local_timer_interrupts
    144558 ±  3%     +16.7%     168727 ±  3%  interrupts.CPU28.RES:Rescheduling_interrupts
    244776 ±  2%     +30.9%     320432        interrupts.CPU29.LOC:Local_timer_interrupts
    144948 ±  3%     +17.1%     169683 ±  2%  interrupts.CPU29.RES:Rescheduling_interrupts
    244671 ±  2%     +31.0%     320441        interrupts.CPU3.LOC:Local_timer_interrupts
    138929 ±  2%     +20.3%     167117 ±  2%  interrupts.CPU3.RES:Rescheduling_interrupts
    244842 ±  2%     +30.9%     320452        interrupts.CPU30.LOC:Local_timer_interrupts
    144551 ±  3%     +16.6%     168546 ±  3%  interrupts.CPU30.RES:Rescheduling_interrupts
    244837 ±  2%     +30.9%     320425        interrupts.CPU31.LOC:Local_timer_interrupts
    144859 ±  3%     +15.8%     167803 ±  2%  interrupts.CPU31.RES:Rescheduling_interrupts
    244848 ±  2%     +30.9%     320444        interrupts.CPU32.LOC:Local_timer_interrupts
    145208 ±  3%     +16.6%     169359 ±  3%  interrupts.CPU32.RES:Rescheduling_interrupts
    244833 ±  2%     +30.9%     320452        interrupts.CPU33.LOC:Local_timer_interrupts
    145195 ±  3%     +16.6%     169281 ±  2%  interrupts.CPU33.RES:Rescheduling_interrupts
    244831 ±  2%     +30.9%     320446        interrupts.CPU34.LOC:Local_timer_interrupts
    146168 ±  3%     +16.2%     169831 ±  2%  interrupts.CPU34.RES:Rescheduling_interrupts
    244854 ±  2%     +30.9%     320444        interrupts.CPU35.LOC:Local_timer_interrupts
    144667 ±  3%     +16.7%     168845 ±  2%  interrupts.CPU35.RES:Rescheduling_interrupts
    244750 ±  2%     +30.9%     320427        interrupts.CPU36.LOC:Local_timer_interrupts
    138547           +20.5%     166904 ±  3%  interrupts.CPU36.RES:Rescheduling_interrupts
    244786 ±  2%     +30.9%     320493        interrupts.CPU37.LOC:Local_timer_interrupts
    138195 ±  2%     +20.1%     166018 ±  3%  interrupts.CPU37.RES:Rescheduling_interrupts
    244712 ±  2%     +31.0%     320501        interrupts.CPU38.LOC:Local_timer_interrupts
    138469           +20.2%     166476 ±  3%  interrupts.CPU38.RES:Rescheduling_interrupts
    244754 ±  2%     +30.9%     320482        interrupts.CPU39.LOC:Local_timer_interrupts
    137673 ±  2%     +20.3%     165625 ±  3%  interrupts.CPU39.RES:Rescheduling_interrupts
    244786 ±  2%     +30.9%     320446        interrupts.CPU4.LOC:Local_timer_interrupts
    138036 ±  2%     +20.7%     166604 ±  4%  interrupts.CPU4.RES:Rescheduling_interrupts
    244778 ±  2%     +30.9%     320475        interrupts.CPU40.LOC:Local_timer_interrupts
    137938 ±  2%     +20.5%     166159 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
    244763 ±  2%     +30.9%     320446        interrupts.CPU41.LOC:Local_timer_interrupts
    137895 ±  2%     +20.5%     166118 ±  3%  interrupts.CPU41.RES:Rescheduling_interrupts
    244718 ±  2%     +30.9%     320433        interrupts.CPU42.LOC:Local_timer_interrupts
    138009 ±  2%     +20.0%     165675 ±  2%  interrupts.CPU42.RES:Rescheduling_interrupts
    244746 ±  2%     +30.9%     320345        interrupts.CPU43.LOC:Local_timer_interrupts
    138578 ±  2%     +20.1%     166482 ±  3%  interrupts.CPU43.RES:Rescheduling_interrupts
    244759 ±  2%     +30.9%     320497        interrupts.CPU44.LOC:Local_timer_interrupts
    137166 ±  2%     +20.9%     165776 ±  3%  interrupts.CPU44.RES:Rescheduling_interrupts
    244747 ±  2%     +30.9%     320460        interrupts.CPU45.LOC:Local_timer_interrupts
    138112 ±  2%     +20.3%     166101 ±  3%  interrupts.CPU45.RES:Rescheduling_interrupts
    244756 ±  2%     +30.9%     320471        interrupts.CPU46.LOC:Local_timer_interrupts
    137526 ±  2%     +20.4%     165589 ±  3%  interrupts.CPU46.RES:Rescheduling_interrupts
    244784 ±  2%     +30.9%     320319        interrupts.CPU47.LOC:Local_timer_interrupts
    137762 ±  2%     +20.3%     165719 ±  3%  interrupts.CPU47.RES:Rescheduling_interrupts
    244692 ±  2%     +31.0%     320475        interrupts.CPU48.LOC:Local_timer_interrupts
    137330 ±  2%     +21.0%     166131 ±  3%  interrupts.CPU48.RES:Rescheduling_interrupts
    244785 ±  2%     +30.9%     320506        interrupts.CPU49.LOC:Local_timer_interrupts
    137448 ±  2%     +20.6%     165801 ±  3%  interrupts.CPU49.RES:Rescheduling_interrupts
    244761 ±  2%     +30.9%     320468        interrupts.CPU5.LOC:Local_timer_interrupts
    137844 ±  2%     +20.5%     166127 ±  3%  interrupts.CPU5.RES:Rescheduling_interrupts
    244779 ±  2%     +30.9%     320444        interrupts.CPU50.LOC:Local_timer_interrupts
    137351 ±  2%     +20.5%     165464 ±  3%  interrupts.CPU50.RES:Rescheduling_interrupts
    244736 ±  2%     +30.9%     320463        interrupts.CPU51.LOC:Local_timer_interrupts
    138344 ±  2%     +20.3%     166444 ±  3%  interrupts.CPU51.RES:Rescheduling_interrupts
    244758 ±  2%     +30.9%     320459        interrupts.CPU52.LOC:Local_timer_interrupts
    137864 ±  2%     +20.0%     165427 ±  3%  interrupts.CPU52.RES:Rescheduling_interrupts
    244750 ±  2%     +30.9%     320389        interrupts.CPU53.LOC:Local_timer_interrupts
    137731           +20.3%     165643 ±  3%  interrupts.CPU53.RES:Rescheduling_interrupts
     65.60 ± 33%     +47.9%      97.00 ± 27%  interrupts.CPU53.TLB:TLB_shootdowns
    244852 ±  2%     +30.9%     320472        interrupts.CPU54.LOC:Local_timer_interrupts
    143350 ±  3%     +16.7%     167283 ±  3%  interrupts.CPU54.RES:Rescheduling_interrupts
    244807 ±  2%     +30.9%     320488        interrupts.CPU55.LOC:Local_timer_interrupts
    144591 ±  3%     +16.7%     168681 ±  3%  interrupts.CPU55.RES:Rescheduling_interrupts
    244797 ±  2%     +30.9%     320440        interrupts.CPU56.LOC:Local_timer_interrupts
    144588 ±  3%     +16.7%     168739 ±  2%  interrupts.CPU56.RES:Rescheduling_interrupts
    244744 ±  2%     +30.9%     320434        interrupts.CPU57.LOC:Local_timer_interrupts
    144408 ±  3%     +16.9%     168832 ±  3%  interrupts.CPU57.RES:Rescheduling_interrupts
    244647 ±  2%     +31.0%     320435        interrupts.CPU58.LOC:Local_timer_interrupts
    144028 ±  3%     +16.5%     167755 ±  3%  interrupts.CPU58.RES:Rescheduling_interrupts
    244836 ±  2%     +30.9%     320452        interrupts.CPU59.LOC:Local_timer_interrupts
    144524 ±  3%     +17.1%     169301 ±  3%  interrupts.CPU59.RES:Rescheduling_interrupts
    244775 ±  2%     +30.9%     320401        interrupts.CPU6.LOC:Local_timer_interrupts
    138013 ±  2%     +20.9%     166802 ±  2%  interrupts.CPU6.RES:Rescheduling_interrupts
    244801 ±  2%     +30.9%     320464        interrupts.CPU60.LOC:Local_timer_interrupts
    144621 ±  3%     +16.9%     169021 ±  3%  interrupts.CPU60.RES:Rescheduling_interrupts
    244644 ±  2%     +31.0%     320444        interrupts.CPU61.LOC:Local_timer_interrupts
    144679 ±  3%     +16.3%     168205 ±  3%  interrupts.CPU61.RES:Rescheduling_interrupts
    244823 ±  2%     +30.9%     320443        interrupts.CPU62.LOC:Local_timer_interrupts
    144008 ±  3%     +16.5%     167829 ±  3%  interrupts.CPU62.RES:Rescheduling_interrupts
    244760 ±  2%     +30.9%     320442        interrupts.CPU63.LOC:Local_timer_interrupts
    144920 ±  3%     +16.3%     168549 ±  2%  interrupts.CPU63.RES:Rescheduling_interrupts
    244704 ±  2%     +30.9%     320340        interrupts.CPU64.LOC:Local_timer_interrupts
    144660 ±  3%     +16.3%     168224 ±  2%  interrupts.CPU64.RES:Rescheduling_interrupts
    244800 ±  2%     +30.9%     320496        interrupts.CPU65.LOC:Local_timer_interrupts
    145098 ±  3%     +16.2%     168585 ±  3%  interrupts.CPU65.RES:Rescheduling_interrupts
    244785 ±  2%     +30.9%     320492        interrupts.CPU66.LOC:Local_timer_interrupts
    144118 ±  3%     +16.5%     167850 ±  3%  interrupts.CPU66.RES:Rescheduling_interrupts
    244762 ±  2%     +30.8%     320108        interrupts.CPU67.LOC:Local_timer_interrupts
    143561 ±  3%     +16.8%     167615 ±  2%  interrupts.CPU67.RES:Rescheduling_interrupts
    244835 ±  2%     +30.9%     320439        interrupts.CPU68.LOC:Local_timer_interrupts
    144364 ±  3%     +16.5%     168118 ±  2%  interrupts.CPU68.RES:Rescheduling_interrupts
    244796 ±  2%     +30.9%     320433        interrupts.CPU69.LOC:Local_timer_interrupts
    144799 ±  3%     +16.6%     168805 ±  3%  interrupts.CPU69.RES:Rescheduling_interrupts
    120.00 ±  3%     +31.7%     158.00 ±  6%  interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
    244755 ±  2%     +30.8%     320143        interrupts.CPU7.LOC:Local_timer_interrupts
    138484 ±  2%     +20.3%     166547 ±  3%  interrupts.CPU7.RES:Rescheduling_interrupts
    244801 ±  2%     +30.9%     320430        interrupts.CPU70.LOC:Local_timer_interrupts
    145070 ±  3%     +16.1%     168429 ±  2%  interrupts.CPU70.RES:Rescheduling_interrupts
    244920 ±  2%     +30.9%     320579        interrupts.CPU71.LOC:Local_timer_interrupts
    142869 ±  3%     +17.0%     167093 ±  3%  interrupts.CPU71.RES:Rescheduling_interrupts
    244737 ±  2%     +30.9%     320475        interrupts.CPU8.LOC:Local_timer_interrupts
    138269 ±  2%     +19.7%     165482 ±  2%  interrupts.CPU8.RES:Rescheduling_interrupts
    244795 ±  2%     +30.9%     320351        interrupts.CPU9.LOC:Local_timer_interrupts
    138266 ±  2%     +20.8%     167001 ±  2%  interrupts.CPU9.RES:Rescheduling_interrupts
  17623367 ±  2%     +30.9%   23071880        interrupts.LOC:Local_timer_interrupts
  10179012           +18.4%   12052313        interrupts.RES:Rescheduling_interrupts
     59774           +13.3%      67746        softirqs.CPU0.RCU
     37946 ±  2%     +20.1%      45590        softirqs.CPU0.SCHED
     47829           +26.4%      60436        softirqs.CPU0.TIMER
     60507           +14.4%      69244        softirqs.CPU1.RCU
     33668           +22.8%      41354        softirqs.CPU1.SCHED
     47253 ±  2%     +26.9%      59977 ±  2%  softirqs.CPU1.TIMER
     60020 ±  2%     +13.4%      68036        softirqs.CPU10.RCU
     33056           +21.8%      40261        softirqs.CPU10.SCHED
     47029           +25.3%      58944 ±  2%  softirqs.CPU10.TIMER
     59641           +14.2%      68092        softirqs.CPU11.RCU
     33134           +20.9%      40073        softirqs.CPU11.SCHED
     46830           +27.3%      59618 ±  3%  softirqs.CPU11.TIMER
     59706           +14.6%      68429        softirqs.CPU12.RCU
     32888           +22.3%      40211        softirqs.CPU12.SCHED
     47136           +26.6%      59669 ±  2%  softirqs.CPU12.TIMER
     60162 ±  2%     +13.0%      67969        softirqs.CPU13.RCU
     33134           +21.3%      40198        softirqs.CPU13.SCHED
     47941 ±  4%     +23.2%      59077 ±  2%  softirqs.CPU13.TIMER
     60344           +14.3%      68973        softirqs.CPU14.RCU
     33110           +21.3%      40172        softirqs.CPU14.SCHED
     47049           +26.2%      59387 ±  3%  softirqs.CPU14.TIMER
     60273           +14.1%      68795        softirqs.CPU15.RCU
     33054           +22.1%      40373        softirqs.CPU15.SCHED
     47069           +26.1%      59346 ±  2%  softirqs.CPU15.TIMER
     60166 ±  2%     +13.6%      68369        softirqs.CPU16.RCU
     33046           +21.8%      40235        softirqs.CPU16.SCHED
     46990 ±  2%     +26.1%      59247        softirqs.CPU16.TIMER
     60154           +14.0%      68556        softirqs.CPU17.RCU
     32933           +22.5%      40336        softirqs.CPU17.SCHED
     46978 ±  2%     +42.0%      66694 ± 17%  softirqs.CPU17.TIMER
     59894           +13.9%      68231        softirqs.CPU18.RCU
     33426           +20.5%      40273 ±  2%  softirqs.CPU18.SCHED
     47204           +25.9%      59417        softirqs.CPU18.TIMER
     60195           +14.0%      68613        softirqs.CPU19.RCU
     33414           +20.8%      40367        softirqs.CPU19.SCHED
     47186           +25.1%      59028        softirqs.CPU19.TIMER
     60071           +13.1%      67938        softirqs.CPU2.RCU
     33237           +21.1%      40243        softirqs.CPU2.SCHED
     47090           +25.4%      59032        softirqs.CPU2.TIMER
     60389           +18.0%      71267 ±  4%  softirqs.CPU20.RCU
     33648 ±  3%     +19.8%      40312        softirqs.CPU20.SCHED
     47027 ±  4%     +29.6%      60947 ±  2%  softirqs.CPU20.TIMER
     62957 ±  5%      +8.8%      68500        softirqs.CPU21.RCU
     33429 ±  2%     +20.1%      40145        softirqs.CPU21.SCHED
     60598 ±  2%     +12.1%      67959        softirqs.CPU22.RCU
     33648 ±  2%     +19.4%      40167        softirqs.CPU22.SCHED
     47744 ±  2%     +23.7%      59059        softirqs.CPU22.TIMER
     60854           +13.2%      68903 ±  3%  softirqs.CPU23.RCU
     33387           +20.3%      40182        softirqs.CPU23.SCHED
     47566           +24.3%      59117 ±  3%  softirqs.CPU23.TIMER
     60080           +13.2%      68021        softirqs.CPU24.RCU
     33546           +19.8%      40187        softirqs.CPU24.SCHED
     59920           +14.3%      68497        softirqs.CPU25.RCU
     33140           +21.0%      40114        softirqs.CPU25.SCHED
     47284           +24.4%      58806        softirqs.CPU25.TIMER
     59773           +14.0%      68172        softirqs.CPU26.RCU
     33317           +20.4%      40114 ±  2%  softirqs.CPU26.SCHED
     47263           +25.5%      59294        softirqs.CPU26.TIMER
     59631           +14.9%      68520        softirqs.CPU27.RCU
     33251           +21.4%      40354        softirqs.CPU27.SCHED
     46817           +26.2%      59088        softirqs.CPU27.TIMER
     60002           +13.7%      68208        softirqs.CPU28.RCU
     33349           +20.9%      40315        softirqs.CPU28.SCHED
     47160           +24.7%      58787        softirqs.CPU28.TIMER
     60241 ±  2%     +12.4%      67696        softirqs.CPU29.RCU
     33361 ±  2%     +20.2%      40099        softirqs.CPU29.SCHED
     47064           +24.1%      58401        softirqs.CPU29.TIMER
     60740 ±  2%     +12.8%      68515        softirqs.CPU3.RCU
     33723 ±  2%     +21.1%      40824        softirqs.CPU3.SCHED
     47511 ±  2%     +24.8%      59304        softirqs.CPU3.TIMER
     60151           +13.1%      68017        softirqs.CPU30.RCU
     33435           +20.7%      40345 ±  2%  softirqs.CPU30.SCHED
     47230           +25.0%      59039        softirqs.CPU30.TIMER
     60136           +12.8%      67833        softirqs.CPU31.RCU
     33502           +19.7%      40114        softirqs.CPU31.SCHED
     47149           +24.9%      58892        softirqs.CPU31.TIMER
     60130 ±  2%     +12.9%      67879        softirqs.CPU32.RCU
     33410           +20.3%      40181        softirqs.CPU32.SCHED
     46794           +25.3%      58620        softirqs.CPU32.TIMER
     60192           +13.5%      68319        softirqs.CPU33.RCU
     33434           +20.3%      40212        softirqs.CPU33.SCHED
     46824           +40.2%      65657 ± 18%  softirqs.CPU33.TIMER
     61827 ±  4%     +10.7%      68415        softirqs.CPU34.RCU
     33412           +20.6%      40288        softirqs.CPU34.SCHED
     48323 ±  3%     +22.3%      59092        softirqs.CPU34.TIMER
     60804 ±  2%     +12.9%      68671        softirqs.CPU35.RCU
     33094           +21.6%      40252        softirqs.CPU35.SCHED
     47545           +23.9%      58905        softirqs.CPU35.TIMER
     60159           +13.4%      68199        softirqs.CPU36.RCU
     32835           +21.1%      39775        softirqs.CPU36.SCHED
     46194 ±  2%     +26.4%      58376        softirqs.CPU36.TIMER
     60311           +12.9%      68091        softirqs.CPU37.RCU
     33217           +20.3%      39949        softirqs.CPU37.SCHED
     46831           +25.0%      58531 ±  2%  softirqs.CPU37.TIMER
     60301           +12.6%      67892        softirqs.CPU38.RCU
     33089           +20.9%      40008        softirqs.CPU38.SCHED
     47090 ±  3%     +23.6%      58218        softirqs.CPU38.TIMER
     60518           +12.0%      67803        softirqs.CPU39.RCU
     33114           +21.0%      40074        softirqs.CPU39.SCHED
     46862 ±  2%     +24.0%      58119        softirqs.CPU39.TIMER
     60909 ±  2%     +15.7%      70450 ±  3%  softirqs.CPU4.RCU
     33335           +21.6%      40539        softirqs.CPU4.SCHED
     62092 ±  4%      +9.3%      67853        softirqs.CPU40.RCU
     32920           +21.5%      39990        softirqs.CPU40.SCHED
     47896 ±  5%     +21.5%      58188        softirqs.CPU40.TIMER
     60103           +12.9%      67833        softirqs.CPU41.RCU
     32839           +22.1%      40100        softirqs.CPU41.SCHED
     45812 ±  4%     +26.6%      58005        softirqs.CPU41.TIMER
     60066           +13.3%      68070        softirqs.CPU42.RCU
     32941           +21.5%      40013        softirqs.CPU42.SCHED
     46516           +41.8%      65965 ± 19%  softirqs.CPU42.TIMER
     59845           +13.3%      67812        softirqs.CPU43.RCU
     32968           +21.5%      40052        softirqs.CPU43.SCHED
     46433           +25.2%      58141        softirqs.CPU43.TIMER
     59708           +14.0%      68086        softirqs.CPU44.RCU
     33024           +21.2%      40037        softirqs.CPU44.SCHED
     46623           +25.4%      58478        softirqs.CPU44.TIMER
     59787           +14.0%      68162        softirqs.CPU45.RCU
     32922           +21.5%      39994        softirqs.CPU45.SCHED
     46034 ±  2%     +26.2%      58107        softirqs.CPU45.TIMER
     60165           +13.0%      67985        softirqs.CPU46.RCU
     32947           +21.6%      40063        softirqs.CPU46.SCHED
     46694           +24.7%      58235        softirqs.CPU46.TIMER
     59563           +13.4%      67574        softirqs.CPU47.RCU
     32990           +21.3%      40011        softirqs.CPU47.SCHED
     46246           +26.3%      58426 ±  2%  softirqs.CPU47.TIMER
     59918           +12.7%      67537        softirqs.CPU48.RCU
     33024           +21.0%      39948        softirqs.CPU48.SCHED
     46677 ±  2%     +24.8%      58234        softirqs.CPU48.TIMER
     59860           +12.9%      67593        softirqs.CPU49.RCU
     32933           +21.4%      39985        softirqs.CPU49.SCHED
     46560           +25.2%      58312        softirqs.CPU49.TIMER
     61231 ±  4%     +11.4%      68197        softirqs.CPU5.RCU
     33019           +22.6%      40491        softirqs.CPU5.SCHED
     47732 ±  2%     +23.9%      59155        softirqs.CPU5.TIMER
     59803 ±  2%     +14.5%      68486        softirqs.CPU50.RCU
     33007           +21.5%      40120        softirqs.CPU50.SCHED
     59720           +13.4%      67711        softirqs.CPU51.RCU
     33094           +21.3%      40129        softirqs.CPU51.SCHED
     46602 ±  2%     +24.5%      58019        softirqs.CPU51.TIMER
     59924           +13.1%      67766        softirqs.CPU52.RCU
     33055           +21.4%      40124        softirqs.CPU52.SCHED
     46612 ±  2%     +26.0%      58714        softirqs.CPU52.TIMER
     59895 ±  2%     +14.2%      68381        softirqs.CPU53.RCU
     32742           +21.7%      39837        softirqs.CPU53.SCHED
     46557 ±  2%     +26.3%      58808 ±  3%  softirqs.CPU53.TIMER
     60227           +12.7%      67866        softirqs.CPU54.RCU
     33032 ±  2%     +21.0%      39959 ±  2%  softirqs.CPU54.SCHED
     46832           +24.5%      58309        softirqs.CPU54.TIMER
     60516 ±  2%     +12.8%      68248        softirqs.CPU55.RCU
     33217           +20.3%      39944        softirqs.CPU55.SCHED
     47131           +23.3%      58136        softirqs.CPU55.TIMER
     60157           +15.6%      69566 ±  4%  softirqs.CPU56.RCU
     33032 ±  2%     +20.5%      39815        softirqs.CPU56.SCHED
     45942 ±  3%     +27.3%      58483 ±  2%  softirqs.CPU56.TIMER
     60827           +12.5%      68422        softirqs.CPU57.RCU
     33268           +22.4%      40712 ±  2%  softirqs.CPU57.SCHED
     47463           +39.7%      66310 ± 18%  softirqs.CPU57.TIMER
     59628           +15.0%      68595 ±  2%  softirqs.CPU58.RCU
     33176 ±  2%     +21.2%      40225 ±  2%  softirqs.CPU58.SCHED
     46488           +27.7%      59385 ±  4%  softirqs.CPU58.TIMER
     59916           +14.5%      68600        softirqs.CPU59.RCU
     33071 ±  2%     +21.7%      40244        softirqs.CPU59.SCHED
     46396           +26.8%      58828 ±  3%  softirqs.CPU59.TIMER
     59746           +14.4%      68373        softirqs.CPU6.RCU
     32897           +21.8%      40067        softirqs.CPU6.SCHED
     46868           +26.6%      59319 ±  2%  softirqs.CPU6.TIMER
     60431 ±  3%     +13.2%      68410        softirqs.CPU60.RCU
     33313           +20.8%      40246        softirqs.CPU60.SCHED
     46976 ±  3%     +25.1%      58788        softirqs.CPU60.TIMER
     60090           +13.5%      68213        softirqs.CPU61.RCU
     33294           +20.4%      40095 ±  2%  softirqs.CPU61.SCHED
     46727           +25.2%      58494 ±  2%  softirqs.CPU61.TIMER
     60141 ±  2%     +12.8%      67826        softirqs.CPU62.RCU
     33031           +21.5%      40143        softirqs.CPU62.SCHED
     46899 ±  2%     +24.7%      58497        softirqs.CPU62.TIMER
     59796           +13.8%      68028 ±  2%  softirqs.CPU63.RCU
     33250 ±  2%     +20.8%      40178 ±  2%  softirqs.CPU63.SCHED
     46526           +25.1%      58192 ±  2%  softirqs.CPU63.TIMER
     59896           +14.6%      68646        softirqs.CPU64.RCU
     33135 ±  2%     +21.5%      40265 ±  2%  softirqs.CPU64.SCHED
     46764           +25.3%      58595        softirqs.CPU64.TIMER
     59708           +13.7%      67902        softirqs.CPU65.RCU
     33072 ±  2%     +21.4%      40142        softirqs.CPU65.SCHED
     46461           +25.0%      58060        softirqs.CPU65.TIMER
     59408           +14.5%      68007        softirqs.CPU66.RCU
     33142           +20.9%      40082 ±  2%  softirqs.CPU66.SCHED
     46352           +25.9%      58363        softirqs.CPU66.TIMER
     59363           +15.2%      68371 ±  2%  softirqs.CPU67.RCU
     33170           +20.9%      40088        softirqs.CPU67.SCHED
     46355           +26.7%      58752 ±  2%  softirqs.CPU67.TIMER
     59490           +14.3%      67991 ±  2%  softirqs.CPU68.RCU
     33158 ±  2%     +21.1%      40166 ±  2%  softirqs.CPU68.SCHED
     46309           +25.8%      58255 ±  2%  softirqs.CPU68.TIMER
     59390           +14.7%      68100        softirqs.CPU69.RCU
     33190           +21.3%      40273 ±  2%  softirqs.CPU69.SCHED
     46202           +25.7%      58065        softirqs.CPU69.TIMER
     60327           +13.1%      68238        softirqs.CPU7.RCU
     33413           +20.2%      40162        softirqs.CPU7.SCHED
     47411 ±  2%     +24.4%      58970        softirqs.CPU7.TIMER
     59647           +14.7%      68413 ±  2%  softirqs.CPU70.RCU
     33080           +22.0%      40356 ±  2%  softirqs.CPU70.SCHED
     47074 ±  4%     +24.4%      58568 ±  2%  softirqs.CPU70.TIMER
     59898           +13.0%      67675        softirqs.CPU71.RCU
     32995 ±  2%     +21.3%      40007 ±  2%  softirqs.CPU71.SCHED
     46638           +23.8%      57743        softirqs.CPU71.TIMER
     60423           +12.0%      67692        softirqs.CPU8.RCU
     33033           +21.9%      40266        softirqs.CPU8.SCHED
     47659 ±  2%     +23.7%      58959        softirqs.CPU8.TIMER
     60535           +12.8%      68295        softirqs.CPU9.RCU
     33106           +22.5%      40548        softirqs.CPU9.SCHED
     47479 ±  5%     +24.0%      58858 ±  2%  softirqs.CPU9.TIMER
   4332816           +13.4%    4915384        softirqs.RCU
   2393271           +21.1%    2899153        softirqs.SCHED
   3405386           +25.3%    4266734        softirqs.TIMER
     69.41            -3.4       66.00        perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     75.66            -2.8       72.84        perf-profile.calltrace.cycles-pp.write
     75.19            -2.8       72.44        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     75.18            -2.8       72.43        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      5.54 ±  6%      -2.7        2.80 ±  2%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
      5.52 ±  6%      -2.7        2.77 ±  2%  perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
      5.52 ±  6%      -2.7        2.78 ±  2%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
     74.87            -2.7       72.18        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     74.84            -2.7       72.16        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     74.61            -2.6       71.98        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.62 ±  6%      -2.6        3.00 ±  2%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     74.51            -2.6       71.89        perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      3.07 ±  6%      -1.5        1.61 ±  2%  perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      3.05 ±  5%      -1.5        1.60 ±  2%  perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
      3.04 ±  6%      -1.4        1.59 ±  2%  perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
      1.84 ±  9%      -1.4        0.41 ± 58%  perf-profile.calltrace.cycles-pp.sb_clear_inode_writeback.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request
      2.95 ±  6%      -1.4        1.52 ±  2%  perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages
      2.05 ±  8%      -1.4        0.64 ±  7%  perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
      1.99 ±  9%      -1.4        0.60 ±  7%  perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio
      1.98 ±  9%      -1.4        0.59 ±  7%  perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request
      1.66 ± 10%      -1.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.sb_mark_inode_writeback.__test_set_page_writeback.iomap_do_writepage
      2.42 ±  8%      -1.3        1.14 ±  3%  perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      2.42 ±  8%      -1.3        1.14 ±  2%  perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
      2.22 ±  8%      -1.2        0.97 ±  3%  perf-profile.calltrace.cycles-pp.iomap_do_writepage.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
      1.93 ± 10%      -1.2        0.69 ±  6%  perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_do_writepage.write_cache_pages.iomap_writepages.xfs_vm_writepages
      1.78 ± 10%      -1.2        0.59 ±  8%  perf-profile.calltrace.cycles-pp.sb_mark_inode_writeback.__test_set_page_writeback.iomap_do_writepage.write_cache_pages.iomap_writepages
      1.74 ± 10%      -1.2        0.56 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.sb_mark_inode_writeback.__test_set_page_writeback.iomap_do_writepage.write_cache_pages
      4.30            -0.7        3.60        perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      2.70            -0.4        2.25        perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      2.60            -0.4        2.17        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
      0.66 ±  7%      -0.4        0.25 ±100%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
      1.72            -0.3        1.45        perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
      1.71            -0.3        1.44        perf-profile.calltrace.cycles-pp.__wake_up_common.complete.process_one_work.worker_thread.kthread
      1.65            -0.3        1.40        perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.complete.process_one_work.worker_thread
      1.00 ±  2%      -0.2        0.81        perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      0.77 ±  6%      -0.2        0.59 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
      1.02 ±  2%      -0.2        0.85 ±  2%  perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      0.84            -0.2        0.68        perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
      0.90 ±  2%      -0.2        0.73 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
      0.87 ±  2%      -0.2        0.70 ±  2%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work
      0.88 ±  2%      -0.2        0.72 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
      1.26            -0.2        1.10        perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      1.25            -0.2        1.10        perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      0.91            -0.1        0.78 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.__wake_up_common
      0.91            -0.1        0.78 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.complete.process_one_work
      0.91            -0.1        0.78 ±  2%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.__wake_up_common.complete
      0.82 ±  2%      -0.1        0.70 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.49            +0.0        1.53        perf-profile.calltrace.cycles-pp.xlog_cil_push.process_one_work.worker_thread.kthread.ret_from_fork
      0.60 ±  3%      +0.1        0.68 ±  2%  perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     15.13            +0.4       15.51        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
     15.10            +0.4       15.48        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn
     15.15            +0.4       15.54        perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      2.84 ±  2%      +0.4        3.24 ±  3%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
      2.92 ±  2%      +0.4        3.33 ±  3%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
     15.59            +0.5       16.05        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     16.06            +0.5       16.55        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     16.08            +0.5       16.57        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     18.27            +0.5       18.78        perf-profile.calltrace.cycles-pp.secondary_startup_64
      0.00            +0.5        0.52        perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
     18.01            +0.5       18.56        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     18.01            +0.6       18.57        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     18.00            +0.6       18.55        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +0.6        0.57        perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
      0.00            +0.6        0.57        perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
      0.00            +0.6        0.58 ±  4%  perf-profile.calltrace.cycles-pp.xfs_inode_item_format.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
      0.56 ±  3%      +0.6        1.15 ± 19%  perf-profile.calltrace.cycles-pp.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
      0.00            +0.9        0.92 ± 25%  perf-profile.calltrace.cycles-pp.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time
      3.75 ±  2%      +1.0        4.71 ±  6%  perf-profile.calltrace.cycles-pp.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      0.00            +1.0        0.96 ± 17%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
      3.65 ±  2%      +1.0        4.62 ±  6%  perf-profile.calltrace.cycles-pp.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      3.60 ±  2%      +1.0        4.58 ±  6%  perf-profile.calltrace.cycles-pp.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write
      0.00            +1.0        0.99 ± 17%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
      0.00            +1.0        1.00 ± 22%  perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
      8.37 ±  2%      +1.3        9.65        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
      8.41 ±  2%      +1.3        9.71        perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      8.40 ±  2%      +1.3        9.69        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      0.00            +2.0        2.04 ± 17%  perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
      4.92            +2.3        7.18 ±  4%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      5.24            +2.3        7.55 ±  4%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      5.25            +2.3        7.56 ±  4%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      5.25            +2.3        7.56 ±  4%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +2.6        2.61 ± 13%  perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
      0.00            +2.6        2.62 ± 13%  perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
     69.41            -3.4       66.00        perf-profile.children.cycles-pp.xfs_file_fsync
     75.72            -2.8       72.90        perf-profile.children.cycles-pp.write
     75.68            -2.8       72.91        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     75.66            -2.8       72.89        perf-profile.children.cycles-pp.do_syscall_64
      5.54 ±  6%      -2.7        2.80 ±  2%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      5.52 ±  6%      -2.7        2.78 ±  2%  perf-profile.children.cycles-pp.do_writepages
      5.52 ±  6%      -2.7        2.77 ±  2%  perf-profile.children.cycles-pp.xfs_vm_writepages
     74.88            -2.7       72.19        perf-profile.children.cycles-pp.ksys_write
     74.85            -2.7       72.16        perf-profile.children.cycles-pp.vfs_write
     74.62            -2.6       71.99        perf-profile.children.cycles-pp.new_sync_write
      5.62 ±  6%      -2.6        3.00 ±  2%  perf-profile.children.cycles-pp.file_write_and_wait_range
     74.51            -2.6       71.89        perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
      3.07 ±  6%      -1.5        1.61 ±  2%  perf-profile.children.cycles-pp.iomap_submit_ioend
      3.16 ±  5%      -1.4        1.74 ±  2%  perf-profile.children.cycles-pp.submit_bio
      3.14 ±  5%      -1.4        1.73 ±  2%  perf-profile.children.cycles-pp.generic_make_request
      3.01 ±  6%      -1.4        1.61 ±  2%  perf-profile.children.cycles-pp.brd_make_request
      2.42 ±  8%      -1.3        1.14 ±  3%  perf-profile.children.cycles-pp.iomap_writepages
      2.42 ±  7%      -1.3        1.14 ±  3%  perf-profile.children.cycles-pp.write_cache_pages
      2.22 ±  8%      -1.2        0.98 ±  3%  perf-profile.children.cycles-pp.iomap_do_writepage
      1.93 ± 10%      -1.2        0.70 ±  6%  perf-profile.children.cycles-pp.__test_set_page_writeback
      2.00 ±  9%      -1.2        0.78 ±  6%  perf-profile.children.cycles-pp.end_page_writeback
      2.00 ±  9%      -1.2        0.77 ±  6%  perf-profile.children.cycles-pp.test_clear_page_writeback
      1.78 ± 10%      -1.2        0.59 ±  8%  perf-profile.children.cycles-pp.sb_mark_inode_writeback
      1.85 ±  9%      -1.2        0.68 ±  6%  perf-profile.children.cycles-pp.sb_clear_inode_writeback
      2.09 ±  8%      -0.9        1.21 ±  4%  perf-profile.children.cycles-pp.iomap_finish_ioend
     66.39            -0.8       65.55        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      4.30            -0.7        3.61        perf-profile.children.cycles-pp.__flush_work
      3.44            -0.6        2.86        perf-profile.children.cycles-pp._raw_spin_lock_irq
     28.08            -0.5       27.55        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.74            -0.3        1.46        perf-profile.children.cycles-pp.complete
      1.01 ±  2%      -0.2        0.81        perf-profile.children.cycles-pp.iomap_write_actor
      1.03 ±  2%      -0.2        0.85 ±  2%  perf-profile.children.cycles-pp.wait_for_completion
      0.90 ±  2%      -0.2        0.73 ±  2%  perf-profile.children.cycles-pp.schedule_timeout
      1.26            -0.2        1.10        perf-profile.children.cycles-pp.iomap_file_buffered_write
      1.25            -0.1        1.10        perf-profile.children.cycles-pp.iomap_apply
      0.89            -0.1        0.75        perf-profile.children.cycles-pp.brd_do_bvec
      0.79 ±  6%      -0.1        0.67 ±  3%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      0.68 ±  7%      -0.1        0.57 ±  4%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.61 ±  6%      -0.1        0.50 ±  4%  perf-profile.children.cycles-pp.osq_lock
      0.39 ±  2%      -0.1        0.31 ±  5%  perf-profile.children.cycles-pp.lseek64
      0.29 ±  5%      -0.1        0.22 ±  3%  perf-profile.children.cycles-pp.xlog_state_get_iclog_space
      0.39            -0.1        0.32        perf-profile.children.cycles-pp.iomap_write_begin
      0.31            -0.1        0.24 ±  3%  perf-profile.children.cycles-pp.iomap_write_end
      0.29            -0.1        0.24        perf-profile.children.cycles-pp.grab_cache_page_write_begin
      0.28            -0.1        0.23        perf-profile.children.cycles-pp.pagecache_get_page
      0.19 ±  4%      -0.0        0.15 ±  4%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      0.58 ±  4%      -0.0        0.54        perf-profile.children.cycles-pp.scheduler_ipi
      0.63 ±  3%      -0.0        0.58        perf-profile.children.cycles-pp.reschedule_interrupt
      0.19 ±  5%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.20            -0.0        0.16 ±  2%  perf-profile.children.cycles-pp.find_get_entry
      0.17 ±  6%      -0.0        0.13 ±  3%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.17 ±  2%      -0.0        0.13 ±  3%  perf-profile.children.cycles-pp.copyin
      0.19 ±  2%      -0.0        0.16 ±  2%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      0.16 ±  2%      -0.0        0.13 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.15 ±  5%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      0.07 ±  5%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.account_page_dirtied
      0.21 ±  2%      -0.0        0.18 ±  2%  perf-profile.children.cycles-pp.blkdev_issue_flush
      0.14 ±  7%      -0.0        0.11 ±  3%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.10 ±  8%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.rcu_core
      0.14 ±  2%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.ksys_lseek
      0.11 ±  7%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_iunlock
      0.10 ±  3%      -0.0        0.08        perf-profile.children.cycles-pp.__set_page_dirty
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.mempool_alloc
      0.12 ±  3%      -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.xas_load
      0.07 ±  9%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.10 ±  5%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.switch_fpu_return
      0.09 ±  5%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.xfs_inode_item_committed
      0.11            -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.insert_work
      0.07 ±  6%      -0.0        0.06        perf-profile.children.cycles-pp.__fdget_pos
      0.07 ±  5%      -0.0        0.06        perf-profile.children.cycles-pp.submit_bio_wait
      0.07            -0.0        0.06        perf-profile.children.cycles-pp.xfs_file_llseek
      0.18 ±  2%      +0.0        0.20 ±  3%  perf-profile.children.cycles-pp.poll_idle
      0.19 ±  5%      +0.0        0.21 ±  3%  perf-profile.children.cycles-pp.kernel_text_address
      0.50            +0.0        0.53 ±  2%  perf-profile.children.cycles-pp.schedule_idle
      0.20 ±  4%      +0.0        0.23 ±  3%  perf-profile.children.cycles-pp.__kernel_text_address
      0.06 ±  7%      +0.0        0.09 ±  9%  perf-profile.children.cycles-pp.xfs_trans_free
      0.32 ±  2%      +0.0        0.34 ±  2%  perf-profile.children.cycles-pp.menu_select
      0.26 ±  3%      +0.0        0.28 ±  4%  perf-profile.children.cycles-pp.xlog_cil_process_committed
      0.26 ±  3%      +0.0        0.28 ±  4%  perf-profile.children.cycles-pp.xlog_cil_committed
      0.06 ±  6%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
      0.10 ±  3%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.__fput
      0.33 ±  2%      +0.0        0.36        perf-profile.children.cycles-pp.pick_next_task_fair
      0.18 ±  6%      +0.0        0.21 ±  5%  perf-profile.children.cycles-pp.xfs_trans_committed_bulk
      0.22 ±  5%      +0.0        0.25 ±  4%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.10 ±  4%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.close
      0.10 ±  4%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.10 ±  4%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.task_work_run
      0.09 ±  5%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.dput
      0.09 ±  5%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.__dentry_kill
      0.12 ±  4%      +0.0        0.15 ±  7%  perf-profile.children.cycles-pp.xfs_map_blocks
      0.44            +0.0        0.47        perf-profile.children.cycles-pp.update_load_avg
      1.49            +0.0        1.53        perf-profile.children.cycles-pp.xlog_cil_push
      0.08 ±  5%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.memcpy_erms
      0.28 ±  5%      +0.1        0.33 ±  3%  perf-profile.children.cycles-pp.xlog_state_release_iclog
      0.19 ±  4%      +0.1        0.24        perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
      1.58            +0.1        1.64        perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.xfs_inactive_truncate
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.xfs_buf_find
      1.45            +0.1        1.51        perf-profile.children.cycles-pp.arch_stack_walk
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_cil_prepare_item
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.xfs_buf_read_map
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.destroy_inode
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.xfs_fs_destroy_inode
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.xfs_inactive
      2.96            +0.1        3.04        perf-profile.children.cycles-pp.__wake_up_common
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.xfs_btree_read_buf_block
      0.00            +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.io_schedule
      0.00            +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.wait_on_page_bit
      0.00            +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
      0.00            +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.xfs_verify_fsbno
      1.98            +0.1        2.13        perf-profile.children.cycles-pp.__account_scheduler_latency
      0.00            +0.2        0.17 ±  3%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      2.81            +0.2        2.98        perf-profile.children.cycles-pp.enqueue_task_fair
      2.83            +0.2        3.00        perf-profile.children.cycles-pp.ttwu_do_activate
      2.83            +0.2        3.00        perf-profile.children.cycles-pp.activate_task
      0.00            +0.2        0.17 ±  4%  perf-profile.children.cycles-pp.xfs_end_bio
      2.54            +0.2        2.72        perf-profile.children.cycles-pp.enqueue_entity
      0.00            +0.2        0.18 ±  2%  perf-profile.children.cycles-pp.__queue_work
      0.00            +0.2        0.19 ±  3%  perf-profile.children.cycles-pp.queue_work_on
      0.00            +0.2        0.22 ±  3%  perf-profile.children.cycles-pp.xfs_btree_lookup
      0.07 ±  9%      +0.2        0.30 ±  4%  perf-profile.children.cycles-pp.xfs_verify_agbno
      3.18            +0.2        3.42        perf-profile.children.cycles-pp.try_to_wake_up
      0.11 ±  7%      +0.3        0.44 ±  3%  perf-profile.children.cycles-pp.xfs_bmap_validate_extent
      0.15 ±  6%      +0.4        0.51 ±  3%  perf-profile.children.cycles-pp.xfs_iextents_copy
      0.00            +0.4        0.36        perf-profile.children.cycles-pp.autoremove_wake_function
      0.18 ±  7%      +0.4        0.55 ±  3%  perf-profile.children.cycles-pp.xfs_inode_item_format_data_fork
      0.26 ±  5%      +0.4        0.63 ±  3%  perf-profile.children.cycles-pp.xfs_inode_item_format
      0.00            +0.4        0.38        perf-profile.children.cycles-pp.wake_up_page_bit
     15.82            +0.4       16.23        perf-profile.children.cycles-pp.intel_idle
      0.00            +0.4        0.43 ±  2%  perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
      0.00            +0.5        0.45 ±  2%  perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
     16.30            +0.5       16.76        perf-profile.children.cycles-pp.cpuidle_enter_state
     16.30            +0.5       16.77        perf-profile.children.cycles-pp.cpuidle_enter
      0.00            +0.5        0.46 ± 37%  perf-profile.children.cycles-pp.xlog_grant_head_wait
     18.27            +0.5       18.78        perf-profile.children.cycles-pp.do_idle
     18.27            +0.5       18.78        perf-profile.children.cycles-pp.secondary_startup_64
     18.27            +0.5       18.78        perf-profile.children.cycles-pp.cpu_startup_entry
      0.00            +0.5        0.52        perf-profile.children.cycles-pp.xfs_bmapi_write
     18.01            +0.6       18.57        perf-profile.children.cycles-pp.start_secondary
      0.01 ±200%      +0.6        0.57        perf-profile.children.cycles-pp.iomap_finish_ioends
      0.25 ±  5%      +0.8        1.10 ± 27%  perf-profile.children.cycles-pp.xfs_log_done
      0.00            +0.9        0.89 ± 35%  perf-profile.children.cycles-pp.xfs_log_space_wake
      3.75 ±  2%      +1.0        4.71 ±  6%  perf-profile.children.cycles-pp.xfs_file_aio_write_checks
      3.65 ±  2%      +1.0        4.62 ±  6%  perf-profile.children.cycles-pp.file_update_time
      3.60 ±  2%      +1.0        4.58 ±  6%  perf-profile.children.cycles-pp.xfs_vn_update_time
      0.00            +1.0        1.01 ± 43%  perf-profile.children.cycles-pp.xlog_grant_head_check
      0.35 ±  5%      +1.0        1.37 ± 31%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.42 ±  4%      +1.1        1.48 ± 28%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.57 ±  3%      +1.1        1.66 ± 25%  perf-profile.children.cycles-pp.xfs_trans_alloc
      2.90 ±  2%      +1.4        4.27 ±  6%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      2.98 ±  2%      +1.4        4.39 ±  6%  perf-profile.children.cycles-pp.__xfs_trans_commit
     23.89            +1.7       25.61        perf-profile.children.cycles-pp.remove_wait_queue
      0.00            +2.0        2.04 ± 17%  perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
      4.92            +2.3        7.18 ±  4%  perf-profile.children.cycles-pp.process_one_work
      5.24            +2.3        7.55 ±  4%  perf-profile.children.cycles-pp.worker_thread
      5.25            +2.3        7.56 ±  4%  perf-profile.children.cycles-pp.kthread
      5.25            +2.3        7.56 ±  4%  perf-profile.children.cycles-pp.ret_from_fork
      0.06 ±  6%      +2.6        2.61 ± 13%  perf-profile.children.cycles-pp.xfs_end_ioend
      0.06 ±  6%      +2.6        2.62 ± 13%  perf-profile.children.cycles-pp.xfs_end_io
     65.86            -0.8       65.06        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.71            -0.1        0.58 ±  2%  perf-profile.self.cycles-pp.brd_do_bvec
      0.60 ±  6%      -0.1        0.49 ±  5%  perf-profile.self.cycles-pp.osq_lock
      0.30 ±  2%      -0.1        0.23 ±  2%  perf-profile.self.cycles-pp.do_syscall_64
      0.55 ±  3%      -0.1        0.49        perf-profile.self.cycles-pp._raw_spin_lock
      0.17 ±  6%      -0.0        0.12 ±  4%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.17 ±  4%      -0.0        0.13 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.16 ±  2%      -0.0        0.13 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.19 ±  3%      -0.0        0.16 ±  2%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      0.17 ±  5%      -0.0        0.15 ±  3%  perf-profile.self.cycles-pp.xlog_cil_force_lsn
      0.10 ±  4%      -0.0        0.07        perf-profile.self.cycles-pp.find_get_entry
      0.08 ±  4%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.brd_lookup_page
      0.09 ±  5%      -0.0        0.08        perf-profile.self.cycles-pp.___might_sleep
      0.09 ±  5%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.xas_load
      0.07            -0.0        0.06        perf-profile.self.cycles-pp.xfs_file_llseek
      0.09            -0.0        0.08        perf-profile.self.cycles-pp.dequeue_task_fair
      0.08 ±  5%      +0.0        0.09        perf-profile.self.cycles-pp.do_idle
      0.09 ±  5%      +0.0        0.10        perf-profile.self.cycles-pp.orc_find
      0.17 ±  2%      +0.0        0.19 ±  3%  perf-profile.self.cycles-pp.poll_idle
      0.12 ±  5%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.menu_select
      0.19 ±  5%      +0.0        0.22 ±  6%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.15 ±  4%      +0.0        0.18 ±  6%  perf-profile.self.cycles-pp.__module_address
      0.07 ± 10%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.xfs_trans_reserve
      0.06 ± 12%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.xfs_trans_free
      0.19 ±  3%      +0.0        0.21 ±  4%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.10 ±  4%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.xfs_trans_alloc
      0.08 ±  6%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.memcpy_erms
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.00            +0.1        0.13 ±  3%  perf-profile.self.cycles-pp.xfs_verify_fsbno
      0.06 ±  7%      +0.2        0.28 ±  3%  perf-profile.self.cycles-pp.xfs_verify_agbno
     15.82            +0.4       16.23        perf-profile.self.cycles-pp.intel_idle


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  35000 +-------------------------------------------------------------------+   
        |                                                                   |   
  30000 |.+.+.+..+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+..+                        |   
        |                                                                   |   
  25000 |-+                                                                 |   
        | O   O    O   O O O O O  O   O O O O O O  O O O O O O O O  O O O O |   
  20000 |-+                                                                 |   
        |                                                                   |   
  15000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
        |                                                                   |   
   5000 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  180 +---------------------------------------------------------------------+   
      |                     O   O                                           |   
  160 |-O    O   O   O  O O   O      O O O O O  O O O O O  O O O O O O  O O |   
  140 |-+                                                                   |   
      |.+.  .+.+.+.+.+..+. .+. .+.  .+.+. .+.  .+.+                         |   
  120 |-+ +.              +   +   +.     +   +.                             |   
  100 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
   60 |-+                                                                   |   
      |                                                                     |   
   40 |-+                                                                   |   
   20 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  180 +---------------------------------------------------------------------+   
      |                     O   O                                           |   
  160 |-O    O   O   O  O O   O      O O O O O  O O O O O  O O O O O O  O O |   
  140 |-+                                                                   |   
      |.+.  .+.+.+.+.+..+. .+. .+.  .+.+. .+.  .+.+                         |   
  120 |-+ +.              +   +   +.     +   +.                             |   
  100 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
   60 |-+                                                                   |   
      |                                                                     |   
   40 |-+                                                                   |   
   20 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        aim7.time.voluntary_context_switches                    
                                                                                
  9e+07 +-------------------------------------------------------------------+   
        | O   O    O   O O O O O  O   O O O O O O  O O O O O O O O  O O O O |   
  8e+07 |-+                                                                 |   
  7e+07 |-+                                                                 |   
        |.+.+.+..+.+.+.+.+. .+.+..+.+.+.+.+.+.+.+..+                        |   
  6e+07 |-+                +                                                |   
  5e+07 |-+                                                                 |   
        |                                                                   |   
  4e+07 |-+                                                                 |   
  3e+07 |-+                                                                 |   
        |                                                                   |   
  2e+07 |-+                                                                 |   
  1e+07 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        aim7.time.involuntary_context_switches                  
                                                                                
  1.4e+06 +-----------------------------------------------------------------+   
          |                                                                 |   
  1.2e+06 |-+                                                               |   
          |                                                                 |   
    1e+06 |-+           O  O O O O O   O O O O O O O O O O  O O O O O O O O |   
          |        .+.    .+.                                               |   
   800000 |.+.+.+.+   +.+.   +.+.+.+.+.+.+.+.+.+.+.+                        |   
          |                                                                 |   
   600000 |-+                                                               |   
          |                                                                 |   
   400000 |-+                                                               |   
          |                                                                 |   
   200000 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc3-00001-g41d44159a3575" of type "text/plain" (203539 bytes)

View attachment "job-script" of type "text/plain" (7697 bytes)

View attachment "job.yaml" of type "text/plain" (5350 bytes)

View attachment "reproduce" of type "text/plain" (714 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ