lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Mon, 1 Mar 2021 15:01:22 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Xiao Ni <xni@...hat.com>
Cc:     Song Liu <songliubraving@...com>,
        David Jeffery <djeffery@...hat.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com
Subject: [md]  dc5d17a3c3:  aim7.jobs-per-min -9.0% regression


Greeting,

FYI, we noticed a -9.0% regression of aim7.jobs-per-min due to commit:


commit: dc5d17a3c39b06aef866afca19245a9cfb533a79 ("md: Set prev_flush_start and flush_bio in an atomic way")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: aim7
on test machine: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: xfs
	test: sync_disk_rw
	load: 300
	cpufreq_governor: performance
	ucode: 0x5003006

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/4BRD_12G/xfs/x86_64-rhel-8.3/300/RAID0/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/sync_disk_rw/aim7/0x5003006

commit: 
  b4f664252f ("Merge tag 'nvme-5.11-2021-01-14' of git://git.infradead.org/nvme into block-5.11")
  dc5d17a3c3 ("md: Set prev_flush_start and flush_bio in an atomic way")

b4f664252f51e119 dc5d17a3c39b06aef866afca192 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:6          -17%            :6     kmsg.XFS(md#):xlog_verify_grant_tail:space>BBTOB(tail_blocks)
         %stddev     %change         %stddev
             \          |                \  
     14981            -9.0%      13633        aim7.jobs-per-min
    120.19            +9.9%     132.07        aim7.time.elapsed_time
    120.19            +9.9%     132.07        aim7.time.elapsed_time.max
   1879621            -9.8%    1694730        aim7.time.involuntary_context_switches
     37994           +10.3%      41920 ±  3%  aim7.time.minor_page_faults
      4772           +16.2%       5544        aim7.time.system_time
     13.56            +5.8%      14.34        aim7.time.user_time
  64176881           +13.4%   72751737        aim7.time.voluntary_context_switches
 7.744e+08 ±  2%     +22.7%  9.498e+08        cpuidle.C1.time
  16895358 ±  2%     +23.1%   20803638        cpuidle.C1.usage
     51.70            -5.4%      48.92        iostat.cpu.idle
     47.93            +5.7%      50.65        iostat.cpu.system
      0.04 ±  2%      -0.0        0.02 ± 21%  mpstat.cpu.all.iowait%
      0.31 ±  2%      +0.1        0.39        mpstat.cpu.all.usr%
      2577 ±  2%     -11.2%       2289 ±  3%  slabinfo.pool_workqueue.active_objs
      2578 ±  2%     -11.2%       2290 ±  3%  slabinfo.pool_workqueue.num_objs
    778.93 ±  2%      -9.0%     708.45 ±  2%  sched_debug.cfs_rq:/.load_avg.avg
   1299356           +16.4%    1513095        sched_debug.cfs_rq:/.min_vruntime.avg
   1362913           +16.9%    1592901        sched_debug.cfs_rq:/.min_vruntime.max
   1283414           +16.4%    1494133        sched_debug.cfs_rq:/.min_vruntime.min
    658.00 ± 13%     +32.3%     870.56 ± 10%  sched_debug.cpu.nr_uninterruptible.max
    279.79 ±  3%     +24.2%     347.63 ±  4%  sched_debug.cpu.nr_uninterruptible.stddev
     51.00            -5.2%      48.33        vmstat.cpu.id
     47.33            +5.6%      50.00        vmstat.cpu.sy
    600413            -8.7%     548390        vmstat.io.bo
   1940090           +16.4%    2257559        vmstat.memory.cache
   1062494            +1.9%    1082293        vmstat.system.cs
    215183 ±  2%      +2.7%     221014        vmstat.system.in
    231787 ±  2%     +54.0%     356922 ±  4%  meminfo.Active
    231664 ±  2%     +54.0%     356800 ±  4%  meminfo.Active(anon)
   1807605           +18.5%    2141278        meminfo.Cached
   1191369           +28.1%    1526251        meminfo.Committed_AS
    791985           +27.0%    1006098        meminfo.Inactive
    648301           +32.8%     860627        meminfo.Inactive(anon)
    175250 ±  2%    +150.7%     439290 ±  6%  meminfo.Mapped
   3601304            +9.8%    3954998        meminfo.Memused
    626026           +53.0%     957922 ±  2%  meminfo.Shmem
     67804 ±  5%     +42.8%      96810 ±  9%  numa-meminfo.node0.Active
     67721 ±  5%     +42.8%      96706 ±  9%  numa-meminfo.node0.Active(anon)
     78240 ±  8%     +99.0%     155706 ± 12%  numa-meminfo.node0.Mapped
    207154 ±  6%     +36.5%     282689 ± 11%  numa-meminfo.node0.Shmem
    165126           +57.2%     259597 ±  6%  numa-meminfo.node1.Active
    165086           +57.2%     259578 ±  6%  numa-meminfo.node1.Active(anon)
   1014050           +25.4%    1271894 ±  3%  numa-meminfo.node1.FilePages
     98637 ±  3%    +186.0%     282083 ± 10%  numa-meminfo.node1.Mapped
   1853602 ±  7%     +24.3%    2303765 ±  3%  numa-meminfo.node1.MemUsed
    422089           +59.7%     674035 ±  6%  numa-meminfo.node1.Shmem
     17214 ±  5%     +41.1%      24284 ± 10%  numa-vmstat.node0.nr_active_anon
     19996 ±  8%     +96.7%      39332 ± 13%  numa-vmstat.node0.nr_mapped
     52597 ±  6%     +35.0%      71000 ± 11%  numa-vmstat.node0.nr_shmem
     17214 ±  5%     +41.1%      24284 ± 10%  numa-vmstat.node0.nr_zone_active_anon
      6311 ±  6%     +13.6%       7170 ±  4%  numa-vmstat.node0.nr_zone_write_pending
     41745           +55.9%      65096 ±  5%  numa-vmstat.node1.nr_active_anon
    255013           +24.9%     318602 ±  2%  numa-vmstat.node1.nr_file_pages
     25382 ±  3%    +180.5%      71206 ±  7%  numa-vmstat.node1.nr_mapped
    106930           +58.2%     169120 ±  5%  numa-vmstat.node1.nr_shmem
     41745           +55.9%      65096 ±  5%  numa-vmstat.node1.nr_zone_active_anon
      6215 ±  5%     +13.5%       7056 ±  5%  numa-vmstat.node1.nr_zone_write_pending
   1891802 ±  4%      +9.7%    2074722 ±  3%  numa-vmstat.node1.numa_hit
   1782036 ±  5%      +9.9%    1959007 ±  2%  numa-vmstat.node1.numa_local
     58321 ±  2%     +52.3%      88836 ±  4%  proc-vmstat.nr_active_anon
     63639            +2.0%      64882        proc-vmstat.nr_anon_pages
    453050           +17.9%     534355        proc-vmstat.nr_file_pages
    162846           +31.7%     214480        proc-vmstat.nr_inactive_anon
     35938            +1.4%      36432        proc-vmstat.nr_inactive_file
     44493 ±  2%    +144.3%     108686 ±  4%  proc-vmstat.nr_mapped
    157643           +51.3%     238448 ±  2%  proc-vmstat.nr_shmem
     58321 ±  2%     +52.3%      88836 ±  4%  proc-vmstat.nr_zone_active_anon
    162846           +31.7%     214480        proc-vmstat.nr_zone_inactive_anon
     35938            +1.4%      36432        proc-vmstat.nr_zone_inactive_file
    254089            +5.1%     266997 ±  2%  proc-vmstat.numa_hint_faults
    131678            +7.9%     142105 ±  3%  proc-vmstat.numa_hint_faults_local
    360157 ±  2%      +5.7%     380551        proc-vmstat.numa_pte_updates
    789220            +9.8%     866813        proc-vmstat.pgfault
     28159 ±  2%      +8.0%      30418        proc-vmstat.pgreuse
 4.216e+09            +2.3%  4.313e+09        perf-stat.i.branch-instructions
      1.24 ± 13%      -0.2        1.08        perf-stat.i.branch-miss-rate%
   1083772            +1.5%    1100299        perf-stat.i.context-switches
      6.47            +3.1%       6.67        perf-stat.i.cpi
 1.236e+11            +5.5%  1.305e+11        perf-stat.i.cpu-cycles
    148806           +18.1%     175739        perf-stat.i.cpu-migrations
      2363            +4.5%       2470        perf-stat.i.cycles-between-cache-misses
 4.961e+09            +1.7%  5.047e+09        perf-stat.i.dTLB-loads
 1.724e+09            -2.3%  1.684e+09        perf-stat.i.dTLB-stores
  10571562            -3.0%   10256419        perf-stat.i.iTLB-load-misses
  17061563            -4.7%   16251956        perf-stat.i.iTLB-loads
 1.886e+10            +1.7%  1.919e+10        perf-stat.i.instructions
      1801            +4.2%       1877        perf-stat.i.instructions-per-iTLB-miss
      1.41            +5.5%       1.48        perf-stat.i.metric.GHz
    126.92            +1.3%     128.56        perf-stat.i.metric.M/sec
  18213137            +4.3%   18990449        perf-stat.i.node-load-misses
     82.21            +1.5       83.72        perf-stat.i.node-store-miss-rate%
   8943783            -1.9%    8776890        perf-stat.i.node-store-misses
   1808751            -8.2%    1659794        perf-stat.i.node-stores
      0.98            -0.0        0.94        perf-stat.overall.branch-miss-rate%
      6.55            +3.7%       6.80        perf-stat.overall.cpi
      2352            +4.7%       2463        perf-stat.overall.cycles-between-cache-misses
      1784            +4.8%       1871        perf-stat.overall.instructions-per-iTLB-miss
      0.15            -3.6%       0.15        perf-stat.overall.ipc
     83.18            +0.9       84.10        perf-stat.overall.node-store-miss-rate%
 4.182e+09            +2.4%  4.281e+09        perf-stat.ps.branch-instructions
   1074838            +1.6%    1092240        perf-stat.ps.context-switches
 1.226e+11            +5.6%  1.295e+11        perf-stat.ps.cpu-cycles
    147589           +18.2%     174459        perf-stat.ps.cpu-migrations
 4.921e+09            +1.8%   5.01e+09        perf-stat.ps.dTLB-loads
  1.71e+09            -2.2%  1.672e+09        perf-stat.ps.dTLB-stores
  10484416            -2.9%   10181177        perf-stat.ps.iTLB-load-misses
  16920684            -4.7%   16132601        perf-stat.ps.iTLB-loads
 1.871e+10            +1.8%  1.905e+10        perf-stat.ps.instructions
  18063320            +4.4%   18851552        perf-stat.ps.node-load-misses
   8870434            -1.8%    8712977        perf-stat.ps.node-store-misses
   1793923            -8.2%    1647689        perf-stat.ps.node-stores
 2.267e+12           +11.9%  2.536e+12        perf-stat.total.instructions
      0.05           -12.9%       0.04 ±  2%  perf-sched.sch_delay.avg.ms.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      0.06           +29.8%       0.08 ±  2%  perf-sched.sch_delay.avg.ms.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range
      0.06 ± 27%     +61.0%       0.09 ± 15%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.06 ± 68%     -74.0%       0.02 ±102%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
      0.04 ±  2%      -9.8%       0.04 ±  4%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.blkdev_issue_flush
      0.06 ± 28%    +102.6%       0.12 ± 26%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      0.06 ± 30%     +53.4%       0.09 ±  8%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
      0.04 ±  2%     -15.9%       0.03 ±  9%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      0.07 ± 14%     +57.3%       0.12 ± 14%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.wait_for_completion.__flush_work.xlog_cil_force_lsn
      0.31 ±  2%     +18.2%       0.37 ±  2%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.wait_for_completion_io.submit_bio_wait.blkdev_issue_flush
      0.17 ±  5%     +13.0%       0.19 ±  6%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
      0.04 ± 16%     +25.1%       0.05 ± 12%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.xlog_cil_push_work.process_one_work.worker_thread
      0.03 ±  8%     -20.7%       0.03 ± 12%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.04 ±  2%     +28.6%       0.06        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
      0.01 ±  4%     +50.7%       0.02 ±  2%  perf-sched.sch_delay.avg.ms.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      0.12           -10.9%       0.11        perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      0.24 ±159%     -99.5%       0.00 ±143%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.dput.terminate_walk.path_openat
      0.37 ± 73%     -88.3%       0.04 ±169%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
      0.89 ± 57%     -68.5%       0.28 ± 87%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
      0.96 ± 22%     -33.9%       0.64 ± 14%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
      0.05           +12.9%       0.06        perf-sched.total_sch_delay.average.ms
      0.38           -21.9%       0.30 ±  2%  perf-sched.wait_and_delay.avg.ms.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range
      2.07 ± 12%    +118.4%       4.51 ± 74%  perf-sched.wait_and_delay.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
      6.38 ± 81%     -62.3%       2.41 ± 17%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      4.32           +38.1%       5.97 ± 14%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.83           +16.1%       0.96        perf-sched.wait_and_delay.avg.ms.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
    222682            -9.6%     201332        perf-sched.wait_and_delay.count.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range
    598621           +32.8%     794704        perf-sched.wait_and_delay.count.md_flush_request.raid0_make_request.md_handle_request.md_submit_bio
     78550           -13.5%      67957        perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.wait_for_completion_io.submit_bio_wait.blkdev_issue_flush
    414.50 ±  4%     -30.6%     287.50 ± 21%  perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
      6852 ±  3%     -37.5%       4281 ±  4%  perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten
    159.67 ±  4%     -27.7%     115.50 ±  4%  perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
      2016           -22.6%       1560 ± 13%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    525713            -8.9%     478695        perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    441532           -11.3%     391796        perf-sched.wait_and_delay.count.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
     18.99 ± 14%   +1006.0%     210.06 ±184%  perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
      6.38 ± 81%     -62.3%       2.41 ± 17%  perf-sched.wait_and_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      6.65 ± 20%     +50.3%      10.00 ±  9%  perf-sched.wait_and_delay.max.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      0.32           -31.6%       0.22 ±  2%  perf-sched.wait_time.avg.ms.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range
      0.81           -10.0%       0.73        perf-sched.wait_time.avg.ms.md_flush_request.raid0_make_request.md_handle_request.md_submit_bio
      0.61           +17.9%       0.71        perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
      0.26 ± 20%     -35.3%       0.17 ± 25%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_read.xfs_file_fsync.xfs_file_buffered_aio_write
      0.18 ± 38%     -54.8%       0.08 ± 30%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_read.xfs_log_commit_cil.__xfs_trans_commit
      0.20 ±  7%     -30.2%       0.14 ±  8%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write.xfs_ilock.xfs_buffered_write_iomap_begin
      0.20 ± 17%     -32.0%       0.14 ± 18%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write.xfs_ilock.xfs_vn_update_time
      0.17 ± 16%     -29.6%       0.12 ± 25%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_vn_update_time
      0.79 ±  9%     -23.6%       0.60 ± 15%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.mempool_alloc.md_submit_bio.submit_bio_noacct
      0.29 ±  8%     -46.7%       0.15 ±  3%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      2.01 ± 12%    +120.5%       4.44 ± 75%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
      0.62 ±  5%     -18.1%       0.51 ±  6%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
      6.32 ± 82%     -62.9%       2.35 ± 15%  perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      0.04 ± 16%     +57.9%       0.07 ± 12%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io.submit_bio_wait
      4.29           +38.6%       5.94 ± 14%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.82           +15.6%       0.94        perf-sched.wait_time.avg.ms.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      4.48 ±  6%     +34.9%       6.05 ± 20%  perf-sched.wait_time.max.ms.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      3.38 ±  8%     +14.6%       3.87 ±  6%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
      2.38 ± 18%     -62.4%       0.90 ± 38%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.down_read.xfs_log_commit_cil.__xfs_trans_commit
      1.40 ± 52%     -83.8%       0.23 ±114%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin
     16.66 ± 15%     -67.9%       5.35 ± 21%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
      1.72 ± 32%    +216.7%       5.46 ± 87%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.vfs_write.ksys_write.do_syscall_64
      1.69 ± 48%     -73.2%       0.45 ± 64%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
     18.98 ± 14%   +1006.6%     210.05 ±184%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_roll
      6.32 ± 82%     -62.9%       2.35 ± 15%  perf-sched.wait_time.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      4.02 ± 15%     +35.9%       5.47 ± 13%  perf-sched.wait_time.max.ms.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
     29255 ±  2%     +11.1%      32503 ±  2%  softirqs.CPU0.SCHED
     27245           +11.4%      30344        softirqs.CPU1.SCHED
     26590           +11.9%      29742        softirqs.CPU10.SCHED
     26268           +13.6%      29836        softirqs.CPU11.SCHED
     26364           +13.3%      29867        softirqs.CPU12.SCHED
     26238           +13.5%      29792        softirqs.CPU13.SCHED
     26397           +12.2%      29607        softirqs.CPU14.SCHED
     26339           +13.3%      29841        softirqs.CPU15.SCHED
     26232           +13.1%      29669        softirqs.CPU16.SCHED
     25904           +14.8%      29749        softirqs.CPU17.SCHED
     26440           +13.4%      29973        softirqs.CPU18.SCHED
     26425           +13.2%      29904        softirqs.CPU19.SCHED
     26571           +13.2%      30069 ±  2%  softirqs.CPU2.SCHED
     26226           +12.8%      29573        softirqs.CPU20.SCHED
     26347           +12.5%      29637        softirqs.CPU21.SCHED
     26813           +10.2%      29548 ±  2%  softirqs.CPU22.SCHED
     26447           +14.3%      30227        softirqs.CPU23.SCHED
     26316           +14.3%      30075        softirqs.CPU24.SCHED
     26356           +14.4%      30153        softirqs.CPU25.SCHED
     26421           +14.2%      30165        softirqs.CPU26.SCHED
     26405           +14.2%      30160        softirqs.CPU28.SCHED
     26426           +14.3%      30215        softirqs.CPU29.SCHED
     26759           +12.0%      29959        softirqs.CPU3.SCHED
     26260           +13.7%      29866        softirqs.CPU30.SCHED
     26456           +14.2%      30201        softirqs.CPU31.SCHED
     26514           +14.2%      30274        softirqs.CPU32.SCHED
     26626           +13.1%      30105        softirqs.CPU33.SCHED
     26531           +13.1%      30001        softirqs.CPU34.SCHED
     26296           +14.9%      30225        softirqs.CPU35.SCHED
     26475           +13.1%      29934        softirqs.CPU36.SCHED
     26219           +15.4%      30265        softirqs.CPU37.SCHED
     26536           +12.2%      29766 ±  3%  softirqs.CPU38.SCHED
     26323           +14.6%      30179        softirqs.CPU39.SCHED
     26914 ±  3%     +10.7%      29801        softirqs.CPU4.SCHED
     26275           +14.5%      30083        softirqs.CPU40.SCHED
     26269 ±  2%     +14.3%      30025        softirqs.CPU41.SCHED
     26443           +14.2%      30196        softirqs.CPU42.SCHED
     23618 ± 13%     +19.2%      28161 ±  4%  softirqs.CPU43.SCHED
     26111           +13.5%      29635        softirqs.CPU44.SCHED
     26100           +14.0%      29751        softirqs.CPU45.SCHED
     26063           +13.0%      29460        softirqs.CPU46.SCHED
     26406           +12.0%      29567        softirqs.CPU47.SCHED
     26584           +12.3%      29845        softirqs.CPU48.SCHED
     26422           +12.4%      29693        softirqs.CPU49.SCHED
     26422           +14.3%      30212 ±  2%  softirqs.CPU5.SCHED
     26587           +12.2%      29842        softirqs.CPU50.SCHED
     26659 ±  2%     +12.2%      29915        softirqs.CPU51.SCHED
     26591           +12.1%      29820 ±  2%  softirqs.CPU52.SCHED
     26430           +12.7%      29788        softirqs.CPU53.SCHED
     26327           +13.6%      29907        softirqs.CPU54.SCHED
     26419           +12.8%      29811        softirqs.CPU55.SCHED
     26522           +13.3%      30046        softirqs.CPU56.SCHED
     26456           +13.8%      30110        softirqs.CPU57.SCHED
     26721 ±  2%     +12.0%      29937        softirqs.CPU58.SCHED
     26373           +13.9%      30046        softirqs.CPU59.SCHED
     26497 ±  2%     +13.4%      30054        softirqs.CPU6.SCHED
     26411           +13.8%      30064        softirqs.CPU60.SCHED
     26476           +13.2%      29969        softirqs.CPU61.SCHED
     26385           +14.3%      30170 ±  2%  softirqs.CPU62.SCHED
     26470           +12.4%      29765        softirqs.CPU63.SCHED
     26251           +13.8%      29869        softirqs.CPU64.SCHED
     26467 ±  2%     +12.8%      29845        softirqs.CPU65.SCHED
     26408           +14.6%      30270        softirqs.CPU66.SCHED
     26374           +14.6%      30224        softirqs.CPU67.SCHED
     26478           +14.3%      30260        softirqs.CPU68.SCHED
     26548           +13.3%      30072        softirqs.CPU69.SCHED
     26421           +13.2%      29899        softirqs.CPU7.SCHED
     26504           +13.7%      30144        softirqs.CPU70.SCHED
     26350           +14.5%      30162        softirqs.CPU71.SCHED
     26507           +14.1%      30248        softirqs.CPU72.SCHED
     26464           +14.2%      30211        softirqs.CPU73.SCHED
     26322           +14.6%      30165        softirqs.CPU74.SCHED
     26576 ±  2%     +13.6%      30182        softirqs.CPU75.SCHED
     26410           +15.0%      30378        softirqs.CPU76.SCHED
     26425           +13.9%      30109 ±  2%  softirqs.CPU77.SCHED
     26383           +14.3%      30157        softirqs.CPU78.SCHED
     26487           +14.5%      30321        softirqs.CPU79.SCHED
     26446           +12.0%      29626        softirqs.CPU8.SCHED
     26243           +15.1%      30200        softirqs.CPU80.SCHED
     26413           +14.6%      30266        softirqs.CPU81.SCHED
     26580           +14.0%      30313        softirqs.CPU82.SCHED
     26574           +13.8%      30254        softirqs.CPU83.SCHED
     26491           +14.7%      30389        softirqs.CPU84.SCHED
     26261           +14.6%      30086        softirqs.CPU85.SCHED
     26178           +15.4%      30219        softirqs.CPU86.SCHED
     25535 ±  2%     +14.4%      29223 ±  3%  softirqs.CPU87.SCHED
     26332           +13.7%      29952        softirqs.CPU9.SCHED
   2325436           +13.5%    2640181        softirqs.SCHED
   4135445           +21.7%    5032854        interrupts.CAL:Function_call_interrupts
     46753 ±  3%     +21.3%      56702 ±  2%  interrupts.CPU0.CAL:Function_call_interrupts
     47519 ±  2%     +21.3%      57624        interrupts.CPU1.CAL:Function_call_interrupts
     47007 ±  2%     +20.0%      56394        interrupts.CPU10.CAL:Function_call_interrupts
    240269 ±  2%     +10.8%     266146        interrupts.CPU10.LOC:Local_timer_interrupts
     47331 ±  2%     +20.6%      57091        interrupts.CPU11.CAL:Function_call_interrupts
      4860           +10.5%       5369 ±  2%  interrupts.CPU11.RES:Rescheduling_interrupts
     47277 ±  2%     +21.4%      57381        interrupts.CPU12.CAL:Function_call_interrupts
    240202 ±  2%     +10.8%     266113        interrupts.CPU12.LOC:Local_timer_interrupts
      4980 ±  3%     +12.9%       5620 ±  2%  interrupts.CPU12.RES:Rescheduling_interrupts
     46974 ±  2%     +20.4%      56541        interrupts.CPU13.CAL:Function_call_interrupts
     47160 ±  2%     +19.6%      56426 ±  2%  interrupts.CPU14.CAL:Function_call_interrupts
      4832 ±  3%      +9.6%       5297 ±  3%  interrupts.CPU14.RES:Rescheduling_interrupts
     47213 ±  2%     +20.1%      56725 ±  2%  interrupts.CPU15.CAL:Function_call_interrupts
      4985           +10.7%       5517 ±  2%  interrupts.CPU15.RES:Rescheduling_interrupts
     47074 ±  2%     +20.7%      56821        interrupts.CPU16.CAL:Function_call_interrupts
     46809           +21.3%      56802        interrupts.CPU17.CAL:Function_call_interrupts
      4800 ±  3%     +12.3%       5388 ±  2%  interrupts.CPU17.RES:Rescheduling_interrupts
     47042 ±  3%     +20.4%      56632        interrupts.CPU18.CAL:Function_call_interrupts
     47551 ±  2%     +21.0%      57551        interrupts.CPU19.CAL:Function_call_interrupts
    240258 ±  2%     +10.8%     266159        interrupts.CPU19.LOC:Local_timer_interrupts
      4960           +10.8%       5496 ±  3%  interrupts.CPU19.RES:Rescheduling_interrupts
     47559           +21.2%      57635        interrupts.CPU2.CAL:Function_call_interrupts
    240226 ±  2%     +10.8%     266226        interrupts.CPU2.LOC:Local_timer_interrupts
     47292 ±  2%     +20.4%      56917        interrupts.CPU20.CAL:Function_call_interrupts
      5133 ±  2%     +10.5%       5673 ±  2%  interrupts.CPU20.RES:Rescheduling_interrupts
     47060 ±  2%     +20.7%      56782        interrupts.CPU21.CAL:Function_call_interrupts
     46345           +19.7%      55480 ±  3%  interrupts.CPU22.CAL:Function_call_interrupts
      5138 ±  3%      +8.5%       5573 ±  3%  interrupts.CPU22.RES:Rescheduling_interrupts
     47128           +22.8%      57889 ±  2%  interrupts.CPU23.CAL:Function_call_interrupts
      4972 ±  3%     +10.5%       5496 ±  2%  interrupts.CPU23.RES:Rescheduling_interrupts
     46946           +22.9%      57700        interrupts.CPU24.CAL:Function_call_interrupts
      5091 ±  3%     +11.3%       5666 ±  3%  interrupts.CPU24.RES:Rescheduling_interrupts
     47346           +22.5%      57989 ±  2%  interrupts.CPU25.CAL:Function_call_interrupts
     46917           +22.9%      57652 ±  2%  interrupts.CPU26.CAL:Function_call_interrupts
      4991 ±  3%     +10.9%       5535        interrupts.CPU26.RES:Rescheduling_interrupts
     46920           +22.3%      57385 ±  2%  interrupts.CPU27.CAL:Function_call_interrupts
     47151           +23.5%      58239 ±  2%  interrupts.CPU28.CAL:Function_call_interrupts
      5061 ±  2%     +11.3%       5632 ±  3%  interrupts.CPU28.RES:Rescheduling_interrupts
     46813           +23.6%      57880        interrupts.CPU29.CAL:Function_call_interrupts
      4992 ±  3%     +11.3%       5556        interrupts.CPU29.RES:Rescheduling_interrupts
     47661 ±  2%     +18.9%      56665        interrupts.CPU3.CAL:Function_call_interrupts
      4962           +10.5%       5480 ±  3%  interrupts.CPU3.RES:Rescheduling_interrupts
     46797           +22.4%      57300 ±  2%  interrupts.CPU30.CAL:Function_call_interrupts
    240067 ±  3%     +10.9%     266197        interrupts.CPU30.LOC:Local_timer_interrupts
      4886 ±  4%     +11.0%       5422 ±  3%  interrupts.CPU30.RES:Rescheduling_interrupts
     47148           +23.5%      58237        interrupts.CPU31.CAL:Function_call_interrupts
      4992 ±  4%     +10.8%       5533 ±  2%  interrupts.CPU31.RES:Rescheduling_interrupts
     46931           +24.2%      58285 ±  2%  interrupts.CPU32.CAL:Function_call_interrupts
     47165           +22.9%      57966 ±  2%  interrupts.CPU33.CAL:Function_call_interrupts
      4844 ±  3%     +11.1%       5384 ±  2%  interrupts.CPU33.RES:Rescheduling_interrupts
     47323           +22.7%      58086 ±  2%  interrupts.CPU34.CAL:Function_call_interrupts
      5146 ±  3%     +11.8%       5751        interrupts.CPU34.RES:Rescheduling_interrupts
     47054           +23.3%      57996        interrupts.CPU35.CAL:Function_call_interrupts
     47029           +23.3%      57995 ±  2%  interrupts.CPU36.CAL:Function_call_interrupts
     46935           +23.2%      57821 ±  2%  interrupts.CPU37.CAL:Function_call_interrupts
      5168 ±  2%     +11.2%       5749 ±  4%  interrupts.CPU37.RES:Rescheduling_interrupts
     46925           +21.2%      56880 ±  4%  interrupts.CPU38.CAL:Function_call_interrupts
    240131 ±  2%     +10.9%     266393        interrupts.CPU38.LOC:Local_timer_interrupts
     47268           +23.6%      58429 ±  2%  interrupts.CPU39.CAL:Function_call_interrupts
     48069 ±  3%     +18.1%      56757        interrupts.CPU4.CAL:Function_call_interrupts
    240161 ±  2%     +10.9%     266236        interrupts.CPU4.LOC:Local_timer_interrupts
     47585           +21.9%      58006        interrupts.CPU40.CAL:Function_call_interrupts
      5004 ±  3%     +11.0%       5553 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
     47024           +23.1%      57901 ±  2%  interrupts.CPU41.CAL:Function_call_interrupts
     47177           +22.5%      57815        interrupts.CPU42.CAL:Function_call_interrupts
      5209 ±  5%     +12.2%       5844 ±  3%  interrupts.CPU42.RES:Rescheduling_interrupts
     46277 ±  2%     +24.9%      57792 ±  2%  interrupts.CPU43.CAL:Function_call_interrupts
      4860 ±  3%     +12.0%       5444 ±  2%  interrupts.CPU43.RES:Rescheduling_interrupts
     47371 ±  2%     +20.3%      56974        interrupts.CPU44.CAL:Function_call_interrupts
      4892 ±  2%     +11.5%       5454 ±  3%  interrupts.CPU44.RES:Rescheduling_interrupts
     46853 ±  2%     +21.2%      56771        interrupts.CPU45.CAL:Function_call_interrupts
     47001 ±  2%     +20.6%      56675        interrupts.CPU46.CAL:Function_call_interrupts
      4815 ±  2%     +11.7%       5376 ±  2%  interrupts.CPU46.RES:Rescheduling_interrupts
     47048 ±  2%     +20.7%      56771        interrupts.CPU47.CAL:Function_call_interrupts
    240255 ±  2%     +10.8%     266252        interrupts.CPU47.LOC:Local_timer_interrupts
     46968 ±  2%     +20.3%      56512        interrupts.CPU48.CAL:Function_call_interrupts
    240263 ±  2%     +10.8%     266324        interrupts.CPU48.LOC:Local_timer_interrupts
     47018 ±  2%     +20.2%      56521        interrupts.CPU49.CAL:Function_call_interrupts
     47235 ±  2%     +21.0%      57169 ±  3%  interrupts.CPU5.CAL:Function_call_interrupts
      4994 ±  2%     +10.3%       5511 ±  2%  interrupts.CPU5.RES:Rescheduling_interrupts
     47431 ±  3%     +20.6%      57192        interrupts.CPU50.CAL:Function_call_interrupts
     46914 ±  2%     +21.2%      56870        interrupts.CPU51.CAL:Function_call_interrupts
     47193 ±  2%     +20.1%      56662        interrupts.CPU52.CAL:Function_call_interrupts
      4814 ±  2%     +11.7%       5377 ±  4%  interrupts.CPU52.RES:Rescheduling_interrupts
     46881 ±  2%     +20.6%      56529        interrupts.CPU53.CAL:Function_call_interrupts
     46681 ±  2%     +20.6%      56285        interrupts.CPU54.CAL:Function_call_interrupts
      3597 ± 35%     +42.7%       5131 ± 20%  interrupts.CPU54.NMI:Non-maskable_interrupts
      3597 ± 35%     +42.7%       5131 ± 20%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
     47277 ±  2%     +20.4%      56915        interrupts.CPU55.CAL:Function_call_interrupts
      5033 ±  3%     +11.1%       5594 ±  4%  interrupts.CPU55.RES:Rescheduling_interrupts
     46881           +21.1%      56753        interrupts.CPU56.CAL:Function_call_interrupts
      4908 ±  2%     +10.3%       5413        interrupts.CPU56.RES:Rescheduling_interrupts
     47067 ±  2%     +20.1%      56525        interrupts.CPU57.CAL:Function_call_interrupts
     47115 ±  2%     +20.4%      56728        interrupts.CPU58.CAL:Function_call_interrupts
      4804 ±  2%     +11.0%       5331 ±  2%  interrupts.CPU58.RES:Rescheduling_interrupts
     47248 ±  2%     +19.4%      56436        interrupts.CPU59.CAL:Function_call_interrupts
     47681 ±  4%     +19.4%      56932        interrupts.CPU6.CAL:Function_call_interrupts
      4809 ±  2%     +10.6%       5318 ±  2%  interrupts.CPU6.RES:Rescheduling_interrupts
     46795 ±  2%     +19.8%      56051        interrupts.CPU60.CAL:Function_call_interrupts
     47107 ±  2%     +20.2%      56628        interrupts.CPU61.CAL:Function_call_interrupts
      4750 ±  2%     +10.5%       5248 ±  2%  interrupts.CPU61.RES:Rescheduling_interrupts
     47410 ±  2%     +19.8%      56793        interrupts.CPU62.CAL:Function_call_interrupts
     47120 ±  2%     +19.9%      56482        interrupts.CPU63.CAL:Function_call_interrupts
     46893 ±  2%     +19.4%      56013        interrupts.CPU64.CAL:Function_call_interrupts
    240182 ±  2%     +10.8%     266224        interrupts.CPU64.LOC:Local_timer_interrupts
      4996 ±  2%     +13.1%       5652        interrupts.CPU64.RES:Rescheduling_interrupts
     46524 ±  3%     +21.2%      56403        interrupts.CPU65.CAL:Function_call_interrupts
      4935 ±  4%     +12.0%       5526 ±  2%  interrupts.CPU65.RES:Rescheduling_interrupts
     46844           +21.9%      57099        interrupts.CPU66.CAL:Function_call_interrupts
      5065 ±  2%     +12.4%       5693 ±  2%  interrupts.CPU66.RES:Rescheduling_interrupts
     47288           +23.5%      58398        interrupts.CPU67.CAL:Function_call_interrupts
     46924           +23.7%      58037        interrupts.CPU68.CAL:Function_call_interrupts
    240135 ±  3%     +10.9%     266219        interrupts.CPU68.LOC:Local_timer_interrupts
     46831           +22.4%      57306 ±  2%  interrupts.CPU69.CAL:Function_call_interrupts
      5284 ±  8%     +12.8%       5962 ±  8%  interrupts.CPU69.RES:Rescheduling_interrupts
     47306 ±  2%     +21.4%      57415 ±  2%  interrupts.CPU7.CAL:Function_call_interrupts
     46943           +22.0%      57289 ±  2%  interrupts.CPU70.CAL:Function_call_interrupts
      5191 ±  6%     +13.8%       5910 ±  7%  interrupts.CPU70.RES:Rescheduling_interrupts
     46789           +22.3%      57228        interrupts.CPU71.CAL:Function_call_interrupts
     46644           +22.6%      57176        interrupts.CPU72.CAL:Function_call_interrupts
      5083 ±  6%     +13.6%       5772 ±  7%  interrupts.CPU72.RES:Rescheduling_interrupts
     47318           +21.6%      57545 ±  2%  interrupts.CPU73.CAL:Function_call_interrupts
     46576           +23.7%      57601        interrupts.CPU74.CAL:Function_call_interrupts
      5187 ±  8%     +13.6%       5893 ±  9%  interrupts.CPU74.RES:Rescheduling_interrupts
     46476 ±  2%     +24.4%      57819        interrupts.CPU75.CAL:Function_call_interrupts
      5295 ±  7%     +13.1%       5990 ±  9%  interrupts.CPU75.RES:Rescheduling_interrupts
     46845           +23.0%      57618        interrupts.CPU76.CAL:Function_call_interrupts
     46773           +21.4%      56773 ±  2%  interrupts.CPU77.CAL:Function_call_interrupts
     46743           +23.1%      57553        interrupts.CPU78.CAL:Function_call_interrupts
     46593           +24.3%      57900        interrupts.CPU79.CAL:Function_call_interrupts
     46816 ±  2%     +21.1%      56703        interrupts.CPU8.CAL:Function_call_interrupts
     46669           +23.2%      57506        interrupts.CPU80.CAL:Function_call_interrupts
     46906           +23.0%      57695 ±  2%  interrupts.CPU81.CAL:Function_call_interrupts
     46704           +23.4%      57627        interrupts.CPU82.CAL:Function_call_interrupts
     47025           +23.9%      58277        interrupts.CPU83.CAL:Function_call_interrupts
     46607           +23.9%      57738        interrupts.CPU84.CAL:Function_call_interrupts
     46541           +23.5%      57476        interrupts.CPU85.CAL:Function_call_interrupts
     46333           +24.0%      57475        interrupts.CPU86.CAL:Function_call_interrupts
     44498 ±  5%     +23.5%      54948 ±  3%  interrupts.CPU87.CAL:Function_call_interrupts
     47148           +20.6%      56851        interrupts.CPU9.CAL:Function_call_interrupts
      4974 ±  3%     +13.5%       5644        interrupts.CPU9.RES:Rescheduling_interrupts
     37.19            -5.2       31.95        perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     22.38            -3.5       18.91        perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
     13.74            -3.2       10.52 ±  2%  perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      9.28 ±  2%      -2.8        6.45        perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      9.17 ±  2%      -2.8        6.36        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      9.02            -2.3        6.75 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
      8.99            -2.3        6.73 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
     14.80            -1.8       13.03 ±  2%  perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
     11.26            -1.6        9.62        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn
      2.68 ±  2%      -1.1        1.63        perf-profile.calltrace.cycles-pp.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      2.60 ±  3%      -1.0        1.56        perf-profile.calltrace.cycles-pp.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write
      2.61 ±  3%      -1.0        1.58        perf-profile.calltrace.cycles-pp.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      4.69            -0.9        3.74 ±  2%  perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
      4.54            -0.9        3.62 ±  2%  perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      4.52            -0.9        3.60 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_file_fsync
      2.11 ±  2%      -0.8        1.28 ±  2%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
      2.09 ±  2%      -0.8        1.27 ±  2%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
      7.01            -0.8        6.25        perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      6.80            -0.7        6.06        perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      6.78            -0.7        6.04        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn
      2.79 ±  4%      -0.7        2.06 ±  3%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      2.65 ±  4%      -0.7        1.94 ±  3%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
      2.64 ±  4%      -0.7        1.93 ±  3%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
      2.63 ±  4%      -0.7        1.93 ±  3%  perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
      3.32            -0.6        2.72        perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      3.31            -0.6        2.71        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      3.30            -0.6        2.70        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn
      1.45 ±  6%      -0.5        0.91 ±  5%  perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
      0.79 ±  7%      -0.5        0.27 ±100%  perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
      0.76 ±  6%      -0.5        0.26 ±100%  perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_submit_bio.submit_bio_noacct.submit_bio.iomap_submit_ioend
      1.17 ±  7%      -0.5        0.69 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
     25.06            -0.5       24.58        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      1.13 ±  8%      -0.5        0.66 ±  7%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit
     24.75            -0.4       24.30        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     24.75            -0.4       24.30        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     24.73            -0.4       24.29        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     23.10            -0.4       22.71        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     23.13            -0.4       22.74        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.54 ±  3%      -0.4        1.16 ±  2%  perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      1.53 ±  3%      -0.4        1.16 ±  2%  perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
      1.50 ±  3%      -0.4        1.12 ±  3%  perf-profile.calltrace.cycles-pp.submit_bio_noacct.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
      3.19 ±  4%      -0.4        2.82 ±  2%  perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      1.28 ±  3%      -0.3        0.94 ±  3%  perf-profile.calltrace.cycles-pp.brd_submit_bio.submit_bio_noacct.submit_bio.iomap_submit_ioend.xfs_vm_writepages
      1.08 ±  6%      -0.3        0.75 ±  4%  perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      1.07 ±  6%      -0.3        0.75 ±  4%  perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
      0.94 ±  6%      -0.3        0.64 ±  5%  perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
     22.40            -0.3       22.11        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      2.28 ±  6%      -0.3        2.00 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      2.20 ±  6%      -0.3        1.93 ±  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
      9.54            -0.2        9.29        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      9.57            -0.2        9.32        perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      9.51            -0.2        9.26        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
      1.14 ±  2%      -0.2        0.92 ±  3%  perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
      1.14 ±  2%      -0.2        0.92 ±  3%  perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
      0.83 ±  2%      -0.1        0.69 ±  2%  perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
      0.90 ±  2%      -0.1        0.78        perf-profile.calltrace.cycles-pp.xlog_ioend_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.66            -0.1        0.56 ±  2%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      0.66            -0.1        0.56 ±  2%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      4.57            -0.1        4.49        perf-profile.calltrace.cycles-pp.ret_from_fork
      4.57            -0.1        4.49        perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.67 ±  2%      -0.1        0.60        perf-profile.calltrace.cycles-pp.xlog_state_do_callback.xlog_ioend_work.process_one_work.worker_thread.kthread
      0.57            -0.1        0.52        perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      0.57            -0.0        0.53        perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
      0.58            +0.0        0.61        perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread
     69.91            +0.6       70.49        perf-profile.calltrace.cycles-pp.write
     69.73            +0.6       70.32        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     69.66            +0.6       70.26        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     69.66            +0.6       70.25        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     69.64            +0.6       70.24        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     69.56            +0.6       70.16        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     69.52            +0.6       70.13        perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.00            +0.7        0.73        perf-profile.calltrace.cycles-pp.md_submit_flush_data.process_one_work.worker_thread.kthread.ret_from_fork
     66.13            +1.8       67.90        perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     11.26           +10.6       21.82 ±  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.md_flush_request.raid0_make_request.md_handle_request
     11.35           +10.6       21.91 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.md_flush_request.raid0_make_request.md_handle_request.md_submit_bio
     12.26           +11.0       23.22 ±  2%  perf-profile.calltrace.cycles-pp.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
     12.22           +11.0       23.18 ±  2%  perf-profile.calltrace.cycles-pp.submit_bio_noacct.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync
     12.33           +11.0       23.29 ±  2%  perf-profile.calltrace.cycles-pp.blkdev_issue_flush.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     12.22           +11.0       23.18 ±  2%  perf-profile.calltrace.cycles-pp.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_buffered_aio_write
     12.17           +11.0       23.14 ±  2%  perf-profile.calltrace.cycles-pp.md_submit_bio.submit_bio_noacct.submit_bio.submit_bio_wait.blkdev_issue_flush
     12.10           +11.0       23.08 ±  2%  perf-profile.calltrace.cycles-pp.md_handle_request.md_submit_bio.submit_bio_noacct.submit_bio.submit_bio_wait
     12.01           +11.0       22.98 ±  2%  perf-profile.calltrace.cycles-pp.md_flush_request.raid0_make_request.md_handle_request.md_submit_bio.submit_bio_noacct
     12.02           +11.0       23.02 ±  2%  perf-profile.calltrace.cycles-pp.raid0_make_request.md_handle_request.md_submit_bio.submit_bio_noacct.submit_bio
     24.91            -6.1       18.77 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock
     37.19            -5.2       31.95        perf-profile.children.cycles-pp.xfs_log_force_lsn
     28.54            -5.0       23.55 ±  2%  perf-profile.children.cycles-pp.__xfs_log_force_lsn
     22.38            -3.5       18.91        perf-profile.children.cycles-pp.xlog_cil_force_lsn
     26.49            -3.0       23.50        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     24.23            -2.5       21.72        perf-profile.children.cycles-pp.remove_wait_queue
     11.70            -1.7        9.99        perf-profile.children.cycles-pp.xlog_wait_on_iclog
      2.68 ±  2%      -1.1        1.63        perf-profile.children.cycles-pp.xfs_file_aio_write_checks
      2.60 ±  2%      -1.0        1.56        perf-profile.children.cycles-pp.xfs_vn_update_time
      2.61 ±  3%      -1.0        1.58        perf-profile.children.cycles-pp.file_update_time
      2.63 ±  2%      -0.9        1.72 ±  2%  perf-profile.children.cycles-pp.__xfs_trans_commit
      2.60 ±  2%      -0.9        1.69 ±  2%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      2.79 ±  4%      -0.7        2.06 ±  3%  perf-profile.children.cycles-pp.file_write_and_wait_range
      2.65 ±  4%      -0.7        1.94 ±  3%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      2.64 ±  4%      -0.7        1.93 ±  3%  perf-profile.children.cycles-pp.do_writepages
      2.63 ±  4%      -0.7        1.93 ±  3%  perf-profile.children.cycles-pp.xfs_vm_writepages
      1.80 ±  6%      -0.6        1.21 ±  6%  perf-profile.children.cycles-pp.xlog_cil_insert_items
     25.06            -0.5       24.58        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     25.06            -0.5       24.58        perf-profile.children.cycles-pp.cpu_startup_entry
     25.05            -0.5       24.57        perf-profile.children.cycles-pp.do_idle
     24.75            -0.4       24.30        perf-profile.children.cycles-pp.start_secondary
     23.42            -0.4       23.00        perf-profile.children.cycles-pp.cpuidle_enter
     23.42            -0.4       23.00        perf-profile.children.cycles-pp.cpuidle_enter_state
      1.54 ±  3%      -0.4        1.16 ±  2%  perf-profile.children.cycles-pp.iomap_submit_ioend
      3.19 ±  4%      -0.4        2.82 ±  2%  perf-profile.children.cycles-pp.__flush_work
      1.34 ±  3%      -0.3        0.99 ±  3%  perf-profile.children.cycles-pp.brd_submit_bio
      1.04 ±  7%      -0.3        0.69 ±  7%  perf-profile.children.cycles-pp.end_page_writeback
      1.06 ±  6%      -0.3        0.71 ±  7%  perf-profile.children.cycles-pp.iomap_finish_ioend
      0.94 ±  7%      -0.3        0.60 ±  7%  perf-profile.children.cycles-pp.test_clear_page_writeback
     22.69            -0.3       22.36        perf-profile.children.cycles-pp.intel_idle
      0.85 ±  8%      -0.3        0.53 ±  8%  perf-profile.children.cycles-pp.sb_clear_inode_writeback
      1.08 ±  6%      -0.3        0.75 ±  4%  perf-profile.children.cycles-pp.iomap_writepages
      1.07 ±  6%      -0.3        0.75 ±  4%  perf-profile.children.cycles-pp.write_cache_pages
      0.94 ±  6%      -0.3        0.64 ±  5%  perf-profile.children.cycles-pp.iomap_writepage_map
      0.79 ±  7%      -0.3        0.51 ±  6%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.72 ±  8%      -0.3        0.46 ±  6%  perf-profile.children.cycles-pp.sb_mark_inode_writeback
      0.49 ± 15%      -0.2        0.26 ± 14%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.44 ± 17%      -0.2        0.21 ± 15%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.45 ± 16%      -0.2        0.22 ± 15%  perf-profile.children.cycles-pp.xfs_trans_reserve
      1.14 ±  2%      -0.2        0.92 ±  3%  perf-profile.children.cycles-pp.xfs_end_io
      1.14 ±  2%      -0.2        0.92 ±  3%  perf-profile.children.cycles-pp.xfs_end_ioend
      0.32 ± 20%      -0.2        0.14 ± 21%  perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
      0.30 ± 21%      -0.2        0.13 ± 22%  perf-profile.children.cycles-pp.xlog_grant_add_space
      0.83 ±  2%      -0.1        0.69 ±  2%  perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
      0.90 ±  2%      -0.1        0.78        perf-profile.children.cycles-pp.xlog_ioend_work
      0.66            -0.1        0.56 ±  2%  perf-profile.children.cycles-pp.iomap_apply
      0.66            -0.1        0.56 ±  2%  perf-profile.children.cycles-pp.iomap_file_buffered_write
      0.53            -0.1        0.45 ±  2%  perf-profile.children.cycles-pp.iomap_write_actor
      0.84 ±  3%      -0.1        0.76        perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.10 ±  9%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      4.57            -0.1        4.49        perf-profile.children.cycles-pp.ret_from_fork
      4.57            -0.1        4.49        perf-profile.children.cycles-pp.kthread
      0.67 ±  2%      -0.1        0.60        perf-profile.children.cycles-pp.xlog_state_do_callback
      0.76 ±  2%      -0.1        0.68        perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.30 ±  7%      -0.1        0.23 ±  7%  perf-profile.children.cycles-pp.iomap_finish_ioends
      0.46 ±  2%      -0.1        0.40 ±  2%  perf-profile.children.cycles-pp.brd_do_bvec
      0.57            -0.1        0.52        perf-profile.children.cycles-pp.wait_for_completion
      0.26 ±  3%      -0.1        0.21 ±  2%  perf-profile.children.cycles-pp.ktime_get
      0.48            -0.0        0.43        perf-profile.children.cycles-pp.schedule_timeout
      0.58            -0.0        0.54        perf-profile.children.cycles-pp.complete
      0.43 ±  2%      -0.0        0.38 ±  3%  perf-profile.children.cycles-pp.xlog_state_clean_iclog
      0.23 ±  5%      -0.0        0.18 ±  5%  perf-profile.children.cycles-pp.xlog_state_done_syncing
      0.51 ±  3%      -0.0        0.47        perf-profile.children.cycles-pp.hrtimer_interrupt
      0.78            -0.0        0.73        perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      0.20 ±  2%      -0.0        0.16 ±  3%  perf-profile.children.cycles-pp.xfs_inode_item_format
      0.53            -0.0        0.49        perf-profile.children.cycles-pp.swake_up_locked
      0.16 ±  3%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.xlog_state_get_iclog_space
      0.21 ±  4%      -0.0        0.18 ±  3%  perf-profile.children.cycles-pp.iomap_write_begin
      0.19 ±  2%      -0.0        0.15 ±  4%  perf-profile.children.cycles-pp.poll_idle
      0.25 ±  2%      -0.0        0.22 ±  2%  perf-profile.children.cycles-pp.xfs_bmapi_write
      0.07 ± 10%      -0.0        0.04 ± 44%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.17 ±  2%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.iomap_write_end
      0.22 ±  3%      -0.0        0.19 ±  2%  perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
      0.11 ±  6%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.xfs_inode_item_format_data_fork
      0.10 ±  4%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.down_read
      0.15 ±  5%      -0.0        0.13 ±  5%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      0.14 ±  4%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.pagecache_get_page
      0.10 ±  8%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.xfs_iextents_copy
      0.38 ±  2%      -0.0        0.36 ±  2%  perf-profile.children.cycles-pp.menu_select
      0.21 ±  3%      -0.0        0.19        perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
      0.13 ±  3%      -0.0        0.11 ±  6%  perf-profile.children.cycles-pp.submit_bio_checks
      0.12 ±  4%      -0.0        0.10 ±  3%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      0.11 ±  4%      -0.0        0.09 ±  9%  perf-profile.children.cycles-pp.queue_work_on
      0.17 ±  5%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp.xlog_cil_process_committed
      0.17 ±  5%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp.xlog_cil_committed
      0.10 ±  5%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__queue_work
      0.08 ±  9%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.task_work_run
      0.10 ±  3%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      0.10 ±  4%      -0.0        0.09        perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      0.08 ±  5%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.find_get_entry
      0.09 ±  4%      -0.0        0.08        perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      0.08            -0.0        0.07        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.12 ±  3%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.set_task_cpu
      0.24            +0.0        0.26        perf-profile.children.cycles-pp.available_idle_cpu
      0.35            +0.0        0.37        perf-profile.children.cycles-pp.select_idle_sibling
      0.50 ±  2%      +0.0        0.53        perf-profile.children.cycles-pp.update_load_avg
      0.59            +0.0        0.62        perf-profile.children.cycles-pp.xlog_write
      0.47            +0.0        0.50        perf-profile.children.cycles-pp.select_task_rq_fair
      1.12            +0.1        1.19        perf-profile.children.cycles-pp.__wake_up_common
      0.28            +0.1        0.39        perf-profile.children.cycles-pp.xlog_state_release_iclog
      0.23 ±  2%      +0.2        0.40        perf-profile.children.cycles-pp.autoremove_wake_function
      0.19 ±  3%      +0.3        0.48 ±  2%  perf-profile.children.cycles-pp.prepare_to_wait_event
      0.31 ±  2%      +0.4        0.73        perf-profile.children.cycles-pp.md_submit_flush_data
     69.95            +0.6       70.52        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     69.94            +0.6       70.52        perf-profile.children.cycles-pp.write
     69.80            +0.6       70.39        perf-profile.children.cycles-pp.do_syscall_64
     69.66            +0.6       70.26        perf-profile.children.cycles-pp.ksys_write
     69.65            +0.6       70.25        perf-profile.children.cycles-pp.vfs_write
     69.56            +0.6       70.17        perf-profile.children.cycles-pp.new_sync_write
     69.52            +0.6       70.14        perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     64.09            +1.6       65.72        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     66.13            +1.8       67.90        perf-profile.children.cycles-pp.xfs_file_fsync
     14.12 ±  2%     +10.6       24.75        perf-profile.children.cycles-pp._raw_spin_lock_irq
     13.90           +10.7       24.60 ±  2%  perf-profile.children.cycles-pp.submit_bio_noacct
     13.87           +10.7       24.58 ±  2%  perf-profile.children.cycles-pp.submit_bio
     12.26           +11.0       23.22 ±  2%  perf-profile.children.cycles-pp.submit_bio_wait
     12.33           +11.0       23.29 ±  2%  perf-profile.children.cycles-pp.blkdev_issue_flush
     12.42           +11.1       23.50 ±  2%  perf-profile.children.cycles-pp.md_submit_bio
     12.32           +11.1       23.41 ±  2%  perf-profile.children.cycles-pp.md_handle_request
     12.20           +11.1       23.31 ±  2%  perf-profile.children.cycles-pp.raid0_make_request
     12.08           +11.1       23.20 ±  2%  perf-profile.children.cycles-pp.md_flush_request
     22.69            -0.3       22.36        perf-profile.self.cycles-pp.intel_idle
      0.30 ± 21%      -0.2        0.12 ± 21%  perf-profile.self.cycles-pp.xlog_grant_add_space
      0.27 ± 21%      -0.1        0.12 ± 21%  perf-profile.self.cycles-pp.xfs_log_ticket_ungrant
      0.54            -0.1        0.46 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.37 ±  2%      -0.0        0.33 ±  2%  perf-profile.self.cycles-pp.brd_do_bvec
      0.20 ±  4%      -0.0        0.15 ±  3%  perf-profile.self.cycles-pp.ktime_get
      0.13 ±  5%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.58 ±  2%      -0.0        0.55        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.18 ±  2%      -0.0        0.14 ±  5%  perf-profile.self.cycles-pp.poll_idle
      0.12 ±  8%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.md_handle_request
      0.10 ±  7%      -0.0        0.07 ±  9%  perf-profile.self.cycles-pp.down_read
      0.07 ±  5%      -0.0        0.05 ±  7%  perf-profile.self.cycles-pp.xlog_cil_insert_items
      0.10 ±  3%      -0.0        0.08 ±  4%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      0.06            -0.0        0.05        perf-profile.self.cycles-pp.__wake_up_common
      0.08            -0.0        0.07        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.24 ±  2%      +0.0        0.26        perf-profile.self.cycles-pp.available_idle_cpu
      0.04 ± 44%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.md_flush_request
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.prepare_to_wait_event
     63.87            +1.7       65.54        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  16000 +-------------------------------------------------------------------+   
        |   +                    +  +        +        +  +          +       |   
  15500 |+++:                    :  :      + :        :  :       +  :       |   
        |    :                   :  :      : ::       :  :  +    :  ::      |   
        |    :              ++ +: :: : +++: : : +    : :: : :+  + :: : +    |   
  15000 |-+  +++++++++++ + +  + + :+ :+   + + ++ +++++ :+ :+  ++  ++ ++ ++++|   
        |               + +       +  +                 +  +                 |   
  14500 |-+                                                                 |   
        |                                                                   |   
  14000 |-+                                                                 |   
        |                       O     O                                     |   
        |OOOOOOO OOO O O O OOOOO OOOOO OOOOO                                |   
  13500 |-+     O   O O O O                                                 |   
        |                                                                   |   
  13000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               aim7.time.system_time                            
                                                                                
  5800 +--------------------------------------------------------------------+   
       |           O   O  O                                                 |   
  5600 |OOOOOOOOOOO OOO OO OOOOO OOOOO OOOO                                 |   
  5400 |-+                      O     O                                     |   
       |                                                                    |   
  5200 |-+                                                                  |   
  5000 |-+                                                                  |   
       |    +             +       +  +                +  +                  |   
  4800 |-+  :+++++++++++++ ++++ + :+ :+++ ++ ++++++++ :+ :++ +++  ++ +++++++|   
  4600 |-+  :                  + :  :    +::::       :  :   +   ++  ::      |   
       |    :                    +  +     + +        +  +           +       |   
  4400 |-+ :                                                                |   
  4200 |++ :                                                                |   
       |  ++                                                                |   
  4000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  134 +---------------------------------------------------------------------+   
  132 |-OOO OO OOO  OO OO OOOOO OOOOO OOOOO                                 |   
      |O   O       O           O     O                                      |   
  130 |-+                                                                   |   
  128 |-+                                                                   |   
  126 |-+                                                                   |   
  124 |-+                                                                   |   
      |                  +                                                  |   
  122 |-+  ++  +  +  +++ ::    + ++ ++                ++ +++       +        |   
  120 |-+  : ++ ++ ++   + ++++ : :: : ++ + + ++++++++ :: :  :+++  +: +++++++|   
  118 |-+  :                  +: :: :   +: : :      : :: :  +   + :: :      |   
  116 |-+ :                     :  :      : ::       :  :        +  ::      |   
      |+ ++                     :  :      + :        :  :           :       |   
  114 |-+                       +  +        +        +  +           +       |   
  112 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  134 +---------------------------------------------------------------------+   
  132 |-OOO OO OOO  OO OO OOOOO OOOOO OOOOO                                 |   
      |O   O       O           O     O                                      |   
  130 |-+                                                                   |   
  128 |-+                                                                   |   
  126 |-+                                                                   |   
  124 |-+                                                                   |   
      |                  +                                                  |   
  122 |-+  ++  +  +  +++ ::    + ++ ++                ++ +++       +        |   
  120 |-+  : ++ ++ ++   + ++++ : :: : ++ + + ++++++++ :: :  :+++  +: +++++++|   
  118 |-+  :                  +: :: :   +: : :      : :: :  +   + :: :      |   
  116 |-+ :                     :  :      : ::       :  :        +  ::      |   
      |+ ++                     :  :      + :        :  :           :       |   
  114 |-+                       +  +        +        +  +           +       |   
  112 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         aim7.time.voluntary_context_switches                   
                                                                                
  7.4e+07 +-----------------------------------------------------------------+   
          |OOOOOOOOOOOOOOOOOOOOOOO OOOO OOOOO                               |   
  7.2e+07 |-+                     O    O                                    |   
          |                                                                 |   
    7e+07 |-+                                                               |   
          |                                                                 |   
  6.8e+07 |-+                                                               |   
          |                                                                 |   
  6.6e+07 |-+                                                               |   
          |    +  +      +  +                                               |   
  6.4e+07 |-+  :+++++++++ ++ ++++ +++ ++++ + + +++++++ ++ +++ +++  +++++++++|   
          |    :                 +::: :   + :: :     : :: :  +   + :::      |   
  6.2e+07 |++ :                    : :      + :       :  :        + :       |   
          |  ++                    + +        +       +  +          +       |   
    6e+07 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        aim7.time.involuntary_context_switches                  
                                                                                
  2.4e+06 +-----------------------------------------------------------------+   
          |++++                                                             |   
  2.3e+06 |-+ :                                                             |   
          |   :                                                             |   
  2.2e+06 |-+ :                                                             |   
  2.1e+06 |-+  :                                                            |   
          |    :                                                            |   
    2e+06 |-+  :                                                            |   
          |    :                                                            |   
  1.9e+06 |-+  + ++++ ++++++ ++++ ++++++++++++ +++++++++++++++++++  ++++++++|   
  1.8e+06 |-+   + +  +      +    +            +                   +++       |   
          |                                                                 |   
  1.7e+06 |-O    OO O O        OOOOOO OOOOO O                               |   
          |O OOOO  O O OOOOOOOO      O     O                                |   
  1.6e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.11.0-rc1-00035-gdc5d17a3c39b" of type "text/plain" (172412 bytes)

View attachment "job-script" of type "text/plain" (8039 bytes)

View attachment "job.yaml" of type "text/plain" (5485 bytes)

View attachment "reproduce" of type "text/plain" (1034 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ