lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Thu, 24 Dec 2020 11:10:32 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Goldwyn Rodrigues <rgoldwyn@...e.de>
Cc:     0day robot <lkp@...el.com>, LKML <linux-kernel@...r.kernel.org>,
        lkp@...ts.01.org, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com, linux-btrfs@...r.kernel.org,
        Goldwyn Rodrigues <rgoldwyn@...e.com>
Subject: [btrfs]  e7a8dd2d95:  fio.write_iops -98.3% regression


Greeting,

FYI, we noticed a -98.3% regression of fio.write_iops due to commit:


commit: e7a8dd2d9537a7ec5aeadb002fc4934be1a396eb ("[PATCH v3] btrfs: Make btrfs_direct_write atomic with respect to inode_lock")
url: https://github.com/0day-ci/linux/commits/Goldwyn-Rodrigues/btrfs-Make-btrfs_direct_write-atomic-with-respect-to-inode_lock/20201219-001114
base: https://git.kernel.org/cgit/linux/kernel/git/kdave/linux.git for-next

in testcase: fio-basic
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
with following parameters:

	runtime: 300s
	disk: 1HDD
	fs: btrfs
	nr_task: 100%
	test_size: 128G
	rw: write
	bs: 4k
	ioengine: sync
	direct: direct
	cpufreq_governor: performance
	ucode: 0x42e

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/direct/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
  4k/gcc-9/performance/direct/1HDD/btrfs/sync/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/write/lkp-ivb-2ep1/128G/fio-basic/0x42e

commit: 
  97847e0652 ("Merge branch 'for-next-next-v5.10-20201211' into for-next-20201211")
  e7a8dd2d95 ("btrfs: Make btrfs_direct_write atomic with respect to inode_lock")

97847e06525b51ea e7a8dd2d9537a7ec5aeadb002fc 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4           -4%           0:4     perf-profile.children.cycles-pp.error_return
          0:4            5%           0:4     perf-profile.children.cycles-pp.error_entry
          0:4            5%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      0.02 ±  3%      +0.1        0.12 ± 13%  fio.latency_1000ms%
      0.01            -0.0        0.00        fio.latency_1000us%
      0.96 ± 13%      -0.8        0.19 ±101%  fio.latency_100ms%
     31.35           -31.4        0.00        fio.latency_10ms%
      8.28 ±  2%      -8.3        0.00        fio.latency_20ms%
      0.04 ± 29%      -0.0        0.02 ±100%  fio.latency_250ms%
      0.01            -0.0        0.00        fio.latency_2ms%
     54.35           -54.3        0.00        fio.latency_4ms%
      0.02 ± 10%     +99.6       99.66        fio.latency_500ms%
      0.01            -0.0        0.00        fio.latency_500us%
      4.97            -5.0        0.00        fio.latency_50ms%
      0.01            -0.0        0.00        fio.latency_750us%
  17301324           -93.9%    1062134 ±  2%  fio.time.file_system_outputs
    316.50 ± 45%    +583.7%       2164 ± 14%  fio.time.involuntary_context_switches
     23.75           -63.2%       8.75 ±  4%  fio.time.percent_of_cpu_this_job_got
     62.17           -58.4%      25.87 ±  2%  fio.time.system_time
     10.95 ±  2%     -83.7%       1.78 ±  9%  fio.time.user_time
   2194671           -93.0%     153935        fio.time.voluntary_context_switches
   2162665           -98.3%      36777        fio.workload
     28.16           -98.3%       0.48        fio.write_bw_MBps
  12156928         +3384.6%  4.236e+08        fio.write_clat_90%_us
  23855104 ±  2%   +1715.4%  4.331e+08        fio.write_clat_95%_us
  50724864 ±  4%    +797.2%  4.551e+08        fio.write_clat_99%_us
   6657995         +5784.2%  3.918e+08        fio.write_clat_mean_us
  15787433 ±  2%    +110.1%   33164741 ±  9%  fio.write_clat_stddev
      7208           -98.3%     122.53        fio.write_iops
     10477           +39.4%      14609        uptime.idle
     18.03 ± 15%     -25.7%      13.39        boot-time.dhcp
      1017 ± 11%     -23.4%     779.50        boot-time.idle
     64.67           +45.5%      94.08        iostat.cpu.idle
     32.22 ±  2%     -92.9%       2.28 ±  3%  iostat.cpu.iowait
    460106 ± 33%     +56.7%     720959 ± 15%  numa-numastat.node0.local_node
    461015 ± 33%     +61.1%     742788 ± 15%  numa-numastat.node0.numa_hit
    598719 ± 25%     -43.3%     339612 ± 34%  numa-numastat.node1.local_node
    639421 ± 24%     -43.4%     361729 ± 31%  numa-numastat.node1.numa_hit
     61123 ± 15%     -82.4%      10785        meminfo.Active(anon)
     10906 ±  2%    +490.4%      64393 ± 15%  meminfo.Active(file)
      1678           -85.8%     238.00 ± 14%  meminfo.Dirty
     31700          +351.3%     143073 ±  5%  meminfo.Inactive(file)
    283671           -52.2%     135570        meminfo.SUnreclaim
    359419           -41.1%     211861        meminfo.Slab
     64.47           +29.6       94.08        mpstat.cpu.all.idle%
     32.42 ±  2%     -30.1        2.29 ±  3%  mpstat.cpu.all.iowait%
      2.04 ±  9%      +1.0        3.08 ± 12%  mpstat.cpu.all.irq%
      0.23 ±  6%      -0.0        0.19 ±  2%  mpstat.cpu.all.soft%
      0.66 ±  2%      -0.4        0.25 ±  3%  mpstat.cpu.all.sys%
      0.17            -0.1        0.11 ±  2%  mpstat.cpu.all.usr%
     64.25           +45.1%      93.25        vmstat.cpu.id
     31.75 ±  2%     -95.3%       1.50 ± 33%  vmstat.cpu.wa
     28682           -89.7%       2962 ±  2%  vmstat.io.bo
     46.50           -98.4%       0.75 ± 57%  vmstat.procs.b
     38969           -92.5%       2923        vmstat.system.cs
    100537            -4.5%      96061        vmstat.system.in
 1.306e+09 ±  7%     -99.1%   11707061 ± 18%  cpuidle.C1.time
   5821744 ±  7%     -96.9%     182330 ± 10%  cpuidle.C1.usage
 6.258e+09 ±  9%     -59.8%  2.514e+09 ±  9%  cpuidle.C1E.time
  18768783 ±  4%     -29.4%   13255968 ± 26%  cpuidle.C1E.usage
 6.397e+09 ±  9%     +77.6%  1.136e+10 ±  2%  cpuidle.C6.time
   9966359 ±  7%     +63.3%   16279745 ± 22%  cpuidle.C6.usage
   5156525 ± 14%     -99.2%      40443 ± 13%  cpuidle.POLL.time
   3217284 ± 17%     -99.4%      19436 ± 12%  cpuidle.POLL.usage
     12395 ±  7%    +201.0%      37310 ± 23%  numa-meminfo.node0.Active
     10314 ±  6%    +235.7%      34621 ± 21%  numa-meminfo.node0.Active(file)
      1636           -91.5%     138.75 ± 16%  numa-meminfo.node0.Dirty
     24039 ± 54%    +264.0%      87496 ±  4%  numa-meminfo.node0.Inactive(file)
    219811 ±  4%     -68.3%      69676 ±  4%  numa-meminfo.node0.SUnreclaim
    258497 ±  4%     -57.4%     110004 ±  6%  numa-meminfo.node0.Slab
     59798 ± 14%     -36.6%      37897 ±  6%  numa-meminfo.node1.Active
     59211 ± 15%     -86.3%       8123 ± 15%  numa-meminfo.node1.Active(anon)
    587.00 ± 72%   +4972.2%      29774 ± 11%  numa-meminfo.node1.Active(file)
    183687 ± 20%     -31.4%     126043 ± 15%  numa-meminfo.node1.AnonPages.max
      7665 ±171%    +624.8%      55557 ±  9%  numa-meminfo.node1.Inactive(file)
    139520 ±120%     -81.7%      25573 ±  4%  sched_debug.cfs_rq:/.load.avg
     50.14 ± 19%     -21.8%      39.22 ±  7%  sched_debug.cfs_rq:/.load_avg.avg
      0.11 ±  6%     -22.2%       0.09 ±  7%  sched_debug.cfs_rq:/.nr_running.avg
      0.29 ±  5%     -15.0%       0.25        sched_debug.cfs_rq:/.nr_running.stddev
    122.82 ±  4%     -16.2%     102.93 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
    122.46 ±  4%     -16.5%     102.27 ±  5%  sched_debug.cfs_rq:/.util_avg.avg
    101814 ± 11%    +277.7%     384526 ± 27%  sched_debug.cpu.avg_idle.min
    220884 ±  5%     -37.6%     137900 ± 13%  sched_debug.cpu.avg_idle.stddev
      1.10 ±  4%     +17.5%       1.30 ±  3%  sched_debug.cpu.clock.stddev
      1669 ±  4%     -63.3%     612.98 ±  6%  sched_debug.cpu.clock_task.stddev
      5275 ±  6%     +10.7%       5839        sched_debug.cpu.curr->pid.max
    106300 ± 10%     -89.3%      11398        sched_debug.cpu.nr_switches.avg
   1209385 ± 11%     -97.4%      31808 ± 14%  sched_debug.cpu.nr_switches.max
    192228 ± 11%     -97.3%       5222 ± 18%  sched_debug.cpu.nr_switches.stddev
   -251.12           -86.0%     -35.25        sched_debug.cpu.nr_uninterruptible.min
     40.50 ± 14%     -64.2%      14.48 ± 24%  sched_debug.cpu.nr_uninterruptible.stddev
      2577 ±  6%    +235.7%       8653 ± 21%  numa-vmstat.node0.nr_active_file
     10194 ± 42%    +216.4%      32253 ±  5%  numa-vmstat.node0.nr_dirtied
    409.50           -91.6%      34.50 ± 15%  numa-vmstat.node0.nr_dirty
      6009 ± 54%    +264.0%      21873 ±  4%  numa-vmstat.node0.nr_inactive_file
     54951 ±  4%     -68.3%      17418 ±  4%  numa-vmstat.node0.nr_slab_unreclaimable
      9778 ± 43%    +229.6%      32228 ±  5%  numa-vmstat.node0.nr_written
      2577 ±  6%    +235.7%       8653 ± 21%  numa-vmstat.node0.nr_zone_active_file
      6009 ± 54%    +264.0%      21873 ±  4%  numa-vmstat.node0.nr_zone_inactive_file
    414.75 ±  2%     -92.5%      31.00 ± 13%  numa-vmstat.node0.nr_zone_write_pending
     14804 ± 15%     -86.3%       2027 ± 15%  numa-vmstat.node1.nr_active_anon
    146.75 ± 72%   +4970.4%       7440 ± 10%  numa-vmstat.node1.nr_active_file
      2882 ±147%    +676.1%      22371 ±  5%  numa-vmstat.node1.nr_dirtied
      1915 ±171%    +625.1%      13888 ±  9%  numa-vmstat.node1.nr_inactive_file
      2871 ±147%    +678.4%      22351 ±  5%  numa-vmstat.node1.nr_written
     14804 ± 15%     -86.3%       2027 ± 15%  numa-vmstat.node1.nr_zone_active_anon
    146.75 ± 72%   +4970.4%       7440 ± 10%  numa-vmstat.node1.nr_zone_active_file
      1915 ±171%    +625.1%      13888 ±  9%  numa-vmstat.node1.nr_zone_inactive_file
     15380 ± 15%     -82.4%       2703        proc-vmstat.nr_active_anon
      2725 ±  2%    +490.4%      16094 ± 16%  proc-vmstat.nr_active_file
     17429          +471.3%      99580 ±  2%  proc-vmstat.nr_dirtied
    416.75           -86.0%      58.25 ± 13%  proc-vmstat.nr_dirty
    567440            +4.9%     595244        proc-vmstat.nr_file_pages
      7926          +351.0%      35749 ±  5%  proc-vmstat.nr_inactive_file
     12783            +1.7%      13005        proc-vmstat.nr_kernel_stack
    297875            -4.5%     284331        proc-vmstat.nr_shmem
     70910           -52.2%      33895        proc-vmstat.nr_slab_unreclaimable
     16905          +488.8%      99538 ±  2%  proc-vmstat.nr_written
     15380 ± 15%     -82.4%       2703        proc-vmstat.nr_zone_active_anon
      2725 ±  2%    +490.4%      16094 ± 16%  proc-vmstat.nr_zone_active_file
      7926          +351.0%      35749 ±  5%  proc-vmstat.nr_zone_inactive_file
    424.75           -87.9%      51.50 ±  9%  proc-vmstat.nr_zone_write_pending
     42826 ± 14%     -40.5%      25476 ±  3%  proc-vmstat.pgactivate
   1239967            -3.7%    1194026        proc-vmstat.pgalloc_normal
   8740013           -89.7%     904228 ±  2%  proc-vmstat.pgpgout
     53472 ±  3%      +5.6%      56482        proc-vmstat.pgreuse
    158.00 ± 12%    +348.6%     708.75 ±  2%  slabinfo.biovec-128.active_objs
    158.00 ± 12%    +348.6%     708.75 ±  2%  slabinfo.biovec-128.num_objs
    374.25 ±  8%    +258.8%       1342 ±  2%  slabinfo.biovec-64.active_objs
    374.25 ±  8%    +258.8%       1342 ±  2%  slabinfo.biovec-64.num_objs
    163.50 ± 13%    +227.8%     536.00 ±  7%  slabinfo.biovec-max.active_objs
    204.75 ± 13%    +168.7%     550.25 ±  7%  slabinfo.biovec-max.num_objs
      1297 ± 14%     -34.9%     845.25 ±  6%  slabinfo.btrfs_delayed_tree_ref.active_objs
      1297 ± 14%     -34.9%     845.25 ±  6%  slabinfo.btrfs_delayed_tree_ref.num_objs
      2006 ± 13%     +59.9%       3209        slabinfo.btrfs_extent_buffer.active_objs
      2006 ± 13%     +59.9%       3209        slabinfo.btrfs_extent_buffer.num_objs
    336.00 ± 23%     -31.2%     231.00 ± 15%  slabinfo.kmalloc-rcl-192.active_objs
    336.00 ± 23%     -31.2%     231.00 ± 15%  slabinfo.kmalloc-rcl-192.num_objs
   1079929           -99.6%       4015        slabinfo.pid_namespace.active_objs
     19284           -99.6%      71.25        slabinfo.pid_namespace.active_slabs
   1079934           -99.6%       4015        slabinfo.pid_namespace.num_objs
     19284           -99.6%      71.25        slabinfo.pid_namespace.num_slabs
    681.25 ± 10%    +233.8%       2274 ± 12%  slabinfo.skbuff_fclone_cache.active_objs
    681.25 ± 10%    +233.8%       2274 ± 12%  slabinfo.skbuff_fclone_cache.num_objs
    705679 ±  2%     -96.8%      22805        softirqs.BLOCK
     56594 ± 12%     -36.3%      36023 ± 19%  softirqs.CPU0.RCU
     61813 ± 12%     -41.2%      36360 ± 13%  softirqs.CPU1.RCU
     61170 ± 12%     -41.9%      35511 ±  4%  softirqs.CPU10.RCU
     59336 ± 10%     -37.7%      36972 ±  5%  softirqs.CPU11.RCU
     58606 ± 14%     -36.7%      37114 ± 17%  softirqs.CPU2.RCU
     54177 ± 12%     -40.1%      32471 ± 12%  softirqs.CPU24.RCU
     55834 ± 11%     -36.9%      35230 ± 10%  softirqs.CPU25.RCU
     53216 ±  9%     -31.9%      36262 ± 15%  softirqs.CPU26.RCU
     53744 ± 12%     -35.7%      34533 ± 16%  softirqs.CPU27.RCU
     51611 ±  8%     -33.8%      34181 ± 13%  softirqs.CPU28.RCU
     53726 ±  7%     -33.5%      35754 ±  9%  softirqs.CPU29.RCU
     61015 ± 14%     -41.3%      35804 ± 18%  softirqs.CPU3.RCU
     54715 ± 12%     -37.5%      34188 ±  3%  softirqs.CPU30.RCU
     51435 ± 10%     -33.5%      34224 ±  7%  softirqs.CPU31.RCU
     52697 ±  8%     -38.8%      32238 ±  5%  softirqs.CPU32.RCU
     27579 ±  7%     +49.8%      41304 ±  3%  softirqs.CPU32.SCHED
     48705 ±  9%     -32.2%      33031 ±  4%  softirqs.CPU33.RCU
     48234 ±  8%     -29.0%      34223 ±  3%  softirqs.CPU34.RCU
     49281 ± 10%     -29.7%      34654 ±  3%  softirqs.CPU35.RCU
     54023 ±  8%     -35.2%      34993 ± 19%  softirqs.CPU4.RCU
     55222 ± 26%     -34.4%      36234 ± 19%  softirqs.CPU40.RCU
     59824 ±  8%     -37.8%      37225 ± 14%  softirqs.CPU5.RCU
     62723 ± 13%     -41.6%      36615 ±  9%  softirqs.CPU6.RCU
     56745 ± 13%     -33.6%      37668 ± 11%  softirqs.CPU7.RCU
    685836           -98.4%      10922 ±  2%  softirqs.CPU8.BLOCK
     63989 ±  9%     -41.2%      37647 ±  5%  softirqs.CPU8.RCU
    693875           -96.6%      23342        softirqs.CPU8.TASKLET
     60701 ± 10%     -41.2%      35709 ±  9%  softirqs.CPU9.RCU
   2511282 ± 18%     -31.4%    1721969 ± 10%  softirqs.RCU
    694098           -96.6%      23565        softirqs.TASKLET
      2089 ± 62%     -80.3%     410.75 ± 91%  interrupts.41:PCI-MSI.2621447-edge.eth0-TxRx-6
   1385835           -96.8%      44728        interrupts.73:PCI-MSI.2097152-edge.isci-msix
    774.25 ± 60%     -87.9%      94.00 ± 35%  interrupts.CPU1.RES:Rescheduling_interrupts
      1167 ± 14%     -85.9%     164.50 ± 63%  interrupts.CPU10.RES:Rescheduling_interrupts
      1041 ± 18%     -82.2%     185.25 ± 48%  interrupts.CPU11.RES:Rescheduling_interrupts
    188.00 ± 23%     +32.0%     248.25 ± 26%  interrupts.CPU14.NMI:Non-maskable_interrupts
    188.00 ± 23%     +32.0%     248.25 ± 26%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
    113.75 ± 29%     -44.2%      63.50 ± 15%  interrupts.CPU16.RES:Rescheduling_interrupts
    470.50 ± 38%     -83.7%      76.50 ± 39%  interrupts.CPU2.RES:Rescheduling_interrupts
    880.75 ± 24%     -87.1%     113.75 ± 54%  interrupts.CPU24.RES:Rescheduling_interrupts
    320.50 ± 27%     -37.6%     200.00 ± 28%  interrupts.CPU25.NMI:Non-maskable_interrupts
    320.50 ± 27%     -37.6%     200.00 ± 28%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    640.00 ± 28%     -85.5%      92.50 ± 59%  interrupts.CPU25.RES:Rescheduling_interrupts
    572.00 ± 30%     -85.7%      81.75 ± 42%  interrupts.CPU26.RES:Rescheduling_interrupts
    582.25 ± 25%     -85.1%      86.50 ± 40%  interrupts.CPU27.RES:Rescheduling_interrupts
    474.00 ± 20%     -82.4%      83.25 ± 24%  interrupts.CPU28.RES:Rescheduling_interrupts
    375.00 ± 13%     -74.7%      95.00 ± 29%  interrupts.CPU29.RES:Rescheduling_interrupts
    527.50 ± 46%     -82.2%      93.75 ± 46%  interrupts.CPU3.RES:Rescheduling_interrupts
    949.00 ± 10%     +56.0%       1480 ± 37%  interrupts.CPU30.CAL:Function_call_interrupts
    447.75 ± 30%     -70.2%     133.50 ± 48%  interrupts.CPU30.RES:Rescheduling_interrupts
    892.25 ±  2%     +26.9%       1132 ± 15%  interrupts.CPU31.CAL:Function_call_interrupts
      2089 ± 62%     -80.3%     410.75 ± 91%  interrupts.CPU32.41:PCI-MSI.2621447-edge.eth0-TxRx-6
    968.00 ± 10%     +94.1%       1878 ± 59%  interrupts.CPU32.CAL:Function_call_interrupts
    826.50 ± 28%     -71.7%     233.75 ± 31%  interrupts.CPU32.NMI:Non-maskable_interrupts
    826.50 ± 28%     -71.7%     233.75 ± 31%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    656.25 ± 15%     -75.5%     160.75 ± 64%  interrupts.CPU32.RES:Rescheduling_interrupts
    558.25 ± 32%     -75.0%     139.50 ± 55%  interrupts.CPU33.RES:Rescheduling_interrupts
    447.25 ± 16%     -61.8%     171.00 ± 47%  interrupts.CPU34.RES:Rescheduling_interrupts
    504.50 ± 33%     -81.9%      91.25 ± 32%  interrupts.CPU35.RES:Rescheduling_interrupts
    301.50 ± 40%     -77.9%      66.75 ± 62%  interrupts.CPU4.RES:Rescheduling_interrupts
    315.25 ± 43%     -65.9%     107.50 ± 81%  interrupts.CPU5.RES:Rescheduling_interrupts
    491.75 ± 57%     -86.3%      67.25 ± 35%  interrupts.CPU6.RES:Rescheduling_interrupts
   1385835           -96.8%      44728        interrupts.CPU8.73:PCI-MSI.2097152-edge.isci-msix
    916.25 ±  9%     +26.0%       1154 ± 18%  interrupts.CPU8.CAL:Function_call_interrupts
    827.25 ± 38%     -75.0%     206.50 ± 39%  interrupts.CPU8.NMI:Non-maskable_interrupts
    827.25 ± 38%     -75.0%     206.50 ± 39%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    906.25 ±  2%     +53.9%       1395 ± 47%  interrupts.CPU9.CAL:Function_call_interrupts
      1107 ± 10%     -87.6%     137.50 ± 49%  interrupts.CPU9.RES:Rescheduling_interrupts
     13410 ±  5%     -16.0%      11263 ±  7%  interrupts.NMI:Non-maskable_interrupts
     13410 ±  5%     -16.0%      11263 ±  7%  interrupts.PMI:Performance_monitoring_interrupts
     15515 ± 16%     -69.0%       4813 ± 14%  interrupts.RES:Rescheduling_interrupts
      0.02 ± 49%    +181.5%       0.05 ± 72%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ±  8%     +35.1%       0.01 ±  8%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.03 ± 59%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.01 ± 14%    +287.0%       0.02 ± 16%  perf-sched.sch_delay.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
      0.01 ± 57%     -58.8%       0.00 ± 14%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ±118%    +350.0%       0.02 ± 68%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
      0.00 ±100%    +412.5%       0.01 ± 21%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      0.01 ± 49%     +86.5%       0.02 ±  8%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ± 57%    +733.3%       0.01 ± 18%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ±  9%     +37.7%       0.02 ± 10%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ±  7%     +73.1%       0.01 ±  3%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.03 ± 34%     +72.1%       0.06 ± 18%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.03 ± 59%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.03 ±110%    +197.4%       0.08 ± 32%  perf-sched.sch_delay.max.ms.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range
      0.01 ±110%    +274.4%       0.04 ± 64%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
      0.03 ± 60%     +91.0%       0.06 ± 10%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.02 ± 31%    +130.6%       0.04 ± 12%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.04 ± 18%     +26.8%       0.05 ±  7%  perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 11%    +125.0%       0.01 ±  3%  perf-sched.total_sch_delay.average.ms
     10.12 ± 21%   +1307.5%     142.39        perf-sched.total_wait_and_delay.average.ms
    172750 ± 17%     -91.4%      14933 ±  2%  perf-sched.total_wait_and_delay.count.ms
      5885 ± 25%     +56.9%       9234 ± 11%  perf-sched.total_wait_and_delay.max.ms
     10.11 ± 21%   +1308.2%     142.38        perf-sched.total_wait_time.average.ms
      5885 ± 25%     +56.9%       9234 ± 11%  perf-sched.total_wait_time.max.ms
      8.01 ± 22%   +1181.9%     102.62 ± 62%  perf-sched.wait_and_delay.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
      1.02 ±173%    +314.1%       4.24 ±  3%  perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    232.13 ± 18%    +103.1%     471.43 ± 36%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      4.54 ±  6%      -8.7%       4.14 ±  4%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    393.96 ± 14%     +23.3%     485.81 ±  5%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      4.09 ± 26%   +6387.8%     265.12 ±  2%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     61985 ± 18%     -98.0%       1211 ±  2%  perf-sched.wait_and_delay.count.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
      2.00 ±173%   +4850.0%      99.00 ± 18%  perf-sched.wait_and_delay.count.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range
     83.50 ± 16%     -71.0%      24.25 ±  6%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      1547 ±  7%     +16.4%       1801 ±  3%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    102506 ± 17%     -97.7%       2395 ±  7%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    943.35 ±  8%     -81.1%     178.11 ± 56%  perf-sched.wait_and_delay.max.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
      1.26 ±173%    +304.5%       5.11        perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      1589 ± 52%    +190.9%       4624 ± 25%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ±173%  +1.2e+05%       5.20 ± 73%  perf-sched.wait_time.avg.ms.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork
      3.92 ± 11%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      8.00 ± 22%   +1182.5%     102.60 ± 62%  perf-sched.wait_time.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
      0.28 ±155%  +24627.1%      69.30 ±106%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
    232.13 ± 18%    +103.1%     471.42 ± 36%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      4.53 ±  6%      -8.9%       4.13 ±  4%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    393.96 ± 14%     +23.3%     485.80 ±  5%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      4.08 ± 26%   +6398.3%     265.11 ±  2%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
      0.00 ±173%    +4e+05%      17.17 ±107%  perf-sched.wait_time.max.ms.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork
      3.92 ± 11%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
    943.35 ±  8%     -81.1%     178.04 ± 56%  perf-sched.wait_time.max.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
      6.02 ±172%   +2236.8%     140.75 ± 90%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      0.01 ± 86%   +7423.8%       0.79 ±167%  perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io.write_all_supers
      1589 ± 52%    +190.9%       4624 ± 25%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     44.03           +20.3%      52.96 ± 12%  perf-stat.i.MPKI
 2.589e+08           -36.7%  1.639e+08        perf-stat.i.branch-instructions
      5.01            +6.7       11.71 ±  7%  perf-stat.i.branch-miss-rate%
  12714258           +31.8%   16758440 ±  7%  perf-stat.i.branch-misses
      9.35 ±  2%     +14.7       24.09 ±  3%  perf-stat.i.cache-miss-rate%
   4375264           +81.1%    7923317 ± 18%  perf-stat.i.cache-misses
  48162798           -32.2%   32638947 ± 14%  perf-stat.i.cache-references
     39282           -92.6%       2916        perf-stat.i.context-switches
      4.10           +83.9%       7.54 ±  6%  perf-stat.i.cpi
     68.61 ±  2%     +64.8%     113.06        perf-stat.i.cpu-migrations
      1061 ±  3%     -42.2%     613.59 ±  8%  perf-stat.i.cycles-between-cache-misses
      1.23 ±  2%      -0.2        1.06 ±  8%  perf-stat.i.dTLB-load-miss-rate%
   3992778 ±  2%     -47.7%    2089367 ± 11%  perf-stat.i.dTLB-load-misses
 3.459e+08           -35.4%  2.234e+08 ±  2%  perf-stat.i.dTLB-loads
      0.17 ±  2%      +0.0        0.20 ±  4%  perf-stat.i.dTLB-store-miss-rate%
     70.32           +21.5       91.83        perf-stat.i.iTLB-load-miss-rate%
   1985999 ±  2%     -39.6%    1199619 ±  2%  perf-stat.i.iTLB-load-misses
    854405           -87.4%     107547 ± 14%  perf-stat.i.iTLB-loads
 1.254e+09           -36.9%  7.911e+08        perf-stat.i.instructions
      0.26           -41.9%       0.15 ±  6%  perf-stat.i.ipc
      1.34 ±  7%     -58.0%       0.56 ± 31%  perf-stat.i.metric.K/sec
     19.36           -25.9%      14.35 ±  6%  perf-stat.i.metric.M/sec
      2893            -2.0%       2835        perf-stat.i.minor-faults
    197998 ± 19%     -32.9%     132808 ±  3%  perf-stat.i.node-load-misses
    228908 ± 17%     -31.8%     156142 ±  3%  perf-stat.i.node-loads
    236295 ±  7%     -17.5%     194871 ±  6%  perf-stat.i.node-stores
      3820            -1.7%       3754        perf-stat.i.page-faults
      4.91            +5.3       10.21 ±  7%  perf-stat.overall.branch-miss-rate%
      9.09 ±  2%     +15.1       24.16 ±  3%  perf-stat.overall.cache-miss-rate%
      3.65           +65.6%       6.04 ±  7%  perf-stat.overall.cpi
      1045 ±  3%     -41.3%     613.83 ±  8%  perf-stat.overall.cycles-between-cache-misses
      1.14 ±  2%      -0.2        0.92 ±  8%  perf-stat.overall.dTLB-load-miss-rate%
      0.17 ±  3%      +0.0        0.19 ±  4%  perf-stat.overall.dTLB-store-miss-rate%
     69.91           +21.9       91.81        perf-stat.overall.iTLB-load-miss-rate%
      0.27           -39.3%       0.17 ±  6%  perf-stat.overall.ipc
    175094         +3617.1%    6508495        perf-stat.overall.path-length
 2.583e+08           -36.7%  1.636e+08        perf-stat.ps.branch-instructions
  12678701           +31.8%   16711512 ±  7%  perf-stat.ps.branch-misses
   4361806           +81.1%    7898239 ± 18%  perf-stat.ps.cache-misses
  48005640           -32.2%   32532931 ± 14%  perf-stat.ps.cache-references
     39151           -92.6%       2907        perf-stat.ps.context-switches
     68.42 ±  2%     +64.7%     112.70        perf-stat.ps.cpu-migrations
   3979673 ±  2%     -47.7%    2082707 ± 11%  perf-stat.ps.dTLB-load-misses
 3.449e+08           -35.4%  2.229e+08 ±  2%  perf-stat.ps.dTLB-loads
   1979429 ±  2%     -39.6%    1195674 ±  2%  perf-stat.ps.iTLB-load-misses
    851561           -87.4%     107209 ± 14%  perf-stat.ps.iTLB-loads
 1.251e+09           -36.9%  7.896e+08        perf-stat.ps.instructions
      2884            -2.0%       2827        perf-stat.ps.minor-faults
    197548 ± 19%     -33.0%     132439 ±  3%  perf-stat.ps.node-load-misses
    228425 ± 17%     -31.8%     155760 ±  3%  perf-stat.ps.node-loads
    236208 ±  7%     -17.3%     195231 ±  6%  perf-stat.ps.node-stores
 3.787e+11           -36.8%  2.393e+11        perf-stat.total.instructions
     10.21 ± 20%      -9.9        0.36 ±100%  perf-profile.calltrace.cycles-pp.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
      8.66 ± 23%      -8.5        0.17 ±173%  perf-profile.calltrace.cycles-pp.iomap_apply.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write.vfs_write
     11.16 ± 19%      -7.3        3.87 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     11.02 ± 19%      -7.2        3.84 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.77 ± 19%      -7.2        3.59 ±  6%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.72 ± 19%      -7.1        3.59 ±  6%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.54 ± 19%      -6.9        3.59 ±  6%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.51 ± 20%      -6.9        3.59 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
      6.28 ± 31%      -6.1        0.14 ±173%  perf-profile.calltrace.cycles-pp.btrfs_dio_iomap_begin.iomap_apply.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
      6.48 ± 12%      -4.6        1.87 ± 23%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      6.63 ± 12%      -4.5        2.11 ± 22%  perf-profile.calltrace.cycles-pp.ret_from_fork
      6.63 ± 12%      -4.5        2.11 ± 22%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      5.90 ± 11%      -4.1        1.82 ± 23%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.91 ±  8%      +0.3        1.17 ±  5%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      1.16 ± 12%      +0.3        1.43 ±  6%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.64 ±  9%      +0.3        0.93 ±  8%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack
      0.99 ±  7%      +0.3        1.32 ± 12%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.65 ± 10%      +0.3        0.98 ± 11%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu
      0.26 ±100%      +0.4        0.68 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_governor_latency_req.menu_select.do_idle.cpu_startup_entry.start_secondary
      1.29 ±  6%      +0.4        1.71 ± 14%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.56 ±  9%      +0.4        1.99 ±  3%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.75 ± 10%      +0.5        2.27 ±  5%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.67 ± 65%      +0.5        1.19 ±  8%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter
      0.67 ± 65%      +0.5        1.19 ±  8%  perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.67 ± 65%      +0.5        1.19 ±  8%  perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter.do_idle
      0.67 ± 65%      +0.5        1.19 ±  8%  perf-profile.calltrace.cycles-pp.__sysvec_irq_work.asm_call_sysvec_on_stack.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state
      0.00            +0.6        0.60 ±  6%  perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file
      0.00            +0.6        0.60 ±  7%  perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.14 ±173%      +0.6        0.74 ±  7%  perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      0.00            +0.6        0.62 ± 13%  perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      0.00            +0.6        0.62 ±  5%  perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
      0.80 ± 58%      +0.6        1.45 ± 17%  perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
      0.80 ± 58%      +0.6        1.45 ± 17%  perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
      0.15 ±173%      +0.7        0.83 ± 12%  perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.7        0.72 ±  5%  perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
      0.00            +0.7        0.73 ± 20%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node
      0.00            +0.8        0.77 ± 18%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      0.00            +0.8        0.77 ± 18%  perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
      0.00            +0.8        0.77 ± 18%  perf-profile.calltrace.cycles-pp.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
      0.00            +0.8        0.77 ± 18%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      2.60 ± 14%      +0.9        3.48 ± 10%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +0.9        0.90 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
      2.73 ±  7%      +1.1        3.81 ±  4%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      2.92 ±  7%      +1.2        4.08 ±  4%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      4.36 ± 15%      +1.2        5.60 ±  9%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.03 ± 13%      +1.2        2.28 ± 33%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
      1.05 ± 13%      +1.3        2.31 ± 33%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
      3.44 ±  8%      +1.3        4.72 ±  4%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      1.15 ± 15%      +1.4        2.54 ± 32%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
      1.15 ± 15%      +1.4        2.54 ± 32%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
      1.15 ± 15%      +1.4        2.54 ± 32%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify
      0.00            +1.7        1.68 ± 20%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
      0.00            +1.7        1.68 ± 20%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file
      0.00            +1.9        1.87 ± 18%  perf-profile.calltrace.cycles-pp.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter
      0.00            +2.0        2.02 ± 17%  perf-profile.calltrace.cycles-pp.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
      0.00            +2.0        2.03 ± 17%  perf-profile.calltrace.cycles-pp.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
      5.73 ±  7%      +2.3        8.03 ±  3%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
      8.28 ± 10%      +2.5       10.75 ±  4%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      8.66 ±  9%      +2.7       11.31 ±  4%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      8.79 ±  9%      +2.7       11.50 ±  4%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +3.0        3.00 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
     13.50 ±  9%      +4.3       17.84 ±  7%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     15.39 ±  9%      +4.7       20.10 ±  7%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     51.90            +8.8       60.72 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     72.26 ±  2%      +9.9       82.17        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     71.31 ±  2%      +9.9       81.25        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     79.02 ±  3%     +10.8       89.79        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     79.10 ±  3%     +10.8       89.91        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     79.10 ±  3%     +10.8       89.91        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     80.25 ±  2%     +12.2       92.45        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     10.22 ± 20%      -9.6        0.58 ± 26%  perf-profile.children.cycles-pp.__iomap_dio_rw
      8.66 ± 23%      -8.2        0.45 ± 32%  perf-profile.children.cycles-pp.iomap_apply
     10.79 ± 19%      -7.2        3.62 ±  6%  perf-profile.children.cycles-pp.ksys_write
     12.01 ± 17%      -7.1        4.86 ±  8%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     10.74 ± 19%      -7.1        3.62 ±  6%  perf-profile.children.cycles-pp.vfs_write
     11.77 ± 17%      -7.0        4.73 ±  8%  perf-profile.children.cycles-pp.do_syscall_64
     10.56 ± 19%      -6.9        3.62 ±  6%  perf-profile.children.cycles-pp.new_sync_write
     10.52 ± 20%      -6.9        3.59 ±  6%  perf-profile.children.cycles-pp.btrfs_file_write_iter
      6.28 ± 31%      -5.9        0.38 ± 36%  perf-profile.children.cycles-pp.btrfs_dio_iomap_begin
      6.48 ± 12%      -4.6        1.87 ± 23%  perf-profile.children.cycles-pp.worker_thread
      6.63 ± 12%      -4.5        2.11 ± 22%  perf-profile.children.cycles-pp.kthread
      6.63 ± 12%      -4.5        2.12 ± 22%  perf-profile.children.cycles-pp.ret_from_fork
      5.91 ± 11%      -4.1        1.82 ± 23%  perf-profile.children.cycles-pp.process_one_work
      3.45 ± 24%      -3.3        0.14 ± 55%  perf-profile.children.cycles-pp.btrfs_work_helper
      3.43 ± 24%      -3.3        0.14 ± 55%  perf-profile.children.cycles-pp.btrfs_finish_ordered_io
      2.77 ± 43%      -2.7        0.04 ±106%  perf-profile.children.cycles-pp.asm_common_interrupt
      2.77 ± 43%      -2.7        0.04 ±106%  perf-profile.children.cycles-pp.common_interrupt
      2.33 ±  8%      -2.3        0.04 ±107%  perf-profile.children.cycles-pp.can_nocow_extent
      2.21 ± 12%      -2.1        0.06 ± 67%  perf-profile.children.cycles-pp.iomap_dio_bio_actor
      2.01 ± 42%      -1.9        0.08 ± 17%  perf-profile.children.cycles-pp.blk_done_softirq
      1.95 ± 42%      -1.9        0.07 ± 17%  perf-profile.children.cycles-pp.scsi_io_completion
      1.95 ± 42%      -1.9        0.07 ± 17%  perf-profile.children.cycles-pp.scsi_end_request
      2.13 ± 94%      -1.9        0.27 ± 49%  perf-profile.children.cycles-pp.btrfs_delalloc_reserve_space
      2.17 ± 12%      -1.8        0.37 ± 27%  perf-profile.children.cycles-pp.__schedule
      2.51 ± 96%      -1.7        0.77 ± 25%  perf-profile.children.cycles-pp._raw_spin_lock
      2.00 ±100%      -1.7        0.26 ± 51%  perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
      1.92 ±104%      -1.6        0.29 ± 47%  perf-profile.children.cycles-pp.__reserve_bytes
      1.68 ± 13%      -1.6        0.05 ± 62%  perf-profile.children.cycles-pp.iomap_dio_submit_bio
      1.90 ±105%      -1.6        0.28 ± 48%  perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes
      1.66 ± 12%      -1.6        0.05 ± 62%  perf-profile.children.cycles-pp.btrfs_submit_direct
      1.43 ± 41%      -1.4        0.04 ±100%  perf-profile.children.cycles-pp.blk_update_request
      1.35 ± 41%      -1.3        0.03 ±100%  perf-profile.children.cycles-pp.btrfs_end_bio
      1.50 ± 13%      -1.3        0.24 ± 17%  perf-profile.children.cycles-pp.schedule
      1.11 ± 12%      -1.1        0.04 ±100%  perf-profile.children.cycles-pp.btrfs_map_bio
      0.97 ± 18%      -0.9        0.03 ±100%  perf-profile.children.cycles-pp.io_schedule
      0.98 ± 32%      -0.9        0.09 ± 16%  perf-profile.children.cycles-pp.__blk_mq_run_hw_queue
      0.97 ± 32%      -0.9        0.09 ± 16%  perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
      0.96 ± 33%      -0.9        0.09 ± 16%  perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
      0.92 ± 33%      -0.8        0.09 ± 21%  perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched
      0.89 ±  4%      -0.8        0.05 ± 61%  perf-profile.children.cycles-pp.read_block_for_search
      0.94 ± 32%      -0.8        0.15 ± 39%  perf-profile.children.cycles-pp.try_to_wake_up
      0.85 ± 16%      -0.7        0.16 ± 67%  perf-profile.children.cycles-pp.btrfs_csum_file_blocks
      0.69 ± 34%      -0.6        0.06 ± 17%  perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list
      0.73 ±  9%      -0.6        0.14 ± 47%  perf-profile.children.cycles-pp.schedule_idle
      0.65 ± 33%      -0.6        0.06 ± 17%  perf-profile.children.cycles-pp.scsi_queue_rq
      0.65 ± 30%      -0.5        0.12 ± 33%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.56 ± 10%      -0.5        0.09 ± 13%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.42 ± 32%      -0.4        0.04 ± 58%  perf-profile.children.cycles-pp.sas_queuecommand
      0.39 ± 33%      -0.4        0.03 ±100%  perf-profile.children.cycles-pp.__ata_scsi_queuecmd
      0.43 ±  8%      -0.4        0.07 ± 22%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.43 ± 18%      -0.3        0.09 ± 28%  perf-profile.children.cycles-pp.blk_finish_plug
      0.42 ± 19%      -0.3        0.09 ± 28%  perf-profile.children.cycles-pp.blk_flush_plug_list
      0.42 ± 18%      -0.3        0.09 ± 28%  perf-profile.children.cycles-pp.blk_mq_flush_plug_list
      0.40 ± 22%      -0.3        0.09 ± 28%  perf-profile.children.cycles-pp.blk_mq_sched_insert_requests
      0.38 ± 25%      -0.3        0.08 ± 30%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.37 ± 40%      -0.3        0.07 ± 20%  perf-profile.children.cycles-pp.newidle_balance
      0.34 ± 26%      -0.3        0.05 ± 60%  perf-profile.children.cycles-pp.kmem_cache_free
      0.35 ±  6%      -0.3        0.07 ± 21%  perf-profile.children.cycles-pp.dequeue_entity
      0.33 ± 22%      -0.2        0.08 ± 13%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.36 ± 24%      -0.2        0.16 ±  9%  perf-profile.children.cycles-pp.update_load_avg
      0.26 ± 23%      -0.2        0.07 ±  7%  perf-profile.children.cycles-pp.enqueue_entity
      0.29 ± 22%      -0.2        0.12 ± 81%  perf-profile.children.cycles-pp.btrfs_lookup_csum
      0.18 ± 20%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.set_next_entity
      0.12 ± 22%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.21 ±  9%      -0.1        0.13 ± 31%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.12 ± 15%      -0.1        0.05 ± 67%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.11 ± 20%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.15 ± 20%      -0.1        0.08 ± 13%  perf-profile.children.cycles-pp.__blk_mq_delay_run_hw_queue
      0.79 ±  4%      -0.1        0.73 ±  4%  perf-profile.children.cycles-pp.read_tsc
      0.12 ± 23%      -0.0        0.08 ± 30%  perf-profile.children.cycles-pp.read
      0.07 ±  7%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.path_openat
      0.09 ± 16%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.rcu_nmi_exit
      0.07 ±  7%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.do_filp_open
      0.07 ± 19%      +0.0        0.11 ± 23%  perf-profile.children.cycles-pp.rcu_segcblist_ready_cbs
      0.08 ± 14%      +0.0        0.12 ±  6%  perf-profile.children.cycles-pp.tick_program_event
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.run_timer_softirq
      0.05 ± 62%      +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.perf_pmu_disable
      0.01 ±173%      +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.cpuidle_get_cpu_driver
      0.03 ±100%      +0.1        0.09 ± 23%  perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick
      0.21 ± 10%      +0.1        0.27 ±  9%  perf-profile.children.cycles-pp.menu_reflect
      0.03 ±102%      +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.rcu_is_cpu_rrupt_from_idle
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.btrfs_alloc_tree_block
      0.19 ± 12%      +0.1        0.26 ± 11%  perf-profile.children.cycles-pp.rb_next
      0.16 ± 14%      +0.1        0.23 ± 10%  perf-profile.children.cycles-pp.irqentry_exit
      0.07 ± 11%      +0.1        0.14 ± 25%  perf-profile.children.cycles-pp.rcu_needs_cpu
      0.03 ±102%      +0.1        0.10 ± 30%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.01 ±173%      +0.1        0.09 ± 45%  perf-profile.children.cycles-pp.leave_mm
      0.14 ± 19%      +0.1        0.22 ± 16%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.00            +0.1        0.08 ± 37%  perf-profile.children.cycles-pp.cpu_latency_qos_limit
      0.22 ± 18%      +0.1        0.30 ± 14%  perf-profile.children.cycles-pp.get_cpu_device
      0.02 ±173%      +0.1        0.10 ± 35%  perf-profile.children.cycles-pp.sched_idle_set_state
      0.07 ± 69%      +0.1        0.15 ± 19%  perf-profile.children.cycles-pp.hrtimer_forward
      0.14 ± 10%      +0.1        0.23 ± 14%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.10 ± 15%      +0.1        0.19 ±  9%  perf-profile.children.cycles-pp.x86_pmu_disable
      0.08 ± 67%      +0.1        0.17 ± 15%  perf-profile.children.cycles-pp.cpuidle_reflect
      0.00            +0.1        0.09 ± 27%  perf-profile.children.cycles-pp.check_leaf
      0.07 ± 58%      +0.1        0.17 ± 26%  perf-profile.children.cycles-pp.run_local_timers
      0.00            +0.1        0.10 ± 27%  perf-profile.children.cycles-pp.btrfs_cow_block
      0.00            +0.1        0.10 ± 27%  perf-profile.children.cycles-pp.__btrfs_cow_block
      0.00            +0.1        0.10 ± 25%  perf-profile.children.cycles-pp.btree_csum_one_bio
      0.17 ± 14%      +0.1        0.27 ± 33%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.00            +0.1        0.10 ± 35%  perf-profile.children.cycles-pp.write_one_eb
      0.00            +0.1        0.10 ± 27%  perf-profile.children.cycles-pp.btrfs_submit_metadata_bio
      0.43 ± 13%      +0.1        0.53 ±  5%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.10 ± 18%      +0.1        0.21 ± 14%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.00            +0.1        0.11 ± 23%  perf-profile.children.cycles-pp.submit_one_bio
      0.16 ± 35%      +0.1        0.27 ± 25%  perf-profile.children.cycles-pp.wait_consider_task
      0.16 ± 19%      +0.1        0.28 ± 24%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.18 ± 32%      +0.1        0.30 ± 19%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.00            +0.1        0.12 ± 23%  perf-profile.children.cycles-pp.btrfs_write_marked_extents
      0.00            +0.1        0.12 ± 23%  perf-profile.children.cycles-pp.btree_write_cache_pages
      0.00            +0.1        0.13 ± 17%  perf-profile.children.cycles-pp.do_writepages
      0.00            +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      1.08 ± 11%      +0.2        1.23 ±  7%  perf-profile.children.cycles-pp.io_serial_in
      0.39 ± 10%      +0.2        0.55 ±  3%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.00            +0.2        0.17 ± 55%  perf-profile.children.cycles-pp.btrfs_log_changed_extents
      0.42 ±  5%      +0.2        0.63 ±  6%  perf-profile.children.cycles-pp.timerqueue_del
      0.32 ± 14%      +0.2        0.54 ± 11%  perf-profile.children.cycles-pp.timerqueue_add
      0.72 ± 11%      +0.2        0.95 ±  8%  perf-profile.children.cycles-pp.update_blocked_averages
      0.49 ±  8%      +0.2        0.72 ± 13%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.52 ±  6%      +0.2        0.76 ±  7%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.38 ± 16%      +0.3        0.64 ± 13%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.96 ±  7%      +0.3        1.22 ±  5%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.68 ± 11%      +0.3        1.00 ± 11%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.53 ± 11%      +0.3        0.86 ± 13%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      1.22 ± 10%      +0.3        1.56 ±  6%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.03 ±  6%      +0.4        1.39 ± 13%  perf-profile.children.cycles-pp.tick_irq_enter
      1.38 ±  5%      +0.4        1.78 ± 14%  perf-profile.children.cycles-pp.irq_enter_rcu
      1.64 ±  9%      +0.4        2.08 ±  3%  perf-profile.children.cycles-pp.scheduler_tick
      1.81 ± 11%      +0.6        2.36 ±  5%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.23 ± 18%      +0.6        0.83 ± 36%  perf-profile.children.cycles-pp.__btrfs_tree_read_lock
      0.00            +0.7        0.72 ±  5%  perf-profile.children.cycles-pp.wait_log_commit
      0.06 ± 74%      +0.8        0.81 ± 37%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      0.00            +0.8        0.76 ±  3%  perf-profile.children.cycles-pp.__mutex_lock
      0.03 ±100%      +0.8        0.79 ± 14%  perf-profile.children.cycles-pp.__btrfs_tree_lock
      0.03 ±100%      +0.8        0.79 ± 14%  perf-profile.children.cycles-pp.btrfs_lock_root_node
      0.00            +0.8        0.79 ± 14%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      0.00            +0.9        0.90 ±  5%  perf-profile.children.cycles-pp.btrfs_sync_log
      2.83 ±  6%      +1.1        3.93 ±  3%  perf-profile.children.cycles-pp.update_process_times
      3.01 ±  6%      +1.2        4.20 ±  4%  perf-profile.children.cycles-pp.tick_sched_handle
      3.54 ±  7%      +1.3        4.85 ±  3%  perf-profile.children.cycles-pp.tick_sched_timer
      4.44 ± 14%      +1.3        5.78 ± 10%  perf-profile.children.cycles-pp.menu_select
      1.15 ± 15%      +1.4        2.54 ± 32%  perf-profile.children.cycles-pp.start_kernel
      0.08 ± 78%      +1.5        1.60 ± 20%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.00            +1.7        1.71 ± 19%  perf-profile.children.cycles-pp.btrfs_insert_empty_items
      0.00            +1.9        1.87 ± 18%  perf-profile.children.cycles-pp.btrfs_log_inode
      0.00            +2.0        2.02 ± 17%  perf-profile.children.cycles-pp.btrfs_log_inode_parent
      0.00            +2.0        2.03 ± 17%  perf-profile.children.cycles-pp.btrfs_log_dentry_safe
      0.04 ±115%      +2.2        2.23 ± 14%  perf-profile.children.cycles-pp.osq_lock
      5.92 ±  6%      +2.3        8.26 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      8.53 ±  9%      +2.5       11.05 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      8.91 ±  9%      +2.7       11.63 ±  5%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.00            +3.0        3.00 ± 11%  perf-profile.children.cycles-pp.btrfs_sync_file
     13.89 ±  9%      +4.2       18.09 ±  6%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
     15.08 ±  9%      +4.4       19.51 ±  6%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     51.91            +8.8       60.72 ±  2%  perf-profile.children.cycles-pp.intel_idle
     79.10 ±  3%     +10.8       89.91        perf-profile.children.cycles-pp.start_secondary
     73.22 ±  2%     +11.2       84.38        perf-profile.children.cycles-pp.cpuidle_enter_state
     73.32 ±  2%     +11.2       84.49        perf-profile.children.cycles-pp.cpuidle_enter
     80.25 ±  2%     +12.2       92.45        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     80.25 ±  2%     +12.2       92.45        perf-profile.children.cycles-pp.cpu_startup_entry
     80.23 ±  2%     +12.2       92.45        perf-profile.children.cycles-pp.do_idle
      0.97 ±  2%      -0.5        0.44 ± 19%  perf-profile.self.cycles-pp._raw_spin_lock
      0.36 ± 16%      -0.3        0.05 ± 58%  perf-profile.self.cycles-pp.__schedule
      0.35 ±  8%      -0.3        0.07 ± 13%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.12 ± 22%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.15 ± 25%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.update_load_avg
      0.11 ± 20%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.18 ± 14%      -0.1        0.12 ± 32%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.08 ± 17%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.cpuidle_enter
      0.07 ± 15%      +0.0        0.10 ± 21%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.07 ± 15%      +0.0        0.11 ± 11%  perf-profile.self.cycles-pp.rcu_nmi_exit
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.arch_cpu_idle_enter
      0.04 ± 58%      +0.0        0.08 ± 33%  perf-profile.self.cycles-pp.perf_pmu_disable
      0.10 ± 18%      +0.0        0.14 ± 18%  perf-profile.self.cycles-pp.irq_enter_rcu
      0.12 ±  6%      +0.0        0.17 ± 10%  perf-profile.self.cycles-pp.menu_reflect
      0.08 ± 14%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.tick_program_event
      0.03 ±100%      +0.0        0.07 ± 27%  perf-profile.self.cycles-pp.rcu_is_cpu_rrupt_from_idle
      0.15 ± 18%      +0.0        0.20 ±  7%  perf-profile.self.cycles-pp.asm_call_sysvec_on_stack
      0.06 ± 14%      +0.1        0.11 ± 23%  perf-profile.self.cycles-pp.rcu_segcblist_ready_cbs
      0.01 ±173%      +0.1        0.07 ± 26%  perf-profile.self.cycles-pp.cpuidle_get_cpu_driver
      0.20 ± 17%      +0.1        0.27 ± 10%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.10 ± 23%      +0.1        0.17 ± 25%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.03 ±102%      +0.1        0.10 ± 30%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.01 ±173%      +0.1        0.08 ± 37%  perf-profile.self.cycles-pp.sched_idle_set_state
      0.22 ± 17%      +0.1        0.29 ± 12%  perf-profile.self.cycles-pp.scheduler_tick
      0.17 ± 14%      +0.1        0.24 ± 14%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.1        0.07 ± 24%  perf-profile.self.cycles-pp.tick_nohz_idle_retain_tick
      0.13 ± 15%      +0.1        0.21 ±  7%  perf-profile.self.cycles-pp.irq_exit_rcu
      0.00            +0.1        0.08 ± 37%  perf-profile.self.cycles-pp.cpu_latency_qos_limit
      0.22 ± 18%      +0.1        0.30 ± 14%  perf-profile.self.cycles-pp.get_cpu_device
      0.10 ± 30%      +0.1        0.19 ± 21%  perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
      0.14 ± 10%      +0.1        0.23 ± 14%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.11 ± 26%      +0.1        0.20 ± 12%  perf-profile.self.cycles-pp.clockevents_program_event
      0.09 ± 20%      +0.1        0.18 ± 10%  perf-profile.self.cycles-pp.x86_pmu_disable
      0.05 ±102%      +0.1        0.14 ± 16%  perf-profile.self.cycles-pp.hrtimer_forward
      0.10 ± 18%      +0.1        0.20 ± 18%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.05 ± 60%      +0.1        0.14 ± 23%  perf-profile.self.cycles-pp.run_local_timers
      0.24 ±  8%      +0.1        0.35 ± 19%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.16 ± 18%      +0.1        0.28 ± 24%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.17 ± 26%      +0.1        0.30 ± 21%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.40 ±  7%      +0.1        0.55 ± 17%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.24 ± 10%      +0.1        0.39 ±  6%  perf-profile.self.cycles-pp.timerqueue_add
      0.39 ± 10%      +0.2        0.55 ±  3%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.32 ± 12%      +0.2        0.49 ±  9%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.28 ± 13%      +0.2        0.45 ±  9%  perf-profile.self.cycles-pp.update_blocked_averages
      0.40 ± 14%      +0.2        0.65 ± 19%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.98 ± 10%      +0.3        1.23 ±  7%  perf-profile.self.cycles-pp.io_serial_in
      0.04 ±113%      +2.2        2.21 ± 14%  perf-profile.self.cycles-pp.osq_lock
     51.91            +8.8       60.71 ±  2%  perf-profile.self.cycles-pp.intel_idle


                                                                                
                                fio.write_bw_MBps                               
                                                                                
  30 +----------------------------------------------------------------------+   
     |..+..+..+.+.    +..+. :    +.    +..+..+    +..+..+..+..+..+.+..+..+..|   
  25 |-+                    :    :           :    :                         |   
     |                       :   :           :    :                         |   
     |                       :   :            :   :                         |   
  20 |-+                     :  :             :  :                          |   
     |                       :  :             :  :                          |   
  15 |-+                      : :             :  :                          |   
     |                        : :              : :                          |   
  10 |-+                      : :              : :                          |   
     |                        : :              : :                          |   
     |                        ::               ::                           |   
   5 |-+                       :                :                           |   
     |                         :                :                           |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                  fio.write_iops                                
                                                                                
  8000 +--------------------------------------------------------------------+   
       |..+..+.+..+..+..+.+..+     +.+..+..+..+    +..+..+.+..+..+..+.+..  .|   
  7000 |-+                   :     :          :    :                     +. |   
  6000 |-+                   :     :          :    :                        |   
       |                      :   :           :   :                         |   
  5000 |-+                    :   :            :  :                         |   
       |                      :   :            :  :                         |   
  4000 |-+                    :   :            :  :                         |   
       |                       : :             : :                          |   
  3000 |-+                     : :             : :                          |   
  2000 |-+                     : :             : :                          |   
       |                       : :              ::                          |   
  1000 |-+                      :               :                           |   
       |                        :               :                           |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.write_clat_mean_us                          
                                                                                
  4.5e+08 +-----------------------------------------------------------------+   
          |            O                                                    |   
    4e+08 |-+O O  O  O    O    O  O O  O  O O  O  O O  O  O O  O            |   
  3.5e+08 |-+                                                               |   
          |                                                                 |   
    3e+08 |-+                                                               |   
  2.5e+08 |-+                                                               |   
          |                                                                 |   
    2e+08 |-+                                                               |   
  1.5e+08 |-+                                                               |   
          |                                                                 |   
    1e+08 |-+                                                               |   
    5e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.write_clat_stddev                          
                                                                                
  4.5e+07 +-----------------------------------------------------------------+   
          |    O  O                    O                                    |   
    4e+07 |-+             O       O                                         |   
  3.5e+07 |-+        O              O       O  O       O  O                 |   
          |                               O                                 |   
    3e+07 |-+O         O       O                  O O       O  O            |   
  2.5e+07 |-+                                                               |   
          |                                                                 |   
    2e+07 |-+                                                               |   
  1.5e+07 |..+.+..+..+.+..+.+..+    +..+..+.+..+    +..+..+.+..+.+..+..+.+..|   
          |                     :   :           :   :                       |   
    1e+07 |-+                   :  :            :  :                        |   
    5e+06 |-+                    : :             : :                        |   
          |                      ::              ::                         |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.write_clat_90__us                          
                                                                                
  4.5e+08 +-----------------------------------------------------------------+   
          |  O                 O  O O     O O  O  O O  O  O O  O            |   
    4e+08 |-+                                                               |   
  3.5e+08 |-+                                                               |   
          |                                                                 |   
    3e+08 |-+                                                               |   
  2.5e+08 |-+                                                               |   
          |                                                                 |   
    2e+08 |-+                                                               |   
  1.5e+08 |-+                                                               |   
          |                                                                 |   
    1e+08 |-+                                                               |   
    5e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.write_clat_95__us                          
                                                                                
    5e+08 +-----------------------------------------------------------------+   
  4.5e+08 |-+  O                       O                                    |   
          |  O    O  O O  O    O  O O     O O  O  O O  O  O O  O            |   
    4e+08 |-+                                                               |   
  3.5e+08 |-+                                                               |   
          |                                                                 |   
    3e+08 |-+                                                               |   
  2.5e+08 |-+                                                               |   
    2e+08 |-+                                                               |   
          |                                                                 |   
  1.5e+08 |-+                                                               |   
    1e+08 |-+                                                               |   
          |                                                                 |   
    5e+07 |..+.+..+..+.+..+.+..+.. .+..+..+.+..+.. .+..+..+.+..+.+..+..+.+..|   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.write_clat_99__us                          
                                                                                
    5e+08 +-----------------------------------------------------------------+   
  4.5e+08 |-+O O  O  O O  O    O  O O  O  O O  O    O  O  O    O            |   
          |                                       O         O               |   
    4e+08 |-+                                                               |   
  3.5e+08 |-+                                                               |   
          |                                                                 |   
    3e+08 |-+                                                               |   
  2.5e+08 |-+                                                               |   
    2e+08 |-+                                                               |   
          |                                                                 |   
  1.5e+08 |-+                                                               |   
    1e+08 |-+                                                               |   
          |                                                             .+..|   
    5e+07 |..+.+..+..+.+..+.+..+.. .+..+..+.+..+.. .+..+..+.+..+.+..+..+    |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_500ms_                              
                                                                                
  100 +---------------------------------------------------------------------+   
   90 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
   70 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
   50 |-+                                                                   |   
   40 |-+                                                                   |   
      |                                                                     |   
   30 |-+                                                                   |   
   20 |-+                                                                   |   
      |                                                                     |   
   10 |-+                                                                   |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                     fio.workload                               
                                                                                
  2.5e+06 +-----------------------------------------------------------------+   
          |                           .+..                                  |   
          |..+.+..+..+.+..+.+..+    +.    +.+..+    +..+..+.+..+.+..+..+.+..|   
    2e+06 |-+                  :    :          :    :                       |   
          |                     :   :           :   :                       |   
          |                     :   :           :   :                       |   
  1.5e+06 |-+                   :  :            :  :                        |   
          |                     :  :            :  :                        |   
    1e+06 |-+                    : :             : :                        |   
          |                      : :             : :                        |   
          |                      : :             : :                        |   
   500000 |-+                    : :             : :                        |   
          |                       :               :                         |   
          |                       :               :                         |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.time.user_time                               
                                                                                
  16 +----------------------------------------------------------------------+   
     |                                   : :                                |   
  14 |-+                                 : :                                |   
  12 |-+           +..              +.. :   :                               |   
     |        +. ..   +..  .+     ..   +    :     +..  .+..  .+.. .+..+..   |   
  10 |..    ..  +        +. :    +           +    :  +.    +.    +       +..|   
     |  +..+                 :   :           :    :                         |   
   8 |-+                     :   :            :  :                          |   
     |                       :  :             :  :                          |   
   6 |-+                      : :             :  :                          |   
   4 |-+                      : :              : :                          |   
     |                        : :              : :                          |   
   2 |-+O  O    O           O :: O  O        O ::       O                   |   
     |        O    O  O        :       O  O     : O  O     O  O             |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              fio.time.system_time                              
                                                                                
  70 +----------------------------------------------------------------------+   
     | .+..                      +..+..      +                    .+..+..   |   
  60 |.+   +..+.+..+..+..+..+    :     +.. ..:    +..+..+..+..+..+       +..|   
     |                      :    :        +  :    :                         |   
  50 |-+                     :   :            :   :                         |   
     |                       :  :             :   :                         |   
  40 |-+                     :  :             :  :                          |   
     |                       :  :             :  :                          |   
  30 |-+                      : :              : :                          |   
     |        O O             : :         O  O :O:O  O  O  O  O             |   
  20 |-+O  O       O        O : :      O       : :                          |   
     |                O       :: O  O          : :                          |   
  10 |-+                       :                :                           |   
     |                         :                :                           |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      fio.time.percent_of_cpu_this_job_got                      
                                                                                
  30 +----------------------------------------------------------------------+   
     |                                                                      |   
  25 |-+                           .+..     .+                              |   
     |..+..+..+.+..+..+..+..+    +.    +..+. :    +..+..+..+..+..+.+..+..+..|   
     |                      :    :            :   :                         |   
  20 |-+                     :   :            :   :                         |   
     |                       :  :             :  :                          |   
  15 |-+                     :  :             :  :                          |   
     |                       :  :              : :                          |   
  10 |-+                      : :              : :                          |   
     |        O O             : :         O  O :O:O  O  O  O  O             |   
     |  O  O       O  O     O :O:O  O  O       : :                          |   
   5 |-+                      ::                :                           |   
     |                         :                :                           |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          fio.time.voluntary_context_switches                   
                                                                                
  2.5e+06 +-----------------------------------------------------------------+   
          |..+.+..+.. .+..+.  .+      .+..    .+    +..+..+.+..+.+..        |   
          |          +      +. :    +.    +.+. :    :               +..+.+..|   
    2e+06 |-+                  :    :          :    :                       |   
          |                     :   :           :   :                       |   
          |                     :   :           :  :                        |   
  1.5e+06 |-+                   :  :            :  :                        |   
          |                     :  :            :  :                        |   
    1e+06 |-+                    : :             : :                        |   
          |                      : :             : :                        |   
          |                      : :             : :                        |   
   500000 |-+                    : :             ::                         |   
          |                       :               :                         |   
          |  O O  O  O O  O    O  : O  O  O O  O  : O  O  O O  O            |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       fio.time.involuntary_context_switches                    
                                                                                
  3000 +--------------------------------------------------------------------+   
       |                                                                    |   
  2500 |-+   O O                           O          O                     |   
       |                           O O        O          O                  |   
       |  O             O       O                                           |   
  2000 |-+           O                  O       O          O                |   
       |          O          O                     O          O             |   
  1500 |-+                                                                  |   
       |                                                                    |   
  1000 |-+                                                                  |   
       |                                                                    |   
       |                                              +..  +..              |   
   500 |-.+..+.+..                 +.               ..    +             .+..|   
       |.         +..+..+.+..+.. ..  +..+..+..+.  .+     +    +..+..+.+.    |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fio.time.file_system_outputs                       
                                                                                
  1.8e+07 +-----------------------------------------------------------------+   
          |          +      +. :    :     +.+. :    :                    +..|   
  1.6e+07 |-+                  :    :          :    :                       |   
  1.4e+07 |-+                   :   :           :   :                       |   
          |                     :  :            :  :                        |   
  1.2e+07 |-+                   :  :            :  :                        |   
    1e+07 |-+                   :  :            :  :                        |   
          |                      : :             : :                        |   
    8e+06 |-+                    : :             : :                        |   
    6e+06 |-+                    : :             : :                        |   
          |                      : :             : :                        |   
    4e+06 |-+                    ::              ::                         |   
    2e+06 |-+                     :               :                         |   
          |  O O  O  O O  O    O  : O  O  O O  O  : O  O  O O  O            |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       perf-sched.total_wait_time.average.ms                    
                                                                                
  160 +---------------------------------------------------------------------+   
      |                                            O          O             |   
  140 |-+                                 O  O  O     O O  O                |   
  120 |-+     O        O       O  O  O                                      |   
      |  O  O    O  O                   O                                   |   
  100 |-+                   O                                               |   
      |                                                                     |   
   80 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
   40 |-+                                                                   |   
      |                                                                     |   
   20 |-+                                                                   |   
      |..+..+.+..+..+..+..+.+..  .+..+..+.+..+..  .+..+.+..+..+..+..+.+..+..|   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      perf-sched.total_wait_and_delay.count.ms                  
                                                                                
  250000 +------------------------------------------------------------------+   
         |                                                                  |   
         |      .+.. .+..+.. .+     +.+..   +..+                .+..        |   
  200000 |..+.+.    +       + :     :     ..   :      .+.  .+..+    +..     |   
         |                     :    :    +     :    +.   +.            +   :|   
         |                     :   :            :   :                   :  :|   
  150000 |-+                   :   :            :  :                    : : |   
         |                     :   :            :  :                     +  |   
  100000 |-+                    :  :            :  :                        |   
         |                      : :             :  :                        |   
         |                      : :             : :                         |   
   50000 |-+                    : :              ::                         |   
         |                       ::              ::                         |   
         |  O O  O  O O  O    O  :  O O  O  O  O :  O  O O  O  O            |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    perf-sched.total_wait_and_delay.average.ms                  
                                                                                
  160 +---------------------------------------------------------------------+   
      |                                            O          O             |   
  140 |-+                                 O  O  O     O O  O                |   
  120 |-+     O        O       O  O  O                                      |   
      |  O  O    O  O                   O                                   |   
  100 |-+                   O                                               |   
      |                                                                     |   
   80 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
   40 |-+                                                                   |   
      |                                                                     |   
   20 |-+                                                                   |   
      |..+..+.+..+..+..+..+.+..  .+..+..+.+..+..  .+..+.+..+..+..+..+.+..+..|   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
          perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork       
                                                                                
  300 +---------------------------------------------------------------------+   
      |          O  O       O     O                O          O             |   
  250 |-+O  O O        O       O     O  O O  O  O     O O  O                |   
      |                                                                     |   
      |                                                                     |   
  200 |-+                                                                   |   
      |                                                                     |   
  150 |-+                                                                   |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
   50 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
         perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork    
                                                                                
  140000 +------------------------------------------------------------------+   
         |                          +.+..                                   |   
  120000 |-.+.+..+..+.+..+.. .+     :      .+..+      .+.  .+..+.+..        |   
         |.                 + :     :    +.    :    +.   +.         +..    :|   
  100000 |-+                   :   :           :    :                  +   :|   
         |                     :   :            :  :                    : : |   
   80000 |-+                   :   :            :  :                    : : |   
         |                     :   :            :  :                     +  |   
   60000 |-+                    : :             :  :                        |   
         |                      : :             : :                         |   
   40000 |-+                    : :             : :                         |   
         |                      : :              ::                         |   
   20000 |-+                     :               ::                         |   
         |                       :               :                          |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
       perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork     
                                                                                
  300 +---------------------------------------------------------------------+   
      |          O  O       O     O                O          O             |   
  250 |-+O  O O        O       O     O  O O  O  O     O O  O                |   
      |                                                                     |   
      |                                                                     |   
  200 |-+                                                                   |   
      |                                                                     |   
  150 |-+                                                                   |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
   50 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                  fio.latency_2ms_                              
                                                                                
   0.01 +-------------------------------------------------------------------+   
        |                     :    :          :     :                       |   
        |                     :    :          :     :                       |   
  0.008 |-+                   :   :            :   :                        |   
        |                      :  :            :   :                        |   
        |                      :  :            :   :                        |   
  0.006 |-+                    :  :            :   :                        |   
        |                      :  :             :  :                        |   
  0.004 |-+                    : :              : :                         |   
        |                      : :              : :                         |   
        |                      : :              : :                         |   
  0.002 |-+                     ::              : :                         |   
        |                       :                :                          |   
        |                       :                :                          |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_4ms_                                
                                                                                
  60 +----------------------------------------------------------------------+   
     |..+..+..+.+..+..+..+..+    +..+..+..+..+    +..+..+..+..+..+.+..+..+..|   
  50 |-+                    :    :           :    :                         |   
     |                      :    :           :    :                         |   
     |                       :   :            :   :                         |   
  40 |-+                     :  :             :  :                          |   
     |                       :  :             :  :                          |   
  30 |-+                     :  :             :  :                          |   
     |                        : :              : :                          |   
  20 |-+                      : :              : :                          |   
     |                        : :              : :                          |   
     |                        ::               ::                           |   
  10 |-+                       :                :                           |   
     |                         :                :                           |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_10ms_                               
                                                                                
  35 +----------------------------------------------------------------------+   
     | .+..+..+.+..+..+..+..+    +..+..     .+                  .+.+..+..  .|   
  30 |.+                    :    :     +..+. :    +..+..+..+..+.         +. |   
     |                      :    :           :    :                         |   
  25 |-+                     :   :            :   :                         |   
     |                       :  :             :   :                         |   
  20 |-+                     :  :             :  :                          |   
     |                       :  :             :  :                          |   
  15 |-+                      : :              : :                          |   
     |                        : :              : :                          |   
  10 |-+                      : :              : :                          |   
     |                        ::               : :                          |   
   5 |-+                       :                :                           |   
     |                         :                :                           |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_20ms_                               
                                                                                
  9 +-----------------------------------------------------------------------+   
    |.      .+.. .+..  .+..+     +..  .+.+..+     +.    +.    +.+..+..+..  .|   
  8 |-+   +.    +    +.    :     :  +.      :     :                      +. |   
  7 |-+                    :     :          :     :                         |   
    |                       :   :            :   :                          |   
  6 |-+                     :   :            :   :                          |   
  5 |-+                     :   :            :   :                          |   
    |                       :   :            :   :                          |   
  4 |-+                      : :              : :                           |   
  3 |-+                      : :              : :                           |   
    |                        : :              : :                           |   
  2 |-+                      : :              : :                           |   
  1 |-+                       :                :                            |   
    |                         :                :                            |   
  0 +-----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_50ms_                               
                                                                                
  6 +-----------------------------------------------------------------------+   
    |                                             +..  .+..                 |   
  5 |..+..+..+..+.+..+..+..      +..+.. .+..+     :  +.    +..+.+..+..  .+..|   
    |                      +     :     +    :     :                   +.    |   
    |                      :     :           :   :                          |   
  4 |-+                     :   :            :   :                          |   
    |                       :   :            :   :                          |   
  3 |-+                     :   :            :   :                          |   
    |                       :   :             : :                           |   
  2 |-+                      : :              : :                           |   
    |                        : :              : :                           |   
    |                        : :              : :                           |   
  1 |-+                      : :               :                            |   
    |                         :                :                            |   
  0 +-----------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.10.0-rc7-00228-ge7a8dd2d9537" of type "text/plain" (171047 bytes)

View attachment "job-script" of type "text/plain" (8428 bytes)

View attachment "job.yaml" of type "text/plain" (5808 bytes)

View attachment "reproduce" of type "text/plain" (701 bytes)

Powered by blists - more mailing lists