| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20201224031032.GB17790@xsang-OptiPlex-9020>
Date: Thu, 24 Dec 2020 11:10:32 +0800
From: kernel test robot <oliver.sang@...el.com>
To: Goldwyn Rodrigues <rgoldwyn@...e.de>
Cc: 0day robot <lkp@...el.com>, LKML <linux-kernel@...r.kernel.org>,
lkp@...ts.01.org, ying.huang@...el.com, feng.tang@...el.com,
zhengjun.xing@...el.com, linux-btrfs@...r.kernel.org,
Goldwyn Rodrigues <rgoldwyn@...e.com>
Subject: [btrfs] e7a8dd2d95: fio.write_iops -98.3% regression
Greeting,
FYI, we noticed a -98.3% regression of fio.write_iops due to commit:
commit: e7a8dd2d9537a7ec5aeadb002fc4934be1a396eb ("[PATCH v3] btrfs: Make btrfs_direct_write atomic with respect to inode_lock")
url: https://github.com/0day-ci/linux/commits/Goldwyn-Rodrigues/btrfs-Make-btrfs_direct_write-atomic-with-respect-to-inode_lock/20201219-001114
base: https://git.kernel.org/cgit/linux/kernel/git/kdave/linux.git for-next
in testcase: fio-basic
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
with following parameters:
runtime: 300s
disk: 1HDD
fs: btrfs
nr_task: 100%
test_size: 128G
rw: write
bs: 4k
ioengine: sync
direct: direct
cpufreq_governor: performance
ucode: 0x42e
test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
bs/compiler/cpufreq_governor/direct/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
4k/gcc-9/performance/direct/1HDD/btrfs/sync/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/write/lkp-ivb-2ep1/128G/fio-basic/0x42e
commit:
97847e0652 ("Merge branch 'for-next-next-v5.10-20201211' into for-next-20201211")
e7a8dd2d95 ("btrfs: Make btrfs_direct_write atomic with respect to inode_lock")
97847e06525b51ea e7a8dd2d9537a7ec5aeadb002fc
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
0:4 -4% 0:4 perf-profile.children.cycles-pp.error_return
0:4 5% 0:4 perf-profile.children.cycles-pp.error_entry
0:4 5% 0:4 perf-profile.self.cycles-pp.error_entry
%stddev %change %stddev
\ | \
0.02 ± 3% +0.1 0.12 ± 13% fio.latency_1000ms%
0.01 -0.0 0.00 fio.latency_1000us%
0.96 ± 13% -0.8 0.19 ±101% fio.latency_100ms%
31.35 -31.4 0.00 fio.latency_10ms%
8.28 ± 2% -8.3 0.00 fio.latency_20ms%
0.04 ± 29% -0.0 0.02 ±100% fio.latency_250ms%
0.01 -0.0 0.00 fio.latency_2ms%
54.35 -54.3 0.00 fio.latency_4ms%
0.02 ± 10% +99.6 99.66 fio.latency_500ms%
0.01 -0.0 0.00 fio.latency_500us%
4.97 -5.0 0.00 fio.latency_50ms%
0.01 -0.0 0.00 fio.latency_750us%
17301324 -93.9% 1062134 ± 2% fio.time.file_system_outputs
316.50 ± 45% +583.7% 2164 ± 14% fio.time.involuntary_context_switches
23.75 -63.2% 8.75 ± 4% fio.time.percent_of_cpu_this_job_got
62.17 -58.4% 25.87 ± 2% fio.time.system_time
10.95 ± 2% -83.7% 1.78 ± 9% fio.time.user_time
2194671 -93.0% 153935 fio.time.voluntary_context_switches
2162665 -98.3% 36777 fio.workload
28.16 -98.3% 0.48 fio.write_bw_MBps
12156928 +3384.6% 4.236e+08 fio.write_clat_90%_us
23855104 ± 2% +1715.4% 4.331e+08 fio.write_clat_95%_us
50724864 ± 4% +797.2% 4.551e+08 fio.write_clat_99%_us
6657995 +5784.2% 3.918e+08 fio.write_clat_mean_us
15787433 ± 2% +110.1% 33164741 ± 9% fio.write_clat_stddev
7208 -98.3% 122.53 fio.write_iops
10477 +39.4% 14609 uptime.idle
18.03 ± 15% -25.7% 13.39 boot-time.dhcp
1017 ± 11% -23.4% 779.50 boot-time.idle
64.67 +45.5% 94.08 iostat.cpu.idle
32.22 ± 2% -92.9% 2.28 ± 3% iostat.cpu.iowait
460106 ± 33% +56.7% 720959 ± 15% numa-numastat.node0.local_node
461015 ± 33% +61.1% 742788 ± 15% numa-numastat.node0.numa_hit
598719 ± 25% -43.3% 339612 ± 34% numa-numastat.node1.local_node
639421 ± 24% -43.4% 361729 ± 31% numa-numastat.node1.numa_hit
61123 ± 15% -82.4% 10785 meminfo.Active(anon)
10906 ± 2% +490.4% 64393 ± 15% meminfo.Active(file)
1678 -85.8% 238.00 ± 14% meminfo.Dirty
31700 +351.3% 143073 ± 5% meminfo.Inactive(file)
283671 -52.2% 135570 meminfo.SUnreclaim
359419 -41.1% 211861 meminfo.Slab
64.47 +29.6 94.08 mpstat.cpu.all.idle%
32.42 ± 2% -30.1 2.29 ± 3% mpstat.cpu.all.iowait%
2.04 ± 9% +1.0 3.08 ± 12% mpstat.cpu.all.irq%
0.23 ± 6% -0.0 0.19 ± 2% mpstat.cpu.all.soft%
0.66 ± 2% -0.4 0.25 ± 3% mpstat.cpu.all.sys%
0.17 -0.1 0.11 ± 2% mpstat.cpu.all.usr%
64.25 +45.1% 93.25 vmstat.cpu.id
31.75 ± 2% -95.3% 1.50 ± 33% vmstat.cpu.wa
28682 -89.7% 2962 ± 2% vmstat.io.bo
46.50 -98.4% 0.75 ± 57% vmstat.procs.b
38969 -92.5% 2923 vmstat.system.cs
100537 -4.5% 96061 vmstat.system.in
1.306e+09 ± 7% -99.1% 11707061 ± 18% cpuidle.C1.time
5821744 ± 7% -96.9% 182330 ± 10% cpuidle.C1.usage
6.258e+09 ± 9% -59.8% 2.514e+09 ± 9% cpuidle.C1E.time
18768783 ± 4% -29.4% 13255968 ± 26% cpuidle.C1E.usage
6.397e+09 ± 9% +77.6% 1.136e+10 ± 2% cpuidle.C6.time
9966359 ± 7% +63.3% 16279745 ± 22% cpuidle.C6.usage
5156525 ± 14% -99.2% 40443 ± 13% cpuidle.POLL.time
3217284 ± 17% -99.4% 19436 ± 12% cpuidle.POLL.usage
12395 ± 7% +201.0% 37310 ± 23% numa-meminfo.node0.Active
10314 ± 6% +235.7% 34621 ± 21% numa-meminfo.node0.Active(file)
1636 -91.5% 138.75 ± 16% numa-meminfo.node0.Dirty
24039 ± 54% +264.0% 87496 ± 4% numa-meminfo.node0.Inactive(file)
219811 ± 4% -68.3% 69676 ± 4% numa-meminfo.node0.SUnreclaim
258497 ± 4% -57.4% 110004 ± 6% numa-meminfo.node0.Slab
59798 ± 14% -36.6% 37897 ± 6% numa-meminfo.node1.Active
59211 ± 15% -86.3% 8123 ± 15% numa-meminfo.node1.Active(anon)
587.00 ± 72% +4972.2% 29774 ± 11% numa-meminfo.node1.Active(file)
183687 ± 20% -31.4% 126043 ± 15% numa-meminfo.node1.AnonPages.max
7665 ±171% +624.8% 55557 ± 9% numa-meminfo.node1.Inactive(file)
139520 ±120% -81.7% 25573 ± 4% sched_debug.cfs_rq:/.load.avg
50.14 ± 19% -21.8% 39.22 ± 7% sched_debug.cfs_rq:/.load_avg.avg
0.11 ± 6% -22.2% 0.09 ± 7% sched_debug.cfs_rq:/.nr_running.avg
0.29 ± 5% -15.0% 0.25 sched_debug.cfs_rq:/.nr_running.stddev
122.82 ± 4% -16.2% 102.93 ± 4% sched_debug.cfs_rq:/.runnable_avg.avg
122.46 ± 4% -16.5% 102.27 ± 5% sched_debug.cfs_rq:/.util_avg.avg
101814 ± 11% +277.7% 384526 ± 27% sched_debug.cpu.avg_idle.min
220884 ± 5% -37.6% 137900 ± 13% sched_debug.cpu.avg_idle.stddev
1.10 ± 4% +17.5% 1.30 ± 3% sched_debug.cpu.clock.stddev
1669 ± 4% -63.3% 612.98 ± 6% sched_debug.cpu.clock_task.stddev
5275 ± 6% +10.7% 5839 sched_debug.cpu.curr->pid.max
106300 ± 10% -89.3% 11398 sched_debug.cpu.nr_switches.avg
1209385 ± 11% -97.4% 31808 ± 14% sched_debug.cpu.nr_switches.max
192228 ± 11% -97.3% 5222 ± 18% sched_debug.cpu.nr_switches.stddev
-251.12 -86.0% -35.25 sched_debug.cpu.nr_uninterruptible.min
40.50 ± 14% -64.2% 14.48 ± 24% sched_debug.cpu.nr_uninterruptible.stddev
2577 ± 6% +235.7% 8653 ± 21% numa-vmstat.node0.nr_active_file
10194 ± 42% +216.4% 32253 ± 5% numa-vmstat.node0.nr_dirtied
409.50 -91.6% 34.50 ± 15% numa-vmstat.node0.nr_dirty
6009 ± 54% +264.0% 21873 ± 4% numa-vmstat.node0.nr_inactive_file
54951 ± 4% -68.3% 17418 ± 4% numa-vmstat.node0.nr_slab_unreclaimable
9778 ± 43% +229.6% 32228 ± 5% numa-vmstat.node0.nr_written
2577 ± 6% +235.7% 8653 ± 21% numa-vmstat.node0.nr_zone_active_file
6009 ± 54% +264.0% 21873 ± 4% numa-vmstat.node0.nr_zone_inactive_file
414.75 ± 2% -92.5% 31.00 ± 13% numa-vmstat.node0.nr_zone_write_pending
14804 ± 15% -86.3% 2027 ± 15% numa-vmstat.node1.nr_active_anon
146.75 ± 72% +4970.4% 7440 ± 10% numa-vmstat.node1.nr_active_file
2882 ±147% +676.1% 22371 ± 5% numa-vmstat.node1.nr_dirtied
1915 ±171% +625.1% 13888 ± 9% numa-vmstat.node1.nr_inactive_file
2871 ±147% +678.4% 22351 ± 5% numa-vmstat.node1.nr_written
14804 ± 15% -86.3% 2027 ± 15% numa-vmstat.node1.nr_zone_active_anon
146.75 ± 72% +4970.4% 7440 ± 10% numa-vmstat.node1.nr_zone_active_file
1915 ±171% +625.1% 13888 ± 9% numa-vmstat.node1.nr_zone_inactive_file
15380 ± 15% -82.4% 2703 proc-vmstat.nr_active_anon
2725 ± 2% +490.4% 16094 ± 16% proc-vmstat.nr_active_file
17429 +471.3% 99580 ± 2% proc-vmstat.nr_dirtied
416.75 -86.0% 58.25 ± 13% proc-vmstat.nr_dirty
567440 +4.9% 595244 proc-vmstat.nr_file_pages
7926 +351.0% 35749 ± 5% proc-vmstat.nr_inactive_file
12783 +1.7% 13005 proc-vmstat.nr_kernel_stack
297875 -4.5% 284331 proc-vmstat.nr_shmem
70910 -52.2% 33895 proc-vmstat.nr_slab_unreclaimable
16905 +488.8% 99538 ± 2% proc-vmstat.nr_written
15380 ± 15% -82.4% 2703 proc-vmstat.nr_zone_active_anon
2725 ± 2% +490.4% 16094 ± 16% proc-vmstat.nr_zone_active_file
7926 +351.0% 35749 ± 5% proc-vmstat.nr_zone_inactive_file
424.75 -87.9% 51.50 ± 9% proc-vmstat.nr_zone_write_pending
42826 ± 14% -40.5% 25476 ± 3% proc-vmstat.pgactivate
1239967 -3.7% 1194026 proc-vmstat.pgalloc_normal
8740013 -89.7% 904228 ± 2% proc-vmstat.pgpgout
53472 ± 3% +5.6% 56482 proc-vmstat.pgreuse
158.00 ± 12% +348.6% 708.75 ± 2% slabinfo.biovec-128.active_objs
158.00 ± 12% +348.6% 708.75 ± 2% slabinfo.biovec-128.num_objs
374.25 ± 8% +258.8% 1342 ± 2% slabinfo.biovec-64.active_objs
374.25 ± 8% +258.8% 1342 ± 2% slabinfo.biovec-64.num_objs
163.50 ± 13% +227.8% 536.00 ± 7% slabinfo.biovec-max.active_objs
204.75 ± 13% +168.7% 550.25 ± 7% slabinfo.biovec-max.num_objs
1297 ± 14% -34.9% 845.25 ± 6% slabinfo.btrfs_delayed_tree_ref.active_objs
1297 ± 14% -34.9% 845.25 ± 6% slabinfo.btrfs_delayed_tree_ref.num_objs
2006 ± 13% +59.9% 3209 slabinfo.btrfs_extent_buffer.active_objs
2006 ± 13% +59.9% 3209 slabinfo.btrfs_extent_buffer.num_objs
336.00 ± 23% -31.2% 231.00 ± 15% slabinfo.kmalloc-rcl-192.active_objs
336.00 ± 23% -31.2% 231.00 ± 15% slabinfo.kmalloc-rcl-192.num_objs
1079929 -99.6% 4015 slabinfo.pid_namespace.active_objs
19284 -99.6% 71.25 slabinfo.pid_namespace.active_slabs
1079934 -99.6% 4015 slabinfo.pid_namespace.num_objs
19284 -99.6% 71.25 slabinfo.pid_namespace.num_slabs
681.25 ± 10% +233.8% 2274 ± 12% slabinfo.skbuff_fclone_cache.active_objs
681.25 ± 10% +233.8% 2274 ± 12% slabinfo.skbuff_fclone_cache.num_objs
705679 ± 2% -96.8% 22805 softirqs.BLOCK
56594 ± 12% -36.3% 36023 ± 19% softirqs.CPU0.RCU
61813 ± 12% -41.2% 36360 ± 13% softirqs.CPU1.RCU
61170 ± 12% -41.9% 35511 ± 4% softirqs.CPU10.RCU
59336 ± 10% -37.7% 36972 ± 5% softirqs.CPU11.RCU
58606 ± 14% -36.7% 37114 ± 17% softirqs.CPU2.RCU
54177 ± 12% -40.1% 32471 ± 12% softirqs.CPU24.RCU
55834 ± 11% -36.9% 35230 ± 10% softirqs.CPU25.RCU
53216 ± 9% -31.9% 36262 ± 15% softirqs.CPU26.RCU
53744 ± 12% -35.7% 34533 ± 16% softirqs.CPU27.RCU
51611 ± 8% -33.8% 34181 ± 13% softirqs.CPU28.RCU
53726 ± 7% -33.5% 35754 ± 9% softirqs.CPU29.RCU
61015 ± 14% -41.3% 35804 ± 18% softirqs.CPU3.RCU
54715 ± 12% -37.5% 34188 ± 3% softirqs.CPU30.RCU
51435 ± 10% -33.5% 34224 ± 7% softirqs.CPU31.RCU
52697 ± 8% -38.8% 32238 ± 5% softirqs.CPU32.RCU
27579 ± 7% +49.8% 41304 ± 3% softirqs.CPU32.SCHED
48705 ± 9% -32.2% 33031 ± 4% softirqs.CPU33.RCU
48234 ± 8% -29.0% 34223 ± 3% softirqs.CPU34.RCU
49281 ± 10% -29.7% 34654 ± 3% softirqs.CPU35.RCU
54023 ± 8% -35.2% 34993 ± 19% softirqs.CPU4.RCU
55222 ± 26% -34.4% 36234 ± 19% softirqs.CPU40.RCU
59824 ± 8% -37.8% 37225 ± 14% softirqs.CPU5.RCU
62723 ± 13% -41.6% 36615 ± 9% softirqs.CPU6.RCU
56745 ± 13% -33.6% 37668 ± 11% softirqs.CPU7.RCU
685836 -98.4% 10922 ± 2% softirqs.CPU8.BLOCK
63989 ± 9% -41.2% 37647 ± 5% softirqs.CPU8.RCU
693875 -96.6% 23342 softirqs.CPU8.TASKLET
60701 ± 10% -41.2% 35709 ± 9% softirqs.CPU9.RCU
2511282 ± 18% -31.4% 1721969 ± 10% softirqs.RCU
694098 -96.6% 23565 softirqs.TASKLET
2089 ± 62% -80.3% 410.75 ± 91% interrupts.41:PCI-MSI.2621447-edge.eth0-TxRx-6
1385835 -96.8% 44728 interrupts.73:PCI-MSI.2097152-edge.isci-msix
774.25 ± 60% -87.9% 94.00 ± 35% interrupts.CPU1.RES:Rescheduling_interrupts
1167 ± 14% -85.9% 164.50 ± 63% interrupts.CPU10.RES:Rescheduling_interrupts
1041 ± 18% -82.2% 185.25 ± 48% interrupts.CPU11.RES:Rescheduling_interrupts
188.00 ± 23% +32.0% 248.25 ± 26% interrupts.CPU14.NMI:Non-maskable_interrupts
188.00 ± 23% +32.0% 248.25 ± 26% interrupts.CPU14.PMI:Performance_monitoring_interrupts
113.75 ± 29% -44.2% 63.50 ± 15% interrupts.CPU16.RES:Rescheduling_interrupts
470.50 ± 38% -83.7% 76.50 ± 39% interrupts.CPU2.RES:Rescheduling_interrupts
880.75 ± 24% -87.1% 113.75 ± 54% interrupts.CPU24.RES:Rescheduling_interrupts
320.50 ± 27% -37.6% 200.00 ± 28% interrupts.CPU25.NMI:Non-maskable_interrupts
320.50 ± 27% -37.6% 200.00 ± 28% interrupts.CPU25.PMI:Performance_monitoring_interrupts
640.00 ± 28% -85.5% 92.50 ± 59% interrupts.CPU25.RES:Rescheduling_interrupts
572.00 ± 30% -85.7% 81.75 ± 42% interrupts.CPU26.RES:Rescheduling_interrupts
582.25 ± 25% -85.1% 86.50 ± 40% interrupts.CPU27.RES:Rescheduling_interrupts
474.00 ± 20% -82.4% 83.25 ± 24% interrupts.CPU28.RES:Rescheduling_interrupts
375.00 ± 13% -74.7% 95.00 ± 29% interrupts.CPU29.RES:Rescheduling_interrupts
527.50 ± 46% -82.2% 93.75 ± 46% interrupts.CPU3.RES:Rescheduling_interrupts
949.00 ± 10% +56.0% 1480 ± 37% interrupts.CPU30.CAL:Function_call_interrupts
447.75 ± 30% -70.2% 133.50 ± 48% interrupts.CPU30.RES:Rescheduling_interrupts
892.25 ± 2% +26.9% 1132 ± 15% interrupts.CPU31.CAL:Function_call_interrupts
2089 ± 62% -80.3% 410.75 ± 91% interrupts.CPU32.41:PCI-MSI.2621447-edge.eth0-TxRx-6
968.00 ± 10% +94.1% 1878 ± 59% interrupts.CPU32.CAL:Function_call_interrupts
826.50 ± 28% -71.7% 233.75 ± 31% interrupts.CPU32.NMI:Non-maskable_interrupts
826.50 ± 28% -71.7% 233.75 ± 31% interrupts.CPU32.PMI:Performance_monitoring_interrupts
656.25 ± 15% -75.5% 160.75 ± 64% interrupts.CPU32.RES:Rescheduling_interrupts
558.25 ± 32% -75.0% 139.50 ± 55% interrupts.CPU33.RES:Rescheduling_interrupts
447.25 ± 16% -61.8% 171.00 ± 47% interrupts.CPU34.RES:Rescheduling_interrupts
504.50 ± 33% -81.9% 91.25 ± 32% interrupts.CPU35.RES:Rescheduling_interrupts
301.50 ± 40% -77.9% 66.75 ± 62% interrupts.CPU4.RES:Rescheduling_interrupts
315.25 ± 43% -65.9% 107.50 ± 81% interrupts.CPU5.RES:Rescheduling_interrupts
491.75 ± 57% -86.3% 67.25 ± 35% interrupts.CPU6.RES:Rescheduling_interrupts
1385835 -96.8% 44728 interrupts.CPU8.73:PCI-MSI.2097152-edge.isci-msix
916.25 ± 9% +26.0% 1154 ± 18% interrupts.CPU8.CAL:Function_call_interrupts
827.25 ± 38% -75.0% 206.50 ± 39% interrupts.CPU8.NMI:Non-maskable_interrupts
827.25 ± 38% -75.0% 206.50 ± 39% interrupts.CPU8.PMI:Performance_monitoring_interrupts
906.25 ± 2% +53.9% 1395 ± 47% interrupts.CPU9.CAL:Function_call_interrupts
1107 ± 10% -87.6% 137.50 ± 49% interrupts.CPU9.RES:Rescheduling_interrupts
13410 ± 5% -16.0% 11263 ± 7% interrupts.NMI:Non-maskable_interrupts
13410 ± 5% -16.0% 11263 ± 7% interrupts.PMI:Performance_monitoring_interrupts
15515 ± 16% -69.0% 4813 ± 14% interrupts.RES:Rescheduling_interrupts
0.02 ± 49% +181.5% 0.05 ± 72% perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.01 ± 8% +35.1% 0.01 ± 8% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.03 ± 59% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.01 ± 14% +287.0% 0.02 ± 16% perf-sched.sch_delay.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
0.01 ± 57% -58.8% 0.00 ± 14% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.smpboot_thread_fn.kthread.ret_from_fork
0.00 ±118% +350.0% 0.02 ± 68% perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
0.00 ±100% +412.5% 0.01 ± 21% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
0.01 ± 49% +86.5% 0.02 ± 8% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.00 ± 57% +733.3% 0.01 ± 18% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.02 ± 9% +37.7% 0.02 ± 10% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ± 7% +73.1% 0.01 ± 3% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
0.03 ± 34% +72.1% 0.06 ± 18% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.03 ± 59% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.03 ±110% +197.4% 0.08 ± 32% perf-sched.sch_delay.max.ms.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range
0.01 ±110% +274.4% 0.04 ± 64% perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
0.03 ± 60% +91.0% 0.06 ± 10% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.02 ± 31% +130.6% 0.04 ± 12% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.04 ± 18% +26.8% 0.05 ± 7% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ± 11% +125.0% 0.01 ± 3% perf-sched.total_sch_delay.average.ms
10.12 ± 21% +1307.5% 142.39 perf-sched.total_wait_and_delay.average.ms
172750 ± 17% -91.4% 14933 ± 2% perf-sched.total_wait_and_delay.count.ms
5885 ± 25% +56.9% 9234 ± 11% perf-sched.total_wait_and_delay.max.ms
10.11 ± 21% +1308.2% 142.38 perf-sched.total_wait_time.average.ms
5885 ± 25% +56.9% 9234 ± 11% perf-sched.total_wait_time.max.ms
8.01 ± 22% +1181.9% 102.62 ± 62% perf-sched.wait_and_delay.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
1.02 ±173% +314.1% 4.24 ± 3% perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
232.13 ± 18% +103.1% 471.43 ± 36% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
4.54 ± 6% -8.7% 4.14 ± 4% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
393.96 ± 14% +23.3% 485.81 ± 5% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
4.09 ± 26% +6387.8% 265.12 ± 2% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
61985 ± 18% -98.0% 1211 ± 2% perf-sched.wait_and_delay.count.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
2.00 ±173% +4850.0% 99.00 ± 18% perf-sched.wait_and_delay.count.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range
83.50 ± 16% -71.0% 24.25 ± 6% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
1547 ± 7% +16.4% 1801 ± 3% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
102506 ± 17% -97.7% 2395 ± 7% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
943.35 ± 8% -81.1% 178.11 ± 56% perf-sched.wait_and_delay.max.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
1.26 ±173% +304.5% 5.11 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
1589 ± 52% +190.9% 4624 ± 25% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.00 ±173% +1.2e+05% 5.20 ± 73% perf-sched.wait_time.avg.ms.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork
3.92 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
8.00 ± 22% +1182.5% 102.60 ± 62% perf-sched.wait_time.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
0.28 ±155% +24627.1% 69.30 ±106% perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
232.13 ± 18% +103.1% 471.42 ± 36% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
4.53 ± 6% -8.9% 4.13 ± 4% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
393.96 ± 14% +23.3% 485.80 ± 5% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
4.08 ± 26% +6398.3% 265.11 ± 2% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
0.00 ±173% +4e+05% 17.17 ±107% perf-sched.wait_time.max.ms.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork
3.92 ± 11% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
943.35 ± 8% -81.1% 178.04 ± 56% perf-sched.wait_time.max.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
6.02 ±172% +2236.8% 140.75 ± 90% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
0.01 ± 86% +7423.8% 0.79 ±167% perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io.write_all_supers
1589 ± 52% +190.9% 4624 ± 25% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
44.03 +20.3% 52.96 ± 12% perf-stat.i.MPKI
2.589e+08 -36.7% 1.639e+08 perf-stat.i.branch-instructions
5.01 +6.7 11.71 ± 7% perf-stat.i.branch-miss-rate%
12714258 +31.8% 16758440 ± 7% perf-stat.i.branch-misses
9.35 ± 2% +14.7 24.09 ± 3% perf-stat.i.cache-miss-rate%
4375264 +81.1% 7923317 ± 18% perf-stat.i.cache-misses
48162798 -32.2% 32638947 ± 14% perf-stat.i.cache-references
39282 -92.6% 2916 perf-stat.i.context-switches
4.10 +83.9% 7.54 ± 6% perf-stat.i.cpi
68.61 ± 2% +64.8% 113.06 perf-stat.i.cpu-migrations
1061 ± 3% -42.2% 613.59 ± 8% perf-stat.i.cycles-between-cache-misses
1.23 ± 2% -0.2 1.06 ± 8% perf-stat.i.dTLB-load-miss-rate%
3992778 ± 2% -47.7% 2089367 ± 11% perf-stat.i.dTLB-load-misses
3.459e+08 -35.4% 2.234e+08 ± 2% perf-stat.i.dTLB-loads
0.17 ± 2% +0.0 0.20 ± 4% perf-stat.i.dTLB-store-miss-rate%
70.32 +21.5 91.83 perf-stat.i.iTLB-load-miss-rate%
1985999 ± 2% -39.6% 1199619 ± 2% perf-stat.i.iTLB-load-misses
854405 -87.4% 107547 ± 14% perf-stat.i.iTLB-loads
1.254e+09 -36.9% 7.911e+08 perf-stat.i.instructions
0.26 -41.9% 0.15 ± 6% perf-stat.i.ipc
1.34 ± 7% -58.0% 0.56 ± 31% perf-stat.i.metric.K/sec
19.36 -25.9% 14.35 ± 6% perf-stat.i.metric.M/sec
2893 -2.0% 2835 perf-stat.i.minor-faults
197998 ± 19% -32.9% 132808 ± 3% perf-stat.i.node-load-misses
228908 ± 17% -31.8% 156142 ± 3% perf-stat.i.node-loads
236295 ± 7% -17.5% 194871 ± 6% perf-stat.i.node-stores
3820 -1.7% 3754 perf-stat.i.page-faults
4.91 +5.3 10.21 ± 7% perf-stat.overall.branch-miss-rate%
9.09 ± 2% +15.1 24.16 ± 3% perf-stat.overall.cache-miss-rate%
3.65 +65.6% 6.04 ± 7% perf-stat.overall.cpi
1045 ± 3% -41.3% 613.83 ± 8% perf-stat.overall.cycles-between-cache-misses
1.14 ± 2% -0.2 0.92 ± 8% perf-stat.overall.dTLB-load-miss-rate%
0.17 ± 3% +0.0 0.19 ± 4% perf-stat.overall.dTLB-store-miss-rate%
69.91 +21.9 91.81 perf-stat.overall.iTLB-load-miss-rate%
0.27 -39.3% 0.17 ± 6% perf-stat.overall.ipc
175094 +3617.1% 6508495 perf-stat.overall.path-length
2.583e+08 -36.7% 1.636e+08 perf-stat.ps.branch-instructions
12678701 +31.8% 16711512 ± 7% perf-stat.ps.branch-misses
4361806 +81.1% 7898239 ± 18% perf-stat.ps.cache-misses
48005640 -32.2% 32532931 ± 14% perf-stat.ps.cache-references
39151 -92.6% 2907 perf-stat.ps.context-switches
68.42 ± 2% +64.7% 112.70 perf-stat.ps.cpu-migrations
3979673 ± 2% -47.7% 2082707 ± 11% perf-stat.ps.dTLB-load-misses
3.449e+08 -35.4% 2.229e+08 ± 2% perf-stat.ps.dTLB-loads
1979429 ± 2% -39.6% 1195674 ± 2% perf-stat.ps.iTLB-load-misses
851561 -87.4% 107209 ± 14% perf-stat.ps.iTLB-loads
1.251e+09 -36.9% 7.896e+08 perf-stat.ps.instructions
2884 -2.0% 2827 perf-stat.ps.minor-faults
197548 ± 19% -33.0% 132439 ± 3% perf-stat.ps.node-load-misses
228425 ± 17% -31.8% 155760 ± 3% perf-stat.ps.node-loads
236208 ± 7% -17.3% 195231 ± 6% perf-stat.ps.node-stores
3.787e+11 -36.8% 2.393e+11 perf-stat.total.instructions
10.21 ± 20% -9.9 0.36 ±100% perf-profile.calltrace.cycles-pp.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
8.66 ± 23% -8.5 0.17 ±173% perf-profile.calltrace.cycles-pp.iomap_apply.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write.vfs_write
11.16 ± 19% -7.3 3.87 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
11.02 ± 19% -7.2 3.84 ± 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.77 ± 19% -7.2 3.59 ± 6% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.72 ± 19% -7.1 3.59 ± 6% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.54 ± 19% -6.9 3.59 ± 6% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.51 ± 20% -6.9 3.59 ± 6% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
6.28 ± 31% -6.1 0.14 ±173% perf-profile.calltrace.cycles-pp.btrfs_dio_iomap_begin.iomap_apply.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write
6.48 ± 12% -4.6 1.87 ± 23% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
6.63 ± 12% -4.5 2.11 ± 22% perf-profile.calltrace.cycles-pp.ret_from_fork
6.63 ± 12% -4.5 2.11 ± 22% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
5.90 ± 11% -4.1 1.82 ± 23% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
0.91 ± 8% +0.3 1.17 ± 5% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
1.16 ± 12% +0.3 1.43 ± 6% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
0.64 ± 9% +0.3 0.93 ± 8% perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack
0.99 ± 7% +0.3 1.32 ± 12% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.65 ± 10% +0.3 0.98 ± 11% perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu
0.26 ±100% +0.4 0.68 ± 14% perf-profile.calltrace.cycles-pp.cpuidle_governor_latency_req.menu_select.do_idle.cpu_startup_entry.start_secondary
1.29 ± 6% +0.4 1.71 ± 14% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
1.56 ± 9% +0.4 1.99 ± 3% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
1.75 ± 10% +0.5 2.27 ± 5% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
0.67 ± 65% +0.5 1.19 ± 8% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter
0.67 ± 65% +0.5 1.19 ± 8% perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
0.67 ± 65% +0.5 1.19 ± 8% perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter.do_idle
0.67 ± 65% +0.5 1.19 ± 8% perf-profile.calltrace.cycles-pp.__sysvec_irq_work.asm_call_sysvec_on_stack.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state
0.00 +0.6 0.60 ± 6% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file
0.00 +0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
0.14 ±173% +0.6 0.74 ± 7% perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
0.00 +0.6 0.62 ± 13% perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
0.00 +0.6 0.62 ± 5% perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
0.80 ± 58% +0.6 1.45 ± 17% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
0.80 ± 58% +0.6 1.45 ± 17% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
0.15 ±173% +0.7 0.83 ± 12% perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
0.00 +0.7 0.72 ± 5% perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
0.00 +0.7 0.73 ± 20% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node
0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
2.60 ± 14% +0.9 3.48 ± 10% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.00 +0.9 0.90 ± 5% perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
2.73 ± 7% +1.1 3.81 ± 4% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
2.92 ± 7% +1.2 4.08 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
4.36 ± 15% +1.2 5.60 ± 9% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.03 ± 13% +1.2 2.28 ± 33% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
1.05 ± 13% +1.3 2.31 ± 33% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
3.44 ± 8% +1.3 4.72 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
1.15 ± 15% +1.4 2.54 ± 32% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
1.15 ± 15% +1.4 2.54 ± 32% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
1.15 ± 15% +1.4 2.54 ± 32% perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify
0.00 +1.7 1.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
0.00 +1.7 1.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file
0.00 +1.9 1.87 ± 18% perf-profile.calltrace.cycles-pp.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter
0.00 +2.0 2.02 ± 17% perf-profile.calltrace.cycles-pp.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
0.00 +2.0 2.03 ± 17% perf-profile.calltrace.cycles-pp.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
5.73 ± 7% +2.3 8.03 ± 3% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
8.28 ± 10% +2.5 10.75 ± 4% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
8.66 ± 9% +2.7 11.31 ± 4% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
8.79 ± 9% +2.7 11.50 ± 4% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.00 +3.0 3.00 ± 11% perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
13.50 ± 9% +4.3 17.84 ± 7% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
15.39 ± 9% +4.7 20.10 ± 7% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
51.90 +8.8 60.72 ± 2% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
72.26 ± 2% +9.9 82.17 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
71.31 ± 2% +9.9 81.25 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
79.02 ± 3% +10.8 89.79 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
79.10 ± 3% +10.8 89.91 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
79.10 ± 3% +10.8 89.91 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
80.25 ± 2% +12.2 92.45 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
10.22 ± 20% -9.6 0.58 ± 26% perf-profile.children.cycles-pp.__iomap_dio_rw
8.66 ± 23% -8.2 0.45 ± 32% perf-profile.children.cycles-pp.iomap_apply
10.79 ± 19% -7.2 3.62 ± 6% perf-profile.children.cycles-pp.ksys_write
12.01 ± 17% -7.1 4.86 ± 8% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
10.74 ± 19% -7.1 3.62 ± 6% perf-profile.children.cycles-pp.vfs_write
11.77 ± 17% -7.0 4.73 ± 8% perf-profile.children.cycles-pp.do_syscall_64
10.56 ± 19% -6.9 3.62 ± 6% perf-profile.children.cycles-pp.new_sync_write
10.52 ± 20% -6.9 3.59 ± 6% perf-profile.children.cycles-pp.btrfs_file_write_iter
6.28 ± 31% -5.9 0.38 ± 36% perf-profile.children.cycles-pp.btrfs_dio_iomap_begin
6.48 ± 12% -4.6 1.87 ± 23% perf-profile.children.cycles-pp.worker_thread
6.63 ± 12% -4.5 2.11 ± 22% perf-profile.children.cycles-pp.kthread
6.63 ± 12% -4.5 2.12 ± 22% perf-profile.children.cycles-pp.ret_from_fork
5.91 ± 11% -4.1 1.82 ± 23% perf-profile.children.cycles-pp.process_one_work
3.45 ± 24% -3.3 0.14 ± 55% perf-profile.children.cycles-pp.btrfs_work_helper
3.43 ± 24% -3.3 0.14 ± 55% perf-profile.children.cycles-pp.btrfs_finish_ordered_io
2.77 ± 43% -2.7 0.04 ±106% perf-profile.children.cycles-pp.asm_common_interrupt
2.77 ± 43% -2.7 0.04 ±106% perf-profile.children.cycles-pp.common_interrupt
2.33 ± 8% -2.3 0.04 ±107% perf-profile.children.cycles-pp.can_nocow_extent
2.21 ± 12% -2.1 0.06 ± 67% perf-profile.children.cycles-pp.iomap_dio_bio_actor
2.01 ± 42% -1.9 0.08 ± 17% perf-profile.children.cycles-pp.blk_done_softirq
1.95 ± 42% -1.9 0.07 ± 17% perf-profile.children.cycles-pp.scsi_io_completion
1.95 ± 42% -1.9 0.07 ± 17% perf-profile.children.cycles-pp.scsi_end_request
2.13 ± 94% -1.9 0.27 ± 49% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_space
2.17 ± 12% -1.8 0.37 ± 27% perf-profile.children.cycles-pp.__schedule
2.51 ± 96% -1.7 0.77 ± 25% perf-profile.children.cycles-pp._raw_spin_lock
2.00 ±100% -1.7 0.26 ± 51% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
1.92 ±104% -1.6 0.29 ± 47% perf-profile.children.cycles-pp.__reserve_bytes
1.68 ± 13% -1.6 0.05 ± 62% perf-profile.children.cycles-pp.iomap_dio_submit_bio
1.90 ±105% -1.6 0.28 ± 48% perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes
1.66 ± 12% -1.6 0.05 ± 62% perf-profile.children.cycles-pp.btrfs_submit_direct
1.43 ± 41% -1.4 0.04 ±100% perf-profile.children.cycles-pp.blk_update_request
1.35 ± 41% -1.3 0.03 ±100% perf-profile.children.cycles-pp.btrfs_end_bio
1.50 ± 13% -1.3 0.24 ± 17% perf-profile.children.cycles-pp.schedule
1.11 ± 12% -1.1 0.04 ±100% perf-profile.children.cycles-pp.btrfs_map_bio
0.97 ± 18% -0.9 0.03 ±100% perf-profile.children.cycles-pp.io_schedule
0.98 ± 32% -0.9 0.09 ± 16% perf-profile.children.cycles-pp.__blk_mq_run_hw_queue
0.97 ± 32% -0.9 0.09 ± 16% perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
0.96 ± 33% -0.9 0.09 ± 16% perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
0.92 ± 33% -0.8 0.09 ± 21% perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched
0.89 ± 4% -0.8 0.05 ± 61% perf-profile.children.cycles-pp.read_block_for_search
0.94 ± 32% -0.8 0.15 ± 39% perf-profile.children.cycles-pp.try_to_wake_up
0.85 ± 16% -0.7 0.16 ± 67% perf-profile.children.cycles-pp.btrfs_csum_file_blocks
0.69 ± 34% -0.6 0.06 ± 17% perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list
0.73 ± 9% -0.6 0.14 ± 47% perf-profile.children.cycles-pp.schedule_idle
0.65 ± 33% -0.6 0.06 ± 17% perf-profile.children.cycles-pp.scsi_queue_rq
0.65 ± 30% -0.5 0.12 ± 33% perf-profile.children.cycles-pp.pick_next_task_fair
0.56 ± 10% -0.5 0.09 ± 13% perf-profile.children.cycles-pp.kmem_cache_alloc
0.42 ± 32% -0.4 0.04 ± 58% perf-profile.children.cycles-pp.sas_queuecommand
0.39 ± 33% -0.4 0.03 ±100% perf-profile.children.cycles-pp.__ata_scsi_queuecmd
0.43 ± 8% -0.4 0.07 ± 22% perf-profile.children.cycles-pp.dequeue_task_fair
0.43 ± 18% -0.3 0.09 ± 28% perf-profile.children.cycles-pp.blk_finish_plug
0.42 ± 19% -0.3 0.09 ± 28% perf-profile.children.cycles-pp.blk_flush_plug_list
0.42 ± 18% -0.3 0.09 ± 28% perf-profile.children.cycles-pp.blk_mq_flush_plug_list
0.40 ± 22% -0.3 0.09 ± 28% perf-profile.children.cycles-pp.blk_mq_sched_insert_requests
0.38 ± 25% -0.3 0.08 ± 30% perf-profile.children.cycles-pp.ttwu_do_activate
0.37 ± 40% -0.3 0.07 ± 20% perf-profile.children.cycles-pp.newidle_balance
0.34 ± 26% -0.3 0.05 ± 60% perf-profile.children.cycles-pp.kmem_cache_free
0.35 ± 6% -0.3 0.07 ± 21% perf-profile.children.cycles-pp.dequeue_entity
0.33 ± 22% -0.2 0.08 ± 13% perf-profile.children.cycles-pp.enqueue_task_fair
0.36 ± 24% -0.2 0.16 ± 9% perf-profile.children.cycles-pp.update_load_avg
0.26 ± 23% -0.2 0.07 ± 7% perf-profile.children.cycles-pp.enqueue_entity
0.29 ± 22% -0.2 0.12 ± 81% perf-profile.children.cycles-pp.btrfs_lookup_csum
0.18 ± 20% -0.2 0.03 ±100% perf-profile.children.cycles-pp.set_next_entity
0.12 ± 22% -0.1 0.03 ±100% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.21 ± 9% -0.1 0.13 ± 31% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.12 ± 15% -0.1 0.05 ± 67% perf-profile.children.cycles-pp.__wake_up_common_lock
0.11 ± 20% -0.1 0.04 ± 58% perf-profile.children.cycles-pp.__update_load_avg_se
0.15 ± 20% -0.1 0.08 ± 13% perf-profile.children.cycles-pp.__blk_mq_delay_run_hw_queue
0.79 ± 4% -0.1 0.73 ± 4% perf-profile.children.cycles-pp.read_tsc
0.12 ± 23% -0.0 0.08 ± 30% perf-profile.children.cycles-pp.read
0.07 ± 7% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.path_openat
0.09 ± 16% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.rcu_nmi_exit
0.07 ± 7% +0.0 0.10 ± 12% perf-profile.children.cycles-pp.do_filp_open
0.07 ± 19% +0.0 0.11 ± 23% perf-profile.children.cycles-pp.rcu_segcblist_ready_cbs
0.08 ± 14% +0.0 0.12 ± 6% perf-profile.children.cycles-pp.tick_program_event
0.01 ±173% +0.0 0.06 ± 11% perf-profile.children.cycles-pp.run_timer_softirq
0.05 ± 62% +0.1 0.10 ± 18% perf-profile.children.cycles-pp.perf_pmu_disable
0.01 ±173% +0.1 0.07 ± 26% perf-profile.children.cycles-pp.cpuidle_get_cpu_driver
0.03 ±100% +0.1 0.09 ± 23% perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick
0.21 ± 10% +0.1 0.27 ± 9% perf-profile.children.cycles-pp.menu_reflect
0.03 ±102% +0.1 0.09 ± 24% perf-profile.children.cycles-pp.rcu_is_cpu_rrupt_from_idle
0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush
0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.btrfs_alloc_tree_block
0.19 ± 12% +0.1 0.26 ± 11% perf-profile.children.cycles-pp.rb_next
0.16 ± 14% +0.1 0.23 ± 10% perf-profile.children.cycles-pp.irqentry_exit
0.07 ± 11% +0.1 0.14 ± 25% perf-profile.children.cycles-pp.rcu_needs_cpu
0.03 ±102% +0.1 0.10 ± 30% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.01 ±173% +0.1 0.09 ± 45% perf-profile.children.cycles-pp.leave_mm
0.14 ± 19% +0.1 0.22 ± 16% perf-profile.children.cycles-pp.pm_qos_read_value
0.00 +0.1 0.08 ± 37% perf-profile.children.cycles-pp.cpu_latency_qos_limit
0.22 ± 18% +0.1 0.30 ± 14% perf-profile.children.cycles-pp.get_cpu_device
0.02 ±173% +0.1 0.10 ± 35% perf-profile.children.cycles-pp.sched_idle_set_state
0.07 ± 69% +0.1 0.15 ± 19% perf-profile.children.cycles-pp.hrtimer_forward
0.14 ± 10% +0.1 0.23 ± 14% perf-profile.children.cycles-pp.perf_event_task_tick
0.10 ± 15% +0.1 0.19 ± 9% perf-profile.children.cycles-pp.x86_pmu_disable
0.08 ± 67% +0.1 0.17 ± 15% perf-profile.children.cycles-pp.cpuidle_reflect
0.00 +0.1 0.09 ± 27% perf-profile.children.cycles-pp.check_leaf
0.07 ± 58% +0.1 0.17 ± 26% perf-profile.children.cycles-pp.run_local_timers
0.00 +0.1 0.10 ± 27% perf-profile.children.cycles-pp.btrfs_cow_block
0.00 +0.1 0.10 ± 27% perf-profile.children.cycles-pp.__btrfs_cow_block
0.00 +0.1 0.10 ± 25% perf-profile.children.cycles-pp.btree_csum_one_bio
0.17 ± 14% +0.1 0.27 ± 33% perf-profile.children.cycles-pp.__hrtimer_get_next_event
0.00 +0.1 0.10 ± 35% perf-profile.children.cycles-pp.write_one_eb
0.00 +0.1 0.10 ± 27% perf-profile.children.cycles-pp.btrfs_submit_metadata_bio
0.43 ± 13% +0.1 0.53 ± 5% perf-profile.children.cycles-pp.update_irq_load_avg
0.10 ± 18% +0.1 0.21 ± 14% perf-profile.children.cycles-pp.cpuidle_not_available
0.00 +0.1 0.11 ± 23% perf-profile.children.cycles-pp.submit_one_bio
0.16 ± 35% +0.1 0.27 ± 25% perf-profile.children.cycles-pp.wait_consider_task
0.16 ± 19% +0.1 0.28 ± 24% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
0.18 ± 32% +0.1 0.30 ± 19% perf-profile.children.cycles-pp.irqtime_account_process_tick
0.00 +0.1 0.12 ± 23% perf-profile.children.cycles-pp.btrfs_write_marked_extents
0.00 +0.1 0.12 ± 23% perf-profile.children.cycles-pp.btree_write_cache_pages
0.00 +0.1 0.13 ± 17% perf-profile.children.cycles-pp.do_writepages
0.00 +0.1 0.14 ± 19% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
1.08 ± 11% +0.2 1.23 ± 7% perf-profile.children.cycles-pp.io_serial_in
0.39 ± 10% +0.2 0.55 ± 3% perf-profile.children.cycles-pp.arch_scale_freq_tick
0.00 +0.2 0.17 ± 55% perf-profile.children.cycles-pp.btrfs_log_changed_extents
0.42 ± 5% +0.2 0.63 ± 6% perf-profile.children.cycles-pp.timerqueue_del
0.32 ± 14% +0.2 0.54 ± 11% perf-profile.children.cycles-pp.timerqueue_add
0.72 ± 11% +0.2 0.95 ± 8% perf-profile.children.cycles-pp.update_blocked_averages
0.49 ± 8% +0.2 0.72 ± 13% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.52 ± 6% +0.2 0.76 ± 7% perf-profile.children.cycles-pp.__remove_hrtimer
0.38 ± 16% +0.3 0.64 ± 13% perf-profile.children.cycles-pp.enqueue_hrtimer
0.96 ± 7% +0.3 1.22 ± 5% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
0.68 ± 11% +0.3 1.00 ± 11% perf-profile.children.cycles-pp.run_rebalance_domains
0.53 ± 11% +0.3 0.86 ± 13% perf-profile.children.cycles-pp.rcu_sched_clock_irq
1.22 ± 10% +0.3 1.56 ± 6% perf-profile.children.cycles-pp.tick_nohz_next_event
1.03 ± 6% +0.4 1.39 ± 13% perf-profile.children.cycles-pp.tick_irq_enter
1.38 ± 5% +0.4 1.78 ± 14% perf-profile.children.cycles-pp.irq_enter_rcu
1.64 ± 9% +0.4 2.08 ± 3% perf-profile.children.cycles-pp.scheduler_tick
1.81 ± 11% +0.6 2.36 ± 5% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.23 ± 18% +0.6 0.83 ± 36% perf-profile.children.cycles-pp.__btrfs_tree_read_lock
0.00 +0.7 0.72 ± 5% perf-profile.children.cycles-pp.wait_log_commit
0.06 ± 74% +0.8 0.81 ± 37% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
0.00 +0.8 0.76 ± 3% perf-profile.children.cycles-pp.__mutex_lock
0.03 ±100% +0.8 0.79 ± 14% perf-profile.children.cycles-pp.__btrfs_tree_lock
0.03 ±100% +0.8 0.79 ± 14% perf-profile.children.cycles-pp.btrfs_lock_root_node
0.00 +0.8 0.79 ± 14% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
0.00 +0.9 0.90 ± 5% perf-profile.children.cycles-pp.btrfs_sync_log
2.83 ± 6% +1.1 3.93 ± 3% perf-profile.children.cycles-pp.update_process_times
3.01 ± 6% +1.2 4.20 ± 4% perf-profile.children.cycles-pp.tick_sched_handle
3.54 ± 7% +1.3 4.85 ± 3% perf-profile.children.cycles-pp.tick_sched_timer
4.44 ± 14% +1.3 5.78 ± 10% perf-profile.children.cycles-pp.menu_select
1.15 ± 15% +1.4 2.54 ± 32% perf-profile.children.cycles-pp.start_kernel
0.08 ± 78% +1.5 1.60 ± 20% perf-profile.children.cycles-pp.rwsem_optimistic_spin
0.00 +1.7 1.71 ± 19% perf-profile.children.cycles-pp.btrfs_insert_empty_items
0.00 +1.9 1.87 ± 18% perf-profile.children.cycles-pp.btrfs_log_inode
0.00 +2.0 2.02 ± 17% perf-profile.children.cycles-pp.btrfs_log_inode_parent
0.00 +2.0 2.03 ± 17% perf-profile.children.cycles-pp.btrfs_log_dentry_safe
0.04 ±115% +2.2 2.23 ± 14% perf-profile.children.cycles-pp.osq_lock
5.92 ± 6% +2.3 8.26 ± 3% perf-profile.children.cycles-pp.__hrtimer_run_queues
8.53 ± 9% +2.5 11.05 ± 4% perf-profile.children.cycles-pp.hrtimer_interrupt
8.91 ± 9% +2.7 11.63 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.00 +3.0 3.00 ± 11% perf-profile.children.cycles-pp.btrfs_sync_file
13.89 ± 9% +4.2 18.09 ± 6% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
15.08 ± 9% +4.4 19.51 ± 6% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
51.91 +8.8 60.72 ± 2% perf-profile.children.cycles-pp.intel_idle
79.10 ± 3% +10.8 89.91 perf-profile.children.cycles-pp.start_secondary
73.22 ± 2% +11.2 84.38 perf-profile.children.cycles-pp.cpuidle_enter_state
73.32 ± 2% +11.2 84.49 perf-profile.children.cycles-pp.cpuidle_enter
80.25 ± 2% +12.2 92.45 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
80.25 ± 2% +12.2 92.45 perf-profile.children.cycles-pp.cpu_startup_entry
80.23 ± 2% +12.2 92.45 perf-profile.children.cycles-pp.do_idle
0.97 ± 2% -0.5 0.44 ± 19% perf-profile.self.cycles-pp._raw_spin_lock
0.36 ± 16% -0.3 0.05 ± 58% perf-profile.self.cycles-pp.__schedule
0.35 ± 8% -0.3 0.07 ± 13% perf-profile.self.cycles-pp.kmem_cache_alloc
0.12 ± 22% -0.1 0.03 ±100% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.15 ± 25% -0.1 0.07 ± 7% perf-profile.self.cycles-pp.update_load_avg
0.11 ± 20% -0.1 0.04 ± 58% perf-profile.self.cycles-pp.__update_load_avg_se
0.18 ± 14% -0.1 0.12 ± 32% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.08 ± 17% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.cpuidle_enter
0.07 ± 15% +0.0 0.10 ± 21% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
0.07 ± 15% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.rcu_nmi_exit
0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.arch_cpu_idle_enter
0.04 ± 58% +0.0 0.08 ± 33% perf-profile.self.cycles-pp.perf_pmu_disable
0.10 ± 18% +0.0 0.14 ± 18% perf-profile.self.cycles-pp.irq_enter_rcu
0.12 ± 6% +0.0 0.17 ± 10% perf-profile.self.cycles-pp.menu_reflect
0.08 ± 14% +0.0 0.12 ± 6% perf-profile.self.cycles-pp.tick_program_event
0.03 ±100% +0.0 0.07 ± 27% perf-profile.self.cycles-pp.rcu_is_cpu_rrupt_from_idle
0.15 ± 18% +0.0 0.20 ± 7% perf-profile.self.cycles-pp.asm_call_sysvec_on_stack
0.06 ± 14% +0.1 0.11 ± 23% perf-profile.self.cycles-pp.rcu_segcblist_ready_cbs
0.01 ±173% +0.1 0.07 ± 26% perf-profile.self.cycles-pp.cpuidle_get_cpu_driver
0.20 ± 17% +0.1 0.27 ± 10% perf-profile.self.cycles-pp.get_next_timer_interrupt
0.10 ± 23% +0.1 0.17 ± 25% perf-profile.self.cycles-pp.__hrtimer_get_next_event
0.03 ±102% +0.1 0.10 ± 30% perf-profile.self.cycles-pp.rwsem_spin_on_owner
0.01 ±173% +0.1 0.08 ± 37% perf-profile.self.cycles-pp.sched_idle_set_state
0.22 ± 17% +0.1 0.29 ± 12% perf-profile.self.cycles-pp.scheduler_tick
0.17 ± 14% +0.1 0.24 ± 14% perf-profile.self.cycles-pp.rb_next
0.00 +0.1 0.07 ± 24% perf-profile.self.cycles-pp.tick_nohz_idle_retain_tick
0.13 ± 15% +0.1 0.21 ± 7% perf-profile.self.cycles-pp.irq_exit_rcu
0.00 +0.1 0.08 ± 37% perf-profile.self.cycles-pp.cpu_latency_qos_limit
0.22 ± 18% +0.1 0.30 ± 14% perf-profile.self.cycles-pp.get_cpu_device
0.10 ± 30% +0.1 0.19 ± 21% perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
0.14 ± 10% +0.1 0.23 ± 14% perf-profile.self.cycles-pp.perf_event_task_tick
0.11 ± 26% +0.1 0.20 ± 12% perf-profile.self.cycles-pp.clockevents_program_event
0.09 ± 20% +0.1 0.18 ± 10% perf-profile.self.cycles-pp.x86_pmu_disable
0.05 ±102% +0.1 0.14 ± 16% perf-profile.self.cycles-pp.hrtimer_forward
0.10 ± 18% +0.1 0.20 ± 18% perf-profile.self.cycles-pp.cpuidle_not_available
0.05 ± 60% +0.1 0.14 ± 23% perf-profile.self.cycles-pp.run_local_timers
0.24 ± 8% +0.1 0.35 ± 19% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
0.16 ± 18% +0.1 0.28 ± 24% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
0.17 ± 26% +0.1 0.30 ± 21% perf-profile.self.cycles-pp.irqtime_account_process_tick
0.40 ± 7% +0.1 0.55 ± 17% perf-profile.self.cycles-pp.tick_nohz_next_event
0.24 ± 10% +0.1 0.39 ± 6% perf-profile.self.cycles-pp.timerqueue_add
0.39 ± 10% +0.2 0.55 ± 3% perf-profile.self.cycles-pp.arch_scale_freq_tick
0.32 ± 12% +0.2 0.49 ± 9% perf-profile.self.cycles-pp.__hrtimer_run_queues
0.28 ± 13% +0.2 0.45 ± 9% perf-profile.self.cycles-pp.update_blocked_averages
0.40 ± 14% +0.2 0.65 ± 19% perf-profile.self.cycles-pp.rcu_sched_clock_irq
0.98 ± 10% +0.3 1.23 ± 7% perf-profile.self.cycles-pp.io_serial_in
0.04 ±113% +2.2 2.21 ± 14% perf-profile.self.cycles-pp.osq_lock
51.91 +8.8 60.71 ± 2% perf-profile.self.cycles-pp.intel_idle
fio.write_bw_MBps
30 +----------------------------------------------------------------------+
|..+..+..+.+. +..+. : +. +..+..+ +..+..+..+..+..+.+..+..+..|
25 |-+ : : : : |
| : : : : |
| : : : : |
20 |-+ : : : : |
| : : : : |
15 |-+ : : : : |
| : : : : |
10 |-+ : : : : |
| : : : : |
| :: :: |
5 |-+ : : |
| : : |
0 +----------------------------------------------------------------------+
fio.write_iops
8000 +--------------------------------------------------------------------+
|..+..+.+..+..+..+.+..+ +.+..+..+..+ +..+..+.+..+..+..+.+.. .|
7000 |-+ : : : : +. |
6000 |-+ : : : : |
| : : : : |
5000 |-+ : : : : |
| : : : : |
4000 |-+ : : : : |
| : : : : |
3000 |-+ : : : : |
2000 |-+ : : : : |
| : : :: |
1000 |-+ : : |
| : : |
0 +--------------------------------------------------------------------+
fio.write_clat_mean_us
4.5e+08 +-----------------------------------------------------------------+
| O |
4e+08 |-+O O O O O O O O O O O O O O O O O O |
3.5e+08 |-+ |
| |
3e+08 |-+ |
2.5e+08 |-+ |
| |
2e+08 |-+ |
1.5e+08 |-+ |
| |
1e+08 |-+ |
5e+07 |-+ |
| |
0 +-----------------------------------------------------------------+
fio.write_clat_stddev
4.5e+07 +-----------------------------------------------------------------+
| O O O |
4e+07 |-+ O O |
3.5e+07 |-+ O O O O O O |
| O |
3e+07 |-+O O O O O O O |
2.5e+07 |-+ |
| |
2e+07 |-+ |
1.5e+07 |..+.+..+..+.+..+.+..+ +..+..+.+..+ +..+..+.+..+.+..+..+.+..|
| : : : : |
1e+07 |-+ : : : : |
5e+06 |-+ : : : : |
| :: :: |
0 +-----------------------------------------------------------------+
fio.write_clat_90__us
4.5e+08 +-----------------------------------------------------------------+
| O O O O O O O O O O O O O |
4e+08 |-+ |
3.5e+08 |-+ |
| |
3e+08 |-+ |
2.5e+08 |-+ |
| |
2e+08 |-+ |
1.5e+08 |-+ |
| |
1e+08 |-+ |
5e+07 |-+ |
| |
0 +-----------------------------------------------------------------+
fio.write_clat_95__us
5e+08 +-----------------------------------------------------------------+
4.5e+08 |-+ O O |
| O O O O O O O O O O O O O O O O O |
4e+08 |-+ |
3.5e+08 |-+ |
| |
3e+08 |-+ |
2.5e+08 |-+ |
2e+08 |-+ |
| |
1.5e+08 |-+ |
1e+08 |-+ |
| |
5e+07 |..+.+..+..+.+..+.+..+.. .+..+..+.+..+.. .+..+..+.+..+.+..+..+.+..|
0 +-----------------------------------------------------------------+
fio.write_clat_99__us
5e+08 +-----------------------------------------------------------------+
4.5e+08 |-+O O O O O O O O O O O O O O O O O |
| O O |
4e+08 |-+ |
3.5e+08 |-+ |
| |
3e+08 |-+ |
2.5e+08 |-+ |
2e+08 |-+ |
| |
1.5e+08 |-+ |
1e+08 |-+ |
| .+..|
5e+07 |..+.+..+..+.+..+.+..+.. .+..+..+.+..+.. .+..+..+.+..+.+..+..+ |
0 +-----------------------------------------------------------------+
fio.latency_500ms_
100 +---------------------------------------------------------------------+
90 |-+ |
| |
80 |-+ |
70 |-+ |
| |
60 |-+ |
50 |-+ |
40 |-+ |
| |
30 |-+ |
20 |-+ |
| |
10 |-+ |
0 +---------------------------------------------------------------------+
fio.workload
2.5e+06 +-----------------------------------------------------------------+
| .+.. |
|..+.+..+..+.+..+.+..+ +. +.+..+ +..+..+.+..+.+..+..+.+..|
2e+06 |-+ : : : : |
| : : : : |
| : : : : |
1.5e+06 |-+ : : : : |
| : : : : |
1e+06 |-+ : : : : |
| : : : : |
| : : : : |
500000 |-+ : : : : |
| : : |
| : : |
0 +-----------------------------------------------------------------+
fio.time.user_time
16 +----------------------------------------------------------------------+
| : : |
14 |-+ : : |
12 |-+ +.. +.. : : |
| +. .. +.. .+ .. + : +.. .+.. .+.. .+..+.. |
10 |.. .. + +. : + + : +. +. + +..|
| +..+ : : : : |
8 |-+ : : : : |
| : : : : |
6 |-+ : : : : |
4 |-+ : : : : |
| : : : : |
2 |-+O O O O :: O O O :: O |
| O O O : O O : O O O O |
0 +----------------------------------------------------------------------+
fio.time.system_time
70 +----------------------------------------------------------------------+
| .+.. +..+.. + .+..+.. |
60 |.+ +..+.+..+..+..+..+ : +.. ..: +..+..+..+..+..+ +..|
| : : + : : |
50 |-+ : : : : |
| : : : : |
40 |-+ : : : : |
| : : : : |
30 |-+ : : : : |
| O O : : O O :O:O O O O O |
20 |-+O O O O : : O : : |
| O :: O O : : |
10 |-+ : : |
| : : |
0 +----------------------------------------------------------------------+
fio.time.percent_of_cpu_this_job_got
30 +----------------------------------------------------------------------+
| |
25 |-+ .+.. .+ |
|..+..+..+.+..+..+..+..+ +. +..+. : +..+..+..+..+..+.+..+..+..|
| : : : : |
20 |-+ : : : : |
| : : : : |
15 |-+ : : : : |
| : : : : |
10 |-+ : : : : |
| O O : : O O :O:O O O O O |
| O O O O O :O:O O O : : |
5 |-+ :: : |
| : : |
0 +----------------------------------------------------------------------+
fio.time.voluntary_context_switches
2.5e+06 +-----------------------------------------------------------------+
|..+.+..+.. .+..+. .+ .+.. .+ +..+..+.+..+.+.. |
| + +. : +. +.+. : : +..+.+..|
2e+06 |-+ : : : : |
| : : : : |
| : : : : |
1.5e+06 |-+ : : : : |
| : : : : |
1e+06 |-+ : : : : |
| : : : : |
| : : : : |
500000 |-+ : : :: |
| : : |
| O O O O O O O : O O O O O : O O O O O |
0 +-----------------------------------------------------------------+
fio.time.involuntary_context_switches
3000 +--------------------------------------------------------------------+
| |
2500 |-+ O O O O |
| O O O O |
| O O O |
2000 |-+ O O O O |
| O O O O |
1500 |-+ |
| |
1000 |-+ |
| |
| +.. +.. |
500 |-.+..+.+.. +. .. + .+..|
|. +..+..+.+..+.. .. +..+..+..+. .+ + +..+..+.+. |
0 +--------------------------------------------------------------------+
fio.time.file_system_outputs
1.8e+07 +-----------------------------------------------------------------+
| + +. : : +.+. : : +..|
1.6e+07 |-+ : : : : |
1.4e+07 |-+ : : : : |
| : : : : |
1.2e+07 |-+ : : : : |
1e+07 |-+ : : : : |
| : : : : |
8e+06 |-+ : : : : |
6e+06 |-+ : : : : |
| : : : : |
4e+06 |-+ :: :: |
2e+06 |-+ : : |
| O O O O O O O : O O O O O : O O O O O |
0 +-----------------------------------------------------------------+
perf-sched.total_wait_time.average.ms
160 +---------------------------------------------------------------------+
| O O |
140 |-+ O O O O O O |
120 |-+ O O O O O |
| O O O O O |
100 |-+ O |
| |
80 |-+ |
| |
60 |-+ |
40 |-+ |
| |
20 |-+ |
|..+..+.+..+..+..+..+.+.. .+..+..+.+..+.. .+..+.+..+..+..+..+.+..+..|
0 +---------------------------------------------------------------------+
perf-sched.total_wait_and_delay.count.ms
250000 +------------------------------------------------------------------+
| |
| .+.. .+..+.. .+ +.+.. +..+ .+.. |
200000 |..+.+. + + : : .. : .+. .+..+ +.. |
| : : + : +. +. + :|
| : : : : : :|
150000 |-+ : : : : : : |
| : : : : + |
100000 |-+ : : : : |
| : : : : |
| : : : : |
50000 |-+ : : :: |
| :: :: |
| O O O O O O O : O O O O O : O O O O O |
0 +------------------------------------------------------------------+
perf-sched.total_wait_and_delay.average.ms
160 +---------------------------------------------------------------------+
| O O |
140 |-+ O O O O O O |
120 |-+ O O O O O |
| O O O O O |
100 |-+ O |
| |
80 |-+ |
| |
60 |-+ |
40 |-+ |
| |
20 |-+ |
|..+..+.+..+..+..+..+.+.. .+..+..+.+..+.. .+..+.+..+..+..+..+.+..+..|
0 +---------------------------------------------------------------------+
perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
300 +---------------------------------------------------------------------+
| O O O O O O |
250 |-+O O O O O O O O O O O O O |
| |
| |
200 |-+ |
| |
150 |-+ |
| |
100 |-+ |
| |
| |
50 |-+ |
| |
0 +---------------------------------------------------------------------+
perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
140000 +------------------------------------------------------------------+
| +.+.. |
120000 |-.+.+..+..+.+..+.. .+ : .+..+ .+. .+..+.+.. |
|. + : : +. : +. +. +.. :|
100000 |-+ : : : : + :|
| : : : : : : |
80000 |-+ : : : : : : |
| : : : : + |
60000 |-+ : : : : |
| : : : : |
40000 |-+ : : : : |
| : : :: |
20000 |-+ : :: |
| : : |
0 +------------------------------------------------------------------+
perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
300 +---------------------------------------------------------------------+
| O O O O O O |
250 |-+O O O O O O O O O O O O O |
| |
| |
200 |-+ |
| |
150 |-+ |
| |
100 |-+ |
| |
| |
50 |-+ |
| |
0 +---------------------------------------------------------------------+
fio.latency_2ms_
0.01 +-------------------------------------------------------------------+
| : : : : |
| : : : : |
0.008 |-+ : : : : |
| : : : : |
| : : : : |
0.006 |-+ : : : : |
| : : : : |
0.004 |-+ : : : : |
| : : : : |
| : : : : |
0.002 |-+ :: : : |
| : : |
| : : |
0 +-------------------------------------------------------------------+
fio.latency_4ms_
60 +----------------------------------------------------------------------+
|..+..+..+.+..+..+..+..+ +..+..+..+..+ +..+..+..+..+..+.+..+..+..|
50 |-+ : : : : |
| : : : : |
| : : : : |
40 |-+ : : : : |
| : : : : |
30 |-+ : : : : |
| : : : : |
20 |-+ : : : : |
| : : : : |
| :: :: |
10 |-+ : : |
| : : |
0 +----------------------------------------------------------------------+
fio.latency_10ms_
35 +----------------------------------------------------------------------+
| .+..+..+.+..+..+..+..+ +..+.. .+ .+.+..+.. .|
30 |.+ : : +..+. : +..+..+..+..+. +. |
| : : : : |
25 |-+ : : : : |
| : : : : |
20 |-+ : : : : |
| : : : : |
15 |-+ : : : : |
| : : : : |
10 |-+ : : : : |
| :: : : |
5 |-+ : : |
| : : |
0 +----------------------------------------------------------------------+
fio.latency_20ms_
9 +-----------------------------------------------------------------------+
|. .+.. .+.. .+..+ +.. .+.+..+ +. +. +.+..+..+.. .|
8 |-+ +. + +. : : +. : : +. |
7 |-+ : : : : |
| : : : : |
6 |-+ : : : : |
5 |-+ : : : : |
| : : : : |
4 |-+ : : : : |
3 |-+ : : : : |
| : : : : |
2 |-+ : : : : |
1 |-+ : : |
| : : |
0 +-----------------------------------------------------------------------+
fio.latency_50ms_
6 +-----------------------------------------------------------------------+
| +.. .+.. |
5 |..+..+..+..+.+..+..+.. +..+.. .+..+ : +. +..+.+..+.. .+..|
| + : + : : +. |
| : : : : |
4 |-+ : : : : |
| : : : : |
3 |-+ : : : : |
| : : : : |
2 |-+ : : : : |
| : : : : |
| : : : : |
1 |-+ : : : |
| : : |
0 +-----------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Oliver Sang
View attachment "config-5.10.0-rc7-00228-ge7a8dd2d9537" of type "text/plain" (171047 bytes)
View attachment "job-script" of type "text/plain" (8428 bytes)
View attachment "job.yaml" of type "text/plain" (5808 bytes)
View attachment "reproduce" of type "text/plain" (701 bytes)
Powered by blists - more mailing lists