| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 5 Feb 2020 20:28:05 +0800
From: kernel test robot <rong.a.chen@...el.com>
To: Ming Lei <ming.lei@...hat.com>
Cc: 0day robot <lkp@...el.com>,
Sathya Prakash <sathya.prakash@...adcom.com>,
Chaitra P B <chaitra.basappa@...adcom.com>,
Suganath Prabu Subramani
<suganath-prabu.subramani@...adcom.com>,
Kashyap Desai <kashyap.desai@...adcom.com>,
Sumit Saxena <sumit.saxena@...adcom.com>,
Shivasharan S <shivasharan.srikanteshwara@...adcom.com>,
"Ewan D. Milne" <emilne@...hat.com>,
Hannes Reinecke <hare@...e.de>,
Bart Van Assche <bart.vanassche@....com>,
LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [scsi] d4ea3041f3: fsmark.files_per_sec -11.8% regression
Greeting,
FYI, we noticed a -11.8% regression of fsmark.files_per_sec due to commit:
commit: d4ea3041f3fd005c1be676f66031c24d06c9dc9b ("scsi: sd: register request queue after sd_revalidate_disk is done")
https://github.com/0day-ci/linux/commits/Ming-Lei/scsi-support-bypass-device-busy-check-for-some-high-end-HBA-with-SSD/20200119-151701
in testcase: fsmark
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:
iterations: 1x
nr_threads: 32t
disk: 1SSD
fs: xfs
filesize: 9B
test_size: 400M
sync_method: fsyncBeforeClose
nr_directories: 16d
nr_files_per_directory: 256fpd
cpufreq_governor: performance
ucode: 0x500002c
test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/
In addition to that, the commit also has significant impact on the following tests:
+------------------+----------------------------------------------------------------------+
| testcase: change | filebench: filebench.sum_bytes_mb/s -4.1% regression |
| test machine | 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory |
| test parameters | cpufreq_governor=performance |
| | disk=1HDD |
| | fs=xfs |
| | test=singlestreamreaddirect.f |
| | ucode=0x27 |
+------------------+----------------------------------------------------------------------+
| testcase: change | fsmark: fsmark.files_per_sec -11.2% regression |
| test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory |
| test parameters | cpufreq_governor=performance |
| | disk=1SSD |
| | filesize=8K |
| | fs=xfs |
| | iterations=1x |
| | nr_directories=16d |
| | nr_files_per_directory=256fpd |
| | nr_threads=32t |
| | sync_method=fsyncBeforeClose |
| | test_size=400M |
| | ucode=0x500002c |
+------------------+----------------------------------------------------------------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
gcc-7/performance/1SSD/9B/xfs/1x/x86_64-rhel-7.6/16d/256fpd/32t/debian-x86_64-20191114.cgz/fsyncBeforeClose/lkp-csl-2sp7/400M/fsmark/0x500002c
commit:
8186413d49 ("scsi: remove .for_blk_mq")
d4ea3041f3 ("scsi: sd: register request queue after sd_revalidate_disk is done")
8186413d49989aa6 d4ea3041f3fd005c1be676f6603
---------------- ---------------------------
%stddev %change %stddev
\ | \
19966 -11.8% 17603 fsmark.files_per_sec
43027 +13.3% 48749 ± 2% fsmark.time.involuntary_context_switches
237.67 ± 2% -9.5% 215.00 fsmark.time.percent_of_cpu_this_job_got
575974 +0.9% 581111 fsmark.time.voluntary_context_switches
50117 ± 59% +61.6% 81010 ± 34% numa-numastat.node1.numa_hit
1.28 ± 2% +3.2% 1.32 ± 2% perf-stat.overall.cpi
250801 ± 5% +70.1% 426653 ± 50% turbostat.C1
268.67 ±104% +2589.0% 7224 ±120% numa-vmstat.node1.nr_inactive_file
268.67 ±104% +2589.0% 7224 ±120% numa-vmstat.node1.nr_zone_inactive_file
251408 ± 5% +70.1% 427525 ± 50% cpuidle.C1.usage
226827 ± 4% +103.3% 461250 ± 65% cpuidle.POLL.time
42260 ± 6% +102.4% 85537 ± 59% cpuidle.POLL.usage
13.39 ± 71% -87.7% 1.65 ±125% iostat.sda.await.max
127.48 ± 71% -89.6% 13.30 ±136% iostat.sda.r_await.max
2.51 ±141% -70.5% 0.74 ±100% iostat.sdb.await.max
23.30 ±141% -80.7% 4.48 ±100% iostat.sdb.r_await.max
6252 ± 60% -84.3% 982.25 ±125% proc-vmstat.numa_hint_faults
4011 ± 67% +183.0% 11354 ± 22% proc-vmstat.numa_pages_migrated
4011 ± 67% +183.0% 11354 ± 22% proc-vmstat.pgmigrate_success
1102146 -1.7% 1083816 proc-vmstat.pgpgout
123676 ± 6% -13.5% 106962 ± 6% numa-meminfo.node0.Inactive
12985 ± 24% +209.6% 40200 ± 72% numa-meminfo.node1.Inactive
1071 ±104% +2576.6% 28684 ±122% numa-meminfo.node1.Inactive(file)
1219741 ± 8% +9.5% 1335221 ± 3% numa-meminfo.node1.MemUsed
121749 ± 3% +8.8% 132487 ± 6% numa-meminfo.node1.Slab
9101 ± 3% -11.5% 8058 ± 5% slabinfo.kmalloc-1k.active_objs
9173 ± 3% -11.6% 8107 ± 5% slabinfo.kmalloc-1k.num_objs
2287 ± 17% +16.1% 2655 ± 3% slabinfo.pool_workqueue.active_objs
2287 ± 17% +16.2% 2656 ± 3% slabinfo.pool_workqueue.num_objs
640.00 -20.0% 512.00 slabinfo.scsi_sense_cache.active_objs
640.00 -20.0% 512.00 slabinfo.scsi_sense_cache.num_objs
2187 ± 24% +24.0% 2713 ± 5% slabinfo.xfs_btree_cur.active_objs
2187 ± 24% +24.0% 2713 ± 5% slabinfo.xfs_btree_cur.num_objs
3225 ± 15% +16.8% 3766 ± 4% slabinfo.xfs_buf.active_objs
3271 ± 16% +16.8% 3822 ± 4% slabinfo.xfs_buf.num_objs
0.12 ± 20% +29.0% 0.15 ± 10% sched_debug.cfs_rq:/.nr_running.avg
0.33 ± 11% +13.2% 0.37 ± 3% sched_debug.cfs_rq:/.nr_running.stddev
-1551 +293.8% -6110 sched_debug.cfs_rq:/.spread0.min
257.42 +18.6% 305.18 ± 10% sched_debug.cfs_rq:/.util_avg.avg
1814 ± 14% -27.6% 1314 ± 17% sched_debug.cfs_rq:/.util_avg.max
13631 ±123% +419.4% 70797 ± 75% sched_debug.cpu.avg_idle.min
-20.67 +25.8% -26.00 sched_debug.cpu.nr_uninterruptible.min
6.41 ± 92% -93.8% 0.40 ±144% sched_debug.cpu.sched_count.avg
136.67 ± 87% -92.3% 10.50 ±151% sched_debug.cpu.sched_count.max
26.08 ± 89% -92.7% 1.91 ±148% sched_debug.cpu.sched_count.stddev
72.00 ± 85% -92.7% 5.25 ±151% sched_debug.cpu.sched_goidle.max
13.64 ± 87% -92.9% 0.96 ±146% sched_debug.cpu.sched_goidle.stddev
12.85 ± 56% -12.8 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
12.85 ± 56% -12.8 0.00 perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
12.85 ± 56% -12.8 0.00 perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.90 ± 79% -7.9 0.00 perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
7.90 ± 79% -7.9 0.00 perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
7.72 ± 6% -5.8 1.92 ±173% perf-profile.calltrace.cycles-pp.page_fault
7.72 ± 6% -5.8 1.92 ±173% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
7.72 ± 6% -5.8 1.92 ±173% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
7.72 ± 6% -5.8 1.92 ±173% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
4.94 ± 70% -4.9 0.00 perf-profile.calltrace.cycles-pp.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault.__do_page_fault
12.85 ± 56% -12.8 0.00 perf-profile.children.cycles-pp.__x64_sys_exit_group
7.90 ± 79% -6.0 1.92 ±173% perf-profile.children.cycles-pp.mmput
7.90 ± 79% -6.0 1.92 ±173% perf-profile.children.cycles-pp.exit_mmap
4.94 ± 70% -4.9 0.00 perf-profile.children.cycles-pp.do_wp_page
68230 +19.1% 81258 softirqs.BLOCK
11147 ± 3% -15.5% 9418 ± 4% softirqs.CPU12.TIMER
9184 ± 5% +20.9% 11108 ± 11% softirqs.CPU25.TIMER
9146 ± 5% +22.3% 11186 ± 10% softirqs.CPU26.TIMER
9110 ± 3% +23.3% 11233 ± 12% softirqs.CPU28.TIMER
9832 ± 5% +18.8% 11678 ± 6% softirqs.CPU3.TIMER
9172 ± 4% +15.3% 10572 ± 8% softirqs.CPU30.TIMER
8881 ± 6% +23.5% 10967 ± 12% softirqs.CPU35.TIMER
9032 ± 5% +18.8% 10726 ± 6% softirqs.CPU36.TIMER
8979 ± 8% +15.1% 10339 ± 10% softirqs.CPU37.TIMER
9099 ± 8% +19.2% 10843 ± 12% softirqs.CPU38.TIMER
9012 ± 7% +14.6% 10325 ± 9% softirqs.CPU40.TIMER
9027 ± 5% +23.0% 11106 ± 15% softirqs.CPU43.TIMER
8985 ± 8% +27.3% 11433 ± 10% softirqs.CPU44.TIMER
9463 ± 5% +26.4% 11965 ± 19% softirqs.CPU47.TIMER
10177 ± 7% -13.0% 8857 ± 2% softirqs.CPU67.TIMER
9335 ± 6% +16.8% 10904 ± 6% softirqs.CPU78.TIMER
8584 ± 6% +16.0% 9961 ± 9% softirqs.CPU84.TIMER
8580 ± 7% +23.4% 10587 ± 14% softirqs.CPU89.TIMER
8550 ± 8% +18.2% 10104 ± 10% softirqs.CPU90.TIMER
8451 ± 5% +24.6% 10531 ± 11% softirqs.CPU93.TIMER
8436 ± 4% +32.6% 11186 ± 18% softirqs.CPU94.TIMER
8780 ± 8% +31.5% 11546 ± 7% softirqs.CPU95.TIMER
131591 +19.7% 157488 interrupts.316:PCI-MSI.376832-edge.ahci[0000:00:17.0]
15638 ± 11% +224.4% 50724 ± 56% interrupts.CAL:Function_call_interrupts
94.33 ± 14% -43.0% 53.75 ± 43% interrupts.CPU20.RES:Rescheduling_interrupts
262.67 ± 68% -71.2% 75.75 ± 20% interrupts.CPU23.RES:Rescheduling_interrupts
171.33 ± 15% +386.6% 833.75 ± 71% interrupts.CPU24.CAL:Function_call_interrupts
167.67 ± 14% +436.9% 900.25 ± 68% interrupts.CPU25.CAL:Function_call_interrupts
170.33 ± 15% +318.2% 712.25 ± 58% interrupts.CPU26.CAL:Function_call_interrupts
167.67 ± 12% +421.3% 874.00 ± 69% interrupts.CPU27.CAL:Function_call_interrupts
179.00 ± 20% +388.7% 874.75 ± 71% interrupts.CPU29.CAL:Function_call_interrupts
3.67 ±105% +5006.8% 187.25 ±139% interrupts.CPU29.RES:Rescheduling_interrupts
152.00 ± 10% +20.6% 183.25 ± 8% interrupts.CPU3.CAL:Function_call_interrupts
130.33 ± 49% +531.5% 823.00 ± 66% interrupts.CPU30.CAL:Function_call_interrupts
4.00 ±108% +4112.5% 168.50 ±137% interrupts.CPU31.RES:Rescheduling_interrupts
166.67 ± 11% +417.8% 863.00 ± 66% interrupts.CPU32.CAL:Function_call_interrupts
162.00 ± 22% +414.4% 833.25 ± 68% interrupts.CPU33.CAL:Function_call_interrupts
120.33 ± 38% +617.6% 863.50 ± 69% interrupts.CPU34.CAL:Function_call_interrupts
165.67 ± 11% +408.7% 842.75 ± 69% interrupts.CPU35.CAL:Function_call_interrupts
163.33 ± 9% +420.7% 850.50 ± 68% interrupts.CPU36.CAL:Function_call_interrupts
169.33 ± 12% +451.3% 933.50 ± 75% interrupts.CPU37.CAL:Function_call_interrupts
169.00 ± 13% +401.8% 848.00 ± 70% interrupts.CPU38.CAL:Function_call_interrupts
164.67 ± 9% +433.5% 878.50 ± 76% interrupts.CPU39.CAL:Function_call_interrupts
147.67 ± 12% +18.7% 175.25 ± 11% interrupts.CPU4.CAL:Function_call_interrupts
174.67 ± 16% +393.4% 861.75 ± 67% interrupts.CPU40.CAL:Function_call_interrupts
2.67 ±141% +26928.1% 720.75 ±164% interrupts.CPU40.RES:Rescheduling_interrupts
166.00 ± 9% +427.6% 875.75 ± 69% interrupts.CPU41.CAL:Function_call_interrupts
166.67 ± 10% +456.1% 926.75 ± 70% interrupts.CPU42.CAL:Function_call_interrupts
175.00 ± 15% +511.4% 1070 ± 70% interrupts.CPU43.CAL:Function_call_interrupts
173.33 ± 13% +437.1% 931.00 ± 71% interrupts.CPU44.CAL:Function_call_interrupts
178.00 ± 14% +479.1% 1030 ± 77% interrupts.CPU45.CAL:Function_call_interrupts
174.00 ± 13% +455.2% 966.00 ± 77% interrupts.CPU46.CAL:Function_call_interrupts
162.67 ± 8% +335.9% 709.00 ± 64% interrupts.CPU47.CAL:Function_call_interrupts
154.33 ± 12% +13.4% 175.00 ± 11% interrupts.CPU57.CAL:Function_call_interrupts
101.33 ± 14% -34.6% 66.25 ± 36% interrupts.CPU57.RES:Rescheduling_interrupts
101.00 ± 25% -51.0% 49.50 ± 39% interrupts.CPU6.RES:Rescheduling_interrupts
109.00 ± 5% -28.2% 78.25 ± 24% interrupts.CPU60.RES:Rescheduling_interrupts
179.33 ± 18% +428.6% 948.00 ± 69% interrupts.CPU72.CAL:Function_call_interrupts
176.33 ± 18% +382.9% 851.50 ± 66% interrupts.CPU73.CAL:Function_call_interrupts
173.67 ± 15% +317.9% 725.75 ± 58% interrupts.CPU74.CAL:Function_call_interrupts
170.33 ± 13% +422.7% 890.25 ± 72% interrupts.CPU75.CAL:Function_call_interrupts
171.67 ± 14% +493.9% 1019 ± 70% interrupts.CPU76.CAL:Function_call_interrupts
179.00 ± 21% +471.4% 1022 ± 70% interrupts.CPU77.CAL:Function_call_interrupts
170.67 ± 16% +442.1% 925.25 ± 72% interrupts.CPU78.CAL:Function_call_interrupts
173.33 ± 17% +429.2% 917.25 ± 76% interrupts.CPU79.CAL:Function_call_interrupts
171.00 ± 72% -73.5% 45.25 ± 34% interrupts.CPU8.RES:Rescheduling_interrupts
135.00 ± 57% +533.0% 854.50 ± 67% interrupts.CPU80.CAL:Function_call_interrupts
161.67 ± 22% +421.3% 842.75 ± 67% interrupts.CPU81.CAL:Function_call_interrupts
171.00 ± 17% +426.9% 901.00 ± 71% interrupts.CPU82.CAL:Function_call_interrupts
172.33 ± 17% +376.3% 820.75 ± 75% interrupts.CPU83.CAL:Function_call_interrupts
176.00 ± 18% +448.6% 965.50 ± 70% interrupts.CPU84.CAL:Function_call_interrupts
175.67 ± 18% +428.3% 928.00 ± 72% interrupts.CPU85.CAL:Function_call_interrupts
176.67 ± 17% +382.0% 851.50 ± 68% interrupts.CPU86.CAL:Function_call_interrupts
174.00 ± 16% +459.2% 973.00 ± 76% interrupts.CPU87.CAL:Function_call_interrupts
178.00 ± 16% +517.3% 1098 ± 89% interrupts.CPU88.CAL:Function_call_interrupts
181.33 ± 21% +422.7% 947.75 ± 77% interrupts.CPU89.CAL:Function_call_interrupts
173.67 ± 16% +421.7% 906.00 ± 73% interrupts.CPU90.CAL:Function_call_interrupts
173.67 ± 20% +394.2% 858.25 ± 70% interrupts.CPU91.CAL:Function_call_interrupts
173.67 ± 20% +425.7% 913.00 ± 69% interrupts.CPU92.CAL:Function_call_interrupts
170.67 ± 18% +402.7% 858.00 ± 67% interrupts.CPU93.CAL:Function_call_interrupts
173.67 ± 21% +404.8% 876.75 ± 71% interrupts.CPU94.CAL:Function_call_interrupts
158.00 ± 8% +316.0% 657.25 ± 57% interrupts.CPU95.CAL:Function_call_interrupts
fsmark.time.percent_of_cpu_this_job_got
250 +-+-------------------------------------------------------------------+
| + + +.+ : +.+ +.+.+.+ +.+.+.+.+.+ +.++ : + +.+ |
O O O O O O O O O O O O O O O O O O O O O : : : : |
200 +-+ : : : : : : |
| : : : : : : |
| : : : : : : |
150 +-+ : : : : : : |
| : : : : : : |
100 +-+ :: : : : : |
| :: : : : : |
| :: : : : : |
50 +-+ : : : |
| : : : |
| : : : |
0 +-+-----------O---O------O-------O------------------------------------+
fsmark.files_per_sec
25000 +-+-----------------------------------------------------------------+
| |
| |
20000 +-+.+.+.++.+.+.+ +.+.+.++.+.+.+.+.+.+.+.++.+.+.+.+.+ +.++ +.+.|
O O O O OO O O O O O O O O O O O O OO O : : : : |
| : : : : : : |
15000 +-+ : : : : : : |
| : : : : : : |
10000 +-+ : : : : : : |
| : : : : : : |
| : : : : : : |
5000 +-+ : : : : : : |
| : : : |
| : : : |
0 +-+----------O---O-------O------O-----------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
***************************************************************************************************
lkp-hsw-d01: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/1HDD/xfs/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/lkp-hsw-d01/singlestreamreaddirect.f/filebench/0x27
commit:
8186413d49 ("scsi: remove .for_blk_mq")
d4ea3041f3 ("scsi: sd: register request queue after sd_revalidate_disk is done")
8186413d49989aa6 d4ea3041f3fd005c1be676f6603
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%stddev %change %stddev
\ | \
75.95 -4.1% 72.83 filebench.sum_bytes_mb/s
4557 -4.1% 4371 filebench.sum_operations
75.94 -4.1% 72.85 filebench.sum_operations/s
76.00 -3.9% 73.00 filebench.sum_reads/s
13.17 +4.2% 13.72 filebench.sum_time_ms/op
9332769 -4.1% 8952523 filebench.time.file_system_inputs
162436 ± 6% -11.8% 143321 meminfo.DirectMap4k
44022 -4.4% 42098 vmstat.io.bi
3125 ± 5% -6.3% 2928 ± 5% slabinfo.filp.active_objs
7419 ± 3% -8.0% 6825 ± 3% slabinfo.kmalloc-8.active_objs
7419 ± 3% -8.0% 6825 ± 3% slabinfo.kmalloc-8.num_objs
18.34 +2.1% 18.73 iostat.cpu.iowait
1.46 -37.9% 0.90 iostat.sda.avgqu-sz
44025 -4.4% 42098 iostat.sda.rkB/s
1.05 -3.2% 1.01 iostat.sda.svctm
59194 ± 7% -14.2% 50811 ± 8% softirqs.CPU1.TIMER
32902 ± 5% -7.0% 30592 ± 6% softirqs.CPU2.RCU
12304 ± 11% +32.4% 16287 ± 2% softirqs.CPU2.SCHED
54266 ± 5% -8.7% 49545 ± 5% softirqs.CPU2.TIMER
30586 ± 5% -6.0% 28753 ± 2% softirqs.CPU6.RCU
1.057e+08 ± 5% -27.1% 77008220 ± 5% cpuidle.C1E.time
1127971 -11.1% 1002432 ± 2% cpuidle.C1E.usage
2.882e+08 ± 5% +122.2% 6.403e+08 ± 15% cpuidle.C3.time
794565 +73.5% 1378623 ± 9% cpuidle.C3.usage
6567827 ± 68% +1171.0% 83478482 ±114% cpuidle.C6.time
10816 ± 64% +938.8% 112360 ±107% cpuidle.C6.usage
4.024e+08 ± 7% -98.2% 7380622 ± 74% cpuidle.C7s.time
577131 -98.2% 10454 ± 66% cpuidle.C7s.usage
132590 +3.5% 137195 proc-vmstat.nr_dirty
1080688 +1.3% 1095121 proc-vmstat.nr_file_pages
18355 -5.9% 17263 ± 3% proc-vmstat.nr_free_cma
831702 -1.8% 817002 proc-vmstat.nr_free_pages
799686 +1.8% 814042 proc-vmstat.nr_inactive_file
3957 +2.4% 4050 proc-vmstat.nr_kernel_stack
799690 +1.8% 814044 proc-vmstat.nr_zone_inactive_file
139862 +3.2% 144400 proc-vmstat.nr_zone_write_pending
1397097 ± 5% -4.1% 1340270 ± 7% proc-vmstat.pgfree
4666401 -4.1% 4476278 proc-vmstat.pgpgin
104.50 -8.8% 95.33 ± 6% turbostat.Avg_MHz
3.14 ± 2% -0.3 2.86 ± 3% turbostat.Busy%
1127727 -11.1% 1002183 ± 2% turbostat.C1E
12.67 ± 5% -3.5 9.19 ± 5% turbostat.C1E%
794359 +73.5% 1378587 ± 9% turbostat.C3
34.56 ± 6% +41.9 76.42 ± 15% turbostat.C3%
10756 ± 65% +944.5% 112347 ±107% turbostat.C6
0.78 ± 69% +9.2 10.01 ±115% turbostat.C6%
577213 -98.2% 10268 ± 67% turbostat.C7s
48.25 ± 7% -47.4 0.87 ± 76% turbostat.C7s%
28.55 ± 5% -19.7% 22.92 ± 3% turbostat.CPU%c1
35.85 +96.3% 70.39 ± 6% turbostat.CPU%c3
31.56 ± 8% -99.5% 0.17 ± 92% turbostat.CPU%c7
12.76 -4.1% 12.23 turbostat.PkgWatt
434484 ± 25% -32.5% 293281 ± 9% interrupts.CPU0.LOC:Local_timer_interrupts
620.50 ± 24% -64.3% 221.33 ± 15% interrupts.CPU0.RES:Rescheduling_interrupts
423916 ± 18% -24.0% 322268 ± 23% interrupts.CPU2.LOC:Local_timer_interrupts
575.00 ± 67% -71.1% 166.33 ±106% interrupts.CPU2.RES:Rescheduling_interrupts
276350 ± 23% +83.1% 506099 ± 34% interrupts.CPU3.LOC:Local_timer_interrupts
247.50 ± 3% +55.6% 385.00 ± 7% interrupts.CPU3.NMI:Non-maskable_interrupts
247.50 ± 3% +55.6% 385.00 ± 7% interrupts.CPU3.PMI:Performance_monitoring_interrupts
430.00 -41.9% 250.00 ± 29% interrupts.CPU3.RES:Rescheduling_interrupts
288708 ± 18% -9.5% 261253 ± 17% interrupts.CPU5.LOC:Local_timer_interrupts
506.00 ± 52% -57.6% 214.67 ± 60% interrupts.CPU5.RES:Rescheduling_interrupts
212556 +26.4% 268764 ± 16% interrupts.CPU6.LOC:Local_timer_interrupts
121.00 ± 24% +223.1% 391.00 ± 45% interrupts.CPU6.RES:Rescheduling_interrupts
1051 ± 7% -28.4% 752.67 ± 24% interrupts.CPU7.CAL:Function_call_interrupts
166.50 ± 8% +58.6% 264.00 ± 2% interrupts.CPU7.NMI:Non-maskable_interrupts
166.50 ± 8% +58.6% 264.00 ± 2% interrupts.CPU7.PMI:Performance_monitoring_interrupts
1621 +20.1% 1947 ± 10% interrupts.NMI:Non-maskable_interrupts
1621 +20.1% 1947 ± 10% interrupts.PMI:Performance_monitoring_interrupts
3323 -39.7% 2004 ± 26% interrupts.RES:Rescheduling_interrupts
98.93 ± 99% +139.4% 236.87 ± 11% sched_debug.cfs_rq:/.exec_clock.min
1157 ± 25% -32.8% 777.88 ± 15% sched_debug.cfs_rq:/.load_avg.avg
3709 ± 21% -39.3% 2252 ± 12% sched_debug.cfs_rq:/.load_avg.max
1164 ± 11% -29.0% 826.83 ± 18% sched_debug.cfs_rq:/.load_avg.stddev
3315 ± 3% +19.4% 3959 ± 4% sched_debug.cfs_rq:/.min_vruntime.min
609.00 ± 32% -41.8% 354.50 ± 10% sched_debug.cfs_rq:/.util_avg.min
149.59 ± 9% +54.4% 230.96 ± 9% sched_debug.cfs_rq:/.util_avg.stddev
70559 ± 28% -33.1% 47195 ± 49% sched_debug.cpu.avg_idle.min
311669 ± 3% -7.6% 288062 ± 6% sched_debug.cpu.avg_idle.stddev
778.66 ± 17% -20.9% 616.04 ± 8% sched_debug.cpu.curr->pid.avg
1.59 ± 17% -47.7% 0.83 ± 21% sched_debug.cpu.nr_running.avg
3.50 ± 14% -42.9% 2.00 ± 20% sched_debug.cpu.nr_running.max
1.06 ± 9% -37.7% 0.66 ± 21% sched_debug.cpu.nr_running.stddev
8410 ± 78% +382.4% 40568 ± 38% sched_debug.cpu.nr_switches.min
5158 ±100% +590.5% 35618 ± 42% sched_debug.cpu.sched_count.min
2495 ±100% +612.5% 17778 ± 42% sched_debug.cpu.sched_goidle.min
3940 ±100% +349.8% 17726 ± 35% sched_debug.cpu.ttwu_count.min
2288 ±100% +593.7% 15877 ± 43% sched_debug.cpu.ttwu_local.min
28.82 ± 2% -12.1% 25.33 ± 4% perf-stat.i.MPKI
3.44 ± 2% -1.4 2.02 ± 16% perf-stat.i.branch-miss-rate%
3203155 -33.2% 2140684 ± 35% perf-stat.i.branch-misses
10.17 -0.9 9.28 ± 2% perf-stat.i.cache-miss-rate%
698479 -12.8% 609001 ± 5% perf-stat.i.cache-misses
6854696 -6.9% 6382981 ± 4% perf-stat.i.cache-references
2.81 ± 2% -10.4% 2.52 ± 2% perf-stat.i.cpi
8.278e+08 -7.9% 7.621e+08 ± 6% perf-stat.i.cpu-cycles
6.09 ± 4% +62.2% 9.88 ± 23% perf-stat.i.cpu-migrations
1215 +11.8% 1359 ± 2% perf-stat.i.cycles-between-cache-misses
159444 ± 2% -3.8% 153313 ± 2% perf-stat.i.iTLB-load-misses
194692 -2.9% 189082 ± 2% perf-stat.i.iTLB-loads
0.42 +6.9% 0.45 ± 2% perf-stat.i.ipc
196601 ± 3% -6.9% 183066 ± 6% perf-stat.i.node-loads
3.66 ± 2% -1.2 2.49 ± 26% perf-stat.overall.branch-miss-rate%
10.19 -0.7 9.54 ± 2% perf-stat.overall.cache-miss-rate%
1185 +5.5% 1251 perf-stat.overall.cycles-between-cache-misses
3177943 -33.3% 2120771 ± 35% perf-stat.ps.branch-misses
691895 -12.8% 603163 ± 5% perf-stat.ps.cache-misses
6789018 -6.9% 6321639 ± 4% perf-stat.ps.cache-references
8.201e+08 -8.0% 7.549e+08 ± 6% perf-stat.ps.cpu-cycles
6.03 ± 4% +62.2% 9.78 ± 23% perf-stat.ps.cpu-migrations
157910 ± 2% -3.8% 151843 ± 2% perf-stat.ps.iTLB-load-misses
192827 -2.9% 187263 ± 2% perf-stat.ps.iTLB-loads
194815 ± 3% -6.9% 181321 ± 6% perf-stat.ps.node-loads
23.48 ± 6% -5.8 17.70 ± 3% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
13.07 ± 6% -4.4 8.68 ± 35% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
13.07 ± 6% -4.4 8.68 ± 35% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
13.07 ± 6% -4.4 8.68 ± 35% perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
11.43 -4.4 7.04 ± 7% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
11.85 ± 5% -3.9 7.94 ± 40% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
8.30 ± 11% -3.8 4.49 ± 12% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
19.09 ± 5% -3.8 15.32 ± 5% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
11.80 ± 15% -2.8 9.05 ± 5% perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt.cpuidle_enter_state
11.80 ± 15% -2.8 9.05 ± 5% perf-profile.calltrace.cycles-pp.irq_work_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
11.80 ± 15% -2.8 9.05 ± 5% perf-profile.calltrace.cycles-pp.smp_irq_work_interrupt.irq_work_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
11.80 ± 15% -2.8 9.05 ± 5% perf-profile.calltrace.cycles-pp.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt.cpuidle_enter_state.cpuidle_enter
12.91 ± 5% -2.5 10.41 ± 11% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.irq_work_run_list.irq_work_run
12.91 ± 5% -2.5 10.41 ± 11% perf-profile.calltrace.cycles-pp.vprintk_emit.printk.irq_work_run_list.irq_work_run.smp_irq_work_interrupt
12.91 ± 5% -2.5 10.41 ± 11% perf-profile.calltrace.cycles-pp.printk.irq_work_run_list.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt
12.41 ± 5% -2.5 9.92 ± 11% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.printk.irq_work_run_list
12.41 ± 5% -2.5 9.92 ± 11% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk
13.82 ± 7% -2.4 11.38 ± 19% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
12.93 -2.0 10.98 ± 16% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
9.78 ± 7% -1.6 8.14 ± 19% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
3.68 ± 28% -1.2 2.46 ± 10% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
2.80 ± 20% -0.7 2.06 ± 9% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
3.58 ± 13% -0.7 2.85 ± 17% perf-profile.calltrace.cycles-pp.delay_tsc.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
2.62 ± 21% -0.7 1.90 ± 10% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
1.29 ± 5% -0.4 0.92 ± 18% perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
1.03 ± 13% -0.3 0.76 ± 19% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
2.63 -0.2 2.43 perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
0.66 ± 6% +0.4 1.04 ± 11% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
0.28 ±100% +0.5 0.79 ± 19% perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
0.76 ± 9% +0.6 1.31 ± 13% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
0.95 +0.9 1.85 ± 19% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlinkat
0.95 +0.9 1.85 ± 19% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
0.95 +0.9 1.85 ± 19% perf-profile.calltrace.cycles-pp.unlinkat
0.95 +0.9 1.85 ± 19% perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
0.95 +0.9 1.85 ± 19% perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
0.91 ± 2% +0.9 1.85 ± 19% perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
49.55 +8.9 58.46 ± 3% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
21.98 ± 5% -5.3 16.66 ± 4% perf-profile.children.cycles-pp.apic_timer_interrupt
13.07 ± 6% -4.4 8.68 ± 35% perf-profile.children.cycles-pp.start_kernel
11.46 -4.2 7.28 ± 8% perf-profile.children.cycles-pp.hrtimer_interrupt
19.12 ± 4% -3.8 15.34 ± 5% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
8.37 ± 11% -3.6 4.80 ± 10% perf-profile.children.cycles-pp.__hrtimer_run_queues
13.87 ± 12% -3.0 10.89 ± 16% perf-profile.children.cycles-pp.irq_work_run_list
13.95 ± 7% -2.6 11.39 ± 19% perf-profile.children.cycles-pp.serial8250_console_putchar
14.04 ± 7% -2.5 11.50 ± 20% perf-profile.children.cycles-pp.wait_for_xmitr
14.11 ± 8% -2.5 11.60 ± 19% perf-profile.children.cycles-pp.uart_console_write
12.91 ± 5% -2.5 10.41 ± 11% perf-profile.children.cycles-pp.irq_work_interrupt
12.91 ± 5% -2.5 10.41 ± 11% perf-profile.children.cycles-pp.smp_irq_work_interrupt
12.91 ± 5% -2.5 10.41 ± 11% perf-profile.children.cycles-pp.irq_work_run
12.91 ± 5% -2.5 10.41 ± 11% perf-profile.children.cycles-pp.printk
14.20 ± 8% -2.5 11.71 ± 20% perf-profile.children.cycles-pp.serial8250_console_write
14.79 ± 7% -2.4 12.42 ± 20% perf-profile.children.cycles-pp.console_unlock
13.87 -1.9 11.95 ± 17% perf-profile.children.cycles-pp.vprintk_emit
9.93 ± 7% -1.7 8.18 ± 19% perf-profile.children.cycles-pp.io_serial_in
3.97 ± 22% -1.4 2.60 ± 10% perf-profile.children.cycles-pp.tick_sched_timer
1.51 ± 5% -1.0 0.53 ± 33% perf-profile.children.cycles-pp.__remove_hrtimer
1.38 ± 10% -1.0 0.43 ± 28% perf-profile.children.cycles-pp.timerqueue_del
3.12 ± 22% -0.9 2.18 ± 9% perf-profile.children.cycles-pp.tick_sched_handle
2.87 ± 26% -0.9 2.01 ± 9% perf-profile.children.cycles-pp.update_process_times
1.16 -0.8 0.32 ± 42% perf-profile.children.cycles-pp.interrupt_entry
6.37 ± 3% -0.8 5.55 ± 7% perf-profile.children.cycles-pp.menu_select
0.66 ± 12% -0.6 0.09 ± 72% perf-profile.children.cycles-pp.rb_next
2.52 ± 4% -0.5 2.01 ± 13% perf-profile.children.cycles-pp.tick_nohz_next_event
3.10 ± 3% -0.5 2.61 ± 5% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
1.50 -0.5 1.04 ± 18% perf-profile.children.cycles-pp.irq_enter
0.79 -0.4 0.37 ± 22% perf-profile.children.cycles-pp.enqueue_hrtimer
1.23 ± 6% -0.4 0.84 ± 24% perf-profile.children.cycles-pp.tick_irq_enter
0.73 ± 6% -0.4 0.35 ± 16% perf-profile.children.cycles-pp.timerqueue_add
0.47 ± 19% -0.4 0.10 ± 37% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
0.91 ± 12% -0.4 0.55 ± 41% perf-profile.children.cycles-pp.native_write_msr
0.53 ± 28% -0.3 0.20 ± 32% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.50 ± 13% -0.3 0.22 ± 14% perf-profile.children.cycles-pp.tick_nohz_irq_exit
1.25 ± 11% -0.3 0.99 ± 9% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
1.60 ± 4% -0.3 1.34 ± 20% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.38 ± 18% -0.2 0.15 ± 46% perf-profile.children.cycles-pp.__intel_pmu_enable_all
0.81 ± 5% -0.2 0.59 ± 10% perf-profile.children.cycles-pp.sched_clock_cpu
0.34 ± 10% -0.2 0.14 ± 26% perf-profile.children.cycles-pp.irq_work_needs_cpu
0.44 ± 56% -0.2 0.24 ± 71% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.42 ± 54% -0.2 0.22 ± 27% perf-profile.children.cycles-pp.ret_from_fork
0.25 ± 52% -0.2 0.07 ± 25% perf-profile.children.cycles-pp.rcu_needs_cpu
0.44 ± 15% -0.2 0.27 ± 34% perf-profile.children.cycles-pp.tick_sched_do_timer
0.34 ± 10% -0.2 0.18 ± 48% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.56 ± 9% -0.1 0.42 ± 19% perf-profile.children.cycles-pp.native_sched_clock
0.22 ± 45% -0.1 0.08 ± 20% perf-profile.children.cycles-pp.account_process_tick
0.26 -0.1 0.12 ± 32% perf-profile.children.cycles-pp._raw_spin_lock_irq
1.20 ± 12% -0.1 1.06 ± 12% perf-profile.children.cycles-pp.ktime_get
0.56 ± 9% -0.1 0.44 ± 14% perf-profile.children.cycles-pp.sched_clock
0.25 ± 52% -0.1 0.13 ± 91% perf-profile.children.cycles-pp.idle_cpu
0.34 ± 10% -0.1 0.23 ± 29% perf-profile.children.cycles-pp.timekeeping_advance
0.22 ± 40% -0.1 0.11 ±110% perf-profile.children.cycles-pp.do_IRQ
0.44 ± 15% -0.1 0.34 ± 9% perf-profile.children.cycles-pp.try_to_wake_up
0.25 ± 52% -0.1 0.16 ± 91% perf-profile.children.cycles-pp.__x64_sys_exit_group
0.25 ± 52% -0.1 0.16 ± 91% perf-profile.children.cycles-pp.do_group_exit
0.25 ± 52% -0.1 0.16 ± 91% perf-profile.children.cycles-pp.do_exit
0.22 ± 45% -0.1 0.13 ± 55% perf-profile.children.cycles-pp.rcu_irq_exit
0.12 ± 4% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.hrtimer_forward
0.19 -0.1 0.10 ± 40% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.12 ± 52% -0.1 0.05 ± 81% perf-profile.children.cycles-pp.rcu_qs
0.12 ± 52% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.__hrtimer_get_next_event
0.12 ± 52% -0.1 0.06 ± 86% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.12 ± 4% -0.1 0.07 ± 25% perf-profile.children.cycles-pp.unwind_next_frame
0.09 ± 33% -0.1 0.03 ± 70% perf-profile.children.cycles-pp.pipe_write
0.09 ± 33% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.prepare_exit_to_usermode
0.15 ± 22% -0.1 0.10 ± 40% perf-profile.children.cycles-pp.swake_up_one
0.15 ± 22% -0.1 0.10 ± 40% perf-profile.children.cycles-pp.swake_up_locked
0.19 ± 36% -0.0 0.14 ± 47% perf-profile.children.cycles-pp.pm_qos_read_value
0.10 ± 36% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.truncate_cleanup_page
0.10 ± 36% -0.0 0.06 ± 86% perf-profile.children.cycles-pp.do_sys_open
0.06 -0.0 0.03 ± 70% perf-profile.children.cycles-pp.handle_fasteoi_irq
0.06 -0.0 0.03 ± 70% perf-profile.children.cycles-pp.handle_irq_event
0.06 -0.0 0.03 ± 70% perf-profile.children.cycles-pp.handle_irq_event_percpu
0.12 ± 4% +0.1 0.21 ± 11% perf-profile.children.cycles-pp.call_timer_fn
0.00 +0.1 0.09 ± 32% perf-profile.children.cycles-pp.__x64_sys_clone
0.00 +0.1 0.09 ± 32% perf-profile.children.cycles-pp._do_fork
0.00 +0.1 0.09 ± 32% perf-profile.children.cycles-pp.copy_process
0.00 +0.1 0.09 ± 65% perf-profile.children.cycles-pp.workingset_update_node
0.12 ± 4% +0.1 0.22 ± 14% perf-profile.children.cycles-pp.leave_mm
0.10 ± 36% +0.1 0.20 ± 21% perf-profile.children.cycles-pp.xas_init_marks
0.03 ±100% +0.1 0.14 ± 11% perf-profile.children.cycles-pp.timekeeping_max_deferment
0.10 ± 36% +0.1 0.21 ± 11% perf-profile.children.cycles-pp.pagevec_lookup_entries
0.06 +0.1 0.20 ± 39% perf-profile.children.cycles-pp.rcu_gp_kthread
0.12 ± 4% +0.2 0.32 ± 18% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.10 ± 36% +0.2 0.30 ± 42% perf-profile.children.cycles-pp.__delete_from_page_cache
0.47 ± 8% +0.3 0.76 ± 28% perf-profile.children.cycles-pp.find_busiest_group
0.23 ± 15% +0.3 0.52 ± 34% perf-profile.children.cycles-pp.xas_store
0.15 ± 22% +0.3 0.46 ± 30% perf-profile.children.cycles-pp.pm_qos_request
0.34 ± 10% +0.3 0.67 ± 22% perf-profile.children.cycles-pp.delete_from_page_cache_batch
0.84 ± 5% +0.4 1.20 ± 16% perf-profile.children.cycles-pp.load_balance
0.00 +0.4 0.43 ± 43% perf-profile.children.cycles-pp.blk_mq_try_issue_list_directly
0.00 +0.4 0.43 ± 43% perf-profile.children.cycles-pp.blk_mq_request_issue_directly
0.00 +0.4 0.43 ± 43% perf-profile.children.cycles-pp.__blk_mq_try_issue_directly
0.41 ± 9% +0.6 1.01 ± 11% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.94 ± 8% +0.6 1.56 ± 14% perf-profile.children.cycles-pp.rebalance_domains
1.07 ± 13% +0.9 1.95 ± 19% perf-profile.children.cycles-pp.evict
0.95 +0.9 1.85 ± 19% perf-profile.children.cycles-pp.truncate_inode_pages_range
0.95 +0.9 1.85 ± 19% perf-profile.children.cycles-pp.unlinkat
0.95 +0.9 1.85 ± 19% perf-profile.children.cycles-pp.do_unlinkat
49.55 +8.9 58.49 ± 3% perf-profile.children.cycles-pp.intel_idle
9.93 ± 7% -1.7 8.18 ± 19% perf-profile.self.cycles-pp.io_serial_in
2.56 ± 13% -0.8 1.73 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state
1.06 -0.7 0.32 ± 42% perf-profile.self.cycles-pp.interrupt_entry
0.75 ± 24% -0.7 0.06 ± 86% perf-profile.self.cycles-pp.apic_timer_interrupt
2.61 ± 7% -0.6 2.03 ± 3% perf-profile.self.cycles-pp.menu_select
0.66 ± 12% -0.6 0.09 ± 72% perf-profile.self.cycles-pp.rb_next
0.84 ± 24% -0.4 0.43 ± 44% perf-profile.self.cycles-pp.hrtimer_interrupt
0.67 ± 15% -0.4 0.28 ± 12% perf-profile.self.cycles-pp.timerqueue_add
0.91 ± 12% -0.4 0.54 ± 38% perf-profile.self.cycles-pp.native_write_msr
0.47 ± 19% -0.4 0.10 ± 37% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
0.57 ± 20% -0.3 0.22 ± 32% perf-profile.self.cycles-pp.__hrtimer_run_queues
0.62 ± 18% -0.2 0.38 ± 39% perf-profile.self.cycles-pp.do_idle
0.32 ± 17% -0.2 0.09 ± 32% perf-profile.self.cycles-pp.get_next_timer_interrupt
0.38 ± 49% -0.2 0.18 ± 48% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
0.32 ± 20% -0.2 0.13 ± 43% perf-profile.self.cycles-pp.irq_work_needs_cpu
0.25 ± 52% -0.2 0.07 ± 25% perf-profile.self.cycles-pp.rcu_needs_cpu
0.56 ± 9% -0.2 0.38 ± 16% perf-profile.self.cycles-pp.native_sched_clock
0.32 -0.2 0.14 ± 54% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.41 ± 53% -0.2 0.24 ± 71% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.43 ± 12% -0.2 0.28 ± 12% perf-profile.self.cycles-pp.run_timer_softirq
0.19 ± 31% -0.1 0.05 ± 78% perf-profile.self.cycles-pp.clockevents_program_event
0.22 ± 45% -0.1 0.08 ± 20% perf-profile.self.cycles-pp.account_process_tick
0.25 ± 52% -0.1 0.13 ± 91% perf-profile.self.cycles-pp.idle_cpu
0.25 ± 24% -0.1 0.13 ± 55% perf-profile.self.cycles-pp.rcu_sched_clock_irq
0.19 ± 36% -0.1 0.08 ± 20% perf-profile.self.cycles-pp.tick_sched_timer
0.23 ± 15% -0.1 0.12 ± 32% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.26 -0.1 0.15 ± 16% perf-profile.self.cycles-pp.sched_clock_cpu
0.15 ± 22% -0.1 0.06 ± 71% perf-profile.self.cycles-pp.irq_exit
0.22 ± 45% -0.1 0.13 ± 55% perf-profile.self.cycles-pp.rcu_irq_exit
0.23 ± 15% -0.1 0.14 ± 44% perf-profile.self.cycles-pp.release_pages
0.12 ± 4% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.hrtimer_forward
0.19 -0.1 0.10 ± 40% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.12 ± 52% -0.1 0.05 ± 81% perf-profile.self.cycles-pp.rcu_qs
0.09 ± 33% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.prepare_exit_to_usermode
0.19 ± 36% -0.0 0.14 ± 47% perf-profile.self.cycles-pp.pm_qos_read_value
0.10 ± 36% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.run_local_timers
0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.sched_clock
0.12 ± 4% +0.1 0.19 ± 15% perf-profile.self.cycles-pp.leave_mm
0.00 +0.1 0.09 ± 65% perf-profile.self.cycles-pp.workingset_update_node
0.00 +0.1 0.14 ± 11% perf-profile.self.cycles-pp.timekeeping_max_deferment
0.12 ± 4% +0.2 0.32 ± 18% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.34 ± 10% +0.2 0.57 ± 16% perf-profile.self.cycles-pp.find_busiest_group
0.06 +0.3 0.32 ± 75% perf-profile.self.cycles-pp.rebalance_domains
0.15 ± 22% +0.3 0.45 ± 34% perf-profile.self.cycles-pp.pm_qos_request
0.29 ± 12% +0.4 0.69 ± 22% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
49.43 +8.9 58.30 ± 3% perf-profile.self.cycles-pp.intel_idle
***************************************************************************************************
lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
gcc-7/performance/1SSD/8K/xfs/1x/x86_64-rhel-7.6/16d/256fpd/32t/debian-x86_64-20191114.cgz/fsyncBeforeClose/lkp-csl-2sp7/400M/fsmark/0x500002c
commit:
8186413d49 ("scsi: remove .for_blk_mq")
d4ea3041f3 ("scsi: sd: register request queue after sd_revalidate_disk is done")
8186413d49989aa6 d4ea3041f3fd005c1be676f6603
---------------- ---------------------------
%stddev %change %stddev
\ | \
17532 -11.2% 15568 fsmark.files_per_sec
21452 ± 4% +34.3% 28806 fsmark.time.involuntary_context_switches
220.50 -9.6% 199.33 fsmark.time.percent_of_cpu_this_job_got
287664 +1.1% 290828 fsmark.time.voluntary_context_switches
1231 +5.4% 1298 proc-vmstat.nr_page_table_pages
142907 ± 6% +21.4% 173537 ± 5% turbostat.C1
143727 ± 6% +21.3% 174368 ± 5% cpuidle.C1.usage
125251 ± 9% +25.4% 157022 ± 15% cpuidle.POLL.time
24561 ± 10% +27.3% 31275 ± 17% cpuidle.POLL.usage
176229 ± 8% -24.8% 132584 ± 25% numa-meminfo.node1.Active
175236 ± 7% -25.1% 131325 ± 25% numa-meminfo.node1.Active(anon)
174944 ± 7% -25.1% 130998 ± 25% numa-meminfo.node1.AnonPages
43836 ± 7% -25.1% 32847 ± 25% numa-vmstat.node1.nr_active_anon
43761 ± 7% -25.1% 32756 ± 25% numa-vmstat.node1.nr_anon_pages
43836 ± 7% -25.1% 32847 ± 25% numa-vmstat.node1.nr_zone_active_anon
21491 ± 14% -29.9% 15061 ± 26% sched_debug.cfs_rq:/.min_vruntime.avg
485.50 ± 9% -52.9% 228.67 ± 77% sched_debug.cfs_rq:/.removed.util_avg.max
22518 ± 88% -91.6% 1896 ± 2% sched_debug.cpu.avg_idle.min
34973 +19.9% 41935 softirqs.BLOCK
8249 ± 2% +29.3% 10668 ± 4% softirqs.CPU10.TIMER
10384 ± 21% -17.7% 8545 ± 4% softirqs.CPU28.TIMER
9712 ± 10% -13.5% 8400 ± 5% softirqs.CPU8.TIMER
96.09 -0.7% 95.40 iostat.cpu.idle
6.06 ± 16% -60.6% 2.38 ± 71% iostat.sdb.avgqu-sz
10.43 ± 63% -85.6% 1.50 ± 80% iostat.sdb.await.max
96.57 ± 67% -88.9% 10.71 ± 88% iostat.sdb.r_await.max
1436 ± 16% -93.4% 95.39 ± 71% iostat.sdb.wrqm/s
4368 ± 7% -18.5% 3560 ± 19% slabinfo.eventpoll_pwq.active_objs
4368 ± 7% -18.5% 3560 ± 19% slabinfo.eventpoll_pwq.num_objs
752.00 ± 10% -12.0% 661.67 ± 11% slabinfo.file_lock_cache.active_objs
752.00 ± 10% -12.0% 661.67 ± 11% slabinfo.file_lock_cache.num_objs
7200 ± 3% -9.9% 6487 slabinfo.kmalloc-1k.active_objs
7251 ± 3% -9.3% 6574 slabinfo.kmalloc-1k.num_objs
9913 ± 4% -4.8% 9436 ± 4% slabinfo.kmalloc-512.active_objs
9983 ± 4% -4.8% 9505 ± 5% slabinfo.kmalloc-512.num_objs
924.25 ± 8% -13.9% 796.00 ± 2% slabinfo.mnt_cache.active_objs
924.25 ± 8% -13.9% 796.00 ± 2% slabinfo.mnt_cache.num_objs
640.00 -20.0% 512.00 slabinfo.scsi_sense_cache.active_objs
640.00 -20.0% 512.00 slabinfo.scsi_sense_cache.num_objs
687.00 ± 19% -24.6% 518.33 ± 5% slabinfo.skbuff_fclone_cache.active_objs
687.00 ± 19% -24.6% 518.33 ± 5% slabinfo.skbuff_fclone_cache.num_objs
8.56 ± 5% +17.9% 10.10 ± 3% perf-stat.i.MPKI
10.90 ± 26% -5.1 5.75 ± 12% perf-stat.i.cache-miss-rate%
74744370 ± 7% +19.1% 89057238 ± 4% perf-stat.i.cache-references
193622 ± 6% +14.4% 221455 ± 5% perf-stat.i.context-switches
1731 ±109% +227.2% 5664 ± 46% perf-stat.i.cpu-migrations
0.04 ± 20% +0.0 0.05 ± 17% perf-stat.i.dTLB-load-miss-rate%
690430 ± 34% +53.8% 1061755 ± 23% perf-stat.i.dTLB-load-misses
0.02 ± 6% +0.0 0.02 ± 13% perf-stat.i.dTLB-store-miss-rate%
97896 ± 11% +29.2% 126471 ± 10% perf-stat.i.dTLB-store-misses
43.10 +0.7 43.79 perf-stat.i.iTLB-load-miss-rate%
3798429 ± 4% +7.9% 4100118 ± 3% perf-stat.i.iTLB-load-misses
8.97 ± 7% +18.2% 10.61 ± 5% perf-stat.overall.MPKI
5.65 ± 15% -1.6 4.03 ± 7% perf-stat.overall.cache-miss-rate%
1.30 ± 2% +4.1% 1.35 ± 3% perf-stat.overall.cpi
0.03 ± 32% +0.0 0.05 ± 23% perf-stat.overall.dTLB-load-miss-rate%
0.01 ± 11% +0.0 0.01 ± 16% perf-stat.overall.dTLB-store-miss-rate%
41.58 +0.6 42.14 perf-stat.overall.iTLB-load-miss-rate%
0.77 ± 2% -3.9% 0.74 ± 3% perf-stat.overall.ipc
75.22 ± 2% +5.7 80.88 ± 2% perf-stat.overall.node-load-miss-rate%
60564747 ± 7% +15.0% 69654648 ± 6% perf-stat.ps.cache-references
1431 ±110% +208.5% 4415 ± 47% perf-stat.ps.cpu-migrations
79706 ± 12% +24.1% 98906 ± 8% perf-stat.ps.dTLB-store-misses
10.62 ± 63% -10.6 0.00 perf-profile.calltrace.cycles-pp.__libc_start_main
7.27 ±113% -7.3 0.00 perf-profile.calltrace.cycles-pp.page_fault
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.ioctl.__libc_start_main
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl.__libc_start_main
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.smp_call_function_single.event_function_call.perf_event_for_each_child._perf_ioctl.perf_ioctl
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.event_function_call.perf_event_for_each_child._perf_ioctl.perf_ioctl.ksys_ioctl
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl.__libc_start_main
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp._perf_ioctl.perf_ioctl.ksys_ioctl.__x64_sys_ioctl.do_syscall_64
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.ksys_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.ioctl.__libc_start_main
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.perf_ioctl.ksys_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.07 ±103% -6.1 0.00 perf-profile.calltrace.cycles-pp.perf_event_for_each_child._perf_ioctl.perf_ioctl.ksys_ioctl.__x64_sys_ioctl
5.28 ±100% -1.6 3.70 ±141% perf-profile.calltrace.cycles-pp.setlocale
8.34 ± 62% -8.3 0.00 perf-profile.children.cycles-pp.__libc_start_main
6.07 ±103% -6.1 0.00 perf-profile.children.cycles-pp.__x64_sys_ioctl
6.07 ±103% -6.1 0.00 perf-profile.children.cycles-pp._perf_ioctl
6.07 ±103% -6.1 0.00 perf-profile.children.cycles-pp.ksys_ioctl
6.07 ±103% -6.1 0.00 perf-profile.children.cycles-pp.ioctl
6.07 ±103% -6.1 0.00 perf-profile.children.cycles-pp.perf_ioctl
6.07 ±103% -6.1 0.00 perf-profile.children.cycles-pp.perf_event_for_each_child
8.57 ±102% -4.9 3.70 ±141% perf-profile.children.cycles-pp.event_function_call
8.57 ±102% -4.9 3.70 ±141% perf-profile.children.cycles-pp.smp_call_function_single
4.77 ±100% -4.8 0.00 perf-profile.children.cycles-pp.page_fault
5.28 ±100% -1.6 3.70 ±141% perf-profile.children.cycles-pp.setlocale
4.77 ±100% -1.1 3.70 ±141% perf-profile.children.cycles-pp.link_path_walk
5.05 ±100% -0.9 4.17 ±141% perf-profile.children.cycles-pp.mutex_lock
8.57 ±102% -4.9 3.70 ±141% perf-profile.self.cycles-pp.smp_call_function_single
48515 ± 57% +62.1% 78654 interrupts.316:PCI-MSI.376832-edge.ahci[0000:00:17.0]
19956 ± 54% +95.5% 39006 ± 31% interrupts.CAL:Function_call_interrupts
535.00 ± 72% -90.2% 52.67 ± 79% interrupts.CPU2.RES:Rescheduling_interrupts
290.25 ± 78% +139.7% 695.67 ± 32% interrupts.CPU24.CAL:Function_call_interrupts
304.00 ± 76% +189.3% 879.33 ± 42% interrupts.CPU25.CAL:Function_call_interrupts
315.50 ± 77% +124.8% 709.33 ± 36% interrupts.CPU26.CAL:Function_call_interrupts
291.50 ± 72% +150.5% 730.33 ± 44% interrupts.CPU29.CAL:Function_call_interrupts
407.75 ±134% -95.0% 20.33 ± 85% interrupts.CPU3.RES:Rescheduling_interrupts
299.50 ± 68% +147.4% 741.00 ± 44% interrupts.CPU32.CAL:Function_call_interrupts
337.00 ± 78% +131.0% 778.33 ± 39% interrupts.CPU33.CAL:Function_call_interrupts
285.25 ± 71% +249.6% 997.33 ± 42% interrupts.CPU34.CAL:Function_call_interrupts
268.25 ± 88% +179.8% 750.67 ± 38% interrupts.CPU37.CAL:Function_call_interrupts
291.00 ± 75% +133.1% 678.33 ± 36% interrupts.CPU39.CAL:Function_call_interrupts
7.50 ±119% +740.0% 63.00 ± 86% interrupts.CPU40.RES:Rescheduling_interrupts
291.50 ± 74% +156.6% 748.00 ± 33% interrupts.CPU42.CAL:Function_call_interrupts
279.50 ± 69% +139.7% 670.00 ± 39% interrupts.CPU47.CAL:Function_call_interrupts
127.75 ± 9% -13.6% 110.33 ± 2% interrupts.CPU52.CAL:Function_call_interrupts
127.50 ± 9% -12.4% 111.67 ± 2% interrupts.CPU53.CAL:Function_call_interrupts
127.25 ± 9% -13.0% 110.67 ± 2% interrupts.CPU54.CAL:Function_call_interrupts
127.75 ± 9% -14.9% 108.67 ± 2% interrupts.CPU56.CAL:Function_call_interrupts
126.25 ± 5% -8.1% 116.00 ± 5% interrupts.CPU71.CAL:Function_call_interrupts
264.00 ± 68% +174.7% 725.33 ± 41% interrupts.CPU72.CAL:Function_call_interrupts
280.75 ± 70% +157.5% 723.00 ± 39% interrupts.CPU73.CAL:Function_call_interrupts
253.50 ± 89% +186.0% 725.00 ± 37% interrupts.CPU76.CAL:Function_call_interrupts
30.50 ± 59% +3123.0% 983.00 ± 65% interrupts.CPU8.RES:Rescheduling_interrupts
306.25 ± 78% +140.1% 735.33 ± 34% interrupts.CPU82.CAL:Function_call_interrupts
314.00 ± 78% +123.6% 702.00 ± 35% interrupts.CPU83.CAL:Function_call_interrupts
311.00 ± 72% +131.7% 720.67 ± 33% interrupts.CPU84.CAL:Function_call_interrupts
304.75 ± 78% +133.1% 710.33 ± 33% interrupts.CPU89.CAL:Function_call_interrupts
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
View attachment "config-5.5.0-rc4-00142-gd4ea3041f3fd0" of type "text/plain" (202322 bytes)
View attachment "job-script" of type "text/plain" (8219 bytes)
View attachment "job.yaml" of type "text/plain" (5800 bytes)
View attachment "reproduce" of type "text/plain" (1057 bytes)
Powered by blists - more mailing lists