| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Thu, 25 Feb 2021 20:22:41 -0700
From: Andreas Dilger <adilger@...ger.ca>
To: kernel test robot <oliver.sang@...el.com>
Cc: Harshad Shirwadkar <harshadshirwadkar@...il.com>,
0day robot <lkp@...el.com>, lkp@...ts.01.org,
ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com,
Ext4 Developers List <linux-ext4@...r.kernel.org>,
Theodore Ts'o <tytso@....edu>,
Alex Zhuravlev <bzzz@...mcloud.com>,
Благодаренко Артём
<artem.blagodarenko@...il.com>, Shuichi Ihara <sihara@....com>
Subject: Re: [ext4] ef4eebad9c:
fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works/sec -9.8% regression
On Feb 14, 2021, at 7:09 AM, kernel test robot <oliver.sang@...el.com> wrote:
>
>
> Greeting,
>
> FYI, we noticed a -9.8% regression of fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works/sec due to commit:
>
>
> commit: ef4eebad9c018a972a470b7b41e68bc981b31d00 ("ext4: improve cr 0 / cr 1 group scanning")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git Harshad-Shirwadkar/ext4-drop-s_mb_bal_lock-and-convert-protected-fields-to-atomic/20210210-054647
Hello Oliver and Intel team,
thanks for your regression report. It is definitely very useful to have
such a service running against patches before they are landed.
I'd like to make a couple of suggestions on how these emails could be
more useful to the developers and others who see such reports.
- it would be good to positively identify the source of the patch. In
several parts of this email it references the git hash ef4eebad9c,
but (AFAICS) this hash is only relevant in your local repository.
While reviewing this result, we were not totally positive which
version of the "ext4: improve cr 0 / cr 1 group scanning" patch was
being tested, since there more than one version of this patch was
sent to the list. Including the original Message-Id from the email
(I believe <20210209202857.4185846-5-harshadshirwadkar@...il.com> in
this case) would make it more obvious.
- the subject of the email is unrelated to the original patch, so it
is more difficult to tie this report to the original patch, and it
does not sort or thread together in the mail reader or archive. I
would recommend to make the subject match the original patch email
"Re: [PATCH v2 4/5] ext4: improve cr 0 / cr 1 group scanning" and add
"References: <20210209202857.4185846-5-harshadshirwadkar@...il.com>"
in the header so it threads together in the inbox and archives, and
"fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works/sec -9.8% regression"
as the first line in the body.
- while it is clear from the subject that there is a 9.8% regression
in a test case, it isn't totally clear what this test case is, nor
what is actually being measured (what is "works/sec", and does this
patch add "doesn't work/sec"? :-).
- it would be useful to add a URL in every such email pointing to a
general overview page like "So your patch got an email from the
Intel Kernel Test Robot" that explains what this means (maybe with
some nice stats showing how many patches Intel is testing, how many
tests are run on the systems you have, and generally showing what a
good job you are doing), along with a general explanation of how to
interpret the results in the email.
- it would be even more useful to have a link to a specific page that
explains what each test is doing (fxmark MWCL I guess) and what the
specific regression is ("works/sec" doesn't really mean anything
to me, and I have no idea what MWCL_1, _2, _36, _45 are). Maybe
this is already available somewhere, but having a link makes it much
easier to find.
- the performance comparison itself is a bit confusing, as it isn't
clear what "fail:runs" actually means. It _looks_ like the "before"
patch (a932b2b78) had as many test failures as the "after" patch,
both "0 of 4" or "1 of 4" test runs, which doesn't explain the
5%/10%/4%/9% reproduction%.
- the graphs at the end are (I guess) performance metrics vs. different
commit hashes during git bisect to find the broken patch (?), but the
x-axis isn't labelled, so it is hard to know. There is a bit of a
legend, showing "[*] bisect-good sample" and "[O] bisect-bad sample"
but there are no '*' on the graphs, only 'O' and '+' so it is hard
to know how to interpret them.
Thanks in any case for your work on this. My email is only intended
to help us get the most value and understanding of the effort that
you and the many hard-working Xeon Phi cores are doing on our behalf.
Cheers, Andreas
> in testcase: fxmark
> on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
> with following parameters:
>
> disk: 1HDD
> media: hdd
> test: MWCL
> fstype: ext4_no_jnl
> directio: bufferedio
> cpufreq_governor: performance
> ucode: 0x11
>
>
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <oliver.sang@...el.com>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp split-job --compatible job.yaml
> bin/lkp run compatible-job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
> gcc-9/performance/bufferedio/1HDD/ext4_no_jnl/x86_64-rhel-8.3/hdd/debian-10.4-x86_64-20200603.cgz/lkp-knm01/MWCL/fxmark/0x11
>
> commit:
> a932b2b788 ("ext4: add MB_NUM_ORDERS macro")
> ef4eebad9c ("ext4: improve cr 0 / cr 1 group scanning")
>
> a932b2b7885865bd ef4eebad9c018a972a470b7b41e
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> 0:4 5% 0:4 perf-profile.children.cycles-pp.error_return
> 1:4 10% 1:4 perf-profile.children.cycles-pp.error_entry
> :4 4% 0:4 perf-profile.self.cycles-pp.error_return
> 1:4 9% 1:4 perf-profile.self.cycles-pp.error_entry
> %stddev %change %stddev
> \ | \
> 315979 -9.8% 284922 fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works
> 10532 -9.8% 9497 fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works/sec
> 0.01 ±100% +38150.0% 1.91 ± 11% fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.iowait_sec
> 0.01 ±100% +38189.0% 3.16 ± 11% fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.iowait_util
> 5.33 ± 17% +22.5% 6.52 ± 4% fxmark.hdd_ext4_no_jnl_MWCL_36_bufferedio.idle_sec
> 0.49 ± 16% +22.2% 0.60 ± 4% fxmark.hdd_ext4_no_jnl_MWCL_36_bufferedio.idle_util
> 6.50 ± 9% -21.6% 5.09 ± 8% fxmark.hdd_ext4_no_jnl_MWCL_45_bufferedio.idle_sec
> 0.48 ± 9% -22.6% 0.37 ± 10% fxmark.hdd_ext4_no_jnl_MWCL_45_bufferedio.idle_util
> 0.00 ±173% +75800.0% 1.90 ± 22% fxmark.hdd_ext4_no_jnl_MWCL_4_bufferedio.iowait_sec
> 0.00 ±173% +75915.1% 1.57 ± 22% fxmark.hdd_ext4_no_jnl_MWCL_4_bufferedio.iowait_util
> 0.52 ± 6% -11.1% 0.46 ± 4% fxmark.hdd_ext4_no_jnl_MWCL_54_bufferedio.softirq_util
> 1090 +3.2% 1124 fxmark.time.elapsed_time
> 1090 +3.2% 1124 fxmark.time.elapsed_time.max
> 65107 -5.9% 61260 fxmark.time.involuntary_context_switches
> 69.50 -5.8% 65.50 fxmark.time.percent_of_cpu_this_job_got
> 28.28 -4.1% 27.11 ± 2% fxmark.time.user_time
> 5.50 ± 3% +2.8 8.26 ± 4% mpstat.cpu.all.iowait%
> 58.50 -2.6% 57.00 vmstat.cpu.id
> 38021 -6.2% 35647 vmstat.io.bo
> 85553 -4.1% 82045 vmstat.system.in
> 58.98 -2.7% 57.37 iostat.cpu.idle
> 5.57 ± 4% +49.8% 8.34 ± 4% iostat.cpu.iowait
> 30.35 -3.1% 29.41 iostat.cpu.system
> 2.81 -5.3% 2.66 iostat.cpu.user
> 711278 +15.3% 820380 meminfo.Dirty
> 7003710 -9.0% 6376219 meminfo.KReclaimable
> 1840 ± 12% +21.4% 2233 meminfo.Mlocked
> 7003710 -9.0% 6376219 meminfo.SReclaimable
> 710759 +15.4% 820265 numa-meminfo.node0.Dirty
> 6994361 -9.0% 6365487 numa-meminfo.node0.KReclaimable
> 1053 ± 12% +21.6% 1281 numa-meminfo.node0.Mlocked
> 6994361 -9.0% 6365487 numa-meminfo.node0.SReclaimable
> 177664 +15.5% 205237 numa-vmstat.node0.nr_dirty
> 262.75 ± 12% +21.9% 320.25 numa-vmstat.node0.nr_mlock
> 1751239 -9.0% 1594254 numa-vmstat.node0.nr_slab_reclaimable
> 178395 +15.4% 205952 numa-vmstat.node0.nr_zone_write_pending
> 2244 ± 68% -82.7% 387.72 ± 15% sched_debug.cfs_rq:/.load_avg.max
> 309.86 ± 59% -72.6% 84.98 ± 14% sched_debug.cfs_rq:/.load_avg.stddev
> 385204 ± 8% -35.5% 248625 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev
> -681107 -51.9% -327811 sched_debug.cfs_rq:/.spread0.min
> 385220 ± 8% -35.5% 248625 ± 6% sched_debug.cfs_rq:/.spread0.stddev
> 10.05 ± 51% +506.0% 60.92 ± 32% sched_debug.cfs_rq:/.util_est_enqueued.min
> 125.29 ± 14% -18.5% 102.09 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.stddev
> 24.34 ± 8% -21.6% 19.08 ± 2% sched_debug.cpu.clock.stddev
> 61783 ± 8% +33.0% 82157 ± 7% sched_debug.cpu.nr_switches.avg
> 35702 ± 8% +55.3% 55461 ± 11% sched_debug.cpu.nr_switches.min
> 7989 ± 25% +87.6% 14991 ± 24% softirqs.CPU1.BLOCK
> 123512 ± 3% -7.6% 114086 ± 2% softirqs.CPU21.RCU
> 122473 ± 3% -6.6% 114426 ± 2% softirqs.CPU25.RCU
> 66489 ± 5% -11.7% 58718 ± 5% softirqs.CPU29.SCHED
> 99247 ± 3% -8.6% 90723 ± 5% softirqs.CPU33.RCU
> 56394 ± 3% -13.5% 48805 ± 5% softirqs.CPU36.SCHED
> 43799 ± 4% -12.9% 38133 ± 4% softirqs.CPU45.SCHED
> 44447 ± 4% -12.0% 39128 ± 5% softirqs.CPU51.SCHED
> 169512 ± 3% -11.3% 150299 ± 3% softirqs.CPU6.RCU
> 33198 ± 5% -14.9% 28240 ± 11% softirqs.CPU60.SCHED
> 147310 ± 6% -9.0% 134107 ± 2% softirqs.CPU9.RCU
> 0.04 ± 6% -0.0 0.03 ± 14% perf-stat.i.branch-miss-rate%
> 326874 ± 8% -15.0% 277893 ± 12% perf-stat.i.branch-misses
> 41754 -4.6% 39817 perf-stat.i.cpu-clock
> 85.39 -2.9% 82.87 perf-stat.i.cpu-migrations
> 0.38 ± 10% -16.2% 0.32 ± 11% perf-stat.i.instructions-per-iTLB-miss
> 0.00 ± 11% -17.2% 0.00 ± 11% perf-stat.i.ipc
> 1.06 ± 3% -7.8% 0.98 perf-stat.i.major-faults
> 0.35 +4.1% 0.37 perf-stat.i.metric.K/sec
> 41754 -4.6% 39817 perf-stat.i.task-clock
> 348107 ± 7% -14.8% 296451 ± 12% perf-stat.ps.branch-misses
> 41967 -4.6% 40020 perf-stat.ps.cpu-clock
> 85.62 -2.9% 83.09 perf-stat.ps.cpu-migrations
> 1.05 ± 3% -7.7% 0.97 perf-stat.ps.major-faults
> 41967 -4.6% 40020 perf-stat.ps.task-clock
> 0.11 ± 8% -33.2% 0.07 ± 28% perf-sched.sch_delay.avg.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
> 0.02 ± 9% -100.0% 0.00 perf-sched.sch_delay.avg.ms.kthreadd.ret_from_fork
> 0.28 ± 83% -86.7% 0.04 ± 33% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.submit_bh_wbc
> 0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.kthread.ret_from_fork
> 0.06 ± 19% -28.4% 0.04 ± 8% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
> 0.01 ±100% +141.3% 0.03 ± 8% perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
> 0.06 ± 10% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
> 0.35 ±113% -79.7% 0.07 ± 40% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
> 2.77 ± 40% -46.4% 1.49 ± 53% perf-sched.sch_delay.max.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
> 0.03 ± 21% -100.0% 0.00 perf-sched.sch_delay.max.ms.kthreadd.ret_from_fork
> 0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_preempt_disabled.kthread.ret_from_fork
> 0.06 ± 13% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
> 139.75 ± 7% -13.4% 121.00 ± 3% perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.shrink_dentry_list.prune_dcache_sb.super_cache_scan
> 8210 ± 10% -26.3% 6048 ± 12% perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
> 88.37 ± 15% -18.2% 72.31 ± 11% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.submit_bh_wbc
> 79.45 ±109% +329.8% 341.45 ± 42% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.mutex_lock.drm_gem_shmem_vunmap.mgag200_handle_damage
> 129.91 ± 2% +52.5% 198.10 ± 48% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
> 130.18 ± 3% +72.5% 224.52 ± 51% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.write_cache_pages.generic_writepages.do_writepages
> 8210 ± 10% -26.3% 6048 ± 12% perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
> 639.00 -4.1% 613.00 proc-vmstat.nr_active_anon
> 109230 -4.7% 104085 proc-vmstat.nr_active_file
> 9734223 -3.3% 9414937 proc-vmstat.nr_dirtied
> 178266 +15.5% 205864 proc-vmstat.nr_dirty
> 460.75 ± 12% +21.4% 559.50 proc-vmstat.nr_mlock
> 1758100 -8.9% 1601542 proc-vmstat.nr_slab_reclaimable
> 68945 -3.0% 66853 proc-vmstat.nr_slab_unreclaimable
> 9734223 -3.3% 9414937 proc-vmstat.nr_written
> 639.00 -4.1% 613.00 proc-vmstat.nr_zone_active_anon
> 109230 -4.7% 104085 proc-vmstat.nr_zone_active_file
> 179007 +15.4% 206596 proc-vmstat.nr_zone_write_pending
> 24225927 -2.2% 23703313 proc-vmstat.numa_hit
> 24225924 -2.2% 23703311 proc-vmstat.numa_local
> 47793203 -3.0% 46353511 proc-vmstat.pgalloc_normal
> 4923908 +11.4% 5485129 proc-vmstat.pgdeactivate
> 3348086 +2.3% 3425886 proc-vmstat.pgfault
> 47786479 -3.0% 46346216 proc-vmstat.pgfree
> 41377300 -3.3% 40023642 proc-vmstat.pgpgout
> 264776 +2.5% 271513 proc-vmstat.pgreuse
> 4916073 +11.4% 5477332 proc-vmstat.pgrotated
> 1.779e+08 -2.8% 1.729e+08 proc-vmstat.slabs_scanned
> 9334464 +2.8% 9594624 proc-vmstat.unevictable_pgs_scanned
> 662.25 ± 8% -15.7% 558.50 ± 6% slabinfo.Acpi-Parse.active_objs
> 3972051 -9.0% 3616212 slabinfo.dentry.active_objs
> 189593 -8.9% 172660 slabinfo.dentry.active_slabs
> 3981471 -8.9% 3625865 slabinfo.dentry.num_objs
> 189593 -8.9% 172660 slabinfo.dentry.num_slabs
> 3665 +602.8% 25759 slabinfo.ext4_extent_status.active_objs
> 39.75 +558.5% 261.75 slabinfo.ext4_extent_status.active_slabs
> 4090 +554.0% 26752 slabinfo.ext4_extent_status.num_objs
> 39.75 +558.5% 261.75 slabinfo.ext4_extent_status.num_slabs
> 4203 ± 3% -100.0% 0.00 slabinfo.ext4_groupinfo_4k.active_objs
> 4254 ± 2% -100.0% 0.00 slabinfo.ext4_groupinfo_4k.num_objs
> 5178202 -9.1% 4707049 slabinfo.ext4_inode_cache.active_objs
> 191816 -9.1% 174364 slabinfo.ext4_inode_cache.active_slabs
> 5179060 -9.1% 4707847 slabinfo.ext4_inode_cache.num_objs
> 191816 -9.1% 174364 slabinfo.ext4_inode_cache.num_slabs
> 1133 ± 5% -14.9% 965.00 ± 11% slabinfo.kmalloc-rcl-96.num_objs
> 20676 +24.1% 25662 slabinfo.radix_tree_node.active_objs
> 1642 +14.8% 1885 slabinfo.radix_tree_node.active_slabs
> 23002 +14.8% 26403 slabinfo.radix_tree_node.num_objs
> 1642 +14.8% 1885 slabinfo.radix_tree_node.num_slabs
> 1069 ± 7% +16.6% 1246 ± 6% slabinfo.skbuff_fclone_cache.active_objs
> 1250 ± 5% +15.8% 1448 ± 6% slabinfo.skbuff_fclone_cache.num_objs
> 3019 +122.2% 6710 interrupts.CPU0.180:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
> 14733 ± 10% +135.6% 34711 ± 39% interrupts.CPU1.180:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
> 74.25 ± 41% +328.6% 318.25 ± 54% interrupts.CPU1.37:IR-PCI-MSI.4194305-edge.eth0-TxRx-0
> 4354 ± 25% +29.9% 5655 ± 13% interrupts.CPU11.CAL:Function_call_interrupts
> 1283 +25.2% 1607 ± 29% interrupts.CPU127.CAL:Function_call_interrupts
> 10568 ± 28% +34.3% 14193 ± 15% interrupts.CPU2.CAL:Function_call_interrupts
> 985.00 ± 22% +130.9% 2274 ± 42% interrupts.CPU2.RES:Rescheduling_interrupts
> 263.25 ± 4% +24.5% 327.75 ± 20% interrupts.CPU2.TLB:TLB_shootdowns
> 312.00 ± 72% -50.9% 153.25 ± 22% interrupts.CPU20.NMI:Non-maskable_interrupts
> 312.00 ± 72% -50.9% 153.25 ± 22% interrupts.CPU20.PMI:Performance_monitoring_interrupts
> 4243 ± 10% +45.5% 6172 ± 5% interrupts.CPU22.CAL:Function_call_interrupts
> 3434 ± 20% +58.2% 5433 ± 35% interrupts.CPU25.CAL:Function_call_interrupts
> 491.25 ± 29% -55.7% 217.75 ± 35% interrupts.CPU27.NMI:Non-maskable_interrupts
> 491.25 ± 29% -55.7% 217.75 ± 35% interrupts.CPU27.PMI:Performance_monitoring_interrupts
> 390.50 ± 40% -46.4% 209.50 ± 9% interrupts.CPU29.RES:Rescheduling_interrupts
> 189.50 ± 11% +23.9% 234.75 ± 5% interrupts.CPU3.TLB:TLB_shootdowns
> 234.75 ± 32% -39.8% 141.25 ± 29% interrupts.CPU30.NMI:Non-maskable_interrupts
> 234.75 ± 32% -39.8% 141.25 ± 29% interrupts.CPU30.PMI:Performance_monitoring_interrupts
> 639.50 ± 65% -53.0% 300.75 ± 26% interrupts.CPU30.RES:Rescheduling_interrupts
> 371.50 ± 24% -32.5% 250.75 ± 8% interrupts.CPU34.RES:Rescheduling_interrupts
> 246.00 ± 23% -32.5% 166.00 ± 7% interrupts.CPU37.RES:Rescheduling_interrupts
> 550.25 ± 11% +91.9% 1055 ± 28% interrupts.CPU4.RES:Rescheduling_interrupts
> 165.75 ± 20% +108.1% 345.00 ± 47% interrupts.CPU47.NMI:Non-maskable_interrupts
> 165.75 ± 20% +108.1% 345.00 ± 47% interrupts.CPU47.PMI:Performance_monitoring_interrupts
> 2914 ± 10% +50.3% 4380 ± 23% interrupts.CPU48.CAL:Function_call_interrupts
> 6123 ± 9% +43.8% 8808 ± 18% interrupts.CPU5.CAL:Function_call_interrupts
> 146.25 ± 10% +185.0% 416.75 ± 30% interrupts.CPU5.NMI:Non-maskable_interrupts
> 146.25 ± 10% +185.0% 416.75 ± 30% interrupts.CPU5.PMI:Performance_monitoring_interrupts
> 477.50 ± 62% -70.2% 142.50 ± 22% interrupts.CPU6.NMI:Non-maskable_interrupts
> 477.50 ± 62% -70.2% 142.50 ± 22% interrupts.CPU6.PMI:Performance_monitoring_interrupts
> 580.00 ± 27% +127.7% 1320 ± 42% interrupts.CPU6.RES:Rescheduling_interrupts
> 479.50 ± 35% -56.8% 207.25 ± 62% interrupts.CPU62.NMI:Non-maskable_interrupts
> 479.50 ± 35% -56.8% 207.25 ± 62% interrupts.CPU62.PMI:Performance_monitoring_interrupts
> 1816 ± 14% +35.6% 2463 ± 29% interrupts.CPU65.CAL:Function_call_interrupts
> 142.25 ±100% -66.3% 48.00 ± 10% interrupts.CPU66.RES:Rescheduling_interrupts
> 459.50 ± 10% +42.2% 653.50 ± 16% interrupts.CPU7.RES:Rescheduling_interrupts
> 1282 +32.5% 1699 ± 27% interrupts.CPU97.CAL:Function_call_interrupts
> 1301 ± 2% +26.9% 1650 ± 28% interrupts.CPU98.CAL:Function_call_interrupts
> 12.78 ± 2% -1.9 10.92 ± 5% perf-profile.calltrace.cycles-pp.ret_from_fork
> 12.78 ± 2% -1.9 10.92 ± 5% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
> 4.48 ± 6% -1.6 2.90 ± 9% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
> 4.55 ± 6% -1.6 2.98 ± 9% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.calltrace.cycles-pp.write_cache_pages.generic_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.calltrace.cycles-pp.generic_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
> 3.07 ± 8% -1.3 1.80 ± 10% perf-profile.calltrace.cycles-pp.__writepage.write_cache_pages.generic_writepages.do_writepages.__writeback_single_inode
> 2.98 ± 8% -1.2 1.75 ± 10% perf-profile.calltrace.cycles-pp.__block_write_full_page.__writepage.write_cache_pages.generic_writepages.do_writepages
> 2.06 ± 8% -0.9 1.20 ± 11% perf-profile.calltrace.cycles-pp.submit_bh_wbc.__block_write_full_page.__writepage.write_cache_pages.generic_writepages
> 14.52 ± 2% -0.6 13.89 perf-profile.calltrace.cycles-pp.shrink_dentry_list.prune_dcache_sb.super_cache_scan.do_shrink_slab.shrink_slab
> 16.80 -0.6 16.21 perf-profile.calltrace.cycles-pp.prune_dcache_sb.super_cache_scan.do_shrink_slab.shrink_slab.drop_slab_node
> 1.34 ± 9% -0.6 0.78 ± 8% perf-profile.calltrace.cycles-pp.submit_bio.submit_bh_wbc.__block_write_full_page.__writepage.write_cache_pages
> 1.29 ± 10% -0.5 0.77 ± 9% perf-profile.calltrace.cycles-pp.submit_bio_noacct.submit_bio.submit_bh_wbc.__block_write_full_page.__writepage
> 0.94 ± 7% -0.5 0.48 ± 59% perf-profile.calltrace.cycles-pp.end_bio_bh_io_sync.blk_update_request.scsi_end_request.scsi_io_completion.blk_done_softirq
> 1.23 ± 7% -0.4 0.81 ± 14% perf-profile.calltrace.cycles-pp.blk_done_softirq.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
> 1.21 ± 7% -0.4 0.81 ± 14% perf-profile.calltrace.cycles-pp.scsi_io_completion.blk_done_softirq.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
> 1.21 ± 7% -0.4 0.81 ± 14% perf-profile.calltrace.cycles-pp.scsi_end_request.scsi_io_completion.blk_done_softirq.__softirqentry_text_start.run_ksoftirqd
> 1.17 ± 7% -0.4 0.77 ± 14% perf-profile.calltrace.cycles-pp.blk_update_request.scsi_end_request.scsi_io_completion.blk_done_softirq.__softirqentry_text_start
> 2.99 ± 2% -0.2 2.79 ± 3% perf-profile.calltrace.cycles-pp.__d_drop.__dentry_kill.shrink_dentry_list.prune_dcache_sb.super_cache_scan
> 2.93 ± 2% -0.2 2.73 ± 3% perf-profile.calltrace.cycles-pp.___d_drop.__d_drop.__dentry_kill.shrink_dentry_list.prune_dcache_sb
> 2.30 -0.1 2.18 ± 3% perf-profile.calltrace.cycles-pp.shrink_lock_dentry.shrink_dentry_list.prune_dcache_sb.super_cache_scan.do_shrink_slab
> 1.18 ± 4% -0.1 1.09 ± 3% perf-profile.calltrace.cycles-pp.rcu_cblist_dequeue.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
> 0.56 ± 6% +0.1 0.70 ± 8% perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
> 0.58 ± 4% +0.1 0.73 ± 7% perf-profile.calltrace.cycles-pp.ext4_discard_preallocations.ext4_clear_inode.ext4_evict_inode.evict.dispose_list
> 0.79 ± 4% +0.2 0.95 ± 10% perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
> 5.57 +0.3 5.88 ± 4% perf-profile.calltrace.cycles-pp.evict.dispose_list.prune_icache_sb.super_cache_scan.do_shrink_slab
> 7.17 +0.4 7.55 ± 2% perf-profile.calltrace.cycles-pp.dispose_list.prune_icache_sb.super_cache_scan.do_shrink_slab.shrink_slab
> 8.87 +0.5 9.33 ± 2% perf-profile.calltrace.cycles-pp.prune_icache_sb.super_cache_scan.do_shrink_slab.shrink_slab.drop_slab_node
> 5.66 ± 2% +0.5 6.16 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
> 0.00 +0.6 0.57 ± 9% perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
> 8.54 ± 2% +0.9 9.47 ± 3% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
> 26.26 +1.1 27.37 ± 3% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 24.24 ± 2% +1.1 25.38 ± 3% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
> 14.93 ± 2% +1.3 16.23 ± 2% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
> 14.28 ± 2% +1.4 15.63 ± 3% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
> 15.05 ± 2% +1.4 16.42 ± 2% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
> 53.89 ± 2% +2.1 56.02 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
> 53.09 ± 2% +2.2 55.29 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> 42.65 ± 2% +2.2 44.86 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
> 43.61 ± 2% +2.2 45.83 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> 53.16 ± 2% +2.2 55.40 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> 53.16 ± 2% +2.2 55.40 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
> 12.81 ± 2% -1.9 10.94 ± 5% perf-profile.children.cycles-pp.ret_from_fork
> 12.78 ± 2% -1.9 10.92 ± 5% perf-profile.children.cycles-pp.kthread
> 4.48 ± 6% -1.6 2.90 ± 9% perf-profile.children.cycles-pp.process_one_work
> 4.55 ± 6% -1.6 2.98 ± 9% perf-profile.children.cycles-pp.worker_thread
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.children.cycles-pp.wb_workfn
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.children.cycles-pp.wb_writeback
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.children.cycles-pp.__writeback_inodes_wb
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.children.cycles-pp.writeback_sb_inodes
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.children.cycles-pp.__writeback_single_inode
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.children.cycles-pp.do_writepages
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.children.cycles-pp.generic_writepages
> 3.70 ± 7% -1.5 2.19 ± 10% perf-profile.children.cycles-pp.write_cache_pages
> 3.07 ± 8% -1.3 1.80 ± 10% perf-profile.children.cycles-pp.__writepage
> 2.98 ± 8% -1.2 1.75 ± 10% perf-profile.children.cycles-pp.__block_write_full_page
> 2.06 ± 8% -0.9 1.20 ± 10% perf-profile.children.cycles-pp.submit_bh_wbc
> 1.78 ± 6% -0.6 1.13 ± 17% perf-profile.children.cycles-pp.blk_done_softirq
> 1.76 ± 6% -0.6 1.11 ± 17% perf-profile.children.cycles-pp.scsi_io_completion
> 1.76 ± 6% -0.6 1.11 ± 17% perf-profile.children.cycles-pp.scsi_end_request
> 14.55 ± 2% -0.6 13.92 perf-profile.children.cycles-pp.shrink_dentry_list
> 1.68 ± 7% -0.6 1.07 ± 17% perf-profile.children.cycles-pp.blk_update_request
> 16.80 -0.6 16.21 perf-profile.children.cycles-pp.prune_dcache_sb
> 1.34 ± 10% -0.6 0.78 ± 9% perf-profile.children.cycles-pp.submit_bio
> 1.29 ± 10% -0.5 0.77 ± 8% perf-profile.children.cycles-pp.submit_bio_noacct
> 1.35 ± 7% -0.5 0.84 ± 18% perf-profile.children.cycles-pp.end_bio_bh_io_sync
> 0.97 ± 8% -0.3 0.62 ± 17% perf-profile.children.cycles-pp.end_page_writeback
> 0.79 ± 6% -0.3 0.49 ± 9% perf-profile.children.cycles-pp.blk_mq_submit_bio
> 0.67 ± 12% -0.3 0.40 ± 12% perf-profile.children.cycles-pp.__test_set_page_writeback
> 0.57 ± 8% -0.2 0.35 ± 30% perf-profile.children.cycles-pp.sysvec_call_function_single
> 0.57 ± 8% -0.2 0.35 ± 31% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
> 3.01 ± 2% -0.2 2.80 ± 3% perf-profile.children.cycles-pp.__d_drop
> 0.55 ± 10% -0.2 0.34 ± 21% perf-profile.children.cycles-pp.test_clear_page_writeback
> 2.94 ± 2% -0.2 2.75 ± 3% perf-profile.children.cycles-pp.___d_drop
> 0.42 ± 8% -0.2 0.23 ± 17% perf-profile.children.cycles-pp.bio_alloc_bioset
> 0.40 ± 16% -0.2 0.23 ± 14% perf-profile.children.cycles-pp.submit_bio_checks
> 0.51 ± 12% -0.2 0.36 ± 8% perf-profile.children.cycles-pp.kmem_cache_alloc
> 0.32 ± 12% -0.1 0.17 ± 19% perf-profile.children.cycles-pp.mempool_alloc
> 0.32 ± 6% -0.1 0.18 ± 10% perf-profile.children.cycles-pp.clear_page_dirty_for_io
> 0.38 ± 8% -0.1 0.25 ± 17% perf-profile.children.cycles-pp.rotate_reclaimable_page
> 2.31 -0.1 2.19 ± 3% perf-profile.children.cycles-pp.shrink_lock_dentry
> 0.45 ± 11% -0.1 0.33 ± 5% perf-profile.children.cycles-pp.try_to_wake_up
> 0.28 ± 18% -0.1 0.16 ± 27% perf-profile.children.cycles-pp.end_buffer_async_write
> 0.28 ± 7% -0.1 0.18 ± 21% perf-profile.children.cycles-pp.blk_attempt_plug_merge
> 0.19 ± 15% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.percpu_counter_add_batch
> 0.16 ± 16% -0.1 0.08 ± 68% perf-profile.children.cycles-pp.__slab_alloc
> 0.29 ± 11% -0.1 0.21 ± 15% perf-profile.children.cycles-pp.pagevec_lru_move_fn
> 0.21 ± 21% -0.1 0.13 ± 11% perf-profile.children.cycles-pp.open64
> 0.28 ± 14% -0.1 0.20 ± 4% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
> 0.15 ± 19% -0.1 0.07 ± 71% perf-profile.children.cycles-pp.fscrypt_drop_inode
> 0.17 ± 13% -0.1 0.10 ± 11% perf-profile.children.cycles-pp.bio_attempt_back_merge
> 0.15 ± 15% -0.1 0.07 ± 67% perf-profile.children.cycles-pp.___slab_alloc
> 0.24 ± 14% -0.1 0.16 ± 11% perf-profile.children.cycles-pp.pagevec_move_tail_fn
> 0.14 ± 21% -0.1 0.07 ± 19% perf-profile.children.cycles-pp.blk_throtl_bio
> 0.21 ± 14% -0.1 0.15 ± 9% perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list
> 0.10 ± 14% -0.1 0.04 ±101% perf-profile.children.cycles-pp.allocate_slab
> 0.12 ± 25% -0.1 0.06 ± 26% perf-profile.children.cycles-pp.__mod_lruvec_state
> 0.20 ± 13% -0.1 0.15 ± 11% perf-profile.children.cycles-pp.scsi_queue_rq
> 0.10 ± 25% -0.1 0.05 ± 62% perf-profile.children.cycles-pp.__close_nocancel
> 0.08 ± 15% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__split_vma
> 0.17 ± 8% -0.1 0.12 ± 9% perf-profile.children.cycles-pp.can_stop_idle_tick
> 0.15 ± 19% -0.0 0.11 ± 28% perf-profile.children.cycles-pp.get_page_from_freelist
> 0.09 ± 20% -0.0 0.05 ± 62% perf-profile.children.cycles-pp.__vm_munmap
> 0.15 ± 10% -0.0 0.11 ± 11% perf-profile.children.cycles-pp.schedule_timeout
> 0.14 ± 13% -0.0 0.10 ± 25% perf-profile.children.cycles-pp.call_timer_fn
> 0.09 ± 13% -0.0 0.05 ± 58% perf-profile.children.cycles-pp.enqueue_entity
> 0.23 ± 7% -0.0 0.20 ± 4% perf-profile.children.cycles-pp.rcu_segcblist_enqueue
> 0.23 ± 6% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.rcu_gp_kthread
> 0.17 ± 9% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.tick_nohz_idle_got_tick
> 0.10 ± 8% -0.0 0.08 ± 19% perf-profile.children.cycles-pp.enqueue_task_fair
> 0.04 ± 60% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.rcu_irq_enter
> 0.06 ± 11% +0.0 0.10 ± 12% perf-profile.children.cycles-pp.arch_cpu_idle_exit
> 0.14 ± 7% +0.0 0.19 ± 16% perf-profile.children.cycles-pp.update_dl_rq_load_avg
> 0.07 ± 58% +0.1 0.12 ± 12% perf-profile.children.cycles-pp.delay_tsc
> 0.44 ± 5% +0.1 0.49 ± 4% perf-profile.children.cycles-pp.truncate_inode_pages_final
> 0.18 ± 26% +0.1 0.23 ± 5% perf-profile.children.cycles-pp.update_ts_time_stats
> 0.00 +0.1 0.07 ± 17% perf-profile.children.cycles-pp.perf_iterate_sb
> 0.11 ± 17% +0.1 0.19 ± 21% perf-profile.children.cycles-pp.tick_program_event
> 0.17 ± 17% +0.1 0.24 ± 4% perf-profile.children.cycles-pp.cpuidle_not_available
> 0.46 ± 6% +0.1 0.54 ± 6% perf-profile.children.cycles-pp.__x86_retpoline_rax
> 0.02 ±173% +0.1 0.11 ± 25% perf-profile.children.cycles-pp.cpuidle_get_cpu_driver
> 0.80 ± 4% +0.1 0.90 ± 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.58 ± 10% +0.1 0.71 ± 7% perf-profile.children.cycles-pp.enqueue_hrtimer
> 1.74 +0.1 1.87 ± 4% perf-profile.children.cycles-pp.__list_del_entry_valid
> 0.45 ± 12% +0.1 0.59 ± 6% perf-profile.children.cycles-pp.timerqueue_add
> 0.59 ± 4% +0.1 0.73 ± 7% perf-profile.children.cycles-pp.ext4_discard_preallocations
> 0.87 ± 6% +0.2 1.02 ± 10% perf-profile.children.cycles-pp.rcu_sched_clock_irq
> 0.53 ± 6% +0.2 0.71 ± 9% perf-profile.children.cycles-pp.timerqueue_del
> 0.66 ± 9% +0.2 0.84 ± 8% perf-profile.children.cycles-pp.__remove_hrtimer
> 0.26 ± 35% +0.2 0.45 ± 18% perf-profile.children.cycles-pp.timekeeping_max_deferment
> 7.18 +0.4 7.55 ± 2% perf-profile.children.cycles-pp.dispose_list
> 5.14 +0.4 5.53 ± 3% perf-profile.children.cycles-pp.kmem_cache_free
> 8.87 +0.5 9.33 ± 2% perf-profile.children.cycles-pp.prune_icache_sb
> 10.50 ± 2% +1.7 12.19 ± 11% perf-profile.children.cycles-pp.__hrtimer_run_queues
> 44.20 ± 2% +2.1 46.30 perf-profile.children.cycles-pp.cpuidle_enter
> 16.58 ± 2% +2.1 18.70 ± 8% perf-profile.children.cycles-pp.hrtimer_interrupt
> 53.89 ± 2% +2.1 56.02 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
> 53.89 ± 2% +2.1 56.02 perf-profile.children.cycles-pp.cpu_startup_entry
> 53.89 ± 2% +2.1 56.02 perf-profile.children.cycles-pp.do_idle
> 44.06 ± 2% +2.2 46.23 perf-profile.children.cycles-pp.cpuidle_enter_state
> 28.19 ± 2% +2.2 30.37 ± 3% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
> 17.26 +2.2 19.47 ± 7% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
> 53.16 ± 2% +2.2 55.40 perf-profile.children.cycles-pp.start_secondary
> 29.75 ± 2% +2.3 32.02 ± 3% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
> 2.56 ± 2% -0.3 2.25 ± 7% perf-profile.self.cycles-pp.___d_drop
> 0.77 ± 6% -0.2 0.59 ± 10% perf-profile.self.cycles-pp.tick_nohz_next_event
> 0.12 ± 23% -0.1 0.04 ±101% perf-profile.self.cycles-pp.fscrypt_drop_inode
> 0.16 ± 10% -0.1 0.08 ± 10% perf-profile.self.cycles-pp.percpu_counter_add_batch
> 0.15 ± 22% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.__test_set_page_writeback
> 0.09 ± 14% -0.1 0.03 ±100% perf-profile.self.cycles-pp.clear_page_dirty_for_io
> 0.17 ± 8% -0.1 0.11 ± 21% perf-profile.self.cycles-pp.__block_write_full_page
> 0.19 ± 21% -0.1 0.13 ± 3% perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.19 ± 5% -0.1 0.14 ± 10% perf-profile.self.cycles-pp.cpuidle_governor_latency_req
> 0.12 ± 7% -0.1 0.07 ± 62% perf-profile.self.cycles-pp.cpuidle_enter
> 0.10 ± 14% -0.1 0.05 ± 60% perf-profile.self.cycles-pp.end_bio_bh_io_sync
> 0.17 ± 8% -0.1 0.12 ± 9% perf-profile.self.cycles-pp.can_stop_idle_tick
> 0.23 ± 7% -0.0 0.19 ± 3% perf-profile.self.cycles-pp.rcu_segcblist_enqueue
> 0.08 ± 23% -0.0 0.04 ± 63% perf-profile.self.cycles-pp.find_get_pages_range_tag
> 0.08 ± 6% -0.0 0.04 ± 59% perf-profile.self.cycles-pp.__d_drop
> 0.09 ± 13% +0.0 0.12 ± 15% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
> 0.10 ± 10% +0.0 0.14 ± 5% perf-profile.self.cycles-pp.tick_sched_handle
> 0.36 ± 5% +0.0 0.40 ± 2% perf-profile.self.cycles-pp.__x86_retpoline_rax
> 0.09 ± 27% +0.0 0.13 ± 17% perf-profile.self.cycles-pp.tick_nohz_tick_stopped
> 0.16 ± 7% +0.1 0.21 ± 16% perf-profile.self.cycles-pp.timerqueue_del
> 0.07 ± 58% +0.1 0.12 ± 12% perf-profile.self.cycles-pp.delay_tsc
> 0.01 ±173% +0.1 0.07 ± 7% perf-profile.self.cycles-pp.arch_cpu_idle_exit
> 0.18 ± 10% +0.1 0.23 ± 17% perf-profile.self.cycles-pp.update_blocked_averages
> 0.13 ± 8% +0.1 0.19 ± 16% perf-profile.self.cycles-pp.update_dl_rq_load_avg
> 0.11 ± 15% +0.1 0.18 ± 22% perf-profile.self.cycles-pp.tick_program_event
> 0.00 +0.1 0.07 ± 17% perf-profile.self.cycles-pp.rcu_irq_enter
> 0.19 ± 5% +0.1 0.26 ± 11% perf-profile.self.cycles-pp.__hrtimer_get_next_event
> 0.10 ± 27% +0.1 0.17 ± 8% perf-profile.self.cycles-pp.update_ts_time_stats
> 0.27 ± 6% +0.1 0.34 ± 8% perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
> 0.16 ± 19% +0.1 0.24 ± 3% perf-profile.self.cycles-pp.cpuidle_not_available
> 0.40 ± 6% +0.1 0.48 ± 10% perf-profile.self.cycles-pp.ext4_discard_preallocations
> 0.02 ±173% +0.1 0.10 ± 22% perf-profile.self.cycles-pp.cpuidle_get_cpu_driver
> 0.33 ± 11% +0.1 0.42 ± 14% perf-profile.self.cycles-pp.rb_erase
> 0.20 ± 16% +0.1 0.29 ± 7% perf-profile.self.cycles-pp.timerqueue_add
> 0.18 ± 8% +0.1 0.28 ± 18% perf-profile.self.cycles-pp.irq_exit_rcu
> 0.71 ± 5% +0.1 0.84 ± 2% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.67 ± 8% +0.2 0.84 ± 12% perf-profile.self.cycles-pp.rcu_sched_clock_irq
> 0.25 ± 36% +0.2 0.45 ± 18% perf-profile.self.cycles-pp.timekeeping_max_deferment
>
>
>
> fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.works_sec
>
> 23000 +-------------------------------------------------------------------+
> |.+..+ +..+.+.+..+.+..+.+..+.+.+..+.+..+.+..+.+.+..+.+..+.+..+.|
> 22500 |-+ |
> 22000 |-+ |
> | |
> 21500 |-+ |
> | O O O O O O O O |
> 21000 |-+ O O O O O O O O O O O |
> | O |
> 20500 |-+ |
> 20000 |-+ |
> | |
> 19500 |-+ O O |
> | O O |
> 19000 +-------------------------------------------------------------------+
>
>
> fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.iowait_sec
>
> 2.5 +---------------------------------------------------------------------+
> | O |
> | O O O O O O |
> 2 |-+ O O O |
> | O O O O O |
> | O O O O O O O O |
> 1.5 |-+ O |
> | |
> 1 |-+ |
> | |
> | |
> 0.5 |-+ |
> | |
> | |
> 0 +---------------------------------------------------------------------+
>
>
> fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.iowait_util
>
> 4 +---------------------------------------------------------------------+
> | O O O O |
> 3.5 |-+ O O O O O O |
> 3 |-+ |
> | O O O O O O |
> 2.5 |-+ O O O O O O O O |
> | |
> 2 |-+ |
> | |
> 1.5 |-+ |
> 1 |-+ |
> | |
> 0.5 |-+ |
> | |
> 0 +---------------------------------------------------------------------+
>
>
> fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works
>
> 320000 +------------------------------------------------------------------+
> |. .. + .+.+.+.. .+.+.+..+.+..+.+.+..+.+.+..+. .+.+.+..+.+..+.|
> 310000 |-+ +. +.+. +. |
> | |
> 300000 |-+ |
> | |
> 290000 |-+ O O O O |
> | O O O O O O O O O O O O O O |
> 280000 |-+ O O |
> | |
> 270000 |-+ |
> | O |
> 260000 |-O O O |
> | |
> 250000 +------------------------------------------------------------------+
>
>
> fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works_sec
>
> 10800 +-------------------------------------------------------------------+
> 10600 |-+ + .+ .+. .+ +.. |
> |. .. + .+.+. + .+..+.+..+.+..+.+.+..+.+. +. + + +.+..+.+..+.|
> 10400 |-+ +. + + |
> 10200 |-+ |
> 10000 |-+ |
> 9800 |-+ |
> | O O |
> 9600 |-+ O O O O O O O O O O |
> 9400 |-+ O O O O O O O O |
> 9200 |-+ |
> 9000 |-+ |
> | |
> 8800 |-O O O |
> 8600 +-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Oliver Sang
>
> <config-5.11.0-rc6-00009-gef4eebad9c01><job-script.txt><job.yaml><reproduce.txt>
Cheers, Andreas
Download attachment "signature.asc" of type "application/pgp-signature" (874 bytes)
Powered by blists - more mailing lists