[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <a080165c-824f-3d48-44e8-388fa94d32fe@linux.intel.com>
Date: Wed, 10 Jun 2020 11:07:14 +0800
From: Xing Zhengjun <zhengjun.xing@...ux.intel.com>
To: kernel test robot <rong.a.chen@...el.com>,
"Darrick J. Wong" <darrick.wong@...cle.com>
Cc: Christoph Hellwig <hch@....de>, Brian Foster <bfoster@...hat.com>,
LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: Re: [LKP] [xfs] a5949d3fae: aim7.jobs-per-min -33.6% regression
Hi Darrick,
Do you have time to take a look at this? Thanks.
On 6/6/2020 11:48 PM, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a -33.6% regression of aim7.jobs-per-min due to commit:
>
>
> commit: a5949d3faedf492fa7863b914da408047ab46eb0 ("xfs: force writes to delalloc regions to unwritten")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: aim7
> on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
> with following parameters:
>
> disk: 1BRD_48G
> fs: xfs
> test: sync_disk_rw
> load: 600
> cpufreq_governor: performance
> ucode: 0x42e
>
> test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
> test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
>
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <rong.a.chen@...el.com>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
> gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e
>
> commit:
> 590b16516e ("xfs: refactor xfs_iomap_prealloc_size")
> a5949d3fae ("xfs: force writes to delalloc regions to unwritten")
>
> 590b16516ef38e2e a5949d3faedf492fa7863b914da
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> :4 50% 2:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
> %stddev %change %stddev
> \ | \
> 35272 -33.6% 23430 aim7.jobs-per-min
> 102.13 +50.5% 153.75 aim7.time.elapsed_time
> 102.13 +50.5% 153.75 aim7.time.elapsed_time.max
> 1388038 +40.2% 1945838 aim7.time.involuntary_context_switches
> 43420 ± 2% +13.4% 49255 ± 2% aim7.time.minor_page_faults
> 3123 +44.2% 4504 ± 2% aim7.time.system_time
> 59.31 +6.5% 63.18 aim7.time.user_time
> 48595108 +58.6% 77064959 aim7.time.voluntary_context_switches
> 1.44 -28.8% 1.02 iostat.cpu.user
> 0.07 ± 6% +0.4 0.44 ± 7% mpstat.cpu.all.iowait%
> 1.44 -0.4 1.02 mpstat.cpu.all.usr%
> 8632 ± 50% +75.6% 15156 ± 34% numa-meminfo.node0.KernelStack
> 6583 ±136% +106.0% 13562 ± 82% numa-meminfo.node0.PageTables
> 63325 ± 11% +14.3% 72352 ± 12% numa-meminfo.node0.SUnreclaim
> 8647 ± 50% +75.3% 15156 ± 34% numa-vmstat.node0.nr_kernel_stack
> 1656 ±136% +104.6% 3389 ± 82% numa-vmstat.node0.nr_page_table_pages
> 15831 ± 11% +14.3% 18087 ± 12% numa-vmstat.node0.nr_slab_unreclaimable
> 93640 ± 3% +41.2% 132211 ± 2% meminfo.AnonHugePages
> 21641 +39.9% 30271 ± 4% meminfo.KernelStack
> 129269 +12.3% 145114 meminfo.SUnreclaim
> 28000 -31.2% 19275 meminfo.max_used_kB
> 1269307 -26.9% 927657 vmstat.io.bo
> 149.75 ± 3% -17.4% 123.75 ± 4% vmstat.procs.r
> 718992 +13.3% 814567 vmstat.system.cs
> 231397 -9.3% 209881 ± 2% vmstat.system.in
> 6.774e+08 +70.0% 1.152e+09 cpuidle.C1.time
> 18203372 +60.4% 29198744 cpuidle.C1.usage
> 2.569e+08 ± 18% +81.8% 4.672e+08 ± 5% cpuidle.C1E.time
> 2691402 ± 13% +98.7% 5346901 ± 3% cpuidle.C1E.usage
> 990350 +95.0% 1931226 ± 2% cpuidle.POLL.time
> 520061 +97.7% 1028004 ± 2% cpuidle.POLL.usage
> 77231 +1.8% 78602 proc-vmstat.nr_active_anon
> 19868 +3.8% 20615 proc-vmstat.nr_dirty
> 381302 +1.0% 384969 proc-vmstat.nr_file_pages
> 4388 -2.7% 4270 proc-vmstat.nr_inactive_anon
> 69865 +4.7% 73155 proc-vmstat.nr_inactive_file
> 21615 +40.0% 30251 ± 4% proc-vmstat.nr_kernel_stack
> 7363 -3.2% 7127 proc-vmstat.nr_mapped
> 12595 ± 3% +5.2% 13255 ± 4% proc-vmstat.nr_shmem
> 19619 +3.2% 20247 proc-vmstat.nr_slab_reclaimable
> 32316 +12.3% 36280 proc-vmstat.nr_slab_unreclaimable
> 77231 +1.8% 78602 proc-vmstat.nr_zone_active_anon
> 4388 -2.7% 4270 proc-vmstat.nr_zone_inactive_anon
> 69865 +4.7% 73155 proc-vmstat.nr_zone_inactive_file
> 19165 +3.4% 19824 proc-vmstat.nr_zone_write_pending
> 9845 ± 27% +68.0% 16539 ± 37% proc-vmstat.numa_hint_faults_local
> 8195135 +2.5% 8402537 proc-vmstat.numa_hit
> 8179414 +2.5% 8386209 proc-vmstat.numa_local
> 15720 +3.9% 16328 proc-vmstat.numa_other
> 8281098 +2.4% 8477157 proc-vmstat.pgalloc_normal
> 342925 ± 2% +39.8% 479279 proc-vmstat.pgfault
> 8132789 +2.0% 8298189 proc-vmstat.pgfree
> 1.327e+08 +9.7% 1.455e+08 proc-vmstat.pgpgout
> 394.00 +151.7% 991.75 ± 5% slabinfo.biovec-max.active_objs
> 394.00 +152.1% 993.25 ± 5% slabinfo.biovec-max.num_objs
> 21028 ± 6% -7.8% 19389 ± 5% slabinfo.filp.active_objs
> 21520 ± 5% -7.6% 19886 ± 5% slabinfo.filp.num_objs
> 4955 +234.9% 16594 ± 2% slabinfo.kmalloc-256.active_objs
> 77.25 +238.5% 261.50 ± 2% slabinfo.kmalloc-256.active_slabs
> 4955 +238.4% 16768 ± 2% slabinfo.kmalloc-256.num_objs
> 77.25 +238.5% 261.50 ± 2% slabinfo.kmalloc-256.num_slabs
> 3368 ± 2% +22.6% 4128 ± 2% slabinfo.kmalloc-2k.active_objs
> 3446 ± 2% +21.9% 4200 ± 2% slabinfo.kmalloc-2k.num_objs
> 11815 ± 8% +29.0% 15239 ± 3% slabinfo.kmalloc-512.active_objs
> 11824 ± 9% +29.0% 15249 ± 3% slabinfo.kmalloc-512.num_objs
> 6849 ± 5% +9.0% 7463 ± 2% slabinfo.kmalloc-96.active_objs
> 6906 ± 5% +8.9% 7521 ± 2% slabinfo.kmalloc-96.num_objs
> 765.00 ± 4% +25.0% 956.25 ± 5% slabinfo.mnt_cache.active_objs
> 765.00 ± 4% +25.0% 956.25 ± 5% slabinfo.mnt_cache.num_objs
> 13866 +12.5% 15594 slabinfo.proc_inode_cache.active_objs
> 13905 +12.2% 15598 slabinfo.proc_inode_cache.num_objs
> 1954 +24.4% 2431 ± 3% slabinfo.sighand_cache.active_objs
> 1954 +24.4% 2431 ± 3% slabinfo.sighand_cache.num_objs
> 2596 +13.7% 2951 ± 2% slabinfo.signal_cache.active_objs
> 2596 +13.7% 2951 ± 2% slabinfo.signal_cache.num_objs
> 4368 ± 6% -11.7% 3856 ± 2% slabinfo.skbuff_head_cache.num_objs
> 822.25 ± 11% +21.4% 998.50 ± 3% slabinfo.task_group.active_objs
> 822.25 ± 11% +21.4% 998.50 ± 3% slabinfo.task_group.num_objs
> 1612 +32.1% 2129 ± 3% slabinfo.task_struct.active_objs
> 540.50 +31.6% 711.25 ± 3% slabinfo.task_struct.active_slabs
> 1622 +31.6% 2134 ± 3% slabinfo.task_struct.num_objs
> 540.50 +31.6% 711.25 ± 3% slabinfo.task_struct.num_slabs
> 6905 +20.0% 8289 slabinfo.vmap_area.active_objs
> 6933 +20.0% 8322 slabinfo.vmap_area.num_objs
> 646.25 ± 7% +195.2% 1907 slabinfo.xfs_buf.active_objs
> 646.25 ± 7% +195.2% 1907 slabinfo.xfs_buf.num_objs
> 1.78 ± 2% -0.3 1.51 ± 2% perf-stat.i.branch-miss-rate%
> 64079048 +3.9% 66551536 perf-stat.i.branch-misses
> 17.99 -4.2 13.84 perf-stat.i.cache-miss-rate%
> 66630506 -16.5% 55647472 perf-stat.i.cache-misses
> 3.899e+08 +9.4% 4.265e+08 perf-stat.i.cache-references
> 731029 +13.1% 826756 perf-stat.i.context-switches
> 3.79 -1.7% 3.73 perf-stat.i.cpi
> 113320 +8.3% 122693 perf-stat.i.cpu-migrations
> 1498 +20.4% 1803 perf-stat.i.cycles-between-cache-misses
> 50450547 ± 4% -8.0% 46402743 ± 5% perf-stat.i.dTLB-load-misses
> 0.20 ± 3% -0.0 0.16 ± 2% perf-stat.i.dTLB-store-miss-rate%
> 4587096 ± 3% -22.9% 3538422 ± 2% perf-stat.i.dTLB-store-misses
> 2.297e+09 -5.6% 2.168e+09 perf-stat.i.dTLB-stores
> 86.68 -2.4 84.25 perf-stat.i.iTLB-load-miss-rate%
> 19484791 -6.1% 18297125 perf-stat.i.iTLB-load-misses
> 3060023 +14.4% 3501587 ± 4% perf-stat.i.iTLB-loads
> 0.29 -1.5% 0.28 perf-stat.i.ipc
> 0.21 ± 5% -20.5% 0.17 ± 6% perf-stat.i.metric.K/sec
> 3120 -5.8% 2938 perf-stat.i.minor-faults
> 42.61 +1.2 43.83 perf-stat.i.node-load-miss-rate%
> 24887573 -10.1% 22384693 perf-stat.i.node-load-misses
> 31814752 -12.5% 27830232 perf-stat.i.node-loads
> 36.62 +0.8 37.45 perf-stat.i.node-store-miss-rate%
> 22617213 -20.3% 18015607 perf-stat.i.node-store-misses
> 35707079 -19.9% 28604167 perf-stat.i.node-stores
> 3120 -5.8% 2938 perf-stat.i.page-faults
> 13.96 +9.3% 15.25 perf-stat.overall.MPKI
> 0.99 +0.0 1.03 perf-stat.overall.branch-miss-rate%
> 17.09 -4.0 13.05 perf-stat.overall.cache-miss-rate%
> 1555 +19.4% 1856 perf-stat.overall.cycles-between-cache-misses
> 0.69 ± 4% -0.1 0.64 ± 5% perf-stat.overall.dTLB-load-miss-rate%
> 0.20 ± 3% -0.0 0.16 ± 2% perf-stat.overall.dTLB-store-miss-rate%
> 86.43 -2.5 83.94 perf-stat.overall.iTLB-load-miss-rate%
> 1433 +6.7% 1528 perf-stat.overall.instructions-per-iTLB-miss
> 43.89 +0.7 44.58 perf-stat.overall.node-load-miss-rate%
> 6.389e+09 +0.8% 6.438e+09 perf-stat.ps.branch-instructions
> 63452800 +4.2% 66110454 perf-stat.ps.branch-misses
> 65977234 -16.2% 55277568 perf-stat.ps.cache-misses
> 3.861e+08 +9.8% 4.237e+08 perf-stat.ps.cache-references
> 723870 +13.5% 821265 perf-stat.ps.context-switches
> 112209 +8.6% 121881 perf-stat.ps.cpu-migrations
> 49955941 ± 4% -7.7% 46094243 ± 5% perf-stat.ps.dTLB-load-misses
> 4542094 ± 3% -22.6% 3514899 ± 2% perf-stat.ps.dTLB-store-misses
> 2.275e+09 -5.3% 2.154e+09 perf-stat.ps.dTLB-stores
> 19294019 -5.8% 18175580 perf-stat.ps.iTLB-load-misses
> 3030049 +14.8% 3478314 ± 4% perf-stat.ps.iTLB-loads
> 3089 -5.5% 2918 perf-stat.ps.minor-faults
> 24643655 -9.8% 22235787 perf-stat.ps.node-load-misses
> 31502901 -12.2% 27645143 perf-stat.ps.node-loads
> 22395376 -20.1% 17895417 perf-stat.ps.node-store-misses
> 35356897 -19.6% 28413366 perf-stat.ps.node-stores
> 3089 -5.5% 2918 perf-stat.ps.page-faults
> 2.85e+12 +50.6% 4.293e+12 perf-stat.total.instructions
> 84193 ± 36% +135.9% 198575 ± 25% sched_debug.cfs_rq:/.MIN_vruntime.avg
> 919053 +98.2% 1821460 ± 2% sched_debug.cfs_rq:/.MIN_vruntime.max
> 256686 ± 16% +114.7% 551162 ± 13% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 19497 +106.7% 40307 sched_debug.cfs_rq:/.exec_clock.avg
> 21240 +101.1% 42717 sched_debug.cfs_rq:/.exec_clock.max
> 19079 +107.3% 39542 sched_debug.cfs_rq:/.exec_clock.min
> 394.52 ± 2% +54.3% 608.79 ± 13% sched_debug.cfs_rq:/.exec_clock.stddev
> 965.25 ± 15% +59.7% 1541 ± 4% sched_debug.cfs_rq:/.load_avg.avg
> 2866 ± 21% +57.1% 4503 ± 10% sched_debug.cfs_rq:/.load_avg.max
> 659.77 ± 15% +68.9% 1114 ± 9% sched_debug.cfs_rq:/.load_avg.stddev
> 84193 ± 36% +135.9% 198575 ± 25% sched_debug.cfs_rq:/.max_vruntime.avg
> 919053 +98.2% 1821462 ± 2% sched_debug.cfs_rq:/.max_vruntime.max
> 256686 ± 16% +114.7% 551163 ± 13% sched_debug.cfs_rq:/.max_vruntime.stddev
> 901765 +97.1% 1777741 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg
> 941794 +95.8% 1844445 ± 2% sched_debug.cfs_rq:/.min_vruntime.max
> 872925 +96.5% 1715658 ± 2% sched_debug.cfs_rq:/.min_vruntime.min
> 18358 ± 44% +146.6% 45281 ± 34% sched_debug.cfs_rq:/.min_vruntime.stddev
> 1.22 ± 8% +159.9% 3.18 ± 20% sched_debug.cfs_rq:/.nr_spread_over.avg
> 6.75 ± 23% +65.4% 11.17 ± 17% sched_debug.cfs_rq:/.nr_spread_over.max
> 1.38 ± 11% +61.4% 2.22 ± 12% sched_debug.cfs_rq:/.nr_spread_over.stddev
> 18357 ± 44% +146.5% 45250 ± 34% sched_debug.cfs_rq:/.spread0.stddev
> 773.25 ± 3% +13.2% 875.02 sched_debug.cfs_rq:/.util_avg.avg
> 285624 -30.3% 199111 ± 4% sched_debug.cpu.avg_idle.avg
> 576507 ± 2% -26.2% 425271 ± 7% sched_debug.cpu.avg_idle.max
> 51029 ± 68% -83.4% 8469 ± 29% sched_debug.cpu.avg_idle.min
> 135149 ± 7% -23.4% 103460 ± 10% sched_debug.cpu.avg_idle.stddev
> 57492 +51.0% 86835 sched_debug.cpu.clock.avg
> 57495 +51.0% 86841 sched_debug.cpu.clock.max
> 57487 +51.0% 86828 sched_debug.cpu.clock.min
> 1.98 ± 4% +74.8% 3.46 ± 10% sched_debug.cpu.clock.stddev
> 57492 +51.0% 86835 sched_debug.cpu.clock_task.avg
> 57495 +51.0% 86841 sched_debug.cpu.clock_task.max
> 57487 +51.0% 86828 sched_debug.cpu.clock_task.min
> 1.98 ± 4% +74.8% 3.46 ± 10% sched_debug.cpu.clock_task.stddev
> 2831 +43.0% 4049 sched_debug.cpu.curr->pid.max
> 417519 +133.3% 974078 sched_debug.cpu.nr_switches.avg
> 427854 +134.0% 1001016 sched_debug.cpu.nr_switches.max
> 409942 +131.8% 950218 sched_debug.cpu.nr_switches.min
> 4770 ± 42% +238.7% 16156 ± 22% sched_debug.cpu.nr_switches.stddev
> 4.70 ± 23% +37.1% 6.44 ± 4% sched_debug.cpu.nr_uninterruptible.avg
> 776.62 ± 19% +86.1% 1445 ± 12% sched_debug.cpu.nr_uninterruptible.max
> -472.62 +98.3% -937.33 sched_debug.cpu.nr_uninterruptible.min
> 289.80 ± 5% +109.7% 607.68 ± 3% sched_debug.cpu.nr_uninterruptible.stddev
> 415893 +133.8% 972526 sched_debug.cpu.sched_count.avg
> 423960 +135.5% 998457 sched_debug.cpu.sched_count.max
> 409113 +132.0% 949068 sched_debug.cpu.sched_count.min
> 4194 ± 50% +274.3% 15698 ± 22% sched_debug.cpu.sched_count.stddev
> 110094 +125.8% 248571 sched_debug.cpu.sched_goidle.avg
> 112236 +125.6% 253163 sched_debug.cpu.sched_goidle.max
> 108281 +125.8% 244506 sched_debug.cpu.sched_goidle.min
> 772.04 ± 16% +175.8% 2129 ± 18% sched_debug.cpu.sched_goidle.stddev
> 297267 +137.2% 705173 sched_debug.cpu.ttwu_count.avg
> 320112 +135.9% 755096 sched_debug.cpu.ttwu_count.max
> 272487 ± 2% +142.2% 660099 sched_debug.cpu.ttwu_count.min
> 10662 ± 10% +110.9% 22491 ± 18% sched_debug.cpu.ttwu_count.stddev
> 12295 ± 2% +834.9% 114954 ± 2% sched_debug.cpu.ttwu_local.avg
> 13726 ± 2% +769.7% 119376 ± 2% sched_debug.cpu.ttwu_local.max
> 10620 ± 4% +942.4% 110710 sched_debug.cpu.ttwu_local.min
> 693.29 ± 14% +228.7% 2278 ± 18% sched_debug.cpu.ttwu_local.stddev
> 57488 +51.0% 86829 sched_debug.cpu_clk
> 56998 +51.5% 86337 sched_debug.ktime
> 57977 +50.6% 87317 sched_debug.sched_clk
> 72.50 ± 37% +55.5% 112.75 ± 26% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> 53.25 ± 2% +84.5% 98.25 ± 22% interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
> 73.50 ± 26% +116.3% 159.00 ± 60% interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
> 204541 +49.0% 304702 ± 2% interrupts.CPU0.LOC:Local_timer_interrupts
> 290268 ± 2% +29.1% 374641 ± 4% interrupts.CPU0.RES:Rescheduling_interrupts
> 204580 +48.9% 304667 ± 2% interrupts.CPU1.LOC:Local_timer_interrupts
> 290342 ± 2% +28.5% 373110 ± 4% interrupts.CPU1.RES:Rescheduling_interrupts
> 204537 +48.9% 304619 ± 2% interrupts.CPU10.LOC:Local_timer_interrupts
> 291605 ± 2% +28.3% 374190 ± 4% interrupts.CPU10.RES:Rescheduling_interrupts
> 204140 +49.2% 304587 ± 2% interrupts.CPU11.LOC:Local_timer_interrupts
> 291988 ± 2% +28.1% 374024 ± 4% interrupts.CPU11.RES:Rescheduling_interrupts
> 204367 +49.0% 304559 ± 2% interrupts.CPU12.LOC:Local_timer_interrupts
> 288632 +28.4% 370669 ± 3% interrupts.CPU12.RES:Rescheduling_interrupts
> 204555 +48.9% 304669 ± 2% interrupts.CPU13.LOC:Local_timer_interrupts
> 289354 ± 2% +28.5% 371782 ± 3% interrupts.CPU13.RES:Rescheduling_interrupts
> 204536 +49.0% 304726 ± 2% interrupts.CPU14.LOC:Local_timer_interrupts
> 4937 ± 25% +41.6% 6991 ± 2% interrupts.CPU14.NMI:Non-maskable_interrupts
> 4937 ± 25% +41.6% 6991 ± 2% interrupts.CPU14.PMI:Performance_monitoring_interrupts
> 287759 +28.2% 368986 ± 3% interrupts.CPU14.RES:Rescheduling_interrupts
> 204527 +49.0% 304707 ± 2% interrupts.CPU15.LOC:Local_timer_interrupts
> 286950 +28.6% 369104 ± 3% interrupts.CPU15.RES:Rescheduling_interrupts
> 204556 +49.0% 304718 ± 2% interrupts.CPU16.LOC:Local_timer_interrupts
> 5444 ± 32% -19.9% 4360 ± 32% interrupts.CPU16.NMI:Non-maskable_interrupts
> 5444 ± 32% -19.9% 4360 ± 32% interrupts.CPU16.PMI:Performance_monitoring_interrupts
> 287623 +28.1% 368583 ± 3% interrupts.CPU16.RES:Rescheduling_interrupts
> 204528 +49.0% 304728 ± 2% interrupts.CPU17.LOC:Local_timer_interrupts
> 287734 +28.2% 368920 ± 3% interrupts.CPU17.RES:Rescheduling_interrupts
> 204465 +49.0% 304707 ± 2% interrupts.CPU18.LOC:Local_timer_interrupts
> 5382 ± 31% -21.8% 4211 ± 27% interrupts.CPU18.NMI:Non-maskable_interrupts
> 5382 ± 31% -21.8% 4211 ± 27% interrupts.CPU18.PMI:Performance_monitoring_interrupts
> 288778 +28.1% 369871 ± 3% interrupts.CPU18.RES:Rescheduling_interrupts
> 204302 +49.2% 304727 ± 2% interrupts.CPU19.LOC:Local_timer_interrupts
> 289794 +27.8% 370247 ± 3% interrupts.CPU19.RES:Rescheduling_interrupts
> 204552 +49.0% 304821 ± 2% interrupts.CPU2.LOC:Local_timer_interrupts
> 291056 ± 2% +28.3% 373290 ± 4% interrupts.CPU2.RES:Rescheduling_interrupts
> 204468 +48.9% 304484 ± 2% interrupts.CPU20.LOC:Local_timer_interrupts
> 290778 +27.6% 371106 ± 3% interrupts.CPU20.RES:Rescheduling_interrupts
> 204546 +48.9% 304619 ± 2% interrupts.CPU21.LOC:Local_timer_interrupts
> 289974 +28.0% 371074 ± 3% interrupts.CPU21.RES:Rescheduling_interrupts
> 204694 +48.8% 304581 ± 2% interrupts.CPU22.LOC:Local_timer_interrupts
> 290312 +28.0% 371734 ± 3% interrupts.CPU22.RES:Rescheduling_interrupts
> 204374 +49.0% 304493 ± 2% interrupts.CPU23.LOC:Local_timer_interrupts
> 290266 +28.5% 372865 ± 3% interrupts.CPU23.RES:Rescheduling_interrupts
> 204518 +49.0% 304653 ± 2% interrupts.CPU24.LOC:Local_timer_interrupts
> 288815 ± 2% +28.7% 371621 ± 4% interrupts.CPU24.RES:Rescheduling_interrupts
> 204414 +49.0% 304640 ± 2% interrupts.CPU25.LOC:Local_timer_interrupts
> 288779 ± 2% +28.2% 370202 ± 4% interrupts.CPU25.RES:Rescheduling_interrupts
> 204536 +49.0% 304730 ± 2% interrupts.CPU26.LOC:Local_timer_interrupts
> 287615 ± 2% +28.6% 369843 ± 4% interrupts.CPU26.RES:Rescheduling_interrupts
> 72.50 ± 37% +55.5% 112.75 ± 26% interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> 204514 +49.0% 304668 ± 2% interrupts.CPU27.LOC:Local_timer_interrupts
> 287014 ± 2% +28.5% 368825 ± 4% interrupts.CPU27.RES:Rescheduling_interrupts
> 53.25 ± 2% +84.5% 98.25 ± 22% interrupts.CPU28.37:PCI-MSI.2621443-edge.eth0-TxRx-2
> 204331 +49.1% 304759 ± 2% interrupts.CPU28.LOC:Local_timer_interrupts
> 287278 ± 2% +28.4% 368876 ± 4% interrupts.CPU28.RES:Rescheduling_interrupts
> 73.50 ± 26% +116.3% 159.00 ± 60% interrupts.CPU29.38:PCI-MSI.2621444-edge.eth0-TxRx-3
> 204427 +49.0% 304651 ± 2% interrupts.CPU29.LOC:Local_timer_interrupts
> 287382 ± 2% +28.9% 370434 ± 4% interrupts.CPU29.RES:Rescheduling_interrupts
> 204538 +49.0% 304765 ± 2% interrupts.CPU3.LOC:Local_timer_interrupts
> 5034 ± 27% -29.4% 3552 interrupts.CPU3.NMI:Non-maskable_interrupts
> 5034 ± 27% -29.4% 3552 interrupts.CPU3.PMI:Performance_monitoring_interrupts
> 288881 ± 2% +28.3% 370740 ± 4% interrupts.CPU3.RES:Rescheduling_interrupts
> 204750 +48.8% 304674 ± 2% interrupts.CPU30.LOC:Local_timer_interrupts
> 287139 ± 2% +28.8% 369825 ± 4% interrupts.CPU30.RES:Rescheduling_interrupts
> 204429 +48.9% 304416 ± 2% interrupts.CPU31.LOC:Local_timer_interrupts
> 288411 ± 2% +28.4% 370199 ± 4% interrupts.CPU31.RES:Rescheduling_interrupts
> 204395 +49.3% 305106 ± 2% interrupts.CPU32.LOC:Local_timer_interrupts
> 289178 ± 2% +28.6% 371830 ± 4% interrupts.CPU32.RES:Rescheduling_interrupts
> 204347 +49.2% 304799 ± 2% interrupts.CPU33.LOC:Local_timer_interrupts
> 289689 ± 2% +28.2% 371280 ± 4% interrupts.CPU33.RES:Rescheduling_interrupts
> 204280 +49.1% 304566 ± 2% interrupts.CPU34.LOC:Local_timer_interrupts
> 6418 ± 24% -44.5% 3564 interrupts.CPU34.NMI:Non-maskable_interrupts
> 6418 ± 24% -44.5% 3564 interrupts.CPU34.PMI:Performance_monitoring_interrupts
> 290382 ± 2% +28.3% 372606 ± 4% interrupts.CPU34.RES:Rescheduling_interrupts
> 204331 +49.0% 304517 ± 2% interrupts.CPU35.LOC:Local_timer_interrupts
> 289736 ± 2% +28.3% 371869 ± 4% interrupts.CPU35.RES:Rescheduling_interrupts
> 204578 +48.8% 304423 ± 2% interrupts.CPU36.LOC:Local_timer_interrupts
> 7298 -39.0% 4450 ± 34% interrupts.CPU36.NMI:Non-maskable_interrupts
> 7298 -39.0% 4450 ± 34% interrupts.CPU36.PMI:Performance_monitoring_interrupts
> 287187 +28.3% 368516 ± 3% interrupts.CPU36.RES:Rescheduling_interrupts
> 204538 +49.0% 304729 ± 2% interrupts.CPU37.LOC:Local_timer_interrupts
> 288676 +28.1% 369792 ± 3% interrupts.CPU37.RES:Rescheduling_interrupts
> 204504 +49.0% 304687 ± 2% interrupts.CPU38.LOC:Local_timer_interrupts
> 6301 ± 24% -29.8% 4424 ± 34% interrupts.CPU38.NMI:Non-maskable_interrupts
> 6301 ± 24% -29.8% 4424 ± 34% interrupts.CPU38.PMI:Performance_monitoring_interrupts
> 287510 +28.1% 368349 ± 3% interrupts.CPU38.RES:Rescheduling_interrupts
> 204566 +48.9% 304668 ± 2% interrupts.CPU39.LOC:Local_timer_interrupts
> 286909 +28.1% 367622 ± 3% interrupts.CPU39.RES:Rescheduling_interrupts
> 204445 +49.0% 304686 ± 2% interrupts.CPU4.LOC:Local_timer_interrupts
> 288608 ± 2% +28.7% 371493 ± 3% interrupts.CPU4.RES:Rescheduling_interrupts
> 204538 +48.9% 304582 ± 2% interrupts.CPU40.LOC:Local_timer_interrupts
> 286219 +27.9% 365983 ± 3% interrupts.CPU40.RES:Rescheduling_interrupts
> 204551 +49.0% 304739 ± 2% interrupts.CPU41.LOC:Local_timer_interrupts
> 5170 ± 29% -18.5% 4215 ± 27% interrupts.CPU41.NMI:Non-maskable_interrupts
> 5170 ± 29% -18.5% 4215 ± 27% interrupts.CPU41.PMI:Performance_monitoring_interrupts
> 286946 +27.8% 366717 ± 3% interrupts.CPU41.RES:Rescheduling_interrupts
> 204492 +48.9% 304442 ± 2% interrupts.CPU42.LOC:Local_timer_interrupts
> 287384 +28.0% 367805 ± 3% interrupts.CPU42.RES:Rescheduling_interrupts
> 204545 +49.0% 304679 ± 2% interrupts.CPU43.LOC:Local_timer_interrupts
> 289100 +27.3% 368059 ± 3% interrupts.CPU43.RES:Rescheduling_interrupts
> 204489 +49.0% 304635 ± 2% interrupts.CPU44.LOC:Local_timer_interrupts
> 6393 ± 24% -32.4% 4319 ± 31% interrupts.CPU44.NMI:Non-maskable_interrupts
> 6393 ± 24% -32.4% 4319 ± 31% interrupts.CPU44.PMI:Performance_monitoring_interrupts
> 288835 +27.8% 369182 ± 3% interrupts.CPU44.RES:Rescheduling_interrupts
> 204473 +49.2% 304983 ± 2% interrupts.CPU45.LOC:Local_timer_interrupts
> 289778 +27.4% 369059 ± 3% interrupts.CPU45.RES:Rescheduling_interrupts
> 204493 +49.0% 304598 ± 2% interrupts.CPU46.LOC:Local_timer_interrupts
> 289485 +27.9% 370172 ± 3% interrupts.CPU46.RES:Rescheduling_interrupts
> 204511 +49.2% 305090 ± 2% interrupts.CPU47.LOC:Local_timer_interrupts
> 286611 +28.8% 369021 ± 3% interrupts.CPU47.RES:Rescheduling_interrupts
> 204441 +49.1% 304794 ± 2% interrupts.CPU5.LOC:Local_timer_interrupts
> 289245 ± 2% +28.4% 371335 ± 4% interrupts.CPU5.RES:Rescheduling_interrupts
> 204246 +49.0% 304414 ± 2% interrupts.CPU6.LOC:Local_timer_interrupts
> 290006 ± 2% +28.3% 372180 ± 4% interrupts.CPU6.RES:Rescheduling_interrupts
> 204445 +49.0% 304719 ± 2% interrupts.CPU7.LOC:Local_timer_interrupts
> 290524 ± 2% +28.2% 372414 ± 4% interrupts.CPU7.RES:Rescheduling_interrupts
> 204178 +49.2% 304663 ± 2% interrupts.CPU8.LOC:Local_timer_interrupts
> 291625 ± 2% +27.8% 372747 ± 4% interrupts.CPU8.RES:Rescheduling_interrupts
> 204502 +48.9% 304407 ± 3% interrupts.CPU9.LOC:Local_timer_interrupts
> 291615 ± 2% +27.9% 373024 ± 4% interrupts.CPU9.RES:Rescheduling_interrupts
> 9814627 +49.0% 14624016 ± 2% interrupts.LOC:Local_timer_interrupts
> 13869791 +28.2% 17785831 ± 3% interrupts.RES:Rescheduling_interrupts
> 25728 +47.6% 37969 softirqs.CPU0.RCU
> 34944 +41.7% 49503 ± 3% softirqs.CPU0.SCHED
> 42375 +38.5% 58673 ± 2% softirqs.CPU0.TIMER
> 25348 +53.8% 38976 ± 5% softirqs.CPU1.RCU
> 30986 +46.2% 45315 ± 3% softirqs.CPU1.SCHED
> 41265 +44.3% 59538 ± 3% softirqs.CPU1.TIMER
> 25149 +48.3% 37288 softirqs.CPU10.RCU
> 29752 +47.1% 43764 ± 2% softirqs.CPU10.SCHED
> 40609 +41.5% 57449 softirqs.CPU10.TIMER
> 25218 +48.3% 37399 softirqs.CPU11.RCU
> 30294 +44.6% 43800 softirqs.CPU11.SCHED
> 41371 +39.2% 57578 softirqs.CPU11.TIMER
> 24427 +48.9% 36361 softirqs.CPU12.RCU
> 30438 +44.0% 43835 softirqs.CPU12.SCHED
> 41045 ± 3% +43.0% 58687 softirqs.CPU12.TIMER
> 24247 +47.0% 35645 softirqs.CPU13.RCU
> 30155 +44.5% 43577 softirqs.CPU13.SCHED
> 41148 ± 2% +40.3% 57720 softirqs.CPU13.TIMER
> 24506 +46.3% 35856 softirqs.CPU14.RCU
> 29991 +44.1% 43229 softirqs.CPU14.SCHED
> 41190 ± 2% +39.7% 57552 softirqs.CPU14.TIMER
> 24412 +46.7% 35800 softirqs.CPU15.RCU
> 30135 +44.1% 43430 softirqs.CPU15.SCHED
> 45081 ± 15% +28.0% 57684 softirqs.CPU15.TIMER
> 25327 +47.0% 37225 softirqs.CPU16.RCU
> 30230 +43.8% 43461 softirqs.CPU16.SCHED
> 41157 +39.6% 57448 softirqs.CPU16.TIMER
> 25136 +47.8% 37142 softirqs.CPU17.RCU
> 29894 +44.5% 43212 softirqs.CPU17.SCHED
> 41961 ± 5% +36.5% 57281 softirqs.CPU17.TIMER
> 25431 +46.1% 37160 softirqs.CPU18.RCU
> 30062 +44.0% 43281 softirqs.CPU18.SCHED
> 40956 ± 2% +42.1% 58217 softirqs.CPU18.TIMER
> 25496 +44.8% 36923 ± 2% softirqs.CPU19.RCU
> 29989 +46.5% 43946 softirqs.CPU19.SCHED
> 41740 ± 2% +67.8% 70057 ± 17% softirqs.CPU19.TIMER
> 25414 +48.3% 37678 softirqs.CPU2.RCU
> 30905 +43.4% 44309 ± 2% softirqs.CPU2.SCHED
> 41083 +40.6% 57768 ± 2% softirqs.CPU2.TIMER
> 25124 +46.8% 36892 softirqs.CPU20.RCU
> 30001 +44.7% 43416 softirqs.CPU20.SCHED
> 41060 ± 7% +39.2% 57151 softirqs.CPU20.TIMER
> 24992 +47.6% 36896 softirqs.CPU21.RCU
> 30109 +44.4% 43467 softirqs.CPU21.SCHED
> 40055 ± 3% +42.5% 57081 ± 2% softirqs.CPU21.TIMER
> 25072 +46.2% 36656 softirqs.CPU22.RCU
> 30296 +43.3% 43425 softirqs.CPU22.SCHED
> 24882 +48.8% 37031 softirqs.CPU23.RCU
> 29900 +43.7% 42971 softirqs.CPU23.SCHED
> 40388 ± 3% +44.8% 58494 ± 3% softirqs.CPU23.TIMER
> 25007 +48.1% 37036 softirqs.CPU24.RCU
> 29755 +45.9% 43399 softirqs.CPU24.SCHED
> 40786 +40.6% 57353 softirqs.CPU24.TIMER
> 25031 +47.3% 36875 softirqs.CPU25.RCU
> 29841 ± 2% +45.3% 43355 softirqs.CPU25.SCHED
> 40971 +39.8% 57266 softirqs.CPU25.TIMER
> 25145 +47.6% 37114 softirqs.CPU26.RCU
> 29462 ± 2% +48.5% 43739 ± 2% softirqs.CPU26.SCHED
> 40553 +41.8% 57521 softirqs.CPU26.TIMER
> 25184 +47.6% 37169 softirqs.CPU27.RCU
> 29867 +45.5% 43450 ± 3% softirqs.CPU27.SCHED
> 41118 +39.3% 57261 softirqs.CPU27.TIMER
> 25452 +47.1% 37440 softirqs.CPU28.RCU
> 29714 +46.2% 43440 ± 2% softirqs.CPU28.SCHED
> 41209 ± 2% +37.8% 56783 ± 2% softirqs.CPU28.TIMER
> 25334 +48.8% 37696 ± 2% softirqs.CPU29.RCU
> 29894 +46.3% 43749 softirqs.CPU29.SCHED
> 41413 ± 2% +43.3% 59346 ± 5% softirqs.CPU29.TIMER
> 25514 +47.0% 37495 softirqs.CPU3.RCU
> 30016 +46.0% 43831 ± 2% softirqs.CPU3.SCHED
> 41371 ± 2% +39.3% 57611 softirqs.CPU3.TIMER
> 25206 +47.4% 37149 softirqs.CPU30.RCU
> 30167 +44.0% 43451 ± 2% softirqs.CPU30.SCHED
> 41166 ± 2% +58.1% 65092 ± 19% softirqs.CPU30.TIMER
> 25211 +46.8% 37016 softirqs.CPU31.RCU
> 30004 ± 2% +45.2% 43569 softirqs.CPU31.SCHED
> 41541 ± 2% +36.9% 56874 softirqs.CPU31.TIMER
> 24432 +46.9% 35885 softirqs.CPU32.RCU
> 30206 +43.4% 43322 ± 2% softirqs.CPU32.SCHED
> 41069 +40.7% 57781 ± 2% softirqs.CPU32.TIMER
> 24303 +48.0% 35973 softirqs.CPU33.RCU
> 30238 +43.8% 43489 softirqs.CPU33.SCHED
> 41161 +38.2% 56872 softirqs.CPU33.TIMER
> 24308 +47.6% 35876 softirqs.CPU34.RCU
> 29803 +46.5% 43647 ± 2% softirqs.CPU34.SCHED
> 40809 +39.2% 56817 softirqs.CPU34.TIMER
> 24291 +47.7% 35877 softirqs.CPU35.RCU
> 30092 +44.2% 43386 softirqs.CPU35.SCHED
> 40607 +41.5% 57480 ± 2% softirqs.CPU35.TIMER
> 25605 +49.5% 38286 softirqs.CPU36.RCU
> 29900 +44.6% 43238 softirqs.CPU36.SCHED
> 45207 ± 15% +27.4% 57602 softirqs.CPU36.TIMER
> 25555 ± 2% +49.0% 38070 softirqs.CPU37.RCU
> 29926 +45.2% 43460 softirqs.CPU37.SCHED
> 42619 ± 8% +36.7% 58242 softirqs.CPU37.TIMER
> 27683 ± 9% +50.2% 41577 ± 4% softirqs.CPU38.RCU
> 29700 +45.6% 43230 softirqs.CPU38.SCHED
> 41779 ± 4% +43.5% 59948 softirqs.CPU38.TIMER
> 26699 ± 7% +42.1% 37951 softirqs.CPU39.RCU
> 30061 +43.4% 43102 softirqs.CPU39.SCHED
> 42312 ± 5% +36.1% 57581 ± 2% softirqs.CPU39.TIMER
> 25219 +47.7% 37247 softirqs.CPU4.RCU
> 30363 ± 3% +45.9% 44295 softirqs.CPU4.SCHED
> 41108 ± 2% +39.6% 57371 ± 2% softirqs.CPU4.TIMER
> 25413 +50.6% 38268 softirqs.CPU40.RCU
> 29910 +46.0% 43665 softirqs.CPU40.SCHED
> 39987 ± 2% +45.9% 58322 softirqs.CPU40.TIMER
> 25569 +49.5% 38238 softirqs.CPU41.RCU
> 29867 +44.8% 43247 softirqs.CPU41.SCHED
> 25492 +49.8% 38185 softirqs.CPU42.RCU
> 29871 +45.4% 43440 softirqs.CPU42.SCHED
> 40076 +44.8% 58020 softirqs.CPU42.TIMER
> 25281 +48.9% 37640 softirqs.CPU43.RCU
> 30025 +43.9% 43220 softirqs.CPU43.SCHED
> 40222 ± 3% +44.7% 58195 softirqs.CPU43.TIMER
> 25183 +49.3% 37611 softirqs.CPU44.RCU
> 30123 +43.6% 43249 softirqs.CPU44.SCHED
> 41090 ± 6% +42.5% 58537 softirqs.CPU44.TIMER
> 25314 +49.6% 37863 softirqs.CPU45.RCU
> 29998 +43.2% 42957 softirqs.CPU45.SCHED
> 39705 ± 3% +43.6% 57033 softirqs.CPU45.TIMER
> 25500 +46.2% 37274 softirqs.CPU46.RCU
> 29921 +44.7% 43286 softirqs.CPU46.SCHED
> 41088 ± 5% +39.9% 57463 ± 3% softirqs.CPU46.TIMER
> 25564 +48.4% 37931 softirqs.CPU47.RCU
> 30178 +42.4% 42980 softirqs.CPU47.SCHED
> 40738 ± 4% +42.2% 57939 ± 3% softirqs.CPU47.TIMER
> 26467 ± 8% +40.8% 37278 softirqs.CPU5.RCU
> 30334 ± 2% +44.1% 43714 softirqs.CPU5.SCHED
> 42268 ± 4% +35.8% 57394 softirqs.CPU5.TIMER
> 25343 +47.0% 37245 softirqs.CPU6.RCU
> 30100 +45.0% 43643 ± 2% softirqs.CPU6.SCHED
> 41270 +43.6% 59249 ± 5% softirqs.CPU6.TIMER
> 25331 +47.7% 37418 softirqs.CPU7.RCU
> 30210 +44.7% 43727 ± 2% softirqs.CPU7.SCHED
> 41028 +41.9% 58206 ± 2% softirqs.CPU7.TIMER
> 25483 +47.2% 37514 softirqs.CPU8.RCU
> 30324 ± 2% +43.4% 43483 ± 2% softirqs.CPU8.SCHED
> 41027 +40.4% 57591 ± 2% softirqs.CPU8.TIMER
> 25546 +47.5% 37687 softirqs.CPU9.RCU
> 30218 +44.8% 43771 ± 2% softirqs.CPU9.SCHED
> 41193 +40.1% 57716 ± 2% softirqs.CPU9.TIMER
> 1212606 +47.7% 1790808 softirqs.RCU
> 1448198 +44.7% 2095500 softirqs.SCHED
> 1990064 +40.7% 2800202 softirqs.TIMER
> 8.56 ± 3% -4.3 4.25 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
> 8.50 ± 3% -4.3 4.20 perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
> 8.47 ± 3% -4.3 4.17 perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
> 78.55 -4.1 74.44 perf-profile.calltrace.cycles-pp.write
> 8.74 ± 3% -4.0 4.70 perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> 77.69 -3.9 73.75 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
> 77.64 -3.9 73.71 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 77.13 -3.8 73.36 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 77.08 -3.8 73.31 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 76.61 -3.7 72.96 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 76.43 -3.6 72.80 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
> 69.22 -2.6 66.62 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
> 4.70 ± 2% -2.2 2.50 perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> 4.63 ± 2% -2.2 2.45 perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> 4.60 ± 2% -2.2 2.43 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
> 4.32 ± 2% -2.1 2.22 perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages
> 3.72 ± 4% -2.1 1.63 perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> 3.71 ± 4% -2.1 1.62 perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> 2.63 ± 4% -2.0 0.63 ± 3% perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
> 7.72 -2.0 5.74 ± 2% perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> 3.08 ± 5% -1.9 1.13 perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
> 2.49 ± 5% -1.9 0.55 ± 3% perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio
> 2.48 ± 5% -1.9 0.55 ± 3% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request
> 2.41 ± 6% -1.8 0.61 ± 2% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
> 4.90 -1.5 3.40 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 4.78 -1.5 3.30 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
> 2.35 -0.6 1.77 ± 2% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> 2.36 -0.6 1.79 perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
> 3.40 -0.5 2.87 ± 6% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
> 1.83 -0.5 1.33 ± 2% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
> 3.26 -0.5 2.75 ± 6% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
> 2.85 -0.4 2.40 perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
> 2.79 -0.4 2.34 perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.process_one_work.worker_thread
> 2.83 -0.4 2.38 perf-profile.calltrace.cycles-pp.swake_up_locked.complete.process_one_work.worker_thread.kthread
> 1.48 -0.4 1.05 ± 2% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
> 0.61 ± 2% -0.4 0.26 ±100% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
> 1.89 -0.3 1.54 perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 1.71 -0.3 1.38 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
> 1.65 -0.3 1.34 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
> 0.69 -0.3 0.38 ± 57% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
> 1.59 -0.3 1.29 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work
> 0.86 -0.3 0.58 ± 2% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
> 1.52 -0.3 1.26 ± 3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
> 1.75 -0.2 1.50 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.process_one_work
> 1.74 -0.2 1.49 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked
> 1.74 -0.2 1.50 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete
> 2.15 -0.2 1.94 perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
> 0.79 ± 3% -0.2 0.60 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
> 1.81 -0.2 1.62 perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
> 1.87 -0.2 1.68 perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
> 0.70 ± 4% -0.2 0.54 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit
> 0.81 -0.1 0.68 perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
> 0.65 ± 2% -0.1 0.55 ± 2% perf-profile.calltrace.cycles-pp.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn
> 0.62 ± 2% -0.1 0.52 ± 3% perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn
> 1.18 -0.0 1.16 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 1.01 -0.0 0.99 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
> 1.00 -0.0 0.98 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
> 0.96 -0.0 0.93 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
> 1.01 -0.0 0.99 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
> 0.59 ± 3% +0.1 0.68 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 0.58 ± 3% +0.1 0.67 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
> 0.53 +0.1 0.63 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work
> 0.91 ± 3% +0.1 1.02 ± 2% perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread
> 0.54 +0.1 0.65 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread
> 0.57 +0.1 0.69 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread.kthread
> 1.83 +0.2 1.99 perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork
> 0.00 +0.6 0.57 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
> 0.00 +0.6 0.57 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
> 0.00 +0.6 0.57 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
> 0.00 +0.7 0.70 ± 2% perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend
> 0.00 +0.8 0.75 ± 4% perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
> 0.00 +0.8 0.80 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend
> 0.00 +0.8 0.80 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends
> 0.00 +0.8 0.80 ± 30% perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
> 0.00 +0.8 0.82 perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend
> 0.00 +0.8 0.85 perf-profile.calltrace.cycles-pp.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io
> 0.00 +0.9 0.89 ± 2% perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
> 0.00 +0.9 0.92 ± 21% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
> 0.00 +1.0 0.97 ± 20% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
> 35.68 +1.0 36.68 perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
> 0.00 +1.0 1.04 perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
> 0.00 +1.0 1.05 perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
> 9.35 +1.2 10.54 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 9.61 +1.2 10.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
> 9.63 +1.2 10.87 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 12.24 +1.3 13.58 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 12.25 +1.3 13.60 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> 12.25 +1.3 13.60 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> 12.49 +1.4 13.89 perf-profile.calltrace.cycles-pp.secondary_startup_64
> 14.38 +1.4 15.81 ± 2% perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> 13.41 +1.4 14.86 ± 2% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 13.37 +1.5 14.82 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn
> 13.34 +1.5 14.80 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn
> 58.91 +1.6 60.56 perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> 0.00 +2.2 2.24 ± 13% perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
> 5.63 +2.5 8.15 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
> 5.65 +2.5 8.19 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 5.69 +2.5 8.23 perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> 7.04 +2.8 9.80 ± 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
> 7.54 +2.9 10.45 ± 2% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
> 7.55 +2.9 10.46 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork
> 7.55 +2.9 10.46 ± 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
> 0.00 +3.3 3.30 ± 9% perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
> 0.00 +3.3 3.32 ± 8% perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
> 8.56 ± 3% -4.3 4.25 perf-profile.children.cycles-pp.__filemap_fdatawrite_range
> 8.47 ± 3% -4.3 4.18 perf-profile.children.cycles-pp.xfs_vm_writepages
> 8.50 ± 3% -4.3 4.20 perf-profile.children.cycles-pp.do_writepages
> 78.64 -4.1 74.51 perf-profile.children.cycles-pp.write
> 78.55 -4.1 74.44 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 78.50 -4.1 74.41 perf-profile.children.cycles-pp.do_syscall_64
> 8.75 ± 3% -4.1 4.70 perf-profile.children.cycles-pp.file_write_and_wait_range
> 77.14 -3.8 73.37 perf-profile.children.cycles-pp.ksys_write
> 77.09 -3.8 73.33 perf-profile.children.cycles-pp.vfs_write
> 76.62 -3.7 72.97 perf-profile.children.cycles-pp.new_sync_write
> 76.44 -3.6 72.81 perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
> 69.23 -2.6 66.62 perf-profile.children.cycles-pp.xfs_file_fsync
> 4.70 ± 2% -2.2 2.50 perf-profile.children.cycles-pp.iomap_submit_ioend
> 4.89 ± 2% -2.1 2.75 perf-profile.children.cycles-pp.submit_bio
> 4.85 ± 2% -2.1 2.71 perf-profile.children.cycles-pp.generic_make_request
> 3.73 ± 4% -2.1 1.63 perf-profile.children.cycles-pp.iomap_writepages
> 3.72 ± 4% -2.1 1.63 perf-profile.children.cycles-pp.write_cache_pages
> 4.43 ± 2% -2.1 2.38 perf-profile.children.cycles-pp.brd_make_request
> 7.73 -2.0 5.75 ± 2% perf-profile.children.cycles-pp.__flush_work
> 3.08 ± 4% -1.9 1.14 perf-profile.children.cycles-pp.iomap_writepage_map
> 6.00 -1.8 4.20 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 2.50 ± 4% -1.8 0.70 ± 3% perf-profile.children.cycles-pp.end_page_writeback
> 2.50 ± 5% -1.8 0.70 ± 4% perf-profile.children.cycles-pp.test_clear_page_writeback
> 2.42 ± 6% -1.8 0.61 ± 2% perf-profile.children.cycles-pp.__test_set_page_writeback
> 2.08 ± 7% -1.6 0.44 ± 4% perf-profile.children.cycles-pp.sb_mark_inode_writeback
> 2.08 ± 6% -1.6 0.44 ± 5% perf-profile.children.cycles-pp.sb_clear_inode_writeback
> 2.70 ± 4% -1.0 1.67 ± 2% perf-profile.children.cycles-pp.iomap_finish_ioend
> 2.36 -0.6 1.79 perf-profile.children.cycles-pp.iomap_file_buffered_write
> 2.35 -0.6 1.78 perf-profile.children.cycles-pp.iomap_apply
> 1.84 -0.5 1.33 ± 2% perf-profile.children.cycles-pp.iomap_write_actor
> 2.41 ± 4% -0.5 1.94 ± 2% perf-profile.children.cycles-pp.reschedule_interrupt
> 3.44 ± 2% -0.5 2.98 perf-profile.children.cycles-pp.sched_ttwu_pending
> 2.87 -0.5 2.42 perf-profile.children.cycles-pp.complete
> 2.83 -0.4 2.38 perf-profile.children.cycles-pp.swake_up_locked
> 2.28 ± 4% -0.4 1.84 ± 2% perf-profile.children.cycles-pp.scheduler_ipi
> 1.56 -0.4 1.17 ± 2% perf-profile.children.cycles-pp.brd_do_bvec
> 1.89 -0.4 1.54 perf-profile.children.cycles-pp.wait_for_completion
> 1.71 -0.3 1.38 perf-profile.children.cycles-pp.schedule_timeout
> 0.88 -0.2 0.68 ± 2% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
> 0.69 -0.2 0.51 ± 2% perf-profile.children.cycles-pp.iomap_write_begin
> 0.67 ± 2% -0.2 0.49 perf-profile.children.cycles-pp.lseek64
> 1.33 ± 5% -0.2 1.17 ± 3% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.51 -0.2 0.36 perf-profile.children.cycles-pp.grab_cache_page_write_begin
> 0.47 -0.1 0.33 perf-profile.children.cycles-pp.pagecache_get_page
> 0.51 ± 3% -0.1 0.38 ± 2% perf-profile.children.cycles-pp.iomap_write_end
> 0.57 ± 2% -0.1 0.46 perf-profile.children.cycles-pp.blkdev_issue_flush
> 0.38 -0.1 0.26 ± 5% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
> 0.34 -0.1 0.23 ± 6% perf-profile.children.cycles-pp.copyin
> 0.33 -0.1 0.22 ± 6% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
> 0.65 ± 2% -0.1 0.56 ± 2% perf-profile.children.cycles-pp.rwsem_optimistic_spin
> 0.37 -0.1 0.28 ± 2% perf-profile.children.cycles-pp.bio_alloc_bioset
> 0.31 ± 4% -0.1 0.22 perf-profile.children.cycles-pp.syscall_return_via_sysret
> 0.56 ± 3% -0.1 0.48 ± 3% perf-profile.children.cycles-pp.osq_lock
> 0.26 ± 3% -0.1 0.18 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64
> 0.28 ± 2% -0.1 0.21 ± 5% perf-profile.children.cycles-pp.mempool_alloc
> 0.26 ± 6% -0.1 0.18 ± 8% perf-profile.children.cycles-pp.xlog_state_get_iclog_space
> 0.29 -0.1 0.22 ± 3% perf-profile.children.cycles-pp.find_get_entry
> 0.34 ± 2% -0.1 0.27 ± 4% perf-profile.children.cycles-pp.generic_make_request_checks
> 0.30 ± 5% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.iomap_set_page_dirty
> 0.32 ± 2% -0.1 0.26 ± 5% perf-profile.children.cycles-pp.brd_lookup_page
> 0.18 ± 2% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.iomap_set_range_uptodate
> 0.15 ± 2% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.__mod_lruvec_state
> 0.15 ± 5% -0.1 0.10 ± 9% perf-profile.children.cycles-pp.wake_up_q
> 0.21 ± 5% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.xfs_ilock
> 0.13 ± 3% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.rwsem_wake
> 0.20 ± 2% -0.0 0.15 ± 5% perf-profile.children.cycles-pp.ksys_lseek
> 0.20 ± 3% -0.0 0.15 ± 3% perf-profile.children.cycles-pp.xas_load
> 0.20 ± 5% -0.0 0.16 ± 2% perf-profile.children.cycles-pp.__set_page_dirty
> 0.19 ± 2% -0.0 0.15 ± 2% perf-profile.children.cycles-pp.xfs_iunlock
> 0.40 -0.0 0.36 ± 4% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
> 0.17 ± 2% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.clear_page_dirty_for_io
> 0.20 ± 3% -0.0 0.16 ± 4% perf-profile.children.cycles-pp.submit_bio_wait
> 0.18 ± 4% -0.0 0.15 ± 7% perf-profile.children.cycles-pp.security_file_permission
> 0.10 -0.0 0.06 ± 6% perf-profile.children.cycles-pp.evict
> 0.10 -0.0 0.06 ± 6% perf-profile.children.cycles-pp.truncate_inode_pages_range
> 0.06 ± 6% -0.0 0.03 ±100% perf-profile.children.cycles-pp.vfs_unlink
> 0.08 ± 6% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.add_to_page_cache_lru
> 0.26 ± 2% -0.0 0.23 ± 3% perf-profile.children.cycles-pp.reweight_entity
> 0.12 ± 6% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.bio_endio
> 0.10 ± 7% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.do_sys_open
> 0.10 ± 7% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.do_sys_openat2
> 0.17 ± 2% -0.0 0.14 perf-profile.children.cycles-pp.__pagevec_release
> 0.13 ± 5% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.down_write
> 0.10 ± 5% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.do_filp_open
> 0.15 ± 4% -0.0 0.12 ± 5% perf-profile.children.cycles-pp.tag_pages_for_writeback
> 0.11 ± 9% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.bio_associate_blkg
> 0.10 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.open64
> 0.09 ± 4% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.path_openat
> 0.16 ± 8% -0.0 0.13 ± 3% perf-profile.children.cycles-pp.pagevec_lookup_range_tag
> 0.12 ± 8% -0.0 0.10 ± 5% perf-profile.children.cycles-pp.account_page_dirtied
> 0.10 ± 4% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.__fdget_pos
> 0.14 ± 3% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.memset_erms
> 0.11 ± 3% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.switch_fpu_return
> 0.23 -0.0 0.21 ± 3% perf-profile.children.cycles-pp.__radix_tree_lookup
> 0.18 ± 2% -0.0 0.16 ± 9% perf-profile.children.cycles-pp.insert_work
> 0.12 ± 4% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.common_file_perm
> 0.09 ± 4% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.__fget_light
> 0.09 ± 4% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.bvec_alloc
> 0.15 ± 7% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.find_get_pages_range_tag
> 0.08 ± 8% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.iov_iter_fault_in_readable
> 0.09 ± 5% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.blk_throtl_bio
> 0.24 ± 4% -0.0 0.22 ± 3% perf-profile.children.cycles-pp._cond_resched
> 0.13 ± 3% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.release_pages
> 0.10 ± 4% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
> 0.08 ± 10% -0.0 0.06 perf-profile.children.cycles-pp.unlink
> 0.08 ± 10% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.__sb_start_write
> 0.08 ± 5% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.generic_write_checks
> 0.08 ± 8% -0.0 0.06 perf-profile.children.cycles-pp.do_unlinkat
> 0.07 ± 10% -0.0 0.05 perf-profile.children.cycles-pp.xfs_break_layouts
> 0.18 ± 2% -0.0 0.16 ± 2% perf-profile.children.cycles-pp.__list_add_valid
> 0.12 ± 3% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.__might_sleep
> 0.14 ± 3% -0.0 0.12 ± 7% perf-profile.children.cycles-pp.xas_find_marked
> 0.07 ± 10% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.bio_associate_blkg_from_css
> 0.08 ± 5% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.bio_free
> 0.08 ± 6% -0.0 0.06 perf-profile.children.cycles-pp.xfs_file_llseek
> 0.08 ± 5% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.xfs_file_write_iter
> 0.16 ± 2% +0.0 0.17 ± 2% perf-profile.children.cycles-pp.__switch_to_asm
> 0.05 +0.0 0.07 ± 7% perf-profile.children.cycles-pp.cpumask_next_wrap
> 0.10 ± 4% +0.0 0.12 ± 3% perf-profile.children.cycles-pp._find_next_bit
> 0.07 ± 7% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.xlog_state_done_syncing
> 0.16 ± 2% +0.0 0.19 ± 2% perf-profile.children.cycles-pp.check_preempt_curr
> 0.19 ± 4% +0.0 0.22 perf-profile.children.cycles-pp.ttwu_do_wakeup
> 0.56 ± 3% +0.0 0.60 perf-profile.children.cycles-pp.dequeue_entity
> 0.20 ± 6% +0.0 0.23 ± 4% perf-profile.children.cycles-pp.xfs_trans_committed_bulk
> 0.07 ± 7% +0.0 0.10 ± 7% perf-profile.children.cycles-pp.check_preempt_wakeup
> 0.82 ± 2% +0.0 0.86 perf-profile.children.cycles-pp.update_load_avg
> 0.31 ± 3% +0.0 0.35 perf-profile.children.cycles-pp.__unwind_start
> 0.08 ± 8% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.destroy_inode
> 0.08 ± 10% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.xfs_fs_destroy_inode
> 0.08 ± 10% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.xfs_inactive
> 0.08 ± 15% +0.0 0.12 ± 5% perf-profile.children.cycles-pp.xfs_bmap_validate_extent
> 3.96 +0.0 4.01 perf-profile.children.cycles-pp.stack_trace_save_tsk
> 0.28 +0.0 0.33 perf-profile.children.cycles-pp.set_next_entity
> 0.00 +0.1 0.05 perf-profile.children.cycles-pp.xfs_trans_roll
> 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.xfs_btree_delrec
> 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.xfs_btree_insert
> 0.16 ± 4% +0.1 0.21 ± 5% perf-profile.children.cycles-pp.memcpy_erms
> 0.04 ± 57% +0.1 0.10 ± 4% perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
> 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.xfs_buf_item_format_segment
> 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.xfs_btree_delete
> 0.00 +0.1 0.06 perf-profile.children.cycles-pp.xfs_btree_update
> 0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.xfs_cil_prepare_item
> 0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.xfs_bmbt_update
> 0.00 +0.1 0.07 ± 10% perf-profile.children.cycles-pp.xfs_buf_item_format
> 0.56 +0.1 0.64 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair
> 0.03 ±100% +0.1 0.10 ± 4% perf-profile.children.cycles-pp.xfs_inactive_truncate
> 0.08 ± 5% +0.1 0.16 ± 11% perf-profile.children.cycles-pp.xfs_iext_lookup_extent
> 0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.xfs_defer_finish
> 0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents
> 0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.xfs_verify_agbno
> 0.00 +0.1 0.09 ± 9% perf-profile.children.cycles-pp.xfs_defer_finish_noroll
> 0.60 ± 3% +0.1 0.70 perf-profile.children.cycles-pp.schedule_idle
> 0.39 ± 2% +0.1 0.49 perf-profile.children.cycles-pp.xlog_state_release_iclog
> 0.00 +0.1 0.10 ± 9% perf-profile.children.cycles-pp.xfs_buf_find
> 0.93 ± 3% +0.1 1.03 ± 2% perf-profile.children.cycles-pp.xlog_write
> 2.58 +0.1 2.69 perf-profile.children.cycles-pp.schedule
> 1.52 +0.1 1.63 perf-profile.children.cycles-pp.__wake_up_common_lock
> 0.00 +0.1 0.11 ± 7% perf-profile.children.cycles-pp.xfs_buf_get_map
> 4.62 +0.1 4.73 perf-profile.children.cycles-pp.__account_scheduler_latency
> 0.00 +0.1 0.12 ± 10% perf-profile.children.cycles-pp.xfs_buf_read_map
> 1.83 +0.2 1.99 perf-profile.children.cycles-pp.xlog_cil_push_work
> 5.94 +0.2 6.12 perf-profile.children.cycles-pp.ttwu_do_activate
> 0.00 +0.2 0.17 ± 6% perf-profile.children.cycles-pp.xfs_btree_read_buf_block
> 5.93 +0.2 6.11 perf-profile.children.cycles-pp.activate_task
> 5.88 +0.2 6.06 perf-profile.children.cycles-pp.enqueue_task_fair
> 0.00 +0.2 0.18 ± 5% perf-profile.children.cycles-pp.xfs_trans_read_buf_map
> 5.41 +0.2 5.59 perf-profile.children.cycles-pp.enqueue_entity
> 3.14 +0.2 3.33 perf-profile.children.cycles-pp.__sched_text_start
> 0.00 +0.2 0.21 ± 6% perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
> 0.00 +0.2 0.23 ± 57% perf-profile.children.cycles-pp.xlog_grant_head_wait
> 0.00 +0.2 0.24 ± 6% perf-profile.children.cycles-pp.io_schedule
> 0.11 ± 3% +0.3 0.38 ± 4% perf-profile.children.cycles-pp.__filemap_fdatawait_range
> 0.00 +0.3 0.27 ± 8% perf-profile.children.cycles-pp.wait_on_page_bit
> 0.00 +0.3 0.34 ± 5% perf-profile.children.cycles-pp.xfs_btree_lookup
> 0.08 ± 14% +0.4 0.43 perf-profile.children.cycles-pp.__queue_work
> 0.08 ± 8% +0.4 0.44 perf-profile.children.cycles-pp.queue_work_on
> 0.00 +0.4 0.41 perf-profile.children.cycles-pp.xfs_end_bio
> 3.51 +0.4 3.94 ± 9% perf-profile.children.cycles-pp.__xfs_trans_commit
> 0.20 ± 9% +0.5 0.72 ± 54% perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
> 0.06 ± 11% +0.5 0.58 ± 67% perf-profile.children.cycles-pp.xfs_log_space_wake
> 0.45 ± 5% +0.6 1.04 ± 36% perf-profile.children.cycles-pp.xfs_log_reserve
> 0.65 ± 3% +0.6 1.25 ± 29% perf-profile.children.cycles-pp.xfs_trans_alloc
> 0.48 ± 5% +0.6 1.09 ± 34% perf-profile.children.cycles-pp.xfs_trans_reserve
> 0.04 ± 57% +0.6 0.65 ± 60% perf-profile.children.cycles-pp.xlog_grant_head_check
> 4.83 +0.7 5.52 perf-profile.children.cycles-pp.try_to_wake_up
> 0.00 +0.7 0.70 ± 2% perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
> 0.00 +0.8 0.75 ± 4% perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
> 0.04 ± 57% +0.8 0.80 perf-profile.children.cycles-pp.autoremove_wake_function
> 0.05 ± 8% +0.8 0.85 perf-profile.children.cycles-pp.wake_up_page_bit
> 1.50 +0.9 2.36 perf-profile.children.cycles-pp.__wake_up_common
> 0.00 +0.9 0.89 ± 2% perf-profile.children.cycles-pp.xfs_bmapi_write
> 24.29 +0.9 25.18 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.07 ± 5% +1.0 1.05 perf-profile.children.cycles-pp.iomap_finish_ioends
> 35.68 +1.0 36.69 perf-profile.children.cycles-pp.xlog_cil_force_lsn
> 9.53 +1.2 10.76 perf-profile.children.cycles-pp.intel_idle
> 9.81 +1.3 11.10 perf-profile.children.cycles-pp.cpuidle_enter_state
> 9.82 +1.3 11.11 perf-profile.children.cycles-pp.cpuidle_enter
> 12.25 +1.3 13.60 perf-profile.children.cycles-pp.start_secondary
> 12.49 +1.4 13.89 perf-profile.children.cycles-pp.secondary_startup_64
> 12.49 +1.4 13.89 perf-profile.children.cycles-pp.cpu_startup_entry
> 12.48 +1.4 13.88 perf-profile.children.cycles-pp.do_idle
> 14.78 +1.4 16.20 ± 2% perf-profile.children.cycles-pp.xlog_wait_on_iclog
> 58.91 +1.7 60.57 perf-profile.children.cycles-pp.xfs_log_force_lsn
> 0.00 +2.2 2.24 ± 13% perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
> 7.05 +2.8 9.81 ± 2% perf-profile.children.cycles-pp.process_one_work
> 7.54 +2.9 10.45 ± 2% perf-profile.children.cycles-pp.worker_thread
> 7.55 +2.9 10.46 ± 2% perf-profile.children.cycles-pp.ret_from_fork
> 7.55 +2.9 10.46 ± 2% perf-profile.children.cycles-pp.kthread
> 0.10 +3.2 3.30 ± 9% perf-profile.children.cycles-pp.xfs_end_ioend
> 0.10 ± 4% +3.2 3.32 ± 8% perf-profile.children.cycles-pp.xfs_end_io
> 19.57 +4.0 23.62 perf-profile.children.cycles-pp.remove_wait_queue
> 1.20 -0.3 0.90 ± 2% perf-profile.self.cycles-pp.brd_do_bvec
> 0.55 ± 3% -0.2 0.39 ± 4% perf-profile.self.cycles-pp.do_syscall_64
> 0.31 -0.1 0.22 ± 5% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
> 0.31 ± 3% -0.1 0.22 perf-profile.self.cycles-pp.syscall_return_via_sysret
> 0.54 ± 3% -0.1 0.47 ± 3% perf-profile.self.cycles-pp.osq_lock
> 0.23 ± 3% -0.1 0.16 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64
> 0.10 ± 4% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.__mod_lruvec_state
> 0.18 ± 4% -0.1 0.12 ± 5% perf-profile.self.cycles-pp.iomap_set_range_uptodate
> 0.60 ± 4% -0.1 0.54 ± 2% perf-profile.self.cycles-pp._raw_spin_lock
> 0.77 -0.0 0.72 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.14 -0.0 0.10 ± 8% perf-profile.self.cycles-pp.find_get_entry
> 0.14 ± 3% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.xas_load
> 0.17 ± 4% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.write
> 0.30 ± 2% -0.0 0.27 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irq
> 0.18 ± 5% -0.0 0.15 ± 7% perf-profile.self.cycles-pp.__flush_work
> 0.08 ± 5% -0.0 0.05 perf-profile.self.cycles-pp.xfs_inode_item_format_data_fork
> 0.12 ± 3% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.brd_lookup_page
> 0.12 ± 7% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.vfs_write
> 0.18 ± 4% -0.0 0.15 ± 7% perf-profile.self.cycles-pp.insert_work
> 0.13 -0.0 0.10 ± 8% perf-profile.self.cycles-pp.memset_erms
> 0.11 ± 4% -0.0 0.08 perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
> 0.11 ± 6% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
> 0.12 ± 3% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.iomap_writepage_map
> 0.08 ± 8% -0.0 0.06 ± 15% perf-profile.self.cycles-pp.bio_alloc_bioset
> 0.06 ± 6% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
> 0.09 ± 5% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.__fget_light
> 0.06 -0.0 0.04 ± 57% perf-profile.self.cycles-pp.xfs_map_blocks
> 0.18 ± 2% -0.0 0.15 ± 3% perf-profile.self.cycles-pp.__list_add_valid
> 0.08 ± 6% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.xfs_ilock
> 0.08 ± 6% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.iov_iter_fault_in_readable
> 0.11 ± 4% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.switch_fpu_return
> 0.07 -0.0 0.05 perf-profile.self.cycles-pp.iomap_write_actor
> 0.23 ± 2% -0.0 0.21 ± 5% perf-profile.self.cycles-pp.__radix_tree_lookup
> 0.18 ± 4% -0.0 0.16 ± 6% perf-profile.self.cycles-pp.xlog_cil_force_lsn
> 0.20 ± 2% -0.0 0.18 ± 4% perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.10 ± 5% -0.0 0.08 ± 11% perf-profile.self.cycles-pp.xfs_file_fsync
> 0.07 ± 5% -0.0 0.05 ± 9% perf-profile.self.cycles-pp.clear_page_dirty_for_io
> 0.07 ± 6% -0.0 0.05 perf-profile.self.cycles-pp.__test_set_page_writeback
> 0.11 -0.0 0.09 ± 8% perf-profile.self.cycles-pp.common_file_perm
> 0.23 -0.0 0.21 ± 4% perf-profile.self.cycles-pp.update_rq_clock
> 0.08 ± 6% -0.0 0.06 perf-profile.self.cycles-pp.xfs_file_write_iter
> 0.10 ± 5% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.up_read
> 0.08 ± 5% -0.0 0.07 perf-profile.self.cycles-pp.write_cache_pages
> 0.10 -0.0 0.09 perf-profile.self.cycles-pp.reweight_entity
> 0.05 ± 8% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.dequeue_entity
> 0.16 ± 4% +0.0 0.17 ± 2% perf-profile.self.cycles-pp.__switch_to_asm
> 0.11 ± 4% +0.0 0.12 ± 3% perf-profile.self.cycles-pp.set_next_entity
> 0.15 ± 3% +0.0 0.16 ± 2% perf-profile.self.cycles-pp.orc_find
> 0.09 ± 4% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.enqueue_entity
> 0.10 ± 4% +0.0 0.12 ± 3% perf-profile.self.cycles-pp._find_next_bit
> 0.01 ±173% +0.0 0.06 ± 7% perf-profile.self.cycles-pp.check_preempt_wakeup
> 0.01 ±173% +0.0 0.06 ± 11% perf-profile.self.cycles-pp.xfs_trans_committed_bulk
> 0.45 ± 2% +0.1 0.51 ± 2% perf-profile.self.cycles-pp.__sched_text_start
> 0.15 ± 6% +0.1 0.21 ± 3% perf-profile.self.cycles-pp.memcpy_erms
> 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.xfs_verify_agbno
> 0.08 ± 8% +0.1 0.16 ± 9% perf-profile.self.cycles-pp.xfs_iext_lookup_extent
> 9.53 +1.2 10.76 perf-profile.self.cycles-pp.intel_idle
>
>
>
> aim7.jobs-per-min
>
> 45000 +-------------------------------------------------------------------+
> | + + |
> 40000 |.+.+ : + .+. +. .+. |
> 35000 |-+ : : + +.+ +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+ ++.+.|
> | : : |
> 30000 |-+ : : |
> 25000 |-O :O O:O OO |
> | : : O O O OO O O O O |
> 20000 |-+ : : |
> 15000 |-+ : : |
> | : : |
> 10000 |-+ : : |
> 5000 |-+ : : |
> | : : |
> 0 +-------------------------------------------------------------------+
>
>
> aim7.time.system_time
>
> 5000 +--------------------------------------------------------------------+
> 4500 |-+ O OO O O O O |
> | O OO O O O O O O |
> 4000 |-+ |
> 3500 |-+ |
> | .+ .+.+. .++.+.+.+.+.++.+.+.+.++.+.+.+.+ .+.+.+. .++.+.|
> 3000 |.+.+ .+ + + + + |
> 2500 |-+ : +.+ |
> 2000 |-+ : : |
> | : : |
> 1500 |-+ : : |
> 1000 |-+ : : |
> | : : |
> 500 |-+ : : |
> 0 +--------------------------------------------------------------------+
>
>
> aim7.time.elapsed_time
>
> 160 +---------------------------------------------------------------------+
> | O OO O O O O OO O O O O |
> 140 |-O |
> 120 |-+ |
> | |
> 100 |-+ .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
> |.+.+ +. .+ + + + |
> 80 |-+ : : + |
> | : : |
> 60 |-+ : : |
> 40 |-+ : : |
> | : : |
> 20 |-+ : : |
> | :: |
> 0 +---------------------------------------------------------------------+
>
>
> aim7.time.elapsed_time.max
>
> 160 +---------------------------------------------------------------------+
> | O OO O O O O OO O O O O |
> 140 |-O |
> 120 |-+ |
> | |
> 100 |-+ .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
> |.+.+ +. .+ + + + |
> 80 |-+ : : + |
> | : : |
> 60 |-+ : : |
> 40 |-+ : : |
> | : : |
> 20 |-+ : : |
> | :: |
> 0 +---------------------------------------------------------------------+
>
>
> aim7.time.voluntary_context_switches
>
> 8e+07 +-------------------------------------------------------------------+
> | O O O O OO O O |
> 7e+07 |-+ |
> 6e+07 |-+ |
> | |
> 5e+07 |-+ +.+.+.++.+.+.+.++.+.+. .+ .+.+. .+. .+. +.+.|
> |.+.+ .+ + +.+ ++.+ ++.+ +.+ |
> 4e+07 |-+ : +.+ |
> | : : |
> 3e+07 |-+ : : |
> 2e+07 |-+ : : |
> | : : |
> 1e+07 |-+ : : |
> | : : |
> 0 +-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>
>
> _______________________________________________
> LKP mailing list -- lkp@...ts.01.org
> To unsubscribe send an email to lkp-leave@...ts.01.org
>
--
Zhengjun Xing
Powered by blists - more mailing lists