| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <307f0c16-ef39-416a-8625-eb010b6807e5@suse.de>
Date: Tue, 12 Mar 2019 21:59:49 +0800
From: Qu Wenruo <wqu@...e.de>
To: kernel test robot <rong.a.chen@...el.com>
Cc: lkp@...org, Stephen Rothwell <sfr@...b.auug.org.au>,
Leonard Lausen <leonard@...sen.nl>,
David Sterba <DSterba@...e.com>,
LKML <linux-kernel@...r.kernel.org>
Subject: Re: [LKP] [btrfs] 44fe89de7d: aim7.jobs-per-min -15.1% regression
On 2019/3/12 下午9:50, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a -15.1% regression of aim7.jobs-per-min due to commit:
>
>
> commit: 44fe89de7d5157a4f31f13d94802c7619e23f462 ("btrfs: Do mandatory tree block check before submitting bio")
That commit will cause extra check before writing tree block back onto disk.
It's expected to cause regression for metadata heavy workload.
I'm more interesting if there is any new real world performance
regression like database or other more dedicated workload.
Thanks,
Qu
> https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
>
> in testcase: aim7
> on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
> with following parameters:
>
> disk: 4BRD_12G
> md: RAID0
> fs: btrfs
> test: sync_disk_rw
> load: 20
> cpufreq_governor: performance
>
> test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
> test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
>
> In addition to that, the commit also has significant impact on the following tests:
>
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | aim7: aim7.jobs-per-min -20.8% regression |
> | test machine | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory |
> | test parameters | cpufreq_governor=performance |
> | | disk=4BRD_12G |
> | | fs=btrfs |
> | | load=20 |
> | | md=RAID0 |
> | | test=sync_disk_rw |
> +------------------+-----------------------------------------------------------------------+
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
> gcc-8/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-skl-2sp7/sync_disk_rw/aim7
>
> commit:
> a92e9c3a3c ("btrfs: extent_io: Handle error better in extent_writepages()")
> 44fe89de7d ("btrfs: Do mandatory tree block check before submitting bio")
>
> a92e9c3a3cef4c0c 44fe89de7d5157a4f31f13d9480
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> :4 50% 2:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
> %stddev %change %stddev
> \ | \
> 1204 -15.1% 1022 aim7.jobs-per-min
> 99.66 +18.3% 117.93 aim7.time.elapsed_time
> 99.66 +18.3% 117.93 aim7.time.elapsed_time.max
> 3014 ± 2% -3.1% 2921 aim7.time.minor_page_faults
> 1.236e+09 +44.6% 1.787e+09 ± 27% cpuidle.C1.time
> 90473 +15.8% 104758 ± 2% meminfo.AnonHugePages
> 102742 -16.5% 85841 ± 2% meminfo.max_used_kB
> 11.68 -1.8 9.85 mpstat.cpu.-1.sys%
> 0.07 ± 7% -0.0 0.05 ± 17% mpstat.cpu.-1.usr%
> 664.25 ± 4% +18.1% 784.25 ± 6% slabinfo.dmaengine-unmap-16.active_objs
> 664.25 ± 4% +18.1% 784.25 ± 6% slabinfo.dmaengine-unmap-16.num_objs
> 88.00 +2.3% 90.00 vmstat.cpu.id
> 346712 -15.5% 293086 vmstat.io.bo
> 558192 -15.4% 472096 vmstat.system.cs
> 88.42 +2.0% 90.19 iostat.cpu.idle
> 11.50 -15.3% 9.74 iostat.cpu.system
> 19892 -15.2% 16877 iostat.md0.w/s
> 451087 -15.4% 381436 iostat.md0.wkB/s
> 252.75 ± 2% -19.8% 202.67 ± 16% sched_debug.cfs_rq:/.removed.util_avg.max
> 57.62 ± 2% +71.4% 98.79 ± 51% sched_debug.cpu.cpu_load[1].max
> 47.75 ± 13% +81.6% 86.71 ± 43% sched_debug.cpu.cpu_load[2].max
> 7.35 ± 13% -16.4% 6.14 ± 4% sched_debug.cpu.cpu_load[4].avg
> 422.75 -14.3% 362.50 turbostat.Avg_MHz
> 14.00 -2.0 12.04 turbostat.Busy%
> 15068782 +15.6% 17424116 ± 2% turbostat.IRQ
> 163.49 -2.2% 159.86 turbostat.PkgWatt
> 516444 +2.0% 526872 proc-vmstat.nr_active_file
> 890319 +1.4% 902554 proc-vmstat.nr_file_pages
> 4668 -1.7% 4591 proc-vmstat.nr_inactive_anon
> 516444 +2.0% 526872 proc-vmstat.nr_zone_active_file
> 4668 -1.7% 4591 proc-vmstat.nr_zone_inactive_anon
> 288573 +14.8% 331243 proc-vmstat.pgfault
> 716355 +7.5% 770096 proc-vmstat.pgfree
> 24648391 -12.7% 21521966 ± 5% perf-stat.i.branch-misses
> 27.60 -1.9 25.66 ± 2% perf-stat.i.cache-miss-rate%
> 33238901 -22.5% 25763877 ± 2% perf-stat.i.cache-misses
> 1.163e+08 -13.8% 1.002e+08 ± 3% perf-stat.i.cache-references
> 570181 -15.7% 480734 perf-stat.i.context-switches
> 4.53 ± 3% -20.6% 3.60 ± 2% perf-stat.i.cpi
> 3.02e+10 -14.5% 2.581e+10 perf-stat.i.cpu-cycles
> 12.93 ± 11% -23.2% 9.92 ± 7% perf-stat.i.cpu-migrations
> 1.81e+09 +8.7% 1.968e+09 ± 3% perf-stat.i.dTLB-loads
> 1.018e+09 +10.9% 1.129e+09 perf-stat.i.dTLB-stores
> 8329645 -15.2% 7063121 perf-stat.i.iTLB-loads
> 6.662e+09 +9.5% 7.297e+09 perf-stat.i.instructions
> 93411 ± 7% +20.1% 112199 ± 6% perf-stat.i.instructions-per-iTLB-miss
> 0.24 ± 4% +21.3% 0.30 perf-stat.i.ipc
> 2719 -2.6% 2650 perf-stat.i.minor-faults
> 10179150 -22.8% 7861388 ± 2% perf-stat.i.node-load-misses
> 819501 -18.0% 672068 perf-stat.i.node-loads
> 79.98 -3.6 76.35 perf-stat.i.node-store-miss-rate%
> 3971266 -23.9% 3020404 perf-stat.i.node-store-misses
> 760058 ± 3% -7.2% 705596 perf-stat.i.node-stores
> 2720 -2.6% 2650 perf-stat.i.page-faults
> 17.46 -21.4% 13.73 ± 2% perf-stat.overall.MPKI
> 1.87 -0.3 1.59 ± 3% perf-stat.overall.branch-miss-rate%
> 28.58 -2.8 25.75 ± 4% perf-stat.overall.cache-miss-rate%
> 4.53 -22.0% 3.54 perf-stat.overall.cpi
> 908.69 +10.3% 1002 ± 2% perf-stat.overall.cycles-between-cache-misses
> 0.22 +28.1% 0.28 perf-stat.overall.ipc
> 83.93 -2.9 81.06 perf-stat.overall.node-store-miss-rate%
> 24408968 -12.5% 21345782 ± 5% perf-stat.ps.branch-misses
> 32896998 -22.4% 25539476 ± 2% perf-stat.ps.cache-misses
> 1.151e+08 -13.7% 99343827 ± 3% perf-stat.ps.cache-references
> 564308 -15.6% 476544 perf-stat.ps.context-switches
> 2.989e+10 -14.4% 2.558e+10 perf-stat.ps.cpu-cycles
> 12.79 ± 11% -23.1% 9.84 ± 7% perf-stat.ps.cpu-migrations
> 1.792e+09 +8.9% 1.951e+09 ± 3% perf-stat.ps.dTLB-loads
> 1.008e+09 +11.1% 1.12e+09 perf-stat.ps.dTLB-stores
> 8244056 -15.1% 7001712 perf-stat.ps.iTLB-loads
> 6.595e+09 +9.7% 7.235e+09 perf-stat.ps.instructions
> 2693 -2.4% 2628 perf-stat.ps.minor-faults
> 10074272 -22.6% 7792778 ± 2% perf-stat.ps.node-load-misses
> 811079 -17.9% 666222 perf-stat.ps.node-loads
> 3930342 -23.8% 2994037 perf-stat.ps.node-store-misses
> 752248 ± 3% -7.0% 699463 perf-stat.ps.node-stores
> 2693 -2.4% 2628 perf-stat.ps.page-faults
> 6.6e+11 +29.8% 8.564e+11 perf-stat.total.instructions
> 44.42 -3.1 41.36 ± 3% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
> 46.97 -2.2 44.77 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 50.60 -1.9 48.74 ± 2% perf-profile.calltrace.cycles-pp.secondary_startup_64
> 49.88 -1.7 48.21 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 49.90 -1.7 48.24 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> 49.90 -1.7 48.24 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> 4.89 ± 12% -1.0 3.84 ± 6% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
> 4.89 ± 12% -1.0 3.84 ± 6% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
> 5.75 ± 6% -0.7 5.07 ± 3% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
> 2.90 ± 11% -0.6 2.30 ± 5% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
> 5.82 ± 2% -0.5 5.35 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
> 1.78 ± 14% -0.4 1.34 ± 6% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
> 4.25 ± 7% -0.4 3.82 perf-profile.calltrace.cycles-pp.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
> 2.73 ± 7% -0.3 2.43 ± 4% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
> 8.78 -0.3 8.49 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
> 1.78 ± 8% -0.3 1.52 ± 3% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
> 8.54 -0.3 8.28 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node
> 2.15 ± 2% -0.2 1.99 ± 5% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
> 1.43 -0.1 1.36 ± 3% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum
> 1.02 ± 2% -0.1 0.97 perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode
> 1.02 ± 3% -0.0 0.97 perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_csum_file_blocks.btrfs_log_changed_extents
> 0.95 ± 5% +0.2 1.11 ± 7% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range
> 1.12 ± 2% +0.2 1.32 ± 6% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file
> 1.05 ± 3% +0.2 1.24 ± 5% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
> 1.25 ± 2% +0.2 1.46 ± 5% perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
> 0.68 ± 6% +0.2 0.92 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
> 1.45 ± 2% +0.2 1.70 ± 4% perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
> 0.84 ± 13% +0.3 1.10 ± 3% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 0.70 ± 6% +0.3 0.95 ± 6% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 0.25 ±100% +0.3 0.56 ± 5% perf-profile.calltrace.cycles-pp.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
> 1.55 ± 2% +0.3 1.87 ± 5% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file
> 1.55 ± 2% +0.3 1.87 ± 5% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops
> 1.57 ± 2% +0.3 1.89 ± 5% perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
> 1.56 ± 2% +0.3 1.88 ± 5% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter
> 1.56 ± 2% +0.3 1.88 ± 5% perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
> 0.26 ±100% +0.4 0.62 ± 6% perf-profile.calltrace.cycles-pp.submit_one_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
> 0.26 ±100% +0.4 0.62 ± 6% perf-profile.calltrace.cycles-pp.btrfs_submit_bio_hook.submit_one_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range
> 0.00 +0.6 0.56 ± 5% perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
> 0.00 +0.6 0.56 ± 5% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio
> 0.00 +0.6 0.58 ± 6% perf-profile.calltrace.cycles-pp.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio.extent_writepages.do_writepages
> 0.19 ±173% +0.6 0.78 ± 27% perf-profile.calltrace.cycles-pp.btrfs_next_old_leaf.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
> 0.00 +0.8 0.84 ± 5% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook
> 0.00 +1.3 1.32 ± 6% perf-profile.calltrace.cycles-pp.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
> 0.00 +1.4 1.39 ± 5% perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
> 0.00 +1.4 1.39 ± 6% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
> 0.75 ± 4% +1.5 2.23 ± 5% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
> 0.76 ± 3% +1.5 2.24 ± 5% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
> 0.75 ± 4% +1.5 2.23 ± 5% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
> 0.75 ± 4% +1.5 2.23 ± 5% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
> 0.38 ± 57% +1.6 1.96 ± 6% perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
> 0.38 ± 57% +1.6 1.96 ± 5% perf-profile.calltrace.cycles-pp.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
> 2.42 ± 2% +1.8 4.18 ± 4% perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
> 38.62 +2.5 41.15 ± 3% perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write
> 39.95 +2.6 42.52 ± 3% perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 39.95 +2.6 42.52 ± 3% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64
> 39.99 +2.6 42.56 ± 3% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 39.98 +2.6 42.55 ± 3% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 40.19 +2.6 42.81 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 40.19 +2.6 42.81 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 97.75 ± 21% +77.5% 173.50 ± 21% interrupts.70:PCI-MSI.12589059-edge.eth3-TxRx-2
> 63.25 ± 9% +51.4% 95.75 ± 25% interrupts.71:PCI-MSI.12589060-edge.eth3-TxRx-3
> 202.50 +17.8% 238.50 interrupts.9:IO-APIC.9-fasteoi.acpi
> 88724 +12.8% 100056 interrupts.CAL:Function_call_interrupts
> 1255 +12.4% 1410 interrupts.CPU0.CAL:Function_call_interrupts
> 201136 +17.1% 235497 ± 2% interrupts.CPU0.LOC:Local_timer_interrupts
> 202.50 +17.8% 238.50 interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
> 1242 +13.2% 1407 interrupts.CPU1.CAL:Function_call_interrupts
> 200860 +16.6% 234199 ± 2% interrupts.CPU1.LOC:Local_timer_interrupts
> 3195 ± 24% -60.2% 1273 ± 95% interrupts.CPU1.NMI:Non-maskable_interrupts
> 3195 ± 24% -60.2% 1273 ± 95% interrupts.CPU1.PMI:Performance_monitoring_interrupts
> 200768 +16.8% 234566 interrupts.CPU10.LOC:Local_timer_interrupts
> 200322 +17.1% 234482 ± 2% interrupts.CPU11.LOC:Local_timer_interrupts
> 1174 ± 11% +18.3% 1389 interrupts.CPU12.CAL:Function_call_interrupts
> 200965 +16.4% 233853 interrupts.CPU12.LOC:Local_timer_interrupts
> 1209 ± 4% +15.4% 1396 interrupts.CPU13.CAL:Function_call_interrupts
> 199915 +17.6% 235085 ± 2% interrupts.CPU13.LOC:Local_timer_interrupts
> 1187 ± 8% +18.1% 1403 interrupts.CPU14.CAL:Function_call_interrupts
> 200677 +17.2% 235290 ± 2% interrupts.CPU14.LOC:Local_timer_interrupts
> 1208 ± 5% +13.2% 1367 ± 4% interrupts.CPU15.CAL:Function_call_interrupts
> 200566 +17.3% 235226 ± 2% interrupts.CPU15.LOC:Local_timer_interrupts
> 200800 +17.1% 235099 ± 2% interrupts.CPU16.LOC:Local_timer_interrupts
> 1217 ± 4% +15.3% 1403 interrupts.CPU17.CAL:Function_call_interrupts
> 200473 +16.6% 233816 interrupts.CPU17.LOC:Local_timer_interrupts
> 1246 +12.8% 1406 interrupts.CPU18.CAL:Function_call_interrupts
> 200863 +16.4% 233829 ± 3% interrupts.CPU18.LOC:Local_timer_interrupts
> 1243 +12.7% 1400 interrupts.CPU19.CAL:Function_call_interrupts
> 200576 +16.0% 232732 ± 2% interrupts.CPU19.LOC:Local_timer_interrupts
> 1247 +12.9% 1407 interrupts.CPU2.CAL:Function_call_interrupts
> 200971 +16.9% 234895 ± 2% interrupts.CPU2.LOC:Local_timer_interrupts
> 1247 +12.7% 1406 interrupts.CPU20.CAL:Function_call_interrupts
> 200473 +16.6% 233803 ± 3% interrupts.CPU20.LOC:Local_timer_interrupts
> 1247 +12.6% 1404 interrupts.CPU21.CAL:Function_call_interrupts
> 200696 +16.2% 233299 ± 2% interrupts.CPU21.LOC:Local_timer_interrupts
> 1247 +13.1% 1411 interrupts.CPU22.CAL:Function_call_interrupts
> 200814 +16.4% 233758 ± 3% interrupts.CPU22.LOC:Local_timer_interrupts
> 4891 ± 12% -24.7% 3681 ± 12% interrupts.CPU22.RES:Rescheduling_interrupts
> 1247 +13.0% 1410 interrupts.CPU23.CAL:Function_call_interrupts
> 200828 +16.4% 233716 ± 3% interrupts.CPU23.LOC:Local_timer_interrupts
> 1247 +14.3% 1426 ± 3% interrupts.CPU24.CAL:Function_call_interrupts
> 200833 +16.5% 233997 ± 3% interrupts.CPU24.LOC:Local_timer_interrupts
> 1246 +12.9% 1407 interrupts.CPU25.CAL:Function_call_interrupts
> 201055 +16.3% 233920 ± 3% interrupts.CPU25.LOC:Local_timer_interrupts
> 1265 ± 2% +12.7% 1425 ± 2% interrupts.CPU26.CAL:Function_call_interrupts
> 201132 +15.9% 233180 ± 3% interrupts.CPU26.LOC:Local_timer_interrupts
> 1173 ± 10% +20.0% 1407 interrupts.CPU27.CAL:Function_call_interrupts
> 201166 +15.9% 233210 ± 3% interrupts.CPU27.LOC:Local_timer_interrupts
> 1263 ± 2% +11.4% 1407 interrupts.CPU28.CAL:Function_call_interrupts
> 201049 +16.3% 233848 ± 3% interrupts.CPU28.LOC:Local_timer_interrupts
> 1241 +13.5% 1409 interrupts.CPU29.CAL:Function_call_interrupts
> 200963 +16.3% 233774 ± 3% interrupts.CPU29.LOC:Local_timer_interrupts
> 97.75 ± 21% +77.5% 173.50 ± 21% interrupts.CPU3.70:PCI-MSI.12589059-edge.eth3-TxRx-2
> 1210 ± 5% +13.2% 1369 ± 3% interrupts.CPU3.CAL:Function_call_interrupts
> 199954 +17.1% 234167 ± 2% interrupts.CPU3.LOC:Local_timer_interrupts
> 1181 ± 9% +19.1% 1407 interrupts.CPU30.CAL:Function_call_interrupts
> 200986 +16.4% 233892 ± 3% interrupts.CPU30.LOC:Local_timer_interrupts
> 1127 ± 18% +24.7% 1406 interrupts.CPU31.CAL:Function_call_interrupts
> 201110 +16.2% 233756 ± 3% interrupts.CPU31.LOC:Local_timer_interrupts
> 1249 +13.0% 1412 interrupts.CPU32.CAL:Function_call_interrupts
> 200832 +16.4% 233787 ± 3% interrupts.CPU32.LOC:Local_timer_interrupts
> 1249 +13.3% 1416 interrupts.CPU33.CAL:Function_call_interrupts
> 200969 +16.3% 233758 ± 3% interrupts.CPU33.LOC:Local_timer_interrupts
> 1246 +13.4% 1413 interrupts.CPU34.CAL:Function_call_interrupts
> 199891 +17.1% 234012 ± 3% interrupts.CPU34.LOC:Local_timer_interrupts
> 1234 +13.3% 1399 interrupts.CPU35.CAL:Function_call_interrupts
> 200721 +16.2% 233256 ± 2% interrupts.CPU35.LOC:Local_timer_interrupts
> 3994 ± 14% -50.7% 1970 ± 48% interrupts.CPU35.RES:Rescheduling_interrupts
> 1242 +12.9% 1402 ± 2% interrupts.CPU36.CAL:Function_call_interrupts
> 199765 +17.7% 235071 ± 2% interrupts.CPU36.LOC:Local_timer_interrupts
> 200843 +16.6% 234175 ± 2% interrupts.CPU37.LOC:Local_timer_interrupts
> 1248 +13.4% 1415 interrupts.CPU38.CAL:Function_call_interrupts
> 200016 +17.0% 234095 ± 2% interrupts.CPU38.LOC:Local_timer_interrupts
> 1249 +13.0% 1411 interrupts.CPU39.CAL:Function_call_interrupts
> 200714 +17.2% 235250 ± 2% interrupts.CPU39.LOC:Local_timer_interrupts
> 63.25 ± 9% +51.4% 95.75 ± 25% interrupts.CPU4.71:PCI-MSI.12589060-edge.eth3-TxRx-3
> 1246 +13.9% 1419 ± 2% interrupts.CPU4.CAL:Function_call_interrupts
> 200637 +17.2% 235179 ± 2% interrupts.CPU4.LOC:Local_timer_interrupts
> 200593 +17.2% 235093 ± 2% interrupts.CPU40.LOC:Local_timer_interrupts
> 1248 +13.0% 1410 interrupts.CPU41.CAL:Function_call_interrupts
> 200998 +16.6% 234380 interrupts.CPU41.LOC:Local_timer_interrupts
> 1249 +12.9% 1410 interrupts.CPU42.CAL:Function_call_interrupts
> 200934 +16.6% 234357 interrupts.CPU42.LOC:Local_timer_interrupts
> 1246 +13.2% 1411 interrupts.CPU43.CAL:Function_call_interrupts
> 200894 +16.9% 234942 ± 2% interrupts.CPU43.LOC:Local_timer_interrupts
> 1211 ± 5% +16.3% 1408 interrupts.CPU44.CAL:Function_call_interrupts
> 200154 +17.4% 235070 ± 2% interrupts.CPU44.LOC:Local_timer_interrupts
> 2454 ± 23% +50.0% 3682 ± 14% interrupts.CPU44.RES:Rescheduling_interrupts
> 1248 +13.0% 1410 interrupts.CPU45.CAL:Function_call_interrupts
> 200774 +16.7% 234259 ± 2% interrupts.CPU45.LOC:Local_timer_interrupts
> 1246 +13.1% 1410 interrupts.CPU46.CAL:Function_call_interrupts
> 201184 +16.8% 234966 ± 2% interrupts.CPU46.LOC:Local_timer_interrupts
> 1246 +13.1% 1409 interrupts.CPU47.CAL:Function_call_interrupts
> 200672 +16.7% 234200 interrupts.CPU47.LOC:Local_timer_interrupts
> 200070 +17.7% 235463 ± 2% interrupts.CPU48.LOC:Local_timer_interrupts
> 1251 +12.6% 1409 interrupts.CPU49.CAL:Function_call_interrupts
> 200232 +17.4% 235071 ± 2% interrupts.CPU49.LOC:Local_timer_interrupts
> 200450 +17.4% 235328 ± 2% interrupts.CPU5.LOC:Local_timer_interrupts
> 1216 ± 5% +15.7% 1407 interrupts.CPU50.CAL:Function_call_interrupts
> 201051 +16.5% 234316 interrupts.CPU50.LOC:Local_timer_interrupts
> 1252 +12.6% 1410 interrupts.CPU51.CAL:Function_call_interrupts
> 200007 +17.5% 235046 ± 2% interrupts.CPU51.LOC:Local_timer_interrupts
> 1253 +12.4% 1408 interrupts.CPU52.CAL:Function_call_interrupts
> 200770 +17.0% 234811 ± 2% interrupts.CPU52.LOC:Local_timer_interrupts
> 1252 +12.2% 1405 interrupts.CPU53.CAL:Function_call_interrupts
> 200095 +17.4% 235004 ± 2% interrupts.CPU53.LOC:Local_timer_interrupts
> 1252 +12.6% 1410 interrupts.CPU54.CAL:Function_call_interrupts
> 200697 +16.5% 233818 ± 3% interrupts.CPU54.LOC:Local_timer_interrupts
> 1252 +12.6% 1409 interrupts.CPU55.CAL:Function_call_interrupts
> 200852 +16.5% 234024 ± 3% interrupts.CPU55.LOC:Local_timer_interrupts
> 1251 +12.5% 1407 interrupts.CPU56.CAL:Function_call_interrupts
> 200874 +16.4% 233828 ± 3% interrupts.CPU56.LOC:Local_timer_interrupts
> 1251 +12.3% 1404 interrupts.CPU57.CAL:Function_call_interrupts
> 200887 +16.5% 233974 ± 3% interrupts.CPU57.LOC:Local_timer_interrupts
> 1250 +12.8% 1411 interrupts.CPU58.CAL:Function_call_interrupts
> 200837 +16.4% 233702 ± 3% interrupts.CPU58.LOC:Local_timer_interrupts
> 1250 +12.8% 1410 interrupts.CPU59.CAL:Function_call_interrupts
> 199809 +17.1% 233932 ± 3% interrupts.CPU59.LOC:Local_timer_interrupts
> 1262 ± 2% +10.5% 1394 interrupts.CPU6.CAL:Function_call_interrupts
> 201108 +16.6% 234545 interrupts.CPU6.LOC:Local_timer_interrupts
> 1253 +12.6% 1411 interrupts.CPU60.CAL:Function_call_interrupts
> 200941 +16.4% 233857 ± 3% interrupts.CPU60.LOC:Local_timer_interrupts
> 2285 ± 53% -55.3% 1021 ± 85% interrupts.CPU60.NMI:Non-maskable_interrupts
> 2285 ± 53% -55.3% 1021 ± 85% interrupts.CPU60.PMI:Performance_monitoring_interrupts
> 1213 ± 4% +16.2% 1410 interrupts.CPU61.CAL:Function_call_interrupts
> 200849 +16.4% 233812 ± 3% interrupts.CPU61.LOC:Local_timer_interrupts
> 1250 +12.7% 1408 interrupts.CPU62.CAL:Function_call_interrupts
> 200858 +16.4% 233847 ± 3% interrupts.CPU62.LOC:Local_timer_interrupts
> 1249 +12.7% 1408 interrupts.CPU63.CAL:Function_call_interrupts
> 200846 +16.3% 233509 ± 3% interrupts.CPU63.LOC:Local_timer_interrupts
> 1249 +12.6% 1407 interrupts.CPU64.CAL:Function_call_interrupts
> 201010 +16.3% 233868 ± 3% interrupts.CPU64.LOC:Local_timer_interrupts
> 1249 +12.6% 1407 interrupts.CPU65.CAL:Function_call_interrupts
> 201188 +16.2% 233840 ± 3% interrupts.CPU65.LOC:Local_timer_interrupts
> 1249 +12.6% 1405 interrupts.CPU66.CAL:Function_call_interrupts
> 201142 +16.3% 233885 ± 3% interrupts.CPU66.LOC:Local_timer_interrupts
> 1249 +12.6% 1406 interrupts.CPU67.CAL:Function_call_interrupts
> 200840 +16.5% 233913 ± 3% interrupts.CPU67.LOC:Local_timer_interrupts
> 1249 +12.7% 1408 interrupts.CPU68.CAL:Function_call_interrupts
> 200815 +16.4% 233774 ± 3% interrupts.CPU68.LOC:Local_timer_interrupts
> 1249 +12.8% 1409 interrupts.CPU69.CAL:Function_call_interrupts
> 200983 +16.3% 233829 ± 3% interrupts.CPU69.LOC:Local_timer_interrupts
> 200912 +17.1% 235169 ± 2% interrupts.CPU7.LOC:Local_timer_interrupts
> 1249 +13.0% 1412 interrupts.CPU70.CAL:Function_call_interrupts
> 200821 +16.4% 233823 ± 3% interrupts.CPU70.LOC:Local_timer_interrupts
> 1247 +12.9% 1408 interrupts.CPU71.CAL:Function_call_interrupts
> 200673 +16.4% 233586 ± 3% interrupts.CPU71.LOC:Local_timer_interrupts
> 3112 ± 28% -55.8% 1376 ± 92% interrupts.CPU71.NMI:Non-maskable_interrupts
> 3112 ± 28% -55.8% 1376 ± 92% interrupts.CPU71.PMI:Performance_monitoring_interrupts
> 2655 ± 18% +53.3% 4071 ± 21% interrupts.CPU71.RES:Rescheduling_interrupts
> 1088 ± 16% +25.1% 1361 ± 4% interrupts.CPU8.CAL:Function_call_interrupts
> 201291 +16.9% 235317 ± 2% interrupts.CPU8.LOC:Local_timer_interrupts
> 201246 +16.4% 234250 interrupts.CPU9.LOC:Local_timer_interrupts
> 14452726 +16.7% 16865906 ± 2% interrupts.LOC:Local_timer_interrupts
> 25553 ± 7% +26.4% 32295 softirqs.CPU0.RCU
> 17578 +14.6% 20147 ± 2% softirqs.CPU0.SCHED
> 40851 +20.4% 49167 ± 3% softirqs.CPU0.TIMER
> 40546 +17.0% 47455 ± 5% softirqs.CPU1.TIMER
> 25814 +27.1% 32807 ± 10% softirqs.CPU10.RCU
> 38717 ± 2% +14.6% 44386 ± 5% softirqs.CPU10.TIMER
> 38245 +16.3% 44468 ± 5% softirqs.CPU11.TIMER
> 26439 ± 4% +21.3% 32083 ± 4% softirqs.CPU12.RCU
> 15582 ± 4% +14.4% 17827 ± 2% softirqs.CPU12.SCHED
> 38686 +17.5% 45474 ± 4% softirqs.CPU12.TIMER
> 25962 ± 4% +18.3% 30720 ± 4% softirqs.CPU13.RCU
> 16126 ± 5% +17.9% 19013 ± 5% softirqs.CPU13.SCHED
> 25846 ± 2% +20.4% 31108 ± 5% softirqs.CPU14.RCU
> 15629 ± 4% +21.3% 18953 ± 6% softirqs.CPU14.SCHED
> 37591 +17.9% 44326 ± 5% softirqs.CPU14.TIMER
> 24676 ± 2% +20.2% 29658 softirqs.CPU15.RCU
> 38217 +16.0% 44331 ± 4% softirqs.CPU15.TIMER
> 24680 ± 2% +18.6% 29278 ± 5% softirqs.CPU16.RCU
> 38783 +17.9% 45709 ± 4% softirqs.CPU16.TIMER
> 24738 +20.6% 29833 ± 10% softirqs.CPU17.RCU
> 39088 +16.3% 45476 ± 5% softirqs.CPU17.TIMER
> 27971 +24.7% 34880 ± 4% softirqs.CPU18.RCU
> 27800 +19.7% 33276 ± 2% softirqs.CPU19.RCU
> 16069 ± 5% +14.6% 18417 ± 3% softirqs.CPU19.SCHED
> 39339 +18.2% 46516 ± 5% softirqs.CPU19.TIMER
> 26084 +10.4% 28804 ± 5% softirqs.CPU2.RCU
> 16601 ± 8% +15.0% 19085 ± 10% softirqs.CPU2.SCHED
> 38625 +22.4% 47291 ± 5% softirqs.CPU2.TIMER
> 27472 +27.0% 34877 ± 7% softirqs.CPU20.RCU
> 16088 ± 3% +10.7% 17811 ± 4% softirqs.CPU20.SCHED
> 38845 +17.6% 45674 ± 3% softirqs.CPU20.TIMER
> 28205 ± 6% +20.2% 33906 ± 7% softirqs.CPU21.RCU
> 39324 +13.8% 44753 ± 4% softirqs.CPU21.TIMER
> 27282 ± 9% +20.0% 32736 ± 4% softirqs.CPU22.RCU
> 39542 +13.9% 45038 ± 3% softirqs.CPU22.TIMER
> 27295 ± 7% +22.3% 33387 ± 2% softirqs.CPU23.RCU
> 15694 ± 3% +13.2% 17772 ± 3% softirqs.CPU23.SCHED
> 37452 +16.5% 43645 ± 5% softirqs.CPU23.TIMER
> 25529 ± 10% +30.1% 33204 softirqs.CPU24.RCU
> 16068 ± 6% +13.8% 18289 ± 4% softirqs.CPU24.SCHED
> 39129 +29.4% 50648 ± 17% softirqs.CPU24.TIMER
> 27444 ± 3% +18.3% 32463 ± 4% softirqs.CPU25.RCU
> 38658 +14.6% 44286 ± 6% softirqs.CPU25.TIMER
> 27586 ± 2% +21.4% 33496 softirqs.CPU26.RCU
> 39631 +13.9% 45136 ± 4% softirqs.CPU26.TIMER
> 27618 ± 3% +15.9% 32011 ± 2% softirqs.CPU27.RCU
> 26884 +24.1% 33355 ± 7% softirqs.CPU28.RCU
> 39120 +14.6% 44813 ± 5% softirqs.CPU28.TIMER
> 27542 ± 4% +21.4% 33433 ± 5% softirqs.CPU29.RCU
> 38659 ± 2% +15.9% 44815 ± 5% softirqs.CPU29.TIMER
> 27539 ± 3% +14.2% 31446 softirqs.CPU3.RCU
> 16465 ± 4% +16.6% 19192 ± 6% softirqs.CPU3.SCHED
> 38875 +17.9% 45849 ± 5% softirqs.CPU3.TIMER
> 26139 +23.5% 32277 ± 4% softirqs.CPU30.RCU
> 39666 +11.6% 44285 ± 5% softirqs.CPU30.TIMER
> 26559 ± 2% +23.7% 32855 ± 8% softirqs.CPU31.RCU
> 16278 ± 2% +19.3% 19420 ± 10% softirqs.CPU31.SCHED
> 39178 +15.2% 45121 ± 4% softirqs.CPU31.TIMER
> 25781 +20.9% 31163 ± 2% softirqs.CPU32.RCU
> 37670 +13.6% 42811 ± 5% softirqs.CPU32.TIMER
> 27107 ± 5% +16.7% 31625 ± 4% softirqs.CPU33.RCU
> 16098 ± 2% +12.3% 18071 ± 2% softirqs.CPU33.SCHED
> 38565 +15.6% 44572 ± 4% softirqs.CPU33.TIMER
> 26241 ± 3% +18.9% 31209 ± 5% softirqs.CPU34.RCU
> 38223 +18.1% 45160 ± 5% softirqs.CPU34.TIMER
> 25360 ± 6% +27.0% 32210 ± 2% softirqs.CPU35.RCU
> 39986 ± 2% +13.8% 45494 ± 3% softirqs.CPU35.TIMER
> 26626 ± 2% +23.9% 32987 ± 4% softirqs.CPU36.RCU
> 16281 ± 5% +12.6% 18329 softirqs.CPU36.SCHED
> 27227 +23.8% 33703 ± 2% softirqs.CPU37.RCU
> 26701 ± 2% +29.1% 34462 ± 8% softirqs.CPU38.RCU
> 15157 ± 5% +15.5% 17507 ± 4% softirqs.CPU38.SCHED
> 37119 ± 4% +37.9% 51191 ± 27% softirqs.CPU38.TIMER
> 28007 ± 2% +21.4% 33990 ± 2% softirqs.CPU39.RCU
> 15700 ± 2% +11.8% 17545 ± 3% softirqs.CPU39.SCHED
> 37509 ± 2% +14.9% 43109 ± 6% softirqs.CPU39.TIMER
> 25877 +24.6% 32254 ± 3% softirqs.CPU4.RCU
> 38648 +16.9% 45191 ± 5% softirqs.CPU4.TIMER
> 27903 +17.8% 32867 ± 2% softirqs.CPU40.RCU
> 15256 ± 2% +12.8% 17202 ± 7% softirqs.CPU40.SCHED
> 36371 ± 2% +17.0% 42572 ± 4% softirqs.CPU40.TIMER
> 26560 ± 5% +19.9% 31840 ± 6% softirqs.CPU41.RCU
> 15067 ± 3% +13.8% 17142 ± 4% softirqs.CPU41.SCHED
> 36473 ± 3% +14.2% 41657 ± 4% softirqs.CPU41.TIMER
> 27250 +22.5% 33387 ± 2% softirqs.CPU42.RCU
> 15376 ± 4% +15.4% 17744 ± 3% softirqs.CPU42.SCHED
> 37838 +13.4% 42895 ± 3% softirqs.CPU42.TIMER
> 27137 ± 4% +23.2% 33425 softirqs.CPU43.RCU
> 27818 ± 3% +22.5% 34073 ± 6% softirqs.CPU44.RCU
> 14305 ± 4% +22.5% 17522 ± 2% softirqs.CPU44.SCHED
> 36151 ± 4% +16.7% 42187 ± 4% softirqs.CPU44.TIMER
> 25734 ± 3% +25.6% 32319 ± 4% softirqs.CPU45.RCU
> 14522 +14.8% 16674 ± 6% softirqs.CPU45.SCHED
> 36594 +15.8% 42364 ± 5% softirqs.CPU45.TIMER
> 28660 ± 3% +18.8% 34052 ± 7% softirqs.CPU46.RCU
> 15306 ± 4% +13.5% 17370 ± 7% softirqs.CPU46.SCHED
> 37095 ± 3% +14.3% 42388 ± 5% softirqs.CPU46.TIMER
> 28906 ± 5% +15.3% 33324 ± 7% softirqs.CPU47.RCU
> 14758 ± 5% +13.7% 16775 ± 5% softirqs.CPU47.SCHED
> 36736 ± 3% +14.2% 41969 ± 4% softirqs.CPU47.TIMER
> 28049 ± 4% +24.0% 34791 ± 2% softirqs.CPU48.RCU
> 14583 ± 4% +18.9% 17340 ± 8% softirqs.CPU48.SCHED
> 36095 ± 4% +19.5% 43117 ± 6% softirqs.CPU48.TIMER
> 27551 ± 5% +21.0% 33349 ± 4% softirqs.CPU49.RCU
> 36024 +18.1% 42545 ± 4% softirqs.CPU49.TIMER
> 25870 ± 2% +22.4% 31677 ± 3% softirqs.CPU5.RCU
> 15740 ± 6% +26.8% 19951 ± 9% softirqs.CPU5.SCHED
> 37622 +33.5% 50234 ± 18% softirqs.CPU5.TIMER
> 27424 +19.6% 32798 ± 2% softirqs.CPU50.RCU
> 14010 ± 2% +21.8% 17060 ± 6% softirqs.CPU50.SCHED
> 36485 ± 3% +14.7% 41862 ± 4% softirqs.CPU50.TIMER
> 27884 +18.9% 33160 ± 3% softirqs.CPU51.RCU
> 36212 ± 3% +16.6% 42210 ± 4% softirqs.CPU51.TIMER
> 27031 ± 7% +29.8% 35075 ± 5% softirqs.CPU52.RCU
> 14772 ± 2% +18.1% 17451 ± 4% softirqs.CPU52.SCHED
> 37031 ± 4% +17.9% 43648 ± 4% softirqs.CPU52.TIMER
> 25188 ± 12% +28.7% 32423 ± 3% softirqs.CPU53.RCU
> 16289 ± 6% +18.4% 19280 ± 5% softirqs.CPU53.SCHED
> 37616 ± 3% +35.6% 51010 ± 21% softirqs.CPU53.TIMER
> 26693 ± 3% +19.4% 31870 ± 2% softirqs.CPU54.RCU
> 38094 +16.5% 44379 ± 4% softirqs.CPU54.TIMER
> 26082 ± 2% +23.1% 32100 softirqs.CPU55.RCU
> 37589 ± 2% +16.5% 43779 ± 5% softirqs.CPU55.TIMER
> 26352 ± 2% +18.7% 31284 softirqs.CPU56.RCU
> 15591 ± 4% +15.9% 18066 ± 8% softirqs.CPU56.SCHED
> 37070 +16.8% 43309 ± 2% softirqs.CPU56.TIMER
> 27520 ± 4% +15.7% 31841 ± 5% softirqs.CPU57.RCU
> 37916 ± 2% +13.9% 43170 ± 2% softirqs.CPU57.TIMER
> 27067 +20.8% 32709 ± 7% softirqs.CPU58.RCU
> 15059 +19.1% 17935 ± 8% softirqs.CPU58.SCHED
> 37123 +13.8% 42241 softirqs.CPU58.TIMER
> 25199 ± 4% +19.6% 30126 ± 6% softirqs.CPU59.RCU
> 36384 ± 2% +16.8% 42479 ± 2% softirqs.CPU59.TIMER
> 25871 ± 3% +17.5% 30409 ± 3% softirqs.CPU6.RCU
> 16584 ± 2% +14.8% 19032 ± 5% softirqs.CPU6.SCHED
> 38784 +15.6% 44850 ± 4% softirqs.CPU6.TIMER
> 24777 ± 5% +15.0% 28502 ± 5% softirqs.CPU60.RCU
> 37468 +14.2% 42794 ± 6% softirqs.CPU60.TIMER
> 23108 ± 3% +30.1% 30065 ± 2% softirqs.CPU61.RCU
> 15059 ± 6% +16.5% 17540 ± 6% softirqs.CPU61.SCHED
> 37002 +16.8% 43227 ± 3% softirqs.CPU61.TIMER
> 23023 ± 7% +33.6% 30769 ± 6% softirqs.CPU62.RCU
> 37164 ± 2% +16.2% 43203 ± 2% softirqs.CPU62.TIMER
> 23069 ± 2% +25.1% 28850 softirqs.CPU63.RCU
> 14642 ± 5% +15.4% 16900 ± 8% softirqs.CPU63.SCHED
> 36254 ± 2% +15.4% 41854 ± 3% softirqs.CPU63.TIMER
> 24304 ± 2% +25.9% 30593 ± 2% softirqs.CPU64.RCU
> 14666 +15.5% 16938 ± 3% softirqs.CPU64.SCHED
> 37048 +15.7% 42877 ± 4% softirqs.CPU64.TIMER
> 24438 ± 9% +23.6% 30212 ± 5% softirqs.CPU65.RCU
> 37505 +12.8% 42288 ± 2% softirqs.CPU65.TIMER
> 24568 +20.4% 29569 softirqs.CPU66.RCU
> 15290 ± 5% +8.4% 16569 ± 4% softirqs.CPU66.SCHED
> 37381 +14.9% 42968 ± 3% softirqs.CPU66.TIMER
> 23253 ± 9% +29.7% 30150 ± 2% softirqs.CPU67.RCU
> 14645 ± 6% +13.4% 16613 ± 6% softirqs.CPU67.SCHED
> 37012 +14.1% 42221 ± 2% softirqs.CPU67.TIMER
> 36993 +15.0% 42536 ± 4% softirqs.CPU68.TIMER
> 24695 ± 7% +23.7% 30551 ± 3% softirqs.CPU69.RCU
> 14441 ± 3% +11.4% 16089 ± 4% softirqs.CPU69.SCHED
> 36999 +14.3% 42299 softirqs.CPU69.TIMER
> 26022 ± 3% +21.4% 31579 ± 2% softirqs.CPU7.RCU
> 38481 +18.7% 45661 ± 3% softirqs.CPU7.TIMER
> 25083 ± 8% +18.5% 29721 ± 2% softirqs.CPU70.RCU
> 14845 ± 7% +15.7% 17173 ± 4% softirqs.CPU70.SCHED
> 37094 +18.5% 43948 ± 3% softirqs.CPU70.TIMER
> 24438 ± 2% +20.6% 29461 ± 2% softirqs.CPU71.RCU
> 15821 ± 4% +13.6% 17978 ± 4% softirqs.CPU71.SCHED
> 26912 ± 7% +22.3% 32925 ± 6% softirqs.CPU8.RCU
> 39193 +15.2% 45167 ± 4% softirqs.CPU8.TIMER
> 25829 ± 2% +20.7% 31178 softirqs.CPU9.RCU
> 37434 ± 2% +16.6% 43665 ± 5% softirqs.CPU9.TIMER
> 1897274 +21.5% 2304570 ± 2% softirqs.RCU
> 1150548 +11.9% 1287616 softirqs.SCHED
> 2757329 +15.8% 3193002 ± 3% softirqs.TIMER
>
>
>
> aim7.jobs-per-min
>
> 1400 +-+------------------------------------------------------------------+
> | |
> 1200 +-+++.++.++.+++.++.++ + ++.+ + +++.++ + ++.++.++.++.+++.++.++.+|
> | : : : : : : : : : |
> 1000 OO+OO OO OO OOO OO OO OOO OO OO OO OOO OO OO OOO O |
> | : : : : : : : :: : |
> 800 +-+ : : : : :: : : :: : |
> | : : : : :: : : :: : |
> 600 +-+ : ::: :: :: : :: : |
> | :: :: :: :: :: :: |
> 400 +-+ :: :: :: :: :: :: |
> | : :: :: :: : : |
> 200 +-+ : : : : : : |
> | : : : : : : |
> 0 +-+------------------------------------------------------------------+
>
>
> aim7.time.elapsed_time
>
> 140 +-+-------------------------------------------------------------------+
> | |
> 120 OO+OO OO OO OO OO O O OO OO O OO OOO OO OO OO OO O |
> | O O |
> 100 +-+++.++.++.++.++.+++ + +.++ + ++.++.+ + ++.++.+++.++.++.++.++.+|
> | : : : : : : : : : |
> 80 +-+ : : : : : : : : : |
> | : :: : : :: : : :: : |
> 60 +-+ : :: : : :: : : :: : |
> | : :: : : :: : :: :: |
> 40 +-+ :: :: :: :: :: :: |
> | :: :: :: :: :: :: |
> 20 +-+ : : : : : : |
> | : : : : : : |
> 0 +-+-------------------------------------------------------------------+
>
>
> aim7.time.elapsed_time.max
>
> 140 +-+-------------------------------------------------------------------+
> | |
> 120 OO+OO OO OO OO OO O O OO OO O OO OOO OO OO OO OO O |
> | O O |
> 100 +-+++.++.++.++.++.+++ + +.++ + ++.++.+ + ++.++.+++.++.++.++.++.+|
> | : : : : : : : : : |
> 80 +-+ : : : : : : : : : |
> | : :: : : :: : : :: : |
> 60 +-+ : :: : : :: : : :: : |
> | : :: : : :: : :: :: |
> 40 +-+ :: :: :: :: :: :: |
> | :: :: :: :: :: :: |
> 20 +-+ : : : : : : |
> | : : : : : : |
> 0 +-+-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
> ***************************************************************************************************
> lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
> =========================================================================================
> compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
> gcc-8/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/sync_disk_rw/aim7
>
> commit:
> a92e9c3a3c ("btrfs: extent_io: Handle error better in extent_writepages()")
> 44fe89de7d ("btrfs: Do mandatory tree block check before submitting bio")
>
> a92e9c3a3cef4c0c 44fe89de7d5157a4f31f13d9480
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 474.71 -20.8% 376.09 aim7.jobs-per-min
> 252.89 +26.2% 319.19 aim7.time.elapsed_time
> 252.89 +26.2% 319.19 aim7.time.elapsed_time.max
> 8376 ± 3% +9.4% 9167 ± 3% aim7.time.minor_page_faults
> 1760 +6.3% 1872 aim7.time.system_time
> 22.58 -1.5% 22.24 boot-time.dhcp
> 40945 ± 2% -22.3% 31813 ± 3% meminfo.max_used_kB
> 670014 +24.2% 832471 proc-vmstat.pgfault
> 1013780 +13.7% 1152374 proc-vmstat.pgfree
> 3.022e+09 ± 4% +50.2% 4.54e+09 ± 19% cpuidle.C1.time
> 34330913 ± 2% +16.1% 39848597 ± 9% cpuidle.C1.usage
> 2.611e+09 ± 16% +48.9% 3.887e+09 ± 3% cpuidle.C6.time
> 0.02 ± 15% -0.0 0.00 ±113% mpstat.cpu.-1.soft%
> 21.36 -3.4 17.98 mpstat.cpu.-1.sys%
> 0.14 -0.0 0.12 mpstat.cpu.-1.usr%
> 78.00 +4.8% 81.75 vmstat.cpu.id
> 138672 -21.1% 109467 vmstat.io.bo
> 299636 -21.0% 236781 vmstat.system.cs
> 78.64 +4.3% 81.99 iostat.cpu.idle
> 21.21 -15.7% 17.88 iostat.cpu.system
> 7947 -20.5% 6318 iostat.md0.w/s
> 180043 -20.9% 142377 iostat.md0.wkB/s
> 3098 ± 30% +270.4% 11475 ± 34% numa-meminfo.node0.Inactive(anon)
> 9526 +37.0% 13052 ± 15% numa-meminfo.node0.Mapped
> 5474 ± 26% +188.7% 15807 ± 23% numa-meminfo.node0.Shmem
> 75040 ± 2% +10.4% 82807 ± 7% numa-meminfo.node0.Slab
> 16192 ± 5% -51.6% 7838 ± 51% numa-meminfo.node1.Inactive(anon)
> 20174 ± 7% -50.1% 10076 ± 36% numa-meminfo.node1.Shmem
> 774.25 ± 30% +270.4% 2867 ± 34% numa-vmstat.node0.nr_inactive_anon
> 2474 ± 2% +35.6% 3354 ± 14% numa-vmstat.node0.nr_mapped
> 1367 ± 26% +188.9% 3950 ± 23% numa-vmstat.node0.nr_shmem
> 774.25 ± 30% +270.4% 2867 ± 34% numa-vmstat.node0.nr_zone_inactive_anon
> 4046 ± 5% -51.6% 1958 ± 51% numa-vmstat.node1.nr_inactive_anon
> 5042 ± 7% -50.1% 2518 ± 36% numa-vmstat.node1.nr_shmem
> 4046 ± 5% -51.6% 1958 ± 51% numa-vmstat.node1.nr_zone_inactive_anon
> 312.25 -15.1% 265.25 turbostat.Avg_MHz
> 25.87 -3.9 22.02 turbostat.Busy%
> 34326534 ± 2% +16.1% 39845190 ± 9% turbostat.C1
> 33.45 -4.6% 31.92 turbostat.CorWatt
> 21163841 +25.1% 26466542 turbostat.IRQ
> 59.45 -2.7% 57.87 turbostat.PkgWatt
> 32.25 -1.7% 31.71 turbostat.RAMWatt
> 20330 +26.1% 25630 turbostat.SMI
> 6.46 ± 2% -0.6 5.89 ± 3% perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
> 2.75 ± 5% -0.6 2.19 ± 11% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
> 5.46 ± 2% -0.5 4.92 ± 3% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
> 5.46 ± 2% -0.5 4.93 ± 3% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
> 0.69 ± 9% -0.4 0.27 ±100% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.__vfs_write.vfs_write
> 1.74 ± 6% -0.4 1.39 ± 8% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write
> 0.63 ± 3% -0.3 0.30 ±100% perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio.extent_writepages
> 2.35 ± 4% -0.2 2.13 ± 3% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents
> 0.92 ± 9% -0.1 0.80 ± 7% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
> 0.86 ± 10% -0.1 0.74 ± 9% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages
> 0.85 ± 9% -0.1 0.73 ± 8% perf-profile.calltrace.cycles-pp.cow_file_range.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
> 0.70 ± 5% -0.1 0.57 ± 3% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
> 1.40 ± 5% +0.2 1.55 perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 0.65 ± 11% +0.3 0.94 ± 23% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
> 0.79 ± 8% +0.3 1.13 ± 21% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
> 0.26 ±100% +0.3 0.61 ± 7% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
> 1.44 ± 8% +0.4 1.87 ± 17% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
> 0.14 ±173% +0.6 0.70 ± 27% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
> 2.44 ± 5% +0.6 3.04 ± 7% perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
> 0.00 +0.7 0.72 ± 2% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook
> 2.51 ± 6% +0.8 3.30 ± 16% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
> 2.71 ± 6% +0.8 3.53 ± 16% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
> 0.73 ± 6% +1.0 1.69 ± 2% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
> 0.72 ± 6% +1.0 1.69 ± 2% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
> 0.72 ± 6% +1.0 1.69 ± 2% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
> 0.72 ± 6% +1.0 1.69 ± 2% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
> 0.00 +1.0 1.05 perf-profile.calltrace.cycles-pp.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
> 0.00 +1.1 1.08 perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
> 0.00 +1.1 1.09 perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
> 0.00 +1.5 1.46 perf-profile.calltrace.cycles-pp.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
> 0.00 +1.5 1.46 perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
> 71.60 ± 7% -16.2% 59.98 ± 8% sched_debug.cfs_rq:/.load_avg.avg
> 96.11 ± 10% -16.2% 80.58 ± 6% sched_debug.cfs_rq:/.load_avg.stddev
> 187997 -8.8% 171461 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg
> 226472 ± 2% -10.8% 202016 ± 6% sched_debug.cfs_rq:/.min_vruntime.max
> 17261 ± 7% -21.5% 13550 ± 15% sched_debug.cfs_rq:/.min_vruntime.stddev
> 8.89 ± 47% -64.7% 3.14 ± 57% sched_debug.cfs_rq:/.removed.load_avg.avg
> 204.15 -38.5% 125.58 ± 57% sched_debug.cfs_rq:/.removed.load_avg.max
> 40.32 ± 22% -51.4% 19.61 ± 57% sched_debug.cfs_rq:/.removed.load_avg.stddev
> 409.91 ± 47% -64.8% 144.41 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.avg
> 9437 -38.8% 5776 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.max
> 1859 ± 22% -51.5% 901.87 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.stddev
> 2.95 ± 44% -79.5% 0.60 ± 75% sched_debug.cfs_rq:/.removed.util_avg.avg
> 72.25 ± 8% -66.6% 24.17 ± 75% sched_debug.cfs_rq:/.removed.util_avg.max
> 13.53 ± 23% -72.1% 3.77 ± 75% sched_debug.cfs_rq:/.removed.util_avg.stddev
> 9.43 ± 5% +27.0% 11.97 ± 14% sched_debug.cfs_rq:/.runnable_load_avg.avg
> 103.80 ± 10% +62.1% 168.29 ± 27% sched_debug.cfs_rq:/.runnable_load_avg.max
> 22.41 ± 7% +39.0% 31.15 ± 20% sched_debug.cfs_rq:/.runnable_load_avg.stddev
> 12421 ± 59% +64.4% 20419 ± 40% sched_debug.cfs_rq:/.spread0.avg
> -24890 -69.6% -7575 sched_debug.cfs_rq:/.spread0.min
> 17263 ± 7% -21.5% 13551 ± 15% sched_debug.cfs_rq:/.spread0.stddev
> 325.68 ± 2% -14.6% 278.27 ± 3% sched_debug.cfs_rq:/.util_avg.avg
> 186.27 ± 3% -11.0% 165.84 ± 2% sched_debug.cfs_rq:/.util_avg.stddev
> 132.45 -12.4% 116.05 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.stddev
> 155113 +19.3% 185086 sched_debug.cpu.clock.avg
> 155121 +19.3% 185093 sched_debug.cpu.clock.max
> 155106 +19.3% 185079 sched_debug.cpu.clock.min
> 155113 +19.3% 185086 sched_debug.cpu.clock_task.avg
> 155121 +19.3% 185093 sched_debug.cpu.clock_task.max
> 155106 +19.3% 185079 sched_debug.cpu.clock_task.min
> 104.40 ± 7% +61.5% 168.62 ± 27% sched_debug.cpu.cpu_load[0].max
> 96.20 ± 8% +74.1% 167.46 ± 30% sched_debug.cpu.cpu_load[1].max
> 4871 +14.8% 5592 sched_debug.cpu.curr->pid.max
> 127731 +23.5% 157685 sched_debug.cpu.nr_load_updates.avg
> 134899 +20.9% 163058 sched_debug.cpu.nr_load_updates.max
> 125325 +24.0% 155428 sched_debug.cpu.nr_load_updates.min
> 155106 +19.3% 185079 sched_debug.cpu_clk
> 151513 +19.8% 181485 sched_debug.ktime
> 155537 +19.3% 185526 sched_debug.sched_clk
> 19.77 -15.1% 16.78 ± 3% perf-stat.i.MPKI
> 8.783e+08 -9.0% 7.989e+08 perf-stat.i.branch-instructions
> 19.44 -0.2 19.21 perf-stat.i.cache-miss-rate%
> 14932557 -19.1% 12081970 perf-stat.i.cache-misses
> 77250252 -18.0% 63363496 perf-stat.i.cache-references
> 302132 -21.2% 238146 perf-stat.i.context-switches
> 3.13 -11.6% 2.77 ± 2% perf-stat.i.cpi
> 1.254e+10 -15.4% 1.061e+10 perf-stat.i.cpu-cycles
> 229.12 ± 3% -21.3% 180.36 ± 3% perf-stat.i.cpu-migrations
> 840.12 +4.5% 877.89 perf-stat.i.cycles-between-cache-misses
> 1.129e+09 -5.4% 1.068e+09 perf-stat.i.dTLB-loads
> 0.10 ± 4% -0.0 0.09 ± 5% perf-stat.i.dTLB-store-miss-rate%
> 538827 ± 3% -16.1% 452028 ± 4% perf-stat.i.dTLB-store-misses
> 47.32 +3.3 50.67 ± 3% perf-stat.i.iTLB-load-miss-rate%
> 2556829 -20.4% 2034861 ± 2% perf-stat.i.iTLB-loads
> 4.153e+09 -4.5% 3.964e+09 perf-stat.i.instructions
> 0.33 +12.2% 0.37 ± 2% perf-stat.i.ipc
> 6672298 -19.6% 5364393 perf-stat.i.node-load-misses
> 7193961 -19.6% 5786904 perf-stat.i.node-loads
> 4765580 -20.6% 3784168 perf-stat.i.node-store-misses
> 6965089 -20.3% 5548679 perf-stat.i.node-stores
> 18.60 -14.0% 15.99 ± 2% perf-stat.overall.MPKI
> 19.33 -0.3 19.07 perf-stat.overall.cache-miss-rate%
> 3.02 -11.3% 2.68 ± 2% perf-stat.overall.cpi
> 839.71 +4.6% 878.19 perf-stat.overall.cycles-between-cache-misses
> 0.10 ± 4% -0.0 0.08 ± 5% perf-stat.overall.dTLB-store-miss-rate%
> 45.63 +3.8 49.47 ± 3% perf-stat.overall.iTLB-load-miss-rate%
> 0.33 +12.8% 0.37 ± 2% perf-stat.overall.ipc
> 8.748e+08 -9.0% 7.964e+08 perf-stat.ps.branch-instructions
> 14872846 -19.0% 12043692 perf-stat.ps.cache-misses
> 76941024 -17.9% 63162394 perf-stat.ps.cache-references
> 300917 -21.1% 237386 perf-stat.ps.context-switches
> 1.249e+10 -15.3% 1.058e+10 perf-stat.ps.cpu-cycles
> 228.20 ± 3% -21.2% 179.78 ± 3% perf-stat.ps.cpu-migrations
> 1.124e+09 -5.3% 1.065e+09 perf-stat.ps.dTLB-loads
> 536681 ± 3% -16.0% 450604 ± 4% perf-stat.ps.dTLB-store-misses
> 2546543 -20.3% 2028370 ± 2% perf-stat.ps.iTLB-loads
> 4.137e+09 -4.5% 3.952e+09 perf-stat.ps.instructions
> 6645469 -19.5% 5347283 perf-stat.ps.node-load-misses
> 7165042 -19.5% 5768454 perf-stat.ps.node-loads
> 4746415 -20.5% 3772102 perf-stat.ps.node-store-misses
> 6937096 -20.3% 5531006 perf-stat.ps.node-stores
> 1.049e+12 +20.4% 1.263e+12 perf-stat.total.instructions
> 160.75 ± 18% +49.5% 240.25 ± 27% interrupts.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5
> 123.75 +26.3% 156.25 interrupts.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
> 100313 ± 7% +13.4% 113737 interrupts.CAL:Function_call_interrupts
> 508519 +26.0% 640668 interrupts.CPU0.LOC:Local_timer_interrupts
> 2437 ± 12% +18.1% 2877 ± 4% interrupts.CPU1.CAL:Function_call_interrupts
> 508036 +26.1% 640579 interrupts.CPU1.LOC:Local_timer_interrupts
> 2521 ± 10% +14.3% 2882 ± 5% interrupts.CPU10.CAL:Function_call_interrupts
> 507477 +26.1% 639993 interrupts.CPU10.LOC:Local_timer_interrupts
> 2519 ± 2% +17.9% 2969 ± 6% interrupts.CPU11.CAL:Function_call_interrupts
> 508051 +26.1% 640531 interrupts.CPU11.LOC:Local_timer_interrupts
> 508353 +25.8% 639418 interrupts.CPU12.LOC:Local_timer_interrupts
> 508289 +26.0% 640385 interrupts.CPU13.LOC:Local_timer_interrupts
> 508136 +25.8% 639470 interrupts.CPU14.LOC:Local_timer_interrupts
> 2417 ± 5% +20.2% 2906 ± 6% interrupts.CPU15.CAL:Function_call_interrupts
> 508330 +25.9% 640229 interrupts.CPU15.LOC:Local_timer_interrupts
> 508351 +25.9% 639790 interrupts.CPU16.LOC:Local_timer_interrupts
> 2508 ± 10% +18.4% 2968 ± 6% interrupts.CPU17.CAL:Function_call_interrupts
> 508084 +26.0% 640420 interrupts.CPU17.LOC:Local_timer_interrupts
> 17327 ± 18% +26.1% 21856 ± 4% interrupts.CPU17.RES:Rescheduling_interrupts
> 508457 +25.8% 639737 interrupts.CPU18.LOC:Local_timer_interrupts
> 508140 +26.0% 640214 interrupts.CPU19.LOC:Local_timer_interrupts
> 2280 ± 8% +22.2% 2786 ± 5% interrupts.CPU2.CAL:Function_call_interrupts
> 508336 +26.0% 640548 interrupts.CPU2.LOC:Local_timer_interrupts
> 507912 +26.1% 640315 interrupts.CPU20.LOC:Local_timer_interrupts
> 2595 ± 7% +13.4% 2943 ± 5% interrupts.CPU21.CAL:Function_call_interrupts
> 508108 +26.1% 640485 interrupts.CPU21.LOC:Local_timer_interrupts
> 508306 +26.0% 640244 interrupts.CPU22.LOC:Local_timer_interrupts
> 1303 ± 34% -29.2% 922.75 ± 32% interrupts.CPU22.NMI:Non-maskable_interrupts
> 1303 ± 34% -29.2% 922.75 ± 32% interrupts.CPU22.PMI:Performance_monitoring_interrupts
> 2522 ± 9% +17.2% 2956 ± 4% interrupts.CPU23.CAL:Function_call_interrupts
> 508123 +26.1% 640526 interrupts.CPU23.LOC:Local_timer_interrupts
> 508372 +26.0% 640372 interrupts.CPU24.LOC:Local_timer_interrupts
> 507678 +26.1% 640435 interrupts.CPU25.LOC:Local_timer_interrupts
> 508375 +26.0% 640407 interrupts.CPU26.LOC:Local_timer_interrupts
> 2406 ± 11% +19.9% 2885 ± 5% interrupts.CPU27.CAL:Function_call_interrupts
> 508242 +26.0% 640265 interrupts.CPU27.LOC:Local_timer_interrupts
> 2449 ± 11% +21.8% 2982 ± 5% interrupts.CPU28.CAL:Function_call_interrupts
> 508292 +26.0% 640304 interrupts.CPU28.LOC:Local_timer_interrupts
> 2510 ± 6% +14.4% 2870 ± 5% interrupts.CPU29.CAL:Function_call_interrupts
> 508085 +26.0% 640167 interrupts.CPU29.LOC:Local_timer_interrupts
> 508172 +26.0% 640443 interrupts.CPU3.LOC:Local_timer_interrupts
> 508342 +25.9% 639891 interrupts.CPU30.LOC:Local_timer_interrupts
> 508009 +26.1% 640402 interrupts.CPU31.LOC:Local_timer_interrupts
> 2522 ± 11% +16.8% 2946 ± 4% interrupts.CPU32.CAL:Function_call_interrupts
> 508138 +25.9% 639578 interrupts.CPU32.LOC:Local_timer_interrupts
> 508010 +26.1% 640577 interrupts.CPU33.LOC:Local_timer_interrupts
> 508029 +25.9% 639405 interrupts.CPU34.LOC:Local_timer_interrupts
> 2464 ± 8% +15.3% 2840 ± 5% interrupts.CPU35.CAL:Function_call_interrupts
> 508216 +26.0% 640171 interrupts.CPU35.LOC:Local_timer_interrupts
> 160.75 ± 18% +49.5% 240.25 ± 27% interrupts.CPU36.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5
> 507990 +25.9% 639575 interrupts.CPU36.LOC:Local_timer_interrupts
> 508010 +26.0% 640200 interrupts.CPU37.LOC:Local_timer_interrupts
> 2536 ± 8% +13.9% 2888 ± 6% interrupts.CPU38.CAL:Function_call_interrupts
> 507853 +25.9% 639351 interrupts.CPU38.LOC:Local_timer_interrupts
> 2414 ± 7% +19.6% 2887 ± 7% interrupts.CPU39.CAL:Function_call_interrupts
> 507619 +26.0% 639416 interrupts.CPU39.LOC:Local_timer_interrupts
> 508318 +25.9% 640069 interrupts.CPU4.LOC:Local_timer_interrupts
> 2463 ± 12% +26.6% 3119 ± 2% interrupts.CPU5.CAL:Function_call_interrupts
> 508112 +26.1% 640497 interrupts.CPU5.LOC:Local_timer_interrupts
> 123.75 +26.3% 156.25 interrupts.CPU6.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
> 2351 ± 9% +21.5% 2856 ± 5% interrupts.CPU6.CAL:Function_call_interrupts
> 508563 +25.8% 639582 interrupts.CPU6.LOC:Local_timer_interrupts
> 2592 ± 4% +13.6% 2943 ± 4% interrupts.CPU7.CAL:Function_call_interrupts
> 508150 +26.0% 640378 interrupts.CPU7.LOC:Local_timer_interrupts
> 508498 +25.9% 640139 interrupts.CPU8.LOC:Local_timer_interrupts
> 2501 ± 10% +18.4% 2960 ± 7% interrupts.CPU9.CAL:Function_call_interrupts
> 508092 +26.0% 640432 interrupts.CPU9.LOC:Local_timer_interrupts
> 20326190 +26.0% 25605623 interrupts.LOC:Local_timer_interrupts
> 26401 ± 5% -9.9% 23785 ± 4% interrupts.NMI:Non-maskable_interrupts
> 26401 ± 5% -9.9% 23785 ± 4% interrupts.PMI:Performance_monitoring_interrupts
> 69901 ± 3% +27.9% 89422 ± 3% softirqs.CPU0.RCU
> 45503 ± 2% +18.1% 53752 softirqs.CPU0.SCHED
> 90281 ± 2% +17.4% 105987 ± 2% softirqs.CPU0.TIMER
> 72001 ± 4% +30.7% 94130 ± 2% softirqs.CPU1.RCU
> 42373 +18.9% 50374 ± 2% softirqs.CPU1.SCHED
> 89928 ± 3% +18.5% 106554 ± 4% softirqs.CPU1.TIMER
> 73216 ± 6% +24.5% 91156 ± 3% softirqs.CPU10.RCU
> 42693 +16.8% 49881 ± 2% softirqs.CPU10.SCHED
> 86975 ± 2% +19.4% 103822 ± 3% softirqs.CPU10.TIMER
> 71035 +31.2% 93194 ± 2% softirqs.CPU11.RCU
> 42794 +19.6% 51168 ± 2% softirqs.CPU11.SCHED
> 86533 +18.9% 102845 ± 4% softirqs.CPU11.TIMER
> 70358 ± 3% +33.3% 93810 ± 3% softirqs.CPU12.RCU
> 42548 +17.2% 49878 softirqs.CPU12.SCHED
> 87303 ± 2% +21.8% 106341 ± 3% softirqs.CPU12.TIMER
> 70364 ± 3% +31.7% 92648 ± 4% softirqs.CPU13.RCU
> 42421 +18.9% 50428 softirqs.CPU13.SCHED
> 86875 +19.5% 103817 ± 3% softirqs.CPU13.TIMER
> 69128 ± 2% +30.7% 90331 ± 3% softirqs.CPU14.RCU
> 42975 ± 2% +15.4% 49575 softirqs.CPU14.SCHED
> 84688 ± 3% +21.6% 102994 ± 3% softirqs.CPU14.TIMER
> 72010 ± 2% +30.6% 94068 ± 3% softirqs.CPU15.RCU
> 42893 +17.0% 50178 softirqs.CPU15.SCHED
> 86458 +20.4% 104117 ± 4% softirqs.CPU15.TIMER
> 70886 ± 2% +30.3% 92387 ± 2% softirqs.CPU16.RCU
> 42970 +16.3% 49975 softirqs.CPU16.SCHED
> 87944 ± 2% +19.7% 105244 ± 2% softirqs.CPU16.TIMER
> 70757 ± 2% +30.5% 92359 ± 2% softirqs.CPU17.RCU
> 42742 +17.7% 50310 ± 2% softirqs.CPU17.SCHED
> 87096 ± 2% +17.9% 102652 ± 3% softirqs.CPU17.TIMER
> 69077 ± 3% +32.6% 91630 ± 3% softirqs.CPU18.RCU
> 42942 +17.0% 50245 ± 2% softirqs.CPU18.SCHED
> 68722 ± 2% +29.7% 89150 ± 2% softirqs.CPU19.RCU
> 42964 ± 2% +17.2% 50334 ± 2% softirqs.CPU19.SCHED
> 84910 ± 4% +24.0% 105326 ± 7% softirqs.CPU19.TIMER
> 72687 ± 4% +25.6% 91315 ± 3% softirqs.CPU2.RCU
> 42688 +16.2% 49622 softirqs.CPU2.SCHED
> 88869 ± 5% +16.3% 103332 softirqs.CPU2.TIMER
> 70693 ± 2% +26.6% 89525 ± 4% softirqs.CPU20.RCU
> 42228 +17.5% 49611 ± 2% softirqs.CPU20.SCHED
> 88647 ± 2% +16.4% 103227 ± 2% softirqs.CPU20.TIMER
> 70857 ± 3% +32.1% 93633 ± 3% softirqs.CPU21.RCU
> 42878 +17.2% 50239 softirqs.CPU21.SCHED
> 87325 +18.2% 103214 ± 4% softirqs.CPU21.TIMER
> 71099 ± 3% +29.0% 91711 ± 3% softirqs.CPU22.RCU
> 42534 +17.3% 49888 softirqs.CPU22.SCHED
> 87698 ± 3% +19.3% 104642 softirqs.CPU22.TIMER
> 72098 ± 2% +30.3% 93930 ± 3% softirqs.CPU23.RCU
> 42771 +17.0% 50022 ± 2% softirqs.CPU23.SCHED
> 87931 +17.5% 103319 ± 3% softirqs.CPU23.TIMER
> 70804 ± 3% +33.4% 94432 ± 6% softirqs.CPU24.RCU
> 42554 ± 3% +16.4% 49516 ± 2% softirqs.CPU24.SCHED
> 71435 ± 3% +35.0% 96465 ± 3% softirqs.CPU25.RCU
> 42605 +16.9% 49815 softirqs.CPU25.SCHED
> 87163 +21.2% 105672 softirqs.CPU25.TIMER
> 70216 +31.0% 91964 ± 3% softirqs.CPU26.RCU
> 42147 +19.0% 50165 ± 2% softirqs.CPU26.SCHED
> 88215 ± 3% +36.3% 120217 ± 21% softirqs.CPU26.TIMER
> 70713 +32.5% 93695 ± 6% softirqs.CPU27.RCU
> 42528 +18.2% 50249 ± 2% softirqs.CPU27.SCHED
> 88110 +38.5% 122002 ± 24% softirqs.CPU27.TIMER
> 69654 ± 2% +34.1% 93398 ± 3% softirqs.CPU28.RCU
> 42052 +18.0% 49636 ± 2% softirqs.CPU28.SCHED
> 86876 ± 2% +21.6% 105634 ± 2% softirqs.CPU28.TIMER
> 70230 ± 2% +37.8% 96776 ± 5% softirqs.CPU29.RCU
> 42440 +17.0% 49673 ± 2% softirqs.CPU29.SCHED
> 87058 +18.9% 103502 ± 2% softirqs.CPU29.TIMER
> 70519 ± 3% +34.2% 94624 ± 2% softirqs.CPU3.RCU
> 43299 +16.2% 50322 ± 2% softirqs.CPU3.SCHED
> 87572 +19.2% 104386 ± 4% softirqs.CPU3.TIMER
> 70538 ± 2% +31.0% 92391 ± 5% softirqs.CPU30.RCU
> 42038 ± 2% +17.9% 49574 softirqs.CPU30.SCHED
> 86446 ± 3% +20.6% 104284 softirqs.CPU30.TIMER
> 71140 ± 2% +33.6% 95050 ± 3% softirqs.CPU31.RCU
> 42599 +16.5% 49608 ± 2% softirqs.CPU31.SCHED
> 86417 +19.6% 103393 ± 3% softirqs.CPU31.TIMER
> 59293 ± 2% +29.7% 76894 ± 4% softirqs.CPU32.RCU
> 42200 +17.8% 49711 softirqs.CPU32.SCHED
> 86726 +18.8% 103045 ± 3% softirqs.CPU32.TIMER
> 58507 +31.6% 76967 softirqs.CPU33.RCU
> 42587 +16.7% 49683 softirqs.CPU33.SCHED
> 86567 +19.1% 103101 ± 3% softirqs.CPU33.TIMER
> 59394 ± 2% +27.1% 75493 ± 5% softirqs.CPU34.RCU
> 42682 +16.6% 49753 softirqs.CPU34.SCHED
> 59949 ± 3% +29.2% 77459 softirqs.CPU35.RCU
> 42681 ± 2% +17.3% 50047 ± 2% softirqs.CPU35.SCHED
> 85785 +19.9% 102825 ± 2% softirqs.CPU35.TIMER
> 61089 +25.4% 76612 softirqs.CPU36.RCU
> 42662 +16.9% 49870 softirqs.CPU36.SCHED
> 87784 ± 2% +17.3% 102973 ± 3% softirqs.CPU36.TIMER
> 60018 +29.3% 77596 softirqs.CPU37.RCU
> 42378 ± 2% +18.0% 50012 softirqs.CPU37.SCHED
> 85919 +20.3% 103386 ± 2% softirqs.CPU37.TIMER
> 59781 ± 2% +28.6% 76877 softirqs.CPU38.RCU
> 42779 +17.1% 50105 softirqs.CPU38.SCHED
> 85246 ± 4% +20.7% 102859 ± 2% softirqs.CPU38.TIMER
> 59657 +26.9% 75699 ± 2% softirqs.CPU39.RCU
> 43011 ± 2% +17.6% 50580 ± 3% softirqs.CPU39.SCHED
> 98669 ± 20% +36.3% 134491 ± 24% softirqs.CPU39.TIMER
> 68912 ± 2% +32.7% 91479 ± 2% softirqs.CPU4.RCU
> 43101 +15.9% 49946 softirqs.CPU4.SCHED
> 86425 ± 5% +18.4% 102344 ± 3% softirqs.CPU4.TIMER
> 70893 ± 3% +32.1% 93618 ± 3% softirqs.CPU5.RCU
> 42977 +17.2% 50381 softirqs.CPU5.SCHED
> 87301 +17.7% 102788 ± 3% softirqs.CPU5.TIMER
> 71151 +25.0% 88965 ± 3% softirqs.CPU6.RCU
> 42864 +17.3% 50282 softirqs.CPU6.SCHED
> 87741 ± 2% +22.5% 107458 ± 4% softirqs.CPU6.TIMER
> 71242 ± 4% +30.5% 92999 ± 3% softirqs.CPU7.RCU
> 43014 ± 2% +17.7% 50619 softirqs.CPU7.SCHED
> 87907 +15.8% 101781 ± 5% softirqs.CPU7.TIMER
> 69638 ± 2% +32.6% 92309 ± 4% softirqs.CPU8.RCU
> 42479 +17.6% 49947 softirqs.CPU8.SCHED
> 86964 ± 2% +18.6% 103128 ± 3% softirqs.CPU8.TIMER
> 69585 ± 2% +36.1% 94684 ± 2% softirqs.CPU9.RCU
> 42667 +18.0% 50343 softirqs.CPU9.SCHED
> 86913 +19.2% 103596 ± 4% softirqs.CPU9.TIMER
> 2739272 +30.7% 3580875 ± 2% softirqs.RCU
> 1709285 +17.3% 2005342 softirqs.SCHED
> 3539953 +19.4% 4226728 ± 2% softirqs.TIMER
>
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>
Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)
Powered by blists - more mailing lists