lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Tue, 12 Mar 2019 21:59:49 +0800
From:   Qu Wenruo <wqu@...e.de>
To:     kernel test robot <rong.a.chen@...el.com>
Cc:     lkp@...org, Stephen Rothwell <sfr@...b.auug.org.au>,
        Leonard Lausen <leonard@...sen.nl>,
        David Sterba <DSterba@...e.com>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: [LKP] [btrfs] 44fe89de7d: aim7.jobs-per-min -15.1% regression



On 2019/3/12 下午9:50, kernel test robot wrote:
> Greeting,
> 
> FYI, we noticed a -15.1% regression of aim7.jobs-per-min due to commit:
> 
> 
> commit: 44fe89de7d5157a4f31f13d94802c7619e23f462 ("btrfs: Do mandatory tree block check before submitting bio")

That commit will cause extra check before writing tree block back onto disk.

It's expected to cause regression for metadata heavy workload.

I'm more interesting if there is any new real world performance
regression like database or other more dedicated workload.

Thanks,
Qu

> https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
> 
> in testcase: aim7
> on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
> with following parameters:
> 
> 	disk: 4BRD_12G
> 	md: RAID0
> 	fs: btrfs
> 	test: sync_disk_rw
> 	load: 20
> 	cpufreq_governor: performance
> 
> test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
> test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
> 
> In addition to that, the commit also has significant impact on the following tests:
> 
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | aim7: aim7.jobs-per-min -20.8% regression                             |
> | test machine     | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory |
> | test parameters  | cpufreq_governor=performance                                          |
> |                  | disk=4BRD_12G                                                         |
> |                  | fs=btrfs                                                              |
> |                  | load=20                                                               |
> |                  | md=RAID0                                                              |
> |                  | test=sync_disk_rw                                                     |
> +------------------+-----------------------------------------------------------------------+
> 
> 
> Details are as below:
> -------------------------------------------------------------------------------------------------->
> 
> 
> To reproduce:
> 
>         git clone https://github.com/intel/lkp-tests.git
>         cd lkp-tests
>         bin/lkp install job.yaml  # job file is attached in this email
>         bin/lkp run     job.yaml
> 
> =========================================================================================
> compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
>   gcc-8/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-skl-2sp7/sync_disk_rw/aim7
> 
> commit: 
>   a92e9c3a3c ("btrfs: extent_io: Handle error better in extent_writepages()")
>   44fe89de7d ("btrfs: Do mandatory tree block check before submitting bio")
> 
> a92e9c3a3cef4c0c 44fe89de7d5157a4f31f13d9480 
> ---------------- --------------------------- 
>        fail:runs  %reproduction    fail:runs
>            |             |             |    
>            :4           50%           2:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
>          %stddev     %change         %stddev
>              \          |                \  
>       1204           -15.1%       1022        aim7.jobs-per-min
>      99.66           +18.3%     117.93        aim7.time.elapsed_time
>      99.66           +18.3%     117.93        aim7.time.elapsed_time.max
>       3014 ±  2%      -3.1%       2921        aim7.time.minor_page_faults
>  1.236e+09           +44.6%  1.787e+09 ± 27%  cpuidle.C1.time
>      90473           +15.8%     104758 ±  2%  meminfo.AnonHugePages
>     102742           -16.5%      85841 ±  2%  meminfo.max_used_kB
>      11.68            -1.8        9.85        mpstat.cpu.-1.sys%
>       0.07 ±  7%      -0.0        0.05 ± 17%  mpstat.cpu.-1.usr%
>     664.25 ±  4%     +18.1%     784.25 ±  6%  slabinfo.dmaengine-unmap-16.active_objs
>     664.25 ±  4%     +18.1%     784.25 ±  6%  slabinfo.dmaengine-unmap-16.num_objs
>      88.00            +2.3%      90.00        vmstat.cpu.id
>     346712           -15.5%     293086        vmstat.io.bo
>     558192           -15.4%     472096        vmstat.system.cs
>      88.42            +2.0%      90.19        iostat.cpu.idle
>      11.50           -15.3%       9.74        iostat.cpu.system
>      19892           -15.2%      16877        iostat.md0.w/s
>     451087           -15.4%     381436        iostat.md0.wkB/s
>     252.75 ±  2%     -19.8%     202.67 ± 16%  sched_debug.cfs_rq:/.removed.util_avg.max
>      57.62 ±  2%     +71.4%      98.79 ± 51%  sched_debug.cpu.cpu_load[1].max
>      47.75 ± 13%     +81.6%      86.71 ± 43%  sched_debug.cpu.cpu_load[2].max
>       7.35 ± 13%     -16.4%       6.14 ±  4%  sched_debug.cpu.cpu_load[4].avg
>     422.75           -14.3%     362.50        turbostat.Avg_MHz
>      14.00            -2.0       12.04        turbostat.Busy%
>   15068782           +15.6%   17424116 ±  2%  turbostat.IRQ
>     163.49            -2.2%     159.86        turbostat.PkgWatt
>     516444            +2.0%     526872        proc-vmstat.nr_active_file
>     890319            +1.4%     902554        proc-vmstat.nr_file_pages
>       4668            -1.7%       4591        proc-vmstat.nr_inactive_anon
>     516444            +2.0%     526872        proc-vmstat.nr_zone_active_file
>       4668            -1.7%       4591        proc-vmstat.nr_zone_inactive_anon
>     288573           +14.8%     331243        proc-vmstat.pgfault
>     716355            +7.5%     770096        proc-vmstat.pgfree
>   24648391           -12.7%   21521966 ±  5%  perf-stat.i.branch-misses
>      27.60            -1.9       25.66 ±  2%  perf-stat.i.cache-miss-rate%
>   33238901           -22.5%   25763877 ±  2%  perf-stat.i.cache-misses
>  1.163e+08           -13.8%  1.002e+08 ±  3%  perf-stat.i.cache-references
>     570181           -15.7%     480734        perf-stat.i.context-switches
>       4.53 ±  3%     -20.6%       3.60 ±  2%  perf-stat.i.cpi
>   3.02e+10           -14.5%  2.581e+10        perf-stat.i.cpu-cycles
>      12.93 ± 11%     -23.2%       9.92 ±  7%  perf-stat.i.cpu-migrations
>   1.81e+09            +8.7%  1.968e+09 ±  3%  perf-stat.i.dTLB-loads
>  1.018e+09           +10.9%  1.129e+09        perf-stat.i.dTLB-stores
>    8329645           -15.2%    7063121        perf-stat.i.iTLB-loads
>  6.662e+09            +9.5%  7.297e+09        perf-stat.i.instructions
>      93411 ±  7%     +20.1%     112199 ±  6%  perf-stat.i.instructions-per-iTLB-miss
>       0.24 ±  4%     +21.3%       0.30        perf-stat.i.ipc
>       2719            -2.6%       2650        perf-stat.i.minor-faults
>   10179150           -22.8%    7861388 ±  2%  perf-stat.i.node-load-misses
>     819501           -18.0%     672068        perf-stat.i.node-loads
>      79.98            -3.6       76.35        perf-stat.i.node-store-miss-rate%
>    3971266           -23.9%    3020404        perf-stat.i.node-store-misses
>     760058 ±  3%      -7.2%     705596        perf-stat.i.node-stores
>       2720            -2.6%       2650        perf-stat.i.page-faults
>      17.46           -21.4%      13.73 ±  2%  perf-stat.overall.MPKI
>       1.87            -0.3        1.59 ±  3%  perf-stat.overall.branch-miss-rate%
>      28.58            -2.8       25.75 ±  4%  perf-stat.overall.cache-miss-rate%
>       4.53           -22.0%       3.54        perf-stat.overall.cpi
>     908.69           +10.3%       1002 ±  2%  perf-stat.overall.cycles-between-cache-misses
>       0.22           +28.1%       0.28        perf-stat.overall.ipc
>      83.93            -2.9       81.06        perf-stat.overall.node-store-miss-rate%
>   24408968           -12.5%   21345782 ±  5%  perf-stat.ps.branch-misses
>   32896998           -22.4%   25539476 ±  2%  perf-stat.ps.cache-misses
>  1.151e+08           -13.7%   99343827 ±  3%  perf-stat.ps.cache-references
>     564308           -15.6%     476544        perf-stat.ps.context-switches
>  2.989e+10           -14.4%  2.558e+10        perf-stat.ps.cpu-cycles
>      12.79 ± 11%     -23.1%       9.84 ±  7%  perf-stat.ps.cpu-migrations
>  1.792e+09            +8.9%  1.951e+09 ±  3%  perf-stat.ps.dTLB-loads
>  1.008e+09           +11.1%   1.12e+09        perf-stat.ps.dTLB-stores
>    8244056           -15.1%    7001712        perf-stat.ps.iTLB-loads
>  6.595e+09            +9.7%  7.235e+09        perf-stat.ps.instructions
>       2693            -2.4%       2628        perf-stat.ps.minor-faults
>   10074272           -22.6%    7792778 ±  2%  perf-stat.ps.node-load-misses
>     811079           -17.9%     666222        perf-stat.ps.node-loads
>    3930342           -23.8%    2994037        perf-stat.ps.node-store-misses
>     752248 ±  3%      -7.0%     699463        perf-stat.ps.node-stores
>       2693            -2.4%       2628        perf-stat.ps.page-faults
>    6.6e+11           +29.8%  8.564e+11        perf-stat.total.instructions
>      44.42            -3.1       41.36 ±  3%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
>      46.97            -2.2       44.77 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>      50.60            -1.9       48.74 ±  2%  perf-profile.calltrace.cycles-pp.secondary_startup_64
>      49.88            -1.7       48.21 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>      49.90            -1.7       48.24 ±  2%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
>      49.90            -1.7       48.24 ±  2%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
>       4.89 ± 12%      -1.0        3.84 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
>       4.89 ± 12%      -1.0        3.84 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
>       5.75 ±  6%      -0.7        5.07 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
>       2.90 ± 11%      -0.6        2.30 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
>       5.82 ±  2%      -0.5        5.35 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
>       1.78 ± 14%      -0.4        1.34 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
>       4.25 ±  7%      -0.4        3.82        perf-profile.calltrace.cycles-pp.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
>       2.73 ±  7%      -0.3        2.43 ±  4%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
>       8.78            -0.3        8.49 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
>       1.78 ±  8%      -0.3        1.52 ±  3%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
>       8.54            -0.3        8.28 ±  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node
>       2.15 ±  2%      -0.2        1.99 ±  5%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
>       1.43            -0.1        1.36 ±  3%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum
>       1.02 ±  2%      -0.1        0.97        perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode
>       1.02 ±  3%      -0.0        0.97        perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_csum_file_blocks.btrfs_log_changed_extents
>       0.95 ±  5%      +0.2        1.11 ±  7%  perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range
>       1.12 ±  2%      +0.2        1.32 ±  6%  perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file
>       1.05 ±  3%      +0.2        1.24 ±  5%  perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
>       1.25 ±  2%      +0.2        1.46 ±  5%  perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
>       0.68 ±  6%      +0.2        0.92 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
>       1.45 ±  2%      +0.2        1.70 ±  4%  perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
>       0.84 ± 13%      +0.3        1.10 ±  3%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>       0.70 ±  6%      +0.3        0.95 ±  6%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>       0.25 ±100%      +0.3        0.56 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
>       1.55 ±  2%      +0.3        1.87 ±  5%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file
>       1.55 ±  2%      +0.3        1.87 ±  5%  perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops
>       1.57 ±  2%      +0.3        1.89 ±  5%  perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
>       1.56 ±  2%      +0.3        1.88 ±  5%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter
>       1.56 ±  2%      +0.3        1.88 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
>       0.26 ±100%      +0.4        0.62 ±  6%  perf-profile.calltrace.cycles-pp.submit_one_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
>       0.26 ±100%      +0.4        0.62 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_submit_bio_hook.submit_one_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range
>       0.00            +0.6        0.56 ±  5%  perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
>       0.00            +0.6        0.56 ±  5%  perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio
>       0.00            +0.6        0.58 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio.extent_writepages.do_writepages
>       0.19 ±173%      +0.6        0.78 ± 27%  perf-profile.calltrace.cycles-pp.btrfs_next_old_leaf.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
>       0.00            +0.8        0.84 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook
>       0.00            +1.3        1.32 ±  6%  perf-profile.calltrace.cycles-pp.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
>       0.00            +1.4        1.39 ±  5%  perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
>       0.00            +1.4        1.39 ±  6%  perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
>       0.75 ±  4%      +1.5        2.23 ±  5%  perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
>       0.76 ±  3%      +1.5        2.24 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
>       0.75 ±  4%      +1.5        2.23 ±  5%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
>       0.75 ±  4%      +1.5        2.23 ±  5%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
>       0.38 ± 57%      +1.6        1.96 ±  6%  perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
>       0.38 ± 57%      +1.6        1.96 ±  5%  perf-profile.calltrace.cycles-pp.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
>       2.42 ±  2%      +1.8        4.18 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
>      38.62            +2.5       41.15 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write
>      39.95            +2.6       42.52 ±  3%  perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      39.95            +2.6       42.52 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64
>      39.99            +2.6       42.56 ±  3%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      39.98            +2.6       42.55 ±  3%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      40.19            +2.6       42.81 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
>      40.19            +2.6       42.81 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      97.75 ± 21%     +77.5%     173.50 ± 21%  interrupts.70:PCI-MSI.12589059-edge.eth3-TxRx-2
>      63.25 ±  9%     +51.4%      95.75 ± 25%  interrupts.71:PCI-MSI.12589060-edge.eth3-TxRx-3
>     202.50           +17.8%     238.50        interrupts.9:IO-APIC.9-fasteoi.acpi
>      88724           +12.8%     100056        interrupts.CAL:Function_call_interrupts
>       1255           +12.4%       1410        interrupts.CPU0.CAL:Function_call_interrupts
>     201136           +17.1%     235497 ±  2%  interrupts.CPU0.LOC:Local_timer_interrupts
>     202.50           +17.8%     238.50        interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
>       1242           +13.2%       1407        interrupts.CPU1.CAL:Function_call_interrupts
>     200860           +16.6%     234199 ±  2%  interrupts.CPU1.LOC:Local_timer_interrupts
>       3195 ± 24%     -60.2%       1273 ± 95%  interrupts.CPU1.NMI:Non-maskable_interrupts
>       3195 ± 24%     -60.2%       1273 ± 95%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
>     200768           +16.8%     234566        interrupts.CPU10.LOC:Local_timer_interrupts
>     200322           +17.1%     234482 ±  2%  interrupts.CPU11.LOC:Local_timer_interrupts
>       1174 ± 11%     +18.3%       1389        interrupts.CPU12.CAL:Function_call_interrupts
>     200965           +16.4%     233853        interrupts.CPU12.LOC:Local_timer_interrupts
>       1209 ±  4%     +15.4%       1396        interrupts.CPU13.CAL:Function_call_interrupts
>     199915           +17.6%     235085 ±  2%  interrupts.CPU13.LOC:Local_timer_interrupts
>       1187 ±  8%     +18.1%       1403        interrupts.CPU14.CAL:Function_call_interrupts
>     200677           +17.2%     235290 ±  2%  interrupts.CPU14.LOC:Local_timer_interrupts
>       1208 ±  5%     +13.2%       1367 ±  4%  interrupts.CPU15.CAL:Function_call_interrupts
>     200566           +17.3%     235226 ±  2%  interrupts.CPU15.LOC:Local_timer_interrupts
>     200800           +17.1%     235099 ±  2%  interrupts.CPU16.LOC:Local_timer_interrupts
>       1217 ±  4%     +15.3%       1403        interrupts.CPU17.CAL:Function_call_interrupts
>     200473           +16.6%     233816        interrupts.CPU17.LOC:Local_timer_interrupts
>       1246           +12.8%       1406        interrupts.CPU18.CAL:Function_call_interrupts
>     200863           +16.4%     233829 ±  3%  interrupts.CPU18.LOC:Local_timer_interrupts
>       1243           +12.7%       1400        interrupts.CPU19.CAL:Function_call_interrupts
>     200576           +16.0%     232732 ±  2%  interrupts.CPU19.LOC:Local_timer_interrupts
>       1247           +12.9%       1407        interrupts.CPU2.CAL:Function_call_interrupts
>     200971           +16.9%     234895 ±  2%  interrupts.CPU2.LOC:Local_timer_interrupts
>       1247           +12.7%       1406        interrupts.CPU20.CAL:Function_call_interrupts
>     200473           +16.6%     233803 ±  3%  interrupts.CPU20.LOC:Local_timer_interrupts
>       1247           +12.6%       1404        interrupts.CPU21.CAL:Function_call_interrupts
>     200696           +16.2%     233299 ±  2%  interrupts.CPU21.LOC:Local_timer_interrupts
>       1247           +13.1%       1411        interrupts.CPU22.CAL:Function_call_interrupts
>     200814           +16.4%     233758 ±  3%  interrupts.CPU22.LOC:Local_timer_interrupts
>       4891 ± 12%     -24.7%       3681 ± 12%  interrupts.CPU22.RES:Rescheduling_interrupts
>       1247           +13.0%       1410        interrupts.CPU23.CAL:Function_call_interrupts
>     200828           +16.4%     233716 ±  3%  interrupts.CPU23.LOC:Local_timer_interrupts
>       1247           +14.3%       1426 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
>     200833           +16.5%     233997 ±  3%  interrupts.CPU24.LOC:Local_timer_interrupts
>       1246           +12.9%       1407        interrupts.CPU25.CAL:Function_call_interrupts
>     201055           +16.3%     233920 ±  3%  interrupts.CPU25.LOC:Local_timer_interrupts
>       1265 ±  2%     +12.7%       1425 ±  2%  interrupts.CPU26.CAL:Function_call_interrupts
>     201132           +15.9%     233180 ±  3%  interrupts.CPU26.LOC:Local_timer_interrupts
>       1173 ± 10%     +20.0%       1407        interrupts.CPU27.CAL:Function_call_interrupts
>     201166           +15.9%     233210 ±  3%  interrupts.CPU27.LOC:Local_timer_interrupts
>       1263 ±  2%     +11.4%       1407        interrupts.CPU28.CAL:Function_call_interrupts
>     201049           +16.3%     233848 ±  3%  interrupts.CPU28.LOC:Local_timer_interrupts
>       1241           +13.5%       1409        interrupts.CPU29.CAL:Function_call_interrupts
>     200963           +16.3%     233774 ±  3%  interrupts.CPU29.LOC:Local_timer_interrupts
>      97.75 ± 21%     +77.5%     173.50 ± 21%  interrupts.CPU3.70:PCI-MSI.12589059-edge.eth3-TxRx-2
>       1210 ±  5%     +13.2%       1369 ±  3%  interrupts.CPU3.CAL:Function_call_interrupts
>     199954           +17.1%     234167 ±  2%  interrupts.CPU3.LOC:Local_timer_interrupts
>       1181 ±  9%     +19.1%       1407        interrupts.CPU30.CAL:Function_call_interrupts
>     200986           +16.4%     233892 ±  3%  interrupts.CPU30.LOC:Local_timer_interrupts
>       1127 ± 18%     +24.7%       1406        interrupts.CPU31.CAL:Function_call_interrupts
>     201110           +16.2%     233756 ±  3%  interrupts.CPU31.LOC:Local_timer_interrupts
>       1249           +13.0%       1412        interrupts.CPU32.CAL:Function_call_interrupts
>     200832           +16.4%     233787 ±  3%  interrupts.CPU32.LOC:Local_timer_interrupts
>       1249           +13.3%       1416        interrupts.CPU33.CAL:Function_call_interrupts
>     200969           +16.3%     233758 ±  3%  interrupts.CPU33.LOC:Local_timer_interrupts
>       1246           +13.4%       1413        interrupts.CPU34.CAL:Function_call_interrupts
>     199891           +17.1%     234012 ±  3%  interrupts.CPU34.LOC:Local_timer_interrupts
>       1234           +13.3%       1399        interrupts.CPU35.CAL:Function_call_interrupts
>     200721           +16.2%     233256 ±  2%  interrupts.CPU35.LOC:Local_timer_interrupts
>       3994 ± 14%     -50.7%       1970 ± 48%  interrupts.CPU35.RES:Rescheduling_interrupts
>       1242           +12.9%       1402 ±  2%  interrupts.CPU36.CAL:Function_call_interrupts
>     199765           +17.7%     235071 ±  2%  interrupts.CPU36.LOC:Local_timer_interrupts
>     200843           +16.6%     234175 ±  2%  interrupts.CPU37.LOC:Local_timer_interrupts
>       1248           +13.4%       1415        interrupts.CPU38.CAL:Function_call_interrupts
>     200016           +17.0%     234095 ±  2%  interrupts.CPU38.LOC:Local_timer_interrupts
>       1249           +13.0%       1411        interrupts.CPU39.CAL:Function_call_interrupts
>     200714           +17.2%     235250 ±  2%  interrupts.CPU39.LOC:Local_timer_interrupts
>      63.25 ±  9%     +51.4%      95.75 ± 25%  interrupts.CPU4.71:PCI-MSI.12589060-edge.eth3-TxRx-3
>       1246           +13.9%       1419 ±  2%  interrupts.CPU4.CAL:Function_call_interrupts
>     200637           +17.2%     235179 ±  2%  interrupts.CPU4.LOC:Local_timer_interrupts
>     200593           +17.2%     235093 ±  2%  interrupts.CPU40.LOC:Local_timer_interrupts
>       1248           +13.0%       1410        interrupts.CPU41.CAL:Function_call_interrupts
>     200998           +16.6%     234380        interrupts.CPU41.LOC:Local_timer_interrupts
>       1249           +12.9%       1410        interrupts.CPU42.CAL:Function_call_interrupts
>     200934           +16.6%     234357        interrupts.CPU42.LOC:Local_timer_interrupts
>       1246           +13.2%       1411        interrupts.CPU43.CAL:Function_call_interrupts
>     200894           +16.9%     234942 ±  2%  interrupts.CPU43.LOC:Local_timer_interrupts
>       1211 ±  5%     +16.3%       1408        interrupts.CPU44.CAL:Function_call_interrupts
>     200154           +17.4%     235070 ±  2%  interrupts.CPU44.LOC:Local_timer_interrupts
>       2454 ± 23%     +50.0%       3682 ± 14%  interrupts.CPU44.RES:Rescheduling_interrupts
>       1248           +13.0%       1410        interrupts.CPU45.CAL:Function_call_interrupts
>     200774           +16.7%     234259 ±  2%  interrupts.CPU45.LOC:Local_timer_interrupts
>       1246           +13.1%       1410        interrupts.CPU46.CAL:Function_call_interrupts
>     201184           +16.8%     234966 ±  2%  interrupts.CPU46.LOC:Local_timer_interrupts
>       1246           +13.1%       1409        interrupts.CPU47.CAL:Function_call_interrupts
>     200672           +16.7%     234200        interrupts.CPU47.LOC:Local_timer_interrupts
>     200070           +17.7%     235463 ±  2%  interrupts.CPU48.LOC:Local_timer_interrupts
>       1251           +12.6%       1409        interrupts.CPU49.CAL:Function_call_interrupts
>     200232           +17.4%     235071 ±  2%  interrupts.CPU49.LOC:Local_timer_interrupts
>     200450           +17.4%     235328 ±  2%  interrupts.CPU5.LOC:Local_timer_interrupts
>       1216 ±  5%     +15.7%       1407        interrupts.CPU50.CAL:Function_call_interrupts
>     201051           +16.5%     234316        interrupts.CPU50.LOC:Local_timer_interrupts
>       1252           +12.6%       1410        interrupts.CPU51.CAL:Function_call_interrupts
>     200007           +17.5%     235046 ±  2%  interrupts.CPU51.LOC:Local_timer_interrupts
>       1253           +12.4%       1408        interrupts.CPU52.CAL:Function_call_interrupts
>     200770           +17.0%     234811 ±  2%  interrupts.CPU52.LOC:Local_timer_interrupts
>       1252           +12.2%       1405        interrupts.CPU53.CAL:Function_call_interrupts
>     200095           +17.4%     235004 ±  2%  interrupts.CPU53.LOC:Local_timer_interrupts
>       1252           +12.6%       1410        interrupts.CPU54.CAL:Function_call_interrupts
>     200697           +16.5%     233818 ±  3%  interrupts.CPU54.LOC:Local_timer_interrupts
>       1252           +12.6%       1409        interrupts.CPU55.CAL:Function_call_interrupts
>     200852           +16.5%     234024 ±  3%  interrupts.CPU55.LOC:Local_timer_interrupts
>       1251           +12.5%       1407        interrupts.CPU56.CAL:Function_call_interrupts
>     200874           +16.4%     233828 ±  3%  interrupts.CPU56.LOC:Local_timer_interrupts
>       1251           +12.3%       1404        interrupts.CPU57.CAL:Function_call_interrupts
>     200887           +16.5%     233974 ±  3%  interrupts.CPU57.LOC:Local_timer_interrupts
>       1250           +12.8%       1411        interrupts.CPU58.CAL:Function_call_interrupts
>     200837           +16.4%     233702 ±  3%  interrupts.CPU58.LOC:Local_timer_interrupts
>       1250           +12.8%       1410        interrupts.CPU59.CAL:Function_call_interrupts
>     199809           +17.1%     233932 ±  3%  interrupts.CPU59.LOC:Local_timer_interrupts
>       1262 ±  2%     +10.5%       1394        interrupts.CPU6.CAL:Function_call_interrupts
>     201108           +16.6%     234545        interrupts.CPU6.LOC:Local_timer_interrupts
>       1253           +12.6%       1411        interrupts.CPU60.CAL:Function_call_interrupts
>     200941           +16.4%     233857 ±  3%  interrupts.CPU60.LOC:Local_timer_interrupts
>       2285 ± 53%     -55.3%       1021 ± 85%  interrupts.CPU60.NMI:Non-maskable_interrupts
>       2285 ± 53%     -55.3%       1021 ± 85%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
>       1213 ±  4%     +16.2%       1410        interrupts.CPU61.CAL:Function_call_interrupts
>     200849           +16.4%     233812 ±  3%  interrupts.CPU61.LOC:Local_timer_interrupts
>       1250           +12.7%       1408        interrupts.CPU62.CAL:Function_call_interrupts
>     200858           +16.4%     233847 ±  3%  interrupts.CPU62.LOC:Local_timer_interrupts
>       1249           +12.7%       1408        interrupts.CPU63.CAL:Function_call_interrupts
>     200846           +16.3%     233509 ±  3%  interrupts.CPU63.LOC:Local_timer_interrupts
>       1249           +12.6%       1407        interrupts.CPU64.CAL:Function_call_interrupts
>     201010           +16.3%     233868 ±  3%  interrupts.CPU64.LOC:Local_timer_interrupts
>       1249           +12.6%       1407        interrupts.CPU65.CAL:Function_call_interrupts
>     201188           +16.2%     233840 ±  3%  interrupts.CPU65.LOC:Local_timer_interrupts
>       1249           +12.6%       1405        interrupts.CPU66.CAL:Function_call_interrupts
>     201142           +16.3%     233885 ±  3%  interrupts.CPU66.LOC:Local_timer_interrupts
>       1249           +12.6%       1406        interrupts.CPU67.CAL:Function_call_interrupts
>     200840           +16.5%     233913 ±  3%  interrupts.CPU67.LOC:Local_timer_interrupts
>       1249           +12.7%       1408        interrupts.CPU68.CAL:Function_call_interrupts
>     200815           +16.4%     233774 ±  3%  interrupts.CPU68.LOC:Local_timer_interrupts
>       1249           +12.8%       1409        interrupts.CPU69.CAL:Function_call_interrupts
>     200983           +16.3%     233829 ±  3%  interrupts.CPU69.LOC:Local_timer_interrupts
>     200912           +17.1%     235169 ±  2%  interrupts.CPU7.LOC:Local_timer_interrupts
>       1249           +13.0%       1412        interrupts.CPU70.CAL:Function_call_interrupts
>     200821           +16.4%     233823 ±  3%  interrupts.CPU70.LOC:Local_timer_interrupts
>       1247           +12.9%       1408        interrupts.CPU71.CAL:Function_call_interrupts
>     200673           +16.4%     233586 ±  3%  interrupts.CPU71.LOC:Local_timer_interrupts
>       3112 ± 28%     -55.8%       1376 ± 92%  interrupts.CPU71.NMI:Non-maskable_interrupts
>       3112 ± 28%     -55.8%       1376 ± 92%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
>       2655 ± 18%     +53.3%       4071 ± 21%  interrupts.CPU71.RES:Rescheduling_interrupts
>       1088 ± 16%     +25.1%       1361 ±  4%  interrupts.CPU8.CAL:Function_call_interrupts
>     201291           +16.9%     235317 ±  2%  interrupts.CPU8.LOC:Local_timer_interrupts
>     201246           +16.4%     234250        interrupts.CPU9.LOC:Local_timer_interrupts
>   14452726           +16.7%   16865906 ±  2%  interrupts.LOC:Local_timer_interrupts
>      25553 ±  7%     +26.4%      32295        softirqs.CPU0.RCU
>      17578           +14.6%      20147 ±  2%  softirqs.CPU0.SCHED
>      40851           +20.4%      49167 ±  3%  softirqs.CPU0.TIMER
>      40546           +17.0%      47455 ±  5%  softirqs.CPU1.TIMER
>      25814           +27.1%      32807 ± 10%  softirqs.CPU10.RCU
>      38717 ±  2%     +14.6%      44386 ±  5%  softirqs.CPU10.TIMER
>      38245           +16.3%      44468 ±  5%  softirqs.CPU11.TIMER
>      26439 ±  4%     +21.3%      32083 ±  4%  softirqs.CPU12.RCU
>      15582 ±  4%     +14.4%      17827 ±  2%  softirqs.CPU12.SCHED
>      38686           +17.5%      45474 ±  4%  softirqs.CPU12.TIMER
>      25962 ±  4%     +18.3%      30720 ±  4%  softirqs.CPU13.RCU
>      16126 ±  5%     +17.9%      19013 ±  5%  softirqs.CPU13.SCHED
>      25846 ±  2%     +20.4%      31108 ±  5%  softirqs.CPU14.RCU
>      15629 ±  4%     +21.3%      18953 ±  6%  softirqs.CPU14.SCHED
>      37591           +17.9%      44326 ±  5%  softirqs.CPU14.TIMER
>      24676 ±  2%     +20.2%      29658        softirqs.CPU15.RCU
>      38217           +16.0%      44331 ±  4%  softirqs.CPU15.TIMER
>      24680 ±  2%     +18.6%      29278 ±  5%  softirqs.CPU16.RCU
>      38783           +17.9%      45709 ±  4%  softirqs.CPU16.TIMER
>      24738           +20.6%      29833 ± 10%  softirqs.CPU17.RCU
>      39088           +16.3%      45476 ±  5%  softirqs.CPU17.TIMER
>      27971           +24.7%      34880 ±  4%  softirqs.CPU18.RCU
>      27800           +19.7%      33276 ±  2%  softirqs.CPU19.RCU
>      16069 ±  5%     +14.6%      18417 ±  3%  softirqs.CPU19.SCHED
>      39339           +18.2%      46516 ±  5%  softirqs.CPU19.TIMER
>      26084           +10.4%      28804 ±  5%  softirqs.CPU2.RCU
>      16601 ±  8%     +15.0%      19085 ± 10%  softirqs.CPU2.SCHED
>      38625           +22.4%      47291 ±  5%  softirqs.CPU2.TIMER
>      27472           +27.0%      34877 ±  7%  softirqs.CPU20.RCU
>      16088 ±  3%     +10.7%      17811 ±  4%  softirqs.CPU20.SCHED
>      38845           +17.6%      45674 ±  3%  softirqs.CPU20.TIMER
>      28205 ±  6%     +20.2%      33906 ±  7%  softirqs.CPU21.RCU
>      39324           +13.8%      44753 ±  4%  softirqs.CPU21.TIMER
>      27282 ±  9%     +20.0%      32736 ±  4%  softirqs.CPU22.RCU
>      39542           +13.9%      45038 ±  3%  softirqs.CPU22.TIMER
>      27295 ±  7%     +22.3%      33387 ±  2%  softirqs.CPU23.RCU
>      15694 ±  3%     +13.2%      17772 ±  3%  softirqs.CPU23.SCHED
>      37452           +16.5%      43645 ±  5%  softirqs.CPU23.TIMER
>      25529 ± 10%     +30.1%      33204        softirqs.CPU24.RCU
>      16068 ±  6%     +13.8%      18289 ±  4%  softirqs.CPU24.SCHED
>      39129           +29.4%      50648 ± 17%  softirqs.CPU24.TIMER
>      27444 ±  3%     +18.3%      32463 ±  4%  softirqs.CPU25.RCU
>      38658           +14.6%      44286 ±  6%  softirqs.CPU25.TIMER
>      27586 ±  2%     +21.4%      33496        softirqs.CPU26.RCU
>      39631           +13.9%      45136 ±  4%  softirqs.CPU26.TIMER
>      27618 ±  3%     +15.9%      32011 ±  2%  softirqs.CPU27.RCU
>      26884           +24.1%      33355 ±  7%  softirqs.CPU28.RCU
>      39120           +14.6%      44813 ±  5%  softirqs.CPU28.TIMER
>      27542 ±  4%     +21.4%      33433 ±  5%  softirqs.CPU29.RCU
>      38659 ±  2%     +15.9%      44815 ±  5%  softirqs.CPU29.TIMER
>      27539 ±  3%     +14.2%      31446        softirqs.CPU3.RCU
>      16465 ±  4%     +16.6%      19192 ±  6%  softirqs.CPU3.SCHED
>      38875           +17.9%      45849 ±  5%  softirqs.CPU3.TIMER
>      26139           +23.5%      32277 ±  4%  softirqs.CPU30.RCU
>      39666           +11.6%      44285 ±  5%  softirqs.CPU30.TIMER
>      26559 ±  2%     +23.7%      32855 ±  8%  softirqs.CPU31.RCU
>      16278 ±  2%     +19.3%      19420 ± 10%  softirqs.CPU31.SCHED
>      39178           +15.2%      45121 ±  4%  softirqs.CPU31.TIMER
>      25781           +20.9%      31163 ±  2%  softirqs.CPU32.RCU
>      37670           +13.6%      42811 ±  5%  softirqs.CPU32.TIMER
>      27107 ±  5%     +16.7%      31625 ±  4%  softirqs.CPU33.RCU
>      16098 ±  2%     +12.3%      18071 ±  2%  softirqs.CPU33.SCHED
>      38565           +15.6%      44572 ±  4%  softirqs.CPU33.TIMER
>      26241 ±  3%     +18.9%      31209 ±  5%  softirqs.CPU34.RCU
>      38223           +18.1%      45160 ±  5%  softirqs.CPU34.TIMER
>      25360 ±  6%     +27.0%      32210 ±  2%  softirqs.CPU35.RCU
>      39986 ±  2%     +13.8%      45494 ±  3%  softirqs.CPU35.TIMER
>      26626 ±  2%     +23.9%      32987 ±  4%  softirqs.CPU36.RCU
>      16281 ±  5%     +12.6%      18329        softirqs.CPU36.SCHED
>      27227           +23.8%      33703 ±  2%  softirqs.CPU37.RCU
>      26701 ±  2%     +29.1%      34462 ±  8%  softirqs.CPU38.RCU
>      15157 ±  5%     +15.5%      17507 ±  4%  softirqs.CPU38.SCHED
>      37119 ±  4%     +37.9%      51191 ± 27%  softirqs.CPU38.TIMER
>      28007 ±  2%     +21.4%      33990 ±  2%  softirqs.CPU39.RCU
>      15700 ±  2%     +11.8%      17545 ±  3%  softirqs.CPU39.SCHED
>      37509 ±  2%     +14.9%      43109 ±  6%  softirqs.CPU39.TIMER
>      25877           +24.6%      32254 ±  3%  softirqs.CPU4.RCU
>      38648           +16.9%      45191 ±  5%  softirqs.CPU4.TIMER
>      27903           +17.8%      32867 ±  2%  softirqs.CPU40.RCU
>      15256 ±  2%     +12.8%      17202 ±  7%  softirqs.CPU40.SCHED
>      36371 ±  2%     +17.0%      42572 ±  4%  softirqs.CPU40.TIMER
>      26560 ±  5%     +19.9%      31840 ±  6%  softirqs.CPU41.RCU
>      15067 ±  3%     +13.8%      17142 ±  4%  softirqs.CPU41.SCHED
>      36473 ±  3%     +14.2%      41657 ±  4%  softirqs.CPU41.TIMER
>      27250           +22.5%      33387 ±  2%  softirqs.CPU42.RCU
>      15376 ±  4%     +15.4%      17744 ±  3%  softirqs.CPU42.SCHED
>      37838           +13.4%      42895 ±  3%  softirqs.CPU42.TIMER
>      27137 ±  4%     +23.2%      33425        softirqs.CPU43.RCU
>      27818 ±  3%     +22.5%      34073 ±  6%  softirqs.CPU44.RCU
>      14305 ±  4%     +22.5%      17522 ±  2%  softirqs.CPU44.SCHED
>      36151 ±  4%     +16.7%      42187 ±  4%  softirqs.CPU44.TIMER
>      25734 ±  3%     +25.6%      32319 ±  4%  softirqs.CPU45.RCU
>      14522           +14.8%      16674 ±  6%  softirqs.CPU45.SCHED
>      36594           +15.8%      42364 ±  5%  softirqs.CPU45.TIMER
>      28660 ±  3%     +18.8%      34052 ±  7%  softirqs.CPU46.RCU
>      15306 ±  4%     +13.5%      17370 ±  7%  softirqs.CPU46.SCHED
>      37095 ±  3%     +14.3%      42388 ±  5%  softirqs.CPU46.TIMER
>      28906 ±  5%     +15.3%      33324 ±  7%  softirqs.CPU47.RCU
>      14758 ±  5%     +13.7%      16775 ±  5%  softirqs.CPU47.SCHED
>      36736 ±  3%     +14.2%      41969 ±  4%  softirqs.CPU47.TIMER
>      28049 ±  4%     +24.0%      34791 ±  2%  softirqs.CPU48.RCU
>      14583 ±  4%     +18.9%      17340 ±  8%  softirqs.CPU48.SCHED
>      36095 ±  4%     +19.5%      43117 ±  6%  softirqs.CPU48.TIMER
>      27551 ±  5%     +21.0%      33349 ±  4%  softirqs.CPU49.RCU
>      36024           +18.1%      42545 ±  4%  softirqs.CPU49.TIMER
>      25870 ±  2%     +22.4%      31677 ±  3%  softirqs.CPU5.RCU
>      15740 ±  6%     +26.8%      19951 ±  9%  softirqs.CPU5.SCHED
>      37622           +33.5%      50234 ± 18%  softirqs.CPU5.TIMER
>      27424           +19.6%      32798 ±  2%  softirqs.CPU50.RCU
>      14010 ±  2%     +21.8%      17060 ±  6%  softirqs.CPU50.SCHED
>      36485 ±  3%     +14.7%      41862 ±  4%  softirqs.CPU50.TIMER
>      27884           +18.9%      33160 ±  3%  softirqs.CPU51.RCU
>      36212 ±  3%     +16.6%      42210 ±  4%  softirqs.CPU51.TIMER
>      27031 ±  7%     +29.8%      35075 ±  5%  softirqs.CPU52.RCU
>      14772 ±  2%     +18.1%      17451 ±  4%  softirqs.CPU52.SCHED
>      37031 ±  4%     +17.9%      43648 ±  4%  softirqs.CPU52.TIMER
>      25188 ± 12%     +28.7%      32423 ±  3%  softirqs.CPU53.RCU
>      16289 ±  6%     +18.4%      19280 ±  5%  softirqs.CPU53.SCHED
>      37616 ±  3%     +35.6%      51010 ± 21%  softirqs.CPU53.TIMER
>      26693 ±  3%     +19.4%      31870 ±  2%  softirqs.CPU54.RCU
>      38094           +16.5%      44379 ±  4%  softirqs.CPU54.TIMER
>      26082 ±  2%     +23.1%      32100        softirqs.CPU55.RCU
>      37589 ±  2%     +16.5%      43779 ±  5%  softirqs.CPU55.TIMER
>      26352 ±  2%     +18.7%      31284        softirqs.CPU56.RCU
>      15591 ±  4%     +15.9%      18066 ±  8%  softirqs.CPU56.SCHED
>      37070           +16.8%      43309 ±  2%  softirqs.CPU56.TIMER
>      27520 ±  4%     +15.7%      31841 ±  5%  softirqs.CPU57.RCU
>      37916 ±  2%     +13.9%      43170 ±  2%  softirqs.CPU57.TIMER
>      27067           +20.8%      32709 ±  7%  softirqs.CPU58.RCU
>      15059           +19.1%      17935 ±  8%  softirqs.CPU58.SCHED
>      37123           +13.8%      42241        softirqs.CPU58.TIMER
>      25199 ±  4%     +19.6%      30126 ±  6%  softirqs.CPU59.RCU
>      36384 ±  2%     +16.8%      42479 ±  2%  softirqs.CPU59.TIMER
>      25871 ±  3%     +17.5%      30409 ±  3%  softirqs.CPU6.RCU
>      16584 ±  2%     +14.8%      19032 ±  5%  softirqs.CPU6.SCHED
>      38784           +15.6%      44850 ±  4%  softirqs.CPU6.TIMER
>      24777 ±  5%     +15.0%      28502 ±  5%  softirqs.CPU60.RCU
>      37468           +14.2%      42794 ±  6%  softirqs.CPU60.TIMER
>      23108 ±  3%     +30.1%      30065 ±  2%  softirqs.CPU61.RCU
>      15059 ±  6%     +16.5%      17540 ±  6%  softirqs.CPU61.SCHED
>      37002           +16.8%      43227 ±  3%  softirqs.CPU61.TIMER
>      23023 ±  7%     +33.6%      30769 ±  6%  softirqs.CPU62.RCU
>      37164 ±  2%     +16.2%      43203 ±  2%  softirqs.CPU62.TIMER
>      23069 ±  2%     +25.1%      28850        softirqs.CPU63.RCU
>      14642 ±  5%     +15.4%      16900 ±  8%  softirqs.CPU63.SCHED
>      36254 ±  2%     +15.4%      41854 ±  3%  softirqs.CPU63.TIMER
>      24304 ±  2%     +25.9%      30593 ±  2%  softirqs.CPU64.RCU
>      14666           +15.5%      16938 ±  3%  softirqs.CPU64.SCHED
>      37048           +15.7%      42877 ±  4%  softirqs.CPU64.TIMER
>      24438 ±  9%     +23.6%      30212 ±  5%  softirqs.CPU65.RCU
>      37505           +12.8%      42288 ±  2%  softirqs.CPU65.TIMER
>      24568           +20.4%      29569        softirqs.CPU66.RCU
>      15290 ±  5%      +8.4%      16569 ±  4%  softirqs.CPU66.SCHED
>      37381           +14.9%      42968 ±  3%  softirqs.CPU66.TIMER
>      23253 ±  9%     +29.7%      30150 ±  2%  softirqs.CPU67.RCU
>      14645 ±  6%     +13.4%      16613 ±  6%  softirqs.CPU67.SCHED
>      37012           +14.1%      42221 ±  2%  softirqs.CPU67.TIMER
>      36993           +15.0%      42536 ±  4%  softirqs.CPU68.TIMER
>      24695 ±  7%     +23.7%      30551 ±  3%  softirqs.CPU69.RCU
>      14441 ±  3%     +11.4%      16089 ±  4%  softirqs.CPU69.SCHED
>      36999           +14.3%      42299        softirqs.CPU69.TIMER
>      26022 ±  3%     +21.4%      31579 ±  2%  softirqs.CPU7.RCU
>      38481           +18.7%      45661 ±  3%  softirqs.CPU7.TIMER
>      25083 ±  8%     +18.5%      29721 ±  2%  softirqs.CPU70.RCU
>      14845 ±  7%     +15.7%      17173 ±  4%  softirqs.CPU70.SCHED
>      37094           +18.5%      43948 ±  3%  softirqs.CPU70.TIMER
>      24438 ±  2%     +20.6%      29461 ±  2%  softirqs.CPU71.RCU
>      15821 ±  4%     +13.6%      17978 ±  4%  softirqs.CPU71.SCHED
>      26912 ±  7%     +22.3%      32925 ±  6%  softirqs.CPU8.RCU
>      39193           +15.2%      45167 ±  4%  softirqs.CPU8.TIMER
>      25829 ±  2%     +20.7%      31178        softirqs.CPU9.RCU
>      37434 ±  2%     +16.6%      43665 ±  5%  softirqs.CPU9.TIMER
>    1897274           +21.5%    2304570 ±  2%  softirqs.RCU
>    1150548           +11.9%    1287616        softirqs.SCHED
>    2757329           +15.8%    3193002 ±  3%  softirqs.TIMER
> 
> 
>                                                                                 
>                                  aim7.jobs-per-min                              
>                                                                                 
>   1400 +-+------------------------------------------------------------------+   
>        |                                                                    |   
>   1200 +-+++.++.++.+++.++.++  +  ++.+  +  +++.++  +  ++.++.++.++.+++.++.++.+|   
>        |                   :  :  :  :  :  :    :  :  :                      |   
>   1000 OO+OO OO OO OOO OO OO OOO OO OO OO OOO OO OO OOO O                   |   
>        |                    : :  :  :  :  :     : :: :                      |   
>    800 +-+                  : : :   : :: :      : :: :                      |   
>        |                    : : :   : :: :      : :: :                      |   
>    600 +-+                  : :::    :: ::      : :: :                      |   
>        |                    :: ::    :: ::      :: ::                       |   
>    400 +-+                  :: ::    :: ::      :: ::                       |   
>        |                     : ::    :: ::       :  :                       |   
>    200 +-+                   : :     :  :        :  :                       |   
>        |                     : :     :  :        :  :                       |   
>      0 +-+------------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                               aim7.time.elapsed_time                            
>                                                                                 
>   140 +-+-------------------------------------------------------------------+   
>       |                                                                     |   
>   120 OO+OO OO OO OO OO O O OO OO  O OO OOO OO OO OO OO O                   |   
>       |                  O        O                                         |   
>   100 +-+++.++.++.++.++.+++  +  +.++  +  ++.++.+  +  ++.++.+++.++.++.++.++.+|   
>       |                   :  :  :  :  :  :     :  :  :                      |   
>    80 +-+                 :  :  :  :  :  :     :  :  :                      |   
>       |                    : :: :   : :: :     : :: :                       |   
>    60 +-+                  : :: :   : :: :     : :: :                       |   
>       |                    : :: :   : :: :      :: ::                       |   
>    40 +-+                  :: ::    :: ::       :: ::                       |   
>       |                    :: ::    :: ::       :: ::                       |   
>    20 +-+                   :  :     :  :       :  :                        |   
>       |                     :  :     :  :       :  :                        |   
>     0 +-+-------------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                             aim7.time.elapsed_time.max                          
>                                                                                 
>   140 +-+-------------------------------------------------------------------+   
>       |                                                                     |   
>   120 OO+OO OO OO OO OO O O OO OO  O OO OOO OO OO OO OO O                   |   
>       |                  O        O                                         |   
>   100 +-+++.++.++.++.++.+++  +  +.++  +  ++.++.+  +  ++.++.+++.++.++.++.++.+|   
>       |                   :  :  :  :  :  :     :  :  :                      |   
>    80 +-+                 :  :  :  :  :  :     :  :  :                      |   
>       |                    : :: :   : :: :     : :: :                       |   
>    60 +-+                  : :: :   : :: :     : :: :                       |   
>       |                    : :: :   : :: :      :: ::                       |   
>    40 +-+                  :: ::    :: ::       :: ::                       |   
>       |                    :: ::    :: ::       :: ::                       |   
>    20 +-+                   :  :     :  :       :  :                        |   
>       |                     :  :     :  :       :  :                        |   
>     0 +-+-------------------------------------------------------------------+   
>                                                                                 
>                                                                                 
> [*] bisect-good sample
> [O] bisect-bad  sample
> 
> ***************************************************************************************************
> lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
> =========================================================================================
> compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
>   gcc-8/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/sync_disk_rw/aim7
> 
> commit: 
>   a92e9c3a3c ("btrfs: extent_io: Handle error better in extent_writepages()")
>   44fe89de7d ("btrfs: Do mandatory tree block check before submitting bio")
> 
> a92e9c3a3cef4c0c 44fe89de7d5157a4f31f13d9480 
> ---------------- --------------------------- 
>          %stddev     %change         %stddev
>              \          |                \  
>     474.71           -20.8%     376.09        aim7.jobs-per-min
>     252.89           +26.2%     319.19        aim7.time.elapsed_time
>     252.89           +26.2%     319.19        aim7.time.elapsed_time.max
>       8376 ±  3%      +9.4%       9167 ±  3%  aim7.time.minor_page_faults
>       1760            +6.3%       1872        aim7.time.system_time
>      22.58            -1.5%      22.24        boot-time.dhcp
>      40945 ±  2%     -22.3%      31813 ±  3%  meminfo.max_used_kB
>     670014           +24.2%     832471        proc-vmstat.pgfault
>    1013780           +13.7%    1152374        proc-vmstat.pgfree
>  3.022e+09 ±  4%     +50.2%   4.54e+09 ± 19%  cpuidle.C1.time
>   34330913 ±  2%     +16.1%   39848597 ±  9%  cpuidle.C1.usage
>  2.611e+09 ± 16%     +48.9%  3.887e+09 ±  3%  cpuidle.C6.time
>       0.02 ± 15%      -0.0        0.00 ±113%  mpstat.cpu.-1.soft%
>      21.36            -3.4       17.98        mpstat.cpu.-1.sys%
>       0.14            -0.0        0.12        mpstat.cpu.-1.usr%
>      78.00            +4.8%      81.75        vmstat.cpu.id
>     138672           -21.1%     109467        vmstat.io.bo
>     299636           -21.0%     236781        vmstat.system.cs
>      78.64            +4.3%      81.99        iostat.cpu.idle
>      21.21           -15.7%      17.88        iostat.cpu.system
>       7947           -20.5%       6318        iostat.md0.w/s
>     180043           -20.9%     142377        iostat.md0.wkB/s
>       3098 ± 30%    +270.4%      11475 ± 34%  numa-meminfo.node0.Inactive(anon)
>       9526           +37.0%      13052 ± 15%  numa-meminfo.node0.Mapped
>       5474 ± 26%    +188.7%      15807 ± 23%  numa-meminfo.node0.Shmem
>      75040 ±  2%     +10.4%      82807 ±  7%  numa-meminfo.node0.Slab
>      16192 ±  5%     -51.6%       7838 ± 51%  numa-meminfo.node1.Inactive(anon)
>      20174 ±  7%     -50.1%      10076 ± 36%  numa-meminfo.node1.Shmem
>     774.25 ± 30%    +270.4%       2867 ± 34%  numa-vmstat.node0.nr_inactive_anon
>       2474 ±  2%     +35.6%       3354 ± 14%  numa-vmstat.node0.nr_mapped
>       1367 ± 26%    +188.9%       3950 ± 23%  numa-vmstat.node0.nr_shmem
>     774.25 ± 30%    +270.4%       2867 ± 34%  numa-vmstat.node0.nr_zone_inactive_anon
>       4046 ±  5%     -51.6%       1958 ± 51%  numa-vmstat.node1.nr_inactive_anon
>       5042 ±  7%     -50.1%       2518 ± 36%  numa-vmstat.node1.nr_shmem
>       4046 ±  5%     -51.6%       1958 ± 51%  numa-vmstat.node1.nr_zone_inactive_anon
>     312.25           -15.1%     265.25        turbostat.Avg_MHz
>      25.87            -3.9       22.02        turbostat.Busy%
>   34326534 ±  2%     +16.1%   39845190 ±  9%  turbostat.C1
>      33.45            -4.6%      31.92        turbostat.CorWatt
>   21163841           +25.1%   26466542        turbostat.IRQ
>      59.45            -2.7%      57.87        turbostat.PkgWatt
>      32.25            -1.7%      31.71        turbostat.RAMWatt
>      20330           +26.1%      25630        turbostat.SMI
>       6.46 ±  2%      -0.6        5.89 ±  3%  perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
>       2.75 ±  5%      -0.6        2.19 ± 11%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
>       5.46 ±  2%      -0.5        4.92 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
>       5.46 ±  2%      -0.5        4.93 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
>       0.69 ±  9%      -0.4        0.27 ±100%  perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.__vfs_write.vfs_write
>       1.74 ±  6%      -0.4        1.39 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write
>       0.63 ±  3%      -0.3        0.30 ±100%  perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio.extent_writepages
>       2.35 ±  4%      -0.2        2.13 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents
>       0.92 ±  9%      -0.1        0.80 ±  7%  perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
>       0.86 ± 10%      -0.1        0.74 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages
>       0.85 ±  9%      -0.1        0.73 ±  8%  perf-profile.calltrace.cycles-pp.cow_file_range.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
>       0.70 ±  5%      -0.1        0.57 ±  3%  perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
>       1.40 ±  5%      +0.2        1.55        perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>       0.65 ± 11%      +0.3        0.94 ± 23%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
>       0.79 ±  8%      +0.3        1.13 ± 21%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
>       0.26 ±100%      +0.3        0.61 ±  7%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
>       1.44 ±  8%      +0.4        1.87 ± 17%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
>       0.14 ±173%      +0.6        0.70 ± 27%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
>       2.44 ±  5%      +0.6        3.04 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
>       0.00            +0.7        0.72 ±  2%  perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook
>       2.51 ±  6%      +0.8        3.30 ± 16%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
>       2.71 ±  6%      +0.8        3.53 ± 16%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
>       0.73 ±  6%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
>       0.72 ±  6%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
>       0.72 ±  6%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
>       0.72 ±  6%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
>       0.00            +1.0        1.05        perf-profile.calltrace.cycles-pp.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
>       0.00            +1.1        1.08        perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
>       0.00            +1.1        1.09        perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
>       0.00            +1.5        1.46        perf-profile.calltrace.cycles-pp.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
>       0.00            +1.5        1.46        perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
>      71.60 ±  7%     -16.2%      59.98 ±  8%  sched_debug.cfs_rq:/.load_avg.avg
>      96.11 ± 10%     -16.2%      80.58 ±  6%  sched_debug.cfs_rq:/.load_avg.stddev
>     187997            -8.8%     171461 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
>     226472 ±  2%     -10.8%     202016 ±  6%  sched_debug.cfs_rq:/.min_vruntime.max
>      17261 ±  7%     -21.5%      13550 ± 15%  sched_debug.cfs_rq:/.min_vruntime.stddev
>       8.89 ± 47%     -64.7%       3.14 ± 57%  sched_debug.cfs_rq:/.removed.load_avg.avg
>     204.15           -38.5%     125.58 ± 57%  sched_debug.cfs_rq:/.removed.load_avg.max
>      40.32 ± 22%     -51.4%      19.61 ± 57%  sched_debug.cfs_rq:/.removed.load_avg.stddev
>     409.91 ± 47%     -64.8%     144.41 ± 57%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
>       9437           -38.8%       5776 ± 57%  sched_debug.cfs_rq:/.removed.runnable_sum.max
>       1859 ± 22%     -51.5%     901.87 ± 57%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
>       2.95 ± 44%     -79.5%       0.60 ± 75%  sched_debug.cfs_rq:/.removed.util_avg.avg
>      72.25 ±  8%     -66.6%      24.17 ± 75%  sched_debug.cfs_rq:/.removed.util_avg.max
>      13.53 ± 23%     -72.1%       3.77 ± 75%  sched_debug.cfs_rq:/.removed.util_avg.stddev
>       9.43 ±  5%     +27.0%      11.97 ± 14%  sched_debug.cfs_rq:/.runnable_load_avg.avg
>     103.80 ± 10%     +62.1%     168.29 ± 27%  sched_debug.cfs_rq:/.runnable_load_avg.max
>      22.41 ±  7%     +39.0%      31.15 ± 20%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
>      12421 ± 59%     +64.4%      20419 ± 40%  sched_debug.cfs_rq:/.spread0.avg
>     -24890           -69.6%      -7575        sched_debug.cfs_rq:/.spread0.min
>      17263 ±  7%     -21.5%      13551 ± 15%  sched_debug.cfs_rq:/.spread0.stddev
>     325.68 ±  2%     -14.6%     278.27 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
>     186.27 ±  3%     -11.0%     165.84 ±  2%  sched_debug.cfs_rq:/.util_avg.stddev
>     132.45           -12.4%     116.05 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
>     155113           +19.3%     185086        sched_debug.cpu.clock.avg
>     155121           +19.3%     185093        sched_debug.cpu.clock.max
>     155106           +19.3%     185079        sched_debug.cpu.clock.min
>     155113           +19.3%     185086        sched_debug.cpu.clock_task.avg
>     155121           +19.3%     185093        sched_debug.cpu.clock_task.max
>     155106           +19.3%     185079        sched_debug.cpu.clock_task.min
>     104.40 ±  7%     +61.5%     168.62 ± 27%  sched_debug.cpu.cpu_load[0].max
>      96.20 ±  8%     +74.1%     167.46 ± 30%  sched_debug.cpu.cpu_load[1].max
>       4871           +14.8%       5592        sched_debug.cpu.curr->pid.max
>     127731           +23.5%     157685        sched_debug.cpu.nr_load_updates.avg
>     134899           +20.9%     163058        sched_debug.cpu.nr_load_updates.max
>     125325           +24.0%     155428        sched_debug.cpu.nr_load_updates.min
>     155106           +19.3%     185079        sched_debug.cpu_clk
>     151513           +19.8%     181485        sched_debug.ktime
>     155537           +19.3%     185526        sched_debug.sched_clk
>      19.77           -15.1%      16.78 ±  3%  perf-stat.i.MPKI
>  8.783e+08            -9.0%  7.989e+08        perf-stat.i.branch-instructions
>      19.44            -0.2       19.21        perf-stat.i.cache-miss-rate%
>   14932557           -19.1%   12081970        perf-stat.i.cache-misses
>   77250252           -18.0%   63363496        perf-stat.i.cache-references
>     302132           -21.2%     238146        perf-stat.i.context-switches
>       3.13           -11.6%       2.77 ±  2%  perf-stat.i.cpi
>  1.254e+10           -15.4%  1.061e+10        perf-stat.i.cpu-cycles
>     229.12 ±  3%     -21.3%     180.36 ±  3%  perf-stat.i.cpu-migrations
>     840.12            +4.5%     877.89        perf-stat.i.cycles-between-cache-misses
>  1.129e+09            -5.4%  1.068e+09        perf-stat.i.dTLB-loads
>       0.10 ±  4%      -0.0        0.09 ±  5%  perf-stat.i.dTLB-store-miss-rate%
>     538827 ±  3%     -16.1%     452028 ±  4%  perf-stat.i.dTLB-store-misses
>      47.32            +3.3       50.67 ±  3%  perf-stat.i.iTLB-load-miss-rate%
>    2556829           -20.4%    2034861 ±  2%  perf-stat.i.iTLB-loads
>  4.153e+09            -4.5%  3.964e+09        perf-stat.i.instructions
>       0.33           +12.2%       0.37 ±  2%  perf-stat.i.ipc
>    6672298           -19.6%    5364393        perf-stat.i.node-load-misses
>    7193961           -19.6%    5786904        perf-stat.i.node-loads
>    4765580           -20.6%    3784168        perf-stat.i.node-store-misses
>    6965089           -20.3%    5548679        perf-stat.i.node-stores
>      18.60           -14.0%      15.99 ±  2%  perf-stat.overall.MPKI
>      19.33            -0.3       19.07        perf-stat.overall.cache-miss-rate%
>       3.02           -11.3%       2.68 ±  2%  perf-stat.overall.cpi
>     839.71            +4.6%     878.19        perf-stat.overall.cycles-between-cache-misses
>       0.10 ±  4%      -0.0        0.08 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
>      45.63            +3.8       49.47 ±  3%  perf-stat.overall.iTLB-load-miss-rate%
>       0.33           +12.8%       0.37 ±  2%  perf-stat.overall.ipc
>  8.748e+08            -9.0%  7.964e+08        perf-stat.ps.branch-instructions
>   14872846           -19.0%   12043692        perf-stat.ps.cache-misses
>   76941024           -17.9%   63162394        perf-stat.ps.cache-references
>     300917           -21.1%     237386        perf-stat.ps.context-switches
>  1.249e+10           -15.3%  1.058e+10        perf-stat.ps.cpu-cycles
>     228.20 ±  3%     -21.2%     179.78 ±  3%  perf-stat.ps.cpu-migrations
>  1.124e+09            -5.3%  1.065e+09        perf-stat.ps.dTLB-loads
>     536681 ±  3%     -16.0%     450604 ±  4%  perf-stat.ps.dTLB-store-misses
>    2546543           -20.3%    2028370 ±  2%  perf-stat.ps.iTLB-loads
>  4.137e+09            -4.5%  3.952e+09        perf-stat.ps.instructions
>    6645469           -19.5%    5347283        perf-stat.ps.node-load-misses
>    7165042           -19.5%    5768454        perf-stat.ps.node-loads
>    4746415           -20.5%    3772102        perf-stat.ps.node-store-misses
>    6937096           -20.3%    5531006        perf-stat.ps.node-stores
>  1.049e+12           +20.4%  1.263e+12        perf-stat.total.instructions
>     160.75 ± 18%     +49.5%     240.25 ± 27%  interrupts.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5
>     123.75           +26.3%     156.25        interrupts.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
>     100313 ±  7%     +13.4%     113737        interrupts.CAL:Function_call_interrupts
>     508519           +26.0%     640668        interrupts.CPU0.LOC:Local_timer_interrupts
>       2437 ± 12%     +18.1%       2877 ±  4%  interrupts.CPU1.CAL:Function_call_interrupts
>     508036           +26.1%     640579        interrupts.CPU1.LOC:Local_timer_interrupts
>       2521 ± 10%     +14.3%       2882 ±  5%  interrupts.CPU10.CAL:Function_call_interrupts
>     507477           +26.1%     639993        interrupts.CPU10.LOC:Local_timer_interrupts
>       2519 ±  2%     +17.9%       2969 ±  6%  interrupts.CPU11.CAL:Function_call_interrupts
>     508051           +26.1%     640531        interrupts.CPU11.LOC:Local_timer_interrupts
>     508353           +25.8%     639418        interrupts.CPU12.LOC:Local_timer_interrupts
>     508289           +26.0%     640385        interrupts.CPU13.LOC:Local_timer_interrupts
>     508136           +25.8%     639470        interrupts.CPU14.LOC:Local_timer_interrupts
>       2417 ±  5%     +20.2%       2906 ±  6%  interrupts.CPU15.CAL:Function_call_interrupts
>     508330           +25.9%     640229        interrupts.CPU15.LOC:Local_timer_interrupts
>     508351           +25.9%     639790        interrupts.CPU16.LOC:Local_timer_interrupts
>       2508 ± 10%     +18.4%       2968 ±  6%  interrupts.CPU17.CAL:Function_call_interrupts
>     508084           +26.0%     640420        interrupts.CPU17.LOC:Local_timer_interrupts
>      17327 ± 18%     +26.1%      21856 ±  4%  interrupts.CPU17.RES:Rescheduling_interrupts
>     508457           +25.8%     639737        interrupts.CPU18.LOC:Local_timer_interrupts
>     508140           +26.0%     640214        interrupts.CPU19.LOC:Local_timer_interrupts
>       2280 ±  8%     +22.2%       2786 ±  5%  interrupts.CPU2.CAL:Function_call_interrupts
>     508336           +26.0%     640548        interrupts.CPU2.LOC:Local_timer_interrupts
>     507912           +26.1%     640315        interrupts.CPU20.LOC:Local_timer_interrupts
>       2595 ±  7%     +13.4%       2943 ±  5%  interrupts.CPU21.CAL:Function_call_interrupts
>     508108           +26.1%     640485        interrupts.CPU21.LOC:Local_timer_interrupts
>     508306           +26.0%     640244        interrupts.CPU22.LOC:Local_timer_interrupts
>       1303 ± 34%     -29.2%     922.75 ± 32%  interrupts.CPU22.NMI:Non-maskable_interrupts
>       1303 ± 34%     -29.2%     922.75 ± 32%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
>       2522 ±  9%     +17.2%       2956 ±  4%  interrupts.CPU23.CAL:Function_call_interrupts
>     508123           +26.1%     640526        interrupts.CPU23.LOC:Local_timer_interrupts
>     508372           +26.0%     640372        interrupts.CPU24.LOC:Local_timer_interrupts
>     507678           +26.1%     640435        interrupts.CPU25.LOC:Local_timer_interrupts
>     508375           +26.0%     640407        interrupts.CPU26.LOC:Local_timer_interrupts
>       2406 ± 11%     +19.9%       2885 ±  5%  interrupts.CPU27.CAL:Function_call_interrupts
>     508242           +26.0%     640265        interrupts.CPU27.LOC:Local_timer_interrupts
>       2449 ± 11%     +21.8%       2982 ±  5%  interrupts.CPU28.CAL:Function_call_interrupts
>     508292           +26.0%     640304        interrupts.CPU28.LOC:Local_timer_interrupts
>       2510 ±  6%     +14.4%       2870 ±  5%  interrupts.CPU29.CAL:Function_call_interrupts
>     508085           +26.0%     640167        interrupts.CPU29.LOC:Local_timer_interrupts
>     508172           +26.0%     640443        interrupts.CPU3.LOC:Local_timer_interrupts
>     508342           +25.9%     639891        interrupts.CPU30.LOC:Local_timer_interrupts
>     508009           +26.1%     640402        interrupts.CPU31.LOC:Local_timer_interrupts
>       2522 ± 11%     +16.8%       2946 ±  4%  interrupts.CPU32.CAL:Function_call_interrupts
>     508138           +25.9%     639578        interrupts.CPU32.LOC:Local_timer_interrupts
>     508010           +26.1%     640577        interrupts.CPU33.LOC:Local_timer_interrupts
>     508029           +25.9%     639405        interrupts.CPU34.LOC:Local_timer_interrupts
>       2464 ±  8%     +15.3%       2840 ±  5%  interrupts.CPU35.CAL:Function_call_interrupts
>     508216           +26.0%     640171        interrupts.CPU35.LOC:Local_timer_interrupts
>     160.75 ± 18%     +49.5%     240.25 ± 27%  interrupts.CPU36.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5
>     507990           +25.9%     639575        interrupts.CPU36.LOC:Local_timer_interrupts
>     508010           +26.0%     640200        interrupts.CPU37.LOC:Local_timer_interrupts
>       2536 ±  8%     +13.9%       2888 ±  6%  interrupts.CPU38.CAL:Function_call_interrupts
>     507853           +25.9%     639351        interrupts.CPU38.LOC:Local_timer_interrupts
>       2414 ±  7%     +19.6%       2887 ±  7%  interrupts.CPU39.CAL:Function_call_interrupts
>     507619           +26.0%     639416        interrupts.CPU39.LOC:Local_timer_interrupts
>     508318           +25.9%     640069        interrupts.CPU4.LOC:Local_timer_interrupts
>       2463 ± 12%     +26.6%       3119 ±  2%  interrupts.CPU5.CAL:Function_call_interrupts
>     508112           +26.1%     640497        interrupts.CPU5.LOC:Local_timer_interrupts
>     123.75           +26.3%     156.25        interrupts.CPU6.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
>       2351 ±  9%     +21.5%       2856 ±  5%  interrupts.CPU6.CAL:Function_call_interrupts
>     508563           +25.8%     639582        interrupts.CPU6.LOC:Local_timer_interrupts
>       2592 ±  4%     +13.6%       2943 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
>     508150           +26.0%     640378        interrupts.CPU7.LOC:Local_timer_interrupts
>     508498           +25.9%     640139        interrupts.CPU8.LOC:Local_timer_interrupts
>       2501 ± 10%     +18.4%       2960 ±  7%  interrupts.CPU9.CAL:Function_call_interrupts
>     508092           +26.0%     640432        interrupts.CPU9.LOC:Local_timer_interrupts
>   20326190           +26.0%   25605623        interrupts.LOC:Local_timer_interrupts
>      26401 ±  5%      -9.9%      23785 ±  4%  interrupts.NMI:Non-maskable_interrupts
>      26401 ±  5%      -9.9%      23785 ±  4%  interrupts.PMI:Performance_monitoring_interrupts
>      69901 ±  3%     +27.9%      89422 ±  3%  softirqs.CPU0.RCU
>      45503 ±  2%     +18.1%      53752        softirqs.CPU0.SCHED
>      90281 ±  2%     +17.4%     105987 ±  2%  softirqs.CPU0.TIMER
>      72001 ±  4%     +30.7%      94130 ±  2%  softirqs.CPU1.RCU
>      42373           +18.9%      50374 ±  2%  softirqs.CPU1.SCHED
>      89928 ±  3%     +18.5%     106554 ±  4%  softirqs.CPU1.TIMER
>      73216 ±  6%     +24.5%      91156 ±  3%  softirqs.CPU10.RCU
>      42693           +16.8%      49881 ±  2%  softirqs.CPU10.SCHED
>      86975 ±  2%     +19.4%     103822 ±  3%  softirqs.CPU10.TIMER
>      71035           +31.2%      93194 ±  2%  softirqs.CPU11.RCU
>      42794           +19.6%      51168 ±  2%  softirqs.CPU11.SCHED
>      86533           +18.9%     102845 ±  4%  softirqs.CPU11.TIMER
>      70358 ±  3%     +33.3%      93810 ±  3%  softirqs.CPU12.RCU
>      42548           +17.2%      49878        softirqs.CPU12.SCHED
>      87303 ±  2%     +21.8%     106341 ±  3%  softirqs.CPU12.TIMER
>      70364 ±  3%     +31.7%      92648 ±  4%  softirqs.CPU13.RCU
>      42421           +18.9%      50428        softirqs.CPU13.SCHED
>      86875           +19.5%     103817 ±  3%  softirqs.CPU13.TIMER
>      69128 ±  2%     +30.7%      90331 ±  3%  softirqs.CPU14.RCU
>      42975 ±  2%     +15.4%      49575        softirqs.CPU14.SCHED
>      84688 ±  3%     +21.6%     102994 ±  3%  softirqs.CPU14.TIMER
>      72010 ±  2%     +30.6%      94068 ±  3%  softirqs.CPU15.RCU
>      42893           +17.0%      50178        softirqs.CPU15.SCHED
>      86458           +20.4%     104117 ±  4%  softirqs.CPU15.TIMER
>      70886 ±  2%     +30.3%      92387 ±  2%  softirqs.CPU16.RCU
>      42970           +16.3%      49975        softirqs.CPU16.SCHED
>      87944 ±  2%     +19.7%     105244 ±  2%  softirqs.CPU16.TIMER
>      70757 ±  2%     +30.5%      92359 ±  2%  softirqs.CPU17.RCU
>      42742           +17.7%      50310 ±  2%  softirqs.CPU17.SCHED
>      87096 ±  2%     +17.9%     102652 ±  3%  softirqs.CPU17.TIMER
>      69077 ±  3%     +32.6%      91630 ±  3%  softirqs.CPU18.RCU
>      42942           +17.0%      50245 ±  2%  softirqs.CPU18.SCHED
>      68722 ±  2%     +29.7%      89150 ±  2%  softirqs.CPU19.RCU
>      42964 ±  2%     +17.2%      50334 ±  2%  softirqs.CPU19.SCHED
>      84910 ±  4%     +24.0%     105326 ±  7%  softirqs.CPU19.TIMER
>      72687 ±  4%     +25.6%      91315 ±  3%  softirqs.CPU2.RCU
>      42688           +16.2%      49622        softirqs.CPU2.SCHED
>      88869 ±  5%     +16.3%     103332        softirqs.CPU2.TIMER
>      70693 ±  2%     +26.6%      89525 ±  4%  softirqs.CPU20.RCU
>      42228           +17.5%      49611 ±  2%  softirqs.CPU20.SCHED
>      88647 ±  2%     +16.4%     103227 ±  2%  softirqs.CPU20.TIMER
>      70857 ±  3%     +32.1%      93633 ±  3%  softirqs.CPU21.RCU
>      42878           +17.2%      50239        softirqs.CPU21.SCHED
>      87325           +18.2%     103214 ±  4%  softirqs.CPU21.TIMER
>      71099 ±  3%     +29.0%      91711 ±  3%  softirqs.CPU22.RCU
>      42534           +17.3%      49888        softirqs.CPU22.SCHED
>      87698 ±  3%     +19.3%     104642        softirqs.CPU22.TIMER
>      72098 ±  2%     +30.3%      93930 ±  3%  softirqs.CPU23.RCU
>      42771           +17.0%      50022 ±  2%  softirqs.CPU23.SCHED
>      87931           +17.5%     103319 ±  3%  softirqs.CPU23.TIMER
>      70804 ±  3%     +33.4%      94432 ±  6%  softirqs.CPU24.RCU
>      42554 ±  3%     +16.4%      49516 ±  2%  softirqs.CPU24.SCHED
>      71435 ±  3%     +35.0%      96465 ±  3%  softirqs.CPU25.RCU
>      42605           +16.9%      49815        softirqs.CPU25.SCHED
>      87163           +21.2%     105672        softirqs.CPU25.TIMER
>      70216           +31.0%      91964 ±  3%  softirqs.CPU26.RCU
>      42147           +19.0%      50165 ±  2%  softirqs.CPU26.SCHED
>      88215 ±  3%     +36.3%     120217 ± 21%  softirqs.CPU26.TIMER
>      70713           +32.5%      93695 ±  6%  softirqs.CPU27.RCU
>      42528           +18.2%      50249 ±  2%  softirqs.CPU27.SCHED
>      88110           +38.5%     122002 ± 24%  softirqs.CPU27.TIMER
>      69654 ±  2%     +34.1%      93398 ±  3%  softirqs.CPU28.RCU
>      42052           +18.0%      49636 ±  2%  softirqs.CPU28.SCHED
>      86876 ±  2%     +21.6%     105634 ±  2%  softirqs.CPU28.TIMER
>      70230 ±  2%     +37.8%      96776 ±  5%  softirqs.CPU29.RCU
>      42440           +17.0%      49673 ±  2%  softirqs.CPU29.SCHED
>      87058           +18.9%     103502 ±  2%  softirqs.CPU29.TIMER
>      70519 ±  3%     +34.2%      94624 ±  2%  softirqs.CPU3.RCU
>      43299           +16.2%      50322 ±  2%  softirqs.CPU3.SCHED
>      87572           +19.2%     104386 ±  4%  softirqs.CPU3.TIMER
>      70538 ±  2%     +31.0%      92391 ±  5%  softirqs.CPU30.RCU
>      42038 ±  2%     +17.9%      49574        softirqs.CPU30.SCHED
>      86446 ±  3%     +20.6%     104284        softirqs.CPU30.TIMER
>      71140 ±  2%     +33.6%      95050 ±  3%  softirqs.CPU31.RCU
>      42599           +16.5%      49608 ±  2%  softirqs.CPU31.SCHED
>      86417           +19.6%     103393 ±  3%  softirqs.CPU31.TIMER
>      59293 ±  2%     +29.7%      76894 ±  4%  softirqs.CPU32.RCU
>      42200           +17.8%      49711        softirqs.CPU32.SCHED
>      86726           +18.8%     103045 ±  3%  softirqs.CPU32.TIMER
>      58507           +31.6%      76967        softirqs.CPU33.RCU
>      42587           +16.7%      49683        softirqs.CPU33.SCHED
>      86567           +19.1%     103101 ±  3%  softirqs.CPU33.TIMER
>      59394 ±  2%     +27.1%      75493 ±  5%  softirqs.CPU34.RCU
>      42682           +16.6%      49753        softirqs.CPU34.SCHED
>      59949 ±  3%     +29.2%      77459        softirqs.CPU35.RCU
>      42681 ±  2%     +17.3%      50047 ±  2%  softirqs.CPU35.SCHED
>      85785           +19.9%     102825 ±  2%  softirqs.CPU35.TIMER
>      61089           +25.4%      76612        softirqs.CPU36.RCU
>      42662           +16.9%      49870        softirqs.CPU36.SCHED
>      87784 ±  2%     +17.3%     102973 ±  3%  softirqs.CPU36.TIMER
>      60018           +29.3%      77596        softirqs.CPU37.RCU
>      42378 ±  2%     +18.0%      50012        softirqs.CPU37.SCHED
>      85919           +20.3%     103386 ±  2%  softirqs.CPU37.TIMER
>      59781 ±  2%     +28.6%      76877        softirqs.CPU38.RCU
>      42779           +17.1%      50105        softirqs.CPU38.SCHED
>      85246 ±  4%     +20.7%     102859 ±  2%  softirqs.CPU38.TIMER
>      59657           +26.9%      75699 ±  2%  softirqs.CPU39.RCU
>      43011 ±  2%     +17.6%      50580 ±  3%  softirqs.CPU39.SCHED
>      98669 ± 20%     +36.3%     134491 ± 24%  softirqs.CPU39.TIMER
>      68912 ±  2%     +32.7%      91479 ±  2%  softirqs.CPU4.RCU
>      43101           +15.9%      49946        softirqs.CPU4.SCHED
>      86425 ±  5%     +18.4%     102344 ±  3%  softirqs.CPU4.TIMER
>      70893 ±  3%     +32.1%      93618 ±  3%  softirqs.CPU5.RCU
>      42977           +17.2%      50381        softirqs.CPU5.SCHED
>      87301           +17.7%     102788 ±  3%  softirqs.CPU5.TIMER
>      71151           +25.0%      88965 ±  3%  softirqs.CPU6.RCU
>      42864           +17.3%      50282        softirqs.CPU6.SCHED
>      87741 ±  2%     +22.5%     107458 ±  4%  softirqs.CPU6.TIMER
>      71242 ±  4%     +30.5%      92999 ±  3%  softirqs.CPU7.RCU
>      43014 ±  2%     +17.7%      50619        softirqs.CPU7.SCHED
>      87907           +15.8%     101781 ±  5%  softirqs.CPU7.TIMER
>      69638 ±  2%     +32.6%      92309 ±  4%  softirqs.CPU8.RCU
>      42479           +17.6%      49947        softirqs.CPU8.SCHED
>      86964 ±  2%     +18.6%     103128 ±  3%  softirqs.CPU8.TIMER
>      69585 ±  2%     +36.1%      94684 ±  2%  softirqs.CPU9.RCU
>      42667           +18.0%      50343        softirqs.CPU9.SCHED
>      86913           +19.2%     103596 ±  4%  softirqs.CPU9.TIMER
>    2739272           +30.7%    3580875 ±  2%  softirqs.RCU
>    1709285           +17.3%    2005342        softirqs.SCHED
>    3539953           +19.4%    4226728 ±  2%  softirqs.TIMER
> 
> 
> 
> 
> 
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
> 
> 
> Thanks,
> Rong Chen
> 



Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)

Powered by blists - more mailing lists