lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Tue, 12 Mar 2019 21:50:44 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Qu Wenruo <wqu@...e.com>
Cc:     David Sterba <dsterba@...e.com>,
        Leonard Lausen <leonard@...sen.nl>,
        LKML <linux-kernel@...r.kernel.org>,
        Stephen Rothwell <sfr@...b.auug.org.au>, lkp@...org
Subject: [LKP] [btrfs]  44fe89de7d:  aim7.jobs-per-min -15.1% regression

Greeting,

FYI, we noticed a -15.1% regression of aim7.jobs-per-min due to commit:


commit: 44fe89de7d5157a4f31f13d94802c7619e23f462 ("btrfs: Do mandatory tree block check before submitting bio")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

in testcase: aim7
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: btrfs
	test: sync_disk_rw
	load: 20
	cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min -20.8% regression                             |
| test machine     | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory |
| test parameters  | cpufreq_governor=performance                                          |
|                  | disk=4BRD_12G                                                         |
|                  | fs=btrfs                                                              |
|                  | load=20                                                               |
|                  | md=RAID0                                                              |
|                  | test=sync_disk_rw                                                     |
+------------------+-----------------------------------------------------------------------+


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-8/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-skl-2sp7/sync_disk_rw/aim7

commit: 
  a92e9c3a3c ("btrfs: extent_io: Handle error better in extent_writepages()")
  44fe89de7d ("btrfs: Do mandatory tree block check before submitting bio")

a92e9c3a3cef4c0c 44fe89de7d5157a4f31f13d9480 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           50%           2:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      1204           -15.1%       1022        aim7.jobs-per-min
     99.66           +18.3%     117.93        aim7.time.elapsed_time
     99.66           +18.3%     117.93        aim7.time.elapsed_time.max
      3014 ±  2%      -3.1%       2921        aim7.time.minor_page_faults
 1.236e+09           +44.6%  1.787e+09 ± 27%  cpuidle.C1.time
     90473           +15.8%     104758 ±  2%  meminfo.AnonHugePages
    102742           -16.5%      85841 ±  2%  meminfo.max_used_kB
     11.68            -1.8        9.85        mpstat.cpu.-1.sys%
      0.07 ±  7%      -0.0        0.05 ± 17%  mpstat.cpu.-1.usr%
    664.25 ±  4%     +18.1%     784.25 ±  6%  slabinfo.dmaengine-unmap-16.active_objs
    664.25 ±  4%     +18.1%     784.25 ±  6%  slabinfo.dmaengine-unmap-16.num_objs
     88.00            +2.3%      90.00        vmstat.cpu.id
    346712           -15.5%     293086        vmstat.io.bo
    558192           -15.4%     472096        vmstat.system.cs
     88.42            +2.0%      90.19        iostat.cpu.idle
     11.50           -15.3%       9.74        iostat.cpu.system
     19892           -15.2%      16877        iostat.md0.w/s
    451087           -15.4%     381436        iostat.md0.wkB/s
    252.75 ±  2%     -19.8%     202.67 ± 16%  sched_debug.cfs_rq:/.removed.util_avg.max
     57.62 ±  2%     +71.4%      98.79 ± 51%  sched_debug.cpu.cpu_load[1].max
     47.75 ± 13%     +81.6%      86.71 ± 43%  sched_debug.cpu.cpu_load[2].max
      7.35 ± 13%     -16.4%       6.14 ±  4%  sched_debug.cpu.cpu_load[4].avg
    422.75           -14.3%     362.50        turbostat.Avg_MHz
     14.00            -2.0       12.04        turbostat.Busy%
  15068782           +15.6%   17424116 ±  2%  turbostat.IRQ
    163.49            -2.2%     159.86        turbostat.PkgWatt
    516444            +2.0%     526872        proc-vmstat.nr_active_file
    890319            +1.4%     902554        proc-vmstat.nr_file_pages
      4668            -1.7%       4591        proc-vmstat.nr_inactive_anon
    516444            +2.0%     526872        proc-vmstat.nr_zone_active_file
      4668            -1.7%       4591        proc-vmstat.nr_zone_inactive_anon
    288573           +14.8%     331243        proc-vmstat.pgfault
    716355            +7.5%     770096        proc-vmstat.pgfree
  24648391           -12.7%   21521966 ±  5%  perf-stat.i.branch-misses
     27.60            -1.9       25.66 ±  2%  perf-stat.i.cache-miss-rate%
  33238901           -22.5%   25763877 ±  2%  perf-stat.i.cache-misses
 1.163e+08           -13.8%  1.002e+08 ±  3%  perf-stat.i.cache-references
    570181           -15.7%     480734        perf-stat.i.context-switches
      4.53 ±  3%     -20.6%       3.60 ±  2%  perf-stat.i.cpi
  3.02e+10           -14.5%  2.581e+10        perf-stat.i.cpu-cycles
     12.93 ± 11%     -23.2%       9.92 ±  7%  perf-stat.i.cpu-migrations
  1.81e+09            +8.7%  1.968e+09 ±  3%  perf-stat.i.dTLB-loads
 1.018e+09           +10.9%  1.129e+09        perf-stat.i.dTLB-stores
   8329645           -15.2%    7063121        perf-stat.i.iTLB-loads
 6.662e+09            +9.5%  7.297e+09        perf-stat.i.instructions
     93411 ±  7%     +20.1%     112199 ±  6%  perf-stat.i.instructions-per-iTLB-miss
      0.24 ±  4%     +21.3%       0.30        perf-stat.i.ipc
      2719            -2.6%       2650        perf-stat.i.minor-faults
  10179150           -22.8%    7861388 ±  2%  perf-stat.i.node-load-misses
    819501           -18.0%     672068        perf-stat.i.node-loads
     79.98            -3.6       76.35        perf-stat.i.node-store-miss-rate%
   3971266           -23.9%    3020404        perf-stat.i.node-store-misses
    760058 ±  3%      -7.2%     705596        perf-stat.i.node-stores
      2720            -2.6%       2650        perf-stat.i.page-faults
     17.46           -21.4%      13.73 ±  2%  perf-stat.overall.MPKI
      1.87            -0.3        1.59 ±  3%  perf-stat.overall.branch-miss-rate%
     28.58            -2.8       25.75 ±  4%  perf-stat.overall.cache-miss-rate%
      4.53           -22.0%       3.54        perf-stat.overall.cpi
    908.69           +10.3%       1002 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.22           +28.1%       0.28        perf-stat.overall.ipc
     83.93            -2.9       81.06        perf-stat.overall.node-store-miss-rate%
  24408968           -12.5%   21345782 ±  5%  perf-stat.ps.branch-misses
  32896998           -22.4%   25539476 ±  2%  perf-stat.ps.cache-misses
 1.151e+08           -13.7%   99343827 ±  3%  perf-stat.ps.cache-references
    564308           -15.6%     476544        perf-stat.ps.context-switches
 2.989e+10           -14.4%  2.558e+10        perf-stat.ps.cpu-cycles
     12.79 ± 11%     -23.1%       9.84 ±  7%  perf-stat.ps.cpu-migrations
 1.792e+09            +8.9%  1.951e+09 ±  3%  perf-stat.ps.dTLB-loads
 1.008e+09           +11.1%   1.12e+09        perf-stat.ps.dTLB-stores
   8244056           -15.1%    7001712        perf-stat.ps.iTLB-loads
 6.595e+09            +9.7%  7.235e+09        perf-stat.ps.instructions
      2693            -2.4%       2628        perf-stat.ps.minor-faults
  10074272           -22.6%    7792778 ±  2%  perf-stat.ps.node-load-misses
    811079           -17.9%     666222        perf-stat.ps.node-loads
   3930342           -23.8%    2994037        perf-stat.ps.node-store-misses
    752248 ±  3%      -7.0%     699463        perf-stat.ps.node-stores
      2693            -2.4%       2628        perf-stat.ps.page-faults
   6.6e+11           +29.8%  8.564e+11        perf-stat.total.instructions
     44.42            -3.1       41.36 ±  3%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     46.97            -2.2       44.77 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     50.60            -1.9       48.74 ±  2%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     49.88            -1.7       48.21 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     49.90            -1.7       48.24 ±  2%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     49.90            -1.7       48.24 ±  2%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      4.89 ± 12%      -1.0        3.84 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
      4.89 ± 12%      -1.0        3.84 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
      5.75 ±  6%      -0.7        5.07 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
      2.90 ± 11%      -0.6        2.30 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
      5.82 ±  2%      -0.5        5.35 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
      1.78 ± 14%      -0.4        1.34 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
      4.25 ±  7%      -0.4        3.82        perf-profile.calltrace.cycles-pp.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
      2.73 ±  7%      -0.3        2.43 ±  4%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      8.78            -0.3        8.49 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
      1.78 ±  8%      -0.3        1.52 ±  3%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      8.54            -0.3        8.28 ±  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node
      2.15 ±  2%      -0.2        1.99 ±  5%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      1.43            -0.1        1.36 ±  3%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum
      1.02 ±  2%      -0.1        0.97        perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode
      1.02 ±  3%      -0.0        0.97        perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_csum_file_blocks.btrfs_log_changed_extents
      0.95 ±  5%      +0.2        1.11 ±  7%  perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range
      1.12 ±  2%      +0.2        1.32 ±  6%  perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file
      1.05 ±  3%      +0.2        1.24 ±  5%  perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
      1.25 ±  2%      +0.2        1.46 ±  5%  perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
      0.68 ±  6%      +0.2        0.92 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      1.45 ±  2%      +0.2        1.70 ±  4%  perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
      0.84 ± 13%      +0.3        1.10 ±  3%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.70 ±  6%      +0.3        0.95 ±  6%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.25 ±100%      +0.3        0.56 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
      1.55 ±  2%      +0.3        1.87 ±  5%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file
      1.55 ±  2%      +0.3        1.87 ±  5%  perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops
      1.57 ±  2%      +0.3        1.89 ±  5%  perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
      1.56 ±  2%      +0.3        1.88 ±  5%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter
      1.56 ±  2%      +0.3        1.88 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
      0.26 ±100%      +0.4        0.62 ±  6%  perf-profile.calltrace.cycles-pp.submit_one_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
      0.26 ±100%      +0.4        0.62 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_submit_bio_hook.submit_one_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range
      0.00            +0.6        0.56 ±  5%  perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
      0.00            +0.6        0.56 ±  5%  perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio
      0.00            +0.6        0.58 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio.extent_writepages.do_writepages
      0.19 ±173%      +0.6        0.78 ± 27%  perf-profile.calltrace.cycles-pp.btrfs_next_old_leaf.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      0.00            +0.8        0.84 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook
      0.00            +1.3        1.32 ±  6%  perf-profile.calltrace.cycles-pp.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
      0.00            +1.4        1.39 ±  5%  perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
      0.00            +1.4        1.39 ±  6%  perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
      0.75 ±  4%      +1.5        2.23 ±  5%  perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
      0.76 ±  3%      +1.5        2.24 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
      0.75 ±  4%      +1.5        2.23 ±  5%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
      0.75 ±  4%      +1.5        2.23 ±  5%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
      0.38 ± 57%      +1.6        1.96 ±  6%  perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
      0.38 ± 57%      +1.6        1.96 ±  5%  perf-profile.calltrace.cycles-pp.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
      2.42 ±  2%      +1.8        4.18 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
     38.62            +2.5       41.15 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write
     39.95            +2.6       42.52 ±  3%  perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     39.95            +2.6       42.52 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64
     39.99            +2.6       42.56 ±  3%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     39.98            +2.6       42.55 ±  3%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     40.19            +2.6       42.81 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     40.19            +2.6       42.81 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     97.75 ± 21%     +77.5%     173.50 ± 21%  interrupts.70:PCI-MSI.12589059-edge.eth3-TxRx-2
     63.25 ±  9%     +51.4%      95.75 ± 25%  interrupts.71:PCI-MSI.12589060-edge.eth3-TxRx-3
    202.50           +17.8%     238.50        interrupts.9:IO-APIC.9-fasteoi.acpi
     88724           +12.8%     100056        interrupts.CAL:Function_call_interrupts
      1255           +12.4%       1410        interrupts.CPU0.CAL:Function_call_interrupts
    201136           +17.1%     235497 ±  2%  interrupts.CPU0.LOC:Local_timer_interrupts
    202.50           +17.8%     238.50        interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
      1242           +13.2%       1407        interrupts.CPU1.CAL:Function_call_interrupts
    200860           +16.6%     234199 ±  2%  interrupts.CPU1.LOC:Local_timer_interrupts
      3195 ± 24%     -60.2%       1273 ± 95%  interrupts.CPU1.NMI:Non-maskable_interrupts
      3195 ± 24%     -60.2%       1273 ± 95%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
    200768           +16.8%     234566        interrupts.CPU10.LOC:Local_timer_interrupts
    200322           +17.1%     234482 ±  2%  interrupts.CPU11.LOC:Local_timer_interrupts
      1174 ± 11%     +18.3%       1389        interrupts.CPU12.CAL:Function_call_interrupts
    200965           +16.4%     233853        interrupts.CPU12.LOC:Local_timer_interrupts
      1209 ±  4%     +15.4%       1396        interrupts.CPU13.CAL:Function_call_interrupts
    199915           +17.6%     235085 ±  2%  interrupts.CPU13.LOC:Local_timer_interrupts
      1187 ±  8%     +18.1%       1403        interrupts.CPU14.CAL:Function_call_interrupts
    200677           +17.2%     235290 ±  2%  interrupts.CPU14.LOC:Local_timer_interrupts
      1208 ±  5%     +13.2%       1367 ±  4%  interrupts.CPU15.CAL:Function_call_interrupts
    200566           +17.3%     235226 ±  2%  interrupts.CPU15.LOC:Local_timer_interrupts
    200800           +17.1%     235099 ±  2%  interrupts.CPU16.LOC:Local_timer_interrupts
      1217 ±  4%     +15.3%       1403        interrupts.CPU17.CAL:Function_call_interrupts
    200473           +16.6%     233816        interrupts.CPU17.LOC:Local_timer_interrupts
      1246           +12.8%       1406        interrupts.CPU18.CAL:Function_call_interrupts
    200863           +16.4%     233829 ±  3%  interrupts.CPU18.LOC:Local_timer_interrupts
      1243           +12.7%       1400        interrupts.CPU19.CAL:Function_call_interrupts
    200576           +16.0%     232732 ±  2%  interrupts.CPU19.LOC:Local_timer_interrupts
      1247           +12.9%       1407        interrupts.CPU2.CAL:Function_call_interrupts
    200971           +16.9%     234895 ±  2%  interrupts.CPU2.LOC:Local_timer_interrupts
      1247           +12.7%       1406        interrupts.CPU20.CAL:Function_call_interrupts
    200473           +16.6%     233803 ±  3%  interrupts.CPU20.LOC:Local_timer_interrupts
      1247           +12.6%       1404        interrupts.CPU21.CAL:Function_call_interrupts
    200696           +16.2%     233299 ±  2%  interrupts.CPU21.LOC:Local_timer_interrupts
      1247           +13.1%       1411        interrupts.CPU22.CAL:Function_call_interrupts
    200814           +16.4%     233758 ±  3%  interrupts.CPU22.LOC:Local_timer_interrupts
      4891 ± 12%     -24.7%       3681 ± 12%  interrupts.CPU22.RES:Rescheduling_interrupts
      1247           +13.0%       1410        interrupts.CPU23.CAL:Function_call_interrupts
    200828           +16.4%     233716 ±  3%  interrupts.CPU23.LOC:Local_timer_interrupts
      1247           +14.3%       1426 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
    200833           +16.5%     233997 ±  3%  interrupts.CPU24.LOC:Local_timer_interrupts
      1246           +12.9%       1407        interrupts.CPU25.CAL:Function_call_interrupts
    201055           +16.3%     233920 ±  3%  interrupts.CPU25.LOC:Local_timer_interrupts
      1265 ±  2%     +12.7%       1425 ±  2%  interrupts.CPU26.CAL:Function_call_interrupts
    201132           +15.9%     233180 ±  3%  interrupts.CPU26.LOC:Local_timer_interrupts
      1173 ± 10%     +20.0%       1407        interrupts.CPU27.CAL:Function_call_interrupts
    201166           +15.9%     233210 ±  3%  interrupts.CPU27.LOC:Local_timer_interrupts
      1263 ±  2%     +11.4%       1407        interrupts.CPU28.CAL:Function_call_interrupts
    201049           +16.3%     233848 ±  3%  interrupts.CPU28.LOC:Local_timer_interrupts
      1241           +13.5%       1409        interrupts.CPU29.CAL:Function_call_interrupts
    200963           +16.3%     233774 ±  3%  interrupts.CPU29.LOC:Local_timer_interrupts
     97.75 ± 21%     +77.5%     173.50 ± 21%  interrupts.CPU3.70:PCI-MSI.12589059-edge.eth3-TxRx-2
      1210 ±  5%     +13.2%       1369 ±  3%  interrupts.CPU3.CAL:Function_call_interrupts
    199954           +17.1%     234167 ±  2%  interrupts.CPU3.LOC:Local_timer_interrupts
      1181 ±  9%     +19.1%       1407        interrupts.CPU30.CAL:Function_call_interrupts
    200986           +16.4%     233892 ±  3%  interrupts.CPU30.LOC:Local_timer_interrupts
      1127 ± 18%     +24.7%       1406        interrupts.CPU31.CAL:Function_call_interrupts
    201110           +16.2%     233756 ±  3%  interrupts.CPU31.LOC:Local_timer_interrupts
      1249           +13.0%       1412        interrupts.CPU32.CAL:Function_call_interrupts
    200832           +16.4%     233787 ±  3%  interrupts.CPU32.LOC:Local_timer_interrupts
      1249           +13.3%       1416        interrupts.CPU33.CAL:Function_call_interrupts
    200969           +16.3%     233758 ±  3%  interrupts.CPU33.LOC:Local_timer_interrupts
      1246           +13.4%       1413        interrupts.CPU34.CAL:Function_call_interrupts
    199891           +17.1%     234012 ±  3%  interrupts.CPU34.LOC:Local_timer_interrupts
      1234           +13.3%       1399        interrupts.CPU35.CAL:Function_call_interrupts
    200721           +16.2%     233256 ±  2%  interrupts.CPU35.LOC:Local_timer_interrupts
      3994 ± 14%     -50.7%       1970 ± 48%  interrupts.CPU35.RES:Rescheduling_interrupts
      1242           +12.9%       1402 ±  2%  interrupts.CPU36.CAL:Function_call_interrupts
    199765           +17.7%     235071 ±  2%  interrupts.CPU36.LOC:Local_timer_interrupts
    200843           +16.6%     234175 ±  2%  interrupts.CPU37.LOC:Local_timer_interrupts
      1248           +13.4%       1415        interrupts.CPU38.CAL:Function_call_interrupts
    200016           +17.0%     234095 ±  2%  interrupts.CPU38.LOC:Local_timer_interrupts
      1249           +13.0%       1411        interrupts.CPU39.CAL:Function_call_interrupts
    200714           +17.2%     235250 ±  2%  interrupts.CPU39.LOC:Local_timer_interrupts
     63.25 ±  9%     +51.4%      95.75 ± 25%  interrupts.CPU4.71:PCI-MSI.12589060-edge.eth3-TxRx-3
      1246           +13.9%       1419 ±  2%  interrupts.CPU4.CAL:Function_call_interrupts
    200637           +17.2%     235179 ±  2%  interrupts.CPU4.LOC:Local_timer_interrupts
    200593           +17.2%     235093 ±  2%  interrupts.CPU40.LOC:Local_timer_interrupts
      1248           +13.0%       1410        interrupts.CPU41.CAL:Function_call_interrupts
    200998           +16.6%     234380        interrupts.CPU41.LOC:Local_timer_interrupts
      1249           +12.9%       1410        interrupts.CPU42.CAL:Function_call_interrupts
    200934           +16.6%     234357        interrupts.CPU42.LOC:Local_timer_interrupts
      1246           +13.2%       1411        interrupts.CPU43.CAL:Function_call_interrupts
    200894           +16.9%     234942 ±  2%  interrupts.CPU43.LOC:Local_timer_interrupts
      1211 ±  5%     +16.3%       1408        interrupts.CPU44.CAL:Function_call_interrupts
    200154           +17.4%     235070 ±  2%  interrupts.CPU44.LOC:Local_timer_interrupts
      2454 ± 23%     +50.0%       3682 ± 14%  interrupts.CPU44.RES:Rescheduling_interrupts
      1248           +13.0%       1410        interrupts.CPU45.CAL:Function_call_interrupts
    200774           +16.7%     234259 ±  2%  interrupts.CPU45.LOC:Local_timer_interrupts
      1246           +13.1%       1410        interrupts.CPU46.CAL:Function_call_interrupts
    201184           +16.8%     234966 ±  2%  interrupts.CPU46.LOC:Local_timer_interrupts
      1246           +13.1%       1409        interrupts.CPU47.CAL:Function_call_interrupts
    200672           +16.7%     234200        interrupts.CPU47.LOC:Local_timer_interrupts
    200070           +17.7%     235463 ±  2%  interrupts.CPU48.LOC:Local_timer_interrupts
      1251           +12.6%       1409        interrupts.CPU49.CAL:Function_call_interrupts
    200232           +17.4%     235071 ±  2%  interrupts.CPU49.LOC:Local_timer_interrupts
    200450           +17.4%     235328 ±  2%  interrupts.CPU5.LOC:Local_timer_interrupts
      1216 ±  5%     +15.7%       1407        interrupts.CPU50.CAL:Function_call_interrupts
    201051           +16.5%     234316        interrupts.CPU50.LOC:Local_timer_interrupts
      1252           +12.6%       1410        interrupts.CPU51.CAL:Function_call_interrupts
    200007           +17.5%     235046 ±  2%  interrupts.CPU51.LOC:Local_timer_interrupts
      1253           +12.4%       1408        interrupts.CPU52.CAL:Function_call_interrupts
    200770           +17.0%     234811 ±  2%  interrupts.CPU52.LOC:Local_timer_interrupts
      1252           +12.2%       1405        interrupts.CPU53.CAL:Function_call_interrupts
    200095           +17.4%     235004 ±  2%  interrupts.CPU53.LOC:Local_timer_interrupts
      1252           +12.6%       1410        interrupts.CPU54.CAL:Function_call_interrupts
    200697           +16.5%     233818 ±  3%  interrupts.CPU54.LOC:Local_timer_interrupts
      1252           +12.6%       1409        interrupts.CPU55.CAL:Function_call_interrupts
    200852           +16.5%     234024 ±  3%  interrupts.CPU55.LOC:Local_timer_interrupts
      1251           +12.5%       1407        interrupts.CPU56.CAL:Function_call_interrupts
    200874           +16.4%     233828 ±  3%  interrupts.CPU56.LOC:Local_timer_interrupts
      1251           +12.3%       1404        interrupts.CPU57.CAL:Function_call_interrupts
    200887           +16.5%     233974 ±  3%  interrupts.CPU57.LOC:Local_timer_interrupts
      1250           +12.8%       1411        interrupts.CPU58.CAL:Function_call_interrupts
    200837           +16.4%     233702 ±  3%  interrupts.CPU58.LOC:Local_timer_interrupts
      1250           +12.8%       1410        interrupts.CPU59.CAL:Function_call_interrupts
    199809           +17.1%     233932 ±  3%  interrupts.CPU59.LOC:Local_timer_interrupts
      1262 ±  2%     +10.5%       1394        interrupts.CPU6.CAL:Function_call_interrupts
    201108           +16.6%     234545        interrupts.CPU6.LOC:Local_timer_interrupts
      1253           +12.6%       1411        interrupts.CPU60.CAL:Function_call_interrupts
    200941           +16.4%     233857 ±  3%  interrupts.CPU60.LOC:Local_timer_interrupts
      2285 ± 53%     -55.3%       1021 ± 85%  interrupts.CPU60.NMI:Non-maskable_interrupts
      2285 ± 53%     -55.3%       1021 ± 85%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      1213 ±  4%     +16.2%       1410        interrupts.CPU61.CAL:Function_call_interrupts
    200849           +16.4%     233812 ±  3%  interrupts.CPU61.LOC:Local_timer_interrupts
      1250           +12.7%       1408        interrupts.CPU62.CAL:Function_call_interrupts
    200858           +16.4%     233847 ±  3%  interrupts.CPU62.LOC:Local_timer_interrupts
      1249           +12.7%       1408        interrupts.CPU63.CAL:Function_call_interrupts
    200846           +16.3%     233509 ±  3%  interrupts.CPU63.LOC:Local_timer_interrupts
      1249           +12.6%       1407        interrupts.CPU64.CAL:Function_call_interrupts
    201010           +16.3%     233868 ±  3%  interrupts.CPU64.LOC:Local_timer_interrupts
      1249           +12.6%       1407        interrupts.CPU65.CAL:Function_call_interrupts
    201188           +16.2%     233840 ±  3%  interrupts.CPU65.LOC:Local_timer_interrupts
      1249           +12.6%       1405        interrupts.CPU66.CAL:Function_call_interrupts
    201142           +16.3%     233885 ±  3%  interrupts.CPU66.LOC:Local_timer_interrupts
      1249           +12.6%       1406        interrupts.CPU67.CAL:Function_call_interrupts
    200840           +16.5%     233913 ±  3%  interrupts.CPU67.LOC:Local_timer_interrupts
      1249           +12.7%       1408        interrupts.CPU68.CAL:Function_call_interrupts
    200815           +16.4%     233774 ±  3%  interrupts.CPU68.LOC:Local_timer_interrupts
      1249           +12.8%       1409        interrupts.CPU69.CAL:Function_call_interrupts
    200983           +16.3%     233829 ±  3%  interrupts.CPU69.LOC:Local_timer_interrupts
    200912           +17.1%     235169 ±  2%  interrupts.CPU7.LOC:Local_timer_interrupts
      1249           +13.0%       1412        interrupts.CPU70.CAL:Function_call_interrupts
    200821           +16.4%     233823 ±  3%  interrupts.CPU70.LOC:Local_timer_interrupts
      1247           +12.9%       1408        interrupts.CPU71.CAL:Function_call_interrupts
    200673           +16.4%     233586 ±  3%  interrupts.CPU71.LOC:Local_timer_interrupts
      3112 ± 28%     -55.8%       1376 ± 92%  interrupts.CPU71.NMI:Non-maskable_interrupts
      3112 ± 28%     -55.8%       1376 ± 92%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      2655 ± 18%     +53.3%       4071 ± 21%  interrupts.CPU71.RES:Rescheduling_interrupts
      1088 ± 16%     +25.1%       1361 ±  4%  interrupts.CPU8.CAL:Function_call_interrupts
    201291           +16.9%     235317 ±  2%  interrupts.CPU8.LOC:Local_timer_interrupts
    201246           +16.4%     234250        interrupts.CPU9.LOC:Local_timer_interrupts
  14452726           +16.7%   16865906 ±  2%  interrupts.LOC:Local_timer_interrupts
     25553 ±  7%     +26.4%      32295        softirqs.CPU0.RCU
     17578           +14.6%      20147 ±  2%  softirqs.CPU0.SCHED
     40851           +20.4%      49167 ±  3%  softirqs.CPU0.TIMER
     40546           +17.0%      47455 ±  5%  softirqs.CPU1.TIMER
     25814           +27.1%      32807 ± 10%  softirqs.CPU10.RCU
     38717 ±  2%     +14.6%      44386 ±  5%  softirqs.CPU10.TIMER
     38245           +16.3%      44468 ±  5%  softirqs.CPU11.TIMER
     26439 ±  4%     +21.3%      32083 ±  4%  softirqs.CPU12.RCU
     15582 ±  4%     +14.4%      17827 ±  2%  softirqs.CPU12.SCHED
     38686           +17.5%      45474 ±  4%  softirqs.CPU12.TIMER
     25962 ±  4%     +18.3%      30720 ±  4%  softirqs.CPU13.RCU
     16126 ±  5%     +17.9%      19013 ±  5%  softirqs.CPU13.SCHED
     25846 ±  2%     +20.4%      31108 ±  5%  softirqs.CPU14.RCU
     15629 ±  4%     +21.3%      18953 ±  6%  softirqs.CPU14.SCHED
     37591           +17.9%      44326 ±  5%  softirqs.CPU14.TIMER
     24676 ±  2%     +20.2%      29658        softirqs.CPU15.RCU
     38217           +16.0%      44331 ±  4%  softirqs.CPU15.TIMER
     24680 ±  2%     +18.6%      29278 ±  5%  softirqs.CPU16.RCU
     38783           +17.9%      45709 ±  4%  softirqs.CPU16.TIMER
     24738           +20.6%      29833 ± 10%  softirqs.CPU17.RCU
     39088           +16.3%      45476 ±  5%  softirqs.CPU17.TIMER
     27971           +24.7%      34880 ±  4%  softirqs.CPU18.RCU
     27800           +19.7%      33276 ±  2%  softirqs.CPU19.RCU
     16069 ±  5%     +14.6%      18417 ±  3%  softirqs.CPU19.SCHED
     39339           +18.2%      46516 ±  5%  softirqs.CPU19.TIMER
     26084           +10.4%      28804 ±  5%  softirqs.CPU2.RCU
     16601 ±  8%     +15.0%      19085 ± 10%  softirqs.CPU2.SCHED
     38625           +22.4%      47291 ±  5%  softirqs.CPU2.TIMER
     27472           +27.0%      34877 ±  7%  softirqs.CPU20.RCU
     16088 ±  3%     +10.7%      17811 ±  4%  softirqs.CPU20.SCHED
     38845           +17.6%      45674 ±  3%  softirqs.CPU20.TIMER
     28205 ±  6%     +20.2%      33906 ±  7%  softirqs.CPU21.RCU
     39324           +13.8%      44753 ±  4%  softirqs.CPU21.TIMER
     27282 ±  9%     +20.0%      32736 ±  4%  softirqs.CPU22.RCU
     39542           +13.9%      45038 ±  3%  softirqs.CPU22.TIMER
     27295 ±  7%     +22.3%      33387 ±  2%  softirqs.CPU23.RCU
     15694 ±  3%     +13.2%      17772 ±  3%  softirqs.CPU23.SCHED
     37452           +16.5%      43645 ±  5%  softirqs.CPU23.TIMER
     25529 ± 10%     +30.1%      33204        softirqs.CPU24.RCU
     16068 ±  6%     +13.8%      18289 ±  4%  softirqs.CPU24.SCHED
     39129           +29.4%      50648 ± 17%  softirqs.CPU24.TIMER
     27444 ±  3%     +18.3%      32463 ±  4%  softirqs.CPU25.RCU
     38658           +14.6%      44286 ±  6%  softirqs.CPU25.TIMER
     27586 ±  2%     +21.4%      33496        softirqs.CPU26.RCU
     39631           +13.9%      45136 ±  4%  softirqs.CPU26.TIMER
     27618 ±  3%     +15.9%      32011 ±  2%  softirqs.CPU27.RCU
     26884           +24.1%      33355 ±  7%  softirqs.CPU28.RCU
     39120           +14.6%      44813 ±  5%  softirqs.CPU28.TIMER
     27542 ±  4%     +21.4%      33433 ±  5%  softirqs.CPU29.RCU
     38659 ±  2%     +15.9%      44815 ±  5%  softirqs.CPU29.TIMER
     27539 ±  3%     +14.2%      31446        softirqs.CPU3.RCU
     16465 ±  4%     +16.6%      19192 ±  6%  softirqs.CPU3.SCHED
     38875           +17.9%      45849 ±  5%  softirqs.CPU3.TIMER
     26139           +23.5%      32277 ±  4%  softirqs.CPU30.RCU
     39666           +11.6%      44285 ±  5%  softirqs.CPU30.TIMER
     26559 ±  2%     +23.7%      32855 ±  8%  softirqs.CPU31.RCU
     16278 ±  2%     +19.3%      19420 ± 10%  softirqs.CPU31.SCHED
     39178           +15.2%      45121 ±  4%  softirqs.CPU31.TIMER
     25781           +20.9%      31163 ±  2%  softirqs.CPU32.RCU
     37670           +13.6%      42811 ±  5%  softirqs.CPU32.TIMER
     27107 ±  5%     +16.7%      31625 ±  4%  softirqs.CPU33.RCU
     16098 ±  2%     +12.3%      18071 ±  2%  softirqs.CPU33.SCHED
     38565           +15.6%      44572 ±  4%  softirqs.CPU33.TIMER
     26241 ±  3%     +18.9%      31209 ±  5%  softirqs.CPU34.RCU
     38223           +18.1%      45160 ±  5%  softirqs.CPU34.TIMER
     25360 ±  6%     +27.0%      32210 ±  2%  softirqs.CPU35.RCU
     39986 ±  2%     +13.8%      45494 ±  3%  softirqs.CPU35.TIMER
     26626 ±  2%     +23.9%      32987 ±  4%  softirqs.CPU36.RCU
     16281 ±  5%     +12.6%      18329        softirqs.CPU36.SCHED
     27227           +23.8%      33703 ±  2%  softirqs.CPU37.RCU
     26701 ±  2%     +29.1%      34462 ±  8%  softirqs.CPU38.RCU
     15157 ±  5%     +15.5%      17507 ±  4%  softirqs.CPU38.SCHED
     37119 ±  4%     +37.9%      51191 ± 27%  softirqs.CPU38.TIMER
     28007 ±  2%     +21.4%      33990 ±  2%  softirqs.CPU39.RCU
     15700 ±  2%     +11.8%      17545 ±  3%  softirqs.CPU39.SCHED
     37509 ±  2%     +14.9%      43109 ±  6%  softirqs.CPU39.TIMER
     25877           +24.6%      32254 ±  3%  softirqs.CPU4.RCU
     38648           +16.9%      45191 ±  5%  softirqs.CPU4.TIMER
     27903           +17.8%      32867 ±  2%  softirqs.CPU40.RCU
     15256 ±  2%     +12.8%      17202 ±  7%  softirqs.CPU40.SCHED
     36371 ±  2%     +17.0%      42572 ±  4%  softirqs.CPU40.TIMER
     26560 ±  5%     +19.9%      31840 ±  6%  softirqs.CPU41.RCU
     15067 ±  3%     +13.8%      17142 ±  4%  softirqs.CPU41.SCHED
     36473 ±  3%     +14.2%      41657 ±  4%  softirqs.CPU41.TIMER
     27250           +22.5%      33387 ±  2%  softirqs.CPU42.RCU
     15376 ±  4%     +15.4%      17744 ±  3%  softirqs.CPU42.SCHED
     37838           +13.4%      42895 ±  3%  softirqs.CPU42.TIMER
     27137 ±  4%     +23.2%      33425        softirqs.CPU43.RCU
     27818 ±  3%     +22.5%      34073 ±  6%  softirqs.CPU44.RCU
     14305 ±  4%     +22.5%      17522 ±  2%  softirqs.CPU44.SCHED
     36151 ±  4%     +16.7%      42187 ±  4%  softirqs.CPU44.TIMER
     25734 ±  3%     +25.6%      32319 ±  4%  softirqs.CPU45.RCU
     14522           +14.8%      16674 ±  6%  softirqs.CPU45.SCHED
     36594           +15.8%      42364 ±  5%  softirqs.CPU45.TIMER
     28660 ±  3%     +18.8%      34052 ±  7%  softirqs.CPU46.RCU
     15306 ±  4%     +13.5%      17370 ±  7%  softirqs.CPU46.SCHED
     37095 ±  3%     +14.3%      42388 ±  5%  softirqs.CPU46.TIMER
     28906 ±  5%     +15.3%      33324 ±  7%  softirqs.CPU47.RCU
     14758 ±  5%     +13.7%      16775 ±  5%  softirqs.CPU47.SCHED
     36736 ±  3%     +14.2%      41969 ±  4%  softirqs.CPU47.TIMER
     28049 ±  4%     +24.0%      34791 ±  2%  softirqs.CPU48.RCU
     14583 ±  4%     +18.9%      17340 ±  8%  softirqs.CPU48.SCHED
     36095 ±  4%     +19.5%      43117 ±  6%  softirqs.CPU48.TIMER
     27551 ±  5%     +21.0%      33349 ±  4%  softirqs.CPU49.RCU
     36024           +18.1%      42545 ±  4%  softirqs.CPU49.TIMER
     25870 ±  2%     +22.4%      31677 ±  3%  softirqs.CPU5.RCU
     15740 ±  6%     +26.8%      19951 ±  9%  softirqs.CPU5.SCHED
     37622           +33.5%      50234 ± 18%  softirqs.CPU5.TIMER
     27424           +19.6%      32798 ±  2%  softirqs.CPU50.RCU
     14010 ±  2%     +21.8%      17060 ±  6%  softirqs.CPU50.SCHED
     36485 ±  3%     +14.7%      41862 ±  4%  softirqs.CPU50.TIMER
     27884           +18.9%      33160 ±  3%  softirqs.CPU51.RCU
     36212 ±  3%     +16.6%      42210 ±  4%  softirqs.CPU51.TIMER
     27031 ±  7%     +29.8%      35075 ±  5%  softirqs.CPU52.RCU
     14772 ±  2%     +18.1%      17451 ±  4%  softirqs.CPU52.SCHED
     37031 ±  4%     +17.9%      43648 ±  4%  softirqs.CPU52.TIMER
     25188 ± 12%     +28.7%      32423 ±  3%  softirqs.CPU53.RCU
     16289 ±  6%     +18.4%      19280 ±  5%  softirqs.CPU53.SCHED
     37616 ±  3%     +35.6%      51010 ± 21%  softirqs.CPU53.TIMER
     26693 ±  3%     +19.4%      31870 ±  2%  softirqs.CPU54.RCU
     38094           +16.5%      44379 ±  4%  softirqs.CPU54.TIMER
     26082 ±  2%     +23.1%      32100        softirqs.CPU55.RCU
     37589 ±  2%     +16.5%      43779 ±  5%  softirqs.CPU55.TIMER
     26352 ±  2%     +18.7%      31284        softirqs.CPU56.RCU
     15591 ±  4%     +15.9%      18066 ±  8%  softirqs.CPU56.SCHED
     37070           +16.8%      43309 ±  2%  softirqs.CPU56.TIMER
     27520 ±  4%     +15.7%      31841 ±  5%  softirqs.CPU57.RCU
     37916 ±  2%     +13.9%      43170 ±  2%  softirqs.CPU57.TIMER
     27067           +20.8%      32709 ±  7%  softirqs.CPU58.RCU
     15059           +19.1%      17935 ±  8%  softirqs.CPU58.SCHED
     37123           +13.8%      42241        softirqs.CPU58.TIMER
     25199 ±  4%     +19.6%      30126 ±  6%  softirqs.CPU59.RCU
     36384 ±  2%     +16.8%      42479 ±  2%  softirqs.CPU59.TIMER
     25871 ±  3%     +17.5%      30409 ±  3%  softirqs.CPU6.RCU
     16584 ±  2%     +14.8%      19032 ±  5%  softirqs.CPU6.SCHED
     38784           +15.6%      44850 ±  4%  softirqs.CPU6.TIMER
     24777 ±  5%     +15.0%      28502 ±  5%  softirqs.CPU60.RCU
     37468           +14.2%      42794 ±  6%  softirqs.CPU60.TIMER
     23108 ±  3%     +30.1%      30065 ±  2%  softirqs.CPU61.RCU
     15059 ±  6%     +16.5%      17540 ±  6%  softirqs.CPU61.SCHED
     37002           +16.8%      43227 ±  3%  softirqs.CPU61.TIMER
     23023 ±  7%     +33.6%      30769 ±  6%  softirqs.CPU62.RCU
     37164 ±  2%     +16.2%      43203 ±  2%  softirqs.CPU62.TIMER
     23069 ±  2%     +25.1%      28850        softirqs.CPU63.RCU
     14642 ±  5%     +15.4%      16900 ±  8%  softirqs.CPU63.SCHED
     36254 ±  2%     +15.4%      41854 ±  3%  softirqs.CPU63.TIMER
     24304 ±  2%     +25.9%      30593 ±  2%  softirqs.CPU64.RCU
     14666           +15.5%      16938 ±  3%  softirqs.CPU64.SCHED
     37048           +15.7%      42877 ±  4%  softirqs.CPU64.TIMER
     24438 ±  9%     +23.6%      30212 ±  5%  softirqs.CPU65.RCU
     37505           +12.8%      42288 ±  2%  softirqs.CPU65.TIMER
     24568           +20.4%      29569        softirqs.CPU66.RCU
     15290 ±  5%      +8.4%      16569 ±  4%  softirqs.CPU66.SCHED
     37381           +14.9%      42968 ±  3%  softirqs.CPU66.TIMER
     23253 ±  9%     +29.7%      30150 ±  2%  softirqs.CPU67.RCU
     14645 ±  6%     +13.4%      16613 ±  6%  softirqs.CPU67.SCHED
     37012           +14.1%      42221 ±  2%  softirqs.CPU67.TIMER
     36993           +15.0%      42536 ±  4%  softirqs.CPU68.TIMER
     24695 ±  7%     +23.7%      30551 ±  3%  softirqs.CPU69.RCU
     14441 ±  3%     +11.4%      16089 ±  4%  softirqs.CPU69.SCHED
     36999           +14.3%      42299        softirqs.CPU69.TIMER
     26022 ±  3%     +21.4%      31579 ±  2%  softirqs.CPU7.RCU
     38481           +18.7%      45661 ±  3%  softirqs.CPU7.TIMER
     25083 ±  8%     +18.5%      29721 ±  2%  softirqs.CPU70.RCU
     14845 ±  7%     +15.7%      17173 ±  4%  softirqs.CPU70.SCHED
     37094           +18.5%      43948 ±  3%  softirqs.CPU70.TIMER
     24438 ±  2%     +20.6%      29461 ±  2%  softirqs.CPU71.RCU
     15821 ±  4%     +13.6%      17978 ±  4%  softirqs.CPU71.SCHED
     26912 ±  7%     +22.3%      32925 ±  6%  softirqs.CPU8.RCU
     39193           +15.2%      45167 ±  4%  softirqs.CPU8.TIMER
     25829 ±  2%     +20.7%      31178        softirqs.CPU9.RCU
     37434 ±  2%     +16.6%      43665 ±  5%  softirqs.CPU9.TIMER
   1897274           +21.5%    2304570 ±  2%  softirqs.RCU
   1150548           +11.9%    1287616        softirqs.SCHED
   2757329           +15.8%    3193002 ±  3%  softirqs.TIMER


                                                                                
                                 aim7.jobs-per-min                              
                                                                                
  1400 +-+------------------------------------------------------------------+   
       |                                                                    |   
  1200 +-+++.++.++.+++.++.++  +  ++.+  +  +++.++  +  ++.++.++.++.+++.++.++.+|   
       |                   :  :  :  :  :  :    :  :  :                      |   
  1000 OO+OO OO OO OOO OO OO OOO OO OO OO OOO OO OO OOO O                   |   
       |                    : :  :  :  :  :     : :: :                      |   
   800 +-+                  : : :   : :: :      : :: :                      |   
       |                    : : :   : :: :      : :: :                      |   
   600 +-+                  : :::    :: ::      : :: :                      |   
       |                    :: ::    :: ::      :: ::                       |   
   400 +-+                  :: ::    :: ::      :: ::                       |   
       |                     : ::    :: ::       :  :                       |   
   200 +-+                   : :     :  :        :  :                       |   
       |                     : :     :  :        :  :                       |   
     0 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  140 +-+-------------------------------------------------------------------+   
      |                                                                     |   
  120 OO+OO OO OO OO OO O O OO OO  O OO OOO OO OO OO OO O                   |   
      |                  O        O                                         |   
  100 +-+++.++.++.++.++.+++  +  +.++  +  ++.++.+  +  ++.++.+++.++.++.++.++.+|   
      |                   :  :  :  :  :  :     :  :  :                      |   
   80 +-+                 :  :  :  :  :  :     :  :  :                      |   
      |                    : :: :   : :: :     : :: :                       |   
   60 +-+                  : :: :   : :: :     : :: :                       |   
      |                    : :: :   : :: :      :: ::                       |   
   40 +-+                  :: ::    :: ::       :: ::                       |   
      |                    :: ::    :: ::       :: ::                       |   
   20 +-+                   :  :     :  :       :  :                        |   
      |                     :  :     :  :       :  :                        |   
    0 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  140 +-+-------------------------------------------------------------------+   
      |                                                                     |   
  120 OO+OO OO OO OO OO O O OO OO  O OO OOO OO OO OO OO O                   |   
      |                  O        O                                         |   
  100 +-+++.++.++.++.++.+++  +  +.++  +  ++.++.+  +  ++.++.+++.++.++.++.++.+|   
      |                   :  :  :  :  :  :     :  :  :                      |   
   80 +-+                 :  :  :  :  :  :     :  :  :                      |   
      |                    : :: :   : :: :     : :: :                       |   
   60 +-+                  : :: :   : :: :     : :: :                       |   
      |                    : :: :   : :: :      :: ::                       |   
   40 +-+                  :: ::    :: ::       :: ::                       |   
      |                    :: ::    :: ::       :: ::                       |   
   20 +-+                   :  :     :  :       :  :                        |   
      |                     :  :     :  :       :  :                        |   
    0 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-8/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/sync_disk_rw/aim7

commit: 
  a92e9c3a3c ("btrfs: extent_io: Handle error better in extent_writepages()")
  44fe89de7d ("btrfs: Do mandatory tree block check before submitting bio")

a92e9c3a3cef4c0c 44fe89de7d5157a4f31f13d9480 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    474.71           -20.8%     376.09        aim7.jobs-per-min
    252.89           +26.2%     319.19        aim7.time.elapsed_time
    252.89           +26.2%     319.19        aim7.time.elapsed_time.max
      8376 ±  3%      +9.4%       9167 ±  3%  aim7.time.minor_page_faults
      1760            +6.3%       1872        aim7.time.system_time
     22.58            -1.5%      22.24        boot-time.dhcp
     40945 ±  2%     -22.3%      31813 ±  3%  meminfo.max_used_kB
    670014           +24.2%     832471        proc-vmstat.pgfault
   1013780           +13.7%    1152374        proc-vmstat.pgfree
 3.022e+09 ±  4%     +50.2%   4.54e+09 ± 19%  cpuidle.C1.time
  34330913 ±  2%     +16.1%   39848597 ±  9%  cpuidle.C1.usage
 2.611e+09 ± 16%     +48.9%  3.887e+09 ±  3%  cpuidle.C6.time
      0.02 ± 15%      -0.0        0.00 ±113%  mpstat.cpu.-1.soft%
     21.36            -3.4       17.98        mpstat.cpu.-1.sys%
      0.14            -0.0        0.12        mpstat.cpu.-1.usr%
     78.00            +4.8%      81.75        vmstat.cpu.id
    138672           -21.1%     109467        vmstat.io.bo
    299636           -21.0%     236781        vmstat.system.cs
     78.64            +4.3%      81.99        iostat.cpu.idle
     21.21           -15.7%      17.88        iostat.cpu.system
      7947           -20.5%       6318        iostat.md0.w/s
    180043           -20.9%     142377        iostat.md0.wkB/s
      3098 ± 30%    +270.4%      11475 ± 34%  numa-meminfo.node0.Inactive(anon)
      9526           +37.0%      13052 ± 15%  numa-meminfo.node0.Mapped
      5474 ± 26%    +188.7%      15807 ± 23%  numa-meminfo.node0.Shmem
     75040 ±  2%     +10.4%      82807 ±  7%  numa-meminfo.node0.Slab
     16192 ±  5%     -51.6%       7838 ± 51%  numa-meminfo.node1.Inactive(anon)
     20174 ±  7%     -50.1%      10076 ± 36%  numa-meminfo.node1.Shmem
    774.25 ± 30%    +270.4%       2867 ± 34%  numa-vmstat.node0.nr_inactive_anon
      2474 ±  2%     +35.6%       3354 ± 14%  numa-vmstat.node0.nr_mapped
      1367 ± 26%    +188.9%       3950 ± 23%  numa-vmstat.node0.nr_shmem
    774.25 ± 30%    +270.4%       2867 ± 34%  numa-vmstat.node0.nr_zone_inactive_anon
      4046 ±  5%     -51.6%       1958 ± 51%  numa-vmstat.node1.nr_inactive_anon
      5042 ±  7%     -50.1%       2518 ± 36%  numa-vmstat.node1.nr_shmem
      4046 ±  5%     -51.6%       1958 ± 51%  numa-vmstat.node1.nr_zone_inactive_anon
    312.25           -15.1%     265.25        turbostat.Avg_MHz
     25.87            -3.9       22.02        turbostat.Busy%
  34326534 ±  2%     +16.1%   39845190 ±  9%  turbostat.C1
     33.45            -4.6%      31.92        turbostat.CorWatt
  21163841           +25.1%   26466542        turbostat.IRQ
     59.45            -2.7%      57.87        turbostat.PkgWatt
     32.25            -1.7%      31.71        turbostat.RAMWatt
     20330           +26.1%      25630        turbostat.SMI
      6.46 ±  2%      -0.6        5.89 ±  3%  perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
      2.75 ±  5%      -0.6        2.19 ± 11%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      5.46 ±  2%      -0.5        4.92 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
      5.46 ±  2%      -0.5        4.93 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      0.69 ±  9%      -0.4        0.27 ±100%  perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.__vfs_write.vfs_write
      1.74 ±  6%      -0.4        1.39 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write
      0.63 ±  3%      -0.3        0.30 ±100%  perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio.extent_writepages
      2.35 ±  4%      -0.2        2.13 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents
      0.92 ±  9%      -0.1        0.80 ±  7%  perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
      0.86 ± 10%      -0.1        0.74 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages
      0.85 ±  9%      -0.1        0.73 ±  8%  perf-profile.calltrace.cycles-pp.cow_file_range.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
      0.70 ±  5%      -0.1        0.57 ±  3%  perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      1.40 ±  5%      +0.2        1.55        perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.65 ± 11%      +0.3        0.94 ± 23%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.79 ±  8%      +0.3        1.13 ± 21%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.26 ±100%      +0.3        0.61 ±  7%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      1.44 ±  8%      +0.4        1.87 ± 17%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.14 ±173%      +0.6        0.70 ± 27%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      2.44 ±  5%      +0.6        3.04 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write
      0.00            +0.7        0.72 ±  2%  perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook
      2.51 ±  6%      +0.8        3.30 ± 16%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      2.71 ±  6%      +0.8        3.53 ± 16%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.73 ±  6%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write
      0.72 ±  6%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
      0.72 ±  6%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
      0.72 ±  6%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
      0.00            +1.0        1.05        perf-profile.calltrace.cycles-pp.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
      0.00            +1.1        1.08        perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages
      0.00            +1.1        1.09        perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages
      0.00            +1.5        1.46        perf-profile.calltrace.cycles-pp.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
      0.00            +1.5        1.46        perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
     71.60 ±  7%     -16.2%      59.98 ±  8%  sched_debug.cfs_rq:/.load_avg.avg
     96.11 ± 10%     -16.2%      80.58 ±  6%  sched_debug.cfs_rq:/.load_avg.stddev
    187997            -8.8%     171461 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
    226472 ±  2%     -10.8%     202016 ±  6%  sched_debug.cfs_rq:/.min_vruntime.max
     17261 ±  7%     -21.5%      13550 ± 15%  sched_debug.cfs_rq:/.min_vruntime.stddev
      8.89 ± 47%     -64.7%       3.14 ± 57%  sched_debug.cfs_rq:/.removed.load_avg.avg
    204.15           -38.5%     125.58 ± 57%  sched_debug.cfs_rq:/.removed.load_avg.max
     40.32 ± 22%     -51.4%      19.61 ± 57%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    409.91 ± 47%     -64.8%     144.41 ± 57%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      9437           -38.8%       5776 ± 57%  sched_debug.cfs_rq:/.removed.runnable_sum.max
      1859 ± 22%     -51.5%     901.87 ± 57%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      2.95 ± 44%     -79.5%       0.60 ± 75%  sched_debug.cfs_rq:/.removed.util_avg.avg
     72.25 ±  8%     -66.6%      24.17 ± 75%  sched_debug.cfs_rq:/.removed.util_avg.max
     13.53 ± 23%     -72.1%       3.77 ± 75%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      9.43 ±  5%     +27.0%      11.97 ± 14%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    103.80 ± 10%     +62.1%     168.29 ± 27%  sched_debug.cfs_rq:/.runnable_load_avg.max
     22.41 ±  7%     +39.0%      31.15 ± 20%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     12421 ± 59%     +64.4%      20419 ± 40%  sched_debug.cfs_rq:/.spread0.avg
    -24890           -69.6%      -7575        sched_debug.cfs_rq:/.spread0.min
     17263 ±  7%     -21.5%      13551 ± 15%  sched_debug.cfs_rq:/.spread0.stddev
    325.68 ±  2%     -14.6%     278.27 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
    186.27 ±  3%     -11.0%     165.84 ±  2%  sched_debug.cfs_rq:/.util_avg.stddev
    132.45           -12.4%     116.05 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    155113           +19.3%     185086        sched_debug.cpu.clock.avg
    155121           +19.3%     185093        sched_debug.cpu.clock.max
    155106           +19.3%     185079        sched_debug.cpu.clock.min
    155113           +19.3%     185086        sched_debug.cpu.clock_task.avg
    155121           +19.3%     185093        sched_debug.cpu.clock_task.max
    155106           +19.3%     185079        sched_debug.cpu.clock_task.min
    104.40 ±  7%     +61.5%     168.62 ± 27%  sched_debug.cpu.cpu_load[0].max
     96.20 ±  8%     +74.1%     167.46 ± 30%  sched_debug.cpu.cpu_load[1].max
      4871           +14.8%       5592        sched_debug.cpu.curr->pid.max
    127731           +23.5%     157685        sched_debug.cpu.nr_load_updates.avg
    134899           +20.9%     163058        sched_debug.cpu.nr_load_updates.max
    125325           +24.0%     155428        sched_debug.cpu.nr_load_updates.min
    155106           +19.3%     185079        sched_debug.cpu_clk
    151513           +19.8%     181485        sched_debug.ktime
    155537           +19.3%     185526        sched_debug.sched_clk
     19.77           -15.1%      16.78 ±  3%  perf-stat.i.MPKI
 8.783e+08            -9.0%  7.989e+08        perf-stat.i.branch-instructions
     19.44            -0.2       19.21        perf-stat.i.cache-miss-rate%
  14932557           -19.1%   12081970        perf-stat.i.cache-misses
  77250252           -18.0%   63363496        perf-stat.i.cache-references
    302132           -21.2%     238146        perf-stat.i.context-switches
      3.13           -11.6%       2.77 ±  2%  perf-stat.i.cpi
 1.254e+10           -15.4%  1.061e+10        perf-stat.i.cpu-cycles
    229.12 ±  3%     -21.3%     180.36 ±  3%  perf-stat.i.cpu-migrations
    840.12            +4.5%     877.89        perf-stat.i.cycles-between-cache-misses
 1.129e+09            -5.4%  1.068e+09        perf-stat.i.dTLB-loads
      0.10 ±  4%      -0.0        0.09 ±  5%  perf-stat.i.dTLB-store-miss-rate%
    538827 ±  3%     -16.1%     452028 ±  4%  perf-stat.i.dTLB-store-misses
     47.32            +3.3       50.67 ±  3%  perf-stat.i.iTLB-load-miss-rate%
   2556829           -20.4%    2034861 ±  2%  perf-stat.i.iTLB-loads
 4.153e+09            -4.5%  3.964e+09        perf-stat.i.instructions
      0.33           +12.2%       0.37 ±  2%  perf-stat.i.ipc
   6672298           -19.6%    5364393        perf-stat.i.node-load-misses
   7193961           -19.6%    5786904        perf-stat.i.node-loads
   4765580           -20.6%    3784168        perf-stat.i.node-store-misses
   6965089           -20.3%    5548679        perf-stat.i.node-stores
     18.60           -14.0%      15.99 ±  2%  perf-stat.overall.MPKI
     19.33            -0.3       19.07        perf-stat.overall.cache-miss-rate%
      3.02           -11.3%       2.68 ±  2%  perf-stat.overall.cpi
    839.71            +4.6%     878.19        perf-stat.overall.cycles-between-cache-misses
      0.10 ±  4%      -0.0        0.08 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
     45.63            +3.8       49.47 ±  3%  perf-stat.overall.iTLB-load-miss-rate%
      0.33           +12.8%       0.37 ±  2%  perf-stat.overall.ipc
 8.748e+08            -9.0%  7.964e+08        perf-stat.ps.branch-instructions
  14872846           -19.0%   12043692        perf-stat.ps.cache-misses
  76941024           -17.9%   63162394        perf-stat.ps.cache-references
    300917           -21.1%     237386        perf-stat.ps.context-switches
 1.249e+10           -15.3%  1.058e+10        perf-stat.ps.cpu-cycles
    228.20 ±  3%     -21.2%     179.78 ±  3%  perf-stat.ps.cpu-migrations
 1.124e+09            -5.3%  1.065e+09        perf-stat.ps.dTLB-loads
    536681 ±  3%     -16.0%     450604 ±  4%  perf-stat.ps.dTLB-store-misses
   2546543           -20.3%    2028370 ±  2%  perf-stat.ps.iTLB-loads
 4.137e+09            -4.5%  3.952e+09        perf-stat.ps.instructions
   6645469           -19.5%    5347283        perf-stat.ps.node-load-misses
   7165042           -19.5%    5768454        perf-stat.ps.node-loads
   4746415           -20.5%    3772102        perf-stat.ps.node-store-misses
   6937096           -20.3%    5531006        perf-stat.ps.node-stores
 1.049e+12           +20.4%  1.263e+12        perf-stat.total.instructions
    160.75 ± 18%     +49.5%     240.25 ± 27%  interrupts.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5
    123.75           +26.3%     156.25        interrupts.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
    100313 ±  7%     +13.4%     113737        interrupts.CAL:Function_call_interrupts
    508519           +26.0%     640668        interrupts.CPU0.LOC:Local_timer_interrupts
      2437 ± 12%     +18.1%       2877 ±  4%  interrupts.CPU1.CAL:Function_call_interrupts
    508036           +26.1%     640579        interrupts.CPU1.LOC:Local_timer_interrupts
      2521 ± 10%     +14.3%       2882 ±  5%  interrupts.CPU10.CAL:Function_call_interrupts
    507477           +26.1%     639993        interrupts.CPU10.LOC:Local_timer_interrupts
      2519 ±  2%     +17.9%       2969 ±  6%  interrupts.CPU11.CAL:Function_call_interrupts
    508051           +26.1%     640531        interrupts.CPU11.LOC:Local_timer_interrupts
    508353           +25.8%     639418        interrupts.CPU12.LOC:Local_timer_interrupts
    508289           +26.0%     640385        interrupts.CPU13.LOC:Local_timer_interrupts
    508136           +25.8%     639470        interrupts.CPU14.LOC:Local_timer_interrupts
      2417 ±  5%     +20.2%       2906 ±  6%  interrupts.CPU15.CAL:Function_call_interrupts
    508330           +25.9%     640229        interrupts.CPU15.LOC:Local_timer_interrupts
    508351           +25.9%     639790        interrupts.CPU16.LOC:Local_timer_interrupts
      2508 ± 10%     +18.4%       2968 ±  6%  interrupts.CPU17.CAL:Function_call_interrupts
    508084           +26.0%     640420        interrupts.CPU17.LOC:Local_timer_interrupts
     17327 ± 18%     +26.1%      21856 ±  4%  interrupts.CPU17.RES:Rescheduling_interrupts
    508457           +25.8%     639737        interrupts.CPU18.LOC:Local_timer_interrupts
    508140           +26.0%     640214        interrupts.CPU19.LOC:Local_timer_interrupts
      2280 ±  8%     +22.2%       2786 ±  5%  interrupts.CPU2.CAL:Function_call_interrupts
    508336           +26.0%     640548        interrupts.CPU2.LOC:Local_timer_interrupts
    507912           +26.1%     640315        interrupts.CPU20.LOC:Local_timer_interrupts
      2595 ±  7%     +13.4%       2943 ±  5%  interrupts.CPU21.CAL:Function_call_interrupts
    508108           +26.1%     640485        interrupts.CPU21.LOC:Local_timer_interrupts
    508306           +26.0%     640244        interrupts.CPU22.LOC:Local_timer_interrupts
      1303 ± 34%     -29.2%     922.75 ± 32%  interrupts.CPU22.NMI:Non-maskable_interrupts
      1303 ± 34%     -29.2%     922.75 ± 32%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      2522 ±  9%     +17.2%       2956 ±  4%  interrupts.CPU23.CAL:Function_call_interrupts
    508123           +26.1%     640526        interrupts.CPU23.LOC:Local_timer_interrupts
    508372           +26.0%     640372        interrupts.CPU24.LOC:Local_timer_interrupts
    507678           +26.1%     640435        interrupts.CPU25.LOC:Local_timer_interrupts
    508375           +26.0%     640407        interrupts.CPU26.LOC:Local_timer_interrupts
      2406 ± 11%     +19.9%       2885 ±  5%  interrupts.CPU27.CAL:Function_call_interrupts
    508242           +26.0%     640265        interrupts.CPU27.LOC:Local_timer_interrupts
      2449 ± 11%     +21.8%       2982 ±  5%  interrupts.CPU28.CAL:Function_call_interrupts
    508292           +26.0%     640304        interrupts.CPU28.LOC:Local_timer_interrupts
      2510 ±  6%     +14.4%       2870 ±  5%  interrupts.CPU29.CAL:Function_call_interrupts
    508085           +26.0%     640167        interrupts.CPU29.LOC:Local_timer_interrupts
    508172           +26.0%     640443        interrupts.CPU3.LOC:Local_timer_interrupts
    508342           +25.9%     639891        interrupts.CPU30.LOC:Local_timer_interrupts
    508009           +26.1%     640402        interrupts.CPU31.LOC:Local_timer_interrupts
      2522 ± 11%     +16.8%       2946 ±  4%  interrupts.CPU32.CAL:Function_call_interrupts
    508138           +25.9%     639578        interrupts.CPU32.LOC:Local_timer_interrupts
    508010           +26.1%     640577        interrupts.CPU33.LOC:Local_timer_interrupts
    508029           +25.9%     639405        interrupts.CPU34.LOC:Local_timer_interrupts
      2464 ±  8%     +15.3%       2840 ±  5%  interrupts.CPU35.CAL:Function_call_interrupts
    508216           +26.0%     640171        interrupts.CPU35.LOC:Local_timer_interrupts
    160.75 ± 18%     +49.5%     240.25 ± 27%  interrupts.CPU36.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5
    507990           +25.9%     639575        interrupts.CPU36.LOC:Local_timer_interrupts
    508010           +26.0%     640200        interrupts.CPU37.LOC:Local_timer_interrupts
      2536 ±  8%     +13.9%       2888 ±  6%  interrupts.CPU38.CAL:Function_call_interrupts
    507853           +25.9%     639351        interrupts.CPU38.LOC:Local_timer_interrupts
      2414 ±  7%     +19.6%       2887 ±  7%  interrupts.CPU39.CAL:Function_call_interrupts
    507619           +26.0%     639416        interrupts.CPU39.LOC:Local_timer_interrupts
    508318           +25.9%     640069        interrupts.CPU4.LOC:Local_timer_interrupts
      2463 ± 12%     +26.6%       3119 ±  2%  interrupts.CPU5.CAL:Function_call_interrupts
    508112           +26.1%     640497        interrupts.CPU5.LOC:Local_timer_interrupts
    123.75           +26.3%     156.25        interrupts.CPU6.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
      2351 ±  9%     +21.5%       2856 ±  5%  interrupts.CPU6.CAL:Function_call_interrupts
    508563           +25.8%     639582        interrupts.CPU6.LOC:Local_timer_interrupts
      2592 ±  4%     +13.6%       2943 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
    508150           +26.0%     640378        interrupts.CPU7.LOC:Local_timer_interrupts
    508498           +25.9%     640139        interrupts.CPU8.LOC:Local_timer_interrupts
      2501 ± 10%     +18.4%       2960 ±  7%  interrupts.CPU9.CAL:Function_call_interrupts
    508092           +26.0%     640432        interrupts.CPU9.LOC:Local_timer_interrupts
  20326190           +26.0%   25605623        interrupts.LOC:Local_timer_interrupts
     26401 ±  5%      -9.9%      23785 ±  4%  interrupts.NMI:Non-maskable_interrupts
     26401 ±  5%      -9.9%      23785 ±  4%  interrupts.PMI:Performance_monitoring_interrupts
     69901 ±  3%     +27.9%      89422 ±  3%  softirqs.CPU0.RCU
     45503 ±  2%     +18.1%      53752        softirqs.CPU0.SCHED
     90281 ±  2%     +17.4%     105987 ±  2%  softirqs.CPU0.TIMER
     72001 ±  4%     +30.7%      94130 ±  2%  softirqs.CPU1.RCU
     42373           +18.9%      50374 ±  2%  softirqs.CPU1.SCHED
     89928 ±  3%     +18.5%     106554 ±  4%  softirqs.CPU1.TIMER
     73216 ±  6%     +24.5%      91156 ±  3%  softirqs.CPU10.RCU
     42693           +16.8%      49881 ±  2%  softirqs.CPU10.SCHED
     86975 ±  2%     +19.4%     103822 ±  3%  softirqs.CPU10.TIMER
     71035           +31.2%      93194 ±  2%  softirqs.CPU11.RCU
     42794           +19.6%      51168 ±  2%  softirqs.CPU11.SCHED
     86533           +18.9%     102845 ±  4%  softirqs.CPU11.TIMER
     70358 ±  3%     +33.3%      93810 ±  3%  softirqs.CPU12.RCU
     42548           +17.2%      49878        softirqs.CPU12.SCHED
     87303 ±  2%     +21.8%     106341 ±  3%  softirqs.CPU12.TIMER
     70364 ±  3%     +31.7%      92648 ±  4%  softirqs.CPU13.RCU
     42421           +18.9%      50428        softirqs.CPU13.SCHED
     86875           +19.5%     103817 ±  3%  softirqs.CPU13.TIMER
     69128 ±  2%     +30.7%      90331 ±  3%  softirqs.CPU14.RCU
     42975 ±  2%     +15.4%      49575        softirqs.CPU14.SCHED
     84688 ±  3%     +21.6%     102994 ±  3%  softirqs.CPU14.TIMER
     72010 ±  2%     +30.6%      94068 ±  3%  softirqs.CPU15.RCU
     42893           +17.0%      50178        softirqs.CPU15.SCHED
     86458           +20.4%     104117 ±  4%  softirqs.CPU15.TIMER
     70886 ±  2%     +30.3%      92387 ±  2%  softirqs.CPU16.RCU
     42970           +16.3%      49975        softirqs.CPU16.SCHED
     87944 ±  2%     +19.7%     105244 ±  2%  softirqs.CPU16.TIMER
     70757 ±  2%     +30.5%      92359 ±  2%  softirqs.CPU17.RCU
     42742           +17.7%      50310 ±  2%  softirqs.CPU17.SCHED
     87096 ±  2%     +17.9%     102652 ±  3%  softirqs.CPU17.TIMER
     69077 ±  3%     +32.6%      91630 ±  3%  softirqs.CPU18.RCU
     42942           +17.0%      50245 ±  2%  softirqs.CPU18.SCHED
     68722 ±  2%     +29.7%      89150 ±  2%  softirqs.CPU19.RCU
     42964 ±  2%     +17.2%      50334 ±  2%  softirqs.CPU19.SCHED
     84910 ±  4%     +24.0%     105326 ±  7%  softirqs.CPU19.TIMER
     72687 ±  4%     +25.6%      91315 ±  3%  softirqs.CPU2.RCU
     42688           +16.2%      49622        softirqs.CPU2.SCHED
     88869 ±  5%     +16.3%     103332        softirqs.CPU2.TIMER
     70693 ±  2%     +26.6%      89525 ±  4%  softirqs.CPU20.RCU
     42228           +17.5%      49611 ±  2%  softirqs.CPU20.SCHED
     88647 ±  2%     +16.4%     103227 ±  2%  softirqs.CPU20.TIMER
     70857 ±  3%     +32.1%      93633 ±  3%  softirqs.CPU21.RCU
     42878           +17.2%      50239        softirqs.CPU21.SCHED
     87325           +18.2%     103214 ±  4%  softirqs.CPU21.TIMER
     71099 ±  3%     +29.0%      91711 ±  3%  softirqs.CPU22.RCU
     42534           +17.3%      49888        softirqs.CPU22.SCHED
     87698 ±  3%     +19.3%     104642        softirqs.CPU22.TIMER
     72098 ±  2%     +30.3%      93930 ±  3%  softirqs.CPU23.RCU
     42771           +17.0%      50022 ±  2%  softirqs.CPU23.SCHED
     87931           +17.5%     103319 ±  3%  softirqs.CPU23.TIMER
     70804 ±  3%     +33.4%      94432 ±  6%  softirqs.CPU24.RCU
     42554 ±  3%     +16.4%      49516 ±  2%  softirqs.CPU24.SCHED
     71435 ±  3%     +35.0%      96465 ±  3%  softirqs.CPU25.RCU
     42605           +16.9%      49815        softirqs.CPU25.SCHED
     87163           +21.2%     105672        softirqs.CPU25.TIMER
     70216           +31.0%      91964 ±  3%  softirqs.CPU26.RCU
     42147           +19.0%      50165 ±  2%  softirqs.CPU26.SCHED
     88215 ±  3%     +36.3%     120217 ± 21%  softirqs.CPU26.TIMER
     70713           +32.5%      93695 ±  6%  softirqs.CPU27.RCU
     42528           +18.2%      50249 ±  2%  softirqs.CPU27.SCHED
     88110           +38.5%     122002 ± 24%  softirqs.CPU27.TIMER
     69654 ±  2%     +34.1%      93398 ±  3%  softirqs.CPU28.RCU
     42052           +18.0%      49636 ±  2%  softirqs.CPU28.SCHED
     86876 ±  2%     +21.6%     105634 ±  2%  softirqs.CPU28.TIMER
     70230 ±  2%     +37.8%      96776 ±  5%  softirqs.CPU29.RCU
     42440           +17.0%      49673 ±  2%  softirqs.CPU29.SCHED
     87058           +18.9%     103502 ±  2%  softirqs.CPU29.TIMER
     70519 ±  3%     +34.2%      94624 ±  2%  softirqs.CPU3.RCU
     43299           +16.2%      50322 ±  2%  softirqs.CPU3.SCHED
     87572           +19.2%     104386 ±  4%  softirqs.CPU3.TIMER
     70538 ±  2%     +31.0%      92391 ±  5%  softirqs.CPU30.RCU
     42038 ±  2%     +17.9%      49574        softirqs.CPU30.SCHED
     86446 ±  3%     +20.6%     104284        softirqs.CPU30.TIMER
     71140 ±  2%     +33.6%      95050 ±  3%  softirqs.CPU31.RCU
     42599           +16.5%      49608 ±  2%  softirqs.CPU31.SCHED
     86417           +19.6%     103393 ±  3%  softirqs.CPU31.TIMER
     59293 ±  2%     +29.7%      76894 ±  4%  softirqs.CPU32.RCU
     42200           +17.8%      49711        softirqs.CPU32.SCHED
     86726           +18.8%     103045 ±  3%  softirqs.CPU32.TIMER
     58507           +31.6%      76967        softirqs.CPU33.RCU
     42587           +16.7%      49683        softirqs.CPU33.SCHED
     86567           +19.1%     103101 ±  3%  softirqs.CPU33.TIMER
     59394 ±  2%     +27.1%      75493 ±  5%  softirqs.CPU34.RCU
     42682           +16.6%      49753        softirqs.CPU34.SCHED
     59949 ±  3%     +29.2%      77459        softirqs.CPU35.RCU
     42681 ±  2%     +17.3%      50047 ±  2%  softirqs.CPU35.SCHED
     85785           +19.9%     102825 ±  2%  softirqs.CPU35.TIMER
     61089           +25.4%      76612        softirqs.CPU36.RCU
     42662           +16.9%      49870        softirqs.CPU36.SCHED
     87784 ±  2%     +17.3%     102973 ±  3%  softirqs.CPU36.TIMER
     60018           +29.3%      77596        softirqs.CPU37.RCU
     42378 ±  2%     +18.0%      50012        softirqs.CPU37.SCHED
     85919           +20.3%     103386 ±  2%  softirqs.CPU37.TIMER
     59781 ±  2%     +28.6%      76877        softirqs.CPU38.RCU
     42779           +17.1%      50105        softirqs.CPU38.SCHED
     85246 ±  4%     +20.7%     102859 ±  2%  softirqs.CPU38.TIMER
     59657           +26.9%      75699 ±  2%  softirqs.CPU39.RCU
     43011 ±  2%     +17.6%      50580 ±  3%  softirqs.CPU39.SCHED
     98669 ± 20%     +36.3%     134491 ± 24%  softirqs.CPU39.TIMER
     68912 ±  2%     +32.7%      91479 ±  2%  softirqs.CPU4.RCU
     43101           +15.9%      49946        softirqs.CPU4.SCHED
     86425 ±  5%     +18.4%     102344 ±  3%  softirqs.CPU4.TIMER
     70893 ±  3%     +32.1%      93618 ±  3%  softirqs.CPU5.RCU
     42977           +17.2%      50381        softirqs.CPU5.SCHED
     87301           +17.7%     102788 ±  3%  softirqs.CPU5.TIMER
     71151           +25.0%      88965 ±  3%  softirqs.CPU6.RCU
     42864           +17.3%      50282        softirqs.CPU6.SCHED
     87741 ±  2%     +22.5%     107458 ±  4%  softirqs.CPU6.TIMER
     71242 ±  4%     +30.5%      92999 ±  3%  softirqs.CPU7.RCU
     43014 ±  2%     +17.7%      50619        softirqs.CPU7.SCHED
     87907           +15.8%     101781 ±  5%  softirqs.CPU7.TIMER
     69638 ±  2%     +32.6%      92309 ±  4%  softirqs.CPU8.RCU
     42479           +17.6%      49947        softirqs.CPU8.SCHED
     86964 ±  2%     +18.6%     103128 ±  3%  softirqs.CPU8.TIMER
     69585 ±  2%     +36.1%      94684 ±  2%  softirqs.CPU9.RCU
     42667           +18.0%      50343        softirqs.CPU9.SCHED
     86913           +19.2%     103596 ±  4%  softirqs.CPU9.TIMER
   2739272           +30.7%    3580875 ±  2%  softirqs.RCU
   1709285           +17.3%    2005342        softirqs.SCHED
   3539953           +19.4%    4226728 ±  2%  softirqs.TIMER





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.0.0-rc8-00103-g44fe89d" of type "text/plain" (191800 bytes)

View attachment "job-script" of type "text/plain" (7518 bytes)

View attachment "job.yaml" of type "text/plain" (5105 bytes)

View attachment "reproduce" of type "text/plain" (1007 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ