lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Fri, 12 Jun 2020 14:11:44 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Josef Bacik <josef@...icpanda.com>
Cc:     David Sterba <dsterba@...e.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [btrfs] c75e839414: aim7.jobs-per-min -9.1% regression

Greeting,

FYI, we noticed a -9.1% regression of aim7.jobs-per-min due to commit:


commit: c75e839414d3610e6487ae3145199c500d55f7f7 ("btrfs: kill the subvol_srcu")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: aim7
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: btrfs
	test: disk_wrt
	load: 1500
	cpufreq_governor: performance
	ucode: 0x500002c

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/1500/RAID0/debian-x86_64-20191114.cgz/lkp-csl-2ap2/disk_wrt/aim7/0x500002c

commit: 
  efc3453494 ("btrfs: make btrfs_cleanup_fs_roots use the radix tree lock")
  c75e839414 ("btrfs: kill the subvol_srcu")

efc3453494af7818 c75e839414d3610e6487ae31451 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          3:9          -33%            :8     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
         %stddev     %change         %stddev
             \          |                \  
     29509 ±  2%      -9.1%      26837 ±  2%  aim7.jobs-per-min
    305.28 ±  2%     +10.0%     335.72 ±  2%  aim7.time.elapsed_time
    305.28 ±  2%     +10.0%     335.72 ±  2%  aim7.time.elapsed_time.max
   4883135 ± 10%     +37.9%    6735464 ±  7%  aim7.time.involuntary_context_switches
     56288 ±  2%     +10.5%      62202 ±  2%  aim7.time.system_time
   2344783            +6.5%    2497364 ±  2%  aim7.time.voluntary_context_switches
  62337721 ±  2%      +9.8%   68456490 ±  2%  turbostat.IRQ
    431.56 ±  6%     +22.3%     527.88 ±  4%  vmstat.procs.r
     27340 ±  2%     +11.2%      30397 ±  2%  vmstat.system.cs
    226804 ±  6%     +21.7%     276057 ±  4%  meminfo.Active(file)
    221309 ±  6%     +22.3%     270668 ±  4%  meminfo.Dirty
    720.89 ±111%     +49.3%       1076 ± 73%  meminfo.Mlocked
     14278 ±  2%      -8.3%      13094 ±  2%  meminfo.max_used_kB
     57228 ±  6%     +22.7%      70195 ±  5%  numa-meminfo.node0.Active(file)
     55433 ±  6%     +21.6%      67431 ±  4%  numa-meminfo.node0.Dirty
     56152 ±  6%     +21.4%      68180 ±  5%  numa-meminfo.node1.Active(file)
     55001 ±  6%     +22.5%      67397 ±  4%  numa-meminfo.node1.Dirty
     56373 ±  6%     +21.7%      68594 ±  4%  numa-meminfo.node2.Active(file)
     55222 ±  7%     +22.6%      67726 ±  4%  numa-meminfo.node2.Dirty
     56671 ±  6%     +20.5%      68317 ±  3%  numa-meminfo.node3.Active(file)
     55285 ±  6%     +21.8%      67355 ±  4%  numa-meminfo.node3.Dirty
     56694 ±  6%     +21.7%      69019 ±  4%  proc-vmstat.nr_active_file
     55342 ±  6%     +22.3%      67662 ±  4%  proc-vmstat.nr_dirty
    402316            +2.1%     410951        proc-vmstat.nr_file_pages
    180.22 ±111%     +49.4%     269.25 ± 73%  proc-vmstat.nr_mlock
     56694 ±  6%     +21.7%      69019 ±  4%  proc-vmstat.nr_zone_active_file
     54680 ±  6%     +22.8%      67168 ±  4%  proc-vmstat.nr_zone_write_pending
   3144381 ±  2%      +6.1%    3335275        proc-vmstat.pgactivate
   1387558 ±  2%      +7.9%    1496754 ±  2%  proc-vmstat.pgfault
    983.33 ±  4%      +5.4%       1036        proc-vmstat.unevictable_pgs_culled
     14331 ±  6%     +22.6%      17566 ±  5%  numa-vmstat.node0.nr_active_file
     13884 ±  6%     +21.6%      16884 ±  4%  numa-vmstat.node0.nr_dirty
     14330 ±  6%     +22.6%      17566 ±  5%  numa-vmstat.node0.nr_zone_active_file
     13714 ±  6%     +22.2%      16755 ±  4%  numa-vmstat.node0.nr_zone_write_pending
     14047 ±  6%     +21.3%      17043 ±  4%  numa-vmstat.node1.nr_active_file
     13763 ±  6%     +22.3%      16838 ±  4%  numa-vmstat.node1.nr_dirty
     14047 ±  6%     +21.3%      17043 ±  4%  numa-vmstat.node1.nr_zone_active_file
     13599 ±  6%     +23.0%      16726 ±  4%  numa-vmstat.node1.nr_zone_write_pending
     14074 ±  5%     +21.7%      17130 ±  4%  numa-vmstat.node2.nr_active_file
     13786 ±  6%     +22.6%      16906 ±  4%  numa-vmstat.node2.nr_dirty
     14074 ±  5%     +21.7%      17130 ±  4%  numa-vmstat.node2.nr_zone_active_file
     13617 ±  7%     +23.3%      16789 ±  4%  numa-vmstat.node2.nr_zone_write_pending
     14151 ±  6%     +20.7%      17074 ±  3%  numa-vmstat.node3.nr_active_file
     13790 ±  6%     +22.1%      16832 ±  4%  numa-vmstat.node3.nr_dirty
     14152 ±  6%     +20.7%      17074 ±  3%  numa-vmstat.node3.nr_zone_active_file
     13627 ±  6%     +22.5%      16696 ±  4%  numa-vmstat.node3.nr_zone_write_pending
    828631 ± 16%     +49.9%    1241934 ±  8%  sched_debug.cfs_rq:/.min_vruntime.stddev
      1.33 ± 30%     +96.9%       2.61 ± 14%  sched_debug.cfs_rq:/.nr_spread_over.avg
    831152 ± 16%     +49.7%    1244586 ±  8%  sched_debug.cfs_rq:/.spread0.stddev
      1.92 ± 14%     +28.4%       2.47 ±  8%  sched_debug.cpu.nr_running.avg
      5.00 ±  9%     +29.7%       6.48 ±  9%  sched_debug.cpu.nr_running.max
      0.90 ±  9%     +16.0%       1.05 ±  3%  sched_debug.cpu.nr_running.stddev
     21790 ± 11%     +16.7%      25440 ±  2%  sched_debug.cpu.nr_switches.avg
     19761 ± 11%     +17.9%      23289 ±  2%  sched_debug.cpu.nr_switches.min
     20403 ± 12%     +18.1%      24101 ±  2%  sched_debug.cpu.sched_count.avg
     27825 ± 10%     +15.8%      32227 ±  4%  sched_debug.cpu.sched_count.max
     18995 ± 13%     +18.3%      22463 ±  2%  sched_debug.cpu.sched_count.min
      1143 ± 11%     +15.6%       1322 ±  4%  sched_debug.cpu.sched_count.stddev
      1404 ±  7%     -31.0%     969.66 ± 11%  sched_debug.cpu.sched_goidle.avg
      1186 ±  8%     -36.0%     759.25 ± 14%  sched_debug.cpu.sched_goidle.min
     16821 ± 14%     +20.6%      20284 ±  6%  sched_debug.cpu.ttwu_count.max
      2790 ± 11%     +21.7%       3395 ±  7%  sched_debug.cpu.ttwu_count.stddev
    398.19 ± 16%     +28.3%     510.73 ± 10%  sched_debug.cpu.ttwu_local.stddev
  43238516 ±  5%     +19.0%   51459028 ±  4%  perf-stat.i.cache-misses
  83870573 ±  4%     +13.8%   95436555 ±  4%  perf-stat.i.cache-references
     27181 ±  2%     +11.1%      30198 ±  2%  perf-stat.i.context-switches
      3008 ±  2%      -7.1%       2794        perf-stat.i.cpu-migrations
     12612 ±  5%     -15.6%      10641 ±  3%  perf-stat.i.cycles-between-cache-misses
   1.1e+09 ±  2%      -7.3%  1.019e+09 ±  2%  perf-stat.i.dTLB-stores
  10684629 ±  2%     +10.8%   11842781 ±  2%  perf-stat.i.node-load-misses
    641429 ±  2%     +24.4%     798190 ±  2%  perf-stat.i.node-loads
   5915955 ±  2%      -6.3%    5543368 ±  2%  perf-stat.i.node-store-misses
    380971            -6.7%     355629 ±  2%  perf-stat.i.node-stores
      1.66 ±  4%     +14.4%       1.90 ±  3%  perf-stat.overall.MPKI
     13064 ±  5%     -15.8%      10999 ±  4%  perf-stat.overall.cycles-between-cache-misses
  43484558 ±  5%     +19.2%   51837476 ±  4%  perf-stat.ps.cache-misses
  83278267 ±  4%     +14.4%   95253129 ±  3%  perf-stat.ps.cache-references
     27414 ±  2%     +11.2%      30486 ±  2%  perf-stat.ps.context-switches
      3028 ±  2%      -7.0%       2817        perf-stat.ps.cpu-migrations
 1.104e+09 ±  2%      -7.3%  1.023e+09 ±  2%  perf-stat.ps.dTLB-stores
      4394            -1.8%       4315        perf-stat.ps.minor-faults
  10766873 ±  2%     +11.0%   11947268 ±  2%  perf-stat.ps.node-load-misses
    645940 ±  2%     +24.7%     805392 ±  2%  perf-stat.ps.node-loads
   5965618 ±  2%      -6.2%    5595562 ±  2%  perf-stat.ps.node-store-misses
    383092            -6.5%     358124 ±  2%  perf-stat.ps.node-stores
      4395            -1.8%       4315        perf-stat.ps.page-faults
 1.538e+13 ±  2%     +10.0%  1.692e+13 ±  2%  perf-stat.total.instructions
     27.39            -0.1       27.28        perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
     35.13            +0.1       35.23        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
     35.23            +0.1       35.34        perf-profile.calltrace.cycles-pp._raw_spin_lock.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter
     35.48            +0.1       35.60        perf-profile.calltrace.cycles-pp.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
     35.08            +0.1       35.22        perf-profile.calltrace.cycles-pp._raw_spin_lock.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter
     34.97            +0.1       35.12        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write
     35.57            +0.1       35.72        perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
     35.10            +0.2       35.26        perf-profile.calltrace.cycles-pp.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
     35.11            +0.2       35.27        perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
     98.71            +0.2       98.88        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     98.69            +0.2       98.87        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     98.74            +0.2       98.91        perf-profile.calltrace.cycles-pp.write
     98.71            +0.2       98.89        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     98.68            +0.2       98.86        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     98.64            +0.2       98.82        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     98.63            +0.2       98.81        perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     98.58            +0.2       98.77        perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
     27.39            -0.1       27.28        perf-profile.children.cycles-pp.btrfs_dirty_pages
      0.56 ±  5%      -0.1        0.46 ±  9%  perf-profile.children.cycles-pp.btrfs_get_extent
      0.51 ±  5%      -0.1        0.41 ±  9%  perf-profile.children.cycles-pp.btrfs_lookup_file_extent
      0.52 ±  5%      -0.1        0.43 ±  9%  perf-profile.children.cycles-pp.btrfs_search_slot
      0.44 ±  6%      -0.1        0.35 ± 11%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
      0.43 ±  6%      -0.1        0.34 ± 11%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
      0.19 ± 15%      -0.1        0.11 ± 27%  perf-profile.children.cycles-pp.start_secondary
      0.19 ± 16%      -0.1        0.10 ± 27%  perf-profile.children.cycles-pp.cpuidle_enter
      0.19 ± 16%      -0.1        0.10 ± 27%  perf-profile.children.cycles-pp.cpuidle_enter_state
      0.41 ±  7%      -0.1        0.33 ±  8%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.46 ±  7%      -0.1        0.38 ±  8%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      0.19 ± 15%      -0.1        0.11 ± 25%  perf-profile.children.cycles-pp.secondary_startup_64
      0.19 ± 15%      -0.1        0.11 ± 25%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.19 ± 15%      -0.1        0.11 ± 25%  perf-profile.children.cycles-pp.do_idle
      0.18 ± 15%      -0.1        0.10 ± 24%  perf-profile.children.cycles-pp.intel_idle
      0.29 ±  9%      -0.1        0.22 ± 11%  perf-profile.children.cycles-pp.osq_lock
      0.19 ±  6%      -0.0        0.15 ± 16%  perf-profile.children.cycles-pp.finish_wait
      0.39 ±  5%      -0.0        0.35 ±  6%  perf-profile.children.cycles-pp.do_unlinkat
      0.39 ±  6%      -0.0        0.35 ±  7%  perf-profile.children.cycles-pp.unlink
      0.26 ±  5%      -0.0        0.22 ±  6%  perf-profile.children.cycles-pp.do_sys_open
      0.26 ±  5%      -0.0        0.22 ±  6%  perf-profile.children.cycles-pp.do_sys_openat2
      0.26 ±  4%      -0.0        0.22 ±  6%  perf-profile.children.cycles-pp.do_filp_open
      0.26 ±  4%      -0.0        0.22 ±  6%  perf-profile.children.cycles-pp.path_openat
      0.26 ±  4%      -0.0        0.22 ±  6%  perf-profile.children.cycles-pp.creat
      0.00            +0.0        0.05        perf-profile.children.cycles-pp.btrfs_calculate_inode_block_rsv_size
     99.65            +0.1       99.73        perf-profile.children.cycles-pp.do_syscall_64
     99.66            +0.1       99.74        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     35.62            +0.1       35.74        perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes
     35.57            +0.1       35.72        perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
     97.55            +0.2       97.72        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     98.74            +0.2       98.92        perf-profile.children.cycles-pp.write
     98.70            +0.2       98.88        perf-profile.children.cycles-pp.ksys_write
     98.69            +0.2       98.87        perf-profile.children.cycles-pp.vfs_write
     98.65            +0.2       98.83        perf-profile.children.cycles-pp.new_sync_write
     98.63            +0.2       98.81        perf-profile.children.cycles-pp.btrfs_file_write_iter
     98.59            +0.2       98.78        perf-profile.children.cycles-pp.btrfs_buffered_write
     97.60            +0.2       97.79        perf-profile.children.cycles-pp._raw_spin_lock
      0.18 ± 15%      -0.1        0.10 ± 24%  perf-profile.self.cycles-pp.intel_idle
      0.28 ±  9%      -0.1        0.21 ± 12%  perf-profile.self.cycles-pp.osq_lock
      0.45 ±  2%      -0.0        0.41 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
    135788 ±  2%     +13.1%     153639 ±  3%  softirqs.CPU0.RCU
    115547 ±  3%      +8.0%     124768 ±  4%  softirqs.CPU0.TIMER
    135243 ±  3%     +14.0%     154188 ±  2%  softirqs.CPU1.RCU
    136056 ±  3%     +12.4%     152960 ±  3%  softirqs.CPU10.RCU
    134253 ±  3%     +14.2%     153260 ±  3%  softirqs.CPU100.RCU
    134668 ±  4%     +13.7%     153157 ±  2%  softirqs.CPU101.RCU
    109995 ±  3%      +7.9%     118637 ±  3%  softirqs.CPU101.TIMER
    134989 ±  4%     +13.9%     153700 ±  2%  softirqs.CPU102.RCU
    109876 ±  3%      +8.9%     119608 ±  2%  softirqs.CPU102.TIMER
    134868 ±  4%     +13.8%     153427 ±  3%  softirqs.CPU103.RCU
    109933 ±  3%      +8.2%     118946 ±  3%  softirqs.CPU103.TIMER
    134454 ±  3%     +14.0%     153285 ±  3%  softirqs.CPU104.RCU
    109731 ±  2%      +8.4%     118965 ±  3%  softirqs.CPU104.TIMER
    134878 ±  3%     +13.6%     153283 ±  3%  softirqs.CPU105.RCU
    134096 ±  3%     +14.1%     153022 ±  3%  softirqs.CPU106.RCU
    109821 ±  2%      +8.2%     118845 ±  3%  softirqs.CPU106.TIMER
    133905 ±  3%     +15.0%     154004 ±  3%  softirqs.CPU107.RCU
    134117 ±  3%     +14.3%     153232 ±  3%  softirqs.CPU108.RCU
    109545 ±  2%      +8.8%     119182 ±  3%  softirqs.CPU108.TIMER
    134574 ±  3%     +13.8%     153202 ±  3%  softirqs.CPU109.RCU
    109804 ±  3%      +8.2%     118760 ±  3%  softirqs.CPU109.TIMER
    135106 ±  3%     +14.8%     155124 ±  3%  softirqs.CPU11.RCU
    111746 ±  4%      +7.8%     120460 ±  3%  softirqs.CPU11.TIMER
    134482 ±  4%     +14.3%     153687 ±  3%  softirqs.CPU110.RCU
    110314 ±  3%      +8.1%     119241 ±  3%  softirqs.CPU110.TIMER
    134464 ±  3%     +14.3%     153634 ±  3%  softirqs.CPU111.RCU
    110626 ±  3%      +8.3%     119815 ±  3%  softirqs.CPU111.TIMER
    134895 ±  3%     +14.4%     154257 ±  3%  softirqs.CPU112.RCU
    110040 ±  3%     +14.6%     126058 ± 16%  softirqs.CPU112.TIMER
    134644 ±  3%     +13.6%     152993 ±  3%  softirqs.CPU113.RCU
    109693 ±  3%      +8.1%     118569 ±  3%  softirqs.CPU113.TIMER
    134313 ±  3%     +13.9%     152993 ±  3%  softirqs.CPU114.RCU
    109711 ±  3%      +8.6%     119143 ±  3%  softirqs.CPU114.TIMER
    134559 ±  3%     +13.8%     153115 ±  3%  softirqs.CPU115.RCU
    110226 ±  3%      +8.2%     119221 ±  3%  softirqs.CPU115.TIMER
    134087 ±  3%     +14.2%     153077 ±  3%  softirqs.CPU116.RCU
    110225 ±  3%      +9.0%     120184 ±  3%  softirqs.CPU116.TIMER
    134271 ±  3%     +14.1%     153220 ±  3%  softirqs.CPU117.RCU
    109179 ±  2%      +8.3%     118274 ±  3%  softirqs.CPU117.TIMER
    134281 ±  3%     +14.1%     153230 ±  3%  softirqs.CPU118.RCU
    134415 ±  3%     +14.0%     153215 ±  3%  softirqs.CPU119.RCU
    109514 ±  2%      +8.3%     118643 ±  3%  softirqs.CPU119.TIMER
    135560 ±  3%     +13.5%     153874 ±  3%  softirqs.CPU12.RCU
    110316 ±  2%      +8.5%     119734 ±  3%  softirqs.CPU12.TIMER
    134177 ±  3%     +14.6%     153815 ±  3%  softirqs.CPU120.RCU
    134367 ±  3%     +15.0%     154493 ±  3%  softirqs.CPU121.RCU
    133972 ±  3%     +14.3%     153173 ±  3%  softirqs.CPU122.RCU
    134801 ±  4%     +13.6%     153136 ±  3%  softirqs.CPU123.RCU
    134709 ±  3%     +13.9%     153440 ±  3%  softirqs.CPU124.RCU
    135185 ±  3%     +13.1%     152925 ±  3%  softirqs.CPU125.RCU
    134977 ±  3%     +13.8%     153608 ±  3%  softirqs.CPU126.RCU
    133862 ±  4%     +14.7%     153558 ±  3%  softirqs.CPU127.RCU
    134554 ±  3%     +13.9%     153237 ±  3%  softirqs.CPU128.RCU
    134254 ±  3%     +14.1%     153130 ±  3%  softirqs.CPU129.RCU
    135133 ±  3%     +14.0%     154103 ±  3%  softirqs.CPU13.RCU
    110389 ±  2%      +8.1%     119382 ±  3%  softirqs.CPU13.TIMER
    134319 ±  3%     +13.8%     152909 ±  3%  softirqs.CPU130.RCU
    133590 ±  3%     +14.5%     152925 ±  3%  softirqs.CPU131.RCU
    134416 ±  4%     +14.0%     153223 ±  3%  softirqs.CPU132.RCU
    133931 ±  3%     +14.5%     153359 ±  3%  softirqs.CPU133.RCU
    134031 ±  3%     +14.5%     153527 ±  3%  softirqs.CPU134.RCU
    133851 ±  4%     +14.3%     153052 ±  3%  softirqs.CPU135.RCU
    134235 ±  4%     +13.9%     152959 ±  3%  softirqs.CPU136.RCU
    134003 ±  3%     +14.2%     153037 ±  3%  softirqs.CPU137.RCU
    133453 ±  3%     +14.4%     152625 ±  3%  softirqs.CPU138.RCU
    133514 ±  3%     +14.8%     153289 ±  3%  softirqs.CPU139.RCU
    134952 ±  3%     +13.6%     153372 ±  3%  softirqs.CPU14.RCU
    134045 ±  3%     +13.7%     152361 ±  3%  softirqs.CPU140.RCU
    133662 ±  3%     +14.1%     152528 ±  3%  softirqs.CPU141.RCU
    133643 ±  3%     +14.7%     153283 ±  2%  softirqs.CPU142.RCU
    133313 ±  3%     +15.0%     153254 ±  3%  softirqs.CPU143.RCU
    133585 ±  3%     +14.4%     152837 ±  3%  softirqs.CPU144.RCU
    108590 ±  3%     +12.7%     122339 ±  5%  softirqs.CPU144.TIMER
    133471 ±  3%     +14.5%     152883 ±  3%  softirqs.CPU145.RCU
    108095 ±  3%     +14.5%     123726 ±  6%  softirqs.CPU145.TIMER
    133588 ±  3%     +15.4%     154173 ±  3%  softirqs.CPU146.RCU
    108268 ±  3%     +19.8%     129684 ± 16%  softirqs.CPU146.TIMER
    134060 ±  3%     +14.3%     153270 ±  3%  softirqs.CPU147.RCU
    108731 ±  3%     +13.0%     122902 ±  5%  softirqs.CPU147.TIMER
    133778 ±  3%     +14.1%     152681 ±  3%  softirqs.CPU148.RCU
    108208 ±  3%     +12.6%     121875 ±  5%  softirqs.CPU148.TIMER
    133514 ±  3%     +14.2%     152460 ±  3%  softirqs.CPU149.RCU
    108054 ±  3%     +15.6%     124938 ±  9%  softirqs.CPU149.TIMER
    134677 ±  3%     +14.5%     154138 ±  2%  softirqs.CPU15.RCU
    133245 ±  3%     +14.4%     152373 ±  3%  softirqs.CPU150.RCU
    107788 ±  3%     +13.1%     121957 ±  5%  softirqs.CPU150.TIMER
    133747 ±  3%     +14.0%     152473 ±  3%  softirqs.CPU151.RCU
    108258 ±  4%     +12.4%     121662 ±  5%  softirqs.CPU151.TIMER
    133819 ±  3%     +14.1%     152726 ±  3%  softirqs.CPU152.RCU
    108163 ±  3%     +12.7%     121940 ±  5%  softirqs.CPU152.TIMER
    134057 ±  3%     +14.0%     152844 ±  3%  softirqs.CPU153.RCU
    107680 ±  3%     +13.1%     121787 ±  5%  softirqs.CPU153.TIMER
    133660 ±  3%     +14.2%     152675 ±  3%  softirqs.CPU154.RCU
    107911 ±  3%     +15.4%     124485 ±  9%  softirqs.CPU154.TIMER
    133349 ±  3%     +14.6%     152811 ±  3%  softirqs.CPU155.RCU
    107513 ±  3%     +15.1%     123737 ±  6%  softirqs.CPU155.TIMER
    133901 ±  3%     +14.0%     152615 ±  3%  softirqs.CPU156.RCU
    107853 ±  3%     +13.0%     121904 ±  5%  softirqs.CPU156.TIMER
    133621 ±  3%     +14.2%     152615 ±  3%  softirqs.CPU157.RCU
    108040 ±  3%     +12.6%     121684 ±  5%  softirqs.CPU157.TIMER
    133891 ±  3%     +14.2%     152862 ±  3%  softirqs.CPU158.RCU
    108534 ±  3%     +12.7%     122300 ±  5%  softirqs.CPU158.TIMER
    133925 ±  3%     +14.5%     153347 ±  3%  softirqs.CPU159.RCU
    108099 ±  3%     +12.9%     122015 ±  5%  softirqs.CPU159.TIMER
    134893 ±  3%     +14.5%     154389 ±  3%  softirqs.CPU16.RCU
    134169 ±  3%     +13.9%     152812 ±  3%  softirqs.CPU160.RCU
    108300 ±  3%     +12.9%     122318 ±  5%  softirqs.CPU160.TIMER
    133818 ±  3%     +15.1%     154048 ±  4%  softirqs.CPU161.RCU
    107919 ±  3%     +13.6%     122606 ±  6%  softirqs.CPU161.TIMER
    133867 ±  3%     +14.2%     152856 ±  3%  softirqs.CPU162.RCU
    107991 ±  3%     +13.2%     122209 ±  5%  softirqs.CPU162.TIMER
    133579 ±  3%     +14.3%     152654 ±  3%  softirqs.CPU163.RCU
    108317 ±  3%     +12.9%     122271 ±  5%  softirqs.CPU163.TIMER
    133914 ±  3%     +14.4%     153158 ±  3%  softirqs.CPU164.RCU
    133684 ±  3%     +14.4%     152895 ±  3%  softirqs.CPU165.RCU
    107435 ±  3%     +12.5%     120877 ±  5%  softirqs.CPU165.TIMER
    133983 ±  3%     +13.9%     152582 ±  3%  softirqs.CPU166.RCU
    107432 ±  3%     +13.2%     121641 ±  5%  softirqs.CPU166.TIMER
    133947 ±  3%     +13.6%     152137 ±  3%  softirqs.CPU167.RCU
    134005 ±  3%     +14.5%     153385 ±  4%  softirqs.CPU168.RCU
    115752 ±  4%     +12.0%     129634 ±  4%  softirqs.CPU168.TIMER
    133557 ±  3%     +15.2%     153865 ±  4%  softirqs.CPU169.RCU
    115884 ±  3%     +11.7%     129481 ±  4%  softirqs.CPU169.TIMER
    135348 ±  3%     +13.8%     153977 ±  3%  softirqs.CPU17.RCU
    110589 ±  3%      +7.9%     119298 ±  3%  softirqs.CPU17.TIMER
    134463 ±  3%     +14.0%     153229 ±  3%  softirqs.CPU170.RCU
    133740 ±  3%     +14.1%     152630 ±  3%  softirqs.CPU171.RCU
    116524 ±  4%     +10.9%     129237 ±  3%  softirqs.CPU171.TIMER
    133716 ±  3%     +13.8%     152196 ±  3%  softirqs.CPU172.RCU
    116393 ±  4%     +10.9%     129087 ±  4%  softirqs.CPU172.TIMER
    133564 ±  3%     +13.9%     152165 ±  3%  softirqs.CPU173.RCU
    116237 ±  3%     +11.4%     129460 ±  3%  softirqs.CPU173.TIMER
    133411 ±  3%     +14.5%     152764 ±  3%  softirqs.CPU174.RCU
    116018 ±  3%     +11.2%     129006 ±  4%  softirqs.CPU174.TIMER
    133147 ±  3%     +15.0%     153104 ±  2%  softirqs.CPU175.RCU
    116052 ±  4%     +11.7%     129664 ±  3%  softirqs.CPU175.TIMER
    133989 ±  3%     +13.8%     152504 ±  3%  softirqs.CPU176.RCU
    116071 ±  4%     +10.9%     128756 ±  3%  softirqs.CPU176.TIMER
    133902 ±  3%     +14.1%     152811 ±  3%  softirqs.CPU177.RCU
    115994 ±  3%     +11.1%     128849 ±  4%  softirqs.CPU177.TIMER
    133679 ±  3%     +14.4%     152887 ±  3%  softirqs.CPU178.RCU
    115942 ±  4%     +13.4%     131439 ±  5%  softirqs.CPU178.TIMER
    133985 ±  3%     +14.0%     152762 ±  3%  softirqs.CPU179.RCU
    116347 ±  4%     +11.2%     129422 ±  4%  softirqs.CPU179.TIMER
    134982 ±  3%     +14.0%     153838 ±  3%  softirqs.CPU18.RCU
    110310 ±  3%      +8.6%     119851 ±  3%  softirqs.CPU18.TIMER
    133236 ±  3%     +14.7%     152870 ±  3%  softirqs.CPU180.RCU
    115901 ±  4%     +11.3%     129054 ±  4%  softirqs.CPU180.TIMER
    133615 ±  3%     +14.4%     152876 ±  3%  softirqs.CPU181.RCU
    115505 ±  4%     +11.3%     128560 ±  3%  softirqs.CPU181.TIMER
    133171 ±  3%     +14.7%     152810 ±  3%  softirqs.CPU182.RCU
    115955 ±  4%     +11.0%     128687 ±  3%  softirqs.CPU182.TIMER
    133413 ±  3%     +14.4%     152671 ±  3%  softirqs.CPU183.RCU
    116313 ±  4%     +11.0%     129140 ±  3%  softirqs.CPU183.TIMER
    133598 ±  3%     +14.8%     153388 ±  3%  softirqs.CPU184.RCU
    115462 ±  3%     +13.0%     130430 ±  6%  softirqs.CPU184.TIMER
    133884 ±  3%     +13.9%     152525 ±  3%  softirqs.CPU185.RCU
    115598 ±  4%     +11.1%     128397 ±  4%  softirqs.CPU185.TIMER
    133262 ±  3%     +14.5%     152646 ±  3%  softirqs.CPU186.RCU
    115745 ±  3%     +11.1%     128612 ±  4%  softirqs.CPU186.TIMER
    133430 ±  3%     +14.5%     152767 ±  3%  softirqs.CPU187.RCU
    115769 ±  4%     +11.0%     128464 ±  3%  softirqs.CPU187.TIMER
    133259 ±  3%     +14.8%     153036 ±  3%  softirqs.CPU188.RCU
    116108 ±  4%     +11.3%     129232 ±  3%  softirqs.CPU188.TIMER
    133609 ±  3%     +14.1%     152436 ±  3%  softirqs.CPU189.RCU
    115984 ±  4%     +11.0%     128710 ±  3%  softirqs.CPU189.TIMER
    135090 ±  3%     +13.9%     153802 ±  3%  softirqs.CPU19.RCU
    110865 ±  2%      +8.2%     119936 ±  3%  softirqs.CPU19.TIMER
    133685 ±  3%     +14.2%     152631 ±  3%  softirqs.CPU190.RCU
    115642 ±  3%     +11.2%     128581 ±  4%  softirqs.CPU190.TIMER
    131031 ±  3%     +14.6%     150126 ±  3%  softirqs.CPU191.RCU
    114275 ±  4%     +11.4%     127249 ±  4%  softirqs.CPU191.TIMER
    134928 ±  3%     +14.3%     154165 ±  3%  softirqs.CPU2.RCU
    135126 ±  3%     +14.0%     154083 ±  3%  softirqs.CPU20.RCU
    111013 ±  3%      +8.6%     120549 ±  3%  softirqs.CPU20.TIMER
    134663 ±  3%     +14.5%     154251 ±  3%  softirqs.CPU21.RCU
    109671 ±  2%      +8.2%     118708 ±  3%  softirqs.CPU21.TIMER
    135047 ±  3%     +13.9%     153766 ±  3%  softirqs.CPU22.RCU
    134437 ±  3%     +14.2%     153492 ±  3%  softirqs.CPU23.RCU
    109675 ±  2%      +8.3%     118824 ±  3%  softirqs.CPU23.TIMER
    136114 ±  3%     +13.3%     154277 ±  3%  softirqs.CPU24.RCU
    134825 ±  3%     +14.2%     154007 ±  3%  softirqs.CPU25.RCU
    139234 ±  4%     +13.8%     158515 ±  3%  softirqs.CPU26.RCU
    135056 ±  3%     +14.1%     154062 ±  3%  softirqs.CPU27.RCU
    135307 ±  3%     +13.7%     153897 ±  3%  softirqs.CPU28.RCU
    134686 ±  3%     +14.7%     154497 ±  3%  softirqs.CPU29.RCU
    134688 ±  3%     +14.4%     154115 ±  2%  softirqs.CPU3.RCU
    134642 ±  3%     +14.5%     154226 ±  3%  softirqs.CPU30.RCU
    134773 ±  3%     +14.2%     153887 ±  3%  softirqs.CPU31.RCU
    134523 ±  3%     +14.3%     153733 ±  3%  softirqs.CPU32.RCU
    134488 ±  3%     +14.4%     153801 ±  3%  softirqs.CPU33.RCU
    134483 ±  3%     +14.1%     153380 ±  3%  softirqs.CPU34.RCU
    134886 ±  3%     +13.6%     153180 ±  3%  softirqs.CPU35.RCU
    135021 ±  3%     +13.6%     153340 ±  3%  softirqs.CPU36.RCU
    134064 ±  3%     +15.1%     154277 ±  3%  softirqs.CPU37.RCU
    134166 ±  3%     +14.8%     153967 ±  3%  softirqs.CPU38.RCU
    134488 ±  3%     +14.4%     153871 ±  3%  softirqs.CPU39.RCU
    135294 ±  2%     +14.6%     155025 ±  3%  softirqs.CPU4.RCU
    134598 ±  4%     +14.3%     153848 ±  3%  softirqs.CPU40.RCU
    134255 ±  3%     +14.8%     154099 ±  3%  softirqs.CPU41.RCU
    134334 ±  3%     +14.3%     153491 ±  3%  softirqs.CPU42.RCU
    134201 ±  3%     +14.5%     153654 ±  3%  softirqs.CPU43.RCU
    134260 ±  3%     +14.2%     153359 ±  3%  softirqs.CPU44.RCU
    134005 ±  3%     +14.4%     153367 ±  3%  softirqs.CPU45.RCU
    134236 ±  3%     +14.3%     153419 ±  3%  softirqs.CPU46.RCU
    134552 ±  3%     +13.8%     153056 ±  3%  softirqs.CPU47.RCU
    134623 ±  3%     +14.4%     154028 ±  3%  softirqs.CPU48.RCU
    134348 ±  3%     +14.7%     154080 ±  3%  softirqs.CPU49.RCU
    109020 ±  3%     +18.6%     129279 ± 14%  softirqs.CPU49.TIMER
    134334 ±  3%     +15.0%     154455 ±  3%  softirqs.CPU5.RCU
    110260 ±  2%      +8.8%     119951 ±  4%  softirqs.CPU5.TIMER
    134377 ±  3%     +13.8%     152954 ±  3%  softirqs.CPU50.RCU
    109169 ±  3%     +15.0%     125548 ±  9%  softirqs.CPU50.TIMER
    134780 ±  3%     +14.2%     153967 ±  3%  softirqs.CPU51.RCU
    109951 ±  3%     +12.5%     123641 ±  5%  softirqs.CPU51.TIMER
    133929 ±  3%     +15.0%     154012 ±  3%  softirqs.CPU52.RCU
    108665 ±  3%     +12.8%     122613 ±  5%  softirqs.CPU52.TIMER
    133761 ±  3%     +14.9%     153675 ±  3%  softirqs.CPU53.RCU
    108606 ±  3%     +19.6%     129861 ± 16%  softirqs.CPU53.TIMER
    134369 ±  3%     +14.1%     153255 ±  3%  softirqs.CPU54.RCU
    108917 ±  3%     +12.5%     122585 ±  5%  softirqs.CPU54.TIMER
    133810 ±  3%     +14.6%     153282 ±  3%  softirqs.CPU55.RCU
    108875 ±  3%     +12.5%     122468 ±  5%  softirqs.CPU55.TIMER
    134038 ±  3%     +14.3%     153248 ±  3%  softirqs.CPU56.RCU
    108569 ±  3%     +12.8%     122502 ±  5%  softirqs.CPU56.TIMER
    133993 ±  4%     +14.5%     153426 ±  3%  softirqs.CPU57.RCU
    108465 ±  3%     +12.8%     122399 ±  5%  softirqs.CPU57.TIMER
    134049 ±  3%     +14.7%     153708 ±  3%  softirqs.CPU58.RCU
    108634 ±  4%     +18.8%     129107 ± 16%  softirqs.CPU58.TIMER
    134025 ±  3%     +14.9%     154015 ±  3%  softirqs.CPU59.RCU
    108501 ±  3%     +18.6%     128721 ± 13%  softirqs.CPU59.TIMER
    135247 ±  3%     +14.5%     154804 ±  3%  softirqs.CPU6.RCU
    133745 ±  4%     +14.7%     153452 ±  3%  softirqs.CPU60.RCU
    108526 ±  3%     +13.0%     122688 ±  5%  softirqs.CPU60.TIMER
    133931 ±  3%     +14.6%     153549 ±  3%  softirqs.CPU61.RCU
    108764 ±  3%     +12.9%     122787 ±  5%  softirqs.CPU61.TIMER
    134682 ±  3%     +13.8%     153266 ±  3%  softirqs.CPU62.RCU
    109188 ±  3%     +12.5%     122806 ±  5%  softirqs.CPU62.TIMER
    134190 ±  3%     +14.6%     153761 ±  2%  softirqs.CPU63.RCU
    108489 ±  3%     +12.8%     122426 ±  5%  softirqs.CPU63.TIMER
    134280 ±  3%     +14.3%     153538 ±  3%  softirqs.CPU64.RCU
    108493 ±  3%     +13.3%     122953 ±  5%  softirqs.CPU64.TIMER
    134074 ±  3%     +14.7%     153765 ±  3%  softirqs.CPU65.RCU
    108331 ±  3%     +13.0%     122440 ±  5%  softirqs.CPU65.TIMER
    134424 ±  4%     +14.3%     153692 ±  3%  softirqs.CPU66.RCU
    108874 ±  3%     +12.5%     122490 ±  5%  softirqs.CPU66.TIMER
    133801 ±  3%     +14.9%     153746 ±  3%  softirqs.CPU67.RCU
    108903 ±  3%     +13.2%     123240 ±  5%  softirqs.CPU67.TIMER
    134085 ±  3%     +14.4%     153371 ±  3%  softirqs.CPU68.RCU
    109310 ±  3%     +12.3%     122810 ±  5%  softirqs.CPU68.TIMER
    134091 ±  3%     +14.5%     153515 ±  3%  softirqs.CPU69.RCU
    108193 ±  3%     +12.5%     121716 ±  5%  softirqs.CPU69.TIMER
    134883 ±  3%     +13.6%     153181 ±  3%  softirqs.CPU7.RCU
    134392 ±  3%     +14.8%     154240 ±  3%  softirqs.CPU70.RCU
    108056 ±  3%     +13.4%     122578 ±  5%  softirqs.CPU70.TIMER
    133901 ±  3%     +14.2%     152873 ±  3%  softirqs.CPU71.RCU
    134273 ±  3%     +14.4%     153630 ±  3%  softirqs.CPU72.RCU
    116189 ±  3%     +11.5%     129495 ±  3%  softirqs.CPU72.TIMER
    134231 ±  3%     +14.5%     153682 ±  3%  softirqs.CPU73.RCU
    116348 ±  3%     +11.3%     129453 ±  3%  softirqs.CPU73.TIMER
    134235 ±  3%     +14.1%     153158 ±  3%  softirqs.CPU74.RCU
    133896 ±  3%     +14.6%     153477 ±  3%  softirqs.CPU75.RCU
    116933 ±  4%     +11.2%     129980 ±  3%  softirqs.CPU75.TIMER
    134014 ±  3%     +14.3%     153198 ±  3%  softirqs.CPU76.RCU
    116889 ±  3%     +11.0%     129768 ±  3%  softirqs.CPU76.TIMER
    134608 ±  3%     +13.7%     152994 ±  3%  softirqs.CPU77.RCU
    117342 ±  3%     +11.0%     130274 ±  3%  softirqs.CPU77.TIMER
    134551 ±  3%     +13.9%     153253 ±  3%  softirqs.CPU78.RCU
    117229 ±  4%     +10.5%     129549 ±  4%  softirqs.CPU78.TIMER
    134118 ±  3%     +14.3%     153325 ±  3%  softirqs.CPU79.RCU
    116982 ±  3%     +11.0%     129802 ±  3%  softirqs.CPU79.TIMER
    135303 ±  3%     +13.9%     154129 ±  3%  softirqs.CPU8.RCU
    134699 ±  3%     +13.6%     153047 ±  3%  softirqs.CPU80.RCU
    117058 ±  4%     +10.6%     129470 ±  3%  softirqs.CPU80.TIMER
    134088 ±  3%     +15.6%     155063 ±  3%  softirqs.CPU81.RCU
    116539 ±  4%     +12.3%     130886 ±  5%  softirqs.CPU81.TIMER
    133989 ±  3%     +14.6%     153526 ±  3%  softirqs.CPU82.RCU
    116614 ±  4%     +16.8%     136238 ± 12%  softirqs.CPU82.TIMER
    133966 ±  3%     +14.3%     153142 ±  3%  softirqs.CPU83.RCU
    117140 ±  3%     +11.2%     130205 ±  4%  softirqs.CPU83.TIMER
    133561 ±  3%     +15.0%     153661 ±  3%  softirqs.CPU84.RCU
    116648 ±  4%     +11.3%     129844 ±  4%  softirqs.CPU84.TIMER
    134217 ±  3%     +14.2%     153330 ±  3%  softirqs.CPU85.RCU
    116247 ±  4%     +11.3%     129336 ±  3%  softirqs.CPU85.TIMER
    133956 ±  3%     +14.5%     153338 ±  3%  softirqs.CPU86.RCU
    116654 ±  4%     +11.1%     129560 ±  3%  softirqs.CPU86.TIMER
    133834 ±  3%     +15.1%     154049 ±  2%  softirqs.CPU87.RCU
    116898 ±  4%     +12.0%     130912 ±  3%  softirqs.CPU87.TIMER
    133951 ±  3%     +15.1%     154211 ±  3%  softirqs.CPU88.RCU
    116139 ±  3%     +16.0%     134676 ± 13%  softirqs.CPU88.TIMER
    134288 ±  3%     +14.3%     153433 ±  2%  softirqs.CPU89.RCU
    116227 ±  3%     +11.0%     129064 ±  3%  softirqs.CPU89.TIMER
    135289 ±  3%     +13.6%     153744 ±  3%  softirqs.CPU9.RCU
    133598 ±  3%     +14.5%     153019 ±  3%  softirqs.CPU90.RCU
    116395 ±  3%     +10.9%     129084 ±  4%  softirqs.CPU90.TIMER
    133854 ±  4%     +14.9%     153799 ±  3%  softirqs.CPU91.RCU
    116562 ±  4%     +11.1%     129452 ±  3%  softirqs.CPU91.TIMER
    134107 ±  3%     +14.1%     152958 ±  3%  softirqs.CPU92.RCU
    117107 ±  4%     +10.6%     129559 ±  3%  softirqs.CPU92.TIMER
    133800 ±  3%     +14.7%     153465 ±  3%  softirqs.CPU93.RCU
    116415 ±  3%     +11.5%     129823 ±  4%  softirqs.CPU93.TIMER
    134092 ±  3%     +14.0%     152835 ±  3%  softirqs.CPU94.RCU
    116063 ±  3%     +11.4%     129252 ±  4%  softirqs.CPU94.TIMER
    134284 ±  3%     +14.1%     153230 ±  4%  softirqs.CPU95.RCU
    117908 ±  3%     +11.0%     130904 ±  4%  softirqs.CPU95.TIMER
    134264 ±  3%     +14.3%     153455 ±  3%  softirqs.CPU96.RCU
    109819 ±  3%      +8.6%     119258 ±  3%  softirqs.CPU96.TIMER
    135000 ±  3%     +13.7%     153529 ±  3%  softirqs.CPU97.RCU
    134163 ±  3%     +14.2%     153253 ±  3%  softirqs.CPU98.RCU
    109990 ±  3%      +8.1%     118896 ±  3%  softirqs.CPU98.TIMER
    134966 ±  3%     +13.5%     153245 ±  3%  softirqs.CPU99.RCU
  25776297 ±  3%     +14.3%   29452390 ±  3%  softirqs.RCU
  21445010 ±  3%      +9.5%   23488096 ±  3%  softirqs.TIMER
    614.00 ±  2%      +9.9%     674.50 ±  2%  interrupts.9:IO-APIC.9-fasteoi.acpi
    611854 ±  2%     +10.1%     673945 ±  2%  interrupts.CPU0.LOC:Local_timer_interrupts
    614.00 ±  2%      +9.9%     674.50 ±  2%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    611835 ±  2%     +10.1%     673883 ±  2%  interrupts.CPU1.LOC:Local_timer_interrupts
    611859 ±  2%     +10.2%     673975 ±  2%  interrupts.CPU10.LOC:Local_timer_interrupts
      7508 ±  5%     +15.5%       8675 ±  4%  interrupts.CPU10.RES:Rescheduling_interrupts
    611802 ±  2%     +10.2%     673904 ±  2%  interrupts.CPU100.LOC:Local_timer_interrupts
      7323 ±  5%     +16.1%       8503 ±  3%  interrupts.CPU100.RES:Rescheduling_interrupts
    611804 ±  2%     +10.2%     673917 ±  2%  interrupts.CPU101.LOC:Local_timer_interrupts
      7342 ±  3%     +13.8%       8352 ±  3%  interrupts.CPU101.RES:Rescheduling_interrupts
    611780 ±  2%     +10.2%     673877 ±  2%  interrupts.CPU102.LOC:Local_timer_interrupts
      7506 ±  5%     +12.2%       8424 ±  4%  interrupts.CPU102.RES:Rescheduling_interrupts
    611734 ±  2%     +10.2%     673851 ±  2%  interrupts.CPU103.LOC:Local_timer_interrupts
      7436 ±  3%     +15.8%       8607 ±  4%  interrupts.CPU103.RES:Rescheduling_interrupts
    611800 ±  2%     +10.2%     673908 ±  2%  interrupts.CPU104.LOC:Local_timer_interrupts
      7519 ±  5%     +13.1%       8502 ±  2%  interrupts.CPU104.RES:Rescheduling_interrupts
    611790 ±  2%     +10.1%     673864 ±  2%  interrupts.CPU105.LOC:Local_timer_interrupts
      7346 ±  5%     +16.4%       8553 ±  2%  interrupts.CPU105.RES:Rescheduling_interrupts
    611801 ±  2%     +10.1%     673859 ±  2%  interrupts.CPU106.LOC:Local_timer_interrupts
      7352 ±  5%     +16.5%       8567 ±  2%  interrupts.CPU106.RES:Rescheduling_interrupts
    611797 ±  2%     +10.1%     673890 ±  2%  interrupts.CPU107.LOC:Local_timer_interrupts
      7238 ±  6%     +15.3%       8344 ±  4%  interrupts.CPU107.RES:Rescheduling_interrupts
    611778 ±  2%     +10.2%     673882 ±  2%  interrupts.CPU108.LOC:Local_timer_interrupts
      7310 ±  5%     +16.3%       8503 ±  5%  interrupts.CPU108.RES:Rescheduling_interrupts
    611788 ±  2%     +10.1%     673881 ±  2%  interrupts.CPU109.LOC:Local_timer_interrupts
      7324 ±  6%     +13.4%       8307        interrupts.CPU109.RES:Rescheduling_interrupts
    611827 ±  2%     +10.2%     673960 ±  2%  interrupts.CPU11.LOC:Local_timer_interrupts
      7598 ±  6%     +12.6%       8557 ±  5%  interrupts.CPU11.RES:Rescheduling_interrupts
    611796 ±  2%     +10.1%     673865 ±  2%  interrupts.CPU110.LOC:Local_timer_interrupts
      7326 ±  4%     +16.2%       8513 ±  4%  interrupts.CPU110.RES:Rescheduling_interrupts
    611804 ±  2%     +10.1%     673890 ±  2%  interrupts.CPU111.LOC:Local_timer_interrupts
      7312 ±  3%     +14.4%       8365 ±  3%  interrupts.CPU111.RES:Rescheduling_interrupts
    611801 ±  2%     +10.1%     673890 ±  2%  interrupts.CPU112.LOC:Local_timer_interrupts
      7277 ±  2%     +14.6%       8337 ±  5%  interrupts.CPU112.RES:Rescheduling_interrupts
    611789 ±  2%     +10.2%     673896 ±  2%  interrupts.CPU113.LOC:Local_timer_interrupts
      7346 ±  4%     +15.2%       8461 ±  4%  interrupts.CPU113.RES:Rescheduling_interrupts
    611759 ±  2%     +10.2%     673861 ±  2%  interrupts.CPU114.LOC:Local_timer_interrupts
      7290 ±  4%     +17.9%       8594 ±  4%  interrupts.CPU114.RES:Rescheduling_interrupts
    611783 ±  2%     +10.2%     673883 ±  2%  interrupts.CPU115.LOC:Local_timer_interrupts
      7470 ±  4%     +13.6%       8489 ±  5%  interrupts.CPU115.RES:Rescheduling_interrupts
    611807 ±  2%     +10.2%     673925 ±  2%  interrupts.CPU116.LOC:Local_timer_interrupts
      7336 ±  4%     +15.3%       8455 ±  2%  interrupts.CPU116.RES:Rescheduling_interrupts
    611778 ±  2%     +10.2%     673890 ±  2%  interrupts.CPU117.LOC:Local_timer_interrupts
      7326 ±  5%     +15.9%       8493 ±  3%  interrupts.CPU117.RES:Rescheduling_interrupts
    611782 ±  2%     +10.1%     673849 ±  2%  interrupts.CPU118.LOC:Local_timer_interrupts
      7239 ±  4%     +18.4%       8574 ±  5%  interrupts.CPU118.RES:Rescheduling_interrupts
    611778 ±  2%     +10.2%     673881 ±  2%  interrupts.CPU119.LOC:Local_timer_interrupts
      7327 ±  4%     +13.7%       8335 ±  6%  interrupts.CPU119.RES:Rescheduling_interrupts
    611853 ±  2%     +10.1%     673944 ±  2%  interrupts.CPU12.LOC:Local_timer_interrupts
      7516 ±  4%     +17.0%       8798 ±  6%  interrupts.CPU12.RES:Rescheduling_interrupts
    611417 ±  2%     +10.2%     673606 ±  2%  interrupts.CPU120.LOC:Local_timer_interrupts
      7392 ±  4%     +14.2%       8444 ±  5%  interrupts.CPU120.RES:Rescheduling_interrupts
    611457 ±  2%     +10.2%     673608 ±  2%  interrupts.CPU121.LOC:Local_timer_interrupts
      7376 ±  3%     +15.6%       8525 ±  4%  interrupts.CPU121.RES:Rescheduling_interrupts
    611426 ±  2%     +10.2%     673610 ±  2%  interrupts.CPU122.LOC:Local_timer_interrupts
      7466 ±  5%     +11.7%       8336 ±  4%  interrupts.CPU122.RES:Rescheduling_interrupts
    611384 ±  2%     +10.2%     673577 ±  2%  interrupts.CPU123.LOC:Local_timer_interrupts
      7240 ±  5%     +18.2%       8559 ±  3%  interrupts.CPU123.RES:Rescheduling_interrupts
    611388 ±  2%     +10.2%     673597 ±  2%  interrupts.CPU124.LOC:Local_timer_interrupts
      7245 ±  5%     +18.0%       8548 ±  5%  interrupts.CPU124.RES:Rescheduling_interrupts
    611391 ±  2%     +10.2%     673567 ±  2%  interrupts.CPU125.LOC:Local_timer_interrupts
      7318 ±  3%     +15.3%       8436 ±  5%  interrupts.CPU125.RES:Rescheduling_interrupts
    611421 ±  2%     +10.2%     673570 ±  2%  interrupts.CPU126.LOC:Local_timer_interrupts
      7258 ±  5%     +16.4%       8448 ±  4%  interrupts.CPU126.RES:Rescheduling_interrupts
    611468 ±  2%     +10.2%     673614 ±  2%  interrupts.CPU127.LOC:Local_timer_interrupts
      7276 ±  5%     +16.9%       8508 ±  5%  interrupts.CPU127.RES:Rescheduling_interrupts
    611422 ±  2%     +10.2%     673591 ±  2%  interrupts.CPU128.LOC:Local_timer_interrupts
      7278 ±  4%     +17.0%       8518 ±  3%  interrupts.CPU128.RES:Rescheduling_interrupts
    611407 ±  2%     +10.2%     673562 ±  2%  interrupts.CPU129.LOC:Local_timer_interrupts
      7284 ±  3%     +17.0%       8519 ±  5%  interrupts.CPU129.RES:Rescheduling_interrupts
    611825 ±  2%     +10.1%     673920 ±  2%  interrupts.CPU13.LOC:Local_timer_interrupts
      7550 ±  4%     +16.2%       8771 ±  3%  interrupts.CPU13.RES:Rescheduling_interrupts
    611428 ±  2%     +10.2%     673571 ±  2%  interrupts.CPU130.LOC:Local_timer_interrupts
      7337 ±  5%     +16.4%       8541 ±  5%  interrupts.CPU130.RES:Rescheduling_interrupts
    611406 ±  2%     +10.2%     673582 ±  2%  interrupts.CPU131.LOC:Local_timer_interrupts
      7205 ±  5%     +17.1%       8440 ±  6%  interrupts.CPU131.RES:Rescheduling_interrupts
    611449 ±  2%     +10.2%     673586 ±  2%  interrupts.CPU132.LOC:Local_timer_interrupts
      7431 ±  4%     +12.7%       8372 ±  3%  interrupts.CPU132.RES:Rescheduling_interrupts
    611464 ±  2%     +10.2%     673588 ±  2%  interrupts.CPU133.LOC:Local_timer_interrupts
      7289 ±  4%     +15.9%       8446 ±  3%  interrupts.CPU133.RES:Rescheduling_interrupts
    611437 ±  2%     +10.2%     673583 ±  2%  interrupts.CPU134.LOC:Local_timer_interrupts
      7321 ±  4%     +17.7%       8616 ±  4%  interrupts.CPU134.RES:Rescheduling_interrupts
    611401 ±  2%     +10.2%     673595 ±  2%  interrupts.CPU135.LOC:Local_timer_interrupts
      7300 ±  4%     +15.7%       8446 ±  4%  interrupts.CPU135.RES:Rescheduling_interrupts
    611462 ±  2%     +10.2%     673602 ±  2%  interrupts.CPU136.LOC:Local_timer_interrupts
      7389 ±  4%     +13.9%       8417 ±  5%  interrupts.CPU136.RES:Rescheduling_interrupts
    611424 ±  2%     +10.2%     673588 ±  2%  interrupts.CPU137.LOC:Local_timer_interrupts
      7126 ±  4%     +20.0%       8550 ±  5%  interrupts.CPU137.RES:Rescheduling_interrupts
    611383 ±  2%     +10.2%     673556 ±  2%  interrupts.CPU138.LOC:Local_timer_interrupts
      7297 ±  5%     +16.1%       8475 ±  5%  interrupts.CPU138.RES:Rescheduling_interrupts
    611422 ±  2%     +10.2%     673603 ±  2%  interrupts.CPU139.LOC:Local_timer_interrupts
      7160 ±  4%     +17.6%       8423 ±  4%  interrupts.CPU139.RES:Rescheduling_interrupts
    611815 ±  2%     +10.1%     673901 ±  2%  interrupts.CPU14.LOC:Local_timer_interrupts
      7576 ±  5%     +16.3%       8810 ±  3%  interrupts.CPU14.RES:Rescheduling_interrupts
    611388 ±  2%     +10.2%     673577 ±  2%  interrupts.CPU140.LOC:Local_timer_interrupts
      7342 ±  6%     +12.9%       8293 ±  4%  interrupts.CPU140.RES:Rescheduling_interrupts
    611394 ±  2%     +10.2%     673596 ±  2%  interrupts.CPU141.LOC:Local_timer_interrupts
      7257 ±  4%     +16.1%       8423 ±  5%  interrupts.CPU141.RES:Rescheduling_interrupts
    611411 ±  2%     +10.2%     673590 ±  2%  interrupts.CPU142.LOC:Local_timer_interrupts
      7300 ±  6%     +16.1%       8475 ±  3%  interrupts.CPU142.RES:Rescheduling_interrupts
    611396 ±  2%     +10.2%     673622 ±  2%  interrupts.CPU143.LOC:Local_timer_interrupts
      7399 ±  5%     +14.0%       8435 ±  4%  interrupts.CPU143.RES:Rescheduling_interrupts
    610929 ±  2%     +10.3%     673686 ±  2%  interrupts.CPU144.LOC:Local_timer_interrupts
      7281 ±  5%     +13.4%       8257 ±  5%  interrupts.CPU144.RES:Rescheduling_interrupts
    610918 ±  2%     +10.3%     673755 ±  2%  interrupts.CPU145.LOC:Local_timer_interrupts
      7343 ±  5%     +14.5%       8408 ±  4%  interrupts.CPU145.RES:Rescheduling_interrupts
    610910 ±  2%     +10.3%     673705 ±  2%  interrupts.CPU146.LOC:Local_timer_interrupts
      7524 ±  4%     +13.6%       8550 ±  6%  interrupts.CPU146.RES:Rescheduling_interrupts
    610858 ±  2%     +10.3%     673769 ±  2%  interrupts.CPU147.LOC:Local_timer_interrupts
      7433 ±  4%     +10.8%       8237 ±  3%  interrupts.CPU147.RES:Rescheduling_interrupts
    610872 ±  2%     +10.3%     673727 ±  2%  interrupts.CPU148.LOC:Local_timer_interrupts
      7354 ±  3%     +13.1%       8320 ±  4%  interrupts.CPU148.RES:Rescheduling_interrupts
    610909 ±  2%     +10.3%     673750 ±  2%  interrupts.CPU149.LOC:Local_timer_interrupts
      7365 ±  3%     +16.5%       8582 ±  3%  interrupts.CPU149.RES:Rescheduling_interrupts
    611818 ±  2%     +10.2%     673937 ±  2%  interrupts.CPU15.LOC:Local_timer_interrupts
      7632 ±  4%     +12.5%       8587 ±  4%  interrupts.CPU15.RES:Rescheduling_interrupts
    610898 ±  2%     +10.3%     673770 ±  2%  interrupts.CPU150.LOC:Local_timer_interrupts
      7369 ±  4%     +12.4%       8282 ±  3%  interrupts.CPU150.RES:Rescheduling_interrupts
    610887 ±  2%     +10.3%     673754 ±  2%  interrupts.CPU151.LOC:Local_timer_interrupts
      7322 ±  4%     +15.3%       8445 ±  3%  interrupts.CPU151.RES:Rescheduling_interrupts
    610849 ±  2%     +10.3%     673762 ±  2%  interrupts.CPU152.LOC:Local_timer_interrupts
      7305 ±  4%     +15.5%       8439 ±  5%  interrupts.CPU152.RES:Rescheduling_interrupts
    610909 ±  2%     +10.3%     673759 ±  2%  interrupts.CPU153.LOC:Local_timer_interrupts
      7363 ±  6%     +14.5%       8432 ±  2%  interrupts.CPU153.RES:Rescheduling_interrupts
    610878 ±  2%     +10.3%     673730 ±  2%  interrupts.CPU154.LOC:Local_timer_interrupts
      7373 ±  4%     +14.1%       8414 ±  4%  interrupts.CPU154.RES:Rescheduling_interrupts
    610891 ±  2%     +10.3%     673720 ±  2%  interrupts.CPU155.LOC:Local_timer_interrupts
      7385 ±  5%     +11.8%       8260 ±  4%  interrupts.CPU155.RES:Rescheduling_interrupts
    610909 ±  2%     +10.3%     673753 ±  2%  interrupts.CPU156.LOC:Local_timer_interrupts
      7377 ±  5%     +14.0%       8410 ±  5%  interrupts.CPU156.RES:Rescheduling_interrupts
    610886 ±  2%     +10.3%     673754 ±  2%  interrupts.CPU157.LOC:Local_timer_interrupts
      7399 ±  5%     +13.1%       8370 ±  4%  interrupts.CPU157.RES:Rescheduling_interrupts
    610854 ±  2%     +10.3%     673699 ±  2%  interrupts.CPU158.LOC:Local_timer_interrupts
      7489 ±  4%     +10.5%       8278 ±  2%  interrupts.CPU158.RES:Rescheduling_interrupts
    610921 ±  2%     +10.3%     673726 ±  2%  interrupts.CPU159.LOC:Local_timer_interrupts
      7281 ±  5%     +15.4%       8402 ±  4%  interrupts.CPU159.RES:Rescheduling_interrupts
    611836 ±  2%     +10.1%     673889 ±  2%  interrupts.CPU16.LOC:Local_timer_interrupts
      7548 ±  4%     +15.6%       8723 ±  5%  interrupts.CPU16.RES:Rescheduling_interrupts
    610870 ±  2%     +10.3%     673742 ±  2%  interrupts.CPU160.LOC:Local_timer_interrupts
    610856 ±  2%     +10.3%     673722 ±  2%  interrupts.CPU161.LOC:Local_timer_interrupts
      7472 ±  5%     +13.4%       8475 ±  5%  interrupts.CPU161.RES:Rescheduling_interrupts
    610891 ±  2%     +10.3%     673708 ±  2%  interrupts.CPU162.LOC:Local_timer_interrupts
      7177 ±  5%     +14.8%       8239 ±  5%  interrupts.CPU162.RES:Rescheduling_interrupts
    610864 ±  2%     +10.3%     673696 ±  2%  interrupts.CPU163.LOC:Local_timer_interrupts
      7337 ±  5%     +12.7%       8266 ±  4%  interrupts.CPU163.RES:Rescheduling_interrupts
    610858 ±  2%     +10.3%     673724 ±  2%  interrupts.CPU164.LOC:Local_timer_interrupts
      7269 ±  5%     +15.2%       8376 ±  4%  interrupts.CPU164.RES:Rescheduling_interrupts
    610861 ±  2%     +10.3%     673711 ±  2%  interrupts.CPU165.LOC:Local_timer_interrupts
      7431 ±  4%      +9.4%       8131 ±  4%  interrupts.CPU165.RES:Rescheduling_interrupts
    610867 ±  2%     +10.3%     673684 ±  2%  interrupts.CPU166.LOC:Local_timer_interrupts
      7339 ±  2%     +14.1%       8374 ±  4%  interrupts.CPU166.RES:Rescheduling_interrupts
    610913 ±  2%     +10.3%     673730 ±  2%  interrupts.CPU167.LOC:Local_timer_interrupts
      7174 ±  3%     +17.5%       8427 ±  4%  interrupts.CPU167.RES:Rescheduling_interrupts
    611289 ±  2%     +10.2%     673888 ±  2%  interrupts.CPU168.LOC:Local_timer_interrupts
      7311 ±  5%     +14.7%       8382 ±  3%  interrupts.CPU168.RES:Rescheduling_interrupts
    611333 ±  2%     +10.2%     673887 ±  2%  interrupts.CPU169.LOC:Local_timer_interrupts
      7232 ±  2%     +17.8%       8517 ±  5%  interrupts.CPU169.RES:Rescheduling_interrupts
    611838 ±  2%     +10.1%     673879 ±  2%  interrupts.CPU17.LOC:Local_timer_interrupts
      7539 ±  2%     +17.7%       8873 ± 10%  interrupts.CPU17.RES:Rescheduling_interrupts
    611349 ±  2%     +10.2%     673889 ±  2%  interrupts.CPU170.LOC:Local_timer_interrupts
      7374 ±  2%     +14.2%       8424 ±  5%  interrupts.CPU170.RES:Rescheduling_interrupts
    611297 ±  2%     +10.2%     673873 ±  2%  interrupts.CPU171.LOC:Local_timer_interrupts
      7438 ±  4%     +13.9%       8471 ±  5%  interrupts.CPU171.RES:Rescheduling_interrupts
    611304 ±  2%     +10.2%     673845 ±  2%  interrupts.CPU172.LOC:Local_timer_interrupts
      7317 ±  3%     +16.6%       8529 ±  5%  interrupts.CPU172.RES:Rescheduling_interrupts
    611297 ±  2%     +10.2%     673887 ±  2%  interrupts.CPU173.LOC:Local_timer_interrupts
    611309 ±  2%     +10.2%     673858 ±  2%  interrupts.CPU174.LOC:Local_timer_interrupts
      7369 ±  5%     +15.8%       8531 ±  5%  interrupts.CPU174.RES:Rescheduling_interrupts
    611311 ±  2%     +10.2%     673872 ±  2%  interrupts.CPU175.LOC:Local_timer_interrupts
      7352 ±  4%     +15.1%       8465 ±  4%  interrupts.CPU175.RES:Rescheduling_interrupts
    611309 ±  2%     +10.2%     673886 ±  2%  interrupts.CPU176.LOC:Local_timer_interrupts
      7395 ±  2%     +14.0%       8434 ±  5%  interrupts.CPU176.RES:Rescheduling_interrupts
    611322 ±  2%     +10.2%     673868 ±  2%  interrupts.CPU177.LOC:Local_timer_interrupts
    611248 ±  2%     +10.2%     673805 ±  2%  interrupts.CPU178.LOC:Local_timer_interrupts
      7355 ±  5%     +14.1%       8393 ±  4%  interrupts.CPU178.RES:Rescheduling_interrupts
    611319 ±  2%     +10.2%     673906 ±  2%  interrupts.CPU179.LOC:Local_timer_interrupts
      7269 ±  4%     +15.2%       8374 ±  6%  interrupts.CPU179.RES:Rescheduling_interrupts
    611799 ±  2%     +10.1%     673864 ±  2%  interrupts.CPU18.LOC:Local_timer_interrupts
      7585 ±  4%     +14.5%       8683 ±  4%  interrupts.CPU18.RES:Rescheduling_interrupts
    611330 ±  2%     +10.2%     673869 ±  2%  interrupts.CPU180.LOC:Local_timer_interrupts
      7404 ±  4%     +14.0%       8442 ±  5%  interrupts.CPU180.RES:Rescheduling_interrupts
    611324 ±  2%     +10.2%     673889 ±  2%  interrupts.CPU181.LOC:Local_timer_interrupts
      7435 ±  2%     +13.0%       8399 ±  3%  interrupts.CPU181.RES:Rescheduling_interrupts
    611322 ±  2%     +10.2%     673874 ±  2%  interrupts.CPU182.LOC:Local_timer_interrupts
      7250 ±  4%     +14.6%       8305 ±  4%  interrupts.CPU182.RES:Rescheduling_interrupts
    611307 ±  2%     +10.2%     673852 ±  2%  interrupts.CPU183.LOC:Local_timer_interrupts
      7320 ±  4%     +13.2%       8290 ±  5%  interrupts.CPU183.RES:Rescheduling_interrupts
    611329 ±  2%     +10.2%     673877 ±  2%  interrupts.CPU184.LOC:Local_timer_interrupts
      7377 ±  4%     +12.5%       8299 ±  5%  interrupts.CPU184.RES:Rescheduling_interrupts
    611312 ±  2%     +10.2%     673866 ±  2%  interrupts.CPU185.LOC:Local_timer_interrupts
      7333 ±  4%     +13.9%       8351 ±  5%  interrupts.CPU185.RES:Rescheduling_interrupts
    611290 ±  2%     +10.2%     673868 ±  2%  interrupts.CPU186.LOC:Local_timer_interrupts
      7426 ±  4%     +14.1%       8471 ±  3%  interrupts.CPU186.RES:Rescheduling_interrupts
    611304 ±  2%     +10.2%     673853 ±  2%  interrupts.CPU187.LOC:Local_timer_interrupts
      7380 ±  3%     +12.5%       8304 ±  5%  interrupts.CPU187.RES:Rescheduling_interrupts
    611307 ±  2%     +10.2%     673881 ±  2%  interrupts.CPU188.LOC:Local_timer_interrupts
      7277 ±  5%     +16.3%       8467 ±  6%  interrupts.CPU188.RES:Rescheduling_interrupts
    611299 ±  2%     +10.2%     673879 ±  2%  interrupts.CPU189.LOC:Local_timer_interrupts
      7436 ±  3%     +14.5%       8514 ±  6%  interrupts.CPU189.RES:Rescheduling_interrupts
    611761 ±  2%     +10.2%     673928 ±  2%  interrupts.CPU19.LOC:Local_timer_interrupts
      7515 ±  6%     +12.7%       8467 ±  4%  interrupts.CPU19.RES:Rescheduling_interrupts
    611321 ±  2%     +10.2%     673856 ±  2%  interrupts.CPU190.LOC:Local_timer_interrupts
      7188 ±  3%     +16.4%       8366 ±  5%  interrupts.CPU190.RES:Rescheduling_interrupts
    611544 ±  2%     +10.2%     674120 ±  2%  interrupts.CPU191.LOC:Local_timer_interrupts
      7207 ±  3%     +15.3%       8312 ±  3%  interrupts.CPU191.RES:Rescheduling_interrupts
    611805 ±  2%     +10.2%     673920 ±  2%  interrupts.CPU2.LOC:Local_timer_interrupts
    611781 ±  2%     +10.2%     673897 ±  2%  interrupts.CPU20.LOC:Local_timer_interrupts
      7531 ±  5%     +16.4%       8768 ±  3%  interrupts.CPU20.RES:Rescheduling_interrupts
    611715 ±  2%     +10.2%     673897 ±  2%  interrupts.CPU21.LOC:Local_timer_interrupts
      7467 ±  3%     +14.9%       8577 ±  4%  interrupts.CPU21.RES:Rescheduling_interrupts
    611834 ±  2%     +10.1%     673900 ±  2%  interrupts.CPU22.LOC:Local_timer_interrupts
      7427 ±  6%     +17.9%       8754 ±  4%  interrupts.CPU22.RES:Rescheduling_interrupts
    611788 ±  2%     +10.1%     673869 ±  2%  interrupts.CPU23.LOC:Local_timer_interrupts
      7592 ±  4%     +12.2%       8518 ±  4%  interrupts.CPU23.RES:Rescheduling_interrupts
    611426 ±  2%     +10.2%     673619 ±  2%  interrupts.CPU24.LOC:Local_timer_interrupts
      7457 ±  4%     +14.7%       8551 ±  3%  interrupts.CPU24.RES:Rescheduling_interrupts
    611439 ±  2%     +10.2%     673636 ±  2%  interrupts.CPU25.LOC:Local_timer_interrupts
    611402 ±  2%     +10.2%     673620 ±  2%  interrupts.CPU26.LOC:Local_timer_interrupts
      7463 ±  3%     +18.4%       8838 ±  7%  interrupts.CPU26.RES:Rescheduling_interrupts
    611413 ±  2%     +10.2%     673615 ±  2%  interrupts.CPU27.LOC:Local_timer_interrupts
      7412 ±  4%     +16.1%       8607 ±  4%  interrupts.CPU27.RES:Rescheduling_interrupts
    611409 ±  2%     +10.2%     673625 ±  2%  interrupts.CPU28.LOC:Local_timer_interrupts
      7478 ±  5%     +15.6%       8648 ±  3%  interrupts.CPU28.RES:Rescheduling_interrupts
    611415 ±  2%     +10.2%     673620 ±  2%  interrupts.CPU29.LOC:Local_timer_interrupts
      7431 ±  2%     +15.5%       8585 ±  2%  interrupts.CPU29.RES:Rescheduling_interrupts
    611807 ±  2%     +10.1%     673896 ±  2%  interrupts.CPU3.LOC:Local_timer_interrupts
      7659 ±  5%     +20.9%       9264 ± 10%  interrupts.CPU3.RES:Rescheduling_interrupts
    611433 ±  2%     +10.2%     673520 ±  2%  interrupts.CPU30.LOC:Local_timer_interrupts
      7432 ±  3%     +16.5%       8659 ±  4%  interrupts.CPU30.RES:Rescheduling_interrupts
    611450 ±  2%     +10.2%     673604 ±  2%  interrupts.CPU31.LOC:Local_timer_interrupts
      7490 ±  4%     +14.6%       8585 ±  5%  interrupts.CPU31.RES:Rescheduling_interrupts
    611444 ±  2%     +10.2%     673588 ±  2%  interrupts.CPU32.LOC:Local_timer_interrupts
      7393 ±  4%     +19.0%       8799 ±  3%  interrupts.CPU32.RES:Rescheduling_interrupts
    611407 ±  2%     +10.2%     673606 ±  2%  interrupts.CPU33.LOC:Local_timer_interrupts
      7398 ±  4%     +16.8%       8644 ±  4%  interrupts.CPU33.RES:Rescheduling_interrupts
    611452 ±  2%     +10.2%     673609 ±  2%  interrupts.CPU34.LOC:Local_timer_interrupts
      7414 ±  5%     +18.8%       8806 ±  4%  interrupts.CPU34.RES:Rescheduling_interrupts
    611433 ±  2%     +10.2%     673588 ±  2%  interrupts.CPU35.LOC:Local_timer_interrupts
      7486 ±  5%     +18.6%       8878 ±  3%  interrupts.CPU35.RES:Rescheduling_interrupts
    611450 ±  2%     +10.2%     673603 ±  2%  interrupts.CPU36.LOC:Local_timer_interrupts
      7503 ±  3%     +15.0%       8627 ±  3%  interrupts.CPU36.RES:Rescheduling_interrupts
    611465 ±  2%     +10.2%     673611 ±  2%  interrupts.CPU37.LOC:Local_timer_interrupts
      7416 ±  3%     +17.6%       8719 ±  2%  interrupts.CPU37.RES:Rescheduling_interrupts
    611445 ±  2%     +10.2%     673575 ±  2%  interrupts.CPU38.LOC:Local_timer_interrupts
      7477 ±  5%     +14.2%       8540 ±  3%  interrupts.CPU38.RES:Rescheduling_interrupts
    611420 ±  2%     +10.2%     673600 ±  2%  interrupts.CPU39.LOC:Local_timer_interrupts
      7562 ±  5%     +14.0%       8620 ±  4%  interrupts.CPU39.RES:Rescheduling_interrupts
    611823 ±  2%     +10.2%     673949 ±  2%  interrupts.CPU4.LOC:Local_timer_interrupts
      7579 ±  4%     +14.9%       8708 ±  3%  interrupts.CPU4.RES:Rescheduling_interrupts
    611390 ±  2%     +10.2%     673618 ±  2%  interrupts.CPU40.LOC:Local_timer_interrupts
      7444 ±  4%     +15.9%       8626 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
    611429 ±  2%     +10.2%     673612 ±  2%  interrupts.CPU41.LOC:Local_timer_interrupts
      7464 ±  5%     +15.3%       8605 ±  2%  interrupts.CPU41.RES:Rescheduling_interrupts
    611418 ±  2%     +10.2%     673597 ±  2%  interrupts.CPU42.LOC:Local_timer_interrupts
      7394 ±  4%     +18.6%       8772 ±  5%  interrupts.CPU42.RES:Rescheduling_interrupts
    611364 ±  2%     +10.2%     673601 ±  2%  interrupts.CPU43.LOC:Local_timer_interrupts
      7483 ±  5%     +15.3%       8630 ±  3%  interrupts.CPU43.RES:Rescheduling_interrupts
    611421 ±  2%     +10.2%     673593 ±  2%  interrupts.CPU44.LOC:Local_timer_interrupts
      7463 ±  2%     +16.0%       8656 ±  4%  interrupts.CPU44.RES:Rescheduling_interrupts
    611423 ±  2%     +10.2%     673593 ±  2%  interrupts.CPU45.LOC:Local_timer_interrupts
      7455 ±  4%     +14.8%       8558 ±  3%  interrupts.CPU45.RES:Rescheduling_interrupts
    611333 ±  2%     +10.2%     673616 ±  2%  interrupts.CPU46.LOC:Local_timer_interrupts
      7406 ±  3%     +17.4%       8696 ±  3%  interrupts.CPU46.RES:Rescheduling_interrupts
    611421 ±  2%     +10.2%     673605 ±  2%  interrupts.CPU47.LOC:Local_timer_interrupts
      7482 ±  5%     +15.7%       8660 ±  2%  interrupts.CPU47.RES:Rescheduling_interrupts
    610910 ±  2%     +10.3%     673737 ±  2%  interrupts.CPU48.LOC:Local_timer_interrupts
      7521 ±  6%     +15.1%       8655 ±  4%  interrupts.CPU48.RES:Rescheduling_interrupts
    610921 ±  2%     +10.3%     673770 ±  2%  interrupts.CPU49.LOC:Local_timer_interrupts
      7594 ±  5%     +12.7%       8558 ±  3%  interrupts.CPU49.RES:Rescheduling_interrupts
    611796 ±  2%     +10.1%     673831 ±  2%  interrupts.CPU5.LOC:Local_timer_interrupts
    610930 ±  2%     +10.3%     673671 ±  2%  interrupts.CPU50.LOC:Local_timer_interrupts
      7504 ±  2%     +17.6%       8824 ±  4%  interrupts.CPU50.RES:Rescheduling_interrupts
    610882 ±  2%     +10.3%     673738 ±  2%  interrupts.CPU51.LOC:Local_timer_interrupts
    610884 ±  2%     +10.3%     673731 ±  2%  interrupts.CPU52.LOC:Local_timer_interrupts
      7436 ±  4%     +18.1%       8783 ±  3%  interrupts.CPU52.RES:Rescheduling_interrupts
    610885 ±  2%     +10.3%     673774 ±  2%  interrupts.CPU53.LOC:Local_timer_interrupts
      7544 ±  4%     +15.3%       8700 ±  4%  interrupts.CPU53.RES:Rescheduling_interrupts
    610902 ±  2%     +10.3%     673763 ±  2%  interrupts.CPU54.LOC:Local_timer_interrupts
      7534 ±  4%     +15.6%       8709 ±  6%  interrupts.CPU54.RES:Rescheduling_interrupts
    610927 ±  2%     +10.3%     673739 ±  2%  interrupts.CPU55.LOC:Local_timer_interrupts
    610883 ±  2%     +10.3%     673719 ±  2%  interrupts.CPU56.LOC:Local_timer_interrupts
    610921 ±  2%     +10.3%     673689 ±  2%  interrupts.CPU57.LOC:Local_timer_interrupts
      7473 ±  5%     +14.8%       8577 ±  4%  interrupts.CPU57.RES:Rescheduling_interrupts
    610893 ±  2%     +10.3%     673781 ±  2%  interrupts.CPU58.LOC:Local_timer_interrupts
      7407 ±  5%     +16.6%       8635 ±  3%  interrupts.CPU58.RES:Rescheduling_interrupts
    610898 ±  2%     +10.3%     673763 ±  2%  interrupts.CPU59.LOC:Local_timer_interrupts
      7642 ±  5%     +13.1%       8643 ±  3%  interrupts.CPU59.RES:Rescheduling_interrupts
    611823 ±  2%     +10.1%     673914 ±  2%  interrupts.CPU6.LOC:Local_timer_interrupts
      7542 ±  3%     +17.3%       8845 ±  3%  interrupts.CPU6.RES:Rescheduling_interrupts
    610869 ±  2%     +10.3%     673784 ±  2%  interrupts.CPU60.LOC:Local_timer_interrupts
      7535 ±  4%     +14.8%       8647 ±  5%  interrupts.CPU60.RES:Rescheduling_interrupts
    610906 ±  2%     +10.3%     673701 ±  2%  interrupts.CPU61.LOC:Local_timer_interrupts
      7631 ±  4%     +13.5%       8660 ±  4%  interrupts.CPU61.RES:Rescheduling_interrupts
    610863 ±  2%     +10.3%     673705 ±  2%  interrupts.CPU62.LOC:Local_timer_interrupts
      7634 ±  4%     +11.8%       8532 ±  4%  interrupts.CPU62.RES:Rescheduling_interrupts
    610920 ±  2%     +10.3%     673730 ±  2%  interrupts.CPU63.LOC:Local_timer_interrupts
      7549 ±  5%     +14.8%       8663 ±  5%  interrupts.CPU63.RES:Rescheduling_interrupts
    610884 ±  2%     +10.3%     673727 ±  2%  interrupts.CPU64.LOC:Local_timer_interrupts
      7644 ±  3%     +13.4%       8672 ±  3%  interrupts.CPU64.RES:Rescheduling_interrupts
    610853 ±  2%     +10.3%     673708 ±  2%  interrupts.CPU65.LOC:Local_timer_interrupts
      7555 ±  5%     +14.8%       8672 ±  3%  interrupts.CPU65.RES:Rescheduling_interrupts
    610950 ±  2%     +10.3%     673804 ±  2%  interrupts.CPU66.LOC:Local_timer_interrupts
      7457 ±  5%     +16.8%       8710 ±  4%  interrupts.CPU66.RES:Rescheduling_interrupts
    610888 ±  2%     +10.3%     673735 ±  2%  interrupts.CPU67.LOC:Local_timer_interrupts
      7550 ±  4%     +11.5%       8415 ±  5%  interrupts.CPU67.RES:Rescheduling_interrupts
    610891 ±  2%     +10.3%     673727 ±  2%  interrupts.CPU68.LOC:Local_timer_interrupts
      7612 ±  4%     +12.1%       8534 ±  3%  interrupts.CPU68.RES:Rescheduling_interrupts
    610901 ±  2%     +10.3%     673659 ±  2%  interrupts.CPU69.LOC:Local_timer_interrupts
      7493 ±  4%     +14.8%       8601 ±  4%  interrupts.CPU69.RES:Rescheduling_interrupts
    611808 ±  2%     +10.1%     673844 ±  2%  interrupts.CPU7.LOC:Local_timer_interrupts
      7673 ±  5%     +15.0%       8821 ±  4%  interrupts.CPU7.RES:Rescheduling_interrupts
    610906 ±  2%     +10.3%     673733 ±  2%  interrupts.CPU70.LOC:Local_timer_interrupts
      7412 ±  3%     +16.9%       8664 ±  4%  interrupts.CPU70.RES:Rescheduling_interrupts
    610871 ±  2%     +10.3%     673671 ±  2%  interrupts.CPU71.LOC:Local_timer_interrupts
      7519 ±  4%     +13.8%       8557 ±  5%  interrupts.CPU71.RES:Rescheduling_interrupts
    611325 ±  2%     +10.2%     673906 ±  2%  interrupts.CPU72.LOC:Local_timer_interrupts
      7720 ±  4%      +9.6%       8459 ±  4%  interrupts.CPU72.RES:Rescheduling_interrupts
    611328 ±  2%     +10.2%     673908 ±  2%  interrupts.CPU73.LOC:Local_timer_interrupts
      7535 ±  4%     +14.6%       8634 ±  4%  interrupts.CPU73.RES:Rescheduling_interrupts
    611311 ±  2%     +10.2%     673883 ±  2%  interrupts.CPU74.LOC:Local_timer_interrupts
      7526 ±  5%     +14.7%       8635 ±  6%  interrupts.CPU74.RES:Rescheduling_interrupts
    611315 ±  2%     +10.2%     673868 ±  2%  interrupts.CPU75.LOC:Local_timer_interrupts
      7712 ±  5%      +9.8%       8469 ±  4%  interrupts.CPU75.RES:Rescheduling_interrupts
    611304 ±  2%     +10.2%     673860 ±  2%  interrupts.CPU76.LOC:Local_timer_interrupts
      7476 ±  5%     +14.2%       8537 ±  5%  interrupts.CPU76.RES:Rescheduling_interrupts
    611322 ±  2%     +10.2%     673834 ±  2%  interrupts.CPU77.LOC:Local_timer_interrupts
      7670 ±  3%     +12.5%       8630 ±  5%  interrupts.CPU77.RES:Rescheduling_interrupts
    611326 ±  2%     +10.2%     673851 ±  2%  interrupts.CPU78.LOC:Local_timer_interrupts
      7417 ±  3%     +15.6%       8578 ±  4%  interrupts.CPU78.RES:Rescheduling_interrupts
    611328 ±  2%     +10.2%     673820 ±  2%  interrupts.CPU79.LOC:Local_timer_interrupts
      7545 ±  4%     +13.9%       8590 ±  4%  interrupts.CPU79.RES:Rescheduling_interrupts
    611842 ±  2%     +10.1%     673928 ±  2%  interrupts.CPU8.LOC:Local_timer_interrupts
      7673 ±  3%     +14.9%       8818 ±  4%  interrupts.CPU8.RES:Rescheduling_interrupts
    611366 ±  2%     +10.2%     673868 ±  2%  interrupts.CPU80.LOC:Local_timer_interrupts
      7513 ±  3%     +16.1%       8723 ±  5%  interrupts.CPU80.RES:Rescheduling_interrupts
    611322 ±  2%     +10.2%     673860 ±  2%  interrupts.CPU81.LOC:Local_timer_interrupts
    611299 ±  2%     +10.2%     673868 ±  2%  interrupts.CPU82.LOC:Local_timer_interrupts
      7523 ±  4%     +13.8%       8562 ±  4%  interrupts.CPU82.RES:Rescheduling_interrupts
    611320 ±  2%     +10.2%     673727 ±  2%  interrupts.CPU83.LOC:Local_timer_interrupts
      7518 ±  5%     +12.4%       8454 ±  5%  interrupts.CPU83.RES:Rescheduling_interrupts
    611326 ±  2%     +10.2%     673899 ±  2%  interrupts.CPU84.LOC:Local_timer_interrupts
      7452 ±  6%     +12.3%       8365 ±  5%  interrupts.CPU84.RES:Rescheduling_interrupts
    611329 ±  2%     +10.2%     673885 ±  2%  interrupts.CPU85.LOC:Local_timer_interrupts
      7399 ±  5%     +11.1%       8218 ±  5%  interrupts.CPU85.RES:Rescheduling_interrupts
    611278 ±  2%     +10.2%     673875 ±  2%  interrupts.CPU86.LOC:Local_timer_interrupts
      7376 ±  4%     +16.3%       8581 ±  3%  interrupts.CPU86.RES:Rescheduling_interrupts
    611313 ±  2%     +10.2%     673874 ±  2%  interrupts.CPU87.LOC:Local_timer_interrupts
      7544 ±  4%     +12.8%       8508 ±  4%  interrupts.CPU87.RES:Rescheduling_interrupts
    611351 ±  2%     +10.2%     673928 ±  2%  interrupts.CPU88.LOC:Local_timer_interrupts
      7418 ±  6%     +13.3%       8406 ±  4%  interrupts.CPU88.RES:Rescheduling_interrupts
    611324 ±  2%     +10.2%     673795 ±  2%  interrupts.CPU89.LOC:Local_timer_interrupts
      7426 ±  4%     +13.6%       8439 ±  5%  interrupts.CPU89.RES:Rescheduling_interrupts
    611832 ±  2%     +10.2%     673968 ±  2%  interrupts.CPU9.LOC:Local_timer_interrupts
      7538 ±  3%     +15.6%       8715 ±  4%  interrupts.CPU9.RES:Rescheduling_interrupts
    611298 ±  2%     +10.2%     673878 ±  2%  interrupts.CPU90.LOC:Local_timer_interrupts
      7421 ±  3%     +12.7%       8363 ±  4%  interrupts.CPU90.RES:Rescheduling_interrupts
    611305 ±  2%     +10.2%     673874 ±  2%  interrupts.CPU91.LOC:Local_timer_interrupts
      7377 ±  5%     +15.6%       8528 ±  5%  interrupts.CPU91.RES:Rescheduling_interrupts
    611288 ±  2%     +10.2%     673876 ±  2%  interrupts.CPU92.LOC:Local_timer_interrupts
      7349 ±  4%     +12.0%       8234 ±  4%  interrupts.CPU92.RES:Rescheduling_interrupts
    611329 ±  2%     +10.2%     673857 ±  2%  interrupts.CPU93.LOC:Local_timer_interrupts
      7438 ±  3%     +14.7%       8530 ±  4%  interrupts.CPU93.RES:Rescheduling_interrupts
    611276 ±  2%     +10.2%     673866 ±  2%  interrupts.CPU94.LOC:Local_timer_interrupts
      7363 ±  4%     +12.7%       8300 ±  5%  interrupts.CPU94.RES:Rescheduling_interrupts
    611381 ±  2%     +10.2%     673922 ±  2%  interrupts.CPU95.LOC:Local_timer_interrupts
     10549 ±  3%     +14.9%      12117 ±  4%  interrupts.CPU95.RES:Rescheduling_interrupts
    611824 ±  2%     +10.1%     673847 ±  2%  interrupts.CPU96.LOC:Local_timer_interrupts
      7416 ±  3%     +12.9%       8375 ±  5%  interrupts.CPU96.RES:Rescheduling_interrupts
    611751 ±  2%     +10.2%     673870 ±  2%  interrupts.CPU97.LOC:Local_timer_interrupts
      7437 ±  4%     +14.0%       8477 ±  4%  interrupts.CPU97.RES:Rescheduling_interrupts
    611787 ±  2%     +10.1%     673877 ±  2%  interrupts.CPU98.LOC:Local_timer_interrupts
      7555 ±  4%     +13.0%       8534 ±  3%  interrupts.CPU98.RES:Rescheduling_interrupts
    611783 ±  2%     +10.1%     673872 ±  2%  interrupts.CPU99.LOC:Local_timer_interrupts
      7452 ±  3%     +12.8%       8404 ±  5%  interrupts.CPU99.RES:Rescheduling_interrupts
 1.174e+08 ±  2%     +10.2%  1.294e+08 ±  2%  interrupts.LOC:Local_timer_interrupts
   1433875 ±  3%     +14.6%    1643307 ±  3%  interrupts.RES:Rescheduling_interrupts


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  30500 +-------------------------------------------------------------------+   
        |                               ..+....                            .|   
  30000 |-+                       ......       ...                      ... |   
  29500 |................   ......                ..                 ...    |   
        |                +..                        ...            ..       |   
  29000 |-+                                            ...      ...         |   
  28500 |-+                                               .  ...            |   
        |                                                  +.               |   
  28000 |-+              O                                 O                |   
  27500 |-+                                                                 |   
        |                                                                   |   
  27000 |-+                                                                 |   
  26500 |-+                                                                 |   
        |                                 O                                 |   
  26000 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc7-00198-gc75e839414d36" of type "text/plain" (203570 bytes)

View attachment "job-script" of type "text/plain" (7970 bytes)

View attachment "job.yaml" of type "text/plain" (5490 bytes)

View attachment "reproduce" of type "text/plain" (1005 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ