lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Fri, 27 Mar 2020 15:40:02 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Kees Cook <keescook@...omium.org>
Cc:     LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [x86/entry] 8c5045f89b: will-it-scale.per_process_ops -12.9%
 regression

Greeting,

FYI, we noticed a -12.9% regression of will-it-scale.per_process_ops due to commit:


commit: 8c5045f89b181a4412b698cafc9dc257192a3a65 ("x86/entry: Enable random_kstack_offset support")
https://git.kernel.org/cgit/linux/kernel/git/kees/linux.git kspp/stack/offset/agnostic/v1

in testcase: will-it-scale
on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
with following parameters:

	nr_task: 100%
	mode: process
	test: lseek1
	cpufreq_governor: performance
	ucode: 0x11

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/lseek1/will-it-scale/0x11

commit: 
  3dd7e7ac5e ("stack: Optionally randomize kernel stack offset each syscall")
  8c5045f89b ("x86/entry: Enable random_kstack_offset support")

3dd7e7ac5ef51bef 8c5045f89b181a4412b698cafc9 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :4           50%           2:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
         %stddev     %change         %stddev
             \          |                \  
    334540           -12.9%     291311        will-it-scale.per_process_ops
    311.95            +1.5%     316.63        will-it-scale.time.elapsed_time
    311.95            +1.5%     316.63        will-it-scale.time.elapsed_time.max
  96347834           -12.9%   83897839        will-it-scale.workload
    867.00 ± 68%     -45.4%     473.25 ±  2%  cpuidle.POLL.usage
      0.00 ± 22%      +0.0        0.00 ± 29%  mpstat.cpu.all.soft%
     33673 ±  4%     +27.5%      42950        numa-meminfo.node0.Mapped
      8519 ±  4%     +26.3%      10759        numa-vmstat.node0.nr_mapped
     44724 ±  3%     +20.7%      53969        meminfo.Mapped
     11436 ±  4%      +5.8%      12104 ±  5%  meminfo.max_used_kB
     52.25            +2.9%      53.75        vmstat.cpu.sy
     45.50            -4.9%      43.25        vmstat.cpu.us
      1242 ±  8%     +39.1%       1728 ±  9%  slabinfo.pool_workqueue.active_objs
      1258 ±  7%     +38.6%       1744 ±  9%  slabinfo.pool_workqueue.num_objs
    798.25 ±  8%     +32.3%       1056 ±  9%  slabinfo.skbuff_fclone_cache.active_objs
    798.25 ±  8%     +32.3%       1056 ±  9%  slabinfo.skbuff_fclone_cache.num_objs
    133626            +2.2%     136556        proc-vmstat.nr_active_anon
     11260 ±  3%     +20.2%      13540        proc-vmstat.nr_mapped
    133626            +2.2%     136556        proc-vmstat.nr_zone_active_anon
    935337            -1.4%     922644        proc-vmstat.numa_hit
    935337            -1.4%     922644        proc-vmstat.numa_local
      2356 ±  2%     +14.0%       2685 ±  5%  proc-vmstat.pgactivate
   1020296            -2.8%     991613        proc-vmstat.pgalloc_normal
    870609            -2.3%     850589        proc-vmstat.pgfault
      1990 ±  4%     +43.4%       2853 ±  4%  sched_debug.cfs_rq:/.exec_clock.stddev
    851835 ±  3%     +36.6%    1163841 ±  4%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.03 ± 23%     +35.9%       0.05 ± 11%  sched_debug.cfs_rq:/.nr_running.stddev
      1.83 ±  6%     +20.0%       2.20        sched_debug.cfs_rq:/.nr_spread_over.avg
  -6622421           +30.1%   -8616279        sched_debug.cfs_rq:/.spread0.min
    497607 ±  6%     +29.5%     644599 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
      1942 ±  2%     +41.1%       2741 ±  3%  sched_debug.cpu.clock.stddev
      1942 ±  2%     +41.1%       2741 ±  3%  sched_debug.cpu.clock_task.stddev
    349394 ± 11%     +20.8%     422164 ±  7%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ±  2%     +41.6%       0.00 ±  3%  sched_debug.cpu.next_balance.stddev
      0.10 ± 13%     +18.1%       0.12 ±  8%  sched_debug.cpu.nr_running.stddev
     17187 ±  7%     +67.7%      28831 ± 38%  sched_debug.cpu.nr_switches.max
      1836 ±  5%     +34.3%       2467 ± 16%  sched_debug.cpu.nr_switches.stddev
     11711 ± 13%    +107.1%      24250 ± 47%  sched_debug.cpu.sched_count.max
      1413 ±  5%     +46.2%       2066 ± 22%  sched_debug.cpu.sched_count.stddev
      5063 ± 13%    +122.3%      11254 ± 44%  sched_debug.cpu.ttwu_count.max
    639.38 ±  8%     +52.2%     972.84 ± 23%  sched_debug.cpu.ttwu_count.stddev
      4213 ± 12%    +149.2%      10499 ± 48%  sched_debug.cpu.ttwu_local.max
    504.61 ±  6%     +71.7%     866.25 ± 26%  sched_debug.cpu.ttwu_local.stddev
 6.236e+09           -10.9%  5.556e+09        perf-stat.i.branch-instructions
      8.35            -0.5        7.86        perf-stat.i.branch-miss-rate%
 5.211e+08           -16.1%  4.373e+08        perf-stat.i.branch-misses
  50328901 ±  2%      -6.3%   47148698 ±  3%  perf-stat.i.cache-misses
 1.261e+09            -6.7%  1.176e+09 ±  2%  perf-stat.i.cache-references
     14.96            +8.9%      16.29        perf-stat.i.cpi
    258.22            -7.3%     239.33 ±  3%  perf-stat.i.cpu-migrations
      8795            +6.4%       9362 ±  3%  perf-stat.i.cycles-between-cache-misses
      1.44            -0.0        1.40        perf-stat.i.iTLB-load-miss-rate%
 4.339e+08           -11.5%  3.841e+08        perf-stat.i.iTLB-load-misses
 2.961e+10            -8.5%  2.711e+10        perf-stat.i.iTLB-loads
 2.957e+10            -8.4%  2.707e+10        perf-stat.i.instructions
     68.40            +3.3%      70.68        perf-stat.i.instructions-per-iTLB-miss
      0.07            -7.8%       0.06        perf-stat.i.ipc
      2747            -1.8%       2697        perf-stat.i.minor-faults
      2747            -1.8%       2697        perf-stat.i.page-faults
     42.73            +2.0%      43.59        perf-stat.overall.MPKI
      8.35            -0.5        7.87        perf-stat.overall.branch-miss-rate%
     15.00            +9.1%      16.37        perf-stat.overall.cpi
      8793            +6.7%       9386 ±  3%  perf-stat.overall.cycles-between-cache-misses
      1.45            -0.0        1.40        perf-stat.overall.iTLB-load-miss-rate%
     68.14            +3.4%      70.46        perf-stat.overall.instructions-per-iTLB-miss
      0.07            -8.3%       0.06        perf-stat.overall.ipc
     95291            +6.9%     101893        perf-stat.overall.path-length
  6.22e+09           -10.9%  5.539e+09        perf-stat.ps.branch-instructions
 5.192e+08           -16.1%  4.358e+08        perf-stat.ps.branch-misses
  50344143            -6.4%   47117613 ±  3%  perf-stat.ps.cache-misses
  1.26e+09            -6.6%  1.177e+09 ±  2%  perf-stat.ps.cache-references
    240.97           -10.5%     215.71 ±  4%  perf-stat.ps.cpu-migrations
 4.328e+08           -11.5%   3.83e+08        perf-stat.ps.iTLB-load-misses
 2.951e+10            -8.5%  2.699e+10        perf-stat.ps.iTLB-loads
 2.949e+10            -8.5%  2.698e+10        perf-stat.ps.instructions
      2675            -3.7%       2577        perf-stat.ps.minor-faults
      2675            -3.7%       2577        perf-stat.ps.page-faults
 9.181e+12            -6.9%  8.548e+12        perf-stat.total.instructions
     26.34 ± 16%      -7.4       18.91 ± 16%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.lseek64
      6.32            -1.1        5.20        perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.80 ± 22%      +0.4        1.22 ± 26%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.lseek64
      1.29 ± 21%      +0.6        1.84 ± 26%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.lseek64
      0.83 ± 15%      +0.8        1.60 ± 25%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.do_syscall_64
      1.75 ±  4%      +0.8        2.56 ± 13%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.91 ± 14%      +1.1        2.97 ± 27%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
      1.77 ± 14%      +1.1        2.84 ± 27%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
     27.02 ± 17%     +18.1       45.13 ± 17%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.lseek64
     24.70 ± 17%     +18.4       43.07 ± 17%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
     32.19           -11.6       20.58 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64
     20.79 ±  2%      -5.8       14.99 ±  5%  perf-profile.children.cycles-pp.syscall_return_via_sysret
     14.12 ±  2%      -2.5       11.62        perf-profile.children.cycles-pp.ksys_lseek
      6.94            -1.2        5.72        perf-profile.children.cycles-pp.__fdget_pos
      6.40            -1.1        5.26        perf-profile.children.cycles-pp.__fget_light
      4.47 ±  2%      -0.8        3.62 ±  2%  perf-profile.children.cycles-pp.shmem_file_llseek
      3.12            -0.7        2.39 ±  3%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.55 ±  2%      -0.1        0.43 ±  3%  perf-profile.children.cycles-pp.generic_file_llseek_size
      0.40 ±  5%      -0.1        0.29 ± 14%  perf-profile.children.cycles-pp.interrupt_entry
      0.75            -0.1        0.66        perf-profile.children.cycles-pp.__x64_sys_lseek
      0.51            -0.0        0.46 ±  3%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.09 ±  8%      -0.0        0.07        perf-profile.children.cycles-pp.rb_next
      0.15 ±  5%      -0.0        0.14 ±  6%  perf-profile.children.cycles-pp.timerqueue_del
      0.09 ±  5%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.04 ± 58%      +0.0        0.07 ± 15%  perf-profile.children.cycles-pp.do_page_fault
      0.01 ±173%      +0.0        0.06 ± 14%  perf-profile.children.cycles-pp.ret_from_intr
      0.06 ±  6%      +0.0        0.11 ± 23%  perf-profile.children.cycles-pp.raise_softirq
      0.12 ±  8%      +0.0        0.17 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      0.13 ±  8%      +0.0        0.18 ± 16%  perf-profile.children.cycles-pp.ktime_get
      0.08 ±  8%      +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.vfs_read
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.ksys_read
      0.00            +0.1        0.06 ± 28%  perf-profile.children.cycles-pp.rcu_irq_exit
      0.07 ±  7%      +0.1        0.13 ± 28%  perf-profile.children.cycles-pp.timerqueue_add
      0.08 ±  5%      +0.1        0.15 ± 27%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.07 ± 12%      +0.1        0.14 ± 23%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.rcu_segcblist_ready_cbs
      0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.idle_cpu
      0.01 ±173%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.ksoftirqd_running
      0.10 ±  4%      +0.1        0.20 ± 20%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.06 ± 11%      +0.1        0.16 ± 20%  perf-profile.children.cycles-pp.run_timer_softirq
      0.09 ±  8%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.56 ±  3%      +0.1        0.70 ±  7%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.21 ±  7%      +0.2        0.36 ± 25%  perf-profile.children.cycles-pp.update_load_avg
      0.65 ±  4%      +0.2        0.83 ±  8%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      1.04 ±  3%      +0.2        1.29 ±  2%  perf-profile.children.cycles-pp.change_protection
      1.03 ±  3%      +0.3        1.28 ±  2%  perf-profile.children.cycles-pp.change_prot_numa
      1.03 ±  3%      +0.3        1.28 ±  2%  perf-profile.children.cycles-pp.change_p4d_range
      1.04 ±  3%      +0.3        1.29 ±  2%  perf-profile.children.cycles-pp.task_numa_work
      1.04 ±  3%      +0.3        1.29 ±  2%  perf-profile.children.cycles-pp.task_work_run
      1.05 ±  3%      +0.3        1.32 ±  2%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.23 ±  9%      +0.3        0.53 ± 18%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.45 ±  7%      +0.4        0.90 ± 15%  perf-profile.children.cycles-pp.irq_exit
      1.23 ±  9%      +0.9        2.14 ± 25%  perf-profile.children.cycles-pp.tick_sched_handle
      1.36 ±  9%      +1.0        2.37 ± 25%  perf-profile.children.cycles-pp.tick_sched_timer
      2.84 ±  4%      +1.1        3.95 ± 15%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.62 ±  3%      +1.5        6.09 ± 12%  perf-profile.children.cycles-pp.hrtimer_interrupt
      6.52 ±  4%      +1.6        8.12 ± 12%  perf-profile.children.cycles-pp.apic_timer_interrupt
      5.98 ±  4%      +1.7        7.72 ± 12%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     36.69           +17.2       53.84 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     33.69           +17.9       51.58 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     30.42           -11.4       19.00 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64
     20.75 ±  2%      -5.8       14.98 ±  5%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      5.97            -1.2        4.82        perf-profile.self.cycles-pp.__fget_light
      4.16 ±  2%      -0.9        3.31 ±  2%  perf-profile.self.cycles-pp.shmem_file_llseek
      3.00            -0.7        2.28 ±  3%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      2.81            -0.7        2.13 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      2.09 ±  2%      -0.3        1.75 ±  3%  perf-profile.self.cycles-pp.ksys_lseek
      0.52 ± 23%      -0.1        0.38 ± 16%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.40 ±  5%      -0.1        0.29 ± 15%  perf-profile.self.cycles-pp.interrupt_entry
      0.52 ±  2%      -0.1        0.40 ±  3%  perf-profile.self.cycles-pp.generic_file_llseek_size
      0.42            -0.1        0.33 ±  3%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.50 ±  2%      -0.1        0.41 ±  4%  perf-profile.self.cycles-pp.__fdget_pos
      0.22 ± 12%      -0.1        0.17 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.59            -0.0        0.56        perf-profile.self.cycles-pp.__x64_sys_lseek
      0.09 ±  4%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.rb_next
      0.09 ± 14%      +0.0        0.13 ± 14%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.06 ±  6%      +0.0        0.11 ± 23%  perf-profile.self.cycles-pp.raise_softirq
      0.05 ±  8%      +0.1        0.11 ± 28%  perf-profile.self.cycles-pp.timerqueue_add
      0.07 ± 12%      +0.1        0.12 ± 24%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.06 ± 28%  perf-profile.self.cycles-pp.rcu_irq_exit
      0.01 ±173%      +0.1        0.07 ± 19%  perf-profile.self.cycles-pp.task_tick_fair
      0.06 ±  7%      +0.1        0.12 ± 15%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.07 ± 12%      +0.1        0.14 ± 23%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.rcu_segcblist_ready_cbs
      0.00            +0.1        0.08 ± 26%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.idle_cpu
      0.01 ±173%      +0.1        0.10 ± 30%  perf-profile.self.cycles-pp.prepare_exit_to_usermode
      0.01 ±173%      +0.1        0.10 ± 17%  perf-profile.self.cycles-pp.ksoftirqd_running
      0.10 ±  4%      +0.1        0.20 ± 21%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.04 ± 57%      +0.1        0.14 ± 20%  perf-profile.self.cycles-pp.run_timer_softirq
      0.00            +0.1        0.13 ± 19%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.91 ±  3%      +0.2        1.14 ±  2%  perf-profile.self.cycles-pp.change_p4d_range
      1.23 ±  5%      +0.4        1.60 ± 13%  perf-profile.self.cycles-pp.hrtimer_interrupt
     12.36 ±  2%     +19.9       32.29 ±  3%  perf-profile.self.cycles-pp.do_syscall_64
      1234 ± 23%     -49.9%     617.75 ± 30%  interrupts.CPU1.RES:Rescheduling_interrupts
      1494 ± 49%     -67.1%     492.25 ± 23%  interrupts.CPU10.RES:Rescheduling_interrupts
      5500 ± 33%     -22.6%       4256 ± 37%  interrupts.CPU108.NMI:Non-maskable_interrupts
      5500 ± 33%     -22.6%       4256 ± 37%  interrupts.CPU108.PMI:Performance_monitoring_interrupts
     18.75 ± 39%    +365.3%      87.25 ± 93%  interrupts.CPU113.RES:Rescheduling_interrupts
     40.75 ± 37%    +501.8%     245.25 ± 55%  interrupts.CPU114.RES:Rescheduling_interrupts
     44.25 ± 35%    +226.0%     144.25 ± 35%  interrupts.CPU118.RES:Rescheduling_interrupts
      4559 ± 34%     -26.9%       3334        interrupts.CPU121.NMI:Non-maskable_interrupts
      4559 ± 34%     -26.9%       3334        interrupts.CPU121.PMI:Performance_monitoring_interrupts
      5472 ± 33%     -37.8%       3404        interrupts.CPU122.NMI:Non-maskable_interrupts
      5472 ± 33%     -37.8%       3404        interrupts.CPU122.PMI:Performance_monitoring_interrupts
      6396 ± 24%     -32.5%       4318 ± 35%  interrupts.CPU126.NMI:Non-maskable_interrupts
      6396 ± 24%     -32.5%       4318 ± 35%  interrupts.CPU126.PMI:Performance_monitoring_interrupts
      7304           -51.7%       3531        interrupts.CPU127.NMI:Non-maskable_interrupts
      7304           -51.7%       3531        interrupts.CPU127.PMI:Performance_monitoring_interrupts
     10.00 ± 22%    +435.0%      53.50 ± 87%  interrupts.CPU127.RES:Rescheduling_interrupts
      5516 ± 32%     -21.7%       4317 ± 35%  interrupts.CPU128.NMI:Non-maskable_interrupts
      5516 ± 32%     -21.7%       4317 ± 35%  interrupts.CPU128.PMI:Performance_monitoring_interrupts
      5498 ± 33%     -21.8%       4297 ± 34%  interrupts.CPU132.NMI:Non-maskable_interrupts
      5498 ± 33%     -21.8%       4297 ± 34%  interrupts.CPU132.PMI:Performance_monitoring_interrupts
      5527 ± 33%     -22.0%       4313 ± 34%  interrupts.CPU137.NMI:Non-maskable_interrupts
      5527 ± 33%     -22.0%       4313 ± 34%  interrupts.CPU137.PMI:Performance_monitoring_interrupts
      5534 ± 33%     -22.6%       4281 ± 34%  interrupts.CPU138.NMI:Non-maskable_interrupts
      5534 ± 33%     -22.6%       4281 ± 34%  interrupts.CPU138.PMI:Performance_monitoring_interrupts
      1298 ± 41%     -60.1%     518.75 ± 14%  interrupts.CPU14.RES:Rescheduling_interrupts
      6374 ± 24%     -45.4%       3481        interrupts.CPU141.NMI:Non-maskable_interrupts
      6374 ± 24%     -45.4%       3481        interrupts.CPU141.PMI:Performance_monitoring_interrupts
      5454 ± 33%     -36.8%       3446        interrupts.CPU146.NMI:Non-maskable_interrupts
      5454 ± 33%     -36.8%       3446        interrupts.CPU146.PMI:Performance_monitoring_interrupts
    451.75 ± 23%     -54.6%     205.25 ± 43%  interrupts.CPU15.RES:Rescheduling_interrupts
      5486 ± 33%     -23.2%       4214 ± 34%  interrupts.CPU154.NMI:Non-maskable_interrupts
      5486 ± 33%     -23.2%       4214 ± 34%  interrupts.CPU154.PMI:Performance_monitoring_interrupts
     43.00 ± 14%    +113.4%      91.75 ± 14%  interrupts.CPU154.RES:Rescheduling_interrupts
      7360           -42.4%       4239 ± 33%  interrupts.CPU156.NMI:Non-maskable_interrupts
      7360           -42.4%       4239 ± 33%  interrupts.CPU156.PMI:Performance_monitoring_interrupts
      5509 ± 33%     -22.2%       4288 ± 31%  interrupts.CPU158.NMI:Non-maskable_interrupts
      5509 ± 33%     -22.2%       4288 ± 31%  interrupts.CPU158.PMI:Performance_monitoring_interrupts
      6490 ± 25%     -33.4%       4322 ± 33%  interrupts.CPU161.NMI:Non-maskable_interrupts
      6490 ± 25%     -33.4%       4322 ± 33%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
     69.25 ± 60%     -75.1%      17.25 ± 59%  interrupts.CPU169.RES:Rescheduling_interrupts
    775.00 ± 16%     -72.7%     211.75 ± 35%  interrupts.CPU17.RES:Rescheduling_interrupts
      6399 ± 25%     -46.3%       3435        interrupts.CPU170.NMI:Non-maskable_interrupts
      6399 ± 25%     -46.3%       3435        interrupts.CPU170.PMI:Performance_monitoring_interrupts
      5511 ± 33%     -21.2%       4343 ± 34%  interrupts.CPU175.NMI:Non-maskable_interrupts
      5511 ± 33%     -21.2%       4343 ± 34%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
      6487 ± 24%     -47.0%       3438        interrupts.CPU177.NMI:Non-maskable_interrupts
      6487 ± 24%     -47.0%       3438        interrupts.CPU177.PMI:Performance_monitoring_interrupts
      5527 ± 33%     -37.3%       3463        interrupts.CPU178.NMI:Non-maskable_interrupts
      5527 ± 33%     -37.3%       3463        interrupts.CPU178.PMI:Performance_monitoring_interrupts
      6414 ± 24%     -32.8%       4307 ± 33%  interrupts.CPU191.NMI:Non-maskable_interrupts
      6414 ± 24%     -32.8%       4307 ± 33%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
      5465 ± 33%     -22.1%       4259 ± 36%  interrupts.CPU192.NMI:Non-maskable_interrupts
      5465 ± 33%     -22.1%       4259 ± 36%  interrupts.CPU192.PMI:Performance_monitoring_interrupts
      6464 ± 24%     -46.0%       3488        interrupts.CPU193.NMI:Non-maskable_interrupts
      6464 ± 24%     -46.0%       3488        interrupts.CPU193.PMI:Performance_monitoring_interrupts
      5482 ± 33%     -21.9%       4284 ± 32%  interrupts.CPU205.NMI:Non-maskable_interrupts
      5482 ± 33%     -21.9%       4284 ± 32%  interrupts.CPU205.PMI:Performance_monitoring_interrupts
     24.25 ± 28%    +581.4%     165.25 ±119%  interrupts.CPU206.RES:Rescheduling_interrupts
      5503 ± 34%     -20.8%       4357 ± 34%  interrupts.CPU207.NMI:Non-maskable_interrupts
      5503 ± 34%     -20.8%       4357 ± 34%  interrupts.CPU207.PMI:Performance_monitoring_interrupts
      5512 ± 33%     -37.5%       3443        interrupts.CPU208.NMI:Non-maskable_interrupts
      5512 ± 33%     -37.5%       3443        interrupts.CPU208.PMI:Performance_monitoring_interrupts
      5542 ± 33%     -20.4%       4412 ± 34%  interrupts.CPU209.NMI:Non-maskable_interrupts
      5542 ± 33%     -20.4%       4412 ± 34%  interrupts.CPU209.PMI:Performance_monitoring_interrupts
      5558 ± 33%     -20.5%       4418 ± 33%  interrupts.CPU211.NMI:Non-maskable_interrupts
      5558 ± 33%     -20.5%       4418 ± 33%  interrupts.CPU211.PMI:Performance_monitoring_interrupts
      5485 ± 33%     -22.4%       4255 ± 33%  interrupts.CPU213.NMI:Non-maskable_interrupts
      5485 ± 33%     -22.4%       4255 ± 33%  interrupts.CPU213.PMI:Performance_monitoring_interrupts
      5491 ± 33%     -21.1%       4330 ± 35%  interrupts.CPU218.NMI:Non-maskable_interrupts
      5491 ± 33%     -21.1%       4330 ± 35%  interrupts.CPU218.PMI:Performance_monitoring_interrupts
      5548 ± 33%     -37.3%       3479 ±  2%  interrupts.CPU219.NMI:Non-maskable_interrupts
      5548 ± 33%     -37.3%       3479 ±  2%  interrupts.CPU219.PMI:Performance_monitoring_interrupts
      6396 ± 25%     -46.2%       3441        interrupts.CPU222.NMI:Non-maskable_interrupts
      6396 ± 25%     -46.2%       3441        interrupts.CPU222.PMI:Performance_monitoring_interrupts
      5477 ± 32%     -22.7%       4232 ± 36%  interrupts.CPU227.NMI:Non-maskable_interrupts
      5477 ± 32%     -22.7%       4232 ± 36%  interrupts.CPU227.PMI:Performance_monitoring_interrupts
    149.75 ± 98%     -90.0%      15.00 ± 16%  interrupts.CPU229.RES:Rescheduling_interrupts
     29.25 ± 21%   +1021.4%     328.00 ±131%  interrupts.CPU230.RES:Rescheduling_interrupts
      5536 ± 33%     -38.3%       3416        interrupts.CPU232.NMI:Non-maskable_interrupts
      5536 ± 33%     -38.3%       3416        interrupts.CPU232.PMI:Performance_monitoring_interrupts
     76.25 ± 67%     -85.9%      10.75 ± 36%  interrupts.CPU235.RES:Rescheduling_interrupts
      6409 ± 24%     -33.3%       4277 ± 36%  interrupts.CPU236.NMI:Non-maskable_interrupts
      6409 ± 24%     -33.3%       4277 ± 36%  interrupts.CPU236.PMI:Performance_monitoring_interrupts
    119.25 ± 63%     -80.5%      23.25 ± 28%  interrupts.CPU240.RES:Rescheduling_interrupts
      6353 ± 24%     -46.4%       3402        interrupts.CPU241.NMI:Non-maskable_interrupts
      6353 ± 24%     -46.4%       3402        interrupts.CPU241.PMI:Performance_monitoring_interrupts
      5491 ± 33%     -22.3%       4266 ± 33%  interrupts.CPU242.NMI:Non-maskable_interrupts
      5491 ± 33%     -22.3%       4266 ± 33%  interrupts.CPU242.PMI:Performance_monitoring_interrupts
      5496 ± 33%     -37.1%       3455        interrupts.CPU243.NMI:Non-maskable_interrupts
      5496 ± 33%     -37.1%       3455        interrupts.CPU243.PMI:Performance_monitoring_interrupts
      5500 ± 34%     -23.1%       4229 ± 35%  interrupts.CPU245.NMI:Non-maskable_interrupts
      5500 ± 34%     -23.1%       4229 ± 35%  interrupts.CPU245.PMI:Performance_monitoring_interrupts
      5481 ± 32%     -36.9%       3458        interrupts.CPU246.NMI:Non-maskable_interrupts
      5481 ± 32%     -36.9%       3458        interrupts.CPU246.PMI:Performance_monitoring_interrupts
    227.25 ± 43%     -55.6%     101.00 ± 16%  interrupts.CPU25.RES:Rescheduling_interrupts
      5447 ± 33%     -22.1%       4242 ± 33%  interrupts.CPU250.NMI:Non-maskable_interrupts
      5447 ± 33%     -22.1%       4242 ± 33%  interrupts.CPU250.PMI:Performance_monitoring_interrupts
      6428 ± 24%     -47.1%       3398        interrupts.CPU254.NMI:Non-maskable_interrupts
      6428 ± 24%     -47.1%       3398        interrupts.CPU254.PMI:Performance_monitoring_interrupts
     14.25 ± 48%    +715.8%     116.25 ± 72%  interrupts.CPU257.RES:Rescheduling_interrupts
      5523 ± 33%     -37.8%       3436        interrupts.CPU259.NMI:Non-maskable_interrupts
      5523 ± 33%     -37.8%       3436        interrupts.CPU259.PMI:Performance_monitoring_interrupts
      1072 ± 69%     -82.8%     184.25 ± 62%  interrupts.CPU26.RES:Rescheduling_interrupts
      5595 ± 33%     -19.4%       4510 ± 33%  interrupts.CPU267.NMI:Non-maskable_interrupts
      5595 ± 33%     -19.4%       4510 ± 33%  interrupts.CPU267.PMI:Performance_monitoring_interrupts
    163.00 ± 29%     -69.5%      49.75 ± 48%  interrupts.CPU270.RES:Rescheduling_interrupts
      5610 ± 33%     -35.5%       3616        interrupts.CPU272.NMI:Non-maskable_interrupts
      5610 ± 33%     -35.5%       3616        interrupts.CPU272.PMI:Performance_monitoring_interrupts
      5582 ± 33%     -37.8%       3472        interrupts.CPU273.NMI:Non-maskable_interrupts
      5582 ± 33%     -37.8%       3472        interrupts.CPU273.PMI:Performance_monitoring_interrupts
      5606 ± 33%     -20.5%       4454 ± 35%  interrupts.CPU274.NMI:Non-maskable_interrupts
      5606 ± 33%     -20.5%       4454 ± 35%  interrupts.CPU274.PMI:Performance_monitoring_interrupts
      5581 ± 33%     -19.6%       4487 ± 34%  interrupts.CPU275.NMI:Non-maskable_interrupts
      5581 ± 33%     -19.6%       4487 ± 34%  interrupts.CPU275.PMI:Performance_monitoring_interrupts
     14.50 ± 15%    +456.9%      80.75 ±100%  interrupts.CPU277.RES:Rescheduling_interrupts
      6529 ± 24%     -31.0%       4506 ± 32%  interrupts.CPU279.NMI:Non-maskable_interrupts
      6529 ± 24%     -31.0%       4506 ± 32%  interrupts.CPU279.PMI:Performance_monitoring_interrupts
      5582 ± 33%     -36.0%       3573        interrupts.CPU284.NMI:Non-maskable_interrupts
      5582 ± 33%     -36.0%       3573        interrupts.CPU284.PMI:Performance_monitoring_interrupts
     16.25 ± 23%   +1187.7%     209.25 ± 89%  interrupts.CPU287.TLB:TLB_shootdowns
      5595 ± 33%     -20.1%       4470 ± 36%  interrupts.CPU43.NMI:Non-maskable_interrupts
      5595 ± 33%     -20.1%       4470 ± 36%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      6542 ± 24%     -44.5%       3627        interrupts.CPU50.NMI:Non-maskable_interrupts
      6542 ± 24%     -44.5%       3627        interrupts.CPU50.PMI:Performance_monitoring_interrupts
      5611 ± 33%     -35.3%       3630 ±  2%  interrupts.CPU51.NMI:Non-maskable_interrupts
      5611 ± 33%     -35.3%       3630 ±  2%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
    218.50 ± 29%    +162.4%     573.25 ± 55%  interrupts.CPU53.RES:Rescheduling_interrupts
      1495 ± 50%     -65.8%     511.75 ± 24%  interrupts.CPU6.RES:Rescheduling_interrupts
      5614 ± 33%     -20.6%       4457 ± 34%  interrupts.CPU62.NMI:Non-maskable_interrupts
      5614 ± 33%     -20.6%       4457 ± 34%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
    554.50 ± 33%     -58.0%     233.00 ± 20%  interrupts.CPU62.RES:Rescheduling_interrupts
      5605 ± 33%     -19.3%       4521 ± 33%  interrupts.CPU63.NMI:Non-maskable_interrupts
      5605 ± 33%     -19.3%       4521 ± 33%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
      5516 ± 33%     -36.7%       3491        interrupts.CPU72.NMI:Non-maskable_interrupts
      5516 ± 33%     -36.7%       3491        interrupts.CPU72.PMI:Performance_monitoring_interrupts
      7386           -53.1%       3467        interrupts.CPU73.NMI:Non-maskable_interrupts
      7386           -53.1%       3467        interrupts.CPU73.PMI:Performance_monitoring_interrupts
      5475 ± 33%     -21.2%       4314 ± 34%  interrupts.CPU75.NMI:Non-maskable_interrupts
      5475 ± 33%     -21.2%       4314 ± 34%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
     15.00 ± 47%    +295.0%      59.25 ± 67%  interrupts.CPU75.RES:Rescheduling_interrupts
      5502 ± 33%     -21.6%       4311 ± 35%  interrupts.CPU76.NMI:Non-maskable_interrupts
      5502 ± 33%     -21.6%       4311 ± 35%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      5536 ± 34%     -21.6%       4338 ± 33%  interrupts.CPU77.NMI:Non-maskable_interrupts
      5536 ± 34%     -21.6%       4338 ± 33%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      1095 ± 66%     -56.9%     471.75 ± 35%  interrupts.CPU8.RES:Rescheduling_interrupts
      5487 ± 33%     -38.2%       3388        interrupts.CPU81.NMI:Non-maskable_interrupts
      5487 ± 33%     -38.2%       3388        interrupts.CPU81.PMI:Performance_monitoring_interrupts
      5614 ± 33%     -34.0%       3705        interrupts.CPU9.NMI:Non-maskable_interrupts
      5614 ± 33%     -34.0%       3705        interrupts.CPU9.PMI:Performance_monitoring_interrupts
      1409 ± 93%     -79.8%     284.75 ± 18%  interrupts.CPU9.RES:Rescheduling_interrupts
    426.75 ± 48%     -91.3%      37.00 ± 46%  interrupts.CPU92.RES:Rescheduling_interrupts
      6436 ± 25%     -31.0%       4442 ± 35%  interrupts.CPU94.NMI:Non-maskable_interrupts
      6436 ± 25%     -31.0%       4442 ± 35%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      5477 ± 33%     -36.4%       3483        interrupts.CPU96.NMI:Non-maskable_interrupts
      5477 ± 33%     -36.4%       3483        interrupts.CPU96.PMI:Performance_monitoring_interrupts
     46.75 ± 74%     -77.0%      10.75 ± 45%  interrupts.CPU97.RES:Rescheduling_interrupts
    136.25 ± 67%     -77.8%      30.25 ± 25%  interrupts.CPU98.RES:Rescheduling_interrupts
     64096 ±  4%     -20.2%      51160 ±  4%  interrupts.RES:Rescheduling_interrupts
      1147 ±  3%     +48.3%       1701 ± 22%  interrupts.TLB:TLB_shootdowns
    132024           +14.3%     150947 ±  2%  softirqs.CPU0.RCU
      9306 ± 21%     -19.9%       7456 ±  5%  softirqs.CPU0.SCHED
    134070 ±  3%     +12.5%     150780 ±  3%  softirqs.CPU1.RCU
    133661           +14.8%     153386 ±  5%  softirqs.CPU10.RCU
    132307           +15.0%     152108 ±  2%  softirqs.CPU100.RCU
    132603           +13.3%     150262 ±  2%  softirqs.CPU101.RCU
    131269           +13.5%     149009 ±  2%  softirqs.CPU102.RCU
    130873           +14.4%     149753 ±  2%  softirqs.CPU103.RCU
    133305           +13.4%     151223 ±  2%  softirqs.CPU104.RCU
    134079           +13.7%     152466 ±  3%  softirqs.CPU105.RCU
    133189           +13.5%     151170 ±  2%  softirqs.CPU106.RCU
    133518           +14.0%     152197 ±  2%  softirqs.CPU107.RCU
    131420           +14.1%     149940 ±  2%  softirqs.CPU108.RCU
    131999           +13.9%     150367 ±  2%  softirqs.CPU109.RCU
    133435           +12.7%     150399 ±  2%  softirqs.CPU11.RCU
    135866 ±  6%     +12.6%     152943 ±  4%  softirqs.CPU110.RCU
    131917           +13.0%     149068 ±  2%  softirqs.CPU111.RCU
    130445           +14.5%     149396 ±  2%  softirqs.CPU112.RCU
    130778           +14.4%     149649 ±  2%  softirqs.CPU113.RCU
    131832           +13.4%     149477 ±  2%  softirqs.CPU114.RCU
    131101           +16.6%     152883 ±  5%  softirqs.CPU115.RCU
    130414           +14.0%     148720 ±  2%  softirqs.CPU116.RCU
    133604           +13.3%     151434 ±  2%  softirqs.CPU117.RCU
    133959           +12.8%     151143        softirqs.CPU118.RCU
    133158           +13.4%     150943 ±  2%  softirqs.CPU119.RCU
    132153           +13.7%     150297 ±  2%  softirqs.CPU12.RCU
    131245           +13.9%     149434 ±  2%  softirqs.CPU120.RCU
    131301           +14.2%     150000 ±  2%  softirqs.CPU121.RCU
    131357           +13.4%     149008 ±  2%  softirqs.CPU122.RCU
    130888           +14.7%     150097 ±  2%  softirqs.CPU123.RCU
    133574           +13.0%     150877 ±  2%  softirqs.CPU124.RCU
    132738           +13.5%     150675 ±  2%  softirqs.CPU125.RCU
    129995           +14.6%     148930 ±  2%  softirqs.CPU126.RCU
    132559           +13.8%     150859 ±  2%  softirqs.CPU127.RCU
    132611           +13.2%     150099 ±  2%  softirqs.CPU128.RCU
    133130           +13.5%     151138        softirqs.CPU129.RCU
    132941           +12.7%     149857        softirqs.CPU13.RCU
    132399           +14.5%     151641 ±  2%  softirqs.CPU130.RCU
    133664           +13.1%     151195        softirqs.CPU131.RCU
    131354           +13.1%     148547 ±  2%  softirqs.CPU132.RCU
    132153           +11.5%     147333        softirqs.CPU133.RCU
    141243 ±  4%      -8.5%     129259 ±  7%  softirqs.CPU133.TIMER
    130923           +13.8%     148929 ±  2%  softirqs.CPU134.RCU
    130658           +14.9%     150185 ±  2%  softirqs.CPU135.RCU
    131314           +14.6%     150465 ±  2%  softirqs.CPU136.RCU
    131704           +13.5%     149451 ±  2%  softirqs.CPU137.RCU
    129988           +14.6%     148913 ±  2%  softirqs.CPU138.RCU
    129204           +15.2%     148810 ±  2%  softirqs.CPU139.RCU
    132909           +14.0%     151509 ±  3%  softirqs.CPU14.RCU
    133858           +14.4%     153150 ±  3%  softirqs.CPU140.RCU
    133023           +13.6%     151136 ±  2%  softirqs.CPU141.RCU
    131085           +13.8%     149216 ±  2%  softirqs.CPU142.RCU
    132866           +13.8%     151258 ±  2%  softirqs.CPU143.RCU
    134532           +12.5%     151294 ±  2%  softirqs.CPU144.RCU
    132923           +12.9%     150024 ±  3%  softirqs.CPU145.RCU
    132467           +14.3%     151369        softirqs.CPU146.RCU
    135143 ±  2%     +11.8%     151071 ±  2%  softirqs.CPU147.RCU
    131448           +13.5%     149199 ±  2%  softirqs.CPU148.RCU
    131553           +15.4%     151852 ±  2%  softirqs.CPU149.RCU
    132807           +13.8%     151080 ±  2%  softirqs.CPU15.RCU
    131249           +13.8%     149339 ±  2%  softirqs.CPU150.RCU
    130798           +14.6%     149901 ±  2%  softirqs.CPU151.RCU
    133363           +13.4%     151240 ±  2%  softirqs.CPU152.RCU
    133910           +12.9%     151190 ±  2%  softirqs.CPU153.RCU
    133644           +14.0%     152403 ±  3%  softirqs.CPU154.RCU
    134555 ±  2%     +13.1%     152144 ±  2%  softirqs.CPU155.RCU
    131692           +13.6%     149559        softirqs.CPU156.RCU
    131710           +13.3%     149247        softirqs.CPU157.RCU
    131528           +12.9%     148436 ±  2%  softirqs.CPU158.RCU
    131543           +13.9%     149886 ±  2%  softirqs.CPU159.RCU
    134455           +14.0%     153325 ±  2%  softirqs.CPU16.RCU
    131850           +13.8%     150009 ±  2%  softirqs.CPU160.RCU
    131062           +14.2%     149698 ±  2%  softirqs.CPU161.RCU
    132570           +13.8%     150822 ±  3%  softirqs.CPU162.RCU
    131238           +13.5%     148951 ±  2%  softirqs.CPU163.RCU
    130720           +15.4%     150870 ±  2%  softirqs.CPU164.RCU
    133626           +13.0%     151048 ±  2%  softirqs.CPU165.RCU
    132741           +13.5%     150710 ±  2%  softirqs.CPU166.RCU
    133569           +13.0%     150960 ±  2%  softirqs.CPU167.RCU
    133292           +12.9%     150530 ±  2%  softirqs.CPU168.RCU
    132312           +14.2%     151081 ±  2%  softirqs.CPU169.RCU
    132191           +14.0%     150726 ±  2%  softirqs.CPU17.RCU
    133180           +13.1%     150618 ±  2%  softirqs.CPU170.RCU
    132649           +13.7%     150799 ±  2%  softirqs.CPU171.RCU
    131676           +13.7%     149778 ±  2%  softirqs.CPU172.RCU
    131322           +13.9%     149532 ±  2%  softirqs.CPU173.RCU
    131163           +14.2%     149810 ±  2%  softirqs.CPU174.RCU
    130944           +14.2%     149603 ±  2%  softirqs.CPU175.RCU
    129591           +15.0%     149026 ±  2%  softirqs.CPU176.RCU
    130486           +13.8%     148500 ±  2%  softirqs.CPU177.RCU
    130598           +18.8%     155156 ±  6%  softirqs.CPU178.RCU
    131102           +14.3%     149901 ±  2%  softirqs.CPU179.RCU
    131114           +14.3%     149803 ±  3%  softirqs.CPU18.RCU
    130612           +14.0%     148920 ±  2%  softirqs.CPU180.RCU
    130515           +13.6%     148273 ±  2%  softirqs.CPU181.RCU
    132363           +13.9%     150790 ±  2%  softirqs.CPU182.RCU
    132913           +13.5%     150826 ±  2%  softirqs.CPU183.RCU
    130724           +14.4%     149546 ±  2%  softirqs.CPU184.RCU
    131396           +14.0%     149794 ±  2%  softirqs.CPU185.RCU
    129872           +14.3%     148465 ±  2%  softirqs.CPU186.RCU
    130017           +14.0%     148230 ±  2%  softirqs.CPU187.RCU
    132362           +14.3%     151245 ±  2%  softirqs.CPU188.RCU
    132501           +12.3%     148740 ±  4%  softirqs.CPU189.RCU
    132251           +13.9%     150597 ±  2%  softirqs.CPU19.RCU
    132333           +13.7%     150397 ±  2%  softirqs.CPU190.RCU
    132550           +13.2%     150073 ±  2%  softirqs.CPU191.RCU
    132411           +13.2%     149860 ±  2%  softirqs.CPU192.RCU
    131767           +14.6%     151069        softirqs.CPU193.RCU
    131753           +13.7%     149865 ±  2%  softirqs.CPU194.RCU
    132231           +13.8%     150493 ±  2%  softirqs.CPU195.RCU
    131283           +13.4%     148824 ±  2%  softirqs.CPU196.RCU
    130720           +13.8%     148785 ±  2%  softirqs.CPU197.RCU
    130623           +14.4%     149390 ±  2%  softirqs.CPU198.RCU
    130459           +14.4%     149267 ±  2%  softirqs.CPU199.RCU
    134212           +12.7%     151215 ±  3%  softirqs.CPU2.RCU
    132200           +13.6%     150117 ±  2%  softirqs.CPU20.RCU
    130250           +14.8%     149565 ±  2%  softirqs.CPU200.RCU
    128798 ±  2%     +15.2%     148334        softirqs.CPU201.RCU
    129002           +14.8%     148109 ±  2%  softirqs.CPU202.RCU
    131094           +13.4%     148596 ±  2%  softirqs.CPU203.RCU
    132435           +13.5%     150322 ±  2%  softirqs.CPU204.RCU
    132906           +12.5%     149513        softirqs.CPU205.RCU
    130337           +13.7%     148175 ±  2%  softirqs.CPU206.RCU
    130426           +13.5%     148063 ±  2%  softirqs.CPU207.RCU
    132917           +13.1%     150361        softirqs.CPU208.RCU
    131949           +14.1%     150618 ±  2%  softirqs.CPU209.RCU
    133425           +11.5%     148749        softirqs.CPU21.RCU
    132555           +14.0%     151127 ±  2%  softirqs.CPU210.RCU
    131608           +13.9%     149918        softirqs.CPU211.RCU
    131712 ±  2%     +12.5%     148170        softirqs.CPU212.RCU
    140958 ±  3%     -12.7%     122999 ±  9%  softirqs.CPU212.TIMER
    130072           +14.5%     148916 ±  2%  softirqs.CPU213.RCU
    131160           +13.4%     148799 ±  2%  softirqs.CPU214.RCU
    130097           +14.3%     148700 ±  2%  softirqs.CPU215.RCU
    132972           +13.4%     150844 ±  2%  softirqs.CPU216.RCU
    131734           +13.8%     149933 ±  2%  softirqs.CPU217.RCU
    133271           +12.7%     150173 ±  2%  softirqs.CPU218.RCU
    133210           +12.9%     150446 ±  2%  softirqs.CPU219.RCU
    132730           +13.8%     151064 ±  3%  softirqs.CPU22.RCU
    130710           +14.6%     149773 ±  2%  softirqs.CPU220.RCU
    130833           +14.5%     149782 ±  2%  softirqs.CPU221.RCU
    131452           +12.3%     147634 ±  2%  softirqs.CPU222.RCU
    132072           +12.6%     148695 ±  2%  softirqs.CPU223.RCU
    129537           +14.4%     148227 ±  2%  softirqs.CPU224.RCU
    130864           +13.1%     148068        softirqs.CPU225.RCU
    131474           +14.2%     150083 ±  2%  softirqs.CPU226.RCU
    131937           +13.0%     149088 ±  2%  softirqs.CPU227.RCU
    132212           +15.3%     152414 ±  4%  softirqs.CPU228.RCU
    133658           +12.7%     150657 ±  2%  softirqs.CPU229.RCU
    132266           +13.3%     149869 ±  2%  softirqs.CPU23.RCU
    131339           +14.6%     150485 ±  2%  softirqs.CPU230.RCU
    134228           +12.5%     151031 ±  3%  softirqs.CPU231.RCU
    131295           +12.8%     148120 ±  3%  softirqs.CPU232.RCU
    131788           +13.1%     149003 ±  2%  softirqs.CPU233.RCU
    130486           +14.5%     149383 ±  2%  softirqs.CPU234.RCU
    131931           +13.3%     149470 ±  2%  softirqs.CPU235.RCU
    133346           +13.9%     151861        softirqs.CPU236.RCU
    133503 ±  2%     +12.4%     150084 ±  2%  softirqs.CPU237.RCU
    132637           +14.0%     151182 ±  2%  softirqs.CPU238.RCU
    132414           +13.5%     150287 ±  2%  softirqs.CPU239.RCU
    132078           +14.0%     150592 ±  2%  softirqs.CPU24.RCU
    132587           +13.8%     150929 ±  2%  softirqs.CPU240.RCU
    133633           +12.4%     150261 ±  2%  softirqs.CPU241.RCU
    132100           +12.9%     149140 ±  2%  softirqs.CPU242.RCU
    130211           +13.6%     147971 ±  2%  softirqs.CPU243.RCU
    131395           +13.2%     148675 ±  2%  softirqs.CPU244.RCU
    130520           +13.7%     148422 ±  2%  softirqs.CPU245.RCU
    130539           +13.2%     147709 ±  2%  softirqs.CPU246.RCU
    129679           +14.0%     147792 ±  2%  softirqs.CPU247.RCU
    132242           +14.0%     150693 ±  2%  softirqs.CPU248.RCU
    129827           +14.0%     147949 ±  2%  softirqs.CPU249.RCU
    132214           +14.0%     150747 ±  2%  softirqs.CPU25.RCU
    132146           +13.7%     150270 ±  2%  softirqs.CPU250.RCU
    132573           +13.2%     150111 ±  2%  softirqs.CPU251.RCU
    130670           +13.5%     148323 ±  2%  softirqs.CPU252.RCU
    131169           +13.6%     149045 ±  2%  softirqs.CPU253.RCU
    131213           +13.7%     149147 ±  2%  softirqs.CPU254.RCU
    131876           +12.9%     148887 ±  2%  softirqs.CPU255.RCU
    145390 ±  6%     -16.0%     122070 ± 10%  softirqs.CPU255.TIMER
    130441           +14.0%     148724 ±  2%  softirqs.CPU256.RCU
    130541           +14.3%     149242 ±  2%  softirqs.CPU257.RCU
    130412           +13.5%     147957 ±  2%  softirqs.CPU258.RCU
    130939           +13.8%     149027 ±  2%  softirqs.CPU259.RCU
    132247           +10.1%     145539 ±  4%  softirqs.CPU26.RCU
    132713           +14.1%     151424 ±  3%  softirqs.CPU260.RCU
    130372           +15.2%     150132        softirqs.CPU261.RCU
    139063 ±  5%     -12.5%     121745 ±  6%  softirqs.CPU261.TIMER
    132238           +13.7%     150338 ±  3%  softirqs.CPU262.RCU
    132405           +13.2%     149908 ±  2%  softirqs.CPU263.RCU
    131146           +15.1%     151008        softirqs.CPU264.RCU
    129868           +14.4%     148607 ±  2%  softirqs.CPU265.RCU
    130238           +13.7%     148059 ±  2%  softirqs.CPU266.RCU
    131558           +13.4%     149125        softirqs.CPU267.RCU
    129436           +14.0%     147543 ±  2%  softirqs.CPU268.RCU
    129664           +13.3%     146860 ±  2%  softirqs.CPU269.RCU
    132316           +14.3%     151225        softirqs.CPU27.RCU
    129376           +14.4%     147985 ±  2%  softirqs.CPU270.RCU
    129746           +14.0%     147954 ±  2%  softirqs.CPU271.RCU
    129390           +14.0%     147473 ±  3%  softirqs.CPU272.RCU
    129961           +13.0%     146904        softirqs.CPU273.RCU
    129042           +14.3%     147517 ±  3%  softirqs.CPU274.RCU
    129282           +13.7%     147048        softirqs.CPU275.RCU
    121447 ±  5%     -11.0%     108069 ±  6%  softirqs.CPU275.TIMER
    131507           +13.1%     148787        softirqs.CPU276.RCU
    130103           +12.4%     146298        softirqs.CPU277.RCU
    131346           +12.9%     148242 ±  2%  softirqs.CPU278.RCU
    133451 ±  4%     +11.0%     148191 ±  2%  softirqs.CPU279.RCU
    132401           +13.4%     150129        softirqs.CPU28.RCU
    129114           +14.4%     147756 ±  3%  softirqs.CPU280.RCU
    129304           +14.4%     147976 ±  2%  softirqs.CPU281.RCU
    128958           +15.2%     148558 ±  2%  softirqs.CPU282.RCU
    129597           +14.2%     147985 ±  2%  softirqs.CPU283.RCU
    130854           +13.4%     148398 ±  2%  softirqs.CPU284.RCU
    130129           +13.6%     147888 ±  2%  softirqs.CPU285.RCU
    131388           +13.4%     148994 ±  2%  softirqs.CPU286.RCU
    132027           +14.0%     150525 ±  2%  softirqs.CPU29.RCU
    132235           +14.2%     150987 ±  2%  softirqs.CPU3.RCU
    131972           +13.7%     150055 ±  2%  softirqs.CPU30.RCU
    131795           +14.1%     150411 ±  2%  softirqs.CPU31.RCU
    131619           +13.9%     149920 ±  2%  softirqs.CPU32.RCU
    131760           +14.0%     150270 ±  2%  softirqs.CPU33.RCU
    131864           +13.3%     149344 ±  2%  softirqs.CPU34.RCU
    131591           +12.9%     148559 ±  2%  softirqs.CPU35.RCU
    131895           +13.5%     149693 ±  2%  softirqs.CPU36.RCU
    131357           +14.1%     149850 ±  2%  softirqs.CPU38.RCU
    132021           +13.8%     150194 ±  3%  softirqs.CPU39.RCU
    133632           +13.3%     151405 ±  2%  softirqs.CPU4.RCU
    131366           +14.6%     150518 ±  2%  softirqs.CPU40.RCU
    131452           +13.7%     149430 ±  2%  softirqs.CPU41.RCU
    131478           +13.8%     149572 ±  2%  softirqs.CPU42.RCU
    135354 ±  4%     +10.6%     149742 ±  2%  softirqs.CPU43.RCU
    131654           +14.0%     150146 ±  2%  softirqs.CPU44.RCU
    132262           +12.2%     148338 ±  3%  softirqs.CPU46.RCU
    136692 ±  5%     +13.4%     154967 ±  3%  softirqs.CPU47.RCU
    136604 ±  4%     +10.5%     150893 ±  2%  softirqs.CPU48.RCU
    132229           +13.6%     150147 ±  2%  softirqs.CPU49.RCU
    132467           +14.5%     151716 ±  2%  softirqs.CPU5.RCU
    129686 ±  3%     +15.3%     149577 ±  2%  softirqs.CPU50.RCU
    131517           +14.0%     149876 ±  2%  softirqs.CPU51.RCU
    132539           +16.2%     153959 ±  7%  softirqs.CPU52.RCU
    131675           +14.9%     151237 ±  2%  softirqs.CPU53.RCU
    134982 ±  3%     +10.8%     149589 ±  2%  softirqs.CPU54.RCU
    131038           +13.9%     149199 ±  2%  softirqs.CPU55.RCU
    133004           +13.6%     151081        softirqs.CPU56.RCU
    133103           +12.3%     149447 ±  2%  softirqs.CPU57.RCU
    135835 ±  3%     +11.7%     151672        softirqs.CPU58.RCU
    132457           +12.7%     149299        softirqs.CPU59.RCU
    133946           +11.9%     149888 ±  2%  softirqs.CPU6.RCU
    131223           +15.6%     151658 ±  6%  softirqs.CPU60.RCU
    131146           +13.0%     148142        softirqs.CPU61.RCU
    131056           +13.8%     149166 ±  2%  softirqs.CPU62.RCU
    129886           +19.0%     154563 ±  5%  softirqs.CPU63.RCU
    131105           +14.5%     150073 ±  2%  softirqs.CPU64.RCU
    130771           +14.6%     149818 ±  3%  softirqs.CPU65.RCU
    130747           +14.9%     150212 ±  2%  softirqs.CPU66.RCU
    131216           +13.8%     149365 ±  2%  softirqs.CPU67.RCU
    131888           +14.0%     150393 ±  2%  softirqs.CPU68.RCU
    132903           +13.9%     151410 ±  2%  softirqs.CPU69.RCU
    132795           +13.3%     150435 ±  2%  softirqs.CPU7.RCU
    132412           +12.5%     148953 ±  2%  softirqs.CPU70.RCU
    135771 ±  3%     +13.8%     154540 ±  3%  softirqs.CPU71.RCU
    131184           +14.1%     149660 ±  2%  softirqs.CPU73.RCU
    133242           +13.8%     151687        softirqs.CPU74.RCU
    133801           +14.1%     152658 ±  2%  softirqs.CPU75.RCU
    131638           +13.5%     149365 ±  2%  softirqs.CPU76.RCU
    132076           +13.1%     149420        softirqs.CPU77.RCU
    133424           +11.6%     148961        softirqs.CPU78.RCU
    132259           +13.6%     150242 ±  2%  softirqs.CPU79.RCU
    132414           +13.7%     150602 ±  2%  softirqs.CPU8.RCU
    132706           +13.2%     150235 ±  2%  softirqs.CPU80.RCU
    132689           +13.1%     150073 ±  2%  softirqs.CPU81.RCU
    132558           +13.4%     150348 ±  2%  softirqs.CPU82.RCU
    132705           +14.0%     151310 ±  2%  softirqs.CPU83.RCU
    133075           +14.2%     151919 ±  2%  softirqs.CPU84.RCU
    135186           +12.5%     152030 ±  2%  softirqs.CPU85.RCU
    132708           +14.3%     151710 ±  2%  softirqs.CPU86.RCU
    133908           +12.9%     151248 ±  2%  softirqs.CPU87.RCU
    132576           +14.5%     151806 ±  2%  softirqs.CPU88.RCU
    132458           +13.8%     150732 ±  3%  softirqs.CPU89.RCU
    132319           +13.4%     150009        softirqs.CPU9.RCU
    131982           +14.0%     150437 ±  2%  softirqs.CPU90.RCU
    132408           +13.9%     150777 ±  2%  softirqs.CPU91.RCU
    133616           +13.6%     151830 ±  2%  softirqs.CPU92.RCU
    133573           +13.6%     151719        softirqs.CPU93.RCU
    133491           +13.2%     151128 ±  2%  softirqs.CPU94.RCU
    134313           +13.1%     151971 ±  2%  softirqs.CPU95.RCU
    134118           +13.0%     151568 ±  2%  softirqs.CPU96.RCU
    133913           +13.3%     151766 ±  2%  softirqs.CPU97.RCU
    134677           +12.9%     152025 ±  2%  softirqs.CPU98.RCU
    134115           +13.5%     152168 ±  2%  softirqs.CPU99.RCU
  38007998           +13.6%   43194149 ±  2%  softirqs.RCU


                                                                                
                            will-it-scale.per_process_ops                       
                                                                                
  350000 +------------------------------------------------------------------+   
         |                                                                  |   
  340000 |.+.+. .+.   .+.+.     .+.+.      .+. .+.   .+.                    |   
  330000 |-+   +   +.+     +.+.+     +.+.+.   +   +.+   +.+                 |   
         |                                                                  |   
  320000 |-+                                                                |   
         |                                                                  |   
  310000 |-+                                                                |   
         |                             O O                                  |   
  300000 |-+                                O O             O       O O     |   
  290000 |-+     O               O O O          O O                     O   |   
         |   O         O   O                        O   O       O         O |   
  280000 |-+             O   O O                      O   O   O   O         |   
         | O   O   O O                                                      |   
  270000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                will-it-scale.workload                          
                                                                                
    1e+08 +-----------------------------------------------------------------+   
  9.8e+07 |.+.         .+.        +.                                        |   
          |   +. .+. .+   +.+.   +  +.+.   .+. .+.   .+.+.+                 |   
  9.6e+07 |-+   +   +         +.+       +.+   +   +.+                       |   
  9.4e+07 |-+                                                               |   
  9.2e+07 |-+                                                               |   
    9e+07 |-+                                                               |   
          |                               O                                 |   
  8.8e+07 |-+                           O                                   |   
  8.6e+07 |-+                               O O   O         O       O O     |   
  8.4e+07 |-+     O               O O O         O                       O   |   
  8.2e+07 |-+ O         O   O                       O                     O |   
          |               O   O O                     O O O   O O O         |   
    8e+07 |-O   O   O O                                                     |   
  7.8e+07 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc6-00015-g8c5045f89b181" of type "text/plain" (203665 bytes)

View attachment "job-script" of type "text/plain" (7397 bytes)

View attachment "job.yaml" of type "text/plain" (5052 bytes)

View attachment "reproduce" of type "text/plain" (311 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ