lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Tue, 16 Jun 2020 10:02:24 +0800
From:   Xing Zhengjun <zhengjun.xing@...ux.intel.com>
To:     kernel test robot <rong.a.chen@...el.com>,
        "Paul E. McKenney" <paulmck@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: Re: [LKP] [rcu] 276c410448: will-it-scale.per_thread_ops -12.3%
 regression

Hi Paul,

    Do you have time to take a look at this? Thanks.

On 6/15/2020 4:57 PM, kernel test robot wrote:
> Greeting,
> 
> FYI, we noticed a -12.3% regression of will-it-scale.per_thread_ops due to commit:
> 
> 
> commit: 276c410448dbca357a2bc3539acfe04862e5f172 ("rcu-tasks: Split ->trc_reader_need_end")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> 
> in testcase: will-it-scale
> on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
> with following parameters:
> 
> 	nr_task: 100%
> 	mode: thread
> 	test: page_fault3
> 	cpufreq_governor: performance
> 	ucode: 0x11
> 
> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
> test-url: https://github.com/antonblanchard/will-it-scale
> 
> 
> 
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <rong.a.chen@...el.com>
> 
> 
> Details are as below:
> -------------------------------------------------------------------------------------------------->
> 
> 
> To reproduce:
> 
>          git clone https://github.com/intel/lkp-tests.git
>          cd lkp-tests
>          bin/lkp install job.yaml  # job file is attached in this email
>          bin/lkp run     job.yaml
> 
> =========================================================================================
> compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
>    gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/page_fault3/will-it-scale/0x11
> 
> commit:
>    b0afa0f056 ("rcu-tasks: Provide boot parameter to delay IPIs until late in grace period")
>    276c410448 ("rcu-tasks: Split ->trc_reader_need_end")
> 
> b0afa0f056676ffe 276c410448dbca357a2bc3539ac
> ---------------- ---------------------------
>         fail:runs  %reproduction    fail:runs
>             |             |             |
>            2:4          -50%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
>             :4           28%           1:4     perf-profile.calltrace.cycles-pp.error_entry
>            0:4            0%           0:4     perf-profile.children.cycles-pp.error_exit
>            1:4            7%           2:4     perf-profile.children.cycles-pp.error_entry
>            0:4            4%           1:4     perf-profile.self.cycles-pp.error_entry
>           %stddev     %change         %stddev
>               \          |                \
>        1414           -12.3%       1241 ±  2%  will-it-scale.per_thread_ops
>      463.32            +1.7%     470.99        will-it-scale.time.elapsed_time
>      463.32            +1.7%     470.99        will-it-scale.time.elapsed_time.max
>      407566           -12.3%     357573 ±  2%  will-it-scale.workload
>       48.51            -1.5%      47.77        boot-time.boot
>   7.203e+10           +20.0%   8.64e+10 ±  2%  cpuidle.C1.time
>   2.162e+08 ±  2%     +27.7%  2.761e+08 ±  2%  cpuidle.C1.usage
>       60.50           +12.2       72.74 ±  2%  mpstat.cpu.all.idle%
>       39.17           -12.2       26.97 ±  6%  mpstat.cpu.all.sys%
>        2334 ± 12%     +18.8%       2772 ±  5%  slabinfo.khugepaged_mm_slot.active_objs
>        2334 ± 12%     +18.8%       2772 ±  5%  slabinfo.khugepaged_mm_slot.num_objs
>       60.25           +20.3%      72.50 ±  2%  vmstat.cpu.id
>       92.75 ±  3%     -21.6%      72.75 ±  5%  vmstat.procs.r
>      223709           +41.8%     317250 ±  3%  vmstat.system.cs
>      641687 ±  3%      +8.0%     693245 ±  2%  proc-vmstat.nr_inactive_anon
>      641688 ±  3%      +8.0%     693245 ±  2%  proc-vmstat.nr_zone_inactive_anon
>      166782            -3.7%     160632        proc-vmstat.numa_hint_faults
>      166782            -3.7%     160632        proc-vmstat.numa_hint_faults_local
>      984.25           -14.2%     844.75 ±  2%  proc-vmstat.numa_huge_pte_updates
>      710979           -11.2%     631134        proc-vmstat.numa_pte_updates
>   1.967e+08           -10.9%  1.752e+08        proc-vmstat.pgfault
>       58.18            +3.4%      60.17        perf-stat.i.MPKI
>   1.173e+09           +10.5%  1.296e+09        perf-stat.i.branch-instructions
>        6.74            -0.1        6.68        perf-stat.i.branch-miss-rate%
>    72495831           +10.7%   80219684        perf-stat.i.branch-misses
>       14.68            -0.6       14.06        perf-stat.i.cache-miss-rate%
>    43014696           +10.5%   47551690        perf-stat.i.cache-misses
>   2.936e+08           +15.6%  3.393e+08        perf-stat.i.cache-references
>      227441           +42.0%     323034 ±  3%  perf-stat.i.context-switches
>       37.22           -29.0%      26.44 ±  5%  perf-stat.i.cpi
>   1.828e+11           -22.3%  1.421e+11 ±  3%  perf-stat.i.cpu-cycles
>      513.71           +13.6%     583.63        perf-stat.i.cpu-migrations
>        4303           -27.8%       3107 ±  5%  perf-stat.i.cycles-between-cache-misses
>        1.78            -0.0        1.74        perf-stat.i.iTLB-load-miss-rate%
>    90757979            +9.1%   98982336        perf-stat.i.iTLB-load-misses
>   5.518e+09           +10.1%  6.076e+09        perf-stat.i.iTLB-loads
>   5.501e+09           +10.3%  6.065e+09        perf-stat.i.instructions
>       59.50            +2.4%      60.91        perf-stat.i.instructions-per-iTLB-miss
>        0.03           +48.9%       0.05 ±  3%  perf-stat.i.ipc
>        0.61           -22.2%       0.47 ±  3%  perf-stat.i.metric.GHz
>        0.85 ±  2%      +6.3%       0.90 ±  3%  perf-stat.i.metric.K/sec
>       23.19           +10.3%      25.58        perf-stat.i.metric.M/sec
>      398027           -13.3%     345187 ±  2%  perf-stat.i.minor-faults
>      398027           -13.3%     345187 ±  2%  perf-stat.i.page-faults
>       51.63            +4.7%      54.06        perf-stat.overall.MPKI
>       14.82            -0.6       14.26        perf-stat.overall.cache-miss-rate%
>       32.82           -28.9%      23.34 ±  4%  perf-stat.overall.cpi
>        4290           -29.4%       3029 ±  5%  perf-stat.overall.cycles-between-cache-misses
>        0.03           +40.9%       0.04 ±  4%  perf-stat.overall.ipc
>     6447068           +28.0%    8254132 ±  3%  perf-stat.overall.path-length
>   1.214e+09           +10.5%  1.341e+09        perf-stat.ps.branch-instructions
>    72454767           +10.7%   80188768        perf-stat.ps.branch-misses
>    43379426           +11.2%   48228470        perf-stat.ps.cache-misses
>   2.927e+08           +15.5%  3.382e+08 ±  2%  perf-stat.ps.cache-references
>      223136           +42.0%     316818 ±  3%  perf-stat.ps.context-switches
>   1.861e+11           -21.6%  1.459e+11 ±  3%  perf-stat.ps.cpu-cycles
>      498.80           +14.0%     568.54        perf-stat.ps.cpu-migrations
>    90216145            +8.9%   98279965        perf-stat.ps.iTLB-load-misses
>   5.688e+09           +10.2%  6.267e+09        perf-stat.ps.iTLB-loads
>    5.67e+09           +10.3%  6.256e+09        perf-stat.ps.instructions
>      404137           -13.0%     351604 ±  2%  perf-stat.ps.minor-faults
>      404137           -13.0%     351604 ±  2%  perf-stat.ps.page-faults
>   2.627e+12           +12.2%  2.949e+12        perf-stat.total.instructions
>       84210           -25.9%      62375 ±  5%  sched_debug.cfs_rq:/.exec_clock.avg
>       91870 ±  3%     -23.3%      70486 ±  5%  sched_debug.cfs_rq:/.exec_clock.max
>       79683           -23.1%      61285 ±  6%  sched_debug.cfs_rq:/.exec_clock.min
>        2299 ±  7%     +36.5%       3139 ± 12%  sched_debug.cfs_rq:/.load.avg
>      216559           +81.8%     393746 ± 14%  sched_debug.cfs_rq:/.load.max
>       16561 ±  3%     +64.0%      27165 ± 14%  sched_debug.cfs_rq:/.load.stddev
>        2.32 ± 11%     -19.6%       1.87 ± 14%  sched_debug.cfs_rq:/.load_avg.avg
>    18373664 ±  3%     -19.7%   14762699 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
>    18652687 ±  3%     -19.9%   14948853 ±  8%  sched_debug.cfs_rq:/.min_vruntime.max
>    17259651 ±  3%     -20.6%   13706324 ±  9%  sched_debug.cfs_rq:/.min_vruntime.min
>        0.09 ±  4%     -30.2%       0.06 ±  8%  sched_debug.cfs_rq:/.nr_running.avg
>        0.27 ±  2%     -14.3%       0.24 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
>       18.07 ±  2%     -31.0%      12.47 ±  5%  sched_debug.cfs_rq:/.nr_spread_over.avg
>        1.97 ± 34%     -49.2%       1.00 ± 57%  sched_debug.cfs_rq:/.nr_spread_over.min
>      258.66 ±  6%     -38.4%     159.22 ±  5%  sched_debug.cfs_rq:/.runnable_avg.avg
>      231.14           -13.6%     199.77 ±  7%  sched_debug.cfs_rq:/.runnable_avg.stddev
>      302282 ± 40%     -44.4%     168177 ± 29%  sched_debug.cfs_rq:/.spread0.avg
>      568464 ± 36%     -39.3%     344858 ± 16%  sched_debug.cfs_rq:/.spread0.max
>      252.26 ±  6%     -38.7%     154.72 ±  5%  sched_debug.cfs_rq:/.util_avg.avg
>      217.63           -15.4%     184.13 ±  6%  sched_debug.cfs_rq:/.util_avg.stddev
>       18.44 ± 12%     -50.3%       9.17 ±  9%  sched_debug.cfs_rq:/.util_est_enqueued.avg
>      575.84 ± 17%     -22.2%     447.81 ± 16%  sched_debug.cfs_rq:/.util_est_enqueued.max
>       67.91 ±  9%     -35.4%      43.85 ±  9%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
>      417156 ±  7%     -15.2%     353896 ±  5%  sched_debug.cpu.avg_idle.stddev
>      273.72 ±  3%     -11.3%     242.79 ±  4%  sched_debug.cpu.clock.stddev
>      273.72 ±  3%     -11.3%     242.79 ±  4%  sched_debug.cpu.clock_task.stddev
>      426.78 ±  3%     -27.6%     309.10 ± 12%  sched_debug.cpu.curr->pid.avg
>        1282 ±  2%     -12.4%       1123 ±  6%  sched_debug.cpu.curr->pid.stddev
>      351857 ±  3%     -16.4%     294246 ±  4%  sched_debug.cpu.max_idle_balance_cost.stddev
>        0.00           -11.8%       0.00 ±  3%  sched_debug.cpu.next_balance.stddev
>        0.09 ±  3%     -32.0%       0.06 ± 12%  sched_debug.cpu.nr_running.avg
>        0.28           -14.6%       0.24 ±  7%  sched_debug.cpu.nr_running.stddev
>      163725           +37.4%     225040 ±  4%  sched_debug.cpu.nr_switches.avg
>      174783           +35.4%     236604 ±  4%  sched_debug.cpu.nr_switches.max
>      151538           +36.4%     206743 ±  4%  sched_debug.cpu.nr_switches.min
>        2143 ±  4%     +28.5%       2753 ±  4%  sched_debug.cpu.nr_switches.stddev
>      160974           +38.1%     222280 ±  4%  sched_debug.cpu.sched_count.avg
>      170145           +36.7%     232581 ±  4%  sched_debug.cpu.sched_count.max
>      148944           +37.6%     204888 ±  4%  sched_debug.cpu.sched_count.min
>        1847 ±  8%     +38.7%       2563 ±  6%  sched_debug.cpu.sched_count.stddev
>       80136           +38.3%     110820 ±  4%  sched_debug.cpu.sched_goidle.avg
>       84559           +36.9%     115782 ±  4%  sched_debug.cpu.sched_goidle.max
>       74091           +37.8%     102072 ±  4%  sched_debug.cpu.sched_goidle.min
>      891.35 ±  6%     +41.4%       1260 ±  6%  sched_debug.cpu.sched_goidle.stddev
>       80346           +38.2%     111003 ±  4%  sched_debug.cpu.ttwu_count.avg
>      114308 ±  2%     +42.7%     163080 ±  6%  sched_debug.cpu.ttwu_count.max
>       62759 ±  3%     +38.4%      86881 ±  2%  sched_debug.cpu.ttwu_count.min
>        8223 ±  7%     +53.5%      12623 ± 13%  sched_debug.cpu.ttwu_count.stddev
>        2.35 ± 10%      -0.7        1.65 ± 14%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
>        0.94 ± 15%      -0.5        0.44 ± 58%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.rwsem_down_read_slowpath.do_user_addr_fault.page_fault
>        0.90 ± 15%      -0.5        0.42 ± 58%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.rwsem_down_read_slowpath.do_user_addr_fault.page_fault
>        1.38 ±  4%      -0.4        0.94 ± 12%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
>        0.81 ±  4%      +0.1        0.91 ±  4%  perf-profile.calltrace.cycles-pp.file_update_time.fault_dirty_shared_page.do_fault.__handle_mm_fault.handle_mm_fault
>        1.35 ±  6%      +0.2        1.51 ±  3%  perf-profile.calltrace.cycles-pp.find_get_entry.find_lock_entry.shmem_getpage_gfp.shmem_fault.__do_fault
>        1.16 ±  7%      +0.2        1.33 ±  5%  perf-profile.calltrace.cycles-pp.unlock_page.fault_dirty_shared_page.do_fault.__handle_mm_fault.handle_mm_fault
>        1.29 ±  8%      +0.2        1.46 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock.alloc_set_pte.finish_fault.do_fault.__handle_mm_fault
>        1.76 ±  7%      +0.3        2.04 ±  5%  perf-profile.calltrace.cycles-pp.page_add_file_rmap.alloc_set_pte.finish_fault.do_fault.__handle_mm_fault
>        2.13 ±  5%      +0.3        2.41 ±  4%  perf-profile.calltrace.cycles-pp.fault_dirty_shared_page.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
>        2.46 ±  8%      +0.4        2.81 ±  7%  perf-profile.calltrace.cycles-pp.up_read.do_user_addr_fault.page_fault
>        2.86 ±  6%      +0.4        3.21 ±  4%  perf-profile.calltrace.cycles-pp.find_lock_entry.shmem_getpage_gfp.shmem_fault.__do_fault.do_fault
>        3.12 ±  6%      +0.4        3.52 ±  4%  perf-profile.calltrace.cycles-pp.shmem_getpage_gfp.shmem_fault.__do_fault.do_fault.__handle_mm_fault
>        3.27 ±  6%      +0.4        3.67 ±  4%  perf-profile.calltrace.cycles-pp.shmem_fault.__do_fault.do_fault.__handle_mm_fault.handle_mm_fault
>        3.33 ±  6%      +0.4        3.75 ±  4%  perf-profile.calltrace.cycles-pp.__do_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
>        0.13 ±173%      +0.4        0.55 ±  8%  perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode
>        3.39 ±  7%      +0.5        3.89 ±  5%  perf-profile.calltrace.cycles-pp.alloc_set_pte.finish_fault.do_fault.__handle_mm_fault.handle_mm_fault
>        3.44 ±  7%      +0.5        3.94 ±  5%  perf-profile.calltrace.cycles-pp.finish_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
>        8.99 ±  6%      +1.2       10.20 ±  4%  perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.page_fault
>       12.23 ±  6%      +1.6       13.84 ±  5%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.page_fault
>       12.68 ±  6%      +1.7       14.38 ±  5%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.page_fault
>       45.14 ±  9%      +6.1       51.25 ±  6%  perf-profile.calltrace.cycles-pp.down_read_trylock.do_user_addr_fault.page_fault
>       66.96 ±  7%      +7.4       74.37 ±  4%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.page_fault
>       69.19 ±  7%      +7.7       76.89 ±  4%  perf-profile.calltrace.cycles-pp.page_fault
>       17.74 ±  8%      -3.3       14.46 ± 20%  perf-profile.children.cycles-pp.apic_timer_interrupt
>       17.02 ±  8%      -3.1       13.89 ± 20%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
>        2.97 ±  8%      -0.7        2.25 ± 12%  perf-profile.children.cycles-pp.irq_exit
>        1.88 ± 11%      -0.5        1.41 ± 11%  perf-profile.children.cycles-pp.__softirqentry_text_start
>        1.02 ± 14%      -0.4        0.59 ± 15%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
>        1.75 ± 12%      -0.4        1.35 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_irq
>        0.92 ± 12%      -0.2        0.68 ± 17%  perf-profile.children.cycles-pp.ktime_get
>        0.32 ± 20%      -0.2        0.13 ± 30%  perf-profile.children.cycles-pp.munmap
>        0.32 ± 22%      -0.2        0.14 ± 27%  perf-profile.children.cycles-pp.__x64_sys_munmap
>        0.32 ± 22%      -0.2        0.14 ± 28%  perf-profile.children.cycles-pp.__vm_munmap
>        0.29 ± 21%      -0.2        0.13 ± 23%  perf-profile.children.cycles-pp.__do_munmap
>        0.28 ± 21%      -0.2        0.12 ± 25%  perf-profile.children.cycles-pp.unmap_region
>        0.28 ± 22%      -0.2        0.13 ± 23%  perf-profile.children.cycles-pp.unmap_vmas
>        0.28 ± 21%      -0.2        0.13 ± 23%  perf-profile.children.cycles-pp.unmap_page_range
>        0.48 ±  6%      -0.1        0.34 ± 15%  perf-profile.children.cycles-pp.rebalance_domains
>        0.56 ± 12%      -0.1        0.43 ± 16%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
>        0.44 ± 11%      -0.1        0.33 ± 25%  perf-profile.children.cycles-pp.clockevents_program_event
>        0.31 ± 14%      -0.1        0.21 ± 16%  perf-profile.children.cycles-pp.run_rebalance_domains
>        0.34 ±  9%      -0.1        0.25 ± 16%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
>        0.28 ± 14%      -0.1        0.21 ± 22%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
>        0.12 ± 21%      -0.1        0.04 ± 59%  perf-profile.children.cycles-pp.smp_call_function_many_cond
>        0.12 ± 21%      -0.1        0.04 ± 59%  perf-profile.children.cycles-pp.on_each_cpu_cond_mask
>        0.12 ± 21%      -0.1        0.05 ± 60%  perf-profile.children.cycles-pp.flush_tlb_mm_range
>        0.19 ± 16%      -0.1        0.12 ± 25%  perf-profile.children.cycles-pp.irq_work_tick
>        0.31 ± 10%      -0.1        0.24 ± 21%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
>        0.17 ± 13%      -0.0        0.12 ± 15%  perf-profile.children.cycles-pp.rcu_idle_exit
>        0.12 ± 12%      -0.0        0.08 ± 30%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
>        0.14 ±  8%      -0.0        0.10 ± 12%  perf-profile.children.cycles-pp.raise_softirq
>        0.09 ±  7%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp._cond_resched
>        0.06            +0.0        0.08 ± 11%  perf-profile.children.cycles-pp.lock_page_memcg
>        0.15 ±  7%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.__mod_memcg_state
>        0.07            +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.perf_swevent_event
>        0.18 ±  7%      +0.0        0.21 ±  4%  perf-profile.children.cycles-pp.current_time
>        0.23 ±  3%      +0.0        0.26 ±  4%  perf-profile.children.cycles-pp.xas_load
>        0.14 ±  9%      +0.0        0.18 ±  8%  perf-profile.children.cycles-pp.__count_memcg_events
>        0.46 ±  5%      +0.1        0.51 ±  4%  perf-profile.children.cycles-pp.__mod_lruvec_state
>        0.42 ±  5%      +0.1        0.48 ±  7%  perf-profile.children.cycles-pp.__mark_inode_dirty
>        0.55 ±  5%      +0.1        0.61 ±  3%  perf-profile.children.cycles-pp.___perf_sw_event
>        0.47 ±  4%      +0.1        0.55 ±  8%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
>        0.42 ±  9%      +0.1        0.50 ±  8%  perf-profile.children.cycles-pp.__might_sleep
>        0.81 ±  5%      +0.1        0.92 ±  4%  perf-profile.children.cycles-pp.file_update_time
>        1.36 ±  6%      +0.2        1.52 ±  3%  perf-profile.children.cycles-pp.find_get_entry
>        1.17 ±  7%      +0.2        1.34 ±  5%  perf-profile.children.cycles-pp.unlock_page
>        1.77 ±  7%      +0.3        2.05 ±  5%  perf-profile.children.cycles-pp.page_add_file_rmap
>        2.14 ±  5%      +0.3        2.42 ±  4%  perf-profile.children.cycles-pp.fault_dirty_shared_page
>        2.51 ±  8%      +0.4        2.86 ±  7%  perf-profile.children.cycles-pp.up_read
>        2.90 ±  6%      +0.4        3.26 ±  4%  perf-profile.children.cycles-pp.find_lock_entry
>        3.12 ±  6%      +0.4        3.52 ±  4%  perf-profile.children.cycles-pp.shmem_getpage_gfp
>        3.27 ±  6%      +0.4        3.68 ±  4%  perf-profile.children.cycles-pp.shmem_fault
>        3.33 ±  6%      +0.4        3.75 ±  3%  perf-profile.children.cycles-pp.__do_fault
>        3.45 ±  7%      +0.5        3.95 ±  5%  perf-profile.children.cycles-pp.finish_fault
>        3.42 ±  7%      +0.5        3.92 ±  5%  perf-profile.children.cycles-pp.alloc_set_pte
>        9.02 ±  6%      +1.2       10.23 ±  4%  perf-profile.children.cycles-pp.do_fault
>       12.26 ±  6%      +1.6       13.88 ±  5%  perf-profile.children.cycles-pp.__handle_mm_fault
>       12.72 ±  6%      +1.7       14.43 ±  5%  perf-profile.children.cycles-pp.handle_mm_fault
>       45.27 ±  9%      +6.1       51.39 ±  6%  perf-profile.children.cycles-pp.down_read_trylock
>       67.33 ±  7%      +7.5       74.81 ±  4%  perf-profile.children.cycles-pp.do_user_addr_fault
>       69.24 ±  7%      +7.7       76.95 ±  4%  perf-profile.children.cycles-pp.page_fault
>        1.02 ± 14%      -0.4        0.59 ± 15%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>        0.74 ± 13%      -0.2        0.55 ± 18%  perf-profile.self.cycles-pp.ktime_get
>        0.28 ± 10%      -0.1        0.21 ± 17%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
>        0.28 ± 13%      -0.1        0.20 ± 21%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
>        0.19 ± 16%      -0.1        0.12 ± 25%  perf-profile.self.cycles-pp.irq_work_tick
>        0.26 ±  7%      -0.1        0.21 ± 12%  perf-profile.self.cycles-pp.__softirqentry_text_start
>        0.20 ± 15%      -0.0        0.16 ± 11%  perf-profile.self.cycles-pp.__run_timers
>        0.14 ±  7%      -0.0        0.10 ± 12%  perf-profile.self.cycles-pp.raise_softirq
>        0.13 ±  9%      -0.0        0.10 ± 15%  perf-profile.self.cycles-pp.tick_sched_timer
>        0.07 ± 10%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.clockevents_program_event
>        0.09 ±  5%      -0.0        0.07 ± 17%  perf-profile.self.cycles-pp.apic_timer_interrupt
>        0.05 ±  8%      +0.0        0.07 ± 12%  perf-profile.self.cycles-pp.lock_page_memcg
>        0.11 ±  4%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.file_update_time
>        0.10 ±  7%      +0.0        0.12 ±  7%  perf-profile.self.cycles-pp.___might_sleep
>        0.06 ±  6%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.perf_swevent_event
>        0.18 ±  8%      +0.0        0.20 ±  6%  perf-profile.self.cycles-pp.__mod_lruvec_state
>        0.13 ±  9%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.__mod_memcg_state
>        0.24 ±  6%      +0.0        0.27 ±  6%  perf-profile.self.cycles-pp.handle_mm_fault
>        0.41 ±  4%      +0.0        0.44 ±  3%  perf-profile.self.cycles-pp.___perf_sw_event
>        0.14 ±  9%      +0.0        0.17 ±  6%  perf-profile.self.cycles-pp.__count_memcg_events
>        0.21 ±  6%      +0.0        0.25 ±  4%  perf-profile.self.cycles-pp.shmem_getpage_gfp
>        0.23 ± 10%      +0.1        0.29 ± 13%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
>        1.02 ±  6%      +0.1        1.13 ±  6%  perf-profile.self.cycles-pp.find_get_entry
>        0.86 ±  4%      +0.1        1.00 ±  6%  perf-profile.self.cycles-pp.do_user_addr_fault
>        1.10 ±  6%      +0.2        1.26 ±  7%  perf-profile.self.cycles-pp.unlock_page
>        1.26 ±  7%      +0.2        1.44 ±  8%  perf-profile.self.cycles-pp.find_lock_entry
>        1.20 ±  6%      +0.2        1.38 ±  8%  perf-profile.self.cycles-pp.page_add_file_rmap
>        2.35 ±  7%      +0.3        2.69 ±  8%  perf-profile.self.cycles-pp.up_read
>        2.97 ±  6%      +0.4        3.35 ±  8%  perf-profile.self.cycles-pp.__handle_mm_fault
>       42.09 ±  8%      +5.7       47.79 ±  8%  perf-profile.self.cycles-pp.down_read_trylock
>      406.00 ± 37%   +1859.5%       7955 ±151%  interrupts.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2
>     3009113            -4.3%    2880393 ±  2%  interrupts.CAL:Function_call_interrupts
>        1445 ± 50%     +93.0%       2788 ±  9%  interrupts.CPU0.NMI:Non-maskable_interrupts
>        1445 ± 50%     +93.0%       2788 ±  9%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
>      939.50 ± 16%     +23.2%       1157 ±  5%  interrupts.CPU100.RES:Rescheduling_interrupts
>        1223 ± 11%     +43.3%       1753 ± 23%  interrupts.CPU103.NMI:Non-maskable_interrupts
>        1223 ± 11%     +43.3%       1753 ± 23%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
>        1163 ±  6%     +24.5%       1448 ±  6%  interrupts.CPU104.RES:Rescheduling_interrupts
>        1062 ± 15%     +40.5%       1493 ±  5%  interrupts.CPU105.RES:Rescheduling_interrupts
>        1490 ± 28%     +70.1%       2536 ± 20%  interrupts.CPU106.NMI:Non-maskable_interrupts
>        1490 ± 28%     +70.1%       2536 ± 20%  interrupts.CPU106.PMI:Performance_monitoring_interrupts
>        1186 ±  3%     +26.4%       1499 ± 11%  interrupts.CPU106.RES:Rescheduling_interrupts
>        1186 ±  5%     +36.5%       1618 ±  9%  interrupts.CPU107.RES:Rescheduling_interrupts
>        1484 ± 29%     +98.1%       2941 ±  9%  interrupts.CPU108.NMI:Non-maskable_interrupts
>        1484 ± 29%     +98.1%       2941 ±  9%  interrupts.CPU108.PMI:Performance_monitoring_interrupts
>        1087 ± 11%     +35.2%       1470 ± 11%  interrupts.CPU108.RES:Rescheduling_interrupts
>      905.00 ± 14%     +26.9%       1148 ±  8%  interrupts.CPU109.RES:Rescheduling_interrupts
>        1534 ± 35%     +91.6%       2939 ±  9%  interrupts.CPU110.NMI:Non-maskable_interrupts
>        1534 ± 35%     +91.6%       2939 ±  9%  interrupts.CPU110.PMI:Performance_monitoring_interrupts
>        1222 ± 11%     +54.3%       1885 ± 44%  interrupts.CPU111.NMI:Non-maskable_interrupts
>        1222 ± 11%     +54.3%       1885 ± 44%  interrupts.CPU111.PMI:Performance_monitoring_interrupts
>        1066           +27.7%       1360 ±  5%  interrupts.CPU111.RES:Rescheduling_interrupts
>        1214 ± 12%     +51.1%       1833 ± 35%  interrupts.CPU112.NMI:Non-maskable_interrupts
>        1214 ± 12%     +51.1%       1833 ± 35%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
>        1206 ± 12%     +30.8%       1577 ±  9%  interrupts.CPU116.RES:Rescheduling_interrupts
>        1102 ± 10%     +46.7%       1616 ± 11%  interrupts.CPU117.RES:Rescheduling_interrupts
>        1211 ±  3%     +35.2%       1638 ± 15%  interrupts.CPU118.RES:Rescheduling_interrupts
>        1217 ± 11%    +115.4%       2623 ± 29%  interrupts.CPU119.NMI:Non-maskable_interrupts
>        1217 ± 11%    +115.4%       2623 ± 29%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
>      406.00 ± 37%   +1859.5%       7955 ±151%  interrupts.CPU12.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2
>        1125 ± 12%     +27.3%       1432 ± 10%  interrupts.CPU12.NMI:Non-maskable_interrupts
>        1125 ± 12%     +27.3%       1432 ± 10%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
>        1173 ± 12%     +55.5%       1823 ± 35%  interrupts.CPU120.NMI:Non-maskable_interrupts
>        1173 ± 12%     +55.5%       1823 ± 35%  interrupts.CPU120.PMI:Performance_monitoring_interrupts
>        1112 ±  5%     +40.2%       1559 ±  6%  interrupts.CPU121.RES:Rescheduling_interrupts
>        1227 ± 11%     +50.5%       1846 ± 34%  interrupts.CPU122.NMI:Non-maskable_interrupts
>        1227 ± 11%     +50.5%       1846 ± 34%  interrupts.CPU122.PMI:Performance_monitoring_interrupts
>        1036 ± 11%     +36.6%       1416 ±  4%  interrupts.CPU122.RES:Rescheduling_interrupts
>        1164 ± 22%     +40.8%       1639 ± 16%  interrupts.CPU123.RES:Rescheduling_interrupts
>        1419 ± 13%     +20.7%       1712 ±  9%  interrupts.CPU124.RES:Rescheduling_interrupts
>        1209 ± 11%     +21.5%       1469 ±  9%  interrupts.CPU126.NMI:Non-maskable_interrupts
>        1209 ± 11%     +21.5%       1469 ±  9%  interrupts.CPU126.PMI:Performance_monitoring_interrupts
>        1266 ± 16%     +34.9%       1708 ± 13%  interrupts.CPU126.RES:Rescheduling_interrupts
>        1207 ± 24%     +36.5%       1648 ± 11%  interrupts.CPU127.RES:Rescheduling_interrupts
>        1298 ± 12%     +30.8%       1698        interrupts.CPU128.RES:Rescheduling_interrupts
>        1354 ±  7%     +33.5%       1808 ±  9%  interrupts.CPU129.RES:Rescheduling_interrupts
>        1046 ±  7%     +42.3%       1488 ± 17%  interrupts.CPU130.RES:Rescheduling_interrupts
>        1068 ±  6%     +45.9%       1558 ± 13%  interrupts.CPU131.RES:Rescheduling_interrupts
>        1200 ±  2%     +41.7%       1701 ± 14%  interrupts.CPU132.RES:Rescheduling_interrupts
>        1191 ± 11%     +85.4%       2208 ± 39%  interrupts.CPU133.NMI:Non-maskable_interrupts
>        1191 ± 11%     +85.4%       2208 ± 39%  interrupts.CPU133.PMI:Performance_monitoring_interrupts
>        1327 ±  9%     +13.8%       1510 ±  4%  interrupts.CPU134.RES:Rescheduling_interrupts
>        1203 ± 12%     +52.8%       1837 ± 34%  interrupts.CPU136.NMI:Non-maskable_interrupts
>        1203 ± 12%     +52.8%       1837 ± 34%  interrupts.CPU136.PMI:Performance_monitoring_interrupts
>      976.75 ±  8%     +19.0%       1162 ±  8%  interrupts.CPU136.RES:Rescheduling_interrupts
>        1438 ± 34%     +75.9%       2530 ± 27%  interrupts.CPU137.NMI:Non-maskable_interrupts
>        1438 ± 34%     +75.9%       2530 ± 27%  interrupts.CPU137.PMI:Performance_monitoring_interrupts
>      950.75 ±  5%     +33.1%       1265 ± 14%  interrupts.CPU139.RES:Rescheduling_interrupts
>        1054 ±  6%     +26.9%       1338 ±  8%  interrupts.CPU140.RES:Rescheduling_interrupts
>       10703 ±  3%      -8.0%       9849 ±  4%  interrupts.CPU140.TLB:TLB_shootdowns
>        1509 ± 29%     +71.5%       2589 ± 27%  interrupts.CPU141.NMI:Non-maskable_interrupts
>        1509 ± 29%     +71.5%       2589 ± 27%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
>        1334 ±  7%     +14.9%       1533 ±  8%  interrupts.CPU141.RES:Rescheduling_interrupts
>        1505 ± 36%     +71.7%       2585 ± 27%  interrupts.CPU142.NMI:Non-maskable_interrupts
>        1505 ± 36%     +71.7%       2585 ± 27%  interrupts.CPU142.PMI:Performance_monitoring_interrupts
>        1206 ±  7%     +32.3%       1596 ± 13%  interrupts.CPU142.RES:Rescheduling_interrupts
>        1218 ± 11%     +55.0%       1888 ± 44%  interrupts.CPU143.NMI:Non-maskable_interrupts
>        1218 ± 11%     +55.0%       1888 ± 44%  interrupts.CPU143.PMI:Performance_monitoring_interrupts
>        1883 ±  5%     +37.7%       2594 ± 12%  interrupts.CPU143.RES:Rescheduling_interrupts
>        1211 ± 11%     +51.1%       1831 ± 35%  interrupts.CPU144.NMI:Non-maskable_interrupts
>        1211 ± 11%     +51.1%       1831 ± 35%  interrupts.CPU144.PMI:Performance_monitoring_interrupts
>        1266 ± 15%     +43.3%       1815 ±  7%  interrupts.CPU144.RES:Rescheduling_interrupts
>        1226 ± 11%    +109.7%       2572 ± 27%  interrupts.CPU145.NMI:Non-maskable_interrupts
>        1226 ± 11%    +109.7%       2572 ± 27%  interrupts.CPU145.PMI:Performance_monitoring_interrupts
>        1370 ± 27%     +54.3%       2114 ± 35%  interrupts.CPU15.NMI:Non-maskable_interrupts
>        1370 ± 27%     +54.3%       2114 ± 35%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
>      685.25 ±  6%     +33.2%     913.00 ±  9%  interrupts.CPU15.RES:Rescheduling_interrupts
>        1197 ± 11%     +84.2%       2206 ± 39%  interrupts.CPU150.NMI:Non-maskable_interrupts
>        1197 ± 11%     +84.2%       2206 ± 39%  interrupts.CPU150.PMI:Performance_monitoring_interrupts
>        1097 ±  9%     +33.0%       1459 ± 13%  interrupts.CPU152.RES:Rescheduling_interrupts
>        1263 ±  7%     +31.8%       1665 ±  8%  interrupts.CPU154.RES:Rescheduling_interrupts
>        1215 ± 12%     +49.4%       1815 ± 33%  interrupts.CPU155.NMI:Non-maskable_interrupts
>        1215 ± 12%     +49.4%       1815 ± 33%  interrupts.CPU155.PMI:Performance_monitoring_interrupts
>        1222 ±  5%     +29.1%       1578 ±  9%  interrupts.CPU155.RES:Rescheduling_interrupts
>        1065 ±  9%     +30.1%       1386 ± 13%  interrupts.CPU156.RES:Rescheduling_interrupts
>        1017 ± 13%     +23.8%       1259 ±  7%  interrupts.CPU157.RES:Rescheduling_interrupts
>        1055 ± 11%     +28.5%       1356 ± 11%  interrupts.CPU158.RES:Rescheduling_interrupts
>        1016 ±  9%     +39.7%       1420 ± 13%  interrupts.CPU159.RES:Rescheduling_interrupts
>        1084 ±  2%     +24.3%       1347 ± 13%  interrupts.CPU161.RES:Rescheduling_interrupts
>      918.75 ± 13%     +20.4%       1106 ± 11%  interrupts.CPU162.RES:Rescheduling_interrupts
>       10725            -8.9%       9774 ±  2%  interrupts.CPU162.TLB:TLB_shootdowns
>      959.75 ±  9%     +24.8%       1197 ± 11%  interrupts.CPU163.RES:Rescheduling_interrupts
>        1177 ±  8%     +36.8%       1610 ± 15%  interrupts.CPU167.RES:Rescheduling_interrupts
>        1211 ± 12%     +84.4%       2233 ± 39%  interrupts.CPU168.NMI:Non-maskable_interrupts
>        1211 ± 12%     +84.4%       2233 ± 39%  interrupts.CPU168.PMI:Performance_monitoring_interrupts
>        1220 ± 12%     +22.2%       1491 ±  8%  interrupts.CPU169.RES:Rescheduling_interrupts
>      711.00 ±  6%     +33.3%     948.00 ± 13%  interrupts.CPU17.RES:Rescheduling_interrupts
>        1254 ±  8%     +32.6%       1663 ± 18%  interrupts.CPU170.RES:Rescheduling_interrupts
>        1149 ± 10%     +46.0%       1677 ± 11%  interrupts.CPU171.RES:Rescheduling_interrupts
>        1578 ± 49%     +41.6%       2234 ± 40%  interrupts.CPU173.NMI:Non-maskable_interrupts
>        1578 ± 49%     +41.6%       2234 ± 40%  interrupts.CPU173.PMI:Performance_monitoring_interrupts
>        1242 ±  9%     +17.3%       1456 ±  9%  interrupts.CPU174.NMI:Non-maskable_interrupts
>        1242 ±  9%     +17.3%       1456 ±  9%  interrupts.CPU174.PMI:Performance_monitoring_interrupts
>        1008 ±  7%     +26.7%       1278        interrupts.CPU176.RES:Rescheduling_interrupts
>        1015 ±  4%     +33.9%       1360 ±  8%  interrupts.CPU177.RES:Rescheduling_interrupts
>      929.50 ± 13%     +27.4%       1183 ±  6%  interrupts.CPU178.RES:Rescheduling_interrupts
>        1051 ±  8%     +20.7%       1268        interrupts.CPU179.RES:Rescheduling_interrupts
>        1149 ± 13%     +87.3%       2153 ± 34%  interrupts.CPU18.NMI:Non-maskable_interrupts
>        1149 ± 13%     +87.3%       2153 ± 34%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
>      832.75 ±  4%     +23.5%       1028        interrupts.CPU18.RES:Rescheduling_interrupts
>        1215 ± 11%     +21.9%       1481 ±  9%  interrupts.CPU180.NMI:Non-maskable_interrupts
>        1215 ± 11%     +21.9%       1481 ±  9%  interrupts.CPU180.PMI:Performance_monitoring_interrupts
>        1050 ± 15%     +39.3%       1463 ±  6%  interrupts.CPU181.RES:Rescheduling_interrupts
>        1093 ±  5%     +28.7%       1407 ± 11%  interrupts.CPU182.RES:Rescheduling_interrupts
>        1082 ±  5%     +27.6%       1381 ±  3%  interrupts.CPU183.RES:Rescheduling_interrupts
>      990.50 ± 10%     +28.1%       1269 ±  8%  interrupts.CPU184.RES:Rescheduling_interrupts
>      964.00 ± 13%     +28.2%       1235 ±  8%  interrupts.CPU186.RES:Rescheduling_interrupts
>        1498 ± 29%     +68.5%       2524 ± 19%  interrupts.CPU187.NMI:Non-maskable_interrupts
>        1498 ± 29%     +68.5%       2524 ± 19%  interrupts.CPU187.PMI:Performance_monitoring_interrupts
>        1084 ± 11%     +35.9%       1473 ±  9%  interrupts.CPU188.RES:Rescheduling_interrupts
>        1537 ± 34%     +68.0%       2582 ± 26%  interrupts.CPU190.NMI:Non-maskable_interrupts
>        1537 ± 34%     +68.0%       2582 ± 26%  interrupts.CPU190.PMI:Performance_monitoring_interrupts
>        1247 ±  2%     +32.7%       1655 ±  5%  interrupts.CPU190.RES:Rescheduling_interrupts
>        1292 ± 11%     +26.0%       1628 ±  6%  interrupts.CPU191.RES:Rescheduling_interrupts
>        1223 ± 13%     +50.2%       1837 ± 33%  interrupts.CPU192.NMI:Non-maskable_interrupts
>        1223 ± 13%     +50.2%       1837 ± 33%  interrupts.CPU192.PMI:Performance_monitoring_interrupts
>        1263 ±  8%     +30.8%       1651 ±  8%  interrupts.CPU193.RES:Rescheduling_interrupts
>        1369 ±  8%     +25.9%       1724 ± 13%  interrupts.CPU196.RES:Rescheduling_interrupts
>        1127 ±  9%     +27.5%       1437 ±  6%  interrupts.CPU197.RES:Rescheduling_interrupts
>        1134 ± 10%     +25.8%       1427 ± 19%  interrupts.CPU198.RES:Rescheduling_interrupts
>        1205 ± 12%     +47.0%       1772 ± 24%  interrupts.CPU199.NMI:Non-maskable_interrupts
>        1205 ± 12%     +47.0%       1772 ± 24%  interrupts.CPU199.PMI:Performance_monitoring_interrupts
>        1160 ± 12%    +117.2%       2520 ± 25%  interrupts.CPU2.NMI:Non-maskable_interrupts
>        1160 ± 12%    +117.2%       2520 ± 25%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
>        1408 ± 27%     +77.7%       2502 ± 27%  interrupts.CPU20.NMI:Non-maskable_interrupts
>        1408 ± 27%     +77.7%       2502 ± 27%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
>        1077 ± 13%     +28.2%       1381 ±  9%  interrupts.CPU202.RES:Rescheduling_interrupts
>        1512 ± 36%     +41.9%       2147 ± 26%  interrupts.CPU203.NMI:Non-maskable_interrupts
>        1512 ± 36%     +41.9%       2147 ± 26%  interrupts.CPU203.PMI:Performance_monitoring_interrupts
>        1052 ±  5%     +35.0%       1420 ± 20%  interrupts.CPU203.RES:Rescheduling_interrupts
>        1001 ±  3%     +41.0%       1411 ± 19%  interrupts.CPU204.RES:Rescheduling_interrupts
>        1140 ±  4%     +31.4%       1498 ±  7%  interrupts.CPU206.RES:Rescheduling_interrupts
>        1288 ±  5%     +27.6%       1643 ±  3%  interrupts.CPU208.RES:Rescheduling_interrupts
>        1131 ± 13%     +60.9%       1821 ± 44%  interrupts.CPU21.NMI:Non-maskable_interrupts
>        1131 ± 13%     +60.9%       1821 ± 44%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
>        1126 ±  2%     +29.1%       1454 ± 13%  interrupts.CPU210.RES:Rescheduling_interrupts
>       10556 ±  3%      -8.9%       9618 ±  2%  interrupts.CPU210.TLB:TLB_shootdowns
>        1042 ± 12%     +42.8%       1489 ±  9%  interrupts.CPU211.RES:Rescheduling_interrupts
>        1156 ± 10%     +18.5%       1371 ±  9%  interrupts.CPU212.RES:Rescheduling_interrupts
>        1336 ±  4%     +13.8%       1520 ±  5%  interrupts.CPU213.RES:Rescheduling_interrupts
>        1201 ±  5%     +23.0%       1477 ±  8%  interrupts.CPU214.RES:Rescheduling_interrupts
>        1622 ±  6%     +36.7%       2219 ± 17%  interrupts.CPU215.RES:Rescheduling_interrupts
>        1178 ± 16%     +30.5%       1538 ±  6%  interrupts.CPU216.RES:Rescheduling_interrupts
>        1006 ± 13%     +48.5%       1494 ±  7%  interrupts.CPU217.RES:Rescheduling_interrupts
>        1603 ± 48%     +40.3%       2248 ± 38%  interrupts.CPU218.NMI:Non-maskable_interrupts
>        1603 ± 48%     +40.3%       2248 ± 38%  interrupts.CPU218.PMI:Performance_monitoring_interrupts
>        1235 ±  9%     +36.4%       1684 ±  6%  interrupts.CPU218.RES:Rescheduling_interrupts
>        1219 ± 13%     +29.3%       1576 ±  8%  interrupts.CPU219.RES:Rescheduling_interrupts
>        1153 ± 12%    +145.2%       2828 ± 10%  interrupts.CPU22.NMI:Non-maskable_interrupts
>        1153 ± 12%    +145.2%       2828 ± 10%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
>      849.50 ±  4%     +25.5%       1066 ±  6%  interrupts.CPU22.RES:Rescheduling_interrupts
>        1209 ± 12%     +84.1%       2226 ± 40%  interrupts.CPU220.NMI:Non-maskable_interrupts
>        1209 ± 12%     +84.1%       2226 ± 40%  interrupts.CPU220.PMI:Performance_monitoring_interrupts
>        1037 ± 10%     +14.3%       1186 ±  3%  interrupts.CPU220.RES:Rescheduling_interrupts
>        1092 ±  8%     +31.8%       1440 ± 14%  interrupts.CPU221.RES:Rescheduling_interrupts
>        1112 ± 13%     +28.8%       1433 ±  9%  interrupts.CPU222.RES:Rescheduling_interrupts
>        1220 ±  9%     +32.8%       1620 ±  7%  interrupts.CPU223.RES:Rescheduling_interrupts
>        1515 ± 34%     +42.5%       2159 ± 28%  interrupts.CPU226.NMI:Non-maskable_interrupts
>        1515 ± 34%     +42.5%       2159 ± 28%  interrupts.CPU226.PMI:Performance_monitoring_interrupts
>        1083 ±  8%     +33.0%       1441 ± 13%  interrupts.CPU226.RES:Rescheduling_interrupts
>      956.50 ± 11%     +23.7%       1183 ±  8%  interrupts.CPU227.RES:Rescheduling_interrupts
>        1180 ±  3%     +38.2%       1631 ± 18%  interrupts.CPU228.RES:Rescheduling_interrupts
>        1223 ± 11%     +20.1%       1468 ±  9%  interrupts.CPU229.NMI:Non-maskable_interrupts
>        1223 ± 11%     +20.1%       1468 ±  9%  interrupts.CPU229.PMI:Performance_monitoring_interrupts
>        1149 ± 12%     +58.3%       1819 ± 44%  interrupts.CPU23.NMI:Non-maskable_interrupts
>        1149 ± 12%     +58.3%       1819 ± 44%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
>      726.75 ±  4%     +32.6%     963.50 ±  6%  interrupts.CPU23.RES:Rescheduling_interrupts
>        1013 ±  7%     +35.2%       1370 ± 14%  interrupts.CPU230.RES:Rescheduling_interrupts
>        1004 ±  5%     +29.5%       1300 ±  8%  interrupts.CPU231.RES:Rescheduling_interrupts
>        1213 ± 13%    +141.5%       2929 ±  9%  interrupts.CPU233.NMI:Non-maskable_interrupts
>        1213 ± 13%    +141.5%       2929 ±  9%  interrupts.CPU233.PMI:Performance_monitoring_interrupts
>        1013 ±  9%     +22.5%       1241 ±  8%  interrupts.CPU233.RES:Rescheduling_interrupts
>        1177 ± 13%     +80.2%       2121 ± 29%  interrupts.CPU234.NMI:Non-maskable_interrupts
>        1177 ± 13%     +80.2%       2121 ± 29%  interrupts.CPU234.PMI:Performance_monitoring_interrupts
>        1515 ± 35%     +68.0%       2545 ± 26%  interrupts.CPU237.NMI:Non-maskable_interrupts
>        1515 ± 35%     +68.0%       2545 ± 26%  interrupts.CPU237.PMI:Performance_monitoring_interrupts
>        1770 ± 29%     +64.4%       2910 ± 10%  interrupts.CPU238.NMI:Non-maskable_interrupts
>        1770 ± 29%     +64.4%       2910 ± 10%  interrupts.CPU238.PMI:Performance_monitoring_interrupts
>        1000 ± 10%     +23.4%       1233 ±  3%  interrupts.CPU238.RES:Rescheduling_interrupts
>        1178 ±  6%     +23.5%       1456 ±  8%  interrupts.CPU239.RES:Rescheduling_interrupts
>        1151 ± 14%    +115.7%       2483 ± 27%  interrupts.CPU24.NMI:Non-maskable_interrupts
>        1151 ± 14%    +115.7%       2483 ± 27%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
>      823.00 ±  9%     +51.7%       1248 ± 19%  interrupts.CPU24.RES:Rescheduling_interrupts
>        1217 ± 15%     +22.9%       1496 ±  8%  interrupts.CPU240.RES:Rescheduling_interrupts
>        1014 ±  4%     +54.4%       1565 ± 20%  interrupts.CPU242.RES:Rescheduling_interrupts
>       10616 ±  3%      -8.3%       9731        interrupts.CPU242.TLB:TLB_shootdowns
>        1043 ± 11%     +36.3%       1422 ± 16%  interrupts.CPU243.RES:Rescheduling_interrupts
>        1215 ± 11%     +83.3%       2227 ± 40%  interrupts.CPU247.NMI:Non-maskable_interrupts
>        1215 ± 11%     +83.3%       2227 ± 40%  interrupts.CPU247.PMI:Performance_monitoring_interrupts
>        1499 ± 34%     +68.8%       2531 ± 20%  interrupts.CPU248.NMI:Non-maskable_interrupts
>        1499 ± 34%     +68.8%       2531 ± 20%  interrupts.CPU248.PMI:Performance_monitoring_interrupts
>        1034 ±  5%     +28.3%       1327 ± 15%  interrupts.CPU248.RES:Rescheduling_interrupts
>        1103 ± 14%     +25.1%       1380 ±  8%  interrupts.CPU249.RES:Rescheduling_interrupts
>        1143 ± 13%     +49.8%       1712 ± 24%  interrupts.CPU25.NMI:Non-maskable_interrupts
>        1143 ± 13%     +49.8%       1712 ± 24%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
>      749.50 ±  9%     +24.9%     936.25 ±  4%  interrupts.CPU25.RES:Rescheduling_interrupts
>       10842 ±  2%      -8.8%       9885 ±  2%  interrupts.CPU25.TLB:TLB_shootdowns
>        1099 ±  5%     +18.0%       1297 ±  3%  interrupts.CPU251.RES:Rescheduling_interrupts
>      993.25 ± 11%     +23.1%       1223 ±  5%  interrupts.CPU252.RES:Rescheduling_interrupts
>        1017 ± 12%     +34.7%       1369 ± 11%  interrupts.CPU253.RES:Rescheduling_interrupts
>        1532 ± 49%     +44.3%       2211 ± 39%  interrupts.CPU254.NMI:Non-maskable_interrupts
>        1532 ± 49%     +44.3%       2211 ± 39%  interrupts.CPU254.PMI:Performance_monitoring_interrupts
>      979.50 ± 10%     +29.8%       1271 ± 10%  interrupts.CPU254.RES:Rescheduling_interrupts
>      928.50 ±  8%     +43.7%       1334 ± 17%  interrupts.CPU256.RES:Rescheduling_interrupts
>        1049 ± 12%     +23.9%       1300 ±  8%  interrupts.CPU257.RES:Rescheduling_interrupts
>        1183 ±  4%     +12.5%       1331 ± 10%  interrupts.CPU258.RES:Rescheduling_interrupts
>      796.50 ±  6%     +38.3%       1101 ± 23%  interrupts.CPU26.RES:Rescheduling_interrupts
>        1087 ±  8%     +28.4%       1396 ±  6%  interrupts.CPU260.RES:Rescheduling_interrupts
>        1081 ± 11%     +40.3%       1517 ±  6%  interrupts.CPU261.RES:Rescheduling_interrupts
>        1553 ± 48%     +43.8%       2233 ± 39%  interrupts.CPU262.NMI:Non-maskable_interrupts
>        1553 ± 48%     +43.8%       2233 ± 39%  interrupts.CPU262.PMI:Performance_monitoring_interrupts
>        1270 ± 13%     +29.0%       1638 ± 10%  interrupts.CPU262.RES:Rescheduling_interrupts
>        1170 ± 13%     +57.6%       1844 ± 44%  interrupts.CPU264.NMI:Non-maskable_interrupts
>        1170 ± 13%     +57.6%       1844 ± 44%  interrupts.CPU264.PMI:Performance_monitoring_interrupts
>      899.00 ± 10%     +38.8%       1247 ± 14%  interrupts.CPU264.RES:Rescheduling_interrupts
>      912.25 ± 11%     +18.3%       1079 ±  7%  interrupts.CPU267.RES:Rescheduling_interrupts
>      703.25 ±  9%     +48.5%       1044 ± 21%  interrupts.CPU27.RES:Rescheduling_interrupts
>        1198 ± 12%     +82.4%       2185 ± 40%  interrupts.CPU273.NMI:Non-maskable_interrupts
>        1198 ± 12%     +82.4%       2185 ± 40%  interrupts.CPU273.PMI:Performance_monitoring_interrupts
>      847.50 ± 10%     +44.9%       1228 ± 18%  interrupts.CPU273.RES:Rescheduling_interrupts
>      900.50 ± 19%     +34.2%       1208 ± 17%  interrupts.CPU274.RES:Rescheduling_interrupts
>        1175 ± 13%     +82.5%       2145 ± 36%  interrupts.CPU275.NMI:Non-maskable_interrupts
>        1175 ± 13%     +82.5%       2145 ± 36%  interrupts.CPU275.PMI:Performance_monitoring_interrupts
>      800.50 ±  8%     +35.5%       1084 ± 14%  interrupts.CPU275.RES:Rescheduling_interrupts
>      803.00 ±  7%     +55.0%       1244 ± 20%  interrupts.CPU276.RES:Rescheduling_interrupts
>      888.50 ± 11%     +40.6%       1249 ±  9%  interrupts.CPU277.RES:Rescheduling_interrupts
>       10916           -10.1%       9813 ±  2%  interrupts.CPU277.TLB:TLB_shootdowns
>        1035 ±  2%     +23.6%       1280 ±  5%  interrupts.CPU278.RES:Rescheduling_interrupts
>      926.75 ±  8%     +37.3%       1272 ± 16%  interrupts.CPU279.RES:Rescheduling_interrupts
>      753.25 ± 21%     +42.8%       1076 ±  8%  interrupts.CPU282.RES:Rescheduling_interrupts
>        1183 ± 12%     +55.6%       1841 ± 44%  interrupts.CPU283.NMI:Non-maskable_interrupts
>        1183 ± 12%     +55.6%       1841 ± 44%  interrupts.CPU283.PMI:Performance_monitoring_interrupts
>        1187 ± 11%    +112.7%       2525 ± 27%  interrupts.CPU284.NMI:Non-maskable_interrupts
>        1187 ± 11%    +112.7%       2525 ± 27%  interrupts.CPU284.PMI:Performance_monitoring_interrupts
>        2063 ± 28%     +38.7%       2863 ±  9%  interrupts.CPU285.NMI:Non-maskable_interrupts
>        2063 ± 28%     +38.7%       2863 ±  9%  interrupts.CPU285.PMI:Performance_monitoring_interrupts
>      854.25 ±  3%     +27.0%       1084 ±  5%  interrupts.CPU286.RES:Rescheduling_interrupts
>      729.50 ± 11%     +33.3%     972.75 ±  5%  interrupts.CPU287.RES:Rescheduling_interrupts
>        1473 ± 50%     +72.6%       2542 ± 30%  interrupts.CPU29.NMI:Non-maskable_interrupts
>        1473 ± 50%     +72.6%       2542 ± 30%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
>        1133 ± 13%     +92.6%       2183 ± 39%  interrupts.CPU3.NMI:Non-maskable_interrupts
>        1133 ± 13%     +92.6%       2183 ± 39%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
>      853.00 ±  5%     +19.3%       1017 ±  4%  interrupts.CPU3.RES:Rescheduling_interrupts
>      865.00 ±  5%     +27.6%       1103 ±  9%  interrupts.CPU30.RES:Rescheduling_interrupts
>      750.50 ± 10%     +20.9%     907.25 ± 12%  interrupts.CPU31.RES:Rescheduling_interrupts
>        1140 ± 13%     +93.0%       2201 ± 40%  interrupts.CPU32.NMI:Non-maskable_interrupts
>        1140 ± 13%     +93.0%       2201 ± 40%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
>      800.50 ± 10%     +17.9%     944.00 ±  4%  interrupts.CPU32.RES:Rescheduling_interrupts
>      836.50 ±  5%     +18.9%     994.50 ±  4%  interrupts.CPU34.RES:Rescheduling_interrupts
>      797.00 ±  4%     +21.9%     971.75 ±  6%  interrupts.CPU35.RES:Rescheduling_interrupts
>      754.00 ±  4%     +21.2%     914.00 ±  5%  interrupts.CPU36.RES:Rescheduling_interrupts
>      616.25 ±  8%     +44.8%     892.25 ± 15%  interrupts.CPU37.RES:Rescheduling_interrupts
>      791.25 ±  6%     +32.6%       1049 ± 16%  interrupts.CPU38.RES:Rescheduling_interrupts
>      686.75 ±  5%     +34.9%     926.50 ± 14%  interrupts.CPU39.RES:Rescheduling_interrupts
>      878.50 ±  6%     +31.6%       1156 ± 16%  interrupts.CPU4.RES:Rescheduling_interrupts
>        1180 ± 12%     +85.2%       2185 ± 34%  interrupts.CPU42.NMI:Non-maskable_interrupts
>        1180 ± 12%     +85.2%       2185 ± 34%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
>      980.75 ± 11%     +39.5%       1367 ± 14%  interrupts.CPU45.RES:Rescheduling_interrupts
>        1103 ± 14%     +22.1%       1347 ±  6%  interrupts.CPU46.RES:Rescheduling_interrupts
>      925.00 ± 26%     +42.2%       1315 ± 10%  interrupts.CPU47.RES:Rescheduling_interrupts
>      960.25 ± 15%     +41.2%       1356 ± 12%  interrupts.CPU48.RES:Rescheduling_interrupts
>      921.75 ±  8%     +23.2%       1135 ± 11%  interrupts.CPU49.RES:Rescheduling_interrupts
>        1067 ± 15%     +25.3%       1337 ± 10%  interrupts.CPU50.RES:Rescheduling_interrupts
>        1076 ±  7%     +49.1%       1605 ± 17%  interrupts.CPU56.RES:Rescheduling_interrupts
>       10449 ±  2%      -7.7%       9646 ±  3%  interrupts.CPU56.TLB:TLB_shootdowns
>      913.75 ± 13%     +40.9%       1287 ± 13%  interrupts.CPU58.RES:Rescheduling_interrupts
>      767.50 ±  8%     +54.1%       1182 ± 14%  interrupts.CPU59.RES:Rescheduling_interrupts
>        1374 ± 27%     +79.9%       2473 ± 26%  interrupts.CPU6.NMI:Non-maskable_interrupts
>        1374 ± 27%     +79.9%       2473 ± 26%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
>      825.00 ± 11%     +38.8%       1145 ± 18%  interrupts.CPU6.RES:Rescheduling_interrupts
>        1016 ± 21%     +32.2%       1343 ± 12%  interrupts.CPU60.RES:Rescheduling_interrupts
>      802.50 ± 12%     +52.6%       1225 ± 16%  interrupts.CPU61.RES:Rescheduling_interrupts
>      895.50 ±  8%     +21.4%       1087 ±  2%  interrupts.CPU62.RES:Rescheduling_interrupts
>      973.25 ± 22%     +41.4%       1376 ±  6%  interrupts.CPU66.RES:Rescheduling_interrupts
>      854.75 ± 10%     +33.8%       1143 ± 13%  interrupts.CPU67.RES:Rescheduling_interrupts
>      771.25 ± 12%     +40.2%       1081 ± 13%  interrupts.CPU69.RES:Rescheduling_interrupts
>      621.50 ±  7%     +45.4%     903.50 ± 12%  interrupts.CPU7.RES:Rescheduling_interrupts
>        1169 ±  7%     +24.5%       1455 ±  5%  interrupts.CPU70.RES:Rescheduling_interrupts
>       10547 ±  2%      -8.5%       9654        interrupts.CPU70.TLB:TLB_shootdowns
>        1574 ± 12%     +34.4%       2115 ± 11%  interrupts.CPU71.RES:Rescheduling_interrupts
>        1456 ± 29%     +49.9%       2183 ± 35%  interrupts.CPU72.NMI:Non-maskable_interrupts
>        1456 ± 29%     +49.9%       2183 ± 35%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
>        1183 ± 12%     +25.6%       1485 ±  8%  interrupts.CPU72.RES:Rescheduling_interrupts
>        1514 ± 33%     +41.4%       2142 ± 27%  interrupts.CPU73.NMI:Non-maskable_interrupts
>        1514 ± 33%     +41.4%       2142 ± 27%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
>      986.75 ± 21%     +56.7%       1546 ± 14%  interrupts.CPU73.RES:Rescheduling_interrupts
>        1001 ±  4%     +35.5%       1356 ±  7%  interrupts.CPU76.RES:Rescheduling_interrupts
>        1004 ± 10%     +22.4%       1229 ± 11%  interrupts.CPU77.RES:Rescheduling_interrupts
>      897.25 ±  9%     +34.5%       1206 ± 19%  interrupts.CPU78.RES:Rescheduling_interrupts
>      785.75 ± 11%     +25.6%     987.00 ±  4%  interrupts.CPU8.RES:Rescheduling_interrupts
>      943.75 ±  9%     +29.1%       1218 ± 12%  interrupts.CPU81.RES:Rescheduling_interrupts
>      991.75 ±  2%     +37.9%       1367 ± 17%  interrupts.CPU82.RES:Rescheduling_interrupts
>        1177 ± 12%     +58.9%       1870 ± 44%  interrupts.CPU83.NMI:Non-maskable_interrupts
>        1177 ± 12%     +58.9%       1870 ± 44%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
>        1170 ± 15%     +17.1%       1370 ±  6%  interrupts.CPU83.RES:Rescheduling_interrupts
>        1499 ± 30%     +45.9%       2188 ± 34%  interrupts.CPU86.NMI:Non-maskable_interrupts
>        1499 ± 30%     +45.9%       2188 ± 34%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
>        1193 ± 10%    +109.8%       2502 ± 20%  interrupts.CPU87.NMI:Non-maskable_interrupts
>        1193 ± 10%    +109.8%       2502 ± 20%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
>        1145 ± 14%     +31.2%       1502 ±  7%  interrupts.CPU87.RES:Rescheduling_interrupts
>        1177 ±  6%     +21.1%       1426 ±  8%  interrupts.CPU88.RES:Rescheduling_interrupts
>        1140 ± 14%     +91.1%       2179 ± 40%  interrupts.CPU9.NMI:Non-maskable_interrupts
>        1140 ± 14%     +91.1%       2179 ± 40%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
>      676.75 ± 10%     +41.2%     955.25 ± 11%  interrupts.CPU9.RES:Rescheduling_interrupts
>        1195 ± 11%     +56.7%       1872 ± 43%  interrupts.CPU90.NMI:Non-maskable_interrupts
>        1195 ± 11%     +56.7%       1872 ± 43%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
>        2083 ± 27%     +40.0%       2917 ± 10%  interrupts.CPU95.NMI:Non-maskable_interrupts
>        2083 ± 27%     +40.0%       2917 ± 10%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
>        1110 ±  6%     +14.4%       1270 ± 11%  interrupts.CPU95.RES:Rescheduling_interrupts
>        1110 ±  6%     +41.0%       1566 ± 18%  interrupts.CPU98.RES:Rescheduling_interrupts
>      984.75 ±  9%     +58.2%       1557 ± 19%  interrupts.CPU99.RES:Rescheduling_interrupts
>      310787 ±  6%     +23.6%     384165 ±  4%  interrupts.RES:Rescheduling_interrupts
>       33134 ±  2%     +14.7%      38003 ±  6%  softirqs.CPU0.SCHED
>       27079 ±  2%     +17.4%      31779 ±  3%  softirqs.CPU1.SCHED
>      162495            +8.8%     176829 ±  4%  softirqs.CPU1.TIMER
>       26465           +19.2%      31556 ±  3%  softirqs.CPU10.SCHED
>       25261 ±  3%     +16.4%      29411 ±  3%  softirqs.CPU100.SCHED
>       25450           +18.5%      30153 ±  4%  softirqs.CPU101.SCHED
>       25169 ±  2%     +19.3%      30020 ±  3%  softirqs.CPU102.SCHED
>       24805 ±  3%     +20.5%      29893 ±  3%  softirqs.CPU103.SCHED
>       25132 ±  3%     +18.9%      29875 ±  3%  softirqs.CPU104.SCHED
>      177301 ±  2%     +11.3%     197274 ±  4%  softirqs.CPU104.TIMER
>       24869 ±  3%     +18.8%      29537 ±  2%  softirqs.CPU105.SCHED
>       25032 ±  2%     +17.7%      29462 ±  3%  softirqs.CPU106.SCHED
>       24317           +22.3%      29729 ±  3%  softirqs.CPU107.SCHED
>       25241 ±  3%     +18.5%      29917 ±  2%  softirqs.CPU108.SCHED
>       25571 ±  2%     +17.6%      30075 ±  3%  softirqs.CPU109.SCHED
>       25910           +17.9%      30551 ±  3%  softirqs.CPU11.SCHED
>       24927 ±  2%     +18.5%      29538 ±  4%  softirqs.CPU110.SCHED
>       25019 ±  2%     +19.2%      29833 ±  4%  softirqs.CPU111.SCHED
>       25067 ±  2%     +18.7%      29743 ±  4%  softirqs.CPU112.SCHED
>       25105 ±  2%     +19.1%      29892 ±  2%  softirqs.CPU113.SCHED
>       25603 ±  2%     +16.2%      29753 ±  4%  softirqs.CPU114.SCHED
>       25517 ±  2%     +17.9%      30096 ±  3%  softirqs.CPU115.SCHED
>      176346 ±  5%     +11.4%     196390 ±  4%  softirqs.CPU115.TIMER
>       24988 ±  2%     +17.3%      29299 ±  4%  softirqs.CPU116.SCHED
>       24362           +22.1%      29748 ±  2%  softirqs.CPU117.SCHED
>       25420 ±  2%     +18.4%      30101 ±  3%  softirqs.CPU118.SCHED
>       25202           +17.2%      29531 ±  3%  softirqs.CPU119.SCHED
>      246.00 ± 32%   +5886.8%      14727 ±166%  softirqs.CPU12.NET_RX
>       25808 ±  2%     +19.6%      30866 ±  4%  softirqs.CPU12.SCHED
>      161592 ±  4%     +26.1%     203759 ± 17%  softirqs.CPU12.TIMER
>       25550           +16.3%      29721 ±  3%  softirqs.CPU120.SCHED
>      172644 ±  4%     +13.1%     195269 ±  4%  softirqs.CPU120.TIMER
>       24062 ±  5%     +24.3%      29898 ±  3%  softirqs.CPU121.SCHED
>      173612 ±  4%     +12.4%     195124 ±  3%  softirqs.CPU121.TIMER
>       24774 ±  3%     +20.1%      29742 ±  2%  softirqs.CPU122.SCHED
>       25233 ±  2%     +18.4%      29878 ±  2%  softirqs.CPU123.SCHED
>       25038 ±  2%     +18.0%      29548 ±  4%  softirqs.CPU124.SCHED
>       24756           +19.1%      29485 ±  3%  softirqs.CPU125.SCHED
>       25079           +19.2%      29903 ±  3%  softirqs.CPU126.SCHED
>       25110           +17.9%      29598 ±  3%  softirqs.CPU127.SCHED
>       24869 ±  3%     +19.6%      29754 ±  3%  softirqs.CPU128.SCHED
>       25255 ±  2%     +17.6%      29710 ±  3%  softirqs.CPU129.SCHED
>       26257           +22.8%      32254 ±  3%  softirqs.CPU13.SCHED
>       25274 ±  2%     +19.3%      30153 ±  2%  softirqs.CPU130.SCHED
>       25219 ±  2%     +15.4%      29103 ±  2%  softirqs.CPU131.SCHED
>       25034 ±  2%     +19.3%      29869 ±  2%  softirqs.CPU132.SCHED
>       25448           +17.1%      29792        softirqs.CPU133.SCHED
>       25064           +17.5%      29450 ±  4%  softirqs.CPU134.SCHED
>       25561 ±  2%     +16.4%      29742 ±  4%  softirqs.CPU135.SCHED
>       25620           +17.3%      30041 ±  3%  softirqs.CPU136.SCHED
>      174761 ±  6%     +11.1%     194097 ±  7%  softirqs.CPU136.TIMER
>       26235 ±  2%     +14.8%      30124 ±  2%  softirqs.CPU137.SCHED
>      169540 ±  4%     +13.3%     192089 ±  3%  softirqs.CPU137.TIMER
>       25194           +18.3%      29808 ±  3%  softirqs.CPU138.SCHED
>       25038 ±  4%     +19.7%      29982 ±  4%  softirqs.CPU139.SCHED
>       26136 ±  2%     +20.5%      31494 ±  2%  softirqs.CPU14.SCHED
>       25185           +20.2%      30269 ±  3%  softirqs.CPU140.SCHED
>       25032           +20.2%      30098 ±  2%  softirqs.CPU141.SCHED
>       25247 ±  2%     +16.9%      29514        softirqs.CPU142.SCHED
>       24775           +12.9%      27962 ±  6%  softirqs.CPU143.SCHED
>       25196           +19.1%      30016 ±  3%  softirqs.CPU144.SCHED
>       24907 ±  2%     +18.4%      29495 ±  3%  softirqs.CPU145.SCHED
>       25254           +16.3%      29359 ±  4%  softirqs.CPU146.SCHED
>       25123           +19.8%      30089 ±  3%  softirqs.CPU147.SCHED
>       25311 ±  6%     +17.6%      29757 ±  5%  softirqs.CPU148.SCHED
>       25020 ±  2%     +20.4%      30121 ±  2%  softirqs.CPU149.SCHED
>       26013           +19.5%      31094 ±  2%  softirqs.CPU15.SCHED
>       25040           +19.1%      29828 ±  4%  softirqs.CPU150.SCHED
>       25358           +20.1%      30458 ±  2%  softirqs.CPU151.SCHED
>      175966 ±  6%     +11.6%     196369 ±  4%  softirqs.CPU151.TIMER
>       24847 ±  2%     +20.7%      30003 ±  2%  softirqs.CPU152.SCHED
>       24860 ±  4%     +19.9%      29811 ±  3%  softirqs.CPU153.SCHED
>       25102 ±  3%     +18.0%      29611 ±  3%  softirqs.CPU154.SCHED
>       24819           +20.0%      29792 ±  3%  softirqs.CPU155.SCHED
>       25414 ±  2%     +17.3%      29808        softirqs.CPU156.SCHED
>      176417 ±  5%     +15.6%     203967 ±  5%  softirqs.CPU156.TIMER
>       25556           +16.1%      29663 ±  4%  softirqs.CPU157.SCHED
>       25241 ±  2%     +19.9%      30256 ±  3%  softirqs.CPU158.SCHED
>       25462           +17.9%      30017 ±  2%  softirqs.CPU159.SCHED
>       25909           +18.5%      30694 ±  3%  softirqs.CPU16.SCHED
>       25139           +18.1%      29684 ±  4%  softirqs.CPU160.SCHED
>       25431           +17.2%      29812 ±  4%  softirqs.CPU161.SCHED
>       25712           +18.6%      30483 ±  5%  softirqs.CPU162.SCHED
>      172304 ±  4%     +13.1%     194925 ±  4%  softirqs.CPU162.TIMER
>       25736           +16.7%      30040 ±  3%  softirqs.CPU163.SCHED
>      169697 ±  6%     +16.0%     196768 ±  7%  softirqs.CPU163.TIMER
>       25196 ±  2%     +17.4%      29591 ±  2%  softirqs.CPU164.SCHED
>       24976           +19.1%      29753 ±  2%  softirqs.CPU165.SCHED
>       25153 ±  2%     +18.0%      29682 ±  2%  softirqs.CPU166.SCHED
>       24977           +19.7%      29886 ±  2%  softirqs.CPU167.SCHED
>       24784 ±  3%     +18.9%      29457 ±  3%  softirqs.CPU168.SCHED
>       25266 ±  2%     +17.1%      29596 ±  4%  softirqs.CPU169.SCHED
>       25886 ±  2%     +19.3%      30876 ±  3%  softirqs.CPU17.SCHED
>       24920           +17.6%      29295 ±  3%  softirqs.CPU170.SCHED
>       24993           +18.7%      29660 ±  3%  softirqs.CPU171.SCHED
>       25422           +17.1%      29763 ±  2%  softirqs.CPU172.SCHED
>       25491 ±  2%     +17.7%      29994 ±  3%  softirqs.CPU173.SCHED
>       25264 ±  2%     +16.8%      29518 ±  6%  softirqs.CPU174.SCHED
>       25266 ±  2%     +17.9%      29791 ±  2%  softirqs.CPU175.SCHED
>       25231 ±  2%     +17.9%      29754 ±  3%  softirqs.CPU176.SCHED
>      183212 ±  2%     +10.0%     201572 ±  2%  softirqs.CPU176.TIMER
>       25273 ±  2%     +18.3%      29899 ±  3%  softirqs.CPU177.SCHED
>       25155           +19.8%      30145 ±  2%  softirqs.CPU178.SCHED
>       25499 ±  2%     +16.0%      29576 ±  5%  softirqs.CPU179.SCHED
>       26339           +16.3%      30634 ±  2%  softirqs.CPU18.SCHED
>       24757 ±  4%     +19.2%      29500 ±  2%  softirqs.CPU180.SCHED
>       25192 ±  3%     +14.7%      28898 ±  5%  softirqs.CPU181.SCHED
>       25166 ±  2%     +19.2%      29999 ±  2%  softirqs.CPU182.SCHED
>       25259 ±  2%     +18.4%      29910 ±  4%  softirqs.CPU183.SCHED
>       25247 ±  2%     +16.9%      29519 ±  4%  softirqs.CPU184.SCHED
>       25415 ±  2%     +16.9%      29715 ±  2%  softirqs.CPU185.SCHED
>      180421 ±  5%     +11.4%     200946 ±  3%  softirqs.CPU185.TIMER
>       24973 ±  3%     +16.3%      29053        softirqs.CPU186.SCHED
>       25525 ±  2%     +16.1%      29625 ±  2%  softirqs.CPU187.SCHED
>       25461           +18.2%      30088 ±  2%  softirqs.CPU188.SCHED
>       24833           +20.5%      29936 ±  2%  softirqs.CPU189.SCHED
>       25894           +17.1%      30332 ±  4%  softirqs.CPU19.SCHED
>      162878 ±  4%     +18.0%     192168 ± 14%  softirqs.CPU19.TIMER
>       25062           +18.4%      29663 ±  2%  softirqs.CPU190.SCHED
>       24867           +18.3%      29425 ±  3%  softirqs.CPU191.SCHED
>       24926           +18.8%      29620 ±  2%  softirqs.CPU192.SCHED
>      183126 ±  5%     +11.9%     204971 ±  2%  softirqs.CPU192.TIMER
>       24864 ±  2%     +16.2%      28893 ±  4%  softirqs.CPU193.SCHED
>       25242           +17.7%      29702 ±  2%  softirqs.CPU194.SCHED
>       25230 ±  2%     +17.4%      29626 ±  3%  softirqs.CPU195.SCHED
>       24981 ±  2%     +18.8%      29680 ±  2%  softirqs.CPU196.SCHED
>       25020           +17.3%      29348 ±  6%  softirqs.CPU197.SCHED
>       25272           +18.5%      29942 ±  3%  softirqs.CPU198.SCHED
>       25221 ±  2%     +18.8%      29968 ±  3%  softirqs.CPU199.SCHED
>      182100 ±  4%      +8.3%     197211 ±  3%  softirqs.CPU199.TIMER
>       26451           +18.1%      31251 ±  3%  softirqs.CPU2.SCHED
>       26194           +18.4%      31019 ±  2%  softirqs.CPU20.SCHED
>       25645 ±  2%     +17.6%      30168 ±  3%  softirqs.CPU200.SCHED
>       25337           +20.0%      30393 ±  3%  softirqs.CPU201.SCHED
>       25160           +19.0%      29945 ±  2%  softirqs.CPU202.SCHED
>       25322 ±  2%     +18.3%      29944 ±  3%  softirqs.CPU203.SCHED
>       25076           +20.2%      30130 ±  3%  softirqs.CPU204.SCHED
>       25542           +19.2%      30455 ±  2%  softirqs.CPU205.SCHED
>       25245 ±  2%     +16.7%      29455 ±  2%  softirqs.CPU206.SCHED
>       25317 ±  2%     +17.5%      29743 ±  3%  softirqs.CPU207.SCHED
>       25283 ±  2%     +18.5%      29958 ±  4%  softirqs.CPU208.SCHED
>       25533           +12.9%      28839 ±  6%  softirqs.CPU209.SCHED
>      176766 ±  3%     +13.1%     199978 ±  2%  softirqs.CPU209.TIMER
>       26093           +18.6%      30954 ±  3%  softirqs.CPU21.SCHED
>       25426 ±  2%     +17.8%      29963 ±  3%  softirqs.CPU210.SCHED
>       25363 ±  2%     +17.5%      29794        softirqs.CPU211.SCHED
>       25056 ±  2%     +18.3%      29644 ±  5%  softirqs.CPU212.SCHED
>       24915 ±  2%     +18.8%      29602 ±  3%  softirqs.CPU213.SCHED
>       25269 ±  2%     +19.9%      30296 ±  2%  softirqs.CPU214.SCHED
>       24759 ±  2%     +16.4%      28808 ±  5%  softirqs.CPU215.SCHED
>       25090           +19.8%      30061 ±  2%  softirqs.CPU216.SCHED
>       24887 ±  2%     +17.2%      29156        softirqs.CPU217.SCHED
>       25021           +17.2%      29336 ±  4%  softirqs.CPU218.SCHED
>       25320 ±  2%     +17.7%      29806 ±  2%  softirqs.CPU219.SCHED
>       25672 ±  2%     +20.9%      31048        softirqs.CPU22.SCHED
>       25182 ±  2%     +18.8%      29919 ±  3%  softirqs.CPU220.SCHED
>       25152 ±  2%     +20.3%      30248 ±  4%  softirqs.CPU221.SCHED
>       24777           +19.1%      29504 ±  5%  softirqs.CPU222.SCHED
>       25122 ±  3%     +19.6%      30041 ±  2%  softirqs.CPU223.SCHED
>       25079 ±  2%     +19.4%      29954 ±  2%  softirqs.CPU224.SCHED
>       25236 ±  2%     +18.8%      29969 ±  2%  softirqs.CPU225.SCHED
>       25105 ±  3%     +19.0%      29876 ±  2%  softirqs.CPU226.SCHED
>       25609           +15.6%      29613 ±  4%  softirqs.CPU227.SCHED
>      180136 ±  8%     +11.0%     200040 ±  5%  softirqs.CPU227.TIMER
>       25435           +15.9%      29476 ±  2%  softirqs.CPU228.SCHED
>      175813 ±  3%     +16.7%     205108 ±  5%  softirqs.CPU228.TIMER
>       25142 ±  2%     +14.1%      28699 ±  5%  softirqs.CPU229.SCHED
>       25831           +19.2%      30787 ±  2%  softirqs.CPU23.SCHED
>       25460 ±  2%     +18.8%      30242 ±  3%  softirqs.CPU230.SCHED
>       25200 ±  3%     +20.1%      30259 ±  2%  softirqs.CPU231.SCHED
>      168374 ±  3%      +8.7%     183098 ±  3%  softirqs.CPU231.TIMER
>       25379 ±  2%     +17.1%      29711 ±  2%  softirqs.CPU232.SCHED
>       25524 ±  2%     +18.5%      30246 ±  3%  softirqs.CPU233.SCHED
>       25686 ±  2%     +17.5%      30170 ±  3%  softirqs.CPU234.SCHED
>      170330 ±  4%     +11.3%     189542 ±  5%  softirqs.CPU234.TIMER
>       25635 ±  2%     +14.7%      29410 ±  4%  softirqs.CPU235.SCHED
>      176292 ±  7%     +15.9%     204389 ±  6%  softirqs.CPU235.TIMER
>       25549 ±  2%     +16.3%      29718 ±  2%  softirqs.CPU236.SCHED
>      178538 ±  4%      +9.2%     194880 ±  4%  softirqs.CPU236.TIMER
>       25386 ±  2%     +18.4%      30055 ±  2%  softirqs.CPU237.SCHED
>       25191 ±  2%     +17.9%      29707        softirqs.CPU238.SCHED
>       25186 ±  3%     +20.3%      30293        softirqs.CPU239.SCHED
>       25877           +19.0%      30799 ±  3%  softirqs.CPU24.SCHED
>       25506           +17.5%      29967 ±  3%  softirqs.CPU240.SCHED
>       24964 ±  2%     +21.1%      30222 ±  3%  softirqs.CPU241.SCHED
>       25258           +17.7%      29719 ±  2%  softirqs.CPU242.SCHED
>       25162           +18.2%      29740 ±  4%  softirqs.CPU243.SCHED
>       25808           +17.3%      30285 ±  3%  softirqs.CPU244.SCHED
>       25679           +17.4%      30150 ±  3%  softirqs.CPU245.SCHED
>      175664 ±  3%      +8.3%     190327 ±  4%  softirqs.CPU245.TIMER
>       25134 ±  2%     +19.7%      30079 ±  4%  softirqs.CPU246.SCHED
>       24942           +19.9%      29913 ±  3%  softirqs.CPU247.SCHED
>       25395 ±  2%     +17.8%      29906 ±  2%  softirqs.CPU248.SCHED
>      180993 ±  3%     +10.8%     200584 ±  4%  softirqs.CPU248.TIMER
>       25491 ±  2%     +18.0%      30075 ±  4%  softirqs.CPU249.SCHED
>       26099           +17.2%      30595 ±  3%  softirqs.CPU25.SCHED
>       25332           +18.7%      30060 ±  2%  softirqs.CPU250.SCHED
>       25471 ±  2%     +17.4%      29892 ±  3%  softirqs.CPU251.SCHED
>      180242 ±  4%      +9.8%     197819 ±  4%  softirqs.CPU251.TIMER
>       25406 ±  2%     +16.9%      29694 ±  3%  softirqs.CPU252.SCHED
>       25317 ±  3%     +18.2%      29922 ±  3%  softirqs.CPU253.SCHED
>       25103 ±  4%     +19.9%      30106 ±  3%  softirqs.CPU254.SCHED
>       25231 ±  2%     +19.8%      30227 ±  3%  softirqs.CPU255.SCHED
>       25509 ±  2%     +18.2%      30161 ±  3%  softirqs.CPU256.SCHED
>       25559 ±  3%     +16.9%      29888 ±  3%  softirqs.CPU257.SCHED
>       25277 ±  2%     +16.8%      29523 ±  2%  softirqs.CPU258.SCHED
>       24695 ±  5%     +20.4%      29721 ±  2%  softirqs.CPU259.SCHED
>      182550 ±  5%     +10.5%     201781 ±  3%  softirqs.CPU259.TIMER
>       25905           +19.5%      30970 ±  2%  softirqs.CPU26.SCHED
>      164306 ±  3%      +7.6%     176738 ±  5%  softirqs.CPU26.TIMER
>       24829 ±  2%     +20.8%      29990 ±  2%  softirqs.CPU260.SCHED
>       25156           +19.1%      29971 ±  2%  softirqs.CPU261.SCHED
>       25098 ±  3%     +17.6%      29515 ±  4%  softirqs.CPU262.SCHED
>      180807 ±  5%      +9.4%     197856 ±  3%  softirqs.CPU262.TIMER
>       25053 ±  2%     +18.2%      29613 ±  2%  softirqs.CPU263.SCHED
>       25505           +18.9%      30336 ±  3%  softirqs.CPU264.SCHED
>      169958 ±  5%     +11.8%     189979 ±  4%  softirqs.CPU264.TIMER
>       25011 ±  3%     +18.5%      29640 ±  4%  softirqs.CPU265.SCHED
>      169719 ±  4%     +22.2%     207453 ±  8%  softirqs.CPU265.TIMER
>       25489 ±  2%     +18.6%      30231 ±  3%  softirqs.CPU266.SCHED
>       25727           +18.8%      30554 ±  2%  softirqs.CPU267.SCHED
>       25423           +18.1%      30020 ±  2%  softirqs.CPU268.SCHED
>       25451 ±  2%     +19.0%      30292 ±  4%  softirqs.CPU269.SCHED
>       26042           +18.5%      30852 ±  3%  softirqs.CPU27.SCHED
>      163812 ±  3%      +8.6%     177825 ±  5%  softirqs.CPU27.TIMER
>       25649 ±  2%     +18.3%      30337 ±  3%  softirqs.CPU270.SCHED
>       25318           +19.3%      30212 ±  3%  softirqs.CPU271.SCHED
>       25592 ±  2%     +19.6%      30603 ±  3%  softirqs.CPU272.SCHED
>       25440 ±  2%     +20.0%      30520 ±  3%  softirqs.CPU273.SCHED
>       26007           +18.2%      30741 ±  2%  softirqs.CPU274.SCHED
>       25661           +18.7%      30454 ±  3%  softirqs.CPU275.SCHED
>       25227           +21.0%      30536 ±  2%  softirqs.CPU276.SCHED
>       25923           +17.9%      30560 ±  3%  softirqs.CPU277.SCHED
>       25973           +14.6%      29757 ±  5%  softirqs.CPU278.SCHED
>       25905 ±  2%     +16.2%      30089 ±  3%  softirqs.CPU279.SCHED
>       26091           +18.3%      30875 ±  3%  softirqs.CPU28.SCHED
>       25736 ±  2%     +17.4%      30217 ±  4%  softirqs.CPU280.SCHED
>       25964           +13.5%      29482 ±  6%  softirqs.CPU281.SCHED
>      170413 ±  4%     +11.8%     190550 ±  4%  softirqs.CPU281.TIMER
>       25669           +18.8%      30484 ±  3%  softirqs.CPU282.SCHED
>       25462 ±  2%     +18.6%      30208        softirqs.CPU283.SCHED
>      174059 ±  6%     +10.6%     192446 ±  5%  softirqs.CPU283.TIMER
>       25578           +18.4%      30287 ±  3%  softirqs.CPU284.SCHED
>       25703           +19.1%      30617 ±  3%  softirqs.CPU285.SCHED
>       25775 ±  2%     +16.0%      29891 ±  2%  softirqs.CPU286.SCHED
>       25175 ±  2%     +16.0%      29204 ±  4%  softirqs.CPU287.SCHED
>       26076           +17.7%      30684 ±  3%  softirqs.CPU29.SCHED
>       26116           +18.8%      31032 ±  2%  softirqs.CPU3.SCHED
>       26043           +18.2%      30789 ±  3%  softirqs.CPU30.SCHED
>       25884           +18.5%      30681 ±  2%  softirqs.CPU31.SCHED
>      167289 ±  2%      +6.3%     177832 ±  6%  softirqs.CPU31.TIMER
>       26095           +17.1%      30563 ±  3%  softirqs.CPU32.SCHED
>      161186 ±  2%     +10.6%     178192 ±  4%  softirqs.CPU32.TIMER
>       26124           +18.2%      30887 ±  3%  softirqs.CPU33.SCHED
>       25966           +18.0%      30651 ±  2%  softirqs.CPU34.SCHED
>      167402 ±  4%      +6.6%     178425 ±  4%  softirqs.CPU34.TIMER
>       25944           +18.0%      30622 ±  3%  softirqs.CPU35.SCHED
>       26167           +17.9%      30861 ±  2%  softirqs.CPU36.SCHED
>       25900           +20.7%      31271 ±  3%  softirqs.CPU37.SCHED
>       25475 ±  3%     +21.1%      30843 ±  3%  softirqs.CPU38.SCHED
>       25642 ±  2%     +19.4%      30627 ±  3%  softirqs.CPU39.SCHED
>       25494 ±  2%     +20.7%      30774 ±  2%  softirqs.CPU4.SCHED
>       25610           +18.1%      30236 ±  4%  softirqs.CPU40.SCHED
>       26064 ±  3%     +17.2%      30546 ±  3%  softirqs.CPU41.SCHED
>       25739 ±  2%     +15.2%      29643 ±  4%  softirqs.CPU42.SCHED
>       25730 ±  2%     +18.5%      30493 ±  3%  softirqs.CPU43.SCHED
>      175037 ±  6%     +10.6%     193664 ±  4%  softirqs.CPU43.TIMER
>       25729           +18.4%      30470 ±  3%  softirqs.CPU44.SCHED
>       25114 ±  2%     +21.3%      30475 ±  2%  softirqs.CPU45.SCHED
>       25415 ±  3%     +20.1%      30511 ±  2%  softirqs.CPU46.SCHED
>       25558           +17.4%      29994        softirqs.CPU47.SCHED
>       25624           +16.7%      29897 ±  4%  softirqs.CPU48.SCHED
>      167429 ±  5%     +13.0%     189268 ±  5%  softirqs.CPU48.TIMER
>       25871           +17.3%      30344 ±  3%  softirqs.CPU49.SCHED
>      167694 ±  3%     +12.5%     188683 ±  4%  softirqs.CPU49.TIMER
>       25681           +20.0%      30819 ±  3%  softirqs.CPU5.SCHED
>       25661           +17.8%      30220        softirqs.CPU50.SCHED
>       25857           +17.6%      30408 ±  3%  softirqs.CPU51.SCHED
>       25943           +17.7%      30544 ±  3%  softirqs.CPU52.SCHED
>       25398 ±  2%     +19.6%      30387 ±  3%  softirqs.CPU53.SCHED
>       25599 ±  2%     +18.9%      30428 ±  3%  softirqs.CPU54.SCHED
>       25454 ±  2%     +20.5%      30663 ±  3%  softirqs.CPU55.SCHED
>       25603 ±  2%     +18.6%      30374 ±  3%  softirqs.CPU56.SCHED
>       25665           +18.7%      30453 ±  3%  softirqs.CPU57.SCHED
>       25853           +18.9%      30740 ±  2%  softirqs.CPU58.SCHED
>       25916           +17.5%      30454 ±  3%  softirqs.CPU59.SCHED
>       26202           +16.7%      30570 ±  3%  softirqs.CPU6.SCHED
>       25549 ±  2%     +20.5%      30778 ±  2%  softirqs.CPU60.SCHED
>       25854           +18.5%      30626 ±  2%  softirqs.CPU61.SCHED
>       25516 ±  2%     +17.1%      29871 ±  4%  softirqs.CPU62.SCHED
>       25833           +17.2%      30284 ±  3%  softirqs.CPU63.SCHED
>       25510 ±  3%     +19.2%      30420 ±  3%  softirqs.CPU64.SCHED
>       26177           +16.5%      30487 ±  3%  softirqs.CPU65.SCHED
>      166747 ±  3%     +11.5%     185900 ±  3%  softirqs.CPU65.TIMER
>       25637 ±  2%     +19.0%      30503 ±  3%  softirqs.CPU66.SCHED
>       25644 ±  2%     +15.4%      29588        softirqs.CPU67.SCHED
>       25796           +18.6%      30585 ±  3%  softirqs.CPU68.SCHED
>       25503           +20.2%      30654 ±  3%  softirqs.CPU69.SCHED
>       25912           +16.4%      30171 ±  4%  softirqs.CPU7.SCHED
>       25994 ±  6%     +16.2%      30203        softirqs.CPU70.SCHED
>       25255           +17.9%      29766 ±  3%  softirqs.CPU71.SCHED
>       25057           +19.1%      29854 ±  3%  softirqs.CPU72.SCHED
>       24925 ±  2%     +18.7%      29588 ±  2%  softirqs.CPU73.SCHED
>       25774           +16.1%      29929 ±  3%  softirqs.CPU74.SCHED
>      177035 ±  2%      +7.9%     190942 ±  4%  softirqs.CPU74.TIMER
>       25433           +19.2%      30327 ±  3%  softirqs.CPU75.SCHED
>       25274 ±  2%     +15.5%      29184 ±  3%  softirqs.CPU76.SCHED
>       25297           +17.7%      29765 ±  4%  softirqs.CPU77.SCHED
>       25557 ±  2%     +18.2%      30203 ±  3%  softirqs.CPU78.SCHED
>       25604 ±  2%     +17.3%      30034 ±  3%  softirqs.CPU79.SCHED
>      175758 ±  6%      +9.8%     193033 ±  3%  softirqs.CPU79.TIMER
>       26086           +18.4%      30878 ±  2%  softirqs.CPU8.SCHED
>       25297           +18.6%      30013 ±  3%  softirqs.CPU80.SCHED
>       25450           +17.8%      29987 ±  2%  softirqs.CPU81.SCHED
>       25125 ±  3%     +17.0%      29404 ±  3%  softirqs.CPU82.SCHED
>       24840 ±  2%     +18.7%      29490 ±  6%  softirqs.CPU83.SCHED
>       25920           +14.9%      29770 ±  2%  softirqs.CPU84.SCHED
>      168708 ±  5%     +18.4%     199703 ±  7%  softirqs.CPU84.TIMER
>       24980 ±  3%     +19.5%      29850 ±  3%  softirqs.CPU85.SCHED
>       25338           +18.6%      30042 ±  2%  softirqs.CPU86.SCHED
>       25404 ±  2%     +18.6%      30142 ±  2%  softirqs.CPU87.SCHED
>      177099 ±  3%      +9.0%     193072        softirqs.CPU87.TIMER
>       24855 ±  2%     +20.7%      30000 ±  3%  softirqs.CPU88.SCHED
>       25532 ±  2%     +17.2%      29916 ±  3%  softirqs.CPU89.SCHED
>       26321 ±  2%     +14.7%      30186 ±  4%  softirqs.CPU9.SCHED
>       25622 ±  2%     +17.4%      30077 ±  3%  softirqs.CPU90.SCHED
>      177491 ±  4%     +10.9%     196822 ±  4%  softirqs.CPU90.TIMER
>       25638 ±  2%     +14.0%      29228 ±  5%  softirqs.CPU91.SCHED
>      175040 ±  5%     +16.7%     204221 ±  6%  softirqs.CPU91.TIMER
>       25387 ±  2%     +18.0%      29968 ±  2%  softirqs.CPU92.SCHED
>       25316 ±  2%     +19.5%      30262 ±  2%  softirqs.CPU93.SCHED
>       25125 ±  2%     +19.6%      30051        softirqs.CPU94.SCHED
>       25303 ±  2%     +18.1%      29875 ±  2%  softirqs.CPU95.SCHED
>       25481           +18.3%      30133 ±  3%  softirqs.CPU96.SCHED
>       25533           +18.1%      30146 ±  3%  softirqs.CPU97.SCHED
>       24892 ±  3%     +20.4%      29982 ±  3%  softirqs.CPU98.SCHED
>       24611           +20.3%      29607 ±  3%  softirqs.CPU99.SCHED
>     7325587           +18.2%    8660780 ±  3%  softirqs.SCHED
> 
> 
>                                                                                  
>                             will-it-scale.per_thread_ops
>                                                                                  
>    1500 +--------------------------------------------------------------------+
>    1450 |+. +.  .+  +++. +.  .+++. +.  .++. +                                |
>         |  +  ++        +  ++     +  ++    + +   .++.+                       |
>    1400 |-+                                   : +     +                      |
>    1350 |-+                                   : :                     OO     |
>    1300 |-+                                    +                    O        |
>    1250 |-+                                       OO            O OO     OO O|
>         |               O  OO O O OO OO O                                    |
>    1200 |-+              O     O                      OO OO OO O             |
>    1150 |-+                                    O                             |
>    1100 |-+                              O O    O    O                       |
>    1050 |O+O      O O O                      O                               |
>         |     O  O   O                      O                                |
>    1000 |-+ O  O                                                             |
>     950 +--------------------------------------------------------------------+
>                                                                                  
>                                                                                  
> [*] bisect-good sample
> [O] bisect-bad  sample
> 
> 
> 
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
> 
> 
> Thanks,
> Rong Chen
> 
> 
> _______________________________________________
> LKP mailing list -- lkp@...ts.01.org
> To unsubscribe send an email to lkp-leave@...ts.01.org
> 

-- 
Zhengjun Xing

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ