lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Sat, 27 Jun 2020 16:22:22 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Yonghong Song <yhs@...com>
Cc:     Alexei Starovoitov <ast@...nel.org>,
        Andrii Nakryiko <andriin@...com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [bpf] will-it-scale.per_thread_ops 2.6% improvement

Greeting,

FYI, we noticed a 2.6% improvement of will-it-scale.per_thread_ops due to commit:


commit: 492e639f0c222784e2e0f121966375f641c61b15 ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: will-it-scale
on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
with following parameters:

	nr_task: 100%
	mode: thread
	test: pthread_mutex2
	cpufreq_governor: performance
	ucode: 0x11

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops 2.5% improvement        |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | mode=process                                                         |
|                  | nr_task=100%                                                         |
|                  | test=malloc2                                                         |
|                  | ucode=0x11                                                           |
+------------------+----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/pthread_mutex2/will-it-scale/0x11

commit: 
  b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support")
  492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")

b121b341e5983bdc 492e639f0c222784e2e0f121966 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   6105969            +2.6%    6265564        will-it-scale.per_thread_ops
      9855         +1196.5%     127775 ± 92%  will-it-scale.time.maximum_resident_set_size
 1.759e+09            +2.6%  1.804e+09        will-it-scale.workload
    330.50 ± 13%     +71.4%     566.50 ± 21%  cpuidle.POLL.usage
    149.50            +3.3%     154.50 ±  2%  proc-vmstat.nr_anon_transparent_hugepages
     24750            +2.5%      25362        proc-vmstat.nr_shmem
     22056 ±  6%     -21.5%      17321 ± 27%  proc-vmstat.numa_hint_faults
     22056 ±  6%     -21.5%      17321 ± 27%  proc-vmstat.numa_hint_faults_local
    451.64 ± 11%     +27.3%     574.96 ± 21%  sched_debug.cfs_rq:/.util_avg.min
      0.46 ± 18%     -58.7%       0.19 ±105%  sched_debug.rt_rq:/.rt_time.avg
    133.72 ± 18%     -58.7%      55.26 ±105%  sched_debug.rt_rq:/.rt_time.max
      7.87 ± 18%     -58.7%       3.25 ±105%  sched_debug.rt_rq:/.rt_time.stddev
    123790 ± 14%     -13.1%     107602 ±  3%  softirqs.CPU100.TIMER
    124472 ± 14%     -13.0%     108248 ±  5%  softirqs.CPU142.TIMER
    125239 ± 14%     -12.4%     109741 ±  5%  softirqs.CPU166.TIMER
    130036 ± 11%     -15.7%     109609 ±  3%  softirqs.CPU170.TIMER
    123083 ± 12%     -14.6%     105115 ±  3%  softirqs.CPU207.TIMER
    137740 ± 13%     -21.5%     108150 ±  4%  softirqs.CPU209.TIMER
    126203 ± 11%     -11.0%     112261 ±  2%  softirqs.CPU213.TIMER
    126552 ± 13%     -14.7%     107985 ±  2%  softirqs.CPU226.TIMER
    126084 ± 11%     -13.6%     108956 ±  4%  softirqs.CPU73.TIMER
 3.162e+10            +2.8%  3.252e+10        perf-stat.i.branch-instructions
     22.58            +0.3       22.90        perf-stat.i.branch-miss-rate%
 7.165e+09            +4.3%   7.47e+09        perf-stat.i.branch-misses
  82101720            +1.4%   83214202        perf-stat.i.cache-references
 4.361e+11            +2.2%  4.455e+11        perf-stat.i.cpu-cycles
     38454 ±  2%      +5.2%      40469 ±  2%  perf-stat.i.cycles-between-cache-misses
      3.53            +0.0        3.57        perf-stat.i.iTLB-load-miss-rate%
 3.786e+09            +3.8%  3.931e+09        perf-stat.i.iTLB-load-misses
 1.032e+11            +2.4%  1.057e+11        perf-stat.i.iTLB-loads
 1.028e+11            +2.7%  1.055e+11        perf-stat.i.instructions
     27.45            -1.3%      27.08        perf-stat.i.instructions-per-iTLB-miss
      1.51            +2.2%       1.55        perf-stat.i.metric.GHz
    468.30            +2.5%     480.18        perf-stat.i.metric.M/sec
     22.69            +0.3       23.00        perf-stat.overall.branch-miss-rate%
     37831 ±  2%      +5.2%      39799        perf-stat.overall.cycles-between-cache-misses
      3.54            +0.0        3.59        perf-stat.overall.iTLB-load-miss-rate%
     27.14            -1.1%      26.84        perf-stat.overall.instructions-per-iTLB-miss
 3.152e+10            +2.8%  3.241e+10        perf-stat.ps.branch-instructions
 7.152e+09            +4.2%  7.454e+09        perf-stat.ps.branch-misses
  82017742            +1.3%   83120324        perf-stat.ps.cache-references
 4.348e+11            +2.1%  4.441e+11        perf-stat.ps.cpu-cycles
 3.775e+09            +3.8%  3.919e+09        perf-stat.ps.iTLB-load-misses
 1.028e+11            +2.4%  1.053e+11        perf-stat.ps.iTLB-loads
 1.025e+11            +2.6%  1.052e+11        perf-stat.ps.instructions
 3.122e+13            +2.5%    3.2e+13        perf-stat.total.instructions
     10.82            -0.1       10.70        perf-profile.calltrace.cycles-pp.pthread_mutex_unlock
      0.72 ± 10%      +0.2        0.89 ± 10%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
      0.79 ± 11%      +0.2        0.96 ± 11%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt
      0.84 ± 11%      +0.3        1.09 ± 10%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt
      0.48 ± 58%      +0.3        0.76 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.11 ± 11%      +0.3        1.39 ± 10%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt
      1.34 ± 10%      +0.3        1.64 ± 11%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt
      1.47 ± 10%      +0.3        1.78 ± 11%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.__pthread_mutex_unlock_usercnt
      0.14 ±173%      +0.5        0.60 ±  9%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.64 ± 12%      +0.5        2.15 ± 10%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock
      2.17 ± 11%      +0.6        2.74 ± 10%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock
      2.62 ± 10%      +0.6        3.24 ± 11%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock
      2.89 ± 10%      +0.6        3.51 ± 11%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.pthread_mutex_lock
      0.14 ±173%      +0.7        0.86 ± 10%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.47 ± 59%      +1.4        1.83 ± 10%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.48 ± 59%      +1.4        1.90 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.67 ± 17%      +1.5        2.19 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.74 ± 12%      -0.6        0.17 ± 10%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.23 ±  7%      -0.2        0.08 ± 20%  perf-profile.children.cycles-pp.rb_next
      0.29 ±  7%      -0.2        0.14 ± 11%  perf-profile.children.cycles-pp.timerqueue_del
      0.41 ± 10%      -0.1        0.32 ± 12%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      8.81            -0.1        8.72        perf-profile.children.cycles-pp.pthread_mutex_unlock
      0.11 ± 18%      -0.0        0.07 ± 30%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.11 ± 10%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.timerqueue_add
      0.12 ± 11%      +0.0        0.15 ±  5%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.17 ± 14%      +0.0        0.21 ± 12%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.07 ± 15%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.06 ± 20%      +0.0        0.11 ± 10%  perf-profile.children.cycles-pp.cpuacct_charge
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.sched_clock
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.__calc_delta
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.raise_softirq
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.cpuacct_account_field
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.reweight_entity
      0.01 ±173%      +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.05 ± 58%      +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.run_timer_softirq
      0.01 ±173%      +0.1        0.08 ± 29%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.hrtimer_active
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.x86_pmu_disable
      0.14 ± 15%      +0.1        0.22 ±  8%  perf-profile.children.cycles-pp.clockevents_program_event
      0.09 ± 15%      +0.1        0.18 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.06 ± 60%      +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.account_user_time
      0.09 ± 19%      +0.1        0.18 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock
      0.09 ± 20%      +0.1        0.18 ± 10%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.01 ±173%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.profile_tick
      0.14 ± 12%      +0.1        0.25 ± 11%  perf-profile.children.cycles-pp.ktime_get
      0.08 ± 21%      +0.1        0.21 ± 13%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.24 ± 17%      +0.1        0.37 ±  7%  perf-profile.children.cycles-pp.update_curr
      0.16 ± 11%      +0.1        0.30 ± 18%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.01 ±173%      +0.2        0.17 ± 15%  perf-profile.children.cycles-pp.run_local_timers
      0.04 ± 59%      +0.2        0.20 ± 20%  perf-profile.children.cycles-pp.update_rq_clock
      0.26 ± 12%      +0.2        0.43 ± 17%  perf-profile.children.cycles-pp.irq_exit
      0.59 ± 17%      +0.5        1.14 ±  8%  perf-profile.children.cycles-pp.task_tick_fair
      0.88 ± 18%      +0.9        1.76 ±  9%  perf-profile.children.cycles-pp.scheduler_tick
      3.39 ± 12%      +1.1        4.47 ±  9%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.47 ± 11%      +1.2        5.63 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
      5.38 ± 10%      +1.2        6.62 ± 11%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      5.93 ± 10%      +1.3        7.19 ± 11%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.19 ± 18%      +1.3        2.48 ± 10%  perf-profile.children.cycles-pp.update_process_times
      1.23 ± 18%      +1.4        2.58 ± 10%  perf-profile.children.cycles-pp.tick_sched_handle
      1.39 ± 17%      +1.6        2.96 ± 10%  perf-profile.children.cycles-pp.tick_sched_timer
      0.22 ±  6%      -0.1        0.07 ± 21%  perf-profile.self.cycles-pp.rb_next
      5.98            -0.1        5.85        perf-profile.self.cycles-pp.pthread_mutex_unlock
      0.25 ± 10%      -0.1        0.14 ± 15%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.11 ± 18%      -0.1        0.06 ± 65%  perf-profile.self.cycles-pp.rcu_irq_enter
      0.04 ± 58%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.prepare_exit_to_usermode
      0.07 ± 10%      +0.0        0.11 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.15 ± 16%      +0.0        0.19 ±  6%  perf-profile.self.cycles-pp.update_curr
      0.06 ± 20%      +0.0        0.11 ± 10%  perf-profile.self.cycles-pp.cpuacct_charge
      0.04 ± 58%      +0.1        0.09 ± 19%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.raise_softirq
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.cpuacct_account_field
      0.03 ±100%      +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.run_timer_softirq
      0.01 ±173%      +0.1        0.08 ± 21%  perf-profile.self.cycles-pp.account_user_time
      0.08 ± 17%      +0.1        0.15 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.01 ±173%      +0.1        0.08 ± 29%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.hrtimer_active
      0.00            +0.1        0.08 ± 24%  perf-profile.self.cycles-pp.x86_pmu_disable
      0.07 ± 20%      +0.1        0.15 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.09 ± 19%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.01 ±173%      +0.1        0.10 ± 14%  perf-profile.self.cycles-pp.profile_tick
      0.04 ± 58%      +0.1        0.13 ± 12%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.03 ±102%      +0.1        0.12 ± 13%  perf-profile.self.cycles-pp.tick_sched_timer
      0.11 ± 11%      +0.1        0.21 ± 12%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.1        0.13 ± 16%  perf-profile.self.cycles-pp.run_local_timers
      0.00            +0.1        0.14 ± 22%  perf-profile.self.cycles-pp.update_rq_clock
      0.11 ± 20%      +0.1        0.25 ± 10%  perf-profile.self.cycles-pp.task_tick_fair
      3759           +52.3%       5725 ± 33%  interrupts.CPU0.NMI:Non-maskable_interrupts
      3759           +52.3%       5725 ± 33%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      4752 ± 34%     +40.9%       6695 ± 24%  interrupts.CPU1.NMI:Non-maskable_interrupts
      4752 ± 34%     +40.9%       6695 ± 24%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
    607.50 ± 12%     +38.6%     842.25 ± 12%  interrupts.CPU10.RES:Rescheduling_interrupts
      4635 ± 34%     +62.9%       7551        interrupts.CPU104.NMI:Non-maskable_interrupts
      4635 ± 34%     +62.9%       7551        interrupts.CPU104.PMI:Performance_monitoring_interrupts
    896.00 ± 40%     -45.1%     492.25 ± 79%  interrupts.CPU106.RES:Rescheduling_interrupts
     28.50 ± 54%    +860.5%     273.75 ± 64%  interrupts.CPU107.RES:Rescheduling_interrupts
    115.00 ± 48%     -64.8%      40.50 ± 21%  interrupts.CPU119.RES:Rescheduling_interrupts
      4633 ± 34%     +22.5%       5675 ± 33%  interrupts.CPU124.NMI:Non-maskable_interrupts
      4633 ± 34%     +22.5%       5675 ± 33%  interrupts.CPU124.PMI:Performance_monitoring_interrupts
      3724           +52.3%       5673 ± 33%  interrupts.CPU129.NMI:Non-maskable_interrupts
      3724           +52.3%       5673 ± 33%  interrupts.CPU129.PMI:Performance_monitoring_interrupts
      3705           +54.9%       5741 ± 33%  interrupts.CPU13.NMI:Non-maskable_interrupts
      3705           +54.9%       5741 ± 33%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    537.50 ± 57%    +136.8%       1273 ± 53%  interrupts.CPU130.RES:Rescheduling_interrupts
      4635 ± 34%     +22.8%       5692 ± 33%  interrupts.CPU131.NMI:Non-maskable_interrupts
      4635 ± 34%     +22.8%       5692 ± 33%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
      4648 ± 34%     +42.1%       6603 ± 24%  interrupts.CPU147.NMI:Non-maskable_interrupts
      4648 ± 34%     +42.1%       6603 ± 24%  interrupts.CPU147.PMI:Performance_monitoring_interrupts
      3720           +52.9%       5689 ± 33%  interrupts.CPU148.NMI:Non-maskable_interrupts
      3720           +52.9%       5689 ± 33%  interrupts.CPU148.PMI:Performance_monitoring_interrupts
      3725           +52.9%       5695 ± 33%  interrupts.CPU149.NMI:Non-maskable_interrupts
      3725           +52.9%       5695 ± 33%  interrupts.CPU149.PMI:Performance_monitoring_interrupts
      4626 ± 34%     +23.3%       5704 ± 33%  interrupts.CPU152.NMI:Non-maskable_interrupts
      4626 ± 34%     +23.3%       5704 ± 33%  interrupts.CPU152.PMI:Performance_monitoring_interrupts
      4657 ± 34%     +42.4%       6630 ± 24%  interrupts.CPU154.NMI:Non-maskable_interrupts
      4657 ± 34%     +42.4%       6630 ± 24%  interrupts.CPU154.PMI:Performance_monitoring_interrupts
    260.00 ± 59%     -55.9%     114.75 ± 31%  interrupts.CPU156.RES:Rescheduling_interrupts
      3729           +77.2%       6608 ± 24%  interrupts.CPU163.NMI:Non-maskable_interrupts
      3729           +77.2%       6608 ± 24%  interrupts.CPU163.PMI:Performance_monitoring_interrupts
      4634 ± 34%     +22.6%       5681 ± 33%  interrupts.CPU164.NMI:Non-maskable_interrupts
      4634 ± 34%     +22.6%       5681 ± 33%  interrupts.CPU164.PMI:Performance_monitoring_interrupts
     95.25 ± 48%    +415.7%     491.25 ± 64%  interrupts.CPU164.RES:Rescheduling_interrupts
      7422           -48.8%       3798        interrupts.CPU168.NMI:Non-maskable_interrupts
      7422           -48.8%       3798        interrupts.CPU168.PMI:Performance_monitoring_interrupts
      3717           +52.6%       5672 ± 33%  interrupts.CPU173.NMI:Non-maskable_interrupts
      3717           +52.6%       5672 ± 33%  interrupts.CPU173.PMI:Performance_monitoring_interrupts
      4682 ± 34%     +21.4%       5682 ± 33%  interrupts.CPU179.NMI:Non-maskable_interrupts
      4682 ± 34%     +21.4%       5682 ± 33%  interrupts.CPU179.PMI:Performance_monitoring_interrupts
      4675 ± 33%     +23.6%       5778 ± 33%  interrupts.CPU19.NMI:Non-maskable_interrupts
      4675 ± 33%     +23.6%       5778 ± 33%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
      3718           +52.6%       5675 ± 33%  interrupts.CPU197.NMI:Non-maskable_interrupts
      3718           +52.6%       5675 ± 33%  interrupts.CPU197.PMI:Performance_monitoring_interrupts
      3801           +77.3%       6740 ± 24%  interrupts.CPU2.NMI:Non-maskable_interrupts
      3801           +77.3%       6740 ± 24%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      3702           +78.6%       6612 ± 24%  interrupts.CPU204.NMI:Non-maskable_interrupts
      3702           +78.6%       6612 ± 24%  interrupts.CPU204.PMI:Performance_monitoring_interrupts
      3701           +52.9%       5660 ± 33%  interrupts.CPU205.NMI:Non-maskable_interrupts
      3701           +52.9%       5660 ± 33%  interrupts.CPU205.PMI:Performance_monitoring_interrupts
    156.50 ± 49%    +160.7%     408.00 ± 18%  interrupts.CPU206.RES:Rescheduling_interrupts
      4657 ± 34%     +21.6%       5662 ± 33%  interrupts.CPU208.NMI:Non-maskable_interrupts
      4657 ± 34%     +21.6%       5662 ± 33%  interrupts.CPU208.PMI:Performance_monitoring_interrupts
      3725           +51.9%       5659 ± 33%  interrupts.CPU209.NMI:Non-maskable_interrupts
      3725           +51.9%       5659 ± 33%  interrupts.CPU209.PMI:Performance_monitoring_interrupts
      4616 ± 34%     +23.2%       5684 ± 33%  interrupts.CPU210.NMI:Non-maskable_interrupts
      4616 ± 34%     +23.2%       5684 ± 33%  interrupts.CPU210.PMI:Performance_monitoring_interrupts
      4632 ± 34%     +43.5%       6645 ± 24%  interrupts.CPU211.NMI:Non-maskable_interrupts
      4632 ± 34%     +43.5%       6645 ± 24%  interrupts.CPU211.PMI:Performance_monitoring_interrupts
      4570 ± 34%     +43.5%       6557 ± 24%  interrupts.CPU214.NMI:Non-maskable_interrupts
      4570 ± 34%     +43.5%       6557 ± 24%  interrupts.CPU214.PMI:Performance_monitoring_interrupts
      3698           +54.0%       5695 ± 33%  interrupts.CPU217.NMI:Non-maskable_interrupts
      3698           +54.0%       5695 ± 33%  interrupts.CPU217.PMI:Performance_monitoring_interrupts
      4659 ± 34%     +42.1%       6622 ± 24%  interrupts.CPU218.NMI:Non-maskable_interrupts
      4659 ± 34%     +42.1%       6622 ± 24%  interrupts.CPU218.PMI:Performance_monitoring_interrupts
      4647 ± 35%     +42.8%       6636 ± 24%  interrupts.CPU219.NMI:Non-maskable_interrupts
      4647 ± 35%     +42.8%       6636 ± 24%  interrupts.CPU219.PMI:Performance_monitoring_interrupts
     46.75 ± 39%    +387.2%     227.75 ± 78%  interrupts.CPU222.RES:Rescheduling_interrupts
     18.00 ±102%    +212.5%      56.25 ± 71%  interrupts.CPU225.RES:Rescheduling_interrupts
      4667 ± 34%     +21.4%       5668 ± 33%  interrupts.CPU231.NMI:Non-maskable_interrupts
      4667 ± 34%     +21.4%       5668 ± 33%  interrupts.CPU231.PMI:Performance_monitoring_interrupts
      4660 ± 34%     +21.8%       5674 ± 33%  interrupts.CPU232.NMI:Non-maskable_interrupts
      4660 ± 34%     +21.8%       5674 ± 33%  interrupts.CPU232.PMI:Performance_monitoring_interrupts
      4633 ± 34%     +22.6%       5679 ± 33%  interrupts.CPU237.NMI:Non-maskable_interrupts
      4633 ± 34%     +22.6%       5679 ± 33%  interrupts.CPU237.PMI:Performance_monitoring_interrupts
      4646 ± 34%     +21.3%       5638 ± 33%  interrupts.CPU238.NMI:Non-maskable_interrupts
      4646 ± 34%     +21.3%       5638 ± 33%  interrupts.CPU238.PMI:Performance_monitoring_interrupts
      3719           +52.1%       5657 ± 33%  interrupts.CPU245.NMI:Non-maskable_interrupts
      3719           +52.1%       5657 ± 33%  interrupts.CPU245.PMI:Performance_monitoring_interrupts
      4635 ± 34%     +22.1%       5660 ± 32%  interrupts.CPU255.NMI:Non-maskable_interrupts
      4635 ± 34%     +22.1%       5660 ± 32%  interrupts.CPU255.PMI:Performance_monitoring_interrupts
     54.50 ± 28%    +242.2%     186.50 ± 54%  interrupts.CPU256.RES:Rescheduling_interrupts
     39.50 ± 19%    +162.0%     103.50 ± 45%  interrupts.CPU260.RES:Rescheduling_interrupts
      4662 ± 34%     +22.9%       5730 ± 33%  interrupts.CPU267.NMI:Non-maskable_interrupts
      4662 ± 34%     +22.9%       5730 ± 33%  interrupts.CPU267.PMI:Performance_monitoring_interrupts
      4613 ± 34%     +24.1%       5724 ± 33%  interrupts.CPU269.NMI:Non-maskable_interrupts
      4613 ± 34%     +24.1%       5724 ± 33%  interrupts.CPU269.PMI:Performance_monitoring_interrupts
      4669 ± 34%     +22.6%       5723 ± 33%  interrupts.CPU27.NMI:Non-maskable_interrupts
      4669 ± 34%     +22.6%       5723 ± 33%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      4616 ± 34%     +24.1%       5727 ± 33%  interrupts.CPU274.NMI:Non-maskable_interrupts
      4616 ± 34%     +24.1%       5727 ± 33%  interrupts.CPU274.PMI:Performance_monitoring_interrupts
      3732           +52.4%       5689 ± 33%  interrupts.CPU276.NMI:Non-maskable_interrupts
      3732           +52.4%       5689 ± 33%  interrupts.CPU276.PMI:Performance_monitoring_interrupts
      3718           +53.7%       5713 ± 33%  interrupts.CPU277.NMI:Non-maskable_interrupts
      3718           +53.7%       5713 ± 33%  interrupts.CPU277.PMI:Performance_monitoring_interrupts
      4639 ± 34%     +44.0%       6682 ± 24%  interrupts.CPU278.NMI:Non-maskable_interrupts
      4639 ± 34%     +44.0%       6682 ± 24%  interrupts.CPU278.PMI:Performance_monitoring_interrupts
      3707           +79.7%       6663 ± 24%  interrupts.CPU279.NMI:Non-maskable_interrupts
      3707           +79.7%       6663 ± 24%  interrupts.CPU279.PMI:Performance_monitoring_interrupts
    133.50 ± 44%     -80.1%      26.50 ± 36%  interrupts.CPU279.RES:Rescheduling_interrupts
      3722           +79.1%       6668 ± 24%  interrupts.CPU281.NMI:Non-maskable_interrupts
      3722           +79.1%       6668 ± 24%  interrupts.CPU281.PMI:Performance_monitoring_interrupts
      3726           +53.5%       5719 ± 33%  interrupts.CPU283.NMI:Non-maskable_interrupts
      3726           +53.5%       5719 ± 33%  interrupts.CPU283.PMI:Performance_monitoring_interrupts
      4642 ± 34%     +43.8%       6678 ± 24%  interrupts.CPU285.NMI:Non-maskable_interrupts
      4642 ± 34%     +43.8%       6678 ± 24%  interrupts.CPU285.PMI:Performance_monitoring_interrupts
      3775           +77.7%       6709 ± 24%  interrupts.CPU32.NMI:Non-maskable_interrupts
      3775           +77.7%       6709 ± 24%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      4681 ± 33%     +22.9%       5755 ± 33%  interrupts.CPU34.NMI:Non-maskable_interrupts
      4681 ± 33%     +22.9%       5755 ± 33%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
    354.25 ±  9%     +30.3%     461.75 ± 20%  interrupts.CPU36.RES:Rescheduling_interrupts
    337.75 ±  4%     +30.6%     441.25 ± 19%  interrupts.CPU38.RES:Rescheduling_interrupts
      4633 ± 34%     +23.7%       5729 ± 33%  interrupts.CPU48.NMI:Non-maskable_interrupts
      4633 ± 34%     +23.7%       5729 ± 33%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    233.50 ± 18%     +47.9%     345.25 ±  7%  interrupts.CPU54.RES:Rescheduling_interrupts
      4682 ± 34%     +22.9%       5756 ± 33%  interrupts.CPU6.NMI:Non-maskable_interrupts
      4682 ± 34%     +22.9%       5756 ± 33%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    656.50 ± 17%     +54.4%       1013 ± 16%  interrupts.CPU6.RES:Rescheduling_interrupts
      3726           +53.9%       5736 ± 33%  interrupts.CPU64.NMI:Non-maskable_interrupts
      3726           +53.9%       5736 ± 33%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
      3724           +53.3%       5708 ± 33%  interrupts.CPU65.NMI:Non-maskable_interrupts
      3724           +53.3%       5708 ± 33%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
      4543 ± 33%     +23.7%       5618 ± 33%  interrupts.CPU71.NMI:Non-maskable_interrupts
      4543 ± 33%     +23.7%       5618 ± 33%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
    902.50 ± 69%     -68.4%     285.00 ± 27%  interrupts.CPU72.RES:Rescheduling_interrupts
      4647 ± 34%     +22.2%       5677 ± 33%  interrupts.CPU73.NMI:Non-maskable_interrupts
      4647 ± 34%     +22.2%       5677 ± 33%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
      3736           +77.2%       6622 ± 24%  interrupts.CPU79.NMI:Non-maskable_interrupts
      3736           +77.2%       6622 ± 24%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      4705 ± 34%     +22.2%       5750 ± 33%  interrupts.CPU8.NMI:Non-maskable_interrupts
      4705 ± 34%     +22.2%       5750 ± 33%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      4666 ± 34%     +41.7%       6613 ± 24%  interrupts.CPU83.NMI:Non-maskable_interrupts
      4666 ± 34%     +41.7%       6613 ± 24%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
      4644 ± 34%     +22.3%       5680 ± 33%  interrupts.CPU84.NMI:Non-maskable_interrupts
      4644 ± 34%     +22.3%       5680 ± 33%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
      4653 ± 35%     +22.1%       5682 ± 33%  interrupts.CPU86.NMI:Non-maskable_interrupts
      4653 ± 35%     +22.1%       5682 ± 33%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
      3712           +52.5%       5660 ± 33%  interrupts.CPU87.NMI:Non-maskable_interrupts
      3712           +52.5%       5660 ± 33%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      3741           +77.5%       6640 ± 24%  interrupts.CPU88.NMI:Non-maskable_interrupts
      3741           +77.5%       6640 ± 24%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
      1278 ± 17%     -49.1%     650.50 ± 51%  interrupts.CPU88.RES:Rescheduling_interrupts
      3716           +77.9%       6611 ± 24%  interrupts.CPU89.NMI:Non-maskable_interrupts
      3716           +77.9%       6611 ± 24%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
    511.25 ± 14%     +63.9%     837.75 ± 33%  interrupts.CPU9.RES:Rescheduling_interrupts
      4648 ± 34%     +42.8%       6639 ± 24%  interrupts.CPU91.NMI:Non-maskable_interrupts
      4648 ± 34%     +42.8%       6639 ± 24%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
      4648 ± 34%     +22.4%       5689 ± 33%  interrupts.CPU92.NMI:Non-maskable_interrupts
      4648 ± 34%     +22.4%       5689 ± 33%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
      3716          +103.5%       7565        interrupts.CPU95.NMI:Non-maskable_interrupts
      3716          +103.5%       7565        interrupts.CPU95.PMI:Performance_monitoring_interrupts
      4639 ± 34%     +22.1%       5664 ± 33%  interrupts.CPU96.NMI:Non-maskable_interrupts
      4639 ± 34%     +22.1%       5664 ± 33%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
      3722           +53.1%       5699 ± 33%  interrupts.CPU98.NMI:Non-maskable_interrupts
      3722           +53.1%       5699 ± 33%  interrupts.CPU98.PMI:Performance_monitoring_interrupts


                                                                                
                            will-it-scale.per_thread_ops                        
                                                                                
  7e+06 +-------------------------------------------------------------------+   
        | O  O O O  O O O  O O O  O O O  O O    O O  O O                    |   
  6e+06 |.+..+.+.+    +.+..+.+.+..+.+.+..+.+..+.+.+..+.+.+..+.+.+..+.+.+..+.|   
        |        :    :                                                     |   
  5e+06 |-+       :   :                                                     |   
        |         :   :                                                     |   
  4e+06 |-+       :  :                                                      |   
        |         :  :                                                      |   
  3e+06 |-+        : :                                                      |   
        |          : :                                                      |   
  2e+06 |-+        : :                                                      |   
        |          : :                                                      |   
  1e+06 |-+         :                                                       |   
        |           :                                                       |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                will-it-scale.workload                          
                                                                                
    2e+09 +-----------------------------------------------------------------+   
  1.8e+09 |-O  O O O O  O O O O  O O O  O O O O  O O O  O                   |   
          |.+..+.+.+    +.+.+.+..+.+.+..+.+.+.+..+.+.+..+.+.+.+..+.+.+.+..+.|   
  1.6e+09 |-+      :    :                                                   |   
  1.4e+09 |-+      :    :                                                   |   
          |         :  :                                                    |   
  1.2e+09 |-+       :  :                                                    |   
    1e+09 |-+       :  :                                                    |   
    8e+08 |-+       :  :                                                    |   
          |         : :                                                     |   
    6e+08 |-+       : :                                                     |   
    4e+08 |-+        ::                                                     |   
          |          ::                                                     |   
    2e+08 |-+        :                                                      |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/malloc2/will-it-scale/0x11

commit: 
  b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support")
  492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")

b121b341e5983bdc 492e639f0c222784e2e0f121966 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          4:4         -100%            :4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
         %stddev     %change         %stddev
             \          |                \  
    969613            +2.5%     993613        will-it-scale.per_process_ops
      9816            +1.7%       9984        will-it-scale.time.maximum_resident_set_size
 2.792e+08            +2.5%  2.862e+08        will-it-scale.workload
    329.25 ± 13%     +43.7%     473.00        cpuidle.POLL.usage
   7397888 ±  5%     +17.6%    8699904        meminfo.DirectMap2M
      2259 ±  2%      +7.5%       2429        vmstat.system.cs
      5771 ±  5%      -7.8%       5323        numa-vmstat.node1.nr_slab_reclaimable
      3003 ± 13%     +34.0%       4024        numa-vmstat.node1.nr_slab_unreclaimable
     23875            +3.1%      24612        proc-vmstat.nr_shmem
      1959            +3.3%       2024        proc-vmstat.pgactivate
     23087 ±  5%      -7.8%      21292        numa-meminfo.node1.KReclaimable
     23087 ±  5%      -7.8%      21292        numa-meminfo.node1.SReclaimable
     12013 ± 13%     +34.0%      16096        numa-meminfo.node1.SUnreclaim
      1094 ±  9%     +20.5%       1319        slabinfo.dmaengine-unmap-16.active_objs
      1094 ±  9%     +20.5%       1319        slabinfo.dmaengine-unmap-16.num_objs
      1573 ±  4%     -18.9%       1275        slabinfo.file_lock_cache.active_objs
      1573 ±  4%     -18.9%       1275        slabinfo.file_lock_cache.num_objs
     20496 ±  2%     -10.1%      18432        slabinfo.skbuff_head_cache.active_objs
     20496 ±  2%     -10.1%      18432        slabinfo.skbuff_head_cache.num_objs
 4.371e+10            +2.6%  4.486e+10        perf-stat.i.branch-instructions
      4.12            -0.1        4.05        perf-stat.i.branch-miss-rate%
      2.36            -0.0        2.31        perf-stat.i.cache-miss-rate%
 8.193e+08            +2.1%  8.365e+08        perf-stat.i.cache-references
      2224 ±  2%      +7.9%       2400        perf-stat.i.context-switches
  4.33e+11            +2.3%  4.428e+11        perf-stat.i.cpu-cycles
    279.15            -1.5%     275.01        perf-stat.i.cpu-migrations
 9.822e+08            +1.7%  9.991e+08        perf-stat.i.iTLB-load-misses
  1.56e+11            +2.3%  1.596e+11        perf-stat.i.iTLB-loads
 1.554e+11            +2.6%  1.594e+11        perf-stat.i.instructions
      1.52            +2.2%       1.55        perf-stat.i.metric.GHz
      0.03            +3.7%       0.03        perf-stat.i.metric.K/sec
    701.66            +2.3%     717.66        perf-stat.i.metric.M/sec
      2740            +2.8%       2816        perf-stat.i.minor-faults
      2740            +2.8%       2816        perf-stat.i.page-faults
      4.10            -0.1        4.04        perf-stat.overall.branch-miss-rate%
      2.24            -0.1        2.19        perf-stat.overall.cache-miss-rate%
  4.36e+10            +2.6%  4.473e+10        perf-stat.ps.branch-instructions
 8.182e+08            +2.1%   8.35e+08        perf-stat.ps.cache-references
      2185 ±  2%      +7.9%       2359        perf-stat.ps.context-switches
  4.32e+11            +2.3%  4.418e+11        perf-stat.ps.cpu-cycles
    269.08            -1.6%     264.85        perf-stat.ps.cpu-migrations
 9.792e+08            +1.7%  9.958e+08        perf-stat.ps.iTLB-load-misses
 1.556e+11            +2.3%  1.591e+11        perf-stat.ps.iTLB-loads
  1.55e+11            +2.5%   1.59e+11        perf-stat.ps.instructions
      2694            +2.5%       2760        perf-stat.ps.minor-faults
      2694            +2.5%       2760        perf-stat.ps.page-faults
 4.754e+13            +2.7%  4.882e+13        perf-stat.total.instructions
      3888 ±  8%     -14.7%       3317        sched_debug.cfs_rq:/.load.avg
    137829 ± 62%     -80.9%      26264        sched_debug.cfs_rq:/.load.max
      2980           -19.8%       2390        sched_debug.cfs_rq:/.load.min
      9697 ± 53%     -75.3%       2397        sched_debug.cfs_rq:/.load.stddev
      3.52 ±  9%     -15.9%       2.97        sched_debug.cfs_rq:/.load_avg.avg
    105.71 ± 63%     -72.1%      29.50        sched_debug.cfs_rq:/.load_avg.max
      8.21 ± 50%     -66.4%       2.76        sched_debug.cfs_rq:/.load_avg.stddev
      1.30           -13.4%       1.12        sched_debug.cfs_rq:/.nr_spread_over.avg
    407305 ±  5%     +17.4%     478222        sched_debug.cfs_rq:/.spread0.max
  -5404888            -3.2%   -5231353        sched_debug.cfs_rq:/.spread0.min
      0.83          +160.0%       2.17        sched_debug.cfs_rq:/.util_est_enqueued.min
      7576            -9.1%       6888        sched_debug.cpu.curr->pid.max
    335.97 ±  5%     +11.0%     373.05        sched_debug.cpu.curr->pid.stddev
     23888 ± 52%    +214.7%      75185        sched_debug.cpu.nr_switches.max
    878.67 ± 17%     -25.1%     658.33        sched_debug.cpu.nr_switches.min
      2016 ± 30%    +129.0%       4618        sched_debug.cpu.nr_switches.stddev
      0.00 ± 20%    -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.avg
     16513 ± 86%    +329.7%      70964        sched_debug.cpu.sched_count.max
      1557 ± 42%    +176.4%       4305        sched_debug.cpu.sched_count.stddev
      2278           +29.0%       2939        sched_debug.cpu.sched_goidle.max
    202.41           +11.1%     224.95        sched_debug.cpu.sched_goidle.stddev
      6738 ± 44%    +230.4%      22261        sched_debug.cpu.ttwu_count.max
    673.86 ± 23%    +112.5%       1431        sched_debug.cpu.ttwu_count.stddev
      4944 ± 77%    +343.2%      21913        sched_debug.cpu.ttwu_local.max
    463.98 ± 43%    +189.1%       1341        sched_debug.cpu.ttwu_local.stddev
      0.20 ± 74%    -100.0%       0.00        sched_debug.rt_rq:/.rt_time.avg
     57.79 ± 74%    -100.0%       0.00        sched_debug.rt_rq:/.rt_time.max
      3.40 ± 74%    -100.0%       0.00        sched_debug.rt_rq:/.rt_time.stddev
     50.90 ± 10%     -11.0       39.91        perf-profile.calltrace.cycles-pp.malloc
     17.95 ± 10%      -3.5       14.47        perf-profile.calltrace.cycles-pp.free
      2.32 ± 30%      -1.5        0.84        perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      2.02 ± 14%      -1.0        1.02        perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.05 ± 28%      -0.3        0.75        perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.65 ± 17%      +0.3        0.97        perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.13 ±173%      +0.4        0.55        perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.89 ± 13%      +0.4        2.34        perf-profile.calltrace.cycles-pp.change_prot_numa.task_numa_work.task_work_run.exit_to_usermode_loop.prepare_exit_to_usermode
      1.89 ± 13%      +0.5        2.34        perf-profile.calltrace.cycles-pp.change_protection.change_prot_numa.task_numa_work.task_work_run.exit_to_usermode_loop
      0.30 ±101%      +0.5        0.75        perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.92 ± 13%      +0.5        2.38        perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode
      1.90 ± 13%      +0.5        2.36        perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode
      0.00            +0.5        0.50        perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.00            +0.5        0.50        perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +0.5        0.51        perf-profile.calltrace.cycles-pp.__acct_update_integrals.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.6        0.55        perf-profile.calltrace.cycles-pp.__run_timers.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      0.00            +0.6        0.60        perf-profile.calltrace.cycles-pp.cpuacct_charge.update_curr.task_tick_fair.scheduler_tick.update_process_times
      0.00            +0.6        0.64        perf-profile.calltrace.cycles-pp._raw_spin_lock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.13 ±173%      +0.7        0.84        perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.task_tick_fair.scheduler_tick.update_process_times
      0.13 ±173%      +0.8        0.91        perf-profile.calltrace.cycles-pp.__update_load_avg_se.update_load_avg.task_tick_fair.scheduler_tick.update_process_times
      0.00            +0.9        0.89        perf-profile.calltrace.cycles-pp.run_local_timers.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.9        0.95        perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +1.0        0.98        perf-profile.calltrace.cycles-pp.account_user_time.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.02 ± 22%      +1.0        2.01        perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      0.00            +1.0        0.99        perf-profile.calltrace.cycles-pp.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +1.0        1.00        perf-profile.calltrace.cycles-pp.ksoftirqd_running.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      2.62 ± 16%      +1.0        3.63        perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.00            +1.1        1.11        perf-profile.calltrace.cycles-pp.run_posix_cpu_timers.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +1.1        1.15        perf-profile.calltrace.cycles-pp.update_rq_clock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      1.15 ± 23%      +1.6        2.78        perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      0.71 ± 36%      +1.8        2.54        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      1.20 ± 30%      +2.8        4.00        perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      2.75 ± 24%      +3.7        6.47        perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      4.15 ± 25%      +6.5       10.63        perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      6.49 ± 18%      +8.8       15.25        perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
     14.71 ± 23%      +8.8       23.52        perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
     19.68 ± 24%      +9.5       29.17        perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      6.81 ± 19%      +9.8       16.62        perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      7.62 ± 19%     +10.9       18.52        perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
     26.15 ± 25%     +13.0       39.11        perf-profile.calltrace.cycles-pp.apic_timer_interrupt
     23.59 ± 25%     +13.2       36.78        perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
     51.00 ± 10%     -11.0       40.04        perf-profile.children.cycles-pp.malloc
     16.65 ± 10%      -3.3       13.31        perf-profile.children.cycles-pp.free
      2.71 ± 25%      -1.8        0.92        perf-profile.children.cycles-pp.__remove_hrtimer
      2.36 ± 11%      -1.2        1.15        perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      1.86 ± 12%      -0.4        1.43        perf-profile.children.cycles-pp.malloc@plt
      1.26 ± 23%      -0.4        0.84        perf-profile.children.cycles-pp.timerqueue_del
      0.88 ± 29%      -0.4        0.48        perf-profile.children.cycles-pp.rb_next
      0.67 ± 28%      -0.3        0.36        perf-profile.children.cycles-pp.rcu_irq_enter
      0.17 ± 12%      -0.1        0.05        perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.37 ± 12%      -0.0        0.32        perf-profile.children.cycles-pp.native_write_msr
      0.25 ±  9%      -0.0        0.21        perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.14 ±  5%      -0.0        0.12        perf-profile.children.cycles-pp.irq_work_tick
      0.07 ± 13%      +0.0        0.10        perf-profile.children.cycles-pp.irq_work_run_list
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.account_entity_enqueue
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.account_process_tick
      0.14 ± 19%      +0.1        0.19        perf-profile.children.cycles-pp.ret_from_intr
      0.30 ±  7%      +0.1        0.36        perf-profile.children.cycles-pp.vm_normal_page
      0.18 ± 13%      +0.1        0.24        perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.rcu_qs
      0.17 ±  9%      +0.1        0.24        perf-profile.children.cycles-pp.x86_pmu_disable
      0.01 ±173%      +0.1        0.08        perf-profile.children.cycles-pp.nohz_balance_exit_idle
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.__note_gp_changes
      0.23 ± 19%      +0.1        0.30        perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.01 ±173%      +0.1        0.10        perf-profile.children.cycles-pp.update_min_vruntime
      0.03 ±102%      +0.1        0.13        perf-profile.children.cycles-pp.account_entity_dequeue
      0.11 ± 17%      +0.1        0.22        perf-profile.children.cycles-pp.update_cfs_group
      0.10 ± 15%      +0.1        0.20        perf-profile.children.cycles-pp.native_read_msr
      0.39 ± 11%      +0.1        0.50        perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.28 ± 18%      +0.1        0.39        perf-profile.children.cycles-pp.lapic_next_deadline
      0.09 ± 19%      +0.1        0.20        perf-profile.children.cycles-pp.__accumulate_pelt_segments
      0.39 ±  9%      +0.1        0.50        perf-profile.children.cycles-pp.timerqueue_add
      0.48 ± 12%      +0.1        0.61        perf-profile.children.cycles-pp.enqueue_hrtimer
      0.11 ± 23%      +0.1        0.25        perf-profile.children.cycles-pp.__calc_delta
      0.05 ± 67%      +0.2        0.21        perf-profile.children.cycles-pp.note_gp_changes
      0.14 ± 27%      +0.2        0.31        perf-profile.children.cycles-pp.trigger_load_balance
      0.02 ±173%      +0.2        0.19        perf-profile.children.cycles-pp.calc_global_load_tick
      0.09 ± 17%      +0.2        0.27        perf-profile.children.cycles-pp.native_sched_clock
      0.07 ±  6%      +0.2        0.25        perf-profile.children.cycles-pp.hrtimer_run_queues
      0.18 ± 18%      +0.2        0.37        perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.10 ± 17%      +0.2        0.29        perf-profile.children.cycles-pp.sched_clock
      0.16 ± 23%      +0.2        0.39        perf-profile.children.cycles-pp.reweight_entity
      0.68 ± 13%      +0.2        0.91        perf-profile.children.cycles-pp.ktime_get
      0.11 ± 17%      +0.2        0.34        perf-profile.children.cycles-pp.sched_clock_cpu
      0.61 ± 17%      +0.2        0.85        perf-profile.children.cycles-pp.clockevents_program_event
      0.21 ±  7%      +0.3        0.50        perf-profile.children.cycles-pp.cpuacct_account_field
      0.09 ± 19%      +0.3        0.42        perf-profile.children.cycles-pp.hrtimer_active
      2.15 ±  8%      +0.3        2.48        perf-profile.children.cycles-pp.change_p4d_range
      2.13 ±  8%      +0.3        2.46        perf-profile.children.cycles-pp.change_pte_range
      0.81 ± 14%      +0.3        1.15        perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      2.15 ±  8%      +0.3        2.49        perf-profile.children.cycles-pp.change_protection
      2.15 ±  8%      +0.3        2.49        perf-profile.children.cycles-pp.change_prot_numa
      0.15 ± 13%      +0.3        0.49        perf-profile.children.cycles-pp.raise_softirq
      0.23 ±  9%      +0.3        0.57        perf-profile.children.cycles-pp.__acct_update_integrals
      2.16 ±  8%      +0.3        2.51        perf-profile.children.cycles-pp.task_work_run
      2.16 ±  8%      +0.3        2.51        perf-profile.children.cycles-pp.task_numa_work
      0.15 ± 16%      +0.3        0.50        perf-profile.children.cycles-pp.profile_tick
      2.19 ±  8%      +0.3        2.54        perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.31 ±  9%      +0.4        0.67        perf-profile.children.cycles-pp.cpuacct_charge
      0.37 ± 15%      +0.4        0.75        perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.55 ± 10%      +0.4        0.94        perf-profile.children.cycles-pp._raw_spin_lock
      0.14 ± 34%      +0.4        0.58        perf-profile.children.cycles-pp.rcu_core
      2.58 ±  9%      +0.5        3.04        perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.46 ± 18%      +0.5        0.95        perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.44 ± 16%      +0.6        1.02        perf-profile.children.cycles-pp.__update_load_avg_se
      0.24 ± 30%      +0.6        0.84        perf-profile.children.cycles-pp.__run_timers
      0.47 ±  3%      +0.6        1.08        perf-profile.children.cycles-pp.account_user_time
      0.45 ± 14%      +0.6        1.07        perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.30 ±  7%      +0.7        0.99        perf-profile.children.cycles-pp.run_local_timers
      0.28 ± 31%      +0.8        1.10        perf-profile.children.cycles-pp.run_timer_softirq
      0.17 ± 44%      +0.9        1.11        perf-profile.children.cycles-pp.ksoftirqd_running
      2.99 ± 11%      +0.9        3.94        perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.26 ± 45%      +1.0        1.23        perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.25 ± 23%      +1.0        1.27        perf-profile.children.cycles-pp.update_rq_clock
      1.23 ± 19%      +1.1        2.31        perf-profile.children.cycles-pp.update_curr
      1.35 ± 19%      +1.8        3.10        perf-profile.children.cycles-pp.update_load_avg
      0.83 ± 31%      +2.0        2.85        perf-profile.children.cycles-pp.__softirqentry_text_start
      1.40 ± 27%      +3.1        4.46        perf-profile.children.cycles-pp.irq_exit
      3.20 ± 20%      +4.0        7.18        perf-profile.children.cycles-pp.task_tick_fair
      4.84 ± 21%      +6.1       10.97        perf-profile.children.cycles-pp.scheduler_tick
      6.83 ± 17%      +8.9       15.73        perf-profile.children.cycles-pp.update_process_times
     17.23 ± 19%      +9.0       26.21        perf-profile.children.cycles-pp.__hrtimer_run_queues
     22.90 ± 19%      +9.5       32.43        perf-profile.children.cycles-pp.hrtimer_interrupt
      7.16 ± 18%     +10.0       17.12        perf-profile.children.cycles-pp.tick_sched_handle
      8.02 ± 18%     +11.1       19.09        perf-profile.children.cycles-pp.tick_sched_timer
     29.66 ± 21%     +13.1       42.74        perf-profile.children.cycles-pp.apic_timer_interrupt
     27.48 ± 21%     +13.2       40.69        perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     46.00 ± 10%     -10.4       35.62        perf-profile.self.cycles-pp.malloc
     13.39 ± 11%      -2.9       10.47        perf-profile.self.cycles-pp.free
      1.45 ± 27%      -1.4        0.09        perf-profile.self.cycles-pp.__remove_hrtimer
      1.41 ± 12%      -0.9        0.55        perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      1.75 ± 12%      -0.4        1.31        perf-profile.self.cycles-pp.malloc@plt
      0.84 ± 29%      -0.4        0.47        perf-profile.self.cycles-pp.rb_next
      0.66 ± 29%      -0.3        0.36        perf-profile.self.cycles-pp.rcu_irq_enter
      0.25 ± 14%      -0.0        0.20        perf-profile.self.cycles-pp.timerqueue_del
      0.24 ±  9%      -0.0        0.20        perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.36 ± 12%      -0.0        0.32        perf-profile.self.cycles-pp.native_write_msr
      0.14 ±  5%      -0.0        0.11        perf-profile.self.cycles-pp.irq_work_tick
      0.10 ± 17%      +0.0        0.13        perf-profile.self.cycles-pp.lapic_next_deadline
      0.07 ±  7%      +0.0        0.11        perf-profile.self.cycles-pp.update_process_times
      0.26 ±  9%      +0.0        0.31        perf-profile.self.cycles-pp.vm_normal_page
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__note_gp_changes
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.account_entity_enqueue
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.rcu_qs
      0.11 ± 14%      +0.1        0.17        perf-profile.self.cycles-pp.clockevents_program_event
      0.01 ±173%      +0.1        0.08        perf-profile.self.cycles-pp.irq_work_run_list
      0.01 ±173%      +0.1        0.08        perf-profile.self.cycles-pp.nohz_balance_exit_idle
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.sched_clock_cpu
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.17 ± 12%      +0.1        0.24        perf-profile.self.cycles-pp.x86_pmu_disable
      0.01 ±173%      +0.1        0.09        perf-profile.self.cycles-pp.update_min_vruntime
      0.21 ± 18%      +0.1        0.29        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.09 ± 21%      +0.1        0.17        perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.23 ± 13%      +0.1        0.32        perf-profile.self.cycles-pp.prepare_exit_to_usermode
      0.06 ± 20%      +0.1        0.15        perf-profile.self.cycles-pp.tick_sched_handle
      0.00            +0.1        0.09        perf-profile.self.cycles-pp.note_gp_changes
      0.03 ±100%      +0.1        0.12        perf-profile.self.cycles-pp.account_entity_dequeue
      0.30 ±  9%      +0.1        0.39        perf-profile.self.cycles-pp.irq_exit
      0.11 ± 19%      +0.1        0.21        perf-profile.self.cycles-pp.update_cfs_group
      0.39 ± 11%      +0.1        0.49        perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.08 ± 26%      +0.1        0.19        perf-profile.self.cycles-pp.reweight_entity
      0.09 ± 17%      +0.1        0.20        perf-profile.self.cycles-pp.native_read_msr
      0.09 ± 24%      +0.1        0.20        perf-profile.self.cycles-pp.__accumulate_pelt_segments
      0.11 ± 26%      +0.1        0.23        perf-profile.self.cycles-pp.trigger_load_balance
      0.31 ±  7%      +0.1        0.43        perf-profile.self.cycles-pp.timerqueue_add
      0.11 ± 19%      +0.1        0.25        perf-profile.self.cycles-pp.__calc_delta
      0.02 ±173%      +0.2        0.18        perf-profile.self.cycles-pp.rcu_core
      0.02 ±173%      +0.2        0.18        perf-profile.self.cycles-pp.calc_global_load_tick
      0.09 ± 19%      +0.2        0.26        perf-profile.self.cycles-pp.native_sched_clock
      0.07 ±  7%      +0.2        0.25        perf-profile.self.cycles-pp.hrtimer_run_queues
      0.53 ± 14%      +0.2        0.72        perf-profile.self.cycles-pp.ktime_get
      0.21 ±  7%      +0.3        0.50        perf-profile.self.cycles-pp.cpuacct_account_field
      0.27            +0.3        0.58        perf-profile.self.cycles-pp.account_user_time
      0.09 ± 15%      +0.3        0.42        perf-profile.self.cycles-pp.hrtimer_active
      0.23 ±  9%      +0.3        0.57        perf-profile.self.cycles-pp.__acct_update_integrals
      0.15 ± 14%      +0.3        0.49        perf-profile.self.cycles-pp.raise_softirq
      0.15 ± 16%      +0.3        0.50        perf-profile.self.cycles-pp.profile_tick
      0.67 ± 12%      +0.4        1.03        perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.30 ±  9%      +0.4        0.67        perf-profile.self.cycles-pp.cpuacct_charge
      0.12 ± 29%      +0.4        0.49        perf-profile.self.cycles-pp.run_timer_softirq
      0.37 ± 14%      +0.4        0.75        perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.54 ±  9%      +0.4        0.93        perf-profile.self.cycles-pp._raw_spin_lock
      0.16 ± 32%      +0.4        0.59        perf-profile.self.cycles-pp.__run_timers
      0.42 ± 17%      +0.5        0.87        perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.23 ± 11%      +0.5        0.69        perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.27 ± 19%      +0.5        0.75        perf-profile.self.cycles-pp.tick_sched_timer
      0.42 ± 25%      +0.5        0.90        perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.23 ±  8%      +0.5        0.73        perf-profile.self.cycles-pp.run_local_timers
      0.39 ± 16%      +0.5        0.90        perf-profile.self.cycles-pp.__update_load_avg_se
      0.78 ± 24%      +0.5        1.29        perf-profile.self.cycles-pp.update_curr
      0.28 ± 29%      +0.5        0.80        perf-profile.self.cycles-pp.task_tick_fair
      0.27 ± 33%      +0.5        0.81        perf-profile.self.cycles-pp.__softirqentry_text_start
      0.45 ± 24%      +0.7        1.14        perf-profile.self.cycles-pp.update_load_avg
      0.14 ± 31%      +0.8        0.93        perf-profile.self.cycles-pp.update_rq_clock
      0.17 ± 44%      +0.9        1.10        perf-profile.self.cycles-pp.ksoftirqd_running
      0.26 ± 45%      +1.0        1.22        perf-profile.self.cycles-pp.run_posix_cpu_timers
      1251 ±134%   +6253.9%      79487        interrupts.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3
      1867 ± 12%     -27.6%       1351        interrupts.CPU0.RES:Rescheduling_interrupts
    551.25 ± 53%     -45.2%     302.00        interrupts.CPU10.RES:Rescheduling_interrupts
    140.00 ± 87%     -92.9%      10.00        interrupts.CPU101.RES:Rescheduling_interrupts
      1121 ± 88%     -79.4%     231.00        interrupts.CPU102.RES:Rescheduling_interrupts
    118.50 ± 31%     -68.8%      37.00        interrupts.CPU103.RES:Rescheduling_interrupts
    387.25 ± 43%    +187.4%       1113        interrupts.CPU104.RES:Rescheduling_interrupts
      1326 ± 84%     -78.1%     291.00        interrupts.CPU106.RES:Rescheduling_interrupts
     86.50 ± 19%     -52.6%      41.00        interrupts.CPU109.RES:Rescheduling_interrupts
    589.75 ± 18%     -54.6%     268.00        interrupts.CPU11.RES:Rescheduling_interrupts
      1711 ± 39%     -85.9%     242.00        interrupts.CPU110.RES:Rescheduling_interrupts
     97.75 ± 62%     -83.6%      16.00        interrupts.CPU113.RES:Rescheduling_interrupts
    897.75 ± 44%    +145.7%       2206        interrupts.CPU120.RES:Rescheduling_interrupts
      1079 ± 71%    +310.7%       4433        interrupts.CPU122.RES:Rescheduling_interrupts
      4608 ± 34%     +63.4%       7532        interrupts.CPU123.NMI:Non-maskable_interrupts
      4608 ± 34%     +63.4%       7532        interrupts.CPU123.PMI:Performance_monitoring_interrupts
    240.75 ±105%     -93.8%      15.00        interrupts.CPU125.RES:Rescheduling_interrupts
    134.00 ± 62%     -95.5%       6.00        interrupts.CPU129.RES:Rescheduling_interrupts
      1251 ±134%   +6253.9%      79487        interrupts.CPU13.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3
    607.50 ± 34%     -58.7%     251.00        interrupts.CPU13.RES:Rescheduling_interrupts
      3690          +103.5%       7510        interrupts.CPU130.NMI:Non-maskable_interrupts
      3690          +103.5%       7510        interrupts.CPU130.PMI:Performance_monitoring_interrupts
      3693          +103.6%       7521        interrupts.CPU131.NMI:Non-maskable_interrupts
      3693          +103.6%       7521        interrupts.CPU131.PMI:Performance_monitoring_interrupts
      4588 ± 33%     +63.9%       7522        interrupts.CPU132.NMI:Non-maskable_interrupts
      4588 ± 33%     +63.9%       7522        interrupts.CPU132.PMI:Performance_monitoring_interrupts
      1734 ± 91%     -79.5%     356.00        interrupts.CPU132.RES:Rescheduling_interrupts
      1104 ± 87%     -85.2%     163.00        interrupts.CPU134.RES:Rescheduling_interrupts
      3664          +105.1%       7515        interrupts.CPU136.NMI:Non-maskable_interrupts
      3664          +105.1%       7515        interrupts.CPU136.PMI:Performance_monitoring_interrupts
    685.00 ± 73%    +233.7%       2286        interrupts.CPU138.RES:Rescheduling_interrupts
    481.75 ± 25%     -35.4%     311.00        interrupts.CPU14.RES:Rescheduling_interrupts
    110.25 ± 27%     -69.2%      34.00        interrupts.CPU143.RES:Rescheduling_interrupts
    248.25 ± 46%    +102.2%     502.00        interrupts.CPU144.RES:Rescheduling_interrupts
      4612 ± 35%     +62.9%       7514        interrupts.CPU145.NMI:Non-maskable_interrupts
      4612 ± 35%     +62.9%       7514        interrupts.CPU145.PMI:Performance_monitoring_interrupts
    180.00 ± 75%     -79.4%      37.00        interrupts.CPU145.RES:Rescheduling_interrupts
      4620 ± 34%     +62.4%       7506        interrupts.CPU149.NMI:Non-maskable_interrupts
      4620 ± 34%     +62.4%       7506        interrupts.CPU149.PMI:Performance_monitoring_interrupts
    276.75 ±116%     -97.8%       6.00        interrupts.CPU149.RES:Rescheduling_interrupts
      4688 ± 35%     +62.8%       7635        interrupts.CPU15.NMI:Non-maskable_interrupts
      4688 ± 35%     +62.8%       7635        interrupts.CPU15.PMI:Performance_monitoring_interrupts
    495.25 ± 12%     -47.7%     259.00        interrupts.CPU15.RES:Rescheduling_interrupts
    409.25 ± 30%     +97.2%     807.00        interrupts.CPU150.RES:Rescheduling_interrupts
      4628 ± 34%     +62.7%       7530        interrupts.CPU152.NMI:Non-maskable_interrupts
      4628 ± 34%     +62.7%       7530        interrupts.CPU152.PMI:Performance_monitoring_interrupts
      4614 ± 34%     +64.1%       7573        interrupts.CPU153.NMI:Non-maskable_interrupts
      4614 ± 34%     +64.1%       7573        interrupts.CPU153.PMI:Performance_monitoring_interrupts
    212.00 ± 84%    +230.2%     700.00        interrupts.CPU156.RES:Rescheduling_interrupts
      4621 ± 34%     +62.5%       7512        interrupts.CPU159.NMI:Non-maskable_interrupts
      4621 ± 34%     +62.5%       7512        interrupts.CPU159.PMI:Performance_monitoring_interrupts
      4633 ± 34%     +64.9%       7640        interrupts.CPU16.NMI:Non-maskable_interrupts
      4633 ± 34%     +64.9%       7640        interrupts.CPU16.PMI:Performance_monitoring_interrupts
      4623 ± 35%     +62.8%       7525        interrupts.CPU163.NMI:Non-maskable_interrupts
      4623 ± 35%     +62.8%       7525        interrupts.CPU163.PMI:Performance_monitoring_interrupts
      4623 ± 35%     +62.4%       7509        interrupts.CPU164.NMI:Non-maskable_interrupts
      4623 ± 35%     +62.4%       7509        interrupts.CPU164.PMI:Performance_monitoring_interrupts
      4649 ± 34%     +61.9%       7527        interrupts.CPU165.NMI:Non-maskable_interrupts
      4649 ± 34%     +61.9%       7527        interrupts.CPU165.PMI:Performance_monitoring_interrupts
    226.75 ± 63%     -78.4%      49.00        interrupts.CPU166.RES:Rescheduling_interrupts
    100.75 ± 69%     -94.0%       6.00        interrupts.CPU169.RES:Rescheduling_interrupts
    425.25 ± 20%     -42.4%     245.00        interrupts.CPU17.RES:Rescheduling_interrupts
      4603 ± 35%     +63.7%       7537        interrupts.CPU170.NMI:Non-maskable_interrupts
      4603 ± 35%     +63.7%       7537        interrupts.CPU170.PMI:Performance_monitoring_interrupts
      4635 ± 34%     +62.0%       7508        interrupts.CPU171.NMI:Non-maskable_interrupts
      4635 ± 34%     +62.0%       7508        interrupts.CPU171.PMI:Performance_monitoring_interrupts
    165.00 ± 50%    +269.1%     609.00        interrupts.CPU174.RES:Rescheduling_interrupts
    219.50 ± 55%    +134.2%     514.00        interrupts.CPU176.RES:Rescheduling_interrupts
      3689          +104.2%       7535        interrupts.CPU177.NMI:Non-maskable_interrupts
      3689          +104.2%       7535        interrupts.CPU177.PMI:Performance_monitoring_interrupts
    250.50 ± 35%    +108.8%     523.00        interrupts.CPU178.RES:Rescheduling_interrupts
    534.50 ±  7%     -44.1%     299.00        interrupts.CPU18.RES:Rescheduling_interrupts
      1195           -48.7%     613.00        interrupts.CPU181.CAL:Function_call_interrupts
      3684          +104.3%       7525        interrupts.CPU184.NMI:Non-maskable_interrupts
      3684          +104.3%       7525        interrupts.CPU184.PMI:Performance_monitoring_interrupts
    243.75 ± 19%     +35.4%     330.00        interrupts.CPU190.RES:Rescheduling_interrupts
     61.25 ± 59%     -91.8%       5.00        interrupts.CPU191.RES:Rescheduling_interrupts
    114.00 ±120%     -94.7%       6.00        interrupts.CPU195.RES:Rescheduling_interrupts
    538.75 ± 84%     -77.4%     122.00        interrupts.CPU196.RES:Rescheduling_interrupts
    193.50 ± 10%     -95.3%       9.00        interrupts.CPU197.RES:Rescheduling_interrupts
    454.00 ± 22%     -55.5%     202.00        interrupts.CPU198.RES:Rescheduling_interrupts
    764.75 ± 56%     -58.9%     314.00        interrupts.CPU2.RES:Rescheduling_interrupts
    463.50 ±  9%     -23.0%     357.00        interrupts.CPU20.RES:Rescheduling_interrupts
    243.50 ± 48%     -96.3%       9.00        interrupts.CPU200.RES:Rescheduling_interrupts
     82.25 ± 49%     -83.0%      14.00        interrupts.CPU205.RES:Rescheduling_interrupts
    349.75 ± 12%    +120.4%     771.00        interrupts.CPU206.RES:Rescheduling_interrupts
    222.25 ± 56%     -80.7%      43.00        interrupts.CPU208.RES:Rescheduling_interrupts
      4685 ± 35%     +61.5%       7565        interrupts.CPU21.NMI:Non-maskable_interrupts
      4685 ± 35%     +61.5%       7565        interrupts.CPU21.PMI:Performance_monitoring_interrupts
    394.50 ± 48%     -92.1%      31.00        interrupts.CPU210.RES:Rescheduling_interrupts
    457.50 ± 74%     -78.6%      98.00        interrupts.CPU212.RES:Rescheduling_interrupts
    107.50 ± 51%     -86.0%      15.00        interrupts.CPU213.RES:Rescheduling_interrupts
     79.75 ± 53%     -88.7%       9.00        interrupts.CPU215.RES:Rescheduling_interrupts
    137.25 ± 24%     -46.8%      73.00        interrupts.CPU218.RES:Rescheduling_interrupts
     84.25 ± 29%     -90.5%       8.00        interrupts.CPU219.RES:Rescheduling_interrupts
      3707          +105.5%       7619        interrupts.CPU22.NMI:Non-maskable_interrupts
      3707          +105.5%       7619        interrupts.CPU22.PMI:Performance_monitoring_interrupts
    512.75 ± 29%     -43.6%     289.00        interrupts.CPU22.RES:Rescheduling_interrupts
    157.75 ± 42%     -77.8%      35.00        interrupts.CPU220.RES:Rescheduling_interrupts
     93.00 ±103%     -92.5%       7.00        interrupts.CPU221.RES:Rescheduling_interrupts
     73.75 ± 22%     -67.5%      24.00        interrupts.CPU222.RES:Rescheduling_interrupts
      4615 ± 35%     +63.0%       7521        interrupts.CPU225.NMI:Non-maskable_interrupts
      4615 ± 35%     +63.0%       7521        interrupts.CPU225.PMI:Performance_monitoring_interrupts
     75.00 ± 44%     -93.3%       5.00        interrupts.CPU225.RES:Rescheduling_interrupts
    144.75 ± 46%     -76.5%      34.00        interrupts.CPU226.RES:Rescheduling_interrupts
      3740          +104.5%       7648        interrupts.CPU23.NMI:Non-maskable_interrupts
      3740          +104.5%       7648        interrupts.CPU23.PMI:Performance_monitoring_interrupts
    404.75 ± 23%     -38.7%     248.00        interrupts.CPU23.RES:Rescheduling_interrupts
    154.75 ± 62%    +122.9%     345.00        interrupts.CPU232.RES:Rescheduling_interrupts
     69.25 ± 32%     -85.6%      10.00        interrupts.CPU233.RES:Rescheduling_interrupts
    121.50 ± 53%     -76.1%      29.00        interrupts.CPU236.RES:Rescheduling_interrupts
      3772          +102.4%       7636        interrupts.CPU24.NMI:Non-maskable_interrupts
      3772          +102.4%       7636        interrupts.CPU24.PMI:Performance_monitoring_interrupts
    683.25 ± 43%     -58.4%     284.00        interrupts.CPU24.RES:Rescheduling_interrupts
    218.25 ±101%     -78.9%      46.00        interrupts.CPU240.RES:Rescheduling_interrupts
      4616 ± 34%     +62.7%       7509        interrupts.CPU242.NMI:Non-maskable_interrupts
      4616 ± 34%     +62.7%       7509        interrupts.CPU242.PMI:Performance_monitoring_interrupts
    108.50 ± 30%     -39.2%      66.00        interrupts.CPU242.RES:Rescheduling_interrupts
    113.25 ± 44%     -93.8%       7.00        interrupts.CPU243.RES:Rescheduling_interrupts
     60.50 ± 83%     -88.4%       7.00        interrupts.CPU247.RES:Rescheduling_interrupts
    161.75 ±125%     -96.9%       5.00        interrupts.CPU249.RES:Rescheduling_interrupts
    239.50 ± 65%     -80.0%      48.00        interrupts.CPU250.RES:Rescheduling_interrupts
    105.25 ± 51%    +286.7%     407.00        interrupts.CPU252.RES:Rescheduling_interrupts
      4622 ± 35%     +62.5%       7510        interrupts.CPU253.NMI:Non-maskable_interrupts
      4622 ± 35%     +62.5%       7510        interrupts.CPU253.PMI:Performance_monitoring_interrupts
    151.50 ± 36%     -48.5%      78.00        interrupts.CPU256.RES:Rescheduling_interrupts
      3669          +105.7%       7547        interrupts.CPU257.NMI:Non-maskable_interrupts
      3669          +105.7%       7547        interrupts.CPU257.PMI:Performance_monitoring_interrupts
     63.75 ± 52%     -90.6%       6.00        interrupts.CPU257.RES:Rescheduling_interrupts
    168.00 ± 56%     -64.9%      59.00        interrupts.CPU258.RES:Rescheduling_interrupts
     90.75 ± 86%     -91.2%       8.00        interrupts.CPU259.RES:Rescheduling_interrupts
      1164 ± 63%     -75.5%     285.00        interrupts.CPU26.RES:Rescheduling_interrupts
      4592 ± 34%     +63.5%       7507        interrupts.CPU260.NMI:Non-maskable_interrupts
      4592 ± 34%     +63.5%       7507        interrupts.CPU260.PMI:Performance_monitoring_interrupts
    119.25 ± 28%     -74.0%      31.00        interrupts.CPU260.RES:Rescheduling_interrupts
      4595 ± 35%     +63.3%       7505        interrupts.CPU262.NMI:Non-maskable_interrupts
      4595 ± 35%     +63.3%       7505        interrupts.CPU262.PMI:Performance_monitoring_interrupts
      4590 ± 34%     +63.7%       7514        interrupts.CPU263.NMI:Non-maskable_interrupts
      4590 ± 34%     +63.7%       7514        interrupts.CPU263.PMI:Performance_monitoring_interrupts
     62.25 ± 56%     -92.0%       5.00        interrupts.CPU263.RES:Rescheduling_interrupts
     98.50 ± 36%    +383.2%     476.00        interrupts.CPU264.RES:Rescheduling_interrupts
     51.25 ± 80%     -86.3%       7.00        interrupts.CPU265.RES:Rescheduling_interrupts
    244.75 ± 84%     -86.1%      34.00        interrupts.CPU266.RES:Rescheduling_interrupts
    114.50 ± 13%     -79.0%      24.00        interrupts.CPU268.RES:Rescheduling_interrupts
      4610 ± 35%     +64.0%       7559        interrupts.CPU269.NMI:Non-maskable_interrupts
      4610 ± 35%     +64.0%       7559        interrupts.CPU269.PMI:Performance_monitoring_interrupts
    441.50 ± 27%     -36.6%     280.00        interrupts.CPU27.RES:Rescheduling_interrupts
    129.00 ± 62%  +20168.2%      26146        interrupts.CPU272.RES:Rescheduling_interrupts
    109.25 ± 65%     -93.6%       7.00        interrupts.CPU279.RES:Rescheduling_interrupts
      3728          +104.8%       7636        interrupts.CPU28.NMI:Non-maskable_interrupts
      3728          +104.8%       7636        interrupts.CPU28.PMI:Performance_monitoring_interrupts
    556.75 ± 23%     -47.9%     290.00        interrupts.CPU28.RES:Rescheduling_interrupts
    358.50 ± 79%     -86.1%      50.00        interrupts.CPU282.RES:Rescheduling_interrupts
    202.75 ± 54%     -83.2%      34.00        interrupts.CPU284.RES:Rescheduling_interrupts
    242.25 ± 69%     -88.0%      29.00        interrupts.CPU287.RES:Rescheduling_interrupts
      3718          +105.0%       7623        interrupts.CPU29.NMI:Non-maskable_interrupts
      3718          +105.0%       7623        interrupts.CPU29.PMI:Performance_monitoring_interrupts
    389.25 ± 14%     -32.2%     264.00        interrupts.CPU29.RES:Rescheduling_interrupts
      3757          +102.6%       7612        interrupts.CPU30.NMI:Non-maskable_interrupts
      3757          +102.6%       7612        interrupts.CPU30.PMI:Performance_monitoring_interrupts
    941.25 ± 19%     -63.3%     345.00        interrupts.CPU30.RES:Rescheduling_interrupts
      3730          +104.4%       7625        interrupts.CPU31.NMI:Non-maskable_interrupts
      3730          +104.4%       7625        interrupts.CPU31.PMI:Performance_monitoring_interrupts
      3744          +103.8%       7632        interrupts.CPU32.NMI:Non-maskable_interrupts
      3744          +103.8%       7632        interrupts.CPU32.PMI:Performance_monitoring_interrupts
    637.75 ± 41%     -56.6%     277.00        interrupts.CPU32.RES:Rescheduling_interrupts
      4649 ± 34%     +64.3%       7637        interrupts.CPU34.NMI:Non-maskable_interrupts
      4649 ± 34%     +64.3%       7637        interrupts.CPU34.PMI:Performance_monitoring_interrupts
    416.25 ± 16%     -30.3%     290.00        interrupts.CPU34.RES:Rescheduling_interrupts
      3725          +104.1%       7602        interrupts.CPU35.NMI:Non-maskable_interrupts
      3725          +104.1%       7602        interrupts.CPU35.PMI:Performance_monitoring_interrupts
      3695          +104.7%       7562        interrupts.CPU36.NMI:Non-maskable_interrupts
      3695          +104.7%       7562        interrupts.CPU36.PMI:Performance_monitoring_interrupts
      3748          +104.3%       7657        interrupts.CPU37.NMI:Non-maskable_interrupts
      3748          +104.3%       7657        interrupts.CPU37.PMI:Performance_monitoring_interrupts
    324.00 ±  9%     -18.8%     263.00        interrupts.CPU37.RES:Rescheduling_interrupts
    436.00 ±  5%     -27.8%     315.00        interrupts.CPU38.RES:Rescheduling_interrupts
      4671 ± 34%     +61.0%       7521        interrupts.CPU39.NMI:Non-maskable_interrupts
      4671 ± 34%     +61.0%       7521        interrupts.CPU39.PMI:Performance_monitoring_interrupts
    349.25 ± 14%     -30.7%     242.00        interrupts.CPU39.RES:Rescheduling_interrupts
    710.50 ± 32%     -52.9%     335.00        interrupts.CPU4.RES:Rescheduling_interrupts
      3707          +103.9%       7560        interrupts.CPU42.NMI:Non-maskable_interrupts
      3707          +103.9%       7560        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      4597 ± 35%     +63.4%       7513        interrupts.CPU43.NMI:Non-maskable_interrupts
      4597 ± 35%     +63.4%       7513        interrupts.CPU43.PMI:Performance_monitoring_interrupts
      4610 ± 35%     +64.1%       7568        interrupts.CPU44.NMI:Non-maskable_interrupts
      4610 ± 35%     +64.1%       7568        interrupts.CPU44.PMI:Performance_monitoring_interrupts
    887.25 ± 95%     -67.7%     287.00        interrupts.CPU44.RES:Rescheduling_interrupts
      3669          +106.6%       7582        interrupts.CPU45.NMI:Non-maskable_interrupts
      3669          +106.6%       7582        interrupts.CPU45.PMI:Performance_monitoring_interrupts
    604.50 ± 66%     -55.0%     272.00        interrupts.CPU47.RES:Rescheduling_interrupts
      3717          +103.6%       7570        interrupts.CPU49.NMI:Non-maskable_interrupts
      3717          +103.6%       7570        interrupts.CPU49.PMI:Performance_monitoring_interrupts
    160.50 ± 20%     +63.9%     263.00        interrupts.CPU49.RES:Rescheduling_interrupts
      4601 ± 34%     +64.8%       7585        interrupts.CPU50.NMI:Non-maskable_interrupts
      4601 ± 34%     +64.8%       7585        interrupts.CPU50.PMI:Performance_monitoring_interrupts
    170.50 ± 36%     +67.2%     285.00        interrupts.CPU50.RES:Rescheduling_interrupts
      4590 ± 34%     +65.5%       7595        interrupts.CPU51.NMI:Non-maskable_interrupts
      4590 ± 34%     +65.5%       7595        interrupts.CPU51.PMI:Performance_monitoring_interrupts
    146.75 ± 43%     -78.9%      31.00        interrupts.CPU51.RES:Rescheduling_interrupts
      4627 ± 35%     +62.4%       7514        interrupts.CPU52.NMI:Non-maskable_interrupts
      4627 ± 35%     +62.4%       7514        interrupts.CPU52.PMI:Performance_monitoring_interrupts
    266.25 ± 57%     -58.3%     111.00        interrupts.CPU54.RES:Rescheduling_interrupts
    204.00 ± 71%     -87.7%      25.00        interrupts.CPU55.RES:Rescheduling_interrupts
      3703          +105.1%       7596        interrupts.CPU58.NMI:Non-maskable_interrupts
      3703          +105.1%       7596        interrupts.CPU58.PMI:Performance_monitoring_interrupts
    589.00 ± 41%     -45.8%     319.00        interrupts.CPU6.RES:Rescheduling_interrupts
    255.25 ± 57%     -63.6%      93.00        interrupts.CPU60.RES:Rescheduling_interrupts
    174.25 ± 19%     -62.1%      66.00        interrupts.CPU62.RES:Rescheduling_interrupts
    207.25 ± 50%     -65.3%      72.00        interrupts.CPU66.RES:Rescheduling_interrupts
    169.00 ± 54%     -65.1%      59.00        interrupts.CPU68.RES:Rescheduling_interrupts
    103.25 ± 66%     -84.5%      16.00        interrupts.CPU69.RES:Rescheduling_interrupts
    501.75 ± 24%     -43.4%     284.00        interrupts.CPU7.RES:Rescheduling_interrupts
    286.50 ± 53%    +388.3%       1399        interrupts.CPU72.RES:Rescheduling_interrupts
    634.25 ± 71%    +406.0%       3209        interrupts.CPU74.RES:Rescheduling_interrupts
    484.00 ± 55%    +147.3%       1197        interrupts.CPU76.RES:Rescheduling_interrupts
     85.75 ± 53%     -63.8%      31.00        interrupts.CPU77.RES:Rescheduling_interrupts
    539.25 ± 18%     +73.9%     938.00        interrupts.CPU8.RES:Rescheduling_interrupts
    755.25 ± 91%     -77.0%     174.00        interrupts.CPU80.RES:Rescheduling_interrupts
     63.25 ± 53%    +119.8%     139.00        interrupts.CPU81.RES:Rescheduling_interrupts
    519.25 ± 33%     -65.1%     181.00        interrupts.CPU82.RES:Rescheduling_interrupts
    517.25 ± 28%     -60.8%     203.00        interrupts.CPU84.RES:Rescheduling_interrupts
    132.25 ± 50%     -61.4%      51.00        interrupts.CPU85.RES:Rescheduling_interrupts
    382.00 ± 35%     +77.2%     677.00        interrupts.CPU86.RES:Rescheduling_interrupts
     62.50 ± 62%     -74.4%      16.00        interrupts.CPU87.RES:Rescheduling_interrupts
     92.00 ± 32%     -66.3%      31.00        interrupts.CPU89.RES:Rescheduling_interrupts
    436.75 ± 19%     -39.1%     266.00        interrupts.CPU9.RES:Rescheduling_interrupts
      1072 ± 76%     -73.7%     282.00        interrupts.CPU92.RES:Rescheduling_interrupts
    167.00 ± 75%     -82.6%      29.00        interrupts.CPU93.RES:Rescheduling_interrupts
    952.75 ± 66%     -68.1%     304.00        interrupts.CPU96.RES:Rescheduling_interrupts
     96.50 ± 25%    +217.1%     306.00        interrupts.CPU97.RES:Rescheduling_interrupts
     54.25 ± 22%    +351.6%     245.00        interrupts.CPU99.RES:Rescheduling_interrupts
     39164 ± 11%     +56.5%      61282        softirqs.CPU0.RCU
    119211 ± 14%     +35.1%     161073        softirqs.CPU0.TIMER
     39132 ± 15%     +54.9%      60633        softirqs.CPU1.RCU
    106485 ± 10%     +39.0%     148019        softirqs.CPU1.TIMER
     40469 ± 13%     +46.2%      59148        softirqs.CPU10.RCU
     33205 ± 16%     +58.8%      52724        softirqs.CPU100.RCU
     33017 ± 16%     +58.1%      52202        softirqs.CPU101.RCU
     32946 ± 16%     +59.9%      52688        softirqs.CPU102.RCU
     32954 ± 16%     +58.7%      52303        softirqs.CPU103.RCU
     32712 ± 16%     +59.2%      52093        softirqs.CPU104.RCU
     32374 ± 16%     +62.4%      52576        softirqs.CPU105.RCU
     32420 ± 16%     +63.4%      52971        softirqs.CPU106.RCU
     32316 ± 15%     +62.9%      52648        softirqs.CPU107.RCU
     32685 ± 15%     +62.1%      52968        softirqs.CPU108.RCU
    132926 ±  9%     +18.9%     157999        softirqs.CPU108.TIMER
     32436 ± 16%     +61.6%      52414        softirqs.CPU109.RCU
     40413 ± 14%     +60.2%      64751        softirqs.CPU11.RCU
    115232 ± 12%     +21.9%     140473        softirqs.CPU11.TIMER
     32751 ± 16%     +60.0%      52403        softirqs.CPU110.RCU
     32850 ± 16%     +60.6%      52742        softirqs.CPU111.RCU
     32532 ± 17%     +60.8%      52319        softirqs.CPU112.RCU
     32501 ± 16%     +60.8%      52267        softirqs.CPU113.RCU
     31957 ± 16%     +64.6%      52601        softirqs.CPU114.RCU
     32115 ± 15%     +62.2%      52104        softirqs.CPU115.RCU
     32767 ± 16%     +59.3%      52200        softirqs.CPU116.RCU
     32625 ± 15%     +61.7%      52741        softirqs.CPU117.RCU
     32319 ± 16%     +61.3%      52142        softirqs.CPU118.RCU
     32227 ± 15%     +61.7%      52110        softirqs.CPU119.RCU
     43084 ± 21%     +41.9%      61121        softirqs.CPU12.RCU
     31813 ± 17%     +64.6%      52360        softirqs.CPU120.RCU
     31512 ± 15%     +66.8%      52559        softirqs.CPU121.RCU
     32569 ± 15%     +61.6%      52619        softirqs.CPU122.RCU
     32337 ± 16%     +61.7%      52276        softirqs.CPU123.RCU
     32442 ± 15%     +61.9%      52524        softirqs.CPU124.RCU
     32258 ± 15%     +63.3%      52680        softirqs.CPU125.RCU
     32480 ± 16%     +61.6%      52500        softirqs.CPU126.RCU
     32535 ± 17%     +60.8%      52330        softirqs.CPU127.RCU
     32059 ± 16%     +60.4%      51435        softirqs.CPU128.RCU
     32312 ± 16%     +61.8%      52278        softirqs.CPU129.RCU
      1056 ±137%   +3671.0%      39822        softirqs.CPU13.NET_RX
     40034 ± 11%     +51.2%      60514        softirqs.CPU13.RCU
    108802 ± 10%     +29.6%     141012        softirqs.CPU13.TIMER
     32215 ± 16%     +62.1%      52205        softirqs.CPU130.RCU
     32315 ± 16%     +61.1%      52075        softirqs.CPU131.RCU
     33006 ± 15%     +59.0%      52476        softirqs.CPU132.RCU
     32266 ± 17%     +61.9%      52250        softirqs.CPU133.RCU
     32626 ± 16%     +62.9%      53136        softirqs.CPU134.RCU
     32433 ± 18%     +58.8%      51494        softirqs.CPU135.RCU
     32294 ± 14%     +58.7%      51235        softirqs.CPU136.RCU
     31820 ± 16%     +63.5%      52021        softirqs.CPU137.RCU
     32892 ± 16%     +58.8%      52227        softirqs.CPU138.RCU
     32976 ± 15%     +57.9%      52060        softirqs.CPU139.RCU
     40138 ± 12%     +60.0%      64221        softirqs.CPU14.RCU
    117850 ±  9%     +15.9%     136643        softirqs.CPU14.TIMER
     32440 ± 16%     +59.4%      51714        softirqs.CPU140.RCU
     32239 ± 16%     +65.8%      53461        softirqs.CPU141.RCU
     32639 ± 16%     +59.7%      52137        softirqs.CPU142.RCU
     32350 ± 16%     +62.3%      52492        softirqs.CPU143.RCU
     32203 ± 16%     +61.8%      52101        softirqs.CPU144.RCU
     32170 ± 17%     +60.4%      51609        softirqs.CPU145.RCU
     32892 ± 16%     +65.1%      54293        softirqs.CPU146.RCU
     32355 ± 16%     +61.3%      52182        softirqs.CPU147.RCU
     32385 ± 16%     +60.7%      52047        softirqs.CPU148.RCU
     32356 ± 17%     +62.5%      52581        softirqs.CPU149.RCU
     38976 ± 16%     +61.0%      62748        softirqs.CPU15.RCU
    110967 ± 12%     +24.7%     138390        softirqs.CPU15.TIMER
     32399 ± 16%     +63.3%      52903        softirqs.CPU150.RCU
     32238 ± 15%     +57.9%      50889        softirqs.CPU151.RCU
     32360 ± 16%     +62.0%      52413        softirqs.CPU152.RCU
     32475 ± 16%     +60.2%      52015        softirqs.CPU153.RCU
     32209 ± 16%     +61.4%      51998        softirqs.CPU154.RCU
     32373 ± 16%     +61.2%      52181        softirqs.CPU155.RCU
     32295 ± 15%     +65.9%      53590        softirqs.CPU156.RCU
     32127 ± 16%     +57.1%      50483        softirqs.CPU157.RCU
     32514 ± 16%     +60.6%      52204        softirqs.CPU158.RCU
     32857 ± 16%     +58.2%      51977        softirqs.CPU159.RCU
     39215 ± 11%     +54.0%      60374        softirqs.CPU16.RCU
    116903 ± 13%     +28.9%     150659        softirqs.CPU16.TIMER
     32327 ± 16%     +61.5%      52199        softirqs.CPU160.RCU
     32410 ± 16%     +60.4%      51970        softirqs.CPU161.RCU
     32024 ± 16%     +60.8%      51503        softirqs.CPU162.RCU
     31125 ± 14%     +67.1%      52016        softirqs.CPU163.RCU
     32266 ± 16%     +60.0%      51633        softirqs.CPU164.RCU
     32209 ± 17%     +61.9%      52135        softirqs.CPU165.RCU
     32200 ± 16%     +61.4%      51956        softirqs.CPU166.RCU
     31951 ± 16%     +62.5%      51932        softirqs.CPU167.RCU
     31914 ± 16%     +62.4%      51836        softirqs.CPU168.RCU
     32081 ± 16%     +62.3%      52066        softirqs.CPU169.RCU
     39400 ± 11%     +52.2%      59956        softirqs.CPU17.RCU
    107863 ±  7%     +37.2%     147984        softirqs.CPU17.TIMER
     32016 ± 17%     +63.9%      52460        softirqs.CPU170.RCU
     31961 ± 16%     +62.7%      51996        softirqs.CPU171.RCU
     32978 ± 15%     +56.1%      51468        softirqs.CPU172.RCU
     32233 ± 16%     +62.1%      52246        softirqs.CPU173.RCU
     32152 ± 16%     +63.4%      52543        softirqs.CPU174.RCU
     32075 ± 16%     +62.0%      51952        softirqs.CPU175.RCU
     32196 ± 16%     +60.3%      51620        softirqs.CPU176.RCU
     32269 ± 17%     +59.5%      51462        softirqs.CPU177.RCU
     32096 ± 15%     +61.7%      51886        softirqs.CPU178.RCU
     32182 ± 16%     +60.2%      51570        softirqs.CPU179.RCU
     38279 ± 15%     +55.2%      59426        softirqs.CPU18.RCU
    117158 ± 13%     +31.0%     153465        softirqs.CPU18.TIMER
     31735 ± 16%     +62.2%      51480        softirqs.CPU180.RCU
     29335 ±  5%     +75.2%      51409        softirqs.CPU181.RCU
     31981 ± 16%     +60.1%      51205        softirqs.CPU182.RCU
     31948 ± 15%     +57.6%      50365        softirqs.CPU183.RCU
     32189 ± 17%     +59.1%      51205        softirqs.CPU184.RCU
     31957 ± 16%     +60.7%      51344        softirqs.CPU185.RCU
     32298 ± 17%     +58.2%      51080        softirqs.CPU186.RCU
     32335 ± 16%     +58.8%      51344        softirqs.CPU187.RCU
     32287 ± 16%     +59.6%      51540        softirqs.CPU188.RCU
     31965 ± 16%     +59.1%      50857        softirqs.CPU189.RCU
     39495 ± 14%     +53.9%      60800        softirqs.CPU19.RCU
    114131 ±  9%     +20.9%     138034        softirqs.CPU19.TIMER
     31213 ± 17%     +65.4%      51631        softirqs.CPU190.RCU
     31471 ± 17%     +64.1%      51645        softirqs.CPU191.RCU
     31760 ± 17%     +62.9%      51728        softirqs.CPU192.RCU
     32147 ± 16%     +61.8%      52016        softirqs.CPU193.RCU
     32079 ± 17%     +62.9%      52256        softirqs.CPU194.RCU
     31755 ± 16%     +63.2%      51815        softirqs.CPU195.RCU
     32240 ± 16%     +62.6%      52417        softirqs.CPU196.RCU
     32033 ± 16%     +60.6%      51435        softirqs.CPU197.RCU
     32041 ± 17%     +62.4%      52022        softirqs.CPU198.RCU
     31849 ± 17%     +64.2%      52294        softirqs.CPU199.RCU
     39674 ± 10%     +54.8%      61436        softirqs.CPU2.RCU
     40657 ± 17%     +48.6%      60422        softirqs.CPU20.RCU
    113465 ± 13%     +32.5%     150390        softirqs.CPU20.TIMER
     31959 ± 16%     +62.8%      52023        softirqs.CPU200.RCU
     31710 ± 16%     +62.6%      51548        softirqs.CPU201.RCU
     32021 ± 17%     +61.8%      51801        softirqs.CPU202.RCU
     32269 ± 16%     +59.5%      51463        softirqs.CPU203.RCU
     31922 ± 15%     +60.0%      51087        softirqs.CPU204.RCU
     31672 ± 17%     +63.4%      51740        softirqs.CPU205.RCU
     31875 ± 16%     +63.2%      52006        softirqs.CPU206.RCU
     31746 ± 16%     +63.5%      51898        softirqs.CPU207.RCU
     32414 ± 16%     +61.0%      52186        softirqs.CPU208.RCU
     32251 ± 16%     +60.8%      51875        softirqs.CPU209.RCU
     40977 ± 13%     +46.2%      59928        softirqs.CPU21.RCU
    109618 ± 13%     +23.9%     135789        softirqs.CPU21.TIMER
     32763 ± 15%     +58.2%      51825        softirqs.CPU210.RCU
     32054 ± 15%     +62.8%      52195        softirqs.CPU211.RCU
     32462 ± 16%     +61.6%      52468        softirqs.CPU212.RCU
     32438 ± 16%     +61.8%      52499        softirqs.CPU213.RCU
     31689 ± 17%     +62.5%      51498        softirqs.CPU214.RCU
     32181 ± 16%     +63.6%      52657        softirqs.CPU215.RCU
     32243 ± 16%     +60.5%      51745        softirqs.CPU216.RCU
     32122 ± 15%     +59.8%      51345        softirqs.CPU217.RCU
     32194 ± 16%     +62.6%      52340        softirqs.CPU218.RCU
     33401 ± 15%     +58.2%      52834        softirqs.CPU219.RCU
     39309 ± 19%     +53.5%      60334        softirqs.CPU22.RCU
    119248 ± 16%     +23.3%     147000        softirqs.CPU22.TIMER
     31767 ± 16%     +63.9%      52051        softirqs.CPU220.RCU
     32483 ± 15%     +60.4%      52113        softirqs.CPU221.RCU
     32330 ± 16%     +61.1%      52088        softirqs.CPU222.RCU
     32645 ± 15%     +59.9%      52206        softirqs.CPU223.RCU
     32122 ± 17%     +59.0%      51071        softirqs.CPU224.RCU
     32307 ± 16%     +60.0%      51695        softirqs.CPU225.RCU
     32176 ± 16%     +61.2%      51868        softirqs.CPU226.RCU
     32405 ± 16%     +60.4%      51994        softirqs.CPU227.RCU
     32186 ± 20%     +61.2%      51874        softirqs.CPU228.RCU
     31808 ± 16%     +63.5%      52008        softirqs.CPU229.RCU
     40782 ± 17%     +48.6%      60604        softirqs.CPU23.RCU
     32003 ± 16%     +58.8%      50807        softirqs.CPU230.RCU
     32424 ± 16%     +58.7%      51456        softirqs.CPU231.RCU
     32316 ± 16%     +60.7%      51924        softirqs.CPU232.RCU
     32725 ± 15%     +59.2%      52098        softirqs.CPU233.RCU
     31237 ± 15%     +65.4%      51666        softirqs.CPU234.RCU
     32479 ± 15%     +60.3%      52063        softirqs.CPU235.RCU
    133723 ±  7%     +15.4%     154367        softirqs.CPU235.TIMER
     32028 ± 16%     +62.2%      51963        softirqs.CPU236.RCU
     32136 ± 16%     +62.4%      52192        softirqs.CPU237.RCU
     32023 ± 16%     +60.5%      51408        softirqs.CPU238.RCU
     32055 ± 16%     +60.7%      51524        softirqs.CPU239.RCU
     39888 ± 17%     +53.7%      61300        softirqs.CPU24.RCU
    112430 ± 15%     +27.1%     142949        softirqs.CPU24.TIMER
     31745 ± 17%     +63.8%      52007        softirqs.CPU240.RCU
     32474 ± 16%     +60.9%      52242        softirqs.CPU241.RCU
     31904 ± 16%     +62.7%      51899        softirqs.CPU242.RCU
     32032 ± 16%     +61.9%      51868        softirqs.CPU243.RCU
     31357 ± 18%     +64.8%      51685        softirqs.CPU244.RCU
    132812 ±  8%     +15.3%     153157        softirqs.CPU244.TIMER
     31549 ± 16%     +64.3%      51841        softirqs.CPU245.RCU
     32027 ± 16%     +60.8%      51504        softirqs.CPU246.RCU
     32033 ± 16%     +77.1%      56745        softirqs.CPU247.RCU
    141833 ±  5%     +19.5%     169427        softirqs.CPU247.TIMER
     31859 ± 17%     +65.0%      52580        softirqs.CPU248.RCU
     32090 ± 16%     +62.2%      52055        softirqs.CPU249.RCU
     39971 ± 13%     +52.6%      61007        softirqs.CPU25.RCU
    114330 ± 12%     +28.9%     147423        softirqs.CPU25.TIMER
     31491 ± 16%     +64.9%      51918        softirqs.CPU250.RCU
     31766 ± 15%     +64.4%      52234        softirqs.CPU251.RCU
     32019 ± 16%     +61.0%      51544        softirqs.CPU252.RCU
     31922 ± 17%     +59.7%      50971        softirqs.CPU253.RCU
     31514 ± 16%     +64.1%      51705        softirqs.CPU254.RCU
     31765 ± 17%     +64.6%      52276        softirqs.CPU255.RCU
     31738 ± 16%     +60.0%      50780        softirqs.CPU256.RCU
     33985 ± 11%     +48.7%      50523        softirqs.CPU257.RCU
     31187 ± 17%     +57.1%      49000        softirqs.CPU258.RCU
     32031 ± 14%     +50.2%      48097        softirqs.CPU259.RCU
     39939 ± 11%     +53.3%      61220        softirqs.CPU26.RCU
    115081 ± 12%     +24.6%     143406        softirqs.CPU26.TIMER
     31401 ± 16%     +60.2%      50314        softirqs.CPU260.RCU
     30452 ± 16%     +62.8%      49574        softirqs.CPU261.RCU
     30819 ± 16%     +62.0%      49913        softirqs.CPU262.RCU
     31668 ± 22%     +57.4%      49847        softirqs.CPU263.RCU
     32298 ± 16%     +58.0%      51038        softirqs.CPU264.RCU
     31601 ± 13%     +57.3%      49722        softirqs.CPU265.RCU
    123387 ±  6%     +17.4%     144849        softirqs.CPU265.TIMER
     31810 ± 16%     +60.5%      51049        softirqs.CPU266.RCU
     32986 ± 14%     +51.1%      49844        softirqs.CPU267.RCU
     32309 ± 16%     +58.0%      51060        softirqs.CPU268.RCU
     32278 ± 16%     +59.2%      51371        softirqs.CPU269.RCU
     40744 ± 17%     +58.9%      64726        softirqs.CPU27.RCU
    108716 ± 13%     +41.4%     153739        softirqs.CPU27.TIMER
     32431 ± 16%     +56.5%      50753        softirqs.CPU270.RCU
     32242 ± 16%     +55.4%      50109        softirqs.CPU271.RCU
     33991 ± 15%     -25.3%      25388        softirqs.CPU272.RCU
    122760 ± 13%     +26.1%     154830        softirqs.CPU272.TIMER
     33868 ± 16%     +45.9%      49423        softirqs.CPU273.RCU
    125322 ± 12%     +22.9%     154051        softirqs.CPU273.TIMER
     33902 ± 15%     +57.9%      53546        softirqs.CPU274.RCU
     33622 ± 16%     +53.0%      51450        softirqs.CPU275.RCU
    133137 ±  8%     +16.5%     155075        softirqs.CPU275.TIMER
     33881 ± 16%     +59.1%      53894        softirqs.CPU276.RCU
     33983 ± 16%     +60.2%      54449        softirqs.CPU277.RCU
     33462 ± 15%     +57.7%      52778        softirqs.CPU278.RCU
    126550 ±  9%     +18.6%     150046        softirqs.CPU278.TIMER
     34169 ± 16%     +58.3%      54100        softirqs.CPU279.RCU
     40134 ± 12%     +54.1%      61856        softirqs.CPU28.RCU
    106399 ±  7%     +36.8%     145511        softirqs.CPU28.TIMER
     34216 ± 15%     +54.1%      52717        softirqs.CPU280.RCU
     34772 ± 14%     +49.9%      52127        softirqs.CPU281.RCU
     34184 ± 16%     +55.3%      53074        softirqs.CPU282.RCU
     33411 ± 17%     +57.4%      52591        softirqs.CPU283.RCU
     33482 ± 17%     +59.2%      53316        softirqs.CPU284.RCU
    124921 ±  7%     +18.4%     147846        softirqs.CPU284.TIMER
     34232 ± 16%     +59.9%      54735        softirqs.CPU285.RCU
     33099 ± 16%     +61.5%      53454        softirqs.CPU286.RCU
    128032 ±  6%     +15.8%     148252        softirqs.CPU286.TIMER
     33660 ± 14%     +60.2%      53931        softirqs.CPU287.RCU
     39735 ± 14%     +52.4%      60570        softirqs.CPU29.RCU
    109107 ± 10%     +31.4%     143385        softirqs.CPU29.TIMER
     38978 ± 17%     +58.8%      61916        softirqs.CPU3.RCU
    114894 ± 13%     +24.5%     143095        softirqs.CPU3.TIMER
     39122 ± 13%     +57.3%      61555        softirqs.CPU30.RCU
    112105 ± 11%     +29.4%     145113        softirqs.CPU30.TIMER
     40231 ± 15%     +58.2%      63642        softirqs.CPU31.RCU
    106758 ±  9%     +28.5%     137237        softirqs.CPU31.TIMER
     36607 ± 12%     +49.8%      54850        softirqs.CPU32.RCU
    107614 ± 10%     +30.6%     140576        softirqs.CPU32.TIMER
     38637 ± 11%     +41.5%      54675        softirqs.CPU33.RCU
    111703 ± 10%     +25.3%     139965        softirqs.CPU33.TIMER
     37654 ± 14%     +48.0%      55715        softirqs.CPU34.RCU
    108130 ± 10%     +29.2%     139690        softirqs.CPU34.TIMER
     37742 ± 14%     +48.5%      56044        softirqs.CPU35.RCU
    111809 ± 13%     +35.7%     151748        softirqs.CPU35.TIMER
     37760 ± 13%     +50.7%      56894        softirqs.CPU36.RCU
    104801 ±  6%     +41.3%     148074        softirqs.CPU36.TIMER
     38759 ± 10%     +44.3%      55925        softirqs.CPU37.RCU
    110484 ± 10%     +31.8%     145580        softirqs.CPU37.TIMER
     37881 ± 13%     +52.6%      57806        softirqs.CPU38.RCU
    111036 ± 14%     +28.4%     142517        softirqs.CPU38.TIMER
     37751 ± 13%     +53.5%      57963        softirqs.CPU39.RCU
    109975 ± 12%     +29.8%     142703        softirqs.CPU39.TIMER
     40700 ± 15%     +55.2%      63155        softirqs.CPU4.RCU
     33795 ± 15%     +53.5%      51887        softirqs.CPU40.RCU
     32637 ± 15%     +67.7%      54725        softirqs.CPU41.RCU
     33955 ± 15%     +66.4%      56488        softirqs.CPU42.RCU
    128703 ± 11%     +31.3%     168987        softirqs.CPU42.TIMER
     34606 ± 14%     +54.9%      53594        softirqs.CPU43.RCU
    124225 ± 11%     +22.8%     152583        softirqs.CPU43.TIMER
     34280 ± 14%     +61.1%      55211        softirqs.CPU44.RCU
     33700 ± 16%     +64.4%      55416        softirqs.CPU45.RCU
    122489 ±  7%     +23.4%     151156        softirqs.CPU45.TIMER
     34421 ± 14%     +60.7%      55327        softirqs.CPU46.RCU
    124335 ± 12%     +22.7%     152608        softirqs.CPU46.TIMER
     32622 ± 17%     +63.8%      53430        softirqs.CPU47.RCU
     34127 ± 16%     +59.8%      54523        softirqs.CPU48.RCU
     35165 ± 16%     +58.9%      55872        softirqs.CPU49.RCU
     40442 ± 11%     +56.7%      63356        softirqs.CPU5.RCU
    109994 ± 13%     +22.7%     135002        softirqs.CPU5.TIMER
     34820 ± 16%     +61.1%      56093        softirqs.CPU50.RCU
     35464 ± 15%     +59.4%      56524        softirqs.CPU51.RCU
    125051 ± 11%     +24.2%     155354        softirqs.CPU51.TIMER
     36036 ± 18%     +53.7%      55396        softirqs.CPU52.RCU
     35654 ± 16%     +52.6%      54416        softirqs.CPU53.RCU
     35691 ± 11%     +52.4%      54411        softirqs.CPU54.RCU
     34660 ± 13%     +53.3%      53123        softirqs.CPU55.RCU
    126943 ±  9%     +21.1%     153725        softirqs.CPU55.TIMER
     34264 ± 15%     +62.4%      55653        softirqs.CPU56.RCU
    125229 ±  7%     +20.9%     151461        softirqs.CPU56.TIMER
     34576 ± 16%     +62.1%      56057        softirqs.CPU57.RCU
     34536 ± 16%     +61.4%      55726        softirqs.CPU58.RCU
    129172 ±  5%     +16.5%     150459        softirqs.CPU58.TIMER
     35042 ± 16%     +57.1%      55050        softirqs.CPU59.RCU
     40606 ± 12%     +60.2%      65069        softirqs.CPU6.RCU
    109977 ±  9%     +23.9%     136257        softirqs.CPU6.TIMER
     35058 ± 16%     +57.7%      55277        softirqs.CPU60.RCU
     34951 ± 15%     +52.2%      53188        softirqs.CPU61.RCU
     35594 ± 14%     +54.4%      54950        softirqs.CPU62.RCU
     35217 ± 15%     +52.7%      53790        softirqs.CPU63.RCU
     33270 ± 15%     +59.3%      52990        softirqs.CPU64.RCU
     33326 ± 16%     +55.1%      51696        softirqs.CPU65.RCU
     33573 ± 15%     +56.8%      52631        softirqs.CPU66.RCU
     33582 ± 15%     +53.8%      51642        softirqs.CPU67.RCU
     32917 ± 15%     +59.9%      52644        softirqs.CPU68.RCU
     33037 ± 15%     +63.0%      53849        softirqs.CPU69.RCU
     39522 ± 10%     +60.3%      63357        softirqs.CPU7.RCU
     33226 ± 16%     +59.2%      52908        softirqs.CPU70.RCU
     32781 ± 15%     +58.6%      51989        softirqs.CPU71.RCU
     33064 ± 13%     +80.6%      59702        softirqs.CPU72.RCU
     32784 ± 14%     +56.8%      51422        softirqs.CPU73.RCU
     32988 ± 17%     +60.6%      52968        softirqs.CPU74.RCU
     32197 ± 15%     +61.3%      51922        softirqs.CPU75.RCU
     32210 ± 15%     +55.6%      50127        softirqs.CPU76.RCU
     31828 ± 15%     +57.7%      50193        softirqs.CPU77.RCU
     33704 ± 14%     +52.1%      51273        softirqs.CPU78.RCU
     33146 ± 17%     +53.4%      50850        softirqs.CPU79.RCU
     40019 ± 14%     +53.5%      61416        softirqs.CPU8.RCU
    122353 ±  9%     +21.5%     148650        softirqs.CPU8.TIMER
     32597 ± 15%     +64.3%      53549        softirqs.CPU80.RCU
    134840 ±  5%     +15.2%     155292        softirqs.CPU80.TIMER
     32221 ± 15%     +64.6%      53052        softirqs.CPU81.RCU
     33039 ± 15%     +59.6%      52743        softirqs.CPU82.RCU
     32787 ± 16%     +59.1%      52164        softirqs.CPU83.RCU
     32695 ± 15%     +58.8%      51926        softirqs.CPU84.RCU
     32467 ± 16%     +64.0%      53258        softirqs.CPU85.RCU
    136153 ±  7%     +14.7%     156116        softirqs.CPU85.TIMER
     32705 ± 16%     +62.3%      53086        softirqs.CPU86.RCU
     32601 ± 16%     +62.1%      52854        softirqs.CPU87.RCU
     32995 ± 15%     +59.9%      52757        softirqs.CPU88.RCU
     32822 ± 16%     +59.3%      52298        softirqs.CPU89.RCU
     40364 ± 14%     +52.0%      61352        softirqs.CPU9.RCU
    112908 ± 14%     +33.1%     150306        softirqs.CPU9.TIMER
     32824 ± 16%     +62.8%      53450        softirqs.CPU90.RCU
     32906 ± 16%     +64.4%      54093        softirqs.CPU91.RCU
     32596 ± 15%     +61.9%      52772        softirqs.CPU92.RCU
     32472 ± 16%     +66.6%      54103        softirqs.CPU93.RCU
     32897 ± 16%     +60.2%      52703        softirqs.CPU94.RCU
     32672 ± 16%     +62.5%      53086        softirqs.CPU95.RCU
     32311 ± 15%     +57.7%      50960        softirqs.CPU96.RCU
     32505 ± 15%     +58.6%      51551        softirqs.CPU97.RCU
     32357 ± 13%     +64.3%      53148        softirqs.CPU98.RCU
     32434 ± 16%     +63.6%      53073        softirqs.CPU99.RCU
      9259 ±127%    +353.8%      42022        softirqs.NET_RX
   9669747 ± 15%     +59.0%   15377610        softirqs.RCU



***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/context_switch1/will-it-scale/0x11

commit: 
  b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support")
  492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")

b121b341e5983bdc 492e639f0c222784e2e0f121966 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          2:2         -100%            :2     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          2:2         -100%            :2     dmesg.WARNING:stack_recursion





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.7.0-rc2-01182-g492e639f0c222" of type "text/plain" (202466 bytes)

View attachment "job-script" of type "text/plain" (7495 bytes)

View attachment "job.yaml" of type "text/plain" (5146 bytes)

View attachment "reproduce" of type "text/plain" (346 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ