lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200627082443.GI5535@shao2-debian>
Date:   Sat, 27 Jun 2020 16:24:43 +0800
From:   krenel test robot <rong.a.chen@...el.com>
To:     Xiyu Yang <xiyuyang19@...an.edu.cn>
Cc:     Christoph Hellwig <hch@....de>, Xin Tan <tanxin.ctf@...il.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [configfs] 8aebfffacf: will-it-scale.per_process_ops 10.7%
 improvement

Greeting,

FYI, we noticed a 10.7% improvement of will-it-scale.per_process_ops due to commit:


commit: 8aebfffacfa379ba400da573a5bf9e49634e38cb ("configfs: fix config_item refcnt leak in configfs_rmdir()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: will-it-scale
on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
with following parameters:

	nr_task: 100%
	mode: process
	test: mmap1
	cpufreq_governor: performance
	ucode: 0x11

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops 11.9% improvement        |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | mode=thread                                                          |
|                  | nr_task=100%                                                         |
|                  | test=open1                                                           |
|                  | ucode=0x11                                                           |
+------------------+----------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops 17.8% improvement        |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | mode=thread                                                          |
|                  | nr_task=100%                                                         |
|                  | test=sched_yield                                                     |
|                  | ucode=0x11                                                           |
+------------------+----------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -14.1% regression       |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | mode=process                                                         |
|                  | nr_task=100%                                                         |
|                  | test=page_fault2                                                     |
|                  | ucode=0x11                                                           |
+------------------+----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/mmap1/will-it-scale/0x11

commit: 
  v5.7-rc3
  8aebfffacf ("configfs: fix config_item refcnt leak in configfs_rmdir()")

        v5.7-rc3 8aebfffacfa379ba400da573a5b 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          5:6          -83%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :6           17%           1:4     dmesg.WARNING:at_ip_fsnotify_parent/0x
          2:6          -33%            :4     dmesg.WARNING:at_ip_perf_event_mmap_output/0x
         %stddev     %change         %stddev
             \          |                \  
      1471           +10.7%       1628        will-it-scale.per_process_ops
    158244           -93.8%       9886        will-it-scale.time.maximum_resident_set_size
     87613           -93.0%       6106        will-it-scale.time.minor_page_faults
      1.17 ± 31%    -100.0%       0.00        will-it-scale.time.percent_of_cpu_this_job_got
    423766           +10.7%     469037        will-it-scale.workload
     45.02            +4.4%      47.01 ±  2%  boot-time.boot
      0.54 ±  5%      +0.2        0.76 ±  8%  mpstat.cpu.all.usr%
   1381971           +11.0%    1534587        vmstat.memory.cache
    360.06          -100.0%       0.00        uptime.boot
     14088 ±  3%    -100.0%       0.00        uptime.idle
   1261069           +11.3%    1403860        meminfo.Cached
     48276 ± 22%     +48.4%      71641 ± 22%  meminfo.Mapped
   1096270           +12.8%    1236320        meminfo.Unevictable
    315278           +11.3%     350955        proc-vmstat.nr_file_pages
     12192 ± 22%     +47.3%      17953 ± 22%  proc-vmstat.nr_mapped
     30130            +8.1%      32577        proc-vmstat.nr_slab_reclaimable
    274067           +12.8%     309080        proc-vmstat.nr_unevictable
    274067           +12.8%     309080        proc-vmstat.nr_zone_unevictable
   1074698 ± 11%     -17.1%     890766 ±  5%  proc-vmstat.pgfree
    565456           +12.4%     635335        numa-meminfo.node0.Unevictable
    530814           +13.2%     600985        numa-meminfo.node1.FilePages
     17836 ±  6%     +27.3%      22702        numa-meminfo.node1.KReclaimable
     10865           +14.1%      12399        numa-meminfo.node1.Mapped
    623752           +11.9%     697704        numa-meminfo.node1.MemUsed
     17836 ±  6%     +27.3%      22702        numa-meminfo.node1.SReclaimable
    530814           +13.2%     600985        numa-meminfo.node1.Unevictable
    141363           +12.4%     158833        numa-vmstat.node0.nr_unevictable
    141363           +12.4%     158833        numa-vmstat.node0.nr_zone_unevictable
    154650           +11.3%     172062        numa-vmstat.node0.numa_interleave
    132703           +13.2%     150246        numa-vmstat.node1.nr_file_pages
      2715           +14.1%       3099        numa-vmstat.node1.nr_mapped
      4459 ±  6%     +27.3%       5675        numa-vmstat.node1.nr_slab_reclaimable
    132703           +13.2%     150246        numa-vmstat.node1.nr_unevictable
    132703           +13.2%     150246        numa-vmstat.node1.nr_zone_unevictable
    155463           +11.0%     172640        numa-vmstat.node1.numa_hit
    154878           +11.1%     172055        numa-vmstat.node1.numa_interleave
    154891           +11.1%     172068        numa-vmstat.node1.numa_other
    108714 ±  3%      -8.1%      99959 ±  5%  sched_debug.cfs_rq:/.exec_clock.min
  35857851 ±  3%      -7.8%   33046430 ±  5%  sched_debug.cfs_rq:/.min_vruntime.min
    566.50 ±  7%     -11.5%     501.41 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.avg
   2057901 ± 11%     +24.7%    2565766 ± 10%  sched_debug.cpu.avg_idle.avg
   1139755 ±  8%     +48.0%    1686683 ± 11%  sched_debug.cpu.avg_idle.stddev
    608.73 ±  7%     -14.8%     518.47 ±  3%  sched_debug.cpu.clock.stddev
    608.73 ±  7%     -14.8%     518.47 ±  3%  sched_debug.cpu.clock_task.stddev
   2502457 ± 25%     +52.1%    3805340 ± 21%  sched_debug.cpu.max_idle_balance_cost.max
    227444 ± 12%     +57.7%     358618 ± 12%  sched_debug.cpu.max_idle_balance_cost.stddev
      1059 ± 22%     -47.5%     556.70 ±  2%  sched_debug.cpu.nr_switches.min
    418.53 ±  4%     -12.7%     365.55 ±  3%  sched_debug.cpu.sched_count.min
      0.03 ± 82%   +1207.4%       0.45 ± 41%  sched_debug.rt_rq:/.rt_time.avg
      4.65 ±211%   +2609.5%     125.93 ± 46%  sched_debug.rt_rq:/.rt_time.max
      0.30 ±185%   +2337.7%       7.41 ± 46%  sched_debug.rt_rq:/.rt_time.stddev
     65.33 ± 31%    +157.1%     168.00        slabinfo.btrfs_extent_map.active_objs
     65.33 ± 31%    +157.1%     168.00        slabinfo.btrfs_extent_map.num_objs
      8668 ±  8%     +13.9%       9873 ±  2%  slabinfo.eventpoll_pwq.active_objs
      8668 ±  8%     +13.9%       9873 ±  2%  slabinfo.eventpoll_pwq.num_objs
     84377           +10.6%      93315        slabinfo.inode_cache.active_objs
      1567           +10.6%       1733        slabinfo.inode_cache.active_slabs
     84674           +10.6%      93652        slabinfo.inode_cache.num_objs
      1567           +10.6%       1733        slabinfo.inode_cache.num_slabs
    597.33 ± 14%     -27.7%     432.00 ± 19%  slabinfo.kmalloc-rcl-128.active_objs
    597.33 ± 14%     -27.7%     432.00 ± 19%  slabinfo.kmalloc-rcl-128.num_objs
      6319 ±  6%     +23.9%       7831 ±  9%  slabinfo.kmalloc-rcl-64.active_objs
      6319 ±  6%     +23.9%       7831 ±  9%  slabinfo.kmalloc-rcl-64.num_objs
     28831           +15.7%      33359        slabinfo.radix_tree_node.active_objs
     28831           +15.7%      33359        slabinfo.radix_tree_node.num_objs
    986.50 ±  5%    -100.0%       0.00        slabinfo.skbuff_ext_cache.active_objs
    986.50 ±  5%    -100.0%       0.00        slabinfo.skbuff_ext_cache.num_objs
      5697 ± 33%     -34.2%       3746        interrupts.CPU106.NMI:Non-maskable_interrupts
      5697 ± 33%     -34.2%       3746        interrupts.CPU106.PMI:Performance_monitoring_interrupts
      6325 ± 28%     -25.8%       4690 ± 35%  interrupts.CPU112.NMI:Non-maskable_interrupts
      6325 ± 28%     -25.8%       4690 ± 35%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
      5692 ± 33%     -17.6%       4691 ± 35%  interrupts.CPU124.NMI:Non-maskable_interrupts
      5692 ± 33%     -17.6%       4691 ± 35%  interrupts.CPU124.PMI:Performance_monitoring_interrupts
      9.33 ± 23%    +408.9%      47.50 ±111%  interrupts.CPU147.RES:Rescheduling_interrupts
     83.00 ± 32%     +54.2%     128.00 ± 15%  interrupts.CPU150.RES:Rescheduling_interrupts
    170.50 ± 26%     -38.7%     104.50 ± 22%  interrupts.CPU160.RES:Rescheduling_interrupts
    184.83 ± 75%     -52.4%      88.00 ± 51%  interrupts.CPU170.RES:Rescheduling_interrupts
     83.17 ± 15%    +149.8%     207.75 ± 16%  interrupts.CPU188.RES:Rescheduling_interrupts
     97.67 ± 37%     +68.7%     164.75 ± 30%  interrupts.CPU204.RES:Rescheduling_interrupts
    147.50 ± 55%     -52.2%      70.50 ± 81%  interrupts.CPU208.RES:Rescheduling_interrupts
      6327 ± 28%     -26.1%       4675 ± 35%  interrupts.CPU220.NMI:Non-maskable_interrupts
      6327 ± 28%     -26.1%       4675 ± 35%  interrupts.CPU220.PMI:Performance_monitoring_interrupts
      6941 ± 20%     -32.6%       4676 ± 35%  interrupts.CPU227.NMI:Non-maskable_interrupts
      6941 ± 20%     -32.6%       4676 ± 35%  interrupts.CPU227.PMI:Performance_monitoring_interrupts
      5678 ± 33%     -18.5%       4628 ± 31%  interrupts.CPU238.NMI:Non-maskable_interrupts
      5678 ± 33%     -18.5%       4628 ± 31%  interrupts.CPU238.PMI:Performance_monitoring_interrupts
      5669 ± 33%     -34.0%       3744        interrupts.CPU248.NMI:Non-maskable_interrupts
      5669 ± 33%     -34.0%       3744        interrupts.CPU248.PMI:Performance_monitoring_interrupts
     29.83 ± 50%    +202.5%      90.25 ± 59%  interrupts.CPU264.RES:Rescheduling_interrupts
      6324 ± 28%     -40.7%       3752        interrupts.CPU281.NMI:Non-maskable_interrupts
      6324 ± 28%     -40.7%       3752        interrupts.CPU281.PMI:Performance_monitoring_interrupts
     93.67 ± 20%     -56.8%      40.50 ± 33%  interrupts.CPU29.RES:Rescheduling_interrupts
     80.17 ± 22%    +231.8%     266.00 ± 64%  interrupts.CPU70.RES:Rescheduling_interrupts
      5699 ± 33%     -18.6%       4641 ± 32%  interrupts.CPU90.NMI:Non-maskable_interrupts
      5699 ± 33%     -18.6%       4641 ± 32%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
     35.57 ±  8%      -6.3       29.27        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.munmap
     35.56 ±  8%      -6.3       29.26        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     35.69 ±  8%      -6.3       29.40        perf-profile.calltrace.cycles-pp.munmap
     35.34 ±  8%      -6.3       29.05        perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     35.35 ±  8%      -6.3       29.07        perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     34.52 ±  8%      -5.9       28.63        perf-profile.calltrace.cycles-pp.mmap64
     34.37 ±  8%      -5.9       28.48        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
     34.38 ±  8%      -5.9       28.49        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.mmap64
     34.17 ±  8%      -5.9       28.31        perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
     34.16 ±  8%      -5.9       28.30        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
      0.73 ± 15%      -0.5        0.27 ±100%  perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.percpu_counter_add_batch.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.72 ± 13%      -0.5        0.26 ±100%  perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.percpu_counter_add_batch.__vm_enough_memory.mmap_region.do_mmap
      0.91 ±  7%      -0.2        0.68 ±  4%  perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.85 ±  7%      -0.2        0.64 ±  4%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap
     49.96            -0.2       49.76        perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     48.00            -0.2       47.81        perf-profile.calltrace.cycles-pp.percpu_counter_add_batch.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
     46.09            +0.4       46.48        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.percpu_counter_add_batch.__vm_enough_memory.mmap_region
     46.13            +0.4       46.53        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.percpu_counter_add_batch.__vm_enough_memory.mmap_region.do_mmap
     14.25 ± 19%      +6.0       20.24        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.26 ± 19%      +6.0       20.25        perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.75 ± 19%      +6.0       20.79        perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.76 ± 19%      +6.0       20.80        perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.34 ± 19%     +12.2       41.52        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.35 ± 19%     +12.2       41.54        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     35.71 ±  8%      -6.3       29.41        perf-profile.children.cycles-pp.munmap
     34.53 ±  8%      -5.9       28.64        perf-profile.children.cycles-pp.mmap64
      1.92 ±  9%      -0.4        1.57 ± 11%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.66 ±  9%      -0.3        1.33 ± 11%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.38 ±  6%      -0.3        1.11 ± 10%  perf-profile.children.cycles-pp.__hrtimer_run_queues
     50.09            -0.2       49.85        perf-profile.children.cycles-pp.__vm_munmap
     50.11            -0.2       49.88        perf-profile.children.cycles-pp.__x64_sys_munmap
     49.97            -0.2       49.77        perf-profile.children.cycles-pp.__do_munmap
      1.30            -0.1        1.17 ±  3%  perf-profile.children.cycles-pp.unmap_vmas
      1.25            -0.1        1.12 ±  3%  perf-profile.children.cycles-pp.unmap_page_range
      0.94 ±  3%      -0.1        0.83 ±  9%  perf-profile.children.cycles-pp.tick_sched_timer
      0.90 ±  3%      -0.1        0.79 ± 10%  perf-profile.children.cycles-pp.tick_sched_handle
      0.88 ±  3%      -0.1        0.78 ± 10%  perf-profile.children.cycles-pp.update_process_times
      0.70 ±  4%      -0.1        0.60 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.49 ±  2%      -0.1        0.42 ± 10%  perf-profile.children.cycles-pp.task_tick_fair
      0.12 ± 19%      -0.1        0.06 ± 20%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.59            -0.0        0.56 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.15 ±  5%      -0.0        0.12 ± 12%  perf-profile.children.cycles-pp.irq_exit
      0.14 ±  5%      -0.0        0.11 ±  9%  perf-profile.children.cycles-pp.update_curr
      0.11 ±  6%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.07 ±  5%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.remove_vma
      0.10 ±  9%      +0.0        0.13 ±  5%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.13 ±  7%      +0.0        0.16 ±  6%  perf-profile.children.cycles-pp.vm_area_alloc
      0.14 ±  5%      +0.0        0.18 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.kmem_cache_free
      0.01 ±223%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__might_sleep
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.05            +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.lru_add_drain
      0.02 ±141%      +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.lru_add_drain_cpu
     92.90            +0.5       93.36        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     93.01            +0.5       93.48        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.20 ±  6%      -0.1        0.13 ±  9%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.51 ±  2%      -0.1        0.43 ±  3%  perf-profile.self.cycles-pp.unmap_page_range
      0.09 ± 27%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.10 ± 14%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.perf_event_mmap
      0.56            -0.0        0.53 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.24 ±  3%      -0.0        0.22 ±  5%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.09 ±  5%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.12 ±  8%      +0.0        0.15 ± 10%  perf-profile.self.cycles-pp.perf_iterate_sb
      0.14 ±  5%      +0.0        0.18 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.kmem_cache_free
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.01 ±223%      +0.1        0.13 ±  6%  perf-profile.self.cycles-pp.lru_add_drain_cpu
     92.90            +0.5       93.36        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
    134459 ±  2%      -8.7%     122798        softirqs.CPU0.TIMER
    133411 ±  2%      -8.0%     122678        softirqs.CPU100.TIMER
    133333 ±  2%      -8.0%     122724        softirqs.CPU101.TIMER
    133598 ±  2%      -8.4%     122435        softirqs.CPU110.TIMER
    134909            -9.1%     122634        softirqs.CPU113.TIMER
    133179 ±  2%      -7.7%     122926 ±  2%  softirqs.CPU114.TIMER
    133436 ±  2%      -8.3%     122320        softirqs.CPU119.TIMER
    134596 ±  2%      -9.3%     122103        softirqs.CPU12.TIMER
    135545 ±  4%      -9.8%     122253        softirqs.CPU122.TIMER
    134335 ±  2%      -8.6%     122754        softirqs.CPU125.TIMER
    135177 ±  4%      -9.2%     122750        softirqs.CPU128.TIMER
    134010 ±  3%      -8.4%     122696        softirqs.CPU129.TIMER
    133968 ±  2%      -8.5%     122525        softirqs.CPU13.TIMER
    132997 ±  2%      -8.0%     122308        softirqs.CPU134.TIMER
    132866 ±  2%      -8.1%     122135        softirqs.CPU136.TIMER
    133212 ±  2%      -8.4%     122064        softirqs.CPU137.TIMER
    133362 ±  2%      -8.2%     122381        softirqs.CPU138.TIMER
    134902 ±  2%      -8.7%     123132        softirqs.CPU14.TIMER
    133290 ±  2%      -8.2%     122366        softirqs.CPU140.TIMER
    133372 ±  3%      -8.2%     122468        softirqs.CPU141.TIMER
    132826 ±  2%      -8.0%     122159        softirqs.CPU142.TIMER
    132860 ±  2%      -8.6%     121443        softirqs.CPU144.TIMER
    133202 ±  2%      -8.2%     122321        softirqs.CPU146.TIMER
    133355 ±  2%      -8.7%     121769        softirqs.CPU148.TIMER
    133645 ±  2%      -8.3%     122597        softirqs.CPU149.TIMER
    133171 ±  3%      -8.2%     122221        softirqs.CPU156.TIMER
    135643 ±  4%      -9.6%     122677        softirqs.CPU16.TIMER
    133388 ±  2%      -8.1%     122526        softirqs.CPU163.TIMER
    133749 ±  2%      -8.5%     122375        softirqs.CPU17.TIMER
    132919 ±  2%      -8.5%     121670        softirqs.CPU171.TIMER
    133345 ±  2%      -8.3%     122262        softirqs.CPU173.TIMER
    132748 ±  3%      -8.0%     122131        softirqs.CPU176.TIMER
    133989 ±  2%      -8.9%     122054        softirqs.CPU18.TIMER
    133046 ±  2%      -8.1%     122223        softirqs.CPU180.TIMER
    132979 ±  2%      -8.1%     122264        softirqs.CPU181.TIMER
    133378 ±  2%      -8.4%     122150        softirqs.CPU182.TIMER
    133506 ±  3%      -8.7%     121872        softirqs.CPU188.TIMER
    133634 ±  2%      -8.4%     122410        softirqs.CPU19.TIMER
    133179 ±  2%      -8.5%     121804        softirqs.CPU195.TIMER
    132997 ±  3%      -8.6%     121596        softirqs.CPU197.TIMER
    132919 ±  2%      -8.5%     121655        softirqs.CPU198.TIMER
    133502 ±  2%      -8.6%     121974        softirqs.CPU20.TIMER
    132247 ±  2%      -7.7%     122080        softirqs.CPU200.TIMER
    132682 ±  2%      -8.3%     121707        softirqs.CPU201.TIMER
    132349 ±  3%      -7.7%     122114        softirqs.CPU202.TIMER
    132434 ±  3%      -7.7%     122283        softirqs.CPU203.TIMER
    132590 ±  2%      -8.4%     121393        softirqs.CPU208.TIMER
    132398 ±  2%      -8.2%     121482        softirqs.CPU209.TIMER
    133514 ±  2%      -8.7%     121956        softirqs.CPU21.TIMER
    133008 ±  3%      -8.3%     121918        softirqs.CPU210.TIMER
    133510 ±  3%      -8.9%     121656        softirqs.CPU211.TIMER
    132399 ±  2%      -8.2%     121476        softirqs.CPU212.TIMER
    132399 ±  2%      -8.2%     121476        softirqs.CPU213.TIMER
    132523 ±  2%      -8.0%     121977        softirqs.CPU214.TIMER
    132473 ±  2%      -8.5%     121240        softirqs.CPU215.TIMER
    137798 ±  7%     -11.5%     121900        softirqs.CPU22.TIMER
    133101 ±  2%      -8.6%     121611        softirqs.CPU220.TIMER
    134009 ±  3%      -9.2%     121636        softirqs.CPU221.TIMER
    132697 ±  2%      -8.4%     121490        softirqs.CPU222.TIMER
    133397 ±  2%      -8.4%     122135        softirqs.CPU228.TIMER
    133531 ±  2%      -8.7%     121970        softirqs.CPU23.TIMER
    132213 ±  2%      -7.9%     121829        softirqs.CPU230.TIMER
    132975 ±  3%      -8.3%     121978        softirqs.CPU238.TIMER
    133584 ±  3%      -8.7%     121978        softirqs.CPU24.TIMER
    132432 ±  2%      -8.0%     121787        softirqs.CPU242.TIMER
    132561 ±  2%      -8.4%     121491        softirqs.CPU243.TIMER
    132407 ±  2%      -7.8%     122027 ±  2%  softirqs.CPU244.TIMER
    132955 ±  2%      -8.6%     121534        softirqs.CPU245.TIMER
    133660 ±  2%      -8.8%     121922        softirqs.CPU25.TIMER
    132138 ±  2%      -7.8%     121774        softirqs.CPU250.TIMER
    132361 ±  3%      -8.2%     121514        softirqs.CPU252.TIMER
    132422 ±  3%      -8.2%     121586        softirqs.CPU253.TIMER
    137025 ±  7%     -11.3%     121487        softirqs.CPU254.TIMER
    132352 ±  2%      -8.4%     121197        softirqs.CPU255.TIMER
    131979 ±  2%      -8.2%     121176        softirqs.CPU256.TIMER
    132151 ±  2%      -8.4%     121098        softirqs.CPU257.TIMER
    132837 ±  3%      -8.5%     121489        softirqs.CPU260.TIMER
    132653            -8.8%     120999 ±  2%  softirqs.CPU266.TIMER
    132231 ±  2%      -8.9%     120478        softirqs.CPU269.TIMER
    133698 ±  2%      -8.7%     122032        softirqs.CPU27.TIMER
    132110 ±  2%      -8.9%     120375        softirqs.CPU270.TIMER
    132329 ±  2%      -9.0%     120457        softirqs.CPU272.TIMER
    132508 ±  2%      -8.8%     120911        softirqs.CPU273.TIMER
    131921 ±  2%      -8.6%     120544        softirqs.CPU274.TIMER
    132230 ±  2%      -8.7%     120729        softirqs.CPU275.TIMER
    132181 ±  2%      -8.8%     120573        softirqs.CPU276.TIMER
    134398 ±  5%     -10.1%     120833        softirqs.CPU277.TIMER
    131843 ±  2%      -8.5%     120602        softirqs.CPU278.TIMER
    133421 ±  2%      -8.5%     122025        softirqs.CPU28.TIMER
    131856 ±  2%      -8.7%     120359        softirqs.CPU280.TIMER
    132034 ±  2%      -8.7%     120592        softirqs.CPU281.TIMER
    132557 ±  2%      -9.0%     120614        softirqs.CPU282.TIMER
    133019 ±  2%      -9.7%     120142        softirqs.CPU283.TIMER
    131831 ±  2%      -8.5%     120600        softirqs.CPU284.TIMER
    132329 ±  2%      -8.6%     120971        softirqs.CPU285.TIMER
    131836 ±  2%      -8.9%     120073        softirqs.CPU286.TIMER
    126308 ±  3%     -14.3%     108210 ±  7%  softirqs.CPU287.TIMER
    133749 ±  2%      -8.7%     122054        softirqs.CPU29.TIMER
    133821 ±  2%      -9.0%     121831        softirqs.CPU3.TIMER
    133931            -8.9%     122056        softirqs.CPU30.TIMER
    133730 ±  2%      -8.8%     121904        softirqs.CPU31.TIMER
    135936 ±  3%     -10.5%     121705        softirqs.CPU32.TIMER
    133364 ±  2%      -8.7%     121789        softirqs.CPU33.TIMER
    133499            -8.8%     121705        softirqs.CPU34.TIMER
    134336 ±  2%      -9.2%     121940        softirqs.CPU35.TIMER
    133118 ±  2%      -8.3%     122113        softirqs.CPU36.TIMER
    133775 ±  2%      -9.0%     121748        softirqs.CPU37.TIMER
    133045 ±  2%      -8.3%     121998        softirqs.CPU38.TIMER
    132874 ±  2%      -8.0%     122287        softirqs.CPU39.TIMER
    134208 ±  2%      -7.4%     124329 ±  4%  softirqs.CPU4.TIMER
    133599 ±  2%      -8.6%     122060        softirqs.CPU41.TIMER
    133991 ±  2%      -8.9%     122058        softirqs.CPU42.TIMER
    133783 ±  2%      -8.7%     122132        softirqs.CPU45.TIMER
    133698 ±  2%      -8.4%     122501        softirqs.CPU47.TIMER
    133904 ±  2%      -8.9%     121925        softirqs.CPU5.TIMER
    134059 ±  2%      -8.6%     122491        softirqs.CPU53.TIMER
    140548 ± 11%     -13.0%     122285        softirqs.CPU55.TIMER
    134013 ±  2%      -8.8%     122274        softirqs.CPU56.TIMER
    135542 ±  4%      -9.3%     122942        softirqs.CPU57.TIMER
    133683 ±  2%      -8.7%     122070        softirqs.CPU58.TIMER
    133734 ±  2%      -8.2%     122801        softirqs.CPU6.TIMER
    133469 ±  2%      -8.4%     122305        softirqs.CPU60.TIMER
    136154 ±  4%     -10.1%     122353        softirqs.CPU61.TIMER
    133819 ±  2%      -8.9%     121944        softirqs.CPU62.TIMER
    133824 ±  2%      -9.3%     121414        softirqs.CPU63.TIMER
    133697 ±  2%      -8.5%     122288        softirqs.CPU64.TIMER
    133719 ±  2%      -8.8%     122016        softirqs.CPU66.TIMER
    133634 ±  2%      -8.9%     121724        softirqs.CPU67.TIMER
    133851 ±  2%      -8.5%     122495        softirqs.CPU68.TIMER
    133618 ±  2%      -8.5%     122325        softirqs.CPU69.TIMER
    133816 ±  2%      -8.7%     122110        softirqs.CPU70.TIMER
    133338 ±  2%      -8.2%     122340        softirqs.CPU71.TIMER
      8074 ± 32%     -43.6%       4557        softirqs.CPU73.RCU
    139691 ±  5%     -12.2%     122619        softirqs.CPU73.TIMER
    134414 ±  2%      -8.7%     122695        softirqs.CPU76.TIMER
    134661 ±  2%      -8.9%     122674        softirqs.CPU77.TIMER
    133670 ±  2%      -8.7%     122040        softirqs.CPU8.TIMER
    134093 ±  2%      -8.4%     122854        softirqs.CPU80.TIMER
    133919 ±  2%      -8.6%     122458        softirqs.CPU83.TIMER
    133776 ±  2%      -8.2%     122797        softirqs.CPU86.TIMER
    135212 ±  4%      -9.4%     122496        softirqs.CPU89.TIMER
    133802 ±  2%      -8.9%     121958        softirqs.CPU9.TIMER
    134030 ±  2%      -8.6%     122545        softirqs.CPU91.TIMER
    134236 ±  3%      -8.4%     123012        softirqs.CPU94.TIMER
    138956 ±  7%     -11.4%     123126        softirqs.CPU96.TIMER
      1565 ±  5%    +381.2%       7535 ± 49%  softirqs.NET_RX
  38399366 ±  2%      -8.2%   35256509        softirqs.TIMER


                                                                                
                           will-it-scale.per_process_ops                        
                                                                                
  1800 +--------------------------------------------------------------------+   
       |  O   O  O      O   O  O   O  O   O  O  O   O  O   O  O   O  O      |   
  1600 |..+                      ..+..      .+..+...+..+...     ..+..+...+..|   
  1400 |-+:             +...+..+.     +...+.               +..+.            |   
       |   :            :                                                   |   
  1200 |-+ :           :                                                    |   
  1000 |-+ :           :                                                    |   
       |    :          :                                                    |   
   800 |-+  :          :                                                    |   
   600 |-+  :         :                                                     |   
       |     :        :                                                     |   
   400 |-+   :        :                                                     |   
   200 |-+   :        :                                                     |   
       |      :      :                                                      |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               will-it-scale.workload                           
                                                                                
  500000 +------------------------------------------------------------------+   
  450000 |-+O   O  O      O  O  O   O  O   O  O  O   O  O  O   O  O  O      |   
         |..+             +..+..+...+..  ..+..+..+...+..+..  ..+..+..+...+..|   
  400000 |-+:             :            +.                  +.               |   
  350000 |-+ :           :                                                  |   
         |   :           :                                                  |   
  300000 |-+ :           :                                                  |   
  250000 |-+  :         :                                                   |   
  200000 |-+  :         :                                                   |   
         |    :         :                                                   |   
  150000 |-+   :       :                                                    |   
  100000 |-+   :       :                                                    |   
         |     :       :                                                    |   
   50000 |-+    :     :                                                     |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/open1/will-it-scale/0x11

commit: 
  v5.7-rc3
  8aebfffacf ("configfs: fix config_item refcnt leak in configfs_rmdir()")

        v5.7-rc3 8aebfffacfa379ba400da573a5b 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          5:5         -100%            :1     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :5           20%           1:1     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          4:5          -80%            :1     dmesg.WARNING:at_ip_fsnotify_parent/0x
         %stddev     %change         %stddev
             \          |                \  
    888.20           +11.9%     994.00        will-it-scale.per_thread_ops
    542327          -100.0%       6.00        will-it-scale.time.involuntary_context_switches
    244880           -96.0%       9772        will-it-scale.time.maximum_resident_set_size
     52934           -88.5%       6070        will-it-scale.time.minor_page_faults
     28447          -100.0%       0.00        will-it-scale.time.percent_of_cpu_this_job_got
     87059          -100.0%       0.19        will-it-scale.time.system_time
    313.04 ±  3%     -99.9%       0.24        will-it-scale.time.user_time
      2065 ± 21%     -96.8%      67.00        will-it-scale.time.voluntary_context_switches
    255979           +11.9%     286557        will-it-scale.workload
     45.76 ±  2%      +3.8%      47.50        boot-time.boot
     11391 ±  2%      +3.9%      11836        boot-time.idle
      0.06 ±  5%      +0.1        0.11        mpstat.cpu.all.soft%
      0.50 ±  2%      +0.2        0.73        mpstat.cpu.all.usr%
    360.53          -100.0%       0.00        uptime.boot
     14281          -100.0%       0.00        uptime.idle
   1366998           +10.9%    1515912        vmstat.memory.cache
      4322           +40.8%       6087        vmstat.system.cs
   1247353           +11.1%    1386310        meminfo.Cached
    681.20 ±122%    +150.6%       1707        meminfo.Mlocked
   1095965           +12.9%    1237288        meminfo.Unevictable
    565128           +12.6%     636336        numa-meminfo.node0.Unevictable
    530836           +13.2%     600952        numa-meminfo.node1.FilePages
     17615 ±  9%     +32.5%      23348        numa-meminfo.node1.KReclaimable
     10925           +13.8%      12429        numa-meminfo.node1.Mapped
    622609           +12.2%     698516        numa-meminfo.node1.MemUsed
     17615 ±  9%     +32.5%      23348        numa-meminfo.node1.SReclaimable
     31688 ±  8%     +18.7%      37628        numa-meminfo.node1.Slab
    530836           +13.2%     600952        numa-meminfo.node1.Unevictable
    141281           +12.6%     159083        numa-vmstat.node0.nr_unevictable
    141281           +12.6%     159083        numa-vmstat.node0.nr_zone_unevictable
    154743           +11.2%     172123        numa-vmstat.node0.numa_interleave
    132709           +13.2%     150238        numa-vmstat.node1.nr_file_pages
      2731           +13.7%       3105        numa-vmstat.node1.nr_mapped
      4403 ±  9%     +32.5%       5837        numa-vmstat.node1.nr_slab_reclaimable
    132709           +13.2%     150238        numa-vmstat.node1.nr_unevictable
    132709           +13.2%     150238        numa-vmstat.node1.nr_zone_unevictable
    155367           +11.1%     172552        numa-vmstat.node1.numa_hit
    154782           +11.1%     171964        numa-vmstat.node1.numa_interleave
    154795           +11.1%     171976        numa-vmstat.node1.numa_other
    125539            +1.4%     127359        proc-vmstat.nr_anon_pages
    311850           +11.1%     346545        proc-vmstat.nr_file_pages
     11616            +5.4%      12242        proc-vmstat.nr_mapped
     29822            +8.5%      32345        proc-vmstat.nr_slab_reclaimable
     87439            +1.4%      88664        proc-vmstat.nr_slab_unreclaimable
    273990           +12.9%     309322        proc-vmstat.nr_unevictable
    273990           +12.9%     309322        proc-vmstat.nr_zone_unevictable
      6518 ±  3%      -9.2%       5921        proc-vmstat.numa_hint_faults
      6518 ±  3%      -9.2%       5921        proc-vmstat.numa_hint_faults_local
    894126            -1.5%     880458        proc-vmstat.numa_hit
    894126            -1.5%     880458        proc-vmstat.numa_local
    356019            +7.0%     380894        proc-vmstat.numa_pte_updates
    846963            -1.9%     830953        proc-vmstat.pgfault
    962825            -7.8%     887792        proc-vmstat.pgfree
      1511 ±  6%     +27.9%       1933        slabinfo.file_lock_cache.active_objs
      1511 ±  6%     +27.9%       1933        slabinfo.file_lock_cache.num_objs
     84372           +10.6%      93300        slabinfo.inode_cache.active_objs
      1567           +10.6%       1733        slabinfo.inode_cache.active_slabs
     84657           +10.6%      93607        slabinfo.inode_cache.num_objs
      1567           +10.6%       1733        slabinfo.inode_cache.num_slabs
      2465 ± 10%     +27.2%       3137        slabinfo.khugepaged_mm_slot.active_objs
      2465 ± 10%     +27.2%       3137        slabinfo.khugepaged_mm_slot.num_objs
      6099 ±  8%     +16.9%       7129        slabinfo.kmalloc-rcl-64.active_objs
      6099 ±  8%     +16.9%       7129        slabinfo.kmalloc-rcl-64.num_objs
    657.20 ±  7%     +22.6%     806.00        slabinfo.numa_policy.active_objs
    657.20 ±  7%     +22.6%     806.00        slabinfo.numa_policy.num_objs
     28780           +15.5%      33230        slabinfo.radix_tree_node.active_objs
     28780           +15.5%      33230        slabinfo.radix_tree_node.num_objs
      1136 ±  4%    -100.0%       0.00        slabinfo.skbuff_ext_cache.active_objs
      1136 ±  4%    -100.0%       0.00        slabinfo.skbuff_ext_cache.num_objs
      1026 ±  4%     -15.2%     870.00        slabinfo.skbuff_fclone_cache.active_objs
      1026 ±  4%     -15.2%     870.00        slabinfo.skbuff_fclone_cache.num_objs
      1255 ±  3%     +15.5%       1450        slabinfo.task_group.active_objs
      1255 ±  3%     +15.5%       1450        slabinfo.task_group.num_objs
      1472           -18.5%       1200        sched_debug.cfs_rq:/.exec_clock.stddev
      5.09 ±  5%     +24.2%       6.32        sched_debug.cfs_rq:/.load_avg.avg
    147.32 ± 36%     -38.2%      91.00        sched_debug.cfs_rq:/.load_avg.max
    644505           -15.9%     541941        sched_debug.cfs_rq:/.min_vruntime.stddev
    128.50 ± 40%     +80.8%     232.36        sched_debug.cfs_rq:/.nr_spread_over.avg
    254.64 ± 21%     +46.5%     373.00        sched_debug.cfs_rq:/.nr_spread_over.max
     19.72 ±  8%     +84.7%      36.42        sched_debug.cfs_rq:/.nr_spread_over.stddev
     47.08 ±  7%     +13.0%      53.19        sched_debug.cfs_rq:/.util_avg.stddev
   8576310 ± 11%     +23.4%   10584065        sched_debug.cpu.avg_idle.max
      1270           -33.7%     842.03        sched_debug.cpu.clock.stddev
      1270           -33.7%     842.03        sched_debug.cpu.clock_task.stddev
   1969070 ± 10%     +36.3%    2683386        sched_debug.cpu.max_idle_balance_cost.max
    207363 ± 10%     +17.2%     243056        sched_debug.cpu.max_idle_balance_cost.stddev
      0.00           -33.5%       0.00        sched_debug.cpu.next_balance.stddev
      4788           +15.0%       5504        sched_debug.cpu.nr_switches.avg
     22523 ± 12%     +34.6%      30319        sched_debug.cpu.nr_switches.max
      1789 ± 19%     +53.3%       2743        sched_debug.cpu.nr_switches.min
      1922 ±  5%     +15.5%       2220        sched_debug.cpu.nr_switches.stddev
      0.00 ± 54%    -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.avg
      2066           +33.0%       2748        sched_debug.cpu.sched_count.avg
     18124 ± 17%     +43.4%      25987        sched_debug.cpu.sched_count.max
      1336           +45.4%       1943        sched_debug.cpu.sched_count.min
      1489 ±  9%     +21.2%       1805        sched_debug.cpu.sched_count.stddev
    906.60           +38.5%       1255        sched_debug.cpu.ttwu_count.avg
      8824 ± 16%     +42.9%      12610        sched_debug.cpu.ttwu_count.max
    600.40           +51.8%     911.40        sched_debug.cpu.ttwu_count.min
    771.87 ±  8%     +19.8%     924.84        sched_debug.cpu.ttwu_count.stddev
    838.79           +41.8%       1189        sched_debug.cpu.ttwu_local.avg
      8251 ± 17%     +47.4%      12161        sched_debug.cpu.ttwu_local.max
    592.04           +50.8%     893.00        sched_debug.cpu.ttwu_local.min
    671.44 ± 11%     +24.5%     836.15        sched_debug.cpu.ttwu_local.stddev
      0.02 ± 13%   +2158.1%       0.49        sched_debug.rt_rq:/.rt_time.avg
      0.23 ± 23%  +61069.8%     140.99        sched_debug.rt_rq:/.rt_time.max
      0.05 ±  8%  +15455.0%       8.29        sched_debug.rt_rq:/.rt_time.stddev
     17897           +10.0%      19690        softirqs.CPU102.RCU
     17584 ±  3%      +8.6%      19097        softirqs.CPU104.RCU
    118027 ± 13%     +17.3%     138420        softirqs.CPU105.TIMER
      1567 ±146%   +1449.6%      24286        softirqs.CPU11.NET_RX
     20056 ±  8%     +47.9%      29665        softirqs.CPU11.RCU
    105076 ±  9%     +14.9%     120696        softirqs.CPU11.TIMER
    125179 ± 10%     +19.2%     149205        softirqs.CPU112.TIMER
    122851 ± 12%     +19.7%     147077        softirqs.CPU122.TIMER
    295.00 ± 70%   +3917.3%      11851        softirqs.CPU13.NET_RX
     19060 ±  3%     +48.6%      28327        softirqs.CPU13.RCU
    118005 ± 12%     +20.9%     142700        softirqs.CPU134.TIMER
    115236 ±  9%     +17.7%     135682        softirqs.CPU135.TIMER
    124737 ± 12%     +17.5%     146568        softirqs.CPU138.TIMER
    118041 ± 11%     +16.2%     137150        softirqs.CPU146.TIMER
    117207 ± 13%     +22.4%     143500        softirqs.CPU147.TIMER
    123578 ± 12%     +15.2%     142420        softirqs.CPU152.TIMER
    119024 ± 12%     +18.6%     141137        softirqs.CPU153.TIMER
    114261 ± 13%     +23.4%     141010        softirqs.CPU157.TIMER
    127481 ± 11%     +14.9%     146417        softirqs.CPU159.TIMER
     19230 ±  4%      +9.6%      21072        softirqs.CPU16.RCU
    124958 ± 11%     +14.9%     143591        softirqs.CPU161.TIMER
    124030 ± 13%     +16.2%     144177        softirqs.CPU165.TIMER
    115582 ± 14%     +19.0%     137497        softirqs.CPU173.TIMER
     17444 ±  2%     +17.3%      20465        softirqs.CPU182.RCU
    120903 ± 13%     +19.7%     144709        softirqs.CPU187.TIMER
    116780 ± 12%     +17.5%     137227        softirqs.CPU195.TIMER
     17548 ±  3%     +17.4%      20609        softirqs.CPU196.RCU
    116381 ± 13%     +19.5%     139093        softirqs.CPU199.TIMER
     17195 ±  2%      +9.3%      18790        softirqs.CPU200.RCU
    118401 ± 11%     +16.4%     137831        softirqs.CPU200.TIMER
    124742 ± 11%     +15.4%     143893        softirqs.CPU202.TIMER
    117531 ± 11%     +18.9%     139749        softirqs.CPU215.TIMER
    116289 ± 11%     +19.6%     139030        softirqs.CPU216.TIMER
    116587 ± 11%     +23.7%     144193        softirqs.CPU220.TIMER
     17695 ±  3%      +8.1%      19135        softirqs.CPU221.RCU
    117205 ± 11%     +19.4%     139956        softirqs.CPU221.TIMER
    119762 ± 11%     +20.7%     144564        softirqs.CPU225.TIMER
     17214 ±  2%      +9.1%      18776        softirqs.CPU229.RCU
    122398 ± 11%     +17.4%     143709        softirqs.CPU233.TIMER
    112338 ± 13%     +17.5%     131952        softirqs.CPU238.TIMER
     17326           +15.2%      19952        softirqs.CPU239.RCU
    119806 ± 11%     +19.6%     143316        softirqs.CPU246.TIMER
    118281 ± 13%     +17.1%     138534        softirqs.CPU247.TIMER
    110402 ± 13%     +20.4%     132888        softirqs.CPU256.TIMER
     17218 ±  2%     +40.6%      24209        softirqs.CPU265.RCU
    103283 ±  8%     +71.8%     177475        softirqs.CPU265.TIMER
     18905 ±  2%     +11.8%      21138        softirqs.CPU28.RCU
     17230 ±  3%     +10.3%      18999        softirqs.CPU282.RCU
     19392 ±  4%     +21.9%      23632        softirqs.CPU30.RCU
     18951 ±  3%     +18.4%      22433        softirqs.CPU31.RCU
    105516 ±  6%     -11.3%      93580        softirqs.CPU32.TIMER
     18749 ±  2%     +13.1%      21201        softirqs.CPU34.RCU
    108972 ± 10%     +22.1%     133022        softirqs.CPU43.TIMER
     18723 ±  3%     +27.8%      23926        softirqs.CPU53.RCU
    109283 ± 10%     +22.0%     133326        softirqs.CPU64.TIMER
     18586 ±  9%     +45.0%      26960        softirqs.CPU72.RCU
    117359 ± 13%     +30.5%     153156        softirqs.CPU72.TIMER
     17989           +24.6%      22407        softirqs.CPU75.RCU
    112037 ± 12%     +19.8%     134168        softirqs.CPU75.TIMER
    115165 ± 12%     +21.8%     140265        softirqs.CPU80.TIMER
    110504 ± 12%     +17.9%     130255        softirqs.CPU81.TIMER
    122916 ± 12%     +16.8%     143533        softirqs.CPU82.TIMER
    116896 ± 12%     +17.2%     137045        softirqs.CPU84.TIMER
    118538 ± 10%     +13.6%     134673        softirqs.CPU85.TIMER
    116481 ± 12%     +19.4%     139126        softirqs.CPU93.TIMER
      3411 ± 68%   +1005.0%      37695        softirqs.NET_RX
     48.48            -0.3       48.13        perf-profile.calltrace.cycles-pp.__alloc_fd.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     48.22            -0.3       47.94        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__alloc_fd.do_sys_openat2.do_sys_open
     49.17            -0.3       48.91        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_close
     49.15            -0.3       48.90        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close
     49.95            -0.2       49.72        perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_open
     49.94            -0.2       49.71        perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_open
     48.20            -0.2       47.98        perf-profile.calltrace.cycles-pp._raw_spin_lock.__alloc_fd.do_sys_openat2.do_sys_open.do_syscall_64
     48.62            -0.2       48.41        perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close
     50.02            -0.2       49.81        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_open
     50.03            -0.2       49.82        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_open
     48.47            -0.2       48.27        perf-profile.calltrace.cycles-pp.__close_fd.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close
     49.34            -0.2       49.15        perf-profile.calltrace.cycles-pp.__GI___libc_close
     48.28            -0.2       48.10        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__close_fd.__x64_sys_close.do_syscall_64
     50.22            -0.2       50.06        perf-profile.calltrace.cycles-pp.__GI___libc_open
     48.27            -0.1       48.14        perf-profile.calltrace.cycles-pp._raw_spin_lock.__close_fd.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.89 ±  3%      +0.2        3.11        perf-profile.calltrace.cycles-pp.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.__alloc_fd.do_sys_openat2
      2.77 ±  3%      +0.2        3.01        perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.__alloc_fd
      3.55 ±  3%      +0.2        3.80        perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath
      2.91 ±  4%      +0.3        3.18        perf-profile.calltrace.cycles-pp.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.__close_fd.__x64_sys_close
      2.79 ±  4%      +0.3        3.07        perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.__close_fd
      4.56 ±  3%      +0.3        4.91        perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
     96.64            -0.5       96.17        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     96.69            -0.5       96.23        perf-profile.children.cycles-pp._raw_spin_lock
     48.51            -0.3       48.16        perf-profile.children.cycles-pp.__alloc_fd
     99.46            -0.3       99.16        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     99.43            -0.3       99.13        perf-profile.children.cycles-pp.do_syscall_64
     49.95            -0.2       49.72        perf-profile.children.cycles-pp.do_sys_openat2
     49.96            -0.2       49.73        perf-profile.children.cycles-pp.do_sys_open
     48.62            -0.2       48.41        perf-profile.children.cycles-pp.__x64_sys_close
     48.47            -0.2       48.27        perf-profile.children.cycles-pp.__close_fd
     49.35            -0.2       49.16        perf-profile.children.cycles-pp.__GI___libc_close
     50.22            -0.2       50.07        perf-profile.children.cycles-pp.__GI___libc_open
      0.17 ± 13%      -0.1        0.08        perf-profile.children.cycles-pp.update_rq_clock
      0.49            -0.1        0.41        perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.45            -0.1        0.37        perf-profile.children.cycles-pp.task_work_run
      0.48 ±  2%      -0.1        0.41        perf-profile.children.cycles-pp.alloc_empty_file
      0.34            -0.1        0.28        perf-profile.children.cycles-pp.kmem_cache_alloc
      0.16 ± 10%      -0.0        0.11        perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.14 ± 22%      -0.0        0.09        perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.09 ± 10%      -0.0        0.05        perf-profile.children.cycles-pp.expand_files
      0.30            -0.0        0.26        perf-profile.children.cycles-pp.__fput
      0.42 ±  2%      -0.0        0.38        perf-profile.children.cycles-pp.__alloc_file
      0.14 ±  5%      -0.0        0.11        perf-profile.children.cycles-pp.security_file_alloc
      0.11 ±  4%      -0.0        0.10        perf-profile.children.cycles-pp.strncpy_from_user
      0.06            -0.0        0.05        perf-profile.children.cycles-pp.security_file_open
      0.16 ±  2%      +0.0        0.17        perf-profile.children.cycles-pp.link_path_walk
      0.06 ±  6%      +0.0        0.08        perf-profile.children.cycles-pp.tick_program_event
      0.15 ±  4%      +0.0        0.17        perf-profile.children.cycles-pp.do_dentry_open
      0.04 ± 50%      +0.0        0.06        perf-profile.children.cycles-pp.lapic_next_deadline
      0.05            +0.0        0.07        perf-profile.children.cycles-pp.dput
      0.19 ±  2%      +0.0        0.21        perf-profile.children.cycles-pp.getname_flags
      0.05 ± 50%      +0.0        0.07        perf-profile.children.cycles-pp.native_write_msr
      0.13 ±  6%      +0.0        0.15        perf-profile.children.cycles-pp.enqueue_hrtimer
      0.11 ±  7%      +0.0        0.13        perf-profile.children.cycles-pp.timerqueue_add
      0.06 ±  7%      +0.0        0.09        perf-profile.children.cycles-pp.___might_sleep
      0.16 ±  9%      +0.0        0.19        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.03 ± 81%      +0.0        0.06        perf-profile.children.cycles-pp.fsnotify
      0.03 ± 81%      +0.0        0.06        perf-profile.children.cycles-pp._cond_resched
      0.19 ± 29%      +0.0        0.24        perf-profile.children.cycles-pp.rb_next
      0.09 ± 23%      +0.1        0.14        perf-profile.children.cycles-pp.rcu_irq_enter
      0.01 ±200%      +0.1        0.07        perf-profile.children.cycles-pp.shmem_getpage_gfp
      0.03 ± 81%      +0.1        0.09        perf-profile.children.cycles-pp.__slab_free
      0.01 ±200%      +0.1        0.07        perf-profile.children.cycles-pp.fput_many
      0.24 ± 25%      +0.1        0.31        perf-profile.children.cycles-pp.timerqueue_del
      0.13 ±  3%      +0.1        0.20        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.14 ±  2%      +0.1        0.21        perf-profile.children.cycles-pp.kmem_cache_free
      0.48 ±  7%      +0.1        0.55        perf-profile.children.cycles-pp.irq_exit
      0.00            +0.1        0.08        perf-profile.children.cycles-pp.account_process_tick
      0.00            +0.1        0.10        perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.14 ± 34%      +0.1        0.24        perf-profile.children.cycles-pp.generic_perform_write
      0.15 ± 32%      +0.1        0.25        perf-profile.children.cycles-pp.__generic_file_write_iter
      0.19 ±  2%      +0.1        0.30        perf-profile.children.cycles-pp.file_free_rcu
      0.18 ± 21%      +0.1        0.29        perf-profile.children.cycles-pp.irq_enter
      0.50 ± 28%      +0.1        0.62        perf-profile.children.cycles-pp.__remove_hrtimer
      0.15 ± 34%      +0.1        0.27        perf-profile.children.cycles-pp.generic_file_write_iter
      0.16 ± 33%      +0.1        0.28        perf-profile.children.cycles-pp.new_sync_write
      0.18 ± 32%      +0.1        0.32        perf-profile.children.cycles-pp.vfs_write
      0.19 ± 32%      +0.1        0.34        perf-profile.children.cycles-pp.ksys_write
      0.24 ± 22%      +0.1        0.39        perf-profile.children.cycles-pp.update_curr
      0.06            +0.2        0.22        perf-profile.children.cycles-pp.ret_from_fork
      0.06 ±  6%      +0.2        0.22        perf-profile.children.cycles-pp.kthread
      0.29 ±  2%      +0.2        0.45        perf-profile.children.cycles-pp.rcu_core
      0.27            +0.2        0.44        perf-profile.children.cycles-pp.rcu_do_batch
      0.25 ± 32%      +0.2        0.42        perf-profile.children.cycles-pp.main
      0.25 ± 32%      +0.2        0.42        perf-profile.children.cycles-pp.cmd_record
      0.00            +0.2        0.19        perf-profile.children.cycles-pp.run_ksoftirqd
      0.00            +0.2        0.20        perf-profile.children.cycles-pp.smpboot_thread_fn
      0.42 ±  4%      +0.2        0.62        perf-profile.children.cycles-pp.__softirqentry_text_start
      3.69 ±  2%      +0.3        3.97        perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.75 ±  2%      +0.4        5.15        perf-profile.children.cycles-pp.hrtimer_interrupt
      6.02 ±  4%      +0.6        6.59        perf-profile.children.cycles-pp.apic_timer_interrupt
      5.76 ±  3%      +0.6        6.35        perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     90.71            -1.0       89.67        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.27 ±  3%      -0.1        0.16        perf-profile.self.cycles-pp._raw_spin_lock
      0.23            -0.1        0.14        perf-profile.self.cycles-pp.kmem_cache_alloc
      0.15 ±  7%      -0.1        0.10        perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.10 ± 23%      -0.1        0.05        perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.08 ±  4%      -0.0        0.05        perf-profile.self.cycles-pp.do_syscall_64
      0.09 ±  4%      -0.0        0.07        perf-profile.self.cycles-pp.__close_fd
      0.10 ±  5%      -0.0        0.08        perf-profile.self.cycles-pp.__alloc_fd
      0.08 ±  5%      +0.0        0.10        perf-profile.self.cycles-pp.timerqueue_add
      0.09 ±  4%      +0.0        0.11        perf-profile.self.cycles-pp.kmem_cache_free
      0.06 ±  6%      +0.0        0.08        perf-profile.self.cycles-pp.tick_program_event
      0.05 ± 50%      +0.0        0.07        perf-profile.self.cycles-pp.native_write_msr
      0.06 ±  8%      +0.0        0.08        perf-profile.self.cycles-pp.___might_sleep
      0.16 ±  9%      +0.0        0.19        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.02 ±122%      +0.0        0.06        perf-profile.self.cycles-pp.__softirqentry_text_start
      0.08 ±  6%      +0.0        0.12        perf-profile.self.cycles-pp.__alloc_file
      0.18 ± 28%      +0.0        0.23        perf-profile.self.cycles-pp.rb_next
      0.03 ± 81%      +0.0        0.08        perf-profile.self.cycles-pp.__slab_free
      0.01 ±200%      +0.0        0.06        perf-profile.self.cycles-pp.run_timer_softirq
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.do_filp_open
      0.09 ± 19%      +0.1        0.15        perf-profile.self.cycles-pp.irq_enter
      0.08 ± 22%      +0.1        0.14        perf-profile.self.cycles-pp.rcu_irq_enter
      0.13 ±  3%      +0.1        0.20        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.account_process_tick
      0.18 ±  4%      +0.1        0.28        perf-profile.self.cycles-pp.file_free_rcu
      0.00            +0.1        0.10        perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.14 ± 29%      +0.1        0.25        perf-profile.self.cycles-pp.update_curr
      2513 ±155%   +1249.9%      33932        interrupts.31:IR-PCI-MSI.2097154-edge.eth0-TxRx-1
    446.60 ± 74%   +3332.4%      15329        interrupts.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3
    875.40 ± 71%     -68.5%     276.00        interrupts.CPU1.RES:Rescheduling_interrupts
      6595 ± 21%     -44.2%       3683        interrupts.CPU10.NMI:Non-maskable_interrupts
      6595 ± 21%     -44.2%       3683        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      4331 ± 33%     +69.1%       7322        interrupts.CPU101.NMI:Non-maskable_interrupts
      4331 ± 33%     +69.1%       7322        interrupts.CPU101.PMI:Performance_monitoring_interrupts
    115.20 ± 11%     -17.5%      95.00        interrupts.CPU103.RES:Rescheduling_interrupts
    570.60 ± 89%     -83.4%      95.00        interrupts.CPU104.RES:Rescheduling_interrupts
    104.80 ±  8%     +88.0%     197.00        interrupts.CPU105.RES:Rescheduling_interrupts
      3644           +97.8%       7209        interrupts.CPU106.NMI:Non-maskable_interrupts
      3644           +97.8%       7209        interrupts.CPU106.PMI:Performance_monitoring_interrupts
      3619          +101.2%       7282        interrupts.CPU108.NMI:Non-maskable_interrupts
      3619          +101.2%       7282        interrupts.CPU108.PMI:Performance_monitoring_interrupts
    169.00 ± 25%     -55.6%      75.00        interrupts.CPU108.RES:Rescheduling_interrupts
      2513 ±155%   +1249.9%      33932        interrupts.CPU11.31:IR-PCI-MSI.2097154-edge.eth0-TxRx-1
    337.60 ± 11%     -13.5%     292.00        interrupts.CPU11.RES:Rescheduling_interrupts
    241.20 ± 65%     -62.3%      91.00        interrupts.CPU110.RES:Rescheduling_interrupts
    121.60 ± 28%     -56.4%      53.00        interrupts.CPU111.RES:Rescheduling_interrupts
    228.40 ± 53%     -64.1%      82.00        interrupts.CPU112.RES:Rescheduling_interrupts
      5826 ± 30%     -38.1%       3609        interrupts.CPU113.NMI:Non-maskable_interrupts
      5826 ± 30%     -38.1%       3609        interrupts.CPU113.PMI:Performance_monitoring_interrupts
    125.00 ± 14%     -29.6%      88.00        interrupts.CPU115.RES:Rescheduling_interrupts
    199.60 ± 18%     -48.4%     103.00        interrupts.CPU116.RES:Rescheduling_interrupts
    144.80 ± 41%     -62.7%      54.00        interrupts.CPU117.RES:Rescheduling_interrupts
    168.20 ± 10%     -57.8%      71.00        interrupts.CPU119.RES:Rescheduling_interrupts
      5899 ± 30%     -37.7%       3675        interrupts.CPU12.NMI:Non-maskable_interrupts
      5899 ± 30%     -37.7%       3675        interrupts.CPU12.PMI:Performance_monitoring_interrupts
    195.80 ± 44%     -62.7%      73.00        interrupts.CPU120.RES:Rescheduling_interrupts
      4313 ± 33%     +67.8%       7240        interrupts.CPU121.NMI:Non-maskable_interrupts
      4313 ± 33%     +67.8%       7240        interrupts.CPU121.PMI:Performance_monitoring_interrupts
    138.20 ± 17%     -37.8%      86.00        interrupts.CPU121.RES:Rescheduling_interrupts
      6509 ± 21%     -43.9%       3651        interrupts.CPU123.NMI:Non-maskable_interrupts
      6509 ± 21%     -43.9%       3651        interrupts.CPU123.PMI:Performance_monitoring_interrupts
      4330 ± 33%     +67.1%       7235        interrupts.CPU127.NMI:Non-maskable_interrupts
      4330 ± 33%     +67.1%       7235        interrupts.CPU127.PMI:Performance_monitoring_interrupts
    154.20 ± 33%     -31.3%     106.00        interrupts.CPU129.RES:Rescheduling_interrupts
    446.60 ± 74%   +3332.4%      15329        interrupts.CPU13.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3
    162.00 ± 55%     -52.5%      77.00        interrupts.CPU131.RES:Rescheduling_interrupts
    142.60 ± 19%     -32.7%      96.00        interrupts.CPU132.RES:Rescheduling_interrupts
    162.60 ±  8%     +58.1%     257.00        interrupts.CPU134.RES:Rescheduling_interrupts
    160.60 ± 23%     -34.0%     106.00        interrupts.CPU138.RES:Rescheduling_interrupts
      5820 ± 30%     -38.6%       3572        interrupts.CPU139.NMI:Non-maskable_interrupts
      5820 ± 30%     -38.6%       3572        interrupts.CPU139.PMI:Performance_monitoring_interrupts
    324.40 ±  7%     -12.8%     283.00        interrupts.CPU14.RES:Rescheduling_interrupts
    167.80 ± 32%     -70.8%      49.00        interrupts.CPU140.RES:Rescheduling_interrupts
    179.40 ± 29%     -57.1%      77.00        interrupts.CPU142.RES:Rescheduling_interrupts
      7243           -50.2%       3605        interrupts.CPU150.NMI:Non-maskable_interrupts
      7243           -50.2%       3605        interrupts.CPU150.PMI:Performance_monitoring_interrupts
      3644           +98.8%       7246        interrupts.CPU151.NMI:Non-maskable_interrupts
      3644           +98.8%       7246        interrupts.CPU151.PMI:Performance_monitoring_interrupts
     58.20 ± 21%     +92.4%     112.00        interrupts.CPU154.RES:Rescheduling_interrupts
      3598          +101.1%       7237        interrupts.CPU160.NMI:Non-maskable_interrupts
      3598          +101.1%       7237        interrupts.CPU160.PMI:Performance_monitoring_interrupts
      3618          +101.5%       7291        interrupts.CPU165.NMI:Non-maskable_interrupts
      3618          +101.5%       7291        interrupts.CPU165.PMI:Performance_monitoring_interrupts
     63.80 ± 86%    +459.6%     357.00        interrupts.CPU167.RES:Rescheduling_interrupts
      5818 ± 30%     -38.1%       3599        interrupts.CPU170.NMI:Non-maskable_interrupts
      5818 ± 30%     -38.1%       3599        interrupts.CPU170.PMI:Performance_monitoring_interrupts
     63.00 ± 42%     +96.8%     124.00        interrupts.CPU176.RES:Rescheduling_interrupts
     39.60 ± 27%    +185.4%     113.00        interrupts.CPU179.RES:Rescheduling_interrupts
     56.00 ± 28%    +166.1%     149.00        interrupts.CPU180.RES:Rescheduling_interrupts
     35.80 ± 20%    +441.9%     194.00        interrupts.CPU187.RES:Rescheduling_interrupts
     46.60 ± 16%    +116.7%     101.00        interrupts.CPU191.RES:Rescheduling_interrupts
      4318 ± 33%     +68.8%       7291        interrupts.CPU198.NMI:Non-maskable_interrupts
      4318 ± 33%     +68.8%       7291        interrupts.CPU198.PMI:Performance_monitoring_interrupts
      4310 ± 33%     +68.4%       7258        interrupts.CPU201.NMI:Non-maskable_interrupts
      4310 ± 33%     +68.4%       7258        interrupts.CPU201.PMI:Performance_monitoring_interrupts
     37.80 ± 31%    +243.9%     130.00        interrupts.CPU201.RES:Rescheduling_interrupts
      5055 ± 35%     +44.1%       7285        interrupts.CPU202.NMI:Non-maskable_interrupts
      5055 ± 35%     +44.1%       7285        interrupts.CPU202.PMI:Performance_monitoring_interrupts
     75.80 ± 28%    +911.9%     767.00        interrupts.CPU202.RES:Rescheduling_interrupts
      4311 ± 33%     +68.2%       7250        interrupts.CPU203.NMI:Non-maskable_interrupts
      4311 ± 33%     +68.2%       7250        interrupts.CPU203.PMI:Performance_monitoring_interrupts
      3593          +100.4%       7199        interrupts.CPU205.NMI:Non-maskable_interrupts
      3593          +100.4%       7199        interrupts.CPU205.PMI:Performance_monitoring_interrupts
    129.60 ± 11%    +240.3%     441.00        interrupts.CPU21.RES:Rescheduling_interrupts
     74.00 ± 17%    +456.8%     412.00        interrupts.CPU210.RES:Rescheduling_interrupts
      5033 ± 34%     +44.1%       7254        interrupts.CPU213.NMI:Non-maskable_interrupts
      5033 ± 34%     +44.1%       7254        interrupts.CPU213.PMI:Performance_monitoring_interrupts
     43.60 ± 39%   +2166.1%     988.00        interrupts.CPU213.RES:Rescheduling_interrupts
     67.80 ± 88%     -72.0%      19.00        interrupts.CPU216.RES:Rescheduling_interrupts
     27.80 ± 45%    +310.1%     114.00        interrupts.CPU223.RES:Rescheduling_interrupts
      3586          +101.4%       7224        interrupts.CPU224.NMI:Non-maskable_interrupts
      3586          +101.4%       7224        interrupts.CPU224.PMI:Performance_monitoring_interrupts
     22.20 ± 28%    +404.5%     112.00        interrupts.CPU239.RES:Rescheduling_interrupts
      3602          +100.8%       7234        interrupts.CPU240.NMI:Non-maskable_interrupts
      3602          +100.8%       7234        interrupts.CPU240.PMI:Performance_monitoring_interrupts
      3594          +101.4%       7239        interrupts.CPU241.NMI:Non-maskable_interrupts
      3594          +101.4%       7239        interrupts.CPU241.PMI:Performance_monitoring_interrupts
     33.60 ± 51%    +370.2%     158.00        interrupts.CPU241.RES:Rescheduling_interrupts
      4765           -11.6%       4212        interrupts.CPU244.CAL:Function_call_interrupts
      4316 ± 32%     +67.2%       7218        interrupts.CPU248.NMI:Non-maskable_interrupts
      4316 ± 32%     +67.2%       7218        interrupts.CPU248.PMI:Performance_monitoring_interrupts
      4409 ± 32%     +68.4%       7425        interrupts.CPU25.NMI:Non-maskable_interrupts
      4409 ± 32%     +68.4%       7425        interrupts.CPU25.PMI:Performance_monitoring_interrupts
    118.80 ± 17%     -31.0%      82.00        interrupts.CPU25.RES:Rescheduling_interrupts
      3602           +99.0%       7167        interrupts.CPU250.NMI:Non-maskable_interrupts
      3602           +99.0%       7167        interrupts.CPU250.PMI:Performance_monitoring_interrupts
     31.60 ± 66%    +263.9%     115.00        interrupts.CPU251.RES:Rescheduling_interrupts
     38.60 ± 28%    +648.7%     289.00        interrupts.CPU258.RES:Rescheduling_interrupts
      3611          +102.2%       7303        interrupts.CPU259.NMI:Non-maskable_interrupts
      3611          +102.2%       7303        interrupts.CPU259.PMI:Performance_monitoring_interrupts
      4332 ± 33%     +66.8%       7228        interrupts.CPU267.NMI:Non-maskable_interrupts
      4332 ± 33%     +66.8%       7228        interrupts.CPU267.PMI:Performance_monitoring_interrupts
      5778 ± 31%     -39.0%       3526        interrupts.CPU268.NMI:Non-maskable_interrupts
      5778 ± 31%     -39.0%       3526        interrupts.CPU268.PMI:Performance_monitoring_interrupts
      3697           +99.5%       7377        interrupts.CPU27.NMI:Non-maskable_interrupts
      3697           +99.5%       7377        interrupts.CPU27.PMI:Performance_monitoring_interrupts
      5037 ± 35%     +44.5%       7278        interrupts.CPU275.NMI:Non-maskable_interrupts
      5037 ± 35%     +44.5%       7278        interrupts.CPU275.PMI:Performance_monitoring_interrupts
     23.00 ± 22%   +1034.8%     261.00        interrupts.CPU278.RES:Rescheduling_interrupts
      4322 ± 32%     +66.5%       7197        interrupts.CPU280.NMI:Non-maskable_interrupts
      4322 ± 32%     +66.5%       7197        interrupts.CPU280.PMI:Performance_monitoring_interrupts
      3585          +101.5%       7225        interrupts.CPU285.NMI:Non-maskable_interrupts
      3585          +101.5%       7225        interrupts.CPU285.PMI:Performance_monitoring_interrupts
    359.80 ±  7%     +60.6%     578.00        interrupts.CPU3.RES:Rescheduling_interrupts
    131.80 ± 20%     -37.0%      83.00        interrupts.CPU30.RES:Rescheduling_interrupts
    125.40 ± 14%     -30.6%      87.00        interrupts.CPU32.RES:Rescheduling_interrupts
      5948 ± 30%     -38.8%       3639        interrupts.CPU33.NMI:Non-maskable_interrupts
      5948 ± 30%     -38.8%       3639        interrupts.CPU33.PMI:Performance_monitoring_interrupts
      4408 ± 34%     +68.8%       7443        interrupts.CPU34.NMI:Non-maskable_interrupts
      4408 ± 34%     +68.8%       7443        interrupts.CPU34.PMI:Performance_monitoring_interrupts
    131.80 ± 23%     -40.8%      78.00        interrupts.CPU34.RES:Rescheduling_interrupts
      5152 ± 34%     +43.5%       7395        interrupts.CPU35.NMI:Non-maskable_interrupts
      5152 ± 34%     +43.5%       7395        interrupts.CPU35.PMI:Performance_monitoring_interrupts
    127.80 ± 24%     -34.3%      84.00        interrupts.CPU36.RES:Rescheduling_interrupts
    141.00 ± 16%     +26.2%     178.00        interrupts.CPU38.RES:Rescheduling_interrupts
    135.60 ± 36%     -59.4%      55.00        interrupts.CPU39.RES:Rescheduling_interrupts
    175.80 ± 44%     -58.5%      73.00        interrupts.CPU40.RES:Rescheduling_interrupts
    128.00 ±  4%     -32.8%      86.00        interrupts.CPU41.RES:Rescheduling_interrupts
     94.40 ± 10%     -33.3%      63.00        interrupts.CPU43.RES:Rescheduling_interrupts
      5834 ± 30%     -38.5%       3587        interrupts.CPU44.NMI:Non-maskable_interrupts
      5834 ± 30%     -38.5%       3587        interrupts.CPU44.PMI:Performance_monitoring_interrupts
    131.60 ± 15%     -23.3%     101.00        interrupts.CPU44.RES:Rescheduling_interrupts
    170.00 ± 22%     -47.1%      90.00        interrupts.CPU46.RES:Rescheduling_interrupts
    159.20 ± 35%     -49.7%      80.00        interrupts.CPU47.RES:Rescheduling_interrupts
    127.60 ± 34%     -43.6%      72.00        interrupts.CPU49.RES:Rescheduling_interrupts
    179.20 ± 41%     -51.5%      87.00        interrupts.CPU50.RES:Rescheduling_interrupts
    122.80 ± 17%     -39.7%      74.00        interrupts.CPU51.RES:Rescheduling_interrupts
    123.60 ± 18%     -35.3%      80.00        interrupts.CPU53.RES:Rescheduling_interrupts
    141.20 ± 19%     +76.3%     249.00        interrupts.CPU57.RES:Rescheduling_interrupts
      5820 ± 30%     -38.3%       3590        interrupts.CPU58.NMI:Non-maskable_interrupts
      5820 ± 30%     -38.3%       3590        interrupts.CPU58.PMI:Performance_monitoring_interrupts
    148.20 ± 33%     -69.0%      46.00        interrupts.CPU59.RES:Rescheduling_interrupts
    339.40 ± 14%    +158.7%     878.00        interrupts.CPU6.RES:Rescheduling_interrupts
      3594          +100.6%       7211        interrupts.CPU60.NMI:Non-maskable_interrupts
      3594          +100.6%       7211        interrupts.CPU60.PMI:Performance_monitoring_interrupts
    165.40 ± 16%     -46.8%      88.00        interrupts.CPU60.RES:Rescheduling_interrupts
    119.60 ± 13%     -32.3%      81.00        interrupts.CPU65.RES:Rescheduling_interrupts
      6569 ± 22%     -45.8%       3559        interrupts.CPU68.NMI:Non-maskable_interrupts
      6569 ± 22%     -45.8%       3559        interrupts.CPU68.PMI:Performance_monitoring_interrupts
    149.60 ± 18%     -53.9%      69.00        interrupts.CPU68.RES:Rescheduling_interrupts
    124.60 ± 23%     -55.1%      56.00        interrupts.CPU69.RES:Rescheduling_interrupts
    351.40 ± 21%     -19.7%     282.00        interrupts.CPU7.RES:Rescheduling_interrupts
      5826 ± 31%     -38.5%       3584        interrupts.CPU76.NMI:Non-maskable_interrupts
      5826 ± 31%     -38.5%       3584        interrupts.CPU76.PMI:Performance_monitoring_interrupts
      3615          +100.9%       7263        interrupts.CPU80.NMI:Non-maskable_interrupts
      3615          +100.9%       7263        interrupts.CPU80.PMI:Performance_monitoring_interrupts
      4325 ± 32%     +66.8%       7216        interrupts.CPU81.NMI:Non-maskable_interrupts
      4325 ± 32%     +66.8%       7216        interrupts.CPU81.PMI:Performance_monitoring_interrupts
    114.20 ± 22%     -39.6%      69.00        interrupts.CPU81.RES:Rescheduling_interrupts
    120.00 ± 32%     -62.5%      45.00        interrupts.CPU83.RES:Rescheduling_interrupts
    293.40 ± 63%     -73.4%      78.00        interrupts.CPU86.RES:Rescheduling_interrupts
    126.20 ± 24%     -45.3%      69.00        interrupts.CPU89.RES:Rescheduling_interrupts
    311.00 ±  8%    +187.8%     895.00        interrupts.CPU9.RES:Rescheduling_interrupts
    175.60 ± 31%     -55.6%      78.00        interrupts.CPU90.RES:Rescheduling_interrupts
    100.40 ± 29%     -29.3%      71.00        interrupts.CPU91.RES:Rescheduling_interrupts
    177.40 ± 16%     -58.9%      73.00        interrupts.CPU95.RES:Rescheduling_interrupts
    185.20 ± 38%     -50.9%      91.00        interrupts.CPU96.RES:Rescheduling_interrupts
    113.40 ± 12%     +20.8%     137.00        interrupts.CPU99.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/sched_yield/will-it-scale/0x11

commit: 
  v5.7-rc3
  8aebfffacf ("configfs: fix config_item refcnt leak in configfs_rmdir()")

        v5.7-rc3 8aebfffacfa379ba400da573a5b 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:5          -20%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
    129619           +17.8%     152634        will-it-scale.per_thread_ops
  12586764 ±  6%    -100.0%       2.25 ± 72%  will-it-scale.time.involuntary_context_switches
    245608           -96.0%       9887        will-it-scale.time.maximum_resident_set_size
     57640 ±  4%     -89.4%       6105        will-it-scale.time.minor_page_faults
     28478          -100.0%       0.00        will-it-scale.time.percent_of_cpu_this_job_got
     74208          -100.0%       0.19 ±  7%  will-it-scale.time.system_time
     14260          -100.0%       0.25 ±  3%  will-it-scale.time.user_time
  37330584           +17.8%   43958990        will-it-scale.workload
   1348004           +11.0%    1496355        vmstat.memory.cache
   1228847           +11.3%    1368042        meminfo.Cached
   1096118           +12.8%    1236404        meminfo.Unevictable
    363.61          -100.0%       0.00        uptime.boot
     14062 ±  3%    -100.0%       0.00        uptime.idle
     45.55 ±  2%      +4.3%      47.52 ±  2%  boot-time.boot
     27.68 ±  4%      +7.5%      29.77 ±  3%  boot-time.dhcp
     11319 ±  3%      +5.0%      11886 ±  2%  boot-time.idle
    126266            +1.4%     127983        proc-vmstat.nr_anon_pages
    307237           +11.3%     341994        proc-vmstat.nr_file_pages
     29707            +7.7%      31999        proc-vmstat.nr_slab_reclaimable
    274029           +12.8%     309100        proc-vmstat.nr_unevictable
    274029           +12.8%     309100        proc-vmstat.nr_zone_unevictable
    991130            +1.2%    1003460        proc-vmstat.pgalloc_normal
    818717            +1.3%     829269        proc-vmstat.pgfault
    698116            +9.9%     767014        numa-meminfo.node0.FilePages
    565294           +12.4%     635266        numa-meminfo.node0.Unevictable
    530824           +13.2%     601139        numa-meminfo.node1.FilePages
     18760 ±  6%     +19.6%      22432 ±  5%  numa-meminfo.node1.KReclaimable
     10936           +14.3%      12504        numa-meminfo.node1.Mapped
    625411           +11.6%     698082        numa-meminfo.node1.MemUsed
     18760 ±  6%     +19.6%      22432 ±  5%  numa-meminfo.node1.SReclaimable
    530824           +13.2%     601139        numa-meminfo.node1.Unevictable
    174502            +9.9%     191744        numa-vmstat.node0.nr_file_pages
    141323           +12.4%     158816        numa-vmstat.node0.nr_unevictable
    141323           +12.4%     158816        numa-vmstat.node0.nr_zone_unevictable
    154664           +11.3%     172075        numa-vmstat.node0.numa_interleave
    132706           +13.2%     150284        numa-vmstat.node1.nr_file_pages
      2735           +14.3%       3124        numa-vmstat.node1.nr_mapped
      4690 ±  6%     +19.6%       5608 ±  5%  numa-vmstat.node1.nr_slab_reclaimable
    132706           +13.2%     150284        numa-vmstat.node1.nr_unevictable
    132706           +13.2%     150284        numa-vmstat.node1.nr_zone_unevictable
    155440           +11.1%     172714        numa-vmstat.node1.numa_hit
    154855           +11.2%     172128        numa-vmstat.node1.numa_interleave
    154868           +11.2%     172141        numa-vmstat.node1.numa_other
     56.00          +150.0%     140.00 ± 20%  slabinfo.btrfs_extent_map.active_objs
     56.00          +150.0%     140.00 ± 20%  slabinfo.btrfs_extent_map.num_objs
      1568            +9.7%       1720        slabinfo.inode_cache.active_slabs
     84716            +9.7%      92925        slabinfo.inode_cache.num_objs
      1568            +9.7%       1720        slabinfo.inode_cache.num_slabs
    524.80 ± 14%     -29.9%     368.00 ±  7%  slabinfo.kmalloc-rcl-128.active_objs
    524.80 ± 14%     -29.9%     368.00 ±  7%  slabinfo.kmalloc-rcl-128.num_objs
      6284 ±  4%     +28.9%       8098 ±  7%  slabinfo.kmalloc-rcl-64.active_objs
      6284 ±  4%     +28.9%       8098 ±  7%  slabinfo.kmalloc-rcl-64.num_objs
     28669           +16.0%      33247        slabinfo.radix_tree_node.active_objs
     28669           +16.0%      33247        slabinfo.radix_tree_node.num_objs
      1066 ± 10%    -100.0%       0.00        slabinfo.skbuff_ext_cache.active_objs
      1066 ± 10%    -100.0%       0.00        slabinfo.skbuff_ext_cache.num_objs
    945.80 ±  2%     +10.2%       1042 ±  6%  slabinfo.skbuff_fclone_cache.active_objs
    945.80 ±  2%     +10.2%       1042 ±  6%  slabinfo.skbuff_fclone_cache.num_objs
      2289 ±  2%     -14.9%       1947        sched_debug.cfs_rq:/.exec_clock.stddev
      4558 ±  8%     -10.1%       4098 ±  3%  sched_debug.cfs_rq:/.load.avg
     21597 ± 25%     -25.8%      16031 ±  4%  sched_debug.cfs_rq:/.load.stddev
    938759 ±  2%     -13.0%     816467 ±  3%  sched_debug.cfs_rq:/.min_vruntime.stddev
     35.32 ± 32%     +45.2%      51.30 ± 17%  sched_debug.cfs_rq:/.nr_spread_over.max
      2.90 ± 19%     +35.1%       3.92 ±  3%  sched_debug.cfs_rq:/.nr_spread_over.stddev
  -8467500           -16.3%   -7085934        sched_debug.cfs_rq:/.spread0.min
    641459 ±  5%     -12.2%     563287 ±  5%  sched_debug.cfs_rq:/.spread0.stddev
      1937 ±  3%     -14.2%       1662        sched_debug.cpu.clock.stddev
      1937 ±  3%     -14.2%       1662        sched_debug.cpu.clock_task.stddev
    229162 ± 14%     +15.5%     264772 ±  6%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ±  2%     -14.6%       0.00        sched_debug.cpu.next_balance.stddev
  17032335           +15.7%   19705462        sched_debug.cpu.sched_count.avg
  41820934           +10.5%   46197041        sched_debug.cpu.sched_count.max
   9425674 ±  2%     +20.7%   11374378        sched_debug.cpu.sched_count.stddev
  17031308           +15.7%   19704439        sched_debug.cpu.yld_count.avg
  41819326           +10.5%   46194610        sched_debug.cpu.yld_count.max
   9425595 ±  2%     +20.7%   11374229        sched_debug.cpu.yld_count.stddev
      1645 ± 17%     -44.1%     919.00 ± 45%  interrupts.CPU1.RES:Rescheduling_interrupts
    126.40 ± 55%    +337.5%     553.00 ± 97%  interrupts.CPU101.RES:Rescheduling_interrupts
      1103 ± 95%     -77.4%     249.50 ± 29%  interrupts.CPU104.RES:Rescheduling_interrupts
      5548 ± 29%     -37.4%       3473        interrupts.CPU139.NMI:Non-maskable_interrupts
      5548 ± 29%     -37.4%       3473        interrupts.CPU139.PMI:Performance_monitoring_interrupts
      5486 ± 29%     -37.5%       3427        interrupts.CPU147.NMI:Non-maskable_interrupts
      5486 ± 29%     -37.5%       3427        interrupts.CPU147.PMI:Performance_monitoring_interrupts
      5554 ± 29%     -37.6%       3465        interrupts.CPU148.NMI:Non-maskable_interrupts
      5554 ± 29%     -37.6%       3465        interrupts.CPU148.PMI:Performance_monitoring_interrupts
    132.60 ± 33%     -59.5%      53.75 ± 25%  interrupts.CPU148.RES:Rescheduling_interrupts
    454.80 ± 28%     +27.7%     581.00 ± 17%  interrupts.CPU15.RES:Rescheduling_interrupts
     62.00 ± 85%    +127.0%     140.75 ± 69%  interrupts.CPU150.RES:Rescheduling_interrupts
     21.60 ±100%    +841.0%     203.25 ± 73%  interrupts.CPU161.RES:Rescheduling_interrupts
    307.00 ±137%     -95.5%      13.75 ± 48%  interrupts.CPU167.RES:Rescheduling_interrupts
      5547 ± 30%     -38.6%       3404        interrupts.CPU171.NMI:Non-maskable_interrupts
      5547 ± 30%     -38.6%       3404        interrupts.CPU171.PMI:Performance_monitoring_interrupts
    102.20 ± 35%     -90.5%       9.75 ± 18%  interrupts.CPU193.RES:Rescheduling_interrupts
    183.40 ± 70%     -79.4%      37.75 ± 46%  interrupts.CPU196.RES:Rescheduling_interrupts
      5638 ± 29%     -37.7%       3515        interrupts.CPU201.NMI:Non-maskable_interrupts
      5638 ± 29%     -37.7%       3515        interrupts.CPU201.PMI:Performance_monitoring_interrupts
    268.00 ± 45%    +140.2%     643.75 ± 51%  interrupts.CPU21.RES:Rescheduling_interrupts
      5463 ± 29%     -36.5%       3468        interrupts.CPU216.NMI:Non-maskable_interrupts
      5463 ± 29%     -36.5%       3468        interrupts.CPU216.PMI:Performance_monitoring_interrupts
    304.20 ± 34%    +109.8%     638.25 ± 24%  interrupts.CPU22.RES:Rescheduling_interrupts
    131.60 ± 88%     -66.2%      44.50 ± 49%  interrupts.CPU220.RES:Rescheduling_interrupts
     79.00 ± 25%     -50.0%      39.50 ± 64%  interrupts.CPU228.RES:Rescheduling_interrupts
    423.20 ± 41%     +97.7%     836.50 ± 27%  interrupts.CPU24.RES:Rescheduling_interrupts
    136.00 ± 38%     -78.1%      29.75 ± 84%  interrupts.CPU240.RES:Rescheduling_interrupts
     76.80 ± 61%     -74.0%      20.00 ± 68%  interrupts.CPU266.RES:Rescheduling_interrupts
    480.80 ± 54%     +81.8%     874.00 ± 14%  interrupts.CPU28.RES:Rescheduling_interrupts
    229.20 ± 43%    +218.2%     729.25 ± 90%  interrupts.CPU29.RES:Rescheduling_interrupts
    434.40 ± 95%    +105.7%     893.75 ± 47%  interrupts.CPU30.RES:Rescheduling_interrupts
      6433 ± 21%     -44.8%       3552        interrupts.CPU66.NMI:Non-maskable_interrupts
      6433 ± 21%     -44.8%       3552        interrupts.CPU66.PMI:Performance_monitoring_interrupts
    415.00 ± 55%     -79.0%      87.25 ± 16%  interrupts.CPU72.RES:Rescheduling_interrupts
    312.00 ± 34%     -54.6%     141.50 ± 37%  interrupts.CPU75.RES:Rescheduling_interrupts
    841.60 ± 29%     -43.8%     473.00 ± 24%  interrupts.CPU8.RES:Rescheduling_interrupts
      4127 ± 31%     +64.4%       6787        interrupts.CPU89.NMI:Non-maskable_interrupts
      4127 ± 31%     +64.4%       6787        interrupts.CPU89.PMI:Performance_monitoring_interrupts
    217.80 ± 31%     -59.5%      88.25 ± 58%  interrupts.CPU89.RES:Rescheduling_interrupts
    285.00 ± 66%     -79.6%      58.00 ± 63%  interrupts.CPU93.RES:Rescheduling_interrupts
     96.20 ± 53%     +81.1%     174.25 ± 47%  interrupts.CPU95.RES:Rescheduling_interrupts
    471.80 ± 40%    +210.9%       1466 ± 66%  interrupts.CPU98.RES:Rescheduling_interrupts
     10.39            -4.7        5.72 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock.do_sched_yield.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.81            -4.3        7.56 ±  2%  perf-profile.calltrace.cycles-pp.update_rq_clock.__sched_text_start.schedule.__x64_sys_sched_yield.do_syscall_64
      5.25 ±  2%      -2.4        2.82 ±  3%  perf-profile.calltrace.cycles-pp.sched_clock.sched_clock_cpu.update_rq_clock.__sched_text_start.schedule
      6.08 ±  2%      -2.4        3.66 ±  3%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.update_rq_clock.__sched_text_start.schedule.__x64_sys_sched_yield
      5.47 ±  2%      -2.4        3.05 ±  3%  perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock.sched_clock_cpu.update_rq_clock.__sched_text_start
      3.46            -2.1        1.41 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__sched_text_start.schedule.__x64_sys_sched_yield.do_syscall_64
     14.72            -2.0       12.71 ±  2%  perf-profile.calltrace.cycles-pp.do_sched_yield.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
      1.94 ±  3%      -1.3        0.64 ±  4%  perf-profile.calltrace.cycles-pp.rcu_qs.rcu_note_context_switch.__sched_text_start.schedule.__x64_sys_sched_yield
      4.86 ± 12%      -1.2        3.62 ± 13%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.__raw_spin_unlock_irq.__sched_text_start.schedule.__x64_sys_sched_yield
      4.70 ± 11%      -1.2        3.48 ± 13%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.__raw_spin_unlock_irq.__sched_text_start.schedule
      5.60 ± 10%      -1.1        4.54 ± 10%  perf-profile.calltrace.cycles-pp.__raw_spin_unlock_irq.__sched_text_start.schedule.__x64_sys_sched_yield.do_syscall_64
      3.89 ± 11%      -1.0        2.87 ± 15%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__raw_spin_unlock_irq.__sched_text_start
     76.22            -0.9       75.35        perf-profile.calltrace.cycles-pp.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
      2.75 ± 10%      -0.7        2.07 ± 15%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__raw_spin_unlock_irq
      1.48 ± 16%      -0.7        0.81 ± 65%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
     81.87            -0.7       81.21        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
      1.35 ± 18%      -0.6        0.71 ± 65%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      1.28 ± 18%      -0.6        0.67 ± 65%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.93 ± 19%      -0.6        0.36 ±101%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
     83.76            -0.5       83.26        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__sched_yield
      0.71 ± 13%      -0.4        0.30 ±100%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__sched_yield
      2.69 ±  2%      -0.4        2.33 ±  3%  perf-profile.calltrace.cycles-pp.rcu_note_context_switch.__sched_text_start.schedule.__x64_sys_sched_yield.do_syscall_64
      1.28 ±  2%      -0.2        1.04 ±  2%  perf-profile.calltrace.cycles-pp.__list_del_entry_valid.pick_next_task_fair.__sched_text_start.schedule.__x64_sys_sched_yield
      5.32            -0.2        5.09        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__sched_yield
      0.89 ± 14%      -0.2        0.67 ± 15%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.__sched_yield
      0.85 ± 13%      -0.2        0.64 ± 15%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.__sched_yield
      0.42 ± 50%      +0.1        0.55 ±  3%  perf-profile.calltrace.cycles-pp.check_cfs_rq_runtime.pick_next_task_fair.__sched_text_start.schedule.__x64_sys_sched_yield
      0.86 ±  2%      +0.2        1.02 ±  3%  perf-profile.calltrace.cycles-pp.fpregs_assert_state_consistent.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
      1.36 ±  2%      +0.2        1.53 ±  3%  perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__sched_text_start.schedule.__x64_sys_sched_yield
      0.64 ±  2%      +0.2        0.82        perf-profile.calltrace.cycles-pp.update_min_vruntime.update_curr.pick_next_task_fair.__sched_text_start.schedule
      2.44            +0.4        2.80 ±  3%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.pick_next_task_fair.__sched_text_start.schedule
     58.36            +0.5       58.86        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.61            +0.7        6.34        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__sched_yield
     60.37            +1.1       61.43        perf-profile.calltrace.cycles-pp.schedule.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
      6.36 ±  2%      +1.8        8.11 ±  3%  perf-profile.calltrace.cycles-pp.cpuacct_charge.update_curr.pick_next_task_fair.__sched_text_start.schedule
     19.28            +3.6       22.89 ±  2%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule.__x64_sys_sched_yield.do_syscall_64
     12.50            +4.2       16.73 ±  2%  perf-profile.calltrace.cycles-pp.update_curr.pick_next_task_fair.__sched_text_start.schedule.__x64_sys_sched_yield
     14.19            -6.9        7.32 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock
     12.08            -4.4        7.71 ±  2%  perf-profile.children.cycles-pp.update_rq_clock
      5.67 ±  2%      -2.5        3.19 ±  2%  perf-profile.children.cycles-pp.sched_clock
      6.25 ±  2%      -2.5        3.77 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      5.59 ±  2%      -2.5        3.11 ±  2%  perf-profile.children.cycles-pp.native_sched_clock
     14.75            -2.0       12.75 ±  2%  perf-profile.children.cycles-pp.do_sched_yield
      1.99 ±  3%      -1.3        0.67 ±  4%  perf-profile.children.cycles-pp.rcu_qs
      5.74 ± 10%      -1.1        4.66 ± 10%  perf-profile.children.cycles-pp.__raw_spin_unlock_irq
     76.26            -0.9       75.39        perf-profile.children.cycles-pp.__x64_sys_sched_yield
     82.34            -0.7       81.65        perf-profile.children.cycles-pp.do_syscall_64
     84.21            -0.5       83.67        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      2.75 ±  2%      -0.4        2.39 ±  3%  perf-profile.children.cycles-pp.rcu_note_context_switch
      1.32 ±  2%      -0.2        1.08 ±  2%  perf-profile.children.cycles-pp.__list_del_entry_valid
      5.94            -0.2        5.76        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.66 ±  9%      -0.1        0.54 ± 15%  perf-profile.children.cycles-pp.irq_exit
      0.28 ±  8%      -0.1        0.18 ± 16%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.17 ± 19%      -0.1        0.07 ± 72%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.16 ± 13%      -0.1        0.07 ± 64%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.20 ± 11%      -0.1        0.13 ± 10%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.10 ± 13%      -0.1        0.04 ±107%  perf-profile.children.cycles-pp.rcu_core
      0.05 ± 51%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.ret_from_intr
      0.61 ±  2%      +0.0        0.65 ±  2%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.06 ± 57%      +0.0        0.10 ± 25%  perf-profile.children.cycles-pp.irq_enter
      0.00            +0.1        0.10 ± 11%  perf-profile.children.cycles-pp.idle_cpu
      0.89            +0.2        1.05 ±  3%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      1.48 ±  2%      +0.2        1.66 ±  3%  perf-profile.children.cycles-pp.pick_next_entity
      0.70 ±  2%      +0.2        0.89        perf-profile.children.cycles-pp.update_min_vruntime
      2.49            +0.4        2.85 ±  3%  perf-profile.children.cycles-pp.__calc_delta
     58.57            +0.5       59.08        perf-profile.children.cycles-pp.__sched_text_start
      4.87            +0.6        5.50        perf-profile.children.cycles-pp.entry_SYSCALL_64
     60.44            +1.1       61.50        perf-profile.children.cycles-pp.schedule
      6.49 ±  3%      +1.7        8.22 ±  2%  perf-profile.children.cycles-pp.cpuacct_charge
     19.54            +3.6       23.16 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
     12.90 ±  2%      +4.2       17.12 ±  2%  perf-profile.children.cycles-pp.update_curr
     14.15            -6.9        7.28 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      5.30 ±  2%      -2.5        2.81 ±  2%  perf-profile.self.cycles-pp.native_sched_clock
      5.86            -1.9        3.96        perf-profile.self.cycles-pp.update_rq_clock
      1.95 ±  3%      -1.3        0.62 ±  5%  perf-profile.self.cycles-pp.rcu_qs
      3.39            -0.6        2.79 ±  2%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.97 ± 26%      -0.4        0.56 ± 21%  perf-profile.self.cycles-pp.hrtimer_interrupt
      1.29 ±  2%      -0.2        1.05 ±  2%  perf-profile.self.cycles-pp.__list_del_entry_valid
      5.92            -0.2        5.74        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.27 ±  9%      -0.1        0.18 ± 16%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.16 ± 13%      -0.1        0.07 ± 64%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.19 ± 18%      -0.1        0.11 ± 40%  perf-profile.self.cycles-pp.task_tick_fair
      0.10 ± 19%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.20 ± 11%      -0.1        0.13 ± 10%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.20 ±  2%      +0.0        0.22 ±  3%  perf-profile.self.cycles-pp.__list_add_valid
      0.55            +0.0        0.58 ±  2%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.02 ±122%      +0.0        0.06 ±  6%  perf-profile.self.cycles-pp.ret_from_intr
      0.97            +0.1        1.03 ±  2%  perf-profile.self.cycles-pp.yield_task_fair
      1.02            +0.1        1.11 ±  2%  perf-profile.self.cycles-pp.__x64_sys_sched_yield
      0.00            +0.1        0.09 ± 11%  perf-profile.self.cycles-pp.idle_cpu
      0.99 ±  2%      +0.2        1.15 ±  3%  perf-profile.self.cycles-pp.pick_next_entity
      0.83 ±  2%      +0.2        0.99 ±  3%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      1.84            +0.2        2.00 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.73 ±  6%      +0.2        0.90        perf-profile.self.cycles-pp.__raw_spin_unlock_irq
      0.65 ±  2%      +0.2        0.84        perf-profile.self.cycles-pp.update_min_vruntime
      2.44            +0.4        2.79 ±  2%  perf-profile.self.cycles-pp.__calc_delta
      3.20            +0.4        3.61        perf-profile.self.cycles-pp.__sched_yield
      4.05            +0.5        4.58        perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.79 ±  3%      +0.5        2.32 ±  2%  perf-profile.self.cycles-pp.schedule
      0.77            +1.0        1.73 ±  3%  perf-profile.self.cycles-pp.rcu_note_context_switch
      6.44 ±  3%      +1.7        8.17 ±  2%  perf-profile.self.cycles-pp.cpuacct_charge
      3.33 ±  2%      +2.0        5.29 ±  2%  perf-profile.self.cycles-pp.update_curr
      3.11 ±  2%      +2.7        5.79        perf-profile.self.cycles-pp.do_sched_yield
     14.98 ±  4%      +4.6       19.54 ±  2%  perf-profile.self.cycles-pp.__sched_text_start
    113119 ±  7%     +16.3%     131545 ±  4%  softirqs.CPU0.TIMER
    114026 ±  6%     -13.7%      98362 ±  3%  softirqs.CPU10.TIMER
    124842 ± 11%     +15.5%     144145 ±  4%  softirqs.CPU100.TIMER
    120351 ± 11%     +16.0%     139650 ±  4%  softirqs.CPU101.TIMER
    114377 ±  9%     +23.7%     141538        softirqs.CPU103.TIMER
    119810 ±  9%     +20.0%     143803        softirqs.CPU104.TIMER
    124302 ± 10%     +19.0%     147917 ±  2%  softirqs.CPU105.TIMER
    128843 ±  6%     +16.0%     149410 ±  2%  softirqs.CPU106.TIMER
    133894 ±  6%     +14.4%     153196        softirqs.CPU107.TIMER
    121089 ±  8%     +19.4%     144552 ±  2%  softirqs.CPU108.TIMER
    116435 ±  9%     +18.9%     138420 ±  6%  softirqs.CPU109.TIMER
    122605 ±  9%     +20.6%     147851 ±  2%  softirqs.CPU110.TIMER
    125118 ± 11%     +18.0%     147647 ±  3%  softirqs.CPU111.TIMER
    120414 ± 11%     +24.0%     149357 ±  4%  softirqs.CPU112.TIMER
    118061 ±  9%     +24.5%     147002 ±  2%  softirqs.CPU113.TIMER
    113262 ± 12%     +21.7%     137886 ±  4%  softirqs.CPU114.TIMER
    115072 ±  8%     +15.8%     133253 ±  2%  softirqs.CPU115.TIMER
    126037 ±  7%     +17.3%     147845 ±  3%  softirqs.CPU116.TIMER
    125201 ±  9%     +27.9%     160135 ±  5%  softirqs.CPU117.TIMER
    126815 ±  7%     +15.3%     146183 ±  3%  softirqs.CPU119.TIMER
    113033 ± 11%     +22.8%     138784 ±  5%  softirqs.CPU120.TIMER
    124716 ±  8%     +20.6%     150419 ±  9%  softirqs.CPU124.TIMER
    119654 ±  9%     +23.3%     147567 ±  2%  softirqs.CPU125.TIMER
    121689 ± 11%     +17.8%     143395 ±  3%  softirqs.CPU126.TIMER
    124001 ±  7%     +18.0%     146288        softirqs.CPU128.TIMER
    129333 ±  8%     +15.5%     149415 ±  2%  softirqs.CPU129.TIMER
    109263 ±  3%     +24.0%     135462 ±  4%  softirqs.CPU13.TIMER
    120280 ±  8%     +18.5%     142504 ±  2%  softirqs.CPU130.TIMER
    127568 ±  9%     +15.9%     147805 ±  2%  softirqs.CPU131.TIMER
    121492 ± 10%     +18.9%     144478 ±  3%  softirqs.CPU132.TIMER
    121701 ± 12%     +20.0%     146072 ±  4%  softirqs.CPU133.TIMER
    118584 ± 12%     +19.4%     141554 ±  2%  softirqs.CPU134.TIMER
    118139 ± 12%     +19.3%     140941 ±  8%  softirqs.CPU135.TIMER
    111559 ± 10%     +23.7%     137951 ±  8%  softirqs.CPU136.TIMER
    116550 ±  6%     +21.1%     141123 ±  4%  softirqs.CPU138.TIMER
    119925 ± 11%     +20.9%     145018 ±  3%  softirqs.CPU139.TIMER
    110559 ±  8%     +19.2%     131773 ±  4%  softirqs.CPU14.TIMER
    118476 ±  9%     +15.1%     136359 ±  2%  softirqs.CPU140.TIMER
    123705 ±  9%     +17.5%     145309 ±  2%  softirqs.CPU141.TIMER
    119453 ±  9%     +17.7%     140644 ±  2%  softirqs.CPU142.TIMER
    130404 ± 10%     +16.3%     151657 ±  2%  softirqs.CPU143.TIMER
    127286 ±  8%     +17.1%     149108 ±  2%  softirqs.CPU144.TIMER
    125136 ± 10%     +16.3%     145511        softirqs.CPU145.TIMER
    126142 ±  6%     +15.4%     145582 ±  2%  softirqs.CPU146.TIMER
    125707 ±  8%     +14.3%     143729 ±  2%  softirqs.CPU147.TIMER
    120985 ± 10%     +18.4%     143290 ±  3%  softirqs.CPU148.TIMER
    123289 ± 11%     +15.8%     142709 ±  3%  softirqs.CPU149.TIMER
    110288 ±  3%     +23.7%     136419 ±  2%  softirqs.CPU15.TIMER
    118787 ± 14%     +21.0%     143741 ±  3%  softirqs.CPU150.TIMER
    118711 ± 10%     +11.9%     132818 ±  6%  softirqs.CPU151.TIMER
    124926 ±  7%     +18.0%     147353 ±  2%  softirqs.CPU152.TIMER
    123757 ±  7%     +15.6%     143106 ±  2%  softirqs.CPU153.TIMER
    128552 ±  8%     +14.5%     147151 ±  2%  softirqs.CPU154.TIMER
    124377 ±  8%     +17.8%     146546        softirqs.CPU155.TIMER
    125536 ± 10%     +15.9%     145472 ±  2%  softirqs.CPU156.TIMER
    116239 ±  9%     +15.8%     134554 ±  3%  softirqs.CPU157.TIMER
    123726 ± 10%     +16.4%     143992 ±  4%  softirqs.CPU158.TIMER
    123210 ±  9%     +16.6%     143679 ±  2%  softirqs.CPU159.TIMER
    109481 ±  2%     +33.6%     146300 ±  4%  softirqs.CPU16.TIMER
    119827 ±  7%     +23.5%     148008        softirqs.CPU160.TIMER
    123247 ±  9%     +18.8%     146472 ±  2%  softirqs.CPU161.TIMER
    112096 ±  9%     +19.4%     133805 ±  3%  softirqs.CPU162.TIMER
    115002 ±  8%     +24.0%     142655 ±  3%  softirqs.CPU164.TIMER
    124289 ±  6%     +26.0%     156586        softirqs.CPU165.TIMER
    124167 ±  7%     +13.2%     140537        softirqs.CPU166.TIMER
    120607 ±  7%     +25.4%     151242        softirqs.CPU167.TIMER
    124851 ±  7%     +15.4%     144078        softirqs.CPU168.TIMER
    126304 ±  8%     +16.0%     146520        softirqs.CPU169.TIMER
    110273 ±  6%     +22.0%     134573 ±  3%  softirqs.CPU17.TIMER
    125873 ±  8%     +17.0%     147232        softirqs.CPU170.TIMER
    128114 ±  6%     +18.9%     152291        softirqs.CPU171.TIMER
    124202 ± 11%     +13.2%     140567 ±  3%  softirqs.CPU172.TIMER
    124056 ± 12%     +18.0%     146348 ±  3%  softirqs.CPU174.TIMER
    121501 ± 10%     +20.5%     146393 ±  2%  softirqs.CPU175.TIMER
    118121 ± 10%     +19.3%     140906 ±  3%  softirqs.CPU176.TIMER
    117434 ± 11%     +21.4%     142608 ±  3%  softirqs.CPU177.TIMER
    114173 ± 10%     +15.6%     131982 ±  3%  softirqs.CPU178.TIMER
    116001 ±  9%     +17.5%     136283 ±  2%  softirqs.CPU179.TIMER
    113960 ±  5%     +30.4%     148616 ±  2%  softirqs.CPU18.TIMER
    124274 ±  7%     +18.2%     146928 ±  2%  softirqs.CPU180.TIMER
    124996 ±  5%     +17.7%     147104 ±  2%  softirqs.CPU181.TIMER
    111019 ±  6%     +28.1%     142231 ±  3%  softirqs.CPU182.TIMER
    111383 ±  9%     +18.5%     131989 ±  2%  softirqs.CPU183.TIMER
    118735 ±  8%     +21.4%     144187 ±  2%  softirqs.CPU185.TIMER
    119430 ± 10%     +20.0%     143357 ±  2%  softirqs.CPU186.TIMER
    119779 ± 13%     +18.5%     141916 ±  2%  softirqs.CPU187.TIMER
    114844 ± 10%     +19.3%     137044 ±  5%  softirqs.CPU188.TIMER
    112217 ± 10%     +16.6%     130821 ±  4%  softirqs.CPU189.TIMER
    108078 ±  3%     +18.7%     128293 ±  2%  softirqs.CPU19.TIMER
    128086 ±  7%     +18.9%     152236 ±  2%  softirqs.CPU190.TIMER
    127315 ±  9%     +14.8%     146133        softirqs.CPU192.TIMER
    127602 ±  7%     +17.6%     150031 ±  2%  softirqs.CPU193.TIMER
    126385 ±  7%     +14.4%     144530        softirqs.CPU195.TIMER
    119831 ±  7%     +21.2%     145237 ±  2%  softirqs.CPU196.TIMER
    121831 ±  9%     +17.1%     142687 ±  3%  softirqs.CPU197.TIMER
    116601 ± 11%     +15.6%     134796 ±  5%  softirqs.CPU198.TIMER
    113355 ±  7%     +24.9%     141568        softirqs.CPU2.TIMER
    110416 ±  8%     +17.0%     129173 ±  4%  softirqs.CPU200.TIMER
    113086 ± 11%     +19.5%     135112 ±  5%  softirqs.CPU201.TIMER
    119543 ± 12%     +18.6%     141771 ±  2%  softirqs.CPU202.TIMER
    118165 ± 10%     +21.5%     143606 ±  2%  softirqs.CPU203.TIMER
    116981 ±  8%     +19.0%     139231 ±  2%  softirqs.CPU204.TIMER
    118396 ± 10%     +21.8%     144211 ±  2%  softirqs.CPU206.TIMER
    133254 ±  6%     +12.7%     150153 ±  2%  softirqs.CPU207.TIMER
    129492 ±  7%     +16.5%     150874 ±  3%  softirqs.CPU208.TIMER
    126629 ±  8%     +16.3%     147271 ±  2%  softirqs.CPU209.TIMER
    120427 ±  7%     +20.1%     144661 ±  2%  softirqs.CPU210.TIMER
    115844 ±  5%     +23.4%     142908 ±  2%  softirqs.CPU211.TIMER
    123594 ±  8%     +20.2%     148590 ±  3%  softirqs.CPU212.TIMER
    124179 ± 11%     +16.0%     144109 ±  3%  softirqs.CPU213.TIMER
    117713 ± 10%     +20.0%     141292 ±  3%  softirqs.CPU214.TIMER
    121682 ± 11%     +13.7%     138390        softirqs.CPU215.TIMER
    125855 ±  9%     +14.6%     144201 ±  2%  softirqs.CPU216.TIMER
    119457 ±  7%     +18.1%     141135 ±  2%  softirqs.CPU217.TIMER
    129515 ±  7%     +15.2%     149225 ±  2%  softirqs.CPU218.TIMER
    126641 ±  8%     +16.2%     147164        softirqs.CPU219.TIMER
    118461 ± 12%     +16.9%     138460 ±  4%  softirqs.CPU220.TIMER
    118142 ± 12%     +17.8%     139186 ±  4%  softirqs.CPU221.TIMER
    123028 ± 11%     +17.9%     145077 ±  4%  softirqs.CPU223.TIMER
    116760 ± 11%     +21.0%     141309 ±  4%  softirqs.CPU224.TIMER
    117769 ± 13%     +22.9%     144787 ±  2%  softirqs.CPU225.TIMER
    115921 ±  9%     +21.0%     140316 ±  3%  softirqs.CPU226.TIMER
    117756 ± 12%     +23.8%     145835 ±  4%  softirqs.CPU227.TIMER
    127787 ±  7%     +13.1%     144484 ±  3%  softirqs.CPU228.TIMER
    129622 ±  8%     +15.0%     149116        softirqs.CPU229.TIMER
    112443 ±  9%     +17.6%     132259 ±  6%  softirqs.CPU230.TIMER
    111976 ±  8%     +17.5%     131521 ±  4%  softirqs.CPU231.TIMER
    120500 ±  9%     +16.7%     140660 ±  3%  softirqs.CPU233.TIMER
    115604 ± 10%     +15.7%     133702 ±  3%  softirqs.CPU234.TIMER
    125051 ± 11%     +14.2%     142772 ±  2%  softirqs.CPU235.TIMER
    119846 ±  9%     +22.5%     146843 ±  2%  softirqs.CPU236.TIMER
    125549 ± 10%     +18.4%     148672 ±  3%  softirqs.CPU237.TIMER
    121442 ± 10%     +16.6%     141632 ±  2%  softirqs.CPU238.TIMER
    110362 ±  3%     +26.7%     139858 ±  2%  softirqs.CPU24.TIMER
    120794 ±  9%     +17.6%     141997        softirqs.CPU240.TIMER
    113824 ±  9%     +17.1%     133278 ±  3%  softirqs.CPU241.TIMER
    119687 ± 11%     +21.7%     145654        softirqs.CPU242.TIMER
    115912 ± 12%     +19.2%     138121        softirqs.CPU243.TIMER
    110237 ±  8%     +14.0%     125635 ±  7%  softirqs.CPU244.TIMER
    111771 ±  8%     +16.2%     129872 ±  6%  softirqs.CPU245.TIMER
    118060 ± 11%     +18.0%     139342 ±  2%  softirqs.CPU246.TIMER
    116693 ± 13%     +20.5%     140666        softirqs.CPU247.TIMER
    118072 ±  7%     +28.4%     151653        softirqs.CPU248.TIMER
    108763 ±  9%     +26.2%     137221        softirqs.CPU249.TIMER
    122016 ± 10%     +19.8%     146178        softirqs.CPU250.TIMER
    117815 ± 10%     +17.3%     138149        softirqs.CPU251.TIMER
    121858 ± 10%     +19.6%     145741 ±  3%  softirqs.CPU253.TIMER
    107240 ±  6%     +12.1%     120268 ±  4%  softirqs.CPU254.TIMER
    108103 ±  6%     +17.9%     127505 ±  5%  softirqs.CPU255.TIMER
    116381 ± 12%     +15.6%     134529 ±  2%  softirqs.CPU256.TIMER
    122438 ±  9%     +17.1%     143331 ±  3%  softirqs.CPU259.TIMER
    120442 ± 10%     +17.3%     141291 ±  3%  softirqs.CPU260.TIMER
    122688 ± 10%     +18.3%     145140 ±  2%  softirqs.CPU261.TIMER
    120694 ±  8%     +20.5%     145449 ±  3%  softirqs.CPU262.TIMER
    114187 ±  9%     +11.8%     127621 ±  3%  softirqs.CPU266.TIMER
    107478 ±  6%     +10.0%     118190 ±  6%  softirqs.CPU269.TIMER
    107966 ±  5%     +11.4%     120301 ±  7%  softirqs.CPU273.TIMER
    109304 ±  6%     +13.9%     124478 ±  4%  softirqs.CPU276.TIMER
    110399 ±  8%     +16.2%     128241 ±  5%  softirqs.CPU277.TIMER
    106029 ±  6%      +9.9%     116510 ±  7%  softirqs.CPU281.TIMER
    106824 ±  4%     +19.7%     127862 ±  9%  softirqs.CPU283.TIMER
    105328 ±  5%     +12.4%     118346 ±  9%  softirqs.CPU284.TIMER
     98787 ±  5%     +11.0%     109671 ±  4%  softirqs.CPU287.TIMER
    109815 ±  6%     +22.1%     134129 ±  4%  softirqs.CPU36.TIMER
    109122 ±  2%     +29.2%     140981 ±  4%  softirqs.CPU37.TIMER
    108522 ±  2%     +23.9%     134419 ±  4%  softirqs.CPU38.TIMER
    109707 ±  3%     +26.6%     138933 ±  4%  softirqs.CPU39.TIMER
    107644 ±  4%     +15.6%     124484 ±  5%  softirqs.CPU42.TIMER
    108155 ±  4%     +10.8%     119800 ±  7%  softirqs.CPU45.TIMER
    112088 ±  7%     +16.9%     130979 ±  6%  softirqs.CPU47.TIMER
    112818 ±  8%     +14.3%     128922 ±  6%  softirqs.CPU49.TIMER
    110167 ±  3%     +17.6%     129599 ±  4%  softirqs.CPU59.TIMER
    107521 ±  5%     +11.1%     119499 ±  3%  softirqs.CPU62.TIMER
    105396 ±  4%     +11.2%     117248 ±  7%  softirqs.CPU66.TIMER
    109136 ±  5%      +9.1%     119076 ±  3%  softirqs.CPU68.TIMER
    107512 ±  4%     +11.6%     120020 ±  5%  softirqs.CPU69.TIMER
    109104 ±  5%     +20.5%     131425 ±  6%  softirqs.CPU7.TIMER
    109245 ±  6%      +8.5%     118577 ±  3%  softirqs.CPU70.TIMER
    116561 ± 10%     +24.6%     145283 ±  3%  softirqs.CPU72.TIMER
     42353 ±  7%     -12.2%      37180 ±  9%  softirqs.CPU73.RCU
    119948 ± 12%     +15.9%     139063 ±  3%  softirqs.CPU73.TIMER
    110624 ±  7%     +20.1%     132912 ±  3%  softirqs.CPU74.TIMER
    115257 ± 12%     +19.8%     138101 ±  3%  softirqs.CPU75.TIMER
    119711 ± 12%     +16.5%     139480 ±  3%  softirqs.CPU76.TIMER
    117787 ± 12%     +19.3%     140577 ±  4%  softirqs.CPU77.TIMER
    117562 ±  9%     +19.8%     140828 ±  4%  softirqs.CPU80.TIMER
    116478 ±  8%     +18.3%     137819 ±  4%  softirqs.CPU81.TIMER
    124014 ± 11%     +13.9%     141248 ±  3%  softirqs.CPU82.TIMER
    120670 ± 10%     +17.8%     142089 ±  2%  softirqs.CPU83.TIMER
    117102 ±  7%     +23.9%     145050        softirqs.CPU85.TIMER
    125172 ±  8%     +18.8%     148643 ±  2%  softirqs.CPU86.TIMER
    121015 ±  8%     +21.2%     146694 ±  2%  softirqs.CPU87.TIMER
    127523 ± 10%     +14.4%     145924 ±  2%  softirqs.CPU88.TIMER
    119199 ± 11%     +17.0%     139463 ±  3%  softirqs.CPU90.TIMER
    125653 ±  9%     +18.7%     149146 ±  2%  softirqs.CPU92.TIMER
    121635 ± 10%     +21.5%     147752        softirqs.CPU93.TIMER
    121228 ±  8%     +22.1%     147960        softirqs.CPU95.TIMER
    111706 ±  8%     +16.4%     130048        softirqs.CPU96.TIMER
    115196 ± 11%     +16.1%     133731 ±  3%  softirqs.CPU97.TIMER
    121196 ± 13%     +20.3%     145771        softirqs.CPU98.TIMER
    126502 ± 11%     +18.8%     150267        softirqs.CPU99.TIMER
  33761845 ±  7%     +15.1%   38854076 ±  2%  softirqs.TIMER



***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/page_fault2/will-it-scale/0x11

commit: 
  v5.7-rc3
  8aebfffacf ("configfs: fix config_item refcnt leak in configfs_rmdir()")

        v5.7-rc3 8aebfffacfa379ba400da573a5b 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      1945           -14.1%       1671        will-it-scale.per_process_ops
    978.60 ±  7%     -99.8%       2.00 ± 50%  will-it-scale.time.involuntary_context_switches
    158244           -93.7%       9896        will-it-scale.time.maximum_resident_set_size
     87626           -93.0%       6109        will-it-scale.time.minor_page_faults
      6.80 ±  5%    -100.0%       0.00        will-it-scale.time.percent_of_cpu_this_job_got
     22.04           -99.2%       0.18        will-it-scale.time.system_time
    560392           -14.1%     481535        will-it-scale.workload
     90.00            -1.1%      89.00        vmstat.cpu.sy
     45.31 ±  4%      +5.6%      47.83 ±  2%  boot-time.boot
     11253 ±  4%      +6.3%      11962 ±  3%  boot-time.idle
    294444 ± 14%     -26.3%     217051 ±  4%  meminfo.Mapped
   1096009           +12.9%    1236982        meminfo.Unevictable
      7.71            +0.9        8.60        mpstat.cpu.all.idle%
      1.23 ±  4%      +0.3        1.55 ±  4%  mpstat.cpu.all.usr%
 2.043e+08 ±  4%     -15.5%  1.726e+08        numa-numastat.node0.local_node
 2.043e+08 ±  4%     -15.5%  1.726e+08        numa-numastat.node0.numa_hit
    430.09 ±  4%    -100.0%       0.00        uptime.boot
     16131 ±  3%    -100.0%       0.00        uptime.idle
      6498 ± 10%     +16.8%       7587 ±  5%  slabinfo.kmalloc-rcl-64.active_objs
      6498 ± 10%     +16.8%       7587 ±  5%  slabinfo.kmalloc-rcl-64.num_objs
    706.80 ±  4%      +9.6%     775.00 ±  4%  slabinfo.numa_policy.active_objs
    706.80 ±  4%      +9.6%     775.00 ±  4%  slabinfo.numa_policy.num_objs
      1237 ± 13%    -100.0%       0.00        slabinfo.skbuff_ext_cache.active_objs
      1237 ± 13%    -100.0%       0.00        slabinfo.skbuff_ext_cache.num_objs
    282427 ± 14%     -27.3%     205355 ±  5%  numa-meminfo.node0.Mapped
    565175           +12.5%     635871        numa-meminfo.node0.Unevictable
    530833           +13.2%     601112        numa-meminfo.node1.FilePages
     17652 ±  6%     +20.6%      21280 ±  7%  numa-meminfo.node1.KReclaimable
     11323           +13.6%      12867        numa-meminfo.node1.Mapped
    623495           +11.8%     696907        numa-meminfo.node1.MemUsed
     17652 ±  6%     +20.6%      21280 ±  7%  numa-meminfo.node1.SReclaimable
     32701 ±  5%      +9.5%      35810 ±  5%  numa-meminfo.node1.Slab
    530833           +13.2%     601111        numa-meminfo.node1.Unevictable
    107.40 ± 45%     +31.3%     141.00        proc-vmstat.nr_isolated_anon
     74270 ± 15%     -26.0%      54987 ±  4%  proc-vmstat.nr_mapped
     49908            +3.7%      51731        proc-vmstat.nr_slab_reclaimable
    274002           +12.9%     309245        proc-vmstat.nr_unevictable
    274002           +12.9%     309245        proc-vmstat.nr_zone_unevictable
    109.00 ± 40%   +1585.8%       1837 ± 89%  proc-vmstat.numa_hint_faults
    109.00 ± 40%   +1585.8%       1837 ± 89%  proc-vmstat.numa_hint_faults_local
 2.043e+08 ±  4%     -15.5%  1.726e+08        proc-vmstat.numa_hit
 2.043e+08 ±  4%     -15.5%  1.726e+08        proc-vmstat.numa_local
     57344 ± 38%     +89.4%     108606 ±  3%  proc-vmstat.numa_pte_updates
     59052 ±121%     -99.5%     311.50        proc-vmstat.pgactivate
 2.002e+08 ±  4%     -15.3%  1.695e+08        proc-vmstat.pgalloc_normal
 1.942e+08 ±  5%     -16.3%  1.626e+08        proc-vmstat.pgfault
 2.037e+08 ±  4%     -15.4%  1.722e+08        proc-vmstat.pgfree
    113.20 ± 46%     +30.7%     148.00 ±  6%  numa-vmstat.node0.nr_isolated_anon
     70785 ± 14%     -29.7%      49758 ±  3%  numa-vmstat.node0.nr_mapped
    141293           +12.5%     158966        numa-vmstat.node0.nr_unevictable
    141293           +12.5%     158966        numa-vmstat.node0.nr_zone_unevictable
 1.107e+08 ±  4%     -15.3%   93735997        numa-vmstat.node0.numa_hit
    154693           +11.3%     172115        numa-vmstat.node0.numa_interleave
 1.107e+08 ±  4%     -15.3%   93737665        numa-vmstat.node0.numa_local
    132708           +13.2%     150278        numa-vmstat.node1.nr_file_pages
      2836           +13.6%       3222        numa-vmstat.node1.nr_mapped
      4413 ±  6%     +20.6%       5320 ±  7%  numa-vmstat.node1.nr_slab_reclaimable
    132708           +13.2%     150277        numa-vmstat.node1.nr_unevictable
    132708           +13.2%     150277        numa-vmstat.node1.nr_zone_unevictable
    155429           +11.1%     172634        numa-vmstat.node1.numa_hit
    154844           +11.1%     172048        numa-vmstat.node1.numa_interleave
    154857           +11.1%     172060        numa-vmstat.node1.numa_other
     64.47 ±  9%     -26.9%      47.16 ± 12%  sched_debug.cfs_rq:/.nr_spread_over.avg
     39.33 ±  8%     -25.7%      29.23 ± 15%  sched_debug.cfs_rq:/.nr_spread_over.min
 -10184167           -37.1%   -6401034        sched_debug.cfs_rq:/.spread0.min
      1352 ±  6%     +14.2%       1544        sched_debug.cfs_rq:/.util_avg.max
    140.69 ±  9%     +16.3%     163.69 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
   5737899 ±  8%     +22.0%    7002653 ±  2%  sched_debug.cpu.avg_idle.avg
  25342220 ±  7%     +23.9%   31404672        sched_debug.cpu.avg_idle.max
   4341642 ±  5%     +26.7%    5499420        sched_debug.cpu.avg_idle.stddev
    614.61 ±  5%     +11.8%     687.14 ±  2%  sched_debug.cpu.curr->pid.stddev
   2726049 ±  8%     +24.7%    3400632        sched_debug.cpu.max_idle_balance_cost.avg
  11766110 ±  4%     +21.6%   14305126        sched_debug.cpu.max_idle_balance_cost.max
   2097272 ±  6%     +28.6%    2698037        sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ± 12%     -46.1%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
     17846 ± 11%     -17.3%      14763 ±  3%  sched_debug.cpu.nr_switches.max
      0.00 ± 55%    +134.9%       0.00 ±  2%  sched_debug.cpu.nr_uninterruptible.avg
    237.17 ±  8%     -20.9%     187.48 ±  7%  sched_debug.cpu.ttwu_count.min
     11.28 ±  4%      -8.8        2.53 ± 32%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     11.28 ±  4%      -8.8        2.53 ± 32%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.98 ±  4%      -8.7        2.26 ± 35%  perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.98 ±  4%      -8.7        2.26 ± 35%  perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.44 ±  2%      -0.3        3.12 ±  3%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap
      3.44 ±  2%      -0.3        3.12 ±  3%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      3.42 ±  2%      -0.3        3.12 ±  3%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      3.40 ±  2%      -0.3        3.10 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu.tlb_finish_mmu
      3.40 ±  2%      -0.3        3.10 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region
     19.79 ±  2%      +8.4       28.19 ±  4%  perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     19.79 ±  2%      +8.4       28.19 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     19.82 ±  2%      +8.4       28.23 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.munmap
     19.82 ±  2%      +8.4       28.23 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     19.82 ±  2%      +8.4       28.23 ±  4%  perf-profile.calltrace.cycles-pp.munmap
      0.19 ± 34%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.free_unref_page_list
      0.15 ± 35%      -0.1        0.06        perf-profile.children.cycles-pp.free_pcppages_bulk
     19.82 ±  2%      +8.4       28.23 ±  4%  perf-profile.children.cycles-pp.munmap
     30744 ± 22%     +20.4%      37021 ± 19%  softirqs.CPU0.SCHED
     40150 ±  6%     -14.2%      34450 ±  3%  softirqs.CPU10.TIMER
     40980 ±  5%     -12.6%      35798 ±  4%  softirqs.CPU102.TIMER
     41457 ±  9%     -14.0%      35671 ±  3%  softirqs.CPU11.TIMER
     40837 ±  6%     -13.0%      35533 ±  3%  softirqs.CPU116.TIMER
      2991 ±  7%    +137.4%       7100 ± 56%  softirqs.CPU118.SCHED
     40675 ±  4%     -13.3%      35251 ±  3%  softirqs.CPU125.TIMER
      2916 ± 11%    +218.9%       9298 ± 66%  softirqs.CPU153.SCHED
     43448 ±  9%     -15.7%      36639 ±  3%  softirqs.CPU16.TIMER
      2948 ± 12%    +131.9%       6838 ± 56%  softirqs.CPU165.SCHED
     39548 ±  5%     -12.3%      34675        softirqs.CPU187.TIMER
     43043 ± 13%     -18.2%      35210 ±  3%  softirqs.CPU193.TIMER
     44340 ±  8%     -18.1%      36293 ±  2%  softirqs.CPU2.TIMER
     42478 ± 14%     -16.6%      35430        softirqs.CPU200.TIMER
     40184 ±  4%     -11.6%      35535 ±  3%  softirqs.CPU213.TIMER
      2891 ± 15%    +125.4%       6517 ± 57%  softirqs.CPU221.SCHED
      4677 ± 69%     -46.1%       2522 ±  2%  softirqs.CPU239.SCHED
     39582 ±  4%     -12.0%      34848 ±  2%  softirqs.CPU264.TIMER
     39242 ±  3%     -12.1%      34500 ±  2%  softirqs.CPU273.TIMER
     39720 ±  5%     -11.0%      35370        softirqs.CPU274.TIMER
     39463 ±  5%     -14.5%      33747 ±  3%  softirqs.CPU275.TIMER
     39046 ±  5%     -11.1%      34704 ±  2%  softirqs.CPU282.TIMER
     39107 ±  5%     -12.9%      34052 ±  3%  softirqs.CPU284.TIMER
     39427 ±  4%     -12.2%      34614        softirqs.CPU285.TIMER
     38657 ±  5%     -14.5%      33049        softirqs.CPU286.TIMER
     40552 ±  4%     -11.7%      35817 ±  3%  softirqs.CPU32.TIMER
      2980 ±  8%    +292.0%      11681 ± 72%  softirqs.CPU47.SCHED
     41953 ±  9%     -13.9%      36109 ±  4%  softirqs.CPU50.TIMER
     40401 ±  4%     -11.1%      35926 ±  2%  softirqs.CPU56.TIMER
     42468 ±  8%     -14.4%      36338 ±  4%  softirqs.CPU58.TIMER
     43418 ± 10%     -17.0%      36044        softirqs.CPU60.TIMER
      2199 ±  4%    +185.4%       6278 ± 63%  softirqs.CPU71.RCU
     40825 ±  3%     -12.2%      35851 ±  3%  softirqs.CPU72.TIMER
      2814 ± 12%    +154.9%       7174 ± 56%  softirqs.CPU78.SCHED
     44413 ± 31%     +20.6%      53565 ± 24%  interrupts.CPU0.RES:Rescheduling_interrupts
      1311 ± 24%     -89.3%     140.00 ± 71%  interrupts.CPU1.RES:Rescheduling_interrupts
     25362 ± 49%     +47.1%      37320 ± 36%  interrupts.CPU10.NMI:Non-maskable_interrupts
     25362 ± 49%     +47.1%      37320 ± 36%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
     23833 ± 44%     +50.0%      35755 ± 38%  interrupts.CPU100.NMI:Non-maskable_interrupts
     23833 ± 44%     +50.0%      35755 ± 38%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
     21886 ± 55%     +57.9%      34568 ± 26%  interrupts.CPU101.NMI:Non-maskable_interrupts
     21886 ± 55%     +57.9%      34568 ± 26%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
    115.60 ± 67%     -68.9%      36.00 ± 13%  interrupts.CPU102.RES:Rescheduling_interrupts
     17917 ± 40%     +90.8%      34188 ± 25%  interrupts.CPU107.NMI:Non-maskable_interrupts
     17917 ± 40%     +90.8%      34188 ± 25%  interrupts.CPU107.PMI:Performance_monitoring_interrupts
    116731 ±  6%      -9.9%     105211        interrupts.CPU108.LOC:Local_timer_interrupts
    113992 ±  3%     -10.3%     102241        interrupts.CPU109.LOC:Local_timer_interrupts
    117808           -10.7%     105251        interrupts.CPU113.LOC:Local_timer_interrupts
     23944 ± 43%     +51.1%      36178 ± 39%  interrupts.CPU113.NMI:Non-maskable_interrupts
     23944 ± 43%     +51.1%      36178 ± 39%  interrupts.CPU113.PMI:Performance_monitoring_interrupts
    112586 ±  2%      -9.6%     101751        interrupts.CPU115.LOC:Local_timer_interrupts
     17938 ± 39%     +92.1%      34458 ± 25%  interrupts.CPU115.NMI:Non-maskable_interrupts
     17938 ± 39%     +92.1%      34458 ± 25%  interrupts.CPU115.PMI:Performance_monitoring_interrupts
    119013 ±  3%     -12.4%     104204        interrupts.CPU116.LOC:Local_timer_interrupts
     23941 ± 46%     +44.6%      34625 ± 24%  interrupts.CPU116.NMI:Non-maskable_interrupts
     23941 ± 46%     +44.6%      34625 ± 24%  interrupts.CPU116.PMI:Performance_monitoring_interrupts
     62.60 ± 46%     -55.3%      28.00 ±  7%  interrupts.CPU117.RES:Rescheduling_interrupts
     22842 ± 42%     +52.1%      34744 ± 27%  interrupts.CPU118.NMI:Non-maskable_interrupts
     22842 ± 42%     +52.1%      34744 ± 27%  interrupts.CPU118.PMI:Performance_monitoring_interrupts
     55.40 ± 60%  +14820.6%       8266 ± 98%  interrupts.CPU118.RES:Rescheduling_interrupts
    119354 ±  7%     -14.2%     102437        interrupts.CPU119.LOC:Local_timer_interrupts
     21602 ± 17%     +69.2%      36544 ± 39%  interrupts.CPU119.NMI:Non-maskable_interrupts
     21602 ± 17%     +69.2%      36544 ± 39%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
     19795 ± 31%     +85.9%      36795 ± 40%  interrupts.CPU122.NMI:Non-maskable_interrupts
     19795 ± 31%     +85.9%      36795 ± 40%  interrupts.CPU122.PMI:Performance_monitoring_interrupts
      2230 ±182%     -98.3%      37.00        interrupts.CPU122.RES:Rescheduling_interrupts
     23701 ± 46%     +52.3%      36100 ± 39%  interrupts.CPU123.NMI:Non-maskable_interrupts
     23701 ± 46%     +52.3%      36100 ± 39%  interrupts.CPU123.PMI:Performance_monitoring_interrupts
      3061 ±175%     -99.0%      31.50 ± 11%  interrupts.CPU124.RES:Rescheduling_interrupts
    120574 ±  5%     -13.6%     104157 ±  2%  interrupts.CPU126.LOC:Local_timer_interrupts
     23625 ± 46%     +45.5%      34373 ± 25%  interrupts.CPU126.NMI:Non-maskable_interrupts
     23625 ± 46%     +45.5%      34373 ± 25%  interrupts.CPU126.PMI:Performance_monitoring_interrupts
    117690 ±  4%     -14.2%     100992        interrupts.CPU127.LOC:Local_timer_interrupts
     26834 ± 31%     +38.1%      37049 ± 30%  interrupts.CPU13.NMI:Non-maskable_interrupts
     26834 ± 31%     +38.1%      37049 ± 30%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
     23710 ± 46%     +46.9%      34829 ± 26%  interrupts.CPU131.NMI:Non-maskable_interrupts
     23710 ± 46%     +46.9%      34829 ± 26%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
    121993 ±  6%     -19.3%      98444        interrupts.CPU132.LOC:Local_timer_interrupts
    118641 ±  5%     -13.3%     102859        interrupts.CPU133.LOC:Local_timer_interrupts
    116855 ±  5%     -15.2%      99116        interrupts.CPU135.LOC:Local_timer_interrupts
     21994 ± 56%     +65.0%      36281 ± 39%  interrupts.CPU136.NMI:Non-maskable_interrupts
     21994 ± 56%     +65.0%      36281 ± 39%  interrupts.CPU136.PMI:Performance_monitoring_interrupts
     24176 ± 46%     +50.0%      36269 ± 39%  interrupts.CPU137.NMI:Non-maskable_interrupts
     24176 ± 46%     +50.0%      36269 ± 39%  interrupts.CPU137.PMI:Performance_monitoring_interrupts
     19777 ± 31%     +85.4%      36666 ± 40%  interrupts.CPU139.NMI:Non-maskable_interrupts
     19777 ± 31%     +85.4%      36666 ± 40%  interrupts.CPU139.PMI:Performance_monitoring_interrupts
     27476 ± 37%     +74.8%      48017 ±  7%  interrupts.CPU14.NMI:Non-maskable_interrupts
     27476 ± 37%     +74.8%      48017 ±  7%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
     17615 ± 39%     +95.8%      34493 ± 28%  interrupts.CPU141.NMI:Non-maskable_interrupts
     17615 ± 39%     +95.8%      34493 ± 28%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
     19759 ± 31%     +82.6%      36089 ± 38%  interrupts.CPU142.NMI:Non-maskable_interrupts
     19759 ± 31%     +82.6%      36089 ± 38%  interrupts.CPU142.PMI:Performance_monitoring_interrupts
     47.60 ± 22%  +15915.8%       7623 ± 98%  interrupts.CPU143.RES:Rescheduling_interrupts
     21504 ± 56%     +59.1%      34204 ± 28%  interrupts.CPU145.NMI:Non-maskable_interrupts
     21504 ± 56%     +59.1%      34204 ± 28%  interrupts.CPU145.PMI:Performance_monitoring_interrupts
    121982 ±  9%     -14.0%     104947 ±  2%  interrupts.CPU146.LOC:Local_timer_interrupts
    121216 ±  4%     -12.6%     105990        interrupts.CPU148.LOC:Local_timer_interrupts
     66.60 ± 57%    +795.6%     596.50 ± 82%  interrupts.CPU148.RES:Rescheduling_interrupts
    119881 ±  3%     -13.4%     103848        interrupts.CPU15.LOC:Local_timer_interrupts
     58.60 ± 73%   +9627.0%       5700 ± 56%  interrupts.CPU152.RES:Rescheduling_interrupts
     62.60 ± 20%  +19400.8%      12207 ± 99%  interrupts.CPU153.RES:Rescheduling_interrupts
     62.80 ± 44%    +117.4%     136.50 ±  4%  interrupts.CPU154.RES:Rescheduling_interrupts
     81.40 ± 30%     +65.2%     134.50 ± 13%  interrupts.CPU157.RES:Rescheduling_interrupts
    122848 ±  5%     -14.8%     104642        interrupts.CPU158.LOC:Local_timer_interrupts
     18722 ± 29%     +78.7%      33462 ± 28%  interrupts.CPU160.NMI:Non-maskable_interrupts
     18722 ± 29%     +78.7%      33462 ± 28%  interrupts.CPU160.PMI:Performance_monitoring_interrupts
     54.80 ± 81%   +1773.2%       1026 ± 95%  interrupts.CPU164.RES:Rescheduling_interrupts
     51.80 ± 41%  +14703.1%       7668 ± 99%  interrupts.CPU165.RES:Rescheduling_interrupts
    119687 ±  3%     -14.5%     102376        interrupts.CPU167.LOC:Local_timer_interrupts
     19851 ± 18%     +66.0%      32947 ± 27%  interrupts.CPU167.NMI:Non-maskable_interrupts
     19851 ± 18%     +66.0%      32947 ± 27%  interrupts.CPU167.PMI:Performance_monitoring_interrupts
     20526 ± 53%     +62.9%      33435 ± 37%  interrupts.CPU168.NMI:Non-maskable_interrupts
     20526 ± 53%     +62.9%      33435 ± 37%  interrupts.CPU168.PMI:Performance_monitoring_interrupts
     22011 ± 42%     +55.3%      34182 ± 39%  interrupts.CPU169.NMI:Non-maskable_interrupts
     22011 ± 42%     +55.3%      34182 ± 39%  interrupts.CPU169.PMI:Performance_monitoring_interrupts
     16490 ± 43%    +105.5%      33884 ± 38%  interrupts.CPU175.NMI:Non-maskable_interrupts
     16490 ± 43%    +105.5%      33884 ± 38%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
     98.80 ± 49%     -72.7%      27.00 ±  3%  interrupts.CPU175.RES:Rescheduling_interrupts
    119432 ±  6%     -15.3%     101195        interrupts.CPU177.LOC:Local_timer_interrupts
     21582 ± 42%     +97.4%      42614 ±  7%  interrupts.CPU180.NMI:Non-maskable_interrupts
     21582 ± 42%     +97.4%      42614 ±  7%  interrupts.CPU180.PMI:Performance_monitoring_interrupts
    119772 ±  3%     -13.7%     103342        interrupts.CPU183.LOC:Local_timer_interrupts
    114335 ±  5%     -10.8%     101938        interrupts.CPU184.LOC:Local_timer_interrupts
     22474 ± 43%     +91.1%      42954 ±  6%  interrupts.CPU184.NMI:Non-maskable_interrupts
     22474 ± 43%     +91.1%      42954 ±  6%  interrupts.CPU184.PMI:Performance_monitoring_interrupts
     67.00 ± 60%     -61.2%      26.00 ±  3%  interrupts.CPU184.RES:Rescheduling_interrupts
    117913 ±  5%     -10.6%     105418        interrupts.CPU185.LOC:Local_timer_interrupts
     20531 ± 54%     +51.4%      31081 ± 26%  interrupts.CPU185.NMI:Non-maskable_interrupts
     20531 ± 54%     +51.4%      31081 ± 26%  interrupts.CPU185.PMI:Performance_monitoring_interrupts
    114480 ±  2%     -13.8%      98655        interrupts.CPU187.LOC:Local_timer_interrupts
    117946 ±  7%     -13.1%     102502 ±  2%  interrupts.CPU188.LOC:Local_timer_interrupts
    117037 ±  6%     -12.1%     102841        interrupts.CPU189.LOC:Local_timer_interrupts
     20504 ± 54%     +53.6%      31491 ± 25%  interrupts.CPU189.NMI:Non-maskable_interrupts
     20504 ± 54%     +53.6%      31491 ± 25%  interrupts.CPU189.PMI:Performance_monitoring_interrupts
    119458 ±  5%     -16.3%      99979 ±  2%  interrupts.CPU191.LOC:Local_timer_interrupts
     17934 ± 35%     +84.9%      33158 ± 38%  interrupts.CPU191.NMI:Non-maskable_interrupts
     17934 ± 35%     +84.9%      33158 ± 38%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
    117361 ±  4%     -11.9%     103355        interrupts.CPU193.LOC:Local_timer_interrupts
     19054 ± 37%     +65.3%      31490 ± 26%  interrupts.CPU193.NMI:Non-maskable_interrupts
     19054 ± 37%     +65.3%      31490 ± 26%  interrupts.CPU193.PMI:Performance_monitoring_interrupts
    744.20 ±175%     -94.4%      41.50 ± 18%  interrupts.CPU193.RES:Rescheduling_interrupts
    120732 ±  6%     -15.1%     102480        interrupts.CPU194.LOC:Local_timer_interrupts
     16696 ± 45%    +155.6%      42678 ±  7%  interrupts.CPU194.NMI:Non-maskable_interrupts
     16696 ± 45%    +155.6%      42678 ±  7%  interrupts.CPU194.PMI:Performance_monitoring_interrupts
     20990 ± 59%     +51.7%      31840 ± 41%  interrupts.CPU196.NMI:Non-maskable_interrupts
     20990 ± 59%     +51.7%      31840 ± 41%  interrupts.CPU196.PMI:Performance_monitoring_interrupts
    119369 ±  4%     -14.2%     102421 ±  3%  interrupts.CPU199.LOC:Local_timer_interrupts
     16633 ± 45%     +83.9%      30586 ± 27%  interrupts.CPU199.NMI:Non-maskable_interrupts
     16633 ± 45%     +83.9%      30586 ± 27%  interrupts.CPU199.PMI:Performance_monitoring_interrupts
    118932 ±  7%     -15.5%     100519 ±  3%  interrupts.CPU2.LOC:Local_timer_interrupts
     22407 ± 43%     +34.9%      30220 ± 29%  interrupts.CPU200.NMI:Non-maskable_interrupts
     22407 ± 43%     +34.9%      30220 ± 29%  interrupts.CPU200.PMI:Performance_monitoring_interrupts
    123037 ±  4%     -13.5%     106382 ±  3%  interrupts.CPU202.LOC:Local_timer_interrupts
    116719 ±  6%     -15.4%      98794        interrupts.CPU203.LOC:Local_timer_interrupts
    117259 ±  4%     -12.8%     102265        interrupts.CPU204.LOC:Local_timer_interrupts
     17009 ± 47%     +78.4%      30342 ± 24%  interrupts.CPU205.NMI:Non-maskable_interrupts
     17009 ± 47%     +78.4%      30342 ± 24%  interrupts.CPU205.PMI:Performance_monitoring_interrupts
      3714 ±196%     -99.3%      27.50 ± 20%  interrupts.CPU205.RES:Rescheduling_interrupts
     16910 ± 46%     +91.8%      32437 ± 42%  interrupts.CPU206.NMI:Non-maskable_interrupts
     16910 ± 46%     +91.8%      32437 ± 42%  interrupts.CPU206.PMI:Performance_monitoring_interrupts
    120037 ±  5%     -14.5%     102637 ±  2%  interrupts.CPU21.LOC:Local_timer_interrupts
    117963 ±  4%     -12.1%     103659        interrupts.CPU211.LOC:Local_timer_interrupts
     17939 ± 35%     +65.9%      29764 ± 23%  interrupts.CPU212.NMI:Non-maskable_interrupts
     17939 ± 35%     +65.9%      29764 ± 23%  interrupts.CPU212.PMI:Performance_monitoring_interrupts
    115452 ±  3%     -12.9%     100528        interrupts.CPU214.LOC:Local_timer_interrupts
    466.80 ±178%     -93.3%      31.50 ±  4%  interrupts.CPU217.RES:Rescheduling_interrupts
    121828 ±  5%     -18.3%      99577        interrupts.CPU218.LOC:Local_timer_interrupts
      1660 ±192%     -98.5%      24.50 ± 30%  interrupts.CPU218.RES:Rescheduling_interrupts
    122819 ±  5%     -13.2%     106581        interrupts.CPU22.LOC:Local_timer_interrupts
     50.40 ± 22%  +13769.0%       6990 ± 95%  interrupts.CPU221.RES:Rescheduling_interrupts
     17704 ± 35%     +64.8%      29176 ± 27%  interrupts.CPU222.NMI:Non-maskable_interrupts
     17704 ± 35%     +64.8%      29176 ± 27%  interrupts.CPU222.PMI:Performance_monitoring_interrupts
     61.00 ± 47%   +4351.6%       2715 ± 97%  interrupts.CPU222.RES:Rescheduling_interrupts
     41.80 ± 32%  +15075.8%       6343 ± 87%  interrupts.CPU224.RES:Rescheduling_interrupts
    117616 ±  7%     -13.1%     102187        interrupts.CPU225.LOC:Local_timer_interrupts
     56.40 ± 54%    +640.2%     417.50        interrupts.CPU225.RES:Rescheduling_interrupts
     25163 ± 43%     +57.1%      39525 ± 11%  interrupts.CPU227.NMI:Non-maskable_interrupts
     25163 ± 43%     +57.1%      39525 ± 11%  interrupts.CPU227.PMI:Performance_monitoring_interrupts
     76.60 ± 27%     -56.9%      33.00        interrupts.CPU228.RES:Rescheduling_interrupts
     35.20 ± 31%   +1846.0%     685.00 ± 78%  interrupts.CPU229.RES:Rescheduling_interrupts
     17795 ± 35%    +123.2%      39716 ±  8%  interrupts.CPU231.NMI:Non-maskable_interrupts
     17795 ± 35%    +123.2%      39716 ±  8%  interrupts.CPU231.PMI:Performance_monitoring_interrupts
    119991 ±  4%     -13.3%     104031        interrupts.CPU235.LOC:Local_timer_interrupts
     66.20 ± 97%    +659.1%     502.50 ± 90%  interrupts.CPU237.RES:Rescheduling_interrupts
    120331 ±  3%     -16.0%     101114        interrupts.CPU238.LOC:Local_timer_interrupts
      4018 ±122%     -99.3%      27.50        interrupts.CPU238.RES:Rescheduling_interrupts
    121416 ±  3%     -18.1%      99384 ±  3%  interrupts.CPU239.LOC:Local_timer_interrupts
    134.80 ± 74%     -69.6%      41.00 ± 41%  interrupts.CPU24.RES:Rescheduling_interrupts
    119296 ±  6%     -14.1%     102465        interrupts.CPU240.LOC:Local_timer_interrupts
      2321 ±192%     -99.2%      19.50 ±  7%  interrupts.CPU240.RES:Rescheduling_interrupts
     39.00 ± 62%   +2380.8%     967.50 ± 96%  interrupts.CPU243.RES:Rescheduling_interrupts
     16832 ± 48%    +133.0%      39225 ±  9%  interrupts.CPU244.NMI:Non-maskable_interrupts
     16832 ± 48%    +133.0%      39225 ±  9%  interrupts.CPU244.PMI:Performance_monitoring_interrupts
    114446 ±  5%     -12.4%     100302        interrupts.CPU247.LOC:Local_timer_interrupts
     45.80 ± 51%    +183.8%     130.00 ± 43%  interrupts.CPU247.RES:Rescheduling_interrupts
    116970 ±  8%     -15.8%      98458        interrupts.CPU248.LOC:Local_timer_interrupts
    117471 ±  7%     -13.7%     101346        interrupts.CPU249.LOC:Local_timer_interrupts
      2762 ±196%     -99.0%      28.50 ±  5%  interrupts.CPU249.RES:Rescheduling_interrupts
    122057 ±  6%     -12.0%     107449        interrupts.CPU25.LOC:Local_timer_interrupts
    115489 ±  5%     -15.0%      98109        interrupts.CPU253.LOC:Local_timer_interrupts
     24369 ± 41%     +63.0%      39726 ±  7%  interrupts.CPU253.NMI:Non-maskable_interrupts
     24369 ± 41%     +63.0%      39726 ±  7%  interrupts.CPU253.PMI:Performance_monitoring_interrupts
    119795 ±  3%     -13.9%     103127        interrupts.CPU255.LOC:Local_timer_interrupts
    114991 ±  3%     -11.3%     102052        interrupts.CPU257.LOC:Local_timer_interrupts
     19912 ± 49%     +45.4%      28955 ± 25%  interrupts.CPU259.NMI:Non-maskable_interrupts
     19912 ± 49%     +45.4%      28955 ± 25%  interrupts.CPU259.PMI:Performance_monitoring_interrupts
    126.80 ±122%    +917.4%       1290 ± 94%  interrupts.CPU259.RES:Rescheduling_interrupts
     17936 ± 37%     +59.6%      28626 ± 27%  interrupts.CPU260.NMI:Non-maskable_interrupts
     17936 ± 37%     +59.6%      28626 ± 27%  interrupts.CPU260.PMI:Performance_monitoring_interrupts
     21974 ± 51%     +78.4%      39200 ±  8%  interrupts.CPU261.NMI:Non-maskable_interrupts
     21974 ± 51%     +78.4%      39200 ±  8%  interrupts.CPU261.PMI:Performance_monitoring_interrupts
     37.80 ± 59%  +22356.3%       8488 ± 99%  interrupts.CPU261.RES:Rescheduling_interrupts
    115566 ±  4%     -12.6%     101038 ±  4%  interrupts.CPU263.LOC:Local_timer_interrupts
    117096 ±  4%     -13.8%     100936 ±  3%  interrupts.CPU264.LOC:Local_timer_interrupts
    115215 ±  5%     -11.6%     101842 ±  2%  interrupts.CPU265.LOC:Local_timer_interrupts
     16524 ± 46%     +72.2%      28453 ± 24%  interrupts.CPU265.NMI:Non-maskable_interrupts
     16524 ± 46%     +72.2%      28453 ± 24%  interrupts.CPU265.PMI:Performance_monitoring_interrupts
     16588 ± 46%     +75.8%      29154 ± 27%  interrupts.CPU269.NMI:Non-maskable_interrupts
     16588 ± 46%     +75.8%      29154 ± 27%  interrupts.CPU269.PMI:Performance_monitoring_interrupts
    119181 ±  3%     -12.3%     104485        interrupts.CPU270.LOC:Local_timer_interrupts
    126083 ±  3%     -18.6%     102582        interrupts.CPU271.LOC:Local_timer_interrupts
    120889 ±  4%     -10.4%     108294        interrupts.CPU274.LOC:Local_timer_interrupts
     20501 ± 61%     +86.6%      38253 ±  8%  interrupts.CPU274.NMI:Non-maskable_interrupts
     20501 ± 61%     +86.6%      38253 ±  8%  interrupts.CPU274.PMI:Performance_monitoring_interrupts
     43.60 ± 38%    +311.7%     179.50 ± 79%  interrupts.CPU274.RES:Rescheduling_interrupts
     20162 ± 27%     +90.5%      38419 ± 11%  interrupts.CPU276.NMI:Non-maskable_interrupts
     20162 ± 27%     +90.5%      38419 ± 11%  interrupts.CPU276.PMI:Performance_monitoring_interrupts
    469.60 ±180%     -94.5%      26.00 ± 23%  interrupts.CPU276.RES:Rescheduling_interrupts
    119850 ±  2%     -16.4%     100136 ±  2%  interrupts.CPU277.LOC:Local_timer_interrupts
     19782 ± 50%     +39.4%      27572 ± 22%  interrupts.CPU281.NMI:Non-maskable_interrupts
     19782 ± 50%     +39.4%      27572 ± 22%  interrupts.CPU281.PMI:Performance_monitoring_interrupts
     19467 ± 49%     +98.5%      38647 ± 11%  interrupts.CPU283.NMI:Non-maskable_interrupts
     19467 ± 49%     +98.5%      38647 ± 11%  interrupts.CPU283.PMI:Performance_monitoring_interrupts
    118186 ±  4%     -12.5%     103417        interrupts.CPU284.LOC:Local_timer_interrupts
     20890 ± 41%     +31.7%      27517 ± 22%  interrupts.CPU284.NMI:Non-maskable_interrupts
     20890 ± 41%     +31.7%      27517 ± 22%  interrupts.CPU284.PMI:Performance_monitoring_interrupts
     16703 ± 47%     +79.1%      29908 ± 42%  interrupts.CPU285.NMI:Non-maskable_interrupts
     16703 ± 47%     +79.1%      29908 ± 42%  interrupts.CPU285.PMI:Performance_monitoring_interrupts
    120367 ±  3%     -14.3%     103207 ±  3%  interrupts.CPU286.LOC:Local_timer_interrupts
    203105 ± 16%     -21.3%     159888        interrupts.CPU287.LOC:Local_timer_interrupts
    116843           -12.2%     102631        interrupts.CPU29.LOC:Local_timer_interrupts
     19918 ± 30%     +82.9%      36430 ± 26%  interrupts.CPU29.NMI:Non-maskable_interrupts
     19918 ± 30%     +82.9%      36430 ± 26%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    115537 ±  5%     -12.8%     100799        interrupts.CPU31.LOC:Local_timer_interrupts
     24936 ± 14%     +42.8%      35617 ± 28%  interrupts.CPU31.NMI:Non-maskable_interrupts
     24936 ± 14%     +42.8%      35617 ± 28%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
    117133 ±  4%     -13.5%     101372        interrupts.CPU33.LOC:Local_timer_interrupts
     24316 ± 49%     +99.0%      48379 ±  4%  interrupts.CPU33.NMI:Non-maskable_interrupts
     24316 ± 49%     +99.0%      48379 ±  4%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     26337 ± 39%     +86.4%      49100 ±  5%  interrupts.CPU35.NMI:Non-maskable_interrupts
     26337 ± 39%     +86.4%      49100 ±  5%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
     26228 ± 39%     +43.4%      37614 ± 39%  interrupts.CPU36.NMI:Non-maskable_interrupts
     26228 ± 39%     +43.4%      37614 ± 39%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    116348 ±  5%     -11.5%     102962        interrupts.CPU37.LOC:Local_timer_interrupts
     19410 ± 30%     +85.7%      36045 ± 28%  interrupts.CPU37.NMI:Non-maskable_interrupts
     19410 ± 30%     +85.7%      36045 ± 28%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
     22299 ± 28%    +115.7%      48099        interrupts.CPU39.NMI:Non-maskable_interrupts
     22299 ± 28%    +115.7%      48099        interrupts.CPU39.PMI:Performance_monitoring_interrupts
     20744 ± 27%     +72.7%      35832 ± 28%  interrupts.CPU4.NMI:Non-maskable_interrupts
     20744 ± 27%     +72.7%      35832 ± 28%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    378.60 ± 83%    +694.8%       3009 ± 34%  interrupts.CPU4.RES:Rescheduling_interrupts
    118387 ±  4%     -10.7%     105718        interrupts.CPU40.LOC:Local_timer_interrupts
     25688 ± 35%     +47.2%      37822 ± 37%  interrupts.CPU40.NMI:Non-maskable_interrupts
     25688 ± 35%     +47.2%      37822 ± 37%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    118362 ±  4%     -12.7%     103323 ±  2%  interrupts.CPU41.LOC:Local_timer_interrupts
     92.20 ± 94%     -74.0%      24.00 ±  4%  interrupts.CPU44.RES:Rescheduling_interrupts
    121354 ±  7%     -15.5%     102554 ±  2%  interrupts.CPU46.LOC:Local_timer_interrupts
    119439 ±  4%     -12.3%     104763 ±  3%  interrupts.CPU47.LOC:Local_timer_interrupts
    118786 ±  7%     -14.9%     101047        interrupts.CPU49.LOC:Local_timer_interrupts
    123.60 ± 51%   +3720.4%       4722 ± 96%  interrupts.CPU5.RES:Rescheduling_interrupts
    119935 ±  6%     -15.3%     101599 ±  2%  interrupts.CPU50.LOC:Local_timer_interrupts
     25639 ± 42%     +48.5%      38066 ± 38%  interrupts.CPU53.NMI:Non-maskable_interrupts
     25639 ± 42%     +48.5%      38066 ± 38%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    123762 ±  6%     -17.3%     102351        interrupts.CPU55.LOC:Local_timer_interrupts
    120248 ±  6%     -15.6%     101459        interrupts.CPU61.LOC:Local_timer_interrupts
     27467 ± 31%     +33.8%      36765 ± 27%  interrupts.CPU61.NMI:Non-maskable_interrupts
     27467 ± 31%     +33.8%      36765 ± 27%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
     63.20 ± 38%   +2386.6%       1571 ± 90%  interrupts.CPU62.RES:Rescheduling_interrupts
    120083 ±  7%     -13.0%     104487        interrupts.CPU64.LOC:Local_timer_interrupts
     22757 ± 50%     +67.5%      38115 ± 39%  interrupts.CPU67.NMI:Non-maskable_interrupts
     22757 ± 50%     +67.5%      38115 ± 39%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
     25527 ± 54%     +90.2%      48549 ±  9%  interrupts.CPU73.NMI:Non-maskable_interrupts
     25527 ± 54%     +90.2%      48549 ±  9%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
      1079 ±187%     -96.2%      41.00 ±  7%  interrupts.CPU73.RES:Rescheduling_interrupts
    116963 ±  5%     -11.9%     103068        interrupts.CPU74.LOC:Local_timer_interrupts
    119414 ±  5%     -12.3%     104777        interrupts.CPU76.LOC:Local_timer_interrupts
     96.40 ± 93%   +8929.0%       8704 ± 99%  interrupts.CPU78.RES:Rescheduling_interrupts
    119473 ±  4%     -10.1%     107439        interrupts.CPU8.LOC:Local_timer_interrupts
    101.40 ± 40%   +1138.2%       1255 ± 90%  interrupts.CPU8.RES:Rescheduling_interrupts
    100.40 ± 35%   +2870.6%       2982 ±  8%  interrupts.CPU80.RES:Rescheduling_interrupts
     63.80 ± 18%   +1049.7%     733.50 ± 85%  interrupts.CPU81.RES:Rescheduling_interrupts
    121318 ±  6%     -12.7%     105910        interrupts.CPU82.LOC:Local_timer_interrupts
     20126 ± 26%     +67.5%      33715 ± 24%  interrupts.CPU83.NMI:Non-maskable_interrupts
     20126 ± 26%     +67.5%      33715 ± 24%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
     67.20 ± 46%   +8900.7%       6048 ± 98%  interrupts.CPU84.RES:Rescheduling_interrupts
     20042 ± 27%     +79.5%      35969 ± 40%  interrupts.CPU86.NMI:Non-maskable_interrupts
     20042 ± 27%     +79.5%      35969 ± 40%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
     50.80 ± 31%    +566.3%     338.50 ± 85%  interrupts.CPU88.RES:Rescheduling_interrupts
     22713 ± 17%     +66.5%      37828 ± 36%  interrupts.CPU9.NMI:Non-maskable_interrupts
     22713 ± 17%     +66.5%      37828 ± 36%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    250.80 ±119%    +735.3%       2095 ± 70%  interrupts.CPU9.RES:Rescheduling_interrupts
     21922 ± 53%     +53.3%      33598 ± 25%  interrupts.CPU90.NMI:Non-maskable_interrupts
     21922 ± 53%     +53.3%      33598 ± 25%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
     47.20 ± 39%    +280.3%     179.50 ± 70%  interrupts.CPU92.RES:Rescheduling_interrupts
     23428 ± 43%     +53.5%      35964 ± 40%  interrupts.CPU93.NMI:Non-maskable_interrupts
     23428 ± 43%     +53.5%      35964 ± 40%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
    121016 ±  7%     -12.7%     105693 ±  2%  interrupts.CPU94.LOC:Local_timer_interrupts
     17635 ± 39%     +99.2%      35125 ± 40%  interrupts.CPU95.NMI:Non-maskable_interrupts
     17635 ± 39%     +99.2%      35125 ± 40%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
     25621 ± 32%     +33.1%      34090 ± 27%  interrupts.CPU96.NMI:Non-maskable_interrupts
     25621 ± 32%     +33.1%      34090 ± 27%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
    118451 ±  6%     -12.9%     103182        interrupts.CPU97.LOC:Local_timer_interrupts





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.7.0-rc3-00001-g8aebfffacfa37" of type "text/plain" (202604 bytes)

View attachment "job-script" of type "text/plain" (7472 bytes)

View attachment "job.yaml" of type "text/plain" (5114 bytes)

View attachment "reproduce" of type "text/plain" (338 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ