lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Fri, 12 Apr 2019 23:29:55 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Waiman Long <longman@...hat.com>
Cc:     Ingo Molnar <mingo@...nel.org>,
        Peter Zijlstra <a.p.zijlstra@...llo.nl>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Davidlohr Bueso <dave@...olabs.net>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        Will Deacon <will.deacon@....com>,
        LKML <linux-kernel@...r.kernel.org>,
        "H. Peter Anvin" <hpa@...or.com>, tipbuild@...or.com, lkp@...org
Subject: [locking/rwsem] adc32e8877: will-it-scale.per_thread_ops -21.0%
 regression

Greeting,

FYI, we noticed a -21.0% regression of will-it-scale.per_thread_ops due to commit:


commit: adc32e887793f51b32122c66b0e12b93ff4fdb8c ("locking/rwsem: Enable time-based spinning on reader-owned rwsem")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git WIP.locking/core

in testcase: will-it-scale
on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
with following parameters:

	nr_task: 50%
	mode: thread
	test: page_fault1
	cpufreq_governor: performance

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
  gcc-7/performance/x86_64-rhel-7.6/thread/50%/debian-x86_64-2018-04-03-no-ucode.cgz/lkp-bdw-ep3d/page_fault1/will-it-scale

commit: 
  4407b29e19 ("locking/rwsem: Enable readers spinning on writer")
  adc32e8877 ("locking/rwsem: Enable time-based spinning on reader-owned rwsem")

4407b29e19ffd372 adc32e887793f51b32122c66b0e 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          3:4          -75%            :4     kmsg.DHCP/BOOTP:Reply_not_for_us_on_eth#,op[#]xid[#]
         %stddev     %change         %stddev
             \          |                \  
    222958           -21.0%     176176 ±  2%  will-it-scale.per_thread_ops
     14388 ±  2%     -34.4%       9439 ±  5%  will-it-scale.time.involuntary_context_switches
   5682228           -15.6%    4794550 ±  5%  will-it-scale.time.maximum_resident_set_size
  51676661           -20.9%   40893195 ±  2%  will-it-scale.time.minor_page_faults
      3646           -32.2%       2472 ±  3%  will-it-scale.time.percent_of_cpu_this_job_got
     10734           -31.6%       7346 ±  3%  will-it-scale.time.system_time
    247.33           -60.1%      98.68 ±  2%  will-it-scale.time.user_time
   3946863 ±  2%     +11.3%    4394388 ±  3%  will-it-scale.time.voluntary_context_switches
   9810207           -21.0%    7751796 ±  2%  will-it-scale.workload
     33.93 ± 10%      -6.3%      31.80 ±  9%  boot-time.dhcp
     33321 ± 12%     -15.6%      28119 ± 10%  numa-meminfo.node1.KReclaimable
     33321 ± 12%     -15.6%      28119 ± 10%  numa-meminfo.node1.SReclaimable
      3572 ± 16%     -15.4%       3022        slabinfo.eventpoll_pwq.active_objs
      3572 ± 16%     -15.4%       3022        slabinfo.eventpoll_pwq.num_objs
     58.00           +23.3%      71.50        vmstat.cpu.id
     35.25           -39.7%      21.25 ±  6%  vmstat.procs.r
     27023           +10.7%      29914 ±  2%  vmstat.system.cs
   6378018 ± 79%   +1312.6%   90098072 ± 93%  cpuidle.C1.time
    335896 ± 67%    +907.4%    3383766 ± 89%  cpuidle.C1.usage
    283589 ± 28%    +626.2%    2059300 ± 64%  cpuidle.POLL.time
     42961 ± 24%   +2963.0%    1315908 ± 20%  cpuidle.POLL.usage
   6085134 ±  7%      +5.8%    6438411 ±  8%  meminfo.DirectMap2M
    371628 ±  6%     -24.5%     280495 ± 10%  meminfo.DirectMap4k
     55613           -20.6%      44144 ±  5%  meminfo.Shmem
     24543           -23.8%      18709 ±  2%  meminfo.max_used_kB
     58.04           +13.7       71.78        mpstat.cpu.all.idle%
      0.00 ±113%      +0.0        0.01 ± 91%  mpstat.cpu.all.soft%
     40.97           -13.2       27.80 ±  3%  mpstat.cpu.all.sys%
      1.00            -0.6        0.41 ±  2%  mpstat.cpu.all.usr%
  28932236           -20.2%   23101880        numa-numastat.node0.local_node
  28950748           -20.2%   23111830        numa-numastat.node0.numa_hit
  29228231           -21.1%   23046507 ±  2%  numa-numastat.node1.local_node
  29238249           -21.1%   23065051 ±  2%  numa-numastat.node1.numa_hit
  14882447           -19.8%   11942555        numa-vmstat.node0.numa_hit
  14830170           -19.5%   11932491        numa-vmstat.node0.numa_local
      8330 ± 12%     -15.6%       7029 ± 10%  numa-vmstat.node1.nr_slab_reclaimable
  14989784           -20.7%   11887606 ±  2%  numa-vmstat.node1.numa_hit
  14861825           -21.1%   11718577 ±  2%  numa-vmstat.node1.numa_local
      1205           -30.4%     839.00 ±  2%  turbostat.Avg_MHz
     43.35           -13.2       30.17 ±  2%  turbostat.Busy%
    333289 ± 68%    +914.2%    3380326 ± 89%  turbostat.C1
      0.02 ± 96%      +0.3        0.34 ± 93%  turbostat.C1%
      1.70 ± 10%    +202.2%       5.14 ± 11%  turbostat.Pkg%pc2
    209.59           -13.5%     181.27        turbostat.PkgWatt
     22.38           -15.2%      18.97        turbostat.RAMWatt
    785556            +1.9%     800400        proc-vmstat.nr_active_anon
    773283            +1.9%     788100        proc-vmstat.nr_anon_pages
      1421            +2.7%       1460        proc-vmstat.nr_anon_transparent_hugepages
    274260            -1.5%     270044        proc-vmstat.nr_file_pages
      4599            -3.9%       4420        proc-vmstat.nr_inactive_anon
      6904            -3.5%       6661        proc-vmstat.nr_mapped
      2978            -1.7%       2927        proc-vmstat.nr_page_table_pages
     13918           -20.8%      11029 ±  6%  proc-vmstat.nr_shmem
    785553            +1.9%     800400        proc-vmstat.nr_zone_active_anon
      4599            -3.9%       4420        proc-vmstat.nr_zone_inactive_anon
  58104829           -20.6%   46153605 ±  2%  proc-vmstat.numa_hit
      7075 ±  2%     -60.5%       2793 ± 11%  proc-vmstat.numa_huge_pte_updates
  58076290           -20.6%   46125099 ±  2%  proc-vmstat.numa_local
   3708795 ±  2%     -60.5%    1463190 ± 11%  proc-vmstat.numa_pte_updates
     13129 ±  4%     -32.8%       8818 ± 10%  proc-vmstat.pgactivate
 2.957e+09           -21.0%  2.336e+09 ±  2%  proc-vmstat.pgalloc_normal
  52427112           -20.5%   41660634 ±  2%  proc-vmstat.pgfault
 2.956e+09           -21.0%  2.335e+09 ±  2%  proc-vmstat.pgfree
   5670454           -21.0%    4479930 ±  2%  proc-vmstat.thp_deferred_split_page
   5671853           -21.0%    4481069 ±  2%  proc-vmstat.thp_fault_alloc
   2560591 ±  2%     -51.2%    1249187 ±  6%  sched_debug.cfs_rq:/.min_vruntime.avg
   4759988           -50.7%    2345356 ±  5%  sched_debug.cfs_rq:/.min_vruntime.max
   1567395 ±  7%     -56.5%     681789 ± 13%  sched_debug.cfs_rq:/.min_vruntime.stddev
     20.20 ± 62%     +77.7%      35.91 ± 17%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    932.02 ± 62%     +78.0%       1659 ± 17%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
     23.96 ±  4%     +53.2%      36.71 ± 13%  sched_debug.cfs_rq:/.runnable_load_avg.max
   2142363 ±  8%     -53.8%     989520 ± 21%  sched_debug.cfs_rq:/.spread0.avg
   4341759 ±  4%     -52.0%    2085702 ±  8%  sched_debug.cfs_rq:/.spread0.max
   1567398 ±  7%     -56.5%     681790 ± 13%  sched_debug.cfs_rq:/.spread0.stddev
    559.79 ±  5%     -32.7%     376.85 ±  8%  sched_debug.cfs_rq:/.util_avg.avg
    964.92 ±  2%     -12.4%     844.88 ±  2%  sched_debug.cfs_rq:/.util_avg.max
    388.94 ±  3%     -28.1%     279.66 ± 12%  sched_debug.cfs_rq:/.util_avg.stddev
    873.08 ±  2%     -20.7%     692.33 ±  5%  sched_debug.cfs_rq:/.util_est_enqueued.max
    308.97 ± 12%     -32.1%     209.70 ± 26%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    723517 ±  2%     +12.4%     812920 ±  3%  sched_debug.cpu.avg_idle.avg
    247774 ±  4%     -15.9%     208263 ±  6%  sched_debug.cpu.avg_idle.stddev
     23.96 ±  4%    +228.0%      78.58 ± 92%  sched_debug.cpu.cpu_load[0].max
     24.83 ± 18%    +125.7%      56.04 ± 65%  sched_debug.cpu.cpu_load[1].max
     27.92 ± 15%     +26.3%      35.25 ± 16%  sched_debug.cpu.cpu_load[3].max
     47157 ±  2%     +10.1%      51938 ±  3%  sched_debug.cpu.nr_switches.avg
     94844 ±  3%     +11.2%     105419 ±  2%  sched_debug.cpu.nr_switches.max
      0.26 ±100%      +0.4        0.64 ±  7%  perf-profile.calltrace.cycles-pp._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.82 ±  8%      +0.4        1.23 ±  8%  perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      1.34 ±  4%      +0.6        1.97 ±  8%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      1.34 ±  4%      +0.6        1.97 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
      0.00            +0.7        0.66 ± 20%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.00            +0.7        0.66 ±  9%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault
      0.00            +0.7        0.67 ±  9%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.00            +0.7        0.67 ±  8%  perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
      0.00            +0.7        0.72 ±  8%  perf-profile.calltrace.cycles-pp.__free_pages_ok.release_pages.tlb_flush_mmu_free.arch_tlb_finish_mmu.tlb_finish_mmu
      0.00            +0.8        0.82 ±  8%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu_free.arch_tlb_finish_mmu.tlb_finish_mmu.unmap_region
      0.00            +0.8        0.82 ±  8%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu_free.arch_tlb_finish_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      0.42 ± 59%      +0.8        1.25 ± 23%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.8        0.84 ±  8%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.00            +0.8        0.84 ±  8%  perf-profile.calltrace.cycles-pp.arch_tlb_finish_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap
      0.15 ±173%      +1.0        1.16 ± 25%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      0.00            +1.0        1.01 ±  9%  perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      0.00            +1.0        1.04 ±  8%  perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.06 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.munmap
      0.00            +1.1        1.06 ±  8%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      0.00            +1.1        1.06 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      0.00            +1.1        1.06 ±  8%  perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      0.00            +1.1        1.06 ±  9%  perf-profile.calltrace.cycles-pp.munmap
      0.00            +1.4        1.45 ±  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page
      0.00            +1.5        1.46 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault
    240.08           -18.3%     196.23 ±  2%  perf-stat.i.MPKI
 6.188e+08            +3.3%   6.39e+08        perf-stat.i.branch-instructions
      1.05 ± 16%      +0.6        1.68 ± 18%  perf-stat.i.branch-miss-rate%
   6838156 ± 14%     +61.1%   11019347 ± 18%  perf-stat.i.branch-misses
      6.51            -0.8        5.73        perf-stat.i.cache-miss-rate%
  45477357           -27.6%   32915438 ±  2%  perf-stat.i.cache-misses
 6.986e+08           -17.9%  5.736e+08 ±  2%  perf-stat.i.cache-references
     27189           +10.7%      30108 ±  2%  perf-stat.i.context-switches
     35.08           -30.5%      24.39 ±  2%  perf-stat.i.cpi
 1.021e+11           -30.0%  7.141e+10 ±  2%  perf-stat.i.cpu-cycles
     18.74 ±  2%     -58.1%       7.86 ±  6%  perf-stat.i.cpu-migrations
      2246            -3.4%       2168        perf-stat.i.cycles-between-cache-misses
      0.25 ±  7%      +0.1        0.36 ±  8%  perf-stat.i.dTLB-load-miss-rate%
   2037936 ±  7%     +43.1%    2915695 ±  7%  perf-stat.i.dTLB-load-misses
      0.11            +0.0        0.12 ±  7%  perf-stat.i.dTLB-store-miss-rate%
 1.828e+09           -14.4%  1.564e+09        perf-stat.i.dTLB-stores
     35.58 ±  4%     +11.7       47.31 ±  6%  perf-stat.i.iTLB-load-miss-rate%
    650313 ±  6%     +24.5%     809395 ±  4%  perf-stat.i.iTLB-load-misses
   1176138 ±  2%     -22.9%     907288 ± 10%  perf-stat.i.iTLB-loads
      4555 ±  7%     -19.6%       3664 ±  5%  perf-stat.i.instructions-per-iTLB-miss
      0.03           +44.4%       0.04 ±  2%  perf-stat.i.ipc
    173825           -20.6%     138089 ±  2%  perf-stat.i.minor-faults
      8.18 ±  2%      +1.7        9.87 ±  3%  perf-stat.i.node-load-miss-rate%
    728666 ±  4%     -19.8%     584113 ±  2%  perf-stat.i.node-load-misses
   8200847 ±  3%     -34.7%    5351398 ±  2%  perf-stat.i.node-loads
      1.82            -0.2        1.64 ±  2%  perf-stat.i.node-store-miss-rate%
    447002 ±  2%     -35.5%     288326 ±  4%  perf-stat.i.node-store-misses
  24131452           -28.2%   17333253 ±  3%  perf-stat.i.node-stores
    173824           -20.6%     138088 ±  2%  perf-stat.i.page-faults
    237.41           -18.3%     194.01 ±  2%  perf-stat.overall.MPKI
      1.11 ± 15%      +0.6        1.72 ± 18%  perf-stat.overall.branch-miss-rate%
      6.51            -0.8        5.74        perf-stat.overall.cache-miss-rate%
     34.69           -30.4%      24.15 ±  2%  perf-stat.overall.cpi
      2244            -3.4%       2169        perf-stat.overall.cycles-between-cache-misses
      0.25 ±  7%      +0.1        0.36 ±  8%  perf-stat.overall.dTLB-load-miss-rate%
      0.11            +0.0        0.12 ±  7%  perf-stat.overall.dTLB-store-miss-rate%
     35.58 ±  4%     +11.7       47.27 ±  6%  perf-stat.overall.iTLB-load-miss-rate%
      4545 ±  7%     -19.4%       3661 ±  5%  perf-stat.overall.instructions-per-iTLB-miss
      0.03           +43.7%       0.04 ±  2%  perf-stat.overall.ipc
      8.16 ±  2%      +1.7        9.85 ±  3%  perf-stat.overall.node-load-miss-rate%
      1.82            -0.2        1.64        perf-stat.overall.node-store-miss-rate%
     90187           +27.2%     114676        perf-stat.overall.path-length
 6.168e+08            +3.3%  6.369e+08        perf-stat.ps.branch-instructions
   6819807 ± 14%     +61.1%   10987177 ± 18%  perf-stat.ps.branch-misses
  45322958           -27.6%   32804347 ±  2%  perf-stat.ps.cache-misses
 6.962e+08           -17.9%  5.716e+08 ±  2%  perf-stat.ps.cache-references
     27098           +10.7%      30006 ±  2%  perf-stat.ps.context-switches
 1.017e+11           -30.0%  7.117e+10 ±  2%  perf-stat.ps.cpu-cycles
     18.69 ±  2%     -58.0%       7.84 ±  6%  perf-stat.ps.cpu-migrations
   2031217 ±  7%     +43.1%    2905916 ±  7%  perf-stat.ps.dTLB-load-misses
 1.821e+09           -14.4%  1.559e+09        perf-stat.ps.dTLB-stores
    648188 ±  6%     +24.5%     806705 ±  4%  perf-stat.ps.iTLB-load-misses
   1172160 ±  2%     -22.9%     904228 ± 10%  perf-stat.ps.iTLB-loads
    173246           -20.6%     137623 ±  2%  perf-stat.ps.minor-faults
    726221 ±  4%     -19.8%     582154 ±  2%  perf-stat.ps.node-load-misses
   8173121 ±  3%     -34.7%    5333276 ±  2%  perf-stat.ps.node-loads
    445502 ±  2%     -35.5%     287358 ±  4%  perf-stat.ps.node-store-misses
  24049168           -28.2%   17274645 ±  3%  perf-stat.ps.node-stores
    173246           -20.6%     137622 ±  2%  perf-stat.ps.page-faults
     39652 ±  2%      +9.9%      43581 ±  3%  softirqs.CPU0.SCHED
    107352           +13.2%     121534        softirqs.CPU0.TIMER
     30790 ± 19%     +47.1%      45286 ±  3%  softirqs.CPU1.SCHED
    103533 ±  2%     +10.6%     114471 ±  3%  softirqs.CPU1.TIMER
     28329 ±  9%     +53.3%      43421 ±  4%  softirqs.CPU10.SCHED
    104203 ±  2%     +11.9%     116563 ±  3%  softirqs.CPU10.TIMER
     25132 ± 12%     +80.2%      45299 ±  8%  softirqs.CPU11.SCHED
    102967           +12.6%     115922 ±  4%  softirqs.CPU11.TIMER
     26183 ± 15%     +75.0%      45808 ±  5%  softirqs.CPU12.SCHED
    103373           +11.9%     115631 ±  3%  softirqs.CPU12.TIMER
     31612 ± 18%     +48.5%      46932 ±  7%  softirqs.CPU13.SCHED
    104644           +10.3%     115470 ±  2%  softirqs.CPU13.TIMER
     26288 ±  7%     +75.2%      46070 ±  5%  softirqs.CPU14.SCHED
    104066 ±  2%     +12.6%     117183        softirqs.CPU14.TIMER
     24804 ± 16%     +81.9%      45125 ±  2%  softirqs.CPU15.SCHED
    102933 ±  2%     +14.9%     118222        softirqs.CPU15.TIMER
     23752 ± 18%     +83.4%      43562 ±  5%  softirqs.CPU16.SCHED
    102635 ±  2%     +15.0%     118001        softirqs.CPU16.TIMER
     29082 ± 16%     +56.7%      45576 ±  8%  softirqs.CPU17.SCHED
    103248 ±  2%     +14.0%     117737        softirqs.CPU17.TIMER
     22195 ± 10%    +102.4%      44921 ±  6%  softirqs.CPU18.SCHED
    102277 ±  2%     +15.4%     118070        softirqs.CPU18.TIMER
     27975 ± 11%     +66.1%      46461 ±  4%  softirqs.CPU19.SCHED
     27177 ± 26%     +72.7%      46948 ±  3%  softirqs.CPU2.SCHED
    105320           +24.8%     131418 ± 16%  softirqs.CPU2.TIMER
     30379 ± 11%     +48.8%      45203 ±  3%  softirqs.CPU20.SCHED
    104150 ±  3%     +13.8%     118540        softirqs.CPU20.TIMER
     30891 ± 11%     +52.6%      47154        softirqs.CPU21.SCHED
     26164 ± 17%     +71.6%      44900 ±  6%  softirqs.CPU22.SCHED
     31366 ± 17%     +43.1%      44873 ±  7%  softirqs.CPU23.SCHED
     24394 ± 14%     +83.5%      44774 ±  7%  softirqs.CPU24.SCHED
     27442 ± 16%     +67.3%      45907 ±  3%  softirqs.CPU25.SCHED
     31616 ± 23%     +42.9%      45164        softirqs.CPU26.SCHED
     26466 ± 16%     +71.3%      45336 ±  4%  softirqs.CPU27.SCHED
     29151 ± 10%     +45.6%      42433 ±  2%  softirqs.CPU28.SCHED
     28455 ± 18%     +64.1%      46688        softirqs.CPU29.SCHED
     20502 ±  2%     +12.4%      23051 ±  5%  softirqs.CPU3.RCU
     24116 ± 18%     +91.5%      46193 ±  3%  softirqs.CPU3.SCHED
    102093 ±  2%     +17.6%     120036        softirqs.CPU3.TIMER
     27645 ± 26%     +68.6%      46601 ±  6%  softirqs.CPU30.SCHED
     25045 ± 14%     +69.7%      42498 ±  3%  softirqs.CPU31.SCHED
     28097 ±  3%     +54.0%      43277 ±  6%  softirqs.CPU32.SCHED
     22500 ± 18%    +100.2%      45043 ±  6%  softirqs.CPU33.SCHED
     24277 ± 18%     +82.5%      44309 ±  5%  softirqs.CPU34.SCHED
     30861 ± 16%     +52.7%      47113 ±  4%  softirqs.CPU35.SCHED
     25425 ± 30%     +77.4%      45117 ±  6%  softirqs.CPU36.SCHED
     27246 ± 25%     +70.5%      46449 ±  6%  softirqs.CPU37.SCHED
     24448 ± 30%     +84.4%      45094 ±  5%  softirqs.CPU38.SCHED
     25117 ± 34%     +77.0%      44447        softirqs.CPU39.SCHED
     23205 ± 16%     +98.6%      46077 ±  3%  softirqs.CPU4.SCHED
    102105           +13.4%     115741 ±  2%  softirqs.CPU4.TIMER
     28942 ± 29%     +54.1%      44603        softirqs.CPU40.SCHED
     27338 ± 15%     +65.3%      45199 ±  7%  softirqs.CPU41.SCHED
     29373 ± 18%     +52.6%      44834 ±  7%  softirqs.CPU42.SCHED
     32350 ± 14%     +34.0%      43336 ±  3%  softirqs.CPU43.SCHED
     14249 ±  4%     +18.4%      16869 ±  5%  softirqs.CPU44.RCU
     20276 ±  6%    +139.1%      48484 ±  2%  softirqs.CPU44.SCHED
    100697 ±  2%     +15.3%     116116        softirqs.CPU44.TIMER
     27640 ± 19%     +59.8%      44160 ±  4%  softirqs.CPU45.SCHED
    104558 ±  3%     +13.8%     118954 ±  3%  softirqs.CPU45.TIMER
     32781 ± 14%     +34.5%      44095 ±  3%  softirqs.CPU47.SCHED
     33772 ± 12%     +28.2%      43288 ±  4%  softirqs.CPU48.SCHED
    104880 ±  2%     +11.9%     117310 ±  2%  softirqs.CPU48.TIMER
     32556 ± 13%     +39.2%      45330 ±  4%  softirqs.CPU49.SCHED
     24734 ± 13%     +77.3%      43855 ±  3%  softirqs.CPU5.SCHED
    103789           +12.1%     116309 ±  3%  softirqs.CPU5.TIMER
     26588 ± 14%     -18.5%      21657 ±  6%  softirqs.CPU50.RCU
     34458 ±  8%     +31.5%      45315 ±  5%  softirqs.CPU50.SCHED
    105036 ±  2%     +11.9%     117517 ±  2%  softirqs.CPU50.TIMER
     34237 ± 15%     +31.4%      44981 ±  6%  softirqs.CPU51.SCHED
     32473 ± 11%     +35.0%      43834 ±  6%  softirqs.CPU52.SCHED
    104568 ±  2%     +12.4%     117503 ±  3%  softirqs.CPU52.TIMER
     30645 ± 13%     +45.6%      44617 ±  3%  softirqs.CPU53.SCHED
    104251 ±  2%     +12.5%     117286 ±  2%  softirqs.CPU53.TIMER
     29085 ±  6%     +57.5%      45808 ±  4%  softirqs.CPU54.SCHED
     31859 ±  9%     +38.7%      44198 ±  8%  softirqs.CPU55.SCHED
    104882 ±  3%     +11.5%     116985        softirqs.CPU55.TIMER
     30625 ± 13%     +42.0%      43489 ±  5%  softirqs.CPU56.SCHED
    103851 ±  2%     +12.8%     117106        softirqs.CPU56.TIMER
     25529 ± 27%     +65.2%      42183 ±  8%  softirqs.CPU57.SCHED
    102045 ±  4%     +15.8%     118201 ±  2%  softirqs.CPU57.TIMER
     31198 ±  7%     +39.2%      43421 ±  6%  softirqs.CPU58.SCHED
    103835 ±  2%     +14.3%     118680        softirqs.CPU58.TIMER
     33286 ± 13%     +32.3%      44052 ±  3%  softirqs.CPU59.SCHED
    104067 ±  2%     +12.3%     116885        softirqs.CPU59.TIMER
     23032 ±  9%     +92.7%      44375 ±  5%  softirqs.CPU6.SCHED
    102378           +13.9%     116623 ±  3%  softirqs.CPU6.TIMER
     33521 ± 14%     +36.0%      45604 ±  4%  softirqs.CPU60.SCHED
    104127 ±  2%     +11.5%     116107 ±  2%  softirqs.CPU60.TIMER
     28074 ± 16%     +53.9%      43204 ±  8%  softirqs.CPU61.SCHED
    103917 ±  2%     +12.4%     116757 ±  2%  softirqs.CPU61.TIMER
     34955 ±  5%     +25.7%      43944 ±  6%  softirqs.CPU62.SCHED
    106068 ±  2%     +12.2%     119043        softirqs.CPU62.TIMER
     29799 ±  8%     +42.8%      42559 ±  5%  softirqs.CPU63.SCHED
    103835 ±  2%     +13.6%     117936 ±  2%  softirqs.CPU63.TIMER
     27444 ± 11%     +61.0%      44190 ±  2%  softirqs.CPU64.SCHED
    103157 ±  2%     +14.7%     118349        softirqs.CPU64.TIMER
     27539 ± 12%     +53.7%      42331        softirqs.CPU65.SCHED
     33472 ± 12%     +34.2%      44915 ±  4%  softirqs.CPU66.SCHED
     26992 ± 18%     +62.9%      43958 ±  5%  softirqs.CPU67.SCHED
     33076 ±  7%     +32.6%      43849 ±  7%  softirqs.CPU68.SCHED
     30471 ± 16%     +42.1%      43288        softirqs.CPU69.SCHED
     22903 ± 26%     +95.0%      44663 ±  6%  softirqs.CPU7.SCHED
    102386 ±  3%     +13.4%     116076 ±  3%  softirqs.CPU7.TIMER
     26717 ± 26%     +65.7%      44272        softirqs.CPU70.SCHED
     31545 ± 12%     +41.8%      44717 ±  5%  softirqs.CPU71.SCHED
     29237 ± 12%     +58.6%      46373 ±  4%  softirqs.CPU72.SCHED
     29733 ± 16%     +42.1%      42255 ±  3%  softirqs.CPU73.SCHED
     29977 ± 21%     +39.6%      41860 ±  7%  softirqs.CPU74.SCHED
     32778 ±  9%     +41.4%      46341 ±  4%  softirqs.CPU75.SCHED
     29721 ±  5%     +52.3%      45255 ±  3%  softirqs.CPU76.SCHED
     35096 ±  9%     +26.2%      44287 ±  2%  softirqs.CPU77.SCHED
    103979 ±  2%     +22.4%     127298 ± 16%  softirqs.CPU77.TIMER
     33571 ± 12%     +34.9%      45293 ±  7%  softirqs.CPU78.SCHED
     27032 ± 16%     +53.8%      41571 ±  5%  softirqs.CPU79.SCHED
     24602 ± 18%     +85.0%      45505 ±  5%  softirqs.CPU8.SCHED
    102919 ±  2%     +12.7%     115994 ±  2%  softirqs.CPU8.TIMER
     32946 ± 22%     +34.7%      44368 ±  5%  softirqs.CPU80.SCHED
     31490 ± 18%     +37.8%      43391 ±  7%  softirqs.CPU81.SCHED
     33486 ± 20%     +32.6%      44402 ±  5%  softirqs.CPU82.SCHED
     32684 ± 26%     +34.6%      43989 ±  3%  softirqs.CPU83.SCHED
     29156 ± 28%     +52.4%      44431 ±  4%  softirqs.CPU84.SCHED
     30623 ± 15%     +44.0%      44085 ±  4%  softirqs.CPU85.SCHED
     30209 ± 13%     +53.2%      46285 ±  5%  softirqs.CPU86.SCHED
     26704 ± 11%     +76.3%      47091 ±  2%  softirqs.CPU87.SCHED
     26401 ± 15%     +69.0%      44606 ±  3%  softirqs.CPU9.SCHED
    103291 ±  2%     +12.2%     115944 ±  3%  softirqs.CPU9.TIMER
   2546947           +54.4%    3933302        softirqs.SCHED
    506.75 ± 33%     -54.8%     229.00 ± 34%  interrupts.39:PCI-MSI.3145736-edge.eth0-TxRx-7
   1206416          +123.1%    2691966 ±  2%  interrupts.CAL:Function_call_interrupts
     12861 ± 37%    +154.5%      32732 ± 36%  interrupts.CPU1.CAL:Function_call_interrupts
      4735 ± 35%     -39.7%       2855 ± 36%  interrupts.CPU1.NMI:Non-maskable_interrupts
      4735 ± 35%     -39.7%       2855 ± 36%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
     10297 ± 53%    +195.1%      30384 ± 40%  interrupts.CPU1.TLB:TLB_shootdowns
     10943 ± 61%    +288.9%      42564 ± 41%  interrupts.CPU13.CAL:Function_call_interrupts
    413.50 ±102%    +233.3%       1378 ± 49%  interrupts.CPU13.RES:Rescheduling_interrupts
      8158 ± 90%    +396.8%      40527 ± 45%  interrupts.CPU13.TLB:TLB_shootdowns
     17692 ± 22%     +88.8%      33398 ± 19%  interrupts.CPU15.CAL:Function_call_interrupts
      6203 ±  4%     -42.0%       3597 ± 24%  interrupts.CPU15.NMI:Non-maskable_interrupts
      6203 ±  4%     -42.0%       3597 ± 24%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
     15585 ± 28%     +99.7%      31120 ± 20%  interrupts.CPU15.TLB:TLB_shootdowns
      5696 ± 19%     -57.0%       2449 ± 19%  interrupts.CPU16.NMI:Non-maskable_interrupts
      5696 ± 19%     -57.0%       2449 ± 19%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    506.75 ± 33%     -54.8%     229.00 ± 34%  interrupts.CPU18.39:PCI-MSI.3145736-edge.eth0-TxRx-7
      6395           -49.3%       3239 ± 28%  interrupts.CPU18.NMI:Non-maskable_interrupts
      6395           -49.3%       3239 ± 28%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
     14613 ± 18%    +176.2%      40354 ± 29%  interrupts.CPU19.CAL:Function_call_interrupts
    512.00 ± 16%    +153.3%       1296 ± 32%  interrupts.CPU19.RES:Rescheduling_interrupts
     12039 ± 23%    +217.5%      38224 ± 32%  interrupts.CPU19.TLB:TLB_shootdowns
     15894 ± 44%    +137.0%      37661 ± 29%  interrupts.CPU2.CAL:Function_call_interrupts
     13480 ± 58%    +162.8%      35426 ± 32%  interrupts.CPU2.TLB:TLB_shootdowns
     12773 ± 23%    +160.6%      33284 ± 25%  interrupts.CPU20.CAL:Function_call_interrupts
    435.00 ± 26%    +119.5%     955.00 ± 16%  interrupts.CPU20.RES:Rescheduling_interrupts
     10134 ± 34%    +205.4%      30951 ± 27%  interrupts.CPU20.TLB:TLB_shootdowns
     11925 ± 30%    +273.0%      44485 ± 10%  interrupts.CPU21.CAL:Function_call_interrupts
    442.25 ± 19%    +177.0%       1225 ± 14%  interrupts.CPU21.RES:Rescheduling_interrupts
      9240 ± 39%    +360.4%      42544 ± 10%  interrupts.CPU21.TLB:TLB_shootdowns
     11296 ± 43%    +198.7%      33747 ± 49%  interrupts.CPU23.CAL:Function_call_interrupts
    371.00 ± 41%    +150.8%     930.50 ± 34%  interrupts.CPU23.RES:Rescheduling_interrupts
      8604 ± 58%    +266.0%      31493 ± 54%  interrupts.CPU23.TLB:TLB_shootdowns
     15140 ± 35%    +156.1%      38776 ± 15%  interrupts.CPU25.CAL:Function_call_interrupts
    569.50 ± 56%    +109.0%       1190 ± 17%  interrupts.CPU25.RES:Rescheduling_interrupts
     12617 ± 43%    +190.2%      36620 ± 17%  interrupts.CPU25.TLB:TLB_shootdowns
     11134 ± 62%    +201.8%      33609 ± 12%  interrupts.CPU26.CAL:Function_call_interrupts
    356.00 ± 72%    +186.0%       1018 ± 22%  interrupts.CPU26.RES:Rescheduling_interrupts
      8317 ± 85%    +276.2%      31289 ± 13%  interrupts.CPU26.TLB:TLB_shootdowns
     16482 ± 27%     +84.7%      30449 ± 37%  interrupts.CPU27.CAL:Function_call_interrupts
     14166 ± 33%     +97.9%      28027 ± 42%  interrupts.CPU27.TLB:TLB_shootdowns
      4733 ± 33%     -47.4%       2491 ± 19%  interrupts.CPU28.NMI:Non-maskable_interrupts
      4733 ± 33%     -47.4%       2491 ± 19%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
    502.00 ± 43%     +69.5%     851.00 ± 15%  interrupts.CPU28.RES:Rescheduling_interrupts
     14068 ± 34%    +214.8%      44283 ±  8%  interrupts.CPU29.CAL:Function_call_interrupts
    499.25 ± 39%    +168.8%       1341 ± 17%  interrupts.CPU29.RES:Rescheduling_interrupts
     11502 ± 44%    +267.7%      42292 ±  9%  interrupts.CPU29.TLB:TLB_shootdowns
     18770 ± 24%     +89.3%      35532 ± 21%  interrupts.CPU3.CAL:Function_call_interrupts
      4889 ± 30%     -55.3%       2186 ± 34%  interrupts.CPU3.NMI:Non-maskable_interrupts
      4889 ± 30%     -55.3%       2186 ± 34%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
     16584 ± 31%    +100.5%      33245 ± 23%  interrupts.CPU3.TLB:TLB_shootdowns
     14747 ± 43%    +195.7%      43601 ± 34%  interrupts.CPU30.CAL:Function_call_interrupts
     12277 ± 54%    +238.2%      41524 ± 37%  interrupts.CPU30.TLB:TLB_shootdowns
      4960 ± 29%     -55.4%       2213 ± 34%  interrupts.CPU31.NMI:Non-maskable_interrupts
      4960 ± 29%     -55.4%       2213 ± 34%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      5793 ± 17%     -45.8%       3139 ± 32%  interrupts.CPU32.NMI:Non-maskable_interrupts
      5793 ± 17%     -45.8%       3139 ± 32%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      6341           -55.7%       2811 ± 46%  interrupts.CPU33.NMI:Non-maskable_interrupts
      6341           -55.7%       2811 ± 46%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     11612 ± 43%    +301.1%      46572 ± 24%  interrupts.CPU35.CAL:Function_call_interrupts
    494.50 ± 36%    +177.5%       1372 ± 25%  interrupts.CPU35.RES:Rescheduling_interrupts
      8895 ± 60%    +401.5%      44613 ± 26%  interrupts.CPU35.TLB:TLB_shootdowns
     17143 ± 48%     +84.3%      31591 ± 15%  interrupts.CPU39.CAL:Function_call_interrupts
      5747 ± 18%     -46.2%       3093 ± 33%  interrupts.CPU39.NMI:Non-maskable_interrupts
      5747 ± 18%     -46.2%       3093 ± 33%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
     14848 ± 59%     +96.7%      29211 ± 17%  interrupts.CPU39.TLB:TLB_shootdowns
     19066 ± 22%     +97.1%      37572 ± 20%  interrupts.CPU4.CAL:Function_call_interrupts
     16957 ± 28%    +108.3%      35327 ± 22%  interrupts.CPU4.TLB:TLB_shootdowns
     13209 ± 63%    +138.2%      31460 ± 17%  interrupts.CPU40.CAL:Function_call_interrupts
     10995 ± 77%    +164.9%      29121 ± 19%  interrupts.CPU40.TLB:TLB_shootdowns
      5140 ± 24%     -51.3%       2504 ± 18%  interrupts.CPU43.NMI:Non-maskable_interrupts
      5140 ± 24%     -51.3%       2504 ± 18%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
     21954 ±  4%    +135.3%      51667 ± 10%  interrupts.CPU44.CAL:Function_call_interrupts
      6355           -37.7%       3962 ± 27%  interrupts.CPU44.NMI:Non-maskable_interrupts
      6355           -37.7%       3962 ± 27%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
     20036 ±  3%    +148.8%      49854 ± 10%  interrupts.CPU44.TLB:TLB_shootdowns
      4926 ± 30%     -48.5%       2535 ± 21%  interrupts.CPU45.NMI:Non-maskable_interrupts
      4926 ± 30%     -48.5%       2535 ± 21%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    529.00 ± 34%    +144.6%       1293 ± 36%  interrupts.CPU45.RES:Rescheduling_interrupts
      8879 ± 47%    +188.3%      25599 ± 28%  interrupts.CPU47.CAL:Function_call_interrupts
    279.75 ± 59%    +356.0%       1275 ± 83%  interrupts.CPU47.RES:Rescheduling_interrupts
      6031 ± 71%    +283.4%      23124 ± 32%  interrupts.CPU47.TLB:TLB_shootdowns
      8046 ± 52%    +192.6%      23545 ± 38%  interrupts.CPU48.CAL:Function_call_interrupts
      4357 ± 13%     -28.8%       3103 ± 35%  interrupts.CPU48.NMI:Non-maskable_interrupts
      4357 ± 13%     -28.8%       3103 ± 35%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    239.75 ± 71%    +159.5%     622.25 ± 32%  interrupts.CPU48.RES:Rescheduling_interrupts
      5094 ± 89%    +312.0%      20990 ± 44%  interrupts.CPU48.TLB:TLB_shootdowns
      8857 ± 44%    +296.0%      35077 ± 31%  interrupts.CPU49.CAL:Function_call_interrupts
    283.00 ± 51%    +252.7%     998.25 ± 25%  interrupts.CPU49.RES:Rescheduling_interrupts
      5940 ± 69%    +452.4%      32813 ± 34%  interrupts.CPU49.TLB:TLB_shootdowns
      4257 ± 29%     -48.9%       2175 ± 35%  interrupts.CPU5.NMI:Non-maskable_interrupts
      4257 ± 29%     -48.9%       2175 ± 35%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      7494 ± 30%    +345.9%      33416 ± 41%  interrupts.CPU50.CAL:Function_call_interrupts
      3917 ± 15%     -35.6%       2522 ± 19%  interrupts.CPU50.NMI:Non-maskable_interrupts
      3917 ± 15%     -35.6%       2522 ± 19%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      4744 ± 56%    +556.7%      31155 ± 45%  interrupts.CPU50.TLB:TLB_shootdowns
      8245 ± 88%    +267.2%      30274 ± 53%  interrupts.CPU51.CAL:Function_call_interrupts
    309.50 ±115%    +231.7%       1026 ± 43%  interrupts.CPU51.RES:Rescheduling_interrupts
      5362 ±148%    +420.5%      27911 ± 59%  interrupts.CPU51.TLB:TLB_shootdowns
     12505 ± 37%    +145.1%      30652 ± 25%  interrupts.CPU53.CAL:Function_call_interrupts
      5377 ± 19%     -55.1%       2413 ± 58%  interrupts.CPU53.NMI:Non-maskable_interrupts
      5377 ± 19%     -55.1%       2413 ± 58%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    695.75 ± 24%     +54.0%       1071 ± 21%  interrupts.CPU53.RES:Rescheduling_interrupts
      9933 ± 52%    +185.5%      28361 ± 28%  interrupts.CPU53.TLB:TLB_shootdowns
     13095 ± 22%    +184.9%      37305 ± 30%  interrupts.CPU54.CAL:Function_call_interrupts
     10596 ± 28%    +231.8%      35153 ± 33%  interrupts.CPU54.TLB:TLB_shootdowns
     10486 ± 42%    +134.1%      24551 ± 48%  interrupts.CPU56.CAL:Function_call_interrupts
    394.25 ± 50%    +104.7%     807.00 ± 47%  interrupts.CPU56.RES:Rescheduling_interrupts
      8302 ± 53%    +164.8%      21988 ± 55%  interrupts.CPU56.TLB:TLB_shootdowns
      5367 ± 31%     -66.8%       1782 ± 25%  interrupts.CPU57.NMI:Non-maskable_interrupts
      5367 ± 31%     -66.8%       1782 ± 25%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      8980 ± 46%    +209.4%      27786 ± 25%  interrupts.CPU59.CAL:Function_call_interrupts
      1973 ± 52%     -63.4%     723.00 ± 17%  interrupts.CPU59.RES:Rescheduling_interrupts
      6036 ± 71%    +320.3%      25370 ± 29%  interrupts.CPU59.TLB:TLB_shootdowns
      8012 ± 55%    +349.6%      36023 ± 33%  interrupts.CPU60.CAL:Function_call_interrupts
    272.75 ± 62%    +285.1%       1050 ± 32%  interrupts.CPU60.RES:Rescheduling_interrupts
      5226 ± 90%    +553.3%      34144 ± 35%  interrupts.CPU60.TLB:TLB_shootdowns
      5570 ± 24%     -50.2%       2772 ± 34%  interrupts.CPU64.NMI:Non-maskable_interrupts
      5570 ± 24%     -50.2%       2772 ± 34%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
      4892 ± 29%     -51.6%       2367 ± 42%  interrupts.CPU65.NMI:Non-maskable_interrupts
      4892 ± 29%     -51.6%       2367 ± 42%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
     10367 ± 42%    +200.4%      31144 ± 44%  interrupts.CPU66.CAL:Function_call_interrupts
      4480 ± 32%     -39.6%       2705 ± 32%  interrupts.CPU66.NMI:Non-maskable_interrupts
      4480 ± 32%     -39.6%       2705 ± 32%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
      7605 ± 61%    +278.7%      28798 ± 49%  interrupts.CPU66.TLB:TLB_shootdowns
      5495 ± 18%     -49.5%       2774 ± 33%  interrupts.CPU67.NMI:Non-maskable_interrupts
      5495 ± 18%     -49.5%       2774 ± 33%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      4581 ± 26%     -51.6%       2217 ± 35%  interrupts.CPU69.NMI:Non-maskable_interrupts
      4581 ± 26%     -51.6%       2217 ± 35%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
      4668 ± 25%     -43.8%       2624 ± 38%  interrupts.CPU70.NMI:Non-maskable_interrupts
      4668 ± 25%     -43.8%       2624 ± 38%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
     14254 ± 24%    +195.9%      42175 ± 23%  interrupts.CPU72.CAL:Function_call_interrupts
     11753 ± 34%    +241.8%      40168 ± 25%  interrupts.CPU72.TLB:TLB_shootdowns
      9449 ± 27%    +329.0%      40541 ± 25%  interrupts.CPU75.CAL:Function_call_interrupts
    376.00 ± 48%    +206.1%       1150 ± 18%  interrupts.CPU75.RES:Rescheduling_interrupts
      6643 ± 44%    +477.9%      38390 ± 27%  interrupts.CPU75.TLB:TLB_shootdowns
     13559 ± 19%    +158.9%      35109 ± 34%  interrupts.CPU76.CAL:Function_call_interrupts
      4630 ± 25%     -40.9%       2737 ± 32%  interrupts.CPU76.NMI:Non-maskable_interrupts
      4630 ± 25%     -40.9%       2737 ± 32%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
    432.75 ±  5%    +176.4%       1196 ± 51%  interrupts.CPU76.RES:Rescheduling_interrupts
     11045 ± 24%    +196.6%      32758 ± 37%  interrupts.CPU76.TLB:TLB_shootdowns
      6487 ± 50%    +309.7%      26577 ± 37%  interrupts.CPU77.CAL:Function_call_interrupts
    215.75 ± 89%    +279.6%     819.00 ± 62%  interrupts.CPU77.RES:Rescheduling_interrupts
      3747 ± 89%    +541.1%      24023 ± 42%  interrupts.CPU77.TLB:TLB_shootdowns
      5619 ± 24%     -56.0%       2474 ± 22%  interrupts.CPU79.NMI:Non-maskable_interrupts
      5619 ± 24%     -56.0%       2474 ± 22%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    339.25 ± 90%    +214.3%       1066 ± 42%  interrupts.CPU80.RES:Rescheduling_interrupts
     10050 ± 81%    +196.0%      29747 ± 22%  interrupts.CPU83.CAL:Function_call_interrupts
      7212 ±123%    +278.5%      27296 ± 24%  interrupts.CPU83.TLB:TLB_shootdowns
      5387 ± 18%     -53.3%       2513 ± 19%  interrupts.CPU84.NMI:Non-maskable_interrupts
      5387 ± 18%     -53.3%       2513 ± 19%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
    451.75 ± 67%     +93.4%     873.75 ± 17%  interrupts.CPU84.RES:Rescheduling_interrupts
      5362 ± 18%     -50.0%       2681 ± 33%  interrupts.CPU85.NMI:Non-maskable_interrupts
      5362 ± 18%     -50.0%       2681 ± 33%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
    383.00 ± 42%    +414.6%       1970 ± 47%  interrupts.CPU85.RES:Rescheduling_interrupts
      5037 ± 26%     -45.6%       2738 ± 20%  interrupts.CPU86.NMI:Non-maskable_interrupts
      5037 ± 26%     -45.6%       2738 ± 20%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
    494.50 ± 69%    +749.5%       4201 ±124%  interrupts.CPU86.RES:Rescheduling_interrupts
     16215 ± 21%    +156.3%      41560 ± 20%  interrupts.CPU87.CAL:Function_call_interrupts
    646.50 ± 44%     +78.8%       1156 ± 21%  interrupts.CPU87.RES:Rescheduling_interrupts
     13994 ± 28%    +181.9%      39454 ± 22%  interrupts.CPU87.TLB:TLB_shootdowns
     15408 ± 29%     +96.9%      30334 ± 22%  interrupts.CPU9.CAL:Function_call_interrupts
     13083 ± 36%    +114.1%      28011 ± 24%  interrupts.CPU9.TLB:TLB_shootdowns
    419619           -33.5%     278854 ± 12%  interrupts.NMI:Non-maskable_interrupts
    419619           -33.5%     278854 ± 12%  interrupts.PMI:Performance_monitoring_interrupts
     53973 ± 17%     +65.5%      89333 ±  7%  interrupts.RES:Rescheduling_interrupts
    985222 ±  2%    +152.1%    2483749 ±  3%  interrupts.TLB:TLB_shootdowns


                                                                                
                            will-it-scale.per_thread_ops                        
                                                                                
  250000 +-+----------------------------------------------------------------+   
         |+.++++.+++.++++.+++.++++.+++.+++.++++.+++.++++.+++.++++. + .++  .+|   
         |                                                        + +   ++  |   
  200000 +-+                                                                |   
         |  O O        OO  O  OO O  OO OOO OOOO O O                         |   
         O     O  O  O    O O                    O                          |   
  150000 +-+                                                                |   
         |                                                                  |   
  100000 +-+                                                                |   
         |                                                                  |   
         |                                                                  |   
   50000 +-+                                                                |   
         |                                                                  |   
         |                                                                  |   
       0 +O+-O---O-O--O---------O--O----------------------------------------+   
                                                                                
                                                                                                                                                                
                               will-it-scale.workload                           
                                                                                
  1e+07 +-+-----------------------------------------------------------------+   
  9e+06 +-+                                                                 |   
        |                                                                   |   
  8e+06 +-+O O        O O O  OO  O   OOO  OO OOO  O                         |   
  7e+06 O-+    O O  O    O  O      O     O       O                          |   
        |                                                                   |   
  6e+06 +-+                                                                 |   
  5e+06 +-+                                                                 |   
  4e+06 +-+                                                                 |   
        |                                                                   |   
  3e+06 +-+                                                                 |   
  2e+06 +-+                                                                 |   
        |                                                                   |   
  1e+06 +-+                                                                 |   
      0 +O+-O---O-O--O----------O-O-----------------------------------------+   
                                                                                
                                                                                                                                                                
                           will-it-scale.time.user_time                         
                                                                                
  300 +-+-------------------------------------------------------------------+   
      |                                                                     |   
  250 +-+  +.                  .+     +.+  .+          .+     .+        + .+|   
      |+.++  +++.+++.++.+++.+++  ++.++   ++  ++.+++.+++  +.+++  ++.+ +.+ +  |   
      |                                                             +       |   
  200 +-+                                                                   |   
      |                                                                     |   
  150 +-+                                                                   |   
      |                                                                     |   
  100 O-+O O O O  O  OO  O  OO  O    O    O  OO  O                          |   
      |                 O O       O O O OO  O   O                           |   
      |                                                                     |   
   50 +-+                                                                   |   
      |                                                                     |   
    0 +O+-O---O--O-O----------O--O------------------------------------------+   
                                                                                
                                                                                                                                                                
                           will-it-scale.time.system_time                       
                                                                                
  12000 +-+-----------------------------------------------------------------+   
        |+.+ +. +  .+     +.+++. +++.+ +.+++.++ .+ ++.+ +.+++. +   + +.++ .+|   
  10000 +-+ +  + ++  ++.++      +     +        +  +    +      + +.+ +    +  |   
        |                                                                   |   
        |                                                                   |   
   8000 +-+           O   O  OO       O    O  OO  O                         |   
        O  O O O O  O   OO  O    O O O O OO  O   O                          |   
   6000 +-+                                                                 |   
        |                                                                   |   
   4000 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
   2000 +-+                                                                 |   
        |                                                                   |   
      0 +O+-O---O-O--O----------O-O-----------------------------------------+   
                                                                                
                                                                                                                                                                
                  will-it-scale.time.percent_of_cpu_this_job_got                
                                                                                
  4000 +-+------------------------------------------------------------------+   
       |+.+++.+++.+++.+++.+++.+++.+++.+++.++++.+++.+++.+++.+++.+++.+++.+++.+|   
  3500 +-+                                                                  |   
  3000 +-+                                                                  |   
       |                                                                    |   
  2500 +-+  O         O   O O O       O    O O O O                          |   
       O  O   O O  O   OO  O    O  OO  OO O O   O                           |   
  2000 +-+                                                                  |   
       |                                                                    |   
  1500 +-+                                                                  |   
  1000 +-+                                                                  |   
       |                                                                    |   
   500 +-+                                                                  |   
       |                                                                    |   
     0 +O+-O---O--O-O----------O--O-----------------------------------------+   
                                                                                
                                                                                                                                                                
                        will-it-scale.time.minor_page_faults                    
                                                                                
  6e+07 +-+-----------------------------------------------------------------+   
        |                                                                   |   
  5e+07 +-++++.++++.+++.+++.+++.++++.+++.+++.+++.++++.+++.+++.+++.++++.+++.+|   
        |                                                                   |   
        |             O   O  OO       O    O  OO  O                         |   
  4e+07 O-+O O O O  O   OO  O    O O O O OO  O   O                          |   
        |                                                                   |   
  3e+07 +-+                                                                 |   
        |                                                                   |   
  2e+07 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  1e+07 +-+                                                                 |   
        |                                                                   |   
      0 +O+-O---O-O--O----------O-O-----------------------------------------+   
                                                                                
                                                                                                                                                                
                   will-it-scale.time.involuntary_context_switches              
                                                                                
  16000 +-+-----------------------------------------------------------------+   
        |                +              .+     +.                  +    +   |   
  14000 +-++++.++++.+++.+ +.+++.++++.+++  ++.++  ++++.+++.+++.+++.+ ++.+ +.+|   
  12000 +-+                                                                 |   
        |                                                                   |   
  10000 +-+           O O O  O        O    O  OO                            |   
        O  O O   O  O    O  O O  O O O O OO  O   OO                         |   
   8000 +-+    O                                                            |   
        |                                                                   |   
   6000 +-+                                                                 |   
   4000 +-+                                                                 |   
        |                                                                   |   
   2000 +-+                                                                 |   
        |                                                                   |   
      0 +O+-O---O-O--O----------O-O-----------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.1.0-rc4-00080-gadc32e8" of type "text/plain" (188821 bytes)

View attachment "job-script" of type "text/plain" (7207 bytes)

View attachment "job.yaml" of type "text/plain" (4863 bytes)

View attachment "reproduce" of type "text/plain" (314 bytes)

Powered by blists - more mailing lists