lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20191002140622.GD1741@shao2-debian>
Date:   Wed, 2 Oct 2019 22:06:22 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Mike Kravetz <mike.kravetz@...cle.com>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Naoya Horiguchi <n-horiguchi@...jp.nec.com>,
        Davidlohr Bueso <dbueso@...e.de>,
        Joonsoo Kim <iamjoonsoo.kim@....com>,
        "Kirill A . Shutemov" <kirill.shutemov@...ux.intel.com>,
        Michal Hocko <mhocko@...nel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...org
Subject: [hugetlb]  1b426bac66:  vm-scalability.throughput 2.9% improvement

Greeting,

FYI, we noticed a 2.9% improvement of vm-scalability.throughput due to commit:


commit: 1b426bac66e6cc83c9f2d92b96e4e72acf43419a ("hugetlb: use same fault hash key for shared and private mappings")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: vm-scalability
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	runtime: 300s
	size: 8T
	test: anon-cow-seq-hugetlb
	ucode: 0x5000021
	cpufreq_governor: performance

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/8T/lkp-csl-2sp6/anon-cow-seq-hugetlb/vm-scalability/0x5000021

commit: 
  0919e1b69a ("hugetlbfs: on restore reserve error path retain subpool reservation")
  1b426bac66 ("hugetlb: use same fault hash key for shared and private mappings")

0919e1b69ab459e0 1b426bac66e6cc83c9f2d92b96e 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      0.02 ± 14%    +157.1%       0.04 ± 16%  vm-scalability.median_stddev
  36623026            +2.9%   37673550        vm-scalability.throughput
     41394           +75.4%      72611        vm-scalability.time.involuntary_context_switches
   5672082           -25.8%    4210479        vm-scalability.time.minor_page_faults
      6679            +8.4%       7242        vm-scalability.time.percent_of_cpu_this_job_got
     11872           +19.5%      14183        vm-scalability.time.system_time
      8397            -7.5%       7768        vm-scalability.time.user_time
   1683225           -97.1%      48554        vm-scalability.time.voluntary_context_switches
    100516 ± 25%     +32.0%     132685 ±  4%  numa-meminfo.node1.AnonPages
   7952394 ±  5%     -13.6%    6871677 ±  6%  meminfo.DirectMap2M
      1621           -75.1%     404.50 ±173%  meminfo.Mlocked
      1489 ±  7%     -12.8%       1299        slabinfo.skbuff_ext_cache.active_objs
      1496 ±  7%     -12.3%       1311 ±  2%  slabinfo.skbuff_ext_cache.num_objs
      5.87 ±  6%      -5.9        0.02 ± 56%  mpstat.cpu.all.iowait%
      0.00 ± 24%      -0.0        0.00 ± 34%  mpstat.cpu.all.soft%
     41.03            +7.8       48.83        mpstat.cpu.all.sys%
    218.50 ± 12%     -80.5%      42.50 ±173%  numa-vmstat.node0.nr_mlock
     25135 ± 25%     +31.9%      33149 ±  4%  numa-vmstat.node1.nr_anon_pages
   1707812 ±  3%      +8.5%    1853820 ±  5%  numa-vmstat.node1.numa_hit
     28.00            -7.1%      26.00        vmstat.cpu.us
      8.25 ±  5%    -100.0%       0.00        vmstat.procs.b
     12934           -79.2%       2693        vmstat.system.cs
  50616266 ±  7%     -98.3%     855137 ± 23%  cpuidle.C1.time
    902135 ± 81%     -93.6%      57872 ±  5%  cpuidle.C1.usage
   8918453 ±141%     -98.6%     124613 ± 35%  cpuidle.POLL.time
    758814 ±124%     -94.1%      44671 ± 25%  cpuidle.POLL.usage
      1962            +7.3%       2105        turbostat.Avg_MHz
    898182 ± 81%     -94.0%      53799 ±  5%  turbostat.C1
      0.17 ±  8%      -0.2        0.00        turbostat.C1%
     28.87 ±  2%     -17.9%      23.72 ±  2%  turbostat.CPU%c1
    177.27            -0.9%     175.63        turbostat.RAMWatt
   5324754            +1.2%    5387271        proc-vmstat.nr_dirty_background_threshold
  10662757            +1.2%   10788913        proc-vmstat.nr_dirty_threshold
  53611573            +1.2%   54242516        proc-vmstat.nr_free_pages
      7277            -1.1%       7198        proc-vmstat.nr_mapped
    404.75           -75.0%     101.00 ±173%  proc-vmstat.nr_mlock
   6490796           -22.5%    5028063        proc-vmstat.pgfault
      0.63 ± 11%     -25.7%       0.47 ± 31%  sched_debug.cfs_rq:/.nr_running.avg
     12.25 ± 22%     +65.4%      20.27 ± 12%  sched_debug.cfs_rq:/.nr_spread_over.max
      2.26 ±  7%     +32.6%       3.00 ± 12%  sched_debug.cfs_rq:/.nr_spread_over.stddev
   1144616 ± 21%     +56.5%    1791801 ±  9%  sched_debug.cfs_rq:/.spread0.avg
   2350093 ±  3%     +27.6%    2999685 ±  9%  sched_debug.cfs_rq:/.spread0.max
    676.07 ±  9%     -23.6%     516.75 ± 29%  sched_debug.cfs_rq:/.util_avg.avg
    534.68 ± 12%     -32.3%     361.90 ± 30%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    200384 ±  7%     -33.8%     132625 ±  4%  sched_debug.cpu.avg_idle.stddev
      2930 ±  8%     -44.8%       1616 ± 13%  sched_debug.cpu.curr->pid.stddev
      0.62 ± 11%     -24.9%       0.47 ± 32%  sched_debug.cpu.nr_running.avg
     21212           -78.9%       4478 ±  8%  sched_debug.cpu.nr_switches.avg
     36406 ±  5%     -42.7%      20850 ±  5%  sched_debug.cpu.nr_switches.max
     13090 ± 11%     -89.6%       1357 ± 10%  sched_debug.cpu.nr_switches.min
      4386 ±  9%     -17.3%       3630 ±  8%  sched_debug.cpu.nr_switches.stddev
      0.09 ± 15%     -80.2%       0.02 ± 38%  sched_debug.cpu.nr_uninterruptible.avg
     89.92 ± 27%     -74.4%      23.00 ± 12%  sched_debug.cpu.nr_uninterruptible.max
    -54.67           -70.0%     -16.40        sched_debug.cpu.nr_uninterruptible.min
     21.72 ±  9%     -66.7%       7.24        sched_debug.cpu.nr_uninterruptible.stddev
     20648           -80.7%       3986 ± 10%  sched_debug.cpu.sched_count.avg
     36066 ±  3%     -43.7%      20296 ±  7%  sched_debug.cpu.sched_count.max
     12784 ± 12%     -91.7%       1059 ± 11%  sched_debug.cpu.sched_count.min
      4275 ±  9%     -19.2%       3456 ± 11%  sched_debug.cpu.sched_count.stddev
      9888           -88.5%       1137 ±  9%  sched_debug.cpu.sched_goidle.avg
     16456 ±  6%     -58.3%       6859 ± 17%  sched_debug.cpu.sched_goidle.max
      5961 ± 13%     -96.5%     210.67 ±  7%  sched_debug.cpu.sched_goidle.min
      2032 ± 10%     -40.0%       1220 ± 15%  sched_debug.cpu.sched_goidle.stddev
     10204           -81.6%       1876 ± 10%  sched_debug.cpu.ttwu_count.avg
     15307           -41.7%       8917 ± 22%  sched_debug.cpu.ttwu_count.max
      7640 ±  3%     -91.4%     659.22 ± 10%  sched_debug.cpu.ttwu_count.min
    480.90 ±  2%     +73.6%     834.76 ± 12%  sched_debug.cpu.ttwu_local.avg
      1684 ± 20%    +242.6%       5771 ± 11%  sched_debug.cpu.ttwu_local.max
    207.65 ± 11%    +316.5%     864.78 ± 12%  sched_debug.cpu.ttwu_local.stddev
      9.67            -8.7%       8.82        perf-stat.i.MPKI
 3.144e+10            +5.2%  3.308e+10        perf-stat.i.branch-instructions
 7.849e+08            -2.4%  7.664e+08        perf-stat.i.cache-misses
 9.538e+08            -2.0%  9.347e+08        perf-stat.i.cache-references
     12975           -79.5%       2662        perf-stat.i.context-switches
  1.85e+11            +7.5%  1.988e+11        perf-stat.i.cpu-cycles
    681.59 ±  2%     -82.6%     118.52 ±  2%  perf-stat.i.cpu-migrations
    230.52           +11.8%     257.66        perf-stat.i.cycles-between-cache-misses
  2.14e+10           +10.3%   2.36e+10        perf-stat.i.dTLB-loads
     76283 ±  5%     -13.5%      65952 ±  4%  perf-stat.i.dTLB-store-misses
     65.58 ±  3%      +6.3       71.91        perf-stat.i.iTLB-load-miss-rate%
   1158687 ±  2%      -7.8%    1068342        perf-stat.i.iTLB-load-misses
 9.888e+10            +7.5%  1.063e+11        perf-stat.i.instructions
     91782 ±  3%     +18.6%     108861        perf-stat.i.instructions-per-iTLB-miss
      0.56            -1.9%       0.55        perf-stat.i.ipc
     21254           -22.8%      16418        perf-stat.i.minor-faults
     54.29 ±  3%      -3.9       50.37 ±  3%  perf-stat.i.node-load-miss-rate%
 1.203e+08 ±  3%     -13.0%  1.048e+08 ±  3%  perf-stat.i.node-load-misses
  79399232 ±  2%      -9.5%   71891325 ±  2%  perf-stat.i.node-stores
     21255           -22.8%      16418        perf-stat.i.page-faults
      9.65            -8.8%       8.79        perf-stat.overall.MPKI
      0.01 ±  2%      -0.0        0.01 ±  8%  perf-stat.overall.branch-miss-rate%
    235.72           +10.1%     259.42        perf-stat.overall.cycles-between-cache-misses
      0.00 ±  5%      -0.0        0.00 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     85420 ±  2%     +16.5%      99545        perf-stat.overall.instructions-per-iTLB-miss
     58.23 ±  3%      -4.0       54.18 ±  2%  perf-stat.overall.node-load-miss-rate%
      3239            +9.3%       3539        perf-stat.overall.path-length
 3.134e+10            +5.2%  3.298e+10        perf-stat.ps.branch-instructions
 7.824e+08            -2.3%  7.641e+08        perf-stat.ps.cache-misses
 9.507e+08            -2.0%  9.319e+08        perf-stat.ps.cache-references
     12921           -79.5%       2652        perf-stat.ps.context-switches
 1.844e+11            +7.5%  1.982e+11        perf-stat.ps.cpu-cycles
    677.93 ±  2%     -82.6%     117.91 ±  2%  perf-stat.ps.cpu-migrations
 2.133e+10           +10.3%  2.353e+10        perf-stat.ps.dTLB-loads
     76017 ±  5%     -13.5%      65742 ±  4%  perf-stat.ps.dTLB-store-misses
   1154758 ±  2%      -7.8%    1064646        perf-stat.ps.iTLB-load-misses
 9.856e+10            +7.5%   1.06e+11        perf-stat.ps.instructions
     21179           -22.7%      16365        perf-stat.ps.minor-faults
 1.199e+08 ±  3%     -12.9%  1.045e+08 ±  3%  perf-stat.ps.node-load-misses
  79135385 ±  2%      -9.4%   71681660 ±  2%  perf-stat.ps.node-stores
     21179           -22.7%      16366        perf-stat.ps.page-faults
 2.949e+13            +8.0%  3.184e+13        perf-stat.total.instructions
     55.60 ±  4%      -7.4       48.17 ±  6%  perf-profile.calltrace.cycles-pp.hugetlb_cow.hugetlb_fault.handle_mm_fault.__do_page_fault.do_page_fault
     54.13 ±  4%      -7.3       46.83 ±  6%  perf-profile.calltrace.cycles-pp.copy_subpage.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault
     54.27 ±  4%      -7.3       46.97 ±  6%  perf-profile.calltrace.cycles-pp.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault.__do_page_fault
     53.46 ±  5%      -7.2       46.22 ±  6%  perf-profile.calltrace.cycles-pp.copy_page.copy_subpage.copy_user_huge_page.hugetlb_cow.hugetlb_fault
     10.08 ± 10%      -5.9        4.13 ± 13%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     10.08 ± 10%      -5.9        4.13 ± 13%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     10.07 ± 10%      -5.9        4.13 ± 13%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     10.24 ± 10%      -5.9        4.31 ± 13%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      9.79 ± 10%      -5.8        4.03 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      9.77 ± 10%      -5.8        4.02 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      9.38 ± 12%      -5.6        3.81 ± 13%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +6.0        6.03 ± 16%  perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.hugetlb_fault.handle_mm_fault.__do_page_fault
     57.63 ±  4%      +7.2       64.84 ±  5%  perf-profile.calltrace.cycles-pp.hugetlb_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      1.54 ±  7%      +8.8       10.38 ±  7%  perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.hugetlb_fault.handle_mm_fault.__do_page_fault
     41.05 ±  9%     +11.1       52.15 ± 13%  perf-profile.calltrace.cycles-pp.page_fault.do_access
     41.04 ±  9%     +11.1       52.14 ± 13%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.do_access
     41.02 ±  9%     +11.1       52.12 ± 13%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault.do_access
     40.94 ±  9%     +11.1       52.06 ± 13%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.do_access
      1.57 ±  7%     +15.0       16.58 ±  9%  perf-profile.calltrace.cycles-pp.__mutex_lock.hugetlb_fault.handle_mm_fault.__do_page_fault.do_page_fault
     55.60 ±  4%      -7.4       48.17 ±  6%  perf-profile.children.cycles-pp.hugetlb_cow
     54.29 ±  4%      -7.3       46.99 ±  6%  perf-profile.children.cycles-pp.copy_user_huge_page
     54.13 ±  4%      -7.3       46.84 ±  6%  perf-profile.children.cycles-pp.copy_subpage
     53.94 ±  4%      -7.3       46.67 ±  6%  perf-profile.children.cycles-pp.copy_page
     10.08 ± 10%      -5.9        4.13 ± 13%  perf-profile.children.cycles-pp.start_secondary
     10.24 ± 10%      -5.9        4.31 ± 13%  perf-profile.children.cycles-pp.secondary_startup_64
     10.24 ± 10%      -5.9        4.31 ± 13%  perf-profile.children.cycles-pp.cpu_startup_entry
     10.24 ± 10%      -5.9        4.31 ± 13%  perf-profile.children.cycles-pp.do_idle
      9.96 ± 10%      -5.8        4.21 ± 13%  perf-profile.children.cycles-pp.cpuidle_enter
      9.96 ± 10%      -5.8        4.21 ± 13%  perf-profile.children.cycles-pp.cpuidle_enter_state
      9.54 ± 11%      -5.5        3.99 ± 13%  perf-profile.children.cycles-pp.intel_idle
      2.21 ±  3%      -0.3        1.90 ±  4%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.66 ±  7%      -0.3        1.37 ±  5%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.38 ±  6%      -0.2        1.19 ±  6%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.08 ±  6%      -0.2        0.91 ±  8%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.79 ±  8%      -0.1        0.64 ± 12%  perf-profile.children.cycles-pp.tick_sched_handle
      0.78 ±  7%      -0.1        0.63 ± 13%  perf-profile.children.cycles-pp.update_process_times
      0.54 ±  9%      -0.1        0.44 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.79 ±  4%      -0.1        0.69 ±  7%  perf-profile.children.cycles-pp.alloc_fresh_huge_page
      0.71 ±  5%      -0.1        0.62 ±  7%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.69 ±  5%      -0.1        0.60 ±  7%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.43 ±  8%      -0.1        0.34 ±  8%  perf-profile.children.cycles-pp.task_tick_fair
      0.23 ±  7%      -0.1        0.14 ±  5%  perf-profile.children.cycles-pp.irq_exit
      0.18 ± 10%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.14 ± 17%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.menu_select
      0.08 ± 20%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.09 ± 17%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.15 ±  5%      -0.0        0.12 ± 11%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.12 ±  8%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.update_curr
      0.08 ±  5%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.__get_user_pages
      0.08 ±  5%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.__mm_populate
      0.08 ±  5%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.populate_vma_page_range
      0.08 ±  5%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.follow_hugetlb_page
      0.08 ±  5%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.hugetlb_no_page
      0.08 ±  5%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.clear_huge_page
      0.08 ±  5%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.clear_subpage
      0.00            +6.1        6.07 ± 15%  perf-profile.children.cycles-pp.osq_lock
     57.87 ±  4%      +7.2       65.04 ±  5%  perf-profile.children.cycles-pp.page_fault
     57.84 ±  4%      +7.2       65.01 ±  5%  perf-profile.children.cycles-pp.do_page_fault
     57.81 ±  4%      +7.2       64.99 ±  5%  perf-profile.children.cycles-pp.__do_page_fault
     57.72 ±  4%      +7.2       64.91 ±  5%  perf-profile.children.cycles-pp.hugetlb_fault
     57.71 ±  4%      +7.2       64.90 ±  5%  perf-profile.children.cycles-pp.handle_mm_fault
      1.57 ±  4%      +8.9       10.49 ±  6%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      1.60 ±  4%     +15.0       16.58 ±  9%  perf-profile.children.cycles-pp.__mutex_lock
     53.19 ±  4%      -7.1       46.06 ±  6%  perf-profile.self.cycles-pp.copy_page
      9.53 ± 11%      -5.5        3.99 ± 13%  perf-profile.self.cycles-pp.intel_idle
      0.48 ±  5%      -0.1        0.42 ±  6%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.08 ±  5%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock
      0.19 ±  4%      -0.0        0.17 ±  4%  perf-profile.self.cycles-pp.copy_subpage
      0.00            +6.0        6.02 ± 15%  perf-profile.self.cycles-pp.osq_lock
      1.54 ±  4%      +8.8       10.39 ±  6%  perf-profile.self.cycles-pp.mutex_spin_on_owner
     29606 ±  6%     -15.1%      25136 ± 11%  softirqs.CPU0.RCU
     26386 ±  9%     -22.2%      20532 ±  3%  softirqs.CPU0.SCHED
     29555 ±  6%     -20.8%      23414 ±  7%  softirqs.CPU1.RCU
     21505 ± 21%     -35.6%      13859 ±  5%  softirqs.CPU1.SCHED
     29293 ±  5%     -21.2%      23069 ±  6%  softirqs.CPU10.RCU
     20892 ± 21%     -60.6%       8227 ± 40%  softirqs.CPU10.SCHED
     29739 ±  5%     -20.4%      23669 ±  5%  softirqs.CPU11.RCU
     20498 ± 24%     -59.9%       8223 ± 43%  softirqs.CPU11.SCHED
     29435 ±  5%     -20.0%      23535 ±  5%  softirqs.CPU12.RCU
     20502 ± 23%     -59.8%       8235 ± 47%  softirqs.CPU12.SCHED
     29555 ±  6%     -18.4%      24109 ±  5%  softirqs.CPU13.RCU
     20746 ± 25%     -60.4%       8212 ± 48%  softirqs.CPU13.SCHED
     29145 ±  6%     -20.6%      23143 ±  6%  softirqs.CPU14.RCU
     20450 ± 20%     -62.4%       7682 ± 49%  softirqs.CPU14.SCHED
     29122 ±  5%     -21.0%      23005 ±  6%  softirqs.CPU15.RCU
     20223 ± 22%     -63.1%       7469 ± 51%  softirqs.CPU15.SCHED
     30773 ± 10%     -20.2%      24567 ± 12%  softirqs.CPU16.RCU
     19804 ± 21%     -60.8%       7757 ± 51%  softirqs.CPU16.SCHED
     30782 ± 11%     -24.4%      23277 ±  6%  softirqs.CPU17.RCU
     19776 ± 24%     -60.8%       7760 ± 49%  softirqs.CPU17.SCHED
     30397 ± 12%     -22.3%      23611 ±  8%  softirqs.CPU18.RCU
     18953 ± 29%     -57.1%       8133 ± 49%  softirqs.CPU18.SCHED
     30834 ± 10%     -22.5%      23893 ±  9%  softirqs.CPU19.RCU
     19644 ± 25%     -60.4%       7774 ± 48%  softirqs.CPU19.SCHED
     30788 ±  7%     -24.8%      23146 ±  6%  softirqs.CPU2.RCU
     22262 ± 18%     -46.8%      11844 ± 13%  softirqs.CPU2.SCHED
     30851 ± 10%     -20.3%      24581 ± 11%  softirqs.CPU20.RCU
     20365 ± 22%     -61.8%       7787 ± 57%  softirqs.CPU20.SCHED
     30932 ± 11%     -24.8%      23248 ±  7%  softirqs.CPU21.RCU
     20354 ± 23%     -62.1%       7724 ± 55%  softirqs.CPU21.SCHED
     30953 ± 11%     -21.6%      24273 ±  5%  softirqs.CPU22.RCU
     19148 ± 28%     -59.0%       7850 ± 52%  softirqs.CPU22.SCHED
     30985 ± 12%     -24.8%      23299 ±  7%  softirqs.CPU23.RCU
     19553 ± 26%     -58.6%       8092 ± 51%  softirqs.CPU23.SCHED
     32562 ±  9%     -24.4%      24621 ± 12%  softirqs.CPU24.RCU
     21048 ± 15%     -38.5%      12952 ±  9%  softirqs.CPU24.SCHED
     29498 ±  8%     -18.4%      24072 ±  7%  softirqs.CPU25.RCU
     21074 ± 19%     -36.9%      13307 ±  2%  softirqs.CPU25.SCHED
     31280 ± 13%     -23.7%      23861 ±  7%  softirqs.CPU26.RCU
     20952 ± 23%     -31.2%      14418 ± 12%  softirqs.CPU26.SCHED
     30054 ±  8%     -19.2%      24271 ±  7%  softirqs.CPU27.RCU
     21111 ± 20%     -31.4%      14492 ± 17%  softirqs.CPU27.SCHED
     29907 ±  9%     -19.8%      23985 ±  8%  softirqs.CPU28.RCU
     29564 ±  6%     -21.9%      23100 ±  5%  softirqs.CPU3.RCU
     22698 ± 17%     -55.5%      10096 ± 22%  softirqs.CPU3.SCHED
     29786 ±  9%     -19.5%      23991 ±  7%  softirqs.CPU30.RCU
     29689 ±  9%     -19.8%      23811 ±  6%  softirqs.CPU31.RCU
     29464 ±  7%     -19.1%      23849 ± 10%  softirqs.CPU32.RCU
     29526 ±  7%     -19.2%      23857 ± 11%  softirqs.CPU33.RCU
     21153 ± 23%     -33.2%      14128 ± 24%  softirqs.CPU33.SCHED
     30009 ±  7%     -19.7%      24109 ± 11%  softirqs.CPU34.RCU
     20989 ± 25%     -37.1%      13198 ± 27%  softirqs.CPU34.SCHED
     29737 ±  7%     -18.6%      24219 ± 11%  softirqs.CPU35.RCU
     20484 ± 20%     -33.4%      13637 ± 29%  softirqs.CPU35.SCHED
     29558 ±  7%     -19.2%      23893 ±  9%  softirqs.CPU36.RCU
     29495 ±  7%     -20.2%      23528 ±  9%  softirqs.CPU37.RCU
     29520 ±  7%     -18.4%      24091 ± 10%  softirqs.CPU38.RCU
     20410 ± 21%     -35.0%      13267 ± 31%  softirqs.CPU39.SCHED
     29648 ±  5%     -22.4%      23004 ±  5%  softirqs.CPU4.RCU
     21235 ± 24%     -55.2%       9503 ± 31%  softirqs.CPU4.SCHED
     29631 ±  7%     -18.7%      24086 ± 11%  softirqs.CPU40.RCU
     29559 ±  7%     -18.1%      24222 ± 11%  softirqs.CPU41.RCU
     30027 ±  6%     -19.4%      24191 ± 11%  softirqs.CPU42.RCU
     29304 ±  7%     -16.9%      24346 ± 11%  softirqs.CPU43.RCU
     29799 ±  7%     -19.3%      24035 ± 10%  softirqs.CPU44.RCU
     29550 ±  7%     -18.3%      24136 ± 10%  softirqs.CPU45.RCU
     29543 ±  7%     -20.3%      23548 ± 12%  softirqs.CPU46.RCU
     29392 ±  7%     -17.7%      24189 ± 12%  softirqs.CPU47.RCU
     30399 ±  9%     -24.8%      22863 ±  6%  softirqs.CPU48.RCU
     22651 ± 10%     -66.8%       7527 ± 32%  softirqs.CPU48.SCHED
     31188 ± 10%     -23.8%      23769 ±  6%  softirqs.CPU49.RCU
     21037 ± 20%     -60.4%       8322 ± 44%  softirqs.CPU49.SCHED
     29487 ±  6%     -19.3%      23809 ±  5%  softirqs.CPU5.RCU
     20755 ± 26%     -57.6%       8810 ± 35%  softirqs.CPU5.SCHED
     29850 ±  6%     -23.3%      22901 ±  4%  softirqs.CPU50.RCU
     21421 ± 20%     -62.1%       8127 ± 44%  softirqs.CPU50.SCHED
     31761 ±  9%     -26.5%      23331 ±  4%  softirqs.CPU51.RCU
     21198 ± 20%     -62.3%       7987 ± 44%  softirqs.CPU51.SCHED
     29990 ±  7%     -23.6%      22924 ±  5%  softirqs.CPU52.RCU
     21043 ± 23%     -63.9%       7588 ± 47%  softirqs.CPU52.SCHED
     29677 ±  6%     -23.3%      22767 ±  5%  softirqs.CPU53.RCU
     20254 ± 24%     -61.8%       7736 ± 58%  softirqs.CPU53.SCHED
     31327 ±  9%     -26.1%      23157 ±  4%  softirqs.CPU54.RCU
     21084 ± 26%     -64.5%       7489 ± 51%  softirqs.CPU54.SCHED
     30387 ±  9%     -25.0%      22804 ±  4%  softirqs.CPU55.RCU
     21411 ± 19%     -64.1%       7681 ± 47%  softirqs.CPU55.SCHED
     29628 ±  5%     -22.5%      22967 ±  4%  softirqs.CPU56.RCU
     20868 ± 20%     -63.4%       7648 ± 47%  softirqs.CPU56.SCHED
     29066 ±  2%     -21.1%      22938 ±  5%  softirqs.CPU57.RCU
     20372 ± 22%     -63.6%       7418 ± 51%  softirqs.CPU57.SCHED
     29743 ±  5%     -23.2%      22832 ±  5%  softirqs.CPU58.RCU
     21315 ± 21%     -64.4%       7595 ± 49%  softirqs.CPU58.SCHED
     29846 ±  6%     -25.7%      22184 ±  3%  softirqs.CPU59.RCU
     20255 ± 25%     -61.5%       7802 ± 55%  softirqs.CPU59.SCHED
     30040 ±  9%     -22.3%      23355 ±  5%  softirqs.CPU6.RCU
     20823 ± 27%     -58.2%       8708 ± 39%  softirqs.CPU6.SCHED
     29946 ±  6%     -23.3%      22957 ±  4%  softirqs.CPU60.RCU
     20308 ± 25%     -62.6%       7596 ± 50%  softirqs.CPU60.SCHED
     29605 ±  6%     -22.4%      22973 ±  5%  softirqs.CPU61.RCU
     20328 ± 25%     -64.6%       7189 ± 49%  softirqs.CPU61.SCHED
     29627 ±  6%     -22.4%      23000 ±  5%  softirqs.CPU62.RCU
     21086 ± 23%     -65.7%       7232 ± 51%  softirqs.CPU62.SCHED
     29830 ±  6%     -23.2%      22900 ±  5%  softirqs.CPU63.RCU
     20735 ± 23%     -64.6%       7330 ± 52%  softirqs.CPU63.SCHED
     31074 ± 10%     -25.7%      23081 ±  5%  softirqs.CPU64.RCU
     20089 ± 22%     -63.7%       7297 ± 49%  softirqs.CPU64.SCHED
     30783 ± 12%     -25.1%      23064 ±  6%  softirqs.CPU65.RCU
     19830 ± 26%     -62.5%       7429 ± 52%  softirqs.CPU65.SCHED
     30031 ± 13%     -23.0%      23118 ±  6%  softirqs.CPU66.RCU
     19053 ± 31%     -61.0%       7438 ± 48%  softirqs.CPU66.SCHED
     31057 ± 12%     -24.8%      23361 ±  6%  softirqs.CPU67.RCU
     19744 ± 26%     -60.9%       7719 ± 49%  softirqs.CPU67.SCHED
     30750 ± 11%     -25.8%      22804 ±  5%  softirqs.CPU68.RCU
     20174 ± 24%     -64.2%       7230 ± 52%  softirqs.CPU68.SCHED
     30754 ± 11%     -25.2%      23012 ±  6%  softirqs.CPU69.RCU
     20357 ± 24%     -63.7%       7386 ± 49%  softirqs.CPU69.SCHED
     29870 ±  8%     -23.3%      22902 ±  5%  softirqs.CPU7.RCU
     21514 ± 20%     -56.9%       9280 ± 29%  softirqs.CPU7.SCHED
     31135 ± 13%     -26.0%      23038 ±  6%  softirqs.CPU70.RCU
     18844 ± 27%     -60.4%       7465 ± 45%  softirqs.CPU70.SCHED
     30760 ± 11%     -24.9%      23086 ±  6%  softirqs.CPU71.RCU
     19068 ± 26%     -59.7%       7682 ± 51%  softirqs.CPU71.SCHED
     29320 ± 11%     -19.6%      23567 ±  8%  softirqs.CPU72.RCU
     29847 ± 10%     -20.4%      23769 ±  8%  softirqs.CPU73.RCU
     21694 ± 20%     -36.9%      13684 ± 31%  softirqs.CPU73.SCHED
     30143 ± 11%     -21.2%      23749 ±  6%  softirqs.CPU74.RCU
     30010 ±  9%     -21.1%      23671 ±  8%  softirqs.CPU75.RCU
     21039 ± 22%     -35.9%      13491 ± 30%  softirqs.CPU75.SCHED
     30023 ± 10%     -21.1%      23676 ±  8%  softirqs.CPU76.RCU
     29837 ± 10%     -20.3%      23771 ±  9%  softirqs.CPU77.RCU
     30191 ± 10%     -21.5%      23698 ±  8%  softirqs.CPU78.RCU
     29981 ± 10%     -22.0%      23385 ±  7%  softirqs.CPU79.RCU
     29448 ±  4%     -21.8%      23028 ±  5%  softirqs.CPU8.RCU
     21576 ± 18%     -58.6%       8926 ± 46%  softirqs.CPU8.SCHED
     29172 ±  7%     -18.4%      23791 ± 11%  softirqs.CPU80.RCU
     29528 ±  7%     -18.4%      24089 ± 12%  softirqs.CPU81.RCU
     21298 ± 23%     -39.9%      12809 ± 34%  softirqs.CPU81.SCHED
     29519 ±  7%     -19.0%      23908 ± 11%  softirqs.CPU82.RCU
     29600 ±  7%     -18.5%      24111 ± 11%  softirqs.CPU83.RCU
     29075 ±  7%     -19.0%      23561 ± 10%  softirqs.CPU84.RCU
     29134 ±  6%     -19.6%      23421 ±  9%  softirqs.CPU85.RCU
     20802 ± 22%     -35.4%      13437 ± 29%  softirqs.CPU85.SCHED
     29327 ±  7%     -19.3%      23681 ± 11%  softirqs.CPU86.RCU
     20885 ± 23%     -37.1%      13129 ± 33%  softirqs.CPU86.SCHED
     29899 ±  7%     -20.5%      23759 ± 11%  softirqs.CPU87.RCU
     20679 ± 20%     -37.3%      12971 ± 33%  softirqs.CPU87.SCHED
     29511 ±  7%     -18.2%      24138 ± 11%  softirqs.CPU88.RCU
     29536 ±  7%     -18.3%      24136 ± 12%  softirqs.CPU89.RCU
     28743 ±  2%     -20.1%      22969 ±  5%  softirqs.CPU9.RCU
     20443 ± 22%     -58.5%       8483 ± 40%  softirqs.CPU9.SCHED
     29321 ±  7%     -20.5%      23311 ± 10%  softirqs.CPU90.RCU
     29204 ±  7%     -18.4%      23820 ± 10%  softirqs.CPU91.RCU
     20839 ± 24%     -38.2%      12869 ± 30%  softirqs.CPU91.SCHED
     29393 ±  7%     -19.1%      23772 ± 10%  softirqs.CPU92.RCU
     20404 ± 24%     -37.7%      12714 ± 32%  softirqs.CPU93.SCHED
     29875 ±  7%     -20.5%      23740 ± 11%  softirqs.CPU94.RCU
     29387 ±  7%     -17.9%      24137 ± 11%  softirqs.CPU95.RCU
     20129 ± 22%     -38.4%      12403 ± 32%  softirqs.CPU95.SCHED
   2875061 ±  7%     -21.1%    2267715 ±  6%  softirqs.RCU
   1982937           -47.1%    1049647        softirqs.SCHED
     61.25 ±115%     -99.6%       0.25 ±173%  interrupts.103:PCI-MSI.31981636-edge.i40e-eth0-TxRx-67
     29.25 ±157%     -98.3%       0.50 ±173%  interrupts.40:PCI-MSI.31981573-edge.i40e-eth0-TxRx-4
     50.00 ±152%    -100.0%       0.00        interrupts.42:PCI-MSI.31981575-edge.i40e-eth0-TxRx-6
     45.75 ±156%    -100.0%       0.00        interrupts.88:PCI-MSI.31981621-edge.i40e-eth0-TxRx-52
      3846 ±  9%     -15.2%       3263 ± 10%  interrupts.CPU0.CAL:Function_call_interrupts
      3012 ± 12%     -26.0%       2229 ± 23%  interrupts.CPU0.RES:Rescheduling_interrupts
      2473 ± 92%     -90.3%     240.00 ±134%  interrupts.CPU0.TLB:TLB_shootdowns
      2029 ± 37%     -40.4%       1208 ± 62%  interrupts.CPU1.RES:Rescheduling_interrupts
      1756 ± 75%     -88.3%     204.75 ±155%  interrupts.CPU1.TLB:TLB_shootdowns
      3873 ± 10%      -6.3%       3630 ±  9%  interrupts.CPU10.CAL:Function_call_interrupts
      1463 ± 22%     -65.3%     508.25 ± 44%  interrupts.CPU10.RES:Rescheduling_interrupts
      1365 ± 65%     -92.3%     104.75 ±152%  interrupts.CPU10.TLB:TLB_shootdowns
      1246 ± 15%     -76.6%     291.00 ± 14%  interrupts.CPU11.RES:Rescheduling_interrupts
      1345 ± 71%     -97.7%      31.50 ±100%  interrupts.CPU11.TLB:TLB_shootdowns
      1455 ± 22%     -64.9%     511.25 ± 55%  interrupts.CPU12.RES:Rescheduling_interrupts
      1385 ± 65%     -97.2%      38.25 ±146%  interrupts.CPU12.TLB:TLB_shootdowns
      1747 ± 64%     -79.2%     364.25 ± 11%  interrupts.CPU13.RES:Rescheduling_interrupts
      1102 ± 67%     -96.9%      34.25 ±121%  interrupts.CPU13.TLB:TLB_shootdowns
      1304 ± 22%     -74.5%     333.25 ± 19%  interrupts.CPU14.RES:Rescheduling_interrupts
      1316 ± 79%     -97.2%      36.25 ±149%  interrupts.CPU14.TLB:TLB_shootdowns
      1220 ±  9%     -75.8%     296.00 ± 15%  interrupts.CPU15.RES:Rescheduling_interrupts
    915.75 ± 62%     -98.4%      14.25 ± 80%  interrupts.CPU15.TLB:TLB_shootdowns
      1084 ± 12%     -72.8%     295.00 ± 24%  interrupts.CPU16.RES:Rescheduling_interrupts
      1014 ± 69%     -92.2%      78.75 ±153%  interrupts.CPU16.TLB:TLB_shootdowns
      1196 ± 18%     -80.8%     229.75 ± 14%  interrupts.CPU17.RES:Rescheduling_interrupts
      1087 ± 78%     -91.2%      96.00 ±155%  interrupts.CPU17.TLB:TLB_shootdowns
      5015 ± 23%     +33.3%       6686 ±  9%  interrupts.CPU18.NMI:Non-maskable_interrupts
      5015 ± 23%     +33.3%       6686 ±  9%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
    921.25 ± 12%     -69.9%     277.25 ± 16%  interrupts.CPU18.RES:Rescheduling_interrupts
      1340 ± 91%     -97.2%      37.75 ±150%  interrupts.CPU18.TLB:TLB_shootdowns
      1057 ± 19%     -53.9%     487.00 ± 64%  interrupts.CPU19.RES:Rescheduling_interrupts
      1279 ± 82%     -96.4%      45.75 ±129%  interrupts.CPU19.TLB:TLB_shootdowns
      1384 ± 10%     -68.1%     442.00 ± 32%  interrupts.CPU2.RES:Rescheduling_interrupts
      1259 ± 94%     -88.6%     143.00 ±146%  interrupts.CPU2.TLB:TLB_shootdowns
      4271 ± 35%     +54.4%       6595 ±  9%  interrupts.CPU20.NMI:Non-maskable_interrupts
      4271 ± 35%     +54.4%       6595 ±  9%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      1097 ±  9%     -76.9%     253.25 ± 28%  interrupts.CPU20.RES:Rescheduling_interrupts
      1148 ± 71%     -99.3%       7.50 ± 70%  interrupts.CPU20.TLB:TLB_shootdowns
      1108 ± 24%     -73.3%     295.75 ± 21%  interrupts.CPU21.RES:Rescheduling_interrupts
      1183 ± 69%     -90.9%     107.25 ±160%  interrupts.CPU21.TLB:TLB_shootdowns
      1020 ±  5%     -74.1%     264.75 ± 19%  interrupts.CPU22.RES:Rescheduling_interrupts
      1011 ± 81%     -95.0%      50.50 ±152%  interrupts.CPU22.TLB:TLB_shootdowns
      4265 ± 23%     +52.3%       6495 ± 13%  interrupts.CPU23.NMI:Non-maskable_interrupts
      4265 ± 23%     +52.3%       6495 ± 13%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      1262 ± 34%     -80.4%     247.50 ± 17%  interrupts.CPU23.RES:Rescheduling_interrupts
      1019 ± 62%     -94.8%      53.50 ±146%  interrupts.CPU23.TLB:TLB_shootdowns
      1196 ± 14%     -69.4%     365.50 ± 91%  interrupts.CPU24.RES:Rescheduling_interrupts
      5626 ± 86%     -60.0%       2251 ±171%  interrupts.CPU24.TLB:TLB_shootdowns
      1113 ± 17%     -55.9%     491.50 ± 75%  interrupts.CPU25.RES:Rescheduling_interrupts
      1166 ± 19%     -75.0%     291.50 ± 52%  interrupts.CPU26.RES:Rescheduling_interrupts
      1029 ±  9%     -76.9%     237.75 ± 28%  interrupts.CPU27.RES:Rescheduling_interrupts
      1248 ± 30%     -76.9%     288.50 ± 42%  interrupts.CPU29.RES:Rescheduling_interrupts
      1503 ± 58%     -64.2%     538.75 ±170%  interrupts.CPU29.TLB:TLB_shootdowns
      1960 ± 51%     -83.3%     326.75 ± 19%  interrupts.CPU3.RES:Rescheduling_interrupts
      1396 ± 63%     -89.5%     146.50 ±153%  interrupts.CPU3.TLB:TLB_shootdowns
      1088 ± 16%     -74.2%     281.25 ± 34%  interrupts.CPU30.RES:Rescheduling_interrupts
    952.25 ± 11%     -55.4%     425.00 ± 55%  interrupts.CPU31.RES:Rescheduling_interrupts
      1052 ± 19%     -73.7%     276.50 ± 41%  interrupts.CPU32.RES:Rescheduling_interrupts
      5870 ± 13%     -34.8%       3825 ± 28%  interrupts.CPU33.NMI:Non-maskable_interrupts
      5870 ± 13%     -34.8%       3825 ± 28%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
      1271 ± 36%     -77.2%     290.25 ± 38%  interrupts.CPU33.RES:Rescheduling_interrupts
      1551 ± 71%     -79.9%     311.25 ±170%  interrupts.CPU33.TLB:TLB_shootdowns
      1015 ±  7%     -76.8%     235.75 ± 31%  interrupts.CPU34.RES:Rescheduling_interrupts
      1004 ±  7%     -67.2%     329.75 ± 26%  interrupts.CPU35.RES:Rescheduling_interrupts
    987.50 ± 14%     -72.7%     269.75 ± 22%  interrupts.CPU36.RES:Rescheduling_interrupts
      1681 ± 70%     -71.8%     473.50 ±169%  interrupts.CPU36.TLB:TLB_shootdowns
      1042 ±  9%     -73.1%     280.50 ± 12%  interrupts.CPU37.RES:Rescheduling_interrupts
      2108 ± 66%     -76.0%     506.00 ±169%  interrupts.CPU37.TLB:TLB_shootdowns
      1093 ±  6%     -84.2%     173.25 ± 19%  interrupts.CPU38.RES:Rescheduling_interrupts
      1454 ± 64%     -71.4%     415.50 ±172%  interrupts.CPU38.TLB:TLB_shootdowns
      1054 ± 19%     -77.7%     234.75 ± 16%  interrupts.CPU39.RES:Rescheduling_interrupts
      1500 ± 62%     -69.1%     464.25 ±169%  interrupts.CPU39.TLB:TLB_shootdowns
     28.50 ±163%     -99.1%       0.25 ±173%  interrupts.CPU4.40:PCI-MSI.31981573-edge.i40e-eth0-TxRx-4
      3497 ± 26%     +77.3%       6200 ±  6%  interrupts.CPU4.NMI:Non-maskable_interrupts
      3497 ± 26%     +77.3%       6200 ±  6%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
      1549 ± 40%     -78.4%     335.00 ±  8%  interrupts.CPU4.RES:Rescheduling_interrupts
      1225 ± 80%     -95.1%      59.50 ±148%  interrupts.CPU4.TLB:TLB_shootdowns
      1007 ± 16%     -76.4%     238.25 ± 24%  interrupts.CPU40.RES:Rescheduling_interrupts
      1750 ± 68%     -70.0%     524.25 ±172%  interrupts.CPU40.TLB:TLB_shootdowns
    928.25 ± 15%     -77.5%     208.50 ± 28%  interrupts.CPU41.RES:Rescheduling_interrupts
      1986 ± 74%     -63.5%     725.00 ±171%  interrupts.CPU41.TLB:TLB_shootdowns
      1075 ±  7%     -79.6%     219.50 ± 13%  interrupts.CPU42.RES:Rescheduling_interrupts
      1510 ± 71%     -78.8%     320.00 ±169%  interrupts.CPU42.TLB:TLB_shootdowns
    974.00 ± 14%     -74.2%     251.50 ± 25%  interrupts.CPU43.RES:Rescheduling_interrupts
      1709 ± 62%     -72.1%     477.50 ±172%  interrupts.CPU43.TLB:TLB_shootdowns
    958.25 ± 10%     -70.8%     280.25 ± 18%  interrupts.CPU44.RES:Rescheduling_interrupts
      1681 ± 70%     -81.4%     312.00 ±169%  interrupts.CPU44.TLB:TLB_shootdowns
      1056 ±  9%     -79.3%     218.25 ± 18%  interrupts.CPU45.RES:Rescheduling_interrupts
      1663 ± 72%     -80.0%     333.00 ±169%  interrupts.CPU45.TLB:TLB_shootdowns
    962.00 ±  8%     -60.8%     377.00 ± 56%  interrupts.CPU46.RES:Rescheduling_interrupts
      1369 ± 67%     -91.1%     121.25 ±157%  interrupts.CPU46.TLB:TLB_shootdowns
      5775 ±  9%     -30.4%       4018 ± 24%  interrupts.CPU47.NMI:Non-maskable_interrupts
      5775 ±  9%     -30.4%       4018 ± 24%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      1030 ± 60%     -75.4%     253.25 ±164%  interrupts.CPU47.TLB:TLB_shootdowns
      1156 ± 19%     -77.2%     264.25 ± 16%  interrupts.CPU48.RES:Rescheduling_interrupts
      1054 ± 12%     -78.0%     231.50 ± 33%  interrupts.CPU49.RES:Rescheduling_interrupts
    200.25 ± 52%     -97.6%       4.75 ± 89%  interrupts.CPU49.TLB:TLB_shootdowns
      1355 ± 22%     -78.0%     298.00 ± 19%  interrupts.CPU5.RES:Rescheduling_interrupts
      1204 ± 86%     -95.3%      56.50 ±148%  interrupts.CPU5.TLB:TLB_shootdowns
      1174 ±  8%     -78.5%     252.00 ± 25%  interrupts.CPU50.RES:Rescheduling_interrupts
    213.25 ± 58%     -95.5%       9.50 ±131%  interrupts.CPU50.TLB:TLB_shootdowns
      1065 ± 13%     -79.1%     222.75 ± 18%  interrupts.CPU51.RES:Rescheduling_interrupts
    151.00 ± 63%     -97.2%       4.25 ±124%  interrupts.CPU51.TLB:TLB_shootdowns
      4336 ± 35%     +52.4%       6610 ±  8%  interrupts.CPU52.NMI:Non-maskable_interrupts
      4336 ± 35%     +52.4%       6610 ±  8%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      1151 ±  5%     -81.4%     213.75 ±  7%  interrupts.CPU52.RES:Rescheduling_interrupts
    272.00 ± 64%     -86.1%      37.75 ±138%  interrupts.CPU52.TLB:TLB_shootdowns
      4835 ± 26%     +35.7%       6563 ± 12%  interrupts.CPU53.NMI:Non-maskable_interrupts
      4835 ± 26%     +35.7%       6563 ± 12%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    136.25 ± 46%     -95.0%       6.75 ± 48%  interrupts.CPU53.TLB:TLB_shootdowns
      4263 ± 39%     +52.6%       6506 ± 10%  interrupts.CPU54.NMI:Non-maskable_interrupts
      4263 ± 39%     +52.6%       6506 ± 10%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
    988.25 ± 14%     -78.2%     215.25 ± 16%  interrupts.CPU54.RES:Rescheduling_interrupts
    138.00 ± 58%     -99.3%       1.00 ± 70%  interrupts.CPU54.TLB:TLB_shootdowns
      1066 ± 20%     -75.7%     259.00 ± 19%  interrupts.CPU55.RES:Rescheduling_interrupts
    125.25 ± 66%     -89.4%      13.25 ± 88%  interrupts.CPU55.TLB:TLB_shootdowns
      1134 ±  7%     -81.3%     212.50 ± 20%  interrupts.CPU56.RES:Rescheduling_interrupts
    159.25 ± 96%     -92.6%      11.75 ±139%  interrupts.CPU56.TLB:TLB_shootdowns
    941.25 ± 15%     -73.1%     253.50 ± 13%  interrupts.CPU57.RES:Rescheduling_interrupts
     87.75 ± 73%     -94.6%       4.75 ± 93%  interrupts.CPU57.TLB:TLB_shootdowns
      4661 ± 19%     +34.7%       6279 ±  9%  interrupts.CPU58.NMI:Non-maskable_interrupts
      4661 ± 19%     +34.7%       6279 ±  9%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      1156 ± 14%     -76.9%     267.75 ± 11%  interrupts.CPU58.RES:Rescheduling_interrupts
    135.75 ± 80%     -87.1%      17.50 ±137%  interrupts.CPU58.TLB:TLB_shootdowns
      1205 ± 24%     -81.5%     222.75 ± 13%  interrupts.CPU59.RES:Rescheduling_interrupts
     69.25 ± 73%     -96.4%       2.50 ±150%  interrupts.CPU59.TLB:TLB_shootdowns
      1329 ± 28%     -78.6%     284.00 ± 11%  interrupts.CPU6.RES:Rescheduling_interrupts
      1336 ± 75%     -97.2%      37.00 ±120%  interrupts.CPU6.TLB:TLB_shootdowns
      1027 ±  8%     -71.7%     290.50 ± 13%  interrupts.CPU60.RES:Rescheduling_interrupts
    114.25 ± 62%     -94.3%       6.50 ± 85%  interrupts.CPU60.TLB:TLB_shootdowns
      1210 ± 18%     -76.5%     284.50 ± 30%  interrupts.CPU61.RES:Rescheduling_interrupts
    112.00 ± 94%     -96.2%       4.25 ±159%  interrupts.CPU61.TLB:TLB_shootdowns
      1331 ± 18%     -80.6%     258.00 ±  5%  interrupts.CPU62.RES:Rescheduling_interrupts
    181.50 ± 84%     -97.1%       5.25 ± 93%  interrupts.CPU62.TLB:TLB_shootdowns
      3851 ±  8%      -6.7%       3595 ±  9%  interrupts.CPU63.CAL:Function_call_interrupts
      1071 ± 12%     -74.1%     277.25 ± 19%  interrupts.CPU63.RES:Rescheduling_interrupts
    105.75 ± 42%     -98.3%       1.75 ± 62%  interrupts.CPU63.TLB:TLB_shootdowns
      1046 ± 20%     -74.7%     264.50 ± 31%  interrupts.CPU64.RES:Rescheduling_interrupts
    333.50 ±121%     -99.2%       2.75 ± 69%  interrupts.CPU64.TLB:TLB_shootdowns
      3869 ±  8%      -6.2%       3629 ±  8%  interrupts.CPU65.CAL:Function_call_interrupts
    978.75 ± 15%     -75.4%     241.25 ± 40%  interrupts.CPU65.RES:Rescheduling_interrupts
    128.50 ± 86%     -86.6%      17.25 ±127%  interrupts.CPU65.TLB:TLB_shootdowns
      3862 ±  8%      -5.4%       3651 ±  8%  interrupts.CPU66.CAL:Function_call_interrupts
    960.50 ± 22%     -74.2%     247.75 ± 24%  interrupts.CPU66.RES:Rescheduling_interrupts
    130.25 ± 79%     -92.3%      10.00 ±119%  interrupts.CPU66.TLB:TLB_shootdowns
     60.75 ±116%    -100.0%       0.00        interrupts.CPU67.103:PCI-MSI.31981636-edge.i40e-eth0-TxRx-67
      1004 ±  7%     -65.7%     344.25 ± 23%  interrupts.CPU67.RES:Rescheduling_interrupts
    128.75 ±110%     -86.2%      17.75 ±124%  interrupts.CPU67.TLB:TLB_shootdowns
      3859 ±  9%      -5.9%       3630 ±  8%  interrupts.CPU68.CAL:Function_call_interrupts
      1061 ±  9%     -75.6%     259.00 ± 29%  interrupts.CPU68.RES:Rescheduling_interrupts
    275.00 ±107%     -97.8%       6.00 ± 82%  interrupts.CPU68.TLB:TLB_shootdowns
      3865 ±  9%      -5.9%       3638 ±  8%  interrupts.CPU69.CAL:Function_call_interrupts
      1103 ±  9%     -80.2%     218.50 ± 15%  interrupts.CPU69.RES:Rescheduling_interrupts
      1211 ± 26%     -49.2%     615.25 ± 44%  interrupts.CPU7.RES:Rescheduling_interrupts
      1487 ± 83%     -89.3%     159.50 ±163%  interrupts.CPU7.TLB:TLB_shootdowns
    908.25 ± 17%     -79.0%     190.50 ± 18%  interrupts.CPU70.RES:Rescheduling_interrupts
    915.25 ±155%     -99.0%       8.75 ±102%  interrupts.CPU70.TLB:TLB_shootdowns
      1806 ±162%     -99.5%       9.25 ± 85%  interrupts.CPU71.TLB:TLB_shootdowns
      1049 ± 20%     -76.8%     243.25 ± 17%  interrupts.CPU72.RES:Rescheduling_interrupts
    175.75 ± 74%     -95.9%       7.25 ± 51%  interrupts.CPU72.TLB:TLB_shootdowns
      1078 ± 18%     -77.4%     243.75 ± 17%  interrupts.CPU73.RES:Rescheduling_interrupts
    282.00 ± 72%     -97.5%       7.00 ± 33%  interrupts.CPU73.TLB:TLB_shootdowns
    987.25 ±  6%     -65.0%     345.50 ± 17%  interrupts.CPU74.RES:Rescheduling_interrupts
    265.00 ± 71%     -95.8%      11.25 ±117%  interrupts.CPU74.TLB:TLB_shootdowns
      5923 ± 14%     -22.2%       4610 ± 14%  interrupts.CPU75.NMI:Non-maskable_interrupts
      5923 ± 14%     -22.2%       4610 ± 14%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
    989.50 ± 11%     -74.2%     255.75 ± 44%  interrupts.CPU75.RES:Rescheduling_interrupts
    150.75 ± 70%     -90.7%      14.00 ±108%  interrupts.CPU75.TLB:TLB_shootdowns
      3879 ±  9%      -6.6%       3624 ±  9%  interrupts.CPU76.CAL:Function_call_interrupts
    965.25 ±  7%     -63.2%     355.25 ± 57%  interrupts.CPU76.RES:Rescheduling_interrupts
    221.75 ± 90%     -97.5%       5.50 ± 79%  interrupts.CPU76.TLB:TLB_shootdowns
    983.75 ± 15%     -70.7%     288.50 ± 16%  interrupts.CPU77.RES:Rescheduling_interrupts
    147.00 ± 66%     -84.0%      23.50 ±108%  interrupts.CPU77.TLB:TLB_shootdowns
      1193 ± 12%     -76.4%     281.25 ± 15%  interrupts.CPU78.RES:Rescheduling_interrupts
      5918 ± 15%     -22.2%       4604 ± 29%  interrupts.CPU79.NMI:Non-maskable_interrupts
      5918 ± 15%     -22.2%       4604 ± 29%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      1019 ±  9%     -75.5%     249.50 ± 23%  interrupts.CPU79.RES:Rescheduling_interrupts
    122.75 ± 76%     -92.9%       8.75 ± 58%  interrupts.CPU79.TLB:TLB_shootdowns
      1228 ± 77%     -96.9%      38.25 ±142%  interrupts.CPU8.TLB:TLB_shootdowns
    934.25 ± 14%     -76.4%     220.50 ± 32%  interrupts.CPU80.RES:Rescheduling_interrupts
      1043 ± 12%     -78.4%     225.25 ±  7%  interrupts.CPU81.RES:Rescheduling_interrupts
    942.50 ±  8%     -68.8%     293.75 ± 23%  interrupts.CPU82.RES:Rescheduling_interrupts
    948.25 ± 12%     -76.4%     223.75 ± 11%  interrupts.CPU84.RES:Rescheduling_interrupts
    119.25 ± 78%     -86.2%      16.50 ± 89%  interrupts.CPU84.TLB:TLB_shootdowns
      1027 ± 14%     -79.5%     211.00 ±  9%  interrupts.CPU85.RES:Rescheduling_interrupts
    128.75 ± 81%     -95.7%       5.50 ± 37%  interrupts.CPU85.TLB:TLB_shootdowns
      1267 ± 44%     -83.4%     210.25 ± 10%  interrupts.CPU86.RES:Rescheduling_interrupts
      1045 ± 16%     -79.9%     210.25 ± 24%  interrupts.CPU87.RES:Rescheduling_interrupts
    128.75 ± 78%     -84.5%      20.00 ±144%  interrupts.CPU87.TLB:TLB_shootdowns
    921.00 ± 16%     -74.9%     231.50 ± 27%  interrupts.CPU88.RES:Rescheduling_interrupts
      1132 ± 14%     -75.1%     281.50 ± 68%  interrupts.CPU89.RES:Rescheduling_interrupts
      1504 ± 69%     -96.9%      46.25 ±156%  interrupts.CPU9.TLB:TLB_shootdowns
      1030 ±  8%     -75.3%     254.75 ± 40%  interrupts.CPU90.RES:Rescheduling_interrupts
    960.75 ±  6%     -67.3%     313.75 ± 34%  interrupts.CPU91.RES:Rescheduling_interrupts
      1072 ± 13%     -76.1%     255.75 ± 17%  interrupts.CPU92.RES:Rescheduling_interrupts
      5634 ± 10%     -31.3%       3871 ± 24%  interrupts.CPU93.NMI:Non-maskable_interrupts
      5634 ± 10%     -31.3%       3871 ± 24%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
      1086 ± 19%     -71.0%     314.75 ± 48%  interrupts.CPU93.RES:Rescheduling_interrupts
    871.75 ± 86%     -80.1%     173.50 ±168%  interrupts.CPU93.TLB:TLB_shootdowns
    985.00 ±  8%     -76.2%     234.75 ± 13%  interrupts.CPU94.RES:Rescheduling_interrupts
      1050 ± 13%     -82.5%     184.00 ±  3%  interrupts.CPU95.RES:Rescheduling_interrupts
      3477 ± 98%     -86.8%     459.25 ±169%  interrupts.CPU95.TLB:TLB_shootdowns
    110301 ±  4%     -68.6%      34619 ± 13%  interrupts.RES:Rescheduling_interrupts
     90838 ± 55%     -79.2%      18886 ±164%  interrupts.TLB:TLB_shootdowns


                                                                                
                           vm-scalability.time.user_time                        
                                                                                
  8500 +-+------------------------------------------------------------------+   
       |   +.  .+.    +        +.      +..+      .+. .+..+   +   +.+.   +.  |   
  8400 +-+   +.   +  : + .+.. +  +.+. +    + .+.+   +     + +             +.|   
  8300 +-+         + :  +    +       +      +              +                |   
       |            +                                                       |   
  8200 +-+                                                                  |   
  8100 +-+                                                                  |   
       |                                                                    |   
  8000 +-+                                                                  |   
  7900 +-+                   O                                              |   
       |     O                       O          O                O          |   
  7800 +-O      O   O     O               O   O   O      O O       O        |   
  7700 O-+              O      O O     O    O       O          O            |   
       |          O   O            O                  O      O              |   
  7600 +-+-O----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           vm-scalability.time.system_time                      
                                                                                
  14500 +-+------------O----------------------------------------------------+   
        |   O      O O              O   O   O      O   O     O O            |   
  14000 O-O   O  O       O O O O  O   O   O   O O    O   O O     O  O       |   
        |                                                                   |   
        |                                                                   |   
  13500 +-+                                                                 |   
        |                                                                   |   
  13000 +-+                                                                 |   
        |                                                                   |   
  12500 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  12000 +-+.+.+..+.+.+.+. .+.+.+..   .+.+.+.   .+..+.+.+.+.+.+.+.  .+. .+. .|   
        |                +        +.+       +.+                  +.   +   + |   
  11500 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                  vm-scalability.time.percent_of_cpu_this_job_got               
                                                                                
  7300 +-+----------O------------------------------------------O------------+   
       O O      O O   O O O    O O O   O  O O O O   O O  O O     O O        |   
  7200 +-+ O O               O       O            O          O              |   
       |                                                                    |   
  7100 +-+                                                                  |   
       |                                                                    |   
  7000 +-+                                                                  |   
       |                                                                    |   
  6900 +-+                                                                  |   
       |                                                                    |   
  6800 +-+                                                                  |   
       |                                                                    |   
  6700 +-+.+.+..     .+.          .+.+.  .+.+. .+.+.+.+..+. .+.+.    .+. .+.|   
       |        +.+.+   +.+..+.+.+     +.     +            +     +.+.   +   |   
  6600 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         vm-scalability.time.minor_page_faults                  
                                                                                
    6e+06 +-+---------------------------------------------------------------+   
  5.8e+06 +-+             .+.                                               |   
          |  .+.+.   .+.+.   +.+.+.+.+.+.+.     .+.+.+.+.+..+.+.+.+.+.+.+.+.|   
  5.6e+06 +-+     +.+                      +.+.+                            |   
  5.4e+06 +-+                                                               |   
          |                                                                 |   
  5.2e+06 +-+                                                               |   
    5e+06 +-+                                                               |   
  4.8e+06 +-+                                                               |   
          |                                                                 |   
  4.6e+06 +-+                                                               |   
  4.4e+06 +-+                                                               |   
          |           O O                O   O                  O           |   
  4.2e+06 O-O O O O O      O O O O O O O   O   O O O O O O  O O   O O       |   
    4e+06 +-+---------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    vm-scalability.time.voluntary_context_switches              
                                                                                
    2e+06 +-+---------------------------------------------------------------+   
  1.8e+06 +-+            .. +                                               |   
          |  .+.+.   .+.+    +.+.+.+.+.+.+.     .+.+.+.+.+..+.+.+.+.+.+.+.+.|   
  1.6e+06 +-+     +.+                      +.+.+                            |   
  1.4e+06 +-+                                                               |   
          |                                                                 |   
  1.2e+06 +-+                                                               |   
    1e+06 +-+                                                               |   
   800000 +-+                                                               |   
          |                                                                 |   
   600000 +-+                                                               |   
   400000 +-+                                                               |   
          |                                                                 |   
   200000 +-+                                                               |   
        0 O-O-O-O-O-O-O-O--O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O--O-O-O-O-O-------+   
                                                                                
                                                                                                                                                                
                  vm-scalability.time.involuntary_context_switches              
                                                                                
  90000 +-+-----------------------------------------------------------------+   
  85000 +-+                O                                                |   
        |                                       O                           |   
  80000 +-+                                                                 |   
  75000 +-+      O     O     O O                     O   O     O O  O       |   
  70000 O-O   O    O O   O        O O O O O O O    O   O   O O              |   
  65000 +-+ O                                                               |   
        |                                                                   |   
  60000 +-+                                                                 |   
  55000 +-+                                                                 |   
  50000 +-+                                                                 |   
  45000 +-+                                                                 |   
        |.+. .+..+.+. .+.+.        .+.   .+.      .+. .+.     .+.    .+.+.+.|   
  40000 +-+ +        +     +.+.+..+   +.+   +.+.+.   +   +.+.+   +..+       |   
  35000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               vm-scalability.throughput                        
                                                                                
   3.8e+07 +-+--------------------------------------------------------------+   
           |   O                                                            |   
  3.78e+07 +-+   O       O O         O                      O   O           |   
           O O     O O           O O     O  O O O O     O     O     O       |   
  3.76e+07 +-+               O O       O            O O   O       O         |   
  3.74e+07 +-+                                                              |   
           |           O                                                    |   
  3.72e+07 +-+                                                              |   
           |                                                                |   
   3.7e+07 +-+                                                              |   
  3.68e+07 +-+                                                              |   
           |          .+               +.    .+.                +.          |   
  3.66e+07 +-+.+.+. .+  +    +.+.+.   +  +..+   +.+. .+. .+.+. +  +.+.+.+.+.|   
           | +     +     +. +      +.+              +   +     +             |   
  3.64e+07 +-+--------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.1.0-10212-g1b426bac66e6c" of type "text/plain" (190235 bytes)

View attachment "job-script" of type "text/plain" (7728 bytes)

View attachment "job.yaml" of type "text/plain" (5343 bytes)

View attachment "reproduce" of type "text/plain" (6686 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ