lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20190326042507.GA7277@shao2-debian>
Date:   Tue, 26 Mar 2019 12:25:07 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Ming Lei <ming.lei@...hat.com>
Cc:     Jens Axboe <axboe@...nel.dk>, Omar Sandoval <osandov@...com>,
        Christoph Hellwig <hch@....de>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [block] 6861428921: vm-scalability.median 1003.2% improvement

Greeting,

FYI, we noticed a 1003.2% improvement of vm-scalability.median due to commit:


commit: 6861428921b51113520cd47897be6c2774e4fc58 ("block: always define BIO_MAX_PAGES as 256")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: vm-scalability
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	runtime: 300
	thp_enabled: always
	thp_defrag: always
	nr_task: 32
	nr_ssd: 1
	test: swap-w-seq
	cpufreq_governor: performance

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_ssd/nr_task/rootfs/runtime/tbox_group/test/testcase/thp_defrag/thp_enabled:
  gcc-7/performance/x86_64-rhel-7.6/1/32/debian-x86_64-2018-04-03.cgz/300/lkp-skl-2sp7/swap-w-seq/vm-scalability/always/always

commit: 
  07173c3ec2 ("block: enable multipage bvecs")
  6861428921 ("block: always define BIO_MAX_PAGES as 256")

07173c3ec276cbb1 6861428921b51113520cd47897b 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.Out_of_memory:Kill_process
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          2:4          -50%            :4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
      0.12 ± 32%     -65.5%       0.04 ±  4%  vm-scalability.free_time
     89576 ± 28%   +1003.2%     988196 ±  5%  vm-scalability.median
     51.89 ± 13%     -78.0%      11.43 ±  3%  vm-scalability.stddev
   8556830 ±  8%    +310.9%   35161381 ±  5%  vm-scalability.throughput
    141.53 ± 20%     -88.7%      15.95 ±  5%  vm-scalability.time.elapsed_time
    141.53 ± 20%     -88.7%      15.95 ±  5%  vm-scalability.time.elapsed_time.max
      2774 ± 15%     +45.1%       4024 ± 13%  vm-scalability.time.involuntary_context_switches
    204787 ±  3%     +43.0%     292906 ± 13%  vm-scalability.time.minor_page_faults
    167.75 ± 20%   +1210.9%       2199        vm-scalability.time.percent_of_cpu_this_job_got
    106.53 ±  7%    +110.2%     223.98 ±  7%  vm-scalability.time.system_time
    121.58            +4.4%     126.97        vm-scalability.time.user_time
     38699 ± 21%     -97.4%     994.00 ±  2%  vm-scalability.time.voluntary_context_switches
     29.00            +4.0%      30.15 ±  3%  boot-time.boot
     30.57 ±  2%     -29.7        0.87        mpstat.cpu.all.iowait%
      0.10 ±115%      -0.1        0.00 ± 49%  mpstat.cpu.all.soft%
      1.12 ± 14%     +17.6       18.68 ±  2%  mpstat.cpu.all.sys%
      1.29 ± 25%      +9.6       10.89 ±  3%  mpstat.cpu.all.usr%
    363832 ± 11%     -33.5%     242033 ±  3%  numa-numastat.node0.local_node
    372060 ± 11%     -34.5%     243702 ±  3%  numa-numastat.node0.numa_hit
    323271 ±  6%     -36.1%     206635 ±  9%  numa-numastat.node1.local_node
    329122 ±  7%     -33.4%     219052 ±  8%  numa-numastat.node1.numa_hit
     34179 ±  7%     -12.1%      30028 ±  8%  numa-meminfo.node1.KReclaimable
     34179 ±  7%     -12.1%      30028 ±  8%  numa-meminfo.node1.SReclaimable
     56266 ±  3%     -12.8%      49084 ±  6%  numa-meminfo.node1.SUnreclaim
      1880 ± 38%    +329.5%       8076 ± 52%  numa-meminfo.node1.Shmem
     90446 ±  5%     -12.5%      79113 ±  6%  numa-meminfo.node1.Slab
    726448 ±  7%     -11.7%     641589 ± 10%  numa-vmstat.node0.numa_hit
    718153 ±  8%     -10.9%     639936 ± 10%  numa-vmstat.node0.numa_local
    470.75 ± 39%    +328.8%       2018 ± 52%  numa-vmstat.node1.nr_shmem
      8544 ±  7%     -12.1%       7507 ±  8%  numa-vmstat.node1.nr_slab_reclaimable
     14066 ±  3%     -12.8%      12270 ±  6%  numa-vmstat.node1.nr_slab_unreclaimable
 1.455e+08 ±149%     -99.6%     591042 ±  8%  cpuidle.C1.time
   1256451 ±158%     -99.4%       8079        cpuidle.C1.usage
 5.623e+09 ± 20%     -87.0%  7.331e+08 ±  5%  cpuidle.C1E.time
  14043488 ± 11%     -88.8%    1575926 ±  5%  cpuidle.C1E.usage
    241125 ± 36%     -94.8%      12566 ± 11%  cpuidle.POLL.time
     36871 ± 20%     -87.0%       4780 ± 26%  cpuidle.POLL.usage
    112.25 ± 10%    +724.9%     926.00        turbostat.Avg_MHz
      5.44 ±  3%     +24.8       30.25        turbostat.Busy%
      2070 ±  8%     +48.3%       3070        turbostat.Bzy_MHz
   1253728 ±158%     -99.7%       4008 ±  2%  turbostat.C1
      1.18 ±144%      -1.1        0.04 ± 12%  turbostat.C1%
  14041415 ± 11%     -88.8%    1579538 ±  5%  turbostat.C1E
     89.11 ±  6%     -23.0%      68.62        turbostat.CPU%c1
     47.50 ±  4%     +10.2%      52.33 ±  3%  turbostat.CoreTmp
  20865755 ± 20%     -88.0%    2504935 ±  5%  turbostat.IRQ
      1.40 ± 58%     +10.3       11.68 ± 32%  turbostat.PKG_%
    110.81 ±  5%     +99.5%     221.02        turbostat.PkgWatt
     33.30 ±  4%    +139.1%      79.64 ±  2%  turbostat.RAMWatt
     67.00            +7.5%      72.00        vmstat.cpu.id
      0.50 ±100%   +1766.7%       9.33 ±  5%  vmstat.cpu.us
     29.75 ±  2%    -100.0%       0.00        vmstat.cpu.wa
    153.50 ± 48%    +316.1%     638.67 ± 24%  vmstat.io.bi
    394547 ±  3%     -97.3%      10790 ±  4%  vmstat.io.bo
  43779329 ±  9%     +40.0%   61286364        vmstat.memory.free
  23332648 ± 25%     -92.8%    1677588 ±  3%  vmstat.memory.swpd
     22.50 ±  3%    -100.0%       0.00        vmstat.procs.b
      1.25 ± 34%   +1473.3%      19.67 ±  2%  vmstat.procs.r
    137.50 ± 54%    +280.6%     523.33 ± 29%  vmstat.swap.si
    394532 ±  3%     -97.3%      10682 ±  4%  vmstat.swap.so
      2924 ±  2%      -8.3%       2682 ±  2%  vmstat.system.cs
    145508            -8.6%     132947        vmstat.system.in
  84326843 ±  5%     -16.3%   70555519        meminfo.Active
  84326614 ±  5%     -16.3%   70555392        meminfo.Active(anon)
  85966552 ±  5%     -19.8%   68973123        meminfo.AnonHugePages
  86219683 ±  5%     -18.3%   70470647        meminfo.AnonPages
 2.369e+08 ±  2%     -18.5%  1.931e+08        meminfo.Committed_AS
    306860 ±  9%     -12.5%     268446 ±  7%  meminfo.DirectMap4k
   2420185 ±  3%     -32.0%    1645128 ±  2%  meminfo.Inactive
   2418347 ±  3%     -32.0%    1644489 ±  2%  meminfo.Inactive(anon)
  42221972 ± 11%     +34.7%   56854854        meminfo.MemAvailable
  42804230 ± 11%     +34.2%   57438579        meminfo.MemFree
  88905904 ±  5%     -16.5%   74271557        meminfo.Memused
    218034 ±  4%     -34.5%     142816        meminfo.PageTables
     18404 ± 26%     +44.3%      26562 ± 12%  meminfo.Shmem
    540318 ±  9%    +201.2%    1627241 ±  3%  meminfo.SwapCached
    851206 ± 28%    +690.9%    6732372 ±  3%  meminfo.max_used_kB
    527.00 ± 12%     -20.9%     417.00 ±  3%  slabinfo.biovec-128.active_objs
    527.00 ± 12%     -20.9%     417.00 ±  3%  slabinfo.biovec-128.num_objs
    344.25 ± 21%     -72.1%      96.00 ±  3%  slabinfo.biovec-max.active_objs
    346.75 ± 20%     -72.3%      96.00 ±  3%  slabinfo.biovec-max.num_objs
      1943 ±  3%     -20.0%       1555        slabinfo.blkdev_ioc.active_objs
      1943 ±  3%     -20.0%       1555        slabinfo.blkdev_ioc.num_objs
    112.00           +96.1%     219.67 ±  9%  slabinfo.btrfs_extent_map.active_objs
    112.00           +96.1%     219.67 ±  9%  slabinfo.btrfs_extent_map.num_objs
     17394 ±  4%      -7.1%      16157 ±  4%  slabinfo.filp.active_objs
     17608 ±  4%      -7.2%      16333 ±  4%  slabinfo.filp.num_objs
      8561 ±  5%     -15.3%       7252 ±  2%  slabinfo.kmalloc-96.active_objs
      8570 ±  5%     -14.7%       7311 ±  2%  slabinfo.kmalloc-96.num_objs
      1248 ±  3%     -18.2%       1021 ± 15%  slabinfo.task_group.active_objs
      1248 ±  3%     -18.2%       1021 ± 15%  slabinfo.task_group.num_objs
    141.50 ± 43%     -47.5%      74.33 ± 81%  slabinfo.xfs_btree_cur.active_objs
    141.50 ± 43%     -47.5%      74.33 ± 81%  slabinfo.xfs_btree_cur.num_objs
     67.36            +7.8%      72.58        iostat.cpu.idle
     30.11 ±  2%     -97.4%       0.79        iostat.cpu.iowait
      1.23 ±  8%   +1268.7%      16.84 ±  3%  iostat.cpu.system
      1.30 ± 24%    +651.8%       9.79 ±  2%  iostat.cpu.user
     56.50 ±  2%     -97.0%       1.68 ±  5%  iostat.sdc.avgqu-sz
      1857           -97.4%      47.62        iostat.sdc.avgrq-sz
    134.57           -97.4%       3.56 ±  5%  iostat.sdc.await
    166.35 ± 12%     -70.6%      48.83 ±  3%  iostat.sdc.await.max
     24.14 ± 53%    +299.8%      96.54 ± 28%  iostat.sdc.r/s
     74.44 ± 25%     -99.5%       0.37 ± 21%  iostat.sdc.r_await
    125.94 ±  7%     -96.2%       4.73 ± 39%  iostat.sdc.r_await.max
    138.16 ± 53%    +293.3%     543.33 ± 28%  iostat.sdc.rkB/s
      9.66 ± 57%    +258.4%      34.64 ± 33%  iostat.sdc.rrqm/s
     39.63 ±  3%     -90.8%       3.63 ± 13%  iostat.sdc.util
    385.01 ±  3%     -92.8%      27.62 ±  9%  iostat.sdc.w/s
    136.30           -70.9%      39.63 ± 14%  iostat.sdc.w_await
    166.66 ± 12%    +238.0%     563.33 ± 13%  iostat.sdc.w_await.max
    391580 ±  3%     -97.3%      10682 ±  4%  iostat.sdc.wkB/s
    342.04 ± 33%    +672.7%       2643 ±  4%  iostat.sdc.wrqm/s
     19.90 ± 30%     -19.9        0.00        perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     17.41 ± 33%     -17.4        0.00        perf-profile.calltrace.cycles-pp.page_fault
     17.41 ± 33%     -17.4        0.00        perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
     17.40 ± 33%     -17.4        0.00        perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
     17.39 ± 33%     -17.4        0.00        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     17.38 ± 33%     -17.4        0.00        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     17.31 ± 34%     -17.3        0.00        perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
     16.88 ± 30%     -16.9        0.00        perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
     14.20 ± 49%     -14.2        0.00        perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
     13.67 ± 50%     -13.7        0.00        perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
     79.58 ±  9%     -12.3       67.25 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     78.64 ±  8%     -11.4       67.25 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     78.64 ±  8%     -11.4       67.25 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     78.59 ±  8%     -11.3       67.25 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     10.66 ± 30%     -10.7        0.00        perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
     10.32 ± 50%     -10.3        0.00        perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault
      6.14 ± 34%      -6.1        0.00        perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.50 ± 32%      -5.5        0.00        perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      4.78 ± 28%      -4.8        0.00        perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
     19069 ± 39%     -59.4%       7750 ± 90%  proc-vmstat.compact_free_scanned
      9577 ± 42%     -67.8%       3082 ± 97%  proc-vmstat.compact_isolated
  21104747 ±  5%     -16.4%   17643515        proc-vmstat.nr_active_anon
  21583000 ±  5%     -18.4%   17603447        proc-vmstat.nr_anon_pages
     42043 ±  5%     -19.9%      33672        proc-vmstat.nr_anon_transparent_hugepages
   1050704 ± 11%     +34.9%    1417852        proc-vmstat.nr_dirty_background_threshold
   2103983 ± 11%     +34.9%    2839208        proc-vmstat.nr_dirty_threshold
    445307 ±  3%     +61.9%     720882        proc-vmstat.nr_file_pages
  10676807 ± 11%     +34.4%   14354359        proc-vmstat.nr_free_pages
    605531 ±  3%     -32.1%     411181 ±  3%  proc-vmstat.nr_inactive_anon
     11350           -77.0%       2611 ±  8%  proc-vmstat.nr_isolated_anon
     13240            -2.8%      12870        proc-vmstat.nr_kernel_stack
     54623 ±  4%     -34.6%      35718        proc-vmstat.nr_page_table_pages
      4602 ± 26%     +44.3%       6640 ± 12%  proc-vmstat.nr_shmem
     30951            -5.3%      29303        proc-vmstat.nr_slab_unreclaimable
      9.50 ±173%    +777.2%      83.33 ± 50%  proc-vmstat.nr_vmscan_immediate_reclaim
     58616 ± 14%   +4992.7%    2985134 ± 10%  proc-vmstat.nr_vmscan_write
     79569 ± 17%     -36.3%      50713 ±  2%  proc-vmstat.nr_written
  21104747 ±  5%     -16.4%   17643515        proc-vmstat.nr_zone_active_anon
    605531 ±  3%     -32.1%     411155 ±  3%  proc-vmstat.nr_zone_inactive_anon
     21148 ± 26%    +333.6%      91702 ± 24%  proc-vmstat.numa_foreign
    726653 ±  8%     -33.1%     486398 ±  3%  proc-vmstat.numa_hit
     85245 ±  2%     +14.8%      97846 ±  4%  proc-vmstat.numa_huge_pte_updates
    712567 ±  8%     -33.7%     472313 ±  3%  proc-vmstat.numa_local
     21148 ± 26%    +333.6%      91702 ± 24%  proc-vmstat.numa_miss
     35234 ± 15%    +200.2%     105786 ± 21%  proc-vmstat.numa_other
  43663389 ±  2%     +14.9%   50170277 ±  4%  proc-vmstat.numa_pte_updates
  15711286 ± 22%     -91.1%    1393556 ±  6%  proc-vmstat.pgdeactivate
    590550 ± 12%     -39.2%     359010 ± 10%  proc-vmstat.pgfault
  77729133            +2.1%   79393516        proc-vmstat.pgfree
      4505 ± 41%     -54.1%       2067 ± 71%  proc-vmstat.pgmigrate_success
  56874030 ± 22%     -99.6%     204725 ±  2%  proc-vmstat.pgpgout
     96029 ± 51%     -55.6%      42642 ±  4%  proc-vmstat.pgrefill
     78643 ± 17%     -36.6%      49878 ±  2%  proc-vmstat.pgrotated
    197998 ± 28%   +3237.5%    6608253 ±  8%  proc-vmstat.pgscan_direct
     29.75 ±149%  +70918.5%      21128 ± 78%  proc-vmstat.pgscan_kswapd
     79484 ± 17%     -35.8%      51047        proc-vmstat.pgsteal_direct
  14217995 ± 22%     -99.6%      50668 ±  2%  proc-vmstat.pswpout
    150640            +2.4%     154272        proc-vmstat.thp_deferred_split_page
     27686 ± 22%     -90.6%       2603 ±  5%  proc-vmstat.thp_split_pmd
     27668 ± 22%    -100.0%       0.00        proc-vmstat.thp_swpout
 2.878e+09 ± 25%    +815.4%  2.635e+10 ±  3%  perf-stat.i.branch-instructions
      0.78 ± 33%      -0.6        0.22 ± 23%  perf-stat.i.branch-miss-rate%
   7946383 ± 24%    +245.0%   27415662 ±  5%  perf-stat.i.branch-misses
     18.20 ± 24%     +31.4       49.57        perf-stat.i.cache-miss-rate%
  16435347 ± 14%   +2031.0%  3.502e+08 ±  2%  perf-stat.i.cache-misses
  63696811 ± 13%   +1023.7%  7.158e+08        perf-stat.i.cache-references
      2851           -18.5%       2324 ±  3%  perf-stat.i.context-switches
      1.21 ±  6%     -30.8%       0.84 ±  4%  perf-stat.i.cpi
 7.395e+09 ± 11%    +874.9%   7.21e+10        perf-stat.i.cpu-cycles
      7.16 ± 14%    +212.9%      22.39 ±  6%  perf-stat.i.cpu-migrations
    747.18 ± 18%     -68.3%     237.10        perf-stat.i.cycles-between-cache-misses
      0.07 ± 58%      -0.1        0.00 ±  5%  perf-stat.i.dTLB-load-miss-rate%
 2.049e+09 ± 23%    +831.5%  1.909e+10 ±  2%  perf-stat.i.dTLB-loads
      0.02 ± 49%      -0.0        0.00 ± 15%  perf-stat.i.dTLB-store-miss-rate%
     70657 ± 38%    +123.4%     157882 ± 14%  perf-stat.i.dTLB-store-misses
 7.366e+08 ± 22%    +664.9%  5.634e+09 ±  5%  perf-stat.i.dTLB-stores
 9.412e+09 ± 24%    +843.3%  8.878e+10 ±  3%  perf-stat.i.instructions
    192493 ± 70%    +657.2%    1457546 ± 14%  perf-stat.i.instructions-per-iTLB-miss
      1.05 ± 13%     +25.3%       1.31 ±  3%  perf-stat.i.ipc
      8.82 ± 45%    +539.0%      56.34 ± 51%  perf-stat.i.major-faults
      4060 ± 11%    +442.7%      22033 ± 16%  perf-stat.i.minor-faults
    277510 ± 20%   +2857.2%    8206676 ±  9%  perf-stat.i.node-load-misses
   1262778 ± 10%   +4905.1%   63203290        perf-stat.i.node-loads
      8.83 ± 97%      -5.3        3.58 ± 11%  perf-stat.i.node-store-miss-rate%
     42108 ± 20%   +2480.3%    1086527 ±  8%  perf-stat.i.node-store-misses
   3464808 ± 27%    +931.8%   35749595 ±  5%  perf-stat.i.node-stores
      4068 ± 11%    +442.9%      22089 ± 16%  perf-stat.i.page-faults
      0.29 ± 37%      -0.2        0.10        perf-stat.overall.branch-miss-rate%
     26.56 ± 15%     +22.4       48.92        perf-stat.overall.cache-miss-rate%
    446.06 ±  2%     -53.8%     206.15 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.03 ± 61%      -0.0        0.00 ±  2%  perf-stat.overall.dTLB-load-miss-rate%
      0.01 ± 46%      -0.0        0.00 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
    157221 ±113%    +566.3%    1047514 ± 11%  perf-stat.overall.instructions-per-iTLB-miss
     18.23 ± 26%      -6.7       11.49 ±  9%  perf-stat.overall.node-load-miss-rate%
      1.19 ±  8%      +1.8        2.97 ± 13%  perf-stat.overall.node-store-miss-rate%
 2.918e+09 ± 22%    +745.4%  2.467e+10 ±  3%  perf-stat.ps.branch-instructions
   7929699 ± 24%    +225.2%   25790635 ±  4%  perf-stat.ps.branch-misses
  16720398 ± 10%   +1860.8%  3.278e+08 ±  2%  perf-stat.ps.cache-misses
  64027917 ± 13%    +946.5%  6.701e+08        perf-stat.ps.cache-references
      2832           -22.9%       2183 ±  3%  perf-stat.ps.context-switches
 7.452e+09 ±  9%    +806.2%  6.754e+10        perf-stat.ps.cpu-cycles
      7.24 ± 12%    +190.7%      21.05 ±  5%  perf-stat.ps.cpu-migrations
 2.077e+09 ± 20%    +760.4%  1.787e+10 ±  2%  perf-stat.ps.dTLB-loads
     70147 ± 38%    +111.1%     148113 ± 13%  perf-stat.ps.dTLB-store-misses
 7.454e+08 ± 19%    +607.6%  5.275e+09 ±  5%  perf-stat.ps.dTLB-stores
 9.534e+09 ± 21%    +771.8%  8.312e+10 ±  3%  perf-stat.ps.instructions
      8.95 ± 42%    +493.8%      53.12 ± 52%  perf-stat.ps.major-faults
      4079 ±  9%    +406.4%      20658 ± 15%  perf-stat.ps.minor-faults
    142918            -5.6%     134916        perf-stat.ps.msec
    275536 ± 20%   +2688.1%    7682267 ± 10%  perf-stat.ps.node-load-misses
   1256768 ± 10%   +4605.7%   59140271        perf-stat.ps.node-loads
     42074 ± 18%   +2317.7%    1017230 ±  8%  perf-stat.ps.node-store-misses
   3533362 ± 23%    +847.4%   33473753 ±  5%  perf-stat.ps.node-stores
      4088 ±  9%    +406.6%      20711 ± 15%  perf-stat.ps.page-faults
     12346 ± 34%     -81.8%       2243 ± 22%  sched_debug.cfs_rq:/.load.avg
    453397 ± 26%     -95.1%      22090 ± 24%  sched_debug.cfs_rq:/.load.max
     66975 ± 31%     -91.6%       5640 ± 11%  sched_debug.cfs_rq:/.load.stddev
     42.00 ±  3%    +124.3%      94.19 ± 21%  sched_debug.cfs_rq:/.load_avg.avg
    751.00 ± 16%     +37.8%       1034        sched_debug.cfs_rq:/.load_avg.max
    140.48 ± 11%     +87.7%     263.71 ± 13%  sched_debug.cfs_rq:/.load_avg.stddev
     46652 ±  9%     -59.0%      19148 ±  9%  sched_debug.cfs_rq:/.min_vruntime.avg
     92238 ± 11%     -67.0%      30398 ± 11%  sched_debug.cfs_rq:/.min_vruntime.max
     29251 ±  7%     -88.0%       3517 ±  8%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.06 ±  9%    +123.1%       0.13 ± 27%  sched_debug.cfs_rq:/.nr_running.avg
      0.22 ±  3%     +49.8%       0.34 ± 12%  sched_debug.cfs_rq:/.nr_running.stddev
     15.30 ± 13%    +268.0%      56.31 ± 35%  sched_debug.cfs_rq:/.removed.load_avg.avg
    384.00 ± 19%    +166.7%       1024        sched_debug.cfs_rq:/.removed.load_avg.max
     74.20 ±  9%    +207.6%     228.21 ± 16%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    707.25 ± 13%    +267.5%       2599 ± 35%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
     17831 ± 19%    +167.1%      47625        sched_debug.cfs_rq:/.removed.runnable_sum.max
      3429 ±  9%    +207.4%      10541 ± 16%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      5.99 ±  8%    +261.3%      21.65 ± 28%  sched_debug.cfs_rq:/.removed.util_avg.avg
    192.71 ± 19%    +163.1%     507.00        sched_debug.cfs_rq:/.removed.util_avg.max
     30.20 ± 14%    +201.6%      91.09 ± 10%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      5.55 ± 22%     -68.0%       1.78 ± 19%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    341.04 ± 26%     -95.3%      16.00 ±  5%  sched_debug.cfs_rq:/.runnable_load_avg.max
     41.28 ± 25%     -88.7%       4.65 ±  9%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     12298 ± 34%     -82.9%       2098 ± 16%  sched_debug.cfs_rq:/.runnable_weight.avg
    449033 ± 27%     -96.0%      18087 ±  2%  sched_debug.cfs_rq:/.runnable_weight.max
     66673 ± 31%     -92.0%       5347 ±  5%  sched_debug.cfs_rq:/.runnable_weight.stddev
     40040 ± 40%     -58.6%      16557 ± 44%  sched_debug.cfs_rq:/.spread0.max
     29251 ±  7%     -88.0%       3519 ±  8%  sched_debug.cfs_rq:/.spread0.stddev
    173.85 ± 17%    +145.2%     426.28 ±  2%  sched_debug.cfs_rq:/.util_avg.avg
    858.50 ± 14%     +50.7%       1293 ±  6%  sched_debug.cfs_rq:/.util_avg.max
    153.07 ± 12%     +78.4%     273.07 ±  6%  sched_debug.cfs_rq:/.util_avg.stddev
     10.79 ± 21%    +156.9%      27.72 ± 12%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    236.58 ± 15%    +115.7%     510.33 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.max
     43.34 ± 11%    +141.3%     104.56 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    829892 ±  3%     -25.9%     615172        sched_debug.cpu.avg_idle.avg
    228918 ± 73%     -99.4%       1421 ± 21%  sched_debug.cpu.avg_idle.min
    153019 ± 14%     +57.6%     241163 ±  8%  sched_debug.cpu.avg_idle.stddev
     87991 ± 15%     -60.9%      34364 ±  3%  sched_debug.cpu.clock.avg
     88116 ± 15%     -61.0%      34368 ±  3%  sched_debug.cpu.clock.max
     87240 ± 15%     -60.6%      34360 ±  3%  sched_debug.cpu.clock.min
    280.54 ± 26%     -99.2%       2.30 ± 15%  sched_debug.cpu.clock.stddev
     87991 ± 15%     -60.9%      34364 ±  3%  sched_debug.cpu.clock_task.avg
     88116 ± 15%     -61.0%      34368 ±  3%  sched_debug.cpu.clock_task.max
     87240 ± 15%     -60.6%      34360 ±  3%  sched_debug.cpu.clock_task.min
    280.54 ± 26%     -99.2%       2.30 ± 15%  sched_debug.cpu.clock_task.stddev
      5.62 ± 23%     -62.9%       2.09 ± 15%  sched_debug.cpu.cpu_load[0].avg
    341.38 ± 26%     -95.3%      16.00 ±  5%  sched_debug.cpu.cpu_load[0].max
     41.31 ± 25%     -88.2%       4.89 ±  6%  sched_debug.cpu.cpu_load[0].stddev
      5.77 ± 24%     -54.1%       2.65 ± 21%  sched_debug.cpu.cpu_load[1].avg
    334.12 ± 26%     -93.5%      21.67 ± 33%  sched_debug.cpu.cpu_load[1].max
     40.30 ± 26%     -88.1%       4.82 ± 16%  sched_debug.cpu.cpu_load[1].stddev
      5.46 ± 32%     -33.0%       3.66 ± 25%  sched_debug.cpu.cpu_load[2].avg
    299.12 ± 38%     -82.7%      51.67 ± 40%  sched_debug.cpu.cpu_load[2].max
     35.71 ± 37%     -79.4%       7.36 ± 36%  sched_debug.cpu.cpu_load[2].stddev
    259.21 ± 55%     -70.9%      75.33 ± 14%  sched_debug.cpu.cpu_load[3].max
     31.38 ± 53%     -62.7%      11.70 ± 18%  sched_debug.cpu.cpu_load[3].stddev
    120.21 ±  9%    +100.4%     240.89 ± 27%  sched_debug.cpu.curr->pid.avg
      3203 ± 11%     -46.0%       1729        sched_debug.cpu.curr->pid.max
     12408 ± 34%     -80.1%       2471 ± 32%  sched_debug.cpu.load.avg
    454839 ± 27%     -95.1%      22090 ± 24%  sched_debug.cpu.load.max
     67115 ± 31%     -91.2%       5938 ± 18%  sched_debug.cpu.load.stddev
      0.00 ± 25%     -89.5%       0.00 ± 11%  sched_debug.cpu.next_balance.stddev
     62558 ± 21%     -86.5%       8451        sched_debug.cpu.nr_load_updates.avg
     69133 ± 20%     -77.1%      15845 ±  8%  sched_debug.cpu.nr_load_updates.max
     59990 ± 21%     -89.0%       6608 ± 18%  sched_debug.cpu.nr_load_updates.min
      0.06 ±  8%    +137.0%       0.15 ± 26%  sched_debug.cpu.nr_running.avg
      0.23 ±  2%     +54.4%       0.35 ± 11%  sched_debug.cpu.nr_running.stddev
      3321 ± 17%     -66.4%       1114 ± 11%  sched_debug.cpu.nr_switches.avg
     27971 ± 15%     -74.1%       7248 ± 17%  sched_debug.cpu.nr_switches.max
    542.54 ± 12%     -38.8%     332.00 ± 15%  sched_debug.cpu.nr_switches.min
      3735 ±  9%     -63.0%       1383 ±  8%  sched_debug.cpu.nr_switches.stddev
      0.23 ±  9%     -85.9%       0.03 ± 20%  sched_debug.cpu.nr_uninterruptible.avg
     87202 ± 15%     -60.6%      34360 ±  3%  sched_debug.cpu_clk
     84395 ± 16%     -62.6%      31575 ±  3%  sched_debug.ktime
     87780 ± 15%     -60.2%      34931 ±  3%  sched_debug.sched_clk
     30382 ± 22%     -93.9%       1860 ± 15%  softirqs.BLOCK
     33695 ± 24%     -93.1%       2330 ± 17%  softirqs.CPU0.RCU
     21554 ± 18%     -64.6%       7624 ± 10%  softirqs.CPU0.SCHED
     66621 ± 24%     -79.3%      13785 ±  2%  softirqs.CPU0.TIMER
     34498 ± 19%     -89.9%       3496 ± 70%  softirqs.CPU1.RCU
     19179 ± 18%     -80.0%       3833 ± 34%  softirqs.CPU1.SCHED
     65051 ± 25%     -78.2%      14171 ± 38%  softirqs.CPU1.TIMER
     38090 ± 27%     -92.6%       2823 ± 15%  softirqs.CPU10.RCU
     19372 ± 20%     -84.9%       2927 ± 12%  softirqs.CPU10.SCHED
     66363 ± 25%     -81.7%      12126 ±  7%  softirqs.CPU10.TIMER
     35164 ± 26%     -91.6%       2945 ± 33%  softirqs.CPU11.RCU
     19482 ± 22%     -83.0%       3312 ±  6%  softirqs.CPU11.SCHED
     65570 ± 25%     -81.5%      12108 ±  7%  softirqs.CPU11.TIMER
     36102 ± 24%     -94.2%       2099 ±  5%  softirqs.CPU12.RCU
     27007 ± 59%     -88.6%       3090        softirqs.CPU12.SCHED
     66236 ± 25%     -81.8%      12072 ±  5%  softirqs.CPU12.TIMER
     37852 ± 27%     -91.1%       3385 ± 52%  softirqs.CPU13.RCU
     19566 ± 20%     -81.9%       3535 ± 19%  softirqs.CPU13.SCHED
     66208 ± 25%     -79.1%      13818 ± 19%  softirqs.CPU13.TIMER
     33605 ± 24%     -92.8%       2420 ± 28%  softirqs.CPU14.RCU
     19345 ± 20%     -84.0%       3092 ± 17%  softirqs.CPU14.SCHED
     64337 ± 23%     -81.6%      11829 ±  3%  softirqs.CPU14.TIMER
     26910 ± 24%     -92.5%       2012 ± 20%  softirqs.CPU15.RCU
     19239 ± 20%     -85.5%       2789 ± 13%  softirqs.CPU15.SCHED
     65447 ± 25%     -82.3%      11615 ±  4%  softirqs.CPU15.TIMER
     26523 ± 23%     -91.1%       2359 ± 22%  softirqs.CPU16.RCU
     19218 ± 19%     -84.1%       3051 ±  9%  softirqs.CPU16.SCHED
     66088 ± 26%     -81.1%      12511 ±  8%  softirqs.CPU16.TIMER
     26962 ± 24%     -87.9%       3262 ± 69%  softirqs.CPU17.RCU
     19703 ± 20%     -83.5%       3245 ±  5%  softirqs.CPU17.SCHED
     66543 ± 25%     -81.7%      12153 ±  6%  softirqs.CPU17.TIMER
     33228 ± 23%     -92.2%       2597 ± 26%  softirqs.CPU18.RCU
     19627 ± 23%     -81.4%       3652 ± 14%  softirqs.CPU18.SCHED
     51119 ± 21%     -75.6%      12487 ±  9%  softirqs.CPU18.TIMER
     33693 ± 23%     -91.4%       2894 ± 15%  softirqs.CPU19.RCU
     19758 ± 20%     -85.7%       2822 ± 20%  softirqs.CPU19.SCHED
     57958 ± 38%     -74.0%      15055 ± 24%  softirqs.CPU19.TIMER
     36652 ± 27%     -90.2%       3587 ± 51%  softirqs.CPU2.RCU
     19522 ± 19%     -82.7%       3369 ±  9%  softirqs.CPU2.SCHED
     67481 ± 22%     -82.5%      11824 ±  6%  softirqs.CPU2.TIMER
     33741 ± 26%     -94.1%       1987 ± 30%  softirqs.CPU20.RCU
     19347 ± 20%     -83.8%       3130 ± 16%  softirqs.CPU20.SCHED
     49615 ± 19%     -77.1%      11379 ±  4%  softirqs.CPU20.TIMER
     33377 ± 22%     -93.9%       2023 ± 18%  softirqs.CPU21.RCU
     19061 ± 20%     -83.5%       3154 ± 19%  softirqs.CPU21.SCHED
     48446 ± 20%     -76.3%      11478 ±  6%  softirqs.CPU21.TIMER
     32776 ± 24%     -92.0%       2620 ± 55%  softirqs.CPU22.RCU
     19233 ± 19%     -86.1%       2675 ± 26%  softirqs.CPU22.SCHED
     49106 ± 22%     -73.8%      12851 ± 10%  softirqs.CPU22.TIMER
     31660 ± 22%     -94.1%       1883 ± 28%  softirqs.CPU23.RCU
     18764 ± 20%     -82.0%       3386 ± 12%  softirqs.CPU23.SCHED
     46906 ± 19%     -75.3%      11594 ±  7%  softirqs.CPU23.TIMER
     32115 ± 24%     -94.7%       1688 ± 30%  softirqs.CPU24.RCU
     19048 ± 19%     -83.6%       3130 ± 15%  softirqs.CPU24.SCHED
     49073 ± 19%     -76.5%      11533 ±  7%  softirqs.CPU24.TIMER
     31473 ± 23%     -93.2%       2152 ±  3%  softirqs.CPU25.RCU
     19257 ± 19%     -83.0%       3280 ± 13%  softirqs.CPU25.SCHED
     51553 ± 22%     -77.5%      11596 ±  4%  softirqs.CPU25.TIMER
     33810 ± 26%     -91.7%       2813 ± 48%  softirqs.CPU26.RCU
     19343 ± 18%     -83.2%       3244 ± 14%  softirqs.CPU26.SCHED
     49490 ± 21%     -76.0%      11864 ±  5%  softirqs.CPU26.TIMER
     32552 ± 24%     -89.6%       3373 ± 69%  softirqs.CPU27.RCU
     18875 ± 19%     -82.8%       3254 ±  4%  softirqs.CPU27.SCHED
     48454 ± 19%     -76.0%      11626 ±  5%  softirqs.CPU27.TIMER
     30030 ± 46%     -93.2%       2047 ± 25%  softirqs.CPU28.RCU
     19167 ± 17%     -83.5%       3172 ±  6%  softirqs.CPU28.SCHED
     47942 ± 17%     -75.9%      11566 ±  3%  softirqs.CPU28.TIMER
     33103 ± 20%     -87.8%       4025 ± 39%  softirqs.CPU29.RCU
     19403 ± 16%     -84.3%       3047 ±  5%  softirqs.CPU29.SCHED
     54323 ±  7%     -73.7%      14297 ± 29%  softirqs.CPU29.TIMER
     36877 ± 29%     -78.1%       8089 ± 11%  softirqs.CPU3.RCU
     19493 ± 19%     -81.8%       3550 ± 14%  softirqs.CPU3.SCHED
     65990 ± 25%     -75.8%      15969 ±  8%  softirqs.CPU3.TIMER
     28611 ± 23%     -94.3%       1630 ± 11%  softirqs.CPU30.RCU
     19185 ± 19%     -85.7%       2740 ± 23%  softirqs.CPU30.SCHED
     49156 ± 21%     -76.1%      11737 ±  5%  softirqs.CPU30.TIMER
     27996 ± 22%     -92.3%       2145 ± 19%  softirqs.CPU31.RCU
     19271 ± 18%     -82.2%       3434 ±  6%  softirqs.CPU31.SCHED
     50908 ± 22%     -76.5%      11985 ±  2%  softirqs.CPU31.TIMER
     28302 ± 24%     -93.8%       1742 ± 30%  softirqs.CPU32.RCU
     18860 ± 20%     -83.0%       3198 ±  3%  softirqs.CPU32.SCHED
     46783 ± 20%     -74.9%      11725 ±  2%  softirqs.CPU32.TIMER
     30319 ± 23%     -93.4%       2012 ±  9%  softirqs.CPU33.RCU
     19493 ± 20%     -83.7%       3180 ±  4%  softirqs.CPU33.SCHED
     66157 ± 32%     -82.1%      11831 ±  3%  softirqs.CPU33.TIMER
     28390 ± 23%     -91.6%       2376 ± 24%  softirqs.CPU34.RCU
     19349 ± 19%     -82.7%       3348 ±  8%  softirqs.CPU34.SCHED
     48588 ± 21%     -75.7%      11819        softirqs.CPU34.TIMER
     28752 ± 22%     -93.6%       1836 ±  6%  softirqs.CPU35.RCU
     19247 ± 18%     -83.5%       3173 ±  8%  softirqs.CPU35.SCHED
     49822 ± 19%     -76.0%      11964        softirqs.CPU35.TIMER
     32018 ± 22%     -92.8%       2313 ± 11%  softirqs.CPU36.RCU
     19323 ± 20%     -87.5%       2413 ± 12%  softirqs.CPU36.SCHED
     65438 ± 25%     -82.1%      11711 ±  6%  softirqs.CPU36.TIMER
     34935 ± 22%     -90.9%       3191 ±  6%  softirqs.CPU37.RCU
     19599 ± 21%     -79.3%       4055 ± 18%  softirqs.CPU37.SCHED
     66573 ± 27%     -80.5%      12976        softirqs.CPU37.TIMER
     33636 ± 25%     -91.7%       2798 ± 33%  softirqs.CPU38.RCU
     19747 ± 22%     -82.2%       3506 ± 19%  softirqs.CPU38.SCHED
     66313 ± 26%     -81.5%      12280 ±  8%  softirqs.CPU38.TIMER
     33655 ± 26%     -90.7%       3140 ± 58%  softirqs.CPU39.RCU
     18984 ± 21%     -83.0%       3233 ± 14%  softirqs.CPU39.SCHED
     63569 ± 26%     -77.1%      14569 ± 24%  softirqs.CPU39.TIMER
     35962 ± 25%     -89.2%       3900 ± 52%  softirqs.CPU4.RCU
     19414 ± 20%     -85.4%       2828 ± 30%  softirqs.CPU4.SCHED
     65642 ± 25%     -80.4%      12836 ±  2%  softirqs.CPU4.TIMER
     35225 ± 24%     -94.2%       2049 ±  5%  softirqs.CPU40.RCU
     19411 ± 21%     -83.0%       3295 ± 11%  softirqs.CPU40.SCHED
     65189 ± 25%     -81.5%      12076 ±  4%  softirqs.CPU40.TIMER
     28976 ± 38%     -92.8%       2090 ± 22%  softirqs.CPU41.RCU
     18919 ± 21%     -83.2%       3179 ± 16%  softirqs.CPU41.SCHED
     64523 ± 26%     -81.5%      11968 ±  4%  softirqs.CPU41.TIMER
     30996 ± 22%     -94.2%       1803 ± 24%  softirqs.CPU42.RCU
     19314 ± 20%     -83.6%       3163 ± 18%  softirqs.CPU42.SCHED
     67652 ± 29%     -82.9%      11580 ±  4%  softirqs.CPU42.TIMER
     31537 ± 21%     -93.6%       2024 ± 11%  softirqs.CPU43.RCU
     18801 ± 19%     -83.9%       3034 ± 24%  softirqs.CPU43.SCHED
     62622 ± 22%     -80.9%      11973 ±  5%  softirqs.CPU43.TIMER
     34114 ± 25%     -93.6%       2172 ± 11%  softirqs.CPU44.RCU
     19184 ± 20%     -84.6%       2960 ± 23%  softirqs.CPU44.SCHED
     65380 ± 25%     -82.1%      11726 ±  7%  softirqs.CPU44.TIMER
     31132 ± 21%     -93.3%       2077 ± 26%  softirqs.CPU45.RCU
     21106 ± 26%     -83.3%       3517 ±  8%  softirqs.CPU45.SCHED
     65507 ± 26%     -81.7%      11999        softirqs.CPU45.TIMER
     34797 ± 23%     -91.9%       2819 ± 23%  softirqs.CPU46.RCU
     19317 ± 21%     -82.5%       3378 ± 19%  softirqs.CPU46.SCHED
     65578 ± 26%     -81.8%      11941 ±  6%  softirqs.CPU46.TIMER
     25883 ± 41%     -90.4%       2497 ± 23%  softirqs.CPU47.RCU
     19536 ± 20%     -83.8%       3160 ± 23%  softirqs.CPU47.SCHED
     65082 ± 25%     -81.9%      11767 ±  3%  softirqs.CPU47.TIMER
     35469 ± 24%     -93.2%       2416 ± 19%  softirqs.CPU48.RCU
     19539 ± 19%     -83.9%       3141 ± 22%  softirqs.CPU48.SCHED
     65712 ± 26%     -81.7%      12001 ±  4%  softirqs.CPU48.TIMER
     36383 ± 21%     -95.5%       1625 ± 19%  softirqs.CPU49.RCU
     19286 ± 21%     -79.1%       4038 ± 35%  softirqs.CPU49.SCHED
     66192 ± 23%     -82.0%      11883 ±  3%  softirqs.CPU49.TIMER
     36443 ± 26%     -93.8%       2263 ± 16%  softirqs.CPU5.RCU
     19520 ± 20%     -83.8%       3157 ±  7%  softirqs.CPU5.SCHED
     65193 ± 25%     -81.4%      12096 ±  6%  softirqs.CPU5.TIMER
     32092 ± 23%     -94.5%       1754 ± 12%  softirqs.CPU50.RCU
     20294 ± 23%     -85.7%       2892 ± 24%  softirqs.CPU50.SCHED
     64333 ± 24%     -82.2%      11432 ±  2%  softirqs.CPU50.TIMER
     31428 ± 19%     -93.3%       2106 ± 25%  softirqs.CPU51.RCU
     21209 ± 24%     -81.9%       3844 ± 34%  softirqs.CPU51.SCHED
     65230 ± 25%     -81.8%      11888 ±  5%  softirqs.CPU51.TIMER
     33852 ± 23%     -93.6%       2171 ± 21%  softirqs.CPU52.RCU
     19506 ± 20%     -84.6%       3002 ± 21%  softirqs.CPU52.SCHED
     65431 ± 26%     -81.5%      12125 ±  7%  softirqs.CPU52.TIMER
     35488 ± 24%     -91.1%       3145 ± 64%  softirqs.CPU53.RCU
     19538 ± 21%     -85.4%       2855 ± 13%  softirqs.CPU53.SCHED
     65716 ± 26%     -82.3%      11613 ±  4%  softirqs.CPU53.TIMER
     28481 ± 22%     -93.6%       1831 ± 28%  softirqs.CPU54.RCU
     19105 ± 18%     -83.8%       3093 ±  6%  softirqs.CPU54.SCHED
     48345 ± 18%     -75.9%      11629 ±  4%  softirqs.CPU54.TIMER
     27730 ± 20%     -95.9%       1144 ± 26%  softirqs.CPU55.RCU
     18638 ± 17%     -77.1%       4262 ± 32%  softirqs.CPU55.SCHED
     45647 ± 16%     -74.3%      11712 ±  3%  softirqs.CPU55.TIMER
     28531 ± 23%     -94.3%       1624 ± 29%  softirqs.CPU56.RCU
     22300 ± 15%     -86.0%       3125 ±  5%  softirqs.CPU56.SCHED
     46258 ± 17%     -74.5%      11775 ±  5%  softirqs.CPU56.TIMER
     28703 ± 21%     -94.7%       1508 ±  2%  softirqs.CPU57.RCU
     18009 ± 23%     -83.9%       2905 ±  8%  softirqs.CPU57.SCHED
     45957 ± 18%     -75.0%      11491 ±  3%  softirqs.CPU57.TIMER
     27503 ± 21%     -94.5%       1514 ± 22%  softirqs.CPU58.RCU
     18444 ± 20%     -81.6%       3399 ± 14%  softirqs.CPU58.SCHED
     47391 ± 20%     -75.2%      11773 ±  4%  softirqs.CPU58.TIMER
     28356 ± 21%     -94.1%       1681 ± 14%  softirqs.CPU59.RCU
     18172 ± 18%     -84.2%       2871 ± 15%  softirqs.CPU59.SCHED
     47058 ± 18%     -75.6%      11472 ±  6%  softirqs.CPU59.TIMER
     32712 ± 21%     -92.3%       2509 ± 11%  softirqs.CPU6.RCU
     19308 ± 21%     -82.8%       3317 ±  9%  softirqs.CPU6.SCHED
     66288 ± 25%     -81.6%      12223 ±  6%  softirqs.CPU6.TIMER
     32168 ± 24%     -95.3%       1501 ±  2%  softirqs.CPU60.RCU
     18584 ± 18%     -85.2%       2746 ± 20%  softirqs.CPU60.SCHED
     47481 ± 18%     -75.7%      11525 ±  6%  softirqs.CPU60.TIMER
     30453 ± 22%     -90.2%       2989 ± 56%  softirqs.CPU61.RCU
     18686 ± 18%     -85.8%       2655 ± 22%  softirqs.CPU61.SCHED
     50042 ± 22%     -76.6%      11691 ±  5%  softirqs.CPU61.TIMER
     30580 ± 24%     -94.0%       1846 ±  7%  softirqs.CPU62.RCU
     18292 ± 18%     -86.4%       2492 ± 27%  softirqs.CPU62.SCHED
     46824 ± 19%     -75.2%      11627 ±  2%  softirqs.CPU62.TIMER
     31793 ± 19%     -95.6%       1402 ±  8%  softirqs.CPU63.RCU
     18089 ± 17%     -83.9%       2907 ± 14%  softirqs.CPU63.SCHED
     47219 ± 17%     -75.6%      11510 ±  4%  softirqs.CPU63.TIMER
     32333 ± 29%     -90.1%       3201 ± 56%  softirqs.CPU64.RCU
     18330 ± 17%     -85.6%       2643 ± 24%  softirqs.CPU64.SCHED
     45631 ± 17%     -73.4%      12148 ±  5%  softirqs.CPU64.TIMER
     30285 ± 29%     -95.9%       1248 ± 18%  softirqs.CPU65.RCU
     18423 ± 17%     -83.8%       2979 ±  5%  softirqs.CPU65.SCHED
     49334 ± 21%     -77.3%      11222 ±  2%  softirqs.CPU65.TIMER
     31483 ± 26%     -94.7%       1679 ± 32%  softirqs.CPU66.RCU
     18350 ± 19%     -83.9%       2953 ± 26%  softirqs.CPU66.SCHED
     46858 ± 18%     -75.1%      11658 ±  3%  softirqs.CPU66.TIMER
     30388 ± 22%     -95.6%       1338 ±  5%  softirqs.CPU67.RCU
     18446 ± 19%     -85.5%       2668 ± 19%  softirqs.CPU67.SCHED
     49426 ± 22%     -76.7%      11495 ±  4%  softirqs.CPU67.TIMER
     30256 ± 26%     -95.1%       1476 ± 26%  softirqs.CPU68.RCU
     18127 ± 19%     -86.1%       2522 ± 19%  softirqs.CPU68.SCHED
     47197 ± 17%     -75.4%      11612 ±  7%  softirqs.CPU68.TIMER
     27610 ± 27%     -93.2%       1864 ± 19%  softirqs.CPU69.RCU
     18502 ± 19%     -86.5%       2491 ± 18%  softirqs.CPU69.SCHED
     47136 ± 21%     -75.3%      11665 ±  6%  softirqs.CPU69.TIMER
     33062 ± 21%     -92.9%       2357 ± 26%  softirqs.CPU7.RCU
     19330 ± 20%     -84.0%       3088 ± 10%  softirqs.CPU7.SCHED
     63992 ± 23%     -81.3%      11959 ±  5%  softirqs.CPU7.TIMER
     30725 ± 24%     -90.5%       2919 ± 62%  softirqs.CPU70.RCU
     18410 ± 19%     -86.0%       2572 ± 16%  softirqs.CPU70.SCHED
     45716 ± 18%     -74.5%      11635 ±  4%  softirqs.CPU70.TIMER
     31738 ± 27%     -96.1%       1223 ± 19%  softirqs.CPU71.RCU
     18528 ± 19%     -87.1%       2393 ± 16%  softirqs.CPU71.SCHED
     46690 ± 18%     -75.2%      11575 ±  3%  softirqs.CPU71.TIMER
     39101 ± 30%     -90.3%       3805 ± 39%  softirqs.CPU8.RCU
     19596 ± 20%     -76.3%       4640 ± 18%  softirqs.CPU8.SCHED
     66597 ± 25%     -76.5%      15634 ± 26%  softirqs.CPU8.TIMER
     33648 ± 29%     -93.3%       2253        softirqs.CPU9.RCU
     19448 ± 21%     -81.3%       3637 ± 13%  softirqs.CPU9.SCHED
     66104 ± 25%     -81.5%      12215 ±  5%  softirqs.CPU9.TIMER
   2308563 ± 23%     -92.6%     171957 ± 12%  softirqs.RCU
   1393678 ± 20%     -83.4%     231487 ±  3%  softirqs.SCHED
   4127446 ± 22%     -78.8%     876092 ±  2%  softirqs.TIMER
    143.50 ± 40%     -84.4%      22.33 ± 75%  interrupts.69:PCI-MSI.12589058-edge.eth3-TxRx-1
    192.00 ± 61%     -94.8%      10.00 ± 16%  interrupts.70:PCI-MSI.12589059-edge.eth3-TxRx-2
    105.25 ± 20%     -73.1%      28.33 ± 94%  interrupts.71:PCI-MSI.12589060-edge.eth3-TxRx-3
    264.00 ±106%     -94.4%      14.67 ± 17%  interrupts.72:PCI-MSI.12589061-edge.eth3-TxRx-4
    246.25 ± 51%     -89.6%      25.67 ± 70%  interrupts.73:PCI-MSI.12589062-edge.eth3-TxRx-5
    135.75 ± 30%     -82.6%      23.67 ± 78%  interrupts.74:PCI-MSI.12589063-edge.eth3-TxRx-6
    194309 ± 24%     -69.8%      58683 ± 11%  interrupts.CAL:Function_call_interrupts
      1981 ± 20%     -65.9%     675.00 ±  9%  interrupts.CPU0.CAL:Function_call_interrupts
    284946 ± 20%     -88.1%      33839 ±  5%  interrupts.CPU0.LOC:Local_timer_interrupts
    552.00 ± 99%    -100.0%       0.00        interrupts.CPU0.NMI:Non-maskable_interrupts
    552.00 ± 99%    -100.0%       0.00        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      1901 ± 22%     -63.6%     691.33 ± 15%  interrupts.CPU1.CAL:Function_call_interrupts
    284845 ± 20%     -88.2%      33562 ±  6%  interrupts.CPU1.LOC:Local_timer_interrupts
    499.25 ±109%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
    499.25 ±109%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      1849 ± 19%     -54.7%     837.33 ± 27%  interrupts.CPU10.CAL:Function_call_interrupts
    284556 ± 20%     -88.1%      33988 ±  5%  interrupts.CPU10.LOC:Local_timer_interrupts
    151.50 ± 35%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
    151.50 ± 35%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      1659 ± 14%     -60.4%     657.33 ±  3%  interrupts.CPU11.CAL:Function_call_interrupts
    284538 ± 20%     -88.1%      33824 ±  5%  interrupts.CPU11.LOC:Local_timer_interrupts
    170.00 ± 43%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
    170.00 ± 43%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      1763 ±  6%     -64.4%     628.33 ±  6%  interrupts.CPU12.CAL:Function_call_interrupts
    284719 ± 20%     -88.2%      33574 ±  5%  interrupts.CPU12.LOC:Local_timer_interrupts
    200.25 ± 34%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
    200.25 ± 34%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
      1737 ±  9%     -55.7%     769.33 ± 29%  interrupts.CPU13.CAL:Function_call_interrupts
    284593 ± 20%     -88.2%      33557 ±  6%  interrupts.CPU13.LOC:Local_timer_interrupts
    283.00 ± 59%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
    283.00 ± 59%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
      1671 ± 13%     -64.1%     599.67 ±  6%  interrupts.CPU14.CAL:Function_call_interrupts
    284472 ± 20%     -88.2%      33428 ±  6%  interrupts.CPU14.LOC:Local_timer_interrupts
    230.50 ± 44%     -99.9%       0.33 ±141%  interrupts.CPU14.NMI:Non-maskable_interrupts
    230.50 ± 44%     -99.9%       0.33 ±141%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      1800 ±  6%     -51.6%     872.00 ± 28%  interrupts.CPU15.CAL:Function_call_interrupts
    284375 ± 20%     -88.3%      33355 ±  6%  interrupts.CPU15.LOC:Local_timer_interrupts
    151.75 ± 38%    -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
    151.75 ± 38%    -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      5.50 ± 73%   +5803.0%     324.67 ±118%  interrupts.CPU15.RES:Rescheduling_interrupts
      1645 ± 16%     -52.8%     776.00 ± 34%  interrupts.CPU16.CAL:Function_call_interrupts
    284481 ± 20%     -88.2%      33582 ±  4%  interrupts.CPU16.LOC:Local_timer_interrupts
    188.25 ± 31%    -100.0%       0.00        interrupts.CPU16.NMI:Non-maskable_interrupts
    188.25 ± 31%    -100.0%       0.00        interrupts.CPU16.PMI:Performance_monitoring_interrupts
      1648 ± 15%     -63.8%     597.33 ±  9%  interrupts.CPU17.CAL:Function_call_interrupts
    284886 ± 20%     -88.3%      33251 ±  5%  interrupts.CPU17.LOC:Local_timer_interrupts
    568.25 ±120%    -100.0%       0.00        interrupts.CPU17.NMI:Non-maskable_interrupts
    568.25 ±120%    -100.0%       0.00        interrupts.CPU17.PMI:Performance_monitoring_interrupts
      3484 ± 36%     -74.2%     899.00 ± 36%  interrupts.CPU18.CAL:Function_call_interrupts
    284608 ± 20%     -88.2%      33724 ±  5%  interrupts.CPU18.LOC:Local_timer_interrupts
    611.75 ±114%    -100.0%       0.00        interrupts.CPU18.NMI:Non-maskable_interrupts
    611.75 ±114%    -100.0%       0.00        interrupts.CPU18.PMI:Performance_monitoring_interrupts
      3470 ± 26%     -77.8%     771.00 ± 16%  interrupts.CPU19.CAL:Function_call_interrupts
    284822 ± 20%     -88.2%      33555 ±  4%  interrupts.CPU19.LOC:Local_timer_interrupts
    741.00 ±125%    -100.0%       0.00        interrupts.CPU19.NMI:Non-maskable_interrupts
    741.00 ±125%    -100.0%       0.00        interrupts.CPU19.PMI:Performance_monitoring_interrupts
    143.50 ± 40%     -84.4%      22.33 ± 75%  interrupts.CPU2.69:PCI-MSI.12589058-edge.eth3-TxRx-1
      1872 ± 21%     -47.5%     983.00 ± 20%  interrupts.CPU2.CAL:Function_call_interrupts
    284688 ± 20%     -88.3%      33217 ±  6%  interrupts.CPU2.LOC:Local_timer_interrupts
    152.00 ± 32%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
    152.00 ± 32%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
      3306 ± 35%     -73.9%     864.00 ± 43%  interrupts.CPU20.CAL:Function_call_interrupts
    284777 ± 20%     -88.2%      33517 ±  4%  interrupts.CPU20.LOC:Local_timer_interrupts
    943.25 ±137%    -100.0%       0.00        interrupts.CPU20.NMI:Non-maskable_interrupts
    943.25 ±137%    -100.0%       0.00        interrupts.CPU20.PMI:Performance_monitoring_interrupts
      3531 ± 42%     -76.9%     816.00 ± 33%  interrupts.CPU21.CAL:Function_call_interrupts
    284825 ± 20%     -88.3%      33392 ±  4%  interrupts.CPU21.LOC:Local_timer_interrupts
    170.00 ± 27%    -100.0%       0.00        interrupts.CPU21.NMI:Non-maskable_interrupts
    170.00 ± 27%    -100.0%       0.00        interrupts.CPU21.PMI:Performance_monitoring_interrupts
      3357 ± 25%     -73.2%     898.00 ± 44%  interrupts.CPU22.CAL:Function_call_interrupts
    284958 ± 20%     -88.2%      33698 ±  4%  interrupts.CPU22.LOC:Local_timer_interrupts
    172.50 ± 34%    -100.0%       0.00        interrupts.CPU22.NMI:Non-maskable_interrupts
    172.50 ± 34%    -100.0%       0.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
      3454 ± 33%     -77.6%     774.33 ± 48%  interrupts.CPU23.CAL:Function_call_interrupts
    284646 ± 20%     -88.2%      33619 ±  4%  interrupts.CPU23.LOC:Local_timer_interrupts
    205.00 ± 40%    -100.0%       0.00        interrupts.CPU23.NMI:Non-maskable_interrupts
    205.00 ± 40%    -100.0%       0.00        interrupts.CPU23.PMI:Performance_monitoring_interrupts
      3063 ± 31%     -71.0%     887.67 ± 41%  interrupts.CPU24.CAL:Function_call_interrupts
    284624 ± 20%     -88.3%      33398 ±  4%  interrupts.CPU24.LOC:Local_timer_interrupts
    299.50 ± 55%     -99.8%       0.67 ±141%  interrupts.CPU24.NMI:Non-maskable_interrupts
    299.50 ± 55%     -99.8%       0.67 ±141%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      2687 ± 18%     -68.2%     854.33 ± 42%  interrupts.CPU25.CAL:Function_call_interrupts
    284821 ± 20%     -88.3%      33398 ±  4%  interrupts.CPU25.LOC:Local_timer_interrupts
    147.50 ± 20%     -99.5%       0.67 ± 70%  interrupts.CPU25.NMI:Non-maskable_interrupts
    147.50 ± 20%     -99.5%       0.67 ± 70%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    284698 ± 20%     -88.2%      33657 ±  6%  interrupts.CPU26.LOC:Local_timer_interrupts
    207.75 ± 22%    -100.0%       0.00        interrupts.CPU26.NMI:Non-maskable_interrupts
    207.75 ± 22%    -100.0%       0.00        interrupts.CPU26.PMI:Performance_monitoring_interrupts
      3079 ± 29%     -79.4%     633.33 ± 12%  interrupts.CPU27.CAL:Function_call_interrupts
    284642 ± 20%     -88.3%      33407 ±  6%  interrupts.CPU27.LOC:Local_timer_interrupts
    244.25 ± 42%    -100.0%       0.00        interrupts.CPU27.NMI:Non-maskable_interrupts
    244.25 ± 42%    -100.0%       0.00        interrupts.CPU27.PMI:Performance_monitoring_interrupts
    295.50 ± 48%     -82.5%      51.67 ±103%  interrupts.CPU27.TLB:TLB_shootdowns
      2523 ± 33%     -73.5%     668.67 ±  5%  interrupts.CPU28.CAL:Function_call_interrupts
    284601 ± 20%     -88.2%      33635 ±  4%  interrupts.CPU28.LOC:Local_timer_interrupts
    196.00 ± 34%    -100.0%       0.00        interrupts.CPU28.NMI:Non-maskable_interrupts
    196.00 ± 34%    -100.0%       0.00        interrupts.CPU28.PMI:Performance_monitoring_interrupts
      2827 ± 34%     -74.5%     721.67 ± 12%  interrupts.CPU29.CAL:Function_call_interrupts
    284946 ± 20%     -88.4%      33151 ±  5%  interrupts.CPU29.LOC:Local_timer_interrupts
    201.50 ± 30%    -100.0%       0.00        interrupts.CPU29.NMI:Non-maskable_interrupts
    201.50 ± 30%    -100.0%       0.00        interrupts.CPU29.PMI:Performance_monitoring_interrupts
     21.00 ± 22%    +647.6%     157.00 ±113%  interrupts.CPU29.RES:Rescheduling_interrupts
    192.00 ± 61%     -94.8%      10.00 ± 16%  interrupts.CPU3.70:PCI-MSI.12589059-edge.eth3-TxRx-2
      1839 ± 24%     -59.2%     751.00 ± 24%  interrupts.CPU3.CAL:Function_call_interrupts
    284621 ± 20%     -88.2%      33619 ±  6%  interrupts.CPU3.LOC:Local_timer_interrupts
    405.00 ±110%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
    405.00 ±110%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
      3440 ± 36%     -78.1%     753.67 ±  8%  interrupts.CPU30.CAL:Function_call_interrupts
    284824 ± 20%     -88.2%      33596 ±  5%  interrupts.CPU30.LOC:Local_timer_interrupts
    273.75 ± 34%    -100.0%       0.00        interrupts.CPU30.NMI:Non-maskable_interrupts
    273.75 ± 34%    -100.0%       0.00        interrupts.CPU30.PMI:Performance_monitoring_interrupts
      3467 ± 40%     -82.6%     603.33 ±  9%  interrupts.CPU31.CAL:Function_call_interrupts
    284780 ± 20%     -88.3%      33370 ±  4%  interrupts.CPU31.LOC:Local_timer_interrupts
    215.00 ± 38%    -100.0%       0.00        interrupts.CPU31.NMI:Non-maskable_interrupts
    215.00 ± 38%    -100.0%       0.00        interrupts.CPU31.PMI:Performance_monitoring_interrupts
      3950 ± 39%     -82.4%     696.33 ±  9%  interrupts.CPU32.CAL:Function_call_interrupts
    284694 ± 20%     -88.3%      33367 ±  4%  interrupts.CPU32.LOC:Local_timer_interrupts
    305.75 ± 47%    -100.0%       0.00        interrupts.CPU32.NMI:Non-maskable_interrupts
    305.75 ± 47%    -100.0%       0.00        interrupts.CPU32.PMI:Performance_monitoring_interrupts
      7621 ±127%     -91.8%     622.33 ±  9%  interrupts.CPU33.CAL:Function_call_interrupts
    284927 ± 20%     -88.3%      33343 ±  4%  interrupts.CPU33.LOC:Local_timer_interrupts
    287.25 ± 47%    -100.0%       0.00        interrupts.CPU33.NMI:Non-maskable_interrupts
    287.25 ± 47%    -100.0%       0.00        interrupts.CPU33.PMI:Performance_monitoring_interrupts
      6481 ±166%     -98.7%      82.33 ± 76%  interrupts.CPU33.TLB:TLB_shootdowns
      3102 ± 49%     -83.7%     506.00 ± 46%  interrupts.CPU34.CAL:Function_call_interrupts
    284717 ± 20%     -88.2%      33564 ±  4%  interrupts.CPU34.LOC:Local_timer_interrupts
    228.50 ± 38%    -100.0%       0.00        interrupts.CPU34.NMI:Non-maskable_interrupts
    228.50 ± 38%    -100.0%       0.00        interrupts.CPU34.PMI:Performance_monitoring_interrupts
      7056 ± 99%     -90.7%     653.33 ± 11%  interrupts.CPU35.CAL:Function_call_interrupts
    284425 ± 20%     -88.2%      33439 ±  4%  interrupts.CPU35.LOC:Local_timer_interrupts
    250.00 ± 51%    -100.0%       0.00        interrupts.CPU35.NMI:Non-maskable_interrupts
    250.00 ± 51%    -100.0%       0.00        interrupts.CPU35.PMI:Performance_monitoring_interrupts
      1880 ± 13%     -63.5%     685.67 ± 12%  interrupts.CPU36.CAL:Function_call_interrupts
    284607 ± 20%     -88.3%      33236 ±  5%  interrupts.CPU36.LOC:Local_timer_interrupts
    317.50 ± 48%    -100.0%       0.00        interrupts.CPU36.NMI:Non-maskable_interrupts
    317.50 ± 48%    -100.0%       0.00        interrupts.CPU36.PMI:Performance_monitoring_interrupts
      1843 ±  7%     -65.9%     628.33 ± 15%  interrupts.CPU37.CAL:Function_call_interrupts
    284781 ± 20%     -88.3%      33420 ±  5%  interrupts.CPU37.LOC:Local_timer_interrupts
    710.25 ±118%    -100.0%       0.00        interrupts.CPU37.NMI:Non-maskable_interrupts
    710.25 ±118%    -100.0%       0.00        interrupts.CPU37.PMI:Performance_monitoring_interrupts
      1719 ± 12%     -63.7%     624.33 ± 14%  interrupts.CPU38.CAL:Function_call_interrupts
    284571 ± 20%     -88.3%      33395 ±  6%  interrupts.CPU38.LOC:Local_timer_interrupts
    186.00 ± 29%    -100.0%       0.00        interrupts.CPU38.NMI:Non-maskable_interrupts
    186.00 ± 29%    -100.0%       0.00        interrupts.CPU38.PMI:Performance_monitoring_interrupts
      1810 ± 10%     -65.4%     626.00 ±  8%  interrupts.CPU39.CAL:Function_call_interrupts
    284611 ± 20%     -88.3%      33268 ±  6%  interrupts.CPU39.LOC:Local_timer_interrupts
    546.25 ±117%     -99.9%       0.67 ± 70%  interrupts.CPU39.NMI:Non-maskable_interrupts
    546.25 ±117%     -99.9%       0.67 ± 70%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
    105.25 ± 20%     -73.1%      28.33 ± 94%  interrupts.CPU4.71:PCI-MSI.12589060-edge.eth3-TxRx-3
      1886 ± 20%     -51.6%     914.00 ± 28%  interrupts.CPU4.CAL:Function_call_interrupts
    284385 ± 20%     -88.1%      33728 ±  5%  interrupts.CPU4.LOC:Local_timer_interrupts
    542.50 ±117%    -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
    542.50 ±117%    -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
      1793 ±  7%     -65.7%     614.33 ± 12%  interrupts.CPU40.CAL:Function_call_interrupts
    284529 ± 20%     -88.3%      33390 ±  6%  interrupts.CPU40.LOC:Local_timer_interrupts
    797.00 ±125%    -100.0%       0.00        interrupts.CPU40.NMI:Non-maskable_interrupts
    797.00 ±125%    -100.0%       0.00        interrupts.CPU40.PMI:Performance_monitoring_interrupts
    416.50 ±163%     -97.8%       9.33 ±111%  interrupts.CPU40.RES:Rescheduling_interrupts
      1847 ± 15%     -63.7%     669.67 ± 22%  interrupts.CPU41.CAL:Function_call_interrupts
    284478 ± 20%     -88.3%      33379 ±  6%  interrupts.CPU41.LOC:Local_timer_interrupts
    949.25 ± 78%     -99.9%       0.67 ± 70%  interrupts.CPU41.NMI:Non-maskable_interrupts
    949.25 ± 78%     -99.9%       0.67 ± 70%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
      1773 ± 10%     -65.0%     621.00 ± 13%  interrupts.CPU42.CAL:Function_call_interrupts
    284344 ± 20%     -88.2%      33447 ±  6%  interrupts.CPU42.LOC:Local_timer_interrupts
    260.75 ± 30%    -100.0%       0.00        interrupts.CPU42.NMI:Non-maskable_interrupts
    260.75 ± 30%    -100.0%       0.00        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      1871 ± 17%     -63.7%     678.67 ± 25%  interrupts.CPU43.CAL:Function_call_interrupts
    284513 ± 20%     -88.2%      33432 ±  6%  interrupts.CPU43.LOC:Local_timer_interrupts
    751.50 ±135%    -100.0%       0.33 ±141%  interrupts.CPU43.NMI:Non-maskable_interrupts
    751.50 ±135%    -100.0%       0.33 ±141%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      1777 ± 13%     -62.5%     667.00 ± 22%  interrupts.CPU44.CAL:Function_call_interrupts
    284355 ± 20%     -88.3%      33284 ±  4%  interrupts.CPU44.LOC:Local_timer_interrupts
    312.00 ± 47%    -100.0%       0.00        interrupts.CPU44.NMI:Non-maskable_interrupts
    312.00 ± 47%    -100.0%       0.00        interrupts.CPU44.PMI:Performance_monitoring_interrupts
      1786 ±  9%     -68.7%     559.33        interrupts.CPU45.CAL:Function_call_interrupts
    284581 ± 20%     -88.2%      33582 ±  5%  interrupts.CPU45.LOC:Local_timer_interrupts
    314.25 ± 59%    -100.0%       0.00        interrupts.CPU45.NMI:Non-maskable_interrupts
    314.25 ± 59%    -100.0%       0.00        interrupts.CPU45.PMI:Performance_monitoring_interrupts
      4338 ±171%     -99.9%       4.00 ±124%  interrupts.CPU45.RES:Rescheduling_interrupts
      1764 ±  8%     -59.4%     715.67 ± 27%  interrupts.CPU46.CAL:Function_call_interrupts
    284541 ± 20%     -88.2%      33684 ±  5%  interrupts.CPU46.LOC:Local_timer_interrupts
    195.75 ± 41%     -99.7%       0.67 ±141%  interrupts.CPU46.NMI:Non-maskable_interrupts
    195.75 ± 41%     -99.7%       0.67 ±141%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      1664 ± 16%     -58.6%     689.33 ± 19%  interrupts.CPU47.CAL:Function_call_interrupts
    284290 ± 20%     -88.1%      33829 ±  5%  interrupts.CPU47.LOC:Local_timer_interrupts
    180.75 ± 30%     -98.7%       2.33 ±141%  interrupts.CPU47.NMI:Non-maskable_interrupts
    180.75 ± 30%     -98.7%       2.33 ±141%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      1661 ± 16%     -58.9%     683.00 ± 25%  interrupts.CPU48.CAL:Function_call_interrupts
    284400 ± 20%     -88.2%      33626 ±  6%  interrupts.CPU48.LOC:Local_timer_interrupts
    196.00 ± 30%    -100.0%       0.00        interrupts.CPU48.NMI:Non-maskable_interrupts
    196.00 ± 30%    -100.0%       0.00        interrupts.CPU48.PMI:Performance_monitoring_interrupts
      1729 ±  9%     -64.4%     615.33 ± 12%  interrupts.CPU49.CAL:Function_call_interrupts
    284324 ± 20%     -88.1%      33754 ±  5%  interrupts.CPU49.LOC:Local_timer_interrupts
    218.50 ± 15%    -100.0%       0.00        interrupts.CPU49.NMI:Non-maskable_interrupts
    218.50 ± 15%    -100.0%       0.00        interrupts.CPU49.PMI:Performance_monitoring_interrupts
    264.00 ±106%     -94.4%      14.67 ± 17%  interrupts.CPU5.72:PCI-MSI.12589061-edge.eth3-TxRx-4
      1872 ± 23%     -60.8%     734.00 ± 29%  interrupts.CPU5.CAL:Function_call_interrupts
    284343 ± 20%     -88.3%      33274 ±  5%  interrupts.CPU5.LOC:Local_timer_interrupts
    677.75 ± 72%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
    677.75 ± 72%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      1825 ±  6%     -61.4%     704.00 ± 28%  interrupts.CPU50.CAL:Function_call_interrupts
    284502 ± 20%     -88.2%      33464 ±  6%  interrupts.CPU50.LOC:Local_timer_interrupts
    230.75 ± 18%     -99.9%       0.33 ±141%  interrupts.CPU50.NMI:Non-maskable_interrupts
    230.75 ± 18%     -99.9%       0.33 ±141%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      1692 ± 18%     -59.6%     683.33 ± 24%  interrupts.CPU51.CAL:Function_call_interrupts
    284635 ± 20%     -88.2%      33671 ±  5%  interrupts.CPU51.LOC:Local_timer_interrupts
    171.25 ± 47%    -100.0%       0.00        interrupts.CPU51.NMI:Non-maskable_interrupts
    171.25 ± 47%    -100.0%       0.00        interrupts.CPU51.PMI:Performance_monitoring_interrupts
      1715 ± 10%     -59.1%     701.33 ± 25%  interrupts.CPU52.CAL:Function_call_interrupts
    284601 ± 20%     -88.3%      33394 ±  6%  interrupts.CPU52.LOC:Local_timer_interrupts
    198.50 ± 16%    -100.0%       0.00        interrupts.CPU52.NMI:Non-maskable_interrupts
    198.50 ± 16%    -100.0%       0.00        interrupts.CPU52.PMI:Performance_monitoring_interrupts
      1809 ±  6%     -63.9%     652.33 ± 14%  interrupts.CPU53.CAL:Function_call_interrupts
    284563 ± 20%     -88.2%      33441 ±  6%  interrupts.CPU53.LOC:Local_timer_interrupts
    482.25 ± 98%    -100.0%       0.00        interrupts.CPU53.NMI:Non-maskable_interrupts
    482.25 ± 98%    -100.0%       0.00        interrupts.CPU53.PMI:Performance_monitoring_interrupts
      3422 ± 43%     -81.1%     646.00 ±  7%  interrupts.CPU54.CAL:Function_call_interrupts
    284884 ± 20%     -88.3%      33384 ±  4%  interrupts.CPU54.LOC:Local_timer_interrupts
    307.75 ± 57%     -99.9%       0.33 ±141%  interrupts.CPU54.NMI:Non-maskable_interrupts
    307.75 ± 57%     -99.9%       0.33 ±141%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      3743 ± 40%     -83.7%     608.33 ± 14%  interrupts.CPU55.CAL:Function_call_interrupts
    284545 ± 20%     -88.3%      33392 ±  4%  interrupts.CPU55.LOC:Local_timer_interrupts
    396.25 ± 56%     -99.9%       0.33 ±141%  interrupts.CPU55.NMI:Non-maskable_interrupts
    396.25 ± 56%     -99.9%       0.33 ±141%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
     18.75 ± 63%  +10557.8%       1998 ±139%  interrupts.CPU55.RES:Rescheduling_interrupts
      2990 ± 40%     -77.7%     667.33 ± 11%  interrupts.CPU56.CAL:Function_call_interrupts
    284602 ± 20%     -88.3%      33389 ±  4%  interrupts.CPU56.LOC:Local_timer_interrupts
    468.75 ± 72%     -99.9%       0.67 ±141%  interrupts.CPU56.NMI:Non-maskable_interrupts
    468.75 ± 72%     -99.9%       0.67 ±141%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      3768 ± 41%     -82.1%     676.00 ± 11%  interrupts.CPU57.CAL:Function_call_interrupts
    284795 ± 20%     -88.3%      33329 ±  4%  interrupts.CPU57.LOC:Local_timer_interrupts
    221.50 ± 44%     -99.8%       0.33 ±141%  interrupts.CPU57.NMI:Non-maskable_interrupts
    221.50 ± 44%     -99.8%       0.33 ±141%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      2742 ± 33%     -76.8%     636.67 ±  9%  interrupts.CPU58.CAL:Function_call_interrupts
    284604 ± 20%     -88.2%      33549 ±  4%  interrupts.CPU58.LOC:Local_timer_interrupts
    226.00 ± 57%    -100.0%       0.00        interrupts.CPU58.NMI:Non-maskable_interrupts
    226.00 ± 57%    -100.0%       0.00        interrupts.CPU58.PMI:Performance_monitoring_interrupts
      2936 ± 35%     -74.8%     738.67 ± 19%  interrupts.CPU59.CAL:Function_call_interrupts
    284808 ± 20%     -88.2%      33465 ±  4%  interrupts.CPU59.LOC:Local_timer_interrupts
    281.00 ± 60%    -100.0%       0.00        interrupts.CPU59.NMI:Non-maskable_interrupts
    281.00 ± 60%    -100.0%       0.00        interrupts.CPU59.PMI:Performance_monitoring_interrupts
    246.25 ± 51%     -89.6%      25.67 ± 70%  interrupts.CPU6.73:PCI-MSI.12589062-edge.eth3-TxRx-5
      1803 ± 10%     -60.4%     714.33 ± 21%  interrupts.CPU6.CAL:Function_call_interrupts
    284294 ± 20%     -88.3%      33257 ±  5%  interrupts.CPU6.LOC:Local_timer_interrupts
    283.50 ± 57%    -100.0%       0.00        interrupts.CPU6.NMI:Non-maskable_interrupts
    283.50 ± 57%    -100.0%       0.00        interrupts.CPU6.PMI:Performance_monitoring_interrupts
      3451 ± 45%     -77.7%     770.33 ± 24%  interrupts.CPU60.CAL:Function_call_interrupts
    284660 ± 20%     -88.3%      33377 ±  4%  interrupts.CPU60.LOC:Local_timer_interrupts
    218.50 ± 46%     -99.8%       0.33 ±141%  interrupts.CPU60.NMI:Non-maskable_interrupts
    218.50 ± 46%     -99.8%       0.33 ±141%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      2634 ± 36%     -70.9%     767.33 ± 20%  interrupts.CPU61.CAL:Function_call_interrupts
    284804 ± 20%     -88.3%      33392 ±  4%  interrupts.CPU61.LOC:Local_timer_interrupts
    177.50 ± 48%     -99.8%       0.33 ±141%  interrupts.CPU61.NMI:Non-maskable_interrupts
    177.50 ± 48%     -99.8%       0.33 ±141%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      2755 ± 45%     -72.1%     768.67 ± 19%  interrupts.CPU62.CAL:Function_call_interrupts
    284635 ± 20%     -88.3%      33372 ±  4%  interrupts.CPU62.LOC:Local_timer_interrupts
    208.50 ± 56%    -100.0%       0.00        interrupts.CPU62.NMI:Non-maskable_interrupts
    208.50 ± 56%    -100.0%       0.00        interrupts.CPU62.PMI:Performance_monitoring_interrupts
      3960 ± 43%     -82.3%     702.67 ± 15%  interrupts.CPU63.CAL:Function_call_interrupts
    284765 ± 20%     -88.3%      33420 ±  4%  interrupts.CPU63.LOC:Local_timer_interrupts
    288.00 ± 65%    -100.0%       0.00        interrupts.CPU63.NMI:Non-maskable_interrupts
    288.00 ± 65%    -100.0%       0.00        interrupts.CPU63.PMI:Performance_monitoring_interrupts
      4003 ± 41%     -82.2%     713.33 ± 18%  interrupts.CPU64.CAL:Function_call_interrupts
    284828 ± 20%     -88.3%      33462 ±  5%  interrupts.CPU64.LOC:Local_timer_interrupts
    190.25 ± 50%    -100.0%       0.00        interrupts.CPU64.NMI:Non-maskable_interrupts
    190.25 ± 50%    -100.0%       0.00        interrupts.CPU64.PMI:Performance_monitoring_interrupts
    284682 ± 20%     -88.2%      33648 ±  5%  interrupts.CPU65.LOC:Local_timer_interrupts
    200.50 ± 65%    -100.0%       0.00        interrupts.CPU65.NMI:Non-maskable_interrupts
    200.50 ± 65%    -100.0%       0.00        interrupts.CPU65.PMI:Performance_monitoring_interrupts
      3508 ± 37%     -78.8%     744.67 ± 17%  interrupts.CPU66.CAL:Function_call_interrupts
    284752 ± 20%     -88.2%      33619 ±  5%  interrupts.CPU66.LOC:Local_timer_interrupts
    232.75 ± 50%    -100.0%       0.00        interrupts.CPU66.NMI:Non-maskable_interrupts
    232.75 ± 50%    -100.0%       0.00        interrupts.CPU66.PMI:Performance_monitoring_interrupts
      3081 ± 40%     -75.3%     760.00 ± 20%  interrupts.CPU67.CAL:Function_call_interrupts
    284736 ± 20%     -88.3%      33398 ±  4%  interrupts.CPU67.LOC:Local_timer_interrupts
    259.50 ± 53%    -100.0%       0.00        interrupts.CPU67.NMI:Non-maskable_interrupts
    259.50 ± 53%    -100.0%       0.00        interrupts.CPU67.PMI:Performance_monitoring_interrupts
      3458 ± 37%     -77.6%     773.67 ± 20%  interrupts.CPU68.CAL:Function_call_interrupts
    284880 ± 20%     -88.3%      33363 ±  4%  interrupts.CPU68.LOC:Local_timer_interrupts
    262.25 ± 55%    -100.0%       0.00        interrupts.CPU68.NMI:Non-maskable_interrupts
    262.25 ± 55%    -100.0%       0.00        interrupts.CPU68.PMI:Performance_monitoring_interrupts
      3247 ± 36%     -76.5%     764.33 ± 16%  interrupts.CPU69.CAL:Function_call_interrupts
    284772 ± 20%     -88.4%      33084 ±  6%  interrupts.CPU69.LOC:Local_timer_interrupts
    259.25 ± 45%     -99.9%       0.33 ±141%  interrupts.CPU69.NMI:Non-maskable_interrupts
    259.25 ± 45%     -99.9%       0.33 ±141%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
    135.75 ± 30%     -82.6%      23.67 ± 78%  interrupts.CPU7.74:PCI-MSI.12589063-edge.eth3-TxRx-6
      1784 ± 25%     -65.2%     621.00 ±  9%  interrupts.CPU7.CAL:Function_call_interrupts
    284388 ± 20%     -88.3%      33183 ±  5%  interrupts.CPU7.LOC:Local_timer_interrupts
    213.75 ± 68%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
    213.75 ± 68%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
      3907 ± 41%     -80.3%     769.33 ± 19%  interrupts.CPU70.CAL:Function_call_interrupts
    284422 ± 20%     -88.3%      33403 ±  4%  interrupts.CPU70.LOC:Local_timer_interrupts
    240.25 ± 47%     -99.9%       0.33 ±141%  interrupts.CPU70.NMI:Non-maskable_interrupts
    240.25 ± 47%     -99.9%       0.33 ±141%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
      6753 ± 35%     -87.2%     867.33 ±  5%  interrupts.CPU71.CAL:Function_call_interrupts
    284231 ± 20%     -88.3%      33367 ±  4%  interrupts.CPU71.LOC:Local_timer_interrupts
    303.25 ± 58%     -99.9%       0.33 ±141%  interrupts.CPU71.NMI:Non-maskable_interrupts
    303.25 ± 58%     -99.9%       0.33 ±141%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      3778 ±154%     -91.5%     319.33 ± 13%  interrupts.CPU71.TLB:TLB_shootdowns
      1741 ±  7%     -66.4%     585.33 ±  5%  interrupts.CPU8.CAL:Function_call_interrupts
    284450 ± 20%     -88.1%      33716 ±  4%  interrupts.CPU8.LOC:Local_timer_interrupts
    242.00 ± 46%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
    242.00 ± 46%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      1827 ±  7%     -61.9%     696.67 ±  7%  interrupts.CPU9.CAL:Function_call_interrupts
    284698 ± 20%     -88.1%      33873 ±  5%  interrupts.CPU9.LOC:Local_timer_interrupts
    242.50 ± 48%    -100.0%       0.00        interrupts.CPU9.NMI:Non-maskable_interrupts
    242.50 ± 48%    -100.0%       0.00        interrupts.CPU9.PMI:Performance_monitoring_interrupts
  20493273 ± 20%     -88.2%    2410789 ±  5%  interrupts.LOC:Local_timer_interrupts
     23214 ± 24%    -100.0%      10.00 ± 37%  interrupts.NMI:Non-maskable_interrupts
     23214 ± 24%    -100.0%      10.00 ± 37%  interrupts.PMI:Performance_monitoring_interrupts
     42829 ± 45%     -66.4%      14402 ± 11%  interrupts.RES:Rescheduling_interrupts


                                                                                
                               vm-scalability.throughput                        
                                                                                
    4e+07 +-+---------------------------------------------------------------+   
          |           O           O              O              O           |   
  3.5e+07 O-+                   O      O  O O  O    O O    O O       O O    |   
    3e+07 +-+                                                               |   
          |                                                                 |   
  2.5e+07 +-+                                                               |   
          |                                                                 |   
    2e+07 +-+                                    +    +                     |   
          |                            +..       ::   ::                    |   
  1.5e+07 +-+                         :         :  : : :                    |   
    1e+07 +-+      .+     .+..O       :   +.    :  : :  O      .+.O         |   
          |      +.  +  .+       .+..+      +   :   +   +..+.+.   +..+.+..+.|   
    5e+06 +-+   :     +.      +.+            +  :                           |   
          |     :                             +:                            |   
        0 +-O--O-O--O----O-O---------O--------------------------------------+   
                                                                                
                                                                                                                                                                
                          vm-scalability.time.system_time                       
                                                                                
  250 +-+-----------------------------------------------------------O-------+   
      O            O                 O    O    O  O  O    O O          O    |   
      |                      O  O       O    O                              |   
  200 +-+                                                      O            |   
      |                                                                     |   
      |                                                                     |   
  150 +-+          +..                                      +..             |   
      |          ..        +.+..+..                        :     .+.        |   
  100 +-+     +.+     +   +                                :   +.   +..+.+..|   
      |       :        + +         +.    .+    +..+..+.+..+                 |   
      |      :          +            +..+  :   :                            |   
   50 +-+    :                             :  :                             |   
      |     :                               : :                             |   
      |     :                               ::                              |   
    0 +-+O-O--O-O-----O-O--O-------O-------------------O----------O---------+   
                                                                                
                                                                                                                                                                
                  vm-scalability.time.percent_of_cpu_this_job_got               
                                                                                
  2500 +-+------------------------------------------------------------------+   
       |            O         O                      O                      |   
       O                         O    O O  O O  O O       O  O O    O  O    |   
  2000 +-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
  1500 +-+                                                                  |   
       |                                                                    |   
  1000 +-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
   500 +-+                            +.        +    +..                    |   
       |        .+..    .+.         ..  +..    + + ..    .+..               |   
       |      .+    +.+.   +..+..+.+       +. +   +     +    +.+..+.+..+.+..|   
     0 +-+O-O--O-O----O--O-O-------O--------------------O---------O---------+   
                                                                                
                                                                                                                                                                
                                 vm-scalability.median                          
                                                                                
  1.2e+06 +-+---------------------------------------------------------------+   
          |                                                                 |   
    1e+06 +-+                     O    O  O O  O O              O           |   
          O           O         O                   O O    O         O O    |   
          |                                                  O              |   
   800000 +-+                                                               |   
          |                                                                 |   
   600000 +-+                                                               |   
          |                                                                 |   
   400000 +-+                          +..                                  |   
          |                           :          +..  +                     |   
          |                           :   +.     :   + +                    |   
   200000 +-+      .+.    .+..      .+      +.. :   +   +..+.  .+.         .|   
          |     .+.   +..+    O.+.+. O          :            +.   +..+.+..+ |   
        0 +-O--O-O--O----O-O----------------------------O---------O---------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.0.0-rc4-00073-g6861428" of type "text/plain" (191763 bytes)

View attachment "job-script" of type "text/plain" (7765 bytes)

View attachment "job.yaml" of type "text/plain" (5390 bytes)

View attachment "reproduce" of type "text/plain" (933 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ