lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [day] [month] [year] [list]
Date:   Sun, 6 Oct 2019 19:48:32 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Thomas Zimmermann <tzimmermann@...e.de>
Cc:     Noralf Trønnes <noralf@...nnes.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [drm/ast]  8a99de3dc0:  vm-scalability.median -4.9% regression

Greeting,

FYI, we noticed a -4.9% regression of vm-scalability.median due to commit:


commit: 8a99de3dc0dd417e1f46c8e27e63b1363ecfb969 ("drm/ast: Replace struct ast_fbdev with generic framebuffer emulation")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: vm-scalability
on test machine: 104 threads Skylake with 192G memory
with following parameters:

	runtime: 300s
	size: 8T
	test: anon-cow-seq
	cpufreq_governor: performance

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------+
| testcase: change | vm-scalability: vm-scalability.throughput 11.5% improvement               |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | runtime=300s                                                              |
|                  | size=8T                                                                   |
|                  | test=anon-w-seq                                                           |
+------------------+---------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/8T/lkp-skl-fpga01/anon-cow-seq/vm-scalability

commit: 
  f458579ede ("drm/fb-helper: Instanciate shadow FB if configured in device's mode_config")
  8a99de3dc0 ("drm/ast: Replace struct ast_fbdev with generic framebuffer emulation")

f458579ede0b7214 8a99de3dc0dd417e1f46c8e27e6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    443739            -4.9%     422142        vm-scalability.median
      0.02 ± 14%    +104.7%       0.04 ±  9%  vm-scalability.median_stddev
  45979363            -2.1%   44993698        vm-scalability.throughput
    127074 ±  2%     +32.5%     168384        vm-scalability.time.involuntary_context_switches
  13414165            -4.1%   12868046        vm-scalability.time.minor_page_faults
      8075            -4.2%       7734        vm-scalability.time.percent_of_cpu_this_job_got
     13243            -2.8%      12870        vm-scalability.time.system_time
     11163            -6.1%      10484        vm-scalability.time.user_time
    679818 ±  3%     -12.1%     597397 ±  2%  vm-scalability.time.voluntary_context_switches
 1.244e+10            -4.6%  1.187e+10        vm-scalability.workload
   3285339 ± 27%     -26.9%    2401527 ±  6%  cpuidle.POLL.time
     22220 ± 32%     -31.1%      15303 ± 19%  numa-numastat.node0.other_node
      8012            -9.8%       7228 ±  2%  vmstat.system.cs
    122706 ± 54%     -66.8%      40696 ±167%  numa-meminfo.node0.PageTables
    116284 ± 10%     -19.0%      94212 ± 16%  numa-meminfo.node0.SUnreclaim
     30801 ± 54%     -67.3%      10081 ±167%  numa-vmstat.node0.nr_page_table_pages
     29077 ± 10%     -19.0%      23551 ± 16%  numa-vmstat.node0.nr_slab_unreclaimable
     22.73 ±  4%     +19.4%      27.13 ± 11%  turbostat.CPU%c1
    141.37 ± 13%     -39.4      101.94 ± 24%  turbostat.PKG_%
     14081 ±  3%     +16.0%      16329        slabinfo.cred_jar.active_objs
     14094 ±  3%     +15.9%      16335        slabinfo.cred_jar.num_objs
    763.00 ±  5%     +12.0%     854.50 ±  5%  slabinfo.file_lock_cache.active_objs
    763.00 ±  5%     +12.0%     854.50 ±  5%  slabinfo.file_lock_cache.num_objs
     18285           +14.4%      20917 ±  2%  slabinfo.vmap_area.active_objs
    292.75            -3.1%     283.75        proc-vmstat.nr_dirtied
    873792 ±  3%      +7.6%     940363        proc-vmstat.numa_hint_faults_local
  18525216            -4.4%   17706943        proc-vmstat.numa_hit
   5587712            -6.0%    5251014        proc-vmstat.numa_huge_pte_updates
  18491148            -4.4%   17673004        proc-vmstat.numa_local
    747370 ±  7%     -11.2%     663547        proc-vmstat.numa_pages_migrated
 2.862e+09            -6.0%   2.69e+09        proc-vmstat.numa_pte_updates
 2.952e+09            -5.2%    2.8e+09        proc-vmstat.pgalloc_normal
  14208792            -3.6%   13695995        proc-vmstat.pgfault
 2.942e+09            -4.9%    2.8e+09        proc-vmstat.pgfree
 1.552e+08 ±  4%     -16.0%  1.303e+08 ±  7%  proc-vmstat.pgmigrate_fail
    747370 ±  7%     -11.2%     663547        proc-vmstat.pgmigrate_success
   5418839            -4.2%    5189265        proc-vmstat.thp_deferred_split_page
   5436250            -4.6%    5188033        proc-vmstat.thp_fault_alloc
      8014            -9.9%       7218 ±  2%  perf-stat.i.context-switches
    277.91 ±  2%      -9.6%     251.31 ±  5%  perf-stat.i.cpu-migrations
   1238959 ±  3%      -7.4%    1147280 ±  4%  perf-stat.i.dTLB-load-misses
    380964 ±  2%      -3.5%     367548        perf-stat.i.dTLB-store-misses
     40.03 ±  3%      -2.4       37.65 ±  4%  perf-stat.i.iTLB-load-miss-rate%
    646974 ±  5%     +13.0%     730798 ±  4%  perf-stat.i.iTLB-loads
  46335754 ±  8%     +14.9%   53225704 ±  4%  perf-stat.i.node-load-misses
     62.29            +3.0       65.29        perf-stat.overall.cache-miss-rate%
    267.20            -4.3%     255.58        perf-stat.overall.cycles-between-cache-misses
     54.90            +1.9       56.77        perf-stat.overall.node-load-miss-rate%
      7983            -9.8%       7198 ±  2%  perf-stat.ps.context-switches
    277.14 ±  2%      -9.5%     250.79 ±  5%  perf-stat.ps.cpu-migrations
   1234115 ±  3%      -7.4%    1142985 ±  4%  perf-stat.ps.dTLB-load-misses
    379584 ±  2%      -3.5%     366465        perf-stat.ps.dTLB-store-misses
    645205 ±  5%     +13.1%     729425 ±  4%  perf-stat.ps.iTLB-loads
  46174717 ±  8%     +14.9%   53032255 ±  4%  perf-stat.ps.node-load-misses
 4.039e+13            -4.5%  3.857e+13        perf-stat.total.instructions
      7845 ±  9%     +69.0%      13257 ± 11%  sched_debug.cfs_rq:/.exec_clock.stddev
     10135 ± 14%     +31.4%      13320 ±  7%  sched_debug.cfs_rq:/.load.avg
    197996 ± 85%    +135.4%     466130 ± 15%  sched_debug.cfs_rq:/.load.max
     21538 ± 74%    +123.4%      48115 ± 19%  sched_debug.cfs_rq:/.load.stddev
    357.45 ± 17%     +36.5%     487.86 ±  7%  sched_debug.cfs_rq:/.load_avg.max
     47.06 ± 15%     +25.1%      58.88 ± 10%  sched_debug.cfs_rq:/.load_avg.stddev
    447862 ± 10%     +79.8%     805104 ± 10%  sched_debug.cfs_rq:/.min_vruntime.stddev
     62.49 ±  9%     -83.0%      10.64 ±  6%  sched_debug.cfs_rq:/.nr_spread_over.avg
    483.37 ± 17%     -92.2%      37.85 ±  6%  sched_debug.cfs_rq:/.nr_spread_over.max
    100.66 ±  9%     -89.3%      10.79 ±  7%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      8.08 ±  6%     +32.8%      10.73 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    104.28 ± 69%    +209.5%     322.71 ± 19%  sched_debug.cfs_rq:/.runnable_load_avg.max
     12.20 ± 66%    +180.4%      34.20 ± 22%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     10073 ± 15%     +31.7%      13267 ±  7%  sched_debug.cfs_rq:/.runnable_weight.avg
    195732 ± 86%    +137.5%     464835 ± 15%  sched_debug.cfs_rq:/.runnable_weight.max
     21309 ± 75%    +125.1%      47960 ± 19%  sched_debug.cfs_rq:/.runnable_weight.stddev
    599980 ± 34%    +121.8%    1330620 ± 26%  sched_debug.cfs_rq:/.spread0.max
    448195 ± 10%     +79.5%     804638 ± 10%  sched_debug.cfs_rq:/.spread0.stddev
      7.91 ±  6%     +33.4%      10.55 ±  7%  sched_debug.cpu.cpu_load[0].avg
     94.75 ± 77%    +241.4%     323.46 ± 19%  sched_debug.cpu.cpu_load[0].max
     11.25 ± 71%    +197.4%      33.46 ± 21%  sched_debug.cpu.cpu_load[0].stddev
      8.15 ±  6%     +29.6%      10.57 ±  6%  sched_debug.cpu.cpu_load[1].avg
    110.83 ± 74%    +190.7%     322.18 ± 18%  sched_debug.cpu.cpu_load[1].max
     12.10 ± 71%    +173.4%      33.07 ± 20%  sched_debug.cpu.cpu_load[1].stddev
      8.23 ±  5%     +29.6%      10.67 ±  5%  sched_debug.cpu.cpu_load[2].avg
    116.93 ± 68%    +177.4%     324.40 ± 16%  sched_debug.cpu.cpu_load[2].max
     12.40 ± 66%    +168.5%      33.29 ± 19%  sched_debug.cpu.cpu_load[2].stddev
      8.19 ±  3%     +31.0%      10.74 ±  5%  sched_debug.cpu.cpu_load[3].avg
    114.92 ± 62%    +184.2%     326.64 ± 17%  sched_debug.cpu.cpu_load[3].max
     11.91 ± 60%    +183.4%      33.76 ± 20%  sched_debug.cpu.cpu_load[3].stddev
      8.04 ±  2%     +33.9%      10.77 ±  4%  sched_debug.cpu.cpu_load[4].avg
    115.38 ± 53%    +200.9%     347.13 ± 16%  sched_debug.cpu.cpu_load[4].max
     11.87 ± 52%    +199.7%      35.58 ± 19%  sched_debug.cpu.cpu_load[4].stddev
     10027 ± 15%     +29.3%      12967 ±  5%  sched_debug.cpu.load.avg
    197996 ± 85%    +135.4%     466130 ± 15%  sched_debug.cpu.load.max
     21214 ± 75%    +122.2%      47147 ± 19%  sched_debug.cpu.load.stddev
      0.01 ± 39%     -62.4%       0.00 ± 38%  sched_debug.cpu.nr_uninterruptible.avg
      0.00            +0.6        0.58 ±  7%  perf-profile.calltrace.cycles-pp.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +0.6        0.58 ±  8%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.00            +0.6        0.58 ±  8%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +0.6        0.59 ±  8%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +0.6        0.59 ±  8%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      1.13 ± 22%      +1.1        2.21 ± 21%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.38 ± 18%      +1.1        2.49 ± 20%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      1.40 ± 18%      +1.1        2.52 ± 20%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.52 ± 17%      +1.1        2.66 ± 19%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.52 ± 17%      +1.1        2.66 ± 19%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      1.52 ± 17%      +1.1        2.66 ± 19%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      1.54 ± 17%      +1.2        2.70 ± 19%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      1.10 ±  4%      -0.1        0.99 ±  4%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.08 ±  4%      -0.1        0.97 ±  4%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.34 ±  5%      -0.1        0.28 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.39 ±  4%      -0.1        0.33 ±  5%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.09 ±  5%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.wake_up_page_bit
      0.08 ±  5%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__wake_up_common
      0.08 ±  5%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.09 ±  5%      -0.0        0.07        perf-profile.children.cycles-pp.try_to_wake_up
      0.08 ± 10%      +0.0        0.11 ± 19%  perf-profile.children.cycles-pp.menu_select
      0.35 ±  8%      +0.0        0.40 ±  6%  perf-profile.children.cycles-pp.release_pages
      0.33 ± 10%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.ksys_write
      0.34 ±  8%      +0.1        0.40 ±  7%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.33 ± 12%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.vfs_write
      0.31 ±  9%      +0.1        0.36 ±  4%  perf-profile.children.cycles-pp.console_unlock
      0.24 ± 12%      +0.1        0.29 ±  3%  perf-profile.children.cycles-pp.uart_console_write
      0.26 ± 12%      +0.1        0.32 ±  4%  perf-profile.children.cycles-pp.serial8250_console_write
      0.32 ± 11%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.vprintk_emit
      0.20 ± 11%      +0.1        0.26 ±  4%  perf-profile.children.cycles-pp.io_serial_in
      1.00 ±  3%      +0.1        1.13 ±  3%  perf-profile.children.cycles-pp.do_syscall_64
      1.00 ±  3%      +0.1        1.14 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.58 ±  7%  perf-profile.children.cycles-pp.memcpy_erms
      0.00            +0.6        0.58 ±  7%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      0.00            +0.6        0.58 ±  8%  perf-profile.children.cycles-pp.worker_thread
      0.00            +0.6        0.58 ±  8%  perf-profile.children.cycles-pp.process_one_work
      0.00            +0.6        0.59 ±  8%  perf-profile.children.cycles-pp.kthread
      0.00            +0.6        0.60 ±  8%  perf-profile.children.cycles-pp.ret_from_fork
      1.15 ± 23%      +1.1        2.24 ± 21%  perf-profile.children.cycles-pp.intel_idle
      1.42 ± 18%      +1.1        2.55 ± 19%  perf-profile.children.cycles-pp.cpuidle_enter
      1.42 ± 18%      +1.1        2.55 ± 19%  perf-profile.children.cycles-pp.cpuidle_enter_state
      1.52 ± 17%      +1.1        2.66 ± 19%  perf-profile.children.cycles-pp.start_secondary
      1.54 ± 17%      +1.2        2.70 ± 19%  perf-profile.children.cycles-pp.do_idle
      1.54 ± 17%      +1.2        2.70 ± 19%  perf-profile.children.cycles-pp.secondary_startup_64
      1.54 ± 17%      +1.2        2.70 ± 19%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.31 ± 26%      -0.1        0.23 ±  8%  perf-profile.self.cycles-pp.copy_subpage
      0.39 ±  4%      -0.1        0.33 ±  5%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.20 ± 11%      +0.1        0.26 ±  4%  perf-profile.self.cycles-pp.io_serial_in
      0.00            +0.6        0.57 ±  8%  perf-profile.self.cycles-pp.memcpy_erms
      1.15 ± 23%      +1.1        2.24 ± 21%  perf-profile.self.cycles-pp.intel_idle
      1201 ± 17%     -40.9%     709.75 ±  9%  interrupts.CPU100.RES:Rescheduling_interrupts
      1120 ± 20%     -35.7%     720.00 ± 17%  interrupts.CPU101.RES:Rescheduling_interrupts
      5730 ±  6%     -27.4%       4162 ± 23%  interrupts.CPU102.NMI:Non-maskable_interrupts
      5730 ±  6%     -27.4%       4162 ± 23%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
      5946 ±  8%     -13.7%       5130 ± 10%  interrupts.CPU18.NMI:Non-maskable_interrupts
      5946 ±  8%     -13.7%       5130 ± 10%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
      7.75 ± 48%  +19019.4%       1481 ±104%  interrupts.CPU19.TLB:TLB_shootdowns
     31.25 ±147%   +3864.0%       1238 ±101%  interrupts.CPU21.TLB:TLB_shootdowns
    137.75 ±143%   +1319.8%       1955 ±112%  interrupts.CPU23.TLB:TLB_shootdowns
      5867 ±  8%     -15.7%       4944 ± 14%  interrupts.CPU24.NMI:Non-maskable_interrupts
      5867 ±  8%     -15.7%       4944 ± 14%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
     69.25 ±157%   +2356.3%       1701 ± 84%  interrupts.CPU24.TLB:TLB_shootdowns
      5923 ±  8%     -16.6%       4938 ± 15%  interrupts.CPU25.NMI:Non-maskable_interrupts
      5923 ±  8%     -16.6%       4938 ± 15%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      1354 ±  5%     -22.0%       1056 ± 17%  interrupts.CPU26.RES:Rescheduling_interrupts
      5915 ±  9%     -14.9%       5031 ± 14%  interrupts.CPU3.NMI:Non-maskable_interrupts
      5915 ±  9%     -14.9%       5031 ± 14%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
      1343 ± 21%     -43.7%     756.00 ±  8%  interrupts.CPU33.RES:Rescheduling_interrupts
      5866 ±  7%     -23.9%       4462 ± 29%  interrupts.CPU4.NMI:Non-maskable_interrupts
      5866 ±  7%     -23.9%       4462 ± 29%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    289.75 ±107%    +810.4%       2638 ± 84%  interrupts.CPU4.TLB:TLB_shootdowns
      4145 ± 28%     +43.0%       5928 ±  6%  interrupts.CPU40.NMI:Non-maskable_interrupts
      4145 ± 28%     +43.0%       5928 ±  6%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      1336 ± 14%     -37.0%     841.25 ± 11%  interrupts.CPU41.RES:Rescheduling_interrupts
      1282 ± 20%     -39.3%     778.25 ± 13%  interrupts.CPU45.RES:Rescheduling_interrupts
      1207 ± 12%     -37.0%     760.75 ± 20%  interrupts.CPU47.RES:Rescheduling_interrupts
      5946 ±  7%     -35.1%       3858 ± 41%  interrupts.CPU5.NMI:Non-maskable_interrupts
      5946 ±  7%     -35.1%       3858 ± 41%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      5943 ±  6%     -25.8%       4411 ± 28%  interrupts.CPU52.NMI:Non-maskable_interrupts
      5943 ±  6%     -25.8%       4411 ± 28%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      5922 ±  8%     -24.6%       4468 ± 29%  interrupts.CPU53.NMI:Non-maskable_interrupts
      5922 ±  8%     -24.6%       4468 ± 29%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
     61.25 ±155%   +6693.9%       4161 ±124%  interrupts.CPU53.TLB:TLB_shootdowns
      5911 ±  8%     -14.5%       5054 ± 13%  interrupts.CPU54.NMI:Non-maskable_interrupts
      5911 ±  8%     -14.5%       5054 ± 13%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
    108.75 ±159%   +1896.6%       2171 ± 96%  interrupts.CPU54.TLB:TLB_shootdowns
    253.75 ±140%    +750.3%       2157 ±101%  interrupts.CPU55.TLB:TLB_shootdowns
      5885 ±  8%     -13.5%       5088 ± 13%  interrupts.CPU56.NMI:Non-maskable_interrupts
      5885 ±  8%     -13.5%       5088 ± 13%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
     41.50 ±118%   +4375.9%       1857 ± 88%  interrupts.CPU56.TLB:TLB_shootdowns
      5896 ±  8%     -32.1%       4002 ± 42%  interrupts.CPU58.NMI:Non-maskable_interrupts
      5896 ±  8%     -32.1%       4002 ± 42%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
    151.00 ±161%    +550.3%     982.00 ±110%  interrupts.CPU58.TLB:TLB_shootdowns
      6081 ± 12%     -18.5%       4954 ± 14%  interrupts.CPU59.NMI:Non-maskable_interrupts
      6081 ± 12%     -18.5%       4954 ± 14%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
     18.00 ± 71%   +8488.9%       1546 ± 94%  interrupts.CPU61.TLB:TLB_shootdowns
     10.75 ±116%  +17397.7%       1881 ± 86%  interrupts.CPU62.TLB:TLB_shootdowns
    131.50 ±160%   +1435.2%       2018 ±100%  interrupts.CPU63.TLB:TLB_shootdowns
      5923 ±  8%     -15.9%       4979 ± 13%  interrupts.CPU64.NMI:Non-maskable_interrupts
      5923 ±  8%     -15.9%       4979 ± 13%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
      7.25 ± 61%  +18124.1%       1321 ±116%  interrupts.CPU64.TLB:TLB_shootdowns
      5909 ±  8%     -14.9%       5031 ± 14%  interrupts.CPU65.NMI:Non-maskable_interrupts
      5909 ±  8%     -14.9%       5031 ± 14%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
      5913 ±  8%     -13.0%       5147 ± 12%  interrupts.CPU68.NMI:Non-maskable_interrupts
      5913 ±  8%     -13.0%       5147 ± 12%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
     99.00 ±164%    +859.1%     949.50 ± 99%  interrupts.CPU68.TLB:TLB_shootdowns
      5935 ±  9%     -34.9%       3864 ± 42%  interrupts.CPU69.NMI:Non-maskable_interrupts
      5935 ±  9%     -34.9%       3864 ± 42%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
      6098 ± 11%     -26.0%       4510 ± 32%  interrupts.CPU7.NMI:Non-maskable_interrupts
      6098 ± 11%     -26.0%       4510 ± 32%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    132.50 ±153%   +1204.3%       1728 ± 87%  interrupts.CPU7.TLB:TLB_shootdowns
      5917 ±  9%     -24.8%       4450 ± 31%  interrupts.CPU70.NMI:Non-maskable_interrupts
      5917 ±  9%     -24.8%       4450 ± 31%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
      5881 ±  8%     -14.3%       5043 ± 13%  interrupts.CPU71.NMI:Non-maskable_interrupts
      5881 ±  8%     -14.3%       5043 ± 13%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      9.00 ± 93%  +19183.3%       1735 ± 70%  interrupts.CPU71.TLB:TLB_shootdowns
      5987 ±  7%     -17.1%       4964 ± 13%  interrupts.CPU72.NMI:Non-maskable_interrupts
      5987 ±  7%     -17.1%       4964 ± 13%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
      5938 ±  8%     -12.8%       5178 ± 13%  interrupts.CPU73.NMI:Non-maskable_interrupts
      5938 ±  8%     -12.8%       5178 ± 13%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
      8.75 ± 60%  +14048.6%       1238 ± 99%  interrupts.CPU73.TLB:TLB_shootdowns
      5957 ±  8%     -28.0%       4292 ± 31%  interrupts.CPU74.NMI:Non-maskable_interrupts
      5957 ±  8%     -28.0%       4292 ± 31%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
      5877 ±  8%     -13.5%       5086 ± 13%  interrupts.CPU76.NMI:Non-maskable_interrupts
      5877 ±  8%     -13.5%       5086 ± 13%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      5894 ±  9%     -16.1%       4944 ± 16%  interrupts.CPU77.NMI:Non-maskable_interrupts
      5894 ±  9%     -16.1%       4944 ± 16%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      1273 ± 20%     -43.0%     725.75 ± 19%  interrupts.CPU81.RES:Rescheduling_interrupts
      5932 ±  7%     -11.0%       5278 ± 11%  interrupts.CPU9.NMI:Non-maskable_interrupts
      5932 ±  7%     -11.0%       5278 ± 11%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      1157 ± 20%     -33.2%     773.25 ± 28%  interrupts.CPU92.RES:Rescheduling_interrupts
      1275 ± 13%     -35.6%     821.75 ± 23%  interrupts.CPU93.RES:Rescheduling_interrupts
      1388 ± 16%     -48.9%     709.25 ±  5%  interrupts.CPU94.RES:Rescheduling_interrupts
      1325 ±  6%     -40.3%     791.75 ± 20%  interrupts.CPU95.RES:Rescheduling_interrupts
      1172 ± 19%     -41.2%     689.00 ± 13%  interrupts.CPU97.RES:Rescheduling_interrupts
      1157 ± 18%     -43.2%     656.75 ± 15%  interrupts.CPU98.RES:Rescheduling_interrupts
     98.00 ±  8%     -10.5%      87.75 ±  9%  interrupts.IWI:IRQ_work_interrupts
    122039 ±  6%     -18.3%      99715 ±  3%  interrupts.RES:Rescheduling_interrupts
     28612 ±  7%     -27.1%      20844 ±  5%  softirqs.CPU0.RCU
     28418 ±  9%     -26.7%      20818 ±  4%  softirqs.CPU1.RCU
     29295 ±  9%     -24.7%      22051 ±  5%  softirqs.CPU10.RCU
     28659 ±  3%     -32.3%      19398 ±  2%  softirqs.CPU100.RCU
     28765 ±  2%     -32.5%      19408 ±  3%  softirqs.CPU101.RCU
     28957 ±  4%     -32.6%      19514 ±  3%  softirqs.CPU102.RCU
     28971 ±  4%     -32.7%      19490 ±  2%  softirqs.CPU103.RCU
     28919 ±  7%     -26.9%      21127 ±  5%  softirqs.CPU11.RCU
     29003 ±  7%     -27.3%      21074 ±  4%  softirqs.CPU12.RCU
     29170 ±  7%     -27.6%      21110 ±  6%  softirqs.CPU13.RCU
     28977 ±  6%     -28.1%      20845 ±  2%  softirqs.CPU14.RCU
     30540 ±  7%     -28.0%      21982 ±  4%  softirqs.CPU15.RCU
     30136 ±  7%     -25.9%      22331 ±  6%  softirqs.CPU16.RCU
     30602 ±  8%     -25.8%      22700 ±  5%  softirqs.CPU17.RCU
     30824 ±  6%     -27.9%      22215 ±  4%  softirqs.CPU18.RCU
     31594 ±  8%     -29.5%      22270 ±  5%  softirqs.CPU19.RCU
     29072 ±  7%     -26.3%      21430 ±  2%  softirqs.CPU2.RCU
     30322 ±  7%     -27.3%      22035 ±  4%  softirqs.CPU20.RCU
     30302 ±  7%     -27.6%      21936 ±  4%  softirqs.CPU21.RCU
     30518 ±  6%     -27.7%      22058 ±  4%  softirqs.CPU22.RCU
     30338 ±  7%     -26.3%      22362 ±  5%  softirqs.CPU23.RCU
     30235 ±  6%     -25.9%      22415 ±  4%  softirqs.CPU24.RCU
     31019 ±  8%     -28.4%      22209 ±  5%  softirqs.CPU25.RCU
     27942 ±  3%     -24.4%      21133 ±  6%  softirqs.CPU26.RCU
     27305 ±  2%     -26.2%      20160 ±  7%  softirqs.CPU27.RCU
     27118 ±  2%     -27.0%      19785 ±  4%  softirqs.CPU28.RCU
     28452 ±  6%     -31.3%      19546 ±  4%  softirqs.CPU29.RCU
     30999 ±  9%     -25.0%      23262 ± 11%  softirqs.CPU3.RCU
     29470 ±  4%     -29.2%      20857 ±  4%  softirqs.CPU30.RCU
     30095 ±  4%     -28.0%      21677 ±  5%  softirqs.CPU31.RCU
     30081 ±  4%     -32.1%      20414 ±  3%  softirqs.CPU32.RCU
     30545 ±  6%     -32.9%      20484 ±  4%  softirqs.CPU33.RCU
     29443 ±  5%     -30.0%      20612 ±  6%  softirqs.CPU34.RCU
     30357 ±  3%     -32.8%      20399 ±  2%  softirqs.CPU35.RCU
     29733 ±  2%     -32.1%      20177 ±  3%  softirqs.CPU36.RCU
     30365 ±  3%     -33.1%      20302 ±  2%  softirqs.CPU37.RCU
     30076 ±  3%     -32.2%      20396 ±  2%  softirqs.CPU38.RCU
     29966 ±  3%     -33.0%      20077 ±  2%  softirqs.CPU39.RCU
     28974 ±  7%     -28.0%      20871 ±  4%  softirqs.CPU4.RCU
     29434 ±  3%     -31.8%      20069 ±  2%  softirqs.CPU40.RCU
     30660 ± 10%     -34.6%      20056 ±  3%  softirqs.CPU41.RCU
     29952 ±  3%     -32.8%      20141 ±  3%  softirqs.CPU42.RCU
     31326 ±  6%     -35.5%      20195 ±  3%  softirqs.CPU43.RCU
     29590 ±  5%     -30.1%      20670 ±  3%  softirqs.CPU44.RCU
     29642 ±  3%     -32.2%      20088 ±  4%  softirqs.CPU45.RCU
     28965 ±  4%     -30.7%      20081 ±  3%  softirqs.CPU46.RCU
     29285 ±  4%     -29.5%      20639 ±  6%  softirqs.CPU47.RCU
     29120 ±  2%     -31.4%      19970 ±  4%  softirqs.CPU48.RCU
     29051 ±  2%     -31.0%      20050 ±  4%  softirqs.CPU49.RCU
     29755 ±  5%     -26.7%      21801 ± 10%  softirqs.CPU5.RCU
     29359 ±  2%     -31.6%      20076 ±  4%  softirqs.CPU50.RCU
     29089 ±  3%     -30.4%      20251 ±  3%  softirqs.CPU51.RCU
     29220 ±  7%     -27.2%      21284 ±  5%  softirqs.CPU52.RCU
     29302 ±  9%     -27.5%      21257 ±  6%  softirqs.CPU53.RCU
     29437 ±  8%     -28.6%      21027 ±  5%  softirqs.CPU54.RCU
     29037 ±  6%     -27.3%      21106 ±  4%  softirqs.CPU55.RCU
     29496 ±  8%     -30.7%      20447 ±  5%  softirqs.CPU56.RCU
     29759 ±  6%     -25.9%      22060 ± 13%  softirqs.CPU57.RCU
     29657 ±  8%     -29.0%      21058 ±  5%  softirqs.CPU58.RCU
     29285 ±  8%     -28.1%      21045 ±  5%  softirqs.CPU59.RCU
     29133 ±  6%     -28.2%      20919 ±  4%  softirqs.CPU6.RCU
     28950 ±  7%     -25.6%      21542 ±  5%  softirqs.CPU60.RCU
     29050 ±  7%     -21.0%      22936 ± 12%  softirqs.CPU61.RCU
     29033 ±  7%     -29.4%      20505 ±  3%  softirqs.CPU62.RCU
     28975 ±  6%     -28.8%      20635 ±  4%  softirqs.CPU63.RCU
     29172 ±  7%     -28.2%      20931 ±  3%  softirqs.CPU64.RCU
     29371 ±  8%     -29.9%      20582 ±  4%  softirqs.CPU65.RCU
     29182 ±  7%     -29.4%      20615 ±  4%  softirqs.CPU66.RCU
     29009 ±  7%     -27.9%      20905 ±  3%  softirqs.CPU67.RCU
     29030 ±  7%     -28.8%      20670 ±  3%  softirqs.CPU68.RCU
     29096 ±  7%     -27.9%      20965 ±  2%  softirqs.CPU69.RCU
     31133 ±  8%     -31.8%      21237 ±  6%  softirqs.CPU7.RCU
     29295 ±  8%     -29.7%      20599 ±  4%  softirqs.CPU70.RCU
     29212 ±  8%     -29.5%      20605 ±  4%  softirqs.CPU71.RCU
     29174 ±  8%     -28.9%      20735 ±  4%  softirqs.CPU72.RCU
     29015 ±  8%     -28.5%      20748 ±  4%  softirqs.CPU73.RCU
     29319 ±  8%     -30.1%      20490 ±  4%  softirqs.CPU74.RCU
     26434 ±  5%     -30.3%      18428 ±  4%  softirqs.CPU75.RCU
     26450 ±  6%     -30.0%      18521 ±  5%  softirqs.CPU76.RCU
     26526 ±  5%     -28.9%      18858 ±  5%  softirqs.CPU77.RCU
     30013 ±  3%     -31.1%      20665 ±  5%  softirqs.CPU78.RCU
     30161 ±  4%     -31.3%      20715 ±  5%  softirqs.CPU79.RCU
     29127 ±  7%     -28.6%      20796 ±  2%  softirqs.CPU8.RCU
     30869 ±  5%     -34.6%      20189 ±  3%  softirqs.CPU80.RCU
     30397 ±  3%     -31.5%      20828 ±  4%  softirqs.CPU81.RCU
     29630 ±  4%     -30.6%      20557 ±  4%  softirqs.CPU82.RCU
     30453 ±  3%     -32.6%      20518 ±  5%  softirqs.CPU83.RCU
     30353 ±  3%     -32.3%      20539 ±  3%  softirqs.CPU84.RCU
     29308 ±  5%     -30.2%      20458 ±  4%  softirqs.CPU85.RCU
     29752 ±  4%     -31.2%      20457 ±  3%  softirqs.CPU86.RCU
     30518 ±  3%     -32.9%      20491 ±  3%  softirqs.CPU87.RCU
     30185 ±  2%     -32.6%      20359 ±  4%  softirqs.CPU88.RCU
     30329 ±  3%     -32.6%      20445 ±  4%  softirqs.CPU89.RCU
     28785 ±  7%     -27.7%      20824 ±  4%  softirqs.CPU9.RCU
     28868 ±  3%     -28.3%      20690 ±  8%  softirqs.CPU90.RCU
     29066 ±  3%     -33.2%      19418 ±  2%  softirqs.CPU91.RCU
     28024 ±  3%     -31.0%      19350 ±  2%  softirqs.CPU92.RCU
     29079 ±  2%     -33.5%      19352 ±  2%  softirqs.CPU93.RCU
     28617 ±  3%     -31.2%      19696        softirqs.CPU94.RCU
     30527 ±  3%     -36.0%      19530 ±  3%  softirqs.CPU95.RCU
     28895 ±  2%     -31.7%      19738        softirqs.CPU96.RCU
     30157 ± 10%     -34.7%      19694 ±  3%  softirqs.CPU97.RCU
     28831 ±  3%     -32.7%      19389 ±  3%  softirqs.CPU98.RCU
     29039 ±  3%     -32.1%      19704 ±  4%  softirqs.CPU99.RCU
   3062198 ±  5%     -29.8%    2151034 ±  3%  softirqs.RCU


                                                                                
                            vm-scalability.time.user_time                       
                                                                                
  11400 +-+-----------------------------------------------------------------+   
  11300 +-+                                              +          +       |   
        |   + .+ .+   + +   +. ++    +  .+ +      +  +  + +.++     :        |   
  11200 +-+ :+  + :  + + :  : +  +  : ++  + + + .+ + ::+      +.++ :        |   
  11100 +-++       :+    : +      + :        + +    + +           +         |   
  11000 +-+        +      +        +                                        |   
  10900 +-+                                                                 |   
        |                                                                   |   
  10800 +-+                                                                 |   
  10700 +-+                                                                 |   
  10600 +-+ O                  O           O O O        OOO O               |   
  10500 +-+O O OO O O     OO    O                     O      O    O  O O    O   
        O          O   O    O O   OO OOO O    O  OO O         O OO  O   OO O|   
  10400 +O+           O O                 O          O                      |   
  10300 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                  vm-scalability.time.percent_of_cpu_this_job_got               
                                                                                
  8200 +-+------------------------------------------------------------------+   
  8150 +-+       +      +                  +           +.+ +.               |   
       |+.++.   : ++.+ + ++.+       +   +. :  + +.++.++  : : ++   +.        |   
  8100 +-+   ++ :     +      + .++.+ +.+  + :+ ::         :    + +  +       |   
  8050 +-+     +              +             +  +          +     +           |   
  8000 +-+                                                                  |   
  7950 +-+                                                                  |   
       |                                                                    |   
  7900 +-+                                                                  |   
  7850 +-+                                                                  |   
  7800 +-+                                                                  |   
  7750 +O+         O     O                         O         O       O  O   |   
       O       O     OO O O OO  OO OOO O   OO OOO        O O             O O|   
  7700 +-+OO OO  OO           O         O O       O  OOO  O   O OOO O O     O   
  7650 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.median                           
                                                                                
  450000 +-+----------------------------------------------------------------+   
         |                                                                  |   
  445000 +-+ + .++    +.     +    +   +.+    +.+++.        +    ++. +       |   
         |   :+   :   : +    :+.++ + +   :.+ :     +++.++ + + .+   +        |   
  440000 +-++     +.++   ++.+       +    +  +            +   +              |   
         |                                                                  |   
  435000 +-+                                                                |   
         |                                                                  |   
  430000 +-+                                                                |   
         |                                                                  |   
  425000 +-+     O  O       O                        O   O  O   O        O  O   
         O  O O   O  OO OOO   O OO  O O O  OOO OOO O    O  O O   O OOO     O|   
  420000 +O+ O  O            O    O  O   O          O          O       OO   |   
         |                                             O                    |   
  415000 +-+----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.workload                         
                                                                                
  1.27e+10 +-+--------------------------------------------------------------+   
           |                                                +               |   
  1.26e+10 +-+                                              :+              |   
  1.25e+10 +-+ +  + +   +  +   +     ++   +    +  ++.+++.++ : ++   ++       |   
           |:  ::+ :::  ::+ : + :    : +  :+   :+ :        :    :  :        |   
  1.24e+10 +-++ +  + +.+ +  ++  +++.+   ++  +++  +         +    +.+         |   
  1.23e+10 +-+                                                              |   
           |                                                                |   
  1.22e+10 +-+                                                              |   
  1.21e+10 +-+                                                              |   
           |                                                                |   
   1.2e+10 +-+                                                              |   
  1.19e+10 +-+     O   OO  OOO                            O O O       O  O  |   
           OO OOO O OO   O     OOOO OOO OOO OOOO OOO OOO O O   OO OOO  OO  OO   
  1.18e+10 +-+--------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2ap4: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/8T/lkp-csl-2ap4/anon-w-seq/vm-scalability

commit: 
  f458579ede ("drm/fb-helper: Instanciate shadow FB if configured in device's mode_config")
  8a99de3dc0 ("drm/ast: Replace struct ast_fbdev with generic framebuffer emulation")

f458579ede0b7214 8a99de3dc0dd417e1f46c8e27e6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.02 ±  3%      -9.7%       0.01 ±  4%  vm-scalability.free_time
      0.04 ± 15%     +32.2%       0.06 ± 15%  vm-scalability.median_stddev
  40612512           +11.5%   45286761 ±  2%  vm-scalability.throughput
    341832 ±  3%    +106.2%     705019 ± 23%  vm-scalability.time.involuntary_context_switches
     16292           -20.5%      12950 ±  9%  vm-scalability.time.percent_of_cpu_this_job_got
     23353           -20.6%      18546 ±  9%  vm-scalability.time.system_time
     25960           -20.6%      20604 ±  9%  vm-scalability.time.user_time
 1.065e+10           -10.4%  9.537e+09 ±  5%  vm-scalability.workload
      5596 ±  3%      +4.2%       5834        boot-time.idle
      7040 ± 11%     -20.2%       5621 ± 10%  numa-meminfo.node1.Mapped
     15.00 ± 16%    +106.7%      31.00 ± 21%  vmstat.cpu.id
     44.00 ±  2%     -19.9%      35.25 ±  9%  vmstat.cpu.us
     14.99 ± 16%     +16.3       31.30 ± 21%  mpstat.cpu.all.idle%
     40.34 ±  2%      -7.5       32.86 ±  9%  mpstat.cpu.all.sys%
     44.67 ±  2%      -8.8       35.83 ±  9%  mpstat.cpu.all.usr%
   5702492 ± 27%     -37.2%    3580311 ± 14%  numa-numastat.node2.local_node
   5727472 ± 27%     -37.2%    3594296 ± 14%  numa-numastat.node2.numa_hit
     24979 ± 25%     -44.0%      13988 ± 78%  numa-numastat.node2.other_node
   3187783 ± 26%     -36.1%    2037897 ± 12%  numa-vmstat.node2.numa_hit
   3079149 ± 26%     -37.0%    1939817 ± 12%  numa-vmstat.node2.numa_local
    108649 ±  5%      -9.7%      98083 ± 10%  numa-vmstat.node2.numa_other
 3.376e+09 ± 72%    +370.7%  1.589e+10 ± 23%  cpuidle.C1E.time
  11203489 ± 28%    +204.9%   34155602 ± 22%  cpuidle.C1E.usage
    142637 ± 28%    +332.5%     616893 ± 41%  cpuidle.POLL.time
     45901 ± 28%    +481.3%     266828 ± 39%  cpuidle.POLL.usage
      2635 ±  2%     -19.0%       2134 ±  9%  turbostat.Avg_MHz
     85.34 ±  2%     -16.1       69.20 ±  9%  turbostat.Busy%
  11194352 ± 28%    +205.2%   34164734 ± 22%  turbostat.C1E
      5.71 ± 72%     +21.5       27.24 ± 22%  turbostat.C1E%
     13.48 ± 19%    +123.2%      30.09 ± 21%  turbostat.CPU%c1
    515.38            -8.6%     471.31 ±  4%  turbostat.PkgWatt
  42581281 ±  3%     -17.6%   35067509 ±  6%  meminfo.Active
  42579673 ±  3%     -17.6%   35065901 ±  6%  meminfo.Active(anon)
  30535168 ±  3%     -16.4%   25541626 ±  4%  meminfo.AnonHugePages
  42280681 ±  3%     -17.7%   34800948 ±  6%  meminfo.AnonPages
      2473           +11.7%       2763 ±  5%  meminfo.Inactive(file)
  45041684 ±  3%     -16.7%   37498966 ±  5%  meminfo.Memused
     69737 ±  3%     -14.9%      59333 ±  4%  meminfo.PageTables
    302492 ±  3%     -13.5%     261669 ±  6%  meminfo.max_used_kB
      2065 ± 16%     +19.0%       2458 ±  4%  slabinfo.UNIX.active_objs
      2065 ± 16%     +19.0%       2458 ±  4%  slabinfo.UNIX.num_objs
     22849 ±  5%     +24.8%      28514        slabinfo.cred_jar.active_objs
    543.25 ±  5%     +24.9%     678.50        slabinfo.cred_jar.active_slabs
     22849 ±  5%     +24.8%      28514        slabinfo.cred_jar.num_objs
    543.25 ±  5%     +24.9%     678.50        slabinfo.cred_jar.num_slabs
     25887           +16.2%      30072 ±  3%  slabinfo.vmap_area.active_objs
     27392           +10.3%      30220 ±  3%  slabinfo.vmap_area.num_objs
  10520809           -17.5%    8678491 ±  5%  proc-vmstat.nr_active_anon
  10438636           -17.5%    8615390 ±  5%  proc-vmstat.nr_anon_pages
     14698           -15.8%      12371 ±  4%  proc-vmstat.nr_anon_transparent_hugepages
    401.50 ±  3%     +38.5%     556.00 ± 14%  proc-vmstat.nr_dirtied
   3804457            +4.9%    3990179        proc-vmstat.nr_dirty_background_threshold
   7621079            +4.9%    7990896        proc-vmstat.nr_dirty_threshold
  38296490            +4.8%   40145930        proc-vmstat.nr_free_pages
    618.00           +11.7%     690.50 ±  5%  proc-vmstat.nr_inactive_file
     17222           -14.4%      14740 ±  3%  proc-vmstat.nr_page_table_pages
     48514            -3.4%      46855 ±  2%  proc-vmstat.nr_shmem
     24938            +1.4%      25295        proc-vmstat.nr_slab_reclaimable
    371.75 ±  2%     +41.4%     525.75 ± 15%  proc-vmstat.nr_written
  10520792           -17.5%    8678481 ±  5%  proc-vmstat.nr_zone_active_anon
    618.00           +11.7%     690.50 ±  5%  proc-vmstat.nr_zone_inactive_file
    148112 ±  6%     -24.3%     112186 ± 19%  proc-vmstat.numa_hint_faults
     30689 ±  4%     -20.4%      24417 ±  8%  proc-vmstat.pgactivate
 2.372e+09           -10.4%  2.125e+09 ±  5%  proc-vmstat.pgalloc_normal
 2.361e+09           -10.1%  2.123e+09 ±  5%  proc-vmstat.pgfree
   4581091           -10.1%    4118160 ±  5%  proc-vmstat.thp_deferred_split_page
   4602140           -10.4%    4121775 ±  5%  proc-vmstat.thp_fault_alloc
 3.623e+10 ±  2%      -9.3%  3.288e+10 ±  5%  perf-stat.i.branch-instructions
     71.26            -2.5       68.76        perf-stat.i.cache-miss-rate%
 4.147e+08 ±  3%     -12.4%  3.634e+08 ±  7%  perf-stat.i.cache-misses
 5.767e+08 ±  2%      -9.0%  5.249e+08 ±  5%  perf-stat.i.cache-references
      4.18           -11.5%       3.70 ±  3%  perf-stat.i.cpi
 4.808e+11 ±  2%     -19.3%  3.881e+11 ±  8%  perf-stat.i.cpu-cycles
    266.65            +7.9%     287.60 ±  2%  perf-stat.i.cpu-migrations
      1163            -9.0%       1058        perf-stat.i.cycles-between-cache-misses
      0.00 ± 77%      -0.0        0.00 ± 12%  perf-stat.i.dTLB-load-miss-rate%
    404068 ± 21%     -35.3%     261419 ± 13%  perf-stat.i.dTLB-load-misses
 2.485e+10 ±  2%      -9.1%  2.259e+10 ±  5%  perf-stat.i.dTLB-loads
  8.46e+09 ±  2%      -8.4%  7.751e+09 ±  5%  perf-stat.i.dTLB-stores
     80.28 ±  2%     -10.8       69.49 ±  5%  perf-stat.i.iTLB-load-miss-rate%
   1818274 ±  3%     +14.5%    2081935 ±  6%  perf-stat.i.iTLB-load-misses
    538566 ± 16%     +88.3%    1014367 ± 18%  perf-stat.i.iTLB-loads
  1.14e+11 ±  2%      -9.0%  1.038e+11 ±  5%  perf-stat.i.instructions
     66086 ±  4%     -20.4%      52589 ± 13%  perf-stat.i.instructions-per-iTLB-miss
      0.24           +14.3%       0.28 ±  3%  perf-stat.i.ipc
     19.88 ±  7%      +2.8       22.70 ± 13%  perf-stat.i.node-load-miss-rate%
  1.23e+08 ±  2%     -13.4%  1.066e+08 ±  7%  perf-stat.i.node-stores
     71.98            -2.7       69.26        perf-stat.overall.cache-miss-rate%
      4.22           -11.4%       3.74 ±  3%  perf-stat.overall.cpi
      1160            -8.0%       1068        perf-stat.overall.cycles-between-cache-misses
      0.00 ± 21%      -0.0        0.00 ± 13%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  3%      +0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     77.56 ±  2%      -9.8       67.72 ±  4%  perf-stat.overall.iTLB-load-miss-rate%
     63169 ±  5%     -20.0%      50528 ± 12%  perf-stat.overall.instructions-per-iTLB-miss
      0.24           +13.0%       0.27 ±  3%  perf-stat.overall.ipc
     18.75 ±  3%      +4.5       23.29 ± 15%  perf-stat.overall.node-load-miss-rate%
 3.617e+10 ±  2%      -9.2%  3.284e+10 ±  5%  perf-stat.ps.branch-instructions
 4.143e+08 ±  3%     -12.3%  3.634e+08 ±  7%  perf-stat.ps.cache-misses
 5.755e+08 ±  2%      -8.9%  5.242e+08 ±  5%  perf-stat.ps.cache-references
 4.809e+11 ±  2%     -19.2%  3.885e+11 ±  8%  perf-stat.ps.cpu-cycles
    261.39 ±  2%      +8.7%     284.08 ±  2%  perf-stat.ps.cpu-migrations
    396350 ± 22%     -34.4%     260098 ± 13%  perf-stat.ps.dTLB-load-misses
 2.479e+10 ±  2%      -9.0%  2.256e+10 ±  5%  perf-stat.ps.dTLB-loads
  8.44e+09 ±  2%      -8.3%  7.738e+09 ±  5%  perf-stat.ps.dTLB-stores
   1805658 ±  3%     +14.6%    2069128 ±  6%  perf-stat.ps.iTLB-load-misses
    526225 ± 16%     +90.1%    1000111 ± 18%  perf-stat.ps.iTLB-loads
 1.138e+11 ±  2%      -8.9%  1.037e+11 ±  5%  perf-stat.ps.instructions
 1.228e+08 ±  2%     -13.3%  1.065e+08 ±  7%  perf-stat.ps.node-stores
 3.377e+13            -9.7%   3.05e+13 ±  4%  perf-stat.total.instructions
      0.00            +0.8        0.78 ± 15%  perf-profile.calltrace.cycles-pp.memcpy_erms.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread
      0.00            +0.8        0.81 ± 14%  perf-profile.calltrace.cycles-pp.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +0.8        0.82 ± 14%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +0.8        0.83 ± 14%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.00            +0.8        0.85 ± 13%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +0.8        0.85 ± 13%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      2.30 ± 26%      +2.9        5.16 ± 34%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      2.45 ± 27%      +3.2        5.61 ± 30%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      2.46 ± 27%      +3.2        5.65 ± 30%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.53 ± 26%      +3.3        5.86 ± 29%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.53 ± 26%      +3.3        5.86 ± 29%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      2.53 ± 26%      +3.3        5.86 ± 29%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      2.56 ± 27%      +3.4        5.91 ± 30%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      0.06 ± 11%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.find_busiest_group
      0.06 ± 11%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.22 ±  9%      +0.1        0.28 ±  9%  perf-profile.children.cycles-pp.irq_exit
      0.01 ±173%      +0.1        0.08 ± 22%  perf-profile.children.cycles-pp.rebalance_domains
      0.11 ± 14%      +0.1        0.19 ± 19%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.22 ±  3%      +0.1        0.32 ±  8%  perf-profile.children.cycles-pp.vfs_write
      0.22 ±  3%      +0.1        0.32 ±  9%  perf-profile.children.cycles-pp.ksys_write
      0.21 ±  3%      +0.1        0.31 ±  9%  perf-profile.children.cycles-pp.new_sync_write
      0.11 ± 15%      +0.1        0.21 ± 20%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.11 ± 15%      +0.1        0.22 ± 20%  perf-profile.children.cycles-pp.uart_console_write
      0.10 ± 11%      +0.1        0.20 ± 20%  perf-profile.children.cycles-pp.io_serial_in
      0.00            +0.1        0.11 ± 48%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.16 ±  9%      +0.1        0.28 ± 11%  perf-profile.children.cycles-pp.write
      0.13 ± 12%      +0.1        0.25 ± 17%  perf-profile.children.cycles-pp._fini
      0.00            +0.1        0.12 ± 47%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.12 ± 14%      +0.1        0.25 ± 21%  perf-profile.children.cycles-pp.serial8250_console_write
      0.04 ± 58%      +0.1        0.16 ± 42%  perf-profile.children.cycles-pp.menu_select
      0.14 ± 11%      +0.1        0.27 ± 17%  perf-profile.children.cycles-pp.vprintk_emit
      0.12 ± 14%      +0.1        0.25 ± 17%  perf-profile.children.cycles-pp.devkmsg_write
      0.12 ± 14%      +0.1        0.25 ± 17%  perf-profile.children.cycles-pp.devkmsg_emit
      0.14 ± 12%      +0.1        0.26 ± 19%  perf-profile.children.cycles-pp.console_unlock
      0.00            +0.8        0.79 ± 15%  perf-profile.children.cycles-pp.memcpy_erms
      0.04 ± 58%      +0.8        0.85 ± 13%  perf-profile.children.cycles-pp.ret_from_fork
      0.04 ± 58%      +0.8        0.85 ± 13%  perf-profile.children.cycles-pp.kthread
      0.00            +0.8        0.81 ± 14%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      0.00            +0.8        0.82 ± 14%  perf-profile.children.cycles-pp.process_one_work
      0.00            +0.8        0.83 ± 14%  perf-profile.children.cycles-pp.worker_thread
      2.33 ± 27%      +2.9        5.20 ± 34%  perf-profile.children.cycles-pp.intel_idle
      2.49 ± 27%      +3.2        5.69 ± 31%  perf-profile.children.cycles-pp.cpuidle_enter_state
      2.49 ± 27%      +3.2        5.70 ± 31%  perf-profile.children.cycles-pp.cpuidle_enter
      2.53 ± 26%      +3.3        5.86 ± 29%  perf-profile.children.cycles-pp.start_secondary
      2.56 ± 27%      +3.4        5.91 ± 30%  perf-profile.children.cycles-pp.do_idle
      2.56 ± 27%      +3.4        5.91 ± 30%  perf-profile.children.cycles-pp.secondary_startup_64
      2.56 ± 27%      +3.4        5.91 ± 30%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.07            -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.apic_timer_interrupt
      0.09 ± 13%      +0.1        0.19 ± 19%  perf-profile.self.cycles-pp.io_serial_in
      0.00            +0.8        0.78 ± 15%  perf-profile.self.cycles-pp.memcpy_erms
      2.33 ± 27%      +2.9        5.20 ± 34%  perf-profile.self.cycles-pp.intel_idle
    105414 ± 10%     -55.8%      46547 ± 45%  sched_debug.cfs_rq:/.exec_clock.min
      7955 ± 28%     +60.4%      12759 ±  7%  sched_debug.cfs_rq:/.load.avg
    544679 ± 58%    +110.6%    1147127 ±  8%  sched_debug.cfs_rq:/.load.max
     42363 ± 60%    +116.1%      91551 ±  9%  sched_debug.cfs_rq:/.load.stddev
    666.02 ± 14%     +50.6%       1003 ± 11%  sched_debug.cfs_rq:/.load_avg.max
     59.52 ± 17%     +43.3%      85.28 ±  8%  sched_debug.cfs_rq:/.load_avg.stddev
  22565485 ± 11%     -26.3%   16634488 ±  5%  sched_debug.cfs_rq:/.min_vruntime.avg
  20142148 ± 11%     -57.8%    8498225 ± 50%  sched_debug.cfs_rq:/.min_vruntime.min
     87.71 ± 10%     -85.2%      13.00 ± 64%  sched_debug.cfs_rq:/.nr_spread_over.avg
      1375 ± 37%     -91.2%     120.71 ± 95%  sched_debug.cfs_rq:/.nr_spread_over.max
    184.64 ± 13%     -89.2%      19.90 ± 84%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      5.87 ± 18%     +59.9%       9.38 ± 11%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    272.97 ± 55%    +184.4%     776.24 ± 11%  sched_debug.cfs_rq:/.runnable_load_avg.max
     21.42 ± 53%    +177.8%      59.50 ± 14%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
      7936 ± 28%     +60.5%      12741 ±  7%  sched_debug.cfs_rq:/.runnable_weight.avg
    543857 ± 58%    +110.9%    1147127 ±  8%  sched_debug.cfs_rq:/.runnable_weight.max
     42294 ± 59%    +116.5%      91551 ±  9%  sched_debug.cfs_rq:/.runnable_weight.stddev
    -96364         +8618.0%   -8401037        sched_debug.cfs_rq:/.spread0.min
      1107 ±  2%     +18.9%       1316 ±  6%  sched_debug.cfs_rq:/.util_avg.max
    165.38 ±  2%     +18.2%     195.49 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    746.55 ±  7%     +36.6%       1019 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.max
    165.18 ±  6%     +36.1%     224.86 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    389704 ± 12%     -58.3%     162437 ± 41%  sched_debug.cpu.avg_idle.min
    110.33 ± 30%     -70.2%      32.84 ± 74%  sched_debug.cpu.clock.stddev
    110.33 ± 30%     -70.2%      32.84 ± 74%  sched_debug.cpu.clock_task.stddev
      5.68 ± 20%     +50.0%       8.51 ±  8%  sched_debug.cpu.cpu_load[0].avg
    260.01 ± 64%    +198.4%     775.83 ± 11%  sched_debug.cpu.cpu_load[0].max
     20.40 ± 63%    +178.2%      56.75 ± 13%  sched_debug.cpu.cpu_load[0].stddev
      5.56 ± 16%     +55.2%       8.63 ± 11%  sched_debug.cpu.cpu_load[1].avg
    248.49 ± 59%    +216.6%     786.65 ± 16%  sched_debug.cpu.cpu_load[1].max
     18.77 ± 58%    +206.1%      57.46 ± 17%  sched_debug.cpu.cpu_load[1].stddev
      5.52 ± 12%     +58.6%       8.76 ± 11%  sched_debug.cpu.cpu_load[2].avg
    240.52 ± 45%    +230.4%     794.58 ± 19%  sched_debug.cpu.cpu_load[2].max
     18.10 ± 45%    +221.3%      58.15 ± 20%  sched_debug.cpu.cpu_load[2].stddev
      5.55 ±  9%     +59.6%       8.87 ± 11%  sched_debug.cpu.cpu_load[3].avg
    240.31 ± 36%    +232.2%     798.23 ± 20%  sched_debug.cpu.cpu_load[3].max
     17.96 ± 36%    +226.9%      58.72 ± 20%  sched_debug.cpu.cpu_load[3].stddev
      5.69 ±  8%     +59.0%       9.05 ± 10%  sched_debug.cpu.cpu_load[4].avg
    265.29 ± 30%    +211.1%     825.24 ± 19%  sched_debug.cpu.cpu_load[4].max
     19.60 ± 30%    +210.1%      60.76 ± 19%  sched_debug.cpu.cpu_load[4].stddev
     13044 ± 10%     -18.5%      10626 ±  5%  sched_debug.cpu.curr->pid.avg
      3824 ± 15%     +56.2%       5975 ± 15%  sched_debug.cpu.curr->pid.stddev
    596300 ± 46%     +92.3%    1146624 ±  8%  sched_debug.cpu.load.max
     45339 ± 48%     +83.7%      83280 ±  7%  sched_debug.cpu.load.stddev
      0.00 ±  9%     -50.2%       0.00 ± 55%  sched_debug.cpu.next_balance.stddev
      0.74 ±  2%     +12.5%       0.84 ±  2%  sched_debug.cpu.nr_running.avg
      1.92 ± 14%     +73.0%       3.32 ± 22%  sched_debug.cpu.nr_running.max
      0.31 ± 23%    +135.0%       0.72 ± 27%  sched_debug.cpu.nr_running.stddev
      8688 ± 13%    +281.1%      33109 ± 42%  sched_debug.cpu.sched_goidle.max
    880.37 ± 14%    +294.2%       3470 ± 41%  sched_debug.cpu.sched_goidle.stddev
    477.44 ±  8%     -13.7%     411.92 ±  2%  sched_debug.cpu.ttwu_count.min
    991.67 ± 15%     -42.8%     566.76 ± 10%  sched_debug.cpu.ttwu_local.avg
      2202 ± 14%     -43.8%       1236 ± 19%  sched_debug.cpu.ttwu_local.stddev
     11.00 ±131%   +2522.7%     288.50 ± 96%  interrupts.CPU10.TLB:TLB_shootdowns
      7334           -35.4%       4739 ± 39%  interrupts.CPU100.NMI:Non-maskable_interrupts
      7334           -35.4%       4739 ± 39%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
    364.00 ± 30%     -51.9%     175.00 ± 54%  interrupts.CPU102.RES:Rescheduling_interrupts
    449.75 ± 17%     -47.7%     235.00 ± 44%  interrupts.CPU108.RES:Rescheduling_interrupts
     46.50 ±113%    +409.7%     237.00 ± 59%  interrupts.CPU11.TLB:TLB_shootdowns
    592.75 ± 60%     -73.3%     158.25 ± 69%  interrupts.CPU111.RES:Rescheduling_interrupts
    422.75 ± 38%     -56.8%     182.75 ± 93%  interrupts.CPU114.RES:Rescheduling_interrupts
    550.25 ± 27%     -53.5%     255.75 ± 63%  interrupts.CPU117.RES:Rescheduling_interrupts
     19.50 ± 91%   +1457.7%     303.75 ± 43%  interrupts.CPU12.TLB:TLB_shootdowns
      7224 ±  3%     -47.7%       3779 ± 23%  interrupts.CPU120.NMI:Non-maskable_interrupts
      7224 ±  3%     -47.7%       3779 ± 23%  interrupts.CPU120.PMI:Performance_monitoring_interrupts
    294.50 ± 34%     -65.4%     101.75 ± 44%  interrupts.CPU120.RES:Rescheduling_interrupts
    305.50 ± 28%     -71.4%      87.50 ± 42%  interrupts.CPU124.RES:Rescheduling_interrupts
     48.75 ±122%    +357.9%     223.25 ± 32%  interrupts.CPU13.TLB:TLB_shootdowns
    249.50 ± 24%     -52.2%     119.25 ± 75%  interrupts.CPU130.RES:Rescheduling_interrupts
    569.00 ± 29%     -69.9%     171.25 ± 73%  interrupts.CPU132.RES:Rescheduling_interrupts
      7179 ±  2%     -26.8%       5255 ± 22%  interrupts.CPU133.NMI:Non-maskable_interrupts
      7179 ±  2%     -26.8%       5255 ± 22%  interrupts.CPU133.PMI:Performance_monitoring_interrupts
    343.50 ± 20%     -60.7%     135.00 ± 88%  interrupts.CPU134.RES:Rescheduling_interrupts
    371.75 ± 23%     -63.3%     136.50 ±106%  interrupts.CPU135.RES:Rescheduling_interrupts
    477.25 ± 46%     -61.8%     182.25 ±124%  interrupts.CPU138.RES:Rescheduling_interrupts
    648.75 ± 13%     -78.3%     140.75 ± 96%  interrupts.CPU139.RES:Rescheduling_interrupts
    512.75 ± 73%     -66.3%     173.00 ± 73%  interrupts.CPU14.RES:Rescheduling_interrupts
     59.75 ±156%    +528.5%     375.50 ± 78%  interrupts.CPU14.TLB:TLB_shootdowns
    779.00 ± 41%     -65.5%     268.50 ± 89%  interrupts.CPU141.RES:Rescheduling_interrupts
    851.00 ± 15%     -69.7%     257.75 ±119%  interrupts.CPU143.RES:Rescheduling_interrupts
      1.25 ±103%   +7440.0%      94.25 ±114%  interrupts.CPU144.TLB:TLB_shootdowns
     59.25 ±154%    +220.3%     189.75 ± 82%  interrupts.CPU15.TLB:TLB_shootdowns
      7185 ±  5%     -33.4%       4784 ± 34%  interrupts.CPU152.NMI:Non-maskable_interrupts
      7185 ±  5%     -33.4%       4784 ± 34%  interrupts.CPU152.PMI:Performance_monitoring_interrupts
      7056 ±  6%     -34.0%       4657 ± 37%  interrupts.CPU155.NMI:Non-maskable_interrupts
      7056 ±  6%     -34.0%       4657 ± 37%  interrupts.CPU155.PMI:Performance_monitoring_interrupts
      7099 ±  6%     -35.1%       4607 ± 36%  interrupts.CPU156.NMI:Non-maskable_interrupts
      7099 ±  6%     -35.1%       4607 ± 36%  interrupts.CPU156.PMI:Performance_monitoring_interrupts
     20.50 ±112%    +784.1%     181.25 ± 71%  interrupts.CPU16.TLB:TLB_shootdowns
      7244 ±  6%     -44.8%       3995 ± 51%  interrupts.CPU164.NMI:Non-maskable_interrupts
      7244 ±  6%     -44.8%       3995 ± 51%  interrupts.CPU164.PMI:Performance_monitoring_interrupts
     38.75 ±132%    +506.5%     235.00 ± 87%  interrupts.CPU17.TLB:TLB_shootdowns
    463.00 ± 26%     -55.0%     208.25 ± 61%  interrupts.CPU184.RES:Rescheduling_interrupts
    653.75 ± 77%     -76.3%     155.00 ± 76%  interrupts.CPU185.RES:Rescheduling_interrupts
     10.00 ± 44%    +472.5%      57.25 ± 62%  interrupts.CPU189.TLB:TLB_shootdowns
     68.50 ±137%    +254.7%     243.00 ± 90%  interrupts.CPU19.TLB:TLB_shootdowns
     19.25 ± 57%    +207.8%      59.25 ± 78%  interrupts.CPU190.TLB:TLB_shootdowns
     66.75 ±135%    +270.8%     247.50 ± 70%  interrupts.CPU2.TLB:TLB_shootdowns
     13.00 ±125%   +2090.4%     284.75 ± 73%  interrupts.CPU20.TLB:TLB_shootdowns
     83.50 ±166%    +300.3%     334.25 ± 34%  interrupts.CPU22.TLB:TLB_shootdowns
     18.00 ±136%   +1225.0%     238.50 ±120%  interrupts.CPU23.TLB:TLB_shootdowns
    771.25 ± 33%     -56.3%     337.25 ± 82%  interrupts.CPU26.RES:Rescheduling_interrupts
     19.75 ± 52%    +744.3%     166.75 ± 45%  interrupts.CPU26.TLB:TLB_shootdowns
     32.50 ± 68%   +1141.5%     403.50 ± 57%  interrupts.CPU27.TLB:TLB_shootdowns
     25.00 ±114%    +923.0%     255.75 ± 53%  interrupts.CPU28.TLB:TLB_shootdowns
      7108 ±  5%     -27.8%       5133 ± 26%  interrupts.CPU29.NMI:Non-maskable_interrupts
      7108 ±  5%     -27.8%       5133 ± 26%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
     18.50 ± 88%   +1167.6%     234.50 ± 63%  interrupts.CPU29.TLB:TLB_shootdowns
     16.00 ±130%   +1554.7%     264.75 ± 70%  interrupts.CPU3.TLB:TLB_shootdowns
     32.00 ± 61%   +1094.5%     382.25 ± 57%  interrupts.CPU30.TLB:TLB_shootdowns
      6990 ±  8%     -38.0%       4335 ± 18%  interrupts.CPU31.NMI:Non-maskable_interrupts
      6990 ±  8%     -38.0%       4335 ± 18%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
     18.50 ± 45%   +1754.1%     343.00 ± 72%  interrupts.CPU31.TLB:TLB_shootdowns
    371.75 ± 37%     -66.2%     125.75 ± 23%  interrupts.CPU32.RES:Rescheduling_interrupts
     42.50 ±109%    +531.8%     268.50 ± 98%  interrupts.CPU32.TLB:TLB_shootdowns
     25.50 ± 92%    +838.2%     239.25 ± 62%  interrupts.CPU33.TLB:TLB_shootdowns
     20.25 ± 44%    +718.5%     165.75 ± 62%  interrupts.CPU34.TLB:TLB_shootdowns
      8.75 ± 48%   +1511.4%     141.00 ± 57%  interrupts.CPU35.TLB:TLB_shootdowns
    301.25 ± 26%     -53.6%     139.75 ± 28%  interrupts.CPU36.RES:Rescheduling_interrupts
     17.50 ± 62%   +1430.0%     267.75 ± 36%  interrupts.CPU36.TLB:TLB_shootdowns
     23.00 ± 68%    +389.1%     112.50 ± 53%  interrupts.CPU37.TLB:TLB_shootdowns
     26.50 ± 76%   +1270.8%     363.25 ± 68%  interrupts.CPU38.TLB:TLB_shootdowns
     22.00 ±107%   +1322.7%     313.00 ± 82%  interrupts.CPU39.TLB:TLB_shootdowns
      7317 ±  2%     -35.7%       4702 ± 47%  interrupts.CPU4.NMI:Non-maskable_interrupts
      7317 ±  2%     -35.7%       4702 ± 47%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
     17.75 ± 57%   +1370.4%     261.00 ± 38%  interrupts.CPU40.TLB:TLB_shootdowns
      7189 ±  3%     -27.8%       5188 ± 25%  interrupts.CPU41.NMI:Non-maskable_interrupts
      7189 ±  3%     -27.8%       5188 ± 25%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
     37.75 ± 75%    +753.6%     322.25 ± 62%  interrupts.CPU41.TLB:TLB_shootdowns
     13.75 ± 85%   +1834.5%     266.00 ± 30%  interrupts.CPU42.TLB:TLB_shootdowns
    474.50 ± 52%     -67.2%     155.50 ± 53%  interrupts.CPU43.RES:Rescheduling_interrupts
     22.00 ±116%    +868.2%     213.00 ± 21%  interrupts.CPU43.TLB:TLB_shootdowns
     16.25 ± 84%   +1000.0%     178.75 ± 81%  interrupts.CPU44.TLB:TLB_shootdowns
      7289 ±  3%     -39.1%       4439 ± 18%  interrupts.CPU45.NMI:Non-maskable_interrupts
      7289 ±  3%     -39.1%       4439 ± 18%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    418.50 ± 33%     -73.4%     111.50 ± 66%  interrupts.CPU45.RES:Rescheduling_interrupts
     26.75 ±127%    +813.1%     244.25 ± 60%  interrupts.CPU45.TLB:TLB_shootdowns
      6943 ±  8%     -37.4%       4347 ± 16%  interrupts.CPU46.NMI:Non-maskable_interrupts
      6943 ±  8%     -37.4%       4347 ± 16%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
     43.50 ± 86%    +443.1%     236.25 ± 31%  interrupts.CPU46.TLB:TLB_shootdowns
     10.50 ± 48%   +2314.3%     253.50 ± 13%  interrupts.CPU47.TLB:TLB_shootdowns
     46.50 ±141%    +243.0%     159.50 ± 83%  interrupts.CPU5.TLB:TLB_shootdowns
     11.00 ± 49%   +1570.5%     183.75 ±126%  interrupts.CPU50.TLB:TLB_shootdowns
     11.00 ± 42%   +1586.4%     185.50 ± 59%  interrupts.CPU51.TLB:TLB_shootdowns
      7013 ±  8%     -42.7%       4018 ± 46%  interrupts.CPU52.NMI:Non-maskable_interrupts
      7013 ±  8%     -42.7%       4018 ± 46%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      9.50 ± 59%   +1552.6%     157.00 ± 60%  interrupts.CPU52.TLB:TLB_shootdowns
      7107 ±  7%     -56.5%       3095 ± 16%  interrupts.CPU53.NMI:Non-maskable_interrupts
      7107 ±  7%     -56.5%       3095 ± 16%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
     35.50 ±139%    +501.4%     213.50 ± 70%  interrupts.CPU53.TLB:TLB_shootdowns
      9.00 ± 78%   +1583.3%     151.50 ± 89%  interrupts.CPU54.TLB:TLB_shootdowns
     11.50 ± 49%   +1921.7%     232.50 ±110%  interrupts.CPU56.TLB:TLB_shootdowns
    355.75 ± 16%     -49.9%     178.25 ± 56%  interrupts.CPU57.RES:Rescheduling_interrupts
     10.75 ± 51%   +2695.3%     300.50 ±117%  interrupts.CPU57.TLB:TLB_shootdowns
      9.00 ± 68%   +2658.3%     248.25 ± 88%  interrupts.CPU58.TLB:TLB_shootdowns
      9.00 ± 43%   +1963.9%     185.75 ± 69%  interrupts.CPU59.TLB:TLB_shootdowns
    731.75 ± 14%     -77.3%     165.75 ± 49%  interrupts.CPU6.RES:Rescheduling_interrupts
      8.25 ± 51%   +2306.1%     198.50 ± 62%  interrupts.CPU60.TLB:TLB_shootdowns
     14.00 ± 46%   +2475.0%     360.50 ± 76%  interrupts.CPU61.TLB:TLB_shootdowns
     47.25 ±112%    +587.3%     324.75 ± 35%  interrupts.CPU62.TLB:TLB_shootdowns
     39.00 ±152%    +261.5%     141.00 ± 87%  interrupts.CPU64.TLB:TLB_shootdowns
     25.00 ± 94%    +507.0%     151.75 ± 44%  interrupts.CPU65.TLB:TLB_shootdowns
     17.25 ±120%   +1331.9%     247.00 ± 42%  interrupts.CPU66.TLB:TLB_shootdowns
     18.25 ± 64%    +713.7%     148.50 ± 48%  interrupts.CPU67.TLB:TLB_shootdowns
     11.00 ± 87%   +2545.5%     291.00 ± 80%  interrupts.CPU68.TLB:TLB_shootdowns
      7246 ±  4%     -35.2%       4697 ± 39%  interrupts.CPU69.NMI:Non-maskable_interrupts
      7246 ±  4%     -35.2%       4697 ± 39%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
     12.50 ± 66%   +1624.0%     215.50 ±129%  interrupts.CPU69.TLB:TLB_shootdowns
     15.25 ±107%   +1045.9%     174.75 ± 58%  interrupts.CPU7.TLB:TLB_shootdowns
     37.25 ±150%    +538.9%     238.00 ±109%  interrupts.CPU70.TLB:TLB_shootdowns
     25.75 ±155%    +699.0%     205.75 ± 59%  interrupts.CPU71.TLB:TLB_shootdowns
      6833 ±  5%     -13.4%       5917 ± 10%  interrupts.CPU73.NMI:Non-maskable_interrupts
      6833 ±  5%     -13.4%       5917 ± 10%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
     22.75 ± 71%    +917.6%     231.50 ± 57%  interrupts.CPU73.TLB:TLB_shootdowns
     31.50 ± 94%    +283.3%     120.75 ± 64%  interrupts.CPU74.TLB:TLB_shootdowns
     17.75 ± 70%   +1353.5%     258.00 ± 31%  interrupts.CPU77.TLB:TLB_shootdowns
     13.25 ±114%   +1086.8%     157.25 ± 62%  interrupts.CPU78.TLB:TLB_shootdowns
     34.50 ±101%    +774.6%     301.75 ± 96%  interrupts.CPU79.TLB:TLB_shootdowns
     19.75 ±117%    +838.0%     185.25 ± 90%  interrupts.CPU8.TLB:TLB_shootdowns
     29.50 ±121%    +851.7%     280.75 ± 57%  interrupts.CPU80.TLB:TLB_shootdowns
    322.25 ± 26%     -56.0%     141.75 ± 32%  interrupts.CPU81.RES:Rescheduling_interrupts
     53.75 ±122%    +294.4%     212.00 ± 29%  interrupts.CPU81.TLB:TLB_shootdowns
     16.75 ±136%   +1255.2%     227.00 ± 58%  interrupts.CPU83.TLB:TLB_shootdowns
     23.00 ± 90%    +722.8%     189.25 ±101%  interrupts.CPU84.TLB:TLB_shootdowns
     20.25 ± 82%    +921.0%     206.75 ± 53%  interrupts.CPU85.TLB:TLB_shootdowns
     41.00 ±102%    +344.5%     182.25 ± 46%  interrupts.CPU86.TLB:TLB_shootdowns
     15.25 ± 70%   +1208.2%     199.50 ± 48%  interrupts.CPU87.TLB:TLB_shootdowns
     50.75 ±125%    +522.2%     315.75 ± 79%  interrupts.CPU88.TLB:TLB_shootdowns
     48.00 ±132%    +797.9%     431.00 ± 51%  interrupts.CPU89.TLB:TLB_shootdowns
     31.00 ±134%    +802.4%     279.75 ± 36%  interrupts.CPU9.TLB:TLB_shootdowns
     99.75 ±161%    +285.0%     384.00 ± 25%  interrupts.CPU90.TLB:TLB_shootdowns
     72.25 ± 89%    +421.8%     377.00 ± 81%  interrupts.CPU91.TLB:TLB_shootdowns
     43.25 ±156%    +556.1%     283.75 ± 76%  interrupts.CPU94.TLB:TLB_shootdowns
     48.50 ±154%    +313.9%     200.75 ± 67%  interrupts.CPU95.TLB:TLB_shootdowns
    555.75 ± 38%     -60.6%     219.00 ± 46%  interrupts.CPU96.RES:Rescheduling_interrupts
      7303 ±  2%     -24.2%       5538 ± 31%  interrupts.CPU97.NMI:Non-maskable_interrupts
      7303 ±  2%     -24.2%       5538 ± 31%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
    169.75 ±  2%     -16.8%     141.25 ± 10%  interrupts.IWI:IRQ_work_interrupts
   1236629 ±  2%     -17.1%    1025293 ± 10%  interrupts.NMI:Non-maskable_interrupts
   1236629 ±  2%     -17.1%    1025293 ± 10%  interrupts.PMI:Performance_monitoring_interrupts
     86301 ±  7%     -29.2%      61084 ± 21%  interrupts.RES:Rescheduling_interrupts
      5187 ± 75%    +386.0%      25214 ± 15%  interrupts.TLB:TLB_shootdowns
     29816 ±  3%     -63.5%      10876 ± 31%  softirqs.CPU0.RCU
     34588           -54.6%      15700 ± 26%  softirqs.CPU1.RCU
     32246 ± 10%     -64.8%      11337 ± 37%  softirqs.CPU10.RCU
     30856 ±  4%     -62.4%      11612 ± 32%  softirqs.CPU100.RCU
    109780 ±  2%     +19.3%     130913 ± 14%  softirqs.CPU100.TIMER
     30870 ±  4%     -64.4%      10983 ± 37%  softirqs.CPU101.RCU
     30730 ±  2%     -64.7%      10855 ± 38%  softirqs.CPU102.RCU
     30736 ±  4%     -64.5%      10904 ± 37%  softirqs.CPU103.RCU
     30940 ±  3%     -64.7%      10930 ± 38%  softirqs.CPU104.RCU
      7998 ± 15%     +95.2%      15613 ± 47%  softirqs.CPU104.SCHED
     30768 ±  4%     -64.6%      10885 ± 38%  softirqs.CPU105.RCU
     30948 ±  4%     -64.8%      10891 ± 38%  softirqs.CPU106.RCU
     31224 ±  3%     -64.4%      11107 ± 42%  softirqs.CPU107.RCU
     30527 ±  4%     -64.6%      10819 ± 37%  softirqs.CPU108.RCU
     30887 ±  6%     -64.0%      11129 ± 35%  softirqs.CPU109.RCU
     31618 ±  3%     -61.6%      12154 ± 33%  softirqs.CPU11.RCU
     30944 ±  6%     -65.0%      10830 ± 38%  softirqs.CPU110.RCU
     30991 ±  5%     -65.4%      10720 ± 39%  softirqs.CPU111.RCU
     31052 ±  5%     -64.4%      11063 ± 39%  softirqs.CPU112.RCU
     31352 ±  4%     -64.5%      11125 ± 38%  softirqs.CPU113.RCU
     31433 ±  4%     -64.8%      11076 ± 39%  softirqs.CPU114.RCU
     31578 ±  4%     -64.6%      11183 ± 37%  softirqs.CPU115.RCU
     31282 ±  5%     -65.0%      10956 ± 40%  softirqs.CPU116.RCU
      8165 ± 16%     +96.7%      16057 ± 50%  softirqs.CPU116.SCHED
     31269 ±  4%     -65.3%      10859 ± 40%  softirqs.CPU117.RCU
     31179 ±  4%     -64.5%      11060 ± 39%  softirqs.CPU118.RCU
     31111 ±  4%     -64.2%      11126 ± 43%  softirqs.CPU119.RCU
     30940 ±  3%     -63.3%      11341 ± 36%  softirqs.CPU12.RCU
     30915 ±  5%     -64.0%      11134 ± 40%  softirqs.CPU120.RCU
     31078 ±  4%     -64.3%      11108 ± 42%  softirqs.CPU121.RCU
     30876 ±  4%     -64.0%      11102 ± 41%  softirqs.CPU122.RCU
     30986 ±  4%     -64.1%      11134 ± 40%  softirqs.CPU123.RCU
     31046 ±  4%     -64.4%      11041 ± 41%  softirqs.CPU124.RCU
     32058 ±  6%     -64.7%      11301 ± 40%  softirqs.CPU125.RCU
     31077 ±  4%     -64.6%      10997 ± 41%  softirqs.CPU126.RCU
     30992 ±  4%     -62.3%      11674 ± 39%  softirqs.CPU127.RCU
     31146 ±  4%     -65.3%      10807 ± 39%  softirqs.CPU128.RCU
     30009 ±  2%     -64.1%      10772 ± 39%  softirqs.CPU129.RCU
     32302 ±  7%     -64.2%      11576 ± 33%  softirqs.CPU13.RCU
     30453 ±  4%     -64.7%      10745 ± 39%  softirqs.CPU130.RCU
     30806 ±  3%     -65.0%      10781 ± 40%  softirqs.CPU131.RCU
     30588 ±  3%     -64.6%      10818 ± 40%  softirqs.CPU132.RCU
     30210 ±  4%     -64.5%      10735 ± 38%  softirqs.CPU133.RCU
     30601 ±  4%     -64.6%      10819 ± 39%  softirqs.CPU134.RCU
     30534 ±  4%     -64.7%      10779 ± 40%  softirqs.CPU135.RCU
     30554 ±  3%     -64.7%      10784 ± 40%  softirqs.CPU136.RCU
     30567 ±  4%     -65.1%      10666 ± 40%  softirqs.CPU137.RCU
     30828 ±  4%     -65.2%      10725 ± 41%  softirqs.CPU138.RCU
     30696 ±  4%     -65.1%      10707 ± 39%  softirqs.CPU139.RCU
     31276 ±  5%     -62.7%      11651 ± 33%  softirqs.CPU14.RCU
     30435 ±  3%     -64.5%      10791 ± 40%  softirqs.CPU140.RCU
     29203 ±  3%     -63.9%      10541 ± 39%  softirqs.CPU141.RCU
     30610 ±  3%     -65.3%      10636 ± 40%  softirqs.CPU142.RCU
     30571 ±  4%     -63.0%      11320 ± 49%  softirqs.CPU143.RCU
     29631 ±  8%     -64.0%      10676 ± 40%  softirqs.CPU144.RCU
      7890 ± 24%    +101.7%      15917 ± 50%  softirqs.CPU144.SCHED
     29933 ±  7%     -64.9%      10517 ± 40%  softirqs.CPU145.RCU
      7838 ± 29%    +102.7%      15888 ± 51%  softirqs.CPU145.SCHED
     29765 ±  7%     -64.9%      10443 ± 42%  softirqs.CPU146.RCU
      7790 ± 28%    +106.8%      16112 ± 52%  softirqs.CPU146.SCHED
     30082 ±  6%     -64.5%      10684 ± 39%  softirqs.CPU147.RCU
      7622 ± 26%    +114.6%      16354 ± 52%  softirqs.CPU147.SCHED
     29714 ±  6%     -64.6%      10532 ± 40%  softirqs.CPU148.RCU
      7879 ± 28%    +102.7%      15975 ± 51%  softirqs.CPU148.SCHED
     30469 ± 12%     -65.6%      10492 ± 40%  softirqs.CPU149.RCU
      7941 ± 29%    +101.4%      15994 ± 49%  softirqs.CPU149.SCHED
     31179 ±  5%     -63.0%      11523 ± 34%  softirqs.CPU15.RCU
     29920 ±  7%     -64.5%      10625 ± 39%  softirqs.CPU150.RCU
      7900 ± 30%    +108.6%      16483 ± 51%  softirqs.CPU150.SCHED
     29991 ±  6%     -64.4%      10669 ± 40%  softirqs.CPU151.RCU
      7976 ± 31%    +104.9%      16344 ± 51%  softirqs.CPU151.SCHED
     29590 ±  7%     -64.5%      10498 ± 41%  softirqs.CPU152.RCU
      7963 ± 28%    +107.9%      16560 ± 52%  softirqs.CPU152.SCHED
     29553 ±  6%     -65.0%      10346 ± 41%  softirqs.CPU153.RCU
      7998 ± 28%    +106.9%      16545 ± 53%  softirqs.CPU153.SCHED
     29758 ±  7%     -64.4%      10603 ± 39%  softirqs.CPU154.RCU
      8066 ± 31%    +104.5%      16493 ± 52%  softirqs.CPU154.SCHED
     30207 ±  6%     -65.4%      10461 ± 38%  softirqs.CPU155.RCU
      8255 ± 29%     +96.5%      16219 ± 48%  softirqs.CPU155.SCHED
     29913 ±  7%     -64.6%      10577 ± 38%  softirqs.CPU156.RCU
      8117 ± 28%     +98.6%      16117 ± 47%  softirqs.CPU156.SCHED
     29739 ±  7%     -65.0%      10406 ± 39%  softirqs.CPU157.RCU
      8023 ± 31%    +108.8%      16750 ± 49%  softirqs.CPU157.SCHED
     29951 ±  7%     -65.2%      10414 ± 41%  softirqs.CPU158.RCU
      8075 ± 30%    +108.2%      16815 ± 51%  softirqs.CPU158.SCHED
     30338 ±  5%     -65.6%      10434 ± 41%  softirqs.CPU159.RCU
      8096 ± 31%    +105.2%      16613 ± 53%  softirqs.CPU159.SCHED
     32653 ±  3%     -64.2%      11703 ± 38%  softirqs.CPU16.RCU
     32209 ±  8%     -67.7%      10417 ± 43%  softirqs.CPU160.RCU
      8128 ± 25%    +101.6%      16389 ± 53%  softirqs.CPU160.SCHED
     32261 ±  8%     -66.8%      10724 ± 45%  softirqs.CPU161.RCU
      8004 ± 29%    +110.2%      16827 ± 51%  softirqs.CPU161.SCHED
     32193 ±  7%     -67.6%      10439 ± 44%  softirqs.CPU162.RCU
      8126 ± 29%     +98.0%      16092 ± 49%  softirqs.CPU162.SCHED
     32075 ±  8%     -67.2%      10534 ± 44%  softirqs.CPU163.RCU
      8351 ± 28%     +97.7%      16515 ± 51%  softirqs.CPU163.SCHED
     32082 ±  8%     -67.2%      10522 ± 45%  softirqs.CPU164.RCU
      7812 ± 31%    +109.3%      16348 ± 50%  softirqs.CPU164.SCHED
     31908 ±  8%     -67.4%      10402 ± 44%  softirqs.CPU165.RCU
      7840 ± 29%    +110.9%      16533 ± 51%  softirqs.CPU165.SCHED
     32493 ±  9%     -65.8%      11120 ± 40%  softirqs.CPU166.RCU
      7577 ± 31%    +118.8%      16578 ± 52%  softirqs.CPU166.SCHED
     31982 ±  8%     -67.8%      10310 ± 45%  softirqs.CPU167.RCU
      7793 ± 32%    +108.3%      16235 ± 49%  softirqs.CPU167.SCHED
     30811 ±  4%     -65.5%      10633 ± 42%  softirqs.CPU168.RCU
     31032 ±  5%     -66.9%      10276 ± 39%  softirqs.CPU169.RCU
     32029 ±  4%     -63.6%      11647 ± 37%  softirqs.CPU17.RCU
     30978 ±  4%     -65.5%      10686 ± 45%  softirqs.CPU170.RCU
     31018 ±  4%     -63.9%      11205 ± 38%  softirqs.CPU171.RCU
     30917 ±  4%     -64.9%      10863 ± 40%  softirqs.CPU172.RCU
     31187 ±  4%     -65.0%      10903 ± 46%  softirqs.CPU173.RCU
     31009 ±  5%     -65.1%      10833 ± 45%  softirqs.CPU174.RCU
     30923 ±  4%     -65.5%      10677 ± 43%  softirqs.CPU175.RCU
     29732 ±  4%     -64.8%      10476 ± 40%  softirqs.CPU176.RCU
     29864 ±  5%     -65.5%      10311 ± 40%  softirqs.CPU177.RCU
     31273 ±  8%     -66.5%      10477 ± 39%  softirqs.CPU178.RCU
     30307 ±  4%     -65.5%      10470 ± 39%  softirqs.CPU179.RCU
     32332 ±  5%     -63.8%      11700 ± 37%  softirqs.CPU18.RCU
     30372 ±  5%     -65.3%      10536 ± 39%  softirqs.CPU180.RCU
     29664 ±  5%     -64.4%      10558 ± 38%  softirqs.CPU181.RCU
     29989 ±  6%     -65.0%      10505 ± 39%  softirqs.CPU182.RCU
     29804 ±  6%     -64.6%      10537 ± 39%  softirqs.CPU183.RCU
     29919 ±  5%     -64.8%      10523 ± 39%  softirqs.CPU184.RCU
     29803 ±  5%     -65.1%      10401 ± 40%  softirqs.CPU185.RCU
     29526 ±  6%     -65.2%      10281 ± 40%  softirqs.CPU186.RCU
     29087 ±  8%     -65.4%      10065 ± 42%  softirqs.CPU187.RCU
     29704 ±  5%     -64.9%      10412 ± 40%  softirqs.CPU188.RCU
     30645 ±  6%     -66.0%      10420 ± 39%  softirqs.CPU189.RCU
     31752 ±  4%     -63.4%      11632 ± 38%  softirqs.CPU19.RCU
     30599 ±  5%     -64.8%      10771 ± 38%  softirqs.CPU190.RCU
     29683 ±  5%     -65.0%      10377 ± 42%  softirqs.CPU191.RCU
     32882 ±  9%     -65.5%      11330 ± 36%  softirqs.CPU2.RCU
     32022 ±  5%     -63.8%      11579 ± 39%  softirqs.CPU20.RCU
      7907 ± 13%     +99.8%      15796 ± 51%  softirqs.CPU20.SCHED
     31968 ±  4%     -63.9%      11555 ± 38%  softirqs.CPU21.RCU
     32077 ±  3%     -62.4%      12055 ± 41%  softirqs.CPU22.RCU
     31617 ±  4%     -62.8%      11754 ± 37%  softirqs.CPU23.RCU
     32138 ±  4%     -63.7%      11655 ± 39%  softirqs.CPU24.RCU
     31361 ±  4%     -62.7%      11689 ± 41%  softirqs.CPU25.RCU
     31323 ±  4%     -62.9%      11630 ± 40%  softirqs.CPU26.RCU
     31308 ±  4%     -62.7%      11686 ± 40%  softirqs.CPU27.RCU
     31547 ±  3%     -63.0%      11681 ± 41%  softirqs.CPU28.RCU
     31626 ±  3%     -63.3%      11599 ± 40%  softirqs.CPU29.RCU
     31673 ±  6%     -63.9%      11422 ± 35%  softirqs.CPU3.RCU
     31606 ±  4%     -61.5%      12169 ± 37%  softirqs.CPU30.RCU
     31469 ±  3%     -61.4%      12156 ± 38%  softirqs.CPU31.RCU
     31283 ±  4%     -64.0%      11270 ± 38%  softirqs.CPU32.RCU
     31191 ±  4%     -64.0%      11222 ± 37%  softirqs.CPU33.RCU
     31017 ±  4%     -64.1%      11136 ± 37%  softirqs.CPU34.RCU
     31373 ±  4%     -64.1%      11267 ± 37%  softirqs.CPU35.RCU
     31275 ±  4%     -63.9%      11277 ± 37%  softirqs.CPU36.RCU
     30886 ±  4%     -63.6%      11243 ± 38%  softirqs.CPU37.RCU
     31160 ±  4%     -64.1%      11193 ± 38%  softirqs.CPU38.RCU
     31086 ±  5%     -63.8%      11260 ± 38%  softirqs.CPU39.RCU
     31615 ±  6%     -62.6%      11817 ± 33%  softirqs.CPU4.RCU
     30978 ±  4%     -63.8%      11220 ± 39%  softirqs.CPU40.RCU
     31165 ±  5%     -63.9%      11245 ± 37%  softirqs.CPU41.RCU
     31457 ±  4%     -64.4%      11209 ± 38%  softirqs.CPU42.RCU
     31228 ±  4%     -64.2%      11174 ± 39%  softirqs.CPU43.RCU
     32908 ± 12%     -65.9%      11214 ± 38%  softirqs.CPU44.RCU
     30493 ±  6%     -63.4%      11151 ± 36%  softirqs.CPU45.RCU
     31254 ±  4%     -64.2%      11186 ± 38%  softirqs.CPU46.RCU
     31291 ±  4%     -64.1%      11226 ± 38%  softirqs.CPU47.RCU
     30876 ± 10%     -62.0%      11725 ± 36%  softirqs.CPU48.RCU
      7995 ± 21%     +95.7%      15647 ± 51%  softirqs.CPU48.SCHED
     30316 ±  6%     -60.1%      12111 ± 51%  softirqs.CPU49.RCU
      7946 ± 28%     +98.5%      15776 ± 54%  softirqs.CPU49.SCHED
     31492 ±  3%     -61.8%      12025 ± 35%  softirqs.CPU5.RCU
      8761 ± 11%     +80.3%      15801 ± 47%  softirqs.CPU5.SCHED
     30309 ±  7%     -63.9%      10944 ± 39%  softirqs.CPU50.RCU
      8019 ± 27%    +100.3%      16062 ± 53%  softirqs.CPU50.SCHED
     30374 ±  6%     -63.0%      11239 ± 37%  softirqs.CPU51.RCU
      7680 ± 25%    +109.9%      16118 ± 52%  softirqs.CPU51.SCHED
     30120 ±  6%     -63.2%      11084 ± 38%  softirqs.CPU52.RCU
      7862 ± 29%    +100.5%      15763 ± 52%  softirqs.CPU52.SCHED
     30446 ±  8%     -63.9%      11004 ± 38%  softirqs.CPU53.RCU
      7866 ± 28%    +102.7%      15946 ± 52%  softirqs.CPU53.SCHED
     30326 ±  7%     -63.6%      11043 ± 38%  softirqs.CPU54.RCU
      7793 ± 28%    +104.5%      15936 ± 51%  softirqs.CPU54.SCHED
     30591 ±  8%     -63.0%      11329 ± 37%  softirqs.CPU55.RCU
      7940 ± 31%    +102.2%      16059 ± 55%  softirqs.CPU55.SCHED
     29824 ±  7%     -62.4%      11221 ± 38%  softirqs.CPU56.RCU
      7772 ± 27%    +106.4%      16041 ± 52%  softirqs.CPU56.SCHED
     30003 ±  6%     -64.1%      10783 ± 41%  softirqs.CPU57.RCU
      7848 ± 29%    +105.3%      16114 ± 52%  softirqs.CPU57.SCHED
    109110 ±  3%     +17.9%     128620 ± 15%  softirqs.CPU57.TIMER
     30636 ±  6%     -63.3%      11251 ± 35%  softirqs.CPU58.RCU
      8110 ± 24%     +99.3%      16161 ± 51%  softirqs.CPU58.SCHED
     30485 ±  6%     -63.8%      11046 ± 37%  softirqs.CPU59.RCU
      7940 ± 25%    +101.2%      15977 ± 53%  softirqs.CPU59.SCHED
     31296 ±  3%     -64.1%      11250 ± 37%  softirqs.CPU6.RCU
     30473 ±  6%     -62.8%      11325 ± 36%  softirqs.CPU60.RCU
      8033 ± 26%    +100.1%      16072 ± 51%  softirqs.CPU60.SCHED
     30216 ±  6%     -63.2%      11119 ± 38%  softirqs.CPU61.RCU
      7858 ± 28%    +103.9%      16023 ± 52%  softirqs.CPU61.SCHED
     30694 ±  9%     -64.1%      11028 ± 39%  softirqs.CPU62.RCU
      8039 ± 28%    +100.5%      16119 ± 52%  softirqs.CPU62.SCHED
     30404 ±  7%     -63.7%      11051 ± 39%  softirqs.CPU63.RCU
      7897 ± 29%    +100.2%      15812 ± 52%  softirqs.CPU63.SCHED
     30604 ± 11%     -65.6%      10518 ± 39%  softirqs.CPU64.RCU
      7885 ± 24%     +96.9%      15526 ± 49%  softirqs.CPU64.SCHED
     30305 ± 12%     -65.3%      10508 ± 39%  softirqs.CPU65.RCU
      7761 ± 28%    +102.0%      15678 ± 52%  softirqs.CPU65.SCHED
     30441 ± 12%     -65.8%      10401 ± 38%  softirqs.CPU66.RCU
      7894 ± 27%    +102.9%      16016 ± 51%  softirqs.CPU66.SCHED
     30412 ± 11%     -65.4%      10532 ± 40%  softirqs.CPU67.RCU
      7922 ± 27%    +103.0%      16081 ± 53%  softirqs.CPU67.SCHED
     30678 ± 11%     -66.0%      10434 ± 41%  softirqs.CPU68.RCU
      7645 ± 28%    +110.2%      16071 ± 53%  softirqs.CPU68.SCHED
     30428 ± 12%     -61.2%      11797 ± 32%  softirqs.CPU69.RCU
      7668 ± 27%    +108.9%      16022 ± 53%  softirqs.CPU69.SCHED
     31178 ±  4%     -63.6%      11355 ± 36%  softirqs.CPU7.RCU
     30719 ± 12%     -65.8%      10514 ± 39%  softirqs.CPU70.RCU
      7438 ± 28%    +112.4%      15799 ± 51%  softirqs.CPU70.SCHED
     30630 ± 12%     -63.3%      11241 ± 35%  softirqs.CPU71.RCU
      7560 ± 29%    +109.9%      15867 ± 51%  softirqs.CPU71.SCHED
     29629 ±  7%     -63.6%      10783 ± 35%  softirqs.CPU72.RCU
      8795 ± 14%     +84.1%      16189 ± 55%  softirqs.CPU72.SCHED
     29557 ±  7%     -65.5%      10184 ± 34%  softirqs.CPU73.RCU
     29574 ±  7%     -64.4%      10523 ± 39%  softirqs.CPU74.RCU
     31222 ±  8%     -66.2%      10560 ± 38%  softirqs.CPU75.RCU
     31470 ± 12%     -66.0%      10710 ± 37%  softirqs.CPU76.RCU
      9480 ± 19%     +73.5%      16449 ± 53%  softirqs.CPU76.SCHED
     29741 ±  7%     -64.4%      10594 ± 38%  softirqs.CPU77.RCU
     29682 ±  7%     -63.5%      10825 ± 41%  softirqs.CPU78.RCU
     29751 ±  7%     -64.1%      10694 ± 37%  softirqs.CPU79.RCU
     31254 ±  2%     -62.8%      11614 ± 34%  softirqs.CPU8.RCU
      8012 ± 14%     +94.8%      15604 ± 51%  softirqs.CPU8.SCHED
     30481 ±  4%     -64.0%      10967 ± 37%  softirqs.CPU80.RCU
     30079 ±  5%     -63.5%      10984 ± 39%  softirqs.CPU81.RCU
     30437 ±  5%     -64.2%      10911 ± 36%  softirqs.CPU82.RCU
     30650 ±  4%     -64.6%      10864 ± 36%  softirqs.CPU83.RCU
     30324 ±  4%     -63.8%      10979 ± 38%  softirqs.CPU84.RCU
     30167 ±  4%     -63.5%      11012 ± 39%  softirqs.CPU85.RCU
     30627 ±  5%     -64.4%      10910 ± 37%  softirqs.CPU86.RCU
     30343 ±  5%     -63.5%      11077 ± 37%  softirqs.CPU87.RCU
     30428 ±  4%     -63.9%      10973 ± 37%  softirqs.CPU88.RCU
     30348 ±  5%     -64.2%      10862 ± 37%  softirqs.CPU89.RCU
     31425 ±  4%     -62.5%      11772 ± 38%  softirqs.CPU9.RCU
     30022 ±  5%     -63.9%      10847 ± 37%  softirqs.CPU90.RCU
     29730 ±  6%     -64.4%      10584 ± 43%  softirqs.CPU91.RCU
     30359 ±  3%     -64.2%      10866 ± 37%  softirqs.CPU92.RCU
     30269 ±  4%     -64.0%      10882 ± 37%  softirqs.CPU93.RCU
     30371 ±  5%     -64.0%      10919 ± 36%  softirqs.CPU94.RCU
     30195 ±  5%     -64.0%      10885 ± 38%  softirqs.CPU95.RCU
     29811 ±  3%     -64.0%      10733 ± 36%  softirqs.CPU96.RCU
     31071 ±  3%     -64.4%      11046 ± 37%  softirqs.CPU97.RCU
      8191 ± 15%     +93.1%      15813 ± 48%  softirqs.CPU97.SCHED
     31976 ±  6%     -63.3%      11740 ± 34%  softirqs.CPU98.RCU
     31121 ±  4%     -64.3%      11103 ± 36%  softirqs.CPU99.RCU
   5916637 ±  5%     -64.3%    2113584 ± 38%  softirqs.RCU
   1689807 ± 11%     +64.0%    2770803 ± 17%  softirqs.SCHED





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.2.0-rc5-01040-g8a99de3dc0dd4" of type "text/plain" (196530 bytes)

View attachment "job-script" of type "text/plain" (7369 bytes)

View attachment "job.yaml" of type "text/plain" (4996 bytes)

View attachment "reproduce" of type "text/plain" (9780 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ