lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Mon, 20 May 2019 13:16:34 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Thomas Gleixner <tglx@...utronix.de>
Cc:     Borislav Petkov <bp@...e.de>, Jon Masters <jcm@...hat.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [x86/speculation/mds]  bc1241700a:  unixbench.score -49.0% regression

Greeting,

FYI, we noticed a -49.0% regression of unixbench.score due to commit:


commit: bc1241700acd82ec69fde98c5763ce51086269f8 ("x86/speculation/mds: Add mitigation control for MDS")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: unixbench
on test machine: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory
with following parameters:

	runtime: 300s
	nr_task: 1
	test: syscall
	cpufreq_governor: performance
	ucode: 0x7000017

test-description: UnixBench is the original BYTE UNIX benchmark suite aims to test performance of Unix-like system.
test-url: https://github.com/kdlucas/byte-unixbench

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------+
| testcase: change | unixbench: unixbench.score 24.4% improvement                           |
| test machine     | 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory        |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=30%                                                            |
|                  | runtime=300s                                                           |
|                  | test=fstime                                                            |
|                  | ucode=0x7000017                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -49.5% regression         |
| test machine     | 8 threads Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz with 16G memory     |
| test parameters  | cpufreq_governor=performance                                           |
|                  | mode=process                                                           |
|                  | nr_task=100%                                                           |
|                  | test=lseek2                                                            |
|                  | ucode=0x21                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -43.8% regression          |
| test machine     | 8 threads Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz with 16G memory     |
| test parameters  | cpufreq_governor=performance                                           |
|                  | mode=thread                                                            |
|                  | nr_task=100%                                                           |
|                  | test=poll1                                                             |
|                  | ucode=0x21                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -14.8% regression          |
| test machine     | 8 threads Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz with 16G memory     |
| test parameters  | cpufreq_governor=performance                                           |
|                  | mode=thread                                                            |
|                  | nr_task=100%                                                           |
|                  | test=context_switch1                                                   |
|                  | ucode=0x21                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | unixbench: unixbench.score -31.0% regression                           |
| test machine     | 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory        |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=1                                                              |
|                  | runtime=300s                                                           |
|                  | test=pipe                                                              |
|                  | ucode=0x7000017                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | will-it-scale:                                                         |
| test machine     | 192 threads Skylake-SP with 256G memory                                |
| test parameters  | cpufreq_governor=performance                                           |
|                  | mode=process                                                           |
|                  | nr_task=100%                                                           |
|                  | test=getppid1                                                          |
+------------------+------------------------------------------------------------------------+
| testcase: change | vm-scalability: vm-scalability.median -5.8% regression                 |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | runtime=300s                                                           |
|                  | size=1T                                                                |
|                  | test=lru-shm                                                           |
|                  | ucode=0x14                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | aim9: aim9.add_double.ops_per_sec 2.0% improvement                     |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | test=all                                                               |
|                  | testtime=5s                                                            |
|                  | ucode=0x14                                                             |
+------------------+------------------------------------------------------------------------+


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/1/debian-x86_64-2019-05-14.cgz/300s/lkp-bdw-de1/syscall/unixbench/0x7000017

commit: 
  07f07f55a2 ("x86/speculation/mds: Conditionally clear CPU buffers on idle entry")
  bc1241700a ("x86/speculation/mds: Add mitigation control for MDS")

07f07f55a29cb705 bc1241700acd82ec69fde98c576 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    481.75 ±  2%     -49.0%     245.88 ±  2%  unixbench.score
    183.57           -48.8%      93.97        unixbench.time.system_time
    116.79           +76.7%     206.42        unixbench.time.user_time
 2.864e+08 ±  3%     -46.4%  1.535e+08 ±  6%  unixbench.workload
     32.35 ±  2%      -4.9%      30.78        boot-time.boot
    457.90 ±  3%      -5.6%     432.28        boot-time.idle
      0.10 ±136%      +0.4        0.48 ± 80%  mpstat.cpu.all.iowait%
      2.99            -1.5        1.49 ±  5%  mpstat.cpu.all.sys%
      1.93 ±  2%      +1.2        3.18 ±  5%  mpstat.cpu.all.usr%
     52710 ±  2%      +8.4%      57152 ±  5%  softirqs.CPU11.SCHED
     51299 ±  5%     +14.1%      58519 ±  6%  softirqs.CPU12.SCHED
    185332 ±  6%     +16.2%     215439 ±  9%  softirqs.CPU14.TIMER
     52386 ±  4%     +11.0%      58133 ±  5%  softirqs.CPU5.SCHED
      1826 ± 68%     -76.0%     437.75 ± 28%  interrupts.32:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
    678.50 ± 21%     -46.5%     363.25 ± 38%  interrupts.CPU0.NMI:Non-maskable_interrupts
    678.50 ± 21%     -46.5%     363.25 ± 38%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      1826 ± 68%     -76.0%     437.75 ± 28%  interrupts.CPU10.32:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
    208.25 ± 64%     -72.4%      57.50 ± 68%  interrupts.CPU10.RES:Rescheduling_interrupts
    193.50 ± 67%    +137.5%     459.50 ± 29%  interrupts.CPU13.RES:Rescheduling_interrupts
    572790 ±  4%     -10.4%     513159 ±  5%  sched_debug.cfs_rq:/.load.max
    105.66 ±  7%     -12.6%      92.38 ±  3%  sched_debug.cfs_rq:/.load_avg.avg
    812.61 ± 11%     -18.1%     665.25 ±  9%  sched_debug.cfs_rq:/.load_avg.max
    233.72 ±  7%     -12.7%     204.07 ±  4%  sched_debug.cfs_rq:/.load_avg.stddev
     22.78 ± 44%     -53.9%      10.50 ± 29%  sched_debug.cfs_rq:/.removed.load_avg.avg
    219.43 ± 33%     -23.4%     168.03 ± 29%  sched_debug.cfs_rq:/.removed.load_avg.max
     63.81 ± 32%     -36.3%      40.67 ± 29%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    549.11 ±  4%     -11.2%     487.56 ±  5%  sched_debug.cfs_rq:/.runnable_load_avg.max
    560615 ±  4%     -11.2%     497882 ±  5%  sched_debug.cfs_rq:/.runnable_weight.max
      0.99 ± 10%     +15.1%       1.14 ±  9%  sched_debug.cpu.clock.stddev
      0.99 ± 10%     +14.9%       1.13 ± 10%  sched_debug.cpu.clock_task.stddev
    576885 ±  3%     -11.0%     513657 ±  5%  sched_debug.cpu.load.max
     26.28 ±  3%     +39.2%      36.57 ±  9%  perf-stat.i.MPKI
 1.557e+08           -39.2%   94636889 ±  4%  perf-stat.i.branch-instructions
      4.62 ± 13%      +1.1        5.68 ±  7%  perf-stat.i.branch-miss-rate%
   6205768 ±  2%     -23.1%    4774655 ±  6%  perf-stat.i.branch-misses
      3.76 ±  2%     +48.7%       5.59 ±  5%  perf-stat.i.cpi
      0.37 ± 15%      +0.8        1.16 ±  7%  perf-stat.i.dTLB-load-miss-rate%
    590924 ± 16%    +225.8%    1925173 ±  7%  perf-stat.i.dTLB-load-misses
 2.722e+08           -37.5%    1.7e+08 ±  4%  perf-stat.i.dTLB-loads
      0.09 ± 15%      +0.5        0.62 ±  3%  perf-stat.i.dTLB-store-miss-rate%
    170889 ± 23%    +765.6%    1479136 ±  5%  perf-stat.i.dTLB-store-misses
     52.83 ±  4%      -7.1       45.74 ±  5%  perf-stat.i.iTLB-load-miss-rate%
   5709026 ±  2%     -50.0%    2853991 ±  5%  perf-stat.i.iTLB-load-misses
   5526437 ±  2%     -26.8%    4045399 ±  4%  perf-stat.i.iTLB-loads
 8.632e+08           -39.5%  5.221e+08 ±  4%  perf-stat.i.instructions
      0.29 ±  2%     -34.5%       0.19 ±  4%  perf-stat.i.ipc
     15.29 ±  8%     +68.7%      25.80 ± 12%  perf-stat.overall.MPKI
      3.99 ±  3%      +1.1        5.04 ±  4%  perf-stat.overall.branch-miss-rate%
      3.28 ±  2%     +60.7%       5.27 ±  4%  perf-stat.overall.cpi
      0.22 ± 18%      +0.9        1.12 ±  5%  perf-stat.overall.dTLB-load-miss-rate%
      0.07 ± 20%      +0.6        0.70 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     50.81            -9.5       41.36        perf-stat.overall.iTLB-load-miss-rate%
    151.26           +21.0%     183.10 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.31 ±  2%     -37.7%       0.19 ±  4%  perf-stat.overall.ipc
      1206 ±  3%     +19.2%       1438 ±  3%  perf-stat.overall.path-length
 1.553e+08           -39.2%   94431791 ±  4%  perf-stat.ps.branch-instructions
   6190691 ±  2%     -23.1%    4763546 ±  6%  perf-stat.ps.branch-misses
    589453 ± 16%    +225.8%    1920470 ±  7%  perf-stat.ps.dTLB-load-misses
 2.715e+08           -37.5%  1.696e+08 ±  4%  perf-stat.ps.dTLB-loads
    170463 ± 23%    +765.6%    1475492 ±  5%  perf-stat.ps.dTLB-store-misses
   5694375 ±  2%     -50.0%    2846946 ±  5%  perf-stat.ps.iTLB-load-misses
   5512250 ±  2%     -26.8%    4035405 ±  4%  perf-stat.ps.iTLB-loads
  8.61e+08           -39.5%  5.209e+08 ±  4%  perf-stat.ps.instructions
 3.452e+11           -36.1%  2.204e+11 ±  4%  perf-stat.total.instructions
     12.07 ± 22%      -9.1        3.00 ±100%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
     10.62 ± 23%      -7.7        2.95 ±102%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      1.79 ± 22%      -1.3        0.53 ±102%  perf-profile.calltrace.cycles-pp.__x64_sys_dup.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.59 ± 22%      -1.1        0.46 ±103%  perf-profile.calltrace.cycles-pp.ksys_dup.__x64_sys_dup.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.35 ± 23%      -1.0        0.33 ±103%  perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.37 ± 13%      -0.9        2.42 ± 25%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
      3.37 ± 13%      -0.9        2.42 ± 25%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
      3.36 ± 13%      -0.9        2.42 ± 25%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
      2.97 ± 12%      -0.8        2.17 ± 24%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
     13.51 ± 15%      -5.8        7.72 ± 14%  perf-profile.children.cycles-pp.syscall_return_via_sysret
     13.27 ± 15%      -5.7        7.55 ± 20%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      3.37 ± 13%      -0.9        2.42 ± 25%  perf-profile.children.cycles-pp.start_kernel
      2.02 ± 12%      -0.9        1.14 ± 18%  perf-profile.children.cycles-pp.__x64_sys_dup
      1.78 ± 11%      -0.8        1.01 ± 17%  perf-profile.children.cycles-pp.ksys_dup
      1.49 ± 12%      -0.7        0.76 ± 13%  perf-profile.children.cycles-pp.__x64_sys_close
      2.05 ± 12%      -0.4        1.64 ± 18%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.95 ± 19%      -0.4        0.54 ± 20%  perf-profile.children.cycles-pp.__x64_sys_getpid
      0.86 ± 21%      -0.4        0.48 ± 16%  perf-profile.children.cycles-pp.__x64_sys_getuid
      0.69 ± 23%      -0.4        0.33 ± 19%  perf-profile.children.cycles-pp.__x64_sys_umask
      0.77 ± 15%      -0.4        0.42 ± 16%  perf-profile.children.cycles-pp.__alloc_fd
      0.72 ± 17%      -0.3        0.41 ± 27%  perf-profile.children.cycles-pp.__fget
      0.54 ± 18%      -0.3        0.23 ± 18%  perf-profile.children.cycles-pp.__close_fd
      0.55 ± 22%      -0.3        0.29 ± 15%  perf-profile.children.cycles-pp.filp_close
      0.58 ± 15%      -0.3        0.32 ± 23%  perf-profile.children.cycles-pp.__task_pid_nr_ns
      0.75 ± 13%      -0.2        0.51 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock
      0.25 ± 17%      -0.1        0.12 ± 26%  perf-profile.children.cycles-pp.fput
      0.26 ± 33%      -0.1        0.14 ± 20%  perf-profile.children.cycles-pp.from_kuid_munged
      0.24 ± 37%      -0.1        0.14 ± 19%  perf-profile.children.cycles-pp.map_id_up
      0.12 ± 30%      -0.0        0.07 ± 15%  perf-profile.children.cycles-pp.account_process_tick
      8.70 ± 15%     +14.5       23.18 ± 13%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      7.73 ± 15%     +14.7       22.45 ± 12%  perf-profile.children.cycles-pp.do_syscall_64
     13.13 ± 16%      -6.2        6.93 ± 13%  perf-profile.self.cycles-pp.entry_SYSCALL_64
     13.51 ± 15%      -5.8        7.71 ± 14%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.66 ± 22%      -0.3        0.31 ± 23%  perf-profile.self.cycles-pp.__x64_sys_umask
      0.71 ± 16%      -0.3        0.41 ± 27%  perf-profile.self.cycles-pp.__fget
      0.58 ± 14%      -0.3        0.32 ± 23%  perf-profile.self.cycles-pp.__task_pid_nr_ns
      0.73 ± 12%      -0.2        0.48 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock
      0.31 ± 14%      -0.2        0.14 ± 35%  perf-profile.self.cycles-pp.__alloc_fd
      0.35 ±  7%      -0.2        0.19 ±  9%  perf-profile.self.cycles-pp.__x64_sys_close
      0.24 ± 27%      -0.1        0.12 ± 20%  perf-profile.self.cycles-pp.__close_fd
      0.24 ± 17%      -0.1        0.12 ± 26%  perf-profile.self.cycles-pp.fput
      0.23 ± 37%      -0.1        0.13 ± 21%  perf-profile.self.cycles-pp.map_id_up
      0.22 ± 28%      -0.1        0.13 ± 17%  perf-profile.self.cycles-pp.__x64_sys_dup
      0.12 ± 30%      -0.0        0.07 ± 15%  perf-profile.self.cycles-pp.account_process_tick
      0.11 ± 13%      -0.0        0.07 ± 14%  perf-profile.self.cycles-pp.lapic_next_deadline
      1.26 ± 26%     +17.6       18.84 ± 12%  perf-profile.self.cycles-pp.do_syscall_64


                                                                                
                             unixbench.time.user_time                           
                                                                                
  210 +-+O-O-OO-OO-O-OO-O-O--O--O-OO---OO-OO---O--O-O---O-O-----------------+   
  200 O-O                  O  O      O       O  O    O O                    |   
      |                                                                     |   
  190 +-+                                                                   |   
  180 +-+                                                                   |   
      |                                                                     |   
  170 +-+                                                                   |   
  160 +-+                                                                   |   
  150 +-+                                                                   |   
      |                                                                     |   
  140 +-+                                                                   |   
  130 +-+                                                                   |   
      |                                                                     |   
  120 +-++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.|   
  110 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            unixbench.time.system_time                          
                                                                                
  190 +-+-------------------------------------------------------------------+   
  180 +-++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.+.++.++.+.++.|   
      |                                                                     |   
  170 +-+                                                                   |   
  160 +-+                                                                   |   
      |                                                                     |   
  150 +-+                                                                   |   
  140 +-+                                                                   |   
  130 +-+                                                                   |   
      |                                                                     |   
  120 +-+                                                                   |   
  110 +-+                                                                   |   
      |                                                                     |   
  100 O-O                  O  O      O O     O  O O OO O  O                 |   
   90 +-+O-O-OO-OO-O-OO-O-O--O--O-OO----O-OO---O--------O-------------------+   
                                                                                
                                                                                                                                                                
                                  unixbench.score                               
                                                                                
  500 +-+-------------------------------------------------------------------+   
      |  :.+  +.  .+. +.+.+ + +   ++  + +  +.+  :+    + :     + + :.++. .++.|   
  450 +-++      ++   +       +         +        +      +       +  +    +    |   
      |                                                                     |   
      |                                                                     |   
  400 +-+                                                                   |   
      |                                                                     |   
  350 +-+                                                                   |   
      |                                                                     |   
  300 +-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
  250 O-OO O OO OO O OO O OO OO O OO O OO OO O OO O OO OO O                 |   
      |                                                                     |   
  200 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/30%/debian-x86_64-2019-05-14.cgz/300s/lkp-bdw-de1/fstime/unixbench/0x7000017

commit: 
  07f07f55a2 ("x86/speculation/mds: Conditionally clear CPU buffers on idle entry")
  bc1241700a ("x86/speculation/mds: Add mitigation control for MDS")

07f07f55a29cb705 bc1241700acd82ec69fde98c576 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          2:4          -50%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      3769           +24.4%       4690        unixbench.score
      1228           -27.6%     889.80        unixbench.time.system_time
    131.48          +257.9%     470.62 ±  2%  unixbench.time.user_time
     23866 ± 21%     -47.3%      12574 ±  8%  unixbench.time.voluntary_context_switches
 7.275e+08           +23.5%  8.986e+08 ±  2%  unixbench.workload
      4908            -3.0%       4759        proc-vmstat.nr_shmem
   2133531 ± 13%     -65.2%     741612 ± 19%  cpuidle.C1.time
     72587 ±  6%     -21.7%      56857        cpuidle.POLL.usage
     15.87            -4.2       11.67        mpstat.cpu.all.sys%
      1.78            +4.5        6.25 ±  2%  mpstat.cpu.all.usr%
    686.25 ±  2%     -14.6%     585.75 ±  2%  slabinfo.dmaengine-unmap-16.active_objs
    686.25 ±  2%     -14.6%     585.75 ±  2%  slabinfo.dmaengine-unmap-16.num_objs
     29862 ± 10%     +84.8%      55170 ± 68%  softirqs.CPU10.RCU
    238611 ± 10%     -14.2%     204700 ±  9%  softirqs.CPU12.TIMER
      0.03 ± 15%      -0.0        0.01        turbostat.C1%
     24.45            +2.4%      25.04        turbostat.PkgWatt
    125.75 ± 50%    +216.3%     397.75 ± 56%  interrupts.CPU1.RES:Rescheduling_interrupts
    270.75 ± 37%     -54.7%     122.75 ± 40%  interrupts.CPU10.RES:Rescheduling_interrupts
     20130 ±  7%     -30.8%      13934 ± 25%  interrupts.NMI:Non-maskable_interrupts
     20130 ±  7%     -30.8%      13934 ± 25%  interrupts.PMI:Performance_monitoring_interrupts
     92.78 ±  7%     -14.5%      79.29 ±  4%  sched_debug.cfs_rq:/.load_avg.avg
     15.84 ± 37%     -88.8%       1.78 ±173%  sched_debug.cfs_rq:/.removed.load_avg.avg
    169.39 ± 57%     -83.2%      28.44 ±173%  sched_debug.cfs_rq:/.removed.load_avg.max
     47.56 ± 43%     -85.5%       6.89 ±173%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    731.19 ± 37%     -88.8%      81.97 ±173%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      7826 ± 57%     -83.2%       1311 ±173%  sched_debug.cfs_rq:/.removed.runnable_sum.max
      2194 ± 43%     -85.5%     317.46 ±173%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      4.79 ± 58%     -81.2%       0.90 ±173%  sched_debug.cfs_rq:/.removed.util_avg.avg
     15.41 ± 63%     -77.4%       3.49 ±173%  sched_debug.cfs_rq:/.removed.util_avg.stddev
   -632.56         +2487.5%     -16367        sched_debug.cfs_rq:/.spread0.min
    187554 ± 18%     +52.3%     285588 ±  8%  sched_debug.cpu.avg_idle.min
    260817 ±  7%     -15.1%     221319 ±  6%  sched_debug.cpu.avg_idle.stddev
    433438 ± 10%     -21.5%     340049 ± 20%  sched_debug.cpu.load.max
    141067 ±  7%     -16.4%     117948 ± 13%  sched_debug.cpu.load.stddev
      2646 ± 15%     -23.7%       2018 ±  4%  sched_debug.cpu.nr_load_updates.stddev
     11.39 ± 18%     -30.5%       7.92 ± 17%  sched_debug.cpu.nr_uninterruptible.max
     36.46 ±  2%     -10.1%      32.77 ±  3%  perf-stat.i.MPKI
  15449568 ±  3%      +6.6%   16466252        perf-stat.i.branch-misses
 1.061e+08 ±  4%     -21.8%   82968244 ±  7%  perf-stat.i.cache-misses
 1.061e+08 ±  4%     -21.8%   82968244 ±  7%  perf-stat.i.cache-references
      2.71 ±  8%     -22.4%       2.10 ±  8%  perf-stat.i.cpu-migrations
     73.27 ±  2%     +23.2%      90.24 ±  4%  perf-stat.i.cycles-between-cache-misses
      0.29 ± 18%      +0.2        0.45 ± 13%  perf-stat.i.dTLB-load-miss-rate%
    464680 ± 11%    +574.6%    3134877 ±  2%  perf-stat.i.dTLB-load-misses
      0.06 ± 20%      +0.2        0.23 ±  2%  perf-stat.i.dTLB-store-miss-rate%
    135020 ± 19%   +1816.3%    2587411 ±  2%  perf-stat.i.dTLB-store-misses
  8.64e+08 ±  2%     +24.6%  1.077e+09 ±  2%  perf-stat.i.dTLB-stores
   4539544 ±  3%     +14.0%    5175003 ±  2%  perf-stat.i.iTLB-load-misses
   4542106 ±  2%     +71.4%    7784890        perf-stat.i.iTLB-loads
     22.35 ±  3%     -23.1%      17.20 ±  7%  perf-stat.overall.MPKI
      1.62            +0.1        1.71        perf-stat.overall.branch-miss-rate%
      1.58            -1.6%       1.56        perf-stat.overall.cpi
     70.79 ±  2%     +28.4%      90.91 ±  6%  perf-stat.overall.cycles-between-cache-misses
      0.03 ± 13%      +0.2        0.20 ±  2%  perf-stat.overall.dTLB-load-miss-rate%
      0.02 ± 20%      +0.2        0.24        perf-stat.overall.dTLB-store-miss-rate%
     49.98           -10.1       39.93        perf-stat.overall.iTLB-load-miss-rate%
      1046           -10.9%     932.22        perf-stat.overall.instructions-per-iTLB-miss
      3203           -18.8%       2600 ±  2%  perf-stat.overall.path-length
  15418101 ±  3%      +6.6%   16431516        perf-stat.ps.branch-misses
 1.059e+08 ±  4%     -21.8%   82792020 ±  7%  perf-stat.ps.cache-misses
 1.059e+08 ±  4%     -21.8%   82792020 ±  7%  perf-stat.ps.cache-references
      2.70 ±  8%     -22.4%       2.10 ±  8%  perf-stat.ps.cpu-migrations
    463749 ± 11%    +574.6%    3128242 ±  2%  perf-stat.ps.dTLB-load-misses
    134749 ± 19%   +1816.1%    2581911 ±  2%  perf-stat.ps.dTLB-store-misses
 8.622e+08 ±  2%     +24.6%  1.074e+09 ±  2%  perf-stat.ps.dTLB-stores
   4529928 ±  3%     +14.0%    5163989 ±  2%  perf-stat.ps.iTLB-load-misses
   4532481 ±  2%     +71.4%    7768305        perf-stat.ps.iTLB-loads
     13.18 ± 19%      -8.1        5.12 ± 63%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.72 ± 19%      -7.2        4.49 ± 63%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.85 ± 18%      -5.5        3.39 ± 62%  perf-profile.calltrace.cycles-pp.__vfs_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.41 ± 18%      -5.2        3.21 ± 62%  perf-profile.calltrace.cycles-pp.generic_file_read_iter.__vfs_read.vfs_read.ksys_read.do_syscall_64
      5.11 ± 16%      -2.7        2.42 ± 59%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
      4.88 ± 11%      -2.5        2.33 ± 60%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
     11.40 ± 12%      -2.4        8.96 ±  8%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.generic_file_write_iter
      3.63 ± 19%      -2.3        1.34 ± 65%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.generic_file_read_iter.__vfs_read.vfs_read.ksys_read
      2.46 ± 18%      -1.6        0.90 ± 66%  perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.generic_file_read_iter.__vfs_read.vfs_read
      2.32 ± 18%      -1.5        0.85 ± 67%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.generic_file_read_iter.__vfs_read
      1.45 ± 20%      -1.0        0.45 ±100%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.72 ± 21%      -1.0        0.73 ± 61%  perf-profile.calltrace.cycles-pp.pagecache_get_page.generic_file_read_iter.__vfs_read.vfs_read.ksys_read
      1.10 ± 21%      -0.7        0.35 ±100%  perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.82 ± 10%      +0.1        0.95 ±  5%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.simple_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      0.98 ± 10%      +0.1        1.12 ±  3%  perf-profile.calltrace.cycles-pp.simple_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.__vfs_write
      0.76 ± 10%      +0.2        0.92 ±  6%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.simple_write_begin.generic_perform_write.__generic_file_write_iter
      0.30 ±100%      +0.4        0.67 ± 10%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.simple_write_begin.generic_perform_write
      0.00            +0.7        0.65 ± 16%  perf-profile.calltrace.cycles-pp.file_update_time.__generic_file_write_iter.generic_file_write_iter.__vfs_write.vfs_write
     13.29 ± 19%      -8.0        5.26 ± 61%  perf-profile.children.cycles-pp.ksys_read
     11.82 ± 19%      -7.2        4.62 ± 61%  perf-profile.children.cycles-pp.vfs_read
      8.93 ± 18%      -5.4        3.50 ± 60%  perf-profile.children.cycles-pp.__vfs_read
      8.46 ± 18%      -5.2        3.23 ± 62%  perf-profile.children.cycles-pp.generic_file_read_iter
      5.66 ± 12%      -2.8        2.90 ± 39%  perf-profile.children.cycles-pp.syscall_return_via_sysret
     11.43 ± 12%      -2.5        8.98 ±  8%  perf-profile.children.cycles-pp.osq_lock
      5.11 ± 16%      -2.4        2.71 ± 34%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      3.68 ± 19%      -2.3        1.36 ± 65%  perf-profile.children.cycles-pp.copy_page_to_iter
      2.50 ± 17%      -1.6        0.91 ± 66%  perf-profile.children.cycles-pp.copyout
      2.96 ± 12%      -1.6        1.40 ± 43%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      2.52 ± 13%      -0.9        1.67 ± 23%  perf-profile.children.cycles-pp.pagecache_get_page
      1.77 ± 16%      -0.8        0.95 ± 41%  perf-profile.children.cycles-pp.security_file_permission
      1.36 ± 17%      -0.6        0.77 ± 39%  perf-profile.children.cycles-pp.selinux_file_permission
      0.83 ± 24%      -0.6        0.28 ± 59%  perf-profile.children.cycles-pp.fsnotify
      0.69 ± 13%      -0.5        0.22 ± 61%  perf-profile.children.cycles-pp.touch_atime
      0.55 ± 12%      -0.4        0.17 ± 61%  perf-profile.children.cycles-pp.atime_needs_update
      0.71 ± 19%      -0.3        0.39 ± 27%  perf-profile.children.cycles-pp.xas_load
      0.73 ± 10%      -0.3        0.45 ± 21%  perf-profile.children.cycles-pp.___might_sleep
      0.47 ± 20%      -0.3        0.20 ± 61%  perf-profile.children.cycles-pp.__x64_sys_read
      0.44 ±  5%      -0.2        0.21 ± 33%  perf-profile.children.cycles-pp.current_time
      0.46 ± 18%      -0.2        0.23 ± 34%  perf-profile.children.cycles-pp.__might_sleep
      0.36 ± 21%      -0.2        0.16 ± 60%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.31 ± 26%      -0.2        0.12 ± 67%  perf-profile.children.cycles-pp.__might_fault
      0.41 ± 13%      -0.2        0.25 ± 25%  perf-profile.children.cycles-pp._cond_resched
      0.19 ± 37%      -0.1        0.11 ± 19%  perf-profile.children.cycles-pp.set_page_dirty
      0.14 ± 19%      -0.1        0.06 ± 63%  perf-profile.children.cycles-pp.timespec64_trunc
      0.18 ±  8%      -0.1        0.12 ± 32%  perf-profile.children.cycles-pp.rcu_all_qs
      0.18 ± 15%      -0.1        0.12 ± 27%  perf-profile.children.cycles-pp.xas_start
      0.07 ± 12%      +0.0        0.11 ± 10%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.07 ± 26%      +0.0        0.11 ± 24%  perf-profile.children.cycles-pp.cap_inode_need_killpriv
      0.11 ± 19%      +0.0        0.15 ±  7%  perf-profile.children.cycles-pp.__set_page_dirty_no_writeback
      0.10 ± 19%      +0.0        0.14 ± 18%  perf-profile.children.cycles-pp.security_inode_need_killpriv
      0.14 ± 22%      +0.0        0.18 ± 19%  perf-profile.children.cycles-pp.dentry_needs_remove_privs
      0.11 ±  8%      +0.0        0.15 ± 27%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.08 ±  8%      +0.1        0.13 ± 14%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.07 ± 27%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.22 ± 14%      +0.1        0.31 ± 14%  perf-profile.children.cycles-pp.file_remove_privs
      0.01 ±173%      +0.1        0.11 ± 78%  perf-profile.children.cycles-pp.read
      0.29 ± 11%      +0.1        0.40 ± 19%  perf-profile.children.cycles-pp.rebalance_domains
      0.83 ±  9%      +0.1        0.97 ±  5%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      0.98 ± 10%      +0.1        1.13 ±  3%  perf-profile.children.cycles-pp.simple_write_begin
      0.33 ± 18%      +0.2        0.51 ± 17%  perf-profile.children.cycles-pp.delay_tsc
      0.39 ±  5%      +0.3        0.66 ± 15%  perf-profile.children.cycles-pp.file_update_time
      0.47 ± 21%      +0.3        0.77 ± 18%  perf-profile.children.cycles-pp.io_serial_in
      1.17 ±  6%      +0.4        1.53 ± 15%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.47 ±  6%      +0.4        1.87 ± 13%  perf-profile.children.cycles-pp.irq_exit
      0.80 ± 18%      +0.5        1.27 ± 17%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.80 ± 19%      +0.5        1.28 ± 17%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.82 ± 19%      +0.5        1.31 ± 17%  perf-profile.children.cycles-pp.serial8250_console_write
      0.81 ± 18%      +0.5        1.30 ± 17%  perf-profile.children.cycles-pp.uart_console_write
      0.73 ± 14%      +0.5        1.24 ± 17%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.73 ± 14%      +0.5        1.24 ± 17%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.73 ± 14%      +0.5        1.24 ± 17%  perf-profile.children.cycles-pp.irq_work_run
      0.73 ± 14%      +0.5        1.24 ± 17%  perf-profile.children.cycles-pp.printk
      0.73 ± 14%      +0.5        1.24 ± 17%  perf-profile.children.cycles-pp.vprintk_emit
      0.87 ± 19%      +0.5        1.40 ± 18%  perf-profile.children.cycles-pp.console_unlock
      0.85 ± 24%      +0.6        1.41 ± 18%  perf-profile.children.cycles-pp.irq_work_run_list
      5.66 ± 12%      -2.8        2.89 ± 39%  perf-profile.self.cycles-pp.syscall_return_via_sysret
     11.39 ± 12%      -2.5        8.91 ±  8%  perf-profile.self.cycles-pp.osq_lock
      5.11 ± 16%      -2.4        2.68 ± 36%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      2.92 ± 12%      -1.5        1.39 ± 43%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      2.12 ± 19%      -1.3        0.83 ± 59%  perf-profile.self.cycles-pp.generic_file_read_iter
      0.80 ± 24%      -0.5        0.27 ± 60%  perf-profile.self.cycles-pp.fsnotify
      0.78 ± 11%      -0.3        0.44 ± 45%  perf-profile.self.cycles-pp.selinux_file_permission
      0.56 ± 26%      -0.3        0.23 ± 62%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.68 ± 11%      -0.2        0.43 ± 21%  perf-profile.self.cycles-pp.___might_sleep
      0.51 ± 22%      -0.2        0.27 ± 30%  perf-profile.self.cycles-pp.xas_load
      0.39 ± 16%      -0.2        0.16 ± 60%  perf-profile.self.cycles-pp.__x64_sys_read
      0.43 ± 14%      -0.2        0.21 ± 30%  perf-profile.self.cycles-pp.__might_sleep
      0.36 ± 11%      -0.2        0.15 ± 70%  perf-profile.self.cycles-pp.__vfs_read
      0.41 ± 20%      -0.2        0.21 ± 65%  perf-profile.self.cycles-pp.vfs_read
      0.32 ± 21%      -0.2        0.14 ± 60%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.39 ± 15%      -0.2        0.21 ± 67%  perf-profile.self.cycles-pp.ksys_read
      0.28 ± 11%      -0.2        0.11 ± 61%  perf-profile.self.cycles-pp.security_file_permission
      0.22 ± 20%      -0.2        0.07 ± 60%  perf-profile.self.cycles-pp.atime_needs_update
      0.24 ± 17%      -0.1        0.10 ± 30%  perf-profile.self.cycles-pp.current_time
      0.22 ±  6%      -0.1        0.09 ± 62%  perf-profile.self.cycles-pp.__fdget_pos
      0.34 ± 10%      -0.1        0.22 ± 22%  perf-profile.self.cycles-pp.pagecache_get_page
      0.15 ± 24%      -0.1        0.05 ± 63%  perf-profile.self.cycles-pp.touch_atime
      0.12 ±  8%      -0.1        0.05 ± 62%  perf-profile.self.cycles-pp.rcu_all_qs
      0.17 ± 15%      -0.1        0.10 ± 17%  perf-profile.self.cycles-pp.xas_start
      0.06 ±  6%      +0.0        0.10 ± 10%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.10 ± 21%      +0.0        0.14 ± 11%  perf-profile.self.cycles-pp.__set_page_dirty_no_writeback
      0.10 ± 15%      +0.1        0.16 ± 20%  perf-profile.self.cycles-pp.vfs_write
      0.00            +0.1        0.07 ± 27%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.33 ± 18%      +0.2        0.51 ± 17%  perf-profile.self.cycles-pp.delay_tsc
      0.51 ± 14%      +0.2        0.70 ± 14%  perf-profile.self.cycles-pp.down_write
      0.29 ±  7%      +0.3        0.56 ± 18%  perf-profile.self.cycles-pp.file_update_time
      0.47 ± 21%      +0.3        0.77 ± 18%  perf-profile.self.cycles-pp.io_serial_in
      0.45 ± 12%      +6.8        7.21 ± 36%  perf-profile.self.cycles-pp.do_syscall_64



***************************************************************************************************
lkp-ivb-d01: 8 threads Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz with 16G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-2019-05-14.cgz/lkp-ivb-d01/lseek2/will-it-scale/0x21

commit: 
  07f07f55a2 ("x86/speculation/mds: Conditionally clear CPU buffers on idle entry")
  bc1241700a ("x86/speculation/mds: Add mitigation control for MDS")

07f07f55a29cb705 bc1241700acd82ec69fde98c576 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.RIP:get_page_from_freelist
          1:4          -25%            :4     dmesg.RIP:raid#_sse24_xor_syndrome[raid#_pq]
          1:4          -25%            :4     kmsg.c94d030>]usb_hcd_irq
          1:4          -25%            :4     kmsg.cfff0>]usb_hcd_irq
          1:4          -25%            :4     kmsg.d945d>]usb_hcd_irq
           :4           25%           1:4     kmsg.f96e4a>]usb_hcd_irq
         %stddev     %change         %stddev
             \          |                \  
   3581345           -49.5%    1806894        will-it-scale.per_process_ops
  28650765           -49.5%   14455155        will-it-scale.workload
     19168 ±  5%     +17.1%      22438 ±  8%  softirqs.CPU5.RCU
   1267339 ± 54%    +174.1%    3474327 ± 15%  cpuidle.C1E.time
     15021 ± 22%     +98.2%      29779 ± 10%  cpuidle.C1E.usage
     63.44           -19.7       43.74        mpstat.cpu.all.sys%
     35.89           +19.7       55.58        mpstat.cpu.all.usr%
     62.50           -31.6%      42.75        vmstat.cpu.sy
     35.25 ±  2%     +56.0%      55.00        vmstat.cpu.us
     14835 ± 23%     +99.4%      29587 ± 11%  turbostat.C1E
      0.05 ± 52%      +0.1        0.14 ± 15%  turbostat.C1E%
      0.20 ± 12%     +41.8%       0.28 ± 20%  turbostat.CPU%c1
     39.80            -6.7%      37.14        turbostat.CorWatt
     44.74            -6.0%      42.06        turbostat.PkgWatt
      1825 ±173%    +241.1%       6228 ± 60%  interrupts.CPU0.NMI:Non-maskable_interrupts
      1825 ±173%    +241.1%       6228 ± 60%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      1211 ± 38%     +57.2%       1903 ± 18%  interrupts.CPU1.RES:Rescheduling_interrupts
    634870 ±  7%     +14.1%     724680 ± 10%  interrupts.CPU2.LOC:Local_timer_interrupts
    664277 ± 14%     +22.9%     816662 ± 11%  interrupts.CPU3.LOC:Local_timer_interrupts
     10168 ± 12%     -15.1%       8635 ± 19%  interrupts.RES:Rescheduling_interrupts
     18856 ± 11%     +23.0%      23185 ±  7%  sched_debug.cfs_rq:/.min_vruntime.stddev
     12251 ± 32%     +79.8%      22024 ± 20%  sched_debug.cfs_rq:/.spread0.avg
     36631 ± 10%     +49.1%      54621 ± 11%  sched_debug.cfs_rq:/.spread0.max
     18841 ± 11%     +22.9%      23161 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
      3.11 ± 20%     +49.9%       4.66 ±  6%  sched_debug.cpu.clock.stddev
      3.11 ± 20%     +49.9%       4.66 ±  6%  sched_debug.cpu.clock_task.stddev
    180.33 ±  3%      +6.9%     192.71 ±  4%  sched_debug.cpu.cpu_load[3].max
     29.03 ±  8%     +16.0%      33.68 ±  6%  sched_debug.cpu.cpu_load[3].stddev
     23.46 ± 10%     +15.1%      27.00 ±  7%  sched_debug.cpu.cpu_load[4].stddev
   1341621 ± 32%     -57.4%     570980 ±  4%  sched_debug.cpu.nr_switches.max
 1.597e+09           -49.6%  8.051e+08        perf-stat.i.branch-instructions
      3.65            +0.1        3.76        perf-stat.i.branch-miss-rate%
  58311170           -47.9%   30373173        perf-stat.i.branch-misses
      3.65           +94.6%       7.11        perf-stat.i.cpi
  99509625 ± 20%     -50.3%   49457889 ±  8%  perf-stat.i.dTLB-load-misses
  2.39e+09           -45.7%  1.296e+09        perf-stat.i.dTLB-loads
      0.00 ± 52%      +1.9        1.87        perf-stat.i.dTLB-store-miss-rate%
     14216 ± 45%    +2e+05%   28777621        perf-stat.i.dTLB-store-misses
 1.591e+09            -5.7%    1.5e+09        perf-stat.i.dTLB-stores
     73.53 ±  2%     -15.1       58.39 ±  2%  perf-stat.i.iTLB-load-miss-rate%
  23806123 ± 11%     -72.9%    6453671 ±  3%  perf-stat.i.iTLB-load-misses
   8694856 ± 10%     -43.7%    4894589 ±  7%  perf-stat.i.iTLB-loads
 8.014e+09           -48.5%  4.125e+09        perf-stat.i.instructions
    366.71 ± 17%     +85.2%     679.30 ±  4%  perf-stat.i.instructions-per-iTLB-miss
      0.28           -47.9%       0.14        perf-stat.i.ipc
      0.09 ± 15%     +73.9%       0.15 ± 16%  perf-stat.overall.MPKI
      3.65            +0.1        3.77        perf-stat.overall.branch-miss-rate%
      3.65           +94.1%       7.09        perf-stat.overall.cpi
      0.00 ± 46%      +1.9        1.88        perf-stat.overall.dTLB-store-miss-rate%
     73.16 ±  2%     -16.3       56.86 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
    341.86 ± 13%     +87.0%     639.20 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      0.27           -48.5%       0.14        perf-stat.overall.ipc
     84701            +1.8%      86264        perf-stat.overall.path-length
 1.593e+09           -49.6%  8.028e+08        perf-stat.ps.branch-instructions
  58173755           -48.0%   30265784        perf-stat.ps.branch-misses
  99287290 ± 20%     -50.5%   49155670 ±  8%  perf-stat.ps.dTLB-load-misses
 2.384e+09           -45.7%  1.294e+09        perf-stat.ps.dTLB-loads
     14186 ± 45%    +2e+05%   28726446        perf-stat.ps.dTLB-store-misses
 1.588e+09            -5.7%  1.497e+09        perf-stat.ps.dTLB-stores
  23747578 ± 11%     -72.9%    6443410 ±  3%  perf-stat.ps.iTLB-load-misses
   8673919 ± 10%     -43.5%    4897731 ±  7%  perf-stat.ps.iTLB-loads
 7.995e+09           -48.5%  4.114e+09        perf-stat.ps.instructions
 2.427e+12           -48.6%  1.247e+12        perf-stat.total.instructions
     40.84           -19.3       21.55 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.lseek64
     15.25 ±  7%      -8.0        7.25 ±  7%  perf-profile.calltrace.cycles-pp.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
      7.79 ± 10%      -4.2        3.64 ± 10%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
      6.89 ±  7%      -3.7        3.19 ±  5%  perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe
     28.69 ±  2%      -2.8       25.88        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.lseek64
      5.18 ± 17%      -2.8        2.37 ± 10%  perf-profile.calltrace.cycles-pp.shmem_file_llseek.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
      2.68 ±  5%      -1.9        0.79 ±  3%  perf-profile.calltrace.cycles-pp.testcase
      2.60 ±  2%      -1.3        1.32        perf-profile.calltrace.cycles-pp.__x64_sys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
     97.98            +1.1       99.13        perf-profile.calltrace.cycles-pp.lseek64
      0.00            +4.1        4.14 ± 98%  perf-profile.calltrace.cycles-pp.__x86_indirect_thunk_rax.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
     25.14 ±  4%     +17.7       42.83 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
     27.92 ±  3%     +22.4       50.36 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.lseek64
     36.88           -17.2       19.67 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64
     15.46 ±  7%      -8.1        7.36 ±  7%  perf-profile.children.cycles-pp.ksys_lseek
     32.52 ±  2%      -4.6       27.97        perf-profile.children.cycles-pp.syscall_return_via_sysret
      7.98 ±  9%      -4.2        3.76 ± 10%  perf-profile.children.cycles-pp.__fdget_pos
      6.89 ±  7%      -3.7        3.20 ±  5%  perf-profile.children.cycles-pp.__fget_light
      5.18 ± 17%      -2.8        2.38 ± 10%  perf-profile.children.cycles-pp.shmem_file_llseek
      2.60 ±  2%      -1.3        1.32        perf-profile.children.cycles-pp.__x64_sys_lseek
      1.72 ±  4%      -0.9        0.85 ±  4%  perf-profile.children.cycles-pp.testcase
      0.48 ±  6%      -0.2        0.27 ± 11%  perf-profile.children.cycles-pp.generic_file_llseek_size
      0.22 ±  7%      -0.1        0.11 ±  9%  perf-profile.children.cycles-pp.lseek@plt
      0.11 ±  6%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
     99.18            +0.3       99.44        perf-profile.children.cycles-pp.lseek64
      1.09 ±  7%      +1.6        2.68 ± 77%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
     25.23 ±  4%     +19.8       45.06 ±  5%  perf-profile.children.cycles-pp.do_syscall_64
     28.23 ±  3%     +22.4       50.66 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     32.88           -15.1       17.75 ±  6%  perf-profile.self.cycles-pp.entry_SYSCALL_64
     32.49 ±  2%      -4.6       27.93        perf-profile.self.cycles-pp.syscall_return_via_sysret
      6.76 ±  8%      -3.6        3.11 ±  6%  perf-profile.self.cycles-pp.__fget_light
      4.84 ± 18%      -2.6        2.20 ± 11%  perf-profile.self.cycles-pp.shmem_file_llseek
      5.18 ±  2%      -2.3        2.86 ± 18%  perf-profile.self.cycles-pp.lseek64
      2.17 ±  3%      -1.1        1.10        perf-profile.self.cycles-pp.__x64_sys_lseek
      1.89 ±  4%      -0.9        0.99 ±  3%  perf-profile.self.cycles-pp.ksys_lseek
      1.12 ± 34%      -0.5        0.60 ± 36%  perf-profile.self.cycles-pp.__fdget_pos
      0.98 ±  8%      -0.4        0.55 ±  6%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.70 ±  5%      -0.3        0.38 ±  9%  perf-profile.self.cycles-pp.testcase
      0.48 ±  5%      -0.2        0.27 ± 12%  perf-profile.self.cycles-pp.generic_file_llseek_size
      0.16 ±  7%      -0.1        0.06 ± 16%  perf-profile.self.cycles-pp.lseek@plt
      6.78 ±  7%     +29.2       36.02 ±  5%  perf-profile.self.cycles-pp.do_syscall_64



***************************************************************************************************
lkp-ivb-d01: 8 threads Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz with 16G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-2019-05-14.cgz/lkp-ivb-d01/poll1/will-it-scale/0x21

commit: 
  07f07f55a2 ("x86/speculation/mds: Conditionally clear CPU buffers on idle entry")
  bc1241700a ("x86/speculation/mds: Add mitigation control for MDS")

07f07f55a29cb705 bc1241700acd82ec69fde98c576 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.RIP:__do_softirq
          2:4          -50%            :4     dmesg.RIP:cpuidle_enter_state
          1:4          -25%            :4     kmsg.b9>]usb_hcd_irq
          1:4          -25%            :4     kmsg.b9a7595>]usb_hcd_irq
           :4           25%           1:4     kmsg.ebf31>]usb_hcd_irq
          1:4          -25%            :4     kmsg.f9ea22>]usb_hcd_irq
           :4           50%           2:4     kmsg.usb_hcd_irq
         %stddev     %change         %stddev
             \          |                \  
   2814601           -43.8%    1581301        will-it-scale.per_thread_ops
      7861            -1.2%       7764        will-it-scale.time.maximum_resident_set_size
      1583           -23.9%       1204        will-it-scale.time.system_time
    815.42           +46.3%       1193        will-it-scale.time.user_time
  22516811           -43.8%   12650414        will-it-scale.workload
    212.58 ±  6%     +22.2%     259.83 ± 18%  sched_debug.cpu.cpu_load[0].max
     23.02 ±  2%     +17.7%      27.08 ±  8%  boot-time.boot
    163.21 ±  3%     +19.8%     195.45 ±  9%  boot-time.idle
     65.40           -15.7       49.66        mpstat.cpu.all.sys%
     33.97           +15.3       49.27        mpstat.cpu.all.usr%
      1572 ±  8%     +15.0%       1808 ±  4%  slabinfo.kmalloc-512.active_objs
      1589 ±  8%     +13.7%       1808 ±  4%  slabinfo.kmalloc-512.num_objs
     64.75           -24.3%      49.00        vmstat.cpu.sy
     33.00           +47.0%      48.50        vmstat.cpu.us
   2271859 ± 98%    +388.2%   11090568 ± 44%  cpuidle.C3.time
      8257 ± 97%    +252.7%      29122 ± 33%  cpuidle.C3.usage
     15968 ± 25%     +94.4%      31042 ± 37%  cpuidle.C6.usage
     19348 ±  2%     +12.5%      21768        softirqs.CPU1.RCU
    112937 ±  5%     +23.9%     139973 ± 12%  softirqs.CPU3.TIMER
    122073 ±  8%     -11.1%     108556 ±  2%  softirqs.CPU5.TIMER
      8164 ± 98%    +255.1%      28991 ± 33%  turbostat.C3
      0.09 ±100%      +0.4        0.46 ± 43%  turbostat.C3%
     15842 ± 25%     +95.0%      30898 ± 37%  turbostat.C6
      0.25 ±100%    +311.0%       1.03 ± 37%  turbostat.CPU%c3
     42.15            -8.8%      38.45        turbostat.CorWatt
      0.04 ± 30%    +225.0%       0.13 ± 50%  turbostat.Pkg%pc2
      0.15 ±100%    +369.0%       0.68 ± 41%  turbostat.Pkg%pc3
     47.13            -7.9%      43.38        turbostat.PkgWatt
      0.11 ±100%    +225.5%       0.36 ± 51%  perf-stat.i.MPKI
  3.02e+09           -44.2%  1.686e+09        perf-stat.i.branch-instructions
      0.79 ±  2%      +0.0        0.82        perf-stat.i.branch-miss-rate%
  23476838           -42.3%   13556497        perf-stat.i.branch-misses
      2.03           +76.3%       3.58        perf-stat.i.cpi
  84569828 ±  5%     -39.2%   51431068 ± 10%  perf-stat.i.dTLB-load-misses
 3.797e+09           -42.2%  2.194e+09        perf-stat.i.dTLB-loads
      0.00 ± 25%      +1.1        1.13        perf-stat.i.dTLB-store-miss-rate%
     64340 ± 27%  +38891.4%   25087396        perf-stat.i.dTLB-store-misses
 2.814e+09           -22.4%  2.183e+09        perf-stat.i.dTLB-stores
  17567253 ±  2%     -39.8%   10574756 ±  4%  perf-stat.i.iTLB-load-misses
  13899345           -37.2%    8722357 ±  4%  perf-stat.i.iTLB-loads
 1.445e+10           -43.7%  8.143e+09        perf-stat.i.instructions
    843.79 ±  2%      -6.0%     793.14 ±  4%  perf-stat.i.instructions-per-iTLB-miss
      0.49           -43.0%       0.28        perf-stat.i.ipc
      0.05 ±  9%     +76.6%       0.08 ±  6%  perf-stat.overall.MPKI
      0.78            +0.0        0.80        perf-stat.overall.branch-miss-rate%
      2.03           +76.6%       3.58        perf-stat.overall.cpi
      0.00 ± 27%      +1.1        1.14        perf-stat.overall.dTLB-store-miss-rate%
    823.31 ±  2%      -6.3%     771.28 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      0.49           -43.4%       0.28        perf-stat.overall.ipc
 3.012e+09           -44.2%  1.681e+09        perf-stat.ps.branch-instructions
  23408490           -42.2%   13518789        perf-stat.ps.branch-misses
  84320779 ±  5%     -39.2%   51285561 ± 10%  perf-stat.ps.dTLB-load-misses
 3.785e+09           -42.2%  2.188e+09        perf-stat.ps.dTLB-loads
     64144 ± 27%  +38901.1%   25017212        perf-stat.ps.dTLB-store-misses
 2.806e+09           -22.4%  2.177e+09        perf-stat.ps.dTLB-stores
  17515924 ±  2%     -39.8%   10545294 ±  4%  perf-stat.ps.iTLB-load-misses
  13858656           -37.2%    8698260 ±  4%  perf-stat.ps.iTLB-loads
 1.441e+10           -43.7%   8.12e+09        perf-stat.ps.instructions
  4.37e+12           -43.4%  2.472e+12        perf-stat.total.instructions
     33.12            -9.5       23.61 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.poll
     24.11            -6.8       17.32 ±  6%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.poll
     31.52            -6.7       24.84 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.poll
     25.86            -5.8       20.01 ± 13%  perf-profile.calltrace.cycles-pp.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.poll
      6.17 ±  7%      -2.7        3.45 ±  4%  perf-profile.calltrace.cycles-pp.__fget_light.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.97 ±  7%      -2.2        2.78 ±  4%  perf-profile.calltrace.cycles-pp.__fget.__fget_light.do_sys_poll.__x64_sys_poll.do_syscall_64
      2.18 ±  3%      -1.0        1.16 ±  3%  perf-profile.calltrace.cycles-pp.fput.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.97 ± 12%      -0.3        0.71 ± 17%  perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled._copy_from_user.do_sys_poll.__x64_sys_poll.do_syscall_64
     98.77            +0.4       99.19        perf-profile.calltrace.cycles-pp.poll
      0.00            +2.5        2.46 ± 50%  perf-profile.calltrace.cycles-pp.__x86_indirect_thunk_rax.do_syscall_64.entry_SYSCALL_64_after_hwframe.poll
     37.43           +15.7       53.16 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.poll
     40.01           +16.0       56.00 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.poll
     29.78            -8.0       21.73 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
     27.30            -8.0       19.32 ±  6%  perf-profile.children.cycles-pp.syscall_return_via_sysret
     31.68            -6.7       25.02 ±  9%  perf-profile.children.cycles-pp.__x64_sys_poll
     26.48            -6.1       20.37 ± 13%  perf-profile.children.cycles-pp.do_sys_poll
      6.27 ±  6%      -2.8        3.52 ±  4%  perf-profile.children.cycles-pp.__fget_light
      4.97 ±  7%      -2.2        2.78 ±  4%  perf-profile.children.cycles-pp.__fget
      2.29 ±  3%      -1.1        1.23 ±  4%  perf-profile.children.cycles-pp.fput
      1.06 ± 13%      -0.3        0.77 ± 16%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.50 ±  5%      -0.2        0.27 ±  5%  perf-profile.children.cycles-pp.poll_freewait
      0.15 ± 23%      -0.1        0.09 ± 13%  perf-profile.children.cycles-pp.__fdget
      0.20 ±  8%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.45 ±  6%      +1.1        1.52 ± 41%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
     40.31           +15.9       56.24 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     37.53           +17.0       54.57 ±  3%  perf-profile.children.cycles-pp.do_syscall_64
     27.24            -8.0       19.28 ±  6%  perf-profile.self.cycles-pp.syscall_return_via_sysret
     26.41 ±  2%      -6.6       19.82 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64
     10.34            -3.9        6.45 ±  7%  perf-profile.self.cycles-pp.do_sys_poll
      4.87 ±  7%      -2.2        2.71 ±  4%  perf-profile.self.cycles-pp.__fget
      5.24 ±  5%      -1.5        3.75 ±  6%  perf-profile.self.cycles-pp.poll
      2.90            -1.1        1.77 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      2.29 ±  3%      -1.1        1.23 ±  4%  perf-profile.self.cycles-pp.fput
      1.33 ± 10%      -0.6        0.76 ±  6%  perf-profile.self.cycles-pp.__fget_light
      0.98 ± 13%      -0.3        0.71 ± 18%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.39 ±  4%      -0.2        0.21 ±  3%  perf-profile.self.cycles-pp.poll_freewait
      0.45 ±  6%      -0.2        0.29 ±  6%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.51 ±  3%      -0.1        0.42 ± 11%  perf-profile.self.cycles-pp.poll_select_set_timeout
      0.11 ± 27%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.__fdget
      5.53 ±  4%     +23.7       29.27 ±  3%  perf-profile.self.cycles-pp.do_syscall_64



***************************************************************************************************
lkp-ivb-d01: 8 threads Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz with 16G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-2019-05-14.cgz/lkp-ivb-d01/context_switch1/will-it-scale/0x21

commit: 
  07f07f55a2 ("x86/speculation/mds: Conditionally clear CPU buffers on idle entry")
  bc1241700a ("x86/speculation/mds: Add mitigation control for MDS")

07f07f55a29cb705 bc1241700acd82ec69fde98c576 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.RIP:poll_idle
           :4           25%           1:4     dmesg.RIP:raid#_sse24_xor_syndrome[raid#_pq]
           :4           25%           1:4     kmsg.a>]usb_hcd_irq
          1:4          -25%            :4     kmsg.ceba24>]usb_hcd_irq
          1:4          -25%            :4     kmsg.d2070>]usb_hcd_irq
           :4           25%           1:4     kmsg.e6ada9>]usb_hcd_irq
          1:4          -25%            :4     kmsg.ff007>]usb_hcd_irq
           :4           25%           1:4     kmsg.mei_me#:#:#:HBM_haven't_started
           :4           25%           1:4     kmsg.mei_me#:#:#:init_hw_failure
           :4           25%           1:4     kmsg.mei_me#:#:#:initialization_failed
           :4           25%           1:4     kmsg.mei_me#:#:#:link_layer_initialization_failed
           :4           25%           1:4     kmsg.mei_me#:#:#:waiting_for_mei_start_failed
          1:4          -25%            :4     kmsg.usb_hcd_irq
         %stddev     %change         %stddev
             \          |                \  
    316691           -14.8%     269667        will-it-scale.per_thread_ops
      2187           -12.9%       1905        will-it-scale.time.system_time
    210.82          +133.9%     493.00        will-it-scale.time.user_time
 7.521e+08           -14.6%  6.425e+08        will-it-scale.time.voluntary_context_switches
   2533537           -14.8%    2157339        will-it-scale.workload
     35953 ± 59%     -58.2%      15042 ± 47%  cpuidle.C1E.usage
      8.29 ±  7%     +12.1       20.39        mpstat.cpu.all.usr%
     24.37 ±  9%     +17.3%      28.58        boot-time.boot
    173.90 ± 10%     +19.3%     207.43        boot-time.idle
      7.50 ±  6%    +163.3%      19.75 ±  2%  vmstat.cpu.us
   2337694 ±  6%      -9.2%    2121597        vmstat.system.cs
     19667 ±  5%      -8.0%      18092 ±  2%  interrupts.CAL:Function_call_interrupts
      2803 ±  8%     -54.4%       1279 ± 88%  interrupts.CPU2.CAL:Function_call_interrupts
      2852 ±  5%     -10.0%       2566 ±  3%  interrupts.CPU7.CAL:Function_call_interrupts
     93.75 ± 11%     +29.6%     121.50 ± 16%  interrupts.TLB:TLB_shootdowns
    183.50 ± 13%      -8.9%     167.25        proc-vmstat.nr_inactive_file
      4184            +1.7%       4256        proc-vmstat.nr_kernel_stack
    183.50 ± 13%      -8.9%     167.25        proc-vmstat.nr_zone_inactive_file
    326483 ±  6%      -6.6%     304787        proc-vmstat.numa_hit
    326483 ±  6%      -6.6%     304787        proc-vmstat.numa_local
    366904 ±  5%      -6.9%     341455        proc-vmstat.pgalloc_normal
    374582 ±  6%      -7.4%     346888        proc-vmstat.pgfault
      0.75 ± 11%     -33.3%       0.50 ± 23%  sched_debug.cfs_rq:/.nr_running.min
    100712 ±  4%     +12.9%     113754 ± 10%  sched_debug.cfs_rq:/.runnable_weight.avg
    208.86 ±  6%     +11.2%     232.18 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      2.75           +17.9%       3.24 ±  5%  sched_debug.cpu.clock.stddev
      2.75           +17.9%       3.24 ±  5%  sched_debug.cpu.clock_task.stddev
  47169842           -14.7%   40219765        sched_debug.cpu.nr_switches.avg
  47785534           -15.2%   40538142        sched_debug.cpu.nr_switches.max
  46604933           -14.6%   39821656        sched_debug.cpu.nr_switches.min
      5.31 ± 11%     +27.3%       6.77 ± 18%  sched_debug.cpu.nr_uninterruptible.stddev
      4.63 ± 87%     -83.8%       0.75 ± 72%  perf-stat.i.MPKI
 3.741e+09 ±  6%      -9.4%   3.39e+09        perf-stat.i.branch-instructions
  61381829 ±  6%      -9.0%   55879868        perf-stat.i.branch-misses
   2356758 ±  6%      -9.2%    2139880        perf-stat.i.context-switches
      1.52 ±  5%     +10.3%       1.68        perf-stat.i.cpi
 5.652e+09 ±  6%      -9.0%  5.144e+09        perf-stat.i.dTLB-loads
      0.03 ± 27%      +0.3        0.29 ±  4%  perf-stat.i.dTLB-store-miss-rate%
   1112369 ± 24%    +808.0%   10100361 ±  3%  perf-stat.i.dTLB-store-misses
 1.921e+10 ±  6%      -9.2%  1.744e+10        perf-stat.i.instructions
      0.68 ±  3%     -12.1%       0.60        perf-stat.i.ipc
      1.42           +17.2%       1.67        perf-stat.overall.cpi
      0.03 ± 28%      +0.3        0.29 ±  4%  perf-stat.overall.dTLB-store-miss-rate%
      0.70           -14.7%       0.60        perf-stat.overall.ipc
 3.731e+09 ±  6%      -9.4%   3.38e+09        perf-stat.ps.branch-instructions
  61229994 ±  6%      -9.0%   55720998        perf-stat.ps.branch-misses
   2350920 ±  6%      -9.2%    2133847        perf-stat.ps.context-switches
 5.638e+09 ±  6%      -9.0%  5.129e+09        perf-stat.ps.dTLB-loads
   1109678 ± 24%    +807.1%   10065645 ±  3%  perf-stat.ps.dTLB-store-misses
 1.916e+10 ±  6%      -9.3%  1.739e+10        perf-stat.ps.instructions
      1095            -0.7%       1087        perf-stat.ps.minor-faults
      1095            -0.7%       1087        perf-stat.ps.page-faults
 6.223e+12           -14.8%  5.302e+12        perf-stat.total.instructions
     39.52            -5.0       34.56        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     35.23            -4.9       30.34        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     37.71            -4.9       32.83        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     36.70            -4.6       32.14        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     30.50            -4.5       26.01        perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.54            -4.3       25.29        perf-profile.calltrace.cycles-pp.pipe_write.__vfs_write.vfs_write.ksys_write.do_syscall_64
     31.64            -4.1       27.59        perf-profile.calltrace.cycles-pp.__vfs_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     30.85            -4.0       26.88        perf-profile.calltrace.cycles-pp.pipe_read.__vfs_read.vfs_read.ksys_read.do_syscall_64
     24.03            -3.4       20.63        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.__vfs_write.vfs_write.ksys_write
     23.09            -3.3       19.83        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.__vfs_write.vfs_write
     22.72            -3.2       19.48        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.__vfs_write
     22.14            -3.2       18.97        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
     23.07            -3.0       20.04        perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.__vfs_read.vfs_read.ksys_read
     20.29            -2.7       17.55        perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.__vfs_read.vfs_read
     19.74            -2.7       17.08        perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.__vfs_read
     10.77            -1.6        9.22        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     10.29            -1.5        8.80        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      9.45            -1.4        8.03        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read
      6.77 ±  3%      -0.9        5.90 ±  4%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      4.74            -0.7        4.08        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_wait.pipe_read
      5.04 ±  4%      -0.6        4.46 ±  6%  perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      3.74            -0.5        3.25        perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      3.35            -0.5        2.88 ±  4%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_wait
      3.20            -0.5        2.74 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_read
      2.56            -0.4        2.13        perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait
      3.57 ±  4%      -0.4        3.14 ±  2%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.29 ±  2%      -0.4        2.88 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_write
      1.90            -0.3        1.56 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      2.25            -0.3        1.93 ±  4%  perf-profile.calltrace.cycles-pp.__switch_to.__GI___libc_read
      1.34 ±  2%      -0.3        1.03 ±  6%  perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.56 ±  3%      -0.3        0.26 ±100%  perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_wait.pipe_read
      2.72            -0.3        2.42 ±  3%  perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      2.12 ±  4%      -0.3        1.83 ±  3%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.__vfs_write.vfs_write.ksys_write
      1.58            -0.3        1.31 ±  3%  perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      1.18 ±  7%      -0.3        0.93 ± 11%  perf-profile.calltrace.cycles-pp.touch_atime.pipe_read.__vfs_read.vfs_read.ksys_read
      3.16 ±  3%      -0.2        2.92 ±  2%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.89 ±  7%      -0.2        0.66 ± 16%  perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.pipe_read.__vfs_read.vfs_read
      0.61 ±  3%      -0.2        0.38 ± 57%  perf-profile.calltrace.cycles-pp.__inode_security_revalidate.selinux_file_permission.security_file_permission.vfs_read.ksys_read
      1.75 ±  3%      -0.2        1.53 ±  2%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait
      2.58            -0.2        2.38        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__GI___libc_read
      1.30 ±  3%      -0.2        1.10 ±  5%  perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      1.33 ±  2%      -0.2        1.14 ±  7%  perf-profile.calltrace.cycles-pp.native_write_msr.__GI___libc_read
      2.02 ±  2%      -0.2        1.83 ±  3%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.__vfs_read.vfs_read.ksys_read
      1.54 ±  7%      -0.2        1.35 ±  9%  perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
      1.19 ±  3%      -0.2        1.00 ±  2%  perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_wait
      1.12 ±  2%      -0.2        0.94 ±  3%  perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      1.57 ±  3%      -0.2        1.40 ±  2%  perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
      1.25 ±  6%      -0.2        1.07 ±  2%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.53 ±  2%      -0.2        1.36        perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
      1.10 ±  2%      -0.2        0.94 ±  5%  perf-profile.calltrace.cycles-pp.__switch_to_asm.__GI___libc_read
      1.06 ±  2%      -0.2        0.90 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
      0.75            -0.2        0.60        perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.__vfs_write.vfs_write.ksys_write
      0.88 ±  2%      -0.1        0.76        perf-profile.calltrace.cycles-pp.prepare_to_wait.pipe_wait.pipe_read.__vfs_read.vfs_read
      0.82 ±  4%      -0.1        0.71 ±  9%  perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.67 ±  3%      -0.1        0.56 ±  2%  perf-profile.calltrace.cycles-pp.__pthread_enable_asynccancel
      0.77 ±  2%      -0.1        0.65 ±  8%  perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait
      0.69 ±  7%      -0.1        0.58 ±  6%  perf-profile.calltrace.cycles-pp.__pthread_disable_asynccancel
      0.72            -0.1        0.61 ±  5%  perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.__vfs_read.vfs_read.ksys_read
      0.70            -0.1        0.59 ±  5%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.reweight_entity.dequeue_task_fair.__schedule
      0.68 ±  4%      -0.1        0.57 ±  4%  perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.16            -0.1        1.06 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.95 ±  5%      -0.1        0.85 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_read.__vfs_read.vfs_read.ksys_read
      0.90 ±  3%      -0.1        0.82 ±  3%  perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule
      0.74            -0.1        0.67 ±  7%  perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.__vfs_read.vfs_read
      2.59            +0.7        3.33 ±  6%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__GI___libc_write
     40.83            +1.0       41.86        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     41.47            +1.3       42.76        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     39.58            -4.9       34.63        perf-profile.children.cycles-pp.ksys_read
     37.82            -4.9       32.91        perf-profile.children.cycles-pp.vfs_read
     35.31            -4.9       30.42        perf-profile.children.cycles-pp.vfs_write
     36.75            -4.6       32.20        perf-profile.children.cycles-pp.ksys_write
     30.54            -4.5       26.03        perf-profile.children.cycles-pp.__vfs_write
     29.63            -4.3       25.35        perf-profile.children.cycles-pp.pipe_write
     31.67            -4.0       27.64        perf-profile.children.cycles-pp.__vfs_read
     30.93            -4.0       26.96        perf-profile.children.cycles-pp.pipe_read
     25.05            -3.5       21.56        perf-profile.children.cycles-pp.__wake_up_common_lock
     23.23            -3.3       19.94        perf-profile.children.cycles-pp.__wake_up_common
     22.75            -3.2       19.52        perf-profile.children.cycles-pp.autoremove_wake_function
     22.25            -3.2       19.07        perf-profile.children.cycles-pp.try_to_wake_up
     23.20            -3.1       20.15        perf-profile.children.cycles-pp.pipe_wait
     20.40            -2.8       17.62        perf-profile.children.cycles-pp.schedule
     20.06            -2.7       17.32        perf-profile.children.cycles-pp.__schedule
     10.80            -1.6        9.24        perf-profile.children.cycles-pp.ttwu_do_activate
     10.33            -1.5        8.83        perf-profile.children.cycles-pp.enqueue_task_fair
      9.50            -1.4        8.07        perf-profile.children.cycles-pp.dequeue_task_fair
      6.79 ±  3%      -0.9        5.92 ±  4%  perf-profile.children.cycles-pp.select_task_rq_fair
      6.50            -0.9        5.64        perf-profile.children.cycles-pp.entry_SYSCALL_64
      5.39            -0.7        4.65 ±  2%  perf-profile.children.cycles-pp.reweight_entity
      4.98            -0.7        4.27        perf-profile.children.cycles-pp.pick_next_task_fair
      6.78 ±  3%      -0.7        6.10 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      5.13 ±  4%      -0.6        4.54 ±  6%  perf-profile.children.cycles-pp.select_idle_sibling
      4.47            -0.5        3.93 ±  2%  perf-profile.children.cycles-pp.update_curr
      3.83            -0.5        3.31        perf-profile.children.cycles-pp.enqueue_entity
      3.44            -0.5        2.95 ±  4%  perf-profile.children.cycles-pp.dequeue_entity
      3.49 ±  3%      -0.4        3.05        perf-profile.children.cycles-pp.update_load_avg
      2.82 ±  3%      -0.4        2.40 ±  3%  perf-profile.children.cycles-pp.update_cfs_group
      2.34            -0.4        1.99 ±  3%  perf-profile.children.cycles-pp.mutex_lock
      1.84 ±  4%      -0.3        1.49 ±  8%  perf-profile.children.cycles-pp.fsnotify
      1.92            -0.3        1.57 ±  2%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      3.11            -0.3        2.77        perf-profile.children.cycles-pp.selinux_file_permission
      2.42            -0.3        2.08 ±  4%  perf-profile.children.cycles-pp.__switch_to
      2.10 ±  3%      -0.3        1.78 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_se
      2.17 ±  3%      -0.3        1.88 ±  4%  perf-profile.children.cycles-pp.copy_page_from_iter
      1.02 ± 12%      -0.3        0.73 ±  4%  perf-profile.children.cycles-pp.__fsnotify_parent
      1.60            -0.3        1.33 ±  2%  perf-profile.children.cycles-pp.check_preempt_curr
      2.07 ±  2%      -0.3        1.81 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      1.29 ±  2%      -0.3        1.03 ±  4%  perf-profile.children.cycles-pp.account_entity_dequeue
      1.20 ±  6%      -0.3        0.95 ± 11%  perf-profile.children.cycles-pp.touch_atime
      1.19 ±  4%      -0.2        0.95 ±  5%  perf-profile.children.cycles-pp.__inode_security_revalidate
      0.90 ±  7%      -0.2        0.67 ± 16%  perf-profile.children.cycles-pp.atime_needs_update
      1.48 ±  3%      -0.2        1.25 ±  4%  perf-profile.children.cycles-pp.___might_sleep
      1.45 ±  2%      -0.2        1.23 ±  7%  perf-profile.children.cycles-pp.native_write_msr
      1.11 ±  3%      -0.2        0.90 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      1.84 ±  2%      -0.2        1.62 ±  2%  perf-profile.children.cycles-pp.set_next_entity
      2.06 ±  2%      -0.2        1.86 ±  3%  perf-profile.children.cycles-pp.copy_page_to_iter
      1.67 ±  8%      -0.2        1.47 ±  8%  perf-profile.children.cycles-pp.cpumask_next_wrap
      1.51 ±  4%      -0.2        1.32 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.92 ±  3%      -0.2        0.73 ±  2%  perf-profile.children.cycles-pp.mutex_unlock
      1.37 ±  2%      -0.2        1.18 ±  4%  perf-profile.children.cycles-pp.update_rq_clock
      1.17 ±  2%      -0.2        0.98 ±  3%  perf-profile.children.cycles-pp.check_preempt_wakeup
      1.61 ±  2%      -0.2        1.42 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      1.66 ±  2%      -0.2        1.48        perf-profile.children.cycles-pp.sched_clock_cpu
      1.16 ±  7%      -0.2        0.99 ±  8%  perf-profile.children.cycles-pp.find_next_bit
      1.14 ±  2%      -0.2        0.97 ±  5%  perf-profile.children.cycles-pp.__switch_to_asm
      1.36            -0.2        1.20        perf-profile.children.cycles-pp.sched_clock
      1.30 ±  2%      -0.2        1.15        perf-profile.children.cycles-pp.native_sched_clock
      1.34 ±  3%      -0.2        1.18        perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.91            -0.1        0.76 ±  7%  perf-profile.children.cycles-pp.pick_next_entity
      0.91 ±  2%      -0.1        0.78        perf-profile.children.cycles-pp.prepare_to_wait
      0.66 ±  4%      -0.1        0.54 ± 12%  perf-profile.children.cycles-pp.update_min_vruntime
      0.67 ±  3%      -0.1        0.56 ±  2%  perf-profile.children.cycles-pp.__pthread_enable_asynccancel
      1.16 ±  4%      -0.1        1.04 ±  3%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.69 ±  7%      -0.1        0.58 ±  7%  perf-profile.children.cycles-pp.__pthread_disable_asynccancel
      0.54 ±  4%      -0.1        0.44 ±  6%  perf-profile.children.cycles-pp.cpumask_next
      0.50 ±  3%      -0.1        0.41 ±  8%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.60            -0.1        0.51 ±  3%  perf-profile.children.cycles-pp.rb_erase_cached
      0.50 ±  7%      -0.1        0.41 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock
      0.37 ±  6%      -0.1        0.28 ±  6%  perf-profile.children.cycles-pp.put_prev_entity
      0.43 ±  6%      -0.1        0.35 ±  7%  perf-profile.children.cycles-pp.deactivate_task
      0.54 ±  6%      -0.1        0.46 ±  3%  perf-profile.children.cycles-pp.finish_task_switch
      0.90 ±  2%      -0.1        0.83 ±  3%  perf-profile.children.cycles-pp._cond_resched
      0.46 ±  4%      -0.1        0.39 ±  9%  perf-profile.children.cycles-pp.clear_buddies
      0.61 ±  3%      -0.1        0.54 ±  8%  perf-profile.children.cycles-pp.__might_fault
      0.45 ±  6%      -0.1        0.38 ±  2%  perf-profile.children.cycles-pp.rw_verify_area
      0.28 ± 11%      -0.1        0.21 ±  5%  perf-profile.children.cycles-pp.inode_has_perm
      0.25 ± 12%      -0.1        0.19 ±  6%  perf-profile.children.cycles-pp.timespec64_trunc
      0.24 ±  2%      -0.1        0.18 ±  9%  perf-profile.children.cycles-pp.rb_next
      0.27 ±  5%      -0.0        0.22 ±  9%  perf-profile.children.cycles-pp.wakeup_preempt_entity
      0.29 ±  2%      -0.0        0.24 ± 13%  perf-profile.children.cycles-pp.__x64_sys_write
      0.25 ±  9%      -0.0        0.20 ±  7%  perf-profile.children.cycles-pp.activate_task
      0.17 ±  6%      -0.0        0.13 ± 19%  perf-profile.children.cycles-pp.__enqueue_entity
      0.19 ±  7%      -0.0        0.15 ± 14%  perf-profile.children.cycles-pp.set_next_buddy
      0.29 ±  3%      -0.0        0.24 ±  6%  perf-profile.children.cycles-pp.generic_pipe_buf_confirm
      0.14 ±  8%      -0.0        0.10 ± 21%  perf-profile.children.cycles-pp.testcase
      0.21 ± 10%      -0.0        0.18 ±  8%  perf-profile.children.cycles-pp.__sb_end_write
      0.08 ± 13%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.write@plt
      0.11 ±  7%      -0.0        0.09 ±  9%  perf-profile.children.cycles-pp.kill_fasync
      0.57 ±  7%      +0.2        0.79 ±  9%  perf-profile.children.cycles-pp.fput
      5.86            +0.4        6.28 ±  3%  perf-profile.children.cycles-pp.syscall_return_via_sysret
     78.79            +1.4       80.16        perf-profile.children.cycles-pp.do_syscall_64
     79.79            +1.6       81.35        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      5.74            -0.7        5.00        perf-profile.self.cycles-pp.entry_SYSCALL_64
      2.77 ±  3%      -0.4        2.35 ±  3%  perf-profile.self.cycles-pp.update_cfs_group
      2.13 ±  2%      -0.3        1.78 ±  2%  perf-profile.self.cycles-pp.reweight_entity
      2.36 ±  2%      -0.3        2.02 ±  3%  perf-profile.self.cycles-pp.__schedule
      1.79 ±  4%      -0.3        1.45 ±  9%  perf-profile.self.cycles-pp.fsnotify
      2.33            -0.3        2.01 ±  4%  perf-profile.self.cycles-pp.__switch_to
      1.55 ±  5%      -0.3        1.23 ±  3%  perf-profile.self.cycles-pp.enqueue_task_fair
      2.03 ±  2%      -0.3        1.73 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_se
      2.04 ±  2%      -0.3        1.78 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.94 ± 13%      -0.3        0.68 ±  3%  perf-profile.self.cycles-pp.__fsnotify_parent
      1.14 ±  2%      -0.3        0.88 ±  4%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.63 ± 14%      -0.3        0.38 ± 39%  perf-profile.self.cycles-pp.atime_needs_update
      1.62            -0.2        1.38 ±  7%  perf-profile.self.cycles-pp.select_task_rq_fair
      2.00 ±  5%      -0.2        1.77 ±  6%  perf-profile.self.cycles-pp.select_idle_sibling
      1.44 ±  3%      -0.2        1.22 ±  4%  perf-profile.self.cycles-pp.___might_sleep
      1.44 ±  3%      -0.2        1.23 ±  7%  perf-profile.self.cycles-pp.native_write_msr
      1.29 ±  4%      -0.2        1.08 ±  3%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.84 ±  7%      -0.2        0.64 ±  7%  perf-profile.self.cycles-pp.__vfs_write
      1.03 ±  4%      -0.2        0.82 ±  4%  perf-profile.self.cycles-pp.__might_sleep
      1.07 ±  2%      -0.2        0.87 ±  6%  perf-profile.self.cycles-pp.try_to_wake_up
      1.42            -0.2        1.23 ±  3%  perf-profile.self.cycles-pp.pick_next_task_fair
      1.45 ±  4%      -0.2        1.26 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.13 ±  2%      -0.2        0.94        perf-profile.self.cycles-pp.security_file_permission
      1.16 ±  5%      -0.2        0.97        perf-profile.self.cycles-pp.update_load_avg
      1.58 ±  2%      -0.2        1.41 ±  2%  perf-profile.self.cycles-pp.__calc_delta
      0.86 ±  2%      -0.2        0.69 ±  2%  perf-profile.self.cycles-pp.mutex_unlock
      1.14 ±  2%      -0.2        0.97 ±  5%  perf-profile.self.cycles-pp.__switch_to_asm
      0.99 ±  3%      -0.2        0.83 ±  7%  perf-profile.self.cycles-pp.mutex_lock
      1.26            -0.2        1.10        perf-profile.self.cycles-pp.native_sched_clock
      1.04 ±  7%      -0.1        0.90 ±  7%  perf-profile.self.cycles-pp.find_next_bit
      0.95            -0.1        0.80 ±  7%  perf-profile.self.cycles-pp.pipe_write
      1.20 ±  3%      -0.1        1.07 ±  3%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.75 ±  5%      -0.1        0.62 ±  3%  perf-profile.self.cycles-pp.check_preempt_wakeup
      1.02 ±  4%      -0.1        0.89 ±  5%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.95 ±  5%      -0.1        0.83 ±  4%  perf-profile.self.cycles-pp.copy_page_to_iter
      1.01            -0.1        0.89 ±  4%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.61 ±  3%      -0.1        0.50 ± 12%  perf-profile.self.cycles-pp.update_min_vruntime
      0.61 ±  4%      -0.1        0.51 ±  6%  perf-profile.self.cycles-pp.__GI___libc_read
      0.64 ±  6%      -0.1        0.54 ±  6%  perf-profile.self.cycles-pp.__pthread_disable_asynccancel
      0.64 ±  2%      -0.1        0.54 ±  3%  perf-profile.self.cycles-pp.__pthread_enable_asynccancel
      1.85 ±  2%      -0.1        1.76 ±  2%  perf-profile.self.cycles-pp.selinux_file_permission
      0.67 ±  3%      -0.1        0.58 ±  4%  perf-profile.self.cycles-pp.__GI___libc_write
      0.41 ±  7%      -0.1        0.33 ±  5%  perf-profile.self.cycles-pp.deactivate_task
      0.53 ±  5%      -0.1        0.45 ±  6%  perf-profile.self.cycles-pp.file_has_perm
      0.45 ±  3%      -0.1        0.38 ±  7%  perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.67 ±  6%      -0.1        0.59 ±  6%  perf-profile.self.cycles-pp.__vfs_read
      0.43 ±  5%      -0.1        0.35        perf-profile.self.cycles-pp.check_preempt_curr
      0.31 ±  6%      -0.1        0.23 ±  4%  perf-profile.self.cycles-pp.put_prev_entity
      0.32 ±  4%      -0.1        0.24 ±  8%  perf-profile.self.cycles-pp.ttwu_do_wakeup
      0.47 ±  6%      -0.1        0.39 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock
      0.82 ±  4%      -0.1        0.75 ±  5%  perf-profile.self.cycles-pp.account_entity_enqueue
      0.80 ±  4%      -0.1        0.73        perf-profile.self.cycles-pp.vfs_read
      0.43 ±  5%      -0.1        0.36 ±  3%  perf-profile.self.cycles-pp.rw_verify_area
      0.53 ±  4%      -0.1        0.46 ±  8%  perf-profile.self.cycles-pp.prepare_to_wait
      0.50 ±  6%      -0.1        0.44 ±  2%  perf-profile.self.cycles-pp.dequeue_entity
      0.38 ±  6%      -0.1        0.32 ± 10%  perf-profile.self.cycles-pp.clear_buddies
      0.48 ±  6%      -0.1        0.43 ±  7%  perf-profile.self.cycles-pp.__wake_up_common
      0.52 ±  5%      -0.1        0.47 ±  6%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.21 ± 13%      -0.1        0.16 ±  6%  perf-profile.self.cycles-pp.timespec64_trunc
      0.40            -0.0        0.35 ±  3%  perf-profile.self.cycles-pp.rb_erase_cached
      0.40 ±  6%      -0.0        0.35 ±  7%  perf-profile.self.cycles-pp.finish_task_switch
      0.22 ± 12%      -0.0        0.17 ±  8%  perf-profile.self.cycles-pp.inode_has_perm
      0.25 ±  7%      -0.0        0.21 ±  9%  perf-profile.self.cycles-pp.wakeup_preempt_entity
      0.15 ±  5%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.cpumask_next
      0.14 ±  8%      -0.0        0.09 ± 17%  perf-profile.self.cycles-pp.testcase
      0.18 ±  7%      -0.0        0.14 ± 12%  perf-profile.self.cycles-pp.set_next_buddy
      0.18 ±  5%      -0.0        0.14 ± 10%  perf-profile.self.cycles-pp.rb_next
      0.23 ± 10%      -0.0        0.19 ±  5%  perf-profile.self.cycles-pp.activate_task
      0.15 ±  8%      -0.0        0.11 ± 15%  perf-profile.self.cycles-pp.__enqueue_entity
      0.21 ±  8%      -0.0        0.17 ±  9%  perf-profile.self.cycles-pp.__sb_end_write
      0.25 ±  3%      -0.0        0.22 ±  4%  perf-profile.self.cycles-pp.generic_pipe_buf_confirm
      0.09            -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.kill_fasync
      0.07            -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.perf_swevent_event
      0.24 ±  3%      +0.2        0.40 ± 10%  perf-profile.self.cycles-pp.ksys_write
      1.02 ±  4%      +0.2        1.21 ±  6%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.54 ±  8%      +0.2        0.75 ± 10%  perf-profile.self.cycles-pp.fput
      5.84            +0.4        6.25 ±  3%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.68 ±  3%     +11.0       12.66 ±  5%  perf-profile.self.cycles-pp.do_syscall_64



***************************************************************************************************
lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/1/debian-x86_64-2019-05-14.cgz/300s/lkp-bdw-de1/pipe/unixbench/0x7000017

commit: 
  07f07f55a2 ("x86/speculation/mds: Conditionally clear CPU buffers on idle entry")
  bc1241700a ("x86/speculation/mds: Add mitigation control for MDS")

07f07f55a29cb705 bc1241700acd82ec69fde98c576 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
    671.80           -31.0%     463.82 ±  2%  unixbench.score
    240.02           -31.1%     165.41        unixbench.time.system_time
     60.36          +123.6%     134.98        unixbench.time.user_time
 3.874e+08 ±  9%     -40.8%  2.294e+08 ±  3%  unixbench.workload
      3.31 ± 11%      -0.6        2.69        mpstat.cpu.all.sys%
      0.85 ± 14%      +1.4        2.21        mpstat.cpu.all.usr%
    574562 ±  8%     -12.5%     502970        proc-vmstat.pgalloc_normal
    566972 ±  9%     -12.9%     493988        proc-vmstat.pgfree
      5356 ±  2%      +9.6%       5869 ±  4%  slabinfo.filp.num_objs
    997.50 ± 12%     +21.1%       1207 ±  4%  slabinfo.kmalloc-rcl-96.active_objs
    997.50 ± 12%     +21.1%       1207 ±  4%  slabinfo.kmalloc-rcl-96.num_objs
     30498 ±  9%     -22.0%      23789 ±  8%  turbostat.C1
      0.04 ± 17%      -0.0        0.02 ± 47%  turbostat.C1%
   6561845 ± 13%     -35.1%    4258239 ± 56%  turbostat.C6
   3148828 ± 16%     -63.2%    1159832 ± 27%  cpuidle.C1.time
     31545 ±  9%     -21.4%      24807 ±  7%  cpuidle.C1.usage
   6562406 ± 13%     -35.1%    4258751 ± 56%  cpuidle.C6.usage
    104709 ± 12%     -18.9%      84924        cpuidle.POLL.time
     49004 ± 11%     -23.0%      37739 ±  2%  cpuidle.POLL.usage
    822.50 ± 44%     -61.1%     320.25 ± 17%  interrupts.34:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
    184.00 ± 96%     -87.9%      22.25 ± 70%  interrupts.CPU10.RES:Rescheduling_interrupts
    822.50 ± 44%     -61.1%     320.25 ± 17%  interrupts.CPU12.34:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
      4430 ± 10%     -35.1%       2875 ± 52%  interrupts.CPU12.CAL:Function_call_interrupts
      4426 ± 10%     -15.5%       3739        interrupts.CPU13.CAL:Function_call_interrupts
      4419 ± 11%     -25.9%       3276 ± 16%  interrupts.CPU8.CAL:Function_call_interrupts
      2.15 ± 37%     +84.2%       3.96 ± 28%  sched_debug.cfs_rq:/.load_avg.min
      0.18 ± 15%     +26.0%       0.23 ±  7%  sched_debug.cfs_rq:/.nr_running.avg
    171.40 ±  7%     +15.1%     197.22 ±  4%  sched_debug.cfs_rq:/.util_avg.avg
     63.70 ± 12%     +68.5%     107.36 ± 28%  sched_debug.cfs_rq:/.util_est_enqueued.max
     18.20 ± 15%     +56.7%      28.51 ± 20%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    112166 ±  9%     +14.3%     128241 ±  5%  sched_debug.cpu.avg_idle.stddev
      0.19 ± 17%     +30.0%       0.25 ±  4%  sched_debug.cpu.nr_running.avg
     11793 ±  9%     -13.0%      10259        sched_debug.cpu.nr_switches.avg
     12.02 ± 19%     -35.8%       7.71 ± 17%  sched_debug.cpu.nr_uninterruptible.max
    -12.01           -38.1%      -7.43        sched_debug.cpu.nr_uninterruptible.min
      6.24 ± 14%     -31.4%       4.29 ±  5%  sched_debug.cpu.nr_uninterruptible.stddev
     68213 ± 11%     -18.8%      55416 ±  7%  softirqs.CPU0.SCHED
    194288 ±  4%     -14.7%     165789 ± 17%  softirqs.CPU0.TIMER
    175751 ±  6%     -23.6%     134262 ± 10%  softirqs.CPU1.TIMER
     37772 ± 19%     -32.1%      25647 ± 30%  softirqs.CPU10.RCU
    208221 ±  4%     -21.9%     162722 ± 20%  softirqs.CPU10.TIMER
     37931 ±  3%     -36.5%      24099 ± 31%  softirqs.CPU11.RCU
     63147 ±  8%     -16.6%      52638 ±  2%  softirqs.CPU11.SCHED
    212774 ±  8%     -17.4%     175655 ± 13%  softirqs.CPU11.TIMER
     62960 ±  7%     -13.9%      54183 ±  2%  softirqs.CPU12.SCHED
    184073 ±  7%     -15.7%     155114 ± 12%  softirqs.CPU12.TIMER
    186088 ±  3%     -15.8%     156766 ±  7%  softirqs.CPU13.TIMER
     38782 ± 13%     -29.7%      27258 ± 21%  softirqs.CPU14.RCU
    213647 ±  7%     -23.0%     164418 ± 16%  softirqs.CPU14.TIMER
     64956 ± 11%     -24.4%      49133 ±  4%  softirqs.CPU15.SCHED
    232006 ± 14%     -29.0%     164827 ± 13%  softirqs.CPU15.TIMER
     46026 ±  3%     -31.9%      31325 ± 16%  softirqs.CPU3.RCU
     62566 ±  9%     -17.4%      51688 ±  4%  softirqs.CPU3.SCHED
     65438 ± 11%     -20.7%      51914 ±  4%  softirqs.CPU4.SCHED
    192998 ±  4%     -15.6%     162982 ±  7%  softirqs.CPU5.TIMER
    214847 ±  7%     -20.9%     169861 ± 11%  softirqs.CPU6.TIMER
     43140 ± 12%     -35.5%      27816 ± 13%  softirqs.CPU7.RCU
    237031 ± 13%     -27.8%     171052 ±  8%  softirqs.CPU7.TIMER
     39325 ± 24%     -32.2%      26652 ± 18%  softirqs.CPU8.RCU
    177927 ±  6%     -23.7%     135800 ±  9%  softirqs.CPU9.TIMER
   1010072 ± 11%     -16.9%     839840 ±  2%  softirqs.SCHED
   3240647 ±  5%     -19.0%    2625179 ±  9%  softirqs.TIMER
 3.537e+08 ± 12%     -16.0%  2.972e+08        perf-stat.i.branch-instructions
      3.75 ± 15%      -0.9        2.82 ± 26%  perf-stat.i.branch-miss-rate%
    541.96            -1.1%     535.89        perf-stat.i.context-switches
 2.585e+09 ±  5%      +9.8%  2.838e+09 ±  3%  perf-stat.i.cpu-cycles
    689094 ± 11%    +121.1%    1523761 ±  6%  perf-stat.i.dTLB-load-misses
 5.874e+08 ± 12%     -15.5%  4.962e+08        perf-stat.i.dTLB-loads
      0.09 ± 26%      +0.2        0.25 ±  3%  perf-stat.i.dTLB-store-miss-rate%
    200618 ± 19%    +430.6%    1064385 ±  3%  perf-stat.i.dTLB-store-misses
     59.48 ±  7%     -11.5       47.93 ±  6%  perf-stat.i.iTLB-load-miss-rate%
   2459274 ± 15%     -20.1%    1965197        perf-stat.i.iTLB-load-misses
 1.859e+09 ± 12%     -15.9%  1.563e+09        perf-stat.i.instructions
      0.60 ± 12%     -16.4%       0.50 ±  4%  perf-stat.i.ipc
      1.40 ±  6%     +29.5%       1.82 ±  4%  perf-stat.overall.cpi
      0.12 ± 21%      +0.2        0.31 ±  7%  perf-stat.overall.dTLB-load-miss-rate%
      0.05 ± 24%      +0.2        0.26        perf-stat.overall.dTLB-store-miss-rate%
     52.23           -10.4       41.79 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      0.72 ±  7%     -23.0%       0.55 ±  4%  perf-stat.overall.ipc
      2280 ± 10%     +19.6%       2727        perf-stat.overall.path-length
 3.529e+08 ± 12%     -16.0%  2.966e+08        perf-stat.ps.branch-instructions
    540.81            -1.1%     534.78        perf-stat.ps.context-switches
 2.579e+09 ±  5%      +9.8%  2.832e+09 ±  3%  perf-stat.ps.cpu-cycles
    687655 ± 11%    +121.1%    1520522 ±  6%  perf-stat.ps.dTLB-load-misses
 5.861e+08 ± 12%     -15.5%  4.951e+08        perf-stat.ps.dTLB-loads
    200201 ± 19%    +430.5%    1062081 ±  3%  perf-stat.ps.dTLB-store-misses
   2453746 ± 15%     -20.1%    1960934        perf-stat.ps.iTLB-load-misses
 1.855e+09 ± 12%     -15.9%   1.56e+09        perf-stat.ps.instructions
 8.749e+11           -28.5%  6.255e+11 ±  2%  perf-stat.total.instructions
      0.48 ± 60%      +0.8        1.26 ± 49%  perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
      0.48 ± 60%      +0.8        1.26 ± 49%  perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
      0.49 ± 60%      +0.8        1.27 ± 50%  perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk
      0.49 ± 60%      +0.8        1.29 ± 50%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.printk.irq_work_run_list
      0.38 ±100%      +0.8        1.18 ± 33%  perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt.cpuidle_enter_state
      0.38 ±100%      +0.8        1.18 ± 33%  perf-profile.calltrace.cycles-pp.smp_irq_work_interrupt.irq_work_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      0.38 ±100%      +0.8        1.18 ± 33%  perf-profile.calltrace.cycles-pp.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt.cpuidle_enter_state.do_idle
      0.20 ±173%      +0.8        1.04 ± 38%  perf-profile.calltrace.cycles-pp.irq_work_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.52 ± 59%      +0.9        1.37 ± 50%  perf-profile.calltrace.cycles-pp.printk.irq_work_run_list.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt
      0.52 ± 59%      +0.9        1.37 ± 50%  perf-profile.calltrace.cycles-pp.vprintk_emit.printk.irq_work_run_list.irq_work_run.smp_irq_work_interrupt
      0.52 ± 59%      +0.9        1.37 ± 50%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.irq_work_run_list.irq_work_run
      0.93 ± 20%      -0.4        0.56 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.52 ± 11%      -0.2        0.35 ± 21%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.21 ± 25%      -0.1        0.12 ± 18%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.20 ± 25%      -0.1        0.12 ± 27%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.16 ± 19%      -0.1        0.10 ± 24%  perf-profile.children.cycles-pp.native_apic_mem_write
      0.17 ± 23%      -0.1        0.11 ± 17%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.11 ± 17%      -0.0        0.06 ± 58%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.11 ± 14%      -0.0        0.07 ± 17%  perf-profile.children.cycles-pp.account_process_tick
      0.73 ±  7%      -0.2        0.55 ± 10%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.16 ± 17%      -0.1        0.09 ± 24%  perf-profile.self.cycles-pp.native_apic_mem_write
      0.11 ± 26%      -0.1        0.04 ±107%  perf-profile.self.cycles-pp.tick_irq_enter
      0.22 ± 16%      -0.1        0.16 ± 13%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.12 ± 24%      -0.1        0.06 ± 63%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.17 ± 23%      -0.1        0.11 ± 17%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.13 ± 17%      -0.1        0.08 ± 58%  perf-profile.self.cycles-pp.clockevents_program_event
      0.11 ± 14%      -0.0        0.07 ± 17%  perf-profile.self.cycles-pp.account_process_tick
      0.46 ± 47%      +9.3        9.71 ± 14%  perf-profile.self.cycles-pp.do_syscall_64



***************************************************************************************************
lkp-skl-4sp1: 192 threads Skylake-SP with 256G memory


***************************************************************************************************
lkp-hsx04: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/1T/lkp-hsx04/lru-shm/vm-scalability/0x14

commit: 
  07f07f55a2 ("x86/speculation/mds: Conditionally clear CPU buffers on idle entry")
  bc1241700a ("x86/speculation/mds: Add mitigation control for MDS")

07f07f55a29cb705 bc1241700acd82ec69fde98c576 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    668536            -5.8%     629729        vm-scalability.median
  94403858            -5.2%   89480312        vm-scalability.throughput
      1554            +3.5%       1609        vm-scalability.time.percent_of_cpu_this_job_got
      1675 ±  3%      +8.7%       1822        vm-scalability.time.user_time
      3.68            +0.4        4.11        mpstat.cpu.all.usr%
    162606            -5.5%     153720        pmeter.performance_per_watt
    364.75            +2.9%     375.50        turbostat.Avg_MHz
    229000           -16.2%     191881        cpuidle.POLL.time
    123611 ±  3%     -13.9%     106474 ±  2%  cpuidle.POLL.usage
     18988 ± 52%     +59.2%      30232 ± 19%  numa-numastat.node0.other_node
 1.678e+08 ±  4%      -6.3%  1.573e+08        numa-numastat.node2.local_node
 1.678e+08 ±  4%      -6.3%  1.573e+08        numa-numastat.node2.numa_hit
   2894287 ±  4%      +6.7%    3088310 ±  2%  proc-vmstat.nr_mapped
      6729 ±  3%      +5.2%       7078 ±  2%  proc-vmstat.nr_page_table_pages
      3724 ± 24%     +43.6%       5347 ± 11%  proc-vmstat.numa_hint_faults
 5.285e+10            +3.2%  5.455e+10        perf-stat.i.cpu-cycles
   4943134 ± 14%     +32.5%    6549409 ±  2%  perf-stat.i.dTLB-store-misses
      0.11 ± 16%      +0.0        0.15 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
 5.361e+10            +3.4%  5.545e+10        perf-stat.ps.cpu-cycles
   5008194 ± 14%     +32.9%    6654563 ±  2%  perf-stat.ps.dTLB-store-misses
      0.20 ± 10%     -21.4%       0.15 ±  7%  sched_debug.cfs_rq:/.nr_running.stddev
   -265274           -24.8%    -199382        sched_debug.cfs_rq:/.spread0.min
      5.04 ±  9%     +53.8%       7.76 ± 22%  sched_debug.cpu.clock.stddev
      5.05 ±  9%     +53.8%       7.76 ± 22%  sched_debug.cpu.clock_task.stddev
      5.46 ±  4%     -11.3%       4.84 ±  7%  sched_debug.cpu.cpu_load[0].avg
     37.75 ± 10%     -17.9%      30.99 ± 16%  sched_debug.cpu.cpu_load[1].stddev
      1510 ±  9%     -12.1%       1328        sched_debug.cpu.curr->pid.stddev
      3.92 ± 34%      -1.1        2.82 ±  8%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      4.66 ± 28%      -1.1        3.59 ±  6%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      1.25 ± 33%      -0.4        0.89 ±  8%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      1.30 ± 34%      -0.4        0.94 ±  6%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      3.98 ± 33%      -1.1        2.92 ±  6%  perf-profile.children.cycles-pp.__softirqentry_text_start
      4.73 ± 28%      -1.1        3.68 ±  6%  perf-profile.children.cycles-pp.irq_exit
      0.09 ± 24%      -0.0        0.06 ± 15%  perf-profile.children.cycles-pp.update_dl_rq_load_avg
      0.20 ±  5%      +0.1        0.28 ± 31%  perf-profile.children.cycles-pp.io_serial_in
      0.20 ±  5%      +0.1        0.28 ± 31%  perf-profile.self.cycles-pp.io_serial_in
  25081048 ±  7%      +8.3%   27160974 ±  4%  numa-vmstat.node0.nr_free_pages
     30794 ±  3%     -82.2%       5478 ±115%  numa-vmstat.node1.nr_active_anon
     29930           -82.8%       5155 ±124%  numa-vmstat.node1.nr_anon_pages
     11533 ± 10%     -16.6%       9616 ±  5%  numa-vmstat.node1.nr_slab_unreclaimable
     30794 ±  3%     -82.2%       5478 ±115%  numa-vmstat.node1.nr_zone_active_anon
   6507628 ± 18%     +31.6%    8561965 ±  8%  numa-vmstat.node2.nr_file_pages
   6444293 ± 18%     +31.9%    8498408 ±  8%  numa-vmstat.node2.nr_inactive_anon
   6444938 ± 18%     +31.9%    8499176 ±  8%  numa-vmstat.node2.nr_shmem
     18966 ±  9%     +27.7%      24226 ±  9%  numa-vmstat.node2.nr_slab_reclaimable
     11424           -11.4%      10124 ±  7%  numa-vmstat.node2.nr_slab_unreclaimable
   6444263 ± 18%     +31.9%    8498380 ±  8%  numa-vmstat.node2.nr_zone_inactive_anon
  90921041 ±  4%      -5.7%   85740579        numa-vmstat.node2.numa_hit
  90809005 ±  4%      -5.7%   85629666        numa-vmstat.node2.numa_local
      1440 ± 67%   +1336.1%      20687 ± 54%  numa-vmstat.node3.nr_active_anon
    853.00 ± 91%   +2233.6%      19905 ± 55%  numa-vmstat.node3.nr_anon_pages
      1527 ±  2%     +20.7%       1842 ± 11%  numa-vmstat.node3.nr_page_table_pages
      1440 ± 67%   +1336.1%      20687 ± 54%  numa-vmstat.node3.nr_zone_active_anon
    114168 ±  5%     -10.1%     102643 ±  9%  numa-vmstat.node3.numa_other
 1.004e+08 ±  7%      +8.2%  1.086e+08 ±  4%  numa-meminfo.node0.MemFree
    123251 ±  3%     -82.2%      21911 ±115%  numa-meminfo.node1.Active
    123206 ±  3%     -82.2%      21911 ±115%  numa-meminfo.node1.Active(anon)
     96043 ±  8%     -88.9%      10653 ±170%  numa-meminfo.node1.AnonHugePages
    119745           -82.8%      20621 ±124%  numa-meminfo.node1.AnonPages
     46136 ± 10%     -16.6%      38467 ±  5%  numa-meminfo.node1.SUnreclaim
     34945 ±101%    +174.0%      95748 ± 28%  numa-meminfo.node2.AnonHugePages
  26024822 ± 18%     +31.7%   34262235 ±  8%  numa-meminfo.node2.FilePages
  25771603 ± 18%     +32.0%   34008536 ±  8%  numa-meminfo.node2.Inactive
  25771475 ± 18%     +32.0%   34008004 ±  8%  numa-meminfo.node2.Inactive(anon)
     75818 ±  9%     +27.9%      96946 ±  9%  numa-meminfo.node2.KReclaimable
  26360960 ± 17%     +31.6%   34683443 ±  8%  numa-meminfo.node2.MemUsed
     75818 ±  9%     +27.9%      96946 ±  9%  numa-meminfo.node2.SReclaimable
     45699           -11.4%      40495 ±  7%  numa-meminfo.node2.SUnreclaim
  25774049 ± 18%     +32.0%   34011071 ±  8%  numa-meminfo.node2.Shmem
    121518 ±  5%     +13.1%     137441 ±  8%  numa-meminfo.node2.Slab
      5779 ± 66%   +1332.6%      82801 ± 54%  numa-meminfo.node3.Active
      5764 ± 67%   +1335.3%      82735 ± 54%  numa-meminfo.node3.Active(anon)
      3416 ± 90%   +2230.3%      79614 ± 55%  numa-meminfo.node3.AnonPages
      6070 ±  2%     +22.2%       7415 ± 12%  numa-meminfo.node3.PageTables
     38628 ±  8%     -16.9%      32118 ± 11%  softirqs.CPU101.RCU
     40448 ± 19%     -30.0%      28334 ±  9%  softirqs.CPU112.RCU
     37924 ± 23%     -41.8%      22084 ± 15%  softirqs.CPU116.RCU
     34906 ± 10%     -18.1%      28594 ± 11%  softirqs.CPU118.RCU
     17335 ± 16%     -23.7%      13220 ± 22%  softirqs.CPU126.RCU
     34741 ± 17%     -31.2%      23907 ±  4%  softirqs.CPU131.RCU
     31911 ± 19%     -28.0%      22985 ± 12%  softirqs.CPU133.RCU
     31643 ± 13%     -21.1%      24978 ±  9%  softirqs.CPU135.RCU
     37901 ±  4%     -17.0%      31463 ± 10%  softirqs.CPU137.RCU
     38834 ±  8%     -18.3%      31720 ± 13%  softirqs.CPU139.RCU
     36396 ±  3%     -19.9%      29156 ± 14%  softirqs.CPU14.RCU
     39194 ± 13%     -21.4%      30812 ± 13%  softirqs.CPU141.RCU
     37713 ±  7%     -15.8%      31756 ± 12%  softirqs.CPU142.RCU
     21651 ± 37%     -40.1%      12974 ± 28%  softirqs.CPU17.RCU
     36904 ±  4%      -8.7%      33703 ±  6%  softirqs.CPU28.SCHED
     45023 ± 33%     -33.8%      29824 ± 12%  softirqs.CPU3.RCU
     42527 ±  7%     -32.0%      28929 ± 12%  softirqs.CPU41.RCU
     38765 ± 19%     -30.5%      26943 ± 13%  softirqs.CPU45.RCU
     39895 ±  7%     -25.5%      29722 ± 12%  softirqs.CPU5.RCU
     39228 ± 47%     -43.8%      22050 ± 13%  softirqs.CPU53.RCU
     31136 ± 16%     -20.5%      24768 ± 16%  softirqs.CPU57.RCU
     31425 ± 17%     -32.3%      21280 ± 14%  softirqs.CPU59.RCU
     37813 ±  5%      -7.5%      34983 ±  2%  softirqs.CPU6.SCHED
     33121 ± 18%     -35.8%      21249 ± 13%  softirqs.CPU60.RCU
     24397 ± 18%     -26.1%      18039 ± 12%  softirqs.CPU68.RCU
     39954 ± 12%     -27.1%      29109 ± 12%  softirqs.CPU7.RCU
     31958 ± 12%     -21.0%      25233 ±  9%  softirqs.CPU72.RCU
     35085 ± 29%     -36.2%      22395 ± 33%  softirqs.CPU76.RCU
     28998 ± 15%     -24.3%      21961 ± 17%  softirqs.CPU77.RCU
     33062 ± 15%     -29.5%      23324 ± 13%  softirqs.CPU79.RCU
     28922 ± 28%     -30.3%      20168 ± 23%  softirqs.CPU81.RCU
     40998 ± 15%     -30.3%      28561 ± 11%  softirqs.CPU83.RCU
     34733 ± 10%     -17.2%      28746 ± 12%  softirqs.CPU85.RCU
     42182 ± 31%     -31.6%      28857 ± 17%  softirqs.CPU87.RCU
     38705 ±  8%     -19.3%      31249 ± 14%  softirqs.CPU88.RCU
     73718 ±  9%     -65.8%      25223 ± 27%  softirqs.CPU89.RCU
    117739 ±  4%      +9.1%     128473 ±  5%  softirqs.CPU97.TIMER
      3036 ±  5%     +10.8%       3365 ±  8%  interrupts.CPU107.CAL:Function_call_interrupts
    188.50 ±105%    +162.3%     494.50 ± 12%  interrupts.CPU110.NMI:Non-maskable_interrupts
    188.50 ±105%    +162.3%     494.50 ± 12%  interrupts.CPU110.PMI:Performance_monitoring_interrupts
    231.50 ± 54%     -74.4%      59.25 ±103%  interrupts.CPU111.RES:Rescheduling_interrupts
    626.50 ± 17%     -65.4%     216.50 ± 87%  interrupts.CPU112.NMI:Non-maskable_interrupts
    626.50 ± 17%     -65.4%     216.50 ± 87%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
    602.25 ± 17%     -45.7%     327.25 ± 59%  interrupts.CPU116.NMI:Non-maskable_interrupts
    602.25 ± 17%     -45.7%     327.25 ± 59%  interrupts.CPU116.PMI:Performance_monitoring_interrupts
    638.75 ± 16%     -62.2%     241.75 ± 59%  interrupts.CPU117.NMI:Non-maskable_interrupts
    638.75 ± 16%     -62.2%     241.75 ± 59%  interrupts.CPU117.PMI:Performance_monitoring_interrupts
    501.00 ± 46%     -62.4%     188.50 ±113%  interrupts.CPU119.NMI:Non-maskable_interrupts
    501.00 ± 46%     -62.4%     188.50 ±113%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
    207.50 ± 82%     -69.4%      63.50 ±116%  interrupts.CPU119.RES:Rescheduling_interrupts
    139.50 ± 56%    +184.4%     396.75 ± 33%  interrupts.CPU121.NMI:Non-maskable_interrupts
    139.50 ± 56%    +184.4%     396.75 ± 33%  interrupts.CPU121.PMI:Performance_monitoring_interrupts
    642021 ±  2%      -9.6%     580397 ±  4%  interrupts.CPU125.LOC:Local_timer_interrupts
     24.25 ± 59%   +1494.8%     386.75 ± 51%  interrupts.CPU125.NMI:Non-maskable_interrupts
     24.25 ± 59%   +1494.8%     386.75 ± 51%  interrupts.CPU125.PMI:Performance_monitoring_interrupts
    635.75 ± 10%     -65.7%     217.75 ±117%  interrupts.CPU127.NMI:Non-maskable_interrupts
    635.75 ± 10%     -65.7%     217.75 ±117%  interrupts.CPU127.PMI:Performance_monitoring_interrupts
     64.25 ± 56%     -61.9%      24.50 ± 44%  interrupts.CPU142.RES:Rescheduling_interrupts
    237.00 ± 48%     -71.8%      66.75 ± 80%  interrupts.CPU19.RES:Rescheduling_interrupts
    326.25 ± 64%     -81.2%      61.25 ±173%  interrupts.CPU23.NMI:Non-maskable_interrupts
    326.25 ± 64%     -81.2%      61.25 ±173%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    188.50 ± 38%     -59.9%      75.50 ±104%  interrupts.CPU28.RES:Rescheduling_interrupts
    202.00 ± 58%    +143.9%     492.75 ± 25%  interrupts.CPU40.NMI:Non-maskable_interrupts
    202.00 ± 58%    +143.9%     492.75 ± 25%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    430.75 ± 68%     -86.4%      58.50 ±173%  interrupts.CPU49.NMI:Non-maskable_interrupts
    430.75 ± 68%     -86.4%      58.50 ±173%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
    624.50 ± 14%     -51.2%     304.50 ± 80%  interrupts.CPU53.NMI:Non-maskable_interrupts
    624.50 ± 14%     -51.2%     304.50 ± 80%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
     70.50 ± 93%    +137.9%     167.75 ± 65%  interrupts.CPU7.RES:Rescheduling_interrupts
      1465 ± 94%     -85.8%     208.00 ±114%  interrupts.CPU77.NMI:Non-maskable_interrupts
      1465 ± 94%     -85.8%     208.00 ±114%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
    507.50 ± 28%     -89.2%      55.00 ±173%  interrupts.CPU78.NMI:Non-maskable_interrupts
    507.50 ± 28%     -89.2%      55.00 ±173%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
    672.50 ± 19%     -60.4%     266.50 ± 93%  interrupts.CPU79.NMI:Non-maskable_interrupts
    672.50 ± 19%     -60.4%     266.50 ± 93%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    227.00 ± 67%    +190.3%     659.00 ± 31%  interrupts.CPU84.NMI:Non-maskable_interrupts
    227.00 ± 67%    +190.3%     659.00 ± 31%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
    289.25 ± 86%     -84.3%      45.50 ± 62%  interrupts.CPU90.RES:Rescheduling_interrupts
    223.50 ± 71%     -67.7%      72.25 ± 63%  interrupts.CPU91.RES:Rescheduling_interrupts
    417.00 ± 80%     -76.1%      99.75 ± 44%  interrupts.CPU92.RES:Rescheduling_interrupts
     39.50 ± 31%    +190.5%     114.75 ± 40%  interrupts.CPU97.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-hsx04: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/testtime/ucode:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/lkp-hsx04/all/aim9/5s/0x14

commit: 
  07f07f55a2 ("x86/speculation/mds: Conditionally clear CPU buffers on idle entry")
  bc1241700a ("x86/speculation/mds: Add mitigation control for MDS")

07f07f55a29cb705 bc1241700acd82ec69fde98c576 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
   1978200            +2.0%    2017473        aim9.add_double.ops_per_sec
   1979010            +2.1%    2019600        aim9.add_float.ops_per_sec
   8652341            +2.1%    8832000        aim9.add_int.ops_per_sec
   8652371            +2.0%    8824802        aim9.add_long.ops_per_sec
   5281200            +1.9%    5383832        aim9.add_short.ops_per_sec
      3018            +2.0%       3079        aim9.array_rtns.ops_per_sec
   1586174           -25.3%    1184231        aim9.brk_test.ops_per_sec
    510050           -16.6%     425400        aim9.creat-clo.ops_per_sec
    631415           -20.6%     501365        aim9.dgram_pipe.ops_per_sec
  11710000            -4.1%   11233500        aim9.dir_rtns_1.ops_per_sec
    684629           -22.3%     532183        aim9.disk_cp.ops_per_sec
   1808770           -27.7%    1307247        aim9.disk_rd.ops_per_sec
    545997           -25.8%     405152        aim9.disk_rr.ops_per_sec
    518907           -24.7%     390836        aim9.disk_rw.ops_per_sec
    214248            -9.2%     194553        aim9.disk_src.ops_per_sec
   1114880           -18.7%     906496        aim9.disk_wrt.ops_per_sec
    306591            +2.0%     312589        aim9.div_double.ops_per_sec
    312443            +1.9%     318430        aim9.div_float.ops_per_sec
    259560            +1.9%     264426        aim9.div_int.ops_per_sec
    112646            +2.1%     114962        aim9.div_long.ops_per_sec
    254979            +2.0%     260190        aim9.div_short.ops_per_sec
      2149            -5.2%       2037        aim9.exec_test.ops_per_sec
    859080           -26.9%     627695        aim9.fifo_test.ops_per_sec
      6476            -4.8%       6168 ±  2%  aim9.fork_test.ops_per_sec
 4.215e+08            +2.0%  4.301e+08        aim9.fun_cal.ops_per_sec
 4.244e+08            +1.8%  4.322e+08        aim9.fun_cal1.ops_per_sec
 2.447e+08            +2.0%  2.495e+08        aim9.fun_cal15.ops_per_sec
 4.945e+08            +1.9%  5.041e+08        aim9.fun_cal2.ops_per_sec
  45390900            +2.3%   46450750        aim9.jmp_test.ops_per_sec
    338439           -13.5%     292691        aim9.link_test.ops_per_sec
  13017615            +1.9%   13268080        aim9.matrix_rtns.ops_per_sec
     40962 ±  2%     -24.1%      31076        aim9.misc_rtns_1.ops_per_sec
   1188004            +1.9%    1210778        aim9.mul_double.ops_per_sec
   1188020            +2.1%    1213200        aim9.mul_float.ops_per_sec
   1375752            +1.9%    1402476        aim9.mul_int.ops_per_sec
   1375848            +1.9%    1401960        aim9.mul_long.ops_per_sec
   1369350            +2.0%    1396125        aim9.mul_short.ops_per_sec
    448545           -14.8%     382054        aim9.page_test.ops_per_sec
    880460           -27.0%     642798        aim9.pipe_cpy.ops_per_sec
  53361225            +1.8%   54346140        aim9.series_1.ops_per_sec
    514835           -30.8%     356340        aim9.shared_memory.ops_per_sec
    396.60            -6.1%     372.31        aim9.shell_rtns_1.ops_per_sec
    397.60            -6.1%     373.25        aim9.shell_rtns_2.ops_per_sec
    397.70            -6.2%     373.20        aim9.shell_rtns_3.ops_per_sec
    597142 ±  3%     -26.5%     438829        aim9.signal_test.ops_per_sec
      5578            +1.9%       5684        aim9.sort_rtns_1.ops_per_sec
    655180           -21.3%     515600        aim9.stream_pipe.ops_per_sec
     12597            +1.7%      12817        aim9.string_rtns.ops_per_sec
    641466           -19.5%     516090        aim9.sync_disk_cp.ops_per_sec
    515578           -24.5%     389055        aim9.sync_disk_rw.ops_per_sec
   1095424           -18.4%     893440        aim9.sync_disk_wrt.ops_per_sec
    204124            -7.3%     189301        aim9.tcp_test.ops_per_sec
      8164 ±  3%     -17.6%       6723 ±  9%  aim9.time.involuntary_context_switches
      8281            +1.7%       8418        aim9.time.maximum_resident_set_size
   6480568            -9.9%    5838307        aim9.time.minor_page_faults
    108.15           -19.2%      87.41        aim9.time.system_time
    192.20           +10.9%     213.12        aim9.time.user_time
    133840            -5.4%     126673        aim9.time.voluntary_context_switches
   2656000            +1.8%    2703816        aim9.trig_rtns.ops_per_sec
    381260           -13.6%     329435        aim9.udp_test.ops_per_sec
  31242345 ± 24%     +52.8%   47726189 ± 17%  meminfo.Committed_AS
      6519 ± 10%     -11.1%       5794 ±  4%  numa-meminfo.node0.KernelStack
      4393 ± 26%     -52.3%       2094 ± 31%  sched_debug.cpu.max_idle_balance_cost.stddev
      2081 ±  5%      +9.2%       2274 ±  3%  slabinfo.pool_workqueue.num_objs
      2613            -2.6%       2545        vmstat.system.cs
      0.29            -0.0        0.25        mpstat.cpu.all.sys%
      0.44            +0.1        0.49        mpstat.cpu.all.usr%
   6532726 ± 96%     -58.6%    2704978 ±173%  numa-numastat.node1.local_node
   6559876 ± 96%     -58.2%    2741170 ±170%  numa-numastat.node1.numa_hit
     35.57 ±  8%     -29.0%      25.25 ± 19%  turbostat.CPU%c1
     44.29           +19.9%      53.12 ±  7%  turbostat.Pkg%pc2
      6518 ± 10%     -11.1%       5793 ±  4%  numa-vmstat.node0.nr_kernel_stack
   4679133 ± 92%     -55.7%    2072775 ±153%  numa-vmstat.node1.numa_hit
   4570919 ± 94%     -57.2%    1955505 ±162%  numa-vmstat.node1.numa_local
  13464787           -14.6%   11492411        proc-vmstat.numa_hit
  13356279           -14.8%   11383799        proc-vmstat.numa_local
    424155           -21.0%     334931        proc-vmstat.pgactivate
  13838204           -14.7%   11803945        proc-vmstat.pgalloc_normal
   7370472            -8.7%    6729250        proc-vmstat.pgfault
  13767966           -14.8%   11733277        proc-vmstat.pgfree
      2580            -2.8%       2507        perf-stat.i.context-switches
      0.54 ±  8%     -28.2%       0.39 ± 16%  perf-stat.i.cpu-migrations
     24196            -8.9%      22038        perf-stat.i.minor-faults
     24196            -8.9%      22039        perf-stat.i.page-faults
      2571            -2.8%       2498        perf-stat.ps.context-switches
      0.54 ±  8%     -28.3%       0.39 ± 16%  perf-stat.ps.cpu-migrations
     24114            -8.9%      21963        perf-stat.ps.minor-faults
     24115            -8.9%      21963        perf-stat.ps.page-faults
      2.18 ± 17%      +0.5        2.68 ±  7%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      2.71 ± 12%      +2.2        4.88 ± 25%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      3.64 ±  7%      +2.3        5.92 ± 19%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.20 ± 14%      -0.1        0.13 ± 34%  perf-profile.children.cycles-pp.pm_qos_request
      0.14 ± 23%      +0.0        0.19 ±  9%  perf-profile.children.cycles-pp.leave_mm
      0.01 ±173%      +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.update_group_capacity
      0.08 ± 17%      +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.find_next_and_bit
      0.09 ± 19%      +0.1        0.16 ± 23%  perf-profile.children.cycles-pp.cpumask_next_and
      0.36            +0.1        0.46 ± 21%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.08 ± 35%      +0.1        0.19 ± 51%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.17 ± 31%      +0.2        0.36 ± 25%  perf-profile.children.cycles-pp.note_gp_changes
      0.12 ± 28%      +0.3        0.38 ± 53%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      2.24 ± 17%      +0.5        2.73 ±  7%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.35 ± 11%      +0.8        2.14 ± 27%  perf-profile.children.cycles-pp.ktime_get
      2.98 ±  7%      +2.1        5.09 ± 24%  perf-profile.children.cycles-pp.__softirqentry_text_start
     15.14 ± 12%      +2.2       17.32 ±  7%  perf-profile.children.cycles-pp.apic_timer_interrupt
      3.91 ±  3%      +2.2        6.15 ± 18%  perf-profile.children.cycles-pp.irq_exit
     13.96 ± 13%      +2.4       16.40 ±  6%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.72 ±  8%      -0.3        1.41 ± 12%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.26 ±  3%      -0.1        0.17 ± 27%  perf-profile.self.cycles-pp.native_apic_mem_write
      0.20 ± 14%      -0.1        0.13 ± 35%  perf-profile.self.cycles-pp.pm_qos_request
      0.18 ± 10%      -0.1        0.12 ± 29%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.01 ±173%      +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.update_group_capacity
      0.07 ± 14%      +0.1        0.14 ± 21%  perf-profile.self.cycles-pp.find_next_and_bit
      0.08 ± 37%      +0.1        0.19 ± 50%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.08 ± 40%      +0.1        0.20 ± 30%  perf-profile.self.cycles-pp.note_gp_changes
      0.33 ± 11%      +0.2        0.54 ± 43%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.10 ± 46%      +0.3        0.37 ± 56%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.37 ± 15%      +0.3        0.70 ± 26%  perf-profile.self.cycles-pp.update_blocked_averages
    538.00 ± 16%     -78.8%     114.25 ± 78%  interrupts.CPU108.NMI:Non-maskable_interrupts
    538.00 ± 16%     -78.8%     114.25 ± 78%  interrupts.CPU108.PMI:Performance_monitoring_interrupts
      0.50 ±173%  +23650.0%     118.75 ±156%  interrupts.CPU109.RES:Rescheduling_interrupts
    156.75 ± 72%    +118.7%     342.75 ± 22%  interrupts.CPU132.NMI:Non-maskable_interrupts
    156.75 ± 72%    +118.7%     342.75 ± 22%  interrupts.CPU132.PMI:Performance_monitoring_interrupts
    166.00 ± 91%    +149.7%     414.50 ± 35%  interrupts.CPU133.NMI:Non-maskable_interrupts
    166.00 ± 91%    +149.7%     414.50 ± 35%  interrupts.CPU133.PMI:Performance_monitoring_interrupts
      2936 ±  4%      +8.5%       3185 ±  5%  interrupts.CPU14.CAL:Function_call_interrupts
    114.25 ±173%    +251.6%     401.75 ± 40%  interrupts.CPU14.NMI:Non-maskable_interrupts
    114.25 ±173%    +251.6%     401.75 ± 40%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
     64.50 ±171%    +466.3%     365.25 ± 50%  interrupts.CPU22.NMI:Non-maskable_interrupts
     64.50 ±171%    +466.3%     365.25 ± 50%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    103.75 ±173%    +281.9%     396.25 ± 50%  interrupts.CPU24.NMI:Non-maskable_interrupts
    103.75 ±173%    +281.9%     396.25 ± 50%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
    106.50 ±173%    +232.6%     354.25 ± 61%  interrupts.CPU25.NMI:Non-maskable_interrupts
    106.50 ±173%    +232.6%     354.25 ± 61%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
     59.00 ±173%    +763.6%     509.50 ± 34%  interrupts.CPU35.NMI:Non-maskable_interrupts
     59.00 ±173%    +763.6%     509.50 ± 34%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      0.00       +2.5e+105%       2533 ±141%  interrupts.CPU36.NMI:Non-maskable_interrupts
      0.00       +2.5e+105%       2533 ±141%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      0.25 ±173%  +79400.0%     198.75 ±165%  interrupts.CPU37.RES:Rescheduling_interrupts
    105.00 ±100%    +362.4%     485.50 ± 19%  interrupts.CPU38.NMI:Non-maskable_interrupts
    105.00 ±100%    +362.4%     485.50 ± 19%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    307.75 ± 42%     +72.5%     530.75 ± 27%  interrupts.CPU44.NMI:Non-maskable_interrupts
    307.75 ± 42%     +72.5%     530.75 ± 27%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      2324 ± 48%     +52.5%       3545 ± 14%  interrupts.CPU47.CAL:Function_call_interrupts
    143.00 ± 79%    +200.5%     429.75 ± 34%  interrupts.CPU47.NMI:Non-maskable_interrupts
    143.00 ± 79%    +200.5%     429.75 ± 34%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      2901           +16.9%       3391 ± 13%  interrupts.CPU48.CAL:Function_call_interrupts
     65.75 ±173%    +643.3%     488.75 ± 24%  interrupts.CPU49.NMI:Non-maskable_interrupts
     65.75 ±173%    +643.3%     488.75 ± 24%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
    262.75 ± 75%    +102.9%     533.00 ± 11%  interrupts.CPU6.NMI:Non-maskable_interrupts
    262.75 ± 75%    +102.9%     533.00 ± 11%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
     45863 ± 11%     +18.2%      54231 ±  8%  interrupts.NMI:Non-maskable_interrupts
     45863 ± 11%     +18.2%      54231 ±  8%  interrupts.PMI:Performance_monitoring_interrupts
    150919 ±  2%     -23.6%     115358 ±  5%  softirqs.CPU0.TIMER
    148060 ±  5%     -27.7%     107015 ±  9%  softirqs.CPU1.TIMER
    146702 ±  5%     -27.3%     106688 ± 10%  softirqs.CPU10.TIMER
    133811 ±  9%     -24.0%     101741 ± 10%  softirqs.CPU101.TIMER
    142416 ± 12%     -28.7%     101500 ± 11%  softirqs.CPU106.TIMER
    146474 ±  5%     -27.4%     106399 ±  9%  softirqs.CPU11.TIMER
     34294 ±  6%     +17.6%      40334 ±  9%  softirqs.CPU110.RCU
    146844 ±  5%     -26.8%     107534 ± 10%  softirqs.CPU12.TIMER
    126750 ± 18%     -24.3%      95944 ± 16%  softirqs.CPU120.TIMER
    127795 ± 17%     -24.6%      96409 ± 14%  softirqs.CPU124.TIMER
    146594 ±  5%     -26.9%     107230 ± 10%  softirqs.CPU13.TIMER
     30688 ± 19%     +28.7%      39483 ±  3%  softirqs.CPU131.SCHED
     34375 ±  9%     +12.2%      38578 ±  3%  softirqs.CPU137.SCHED
    146022 ±  5%     -26.6%     107155 ±  9%  softirqs.CPU14.TIMER
    147464 ±  5%     -27.4%     107006 ±  9%  softirqs.CPU15.TIMER
    146391 ±  5%     -26.9%     106973 ± 10%  softirqs.CPU16.TIMER
     32189 ±  5%     +18.6%      38174 ±  4%  softirqs.CPU17.RCU
    145769 ±  5%     -26.5%     107152 ± 10%  softirqs.CPU17.TIMER
    145676 ±  5%     -27.9%     104978 ±  8%  softirqs.CPU2.TIMER
     45664 ±  3%     +19.5%      54579 ± 10%  softirqs.CPU25.RCU
    134271 ±  9%     -22.8%     103712 ± 10%  softirqs.CPU29.TIMER
    147086 ±  5%     -28.7%     104821 ±  7%  softirqs.CPU3.TIMER
    146579 ±  5%     -26.8%     107254 ± 10%  softirqs.CPU4.TIMER
    127289 ± 18%     -17.9%     104558 ± 26%  softirqs.CPU40.TIMER
     41433 ±  7%     +45.9%      60459 ± 17%  softirqs.CPU42.RCU
    139773 ± 14%     -25.1%     104733 ± 25%  softirqs.CPU45.TIMER
    127292 ± 18%     -24.6%      95971 ± 15%  softirqs.CPU48.TIMER
    147239 ±  5%     -28.1%     105898 ±  9%  softirqs.CPU5.TIMER
     65633 ± 33%     -31.1%      45253 ± 13%  softirqs.CPU59.RCU
     26435 ± 43%     +48.8%      39322 ±  3%  softirqs.CPU59.SCHED
    146873 ±  5%     -28.1%     105649 ±  8%  softirqs.CPU6.TIMER
     33549 ± 11%     +13.5%      38092 ±  4%  softirqs.CPU60.SCHED
    146984 ±  5%     -26.9%     107382 ± 10%  softirqs.CPU7.TIMER
    149682 ±  6%     -27.1%     109080 ±  8%  softirqs.CPU72.TIMER
    145895 ±  6%     -28.0%     105085 ±  8%  softirqs.CPU73.TIMER
     39346 ±  4%     -13.0%      34218 ±  3%  softirqs.CPU74.SCHED
    145575 ±  5%     -27.9%     104908 ±  9%  softirqs.CPU74.TIMER
     39460 ±  3%     -11.8%      34790 ±  6%  softirqs.CPU75.SCHED
    146290 ±  5%     -28.0%     105320 ±  9%  softirqs.CPU75.TIMER
    146175 ±  5%     -28.2%     104974 ±  8%  softirqs.CPU76.TIMER
    145762 ±  6%     -27.4%     105757 ±  9%  softirqs.CPU77.TIMER
    145193 ±  6%     -27.5%     105277 ±  9%  softirqs.CPU78.TIMER
    145984 ±  6%     -27.8%     105374 ±  9%  softirqs.CPU79.TIMER
    146269 ±  5%     -27.0%     106745 ±  9%  softirqs.CPU8.TIMER
     39112 ±  5%      -9.4%      35428 ±  6%  softirqs.CPU80.SCHED
    145761 ±  6%     -28.6%     104080 ±  9%  softirqs.CPU80.TIMER
    145316 ±  6%     -27.7%     105019 ± 10%  softirqs.CPU81.TIMER
     39480 ±  5%      -8.9%      35977 ±  5%  softirqs.CPU82.SCHED
    146148 ±  5%     -27.5%     105939 ± 10%  softirqs.CPU82.TIMER
     39201 ±  3%      -9.9%      35317 ±  2%  softirqs.CPU83.SCHED
    145638 ±  5%     -27.6%     105426 ±  9%  softirqs.CPU83.TIMER
    145700 ±  5%     -27.9%     105071 ± 10%  softirqs.CPU84.TIMER
    145988 ±  5%     -27.7%     105618 ±  9%  softirqs.CPU85.TIMER
    146574 ±  5%     -28.5%     104782 ± 10%  softirqs.CPU86.TIMER
     39053 ±  3%      -7.5%      36116 ±  4%  softirqs.CPU87.SCHED
    145677 ±  6%     -27.8%     105132 ± 10%  softirqs.CPU87.TIMER
     39381 ±  4%     -10.9%      35075 ±  2%  softirqs.CPU88.SCHED
    145564 ±  6%     -27.9%     105023 ± 10%  softirqs.CPU88.TIMER
    145358 ±  5%     -27.8%     105009 ± 10%  softirqs.CPU89.TIMER
    146739 ±  6%     -27.7%     106061 ±  9%  softirqs.CPU9.TIMER
     36915 ±  4%     +19.4%      44076 ± 13%  softirqs.CPU90.RCU
     34271 ±  4%     +23.8%      42426 ± 12%  softirqs.CPU92.RCU
   1867609           -10.4%    1672893        softirqs.NET_RX





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.0.0-00010-gbc12417" of type "text/plain" (187612 bytes)

View attachment "job-script" of type "text/plain" (7427 bytes)

View attachment "job.yaml" of type "text/plain" (5224 bytes)

View attachment "reproduce" of type "text/plain" (292 bytes)

Powered by blists - more mailing lists