lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200308140200.GO5972@shao2-debian>
Date:   Sun, 8 Mar 2020 22:02:00 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     Jann Horn <jannh@...gle.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        LKML <linux-kernel@...r.kernel.org>, x86-ml <x86@...nel.org>,
        Ingo Molnar <mingo@...nel.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Borislav Petkov <bp@...e.de>, lkp@...ts.01.org
Subject: [futex] 8019ad13ef: will-it-scale.per_process_ops -97.8% regression

Greeting,

FYI, we noticed a -97.8% regression of will-it-scale.per_process_ops due to commit:


commit: 8019ad13ef7f64be44d4f892af9c840179009254 ("futex: Fix inode life-time issue")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git locking/urgent

in testcase: will-it-scale
on test machine: 104 threads Skylake with 192G memory
with following parameters:

	nr_task: 100%
	mode: process
	test: futex2
	cpufreq_governor: performance
	ucode: 0x2000065

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -99.5% regression            |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=50%                                                               |
|                  | test=futex2                                                               |
|                  | ucode=0x500002c                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -99.7% regression            |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=100%                                                              |
|                  | test=futex4                                                               |
|                  | ucode=0x500002c                                                           |
+------------------+---------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-skl-fpga01/futex2/will-it-scale/0x2000065

commit: 
  v5.6-rc4
  8019ad13ef ("futex: Fix inode life-time issue")

        v5.6-rc4 8019ad13ef7f64be44d4f892af9 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   1083510           -97.8%      23904        will-it-scale.per_process_ops
 1.127e+08           -97.8%    2486092        will-it-scale.workload
      3352 ±  2%      +2.3%       3430 ±  2%  boot-time.idle
     81512 ±  2%     +22.6%      99903 ±  3%  meminfo.Shmem
     70106 ± 10%     +20.8%      84714 ± 11%  numa-meminfo.node1.Shmem
     17521 ± 10%     +20.9%      21181 ± 11%  numa-vmstat.node1.nr_shmem
 1.071e+08 ± 65%     -82.5%   18763577 ±  8%  cpuidle.C6.time
    140072 ± 53%     -66.5%      46887 ±  4%  cpuidle.C6.usage
     63.27           +35.5       98.79        mpstat.cpu.all.sys%
     36.35           -35.5        0.84        mpstat.cpu.all.usr%
     62.40           +57.1%      98.00        vmstat.cpu.sy
     35.60           -97.2%       1.00        vmstat.cpu.us
      1710 ±  4%      -9.3%       1551 ±  2%  vmstat.system.cs
     88040            +5.6%      93005        proc-vmstat.nr_active_anon
    289055            +1.7%     293827        proc-vmstat.nr_file_pages
      7012            +1.7%       7134        proc-vmstat.nr_inactive_anon
     20363 ±  2%     +22.6%      24967 ±  3%  proc-vmstat.nr_shmem
     88040            +5.6%      93005        proc-vmstat.nr_zone_active_anon
      7012            +1.7%       7134        proc-vmstat.nr_zone_inactive_anon
     19355 ±  4%     +35.0%      26131 ±  4%  proc-vmstat.pgactivate
    709.60 ± 45%     +60.2%       1136 ± 21%  interrupts.38:PCI-MSI.67633153-edge.eth0-TxRx-0
    388.40 ± 15%    +128.4%     887.00 ± 35%  interrupts.CPU1.RES:Rescheduling_interrupts
    319.20 ±  4%     +69.6%     541.50 ± 22%  interrupts.CPU20.RES:Rescheduling_interrupts
    709.60 ± 45%     +60.2%       1136 ± 21%  interrupts.CPU30.38:PCI-MSI.67633153-edge.eth0-TxRx-0
    349.00 ±  7%     -11.3%     309.50        interrupts.CPU44.RES:Rescheduling_interrupts
      1566 ± 72%     -63.1%     577.50 ± 43%  interrupts.CPU51.RES:Rescheduling_interrupts
    433.80 ± 35%    +137.4%       1029 ± 75%  interrupts.CPU61.RES:Rescheduling_interrupts
    311.80           +57.2%     490.00 ± 20%  interrupts.CPU82.RES:Rescheduling_interrupts
     90.00 ± 12%     +55.0%     139.50 ± 26%  interrupts.IWI:IRQ_work_interrupts
     59788 ±  9%     -23.1%      45970        interrupts.RES:Rescheduling_interrupts
    183.06 ±  2%     -15.5%     154.64 ±  3%  sched_debug.cfs_rq:/.exec_clock.stddev
    253696 ±  9%     -19.7%     203713 ±  8%  sched_debug.cfs_rq:/.min_vruntime.stddev
     61.43 ±  8%     -17.7%      50.54 ± 10%  sched_debug.cfs_rq:/.nr_spread_over.max
    217149 ±115%     -88.5%      24957 ± 23%  sched_debug.cfs_rq:/.spread0.max
    253777 ±  9%     -19.7%     203720 ±  8%  sched_debug.cfs_rq:/.spread0.stddev
      1173 ±  5%     -10.9%       1045        sched_debug.cfs_rq:/.util_avg.max
     68.48 ± 15%     -24.6%      51.65 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
     12.01 ± 13%     -46.2%       6.46 ± 15%  sched_debug.cpu.clock.stddev
     12.01 ± 13%     -46.2%       6.46 ± 15%  sched_debug.cpu.clock_task.stddev
      0.00 ±  8%     -26.7%       0.00 ±  6%  sched_debug.cpu.next_balance.stddev
     44648 ± 29%     -25.4%      33316 ±  9%  sched_debug.cpu.nr_switches.max
     56.73 ± 10%     -19.8%      45.50 ±  8%  sched_debug.cpu.nr_uninterruptible.max
     41543 ± 31%     -30.4%      28900 ±  9%  sched_debug.cpu.sched_count.max
    583.13 ±  7%     +88.4%       1098        sched_debug.cpu.sched_count.min
      1149 ±  4%     -10.4%       1029 ±  3%  sched_debug.cpu.ttwu_count.avg
     20076 ± 29%     -30.7%      13922 ± 10%  sched_debug.cpu.ttwu_count.max
     23.79           -23.8        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.syscall
     12.78           -12.8        0.00        perf-profile.calltrace.cycles-pp.get_futex_key.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
     11.49           -11.5        0.00        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.syscall
      9.84            -9.8        0.00        perf-profile.calltrace.cycles-pp.get_user_pages_fast.get_futex_key.futex_wait_setup.futex_wait.do_futex
      8.66            -8.7        0.00        perf-profile.calltrace.cycles-pp.gup_pgd_range.get_user_pages_fast.get_futex_key.futex_wait_setup.futex_wait
     62.41           +37.0       99.39        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.syscall
     59.79           +39.6       99.37        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     34.82           +64.1       98.89        perf-profile.calltrace.cycles-pp.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     26.16           +72.7       98.85        perf-profile.calltrace.cycles-pp.do_futex.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     24.33           +74.5       98.83        perf-profile.calltrace.cycles-pp.futex_wait.do_futex.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe
     21.19           +77.6       98.79        perf-profile.calltrace.cycles-pp.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex.do_syscall_64
      1.12 ±  3%     +96.5       97.64        perf-profile.calltrace.cycles-pp._raw_spin_lock.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
      0.00           +97.5       97.55        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.futex_wait_setup.futex_wait.do_futex
     22.01           -21.7        0.28        perf-profile.children.cycles-pp.entry_SYSCALL_64
     13.37           -13.1        0.26        perf-profile.children.cycles-pp.syscall_return_via_sysret
     12.80           -12.5        0.27        perf-profile.children.cycles-pp.get_futex_key
      9.86            -9.7        0.21 ±  3%  perf-profile.children.cycles-pp.get_user_pages_fast
      8.75            -8.6        0.18 ±  2%  perf-profile.children.cycles-pp.gup_pgd_range
      2.62 ±  2%      -2.6        0.05 ±  8%  perf-profile.children.cycles-pp.get_futex_value_locked
      0.69 ± 13%      -0.3        0.38 ±  9%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.74 ± 14%      -0.3        0.43 ±  9%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.09 ±  8%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.ktime_get
      0.10 ± 13%      -0.0        0.06 ± 20%  perf-profile.children.cycles-pp.clockevents_program_event
     99.95            +0.0       99.97        perf-profile.children.cycles-pp.syscall
     62.82           +36.6       99.41        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     60.91           +38.5       99.40        perf-profile.children.cycles-pp.do_syscall_64
     34.97           +63.9       98.89        perf-profile.children.cycles-pp.__x64_sys_futex
     26.26           +72.6       98.85        perf-profile.children.cycles-pp.do_futex
     24.38           +74.5       98.84        perf-profile.children.cycles-pp.futex_wait
     21.36           +77.4       98.79        perf-profile.children.cycles-pp.futex_wait_setup
      1.17 ±  3%     +96.5       97.65        perf-profile.children.cycles-pp._raw_spin_lock
      0.19 ± 27%     +97.4       97.59        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     25.10           -24.6        0.47        perf-profile.self.cycles-pp.do_syscall_64
     20.14           -19.9        0.24        perf-profile.self.cycles-pp.entry_SYSCALL_64
     13.34           -13.1        0.26        perf-profile.self.cycles-pp.syscall_return_via_sysret
      8.51            -8.5        0.00        perf-profile.self.cycles-pp.__x64_sys_futex
      8.59            -8.4        0.18 ±  2%  perf-profile.self.cycles-pp.gup_pgd_range
      3.11            -3.1        0.06        perf-profile.self.cycles-pp.syscall
      2.94            -2.9        0.03 ±100%  perf-profile.self.cycles-pp.futex_wait
      2.60 ±  2%      -2.5        0.05        perf-profile.self.cycles-pp.get_futex_value_locked
      3.08 ±  4%      -2.3        0.79        perf-profile.self.cycles-pp.futex_wait_setup
      1.12 ±  2%      -1.1        0.06 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock
      0.19 ± 27%     +97.0       97.19        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.09 ± 17%   +5811.1%       5.46 ± 14%  perf-stat.i.MPKI
 1.698e+10           -85.9%  2.398e+09        perf-stat.i.branch-instructions
      0.71            -0.3        0.40        perf-stat.i.branch-miss-rate%
 1.174e+08           -91.6%    9837603        perf-stat.i.branch-misses
   1437072 ± 93%   +1602.4%   24464880 ± 15%  perf-stat.i.cache-misses
   8173729 ± 12%    +625.4%   59295944 ± 14%  perf-stat.i.cache-references
      1683 ±  4%      -9.8%       1518 ±  2%  perf-stat.i.context-switches
      2.45          +983.6%      26.53        perf-stat.i.cpi
    124.91 ±  4%     -11.3%     110.80        perf-stat.i.cpu-migrations
    338598 ± 39%     -96.4%      12036 ± 15%  perf-stat.i.cycles-between-cache-misses
      0.40            -0.3        0.10        perf-stat.i.dTLB-load-miss-rate%
 1.123e+08           -97.7%    2574834        perf-stat.i.dTLB-load-misses
 2.802e+10           -90.4%  2.691e+09        perf-stat.i.dTLB-loads
      0.00 ± 30%      +0.0        0.00 ±  2%  perf-stat.i.dTLB-store-miss-rate%
 1.807e+10           -97.1%   5.23e+08        perf-stat.i.dTLB-stores
     97.15            -1.2       95.94        perf-stat.i.iTLB-load-miss-rate%
 1.084e+08           -97.7%    2480944        perf-stat.i.iTLB-load-misses
   2935981 ±  3%     -96.3%     107316 ±  8%  perf-stat.i.iTLB-loads
 1.174e+11           -90.7%  1.087e+10        perf-stat.i.instructions
      1126 ±  2%    +296.3%       4463        perf-stat.i.instructions-per-iTLB-miss
      0.41           -90.1%       0.04        perf-stat.i.ipc
     93.84            +5.8       99.68        perf-stat.i.node-load-miss-rate%
    114511 ±  8%   +4108.2%    4818829        perf-stat.i.node-load-misses
     97.73            +1.8       99.51        perf-stat.i.node-store-miss-rate%
    395666 ±174%   +1252.7%    5352037        perf-stat.i.node-store-misses
      0.07 ± 12%   +7691.4%       5.46 ± 14%  perf-stat.overall.MPKI
      0.69            -0.3        0.41        perf-stat.overall.branch-miss-rate%
     16.14 ± 76%     +25.1       41.20        perf-stat.overall.cache-miss-rate%
      2.45          +980.6%      26.49        perf-stat.overall.cpi
    313480 ± 38%     -96.2%      12060 ± 15%  perf-stat.overall.cycles-between-cache-misses
      0.40            -0.3        0.10        perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  2%      +0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     97.35            -1.5       95.85        perf-stat.overall.iTLB-load-miss-rate%
      1082          +304.9%       4383        perf-stat.overall.instructions-per-iTLB-miss
      0.41           -90.7%       0.04        perf-stat.overall.ipc
     88.45           +11.3       99.71        perf-stat.overall.node-load-miss-rate%
     95.64 ±  2%      +4.3       99.94        perf-stat.overall.node-store-miss-rate%
    313682          +320.2%    1318005        perf-stat.overall.path-length
 1.692e+10           -85.9%   2.39e+09        perf-stat.ps.branch-instructions
  1.17e+08           -91.6%    9804610        perf-stat.ps.branch-misses
   1436636 ± 93%   +1597.3%   24384729 ± 15%  perf-stat.ps.cache-misses
   8192823 ± 12%    +621.6%   59120349 ± 14%  perf-stat.ps.cache-references
      1674 ±  4%      -9.7%       1512 ±  2%  perf-stat.ps.context-switches
    124.46 ±  4%     -11.1%     110.64        perf-stat.ps.cpu-migrations
  1.12e+08           -97.7%    2570860        perf-stat.ps.dTLB-load-misses
 2.792e+10           -90.4%  2.683e+09        perf-stat.ps.dTLB-loads
 1.801e+10           -97.1%  5.214e+08        perf-stat.ps.dTLB-stores
 1.081e+08           -97.7%    2472640        perf-stat.ps.iTLB-load-misses
   2943940 ±  3%     -96.4%     107035 ±  7%  perf-stat.ps.iTLB-loads
  1.17e+11           -90.7%  1.084e+10        perf-stat.ps.instructions
    114380 ±  8%   +4098.9%    4802759        perf-stat.ps.node-load-misses
    394413 ±174%   +1252.4%    5334107        perf-stat.ps.node-store-misses
 3.535e+13           -90.7%  3.276e+12        perf-stat.total.instructions
    133538 ± 14%     -38.1%      82644 ±  3%  softirqs.CPU0.RCU
    132834 ± 13%     -50.3%      66075 ± 13%  softirqs.CPU1.RCU
    132513 ± 13%     -58.8%      54575        softirqs.CPU10.RCU
    132124 ± 13%     -60.5%      52249        softirqs.CPU100.RCU
    132258 ± 14%     -58.3%      55157 ±  6%  softirqs.CPU101.RCU
    132418 ± 14%     -54.3%      60552 ± 21%  softirqs.CPU102.RCU
    132676 ± 14%     -48.1%      68839 ±  8%  softirqs.CPU103.RCU
    132612 ± 13%     -58.2%      55489        softirqs.CPU11.RCU
    132754 ± 13%     -58.1%      55632 ±  2%  softirqs.CPU12.RCU
    132826 ± 13%     -54.8%      59984 ± 13%  softirqs.CPU13.RCU
    132635 ± 13%     -59.6%      53586 ±  2%  softirqs.CPU14.RCU
    132400 ± 13%     -55.3%      59235 ± 12%  softirqs.CPU15.RCU
    132348 ± 13%     -56.7%      57370 ±  4%  softirqs.CPU16.RCU
    132695 ± 13%     -52.0%      63693 ± 11%  softirqs.CPU17.RCU
    132428 ± 13%     -57.8%      55824        softirqs.CPU18.RCU
    132665 ± 13%     -56.9%      57154 ±  2%  softirqs.CPU19.RCU
    132981 ± 14%     -54.1%      61052 ±  8%  softirqs.CPU2.RCU
    132907 ± 12%     -51.8%      64068 ±  7%  softirqs.CPU20.RCU
     95558 ±  4%      +4.4%      99774 ±  5%  softirqs.CPU20.TIMER
    132739 ± 13%     -56.6%      57670 ±  8%  softirqs.CPU21.RCU
    132322 ± 13%     -59.2%      54037        softirqs.CPU22.RCU
    132263 ± 13%     -56.4%      57703 ±  6%  softirqs.CPU23.RCU
    133419 ± 14%     -51.7%      64503 ± 14%  softirqs.CPU24.RCU
    132260 ± 14%     -55.7%      58593 ±  5%  softirqs.CPU25.RCU
    133206 ± 13%     -55.9%      58763 ±  5%  softirqs.CPU26.RCU
    132544 ± 13%     -59.0%      54300 ±  5%  softirqs.CPU27.RCU
    135011 ± 12%     -52.8%      63680 ± 14%  softirqs.CPU28.RCU
    136146 ± 14%     -54.1%      62424 ±  4%  softirqs.CPU29.RCU
    133053 ± 13%     -58.8%      54756        softirqs.CPU3.RCU
    132647 ± 13%     -56.2%      58126 ±  7%  softirqs.CPU30.RCU
    133653 ± 14%     -53.9%      61655 ± 13%  softirqs.CPU31.RCU
    132738 ± 13%     -57.0%      57057 ±  2%  softirqs.CPU32.RCU
    132773 ± 14%     -57.2%      56850        softirqs.CPU33.RCU
    132677 ± 14%     -54.5%      60350 ±  7%  softirqs.CPU34.RCU
    133096 ± 13%     -55.2%      59604 ± 13%  softirqs.CPU35.RCU
    132458 ± 13%     -58.0%      55672        softirqs.CPU36.RCU
    134258 ± 14%     -58.2%      56118 ±  2%  softirqs.CPU37.RCU
    132773 ± 13%     -58.7%      54770 ±  2%  softirqs.CPU38.RCU
    133645 ± 13%     -57.6%      56641 ±  5%  softirqs.CPU39.RCU
    132461 ± 13%     -58.4%      55107        softirqs.CPU4.RCU
    133268 ± 14%     -59.7%      53743        softirqs.CPU40.RCU
    132863 ± 13%     -57.6%      56313 ±  4%  softirqs.CPU41.RCU
    132173 ± 13%     -57.8%      55811 ±  5%  softirqs.CPU42.RCU
    132577 ± 14%     -53.6%      61514 ± 10%  softirqs.CPU43.RCU
    133333 ± 12%     -59.3%      54292 ±  2%  softirqs.CPU44.RCU
    132246 ± 14%     -49.7%      66517 ± 23%  softirqs.CPU45.RCU
    132119 ± 13%     -57.8%      55723 ±  2%  softirqs.CPU46.RCU
    132032 ± 14%     -55.9%      58246 ± 10%  softirqs.CPU47.RCU
    133675 ± 14%     -57.2%      57275 ±  5%  softirqs.CPU48.RCU
    133657 ± 14%     -56.4%      58316 ±  9%  softirqs.CPU49.RCU
    132616 ± 13%     -54.4%      60525 ±  3%  softirqs.CPU5.RCU
    133257 ± 14%     -54.3%      60924 ±  8%  softirqs.CPU50.RCU
    135262 ± 14%     -41.9%      78605 ±  3%  softirqs.CPU51.RCU
    133227 ± 13%     -57.9%      56050 ±  3%  softirqs.CPU52.RCU
    132499 ± 13%     -53.5%      61583 ± 14%  softirqs.CPU53.RCU
    132702 ± 13%     -54.9%      59828 ± 13%  softirqs.CPU54.RCU
    132585 ± 13%     -57.5%      56314 ±  6%  softirqs.CPU55.RCU
    132301 ± 13%     -55.7%      58625 ± 11%  softirqs.CPU56.RCU
    132796 ± 14%     -49.5%      67024 ± 19%  softirqs.CPU57.RCU
    132446 ± 13%     -55.6%      58852 ±  8%  softirqs.CPU58.RCU
    132257 ± 13%     -54.3%      60405 ±  8%  softirqs.CPU59.RCU
    132886 ± 13%     -55.7%      58916 ± 14%  softirqs.CPU6.RCU
    132246 ± 13%     -57.0%      56914 ±  7%  softirqs.CPU60.RCU
     94721 ±  2%      +3.8%      98338 ±  4%  softirqs.CPU60.TIMER
    132355 ± 13%     -56.0%      58198 ±  7%  softirqs.CPU61.RCU
    132945 ± 13%     -58.9%      54586 ±  3%  softirqs.CPU62.RCU
    133128 ± 13%     -59.5%      53952 ±  3%  softirqs.CPU63.RCU
    132088 ± 13%     -58.3%      55137        softirqs.CPU64.RCU
    133245 ± 13%     -53.9%      61371 ± 15%  softirqs.CPU65.RCU
    133008 ± 13%     -59.3%      54159 ±  5%  softirqs.CPU66.RCU
    132695 ± 12%     -57.6%      56301 ±  9%  softirqs.CPU67.RCU
    132699 ± 12%     -59.2%      54156 ±  3%  softirqs.CPU68.RCU
    132521 ± 14%     -55.3%      59196 ± 10%  softirqs.CPU69.RCU
    132613 ± 13%     -56.8%      57326 ±  6%  softirqs.CPU7.RCU
    132114 ± 14%     -57.2%      56604        softirqs.CPU70.RCU
    132283 ± 13%     -58.6%      54754        softirqs.CPU71.RCU
    131985 ± 13%     -57.4%      56219 ±  5%  softirqs.CPU72.RCU
    132131 ± 13%     -58.3%      55090 ±  5%  softirqs.CPU73.RCU
    132634 ± 14%     -59.3%      54042 ±  2%  softirqs.CPU74.RCU
    131921 ± 13%     -58.4%      54887 ±  2%  softirqs.CPU75.RCU
    133164 ± 13%     -54.0%      61189 ± 11%  softirqs.CPU76.RCU
    132137 ± 14%     -57.3%      56443 ±  6%  softirqs.CPU77.RCU
    132102 ± 13%     -56.6%      57282 ±  9%  softirqs.CPU78.RCU
    132005 ± 14%     -58.6%      54708 ±  6%  softirqs.CPU79.RCU
    132056 ± 13%     -57.9%      55577 ±  3%  softirqs.CPU8.RCU
    133654 ± 14%     -56.1%      58668 ±  7%  softirqs.CPU80.RCU
    132611 ± 14%     -58.8%      54679 ±  5%  softirqs.CPU81.RCU
    132833 ± 13%     -53.5%      61722 ±  3%  softirqs.CPU82.RCU
    132671 ± 13%     -54.5%      60333 ± 12%  softirqs.CPU83.RCU
    132128 ± 13%     -56.6%      57349 ±  4%  softirqs.CPU84.RCU
    132394 ± 14%     -58.9%      54349        softirqs.CPU85.RCU
    132248 ± 14%     -56.0%      58222 ±  4%  softirqs.CPU86.RCU
    132741 ± 13%     -54.8%      60029 ± 13%  softirqs.CPU87.RCU
    132620 ± 14%     -57.1%      56887 ±  6%  softirqs.CPU88.RCU
    132853 ± 14%     -55.7%      58902 ±  6%  softirqs.CPU89.RCU
    132754 ± 14%     -60.0%      53118        softirqs.CPU9.RCU
    131899 ± 13%     -58.3%      54938 ±  3%  softirqs.CPU90.RCU
    132085 ± 13%     -57.7%      55886 ±  4%  softirqs.CPU91.RCU
    131887 ± 14%     -58.1%      55210 ±  5%  softirqs.CPU92.RCU
    132891 ± 12%     -59.2%      54177 ±  2%  softirqs.CPU93.RCU
    131918 ± 14%     -58.5%      54802 ±  2%  softirqs.CPU94.RCU
    132357 ± 13%     -55.2%      59242 ± 10%  softirqs.CPU95.RCU
    131921 ± 14%     -59.3%      53676 ±  3%  softirqs.CPU96.RCU
    131814 ± 13%     -56.5%      57361 ± 11%  softirqs.CPU97.RCU
    132857 ± 13%     -59.6%      53711 ±  3%  softirqs.CPU98.RCU
    131781 ± 13%     -52.0%      63235 ± 10%  softirqs.CPU99.RCU
  13801586 ± 13%     -56.2%    6051612        softirqs.RCU


                                                                                
                             will-it-scale.per_process_ops                      
                                                                                
  1.2e+06 +-----------------------------------------------------------------+   
          |..+.+..+.+..+.+..+.+..    .+.+..+..+.+..+.+..+.+..+              |   
    1e+06 |-+                    +.+.                                       |   
          |                                                                 |   
          |                                                                 |   
   800000 |-+                                                               |   
          |                                                                 |   
   600000 |-+                                                               |   
          |                                                                 |   
   400000 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
   200000 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                will-it-scale.workload                          
                                                                                
  1.2e+08 +-----------------------------------------------------------------+   
          |..+.+..+.+..+.+..+.+..+.+..+.+..+..+.+..+.+..+.+..+              |   
    1e+08 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    8e+07 |-+                                                               |   
          |                                                                 |   
    6e+07 |-+                                                               |   
          |                                                                 |   
    4e+07 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    2e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2ap3: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/50%/debian-x86_64-20191114.cgz/lkp-csl-2ap3/futex2/will-it-scale/0x500002c

commit: 
  v5.6-rc4
  8019ad13ef ("futex: Fix inode life-time issue")

        v5.6-rc4 8019ad13ef7f64be44d4f892af9 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :5           20%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
          5:5         -100%            :4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          5:5         -100%            :4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
   4514004           -99.5%      21586        will-it-scale.per_process_ops
 4.333e+08           -99.5%    2072322        will-it-scale.workload
    188407           +34.9%     254228 ± 12%  cpuidle.POLL.time
     65937           +30.9%      86333 ±  2%  cpuidle.POLL.usage
     43.22            +6.8       50.01        mpstat.cpu.all.sys%
      6.89            -6.8        0.07 ± 31%  mpstat.cpu.all.usr%
      3615 ±  5%     -24.1%       2744 ±  4%  numa-meminfo.node1.PageTables
      6113 ±  2%     +45.5%       8894 ± 37%  numa-meminfo.node2.Mapped
    901.80 ±  5%     -24.0%     685.25 ±  4%  numa-vmstat.node1.nr_page_table_pages
    423419 ±  9%     +19.5%     505890 ± 10%  numa-vmstat.node3.numa_hit
    314732 ± 13%     +23.1%     387480 ± 14%  numa-vmstat.node3.numa_local
      8088 ± 14%     -54.5%       3678 ± 92%  proc-vmstat.numa_pages_migrated
     39243 ± 31%     -50.9%      19259 ± 58%  proc-vmstat.numa_pte_updates
      8088 ± 14%     -54.5%       3678 ± 92%  proc-vmstat.pgmigrate_success
    140728 ± 22%     +39.5%     196306 ± 33%  sched_debug.cpu.avg_idle.stddev
      8.29 ±  4%     -11.8%       7.31 ±  2%  sched_debug.cpu.clock.stddev
      8.29 ±  4%     -11.8%       7.31 ±  2%  sched_debug.cpu.clock_task.stddev
      1757 ±  7%     +28.2%       2253 ± 10%  sched_debug.cpu.sched_count.avg
      5883 ± 52%    +123.8%      13167 ± 24%  sched_debug.cpu.ttwu_local.max
    645.89 ± 27%     +81.7%       1173 ± 19%  sched_debug.cpu.ttwu_local.stddev
     24.37 ±  8%     -24.4        0.00        perf-profile.calltrace.cycles-pp.get_futex_key.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
     17.29 ±  8%     -17.3        0.00        perf-profile.calltrace.cycles-pp.get_user_pages_fast.get_futex_key.futex_wait_setup.futex_wait.do_futex
     14.67 ±  8%     -14.7        0.00        perf-profile.calltrace.cycles-pp.gup_pgd_range.get_user_pages_fast.get_futex_key.futex_wait_setup.futex_wait
     12.78 ±  8%     -12.8        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.syscall
      5.58 ±  9%      -5.6        0.00        perf-profile.calltrace.cycles-pp.get_futex_value_locked.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
     55.97 ±  9%     +19.1       75.03        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.syscall
     53.89 ±  9%     +21.1       75.02        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     51.78 ±  9%     +23.2       75.00        perf-profile.calltrace.cycles-pp.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     49.52 ±  9%     +25.5       75.00        perf-profile.calltrace.cycles-pp.do_futex.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     47.79 ±  9%     +27.2       74.98        perf-profile.calltrace.cycles-pp.futex_wait.do_futex.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe
     43.49 ±  9%     +31.5       74.96        perf-profile.calltrace.cycles-pp.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex.do_syscall_64
      2.72 ±  9%     +71.3       74.06        perf-profile.calltrace.cycles-pp._raw_spin_lock.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
      0.00           +74.0       73.98        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.futex_wait_setup.futex_wait.do_futex
     24.38 ±  8%     -24.2        0.13 ±  3%  perf-profile.children.cycles-pp.get_futex_key
     17.30 ±  8%     -17.2        0.09        perf-profile.children.cycles-pp.get_user_pages_fast
     14.80 ±  8%     -14.7        0.07 ±  5%  perf-profile.children.cycles-pp.gup_pgd_range
      8.61 ±  8%      -8.6        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64
      5.69 ±  9%      -5.7        0.00        perf-profile.children.cycles-pp.get_futex_value_locked
      5.19 ±  9%      -5.2        0.00        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.40 ±  8%      +0.1        0.45 ±  7%  perf-profile.children.cycles-pp.hrtimer_interrupt
     55.99 ±  9%     +19.1       75.05        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     54.13 ±  9%     +20.9       75.04        perf-profile.children.cycles-pp.do_syscall_64
     51.84 ±  9%     +23.2       75.00        perf-profile.children.cycles-pp.__x64_sys_futex
     49.62 ±  9%     +25.4       75.00        perf-profile.children.cycles-pp.do_futex
     47.90 ±  9%     +27.1       74.98        perf-profile.children.cycles-pp.futex_wait
     43.84 ±  9%     +31.1       74.97        perf-profile.children.cycles-pp.futex_wait_setup
      2.84 ±  9%     +71.2       74.06        perf-profile.children.cycles-pp._raw_spin_lock
      0.05 ± 55%     +74.0       74.02        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     14.38 ±  8%     -14.3        0.07 ±  5%  perf-profile.self.cycles-pp.gup_pgd_range
      6.63 ±  9%      -6.6        0.00        perf-profile.self.cycles-pp.syscall
      6.68 ±  9%      -6.0        0.72        perf-profile.self.cycles-pp.futex_wait_setup
      5.65 ±  9%      -5.6        0.00        perf-profile.self.cycles-pp.get_futex_value_locked
      5.19 ±  9%      -5.2        0.00        perf-profile.self.cycles-pp.syscall_return_via_sysret
      4.89 ±  8%      -4.9        0.00        perf-profile.self.cycles-pp.get_futex_key
      2.78 ±  9%      -2.8        0.03 ±100%  perf-profile.self.cycles-pp._raw_spin_lock
      0.05 ± 55%     +73.5       73.53        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.04 ± 20%   +3874.8%       1.78 ±  2%  perf-stat.i.MPKI
 6.553e+10           -89.8%  6.669e+09        perf-stat.i.branch-instructions
      0.34            -0.2        0.19 ±  2%  perf-stat.i.branch-miss-rate%
  2.21e+08           -94.4%   12326549 ±  2%  perf-stat.i.branch-misses
     12.05 ±  4%     +39.5       51.53 ±  2%  perf-stat.i.cache-miss-rate%
   1593189 ±  2%   +1466.6%   24959164        perf-stat.i.cache-misses
  13509037 ±  2%    +262.1%   48912007        perf-stat.i.cache-references
      0.68         +1485.1%      10.71        perf-stat.i.cpi
    198371 ±  3%     -94.0%      11944        perf-stat.i.cycles-between-cache-misses
      0.00 ± 35%      +0.0        0.00 ± 15%  perf-stat.i.dTLB-load-miss-rate%
    166434 ±  7%     -18.8%     135118 ± 10%  perf-stat.i.dTLB-load-misses
 1.053e+11           -93.4%  6.934e+09        perf-stat.i.dTLB-loads
      0.00 ± 43%      +0.0        0.00 ±  6%  perf-stat.i.dTLB-store-miss-rate%
 6.842e+10           -99.2%  5.571e+08        perf-stat.i.dTLB-stores
     94.80           -36.5       58.27        perf-stat.i.iTLB-load-miss-rate%
 2.182e+08           -97.8%    4752410 ±  3%  perf-stat.i.iTLB-load-misses
  11807610 ±  8%     -71.2%    3402406 ±  4%  perf-stat.i.iTLB-loads
 4.425e+11           -93.7%  2.785e+10        perf-stat.i.instructions
      2028          +189.4%       5869 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      1.48           -93.7%       0.09        perf-stat.i.ipc
     94.41            +5.3       99.67        perf-stat.i.node-load-miss-rate%
    295303 ±  4%   +2177.2%    6724758        perf-stat.i.node-load-misses
     26308 ±  4%     -18.2%      21515 ±  5%  perf-stat.i.node-loads
     95.05 ±  2%      +4.8       99.83        perf-stat.i.node-store-miss-rate%
    146163 ±  7%   +3894.7%    5838703        perf-stat.i.node-store-misses
      0.03 ±  2%   +5662.4%       1.76        perf-stat.overall.MPKI
      0.34            -0.2        0.18 ±  2%  perf-stat.overall.branch-miss-rate%
     11.81 ±  3%     +39.3       51.06 ±  2%  perf-stat.overall.cache-miss-rate%
      0.67         +1490.7%      10.72        perf-stat.overall.cpi
    187570 ±  2%     -93.6%      11963        perf-stat.overall.cycles-between-cache-misses
      0.00 ±  6%      +0.0        0.00 ± 11%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  5%      +0.0        0.00 ±  7%  perf-stat.overall.dTLB-store-miss-rate%
     94.87           -36.6       58.29        perf-stat.overall.iTLB-load-miss-rate%
      2027          +189.4%       5868 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      1.48           -93.7%       0.09        perf-stat.overall.ipc
     91.75            +7.9       99.68        perf-stat.overall.node-load-miss-rate%
     93.96 ±  2%      +5.9       99.90        perf-stat.overall.node-store-miss-rate%
    307347         +1214.6%    4040286        perf-stat.overall.path-length
 6.531e+10           -89.8%  6.645e+09        perf-stat.ps.branch-instructions
 2.202e+08           -94.5%   12216378 ±  2%  perf-stat.ps.branch-misses
   1585478 ±  2%   +1468.7%   24871260        perf-stat.ps.cache-misses
  13437247 ±  2%    +262.6%   48727884        perf-stat.ps.cache-references
    165108 ±  6%     -17.9%     135632 ± 11%  perf-stat.ps.dTLB-load-misses
 1.049e+11           -93.4%  6.909e+09        perf-stat.ps.dTLB-loads
 6.819e+10           -99.2%  5.548e+08        perf-stat.ps.dTLB-stores
 2.175e+08           -97.8%    4735541 ±  3%  perf-stat.ps.iTLB-load-misses
  11774264 ±  8%     -71.2%    3389286 ±  4%  perf-stat.ps.iTLB-loads
  4.41e+11           -93.7%  2.775e+10        perf-stat.ps.instructions
    293651 ±  4%   +2182.0%    6701184        perf-stat.ps.node-load-misses
     26314 ±  5%     -17.9%      21608 ±  6%  perf-stat.ps.node-loads
    145411 ±  7%   +3901.6%    5818746        perf-stat.ps.node-store-misses
 1.332e+14           -93.7%  8.373e+12        perf-stat.total.instructions
    112618 ±  7%     +10.7%     124644 ±  6%  softirqs.CPU10.RCU
    117775 ±  8%     -80.5%      22957 ± 17%  softirqs.CPU100.RCU
    115198 ±  4%     -79.8%      23223 ± 18%  softirqs.CPU101.RCU
    113368 ±  5%     -80.2%      22457 ± 16%  softirqs.CPU102.RCU
    113292 ±  5%     -80.1%      22528 ± 19%  softirqs.CPU103.RCU
     24241 ± 69%     -84.6%       3741 ±  2%  softirqs.CPU103.SCHED
    112759 ±  5%     -79.3%      23317 ± 12%  softirqs.CPU105.RCU
    112518 ±  5%     -80.0%      22531 ± 16%  softirqs.CPU106.RCU
    112793 ±  5%     -80.0%      22557 ± 17%  softirqs.CPU107.RCU
    112782 ±  5%     -80.4%      22069 ± 16%  softirqs.CPU108.RCU
    117231 ±  9%     -80.5%      22866 ± 14%  softirqs.CPU109.RCU
    112179 ±  7%      +9.6%     122920 ±  4%  softirqs.CPU11.RCU
    114710 ±  9%     -80.7%      22096 ± 14%  softirqs.CPU110.RCU
    113984 ±  4%     -79.3%      23552 ± 20%  softirqs.CPU111.RCU
    116588 ±  5%     -81.2%      21871 ± 19%  softirqs.CPU113.RCU
    115362 ±  5%     -81.3%      21628 ± 18%  softirqs.CPU115.RCU
    112419 ±  5%     -80.8%      21577 ± 17%  softirqs.CPU116.RCU
    113307 ±  5%     -63.3%      41533 ± 80%  softirqs.CPU117.RCU
    115280 ±  5%     -80.8%      22154 ± 18%  softirqs.CPU118.RCU
    116348 ±  5%     -81.3%      21755 ± 16%  softirqs.CPU121.RCU
    115888 ±  6%     -80.3%      22870 ± 13%  softirqs.CPU122.RCU
    112896 ±  5%     -80.7%      21832 ± 16%  softirqs.CPU123.RCU
    112584 ±  5%     -79.5%      23109 ± 15%  softirqs.CPU124.RCU
    116469 ±  4%     -81.1%      22033 ± 15%  softirqs.CPU125.RCU
    112576 ±  5%     -80.2%      22272 ± 17%  softirqs.CPU126.RCU
    115575 ±  5%     -80.4%      22680 ± 13%  softirqs.CPU127.RCU
    120103 ±  9%     -80.8%      23004 ± 12%  softirqs.CPU128.RCU
    111209 ±  6%     +12.4%     125045 ±  5%  softirqs.CPU13.RCU
     98953 ± 15%     +22.1%     120812 ± 18%  softirqs.CPU13.TIMER
    112711 ±  5%     -79.1%      23540 ± 15%  softirqs.CPU130.RCU
    113535 ±  5%     -63.2%      41737 ± 90%  softirqs.CPU131.RCU
    115859 ±  8%     -63.7%      42102 ± 91%  softirqs.CPU132.RCU
    117625 ± 10%     -81.4%      21838 ± 18%  softirqs.CPU133.RCU
    114713 ±  6%     -68.7%      35855 ± 76%  softirqs.CPU134.RCU
    112842 ±  5%     -81.0%      21468 ± 16%  softirqs.CPU136.RCU
    116397 ±  5%     -72.6%      31852 ± 45%  softirqs.CPU139.RCU
    115914 ±  5%     -80.9%      22197 ± 15%  softirqs.CPU142.RCU
    112669 ±  5%     -64.3%      40215 ± 74%  softirqs.CPU144.RCU
    112634 ±  5%     -80.3%      22148 ± 16%  softirqs.CPU145.RCU
    112501 ±  5%     -62.7%      41999 ± 78%  softirqs.CPU146.RCU
    114965 ±  4%     -74.1%      29769 ± 47%  softirqs.CPU148.RCU
    114777 ±  5%     -70.3%      34123 ± 37%  softirqs.CPU149.RCU
    113256 ±  4%     -80.0%      22652 ± 17%  softirqs.CPU150.RCU
    113059 ±  4%     -79.9%      22741 ± 14%  softirqs.CPU151.RCU
    116398 ± 11%     -81.3%      21806 ± 14%  softirqs.CPU152.RCU
    112239 ±  5%     -80.4%      21997 ± 17%  softirqs.CPU153.RCU
    113272 ±  5%     -80.6%      22002 ± 17%  softirqs.CPU155.RCU
    116053 ±  6%     -80.8%      22224 ± 18%  softirqs.CPU156.RCU
    113684 ±  6%     -76.1%      27218 ± 45%  softirqs.CPU157.RCU
    112696 ±  5%     -47.0%      59756 ± 61%  softirqs.CPU158.RCU
      3460 ± 18%    +322.2%      14610 ± 73%  softirqs.CPU158.SCHED
    115678 ±  6%     -67.4%      37661 ± 73%  softirqs.CPU159.RCU
    112444 ±  5%     -80.3%      22114 ± 15%  softirqs.CPU160.RCU
    116104 ± 10%     -81.0%      22030 ± 17%  softirqs.CPU161.RCU
    114027 ±  5%     -79.9%      22911 ± 18%  softirqs.CPU162.RCU
    113360 ±  5%     -80.5%      22143 ± 15%  softirqs.CPU163.RCU
    112313 ±  5%     -79.6%      22896 ± 13%  softirqs.CPU164.RCU
    111908 ±  5%     -79.9%      22538 ± 19%  softirqs.CPU165.RCU
    112378 ±  6%     -79.6%      22947 ± 14%  softirqs.CPU166.RCU
    112227 ±  5%     -80.5%      21861 ± 16%  softirqs.CPU169.RCU
    117834 ±  4%     -81.7%      21603 ± 14%  softirqs.CPU171.RCU
    112132 ±  5%     -80.5%      21860 ± 18%  softirqs.CPU172.RCU
    112088 ±  5%     -69.8%      33869 ± 59%  softirqs.CPU173.RCU
    115719 ±  6%     -81.4%      21553 ± 18%  softirqs.CPU175.RCU
    114346 ±  6%     -80.6%      22154 ± 24%  softirqs.CPU176.RCU
    114858 ±  5%     -81.5%      21231 ± 16%  softirqs.CPU177.RCU
    117242 ± 10%     -82.1%      21014 ± 18%  softirqs.CPU178.RCU
    114504 ±  5%     -81.2%      21560 ± 17%  softirqs.CPU179.RCU
    116594 ±  4%     -81.5%      21529 ± 17%  softirqs.CPU180.RCU
    116086 ±  3%     -81.6%      21353 ± 19%  softirqs.CPU181.RCU
    111841 ±  5%     -81.2%      20990 ± 18%  softirqs.CPU182.RCU
    112945 ±  5%     -81.2%      21268 ± 16%  softirqs.CPU183.RCU
    114480 ±  8%     -81.4%      21286 ± 16%  softirqs.CPU184.RCU
    112221 ±  4%     -80.8%      21540 ± 16%  softirqs.CPU185.RCU
    111925 ±  5%     -80.9%      21354 ± 18%  softirqs.CPU186.RCU
    113458 ±  9%     -80.9%      21711 ± 19%  softirqs.CPU187.RCU
    114050 ±  5%     -81.4%      21172 ± 19%  softirqs.CPU188.RCU
    114654 ±  6%     -63.5%      41800 ± 84%  softirqs.CPU189.RCU
    108488 ±  7%     +12.8%     122383 ±  5%  softirqs.CPU19.RCU
    112504 ±  5%     -81.4%      20953 ± 19%  softirqs.CPU190.RCU
    114735 ±  6%      +9.7%     125808 ±  3%  softirqs.CPU2.RCU
     24178 ± 66%     +60.9%      38913 ±  4%  softirqs.CPU2.SCHED
     98809 ± 11%     +22.6%     121169 ± 18%  softirqs.CPU2.TIMER
    110432 ±  7%     +10.3%     121816 ±  4%  softirqs.CPU20.RCU
     23656 ± 55%     +57.0%      37152 ±  4%  softirqs.CPU26.SCHED
    101821 ± 12%     +16.9%     119033 ± 17%  softirqs.CPU26.TIMER
     18058 ± 92%    +115.1%      38842 ±  4%  softirqs.CPU29.SCHED
     97618 ± 13%     +23.6%     120677 ± 18%  softirqs.CPU29.TIMER
    108935 ±  8%     +15.2%     125507 ±  7%  softirqs.CPU32.RCU
     98043 ± 14%     +23.5%     121113 ± 18%  softirqs.CPU32.TIMER
    113190 ±  8%     +12.6%     127465 ±  6%  softirqs.CPU37.RCU
     17234 ± 91%    +123.7%      38560 ±  4%  softirqs.CPU37.SCHED
     93740 ±  9%     +27.4%     119428 ± 17%  softirqs.CPU37.TIMER
     24505 ± 31%     +44.5%      35415 ± 16%  softirqs.CPU38.SCHED
     99039 ±  4%     +22.7%     121555 ± 18%  softirqs.CPU38.TIMER
    110206 ±  7%     +11.0%     122366 ±  4%  softirqs.CPU5.RCU
    107371 ±  7%     +14.6%     123069 ±  6%  softirqs.CPU55.RCU
    107806 ±  7%      +9.0%     117493 ±  8%  softirqs.CPU59.RCU
    114495 ±  7%     +12.0%     128201 ±  4%  softirqs.CPU6.RCU
    113723 ±  6%     +12.0%     127422 ±  4%  softirqs.CPU7.RCU
     17884 ± 96%    +184.9%      50954 ± 42%  softirqs.CPU7.SCHED
     95863 ± 12%     +25.5%     120313 ± 17%  softirqs.CPU7.TIMER
    110353 ±  7%     +11.6%     123116 ±  5%  softirqs.CPU73.RCU
     97912 ± 10%     +22.2%     119692 ± 18%  softirqs.CPU80.TIMER
    110219 ±  7%     +12.2%     123674 ±  7%  softirqs.CPU81.RCU
     99148 ± 14%     +21.1%     120084 ± 18%  softirqs.CPU82.TIMER
    111660 ±  7%      +9.4%     122104 ±  4%  softirqs.CPU88.RCU
    102994 ± 13%     +16.0%     119475 ± 18%  softirqs.CPU88.TIMER
    113412 ±  6%     +10.8%     125620 ±  5%  softirqs.CPU9.RCU
     29945 ± 28%     +28.9%      38610 ±  4%  softirqs.CPU9.SCHED
    112694 ±  7%     +11.1%     125240 ±  5%  softirqs.CPU90.RCU
     31112 ± 39%     +25.3%      38999 ±  5%  softirqs.CPU90.SCHED
     26420 ± 55%     +47.7%      39013 ±  4%  softirqs.CPU91.SCHED
    100775 ± 11%     +18.7%     119610 ± 18%  softirqs.CPU91.TIMER
    110721 ±  5%      +9.7%     121414 ±  6%  softirqs.CPU92.RCU
    111377 ±  7%     +10.5%     123052 ±  6%  softirqs.CPU94.RCU
    113359 ±  5%     -63.9%      40924 ± 38%  softirqs.CPU95.RCU
    113446 ±  5%     -61.4%      43794 ± 87%  softirqs.CPU96.RCU
    112028 ±  5%     -79.8%      22627 ± 20%  softirqs.CPU98.RCU
  21521903 ±  6%     -35.3%   13923062 ±  4%  softirqs.RCU
    248463 ± 14%     -27.1%     181187 ± 14%  interrupts.CAL:Function_call_interrupts
    907.80 ± 20%     -38.8%     556.00 ± 25%  interrupts.CPU0.CAL:Function_call_interrupts
      1298 ± 14%     -27.3%     944.00 ± 15%  interrupts.CPU1.CAL:Function_call_interrupts
      1293 ± 14%     -26.7%     947.25 ± 14%  interrupts.CPU10.CAL:Function_call_interrupts
      1294 ± 14%     -27.0%     945.00 ± 14%  interrupts.CPU100.CAL:Function_call_interrupts
      5511 ± 48%     +38.8%       7650 ± 24%  interrupts.CPU100.NMI:Non-maskable_interrupts
      5511 ± 48%     +38.8%       7650 ± 24%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
      1293 ± 14%     -26.9%     945.00 ± 14%  interrupts.CPU101.CAL:Function_call_interrupts
      1295 ± 14%     -27.0%     945.00 ± 14%  interrupts.CPU102.CAL:Function_call_interrupts
      1291 ± 14%     -26.8%     945.00 ± 14%  interrupts.CPU103.CAL:Function_call_interrupts
      4328 ± 52%     +76.8%       7650 ± 24%  interrupts.CPU103.NMI:Non-maskable_interrupts
      4328 ± 52%     +76.8%       7650 ± 24%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
      1294 ± 14%     -26.9%     945.50 ± 14%  interrupts.CPU104.CAL:Function_call_interrupts
      1297 ± 14%     -27.2%     945.00 ± 14%  interrupts.CPU105.CAL:Function_call_interrupts
      5501 ± 48%     +39.1%       7650 ± 24%  interrupts.CPU105.NMI:Non-maskable_interrupts
      5501 ± 48%     +39.1%       7650 ± 24%  interrupts.CPU105.PMI:Performance_monitoring_interrupts
      1294 ± 14%     -27.0%     945.00 ± 14%  interrupts.CPU106.CAL:Function_call_interrupts
      1295 ± 14%     -27.0%     945.25 ± 14%  interrupts.CPU107.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     945.25 ± 14%  interrupts.CPU108.CAL:Function_call_interrupts
      1297 ± 14%     -27.2%     945.25 ± 14%  interrupts.CPU109.CAL:Function_call_interrupts
      4336 ± 52%    +101.6%       8744        interrupts.CPU109.NMI:Non-maskable_interrupts
      4336 ± 52%    +101.6%       8744        interrupts.CPU109.PMI:Performance_monitoring_interrupts
      1294 ± 14%     -27.0%     945.50 ± 14%  interrupts.CPU11.CAL:Function_call_interrupts
      1297 ± 14%     -27.2%     945.25 ± 14%  interrupts.CPU110.CAL:Function_call_interrupts
      5510 ± 48%     +58.7%       8743        interrupts.CPU110.NMI:Non-maskable_interrupts
      5510 ± 48%     +58.7%       8743        interrupts.CPU110.PMI:Performance_monitoring_interrupts
      1297 ± 14%     -27.2%     945.25 ± 14%  interrupts.CPU111.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     945.50 ± 14%  interrupts.CPU112.CAL:Function_call_interrupts
      1294 ± 14%     -27.1%     943.50 ± 14%  interrupts.CPU113.CAL:Function_call_interrupts
      5502 ± 48%     +58.9%       8744        interrupts.CPU113.NMI:Non-maskable_interrupts
      5502 ± 48%     +58.9%       8744        interrupts.CPU113.PMI:Performance_monitoring_interrupts
      1294 ± 14%     -26.9%     946.50 ± 14%  interrupts.CPU114.CAL:Function_call_interrupts
      1293 ± 14%     -26.9%     945.50 ± 14%  interrupts.CPU115.CAL:Function_call_interrupts
      5499 ± 48%     +59.0%       8743        interrupts.CPU115.NMI:Non-maskable_interrupts
      5499 ± 48%     +59.0%       8743        interrupts.CPU115.PMI:Performance_monitoring_interrupts
      1297 ± 14%     -27.1%     945.50 ± 14%  interrupts.CPU116.CAL:Function_call_interrupts
      1296 ± 14%     -27.0%     945.75 ± 14%  interrupts.CPU117.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     945.25 ± 14%  interrupts.CPU118.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     946.00 ± 14%  interrupts.CPU119.CAL:Function_call_interrupts
      1291 ± 14%     -26.6%     947.25 ± 14%  interrupts.CPU12.CAL:Function_call_interrupts
     82.40 ±170%     -98.8%       1.00 ±173%  interrupts.CPU12.RES:Rescheduling_interrupts
      1297 ± 14%     -27.1%     946.00 ± 14%  interrupts.CPU120.CAL:Function_call_interrupts
      1297 ± 14%     -27.2%     944.00 ± 14%  interrupts.CPU121.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     946.00 ± 14%  interrupts.CPU122.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     945.75 ± 14%  interrupts.CPU123.CAL:Function_call_interrupts
      1297 ± 14%     -27.2%     944.00 ± 14%  interrupts.CPU124.CAL:Function_call_interrupts
      1293 ± 14%     -26.9%     945.75 ± 14%  interrupts.CPU125.CAL:Function_call_interrupts
      4626 ± 72%     +41.7%       6556 ± 33%  interrupts.CPU125.NMI:Non-maskable_interrupts
      4626 ± 72%     +41.7%       6556 ± 33%  interrupts.CPU125.PMI:Performance_monitoring_interrupts
      1297 ± 14%     -27.1%     945.75 ± 14%  interrupts.CPU126.CAL:Function_call_interrupts
      1295 ± 14%     -27.0%     945.75 ± 14%  interrupts.CPU127.CAL:Function_call_interrupts
      1296 ± 14%     -27.0%     945.75 ± 14%  interrupts.CPU128.CAL:Function_call_interrupts
      4621 ± 72%     +65.5%       7650 ± 24%  interrupts.CPU128.NMI:Non-maskable_interrupts
      4621 ± 72%     +65.5%       7650 ± 24%  interrupts.CPU128.PMI:Performance_monitoring_interrupts
      1296 ± 14%     -27.0%     946.25 ± 14%  interrupts.CPU129.CAL:Function_call_interrupts
      1292 ± 14%     -26.7%     947.25 ± 14%  interrupts.CPU13.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     946.00 ± 14%  interrupts.CPU130.CAL:Function_call_interrupts
      1294 ± 14%     -26.9%     946.25 ± 14%  interrupts.CPU131.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     946.00 ± 14%  interrupts.CPU132.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     946.00 ± 14%  interrupts.CPU133.CAL:Function_call_interrupts
      5192 ± 55%     +68.4%       8744        interrupts.CPU133.NMI:Non-maskable_interrupts
      5192 ± 55%     +68.4%       8744        interrupts.CPU133.PMI:Performance_monitoring_interrupts
      1293 ± 14%     -26.9%     946.00 ± 14%  interrupts.CPU134.CAL:Function_call_interrupts
      1296 ± 14%     -27.1%     945.75 ± 14%  interrupts.CPU135.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     946.00 ± 14%  interrupts.CPU136.CAL:Function_call_interrupts
      1297 ± 14%     -27.2%     945.00 ± 14%  interrupts.CPU137.CAL:Function_call_interrupts
      1296 ± 14%     -27.3%     943.00 ± 14%  interrupts.CPU138.CAL:Function_call_interrupts
      1292 ± 15%     -26.8%     945.75 ± 14%  interrupts.CPU139.CAL:Function_call_interrupts
      5490 ± 48%     +39.4%       7651 ± 24%  interrupts.CPU139.NMI:Non-maskable_interrupts
      5490 ± 48%     +39.4%       7651 ± 24%  interrupts.CPU139.PMI:Performance_monitoring_interrupts
      1290 ± 14%     -26.6%     947.25 ± 14%  interrupts.CPU14.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     945.75 ± 14%  interrupts.CPU140.CAL:Function_call_interrupts
      1296 ± 14%     -27.0%     946.00 ± 14%  interrupts.CPU141.CAL:Function_call_interrupts
      1296 ± 14%     -27.2%     944.00 ± 14%  interrupts.CPU142.CAL:Function_call_interrupts
      1295 ± 15%     -26.9%     947.00 ± 14%  interrupts.CPU143.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     946.25 ± 14%  interrupts.CPU144.CAL:Function_call_interrupts
      1297 ± 14%     -34.5%     849.75 ± 32%  interrupts.CPU145.CAL:Function_call_interrupts
      1296 ± 14%     -27.0%     946.50 ± 14%  interrupts.CPU146.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     946.50 ± 14%  interrupts.CPU147.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     946.25 ± 14%  interrupts.CPU148.CAL:Function_call_interrupts
      1294 ± 14%     -26.9%     946.25 ± 14%  interrupts.CPU149.CAL:Function_call_interrupts
      1290 ± 14%     -26.9%     943.50 ± 14%  interrupts.CPU15.CAL:Function_call_interrupts
      1298 ± 14%     -27.2%     946.00 ± 14%  interrupts.CPU150.CAL:Function_call_interrupts
      1296 ± 14%     -27.1%     945.25 ± 14%  interrupts.CPU152.CAL:Function_call_interrupts
      1294 ± 14%     -26.9%     946.00 ± 14%  interrupts.CPU153.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     946.50 ± 14%  interrupts.CPU154.CAL:Function_call_interrupts
      1295 ± 14%     -27.0%     945.25 ± 14%  interrupts.CPU155.CAL:Function_call_interrupts
      1294 ± 14%     -27.0%     945.75 ± 14%  interrupts.CPU156.CAL:Function_call_interrupts
      5811 ± 42%     +50.4%       8741        interrupts.CPU156.NMI:Non-maskable_interrupts
      5811 ± 42%     +50.4%       8741        interrupts.CPU156.PMI:Performance_monitoring_interrupts
      1294 ± 14%     -26.9%     946.25 ± 14%  interrupts.CPU157.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     945.25 ± 14%  interrupts.CPU158.CAL:Function_call_interrupts
      1294 ± 14%     -26.9%     945.75 ± 14%  interrupts.CPU159.CAL:Function_call_interrupts
      1292 ± 14%     -26.9%     945.00 ± 14%  interrupts.CPU16.CAL:Function_call_interrupts
      1296 ± 14%     -27.0%     946.00 ± 14%  interrupts.CPU160.CAL:Function_call_interrupts
      1296 ± 14%     -27.1%     945.75 ± 14%  interrupts.CPU161.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     946.00 ± 14%  interrupts.CPU162.CAL:Function_call_interrupts
      1293 ± 14%     -26.9%     945.75 ± 14%  interrupts.CPU163.CAL:Function_call_interrupts
      1299 ± 13%     -27.2%     946.00 ± 14%  interrupts.CPU164.CAL:Function_call_interrupts
      1302 ± 13%     -27.4%     945.50 ± 14%  interrupts.CPU165.CAL:Function_call_interrupts
      1301 ± 13%     -27.3%     946.50 ± 14%  interrupts.CPU166.CAL:Function_call_interrupts
      1303 ± 13%     -27.4%     946.50 ± 14%  interrupts.CPU167.CAL:Function_call_interrupts
      1302 ± 13%     -27.3%     946.50 ± 14%  interrupts.CPU168.CAL:Function_call_interrupts
      1071 ± 12%     -38.5%     658.50 ± 21%  interrupts.CPU169.CAL:Function_call_interrupts
      1296 ± 14%     -27.3%     943.25 ± 14%  interrupts.CPU17.CAL:Function_call_interrupts
      1304 ± 13%     -27.5%     945.50 ± 14%  interrupts.CPU170.CAL:Function_call_interrupts
      1307 ± 13%     -27.2%     951.50 ± 14%  interrupts.CPU171.CAL:Function_call_interrupts
      1306 ± 13%     -26.8%     955.75 ± 15%  interrupts.CPU172.CAL:Function_call_interrupts
      1307 ± 13%     -26.8%     957.75 ± 15%  interrupts.CPU173.CAL:Function_call_interrupts
      1309 ± 14%     -26.9%     958.00 ± 15%  interrupts.CPU174.CAL:Function_call_interrupts
      1316 ± 14%     -27.3%     957.50 ± 15%  interrupts.CPU175.CAL:Function_call_interrupts
      1321 ± 14%     -27.3%     960.00 ± 15%  interrupts.CPU176.CAL:Function_call_interrupts
      1321 ± 14%     -27.1%     963.25 ± 15%  interrupts.CPU177.CAL:Function_call_interrupts
      1322 ± 14%     -27.3%     961.75 ± 15%  interrupts.CPU178.CAL:Function_call_interrupts
      4923 ± 64%     +55.4%       7648 ± 24%  interrupts.CPU178.NMI:Non-maskable_interrupts
      4923 ± 64%     +55.4%       7648 ± 24%  interrupts.CPU178.PMI:Performance_monitoring_interrupts
      1322 ± 14%     -26.7%     969.00 ± 14%  interrupts.CPU179.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     946.25 ± 14%  interrupts.CPU18.CAL:Function_call_interrupts
      1317 ± 14%     -26.4%     969.00 ± 14%  interrupts.CPU180.CAL:Function_call_interrupts
      1321 ± 14%     -26.7%     969.00 ± 14%  interrupts.CPU182.CAL:Function_call_interrupts
      1321 ± 14%     -26.7%     969.25 ± 14%  interrupts.CPU183.CAL:Function_call_interrupts
      1319 ± 14%     -26.5%     969.50 ± 14%  interrupts.CPU184.CAL:Function_call_interrupts
      1316 ± 13%     -26.3%     969.75 ± 14%  interrupts.CPU185.CAL:Function_call_interrupts
      1319 ± 14%     -26.8%     965.75 ± 13%  interrupts.CPU186.CAL:Function_call_interrupts
      1316 ± 14%     -26.8%     963.75 ± 13%  interrupts.CPU187.CAL:Function_call_interrupts
      1318 ± 14%     -26.4%     969.75 ± 14%  interrupts.CPU188.CAL:Function_call_interrupts
      1322 ± 14%     -26.8%     968.50 ± 14%  interrupts.CPU189.CAL:Function_call_interrupts
      1293 ± 14%     -27.1%     943.50 ± 14%  interrupts.CPU19.CAL:Function_call_interrupts
     50.20 ± 91%     -97.0%       1.50 ± 74%  interrupts.CPU19.RES:Rescheduling_interrupts
      1321 ± 14%     -26.6%     969.75 ± 14%  interrupts.CPU190.CAL:Function_call_interrupts
      1292 ± 14%     -26.7%     947.25 ± 14%  interrupts.CPU2.CAL:Function_call_interrupts
      1295 ± 14%     -27.0%     945.50 ± 14%  interrupts.CPU20.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     947.00 ± 14%  interrupts.CPU21.CAL:Function_call_interrupts
      1289 ± 14%     -26.6%     947.00 ± 14%  interrupts.CPU22.CAL:Function_call_interrupts
      1289 ± 14%     -26.6%     946.75 ± 14%  interrupts.CPU23.CAL:Function_call_interrupts
      1295 ± 14%     -27.1%     944.50 ± 14%  interrupts.CPU24.CAL:Function_call_interrupts
      7.20 ± 55%    +466.0%      40.75 ± 98%  interrupts.CPU24.RES:Rescheduling_interrupts
      1291 ± 15%     -26.7%     947.00 ± 14%  interrupts.CPU25.CAL:Function_call_interrupts
      1295 ± 14%     -27.1%     944.25 ± 14%  interrupts.CPU26.CAL:Function_call_interrupts
      1293 ± 14%     -26.9%     945.75 ± 14%  interrupts.CPU27.CAL:Function_call_interrupts
      1296 ± 14%     -27.2%     943.75 ± 14%  interrupts.CPU28.CAL:Function_call_interrupts
      1295 ± 14%     -27.1%     944.75 ± 14%  interrupts.CPU29.CAL:Function_call_interrupts
      1291 ± 14%     -26.7%     946.50 ± 14%  interrupts.CPU3.CAL:Function_call_interrupts
      1292 ± 14%     -27.3%     939.25 ± 14%  interrupts.CPU30.CAL:Function_call_interrupts
      1295 ± 14%     -27.4%     940.50 ± 14%  interrupts.CPU31.CAL:Function_call_interrupts
      1297 ± 14%     -27.2%     943.75 ± 14%  interrupts.CPU32.CAL:Function_call_interrupts
     35.00 ±107%     -97.9%       0.75 ± 57%  interrupts.CPU32.RES:Rescheduling_interrupts
      1294 ± 14%     -27.1%     943.00 ± 14%  interrupts.CPU33.CAL:Function_call_interrupts
      1293 ± 14%     -27.0%     943.75 ± 15%  interrupts.CPU34.CAL:Function_call_interrupts
      1292 ± 14%     -26.8%     946.50 ± 14%  interrupts.CPU35.CAL:Function_call_interrupts
      1295 ± 14%     -27.2%     943.25 ± 14%  interrupts.CPU36.CAL:Function_call_interrupts
      1293 ± 14%     -26.9%     945.50 ± 14%  interrupts.CPU37.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     947.25 ± 14%  interrupts.CPU38.CAL:Function_call_interrupts
      1297 ± 14%     -27.0%     947.00 ± 14%  interrupts.CPU39.CAL:Function_call_interrupts
      1296 ± 14%     -26.9%     947.25 ± 14%  interrupts.CPU4.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     947.50 ± 14%  interrupts.CPU40.CAL:Function_call_interrupts
      1293 ± 14%     -26.8%     947.00 ± 14%  interrupts.CPU41.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     947.00 ± 14%  interrupts.CPU42.CAL:Function_call_interrupts
      1294 ± 14%     -26.9%     945.50 ± 14%  interrupts.CPU43.CAL:Function_call_interrupts
      1294 ± 14%     -27.1%     943.00 ± 14%  interrupts.CPU44.CAL:Function_call_interrupts
      1294 ± 14%     -27.0%     945.00 ± 14%  interrupts.CPU45.CAL:Function_call_interrupts
      1295 ± 14%     -26.8%     947.50 ± 14%  interrupts.CPU46.CAL:Function_call_interrupts
      1298 ± 14%     -27.4%     943.25 ± 14%  interrupts.CPU47.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     947.25 ± 14%  interrupts.CPU48.CAL:Function_call_interrupts
      1299 ± 14%     -27.1%     947.25 ± 14%  interrupts.CPU49.CAL:Function_call_interrupts
      1296 ± 14%     -27.0%     947.00 ± 14%  interrupts.CPU5.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     946.25 ± 14%  interrupts.CPU50.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     946.50 ± 14%  interrupts.CPU51.CAL:Function_call_interrupts
      1296 ± 14%     -27.1%     945.75 ± 14%  interrupts.CPU52.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     946.50 ± 14%  interrupts.CPU53.CAL:Function_call_interrupts
      1296 ± 14%     -27.0%     946.25 ± 14%  interrupts.CPU54.CAL:Function_call_interrupts
      1298 ± 14%     -27.2%     945.25 ± 15%  interrupts.CPU55.CAL:Function_call_interrupts
      1295 ± 14%     -27.0%     946.25 ± 14%  interrupts.CPU56.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     947.00 ± 14%  interrupts.CPU57.CAL:Function_call_interrupts
      1299 ± 14%     -27.1%     947.00 ± 14%  interrupts.CPU58.CAL:Function_call_interrupts
      1298 ± 14%     -27.3%     944.00 ± 14%  interrupts.CPU59.CAL:Function_call_interrupts
      1291 ± 14%     -26.6%     947.50 ± 14%  interrupts.CPU6.CAL:Function_call_interrupts
      1293 ± 14%     -26.8%     946.00 ± 14%  interrupts.CPU60.CAL:Function_call_interrupts
      1297 ± 14%     -26.9%     948.00 ± 14%  interrupts.CPU61.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     946.25 ± 14%  interrupts.CPU62.CAL:Function_call_interrupts
      1296 ± 14%     -26.9%     947.50 ± 14%  interrupts.CPU63.CAL:Function_call_interrupts
      1297 ± 14%     -27.2%     945.25 ± 14%  interrupts.CPU64.CAL:Function_call_interrupts
      1298 ± 14%     -27.1%     947.00 ± 14%  interrupts.CPU65.CAL:Function_call_interrupts
      1299 ± 14%     -27.2%     946.25 ± 14%  interrupts.CPU66.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     947.25 ± 14%  interrupts.CPU67.CAL:Function_call_interrupts
      1299 ± 14%     -27.1%     946.75 ± 14%  interrupts.CPU68.CAL:Function_call_interrupts
      1299 ± 14%     -27.1%     947.00 ± 14%  interrupts.CPU69.CAL:Function_call_interrupts
      1293 ± 14%     -26.7%     947.50 ± 14%  interrupts.CPU7.CAL:Function_call_interrupts
      1297 ± 14%     -27.0%     946.75 ± 14%  interrupts.CPU70.CAL:Function_call_interrupts
      1297 ± 14%     -27.1%     945.75 ± 14%  interrupts.CPU71.CAL:Function_call_interrupts
      1292 ± 14%     -26.8%     946.50 ± 14%  interrupts.CPU72.CAL:Function_call_interrupts
      1298 ± 14%     -27.4%     942.50 ± 14%  interrupts.CPU73.CAL:Function_call_interrupts
      1298 ± 14%     -27.5%     942.00 ± 14%  interrupts.CPU74.CAL:Function_call_interrupts
      1299 ± 14%     -27.2%     946.50 ± 14%  interrupts.CPU75.CAL:Function_call_interrupts
      1298 ± 14%     -27.4%     942.75 ± 14%  interrupts.CPU76.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     946.75 ± 14%  interrupts.CPU77.CAL:Function_call_interrupts
      1293 ± 14%     -27.0%     944.25 ± 14%  interrupts.CPU78.CAL:Function_call_interrupts
      1296 ± 14%     -27.0%     946.75 ± 14%  interrupts.CPU79.CAL:Function_call_interrupts
      1298 ± 14%     -27.0%     947.25 ± 14%  interrupts.CPU8.CAL:Function_call_interrupts
      1297 ± 14%     -27.0%     947.00 ± 14%  interrupts.CPU80.CAL:Function_call_interrupts
      1297 ± 14%     -27.0%     947.25 ± 14%  interrupts.CPU81.CAL:Function_call_interrupts
      1291 ± 14%     -26.6%     947.25 ± 14%  interrupts.CPU82.CAL:Function_call_interrupts
      6958 ± 33%     -58.9%       2860        interrupts.CPU82.NMI:Non-maskable_interrupts
      6958 ± 33%     -58.9%       2860        interrupts.CPU82.PMI:Performance_monitoring_interrupts
      1291 ± 14%     -27.0%     943.00 ± 14%  interrupts.CPU83.CAL:Function_call_interrupts
      1293 ± 14%     -26.8%     947.25 ± 14%  interrupts.CPU84.CAL:Function_call_interrupts
      1293 ± 14%     -26.8%     946.75 ± 14%  interrupts.CPU85.CAL:Function_call_interrupts
      1295 ± 14%     -26.9%     947.00 ± 14%  interrupts.CPU86.CAL:Function_call_interrupts
      1296 ± 14%     -27.2%     943.50 ± 14%  interrupts.CPU87.CAL:Function_call_interrupts
      1292 ± 14%     -27.0%     943.50 ± 14%  interrupts.CPU88.CAL:Function_call_interrupts
      1294 ± 14%     -26.8%     947.25 ± 14%  interrupts.CPU89.CAL:Function_call_interrupts
      1291 ± 14%     -26.7%     946.50 ± 14%  interrupts.CPU9.CAL:Function_call_interrupts
      1296 ± 14%     -27.1%     945.50 ± 14%  interrupts.CPU90.CAL:Function_call_interrupts
      1286 ± 13%     -26.5%     945.75 ± 14%  interrupts.CPU91.CAL:Function_call_interrupts
      1293 ± 14%     -26.9%     945.50 ± 14%  interrupts.CPU92.CAL:Function_call_interrupts
      1287 ± 13%     -26.6%     945.00 ± 14%  interrupts.CPU93.CAL:Function_call_interrupts
      1293 ± 14%     -26.9%     946.25 ± 14%  interrupts.CPU94.CAL:Function_call_interrupts
      1293 ± 14%     -27.2%     941.75 ± 14%  interrupts.CPU95.CAL:Function_call_interrupts
      1294 ± 14%     -27.0%     945.25 ± 14%  interrupts.CPU96.CAL:Function_call_interrupts
      1291 ± 14%     -26.8%     944.75 ± 14%  interrupts.CPU97.CAL:Function_call_interrupts
      1296 ± 14%     -27.1%     945.00 ± 14%  interrupts.CPU98.CAL:Function_call_interrupts
      5502 ± 48%     +39.1%       7651 ± 24%  interrupts.CPU98.NMI:Non-maskable_interrupts
      5502 ± 48%     +39.1%       7651 ± 24%  interrupts.CPU98.PMI:Performance_monitoring_interrupts
      1297 ± 14%     -27.1%     945.50 ± 14%  interrupts.CPU99.CAL:Function_call_interrupts



***************************************************************************************************
lkp-csl-2ap4: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-csl-2ap4/futex4/will-it-scale/0x500002c

commit: 
  v5.6-rc4
  8019ad13ef ("futex: Fix inode life-time issue")

        v5.6-rc4 8019ad13ef7f64be44d4f892af9 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :5           60%           3:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
   4449621           -99.7%      11551 ±  2%  will-it-scale.per_process_ops
 8.543e+08           -99.7%    2217968 ±  2%  will-it-scale.workload
    135523 ±  2%     +24.3%     168452 ±  6%  meminfo.Shmem
     35.14 ±  3%      +5.0%      36.91 ±  3%  boot-time.boot
      5869 ±  3%      +5.6%       6200 ±  4%  boot-time.idle
     79.12           +20.4       99.49        mpstat.cpu.all.sys%
     20.36           -20.3        0.09 ±  9%  mpstat.cpu.all.usr%
    376687           +11.2%     418980 ±  4%  numa-meminfo.node3.FilePages
    115214 ±  4%     +26.9%     146170 ±  8%  numa-meminfo.node3.Shmem
     94222           +11.2%     104763 ±  4%  numa-vmstat.node3.nr_file_pages
     28854 ±  4%     +26.7%      36561 ±  8%  numa-vmstat.node3.nr_shmem
      1004 ±  8%     +19.2%       1196 ± 11%  slabinfo.skbuff_ext_cache.active_objs
      1004 ±  8%     +19.2%       1196 ± 11%  slabinfo.skbuff_ext_cache.num_objs
     78.00           +26.9%      99.00        vmstat.cpu.sy
     19.80 ±  2%    -100.0%       0.00        vmstat.cpu.us
      2339            -6.0%       2200 ±  2%  vmstat.system.cs
    125496            +8.4%     136025 ±  2%  proc-vmstat.nr_active_anon
     99409            +2.5%     101907        proc-vmstat.nr_anon_pages
    303648            +2.6%     311648        proc-vmstat.nr_file_pages
      7070 ±  2%      +8.4%       7662 ±  2%  proc-vmstat.nr_inactive_anon
      8690            +1.3%       8802        proc-vmstat.nr_page_table_pages
     33874 ±  2%     +24.3%      42116 ±  6%  proc-vmstat.nr_shmem
     25155            +2.3%      25729        proc-vmstat.nr_slab_reclaimable
    125496            +8.4%     136025 ±  2%  proc-vmstat.nr_zone_active_anon
      7070 ±  2%      +8.4%       7662 ±  2%  proc-vmstat.nr_zone_inactive_anon
     40572 ±  2%     +30.2%      52830 ±  7%  proc-vmstat.pgactivate
     19.13 ± 23%     -27.4%      13.89 ± 15%  sched_debug.cfs_rq:/.load_avg.stddev
     24.04 ±  9%     -17.3%      19.89 ±  4%  sched_debug.cpu.clock.stddev
     24.04 ±  9%     -17.3%      19.89 ±  4%  sched_debug.cpu.clock_task.stddev
      0.00 ±  4%     +53.4%       0.00 ± 16%  sched_debug.cpu.next_balance.stddev
     48116 ±  9%     +36.4%      65645 ± 13%  sched_debug.cpu.nr_switches.max
      4983 ±  5%     +24.0%       6181 ±  6%  sched_debug.cpu.nr_switches.stddev
      1865           +55.4%       2897        sched_debug.cpu.sched_count.avg
     44924 ± 10%     +39.1%      62510 ± 13%  sched_debug.cpu.sched_count.max
    644.60 ±  4%    +160.9%       1682        sched_debug.cpu.sched_count.min
      4704 ±  5%     +24.7%       5868 ±  7%  sched_debug.cpu.sched_count.stddev
     20958 ±  5%     +42.6%      29886 ± 13%  sched_debug.cpu.ttwu_count.max
    240.90 ±  4%     -22.6%     186.42        sched_debug.cpu.ttwu_count.min
      2283 ±  4%     +27.9%       2919 ±  7%  sched_debug.cpu.ttwu_count.stddev
     18360 ±  8%     +61.5%      29654 ± 14%  sched_debug.cpu.ttwu_local.max
    236.17 ±  4%     -22.9%     182.12        sched_debug.cpu.ttwu_local.min
      1932 ±  6%     +47.2%       2845 ±  7%  sched_debug.cpu.ttwu_local.stddev
     23.00           -23.0        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.syscall
      8.88            -8.9        0.00        perf-profile.calltrace.cycles-pp.get_futex_value_locked.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
      7.82            -7.8        0.00        perf-profile.calltrace.cycles-pp.get_futex_key.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
      5.48            -5.5        0.00        perf-profile.calltrace.cycles-pp.hash_futex.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
     99.85            +0.1       99.96        perf-profile.calltrace.cycles-pp.syscall
      0.00            +0.6        0.61 ± 13%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
      0.00            +0.7        0.66 ± 13%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.futex_wait_setup
      0.00            +0.7        0.70 ± 11%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.futex_wait_setup.futex_wait
     68.57           +31.3       99.90        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.syscall
     64.65           +35.2       99.89        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     59.06           +40.8       99.86        perf-profile.calltrace.cycles-pp.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     54.68           +45.2       99.85        perf-profile.calltrace.cycles-pp.do_futex.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     50.54           +49.3       99.84        perf-profile.calltrace.cycles-pp.futex_wait.do_futex.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe
     38.82           +61.0       99.82        perf-profile.calltrace.cycles-pp.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex.do_syscall_64
      4.65           +94.7       99.32        perf-profile.calltrace.cycles-pp._raw_spin_lock.futex_wait_setup.futex_wait.do_futex.__x64_sys_futex
      0.00           +99.3       99.26        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.futex_wait_setup.futex_wait.do_futex
     15.43           -15.4        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64
     11.72           -11.7        0.00        perf-profile.children.cycles-pp.syscall_return_via_sysret
      9.09            -9.1        0.00        perf-profile.children.cycles-pp.get_futex_value_locked
      8.08            -8.1        0.00        perf-profile.children.cycles-pp.get_futex_key
      5.53            -5.5        0.00        perf-profile.children.cycles-pp.hash_futex
      1.11 ±  4%      -0.4        0.70 ± 12%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.07 ±  4%      -0.4        0.66 ± 12%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.85 ±  5%      -0.2        0.62 ± 14%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.49 ±  5%      -0.1        0.36 ± 11%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.40 ±  6%      -0.1        0.30 ± 13%  perf-profile.children.cycles-pp.tick_sched_timer
      0.14 ±  9%      -0.1        0.08 ± 26%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.24 ±  7%      -0.1        0.18 ±  6%  perf-profile.children.cycles-pp.tick_sched_handle
      0.24 ±  7%      -0.1        0.18 ±  6%  perf-profile.children.cycles-pp.update_process_times
      0.20 ±  3%      -0.0        0.15 ± 14%  perf-profile.children.cycles-pp.clockevents_program_event
      0.14 ±  3%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.scheduler_tick
     99.96            +0.0       99.97        perf-profile.children.cycles-pp.syscall
     68.72           +31.2       99.92        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     64.96           +34.9       99.91        perf-profile.children.cycles-pp.do_syscall_64
     59.56           +40.3       99.86        perf-profile.children.cycles-pp.__x64_sys_futex
     55.11           +44.7       99.85        perf-profile.children.cycles-pp.do_futex
     50.74           +49.1       99.84        perf-profile.children.cycles-pp.futex_wait
     39.54           +60.3       99.82        perf-profile.children.cycles-pp.futex_wait_setup
      4.81           +94.5       99.33        perf-profile.children.cycles-pp._raw_spin_lock
      0.11 ± 15%     +99.2       99.31        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     12.10           -11.7        0.45 ±  3%  perf-profile.self.cycles-pp.futex_wait_setup
     11.64           -11.6        0.00        perf-profile.self.cycles-pp.syscall_return_via_sysret
     11.48           -11.5        0.00        perf-profile.self.cycles-pp.syscall
     10.84           -10.8        0.00        perf-profile.self.cycles-pp.futex_wait
      8.99            -9.0        0.00        perf-profile.self.cycles-pp.get_futex_value_locked
      7.78            -7.8        0.00        perf-profile.self.cycles-pp.entry_SYSCALL_64
      5.22            -5.2        0.00        perf-profile.self.cycles-pp.hash_futex
      0.14 ± 11%      -0.1        0.08 ± 29%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.11 ± 15%     +98.5       98.62        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.07 ± 24%   +1501.3%       1.10        perf-stat.i.MPKI
 7.602e+10           -84.5%  1.175e+10        perf-stat.i.branch-instructions
      0.30            -0.2        0.13 ±  3%  perf-stat.i.branch-miss-rate%
 2.194e+08           -93.9%   13290093 ±  3%  perf-stat.i.branch-misses
      8.54 ±  4%     +38.8       47.37        perf-stat.i.cache-miss-rate%
   1506998 ±  2%   +1518.6%   24392848        perf-stat.i.cache-misses
  17846478 ±  3%    +192.2%   52140540        perf-stat.i.cache-references
      2303            -6.3%       2158 ±  2%  perf-stat.i.context-switches
      1.17          +953.9%      12.30        perf-stat.i.cpi
 5.704e+11            +3.3%   5.89e+11        perf-stat.i.cpu-cycles
    221.78           -14.4%     189.85        perf-stat.i.cpu-migrations
    403166 ±  3%     -94.0%      24108        perf-stat.i.cycles-between-cache-misses
      0.00 ± 43%      +0.0        0.00 ±  6%  perf-stat.i.dTLB-load-miss-rate%
 1.288e+11           -90.7%  1.196e+10        perf-stat.i.dTLB-loads
      0.00 ± 44%      +0.0        0.00 ±  4%  perf-stat.i.dTLB-store-miss-rate%
     15307 ±  9%     +16.6%      17843 ±  4%  perf-stat.i.dTLB-store-misses
 9.206e+10           -99.5%  4.323e+08        perf-stat.i.dTLB-stores
     87.34            +9.4       96.71        perf-stat.i.iTLB-load-miss-rate%
 2.158e+08           -97.6%    5097456 ±  4%  perf-stat.i.iTLB-load-misses
  31066533 ±  9%     -99.4%     182948 ±  9%  perf-stat.i.iTLB-loads
 4.899e+11           -90.2%  4.777e+10        perf-stat.i.instructions
      2270          +313.9%       9398 ±  4%  perf-stat.i.instructions-per-iTLB-miss
      0.86           -90.3%       0.08        perf-stat.i.ipc
     94.32            +5.3       99.61        perf-stat.i.node-load-miss-rate%
    245628         +2423.6%    6198703 ±  3%  perf-stat.i.node-load-misses
     98.49            +1.4       99.86        perf-stat.i.node-store-miss-rate%
    132981 ±  2%   +4410.0%    5997422 ±  2%  perf-stat.i.node-store-misses
      0.04 ±  3%   +2872.5%       1.09        perf-stat.overall.MPKI
      0.29            -0.2        0.11 ±  3%  perf-stat.overall.branch-miss-rate%
      8.44 ±  4%     +38.3       46.74        perf-stat.overall.cache-miss-rate%
      1.16          +959.1%      12.33        perf-stat.overall.cpi
    375939 ±  2%     -93.6%      24144        perf-stat.overall.cycles-between-cache-misses
      0.00 ±  7%      +0.0        0.00 ± 12%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  5%      +0.0        0.00 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
     87.40            +9.1       96.53        perf-stat.overall.iTLB-load-miss-rate%
      2269          +313.6%       9388 ±  4%  perf-stat.overall.instructions-per-iTLB-miss
      0.86           -90.6%       0.08        perf-stat.overall.ipc
     88.60           +11.0       99.57        perf-stat.overall.node-load-miss-rate%
     96.31            +3.6       99.92        perf-stat.overall.node-store-miss-rate%
    172790         +3653.6%    6485913 ±  2%  perf-stat.overall.path-length
 7.576e+10           -84.5%  1.171e+10        perf-stat.ps.branch-instructions
 2.187e+08           -94.0%   13209615 ±  3%  perf-stat.ps.branch-misses
   1513342 ±  2%   +1507.0%   24320096        perf-stat.ps.cache-misses
  17954987 ±  3%    +189.8%   52036255        perf-stat.ps.cache-references
      2279            -6.2%       2138 ±  2%  perf-stat.ps.context-switches
 5.685e+11            +3.3%   5.87e+11        perf-stat.ps.cpu-cycles
    217.89           -14.2%     187.03        perf-stat.ps.cpu-migrations
 1.283e+11           -90.7%  1.192e+10        perf-stat.ps.dTLB-loads
     16095 ±  5%     +14.4%      18412 ±  4%  perf-stat.ps.dTLB-store-misses
 9.175e+10           -99.5%  4.307e+08        perf-stat.ps.dTLB-stores
 2.151e+08           -97.6%    5080544 ±  4%  perf-stat.ps.iTLB-load-misses
  31045585 ±  9%     -99.4%     181538 ±  9%  perf-stat.ps.iTLB-loads
 4.882e+11           -90.2%   4.76e+10        perf-stat.ps.instructions
    245445         +2416.9%    6177717 ±  3%  perf-stat.ps.node-load-misses
     31588 ± 10%     -16.1%      26491 ±  3%  perf-stat.ps.node-loads
    132763 ±  2%   +4402.6%    5977862 ±  2%  perf-stat.ps.node-store-misses
 1.476e+14           -90.3%  1.438e+13        perf-stat.total.instructions
    340.00 ±  8%     -10.8%     303.25 ±  4%  interrupts.CPU11.RES:Rescheduling_interrupts
    500.20 ± 77%     -44.7%     276.50 ±  2%  interrupts.CPU116.RES:Rescheduling_interrupts
    792.00 ±122%     -66.0%     269.25 ±  3%  interrupts.CPU119.RES:Rescheduling_interrupts
    306.00           -12.7%     267.25 ±  4%  interrupts.CPU120.RES:Rescheduling_interrupts
    309.20 ±  2%     -18.4%     252.25 ±  7%  interrupts.CPU121.RES:Rescheduling_interrupts
    309.80 ±  3%     -21.2%     244.25 ± 11%  interrupts.CPU122.RES:Rescheduling_interrupts
    305.20           -24.6%     230.25 ± 13%  interrupts.CPU123.RES:Rescheduling_interrupts
    302.60           -21.3%     238.25 ± 18%  interrupts.CPU124.RES:Rescheduling_interrupts
    307.80           -30.8%     213.00 ± 20%  interrupts.CPU125.RES:Rescheduling_interrupts
    303.80           -37.0%     191.50 ± 25%  interrupts.CPU126.RES:Rescheduling_interrupts
    488.00 ± 75%     -55.5%     217.00 ± 32%  interrupts.CPU127.RES:Rescheduling_interrupts
    303.80           -49.7%     152.75 ± 33%  interrupts.CPU128.RES:Rescheduling_interrupts
    303.20           -58.5%     125.75 ± 37%  interrupts.CPU129.RES:Rescheduling_interrupts
    348.40 ± 16%     -18.0%     285.75        interrupts.CPU13.RES:Rescheduling_interrupts
    302.60           -65.2%     105.25 ± 42%  interrupts.CPU130.RES:Rescheduling_interrupts
    303.20           -74.4%      77.50 ± 53%  interrupts.CPU132.RES:Rescheduling_interrupts
    302.40           -74.9%      76.00 ± 50%  interrupts.CPU133.RES:Rescheduling_interrupts
      1682 ±163%     -97.1%      49.50 ± 58%  interrupts.CPU134.RES:Rescheduling_interrupts
    302.00           -90.8%      27.75 ± 62%  interrupts.CPU137.RES:Rescheduling_interrupts
    305.80           -93.0%      21.50 ± 62%  interrupts.CPU138.RES:Rescheduling_interrupts
    302.00           -94.0%      18.00 ± 51%  interrupts.CPU139.RES:Rescheduling_interrupts
    378.60 ± 40%     -91.9%      30.50 ± 49%  interrupts.CPU140.RES:Rescheduling_interrupts
    307.40           -95.3%      14.50 ± 61%  interrupts.CPU141.RES:Rescheduling_interrupts
    606.80 ± 99%     -98.6%       8.75 ± 16%  interrupts.CPU143.RES:Rescheduling_interrupts
    329.40 ± 15%     -97.9%       7.00 ± 34%  interrupts.CPU144.RES:Rescheduling_interrupts
    489.80 ± 76%     -94.9%      25.00 ±113%  interrupts.CPU145.RES:Rescheduling_interrupts
    764.80 ±114%     -90.7%      70.75 ±104%  interrupts.CPU146.RES:Rescheduling_interrupts
      1669 ±143%     -95.9%      68.00 ± 98%  interrupts.CPU147.RES:Rescheduling_interrupts
      6027 ± 35%     +26.8%       7640 ± 24%  interrupts.CPU148.NMI:Non-maskable_interrupts
      6027 ± 35%     +26.8%       7640 ± 24%  interrupts.CPU148.PMI:Performance_monitoring_interrupts
    323.20 ± 14%     -98.0%       6.50 ± 23%  interrupts.CPU148.RES:Rescheduling_interrupts
    301.00           -94.8%      15.75 ± 92%  interrupts.CPU149.RES:Rescheduling_interrupts
    435.20 ± 59%     -84.4%      68.00 ±128%  interrupts.CPU151.RES:Rescheduling_interrupts
    319.20 ±  6%     -97.0%       9.50 ± 46%  interrupts.CPU152.RES:Rescheduling_interrupts
    307.40 ±  3%     -97.7%       7.00 ± 22%  interrupts.CPU153.RES:Rescheduling_interrupts
    506.40 ± 70%     -98.4%       8.00 ±  8%  interrupts.CPU156.RES:Rescheduling_interrupts
      1265 ± 68%     -97.6%      30.50 ± 84%  interrupts.CPU157.RES:Rescheduling_interrupts
    309.80 ±  4%     -95.2%      15.00 ± 85%  interrupts.CPU158.RES:Rescheduling_interrupts
    399.20 ± 45%     -98.0%       8.00 ± 27%  interrupts.CPU159.RES:Rescheduling_interrupts
    415.20 ± 26%     -93.3%      27.75 ± 71%  interrupts.CPU160.RES:Rescheduling_interrupts
    309.00 ±  3%     -96.3%      11.50 ± 53%  interrupts.CPU162.RES:Rescheduling_interrupts
    320.20 ±  6%     -97.2%       9.00 ± 50%  interrupts.CPU163.RES:Rescheduling_interrupts
    308.80 ±  3%     -91.8%      25.25 ±138%  interrupts.CPU164.RES:Rescheduling_interrupts
    328.60 ± 15%     -97.9%       7.00 ± 17%  interrupts.CPU165.RES:Rescheduling_interrupts
    467.80 ± 54%     -95.4%      21.75 ±115%  interrupts.CPU166.RES:Rescheduling_interrupts
    355.20 ± 23%     -97.7%       8.00 ± 17%  interrupts.CPU167.RES:Rescheduling_interrupts
    365.40 ± 34%     -98.2%       6.75 ± 12%  interrupts.CPU169.RES:Rescheduling_interrupts
    319.40 ±  9%     -97.0%       9.50 ± 52%  interrupts.CPU170.RES:Rescheduling_interrupts
    342.40 ± 21%     -94.5%      19.00 ±106%  interrupts.CPU171.RES:Rescheduling_interrupts
    303.40           -84.8%      46.25 ±142%  interrupts.CPU172.RES:Rescheduling_interrupts
    304.20           -89.4%      32.25 ± 77%  interrupts.CPU173.RES:Rescheduling_interrupts
    316.60 ±  7%     -97.2%       9.00 ± 36%  interrupts.CPU175.RES:Rescheduling_interrupts
    487.80 ± 75%     -98.2%       8.75 ± 24%  interrupts.CPU176.RES:Rescheduling_interrupts
    615.00 ± 99%     -96.8%      19.75 ±106%  interrupts.CPU177.RES:Rescheduling_interrupts
    309.00 ±  3%     -97.3%       8.25 ± 15%  interrupts.CPU178.RES:Rescheduling_interrupts
    317.00 ±  9%     -91.5%      27.00 ±121%  interrupts.CPU180.RES:Rescheduling_interrupts
    407.60 ± 40%     -82.5%      71.50 ±102%  interrupts.CPU182.RES:Rescheduling_interrupts
    306.00 ±  3%     -73.5%      81.00 ± 94%  interrupts.CPU183.RES:Rescheduling_interrupts
    323.40 ±  8%     -81.8%      59.00 ±147%  interrupts.CPU185.RES:Rescheduling_interrupts
    341.20 ± 14%     -98.0%       6.75 ± 12%  interrupts.CPU186.RES:Rescheduling_interrupts
    303.40           -89.6%      31.50 ±136%  interrupts.CPU187.RES:Rescheduling_interrupts
    691.20 ±112%     -98.2%      12.75 ± 74%  interrupts.CPU188.RES:Rescheduling_interrupts
      1793 ±166%     -98.7%      22.75 ± 95%  interrupts.CPU189.RES:Rescheduling_interrupts
    309.20 ±  3%     -97.7%       7.25 ±  5%  interrupts.CPU190.RES:Rescheduling_interrupts
    679.00 ± 75%     -75.7%     165.00 ± 52%  interrupts.CPU191.RES:Rescheduling_interrupts
    196.40 ±194%     -98.2%       3.50 ± 31%  interrupts.CPU191.TLB:TLB_shootdowns
      5949 ± 35%     +28.4%       7640 ± 24%  interrupts.CPU28.NMI:Non-maskable_interrupts
      5949 ± 35%     +28.4%       7640 ± 24%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
      5948 ± 35%     +28.4%       7640 ± 24%  interrupts.CPU29.NMI:Non-maskable_interrupts
      5948 ± 35%     +28.4%       7640 ± 24%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      1236 ± 90%     -76.4%     292.50 ±  4%  interrupts.CPU61.RES:Rescheduling_interrupts
    803.00 ±118%     -64.6%     284.25        interrupts.CPU97.RES:Rescheduling_interrupts
    161.20 ±  6%    +112.2%     342.00 ±  7%  interrupts.IWI:IRQ_work_interrupts
     90311           -45.6%      49127 ±  2%  interrupts.RES:Rescheduling_interrupts
    123131 ±  3%     -49.4%      62277 ±  3%  softirqs.CPU0.RCU
    125079 ±  5%     -81.9%      22602 ±  5%  softirqs.CPU1.RCU
    122475 ±  3%     -77.4%      27684 ± 19%  softirqs.CPU10.RCU
    123527 ±  4%     -77.6%      27661 ± 38%  softirqs.CPU100.RCU
    121892 ±  3%     -78.6%      26097 ± 26%  softirqs.CPU101.RCU
    122471 ±  3%     -80.7%      23655 ± 13%  softirqs.CPU102.RCU
    121184 ±  3%     -69.9%      36456 ± 69%  softirqs.CPU103.RCU
    121548 ±  4%     -82.3%      21489 ±  3%  softirqs.CPU104.RCU
    121300 ±  3%     -82.3%      21516 ±  5%  softirqs.CPU105.RCU
    121433 ±  3%     -78.2%      26468 ± 32%  softirqs.CPU106.RCU
    121283 ±  3%     -82.2%      21609 ±  2%  softirqs.CPU107.RCU
    121406 ±  3%     -82.6%      21107 ±  2%  softirqs.CPU108.RCU
    121424 ±  3%     -82.3%      21510 ±  6%  softirqs.CPU109.RCU
    122018 ±  3%     -80.4%      23907 ±  8%  softirqs.CPU11.RCU
    121771 ±  4%     -82.2%      21621 ±  3%  softirqs.CPU110.RCU
    122368 ±  2%     -82.7%      21181 ±  2%  softirqs.CPU111.RCU
    121481 ±  3%     -82.3%      21498 ±  2%  softirqs.CPU112.RCU
    122125 ±  3%     -80.5%      23856 ± 16%  softirqs.CPU113.RCU
    121734 ±  4%     -82.0%      21858 ±  6%  softirqs.CPU114.RCU
    121186 ±  3%     -75.8%      29356 ± 49%  softirqs.CPU115.RCU
    121854 ±  3%     -82.9%      20847 ±  2%  softirqs.CPU116.RCU
    121615 ±  3%     -80.7%      23490 ± 13%  softirqs.CPU117.RCU
    122264 ±  4%     -82.8%      21063 ±  5%  softirqs.CPU118.RCU
    121476 ±  3%     -82.3%      21481 ±  3%  softirqs.CPU119.RCU
    122098 ±  3%     -79.1%      25518 ± 22%  softirqs.CPU12.RCU
    121098 ±  3%     -81.8%      22045 ±  4%  softirqs.CPU120.RCU
    121343 ±  3%     -82.7%      20985 ±  3%  softirqs.CPU121.RCU
    121099 ±  3%     -82.5%      21194 ±  3%  softirqs.CPU122.RCU
    121063 ±  3%     -82.9%      20743 ±  3%  softirqs.CPU123.RCU
    122028 ±  3%     -80.9%      23297 ± 12%  softirqs.CPU124.RCU
    121270 ±  4%     -83.0%      20662 ±  7%  softirqs.CPU125.RCU
    120914 ±  3%     -82.1%      21597 ±  4%  softirqs.CPU126.RCU
    120970 ±  3%     -80.9%      23094 ± 21%  softirqs.CPU127.RCU
    121118 ±  3%     -80.9%      23134 ± 17%  softirqs.CPU128.RCU
    121530 ±  4%     -80.3%      23883 ± 19%  softirqs.CPU129.RCU
    122083 ±  3%     -81.2%      22964 ±  5%  softirqs.CPU13.RCU
    121061 ±  3%     -82.7%      20962 ±  4%  softirqs.CPU130.RCU
    120709 ±  3%     -74.9%      30270 ± 42%  softirqs.CPU131.RCU
    121808 ±  3%     -83.2%      20478 ±  3%  softirqs.CPU132.RCU
    122099 ±  3%     -82.7%      21104 ±  4%  softirqs.CPU133.RCU
    121082 ±  3%     -82.6%      21101 ±  4%  softirqs.CPU134.RCU
    121698 ±  2%     -81.1%      22966 ± 12%  softirqs.CPU135.RCU
    121399 ±  3%     -77.7%      27040 ± 22%  softirqs.CPU136.RCU
    121124 ±  4%     -82.5%      21160 ±  2%  softirqs.CPU137.RCU
    121290 ±  3%     -82.3%      21514 ±  3%  softirqs.CPU138.RCU
    120682 ±  3%     -82.7%      20840        softirqs.CPU139.RCU
    121761 ±  3%     -80.4%      23841 ± 13%  softirqs.CPU14.RCU
    122085 ±  3%     -81.3%      22863 ± 13%  softirqs.CPU140.RCU
    121223 ±  3%     -80.5%      23681 ± 11%  softirqs.CPU141.RCU
    120729 ±  3%     -81.3%      22522 ± 13%  softirqs.CPU142.RCU
    121370 ±  3%     -82.2%      21619 ±  3%  softirqs.CPU143.RCU
    120990 ±  3%     -81.9%      21958 ±  7%  softirqs.CPU144.RCU
     95231           +12.6%     107199 ±  4%  softirqs.CPU144.TIMER
    121571 ±  3%     -81.6%      22369 ±  7%  softirqs.CPU145.RCU
     94953           +12.5%     106814 ±  4%  softirqs.CPU145.TIMER
    122342 ±  4%     -81.1%      23170 ±  3%  softirqs.CPU146.RCU
     95703           +12.9%     108056 ±  2%  softirqs.CPU146.TIMER
    121657 ±  3%     -81.4%      22590 ±  6%  softirqs.CPU147.RCU
     95695           +11.7%     106918 ±  3%  softirqs.CPU147.TIMER
    121974 ±  3%     -83.1%      20648        softirqs.CPU148.RCU
     95351           +10.5%     105384 ±  3%  softirqs.CPU148.TIMER
    121874 ±  3%     -81.9%      22015 ±  2%  softirqs.CPU149.RCU
     94424           +12.5%     106203 ±  3%  softirqs.CPU149.TIMER
    122732 ±  4%     -80.7%      23727 ± 15%  softirqs.CPU15.RCU
    121514 ±  4%     -78.6%      25958 ± 29%  softirqs.CPU150.RCU
     95172           +11.3%     105935 ±  3%  softirqs.CPU150.TIMER
    121503 ±  3%     -82.2%      21679 ±  3%  softirqs.CPU151.RCU
     94827           +11.0%     105253 ±  3%  softirqs.CPU151.TIMER
    120688 ±  3%     -81.4%      22437 ±  8%  softirqs.CPU152.RCU
     94780           +26.3%     119697 ± 19%  softirqs.CPU152.TIMER
    120955 ±  3%     -82.8%      20807        softirqs.CPU153.RCU
    121227 ±  4%     -75.5%      29701 ± 52%  softirqs.CPU154.RCU
     94477           +12.4%     106161 ±  4%  softirqs.CPU154.TIMER
    121222 ±  3%     -74.4%      31088 ± 58%  softirqs.CPU155.RCU
     94264           +12.5%     106023 ±  4%  softirqs.CPU155.TIMER
    121673 ±  4%     -82.1%      21804 ±  9%  softirqs.CPU156.RCU
     95436           +11.4%     106320 ±  4%  softirqs.CPU156.TIMER
    122638 ±  2%     -80.6%      23746 ± 12%  softirqs.CPU157.RCU
     95181           +12.4%     106985 ±  5%  softirqs.CPU157.TIMER
    121113 ±  4%     -81.4%      22503 ± 12%  softirqs.CPU158.RCU
     95392           +12.3%     107142 ±  5%  softirqs.CPU158.TIMER
    121195 ±  3%     -82.4%      21313 ±  7%  softirqs.CPU159.RCU
     94542           +12.8%     106673 ±  4%  softirqs.CPU159.TIMER
    121970 ±  4%     -82.0%      21902 ±  4%  softirqs.CPU16.RCU
    120856 ±  4%     -80.5%      23604 ± 10%  softirqs.CPU160.RCU
     94607           +11.5%     105465 ±  4%  softirqs.CPU160.TIMER
    122697 ±  2%     -76.4%      28950 ± 42%  softirqs.CPU161.RCU
     95338 ±  2%     +24.4%     118602 ± 20%  softirqs.CPU161.TIMER
    121008 ±  3%     -81.1%      22844 ±  7%  softirqs.CPU162.RCU
     95153 ±  2%     +11.5%     106136 ±  4%  softirqs.CPU162.TIMER
    120809 ±  3%     -82.2%      21533 ±  7%  softirqs.CPU163.RCU
     94804           +11.1%     105335 ±  4%  softirqs.CPU163.TIMER
    121109 ±  4%     -81.3%      22631 ±  8%  softirqs.CPU164.RCU
     94068 ±  2%     +12.7%     105982 ±  4%  softirqs.CPU164.TIMER
    121739 ±  3%     -82.2%      21661        softirqs.CPU165.RCU
     94265           +12.2%     105745 ±  4%  softirqs.CPU165.TIMER
    120557 ±  3%     -80.3%      23712 ± 11%  softirqs.CPU166.RCU
     94101           +12.1%     105500 ±  4%  softirqs.CPU166.TIMER
    120949 ±  3%     -82.2%      21560 ±  3%  softirqs.CPU167.RCU
    120800 ±  3%     -73.9%      31526 ± 35%  softirqs.CPU168.RCU
    120954 ±  3%     -82.2%      21510 ±  4%  softirqs.CPU169.RCU
    122846 ±  3%     -80.1%      24404 ±  7%  softirqs.CPU17.RCU
    120415 ±  3%     -81.4%      22402 ±  7%  softirqs.CPU170.RCU
    120815 ±  3%     -81.9%      21841 ±  8%  softirqs.CPU171.RCU
    120815 ±  3%     -78.4%      26108 ± 23%  softirqs.CPU172.RCU
    120549 ±  4%     -81.8%      21923 ±  4%  softirqs.CPU173.RCU
    121183 ±  4%     -77.9%      26774 ± 29%  softirqs.CPU174.RCU
    120427 ±  3%     -82.4%      21251 ±  4%  softirqs.CPU175.RCU
    120157 ±  3%     -82.0%      21622 ±  4%  softirqs.CPU176.RCU
    121995 ±  3%     -77.7%      27248 ± 35%  softirqs.CPU177.RCU
    120342 ±  3%     -82.3%      21340 ±  6%  softirqs.CPU178.RCU
    120428 ±  3%     -79.8%      24322 ± 25%  softirqs.CPU179.RCU
    121835 ±  3%     -81.4%      22626 ±  5%  softirqs.CPU18.RCU
    121482 ±  4%     -82.3%      21524 ±  4%  softirqs.CPU180.RCU
    120787 ±  3%     -72.2%      33551 ± 62%  softirqs.CPU181.RCU
    119994 ±  3%     -75.9%      28883 ± 41%  softirqs.CPU182.RCU
    120367 ±  4%     -81.0%      22875 ±  8%  softirqs.CPU183.RCU
    119780 ±  3%     -76.3%      28333 ± 45%  softirqs.CPU184.RCU
    120497 ±  3%     -81.3%      22501 ± 16%  softirqs.CPU185.RCU
    119904 ±  3%     -82.9%      20445 ±  3%  softirqs.CPU186.RCU
    121045 ±  3%     -82.5%      21187 ±  8%  softirqs.CPU187.RCU
    120328 ±  3%     -81.9%      21766 ± 12%  softirqs.CPU188.RCU
    120350 ±  3%     -82.8%      20693 ±  5%  softirqs.CPU189.RCU
    123716 ±  5%     -75.9%      29872 ± 44%  softirqs.CPU19.RCU
    120381 ±  3%     -82.0%      21612 ±  6%  softirqs.CPU190.RCU
    118142 ±  3%     -72.7%      32219 ± 29%  softirqs.CPU191.RCU
    122126 ±  4%     -80.9%      23356 ± 10%  softirqs.CPU2.RCU
    122167 ±  3%     -78.9%      25813 ± 23%  softirqs.CPU20.RCU
    123162 ±  4%     -79.5%      25213 ± 16%  softirqs.CPU21.RCU
    122693 ±  4%     -64.0%      44179 ± 87%  softirqs.CPU22.RCU
    121919 ±  4%     -82.3%      21589 ±  4%  softirqs.CPU23.RCU
    121797 ±  3%     -80.6%      23685 ±  9%  softirqs.CPU24.RCU
    122393 ±  3%     -72.2%      34009 ± 56%  softirqs.CPU25.RCU
    126232 ±  5%     -79.9%      25386 ± 13%  softirqs.CPU26.RCU
    122998 ±  2%     -80.1%      24532 ± 14%  softirqs.CPU27.RCU
    122437 ±  3%     -80.4%      23973 ±  6%  softirqs.CPU28.RCU
    121849 ±  3%     -82.4%      21435        softirqs.CPU29.RCU
    122433 ±  2%     -81.9%      22102 ±  5%  softirqs.CPU3.RCU
    121720 ±  3%     -81.8%      22096 ±  7%  softirqs.CPU30.RCU
    122090 ±  3%     -79.9%      24532 ± 19%  softirqs.CPU31.RCU
    121948 ±  4%     -76.8%      28316 ± 38%  softirqs.CPU32.RCU
    122464 ±  3%     -80.7%      23628 ±  7%  softirqs.CPU33.RCU
    121676 ±  3%     -79.5%      24919 ± 13%  softirqs.CPU34.RCU
    122035 ±  4%     -75.6%      29752 ± 46%  softirqs.CPU35.RCU
    123266 ±  4%     -82.0%      22167 ±  5%  softirqs.CPU36.RCU
    122907 ±  4%     -82.1%      22050 ±  3%  softirqs.CPU37.RCU
    121814 ±  3%     -81.1%      22970 ±  2%  softirqs.CPU38.RCU
    121677 ±  3%     -80.6%      23665 ± 12%  softirqs.CPU39.RCU
    121971 ±  3%     -76.1%      29178 ± 40%  softirqs.CPU4.RCU
    122527 ±  4%     -78.0%      26924 ± 33%  softirqs.CPU40.RCU
    121714 ±  4%     -82.1%      21763 ±  4%  softirqs.CPU41.RCU
    121562 ±  3%     -82.3%      21485 ±  5%  softirqs.CPU42.RCU
    121462 ±  3%     -82.4%      21383 ±  3%  softirqs.CPU43.RCU
    122502 ±  3%     -82.4%      21501 ±  3%  softirqs.CPU44.RCU
    121816 ±  3%     -81.3%      22769        softirqs.CPU45.RCU
    121634 ±  3%     -82.0%      21879 ±  3%  softirqs.CPU46.RCU
    122430 ±  4%     -81.7%      22355 ±  7%  softirqs.CPU47.RCU
    121830 ±  3%     -82.7%      21130 ±  4%  softirqs.CPU48.RCU
     95769           +11.3%     106574 ±  3%  softirqs.CPU48.TIMER
    121991 ±  3%     -81.1%      23044 ± 11%  softirqs.CPU49.RCU
     95523           +11.4%     106432 ±  3%  softirqs.CPU49.TIMER
    122137 ±  3%     -78.2%      26669 ± 34%  softirqs.CPU5.RCU
    122456 ±  3%     -80.0%      24462 ± 10%  softirqs.CPU50.RCU
     96131           +12.5%     108119 ±  2%  softirqs.CPU50.TIMER
    121969 ±  3%     -81.7%      22265 ±  6%  softirqs.CPU51.RCU
     95983           +12.3%     107747 ±  2%  softirqs.CPU51.TIMER
    122703 ±  3%     -82.2%      21820 ±  2%  softirqs.CPU52.RCU
     95916           +10.5%     106003 ±  3%  softirqs.CPU52.TIMER
    122419 ±  3%     -80.9%      23345 ±  6%  softirqs.CPU53.RCU
     94966           +12.6%     106919 ±  3%  softirqs.CPU53.TIMER
    122291 ±  4%     -77.7%      27306 ± 29%  softirqs.CPU54.RCU
     96020 ±  2%     +10.7%     106300 ±  4%  softirqs.CPU54.TIMER
    122127 ±  3%     -82.1%      21900 ±  3%  softirqs.CPU55.RCU
     95535           +11.6%     106587 ±  4%  softirqs.CPU55.TIMER
     96312           +15.3%     111006 ±  9%  softirqs.CPU56.TIMER
    121902 ±  3%     -82.7%      21117 ±  2%  softirqs.CPU57.RCU
     95488           +10.5%     105536 ±  3%  softirqs.CPU57.TIMER
    122421 ±  4%     -74.0%      31825 ± 54%  softirqs.CPU58.RCU
     95739           +11.6%     106810 ±  4%  softirqs.CPU58.TIMER
    122098 ±  3%     -74.6%      30978 ± 52%  softirqs.CPU59.RCU
     94942           +12.2%     106502 ±  4%  softirqs.CPU59.TIMER
    122835 ±  2%     -82.4%      21670 ±  3%  softirqs.CPU6.RCU
    121413 ±  3%     -82.4%      21346        softirqs.CPU60.RCU
     95319           +11.0%     105777 ±  3%  softirqs.CPU60.TIMER
    122926 ±  2%     -82.1%      21943 ±  3%  softirqs.CPU61.RCU
     95741           +10.7%     105942 ±  4%  softirqs.CPU61.TIMER
    121526 ±  3%     -81.6%      22340 ±  5%  softirqs.CPU62.RCU
     95950           +11.9%     107336 ±  4%  softirqs.CPU62.TIMER
    121292 ±  3%     -82.5%      21176 ±  3%  softirqs.CPU63.RCU
     94965           +12.0%     106386 ±  3%  softirqs.CPU63.TIMER
    121540 ±  3%     -81.2%      22893 ± 10%  softirqs.CPU64.RCU
     95189           +11.6%     106251 ±  3%  softirqs.CPU64.TIMER
    122504 ±  3%     -76.0%      29413 ± 42%  softirqs.CPU65.RCU
     95180           +17.4%     111714 ± 10%  softirqs.CPU65.TIMER
    121587 ±  3%     -81.2%      22902 ±  9%  softirqs.CPU66.RCU
     95668 ±  2%     +11.5%     106700 ±  3%  softirqs.CPU66.TIMER
    121206 ±  3%     -81.6%      22244 ±  3%  softirqs.CPU67.RCU
     95408           +11.2%     106053 ±  4%  softirqs.CPU67.TIMER
    122606 ±  4%     -81.4%      22794 ±  6%  softirqs.CPU68.RCU
    122439 ±  3%     -81.7%      22451 ±  7%  softirqs.CPU69.RCU
     95026           +11.7%     106157 ±  3%  softirqs.CPU69.TIMER
    121135 ±  3%     -82.2%      21585 ±  3%  softirqs.CPU70.RCU
     94683           +11.9%     105921 ±  3%  softirqs.CPU70.TIMER
    121336 ±  3%     -81.7%      22194 ±  4%  softirqs.CPU71.RCU
     94589           +12.6%     106477 ±  3%  softirqs.CPU71.TIMER
    121229 ±  3%     -74.4%      31015 ± 30%  softirqs.CPU72.RCU
    121269 ±  3%     -81.3%      22633 ±  7%  softirqs.CPU73.RCU
    121888 ±  3%     -79.9%      24499 ± 20%  softirqs.CPU74.RCU
    121529 ±  3%     -80.6%      23547 ± 16%  softirqs.CPU76.RCU
    121040 ±  3%     -81.0%      22966 ± 12%  softirqs.CPU77.RCU
    121399 ±  3%     -80.4%      23790 ± 19%  softirqs.CPU78.RCU
    121097 ±  3%     -82.8%      20820        softirqs.CPU79.RCU
    122529 ±  3%     -79.3%      25339 ±  3%  softirqs.CPU8.RCU
    120997 ±  3%     -82.3%      21360 ±  3%  softirqs.CPU80.RCU
    122407 ±  3%     -79.8%      24728 ± 23%  softirqs.CPU81.RCU
    121216 ±  3%     -81.5%      22402 ±  9%  softirqs.CPU82.RCU
    100697            +8.7%     109454 ±  4%  softirqs.CPU82.TIMER
    121722 ±  3%     -77.6%      27215 ± 30%  softirqs.CPU83.RCU
    122208 ±  4%     -82.6%      21216 ±  3%  softirqs.CPU84.RCU
    122063 ±  3%     -73.0%      32931 ± 61%  softirqs.CPU85.RCU
    120977 ±  3%     -77.0%      27765 ± 41%  softirqs.CPU86.RCU
    120869 ±  3%     -81.3%      22622 ±  7%  softirqs.CPU87.RCU
    121449 ±  4%     -75.1%      30300 ± 27%  softirqs.CPU88.RCU
    121763 ±  3%     -80.6%      23666 ± 20%  softirqs.CPU89.RCU
    123584 ±  4%     -79.7%      25034 ±  7%  softirqs.CPU9.RCU
    121079 ±  3%     -82.4%      21281        softirqs.CPU90.RCU
    121422 ±  3%     -79.6%      24800 ± 22%  softirqs.CPU91.RCU
    121365 ±  3%     -80.9%      23228 ± 15%  softirqs.CPU92.RCU
    121458 ±  3%     -82.6%      21100 ±  3%  softirqs.CPU93.RCU
    120910 ±  3%     -80.8%      23238 ± 18%  softirqs.CPU94.RCU
    121727 ±  3%     -73.5%      32316 ± 18%  softirqs.CPU95.RCU
    122200 ±  3%     -82.2%      21697        softirqs.CPU96.RCU
    124724 ±  5%     -82.2%      22147 ±  3%  softirqs.CPU97.RCU
    121940 ±  4%     -82.1%      21783 ±  5%  softirqs.CPU98.RCU
    121481 ±  4%     -82.0%      21885 ±  3%  softirqs.CPU99.RCU
  23360980 ±  3%     -79.8%    4709898 ±  2%  softirqs.RCU
  18735607            +7.0%   20055373 ±  3%  softirqs.TIMER





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc4-00001-g8019ad13ef7f6" of type "text/plain" (203581 bytes)

View attachment "job-script" of type "text/plain" (7391 bytes)

View attachment "job.yaml" of type "text/plain" (4992 bytes)

View attachment "reproduce" of type "text/plain" (311 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ