lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Sun, 8 Mar 2020 22:03:14 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     yangerkun <yangerkun@...wei.com>
Cc:     Jeff Layton <jlayton@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        lkp@...ts.01.org
Subject: [locks] 6d390e4b5d: will-it-scale.per_process_ops -96.6% regression

Greeting,

FYI, we noticed a -96.6% regression of will-it-scale.per_process_ops due to commit:


commit: 6d390e4b5d48ec03bb87e63cf0a2bff5f4e116da ("locks: fix a potential use-after-free problem when wakeup a waiter")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: will-it-scale
on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
with following parameters:

	nr_task: 100%
	mode: process
	test: lock1
	cpufreq_governor: performance
	ucode: 0x11

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -51.3% regression        |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | mode=thread                                                          |
|                  | nr_task=100%                                                         |
|                  | test=lock1                                                           |
|                  | ucode=0x11                                                           |
+------------------+----------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/lock1/will-it-scale/0x11

commit: 
  0a68ff5e2e ("fcntl: Distribute switch variables for initialization")
  6d390e4b5d ("locks: fix a potential use-after-free problem when wakeup a waiter")

0a68ff5e2e7cf226 6d390e4b5d48ec03bb87e63cf0a 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     66597 ±  3%     -96.6%       2260        will-it-scale.per_process_ops
    313.70            -1.2%     310.06        will-it-scale.time.elapsed_time
    313.70            -1.2%     310.06        will-it-scale.time.elapsed_time.max
  19180216 ±  3%     -96.6%     651090        will-it-scale.workload
     63324 ±  9%     -27.5%      45902        meminfo.Mapped
     52611 ± 11%     -33.5%      35010 ±  2%  numa-meminfo.node0.Mapped
     13299 ± 11%     -33.6%       8829        numa-vmstat.node0.nr_mapped
      1440 ±  3%      -8.9%       1312 ±  4%  slabinfo.fsnotify_mark_connector.active_objs
      1440 ±  3%      -8.9%       1312 ±  4%  slabinfo.fsnotify_mark_connector.num_objs
      0.00 ± 10%      -0.0        0.00 ± 17%  mpstat.cpu.all.soft%
     86.17           +11.7       97.88        mpstat.cpu.all.sys%
     12.62 ±  8%     -11.7        0.89 ±  6%  mpstat.cpu.all.usr%
     85.00           +13.8%      96.75        vmstat.cpu.sy
     12.00 ± 10%    -100.0%       0.00        vmstat.cpu.us
      2274            -2.9%       2208        vmstat.system.cs
     15943 ±  9%     -27.5%      11561        proc-vmstat.nr_mapped
      1809 ± 75%     -91.3%     157.00 ±  6%  proc-vmstat.numa_hint_faults
      1809 ± 75%     -91.3%     157.00 ±  6%  proc-vmstat.numa_hint_faults_local
    914333            +3.7%     948222        proc-vmstat.numa_hit
    914333            +3.7%     948222        proc-vmstat.numa_local
      3736 ±  6%     +18.5%       4427 ±  2%  proc-vmstat.pgactivate
    990333            +4.3%    1032696        proc-vmstat.pgalloc_normal
    862745            +3.7%     894537        proc-vmstat.pgfault
      2383 ±  7%     -55.4%       1064        sched_debug.cfs_rq:/.exec_clock.stddev
   1089611 ±  4%     -45.1%     597726        sched_debug.cfs_rq:/.min_vruntime.stddev
      1.52 ± 10%     +58.6%       2.41 ±  8%  sched_debug.cfs_rq:/.nr_spread_over.avg
  -9694655           -44.6%   -5372610        sched_debug.cfs_rq:/.spread0.min
    885953 ±  2%     -36.8%     560044 ±  2%  sched_debug.cfs_rq:/.spread0.stddev
    493.40 ±  8%     +32.0%     651.30 ± 10%  sched_debug.cfs_rq:/.util_avg.min
     62.39 ±  8%     -21.1%      49.24 ± 13%  sched_debug.cfs_rq:/.util_avg.stddev
    131.46 ±  4%     -12.4%     115.20 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    782383 ±  2%     -13.4%     677845 ±  8%  sched_debug.cpu.avg_idle.min
      2142 ± 12%     -55.3%     957.56 ±  4%  sched_debug.cpu.clock.stddev
      2142 ± 12%     -55.3%     957.56 ±  4%  sched_debug.cpu.clock_task.stddev
    289492 ± 13%     -27.7%     209254 ± 17%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ± 11%     -55.5%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
     40490 ± 16%     -41.8%      23551 ±  4%  sched_debug.cpu.nr_switches.max
      3333 ±  9%     -28.6%       2380        sched_debug.cpu.nr_switches.stddev
      1152            -9.2%       1045        sched_debug.cpu.sched_count.avg
     36522 ± 17%     -46.5%      19545 ±  5%  sched_debug.cpu.sched_count.max
      3083 ± 10%     -32.8%       2072        sched_debug.cpu.sched_count.stddev
     17142 ± 17%     -44.3%       9552 ±  5%  sched_debug.cpu.ttwu_count.max
      1439 ±  8%     -31.9%     981.06        sched_debug.cpu.ttwu_count.stddev
     16805 ± 18%     -46.5%       8998 ±  5%  sched_debug.cpu.ttwu_local.max
      1371 ± 10%     -33.5%     912.34        sched_debug.cpu.ttwu_local.stddev
      0.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_nr_migratory.avg
      0.20          -100.0%       0.00        sched_debug.rt_rq:/.rt_nr_migratory.max
      0.01          -100.0%       0.00        sched_debug.rt_rq:/.rt_nr_migratory.stddev
      0.14 ± 41%     -83.3%       0.02 ± 23%  sched_debug.rt_rq:/.rt_time.avg
     39.32 ± 41%     -99.1%       0.35 ± 22%  sched_debug.rt_rq:/.rt_time.max
      2.31 ± 41%     -97.4%       0.06 ±  4%  sched_debug.rt_rq:/.rt_time.stddev
     33.60           -82.4%       5.92        perf-stat.i.MPKI
 7.654e+09 ±  3%     +10.5%  8.458e+09        perf-stat.i.branch-instructions
      7.01            -6.2        0.84        perf-stat.i.branch-miss-rate%
 5.368e+08 ±  4%     -87.3%   67920479        perf-stat.i.branch-misses
      5.18           +17.7       22.86        perf-stat.i.cache-miss-rate%
  65182049 ±  2%     -29.9%   45680888        perf-stat.i.cache-misses
 1.277e+09 ±  3%     -84.3%      2e+08        perf-stat.i.cache-references
      2266            -4.2%       2170        perf-stat.i.context-switches
     11.58 ±  3%     +11.0%      12.85        perf-stat.i.cpi
   4.4e+11            +1.1%  4.447e+11        perf-stat.i.cpu-cycles
    242.58 ±  2%      +3.9%     252.10        perf-stat.i.cpu-migrations
      6729 ±  2%     +44.4%       9715        perf-stat.i.cycles-between-cache-misses
      2.30            -2.1        0.19        perf-stat.i.iTLB-load-miss-rate%
 8.993e+08 ±  3%     -92.9%   63736147        perf-stat.i.iTLB-load-misses
 3.808e+10 ±  3%      -9.2%  3.458e+10        perf-stat.i.iTLB-loads
 3.797e+10 ±  3%      -9.1%  3.452e+10        perf-stat.i.instructions
     42.78         +1170.0%     543.24        perf-stat.i.instructions-per-iTLB-miss
      0.09 ±  3%     -10.5%       0.08        perf-stat.i.ipc
     33.74           -82.8%       5.80        perf-stat.overall.MPKI
      7.01            -6.2        0.80        perf-stat.overall.branch-miss-rate%
      5.11           +17.7       22.85        perf-stat.overall.cache-miss-rate%
     11.62 ±  3%     +10.9%      12.88        perf-stat.overall.cpi
      6738 ±  2%     +44.4%       9728        perf-stat.overall.cycles-between-cache-misses
      2.31            -2.1        0.18        perf-stat.overall.iTLB-load-miss-rate%
     42.21         +1186.2%     542.88        perf-stat.overall.instructions-per-iTLB-miss
      0.09 ±  3%      -9.9%       0.08        perf-stat.overall.ipc
    618557         +2550.0%   16391579        perf-stat.overall.path-length
 7.631e+09 ±  3%     +10.6%   8.44e+09        perf-stat.ps.branch-instructions
 5.353e+08 ±  4%     -87.4%   67223267        perf-stat.ps.branch-misses
  65236586 ±  2%     -30.1%   45607616        perf-stat.ps.cache-misses
 1.277e+09 ±  3%     -84.4%  1.997e+08        perf-stat.ps.cache-references
      2189            -2.7%       2129        perf-stat.ps.context-switches
 4.393e+11            +1.0%  4.436e+11        perf-stat.ps.cpu-cycles
    217.99 ±  2%     +10.2%     240.17        perf-stat.ps.cpu-migrations
 8.968e+08 ±  3%     -92.9%   63451935        perf-stat.ps.iTLB-load-misses
  3.79e+10 ±  3%      -9.0%  3.448e+10        perf-stat.ps.iTLB-loads
 3.785e+10 ±  3%      -9.0%  3.445e+10        perf-stat.ps.instructions
      2653            +4.3%       2766        perf-stat.ps.minor-faults
      2653            +4.3%       2766        perf-stat.ps.page-faults
 1.186e+13 ±  2%     -10.0%  1.067e+13        perf-stat.total.instructions
     30.15           -29.6        0.52 ±  3%  perf-profile.calltrace.cycles-pp.posix_lock_inode.do_lock_file_wait.fcntl_setlk.do_fcntl.__x64_sys_fcntl
     10.22 ±  7%     -10.2        0.00        perf-profile.calltrace.cycles-pp.locks_alloc_lock.posix_lock_inode.do_lock_file_wait.fcntl_setlk.do_fcntl
      9.82 ± 21%      -9.8        0.00        perf-profile.calltrace.cycles-pp._copy_from_user.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.34            -9.3        0.00        perf-profile.calltrace.cycles-pp.locks_alloc_lock.fcntl_setlk.do_fcntl.__x64_sys_fcntl.do_syscall_64
      8.91 ±  8%      -8.9        0.00        perf-profile.calltrace.cycles-pp.kmem_cache_alloc.locks_alloc_lock.posix_lock_inode.do_lock_file_wait.fcntl_setlk
      7.83 ±  3%      -7.8        0.00        perf-profile.calltrace.cycles-pp.kmem_cache_alloc.locks_alloc_lock.fcntl_setlk.do_fcntl.__x64_sys_fcntl
      7.46 ± 26%      -7.5        0.00        perf-profile.calltrace.cycles-pp.__might_fault._copy_from_user.do_fcntl.__x64_sys_fcntl.do_syscall_64
      7.17 ±  9%      -7.2        0.00        perf-profile.calltrace.cycles-pp.security_file_lock.do_lock_file_wait.fcntl_setlk.do_fcntl.__x64_sys_fcntl
      5.86 ± 35%      -5.9        0.00        perf-profile.calltrace.cycles-pp.___might_sleep.__might_fault._copy_from_user.do_fcntl.__x64_sys_fcntl
      5.71 ± 12%      -5.7        0.00        perf-profile.calltrace.cycles-pp.common_file_perm.security_file_lock.do_lock_file_wait.fcntl_setlk.do_fcntl
      4.75 ± 16%      -4.8        0.00        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      0.92 ±  3%      -0.1        0.81        perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.91 ±  3%      -0.1        0.81        perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.91 ±  3%      -0.1        0.81        perf-profile.calltrace.cycles-pp.task_numa_work.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.91 ±  2%      -0.1        0.80        perf-profile.calltrace.cycles-pp.change_protection.change_prot_numa.task_numa_work.task_work_run.exit_to_usermode_loop
      0.91 ±  2%      -0.1        0.80        perf-profile.calltrace.cycles-pp.change_prot_numa.task_numa_work.task_work_run.exit_to_usermode_loop.do_syscall_64
      0.90 ±  3%      -0.1        0.79        perf-profile.calltrace.cycles-pp.change_p4d_range.change_protection.change_prot_numa.task_numa_work.task_work_run
      0.00            +1.0        1.01 ± 25%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +1.0        1.04 ± 25%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.00            +1.1        1.14 ± 26%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +2.1        2.09 ± 36%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath
      0.00            +3.7        3.74 ± 38%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
      0.00            +5.1        5.14 ± 37%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.locks_delete_block
      0.00            +5.5        5.54 ± 36%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.locks_delete_block.do_lock_file_wait
     87.63           +11.5       99.09        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     86.40           +12.7       99.06        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     80.12           +18.0       98.08        perf-profile.calltrace.cycles-pp.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     76.44           +21.5       97.98        perf-profile.calltrace.cycles-pp.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     59.98           +37.9       97.85        perf-profile.calltrace.cycles-pp.fcntl_setlk.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     41.88           +55.6       97.44        perf-profile.calltrace.cycles-pp.do_lock_file_wait.fcntl_setlk.do_fcntl.__x64_sys_fcntl.do_syscall_64
      0.52 ± 60%     +96.2       96.76        perf-profile.calltrace.cycles-pp.locks_delete_block.do_lock_file_wait.fcntl_setlk.do_fcntl.__x64_sys_fcntl
      0.00           +96.3       96.31        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.locks_delete_block.do_lock_file_wait.fcntl_setlk
      0.00           +96.5       96.52        perf-profile.calltrace.cycles-pp._raw_spin_lock.locks_delete_block.do_lock_file_wait.fcntl_setlk.do_fcntl
     30.44           -29.9        0.53 ±  3%  perf-profile.children.cycles-pp.posix_lock_inode
     19.64 ±  4%     -19.3        0.32 ±  3%  perf-profile.children.cycles-pp.locks_alloc_lock
     16.97 ±  5%     -16.7        0.25 ±  5%  perf-profile.children.cycles-pp.kmem_cache_alloc
      9.99 ± 32%      -9.9        0.07 ± 10%  perf-profile.children.cycles-pp.___might_sleep
      9.87 ± 21%      -9.8        0.10 ±  5%  perf-profile.children.cycles-pp._copy_from_user
      7.51 ± 25%      -7.5        0.00        perf-profile.children.cycles-pp.__might_fault
      7.19 ±  9%      -7.1        0.12 ± 11%  perf-profile.children.cycles-pp.security_file_lock
      5.73 ± 12%      -5.6        0.10 ± 10%  perf-profile.children.cycles-pp.common_file_perm
      5.06 ± 15%      -4.9        0.20 ±  5%  perf-profile.children.cycles-pp.syscall_return_via_sysret
     10.41 ± 29%      -4.6        5.80 ± 35%  perf-profile.children.cycles-pp.apic_timer_interrupt
      9.58 ± 29%      -4.2        5.35 ± 37%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      3.79 ±  7%      -3.7        0.09 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      7.66 ± 32%      -3.7        3.97 ± 38%  perf-profile.children.cycles-pp.hrtimer_interrupt
      3.73            -3.6        0.13 ±  5%  perf-profile.children.cycles-pp.kmem_cache_free
      2.77 ±  2%      -2.7        0.07 ±  6%  perf-profile.children.cycles-pp.memset_erms
      4.70 ± 27%      -2.5        2.18 ± 35%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      2.72 ± 33%      -1.5        1.18 ± 26%  perf-profile.children.cycles-pp.tick_sched_timer
      1.52 ±  4%      -1.5        0.07 ±  7%  perf-profile.children.cycles-pp.__fget_light
      1.51 ±  6%      -1.5        0.05        perf-profile.children.cycles-pp.locks_delete_lock_ctx
      2.44 ± 32%      -1.4        1.08 ± 25%  perf-profile.children.cycles-pp.tick_sched_handle
      2.32 ± 33%      -1.3        1.05 ± 25%  perf-profile.children.cycles-pp.update_process_times
      1.11 ±  5%      -1.1        0.06 ±  7%  perf-profile.children.cycles-pp.fcntl
      1.50 ± 29%      -0.9        0.64 ± 16%  perf-profile.children.cycles-pp.scheduler_tick
      0.91 ± 28%      -0.7        0.24 ± 12%  perf-profile.children.cycles-pp.irq_exit
      0.71 ±  7%      -0.6        0.07 ± 12%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      1.08 ± 29%      -0.6        0.48 ± 15%  perf-profile.children.cycles-pp.task_tick_fair
      0.53 ± 45%      -0.4        0.13 ±  8%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.36 ±  3%      -0.3        0.03 ±100%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.49 ± 33%      -0.3        0.21 ± 17%  perf-profile.children.cycles-pp.update_curr
      1.07 ±  3%      -0.2        0.82        perf-profile.children.cycles-pp.exit_to_usermode_loop
      1.05 ±  3%      -0.2        0.82        perf-profile.children.cycles-pp.task_work_run
      1.04 ±  3%      -0.2        0.81        perf-profile.children.cycles-pp.change_protection
      1.04 ±  3%      -0.2        0.81        perf-profile.children.cycles-pp.change_prot_numa
      1.04 ±  3%      -0.2        0.81        perf-profile.children.cycles-pp.change_p4d_range
      1.05 ±  3%      -0.2        0.82        perf-profile.children.cycles-pp.task_numa_work
      0.38 ± 28%      -0.2        0.18 ± 14%  perf-profile.children.cycles-pp.update_load_avg
      0.35 ± 24%      -0.2        0.16 ± 35%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.36 ± 17%      -0.2        0.18 ± 46%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.22 ± 11%      -0.1        0.08 ± 19%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.18 ± 28%      -0.1        0.06 ± 71%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.21 ± 42%      -0.1        0.09 ± 37%  perf-profile.children.cycles-pp.ktime_get
      0.15 ± 53%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.run_timer_softirq
      0.19 ± 41%      -0.1        0.08 ± 40%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.14 ± 38%      -0.1        0.07 ± 16%  perf-profile.children.cycles-pp.cpuacct_charge
      0.17 ± 28%      -0.1        0.10 ± 44%  perf-profile.children.cycles-pp.read_tsc
      0.15 ±  3%      -0.1        0.08 ± 23%  perf-profile.children.cycles-pp.clockevents_program_event
      0.12 ± 26%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.11 ± 28%      -0.1        0.06 ± 14%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.08 ±  8%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.page_fault
      0.08 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.do_page_fault
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.__locks_wake_up_blocks
     88.01           +11.4       99.42        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     86.90           +12.5       99.39        perf-profile.children.cycles-pp.do_syscall_64
     80.17           +17.9       98.08        perf-profile.children.cycles-pp.__x64_sys_fcntl
     76.53           +21.5       97.98        perf-profile.children.cycles-pp.do_fcntl
     60.08           +37.8       97.86        perf-profile.children.cycles-pp.fcntl_setlk
     41.97           +55.5       97.45        perf-profile.children.cycles-pp.do_lock_file_wait
      3.80 ± 11%     +92.9       96.69        perf-profile.children.cycles-pp._raw_spin_lock
      0.65 ± 19%     +96.1       96.76        perf-profile.children.cycles-pp.locks_delete_block
      0.00           +96.5       96.52        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      8.92 ± 30%      -8.9        0.07 ±  7%  perf-profile.self.cycles-pp.___might_sleep
      8.75 ±  9%      -8.7        0.09 ±  4%  perf-profile.self.cycles-pp.posix_lock_inode
      5.96 ± 21%      -6.0        0.00        perf-profile.self.cycles-pp.do_fcntl
      5.31 ±  6%      -5.2        0.11 ±  4%  perf-profile.self.cycles-pp.fcntl_setlk
      5.25 ±  5%      -5.2        0.10 ±  5%  perf-profile.self.cycles-pp.kmem_cache_alloc
      5.05 ± 15%      -4.9        0.20 ±  5%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      4.01 ± 10%      -3.9        0.09 ±  5%  perf-profile.self.cycles-pp.do_syscall_64
      3.79 ±  7%      -3.7        0.09 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      3.32 ±  4%      -3.2        0.12 ±  3%  perf-profile.self.cycles-pp.kmem_cache_free
      3.39 ±  8%      -3.2        0.21 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock
      2.02 ±  3%      -2.0        0.06        perf-profile.self.cycles-pp.memset_erms
      1.75 ±  4%      -1.7        0.06 ±  7%  perf-profile.self.cycles-pp.common_file_perm
      1.65 ±  4%      -1.6        0.05 ±  8%  perf-profile.self.cycles-pp.locks_alloc_lock
      1.35 ±  6%      -1.3        0.05 ±  9%  perf-profile.self.cycles-pp.__fget_light
      0.59 ±  5%      -0.5        0.07 ± 12%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.31 ±  6%      -0.3        0.03 ±100%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.90 ±  5%      -0.2        0.70 ±  2%  perf-profile.self.cycles-pp.change_p4d_range
      0.33 ± 28%      -0.2        0.15 ± 38%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.28 ± 32%      -0.2        0.11 ± 15%  perf-profile.self.cycles-pp.update_curr
      0.22 ± 11%      -0.1        0.08 ± 19%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.18 ± 26%      -0.1        0.05 ± 67%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.17 ± 49%      -0.1        0.06 ± 68%  perf-profile.self.cycles-pp.ktime_get
      0.17 ± 24%      -0.1        0.06 ± 22%  perf-profile.self.cycles-pp.irq_exit
      0.16 ± 28%      -0.1        0.07 ± 42%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.12 ± 43%      -0.1        0.04 ±107%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.15 ± 30%      -0.1        0.07 ± 16%  perf-profile.self.cycles-pp.update_load_avg
      0.14 ± 38%      -0.1        0.07 ± 16%  perf-profile.self.cycles-pp.cpuacct_charge
      0.15 ± 29%      -0.1        0.09 ± 43%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__locks_wake_up_blocks
      0.00           +90.8       90.84 ±  2%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      4528 ± 34%     +23.5%       5591 ± 33%  interrupts.CPU1.NMI:Non-maskable_interrupts
      4528 ± 34%     +23.5%       5591 ± 33%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
      3384           +61.1%       5453 ± 33%  interrupts.CPU104.NMI:Non-maskable_interrupts
      3384           +61.1%       5453 ± 33%  interrupts.CPU104.PMI:Performance_monitoring_interrupts
      3355           +35.9%       4558 ± 34%  interrupts.CPU107.NMI:Non-maskable_interrupts
      3355           +35.9%       4558 ± 34%  interrupts.CPU107.PMI:Performance_monitoring_interrupts
    210.50 ±  3%     +15.4%     243.00 ±  7%  interrupts.CPU11.RES:Rescheduling_interrupts
      4217 ± 34%     +29.9%       5479 ± 33%  interrupts.CPU111.NMI:Non-maskable_interrupts
      4217 ± 34%     +29.9%       5479 ± 33%  interrupts.CPU111.PMI:Performance_monitoring_interrupts
      4238 ± 35%     +51.0%       6400 ± 24%  interrupts.CPU112.NMI:Non-maskable_interrupts
      4238 ± 35%     +51.0%       6400 ± 24%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
      3393           +60.6%       5450 ± 33%  interrupts.CPU113.NMI:Non-maskable_interrupts
      3393           +60.6%       5450 ± 33%  interrupts.CPU113.PMI:Performance_monitoring_interrupts
      4311 ± 33%     +27.0%       5475 ± 33%  interrupts.CPU114.NMI:Non-maskable_interrupts
      4311 ± 33%     +27.0%       5475 ± 33%  interrupts.CPU114.PMI:Performance_monitoring_interrupts
     10.25 ± 88%    +865.9%      99.00 ± 96%  interrupts.CPU117.RES:Rescheduling_interrupts
      4265 ± 35%     +27.9%       5457 ± 33%  interrupts.CPU121.NMI:Non-maskable_interrupts
      4265 ± 35%     +27.9%       5457 ± 33%  interrupts.CPU121.PMI:Performance_monitoring_interrupts
      4205 ± 33%     +30.1%       5473 ± 33%  interrupts.CPU127.NMI:Non-maskable_interrupts
      4205 ± 33%     +30.1%       5473 ± 33%  interrupts.CPU127.PMI:Performance_monitoring_interrupts
      4207 ± 33%     +30.0%       5469 ± 33%  interrupts.CPU133.NMI:Non-maskable_interrupts
      4207 ± 33%     +30.0%       5469 ± 33%  interrupts.CPU133.PMI:Performance_monitoring_interrupts
      4393 ± 32%     +45.9%       6412 ± 24%  interrupts.CPU135.NMI:Non-maskable_interrupts
      4393 ± 32%     +45.9%       6412 ± 24%  interrupts.CPU135.PMI:Performance_monitoring_interrupts
      4320 ± 35%     +26.6%       5468 ± 33%  interrupts.CPU137.NMI:Non-maskable_interrupts
      4320 ± 35%     +26.6%       5468 ± 33%  interrupts.CPU137.PMI:Performance_monitoring_interrupts
      3305           +37.8%       4554 ± 35%  interrupts.CPU141.NMI:Non-maskable_interrupts
      3305           +37.8%       4554 ± 35%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
    115.50 ± 56%     -94.6%       6.25 ± 17%  interrupts.CPU142.RES:Rescheduling_interrupts
      4205 ± 34%     +29.8%       5459 ± 33%  interrupts.CPU147.NMI:Non-maskable_interrupts
      4205 ± 34%     +29.8%       5459 ± 33%  interrupts.CPU147.PMI:Performance_monitoring_interrupts
      3316           +64.9%       5468 ± 33%  interrupts.CPU154.NMI:Non-maskable_interrupts
      3316           +64.9%       5468 ± 33%  interrupts.CPU154.PMI:Performance_monitoring_interrupts
      4250 ± 34%     +28.7%       5468 ± 33%  interrupts.CPU156.NMI:Non-maskable_interrupts
      4250 ± 34%     +28.7%       5468 ± 33%  interrupts.CPU156.PMI:Performance_monitoring_interrupts
      4315 ± 34%     +48.4%       6405 ± 24%  interrupts.CPU159.NMI:Non-maskable_interrupts
      4315 ± 34%     +48.4%       6405 ± 24%  interrupts.CPU159.PMI:Performance_monitoring_interrupts
      4267 ± 32%     +27.8%       5453 ± 33%  interrupts.CPU161.NMI:Non-maskable_interrupts
      4267 ± 32%     +27.8%       5453 ± 33%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
      3447           +58.7%       5472 ± 33%  interrupts.CPU162.NMI:Non-maskable_interrupts
      3447           +58.7%       5472 ± 33%  interrupts.CPU162.PMI:Performance_monitoring_interrupts
      4334 ± 32%     +26.1%       5467 ± 33%  interrupts.CPU169.NMI:Non-maskable_interrupts
      4334 ± 32%     +26.1%       5467 ± 33%  interrupts.CPU169.PMI:Performance_monitoring_interrupts
      6.25 ±  6%   +2788.0%     180.50 ±155%  interrupts.CPU173.RES:Rescheduling_interrupts
      3370 ±  2%     +61.9%       5457 ± 33%  interrupts.CPU175.NMI:Non-maskable_interrupts
      3370 ±  2%     +61.9%       5457 ± 33%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
      4256 ± 32%     +28.4%       5464 ± 33%  interrupts.CPU179.NMI:Non-maskable_interrupts
      4256 ± 32%     +28.4%       5464 ± 33%  interrupts.CPU179.PMI:Performance_monitoring_interrupts
      4199 ± 34%     +30.6%       5484 ± 33%  interrupts.CPU180.NMI:Non-maskable_interrupts
      4199 ± 34%     +30.6%       5484 ± 33%  interrupts.CPU180.PMI:Performance_monitoring_interrupts
      4320 ± 34%     +26.6%       5471 ± 33%  interrupts.CPU182.NMI:Non-maskable_interrupts
      4320 ± 34%     +26.6%       5471 ± 33%  interrupts.CPU182.PMI:Performance_monitoring_interrupts
     20.25 ± 75%    +840.7%     190.50 ± 92%  interrupts.CPU186.RES:Rescheduling_interrupts
     11.75 ± 79%   +1342.6%     169.50 ±107%  interrupts.CPU187.RES:Rescheduling_interrupts
      4340 ± 32%     +25.7%       5457 ± 33%  interrupts.CPU189.NMI:Non-maskable_interrupts
      4340 ± 32%     +25.7%       5457 ± 33%  interrupts.CPU189.PMI:Performance_monitoring_interrupts
      4258 ± 36%     +28.2%       5458 ± 33%  interrupts.CPU191.NMI:Non-maskable_interrupts
      4258 ± 36%     +28.2%       5458 ± 33%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
      6.50 ± 23%   +1034.6%      73.75 ±104%  interrupts.CPU199.RES:Rescheduling_interrupts
    153.00 ± 57%     -72.5%      42.00 ±144%  interrupts.CPU200.RES:Rescheduling_interrupts
      3332           +64.2%       5472 ± 33%  interrupts.CPU202.NMI:Non-maskable_interrupts
      3332           +64.2%       5472 ± 33%  interrupts.CPU202.PMI:Performance_monitoring_interrupts
      3397 ±  2%     +60.9%       5464 ± 33%  interrupts.CPU203.NMI:Non-maskable_interrupts
      3397 ±  2%     +60.9%       5464 ± 33%  interrupts.CPU203.PMI:Performance_monitoring_interrupts
      4299 ± 31%     +48.0%       6364 ± 24%  interrupts.CPU205.NMI:Non-maskable_interrupts
      4299 ± 31%     +48.0%       6364 ± 24%  interrupts.CPU205.PMI:Performance_monitoring_interrupts
      4030 ± 33%     +34.3%       5413 ± 33%  interrupts.CPU214.NMI:Non-maskable_interrupts
      4030 ± 33%     +34.3%       5413 ± 33%  interrupts.CPU214.PMI:Performance_monitoring_interrupts
      4060 ± 35%     +54.4%       6267 ± 24%  interrupts.CPU215.NMI:Non-maskable_interrupts
      4060 ± 35%     +54.4%       6267 ± 24%  interrupts.CPU215.PMI:Performance_monitoring_interrupts
      4242 ± 37%     +50.4%       6382 ± 24%  interrupts.CPU225.NMI:Non-maskable_interrupts
      4242 ± 37%     +50.4%       6382 ± 24%  interrupts.CPU225.PMI:Performance_monitoring_interrupts
      3351           +63.0%       5462 ± 33%  interrupts.CPU226.NMI:Non-maskable_interrupts
      3351           +63.0%       5462 ± 33%  interrupts.CPU226.PMI:Performance_monitoring_interrupts
      7.25 ±  5%    +455.2%      40.25 ±117%  interrupts.CPU237.RES:Rescheduling_interrupts
      4169 ± 34%     +30.8%       5453 ± 33%  interrupts.CPU242.NMI:Non-maskable_interrupts
      4169 ± 34%     +30.8%       5453 ± 33%  interrupts.CPU242.PMI:Performance_monitoring_interrupts
      3645 ±  2%     +80.2%       6568 ± 24%  interrupts.CPU25.NMI:Non-maskable_interrupts
      3645 ±  2%     +80.2%       6568 ± 24%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
     26.00 ± 93%    +714.4%     211.75 ± 70%  interrupts.CPU258.RES:Rescheduling_interrupts
      8.00 ± 15%   +1962.5%     165.00 ±156%  interrupts.CPU260.RES:Rescheduling_interrupts
      4144 ± 33%     +75.5%       7274        interrupts.CPU262.NMI:Non-maskable_interrupts
      4144 ± 33%     +75.5%       7274        interrupts.CPU262.PMI:Performance_monitoring_interrupts
      4391 ± 31%     +24.6%       5470 ± 33%  interrupts.CPU264.NMI:Non-maskable_interrupts
      4391 ± 31%     +24.6%       5470 ± 33%  interrupts.CPU264.PMI:Performance_monitoring_interrupts
      3475 ±  2%     +83.4%       6373 ± 24%  interrupts.CPU274.NMI:Non-maskable_interrupts
      3475 ±  2%     +83.4%       6373 ± 24%  interrupts.CPU274.PMI:Performance_monitoring_interrupts
      4454 ± 35%     +22.7%       5465 ± 32%  interrupts.CPU276.NMI:Non-maskable_interrupts
      4454 ± 35%     +22.7%       5465 ± 32%  interrupts.CPU276.PMI:Performance_monitoring_interrupts
      4407 ± 35%     +44.3%       6361 ± 24%  interrupts.CPU277.NMI:Non-maskable_interrupts
      4407 ± 35%     +44.3%       6361 ± 24%  interrupts.CPU277.PMI:Performance_monitoring_interrupts
      4466 ± 31%     +67.9%       7497        interrupts.CPU28.NMI:Non-maskable_interrupts
      4466 ± 31%     +67.9%       7497        interrupts.CPU28.PMI:Performance_monitoring_interrupts
      4392 ± 31%     +24.5%       5467 ± 33%  interrupts.CPU284.NMI:Non-maskable_interrupts
      4392 ± 31%     +24.5%       5467 ± 33%  interrupts.CPU284.PMI:Performance_monitoring_interrupts
      3476 ±  2%     +57.0%       5456 ± 33%  interrupts.CPU285.NMI:Non-maskable_interrupts
      3476 ±  2%     +57.0%       5456 ± 33%  interrupts.CPU285.PMI:Performance_monitoring_interrupts
      4306 ± 31%     +47.0%       6329 ± 24%  interrupts.CPU286.NMI:Non-maskable_interrupts
      4306 ± 31%     +47.0%       6329 ± 24%  interrupts.CPU286.PMI:Performance_monitoring_interrupts
      3471 ±  2%     +55.4%       5394 ± 33%  interrupts.CPU287.NMI:Non-maskable_interrupts
      3471 ±  2%     +55.4%       5394 ± 33%  interrupts.CPU287.PMI:Performance_monitoring_interrupts
    189.00 ± 88%     -93.1%      13.00 ±  5%  interrupts.CPU287.TLB:TLB_shootdowns
      7308           -48.9%       3736        interrupts.CPU31.NMI:Non-maskable_interrupts
      7308           -48.9%       3736        interrupts.CPU31.PMI:Performance_monitoring_interrupts
    203.50 ±144%     -96.3%       7.50 ± 42%  interrupts.CPU31.RES:Rescheduling_interrupts
    410.25 ±133%     -92.8%      29.50 ±120%  interrupts.CPU41.RES:Rescheduling_interrupts
     33.25 ±100%    +740.6%     279.50 ± 41%  interrupts.CPU42.RES:Rescheduling_interrupts
      3654           +77.2%       6476 ± 24%  interrupts.CPU5.NMI:Non-maskable_interrupts
      3654           +77.2%       6476 ± 24%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      3519 ±  2%     +83.1%       6442 ± 24%  interrupts.CPU53.NMI:Non-maskable_interrupts
      3519 ±  2%     +83.1%       6442 ± 24%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      4446 ± 35%     +24.3%       5527 ± 33%  interrupts.CPU55.NMI:Non-maskable_interrupts
      4446 ± 35%     +24.3%       5527 ± 33%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
    697.50 ± 79%     -94.2%      40.75 ±117%  interrupts.CPU56.RES:Rescheduling_interrupts
      4437 ± 36%     +24.2%       5512 ± 33%  interrupts.CPU59.NMI:Non-maskable_interrupts
      4437 ± 36%     +24.2%       5512 ± 33%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
    204.00 ±  3%     +18.3%     241.25 ±  7%  interrupts.CPU6.RES:Rescheduling_interrupts
      4471 ± 35%     +43.9%       6433 ± 24%  interrupts.CPU61.NMI:Non-maskable_interrupts
      4471 ± 35%     +43.9%       6433 ± 24%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      4440 ± 35%     +45.3%       6453 ± 24%  interrupts.CPU63.NMI:Non-maskable_interrupts
      4440 ± 35%     +45.3%       6453 ± 24%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
     98.00 ± 68%     -63.5%      35.75 ± 37%  interrupts.CPU71.RES:Rescheduling_interrupts
      4189 ± 34%     +53.1%       6413 ± 24%  interrupts.CPU76.NMI:Non-maskable_interrupts
      4189 ± 34%     +53.1%       6413 ± 24%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      4307 ± 35%     +48.3%       6389 ± 24%  interrupts.CPU79.NMI:Non-maskable_interrupts
      4307 ± 35%     +48.3%       6389 ± 24%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      4535 ± 33%     +23.9%       5620 ± 33%  interrupts.CPU8.NMI:Non-maskable_interrupts
      4535 ± 33%     +23.9%       5620 ± 33%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      4156 ± 33%     +31.3%       5459 ± 33%  interrupts.CPU81.NMI:Non-maskable_interrupts
      4156 ± 33%     +31.3%       5459 ± 33%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      3514 ±  2%     +56.1%       5487 ± 33%  interrupts.CPU85.NMI:Non-maskable_interrupts
      3514 ±  2%     +56.1%       5487 ± 33%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
    157.25 ±140%     -96.0%       6.25 ± 13%  interrupts.CPU94.RES:Rescheduling_interrupts
      3450           +58.9%       5481 ± 33%  interrupts.CPU96.NMI:Non-maskable_interrupts
      3450           +58.9%       5481 ± 33%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
      6757 ±  2%     -46.0%       3648        interrupts.CPU99.NMI:Non-maskable_interrupts
      6757 ±  2%     -46.0%       3648        interrupts.CPU99.PMI:Performance_monitoring_interrupts
     32590 ±  8%     -29.7%      22912        interrupts.RES:Rescheduling_interrupts
    149331           -44.0%      83638 ± 18%  softirqs.CPU0.RCU
    154172           -65.3%      53549 ± 52%  softirqs.CPU1.RCU
    149705           -75.4%      36779 ±  3%  softirqs.CPU10.RCU
    137249 ±  6%     -52.8%      64842 ± 22%  softirqs.CPU100.RCU
    144358 ±  3%     -57.4%      61477 ±  7%  softirqs.CPU101.RCU
    139086 ±  2%     -53.8%      64278 ± 22%  softirqs.CPU102.RCU
    141535           -52.7%      66922 ± 15%  softirqs.CPU103.RCU
    145647 ±  3%     -62.2%      55051 ± 20%  softirqs.CPU104.RCU
    143618           -55.7%      63585 ± 21%  softirqs.CPU105.RCU
    142656 ±  2%     -42.5%      82020 ± 12%  softirqs.CPU106.RCU
    142621 ±  3%     -47.9%      74257 ±  4%  softirqs.CPU107.RCU
    143977           -73.0%      38851 ±  7%  softirqs.CPU108.RCU
    148186           -73.3%      39616 ± 16%  softirqs.CPU109.RCU
    148411           -66.0%      50432 ± 38%  softirqs.CPU11.RCU
    131733 ±  3%     -47.7%      68907 ± 33%  softirqs.CPU110.RCU
    135896 ±  2%     -51.8%      65445 ± 10%  softirqs.CPU111.RCU
    146748 ±  4%     -57.3%      62708 ± 12%  softirqs.CPU112.RCU
    143644 ±  2%     -54.0%      66035 ± 20%  softirqs.CPU113.RCU
    148131           -43.6%      83565 ±  6%  softirqs.CPU114.RCU
    142262 ±  5%     -51.0%      69776 ± 14%  softirqs.CPU115.RCU
    137307 ±  2%     -36.9%      86610 ± 17%  softirqs.CPU116.RCU
    141385 ±  3%     -45.0%      77750 ±  5%  softirqs.CPU117.RCU
    145208 ±  2%     -48.2%      75160 ±  2%  softirqs.CPU118.RCU
    142544 ±  2%     -43.5%      80535 ± 12%  softirqs.CPU119.RCU
    150060           -72.0%      41986 ± 11%  softirqs.CPU12.RCU
    146692           -56.6%      63652 ±  5%  softirqs.CPU120.RCU
    147222           -52.3%      70283 ± 10%  softirqs.CPU121.RCU
    143793           -55.3%      64344 ± 10%  softirqs.CPU122.RCU
    142026 ±  4%     -55.8%      62711 ±  7%  softirqs.CPU123.RCU
    143670           -43.6%      81073 ± 12%  softirqs.CPU124.RCU
    142780 ±  3%     -46.2%      76838 ± 11%  softirqs.CPU125.RCU
    133321 ±  3%     -40.2%      79755 ± 12%  softirqs.CPU126.RCU
    145916           -45.7%      79185 ±  9%  softirqs.CPU127.RCU
    136628 ±  3%     -41.6%      79814 ±  7%  softirqs.CPU128.RCU
    137059           -41.0%      80805 ± 12%  softirqs.CPU129.RCU
    149696           -63.0%      55348 ± 10%  softirqs.CPU13.RCU
    147229 ±  2%     -49.3%      74690 ±  5%  softirqs.CPU130.RCU
    147899 ±  3%     -54.4%      67418 ±  4%  softirqs.CPU131.RCU
    132512 ± 14%     -47.0%      70285 ± 28%  softirqs.CPU132.RCU
    133682 ±  2%     -52.9%      62936 ±  9%  softirqs.CPU133.RCU
    146490           -71.7%      41517 ±  5%  softirqs.CPU134.RCU
    148518 ±  3%     -72.3%      41066 ±  3%  softirqs.CPU135.RCU
    145103 ±  5%     -51.8%      69963 ± 24%  softirqs.CPU136.RCU
    149436           -62.2%      56443 ±  9%  softirqs.CPU137.RCU
    142993 ±  2%     -57.3%      60995 ± 20%  softirqs.CPU138.RCU
    147013           -58.0%      61805 ±  4%  softirqs.CPU139.RCU
    150367           -69.6%      45759 ± 21%  softirqs.CPU14.RCU
    148173           -47.1%      78334 ±  8%  softirqs.CPU140.RCU
    143437           -49.1%      73055 ±  8%  softirqs.CPU141.RCU
    139227 ±  4%     -43.4%      78863 ± 18%  softirqs.CPU142.RCU
    143866           -48.7%      73768 ±  6%  softirqs.CPU143.RCU
    142245           -47.0%      75426 ± 13%  softirqs.CPU144.RCU
    139469 ±  8%     -37.3%      87475 ± 15%  softirqs.CPU145.RCU
    143762           -68.9%      44695 ±  5%  softirqs.CPU146.RCU
    145731 ±  5%     -65.5%      50262 ± 16%  softirqs.CPU147.RCU
    139469 ±  2%     -59.9%      55960 ± 12%  softirqs.CPU148.RCU
    132710 ±  2%     -56.2%      58180 ± 15%  softirqs.CPU149.RCU
    149868           -61.2%      58162 ± 42%  softirqs.CPU15.RCU
    147153 ±  2%     -70.8%      43017 ± 23%  softirqs.CPU150.RCU
    148657           -65.8%      50882 ± 44%  softirqs.CPU151.RCU
    143775 ±  4%     -37.5%      89910 ± 15%  softirqs.CPU152.RCU
    144799 ±  3%     -49.8%      72653 ± 11%  softirqs.CPU153.RCU
    138303 ±  2%     -42.7%      79258 ± 20%  softirqs.CPU154.RCU
    138800           -42.1%      80362 ± 14%  softirqs.CPU155.RCU
    147270           -73.8%      38514 ±  5%  softirqs.CPU156.RCU
    147722 ±  3%     -72.1%      41276 ± 11%  softirqs.CPU157.RCU
    134079 ± 10%     -57.7%      56667 ± 12%  softirqs.CPU158.RCU
    133037 ±  7%     -50.8%      65481 ± 13%  softirqs.CPU159.RCU
    152164           -69.0%      47165 ± 20%  softirqs.CPU16.RCU
    144543 ±  2%     -58.0%      60684 ± 13%  softirqs.CPU160.RCU
    147968           -59.9%      59261 ± 12%  softirqs.CPU161.RCU
    148447 ±  2%     -64.2%      53086 ±  7%  softirqs.CPU162.RCU
    149657           -62.1%      56720 ±  5%  softirqs.CPU163.RCU
    137962 ±  5%     -47.7%      72115 ±  4%  softirqs.CPU164.RCU
    142832           -50.0%      71411 ±  8%  softirqs.CPU165.RCU
    143128 ±  3%     -48.4%      73878 ±  4%  softirqs.CPU166.RCU
    141600           -48.7%      72637 ±  4%  softirqs.CPU167.RCU
    147401 ±  3%     -68.9%      45899 ± 15%  softirqs.CPU168.RCU
    146766           -70.6%      43172 ± 12%  softirqs.CPU169.RCU
    147087 ±  3%     -70.1%      43940 ± 32%  softirqs.CPU17.RCU
    137946 ±  2%     -48.7%      70789 ±  2%  softirqs.CPU170.RCU
    132854 ±  5%     -43.1%      75539 ±  6%  softirqs.CPU171.RCU
    145481           -68.6%      45673 ±  8%  softirqs.CPU172.RCU
    149354           -67.4%      48727 ±  3%  softirqs.CPU173.RCU
    145270 ±  2%     -59.3%      59184 ± 16%  softirqs.CPU174.RCU
    145388 ±  4%     -59.8%      58494 ±  9%  softirqs.CPU175.RCU
    143228 ±  6%     -54.1%      65769 ± 21%  softirqs.CPU176.RCU
    136993 ±  6%     -59.3%      55694 ± 10%  softirqs.CPU177.RCU
    149681           -62.2%      56645 ±  7%  softirqs.CPU178.RCU
    147550 ±  2%     -55.4%      65788 ± 30%  softirqs.CPU179.RCU
    137404 ±  4%     -66.5%      46040 ± 17%  softirqs.CPU18.RCU
    134946 ±  5%     -43.2%      76653 ± 11%  softirqs.CPU180.RCU
    130799 ±  9%     -40.9%      77282 ± 13%  softirqs.CPU181.RCU
    146835           -50.3%      73011 ±  9%  softirqs.CPU182.RCU
    148394           -47.3%      78163 ±  5%  softirqs.CPU183.RCU
    148806           -63.0%      55081 ±  7%  softirqs.CPU184.RCU
    142254 ±  5%     -60.0%      56889 ± 12%  softirqs.CPU185.RCU
    146921           -48.3%      75892 ± 10%  softirqs.CPU186.RCU
    142122 ±  4%     -57.1%      61031 ± 11%  softirqs.CPU187.RCU
    149177           -49.4%      75525 ± 14%  softirqs.CPU188.RCU
    148051           -44.2%      82643 ± 14%  softirqs.CPU189.RCU
    149028           -75.5%      36446 ±  4%  softirqs.CPU19.RCU
    141030           -48.3%      72850 ±  6%  softirqs.CPU190.RCU
    142660 ±  3%     -42.6%      81944 ± 15%  softirqs.CPU191.RCU
    134522           -43.3%      76270 ±  4%  softirqs.CPU192.RCU
    136038 ±  2%     -35.5%      87779 ± 11%  softirqs.CPU193.RCU
    146055 ±  3%     -51.9%      70309 ±  9%  softirqs.CPU194.RCU
    146754 ±  2%     -72.7%      40050 ± 17%  softirqs.CPU195.RCU
    135458 ±  3%     -57.1%      58154 ± 21%  softirqs.CPU196.RCU
    146622           -61.9%      55807 ±  9%  softirqs.CPU197.RCU
    148294 ±  2%     -55.8%      65572 ± 32%  softirqs.CPU198.RCU
    150500           -59.3%      61199 ± 10%  softirqs.CPU199.RCU
    144541 ±  2%     -73.1%      38859 ± 12%  softirqs.CPU2.RCU
    150631           -75.9%      36230 ±  5%  softirqs.CPU20.RCU
    146123           -51.9%      70303 ± 15%  softirqs.CPU200.RCU
    147081           -51.4%      71416 ± 14%  softirqs.CPU201.RCU
    141476 ±  2%     -53.6%      65702 ± 14%  softirqs.CPU202.RCU
    136300 ±  5%     -53.2%      63846 ±  7%  softirqs.CPU203.RCU
    146788 ±  2%     -48.6%      75440 ±  4%  softirqs.CPU204.RCU
    148322           -47.1%      78433 ± 10%  softirqs.CPU205.RCU
    138706 ±  3%     -45.5%      75583 ±  5%  softirqs.CPU206.RCU
    137361           -45.0%      75562 ±  3%  softirqs.CPU207.RCU
    138359           -47.0%      73342 ±  2%  softirqs.CPU208.RCU
    136332           -45.0%      74917 ±  3%  softirqs.CPU209.RCU
    150277           -73.0%      40538 ±  6%  softirqs.CPU21.RCU
    145124 ±  3%     -47.9%      75589 ±  6%  softirqs.CPU210.RCU
    146271           -71.6%      41471 ± 13%  softirqs.CPU211.RCU
    139056 ±  2%     -46.7%      74081 ± 18%  softirqs.CPU212.RCU
    142892           -46.3%      76733 ± 23%  softirqs.CPU213.RCU
    146304 ±  2%     -68.4%      46184 ±  3%  softirqs.CPU214.RCU
    143095           -63.6%      52118 ± 15%  softirqs.CPU215.RCU
    147312           -56.8%      63666 ±  4%  softirqs.CPU216.RCU
    143855 ±  2%     -74.1%      37305 ±  8%  softirqs.CPU217.RCU
    137220 ±  2%     -42.2%      79354 ±  8%  softirqs.CPU218.RCU
    140877 ±  2%     -44.6%      78032 ±  9%  softirqs.CPU219.RCU
    149192           -75.1%      37180 ±  9%  softirqs.CPU22.RCU
    148680           -63.8%      53862 ±  9%  softirqs.CPU220.RCU
    149030           -59.0%      61029 ± 24%  softirqs.CPU221.RCU
    129255 ±  5%     -54.2%      59214 ±  8%  softirqs.CPU222.RCU
    141081 ±  3%     -57.2%      60382 ±  4%  softirqs.CPU223.RCU
    141349 ±  2%     -52.5%      67175 ± 16%  softirqs.CPU224.RCU
    140947           -51.2%      68741 ± 26%  softirqs.CPU225.RCU
    142097           -56.7%      61538 ±  9%  softirqs.CPU226.RCU
    141631           -55.9%      62393 ±  9%  softirqs.CPU227.RCU
    138989 ±  5%     -44.5%      77152 ±  4%  softirqs.CPU228.RCU
    142158           -46.3%      76380 ±  2%  softirqs.CPU229.RCU
    149213           -74.0%      38726 ± 11%  softirqs.CPU23.RCU
    148070           -45.3%      80985 ±  3%  softirqs.CPU230.RCU
    147054           -47.0%      77910        softirqs.CPU231.RCU
    132579           -51.7%      63982 ± 22%  softirqs.CPU232.RCU
    139684           -54.5%      63535 ± 16%  softirqs.CPU233.RCU
    143487 ±  5%     -72.6%      39378 ± 11%  softirqs.CPU234.RCU
    142380           -67.3%      46619 ± 27%  softirqs.CPU235.RCU
    143210 ±  3%     -45.7%      77828 ±  7%  softirqs.CPU236.RCU
    142725 ±  2%     -49.5%      72074 ±  7%  softirqs.CPU237.RCU
    144390 ±  2%     -64.4%      51394 ±  2%  softirqs.CPU238.RCU
    138946 ±  4%     -64.9%      48756 ±  7%  softirqs.CPU239.RCU
    139948 ±  6%     -66.7%      46574 ± 22%  softirqs.CPU24.RCU
    139319 ±  3%     -46.0%      75238 ± 11%  softirqs.CPU240.RCU
    145038 ±  4%     -44.2%      80861 ±  5%  softirqs.CPU241.RCU
    139431 ±  4%     -41.8%      81101 ± 11%  softirqs.CPU242.RCU
    142230 ±  2%     -44.9%      78421 ±  7%  softirqs.CPU243.RCU
    146946           -52.7%      69526 ± 26%  softirqs.CPU244.RCU
    143217 ±  2%     -55.5%      63802 ± 20%  softirqs.CPU245.RCU
    142411           -56.5%      61903 ± 14%  softirqs.CPU246.RCU
    141483 ±  5%     -59.1%      57856 ± 14%  softirqs.CPU247.RCU
    142446           -46.3%      76539 ± 10%  softirqs.CPU248.RCU
    142154 ±  2%     -41.4%      83313 ±  6%  softirqs.CPU249.RCU
    150478           -75.0%      37572 ±  4%  softirqs.CPU25.RCU
    139540 ±  3%     -44.7%      77146 ±  7%  softirqs.CPU250.RCU
    146768           -46.4%      78658 ±  6%  softirqs.CPU251.RCU
    144365           -59.0%      59236 ±  8%  softirqs.CPU252.RCU
    139798 ±  3%     -45.3%      76470 ± 21%  softirqs.CPU253.RCU
    146669           -58.7%      60580 ± 11%  softirqs.CPU254.RCU
    142679           -56.8%      61582 ± 14%  softirqs.CPU255.RCU
    147315 ±  2%     -69.7%      44583 ± 31%  softirqs.CPU256.RCU
    143467           -75.1%      35694 ±  3%  softirqs.CPU257.RCU
    133596 ±  2%     -46.6%      71299 ±  8%  softirqs.CPU258.RCU
    130651 ±  4%     -42.0%      75759 ± 23%  softirqs.CPU259.RCU
    150875           -69.9%      45352 ± 40%  softirqs.CPU26.RCU
    145742 ±  2%     -50.0%      72835 ± 17%  softirqs.CPU260.RCU
    147088           -47.0%      77954 ±  9%  softirqs.CPU261.RCU
    136092 ±  2%     -42.5%      78240 ±  5%  softirqs.CPU262.RCU
    137450 ±  2%     -47.0%      72861 ±  8%  softirqs.CPU263.RCU
    147266           -67.4%      48024 ± 11%  softirqs.CPU264.RCU
    150209 ±  7%     -58.9%      61693 ± 17%  softirqs.CPU265.RCU
    138347 ±  2%     -64.4%      49208 ± 12%  softirqs.CPU266.RCU
    152793 ±  2%     -68.8%      47730 ±  8%  softirqs.CPU267.RCU
    145801           -67.8%      47013 ± 20%  softirqs.CPU268.RCU
    138275 ±  6%     -66.3%      46622 ± 25%  softirqs.CPU269.RCU
    148179 ±  3%     -73.9%      38702 ±  2%  softirqs.CPU27.RCU
    144943           -70.5%      42692 ± 17%  softirqs.CPU270.RCU
    144114 ±  2%     -65.2%      50135 ± 15%  softirqs.CPU271.RCU
    143648 ±  2%     -59.7%      57848 ± 25%  softirqs.CPU272.RCU
    136667 ±  9%     -60.0%      54600 ± 42%  softirqs.CPU273.RCU
    134889 ±  4%     -66.8%      44719 ± 15%  softirqs.CPU274.RCU
    141352 ±  2%     -66.7%      47078 ± 17%  softirqs.CPU275.RCU
    149548 ±  3%     -72.8%      40747 ±  5%  softirqs.CPU276.RCU
    144559           -68.6%      45419 ± 11%  softirqs.CPU277.RCU
    152609 ±  3%     -65.5%      52614 ± 21%  softirqs.CPU278.RCU
    144128           -69.2%      44404 ±  5%  softirqs.CPU279.RCU
    149528 ±  2%     -72.2%      41632 ± 19%  softirqs.CPU28.RCU
    138839 ±  2%     -72.9%      37575 ±  5%  softirqs.CPU280.RCU
    144458           -72.4%      39807 ±  6%  softirqs.CPU281.RCU
    142443 ±  2%     -65.3%      49439 ± 18%  softirqs.CPU282.RCU
    144386           -63.1%      53283 ± 38%  softirqs.CPU283.RCU
    149196           -63.1%      55034 ± 15%  softirqs.CPU284.RCU
    142301           -66.0%      48379 ±  9%  softirqs.CPU285.RCU
    150683           -67.8%      48488 ± 17%  softirqs.CPU286.RCU
    130981 ±  3%     -57.6%      55596 ± 14%  softirqs.CPU287.RCU
    152743 ±  2%     -75.2%      37816 ±  8%  softirqs.CPU29.RCU
    146420 ±  2%     -71.4%      41827 ±  7%  softirqs.CPU3.RCU
    149996           -61.5%      57714 ± 53%  softirqs.CPU30.RCU
    150934           -65.8%      51629 ± 24%  softirqs.CPU31.RCU
    150459 ±  2%     -75.8%      36486 ±  2%  softirqs.CPU32.RCU
    149892           -69.8%      45277 ± 32%  softirqs.CPU33.RCU
    149804 ±  5%     -66.8%      49681 ± 36%  softirqs.CPU34.RCU
    137898 ±  5%     -74.2%      35608 ±  3%  softirqs.CPU35.RCU
    152001 ±  2%     -74.9%      38088 ± 10%  softirqs.CPU36.RCU
    148600 ±  2%     -69.7%      45067 ± 22%  softirqs.CPU37.RCU
    150712 ±  5%     -72.6%      41356 ± 21%  softirqs.CPU38.RCU
    148916           -48.6%      76615 ± 32%  softirqs.CPU39.RCU
    151563 ±  3%     -72.7%      41428 ±  5%  softirqs.CPU4.RCU
    147078           -68.2%      46806 ± 29%  softirqs.CPU40.RCU
    147330 ±  5%     -62.1%      55909 ± 34%  softirqs.CPU41.RCU
    146539           -59.0%      60100 ± 18%  softirqs.CPU42.RCU
    143115 ±  2%     -66.1%      48457 ± 22%  softirqs.CPU43.RCU
    145713 ±  2%     -62.2%      55128 ± 25%  softirqs.CPU44.RCU
    150433           -67.1%      49527 ± 11%  softirqs.CPU45.RCU
    148077           -73.6%      39126 ±  4%  softirqs.CPU46.RCU
    158449 ±  5%     -53.1%      74380 ±  9%  softirqs.CPU47.RCU
    152460           -65.6%      52407 ±  7%  softirqs.CPU48.RCU
    148122           -62.2%      56012 ± 19%  softirqs.CPU49.RCU
    149452           -73.2%      40044 ± 17%  softirqs.CPU5.RCU
    148740 ±  3%     -62.4%      55890 ± 15%  softirqs.CPU50.RCU
    146229           -64.5%      51882 ± 23%  softirqs.CPU51.RCU
    146880 ±  2%     -71.0%      42540 ±  6%  softirqs.CPU52.RCU
    152004 ±  3%     -70.9%      44225 ± 10%  softirqs.CPU53.RCU
    137981 ±  3%     -65.5%      47653 ± 13%  softirqs.CPU54.RCU
    144774           -58.7%      59763 ± 37%  softirqs.CPU55.RCU
    152584 ±  2%     -69.0%      47365 ± 15%  softirqs.CPU56.RCU
    145585 ±  2%     -58.7%      60148 ± 28%  softirqs.CPU57.RCU
    152466           -62.7%      56864 ±  7%  softirqs.CPU58.RCU
    148066           -65.8%      50694 ± 10%  softirqs.CPU59.RCU
    147809 ±  2%     -71.2%      42539 ± 24%  softirqs.CPU6.RCU
    139054 ±  4%     -69.7%      42144 ± 16%  softirqs.CPU60.RCU
    146609           -65.6%      50385 ± 13%  softirqs.CPU61.RCU
    140736 ±  4%     -70.2%      41911 ±  4%  softirqs.CPU62.RCU
    150042 ±  4%     -72.9%      40669 ± 10%  softirqs.CPU63.RCU
    149494           -69.7%      45283 ± 18%  softirqs.CPU64.RCU
    146783 ±  4%     -67.3%      48007 ± 15%  softirqs.CPU65.RCU
    150611           -68.2%      47965 ± 13%  softirqs.CPU66.RCU
    139234           -67.4%      45407 ± 12%  softirqs.CPU67.RCU
    148960 ±  3%     -69.7%      45081 ± 13%  softirqs.CPU68.RCU
    151217           -49.4%      76556 ±  8%  softirqs.CPU69.RCU
    149167           -70.5%      44038 ± 10%  softirqs.CPU7.RCU
    141711 ±  3%     -66.5%      47481 ±  5%  softirqs.CPU70.RCU
    150495           -64.0%      54186 ±  3%  softirqs.CPU71.RCU
    144744 ±  3%     -44.2%      80838 ±  8%  softirqs.CPU72.RCU
    136696 ±  2%     -38.0%      84779 ± 20%  softirqs.CPU73.RCU
    149733           -46.8%      79645 ±  5%  softirqs.CPU74.RCU
    147438 ±  2%     -46.8%      78397 ± 17%  softirqs.CPU75.RCU
    133887 ±  9%     -54.9%      60372 ± 20%  softirqs.CPU76.RCU
    143925 ±  2%     -48.1%      74722 ± 26%  softirqs.CPU77.RCU
    146567           -59.8%      58886 ± 16%  softirqs.CPU78.RCU
    147427           -57.3%      62883 ± 10%  softirqs.CPU79.RCU
    150664           -68.1%      48076 ± 27%  softirqs.CPU8.RCU
    146030           -63.8%      52917 ± 18%  softirqs.CPU80.RCU
    139080 ±  2%     -64.0%      50007 ± 24%  softirqs.CPU81.RCU
    144590 ±  2%     -43.9%      81176 ± 17%  softirqs.CPU82.RCU
    141457 ±  2%     -48.2%      73232 ± 21%  softirqs.CPU83.RCU
    146686 ±  2%     -73.7%      38572 ±  8%  softirqs.CPU84.RCU
    149314           -61.9%      56963 ± 32%  softirqs.CPU85.RCU
    136532 ±  5%     -45.9%      73883 ±  8%  softirqs.CPU86.RCU
    136856 ±  6%     -42.2%      79101 ±  6%  softirqs.CPU87.RCU
    147431           -53.4%      68679 ± 17%  softirqs.CPU88.RCU
    128555 ±  5%     -50.4%      63740 ± 19%  softirqs.CPU89.RCU
    150064           -67.4%      48899 ± 33%  softirqs.CPU9.RCU
    149543           -59.9%      59930 ± 11%  softirqs.CPU90.RCU
    148388 ±  3%     -60.4%      58758 ± 21%  softirqs.CPU91.RCU
    134059 ±  7%     -45.0%      73673 ±  9%  softirqs.CPU92.RCU
    139940           -44.7%      77438 ±  6%  softirqs.CPU93.RCU
    145683           -72.5%      40029 ±  7%  softirqs.CPU94.RCU
    149777           -47.4%      78782 ± 18%  softirqs.CPU95.RCU
    149632           -45.1%      82115 ±  2%  softirqs.CPU96.RCU
    146641 ±  2%     -46.2%      78878 ± 13%  softirqs.CPU97.RCU
    146629           -43.5%      82815 ± 13%  softirqs.CPU98.RCU
    138969 ±  6%     -42.3%      80249 ±  5%  softirqs.CPU99.RCU
  41531880           -58.0%   17433667 ±  3%  softirqs.RCU


                                                                                
                            will-it-scale.per_process_ops                       
                                                                                
  80000 +-------------------------------------------------------------------+   
        |                                                                   |   
  70000 |.+ .+.+..+.    .+.  .+.+..+.+.. .+.+..+.+..+.+..                   |   
  60000 |-+.        +..+   +.           +                +                  |   
        |                                                                   |   
  50000 |-+                                                                 |   
        |                                                                   |   
  40000 |-+                                                                 |   
        |                                                                   |   
  30000 |-+                                                                 |   
  20000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                will-it-scale.workload                          
                                                                                
  2.2e+07 +-----------------------------------------------------------------+   
    2e+07 |-+   .+.+..           .+.+.      .+..+. .+..                     |   
          |.+..+      +.+..+.+.+.     +..+.+      +    +.+                  |   
  1.8e+07 |-+                                                               |   
  1.6e+07 |-+                                                               |   
  1.4e+07 |-+                                                               |   
  1.2e+07 |-+                                                               |   
          |                                                                 |   
    1e+07 |-+                                                               |   
    8e+06 |-+                                                               |   
    6e+06 |-+                                                               |   
    4e+06 |-+                                                               |   
          |                                                                 |   
    2e+06 |-+                                                               |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/lock1/will-it-scale/0x11

commit: 
  0a68ff5e2e ("fcntl: Distribute switch variables for initialization")
  6d390e4b5d ("locks: fix a potential use-after-free problem when wakeup a waiter")

0a68ff5e2e7cf226 6d390e4b5d48ec03bb87e63cf0a 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           50%           2:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          2:4          -50%            :4     dmesg.WARNING:at_ip__fsnotify_parent/0x
         %stddev     %change         %stddev
             \          |                \  
      4555           -51.3%       2218 ±  3%  will-it-scale.per_thread_ops
    760.80 ±  6%     -26.1%     561.95 ±  7%  will-it-scale.time.user_time
   1312253           -51.3%     639028 ±  3%  will-it-scale.workload
    153716           +10.6%     170015 ±  2%  meminfo.Shmem
      1.01 ±  5%      -0.2        0.79 ±  5%  mpstat.cpu.all.usr%
      6359 ±  5%     -11.1%       5656 ±  3%  sched_debug.cpu.curr->pid.max
    153760           +10.5%     169943 ±  2%  numa-meminfo.node0.Shmem
     11621 ± 13%     +19.8%      13920 ±  9%  numa-meminfo.node1.SUnreclaim
     38459           +10.6%      42548 ±  2%  numa-vmstat.node0.nr_shmem
      2905 ± 13%     +19.8%       3480 ±  9%  numa-vmstat.node1.nr_slab_unreclaimable
      2228 ±  2%     +13.6%       2530 ±  6%  slabinfo.UNIX.active_objs
      2228 ±  2%     +13.6%       2530 ±  6%  slabinfo.UNIX.num_objs
    160690            +2.4%     164524        proc-vmstat.nr_active_anon
    306982            +1.3%     311087        proc-vmstat.nr_file_pages
     38387           +10.7%      42492 ±  2%  proc-vmstat.nr_shmem
    160690            +2.4%     164524        proc-vmstat.nr_zone_active_anon
      4048 ±  2%     +13.8%       4607 ±  3%  proc-vmstat.pgactivate
      5.69 ±  2%      -5.1%       5.40 ±  2%  perf-stat.i.MPKI
 8.616e+09            -2.7%   8.38e+09        perf-stat.i.branch-instructions
      1.03            -0.2        0.81        perf-stat.i.branch-miss-rate%
  86798208           -24.9%   65209294        perf-stat.i.branch-misses
     20.35            +2.2       22.56        perf-stat.i.cache-miss-rate%
  40221638            +1.5%   40843608        perf-stat.i.cache-misses
 1.977e+08 ±  2%      -8.4%  1.812e+08 ±  2%  perf-stat.i.cache-references
     12.59            +3.4%      13.01        perf-stat.i.cpi
     11067            -1.6%      10894        perf-stat.i.cycles-between-cache-misses
      0.21            -0.0        0.19        perf-stat.i.iTLB-load-miss-rate%
  74059260           -13.5%   64033307        perf-stat.i.iTLB-load-misses
 3.541e+10            -3.3%  3.424e+10        perf-stat.i.iTLB-loads
 3.536e+10            -3.3%  3.419e+10        perf-stat.i.instructions
    477.71           +11.9%     534.66        perf-stat.i.instructions-per-iTLB-miss
      0.08            -3.2%       0.08        perf-stat.i.ipc
      1.00            -0.2        0.77        perf-stat.overall.branch-miss-rate%
     20.34            +2.2       22.52 ±  2%  perf-stat.overall.cache-miss-rate%
     12.61            +3.4%      13.03        perf-stat.overall.cpi
     11066            -1.5%      10895        perf-stat.overall.cycles-between-cache-misses
      0.21            -0.0        0.19        perf-stat.overall.iTLB-load-miss-rate%
    477.71           +11.9%     534.68        perf-stat.overall.instructions-per-iTLB-miss
      0.08            -3.3%       0.08        perf-stat.overall.ipc
   8253689           +98.9%   16418473 ±  2%  perf-stat.overall.path-length
 8.596e+09            -2.7%  8.362e+09        perf-stat.ps.branch-instructions
  86094437           -25.0%   64597658        perf-stat.ps.branch-misses
  40178634            +1.5%   40798401        perf-stat.ps.cache-misses
 1.976e+08 ±  2%      -8.3%  1.812e+08 ±  2%  perf-stat.ps.cache-references
  73832341           -13.6%   63794241        perf-stat.ps.iTLB-load-misses
  3.53e+10            -3.3%  3.413e+10        perf-stat.ps.iTLB-loads
 3.527e+10            -3.3%  3.411e+10        perf-stat.ps.instructions
 1.083e+13            -3.2%  1.048e+13        perf-stat.total.instructions
     96.22           -96.2        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock.fcntl_setlk.do_fcntl.__x64_sys_fcntl.do_syscall_64
     96.11           -96.1        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.fcntl_setlk.do_fcntl.__x64_sys_fcntl
      0.94            -0.6        0.39 ± 57%  perf-profile.calltrace.cycles-pp.posix_lock_inode.do_lock_file_wait.fcntl_setlk.do_fcntl.__x64_sys_fcntl
     98.41            +0.4       98.78        perf-profile.calltrace.cycles-pp.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe.do_fcntl
     98.14            +0.5       98.67        perf-profile.calltrace.cycles-pp.fcntl_setlk.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +4.8        4.82 ± 29%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.locks_delete_block
      0.00            +5.1        5.06 ± 28%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.locks_delete_block.do_lock_file_wait
      1.19           +97.1       98.29        perf-profile.calltrace.cycles-pp.do_lock_file_wait.fcntl_setlk.do_fcntl.__x64_sys_fcntl.do_syscall_64
      0.00           +97.3       97.27        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.locks_delete_block.do_lock_file_wait.fcntl_setlk
      0.00           +97.4       97.39        perf-profile.calltrace.cycles-pp._raw_spin_lock.locks_delete_block.do_lock_file_wait.fcntl_setlk.do_fcntl
      0.00           +97.6       97.64        perf-profile.calltrace.cycles-pp.locks_delete_block.do_lock_file_wait.fcntl_setlk.do_fcntl.__x64_sys_fcntl
      0.95            -0.4        0.52 ±  6%  perf-profile.children.cycles-pp.posix_lock_inode
      0.60            -0.3        0.30 ±  4%  perf-profile.children.cycles-pp.locks_alloc_lock
      0.49            -0.3        0.23 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.20 ±  2%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
      0.18 ±  6%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.security_file_lock
      0.18 ±  4%      -0.1        0.09 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.12 ±  3%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.locks_delete_lock_ctx
      0.14            -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.memset_erms
      0.15 ±  8%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.common_file_perm
      0.13 ±  3%      -0.1        0.06 ± 13%  perf-profile.children.cycles-pp.___might_sleep
      0.15 ±  3%      -0.1        0.09 ± 10%  perf-profile.children.cycles-pp._copy_from_user
      0.25 ±  4%      -0.1        0.19 ± 16%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.09 ±  5%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.__fget_files
      0.12 ±  3%      -0.0        0.08 ± 11%  perf-profile.children.cycles-pp.__fget_light
      0.11            -0.0        0.07        perf-profile.children.cycles-pp.__libc_fcntl
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.__locks_wake_up_blocks
     99.20            +0.2       99.40        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     99.16            +0.2       99.38        perf-profile.children.cycles-pp.do_syscall_64
     98.22            +0.4       98.67        perf-profile.children.cycles-pp.fcntl_setlk
     96.40            +1.2       97.57        perf-profile.children.cycles-pp._raw_spin_lock
     96.22            +1.2       97.42        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.19           +97.1       98.29        perf-profile.children.cycles-pp.do_lock_file_wait
      0.00           +97.6       97.64        perf-profile.children.cycles-pp.locks_delete_block
      0.30 ±  4%      -0.2        0.11 ± 10%  perf-profile.self.cycles-pp.fcntl_setlk
      0.18 ±  2%      -0.1        0.09 ±  7%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.19            -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.kmem_cache_free
      0.15 ±  4%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.12 ±  4%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.memset_erms
      0.15            -0.1        0.09 ±  5%  perf-profile.self.cycles-pp.posix_lock_inode
      0.12            -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.___might_sleep
      0.25 ±  4%      -0.1        0.19 ± 17%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.14 ±  3%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.do_fcntl
      0.08 ±  5%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.__fget_files
      0.11            -0.0        0.07 ± 14%  perf-profile.self.cycles-pp.do_syscall_64
      0.09 ±  4%      -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.locks_alloc_lock
      0.00            +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.__locks_wake_up_blocks
      5550 ± 33%     -33.8%       3672        interrupts.CPU102.NMI:Non-maskable_interrupts
      5550 ± 33%     -33.8%       3672        interrupts.CPU102.PMI:Performance_monitoring_interrupts
      5548 ± 33%     -33.7%       3677        interrupts.CPU103.NMI:Non-maskable_interrupts
      5548 ± 33%     -33.7%       3677        interrupts.CPU103.PMI:Performance_monitoring_interrupts
      5544 ± 33%     -17.3%       4586 ± 34%  interrupts.CPU105.NMI:Non-maskable_interrupts
      5544 ± 33%     -17.3%       4586 ± 34%  interrupts.CPU105.PMI:Performance_monitoring_interrupts
    236.00 ±  2%     +28.8%     304.00 ± 20%  interrupts.CPU11.RES:Rescheduling_interrupts
    115.00 ± 84%     -82.4%      20.25 ±110%  interrupts.CPU111.RES:Rescheduling_interrupts
      5545 ± 33%     -33.7%       3678        interrupts.CPU112.NMI:Non-maskable_interrupts
      5545 ± 33%     -33.7%       3678        interrupts.CPU112.PMI:Performance_monitoring_interrupts
      6470 ± 24%     -29.1%       4587 ± 34%  interrupts.CPU121.NMI:Non-maskable_interrupts
      6470 ± 24%     -29.1%       4587 ± 34%  interrupts.CPU121.PMI:Performance_monitoring_interrupts
      5548 ± 33%     -17.1%       4598 ± 34%  interrupts.CPU122.NMI:Non-maskable_interrupts
      5548 ± 33%     -17.1%       4598 ± 34%  interrupts.CPU122.PMI:Performance_monitoring_interrupts
      6455 ± 24%     -29.0%       4581 ± 34%  interrupts.CPU129.NMI:Non-maskable_interrupts
      6455 ± 24%     -29.0%       4581 ± 34%  interrupts.CPU129.PMI:Performance_monitoring_interrupts
      5535 ± 33%     -17.0%       4593 ± 34%  interrupts.CPU131.NMI:Non-maskable_interrupts
      5535 ± 33%     -17.0%       4593 ± 34%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
      5546 ± 33%     -33.7%       3678        interrupts.CPU135.NMI:Non-maskable_interrupts
      5546 ± 33%     -33.7%       3678        interrupts.CPU135.PMI:Performance_monitoring_interrupts
      6451 ± 24%     -28.9%       4587 ± 34%  interrupts.CPU136.NMI:Non-maskable_interrupts
      6451 ± 24%     -28.9%       4587 ± 34%  interrupts.CPU136.PMI:Performance_monitoring_interrupts
     71.50 ±129%     -88.5%       8.25 ± 40%  interrupts.CPU136.RES:Rescheduling_interrupts
      5524 ± 33%     -17.0%       4586 ± 34%  interrupts.CPU137.NMI:Non-maskable_interrupts
      5524 ± 33%     -17.0%       4586 ± 34%  interrupts.CPU137.PMI:Performance_monitoring_interrupts
    108.50 ±135%     -93.5%       7.00        interrupts.CPU137.RES:Rescheduling_interrupts
      6464 ± 24%     -28.9%       4596 ± 34%  interrupts.CPU138.NMI:Non-maskable_interrupts
      6464 ± 24%     -28.9%       4596 ± 34%  interrupts.CPU138.PMI:Performance_monitoring_interrupts
      5537 ± 33%     -17.2%       4584 ± 34%  interrupts.CPU141.NMI:Non-maskable_interrupts
      5537 ± 33%     -17.2%       4584 ± 34%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
    148.25 ±110%     -93.8%       9.25 ± 29%  interrupts.CPU152.RES:Rescheduling_interrupts
      5534 ± 33%     -33.6%       3673        interrupts.CPU156.NMI:Non-maskable_interrupts
      5534 ± 33%     -33.6%       3673        interrupts.CPU156.PMI:Performance_monitoring_interrupts
      7360           -37.7%       4584 ± 34%  interrupts.CPU169.NMI:Non-maskable_interrupts
      7360           -37.7%       4584 ± 34%  interrupts.CPU169.PMI:Performance_monitoring_interrupts
    114.25 ±115%     -74.2%      29.50 ±118%  interrupts.CPU170.RES:Rescheduling_interrupts
     38.00 ±115%     -75.7%       9.25 ± 11%  interrupts.CPU177.RES:Rescheduling_interrupts
      7367           -50.3%       3663        interrupts.CPU182.NMI:Non-maskable_interrupts
      7367           -50.3%       3663        interrupts.CPU182.PMI:Performance_monitoring_interrupts
      4597 ± 34%     +59.7%       7343        interrupts.CPU185.NMI:Non-maskable_interrupts
      4597 ± 34%     +59.7%       7343        interrupts.CPU185.PMI:Performance_monitoring_interrupts
    110.00 ±134%     -91.1%       9.75 ± 25%  interrupts.CPU209.RES:Rescheduling_interrupts
      5514 ± 33%     -17.1%       4572 ± 34%  interrupts.CPU211.NMI:Non-maskable_interrupts
      5514 ± 33%     -17.1%       4572 ± 34%  interrupts.CPU211.PMI:Performance_monitoring_interrupts
      5646 ± 33%     -33.5%       3755        interrupts.CPU22.NMI:Non-maskable_interrupts
      5646 ± 33%     -33.5%       3755        interrupts.CPU22.PMI:Performance_monitoring_interrupts
      5518 ± 33%     -17.1%       4573 ± 34%  interrupts.CPU228.NMI:Non-maskable_interrupts
      5518 ± 33%     -17.1%       4573 ± 34%  interrupts.CPU228.PMI:Performance_monitoring_interrupts
      5642 ± 33%     -16.9%       4688 ± 34%  interrupts.CPU23.NMI:Non-maskable_interrupts
      5642 ± 33%     -16.9%       4688 ± 34%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      6438 ± 24%     -29.1%       4568 ± 34%  interrupts.CPU236.NMI:Non-maskable_interrupts
      6438 ± 24%     -29.1%       4568 ± 34%  interrupts.CPU236.PMI:Performance_monitoring_interrupts
      5524 ± 33%     -33.8%       3659        interrupts.CPU241.NMI:Non-maskable_interrupts
      5524 ± 33%     -33.8%       3659        interrupts.CPU241.PMI:Performance_monitoring_interrupts
     69.50 ±135%     -87.8%       8.50 ± 13%  interrupts.CPU256.RES:Rescheduling_interrupts
      5513 ± 33%     -17.4%       4556 ± 33%  interrupts.CPU261.NMI:Non-maskable_interrupts
      5513 ± 33%     -17.4%       4556 ± 33%  interrupts.CPU261.PMI:Performance_monitoring_interrupts
      6428 ± 24%     -43.2%       3654        interrupts.CPU263.NMI:Non-maskable_interrupts
      6428 ± 24%     -43.2%       3654        interrupts.CPU263.PMI:Performance_monitoring_interrupts
      5543 ± 33%     -33.8%       3670        interrupts.CPU264.NMI:Non-maskable_interrupts
      5543 ± 33%     -33.8%       3670        interrupts.CPU264.PMI:Performance_monitoring_interrupts
     44.50 ±121%     -81.5%       8.25 ± 15%  interrupts.CPU267.RES:Rescheduling_interrupts
      3754           +98.2%       7440        interrupts.CPU35.NMI:Non-maskable_interrupts
      3754           +98.2%       7440        interrupts.CPU35.PMI:Performance_monitoring_interrupts
      5605 ± 33%     -33.9%       3707        interrupts.CPU42.NMI:Non-maskable_interrupts
      5605 ± 33%     -33.9%       3707        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      5576 ± 32%     -17.3%       4610 ± 34%  interrupts.CPU44.NMI:Non-maskable_interrupts
      5576 ± 32%     -17.3%       4610 ± 34%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      7394           -50.1%       3693        interrupts.CPU56.NMI:Non-maskable_interrupts
      7394           -50.1%       3693        interrupts.CPU56.PMI:Performance_monitoring_interrupts
      7418           -37.6%       4626 ± 34%  interrupts.CPU61.NMI:Non-maskable_interrupts
      7418           -37.6%       4626 ± 34%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      7.25 ± 31%   +1051.7%      83.50 ± 62%  interrupts.CPU62.RES:Rescheduling_interrupts
      6485 ± 24%     -28.8%       4619 ± 34%  interrupts.CPU65.NMI:Non-maskable_interrupts
      6485 ± 24%     -28.8%       4619 ± 34%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
    134.75 ±105%     -93.5%       8.75 ± 16%  interrupts.CPU65.RES:Rescheduling_interrupts
      6487 ± 24%     -28.9%       4613 ± 34%  interrupts.CPU68.NMI:Non-maskable_interrupts
      6487 ± 24%     -28.9%       4613 ± 34%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
      6498 ± 24%     -43.2%       3691        interrupts.CPU69.NMI:Non-maskable_interrupts
      6498 ± 24%     -43.2%       3691        interrupts.CPU69.PMI:Performance_monitoring_interrupts
      5550 ± 33%     -17.0%       4607 ± 34%  interrupts.CPU80.NMI:Non-maskable_interrupts
      5550 ± 33%     -17.0%       4607 ± 34%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      5547 ± 33%     -33.6%       3683        interrupts.CPU81.NMI:Non-maskable_interrupts
      5547 ± 33%     -33.6%       3683        interrupts.CPU81.PMI:Performance_monitoring_interrupts
      5545 ± 33%     -17.1%       4597 ± 34%  interrupts.CPU84.NMI:Non-maskable_interrupts
      5545 ± 33%     -17.1%       4597 ± 34%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
      5558 ± 33%     -17.1%       4607 ± 34%  interrupts.CPU90.NMI:Non-maskable_interrupts
      5558 ± 33%     -17.1%       4607 ± 34%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      5550 ± 33%     -33.7%       3679        interrupts.CPU95.NMI:Non-maskable_interrupts
      5550 ± 33%     -33.7%       3679        interrupts.CPU95.PMI:Performance_monitoring_interrupts
     54559 ±  3%     -22.4%      42328 ±  7%  softirqs.CPU104.RCU
     65316 ± 16%     -25.4%      48704        softirqs.CPU105.RCU
     49154 ±  3%     -26.3%      36235 ±  4%  softirqs.CPU108.RCU
     48474 ±  4%     -20.4%      38585 ±  8%  softirqs.CPU109.RCU
     56182 ±  9%     -24.4%      42449 ± 10%  softirqs.CPU11.RCU
     69518 ±  7%     -17.7%      57209 ±  9%  softirqs.CPU114.RCU
     70411 ±  7%     -16.9%      58494 ± 14%  softirqs.CPU115.RCU
     82704 ±  7%     -11.9%      72902 ±  5%  softirqs.CPU117.RCU
     49949 ±  5%     -24.5%      37732 ±  6%  softirqs.CPU12.RCU
     71866 ± 12%     -21.0%      56762 ± 10%  softirqs.CPU120.RCU
     69964 ±  9%     -16.7%      58296 ± 13%  softirqs.CPU121.RCU
     74258 ± 11%     -19.0%      60167 ± 16%  softirqs.CPU123.RCU
     64097 ±  4%     -22.3%      49819 ±  4%  softirqs.CPU13.RCU
     71335 ± 11%     -18.4%      58212 ± 13%  softirqs.CPU132.RCU
     47661 ±  3%     -19.2%      38499 ±  7%  softirqs.CPU134.RCU
     71835 ±  9%     -17.8%      59082 ±  6%  softirqs.CPU136.RCU
     73184 ± 12%     -20.0%      58549 ± 12%  softirqs.CPU137.RCU
     50549 ±  5%     -27.2%      36822 ±  5%  softirqs.CPU14.RCU
     87211 ±  5%     -11.5%      77224 ±  8%  softirqs.CPU143.RCU
     58973 ± 18%     -37.5%      36832 ±  4%  softirqs.CPU15.RCU
     54512 ±  9%     -23.5%      41675 ±  6%  softirqs.CPU150.RCU
     48970 ±  4%     -14.9%      41693 ±  7%  softirqs.CPU151.RCU
     51441 ±  8%     -25.4%      38354 ±  4%  softirqs.CPU156.RCU
     49719 ±  7%     -21.0%      39301 ±  3%  softirqs.CPU157.RCU
     51413 ±  5%     -27.3%      37356 ±  2%  softirqs.CPU16.RCU
     69966 ±  6%     -14.7%      59665 ± 11%  softirqs.CPU160.RCU
     47327 ±  4%     -25.4%      35304 ±  3%  softirqs.CPU169.RCU
     49442 ±  4%     -28.5%      35364 ±  4%  softirqs.CPU17.RCU
     54983 ±  2%     -13.7%      47425 ±  4%  softirqs.CPU172.RCU
     71486 ± 13%     -18.8%      58033 ±  8%  softirqs.CPU175.RCU
     78719 ± 26%     -24.9%      59143 ± 10%  softirqs.CPU176.RCU
     66339 ±  6%     -15.1%      56337 ± 10%  softirqs.CPU177.RCU
     70598 ±  5%     -12.9%      61514 ±  4%  softirqs.CPU178.RCU
     49000 ±  5%     -25.9%      36287 ±  6%  softirqs.CPU18.RCU
     69522 ±  2%     -17.5%      57331 ± 11%  softirqs.CPU184.RCU
     53616 ± 15%     -29.7%      37690 ± 11%  softirqs.CPU19.RCU
     46530 ±  3%     -23.6%      35527 ±  5%  softirqs.CPU195.RCU
     65960 ±  9%     -13.9%      56772 ±  5%  softirqs.CPU196.RCU
     65948 ±  3%     -13.8%      56818 ±  9%  softirqs.CPU197.RCU
     61753 ± 11%     -18.4%      50375 ±  9%  softirqs.CPU198.RCU
     49063 ±  5%     -26.4%      36102 ±  3%  softirqs.CPU20.RCU
     69682 ±  8%     -17.0%      57840 ± 12%  softirqs.CPU200.RCU
     85131 ±  6%     -16.5%      71117 ±  8%  softirqs.CPU209.RCU
     48180 ±  4%     -24.9%      36169 ±  5%  softirqs.CPU21.RCU
     49795 ±  3%     -24.2%      37734 ± 11%  softirqs.CPU211.RCU
     74862 ±  2%     -20.0%      59923 ± 15%  softirqs.CPU213.RCU
     67497 ± 13%     -22.9%      52016 ±  8%  softirqs.CPU215.RCU
     70217 ±  8%     -25.2%      52488 ± 13%  softirqs.CPU221.RCU
     73106 ±  3%     -20.5%      58122 ±  8%  softirqs.CPU222.RCU
     78826 ±  7%     -26.6%      57843 ± 13%  softirqs.CPU224.RCU
     80792           -12.3%      70815 ±  7%  softirqs.CPU230.RCU
     60982 ±  7%     -20.4%      48548 ± 10%  softirqs.CPU239.RCU
     46118 ±  4%     -23.1%      35479 ±  7%  softirqs.CPU24.RCU
     88583 ±  4%     -20.6%      70347 ± 12%  softirqs.CPU242.RCU
     86647 ±  3%     -20.5%      68926 ± 13%  softirqs.CPU248.RCU
     47652 ±  4%     -26.2%      35166 ±  4%  softirqs.CPU25.RCU
     88271 ±  2%     -19.1%      71391 ± 16%  softirqs.CPU250.RCU
     47416 ±  4%     -20.2%      37859 ± 16%  softirqs.CPU256.RCU
     52441 ± 14%     -32.8%      35259 ±  2%  softirqs.CPU26.RCU
     54370 ±  5%     -19.1%      43968 ± 14%  softirqs.CPU265.RCU
     53131 ±  7%     -23.2%      40803 ±  8%  softirqs.CPU266.RCU
     48354 ±  4%     -26.4%      35588 ±  4%  softirqs.CPU268.RCU
     51934 ± 14%     -33.3%      34636 ±  5%  softirqs.CPU269.RCU
     53543 ± 13%     -33.5%      35599 ±  3%  softirqs.CPU27.RCU
     47194 ±  6%     -20.0%      37759        softirqs.CPU270.RCU
     49457 ± 14%     -29.0%      35136        softirqs.CPU271.RCU
     45491 ±  5%     -26.2%      33559 ±  2%  softirqs.CPU272.RCU
     45784 ±  3%     -25.1%      34298 ±  4%  softirqs.CPU273.RCU
     52291 ± 26%     -30.2%      36488 ±  6%  softirqs.CPU274.RCU
     44673 ±  5%     -21.3%      35144 ±  2%  softirqs.CPU275.RCU
     49716 ±  5%     -26.6%      36487 ±  5%  softirqs.CPU276.RCU
     52399 ± 14%     -24.7%      39459 ±  7%  softirqs.CPU277.RCU
     48252 ±  7%     -25.0%      36188 ±  7%  softirqs.CPU28.RCU
     50299 ±  8%     -27.5%      36457 ±  9%  softirqs.CPU280.RCU
     50958 ± 12%     -28.7%      36316 ±  8%  softirqs.CPU281.RCU
     48188 ±  3%     -24.5%      36399 ±  7%  softirqs.CPU283.RCU
     53991 ±  5%     -21.1%      42600 ±  9%  softirqs.CPU285.RCU
     63902 ±  9%     -20.6%      50708 ±  3%  softirqs.CPU287.RCU
     48867 ±  3%     -26.7%      35816 ±  4%  softirqs.CPU29.RCU
     64188 ± 22%     -36.3%      40910 ± 15%  softirqs.CPU3.RCU
     49075 ±  7%     -24.1%      37237 ± 10%  softirqs.CPU30.RCU
     55112 ± 19%     -33.5%      36654 ±  5%  softirqs.CPU31.RCU
     49696 ±  7%     -23.3%      38094 ± 10%  softirqs.CPU32.RCU
     47263 ±  4%     -26.6%      34695 ±  3%  softirqs.CPU33.RCU
     49828 ±  7%     -25.0%      37346 ± 13%  softirqs.CPU34.RCU
     48569 ±  2%     -22.8%      37473 ± 14%  softirqs.CPU35.RCU
     53596 ± 15%     -31.7%      36597 ±  5%  softirqs.CPU36.RCU
     65963 ±  8%     -19.1%      53397 ±  2%  softirqs.CPU39.RCU
     48592 ±  6%     -21.8%      38015        softirqs.CPU41.RCU
     54962 ± 12%     -23.0%      42297 ±  7%  softirqs.CPU44.RCU
     56343 ±  5%     -17.9%      46282 ± 12%  softirqs.CPU48.RCU
     49455           -12.9%      43073 ±  8%  softirqs.CPU49.RCU
     51678 ± 11%     -24.4%      39057 ±  4%  softirqs.CPU5.RCU
     48164 ±  3%     -24.2%      36493 ±  2%  softirqs.CPU52.RCU
     51613 ±  5%     -25.3%      38554 ±  4%  softirqs.CPU53.RCU
     48022 ±  7%     -23.9%      36525 ±  3%  softirqs.CPU54.RCU
     57159 ±  7%     -16.9%      47503 ± 12%  softirqs.CPU56.RCU
     51776 ±  6%     -18.3%      42303 ±  4%  softirqs.CPU59.RCU
     54240 ±  9%     -24.5%      40949 ±  6%  softirqs.CPU6.RCU
     49715 ±  6%     -24.6%      37501 ±  2%  softirqs.CPU60.RCU
     46880 ±  4%     -20.4%      37312 ±  5%  softirqs.CPU61.RCU
     48634 ±  6%     -11.9%      42833 ±  8%  softirqs.CPU62.RCU
     47966 ±  6%     -20.0%      38393 ±  9%  softirqs.CPU63.RCU
     51800 ±  9%     -22.7%      40039 ± 10%  softirqs.CPU64.RCU
     53752 ± 15%     -32.3%      36411 ±  5%  softirqs.CPU65.RCU
     52624 ± 13%     -24.8%      39585 ±  6%  softirqs.CPU68.RCU
     78620 ±  4%     -14.0%      67609 ±  6%  softirqs.CPU69.RCU
     95100 ±  7%     -17.3%      78635 ± 13%  softirqs.CPU73.RCU
     92258 ±  5%     -19.8%      74034 ± 13%  softirqs.CPU75.RCU
     54483 ± 13%     -32.8%      36617 ±  7%  softirqs.CPU8.RCU
     57971 ± 10%     -24.3%      43865 ±  7%  softirqs.CPU80.RCU
     47545 ±  6%     -25.2%      35553 ±  7%  softirqs.CPU84.RCU
     48658 ±  4%     -17.5%      40119 ±  9%  softirqs.CPU85.RCU
     85838 ±  4%     -14.1%      73715 ± 11%  softirqs.CPU86.RCU
     86626 ±  4%     -18.1%      70970 ± 11%  softirqs.CPU92.RCU
     84850 ±  2%     -17.5%      69983 ± 10%  softirqs.CPU97.RCU
     86178 ±  2%     -18.8%      69974 ± 10%  softirqs.CPU99.RCU
  18827858 ±  3%     -14.5%   16102740 ±  4%  softirqs.RCU





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc4-00002-g6d390e4b5d48e" of type "text/plain" (203581 bytes)

View attachment "job-script" of type "text/plain" (7416 bytes)

View attachment "job.yaml" of type "text/plain" (5019 bytes)

View attachment "reproduce" of type "text/plain" (310 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ