lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:	Mon, 13 Jun 2016 09:31:40 +0800
From:	kernel test robot <xiaolong.ye@...el.com>
To:	Al Viro <viro@...iv.linux.org.uk>
Cc:	LKML <linux-kernel@...r.kernel.org>,
	Al Viro <viro@...iv.linux.org.uk>,
	linux-fsdevel@...r.kernel.org, lkp@...org
Subject: [lkp] 64e3f9744b: reaim.jobs_per_min +34.3% improvement


FYI, we noticed a +34.3% improvement of reaim.jobs_per_min due to commit:

commit 64e3f9744bd3989457502ba79c09445a6909aac1 ("lockless next_positive()")
https://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git untested.dcache_readdir

in testcase: reaim
on test machine: lkp-hsw-ep2: 72 threads Brickland Haswell-EP with 128G memory
with following parameters: cpufreq_governor=performance/nr_task=1600%/runtime=300s/test=custom

In addition to that, the commit also has significant impact on the following tests:

reaim:	reaim.jobs_per_min +78.6% improvement	on test machine - lkp-hsw-ep2: 72 threads Brickland Haswell-EP with 128G memory
with test parameters: cpufreq_governor=performance/nr_job=3000/nr_task=100%/runtime=300s/test=custom


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase:
  gcc-4.9/performance/x86_64-rhel/1600%/debian-x86_64-2015-02-07.cgz/300s/lkp-hsw-ep2/custom/reaim

commit: 
  913f62db9f9ebb7be47ce8f87f2dcc2452dbfe6a
  64e3f9744bd3989457502ba79c09445a6909aac1

913f62db9f9ebb7b 64e3f9744bd3989457502ba79c 
---------------- -------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    410638 ±  2%     +34.3%     551598 ±  3%  reaim.jobs_per_min
    356.46 ±  2%     +34.3%     478.82 ±  3%  reaim.jobs_per_min_child
    433813 ±  4%     +29.1%     559846 ±  3%  reaim.max_jobs_per_min
     16.01 ±  2%     -25.6%      11.92 ±  3%  reaim.parent_time
    312.55 ±  2%      -3.5%     301.49 ±  0%  reaim.time.elapsed_time
    312.55 ±  2%      -3.5%     301.49 ±  0%  reaim.time.elapsed_time.max
 4.143e+08 ±  2%     +24.7%  5.165e+08 ±  2%  reaim.time.minor_page_faults
      4843 ±  1%      -9.4%       4388 ±  3%  reaim.time.percent_of_cpu_this_job_got
      8683 ±  2%     -42.2%       5017 ±  1%  reaim.time.system_time
      6455 ±  4%     +27.2%       8213 ±  3%  reaim.time.user_time
  19183296 ±  2%     +33.9%   25685969 ±  2%  reaim.time.voluntary_context_switches
   4865548 ±  2%     +24.6%    6061029 ±  2%  softirqs.NET_RX
   2315972 ±  2%     +41.7%    3282342 ±  1%  softirqs.RCU
   1516032 ±  3%     +36.9%    2075415 ±  3%  softirqs.SCHED
     24.75 ± 16%    -100.0%       0.00 ±  0%  vmstat.procs.b
    112.00 ±  6%     -42.9%      64.00 ±  8%  vmstat.procs.r
    133037 ±  1%     +39.4%     185465 ±  3%  vmstat.system.cs
 1.929e+08 ±  2%     +21.3%  2.339e+08 ±  9%  numa-numastat.node0.local_node
 1.929e+08 ±  2%     +21.3%  2.339e+08 ±  9%  numa-numastat.node0.numa_hit
   1.9e+08 ±  3%     +27.3%  2.418e+08 ±  9%  numa-numastat.node1.local_node
   1.9e+08 ±  3%     +27.3%  2.418e+08 ±  9%  numa-numastat.node1.numa_hit
   5285933 ±  3%     +49.6%    7907383 ±  8%  cpuidle.C1-HSW.usage
 5.168e+08 ±  5%     +15.0%  5.944e+08 ±  3%  cpuidle.C1E-HSW.time
   5520392 ±  3%     +39.0%    7672760 ±  1%  cpuidle.C1E-HSW.usage
 1.021e+09 ±  9%     +36.1%  1.389e+09 ± 14%  cpuidle.C3-HSW.time
   4974426 ±  8%     +48.5%    7388652 ± 10%  cpuidle.C3-HSW.usage
   5982360 ±  4%     +13.6%    6797802 ±  4%  cpuidle.C6-HSW.usage
    534471 ±  5%     +17.1%     625997 ± 12%  meminfo.Active
    518148 ±  6%     +17.7%     609663 ± 12%  meminfo.Active(anon)
      4823 ±  0%      +9.0%       5258 ±  7%  meminfo.AnonHugePages
    109895 ±  4%     -59.6%      44344 ± 12%  meminfo.KernelStack
     68650 ±  0%     -13.1%      59631 ±  0%  meminfo.SReclaimable
    283101 ±  1%     -22.8%     218669 ±  2%  meminfo.SUnreclaim
     66113 ±  6%     +14.2%      75483 ±  9%  meminfo.Shmem
    351753 ±  1%     -20.9%     278301 ±  1%  meminfo.Slab
     68.82 ±  1%      -8.7%      62.86 ±  2%  turbostat.%Busy
      1902 ±  1%      -9.1%       1729 ±  2%  turbostat.Avg_MHz
     22.15 ±  6%     +16.5%      25.80 ±  7%  turbostat.CPU%c1
      8.89 ±  2%     +26.3%      11.22 ±  2%  turbostat.CPU%c6
      2.96 ±  2%     +35.5%       4.00 ±  2%  turbostat.Pkg%pc2
      3.77 ±  6%     +22.9%       4.63 ±  4%  turbostat.Pkg%pc6
    249.09 ±  0%      +1.8%     253.56 ±  0%  turbostat.PkgWatt
     49.31 ±  0%      +3.8%      51.21 ±  1%  turbostat.RAMWatt
    227252 ±  5%     +21.6%     276314 ± 17%  numa-meminfo.node0.AnonPages
     53783 ±  4%     -53.4%      25067 ± 19%  numa-meminfo.node0.KernelStack
     36046 ±  4%     -30.5%      25057 ±  2%  numa-meminfo.node0.SReclaimable
    137804 ±  2%     -19.7%     110705 ±  5%  numa-meminfo.node0.SUnreclaim
    173851 ±  2%     -21.9%     135763 ±  4%  numa-meminfo.node0.Slab
    254095 ±  9%     +26.8%     322096 ±  7%  numa-meminfo.node1.Active
    245954 ±  9%     +27.6%     313937 ±  7%  numa-meminfo.node1.Active(anon)
    262651 ±  9%     +13.8%     298895 ±  8%  numa-meminfo.node1.FilePages
     52285 ±  4%     -62.8%      19433 ± 17%  numa-meminfo.node1.KernelStack
    142139 ±  3%     -24.0%     108022 ±  3%  numa-meminfo.node1.SUnreclaim
     19032 ±137%    +190.3%      55243 ± 47%  numa-meminfo.node1.Shmem
    174734 ±  3%     -18.4%     142593 ±  2%  numa-meminfo.node1.Slab
     56738 ±  4%     +22.0%      69204 ± 18%  numa-vmstat.node0.nr_anon_pages
      3350 ±  6%     -53.3%       1564 ± 18%  numa-vmstat.node0.nr_kernel_stack
      9008 ±  4%     -30.5%       6263 ±  2%  numa-vmstat.node0.nr_slab_reclaimable
     34418 ±  2%     -19.6%      27678 ±  5%  numa-vmstat.node0.nr_slab_unreclaimable
  95691640 ±  2%     +22.2%  1.169e+08 ±  8%  numa-vmstat.node0.numa_hit
  95691640 ±  2%     +22.2%  1.169e+08 ±  8%  numa-vmstat.node0.numa_local
     61413 ±  9%     +27.6%      78387 ±  6%  numa-vmstat.node1.nr_active_anon
     65653 ±  9%     +13.8%      74684 ±  8%  numa-vmstat.node1.nr_file_pages
      3239 ±  5%     -62.3%       1220 ± 16%  numa-vmstat.node1.nr_kernel_stack
      4749 ±137%    +190.0%      13771 ± 47%  numa-vmstat.node1.nr_shmem
     35475 ±  3%     -23.9%      26988 ±  3%  numa-vmstat.node1.nr_slab_unreclaimable
  95248915 ±  3%     +27.4%  1.213e+08 ±  9%  numa-vmstat.node1.numa_hit
  95248915 ±  3%     +27.4%  1.213e+08 ±  9%  numa-vmstat.node1.numa_local
    129524 ±  6%     +18.2%     153070 ± 12%  proc-vmstat.nr_active_anon
      6734 ±  4%     -58.7%       2780 ± 13%  proc-vmstat.nr_kernel_stack
     16608 ±  7%     +13.4%      18830 ±  9%  proc-vmstat.nr_shmem
     17158 ±  0%     -13.1%      14906 ±  0%  proc-vmstat.nr_slab_reclaimable
     70309 ±  1%     -22.2%      54705 ±  2%  proc-vmstat.nr_slab_unreclaimable
     50273 ±  2%     +16.2%      58426 ±  2%  proc-vmstat.numa_foreign
     80851 ±  7%    -100.0%      10.75 ±  7%  proc-vmstat.numa_hint_faults
     56684 ±  8%    -100.0%       5.00 ±100%  proc-vmstat.numa_hint_faults_local
 3.828e+08 ±  2%     +24.3%  4.757e+08 ±  2%  proc-vmstat.numa_hit
 3.828e+08 ±  2%     +24.3%  4.757e+08 ±  2%  proc-vmstat.numa_local
     50273 ±  2%     +16.2%      58426 ±  2%  proc-vmstat.numa_miss
      9919 ± 13%     -99.9%       5.75 ± 83%  proc-vmstat.numa_pages_migrated
     76071 ±  5%    -100.0%      21.25 ±  2%  proc-vmstat.numa_pte_updates
 4.026e+08 ±  2%     +22.8%  4.945e+08 ±  2%  proc-vmstat.pgalloc_normal
 4.152e+08 ±  2%     +24.6%  5.175e+08 ±  2%  proc-vmstat.pgfault
 4.088e+08 ±  2%     +22.8%  5.019e+08 ±  2%  proc-vmstat.pgfree
    570.00 ±  3%    -100.0%       0.00 ± -1%  proc-vmstat.pgmigrate_fail
      9919 ± 13%     -99.9%       5.75 ± 83%  proc-vmstat.pgmigrate_success
 1.096e+12 ±  2%     +24.9%  1.369e+12 ±  1%  perf-stat.L1-dcache-load-misses
 3.306e+12 ±  2%     +15.8%  3.829e+12 ±  1%  perf-stat.L1-dcache-stores
 7.901e+10 ±  2%     +25.0%  9.876e+10 ±  3%  perf-stat.L1-icache-load-misses
 1.077e+10 ±  3%     +31.4%  1.415e+10 ±  4%  perf-stat.LLC-load-misses
 8.757e+10 ±  1%     +20.7%  1.057e+11 ±  0%  perf-stat.LLC-loads
 4.785e+09 ±  3%     +17.2%  5.606e+09 ±  1%  perf-stat.LLC-store-misses
  7.44e+11 ±  2%     +26.1%   9.38e+11 ±  1%  perf-stat.LLC-stores
 5.417e+12 ±  2%     +10.0%  5.957e+12 ±  2%  perf-stat.branch-instructions
 8.637e+10 ±  2%     +26.1%  1.089e+11 ±  1%  perf-stat.branch-load-misses
 5.445e+12 ±  2%     +10.6%  6.021e+12 ±  1%  perf-stat.branch-loads
 8.649e+10 ±  2%     +26.0%   1.09e+11 ±  1%  perf-stat.branch-misses
  1.53e+12 ±  2%     -12.4%   1.34e+12 ±  2%  perf-stat.bus-cycles
 1.564e+10 ±  2%     +27.1%  1.988e+10 ±  3%  perf-stat.cache-misses
 9.013e+11 ±  2%     +25.5%  1.131e+12 ±  1%  perf-stat.cache-references
  41339114 ±  3%     +36.2%   56285165 ±  3%  perf-stat.context-switches
 4.254e+13 ±  2%     -13.0%  3.701e+13 ±  2%  perf-stat.cpu-cycles
  15328107 ±  2%     +35.3%   20739073 ±  0%  perf-stat.cpu-migrations
 1.023e+10 ±  5%     +22.1%  1.249e+10 ±  2%  perf-stat.dTLB-load-misses
 2.122e+09 ±  3%     +25.2%  2.656e+09 ±  1%  perf-stat.dTLB-store-misses
 3.304e+12 ±  2%     +15.7%  3.822e+12 ±  1%  perf-stat.dTLB-stores
 1.921e+09 ±  2%     +24.3%  2.388e+09 ±  2%  perf-stat.iTLB-load-misses
 1.496e+09 ±  2%     +25.9%  1.884e+09 ±  0%  perf-stat.iTLB-loads
 2.521e+13 ±  2%     +12.9%  2.847e+13 ±  2%  perf-stat.instructions
 4.129e+08 ±  2%     +24.6%  5.146e+08 ±  2%  perf-stat.minor-faults
  8.91e+09 ±  2%     +30.2%   1.16e+10 ±  3%  perf-stat.node-load-misses
  1.85e+09 ±  6%     +36.1%  2.518e+09 ± 12%  perf-stat.node-loads
 2.333e+09 ±  2%     +19.7%  2.794e+09 ±  0%  perf-stat.node-store-misses
 2.458e+09 ±  4%     +14.3%  2.809e+09 ±  2%  perf-stat.node-stores
 4.129e+08 ±  2%     +24.6%  5.146e+08 ±  2%  perf-stat.page-faults
 3.518e+13 ±  2%     -12.0%  3.097e+13 ±  2%  perf-stat.ref-cycles
     18.71 ± 54%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
     18.57 ± 54%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath
      0.99 ± 23%    +334.2%       4.29 ± 72%  perf-profile.cycles-pp._do_fork.sys_clone.do_syscall_64.return_from_SYSCALL_64.__libc_fork
      8.93 ± 51%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp._raw_spin_lock.next_positive.isra.13.dcache_readdir.iterate_dir.sys_getdents
     14.62 ± 54%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp._raw_spin_trylock.dput.dcache_dir_close.__fput.____fput
      1.66 ± 21%     +48.9%       2.48 ± 20%  perf-profile.cycles-pp.anon_vma_clone.anon_vma_fork.copy_process.part.30._do_fork.sys_clone
      2.09 ± 22%     +72.0%       3.60 ± 10%  perf-profile.cycles-pp.anon_vma_fork.copy_process.part.30._do_fork.sys_clone.do_syscall_64
      0.99 ± 17%     +42.1%       1.41 ± 19%  perf-profile.cycles-pp.anon_vma_interval_tree_insert.anon_vma_clone.anon_vma_fork.copy_process.part.30._do_fork
      7.06 ± 20%     +39.4%       9.84 ± 14%  perf-profile.cycles-pp.copy_process.part.30._do_fork.sys_clone.do_syscall_64.return_from_SYSCALL_64
     18.45 ± 54%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.dcache_dir_close.__fput.____fput.task_work_run.exit_to_usermode_loop
      9.89 ± 48%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.dcache_readdir.iterate_dir.sys_getdents.entry_SYSCALL_64_fastpath
      2.35 ± 19%     +30.7%       3.08 ±  6%  perf-profile.cycles-pp.do_execveat_common.isra.34.sys_execve.do_syscall_64.return_from_SYSCALL_64.execve
      7.51 ± 18%     +37.5%      10.33 ± 17%  perf-profile.cycles-pp.do_exit.do_group_exit.sys_exit_group.entry_SYSCALL_64_fastpath
      7.54 ± 18%     +37.5%      10.36 ± 18%  perf-profile.cycles-pp.do_group_exit.sys_exit_group.entry_SYSCALL_64_fastpath
      2.37 ± 19%     +31.1%       3.11 ±  7%  perf-profile.cycles-pp.do_syscall_64.return_from_SYSCALL_64.execve
     17.25 ± 54%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.dput.dcache_dir_close.__fput.____fput.task_work_run
      2.37 ± 19%     +31.9%       3.13 ±  8%  perf-profile.cycles-pp.execve
      6.08 ± 18%     +42.6%       8.67 ± 16%  perf-profile.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.sys_exit_group
      1.41 ± 19%     +40.9%       1.98 ± 17%  perf-profile.cycles-pp.free_pages_and_swap_cache.tlb_flush_mmu_free.tlb_finish_mmu.exit_mmap.mmput
      1.66 ± 26%     +66.3%       2.77 ±  6%  perf-profile.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
      9.96 ± 48%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.iterate_dir.sys_getdents.entry_SYSCALL_64_fastpath
      6.11 ± 18%     +42.5%       8.70 ± 15%  perf-profile.cycles-pp.mmput.do_exit.do_group_exit.sys_exit_group.entry_SYSCALL_64_fastpath
      8.91 ± 51%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.next_positive.isra.13.dcache_readdir.iterate_dir
      9.00 ± 51%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.next_positive.isra.13.dcache_readdir.iterate_dir.sys_getdents.entry_SYSCALL_64_fastpath
      0.76 ± 16%     +38.6%       1.05 ± 21%  perf-profile.cycles-pp.page_remove_rmap.unmap_page_range.unmap_single_vma.unmap_vmas.exit_mmap
      1.07 ± 15%     +41.0%       1.50 ± 13%  perf-profile.cycles-pp.perf_event_aux.part.51.perf_event_mmap.do_brk.sys_brk.entry_SYSCALL_64_fastpath
      0.79 ± 15%     +42.4%       1.12 ± 14%  perf-profile.cycles-pp.perf_event_aux_ctx.perf_event_aux.part.51.perf_event_mmap.do_brk.sys_brk
      1.16 ± 19%     +43.9%       1.66 ± 15%  perf-profile.cycles-pp.release_pages.free_pages_and_swap_cache.tlb_flush_mmu_free.tlb_finish_mmu.exit_mmap
      2.37 ± 19%     +31.1%       3.11 ±  7%  perf-profile.cycles-pp.return_from_SYSCALL_64.execve
      2.37 ± 19%     +31.3%       3.11 ±  7%  perf-profile.cycles-pp.sys_execve.do_syscall_64.return_from_SYSCALL_64.execve
      7.54 ± 18%     +37.5%      10.36 ± 18%  perf-profile.cycles-pp.sys_exit_group.entry_SYSCALL_64_fastpath
      9.97 ± 48%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.sys_getdents.entry_SYSCALL_64_fastpath
      1.44 ± 19%     +39.9%       2.01 ± 17%  perf-profile.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit
      1.41 ± 19%     +40.6%       1.98 ± 17%  perf-profile.cycles-pp.tlb_flush_mmu_free.tlb_finish_mmu.exit_mmap.mmput.do_exit
      0.92 ± 22%     +88.0%       1.73 ± 12%  perf-profile.cycles-pp.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.do_exit
      1694 ±  3%     -27.2%       1233 ±  2%  slabinfo.TCP.active_objs
      1694 ±  3%     -27.2%       1233 ±  2%  slabinfo.TCP.num_objs
      2604 ±  1%     -12.4%       2282 ±  0%  slabinfo.UDP.active_objs
      2604 ±  1%     -12.4%       2282 ±  0%  slabinfo.UDP.num_objs
     30612 ±  2%     -21.5%      24043 ±  3%  slabinfo.cred_jar.active_objs
    743.50 ±  2%     -21.3%     585.25 ±  4%  slabinfo.cred_jar.active_slabs
     31245 ±  2%     -21.3%      24595 ±  4%  slabinfo.cred_jar.num_objs
    743.50 ±  2%     -21.3%     585.25 ±  4%  slabinfo.cred_jar.num_slabs
     81166 ±  1%     -11.6%      71713 ±  0%  slabinfo.dentry.active_objs
      1965 ±  0%     -12.1%       1728 ±  0%  slabinfo.dentry.active_slabs
     82563 ±  0%     -12.1%      72601 ±  0%  slabinfo.dentry.num_objs
      1965 ±  0%     -12.1%       1728 ±  0%  slabinfo.dentry.num_slabs
      3979 ±  1%     -28.8%       2832 ±  1%  slabinfo.idr_layer_cache.active_objs
      3996 ±  2%     -29.1%       2832 ±  1%  slabinfo.idr_layer_cache.num_objs
     60008 ±  3%     -59.4%      24377 ±  2%  slabinfo.kmalloc-256.active_objs
      1089 ±  3%     -59.1%     446.00 ±  4%  slabinfo.kmalloc-256.active_slabs
     69755 ±  3%     -59.1%      28560 ±  4%  slabinfo.kmalloc-256.num_objs
      1089 ±  3%     -59.1%     446.00 ±  4%  slabinfo.kmalloc-256.num_slabs
     57703 ±  1%     -12.6%      50436 ±  0%  slabinfo.kmalloc-64.active_objs
    903.00 ±  1%     -12.7%     788.50 ±  0%  slabinfo.kmalloc-64.active_slabs
     57809 ±  1%     -12.7%      50494 ±  0%  slabinfo.kmalloc-64.num_objs
    903.00 ±  1%     -12.7%     788.50 ±  0%  slabinfo.kmalloc-64.num_slabs
     15224 ±  2%     -13.3%      13204 ±  3%  slabinfo.proc_inode_cache.active_objs
     15229 ±  1%     -13.3%      13204 ±  3%  slabinfo.proc_inode_cache.num_objs
     22960 ±  1%     -43.7%      12932 ±  2%  slabinfo.radix_tree_node.active_objs
    410.75 ±  1%     -43.9%     230.50 ±  2%  slabinfo.radix_tree_node.active_slabs
     23021 ±  1%     -43.8%      12932 ±  2%  slabinfo.radix_tree_node.num_objs
    410.75 ±  1%     -43.9%     230.50 ±  2%  slabinfo.radix_tree_node.num_slabs
     10853 ±  1%     -32.4%       7334 ±  0%  slabinfo.signal_cache.active_objs
    370.50 ±  1%     -32.3%     250.75 ±  1%  slabinfo.signal_cache.active_slabs
     11129 ±  1%     -32.3%       7539 ±  1%  slabinfo.signal_cache.num_objs
    370.50 ±  1%     -32.3%     250.75 ±  1%  slabinfo.signal_cache.num_slabs
     12578 ±  2%     +14.6%      14415 ±  2%  slabinfo.sigqueue.active_objs
     12578 ±  2%     +14.6%      14415 ±  2%  slabinfo.sigqueue.num_objs
      7307 ±  4%     -59.4%       2969 ± 11%  slabinfo.task_struct.active_objs
      2470 ±  4%     -58.3%       1030 ± 12%  slabinfo.task_struct.active_slabs
      7412 ±  4%     -58.3%       3091 ± 12%  slabinfo.task_struct.num_objs
      2470 ±  4%     -58.3%       1030 ± 12%  slabinfo.task_struct.num_slabs
     10188 ±  2%     +19.8%      12203 ±  4%  slabinfo.tw_sock_TCP.active_objs
     10188 ±  2%     +19.8%      12203 ±  4%  slabinfo.tw_sock_TCP.num_objs
     30284 ± 58%     +65.6%      50153 ± 11%  sched_debug.cfs_rq:/.MIN_vruntime.max
     93895 ±  2%     -12.1%      82501 ±  5%  sched_debug.cfs_rq:/.exec_clock.min
      5433 ±  7%     +29.2%       7017 ± 15%  sched_debug.cfs_rq:/.exec_clock.stddev
    579024 ± 12%     -14.8%     493610 ± 11%  sched_debug.cfs_rq:/.load.avg
    507.52 ± 16%     -27.0%     370.57 ± 13%  sched_debug.cfs_rq:/.load_avg.avg
    229.29 ±  3%     -26.3%     169.05 ± 11%  sched_debug.cfs_rq:/.load_avg.stddev
     30285 ± 58%     +65.6%      50153 ± 11%  sched_debug.cfs_rq:/.max_vruntime.max
    109999 ±  1%      -9.2%      99885 ±  1%  sched_debug.cfs_rq:/.min_vruntime.min
     20.25 ±  6%    -100.0%       0.00 ± -1%  sched_debug.cfs_rq:/.nr_spread_over.avg
     81.92 ±  8%    -100.0%       0.00 ± -1%  sched_debug.cfs_rq:/.nr_spread_over.max
     10.67 ±  4%    -100.0%       0.00 ± -1%  sched_debug.cfs_rq:/.nr_spread_over.min
      8.59 ±  8%    -100.0%       0.00 ± -1%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    406.16 ± 16%     -29.5%     286.34 ± 20%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    900.29 ± 13%     -26.6%     660.79 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.max
    260.94 ±  9%     -28.2%     187.23 ± 10%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
    374.86 ± 15%     -34.8%     244.51 ± 23%  sched_debug.cpu.cpu_load[0].avg
    867.92 ± 16%     -26.7%     635.83 ± 12%  sched_debug.cpu.cpu_load[0].max
    274.55 ± 11%     -28.0%     197.59 ± 11%  sched_debug.cpu.cpu_load[0].stddev
    461.43 ± 16%     -26.7%     338.34 ± 16%  sched_debug.cpu.cpu_load[1].avg
    907.79 ± 15%     -21.3%     714.67 ±  7%  sched_debug.cpu.cpu_load[1].max
    209.33 ± 10%     -27.0%     152.83 ± 11%  sched_debug.cpu.cpu_load[1].stddev
    457.05 ± 16%     -27.3%     332.36 ± 16%  sched_debug.cpu.cpu_load[2].avg
    206.50 ± 10%     -27.1%     150.57 ± 11%  sched_debug.cpu.cpu_load[2].stddev
    456.59 ± 17%     -28.4%     326.85 ± 17%  sched_debug.cpu.cpu_load[3].avg
    201.79 ± 11%     -27.3%     146.66 ± 11%  sched_debug.cpu.cpu_load[3].stddev
    457.79 ± 17%     -29.7%     321.94 ± 17%  sched_debug.cpu.cpu_load[4].avg
    840.62 ± 16%     -22.2%     653.79 ±  7%  sched_debug.cpu.cpu_load[4].max
    194.24 ± 11%     -28.8%     138.39 ± 11%  sched_debug.cpu.cpu_load[4].stddev
    583099 ± 10%     -16.4%     487499 ±  9%  sched_debug.cpu.load.avg
      1.19 ± 20%     -41.9%       0.69 ±  7%  sched_debug.cpu.nr_running.avg
     11.62 ± 32%     -68.1%       3.71 ± 11%  sched_debug.cpu.nr_running.max
      1.76 ± 30%     -60.2%       0.70 ± 10%  sched_debug.cpu.nr_running.stddev
    274985 ±  1%     +41.5%     389135 ±  3%  sched_debug.cpu.nr_switches.avg
    327724 ±  3%     +53.9%     504330 ±  9%  sched_debug.cpu.nr_switches.max
    225645 ±  0%     +29.8%     292787 ±  5%  sched_debug.cpu.nr_switches.min
     28442 ± 10%    +132.6%      66147 ± 29%  sched_debug.cpu.nr_switches.stddev
    276454 ±  1%     +41.9%     392229 ±  3%  sched_debug.cpu.sched_count.avg
    332344 ±  3%     +54.2%     512350 ±  8%  sched_debug.cpu.sched_count.max
    227293 ±  1%     +29.9%     295185 ±  5%  sched_debug.cpu.sched_count.min
     28564 ± 10%    +133.4%      66678 ± 28%  sched_debug.cpu.sched_count.stddev
     95059 ±  3%     +54.6%     146990 ±  6%  sched_debug.cpu.sched_goidle.avg
    118703 ±  5%     +68.0%     199381 ± 12%  sched_debug.cpu.sched_goidle.max
     72588 ±  2%     +42.5%     103446 ±  3%  sched_debug.cpu.sched_goidle.min
     14675 ± 10%    +110.9%      30957 ± 26%  sched_debug.cpu.sched_goidle.stddev
    112624 ±  3%     +44.3%     162510 ±  4%  sched_debug.cpu.ttwu_count.avg
    131627 ±  3%     +53.3%     201807 ± 11%  sched_debug.cpu.ttwu_count.max
     96249 ±  2%     +31.7%     126770 ±  6%  sched_debug.cpu.ttwu_count.min
      7968 ±  9%    +172.9%      21742 ± 47%  sched_debug.cpu.ttwu_count.stddev
     40592 ±  3%     +17.0%      47486 ±  8%  sched_debug.cpu.ttwu_local.avg
    279653 ± 66%     -90.5%      26664 ±100%  latency_stats.avg.call_rwsem_down_read_failed.lookup_slow.walk_component.path_lookupat.filename_lookup.user_path_at_empty.vfs_fstatat.SYSC_newlstat.SyS_newlstat.do_syscall_64.return_from_SYSCALL_64
     18032 ± 22%    -100.0%       0.00 ± -1%  latency_stats.avg.call_rwsem_down_read_failed.page_lock_anon_vma_read.rmap_walk_anon.rmap_walk.try_to_unmap.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     10312 ± 15%    -100.0%       0.00 ± -1%  latency_stats.avg.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     19871 ± 62%    -100.0%       0.00 ± -1%  latency_stats.avg.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.dcache_readdir
     44155 ± 26%     -97.7%       1018 ± 63%  latency_stats.avg.call_rwsem_down_write_failed.expand_downwards.expand_stack.__do_page_fault.do_page_fault.page_fault
      9568 ± 35%     -91.5%     812.75 ± 51%  latency_stats.avg.call_rwsem_down_write_failed.expand_downwards.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     76958 ± 22%     -79.2%      16017 ± 12%  latency_stats.avg.call_rwsem_down_write_failed.filename_create.SyS_mknod.entry_SYSCALL_64_fastpath
     46182 ±129%     -92.0%       3689 ±173%  latency_stats.avg.call_rwsem_down_write_failed.ipcget.SyS_semget.entry_SYSCALL_64_fastpath
      8390 ±101%    +327.0%      35827 ± 77%  latency_stats.avg.call_rwsem_down_write_failed.ipcget.SyS_shmget.entry_SYSCALL_64_fastpath
      9770 ± 98%    +547.2%      63237 ± 74%  latency_stats.avg.call_rwsem_down_write_failed.vma_adjust.__split_vma.split_vma.mprotect_fixup.SyS_mprotect.do_syscall_64.return_from_SYSCALL_64
     10179 ±103%    -100.0%       1.50 ±137%  latency_stats.avg.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
     32426 ± 13%     -99.8%      49.50 ±159%  latency_stats.avg.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
    169286 ± 21%    -100.0%       0.00 ± -1%  latency_stats.avg.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     15314 ± 10%    -100.0%       0.00 ± -1%  latency_stats.avg.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     14264 ±  9%    -100.0%       0.00 ± -1%  latency_stats.avg.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings.do_execveat_common.SyS_execve.do_syscall_64
     17744 ± 27%    -100.0%       0.00 ± -1%  latency_stats.avg.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count.do_execveat_common.SyS_execve.do_syscall_64
      6583 ± 58%    -100.0%       0.00 ± -1%  latency_stats.avg.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
     18105 ± 18%    -100.0%       1.50 ± 74%  latency_stats.hits.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
    197397 ±  6%    -100.0%       0.00 ± -1%  latency_stats.hits.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     47665 ±  5%    -100.0%       0.00 ± -1%  latency_stats.hits.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings.do_execveat_common.SyS_execve.do_syscall_64
     24308 ± 11%    -100.0%       0.00 ± -1%  latency_stats.hits.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count.do_execveat_common.SyS_execve.do_syscall_64
    360829 ± 51%     -90.4%      34507 ±105%  latency_stats.max.call_rwsem_down_read_failed.lookup_slow.walk_component.path_lookupat.filename_lookup.user_path_at_empty.vfs_fstatat.SYSC_newlstat.SyS_newlstat.do_syscall_64.return_from_SYSCALL_64
   1929874 ±  5%     -76.1%     460926 ± 38%  latency_stats.max.call_rwsem_down_read_failed.lookup_slow.walk_component.path_lookupat.filename_lookup.user_path_at_empty.vfs_fstatat.SYSC_newlstat.SyS_newlstat.entry_SYSCALL_64_fastpath
    955542 ± 12%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.page_lock_anon_vma_read.rmap_walk_anon.rmap_walk.try_to_unmap.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    941334 ±  6%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     66554 ± 68%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_page_from_iter
    283117 ± 21%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings
    357007 ± 50%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count
     67532 ± 41%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.dcache_readdir
    107233 ±166%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.ip_generic_getfrag
   1375657 ± 18%     -91.9%     111937 ± 19%  latency_stats.max.call_rwsem_down_write_failed.__put_anon_vma.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64
   1491740 ± 11%     -92.1%     118452 ± 14%  latency_stats.max.call_rwsem_down_write_failed.anon_vma_clone.anon_vma_fork.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
   1487474 ± 11%     -91.1%     132833 ±  9%  latency_stats.max.call_rwsem_down_write_failed.anon_vma_fork.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
   1829468 ± 14%     -84.4%     285222 ± 39%  latency_stats.max.call_rwsem_down_write_failed.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
   1921558 ±  6%     -76.2%     457033 ± 38%  latency_stats.max.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath
    881035 ± 24%     -95.9%      36350 ± 74%  latency_stats.max.call_rwsem_down_write_failed.expand_downwards.expand_stack.__do_page_fault.do_page_fault.page_fault
    527677 ± 61%     -92.4%      39967 ± 47%  latency_stats.max.call_rwsem_down_write_failed.expand_downwards.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    863996 ± 17%     -88.0%     103570 ± 15%  latency_stats.max.call_rwsem_down_write_failed.filename_create.SyS_mknod.entry_SYSCALL_64_fastpath
     69748 ±118%     -94.0%       4212 ±173%  latency_stats.max.call_rwsem_down_write_failed.ipcget.SyS_semget.entry_SYSCALL_64_fastpath
   1929879 ±  5%     -76.1%     460953 ± 38%  latency_stats.max.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath
   1902623 ±  4%     -76.8%     441073 ± 39%  latency_stats.max.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_open.entry_SYSCALL_64_fastpath
   1312701 ± 21%     -92.1%     103911 ± 10%  latency_stats.max.call_rwsem_down_write_failed.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
   1841795 ± 13%     -84.6%     283384 ± 40%  latency_stats.max.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
   1804477 ± 15%     -84.2%     284970 ± 39%  latency_stats.max.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.unmap_region.do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.SyS_mmap_pgoff.SyS_mmap.entry_SYSCALL_64_fastpath
   1272375 ± 16%     -89.4%     134784 ± 21%  latency_stats.max.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.unmap_region.do_munmap.vm_munmap.elf_map.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64
   1807510 ± 14%     -84.4%     282535 ± 40%  latency_stats.max.call_rwsem_down_write_failed.vma_adjust.__split_vma.do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.SyS_mmap_pgoff.SyS_mmap.entry_SYSCALL_64_fastpath
   1026438 ± 44%     -87.0%     133228 ± 19%  latency_stats.max.call_rwsem_down_write_failed.vma_adjust.__split_vma.do_munmap.vm_munmap.elf_map.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
     23314 ± 98%    +463.0%     131264 ± 58%  latency_stats.max.call_rwsem_down_write_failed.vma_adjust.__split_vma.split_vma.mprotect_fixup.SyS_mprotect.do_syscall_64.return_from_SYSCALL_64
   1822735 ± 14%     -84.3%     285740 ± 39%  latency_stats.max.call_rwsem_down_write_failed.vma_adjust.__split_vma.split_vma.mprotect_fixup.SyS_mprotect.entry_SYSCALL_64_fastpath
   1787371 ± 16%     -84.1%     284416 ± 39%  latency_stats.max.call_rwsem_down_write_failed.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.SyS_mmap_pgoff.SyS_mmap.entry_SYSCALL_64_fastpath
   1201732 ± 21%     -88.2%     142348 ± 23%  latency_stats.max.call_rwsem_down_write_failed.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.vm_mmap.elf_map.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64
     93165 ±125%    -100.0%       1.50 ±137%  latency_stats.max.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
    282905 ± 20%     -98.2%       4998 ±  0%  latency_stats.max.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
    380767 ± 13%    -100.0%      51.75 ±150%  latency_stats.max.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
    663345 ± 24%    -100.0%       0.00 ± -1%  latency_stats.max.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    906055 ±  6%    -100.0%       0.00 ± -1%  latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    531562 ± 33%    -100.0%       0.00 ± -1%  latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings.do_execveat_common.SyS_execve.do_syscall_64
    465940 ± 51%    -100.0%       0.00 ± -1%  latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count.do_execveat_common.SyS_execve.do_syscall_64
     16056 ± 52%    -100.0%       0.00 ± -1%  latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
    483318 ± 51%     -91.7%      40113 ±111%  latency_stats.sum.call_rwsem_down_read_failed.lookup_slow.walk_component.path_lookupat.filename_lookup.user_path_at_empty.vfs_fstatat.SYSC_newlstat.SyS_newlstat.do_syscall_64.return_from_SYSCALL_64
  94369195 ± 22%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.page_lock_anon_vma_read.rmap_walk_anon.rmap_walk.try_to_unmap.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
  52601473 ± 15%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    262253 ± 18%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_page_from_iter
   4620955 ±  8%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings
   3112733 ± 16%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count
     94113 ± 48%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.dcache_readdir
    110333 ±166%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.ip_generic_getfrag
  16110965 ± 29%     -99.7%      47942 ± 61%  latency_stats.sum.call_rwsem_down_write_failed.expand_downwards.expand_stack.__do_page_fault.do_page_fault.page_fault
   1716751 ± 37%     -92.4%     129775 ± 52%  latency_stats.sum.call_rwsem_down_write_failed.expand_downwards.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
   1847337 ±161%     -98.2%      33204 ±173%  latency_stats.sum.call_rwsem_down_write_failed.ipcget.SyS_semget.entry_SYSCALL_64_fastpath
    152844 ±152%   +1516.5%    2470754 ±123%  latency_stats.sum.call_rwsem_down_write_failed.ipcget.SyS_shmget.entry_SYSCALL_64_fastpath
     23897 ± 99%   +1086.7%     283584 ± 71%  latency_stats.sum.call_rwsem_down_write_failed.vma_adjust.__split_vma.split_vma.mprotect_fixup.SyS_mprotect.do_syscall_64.return_from_SYSCALL_64
    114082 ±102%    -100.0%       1.50 ±137%  latency_stats.sum.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
    268539 ±122%     -90.2%      26414 ± 34%  latency_stats.sum.pipe_wait.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
  5.81e+08 ± 20%    -100.0%      53.50 ±143%  latency_stats.sum.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
   7175164 ± 30%    -100.0%       0.00 ± -1%  latency_stats.sum.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
 3.038e+09 ± 15%    -100.0%       0.00 ± -1%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
 6.801e+08 ± 11%    -100.0%       0.00 ± -1%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings.do_execveat_common.SyS_execve.do_syscall_64
  4.37e+08 ± 31%    -100.0%       0.00 ± -1%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count.do_execveat_common.SyS_execve.do_syscall_64
     51967 ± 85%    -100.0%       0.00 ± -1%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64



                                 reaim.jobs_per_min

  600000 O+O--O-O--O-O-O--O-O--O-O-O--O-O--O-O-O--O-O-O--O------------------+
         |                                                 O  O   O  O O    O
  500000 ++                                                     O         O |
         |                                                                  |
         |                                                          .*.  .* |
  400000 ++                                                   *. .*.   *.   |
         |                                                  ..  *           |
  300000 *+*..*.*..*.*.*..*.*..*. .*..    .*   *..*   *..*.*                |
         |                       *    *.*. :   :  :   :                     |
  200000 ++                                 : :    : :                      |
         |                                  : :    : :                      |
         |                                  : :    : :                      |
  100000 ++                                 : :    : :                      |
         |                                   :      :                       |
       0 ++----------------------------------*------*-----------------------+



	[*] bisect-good sample
	[O] bisect-bad  sample

***************************************************************************************************
lkp-hsw-ep2: 72 threads Brickland Haswell-EP with 128G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_job/nr_task/rootfs/runtime/tbox_group/test/testcase:
  gcc-4.9/performance/x86_64-rhel/3000/100%/debian-x86_64-2015-02-07.cgz/300s/lkp-hsw-ep2/custom/reaim

commit: 
  913f62db9f9ebb7be47ce8f87f2dcc2452dbfe6a
  64e3f9744bd3989457502ba79c09445a6909aac1

913f62db9f9ebb7b 64e3f9744bd3989457502ba79c 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
         %stddev     %change         %stddev
             \          |                \  
    280131 ±  2%     +78.6%     500276 ±  0%  reaim.jobs_per_min
      3890 ±  2%     +78.6%       6948 ±  0%  reaim.jobs_per_min_child
     96.93 ±  0%      +1.1%      97.98 ±  0%  reaim.jti
    293728 ±  1%     +72.8%     507655 ±  0%  reaim.max_jobs_per_min
     46.24 ±  2%     -44.1%      25.86 ±  0%  reaim.parent_time
      2.63 ±  2%     -40.4%       1.57 ±  3%  reaim.std_dev_percent
      1.17 ±  4%     -66.3%       0.39 ±  4%  reaim.std_dev_time
    337.87 ±  2%      -9.2%     306.74 ±  0%  reaim.time.elapsed_time
    337.87 ±  2%      -9.2%     306.74 ±  0%  reaim.time.elapsed_time.max
   3590541 ±  1%     +11.0%    3984163 ±  1%  reaim.time.involuntary_context_switches
 3.566e+08 ±  0%     +57.4%  5.612e+08 ±  0%  reaim.time.minor_page_faults
      4893 ±  1%      -9.5%       4426 ±  0%  reaim.time.percent_of_cpu_this_job_got
     11257 ±  4%     -53.6%       5227 ±  0%  reaim.time.system_time
      5279 ±  1%     +58.2%       8350 ±  0%  reaim.time.user_time
  15174665 ±  2%     +94.5%   29512552 ±  2%  reaim.time.voluntary_context_switches
   4507715 ±  0%     +57.0%    7079258 ±  0%  softirqs.NET_RX
   1911113 ±  1%     +75.0%    3345252 ±  1%  softirqs.RCU
   1276144 ±  1%     +77.3%    2262545 ±  1%  softirqs.SCHED
   8661562 ±  3%     -17.3%    7166459 ±  1%  softirqs.TIMER
    992712 ±  2%     -20.3%     790775 ±  0%  vmstat.memory.cache
     61.75 ±  2%     -21.9%      48.25 ±  0%  vmstat.procs.r
     99182 ±  3%    +110.4%     208695 ±  1%  vmstat.system.cs
     79699 ±  0%      +1.6%      80997 ±  0%  vmstat.system.in
 1.674e+08 ±  6%     +55.7%  2.607e+08 ±  1%  numa-numastat.node0.local_node
     16061 ±  9%    +114.4%      34439 ± 16%  numa-numastat.node0.numa_foreign
 1.674e+08 ±  6%     +55.7%  2.607e+08 ±  1%  numa-numastat.node0.numa_hit
     16392 ± 18%    +115.4%      35317 ± 15%  numa-numastat.node0.numa_miss
 1.691e+08 ±  6%     +57.0%  2.654e+08 ±  1%  numa-numastat.node1.local_node
 1.691e+08 ±  6%     +57.0%  2.654e+08 ±  1%  numa-numastat.node1.numa_hit
  1.83e+08 ±  6%    +217.4%   5.81e+08 ±  2%  cpuidle.C1-HSW.time
   3293822 ±  4%    +219.1%   10512074 ±  2%  cpuidle.C1-HSW.usage
   4.1e+08 ±  1%     +71.1%  7.015e+08 ±  1%  cpuidle.C1E-HSW.time
   5133713 ±  1%     +77.1%    9089940 ±  0%  cpuidle.C1E-HSW.usage
 8.337e+08 ±  4%    +100.0%  1.667e+09 ±  2%  cpuidle.C3-HSW.time
   4449521 ±  4%    +106.4%    9183524 ±  2%  cpuidle.C3-HSW.usage
 5.916e+09 ±  2%     -15.2%  5.017e+09 ±  1%  cpuidle.C6-HSW.time
     35185 ±  3%    +119.4%      77213 ±  1%  cpuidle.POLL.usage
     69.17 ±  1%      -8.7%      63.18 ±  0%  turbostat.%Busy
      1923 ±  1%      -8.8%       1753 ±  0%  turbostat.Avg_MHz
     25.48 ±  3%     +18.7%      30.24 ±  1%  turbostat.CPU%c1
      0.11 ±  9%     -37.8%       0.07 ± 17%  turbostat.CPU%c3
      5.24 ±  1%     +24.5%       6.52 ±  1%  turbostat.CPU%c6
      1.30 ±  3%     +69.8%       2.21 ±  1%  turbostat.Pkg%pc2
      1.54 ±  2%     +38.1%       2.13 ± 14%  turbostat.Pkg%pc6
    249.36 ±  0%      +7.5%     267.97 ±  0%  turbostat.PkgWatt
     48.62 ±  0%     +10.5%      53.72 ±  0%  turbostat.RAMWatt
    159239 ±  2%     +11.9%     178110 ±  1%  meminfo.Active
    142976 ±  2%     +13.2%     161883 ±  1%  meminfo.Active(anon)
    246414 ±  1%     +24.7%     307160 ±  1%  meminfo.Committed_AS
     10533 ±  1%     +12.7%      11870 ±  0%  meminfo.Inactive(anon)
    248131 ±  8%     -89.4%      26316 ±  0%  meminfo.KernelStack
      7976 ±  1%     +11.6%       8898 ±  1%  meminfo.PageTables
     70810 ±  1%     -18.1%      58015 ±  0%  meminfo.SReclaimable
    381947 ±  5%     -54.9%     172231 ±  0%  meminfo.SUnreclaim
     53714 ±  7%     +36.4%      73271 ±  3%  meminfo.Shmem
    452758 ±  4%     -49.1%     230247 ±  0%  meminfo.Slab
    127032 ± 14%     -88.7%      14333 ±  0%  numa-meminfo.node0.KernelStack
    947497 ±  4%     -28.3%     679299 ±  3%  numa-meminfo.node0.MemUsed
     39330 ±  2%     -34.2%      25890 ± 11%  numa-meminfo.node0.SReclaimable
    196101 ± 10%     -57.5%      83439 ±  4%  numa-meminfo.node0.SUnreclaim
    235431 ±  8%     -53.6%     109330 ±  4%  numa-meminfo.node0.Slab
     57879 ±  3%     +80.9%     104724 ± 24%  numa-meminfo.node1.Active
     49709 ±  3%     +94.3%      96605 ± 26%  numa-meminfo.node1.Active(anon)
    248606 ±  1%     +18.9%     295508 ±  8%  numa-meminfo.node1.FilePages
    124588 ± 11%     -90.3%      12062 ±  0%  numa-meminfo.node1.KernelStack
    947115 ±  2%     -17.3%     782949 ±  3%  numa-meminfo.node1.MemUsed
    187971 ±  6%     -52.7%      88852 ±  4%  numa-meminfo.node1.SUnreclaim
      5013 ± 85%    +934.1%      51847 ± 47%  numa-meminfo.node1.Shmem
    219480 ±  6%     -44.9%     120973 ±  4%  numa-meminfo.node1.Slab
      7944 ± 14%     -88.7%     897.75 ±  0%  numa-vmstat.node0.nr_kernel_stack
      9834 ±  2%     -34.2%       6471 ± 11%  numa-vmstat.node0.nr_slab_reclaimable
     49075 ± 10%     -57.5%      20865 ±  4%  numa-vmstat.node0.nr_slab_unreclaimable
     75758 ±  3%     +16.9%      88560 ±  6%  numa-vmstat.node0.numa_foreign
  85590888 ±  6%     +55.5%  1.331e+08 ±  1%  numa-vmstat.node0.numa_hit
  85590888 ±  6%     +55.5%  1.331e+08 ±  1%  numa-vmstat.node0.numa_local
     75898 ±  4%     +17.4%      89083 ±  6%  numa-vmstat.node0.numa_miss
     12439 ±  3%     +94.2%      24153 ± 26%  numa-vmstat.node1.nr_active_anon
     62151 ±  1%     +18.9%      73881 ±  8%  numa-vmstat.node1.nr_file_pages
      7735 ± 11%     -90.3%     751.75 ±  1%  numa-vmstat.node1.nr_kernel_stack
      1253 ± 85%    +934.6%      12965 ± 47%  numa-vmstat.node1.nr_shmem
     46890 ±  6%     -52.6%      22218 ±  4%  numa-vmstat.node1.nr_slab_unreclaimable
  86095923 ±  5%     +57.8%  1.359e+08 ±  1%  numa-vmstat.node1.numa_hit
  86095923 ±  5%     +57.8%  1.359e+08 ±  1%  numa-vmstat.node1.numa_local
     35750 ±  2%     +13.2%      40451 ±  1%  proc-vmstat.nr_active_anon
      2639 ±  0%     +12.2%       2961 ±  1%  proc-vmstat.nr_inactive_anon
     15787 ±  8%     -89.6%       1648 ±  0%  proc-vmstat.nr_kernel_stack
      2001 ±  1%     +10.8%       2217 ±  1%  proc-vmstat.nr_page_table_pages
     13425 ±  7%     +36.5%      18327 ±  3%  proc-vmstat.nr_shmem
     17716 ±  1%     -18.1%      14503 ±  0%  proc-vmstat.nr_slab_reclaimable
     96392 ±  5%     -55.3%      43067 ±  0%  proc-vmstat.nr_slab_unreclaimable
     44179 ±  1%     +47.4%      65119 ±  0%  proc-vmstat.numa_foreign
 3.365e+08 ±  0%     +56.3%   5.26e+08 ±  0%  proc-vmstat.numa_hit
 3.365e+08 ±  0%     +56.3%   5.26e+08 ±  0%  proc-vmstat.numa_local
     44179 ±  1%     +47.4%      65119 ±  0%  proc-vmstat.numa_miss
     56553 ±  3%     +77.2%     100208 ±  3%  proc-vmstat.numa_pages_migrated
    261372 ±  6%     +45.5%     380305 ±  1%  proc-vmstat.numa_pte_updates
     14445 ±  8%     +35.5%      19578 ±  4%  proc-vmstat.pgactivate
   5492871 ± 10%     +58.9%    8725735 ±  1%  proc-vmstat.pgalloc_dma32
 3.574e+08 ±  0%     +53.3%  5.478e+08 ±  0%  proc-vmstat.pgalloc_normal
 3.574e+08 ±  0%     +57.3%  5.622e+08 ±  0%  proc-vmstat.pgfault
 3.629e+08 ±  0%     +53.4%  5.565e+08 ±  0%  proc-vmstat.pgfree
    461.50 ± 27%     -99.8%       0.75 ±110%  proc-vmstat.pgmigrate_fail
     56553 ±  3%     +77.2%     100208 ±  3%  proc-vmstat.pgmigrate_success
 8.853e+11 ±  0%     +55.0%  1.372e+12 ±  0%  perf-stat.L1-dcache-load-misses
 3.302e+12 ±  2%      +9.9%   3.63e+12 ±  0%  perf-stat.L1-dcache-loads
 2.996e+12 ±  1%     +30.8%  3.918e+12 ±  0%  perf-stat.L1-dcache-stores
 6.154e+10 ±  0%     +49.9%  9.225e+10 ±  0%  perf-stat.L1-icache-load-misses
 7.337e+09 ±  2%    +107.9%  1.526e+10 ±  2%  perf-stat.LLC-load-misses
 7.769e+10 ±  0%     +41.5%  1.099e+11 ±  0%  perf-stat.LLC-loads
  4.06e+09 ±  1%     +72.4%  6.998e+09 ±  1%  perf-stat.LLC-store-misses
 5.947e+11 ±  0%     +57.7%  9.376e+11 ±  0%  perf-stat.LLC-stores
 4.994e+12 ±  1%     +22.8%  6.131e+12 ±  1%  perf-stat.branch-instructions
 6.768e+10 ±  0%     +60.7%  1.088e+11 ±  0%  perf-stat.branch-load-misses
 5.034e+12 ±  1%     +22.2%   6.15e+12 ±  0%  perf-stat.branch-loads
 6.827e+10 ±  0%     +58.9%  1.085e+11 ±  0%  perf-stat.branch-misses
 1.662e+12 ±  3%     -17.6%  1.371e+12 ±  0%  perf-stat.bus-cycles
 1.144e+10 ±  1%     +92.3%    2.2e+10 ±  2%  perf-stat.cache-misses
 7.258e+11 ±  0%     +58.8%  1.152e+12 ±  0%  perf-stat.cache-references
  33449933 ±  1%     +92.6%   64433327 ±  1%  perf-stat.context-switches
 4.642e+13 ±  3%     -18.4%  3.787e+13 ±  0%  perf-stat.cpu-cycles
  11871609 ±  2%    +104.9%   24323849 ±  1%  perf-stat.cpu-migrations
 8.093e+09 ±  4%     +45.9%  1.181e+10 ±  1%  perf-stat.dTLB-load-misses
 3.287e+12 ±  2%      +8.6%  3.569e+12 ±  0%  perf-stat.dTLB-loads
  1.66e+09 ±  1%     +61.0%  2.672e+09 ±  0%  perf-stat.dTLB-store-misses
 2.981e+12 ±  0%     +33.9%  3.991e+12 ±  0%  perf-stat.dTLB-stores
 1.596e+09 ±  2%     +48.6%  2.371e+09 ±  1%  perf-stat.iTLB-load-misses
 1.353e+09 ±  0%     +57.3%  2.129e+09 ±  0%  perf-stat.iTLB-loads
 2.258e+13 ±  1%     +29.2%  2.918e+13 ±  0%  perf-stat.instructions
 3.558e+08 ±  0%     +57.3%  5.595e+08 ±  0%  perf-stat.minor-faults
 6.653e+09 ±  2%    +110.5%  1.401e+10 ±  2%  perf-stat.node-load-misses
 7.252e+08 ±  0%     +73.5%  1.259e+09 ±  1%  perf-stat.node-loads
 1.971e+09 ±  1%     +72.3%  3.396e+09 ±  2%  perf-stat.node-store-misses
  2.11e+09 ±  1%     +69.6%  3.579e+09 ±  1%  perf-stat.node-stores
 3.558e+08 ±  0%     +57.3%  5.595e+08 ±  0%  perf-stat.page-faults
 3.828e+13 ±  3%     -17.4%  3.164e+13 ±  0%  perf-stat.ref-cycles
      1345 ±  2%     -15.2%       1141 ±  0%  slabinfo.TCP.active_objs
      1345 ±  2%     -15.2%       1141 ±  0%  slabinfo.TCP.num_objs
     49802 ±  6%     -50.5%      24671 ±  1%  slabinfo.cred_jar.active_objs
      1269 ±  7%     -53.5%     590.25 ±  1%  slabinfo.cred_jar.active_slabs
     53315 ±  7%     -53.5%      24799 ±  1%  slabinfo.cred_jar.num_objs
      1269 ±  7%     -53.5%     590.25 ±  1%  slabinfo.cred_jar.num_slabs
    100464 ±  2%     -26.9%      73397 ±  0%  slabinfo.dentry.active_objs
      2410 ±  2%     -27.2%       1755 ±  0%  slabinfo.dentry.active_slabs
    101236 ±  2%     -27.2%      73743 ±  0%  slabinfo.dentry.num_objs
      2410 ±  2%     -27.2%       1755 ±  0%  slabinfo.dentry.num_slabs
      4688 ±  1%     -36.2%       2990 ±  1%  slabinfo.idr_layer_cache.active_objs
    315.00 ±  1%     -36.8%     199.00 ±  1%  slabinfo.idr_layer_cache.active_slabs
      4729 ±  1%     -36.8%       2990 ±  1%  slabinfo.idr_layer_cache.num_objs
    315.00 ±  1%     -36.8%     199.00 ±  1%  slabinfo.idr_layer_cache.num_slabs
      8258 ±  7%     -19.9%       6618 ±  0%  slabinfo.kmalloc-128.active_objs
      8258 ±  7%     -19.9%       6618 ±  0%  slabinfo.kmalloc-128.num_objs
    158539 ±  7%     -84.4%      24685 ±  0%  slabinfo.kmalloc-256.active_objs
      2500 ±  7%     -84.1%     396.50 ±  0%  slabinfo.kmalloc-256.active_slabs
    160076 ±  7%     -84.1%      25403 ±  0%  slabinfo.kmalloc-256.num_objs
      2500 ±  7%     -84.1%     396.50 ±  0%  slabinfo.kmalloc-256.num_slabs
     61717 ±  2%     -15.4%      52206 ±  0%  slabinfo.kmalloc-64.active_objs
    967.25 ±  1%     -15.7%     815.25 ±  0%  slabinfo.kmalloc-64.active_slabs
     61936 ±  1%     -15.7%      52213 ±  0%  slabinfo.kmalloc-64.num_objs
    967.25 ±  1%     -15.7%     815.25 ±  0%  slabinfo.kmalloc-64.num_slabs
     25220 ±  2%     -51.7%      12173 ±  0%  slabinfo.radix_tree_node.active_objs
    452.00 ±  3%     -52.0%     217.00 ±  0%  slabinfo.radix_tree_node.active_slabs
     25330 ±  3%     -51.9%      12173 ±  0%  slabinfo.radix_tree_node.num_objs
    452.00 ±  3%     -52.0%     217.00 ±  0%  slabinfo.radix_tree_node.num_slabs
     20109 ±  1%     -12.2%      17653 ±  1%  slabinfo.shmem_inode_cache.active_objs
     20346 ±  1%     -12.4%      17825 ±  1%  slabinfo.shmem_inode_cache.num_objs
     18964 ±  6%     -59.4%       7701 ±  1%  slabinfo.signal_cache.active_objs
    673.00 ±  6%     -61.3%     260.50 ±  1%  slabinfo.signal_cache.active_slabs
     20203 ±  6%     -61.2%       7834 ±  1%  slabinfo.signal_cache.num_objs
    673.00 ±  6%     -61.3%     260.50 ±  1%  slabinfo.signal_cache.num_slabs
     16082 ±  7%     -88.6%       1826 ±  0%  slabinfo.task_struct.active_objs
      5457 ±  7%     -88.4%     634.25 ±  0%  slabinfo.task_struct.active_slabs
     16371 ±  7%     -88.4%       1903 ±  0%  slabinfo.task_struct.num_objs
      5457 ±  7%     -88.4%     634.25 ±  0%  slabinfo.task_struct.num_slabs
      9601 ±  2%     +46.7%      14086 ±  0%  slabinfo.tw_sock_TCP.active_objs
      9601 ±  2%     +46.7%      14086 ±  0%  slabinfo.tw_sock_TCP.num_objs
     32772 ±  0%     +11.6%      36565 ±  0%  slabinfo.vm_area_struct.active_objs
     32955 ±  0%     +11.4%      36722 ±  0%  slabinfo.vm_area_struct.num_objs
    178185 ± 20%    -100.0%       0.00 ± -1%  latency_stats.avg.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    116900 ±102%    -100.0%       0.00 ± -1%  latency_stats.avg.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.dcache_readdir.iterate_dir
      7286 ±  4%    +474.3%      41845 ±  5%  latency_stats.hits.call_rwsem_down_write_failed.__put_anon_vma.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64
    203768 ±  5%    +418.3%    1056152 ±  3%  latency_stats.hits.call_rwsem_down_write_failed.anon_vma_clone.anon_vma_fork.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
    198735 ±  4%    +480.4%    1153519 ±  4%  latency_stats.hits.call_rwsem_down_write_failed.anon_vma_fork.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
    234689 ± 37%    +597.0%    1635843 ± 18%  latency_stats.hits.call_rwsem_down_write_failed.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
    346784 ± 10%    +217.9%    1102503 ± 10%  latency_stats.hits.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath
     25145 ±  4%    +318.7%     105283 ±  3%  latency_stats.hits.call_rwsem_down_write_failed.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
     14012 ± 29%     -99.9%      16.75 ±108%  latency_stats.hits.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    147182 ± 20%     -80.9%      28123 ±  8%  latency_stats.max.call_rwsem_down_read_failed.lookup_slow.walk_component.path_lookupat.filename_lookup.user_path_at_empty.vfs_fstatat.SYSC_newlstat.SyS_newlstat.entry_SYSCALL_64_fastpath
     56867 ± 29%     -94.8%       2930 ±153%  latency_stats.max.call_rwsem_down_read_failed.page_lock_anon_vma_read.rmap_walk_anon.rmap_walk.try_to_unmap.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     63761 ± 48%     -98.3%       1101 ±106%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     48873 ± 44%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings
     28133 ± 16%    -100.0%       0.00 ± -1%  latency_stats.max.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count
    129450 ± 20%     -89.5%      13528 ±  6%  latency_stats.max.call_rwsem_down_write_failed.__put_anon_vma.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64
    176508 ± 20%     -90.5%      16735 ± 19%  latency_stats.max.call_rwsem_down_write_failed.anon_vma_clone.anon_vma_fork.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
    159244 ± 28%     -84.2%      25136 ± 52%  latency_stats.max.call_rwsem_down_write_failed.anon_vma_fork.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
    133637 ± 26%     -74.6%      33991 ± 11%  latency_stats.max.call_rwsem_down_write_failed.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
    141731 ± 23%     -81.4%      26362 ±  2%  latency_stats.max.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath
    102886 ± 18%     -76.5%      24133 ±  7%  latency_stats.max.call_rwsem_down_write_failed.filename_create.SyS_link.entry_SYSCALL_64_fastpath
    149381 ± 20%     -81.3%      27940 ± 10%  latency_stats.max.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath
     84090 ± 16%     -78.0%      18541 ± 18%  latency_stats.max.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_open.entry_SYSCALL_64_fastpath
    136819 ± 20%     -89.4%      14560 ± 11%  latency_stats.max.call_rwsem_down_write_failed.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
    130486 ± 27%     -75.4%      32109 ±  5%  latency_stats.max.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
     80055 ± 69%     -58.5%      33246 ± 12%  latency_stats.max.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.unmap_region.do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.SyS_mmap_pgoff.SyS_mmap.entry_SYSCALL_64_fastpath
     60847 ± 27%     -64.7%      21497 ± 15%  latency_stats.max.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.unmap_region.do_munmap.vm_munmap.elf_map.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64
     62740 ± 78%     -47.6%      32897 ± 18%  latency_stats.max.call_rwsem_down_write_failed.vma_adjust.__split_vma.do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.SyS_mmap_pgoff.SyS_mmap.entry_SYSCALL_64_fastpath
     84706 ± 55%     -73.2%      22731 ± 16%  latency_stats.max.call_rwsem_down_write_failed.vma_adjust.__split_vma.do_munmap.vm_munmap.elf_map.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
     90671 ± 43%     -62.1%      34363 ±  7%  latency_stats.max.call_rwsem_down_write_failed.vma_adjust.__split_vma.split_vma.mprotect_fixup.SyS_mprotect.entry_SYSCALL_64_fastpath
     75237 ± 72%     -54.3%      34375 ± 13%  latency_stats.max.call_rwsem_down_write_failed.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.SyS_mmap_pgoff.SyS_mmap.entry_SYSCALL_64_fastpath
     13483 ± 69%     -63.0%       4995 ±  0%  latency_stats.max.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
     31990 ± 38%    -100.0%       3.50 ± 76%  latency_stats.max.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
    857985 ± 15%    -100.0%       0.00 ± -1%  latency_stats.max.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    116900 ±102%    -100.0%       0.00 ± -1%  latency_stats.max.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.dcache_readdir.iterate_dir
     75015 ± 22%     -95.6%       3276 ±155%  latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     48783 ± 42%    -100.0%       0.00 ± -1%  latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings.do_execveat_common.SyS_execve.do_syscall_64
     36423 ± 29%    -100.0%       0.00 ± -1%  latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count.do_execveat_common.SyS_execve.do_syscall_64
   1050731 ± 31%     -99.7%       3113 ±146%  latency_stats.sum.call_rwsem_down_read_failed.page_lock_anon_vma_read.rmap_walk_anon.rmap_walk.try_to_unmap.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
   1090211 ± 36%     -99.8%       1956 ± 91%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    344146 ± 25%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings
    161734 ± 42%    -100.0%       0.00 ± -1%  latency_stats.sum.call_rwsem_down_read_failed.rmap_walk_anon.rmap_walk.remove_migration_ptes.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count
 1.379e+08 ± 55%    +467.5%  7.826e+08 ± 28%  latency_stats.sum.call_rwsem_down_write_failed.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
    412.25 ±173%   +2400.7%      10309 ± 23%  latency_stats.sum.call_rwsem_down_write_failed.vma_adjust.__split_vma.split_vma.mprotect_fixup.SyS_mprotect.do_syscall_64.return_from_SYSCALL_64
   6913089 ± 74%     -43.0%    3942273 ± 24%  latency_stats.sum.call_rwsem_down_write_failed.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.vm_mmap.elf_map.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64
    612938 ± 13%    -100.0%       4.25 ± 82%  latency_stats.sum.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
   9063364 ± 40%    -100.0%       0.00 ± -1%  latency_stats.sum.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    116900 ±102%    -100.0%       0.00 ± -1%  latency_stats.sum.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.dcache_readdir.iterate_dir
  10312452 ± 26%     -99.8%      16505 ±157%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
   3160481 ± 20%    -100.0%       0.00 ± -1%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.copy_strings.do_execveat_common.SyS_execve.do_syscall_64
   1588758 ± 45%    -100.0%       0.00 ± -1%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.count.do_execveat_common.SyS_execve.do_syscall_64
    101538 ±  1%      -9.6%      91752 ±  0%  sched_debug.cfs_rq:/.exec_clock.avg
     89558 ±  1%     -12.4%      78426 ±  1%  sched_debug.cfs_rq:/.exec_clock.min
      6990 ±  1%     +59.1%      11125 ±  5%  sched_debug.cfs_rq:/.exec_clock.stddev
    657444 ±  4%     -14.8%     560319 ±  9%  sched_debug.cfs_rq:/.load.avg
    581.90 ±  6%     -37.3%     365.03 ± 12%  sched_debug.cfs_rq:/.load_avg.avg
      1251 ±  9%     -33.9%     827.62 ±  2%  sched_debug.cfs_rq:/.load_avg.max
     36.62 ± 73%    +197.2%     108.83 ± 58%  sched_debug.cfs_rq:/.load_avg.min
    326.10 ± 13%     -46.6%     174.00 ±  8%  sched_debug.cfs_rq:/.load_avg.stddev
     97385 ±  1%     -10.1%      87521 ±  1%  sched_debug.cfs_rq:/.min_vruntime.min
      8236 ±  2%     +53.7%      12656 ±  4%  sched_debug.cfs_rq:/.min_vruntime.stddev
     40.00 ±  4%    -100.0%       0.00 ± -1%  sched_debug.cfs_rq:/.nr_spread_over.avg
    167.08 ± 10%    -100.0%       0.00 ± -1%  sched_debug.cfs_rq:/.nr_spread_over.max
     21.62 ±  6%    -100.0%       0.00 ± -1%  sched_debug.cfs_rq:/.nr_spread_over.min
     17.35 ±  9%    -100.0%       0.00 ± -1%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    491.37 ±  2%     -46.3%     263.87 ± 24%  sched_debug.cfs_rq:/.runnable_load_avg.avg
      1106 ± 11%     -35.4%     714.37 ±  8%  sched_debug.cfs_rq:/.runnable_load_avg.max
    345.25 ± 11%     -41.2%     202.85 ± 11%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
    -19886 ±-14%     +21.2%     -24096 ± -6%  sched_debug.cfs_rq:/.spread0.avg
    -33046 ± -9%     +17.6%     -38869 ± -5%  sched_debug.cfs_rq:/.spread0.min
      8240 ±  2%     +53.6%      12660 ±  4%  sched_debug.cfs_rq:/.spread0.stddev
    626.23 ±  6%     -11.8%     552.63 ±  6%  sched_debug.cfs_rq:/.util_avg.avg
     46.42 ± 75%    +303.1%     187.12 ± 37%  sched_debug.cfs_rq:/.util_avg.min
    272.45 ±  9%     -35.1%     176.74 ±  9%  sched_debug.cfs_rq:/.util_avg.stddev
    549467 ±  6%     -28.5%     392724 ±  8%  sched_debug.cpu.avg_idle.avg
     67395 ± 15%     -41.7%      39264 ± 24%  sched_debug.cpu.avg_idle.min
    465.95 ±  2%     -56.3%     203.55 ± 30%  sched_debug.cpu.cpu_load[0].avg
      1106 ± 11%     -35.8%     710.12 ±  9%  sched_debug.cpu.cpu_load[0].max
    357.88 ± 11%     -43.1%     203.56 ± 11%  sched_debug.cpu.cpu_load[0].stddev
    548.09 ±  5%     -39.4%     331.94 ± 14%  sched_debug.cpu.cpu_load[1].avg
      1126 ± 13%     -32.8%     757.50 ±  8%  sched_debug.cpu.cpu_load[1].max
    311.62 ± 16%     -45.6%     169.54 ± 10%  sched_debug.cpu.cpu_load[1].stddev
    544.09 ±  5%     -40.1%     325.95 ± 15%  sched_debug.cpu.cpu_load[2].avg
      1099 ± 12%     -33.6%     730.46 ±  9%  sched_debug.cpu.cpu_load[2].max
      2.79 ±163%   +1625.4%      48.17 ±130%  sched_debug.cpu.cpu_load[2].min
    311.03 ± 15%     -46.5%     166.42 ± 11%  sched_debug.cpu.cpu_load[2].stddev
    541.38 ±  5%     -40.9%     319.81 ± 15%  sched_debug.cpu.cpu_load[3].avg
      1086 ± 12%     -34.8%     708.92 ±  8%  sched_debug.cpu.cpu_load[3].max
      7.17 ±138%    +684.3%      56.21 ±114%  sched_debug.cpu.cpu_load[3].min
    310.30 ± 15%     -47.6%     162.65 ± 12%  sched_debug.cpu.cpu_load[3].stddev
    539.78 ±  5%     -41.8%     314.24 ± 15%  sched_debug.cpu.cpu_load[4].avg
      1072 ± 12%     -36.5%     680.42 ±  6%  sched_debug.cpu.cpu_load[4].max
      9.75 ± 95%    +557.7%      64.12 ± 97%  sched_debug.cpu.cpu_load[4].min
    307.15 ± 15%     -49.5%     155.25 ± 12%  sched_debug.cpu.cpu_load[4].stddev
    660005 ±  3%     -18.0%     541012 ±  9%  sched_debug.cpu.load.avg
      6919 ±  3%     +30.9%       9055 ±  4%  sched_debug.cpu.nr_load_updates.stddev
      0.80 ±  5%     -20.7%       0.64 ±  8%  sched_debug.cpu.nr_running.avg
    210549 ±  3%    +110.4%     443009 ±  1%  sched_debug.cpu.nr_switches.avg
    270771 ±  6%    +111.7%     573267 ±  2%  sched_debug.cpu.nr_switches.max
    167884 ±  4%     +98.0%     332477 ±  1%  sched_debug.cpu.nr_switches.min
     22588 ±  3%    +310.3%      92676 ±  3%  sched_debug.cpu.nr_switches.stddev
      0.16 ± 36%    +153.6%       0.41 ± 18%  sched_debug.cpu.nr_uninterruptible.avg
      1020 ±  9%    +426.3%       5371 ± 14%  sched_debug.cpu.nr_uninterruptible.max
     -1120 ±-13%    +422.9%      -5858 ±-18%  sched_debug.cpu.nr_uninterruptible.min
    648.41 ± 15%    +519.4%       4015 ±  7%  sched_debug.cpu.nr_uninterruptible.stddev
    211134 ±  3%    +111.9%     447352 ±  1%  sched_debug.cpu.sched_count.avg
    274783 ±  5%    +114.4%     589174 ±  1%  sched_debug.cpu.sched_count.max
    168465 ±  4%     +99.1%     335338 ±  1%  sched_debug.cpu.sched_count.min
     22822 ±  2%    +312.2%      94072 ±  3%  sched_debug.cpu.sched_count.stddev
     75339 ±  3%    +139.9%     180754 ±  1%  sched_debug.cpu.sched_goidle.avg
    102100 ±  9%    +139.1%     244171 ±  2%  sched_debug.cpu.sched_goidle.max
     57725 ±  4%    +122.0%     128167 ±  0%  sched_debug.cpu.sched_goidle.min
     10268 ±  5%    +341.2%      45302 ±  3%  sched_debug.cpu.sched_goidle.stddev
     79906 ±  3%    +134.4%     187304 ±  2%  sched_debug.cpu.ttwu_count.avg
    101904 ±  2%    +125.4%     229689 ±  3%  sched_debug.cpu.ttwu_count.max
     66774 ±  5%    +132.6%     155346 ±  2%  sched_debug.cpu.ttwu_count.min
      7049 ± 16%    +236.1%      23693 ±  4%  sched_debug.cpu.ttwu_count.stddev
     30661 ±  2%     +47.1%      45116 ±  1%  sched_debug.cpu.ttwu_local.avg
     46199 ±  6%     +27.0%      58654 ± 10%  sched_debug.cpu.ttwu_local.max
     25114 ±  3%     +62.4%      40776 ±  1%  sched_debug.cpu.ttwu_local.min
      0.01 ± 13%     -72.0%       0.00 ± 24%  sched_debug.rt_rq:/.rt_nr_running.avg
      0.54 ± 25%     -69.2%       0.17 ±  0%  sched_debug.rt_rq:/.rt_nr_running.max
      0.08 ± 14%     -69.6%       0.03 ± 13%  sched_debug.rt_rq:/.rt_nr_running.stddev
      0.04 ± 14%     -34.3%       0.02 ± 17%  sched_debug.rt_rq:/.rt_time.avg
      1.47 ± 11%     -23.9%       1.12 ± 17%  sched_debug.rt_rq:/.rt_time.max
      0.21 ± 13%     -29.6%       0.15 ± 17%  sched_debug.rt_rq:/.rt_time.stddev
     44.29 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      3.35 ±  9%    +120.7%       7.39 ±  1%  perf-profile.cycles-pp.__do_page_fault.do_page_fault.page_fault
     44.28 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath
      0.00 ± -1%      +Inf%       2.41 ±  5%  perf-profile.cycles-pp.__libc_fork
      0.00 ± -1%      +Inf%       1.15 ±  7%  perf-profile.cycles-pp.__split_vma.isra.36.split_vma.mprotect_fixup.sys_mprotect.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.54 ±  7%  perf-profile.cycles-pp._dl_addr
      3.56 ± 10%     +42.2%       5.06 ±  4%  perf-profile.cycles-pp._do_fork.sys_clone.do_syscall_64.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%       1.88 ±  5%  perf-profile.cycles-pp._do_fork.sys_clone.do_syscall_64.return_from_SYSCALL_64.__libc_fork
      2.74 ±  8%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp._raw_spin_lock.dcache_dir_close.__fput.____fput.task_work_run
     17.16 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp._raw_spin_lock.next_positive.isra.13.dcache_readdir.iterate_dir.sys_getdents
     34.60 ±  6%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp._raw_spin_trylock.dput.dcache_dir_close.__fput.____fput
      0.00 ± -1%      +Inf%       1.00 ± 20%  perf-profile.cycles-pp.apic_timer_interrupt
     17.77 ± 10%    +152.2%      44.81 ±  3%  perf-profile.cycles-pp.call_cpuidle.cpu_startup_entry.start_secondary
      1.24 ±  7%     +43.4%       1.79 ±  2%  perf-profile.cycles-pp.copy_page_range.copy_process.part.30._do_fork.sys_clone.do_syscall_64
      3.37 ± 10%     +90.7%       6.42 ±  2%  perf-profile.cycles-pp.copy_process.part.30._do_fork.sys_clone.do_syscall_64.return_from_SYSCALL_64
     18.16 ± 10%    +153.0%      45.94 ±  3%  perf-profile.cycles-pp.cpu_startup_entry.start_secondary
     17.77 ± 10%    +152.2%      44.81 ±  3%  perf-profile.cycles-pp.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
     17.72 ± 10%    +151.8%      44.62 ±  3%  perf-profile.cycles-pp.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
     44.20 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.dcache_dir_close.__fput.____fput.task_work_run.exit_to_usermode_loop
     18.78 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.dcache_readdir.iterate_dir.sys_getdents.entry_SYSCALL_64_fastpath
      0.92 ±  8%     -66.8%       0.30 ±100%  perf-profile.cycles-pp.do_brk.sys_brk.entry_SYSCALL_64_fastpath
      0.64 ±  7%    +828.0%       5.96 ±  6%  perf-profile.cycles-pp.do_execveat_common.isra.34.sys_execve.do_syscall_64.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%       4.21 ±  6%  perf-profile.cycles-pp.do_execveat_common.isra.34.sys_execve.do_syscall_64.return_from_SYSCALL_64.execve
      3.94 ±  9%     +66.8%       6.58 ±  1%  perf-profile.cycles-pp.do_exit.do_group_exit.sys_exit_group.entry_SYSCALL_64_fastpath
      0.70 ± 13%     +42.2%       1.00 ±  2%  perf-profile.cycles-pp.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      3.96 ±  9%     +66.8%       6.60 ±  1%  perf-profile.cycles-pp.do_group_exit.sys_exit_group.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       2.79 ±  6%  perf-profile.cycles-pp.do_mmap.vm_mmap_pgoff.sys_mmap_pgoff.sys_mmap.entry_SYSCALL_64_fastpath
      3.38 ±  9%    +120.5%       7.45 ±  1%  perf-profile.cycles-pp.do_page_fault.page_fault
      0.76 ± 13%     +40.8%       1.08 ±  1%  perf-profile.cycles-pp.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      4.29 ±  9%    +163.2%      11.29 ±  2%  perf-profile.cycles-pp.do_syscall_64.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%       1.89 ±  5%  perf-profile.cycles-pp.do_syscall_64.return_from_SYSCALL_64.__libc_fork
      0.00 ± -1%      +Inf%       4.23 ±  6%  perf-profile.cycles-pp.do_syscall_64.return_from_SYSCALL_64.execve
      0.79 ± 10%     +83.0%       1.46 ±  4%  perf-profile.cycles-pp.do_wp_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault
     41.17 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.dput.dcache_dir_close.__fput.____fput.task_work_run
      0.00 ± -1%      +Inf%       1.50 ±  4%  perf-profile.cycles-pp.elf_map.load_elf_binary.search_binary_handler.do_execveat_common.isra.34.sys_execve
     71.57 ±  4%     -74.9%      17.98 ±  1%  perf-profile.cycles-pp.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       4.24 ±  6%  perf-profile.cycles-pp.execve
      3.37 ±  9%     +55.3%       5.23 ±  1%  perf-profile.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.sys_exit_group
      0.00 ± -1%      +Inf%       3.83 ±  6%  perf-profile.cycles-pp.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler
     44.45 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      1.47 ±  8%    +116.2%       3.17 ±  1%  perf-profile.cycles-pp.filemap_map_pages.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.00 ± -1%      +Inf%       3.91 ±  6%  perf-profile.cycles-pp.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.isra.34.sys_execve
      0.74 ± 11%     +68.2%       1.25 ±  3%  perf-profile.cycles-pp.free_pages_and_swap_cache.tlb_flush_mmu_free.tlb_finish_mmu.exit_mmap.mmput
      0.79 ±  9%     +64.8%       1.31 ±  5%  perf-profile.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
      0.00 ± -1%      +Inf%       1.03 ± 17%  perf-profile.cycles-pp.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary
      2.92 ±  9%    +121.6%       6.46 ±  1%  perf-profile.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      2.68 ±  8%    +116.5%       5.81 ±  1%  perf-profile.cycles-pp.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.00 ± -1%      +Inf%       0.79 ± 18%  perf-profile.cycles-pp.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
     17.58 ± 10%    +153.8%      44.61 ±  3%  perf-profile.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry
     19.14 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.iterate_dir.sys_getdents.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.05 ±  4%  perf-profile.cycles-pp.kthread.ret_from_fork
      0.26 ±100%   +3007.7%       8.08 ±  6%  perf-profile.cycles-pp.load_elf_binary.search_binary_handler.do_execveat_common.isra.34.sys_execve.do_syscall_64
      0.00 ± -1%      +Inf%       0.81 ± 19%  perf-profile.cycles-pp.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      2.77 ± 10%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.lockref_put_return.dput.dcache_dir_close.__fput.____fput
      0.00 ± -1%      +Inf%       2.62 ±  6%  perf-profile.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.sys_mmap_pgoff.sys_mmap
      3.38 ±  9%     +55.5%       5.25 ±  1%  perf-profile.cycles-pp.mmput.do_exit.do_group_exit.sys_exit_group.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       3.86 ±  6%  perf-profile.cycles-pp.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.isra.34
      0.00 ± -1%      +Inf%       2.09 ± 10%  perf-profile.cycles-pp.mprotect_fixup.sys_mprotect.entry_SYSCALL_64_fastpath
     17.10 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.next_positive.isra.13.dcache_readdir.iterate_dir
     17.23 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.next_positive.isra.13.dcache_readdir.iterate_dir.sys_getdents.entry_SYSCALL_64_fastpath
      3.39 ±  9%    +120.4%       7.48 ±  1%  perf-profile.cycles-pp.page_fault
      0.69 ± 13%     +42.2%       0.98 ±  1%  perf-profile.cycles-pp.path_openat.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.11 ± 10%  perf-profile.cycles-pp.perf_event_aux.part.51.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff
      0.00 ± -1%      +Inf%       1.22 ± 10%  perf-profile.cycles-pp.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff.sys_mmap_pgoff
      0.61 ± 11%     +63.0%       1.00 ±  4%  perf-profile.cycles-pp.release_pages.free_pages_and_swap_cache.tlb_flush_mmu_free.tlb_finish_mmu.exit_mmap
      0.12 ±173%    +788.0%       1.11 ±  4%  perf-profile.cycles-pp.ret_from_fork
      4.30 ±  9%    +163.1%      11.30 ±  2%  perf-profile.cycles-pp.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%       1.89 ±  5%  perf-profile.cycles-pp.return_from_SYSCALL_64.__libc_fork
      0.00 ± -1%      +Inf%       4.23 ±  6%  perf-profile.cycles-pp.return_from_SYSCALL_64.execve
      0.26 ±100%   +3017.3%       8.11 ±  6%  perf-profile.cycles-pp.search_binary_handler.do_execveat_common.isra.34.sys_execve.do_syscall_64.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%       0.99 ± 20%  perf-profile.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00 ± -1%      +Inf%       1.15 ±  6%  perf-profile.cycles-pp.split_vma.mprotect_fixup.sys_mprotect.entry_SYSCALL_64_fastpath
     18.17 ± 10%    +153.1%      45.99 ±  3%  perf-profile.cycles-pp.start_secondary
      1.16 ±  9%     -54.8%       0.52 ± 58%  perf-profile.cycles-pp.sys_brk.entry_SYSCALL_64_fastpath
      3.57 ± 10%     +44.5%       5.16 ±  4%  perf-profile.cycles-pp.sys_clone.do_syscall_64.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%       1.88 ±  5%  perf-profile.cycles-pp.sys_clone.do_syscall_64.return_from_SYSCALL_64.__libc_fork
      0.77 ± 13%     +40.9%       1.08 ±  1%  perf-profile.cycles-pp.sys_creat.entry_SYSCALL_64_fastpath
      0.65 ±  7%    +832.2%       6.08 ±  7%  perf-profile.cycles-pp.sys_execve.do_syscall_64.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%       4.23 ±  6%  perf-profile.cycles-pp.sys_execve.do_syscall_64.return_from_SYSCALL_64.execve
      3.96 ±  9%     +66.8%       6.60 ±  1%  perf-profile.cycles-pp.sys_exit_group.entry_SYSCALL_64_fastpath
     19.15 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.sys_getdents.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       2.90 ±  6%  perf-profile.cycles-pp.sys_mmap.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       2.90 ±  6%  perf-profile.cycles-pp.sys_mmap_pgoff.sys_mmap.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       2.21 ± 10%  perf-profile.cycles-pp.sys_mprotect.entry_SYSCALL_64_fastpath
     44.46 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.syscall_return_slowpath.entry_SYSCALL_64_fastpath
     44.31 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      0.76 ± 11%     +68.6%       1.28 ±  3%  perf-profile.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit
      0.74 ± 11%     +68.4%       1.25 ±  3%  perf-profile.cycles-pp.tlb_flush_mmu_free.tlb_finish_mmu.exit_mmap.mmput.do_exit
      1.54 ±  8%    +107.8%       3.21 ±  2%  perf-profile.cycles-pp.unmap_page_range.unmap_single_vma.unmap_vmas.exit_mmap.mmput
      1.62 ±  8%     +44.5%       2.35 ±  1%  perf-profile.cycles-pp.unmap_single_vma.unmap_vmas.exit_mmap.mmput.do_exit
      0.00 ± -1%      +Inf%       1.23 ± 23%  perf-profile.cycles-pp.unmap_single_vma.unmap_vmas.exit_mmap.mmput.flush_old_exec
      1.64 ±  8%     +44.1%       2.37 ±  1%  perf-profile.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      0.00 ± -1%      +Inf%       1.28 ± 24%  perf-profile.cycles-pp.unmap_vmas.exit_mmap.mmput.flush_old_exec.load_elf_binary
      0.00 ± -1%      +Inf%       0.96 ± 27%  perf-profile.cycles-pp.vm_mmap.elf_map.load_elf_binary.search_binary_handler.do_execveat_common.isra.34
      0.00 ± -1%      +Inf%       2.87 ±  6%  perf-profile.cycles-pp.vm_mmap_pgoff.sys_mmap_pgoff.sys_mmap.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       0.95 ± 27%  perf-profile.cycles-pp.vm_mmap_pgoff.vm_mmap.elf_map.load_elf_binary.search_binary_handler




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Xiaolong

View attachment "config-4.7.0-rc1-00008-g64e3f97" of type "text/plain" (150857 bytes)

View attachment "job.yaml" of type "text/plain" (3546 bytes)

View attachment "reproduce" of type "text/plain" (9468 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ