lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Mon, 22 Jun 2020 10:23:00 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
Cc:     linux-kernel@...r.kernel.org,
        Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Peter Zijlstra <peterz@...radead.org>,
        Joel Fernandes <joelaf@...gle.com>,
        "H . Peter Anvin" <hpa@...or.com>, Paul Turner <pjt@...gle.com>,
        Boqun Feng <boqun.feng@...il.com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Andy Lutomirski <luto@...capital.net>,
        "Paul E. McKenney" <paulmck@...nel.org>,
        Julien Desfossez <ju@...pix.org>, lkp@...ts.01.org
Subject: [sched_pair_cpu] e9f2fb8893: will-it-scale.per_thread_ops -67.5%
 regression

Greeting,

FYI, we noticed a -67.5% regression of will-it-scale.per_thread_ops due to commit:


commit: e9f2fb8893b705bcabebf3c62b663e030faf1f4a ("[RFC PATCH v2] sched_pair_cpu: Introduce scheduler task pairing system call")
url: https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/sched_pair_cpu-Introduce-scheduler-task-pairing-system-call/20200620-042752


in testcase: will-it-scale
on test machine: 192 threads Cooper Lake with 128G memory
with following parameters:

	nr_task: 100%
	mode: thread
	test: context_switch1
	cpufreq_governor: performance
	ucode: 0x86000017

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -37.5% regression |
| test machine     | 104 threads Skylake with 192G memory                          |
| test parameters  | cpufreq_governor=performance                                  |
|                  | mode=thread                                                   |
|                  | nr_task=100%                                                  |
|                  | test=context_switch1                                          |
|                  | ucode=0x2000065                                               |
+------------------+---------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-cpx-4s1/context_switch1/will-it-scale/0x86000017

commit: 
  87e867b426 ("sched/pelt: Cleanup PELT divider")
  e9f2fb8893 ("sched_pair_cpu: Introduce scheduler task pairing system call")

87e867b4269f29da e9f2fb8893b705bcabebf3c62b6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    381109           -67.5%     123961        will-it-scale.per_thread_ops
  73173098           -67.5%   23800625        will-it-scale.workload
    629488 ± 17%     +58.6%     998073 ± 42%  cpuidle.C1E.usage
    399957 ± 69%     -66.4%     134211 ±162%  numa-numastat.node2.local_node
    412420 ± 67%     -59.9%     165247 ±131%  numa-numastat.node2.numa_hit
     89.50            +6.4%      95.25        vmstat.cpu.sy
  72482192           -67.6%   23500061        vmstat.system.cs
      0.00 ±117%      -0.0        0.00 ±173%  mpstat.cpu.all.iowait%
      0.00 ± 48%      +0.0        0.00 ± 35%  mpstat.cpu.all.soft%
      9.34            -6.5        2.82        mpstat.cpu.all.usr%
    672603 ±  5%      -9.7%     607172 ±  9%  meminfo.AnonPages
     48453           +27.4%      61725        meminfo.Inactive
     48054           +27.6%      61327        meminfo.Inactive(anon)
     49405 ±  2%     +27.7%      63074        meminfo.Mapped
    205187 ±  2%     +28.0%     262594 ±  3%  meminfo.Shmem
    746.00 ± 12%     +21.7%     908.00 ±  4%  slabinfo.khugepaged_mm_slot.active_objs
    746.00 ± 12%     +21.7%     908.00 ±  4%  slabinfo.khugepaged_mm_slot.num_objs
    165639           +40.7%     233094        slabinfo.kmalloc-64.active_objs
      2590           +40.7%       3646        slabinfo.kmalloc-64.active_slabs
    165819           +40.7%     233344        slabinfo.kmalloc-64.num_objs
      2590           +40.7%       3646        slabinfo.kmalloc-64.num_slabs
      8840 ±  5%     -28.4%       6331 ± 15%  numa-vmstat.node2.nr_slab_reclaimable
    113913 ± 11%     +31.6%     149917        numa-vmstat.node2.numa_other
     65522 ±  7%     +23.1%      80627 ± 13%  numa-vmstat.node3.nr_active_anon
    125181           +11.6%     139689 ±  2%  numa-vmstat.node3.nr_file_pages
      3470 ± 20%     +87.3%       6500 ±  3%  numa-vmstat.node3.nr_inactive_anon
      6154 ±  2%     +20.8%       7437 ±  5%  numa-vmstat.node3.nr_kernel_stack
      3934 ± 12%     +50.3%       5912        numa-vmstat.node3.nr_mapped
     42101 ±  2%     +33.2%      56082 ±  3%  numa-vmstat.node3.nr_shmem
     65522 ±  7%     +23.1%      80627 ± 13%  numa-vmstat.node3.nr_zone_active_anon
      3470 ± 20%     +87.3%       6500 ±  3%  numa-vmstat.node3.nr_zone_inactive_anon
     35363 ±  5%     -28.4%      25329 ± 15%  numa-meminfo.node2.KReclaimable
     35363 ±  5%     -28.4%      25329 ± 15%  numa-meminfo.node2.SReclaimable
    262207 ±  7%     +23.0%     322429 ± 13%  numa-meminfo.node3.Active
    262207 ±  7%     +23.0%     322429 ± 13%  numa-meminfo.node3.Active(anon)
    501055           +11.5%     558705 ±  2%  numa-meminfo.node3.FilePages
     14049 ± 22%     +84.6%      25936 ±  2%  numa-meminfo.node3.Inactive
     13989 ± 23%     +85.4%      25936 ±  2%  numa-meminfo.node3.Inactive(anon)
      6154 ±  2%     +20.9%       7438 ±  5%  numa-meminfo.node3.KernelStack
     15606 ± 14%     +49.0%      23254 ±  2%  numa-meminfo.node3.Mapped
    985437 ±  2%     +17.7%    1159682 ±  8%  numa-meminfo.node3.MemUsed
    168735 ±  2%     +32.9%     224275 ±  3%  numa-meminfo.node3.Shmem
    168135 ±  5%      -9.8%     151702 ±  9%  proc-vmstat.nr_anon_pages
    388347            +3.7%     402665        proc-vmstat.nr_file_pages
     11913           +28.7%      15327        proc-vmstat.nr_inactive_anon
     32244            +9.5%      35299        proc-vmstat.nr_kernel_stack
     12356 ±  2%     +28.3%      15853        proc-vmstat.nr_mapped
     51267 ±  2%     +28.1%      65649 ±  3%  proc-vmstat.nr_shmem
     29556            +1.8%      30086        proc-vmstat.nr_slab_reclaimable
     80332            +3.3%      82953        proc-vmstat.nr_slab_unreclaimable
     11913           +28.7%      15327        proc-vmstat.nr_zone_inactive_anon
   1007396            +1.9%    1026170        proc-vmstat.numa_hit
    914187            +2.1%     932960        proc-vmstat.numa_local
     60211 ±  3%     +39.0%      83671 ±  3%  proc-vmstat.pgactivate
   1010160            +7.5%    1085798 ±  4%  proc-vmstat.pgfree
      0.17 ± 99%    +975.0%       1.79 ± 42%  sched_debug.cfs_rq:/.load_avg.min
    132891 ±  9%     +30.1%     172957 ±  3%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.14 ±  6%     +27.5%       0.18 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
    -33482          +404.5%    -168935        sched_debug.cfs_rq:/.spread0.avg
    196291 ± 15%     -82.8%      33829 ± 82%  sched_debug.cfs_rq:/.spread0.max
  -1178050           +78.9%   -2107947        sched_debug.cfs_rq:/.spread0.min
    132832 ±  9%     +29.8%     172431 ±  3%  sched_debug.cfs_rq:/.spread0.stddev
    581.51           +17.2%     681.44        sched_debug.cfs_rq:/.util_est_enqueued.avg
      2094 ±  4%     +70.7%       3574 ±  3%  sched_debug.cpu.avg_idle.min
     37.62 ±  2%     +63.8%      61.62 ±  2%  sched_debug.cpu.clock.stddev
     37.62 ±  2%     +63.8%      61.62 ±  2%  sched_debug.cpu.clock_task.stddev
      7029 ±  5%     +11.3%       7822        sched_debug.cpu.curr->pid.max
  56823172           -67.4%   18533495        sched_debug.cpu.nr_switches.avg
  62261802 ±  2%     -66.7%   20760559        sched_debug.cpu.nr_switches.max
  52691982           -72.7%   14402197        sched_debug.cpu.nr_switches.min
     17.36 ±  3%     +12.0%      19.44 ±  3%  sched_debug.cpu.nr_uninterruptible.stddev
 1.176e+11           -57.9%  4.952e+10        perf-stat.i.branch-instructions
      1.14            -0.2        0.98        perf-stat.i.branch-miss-rate%
 1.324e+09           -63.8%  4.798e+08        perf-stat.i.branch-misses
      0.14 ±  5%      +8.1        8.23 ±  2%  perf-stat.i.cache-miss-rate%
   1925606 ±  8%   +4653.6%   91535573        perf-stat.i.cache-misses
  2.62e+09 ±  8%     -57.3%  1.119e+09 ±  2%  perf-stat.i.cache-references
  72846121           -67.6%   23638038        perf-stat.i.context-switches
      1.13          +185.5%       3.22        perf-stat.i.cpi
 6.637e+11           +14.4%   7.59e+11        perf-stat.i.cpu-cycles
    404127 ± 11%     -98.0%       8282        perf-stat.i.cycles-between-cache-misses
      0.00 ±  6%      +0.0        0.00 ± 76%  perf-stat.i.dTLB-load-miss-rate%
 1.707e+11           -61.8%  6.516e+10        perf-stat.i.dTLB-loads
      0.00            +0.0        0.00 ±  9%  perf-stat.i.dTLB-store-miss-rate%
    114494 ±  2%     -30.8%      79221 ±  8%  perf-stat.i.dTLB-store-misses
 1.046e+11           -61.5%  4.032e+10        perf-stat.i.dTLB-stores
 4.538e+08           -59.7%  1.827e+08        perf-stat.i.iTLB-load-misses
 5.902e+11           -60.1%  2.357e+11        perf-stat.i.instructions
      0.89           -64.9%       0.31        perf-stat.i.ipc
      3.46           +14.5%       3.96        perf-stat.i.metric.GHz
      1.44           -57.0%       0.62 ±  8%  perf-stat.i.metric.K/sec
      2061           -60.4%     815.41        perf-stat.i.metric.M/sec
    298276 ±  4%   +2620.5%    8114651        perf-stat.i.node-load-misses
     47637 ± 36%   +1216.0%     626889        perf-stat.i.node-loads
     92.96           -11.0       81.95        perf-stat.i.node-store-miss-rate%
     87054 ±  7%   +1416.2%    1319891        perf-stat.i.node-store-misses
     10937 ± 12%   +2569.1%     291918        perf-stat.i.node-stores
      1.13            -0.2        0.97        perf-stat.overall.branch-miss-rate%
      0.07 ± 12%      +8.1        8.18 ±  2%  perf-stat.overall.cache-miss-rate%
      1.12          +186.1%       3.22        perf-stat.overall.cpi
    349097 ±  8%     -97.6%       8304        perf-stat.overall.cycles-between-cache-misses
      0.00 ±  8%      +0.0        0.00 ± 71%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  2%      +0.0        0.00 ±  7%  perf-stat.overall.dTLB-store-miss-rate%
      0.89           -65.0%       0.31        perf-stat.overall.ipc
     85.95 ±  5%      +6.8       92.78        perf-stat.overall.node-load-miss-rate%
     88.64            -7.0       81.67        perf-stat.overall.node-store-miss-rate%
   2429767           +23.4%    2998949        perf-stat.overall.path-length
 1.172e+11           -57.9%   4.94e+10        perf-stat.ps.branch-instructions
 1.319e+09           -63.8%  4.782e+08        perf-stat.ps.branch-misses
   1909280 ±  8%   +4671.3%   91098325        perf-stat.ps.cache-misses
 2.609e+09 ±  8%     -57.3%  1.115e+09 ±  2%  perf-stat.ps.cache-references
  72607352           -67.5%   23581726        perf-stat.ps.context-switches
 6.615e+11           +14.4%  7.565e+11        perf-stat.ps.cpu-cycles
 1.701e+11           -61.8%    6.5e+10        perf-stat.ps.dTLB-loads
    114417 ±  2%     -30.1%      79978 ±  7%  perf-stat.ps.dTLB-store-misses
 1.043e+11           -61.4%  4.022e+10        perf-stat.ps.dTLB-stores
 4.523e+08           -59.7%  1.823e+08        perf-stat.ps.iTLB-load-misses
 5.882e+11           -60.0%  2.352e+11        perf-stat.ps.instructions
    296909 ±  3%   +2620.2%    8076510        perf-stat.ps.node-load-misses
     49167 ± 37%   +1177.6%     628163        perf-stat.ps.node-loads
     86332 ±  7%   +1424.9%    1316449        perf-stat.ps.node-store-misses
     10984 ± 12%   +2590.3%     295531        perf-stat.ps.node-stores
 1.778e+14           -59.9%  7.138e+13        perf-stat.total.instructions
     13.91 ±  2%      -7.8        6.16 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     13.78 ±  2%      -7.7        6.08 ±  2%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
     11.75 ±  2%      -7.7        4.07        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.new_sync_read
     13.56 ±  2%      -7.6        6.00 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      7.27            -4.8        2.45        perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      6.04            -4.0        2.08        perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      4.35            -3.0        1.40        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.new_sync_read
      4.48 ±  4%      -2.8        1.69 ±  3%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      5.66 ±  5%      -2.8        2.89 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
     56.05            -2.4       53.61        perf-profile.calltrace.cycles-pp.__GI___libc_read
      3.38            -2.4        0.97        perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      3.10            -2.2        0.89        perf-profile.calltrace.cycles-pp.switch_fpu_return.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      2.94            -2.0        0.91        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read
      2.68            -1.9        0.81        perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      2.80            -1.7        1.06 ±  2%  perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.82            -1.6        0.26 ±100%  perf-profile.calltrace.cycles-pp.__switch_to_asm.__GI___libc_read
      2.09            -1.5        0.57 ±  3%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
      2.09            -1.5        0.63 ±  3%  perf-profile.calltrace.cycles-pp.__switch_to.__GI___libc_read
      1.98 ±  2%      -1.4        0.55        perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
      2.08            -1.4        0.73        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_read
      2.06 ±  6%      -1.2        0.82 ±  6%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.70            -1.1        0.56 ±  2%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
      1.92            -1.1        0.80 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      1.53 ±  5%      -1.0        0.52 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.39            -1.0        0.39 ± 57%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
      2.12 ±  6%      -0.9        1.25 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      1.42            -0.8        0.64        perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.72 ±  2%      -0.7        0.99 ±  2%  perf-profile.calltrace.cycles-pp.native_write_msr.__GI___libc_read
      1.00 ± 13%      -0.7        0.28 ±100%  perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task
      1.23 ±  2%      -0.7        0.55        perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.00            +0.6        0.60 ±  2%  perf-profile.calltrace.cycles-pp.ring_buffer_lock_reserve.trace_vbprintk.__trace_bprintk.__schedule.schedule
      0.00            +1.2        1.21 ±  2%  perf-profile.calltrace.cycles-pp.format_decode.vbin_printf.trace_vbprintk.__trace_bprintk.__schedule
      0.00            +1.7        1.69 ±  2%  perf-profile.calltrace.cycles-pp.vbin_printf.trace_vbprintk.__trace_bprintk.__schedule.schedule
     43.48            +2.3       45.82        perf-profile.calltrace.cycles-pp.__GI___libc_write
     47.40            +3.4       50.76        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     46.70            +3.8       50.53        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     40.47            +4.6       45.12        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      0.00            +4.6        4.65 ±  2%  perf-profile.calltrace.cycles-pp.trace_vbprintk.__trace_bprintk.__schedule.schedule.pipe_read
      0.00            +4.7        4.71 ±  2%  perf-profile.calltrace.cycles-pp.__trace_bprintk.__schedule.schedule.pipe_read.new_sync_read
     40.14            +4.9       45.03        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     38.64            +5.9       44.52        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     36.98            +7.0       43.95        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     42.04            +7.2       49.19        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      0.00            +8.0        8.02        perf-profile.calltrace.cycles-pp.probe_sched_switch.__schedule.schedule.pipe_read.new_sync_read
      0.00            +8.1        8.06        perf-profile.calltrace.cycles-pp.probe_sched_wakeup.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
     40.24            +8.4       48.67        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     34.42            +8.7       43.09        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.75            +9.1       42.86        perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     36.58           +11.0       47.62        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     34.79           +12.5       47.26        perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
     27.84           +13.2       41.02        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
     26.69           +13.9       40.55        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
     26.46           +14.0       40.47        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
     26.05           +14.3       40.35        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
     22.82           +21.0       43.78        perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
     22.05           +21.6       43.61        perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.new_sync_read.vfs_read
      0.00           +22.0       21.99        perf-profile.calltrace.cycles-pp.trace_save_cmdline.tracing_record_taskinfo.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      0.00           +22.0       22.05        perf-profile.calltrace.cycles-pp.tracing_record_taskinfo.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00           +23.3       23.30        perf-profile.calltrace.cycles-pp.trace_save_cmdline.tracing_record_taskinfo_sched_switch.__schedule.schedule.pipe_read
      0.00           +23.4       23.35        perf-profile.calltrace.cycles-pp.tracing_record_taskinfo_sched_switch.__schedule.schedule.pipe_read.new_sync_read
      1.71           +29.0       30.73        perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     13.91 ±  2%      -7.8        6.16 ±  2%  perf-profile.children.cycles-pp.ttwu_do_activate
     11.79 ±  2%      -7.7        4.08        perf-profile.children.cycles-pp.dequeue_task_fair
     13.80 ±  2%      -7.7        6.09 ±  2%  perf-profile.children.cycles-pp.activate_task
     13.59 ±  2%      -7.6        6.00 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
      8.53 ±  3%      -5.1        3.44 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      7.29            -4.8        2.46        perf-profile.children.cycles-pp.select_task_rq_fair
      7.02            -4.4        2.62        perf-profile.children.cycles-pp.update_curr
      6.21            -4.1        2.12        perf-profile.children.cycles-pp.select_idle_sibling
      5.56            -3.7        1.90 ±  2%  perf-profile.children.cycles-pp.reweight_entity
      4.62            -3.2        1.46        perf-profile.children.cycles-pp.pick_next_task_fair
      5.79 ±  4%      -2.9        2.92 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      4.56 ±  4%      -2.9        1.71 ±  3%  perf-profile.children.cycles-pp.dequeue_entity
      3.87            -2.7        1.16        perf-profile.children.cycles-pp.__prepare_exit_to_usermode
     56.06            -2.3       53.76        perf-profile.children.cycles-pp.__GI___libc_read
      3.13            -2.2        0.89        perf-profile.children.cycles-pp.switch_fpu_return
      3.05            -2.1        0.94        perf-profile.children.cycles-pp.prepare_to_wait_event
      2.93            -2.0        0.89 ±  2%  perf-profile.children.cycles-pp.__switch_to
      2.55            -1.9        0.70 ±  2%  perf-profile.children.cycles-pp.mutex_lock
      2.87 ±  8%      -1.8        1.09 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      2.46 ±  3%      -1.7        0.74 ±  3%  perf-profile.children.cycles-pp.security_file_permission
      2.68            -1.6        1.07        perf-profile.children.cycles-pp.update_cfs_group
      2.40            -1.6        0.80        perf-profile.children.cycles-pp.sched_clock_cpu
      2.26            -1.6        0.69        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      2.13            -1.5        0.58 ±  3%  perf-profile.children.cycles-pp.copy_page_to_iter
      2.09            -1.5        0.59 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      2.04 ±  2%      -1.5        0.57        perf-profile.children.cycles-pp.copy_page_from_iter
      2.23            -1.4        0.78 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      2.04 ±  8%      -1.4        0.63        perf-profile.children.cycles-pp.__fdget_pos
      2.16            -1.4        0.75        perf-profile.children.cycles-pp.set_next_entity
      1.92            -1.4        0.54        perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.90            -1.4        0.52 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      1.94 ±  8%      -1.3        0.60        perf-profile.children.cycles-pp.__fget_light
      1.85            -1.3        0.52 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      2.17            -1.3        0.88        perf-profile.children.cycles-pp.sched_clock
      1.80            -1.3        0.53        perf-profile.children.cycles-pp.__update_load_avg_se
      2.09            -1.2        0.85        perf-profile.children.cycles-pp.native_sched_clock
      1.74            -1.2        0.51 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      1.55            -1.1        0.45        perf-profile.children.cycles-pp.check_preempt_curr
      1.52            -1.1        0.43        perf-profile.children.cycles-pp.update_rq_clock
      1.57 ±  5%      -1.1        0.49 ±  5%  perf-profile.children.cycles-pp.common_file_perm
      1.43 ± 12%      -1.0        0.43 ±  3%  perf-profile.children.cycles-pp.__fget_files
      1.50            -1.0        0.53 ±  3%  perf-profile.children.cycles-pp.cpumask_next_wrap
      1.50            -1.0        0.53 ±  2%  perf-profile.children.cycles-pp._find_next_bit
      1.33 ±  2%      -0.9        0.38        perf-profile.children.cycles-pp.fsnotify
      1.30            -0.9        0.36        perf-profile.children.cycles-pp.check_preempt_wakeup
      1.90 ±  2%      -0.8        1.06 ±  3%  perf-profile.children.cycles-pp.native_write_msr
      1.02            -0.7        0.28        perf-profile.children.cycles-pp.__might_sleep
      1.27            -0.7        0.58 ±  4%  perf-profile.children.cycles-pp.update_min_vruntime
      0.98 ±  3%      -0.7        0.29 ±  2%  perf-profile.children.cycles-pp.touch_atime
      0.83            -0.6        0.19 ±  2%  perf-profile.children.cycles-pp._cond_resched
      0.94 ±  5%      -0.6        0.31 ±  4%  perf-profile.children.cycles-pp.current_time
      0.86            -0.6        0.24 ±  5%  perf-profile.children.cycles-pp.__might_fault
      0.89            -0.6        0.27 ±  2%  perf-profile.children.cycles-pp.pick_next_entity
      0.85 ±  2%      -0.6        0.25        perf-profile.children.cycles-pp.fsnotify_parent
      0.86            -0.6        0.26        perf-profile.children.cycles-pp.mutex_unlock
      0.85            -0.6        0.25 ±  3%  perf-profile.children.cycles-pp.__pthread_disable_asynccancel
      0.83            -0.6        0.23 ±  2%  perf-profile.children.cycles-pp.cpumask_next
      0.82            -0.6        0.23 ±  2%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.84            -0.6        0.26        perf-profile.children.cycles-pp.account_entity_dequeue
      0.82 ±  2%      -0.6        0.24 ±  2%  perf-profile.children.cycles-pp.atime_needs_update
      0.75 ± 13%      -0.6        0.20 ± 12%  perf-profile.children.cycles-pp.cpuacct_charge
      0.75            -0.5        0.21 ±  2%  perf-profile.children.cycles-pp.copyout
      0.78 ±  2%      -0.5        0.26 ±  2%  perf-profile.children.cycles-pp.___perf_sw_event
      0.76            -0.5        0.26 ±  3%  perf-profile.children.cycles-pp.account_entity_enqueue
      0.71            -0.5        0.21 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.69            -0.5        0.20        perf-profile.children.cycles-pp.copyin
      0.68            -0.5        0.21 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.79            -0.5        0.33 ±  5%  perf-profile.children.cycles-pp.set_next_buddy
      0.62 ±  2%      -0.4        0.19 ±  3%  perf-profile.children.cycles-pp.available_idle_cpu
      0.85 ±  6%      -0.4        0.42 ±  6%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.88 ±  6%      -0.4        0.46 ±  6%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.80 ±  5%      -0.4        0.38 ±  6%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.80 ±  6%      -0.4        0.38 ±  6%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.52            -0.4        0.12        perf-profile.children.cycles-pp.rcu_all_qs
      0.64            -0.4        0.26 ±  3%  perf-profile.children.cycles-pp.__pthread_enable_asynccancel
      0.53            -0.4        0.15 ±  2%  perf-profile.children.cycles-pp.finish_wait
      0.82            -0.4        0.45 ±  2%  perf-profile.children.cycles-pp.finish_task_switch
      0.73 ±  4%      -0.4        0.37        perf-profile.children.cycles-pp.file_update_time
      0.77 ±  5%      -0.4        0.41 ±  6%  perf-profile.children.cycles-pp.asm_call_on_stack
      0.55            -0.3        0.20 ±  2%  perf-profile.children.cycles-pp.fput_many
      0.46 ±  2%      -0.3        0.13 ±  3%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.47 ±  4%      -0.3        0.17 ±  8%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.41            -0.3        0.12 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.39            -0.3        0.12 ±  3%  perf-profile.children.cycles-pp.clear_buddies
      0.34 ±  2%      -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.__syscall_return_slowpath
      0.35 ± 11%      -0.2        0.11 ±  3%  perf-profile.children.cycles-pp.__sb_start_write
      0.32 ±  2%      -0.2        0.09 ±  4%  perf-profile.children.cycles-pp.wakeup_preempt_entity
      0.31 ±  2%      -0.2        0.09        perf-profile.children.cycles-pp.iov_iter_init
      0.30            -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.put_prev_entity
      0.27 ±  9%      -0.2        0.07        perf-profile.children.cycles-pp.apparmor_file_permission
      0.26 ±  7%      -0.2        0.06 ± 13%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.28 ±  3%      -0.2        0.09 ±  4%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.28            -0.2        0.09        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.24 ±  6%      -0.2        0.06        perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.28            -0.2        0.10        perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.38 ±  3%      -0.2        0.21 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.23            -0.2        0.07 ±  6%  perf-profile.children.cycles-pp.__list_add_valid
      0.23 ± 13%      -0.2        0.08 ±  6%  perf-profile.children.cycles-pp.timestamp_truncate
      0.25 ±  6%      -0.2        0.10        perf-profile.children.cycles-pp.aa_file_perm
      0.21 ±  2%      -0.2        0.06 ±  6%  perf-profile.children.cycles-pp.__sb_end_write
      0.22            -0.1        0.07        perf-profile.children.cycles-pp.find_next_bit
      0.21 ±  2%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__x64_sys_read
      0.30 ±  6%      -0.1        0.16 ±  8%  perf-profile.children.cycles-pp.tick_sched_timer
      0.20            -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.place_entity
      0.21            -0.1        0.07        perf-profile.children.cycles-pp.__x64_sys_write
      0.23 ± 13%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.ktime_get
      0.14 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__bitmap_and
      0.23 ±  2%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.deactivate_task
      0.20 ±  2%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.15 ±  3%      -0.1        0.06        perf-profile.children.cycles-pp.fput
      0.30 ±  2%      -0.1        0.23 ±  3%  perf-profile.children.cycles-pp.rb_insert_color
      0.20 ±  2%      -0.1        0.13 ± 10%  perf-profile.children.cycles-pp.tick_sched_handle
      0.20 ±  2%      -0.1        0.13 ± 10%  perf-profile.children.cycles-pp.update_process_times
      0.15 ± 10%      -0.1        0.09 ±  7%  perf-profile.children.cycles-pp.clockevents_program_event
      0.13 ±  3%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.rb_erase
      0.13 ±  5%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.09 ±  8%      -0.0        0.06 ± 16%  perf-profile.children.cycles-pp.task_tick_fair
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ring_buffer_nest_end
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.tracing_record_taskinfo_skip
      0.00            +0.1        0.08        perf-profile.children.cycles-pp.tracing_generic_entry_update
      0.00            +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.rb_commit
      0.00            +0.2        0.18 ±  6%  perf-profile.children.cycles-pp.ring_buffer_nest_start
      0.00            +0.2        0.20 ±  2%  perf-profile.children.cycles-pp.__rb_reserve_next
      0.00            +0.2        0.21 ±  2%  perf-profile.children.cycles-pp.trace_clock_local
      0.00            +0.2        0.21 ±  2%  perf-profile.children.cycles-pp.ring_buffer_unlock_commit
      0.00            +0.6        0.60 ±  2%  perf-profile.children.cycles-pp.ring_buffer_lock_reserve
      0.00            +1.3        1.26        perf-profile.children.cycles-pp.format_decode
      0.00            +1.7        1.74 ±  2%  perf-profile.children.cycles-pp.vbin_printf
     43.51            +2.5       46.00        perf-profile.children.cycles-pp.__GI___libc_write
      0.00            +4.7        4.70 ±  2%  perf-profile.children.cycles-pp.trace_vbprintk
      0.00            +4.7        4.72 ±  2%  perf-profile.children.cycles-pp.__trace_bprintk
     38.70            +5.9       44.55        perf-profile.children.cycles-pp.ksys_write
     37.07            +6.9       43.99        perf-profile.children.cycles-pp.vfs_write
     42.15            +7.1       49.22        perf-profile.children.cycles-pp.ksys_read
     87.93            +8.0       95.93        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +8.0        8.04        perf-profile.children.cycles-pp.probe_sched_switch
      0.00            +8.1        8.07        perf-profile.children.cycles-pp.probe_sched_wakeup
     40.35            +8.4       48.70        perf-profile.children.cycles-pp.vfs_read
     86.96            +8.7       95.63        perf-profile.children.cycles-pp.do_syscall_64
     34.46            +8.7       43.13        perf-profile.children.cycles-pp.new_sync_write
     33.82            +9.1       42.89        perf-profile.children.cycles-pp.pipe_write
     36.63           +11.0       47.64        perf-profile.children.cycles-pp.new_sync_read
     34.98           +12.3       47.32        perf-profile.children.cycles-pp.pipe_read
     27.87           +13.2       41.03        perf-profile.children.cycles-pp.__wake_up_common_lock
     26.71           +13.8       40.55        perf-profile.children.cycles-pp.__wake_up_common
     26.49           +14.0       40.48        perf-profile.children.cycles-pp.autoremove_wake_function
     26.07           +14.3       40.36        perf-profile.children.cycles-pp.try_to_wake_up
     22.95           +20.9       43.86        perf-profile.children.cycles-pp.schedule
     22.35           +21.4       43.73        perf-profile.children.cycles-pp.__schedule
      0.00           +22.1       22.07        perf-profile.children.cycles-pp.tracing_record_taskinfo
      0.00           +23.4       23.41        perf-profile.children.cycles-pp.tracing_record_taskinfo_sched_switch
      1.74           +29.0       30.74        perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.00           +45.3       45.34        perf-profile.children.cycles-pp.trace_save_cmdline
      3.11            -2.2        0.89        perf-profile.self.cycles-pp.switch_fpu_return
      3.93            -2.1        1.83 ±  3%  perf-profile.self.cycles-pp.update_load_avg
      3.36 ±  3%      -2.0        1.36 ±  2%  perf-profile.self.cycles-pp.update_curr
      2.76            -1.9        0.84 ±  2%  perf-profile.self.cycles-pp.__switch_to
      2.78 ±  8%      -1.7        1.07 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      2.64            -1.6        1.06        perf-profile.self.cycles-pp.update_cfs_group
      2.17            -1.5        0.66        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.89            -1.4        0.52 ±  3%  perf-profile.self.cycles-pp.__switch_to_asm
      1.89            -1.4        0.53        perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.88            -1.3        0.55 ±  3%  perf-profile.self.cycles-pp.__schedule
      1.89            -1.3        0.57 ±  2%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      1.81            -1.3        0.51 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      1.75            -1.3        0.50 ±  2%  perf-profile.self.cycles-pp.pipe_read
      1.75            -1.2        0.52        perf-profile.self.cycles-pp.__update_load_avg_se
      1.72            -1.2        0.51 ±  2%  perf-profile.self.cycles-pp.__calc_delta
      1.85            -1.2        0.64        perf-profile.self.cycles-pp.select_idle_sibling
      1.65            -1.2        0.44        perf-profile.self.cycles-pp.reweight_entity
      2.03            -1.2        0.83        perf-profile.self.cycles-pp.native_sched_clock
      2.08            -1.2        0.90 ±  3%  perf-profile.self.cycles-pp.enqueue_task_fair
      1.44            -1.2        0.28 ±  4%  perf-profile.self.cycles-pp.new_sync_read
      1.40 ± 12%      -1.0        0.42 ±  4%  perf-profile.self.cycles-pp.__fget_files
      1.39 ±  2%      -1.0        0.41 ±  2%  perf-profile.self.cycles-pp.do_syscall_64
      1.49            -1.0        0.52 ±  2%  perf-profile.self.cycles-pp._find_next_bit
      1.32 ±  7%      -0.9        0.39 ±  6%  perf-profile.self.cycles-pp.common_file_perm
      1.28 ±  2%      -0.9        0.37 ±  2%  perf-profile.self.cycles-pp.fsnotify
      1.33            -0.9        0.42        perf-profile.self.cycles-pp.prepare_to_wait_event
      1.90 ±  2%      -0.8        1.06 ±  3%  perf-profile.self.cycles-pp.native_write_msr
      1.06            -0.8        0.30 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.08            -0.7        0.34        perf-profile.self.cycles-pp.select_task_rq_fair
      1.05            -0.7        0.31 ±  3%  perf-profile.self.cycles-pp.vfs_read
      1.03            -0.7        0.32        perf-profile.self.cycles-pp.dequeue_task_fair
      1.00            -0.7        0.29 ±  2%  perf-profile.self.cycles-pp.pipe_write
      1.25            -0.7        0.57 ±  4%  perf-profile.self.cycles-pp.update_min_vruntime
      1.26 ± 10%      -0.7        0.59 ±  7%  perf-profile.self.cycles-pp.enqueue_entity
      0.97            -0.7        0.31 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.91            -0.7        0.25 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      0.95            -0.6        0.30        perf-profile.self.cycles-pp.mutex_lock
      0.88            -0.6        0.25        perf-profile.self.cycles-pp.pick_next_task_fair
      0.80 ±  3%      -0.6        0.21 ±  3%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.90            -0.6        0.31 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.83            -0.6        0.26        perf-profile.self.cycles-pp.mutex_unlock
      0.82            -0.6        0.25        perf-profile.self.cycles-pp.__pthread_disable_asynccancel
      0.79 ±  2%      -0.6        0.23        perf-profile.self.cycles-pp.__GI___libc_write
      0.78            -0.6        0.23 ±  2%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.76 ±  2%      -0.6        0.21 ±  2%  perf-profile.self.cycles-pp.__GI___libc_read
      0.75 ± 13%      -0.6        0.20 ± 13%  perf-profile.self.cycles-pp.cpuacct_charge
      0.76 ±  2%      -0.5        0.23 ±  3%  perf-profile.self.cycles-pp.fsnotify_parent
      0.74 ±  3%      -0.5        0.26 ±  3%  perf-profile.self.cycles-pp.vfs_write
      0.71            -0.5        0.23        perf-profile.self.cycles-pp.account_entity_dequeue
      0.68            -0.5        0.21 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.58            -0.5        0.11 ±  4%  perf-profile.self.cycles-pp.schedule
      0.69 ±  2%      -0.5        0.23        perf-profile.self.cycles-pp.___perf_sw_event
      0.70            -0.5        0.24 ±  3%  perf-profile.self.cycles-pp.account_entity_enqueue
      0.78            -0.5        0.33 ±  5%  perf-profile.self.cycles-pp.set_next_buddy
      0.66            -0.4        0.21 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.58            -0.4        0.16        perf-profile.self.cycles-pp.ksys_read
      0.56            -0.4        0.16 ±  2%  perf-profile.self.cycles-pp.cpumask_next_wrap
      0.51 ±  3%      -0.4        0.12 ±  3%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.53 ±  2%      -0.4        0.15 ±  5%  perf-profile.self.cycles-pp.update_rq_clock
      0.55 ±  2%      -0.4        0.17 ±  4%  perf-profile.self.cycles-pp.available_idle_cpu
      0.61            -0.4        0.25 ±  3%  perf-profile.self.cycles-pp.__pthread_enable_asynccancel
      0.49 ±  2%      -0.4        0.13 ±  3%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.51            -0.4        0.15 ±  3%  perf-profile.self.cycles-pp.pick_next_entity
      0.54 ±  4%      -0.4        0.19 ±  6%  perf-profile.self.cycles-pp.new_sync_write
      0.52            -0.3        0.20 ±  2%  perf-profile.self.cycles-pp.fput_many
      0.42            -0.3        0.10        perf-profile.self.cycles-pp.rcu_all_qs
      0.45 ±  3%      -0.3        0.13        perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.47 ± 14%      -0.3        0.16 ±  4%  perf-profile.self.cycles-pp.__fget_light
      0.47 ±  4%      -0.3        0.17 ±  8%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.39 ± 10%      -0.3        0.11        perf-profile.self.cycles-pp.security_file_permission
      0.40            -0.3        0.12 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.38            -0.3        0.11 ±  7%  perf-profile.self.cycles-pp.__prepare_exit_to_usermode
      0.42            -0.3        0.15 ±  2%  perf-profile.self.cycles-pp.set_next_entity
      0.34            -0.3        0.07        perf-profile.self.cycles-pp._cond_resched
      0.46            -0.2        0.21 ±  2%  perf-profile.self.cycles-pp.finish_task_switch
      0.33 ±  8%      -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.atime_needs_update
      0.35 ± 11%      -0.2        0.11 ±  3%  perf-profile.self.cycles-pp.__sb_start_write
      0.33            -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.clear_buddies
      0.31            -0.2        0.10        perf-profile.self.cycles-pp.__syscall_return_slowpath
      0.28            -0.2        0.07        perf-profile.self.cycles-pp.iov_iter_init
      0.30 ±  3%      -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.wakeup_preempt_entity
      0.26 ±  8%      -0.2        0.06 ± 11%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.25 ± 10%      -0.2        0.06        perf-profile.self.cycles-pp.apparmor_file_permission
      0.26            -0.2        0.08 ±  6%  perf-profile.self.cycles-pp.autoremove_wake_function
      0.23 ±  7%      -0.2        0.06 ±  7%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.27            -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.ksys_write
      0.23 ±  6%      -0.2        0.06 ±  9%  perf-profile.self.cycles-pp.current_time
      0.25            -0.2        0.08        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.26 ±  2%      -0.2        0.10 ±  5%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.21 ±  2%      -0.2        0.05        perf-profile.self.cycles-pp.sched_clock_cpu
      0.23            -0.2        0.07 ±  6%  perf-profile.self.cycles-pp.put_prev_entity
      0.23 ±  3%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.check_preempt_curr
      0.21 ±  2%      -0.2        0.06        perf-profile.self.cycles-pp.__sb_end_write
      0.22 ± 13%      -0.1        0.07        perf-profile.self.cycles-pp.timestamp_truncate
      0.32 ±  3%      -0.1        0.17 ±  2%  perf-profile.self.cycles-pp.dequeue_entity
      0.22 ±  3%      -0.1        0.07        perf-profile.self.cycles-pp.__wake_up_common
      0.20 ±  2%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__list_add_valid
      0.19 ±  2%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__x64_sys_read
      0.23 ± 15%      -0.1        0.09 ± 12%  perf-profile.self.cycles-pp.ktime_get
      0.23 ±  8%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.aa_file_perm
      0.20            -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.__x64_sys_write
      0.20 ±  2%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.18 ±  2%      -0.1        0.05        perf-profile.self.cycles-pp.place_entity
      0.18 ±  2%      -0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.23 ±  2%      -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.deactivate_task
      0.18 ±  6%      -0.1        0.07        perf-profile.self.cycles-pp.activate_task
      0.16 ±  5%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__might_fault
      0.20 ±  2%      -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.27            -0.1        0.20 ±  2%  perf-profile.self.cycles-pp.file_update_time
      0.29 ±  2%      -0.1        0.22 ±  4%  perf-profile.self.cycles-pp.rb_insert_color
      0.12 ±  3%      -0.0        0.07        perf-profile.self.cycles-pp.ttwu_do_activate
      0.11 ±  3%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.rb_erase
      0.21 ±  2%      -0.0        0.17 ± 10%  perf-profile.self.cycles-pp.ttwu_do_wakeup
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.tracing_record_taskinfo_skip
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.ring_buffer_nest_end
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.trace_clock_local
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.tracing_generic_entry_update
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.ring_buffer_unlock_commit
      0.00            +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.rb_commit
      0.00            +0.2        0.18 ±  6%  perf-profile.self.cycles-pp.ring_buffer_nest_start
      0.00            +0.2        0.18 ±  4%  perf-profile.self.cycles-pp.ring_buffer_lock_reserve
      0.00            +0.2        0.19 ±  2%  perf-profile.self.cycles-pp.__rb_reserve_next
      0.00            +0.5        0.51 ±  3%  perf-profile.self.cycles-pp.vbin_printf
      0.00            +1.2        1.22 ±  2%  perf-profile.self.cycles-pp.format_decode
      0.00            +1.8        1.81 ±  9%  perf-profile.self.cycles-pp.trace_vbprintk
      0.00            +8.0        8.03        perf-profile.self.cycles-pp.probe_sched_switch
      0.00            +8.1        8.06        perf-profile.self.cycles-pp.probe_sched_wakeup
      0.00           +45.3       45.33        perf-profile.self.cycles-pp.trace_save_cmdline
    217.75 ± 14%   +2074.2%       4734 ± 89%  interrupts.33:PCI-MSI.524291-edge.eth0-TxRx-2
      9726           +14.6%      11141        interrupts.CPU0.NMI:Non-maskable_interrupts
      9726           +14.6%      11141        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      8509 ± 24%     +30.9%      11137        interrupts.CPU1.NMI:Non-maskable_interrupts
      8509 ± 24%     +30.9%      11137        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      9727           +14.5%      11137        interrupts.CPU10.NMI:Non-maskable_interrupts
      9727           +14.5%      11137        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      9730           +14.5%      11138        interrupts.CPU100.NMI:Non-maskable_interrupts
      9730           +14.5%      11138        interrupts.CPU100.PMI:Performance_monitoring_interrupts
      9727           +14.5%      11140        interrupts.CPU101.NMI:Non-maskable_interrupts
      9727           +14.5%      11140        interrupts.CPU101.PMI:Performance_monitoring_interrupts
      9724           +14.6%      11141        interrupts.CPU102.NMI:Non-maskable_interrupts
      9724           +14.6%      11141        interrupts.CPU102.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11138        interrupts.CPU103.NMI:Non-maskable_interrupts
      9725           +14.5%      11138        interrupts.CPU103.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11139        interrupts.CPU104.NMI:Non-maskable_interrupts
      9725           +14.5%      11139        interrupts.CPU104.PMI:Performance_monitoring_interrupts
      9726           +14.5%      11139        interrupts.CPU105.NMI:Non-maskable_interrupts
      9726           +14.5%      11139        interrupts.CPU105.PMI:Performance_monitoring_interrupts
      9726           +14.5%      11139        interrupts.CPU106.NMI:Non-maskable_interrupts
      9726           +14.5%      11139        interrupts.CPU106.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11140        interrupts.CPU109.NMI:Non-maskable_interrupts
      9723           +14.6%      11140        interrupts.CPU109.PMI:Performance_monitoring_interrupts
    217.75 ± 14%   +2074.2%       4734 ± 89%  interrupts.CPU11.33:PCI-MSI.524291-edge.eth0-TxRx-2
      9728           +14.5%      11137        interrupts.CPU11.NMI:Non-maskable_interrupts
      9728           +14.5%      11137        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11141        interrupts.CPU110.NMI:Non-maskable_interrupts
      9723           +14.6%      11141        interrupts.CPU110.PMI:Performance_monitoring_interrupts
      9725           +14.6%      11140        interrupts.CPU111.NMI:Non-maskable_interrupts
      9725           +14.6%      11140        interrupts.CPU111.PMI:Performance_monitoring_interrupts
      9724           +14.5%      11139        interrupts.CPU112.NMI:Non-maskable_interrupts
      9724           +14.5%      11139        interrupts.CPU112.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11139        interrupts.CPU113.NMI:Non-maskable_interrupts
      9723           +14.6%      11139        interrupts.CPU113.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11141        interrupts.CPU114.NMI:Non-maskable_interrupts
      9723           +14.6%      11141        interrupts.CPU114.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11142        interrupts.CPU115.NMI:Non-maskable_interrupts
      9723           +14.6%      11142        interrupts.CPU115.PMI:Performance_monitoring_interrupts
      9725           +14.6%      11141        interrupts.CPU116.NMI:Non-maskable_interrupts
      9725           +14.6%      11141        interrupts.CPU116.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11140        interrupts.CPU118.NMI:Non-maskable_interrupts
      9725           +14.5%      11140        interrupts.CPU118.PMI:Performance_monitoring_interrupts
      9728           +14.5%      11138        interrupts.CPU119.NMI:Non-maskable_interrupts
      9728           +14.5%      11138        interrupts.CPU119.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU120.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU120.PMI:Performance_monitoring_interrupts
      9887           +13.2%      11192        interrupts.CPU121.NMI:Non-maskable_interrupts
      9887           +13.2%      11192        interrupts.CPU121.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11196        interrupts.CPU122.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11196        interrupts.CPU122.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11195        interrupts.CPU123.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11195        interrupts.CPU123.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11197        interrupts.CPU124.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11197        interrupts.CPU124.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11197        interrupts.CPU125.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11197        interrupts.CPU125.PMI:Performance_monitoring_interrupts
      8514 ± 24%     +30.8%      11134        interrupts.CPU13.NMI:Non-maskable_interrupts
      8514 ± 24%     +30.8%      11134        interrupts.CPU13.PMI:Performance_monitoring_interrupts
      9887           +13.2%      11196        interrupts.CPU132.NMI:Non-maskable_interrupts
      9887           +13.2%      11196        interrupts.CPU132.PMI:Performance_monitoring_interrupts
     18958 ±153%     -98.1%     355.75 ± 47%  interrupts.CPU132.RES:Rescheduling_interrupts
      9886           +13.3%      11197        interrupts.CPU133.NMI:Non-maskable_interrupts
      9886           +13.3%      11197        interrupts.CPU133.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11197        interrupts.CPU134.NMI:Non-maskable_interrupts
      9886           +13.3%      11197        interrupts.CPU134.PMI:Performance_monitoring_interrupts
      9886           +13.2%      11194        interrupts.CPU135.NMI:Non-maskable_interrupts
      9886           +13.2%      11194        interrupts.CPU135.PMI:Performance_monitoring_interrupts
      9886           +13.2%      11196        interrupts.CPU136.NMI:Non-maskable_interrupts
      9886           +13.2%      11196        interrupts.CPU136.PMI:Performance_monitoring_interrupts
      9884           +13.3%      11197        interrupts.CPU137.NMI:Non-maskable_interrupts
      9884           +13.3%      11197        interrupts.CPU137.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11197        interrupts.CPU138.NMI:Non-maskable_interrupts
      9886           +13.3%      11197        interrupts.CPU138.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11195        interrupts.CPU139.NMI:Non-maskable_interrupts
      9885           +13.3%      11195        interrupts.CPU139.PMI:Performance_monitoring_interrupts
      3747 ±142%     -91.7%     312.50 ± 41%  interrupts.CPU139.RES:Rescheduling_interrupts
      9729           +14.4%      11133        interrupts.CPU14.NMI:Non-maskable_interrupts
      9729           +14.4%      11133        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11196        interrupts.CPU140.NMI:Non-maskable_interrupts
      9885           +13.3%      11196        interrupts.CPU140.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11196        interrupts.CPU141.NMI:Non-maskable_interrupts
      9886           +13.3%      11196        interrupts.CPU141.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11198        interrupts.CPU142.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11198        interrupts.CPU142.PMI:Performance_monitoring_interrupts
      7413 ± 33%     +51.0%      11194        interrupts.CPU143.NMI:Non-maskable_interrupts
      7413 ± 33%     +51.0%      11194        interrupts.CPU143.PMI:Performance_monitoring_interrupts
      5609 ±160%     -95.0%     279.75 ± 26%  interrupts.CPU143.RES:Rescheduling_interrupts
      8737 ± 24%     +28.1%      11194        interrupts.CPU144.NMI:Non-maskable_interrupts
      8737 ± 24%     +28.1%      11194        interrupts.CPU144.PMI:Performance_monitoring_interrupts
    712.50 ± 58%     -64.7%     251.25 ± 35%  interrupts.CPU144.RES:Rescheduling_interrupts
      7490 ± 33%     +49.5%      11195        interrupts.CPU145.NMI:Non-maskable_interrupts
      7490 ± 33%     +49.5%      11195        interrupts.CPU145.PMI:Performance_monitoring_interrupts
    672.00 ± 49%     -57.4%     286.00 ± 52%  interrupts.CPU145.RES:Rescheduling_interrupts
    501.75 ± 11%     -52.3%     239.50 ± 36%  interrupts.CPU146.RES:Rescheduling_interrupts
    474.25 ± 23%     -43.4%     268.25 ± 38%  interrupts.CPU147.RES:Rescheduling_interrupts
    455.25 ± 22%     -53.7%     210.75 ± 29%  interrupts.CPU149.RES:Rescheduling_interrupts
      9732           +14.4%      11135        interrupts.CPU15.NMI:Non-maskable_interrupts
      9732           +14.4%      11135        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      6244 ± 34%     +56.9%       9796 ± 24%  interrupts.CPU150.NMI:Non-maskable_interrupts
      6244 ± 34%     +56.9%       9796 ± 24%  interrupts.CPU150.PMI:Performance_monitoring_interrupts
    649.75 ± 36%     -65.6%     223.50 ± 25%  interrupts.CPU150.RES:Rescheduling_interrupts
      6243 ± 34%     +34.5%       8395 ± 33%  interrupts.CPU151.NMI:Non-maskable_interrupts
      6243 ± 34%     +34.5%       8395 ± 33%  interrupts.CPU151.PMI:Performance_monitoring_interrupts
    722.50 ± 33%     -72.4%     199.50 ± 38%  interrupts.CPU151.RES:Rescheduling_interrupts
    457.75 ± 20%     -57.2%     195.75 ± 50%  interrupts.CPU152.RES:Rescheduling_interrupts
    816.00 ± 52%     -79.2%     169.75 ± 41%  interrupts.CPU153.RES:Rescheduling_interrupts
    682.50 ± 59%     -77.5%     153.25 ± 36%  interrupts.CPU154.RES:Rescheduling_interrupts
    453.75 ± 19%     -66.4%     152.25 ± 42%  interrupts.CPU155.RES:Rescheduling_interrupts
    847.00 ± 70%     -80.9%     161.75 ± 30%  interrupts.CPU156.RES:Rescheduling_interrupts
      7489 ± 33%     +49.5%      11197        interrupts.CPU157.NMI:Non-maskable_interrupts
      7489 ± 33%     +49.5%      11197        interrupts.CPU157.PMI:Performance_monitoring_interrupts
    506.75 ± 27%     -69.4%     155.00 ± 87%  interrupts.CPU157.RES:Rescheduling_interrupts
      7488 ± 33%     +49.5%      11196        interrupts.CPU158.NMI:Non-maskable_interrupts
      7488 ± 33%     +49.5%      11196        interrupts.CPU158.PMI:Performance_monitoring_interrupts
    691.50 ± 69%     -80.6%     134.00 ± 75%  interrupts.CPU158.RES:Rescheduling_interrupts
      7490 ± 33%     +49.5%      11198        interrupts.CPU159.NMI:Non-maskable_interrupts
      7490 ± 33%     +49.5%      11198        interrupts.CPU159.PMI:Performance_monitoring_interrupts
    753.00 ± 72%     -74.7%     190.25 ± 79%  interrupts.CPU159.RES:Rescheduling_interrupts
      9731           +14.4%      11133        interrupts.CPU16.NMI:Non-maskable_interrupts
      9731           +14.4%      11133        interrupts.CPU16.PMI:Performance_monitoring_interrupts
      8738 ± 24%     +28.2%      11198        interrupts.CPU160.NMI:Non-maskable_interrupts
      8738 ± 24%     +28.2%      11198        interrupts.CPU160.PMI:Performance_monitoring_interrupts
    717.00 ± 65%     -80.0%     143.50 ± 39%  interrupts.CPU160.RES:Rescheduling_interrupts
      6239 ± 34%     +79.5%      11198        interrupts.CPU161.NMI:Non-maskable_interrupts
      6239 ± 34%     +79.5%      11198        interrupts.CPU161.PMI:Performance_monitoring_interrupts
    481.25 ± 25%     -77.5%     108.50 ± 56%  interrupts.CPU161.RES:Rescheduling_interrupts
      6238 ± 34%     +79.5%      11198        interrupts.CPU162.NMI:Non-maskable_interrupts
      6238 ± 34%     +79.5%      11198        interrupts.CPU162.PMI:Performance_monitoring_interrupts
      4993          +124.2%      11196        interrupts.CPU163.NMI:Non-maskable_interrupts
      4993          +124.2%      11196        interrupts.CPU163.PMI:Performance_monitoring_interrupts
    701.75 ± 60%     -82.9%     120.25 ± 67%  interrupts.CPU163.RES:Rescheduling_interrupts
      6245 ± 34%     +79.3%      11196        interrupts.CPU164.NMI:Non-maskable_interrupts
      6245 ± 34%     +79.3%      11196        interrupts.CPU164.PMI:Performance_monitoring_interrupts
    465.00 ± 21%     -65.6%     159.75 ± 45%  interrupts.CPU164.RES:Rescheduling_interrupts
      6244 ± 34%     +79.3%      11195        interrupts.CPU165.NMI:Non-maskable_interrupts
      6244 ± 34%     +79.3%      11195        interrupts.CPU165.PMI:Performance_monitoring_interrupts
    503.50 ± 23%     -75.0%     126.00 ± 86%  interrupts.CPU165.RES:Rescheduling_interrupts
      6244 ± 34%     +79.3%      11194        interrupts.CPU166.NMI:Non-maskable_interrupts
      6244 ± 34%     +79.3%      11194        interrupts.CPU166.PMI:Performance_monitoring_interrupts
    571.25 ± 29%     -76.5%     134.50 ± 76%  interrupts.CPU166.RES:Rescheduling_interrupts
      6243 ± 34%     +56.9%       9794 ± 24%  interrupts.CPU167.NMI:Non-maskable_interrupts
      6243 ± 34%     +56.9%       9794 ± 24%  interrupts.CPU167.PMI:Performance_monitoring_interrupts
    759.75 ± 70%     -84.1%     121.00 ± 45%  interrupts.CPU167.RES:Rescheduling_interrupts
      5615 ±  3%      +5.5%       5924 ±  5%  interrupts.CPU168.CAL:Function_call_interrupts
      9728           +14.5%      11135        interrupts.CPU17.NMI:Non-maskable_interrupts
      9728           +14.5%      11135        interrupts.CPU17.PMI:Performance_monitoring_interrupts
    601.25 ± 41%     -89.1%      65.25 ± 71%  interrupts.CPU170.RES:Rescheduling_interrupts
      5583           +12.7%       6294 ±  8%  interrupts.CPU171.CAL:Function_call_interrupts
    446.00 ± 26%     -91.0%      40.25 ± 28%  interrupts.CPU172.RES:Rescheduling_interrupts
      6031 ± 34%     +84.6%      11136        interrupts.CPU173.NMI:Non-maskable_interrupts
      6031 ± 34%     +84.6%      11136        interrupts.CPU173.PMI:Performance_monitoring_interrupts
      6032 ± 34%     +61.6%       9745 ± 24%  interrupts.CPU174.NMI:Non-maskable_interrupts
      6032 ± 34%     +61.6%       9745 ± 24%  interrupts.CPU174.PMI:Performance_monitoring_interrupts
      6031 ± 34%     +61.5%       9742 ± 24%  interrupts.CPU175.NMI:Non-maskable_interrupts
      6031 ± 34%     +61.5%       9742 ± 24%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
      2235 ±125%     -97.1%      65.75 ± 65%  interrupts.CPU175.RES:Rescheduling_interrupts
      6032 ± 34%     +61.5%       9742 ± 24%  interrupts.CPU176.NMI:Non-maskable_interrupts
      6032 ± 34%     +61.5%       9742 ± 24%  interrupts.CPU176.PMI:Performance_monitoring_interrupts
    444.50 ± 33%     -89.0%      48.75 ± 67%  interrupts.CPU176.RES:Rescheduling_interrupts
      6031 ± 34%     +84.7%      11139        interrupts.CPU177.NMI:Non-maskable_interrupts
      6031 ± 34%     +84.7%      11139        interrupts.CPU177.PMI:Performance_monitoring_interrupts
      6031 ± 34%     +84.6%      11136        interrupts.CPU178.NMI:Non-maskable_interrupts
      6031 ± 34%     +84.6%      11136        interrupts.CPU178.PMI:Performance_monitoring_interrupts
      5559 ±  3%      +8.0%       6004 ±  5%  interrupts.CPU179.CAL:Function_call_interrupts
      4829          +130.7%      11139        interrupts.CPU179.NMI:Non-maskable_interrupts
      4829          +130.7%      11139        interrupts.CPU179.PMI:Performance_monitoring_interrupts
      9729           +14.5%      11136        interrupts.CPU18.NMI:Non-maskable_interrupts
      9729           +14.5%      11136        interrupts.CPU18.PMI:Performance_monitoring_interrupts
      6040 ± 34%     +84.4%      11137        interrupts.CPU180.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11137        interrupts.CPU180.PMI:Performance_monitoring_interrupts
      6041 ± 34%     +84.4%      11139        interrupts.CPU181.NMI:Non-maskable_interrupts
      6041 ± 34%     +84.4%      11139        interrupts.CPU181.PMI:Performance_monitoring_interrupts
      6040 ± 34%     +84.4%      11138        interrupts.CPU182.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11138        interrupts.CPU182.PMI:Performance_monitoring_interrupts
    410.50 ± 30%     -81.1%      77.75 ± 99%  interrupts.CPU182.RES:Rescheduling_interrupts
      6040 ± 34%     +84.4%      11134        interrupts.CPU183.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11134        interrupts.CPU183.PMI:Performance_monitoring_interrupts
    613.25 ± 57%     -95.1%      30.25 ± 82%  interrupts.CPU183.RES:Rescheduling_interrupts
      6040 ± 34%     +84.4%      11136        interrupts.CPU184.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11136        interrupts.CPU184.PMI:Performance_monitoring_interrupts
      5181 ±149%     -96.2%     197.00 ±127%  interrupts.CPU184.RES:Rescheduling_interrupts
      6039 ± 34%     +84.4%      11135        interrupts.CPU185.NMI:Non-maskable_interrupts
      6039 ± 34%     +84.4%      11135        interrupts.CPU185.PMI:Performance_monitoring_interrupts
    461.25 ± 43%     -88.1%      55.00 ± 53%  interrupts.CPU185.RES:Rescheduling_interrupts
      6040 ± 34%     +84.4%      11137        interrupts.CPU186.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11137        interrupts.CPU186.PMI:Performance_monitoring_interrupts
    800.25 ± 56%     -88.3%      94.00 ± 84%  interrupts.CPU186.RES:Rescheduling_interrupts
      6039 ± 34%     +84.4%      11136        interrupts.CPU187.NMI:Non-maskable_interrupts
      6039 ± 34%     +84.4%      11136        interrupts.CPU187.PMI:Performance_monitoring_interrupts
    808.00 ± 53%     -77.1%     185.25 ±132%  interrupts.CPU187.RES:Rescheduling_interrupts
      6040 ± 34%     +84.5%      11144        interrupts.CPU188.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.5%      11144        interrupts.CPU188.PMI:Performance_monitoring_interrupts
    769.00 ± 51%     -79.7%     156.00 ±143%  interrupts.CPU188.RES:Rescheduling_interrupts
      6039 ± 34%     +84.4%      11138        interrupts.CPU189.NMI:Non-maskable_interrupts
      6039 ± 34%     +84.4%      11138        interrupts.CPU189.PMI:Performance_monitoring_interrupts
    851.50 ± 45%     -81.6%     156.50 ±126%  interrupts.CPU189.RES:Rescheduling_interrupts
      9728           +14.5%      11135        interrupts.CPU19.NMI:Non-maskable_interrupts
      9728           +14.5%      11135        interrupts.CPU19.PMI:Performance_monitoring_interrupts
      6039 ± 34%     +84.4%      11137        interrupts.CPU190.NMI:Non-maskable_interrupts
      6039 ± 34%     +84.4%      11137        interrupts.CPU190.PMI:Performance_monitoring_interrupts
    746.00 ± 39%     -94.9%      37.75 ± 73%  interrupts.CPU190.RES:Rescheduling_interrupts
      7241 ± 33%     +53.7%      11132        interrupts.CPU191.NMI:Non-maskable_interrupts
      7241 ± 33%     +53.7%      11132        interrupts.CPU191.PMI:Performance_monitoring_interrupts
      9730           +14.4%      11135        interrupts.CPU2.NMI:Non-maskable_interrupts
      9730           +14.4%      11135        interrupts.CPU2.PMI:Performance_monitoring_interrupts
      9729           +14.4%      11133        interrupts.CPU20.NMI:Non-maskable_interrupts
      9729           +14.4%      11133        interrupts.CPU20.PMI:Performance_monitoring_interrupts
      9729           +14.4%      11135        interrupts.CPU21.NMI:Non-maskable_interrupts
      9729           +14.4%      11135        interrupts.CPU21.PMI:Performance_monitoring_interrupts
      9730           +14.4%      11133        interrupts.CPU22.NMI:Non-maskable_interrupts
      9730           +14.4%      11133        interrupts.CPU22.PMI:Performance_monitoring_interrupts
      9727           +14.5%      11134        interrupts.CPU23.NMI:Non-maskable_interrupts
      9727           +14.5%      11134        interrupts.CPU23.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU24.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU24.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU25.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU25.PMI:Performance_monitoring_interrupts
      9884           +13.2%      11193        interrupts.CPU26.NMI:Non-maskable_interrupts
      9884           +13.2%      11193        interrupts.CPU26.PMI:Performance_monitoring_interrupts
    388.25 ± 25%    +635.5%       2855 ±139%  interrupts.CPU26.RES:Rescheduling_interrupts
      9886           +13.2%      11191        interrupts.CPU27.NMI:Non-maskable_interrupts
      9886           +13.2%      11191        interrupts.CPU27.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11195        interrupts.CPU28.NMI:Non-maskable_interrupts
      9885           +13.3%      11195        interrupts.CPU28.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11194        interrupts.CPU29.NMI:Non-maskable_interrupts
      9885           +13.2%      11194        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      9730           +14.5%      11138        interrupts.CPU3.NMI:Non-maskable_interrupts
      9730           +14.5%      11138        interrupts.CPU3.PMI:Performance_monitoring_interrupts
      5512 ±  2%     +29.1%       7118 ± 30%  interrupts.CPU30.CAL:Function_call_interrupts
    396.00 ± 23%     +64.1%     650.00 ± 31%  interrupts.CPU30.RES:Rescheduling_interrupts
      8648 ± 24%     +29.5%      11196        interrupts.CPU31.NMI:Non-maskable_interrupts
      8648 ± 24%     +29.5%      11196        interrupts.CPU31.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11196        interrupts.CPU32.NMI:Non-maskable_interrupts
      9885           +13.3%      11196        interrupts.CPU32.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11198        interrupts.CPU33.NMI:Non-maskable_interrupts
      9886           +13.3%      11198        interrupts.CPU33.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11197        interrupts.CPU34.NMI:Non-maskable_interrupts
      9885           +13.3%      11197        interrupts.CPU34.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11195        interrupts.CPU35.NMI:Non-maskable_interrupts
      9885           +13.2%      11195        interrupts.CPU35.PMI:Performance_monitoring_interrupts
      9886           +13.2%      11195        interrupts.CPU36.NMI:Non-maskable_interrupts
      9886           +13.2%      11195        interrupts.CPU36.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11196        interrupts.CPU37.NMI:Non-maskable_interrupts
      9886           +13.3%      11196        interrupts.CPU37.PMI:Performance_monitoring_interrupts
      8648 ± 24%     +29.5%      11196        interrupts.CPU38.NMI:Non-maskable_interrupts
      8648 ± 24%     +29.5%      11196        interrupts.CPU38.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11190        interrupts.CPU39.NMI:Non-maskable_interrupts
      9885           +13.2%      11190        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      9729           +14.5%      11137        interrupts.CPU4.NMI:Non-maskable_interrupts
      9729           +14.5%      11137        interrupts.CPU4.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU40.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU40.PMI:Performance_monitoring_interrupts
      9884           +13.2%      11193        interrupts.CPU41.NMI:Non-maskable_interrupts
      9884           +13.2%      11193        interrupts.CPU41.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11194        interrupts.CPU42.NMI:Non-maskable_interrupts
      9885           +13.2%      11194        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      9884           +13.2%      11190        interrupts.CPU43.NMI:Non-maskable_interrupts
      9884           +13.2%      11190        interrupts.CPU43.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU44.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU44.PMI:Performance_monitoring_interrupts
      9884           +13.3%      11194        interrupts.CPU45.NMI:Non-maskable_interrupts
      9884           +13.3%      11194        interrupts.CPU45.PMI:Performance_monitoring_interrupts
      9883           +13.3%      11194        interrupts.CPU46.NMI:Non-maskable_interrupts
      9883           +13.3%      11194        interrupts.CPU46.PMI:Performance_monitoring_interrupts
      9880           +13.3%      11194        interrupts.CPU47.NMI:Non-maskable_interrupts
      9880           +13.3%      11194        interrupts.CPU47.PMI:Performance_monitoring_interrupts
      9997           +12.0%      11196        interrupts.CPU48.NMI:Non-maskable_interrupts
      9997           +12.0%      11196        interrupts.CPU48.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11194        interrupts.CPU49.NMI:Non-maskable_interrupts
      9987           +12.1%      11194        interrupts.CPU49.PMI:Performance_monitoring_interrupts
     10211 ±156%     -96.1%     402.50 ± 25%  interrupts.CPU49.RES:Rescheduling_interrupts
      9730           +14.4%      11136        interrupts.CPU5.NMI:Non-maskable_interrupts
      9730           +14.4%      11136        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11195        interrupts.CPU50.NMI:Non-maskable_interrupts
      9987           +12.1%      11195        interrupts.CPU50.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11195        interrupts.CPU51.NMI:Non-maskable_interrupts
      9989           +12.1%      11195        interrupts.CPU51.PMI:Performance_monitoring_interrupts
      9986           +12.1%      11194        interrupts.CPU52.NMI:Non-maskable_interrupts
      9986           +12.1%      11194        interrupts.CPU52.PMI:Performance_monitoring_interrupts
    848.00 ± 42%     -49.0%     432.50 ± 38%  interrupts.CPU52.RES:Rescheduling_interrupts
      9990           +12.1%      11198        interrupts.CPU53.NMI:Non-maskable_interrupts
      9990           +12.1%      11198        interrupts.CPU53.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11196        interrupts.CPU54.NMI:Non-maskable_interrupts
      9989           +12.1%      11196        interrupts.CPU54.PMI:Performance_monitoring_interrupts
    651.25 ± 34%     -35.5%     420.00 ± 23%  interrupts.CPU54.RES:Rescheduling_interrupts
      9990           +12.1%      11196        interrupts.CPU55.NMI:Non-maskable_interrupts
      9990           +12.1%      11196        interrupts.CPU55.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11196        interrupts.CPU56.NMI:Non-maskable_interrupts
      9987           +12.1%      11196        interrupts.CPU56.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11195        interrupts.CPU57.NMI:Non-maskable_interrupts
      9989           +12.1%      11195        interrupts.CPU57.PMI:Performance_monitoring_interrupts
      9990           +12.1%      11197        interrupts.CPU58.NMI:Non-maskable_interrupts
      9990           +12.1%      11197        interrupts.CPU58.PMI:Performance_monitoring_interrupts
    572.75 ± 34%     -32.4%     387.00 ± 24%  interrupts.CPU58.RES:Rescheduling_interrupts
      9988           +12.1%      11198        interrupts.CPU59.NMI:Non-maskable_interrupts
      9988           +12.1%      11198        interrupts.CPU59.PMI:Performance_monitoring_interrupts
      9727           +14.5%      11137        interrupts.CPU6.NMI:Non-maskable_interrupts
      9727           +14.5%      11137        interrupts.CPU6.PMI:Performance_monitoring_interrupts
      9990           +12.1%      11197        interrupts.CPU60.NMI:Non-maskable_interrupts
      9990           +12.1%      11197        interrupts.CPU60.PMI:Performance_monitoring_interrupts
      2296 ±128%     -80.8%     440.50 ± 28%  interrupts.CPU60.RES:Rescheduling_interrupts
      9988           +12.1%      11198        interrupts.CPU61.NMI:Non-maskable_interrupts
      9988           +12.1%      11198        interrupts.CPU61.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11197        interrupts.CPU62.NMI:Non-maskable_interrupts
      9987           +12.1%      11197        interrupts.CPU62.PMI:Performance_monitoring_interrupts
      9988           +12.1%      11198        interrupts.CPU63.NMI:Non-maskable_interrupts
      9988           +12.1%      11198        interrupts.CPU63.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11199        interrupts.CPU64.NMI:Non-maskable_interrupts
      9989           +12.1%      11199        interrupts.CPU64.PMI:Performance_monitoring_interrupts
      9988           +12.1%      11200        interrupts.CPU65.NMI:Non-maskable_interrupts
      9988           +12.1%      11200        interrupts.CPU65.PMI:Performance_monitoring_interrupts
      9988           +12.1%      11197        interrupts.CPU66.NMI:Non-maskable_interrupts
      9988           +12.1%      11197        interrupts.CPU66.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11197        interrupts.CPU67.NMI:Non-maskable_interrupts
      9987           +12.1%      11197        interrupts.CPU67.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11197        interrupts.CPU68.NMI:Non-maskable_interrupts
      9989           +12.1%      11197        interrupts.CPU68.PMI:Performance_monitoring_interrupts
      9990           +12.1%      11199        interrupts.CPU69.NMI:Non-maskable_interrupts
      9990           +12.1%      11199        interrupts.CPU69.PMI:Performance_monitoring_interrupts
      5435            +7.2%       5825 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
      9727           +14.6%      11146        interrupts.CPU7.NMI:Non-maskable_interrupts
      9727           +14.6%      11146        interrupts.CPU7.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11197        interrupts.CPU70.NMI:Non-maskable_interrupts
      9989           +12.1%      11197        interrupts.CPU70.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11195        interrupts.CPU71.NMI:Non-maskable_interrupts
      9987           +12.1%      11195        interrupts.CPU71.PMI:Performance_monitoring_interrupts
      9652           +15.4%      11141        interrupts.CPU72.NMI:Non-maskable_interrupts
      9652           +15.4%      11141        interrupts.CPU72.PMI:Performance_monitoring_interrupts
      9654           +15.6%      11160        interrupts.CPU74.NMI:Non-maskable_interrupts
      9654           +15.6%      11160        interrupts.CPU74.PMI:Performance_monitoring_interrupts
      5564           +12.3%       6246 ±  7%  interrupts.CPU75.CAL:Function_call_interrupts
      9654           +15.3%      11133        interrupts.CPU75.NMI:Non-maskable_interrupts
      9654           +15.3%      11133        interrupts.CPU75.PMI:Performance_monitoring_interrupts
      9653           +15.4%      11135        interrupts.CPU76.NMI:Non-maskable_interrupts
      9653           +15.4%      11135        interrupts.CPU76.PMI:Performance_monitoring_interrupts
    447.50 ± 32%     -31.8%     305.00 ±  9%  interrupts.CPU76.RES:Rescheduling_interrupts
      9654           +15.4%      11139        interrupts.CPU78.NMI:Non-maskable_interrupts
      9654           +15.4%      11139        interrupts.CPU78.PMI:Performance_monitoring_interrupts
      9729           +14.4%      11134        interrupts.CPU8.NMI:Non-maskable_interrupts
      9729           +14.4%      11134        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      9652           +15.4%      11137        interrupts.CPU82.NMI:Non-maskable_interrupts
      9652           +15.4%      11137        interrupts.CPU82.PMI:Performance_monitoring_interrupts
      9654           +15.4%      11137        interrupts.CPU85.NMI:Non-maskable_interrupts
      9654           +15.4%      11137        interrupts.CPU85.PMI:Performance_monitoring_interrupts
    537.25 ± 45%     -50.3%     267.25 ± 13%  interrupts.CPU87.RES:Rescheduling_interrupts
      3077 ±132%     -86.5%     416.00 ± 65%  interrupts.CPU88.RES:Rescheduling_interrupts
      9652           +15.4%      11138        interrupts.CPU89.NMI:Non-maskable_interrupts
      9652           +15.4%      11138        interrupts.CPU89.PMI:Performance_monitoring_interrupts
    494.25 ± 48%     -44.9%     272.25 ± 16%  interrupts.CPU89.RES:Rescheduling_interrupts
      9730           +14.5%      11136        interrupts.CPU9.NMI:Non-maskable_interrupts
      9730           +14.5%      11136        interrupts.CPU9.PMI:Performance_monitoring_interrupts
      9654           +15.4%      11141        interrupts.CPU90.NMI:Non-maskable_interrupts
      9654           +15.4%      11141        interrupts.CPU90.PMI:Performance_monitoring_interrupts
      9653           +15.4%      11140        interrupts.CPU91.NMI:Non-maskable_interrupts
      9653           +15.4%      11140        interrupts.CPU91.PMI:Performance_monitoring_interrupts
    707.50 ± 29%     -44.2%     395.00 ± 66%  interrupts.CPU91.RES:Rescheduling_interrupts
      9652           +15.5%      11149        interrupts.CPU92.NMI:Non-maskable_interrupts
      9652           +15.5%      11149        interrupts.CPU92.PMI:Performance_monitoring_interrupts
    666.50 ± 30%     -46.6%     356.00 ± 58%  interrupts.CPU92.RES:Rescheduling_interrupts
      9651           +15.6%      11153        interrupts.CPU93.NMI:Non-maskable_interrupts
      9651           +15.6%      11153        interrupts.CPU93.PMI:Performance_monitoring_interrupts
    744.75 ± 26%     -49.6%     375.00 ± 62%  interrupts.CPU93.RES:Rescheduling_interrupts
      9651           +15.4%      11141        interrupts.CPU94.NMI:Non-maskable_interrupts
      9651           +15.4%      11141        interrupts.CPU94.PMI:Performance_monitoring_interrupts
    760.50 ± 43%     -68.5%     239.25 ± 16%  interrupts.CPU94.RES:Rescheduling_interrupts
      9658           +15.5%      11155        interrupts.CPU95.NMI:Non-maskable_interrupts
      9658           +15.5%      11155        interrupts.CPU95.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11140        interrupts.CPU96.NMI:Non-maskable_interrupts
      9725           +14.5%      11140        interrupts.CPU96.PMI:Performance_monitoring_interrupts
      8510 ± 24%     +30.9%      11138        interrupts.CPU97.NMI:Non-maskable_interrupts
      8510 ± 24%     +30.9%      11138        interrupts.CPU97.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11136        interrupts.CPU98.NMI:Non-maskable_interrupts
      9725           +14.5%      11136        interrupts.CPU98.PMI:Performance_monitoring_interrupts
    350.50 ±  3%     +58.9%     557.00 ±  5%  interrupts.IWI:IRQ_work_interrupts
   1715066 ±  3%     +21.3%    2079884 ±  3%  interrupts.NMI:Non-maskable_interrupts
   1715066 ±  3%     +21.3%    2079884 ±  3%  interrupts.PMI:Performance_monitoring_interrupts
    532498 ± 17%     -67.4%     173726 ± 13%  interrupts.RES:Rescheduling_interrupts


                                                                                
                            will-it-scale.per_thread_ops                        
                                                                                
  400000 +------------------------------------------------------------------+   
         |.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.|   
  350000 |-+                                                                |   
         |                                                                  |   
         |                                                                  |   
  300000 |-+                                                                |   
         |                                                                  |   
  250000 |-+                                                                |   
         |                                                                  |   
  200000 |-+                                                                |   
         |                                                                  |   
         |                                                                  |   
  150000 |-+       O  O O O O O O O O O O  O                                |   
         | O O O O                           O O O O O O O O                |   
  100000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               will-it-scale.workload                           
                                                                                
  8e+07 +-------------------------------------------------------------------+   
        |                                                   .+. .+.. .+. .+.|   
  7e+07 |.+.+.+..+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+..+.+.+.+.+   +    +   +   |   
        |                                                                   |   
        |                                                                   |   
  6e+07 |-+                                                                 |   
        |                                                                   |   
  5e+07 |-+                                                                 |   
        |                                                                   |   
  4e+07 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  3e+07 |-+        O O O O O O O  O O O O O                                 |   
        | O O O  O                          O O O  O O O O O                |   
  2e+07 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-skl-fpga01/context_switch1/will-it-scale/0x2000065

commit: 
  87e867b426 ("sched/pelt: Cleanup PELT divider")
  e9f2fb8893 ("sched_pair_cpu: Introduce scheduler task pairing system call")

87e867b4269f29da e9f2fb8893b705bcabebf3c62b6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    218931           -37.5%     136756        will-it-scale.per_thread_ops
  22768896           -37.5%   14222677        will-it-scale.workload
      3418 ±  2%      -2.3%       3338        boot-time.idle
     22.52 ±  4%      -6.2       16.35        mpstat.cpu.all.usr%
      3559 ± 23%     +42.6%       5074 ± 17%  numa-vmstat.node0.nr_inactive_anon
      3933 ± 24%     +39.8%       5496 ± 11%  numa-vmstat.node0.nr_shmem
      3559 ± 23%     +42.6%       5074 ± 17%  numa-vmstat.node0.nr_zone_inactive_anon
     20298            +8.5%      22033        proc-vmstat.nr_kernel_stack
     23006            +9.1%      25092 ±  3%  proc-vmstat.nr_shmem
     23193           +13.9%      26421 ±  4%  proc-vmstat.pgactivate
     76.25            +7.9%      82.25        vmstat.cpu.sy
     21.75 ±  3%     -27.6%      15.75 ±  2%  vmstat.cpu.us
  22550823           -37.5%   14090838        vmstat.system.cs
     14397 ± 23%     +41.5%      20378 ± 16%  numa-meminfo.node0.Inactive
     14237 ± 23%     +42.6%      20297 ± 17%  numa-meminfo.node0.Inactive(anon)
     15734 ± 24%     +39.7%      21988 ± 11%  numa-meminfo.node0.Shmem
     14514 ± 23%     -41.2%       8541 ± 37%  numa-meminfo.node1.Inactive(anon)
    -38316          +109.2%     -80155        sched_debug.cfs_rq:/.spread0.avg
     56956 ± 57%     -77.7%      12727 ±151%  sched_debug.cfs_rq:/.spread0.max
   -216970           +38.1%    -299693        sched_debug.cfs_rq:/.spread0.min
    311271 ± 16%     +40.3%     436737 ±  9%  sched_debug.cpu.avg_idle.max
      2480 ±  4%     +38.1%       3425 ±  2%  sched_debug.cpu.avg_idle.min
     69363 ±  3%     +12.9%      78335 ±  9%  sched_debug.cpu.avg_idle.stddev
      2783 ± 68%    +176.8%       7705 ± 17%  sched_debug.cpu.max_idle_balance_cost.stddev
  32676933           -37.3%   20482288        sched_debug.cpu.nr_switches.avg
  34658910           -39.9%   20829486        sched_debug.cpu.nr_switches.max
  30785157 ±  2%     -35.0%   20010884        sched_debug.cpu.nr_switches.min
   1055140 ± 18%     -84.8%     159933 ± 19%  sched_debug.cpu.nr_switches.stddev
    782.00 ±  6%     -19.6%     628.50 ± 11%  slabinfo.file_lock_cache.active_objs
    782.00 ±  6%     -19.6%     628.50 ± 11%  slabinfo.file_lock_cache.num_objs
     47986           +77.6%      85231        slabinfo.kmalloc-64.active_objs
    750.50           +77.5%       1332        slabinfo.kmalloc-64.active_slabs
     48043           +77.5%      85274        slabinfo.kmalloc-64.num_objs
    750.50           +77.5%       1332        slabinfo.kmalloc-64.num_slabs
    263.50 ± 10%     +29.4%     341.00 ±  9%  slabinfo.numa_policy.active_objs
    263.50 ± 10%     +29.4%     341.00 ±  9%  slabinfo.numa_policy.num_objs
      2385 ±  3%      +9.0%       2600 ±  2%  slabinfo.sighand_cache.active_objs
      2386 ±  3%      +9.0%       2600 ±  2%  slabinfo.sighand_cache.num_objs
      3693 ±  4%      +8.9%       4021 ±  2%  slabinfo.signal_cache.active_objs
      3693 ±  4%      +8.9%       4021 ±  2%  slabinfo.signal_cache.num_objs
      1469 ±  2%     +10.0%       1616 ±  2%  slabinfo.task_struct.active_objs
      1473 ±  2%     +10.0%       1621 ±  2%  slabinfo.task_struct.active_slabs
      1473 ±  2%     +10.0%       1621 ±  2%  slabinfo.task_struct.num_objs
      1473 ±  2%     +10.0%       1621 ±  2%  slabinfo.task_struct.num_slabs
 3.699e+10           -19.1%  2.991e+10        perf-stat.i.branch-instructions
      1.40            -0.1        1.28        perf-stat.i.branch-miss-rate%
 5.127e+08           -26.0%  3.794e+08        perf-stat.i.branch-misses
      0.36 ± 30%      +7.1        7.43 ±  9%  perf-stat.i.cache-miss-rate%
   1649484 ± 31%   +2393.0%   41121274 ±  2%  perf-stat.i.cache-misses
  22684327           -37.5%   14176299        perf-stat.i.context-switches
      1.49           +34.4%       2.01        perf-stat.i.cpi
 2.771e+11            +2.9%  2.851e+11        perf-stat.i.cpu-cycles
    223279 ± 25%     -96.9%       6941        perf-stat.i.cycles-between-cache-misses
      0.08            -0.0        0.07        perf-stat.i.dTLB-load-miss-rate%
  45500729           -37.4%   28505101        perf-stat.i.dTLB-load-misses
 5.367e+10           -26.7%  3.931e+10        perf-stat.i.dTLB-loads
     67056 ±  3%     -20.2%      53500        perf-stat.i.dTLB-store-misses
 3.325e+10           -26.0%   2.46e+10        perf-stat.i.dTLB-stores
     57.74 ±  2%     +10.8       68.52 ±  4%  perf-stat.i.iTLB-load-miss-rate%
  67964076           -42.7%   38909739        perf-stat.i.iTLB-load-misses
  49449215 ±  4%     -63.9%   17843583 ± 14%  perf-stat.i.iTLB-loads
 1.853e+11           -23.5%  1.418e+11        perf-stat.i.instructions
      2817 ±  2%     +32.4%       3729        perf-stat.i.instructions-per-iTLB-miss
      0.67           -25.5%       0.50        perf-stat.i.ipc
      2.66            +2.9%       2.74        perf-stat.i.metric.GHz
      1.28 ±  9%     -59.7%       0.52        perf-stat.i.metric.K/sec
      1197           -24.2%     907.93        perf-stat.i.metric.M/sec
     90.52            -2.0       88.47        perf-stat.i.node-load-miss-rate%
    138585 ±  7%   +4081.5%    5794873        perf-stat.i.node-load-misses
     22237 ±  7%   +3283.6%     752443        perf-stat.i.node-loads
     76.13 ±  5%     +12.6       88.73        perf-stat.i.node-store-miss-rate%
     14722 ± 26%   +8185.9%    1219857        perf-stat.i.node-store-misses
      5757 ±  5%   +2559.4%     153105        perf-stat.i.node-stores
      1.39            -0.1        1.27        perf-stat.overall.branch-miss-rate%
      0.32 ± 35%      +7.1        7.39 ±  9%  perf-stat.overall.cache-miss-rate%
      1.50           +34.5%       2.01        perf-stat.overall.cpi
    187391 ± 33%     -96.3%       6935 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.08            -0.0        0.07        perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  3%      +0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     57.90 ±  2%     +10.8       68.71 ±  4%  perf-stat.overall.iTLB-load-miss-rate%
      2727           +33.6%       3644        perf-stat.overall.instructions-per-iTLB-miss
      0.67           -25.6%       0.50        perf-stat.overall.ipc
     85.88            +2.6       88.50        perf-stat.overall.node-load-miss-rate%
     70.96 ±  5%     +17.9       88.82        perf-stat.overall.node-store-miss-rate%
   2452546           +22.2%    2998056        perf-stat.overall.path-length
 3.687e+10           -19.1%  2.981e+10        perf-stat.ps.branch-instructions
  5.11e+08           -26.0%  3.781e+08        perf-stat.ps.branch-misses
   1643802 ± 31%   +2393.3%   40984510 ±  2%  perf-stat.ps.cache-misses
  22608494           -37.5%   14128772        perf-stat.ps.context-switches
 2.762e+11            +2.9%  2.841e+11        perf-stat.ps.cpu-cycles
  45350504           -37.3%   28413918        perf-stat.ps.dTLB-load-misses
 5.349e+10           -26.7%  3.918e+10        perf-stat.ps.dTLB-loads
     66900 ±  3%     -20.0%      53521        perf-stat.ps.dTLB-store-misses
 3.314e+10           -26.0%  2.452e+10        perf-stat.ps.dTLB-stores
  67733698           -42.7%   38786731        perf-stat.ps.iTLB-load-misses
  49282420 ±  3%     -63.9%   17783730 ± 14%  perf-stat.ps.iTLB-loads
 1.847e+11           -23.5%  1.413e+11        perf-stat.ps.instructions
    138052 ±  7%   +4083.5%    5775416        perf-stat.ps.node-load-misses
     22671 ±  7%   +3210.8%     750607        perf-stat.ps.node-loads
     14669 ± 26%   +8188.4%    1215837        perf-stat.ps.node-store-misses
      5772 ±  5%   +2550.7%     153020        perf-stat.ps.node-stores
 5.584e+13           -23.6%  4.264e+13        perf-stat.total.instructions
    407830 ±  6%     +24.8%     509113 ±  2%  interrupts.CAL:Function_call_interrupts
      6674 ±  6%      +8.2%       7220 ±  6%  interrupts.CPU0.NMI:Non-maskable_interrupts
      6674 ±  6%      +8.2%       7220 ±  6%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.1%       7219 ±  6%  interrupts.CPU10.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.1%       7219 ±  6%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU100.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
      7059 ±  6%      +4.4%       7366 ±  6%  interrupts.CPU101.NMI:Non-maskable_interrupts
      7059 ±  6%      +4.4%       7366 ±  6%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
      7058 ±  6%      +4.3%       7365 ±  6%  interrupts.CPU102.NMI:Non-maskable_interrupts
      7058 ±  6%      +4.3%       7365 ±  6%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU103.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.1%       7220 ±  6%  interrupts.CPU11.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.1%       7220 ±  6%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      6674 ±  6%      +8.2%       7220 ±  6%  interrupts.CPU12.NMI:Non-maskable_interrupts
      6674 ±  6%      +8.2%       7220 ±  6%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      6670 ±  6%      +8.3%       7221 ±  6%  interrupts.CPU13.NMI:Non-maskable_interrupts
      6670 ±  6%      +8.3%       7221 ±  6%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      6671 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU14.NMI:Non-maskable_interrupts
      6671 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      6672 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU15.NMI:Non-maskable_interrupts
      6672 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      6672 ±  6%      +8.2%       7219 ±  6%  interrupts.CPU16.NMI:Non-maskable_interrupts
      6672 ±  6%      +8.2%       7219 ±  6%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
      6672 ±  6%      +8.2%       7219 ±  6%  interrupts.CPU17.NMI:Non-maskable_interrupts
      6672 ±  6%      +8.2%       7219 ±  6%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU2.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
     12615 ±150%     -94.4%     712.25 ± 38%  interrupts.CPU2.RES:Rescheduling_interrupts
      6665 ±  6%      +8.3%       7220 ±  6%  interrupts.CPU24.NMI:Non-maskable_interrupts
      6665 ±  6%      +8.3%       7220 ±  6%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      6665 ±  6%      +8.3%       7219 ±  6%  interrupts.CPU25.NMI:Non-maskable_interrupts
      6665 ±  6%      +8.3%       7219 ±  6%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU26.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU27.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU28.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU29.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU30.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU31.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU32.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU33.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU34.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.4%       7373 ±  6%  interrupts.CPU35.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.4%       7373 ±  6%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU36.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU37.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.4%       7370 ±  6%  interrupts.CPU38.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.4%       7370 ±  6%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU39.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.2%       7222 ±  6%  interrupts.CPU4.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.2%       7222 ±  6%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.4%       7369 ±  6%  interrupts.CPU40.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.4%       7369 ±  6%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.4%       7368 ±  6%  interrupts.CPU41.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.4%       7368 ±  6%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.4%       7367 ±  6%  interrupts.CPU42.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.4%       7367 ±  6%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU43.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      6085 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU44.NMI:Non-maskable_interrupts
      6085 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      6087 ± 20%     +21.0%       7368 ±  6%  interrupts.CPU45.NMI:Non-maskable_interrupts
      6087 ± 20%     +21.0%       7368 ±  6%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
      6086 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU46.NMI:Non-maskable_interrupts
      6086 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      6086 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU47.NMI:Non-maskable_interrupts
      6086 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      6086 ± 20%     +21.1%       7367 ±  6%  interrupts.CPU48.NMI:Non-maskable_interrupts
      6086 ± 20%     +21.1%       7367 ±  6%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      6084 ± 20%     +21.1%       7367 ±  6%  interrupts.CPU49.NMI:Non-maskable_interrupts
      6084 ± 20%     +21.1%       7367 ±  6%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
      6675 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU5.NMI:Non-maskable_interrupts
      6675 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      6088 ± 20%     +21.0%       7365 ±  6%  interrupts.CPU51.NMI:Non-maskable_interrupts
      6088 ± 20%     +21.0%       7365 ±  6%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU53.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU54.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU55.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU56.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      6651 ±  6%      +8.4%       7210 ±  6%  interrupts.CPU57.NMI:Non-maskable_interrupts
      6651 ±  6%      +8.4%       7210 ±  6%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU58.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      6675 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU6.NMI:Non-maskable_interrupts
      6675 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      5722 ± 20%     +26.1%       7213 ±  6%  interrupts.CPU63.NMI:Non-maskable_interrupts
      5722 ± 20%     +26.1%       7213 ±  6%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
     41159 ±147%     -98.7%     543.50 ± 13%  interrupts.CPU66.RES:Rescheduling_interrupts
      6675 ±  6%      +8.3%       7233 ±  6%  interrupts.CPU7.NMI:Non-maskable_interrupts
      6675 ±  6%      +8.3%       7233 ±  6%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      6645 ±  6%      +8.5%       7209 ±  6%  interrupts.CPU75.NMI:Non-maskable_interrupts
      6645 ±  6%      +8.5%       7209 ±  6%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
      6644 ±  6%      +8.5%       7210 ±  6%  interrupts.CPU76.NMI:Non-maskable_interrupts
      6644 ±  6%      +8.5%       7210 ±  6%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      6641 ±  6%      +8.5%       7209 ±  6%  interrupts.CPU77.NMI:Non-maskable_interrupts
      6641 ±  6%      +8.5%       7209 ±  6%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU78.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU79.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      6686 ±  6%      +8.0%       7221 ±  6%  interrupts.CPU8.NMI:Non-maskable_interrupts
      6686 ±  6%      +8.0%       7221 ±  6%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU80.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.4%       7367 ±  6%  interrupts.CPU81.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.4%       7367 ±  6%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      7056 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU82.NMI:Non-maskable_interrupts
      7056 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
      7057 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU83.NMI:Non-maskable_interrupts
      7057 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU84.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
      7057 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU85.NMI:Non-maskable_interrupts
      7057 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU86.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU87.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU88.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU89.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU9.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU90.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU91.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU92.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU93.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.4%       7366 ±  6%  interrupts.CPU94.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.4%       7366 ±  6%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      7059 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU95.NMI:Non-maskable_interrupts
      7059 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU96.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
      7072 ±  5%      +4.1%       7365 ±  6%  interrupts.CPU97.NMI:Non-maskable_interrupts
      7072 ±  5%      +4.1%       7365 ±  6%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU98.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU98.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU99.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU99.PMI:Performance_monitoring_interrupts
    117.75 ± 15%     +52.0%     179.00 ± 10%  interrupts.IWI:IRQ_work_interrupts
   1034638 ± 10%     -35.2%     670935 ± 11%  interrupts.RES:Rescheduling_interrupts
      9.66 ±  4%      -4.0        5.69 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      9.57 ±  4%      -3.9        5.64 ±  2%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      9.34 ±  4%      -3.9        5.42 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      8.19 ±  4%      -3.4        4.80 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.new_sync_read
      6.05            -2.5        3.59        perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      4.73            -1.9        2.82        perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      3.66            -1.7        1.97        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.new_sync_read
      3.09 ±  4%      -1.5        1.61        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__GI___libc_read
      3.75 ±  6%      -1.4        2.31 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      3.07            -1.4        1.69        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_read
     44.10            -1.4       42.73        perf-profile.calltrace.cycles-pp.__GI___libc_write
      3.06            -1.3        1.73        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_write
      3.00 ±  6%      -1.2        1.78 ±  4%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      3.02 ±  4%      -1.2        1.85        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__GI___libc_write
      1.89            -1.0        0.92        perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
      3.26 ±  5%      -0.9        2.38        perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      2.36            -0.8        1.51        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read
      1.89            -0.8        1.05        perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.69            -0.8        0.90        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_read
      1.71            -0.8        0.92        perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
      1.79            -0.8        1.01        perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      3.01 ±  6%      -0.7        2.28        perf-profile.calltrace.cycles-pp.switch_fpu_return.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      1.40 ±  2%      -0.7        0.75        perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      1.03            -0.6        0.39 ± 57%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.30 ± 10%      -0.6        0.74 ±  6%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      1.25            -0.6        0.70        perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.new_sync_read.vfs_read.ksys_read
      1.73            -0.6        1.18        perf-profile.calltrace.cycles-pp.__switch_to.__GI___libc_read
      1.13 ±  2%      -0.5        0.61        perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      1.20 ±  3%      -0.5        0.69 ±  2%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      1.34 ±  2%      -0.5        0.83 ±  3%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.23 ± 10%      -0.5        0.72 ±  6%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.06 ±  2%      -0.5        0.57        perf-profile.calltrace.cycles-pp.sched_clock_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
      1.04            -0.5        0.56        perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_task_fair.__schedule.schedule.pipe_read
      1.11 ±  2%      -0.5        0.64        perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
      1.48 ±  2%      -0.5        1.01        perf-profile.calltrace.cycles-pp.__switch_to_asm.__GI___libc_read
      1.11 ±  2%      -0.4        0.67        perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
      0.97 ±  3%      -0.4        0.55 ±  2%  perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule
      1.08 ±  6%      -0.4        0.66 ±  6%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      0.98 ±  2%      -0.4        0.56        perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      1.07 ±  3%      -0.4        0.68        perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.80 ±  5%      -0.2        0.56 ±  5%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      0.78 ±  5%      -0.2        0.55 ±  4%  perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.27            -0.2        1.09        perf-profile.calltrace.cycles-pp.native_write_msr.__GI___libc_read
      0.58            -0.0        0.56        perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.pipe_read.new_sync_read
     37.18            +1.5       38.69        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     36.93            +1.6       38.55        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      0.00            +1.7        1.66        perf-profile.calltrace.cycles-pp.ring_buffer_lock_reserve.trace_vbprintk.__trace_bprintk.__schedule.schedule
     54.55            +1.7       56.24        perf-profile.calltrace.cycles-pp.__GI___libc_read
      0.00            +2.1        2.12 ±  2%  perf-profile.calltrace.cycles-pp.format_decode.vbin_printf.trace_vbprintk.__trace_bprintk.__schedule
     31.13            +2.6       33.70        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     29.77            +2.9       32.67        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      0.00            +2.9        2.94 ±  2%  perf-profile.calltrace.cycles-pp.vbin_printf.trace_vbprintk.__trace_bprintk.__schedule.schedule
      0.00            +3.4        3.35        perf-profile.calltrace.cycles-pp.probe_sched_switch.__schedule.schedule.pipe_read.new_sync_read
      0.00            +3.4        3.41        perf-profile.calltrace.cycles-pp.probe_sched_wakeup.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
     27.33            +3.8       31.13        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     26.62            +4.1       30.69        perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     42.39            +5.8       48.18        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     43.08            +6.1       49.15        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     21.46 ±  2%      +6.2       27.62        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
     20.61 ±  2%      +6.4       27.05        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
      0.00            +6.5        6.46        perf-profile.calltrace.cycles-pp.trace_vbprintk.__trace_bprintk.__schedule.schedule.pipe_read
     20.38 ±  2%      +6.5       26.91        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
      0.00            +6.6        6.56        perf-profile.calltrace.cycles-pp.__trace_bprintk.__schedule.schedule.pipe_read.new_sync_read
     19.91 ±  2%      +6.7       26.63        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
     33.81            +8.2       42.00        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     32.21            +8.9       41.09        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     28.99           +10.2       39.15        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     27.48           +11.0       38.50        perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      0.00           +11.3       11.28        perf-profile.calltrace.cycles-pp.trace_save_cmdline.tracing_record_taskinfo.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      0.00           +11.4       11.37        perf-profile.calltrace.cycles-pp.tracing_record_taskinfo.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00           +12.2       12.22        perf-profile.calltrace.cycles-pp.trace_save_cmdline.tracing_record_taskinfo_sched_switch.__schedule.schedule.pipe_read
      0.00           +12.3       12.29        perf-profile.calltrace.cycles-pp.tracing_record_taskinfo_sched_switch.__schedule.schedule.pipe_read.new_sync_read
      1.56           +14.3       15.82        perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     17.22 ±  2%     +15.4       32.65        perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
     16.60 ±  2%     +15.7       32.32        perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.new_sync_read.vfs_read
      9.66 ±  4%      -4.0        5.70 ±  2%  perf-profile.children.cycles-pp.ttwu_do_activate
      9.58 ±  4%      -3.9        5.65 ±  2%  perf-profile.children.cycles-pp.activate_task
      9.36 ±  4%      -3.9        5.43 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
      8.21 ±  4%      -3.4        4.82 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      6.82 ±  4%      -2.9        3.89        perf-profile.children.cycles-pp.syscall_return_via_sysret
      6.14            -2.7        3.42        perf-profile.children.cycles-pp.entry_SYSCALL_64
      5.76 ±  5%      -2.6        3.20 ±  2%  perf-profile.children.cycles-pp.update_load_avg
      6.07            -2.5        3.61        perf-profile.children.cycles-pp.select_task_rq_fair
      4.88 ±  2%      -2.0        2.86 ±  2%  perf-profile.children.cycles-pp.update_curr
      4.86            -2.0        2.89        perf-profile.children.cycles-pp.select_idle_sibling
      3.91            -1.8        2.09        perf-profile.children.cycles-pp.pick_next_task_fair
      3.74            -1.6        2.10        perf-profile.children.cycles-pp.reweight_entity
     44.51            -1.5       42.99        perf-profile.children.cycles-pp.__GI___libc_write
      3.87 ±  6%      -1.5        2.37 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      3.07 ±  6%      -1.3        1.82 ±  4%  perf-profile.children.cycles-pp.dequeue_entity
      1.92            -1.0        0.94        perf-profile.children.cycles-pp.copy_page_to_iter
      2.46 ±  2%      -0.9        1.54 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      2.44            -0.9        1.56        perf-profile.children.cycles-pp.prepare_to_wait_event
      2.04            -0.9        1.17        perf-profile.children.cycles-pp.mutex_lock
      1.96 ±  7%      -0.8        1.13 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      1.77            -0.8        0.93        perf-profile.children.cycles-pp.set_next_entity
      1.76            -0.8        0.95        perf-profile.children.cycles-pp.copy_page_from_iter
      3.88 ±  8%      -0.8        3.08        perf-profile.children.cycles-pp.__prepare_exit_to_usermode
      2.37            -0.8        1.60        perf-profile.children.cycles-pp.__switch_to
      1.86            -0.8        1.10        perf-profile.children.cycles-pp.sched_clock_cpu
      3.03 ±  6%      -0.7        2.29        perf-profile.children.cycles-pp.switch_fpu_return
      1.85            -0.7        1.11        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.70 ±  6%      -0.7        0.96        perf-profile.children.cycles-pp.update_cfs_group
      1.45            -0.7        0.73        perf-profile.children.cycles-pp.__update_load_avg_se
      1.54            -0.7        0.83        perf-profile.children.cycles-pp.___might_sleep
      1.42 ±  2%      -0.7        0.76        perf-profile.children.cycles-pp.check_preempt_curr
      1.75            -0.7        1.10        perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      1.36            -0.6        0.77 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      1.42            -0.6        0.83 ±  2%  perf-profile.children.cycles-pp.common_file_perm
      1.62 ±  5%      -0.6        1.04 ±  5%  perf-profile.children.cycles-pp.__fdget_pos
      1.17 ±  2%      -0.5        0.62        perf-profile.children.cycles-pp.check_preempt_wakeup
      1.55 ±  2%      -0.5        1.04        perf-profile.children.cycles-pp.__switch_to_asm
      1.16 ±  3%      -0.5        0.64 ±  2%  perf-profile.children.cycles-pp.fsnotify
      0.99 ±  2%      -0.5        0.48        perf-profile.children.cycles-pp.__might_sleep
      1.51 ±  5%      -0.5        1.00 ±  5%  perf-profile.children.cycles-pp.__fget_light
      0.90 ±  2%      -0.5        0.40        perf-profile.children.cycles-pp.__might_fault
      1.20 ±  2%      -0.5        0.71 ±  2%  perf-profile.children.cycles-pp.cpumask_next_wrap
      0.95 ±  9%      -0.5        0.47 ±  2%  perf-profile.children.cycles-pp.touch_atime
      1.16 ±  2%      -0.5        0.69        perf-profile.children.cycles-pp._find_next_bit
      0.82 ±  8%      -0.5        0.35 ±  6%  perf-profile.children.cycles-pp.current_time
      0.90            -0.5        0.43        perf-profile.children.cycles-pp.pick_next_entity
      1.69            -0.4        1.25        perf-profile.children.cycles-pp.sched_clock
      0.75 ±  7%      -0.4        0.32 ±  2%  perf-profile.children.cycles-pp.atime_needs_update
      1.61            -0.4        1.21        perf-profile.children.cycles-pp.native_sched_clock
      1.21            -0.4        0.81        perf-profile.children.cycles-pp.update_rq_clock
      1.09 ±  6%      -0.4        0.72 ±  7%  perf-profile.children.cycles-pp.__fget_files
      0.67 ±  2%      -0.4        0.32        perf-profile.children.cycles-pp._cond_resched
      0.72 ± 11%      -0.3        0.40 ±  5%  perf-profile.children.cycles-pp.cpuacct_charge
      0.78 ±  7%      -0.3        0.47 ±  2%  perf-profile.children.cycles-pp.update_min_vruntime
      0.70            -0.3        0.40 ±  2%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.65 ±  2%      -0.3        0.35 ±  2%  perf-profile.children.cycles-pp.cpumask_next
      0.71 ±  2%      -0.3        0.42 ±  3%  perf-profile.children.cycles-pp.fsnotify_parent
      0.91            -0.3        0.63 ±  2%  perf-profile.children.cycles-pp.__x86_retpoline_rax
      0.61 ±  2%      -0.3        0.33 ±  3%  perf-profile.children.cycles-pp.copyout
      0.76            -0.3        0.48 ±  2%  perf-profile.children.cycles-pp.__pthread_disable_asynccancel
      0.58            -0.3        0.31 ±  2%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.63 ±  2%      -0.3        0.37 ±  4%  perf-profile.children.cycles-pp.account_entity_dequeue
      0.44            -0.2        0.19 ±  3%  perf-profile.children.cycles-pp.clear_buddies
      0.68            -0.2        0.45        perf-profile.children.cycles-pp.mutex_unlock
      0.58            -0.2        0.36        perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.53            -0.2        0.31 ±  2%  perf-profile.children.cycles-pp.copyin
      1.39            -0.2        1.17        perf-profile.children.cycles-pp.native_write_msr
      0.41 ±  2%      -0.2        0.20 ±  2%  perf-profile.children.cycles-pp.rcu_all_qs
      0.58            -0.2        0.38 ±  2%  perf-profile.children.cycles-pp.account_entity_enqueue
      0.33 ± 15%      -0.2        0.12 ±  6%  perf-profile.children.cycles-pp.timestamp_truncate
      0.51 ±  2%      -0.2        0.31 ±  3%  perf-profile.children.cycles-pp.available_idle_cpu
      0.36 ± 11%      -0.2        0.17 ±  6%  perf-profile.children.cycles-pp.__sb_start_write
      0.54            -0.2        0.36 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
      0.42 ±  3%      -0.2        0.25 ±  3%  perf-profile.children.cycles-pp.finish_wait
      0.64 ±  2%      -0.2        0.47        perf-profile.children.cycles-pp.___perf_sw_event
      0.35 ±  5%      -0.2        0.18 ±  2%  perf-profile.children.cycles-pp.__syscall_return_slowpath
      0.44 ±  2%      -0.2        0.28 ±  3%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.40 ± 15%      -0.1        0.25 ±  9%  perf-profile.children.cycles-pp.set_next_buddy
      0.69 ± 14%      -0.1        0.55 ±  2%  perf-profile.children.cycles-pp.file_update_time
      0.33            -0.1        0.18 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.29            -0.1        0.15 ±  5%  perf-profile.children.cycles-pp.__x64_sys_read
      0.28            -0.1        0.15 ±  2%  perf-profile.children.cycles-pp.wakeup_preempt_entity
      0.22 ± 11%      -0.1        0.10 ±  7%  perf-profile.children.cycles-pp.__sb_end_write
      0.62 ±  2%      -0.1        0.52 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.51 ±  2%      -0.1        0.41 ±  5%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.50 ±  2%      -0.1        0.40 ±  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.22 ± 13%      -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.28 ±  2%      -0.1        0.18 ±  2%  perf-profile.children.cycles-pp.__x64_sys_write
      0.56 ±  2%      -0.1        0.46 ±  3%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.26            -0.1        0.16 ±  4%  perf-profile.children.cycles-pp.iov_iter_init
      0.24 ±  4%      -0.1        0.15 ±  4%  perf-profile.children.cycles-pp.put_prev_entity
      0.17 ±  2%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.rb_erase
      0.18 ±  8%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.49            -0.1        0.40 ±  2%  perf-profile.children.cycles-pp.fput_many
      0.17 ±  6%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.place_entity
      0.19 ±  2%      -0.1        0.10        perf-profile.children.cycles-pp.__list_add_valid
      0.18 ±  2%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.find_next_bit
      0.14            -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.__enqueue_entity
      0.22 ±  5%      -0.1        0.14 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.20 ±  3%      -0.1        0.12 ±  5%  perf-profile.children.cycles-pp.aa_file_perm
      0.15 ±  5%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.11 ±  4%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.11 ±  3%      -0.1        0.05        perf-profile.children.cycles-pp.__bitmap_and
      0.21 ±  2%      -0.1        0.15 ±  9%  perf-profile.children.cycles-pp.tick_sched_handle
      0.19 ±  3%      -0.1        0.13        perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.24            -0.1        0.18 ± 11%  perf-profile.children.cycles-pp.tick_sched_timer
      0.29 ±  2%      -0.1        0.24 ± 10%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.21 ±  2%      -0.1        0.15 ±  9%  perf-profile.children.cycles-pp.update_process_times
      0.45 ±  2%      -0.1        0.39 ±  4%  perf-profile.children.cycles-pp.asm_call_on_stack
      0.16 ±  4%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.clockevents_program_event
      0.16 ±  2%      -0.1        0.11        perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.08 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.rw_verify_area
      0.14 ±  5%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.ktime_get
      0.53            -0.0        0.49        perf-profile.children.cycles-pp.__pthread_enable_asynccancel
      0.09            -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.default_wake_function
      0.20 ±  4%      -0.0        0.17        perf-profile.children.cycles-pp.deactivate_task
      0.12 ±  4%      -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.fput
      0.12 ±  7%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.scheduler_tick
      0.08 ±  8%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.__vfs_write
      0.07 ±  5%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.ring_buffer_event_data
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.hrtick_update
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.perf_exclude_event
      0.00            +0.1        0.09        perf-profile.children.cycles-pp.ring_buffer_nest_end
      0.00            +0.1        0.10 ±  5%  perf-profile.children.cycles-pp.tracing_record_taskinfo_skip
      0.00            +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.tracing_generic_entry_update
      0.00            +0.2        0.20 ±  4%  perf-profile.children.cycles-pp.ring_buffer_nest_start
      0.00            +0.2        0.21 ±  2%  perf-profile.children.cycles-pp.rb_commit
      0.00            +0.3        0.27        perf-profile.children.cycles-pp.__rb_reserve_next
      0.00            +0.3        0.32        perf-profile.children.cycles-pp.trace_clock_local
      0.00            +0.3        0.34        perf-profile.children.cycles-pp.ring_buffer_unlock_commit
     54.94            +1.5       56.47        perf-profile.children.cycles-pp.__GI___libc_read
      0.00            +1.7        1.66        perf-profile.children.cycles-pp.ring_buffer_lock_reserve
      0.00            +2.2        2.22 ±  2%  perf-profile.children.cycles-pp.format_decode
     31.17            +2.6       33.74        perf-profile.children.cycles-pp.ksys_write
     29.84            +2.9       32.72        perf-profile.children.cycles-pp.vfs_write
      0.00            +3.0        3.02 ±  2%  perf-profile.children.cycles-pp.vbin_printf
      0.00            +3.4        3.37        perf-profile.children.cycles-pp.probe_sched_switch
      0.00            +3.4        3.41        perf-profile.children.cycles-pp.probe_sched_wakeup
     27.37            +3.8       31.18        perf-profile.children.cycles-pp.new_sync_write
     26.69            +4.0       30.74        perf-profile.children.cycles-pp.pipe_write
     21.49 ±  2%      +6.1       27.64        perf-profile.children.cycles-pp.__wake_up_common_lock
     20.61 ±  2%      +6.4       27.05        perf-profile.children.cycles-pp.__wake_up_common
     20.41 ±  2%      +6.5       26.94        perf-profile.children.cycles-pp.autoremove_wake_function
      0.00            +6.5        6.54        perf-profile.children.cycles-pp.trace_vbprintk
      0.00            +6.6        6.59        perf-profile.children.cycles-pp.__trace_bprintk
     19.93 ±  2%      +6.7       26.64        perf-profile.children.cycles-pp.try_to_wake_up
     79.41            +7.4       86.80        perf-profile.children.cycles-pp.do_syscall_64
     80.32            +7.6       87.91        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     33.89            +8.2       42.05        perf-profile.children.cycles-pp.ksys_read
     32.29            +8.9       41.15        perf-profile.children.cycles-pp.vfs_read
     29.04           +10.1       39.18        perf-profile.children.cycles-pp.new_sync_read
     27.63           +11.0       38.59        perf-profile.children.cycles-pp.pipe_read
      0.00           +11.4       11.39        perf-profile.children.cycles-pp.tracing_record_taskinfo
      0.00           +12.3       12.35        perf-profile.children.cycles-pp.tracing_record_taskinfo_sched_switch
      1.58 ±  2%     +14.2       15.83        perf-profile.children.cycles-pp.ttwu_do_wakeup
     17.33           +15.4       32.74        perf-profile.children.cycles-pp.schedule
     16.84 ±  2%     +15.7       32.50        perf-profile.children.cycles-pp.__schedule
      0.00           +23.6       23.55        perf-profile.children.cycles-pp.trace_save_cmdline
      6.81 ±  4%      -2.9        3.88        perf-profile.self.cycles-pp.syscall_return_via_sysret
      5.44            -2.4        3.00        perf-profile.self.cycles-pp.entry_SYSCALL_64
      9.40 ±  2%      -2.1        7.32        perf-profile.self.cycles-pp.do_syscall_64
      2.39 ±  6%      -1.0        1.38 ±  4%  perf-profile.self.cycles-pp.update_load_avg
      1.89 ±  8%      -0.8        1.08 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      2.06 ±  9%      -0.8        1.27 ±  7%  perf-profile.self.cycles-pp.update_curr
      3.01 ±  6%      -0.8        2.24        perf-profile.self.cycles-pp.switch_fpu_return
      1.67 ±  6%      -0.7        0.94        perf-profile.self.cycles-pp.update_cfs_group
      2.23            -0.7        1.52        perf-profile.self.cycles-pp.__switch_to
      1.78            -0.7        1.07        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.42            -0.7        0.72        perf-profile.self.cycles-pp.__update_load_avg_se
      1.51            -0.7        0.82        perf-profile.self.cycles-pp.___might_sleep
      1.14 ±  2%      -0.7        0.46 ±  2%  perf-profile.self.cycles-pp.new_sync_read
      1.53            -0.6        0.88        perf-profile.self.cycles-pp.__schedule
      1.57            -0.6        0.93        perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      1.45            -0.6        0.85        perf-profile.self.cycles-pp.pipe_read
      1.35            -0.6        0.76 ±  3%  perf-profile.self.cycles-pp.__calc_delta
      1.50 ±  5%      -0.6        0.94        perf-profile.self.cycles-pp.enqueue_task_fair
      1.19            -0.5        0.65        perf-profile.self.cycles-pp.reweight_entity
      1.55 ±  2%      -0.5        1.03        perf-profile.self.cycles-pp.__switch_to_asm
      1.12 ±  3%      -0.5        0.62 ±  2%  perf-profile.self.cycles-pp.fsnotify
      1.22 ±  2%      -0.5        0.72 ±  3%  perf-profile.self.cycles-pp.common_file_perm
      1.21            -0.5        0.72        perf-profile.self.cycles-pp.select_task_rq_fair
      1.42            -0.5        0.93 ±  3%  perf-profile.self.cycles-pp.select_idle_sibling
      1.15 ±  2%      -0.5        0.69        perf-profile.self.cycles-pp._find_next_bit
      0.89 ±  2%      -0.5        0.43        perf-profile.self.cycles-pp.__might_sleep
      1.58            -0.4        1.17        perf-profile.self.cycles-pp.native_sched_clock
      0.98            -0.4        0.61        perf-profile.self.cycles-pp.dequeue_task_fair
      0.78            -0.4        0.41        perf-profile.self.cycles-pp.try_to_wake_up
      0.73 ±  3%      -0.4        0.36        perf-profile.self.cycles-pp.check_preempt_wakeup
      1.05            -0.4        0.69        perf-profile.self.cycles-pp.prepare_to_wait_event
      1.06 ±  7%      -0.4        0.71 ±  7%  perf-profile.self.cycles-pp.__fget_files
      0.88            -0.3        0.54        perf-profile.self.cycles-pp.pipe_write
      0.55            -0.3        0.21 ±  3%  perf-profile.self.cycles-pp.set_next_entity
      0.72 ± 11%      -0.3        0.40 ±  4%  perf-profile.self.cycles-pp.cpuacct_charge
      0.83            -0.3        0.52 ±  2%  perf-profile.self.cycles-pp.vfs_read
      0.76 ±  7%      -0.3        0.45 ±  2%  perf-profile.self.cycles-pp.update_min_vruntime
      0.77 ± 14%      -0.3        0.48 ±  9%  perf-profile.self.cycles-pp.enqueue_entity
      0.66 ±  2%      -0.3        0.38        perf-profile.self.cycles-pp.pick_next_task_fair
      0.62            -0.3        0.34        perf-profile.self.cycles-pp.__GI___libc_write
      0.66            -0.3        0.38 ±  2%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.60 ±  2%      -0.3        0.32 ±  2%  perf-profile.self.cycles-pp.__GI___libc_read
      0.64            -0.3        0.37        perf-profile.self.cycles-pp.fsnotify_parent
      0.47 ±  2%      -0.3        0.21 ±  2%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.69            -0.3        0.44 ±  2%  perf-profile.self.cycles-pp.__pthread_disable_asynccancel
      0.54            -0.3        0.29 ±  2%  perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.55 ±  2%      -0.2        0.30        perf-profile.self.cycles-pp.ksys_read
      0.79 ±  2%      -0.2        0.55 ±  2%  perf-profile.self.cycles-pp.__x86_retpoline_rax
      0.76            -0.2        0.52        perf-profile.self.cycles-pp.mutex_lock
      0.47            -0.2        0.24 ±  2%  perf-profile.self.cycles-pp.schedule
      0.39            -0.2        0.16 ±  2%  perf-profile.self.cycles-pp.clear_buddies
      1.39            -0.2        1.17        perf-profile.self.cycles-pp.native_write_msr
      0.66            -0.2        0.44        perf-profile.self.cycles-pp.mutex_unlock
      0.45            -0.2        0.23        perf-profile.self.cycles-pp.pick_next_entity
      0.64            -0.2        0.43 ±  5%  perf-profile.self.cycles-pp.vfs_write
      0.53 ±  2%      -0.2        0.32 ±  3%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.56            -0.2        0.35        perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.48            -0.2        0.27 ±  2%  perf-profile.self.cycles-pp.cpumask_next_wrap
      0.43            -0.2        0.23        perf-profile.self.cycles-pp.copy_page_from_iter
      0.35 ± 10%      -0.2        0.16 ±  5%  perf-profile.self.cycles-pp.__sb_start_write
      0.31 ± 16%      -0.2        0.12 ± 10%  perf-profile.self.cycles-pp.timestamp_truncate
      0.53            -0.2        0.35 ±  2%  perf-profile.self.cycles-pp.account_entity_enqueue
      0.34            -0.2        0.16        perf-profile.self.cycles-pp.rcu_all_qs
      0.45 ±  4%      -0.2        0.28 ±  4%  perf-profile.self.cycles-pp.available_idle_cpu
      0.52            -0.2        0.35 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.28 ±  4%      -0.2        0.11 ±  7%  perf-profile.self.cycles-pp.current_time
      0.33 ±  6%      -0.2        0.17 ±  2%  perf-profile.self.cycles-pp.__syscall_return_slowpath
      0.28 ±  3%      -0.2        0.12        perf-profile.self.cycles-pp._cond_resched
      0.57 ±  3%      -0.2        0.42        perf-profile.self.cycles-pp.___perf_sw_event
      0.49 ±  4%      -0.1        0.34 ±  2%  perf-profile.self.cycles-pp.new_sync_write
      0.39 ± 15%      -0.1        0.25 ±  9%  perf-profile.self.cycles-pp.set_next_buddy
      0.32 ±  2%      -0.1        0.18 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.39 ±  3%      -0.1        0.26 ±  3%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.41            -0.1        0.28        perf-profile.self.cycles-pp.update_rq_clock
      0.33 ±  2%      -0.1        0.21 ±  2%  perf-profile.self.cycles-pp.autoremove_wake_function
      0.29 ±  9%      -0.1        0.17 ±  4%  perf-profile.self.cycles-pp.atime_needs_update
      0.27 ±  3%      -0.1        0.15 ±  5%  perf-profile.self.cycles-pp.dequeue_entity
      0.26            -0.1        0.14 ±  5%  perf-profile.self.cycles-pp.__x64_sys_read
      0.38 ± 11%      -0.1        0.27 ± 10%  perf-profile.self.cycles-pp.__fget_light
      0.21 ± 11%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__sb_end_write
      0.26 ±  3%      -0.1        0.15        perf-profile.self.cycles-pp.wakeup_preempt_entity
      0.21 ± 11%      -0.1        0.11 ± 15%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.15 ±  3%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__fdget_pos
      0.23            -0.1        0.13 ±  5%  perf-profile.self.cycles-pp.iov_iter_init
      0.26            -0.1        0.16 ±  4%  perf-profile.self.cycles-pp.ksys_write
      0.19 ±  4%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.check_preempt_curr
      0.12            -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__enqueue_entity
      0.17 ±  8%      -0.1        0.08        perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.15 ±  3%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.rb_erase
      0.25            -0.1        0.17 ±  2%  perf-profile.self.cycles-pp.__x64_sys_write
      0.47            -0.1        0.39 ±  2%  perf-profile.self.cycles-pp.fput_many
      0.37 ±  8%      -0.1        0.29 ± 14%  perf-profile.self.cycles-pp.security_file_permission
      0.15 ±  7%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.place_entity
      0.18 ±  5%      -0.1        0.10 ±  7%  perf-profile.self.cycles-pp.__wake_up_common
      0.17 ±  2%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.16 ±  4%      -0.1        0.09 ±  5%  perf-profile.self.cycles-pp.__might_fault
      0.20 ±  7%      -0.1        0.12 ±  6%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.16 ±  5%      -0.1        0.09        perf-profile.self.cycles-pp.__list_add_valid
      0.18 ±  5%      -0.1        0.11 ±  3%  perf-profile.self.cycles-pp.aa_file_perm
      0.18 ±  3%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.put_prev_entity
      0.16 ±  2%      -0.1        0.10        perf-profile.self.cycles-pp.__wake_up_common_lock
      0.11 ±  4%      -0.1        0.05        perf-profile.self.cycles-pp.__bitmap_and
      0.16 ±  2%      -0.1        0.11        perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.11            -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.cpumask_next
      0.09 ±  4%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.13 ±  6%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.ktime_get
      0.11 ±  4%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.finish_wait
      0.07            -0.0        0.03 ±100%  perf-profile.self.cycles-pp.rw_verify_area
      0.09 ±  4%      -0.0        0.05        perf-profile.self.cycles-pp.ttwu_do_activate
      0.39            -0.0        0.35 ±  3%  perf-profile.self.cycles-pp.finish_task_switch
      0.12            -0.0        0.08        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.51            -0.0        0.47        perf-profile.self.cycles-pp.__pthread_enable_asynccancel
      0.09            -0.0        0.06        perf-profile.self.cycles-pp.sched_clock
      0.20 ±  4%      -0.0        0.17        perf-profile.self.cycles-pp.deactivate_task
      0.15 ±  2%      -0.0        0.13 ±  5%  perf-profile.self.cycles-pp.activate_task
      0.07 ±  7%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.__vfs_write
      0.10 ± 14%      +0.0        0.14 ±  9%  perf-profile.self.cycles-pp.rb_insert_color
      0.07 ±  7%      +0.0        0.10        perf-profile.self.cycles-pp.rb_next
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.tracing_record_taskinfo_sched_switch
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.hrtick_update
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.trace_clock_local
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.tracing_record_taskinfo_skip
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.__trace_bprintk
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.ring_buffer_nest_end
      0.17            +0.1        0.27 ±  6%  perf-profile.self.cycles-pp.ttwu_do_wakeup
      0.00            +0.1        0.13 ±  3%  perf-profile.self.cycles-pp.tracing_generic_entry_update
      0.00            +0.1        0.14 ±  3%  perf-profile.self.cycles-pp.ring_buffer_unlock_commit
      0.00            +0.2        0.20 ±  5%  perf-profile.self.cycles-pp.ring_buffer_nest_start
      0.92 ±  4%      +0.2        1.11 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +0.2        0.21        perf-profile.self.cycles-pp.rb_commit
      0.00            +0.3        0.27        perf-profile.self.cycles-pp.__rb_reserve_next
      0.00            +0.9        0.88 ±  2%  perf-profile.self.cycles-pp.vbin_printf
      0.00            +1.0        1.04 ±  2%  perf-profile.self.cycles-pp.ring_buffer_lock_reserve
      0.00            +1.0        1.04 ±  4%  perf-profile.self.cycles-pp.trace_vbprintk
      0.00            +2.1        2.15 ±  2%  perf-profile.self.cycles-pp.format_decode
      0.00            +3.4        3.36        perf-profile.self.cycles-pp.probe_sched_switch
      0.00            +3.4        3.40        perf-profile.self.cycles-pp.probe_sched_wakeup
      0.00           +23.5       23.51        perf-profile.self.cycles-pp.trace_save_cmdline





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.8.0-rc1-00023-ge9f2fb8893b70" of type "text/plain" (206263 bytes)

View attachment "job-script" of type "text/plain" (7786 bytes)

View attachment "job.yaml" of type "text/plain" (5337 bytes)

View attachment "reproduce" of type "text/plain" (347 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ