lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Wed, 27 Jan 2021 10:16:51 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Roman Gushchin <guro@...com>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Imran Khan <imran.f.khan@...cle.com>,
        Shakeel Butt <shakeelb@...gle.com>,
        Michal Koutn <mkoutny@...e.com>,
        Johannes Weiner <hannes@...xchg.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com
Subject: [mm]  3de7d4f25a:  hackbench.throughput 45.2% improvement


Greeting,

FYI, we noticed a 45.2% improvement of hackbench.throughput due to commit:


commit: 3de7d4f25a7438f09fef4e71ef111f1805cd8e7c ("mm: memcg/slab: optimize objcg stock draining")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: hackbench
on test machine: 144 threads Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory
with following parameters:

	nr_threads: 100%
	iterations: 4
	mode: threads
	ipc: socket
	cpufreq_governor: performance
	ucode: 0x700001e

test-description: Hackbench is both a benchmark and a stress test for the Linux kernel scheduler.
test-url: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/sched/cfs-scheduler/hackbench.c

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.bad-altstack.ops_per_sec 14.8% improvement      |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory |
| test parameters  | class=memory                                                         |
|                  | cpufreq_governor=performance                                         |
|                  | disk=1HDD                                                            |
|                  | nr_threads=100%                                                      |
|                  | testtime=10s                                                         |
|                  | ucode=0x5003003                                                      |
+------------------+----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.procfs.ops_per_sec 24.2% improvement            |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory |
| test parameters  | class=filesystem                                                     |
|                  | cpufreq_governor=performance                                         |
|                  | disk=1HDD                                                            |
|                  | fs=ext4                                                              |
|                  | nr_threads=10%                                                       |
|                  | testtime=30s                                                         |
|                  | ucode=0x5003003                                                      |
+------------------+----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
  gcc-9/performance/socket/4/x86_64-rhel-8.3/threads/100%/debian-10.4-x86_64-20200603.cgz/lkp-cpl-4sp1/hackbench/0x700001e

commit: 
  d3921cb8be ("mm: fix initialization of struct page for holes in memory layout")
  3de7d4f25a ("mm: memcg/slab: optimize objcg stock draining")

d3921cb8be29ce56 3de7d4f25a7438f09fef4e71ef1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     51479           +45.2%      74753        hackbench.throughput
      1315           -29.2%     931.13        hackbench.time.elapsed_time
      1315           -29.2%     931.13        hackbench.time.elapsed_time.max
 5.546e+08 ± 11%    +609.0%  3.932e+09        hackbench.time.involuntary_context_switches
      2582 ± 17%     -26.1%       1907 ± 18%  hackbench.time.major_page_faults
    184500           -30.7%     127875        hackbench.time.system_time
      1816 ±  2%     +59.7%       2900        hackbench.time.user_time
 1.245e+09 ±  8%    +352.7%  5.638e+09        hackbench.time.voluntary_context_switches
      1356           -28.0%     976.42        uptime.boot
   1292957 ± 13%     +85.0%    2392608 ± 14%  numa-numastat.node3.local_node
   1329025 ± 13%     +83.9%    2443792 ± 13%  numa-numastat.node3.numa_hit
     98.00            -1.0%      97.00        vmstat.cpu.sy
   1404918 ±  7%    +631.6%   10277830 ±  2%  vmstat.system.cs
    510555          +178.1%    1419769 ±  2%  vmstat.system.in
      0.67 ±  4%      -0.1        0.59 ±  3%  mpstat.cpu.all.idle%
      0.91 ±  2%      +0.9        1.82 ±  3%  mpstat.cpu.all.irq%
      0.03 ±  6%      -0.0        0.03 ±  3%  mpstat.cpu.all.soft%
      0.98            +1.2        2.20        mpstat.cpu.all.usr%
 5.767e+08 ±  4%     -63.5%  2.106e+08 ± 12%  cpuidle.C1.time
  38987487 ±  2%     -54.7%   17648341 ±  7%  cpuidle.C1.usage
 6.814e+08 ±  5%     -25.4%  5.083e+08 ± 16%  cpuidle.C1E.time
   4627972 ±  6%     -47.1%    2445949 ± 11%  cpuidle.C1E.usage
  28364267 ± 49%    +402.5%  1.425e+08 ± 52%  cpuidle.C6.time
     94344 ± 20%    +226.6%     308127 ± 57%  cpuidle.C6.usage
   7116263 ±  4%     -16.8%    5920458 ± 10%  cpuidle.POLL.time
    875057 ±  5%     +57.2%    1375770 ± 16%  cpuidle.POLL.usage
    139169 ±  2%     -82.2%      24826 ±  3%  slabinfo.kmalloc-512.active_objs
     19526 ±  2%     -82.3%       3451 ±  2%  slabinfo.kmalloc-512.active_slabs
    156219 ±  2%     -82.3%      27615 ±  2%  slabinfo.kmalloc-512.num_objs
     19526 ±  2%     -82.3%       3451 ±  2%  slabinfo.kmalloc-512.num_slabs
    134115 ±  2%     -59.3%      54639        slabinfo.skbuff_head_cache.active_objs
      2659 ±  2%     -66.2%     898.25        slabinfo.skbuff_head_cache.active_slabs
    170199 ±  2%     -66.2%      57527        slabinfo.skbuff_head_cache.num_objs
      2659 ±  2%     -66.2%     898.25        slabinfo.skbuff_head_cache.num_slabs
   1238587           -21.0%     978955 ± 10%  meminfo.Active
   1238587           -21.0%     978955 ± 10%  meminfo.Active(anon)
    364097 ±  2%     +55.0%     564366 ± 19%  meminfo.Inactive
    364097 ±  2%     +55.0%     564366 ± 19%  meminfo.Inactive(anon)
    105342 ±  9%    +151.7%     265155 ± 44%  meminfo.Mapped
      4223           +11.5%       4709 ±  6%  meminfo.PageTables
    475353           -19.5%     382718        meminfo.SUnreclaim
    588608           -16.1%     493928        meminfo.Slab
      4524           +54.0%       6966 ±  4%  meminfo.max_used_kB
     13678 ±  7%    +197.3%      40667 ± 32%  numa-meminfo.node0.Mapped
     14341 ±141%     -97.5%     364.00 ±105%  numa-meminfo.node1.Active
     14341 ±141%     -97.5%     364.00 ±105%  numa-meminfo.node1.Active(anon)
     31149 ± 30%     -84.2%       4921 ±  6%  numa-meminfo.node1.KernelStack
     16598 ± 23%    +149.0%      41333 ± 26%  numa-meminfo.node1.Mapped
    121998 ± 11%     -45.4%      66637 ± 10%  numa-meminfo.node1.SUnreclaim
    144326 ±  7%     -42.0%      83773 ±  9%  numa-meminfo.node1.Slab
     21517 ± 46%    +133.7%      50287 ± 39%  numa-meminfo.node2.Mapped
    103232 ± 42%    +236.8%     347717 ± 40%  numa-meminfo.node3.Inactive
    103232 ± 42%    +236.8%     347717 ± 40%  numa-meminfo.node3.Inactive(anon)
     51111 ± 27%    +155.3%     130468 ± 54%  numa-meminfo.node3.Mapped
      3443 ±  8%    +193.2%      10094 ± 32%  numa-vmstat.node0.nr_mapped
    177.25 ± 11%     -28.2%     127.25 ±  9%  numa-vmstat.node0.nr_mlock
      3560 ±141%     -97.5%      90.75 ±105%  numa-vmstat.node1.nr_active_anon
     30917 ± 30%     -84.1%       4926 ±  6%  numa-vmstat.node1.nr_kernel_stack
      4110 ± 24%    +149.6%      10260 ± 27%  numa-vmstat.node1.nr_mapped
     30392 ± 11%     -45.2%      16659 ± 10%  numa-vmstat.node1.nr_slab_unreclaimable
      3560 ±141%     -97.5%      90.75 ±105%  numa-vmstat.node1.nr_zone_active_anon
      5274 ± 42%    +136.2%      12460 ± 39%  numa-vmstat.node2.nr_mapped
     25911 ± 43%    +233.2%      86346 ± 40%  numa-vmstat.node3.nr_inactive_anon
     12950 ± 31%    +151.1%      32518 ± 54%  numa-vmstat.node3.nr_mapped
     25911 ± 43%    +233.2%      86346 ± 40%  numa-vmstat.node3.nr_zone_inactive_anon
   1193785 ± 12%     +54.6%    1846002 ± 14%  numa-vmstat.node3.numa_hit
   1078977 ± 15%     +60.6%    1732973 ± 14%  numa-vmstat.node3.numa_local
    310546           -20.9%     245673 ± 10%  proc-vmstat.nr_active_anon
    571656            -3.2%     553385        proc-vmstat.nr_file_pages
     90935 ±  2%     +54.8%     140792 ± 19%  proc-vmstat.nr_inactive_anon
    106363            -3.9%     102168        proc-vmstat.nr_kernel_stack
     26325 ± 11%    +151.1%      66102 ± 43%  proc-vmstat.nr_mapped
      1053           +12.0%       1180 ±  6%  proc-vmstat.nr_page_table_pages
    329411            -5.5%     311140        proc-vmstat.nr_shmem
     28314            -1.8%      27797        proc-vmstat.nr_slab_reclaimable
    118822           -19.5%      95612        proc-vmstat.nr_slab_unreclaimable
    310546           -20.9%     245673 ± 10%  proc-vmstat.nr_zone_active_anon
     90935 ±  2%     +54.8%     140792 ± 19%  proc-vmstat.nr_zone_inactive_anon
   5496694           +20.5%    6621551 ±  3%  proc-vmstat.numa_hit
   5301424           +21.2%    6426191 ±  3%  proc-vmstat.numa_local
     70257 ± 27%     +41.1%      99155 ±  4%  proc-vmstat.numa_pages_migrated
    351223            -2.0%     344063        proc-vmstat.pgactivate
   6011943           +64.8%    9905622 ±  4%  proc-vmstat.pgalloc_normal
   4912356 ±  3%     -27.7%    3551827 ±  2%  proc-vmstat.pgfault
   5704926           +68.5%    9615410 ±  4%  proc-vmstat.pgfree
     70257 ± 27%     +41.1%      99155 ±  4%  proc-vmstat.pgmigrate_success
    209094           -32.5%     141143 ±  2%  proc-vmstat.pgreuse
    123449 ± 62%     -59.0%      50635 ±134%  sched_debug.cfs_rq:/.MIN_vruntime.avg
  13988272 ± 52%     -68.3%    4431193 ±115%  sched_debug.cfs_rq:/.MIN_vruntime.max
   1283183 ± 57%     -64.2%     459455 ±124%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      8641 ±  5%     -15.9%       7264        sched_debug.cfs_rq:/.load.avg
     66297 ± 91%     -74.3%      17061 ± 17%  sched_debug.cfs_rq:/.load.max
      7145 ± 67%     -63.0%       2647 ± 22%  sched_debug.cfs_rq:/.load.stddev
      1.62 ±  5%     +43.6%       2.32 ± 12%  sched_debug.cfs_rq:/.load_avg.min
    123449 ± 62%     -59.0%      50635 ±134%  sched_debug.cfs_rq:/.max_vruntime.avg
  13988272 ± 52%     -68.3%    4431193 ±115%  sched_debug.cfs_rq:/.max_vruntime.max
   1283183 ± 57%     -64.2%     459455 ±124%  sched_debug.cfs_rq:/.max_vruntime.stddev
  95742524 ±  2%     -26.5%   70405887 ±  4%  sched_debug.cfs_rq:/.min_vruntime.avg
 1.052e+08 ±  4%     -19.5%   84632324 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
  87247166 ±  2%     -42.7%   49960558        sched_debug.cfs_rq:/.min_vruntime.min
   4722211 ± 22%    +128.4%   10787549 ± 12%  sched_debug.cfs_rq:/.min_vruntime.stddev
     19408 ±  2%      -6.9%      18075 ±  5%  sched_debug.cfs_rq:/.runnable_avg.avg
     34906 ±  3%     -15.4%      29523 ±  4%  sched_debug.cfs_rq:/.runnable_avg.max
      4281 ± 12%     +86.7%       7992 ± 11%  sched_debug.cfs_rq:/.runnable_avg.min
      6067 ±  3%     -28.9%       4314 ± 12%  sched_debug.cfs_rq:/.runnable_avg.stddev
   4709775 ± 22%    +125.9%   10640405 ± 12%  sched_debug.cfs_rq:/.spread0.stddev
      1605           -13.4%       1390 ±  3%  sched_debug.cfs_rq:/.util_avg.max
    479.64 ±  2%     +24.3%     596.13 ±  7%  sched_debug.cfs_rq:/.util_avg.min
    170.56 ±  3%     -19.7%     136.98 ±  7%  sched_debug.cfs_rq:/.util_avg.stddev
    959.94           -26.0%     710.39 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      2196           -31.0%       1516 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.max
     17.92 ± 36%   +1069.6%     209.55 ± 21%  sched_debug.cfs_rq:/.util_est_enqueued.min
    498.25 ±  2%     -54.5%     226.77 ±  8%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    360873 ±  5%     -18.2%     295164 ±  5%  sched_debug.cpu.avg_idle.avg
      4018 ±  5%     -42.4%       2313 ±  9%  sched_debug.cpu.avg_idle.min
    357106           -10.4%     320047 ±  5%  sched_debug.cpu.avg_idle.stddev
    677491 ±  2%     -27.6%     490674 ±  3%  sched_debug.cpu.clock.avg
    678424 ±  2%     -26.9%     495776 ±  3%  sched_debug.cpu.clock.max
    676540 ±  2%     -28.3%     485260 ±  3%  sched_debug.cpu.clock.min
    550.30 ± 23%    +453.0%       3043 ± 15%  sched_debug.cpu.clock.stddev
    671354 ±  2%     -28.1%     482587 ±  3%  sched_debug.cpu.clock_task.avg
    672474 ±  2%     -27.2%     489370 ±  3%  sched_debug.cpu.clock_task.max
    663609 ±  2%     -30.1%     463559 ±  2%  sched_debug.cpu.clock_task.min
      1024 ± 25%    +319.4%       4295 ± 12%  sched_debug.cpu.clock_task.stddev
     24423           -17.6%      20122        sched_debug.cpu.curr->pid.avg
     30114           -17.5%      24845        sched_debug.cpu.curr->pid.max
      0.00 ± 22%    +452.1%       0.00 ± 15%  sched_debug.cpu.next_balance.stddev
     18.98 ±  3%      -6.6%      17.74 ±  4%  sched_debug.cpu.nr_running.avg
     41.70 ±  3%     -23.3%      31.97 ±  4%  sched_debug.cpu.nr_running.max
      0.74 ± 10%    +630.5%       5.44 ± 15%  sched_debug.cpu.nr_running.min
     10.41 ±  3%     -52.6%       4.93 ± 12%  sched_debug.cpu.nr_running.stddev
   6229084 ±  8%    +412.1%   31899136        sched_debug.cpu.nr_switches.avg
   7584168 ± 10%    +486.3%   44466821 ±  4%  sched_debug.cpu.nr_switches.max
   5190559 ±  8%    +291.7%   20332558 ±  4%  sched_debug.cpu.nr_switches.min
    476341 ±  8%    +839.3%    4474464 ± 16%  sched_debug.cpu.nr_switches.stddev
    676515 ±  2%     -28.3%     485102 ±  3%  sched_debug.cpu_clk
    675769 ±  2%     -28.3%     484355 ±  3%  sched_debug.ktime
    678151 ±  2%     -28.4%     485464 ±  3%  sched_debug.sched_clk
     16.98 ±  2%     -17.2%      14.07        perf-stat.i.MPKI
 7.605e+09          +151.1%   1.91e+10        perf-stat.i.branch-instructions
      1.20            +0.2        1.37        perf-stat.i.branch-miss-rate%
  90406559          +178.8%   2.52e+08        perf-stat.i.branch-misses
     30.10 ±  2%     -15.8       14.29 ±  3%  perf-stat.i.cache-miss-rate%
 1.931e+08 ±  5%     -11.4%   1.71e+08 ±  3%  perf-stat.i.cache-misses
 6.418e+08 ±  2%     +97.6%  1.268e+09 ±  2%  perf-stat.i.cache-references
   1370491 ±  7%    +659.9%   10413883 ±  2%  perf-stat.i.context-switches
     12.40           -57.2%       5.31 ±  2%  perf-stat.i.cpi
    103561          +788.1%     919723 ±  4%  perf-stat.i.cpu-migrations
      2441 ±  5%     +25.2%       3057 ±  4%  perf-stat.i.cycles-between-cache-misses
      0.06 ±  5%      -0.0        0.02 ±  9%  perf-stat.i.dTLB-load-miss-rate%
   6599491 ±  5%     -44.4%    3668259 ±  7%  perf-stat.i.dTLB-load-misses
 1.092e+10          +139.7%  2.618e+10 ±  2%  perf-stat.i.dTLB-loads
      0.03 ±  2%      -0.0        0.00 ±  8%  perf-stat.i.dTLB-store-miss-rate%
   2188293 ±  3%     -75.8%     528948 ±  6%  perf-stat.i.dTLB-store-misses
 6.711e+09          +133.3%  1.566e+10        perf-stat.i.dTLB-stores
     96.13            +2.2       98.35        perf-stat.i.iTLB-load-miss-rate%
  46313820           +94.1%   89917841        perf-stat.i.iTLB-load-misses
   1862878 ±  9%     -25.2%    1393685 ±  7%  perf-stat.i.iTLB-loads
 3.826e+10          +143.7%  9.324e+10 ±  2%  perf-stat.i.instructions
    830.21           +23.3%       1023        perf-stat.i.instructions-per-iTLB-miss
      0.08          +141.3%       0.20 ±  2%  perf-stat.i.ipc
      0.51 ±  5%     -85.5%       0.07 ±  3%  perf-stat.i.metric.K/sec
    179.45          +141.2%     432.87        perf-stat.i.metric.M/sec
     82.77           +12.6       95.35        perf-stat.i.node-load-miss-rate%
  34027931 ±  3%    +141.1%   82055789 ±  4%  perf-stat.i.node-load-misses
   6688784 ±  4%     -40.8%    3963085 ±  3%  perf-stat.i.node-loads
     87.92           +10.3       98.23        perf-stat.i.node-store-miss-rate%
  27978556 ±  3%     +18.3%   33091342 ±  3%  perf-stat.i.node-store-misses
   3607145 ±  5%     -85.8%     512541 ±  4%  perf-stat.i.node-stores
     16.58 ±  2%     -18.1%      13.59        perf-stat.overall.MPKI
      1.18            +0.1        1.32        perf-stat.overall.branch-miss-rate%
     29.92 ±  2%     -16.4       13.51 ±  4%  perf-stat.overall.cache-miss-rate%
     12.20           -58.6%       5.05 ±  2%  perf-stat.overall.cpi
      2464 ±  4%     +11.7%       2754 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.06 ±  5%      -0.0        0.01 ± 10%  perf-stat.overall.dTLB-load-miss-rate%
      0.03 ±  3%      -0.0        0.00 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
     96.94            +1.5       98.48        perf-stat.overall.iTLB-load-miss-rate%
    829.34           +24.7%       1034        perf-stat.overall.instructions-per-iTLB-miss
      0.08          +141.8%       0.20 ±  2%  perf-stat.overall.ipc
     81.73           +13.6       95.35        perf-stat.overall.node-load-miss-rate%
     87.05           +11.3       98.38        perf-stat.overall.node-store-miss-rate%
  7.65e+09          +147.5%  1.893e+10 ±  2%  perf-stat.ps.branch-instructions
  90421083          +176.5%    2.5e+08        perf-stat.ps.branch-misses
  1.91e+08 ±  5%     -11.2%  1.696e+08 ±  3%  perf-stat.ps.cache-misses
 6.379e+08 ±  2%     +96.9%  1.256e+09 ±  2%  perf-stat.ps.cache-references
   1399868 ±  7%    +635.2%   10291374 ±  2%  perf-stat.ps.context-switches
     90581 ±  3%    +894.8%     901133 ±  4%  perf-stat.ps.cpu-migrations
   6425070 ±  5%     -42.1%    3719389 ±  7%  perf-stat.ps.dTLB-load-misses
 1.098e+10          +136.5%  2.597e+10 ±  2%  perf-stat.ps.dTLB-loads
   2100206 ±  3%     -73.8%     549733 ±  7%  perf-stat.ps.dTLB-store-misses
 6.747e+09          +130.3%  1.554e+10 ±  2%  perf-stat.ps.dTLB-stores
  46396702           +92.7%   89397839        perf-stat.ps.iTLB-load-misses
 3.848e+10          +140.3%  9.245e+10 ±  2%  perf-stat.ps.instructions
  31880617 ±  3%    +154.3%   81059404 ±  4%  perf-stat.ps.node-load-misses
   7131897 ±  5%     -44.6%    3951501 ±  3%  perf-stat.ps.node-loads
  26285797 ±  2%     +25.0%   32847276 ±  3%  perf-stat.ps.node-store-misses
   3915250 ±  6%     -86.2%     538400 ±  4%  perf-stat.ps.node-stores
 5.065e+13           +70.1%  8.613e+13        perf-stat.total.instructions
    267.16 ± 61%    -100.0%       0.00 ± 80%  perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    167.08 ± 23%    -100.0%       0.02 ±173%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     13.40 ± 25%     -96.4%       0.48 ± 13%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      9.09 ± 63%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.__alloc_pages_nodemask.allocate_slab.___slab_alloc
     33.47 ± 17%     -99.2%       0.27 ± 98%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.__kmalloc_node_track_caller.__kmalloc_reserve.isra
     45.86 ± 30%    -100.0%       0.01 ± 62%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
     26.92 ± 40%     -99.9%       0.03 ±158%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
     66.58 ± 52%     -94.2%       3.89 ±173%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg
      2.71 ±102%     -99.9%       0.00 ±173%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.01 ± 21%     -52.5%       0.01 ± 58%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.29 ±167%     -97.6%       0.01        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.04 ± 73%    +326.7%       0.18 ± 47%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
    262.60 ±164%    -100.0%       0.01 ± 59%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      4444 ± 85%    -100.0%       0.01 ± 82%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      8007 ± 38%    -100.0%       0.59 ±172%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      8241 ± 23%     -71.6%       2342 ±173%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     35.21 ± 44%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.__alloc_pages_nodemask.allocate_slab.___slab_alloc
     14945 ± 33%     -64.5%       5311 ±101%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.__kmalloc_node_track_caller.__kmalloc_reserve.isra
      9248 ± 12%     -99.9%      13.69 ± 71%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
      5533 ± 28%     -99.8%      13.74 ±155%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
      1599 ±161%     -98.2%      28.15 ± 85%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      6536 ± 49%     -89.2%     704.97 ±173%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg
      9.55 ±108%    -100.0%       0.00 ±173%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ± 16%     -60.0%       0.01 ± 63%  perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
     16049 ± 32%     -71.3%       4611 ±109%  perf-sched.sch_delay.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     67.60 ±172%    -100.0%       0.02 ±  7%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     15.93 ± 22%     -32.3%      10.78 ±  8%  perf-sched.total_sch_delay.average.ms
   1327113 ±  2%     +17.3%    1556514        perf-sched.total_wait_and_delay.count.ms
    781.56 ± 17%     -94.7%      41.26 ±105%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     55.19 ± 25%     -59.5%      22.38 ± 15%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    140.67 ± 19%     -63.2%      51.83 ± 21%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.__kmalloc_node_track_caller.__kmalloc_reserve.isra
    210.24 ± 35%     -72.9%      57.03 ±101%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
    180.93 ± 43%     -71.9%      50.92 ±173%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg
     97.48 ±100%    +761.5%     839.86 ± 34%  perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      1966 ± 53%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      2273 ± 52%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    125.44 ± 35%    +356.0%     571.96 ± 38%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1671 ± 14%     -39.9%       1004 ± 11%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      2041 ± 13%     -70.8%     596.67 ± 63%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
    165.50 ±  7%     -92.4%      12.50 ±100%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    350168 ±  3%     +69.1%     592290        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
     37758 ±  2%     -43.2%      21430 ±  7%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.__kmalloc_node_track_caller.__kmalloc_reserve.isra
      1470           -59.3%     598.25 ±100%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
    969.25           -70.4%     287.00 ±100%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
    209.50 ±  3%     -78.4%      45.25 ±173%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg
      3.50 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      6.75 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
    168.75 ± 21%     -76.0%      40.50 ± 30%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
     28959 ± 13%     -99.4%     188.00 ± 51%  perf-sched.wait_and_delay.count.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
    260.75 ±  9%     -36.1%     166.50        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
    179.25 ±  8%     -86.6%      24.00 ± 71%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
     16646 ± 33%     -93.9%       1020 ±106%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     30036 ± 33%     -30.0%      21037 ± 10%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__kmalloc_node_track_caller.__kmalloc_reserve.isra
     16333 ± 30%     -78.6%       3491 ±173%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg
    856.76 ±143%    +704.7%       6893 ± 66%  perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      6765 ± 56%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     11087 ± 44%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
     32157 ± 31%     -68.9%       9991 ± 94%  perf-sched.wait_and_delay.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
    145.65 ±169%    -100.0%       0.03 ± 48%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    104.57 ± 92%    -100.0%       0.04 ±100%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    614.48 ± 20%     -92.7%      44.98 ± 88%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     32.52 ± 29%    +135.2%      76.50 ± 30%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     41.78 ± 25%     -47.6%      21.90 ± 15%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    266.86 ±147%    -100.0%       0.00 ±173%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__alloc_pages_nodemask.allocate_slab.___slab_alloc
      0.18 ± 32%     -98.2%       0.00 ±173%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc
      0.51 ± 79%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__get_user_pages.__get_user_pages_remote.get_arg_page
    107.20 ± 20%     -51.9%      51.56 ± 21%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__kmalloc_node_track_caller.__kmalloc_reserve.isra
      0.19 ±132%     -93.2%       0.01 ±129%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__split_huge_pud.__handle_mm_fault.handle_mm_fault
    164.38 ± 38%     -56.3%      71.84 ± 60%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
      0.15 ± 60%     -89.2%       0.02 ±119%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write.__anon_vma_prepare.do_anonymous_page
      0.20 ± 57%     -98.4%       0.00 ±173%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write.__anon_vma_prepare.do_fault
      0.22 ± 33%     -77.5%       0.05 ±139%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write.vma_link.mmap_region
      0.79 ±104%     -98.4%       0.01 ±173%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write_killable.vm_mmap_pgoff.elf_map
      0.09 ± 66%     -76.5%       0.02 ±173%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_anonymous_page
      2.41 ± 96%     -88.5%       0.28 ±169%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.security_file_alloc.__alloc_file
      0.11 ± 78%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.vm_area_alloc.__install_special_mapping
      0.25 ± 70%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.vm_area_alloc.alloc_bprm
      1.96 ±101%     -97.1%       0.06 ± 80%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region
     56.24 ± 73%     -97.2%       1.60 ± 61%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      0.21 ± 87%     -76.3%       0.05 ± 99%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.wp_page_copy.__handle_mm_fault.handle_mm_fault
     71.97 ± 74%    +647.1%     537.69 ± 14%  perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      1963 ± 53%    -100.0%       0.04 ±103%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      2273 ± 52%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
     67.55 ± 33%    +481.0%     392.45 ± 32%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1671 ± 14%     -39.9%       1004 ± 11%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      2041 ± 13%     -70.7%     598.01 ± 62%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
      2173 ±170%    -100.0%       0.05 ± 65%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1121 ± 96%    -100.0%       0.12 ±102%  perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     12538 ± 31%     -91.1%       1120 ± 88%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      1543 ±156%    -100.0%       0.00 ±173%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__alloc_pages_nodemask.allocate_slab.___slab_alloc
      0.45 ± 56%     -99.3%       0.00 ±173%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc
      0.95 ± 68%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__get_user_pages.__get_user_pages_remote.get_arg_page
      0.33 ±149%     -96.1%       0.01 ±129%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__split_huge_pud.__handle_mm_fault.handle_mm_fault
      0.40 ± 96%     -92.5%       0.03 ±142%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.down_write.__anon_vma_prepare.do_anonymous_page
      0.32 ± 33%     -99.0%       0.00 ±173%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.down_write.__anon_vma_prepare.do_fault
      0.81 ± 38%     -76.5%       0.19 ±163%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.down_write.vma_link.mmap_region
      0.43 ± 24%     -70.5%       0.13 ±141%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.down_write_killable.setup_arg_pages.load_elf_binary
      5.10 ±158%     -99.8%       0.01 ±173%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.down_write_killable.vm_mmap_pgoff.elf_map
    179.23 ± 24%     -82.3%      31.71 ±128%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      0.22 ± 83%     -75.4%       0.05 ±173%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_anonymous_page
     40.72 ±125%     -91.8%       3.35 ±172%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.security_file_alloc.__alloc_file
      0.11 ± 78%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.vm_area_alloc.__install_special_mapping
      0.28 ± 58%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.vm_area_alloc.alloc_bprm
     11.44 ± 98%     -98.7%       0.15 ±112%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region
      4012 ± 63%     -99.3%      28.71 ± 83%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      0.70 ± 74%     -87.6%       0.09 ±106%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.wp_page_copy.__handle_mm_fault.handle_mm_fault
    640.25 ±106%    +521.6%       3979 ± 53%  perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      6765 ± 56%    -100.0%       0.09 ±100%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     11087 ± 44%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
     16209 ± 31%     -65.9%       5524 ± 79%  perf-sched.wait_time.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
    177132 ± 14%     -19.4%     142720 ±  5%  softirqs.CPU0.RCU
     28719 ± 11%     -44.4%      15962 ±  5%  softirqs.CPU0.SCHED
    179239 ± 14%     -19.1%     145006 ±  4%  softirqs.CPU1.RCU
     26049 ± 13%     -43.4%      14751 ± 12%  softirqs.CPU1.SCHED
    179205 ± 14%     -19.3%     144679 ±  4%  softirqs.CPU10.RCU
     26093 ± 12%     -45.9%      14129 ± 15%  softirqs.CPU10.SCHED
    185731 ± 14%     -17.7%     152796 ±  5%  softirqs.CPU100.RCU
     29921 ±  6%     -57.3%      12784 ± 20%  softirqs.CPU100.SCHED
    187257 ± 14%     -18.1%     153408 ±  4%  softirqs.CPU101.RCU
     29978 ±  4%     -58.2%      12538 ± 19%  softirqs.CPU101.SCHED
    187195 ± 14%     -17.7%     154083 ±  4%  softirqs.CPU102.RCU
     30458 ±  4%     -58.4%      12680 ± 19%  softirqs.CPU102.SCHED
    186955 ± 14%     -21.7%     146397 ± 11%  softirqs.CPU103.RCU
     30447 ±  4%     -57.7%      12870 ± 23%  softirqs.CPU103.SCHED
    185876 ± 14%     -18.3%     151804 ±  4%  softirqs.CPU104.RCU
     30604 ±  4%     -58.7%      12654 ± 19%  softirqs.CPU104.SCHED
    186935 ± 14%     -18.3%     152759 ±  4%  softirqs.CPU105.RCU
     30108 ±  6%     -57.7%      12727 ± 22%  softirqs.CPU105.SCHED
    187644 ± 14%     -17.5%     154734 ±  5%  softirqs.CPU106.RCU
     30453 ±  6%     -58.8%      12537 ± 18%  softirqs.CPU106.SCHED
    188233 ± 13%     -17.6%     155096 ±  5%  softirqs.CPU107.RCU
     29734 ±  5%     -58.6%      12317 ± 15%  softirqs.CPU107.SCHED
    170427 ± 14%     -18.6%     138703 ±  6%  softirqs.CPU108.RCU
     28367 ± 12%     -49.6%      14287 ±  5%  softirqs.CPU108.SCHED
    172104 ± 13%     -19.5%     138548 ±  6%  softirqs.CPU109.RCU
     28657 ± 11%     -46.9%      15225 ±  5%  softirqs.CPU109.SCHED
    180624 ± 14%     -18.6%     147079 ±  3%  softirqs.CPU11.RCU
     26035 ± 12%     -45.5%      14188 ± 12%  softirqs.CPU11.SCHED
    171891 ± 13%     -19.1%     139141 ±  6%  softirqs.CPU110.RCU
     29132 ± 12%     -47.2%      15393 ±  7%  softirqs.CPU110.SCHED
    172220 ± 13%     -19.0%     139503 ±  6%  softirqs.CPU111.RCU
     28568 ± 11%     -44.6%      15817 ±  3%  softirqs.CPU111.SCHED
    184054 ± 11%     -15.3%     155862 ±  6%  softirqs.CPU112.RCU
     28850 ± 11%     -47.3%      15197 ±  3%  softirqs.CPU112.SCHED
    185966 ± 11%     -13.9%     160107 ±  6%  softirqs.CPU113.RCU
     28961 ± 13%     -46.2%      15572 ±  5%  softirqs.CPU113.SCHED
    185695 ± 11%     -15.9%     156146 ±  7%  softirqs.CPU114.RCU
     29351 ± 15%     -48.2%      15198 ± 12%  softirqs.CPU114.SCHED
    185141 ± 11%     -14.7%     157862 ±  5%  softirqs.CPU115.RCU
     28897 ± 11%     -44.6%      16019 ± 10%  softirqs.CPU115.SCHED
    185476 ± 11%     -14.7%     158185 ±  6%  softirqs.CPU116.RCU
     29013 ± 11%     -47.3%      15296 ± 10%  softirqs.CPU116.SCHED
    186091 ± 11%     -14.3%     159414 ±  6%  softirqs.CPU117.RCU
     29035 ± 11%     -46.2%      15630 ±  5%  softirqs.CPU117.SCHED
     29187 ± 11%     -46.0%      15769 ±  7%  softirqs.CPU118.SCHED
    185601 ± 11%     -14.8%     158203 ±  5%  softirqs.CPU119.RCU
     28774 ± 12%     -48.3%      14882 ±  9%  softirqs.CPU119.SCHED
    176896 ± 11%     -17.7%     145584        softirqs.CPU12.RCU
     26941 ± 14%     -43.6%      15184 ± 17%  softirqs.CPU12.SCHED
      8300 ±120%     -87.1%       1074 ± 36%  softirqs.CPU12.TIMER
    185069 ± 11%     -15.0%     157391 ±  5%  softirqs.CPU120.RCU
     28586 ± 11%     -44.1%      15980 ±  4%  softirqs.CPU120.SCHED
    183879 ± 11%     -14.9%     156433 ±  5%  softirqs.CPU121.RCU
     28973 ± 11%     -44.2%      16171 ±  5%  softirqs.CPU121.SCHED
    185885 ± 11%     -14.7%     158619 ±  5%  softirqs.CPU122.RCU
     29016 ± 11%     -47.8%      15160 ±  6%  softirqs.CPU122.SCHED
    185820 ± 11%     -15.1%     157764 ±  5%  softirqs.CPU123.RCU
     29106 ± 10%     -49.3%      14755 ±  5%  softirqs.CPU123.SCHED
    185355 ± 11%     -14.9%     157656 ±  5%  softirqs.CPU124.RCU
     28433 ± 10%     -46.2%      15296 ±  6%  softirqs.CPU124.SCHED
    184495 ± 11%     -15.3%     156217 ±  5%  softirqs.CPU125.RCU
     29562 ± 10%     -47.7%      15454 ±  7%  softirqs.CPU125.SCHED
    162831 ± 14%     -22.6%     126036 ±  5%  softirqs.CPU126.RCU
     27983 ± 12%     -45.6%      15218 ±  3%  softirqs.CPU126.SCHED
    163641 ± 15%     -22.5%     126890 ±  4%  softirqs.CPU127.RCU
     28238 ± 11%     -41.6%      16490 ±  4%  softirqs.CPU127.SCHED
    175578 ± 16%     -18.2%     143625 ±  4%  softirqs.CPU128.RCU
     28422 ± 12%     -42.6%      16327 ±  2%  softirqs.CPU128.SCHED
    175330 ± 16%     -18.3%     143283 ±  5%  softirqs.CPU129.RCU
     28219 ± 11%     -43.4%      15964        softirqs.CPU129.SCHED
    181606 ± 14%     -17.5%     149780 ±  4%  softirqs.CPU13.RCU
     25966 ± 16%     -43.6%      14655 ± 15%  softirqs.CPU13.SCHED
    174309 ± 15%     -18.0%     142865 ±  4%  softirqs.CPU130.RCU
     27919 ± 13%     -39.4%      16926 ±  3%  softirqs.CPU130.SCHED
    175944 ± 16%     -18.0%     144359 ±  4%  softirqs.CPU131.RCU
     28807 ± 12%     -44.6%      15955 ±  8%  softirqs.CPU131.SCHED
    175957 ± 15%     -18.3%     143702 ±  4%  softirqs.CPU132.RCU
     27781 ± 12%     -40.4%      16566 ±  3%  softirqs.CPU132.SCHED
    175102 ± 15%     -18.1%     143330 ±  4%  softirqs.CPU133.RCU
     28551 ± 12%     -42.6%      16392 ±  5%  softirqs.CPU133.SCHED
    174439 ± 15%     -18.4%     142258 ±  5%  softirqs.CPU134.RCU
     27947 ± 13%     -41.7%      16293 ±  4%  softirqs.CPU134.SCHED
    175086 ± 16%     -17.4%     144665 ±  4%  softirqs.CPU135.RCU
     28336 ± 13%     -41.8%      16486 ±  4%  softirqs.CPU135.SCHED
    175640 ± 15%     -17.8%     144423 ±  4%  softirqs.CPU136.RCU
     28047 ± 12%     -40.2%      16761        softirqs.CPU136.SCHED
    175088 ± 15%     -18.2%     143220 ±  4%  softirqs.CPU137.RCU
     28025 ± 13%     -42.0%      16241 ±  9%  softirqs.CPU137.SCHED
    174745 ± 15%     -18.4%     142566 ±  4%  softirqs.CPU138.RCU
     28137 ± 12%     -38.7%      17238 ±  7%  softirqs.CPU138.SCHED
    175002 ± 15%     -17.4%     144527 ±  4%  softirqs.CPU139.RCU
     27781 ± 12%     -37.7%      17306 ±  7%  softirqs.CPU139.SCHED
    179978 ± 14%     -18.7%     146308 ±  4%  softirqs.CPU14.RCU
     25767 ± 12%     -43.4%      14574 ± 11%  softirqs.CPU14.SCHED
    176000 ± 15%     -17.8%     144716 ±  4%  softirqs.CPU140.RCU
     28013 ± 13%     -42.6%      16067 ±  4%  softirqs.CPU140.SCHED
    175311 ± 15%     -20.7%     139100 ±  9%  softirqs.CPU141.RCU
     28762 ± 10%     -41.6%      16796 ±  8%  softirqs.CPU141.SCHED
    174834 ± 15%     -18.3%     142791 ±  4%  softirqs.CPU142.RCU
     28189 ± 13%     -41.3%      16558 ±  4%  softirqs.CPU142.SCHED
    175073 ± 15%     -17.6%     144177 ±  5%  softirqs.CPU143.RCU
     28351 ± 13%     -36.0%      18135 ±  6%  softirqs.CPU143.SCHED
    180119 ± 14%     -18.7%     146416 ±  4%  softirqs.CPU15.RCU
     25693 ± 14%     -48.0%      13371 ± 10%  softirqs.CPU15.SCHED
    163622 ± 14%     -22.3%     127130 ±  4%  softirqs.CPU16.RCU
     25785 ± 11%     -45.1%      14167 ± 13%  softirqs.CPU16.SCHED
    163358 ± 14%     -22.3%     126921 ±  4%  softirqs.CPU17.RCU
     25737 ± 12%     -42.2%      14878 ± 16%  softirqs.CPU17.SCHED
    188067 ± 14%     -19.9%     150707 ±  5%  softirqs.CPU18.RCU
     29719 ±  5%     -58.3%      12378 ± 13%  softirqs.CPU18.SCHED
    186013 ± 13%     -19.5%     149832 ±  5%  softirqs.CPU19.RCU
     29958 ±  5%     -59.6%      12113 ± 12%  softirqs.CPU19.SCHED
    180540 ± 14%     -18.8%     146598 ±  4%  softirqs.CPU2.RCU
     26120 ± 13%     -47.4%      13745 ± 11%  softirqs.CPU2.SCHED
    185904 ± 14%     -19.4%     149825 ±  4%  softirqs.CPU20.RCU
     30088 ±  5%     -56.9%      12960 ± 24%  softirqs.CPU20.SCHED
    184534 ± 14%     -19.4%     148735 ±  5%  softirqs.CPU21.RCU
     29880 ±  6%     -57.9%      12592 ± 18%  softirqs.CPU21.SCHED
    185131 ± 14%     -20.0%     148055 ±  5%  softirqs.CPU22.RCU
     30088 ±  3%     -57.5%      12798 ± 20%  softirqs.CPU22.SCHED
    186267 ± 14%     -19.2%     150471 ±  4%  softirqs.CPU23.RCU
     29965 ±  6%     -58.4%      12466 ± 17%  softirqs.CPU23.SCHED
    185845 ± 14%     -19.3%     150035 ±  5%  softirqs.CPU24.RCU
     30104 ±  5%     -58.6%      12468 ± 17%  softirqs.CPU24.SCHED
    185365 ± 14%     -19.5%     149138 ±  4%  softirqs.CPU25.RCU
     29595 ±  2%     -57.1%      12689 ± 20%  softirqs.CPU25.SCHED
    180270 ± 10%     -17.8%     148219 ±  5%  softirqs.CPU26.RCU
     30090 ±  5%     -57.9%      12674 ± 19%  softirqs.CPU26.SCHED
    186037 ± 14%     -19.2%     150343 ±  5%  softirqs.CPU27.RCU
     30357 ±  3%     -59.8%      12193 ± 13%  softirqs.CPU27.SCHED
    185932 ± 14%     -19.3%     150046 ±  5%  softirqs.CPU28.RCU
     30387 ±  5%     -58.2%      12689 ± 19%  softirqs.CPU28.SCHED
    185591 ± 14%     -20.0%     148507 ±  5%  softirqs.CPU29.RCU
     29912 ±  5%     -57.5%      12709 ± 18%  softirqs.CPU29.SCHED
    180779 ± 14%     -18.3%     147750 ±  4%  softirqs.CPU3.RCU
     25411 ± 14%     -43.2%      14429 ± 12%  softirqs.CPU3.SCHED
    183707 ± 14%     -19.6%     147663 ±  4%  softirqs.CPU30.RCU
     30005 ±  6%     -57.8%      12647 ± 18%  softirqs.CPU30.SCHED
    184172 ± 13%     -21.4%     144846 ±  6%  softirqs.CPU31.RCU
     29903 ±  5%     -56.2%      13084 ± 25%  softirqs.CPU31.SCHED
    171778 ± 14%     -21.6%     134609 ±  5%  softirqs.CPU32.RCU
     30037 ±  5%     -57.9%      12643 ± 18%  softirqs.CPU32.SCHED
    171057 ± 14%     -21.7%     133968 ±  5%  softirqs.CPU33.RCU
     30219 ±  5%     -58.2%      12624 ± 20%  softirqs.CPU33.SCHED
    171389 ± 15%     -21.4%     134774 ±  6%  softirqs.CPU34.RCU
     30516 ±  5%     -59.2%      12457 ± 14%  softirqs.CPU34.SCHED
    171494 ± 14%     -21.1%     135250 ±  5%  softirqs.CPU35.RCU
     30023 ±  4%     -59.1%      12289 ± 16%  softirqs.CPU35.SCHED
    181290 ± 13%     -14.9%     154258 ±  5%  softirqs.CPU36.RCU
     29053 ± 11%     -51.6%      14058 ±  2%  softirqs.CPU36.SCHED
    181334 ± 12%     -14.2%     155511 ±  5%  softirqs.CPU37.RCU
     28250 ± 11%     -43.2%      16054 ±  6%  softirqs.CPU37.SCHED
    181030 ± 12%     -14.1%     155541 ±  5%  softirqs.CPU38.RCU
     28707 ± 11%     -45.7%      15599 ±  7%  softirqs.CPU38.SCHED
    180274 ± 12%     -14.8%     153606 ±  5%  softirqs.CPU39.RCU
     28787 ± 13%     -46.1%      15507        softirqs.CPU39.SCHED
    181956 ± 14%     -18.4%     148545 ±  4%  softirqs.CPU4.RCU
     25391 ± 15%     -44.3%      14130 ± 13%  softirqs.CPU4.SCHED
    180286 ± 12%     -14.9%     153441 ±  5%  softirqs.CPU40.RCU
     28735 ± 12%     -47.0%      15223 ±  4%  softirqs.CPU40.SCHED
    181233 ± 12%     -13.8%     156159 ±  5%  softirqs.CPU41.RCU
     28492 ± 13%     -45.8%      15443 ±  4%  softirqs.CPU41.SCHED
    181415 ± 12%     -19.0%     146945 ± 13%  softirqs.CPU42.RCU
     29239 ± 11%     -48.2%      15138 ±  3%  softirqs.CPU42.SCHED
    181162 ± 12%     -14.8%     154386 ±  5%  softirqs.CPU43.RCU
     28380 ± 14%     -43.7%      15973 ±  5%  softirqs.CPU43.SCHED
    180827 ± 12%     -14.6%     154401 ±  5%  softirqs.CPU44.RCU
     28768 ± 13%     -47.8%      15031 ±  9%  softirqs.CPU44.SCHED
    181654 ± 12%     -14.0%     156187 ±  5%  softirqs.CPU45.RCU
     28695 ± 11%     -44.3%      15969 ±  5%  softirqs.CPU45.SCHED
    179648 ± 13%     -13.8%     154805 ±  6%  softirqs.CPU46.RCU
     30075 ± 13%     -50.8%      14806 ±  7%  softirqs.CPU46.SCHED
    181072 ± 12%     -14.6%     154578 ±  5%  softirqs.CPU47.RCU
     28651 ± 12%     -45.2%      15697 ±  7%  softirqs.CPU47.SCHED
    177712 ± 12%     -17.1%     147242 ±  5%  softirqs.CPU48.RCU
     28855 ± 13%     -45.5%      15740 ±  5%  softirqs.CPU48.SCHED
    178280 ± 12%     -16.9%     148081 ±  5%  softirqs.CPU49.RCU
     28817 ± 10%     -47.5%      15140 ±  3%  softirqs.CPU49.SCHED
    180504 ± 15%     -18.2%     147679 ±  3%  softirqs.CPU5.RCU
     25244 ± 14%     -41.9%      14668 ± 14%  softirqs.CPU5.SCHED
    178681 ± 12%     -17.0%     148336 ±  5%  softirqs.CPU50.RCU
     29000 ± 12%     -49.2%      14743 ±  5%  softirqs.CPU50.SCHED
    178360 ± 12%     -16.8%     148318 ±  5%  softirqs.CPU51.RCU
     28759 ± 11%     -48.3%      14880 ±  7%  softirqs.CPU51.SCHED
    177584 ± 12%     -17.3%     146831 ±  5%  softirqs.CPU52.RCU
     28735 ± 12%     -44.8%      15865 ±  7%  softirqs.CPU52.SCHED
    177986 ± 12%     -17.0%     147807 ±  5%  softirqs.CPU53.RCU
     28925 ± 12%     -42.3%      16695 ± 17%  softirqs.CPU53.SCHED
    177577 ± 15%     -17.9%     145728 ±  4%  softirqs.CPU54.RCU
     28475 ± 12%     -46.1%      15358 ±  8%  softirqs.CPU54.SCHED
    178287 ± 15%     -17.8%     146612 ±  4%  softirqs.CPU55.RCU
     28305 ± 10%     -41.9%      16458 ±  6%  softirqs.CPU55.SCHED
    176670 ± 15%     -17.5%     145691 ±  4%  softirqs.CPU56.RCU
     27920 ± 12%     -41.8%      16243 ±  5%  softirqs.CPU56.SCHED
    177389 ± 15%     -17.5%     146407 ±  5%  softirqs.CPU57.RCU
     28222 ± 12%     -42.9%      16124        softirqs.CPU57.SCHED
    177331 ± 15%     -17.5%     146382 ±  5%  softirqs.CPU58.RCU
     28374 ± 12%     -42.3%      16382 ±  3%  softirqs.CPU58.SCHED
    177412 ± 15%     -17.5%     146355 ±  5%  softirqs.CPU59.RCU
     28230 ± 10%     -41.8%      16440 ±  8%  softirqs.CPU59.SCHED
    179682 ± 14%     -18.6%     146276 ±  4%  softirqs.CPU6.RCU
     26134 ± 13%     -46.5%      13973 ± 15%  softirqs.CPU6.SCHED
    177023 ± 15%     -17.5%     145977 ±  5%  softirqs.CPU60.RCU
     28521 ± 13%     -42.3%      16448 ±  3%  softirqs.CPU60.SCHED
    176681 ± 15%     -17.4%     145896 ±  5%  softirqs.CPU61.RCU
     27761 ± 11%     -42.8%      15891 ±  3%  softirqs.CPU61.SCHED
    176878 ± 15%     -17.5%     146004 ±  4%  softirqs.CPU62.RCU
     28559 ± 11%     -41.7%      16656 ±  5%  softirqs.CPU62.SCHED
    177058 ± 15%     -17.0%     146874 ±  5%  softirqs.CPU63.RCU
     28151 ± 11%     -42.4%      16212 ±  3%  softirqs.CPU63.SCHED
    174751 ± 15%     -17.2%     144699 ±  4%  softirqs.CPU64.RCU
     28262 ± 12%     -43.0%      16100 ±  4%  softirqs.CPU64.SCHED
    175144 ± 15%     -17.2%     145013 ±  4%  softirqs.CPU65.RCU
     28649 ± 12%     -44.8%      15808 ±  2%  softirqs.CPU65.SCHED
    175283 ± 15%     -17.2%     145072 ±  5%  softirqs.CPU66.RCU
     28249 ± 12%     -40.7%      16744 ±  4%  softirqs.CPU66.SCHED
    175245 ± 15%     -17.0%     145418 ±  4%  softirqs.CPU67.RCU
     28533 ± 13%     -41.5%      16691 ±  7%  softirqs.CPU67.SCHED
    174723 ± 15%     -17.2%     144749 ±  4%  softirqs.CPU68.RCU
     28528 ± 11%     -43.6%      16087        softirqs.CPU68.SCHED
    175212 ± 15%     -17.7%     144264 ±  5%  softirqs.CPU69.RCU
     28203 ± 13%     -42.4%      16247 ±  7%  softirqs.CPU69.SCHED
    180615 ± 14%     -18.5%     147211 ±  4%  softirqs.CPU7.RCU
     25771 ± 14%     -45.3%      14095 ± 13%  softirqs.CPU7.SCHED
    175210 ± 15%     -17.2%     145132 ±  4%  softirqs.CPU70.RCU
     27564 ± 11%     -41.4%      16154 ±  2%  softirqs.CPU70.SCHED
    175376 ± 15%     -17.0%     145643 ±  4%  softirqs.CPU71.RCU
     29266 ± 14%     -41.6%      17089 ±  9%  softirqs.CPU71.SCHED
    181207 ± 14%     -16.2%     151800 ±  4%  softirqs.CPU72.RCU
     25921 ± 14%     -47.6%      13592 ± 12%  softirqs.CPU72.SCHED
    183308 ± 13%     -16.2%     153694 ±  3%  softirqs.CPU73.RCU
     26080 ± 11%     -44.0%      14595 ± 14%  softirqs.CPU73.SCHED
    182677 ± 14%     -15.4%     154580 ±  4%  softirqs.CPU74.RCU
     25553 ± 12%     -42.2%      14760 ± 15%  softirqs.CPU74.SCHED
    183155 ± 14%     -15.4%     154904 ±  3%  softirqs.CPU75.RCU
     25521 ± 12%     -44.8%      14096 ± 13%  softirqs.CPU75.SCHED
    183730 ± 13%     -15.0%     156196 ±  3%  softirqs.CPU76.RCU
     26061 ± 11%     -43.6%      14692 ± 13%  softirqs.CPU76.SCHED
    182416 ± 14%     -15.6%     154006 ±  3%  softirqs.CPU77.RCU
     25393 ± 12%     -41.7%      14809 ± 15%  softirqs.CPU77.SCHED
    182222 ± 14%     -15.1%     154646 ±  3%  softirqs.CPU78.RCU
     25937 ± 13%     -44.4%      14419 ± 17%  softirqs.CPU78.SCHED
    183080 ± 14%     -15.3%     155092 ±  3%  softirqs.CPU79.RCU
     25554 ± 12%     -42.8%      14608 ± 15%  softirqs.CPU79.SCHED
    181316 ± 14%     -18.2%     148389 ±  4%  softirqs.CPU8.RCU
     25837 ± 10%     -46.0%      13958 ± 11%  softirqs.CPU8.SCHED
    185007 ± 14%     -15.5%     156261 ±  4%  softirqs.CPU80.RCU
     26157 ± 11%     -45.5%      14252 ± 13%  softirqs.CPU80.SCHED
    184321 ± 14%     -15.8%     155195 ±  4%  softirqs.CPU81.RCU
     25773 ± 14%     -44.2%      14380 ± 12%  softirqs.CPU81.SCHED
    184135 ± 15%     -15.6%     155363 ±  3%  softirqs.CPU82.RCU
     25906 ± 14%     -44.4%      14400 ± 13%  softirqs.CPU82.SCHED
    184376 ± 14%     -15.0%     156683 ±  3%  softirqs.CPU83.RCU
     25995 ± 13%     -44.2%      14496 ± 14%  softirqs.CPU83.SCHED
    185380 ± 14%     -19.7%     148785 ±  6%  softirqs.CPU84.RCU
     25964 ± 13%     -43.3%      14726 ±  9%  softirqs.CPU84.SCHED
    184936 ± 14%     -14.9%     157350 ±  4%  softirqs.CPU85.RCU
     25850 ± 11%     -41.8%      15043 ± 16%  softirqs.CPU85.SCHED
    184914 ± 14%     -15.8%     155667 ±  4%  softirqs.CPU86.RCU
     25454 ± 14%     -43.0%      14513 ± 14%  softirqs.CPU86.SCHED
    184043 ± 14%     -16.3%     153968 ±  4%  softirqs.CPU87.RCU
     25884 ± 11%     -45.9%      14015 ± 14%  softirqs.CPU87.SCHED
    183877 ± 14%     -15.6%     155109 ±  4%  softirqs.CPU88.RCU
     25795 ± 12%     -43.4%      14599 ± 14%  softirqs.CPU88.SCHED
    185255 ± 14%     -15.0%     157552 ±  4%  softirqs.CPU89.RCU
     26084 ± 15%     -41.7%      15199 ± 16%  softirqs.CPU89.SCHED
    179588 ± 14%     -18.6%     146226 ±  4%  softirqs.CPU9.RCU
     25509 ± 17%     -42.9%      14571 ± 11%  softirqs.CPU9.SCHED
    178631 ± 14%     -19.1%     144510 ±  5%  softirqs.CPU90.RCU
     29996 ±  4%     -58.8%      12357 ± 16%  softirqs.CPU90.SCHED
    178675 ± 14%     -19.2%     144375 ±  5%  softirqs.CPU91.RCU
     29809 ±  6%     -59.8%      11995 ± 11%  softirqs.CPU91.SCHED
    177752 ± 14%     -19.6%     142997 ±  5%  softirqs.CPU92.RCU
     30877 ±  5%     -57.4%      13148 ± 24%  softirqs.CPU92.SCHED
    178443 ± 14%     -19.0%     144569 ±  5%  softirqs.CPU93.RCU
     30033 ±  5%     -58.2%      12545 ± 19%  softirqs.CPU93.SCHED
    178791 ± 14%     -19.4%     144044 ±  5%  softirqs.CPU94.RCU
     30651 ±  6%     -57.9%      12895 ± 23%  softirqs.CPU94.SCHED
    178749 ± 14%     -19.1%     144575 ±  5%  softirqs.CPU95.RCU
     30319 ±  4%     -58.8%      12492 ± 18%  softirqs.CPU95.SCHED
    187353 ± 14%     -17.7%     154176 ±  5%  softirqs.CPU96.RCU
     29911 ±  5%     -59.3%      12171 ± 15%  softirqs.CPU96.SCHED
    187309 ± 14%     -17.9%     153842 ±  4%  softirqs.CPU97.RCU
     30350 ±  5%     -57.1%      13022 ± 24%  softirqs.CPU97.SCHED
    185522 ± 12%     -17.3%     153452 ±  5%  softirqs.CPU98.RCU
     29864 ±  3%     -57.6%      12661 ± 21%  softirqs.CPU98.SCHED
    185411 ± 13%     -18.0%     152065 ±  5%  softirqs.CPU99.RCU
     29872 ±  3%     -58.8%      12302 ± 13%  softirqs.CPU99.SCHED
  25926379 ± 14%     -17.5%   21381106 ±  4%  softirqs.RCU
   4074069 ±  2%     -48.0%    2119394        softirqs.SCHED
    252087 ±  8%     -31.4%     173043 ±  4%  softirqs.TIMER
     47.16            -7.1       40.04        perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read
     47.22            -7.1       40.14        perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read.ksys_read
     47.42            -6.9       40.53        perf-profile.calltrace.cycles-pp.sock_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
     47.51            -6.8       40.76        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     17.75 ±  2%      -6.7       11.02 ±  3%  perf-profile.calltrace.cycles-pp.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
     47.80            -6.4       41.44        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     48.07            -6.2       41.91        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     48.12            -6.1       42.02        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      8.74 ±  5%      -5.8        2.98 ±  7%  perf-profile.calltrace.cycles-pp.refill_obj_stock.kfree.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
     48.48            -5.7       42.81        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
     22.27 ±  2%      -5.6       16.70 ±  2%  perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      5.55 ± 21%      -5.5        0.00        perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event
     20.96 ±  2%      -5.4       15.56 ±  2%  perf-profile.calltrace.cycles-pp.kfree.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     19.20 ±  2%      -5.2       13.98 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
     19.27 ±  2%      -5.2       14.06 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
     48.92            -5.1       43.78        perf-profile.calltrace.cycles-pp.__libc_read
      5.12 ± 21%      -5.1        0.00        perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow
      5.10 ± 21%      -5.1        0.00        perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
      7.18 ±  6%      -5.0        2.18 ±  6%  perf-profile.calltrace.cycles-pp.drain_obj_stock.refill_obj_stock.kfree.consume_skb.unix_stream_read_generic
      7.61 ±  5%      -4.7        2.90 ±  5%  perf-profile.calltrace.cycles-pp.refill_obj_stock.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     38.75            -4.6       34.12 ±  2%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     39.28            -4.6       34.70 ±  2%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
     38.79            -4.6       34.20 ±  2%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      5.14 ±  9%      -4.4        0.70 ± 68%  perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock
      5.96 ±  6%      -4.3        1.63 ±  7%  perf-profile.calltrace.cycles-pp.drain_obj_stock.refill_obj_stock.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg
      3.04 ±  8%      -2.4        0.63 ± 10%  perf-profile.calltrace.cycles-pp.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock.kfree.consume_skb
      2.92 ±  8%      -2.4        0.57 ± 10%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock.kfree
      2.54 ±  9%      -2.3        0.28 ±100%  perf-profile.calltrace.cycles-pp.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock.kmem_cache_free.unix_stream_read_generic
      2.48 ±  9%      -2.2        0.27 ±100%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock.kmem_cache_free
      1.98 ± 16%      -1.2        0.79 ± 21%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      2.12 ±  8%      -1.0        1.12 ±  7%  perf-profile.calltrace.cycles-pp.obj_cgroup_charge.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
      1.46 ± 16%      -1.0        0.47 ± 61%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.66 ±  9%      -1.0        0.69 ± 11%  perf-profile.calltrace.cycles-pp.__memcg_kmem_charge.obj_cgroup_charge.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb
      1.52 ± 10%      -0.9        0.66 ± 11%  perf-profile.calltrace.cycles-pp.page_counter_try_charge.__memcg_kmem_charge.obj_cgroup_charge.__kmalloc_node_track_caller.__kmalloc_reserve
      4.83            -0.8        4.02 ±  2%  perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
      2.14 ± 15%      -0.7        1.43 ± 13%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      0.89 ±  9%      -0.6        0.27 ±100%  perf-profile.calltrace.cycles-pp.__memcg_kmem_charge.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
      0.82 ±  9%      -0.6        0.26 ±100%  perf-profile.calltrace.cycles-pp.page_counter_try_charge.__memcg_kmem_charge.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb
      0.54 ±  3%      +0.1        0.64 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      0.53 ±  3%      +0.1        0.63 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      0.39 ± 57%      +0.2        0.61 ±  4%  perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic
      0.77 ±  2%      +0.3        1.04        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg
      0.81            +0.3        1.09        perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.79            +0.3        1.07        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     19.29            +0.5       19.77 ±  2%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      0.00            +0.6        0.58 ±  2%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.schedule_timeout
      0.00            +0.6        0.59 ±  3%  perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read
      0.00            +0.6        0.59        perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      2.52 ± 13%      +0.6        3.12 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.60        perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.14 ±173%      +0.6        0.75 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      2.57 ± 13%      +0.6        3.19 ±  4%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      2.45 ± 12%      +0.6        3.07 ±  6%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.unix_stream_read_generic
      0.00            +0.6        0.64        perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +0.7        0.67 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read
      0.00            +0.7        0.71 ±  3%  perf-profile.calltrace.cycles-pp.refill_obj_stock.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
      0.00            +0.7        0.72 ±  3%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read
      0.00            +0.7        0.72 ±  4%  perf-profile.calltrace.cycles-pp.prepare_to_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.00            +0.8        0.75 ±  2%  perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      0.00            +1.2        1.17 ±  2%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.unix_stream_read_generic
      0.00            +1.2        1.24 ±  3%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      1.07 ± 15%      +1.4        2.46 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      1.11 ± 15%      +1.5        2.60 ±  5%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_write
     46.94            +1.7       48.64        perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write
      0.00            +1.7        1.73 ± 13%  perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
      1.27 ± 14%      +1.8        3.05 ±  5%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_write
      1.30 ± 14%      +1.8        3.11 ±  5%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_write
      8.27            +1.9       10.13 ±  2%  perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
     47.13            +1.9       49.03        perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write.ksys_write
     47.24            +2.0       49.23        perf-profile.calltrace.cycles-pp.sock_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     47.46            +2.0       49.47        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     47.86            +2.3       50.16        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     48.33            +2.6       50.89        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     48.42            +2.6       51.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      4.47 ± 12%      +2.7        7.14 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg
      0.59 ±  4%      +2.7        3.32 ± 11%  perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      4.52 ± 12%      +2.7        7.26 ±  6%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      4.61 ± 12%      +2.9        7.46 ±  6%  perf-profile.calltrace.cycles-pp.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.79 ±  5%      +3.3        4.11 ± 10%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     49.78            +4.4       54.22        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
      5.84 ± 12%      +4.7       10.57 ±  6%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      5.87 ± 12%      +4.8       10.67 ±  6%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg
      5.92 ± 12%      +4.9       10.80 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg
     50.20            +5.1       55.28        perf-profile.calltrace.cycles-pp.__libc_write
      6.15 ± 12%      +5.2       11.30 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      6.37 ± 11%      +5.2       11.61 ±  6%  perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
     16.66 ±  5%      -9.9        6.78 ±  5%  perf-profile.children.cycles-pp.refill_obj_stock
     13.46 ±  6%      -9.3        4.12 ±  6%  perf-profile.children.cycles-pp.drain_obj_stock
      9.50 ±  8%      -8.4        1.14 ± 10%  perf-profile.children.cycles-pp.page_counter_uncharge
      8.81 ±  8%      -7.7        1.13 ± 10%  perf-profile.children.cycles-pp.page_counter_cancel
     47.18            -7.1       40.06        perf-profile.children.cycles-pp.unix_stream_read_generic
     47.22            -7.1       40.14        perf-profile.children.cycles-pp.unix_stream_recvmsg
     47.43            -6.9       40.53        perf-profile.children.cycles-pp.sock_read_iter
     47.52            -6.7       40.77        perf-profile.children.cycles-pp.new_sync_read
     17.81 ±  2%      -6.7       11.07 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
     47.82            -6.4       41.45        perf-profile.children.cycles-pp.vfs_read
     48.09            -6.1       41.94        perf-profile.children.cycles-pp.ksys_read
      6.73 ± 17%      -6.1        0.62 ±108%  perf-profile.children.cycles-pp.perf_tp_event
      6.48 ± 17%      -6.1        0.38 ±173%  perf-profile.children.cycles-pp.perf_swevent_overflow
      6.42 ± 17%      -6.0        0.38 ±173%  perf-profile.children.cycles-pp.__perf_event_overflow
      6.37 ± 17%      -6.0        0.38 ±173%  perf-profile.children.cycles-pp.perf_event_output_forward
      6.17 ± 17%      -5.8        0.37 ±173%  perf-profile.children.cycles-pp.perf_prepare_sample
     22.27 ±  2%      -5.6       16.70 ±  2%  perf-profile.children.cycles-pp.consume_skb
     21.02 ±  2%      -5.4       15.63 ±  2%  perf-profile.children.cycles-pp.kfree
      5.69 ± 17%      -5.3        0.34 ±173%  perf-profile.children.cycles-pp.perf_callchain
      5.66 ± 17%      -5.3        0.34 ±173%  perf-profile.children.cycles-pp.get_perf_callchain
     19.26 ±  2%      -5.2       14.05 ±  4%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
     19.27 ±  2%      -5.2       14.06 ±  4%  perf-profile.children.cycles-pp.__kmalloc_reserve
     48.98            -5.1       43.88        perf-profile.children.cycles-pp.__libc_read
      5.11 ± 17%      -4.8        0.31 ±173%  perf-profile.children.cycles-pp.perf_callchain_kernel
     38.75            -4.6       34.12 ±  2%  perf-profile.children.cycles-pp.__alloc_skb
     39.28            -4.6       34.70 ±  2%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
     38.79            -4.6       34.20 ±  2%  perf-profile.children.cycles-pp.alloc_skb_with_frags
      5.64 ±  8%      -4.4        1.21 ±  9%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge
     96.60            -3.5       93.06        perf-profile.children.cycles-pp.do_syscall_64
      3.58 ± 16%      -3.2        0.38 ± 88%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      2.92 ±  9%      -2.5        0.47 ±  7%  perf-profile.children.cycles-pp.propagate_protected_usage
      4.01 ± 15%      -1.7        2.35 ± 15%  perf-profile.children.cycles-pp.update_curr
      1.89 ± 18%      -1.6        0.31 ± 69%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      2.55 ±  9%      -1.4        1.18 ± 10%  perf-profile.children.cycles-pp.__memcg_kmem_charge
      2.51 ±  9%      -1.3        1.17 ± 10%  perf-profile.children.cycles-pp.page_counter_try_charge
     98.33            -1.3       97.07        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.47 ±  7%      -0.9        2.52 ±  3%  perf-profile.children.cycles-pp.obj_cgroup_charge
      1.49 ± 16%      -0.9        0.58 ± 23%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      1.05 ±  7%      -0.9        0.16 ± 37%  perf-profile.children.cycles-pp.__slab_alloc
      1.01 ±  8%      -0.9        0.15 ± 35%  perf-profile.children.cycles-pp.___slab_alloc
      1.09 ±  3%      -0.8        0.33 ± 13%  perf-profile.children.cycles-pp.__mod_memcg_state
      1.16 ±  3%      -0.7        0.42 ±  9%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      2.18 ± 14%      -0.7        1.46 ± 13%  perf-profile.children.cycles-pp.dequeue_entity
      1.06            -0.5        0.56 ±  2%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.94 ±  4%      -0.3        0.60 ±  2%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.41 ± 10%      -0.3        0.09        perf-profile.children.cycles-pp.task_tick_fair
      0.56 ±  7%      -0.3        0.24 ±  2%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.44 ±  9%      -0.3        0.13        perf-profile.children.cycles-pp.scheduler_tick
      0.48 ±  9%      -0.3        0.18        perf-profile.children.cycles-pp.tick_sched_handle
      0.50 ±  8%      -0.3        0.20        perf-profile.children.cycles-pp.tick_sched_timer
      0.47 ±  8%      -0.3        0.18 ±  2%  perf-profile.children.cycles-pp.update_process_times
      0.76 ±  6%      -0.3        0.51        perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.70 ±  6%      -0.2        0.45        perf-profile.children.cycles-pp.hrtimer_interrupt
      0.71 ±  6%      -0.2        0.46        perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.40 ±  7%      -0.1        0.33 ±  3%  perf-profile.children.cycles-pp.__slab_free
      0.22 ±  8%      -0.1        0.15 ±  3%  perf-profile.children.cycles-pp.mutex_unlock
      0.19 ±  3%      -0.0        0.16 ±  4%  perf-profile.children.cycles-pp.unix_write_space
      0.11 ±  7%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.skb_release_data
      0.06 ±  9%      +0.0        0.07        perf-profile.children.cycles-pp.__list_add_valid
      0.06 ± 11%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.start_thread
      0.07 ±  7%      +0.0        0.09        perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.14 ± 13%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.preempt_schedule_common
      0.10 ±  5%      +0.0        0.14 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.42 ±  3%      +0.0        0.47 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.__ksize
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.place_entity
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.apparmor_socket_sendmsg
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.rb_next
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.security_socket_getpeersec_dgram
      0.12 ±  3%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp.copyin
      0.05            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.aa_file_perm
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.__x64_sys_write
      0.05 ±  8%      +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.clockevents_program_event
      0.45 ±  3%      +0.1        0.51 ±  6%  perf-profile.children.cycles-pp.sock_wfree
      0.10            +0.1        0.16 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.05 ±  9%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.ktime_get
      0.09 ±  7%      +0.1        0.15 ±  3%  perf-profile.children.cycles-pp.rb_insert_color
      0.07 ±  7%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.__might_fault
      0.00            +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.remove_entity_load_avg
      0.22 ±  5%      +0.1        0.29 ±  2%  perf-profile.children.cycles-pp.fput_many
      0.19            +0.1        0.26 ±  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.11 ±  6%      +0.1        0.18 ±  3%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.33 ±  2%      +0.1        0.40 ±  4%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_from_task
      0.19 ±  2%      +0.1        0.27        perf-profile.children.cycles-pp.__entry_text_start
      0.12 ±  5%      +0.1        0.20 ±  2%  perf-profile.children.cycles-pp.sock_recvmsg
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.wakeup_preempt_entity
      0.00            +0.1        0.08        perf-profile.children.cycles-pp.cpumask_next
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.rcu_qs
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.10 ±  7%      +0.1        0.19 ±  6%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.09 ± 12%  perf-profile.children.cycles-pp.get_xsave_addr
      0.12 ±  8%      +0.1        0.21 ±  2%  perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
      0.10 ±  7%      +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
      0.11 ±  3%      +0.1        0.21 ±  2%  perf-profile.children.cycles-pp.__check_heap_object
      0.06            +0.1        0.15 ±  3%  perf-profile.children.cycles-pp.rcu_all_qs
      0.52 ±  3%      +0.1        0.61 ±  5%  perf-profile.children.cycles-pp.unix_destruct_scm
      0.04 ± 58%      +0.1        0.14 ± 18%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.06 ±  7%      +0.1        0.16 ±  2%  perf-profile.children.cycles-pp.wait_for_unix_gc
      0.10 ±  8%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.sched_clock
      0.53 ±  2%      +0.1        0.63 ±  5%  perf-profile.children.cycles-pp.skb_release_head_state
      0.12 ± 10%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.54 ±  3%      +0.1        0.64 ±  5%  perf-profile.children.cycles-pp.skb_release_all
      0.20 ±  4%      +0.1        0.31 ±  2%  perf-profile.children.cycles-pp.aa_sk_perm
      0.35 ±  2%      +0.1        0.46        perf-profile.children.cycles-pp.simple_copy_to_iter
      0.34            +0.1        0.45 ±  2%  perf-profile.children.cycles-pp.__pthread_disable_asynccancel
      0.00            +0.1        0.12 ±  5%  perf-profile.children.cycles-pp.finish_wait
      0.04 ± 59%      +0.1        0.17 ± 22%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.41 ±  2%      +0.1        0.53 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
      0.12 ± 14%      +0.1        0.25 ± 15%  perf-profile.children.cycles-pp.set_task_cpu
      0.18 ±  4%      +0.1        0.31        perf-profile.children.cycles-pp._copy_from_iter
      0.00            +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.23 ±  4%      +0.1        0.36 ±  7%  perf-profile.children.cycles-pp.skb_queue_tail
      0.16 ±  5%      +0.1        0.30        perf-profile.children.cycles-pp.fsnotify
      0.05            +0.1        0.19 ±  6%  perf-profile.children.cycles-pp._find_next_bit
      0.19 ± 16%      +0.1        0.34 ± 14%  perf-profile.children.cycles-pp.finish_task_switch
      0.20 ±  4%      +0.2        0.35 ±  3%  perf-profile.children.cycles-pp.common_file_perm
      0.00            +0.2        0.16 ±  5%  perf-profile.children.cycles-pp.resched_curr
      0.24 ±  7%      +0.2        0.40        perf-profile.children.cycles-pp._cond_resched
      0.00            +0.2        0.16 ± 21%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.00            +0.2        0.17 ±  3%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.04 ± 57%      +0.2        0.21 ±  8%  perf-profile.children.cycles-pp.set_next_buddy
      0.04 ± 58%      +0.2        0.22 ± 11%  perf-profile.children.cycles-pp.cpus_share_cache
      0.06 ±  9%      +0.2        0.24 ± 10%  perf-profile.children.cycles-pp.update_cfs_rq_h_load
      0.05 ±  8%      +0.2        0.24 ±  8%  perf-profile.children.cycles-pp.update_min_vruntime
      0.00            +0.2        0.19 ± 12%  perf-profile.children.cycles-pp.migrate_task_rq_fair
      0.15 ±  3%      +0.2        0.34 ±  3%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.00            +0.2        0.20 ±  2%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.08 ± 15%      +0.2        0.29 ±  3%  perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
      0.15 ±  3%      +0.2        0.35 ±  5%  perf-profile.children.cycles-pp.mutex_lock
      0.08 ±  5%      +0.2        0.30 ±  9%  perf-profile.children.cycles-pp.cpumask_next_wrap
      0.31 ±  6%      +0.2        0.53 ±  4%  perf-profile.children.cycles-pp.__fget_files
      0.27 ±  5%      +0.2        0.49 ±  7%  perf-profile.children.cycles-pp.update_cfs_group
      0.12 ±  3%      +0.2        0.35 ±  6%  perf-profile.children.cycles-pp.cpuacct_charge
      0.18 ±  2%      +0.2        0.41 ±  6%  perf-profile.children.cycles-pp.___might_sleep
      0.47            +0.2        0.70        perf-profile.children.cycles-pp.__check_object_size
      0.30 ±  2%      +0.2        0.53 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      0.00            +0.3        0.26        perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.33 ±  2%      +0.3        0.60        perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
      0.10 ±  9%      +0.3        0.36        perf-profile.children.cycles-pp.__enqueue_entity
      0.78 ±  2%      +0.3        1.04        perf-profile.children.cycles-pp.__skb_datagram_iter
      0.82            +0.3        1.09        perf-profile.children.cycles-pp.unix_stream_read_actor
      0.00            +0.3        0.28 ±  5%  perf-profile.children.cycles-pp.clear_buddies
      0.79            +0.3        1.07        perf-profile.children.cycles-pp.skb_copy_datagram_iter
      0.80 ±  2%      +0.3        1.13 ± 15%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      0.39 ±  4%      +0.3        0.71 ±  3%  perf-profile.children.cycles-pp.__fget_light
      0.08 ±  8%      +0.3        0.41 ±  2%  perf-profile.children.cycles-pp.rb_erase
      0.08 ±  6%      +0.3        0.41 ±  3%  perf-profile.children.cycles-pp.put_prev_entity
      0.41 ±  4%      +0.3        0.75 ±  3%  perf-profile.children.cycles-pp.__fdget_pos
      0.22 ±  4%      +0.3        0.56 ±  8%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.13 ±  3%      +0.3        0.48 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.13 ±  5%      +0.3        0.47 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.05 ± 62%      +0.3        0.39 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      1.87            +0.4        2.23 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
      0.00            +0.4        0.36 ±  6%  perf-profile.children.cycles-pp.___perf_sw_event
      0.14 ±  8%      +0.4        0.55 ±  2%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.31 ±  8%      +0.4        0.76 ±  3%  perf-profile.children.cycles-pp.__switch_to
      0.17 ±  8%      +0.5        0.63        perf-profile.children.cycles-pp.check_preempt_curr
      0.20 ±  9%      +0.5        0.67        perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.07 ±  7%      +0.5        0.54 ±  4%  perf-profile.children.cycles-pp.pick_next_entity
      0.10 ± 10%      +0.5        0.58 ±  6%  perf-profile.children.cycles-pp.__switch_to_asm
      0.00            +0.5        0.48 ± 29%  perf-profile.children.cycles-pp.sched_ttwu_pending
     19.34            +0.5       19.84 ±  2%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.23 ±  5%      +0.5        0.73 ±  4%  perf-profile.children.cycles-pp.prepare_to_wait
      0.17 ±  7%      +0.5        0.67 ±  7%  perf-profile.children.cycles-pp.update_rq_clock
      2.50 ± 12%      +0.6        3.07 ±  6%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.00            +0.6        0.64 ± 29%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.00            +0.7        0.67 ± 29%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.25 ±  7%      +0.7        0.95 ±  4%  perf-profile.children.cycles-pp.switch_fpu_return
      0.00            +0.8        0.82 ± 28%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      2.57 ± 13%      +0.9        3.44 ±  6%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.21 ±  5%      +0.9        1.10 ±  2%  perf-profile.children.cycles-pp.set_next_entity
      0.71            +0.9        1.60 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      2.62 ± 13%      +0.9        3.52 ±  6%  perf-profile.children.cycles-pp.ttwu_do_activate
     13.18            +1.1       14.27 ±  2%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.20 ± 13%      +1.1        1.29 ±  4%  perf-profile.children.cycles-pp.reweight_entity
      0.48 ±  4%      +1.6        2.08 ±  6%  perf-profile.children.cycles-pp.update_load_avg
      0.26 ±  4%      +1.6        1.87 ± 13%  perf-profile.children.cycles-pp.available_idle_cpu
     46.95            +1.7       48.66        perf-profile.children.cycles-pp.unix_stream_sendmsg
     47.13            +1.9       49.03        perf-profile.children.cycles-pp.sock_sendmsg
     47.25            +2.0       49.24        perf-profile.children.cycles-pp.sock_write_iter
     47.46            +2.0       49.47        perf-profile.children.cycles-pp.new_sync_write
      0.49 ±  3%      +2.1        2.55 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.58 ± 12%      +2.1        3.73 ±  4%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      1.63 ± 12%      +2.2        3.84 ±  4%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     47.87            +2.3       50.17        perf-profile.children.cycles-pp.vfs_write
     48.36            +2.5       50.89        perf-profile.children.cycles-pp.ksys_write
      0.60 ±  4%      +2.7        3.33 ± 11%  perf-profile.children.cycles-pp.select_idle_sibling
      4.70 ± 12%      +2.8        7.47 ±  6%  perf-profile.children.cycles-pp.schedule_timeout
      0.80 ±  5%      +3.3        4.11 ± 10%  perf-profile.children.cycles-pp.select_task_rq_fair
      5.94 ± 12%      +3.9        9.88 ±  5%  perf-profile.children.cycles-pp.__schedule
      5.87 ± 12%      +4.0        9.92 ±  5%  perf-profile.children.cycles-pp.schedule
      5.93 ± 12%      +4.7       10.59 ±  6%  perf-profile.children.cycles-pp.try_to_wake_up
      5.97 ± 12%      +4.7       10.67 ±  6%  perf-profile.children.cycles-pp.autoremove_wake_function
      6.03 ± 12%      +4.8       10.80 ±  6%  perf-profile.children.cycles-pp.__wake_up_common
      6.25 ± 12%      +5.1       11.31 ±  6%  perf-profile.children.cycles-pp.__wake_up_common_lock
     50.30            +5.1       55.41        perf-profile.children.cycles-pp.__libc_write
      6.37 ± 11%      +5.2       11.61 ±  6%  perf-profile.children.cycles-pp.sock_def_readable
      6.47 ±  8%      -5.6        0.85 ± 11%  perf-profile.self.cycles-pp.page_counter_cancel
     10.75            -2.5        8.30 ±  4%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      2.92 ±  9%      -2.4        0.47 ±  8%  perf-profile.self.cycles-pp.propagate_protected_usage
      9.63            -1.9        7.77 ±  2%  perf-profile.self.cycles-pp.kmem_cache_free
      9.16            -1.2        7.94 ±  2%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      3.96            -1.0        2.97 ±  5%  perf-profile.self.cycles-pp.drain_obj_stock
      1.91 ±  9%      -0.9        0.97 ± 10%  perf-profile.self.cycles-pp.page_counter_try_charge
      1.09 ±  3%      -0.8        0.33 ± 13%  perf-profile.self.cycles-pp.__mod_memcg_state
      3.05 ±  2%      -0.5        2.53 ±  4%  perf-profile.self.cycles-pp.refill_obj_stock
      1.06            -0.5        0.55 ±  3%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.42 ±  6%      -0.3        0.11 ± 25%  perf-profile.self.cycles-pp.___slab_alloc
      0.63 ±  3%      -0.2        0.44 ±  2%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.40 ±  6%      -0.1        0.32 ±  2%  perf-profile.self.cycles-pp.__slab_free
      0.22 ±  8%      -0.1        0.15 ±  3%  perf-profile.self.cycles-pp.mutex_unlock
      0.11 ±  6%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.skb_release_data
      0.07 ±  7%      +0.0        0.09        perf-profile.self.cycles-pp.unix_destruct_scm
      0.07 ±  6%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.10            +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.__entry_text_start
      0.09 ±  4%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.03 ±100%      +0.0        0.06        perf-profile.self.cycles-pp.skb_unlink
      0.05            +0.0        0.09        perf-profile.self.cycles-pp.sock_alloc_send_pskb
      0.41 ±  4%      +0.0        0.46 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.08 ± 10%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.__libc_read
      0.01 ±173%      +0.0        0.06        perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__skb_datagram_iter
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__ksize
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.rb_next
      0.14 ± 11%      +0.1        0.19 ±  8%  perf-profile.self.cycles-pp.perf_tp_event
      0.06 ± 11%      +0.1        0.11 ±  3%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__might_fault
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__x64_sys_write
      0.22 ±  5%      +0.1        0.27 ±  2%  perf-profile.self.cycles-pp.__check_object_size
      0.14 ±  7%      +0.1        0.19 ±  2%  perf-profile.self.cycles-pp.new_sync_write
      0.11 ±  4%      +0.1        0.16 ±  2%  perf-profile.self.cycles-pp.__alloc_skb
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.rcu_note_context_switch
      0.09            +0.1        0.15 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      0.22 ±  5%      +0.1        0.28 ±  2%  perf-profile.self.cycles-pp.fput_many
      0.09            +0.1        0.15 ±  5%  perf-profile.self.cycles-pp.unix_write_space
      0.09 ±  7%      +0.1        0.15 ±  2%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.do_syscall_64
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp._copy_from_iter
      0.33 ±  2%      +0.1        0.40 ±  3%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.19 ±  2%      +0.1        0.26        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.01 ±173%      +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.security_file_permission
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.mem_cgroup_from_task
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp._cond_resched
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.09 ±  4%      +0.1        0.16 ±  2%  perf-profile.self.cycles-pp.aa_sk_perm
      0.10 ±  4%      +0.1        0.17 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.unix_stream_recvmsg
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.wakeup_preempt_entity
      0.05 ±  8%      +0.1        0.13 ±  9%  perf-profile.self.cycles-pp.__wake_up_common
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.ksys_read
      0.03 ±100%      +0.1        0.11 ±  7%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.check_preempt_curr
      0.25 ±  3%      +0.1        0.33 ±  7%  perf-profile.self.cycles-pp.sock_wfree
      0.21 ±  6%      +0.1        0.30 ±  5%  perf-profile.self.cycles-pp.sock_def_readable
      0.11 ±  3%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.sock_write_iter
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.alloc_skb_with_frags
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.rcu_qs
      0.00            +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.06            +0.1        0.15 ±  3%  perf-profile.self.cycles-pp.consume_skb
      0.10 ±  5%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.09 ± 12%  perf-profile.self.cycles-pp.get_xsave_addr
      0.11            +0.1        0.20        perf-profile.self.cycles-pp.__check_heap_object
      0.05            +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.rcu_all_qs
      0.10 ±  9%      +0.1        0.19 ±  7%  perf-profile.self.cycles-pp.native_sched_clock
      0.07 ±  7%      +0.1        0.16 ±  2%  perf-profile.self.cycles-pp._copy_to_iter
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.wait_for_unix_gc
      0.00            +0.1        0.10 ± 28%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.00            +0.1        0.10 ±  9%  perf-profile.self.cycles-pp.aa_file_perm
      0.15 ±  5%      +0.1        0.25 ±  2%  perf-profile.self.cycles-pp.common_file_perm
      0.07 ±  5%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.__fget_light
      0.09 ±  4%      +0.1        0.19 ±  2%  perf-profile.self.cycles-pp.sock_read_iter
      0.00            +0.1        0.10        perf-profile.self.cycles-pp.put_prev_entity
      0.05            +0.1        0.16 ±  7%  perf-profile.self.cycles-pp.dequeue_entity
      0.33 ±  2%      +0.1        0.44 ±  2%  perf-profile.self.cycles-pp.__pthread_disable_asynccancel
      0.00            +0.1        0.11 ± 11%  perf-profile.self.cycles-pp.migrate_task_rq_fair
      0.17 ±  4%      +0.1        0.29 ±  2%  perf-profile.self.cycles-pp.vfs_write
      0.00            +0.1        0.12 ±  3%  perf-profile.self.cycles-pp.__cgroup_account_cputime
      0.08 ± 13%      +0.1        0.21 ±  3%  perf-profile.self.cycles-pp.schedule
      0.08 ± 13%      +0.1        0.21 ±  6%  perf-profile.self.cycles-pp.schedule_timeout
      0.06 ± 11%      +0.1        0.18 ±  8%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.03 ±100%      +0.1        0.15 ±  8%  perf-profile.self.cycles-pp.cpumask_next_wrap
      0.00            +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.15 ±  5%      +0.1        0.29        perf-profile.self.cycles-pp.fsnotify
      0.22 ± 14%      +0.1        0.35 ±  5%  perf-profile.self.cycles-pp.enqueue_entity
      0.05            +0.1        0.19 ±  6%  perf-profile.self.cycles-pp._find_next_bit
      0.08 ±  6%      +0.1        0.22 ±  5%  perf-profile.self.cycles-pp.new_sync_read
      0.00            +0.2        0.16 ±  7%  perf-profile.self.cycles-pp.resched_curr
      0.04 ± 57%      +0.2        0.20 ±  9%  perf-profile.self.cycles-pp.set_next_buddy
      0.08 ±  6%      +0.2        0.23 ±  7%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.06 ±  9%      +0.2        0.21 ±  4%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.11 ±  4%      +0.2        0.27 ±  6%  perf-profile.self.cycles-pp.mutex_lock
      0.00            +0.2        0.17 ±  3%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.04 ± 57%      +0.2        0.22 ± 11%  perf-profile.self.cycles-pp.cpus_share_cache
      0.06 ±  9%      +0.2        0.24 ± 10%  perf-profile.self.cycles-pp.update_cfs_rq_h_load
      0.05 ±  8%      +0.2        0.24 ±  7%  perf-profile.self.cycles-pp.update_min_vruntime
      0.08 ± 15%      +0.2        0.29 ±  3%  perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
      0.10 ±  5%      +0.2        0.30 ±  4%  perf-profile.self.cycles-pp.vfs_read
      0.14 ± 10%      +0.2        0.35 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.31 ±  5%      +0.2        0.53 ±  4%  perf-profile.self.cycles-pp.__fget_files
      0.27 ±  5%      +0.2        0.48 ±  8%  perf-profile.self.cycles-pp.update_cfs_group
      0.18 ±  2%      +0.2        0.40 ±  5%  perf-profile.self.cycles-pp.___might_sleep
      0.07 ±  7%      +0.2        0.29 ±  8%  perf-profile.self.cycles-pp.try_to_wake_up
      0.12 ±  5%      +0.2        0.35 ±  6%  perf-profile.self.cycles-pp.cpuacct_charge
      0.00            +0.2        0.23 ±  4%  perf-profile.self.cycles-pp.pick_next_entity
      0.00            +0.2        0.23 ± 12%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.00            +0.2        0.23 ±  2%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.12 ±  7%      +0.2        0.36        perf-profile.self.cycles-pp.prepare_to_wait
      0.15 ±  2%      +0.3        0.41 ±  4%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.09 ± 11%      +0.3        0.36 ±  3%  perf-profile.self.cycles-pp.__enqueue_entity
      0.10 ±  7%      +0.3        0.37 ±  2%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.00            +0.3        0.27 ±  5%  perf-profile.self.cycles-pp.clear_buddies
      0.13 ± 10%      +0.3        0.40 ±  8%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.00            +0.3        0.31 ±  6%  perf-profile.self.cycles-pp.___perf_sw_event
      0.06            +0.3        0.38        perf-profile.self.cycles-pp.set_next_entity
      0.13 ±  6%      +0.3        0.45 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.08 ± 10%      +0.3        0.41 ±  2%  perf-profile.self.cycles-pp.rb_erase
      0.05 ± 62%      +0.3        0.39        perf-profile.self.cycles-pp.__calc_delta
      0.13 ±  5%      +0.3        0.47 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.29 ±  8%      +0.4        0.65 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.24 ±  2%      +0.4        0.62 ±  5%  perf-profile.self.cycles-pp.unix_stream_sendmsg
      0.11 ±  9%      +0.4        0.50 ±  9%  perf-profile.self.cycles-pp.update_rq_clock
      0.09 ±  9%      +0.5        0.54 ±  6%  perf-profile.self.cycles-pp.reweight_entity
     11.85            +0.5       12.32 ±  2%  perf-profile.self.cycles-pp.kfree
      0.10 ± 12%      +0.5        0.58 ±  6%  perf-profile.self.cycles-pp.__switch_to_asm
      0.16 ±  8%      +0.6        0.80 ±  5%  perf-profile.self.cycles-pp.update_curr
      0.25 ±  7%      +0.7        0.94 ±  4%  perf-profile.self.cycles-pp.switch_fpu_return
      0.46 ±  2%      +0.7        1.19 ±  4%  perf-profile.self.cycles-pp.unix_stream_read_generic
      0.20 ±  6%      +0.7        0.93 ± 11%  perf-profile.self.cycles-pp.select_idle_sibling
      0.21 ±  2%      +0.8        1.02 ±  9%  perf-profile.self.cycles-pp.update_load_avg
      0.89 ±  2%      +0.8        1.70 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.62            +0.9        1.54 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
     13.03            +1.0       14.04 ±  2%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.31 ±  3%      +1.2        1.53 ±  3%  perf-profile.self.cycles-pp.__schedule
      0.25 ±  5%      +1.6        1.86 ± 13%  perf-profile.self.cycles-pp.available_idle_cpu
  80845791 ± 15%    +593.4%  5.606e+08 ±  8%  interrupts.CAL:Function_call_interrupts
    554100 ± 12%   +1379.7%    8199171 ± 55%  interrupts.CPU0.CAL:Function_call_interrupts
   2633873           -29.2%    1865030        interrupts.CPU0.LOC:Local_timer_interrupts
   1482255 ± 11%     +64.5%    2439004 ± 16%  interrupts.CPU0.RES:Rescheduling_interrupts
    550298 ± 11%    +680.9%    4297484 ± 48%  interrupts.CPU1.CAL:Function_call_interrupts
   2633863           -29.2%    1865036        interrupts.CPU1.LOC:Local_timer_interrupts
   1412849 ± 12%    +130.8%    3261354 ±  8%  interrupts.CPU1.RES:Rescheduling_interrupts
    535367 ± 13%    +713.8%    4357060 ± 45%  interrupts.CPU10.CAL:Function_call_interrupts
   2633812           -29.2%    1865326        interrupts.CPU10.LOC:Local_timer_interrupts
   1398394 ± 14%    +142.1%    3386097 ± 15%  interrupts.CPU10.RES:Rescheduling_interrupts
    577438 ± 16%    +199.2%    1727584 ± 91%  interrupts.CPU100.CAL:Function_call_interrupts
   2633810           -29.2%    1865135        interrupts.CPU100.LOC:Local_timer_interrupts
   1410125 ±  3%    +110.5%    2968345 ± 20%  interrupts.CPU100.RES:Rescheduling_interrupts
    576842 ± 15%    +219.4%    1842591 ± 75%  interrupts.CPU101.CAL:Function_call_interrupts
   2633820           -29.2%    1864888        interrupts.CPU101.LOC:Local_timer_interrupts
   1449967 ±  9%    +105.9%    2985344 ± 22%  interrupts.CPU101.RES:Rescheduling_interrupts
    581564 ± 16%    +277.8%    2197241 ± 82%  interrupts.CPU102.CAL:Function_call_interrupts
   2633820           -29.2%    1865122        interrupts.CPU102.LOC:Local_timer_interrupts
   1450315 ±  6%    +118.0%    3161937 ± 16%  interrupts.CPU102.RES:Rescheduling_interrupts
    572128 ± 16%    +189.3%    1654974 ± 66%  interrupts.CPU103.CAL:Function_call_interrupts
   2633815           -29.2%    1865084        interrupts.CPU103.LOC:Local_timer_interrupts
   1485537 ±  7%    +101.5%    2993556 ± 19%  interrupts.CPU103.RES:Rescheduling_interrupts
    571702 ± 16%    +254.5%    2026666 ± 86%  interrupts.CPU104.CAL:Function_call_interrupts
   2633807           -29.2%    1865070        interrupts.CPU104.LOC:Local_timer_interrupts
   1505692 ±  7%    +121.3%    3332814 ± 15%  interrupts.CPU104.RES:Rescheduling_interrupts
    571646 ± 15%    +246.3%    1979757 ± 85%  interrupts.CPU105.CAL:Function_call_interrupts
   2633752           -29.2%    1865102        interrupts.CPU105.LOC:Local_timer_interrupts
   1476944 ±  5%    +126.3%    3342064 ± 16%  interrupts.CPU105.RES:Rescheduling_interrupts
    569567 ± 16%    +207.9%    1753774 ± 70%  interrupts.CPU106.CAL:Function_call_interrupts
   2633740           -29.2%    1865098        interrupts.CPU106.LOC:Local_timer_interrupts
   1458855 ±  5%    +112.9%    3105708 ± 11%  interrupts.CPU106.RES:Rescheduling_interrupts
    580876 ± 16%    +242.0%    1986529 ± 89%  interrupts.CPU107.CAL:Function_call_interrupts
   2633811           -29.2%    1865005        interrupts.CPU107.LOC:Local_timer_interrupts
   1418276 ±  8%    +121.3%    3139084 ± 18%  interrupts.CPU107.RES:Rescheduling_interrupts
    595179 ± 14%   +1293.0%    8291002 ± 26%  interrupts.CPU108.CAL:Function_call_interrupts
   2633789           -29.2%    1865051        interrupts.CPU108.LOC:Local_timer_interrupts
   1436203 ± 11%    +110.4%    3021112 ±  8%  interrupts.CPU108.RES:Rescheduling_interrupts
    576009 ± 16%    +562.3%    3815028 ± 12%  interrupts.CPU109.CAL:Function_call_interrupts
   2633796           -29.2%    1865119        interrupts.CPU109.LOC:Local_timer_interrupts
   1474371 ± 10%    +152.8%    3726667 ±  7%  interrupts.CPU109.RES:Rescheduling_interrupts
    537884 ± 12%    +604.1%    3787128 ± 45%  interrupts.CPU11.CAL:Function_call_interrupts
   2633889           -29.2%    1865183        interrupts.CPU11.LOC:Local_timer_interrupts
   1353579 ± 12%    +143.9%    3301148 ± 13%  interrupts.CPU11.RES:Rescheduling_interrupts
    579505 ± 15%    +602.3%    4069904 ± 25%  interrupts.CPU110.CAL:Function_call_interrupts
   2633866           -29.2%    1865154        interrupts.CPU110.LOC:Local_timer_interrupts
   1485870 ± 12%    +151.8%    3741307 ± 13%  interrupts.CPU110.RES:Rescheduling_interrupts
    574480 ± 15%    +726.8%    4750053 ± 23%  interrupts.CPU111.CAL:Function_call_interrupts
   2633731           -29.2%    1865194        interrupts.CPU111.LOC:Local_timer_interrupts
   1453539 ± 11%    +206.5%    4454917 ± 10%  interrupts.CPU111.RES:Rescheduling_interrupts
    574177 ± 16%    +583.1%    3922097 ± 28%  interrupts.CPU112.CAL:Function_call_interrupts
   2633698           -29.2%    1865157        interrupts.CPU112.LOC:Local_timer_interrupts
   1449339 ±  6%    +171.8%    3938937 ± 10%  interrupts.CPU112.RES:Rescheduling_interrupts
    572339 ± 16%    +623.2%    4139018 ±  2%  interrupts.CPU113.CAL:Function_call_interrupts
   2633574           -29.2%    1865169        interrupts.CPU113.LOC:Local_timer_interrupts
   1461578 ± 10%    +168.2%    3920285 ±  4%  interrupts.CPU113.RES:Rescheduling_interrupts
    581846 ± 15%    +747.3%    4930055 ± 24%  interrupts.CPU114.CAL:Function_call_interrupts
   2633735           -29.2%    1865175        interrupts.CPU114.LOC:Local_timer_interrupts
   1487035 ± 15%    +155.8%    3803822 ±  5%  interrupts.CPU114.RES:Rescheduling_interrupts
    568185 ± 16%    +590.8%    3924748 ± 18%  interrupts.CPU115.CAL:Function_call_interrupts
   2633765           -29.2%    1865158        interrupts.CPU115.LOC:Local_timer_interrupts
   1421770 ±  5%    +184.4%    4043182 ±  9%  interrupts.CPU115.RES:Rescheduling_interrupts
    574231 ± 15%    +601.3%    4027085 ± 16%  interrupts.CPU116.CAL:Function_call_interrupts
   2633606           -29.2%    1865145        interrupts.CPU116.LOC:Local_timer_interrupts
   1463960 ± 12%    +139.2%    3502130 ±  8%  interrupts.CPU116.RES:Rescheduling_interrupts
    567120 ± 15%    +607.9%    4014482 ± 29%  interrupts.CPU117.CAL:Function_call_interrupts
   2633557           -29.2%    1865129        interrupts.CPU117.LOC:Local_timer_interrupts
   1495359 ± 13%    +186.9%    4290630 ±  7%  interrupts.CPU117.RES:Rescheduling_interrupts
    578676 ± 16%    +550.9%    3766680 ± 16%  interrupts.CPU118.CAL:Function_call_interrupts
   2633683           -29.2%    1865186        interrupts.CPU118.LOC:Local_timer_interrupts
   1512609 ± 11%    +159.9%    3931438 ±  6%  interrupts.CPU118.RES:Rescheduling_interrupts
    566333 ± 17%    +595.5%    3938821 ±  6%  interrupts.CPU119.CAL:Function_call_interrupts
   2633592           -29.2%    1865154        interrupts.CPU119.LOC:Local_timer_interrupts
   1488137 ± 14%    +144.6%    3640022 ±  7%  interrupts.CPU119.RES:Rescheduling_interrupts
    538190 ± 13%    +569.6%    3603715 ± 48%  interrupts.CPU12.CAL:Function_call_interrupts
   2633798           -29.2%    1865228        interrupts.CPU12.LOC:Local_timer_interrupts
   1471083 ± 18%    +126.4%    3330734 ± 12%  interrupts.CPU12.RES:Rescheduling_interrupts
    574035 ± 14%    +686.0%    4511762 ± 25%  interrupts.CPU120.CAL:Function_call_interrupts
   2633804           -29.2%    1865123        interrupts.CPU120.LOC:Local_timer_interrupts
   1431892 ± 10%    +167.6%    3831695 ±  5%  interrupts.CPU120.RES:Rescheduling_interrupts
    566905 ± 16%    +781.2%    4995501 ± 24%  interrupts.CPU121.CAL:Function_call_interrupts
   2633711           -29.2%    1865103        interrupts.CPU121.LOC:Local_timer_interrupts
   1464668 ± 12%    +184.4%    4164954 ±  6%  interrupts.CPU121.RES:Rescheduling_interrupts
    574493 ± 16%    +767.3%    4982480 ± 21%  interrupts.CPU122.CAL:Function_call_interrupts
   2633766           -29.2%    1865160        interrupts.CPU122.LOC:Local_timer_interrupts
   1465207 ± 11%    +157.0%    3765515 ± 15%  interrupts.CPU122.RES:Rescheduling_interrupts
    574985 ± 15%    +578.3%    3899889 ± 39%  interrupts.CPU123.CAL:Function_call_interrupts
   2633799           -29.2%    1865115        interrupts.CPU123.LOC:Local_timer_interrupts
   1528615 ±  8%    +152.1%    3853358 ±  5%  interrupts.CPU123.RES:Rescheduling_interrupts
    573524 ± 15%    +603.1%    4032416 ± 23%  interrupts.CPU124.CAL:Function_call_interrupts
   2633643           -29.2%    1865025        interrupts.CPU124.LOC:Local_timer_interrupts
   1438497 ± 12%    +178.0%    3999107 ±  8%  interrupts.CPU124.RES:Rescheduling_interrupts
    572476 ± 16%    +743.8%    4830789 ± 26%  interrupts.CPU125.CAL:Function_call_interrupts
   2633770           -29.2%    1865163        interrupts.CPU125.LOC:Local_timer_interrupts
   1487222 ± 11%    +165.9%    3954967 ±  4%  interrupts.CPU125.RES:Rescheduling_interrupts
    571725 ± 16%   +1191.8%    7385638 ± 13%  interrupts.CPU126.CAL:Function_call_interrupts
   2633775           -29.2%    1865098        interrupts.CPU126.LOC:Local_timer_interrupts
   1401581 ±  3%    +105.1%    2874635 ± 12%  interrupts.CPU126.RES:Rescheduling_interrupts
    566626 ± 16%    +669.8%    4361710 ± 11%  interrupts.CPU127.CAL:Function_call_interrupts
   2633816           -29.2%    1864935        interrupts.CPU127.LOC:Local_timer_interrupts
   1419656          +190.3%    4121899 ±  3%  interrupts.CPU127.RES:Rescheduling_interrupts
    563855 ± 16%    +680.2%    4399187 ± 13%  interrupts.CPU128.CAL:Function_call_interrupts
   2633791           -29.2%    1865117        interrupts.CPU128.LOC:Local_timer_interrupts
   1437346 ±  4%    +174.2%    3941330 ±  7%  interrupts.CPU128.RES:Rescheduling_interrupts
    562917 ± 16%    +682.1%    4402639 ± 32%  interrupts.CPU129.CAL:Function_call_interrupts
   2633617           -29.2%    1865180        interrupts.CPU129.LOC:Local_timer_interrupts
   1404082 ±  4%    +178.4%    3909629 ±  2%  interrupts.CPU129.RES:Rescheduling_interrupts
    535349 ± 13%    +758.1%    4593651 ± 51%  interrupts.CPU13.CAL:Function_call_interrupts
   2633788           -29.2%    1865227        interrupts.CPU13.LOC:Local_timer_interrupts
   1410172 ± 17%    +132.2%    3274335 ± 18%  interrupts.CPU13.RES:Rescheduling_interrupts
    557248 ± 17%    +655.3%    4208639 ± 15%  interrupts.CPU130.CAL:Function_call_interrupts
   2633723           -29.2%    1865055        interrupts.CPU130.LOC:Local_timer_interrupts
   1392351 ±  2%    +187.2%    3998949 ±  6%  interrupts.CPU130.RES:Rescheduling_interrupts
    555534 ± 16%    +718.3%    4545984 ± 32%  interrupts.CPU131.CAL:Function_call_interrupts
   2633752           -29.2%    1865092        interrupts.CPU131.LOC:Local_timer_interrupts
   1489083 ±  8%    +178.4%    4145278 ±  3%  interrupts.CPU131.RES:Rescheduling_interrupts
    558117 ± 16%    +745.2%    4717120 ± 19%  interrupts.CPU132.CAL:Function_call_interrupts
   2633542           -29.2%    1865057        interrupts.CPU132.LOC:Local_timer_interrupts
   1377068 ±  5%    +181.2%    3872442 ±  4%  interrupts.CPU132.RES:Rescheduling_interrupts
    560564 ± 16%    +695.7%    4460292 ± 29%  interrupts.CPU133.CAL:Function_call_interrupts
   2633715           -29.2%    1865108        interrupts.CPU133.LOC:Local_timer_interrupts
   1458375          +152.0%    3674582 ±  8%  interrupts.CPU133.RES:Rescheduling_interrupts
    553770 ± 16%    +677.8%    4307149 ± 14%  interrupts.CPU134.CAL:Function_call_interrupts
   2633784           -29.2%    1865070        interrupts.CPU134.LOC:Local_timer_interrupts
   1386197 ±  3%    +173.3%    3788186 ± 10%  interrupts.CPU134.RES:Rescheduling_interrupts
    559795 ± 15%    +658.3%    4245180 ± 29%  interrupts.CPU135.CAL:Function_call_interrupts
   2633732           -29.2%    1865135        interrupts.CPU135.LOC:Local_timer_interrupts
   1437159          +201.7%    4336194 ±  6%  interrupts.CPU135.RES:Rescheduling_interrupts
    565365 ± 16%    +595.2%    3930650 ± 15%  interrupts.CPU136.CAL:Function_call_interrupts
   2633726           -29.2%    1864945        interrupts.CPU136.LOC:Local_timer_interrupts
   1366297 ±  4%    +188.7%    3943869 ±  3%  interrupts.CPU136.RES:Rescheduling_interrupts
    560228 ± 15%    +785.3%    4959766 ± 11%  interrupts.CPU137.CAL:Function_call_interrupts
   2633801           -29.2%    1865074        interrupts.CPU137.LOC:Local_timer_interrupts
   1378808 ±  3%    +178.5%    3839326 ±  5%  interrupts.CPU137.RES:Rescheduling_interrupts
    557473 ± 18%    +711.1%    4521407 ± 18%  interrupts.CPU138.CAL:Function_call_interrupts
   2633801           -29.2%    1865014        interrupts.CPU138.LOC:Local_timer_interrupts
   1382141          +186.2%    3955521 ±  5%  interrupts.CPU138.RES:Rescheduling_interrupts
    557471 ± 16%    +676.9%    4331168 ± 33%  interrupts.CPU139.CAL:Function_call_interrupts
   2633711           -29.2%    1864983        interrupts.CPU139.LOC:Local_timer_interrupts
   1351197          +178.0%    3756469 ± 11%  interrupts.CPU139.RES:Rescheduling_interrupts
    540897 ± 12%    +522.4%    3366732 ± 51%  interrupts.CPU14.CAL:Function_call_interrupts
   2633694           -29.2%    1865533        interrupts.CPU14.LOC:Local_timer_interrupts
   1428323 ± 12%    +127.1%    3244079 ± 19%  interrupts.CPU14.RES:Rescheduling_interrupts
    568254 ± 16%    +772.3%    4957094 ± 31%  interrupts.CPU140.CAL:Function_call_interrupts
   2633781           -29.2%    1864986        interrupts.CPU140.LOC:Local_timer_interrupts
   1384947 ±  4%    +185.1%    3948921 ±  6%  interrupts.CPU140.RES:Rescheduling_interrupts
    565567 ± 16%    +782.9%    4993387 ± 17%  interrupts.CPU141.CAL:Function_call_interrupts
   2633738           -29.2%    1864968        interrupts.CPU141.LOC:Local_timer_interrupts
   1480611 ±  2%    +168.4%    3973399 ±  5%  interrupts.CPU141.RES:Rescheduling_interrupts
    560572 ± 16%    +795.9%    5022300 ± 25%  interrupts.CPU142.CAL:Function_call_interrupts
   2633708           -29.2%    1865072        interrupts.CPU142.LOC:Local_timer_interrupts
   1421927 ±  2%    +161.2%    3713622 ± 12%  interrupts.CPU142.RES:Rescheduling_interrupts
    557943 ± 16%    +578.3%    3784321 ± 26%  interrupts.CPU143.CAL:Function_call_interrupts
   2633702           -29.2%    1865057        interrupts.CPU143.LOC:Local_timer_interrupts
   1360011 ±  4%    +183.5%    3856151 ±  3%  interrupts.CPU143.RES:Rescheduling_interrupts
    537700 ± 12%    +679.7%    4192322 ± 52%  interrupts.CPU15.CAL:Function_call_interrupts
   2633763           -29.2%    1865110        interrupts.CPU15.LOC:Local_timer_interrupts
   1408175 ± 17%    +141.5%    3400976 ±  2%  interrupts.CPU15.RES:Rescheduling_interrupts
    534343 ± 13%    +635.0%    3927692 ± 46%  interrupts.CPU16.CAL:Function_call_interrupts
   2633738           -29.2%    1865093        interrupts.CPU16.LOC:Local_timer_interrupts
   1406704 ± 11%    +158.0%    3628791 ±  5%  interrupts.CPU16.RES:Rescheduling_interrupts
    534677 ± 13%    +714.1%    4353035 ± 54%  interrupts.CPU17.CAL:Function_call_interrupts
   2633799           -29.2%    1865188        interrupts.CPU17.LOC:Local_timer_interrupts
   1375046 ± 10%    +142.3%    3331891 ± 11%  interrupts.CPU17.RES:Rescheduling_interrupts
    593706 ± 15%    +288.1%    2304241 ±103%  interrupts.CPU18.CAL:Function_call_interrupts
   2633800           -29.2%    1865119        interrupts.CPU18.LOC:Local_timer_interrupts
   1396394 ±  9%     +77.3%    2475349 ± 18%  interrupts.CPU18.RES:Rescheduling_interrupts
    584823 ± 14%    +196.2%    1732209 ± 70%  interrupts.CPU19.CAL:Function_call_interrupts
   2633760           -29.2%    1865138        interrupts.CPU19.LOC:Local_timer_interrupts
   1463821 ±  4%    +102.4%    2963448 ± 12%  interrupts.CPU19.RES:Rescheduling_interrupts
    537532 ± 14%    +659.5%    4082710 ± 49%  interrupts.CPU2.CAL:Function_call_interrupts
   2633695           -29.2%    1865040        interrupts.CPU2.LOC:Local_timer_interrupts
   1398069 ± 10%    +120.9%    3088668 ±  8%  interrupts.CPU2.RES:Rescheduling_interrupts
    577827 ± 16%    +165.3%    1533212 ± 72%  interrupts.CPU20.CAL:Function_call_interrupts
   2633830           -29.2%    1865061        interrupts.CPU20.LOC:Local_timer_interrupts
   1462963 ±  6%     +96.3%    2872097 ± 14%  interrupts.CPU20.RES:Rescheduling_interrupts
    568745 ± 17%    +156.5%    1458657 ± 63%  interrupts.CPU21.CAL:Function_call_interrupts
   2633790           -29.2%    1865125        interrupts.CPU21.LOC:Local_timer_interrupts
   1455888 ±  4%    +107.8%    3025217 ± 12%  interrupts.CPU21.RES:Rescheduling_interrupts
    569078 ± 17%    +242.1%    1946561 ± 90%  interrupts.CPU22.CAL:Function_call_interrupts
   2633826           -29.2%    1865145        interrupts.CPU22.LOC:Local_timer_interrupts
   1457448 ±  6%    +108.2%    3034926 ± 12%  interrupts.CPU22.RES:Rescheduling_interrupts
    566117 ± 15%    +194.9%    1669379 ± 84%  interrupts.CPU23.CAL:Function_call_interrupts
   2633849           -29.2%    1865132        interrupts.CPU23.LOC:Local_timer_interrupts
   1466517 ±  5%     +99.2%    2921450 ± 23%  interrupts.CPU23.RES:Rescheduling_interrupts
    567257 ± 17%    +236.8%    1910543 ± 94%  interrupts.CPU24.CAL:Function_call_interrupts
   2633857           -29.2%    1865114        interrupts.CPU24.LOC:Local_timer_interrupts
   1473726 ±  5%    +111.6%    3118669 ±  9%  interrupts.CPU24.RES:Rescheduling_interrupts
    571873 ± 17%    +207.7%    1759796 ± 81%  interrupts.CPU25.CAL:Function_call_interrupts
   2633844           -29.2%    1865127        interrupts.CPU25.LOC:Local_timer_interrupts
   1456649 ±  9%     +92.6%    2805286 ± 11%  interrupts.CPU25.RES:Rescheduling_interrupts
    578305 ± 18%    +185.9%    1653215 ± 76%  interrupts.CPU26.CAL:Function_call_interrupts
   2633838           -29.2%    1865119        interrupts.CPU26.LOC:Local_timer_interrupts
   1424045 ±  5%    +102.9%    2889535 ± 15%  interrupts.CPU26.RES:Rescheduling_interrupts
    562598 ± 17%    +236.8%    1894614 ± 83%  interrupts.CPU27.CAL:Function_call_interrupts
   2633855           -29.2%    1865066        interrupts.CPU27.LOC:Local_timer_interrupts
   1521521 ±  9%     +99.0%    3028018 ± 14%  interrupts.CPU27.RES:Rescheduling_interrupts
    573439 ± 17%    +169.9%    1547685 ± 77%  interrupts.CPU28.CAL:Function_call_interrupts
   2633841           -29.2%    1865133        interrupts.CPU28.LOC:Local_timer_interrupts
   1476417 ±  7%     +96.8%    2905945 ± 10%  interrupts.CPU28.RES:Rescheduling_interrupts
    569303 ± 16%    +219.8%    1820704 ± 74%  interrupts.CPU29.CAL:Function_call_interrupts
   2633840           -29.2%    1865144        interrupts.CPU29.LOC:Local_timer_interrupts
   1463402 ±  5%     +99.6%    2920924 ± 11%  interrupts.CPU29.RES:Rescheduling_interrupts
    540299 ± 13%    +572.8%    3635184 ± 46%  interrupts.CPU3.CAL:Function_call_interrupts
   2633834           -29.2%    1865068        interrupts.CPU3.LOC:Local_timer_interrupts
   1357363 ± 11%    +128.9%    3106456 ±  8%  interrupts.CPU3.RES:Rescheduling_interrupts
    566720 ± 17%    +249.9%    1983177 ± 90%  interrupts.CPU30.CAL:Function_call_interrupts
   2633849           -29.2%    1865111        interrupts.CPU30.LOC:Local_timer_interrupts
   1451386 ±  6%    +102.3%    2936684 ± 11%  interrupts.CPU30.RES:Rescheduling_interrupts
    572101 ± 17%    +207.0%    1756300 ± 81%  interrupts.CPU31.CAL:Function_call_interrupts
   2633872           -29.2%    1865173        interrupts.CPU31.LOC:Local_timer_interrupts
   1442117 ±  5%    +101.5%    2906320 ± 14%  interrupts.CPU31.RES:Rescheduling_interrupts
    566117 ± 16%    +221.8%    1821798 ± 79%  interrupts.CPU32.CAL:Function_call_interrupts
   2633836           -29.2%    1865082        interrupts.CPU32.LOC:Local_timer_interrupts
   1520018 ±  6%    +111.8%    3219552 ± 13%  interrupts.CPU32.RES:Rescheduling_interrupts
    572723 ± 17%    +219.3%    1828506 ± 87%  interrupts.CPU33.CAL:Function_call_interrupts
   2633890           -29.2%    1865098        interrupts.CPU33.LOC:Local_timer_interrupts
   1469532 ±  4%    +109.2%    3074482 ± 14%  interrupts.CPU33.RES:Rescheduling_interrupts
    562451 ± 16%    +198.3%    1677546 ± 86%  interrupts.CPU34.CAL:Function_call_interrupts
   2633860           -29.2%    1865114        interrupts.CPU34.LOC:Local_timer_interrupts
   1485260 ±  7%     +84.9%    2746634 ± 24%  interrupts.CPU34.RES:Rescheduling_interrupts
    574437 ± 17%    +192.2%    1678665 ± 84%  interrupts.CPU35.CAL:Function_call_interrupts
   2633873           -29.2%    1865198        interrupts.CPU35.LOC:Local_timer_interrupts
   1454846 ±  9%    +106.8%    3009342 ±  8%  interrupts.CPU35.RES:Rescheduling_interrupts
    603848 ± 16%   +1507.1%    9704505 ± 20%  interrupts.CPU36.CAL:Function_call_interrupts
   2633701           -29.2%    1864749        interrupts.CPU36.LOC:Local_timer_interrupts
   1484946 ±  9%     +72.6%    2562490 ±  7%  interrupts.CPU36.RES:Rescheduling_interrupts
    571784 ± 17%    +758.6%    4909125 ± 14%  interrupts.CPU37.CAL:Function_call_interrupts
   2633716           -29.2%    1865119        interrupts.CPU37.LOC:Local_timer_interrupts
   1483209 ± 10%    +150.6%    3716519 ± 14%  interrupts.CPU37.RES:Rescheduling_interrupts
    570497 ± 16%    +667.3%    4377236 ± 33%  interrupts.CPU38.CAL:Function_call_interrupts
   2633719           -29.2%    1865133        interrupts.CPU38.LOC:Local_timer_interrupts
   1511144 ± 11%    +125.5%    3408141 ±  9%  interrupts.CPU38.RES:Rescheduling_interrupts
    566973 ± 15%    +657.0%    4292058 ± 17%  interrupts.CPU39.CAL:Function_call_interrupts
   2633789           -29.2%    1865151        interrupts.CPU39.LOC:Local_timer_interrupts
   1495142 ± 11%    +134.1%    3499448 ±  6%  interrupts.CPU39.RES:Rescheduling_interrupts
    535647 ± 13%    +564.3%    3558480 ± 42%  interrupts.CPU4.CAL:Function_call_interrupts
   2633852           -29.2%    1865145        interrupts.CPU4.LOC:Local_timer_interrupts
   1390793 ± 14%    +145.7%    3416812 ±  4%  interrupts.CPU4.RES:Rescheduling_interrupts
    569331 ± 16%    +784.2%    5033925 ± 11%  interrupts.CPU40.CAL:Function_call_interrupts
   2633683           -29.2%    1865105        interrupts.CPU40.LOC:Local_timer_interrupts
   1444489 ± 13%    +128.6%    3302235 ±  7%  interrupts.CPU40.RES:Rescheduling_interrupts
    561437 ± 16%    +791.8%    5006730 ± 30%  interrupts.CPU41.CAL:Function_call_interrupts
   2633767           -29.2%    1865127        interrupts.CPU41.LOC:Local_timer_interrupts
   1482967 ± 12%    +140.4%    3565126 ±  3%  interrupts.CPU41.RES:Rescheduling_interrupts
    569466 ± 15%    +717.2%    4653562 ± 22%  interrupts.CPU42.CAL:Function_call_interrupts
   2633766           -29.2%    1865143        interrupts.CPU42.LOC:Local_timer_interrupts
   1557689 ± 10%    +133.7%    3640722 ± 10%  interrupts.CPU42.RES:Rescheduling_interrupts
    570194 ± 16%    +640.8%    4224074 ± 16%  interrupts.CPU43.CAL:Function_call_interrupts
   2633582           -29.2%    1865069        interrupts.CPU43.LOC:Local_timer_interrupts
   1427263 ± 10%    +139.6%    3420407 ± 12%  interrupts.CPU43.RES:Rescheduling_interrupts
    565434 ± 16%    +891.8%    5608254 ± 22%  interrupts.CPU44.CAL:Function_call_interrupts
   2633573           -29.2%    1865201        interrupts.CPU44.LOC:Local_timer_interrupts
   1481005 ± 14%    +117.4%    3219881 ±  4%  interrupts.CPU44.RES:Rescheduling_interrupts
    570693 ± 15%    +632.7%    4181757 ± 12%  interrupts.CPU45.CAL:Function_call_interrupts
   2633516           -29.2%    1865185        interrupts.CPU45.LOC:Local_timer_interrupts
   1497744 ± 15%    +137.6%    3557955 ±  2%  interrupts.CPU45.RES:Rescheduling_interrupts
    572207 ± 15%    +685.8%    4496592 ± 10%  interrupts.CPU46.CAL:Function_call_interrupts
   2633693           -29.2%    1865179        interrupts.CPU46.LOC:Local_timer_interrupts
   1533341 ± 13%    +128.5%    3503837 ±  6%  interrupts.CPU46.RES:Rescheduling_interrupts
    565721 ± 16%    +690.5%    4471782 ± 25%  interrupts.CPU47.CAL:Function_call_interrupts
   2633757           -29.2%    1865153        interrupts.CPU47.LOC:Local_timer_interrupts
   1490647 ± 10%    +150.6%    3735066 ± 16%  interrupts.CPU47.RES:Rescheduling_interrupts
    569497 ± 15%    +822.3%    5252711 ± 28%  interrupts.CPU48.CAL:Function_call_interrupts
   2633765           -29.2%    1865117        interrupts.CPU48.LOC:Local_timer_interrupts
   1520386 ± 10%    +140.1%    3650726 ±  9%  interrupts.CPU48.RES:Rescheduling_interrupts
    564031 ± 16%    +729.1%    4676576 ± 31%  interrupts.CPU49.CAL:Function_call_interrupts
   2633799           -29.2%    1865164        interrupts.CPU49.LOC:Local_timer_interrupts
   1490515 ±  8%    +134.5%    3495727 ±  5%  interrupts.CPU49.RES:Rescheduling_interrupts
    531201 ± 12%    +598.6%    3711227 ± 56%  interrupts.CPU5.CAL:Function_call_interrupts
   2633842           -29.2%    1865170        interrupts.CPU5.LOC:Local_timer_interrupts
   1343122 ± 15%    +145.9%    3303020 ± 11%  interrupts.CPU5.RES:Rescheduling_interrupts
    566959 ± 15%    +832.5%    5287164 ± 18%  interrupts.CPU50.CAL:Function_call_interrupts
   2633824           -29.2%    1865165        interrupts.CPU50.LOC:Local_timer_interrupts
   1503736 ± 11%    +141.6%    3632972 ±  7%  interrupts.CPU50.RES:Rescheduling_interrupts
    560012 ± 15%    +645.3%    4174036 ±  8%  interrupts.CPU51.CAL:Function_call_interrupts
   2633772           -29.2%    1865146        interrupts.CPU51.LOC:Local_timer_interrupts
   1504724 ± 13%    +149.6%    3756539 ±  5%  interrupts.CPU51.RES:Rescheduling_interrupts
    562790 ± 14%    +736.1%    4705640 ± 16%  interrupts.CPU52.CAL:Function_call_interrupts
   2633752           -29.2%    1865090        interrupts.CPU52.LOC:Local_timer_interrupts
   1491058 ±  8%    +150.3%    3732022 ± 10%  interrupts.CPU52.RES:Rescheduling_interrupts
    571396 ± 16%    +729.7%    4740766 ± 15%  interrupts.CPU53.CAL:Function_call_interrupts
   2633685           -29.2%    1865142        interrupts.CPU53.LOC:Local_timer_interrupts
   1468809 ± 13%    +128.9%    3361840 ±  7%  interrupts.CPU53.RES:Rescheduling_interrupts
    593346 ± 15%   +1289.7%    8245619 ± 11%  interrupts.CPU54.CAL:Function_call_interrupts
   2633797           -29.2%    1864977        interrupts.CPU54.LOC:Local_timer_interrupts
   1475562 ±  5%     +86.3%    2749109 ±  7%  interrupts.CPU54.RES:Rescheduling_interrupts
    564926 ± 16%    +813.2%    5158877 ± 21%  interrupts.CPU55.CAL:Function_call_interrupts
   2633823           -29.2%    1865088        interrupts.CPU55.LOC:Local_timer_interrupts
   1445269 ±  6%    +127.3%    3285587 ±  4%  interrupts.CPU55.RES:Rescheduling_interrupts
    563415 ± 17%    +679.2%    4390171 ± 13%  interrupts.CPU56.CAL:Function_call_interrupts
   2633811           -29.2%    1865068        interrupts.CPU56.LOC:Local_timer_interrupts
   1408960 ±  5%    +146.0%    3466532 ± 10%  interrupts.CPU56.RES:Rescheduling_interrupts
    563764 ± 15%    +742.7%    4750963 ± 26%  interrupts.CPU57.CAL:Function_call_interrupts
   2633736           -29.2%    1864961        interrupts.CPU57.LOC:Local_timer_interrupts
   1461158 ±  2%    +133.2%    3407830 ±  3%  interrupts.CPU57.RES:Rescheduling_interrupts
    553457 ± 16%    +810.1%    5037149 ± 22%  interrupts.CPU58.CAL:Function_call_interrupts
   2633812           -29.2%    1864999        interrupts.CPU58.LOC:Local_timer_interrupts
   1462890 ±  4%    +131.4%    3385299 ±  6%  interrupts.CPU58.RES:Rescheduling_interrupts
    554838 ± 15%    +771.7%    4836459 ± 19%  interrupts.CPU59.CAL:Function_call_interrupts
   2633675           -29.2%    1865147        interrupts.CPU59.LOC:Local_timer_interrupts
   1448096 ±  5%    +164.5%    3830835 ±  2%  interrupts.CPU59.RES:Rescheduling_interrupts
    536623 ± 13%    +634.3%    3940672 ± 49%  interrupts.CPU6.CAL:Function_call_interrupts
   2633823           -29.2%    1865043        interrupts.CPU6.LOC:Local_timer_interrupts
   1466299 ± 14%    +139.6%    3513370 ±  8%  interrupts.CPU6.RES:Rescheduling_interrupts
    557353 ± 16%    +906.0%    5607132 ± 18%  interrupts.CPU60.CAL:Function_call_interrupts
   2633707           -29.2%    1864994        interrupts.CPU60.LOC:Local_timer_interrupts
   1485961 ±  4%    +111.3%    3140077 ±  5%  interrupts.CPU60.RES:Rescheduling_interrupts
    555243 ± 16%    +768.6%    4822767 ± 32%  interrupts.CPU61.CAL:Function_call_interrupts
   2633799           -29.2%    1865079        interrupts.CPU61.LOC:Local_timer_interrupts
   1437825 ±  4%    +132.8%    3347200 ±  7%  interrupts.CPU61.RES:Rescheduling_interrupts
    551697 ± 16%    +727.8%    4566738 ± 20%  interrupts.CPU62.CAL:Function_call_interrupts
   2633795           -29.2%    1864957        interrupts.CPU62.LOC:Local_timer_interrupts
   1478294 ±  3%    +133.8%    3456670 ±  5%  interrupts.CPU62.RES:Rescheduling_interrupts
    552353 ± 17%    +709.9%    4473328 ± 22%  interrupts.CPU63.CAL:Function_call_interrupts
   2633799           -29.2%    1865043        interrupts.CPU63.LOC:Local_timer_interrupts
   1465237 ±  7%    +154.1%    3723197 ± 13%  interrupts.CPU63.RES:Rescheduling_interrupts
    558022 ± 15%    +689.9%    4407595 ± 26%  interrupts.CPU64.CAL:Function_call_interrupts
   2633709           -29.2%    1865086        interrupts.CPU64.LOC:Local_timer_interrupts
   1448988 ±  6%    +147.4%    3585414 ±  7%  interrupts.CPU64.RES:Rescheduling_interrupts
    553929 ± 16%    +761.2%    4770656 ± 20%  interrupts.CPU65.CAL:Function_call_interrupts
   2633780           -29.2%    1865052        interrupts.CPU65.LOC:Local_timer_interrupts
   1477139 ±  3%    +149.0%    3677909 ±  5%  interrupts.CPU65.RES:Rescheduling_interrupts
    552322 ± 17%    +838.0%    5180852 ± 18%  interrupts.CPU66.CAL:Function_call_interrupts
   2633753           -29.2%    1864927        interrupts.CPU66.LOC:Local_timer_interrupts
   1435174 ±  2%    +159.6%    3725669 ±  6%  interrupts.CPU66.RES:Rescheduling_interrupts
    553190 ± 16%    +768.5%    4804519 ± 13%  interrupts.CPU67.CAL:Function_call_interrupts
   2633697           -29.2%    1865007        interrupts.CPU67.LOC:Local_timer_interrupts
   1453325 ±  3%    +136.5%    3437075 ± 14%  interrupts.CPU67.RES:Rescheduling_interrupts
    552397 ± 16%    +764.3%    4774301 ± 26%  interrupts.CPU68.CAL:Function_call_interrupts
   2633789           -29.2%    1864964        interrupts.CPU68.LOC:Local_timer_interrupts
   1558186 ±  4%    +118.4%    3403031 ± 11%  interrupts.CPU68.RES:Rescheduling_interrupts
    554418 ± 16%    +961.7%    5886167 ±  8%  interrupts.CPU69.CAL:Function_call_interrupts
   2633764           -29.2%    1865051        interrupts.CPU69.LOC:Local_timer_interrupts
   1423294 ±  3%    +138.4%    3393400 ±  6%  interrupts.CPU69.RES:Rescheduling_interrupts
    541109 ± 11%    +668.2%    4156638 ± 44%  interrupts.CPU7.CAL:Function_call_interrupts
   2633836           -29.2%    1865155        interrupts.CPU7.LOC:Local_timer_interrupts
   1412135 ± 12%    +155.1%    3601951 ± 11%  interrupts.CPU7.RES:Rescheduling_interrupts
    550414 ± 16%    +834.5%    5143694 ± 13%  interrupts.CPU70.CAL:Function_call_interrupts
   2633653           -29.2%    1865092        interrupts.CPU70.LOC:Local_timer_interrupts
   1384755 ±  6%    +138.9%    3308775 ±  2%  interrupts.CPU70.RES:Rescheduling_interrupts
    559074 ± 16%    +653.3%    4211561 ± 13%  interrupts.CPU71.CAL:Function_call_interrupts
   2633748           -29.2%    1865021        interrupts.CPU71.LOC:Local_timer_interrupts
   1472210 ±  3%    +124.4%    3303060 ±  9%  interrupts.CPU71.RES:Rescheduling_interrupts
    542050 ± 11%   +1099.4%    6501414 ± 55%  interrupts.CPU72.CAL:Function_call_interrupts
   2633763           -29.2%    1865050        interrupts.CPU72.LOC:Local_timer_interrupts
   1367191 ± 16%     +80.6%    2469066 ±  9%  interrupts.CPU72.RES:Rescheduling_interrupts
    543872 ± 12%    +576.7%    3680170 ± 50%  interrupts.CPU73.CAL:Function_call_interrupts
   2633769           -29.2%    1865189        interrupts.CPU73.LOC:Local_timer_interrupts
   1409409 ± 12%    +150.9%    3536397 ± 10%  interrupts.CPU73.RES:Rescheduling_interrupts
    542479 ± 12%    +694.7%    4310839 ± 49%  interrupts.CPU74.CAL:Function_call_interrupts
   2633797           -29.2%    1865184        interrupts.CPU74.LOC:Local_timer_interrupts
   1345440 ±  8%    +148.6%    3345236 ± 13%  interrupts.CPU74.RES:Rescheduling_interrupts
    540242 ± 13%    +495.0%    3214652 ± 37%  interrupts.CPU75.CAL:Function_call_interrupts
   2633779           -29.2%    1865130        interrupts.CPU75.LOC:Local_timer_interrupts
   1341464 ± 11%    +177.8%    3726421 ± 18%  interrupts.CPU75.RES:Rescheduling_interrupts
    541422 ± 12%    +607.3%    3829357 ± 39%  interrupts.CPU76.CAL:Function_call_interrupts
   2633800           -29.2%    1865089        interrupts.CPU76.LOC:Local_timer_interrupts
   1366092 ± 12%    +155.4%    3489422 ± 19%  interrupts.CPU76.RES:Rescheduling_interrupts
    536776 ± 13%    +740.7%    4512417 ± 65%  interrupts.CPU77.CAL:Function_call_interrupts
   2633767           -29.2%    1865051        interrupts.CPU77.LOC:Local_timer_interrupts
   1368135 ± 11%    +150.2%    3423438 ± 11%  interrupts.CPU77.RES:Rescheduling_interrupts
    542490 ± 13%    +543.1%    3489024 ± 46%  interrupts.CPU78.CAL:Function_call_interrupts
   2633796           -29.2%    1865113        interrupts.CPU78.LOC:Local_timer_interrupts
   1398379 ± 15%    +158.6%    3615932 ± 13%  interrupts.CPU78.RES:Rescheduling_interrupts
    541330 ± 12%    +613.6%    3863188 ± 49%  interrupts.CPU79.CAL:Function_call_interrupts
   2633706           -29.2%    1865314        interrupts.CPU79.LOC:Local_timer_interrupts
   1345617 ± 11%    +166.4%    3584373 ± 16%  interrupts.CPU79.RES:Rescheduling_interrupts
    544923 ± 12%    +637.9%    4020972 ± 45%  interrupts.CPU8.CAL:Function_call_interrupts
   2633804           -29.2%    1865185        interrupts.CPU8.LOC:Local_timer_interrupts
   1458600 ± 12%    +144.6%    3567390 ± 10%  interrupts.CPU8.RES:Rescheduling_interrupts
    545648 ± 11%    +674.3%    4224816 ± 47%  interrupts.CPU80.CAL:Function_call_interrupts
   2633766           -29.2%    1864933        interrupts.CPU80.LOC:Local_timer_interrupts
   1378101 ± 12%    +169.8%    3718119 ±  6%  interrupts.CPU80.RES:Rescheduling_interrupts
    535168 ± 13%    +665.4%    4096010 ± 49%  interrupts.CPU81.CAL:Function_call_interrupts
   2633793           -29.2%    1865047        interrupts.CPU81.LOC:Local_timer_interrupts
   1368252 ± 13%    +163.1%    3599458 ± 10%  interrupts.CPU81.RES:Rescheduling_interrupts
    540163 ± 12%    +653.4%    4069365 ± 52%  interrupts.CPU82.CAL:Function_call_interrupts
   2633657           -29.2%    1865212        interrupts.CPU82.LOC:Local_timer_interrupts
   1401643 ± 13%    +176.1%    3869805 ± 11%  interrupts.CPU82.RES:Rescheduling_interrupts
    534505 ± 12%    +630.2%    3902850 ± 54%  interrupts.CPU83.CAL:Function_call_interrupts
   2633823           -29.2%    1865163        interrupts.CPU83.LOC:Local_timer_interrupts
   1384886 ± 14%    +167.7%    3707213 ± 17%  interrupts.CPU83.RES:Rescheduling_interrupts
    537983 ± 13%    +532.0%    3399977 ± 54%  interrupts.CPU84.CAL:Function_call_interrupts
   2633802           -29.2%    1865219        interrupts.CPU84.LOC:Local_timer_interrupts
   1396334 ± 15%    +167.1%    3729434 ± 25%  interrupts.CPU84.RES:Rescheduling_interrupts
    539715 ± 12%    +735.9%    4511522 ± 45%  interrupts.CPU85.CAL:Function_call_interrupts
   2633658           -29.2%    1865196        interrupts.CPU85.LOC:Local_timer_interrupts
   1351518 ± 12%    +152.8%    3416707 ±  8%  interrupts.CPU85.RES:Rescheduling_interrupts
    543234 ± 13%    +591.1%    3754561 ± 44%  interrupts.CPU86.CAL:Function_call_interrupts
   2633610           -29.2%    1865113        interrupts.CPU86.LOC:Local_timer_interrupts
   1362882 ± 15%    +170.3%    3684510 ± 13%  interrupts.CPU86.RES:Rescheduling_interrupts
    542751 ± 12%    +543.7%    3493567 ± 46%  interrupts.CPU87.CAL:Function_call_interrupts
   2633778           -29.2%    1865136        interrupts.CPU87.LOC:Local_timer_interrupts
   1363815 ± 13%    +157.8%    3515677 ± 21%  interrupts.CPU87.RES:Rescheduling_interrupts
    540758 ± 12%    +692.6%    4286070 ± 47%  interrupts.CPU88.CAL:Function_call_interrupts
   2633713           -29.2%    1865126        interrupts.CPU88.LOC:Local_timer_interrupts
   1386349 ± 15%    +154.7%    3531157 ± 18%  interrupts.CPU88.RES:Rescheduling_interrupts
    543290 ± 12%    +627.0%    3949703 ± 50%  interrupts.CPU89.CAL:Function_call_interrupts
   2633805           -29.2%    1864993        interrupts.CPU89.LOC:Local_timer_interrupts
   1391863 ± 16%    +161.4%    3638348 ±  9%  interrupts.CPU89.RES:Rescheduling_interrupts
    535660 ± 12%    +693.4%    4250141 ± 51%  interrupts.CPU9.CAL:Function_call_interrupts
   2633794           -29.2%    1865192        interrupts.CPU9.LOC:Local_timer_interrupts
   1430205 ± 18%    +138.5%    3411425 ± 13%  interrupts.CPU9.RES:Rescheduling_interrupts
    592065 ± 17%    +315.3%    2458617 ±103%  interrupts.CPU90.CAL:Function_call_interrupts
   2633796           -29.2%    1865066        interrupts.CPU90.LOC:Local_timer_interrupts
   1374748 ± 10%    +101.9%    2775876 ±  9%  interrupts.CPU90.RES:Rescheduling_interrupts
    583740 ± 15%    +195.1%    1722772 ± 71%  interrupts.CPU91.CAL:Function_call_interrupts
   2633827           -29.2%    1865115        interrupts.CPU91.LOC:Local_timer_interrupts
   1423080 ±  7%    +111.0%    3002050 ±  8%  interrupts.CPU91.RES:Rescheduling_interrupts
    578602 ± 17%    +178.2%    1609750 ± 63%  interrupts.CPU92.CAL:Function_call_interrupts
   2633804           -29.2%    1865110        interrupts.CPU92.LOC:Local_timer_interrupts
   1551851 ±  6%    +100.2%    3106189 ± 21%  interrupts.CPU92.RES:Rescheduling_interrupts
    577351 ± 17%    +190.8%    1678753 ± 53%  interrupts.CPU93.CAL:Function_call_interrupts
   2633846           -29.2%    1864789        interrupts.CPU93.LOC:Local_timer_interrupts
   1423206 ±  8%    +123.2%    3175923 ± 15%  interrupts.CPU93.RES:Rescheduling_interrupts
    570143 ± 17%    +244.0%    1961550 ± 91%  interrupts.CPU94.CAL:Function_call_interrupts
   2633853           -29.2%    1865127        interrupts.CPU94.LOC:Local_timer_interrupts
   1429404 ±  2%    +111.9%    3028819 ± 19%  interrupts.CPU94.RES:Rescheduling_interrupts
    572010 ± 15%    +212.9%    1789889 ± 88%  interrupts.CPU95.CAL:Function_call_interrupts
   2633753           -29.2%    1865079        interrupts.CPU95.LOC:Local_timer_interrupts
   1509973 ±  7%     +97.0%    2974377 ± 10%  interrupts.CPU95.RES:Rescheduling_interrupts
    574571 ± 17%    +244.6%    1979836 ± 96%  interrupts.CPU96.CAL:Function_call_interrupts
   2633812           -29.2%    1864939        interrupts.CPU96.LOC:Local_timer_interrupts
   1439328 ±  7%     +73.1%    2490884 ± 15%  interrupts.CPU96.RES:Rescheduling_interrupts
    575903 ± 15%    +211.7%    1794999 ± 62%  interrupts.CPU97.CAL:Function_call_interrupts
   2633783           -29.2%    1865089        interrupts.CPU97.LOC:Local_timer_interrupts
   1435933 ±  9%     +97.1%    2830079 ± 18%  interrupts.CPU97.RES:Rescheduling_interrupts
    576264 ± 16%    +242.8%    1975455 ± 67%  interrupts.CPU98.CAL:Function_call_interrupts
   2633879           -29.2%    1865106        interrupts.CPU98.LOC:Local_timer_interrupts
   1424628 ±  8%    +113.2%    3037568 ± 14%  interrupts.CPU98.RES:Rescheduling_interrupts
    566075 ± 17%    +271.2%    2101485 ± 88%  interrupts.CPU99.CAL:Function_call_interrupts
   2633879           -29.2%    1865196        interrupts.CPU99.LOC:Local_timer_interrupts
   1420395 ±  7%    +124.9%    3195087 ± 12%  interrupts.CPU99.RES:Rescheduling_interrupts
 3.793e+08           -29.2%  2.686e+08        interrupts.LOC:Local_timer_interrupts
    576.00           -43.8%     324.00 ± 19%  interrupts.MCP:Machine_check_polls
 2.074e+08 ±  6%    +138.3%  4.943e+08        interrupts.RES:Rescheduling_interrupts


                                                                                
                                hackbench.throughput                            
                                                                                
  85000 +-------------------------------------------------------------------+   
        |               O                                                   |   
  80000 |-+                                                                 |   
  75000 |-+O                         O O  O  O    O    O    O               |   
        |       O O  O    O  O  O O            O    O     O                 |   
  70000 |-+  O                                                              |   
        |                                                                   |   
  65000 |-+                                                                 |   
        |                                                                   |   
  60000 |-+                                                                 |   
  55000 |-+                                                                 |   
        |.. .+..            .+..            .+.  .+.     .+.          .+.   |   
  50000 |-++    +.+..+..+.+.    +.+..+.+..+.   +.   +..+.   +..+..+.+.   +..|   
        |                                                                   |   
  45000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  20 +----------------------------------------------------------------------+   
  18 |-+                                                            +       |   
     |                                                             : :      |   
  16 |-+                                                           : :      |   
  14 |..        +..           +                   .+              :   :     |   
     |  +.+..  +       .+.. .. :           +..   +  :       +    +    :     |   
  12 |-+      +    +.+.    +   :   +..+.. +    ..   :     .. + ..      +.+..|   
  10 |-+     +                  :..      +    +      :   +    +             |   
   8 |-+                        +                    : ..                   |   
     |                                                +                     |   
   6 |-+                                                                    |   
   4 |-+                                                                    |   
     |                                                                      |   
   2 |-+  O     O  O O  O          O     O       O                          |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  45 +----------------------------------------------------------------------+   
     |          +             +                                     +       |   
  40 |-+       : :           +:                   .+               + +      |   
  35 |..+.     :  :     +.. +  :           +     +  :       +.    +   +     |   
     |    +.. :   :   ..   +   :           ::   :   :      +  +..+     +.+..|   
  30 |-+      :    +.+          : .+..    :  :  :    :    +                 |   
  25 |-+     +                  +.    +.. :  : :     :  .+                  |   
     |                                   +    +       +.                    |   
  20 |-+                                                                    |   
  15 |-+                                                                    |   
     |                                                                      |   
  10 |-+                                                                    |   
   5 |-+                                                                    |   
     |          O  O O  O                     O  O                          |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  60000 +-------------------------------------------------------------------+   
        |                 +..+       +                 +                    |   
  55000 |-+  +    +       :   :     ::                 ::                   |   
  50000 |-+  ::   ::     :    :     : :               : :                   |   
        |   : :  : :     :     :   :  :               :  :                  |   
  45000 |-+ :  : :  :    :     :   :  :              :   :                  |   
  40000 |..:   ::   :   :       :.+    : .+.. .+..   :    :.+..             |   
        |  +    +    +..+       +      +.    +    +.+     +    +..+.+..+.+..|   
  35000 |-+                                                                 |   
  30000 |-+                                                                 |   
        |                                                                   |   
  25000 |-+O O    O       O  O       O    O            O                    |   
  20000 |-+     O    O            O    O     O O  O O     O                 |   
        |               O       O                           O               |   
  15000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  45000 +-------------------------------------------------------------------+   
        |                           ::                 +                    |   
  40000 |-+  +               +      : :               : :                   |   
  35000 |-+ : +            .. :    :  :               :  :             +    |   
        |.. :  +    .+..+.+    :   :   :             :   :     +     .. +   |   
  30000 |-++    +.+.           : .+    +..         .+     +. .. +   +    +..|   
  25000 |-+                     +         +..+.+..+         +    + +        |   
        |                                                         +         |   
  20000 |-+                                                                 |   
  15000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
   5000 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             hackbench.time.user_time                           
                                                                                
  3200 +--------------------------------------------------------------------+   
       |                                                                    |   
  3000 |-+  O     O         O                               O               |   
  2800 |-+O                    O  O    O    O  O    O  O O                  |   
       |            O  O  O         O     O      O                          |   
  2600 |-+                                                                  |   
       |                                                                    |   
  2400 |-+                                                                  |   
       |                                                                    |   
  2200 |-+    .+..+       +                                                 |   
  2000 |-+ .+.     +     + +  .+..+.+..+..+.  .+.                           |   
       |..+         +.. +   +.              +.   +..+..         .+..        |   
  1800 |-+             +                               +.+..+..+    +.. .+..|   
       |                                                               +    |   
  1600 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             hackbench.time.system_time                         
                                                                                
  200000 +------------------------------------------------------------------+   
         |                                                                  |   
  190000 |-+    .+.+..+.  .+..    .+.     .+.  .+.    .+..                 .|   
  180000 |..+.+.        +.    +.+.   +..+.   +.   +..+    +..+.+..+.+..+.+. |   
         |                                                                  |   
  170000 |-+                                                                |   
         |                                                                  |   
  160000 |-+                                                                |   
         |                                                                  |   
  150000 |-+                                                                |   
  140000 |-+                                                                |   
         |                                                                  |   
  130000 |-+  O  O O       O  O O  O            O    O                      |   
         |  O         O              O  O  O O    O    O  O  O              |   
  120000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            hackbench.time.elapsed_time                         
                                                                                
  1400 +--------------------------------------------------------------------+   
  1350 |-+       .+.                                                       .|   
       | .+.  .+.   +..+..+.  .+..+.  .+..+.  .+.  .+..+.  .+..+.+..+.. .+. |   
  1300 |.+  +.              +.      +.      +.   +.      +.            +    |   
  1250 |-+                                                                  |   
  1200 |-+                                                                  |   
  1150 |-+                                                                  |   
       |                                                                    |   
  1100 |-+                                                                  |   
  1050 |-+                                                                  |   
  1000 |-+                                                                  |   
   950 |-+  O  O  O         O  O  O            O    O                       |   
       |  O         O     O         O     O      O     O O  O               |   
   900 |-+             O               O    O                               |   
   850 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          hackbench.time.elapsed_time.max                       
                                                                                
  1400 +--------------------------------------------------------------------+   
  1350 |-+       .+.                                                       .|   
       | .+.  .+.   +..+..+.  .+..+.  .+..+.  .+.  .+..+.  .+..+.+..+.. .+. |   
  1300 |.+  +.              +.      +.      +.   +.      +.            +    |   
  1250 |-+                                                                  |   
  1200 |-+                                                                  |   
  1150 |-+                                                                  |   
       |                                                                    |   
  1100 |-+                                                                  |   
  1050 |-+                                                                  |   
  1000 |-+                                                                  |   
   950 |-+  O  O  O         O  O  O            O    O                       |   
       |  O         O     O         O     O      O     O O  O               |   
   900 |-+             O               O    O                               |   
   850 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       hackbench.time.voluntary_context_switches                
                                                                                
    6e+09 +-----------------------------------------------------------------+   
  5.5e+09 |-+       O    O       O O    O     O      O  O O  O              |   
          |  O O  O    O    O         O    O    O  O                        |   
    5e+09 |-+                                                               |   
  4.5e+09 |-+                                                               |   
          |                                                                 |   
    4e+09 |-+                                                               |   
  3.5e+09 |-+                                                               |   
    3e+09 |-+                                                               |   
          |                                                                 |   
  2.5e+09 |-+                                                               |   
    2e+09 |-+                                                               |   
          |                                                                 |   
  1.5e+09 |-+ .+..+.+..+.+..+.+..+.+..+.+..+..+.+..+.+..+.    .+..+.+.. .+..|   
    1e+09 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      hackbench.time.involuntary_context_switches               
                                                                                
  4.5e+09 +-----------------------------------------------------------------+   
          |              O    O    O                                        |   
    4e+09 |-+     O O  O         O      O     O      O  O O  O              |   
  3.5e+09 |-+O O            O         O    O    O  O                        |   
          |                                                                 |   
    3e+09 |-+                                                               |   
  2.5e+09 |-+                                                               |   
          |                                                                 |   
    2e+09 |-+                                                               |   
  1.5e+09 |-+                                                               |   
          |                                                                 |   
    1e+09 |-+                                                               |   
    5e+08 |..+.+..+.+..+.+..+.+..+.+..+.+..+..+.+..+.+..+.+..+.+..+.+..+.+..|   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp5: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
  memory/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/stress-ng/10s/0x5003003

commit: 
  d3921cb8be ("mm: fix initialization of struct page for holes in memory layout")
  3de7d4f25a ("mm: memcg/slab: optimize objcg stock draining")

d3921cb8be29ce56 3de7d4f25a7438f09fef4e71ef1 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4            1%           1:4     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    415967           +14.8%     477538        stress-ng.bad-altstack.ops
     41593           +14.8%      47751        stress-ng.bad-altstack.ops_per_sec
      1647            -2.3%       1609        boot-time.idle
      0.61 ±  2%      -0.2        0.45 ±  3%  mpstat.cpu.all.soft%
     73.25 ± 35%     -40.3%      43.75 ± 34%  numa-numastat.node0.interleave_hit
  25486494            +3.8%   26458177        proc-vmstat.pgactivate
     19886 ± 24%     +89.8%      37737 ± 26%  sched_debug.cpu.curr->pid.min
     32.22 ±  3%      -3.4%      31.13        perf-stat.i.MPKI
    208532            -8.0%     191893 ±  2%  perf-stat.i.cycles-between-cache-misses
     12514 ±  8%     -14.5%      10704 ±  6%  numa-meminfo.node0.KernelStack
     20560 ± 29%     -37.4%      12871 ± 17%  numa-meminfo.node0.PageTables
    654926 ±  5%     +11.3%     728915 ±  3%  numa-meminfo.node1.AnonHugePages
    723641 ±  8%     +16.8%     845066 ±  3%  numa-meminfo.node1.AnonPages
    745431 ±  7%     +15.6%     861404 ±  3%  numa-meminfo.node1.Inactive
    745431 ±  7%     +15.6%     861404 ±  3%  numa-meminfo.node1.Inactive(anon)
     10918 ±  7%     +15.1%      12569 ±  8%  numa-meminfo.node1.KernelStack
    120703 ±  3%      +9.4%     132082 ±  4%  slabinfo.kmalloc-32.active_objs
    944.25 ±  3%      +9.5%       1034 ±  4%  slabinfo.kmalloc-32.active_slabs
    120960 ±  3%      +9.5%     132447 ±  4%  slabinfo.kmalloc-32.num_objs
    944.25 ±  3%      +9.5%       1034 ±  4%  slabinfo.kmalloc-32.num_slabs
    166919 ±  4%     +13.1%     188713 ±  4%  slabinfo.kmalloc-8.active_objs
    167749 ±  3%     +13.4%     190188 ±  4%  slabinfo.kmalloc-8.num_objs
     48373 ±  4%      -7.1%      44932        slabinfo.lsm_file_cache.active_objs
     48373 ±  4%      -7.1%      44932        slabinfo.lsm_file_cache.num_objs
    206078 ±  6%     -15.2%     174793 ±  5%  numa-vmstat.node0.nr_anon_pages
    209130 ±  6%     -14.5%     178872 ±  5%  numa-vmstat.node0.nr_inactive_anon
    666.50 ±  5%     +15.3%     768.25 ± 11%  numa-vmstat.node0.nr_isolated_anon
     12516 ±  7%     -14.6%      10689 ±  6%  numa-vmstat.node0.nr_kernel_stack
      5136 ± 27%     -37.5%       3211 ± 17%  numa-vmstat.node0.nr_page_table_pages
    209124 ±  6%     -14.5%     178873 ±  5%  numa-vmstat.node0.nr_zone_inactive_anon
    186466 ±  7%     +13.4%     211536 ±  4%  numa-vmstat.node1.nr_anon_pages
    330.00 ±  6%      +8.5%     358.00 ±  3%  numa-vmstat.node1.nr_anon_transparent_hugepages
    191216 ±  6%     +12.7%     215422 ±  3%  numa-vmstat.node1.nr_inactive_anon
    191212 ±  6%     +12.7%     215425 ±  3%  numa-vmstat.node1.nr_zone_inactive_anon
      1.24 ±160%     -99.0%       0.01 ± 56%  perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.61 ±173%    +409.2%       3.11 ± 18%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.02 ± 15%     +30.2%       0.02 ± 19%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.04 ± 50%     -53.5%       0.02 ±  2%  perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      9.12 ± 64%     -94.1%       0.54 ± 80%  perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1.08 ±173%    +611.4%       7.68 ± 19%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     61.86 ±  5%     +16.4%      72.02 ±  7%  perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    475.75 ± 45%     -47.5%     250.00 ± 52%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
     61.77 ±  5%     +16.6%      72.01 ±  7%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.05 ± 12%     +26.2%       0.06 ± 10%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
      0.03 ± 62%     -46.3%       0.02 ±108%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.__put_anon_vma.unlink_anon_vmas.free_pgtables
      0.12 ± 40%    +379.0%       0.55 ± 62%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
      0.03 ± 62%     -46.3%       0.02 ±108%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.__put_anon_vma.unlink_anon_vmas.free_pgtables
    103405 ± 25%     -19.2%      83544 ±  2%  interrupts.CPU1.CAL:Function_call_interrupts
      1027 ± 26%     -28.8%     731.50 ± 33%  interrupts.CPU21.TLB:TLB_shootdowns
    918.75 ± 32%     +49.1%       1370 ± 18%  interrupts.CPU24.TLB:TLB_shootdowns
      9601 ±  2%     -15.1%       8152 ±  9%  interrupts.CPU26.RES:Rescheduling_interrupts
    136411 ± 51%     -38.2%      84339 ±  4%  interrupts.CPU27.CAL:Function_call_interrupts
     92692 ±  3%      -9.2%      84205 ±  4%  interrupts.CPU30.CAL:Function_call_interrupts
      1291 ± 33%     -43.6%     727.75 ± 37%  interrupts.CPU31.TLB:TLB_shootdowns
     96125 ± 12%     -13.6%      83073        interrupts.CPU35.CAL:Function_call_interrupts
    101106 ± 17%     -15.5%      85423 ±  2%  interrupts.CPU47.CAL:Function_call_interrupts
     11054 ± 20%     -30.0%       7735 ± 17%  interrupts.CPU47.RES:Rescheduling_interrupts
     87756 ± 10%     +27.4%     111827 ± 24%  interrupts.CPU48.CAL:Function_call_interrupts
     10119 ±  6%     -17.0%       8402 ± 12%  interrupts.CPU50.RES:Rescheduling_interrupts
     86234 ±  7%     +39.2%     120069 ± 30%  interrupts.CPU66.CAL:Function_call_interrupts
    104978           -12.7%      91675 ±  3%  interrupts.CPU73.CAL:Function_call_interrupts
      9451 ±  5%     -16.4%       7904 ± 10%  interrupts.CPU75.RES:Rescheduling_interrupts
      4681 ± 35%     +42.0%       6646 ± 24%  interrupts.CPU82.NMI:Non-maskable_interrupts
      4681 ± 35%     +42.0%       6646 ± 24%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
    141092 ± 53%     -35.1%      91507        interrupts.CPU83.CAL:Function_call_interrupts
    932.25 ± 32%     -27.4%     676.75 ± 41%  interrupts.CPU85.TLB:TLB_shootdowns
    111069 ± 13%     -13.3%      96344 ±  7%  interrupts.CPU92.CAL:Function_call_interrupts
    110115 ± 12%     -15.7%      92791 ±  2%  interrupts.CPU93.CAL:Function_call_interrupts
      3.62            -1.2        2.40 ±  3%  perf-profile.calltrace.cycles-pp.kmem_cache_free.remove_vma.exit_mmap.mmput.do_exit
      4.09            -1.2        2.88 ±  3%  perf-profile.calltrace.cycles-pp.remove_vma.exit_mmap.mmput.do_exit.do_group_exit
      3.62 ±  2%      -1.0        2.63 ±  3%  perf-profile.calltrace.cycles-pp.kmem_cache_free.unlink_anon_vmas.free_pgtables.exit_mmap.mmput
      1.67            -0.9        0.75 ±  3%  perf-profile.calltrace.cycles-pp.refill_obj_stock.kmem_cache_free.remove_vma.exit_mmap.mmput
      3.79 ±  2%      -0.9        2.88 ±  5%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.vm_area_dup.dup_mmap.dup_mm.copy_process
      4.29 ±  2%      -0.8        3.47 ±  4%  perf-profile.calltrace.cycles-pp.vm_area_dup.dup_mmap.dup_mm.copy_process.kernel_clone
      2.69 ±  2%      -0.8        1.92 ±  6%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm
      2.45 ±  2%      -0.6        1.85 ±  5%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.anon_vma_fork.dup_mmap.dup_mm.copy_process
      2.25 ± 10%      -0.4        1.81 ±  8%  perf-profile.calltrace.cycles-pp.ret_from_fork
      1.67 ±  3%      -0.4        1.26 ±  7%  perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.kmem_cache_alloc.vm_area_dup.dup_mmap.dup_mm
      1.33 ±  3%      -0.4        0.94 ±  3%  perf-profile.calltrace.cycles-pp.kmem_cache_free.__put_anon_vma.unlink_anon_vmas.free_pgtables.exit_mmap
      1.29 ±  3%      -0.4        0.92 ±  7%  perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.kmem_cache_alloc.anon_vma_clone.anon_vma_fork.dup_mmap
      1.03 ± 12%      -0.2        0.78 ± 10%  perf-profile.calltrace.cycles-pp.__memcg_kmem_charge.__memcg_kmem_charge_page.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc
      1.10 ± 11%      -0.2        0.86 ±  9%  perf-profile.calltrace.cycles-pp.__memcg_kmem_charge_page.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc.copy_pte_range
      0.90 ± 13%      -0.2        0.67 ± 11%  perf-profile.calltrace.cycles-pp.page_counter_try_charge.__memcg_kmem_charge.__memcg_kmem_charge_page.__alloc_pages_nodemask.pte_alloc_one
      0.88 ±  4%      -0.2        0.66 ±  5%  perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.kmem_cache_alloc.anon_vma_fork.dup_mmap.dup_mm
      1.27 ±  2%      -0.1        1.16 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.anon_vma_clone.anon_vma_fork.dup_mmap
      1.18 ±  2%      -0.1        1.06 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.unlink_anon_vmas.free_pgtables.exit_mmap
      1.73 ±  3%      -0.1        1.62 ±  3%  perf-profile.calltrace.cycles-pp.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.75 ±  3%      -0.1        1.64 ±  3%  perf-profile.calltrace.cycles-pp.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.75 ±  3%      -0.1        1.65 ±  3%  perf-profile.calltrace.cycles-pp.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.68            +0.0        0.70 ±  2%  perf-profile.calltrace.cycles-pp.wp_page_copy.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      0.52 ±  2%      +0.1        0.60 ±  5%  perf-profile.calltrace.cycles-pp.free_pages_and_swap_cache.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput
      0.61 ±  6%      +0.1        0.70 ±  3%  perf-profile.calltrace.cycles-pp.alloc_set_pte.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault
      0.60 ±  3%      +0.1        0.70 ±  2%  perf-profile.calltrace.cycles-pp.anon_vma_interval_tree_remove.unlink_anon_vmas.free_pgtables.exit_mmap.mmput
      0.63 ±  2%      +0.1        0.74        perf-profile.calltrace.cycles-pp.wake_up_new_task.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.98            +0.3        2.32 ±  2%  perf-profile.calltrace.cycles-pp.anon_vma_interval_tree_insert.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm
      2.25            +0.4        2.60 ±  3%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput
      8.22            +0.4        8.66 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.anon_vma_fork.dup_mmap.dup_mm.copy_process
      2.80            +0.4        3.24 ±  3%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit
      6.52 ±  2%      +0.4        6.96 ±  3%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.anon_vma_fork.dup_mmap.dup_mm
      2.79            +0.4        3.23 ±  3%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit
      7.79 ±  2%      +0.5        8.25 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.unlink_anon_vmas.free_pgtables.exit_mmap.mmput
      0.12 ±173%      +0.5        0.58        perf-profile.calltrace.cycles-pp.select_task_rq_fair.wake_up_new_task.kernel_clone.__do_sys_clone.do_syscall_64
      6.35 ±  3%      +0.5        6.89 ±  3%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.unlink_anon_vmas.free_pgtables.exit_mmap
     10.69            +0.7       11.38        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     10.55            +0.7       11.25        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     10.55            +0.7       11.25        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     10.55            +0.7       11.25        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
      9.32            +0.9       10.26        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      9.66            +1.0       10.66        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      9.66            +1.0       10.68        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     11.08            -3.3        7.78 ±  2%  perf-profile.children.cycles-pp.kmem_cache_free
      4.26            -3.2        1.10 ±  2%  perf-profile.children.cycles-pp.page_counter_uncharge
      4.06            -3.0        1.09 ±  2%  perf-profile.children.cycles-pp.page_counter_cancel
      4.87            -2.5        2.34        perf-profile.children.cycles-pp.refill_obj_stock
     10.21            -2.5        7.69 ±  5%  perf-profile.children.cycles-pp.kmem_cache_alloc
      3.82            -2.5        1.36        perf-profile.children.cycles-pp.drain_obj_stock
      2.42 ±  2%      -1.5        0.89 ±  3%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge
      4.11            -1.2        2.90 ±  3%  perf-profile.children.cycles-pp.remove_vma
      4.33 ±  3%      -1.1        3.26 ±  6%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      1.50 ±  6%      -1.0        0.54 ± 11%  perf-profile.children.cycles-pp.propagate_protected_usage
      4.29 ±  2%      -0.8        3.47 ±  4%  perf-profile.children.cycles-pp.vm_area_dup
      2.79 ±  8%      -0.8        1.98 ± 10%  perf-profile.children.cycles-pp.page_counter_try_charge
      2.93 ±  8%      -0.8        2.14 ±  9%  perf-profile.children.cycles-pp.__memcg_kmem_charge
      1.20            -0.6        0.55 ±  2%  perf-profile.children.cycles-pp.finish_task_switch
      0.89            -0.6        0.27 ±  4%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge_page
      2.12 ±  9%      -0.5        1.58 ± 10%  perf-profile.children.cycles-pp.__memcg_kmem_charge_page
      0.76            -0.5        0.28 ±  4%  perf-profile.children.cycles-pp.put_task_stack
      1.56 ±  2%      -0.5        1.09 ±  4%  perf-profile.children.cycles-pp.obj_cgroup_charge
      2.40 ±  2%      -0.5        1.93 ±  4%  perf-profile.children.cycles-pp.rcu_core
      2.33            -0.5        1.86 ±  4%  perf-profile.children.cycles-pp.rcu_do_batch
      1.22 ±  2%      -0.5        0.77 ±  2%  perf-profile.children.cycles-pp.__schedule
      2.26 ± 10%      -0.4        1.82 ±  8%  perf-profile.children.cycles-pp.ret_from_fork
      2.64 ±  3%      -0.4        2.21 ±  3%  perf-profile.children.cycles-pp.__softirqentry_text_start
      2.46 ±  3%      -0.4        2.07 ±  4%  perf-profile.children.cycles-pp.do_softirq_own_stack
      2.50 ±  3%      -0.4        2.12 ±  4%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.75            -0.4        0.38 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      4.36            -0.2        4.11        perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.43 ±  3%      -0.2        0.22 ±  3%  perf-profile.children.cycles-pp.__kmalloc_node
      0.70            -0.2        0.49 ±  2%  perf-profile.children.cycles-pp.task_work_run
      0.48            -0.2        0.28 ±  3%  perf-profile.children.cycles-pp.proc_reg_release
      0.62            -0.2        0.42 ±  3%  perf-profile.children.cycles-pp.__fput
      0.46            -0.2        0.26 ±  4%  perf-profile.children.cycles-pp.close_pdeo
      0.71            -0.2        0.51 ±  2%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.43            -0.2        0.23 ±  3%  perf-profile.children.cycles-pp.single_release
      0.59 ±  2%      -0.2        0.40 ±  2%  perf-profile.children.cycles-pp.proc_reg_read_iter
      0.54            -0.2        0.35 ±  3%  perf-profile.children.cycles-pp.kfree
      0.72 ±  2%      -0.2        0.53        perf-profile.children.cycles-pp.vfs_read
      0.73 ±  2%      -0.2        0.54        perf-profile.children.cycles-pp.ksys_read
      0.62 ±  2%      -0.2        0.44        perf-profile.children.cycles-pp.seq_read_iter
      0.63 ±  2%      -0.2        0.44 ±  2%  perf-profile.children.cycles-pp.new_sync_read
      0.42 ±  3%      -0.2        0.25 ±  4%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.62 ±  2%      -0.2        0.46 ±  2%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.35 ±  3%      -0.2        0.20 ±  2%  perf-profile.children.cycles-pp.uncharge_batch
      0.36 ±  4%      -0.1        0.21 ±  2%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.57            -0.1        0.42 ±  2%  perf-profile.children.cycles-pp.schedule_tail
      0.36 ±  8%      -0.1        0.24 ± 28%  perf-profile.children.cycles-pp.__pud_alloc
      0.26 ±  4%      -0.1        0.15 ±  2%  perf-profile.children.cycles-pp.__mmdrop
      0.23 ±  3%      -0.1        0.13        perf-profile.children.cycles-pp.__free_pages_ok
      0.36 ±  4%      -0.1        0.26 ±  3%  perf-profile.children.cycles-pp.try_charge
      1.76 ±  3%      -0.1        1.66 ±  2%  perf-profile.children.cycles-pp.kernel_wait4
      1.73 ±  3%      -0.1        1.64 ±  3%  perf-profile.children.cycles-pp.do_wait
      1.76 ±  3%      -0.1        1.66 ±  3%  perf-profile.children.cycles-pp.__do_sys_wait4
      0.37 ±  3%      -0.1        0.30 ±  3%  perf-profile.children.cycles-pp.schedule
      0.32 ±  2%      -0.1        0.25 ±  2%  perf-profile.children.cycles-pp.__put_task_struct
      1.05 ±  2%      -0.1        0.98 ±  2%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      0.21 ±  7%      -0.0        0.16 ±  2%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.19 ±  5%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.24 ±  5%      -0.0        0.20 ±  7%  perf-profile.children.cycles-pp.prepare_creds
      0.32            -0.0        0.28 ±  2%  perf-profile.children.cycles-pp.alloc_empty_file
      0.32            -0.0        0.28 ±  2%  perf-profile.children.cycles-pp.__alloc_file
      0.16 ±  5%      -0.0        0.13 ±  6%  perf-profile.children.cycles-pp.proc_reg_open
      0.21 ±  3%      -0.0        0.18 ±  6%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.24 ±  6%      -0.0        0.21 ±  3%  perf-profile.children.cycles-pp.lock_page_memcg
      0.11 ±  6%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__count_memcg_events
      0.15 ±  5%      -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.mem_cgroup_charge
      0.22 ±  5%      -0.0        0.19 ±  4%  perf-profile.children.cycles-pp.do_dentry_open
      0.30 ±  5%      -0.0        0.27 ±  2%  perf-profile.children.cycles-pp.__x64_sys_setpgid
      0.13 ±  5%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.copy_creds
      0.22            -0.0        0.20 ±  2%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.15            -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.proc_alloc_inode
      0.15 ±  7%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.15 ±  7%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.do_execveat_common
      0.15 ±  7%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.execve
      0.10 ±  7%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.exec_binprm
      0.16            -0.0        0.14 ±  3%  perf-profile.children.cycles-pp.__d_alloc
      0.12 ±  6%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.bprm_execve
      0.09 ±  4%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.single_open
      0.19            -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.d_alloc
      0.23            -0.0        0.21 ±  3%  perf-profile.children.cycles-pp.new_inode
      0.09 ±  4%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.load_elf_binary
      0.21 ±  2%      -0.0        0.19 ±  2%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.19 ±  2%      -0.0        0.18 ±  2%  perf-profile.children.cycles-pp.alloc_inode
      0.09 ±  4%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.__kmalloc
      0.10            -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.__libc_fork
      0.05            +0.0        0.06        perf-profile.children.cycles-pp.seq_vprintf
      0.07 ±  5%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.select_idle_sibling
      0.07            +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.vsnprintf
      0.06 ±  6%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp._find_next_bit
      0.18 ±  2%      +0.0        0.19        perf-profile.children.cycles-pp.___perf_sw_event
      0.10 ±  5%      +0.0        0.11        perf-profile.children.cycles-pp.alloc_pages_vma
      0.13            +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      0.09 ±  4%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.nr_running
      0.07            +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.08 ±  8%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.put_cpu_partial
      0.07 ±  7%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.perf_event_task
      0.06 ±  6%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.available_idle_cpu
      0.12 ±  3%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
      0.13 ±  3%      +0.0        0.15 ±  5%  perf-profile.children.cycles-pp.sched_move_task
      0.08 ±  5%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.unlock_page_memcg
      0.08 ±  5%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.arch_dup_task_struct
      0.20 ±  2%      +0.0        0.22        perf-profile.children.cycles-pp.__put_user_nocheck_4
      0.15 ±  2%      +0.0        0.17        perf-profile.children.cycles-pp.loadavg_proc_show
      0.14            +0.0        0.16 ±  5%  perf-profile.children.cycles-pp.__anon_vma_interval_tree_augment_rotate
      0.07 ±  5%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.__list_add_valid
      0.13 ±  6%      +0.0        0.15 ±  6%  perf-profile.children.cycles-pp.__setup_rt_frame
      0.14            +0.0        0.17 ±  3%  perf-profile.children.cycles-pp.lookup_fast
      0.10            +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.memcpy_erms
      0.11 ±  4%      +0.0        0.13 ± 16%  perf-profile.children.cycles-pp.update_rq_clock
      0.18 ±  6%      +0.0        0.21 ±  2%  perf-profile.children.cycles-pp.free_unref_page_list
      0.15 ±  7%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.17 ±  5%      +0.0        0.20 ±  2%  perf-profile.children.cycles-pp.unmap_single_vma
      0.14 ±  6%      +0.0        0.17 ±  6%  perf-profile.children.cycles-pp.allocate_slab
      0.06 ±  7%      +0.0        0.09 ± 20%  perf-profile.children.cycles-pp.menu_select
      0.76            +0.0        0.79        perf-profile.children.cycles-pp.wp_page_copy
      0.38 ±  4%      +0.0        0.41 ±  2%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.38 ±  3%      +0.0        0.41 ±  2%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.21 ±  3%      +0.0        0.24        perf-profile.children.cycles-pp.__wake_up_common_lock
      0.25            +0.0        0.28 ±  2%  perf-profile.children.cycles-pp.__rb_erase_color
      0.26            +0.0        0.29 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      0.20 ±  4%      +0.0        0.23        perf-profile.children.cycles-pp.__wake_up_common
      0.24 ±  4%      +0.0        0.28 ±  5%  perf-profile.children.cycles-pp.update_load_avg
      0.29 ±  2%      +0.0        0.33        perf-profile.children.cycles-pp.___might_sleep
      0.53 ±  3%      +0.0        0.57 ±  3%  perf-profile.children.cycles-pp.__slab_free
      0.21 ±  2%      +0.0        0.25        perf-profile.children.cycles-pp.do_notify_parent
      0.17 ±  8%      +0.0        0.21 ±  4%  perf-profile.children.cycles-pp.vma_interval_tree_insert_after
      0.25 ±  7%      +0.0        0.30 ±  4%  perf-profile.children.cycles-pp.___slab_alloc
      0.22 ±  3%      +0.0        0.27 ± 13%  perf-profile.children.cycles-pp.getname_flags
      0.27 ±  5%      +0.0        0.31 ±  4%  perf-profile.children.cycles-pp.__slab_alloc
      0.01 ±173%      +0.0        0.06        perf-profile.children.cycles-pp.free_one_page
      0.21 ±  3%      +0.0        0.26 ± 15%  perf-profile.children.cycles-pp.strncpy_from_user
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.29 ±  4%      +0.1        0.34        perf-profile.children.cycles-pp.__rb_insert_augmented
      0.33            +0.1        0.38        perf-profile.children.cycles-pp.dup_userfaultfd
      0.45 ±  6%      +0.1        0.51 ±  3%  perf-profile.children.cycles-pp.osq_unlock
      0.15 ±  3%      +0.1        0.20 ±  6%  perf-profile.children.cycles-pp.lock_page_lruvec_irqsave
      0.24 ±  4%      +0.1        0.30 ± 13%  perf-profile.children.cycles-pp.copy_page
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.refill_stock
      0.38 ±  7%      +0.1        0.44        perf-profile.children.cycles-pp.page_add_file_rmap
      0.38 ±  2%      +0.1        0.45        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.28 ±  4%      +0.1        0.35 ±  5%  perf-profile.children.cycles-pp.rmqueue
      0.51 ±  4%      +0.1        0.58 ±  5%  perf-profile.children.cycles-pp.clear_page_erms
      0.53 ±  3%      +0.1        0.61 ±  4%  perf-profile.children.cycles-pp.prep_new_page
      0.44            +0.1        0.52        perf-profile.children.cycles-pp.find_idlest_group
      0.53 ±  2%      +0.1        0.61 ±  5%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.38 ±  5%      +0.1        0.47 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.65 ±  5%      +0.1        0.75 ±  3%  perf-profile.children.cycles-pp.alloc_set_pte
      0.60 ±  3%      +0.1        0.71 ±  2%  perf-profile.children.cycles-pp.anon_vma_interval_tree_remove
      0.64            +0.1        0.75        perf-profile.children.cycles-pp.wake_up_new_task
      0.61            +0.1        0.73        perf-profile.children.cycles-pp.select_task_rq_fair
      1.12 ±  4%      +0.1        1.27 ±  4%  perf-profile.children.cycles-pp.up_write
      0.84 ±  3%      +0.2        1.00 ±  4%  perf-profile.children.cycles-pp.get_page_from_freelist
      1.59 ±  3%      +0.2        1.76 ±  2%  perf-profile.children.cycles-pp.down_write
      2.30            +0.4        2.66 ±  3%  perf-profile.children.cycles-pp.release_pages
      2.00            +0.4        2.35        perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      2.83            +0.4        3.27 ±  3%  perf-profile.children.cycles-pp.tlb_finish_mmu
      2.81            +0.4        3.25 ±  3%  perf-profile.children.cycles-pp.tlb_flush_mmu
     10.69            +0.7       11.38        perf-profile.children.cycles-pp.do_idle
     10.69            +0.7       11.38        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     10.69            +0.7       11.38        perf-profile.children.cycles-pp.cpu_startup_entry
     10.55            +0.7       11.25        perf-profile.children.cycles-pp.start_secondary
      9.44            +0.9       10.38        perf-profile.children.cycles-pp.intel_idle
      9.79            +1.0       10.80        perf-profile.children.cycles-pp.cpuidle_enter_state
      9.79            +1.0       10.80        perf-profile.children.cycles-pp.cpuidle_enter
     27.32            +1.8       29.12        perf-profile.children.cycles-pp.rwsem_down_write_slowpath
     21.75 ±  2%      +1.8       23.60        perf-profile.children.cycles-pp.osq_lock
      3.03            -2.2        0.83 ±  3%  perf-profile.self.cycles-pp.page_counter_cancel
      6.03 ±  2%      -1.2        4.85 ±  2%  perf-profile.self.cycles-pp.kmem_cache_free
      4.13            -1.2        2.96 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc
      4.16 ±  3%      -1.0        3.13 ±  5%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      1.47 ±  6%      -0.9        0.52 ± 11%  perf-profile.self.cycles-pp.propagate_protected_usage
      2.23 ±  8%      -0.6        1.64 ±  8%  perf-profile.self.cycles-pp.page_counter_try_charge
      4.21            -0.2        3.98        perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.41 ±  2%      -0.2        0.25 ±  4%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.75 ±  3%      -0.1        0.67 ±  3%  perf-profile.self.cycles-pp.drain_obj_stock
      0.11 ±  6%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.__count_memcg_events
      0.05 ±  8%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.copy_process
      0.05 ±  8%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.08 ±  5%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.memset_erms
      0.06 ±  6%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.__schedule
      0.15 ±  2%      +0.0        0.16 ±  2%  perf-profile.self.cycles-pp.___perf_sw_event
      0.06            +0.0        0.08 ±  6%  perf-profile.self.cycles-pp._find_next_bit
      0.09 ±  4%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.acct_collect
      0.07 ±  6%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.do_user_addr_fault
      0.12 ±  7%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.update_load_avg
      0.08 ±  5%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.unlock_page_memcg
      0.13 ±  3%      +0.0        0.15 ±  5%  perf-profile.self.cycles-pp.__anon_vma_interval_tree_augment_rotate
      0.07            +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.__list_add_valid
      0.06            +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.available_idle_cpu
      0.10 ±  4%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.memcpy_erms
      0.23 ±  4%      +0.0        0.26 ±  3%  perf-profile.self.cycles-pp.anon_vma_fork
      0.16 ±  5%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.unmap_single_vma
      0.15 ±  3%      +0.0        0.17 ±  6%  perf-profile.self.cycles-pp.copy_page_range
      0.38 ±  4%      +0.0        0.41 ±  2%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.38 ±  2%      +0.0        0.40 ±  2%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.29 ±  5%      +0.0        0.32 ±  2%  perf-profile.self.cycles-pp.fput_many
      0.28 ±  2%      +0.0        0.31 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      0.21            +0.0        0.24 ±  3%  perf-profile.self.cycles-pp.__rb_erase_color
      0.39 ±  2%      +0.0        0.43        perf-profile.self.cycles-pp.unlink_anon_vmas
      0.17 ±  9%      +0.0        0.21 ±  2%  perf-profile.self.cycles-pp.vma_interval_tree_insert_after
      0.51 ±  3%      +0.0        0.55 ±  2%  perf-profile.self.cycles-pp.__slab_free
      0.31 ±  2%      +0.1        0.36 ±  2%  perf-profile.self.cycles-pp.dup_userfaultfd
      0.28 ±  5%      +0.1        0.33        perf-profile.self.cycles-pp.__rb_insert_augmented
      0.29 ±  6%      +0.1        0.35 ±  3%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.43 ±  6%      +0.1        0.49 ±  4%  perf-profile.self.cycles-pp.osq_unlock
      0.23 ±  2%      +0.1        0.30 ± 12%  perf-profile.self.cycles-pp.copy_page
      0.72 ±  2%      +0.1        0.79        perf-profile.self.cycles-pp._raw_spin_lock
      0.37 ±  3%      +0.1        0.44        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.39            +0.1        0.46        perf-profile.self.cycles-pp.find_idlest_group
      0.48 ±  4%      +0.1        0.56 ±  5%  perf-profile.self.cycles-pp.clear_page_erms
      0.51 ±  2%      +0.1        0.58 ±  5%  perf-profile.self.cycles-pp.free_pages_and_swap_cache
      0.46 ±  3%      +0.1        0.55 ±  2%  perf-profile.self.cycles-pp.vm_area_dup
      0.59 ±  3%      +0.1        0.68 ±  2%  perf-profile.self.cycles-pp.anon_vma_interval_tree_remove
      0.68 ±  2%      +0.1        0.79 ±  3%  perf-profile.self.cycles-pp.dup_mmap
      0.47 ±  4%      +0.1        0.59 ±  4%  perf-profile.self.cycles-pp.anon_vma_clone
      1.09 ±  4%      +0.1        1.22 ±  4%  perf-profile.self.cycles-pp.up_write
      0.74 ±  7%      +0.1        0.88 ±  8%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      1.42 ±  3%      +0.2        1.57 ±  2%  perf-profile.self.cycles-pp.down_write
      1.22 ±  6%      +0.3        1.50 ± 10%  perf-profile.self.cycles-pp.copy_pte_range
      1.93            +0.3        2.26 ±  2%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      1.56 ±  2%      +0.4        1.96 ±  4%  perf-profile.self.cycles-pp.release_pages
      9.44            +0.9       10.38        perf-profile.self.cycles-pp.intel_idle
     21.21 ±  2%      +1.8       22.97        perf-profile.self.cycles-pp.osq_lock



***************************************************************************************************
lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
  filesystem/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/stress-ng/30s/0x5003003

commit: 
  d3921cb8be ("mm: fix initialization of struct page for holes in memory layout")
  3de7d4f25a ("mm: memcg/slab: optimize objcg stock draining")

d3921cb8be29ce56 3de7d4f25a7438f09fef4e71ef1 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     last_state.is_incomplete_run
         %stddev     %change         %stddev
             \          |                \  
    720028 ± 11%     +12.8%     812528 ±  8%  stress-ng.dir.ops
  10642917           +54.8%   16476318        stress-ng.eventfd.ops
    354761           +54.8%     549206        stress-ng.eventfd.ops_per_sec
      3797           +24.3%       4718 ± 17%  stress-ng.procfs.ops
    126.50           +24.2%     157.15 ± 17%  stress-ng.procfs.ops_per_sec
   1405697            +2.4%    1438971        stress-ng.time.involuntary_context_switches
   7720769 ±  2%      +4.8%    8092400 ±  3%  stress-ng.time.minor_page_faults
    579.27 ±  3%      +4.8%     606.97 ±  2%  stress-ng.time.user_time
  25871459           +46.0%   37766175        stress-ng.time.voluntary_context_switches
      0.26 ±  4%      -0.1        0.18 ±  5%  mpstat.cpu.all.soft%
      1072 ± 12%     +18.1%       1266        numa-meminfo.node0.Mlocked
     56223           +35.3%      76091        vmstat.system.cs
  15061939          +274.1%   56347492 ± 60%  cpuidle.C1.usage
  22928095          +163.3%   60363254 ± 55%  cpuidle.POLL.time
  10134542 ±  2%     +62.5%   16466628 ± 11%  cpuidle.POLL.usage
    267.33 ± 12%     +18.3%     316.25        numa-vmstat.node0.nr_mlock
     49203 ± 71%    +221.1%     158009 ± 35%  numa-vmstat.node0.numa_other
    207347 ± 17%     -52.4%      98738 ± 56%  numa-vmstat.node1.numa_other
     48870 ±  7%     +12.2%      54827 ±  6%  slabinfo.Acpi-Parse.active_objs
    670.67 ±  7%     +12.5%     754.50 ±  6%  slabinfo.Acpi-Parse.active_slabs
     48999 ±  7%     +12.5%      55103 ±  6%  slabinfo.Acpi-Parse.num_objs
    670.67 ±  7%     +12.5%     754.50 ±  6%  slabinfo.Acpi-Parse.num_slabs
  45113239            +4.8%   47271692        proc-vmstat.numa_hit
  45026191            +4.8%   47184765        proc-vmstat.numa_local
  61034072            +5.7%   64517404        proc-vmstat.pgalloc_normal
    271479 ±  2%      -5.2%     257449 ±  4%  proc-vmstat.pgdeactivate
  11329788            +2.7%   11638804 ±  2%  proc-vmstat.pgfault
  60815575            +5.7%   64301056        proc-vmstat.pgfree
    520143 ± 17%     +40.1%     728832 ±  9%  proc-vmstat.slabs_scanned
    167.77 ±  5%      +9.9%     184.37 ±  5%  sched_debug.cfs_rq:/.runnable_avg.avg
    166.98 ±  5%     +10.0%     183.72 ±  5%  sched_debug.cfs_rq:/.util_avg.avg
     79.75 ±  6%     +16.7%      93.08 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    159.54 ±  4%      +7.0%     170.73 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    242127 ± 17%     -38.5%     148844 ± 11%  sched_debug.cpu.avg_idle.min
    445675           +38.3%     616501        sched_debug.cpu.nr_switches.avg
   1116396 ±  3%     +37.8%    1538914 ±  3%  sched_debug.cpu.nr_switches.max
     87188 ±  9%     +22.3%     106625 ±  9%  sched_debug.cpu.nr_switches.min
    257102 ±  7%     +29.8%     333728 ± 13%  sched_debug.cpu.nr_switches.stddev
     56712           +35.6%      76878        perf-stat.i.context-switches
      0.10 ±115%      +0.1        0.20 ± 65%  perf-stat.i.dTLB-load-miss-rate%
      0.03 ±105%      +0.0        0.05 ± 62%  perf-stat.i.dTLB-store-miss-rate%
      0.73 ±  5%      +6.6%       0.77 ±  4%  perf-stat.i.metric.K/sec
      8929            +2.9%       9184 ±  2%  perf-stat.i.minor-faults
   4355263            -7.8%    4016493        perf-stat.i.node-store-misses
      8934            +2.9%       9189 ±  2%  perf-stat.i.page-faults
      2.26            -2.4%       2.21        perf-stat.overall.cpi
      0.44            +2.5%       0.45        perf-stat.overall.ipc
     56281           +35.4%      76201        perf-stat.ps.context-switches
      8895            +2.9%       9153 ±  2%  perf-stat.ps.minor-faults
   4356106            -7.7%    4021578        perf-stat.ps.node-store-misses
      8900            +2.9%       9157 ±  2%  perf-stat.ps.page-faults
      0.01 ± 32%     +36.0%       0.02 ± 10%  perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.01           +62.5%       0.01 ± 25%  perf-sched.sch_delay.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
      3525 ±  9%     +37.0%       4827 ± 18%  perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.chmod_common.__x64_sys_fchmod.do_syscall_64
      4394 ± 12%     +44.9%       6368 ± 19%  perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.chmod_common.do_fchmodat.__x64_sys_chmod
     32.67 ±  3%     +10.2%      36.00 ±  4%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      1035          +185.7%       2959 ± 65%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      1.76 ±  3%     -19.0%       1.42 ±  7%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1.77 ±  3%     -18.9%       1.43 ±  7%  perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      1.19 ± 70%    +152.1%       3.01 ± 26%  perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      6.45 ±  6%     +34.7%       8.69 ±  3%  perf-sched.wait_time.avg.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
      3.51 ±  3%     -19.0%       2.85 ±  7%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      3.53 ±  3%     -19.0%       2.86 ±  7%  perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.00 ± 43%   +1584.6%       0.07 ±132%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      1.19 ± 70%    +152.1%       3.01 ± 26%  perf-sched.wait_time.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
     17.42 ± 14%     +98.7%      34.61 ± 12%  perf-sched.wait_time.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
      1035          +185.7%       2959 ± 65%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
    109028 ± 13%     -19.9%      87359 ±  7%  softirqs.CPU11.RCU
    108501 ± 10%     -20.0%      86855 ±  9%  softirqs.CPU12.RCU
    110305 ± 14%     -20.7%      87480 ± 10%  softirqs.CPU14.RCU
    108707 ±  9%     -22.3%      84483 ±  6%  softirqs.CPU15.RCU
    109869 ± 11%     -22.7%      84920 ±  5%  softirqs.CPU16.RCU
    109589 ± 10%     -20.3%      87352 ±  6%  softirqs.CPU19.RCU
    111095 ±  4%     -21.7%      87035 ±  6%  softirqs.CPU20.RCU
    112327 ±  5%     -12.3%      98484 ±  4%  softirqs.CPU26.RCU
    110285 ± 11%     -20.3%      87908 ±  6%  softirqs.CPU28.RCU
    116821 ± 11%     -19.8%      93714 ±  6%  softirqs.CPU29.RCU
    116416 ± 14%     -21.5%      91381 ± 13%  softirqs.CPU52.RCU
    112023 ± 14%     -22.0%      87433 ±  3%  softirqs.CPU55.RCU
    105612 ± 13%     -16.7%      87988 ±  8%  softirqs.CPU60.RCU
    110826 ±  8%     -21.2%      87282 ±  8%  softirqs.CPU61.RCU
    107047 ±  9%     -21.2%      84308 ±  8%  softirqs.CPU64.RCU
    107421 ±  7%     -19.8%      86135 ±  4%  softirqs.CPU67.RCU
    107038 ±  6%     -18.2%      87573 ± 10%  softirqs.CPU69.RCU
    109282 ± 16%     -17.7%      89900 ±  5%  softirqs.CPU7.RCU
    107866 ± 16%     -22.1%      84000 ± 12%  softirqs.CPU71.RCU
    116855 ± 12%     -18.2%      95536 ±  6%  softirqs.CPU76.RCU
      0.62 ±  6%      +0.1        0.77 ±  5%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
      1.36 ±  3%      +0.3        1.64 ±  4%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.36 ±  3%      +0.3        1.64 ±  4%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.33 ±  3%      +0.3        1.61 ±  4%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      2.99 ±  7%      +0.4        3.35 ±  8%  perf-profile.calltrace.cycles-pp.apparmor_cred_prepare.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
      0.17 ±141%      +0.4        0.59 ±  2%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
      0.00            +0.6        0.55 ±  6%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      6.42 ±  8%      +0.6        7.03 ±  4%  perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat
      6.45 ±  8%      +0.6        7.06 ±  4%  perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat.do_syscall_64
      6.48 ±  8%      +0.6        7.11 ±  4%  perf-profile.calltrace.cycles-pp.free_uid.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.15 ±  3%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.override_creds
      0.07 ± 17%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.11 ±  7%      +0.0        0.14 ±  9%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.15 ±  3%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.23 ± 10%      +0.1        0.29 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.41 ± 13%      +0.1        0.54 ± 11%  perf-profile.children.cycles-pp.tick_sched_handle
      0.40 ± 13%      +0.1        0.53 ± 11%  perf-profile.children.cycles-pp.update_process_times
      0.54 ±  8%      +0.1        0.67 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
      0.77 ±  7%      +0.1        0.92 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.54 ± 13%      +0.1        0.69 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      1.59 ±  5%      +0.3        1.87 ±  8%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.62 ±  5%      +0.3        1.90 ±  8%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      2.00 ±  7%      +0.3        2.29 ±  7%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      2.99 ±  7%      +0.4        3.35 ±  8%  perf-profile.children.cycles-pp.apparmor_cred_prepare
      6.43 ±  8%      +0.6        7.04 ±  4%  perf-profile.children.cycles-pp.refcount_dec_not_one
      6.46 ±  8%      +0.6        7.07 ±  4%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
      6.49 ±  8%      +0.6        7.11 ±  4%  perf-profile.children.cycles-pp.free_uid
      0.08 ± 14%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.07 ± 14%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.15 ±  3%      +0.0        0.18 ±  3%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.11 ±  7%      +0.0        0.14 ±  9%  perf-profile.self.cycles-pp.lapic_next_deadline
      2.97 ±  7%      +0.4        3.33 ±  8%  perf-profile.self.cycles-pp.apparmor_cred_prepare
      5.19 ±  7%      +0.5        5.74 ±  5%  perf-profile.self.cycles-pp.put_cred_rcu
    986.67 ±124%     -99.9%       0.50 ±173%  interrupts.54:PCI-MSI.31981587-edge.i40e-eth0-TxRx-18
      6594 ±108%    -100.0%       0.00        interrupts.67:PCI-MSI.31981600-edge.i40e-eth0-TxRx-31
     15361 ±  8%     +12.6%      17290 ±  8%  interrupts.CPU0.CAL:Function_call_interrupts
     15411 ±  6%     +18.2%      18216 ± 11%  interrupts.CPU13.CAL:Function_call_interrupts
     21697 ±  9%     +11.2%      24133 ±  8%  interrupts.CPU13.TLB:TLB_shootdowns
      6242 ± 48%     -66.3%       2106 ± 42%  interrupts.CPU16.RES:Rescheduling_interrupts
     15155 ±  5%     +20.3%      18225 ±  9%  interrupts.CPU17.CAL:Function_call_interrupts
     21635 ±  8%     +11.4%      24100 ±  6%  interrupts.CPU17.TLB:TLB_shootdowns
    986.33 ±124%    -100.0%       0.25 ±173%  interrupts.CPU18.54:PCI-MSI.31981587-edge.i40e-eth0-TxRx-18
      7537 ± 75%     -76.7%       1756 ± 18%  interrupts.CPU18.RES:Rescheduling_interrupts
     19872 ±  4%     -10.8%      17734 ±  8%  interrupts.CPU21.CAL:Function_call_interrupts
      7630 ± 38%     -50.9%       3750 ± 41%  interrupts.CPU25.RES:Rescheduling_interrupts
     12745 ± 44%     -70.0%       3828 ± 33%  interrupts.CPU30.RES:Rescheduling_interrupts
      6594 ±108%    -100.0%       0.00        interrupts.CPU31.67:PCI-MSI.31981600-edge.i40e-eth0-TxRx-31
      2514 ± 34%    +161.7%       6579 ± 36%  interrupts.CPU31.RES:Rescheduling_interrupts
    151.67 ±  4%     -22.5%     117.50 ± 25%  interrupts.CPU32.NMI:Non-maskable_interrupts
    151.67 ±  4%     -22.5%     117.50 ± 25%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      2186 ± 15%    +168.5%       5871 ± 47%  interrupts.CPU4.RES:Rescheduling_interrupts
     18838 ±  8%     +17.8%      22200 ± 13%  interrupts.CPU40.CAL:Function_call_interrupts
      1042 ±122%    +283.2%       3995 ± 96%  interrupts.CPU40.NMI:Non-maskable_interrupts
      1042 ±122%    +283.2%       3995 ± 96%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      4994 ± 22%     -69.4%       1527 ± 91%  interrupts.CPU46.NMI:Non-maskable_interrupts
      4994 ± 22%     -69.4%       1527 ± 91%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      1499 ±131%    +236.0%       5037 ± 59%  interrupts.CPU48.NMI:Non-maskable_interrupts
      1499 ±131%    +236.0%       5037 ± 59%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
     16663 ±  7%     +23.0%      20495 ± 11%  interrupts.CPU49.CAL:Function_call_interrupts
    114.67 ± 27%   +2932.0%       3476 ± 72%  interrupts.CPU49.NMI:Non-maskable_interrupts
    114.67 ± 27%   +2932.0%       3476 ± 72%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
     22069 ± 11%     +21.1%      26736 ± 12%  interrupts.CPU49.TLB:TLB_shootdowns
      2733 ±133%     -95.8%     114.00 ± 25%  interrupts.CPU5.NMI:Non-maskable_interrupts
      2733 ±133%     -95.8%     114.00 ± 25%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      4574 ± 30%     -47.9%       2382 ± 44%  interrupts.CPU50.RES:Rescheduling_interrupts
     15867 ± 12%     +16.6%      18496 ±  8%  interrupts.CPU52.CAL:Function_call_interrupts
      1307 ±125%     -89.8%     133.50 ±  3%  interrupts.CPU58.NMI:Non-maskable_interrupts
      1307 ±125%     -89.8%     133.50 ±  3%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      6139 ± 46%     -60.5%       2427 ± 76%  interrupts.CPU64.RES:Rescheduling_interrupts
      7489 ± 67%     -66.1%       2541 ± 43%  interrupts.CPU66.RES:Rescheduling_interrupts
     24235 ± 14%     +19.0%      28848 ± 11%  interrupts.CPU67.TLB:TLB_shootdowns
     16186 ± 10%     +19.0%      19269 ± 11%  interrupts.CPU69.CAL:Function_call_interrupts
     21972 ± 12%     +13.4%      24921 ±  9%  interrupts.CPU69.TLB:TLB_shootdowns
    172.33 ± 15%     -43.3%      97.75 ± 34%  interrupts.CPU7.NMI:Non-maskable_interrupts
    172.33 ± 15%     -43.3%      97.75 ± 34%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      4791 ± 19%     -48.5%       2467 ± 23%  interrupts.CPU70.RES:Rescheduling_interrupts
      7892 ± 39%     -58.7%       3259 ± 46%  interrupts.CPU73.RES:Rescheduling_interrupts
     28136 ±  8%     -13.1%      24437 ±  8%  interrupts.CPU73.TLB:TLB_shootdowns
     12028 ± 42%     -60.8%       4715 ± 40%  interrupts.CPU78.RES:Rescheduling_interrupts
      2926 ± 30%    +126.8%       6636 ± 19%  interrupts.CPU79.RES:Rescheduling_interrupts
     21660 ±  3%     -12.3%      18992 ± 10%  interrupts.CPU82.CAL:Function_call_interrupts
     28681 ±  4%     -16.7%      23879 ±  4%  interrupts.CPU82.TLB:TLB_shootdowns
     22381 ± 11%     -21.6%      17554 ±  7%  interrupts.CPU83.CAL:Function_call_interrupts
     28764 ± 12%     -19.5%      23162 ±  4%  interrupts.CPU83.TLB:TLB_shootdowns
     27733 ± 10%     -13.2%      24069 ±  3%  interrupts.CPU86.TLB:TLB_shootdowns
     21889 ± 11%     -19.2%      17686 ±  6%  interrupts.CPU87.CAL:Function_call_interrupts
     29488 ±  9%     -19.8%      23643 ±  5%  interrupts.CPU87.TLB:TLB_shootdowns
     25340 ±  6%     -22.4%      19673 ±  6%  interrupts.CPU90.CAL:Function_call_interrupts
     29230 ± 12%     -15.7%      24646 ±  2%  interrupts.CPU90.TLB:TLB_shootdowns





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.11.0-rc4-g3de7d4f25a74" of type "text/plain" (172414 bytes)

View attachment "job-script" of type "text/plain" (7891 bytes)

View attachment "job.yaml" of type "text/plain" (5474 bytes)

View attachment "reproduce" of type "text/plain" (530 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ