lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <202211221125.64e73ddf-yujie.liu@intel.com>
Date:   Tue, 22 Nov 2022 13:52:29 +0800
From:   kernel test robot <yujie.liu@...el.com>
To:     Andrei Vagin <avagin@...gle.com>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Juri Lelli <juri.lelli@...hat.com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
        "Daniel Bristot de Oliveira" <bristot@...hat.com>,
        Valentin Schneider <vschneid@...hat.com>,
        <linux-kernel@...r.kernel.org>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <zhengjun.xing@...ux.intel.com>,
        <fengwei.yin@...el.com>, Andrei Vagin <avagin@...il.com>,
        Chen Yu <yu.c.chen@...el.com>
Subject: Re: [PATCH] sched: consider WF_SYNC to find idle siblings


Please be noted that we observed various performance changes in
different benchmarks. The impact is well discussed in the mail thread,
e.g. "strictly synchronous jobs may benefit, but netperf will get
punished". We are just sending this report to provide some additional
test data for your reference. Thanks.

Greeting,

FYI, we noticed a -66.0% regression of netperf.Throughput_Mbps due to commit:

commit: b6aabb01e3004b0285d953761e1592974b4b50fa ("[PATCH] sched: consider WF_SYNC to find idle siblings")
url: https://github.com/intel-lab-lkp/linux/commits/Andrei-Vagin/sched-consider-WF_SYNC-to-find-idle-siblings/20221028-042823
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git c36eae5a86d15a694968be35d7ff056854997a72
patch link: https://lore.kernel.org/all/20221027202603.670616-1-avagin@google.com/
patch subject: [PATCH] sched: consider WF_SYNC to find idle siblings

in testcase: netperf
on test machine: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 50%
	cluster: cs-localhost
	send_size: 10K
	test: SCTP_STREAM_MANY
	cpufreq_governor: performance

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.sctp.ops_per_sec -48.4% regression                                        |
| test machine     | 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory |
| test parameters  | class=network                                                                                  |
|                  | cpufreq_governor=performance                                                                   |
|                  | nr_threads=100%                                                                                |
|                  | test=sctp                                                                                      |
|                  | testtime=60s                                                                                   |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | unixbench: unixbench.score 36.2% improvement                                                   |
| test machine     | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory      |
| test parameters  | cpufreq_governor=performance                                                                   |
|                  | nr_task=30%                                                                                    |
|                  | runtime=300s                                                                                   |
|                  | test=context1                                                                                  |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_tps 53.3% improvement                                              |
| test machine     | 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz (Cooper Lake) with 128G memory |
| test parameters  | cluster=cs-localhost                                                                           |
|                  | cpufreq_governor=performance                                                                   |
|                  | ip=ipv4                                                                                        |
|                  | nr_threads=25%                                                                                 |
|                  | runtime=300s                                                                                   |
|                  | test=TCP_RR                                                                                    |
+------------------+------------------------------------------------------------------------------------------------+


Details are as below:

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/50%/debian-11.1-x86_64-20220510.cgz/300s/10K/lkp-icl-2sp4/SCTP_STREAM_MANY/netperf

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      7844           -66.0%       2669 ± 23%  netperf.Throughput_Mbps
    502046           -66.0%     170845 ± 23%  netperf.Throughput_total_Mbps
     96866        +87957.6%   85298188 ± 23%  netperf.time.involuntary_context_switches
     22262            -5.6%      21022        netperf.time.minor_page_faults
      1674           -79.6%     341.00 ± 23%  netperf.time.percent_of_cpu_this_job_got
      4913           -79.7%     995.08 ± 23%  netperf.time.system_time
    133.78 ±  2%     -74.6%      34.03 ± 23%  netperf.time.user_time
    345136          +237.6%    1165282 ±  9%  netperf.time.voluntary_context_switches
 1.839e+09           -66.0%  6.257e+08 ± 23%  netperf.workload
     34569           +22.0%      42159        uptime.idle
 2.841e+10           +27.0%  3.608e+10        cpuidle..time
 5.644e+08           -86.9%   73779743        cpuidle..usage
 1.201e+09 ±  2%     -66.7%  4.002e+08 ± 21%  numa-numastat.node0.local_node
 1.201e+09 ±  2%     -66.7%  4.003e+08 ± 21%  numa-numastat.node0.numa_hit
 1.171e+09 ±  3%     -65.3%  4.064e+08 ± 25%  numa-numastat.node1.local_node
 1.171e+09 ±  3%     -65.3%  4.064e+08 ± 25%  numa-numastat.node1.numa_hit
     74.00           +25.0%      92.50        vmstat.cpu.id
   4716732           -15.7%    3974601        vmstat.memory.cache
     31.83           -65.4%      11.00 ± 21%  vmstat.procs.r
   3346244           -82.4%     589168 ± 23%  vmstat.system.cs
     74.13           +19.0       93.17        mpstat.cpu.all.idle%
      0.60 ±  2%      +0.3        0.95 ± 24%  mpstat.cpu.all.irq%
      3.85            -2.9        0.93 ± 22%  mpstat.cpu.all.soft%
     20.66           -15.9        4.72 ± 23%  mpstat.cpu.all.sys%
      0.77            -0.5        0.24 ± 20%  mpstat.cpu.all.usr%
    147463 ± 10%     +15.6%     170513 ±  6%  numa-meminfo.node0.SUnreclaim
    803425 ±  5%     -95.6%      35689 ± 56%  numa-meminfo.node1.Active
    803380 ±  5%     -95.6%      35646 ± 56%  numa-meminfo.node1.Active(anon)
    330592 ±  9%     -90.1%      32819 ± 45%  numa-meminfo.node1.Mapped
    815993 ±  5%     -90.1%      80396 ± 35%  numa-meminfo.node1.Shmem
    832708 ±  2%     -95.0%      41879 ± 52%  meminfo.Active
    832610 ±  2%     -95.0%      41793 ± 52%  meminfo.Active(anon)
   4598130           -16.3%    3847596        meminfo.Cached
   1436261           -53.0%     674663 ±  3%  meminfo.Committed_AS
    381642 ±  3%     -81.9%      69247 ± 10%  meminfo.Mapped
   6510432           -10.7%    5812535        meminfo.Memused
    846123 ±  2%     -88.6%      96290 ± 24%  meminfo.Shmem
  10463627           -32.4%    7073752 ±  3%  meminfo.max_used_kB
     36887 ± 10%     +15.4%      42580 ±  6%  numa-vmstat.node0.nr_slab_unreclaimable
 1.201e+09 ±  2%     -66.7%  4.003e+08 ± 21%  numa-vmstat.node0.numa_hit
 1.201e+09 ±  2%     -66.7%  4.002e+08 ± 21%  numa-vmstat.node0.numa_local
    200557 ±  5%     -95.5%       8951 ± 56%  numa-vmstat.node1.nr_active_anon
     83108 ±  9%     -90.0%       8342 ± 44%  numa-vmstat.node1.nr_mapped
    203712 ±  5%     -90.1%      20228 ± 35%  numa-vmstat.node1.nr_shmem
    200557 ±  5%     -95.5%       8951 ± 56%  numa-vmstat.node1.nr_zone_active_anon
 1.171e+09 ±  3%     -65.3%  4.064e+08 ± 25%  numa-vmstat.node1.numa_hit
 1.171e+09 ±  3%     -65.3%  4.064e+08 ± 25%  numa-vmstat.node1.numa_local
    957.83           -73.0%     258.33 ± 21%  turbostat.Avg_MHz
     29.11           -21.2        7.89 ± 20%  turbostat.Busy%
 4.303e+08          -100.0%     107318 ± 26%  turbostat.C1
      5.52            -5.5        0.03 ±100%  turbostat.C1%
 1.317e+08 ±  2%     -62.3%   49626627 ± 35%  turbostat.C1E
     74.33           -24.7%      56.00 ±  4%  turbostat.CoreTmp
      0.30           +20.0%       0.36 ±  4%  turbostat.IPC
   1105923           -99.4%       6441 ± 46%  turbostat.POLL
      0.03            -0.0        0.00        turbostat.POLL%
     74.33 ±  2%     -24.4%      56.17 ±  4%  turbostat.PkgTmp
    455.30           -38.7%     279.16 ±  5%  turbostat.PkgWatt
     46.51            -6.1%      43.69        turbostat.RAMWatt
    208073 ±  2%     -95.0%      10499 ± 52%  proc-vmstat.nr_active_anon
    101503            -1.2%     100305        proc-vmstat.nr_anon_pages
   1149456           -16.3%     962060        proc-vmstat.nr_file_pages
    106787            +8.5%     115818        proc-vmstat.nr_inactive_anon
     95661 ±  3%     -81.7%      17510 ± 10%  proc-vmstat.nr_mapped
      2428 ±  2%      -6.0%       2282        proc-vmstat.nr_page_table_pages
    211454 ±  2%     -88.5%      24233 ± 24%  proc-vmstat.nr_shmem
     31087            -2.0%      30451        proc-vmstat.nr_slab_reclaimable
     74377            +7.9%      80233        proc-vmstat.nr_slab_unreclaimable
    208073 ±  2%     -95.0%      10499 ± 52%  proc-vmstat.nr_zone_active_anon
    106787            +8.5%     115818        proc-vmstat.nr_zone_inactive_anon
 2.372e+09           -66.0%  8.068e+08 ± 23%  proc-vmstat.numa_hit
 2.372e+09           -66.0%  8.066e+08 ± 23%  proc-vmstat.numa_local
    242358 ±  2%     +95.5%     473925 ±  7%  proc-vmstat.pgactivate
 1.366e+10           -66.0%  4.643e+09 ± 23%  proc-vmstat.pgalloc_normal
   1461389           -16.3%    1222803 ±  4%  proc-vmstat.pgfault
 1.366e+10           -66.0%  4.643e+09 ± 23%  proc-vmstat.pgfree
     12556 ± 50%     -99.9%      17.47 ±159%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    766470 ± 38%     -99.7%       2236 ±159%  sched_debug.cfs_rq:/.MIN_vruntime.max
     93742 ± 44%     -99.8%     196.88 ±159%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.24 ±  6%     -58.4%       0.10 ± 37%  sched_debug.cfs_rq:/.h_nr_running.avg
      1.06 ±  7%     +66.3%       1.76 ±  5%  sched_debug.cfs_rq:/.h_nr_running.max
    218.12 ±  7%    +153.6%     553.21 ± 76%  sched_debug.cfs_rq:/.load_avg.max
     12556 ± 50%     -99.9%      17.47 ±159%  sched_debug.cfs_rq:/.max_vruntime.avg
    766470 ± 38%     -99.7%       2236 ±159%  sched_debug.cfs_rq:/.max_vruntime.max
     93742 ± 44%     -99.8%     196.88 ±159%  sched_debug.cfs_rq:/.max_vruntime.stddev
    998821 ± 10%     -90.9%      90606 ± 16%  sched_debug.cfs_rq:/.min_vruntime.avg
   1154960 ± 10%     -88.8%     128907 ± 11%  sched_debug.cfs_rq:/.min_vruntime.max
    874022 ± 13%     -92.2%      67934 ± 21%  sched_debug.cfs_rq:/.min_vruntime.min
     49122 ± 17%     -80.4%       9626 ± 13%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.24 ±  6%     -70.0%       0.07 ± 32%  sched_debug.cfs_rq:/.nr_running.avg
      0.42 ±  2%     -40.3%       0.25 ± 14%  sched_debug.cfs_rq:/.nr_running.stddev
    269.97 ±  2%     -53.3%     126.18 ± 19%  sched_debug.cfs_rq:/.runnable_avg.avg
      1166 ±  4%     +24.6%       1453 ±  4%  sched_debug.cfs_rq:/.runnable_avg.max
    116926 ± 27%     -84.1%      18583 ± 50%  sched_debug.cfs_rq:/.spread0.max
   -164017           -74.1%     -42414        sched_debug.cfs_rq:/.spread0.min
     49129 ± 17%     -80.4%       9632 ± 13%  sched_debug.cfs_rq:/.spread0.stddev
    269.72 ±  2%     -61.9%     102.78 ± 15%  sched_debug.cfs_rq:/.util_avg.avg
      1166 ±  4%      -7.0%       1084 ±  3%  sched_debug.cfs_rq:/.util_avg.max
    323.32 ±  2%     -32.6%     217.82 ± 13%  sched_debug.cfs_rq:/.util_avg.stddev
    146.06 ±  6%     -77.0%      33.66 ± 44%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    989.44           -13.3%     857.96 ±  8%  sched_debug.cfs_rq:/.util_est_enqueued.max
    276.89 ±  2%     -50.8%     136.28 ± 23%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    574938 ±  3%     +26.2%     725536 ±  3%  sched_debug.cpu.avg_idle.avg
      3945 ±  3%    +867.5%      38177 ± 80%  sched_debug.cpu.avg_idle.min
    406249 ±  2%     -21.9%     317312 ±  6%  sched_debug.cpu.avg_idle.stddev
      3.57 ±  5%     +54.4%       5.51 ±  6%  sched_debug.cpu.clock.stddev
      1612 ±  2%     -11.4%       1429        sched_debug.cpu.clock_task.stddev
      1391 ±  5%     -69.5%     424.83 ± 33%  sched_debug.cpu.curr->pid.avg
      8941 ±  4%     -17.6%       7367 ±  4%  sched_debug.cpu.curr->pid.max
      2392 ±  2%     -39.0%       1459 ± 14%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  6%     -27.8%       0.00 ±  7%  sched_debug.cpu.next_balance.stddev
      0.24 ±  5%     -56.8%       0.10 ± 37%  sched_debug.cpu.nr_running.avg
      1.06 ±  7%     +86.8%       1.97 ±  3%  sched_debug.cpu.nr_running.max
   3581710 ± 11%     -84.7%     549192 ± 15%  sched_debug.cpu.nr_switches.avg
   4987997 ± 13%     -85.8%     708118 ± 15%  sched_debug.cpu.nr_switches.max
   2372208 ± 19%     -82.6%     411758 ± 18%  sched_debug.cpu.nr_switches.min
    497412 ± 10%     -88.1%      59277 ± 14%  sched_debug.cpu.nr_switches.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     25.06           -64.2%       8.97 ± 18%  perf-stat.i.MPKI
 2.221e+10           -67.9%   7.14e+09 ± 22%  perf-stat.i.branch-instructions
 1.197e+08           -59.5%   48489409 ± 32%  perf-stat.i.branch-misses
      1.48 ±  3%      +1.5        2.96 ± 20%  perf-stat.i.cache-miss-rate%
  39815902 ±  3%     -78.9%    8385204 ± 16%  perf-stat.i.cache-misses
 2.787e+09           -88.6%  3.168e+08 ± 30%  perf-stat.i.cache-references
   3386965           -82.4%     596733 ± 23%  perf-stat.i.context-switches
      1.17           -19.7%       0.94 ±  4%  perf-stat.i.cpi
 1.291e+11           -74.7%  3.267e+10 ± 21%  perf-stat.i.cpu-cycles
    744.79          +888.7%       7363 ± 16%  perf-stat.i.cpu-migrations
      3317 ±  2%     +18.6%       3933 ±  5%  perf-stat.i.cycles-between-cache-misses
 3.178e+10           -67.6%  1.029e+10 ± 22%  perf-stat.i.dTLB-loads
 1.882e+10           -67.8%  6.057e+09 ± 22%  perf-stat.i.dTLB-stores
  1.11e+11           -67.7%  3.585e+10 ± 22%  perf-stat.i.instructions
      0.86           +24.8%       1.07 ±  4%  perf-stat.i.ipc
      1.01           -74.7%       0.26 ± 21%  perf-stat.i.metric.GHz
    590.56           -68.5%     185.89 ± 22%  perf-stat.i.metric.M/sec
      4085           -19.2%       3300 ±  6%  perf-stat.i.minor-faults
     91.59            -1.9       89.65        perf-stat.i.node-load-miss-rate%
   3843758 ±  8%     -56.4%    1676424 ± 17%  perf-stat.i.node-load-misses
    345198 ±  4%     -42.8%     197478 ± 17%  perf-stat.i.node-loads
     70.70           -31.5       39.23 ±  6%  perf-stat.i.node-store-miss-rate%
   3113677 ±  4%     -64.0%    1121141 ± 17%  perf-stat.i.node-store-misses
      4085           -19.2%       3300 ±  6%  perf-stat.i.page-faults
     25.10           -65.2%       8.73 ± 16%  perf-stat.overall.MPKI
      1.43 ±  3%      +1.4        2.79 ± 17%  perf-stat.overall.cache-miss-rate%
      1.16           -21.4%       0.91 ±  4%  perf-stat.overall.cpi
      3244 ±  2%     +19.3%       3871 ±  5%  perf-stat.overall.cycles-between-cache-misses
      0.86           +27.4%       1.10 ±  4%  perf-stat.overall.ipc
     91.72            -2.3       89.42        perf-stat.overall.node-load-miss-rate%
     68.26           -29.8       38.51 ±  6%  perf-stat.overall.node-store-miss-rate%
     18276            -5.2%      17333        perf-stat.overall.path-length
 2.214e+10           -67.8%  7.118e+09 ± 22%  perf-stat.ps.branch-instructions
 1.192e+08           -59.5%   48334433 ± 32%  perf-stat.ps.branch-misses
  39676032 ±  3%     -78.9%    8357755 ± 16%  perf-stat.ps.cache-misses
 2.777e+09           -88.6%  3.158e+08 ± 30%  perf-stat.ps.cache-references
   3374855           -82.4%     594908 ± 23%  perf-stat.ps.context-switches
 1.286e+11           -74.7%  3.257e+10 ± 21%  perf-stat.ps.cpu-cycles
    742.42          +888.6%       7339 ± 16%  perf-stat.ps.cpu-migrations
 3.166e+10           -67.6%  1.026e+10 ± 22%  perf-stat.ps.dTLB-loads
 1.876e+10           -67.8%  6.038e+09 ± 22%  perf-stat.ps.dTLB-stores
 1.106e+11           -67.7%  3.574e+10 ± 22%  perf-stat.ps.instructions
      4069           -19.2%       3290 ±  6%  perf-stat.ps.minor-faults
   3830257 ±  8%     -56.4%    1670802 ± 17%  perf-stat.ps.node-load-misses
    344097 ±  4%     -42.8%     196864 ± 17%  perf-stat.ps.node-loads
   3102660 ±  4%     -64.0%    1117149 ± 17%  perf-stat.ps.node-store-misses
      4069           -19.2%       3290 ±  6%  perf-stat.ps.page-faults
  3.36e+13           -67.8%  1.082e+13 ± 22%  perf-stat.total.instructions
     22.54            -9.4       13.14 ±  5%  perf-profile.calltrace.cycles-pp.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg
     22.62            -9.4       13.26 ±  5%  perf-profile.calltrace.cycles-pp.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64
     23.55            -9.2       14.36 ±  5%  perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.92            -9.0       15.96 ±  4%  perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     25.25            -8.9       16.31 ±  4%  perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     26.64            -8.9       17.72 ±  4%  perf-profile.calltrace.cycles-pp.recvmsg
     25.74            -8.9       16.87 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvmsg
     25.56            -8.8       16.74 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
      8.60            -8.6        0.00        perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      8.45            -8.5        0.00        perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     10.10            -4.9        5.19 ±  3%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
     10.13            -4.9        5.24 ±  3%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      8.62            -4.3        4.35 ±  3%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg
      8.58            -4.3        4.30 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      8.87            -4.2        4.66 ±  3%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg
      4.71            -2.5        2.22 ±  4%  perf-profile.calltrace.cycles-pp.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      3.72 ±  2%      -1.9        1.83 ±  4%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve
      4.19 ±  2%      -1.8        2.36 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
      2.89 ±  2%      -1.8        1.09 ±  5%  perf-profile.calltrace.cycles-pp.skb_release_data.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
     10.94            -1.8        9.18 ±  4%  perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
     11.10            -1.7        9.37 ±  4%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
     11.08            -1.7        9.35 ±  3%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
     11.12            -1.7        9.44 ±  4%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     13.03            -1.6       11.41 ±  3%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc
     11.43            -1.6        9.85 ±  4%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
     11.76            -1.5       10.22 ±  4%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
     11.65            -1.5       10.11 ±  4%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
     11.62            -1.5       10.10 ±  4%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
     12.02            -1.5       10.50 ±  4%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
     12.10            -1.5       10.59 ±  4%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit
     12.16            -1.5       10.67 ±  4%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
      2.44 ±  2%      -1.4        1.00 ± 17%  perf-profile.calltrace.cycles-pp.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
     12.44            -1.4       11.00 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
     14.88            -1.4       13.45 ±  3%  perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
     12.62            -1.4       11.21 ±  4%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      8.62            -1.4        7.21 ±  4%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      2.17 ±  2%      -1.4        0.79 ±  4%  perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
      2.16 ±  2%      -1.4        0.78 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush
      2.15 ±  2%      -1.4        0.77 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.sctp_packet_transmit
      9.20            -1.4        7.83 ±  4%  perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      5.66            -1.3        4.33 ±  5%  perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
      2.18 ±  2%      -1.3        0.87 ± 24%  perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller
      7.72            -1.3        6.46 ±  5%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu
     10.04            -1.2        8.80 ±  3%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg
      2.33            -1.2        1.14 ±  5%  perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      9.78            -1.1        8.67 ±  3%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      2.14            -1.1        1.08 ±  5%  perf-profile.calltrace.cycles-pp.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm
      4.22 ±  2%      -0.9        3.28 ±  4%  perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
     19.14            -0.9       18.21 ±  3%  perf-profile.calltrace.cycles-pp.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg
      3.72 ±  2%      -0.8        2.90 ±  4%  perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm
      1.77            -0.8        1.00 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter
      1.69            -0.7        0.96 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data
      5.43 ±  2%      -0.7        4.72 ±  3%  perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
      1.35            -0.7        0.65 ±  9%  perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      1.80            -0.7        1.10 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_head_state.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      1.38            -0.7        0.68 ±  9%  perf-profile.calltrace.cycles-pp.__sk_mem_schedule.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      0.98 ±  2%      -0.6        0.35 ± 70%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.29 ±  2%      -0.6        0.69 ±  5%  perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.skb_release_head_state.kfree_skb_reason.sctp_recvmsg.inet_recvmsg
      2.42 ±  2%      -0.6        1.84 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb._sctp_make_chunk
      2.48 ±  2%      -0.6        1.92 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty
      2.52 ±  2%      -0.6        1.97 ±  4%  perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user
     13.06            -0.5       12.53 ±  3%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      1.04 ±  2%      -0.5        0.54 ±  5%  perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.sctp_wfree.skb_release_head_state.consume_skb.sctp_chunk_put
      1.42            -0.5        0.94 ±  4%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event
      1.40            -0.5        0.92 ±  4%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready
      1.58 ±  2%      -0.4        1.14 ±  5%  perf-profile.calltrace.cycles-pp.sctp_wfree.skb_release_head_state.consume_skb.sctp_chunk_put.sctp_outq_sack
      1.74 ±  2%      -0.4        1.32 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_head_state.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter
      3.02 ±  2%      -0.4        2.61 ±  4%  perf-profile.calltrace.cycles-pp.sctp_ulpevent_make_rcvmsg.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      3.27            -0.4        2.87 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      1.42 ±  2%      -0.4        1.04 ±  5%  perf-profile.calltrace.cycles-pp.free_unref_page.skb_release_data.consume_skb.sctp_chunk_put.sctp_outq_sack
      1.92 ±  2%      -0.4        1.55 ±  5%  perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter
      0.91            -0.3        0.60 ±  4%  perf-profile.calltrace.cycles-pp.sctp_outq_select_transport.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      0.75            -0.3        0.47 ± 44%  perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      3.92            -0.2        3.70 ±  4%  perf-profile.calltrace.cycles-pp._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.38            -0.2        1.23 ±  3%  perf-profile.calltrace.cycles-pp.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      0.52 ±  2%      +0.1        0.64 ±  4%  perf-profile.calltrace.cycles-pp.sock_kmalloc.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64
      0.59 ±  3%      +0.1        0.74 ±  5%  perf-profile.calltrace.cycles-pp.__import_iovec.import_iovec.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg
      0.58 ±  4%      +0.2        0.73 ±  5%  perf-profile.calltrace.cycles-pp.__import_iovec.import_iovec.copy_msghdr_from_user.___sys_recvmsg.__sys_recvmsg
      0.61 ±  2%      +0.2        0.77 ±  4%  perf-profile.calltrace.cycles-pp.import_iovec.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg.do_syscall_64
      0.62 ±  3%      +0.2        0.78 ±  5%  perf-profile.calltrace.cycles-pp.__copy_msghdr.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg.do_syscall_64
      0.60 ±  4%      +0.2        0.76 ±  5%  perf-profile.calltrace.cycles-pp.import_iovec.copy_msghdr_from_user.___sys_recvmsg.__sys_recvmsg.do_syscall_64
      0.62 ±  5%      +0.2        0.78 ±  9%  perf-profile.calltrace.cycles-pp.__rhashtable_lookup.sctp_epaddr_lookup_transport.sctp_endpoint_lookup_assoc.sctp_sendmsg.sock_sendmsg
      0.79            +0.2        0.97 ±  4%  perf-profile.calltrace.cycles-pp.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      0.68 ±  4%      +0.2        0.86 ±  8%  perf-profile.calltrace.cycles-pp.sctp_epaddr_lookup_transport.sctp_endpoint_lookup_assoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      0.72 ±  4%      +0.2        0.92 ±  8%  perf-profile.calltrace.cycles-pp.sctp_endpoint_lookup_assoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
      0.45 ± 44%      +0.2        0.67 ±  4%  perf-profile.calltrace.cycles-pp.move_addr_to_kernel.__copy_msghdr.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg
      1.20            +0.2        1.43 ±  3%  perf-profile.calltrace.cycles-pp.copy_msghdr_from_user.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.34 ± 70%      +0.3        0.64 ±  7%  perf-profile.calltrace.cycles-pp.sctp_chunkify._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      1.82 ±  2%      +0.4        2.18 ±  4%  perf-profile.calltrace.cycles-pp.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      0.26 ±100%      +0.4        0.64 ±  6%  perf-profile.calltrace.cycles-pp.iovec_from_user.__import_iovec.import_iovec.copy_msghdr_from_user.___sys_sendmsg
      1.69 ±  2%      +0.4        2.14 ±  3%  perf-profile.calltrace.cycles-pp.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.09 ±223%      +0.5        0.62 ±  5%  perf-profile.calltrace.cycles-pp.iovec_from_user.__import_iovec.import_iovec.copy_msghdr_from_user.___sys_recvmsg
      0.00            +0.6        0.56 ±  6%  perf-profile.calltrace.cycles-pp.release_sock.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      0.00            +0.6        0.59 ±  5%  perf-profile.calltrace.cycles-pp.sctp_chunk_free.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      0.00            +0.6        0.62 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.__cond_resched.__release_sock.release_sock.sctp_sendmsg
      0.00            +0.6        0.62 ±  3%  perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance
      0.00            +0.6        0.62 ±  5%  perf-profile.calltrace.cycles-pp.sctp_addto_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.50 ±  2%      +0.6        2.13 ±  3%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg
      1.44 ±  2%      +0.6        2.08 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg
      0.00            +0.7        0.67 ±  3%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
      0.00            +0.7        0.67 ±  4%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
      0.00            +0.7        0.68 ± 14%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +0.7        0.68 ±  3%  perf-profile.calltrace.cycles-pp.__cond_resched.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg
      0.00            +0.7        0.73 ± 18%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.7        0.74 ±  4%  perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
      0.00            +0.8        0.79 ±  4%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
      3.36            +0.9        4.22 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user
      3.40            +0.9        4.27 ±  3%  perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      0.68 ±  2%      +0.9        1.55 ± 19%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.9        0.90 ± 15%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +0.9        0.93 ±  3%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.sctp_skb_recv_datagram
      3.65            +0.9        4.59 ±  3%  perf-profile.calltrace.cycles-pp._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      4.08            +1.1        5.13 ±  3%  perf-profile.calltrace.cycles-pp.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      0.00            +1.1        1.14 ± 14%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.4        1.36 ± 26%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      9.56            +1.4       10.94 ±  3%  perf-profile.calltrace.cycles-pp.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      0.00            +1.5        1.49 ± 30%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +1.7        1.66 ± 27%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +2.5        2.46 ± 26%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +3.5        3.48 ± 19%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +3.6        3.57 ± 20%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.66 ± 10%      +4.9        5.60 ± 18%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.75 ±  9%      +5.9        6.66 ± 17%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     17.20            +6.5       23.70 ± 13%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     17.19            +6.5       23.70 ± 13%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.15            +6.5       23.67 ± 13%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.32            +6.6       23.88 ± 13%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     13.07            +7.7       20.79 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     13.17            +8.2       21.38 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     14.06            +9.2       23.24 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.18 ±  4%     +10.6       13.78 ± 12%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      3.20 ±  4%     +10.6       13.84 ± 12%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     22.86            -9.4       13.47 ±  5%  perf-profile.children.cycles-pp.sctp_recvmsg
     22.63            -9.4       13.27 ±  5%  perf-profile.children.cycles-pp.inet_recvmsg
     23.57            -9.2       14.38 ±  5%  perf-profile.children.cycles-pp.____sys_recvmsg
     24.96            -9.0       16.00 ±  4%  perf-profile.children.cycles-pp.___sys_recvmsg
     25.27            -8.9       16.34 ±  4%  perf-profile.children.cycles-pp.__sys_recvmsg
     26.87            -8.9       18.01 ±  4%  perf-profile.children.cycles-pp.recvmsg
      8.67            -8.7        0.00        perf-profile.children.cycles-pp.intel_idle_irq
     79.11            -7.0       72.07 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     78.72            -7.0       71.69 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
     10.11            -4.9        5.20 ±  3%  perf-profile.children.cycles-pp.__skb_datagram_iter
     10.14            -4.9        5.25 ±  3%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      8.64            -4.3        4.36 ±  3%  perf-profile.children.cycles-pp.copyout
      8.88            -4.2        4.68 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
     13.72            -2.9       10.78 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      4.73            -2.5        2.25 ±  4%  perf-profile.children.cycles-pp.kfree_skb_reason
     33.66            -2.3       31.40 ±  3%  perf-profile.children.cycles-pp.sctp_cmd_interpreter
     33.91            -2.2       31.71 ±  3%  perf-profile.children.cycles-pp.sctp_do_sm
      4.98            -2.1        2.88 ±  4%  perf-profile.children.cycles-pp.skb_release_data
      2.42 ±  2%      -2.0        0.45 ± 71%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     24.48            -1.9       22.58 ±  3%  perf-profile.children.cycles-pp.sctp_packet_transmit
     27.60            -1.6       25.95 ±  3%  perf-profile.children.cycles-pp.sctp_outq_flush
      3.79            -1.6        2.16 ±  4%  perf-profile.children.cycles-pp.get_page_from_freelist
      4.66            -1.6        3.04 ±  3%  perf-profile.children.cycles-pp.__kmalloc_large_node
      4.81            -1.6        3.25 ±  3%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      4.87            -1.5        3.33 ±  3%  perf-profile.children.cycles-pp.kmalloc_reserve
      4.28 ±  2%      -1.5        2.76 ±  3%  perf-profile.children.cycles-pp.__alloc_pages
     15.05            -1.5       13.56 ±  4%  perf-profile.children.cycles-pp.sctp_primitive_SEND
      2.22 ±  2%      -1.5        0.75 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     11.15            -1.4        9.72 ±  4%  perf-profile.children.cycles-pp.sctp_rcv
      2.74 ±  2%      -1.4        1.34 ±  5%  perf-profile.children.cycles-pp.free_unref_page
     11.28            -1.4        9.89 ±  4%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      6.14            -1.4        4.76 ±  3%  perf-profile.children.cycles-pp.__alloc_skb
     11.30            -1.4        9.96 ±  4%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      5.81            -1.3        4.49 ±  5%  perf-profile.children.cycles-pp.sctp_ulpq_tail_data
      1.36 ±  2%      -1.3        0.06 ±  6%  perf-profile.children.cycles-pp.__slab_free
      2.67 ±  2%      -1.3        1.40 ±  4%  perf-profile.children.cycles-pp.rmqueue
     11.61            -1.2       10.38 ±  4%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     12.28            -1.2       11.06 ±  4%  perf-profile.children.cycles-pp.do_softirq
     11.95            -1.2       10.76 ±  4%  perf-profile.children.cycles-pp.net_rx_action
     11.83            -1.2       10.64 ±  4%  perf-profile.children.cycles-pp.__napi_poll
     11.81            -1.2       10.63 ±  4%  perf-profile.children.cycles-pp.process_backlog
      5.74            -1.2        4.56 ±  4%  perf-profile.children.cycles-pp.sctp_chunk_put
     12.49            -1.2       11.32 ±  4%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      1.62 ±  2%      -1.2        0.45 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
      2.60            -1.2        1.43 ±  7%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      3.83            -1.2        2.66 ±  4%  perf-profile.children.cycles-pp.skb_release_head_state
      2.66            -1.2        1.51 ±  6%  perf-profile.children.cycles-pp.__sk_mem_schedule
     24.83            -1.1       23.72 ±  4%  perf-profile.children.cycles-pp.sctp_assoc_bh_rcv
      2.37            -1.1        1.27 ±  5%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
     12.86            -1.1       11.80 ±  4%  perf-profile.children.cycles-pp.ip_finish_output2
      1.10            -1.1        0.04 ± 44%  perf-profile.children.cycles-pp.sched_ttwu_pending
     13.14            -1.0       12.13 ±  4%  perf-profile.children.cycles-pp.__ip_queue_xmit
      2.38            -1.0        1.41 ±  5%  perf-profile.children.cycles-pp.sctp_ulpq_tail_event
     19.52            -1.0       18.56 ±  3%  perf-profile.children.cycles-pp.sctp_backlog_rcv
      2.02            -0.9        1.09 ±  5%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
      2.17            -0.9        1.25 ±  5%  perf-profile.children.cycles-pp.sctp_data_ready
      4.27            -0.9        3.40 ±  4%  perf-profile.children.cycles-pp.consume_skb
      0.94 ±  2%      -0.8        0.12 ± 17%  perf-profile.children.cycles-pp.schedule_idle
      1.10 ±  3%      -0.8        0.34 ±  7%  perf-profile.children.cycles-pp.try_charge_memcg
      1.80            -0.7        1.07 ±  4%  perf-profile.children.cycles-pp.__wake_up_common_lock
      1.72            -0.7        1.02 ±  4%  perf-profile.children.cycles-pp.__wake_up_common
      5.54            -0.6        4.92 ±  3%  perf-profile.children.cycles-pp.sctp_outq_sack
      1.21 ±  3%      -0.6        0.65 ±  7%  perf-profile.children.cycles-pp.__mod_memcg_state
      1.06 ±  2%      -0.5        0.51 ±  9%  perf-profile.children.cycles-pp.__free_pages_ok
      0.90 ±  2%      -0.5        0.44 ±  6%  perf-profile.children.cycles-pp.refill_stock
     12.34            -0.5       11.88 ±  2%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.62 ±  3%      -0.5        0.17 ± 20%  perf-profile.children.cycles-pp.ipv4_dst_check
      1.45            -0.5        1.00 ±  3%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.09            -0.4        0.65 ±  6%  perf-profile.children.cycles-pp.sctp_packet_config
      1.43            -0.4        0.99 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      0.77 ±  2%      -0.4        0.33 ±  7%  perf-profile.children.cycles-pp.free_unref_page_commit
      1.24            -0.4        0.82 ±  5%  perf-profile.children.cycles-pp.sctp_outq_select_transport
      1.33            -0.4        0.92 ±  5%  perf-profile.children.cycles-pp.kmem_cache_free
      1.62            -0.4        1.22 ±  4%  perf-profile.children.cycles-pp.sctp_wfree
      3.15            -0.4        2.76 ±  5%  perf-profile.children.cycles-pp.sctp_ulpevent_make_rcvmsg
      0.40            -0.3        0.07 ±  6%  perf-profile.children.cycles-pp.select_task_rq
      0.46 ±  2%      -0.3        0.16 ±  7%  perf-profile.children.cycles-pp.sctp_inet_skb_msgname
      0.57 ±  2%      -0.3        0.29 ±  8%  perf-profile.children.cycles-pp.dequeue_entity
      0.33            -0.3        0.06 ±  9%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.49 ±  2%      -0.3        0.23 ±  8%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.76 ±  3%      -0.2        0.51 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.55 ±  3%      -0.2        0.33 ± 16%  perf-profile.children.cycles-pp.dst_release
      0.49            -0.2        0.28 ±  6%  perf-profile.children.cycles-pp.enqueue_entity
      0.38 ±  3%      -0.2        0.18 ±  9%  perf-profile.children.cycles-pp.__free_one_page
      4.26            -0.2        4.07 ±  4%  perf-profile.children.cycles-pp._sctp_make_chunk
      0.70 ±  2%      -0.2        0.51 ±  5%  perf-profile.children.cycles-pp.sctp_eat_data
      0.79 ±  2%      -0.2        0.61 ±  4%  perf-profile.children.cycles-pp.sctp_sf_eat_data_6_2
      0.25 ±  3%      -0.2        0.07 ± 11%  perf-profile.children.cycles-pp.prepare_to_wait_exclusive
      0.60 ±  2%      -0.2        0.42 ±  8%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.58 ±  4%      -0.2        0.42 ±  9%  perf-profile.children.cycles-pp.__copy_skb_header
      0.79 ±  3%      -0.2        0.64 ±  7%  perf-profile.children.cycles-pp.__skb_clone
      1.40            -0.1        1.26 ±  3%  perf-profile.children.cycles-pp.sctp_ulpevent_free
      0.76 ±  2%      -0.1        0.63 ±  5%  perf-profile.children.cycles-pp.sctp_addrs_lookup_transport
      0.79 ±  2%      -0.1        0.66 ±  5%  perf-profile.children.cycles-pp.__sctp_rcv_lookup
      0.45 ±  2%      -0.1        0.32 ±  4%  perf-profile.children.cycles-pp.free_large_kmalloc
      0.53 ±  2%      -0.1        0.40 ±  4%  perf-profile.children.cycles-pp.__zone_watermark_ok
      0.28            -0.1        0.16 ±  8%  perf-profile.children.cycles-pp.prepare_task_switch
      0.60            -0.1        0.49 ±  3%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.84            -0.1        0.72 ±  6%  perf-profile.children.cycles-pp.check_heap_object
      0.58            -0.1        0.49 ±  3%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.62 ±  2%      -0.1        0.52 ±  6%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.33            -0.1        0.24 ±  5%  perf-profile.children.cycles-pp.__switch_to
      0.43            -0.1        0.34 ±  4%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.20            -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.___perf_sw_event
      0.20 ±  2%      -0.1        0.13 ±  7%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.13            -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.perf_tp_event
      0.60            -0.1        0.53 ±  5%  perf-profile.children.cycles-pp.sctp_inq_pop
      0.12 ±  3%      -0.0        0.08 ±  9%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.20 ±  3%      -0.0        0.16 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.15 ±  3%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.__build_skb_around
      0.10 ±  3%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.check_new_pages
      0.05            +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.sctp_sendmsg_check_sflags
      0.06 ±  7%      +0.0        0.08 ±  7%  perf-profile.children.cycles-pp.sctp_assoc_rwnd_decrease
      0.05 ±  7%      +0.0        0.07 ± 10%  perf-profile.children.cycles-pp.skb_pull
      0.10 ±  3%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.07 ±  5%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.ip_send_check
      0.05            +0.0        0.07 ±  9%  perf-profile.children.cycles-pp.sctp_tsnmap_check
      0.08 ±  5%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.sctp_validate_data
      0.14            +0.0        0.16 ±  3%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.09 ±  5%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.07 ±  7%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.09 ±  5%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.__ip_local_out
      0.12 ±  4%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp.sctp_sock_rfree
      0.17 ±  2%      +0.0        0.20 ±  5%  perf-profile.children.cycles-pp.sctp_ulpq_order
      0.06 ± 11%      +0.0        0.09 ± 19%  perf-profile.children.cycles-pp.sctp_get_af_specific
      0.08 ±  4%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.ip_rcv_core
      0.11 ±  6%      +0.0        0.14 ±  9%  perf-profile.children.cycles-pp.ip_local_out
      0.14 ±  5%      +0.0        0.16 ±  7%  perf-profile.children.cycles-pp.check_stack_object
      0.03 ± 70%      +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.sctp_auth_send_cid
      0.10            +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.11 ±  3%      +0.0        0.14 ±  9%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.30 ±  2%      +0.0        0.33 ±  6%  perf-profile.children.cycles-pp.sctp_transport_put
      0.07            +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.16 ±  2%      +0.0        0.20 ±  5%  perf-profile.children.cycles-pp.__put_user_nocheck_8
      0.12 ±  5%      +0.0        0.16 ±  7%  perf-profile.children.cycles-pp.os_xsave
      0.04 ± 44%      +0.0        0.08 ± 41%  perf-profile.children.cycles-pp.inet_sendmsg
      0.12 ±  6%      +0.0        0.16 ±  7%  perf-profile.children.cycles-pp.sctp_cmp_addr_exact
      0.27 ±  2%      +0.0        0.31 ±  6%  perf-profile.children.cycles-pp.sctp_make_control
      0.25 ±  3%      +0.0        0.29 ±  6%  perf-profile.children.cycles-pp.sctp_outq_tail
      0.12 ±  4%      +0.0        0.16 ±  8%  perf-profile.children.cycles-pp.sctp_chunk_assign_ssn
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.update_min_vruntime
      0.05 ±  8%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.check_preempt_curr
      0.11 ±  5%      +0.0        0.15 ± 22%  perf-profile.children.cycles-pp.sctp_hash_key
      0.08 ±  5%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.__sctp_write_space
      0.24 ±  4%      +0.0        0.29 ±  3%  perf-profile.children.cycles-pp.skb_clone
      0.01 ±223%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.18 ±  2%      +0.0        0.23 ±  7%  perf-profile.children.cycles-pp.rcu_all_qs
      0.16 ±  3%      +0.0        0.20 ±  3%  perf-profile.children.cycles-pp.ip_rcv
      0.20            +0.0        0.25 ±  9%  perf-profile.children.cycles-pp.sctp_datamsg_destroy
      0.23 ±  3%      +0.1        0.28 ±  7%  perf-profile.children.cycles-pp.sctp_association_hold
      0.18 ±  5%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.01 ±223%      +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.netif_skb_features
      0.34 ±  3%      +0.1        0.39 ±  5%  perf-profile.children.cycles-pp.sctp_sched_fcfs_dequeue
      0.00            +0.1        0.05 ± 13%  perf-profile.children.cycles-pp.sctp_ulpq_retrieve_ordered
      0.23 ±  4%      +0.1        0.29 ±  5%  perf-profile.children.cycles-pp.sctp_chunk_hold
      0.41 ±  4%      +0.1        0.46 ±  5%  perf-profile.children.cycles-pp.sock_recvmsg
      0.39            +0.1        0.45 ±  5%  perf-profile.children.cycles-pp.send_sctp_stream_1toMany
      0.06            +0.1        0.12 ± 10%  perf-profile.children.cycles-pp.reweight_entity
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.sctp_transport_burst_reset
      0.45            +0.1        0.51 ±  7%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.15 ±  2%      +0.1        0.21 ±  9%  perf-profile.children.cycles-pp.__mod_timer
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.error_entry
      0.17 ±  2%      +0.1        0.24 ±  6%  perf-profile.children.cycles-pp.__netif_rx
      0.15 ±  4%      +0.1        0.21 ±  6%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.16 ±  4%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.netif_rx_internal
      0.27 ±  3%      +0.1        0.34 ±  8%  perf-profile.children.cycles-pp.sctp_sendmsg_parse
      0.00            +0.1        0.07 ± 16%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.46 ±  2%      +0.1        0.53 ±  6%  perf-profile.children.cycles-pp.sctp_check_transmitted
      0.21 ±  2%      +0.1        0.28 ±  8%  perf-profile.children.cycles-pp.__put_user_nocheck_4
      0.12 ± 36%      +0.1        0.19 ± 25%  perf-profile.children.cycles-pp.start_kernel
      0.12 ± 36%      +0.1        0.19 ± 25%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.12 ± 36%      +0.1        0.19 ± 25%  perf-profile.children.cycles-pp.rest_init
      0.24 ±  3%      +0.1        0.31 ±  4%  perf-profile.children.cycles-pp.kfree
      0.25 ±  2%      +0.1        0.33 ±  6%  perf-profile.children.cycles-pp.sctp_datamsg_put
      0.08 ±  4%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.sctp_transport_reset_t3_rtx
      0.59            +0.1        0.66 ±  3%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.28 ±  3%      +0.1        0.36 ±  6%  perf-profile.children.cycles-pp.__genradix_ptr
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.48 ±  2%      +0.1        0.56 ±  3%  perf-profile.children.cycles-pp.free_pcp_prepare
      0.33            +0.1        0.41 ±  6%  perf-profile.children.cycles-pp.kmalloc_trace
      0.30            +0.1        0.38 ±  5%  perf-profile.children.cycles-pp.sctp_association_put
      0.00            +0.1        0.09 ±  7%  perf-profile.children.cycles-pp.put_prev_entity
      0.01 ±223%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.ret_from_fork
      0.01 ±223%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.kthread
      0.03 ±100%      +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.40            +0.1        0.49 ±  6%  perf-profile.children.cycles-pp.sctp_ulpevent_receive_data
      0.30            +0.1        0.40 ±  8%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.42 ±  2%      +0.1        0.52 ±  5%  perf-profile.children.cycles-pp.__kmalloc
      0.33 ±  2%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp.__sctp_packet_append_chunk
      0.33 ±  2%      +0.1        0.43 ±  7%  perf-profile.children.cycles-pp.loopback_xmit
      0.27 ±  4%      +0.1        0.38 ±  7%  perf-profile.children.cycles-pp.update_curr
      0.00            +0.1        0.10 ±  7%  perf-profile.children.cycles-pp._find_next_and_bit
      0.01 ±223%      +0.1        0.11 ± 27%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.26 ±  3%      +0.1        0.37 ±  7%  perf-profile.children.cycles-pp.sock_kfree_s
      0.04 ± 45%      +0.1        0.15 ± 35%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.11            +0.1        0.22 ± 29%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.28 ±  3%      +0.1        0.39 ±  6%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.29 ±  2%      +0.1        0.40 ± 48%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.79 ±  2%      +0.1        0.91 ±  4%  perf-profile.children.cycles-pp.__copy_msghdr
      0.00            +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.36 ±  2%      +0.1        0.48 ±  4%  perf-profile.children.cycles-pp.__fdget
      0.34 ±  2%      +0.1        0.45 ±  8%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.36 ±  2%      +0.1        0.48 ±  8%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.43 ±  3%      +0.1        0.55 ±  6%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.02 ± 99%      +0.1        0.15 ± 33%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.54            +0.1        0.66 ±  5%  perf-profile.children.cycles-pp.sock_kmalloc
      0.33            +0.1        0.46 ±  3%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.1        0.12 ± 40%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.44 ±  2%      +0.1        0.57 ±  4%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.00            +0.1        0.13 ± 42%  perf-profile.children.cycles-pp.wait_for_lsr
      0.50            +0.1        0.63 ±  4%  perf-profile.children.cycles-pp.sctp_chunk_free
      0.00            +0.1        0.13 ± 27%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.13 ± 43%  perf-profile.children.cycles-pp.serial8250_console_write
      0.00            +0.1        0.14 ± 43%  perf-profile.children.cycles-pp.irq_work_single
      0.00            +0.1        0.14 ± 43%  perf-profile.children.cycles-pp.console_unlock
      0.00            +0.1        0.14 ± 43%  perf-profile.children.cycles-pp.console_emit_next_record
      0.04 ± 71%      +0.1        0.17 ± 67%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.00            +0.1        0.14 ± 40%  perf-profile.children.cycles-pp.irq_work_run_list
      0.56 ±  2%      +0.2        0.71 ±  4%  perf-profile.children.cycles-pp.move_addr_to_kernel
      0.48 ±  2%      +0.2        0.65 ± 19%  perf-profile.children.cycles-pp.aa_sk_perm
      0.65            +0.2        0.81 ±  3%  perf-profile.children.cycles-pp.__entry_text_start
      0.54            +0.2        0.70        perf-profile.children.cycles-pp.__might_sleep
      0.78            +0.2        0.94 ±  5%  perf-profile.children.cycles-pp.sctp_chunkify
      0.60 ±  2%      +0.2        0.76 ±  5%  perf-profile.children.cycles-pp.sctp_addto_chunk
      0.66            +0.2        0.82 ±  3%  perf-profile.children.cycles-pp.__might_fault
      0.50            +0.2        0.66 ±  6%  perf-profile.children.cycles-pp.memset_erms
      0.60 ±  2%      +0.2        0.77 ±  4%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.00            +0.2        0.17 ± 52%  perf-profile.children.cycles-pp.rcu_pending
      0.00            +0.2        0.17 ± 29%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.57 ±  2%      +0.2        0.75 ±  6%  perf-profile.children.cycles-pp.sctp_packet_append_chunk
      0.18 ±  3%      +0.2        0.36 ±  5%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.05 ±  8%      +0.2        0.24 ± 21%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.89            +0.2        1.08 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.70 ±  3%      +0.2        0.89 ±  8%  perf-profile.children.cycles-pp.sctp_epaddr_lookup_transport
      0.00            +0.2        0.20 ± 26%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.73 ±  3%      +0.2        0.93 ±  9%  perf-profile.children.cycles-pp.sctp_endpoint_lookup_assoc
      0.74            +0.2        0.94 ±  4%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.06 ± 14%      +0.2        0.28 ± 30%  perf-profile.children.cycles-pp.update_blocked_averages
      0.00            +0.2        0.21 ± 51%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.07            +0.2        0.30 ± 15%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +0.2        0.24 ±  8%  perf-profile.children.cycles-pp.idle_cpu
      0.06 ±  6%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase
      0.09 ±  6%      +0.3        0.34 ±  8%  perf-profile.children.cycles-pp.switch_fpu_return
      0.00            +0.3        0.26 ± 32%  perf-profile.children.cycles-pp.run_rebalance_domains
      1.04 ±  2%      +0.3        1.31 ±  4%  perf-profile.children.cycles-pp.iovec_from_user
      0.05 ±  7%      +0.3        0.32 ± 15%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.3        0.27 ±  7%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.24 ±  3%      +0.3        0.52 ± 18%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.06 ±  7%      +0.3        0.34 ± 15%  perf-profile.children.cycles-pp.sctp_wait_for_sndbuf
      0.18 ±  5%      +0.3        0.46 ±  6%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      1.20 ±  2%      +0.3        1.51 ±  4%  perf-profile.children.cycles-pp.__import_iovec
      1.24 ±  2%      +0.3        1.56 ±  4%  perf-profile.children.cycles-pp.import_iovec
      0.36 ±  4%      +0.3        0.69 ± 14%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.35 ±  2%      +0.4        0.71 ±  5%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.09 ±  6%      +0.4        0.45 ± 10%  perf-profile.children.cycles-pp.rebalance_domains
      0.09 ± 30%      +0.4        0.48 ± 32%  perf-profile.children.cycles-pp.tick_irq_enter
      0.10 ± 27%      +0.4        0.49 ± 31%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.65 ±  2%      +0.4        1.08 ±  6%  perf-profile.children.cycles-pp.sctp_packet_transmit_chunk
      2.88            +0.4        3.31 ±  4%  perf-profile.children.cycles-pp.sctp_outq_flush_data
      0.18 ± 12%      +0.5        0.63 ± 21%  perf-profile.children.cycles-pp.clockevents_program_event
      0.71 ±  7%      +0.5        1.20 ± 27%  perf-profile.children.cycles-pp.ktime_get
      0.01 ±223%      +0.5        0.51 ±  4%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      1.84 ±  2%      +0.5        2.37 ±  2%  perf-profile.children.cycles-pp.schedule_timeout
      0.17 ±  4%      +0.6        0.78 ± 16%  perf-profile.children.cycles-pp.scheduler_tick
      2.44            +0.7        3.10 ±  3%  perf-profile.children.cycles-pp._copy_from_user
      2.95            +0.7        3.64 ±  3%  perf-profile.children.cycles-pp.copy_msghdr_from_user
      0.23 ± 10%      +0.7        0.96 ±  2%  perf-profile.children.cycles-pp.newidle_balance
      0.12 ±  9%      +0.8        0.89 ±  2%  perf-profile.children.cycles-pp.update_sg_lb_stats
      1.53 ±  2%      +0.8        2.35 ±  2%  perf-profile.children.cycles-pp.schedule
      0.13 ±  9%      +0.9        0.98 ±  2%  perf-profile.children.cycles-pp.update_sd_lb_stats
      3.45            +0.9        4.31 ±  3%  perf-profile.children.cycles-pp.copyin
      0.14 ± 10%      +0.9        1.00 ±  2%  perf-profile.children.cycles-pp.find_busiest_group
      0.70 ±  2%      +0.9        1.58 ± 19%  perf-profile.children.cycles-pp.menu_select
      0.47 ±  4%      +0.9        1.36 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      2.38            +0.9        3.28 ±  3%  perf-profile.children.cycles-pp.__schedule
      3.70            +0.9        4.65 ±  2%  perf-profile.children.cycles-pp._copy_from_iter
      0.38            +1.0        1.36 ±  4%  perf-profile.children.cycles-pp.__cond_resched
      0.18 ±  4%      +1.0        1.16 ± 14%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.16 ±  8%      +1.0        1.20 ±  3%  perf-profile.children.cycles-pp.load_balance
      4.16            +1.0        5.20 ±  2%  perf-profile.children.cycles-pp.sctp_user_addto_chunk
      0.24 ±  5%      +1.2        1.44 ± 25%  perf-profile.children.cycles-pp.update_process_times
      0.25 ±  6%      +1.3        1.56 ± 28%  perf-profile.children.cycles-pp.tick_sched_handle
      9.70            +1.3       11.05 ±  3%  perf-profile.children.cycles-pp.sctp_datamsg_from_user
      0.29 ±  5%      +1.4        1.74 ± 25%  perf-profile.children.cycles-pp.tick_sched_timer
      0.38 ±  4%      +2.2        2.57 ± 25%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.63 ±  5%      +3.0        3.63 ± 18%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.64 ±  5%      +3.1        3.72 ± 19%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.99 ±  7%      +4.8        5.79 ± 18%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      1.24 ±  5%      +5.3        6.52 ± 16%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     17.20            +6.5       23.70 ± 13%  perf-profile.children.cycles-pp.start_secondary
     17.32            +6.6       23.88 ± 13%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     17.32            +6.6       23.88 ± 13%  perf-profile.children.cycles-pp.cpu_startup_entry
     17.29            +6.6       23.88 ± 13%  perf-profile.children.cycles-pp.do_idle
     13.26            +8.3       21.54 ± 12%  perf-profile.children.cycles-pp.cpuidle_enter_state
     13.27            +8.3       21.56 ± 12%  perf-profile.children.cycles-pp.cpuidle_enter
     14.17            +9.3       23.44 ± 13%  perf-profile.children.cycles-pp.cpuidle_idle_call
      3.22 ±  4%     +10.7       13.95 ± 12%  perf-profile.children.cycles-pp.intel_idle
     13.62            -2.9       10.68 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      2.42 ±  2%      -2.0        0.45 ± 71%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.36 ±  2%      -1.3        0.06 ±  8%  perf-profile.self.cycles-pp.__slab_free
      1.03 ±  3%      -0.6        0.43 ± 10%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.63 ±  2%      -0.5        0.11 ± 11%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.61 ±  2%      -0.5        0.16 ± 23%  perf-profile.self.cycles-pp.ipv4_dst_check
      1.11 ±  2%      -0.4        0.66 ±  6%  perf-profile.self.cycles-pp.rmqueue
      0.74 ±  2%      -0.4        0.30 ±  8%  perf-profile.self.cycles-pp.free_unref_page_commit
      1.09            -0.4        0.66 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.30            -0.4        0.90 ±  5%  perf-profile.self.cycles-pp.kmem_cache_free
      0.73            -0.4        0.34 ±  7%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.79            -0.4        0.42 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.44 ±  2%      -0.3        0.14 ±  7%  perf-profile.self.cycles-pp.sctp_inet_skb_msgname
      0.45            -0.3        0.16 ±  5%  perf-profile.self.cycles-pp.sctp_skb_recv_datagram
      0.86            -0.3        0.59 ±  8%  perf-profile.self.cycles-pp.sctp_recvmsg
      0.47            -0.3        0.21 ±  9%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.32            -0.3        0.07 ± 11%  perf-profile.self.cycles-pp.schedule_timeout
      0.56 ±  2%      -0.2        0.31 ±  5%  perf-profile.self.cycles-pp.sctp_eat_data
      0.58 ±  2%      -0.2        0.34 ± 13%  perf-profile.self.cycles-pp.__sk_mem_raise_allocated
      0.60            -0.2        0.37 ±  6%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.53 ±  2%      -0.2        0.30 ±  7%  perf-profile.self.cycles-pp.try_charge_memcg
      0.55 ±  3%      -0.2        0.32 ± 17%  perf-profile.self.cycles-pp.dst_release
      0.50            -0.2        0.30 ±  5%  perf-profile.self.cycles-pp.check_heap_object
      0.31            -0.2        0.13 ± 12%  perf-profile.self.cycles-pp.sctp_hash_cmp
      0.36            -0.2        0.19 ± 11%  perf-profile.self.cycles-pp.sctp_data_ready
      0.57 ±  4%      -0.2        0.41 ±  9%  perf-profile.self.cycles-pp.__copy_skb_header
      0.52 ±  2%      -0.1        0.40 ±  4%  perf-profile.self.cycles-pp.__zone_watermark_ok
      0.31 ±  2%      -0.1        0.20 ±  5%  perf-profile.self.cycles-pp.sctp_rcv
      0.17 ±  2%      -0.1        0.06 ± 14%  perf-profile.self.cycles-pp.enqueue_entity
      0.18 ±  4%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.consume_skb
      0.13 ±  5%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.__free_one_page
      0.24 ±  2%      -0.1        0.15 ± 22%  perf-profile.self.cycles-pp.update_rq_clock
      0.32            -0.1        0.24 ±  4%  perf-profile.self.cycles-pp.__switch_to
      0.18            -0.1        0.09 ± 10%  perf-profile.self.cycles-pp.___perf_sw_event
      0.42            -0.1        0.33 ±  4%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.38 ±  2%      -0.1        0.30 ±  4%  perf-profile.self.cycles-pp.skb_release_data
      0.15 ±  4%      -0.1        0.07 ± 24%  perf-profile.self.cycles-pp.do_idle
      0.20 ±  4%      -0.1        0.14 ±  8%  perf-profile.self.cycles-pp.sctp_ulpq_tail_event
      0.29 ±  3%      -0.1        0.24 ±  7%  perf-profile.self.cycles-pp.__schedule
      0.11 ±  6%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.14 ±  3%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.__build_skb_around
      0.12 ±  4%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.09 ±  5%      -0.0        0.04 ± 44%  perf-profile.self.cycles-pp.try_to_wake_up
      0.19 ±  3%      -0.0        0.15 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.14 ±  3%      -0.0        0.10 ± 10%  perf-profile.self.cycles-pp.__sys_recvmsg
      0.26 ±  2%      -0.0        0.22 ±  3%  perf-profile.self.cycles-pp.sctp_assoc_bh_rcv
      0.23 ±  4%      -0.0        0.19 ±  7%  perf-profile.self.cycles-pp.__copy_msghdr
      0.08 ±  6%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.prepare_task_switch
      0.10 ±  5%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.security_socket_recvmsg
      0.10 ±  3%      -0.0        0.07 ±  9%  perf-profile.self.cycles-pp.check_new_pages
      0.08 ±  6%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.finish_task_switch
      0.06 ±  6%      +0.0        0.08 ±  9%  perf-profile.self.cycles-pp.kmalloc_reserve
      0.06 ±  6%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__sk_mem_schedule
      0.16 ±  2%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.sctp_inq_pop
      0.06 ±  7%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.sctp_epaddr_lookup_transport
      0.12 ±  3%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp.sock_kmalloc
      0.09 ±  6%      +0.0        0.11 ± 10%  perf-profile.self.cycles-pp.__sys_sendmsg
      0.07 ±  6%      +0.0        0.09 ± 10%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.07 ±  8%      +0.0        0.09 ± 11%  perf-profile.self.cycles-pp.__sctp_write_space
      0.09 ±  5%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.___sys_sendmsg
      0.08 ±  6%      +0.0        0.10 ±  6%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.11 ±  3%      +0.0        0.13 ±  4%  perf-profile.self.cycles-pp.sctp_sock_rfree
      0.08 ±  5%      +0.0        0.10 ±  9%  perf-profile.self.cycles-pp.lock_sock_nested
      0.08 ±  4%      +0.0        0.10 ±  9%  perf-profile.self.cycles-pp.move_addr_to_kernel
      0.07 ±  5%      +0.0        0.09 ± 10%  perf-profile.self.cycles-pp.sctp_outq_flush_ctrl
      0.11 ±  5%      +0.0        0.13 ±  5%  perf-profile.self.cycles-pp.check_stack_object
      0.08            +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.sctp_packet_transmit_chunk
      0.07 ±  5%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.05 ±  8%      +0.0        0.08 ± 13%  perf-profile.self.cycles-pp.sctp_assoc_rwnd_increase
      0.12            +0.0        0.14 ±  8%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.08 ±  6%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.ip_rcv_core
      0.07 ±  5%      +0.0        0.10 ± 25%  perf-profile.self.cycles-pp.sock_sendmsg
      0.12 ±  3%      +0.0        0.15 ±  4%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.15 ±  3%      +0.0        0.18 ±  5%  perf-profile.self.cycles-pp.update_load_avg
      0.10 ±  4%      +0.0        0.13 ±  7%  perf-profile.self.cycles-pp.process_backlog
      0.06 ± 11%      +0.0        0.09 ± 11%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.11 ±  4%      +0.0        0.14 ±  8%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.08 ±  4%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.sctp_chunk_assign_ssn
      0.20 ±  3%      +0.0        0.23 ±  7%  perf-profile.self.cycles-pp.sctp_check_transmitted
      0.05 ±  8%      +0.0        0.08 ± 11%  perf-profile.self.cycles-pp.sctp_sched_fcfs_dequeue
      0.05            +0.0        0.08 ± 12%  perf-profile.self.cycles-pp.inet_recvmsg
      0.05            +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.12            +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.copy_msghdr_from_user
      0.11 ±  4%      +0.0        0.14 ±  9%  perf-profile.self.cycles-pp._sctp_make_chunk
      0.19 ±  3%      +0.0        0.22 ±  8%  perf-profile.self.cycles-pp.free_unref_page
      0.14 ±  3%      +0.0        0.18 ±  5%  perf-profile.self.cycles-pp.sctp_user_addto_chunk
      0.08 ±  6%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.08 ±  8%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.sctp_cmp_addr_exact
      0.13 ±  3%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.12 ±  4%      +0.0        0.15 ±  5%  perf-profile.self.cycles-pp.__might_fault
      0.16 ±  3%      +0.0        0.19 ±  6%  perf-profile.self.cycles-pp.__put_user_nocheck_8
      0.05            +0.0        0.08 ± 11%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.13 ±  4%      +0.0        0.16 ±  7%  perf-profile.self.cycles-pp.skb_put
      0.12 ±  5%      +0.0        0.16 ±  7%  perf-profile.self.cycles-pp.os_xsave
      0.36            +0.0        0.40 ±  6%  perf-profile.self.cycles-pp.native_sched_clock
      0.27 ±  2%      +0.0        0.31 ±  3%  perf-profile.self.cycles-pp.recvmsg
      0.18 ±  2%      +0.0        0.21 ±  6%  perf-profile.self.cycles-pp.__import_iovec
      0.08 ±  5%      +0.0        0.12 ±  9%  perf-profile.self.cycles-pp.sctp_make_datafrag_empty
      0.18 ±  2%      +0.0        0.21 ±  6%  perf-profile.self.cycles-pp._copy_to_iter
      0.22 ±  4%      +0.0        0.26 ±  7%  perf-profile.self.cycles-pp.do_syscall_64
      0.10 ±  4%      +0.0        0.14 ± 24%  perf-profile.self.cycles-pp.sctp_hash_key
      0.13 ±  3%      +0.0        0.17 ±  6%  perf-profile.self.cycles-pp.sctp_ulpevent_free
      0.11 ±  3%      +0.0        0.16 ± 10%  perf-profile.self.cycles-pp.sctp_outq_sack
      0.11 ±  6%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.update_curr
      0.12 ±  4%      +0.0        0.17 ±  4%  perf-profile.self.cycles-pp.ip_finish_output2
      0.09 ±  7%      +0.0        0.14 ± 11%  perf-profile.self.cycles-pp.sock_kfree_s
      0.19 ±  3%      +0.0        0.24 ±  5%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.32 ±  2%      +0.0        0.36 ±  5%  perf-profile.self.cycles-pp.__alloc_pages
      0.23 ±  2%      +0.0        0.27 ±  5%  perf-profile.self.cycles-pp.__entry_text_start
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.sctp_tsnmap_check
      0.01 ±223%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.16 ±  2%      +0.0        0.21 ± 10%  perf-profile.self.cycles-pp.__sctp_packet_append_chunk
      0.16            +0.0        0.21 ±  5%  perf-profile.self.cycles-pp._copy_from_iter
      0.14 ±  5%      +0.0        0.19 ±  5%  perf-profile.self.cycles-pp.iovec_from_user
      0.22 ±  4%      +0.1        0.27 ±  5%  perf-profile.self.cycles-pp.sctp_chunk_hold
      0.31 ±  5%      +0.1        0.36 ±  8%  perf-profile.self.cycles-pp.sctp_transport_hold
      0.34 ±  3%      +0.1        0.40 ±  4%  perf-profile.self.cycles-pp.sctp_packet_pack
      0.21 ±  2%      +0.1        0.26 ±  5%  perf-profile.self.cycles-pp.____sys_sendmsg
      0.21 ±  3%      +0.1        0.26 ±  6%  perf-profile.self.cycles-pp.sctp_association_hold
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.skb_pull
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.sctp_sendmsg_check_sflags
      0.20 ±  3%      +0.1        0.25 ±  5%  perf-profile.self.cycles-pp.kfree
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.error_entry
      0.17 ±  2%      +0.1        0.23 ±  8%  perf-profile.self.cycles-pp.mem_cgroup_uncharge_skmem
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.loopback_xmit
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.12 ±  4%      +0.1        0.18 ±  9%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.24            +0.1        0.30 ±  6%  perf-profile.self.cycles-pp.sctp_ulpevent_receive_data
      0.23 ±  3%      +0.1        0.29 ±  6%  perf-profile.self.cycles-pp.sctp_datamsg_put
      0.38            +0.1        0.44 ±  6%  perf-profile.self.cycles-pp.send_sctp_stream_1toMany
      0.35 ±  2%      +0.1        0.41 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.06 ±  7%      +0.1        0.13 ±  7%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.23 ±  3%      +0.1        0.29 ±  7%  perf-profile.self.cycles-pp.__genradix_ptr
      0.21 ±  2%      +0.1        0.27 ±  8%  perf-profile.self.cycles-pp.__put_user_nocheck_4
      0.00            +0.1        0.06 ± 19%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.26 ±  3%      +0.1        0.33 ±  8%  perf-profile.self.cycles-pp.sctp_sendmsg_parse
      0.00            +0.1        0.07 ± 25%  perf-profile.self.cycles-pp.intel_idle
      0.01 ±223%      +0.1        0.08 ± 34%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.1        0.07 ± 18%  perf-profile.self.cycles-pp.tick_sched_do_timer
      0.23 ±  5%      +0.1        0.30 ±  3%  perf-profile.self.cycles-pp.sctp_packet_append_chunk
      0.29 ±  3%      +0.1        0.36 ±  4%  perf-profile.self.cycles-pp.sctp_do_sm
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.load_balance
      0.22 ±  2%      +0.1        0.30 ±  4%  perf-profile.self.cycles-pp.sctp_outq_flush
      0.29            +0.1        0.37 ±  5%  perf-profile.self.cycles-pp.sctp_association_put
      0.06 ± 11%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.47 ±  2%      +0.1        0.55 ±  4%  perf-profile.self.cycles-pp.free_pcp_prepare
      0.00            +0.1        0.09 ± 40%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.30 ±  2%      +0.1        0.40 ±  4%  perf-profile.self.cycles-pp._copy_from_user
      0.29            +0.1        0.38 ±  9%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.09 ± 32%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.26 ±  2%      +0.1        0.36 ±  8%  perf-profile.self.cycles-pp.sctp_wfree
      0.00            +0.1        0.10 ±  8%  perf-profile.self.cycles-pp._find_next_and_bit
      0.34            +0.1        0.44 ±  7%  perf-profile.self.cycles-pp.__check_object_size
      0.29 ±  3%      +0.1        0.39 ±  5%  perf-profile.self.cycles-pp.sendmsg
      0.00            +0.1        0.10 ±  7%  perf-profile.self.cycles-pp.__release_sock
      0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.29 ±  5%      +0.1        0.39 ± 25%  perf-profile.self.cycles-pp.aa_sk_perm
      0.56 ±  2%      +0.1        0.66 ±  4%  perf-profile.self.cycles-pp.sctp_sendmsg
      0.32 ±  2%      +0.1        0.42 ±  4%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.27 ±  3%      +0.1        0.37 ±  6%  perf-profile.self.cycles-pp.__kmem_cache_free
      0.11 ±  3%      +0.1        0.21 ± 31%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.32 ±  2%      +0.1        0.43 ±  8%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.10 ± 33%  perf-profile.self.cycles-pp.__update_blocked_fair
      0.46            +0.1        0.56 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.23 ±  2%      +0.1        0.34 ±  4%  perf-profile.self.cycles-pp.__cond_resched
      0.00            +0.1        0.11 ± 32%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.1        0.11 ± 21%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.40 ±  3%      +0.1        0.51 ±  9%  perf-profile.self.cycles-pp.sctp_cmd_interpreter
      0.00            +0.1        0.11 ± 19%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.35 ±  3%      +0.1        0.46 ±  4%  perf-profile.self.cycles-pp.__fdget
      0.32            +0.1        0.44 ±  3%  perf-profile.self.cycles-pp.read_tsc
      0.42            +0.1        0.54 ±  4%  perf-profile.self.cycles-pp.__alloc_skb
      0.41 ±  2%      +0.1        0.52 ±  3%  perf-profile.self.cycles-pp.sctp_sendmsg_to_asoc
      0.42 ±  2%      +0.1        0.54 ±  5%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.01 ±223%      +0.1        0.13 ± 23%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.01 ±223%      +0.1        0.13 ± 32%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.04 ± 71%      +0.1        0.16 ± 71%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.44            +0.1        0.58 ±  4%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.55            +0.1        0.69 ±  3%  perf-profile.self.cycles-pp.sctp_chunk_put
      0.44 ±  2%      +0.1        0.59        perf-profile.self.cycles-pp.__might_sleep
      0.48 ±  2%      +0.2        0.64 ±  5%  perf-profile.self.cycles-pp.memset_erms
      0.47            +0.2        0.64 ±  6%  perf-profile.self.cycles-pp.sctp_outq_flush_data
      0.57            +0.2        0.75 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.05            +0.2        0.24 ± 21%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +0.2        0.20 ± 26%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +0.2        0.23 ±  7%  perf-profile.self.cycles-pp.idle_cpu
      0.07            +0.2        0.30 ± 15%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.63            +0.2        0.88 ±  3%  perf-profile.self.cycles-pp.sctp_datamsg_from_user
      0.00            +0.3        0.27 ±  7%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.38 ± 13%      +0.4        0.78 ± 42%  perf-profile.self.cycles-pp.ktime_get
      0.24 ±  3%      +0.5        0.71 ± 30%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.5        0.50 ±  4%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.10 ± 11%      +0.6        0.70        perf-profile.self.cycles-pp.update_sg_lb_stats
      0.28 ±  3%      +0.7        0.97 ± 14%  perf-profile.self.cycles-pp.cpuidle_enter_state


***************************************************************************************************
lkp-icl-2sp6: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory
=========================================================================================
class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  network/gcc-11/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp6/sctp/stress-ng/60s

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   9399721           -48.4%    4850253 ±  5%  stress-ng.sctp.ops
    156661           -48.4%      80837 ±  5%  stress-ng.sctp.ops_per_sec
    514.00 ±  3%  +41004.5%     211277 ± 10%  stress-ng.time.involuntary_context_switches
     59.67 ±  2%     -51.1%      29.17 ± 10%  stress-ng.time.percent_of_cpu_this_job_got
     36.02 ±  2%     -50.6%      17.80 ± 10%  stress-ng.time.system_time
    970641           -63.3%     356134 ±  5%  stress-ng.time.voluntary_context_switches
     31732           -60.7%      12482 ±  3%  vmstat.system.cs
      0.43            -0.2        0.22 ±  9%  mpstat.cpu.all.sys%
      0.10 ±  3%      -0.0        0.09 ±  9%  mpstat.cpu.all.usr%
      1544 ±  2%     +16.4%       1797 ± 11%  numa-meminfo.node0.Active
      3914 ±  5%     -16.9%       3253 ± 12%  numa-meminfo.node1.Active
      3834 ±  5%     -15.6%       3237 ± 12%  numa-meminfo.node1.Active(anon)
   1154373 ±  2%     -31.7%     788904 ±  8%  numa-numastat.node0.local_node
   1239775 ±  2%     -32.6%     835766 ±  7%  numa-numastat.node0.numa_hit
   1300505 ±  2%     -40.6%     772809 ± 12%  numa-numastat.node1.local_node
   1330886 ±  2%     -36.8%     841742 ± 10%  numa-numastat.node1.numa_hit
    213291 ± 17%     -99.3%       1496 ± 33%  turbostat.C1
      0.35 ± 78%      -0.3        0.00 ±141%  turbostat.C1%
    389879 ±129%   +1006.6%    4314548 ± 91%  turbostat.C6
     16517 ±  3%     -97.4%     425.50 ± 54%  turbostat.POLL
   1239630 ±  2%     -32.6%     835988 ±  7%  numa-vmstat.node0.numa_hit
   1154228 ±  2%     -31.6%     789126 ±  8%  numa-vmstat.node0.numa_local
    958.33 ±  5%     -15.6%     809.17 ± 12%  numa-vmstat.node1.nr_active_anon
    958.33 ±  5%     -15.6%     809.17 ± 12%  numa-vmstat.node1.nr_zone_active_anon
   1330563 ±  2%     -36.7%     841771 ± 10%  numa-vmstat.node1.numa_hit
   1300182 ±  2%     -40.6%     772838 ± 12%  numa-vmstat.node1.numa_local
      2.67 ± 35%  +59431.2%       1587 ±116%  proc-vmstat.numa_hint_faults_local
   2580484           -34.9%    1679518 ±  2%  proc-vmstat.numa_hit
   2464701           -36.6%    1563724 ±  2%  proc-vmstat.numa_local
     14670 ± 55%     +64.3%      24106 ± 21%  proc-vmstat.numa_pte_updates
  18610165           -48.5%    9577564 ±  4%  proc-vmstat.pgalloc_normal
  18478395           -48.9%    9435687 ±  4%  proc-vmstat.pgfree
     18554            -3.3%      17937        proc-vmstat.pgreuse
      6103 ± 43%     -72.4%       1682 ± 36%  sched_debug.cfs_rq:/.load.avg
     50624 ± 58%     -81.4%       9439 ± 83%  sched_debug.cfs_rq:/.load.stddev
    853.50 ± 14%   +3161.6%      27837 ±136%  sched_debug.cfs_rq:/.load_avg.max
    965.00 ±  5%     +52.0%       1466 ± 18%  sched_debug.cfs_rq:/.runnable_avg.max
    223.02 ± 23%     +36.3%     303.95 ± 15%  sched_debug.cfs_rq:/.runnable_avg.stddev
    964.33 ±  5%     +51.4%       1460 ± 18%  sched_debug.cfs_rq:/.util_avg.max
    222.86 ± 23%     +35.4%     301.73 ± 15%  sched_debug.cfs_rq:/.util_avg.stddev
    351971 ± 38%     -41.6%     205390 ± 25%  sched_debug.cpu.avg_idle.stddev
      4.02 ±  8%     +18.1%       4.75 ±  5%  sched_debug.cpu.clock.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
 7.447e+08           -27.7%  5.387e+08 ±  3%  perf-stat.i.branch-instructions
      0.79 ± 20%      +2.6        3.35 ± 75%  perf-stat.i.branch-miss-rate%
   4421153 ±  3%     -29.9%    3100372 ±  5%  perf-stat.i.cache-misses
     32967           -61.7%      12612 ±  4%  perf-stat.i.context-switches
      1.67           +50.8%       2.52 ± 21%  perf-stat.i.cpi
    313.73 ±  3%    +253.6%       1109 ±  4%  perf-stat.i.cpu-migrations
      1477 ±  4%     +55.0%       2289 ± 19%  perf-stat.i.cycles-between-cache-misses
      0.06 ± 42%      +0.3        0.36 ± 77%  perf-stat.i.dTLB-load-miss-rate%
 1.027e+09           -28.5%   7.34e+08 ±  2%  perf-stat.i.dTLB-loads
 5.512e+08           -31.7%  3.762e+08 ±  2%  perf-stat.i.dTLB-stores
 3.762e+09           -27.1%  2.742e+09 ±  3%  perf-stat.i.instructions
      0.61           -29.2%       0.43 ± 21%  perf-stat.i.ipc
    393.08 ±  7%     -73.0%     106.27 ± 76%  perf-stat.i.metric.K/sec
     18.00           -26.4%      13.25 ±  5%  perf-stat.i.metric.M/sec
     51.73 ±  4%     +22.8       74.57 ±  6%  perf-stat.i.node-load-miss-rate%
    225775 ±  5%     +85.3%     418438 ±  3%  perf-stat.i.node-load-misses
     14.54 ±  2%     +17.0       31.53 ± 11%  perf-stat.i.node-store-miss-rate%
    213471 ±  5%     +19.7%     255432 ±  7%  perf-stat.i.node-store-misses
   1489600           -47.6%     780285 ±  7%  perf-stat.i.node-stores
      1.03 ± 14%      +2.2        3.19 ± 64%  perf-stat.overall.branch-miss-rate%
      1.61           +42.0%       2.29 ± 19%  perf-stat.overall.cpi
      1370           +46.6%       2008 ± 17%  perf-stat.overall.cycles-between-cache-misses
      0.06 ± 42%      +0.3        0.31 ± 76%  perf-stat.overall.dTLB-load-miss-rate%
      0.62           -26.7%       0.46 ± 20%  perf-stat.overall.ipc
     53.37 ±  3%     +20.7       74.08 ±  5%  perf-stat.overall.node-load-miss-rate%
     12.54 ±  4%     +12.2       24.74 ±  9%  perf-stat.overall.node-store-miss-rate%
 7.319e+08           -27.8%  5.286e+08 ±  3%  perf-stat.ps.branch-instructions
   4344607 ±  3%     -29.8%    3051326 ±  5%  perf-stat.ps.cache-misses
     32395           -61.7%      12403 ±  3%  perf-stat.ps.context-switches
    308.45 ±  4%    +253.7%       1091 ±  4%  perf-stat.ps.cpu-migrations
 1.009e+09           -28.6%  7.207e+08 ±  2%  perf-stat.ps.dTLB-loads
 5.417e+08           -31.8%  3.695e+08 ±  2%  perf-stat.ps.dTLB-stores
 3.697e+09           -27.2%  2.691e+09 ±  3%  perf-stat.ps.instructions
    222119 ±  5%     +85.2%     411273 ±  3%  perf-stat.ps.node-load-misses
    209895 ±  5%     +19.8%     251498 ±  7%  perf-stat.ps.node-store-misses
   1463189           -47.5%     768638 ±  7%  perf-stat.ps.node-stores
 2.374e+11           -26.9%  1.736e+11 ±  2%  perf-stat.total.instructions
     25.09 ±  2%     -14.3       10.78 ± 29%  perf-profile.calltrace.cycles-pp.stress_sctp
     22.22 ±  2%     -12.6        9.64 ± 30%  perf-profile.calltrace.cycles-pp.sendmsg.stress_sctp
     21.90 ±  2%     -12.4        9.50 ± 29%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendmsg.stress_sctp
     21.80 ±  2%     -12.3        9.47 ± 30%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg.stress_sctp
     21.67 ±  2%     -12.3        9.40 ± 30%  perf-profile.calltrace.cycles-pp.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg.stress_sctp
     21.51 ±  2%     -12.2        9.32 ± 30%  perf-profile.calltrace.cycles-pp.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg
     20.62 ±  2%     -11.7        8.96 ± 30%  perf-profile.calltrace.cycles-pp.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.56 ±  2%     -11.1        8.47 ± 30%  perf-profile.calltrace.cycles-pp.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64
     19.33 ±  2%     -11.0        8.37 ± 30%  perf-profile.calltrace.cycles-pp.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg
     11.82 ±  4%      -8.0        3.85 ± 29%  perf-profile.calltrace.cycles-pp.recvmsg
     11.46 ±  4%      -7.8        3.68 ± 29%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvmsg
     11.40 ±  4%      -7.8        3.65 ± 29%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     11.23 ±  5%      -7.7        3.58 ± 29%  perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     11.04 ±  4%      -7.5        3.50 ± 29%  perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     13.15 ±  3%      -7.2        5.97 ± 32%  perf-profile.calltrace.cycles-pp.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
     10.28 ±  4%      -7.2        3.13 ± 29%  perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.78 ±  4%      -6.9        2.89 ± 29%  perf-profile.calltrace.cycles-pp.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64
      9.74 ±  4%      -6.9        2.85 ± 29%  perf-profile.calltrace.cycles-pp.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg
      6.57 ±  5%      -4.1        2.45 ± 30%  perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      5.83 ±  5%      -3.7        2.14 ± 31%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc
      5.48 ±  2%      -3.5        2.00 ± 26%  perf-profile.calltrace.cycles-pp.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg
      5.59 ±  2%      -3.5        2.13 ± 26%  perf-profile.calltrace.cycles-pp.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
      5.49 ±  2%      -3.4        2.09 ± 26%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      4.61            -3.0        1.66 ± 27%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock
      4.57            -2.9        1.62 ± 27%  perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg
      4.49            -2.9        1.61 ± 27%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock
      4.22 ±  2%      -2.9        1.34 ± 53%  perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      5.33            -2.8        2.50 ± 35%  perf-profile.calltrace.cycles-pp.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      4.28 ±  2%      -2.8        1.45 ± 54%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      4.27 ±  2%      -2.8        1.44 ± 54%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
      4.36 ±  2%      -2.8        1.57 ± 46%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
      4.32 ±  4%      -2.7        1.58 ± 30%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg
      3.96 ±  2%      -2.7        1.24 ± 49%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      3.92 ±  2%      -2.7        1.23 ± 50%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
      3.67 ±  2%      -2.7        0.98 ± 67%  perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      3.72 ±  3%      -2.7        1.06 ± 52%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
      3.72 ±  3%      -2.7        1.06 ± 53%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
      4.00 ±  3%      -2.7        1.34 ± 30%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
      3.83 ±  3%      -2.6        1.19 ± 51%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
      3.81 ±  3%      -2.6        1.19 ± 50%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit
      3.47            -2.6        0.84 ± 83%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      3.80 ±  3%      -2.6        1.18 ± 50%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
      3.21 ±  2%      -2.6        0.61 ± 55%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      4.13 ±  4%      -2.6        1.54 ± 30%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      3.74 ±  3%      -2.6        1.15 ± 50%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
      3.23 ±  2%      -2.5        0.71 ± 29%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      3.09 ±  2%      -2.4        0.71 ±100%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu
      2.51 ±  2%      -2.2        0.36 ±103%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg
      2.38 ±  2%      -2.1        0.32 ±101%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg
      2.35 ±  2%      -2.0        0.31 ±102%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      2.85            -1.8        1.02 ± 23%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
      2.56            -1.7        0.87 ± 25%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      2.24 ±  8%      -1.7        0.57 ± 52%  perf-profile.calltrace.cycles-pp.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      2.40 ±  3%      -1.5        0.89 ± 31%  perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
      1.72 ±  4%      -1.4        0.37 ±104%  perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      2.57 ±  2%      -1.3        1.24 ± 42%  perf-profile.calltrace.cycles-pp.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      1.77 ±  9%      -1.2        0.60 ± 75%  perf-profile.calltrace.cycles-pp.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
      2.05 ±  3%      -1.1        0.95 ± 38%  perf-profile.calltrace.cycles-pp._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.47 ±  4%      -1.0        0.48 ±105%  perf-profile.calltrace.cycles-pp.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      1.54 ±  3%      -0.9        0.60 ± 53%  perf-profile.calltrace.cycles-pp.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      1.19 ±  4%      -0.9        0.32 ±102%  perf-profile.calltrace.cycles-pp._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.22 ±  3%      +0.2        1.39 ±  7%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.02 ±  5%      +0.5        1.47 ±  6%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.19 ±141%      +0.5        0.67 ± 14%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call
      0.17 ±141%      +0.5        0.69 ± 19%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.__irq_exit_rcu
      0.40 ± 71%      +0.6        0.95 ± 18%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.__irq_exit_rcu.sysvec_apic_timer_interrupt
      2.27 ± 14%      +0.6        2.82 ± 10%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +0.6        0.57 ±  3%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.6        0.60 ±  7%  perf-profile.calltrace.cycles-pp.update_rq_clock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.00            +0.6        0.64 ± 18%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start
      0.00            +0.6        0.64 ±  6%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +0.6        0.64 ±  6%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.44 ± 72%      +0.8        1.22 ± 39%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.48 ±  3%      +0.8        2.27 ± 11%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      3.20 ±  3%      +0.8        4.00 ±  7%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.70 ±  2%      +1.1        4.80 ±  7%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.36 ±  4%      +1.9        4.25 ± 23%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      2.45 ±  5%      +2.2        4.70 ± 28%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      3.99 ±  8%      +2.4        6.41 ± 15%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      2.99 ±  2%      +2.4        5.42 ± 24%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      4.96 ±  2%      +3.6        8.54 ± 25%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      9.02 ±  2%      +3.7       12.76 ± 12%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      9.12 ±  2%      +3.9       13.04 ± 12%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     14.90            +6.0       20.88 ± 12%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     20.77            +6.3       27.04 ±  7%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     30.56           +12.3       42.87 ±  5%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     30.80           +12.4       43.18 ±  5%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     54.38           +19.1       73.50 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     53.23           +19.3       72.57 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     60.54           +21.9       82.44 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     60.62           +21.9       82.56 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     60.62           +21.9       82.56 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     59.19           +22.2       81.34 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     61.07           +22.3       83.39 ±  5%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     36.24 ±  2%     -21.4       14.80 ± 28%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     36.09 ±  2%     -21.4       14.74 ± 28%  perf-profile.children.cycles-pp.do_syscall_64
     25.09 ±  2%     -14.3       10.78 ± 29%  perf-profile.children.cycles-pp.stress_sctp
     22.40 ±  2%     -12.6        9.77 ± 30%  perf-profile.children.cycles-pp.sendmsg
     21.69 ±  2%     -12.3        9.44 ± 30%  perf-profile.children.cycles-pp.__sys_sendmsg
     21.54 ±  2%     -12.2        9.36 ± 30%  perf-profile.children.cycles-pp.___sys_sendmsg
     20.65 ±  2%     -11.7        9.00 ± 30%  perf-profile.children.cycles-pp.____sys_sendmsg
     19.57 ±  2%     -11.1        8.50 ± 30%  perf-profile.children.cycles-pp.sctp_sendmsg
     19.56 ±  2%     -11.1        8.50 ± 30%  perf-profile.children.cycles-pp.sock_sendmsg
     14.14            -8.2        5.89 ± 26%  perf-profile.children.cycles-pp.sctp_do_sm
     13.90            -8.1        5.79 ± 26%  perf-profile.children.cycles-pp.sctp_cmd_interpreter
     11.97 ±  4%      -8.1        3.92 ± 29%  perf-profile.children.cycles-pp.recvmsg
     11.24 ±  4%      -7.7        3.58 ± 29%  perf-profile.children.cycles-pp.__sys_recvmsg
     11.06 ±  4%      -7.6        3.51 ± 29%  perf-profile.children.cycles-pp.___sys_recvmsg
     13.19 ±  3%      -7.2        6.00 ± 32%  perf-profile.children.cycles-pp.sctp_sendmsg_to_asoc
     10.29 ±  4%      -7.2        3.13 ± 29%  perf-profile.children.cycles-pp.____sys_recvmsg
      9.97 ±  4%      -7.0        2.95 ± 29%  perf-profile.children.cycles-pp.sctp_recvmsg
      9.79 ±  4%      -6.9        2.89 ± 29%  perf-profile.children.cycles-pp.inet_recvmsg
     10.33 ±  2%      -6.1        4.23 ± 27%  perf-profile.children.cycles-pp.sctp_outq_flush
      9.94            -5.7        4.22 ± 26%  perf-profile.children.cycles-pp.sctp_assoc_bh_rcv
      8.54            -5.0        3.51 ± 27%  perf-profile.children.cycles-pp.sctp_packet_transmit
      7.06 ±  2%      -4.3        2.76 ± 27%  perf-profile.children.cycles-pp.release_sock
      6.76 ±  2%      -4.2        2.52 ± 27%  perf-profile.children.cycles-pp.sctp_backlog_rcv
      6.79 ±  2%      -4.2        2.64 ± 27%  perf-profile.children.cycles-pp.__release_sock
      6.58 ±  5%      -4.1        2.45 ± 31%  perf-profile.children.cycles-pp.sctp_primitive_SEND
      4.28 ±  2%      -3.0        1.32 ± 29%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      5.24 ±  2%      -2.9        2.37 ± 29%  perf-profile.children.cycles-pp.__ip_queue_xmit
      5.36            -2.8        2.51 ± 35%  perf-profile.children.cycles-pp.sctp_datamsg_from_user
      5.15 ±  3%      -2.8        2.32 ± 29%  perf-profile.children.cycles-pp.ip_finish_output2
      5.03 ±  3%      -2.8        2.25 ± 31%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      5.57 ±  3%      -2.8        2.80 ± 26%  perf-profile.children.cycles-pp.net_rx_action
      5.53 ±  3%      -2.8        2.78 ± 26%  perf-profile.children.cycles-pp.process_backlog
      5.54 ±  3%      -2.8        2.78 ± 26%  perf-profile.children.cycles-pp.__napi_poll
      4.93 ±  3%      -2.7        2.20 ± 30%  perf-profile.children.cycles-pp.do_softirq
      5.47 ±  3%      -2.7        2.74 ± 26%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      5.35 ±  3%      -2.7        2.68 ± 26%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      5.35 ±  3%      -2.7        2.68 ± 26%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      5.30 ±  3%      -2.6        2.65 ± 26%  perf-profile.children.cycles-pp.sctp_rcv
      3.24 ±  2%      -2.5        0.71 ± 29%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      3.22 ±  2%      -2.5        0.70 ± 30%  perf-profile.children.cycles-pp.__skb_datagram_iter
      2.36 ±  7%      -2.0        0.35 ± 36%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      2.54 ±  2%      -2.0        0.57 ± 33%  perf-profile.children.cycles-pp._copy_to_iter
      8.20            -1.9        6.30 ±  7%  perf-profile.children.cycles-pp.__softirqentry_text_start
      2.40 ±  2%      -1.9        0.49 ± 34%  perf-profile.children.cycles-pp.copyout
      2.14 ±  6%      -1.7        0.46 ± 30%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      2.25 ±  8%      -1.6        0.65 ± 25%  perf-profile.children.cycles-pp.sctp_skb_recv_datagram
      2.01 ±  5%      -1.5        0.46 ± 30%  perf-profile.children.cycles-pp.lock_sock_nested
      2.60 ±  3%      -1.5        1.05 ± 27%  perf-profile.children.cycles-pp.sctp_ulpq_tail_data
      2.52 ±  3%      -1.5        1.00 ± 25%  perf-profile.children.cycles-pp.sctp_chunk_put
      2.70 ±  4%      -1.5        1.23 ± 24%  perf-profile.children.cycles-pp.sctp_outq_sack
      2.44 ±  3%      -1.4        1.00 ± 28%  perf-profile.children.cycles-pp.sctp_packet_pack
      2.33 ±  6%      -1.4        0.97 ± 34%  perf-profile.children.cycles-pp.__alloc_skb
      2.58 ±  2%      -1.3        1.25 ± 42%  perf-profile.children.cycles-pp.sctp_make_datafrag_empty
      2.10 ±  4%      -1.2        0.93 ± 32%  perf-profile.children.cycles-pp.memcpy_erms
      2.19 ±  4%      -1.2        1.03 ± 38%  perf-profile.children.cycles-pp._sctp_make_chunk
      1.76 ±  5%      -1.1        0.68 ± 27%  perf-profile.children.cycles-pp.sctp_ulpevent_make_rcvmsg
      2.17 ±  7%      -1.1        1.11 ± 27%  perf-profile.children.cycles-pp.sctp_outq_flush_data
      1.74 ±  3%      -1.1        0.67 ± 26%  perf-profile.children.cycles-pp.skb_release_head_state
      1.75 ±  3%      -1.0        0.74 ± 24%  perf-profile.children.cycles-pp.consume_skb
      1.38 ±  7%      -1.0        0.39 ± 34%  perf-profile.children.cycles-pp.kfree_skb_reason
      1.60 ±  8%      -1.0        0.62 ± 41%  perf-profile.children.cycles-pp.kmalloc_reserve
      1.59 ±  9%      -1.0        0.60 ± 41%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      1.57 ±  3%      -0.9        0.68 ± 29%  perf-profile.children.cycles-pp.sctp_user_addto_chunk
      1.27 ±  2%      -0.8        0.43 ± 31%  perf-profile.children.cycles-pp.skb_release_data
      1.49 ±  4%      -0.8        0.67 ± 26%  perf-profile.children.cycles-pp.copy_msghdr_from_user
      1.30 ±  9%      -0.7        0.60 ± 28%  perf-profile.children.cycles-pp._copy_from_user
      1.20 ±  5%      -0.7        0.52 ± 27%  perf-profile.children.cycles-pp._copy_from_iter
      1.32 ±  5%      -0.7        0.65 ± 39%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      1.52 ±  9%      -0.7        0.86 ± 22%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.05 ±  5%      -0.6        0.40 ± 25%  perf-profile.children.cycles-pp.copyin
      0.96 ±  5%      -0.6        0.34 ± 32%  perf-profile.children.cycles-pp.__sk_mem_schedule
      0.92 ±  5%      -0.6        0.32 ± 33%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      0.90 ±  8%      -0.6        0.34 ± 23%  perf-profile.children.cycles-pp.__check_object_size
      0.83 ±  3%      -0.5        0.28 ± 28%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.72 ± 13%      -0.5        0.19 ± 28%  perf-profile.children.cycles-pp.__alloc_pages
      0.68 ± 12%      -0.5        0.16 ± 26%  perf-profile.children.cycles-pp.__kmalloc_large_node
      0.90 ±  6%      -0.5        0.38 ± 28%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.77 ±  5%      -0.5        0.27 ± 38%  perf-profile.children.cycles-pp.__close
      0.72 ±  4%      -0.5        0.22 ± 33%  perf-profile.children.cycles-pp.kmem_cache_free
      0.75 ±  5%      -0.5        0.26 ± 38%  perf-profile.children.cycles-pp.task_work_run
      0.65 ± 16%      -0.5        0.16 ± 33%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.74 ±  6%      -0.5        0.26 ± 39%  perf-profile.children.cycles-pp.__fput
      0.83 ±  5%      -0.5        0.35 ± 29%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.75 ±  6%      -0.5        0.27 ± 36%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.71 ±  5%      -0.5        0.23 ± 34%  perf-profile.children.cycles-pp.__slab_free
      0.64 ± 10%      -0.5        0.17 ± 23%  perf-profile.children.cycles-pp.sctp_outq_select_transport
      0.68 ±  5%      -0.5        0.22 ± 46%  perf-profile.children.cycles-pp.sock_close
      0.68 ±  5%      -0.5        0.22 ± 46%  perf-profile.children.cycles-pp.__sock_release
      0.68 ±  5%      -0.5        0.22 ± 46%  perf-profile.children.cycles-pp.inet_release
      0.82 ± 10%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.connect
      0.81 ±  9%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.__x64_sys_connect
      0.81 ±  9%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.__sys_connect
      0.68 ±  5%      -0.5        0.22 ± 46%  perf-profile.children.cycles-pp.sctp_close
      0.81 ±  9%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.sctp_inet_connect
      0.81 ±  9%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.__sctp_connect
      0.81 ±  3%      -0.4        0.37 ± 21%  perf-profile.children.cycles-pp.sctp_wfree
      0.55 ± 17%      -0.4        0.12 ± 38%  perf-profile.children.cycles-pp.rmqueue
      0.55 ± 15%      -0.4        0.13 ± 29%  perf-profile.children.cycles-pp.sctp_packet_config
      0.78 ±  2%      -0.4        0.36 ± 25%  perf-profile.children.cycles-pp.import_iovec
      0.66 ± 11%      -0.4        0.25 ± 34%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
      0.75 ±  3%      -0.4        0.35 ± 24%  perf-profile.children.cycles-pp.__import_iovec
      0.69 ±  6%      -0.4        0.29 ± 27%  perf-profile.children.cycles-pp.sctp_ulpq_tail_event
      0.69 ±  5%      -0.4        0.30 ± 32%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.64 ± 10%      -0.4        0.27 ± 28%  perf-profile.children.cycles-pp.__might_resched
      0.60 ±  6%      -0.4        0.25 ± 36%  perf-profile.children.cycles-pp.sctp_chunkify
      0.38 ±  4%      -0.4        0.03 ±102%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.63 ±  3%      -0.3        0.29 ± 25%  perf-profile.children.cycles-pp.iovec_from_user
      0.50 ±  7%      -0.3        0.16 ± 30%  perf-profile.children.cycles-pp.check_heap_object
      0.56 ±  6%      -0.3        0.25 ± 44%  perf-profile.children.cycles-pp.___slab_alloc
      0.51 ± 14%      -0.3        0.20 ± 28%  perf-profile.children.cycles-pp.__skb_clone
      0.94 ±  8%      -0.3        0.65 ± 29%  perf-profile.children.cycles-pp.sctp_packet_transmit_chunk
      0.41 ±  3%      -0.3        0.13 ± 23%  perf-profile.children.cycles-pp.sctp_sf_eat_data_6_2
      0.53 ±  5%      -0.3        0.25 ± 33%  perf-profile.children.cycles-pp.sctp_data_ready
      0.33 ±  5%      -0.3        0.05 ± 59%  perf-profile.children.cycles-pp.__free_pages_ok
      0.40 ±  2%      -0.3        0.13 ± 37%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.35 ±  4%      -0.3        0.10 ± 21%  perf-profile.children.cycles-pp.sctp_eat_data
      0.51 ± 12%      -0.2        0.26 ± 48%  perf-profile.children.cycles-pp.sctp_outq_flush_ctrl
      0.43 ± 12%      -0.2        0.19 ± 28%  perf-profile.children.cycles-pp.__entry_text_start
      0.43 ±  5%      -0.2        0.19 ± 27%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.33 ±  7%      -0.2        0.10 ± 19%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.32 ± 30%      -0.2        0.09 ± 36%  perf-profile.children.cycles-pp.try_charge_memcg
      0.39 ± 16%      -0.2        0.16 ± 24%  perf-profile.children.cycles-pp.__might_sleep
      0.32 ± 11%      -0.2        0.10 ± 16%  perf-profile.children.cycles-pp.schedule_idle
      0.34 ± 20%      -0.2        0.13 ± 22%  perf-profile.children.cycles-pp.__copy_skb_header
      0.44 ±  4%      -0.2        0.24 ± 22%  perf-profile.children.cycles-pp.__wake_up_common
      0.29 ±  7%      -0.2        0.08 ± 21%  perf-profile.children.cycles-pp.sctp_inq_pop
      0.37 ±  9%      -0.2        0.16 ± 38%  perf-profile.children.cycles-pp.memset_erms
      0.38 ± 12%      -0.2        0.18 ± 29%  perf-profile.children.cycles-pp.__might_fault
      0.25 ±  8%      -0.2        0.05 ± 72%  perf-profile.children.cycles-pp.dst_release
      0.46 ±  5%      -0.2        0.26 ± 20%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.36 ±  3%      -0.2        0.16 ± 25%  perf-profile.children.cycles-pp.sock_kmalloc
      0.29 ± 14%      -0.2        0.10 ± 42%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.33 ±  5%      -0.2        0.15 ± 41%  perf-profile.children.cycles-pp.sctp_wait_for_connect
      0.29 ± 12%      -0.2        0.12 ± 33%  perf-profile.children.cycles-pp.sctp_ulpevent_receive_data
      0.28 ± 16%      -0.2        0.11 ± 31%  perf-profile.children.cycles-pp.refill_stock
      0.39 ±  3%      -0.2        0.22 ± 20%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.32 ±  9%      -0.2        0.15 ± 22%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.34 ±  4%      -0.2        0.18 ± 31%  perf-profile.children.cycles-pp.sctp_packet_append_chunk
      0.42 ±  3%      -0.2        0.26 ± 17%  perf-profile.children.cycles-pp.try_to_wake_up
      0.29 ±  8%      -0.2        0.13 ± 31%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.35 ±  4%      -0.2        0.19 ± 47%  perf-profile.children.cycles-pp.sctp_endpoint_bh_rcv
      0.18 ± 10%      -0.1        0.04 ±101%  perf-profile.children.cycles-pp.free_unref_page
      0.23 ±  4%      -0.1        0.09 ± 36%  perf-profile.children.cycles-pp.stress_sctp_server
      0.28 ±  8%      -0.1        0.13 ± 31%  perf-profile.children.cycles-pp.__kmalloc
      0.26 ±  9%      -0.1        0.12 ± 33%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.25 ±  3%      -0.1        0.12 ± 42%  perf-profile.children.cycles-pp.kmalloc_trace
      0.31 ± 12%      -0.1        0.18 ± 19%  perf-profile.children.cycles-pp.sctp_chunk_free
      0.20 ± 16%      -0.1        0.07 ± 69%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.27 ±  4%      -0.1        0.14 ± 33%  perf-profile.children.cycles-pp.aa_sk_perm
      0.24 ±  3%      -0.1        0.12 ± 19%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.18 ± 11%      -0.1        0.06 ± 84%  perf-profile.children.cycles-pp.__genradix_ptr
      0.20 ± 12%      -0.1        0.08 ± 25%  perf-profile.children.cycles-pp.skb_clone
      0.28 ± 10%      -0.1        0.16 ± 27%  perf-profile.children.cycles-pp.sctp_ulpevent_read_sndrcvinfo
      0.21 ±  5%      -0.1        0.10 ± 39%  perf-profile.children.cycles-pp.__fdget
      0.19 ±  9%      -0.1        0.07 ± 21%  perf-profile.children.cycles-pp.setsockopt
      0.25 ±  8%      -0.1        0.14 ± 28%  perf-profile.children.cycles-pp.sctp_check_transmitted
      0.15 ± 14%      -0.1        0.04 ±101%  perf-profile.children.cycles-pp.do_accept
      0.18 ± 26%      -0.1        0.06 ± 79%  perf-profile.children.cycles-pp.sctp_sendmsg_parse
      0.16 ± 13%      -0.1        0.05 ± 76%  perf-profile.children.cycles-pp.accept
      0.16 ± 16%      -0.1        0.05 ± 74%  perf-profile.children.cycles-pp.__x64_sys_accept
      0.16 ± 16%      -0.1        0.05 ± 74%  perf-profile.children.cycles-pp.__sys_accept4
      0.16 ± 16%      -0.1        0.05 ± 85%  perf-profile.children.cycles-pp.get_partial_node
      0.20 ±  6%      -0.1        0.09 ± 29%  perf-profile.children.cycles-pp.__sctp_packet_append_chunk
      0.15 ± 14%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.__x64_sys_setsockopt
      0.13 ± 12%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__copy_msghdr
      0.18 ± 16%      -0.1        0.08 ± 29%  perf-profile.children.cycles-pp.sock_kfree_s
      0.15 ± 17%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.__sys_setsockopt
      0.21 ±  2%      -0.1        0.10 ± 24%  perf-profile.children.cycles-pp.sock_recvmsg
      0.19 ±  5%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.enqueue_entity
      0.24 ± 12%      -0.1        0.14 ± 29%  perf-profile.children.cycles-pp.put_cmsg
      0.22 ± 12%      -0.1        0.12 ± 14%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.16 ±  6%      -0.1        0.06 ± 47%  perf-profile.children.cycles-pp.sock_wfree
      0.19 ± 10%      -0.1        0.09 ± 43%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.18 ±  5%      -0.1        0.09 ± 26%  perf-profile.children.cycles-pp.sctp_sched_fcfs_dequeue
      0.20 ± 15%      -0.1        0.10 ± 32%  perf-profile.children.cycles-pp.dequeue_entity
      0.21 ±  8%      -0.1        0.11 ± 18%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.21 ±  9%      -0.1        0.11 ± 37%  perf-profile.children.cycles-pp.sctp_association_put
      0.19 ±  7%      -0.1        0.10 ± 29%  perf-profile.children.cycles-pp.skb_put
      0.17 ±  8%      -0.1        0.08 ± 31%  perf-profile.children.cycles-pp.kfree
      0.16 ±  5%      -0.1        0.07 ± 56%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.15 ±  5%      -0.1        0.06 ± 51%  perf-profile.children.cycles-pp.sctp_sched_dequeue_common
      0.18 ±  2%      -0.1        0.09 ± 22%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.12 ± 11%      -0.1        0.03 ±101%  perf-profile.children.cycles-pp.inet_accept
      0.13 ± 16%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.sctp_setsockopt
      0.13 ± 18%      -0.1        0.04 ± 77%  perf-profile.children.cycles-pp.sctp_ulpq_order
      0.12 ± 16%      -0.1        0.03 ±106%  perf-profile.children.cycles-pp.sctp_ulpevent_init
      0.14 ± 17%      -0.1        0.05 ± 55%  perf-profile.children.cycles-pp.sctp_connect_new_asoc
      0.14 ±  3%      -0.1        0.05 ± 73%  perf-profile.children.cycles-pp.sctp_outq_tail
      0.11 ± 12%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.sctp_accept
      0.21 ±  8%      -0.1        0.13 ± 17%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.15 ± 10%      -0.1        0.08 ± 33%  perf-profile.children.cycles-pp.sctp_association_hold
      0.13 ±  3%      -0.1        0.06 ± 53%  perf-profile.children.cycles-pp.__ksize
      0.17 ± 12%      -0.1        0.10 ± 34%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.12 ± 27%      -0.1        0.05 ± 46%  perf-profile.children.cycles-pp.__check_heap_object
      0.14 ± 21%      -0.1        0.07 ± 18%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.13 ± 16%      -0.1        0.06 ± 78%  perf-profile.children.cycles-pp.sctp_chunk_hold
      0.12 ±  3%      -0.1        0.05 ± 49%  perf-profile.children.cycles-pp.__x64_sys_getsockopt
      0.11 ± 11%      -0.1        0.04 ±109%  perf-profile.children.cycles-pp.__put_user_nocheck_8
      0.15 ±  8%      -0.1        0.08 ± 17%  perf-profile.children.cycles-pp.getsockopt
      0.13 ± 26%      -0.1        0.06 ± 20%  perf-profile.children.cycles-pp.loopback_xmit
      0.10 ±  8%      -0.1        0.03 ±108%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.15 ± 14%      -0.1        0.08 ± 24%  perf-profile.children.cycles-pp.sctp_gen_sack
      0.09 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__socket
      0.09 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__sys_socket
      0.09 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__x64_sys_socket
      0.21 ±  2%      -0.1        0.15 ±  8%  perf-profile.children.cycles-pp.update_load_avg
      0.12 ± 14%      -0.1        0.05 ± 76%  perf-profile.children.cycles-pp.rcu_all_qs
      0.11 ±  8%      -0.1        0.05 ± 49%  perf-profile.children.cycles-pp.__sys_getsockopt
      0.15 ± 11%      -0.1        0.09 ± 29%  perf-profile.children.cycles-pp.sctp_make_sack
      0.09 ±  9%      -0.1        0.03 ±103%  perf-profile.children.cycles-pp.sctp_chunk_abandoned
      0.09 ± 30%      -0.1        0.04 ± 75%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.09 ± 10%      -0.1        0.04 ± 75%  perf-profile.children.cycles-pp.sctp_getsockopt
      0.10 ±  9%      -0.0        0.05 ± 45%  perf-profile.children.cycles-pp.sctp_primitive_SHUTDOWN
      0.08 ±  5%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.prepare_task_switch
      0.13 ± 12%      +0.0        0.16 ± 12%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.04 ± 71%      +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.02 ±141%      +0.1        0.09 ± 14%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.23 ±  5%      +0.1        0.31 ± 11%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.02 ±141%      +0.1        0.10 ± 33%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.07 ± 18%      +0.1        0.16 ± 52%  perf-profile.children.cycles-pp.ct_nmi_enter
      0.25 ± 13%      +0.1        0.35 ± 14%  perf-profile.children.cycles-pp.memcpy_toio
      0.25 ± 13%      +0.1        0.35 ± 15%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.25 ± 13%      +0.1        0.35 ± 15%  perf-profile.children.cycles-pp.drm_fb_helper_damage_blit_real
      0.11 ±  4%      +0.1        0.21 ± 33%  perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase
      0.30 ± 13%      +0.1        0.40 ± 13%  perf-profile.children.cycles-pp.process_one_work
      0.09 ± 15%      +0.1        0.20 ± 28%  perf-profile.children.cycles-pp._find_next_and_bit
      0.32 ±  8%      +0.1        0.44 ± 10%  perf-profile.children.cycles-pp.worker_thread
      0.25            +0.1        0.38 ± 22%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.32 ± 19%      +0.1        0.45 ±  9%  perf-profile.children.cycles-pp.io_serial_in
      0.46 ± 16%      +0.1        0.60 ± 10%  perf-profile.children.cycles-pp.rcu_core
      0.23 ±  7%      +0.1        0.38 ± 14%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.18 ±  9%      +0.2        0.33 ± 20%  perf-profile.children.cycles-pp.idle_cpu
      0.26            +0.2        0.42 ± 21%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.25 ± 10%      +0.2        0.41 ± 13%  perf-profile.children.cycles-pp.note_gp_changes
      0.20 ± 10%      +0.2        0.36 ± 17%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.43 ±  3%      +0.2        0.60 ±  4%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      1.27 ±  3%      +0.2        1.44 ±  8%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.58 ±  4%      +0.2        0.76 ±  8%  perf-profile.children.cycles-pp.update_rq_clock
      0.34            +0.2        0.52 ± 26%  perf-profile.children.cycles-pp.ct_idle_exit
      0.44 ± 18%      +0.2        0.63 ± 12%  perf-profile.children.cycles-pp.wait_for_lsr
      0.45 ±  9%      +0.2        0.64 ±  6%  perf-profile.children.cycles-pp.ret_from_fork
      0.44 ± 10%      +0.2        0.64 ±  6%  perf-profile.children.cycles-pp.kthread
      0.45 ± 16%      +0.2        0.65 ± 13%  perf-profile.children.cycles-pp.serial8250_console_write
      0.50 ± 12%      +0.2        0.71 ± 13%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.irq_work_run
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.irq_work_single
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp._printk
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.vprintk_emit
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.console_unlock
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.console_emit_next_record
      0.40 ± 10%      +0.2        0.62 ± 23%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      1.05 ±  3%      +0.2        1.28 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
      0.48 ± 19%      +0.2        0.71 ± 11%  perf-profile.children.cycles-pp.irq_work_run_list
      1.29 ±  2%      +0.2        1.54 ±  9%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.08 ± 20%      +0.3        0.37 ± 63%  perf-profile.children.cycles-pp.check_cpu_stall
      0.71 ±  4%      +0.3        1.01 ± 11%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.74 ±  4%      +0.3        1.06 ± 24%  perf-profile.children.cycles-pp.schedule
      0.34 ±  2%      +0.3        0.65 ± 39%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.46 ± 34%      +0.4        0.83 ± 33%  perf-profile.children.cycles-pp.start_kernel
      0.46 ± 34%      +0.4        0.83 ± 33%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.46 ± 34%      +0.4        0.83 ± 33%  perf-profile.children.cycles-pp.rest_init
      0.37 ±  8%      +0.4        0.76 ± 25%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.19 ± 16%      +0.4        0.58 ± 26%  perf-profile.children.cycles-pp.sctp_wait_for_sndbuf
      0.31 ±  9%      +0.4        0.75 ± 27%  perf-profile.children.cycles-pp.newidle_balance
      0.26 ±  6%      +0.4        0.70 ± 40%  perf-profile.children.cycles-pp.rcu_pending
      1.04 ±  4%      +0.5        1.50 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
      0.51 ± 15%      +0.5        1.03 ± 14%  perf-profile.children.cycles-pp.update_sg_lb_stats
      2.33 ± 14%      +0.6        2.89 ± 10%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.34 ±  2%      +0.6        0.91 ± 41%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.62 ± 12%      +0.6        1.26 ±  9%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.64 ± 18%      +0.6        1.27 ± 37%  perf-profile.children.cycles-pp.tick_irq_enter
      0.65 ± 10%      +0.6        1.30 ±  8%  perf-profile.children.cycles-pp.find_busiest_group
      1.59 ±  3%      +0.8        2.36 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.85 ±  8%      +0.8        1.66 ±  7%  perf-profile.children.cycles-pp.load_balance
      3.82 ±  3%      +1.1        4.90 ±  7%  perf-profile.children.cycles-pp.__irq_exit_rcu
      2.48 ±  5%      +1.9        4.36 ± 23%  perf-profile.children.cycles-pp.update_process_times
      2.56 ±  5%      +2.2        4.76 ± 28%  perf-profile.children.cycles-pp.tick_sched_handle
      3.13 ±  3%      +2.4        5.52 ± 24%  perf-profile.children.cycles-pp.tick_sched_timer
      4.07 ±  8%      +2.5        6.53 ± 15%  perf-profile.children.cycles-pp.menu_select
      5.17 ±  3%      +3.5        8.70 ± 25%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      9.32 ±  2%      +3.7       12.99 ± 11%  perf-profile.children.cycles-pp.hrtimer_interrupt
      9.38 ±  2%      +3.8       13.22 ± 12%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
     15.33 ±  2%      +5.9       21.22 ± 12%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
     18.70            +6.0       24.66 ±  9%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     31.40           +11.9       43.30 ±  5%  perf-profile.children.cycles-pp.mwait_idle_with_hints
     31.04           +12.6       43.59 ±  5%  perf-profile.children.cycles-pp.intel_idle
     54.74           +19.4       74.17 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter_state
     54.81           +19.5       74.31 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter
     60.62           +21.9       82.56 ±  5%  perf-profile.children.cycles-pp.start_secondary
     61.07           +22.3       83.39 ±  5%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     61.07           +22.3       83.39 ±  5%  perf-profile.children.cycles-pp.cpu_startup_entry
     61.07           +22.3       83.38 ±  5%  perf-profile.children.cycles-pp.do_idle
     59.68           +22.6       82.26 ±  5%  perf-profile.children.cycles-pp.cpuidle_idle_call
      4.21 ±  2%      -2.9        1.30 ± 29%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      2.35 ±  7%      -2.0        0.35 ± 36%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      2.09 ±  4%      -1.2        0.90 ± 29%  perf-profile.self.cycles-pp.memcpy_erms
      0.99 ±  6%      -0.5        0.47 ± 28%  perf-profile.self.cycles-pp.sctp_datamsg_from_user
      0.68 ±  7%      -0.5        0.21 ± 33%  perf-profile.self.cycles-pp.kmem_cache_free
      0.69 ±  5%      -0.5        0.22 ± 34%  perf-profile.self.cycles-pp.__slab_free
      0.58 ± 10%      -0.3        0.24 ± 29%  perf-profile.self.cycles-pp.__might_resched
      0.45 ±  6%      -0.3        0.14 ± 39%  perf-profile.self.cycles-pp.sctp_recvmsg
      0.50 ±  4%      -0.3        0.24 ± 28%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.30 ± 10%      -0.2        0.07 ± 47%  perf-profile.self.cycles-pp.check_heap_object
      0.38 ± 10%      -0.2        0.16 ± 32%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.34 ± 20%      -0.2        0.12 ± 23%  perf-profile.self.cycles-pp.__copy_skb_header
      0.38 ±  3%      -0.2        0.17 ± 28%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.37 ±  9%      -0.2        0.16 ± 40%  perf-profile.self.cycles-pp.memset_erms
      0.25 ±  6%      -0.2        0.05 ± 72%  perf-profile.self.cycles-pp.dst_release
      0.32 ±  7%      -0.2        0.13 ± 25%  perf-profile.self.cycles-pp.sctp_chunk_put
      0.26 ± 15%      -0.2        0.07 ± 31%  perf-profile.self.cycles-pp.__sk_mem_raise_allocated
      0.27 ±  6%      -0.2        0.08 ± 36%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.27 ±  3%      -0.2        0.08 ± 30%  perf-profile.self.cycles-pp.sctp_packet_pack
      0.30 ±  8%      -0.2        0.12 ± 38%  perf-profile.self.cycles-pp.sctp_sendmsg
      0.25 ± 11%      -0.2        0.08 ± 27%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.22 ± 11%      -0.2        0.04 ± 72%  perf-profile.self.cycles-pp.sctp_eat_data
      0.24 ± 13%      -0.2        0.06 ± 47%  perf-profile.self.cycles-pp.sctp_packet_config
      0.31 ±  3%      -0.2        0.14 ± 31%  perf-profile.self.cycles-pp.sctp_outq_flush_data
      0.28 ±  7%      -0.2        0.13 ± 29%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.30 ±  4%      -0.2        0.15 ± 52%  perf-profile.self.cycles-pp.___slab_alloc
      0.28            -0.2        0.13 ± 28%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.29 ±  2%      -0.2        0.14 ± 21%  perf-profile.self.cycles-pp.__might_sleep
      0.22 ±  3%      -0.1        0.08 ± 54%  perf-profile.self.cycles-pp.stress_sctp_server
      0.22 ±  4%      -0.1        0.08 ± 75%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.27 ± 10%      -0.1        0.13 ± 26%  perf-profile.self.cycles-pp.sctp_sendmsg_to_asoc
      0.20 ± 16%      -0.1        0.06 ± 88%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.17 ± 12%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.sctp_skb_recv_datagram
      0.25 ± 11%      -0.1        0.12 ± 34%  perf-profile.self.cycles-pp.__alloc_skb
      0.24 ±  5%      -0.1        0.11 ± 20%  perf-profile.self.cycles-pp.__kmem_cache_free
      0.20 ± 12%      -0.1        0.08 ± 41%  perf-profile.self.cycles-pp.sendmsg
      0.18 ±  9%      -0.1        0.06 ± 54%  perf-profile.self.cycles-pp.sctp_ulpevent_receive_data
      0.20 ± 12%      -0.1        0.08 ± 52%  perf-profile.self.cycles-pp._copy_from_user
      0.16 ±  6%      -0.1        0.04 ± 73%  perf-profile.self.cycles-pp.sock_wfree
      0.15 ± 21%      -0.1        0.04 ±102%  perf-profile.self.cycles-pp.sctp_chunkify
      0.21 ±  6%      -0.1        0.10 ± 41%  perf-profile.self.cycles-pp.__fdget
      0.24 ± 13%      -0.1        0.13 ± 24%  perf-profile.self.cycles-pp.sctp_cmd_interpreter
      0.19 ± 18%      -0.1        0.08 ± 39%  perf-profile.self.cycles-pp.__list_add_valid
      0.17 ± 17%      -0.1        0.06 ± 68%  perf-profile.self.cycles-pp.try_charge_memcg
      0.17 ± 24%      -0.1        0.06 ± 79%  perf-profile.self.cycles-pp.sctp_sendmsg_parse
      0.24 ± 12%      -0.1        0.13 ± 26%  perf-profile.self.cycles-pp.put_cmsg
      0.16 ±  5%      -0.1        0.06 ± 47%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.19 ±  2%      -0.1        0.08 ± 22%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.15 ± 17%      -0.1        0.04 ±101%  perf-profile.self.cycles-pp.sctp_data_ready
      0.19 ± 17%      -0.1        0.09 ± 23%  perf-profile.self.cycles-pp.__check_object_size
      0.13 ±  9%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__copy_msghdr
      0.19 ± 22%      -0.1        0.09 ± 27%  perf-profile.self.cycles-pp.refill_stock
      0.17 ±  9%      -0.1        0.08 ± 60%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.16 ± 17%      -0.1        0.06 ± 60%  perf-profile.self.cycles-pp.sctp_do_sm
      0.16 ±  7%      -0.1        0.07 ± 58%  perf-profile.self.cycles-pp.__skb_clone
      0.19 ±  6%      -0.1        0.09 ± 38%  perf-profile.self.cycles-pp.sctp_wfree
      0.17 ±  4%      -0.1        0.08 ± 28%  perf-profile.self.cycles-pp.recvmsg
      0.15 ±  3%      -0.1        0.06 ± 82%  perf-profile.self.cycles-pp.__genradix_ptr
      0.16 ± 10%      -0.1        0.07 ± 62%  perf-profile.self.cycles-pp.skb_release_data
      0.16 ± 10%      -0.1        0.07 ± 52%  perf-profile.self.cycles-pp.__cond_resched
      0.11 ± 14%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.do_syscall_64
      0.16 ± 10%      -0.1        0.08 ± 51%  perf-profile.self.cycles-pp.skb_put
      0.12 ± 26%      -0.1        0.04 ± 72%  perf-profile.self.cycles-pp.__check_heap_object
      0.13 ± 22%      -0.1        0.05 ± 56%  perf-profile.self.cycles-pp.__entry_text_start
      0.14 ±  8%      -0.1        0.07 ± 31%  perf-profile.self.cycles-pp.kfree
      0.12 ± 20%      -0.1        0.05 ± 78%  perf-profile.self.cycles-pp.sctp_chunk_hold
      0.13 ±  6%      -0.1        0.06 ± 51%  perf-profile.self.cycles-pp.__import_iovec
      0.10 ±  9%      -0.1        0.03 ±111%  perf-profile.self.cycles-pp.__put_user_nocheck_8
      0.13 ±  7%      -0.1        0.06 ± 53%  perf-profile.self.cycles-pp.__ksize
      0.14 ±  9%      -0.1        0.07 ± 52%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.09 ± 26%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__sctp_packet_append_chunk
      0.14 ±  6%      -0.1        0.08 ± 39%  perf-profile.self.cycles-pp.aa_sk_perm
      0.13 ± 18%      -0.1        0.07 ± 35%  perf-profile.self.cycles-pp.sctp_association_hold
      0.16 ± 13%      -0.1        0.10 ± 34%  perf-profile.self.cycles-pp.sctp_association_put
      0.10 ±  9%      -0.1        0.04 ± 75%  perf-profile.self.cycles-pp.sctp_outq_tail
      0.09 ± 14%      -0.1        0.03 ±105%  perf-profile.self.cycles-pp._sctp_make_chunk
      0.10 ± 21%      -0.1        0.04 ± 76%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.09 ± 30%      -0.1        0.04 ± 75%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.12 ±  6%      -0.1        0.06 ± 24%  perf-profile.self.cycles-pp.sctp_check_transmitted
      0.10 ± 12%      -0.0        0.06 ± 49%  perf-profile.self.cycles-pp.____sys_sendmsg
      0.12 ± 13%      +0.0        0.15 ± 11%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.09 ± 24%      +0.0        0.12 ± 22%  perf-profile.self.cycles-pp.tick_sched_timer
      0.07 ± 23%      +0.0        0.11 ± 17%  perf-profile.self.cycles-pp.trigger_load_balance
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.hrtimer_forward
      0.22 ±  5%      +0.0        0.27 ±  9%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.15 ± 17%      +0.1        0.22 ± 19%  perf-profile.self.cycles-pp.update_rq_clock
      0.02 ±141%      +0.1        0.09 ± 14%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.02 ±141%      +0.1        0.10 ± 35%  perf-profile.self.cycles-pp.update_rt_rq_load_avg
      0.14 ± 11%      +0.1        0.22 ± 23%  perf-profile.self.cycles-pp.ct_kernel_enter
      0.07 ± 11%      +0.1        0.15 ± 42%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.09 ± 14%      +0.1        0.17 ± 10%  perf-profile.self.cycles-pp.note_gp_changes
      0.08 ± 14%      +0.1        0.17 ± 43%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.1        0.09 ± 37%  perf-profile.self.cycles-pp.rcu_core
      0.11 ± 25%      +0.1        0.20 ± 22%  perf-profile.self.cycles-pp.rebalance_domains
      0.18 ±  9%      +0.1        0.28 ± 11%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.02 ±141%      +0.1        0.12 ± 28%  perf-profile.self.cycles-pp.cpuidle_enter
      0.24 ±  7%      +0.1        0.35 ±  4%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.09 ± 15%      +0.1        0.20 ± 14%  perf-profile.self.cycles-pp.load_balance
      0.08 ± 22%      +0.1        0.19 ± 31%  perf-profile.self.cycles-pp._find_next_and_bit
      0.17 ± 12%      +0.1        0.29 ± 20%  perf-profile.self.cycles-pp.irq_work_tick
      0.29 ±  4%      +0.1        0.42 ± 16%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.17 ±  7%      +0.1        0.30 ± 17%  perf-profile.self.cycles-pp.idle_cpu
      0.32 ± 19%      +0.1        0.45 ±  9%  perf-profile.self.cycles-pp.io_serial_in
      0.26            +0.2        0.41 ± 22%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.12 ± 17%      +0.2        0.27 ± 40%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.16 ±  5%      +0.2        0.32 ± 18%  perf-profile.self.cycles-pp.rcu_pending
      0.10 ± 12%      +0.2        0.28 ± 54%  perf-profile.self.cycles-pp.update_process_times
      1.01 ±  3%      +0.2        1.21 ±  6%  perf-profile.self.cycles-pp.native_sched_clock
      0.26 ± 12%      +0.2        0.46 ± 14%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.09 ± 18%      +0.3        0.34 ± 67%  perf-profile.self.cycles-pp.timerqueue_add
      0.08 ± 20%      +0.3        0.36 ± 65%  perf-profile.self.cycles-pp.check_cpu_stall
      0.58            +0.3        0.87 ±  7%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.35 ±  8%      +0.4        0.78 ± 16%  perf-profile.self.cycles-pp.update_sg_lb_stats
      4.33            +0.4        4.78 ±  9%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.43 ± 34%      +1.6        3.07 ± 33%  perf-profile.self.cycles-pp.menu_select
     31.31           +12.0       43.29 ±  5%  perf-profile.self.cycles-pp.mwait_idle_with_hints



***************************************************************************************************
lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/30%/debian-11.1-x86_64-20220510.cgz/300s/lkp-cfl-e1/context1/unixbench

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      3547           +36.2%       4834        unixbench.score
      1202 ±  4%  +3.2e+05%    3838646 ± 17%  unixbench.time.involuntary_context_switches
    146.17            +4.7%     153.00        unixbench.time.percent_of_cpu_this_job_got
    533.06            +1.7%     542.03        unixbench.time.system_time
     40.02           +40.3%      56.14        unixbench.time.user_time
 4.156e+08           +34.9%  5.604e+08        unixbench.time.voluntary_context_switches
 5.537e+08           +36.3%  7.548e+08        unixbench.workload
      1.53            +0.5        2.01        mpstat.cpu.all.usr%
 4.434e+09           +13.2%  5.022e+09        cpuidle..time
 8.352e+08           -98.7%   11131648 ± 12%  cpuidle..usage
      3.00           +33.3%       4.00        vmstat.procs.r
   4212747           -32.1%    2861762        vmstat.system.cs
     40960           -29.9%      28703        vmstat.system.in
    161422           -32.1%     109641        meminfo.Active
    161345           -32.1%     109569        meminfo.Active(anon)
    530075           -10.9%     472307        meminfo.Committed_AS
     80181           -27.2%      58384        meminfo.Mapped
    162280           -32.2%     110057        meminfo.Shmem
     40427           -31.8%      27576        proc-vmstat.nr_active_anon
    978441            -1.3%     965308        proc-vmstat.nr_file_pages
     20137           -26.7%      14769        proc-vmstat.nr_mapped
     40661           -31.9%      27701        proc-vmstat.nr_shmem
     13329            -0.9%      13206        proc-vmstat.nr_slab_unreclaimable
     40427           -31.8%      27576        proc-vmstat.nr_zone_active_anon
   1751058           -14.2%    1502883        proc-vmstat.numa_hit
   1751425           -14.1%    1504002        proc-vmstat.numa_local
     92726           -17.3%      76685        proc-vmstat.pgactivate
   1819543           -13.7%    1570329        proc-vmstat.pgalloc_normal
    684646            -2.4%     668006        proc-vmstat.pgfault
   1807696           -13.7%    1559418        proc-vmstat.pgfree
     40680            -1.8%      39968        proc-vmstat.pgreuse
      1740           -47.5%     912.83        turbostat.Avg_MHz
     38.54           -18.6       19.90        turbostat.Busy%
      4515            +1.6%       4586        turbostat.Bzy_MHz
   3885351 ±  9%     -82.8%     667034 ± 73%  turbostat.C1
     26.41 ±  6%     +50.7%      39.80 ± 19%  turbostat.CPU%c6
     74.05           -31.1%      50.99 ±  2%  turbostat.CorWatt
     60.17           -18.3%      49.17 ±  2%  turbostat.CoreTmp
      0.22 ±  2%     +83.6%       0.41        turbostat.IPC
  16161117           -30.0%   11319624        turbostat.IRQ
 8.241e+08          -100.0%       3881 ± 26%  turbostat.POLL
      9.50            -9.5        0.00        turbostat.POLL%
     58.67 ±  2%     -17.0%      48.67 ±  2%  turbostat.PkgTmp
     74.52           -30.9%      51.46 ±  2%  turbostat.PkgWatt
    489.94           -35.7%     315.24        turbostat.Totl%C0
      4851 ± 56%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.avg
     56902 ± 44%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.max
     15406 ± 47%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.44 ±  9%     -14.4%       0.38 ±  2%  sched_debug.cfs_rq:/.h_nr_running.avg
    127353           +10.5%     140708 ±  3%  sched_debug.cfs_rq:/.load.stddev
    120.76 ±  8%     +25.4%     151.41 ± 17%  sched_debug.cfs_rq:/.load_avg.avg
      4851 ± 56%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.avg
     56902 ± 44%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.max
     15406 ± 47%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.stddev
    197189            -9.1%     179171 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
    116754 ±  6%     +12.9%     131851 ±  3%  sched_debug.cfs_rq:/.min_vruntime.min
     21217 ± 10%     -39.9%      12759 ± 18%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.44 ±  7%     -18.5%       0.36 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
    366.23           +21.1%     443.60 ±  3%  sched_debug.cfs_rq:/.runnable_avg.avg
    714.74 ±  4%     +81.8%       1299 ±  3%  sched_debug.cfs_rq:/.runnable_avg.max
    246.75 ±  5%    +101.3%     496.81 ±  3%  sched_debug.cfs_rq:/.runnable_avg.stddev
     51496 ± 44%     -66.9%      17048 ± 49%  sched_debug.cfs_rq:/.spread0.max
     21217 ± 10%     -39.9%      12760 ± 18%  sched_debug.cfs_rq:/.spread0.stddev
    708.50 ±  4%     +49.8%       1061 ±  4%  sched_debug.cfs_rq:/.util_avg.max
    245.43 ±  5%     +63.7%     401.71 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    107.96 ± 11%     +42.0%     153.34 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    499.12 ±  2%     +65.3%     825.00 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.max
    181.05 ±  4%     +58.9%     287.67 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    407511 ±  4%    +110.5%     857930        sched_debug.cpu.avg_idle.avg
    104621          +462.1%     588066 ± 11%  sched_debug.cpu.avg_idle.min
    350206 ±  2%     -68.5%     110282 ± 21%  sched_debug.cpu.avg_idle.stddev
      0.35 ± 18%     +89.2%       0.67 ±  3%  sched_debug.cpu.clock.stddev
      0.00 ± 37%     +68.1%       0.00 ±  9%  sched_debug.cpu.next_balance.stddev
      1.07 ±  6%     +55.6%       1.67 ±  9%  sched_debug.cpu.nr_running.max
      0.46 ±  3%     +24.0%       0.57 ±  4%  sched_debug.cpu.nr_running.stddev
  47178388           -32.4%   31875847        sched_debug.cpu.nr_switches.avg
  56953632 ±  2%     -36.6%   36093819 ±  2%  sched_debug.cpu.nr_switches.max
  31935712 ±  9%     -13.3%   27680720 ±  3%  sched_debug.cpu.nr_switches.min
   6579393 ± 14%     -64.6%    2330670 ± 21%  sched_debug.cpu.nr_switches.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     65.64           -30.8%      45.42        perf-stat.i.MPKI
 4.603e+09            -7.6%  4.252e+09        perf-stat.i.branch-instructions
      2.81            +0.1        2.89        perf-stat.i.branch-miss-rate%
  49172917            -1.4%   48506007        perf-stat.i.branch-misses
      0.79            +2.6        3.39 ±  7%  perf-stat.i.cache-miss-rate%
   1887466            -4.2%    1808731 ±  2%  perf-stat.i.cache-misses
 5.404e+08           -94.1%   32037290 ±  3%  perf-stat.i.cache-references
   4247507           -32.1%    2884302        perf-stat.i.context-switches
      1.99           -24.4%       1.50        perf-stat.i.cpi
 2.784e+10           -47.7%  1.457e+10        perf-stat.i.cpu-cycles
     52.17 ±  2%    +634.1%     383.01 ±  4%  perf-stat.i.cpu-migrations
     56513 ±  2%     -59.7%      22750 ±  9%  perf-stat.i.cycles-between-cache-misses
    410439 ±  2%     +23.7%     507559 ±  7%  perf-stat.i.dTLB-load-misses
 6.418e+09            -3.1%   6.22e+09        perf-stat.i.dTLB-loads
 3.826e+09            -4.0%  3.674e+09        perf-stat.i.dTLB-stores
     95.43           -38.9       56.58        perf-stat.i.iTLB-load-miss-rate%
  16498563            -5.9%   15523524        perf-stat.i.iTLB-load-misses
    239550 ±  3%   +5782.0%   14090457        perf-stat.i.iTLB-loads
 2.197e+10            -4.8%  2.092e+10        perf-stat.i.instructions
      0.70           +78.1%       1.25        perf-stat.i.ipc
      1.74           -47.7%       0.91        perf-stat.i.metric.GHz
    404.88 ±  2%      -4.8%     385.49 ±  2%  perf-stat.i.metric.K/sec
    961.54            -7.8%     886.77        perf-stat.i.metric.M/sec
      1575            -2.7%       1533        perf-stat.i.minor-faults
      0.40 ±  4%      +0.1        0.47 ±  7%  perf-stat.i.node-load-miss-rate%
    261859           -14.5%     223787        perf-stat.i.node-stores
      1575            -2.7%       1533        perf-stat.i.page-faults
     24.60           -93.8%       1.53 ±  3%  perf-stat.overall.MPKI
      1.07            +0.1        1.14        perf-stat.overall.branch-miss-rate%
      0.35            +5.3        5.67 ±  5%  perf-stat.overall.cache-miss-rate%
      1.27           -45.0%       0.70        perf-stat.overall.cpi
     14726           -45.3%       8048 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.01 ±  2%      +0.0        0.01 ±  7%  perf-stat.overall.dTLB-load-miss-rate%
     98.57           -46.1       52.42        perf-stat.overall.iTLB-load-miss-rate%
      1331            +1.2%       1347        perf-stat.overall.instructions-per-iTLB-miss
      0.79           +81.9%       1.44        perf-stat.overall.ipc
      0.00 ± 11%      +0.0        0.01 ± 10%  perf-stat.overall.node-store-miss-rate%
     15528           -30.1%      10847        perf-stat.overall.path-length
 4.591e+09            -7.6%   4.24e+09        perf-stat.ps.branch-instructions
  49052787            -1.4%   48388408        perf-stat.ps.branch-misses
   1885579            -4.2%    1807271 ±  2%  perf-stat.ps.cache-misses
 5.389e+08           -94.1%   31961586 ±  3%  perf-stat.ps.cache-references
   4235546           -32.1%    2876157        perf-stat.ps.context-switches
 2.777e+10           -47.6%  1.454e+10        perf-stat.ps.cpu-cycles
     52.05 ±  2%    +633.9%     381.95 ±  4%  perf-stat.ps.cpu-migrations
    409410 ±  2%     +23.7%     506263 ±  7%  perf-stat.ps.dTLB-load-misses
   6.4e+09            -3.1%  6.203e+09        perf-stat.ps.dTLB-loads
 3.815e+09            -4.0%  3.664e+09        perf-stat.ps.dTLB-stores
  16452159            -5.9%   15479733        perf-stat.ps.iTLB-load-misses
    238902 ±  3%   +5781.4%   14050699        perf-stat.ps.iTLB-loads
  2.19e+10            -4.8%  2.086e+10        perf-stat.ps.instructions
      1571            -2.7%       1529        perf-stat.ps.minor-faults
    261165           -14.5%     223199        perf-stat.ps.node-stores
      1571            -2.7%       1529        perf-stat.ps.page-faults
 8.598e+12            -4.8%  8.188e+12        perf-stat.total.instructions
     57.34           -35.5       21.87        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     53.70 ±  2%     -32.8       20.91        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     53.66 ±  2%     -32.8       20.91        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     53.51 ±  2%     -32.6       20.91        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     21.60           -21.6        0.00        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     38.36 ±  2%     -17.5       20.86        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     35.56 ±  2%     -14.1       21.44        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     33.37 ±  2%     -12.8       20.57        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      7.88 ±  2%      -7.9        0.00        perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      5.64            -5.6        0.00        perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.62 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
      2.64 ± 18%      -1.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      2.31 ± 15%      -1.4        0.94 ± 31%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      4.08 ±  2%      -1.4        2.73        perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      1.90 ±  3%      -0.8        1.11 ±  5%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common
      2.12 ±  2%      -0.8        1.32 ±  3%  perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.86 ±  4%      -0.6        0.26 ±100%  perf-profile.calltrace.cycles-pp.copy_user_short_string.copyout._copy_to_iter.copy_page_to_iter.pipe_read
      1.60 ±  2%      -0.4        1.16 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.12 ±  2%      -0.4        0.70 ±  3%  perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.97 ±  4%      -0.2        0.75 ±  5%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.20 ±  4%      -0.2        1.00 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.pipe_read.vfs_read.ksys_read
      1.04 ±  4%      -0.1        0.90 ±  5%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.copy_page_to_iter.pipe_read.vfs_read
      1.31 ±  3%      +0.1        1.40 ±  2%  perf-profile.calltrace.cycles-pp._copy_to_iter.copy_page_to_iter.pipe_read.vfs_read.ksys_read
      1.41            +0.1        1.53        perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.vfs_read.ksys_read.do_syscall_64
      1.04            +0.2        1.21 ±  4%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      2.68 ±  2%      +0.2        2.85        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.56 ±  4%      +0.2        0.80 ±  7%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.42 ±  2%      +0.3        1.70        perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.26 ±100%      +0.4        0.69 ±  6%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
      0.56 ±  3%      +0.5        1.07 ±  8%  perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.pipe_read.vfs_read
      0.00            +0.6        0.56 ± 11%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +0.6        0.56 ±  7%  perf-profile.calltrace.cycles-pp.__might_resched.__might_fault._copy_from_iter.copy_page_from_iter.pipe_write
      0.00            +0.6        0.58 ±  3%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.00            +0.6        0.58 ±  5%  perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock_cpu.update_rq_clock.__schedule.schedule
      0.00            +0.6        0.59 ±  3%  perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock_cpu.update_rq_clock.try_to_wake_up.autoremove_wake_function
      0.00            +0.6        0.62 ±  6%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.update_rq_clock.__schedule.schedule.pipe_read
      0.00            +0.6        0.63 ±  2%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.63 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.pipe_read.vfs_read.ksys_read
      0.00            +0.7        0.66 ± 10%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.00            +0.7        0.66 ±  4%  perf-profile.calltrace.cycles-pp.__wrgsbase_inactive.read
      0.00            +0.7        0.68 ±  8%  perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.pipe_read
      0.00            +0.7        0.69 ±  4%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.70 ±  6%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +0.7        0.70 ±  8%  perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.65 ±  5%      +0.7        1.37 ±  3%  perf-profile.calltrace.cycles-pp.os_xsave.read
      1.26 ±  4%      +0.7        1.98 ±  2%  perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write.ksys_write
      0.00            +0.7        0.72 ± 10%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +0.7        0.74 ±  5%  perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write
      0.00            +0.8        0.76 ±  3%  perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule
      0.00            +0.8        0.76 ±  6%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.reweight_entity.dequeue_task_fair.__schedule
      0.00            +0.8        0.80 ±  4%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.reweight_entity.enqueue_task_fair.ttwu_do_activate
      0.00            +0.8        0.81 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +0.8        0.82 ±  9%  perf-profile.calltrace.cycles-pp.__switch_to.read
      0.00            +0.8        0.82 ±  4%  perf-profile.calltrace.cycles-pp.finish_wait.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.00            +0.9        0.87 ±  3%  perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_read.vfs_read
      0.00            +0.9        0.88 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.9        0.89 ±  3%  perf-profile.calltrace.cycles-pp.main
      0.00            +0.9        0.91 ±  5%  perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.9        0.91 ±  5%  perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +0.9        0.94 ±  2%  perf-profile.calltrace.cycles-pp.__switch_to.__schedule.schedule.pipe_read.vfs_read
      1.43 ±  3%      +1.0        2.38 ±  2%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.00            +1.0        0.95 ±  2%  perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +1.0        1.01 ±  5%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
      0.00            +1.0        1.01 ±  4%  perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      0.91 ±  2%      +1.0        1.93 ±  3%  perf-profile.calltrace.cycles-pp.__switch_to_asm.read
      0.00            +1.0        1.04 ±  4%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +1.2        1.18 ±  3%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_read
      0.00            +1.3        1.27 ±  4%  perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +1.3        1.30 ±  4%  perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +1.4        1.38 ±  3%  perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.54 ±  3%      +1.4        1.94 ±  3%  perf-profile.calltrace.cycles-pp.__entry_text_start.read
      0.00            +1.6        1.58 ±  3%  perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.17 ±141%      +1.7        1.84 ±  3%  perf-profile.calltrace.cycles-pp.__entry_text_start.write
      0.00            +1.7        1.67 ±  3%  perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +1.7        1.68 ±  2%  perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.60 ±  5%      +2.1        2.66 ±  2%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +2.2        2.21 ±  2%  perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.58 ±  6%      +2.2        2.82        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.vfs_read
      0.00            +2.3        2.30 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      4.70            +2.4        7.11        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.vfs_read
      0.00            +3.1        3.15 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +5.1        5.05 ±  2%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.pipe_read.vfs_read
      0.00            +7.3        7.32        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      7.83            +7.3       15.18        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write.ksys_write
      0.00            +7.5        7.52        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      6.88 ±  2%      +7.6       14.49        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      8.24            +7.8       16.01        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
      7.11            +7.8       14.94        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write
     11.86 ±  6%      +8.6       20.50 ±  2%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     11.86 ±  6%      +8.6       20.51 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     12.29            +9.5       21.75        perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.40           +10.0       23.36        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     13.83           +10.4       24.21        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     14.17           +10.9       25.10        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     14.33           +11.1       25.46        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      9.16           +11.3       20.49        perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
      9.44           +11.9       21.39        perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
     15.53           +13.3       28.86        perf-profile.calltrace.cycles-pp.write
     18.03           +14.7       32.74        perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.37           +15.7       35.10        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     19.90           +16.3       36.21        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     20.88           +18.6       39.51        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     21.30           +19.5       40.84        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     24.96           +24.5       49.46        perf-profile.calltrace.cycles-pp.read
     57.34           -35.5       21.87        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     57.34           -35.5       21.87        perf-profile.children.cycles-pp.cpu_startup_entry
     57.17           -35.3       21.87        perf-profile.children.cycles-pp.do_idle
     53.70 ±  2%     -32.8       20.91        perf-profile.children.cycles-pp.start_secondary
     21.62           -21.6        0.00        perf-profile.children.cycles-pp.poll_idle
     41.05           -19.2       21.83        perf-profile.children.cycles-pp.cpuidle_idle_call
     35.70 ±  2%     -14.2       21.51        perf-profile.children.cycles-pp.cpuidle_enter
     35.64 ±  2%     -14.1       21.51        perf-profile.children.cycles-pp.cpuidle_enter_state
      8.40            -8.4        0.00        perf-profile.children.cycles-pp.flush_smp_call_function_queue
      6.02            -6.0        0.00        perf-profile.children.cycles-pp.sched_ttwu_pending
      5.20 ±  2%      -5.2        0.00        perf-profile.children.cycles-pp.schedule_idle
      4.01            -3.8        0.25 ± 11%  perf-profile.children.cycles-pp.menu_select
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.start_kernel
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.arch_call_rest_init
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.rest_init
      2.59 ±  3%      -2.3        0.32 ± 10%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      1.83 ±  2%      -1.7        0.09 ± 20%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      4.18 ±  2%      -1.4        2.81        perf-profile.children.cycles-pp.dequeue_entity
      1.13 ±  2%      -1.1        0.04 ± 72%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.92 ±  3%      -0.8        1.11 ±  5%  perf-profile.children.cycles-pp.select_task_rq_fair
      2.12 ±  2%      -0.8        1.34 ±  4%  perf-profile.children.cycles-pp.select_task_rq
      0.72 ±  3%      -0.7        0.06 ± 47%  perf-profile.children.cycles-pp.ktime_get
      1.53 ±  2%      -0.6        0.90 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      1.86            -0.5        1.33        perf-profile.children.cycles-pp.sched_clock_cpu
      1.63 ±  2%      -0.4        1.20        perf-profile.children.cycles-pp.native_sched_clock
      3.03 ±  3%      -0.4        2.65 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.45 ±  3%      -0.4        1.09 ±  8%  perf-profile.children.cycles-pp.prepare_task_switch
      2.58            -0.3        2.30        perf-profile.children.cycles-pp.mutex_lock
      0.53 ±  4%      -0.2        0.34 ±  6%  perf-profile.children.cycles-pp.finish_task_switch
      1.05 ±  4%      -0.1        0.90 ±  5%  perf-profile.children.cycles-pp.copyout
      1.38 ±  5%      -0.1        1.26 ±  3%  perf-profile.children.cycles-pp.set_next_entity
      0.17 ± 10%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.18 ±  9%      -0.0        0.13 ± 16%  perf-profile.children.cycles-pp.place_entity
      0.10 ±  6%      -0.0        0.05 ± 48%  perf-profile.children.cycles-pp.rb_next
      0.08 ± 11%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.sched_clock
      0.06 ± 11%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.rw_verify_area
      0.08 ± 11%      +0.0        0.12 ± 11%  perf-profile.children.cycles-pp.__x64_sys_read
      0.30 ±  8%      +0.0        0.34 ±  8%  perf-profile.children.cycles-pp.perf_trace_buf_update
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      0.06 ± 45%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.04 ± 71%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
      0.04 ± 71%      +0.1        0.09 ± 14%  perf-profile.children.cycles-pp.inode_needs_update_time
      0.03 ±102%      +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.save_fpregs_to_fpstate
      0.22 ±  6%      +0.1        0.28 ± 11%  perf-profile.children.cycles-pp.put_prev_entity
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.08 ± 14%      +0.1        0.13 ± 16%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.12 ±  6%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.perf_trace_buf_alloc
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.default_wake_function
      0.31 ±  5%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.09 ±  9%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.__x64_sys_write
      0.09 ± 10%      +0.1        0.16 ±  8%  perf-profile.children.cycles-pp.rcu_all_qs
      0.04 ± 44%      +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.page_copy_sane
      0.04 ± 71%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.scheduler_tick
      0.30 ±  8%      +0.1        0.38 ±  4%  perf-profile.children.cycles-pp.pick_next_entity
      0.11 ± 15%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.09 ± 15%      +0.1        0.18 ± 17%  perf-profile.children.cycles-pp.update_process_times
      0.05 ± 46%      +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.07 ± 11%      +0.1        0.15 ± 15%  perf-profile.children.cycles-pp.perf_trace_run_bpf_submit
      0.06 ± 11%      +0.1        0.15 ± 12%  perf-profile.children.cycles-pp.kill_fasync
      0.10 ± 16%      +0.1        0.20 ± 17%  perf-profile.children.cycles-pp.tick_sched_handle
      0.04 ± 72%      +0.1        0.14 ± 34%  perf-profile.children.cycles-pp.write@plt
      0.12 ± 19%      +0.1        0.23 ± 18%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.1        0.12 ± 36%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.19 ±  9%      +0.1        0.31 ±  9%  perf-profile.children.cycles-pp.__get_task_ioprio
      0.57 ±  6%      +0.1        0.68 ±  4%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.13 ± 15%      +0.1        0.25 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      1.32 ±  3%      +0.1        1.44 ±  2%  perf-profile.children.cycles-pp._copy_to_iter
      1.54 ±  2%      +0.1        1.67        perf-profile.children.cycles-pp.__might_resched
      0.00            +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.14 ±  9%      +0.2        0.30 ±  6%  perf-profile.children.cycles-pp.aa_file_perm
      0.09 ± 20%      +0.2        0.26 ±  5%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.14 ± 13%      +0.2        0.31 ± 12%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.14 ± 11%      +0.2        0.31 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.20 ± 14%      +0.2        0.37 ± 12%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.21 ±  9%      +0.2        0.38 ±  5%  perf-profile.children.cycles-pp.__cond_resched
      1.06            +0.2        1.23 ±  4%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.06 ± 19%      +0.2        0.24 ±  8%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.22 ±  7%      +0.2        0.41 ±  6%  perf-profile.children.cycles-pp.atime_needs_update
      0.00            +0.2        0.22 ±  6%  perf-profile.children.cycles-pp.set_next_buddy
      2.70 ±  2%      +0.2        2.92        perf-profile.children.cycles-pp.prepare_to_wait_event
      0.28 ± 11%      +0.2        0.51 ±  9%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.88 ±  5%      +0.2        1.12 ±  4%  perf-profile.children.cycles-pp.__might_fault
      0.22 ±  8%      +0.2        0.46 ±  6%  perf-profile.children.cycles-pp.current_time
      0.28 ±  7%      +0.2        0.52 ±  5%  perf-profile.children.cycles-pp.touch_atime
      0.30 ± 11%      +0.2        0.55 ±  8%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.21 ±  9%      +0.2        0.46 ±  8%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.22 ±  8%      +0.2        0.47 ±  6%  perf-profile.children.cycles-pp.file_update_time
      0.62 ±  2%      +0.3        0.88 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      2.96 ±  2%      +0.3        3.23 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      1.44 ±  2%      +0.3        1.72        perf-profile.children.cycles-pp.copy_page_to_iter
      1.53 ±  3%      +0.3        1.81 ±  3%  perf-profile.children.cycles-pp.__switch_to
      0.26 ±  6%      +0.3        0.57 ±  5%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.21 ±  7%      +0.3        0.53 ±  7%  perf-profile.children.cycles-pp.update_min_vruntime
      2.68 ±  5%      +0.3        3.02        perf-profile.children.cycles-pp.pick_next_task_fair
      0.12 ±  8%      +0.3        0.46 ±  4%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.42 ±  8%      +0.4        0.78 ±  9%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.73 ±  6%      +0.4        1.11 ±  5%  perf-profile.children.cycles-pp.mutex_unlock
      0.48 ±  7%      +0.4        0.86 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.87 ±  2%      +0.4        1.27 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.90 ±  3%      +0.4        1.30 ±  4%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.14 ± 15%      +0.4        0.54 ±  2%  perf-profile.children.cycles-pp.__list_add_valid
      0.89 ±  4%      +0.4        1.30 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.37 ±  7%      +0.4        0.78 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.33 ±  4%      +0.4        0.75 ±  5%  perf-profile.children.cycles-pp.copyin
      0.36 ±  4%      +0.5        0.82 ±  4%  perf-profile.children.cycles-pp.finish_wait
      1.03 ±  3%      +0.5        1.50 ±  4%  perf-profile.children.cycles-pp.security_file_permission
      0.18 ± 16%      +0.5        0.68 ±  7%  perf-profile.children.cycles-pp.cpuacct_charge
      0.40 ±  6%      +0.5        0.93 ±  3%  perf-profile.children.cycles-pp.main
      0.53 ±  2%      +0.6        1.12 ±  7%  perf-profile.children.cycles-pp.__fdget_pos
      0.65 ±  5%      +0.7        1.38 ±  3%  perf-profile.children.cycles-pp.os_xsave
      1.30 ±  4%      +0.7        2.04 ±  2%  perf-profile.children.cycles-pp._copy_from_iter
      0.65 ±  3%      +0.8        1.41 ±  2%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      1.02 ±  2%      +0.8        1.82 ±  2%  perf-profile.children.cycles-pp.update_rq_clock
      1.44 ±  3%      +1.0        2.42 ±  2%  perf-profile.children.cycles-pp.copy_page_from_iter
      0.35 ±  5%      +1.0        1.38 ±  3%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.00            +1.0        1.04 ±  5%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.21 ±  7%      +1.1        1.28 ±  4%  perf-profile.children.cycles-pp.check_preempt_curr
      0.46 ±  5%      +1.1        1.56 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      0.83 ±  5%      +1.1        1.96 ±  2%  perf-profile.children.cycles-pp.update_cfs_group
      2.68 ±  4%      +1.3        3.98 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      1.02            +1.4        2.47 ±  2%  perf-profile.children.cycles-pp.__entry_text_start
      0.00            +1.6        1.59 ±  3%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.36 ±  6%      +1.9        2.23 ±  3%  perf-profile.children.cycles-pp.switch_fpu_return
      0.49 ±  4%      +2.1        2.61 ±  4%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.72            +2.4        7.12        perf-profile.children.cycles-pp.dequeue_task_fair
      0.74 ±  6%      +2.4        3.15 ±  3%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.63 ±  4%      +2.7        3.35        perf-profile.children.cycles-pp.reweight_entity
      1.74 ±  3%      +3.2        4.91 ±  2%  perf-profile.children.cycles-pp.update_curr
      3.34 ±  2%      +4.0        7.34        perf-profile.children.cycles-pp.enqueue_task_fair
      3.44 ±  2%      +4.1        7.54        perf-profile.children.cycles-pp.ttwu_do_activate
      0.26 ±  6%      +4.9        5.12 ±  3%  perf-profile.children.cycles-pp.switch_mm_irqs_off
     14.17 ±  2%      +6.6       20.77        perf-profile.children.cycles-pp.__schedule
      7.84            +7.4       15.19        perf-profile.children.cycles-pp.__wake_up_common
      6.90            +7.7       14.56        perf-profile.children.cycles-pp.try_to_wake_up
      8.26            +7.8       16.03        perf-profile.children.cycles-pp.__wake_up_common_lock
      7.12            +7.9       14.98        perf-profile.children.cycles-pp.autoremove_wake_function
     11.94 ±  6%      +8.6       20.59 ±  2%  perf-profile.children.cycles-pp.mwait_idle_with_hints
     11.95 ±  6%      +8.6       20.59 ±  2%  perf-profile.children.cycles-pp.intel_idle
     12.32            +9.5       21.81        perf-profile.children.cycles-pp.pipe_write
     13.43           +10.0       23.43        perf-profile.children.cycles-pp.vfs_write
     13.85           +10.4       24.25        perf-profile.children.cycles-pp.ksys_write
      9.45           +12.1       21.52        perf-profile.children.cycles-pp.schedule
     15.82           +13.1       28.92        perf-profile.children.cycles-pp.write
     18.16           +14.9       33.02        perf-profile.children.cycles-pp.pipe_read
     19.41           +15.7       35.11        perf-profile.children.cycles-pp.vfs_read
     19.92           +16.3       36.24        perf-profile.children.cycles-pp.ksys_read
     25.25           +24.2       49.48        perf-profile.children.cycles-pp.read
     35.17           +29.7       64.91        perf-profile.children.cycles-pp.do_syscall_64
     35.72           +30.8       66.50        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     21.21           -21.2        0.00        perf-profile.self.cycles-pp.poll_idle
      1.76 ±  2%      -1.6        0.14 ± 11%  perf-profile.self.cycles-pp.menu_select
      1.52 ±  2%      -0.6        0.90 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.72 ±  5%      -0.5        0.23 ± 11%  perf-profile.self.cycles-pp.__wake_up_common
      2.98 ±  3%      -0.4        2.55 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.58 ±  2%      -0.4        1.16        perf-profile.self.cycles-pp.native_sched_clock
      0.78 ±  2%      -0.4        0.40 ± 14%  perf-profile.self.cycles-pp.prepare_task_switch
      0.32 ±  8%      -0.2        0.11 ± 13%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.47 ±  5%      -0.2        0.28 ±  8%  perf-profile.self.cycles-pp.finish_task_switch
      1.24            -0.2        1.06 ±  3%  perf-profile.self.cycles-pp.mutex_lock
      1.03 ±  3%      -0.2        0.86 ±  2%  perf-profile.self.cycles-pp.enqueue_entity
      0.46 ± 13%      -0.2        0.31 ± 12%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.20 ±  5%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.16 ±  8%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.rcu_note_context_switch
      0.38 ±  9%      -0.1        0.31 ± 10%  perf-profile.self.cycles-pp.set_next_entity
      0.35 ±  6%      -0.1        0.29 ±  7%  perf-profile.self.cycles-pp.perf_tp_event
      0.16 ±  6%      -0.0        0.11 ± 13%  perf-profile.self.cycles-pp.put_prev_entity
      0.16 ±  7%      -0.0        0.12 ± 14%  perf-profile.self.cycles-pp.place_entity
      0.09 ±  7%      -0.0        0.05 ± 47%  perf-profile.self.cycles-pp.rb_next
      0.07 ± 13%      +0.0        0.09 ± 11%  perf-profile.self.cycles-pp.perf_trace_buf_alloc
      0.08 ± 12%      +0.0        0.11 ± 13%  perf-profile.self.cycles-pp.__x64_sys_read
      0.14 ±  3%      +0.0        0.17 ± 10%  perf-profile.self.cycles-pp.__might_fault
      0.20 ±  9%      +0.0        0.24 ±  8%  perf-profile.self.cycles-pp.select_task_rq
      0.02 ±141%      +0.0        0.06 ±  8%  perf-profile.self.cycles-pp.perf_swevent_event
      0.03 ± 70%      +0.0        0.08 ± 14%  perf-profile.self.cycles-pp.inode_needs_update_time
      0.07 ± 11%      +0.0        0.12 ± 14%  perf-profile.self.cycles-pp.perf_trace_buf_update
      0.06 ± 17%      +0.1        0.12 ±  9%  perf-profile.self.cycles-pp.rcu_all_qs
      0.23 ±  7%      +0.1        0.28 ±  3%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.06 ±  7%      +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.finish_wait
      0.08 ± 11%      +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.__x64_sys_write
      0.11 ± 11%      +0.1        0.18 ±  5%  perf-profile.self.cycles-pp.atime_needs_update
      0.31 ±  4%      +0.1        0.38 ±  6%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.05 ± 51%      +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.__cgroup_account_cputime
      0.01 ±223%      +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
      0.05 ± 46%      +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.touch_atime
      0.01 ±223%      +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.rw_verify_area
      0.15 ±  8%      +0.1        0.23 ±  6%  perf-profile.self.cycles-pp.check_preempt_curr
      0.13 ± 11%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.security_file_permission
      0.06 ± 11%      +0.1        0.14 ± 13%  perf-profile.self.cycles-pp.perf_trace_run_bpf_submit
      0.04 ± 71%      +0.1        0.12 ± 34%  perf-profile.self.cycles-pp.write@plt
      0.10 ±  6%      +0.1        0.19 ± 13%  perf-profile.self.cycles-pp.ttwu_do_activate
      0.05 ± 46%      +0.1        0.14 ± 18%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.42 ±  7%      +0.1        0.50 ±  7%  perf-profile.self.cycles-pp.update_rq_clock
      0.10 ± 11%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.02 ± 99%      +0.1        0.12 ± 15%  perf-profile.self.cycles-pp.kill_fasync
      0.11 ± 12%      +0.1        0.20 ±  9%  perf-profile.self.cycles-pp.__cond_resched
      0.10 ±  3%      +0.1        0.20 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.06 ± 11%      +0.1        0.16 ± 14%  perf-profile.self.cycles-pp.file_update_time
      0.00            +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.page_copy_sane
      0.18 ±  7%      +0.1        0.28 ±  8%  perf-profile.self.cycles-pp.__get_task_ioprio
      0.11 ±  8%      +0.1        0.22 ± 20%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.12 ± 17%      +0.1        0.23 ±  5%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.57 ±  6%      +0.1        0.68 ±  4%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.12 ±  6%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.10 ±  8%      +0.1        0.22 ±  4%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.00            +0.1        0.13 ± 10%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.12 ± 10%      +0.1        0.25 ±  8%  perf-profile.self.cycles-pp.ksys_write
      0.17 ±  9%      +0.1        0.30 ±  7%  perf-profile.self.cycles-pp.autoremove_wake_function
      1.54 ±  2%      +0.1        1.67        perf-profile.self.cycles-pp.__might_resched
      0.14 ± 14%      +0.1        0.28 ± 13%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.13 ± 12%      +0.1        0.27 ±  6%  perf-profile.self.cycles-pp.aa_file_perm
      0.16 ± 13%      +0.1        0.30 ±  4%  perf-profile.self.cycles-pp.current_time
      0.14 ±  3%      +0.2        0.30 ±  8%  perf-profile.self.cycles-pp._copy_to_iter
      0.14 ± 11%      +0.2        0.31 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.03 ±101%      +0.2        0.21 ± 10%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.12 ± 10%      +0.2        0.30 ±  7%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.41 ±  4%      +0.2        0.60 ±  5%  perf-profile.self.cycles-pp.vfs_write
      0.54 ±  4%      +0.2        0.76        perf-profile.self.cycles-pp.dequeue_task_fair
      0.00            +0.2        0.22 ±  6%  perf-profile.self.cycles-pp.set_next_buddy
      0.38 ±  9%      +0.2        0.62 ±  5%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.25 ±  5%      +0.2        0.50        perf-profile.self.cycles-pp._copy_from_iter
      0.21 ±  7%      +0.3        0.46 ±  8%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.61            +0.3        0.87 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.76 ±  5%      +0.3        1.02 ±  4%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.35 ±  6%      +0.3        0.62 ±  4%  perf-profile.self.cycles-pp.switch_fpu_return
      1.48 ±  3%      +0.3        1.76 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.18 ±  7%      +0.3        0.46 ±  6%  perf-profile.self.cycles-pp.update_min_vruntime
      0.44 ±  7%      +0.3        0.73 ±  6%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.22 ±  7%      +0.3        0.54 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.28 ±  7%      +0.3        0.61 ±  7%  perf-profile.self.cycles-pp.ksys_read
      0.12 ± 11%      +0.3        0.46 ±  4%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.73 ±  6%      +0.4        1.09 ±  6%  perf-profile.self.cycles-pp.mutex_unlock
      0.36 ±  8%      +0.4        0.74 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.85            +0.4        1.24 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.35 ±  6%      +0.4        0.74 ±  4%  perf-profile.self.cycles-pp.try_to_wake_up
      0.14 ± 16%      +0.4        0.54 ±  2%  perf-profile.self.cycles-pp.__list_add_valid
      0.84 ±  4%      +0.4        1.27 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.38 ±  3%      +0.5        0.84 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
      0.39 ±  8%      +0.5        0.86 ±  5%  perf-profile.self.cycles-pp.read
      0.40 ±  6%      +0.5        0.88 ±  4%  perf-profile.self.cycles-pp.write
      0.94 ±  5%      +0.5        1.42 ±  5%  perf-profile.self.cycles-pp.update_load_avg
      0.36 ±  6%      +0.5        0.86 ±  4%  perf-profile.self.cycles-pp.main
      0.17 ± 16%      +0.5        0.67 ±  7%  perf-profile.self.cycles-pp.cpuacct_charge
      0.22 ±  4%      +0.5        0.76 ±  3%  perf-profile.self.cycles-pp.schedule
      0.52 ±  3%      +0.6        1.08 ±  7%  perf-profile.self.cycles-pp.__fdget_pos
      0.58 ±  5%      +0.6        1.16 ±  6%  perf-profile.self.cycles-pp.vfs_read
      0.63 ±  4%      +0.6        1.26 ±  3%  perf-profile.self.cycles-pp.reweight_entity
      0.64 ±  6%      +0.7        1.31        perf-profile.self.cycles-pp.pipe_write
      0.46 ±  4%      +0.7        1.15 ±  4%  perf-profile.self.cycles-pp.__entry_text_start
      0.65 ±  5%      +0.7        1.37 ±  2%  perf-profile.self.cycles-pp.os_xsave
      0.62 ±  3%      +0.7        1.35 ±  3%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.35 ±  7%      +0.7        1.09 ±  4%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.77 ±  6%      +0.8        1.52 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.9        0.90 ±  6%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.94 ±  5%      +1.0        1.94        perf-profile.self.cycles-pp.pipe_read
      0.55 ±  5%      +1.1        1.60 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.45 ±  5%      +1.1        1.56 ±  2%  perf-profile.self.cycles-pp.__calc_delta
      0.81 ±  5%      +1.1        1.93 ±  2%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +1.6        1.59 ±  3%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.25 ±  7%      +4.8        5.07 ±  2%  perf-profile.self.cycles-pp.switch_mm_irqs_off
     11.94 ±  6%      +8.6       20.59 ±  2%  perf-profile.self.cycles-pp.mwait_idle_with_hints



***************************************************************************************************
lkp-cpl-4sp1: 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz (Cooper Lake) with 128G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/25%/debian-11.1-x86_64-20220510.cgz/300s/lkp-cpl-4sp1/TCP_RR/netperf

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   3611595           +53.3%    5537882        netperf.Throughput_total_tps
    100322           +53.3%     153830        netperf.Throughput_tps
    130329        +8.7e+05%  1.128e+09 ± 19%  netperf.time.involuntary_context_switches
     16743 ±  2%     -22.6%      12962        netperf.time.minor_page_faults
      1202           +16.4%       1399 ±  2%  netperf.time.percent_of_cpu_this_job_got
      3432           +12.6%       3865 ±  2%  netperf.time.system_time
    188.48           +85.8%     350.16 ±  4%  netperf.time.user_time
 1.083e+09           -50.8%  5.334e+08 ± 39%  netperf.time.voluntary_context_switches
 1.083e+09           +53.3%  1.661e+09        netperf.workload
 2.212e+09           -97.0%   65981036        cpuidle..usage
     84551 ±  5%     -18.5%      68943 ±  3%  meminfo.Active
     84433 ±  5%     -18.5%      68833 ±  3%  meminfo.Active(anon)
     79312 ±  6%     -21.0%      62677 ±  5%  numa-meminfo.node3.Active
     79280 ±  6%     -21.0%      62645 ±  5%  numa-meminfo.node3.Active(anon)
     19806 ±  6%     -20.8%      15687 ±  5%  numa-vmstat.node3.nr_active_anon
     19806 ±  6%     -20.8%      15687 ±  5%  numa-vmstat.node3.nr_zone_active_anon
      8.60            -2.4        6.16        mpstat.cpu.all.soft%
     18.97            -1.9       17.04        mpstat.cpu.all.sys%
      1.30            +0.5        1.85 ±  3%  mpstat.cpu.all.usr%
     37.00           +47.7%      54.67        vmstat.procs.r
  14204693           -23.0%   10930950        vmstat.system.cs
    295062            +6.5%     314162        vmstat.system.in
     21070 ±  5%     -18.5%      17172 ±  3%  proc-vmstat.nr_active_anon
    106997            -2.5%     104321        proc-vmstat.nr_anon_pages
    409349 ±  4%     +10.0%     450452 ±  4%  proc-vmstat.nr_mapped
     83608            -1.5%      82352        proc-vmstat.nr_slab_unreclaimable
     21070 ±  5%     -18.5%      17172 ±  3%  proc-vmstat.nr_zone_active_anon
    531984 ±  9%     -15.9%     447458 ±  6%  proc-vmstat.numa_hint_faults_local
     24368 ±  7%     -29.6%      17161 ±  4%  proc-vmstat.pgactivate
   7844373            -3.4%    7576615        proc-vmstat.pgalloc_normal
   1450970 ±  2%     -18.0%    1189301        proc-vmstat.pgfree
      1358           -36.3%     864.67        turbostat.Avg_MHz
     41.28           -15.0       26.27        turbostat.Busy%
 1.281e+09          -100.0%      15898 ± 95%  turbostat.C1
     12.60           -12.6        0.00        turbostat.C1%
     58.66           +25.5%      73.65        turbostat.CPU%c1
      0.21           +77.0%       0.37        turbostat.IPC
     14.47 ± 21%     -14.5        0.00        turbostat.PKG_%
 8.861e+08          -100.0%       4493 ± 14%  turbostat.POLL
      6.47            -6.5        0.00        turbostat.POLL%
    571.65            -8.5%     522.90        turbostat.PkgWatt
      8.27            -4.0%       7.94        turbostat.RAMWatt
     27233 ± 32%     -95.4%       1255 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    750764 ±  9%     -76.0%     180343 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.max
    133981 ± 19%     -88.8%      14993 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.26           +23.0%       0.32 ±  6%  sched_debug.cfs_rq:/.h_nr_running.avg
      1.03 ±  6%     +88.1%       1.93 ±  4%  sched_debug.cfs_rq:/.h_nr_running.max
      0.42           +46.9%       0.62 ±  5%  sched_debug.cfs_rq:/.h_nr_running.stddev
      9751 ± 10%     -32.8%       6552 ± 13%  sched_debug.cfs_rq:/.load.avg
     27233 ± 32%     -95.4%       1255 ±136%  sched_debug.cfs_rq:/.max_vruntime.avg
    750764 ±  9%     -76.0%     180343 ±136%  sched_debug.cfs_rq:/.max_vruntime.max
    133981 ± 19%     -88.8%      14993 ±136%  sched_debug.cfs_rq:/.max_vruntime.stddev
    724533 ±  8%     +35.8%     983881 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
    845404 ± 10%     +54.1%    1302855 ±  8%  sched_debug.cfs_rq:/.min_vruntime.max
     27934 ± 13%    +297.8%     111130 ± 11%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.26           -18.8%       0.21 ±  4%  sched_debug.cfs_rq:/.nr_running.avg
    197.05           +90.2%     374.73 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
    849.80 ±  4%     +77.3%       1506 ±  2%  sched_debug.cfs_rq:/.runnable_avg.max
    209.48          +161.4%     547.51        sched_debug.cfs_rq:/.runnable_avg.stddev
    114133 ± 29%    +172.2%     310722 ± 39%  sched_debug.cfs_rq:/.spread0.max
    -65385          +469.6%    -372412        sched_debug.cfs_rq:/.spread0.min
     27949 ± 13%    +297.7%     111154 ± 11%  sched_debug.cfs_rq:/.spread0.stddev
    196.98           +35.1%     266.15 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
    849.66 ±  4%     +35.1%       1147 ±  3%  sched_debug.cfs_rq:/.util_avg.max
    209.42           +86.8%     391.19        sched_debug.cfs_rq:/.util_avg.stddev
     93.07 ±  4%     +76.1%     163.93 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    664.98 ±  8%     +48.2%     985.29 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.max
    162.79           +91.6%     311.99        sched_debug.cfs_rq:/.util_est_enqueued.stddev
    250834 ±  9%     +84.0%     461429 ±  5%  sched_debug.cpu.avg_idle.avg
      3157 ±  6%    +939.5%      32825 ± 45%  sched_debug.cpu.avg_idle.min
    282325 ±  5%     +16.5%     328926        sched_debug.cpu.avg_idle.stddev
      7.58 ±  3%     +34.3%      10.18 ±  2%  sched_debug.cpu.clock.stddev
     10236 ±  4%     -13.5%       8854 ±  5%  sched_debug.cpu.curr->pid.max
      0.00 ± 13%     +55.3%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
      0.23 ±  3%     +41.6%       0.32 ±  5%  sched_debug.cpu.nr_running.avg
      1.03 ±  6%     +81.6%       1.87 ±  5%  sched_debug.cpu.nr_running.max
      0.40 ±  2%     +53.6%       0.62 ±  4%  sched_debug.cpu.nr_running.stddev
  13964454 ± 10%     -31.5%    9562330 ±  9%  sched_debug.cpu.nr_switches.avg
  15141338 ± 10%     -19.0%   12257576 ±  7%  sched_debug.cpu.nr_switches.max
  12639092 ±  9%     -54.5%    5746871 ± 11%  sched_debug.cpu.nr_switches.min
    444327 ± 13%    +150.9%    1114942 ± 10%  sched_debug.cpu.nr_switches.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     12.33           -96.1%       0.48 ±  7%  perf-stat.i.MPKI
 2.876e+10           +11.9%  3.217e+10        perf-stat.i.branch-instructions
      1.21            +0.1        1.26        perf-stat.i.branch-miss-rate%
 3.406e+08           +17.8%  4.012e+08        perf-stat.i.branch-misses
      1.90 ± 15%     +34.8       36.70 ± 16%  perf-stat.i.cache-miss-rate%
 1.739e+09           -95.6%   76617839 ±  7%  perf-stat.i.cache-references
  14360622           -23.1%   11042127        perf-stat.i.context-switches
      1.43           -46.7%       0.76        perf-stat.i.cpi
 2.023e+11           -38.7%  1.239e+11        perf-stat.i.cpu-cycles
      1322         +1109.9%      15995 ±  2%  perf-stat.i.cpu-migrations
      7928 ± 19%     -35.8%       5089 ± 19%  perf-stat.i.cycles-between-cache-misses
      0.02 ±  6%      -0.0        0.01 ±  4%  perf-stat.i.dTLB-load-miss-rate%
   6843508 ±  6%     -32.1%    4644002 ±  3%  perf-stat.i.dTLB-load-misses
 4.213e+10           +15.6%   4.87e+10        perf-stat.i.dTLB-loads
      0.00 ±  5%      -0.0        0.00 ±  4%  perf-stat.i.dTLB-store-miss-rate%
 2.439e+10           +14.5%  2.793e+10        perf-stat.i.dTLB-stores
     62.06            +2.8       64.90        perf-stat.i.iTLB-load-miss-rate%
 1.596e+08 ±  2%     +44.2%  2.302e+08        perf-stat.i.iTLB-load-misses
  97319700           +27.8%  1.244e+08        perf-stat.i.iTLB-loads
 1.429e+11           +14.0%  1.629e+11        perf-stat.i.instructions
    918.99 ±  2%     -19.1%     743.46        perf-stat.i.instructions-per-iTLB-miss
      0.71           +85.9%       1.31        perf-stat.i.ipc
      1.40           -38.7%       0.86        perf-stat.i.metric.GHz
    873.16           +76.2%       1538 ±  3%  perf-stat.i.metric.K/sec
    673.72           +12.2%     755.60        perf-stat.i.metric.M/sec
     82.06 ±  3%      +9.3       91.32        perf-stat.i.node-store-miss-rate%
   1049204 ± 16%     +53.8%    1613542 ± 18%  perf-stat.i.node-store-misses
    443904 ±  6%     -30.0%     310869 ±  6%  perf-stat.i.node-stores
     12.17           -96.1%       0.47 ±  7%  perf-stat.overall.MPKI
      1.18            +0.1        1.25        perf-stat.overall.branch-miss-rate%
      1.75 ± 17%     +36.3       38.07 ± 15%  perf-stat.overall.cache-miss-rate%
      1.42           -46.3%       0.76        perf-stat.overall.cpi
      6820 ± 15%     -35.7%       4383 ± 17%  perf-stat.overall.cycles-between-cache-misses
      0.02 ±  6%      -0.0        0.01 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  5%      -0.0        0.00 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     62.11            +2.8       64.92        perf-stat.overall.iTLB-load-miss-rate%
    895.64 ±  2%     -21.0%     707.76        perf-stat.overall.instructions-per-iTLB-miss
      0.71           +86.1%       1.31        perf-stat.overall.ipc
     69.85 ±  6%     +13.7       83.54 ±  2%  perf-stat.overall.node-store-miss-rate%
     39789           -25.6%      29606        perf-stat.overall.path-length
 2.866e+10           +11.9%  3.206e+10        perf-stat.ps.branch-instructions
 3.395e+08           +17.8%  3.999e+08        perf-stat.ps.branch-misses
 1.733e+09           -95.6%   76361868 ±  7%  perf-stat.ps.cache-references
  14312085           -23.1%   11004648        perf-stat.ps.context-switches
 2.016e+11           -38.7%  1.235e+11        perf-stat.ps.cpu-cycles
      1317         +1109.8%      15942 ±  2%  perf-stat.ps.cpu-migrations
   6808921 ±  6%     -32.1%    4621256 ±  3%  perf-stat.ps.dTLB-load-misses
 4.199e+10           +15.6%  4.854e+10        perf-stat.ps.dTLB-loads
 2.431e+10           +14.5%  2.784e+10        perf-stat.ps.dTLB-stores
 1.591e+08 ±  2%     +44.2%  2.294e+08        perf-stat.ps.iTLB-load-misses
  96991959           +27.8%  1.239e+08        perf-stat.ps.iTLB-loads
 1.424e+11           +14.0%  1.624e+11        perf-stat.ps.instructions
      0.01 ± 50%     -72.0%       0.00 ±115%  perf-stat.ps.major-faults
   1046642 ± 16%     +53.7%    1608395 ± 18%  perf-stat.ps.node-store-misses
    442450 ±  6%     -29.9%     309961 ±  7%  perf-stat.ps.node-stores
 4.311e+13           +14.1%  4.919e+13        perf-stat.total.instructions
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     55.50 ±  5%     -20.9       34.61 ± 14%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     55.49 ±  5%     -20.9       34.61 ± 14%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     55.40 ±  5%     -20.8       34.61 ± 14%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     48.23 ±  7%     -13.7       34.56 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     46.09 ±  7%     -11.7       34.40 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     46.05 ±  7%     -11.7       34.36 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     10.29 ±  7%     -10.3        0.00        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     10.00 ±  6%     -10.0        0.00        perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      9.36 ±  6%      -9.4        0.00        perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      1.58 ±  7%      -1.2        0.40 ± 72%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      6.53 ±  6%      -1.1        5.42 ± 13%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.send_omni_inner
      0.00            +0.8        0.82 ±  8%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +0.9        0.88 ± 42%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni
      0.00            +0.9        0.93 ±  9%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.wait_woken
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        1.03 ± 22%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.2        1.15 ±  8%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
      1.54 ±  6%      +1.2        2.76 ±  8%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.00            +1.3        1.32 ±  7%  perf-profile.calltrace.cycles-pp.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      5.02 ±  7%      +1.3        6.35 ±  8%  perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      1.33 ±  7%      +1.4        2.74 ±  7%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     10.51 ±  5%      +1.4       11.96 ±  8%  perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      8.66 ±  7%      +1.5       10.14 ±  8%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     14.38 ±  6%      +1.5       15.87 ±  7%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
      8.75 ±  7%      +1.5       10.27 ±  8%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      1.06 ± 22%      +1.5        2.60 ±  8%  perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     14.55 ±  6%      +1.5       16.09 ±  7%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
     14.67 ±  6%      +1.6       16.26 ±  7%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     13.28 ±  6%      +1.7       14.93 ±  8%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.7        1.67 ±  8%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
     10.67 ±  5%      +1.7       12.38 ±  7%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
      0.00            +1.7        1.75 ± 14%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      2.39 ±  6%      +1.8        4.16 ±  8%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
     10.70 ±  5%      +1.8       12.47 ±  7%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      0.00            +1.8        1.78 ±  8%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
      0.00            +1.8        1.79 ± 13%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      0.00            +1.8        1.84 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_skb.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      3.91 ±  7%      +1.9        5.78 ±  8%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      3.68 ±  7%      +1.9        5.58 ±  8%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
      4.03 ±  7%      +1.9        5.97 ±  8%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
      0.00            +2.0        1.99 ±  8%  perf-profile.calltrace.cycles-pp.__send
     11.19 ±  6%      +2.2       13.36 ±  7%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
      0.00            +2.3        2.26 ± 21%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner
      0.00            +2.3        2.29 ±  9%  perf-profile.calltrace.cycles-pp.recv
     11.56 ±  6%      +2.5       14.09 ±  7%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
      0.00            +2.6        2.59 ±  8%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
     11.58 ±  6%      +2.6       14.18 ±  7%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
      0.00            +2.7        2.72 ±  8%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +2.9        2.90 ±  8%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.38 ±  6%      +3.4       19.76 ±  7%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      0.00            +3.9        3.86 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     17.18 ±  6%      +4.3       21.44 ±  6%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
     12.91 ±  6%      +4.6       17.50 ±  7%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni
     18.67 ±  6%      +5.3       24.00 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
     13.02 ±  6%      +5.5       18.56 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni.process_requests
     13.10 ±  6%      +5.7       18.77 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.recv_omni.process_requests.spawn_child
     13.24 ±  6%      +5.8       19.06 ±  7%  perf-profile.calltrace.cycles-pp.__send.recv_omni.process_requests.spawn_child.accept_connection
     20.18 ±  6%      +6.3       26.50 ±  6%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
     20.27 ±  6%      +6.5       26.78 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     12.70 ±  6%      +6.5       19.22 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner
     21.45 ±  6%      +8.3       29.75 ±  7%  perf-profile.calltrace.cycles-pp.recv_omni.process_requests.spawn_child.accept_connection.accept_connections
     12.80 ±  6%      +8.8       21.60 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner.send_tcp_rr
     19.60 ±  7%      +8.9       28.49 ±  7%  perf-profile.calltrace.cycles-pp.send_omni_inner.send_tcp_rr.main.__libc_start_main
     24.83 ± 19%      +8.9       33.77 ± 15%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     24.83 ± 19%      +8.9       33.77 ± 15%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     19.76 ±  7%      +9.0       28.80 ±  7%  perf-profile.calltrace.cycles-pp.send_tcp_rr.main.__libc_start_main
     11.96 ±  7%      +9.4       21.31 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner.send_tcp_rr.main
     24.07 ±  6%      +9.4       33.44 ±  7%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     12.09 ±  7%      +9.4       21.47 ±  9%  perf-profile.calltrace.cycles-pp.__send.send_omni_inner.send_tcp_rr.main.__libc_start_main
     24.63 ±  6%      +9.8       34.41 ±  7%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     25.07 ±  6%     +10.3       35.41 ±  7%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     25.49 ±  6%     +11.0       36.46 ±  7%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.children.cycles-pp.cpu_startup_entry
     55.50 ±  5%     -20.9       34.61 ± 14%  perf-profile.children.cycles-pp.start_secondary
     55.74 ±  5%     -20.9       34.84 ± 14%  perf-profile.children.cycles-pp.do_idle
     48.52 ±  7%     -13.7       34.79 ± 14%  perf-profile.children.cycles-pp.cpuidle_idle_call
     46.36 ±  7%     -11.7       34.63 ± 14%  perf-profile.children.cycles-pp.cpuidle_enter
     46.34 ±  7%     -11.7       34.63 ± 14%  perf-profile.children.cycles-pp.cpuidle_enter_state
     10.35 ±  7%     -10.4        0.00        perf-profile.children.cycles-pp.poll_idle
     10.06 ±  6%     -10.1        0.00        perf-profile.children.cycles-pp.intel_idle_irq
      3.04 ±  7%      -2.3        0.72 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.70 ±  5%      -1.6        0.15 ± 16%  perf-profile.children.cycles-pp.menu_select
      1.33 ±  6%      -1.3        0.03 ± 70%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      1.68 ±  7%      -0.9        0.75 ±  8%  perf-profile.children.cycles-pp.dequeue_entity
      0.92 ±  7%      -0.6        0.29 ± 18%  perf-profile.children.cycles-pp.select_task_rq
      0.73 ±  4%      -0.6        0.10 ± 18%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.90 ±  8%      -0.6        0.28 ±  7%  perf-profile.children.cycles-pp.skb_attempt_defer_free
      0.79 ±  7%      -0.6        0.24 ± 19%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.66 ±  7%      -0.5        0.19 ± 10%  perf-profile.children.cycles-pp.sock_rfree
      0.48 ±  3%      -0.4        0.09 ± 20%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.11 ± 10%      -0.4        0.74 ± 18%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.64 ±  5%      -0.4        0.28 ± 11%  perf-profile.children.cycles-pp.remove_wait_queue
      0.96 ±  7%      -0.3        0.62 ± 11%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.81 ±  5%      -0.3        0.47 ± 13%  perf-profile.children.cycles-pp.prepare_task_switch
      0.82 ±  7%      -0.3        0.52 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock
      0.57 ±  9%      -0.2        0.33 ± 11%  perf-profile.children.cycles-pp.tcp_check_space
      1.10 ±  7%      -0.2        0.88 ±  8%  perf-profile.children.cycles-pp.enqueue_entity
      0.56 ±  8%      -0.2        0.35 ±  9%  perf-profile.children.cycles-pp.copyout
      0.55 ±  9%      -0.2        0.34 ±  7%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.35 ±  8%      -0.2        0.16 ±  8%  perf-profile.children.cycles-pp.kfree_skbmem
      0.58 ±  7%      -0.2        0.41 ±  7%  perf-profile.children.cycles-pp.copy_user_short_string
      0.31 ±  8%      -0.2        0.15 ± 11%  perf-profile.children.cycles-pp.perf_tp_event
      0.20 ±  8%      -0.1        0.05 ±  7%  perf-profile.children.cycles-pp.skb_release_head_state
      0.58 ±  7%      -0.1        0.47 ±  8%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.09 ± 15%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.rebalance_domains
      0.07 ± 11%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
      0.07 ±  7%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.tcp_rate_skb_delivered
      0.15 ±  7%      +0.0        0.18 ±  9%  perf-profile.children.cycles-pp.add_wait_queue
      0.07 ±  7%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.tcp_event_data_recv
      0.04 ± 44%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.place_entity
      0.11 ± 11%      +0.0        0.15 ± 15%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.06 ± 14%      +0.0        0.11 ± 12%  perf-profile.children.cycles-pp.ip_skb_dst_mtu
      0.02 ±141%      +0.0        0.06 ± 21%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.05 ± 47%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.nf_hook_slow
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.22 ± 10%      +0.1        0.27 ±  9%  perf-profile.children.cycles-pp.__mod_timer
      0.09 ± 14%      +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.minmax_running_min
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ip_local_deliver
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.tcp_options_write
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.inet_sendmsg
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.skb_clone
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.tcp_small_queue_check
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.cubictcp_cong_avoid
      0.00            +0.1        0.06 ± 18%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.tcp_chrono_stop
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.eth_type_trans
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.kmalloc_slab
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__raise_softirq_irqoff
      0.06 ±  6%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.switch_ldt
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.rb_insert_color
      0.27 ±  9%      +0.1        0.33 ±  7%  perf-profile.children.cycles-pp.sk_reset_timer
      0.10 ± 15%      +0.1        0.17 ± 11%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.10 ±  8%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.__ip_finish_output
      0.10 ± 14%      +0.1        0.17 ± 11%  perf-profile.children.cycles-pp.__tcp_cleanup_rbuf
      0.08 ±  8%      +0.1        0.15 ± 11%  perf-profile.children.cycles-pp.update_min_vruntime
      0.06 ±  9%      +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.09 ±  6%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.try_charge_memcg
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.clear_buddies
      0.04 ± 44%      +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.tcp_established_options
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.apparmor_socket_recvmsg
      0.61 ±  7%      +0.1        0.68 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.08 ±  6%      +0.1        0.15 ±  9%  perf-profile.children.cycles-pp.__copy_skb_header
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.cubictcp_acked
      0.06 ±  9%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.06 ± 11%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.tcp_rate_skb_sent
      0.08 ± 13%      +0.1        0.16 ±  8%  perf-profile.children.cycles-pp.rcu_all_qs
      0.12 ± 12%      +0.1        0.20 ±  8%  perf-profile.children.cycles-pp.ip_send_check
      0.06 ±  9%      +0.1        0.13 ±  7%  perf-profile.children.cycles-pp.inet_ehashfn
      0.03 ± 70%      +0.1        0.11 ±  8%  perf-profile.children.cycles-pp.sk_free
      0.06 ± 11%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.raw_v4_input
      0.00            +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.tcp_inbound_md5_hash
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.neigh_hh_output
      0.19 ± 17%      +0.1        0.28 ± 18%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.tcp_rate_gen
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.apparmor_socket_sendmsg
      0.58 ±  8%      +0.1        0.67 ±  8%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.00            +0.1        0.09 ±  6%  perf-profile.children.cycles-pp.check_stack_object
      0.00            +0.1        0.09 ±  6%  perf-profile.children.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.tcp_newly_delivered
      0.07 ± 13%      +0.1        0.16 ±  5%  perf-profile.children.cycles-pp.sock_put
      0.10 ±  8%      +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.__ksize
      0.05 ± 45%      +0.1        0.14 ± 13%  perf-profile.children.cycles-pp.netif_skb_features
      0.10 ±  9%      +0.1        0.20 ± 14%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      0.04 ± 71%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.__usecs_to_jiffies
      0.09 ± 14%      +0.1        0.19 ± 14%  perf-profile.children.cycles-pp.tcp_rtt_estimator
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.tcp_push
      0.10 ± 10%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.__tcp_select_window
      0.63 ±  6%      +0.1        0.74 ±  7%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.00            +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.resched_curr
      0.07 ± 12%      +0.1        0.19 ± 22%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.43 ±  6%      +0.1        0.54 ±  9%  perf-profile.children.cycles-pp.update_rq_clock
      0.12 ±  9%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.ip_output
      0.18 ±  9%      +0.1        0.29 ± 14%  perf-profile.children.cycles-pp.__calc_delta
      0.50 ±  6%      +0.1        0.63 ±  9%  perf-profile.children.cycles-pp.set_next_entity
      0.09 ±  6%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.28 ±  7%      +0.1        0.41 ±  8%  perf-profile.children.cycles-pp.recv_data
      0.00            +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.cubictcp_cwnd_event
      0.04 ± 71%      +0.1        0.17 ± 13%  perf-profile.children.cycles-pp.skb_page_frag_refill
      0.11 ±  8%      +0.1        0.24 ± 10%  perf-profile.children.cycles-pp.tcp_tso_segs
      0.07 ± 14%      +0.1        0.20 ± 10%  perf-profile.children.cycles-pp.sk_page_frag_refill
      0.11 ±  6%      +0.1        0.25 ±  6%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.30 ±  6%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.08 ± 18%      +0.1        0.23 ± 19%  perf-profile.children.cycles-pp.cpuacct_charge
      0.29 ±  9%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.08 ±  8%      +0.1        0.23 ±  9%  perf-profile.children.cycles-pp.ip_rcv_core
      0.16 ±  9%      +0.2        0.32 ±  8%  perf-profile.children.cycles-pp.__ip_local_out
      0.29 ±  6%      +0.2        0.46 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.12 ±  6%      +0.2        0.28 ± 10%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.35 ±  4%      +0.2        0.52 ±  8%  perf-profile.children.cycles-pp.update_cfs_group
      0.14 ±  5%      +0.2        0.31 ±  7%  perf-profile.children.cycles-pp.copyin
      0.11 ±  6%      +0.2        0.28 ±  8%  perf-profile.children.cycles-pp.refill_stock
      0.18 ±  7%      +0.2        0.35 ±  8%  perf-profile.children.cycles-pp.__cond_resched
      0.14 ±  7%      +0.2        0.32 ±  8%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.11 ±  9%      +0.2        0.28 ±  8%  perf-profile.children.cycles-pp.tcp_update_skb_after_send
      0.15 ± 10%      +0.2        0.32 ± 15%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.17 ±  6%      +0.2        0.34 ±  7%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.12 ±  9%      +0.2        0.30 ± 14%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.15 ± 20%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.find_busiest_group
      0.07 ±  9%      +0.2        0.26 ±  8%  perf-profile.children.cycles-pp.__list_add_valid
      0.18 ±  8%      +0.2        0.36 ± 10%  perf-profile.children.cycles-pp.ip_local_out
      0.17 ±  6%      +0.2        0.36 ± 17%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.16 ±  4%      +0.2        0.35 ±  9%  perf-profile.children.cycles-pp.tcp_wfree
      0.46 ±  6%      +0.2        0.66 ±  7%  perf-profile.children.cycles-pp.read_tsc
      0.11 ±  9%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.__might_fault
      0.00            +0.2        0.20 ± 10%  perf-profile.children.cycles-pp.set_next_buddy
      0.12 ± 21%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.19 ± 17%      +0.2        0.40 ± 17%  perf-profile.children.cycles-pp.load_balance
      0.06 ±  7%      +0.2        0.27 ±  8%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.10 ± 23%      +0.2        0.32 ± 17%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.18 ±  8%      +0.2        0.39 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.23 ±  9%      +0.2        0.45 ±  7%  perf-profile.children.cycles-pp.send_data
      0.35 ±  8%      +0.2        0.58 ±  8%  perf-profile.children.cycles-pp.__skb_clone
      0.67 ±  6%      +0.2        0.90 ±  9%  perf-profile.children.cycles-pp.__switch_to_asm
      0.07 ± 10%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.kmem_cache_free
      0.07 ± 12%      +0.2        0.32 ±  6%  perf-profile.children.cycles-pp.put_prev_entity
      0.12 ± 10%      +0.2        0.37 ±  8%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.24 ±  7%      +0.3        0.50 ±  9%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
      0.27 ± 12%      +0.3        0.54 ± 12%  perf-profile.children.cycles-pp.ip_rcv
      0.48 ±  9%      +0.3        0.75 ±  7%  perf-profile.children.cycles-pp.release_sock
      0.22 ±  8%      +0.3        0.52 ±  6%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.25 ±  7%      +0.3        0.56 ±  8%  perf-profile.children.cycles-pp.os_xsave
      1.01 ±  6%      +0.3        1.33 ±  8%  perf-profile.children.cycles-pp.skb_release_data
      0.56 ±  8%      +0.3        0.87 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.24 ±  8%      +0.3        0.56 ±  9%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.14 ±  3%      +0.3        0.47 ±  7%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.20 ±  8%      +0.3        0.54 ±  8%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.83 ±  5%      +0.3        1.17 ±  7%  perf-profile.children.cycles-pp.__check_object_size
      0.06 ±  9%      +0.4        0.41 ±  9%  perf-profile.children.cycles-pp.import_single_range
      0.35 ±  7%      +0.4        0.71 ±  7%  perf-profile.children.cycles-pp.lock_sock_nested
      0.13 ± 10%      +0.4        0.50 ± 11%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.10 ± 13%      +0.4        0.47 ± 11%  perf-profile.children.cycles-pp.check_preempt_curr
      0.27 ±  6%      +0.4        0.66 ±  9%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.20 ±  9%      +0.4        0.59 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.84 ±  2%      +0.4        1.23 ± 11%  perf-profile.children.cycles-pp.ktime_get
      0.30 ±  5%      +0.4        0.70 ±  8%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.28 ±  7%      +0.4        0.68 ±  9%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.19 ±  5%      +0.4        0.60 ±  9%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.94 ±  6%      +0.4        1.34 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.4        0.41 ± 12%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.21 ±  6%      +0.4        0.63 ±  8%  perf-profile.children.cycles-pp.netif_rx_internal
      0.21 ±  5%      +0.4        0.64 ±  9%  perf-profile.children.cycles-pp.__netif_rx
      0.30 ±  8%      +0.4        0.74 ±  9%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.34 ±  5%      +0.4        0.78 ±  7%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.26 ±  8%      +0.4        0.71 ± 10%  perf-profile.children.cycles-pp.reweight_entity
      0.27 ± 11%      +0.5        0.75 ±  9%  perf-profile.children.cycles-pp.__fdget
      0.41 ±  5%      +0.5        0.89 ±  8%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.00            +0.5        0.50 ± 10%  perf-profile.children.cycles-pp.kfree
      0.22 ±  7%      +0.5        0.76 ±  8%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.29 ±  6%      +0.5        0.84 ±  7%  perf-profile.children.cycles-pp._copy_from_iter
      0.77 ±  8%      +0.5        1.31 ±  9%  perf-profile.children.cycles-pp.update_curr
      0.35 ±  6%      +0.6        0.91 ±  7%  perf-profile.children.cycles-pp.aa_sk_perm
      0.24 ±  8%      +0.6        0.80 ±  8%  perf-profile.children.cycles-pp.sock_recvmsg
      0.43 ±  7%      +0.6        0.99 ±  7%  perf-profile.children.cycles-pp.__entry_text_start
      0.33 ± 10%      +0.6        0.91 ±  9%  perf-profile.children.cycles-pp.sockfd_lookup_light
      1.23 ±  8%      +0.6        1.82 ±  7%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.54 ±  7%      +0.6        1.18 ±  8%  perf-profile.children.cycles-pp.__kfree_skb
      1.28 ±  7%      +0.8        2.04 ±  8%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.32 ±  7%      +0.8        2.09 ±  8%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.54 ±  6%      +0.8        1.36 ±  7%  perf-profile.children.cycles-pp.skb_do_copy_data_nocache
      0.00            +1.0        1.01 ±  8%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.63 ±  6%      +1.1        1.70 ±  8%  perf-profile.children.cycles-pp.loopback_xmit
      0.82 ±  7%      +1.1        1.91 ±  8%  perf-profile.children.cycles-pp.__alloc_skb
      5.25 ±  7%      +1.1        6.39 ±  8%  perf-profile.children.cycles-pp.wait_woken
      0.68 ±  5%      +1.1        1.82 ±  8%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.15 ±  8%      +1.2        1.33 ±  8%  perf-profile.children.cycles-pp.switch_fpu_return
      1.65 ±  6%      +1.2        2.87 ±  8%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      9.07 ±  7%      +1.4       10.45 ±  8%  perf-profile.children.cycles-pp.tcp_rcv_established
     10.89 ±  6%      +1.4       12.30 ±  8%  perf-profile.children.cycles-pp.tcp_v4_rcv
      1.40 ±  6%      +1.4        2.82 ±  8%  perf-profile.children.cycles-pp.__dev_queue_xmit
      9.14 ±  6%      +1.4       10.56 ±  8%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      1.20 ±  7%      +1.5        2.68 ±  8%  perf-profile.children.cycles-pp.tcp_stream_alloc_skb
     13.56 ±  7%      +1.6       15.16 ±  8%  perf-profile.children.cycles-pp.__sys_recvfrom
     11.05 ±  6%      +1.7       12.71 ±  8%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     13.66 ±  7%      +1.7       15.36 ±  8%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
     11.08 ±  6%      +1.7       12.80 ±  8%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      2.51 ±  6%      +1.8        4.28 ±  8%  perf-profile.children.cycles-pp.tcp_ack
      4.22 ±  7%      +1.8        6.02 ±  7%  perf-profile.children.cycles-pp.schedule_timeout
      0.10 ±  9%      +2.1        2.19 ±  8%  perf-profile.children.cycles-pp.switch_mm_irqs_off
     11.50 ±  6%      +2.2       13.70 ±  7%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     11.89 ±  6%      +2.5       14.38 ±  7%  perf-profile.children.cycles-pp.process_backlog
     11.90 ±  6%      +2.6       14.46 ±  7%  perf-profile.children.cycles-pp.__napi_poll
      6.04 ±  7%      +2.6        8.61 ±  7%  perf-profile.children.cycles-pp.__schedule
      0.00            +3.2        3.24 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
     16.72 ±  6%      +3.3       20.05 ±  7%  perf-profile.children.cycles-pp.ip_finish_output2
     15.81 ±  7%      +3.7       19.46 ±  8%  perf-profile.children.cycles-pp.recv
     17.54 ±  6%      +4.2       21.74 ±  7%  perf-profile.children.cycles-pp.__ip_queue_xmit
      0.20 ±  8%      +4.7        4.87 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.10 ±  7%      +4.8        8.90 ±  7%  perf-profile.children.cycles-pp.schedule
      0.32 ±  6%      +4.8        5.12 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     18.90 ±  6%      +5.5       24.36 ±  7%  perf-profile.children.cycles-pp.__tcp_transmit_skb
     20.44 ±  6%      +6.5       26.93 ±  7%  perf-profile.children.cycles-pp.tcp_write_xmit
     20.51 ±  6%      +6.6       27.14 ±  7%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
     21.50 ±  6%      +8.3       29.79 ±  6%  perf-profile.children.cycles-pp.send_tcp_rr
     21.57 ±  6%      +8.5       30.03 ±  7%  perf-profile.children.cycles-pp.send_omni_inner
     21.58 ±  6%      +8.5       30.06 ±  7%  perf-profile.children.cycles-pp.recv_omni
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.accept_connections
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.accept_connection
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.spawn_child
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.process_requests
     24.97 ± 19%      +9.0       34.00 ± 15%  perf-profile.children.cycles-pp.intel_idle
     24.26 ±  6%      +9.3       33.55 ±  7%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
     24.84 ±  6%      +9.7       34.54 ±  7%  perf-profile.children.cycles-pp.tcp_sendmsg
     25.26 ±  6%     +10.2       35.49 ±  7%  perf-profile.children.cycles-pp.sock_sendmsg
     25.68 ±  6%     +10.9       36.56 ±  7%  perf-profile.children.cycles-pp.__sys_sendto
     25.80 ±  6%     +11.0       36.82 ±  7%  perf-profile.children.cycles-pp.__x64_sys_sendto
     26.86 ±  6%     +17.5       44.40 ±  7%  perf-profile.children.cycles-pp.__send
     40.08 ±  6%     +17.8       57.92 ±  7%  perf-profile.children.cycles-pp.do_syscall_64
     40.47 ±  6%     +18.4       58.88 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     10.22 ±  7%     -10.2        0.00        perf-profile.self.cycles-pp.poll_idle
      3.03 ±  7%      -2.3        0.70 ± 10%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.03 ±  7%      -0.7        0.37 ±  9%  perf-profile.self.cycles-pp.wait_woken
      0.66 ±  7%      -0.5        0.19 ± 10%  perf-profile.self.cycles-pp.sock_rfree
      0.91 ±  7%      -0.4        0.49 ± 12%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.64 ±  8%      -0.4        0.26 ±  8%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.92 ±  6%      -0.4        0.55 ±  6%  perf-profile.self.cycles-pp.skb_release_data
      0.40 ±  9%      -0.3        0.06 ±  6%  perf-profile.self.cycles-pp.skb_attempt_defer_free
      0.40 ±  8%      -0.3        0.08 ±  9%  perf-profile.self.cycles-pp.tcp_rcv_space_adjust
      0.82 ±  7%      -0.3        0.50 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock
      0.54 ±  5%      -0.3        0.24 ±  9%  perf-profile.self.cycles-pp.check_heap_object
      0.43 ±  6%      -0.3        0.16 ± 14%  perf-profile.self.cycles-pp.prepare_task_switch
      0.57 ±  6%      -0.3        0.32 ± 11%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.56 ±  9%      -0.2        0.32 ± 10%  perf-profile.self.cycles-pp.tcp_check_space
      0.39 ±  7%      -0.2        0.19 ± 11%  perf-profile.self.cycles-pp.enqueue_entity
      0.35 ±  8%      -0.2        0.16 ±  8%  perf-profile.self.cycles-pp.kfree_skbmem
      0.57 ±  7%      -0.2        0.40 ±  8%  perf-profile.self.cycles-pp.copy_user_short_string
      0.26 ±  9%      -0.2        0.09 ±  7%  perf-profile.self.cycles-pp.__wake_up_common
      0.19 ±  7%      -0.2        0.03 ± 70%  perf-profile.self.cycles-pp.skb_release_head_state
      0.22 ±  8%      -0.2        0.07 ± 20%  perf-profile.self.cycles-pp.perf_tp_event
      0.21 ±  4%      -0.2        0.06 ± 19%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.67 ±  7%      -0.2        0.52 ±  8%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.25 ±  9%      -0.1        0.14 ± 12%  perf-profile.self.cycles-pp.dequeue_entity
      0.14 ±  9%      -0.1        0.05 ± 46%  perf-profile.self.cycles-pp.select_task_rq
      0.34 ±  3%      -0.1        0.27 ± 14%  perf-profile.self.cycles-pp.___perf_sw_event
      0.16 ± 12%      -0.1        0.10 ± 16%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.20 ±  7%      -0.0        0.16 ±  7%  perf-profile.self.cycles-pp.finish_task_switch
      0.09 ± 11%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.check_preempt_curr
      0.06 ±  7%      -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
      0.06 ± 11%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.tcp_event_data_recv
      0.05 ± 45%      +0.0        0.08 ±  7%  perf-profile.self.cycles-pp.put_prev_entity
      0.22 ±  7%      +0.0        0.25 ±  7%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.10 ± 10%      +0.0        0.14 ± 15%  perf-profile.self.cycles-pp.tcp_rearm_rto
      0.22 ±  6%      +0.0        0.26 ±  8%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.13 ±  6%      +0.0        0.17 ± 11%  perf-profile.self.cycles-pp.update_rq_clock
      0.04 ± 47%      +0.0        0.09 ± 18%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.06 ± 11%      +0.0        0.10 ± 13%  perf-profile.self.cycles-pp.ip_skb_dst_mtu
      0.02 ±142%      +0.0        0.06 ± 21%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.place_entity
      0.01 ±223%      +0.0        0.06 ± 13%  perf-profile.self.cycles-pp.skb_clone
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.minmax_running_min
      0.06 ± 13%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.clear_buddies
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.tcp_small_queue_check
      0.18 ±  7%      +0.1        0.23 ± 10%  perf-profile.self.cycles-pp.do_softirq
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.ip_local_deliver
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.skb_copy_datagram_iter
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.cubictcp_cong_avoid
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.sk_reset_timer
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.switch_ldt
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.remove_wait_queue
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__ip_finish_output
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.nf_hook_slow
      0.02 ±141%      +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.tcp_send_mss
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.eth_type_trans
      0.13 ±  8%      +0.1        0.19 ± 10%  perf-profile.self.cycles-pp.schedule_timeout
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.kmalloc_reserve
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.kmalloc_slab
      0.06 ± 13%      +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_inbound_md5_hash
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.apparmor_socket_recvmsg
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.07 ± 12%      +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.update_min_vruntime
      0.06 ± 11%      +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.tcp_rate_skb_sent
      0.04 ± 45%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.08 ±  6%      +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.ip_output
      0.04 ± 44%      +0.1        0.11 ±  5%  perf-profile.self.cycles-pp.tcp_established_options
      0.01 ±223%      +0.1        0.08 ±  9%  perf-profile.self.cycles-pp.lock_sock_nested
      0.10 ± 16%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.08 ±  8%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.try_charge_memcg
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.tcp_stream_alloc_skb
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.cubictcp_acked
      0.59 ±  6%      +0.1        0.66 ±  6%  perf-profile.self.cycles-pp.__switch_to
      0.07 ±  7%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.07 ±  8%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.__copy_skb_header
      0.06 ±  9%      +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.skb_do_copy_data_nocache
      0.04 ± 44%      +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.11 ±  6%      +0.1        0.19 ± 11%  perf-profile.self.cycles-pp._copy_to_iter
      0.04 ± 71%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.mem_cgroup_uncharge_skmem
      0.03 ± 70%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.06 ±  8%      +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.raw_v4_input
      0.03 ± 70%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.sk_free
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.neigh_hh_output
      0.26 ±  7%      +0.1        0.34 ±  9%  perf-profile.self.cycles-pp.recv_data
      0.12 ± 11%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.ip_send_check
      0.00            +0.1        0.08 ±  7%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.06 ± 11%      +0.1        0.15 ± 18%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.__napi_poll
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.08 ± 11%  perf-profile.self.cycles-pp.apparmor_socket_sendmsg
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.inet_ehashfn
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.tcp_rate_gen
      0.02 ±141%      +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.tcp_mstamp_refresh
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.netif_skb_features
      0.00            +0.1        0.09 ± 15%  perf-profile.self.cycles-pp.__might_fault
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.tcp_newly_delivered
      0.10 ±  6%      +0.1        0.19 ± 14%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      0.09 ±  7%      +0.1        0.18 ±  9%  perf-profile.self.cycles-pp.__ksize
      0.00            +0.1        0.09 ±  6%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.00            +0.1        0.09 ±  6%  perf-profile.self.cycles-pp.check_stack_object
      0.08 ± 14%      +0.1        0.18 ± 16%  perf-profile.self.cycles-pp.tcp_rtt_estimator
      0.06 ± 11%      +0.1        0.16 ±  5%  perf-profile.self.cycles-pp.sock_put
      0.10 ± 10%      +0.1        0.20 ± 10%  perf-profile.self.cycles-pp.__cond_resched
      0.22 ±  8%      +0.1        0.32 ±  9%  perf-profile.self.cycles-pp.schedule
      0.06 ±  9%      +0.1        0.16 ±  9%  perf-profile.self.cycles-pp.sockfd_lookup_light
      0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.resched_curr
      0.11 ±  6%      +0.1        0.21 ± 20%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.13 ±  9%      +0.1        0.23 ± 19%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.10 ± 10%      +0.1        0.21 ±  7%  perf-profile.self.cycles-pp.__tcp_select_window
      0.07 ±  5%      +0.1        0.17 ±  8%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.09 ±  5%      +0.1        0.20 ± 12%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.06 ± 14%      +0.1        0.17 ± 25%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_push
      0.01 ±223%      +0.1        0.11 ±  6%  perf-profile.self.cycles-pp.__usecs_to_jiffies
      0.35 ±  7%      +0.1        0.45 ±  7%  perf-profile.self.cycles-pp.update_load_avg
      0.09 ±  7%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.sock_sendmsg
      0.00            +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.ip_rcv
      0.12 ±  6%      +0.1        0.23 ±  9%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.16 ± 10%      +0.1        0.27 ±  8%  perf-profile.self.cycles-pp.sk_wait_data
      0.00            +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.__ip_local_out
      0.09 ± 14%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.18 ± 10%      +0.1        0.29 ± 14%  perf-profile.self.cycles-pp.__calc_delta
      0.09 ±  6%      +0.1        0.21 ±  8%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.24 ±  6%      +0.1        0.36 ± 11%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.00            +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.cubictcp_cwnd_event
      0.14 ±  7%      +0.1        0.27 ±  5%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.04 ± 71%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.skb_page_frag_refill
      0.15 ±  7%      +0.1        0.28 ±  8%  perf-profile.self.cycles-pp.switch_fpu_return
      0.11 ±  6%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.tcp_sendmsg
      0.07 ± 14%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.tcp_update_skb_after_send
      0.15 ±  4%      +0.1        0.29 ±  9%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.29 ±  6%      +0.1        0.42 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.29 ±  9%      +0.1        0.42 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.10 ±  7%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.tcp_tso_segs
      0.11 ±  6%      +0.1        0.25 ±  7%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.07 ± 12%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.__tcp_push_pending_frames
      0.34 ±  9%      +0.1        0.48 ±  8%  perf-profile.self.cycles-pp.update_curr
      0.18 ±  9%      +0.1        0.32 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.08 ± 19%      +0.1        0.22 ± 18%  perf-profile.self.cycles-pp.cpuacct_charge
      0.08 ±  8%      +0.1        0.23 ±  9%  perf-profile.self.cycles-pp.ip_rcv_core
      0.01 ±223%      +0.1        0.16 ± 12%  perf-profile.self.cycles-pp.inet_recvmsg
      0.29 ±  5%      +0.2        0.44 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.10 ±  5%      +0.2        0.25 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.10 ±  6%      +0.2        0.26 ±  8%  perf-profile.self.cycles-pp.process_backlog
      0.12 ±  8%      +0.2        0.27 ±  7%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.28 ±  8%      +0.2        0.44 ±  6%  perf-profile.self.cycles-pp.__skb_clone
      0.12 ±  8%      +0.2        0.28 ±  7%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.34 ±  4%      +0.2        0.51 ±  8%  perf-profile.self.cycles-pp.update_cfs_group
      0.16 ±  6%      +0.2        0.33 ±  7%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.14 ±  7%      +0.2        0.31 ±  8%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.11 ±  6%      +0.2        0.28 ±  8%  perf-profile.self.cycles-pp.refill_stock
      0.09 ± 13%      +0.2        0.26 ±  8%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.00            +0.2        0.17 ± 11%  perf-profile.self.cycles-pp.security_socket_recvmsg
      0.17 ±  9%      +0.2        0.34 ±  8%  perf-profile.self.cycles-pp.__alloc_skb
      0.08 ± 25%      +0.2        0.26 ± 17%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.27 ±  5%      +0.2        0.46 ±  7%  perf-profile.self.cycles-pp.send_omni_inner
      0.06 ±  7%      +0.2        0.25 ±  8%  perf-profile.self.cycles-pp.__list_add_valid
      0.22 ±  8%      +0.2        0.40 ± 11%  perf-profile.self.cycles-pp.recv
      0.00            +0.2        0.19 ±  8%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.20 ±  8%      +0.2        0.39 ±  8%  perf-profile.self.cycles-pp.send_data
      0.38 ±  8%      +0.2        0.57 ± 23%  perf-profile.self.cycles-pp.ktime_get
      0.45 ±  7%      +0.2        0.65 ±  7%  perf-profile.self.cycles-pp.read_tsc
      0.16 ±  4%      +0.2        0.35 ±  9%  perf-profile.self.cycles-pp.tcp_wfree
      0.00            +0.2        0.20 ±  9%  perf-profile.self.cycles-pp.set_next_buddy
      0.16 ±  9%      +0.2        0.36 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.06 ±  7%      +0.2        0.27 ±  8%  perf-profile.self.cycles-pp.__kmem_cache_free
      0.32 ±  5%      +0.2        0.53 ±  9%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.25 ± 14%      +0.2        0.46 ±  9%  perf-profile.self.cycles-pp.recv_omni
      0.62 ±  7%      +0.2        0.84 ±  8%  perf-profile.self.cycles-pp.__schedule
      0.16 ±  7%      +0.2        0.38 ±  9%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.19 ±  8%      +0.2        0.42 ±  7%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.67 ±  6%      +0.2        0.89 ±  8%  perf-profile.self.cycles-pp.__switch_to_asm
      0.09 ±  7%      +0.2        0.32 ±  8%  perf-profile.self.cycles-pp._copy_from_iter
      0.18 ±  6%      +0.2        0.42 ±  5%  perf-profile.self.cycles-pp.__send
      0.10 ± 11%      +0.2        0.34 ± 10%  perf-profile.self.cycles-pp.__check_object_size
      0.07 ± 10%      +0.2        0.31 ±  8%  perf-profile.self.cycles-pp.kmem_cache_free
      0.12 ±  7%      +0.3        0.38 ±  8%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.26 ±  6%      +0.3        0.52 ±  7%  perf-profile.self.cycles-pp.do_syscall_64
      0.21 ±  7%      +0.3        0.48 ±  7%  perf-profile.self.cycles-pp.__entry_text_start
      0.22 ±  7%      +0.3        0.50 ±  8%  perf-profile.self.cycles-pp.__sys_sendto
      0.22 ±  8%      +0.3        0.50 ±  6%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.53 ±  7%      +0.3        0.84 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.25 ±  7%      +0.3        0.56 ±  8%  perf-profile.self.cycles-pp.os_xsave
      0.13 ±  4%      +0.3        0.46 ±  7%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.16 ±  7%      +0.3        0.49 ±  7%  perf-profile.self.cycles-pp.loopback_xmit
      0.05 ±  7%      +0.3        0.40 ±  9%  perf-profile.self.cycles-pp.import_single_range
      0.32 ±  6%      +0.4        0.67 ±  9%  perf-profile.self.cycles-pp.ip_finish_output2
      0.01 ±223%      +0.4        0.36 ±  9%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.21 ±  8%      +0.4        0.59 ±  8%  perf-profile.self.cycles-pp.aa_sk_perm
      0.27 ±  6%      +0.4        0.69 ±  6%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.29 ±  6%      +0.4        0.73 ±  8%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.27 ± 10%      +0.5        0.74 ± 10%  perf-profile.self.cycles-pp.__fdget
      0.00            +0.5        0.49 ± 10%  perf-profile.self.cycles-pp.kfree
      0.66 ±  6%      +0.5        1.16 ±  8%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.38 ±  6%      +0.6        0.96 ±  8%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        1.01 ±  8%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.10 ±  9%      +2.1        2.17 ±  7%  perf-profile.self.cycles-pp.switch_mm_irqs_off



If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <yujie.liu@...el.com>
| Link: https://lore.kernel.org/oe-lkp/202211221125.64e73ddf-yujie.liu@intel.com


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

View attachment "config-6.1.0-rc2-00018-gb6aabb01e300" of type "text/plain" (166032 bytes)

View attachment "job-script" of type "text/plain" (8518 bytes)

View attachment "job.yaml" of type "text/plain" (5757 bytes)

View attachment "reproduce" of type "text/plain" (4769 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ