lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Mon, 31 Oct 2022 10:37:50 +0800
From:   kernel test robot <yujie.liu@...el.com>
To:     Vincent Guittot <vincent.guittot@...aro.org>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        Peter Zijlstra <peterz@...radead.org>,
        David Chen <david.chen@...anix.com>,
        Zhang Qiao <zhangqiao22@...wei.com>,
        <linux-kernel@...r.kernel.org>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <zhengjun.xing@...ux.intel.com>,
        <fengwei.yin@...el.com>
Subject: [linus:master] [sched/fair] c82a69629c: tbench.throughput-MB/sec
 31.4% improvement

Greeting,

FYI, we noticed a 31.4% improvement of tbench.throughput-MB/sec due to commit:

commit: c82a69629c53eda5233f13fc11c3c01585ef48a2 ("sched/fair: fix case with reduced capacity CPU")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: tbench
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
with following parameters:

	nr_threads: 100%
	cluster: cs-localhost
	cpufreq_governor: performance


Details are as below:

=========================================================================================
cluster/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/testcase:
  cs-localhost/gcc-11/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp7/tbench

commit: 
  c02d5546ea ("sched/core: Use try_cmpxchg in set_nr_{and_not,if}_polling")
  c82a69629c ("sched/fair: fix case with reduced capacity CPU")

c02d5546ea34d589 c82a69629c53eda5233f13fc11c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     11.49 ± 10%     +93.6%      22.24 ±  7%  tbench.max_latency
     10095           +31.4%      13264 ± 10%  tbench.throughput-MB/sec
  32056675 ± 23%   +2751.4%  9.141e+08 ± 34%  tbench.time.involuntary_context_switches
      2521           +42.4%       3592 ± 11%  tbench.time.percent_of_cpu_this_job_got
     13943           +41.9%      19781 ± 10%  tbench.time.system_time
      4224 ±  3%     +44.3%       6096 ± 20%  tbench.time.user_time
 1.318e+09           -31.5%  9.033e+08 ± 18%  tbench.time.voluntary_context_switches
     17117           -28.3%      12273 ± 16%  uptime.idle
 1.364e+10           -36.4%  8.685e+09 ± 22%  cpuidle..time
 1.731e+09           -31.8%  1.181e+09 ± 18%  cpuidle..usage
     29.32           +12.2%      32.90        boot-time.boot
     14.26           +24.6%      17.77        boot-time.dhcp
      2339           +13.7%       2660        boot-time.idle
     22.67            -8.4       14.25 ± 23%  mpstat.cpu.all.idle%
      3.11 ±  3%      -1.1        2.06 ± 19%  mpstat.cpu.all.irq%
     43.53            +8.1       51.66 ±  5%  mpstat.cpu.all.sys%
      8.80 ±  2%      +2.4       11.22 ± 23%  mpstat.cpu.all.usr%
   1910029 ±  2%     -41.7%    1114357 ± 32%  numa-vmstat.node1.nr_inactive_anon
    952400 ±  2%     -45.2%     521490 ± 39%  numa-vmstat.node1.nr_mapped
   1908572 ±  2%     -41.5%    1116461 ± 32%  numa-vmstat.node1.nr_shmem
   1910028 ±  2%     -41.7%    1114358 ± 32%  numa-vmstat.node1.nr_zone_inactive_anon
   7641189 ±  2%     -41.6%    4459143 ± 32%  numa-meminfo.node1.Inactive
   7640015 ±  2%     -41.6%    4458204 ± 32%  numa-meminfo.node1.Inactive(anon)
   3809092 ±  2%     -45.2%    2086458 ± 39%  numa-meminfo.node1.Mapped
  10329178 ± 12%     -24.6%    7787191 ± 27%  numa-meminfo.node1.MemUsed
   7635879 ±  2%     -41.5%    4466108 ± 32%  numa-meminfo.node1.Shmem
     22.17 ±  3%     -36.1%      14.17 ± 23%  vmstat.cpu.id
  10481018 ±  2%     -18.3%    8564745 ± 14%  vmstat.memory.cache
     97.00           +14.4%     111.00 ±  5%  vmstat.procs.r
   6128694            +9.4%    6706040 ±  3%  vmstat.system.cs
    810000           -31.2%     557330 ± 17%  vmstat.system.in
      2718            -2.6%       2648        turbostat.Bzy_MHz
 1.681e+09           -33.0%  1.126e+09 ± 19%  turbostat.C1
     18.69            -7.1       11.58 ± 23%  turbostat.C1%
     16.59           -36.3%      10.56 ± 22%  turbostat.CPU%c1
      0.12 ±  3%     +16.2%       0.14 ±  3%  turbostat.IPC
 5.867e+08           -31.2%  4.036e+08 ± 17%  turbostat.IRQ
  10364034 ±  2%     -18.5%    8447939 ± 14%  meminfo.Cached
   9993526 ±  2%     -29.3%    7066169 ± 18%  meminfo.Committed_AS
    551364 ±  3%     +73.8%     958233 ±  6%  meminfo.DirectMap4k
   7806682 ±  2%     -37.4%    4885485 ± 25%  meminfo.Inactive
   7805273 ±  2%     -37.4%    4884076 ± 25%  meminfo.Inactive(anon)
   3900533 ±  2%     -39.1%    2377370 ± 27%  meminfo.Mapped
  12167854           -16.2%   10198035 ± 12%  meminfo.Memused
     25096 ±  2%     -11.5%      22220 ±  5%  meminfo.PageTables
   7642756 ±  2%     -38.1%    4733449 ± 26%  meminfo.Shmem
   2719315           +36.5%    3712526        meminfo.Unevictable
     90730            -5.4%      85831        proc-vmstat.nr_anon_pages
   2590987 ±  2%     -18.5%    2111929 ± 14%  proc-vmstat.nr_file_pages
   1951253 ±  2%     -37.4%    1221001 ± 25%  proc-vmstat.nr_inactive_anon
    975082 ±  2%     -39.0%     594342 ± 27%  proc-vmstat.nr_mapped
      6273 ±  2%     -11.5%       5554 ±  5%  proc-vmstat.nr_page_table_pages
   1910667 ±  2%     -38.1%    1183305 ± 26%  proc-vmstat.nr_shmem
     31783            -3.3%      30743 ±  2%  proc-vmstat.nr_slab_reclaimable
    679828           +36.5%     928131        proc-vmstat.nr_unevictable
   1951253 ±  2%     -37.4%    1221001 ± 25%  proc-vmstat.nr_zone_inactive_anon
    679828           +36.5%     928131        proc-vmstat.nr_zone_unevictable
    627539 ± 10%     +54.4%     968981 ± 39%  proc-vmstat.numa_hint_faults
    449238 ± 12%     +74.9%     785532 ± 49%  proc-vmstat.numa_hint_faults_local
 1.505e+08           +23.8%  1.863e+08 ±  6%  proc-vmstat.numa_hit
      4282 ±  3%     -53.2%       2003 ± 52%  proc-vmstat.numa_huge_pte_updates
 1.505e+08           +23.7%  1.862e+08 ±  6%  proc-vmstat.numa_local
   2952837 ±  4%     -28.1%    2122117 ± 13%  proc-vmstat.numa_pte_updates
 1.505e+08           +23.7%  1.862e+08 ±  6%  proc-vmstat.pgalloc_normal
   2727576 ±  2%     +11.5%    3040426 ± 12%  proc-vmstat.pgfault
 1.505e+08           +23.7%  1.862e+08 ±  6%  proc-vmstat.pgfree
  10982195           +41.3%   15515107 ±  7%  sched_debug.cfs_rq:/.MIN_vruntime.max
      1.00 ±  2%     +11.8%       1.12 ±  5%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.88 ±  3%     -14.7%       0.75 ±  6%  sched_debug.cfs_rq:/.h_nr_running.stddev
     56664 ±  6%     -23.8%      43171 ± 10%  sched_debug.cfs_rq:/.load.max
     12752 ±  3%     -29.3%       9016 ± 16%  sched_debug.cfs_rq:/.load.stddev
      5.00 ±  3%     -25.1%       3.74 ± 14%  sched_debug.cfs_rq:/.load_avg.min
  10982195           +41.3%   15515107 ±  7%  sched_debug.cfs_rq:/.max_vruntime.max
  10265230           +62.3%   16659900 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
  12960623 ±  2%     +49.2%   19341384 ±  9%  sched_debug.cfs_rq:/.min_vruntime.max
   9962156           +47.3%   14674448 ±  7%  sched_debug.cfs_rq:/.min_vruntime.min
    396620 ±  6%    +384.1%    1919935 ± 15%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.67 ±  2%     +15.5%       0.77 ±  5%  sched_debug.cfs_rq:/.nr_running.avg
      0.44           -18.3%       0.36 ± 10%  sched_debug.cfs_rq:/.nr_running.stddev
    753.58           +23.9%     933.72 ±  8%  sched_debug.cfs_rq:/.runnable_avg.avg
      1237 ±  2%     +32.0%       1633 ±  6%  sched_debug.cfs_rq:/.runnable_avg.max
    499.14 ±  2%     -13.9%     429.87 ±  5%  sched_debug.cfs_rq:/.runnable_avg.min
    154.08           +67.7%     258.43 ± 15%  sched_debug.cfs_rq:/.runnable_avg.stddev
    396613 ±  6%    +384.1%    1919923 ± 15%  sched_debug.cfs_rq:/.spread0.stddev
    467.16           +38.5%     647.14 ± 12%  sched_debug.cfs_rq:/.util_avg.avg
    912.23           +16.9%       1066 ±  7%  sched_debug.cfs_rq:/.util_avg.max
     69.44          +179.8%     194.29 ± 28%  sched_debug.cfs_rq:/.util_avg.stddev
     14.94 ±  2%   +1631.7%     258.71 ± 40%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     55.72 ±  3%    +364.2%     258.67 ± 40%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     70960           +46.2%     103773 ±  5%  sched_debug.cpu.avg_idle.avg
    175370 ± 25%    +162.2%     459798 ± 14%  sched_debug.cpu.avg_idle.max
     26563 ± 11%    +233.4%      88569 ± 14%  sched_debug.cpu.avg_idle.stddev
      3415           +18.2%       4038 ±  6%  sched_debug.cpu.curr->pid.avg
      2608           -15.4%       2206 ±  7%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  5%     +61.3%       0.00 ± 20%  sched_debug.cpu.next_balance.stddev
      0.95 ±  2%     +16.1%       1.10 ±  5%  sched_debug.cpu.nr_running.avg
      0.84           -11.9%       0.74 ±  5%  sched_debug.cpu.nr_running.stddev
  22978694           +11.4%   25589455 ±  3%  sched_debug.cpu.nr_switches.avg
  24010750 ±  2%     +16.0%   27851318 ±  5%  sched_debug.cpu.nr_switches.max
  19490849 ±  5%     +19.8%   23353131        sched_debug.cpu.nr_switches.min
     20.18           -30.9%      13.94 ± 16%  perf-stat.i.MPKI
 1.872e+10           +18.2%  2.213e+10 ±  5%  perf-stat.i.branch-instructions
      1.75            -0.1        1.69 ±  2%  perf-stat.i.branch-miss-rate%
 3.247e+08           +13.4%  3.683e+08 ±  4%  perf-stat.i.branch-misses
      3.10 ± 11%      +2.0        5.09 ± 14%  perf-stat.i.cache-miss-rate%
  57504764 ± 10%     +23.7%   71148870 ± 11%  perf-stat.i.cache-misses
  1.89e+09 ±  2%     -20.9%  1.494e+09 ± 10%  perf-stat.i.cache-references
   6156451            +9.4%    6734364 ±  3%  perf-stat.i.context-switches
      2.35           -11.7%       2.08 ±  4%  perf-stat.i.cpi
  2.21e+11            +3.6%  2.289e+11        perf-stat.i.cpu-cycles
   2487332 ±  3%     -28.2%    1786056 ± 15%  perf-stat.i.cpu-migrations
      0.13 ±  3%      -0.0        0.10 ± 17%  perf-stat.i.dTLB-load-miss-rate%
 2.827e+10           +19.6%  3.382e+10 ±  5%  perf-stat.i.dTLB-loads
      0.06            -0.0        0.04 ± 20%  perf-stat.i.dTLB-store-miss-rate%
   8801520 ±  3%     -23.1%    6766285 ± 13%  perf-stat.i.dTLB-store-misses
 1.521e+10           +18.6%  1.803e+10 ±  5%  perf-stat.i.dTLB-stores
     55.17            +3.5       58.69 ±  2%  perf-stat.i.iTLB-load-miss-rate%
 1.201e+08 ±  3%     +12.7%  1.354e+08 ±  4%  perf-stat.i.iTLB-load-misses
  9.46e+10           +18.4%   1.12e+11 ±  5%  perf-stat.i.instructions
    792.14            +4.3%     826.31 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.43           +14.1%       0.49 ±  4%  perf-stat.i.ipc
      0.86 ±  4%     +47.8%       1.27 ± 20%  perf-stat.i.major-faults
      2.30            +3.6%       2.38        perf-stat.i.metric.GHz
    668.48           +17.7%     786.62 ±  5%  perf-stat.i.metric.M/sec
      3502 ±  2%     +12.4%       3935 ± 13%  perf-stat.i.minor-faults
     78.57            +8.8       87.35 ±  2%  perf-stat.i.node-load-miss-rate%
  16081011 ± 12%     +30.0%   20898766 ± 11%  perf-stat.i.node-load-misses
   5543925 ±  2%     -40.0%    3326330 ± 10%  perf-stat.i.node-loads
     83.01 ±  3%      +8.8       91.80 ±  2%  perf-stat.i.node-store-miss-rate%
   5420293 ±  8%     +25.2%    6788239 ± 10%  perf-stat.i.node-store-misses
   1212242 ± 11%     -48.9%     619862 ± 27%  perf-stat.i.node-stores
      3503 ±  2%     +12.4%       3937 ± 13%  perf-stat.i.page-faults
     19.97           -32.8%      13.43 ± 14%  perf-stat.overall.MPKI
      1.73            -0.1        1.67 ±  2%  perf-stat.overall.branch-miss-rate%
      3.05 ± 11%      +1.8        4.84 ± 18%  perf-stat.overall.cache-miss-rate%
      2.34           -12.3%       2.05 ±  4%  perf-stat.overall.cpi
      0.13 ±  3%      -0.0        0.10 ± 16%  perf-stat.overall.dTLB-load-miss-rate%
      0.06            -0.0        0.04 ± 17%  perf-stat.overall.dTLB-store-miss-rate%
     55.17            +3.6       58.72 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
    787.99            +4.9%     826.86        perf-stat.overall.instructions-per-iTLB-miss
      0.43           +14.3%       0.49 ±  4%  perf-stat.overall.ipc
     74.13 ±  3%     +12.0       86.08 ±  2%  perf-stat.overall.node-load-miss-rate%
     81.61 ±  3%      +9.9       91.52 ±  2%  perf-stat.overall.node-store-miss-rate%
 1.869e+10           +18.2%   2.21e+10 ±  5%  perf-stat.ps.branch-instructions
 3.243e+08           +13.4%  3.678e+08 ±  4%  perf-stat.ps.branch-misses
  57432021 ± 10%     +23.8%   71077827 ± 11%  perf-stat.ps.cache-misses
 1.887e+09 ±  2%     -20.9%  1.492e+09 ± 10%  perf-stat.ps.cache-references
   6147733            +9.4%    6725226 ±  3%  perf-stat.ps.context-switches
 2.207e+11            +3.6%  2.286e+11        perf-stat.ps.cpu-cycles
   2483875 ±  2%     -28.2%    1783069 ± 15%  perf-stat.ps.cpu-migrations
 2.824e+10           +19.6%  3.377e+10 ±  5%  perf-stat.ps.dTLB-loads
   8789276 ±  3%     -23.1%    6755194 ± 13%  perf-stat.ps.dTLB-store-misses
 1.519e+10           +18.6%  1.801e+10 ±  5%  perf-stat.ps.dTLB-stores
   1.2e+08 ±  3%     +12.7%  1.352e+08 ±  4%  perf-stat.ps.iTLB-load-misses
 9.447e+10           +18.4%  1.119e+11 ±  5%  perf-stat.ps.instructions
      0.86 ±  4%     +47.9%       1.27 ± 20%  perf-stat.ps.major-faults
      3503 ±  2%     +12.4%       3937 ± 13%  perf-stat.ps.minor-faults
  16057997 ± 12%     +30.0%   20872834 ± 11%  perf-stat.ps.node-load-misses
   5542818 ±  2%     -39.8%    3334248 ± 10%  perf-stat.ps.node-loads
   5412847 ±  8%     +25.3%    6779940 ± 10%  perf-stat.ps.node-store-misses
   1210963 ± 11%     -48.9%     618995 ± 27%  perf-stat.ps.node-stores
      3503 ±  2%     +12.4%       3938 ± 13%  perf-stat.ps.page-faults
 6.822e+13           +18.4%  8.077e+13 ±  5%  perf-stat.total.instructions
     24.89           -13.1       11.78 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     24.65           -13.1       11.58 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     24.64           -13.1       11.57 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     24.62           -13.1       11.56 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     16.56            -8.5        8.06 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     15.49            -8.0        7.53 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     15.47            -8.0        7.52 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     15.06            -7.8        7.29 ±  4%  perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     14.73            -7.6        7.14 ±  4%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      9.84            -4.3        5.52 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.do_packets
      5.15            -3.6        1.56 ± 17%  perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
     19.38            -3.3       16.08        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.server
     19.45            -3.1       16.32 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.server.listener
     19.50            -3.1       16.38 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.server.listener.main
     19.62            -3.1       16.57 ±  2%  perf-profile.calltrace.cycles-pp.__send.server.listener.main.__libc_start_main
      3.59 ±  5%      -2.4        1.19 ±  5%  perf-profile.calltrace.cycles-pp.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      3.69 ±  2%      -2.3        1.34 ±  4%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.31 ±  7%      -2.1        1.18 ±  5%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up.__wake_up_common
      3.26 ±  7%      -2.1        1.16 ±  5%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up
      3.16 ±  6%      -2.0        1.15 ±  5%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu
      3.86            -2.0        1.89 ±  3%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.80            -1.9        1.86 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
     18.24            -1.9       16.33        perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      2.84            -1.8        1.07 ±  5%  perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task
     17.28            -1.7       15.57        perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      2.65 ±  3%      -1.7        0.95 ±  4%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
     16.94            -1.6       15.32        perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
     16.80            -1.6       15.23        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
     16.67            -1.5       15.14        perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
     21.29            -1.5       19.76        perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      7.69            -1.5        6.16 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      8.46            -1.5        6.96 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken
      2.24 ±  4%      -1.5        0.76 ±  4%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry
      2.20 ±  4%      -1.5        0.74 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle
     21.74            -1.3       20.43        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     21.80            -1.3       20.50        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.93 ±  4%      -1.3        0.64 ±  5%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue
      3.31            -1.3        2.04 ±  3%  perf-profile.calltrace.cycles-pp.recv.do_packets.sio_readx.child_op.child_run
      3.21            -1.3        1.96 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_readx.child_op
      3.19            -1.3        1.94 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_readx
      2.26            -1.2        1.12 ±  3%  perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
      2.15            -1.1        1.06 ±  3%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.do_idle
      2.08            -1.1        1.02 ±  3%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle
      2.10            -0.8        1.30 ±  4%  perf-profile.calltrace.cycles-pp.recv.do_packets.sio_createx.child_op.child_run
      2.03            -0.8        1.24 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_createx.child_op
      2.02            -0.8        1.24 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_createx
      1.91            -0.7        1.18 ±  4%  perf-profile.calltrace.cycles-pp.recv.do_packets.sio_qpathinfo.child_op.child_run
      1.85            -0.7        1.13 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_qpathinfo.child_op
      1.84            -0.7        1.12 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_qpathinfo
      2.89            -0.7        2.19 ±  2%  perf-profile.calltrace.cycles-pp.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
      0.98 ±  2%      -0.6        0.35 ± 70%  perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.__wake_up_common
      1.54 ±  2%      -0.6        0.96 ±  4%  perf-profile.calltrace.cycles-pp.recv.do_packets.sio_close.child_op.child_run
      1.24            -0.6        0.66 ±  3%  perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      1.49 ±  2%      -0.6        0.92 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_close.child_op
      2.47            -0.6        1.90 ±  2%  perf-profile.calltrace.cycles-pp.unwind_next_frame.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain
      1.48 ±  2%      -0.6        0.91 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_close
      1.19            -0.5        0.64 ±  4%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      1.06            -0.4        0.66 ±  4%  perf-profile.calltrace.cycles-pp.recv.do_packets.sio_writex.child_op.child_run
      1.02            -0.4        0.63 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_writex
      1.02            -0.4        0.64 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.do_packets.sio_writex.child_op
      0.70 ±  2%      -0.1        0.65 ±  2%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
      0.58            -0.0        0.54 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter.skb_do_copy_data_nocache.tcp_sendmsg_locked
      0.75            +0.1        0.81 ±  2%  perf-profile.calltrace.cycles-pp.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      1.23            +0.1        1.35 ±  3%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      1.24            +0.1        1.38 ±  3%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      0.52            +0.2        0.71 ±  5%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      0.54            +0.2        0.74 ±  4%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      0.57            +0.2        0.78 ±  3%  perf-profile.calltrace.cycles-pp.recv
      0.51            +0.2        0.73 ±  2%  perf-profile.calltrace.cycles-pp.vsnprintf.snprintf.child_run.create_procs.main
      0.52            +0.2        0.74 ±  2%  perf-profile.calltrace.cycles-pp.snprintf.child_run.create_procs.main.__libc_start_main
      0.91            +0.2        1.14 ±  3%  perf-profile.calltrace.cycles-pp.do_packets.sio_qfileinfo.child_op.child_run.create_procs
      0.94            +0.2        1.18 ±  3%  perf-profile.calltrace.cycles-pp.sio_qfileinfo.child_op.child_run.create_procs.main
      0.95            +0.2        1.20 ±  3%  perf-profile.calltrace.cycles-pp.do_packets.sio_qfsinfo.child_op.child_run.create_procs
      0.67            +0.3        0.92 ±  3%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
      0.98            +0.3        1.24 ±  2%  perf-profile.calltrace.cycles-pp.sio_qfsinfo.child_op.child_run.create_procs.main
      0.58            +0.3        0.86 ±  3%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      1.16            +0.3        1.46 ±  3%  perf-profile.calltrace.cycles-pp.do_packets.sio_unlink.child_op.child_run.create_procs
      1.20            +0.3        1.51 ±  2%  perf-profile.calltrace.cycles-pp.sio_unlink.child_op.child_run.create_procs.main
      0.34 ± 70%      +0.3        0.69 ±  5%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
      0.56            +0.4        0.91 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_qfileinfo.child_op
      0.55            +0.4        0.91 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_qfileinfo
      0.56            +0.4        0.92 ±  4%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_qfileinfo.child_op.child_run
      0.58            +0.4        0.95 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_qfsinfo
      0.59            +0.4        0.96 ±  4%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_qfsinfo.child_op.child_run
      0.58            +0.4        0.96 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_qfsinfo.child_op
      0.97            +0.4        1.36 ±  3%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      4.25            +0.4        4.68 ±  3%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
      5.11            +0.4        5.54 ±  4%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
      0.71            +0.4        1.16 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_unlink
      0.72            +0.4        1.17 ±  4%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_unlink.child_op.child_run
      0.71            +0.5        1.16 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_unlink.child_op
      4.47            +0.5        4.93 ±  2%  perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch
      4.07 ±  5%      +0.5        4.54 ±  3%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
      2.04            +0.5        2.52 ±  2%  perf-profile.calltrace.cycles-pp.do_packets.sio_findfirst.child_op.child_run.create_procs
      0.17 ±141%      +0.5        0.66 ±  2%  perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      2.11            +0.5        2.61 ±  2%  perf-profile.calltrace.cycles-pp.sio_findfirst.child_op.child_run.create_procs.main
      0.17 ±141%      +0.5        0.67 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
      0.08 ±223%      +0.5        0.61 ±  2%  perf-profile.calltrace.cycles-pp.sio_sfileinfo.child_op.child_run.create_procs.main
      0.00            +0.6        0.58 ±  2%  perf-profile.calltrace.cycles-pp.do_packets.sio_sfileinfo.child_op.child_run.create_procs
      0.08 ±223%      +0.6        0.70        perf-profile.calltrace.cycles-pp.dopr.vsnprintf.snprintf.child_run.create_procs
      0.00            +0.7        0.65 ±  2%  perf-profile.calltrace.cycles-pp.next_token.child_run.create_procs.main.__libc_start_main
      3.27            +0.7        3.98 ±  2%  perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.schedule_timeout.wait_woken
      3.15            +0.7        3.87 ±  2%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.schedule_timeout
      1.25            +0.8        2.00 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_findfirst
      0.00            +0.8        0.76 ±  8%  perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.kill_pid_info.kill_something_info
      1.25            +0.8        2.01 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_findfirst.child_op
      0.00            +0.8        0.76 ± 14%  perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      1.26            +0.8        2.02 ±  3%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_findfirst.child_op.child_run
      4.22            +0.8        5.04 ±  4%  perf-profile.calltrace.cycles-pp.unwind_next_frame.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
      0.00            +0.8        0.82 ± 13%  perf-profile.calltrace.cycles-pp.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      0.00            +0.8        0.82 ± 13%  perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.91 ± 34%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.exit_to_user_mode_loop
      5.86            +1.0        6.85 ±  9%  perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template
      0.00            +1.0        1.01 ± 45%  perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      4.20            +1.0        5.24 ±  3%  perf-profile.calltrace.cycles-pp.do_packets.sio_close.child_op.child_run.create_procs
      4.34            +1.1        5.42 ±  2%  perf-profile.calltrace.cycles-pp.sio_close.child_op.child_run.create_procs.main
     33.90            +1.2       35.11 ±  2%  perf-profile.calltrace.cycles-pp.server.listener.main.__libc_start_main
     33.95            +1.2       35.18 ±  2%  perf-profile.calltrace.cycles-pp.listener.main.__libc_start_main
      3.50            +1.3        4.77 ±  3%  perf-profile.calltrace.cycles-pp.do_packets.sio_writex.child_op.child_run.create_procs
      5.26            +1.3        6.56 ±  2%  perf-profile.calltrace.cycles-pp.do_packets.sio_qpathinfo.child_op.child_run.create_procs
      3.62            +1.3        4.92 ±  3%  perf-profile.calltrace.cycles-pp.sio_writex.child_op.child_run.create_procs.main
      5.43            +1.4        6.79 ±  2%  perf-profile.calltrace.cycles-pp.sio_qpathinfo.child_op.child_run.create_procs.main
      5.80            +1.4        7.22 ±  2%  perf-profile.calltrace.cycles-pp.do_packets.sio_createx.child_op.child_run.create_procs
      3.07            +1.5        4.55 ±  6%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
      5.99            +1.5        7.47 ±  2%  perf-profile.calltrace.cycles-pp.sio_createx.child_op.child_run.create_procs.main
      0.61 ±  2%      +1.5        2.10 ±  6%  perf-profile.calltrace.cycles-pp.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
      0.75            +1.5        2.28 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill.create_procs
      2.57            +1.6        4.17 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_close
      2.57            +1.6        4.18 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_close.child_op
      2.59            +1.6        4.21 ±  3%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_close.child_op.child_run
      0.82            +1.6        2.46 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill.create_procs.main
      0.84            +1.7        2.50 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill.create_procs.main.__libc_start_main
      2.38            +1.7        4.03 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_writex
      2.38            +1.7        4.04 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_writex.child_op
      2.39            +1.7        4.06 ±  4%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_writex.child_op.child_run
      0.26 ±100%      +1.7        1.94 ±  6%  perf-profile.calltrace.cycles-pp.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.96            +1.7        2.68 ±  5%  perf-profile.calltrace.cycles-pp.kill.create_procs.main.__libc_start_main
      0.00            +1.8        1.80 ±  6%  perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.kill_pid_info.kill_something_info.__x64_sys_kill
      0.00            +1.8        1.82 ±  6%  perf-profile.calltrace.cycles-pp.security_task_kill.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64
      3.23            +2.0        5.25 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_qpathinfo
      3.24            +2.0        5.26 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_qpathinfo.child_op
      3.26            +2.0        5.29 ±  3%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_qpathinfo.child_op.child_run
      9.03            +2.2       11.21 ±  2%  perf-profile.calltrace.cycles-pp.do_packets.sio_readx.child_op.child_run.create_procs
      0.00            +2.2        2.19 ± 21%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      3.57            +2.2        5.77 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_createx
      3.58            +2.2        5.78 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_createx.child_op
      3.60            +2.2        5.82 ±  3%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_createx.child_op.child_run
      9.38            +2.3       11.64 ±  2%  perf-profile.calltrace.cycles-pp.sio_readx.child_op.child_run.create_procs.main
      0.00            +2.4        2.39 ± 16%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +2.4        2.42 ± 16%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.58            +2.5       15.12 ±  3%  perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
      0.00            +2.6        2.55 ± 16%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.00            +2.9        2.89 ±  9%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets
     26.14            +3.1       29.26 ±  2%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      0.52 ±  2%      +3.2        3.69 ±  7%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.enqueue_entity
     26.00            +3.2       29.20 ±  2%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
      0.53 ±  2%      +3.2        3.76 ±  7%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair
     28.21            +3.2       31.44 ±  2%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
      0.54            +3.3        3.81 ±  7%  perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate
     25.78            +3.3       29.06 ±  2%  perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
     24.12            +3.3       27.43 ±  3%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     28.57            +3.4       31.94 ±  2%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
     24.20            +3.4       27.58 ±  3%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      0.59            +3.4        4.00 ± 10%  perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
     28.69            +3.4       32.10 ±  2%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
      5.50            +3.4        8.93 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_readx
      5.51            +3.4        8.95 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_readx.child_op
     28.74            +3.4       32.19 ±  2%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
      5.54            +3.5        9.00 ±  3%  perf-profile.calltrace.cycles-pp.__send.do_packets.sio_readx.child_op.child_run
     30.27            +3.5       33.74 ±  2%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      0.72            +3.6        4.29 ± 10%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common
     32.66            +3.6       36.23 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
     26.74            +3.6       30.32 ±  2%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
     26.98            +3.6       30.62 ±  2%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
     27.02            +3.6       30.67 ±  2%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
     31.17            +3.7       34.88 ±  2%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      4.48 ±  6%      +3.7        8.20 ±  4%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
     15.58            +3.8       19.42 ±  5%  perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
     15.64            +3.9       19.49 ±  5%  perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event
     33.80            +3.9       37.72 ±  2%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
     33.84            +4.0       37.82 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     12.68            +4.1       16.75 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.server
     13.21            +4.2       17.46 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.server.listener
     13.29            +4.3       17.57 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.server.listener.main
     13.75            +4.3       18.04 ±  2%  perf-profile.calltrace.cycles-pp.recv.server.listener.main.__libc_start_main
      3.25            +4.3        7.60 ±  4%  perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime
      0.87            +4.4        5.29 ±  7%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      0.87            +4.4        5.32 ±  7%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
     38.72            +4.7       43.44 ±  2%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     39.07            +5.0       44.08 ±  2%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     39.38            +5.6       44.97 ±  2%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     39.57            +5.7       45.27 ±  2%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     20.26            +9.0       29.28 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets
     35.82            +9.2       44.97 ±  2%  perf-profile.calltrace.cycles-pp.child_op.child_run.create_procs.main.__libc_start_main
     37.59            +9.8       47.36 ±  2%  perf-profile.calltrace.cycles-pp.child_run.create_procs.main.__libc_start_main
     38.72           +11.5       50.25        perf-profile.calltrace.cycles-pp.create_procs.main.__libc_start_main
     72.68           +12.8       85.43        perf-profile.calltrace.cycles-pp.main.__libc_start_main
     72.71           +12.8       85.47        perf-profile.calltrace.cycles-pp.__libc_start_main
     24.89           -13.1       11.78 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     24.89           -13.1       11.78 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
     24.86           -13.1       11.77 ±  4%  perf-profile.children.cycles-pp.do_idle
     24.65           -13.1       11.58 ±  3%  perf-profile.children.cycles-pp.start_secondary
     16.73            -8.5        8.21 ±  4%  perf-profile.children.cycles-pp.cpuidle_idle_call
     15.64            -8.0        7.66 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter
     15.62            -8.0        7.66 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter_state
     15.20            -7.8        7.42 ±  4%  perf-profile.children.cycles-pp.intel_idle_irq
     15.15            -7.7        7.40 ±  4%  perf-profile.children.cycles-pp.mwait_idle_with_hints
      6.70            -3.5        3.24 ±  5%  perf-profile.children.cycles-pp.set_task_cpu
      5.37            -2.6        2.80 ±  5%  perf-profile.children.cycles-pp.perf_trace_sched_migrate_task
      3.79 ±  2%      -2.4        1.39 ±  4%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      3.57 ±  3%      -2.3        1.30 ±  3%  perf-profile.children.cycles-pp.sched_ttwu_pending
      3.90            -2.0        1.92 ±  4%  perf-profile.children.cycles-pp.schedule_idle
     19.65            -1.6       18.00        perf-profile.children.cycles-pp.sk_wait_data
     23.24            -1.6       21.63        perf-profile.children.cycles-pp.tcp_recvmsg_locked
     18.61            -1.5       17.16        perf-profile.children.cycles-pp.wait_woken
      3.26 ±  3%      -1.4        1.83 ±  3%  perf-profile.children.cycles-pp.update_load_avg
     23.79            -1.4       22.40        perf-profile.children.cycles-pp.tcp_recvmsg
      2.50 ±  7%      -1.4        1.11 ±  9%  perf-profile.children.cycles-pp.update_cfs_group
     23.86            -1.4       22.47        perf-profile.children.cycles-pp.inet_recvmsg
     18.25            -1.4       16.89        perf-profile.children.cycles-pp.schedule_timeout
      8.98            -1.1        7.85        perf-profile.children.cycles-pp.dequeue_entity
      9.85            -1.1        8.75        perf-profile.children.cycles-pp.dequeue_task_fair
      2.68 ±  2%      -1.0        1.64 ±  3%  perf-profile.children.cycles-pp.select_task_rq
      2.58 ±  2%      -1.0        1.55 ±  3%  perf-profile.children.cycles-pp.select_task_rq_fair
      2.15 ±  2%      -0.9        1.22 ±  3%  perf-profile.children.cycles-pp.select_idle_sibling
      2.01            -0.9        1.15 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
      1.79 ±  2%      -0.8        0.98 ±  4%  perf-profile.children.cycles-pp.select_idle_cpu
      1.52 ±  2%      -0.7        0.79 ±  5%  perf-profile.children.cycles-pp.select_idle_core
      1.16 ±  3%      -0.7        0.44 ±  3%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      1.05 ±  3%      -0.7        0.39 ±  3%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.98 ±  3%      -0.6        0.37 ±  3%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      1.62            -0.6        1.03 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.16            -0.6        0.60 ±  5%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.13            -0.5        0.60 ±  5%  perf-profile.children.cycles-pp.available_idle_cpu
      0.81 ±  2%      -0.4        0.42 ±  5%  perf-profile.children.cycles-pp.menu_select
      0.93 ±  2%      -0.3        0.61        perf-profile.children.cycles-pp.finish_task_switch
      1.29 ±  2%      -0.3        1.02        perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.52            -0.2        0.28 ±  5%  perf-profile.children.cycles-pp.migrate_task_rq_fair
      0.52            -0.2        0.28 ±  5%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.40            -0.2        0.20 ±  6%  perf-profile.children.cycles-pp.napi_consume_skb
      0.36            -0.2        0.18 ±  7%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.33 ±  2%      -0.2        0.16 ±  5%  perf-profile.children.cycles-pp.tcp_data_queue
      1.61            -0.2        1.44 ±  6%  perf-profile.children.cycles-pp.perf_output_sample
      0.36 ±  3%      -0.2        0.20 ±  8%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.54 ±  5%      -0.2        0.38 ±  4%  perf-profile.children.cycles-pp.read_sock
      0.31            -0.2        0.16 ±  7%  perf-profile.children.cycles-pp.__smp_call_single_queue
      0.30            -0.2        0.15 ±  7%  perf-profile.children.cycles-pp.llist_add_batch
      0.22 ±  6%      -0.1        0.08 ±  9%  perf-profile.children.cycles-pp.bpf_ksym_find
      0.34 ±  2%      -0.1        0.20 ±  7%  perf-profile.children.cycles-pp.perf_output_end
      0.95 ±  2%      -0.1        0.82        perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.30            -0.1        0.16 ±  4%  perf-profile.children.cycles-pp.wake_affine
      0.24            -0.1        0.12 ±  5%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.24 ±  5%      -0.1        0.13 ±  3%  perf-profile.children.cycles-pp._find_next_bit
      1.01            -0.1        0.90 ±  5%  perf-profile.children.cycles-pp.memcpy_erms
      0.27 ±  6%      -0.1        0.16 ±  3%  perf-profile.children.cycles-pp.is_bpf_text_address
      0.39            -0.1        0.29 ±  3%  perf-profile.children.cycles-pp.get_stack_info
      0.20 ±  5%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.cpumask_next
      0.28 ±  2%      -0.1        0.19 ±  3%  perf-profile.children.cycles-pp.get_stack_info_noinstr
      0.19 ±  3%      -0.1        0.10 ±  6%  perf-profile.children.cycles-pp.remove_entity_load_avg
      0.20 ±  3%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.88            -0.1        0.80 ±  3%  perf-profile.children.cycles-pp.copyout
      0.17 ±  2%      -0.1        0.09 ± 12%  perf-profile.children.cycles-pp.llist_reverse_order
      0.84            -0.1        0.76 ±  2%  perf-profile.children.cycles-pp.switch_fpu_return
      0.28 ±  2%      -0.1        0.20 ±  5%  perf-profile.children.cycles-pp.remove_wait_queue
      0.60            -0.1        0.53 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.14 ±  5%      -0.1        0.07 ±  8%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.14 ±  3%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.task_h_load
      0.23 ±  3%      -0.1        0.16 ±  2%  perf-profile.children.cycles-pp.kfree_skbmem
      0.13 ±  2%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.cpumask_next_wrap
      0.51 ±  2%      -0.1        0.44 ±  2%  perf-profile.children.cycles-pp.update_process_times
      0.74 ±  2%      -0.1        0.68 ±  2%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.15 ±  3%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.newidle_balance
      0.51            -0.1        0.45        perf-profile.children.cycles-pp.tick_sched_handle
      0.45            -0.1        0.39        perf-profile.children.cycles-pp.scheduler_tick
      0.09 ±  4%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.13 ±  3%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.12 ±  6%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.rebalance_domains
      0.61 ±  2%      -0.1        0.55        perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.75 ±  3%      -0.1        0.70 ±  2%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.54            -0.1        0.48        perf-profile.children.cycles-pp.tick_sched_timer
      0.32 ±  2%      -0.1        0.27 ±  3%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.22            -0.1        0.17 ±  2%  perf-profile.children.cycles-pp.sock_rfree
      0.11            -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__update_idle_core
      0.16 ±  5%      -0.1        0.11 ±  5%  perf-profile.children.cycles-pp.poll_idle
      0.39 ±  2%      -0.0        0.34 ±  2%  perf-profile.children.cycles-pp.task_tick_fair
      0.11 ±  8%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.load_balance
      0.10 ±  4%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__slab_free
      0.44 ±  2%      -0.0        0.40 ±  3%  perf-profile.children.cycles-pp.__switch_to
      0.28 ±  3%      -0.0        0.24 ±  3%  perf-profile.children.cycles-pp.skb_page_frag_refill
      0.09 ±  5%      -0.0        0.05 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.53            -0.0        0.50 ±  3%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.57 ±  2%      -0.0        0.54        perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.09 ± 12%      -0.0        0.06 ± 46%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.23 ±  2%      -0.0        0.20 ±  3%  perf-profile.children.cycles-pp.tcp_queue_rcv
      0.22 ±  3%      -0.0        0.19        perf-profile.children.cycles-pp.update_rq_clock_task
      0.06 ±  7%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.rb_first
      0.25 ±  3%      -0.0        0.22 ±  3%  perf-profile.children.cycles-pp.skb_attempt_defer_free
      0.08 ±  7%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.__build_skb_around
      0.29 ±  2%      -0.0        0.27 ±  3%  perf-profile.children.cycles-pp.sk_page_frag_refill
      0.17 ±  4%      -0.0        0.15 ±  3%  perf-profile.children.cycles-pp.cpuacct_charge
      0.08 ±  6%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.cubictcp_cwnd_event
      0.07 ±  6%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.ntohl@plt
      0.05            +0.0        0.06        perf-profile.children.cycles-pp.audit_signal_info_syscall
      0.05            +0.0        0.06 ±  6%  perf-profile.children.cycles-pp.perf_trace_run_bpf_submit
      0.07            +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.06 ±  7%      +0.0        0.08        perf-profile.children.cycles-pp.tcp_rate_check_app_limited
      0.05 ±  8%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.clear_buddies
      0.05 ±  7%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.__list_add_valid
      0.12 ±  4%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.06            +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
      0.07            +0.0        0.09        perf-profile.children.cycles-pp.skb_release_head_state
      0.25 ±  3%      +0.0        0.27 ±  4%  perf-profile.children.cycles-pp.__mod_timer
      0.24 ±  3%      +0.0        0.27 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.05 ±  7%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.strchr@plt
      0.52            +0.0        0.55 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.07 ±  6%      +0.0        0.10 ±  6%  perf-profile.children.cycles-pp.is_ftrace_trampoline
      0.06 ±  7%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.cubictcp_acked
      0.06 ±  7%      +0.0        0.09 ±  6%  perf-profile.children.cycles-pp.rb_erase
      0.05 ±  7%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.rcu_all_qs
      0.04 ± 45%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.__usecs_to_jiffies
      0.07 ±  5%      +0.0        0.10 ±  6%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.12 ±  4%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.__ip_local_out
      0.07 ±  5%      +0.0        0.10 ±  6%  perf-profile.children.cycles-pp.check_kill_permission
      0.10 ±  5%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.nb_target_rate
      0.08 ±  8%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.import_single_range
      0.15 ±  7%      +0.0        0.18 ±  5%  perf-profile.children.cycles-pp.perf_trace_buf_alloc
      0.09 ±  4%      +0.0        0.12 ±  5%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.06 ±  6%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.sock_put
      1.22            +0.0        1.26        perf-profile.children.cycles-pp._copy_from_iter
      0.08 ±  4%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.tcp_event_data_recv
      0.09 ±  6%      +0.0        0.12 ±  6%  perf-profile.children.cycles-pp.__is_insn_slot_addr
      0.08 ±  6%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.09 ±  6%      +0.0        0.13 ±  5%  perf-profile.children.cycles-pp.perf_instruction_pointer
      0.36            +0.0        0.40 ±  2%  perf-profile.children.cycles-pp.ftrace_graph_ret_addr
      0.15 ±  2%      +0.0        0.19 ±  5%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.13 ±  2%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.02 ±141%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.skb_clone
      0.34 ±  3%      +0.0        0.38 ±  2%  perf-profile.children.cycles-pp.__task_pid_nr_ns
      0.02 ±141%      +0.0        0.06 ±  8%  perf-profile.children.cycles-pp.__xfrm_policy_check2
      0.04 ± 44%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.inet_ehashfn
      0.09 ±  8%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.44            +0.0        0.48 ±  2%  perf-profile.children.cycles-pp.native_sched_clock
      0.12 ±  4%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.ip_local_out
      0.38 ±  2%      +0.0        0.43 ±  3%  perf-profile.children.cycles-pp.set_next_entity
      0.19 ±  5%      +0.0        0.23 ±  5%  perf-profile.children.cycles-pp.kvm_guest_state
      0.04 ± 44%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.rb_insert_color
      0.06 ± 13%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.netif_skb_features
      0.01 ±223%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.kmalloc_slab
      0.97            +0.0        1.02 ±  2%  perf-profile.children.cycles-pp._copy_to_iter
      0.40 ±  2%      +0.0        0.45 ±  2%  perf-profile.children.cycles-pp.__fdget
      0.08 ±  7%      +0.1        0.13 ±  4%  perf-profile.children.cycles-pp.pick_next_entity
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.tcp_data_ready
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.schedule_debug
      0.26 ±  2%      +0.1        0.31 ±  2%  perf-profile.children.cycles-pp.__skb_clone
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb
      0.08 ±  5%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.nf_hook_slow
      0.09 ±  5%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.ip_rcv_core
      0.05 ±  8%      +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.__copy_skb_header
      0.11 ±  8%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.write_sock
      0.13            +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.perf_misc_flags
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.tcp_chrono_start
      0.01 ±223%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.09 ±  8%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.ip_skb_dst_mtu
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.tcp_established_options
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.save_fpregs_to_fpstate
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.eth_type_trans
      0.09 ±  5%      +0.1        0.15 ±  4%  perf-profile.children.cycles-pp.all_string_sub
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.is_module_text_address
      0.25 ±  8%      +0.1        0.31 ±  6%  perf-profile.children.cycles-pp.ipv4_mtu
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.apparmor_ip_postroute
      0.19 ±  7%      +0.1        0.25 ± 10%  perf-profile.children.cycles-pp.dst_release
      0.26            +0.1        0.32 ±  3%  perf-profile.children.cycles-pp.sio_rename
      0.18 ±  5%      +0.1        0.25        perf-profile.children.cycles-pp.os_xsave
      0.38            +0.1        0.45 ±  2%  perf-profile.children.cycles-pp.perf_output_begin_forward
      0.44 ±  2%      +0.1        0.52 ±  3%  perf-profile.children.cycles-pp.check_heap_object
      0.42 ±  4%      +0.1        0.49 ±  2%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.__tcp_select_window
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.tcp_v4_fill_cb
      0.16 ±  5%      +0.1        0.24 ±  3%  perf-profile.children.cycles-pp.tcp_skb_entail
      0.64            +0.1        0.72 ±  3%  perf-profile.children.cycles-pp.__perf_event_header__init_id
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.cubictcp_cong_avoid
      0.07 ±  5%      +0.1        0.15 ±  5%  perf-profile.children.cycles-pp.tcp_tso_segs
      0.00            +0.1        0.08 ±  4%  perf-profile.children.cycles-pp.memcg_slab_free_hook
      0.46 ±  2%      +0.1        0.54 ±  3%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.08            +0.1        0.16 ±  2%  perf-profile.children.cycles-pp.tcp_wfree
      0.08 ±  6%      +0.1        0.16 ±  6%  perf-profile.children.cycles-pp.__calc_delta
      0.33 ±  2%      +0.1        0.41 ±  2%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.15 ±  3%      +0.1        0.24 ±  5%  perf-profile.children.cycles-pp.__might_sleep
      0.26            +0.1        0.34 ±  3%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.29            +0.1        0.38        perf-profile.children.cycles-pp.__might_resched
      0.18 ± 12%      +0.1        0.27 ± 10%  perf-profile.children.cycles-pp.__sk_dst_check
      0.18 ±  2%      +0.1        0.27 ±  3%  perf-profile.children.cycles-pp.dopr_outch
      0.13 ±  2%      +0.1        0.22 ±  4%  perf-profile.children.cycles-pp.kmem_cache_free
      0.20 ±  5%      +0.1        0.29 ±  3%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.00            +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.tcp_inbound_md5_hash
      0.30            +0.1        0.39 ±  3%  perf-profile.children.cycles-pp.__netif_rx
      0.28            +0.1        0.38 ±  3%  perf-profile.children.cycles-pp.netif_rx_internal
      0.12 ±  5%      +0.1        0.22 ±  8%  perf-profile.children.cycles-pp.__ip_finish_output
      0.20 ±  5%      +0.1        0.29 ±  2%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.19 ± 14%      +0.1        0.28 ± 10%  perf-profile.children.cycles-pp.ipv4_dst_check
      0.12 ±  3%      +0.1        0.22 ±  7%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.39 ±  2%      +0.1        0.50 ±  3%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.42            +0.1        0.52 ±  3%  perf-profile.children.cycles-pp.sio_flush
      0.10 ±  3%      +0.1        0.20 ±  6%  perf-profile.children.cycles-pp.check_preempt_curr
      0.19 ±  5%      +0.1        0.29 ±  3%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.19 ± 11%      +0.1        0.30 ± 11%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.09 ±  6%      +0.1        0.20 ±  8%  perf-profile.children.cycles-pp.tcp_rtt_estimator
      0.01 ±223%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.set_next_buddy
      0.00            +0.1        0.11 ±  5%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.20 ±  2%      +0.1        0.32 ±  6%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.13 ±  4%      +0.1        0.24 ±  5%  perf-profile.children.cycles-pp.kfree
      2.34            +0.1        2.45        perf-profile.children.cycles-pp.__kernel_text_address
      0.29 ±  2%      +0.1        0.40 ±  3%  perf-profile.children.cycles-pp.lock_sock_nested
      0.14 ±  3%      +0.1        0.26 ±  2%  perf-profile.children.cycles-pp.ip_output
      0.23 ±  2%      +0.1        0.35 ±  2%  perf-profile.children.cycles-pp.read_tsc
      0.48 ±  2%      +0.1        0.61 ±  2%  perf-profile.children.cycles-pp.sio_sfileinfo
      0.16 ±  5%      +0.1        0.29 ±  7%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.08 ±  4%      +0.1        0.21 ±  4%  perf-profile.children.cycles-pp.__might_fault
      1.39            +0.1        1.52        perf-profile.children.cycles-pp.skb_do_copy_data_nocache
      1.73            +0.1        1.87 ±  2%  perf-profile.children.cycles-pp.__skb_datagram_iter
      0.24 ±  2%      +0.1        0.38 ±  4%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.43 ±  2%      +0.1        0.58 ±  3%  perf-profile.children.cycles-pp.__kfree_skb
      0.00            +0.2        0.15 ±  6%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.34 ±  2%      +0.2        0.49 ±  3%  perf-profile.children.cycles-pp.fmtstr
      0.22 ±  3%      +0.2        0.38 ±  5%  perf-profile.children.cycles-pp.reweight_entity
      0.02 ± 99%      +0.2        0.18 ±  5%  perf-profile.children.cycles-pp.put_prev_entity
      1.75            +0.2        1.91 ±  3%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      0.54            +0.2        0.70 ±  3%  perf-profile.children.cycles-pp.__check_object_size
      0.27 ±  3%      +0.2        0.44 ±  4%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.35 ±  5%      +0.2        0.52 ±  6%  perf-profile.children.cycles-pp.tcp_current_mss
      0.13 ±  2%      +0.2        0.30 ±  6%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.38 ±  5%      +0.2        0.55 ±  5%  perf-profile.children.cycles-pp.tcp_send_mss
      0.21 ±  6%      +0.2        0.39 ±  3%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      2.86            +0.2        3.04        perf-profile.children.cycles-pp.unwind_get_return_address
      0.22 ±  2%      +0.2        0.41 ±  3%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.41 ±  3%      +0.2        0.59 ±  4%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.2        0.18 ±  6%  perf-profile.children.cycles-pp.aa_may_signal
      0.00            +0.2        0.19 ±  7%  perf-profile.children.cycles-pp.__release_sock
      0.29            +0.2        0.48 ±  3%  perf-profile.children.cycles-pp.__entry_text_start
      0.33 ±  6%      +0.2        0.53 ±  7%  perf-profile.children.cycles-pp.ip_rcv
      0.49 ±  2%      +0.2        0.70 ±  2%  perf-profile.children.cycles-pp.next_token
      0.50 ±  2%      +0.2        0.71 ±  2%  perf-profile.children.cycles-pp.dopr
      0.52 ±  2%      +0.2        0.73 ±  2%  perf-profile.children.cycles-pp.vsnprintf
      0.52            +0.2        0.74 ±  2%  perf-profile.children.cycles-pp.snprintf
      0.94            +0.2        1.18 ±  3%  perf-profile.children.cycles-pp.sio_qfileinfo
      0.15 ±  3%      +0.2        0.39 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.83            +0.2        1.07 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.46            +0.2        0.70 ±  4%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.98            +0.3        1.24 ±  2%  perf-profile.children.cycles-pp.sio_qfsinfo
      0.62            +0.3        0.90 ±  2%  perf-profile.children.cycles-pp.loopback_xmit
      0.54 ±  5%      +0.3        0.82 ±  4%  perf-profile.children.cycles-pp.release_sock
      0.12 ±  3%      +0.3        0.42 ±  4%  perf-profile.children.cycles-pp.__cond_resched
      1.20            +0.3        1.51 ±  2%  perf-profile.children.cycles-pp.sio_unlink
      0.66 ±  2%      +0.3        0.97 ±  2%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      2.34            +0.4        2.70 ±  2%  perf-profile.children.cycles-pp.orc_find
      0.77            +0.4        1.18 ±  2%  perf-profile.children.cycles-pp.__alloc_skb
      3.81            +0.4        4.23 ±  2%  perf-profile.children.cycles-pp.__orc_find
      1.00            +0.5        1.46 ±  2%  perf-profile.children.cycles-pp.tcp_stream_alloc_skb
      2.36            +0.5        2.85 ±  2%  perf-profile.children.cycles-pp.stack_access_ok
      0.26 ±  3%      +0.5        0.76 ±  8%  perf-profile.children.cycles-pp.aa_get_task_label
      2.11            +0.5        2.61 ±  2%  perf-profile.children.cycles-pp.sio_findfirst
      1.41            +0.5        1.95 ±  4%  perf-profile.children.cycles-pp.__dev_queue_xmit
      1.36            +0.6        1.94 ±  2%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.17 ±  4%      +0.6        0.76 ± 14%  perf-profile.children.cycles-pp.security_socket_sendmsg
      2.17            +0.8        2.98 ±  2%  perf-profile.children.cycles-pp.tcp_ack
      6.14            +0.9        7.00        perf-profile.children.cycles-pp.__unwind_start
      8.64            +0.9        9.55 ±  5%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      4.35            +1.1        5.43 ±  2%  perf-profile.children.cycles-pp.sio_close
      0.42 ±  4%      +1.2        1.60 ± 13%  perf-profile.children.cycles-pp.sock_recvmsg
      0.39 ±  3%      +1.2        1.57 ± 13%  perf-profile.children.cycles-pp.security_socket_recvmsg
     33.90            +1.2       35.11 ±  2%  perf-profile.children.cycles-pp.server
     33.95            +1.2       35.18 ±  2%  perf-profile.children.cycles-pp.listener
      3.62            +1.3        4.92 ±  3%  perf-profile.children.cycles-pp.sio_writex
      5.44            +1.4        6.80 ±  2%  perf-profile.children.cycles-pp.sio_qpathinfo
      0.41 ±  2%      +1.4        1.81 ±  6%  perf-profile.children.cycles-pp.apparmor_task_kill
      0.41 ±  2%      +1.4        1.82 ±  6%  perf-profile.children.cycles-pp.security_task_kill
      0.50 ±  2%      +1.4        1.94 ±  6%  perf-profile.children.cycles-pp.kill_pid_info
      6.00            +1.5        7.48 ±  2%  perf-profile.children.cycles-pp.sio_createx
      0.61 ±  2%      +1.5        2.11 ±  5%  perf-profile.children.cycles-pp.kill_something_info
      0.76 ±  2%      +1.5        2.28 ±  5%  perf-profile.children.cycles-pp.__x64_sys_kill
      1.00            +1.7        2.73 ±  5%  perf-profile.children.cycles-pp.kill
      0.42 ±  3%      +1.8        2.19 ± 14%  perf-profile.children.cycles-pp.aa_sk_perm
     26.32            +1.9       28.24 ±  3%  perf-profile.children.cycles-pp.tcp_rcv_established
     26.42            +2.0       28.39 ±  3%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      7.52            +2.0        9.53 ±  4%  perf-profile.children.cycles-pp.perf_trace_sched_switch
     30.22            +2.1       32.28 ±  2%  perf-profile.children.cycles-pp.net_rx_action
     27.78            +2.1       29.85 ±  3%  perf-profile.children.cycles-pp.tcp_v4_rcv
     27.92            +2.1       29.98 ±  3%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     27.98            +2.1       30.04 ±  2%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      2.28            +2.1        4.37 ± 36%  perf-profile.children.cycles-pp.__get_user_nocheck_8
     30.76            +2.1       32.89 ±  2%  perf-profile.children.cycles-pp.__softirqentry_text_start
     30.73            +2.2       32.95 ±  2%  perf-profile.children.cycles-pp.do_softirq
      2.85            +2.2        5.08 ± 30%  perf-profile.children.cycles-pp.perf_callchain_user
      9.38            +2.3       11.64 ±  2%  perf-profile.children.cycles-pp.sio_readx
     30.88            +2.4       33.23 ±  2%  perf-profile.children.cycles-pp.__local_bh_enable_ip
     15.50            +2.4       17.88 ±  2%  perf-profile.children.cycles-pp.unwind_next_frame
     18.11            +2.5       20.60 ±  2%  perf-profile.children.cycles-pp.schedule
     28.64            +2.5       31.14 ±  2%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     28.89            +2.5       31.44 ±  2%  perf-profile.children.cycles-pp.process_backlog
     28.93            +2.6       31.49 ±  2%  perf-profile.children.cycles-pp.__napi_poll
     20.63            +2.8       23.39 ±  2%  perf-profile.children.cycles-pp.perf_callchain_kernel
     32.43            +2.9       35.34 ±  2%  perf-profile.children.cycles-pp.ip_finish_output2
      4.89 ±  2%      +3.1        8.02 ±  4%  perf-profile.children.cycles-pp.enqueue_task_fair
      4.04 ±  2%      +3.1        7.17 ±  4%  perf-profile.children.cycles-pp.enqueue_entity
      4.93 ±  2%      +3.2        8.08 ±  4%  perf-profile.children.cycles-pp.ttwu_do_activate
     33.32            +3.2       36.53 ±  2%  perf-profile.children.cycles-pp.__ip_queue_xmit
     34.40            +3.6       37.96 ±  2%  perf-profile.children.cycles-pp.__tcp_transmit_skb
      0.93 ±  2%      +3.8        4.77 ±  5%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.00            +3.9        3.86 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
     34.80            +3.9       38.73 ±  2%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
     35.32            +4.1       39.41 ±  2%  perf-profile.children.cycles-pp.tcp_write_xmit
      1.06            +4.6        5.62 ± 23%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      7.76            +4.7       12.47 ±  3%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      8.48            +4.9       13.33 ±  3%  perf-profile.children.cycles-pp.update_curr
     27.72            +4.9       32.66 ±  4%  perf-profile.children.cycles-pp.perf_event_output_forward
     27.82            +5.0       32.77 ±  4%  perf-profile.children.cycles-pp.__perf_event_overflow
     39.42            +5.0       44.44 ±  2%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
     23.95            +5.0       29.00 ±  4%  perf-profile.children.cycles-pp.get_perf_callchain
     24.04            +5.1       29.11 ±  4%  perf-profile.children.cycles-pp.perf_callchain
     28.56            +5.1       33.66 ±  4%  perf-profile.children.cycles-pp.perf_tp_event
     25.16            +5.2       30.40 ±  4%  perf-profile.children.cycles-pp.perf_prepare_sample
     39.78            +5.3       45.10 ±  2%  perf-profile.children.cycles-pp.tcp_sendmsg
     40.09            +5.9       45.99 ±  2%  perf-profile.children.cycles-pp.sock_sendmsg
     40.27            +6.0       46.29 ±  2%  perf-profile.children.cycles-pp.__sys_sendto
     40.33            +6.1       46.39 ±  2%  perf-profile.children.cycles-pp.__x64_sys_sendto
     34.03            +8.7       42.75 ±  2%  perf-profile.children.cycles-pp.do_packets
     35.85            +9.2       45.03 ±  2%  perf-profile.children.cycles-pp.child_op
     37.62            +9.8       47.41 ±  2%  perf-profile.children.cycles-pp.child_run
     40.90           +10.9       51.79        perf-profile.children.cycles-pp.__send
     38.72           +11.5       50.25        perf-profile.children.cycles-pp.create_procs
     68.40           +12.0       80.44        perf-profile.children.cycles-pp.do_syscall_64
     68.64           +12.1       80.78        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     72.71           +12.8       85.46        perf-profile.children.cycles-pp.main
     72.71           +12.8       85.47        perf-profile.children.cycles-pp.__libc_start_main
     15.01            -7.7        7.34 ±  4%  perf-profile.self.cycles-pp.mwait_idle_with_hints
      2.49 ±  7%      -1.4        1.10 ±  9%  perf-profile.self.cycles-pp.update_cfs_group
      2.48 ±  4%      -1.3        1.16 ±  3%  perf-profile.self.cycles-pp.update_load_avg
      0.80 ±  8%      -0.6        0.16 ±  8%  perf-profile.self.cycles-pp.set_task_cpu
      1.61            -0.6        1.02 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.16 ±  2%      -0.6        0.60 ±  4%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.13            -0.5        0.60 ±  5%  perf-profile.self.cycles-pp.available_idle_cpu
      0.87            -0.3        0.56 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.46 ±  3%      -0.2        0.24 ±  5%  perf-profile.self.cycles-pp.select_idle_core
      0.87            -0.2        0.71        perf-profile.self.cycles-pp.__schedule
      0.32 ±  2%      -0.2        0.16 ±  5%  perf-profile.self.cycles-pp.menu_select
      0.30            -0.1        0.15 ±  7%  perf-profile.self.cycles-pp.llist_add_batch
      0.32 ±  2%      -0.1        0.17 ±  6%  perf-profile.self.cycles-pp.migrate_task_rq_fair
      0.22 ±  6%      -0.1        0.08 ±  9%  perf-profile.self.cycles-pp.bpf_ksym_find
      0.32 ±  2%      -0.1        0.19 ±  8%  perf-profile.self.cycles-pp.perf_output_end
      0.23 ±  4%      -0.1        0.12 ±  6%  perf-profile.self.cycles-pp._find_next_bit
      0.99            -0.1        0.88 ±  6%  perf-profile.self.cycles-pp.memcpy_erms
      0.36            -0.1        0.26        perf-profile.self.cycles-pp.wait_woken
      0.19 ±  3%      -0.1        0.09 ±  7%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.55            -0.1        0.45 ±  3%  perf-profile.self.cycles-pp.skb_release_data
      0.18 ±  5%      -0.1        0.09 ±  5%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.18 ±  2%      -0.1        0.09 ±  7%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.54            -0.1        0.45 ±  3%  perf-profile.self.cycles-pp.recv
      0.28 ±  3%      -0.1        0.19 ±  3%  perf-profile.self.cycles-pp.perf_event_output_forward
      0.14 ±  2%      -0.1        0.06 ±  8%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.17 ±  2%      -0.1        0.09 ± 10%  perf-profile.self.cycles-pp.llist_reverse_order
      0.36 ±  7%      -0.1        0.27 ±  5%  perf-profile.self.cycles-pp.read_sock
      0.24 ±  3%      -0.1        0.17 ±  5%  perf-profile.self.cycles-pp.finish_task_switch
      0.48            -0.1        0.41 ±  4%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.60            -0.1        0.52        perf-profile.self.cycles-pp.__switch_to_asm
      0.14 ±  3%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.task_h_load
      0.23 ±  4%      -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.kfree_skbmem
      0.16 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.66            -0.1        0.59        perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.39 ±  3%      -0.1        0.32 ±  3%  perf-profile.self.cycles-pp.net_rx_action
      0.27 ±  2%      -0.1        0.21 ±  3%  perf-profile.self.cycles-pp.check_heap_object
      0.18 ±  2%      -0.1        0.11 ±  6%  perf-profile.self.cycles-pp.get_stack_info_noinstr
      0.27 ±  3%      -0.1        0.21 ±  4%  perf-profile.self.cycles-pp.switch_fpu_return
      0.23 ±  2%      -0.1        0.17 ±  4%  perf-profile.self.cycles-pp.skb_page_frag_refill
      0.08 ±  5%      -0.1        0.02 ± 99%  perf-profile.self.cycles-pp.__update_idle_core
      0.22 ±  2%      -0.1        0.16 ±  3%  perf-profile.self.cycles-pp.sock_rfree
      0.10 ±  4%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.do_idle
      0.24 ±  3%      -0.0        0.19 ±  3%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.10 ±  4%      -0.0        0.05 ±  7%  perf-profile.self.cycles-pp.__slab_free
      0.15 ±  4%      -0.0        0.11        perf-profile.self.cycles-pp.select_idle_sibling
      0.18 ±  5%      -0.0        0.13 ±  2%  perf-profile.self.cycles-pp.tcp_rcv_space_adjust
      0.16 ±  2%      -0.0        0.12 ±  4%  perf-profile.self.cycles-pp.get_stack_info
      0.43 ±  3%      -0.0        0.39 ±  3%  perf-profile.self.cycles-pp.__switch_to
      0.10 ±  3%      -0.0        0.06        perf-profile.self.cycles-pp.newidle_balance
      0.15            -0.0        0.11 ±  6%  perf-profile.self.cycles-pp.schedule_timeout
      0.33 ±  3%      -0.0        0.29 ±  4%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.19            -0.0        0.15 ±  4%  perf-profile.self.cycles-pp.try_to_wake_up
      0.10 ±  3%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.23            -0.0        0.20 ±  3%  perf-profile.self.cycles-pp.tcp_queue_rcv
      0.57 ±  2%      -0.0        0.54        perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.16 ±  3%      -0.0        0.12 ±  4%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.12 ±  7%      -0.0        0.09 ±  6%  perf-profile.self.cycles-pp.poll_idle
      0.11 ±  3%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.select_task_rq
      0.08 ±  6%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.cubictcp_cwnd_event
      0.08 ±  6%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.__build_skb_around
      0.13 ±  3%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.dequeue_entity
      0.07 ±  6%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.ntohl@plt
      0.21 ±  2%      -0.0        0.19 ±  3%  perf-profile.self.cycles-pp.prepare_task_switch
      0.05            +0.0        0.06        perf-profile.self.cycles-pp.audit_signal_info_syscall
      0.06 ±  7%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__perf_event_overflow
      0.05            +0.0        0.06 ±  6%  perf-profile.self.cycles-pp.clear_buddies
      0.05            +0.0        0.06 ±  6%  perf-profile.self.cycles-pp.perf_trace_run_bpf_submit
      0.07 ±  5%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.skb_release_head_state
      0.05            +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.__list_add_valid
      0.08            +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.set_next_entity
      0.26 ±  2%      +0.0        0.28 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.05 ±  7%      +0.0        0.07 ± 14%  perf-profile.self.cycles-pp.kill
      0.08 ±  6%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.sock_sendmsg
      0.06 ±  8%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.nb_target_rate
      0.27 ±  3%      +0.0        0.30 ±  3%  perf-profile.self.cycles-pp.perf_prepare_sample
      0.13            +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.__alloc_skb
      0.07 ±  6%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.is_ftrace_trampoline
      0.06 ±  9%      +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
      0.12 ±  4%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.17 ±  4%      +0.0        0.19 ±  5%  perf-profile.self.cycles-pp.__mod_timer
      0.06 ±  7%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.cubictcp_acked
      0.06 ± 11%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.rb_erase
      0.12 ±  3%      +0.0        0.15 ±  4%  perf-profile.self.cycles-pp.do_softirq
      0.07 ±  7%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.18 ±  5%      +0.0        0.21 ±  5%  perf-profile.self.cycles-pp.kvm_guest_state
      0.12 ±  7%      +0.0        0.15 ±  7%  perf-profile.self.cycles-pp.release_sock
      0.18 ±  2%      +0.0        0.20 ±  3%  perf-profile.self.cycles-pp.__perf_event_header__init_id
      0.08 ±  6%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_event_data_recv
      0.08 ±  6%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.import_single_range
      0.09 ±  4%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.04 ± 44%      +0.0        0.07 ±  9%  perf-profile.self.cycles-pp.is_bpf_text_address
      0.06 ±  6%      +0.0        0.09 ± 12%  perf-profile.self.cycles-pp.sock_put
      0.07 ±  7%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.20 ±  3%      +0.0        0.23 ±  5%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.34 ±  2%      +0.0        0.37 ±  2%  perf-profile.self.cycles-pp.__task_pid_nr_ns
      0.15 ±  4%      +0.0        0.18 ±  6%  perf-profile.self.cycles-pp.reweight_entity
      0.06 ±  6%      +0.0        0.09 ±  6%  perf-profile.self.cycles-pp.sockfd_lookup_light
      0.32            +0.0        0.35 ±  2%  perf-profile.self.cycles-pp.ftrace_graph_ret_addr
      0.08 ±  4%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp._copy_from_iter
      0.15 ±  3%      +0.0        0.18 ±  4%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.30 ±  4%      +0.0        0.33 ±  3%  perf-profile.self.cycles-pp.do_packets
      0.14 ±  5%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.tcp_skb_entail
      0.07 ±  6%      +0.0        0.11 ±  5%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.12 ±  3%      +0.0        0.16 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.08 ±  5%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.__is_insn_slot_addr
      0.06 ±  7%      +0.0        0.10        perf-profile.self.cycles-pp.__x64_sys_sendto
      0.42            +0.0        0.46 ±  2%  perf-profile.self.cycles-pp.native_sched_clock
      0.08 ±  9%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.tcp_rearm_rto
      0.30 ±  4%      +0.0        0.34 ±  2%  perf-profile.self.cycles-pp.get_perf_callchain
      0.08 ±  6%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.schedule
      0.07 ±  8%      +0.0        0.11 ±  3%  perf-profile.self.cycles-pp.pick_next_entity
      0.06 ±  7%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.__cond_resched
      0.06 ± 11%      +0.0        0.10 ±  6%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.04 ± 44%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.__x64_sys_kill
      0.16 ±  4%      +0.0        0.20 ±  3%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.03 ± 70%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.nf_hook_slow
      0.06 ±  9%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.ip_rcv
      0.24 ±  5%      +0.0        0.28 ±  3%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.10 ±  4%      +0.0        0.14 ±  6%  perf-profile.self.cycles-pp.write_sock
      0.39 ±  2%      +0.0        0.44 ±  3%  perf-profile.self.cycles-pp.__fdget
      0.01 ±223%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.kmalloc_slab
      0.01 ±223%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.__xfrm_policy_check2
      0.88            +0.0        0.92 ±  2%  perf-profile.self.cycles-pp.kernel_text_address
      0.10 ±  7%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.__send
      0.02 ± 99%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.tcp_sendmsg
      0.08 ±  6%      +0.1        0.12 ±  6%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.08 ±  5%      +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.ip_rcv_core
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.skb_clone
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.tcp_data_ready
      0.20 ±  2%      +0.1        0.26 ±  3%  perf-profile.self.cycles-pp.do_syscall_64
      0.16 ±  4%      +0.1        0.21 ±  4%  perf-profile.self.cycles-pp.dopr
      0.22 ±  2%      +0.1        0.27 ±  3%  perf-profile.self.cycles-pp.tcp_stream_alloc_skb
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.skb_do_copy_data_nocache
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.kmalloc_reserve
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.save_fpregs_to_fpstate
      0.09            +0.1        0.14 ±  4%  perf-profile.self.cycles-pp.__sys_sendto
      0.05 ±  8%      +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__copy_skb_header
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.perf_instruction_pointer
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__napi_poll
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.netif_skb_features
      0.09 ±  5%      +0.1        0.15 ±  6%  perf-profile.self.cycles-pp.all_string_sub
      0.08 ±  8%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.ip_skb_dst_mtu
      0.06            +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.ip_output
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.security_sock_rcv_skb
      0.19 ±  7%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.dst_release
      0.04 ± 44%      +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.perf_misc_flags
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.is_module_text_address
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_established_options
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.tcp_chrono_start
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.eth_type_trans
      0.07 ±  5%      +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.apparmor_ip_postroute
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__tcp_push_pending_frames
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.__sk_dst_check
      0.18 ±  5%      +0.1        0.25 ±  9%  perf-profile.self.cycles-pp.ktime_get
      0.14 ±  3%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.__might_sleep
      0.50 ±  3%      +0.1        0.56        perf-profile.self.cycles-pp.tcp_ack
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.tcp_inbound_md5_hash
      0.15 ±  3%      +0.1        0.22 ±  3%  perf-profile.self.cycles-pp.loopback_xmit
      0.36 ±  2%      +0.1        0.43 ±  3%  perf-profile.self.cycles-pp.perf_output_begin_forward
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.52            +0.1        0.59 ±  2%  perf-profile.self.cycles-pp.unwind_get_return_address
      0.24 ±  9%      +0.1        0.31 ±  6%  perf-profile.self.cycles-pp.ipv4_mtu
      0.18 ±  5%      +0.1        0.25        perf-profile.self.cycles-pp.os_xsave
      0.01 ±223%      +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.rb_insert_color
      0.01 ±223%      +0.1        0.08 ±  4%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.__tcp_select_window
      0.07 ±  7%      +0.1        0.14 ±  5%  perf-profile.self.cycles-pp._copy_to_iter
      0.10 ±  3%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.07 ±  5%      +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.tcp_tso_segs
      0.01 ±223%      +0.1        0.08 ±  7%  perf-profile.self.cycles-pp.inet_ehashfn
      0.06 ±  7%      +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.__check_object_size
      0.17 ±  2%      +0.1        0.24 ±  3%  perf-profile.self.cycles-pp.fmtstr
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.cubictcp_cong_avoid
      0.08            +0.1        0.16 ±  3%  perf-profile.self.cycles-pp.tcp_wfree
      0.12 ±  4%      +0.1        0.20 ±  6%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.__ip_finish_output
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.memcg_slab_free_hook
      0.12 ±  5%      +0.1        0.20 ±  6%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.00            +0.1        0.08 ±  4%  perf-profile.self.cycles-pp.__might_fault
      0.50 ±  2%      +0.1        0.58 ±  2%  perf-profile.self.cycles-pp.child_op
      0.38            +0.1        0.46 ±  2%  perf-profile.self.cycles-pp.__kernel_text_address
      0.29 ±  2%      +0.1        0.37 ±  2%  perf-profile.self.cycles-pp.__might_resched
      0.18 ±  3%      +0.1        0.26 ±  2%  perf-profile.self.cycles-pp.dopr_outch
      0.08 ±  6%      +0.1        0.16 ±  6%  perf-profile.self.cycles-pp.__calc_delta
      0.40 ±  4%      +0.1        0.48        perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.32 ±  2%      +0.1        0.40 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.put_prev_entity
      0.07 ±  7%      +0.1        0.16 ±  3%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.06 ±  6%      +0.1        0.15 ±  4%  perf-profile.self.cycles-pp.kmem_cache_free
      0.12 ±  3%      +0.1        0.21 ±  4%  perf-profile.self.cycles-pp.__entry_text_start
      0.18 ± 14%      +0.1        0.28 ± 11%  perf-profile.self.cycles-pp.ipv4_dst_check
      0.08 ±  8%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp.tcp_current_mss
      0.37 ±  4%      +0.1        0.47 ±  3%  perf-profile.self.cycles-pp.child_run
      0.24 ±  2%      +0.1        0.33 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.08 ±  7%      +0.1        0.18 ±  8%  perf-profile.self.cycles-pp.tcp_rtt_estimator
      0.01 ±223%      +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.set_next_buddy
      0.18 ±  4%      +0.1        0.28 ±  3%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.18 ± 12%      +0.1        0.29 ± 11%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.11 ±  6%      +0.1        0.22 ±  3%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.54            +0.1        0.66 ±  4%  perf-profile.self.cycles-pp.perf_callchain_user
      0.43 ±  2%      +0.1        0.55 ±  7%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.22 ±  4%      +0.1        0.34 ±  2%  perf-profile.self.cycles-pp.read_tsc
      0.50            +0.1        0.61 ±  2%  perf-profile.self.cycles-pp.__unwind_start
      0.20 ±  3%      +0.1        0.32 ±  4%  perf-profile.self.cycles-pp.ip_finish_output2
      0.06 ±  6%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp.kfree
      0.15 ±  4%      +0.1        0.28 ±  7%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.14 ±  3%      +0.1        0.27 ±  3%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.53 ±  2%      +0.1        0.66 ±  3%  perf-profile.self.cycles-pp.perf_tp_event
      0.22 ±  2%      +0.2        0.38        perf-profile.self.cycles-pp.tcp_write_xmit
      0.21 ±  6%      +0.2        0.38 ±  3%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.42 ±  2%      +0.2        0.60 ±  3%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.00            +0.2        0.18 ±  5%  perf-profile.self.cycles-pp.aa_may_signal
      0.49            +0.2        0.69        perf-profile.self.cycles-pp.next_token
      1.66            +0.2        1.88 ±  3%  perf-profile.self.cycles-pp.perf_callchain_kernel
      0.46            +0.2        0.70 ±  4%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      2.24            +0.3        2.58 ±  2%  perf-profile.self.cycles-pp.orc_find
      3.80            +0.4        4.22 ±  2%  perf-profile.self.cycles-pp.__orc_find
      2.10            +0.4        2.53 ±  2%  perf-profile.self.cycles-pp.stack_access_ok
      0.26 ±  2%      +0.5        0.75 ±  8%  perf-profile.self.cycles-pp.aa_get_task_label
      0.06 ± 11%      +0.7        0.76 ±200%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.10 ±  4%      +0.8        0.86 ±  9%  perf-profile.self.cycles-pp.apparmor_task_kill
      6.86            +1.1        7.99 ±  2%  perf-profile.self.cycles-pp.unwind_next_frame
      2.24            +1.2        3.46 ± 20%  perf-profile.self.cycles-pp.__get_user_nocheck_8
      0.34 ±  3%      +1.7        2.05 ± 15%  perf-profile.self.cycles-pp.aa_sk_perm


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

View attachment "config-5.19.0-rc2-00027-gc82a69629c53" of type "text/plain" (163467 bytes)

View attachment "job-script" of type "text/plain" (8271 bytes)

View attachment "job.yaml" of type "text/plain" (5588 bytes)

View attachment "reproduce" of type "text/plain" (305 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ