lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Wed, 28 Sep 2022 15:50:23 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Shakeel Butt <shakeelb@...gle.com>
CC:     <lkp@...ts.01.org>, <lkp@...el.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        kernel test robot <oliver.sang@...el.com>,
        Soheil Hassas Yeganeh <soheil@...gle.com>,
        Feng Tang <feng.tang@...el.com>,
        Roman Gushchin <roman.gushchin@...ux.dev>,
        Muchun Song <songmuchun@...edance.com>,
        Michal Hocko <mhocko@...e.com>,
        Eric Dumazet <edumazet@...gle.com>,
        Johannes Weiner <hannes@...xchg.org>,
        Michal Koutný <mkoutny@...e.com>,
        Linux Memory Management List <linux-mm@...ck.org>,
        <linux-kernel@...r.kernel.org>, <ying.huang@...el.com>,
        <zhengjun.xing@...ux.intel.com>, <fengwei.yin@...el.com>
Subject: [memcg]  1813e51eec:  netperf.Throughput_Mbps 37.5% improvement


Greeting,

FYI, we noticed a 37.5% improvement of netperf.Throughput_Mbps due to commit:


commit: 1813e51eece0ad6f4aacaeb738e7cced46feb470 ("memcg: increase MEMCG_CHARGE_BATCH to 64")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

in testcase: netperf
on test machine: 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz (Cooper Lake) with 128G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 50%
	cluster: cs-localhost
	send_size: 5K
	test: TCP_SENDFILE
	cpufreq_governor: performance

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_Mbps 54.7% improvement                                             |
| test machine     | 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz (Cooper Lake) with 128G memory |
| test parameters  | cluster=cs-localhost                                                                           |
|                  | cpufreq_governor=performance                                                                   |
|                  | ip=ipv4                                                                                        |
|                  | nr_threads=25%                                                                                 |
|                  | runtime=900s                                                                                   |
|                  | test=TCP_MAERTS                                                                                |
+------------------+------------------------------------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/50%/debian-11.1-x86_64-20220510.cgz/300s/5K/lkp-cpl-4sp1/TCP_SENDFILE/netperf

commit: 
  408587baee ("mm: page_counter: rearrange struct page_counter fields")
  1813e51eec ("memcg: increase MEMCG_CHARGE_BATCH to 64")

408587baee397533 1813e51eece0ad6f4aacaeb738e 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      9510           +37.5%      13080        netperf.Throughput_Mbps
    684758           +37.5%     941808        netperf.Throughput_total_Mbps
    172773 ± 10%     +28.9%     222638 ±  7%  netperf.time.involuntary_context_switches
      5829            +3.5%       6034        netperf.time.percent_of_cpu_this_job_got
     17097            +2.4%      17502        netperf.time.system_time
    457.00 ±  4%     +46.3%     668.80        netperf.time.user_time
      4316 ± 22%   +1980.8%      89807 ± 19%  netperf.time.voluntary_context_switches
 5.015e+09           +37.5%  6.898e+09        netperf.workload
    164695 ± 10%     +28.0%     210747 ± 13%  numa-numastat.node1.local_node
    164678 ± 10%     +27.9%     210694 ± 13%  numa-vmstat.node1.numa_local
     15906 ±  2%      -9.1%      14462 ±  2%  uptime.idle
     52.99 ±  4%      -7.5%      48.99 ±  4%  boot-time.boot
      7009 ±  4%      -8.2%       6432 ±  5%  boot-time.idle
 8.407e+09           -11.4%  7.453e+09        cpuidle..time
 4.312e+08           +65.6%  7.142e+08        cpuidle..usage
     19.48            -1.9       17.59        mpstat.cpu.all.idle%
      1.59 ±  3%      +0.6        2.19        mpstat.cpu.all.usr%
   8546062 ±  2%     +14.6%    9793462        vmstat.memory.cache
   2802768           +66.1%    4656023        vmstat.system.cs
    366202            -5.4%     346419        vmstat.system.in
   8366906 ±  2%     +14.8%    9607481        meminfo.Cached
   6311711 ±  3%     +19.7%    7554689        meminfo.Committed_AS
   5964785 ±  3%     +21.0%    7215654        meminfo.Inactive
   5964605 ±  3%     +21.0%    7215134        meminfo.Inactive(anon)
   1182234 ±  2%     +10.8%    1309581 ±  2%  meminfo.Mapped
  10599562 ±  2%     +11.8%   11847313        meminfo.Memused
   5704758 ±  3%     +21.7%    6945032        meminfo.Shmem
  10715816 ±  2%     +11.6%   11962837        meminfo.max_used_kB
      3234            -6.6%       3021        turbostat.Bzy_MHz
 4.024e+08 ±  2%     +74.2%  7.011e+08        turbostat.C1
     15.93            -1.4       14.51        turbostat.C1%
  26648656 ± 15%     -95.2%    1288309 ± 35%  turbostat.C1E
      2.78 ± 19%      -1.8        0.96 ± 58%  turbostat.C1E%
    256905 ± 92%    +180.8%     721482 ± 27%  turbostat.C6
      0.45 ± 76%      +0.9        1.31 ± 32%  turbostat.C6%
     17.80           -17.3%      14.72        turbostat.CPU%c1
      0.12           +50.0%       0.18        turbostat.IPC
   1876464 ±  2%    +490.8%   11085230 ±  4%  turbostat.POLL
      0.07            +0.2        0.27 ±  3%  turbostat.POLL%
     11.38            -4.9%      10.82        turbostat.RAMWatt
     34234 ±  4%      -7.1%      31786 ±  2%  proc-vmstat.nr_active_anon
   3008539            -1.0%    2977532        proc-vmstat.nr_dirty_background_threshold
   6024435            -1.0%    5962346        proc-vmstat.nr_dirty_threshold
   2092764 ±  2%     +14.8%    2401568        proc-vmstat.nr_file_pages
  30262657            -1.0%   29952035        proc-vmstat.nr_free_pages
   1492182 ±  3%     +20.9%    1803494        proc-vmstat.nr_inactive_anon
    296002 ±  2%     +10.6%     327264 ±  2%  proc-vmstat.nr_mapped
   1427226 ±  3%     +21.6%    1735955        proc-vmstat.nr_shmem
     35204            +2.0%      35905        proc-vmstat.nr_slab_reclaimable
     34234 ±  4%      -7.1%      31786 ±  2%  proc-vmstat.nr_zone_active_anon
   1492182 ±  3%     +20.9%    1803494        proc-vmstat.nr_zone_inactive_anon
   4677653 ±  2%     +17.5%    5495038        proc-vmstat.numa_hit
   4472513 ±  3%     +18.8%    5312209        proc-vmstat.numa_local
   4668058 ±  3%     +17.7%    5495063        proc-vmstat.pgalloc_normal
    828068            +3.0%     852870        proc-vmstat.pgfree
    132834 ± 27%     +67.3%     222170 ± 18%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   1016263 ± 14%     +44.6%    1469946 ± 10%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    132834 ± 27%     +67.3%     222170 ± 18%  sched_debug.cfs_rq:/.max_vruntime.avg
   1016263 ± 14%     +44.6%    1469946 ± 10%  sched_debug.cfs_rq:/.max_vruntime.stddev
      0.80 ± 79%    +197.9%       2.38 ± 33%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
      6.99 ± 52%     +97.6%      13.81 ± 23%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      0.80 ± 79%    +197.9%       2.38 ± 33%  sched_debug.cfs_rq:/.removed.util_avg.avg
      6.99 ± 52%     +97.6%      13.81 ± 23%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     80.94 ± 33%     -53.3%      37.78 ± 33%  sched_debug.cfs_rq:/.runnable_avg.min
    454528 ± 60%    -128.4%    -129100        sched_debug.cfs_rq:/.spread0.avg
     79.06 ± 32%     -53.4%      36.81 ± 33%  sched_debug.cfs_rq:/.util_avg.min
    266000 ±  5%      -9.9%     239641 ±  4%  sched_debug.cpu.avg_idle.avg
      9882 ±  7%     -38.4%       6092 ±  2%  sched_debug.cpu.avg_idle.min
      1010 ±  4%     -12.1%     887.63 ±  2%  sched_debug.cpu.clock_task.stddev
   2939459           +67.0%    4908564        sched_debug.cpu.nr_switches.avg
   4503634 ±  4%     +43.6%    6465547 ±  2%  sched_debug.cpu.nr_switches.max
   1896742 ±  7%     +76.9%    3355314 ±  4%  sched_debug.cpu.nr_switches.min
    449995 ±  4%     +30.4%     586586 ±  4%  sched_debug.cpu.nr_switches.stddev
 3.187e+10           +44.5%  4.605e+10        perf-stat.i.branch-instructions
      1.33            -0.0        1.29        perf-stat.i.branch-miss-rate%
 4.183e+08           +40.1%  5.862e+08        perf-stat.i.branch-misses
     13.69            -4.5        9.19 ±  4%  perf-stat.i.cache-miss-rate%
 1.164e+09           +40.9%  1.639e+09        perf-stat.i.cache-references
   2831754           +65.9%    4699259        perf-stat.i.context-switches
      2.40           -32.9%       1.61        perf-stat.i.cpi
 3.862e+11            -3.0%  3.748e+11        perf-stat.i.cpu-cycles
     43725 ±  9%     -41.3%      25665 ±  9%  perf-stat.i.cpu-migrations
      0.01 ±  2%      -0.0        0.01 ±  5%  perf-stat.i.dTLB-load-miss-rate%
 4.854e+10           +44.6%  7.019e+10        perf-stat.i.dTLB-loads
      0.00 ±  7%      -0.0        0.00 ±  7%  perf-stat.i.dTLB-store-miss-rate%
   1260320 ±  8%     -36.7%     798303 ±  8%  perf-stat.i.dTLB-store-misses
 2.637e+10           +45.4%  3.833e+10        perf-stat.i.dTLB-stores
     88.58            -1.8       86.79        perf-stat.i.iTLB-load-miss-rate%
 2.906e+08           +43.0%  4.155e+08        perf-stat.i.iTLB-load-misses
  36764691           +69.5%   62323169        perf-stat.i.iTLB-loads
 1.608e+11           +44.6%  2.325e+11        perf-stat.i.instructions
      0.42           +48.6%       0.62        perf-stat.i.ipc
      2.68            -3.0%       2.60        perf-stat.i.metric.GHz
    591.96           +31.1%     775.86        perf-stat.i.metric.K/sec
    749.57           +44.7%       1084        perf-stat.i.metric.M/sec
   5402586            -9.4%    4893490 ±  2%  perf-stat.i.node-loads
  10881630            +7.5%   11693181 ±  3%  perf-stat.i.node-store-misses
    381960 ±  7%     +18.5%     452668 ± 11%  perf-stat.i.node-stores
      1.31            -0.0        1.27        perf-stat.overall.branch-miss-rate%
     13.66            -4.5        9.14 ±  4%  perf-stat.overall.cache-miss-rate%
      2.40           -32.9%       1.61        perf-stat.overall.cpi
      0.01            -0.0        0.01 ±  5%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  7%      -0.0        0.00 ±  8%  perf-stat.overall.dTLB-store-miss-rate%
     88.77            -1.8       86.96        perf-stat.overall.iTLB-load-miss-rate%
    553.39            +1.1%     559.60        perf-stat.overall.instructions-per-iTLB-miss
      0.42           +49.0%       0.62        perf-stat.overall.ipc
      9673            +5.3%      10185        perf-stat.overall.path-length
 3.177e+10           +44.5%   4.59e+10        perf-stat.ps.branch-instructions
 4.169e+08           +40.1%  5.842e+08        perf-stat.ps.branch-misses
  1.16e+09           +40.9%  1.634e+09        perf-stat.ps.cache-references
   2822240           +65.9%    4683352        perf-stat.ps.context-switches
  3.85e+11            -3.0%  3.735e+11        perf-stat.ps.cpu-cycles
     43565 ±  9%     -41.3%      25585 ±  9%  perf-stat.ps.cpu-migrations
   5110621 ±  2%     +11.6%    5703476 ±  5%  perf-stat.ps.dTLB-load-misses
 4.838e+10           +44.6%  6.996e+10        perf-stat.ps.dTLB-loads
   1255754 ±  8%     -36.6%     795724 ±  8%  perf-stat.ps.dTLB-store-misses
 2.628e+10           +45.4%  3.821e+10        perf-stat.ps.dTLB-stores
 2.896e+08           +43.0%  4.142e+08        perf-stat.ps.iTLB-load-misses
  36640329           +69.5%   62113143        perf-stat.ps.iTLB-loads
 1.603e+11           +44.6%  2.318e+11        perf-stat.ps.instructions
   5374412            -9.4%    4871362 ±  2%  perf-stat.ps.node-loads
  10848893            +7.4%   11655163 ±  3%  perf-stat.ps.node-store-misses
    382976 ±  7%     +17.9%     451586 ± 11%  perf-stat.ps.node-stores
 4.852e+13           +44.8%  7.026e+13        perf-stat.total.instructions
     10.24 ±  2%      -5.3        4.90 ±  7%  perf-profile.calltrace.cycles-pp.try_charge_memcg.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
      8.22 ±  3%      -5.3        2.97 ±  2%  perf-profile.calltrace.cycles-pp.try_charge_memcg.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule
     38.02            -5.1       32.90        perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage
      8.61 ±  2%      -5.1        3.50 ±  2%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_build_frag
     38.35            -4.9       33.50        perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe
     38.51            -4.8       33.73        perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.generic_splice_sendpage
     38.61            -4.7       33.87        perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.generic_splice_sendpage.direct_splice_actor
     38.80            -4.7       34.10        perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.generic_splice_sendpage.direct_splice_actor.splice_direct_to_actor
     40.34            -4.6       35.79        perf-profile.calltrace.cycles-pp.__splice_from_pipe.generic_splice_sendpage.direct_splice_actor.splice_direct_to_actor.do_splice_direct
     40.54            -4.5       36.09        perf-profile.calltrace.cycles-pp.generic_splice_sendpage.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile
     40.63            -4.4       36.23        perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      7.02 ±  2%      -3.8        3.17 ±  8%  perf-profile.calltrace.cycles-pp.page_counter_try_charge.try_charge_memcg.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established
      5.56 ±  3%      -3.7        1.91 ±  3%  perf-profile.calltrace.cycles-pp.page_counter_try_charge.try_charge_memcg.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule
      9.62 ±  2%      -3.6        6.00        perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_build_frag.do_tcp_sendpages
      9.66 ±  2%      -3.6        6.06        perf-profile.calltrace.cycles-pp.__sk_mem_schedule.tcp_wmem_schedule.tcp_build_frag.do_tcp_sendpages.tcp_sendpage
      9.76 ±  2%      -3.6        6.20        perf-profile.calltrace.cycles-pp.tcp_wmem_schedule.tcp_build_frag.do_tcp_sendpages.tcp_sendpage.inet_sendpage
     47.53            -3.5       44.08        perf-profile.calltrace.cycles-pp.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
      6.11 ±  2%      -3.4        2.76 ±  3%  perf-profile.calltrace.cycles-pp.drain_stock.refill_stock.__sk_mem_reduce_allocated.tcp_clean_rtx_queue.tcp_ack
      6.19 ±  2%      -3.4        2.84 ±  3%  perf-profile.calltrace.cycles-pp.refill_stock.__sk_mem_reduce_allocated.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      6.09 ±  2%      -3.3        2.75 ±  3%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.drain_stock.refill_stock.__sk_mem_reduce_allocated.tcp_clean_rtx_queue
      6.38 ±  2%      -3.3        3.08 ±  2%  perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
     13.58            -3.2       10.34        perf-profile.calltrace.cycles-pp.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
     47.96            -3.2       44.73        perf-profile.calltrace.cycles-pp.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.22            -3.2        9.00        perf-profile.calltrace.cycles-pp.tcp_build_frag.do_tcp_sendpages.tcp_sendpage.inet_sendpage.kernel_sendpage
     12.19            -3.2        9.02        perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
     18.68            -3.0       15.70        perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
      5.02 ±  3%      -2.7        2.34 ±  2%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      5.29            -2.6        2.66 ± 10%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
     49.55            -2.5       47.06        perf-profile.calltrace.cycles-pp.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendfile
      6.67 ±  2%      -2.5        4.22        perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      4.50 ±  3%      -2.5        2.05        perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      4.62 ±  2%      -2.4        2.24        perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      4.24            -2.1        2.13 ±  3%  perf-profile.calltrace.cycles-pp.drain_stock.refill_stock.__sk_mem_reduce_allocated.tcp_recvmsg_locked.tcp_recvmsg
      4.23            -2.1        2.12 ±  3%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.drain_stock.refill_stock.__sk_mem_reduce_allocated.tcp_recvmsg_locked
      4.32            -2.1        2.22 ±  3%  perf-profile.calltrace.cycles-pp.refill_stock.__sk_mem_reduce_allocated.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
     50.40            -2.1       48.32        perf-profile.calltrace.cycles-pp.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendfile.sendfile_tcp_stream
      4.54            -2.1        2.48 ±  2%  perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      2.09 ±  4%      -1.8        0.28 ±100%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog
      9.23            -1.8        7.44        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
      2.18 ±  4%      -1.8        0.39 ± 71%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog.tcp_recvmsg_locked.tcp_recvmsg
      2.15 ±  4%      -1.8        0.38 ± 71%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog.tcp_recvmsg_locked
      2.21 ±  3%      -1.7        0.48 ± 45%  perf-profile.calltrace.cycles-pp.__release_sock.__sk_flush_backlog.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      2.23 ±  4%      -1.7        0.57 ± 10%  perf-profile.calltrace.cycles-pp.__sk_flush_backlog.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      7.20 ±  2%      -1.5        5.66        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage
      7.20 ±  2%      -1.5        5.68        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage.inet_sendpage
     11.57            -1.5       10.08        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     11.64            -1.5       10.17        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      9.93            -1.4        8.55 ±  2%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      3.76            -1.0        2.80        perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      3.94            -0.9        3.06        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg
      3.99            -0.8        3.16        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg
      6.84 ±  2%      -0.8        6.01        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
     15.02            -0.8       14.20        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      6.85 ±  2%      -0.8        6.03        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
     15.03            -0.8       14.22        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      4.05            -0.8        3.27 ±  2%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
     13.48            -0.8       12.70        perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      4.15            -0.8        3.40 ±  2%  perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      3.92            -0.8        3.17 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage
      3.70 ±  2%      -0.7        2.96 ±  2%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages
     13.60            -0.7       12.93        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
     13.64            -0.7       12.98        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     15.55            -0.6       14.90        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.04            -0.5        2.50        perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
     14.03            -0.4       13.58        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
     14.32            -0.4       13.90        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
     14.34            -0.4       13.94        perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
      3.22            -0.4        2.87        perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      5.12 ±  2%      -0.3        4.79        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage.inet_sendpage
      5.18 ±  2%      -0.3        4.86        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage.inet_sendpage.kernel_sendpage
      1.64 ±  5%      -0.2        1.47        perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
      1.66 ±  5%      -0.2        1.51        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      0.96            -0.2        0.81        perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_release.__splice_from_pipe.generic_splice_sendpage.direct_splice_actor.splice_direct_to_actor
      1.20            -0.1        1.07        perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
      0.70            +0.1        0.75 ±  3%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.85            +0.1        0.93 ±  2%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked
      1.03 ±  2%      +0.1        1.18        perf-profile.calltrace.cycles-pp.copy_page_to_iter.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
      0.53 ±  3%      +0.2        0.72 ±  3%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_data_queue
      0.60 ±  2%      +0.2        0.79 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established
      0.68 ±  2%      +0.2        0.90 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
      0.75 ±  2%      +0.2        0.98 ±  3%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.58 ±  5%      +0.3        0.83        perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      0.83            +0.3        1.09        perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_build_frag.do_tcp_sendpages.tcp_sendpage.inet_sendpage
      0.69 ±  2%      +0.4        1.06 ±  3%  perf-profile.calltrace.cycles-pp.tcp_send_mss.do_tcp_sendpages.tcp_sendpage.inet_sendpage.kernel_sendpage
      0.83 ±  3%      +0.4        1.20        perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      5.26 ±  2%      +0.4        5.65        perf-profile.calltrace.cycles-pp.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile
      0.45 ± 44%      +0.4        0.85 ±  4%  perf-profile.calltrace.cycles-pp.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage.inet_sendpage
      0.00            +0.5        0.51        perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry
      0.00            +0.5        0.52 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_send_ack.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      0.00            +0.5        0.54 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      0.00            +0.6        0.56 ±  2%  perf-profile.calltrace.cycles-pp.__inet_lookup_established.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      4.01 ±  2%      +0.6        4.59        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage
      5.78 ±  2%      +0.6        6.37        perf-profile.calltrace.cycles-pp.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      0.00            +0.6        0.59        perf-profile.calltrace.cycles-pp.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.60 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken
      0.00            +0.6        0.62 ±  2%  perf-profile.calltrace.cycles-pp.__alloc_skb.tcp_stream_alloc_skb.tcp_build_frag.do_tcp_sendpages.tcp_sendpage
      0.00            +0.7        0.66 ±  2%  perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      4.58 ±  2%      +0.7        5.26        perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage
      1.15 ±  4%      +0.8        1.93        perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      4.89 ±  2%      +0.8        5.72        perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
     28.74            +0.9       29.60        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      1.48 ±  4%      +0.9        2.34        perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
     28.79            +0.9       29.67        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.55 ±  4%      +0.9        2.45        perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      1.08 ±  2%      +0.9        2.01        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      1.08 ±  2%      +0.9        2.02 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
     29.00            +1.1       30.06        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      2.79 ±  5%      +1.1        3.86 ±  2%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
     29.03            +1.1       30.11        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.process_requests
      2.88 ±  5%      +1.1        4.01 ±  2%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
     29.15            +1.1       30.28        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.process_requests.spawn_child
     29.21            +1.2       30.40        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.process_requests.spawn_child.accept_connection
      3.36 ±  5%      +1.3        4.70 ±  2%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
     29.48            +1.4       30.86        perf-profile.calltrace.cycles-pp.recv.process_requests.spawn_child.accept_connection.accept_connections
     24.29            +1.6       25.94        perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      7.31            +1.8        9.11        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      7.22            +2.0        9.18        perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg
      6.62            +2.0        8.60        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked
      7.34            +2.0        9.35        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      5.54 ±  4%      +2.6        8.12        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      5.58 ±  4%      +2.6        8.17        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
     19.54           -10.9        8.60 ±  3%  perf-profile.children.cycles-pp.try_charge_memcg
     20.03           -10.7        9.30 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
     13.31            -7.5        5.78 ±  3%  perf-profile.children.cycles-pp.page_counter_try_charge
     18.63            -5.6       13.01        perf-profile.children.cycles-pp.__release_sock
     25.18            -5.6       19.58        perf-profile.children.cycles-pp.tcp_rcv_established
     25.33            -5.5       19.79        perf-profile.children.cycles-pp.tcp_v4_do_rcv
     10.43            -5.5        4.92 ±  3%  perf-profile.children.cycles-pp.drain_stock
     10.62            -5.5        5.12 ±  3%  perf-profile.children.cycles-pp.refill_stock
     10.39            -5.5        4.90 ±  3%  perf-profile.children.cycles-pp.page_counter_uncharge
     11.00            -5.4        5.60 ±  2%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
     12.69            -5.1        7.63 ±  2%  perf-profile.children.cycles-pp.tcp_data_queue
     38.12            -5.0       33.12        perf-profile.children.cycles-pp.tcp_sendpage
     38.44            -4.8       33.62        perf-profile.children.cycles-pp.inet_sendpage
     38.59            -4.8       33.83        perf-profile.children.cycles-pp.kernel_sendpage
     38.69            -4.7       33.97        perf-profile.children.cycles-pp.sock_sendpage
     38.88            -4.7       34.20        perf-profile.children.cycles-pp.pipe_to_sendpage
     40.43            -4.6       35.84        perf-profile.children.cycles-pp.__splice_from_pipe
     40.62            -4.5       36.11        perf-profile.children.cycles-pp.generic_splice_sendpage
     40.71            -4.5       36.24        perf-profile.children.cycles-pp.direct_splice_actor
     18.00            -4.0       13.99        perf-profile.children.cycles-pp.release_sock
     10.68            -4.0        6.68 ±  2%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
     10.73            -4.0        6.75 ±  2%  perf-profile.children.cycles-pp.__sk_mem_schedule
      9.94            -3.6        6.31        perf-profile.children.cycles-pp.tcp_wmem_schedule
     47.63            -3.5       44.10        perf-profile.children.cycles-pp.splice_direct_to_actor
     48.05            -3.3       44.75        perf-profile.children.cycles-pp.do_splice_direct
     12.36            -3.2        9.16        perf-profile.children.cycles-pp.tcp_build_frag
      7.64            -3.0        4.61        perf-profile.children.cycles-pp.tcp_clean_rtx_queue
     18.90            -2.9       15.99        perf-profile.children.cycles-pp.do_tcp_sendpages
      8.16            -2.6        5.52        perf-profile.children.cycles-pp.tcp_ack
     49.65            -2.6       47.09        perf-profile.children.cycles-pp.do_sendfile
     50.53            -2.1       48.39        perf-profile.children.cycles-pp.__x64_sys_sendfile64
      2.23 ±  4%      -1.7        0.57 ± 10%  perf-profile.children.cycles-pp.__sk_flush_backlog
     51.91            -1.6       50.34        perf-profile.children.cycles-pp.sendfile
     52.55            -1.3       51.22        perf-profile.children.cycles-pp.sendfile_tcp_stream
     14.86            -1.1       13.75        perf-profile.children.cycles-pp.mwait_idle_with_hints
      2.35 ±  5%      -1.1        1.30 ±  4%  perf-profile.children.cycles-pp.propagate_protected_usage
      1.48 ± 12%      -0.9        0.57 ± 40%  perf-profile.children.cycles-pp.intel_idle
     80.10            -0.9       79.24        perf-profile.children.cycles-pp.do_syscall_64
     15.17            -0.8       14.33        perf-profile.children.cycles-pp.cpuidle_enter
     15.16            -0.8       14.32        perf-profile.children.cycles-pp.cpuidle_enter_state
     13.64            -0.8       12.84        perf-profile.children.cycles-pp.tcp_v4_rcv
     80.40            -0.7       79.70        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     13.77            -0.7       13.07        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     13.80            -0.7       13.12        perf-profile.children.cycles-pp.ip_local_deliver_finish
     15.69            -0.7       15.02        perf-profile.children.cycles-pp.cpuidle_idle_call
     14.19            -0.5       13.72        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     14.49            -0.4       14.05        perf-profile.children.cycles-pp.process_backlog
     14.50            -0.4       14.08        perf-profile.children.cycles-pp.__napi_poll
      0.64 ± 11%      -0.4        0.28 ±  8%  perf-profile.children.cycles-pp.newidle_balance
      0.50 ± 11%      -0.3        0.20 ± 35%  perf-profile.children.cycles-pp.tcp_try_rmem_schedule
      0.77 ±  9%      -0.3        0.50 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.37 ± 12%      -0.3        0.11 ± 11%  perf-profile.children.cycles-pp.load_balance
     13.27            -0.3       13.02        perf-profile.children.cycles-pp.tcp_write_xmit
      0.32 ± 12%      -0.2        0.08 ± 11%  perf-profile.children.cycles-pp.find_busiest_group
     13.33            -0.2       13.09        perf-profile.children.cycles-pp.__tcp_push_pending_frames
      0.31 ± 13%      -0.2        0.08 ± 10%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.27 ± 13%      -0.2        0.07 ± 10%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.98            -0.2        0.82 ±  2%  perf-profile.children.cycles-pp.page_cache_pipe_buf_release
      1.67 ±  5%      -0.2        1.52        perf-profile.children.cycles-pp.schedule
      2.34 ±  3%      -0.1        2.21        perf-profile.children.cycles-pp.__schedule
      1.03 ±  4%      -0.1        0.92 ±  3%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.28            -0.1        0.20 ±  2%  perf-profile.children.cycles-pp.tcp_queue_rcv
      0.49 ±  2%      -0.1        0.43 ±  2%  perf-profile.children.cycles-pp.skb_attempt_defer_free
      0.11 ±  3%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.send_call_function_single_ipi
      0.08 ± 11%      -0.0        0.05        perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.14 ±  8%      -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.finish_task_switch
      0.09 ±  4%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.14 ±  4%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.__sock_wfree
      0.09 ±  5%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.sk_free
      0.10 ±  3%      -0.0        0.08 ±  4%  perf-profile.children.cycles-pp.add_wait_queue
      0.06            +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.rebalance_domains
      0.23 ±  2%      +0.0        0.24        perf-profile.children.cycles-pp.sock_rfree
      0.10            +0.0        0.12 ±  5%  perf-profile.children.cycles-pp.xas_start
      0.10 ±  8%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.05            +0.0        0.07        perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.07 ±  6%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.11 ±  3%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.07 ±  5%      +0.0        0.09        perf-profile.children.cycles-pp.demo_stream_interval
      0.13 ±  3%      +0.0        0.15 ±  2%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.05            +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.05            +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.folio_mark_accessed
      0.11 ±  4%      +0.0        0.13 ±  2%  perf-profile.children.cycles-pp.sanity
      0.09 ±  5%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.05            +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.check_stack_object
      0.05            +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.splice_from_pipe_next
      0.04 ± 44%      +0.0        0.07        perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.11 ±  4%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp.update_rq_clock
      0.07            +0.0        0.10        perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.36            +0.0        0.39        perf-profile.children.cycles-pp.kfree_skbmem
      0.07 ±  5%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.__get_task_ioprio
      0.07            +0.0        0.10 ±  3%  perf-profile.children.cycles-pp.rw_verify_area
      0.06 ±  6%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.__tcp_select_window
      0.06 ±  6%      +0.0        0.09        perf-profile.children.cycles-pp.__ip_finish_output
      0.06 ±  6%      +0.0        0.09        perf-profile.children.cycles-pp._raw_spin_unlock_bh
      0.14 ±  5%      +0.0        0.17 ±  3%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.15 ±  3%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.14 ±  3%      +0.0        0.17 ±  4%  perf-profile.children.cycles-pp.clockevents_program_event
      0.09            +0.0        0.13 ±  2%  perf-profile.children.cycles-pp.fsnotify_perm
      0.10 ±  7%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp.page_cache_pipe_buf_confirm
      0.06            +0.0        0.10 ±  3%  perf-profile.children.cycles-pp.kfree
      0.05 ±  7%      +0.0        0.09 ±  6%  perf-profile.children.cycles-pp.tcp_update_skb_after_send
      0.08 ±  4%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.___perf_sw_event
      0.05            +0.0        0.09 ±  6%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.04 ± 44%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.perf_tp_event
      0.03 ± 70%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.aa_sk_perm
      0.07 ±  7%      +0.0        0.11 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.26 ±  3%      +0.0        0.30 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.20 ±  5%      +0.0        0.24 ±  4%  perf-profile.children.cycles-pp.tick_sched_handle
      0.20 ±  5%      +0.0        0.24 ±  4%  perf-profile.children.cycles-pp.update_process_times
      0.07 ± 10%      +0.0        0.12 ±  9%  perf-profile.children.cycles-pp.ipv4_dst_check
      0.05 ±  8%      +0.0        0.10 ± 16%  perf-profile.children.cycles-pp.inet_send_prepare
      0.14 ±  2%      +0.0        0.18 ±  3%  perf-profile.children.cycles-pp.__smp_call_single_queue
      0.14 ±  2%      +0.0        0.18 ±  3%  perf-profile.children.cycles-pp.llist_add_batch
      0.08            +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.tcp_release_cb
      0.22 ±  3%      +0.0        0.27 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
      0.02 ± 99%      +0.0        0.07        perf-profile.children.cycles-pp.tcp_skb_entail
      0.11 ±  5%      +0.0        0.16 ±  3%  perf-profile.children.cycles-pp.copy_user_short_string
      0.16 ±  3%      +0.0        0.20 ±  3%  perf-profile.children.cycles-pp.xas_load
      0.06 ±  8%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.sock_recvmsg
      0.06 ±  6%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.tcp_rtt_estimator
      0.06 ±  7%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.ip_output
      0.21 ±  2%      +0.0        0.26        perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.07 ±  6%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.11 ±  4%      +0.0        0.16 ±  2%  perf-profile.children.cycles-pp.remove_wait_queue
      0.05 ±  7%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.tcp_mtu_probe
      0.26            +0.0        0.31        perf-profile.children.cycles-pp.__mod_timer
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.sk_forced_mem_schedule
      0.00            +0.1        0.05        perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.tcp_ack_update_window
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.tcp_rate_skb_delivered
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.__list_add_valid
      0.08 ±  4%      +0.1        0.13        perf-profile.children.cycles-pp.ip_rcv_core
      0.08 ±  6%      +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.ip_send_check
      0.08            +0.1        0.13 ±  2%  perf-profile.children.cycles-pp.llist_reverse_order
      0.10 ±  4%      +0.1        0.16 ±  3%  perf-profile.children.cycles-pp.rcu_all_qs
      0.08 ±  5%      +0.1        0.14 ±  4%  perf-profile.children.cycles-pp.skb_release_head_state
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ct_idle_exit
      0.08 ±  4%      +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.71            +0.1        0.76 ±  2%  perf-profile.children.cycles-pp.schedule_idle
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.eth_type_trans
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.demo_interval_tick
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.__xfrm_policy_check2
      0.44            +0.1        0.50 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.32            +0.1        0.38 ±  3%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.28 ±  2%      +0.1        0.34 ±  2%  perf-profile.children.cycles-pp.sk_reset_timer
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.rb_erase
      0.12 ±  8%      +0.1        0.17 ±  2%  perf-profile.children.cycles-pp.select_idle_sibling
      0.10 ±  5%      +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.recv_tcp_stream
      0.08 ±  5%      +0.1        0.14 ±  4%  perf-profile.children.cycles-pp.available_idle_cpu
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.ip_skb_dst_mtu
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.01 ±223%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.45 ±  2%      +0.1        0.51 ±  3%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.20            +0.1        0.26        perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.rb_first
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.reweight_entity
      0.13            +0.1        0.19 ±  4%  perf-profile.children.cycles-pp.__switch_to_asm
      0.08 ±  4%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.__sk_dst_check
      0.07            +0.1        0.13 ±  2%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.17 ±  3%      +0.1        0.23 ±  5%  perf-profile.children.cycles-pp.prepare_task_switch
      0.07 ±  7%      +0.1        0.13 ±  4%  perf-profile.children.cycles-pp.raw_v4_input
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.tcp_v4_send_check
      0.26 ±  4%      +0.1        0.32 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.56            +0.1        0.62 ±  2%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.10            +0.1        0.17 ±  2%  perf-profile.children.cycles-pp.__ip_local_out
      0.06 ±  9%      +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.skb_clone
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.12 ±  3%      +0.1        0.19 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.11 ±  3%      +0.1        0.18 ± 12%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.08            +0.1        0.15        perf-profile.children.cycles-pp.set_next_entity
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.os_xsave
      0.18 ±  2%      +0.1        0.25        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.15 ±  2%      +0.1        0.22 ±  2%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.11 ±  3%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp.ip_local_out
      0.17 ±  2%      +0.1        0.24 ±  2%  perf-profile.children.cycles-pp.aa_file_perm
      0.15 ±  6%      +0.1        0.23 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.01 ±223%      +0.1        0.08 ±  4%  perf-profile.children.cycles-pp.__build_skb_around
      0.10 ±  3%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp.kmem_cache_free
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.14 ±  8%      +0.1        0.21 ±  5%  perf-profile.children.cycles-pp.tcp_established_options
      0.17 ±  3%      +0.1        0.24 ±  2%  perf-profile.children.cycles-pp.__put_user_nocheck_8
      0.21 ±  2%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.11            +0.1        0.18 ±  5%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.22            +0.1        0.30 ±  2%  perf-profile.children.cycles-pp.tcp_add_backlog
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.inet_ehashfn
      0.24 ±  2%      +0.1        0.32 ±  2%  perf-profile.children.cycles-pp.sock_put
      0.13 ±  3%      +0.1        0.20 ±  2%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.14 ±  2%      +0.1        0.22 ±  2%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.24 ±  2%      +0.1        0.33        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.08 ±  6%      +0.1        0.16 ±  4%  perf-profile.children.cycles-pp.__copy_skb_header
      0.18 ±  3%      +0.1        0.27        perf-profile.children.cycles-pp.native_sched_clock
      0.27 ±  5%      +0.1        0.36 ±  2%  perf-profile.children.cycles-pp.update_load_avg
      0.18 ±  2%      +0.1        0.27 ±  2%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.16 ±  4%      +0.1        0.26        perf-profile.children.cycles-pp.update_curr
      0.20 ±  4%      +0.1        0.29 ±  2%  perf-profile.children.cycles-pp.select_task_rq
      0.18 ±  2%      +0.1        0.27 ±  2%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.20 ±  2%      +0.1        0.30 ±  3%  perf-profile.children.cycles-pp.ip_rcv
      0.13 ±  3%      +0.1        0.23 ±  3%  perf-profile.children.cycles-pp.__ksize
      0.26            +0.1        0.37 ±  2%  perf-profile.children.cycles-pp.current_time
      0.30 ±  3%      +0.1        0.40        perf-profile.children.cycles-pp.netperf_sendfile
      0.21 ±  3%      +0.1        0.32        perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.12 ±  8%      +0.1        0.23 ±  4%  perf-profile.children.cycles-pp.tcp_rate_check_app_limited
      0.72            +0.1        0.84        perf-profile.children.cycles-pp.__kfree_skb
      0.40 ±  8%      +0.1        0.52        perf-profile.children.cycles-pp.enqueue_task_fair
      0.24 ±  3%      +0.1        0.36 ±  2%  perf-profile.children.cycles-pp.__cond_resched
      0.26 ±  3%      +0.1        0.38 ±  3%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.63            +0.1        0.75        perf-profile.children.cycles-pp.napi_consume_skb
      0.41 ±  7%      +0.1        0.54        perf-profile.children.cycles-pp.ttwu_do_activate
      0.35            +0.1        0.48 ±  2%  perf-profile.children.cycles-pp.dst_release
      0.27 ±  7%      +0.1        0.40 ±  2%  perf-profile.children.cycles-pp.enqueue_entity
      0.26 ±  2%      +0.1        0.39 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.18 ±  3%      +0.1        0.31 ±  3%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.23 ±  3%      +0.1        0.36 ±  2%  perf-profile.children.cycles-pp.tcp_tso_segs
      1.06            +0.1        1.20        perf-profile.children.cycles-pp.copy_page_to_iter
      0.31 ±  2%      +0.1        0.46 ±  2%  perf-profile.children.cycles-pp.__skb_clone
      0.24 ±  3%      +0.1        0.39 ±  3%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.32 ±  2%      +0.1        0.46 ±  2%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.19 ±  3%      +0.1        0.34 ±  2%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.37            +0.2        0.52 ±  3%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.34            +0.2        0.50 ±  2%  perf-profile.children.cycles-pp.menu_select
      0.24 ±  6%      +0.2        0.40 ±  8%  perf-profile.children.cycles-pp.ipv4_mtu
      0.35 ±  3%      +0.2        0.51        perf-profile.children.cycles-pp._copy_from_user
      0.38 ±  6%      +0.2        0.54 ±  2%  perf-profile.children.cycles-pp.dequeue_entity
      0.37            +0.2        0.54        perf-profile.children.cycles-pp.__entry_text_start
      1.81            +0.2        1.98        perf-profile.children.cycles-pp.__dev_queue_xmit
      1.23            +0.2        1.40        perf-profile.children.cycles-pp.skb_release_data
      0.30 ±  2%      +0.2        0.48 ±  2%  perf-profile.children.cycles-pp.read_tsc
      0.42 ±  5%      +0.2        0.60 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.32 ±  2%      +0.2        0.51        perf-profile.children.cycles-pp.__fdget
      0.48 ±  2%      +0.2        0.67 ±  2%  perf-profile.children.cycles-pp.do_splice_to
      0.07 ±  7%      +0.2        0.26 ±  2%  perf-profile.children.cycles-pp.poll_idle
      0.46 ±  2%      +0.2        0.67 ±  2%  perf-profile.children.cycles-pp.__might_fault
      0.55            +0.2        0.77        perf-profile.children.cycles-pp.atime_needs_update
      0.30 ±  4%      +0.2        0.52 ±  2%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.63 ±  5%      +0.2        0.86        perf-profile.children.cycles-pp.sched_ttwu_pending
      0.85            +0.3        1.11        perf-profile.children.cycles-pp.tcp_stream_alloc_skb
      0.78 ±  2%      +0.3        1.05 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
     16.22            +0.3       16.49        perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.88            +0.3        1.16 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      0.56 ±  2%      +0.3        0.84 ±  2%  perf-profile.children.cycles-pp.__inet_lookup_established
      1.21            +0.3        1.50 ±  2%  perf-profile.children.cycles-pp.sock_def_readable
      0.69            +0.3        0.99        perf-profile.children.cycles-pp.touch_atime
      0.63            +0.3        0.93        perf-profile.children.cycles-pp.ktime_get
     17.70            +0.3       18.01        perf-profile.children.cycles-pp.ip_finish_output2
      1.00            +0.3        1.31 ±  2%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.64            +0.3        0.96        perf-profile.children.cycles-pp.__might_resched
      0.60 ±  4%      +0.3        0.93 ±  3%  perf-profile.children.cycles-pp.tcp_current_mss
      6.54            +0.4        6.90        perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.72 ±  3%      +0.4        1.10 ±  2%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.84 ±  3%      +0.4        1.22        perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.71 ±  3%      +0.4        1.09 ±  2%  perf-profile.children.cycles-pp.tcp_send_mss
      5.39            +0.4        5.78        perf-profile.children.cycles-pp.filemap_read
      4.42            +0.4        4.84        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.83 ±  2%      +0.4        1.26        perf-profile.children.cycles-pp.security_file_permission
      0.65 ±  2%      +0.5        1.10        perf-profile.children.cycles-pp.__alloc_skb
     18.09            +0.5       18.62        perf-profile.children.cycles-pp.__ip_queue_xmit
      5.89            +0.6        6.48        perf-profile.children.cycles-pp.generic_file_splice_read
      5.24            +0.8        6.01        perf-profile.children.cycles-pp.lock_sock_nested
     18.89            +0.8       19.67        perf-profile.children.cycles-pp.__tcp_transmit_skb
      1.17 ±  3%      +0.8        1.96        perf-profile.children.cycles-pp.check_heap_object
     28.76            +0.9       29.61        perf-profile.children.cycles-pp.tcp_recvmsg
     28.79            +0.9       29.67        perf-profile.children.cycles-pp.inet_recvmsg
      1.52 ±  4%      +0.9        2.40        perf-profile.children.cycles-pp.__check_object_size
      1.57 ±  4%      +0.9        2.47        perf-profile.children.cycles-pp.simple_copy_to_iter
     29.00            +1.1       30.06        perf-profile.children.cycles-pp.__sys_recvfrom
     29.04            +1.1       30.12        perf-profile.children.cycles-pp.__x64_sys_recvfrom
      2.89 ±  5%      +1.1        4.02 ±  2%  perf-profile.children.cycles-pp.copyout
      2.92 ±  5%      +1.1        4.06 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      3.37 ±  5%      +1.3        4.71 ±  2%  perf-profile.children.cycles-pp._copy_to_iter
     29.54            +1.4       30.94        perf-profile.children.cycles-pp.recv
     29.61            +1.5       31.07        perf-profile.children.cycles-pp.accept_connections
     29.61            +1.5       31.07        perf-profile.children.cycles-pp.accept_connection
     29.61            +1.5       31.07        perf-profile.children.cycles-pp.spawn_child
     29.61            +1.5       31.07        perf-profile.children.cycles-pp.process_requests
     24.30            +1.6       25.95        perf-profile.children.cycles-pp.tcp_recvmsg_locked
      5.56 ±  4%      +2.6        8.14        perf-profile.children.cycles-pp.__skb_datagram_iter
      5.58 ±  4%      +2.6        8.17        perf-profile.children.cycles-pp.skb_copy_datagram_iter
     12.05            -6.9        5.12 ±  3%  perf-profile.self.cycles-pp.page_counter_try_charge
      9.22            -5.0        4.22 ±  3%  perf-profile.self.cycles-pp.page_counter_uncharge
      6.17 ±  2%      -3.4        2.78 ±  2%  perf-profile.self.cycles-pp.try_charge_memcg
     14.75            -1.1       13.65        perf-profile.self.cycles-pp.mwait_idle_with_hints
      2.34 ±  5%      -1.0        1.30 ±  4%  perf-profile.self.cycles-pp.propagate_protected_usage
      0.22 ± 14%      -0.2        0.06 ± 13%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.96            -0.2        0.81 ±  2%  perf-profile.self.cycles-pp.page_cache_pipe_buf_release
      0.38 ±  2%      -0.1        0.30 ±  2%  perf-profile.self.cycles-pp.__schedule
      2.12            -0.1        2.04        perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.28            -0.1        0.20        perf-profile.self.cycles-pp.tcp_queue_rcv
      0.11 ±  3%      -0.0        0.07 ±  8%  perf-profile.self.cycles-pp.send_call_function_single_ipi
      0.14 ±  4%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.__sock_wfree
      0.08 ±  5%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.09 ±  5%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.sk_free
      0.21 ±  2%      -0.0        0.18 ±  2%  perf-profile.self.cycles-pp.sock_def_readable
      0.12 ±  3%      -0.0        0.11 ±  3%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.05            +0.0        0.06        perf-profile.self.cycles-pp.update_rq_clock
      0.09 ±  5%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.xas_start
      0.13 ±  3%      +0.0        0.15 ±  2%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.11 ±  6%      +0.0        0.13 ±  2%  perf-profile.self.cycles-pp.do_splice_to
      0.10 ±  3%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.sanity
      0.09 ±  4%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.07 ±  5%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.tcp_data_queue
      0.07 ±  6%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.05 ±  8%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.06 ±  6%      +0.0        0.08        perf-profile.self.cycles-pp.simple_copy_to_iter
      0.10 ±  4%      +0.0        0.12 ±  7%  perf-profile.self.cycles-pp.prepare_task_switch
      0.06            +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.rw_verify_area
      0.11 ±  6%      +0.0        0.14 ±  4%  perf-profile.self.cycles-pp.update_load_avg
      0.05 ±  8%      +0.0        0.08        perf-profile.self.cycles-pp.ip_rcv
      0.07 ±  7%      +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.__get_task_ioprio
      0.06 ±  7%      +0.0        0.09        perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.28 ±  3%      +0.0        0.31 ±  4%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.05 ±  8%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.11 ±  4%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp.__release_sock
      0.08 ±  8%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.06            +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.05 ±  7%      +0.0        0.08        perf-profile.self.cycles-pp._raw_spin_unlock_bh
      0.06 ±  6%      +0.0        0.09        perf-profile.self.cycles-pp.xas_load
      0.06 ±  8%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp._copy_from_user
      0.05            +0.0        0.08        perf-profile.self.cycles-pp.__tcp_push_pending_frames
      0.06 ±  6%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.__tcp_select_window
      0.06 ±  8%      +0.0        0.09 ±  8%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.07            +0.0        0.10        perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.13 ±  3%      +0.0        0.16 ±  2%  perf-profile.self.cycles-pp.refill_stock
      0.05            +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.09 ±  7%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.page_cache_pipe_buf_confirm
      0.07 ±  5%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.___perf_sw_event
      0.03 ± 70%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.folio_mark_accessed
      0.03 ± 70%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.splice_from_pipe_next
      0.35            +0.0        0.39        perf-profile.self.cycles-pp.kfree_skbmem
      0.08 ±  6%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.rcu_all_qs
      0.06 ±  7%      +0.0        0.10 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.03 ± 70%      +0.0        0.07        perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.19 ±  2%      +0.0        0.22 ±  2%  perf-profile.self.cycles-pp.pipe_to_sendpage
      0.06 ±  6%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.tcp_rcv_space_adjust
      0.06 ±  6%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.process_backlog
      0.14 ±  3%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.filemap_get_pages
      0.03 ± 70%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.09 ±  4%      +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.fsnotify_perm
      0.06 ±  6%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.kfree
      0.09 ±  4%      +0.0        0.13 ±  2%  perf-profile.self.cycles-pp.__mod_timer
      0.02 ± 99%      +0.0        0.06 ±  7%  perf-profile.self.cycles-pp.copyout
      0.16 ± 15%      +0.0        0.20 ± 11%  perf-profile.self.cycles-pp.update_cfs_group
      0.05            +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.tcp_mtu_probe
      0.08 ±  6%      +0.0        0.12        perf-profile.self.cycles-pp.do_softirq
      0.08 ±  4%      +0.0        0.12 ±  5%  perf-profile.self.cycles-pp.tcp_release_cb
      0.14 ±  2%      +0.0        0.18 ±  3%  perf-profile.self.cycles-pp.llist_add_batch
      0.10 ±  4%      +0.0        0.15 ±  4%  perf-profile.self.cycles-pp.copy_user_short_string
      0.08 ±  5%      +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.direct_splice_actor
      0.11 ±  3%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.generic_file_splice_read
      0.07 ±  6%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.lock_sock_nested
      0.06 ±  9%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.10 ±  3%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.sock_sendpage
      0.07 ±  8%      +0.0        0.12 ± 10%  perf-profile.self.cycles-pp.ipv4_dst_check
      0.02 ± 99%      +0.0        0.07        perf-profile.self.cycles-pp.irqtime_account_irq
      0.05 ±  8%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.tcp_rtt_estimator
      0.06 ±  6%      +0.0        0.11 ±  3%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.07 ± 11%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.tcp_event_new_data_sent
      0.08 ±  4%      +0.0        0.13 ±  4%  perf-profile.self.cycles-pp.recv_tcp_stream
      0.02 ±141%      +0.0        0.06 ±  7%  perf-profile.self.cycles-pp.__sk_mem_schedule
      0.06 ±  7%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.skb_attempt_defer_free
      0.07 ±  5%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.enqueue_entity
      0.08 ±  6%      +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.recv
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.demo_stream_interval
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.security_sock_rcv_skb
      0.00            +0.1        0.05        perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.demo_interval_tick
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.05        perf-profile.self.cycles-pp._raw_spin_trylock
      0.07 ±  6%      +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.ip_send_check
      0.12 ±  4%      +0.1        0.17 ±  2%  perf-profile.self.cycles-pp.security_file_permission
      0.09 ±  5%      +0.1        0.14 ±  2%  perf-profile.self.cycles-pp.loopback_xmit
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.__wake_up_common
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.tcp_rearm_rto
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.__xfrm_policy_check2
      0.06 ±  8%      +0.1        0.11 ±  6%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.11 ±  4%      +0.1        0.16 ±  6%  perf-profile.self.cycles-pp.tcp_wmem_schedule
      0.08 ±  6%      +0.1        0.13 ±  2%  perf-profile.self.cycles-pp.ip_rcv_core
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.rb_first
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.napi_consume_skb
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.dequeue_entity
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.mem_cgroup_uncharge_skmem
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.tcp_skb_entail
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.ip_skb_dst_mtu
      0.08 ±  5%      +0.1        0.14 ±  2%  perf-profile.self.cycles-pp.update_curr
      0.08 ±  5%      +0.1        0.14 ±  2%  perf-profile.self.cycles-pp.available_idle_cpu
      0.26            +0.1        0.32        perf-profile.self.cycles-pp.splice_direct_to_actor
      0.14 ±  5%      +0.1        0.19 ±  4%  perf-profile.self.cycles-pp.release_sock
      0.14 ±  2%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.tcp_send_mss
      0.11 ±  3%      +0.1        0.17 ±  2%  perf-profile.self.cycles-pp.__might_fault
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.perf_tp_event
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.inet_recvmsg
      0.01 ±223%      +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.check_stack_object
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.07 ±  6%      +0.1        0.13 ±  2%  perf-profile.self.cycles-pp.llist_reverse_order
      0.01 ±223%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.15 ±  3%      +0.1        0.21 ±  3%  perf-profile.self.cycles-pp.kernel_sendpage
      0.23 ±  2%      +0.1        0.29 ±  2%  perf-profile.self.cycles-pp.__skb_clone
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.reweight_entity
      0.13            +0.1        0.19 ±  4%  perf-profile.self.cycles-pp.__switch_to_asm
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_v4_send_check
      0.06 ±  7%      +0.1        0.13 ±  3%  perf-profile.self.cycles-pp.raw_v4_input
      0.08            +0.1        0.14 ±  3%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.01 ±223%      +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.tcp_add_backlog
      0.29 ±  9%      +0.1        0.36 ±  6%  perf-profile.self.cycles-pp.__check_object_size
      0.13 ±  2%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.__cond_resched
      0.02 ±141%      +0.1        0.08 ± 21%  perf-profile.self.cycles-pp.inet_send_prepare
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__tcp_send_ack
      0.01 ±223%      +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.tcp_stream_alloc_skb
      0.10 ±  4%      +0.1        0.17 ± 13%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.05 ±  7%      +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.skb_clone
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.select_task_rq
      0.15 ±  3%      +0.1        0.22 ±  4%  perf-profile.self.cycles-pp.aa_file_perm
      0.10 ±  3%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.kmem_cache_free
      0.00            +0.1        0.07 ±  8%  perf-profile.self.cycles-pp.ip_output
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.os_xsave
      0.12 ±  3%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.17 ±  3%      +0.1        0.24 ±  3%  perf-profile.self.cycles-pp.__entry_text_start
      0.17 ±  2%      +0.1        0.24 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.01 ±223%      +0.1        0.08 ±  7%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.inet_ehashfn
      0.00            +0.1        0.07 ±  9%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.13 ±  6%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.tcp_established_options
      0.17 ±  2%      +0.1        0.24 ±  3%  perf-profile.self.cycles-pp.current_time
      0.13 ±  4%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.touch_atime
      0.12 ±  4%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.11 ±  4%      +0.1        0.18 ±  3%  perf-profile.self.cycles-pp.menu_select
      0.11 ±  3%      +0.1        0.18 ±  2%  perf-profile.self.cycles-pp.__alloc_skb
      0.24 ±  2%      +0.1        0.32        perf-profile.self.cycles-pp.sock_put
      0.17 ±  2%      +0.1        0.24 ±  2%  perf-profile.self.cycles-pp.__put_user_nocheck_8
      0.00            +0.1        0.08 ±  4%  perf-profile.self.cycles-pp.__build_skb_around
      1.20            +0.1        1.28 ±  2%  perf-profile.self.cycles-pp.filemap_read
      0.60            +0.1        0.68        perf-profile.self.cycles-pp.__dev_queue_xmit
      0.17 ±  4%      +0.1        0.25 ±  2%  perf-profile.self.cycles-pp.generic_splice_sendpage
      0.14 ±  2%      +0.1        0.22 ±  3%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.26 ±  2%      +0.1        0.34 ±  4%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.17 ±  2%      +0.1        0.26 ±  3%  perf-profile.self.cycles-pp.wait_woken
      0.08 ±  6%      +0.1        0.16 ±  3%  perf-profile.self.cycles-pp.__copy_skb_header
      0.17 ±  3%      +0.1        0.26        perf-profile.self.cycles-pp.native_sched_clock
      0.20 ±  2%      +0.1        0.29 ±  2%  perf-profile.self.cycles-pp.do_splice_direct
      0.11 ±  3%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.12 ±  4%      +0.1        0.22 ±  3%  perf-profile.self.cycles-pp.__ksize
      0.28 ±  2%      +0.1        0.38 ±  2%  perf-profile.self.cycles-pp.netperf_sendfile
      0.20 ±  2%      +0.1        0.31 ±  3%  perf-profile.self.cycles-pp.__x64_sys_sendfile64
      0.27            +0.1        0.38        perf-profile.self.cycles-pp.sendfile
      0.22 ±  3%      +0.1        0.33 ±  2%  perf-profile.self.cycles-pp.tcp_sendpage
      0.22 ±  2%      +0.1        0.33        perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.20 ±  2%      +0.1        0.31        perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.12 ±  6%      +0.1        0.22 ±  4%  perf-profile.self.cycles-pp.tcp_rate_check_app_limited
      1.06            +0.1        1.17        perf-profile.self.cycles-pp.skb_release_data
      0.29 ±  2%      +0.1        0.41 ±  2%  perf-profile.self.cycles-pp.atime_needs_update
      0.22 ±  2%      +0.1        0.33 ±  2%  perf-profile.self.cycles-pp.do_syscall_64
      0.34            +0.1        0.46        perf-profile.self.cycles-pp.sendfile_tcp_stream
      0.94            +0.1        1.06        perf-profile.self.cycles-pp.copy_page_to_iter
      0.23 ±  3%      +0.1        0.35 ±  3%  perf-profile.self.cycles-pp.__might_sleep
      0.16 ±  4%      +0.1        0.28 ±  2%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.16 ±  6%      +0.1        0.28 ±  2%  perf-profile.self.cycles-pp.ip_finish_output2
      0.21            +0.1        0.33 ±  2%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.19 ±  3%      +0.1        0.32 ±  3%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
      1.48            +0.1        1.60        perf-profile.self.cycles-pp.tcp_build_frag
      0.34 ±  2%      +0.1        0.47 ±  2%  perf-profile.self.cycles-pp.dst_release
      0.33 ±  3%      +0.1        0.46        perf-profile.self.cycles-pp.ktime_get
      0.23 ±  2%      +0.1        0.36        perf-profile.self.cycles-pp.net_rx_action
      0.23 ±  3%      +0.1        0.36        perf-profile.self.cycles-pp.tcp_tso_segs
      0.26 ±  4%      +0.1        0.39        perf-profile.self.cycles-pp.tcp_current_mss
      0.23 ±  3%      +0.1        0.37 ±  3%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.32 ±  4%      +0.1        0.46        perf-profile.self.cycles-pp._copy_to_iter
      0.32            +0.1        0.46        perf-profile.self.cycles-pp.__fsnotify_parent
      0.27 ±  2%      +0.1        0.42 ±  2%  perf-profile.self.cycles-pp.inet_sendpage
      0.35            +0.1        0.49        perf-profile.self.cycles-pp.__splice_from_pipe
      0.23 ±  6%      +0.2        0.38 ±  8%  perf-profile.self.cycles-pp.ipv4_mtu
      0.30 ±  2%      +0.2        0.46        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.28 ±  2%      +0.2        0.45        perf-profile.self.cycles-pp.tcp_rcv_established
      0.29 ±  2%      +0.2        0.46 ±  2%  perf-profile.self.cycles-pp.read_tsc
      0.28 ±  2%      +0.2        0.45 ±  2%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.32 ±  3%      +0.2        0.50        perf-profile.self.cycles-pp.__fdget
      0.05 ±  8%      +0.2        0.24        perf-profile.self.cycles-pp.poll_idle
      0.25 ±  3%      +0.2        0.44        perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.40 ±  3%      +0.2        0.61 ±  2%  perf-profile.self.cycles-pp.do_tcp_sendpages
      0.52 ±  2%      +0.2        0.77 ±  2%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.58 ±  2%      +0.2        0.83        perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.44 ±  4%      +0.3        0.70        perf-profile.self.cycles-pp.__skb_datagram_iter
      0.60            +0.3        0.86        perf-profile.self.cycles-pp.tcp_write_xmit
      0.35 ±  2%      +0.3        0.61 ±  2%  perf-profile.self.cycles-pp.tcp_ack
      0.62 ±  2%      +0.3        0.91        perf-profile.self.cycles-pp.do_sendfile
      0.63 ±  2%      +0.3        0.93        perf-profile.self.cycles-pp.__might_resched
      0.55 ±  3%      +0.3        0.86 ±  2%  perf-profile.self.cycles-pp.apparmor_file_permission
      4.39            +0.4        4.80        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.91 ±  4%      +0.6        1.54        perf-profile.self.cycles-pp.check_heap_object
      2.90 ±  5%      +1.1        4.02 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.06 ±  2%      +1.6        2.67        perf-profile.self.cycles-pp.__sk_mem_raise_allocated


***************************************************************************************************
lkp-cpl-4sp1: 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz (Cooper Lake) with 128G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/25%/debian-11.1-x86_64-20220510.cgz/900s/lkp-cpl-4sp1/TCP_MAERTS/netperf

commit: 
  408587baee ("mm: page_counter: rearrange struct page_counter fields")
  1813e51eec ("memcg: increase MEMCG_CHARGE_BATCH to 64")

408587baee397533 1813e51eece0ad6f4aacaeb738e 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     19695           +54.7%      30475        netperf.Throughput_Mbps
    709022           +54.7%    1097123        netperf.Throughput_total_Mbps
     86140           -12.1%      75736        netperf.time.involuntary_context_switches
    110235 ±  8%     +49.4%     164698 ±  9%  netperf.time.minor_page_faults
      1937           +10.7%       2145        netperf.time.percent_of_cpu_this_job_got
     17232           +10.3%      19014        netperf.time.system_time
    231.11           +40.4%     324.37        netperf.time.user_time
 9.775e+08           +44.8%  1.416e+09        netperf.time.voluntary_context_switches
 4.868e+09           +54.7%  7.533e+09        netperf.workload
 1.114e+09           +38.2%  1.539e+09        cpuidle..usage
      0.43            +0.2        0.67 ±  2%  mpstat.cpu.all.usr%
   3381208           +40.7%    4758877 ±  6%  vmstat.memory.cache
   2163732           +44.7%    3131855        vmstat.system.cs
    550183 ± 16%    +133.8%    1286322 ± 36%  numa-meminfo.node3.FilePages
    169695 ± 29%    +424.5%     890117 ± 39%  numa-meminfo.node3.Inactive
    169663 ± 29%    +424.6%     890057 ± 39%  numa-meminfo.node3.Inactive(anon)
    190385 ± 20%    +209.2%     588614 ± 34%  numa-meminfo.node3.Mapped
    983178 ±  9%     +73.7%    1708152 ± 27%  numa-meminfo.node3.MemUsed
    500584 ± 15%    +145.5%    1228735 ± 39%  numa-meminfo.node3.Shmem
 5.838e+08 ±  4%     +63.9%  9.566e+08 ±  4%  numa-numastat.node0.local_node
 5.839e+08 ±  4%     +63.9%  9.568e+08 ±  4%  numa-numastat.node0.numa_hit
 6.038e+08 ±  4%     +49.6%  9.034e+08 ±  8%  numa-numastat.node1.local_node
  6.04e+08 ±  4%     +49.6%  9.036e+08 ±  8%  numa-numastat.node1.numa_hit
 6.425e+08 ±  4%     +46.8%   9.43e+08 ±  7%  numa-numastat.node2.local_node
 6.426e+08 ±  4%     +46.8%  9.431e+08 ±  7%  numa-numastat.node2.numa_hit
 6.085e+08 ±  4%     +59.5%  9.706e+08 ±  6%  numa-numastat.node3.local_node
 6.086e+08 ±  4%     +59.5%  9.707e+08 ±  6%  numa-numastat.node3.numa_hit
      3299            -1.3%       3255        turbostat.Bzy_MHz
 9.775e+08           +38.7%  1.356e+09        turbostat.C1
      9.49            -2.3        7.14 ±  2%  turbostat.C1%
      0.07 ±  6%     +50.0%       0.11        turbostat.IPC
     14.52 ± 29%    +134.3      148.84 ± 16%  turbostat.PKG_%
   4275196         +1141.6%   53079630 ±  4%  turbostat.POLL
      0.05            +0.3        0.30 ±  3%  turbostat.POLL%
     58.67            +3.7%      60.83        turbostat.PkgTmp
    563.59            +4.2%     587.47        turbostat.PkgWatt
     10.71            -7.2%       9.94        turbostat.RAMWatt
    479946 ±  4%     +35.3%     649214 ±  5%  meminfo.Active
    479850 ±  4%     +35.3%     649124 ±  5%  meminfo.Active(anon)
   3253347           +42.3%    4630477 ±  6%  meminfo.Cached
   1091348 ±  2%    +128.2%    2490957 ± 12%  meminfo.Committed_AS
    434172 ±  3%    +280.2%    1650741 ± 18%  meminfo.Inactive
    433992 ±  3%    +280.3%    1650255 ± 18%  meminfo.Inactive(anon)
    287125 ±  5%    +220.9%     921428 ± 19%  meminfo.Mapped
   5021697           +28.5%    6451110 ±  5%  meminfo.Memused
      7538           +16.2%       8762 ±  4%  meminfo.PageTables
    592660 ±  5%    +232.3%    1969256 ± 15%  meminfo.Shmem
   9600917           +22.5%   11765453        meminfo.max_used_kB
 5.839e+08 ±  4%     +63.9%  9.568e+08 ±  4%  numa-vmstat.node0.numa_hit
 5.838e+08 ±  4%     +63.9%  9.566e+08 ±  4%  numa-vmstat.node0.numa_local
  6.04e+08 ±  4%     +49.6%  9.036e+08 ±  8%  numa-vmstat.node1.numa_hit
 6.038e+08 ±  4%     +49.6%  9.034e+08 ±  8%  numa-vmstat.node1.numa_local
 6.426e+08 ±  4%     +46.8%  9.431e+08 ±  7%  numa-vmstat.node2.numa_hit
 6.425e+08 ±  4%     +46.8%   9.43e+08 ±  7%  numa-vmstat.node2.numa_local
    137447 ± 16%    +134.1%     321703 ± 36%  numa-vmstat.node3.nr_file_pages
     42481 ± 29%    +422.7%     222054 ± 39%  numa-vmstat.node3.nr_inactive_anon
     47927 ± 20%    +205.7%     146502 ± 35%  numa-vmstat.node3.nr_mapped
    125048 ± 15%    +145.8%     307306 ± 39%  numa-vmstat.node3.nr_shmem
     42481 ± 29%    +422.7%     222053 ± 39%  numa-vmstat.node3.nr_zone_inactive_anon
 6.086e+08 ±  4%     +59.5%  9.707e+08 ±  6%  numa-vmstat.node3.numa_hit
 6.085e+08 ±  4%     +59.5%  9.706e+08 ±  6%  numa-vmstat.node3.numa_local
     48109 ± 15%    +101.3%      96864 ± 19%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   4590777 ± 12%     +59.4%    7315684 ±  8%  sched_debug.cfs_rq:/.MIN_vruntime.max
    455022 ± 12%     +75.2%     797256 ± 11%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     65016 ± 74%     -69.4%      19883        sched_debug.cfs_rq:/.load.max
     12469 ± 31%     -30.4%       8683        sched_debug.cfs_rq:/.load.stddev
     48109 ± 15%    +101.3%      96864 ± 19%  sched_debug.cfs_rq:/.max_vruntime.avg
   4590777 ± 12%     +59.4%    7315684 ±  8%  sched_debug.cfs_rq:/.max_vruntime.max
    455022 ± 12%     +75.2%     797256 ± 11%  sched_debug.cfs_rq:/.max_vruntime.stddev
   9592079 ±  3%     +13.2%   10853663 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
    582405 ± 27%     +78.9%    1041757 ± 20%  sched_debug.cfs_rq:/.min_vruntime.stddev
    582382 ± 27%     +78.9%    1041742 ± 20%  sched_debug.cfs_rq:/.spread0.stddev
     10099 ± 16%     -48.8%       5175 ±  2%  sched_debug.cpu.avg_idle.min
   6677677 ±  2%     +42.7%    9526098 ±  4%  sched_debug.cpu.nr_switches.avg
  10540588 ±  4%     +28.7%   13567814 ±  4%  sched_debug.cpu.nr_switches.max
   4200206 ± 11%     +37.2%    5760605 ± 11%  sched_debug.cpu.nr_switches.min
   1099524 ±  4%     +43.4%    1576281 ±  7%  sched_debug.cpu.nr_switches.stddev
    119976 ±  4%     +34.7%     161660 ±  5%  proc-vmstat.nr_active_anon
     78578            +3.4%      81233        proc-vmstat.nr_anon_pages
   3147774            -1.1%    3112160        proc-vmstat.nr_dirty_background_threshold
   6303246            -1.1%    6231930        proc-vmstat.nr_dirty_threshold
    813360           +42.3%    1157555 ±  6%  proc-vmstat.nr_file_pages
  31657054            -1.1%   31300308        proc-vmstat.nr_free_pages
    108496 ±  3%    +280.8%     413108 ± 18%  proc-vmstat.nr_inactive_anon
     71937 ±  4%    +221.3%     231103 ± 19%  proc-vmstat.nr_mapped
      1884           +16.2%       2189 ±  4%  proc-vmstat.nr_page_table_pages
    148187 ±  5%    +232.2%     492249 ± 15%  proc-vmstat.nr_shmem
     32122            +2.5%      32934        proc-vmstat.nr_slab_reclaimable
    119976 ±  4%     +34.7%     161660 ±  5%  proc-vmstat.nr_zone_active_anon
    108496 ±  3%    +280.8%     413108 ± 18%  proc-vmstat.nr_zone_inactive_anon
    453708 ±  8%     +67.4%     759696 ± 19%  proc-vmstat.numa_hint_faults
    168549 ±  4%    +180.3%     472515 ± 27%  proc-vmstat.numa_hint_faults_local
 2.439e+09           +54.7%  3.774e+09        proc-vmstat.numa_hit
 2.439e+09           +54.7%  3.774e+09        proc-vmstat.numa_local
    706052 ±  6%     +49.8%    1057746 ± 15%  proc-vmstat.numa_pte_updates
    814467 ±  9%    +282.4%    3114371 ±  6%  proc-vmstat.pgactivate
 2.439e+09           +54.7%  3.773e+09        proc-vmstat.pgalloc_normal
   3338201           +11.9%    3735711 ±  4%  proc-vmstat.pgfault
 2.439e+09           +54.7%  3.773e+09        proc-vmstat.pgfree
     95321            +1.3%      96586        proc-vmstat.pgreuse
     45.10            -4.4%      43.10        perf-stat.i.MPKI
 1.035e+10           +48.6%  1.538e+10        perf-stat.i.branch-instructions
      1.15            -0.1        1.09        perf-stat.i.branch-miss-rate%
 1.187e+08           +40.5%  1.668e+08        perf-stat.i.branch-misses
      6.36 ±  4%      -2.4        3.91 ±  5%  perf-stat.i.cache-miss-rate%
 1.493e+08 ±  3%     -12.9%    1.3e+08 ±  5%  perf-stat.i.cache-misses
 2.354e+09           +42.1%  3.344e+09        perf-stat.i.cache-references
   2171367           +44.7%    3142825        perf-stat.i.context-switches
      3.84           -29.5%       2.71        perf-stat.i.cpi
 2.005e+11            +4.8%  2.101e+11        perf-stat.i.cpu-cycles
    951.89           -17.7%     783.00        perf-stat.i.cpu-migrations
      1347 ±  4%     +20.6%       1624 ±  5%  perf-stat.i.cycles-between-cache-misses
      0.01 ±  5%      +0.0        0.01 ± 10%  perf-stat.i.dTLB-load-miss-rate%
   1268458 ±  5%    +133.5%    2961820 ± 11%  perf-stat.i.dTLB-load-misses
 1.546e+10           +48.4%  2.294e+10        perf-stat.i.dTLB-loads
      0.00 ±  5%      +0.0        0.00 ±  3%  perf-stat.i.dTLB-store-miss-rate%
    153005 ±  6%    +108.2%     318484 ±  3%  perf-stat.i.dTLB-store-misses
 8.751e+09           +49.0%  1.304e+10        perf-stat.i.dTLB-stores
     71.40            +2.6       73.97        perf-stat.i.iTLB-load-miss-rate%
  64740777           +48.4%   96068625        perf-stat.i.iTLB-load-misses
  25887225           +30.2%   33713621        perf-stat.i.iTLB-loads
 5.232e+10           +48.6%  7.776e+10        perf-stat.i.instructions
      0.26           +41.7%       0.37        perf-stat.i.ipc
      1.39            +4.8%       1.46        perf-stat.i.metric.GHz
    486.44 ±  2%      +9.5%     532.44 ±  3%  perf-stat.i.metric.K/sec
    256.37           +48.2%     379.89        perf-stat.i.metric.M/sec
      3530           +12.4%       3967 ±  4%  perf-stat.i.minor-faults
      3530           +12.4%       3967 ±  4%  perf-stat.i.page-faults
     44.99            -4.4%      43.01        perf-stat.overall.MPKI
      1.15            -0.1        1.08        perf-stat.overall.branch-miss-rate%
      6.34 ±  3%      -2.5        3.89 ±  5%  perf-stat.overall.cache-miss-rate%
      3.83           -29.5%       2.70        perf-stat.overall.cpi
      1345 ±  4%     +20.4%       1620 ±  5%  perf-stat.overall.cycles-between-cache-misses
      0.01 ±  5%      +0.0        0.01 ± 10%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  6%      +0.0        0.00 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     71.44            +2.6       74.02        perf-stat.overall.iTLB-load-miss-rate%
      0.26           +41.8%       0.37        perf-stat.overall.ipc
      9691            -4.0%       9308        perf-stat.overall.path-length
 1.034e+10           +48.6%  1.536e+10        perf-stat.ps.branch-instructions
 1.186e+08           +40.5%  1.666e+08        perf-stat.ps.branch-misses
 1.491e+08 ±  3%     -12.9%  1.298e+08 ±  5%  perf-stat.ps.cache-misses
 2.351e+09           +42.1%   3.34e+09        perf-stat.ps.cache-references
   2168750           +44.7%    3139234        perf-stat.ps.context-switches
 2.003e+11            +4.8%  2.099e+11        perf-stat.ps.cpu-cycles
    950.79           -17.7%     782.12        perf-stat.ps.cpu-migrations
   1267006 ±  5%    +133.5%    2958178 ± 11%  perf-stat.ps.dTLB-load-misses
 1.544e+10           +48.4%  2.292e+10        perf-stat.ps.dTLB-loads
    152841 ±  6%    +108.1%     318103 ±  3%  perf-stat.ps.dTLB-store-misses
 8.741e+09           +49.0%  1.303e+10        perf-stat.ps.dTLB-stores
  64666218           +48.4%   95960247        perf-stat.ps.iTLB-load-misses
  25856385           +30.2%   33675347        perf-stat.ps.iTLB-loads
 5.226e+10           +48.6%  7.767e+10        perf-stat.ps.instructions
      3526           +12.4%       3962 ±  4%  perf-stat.ps.minor-faults
      3526           +12.4%       3962 ±  4%  perf-stat.ps.page-faults
 4.718e+13           +48.6%  7.012e+13        perf-stat.total.instructions
     10.40 ±  5%      -6.3        4.13 ±  3%  perf-profile.calltrace.cycles-pp.try_charge_memcg.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule
     10.56 ±  5%      -6.2        4.36 ±  3%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked
     10.87 ±  5%      -5.8        5.07 ±  3%  perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked.tcp_sendmsg
     10.88 ±  5%      -5.8        5.08 ±  3%  perf-profile.calltrace.cycles-pp.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
     10.90 ±  5%      -5.8        5.12 ±  3%  perf-profile.calltrace.cycles-pp.tcp_wmem_schedule.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      7.23 ±  6%      -4.4        2.86 ±  4%  perf-profile.calltrace.cycles-pp.page_counter_try_charge.try_charge_memcg.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule
     14.91 ±  7%      -3.7       11.19 ±  3%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
     15.13 ±  7%      -3.6       11.50 ±  4%  perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      6.05 ±  6%      -3.2        2.80 ±  3%  perf-profile.calltrace.cycles-pp.refill_stock.__sk_mem_reduce_allocated.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      9.86 ±  5%      -3.2        6.63 ±  4%  perf-profile.calltrace.cycles-pp.try_charge_memcg.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
      6.14 ±  6%      -3.2        2.91 ±  3%  perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
      5.96 ±  6%      -3.2        2.75 ±  3%  perf-profile.calltrace.cycles-pp.drain_stock.refill_stock.__sk_mem_reduce_allocated.tcp_clean_rtx_queue.tcp_ack
      5.95 ±  6%      -3.2        2.74 ±  3%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.drain_stock.refill_stock.__sk_mem_reduce_allocated.tcp_clean_rtx_queue
      6.90 ±  6%      -2.9        3.97 ±  3%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      7.03 ±  6%      -2.9        4.17 ±  3%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      5.62 ±  7%      -2.4        3.23 ±  4%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      6.68 ±  7%      -2.0        4.66 ±  4%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      6.45 ±  6%      -2.0        4.46 ±  9%  perf-profile.calltrace.cycles-pp.page_counter_try_charge.try_charge_memcg.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established
      9.54 ±  6%      -1.8        7.78 ±  3%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
      9.53 ±  6%      -1.7        7.79 ±  3%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
      4.75 ±  6%      -1.6        3.15 ±  5%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.drain_stock.refill_stock.__sk_mem_reduce_allocated.tcp_recvmsg_locked
      4.76 ±  6%      -1.6        3.16 ±  5%  perf-profile.calltrace.cycles-pp.drain_stock.refill_stock.__sk_mem_reduce_allocated.tcp_recvmsg_locked.tcp_recvmsg
      4.83 ±  6%      -1.6        3.23 ±  5%  perf-profile.calltrace.cycles-pp.refill_stock.__sk_mem_reduce_allocated.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      4.96 ±  6%      -1.5        3.42 ±  5%  perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      4.38 ±  5%      -1.4        3.02 ±  7%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog
      6.74 ±  8%      -1.4        5.38 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      6.92 ±  7%      -1.3        5.59 ±  4%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      7.01 ±  7%      -1.3        5.70 ±  4%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      4.53 ±  5%      -1.3        3.23 ±  7%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog.tcp_recvmsg_locked
      4.66 ±  5%      -1.2        3.42 ±  7%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog.tcp_recvmsg_locked.tcp_recvmsg
      4.72 ±  5%      -1.2        3.52 ±  7%  perf-profile.calltrace.cycles-pp.__release_sock.__sk_flush_backlog.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      4.75 ±  5%      -1.2        3.56 ±  7%  perf-profile.calltrace.cycles-pp.__sk_flush_backlog.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      4.29 ±  6%      -0.8        3.48 ±  6%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      1.43 ±  4%      -0.6        0.84 ±  5%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      1.54 ±  4%      -0.6        0.96 ±  5%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg
      1.51 ±  4%      -0.6        0.93 ±  5%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg
      1.57 ±  4%      -0.6        0.99 ±  5%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      1.65 ±  4%      -0.6        1.09 ±  5%  perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.76 ±  4%      -0.1        0.65 ±  5%  perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      0.64 ±  8%      +0.1        0.75 ±  5%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      1.08 ±  9%      +0.1        1.22 ±  5%  perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      0.59 ±  6%      +0.2        0.79 ±  4%  perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      1.54 ±  9%      +0.2        1.77 ±  5%  perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      0.46 ± 44%      +0.3        0.72 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.tcp_sendmsg.sock_sendmsg.__sys_sendto
      0.70 ±  8%      +0.3        1.00 ±  5%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.35 ± 70%      +0.3        0.68 ±  4%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.08 ±223%      +0.5        0.60 ±  4%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +0.5        0.54 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established
      1.39 ±  8%      +0.6        1.94 ±  4%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked
      0.00            +0.6        0.57 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.6        0.60 ±  5%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      0.00            +0.6        0.61 ±  5%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
      0.00            +0.6        0.62 ±  6%  perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      0.00            +0.6        0.62 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.tcp_sendmsg.sock_sendmsg
      0.00            +0.6        0.63 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
      0.00            +0.6        0.63 ±  4%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      1.78 ±  8%      +0.6        2.42 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg
      0.00            +0.7        0.66 ±  6%  perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
      0.00            +0.7        0.68 ±  7%  perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      1.92 ±  9%      +0.7        2.63 ±  4%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      2.85 ±  4%      +0.7        3.59 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
      2.10 ±  8%      +0.8        2.87 ±  4%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      2.98 ±  4%      +0.8        3.77 ±  4%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
      3.18 ±  4%      +0.9        4.08 ±  4%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      3.43 ±  4%      +1.0        4.48 ±  4%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      6.87 ±  7%      +2.7        9.56 ±  5%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      6.94 ±  7%      +2.7        9.66 ±  5%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
      7.03 ±  7%      +2.8        9.80 ±  5%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      7.71 ±  6%      +3.2       10.88 ±  5%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      7.70 ±  6%      +3.2       10.86 ±  5%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      8.82 ±  3%      +4.8       13.64 ±  4%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter.skb_do_copy_data_nocache.tcp_sendmsg_locked
      8.92 ±  3%      +4.8       13.76 ±  4%  perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg
      9.25 ±  3%      +5.0       14.28 ±  4%  perf-profile.calltrace.cycles-pp._copy_from_iter.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      9.48 ±  3%      +5.1       14.62 ±  4%  perf-profile.calltrace.cycles-pp.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     22.43 ±  5%     -11.2       11.20 ±  4%  perf-profile.children.cycles-pp.try_charge_memcg
     22.66 ±  5%     -11.1       11.54 ±  4%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
     15.56 ±  5%      -7.8        7.77 ±  4%  perf-profile.children.cycles-pp.page_counter_try_charge
     12.74 ±  5%      -7.3        5.46 ±  4%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
     12.76 ±  5%      -7.3        5.49 ±  4%  perf-profile.children.cycles-pp.__sk_mem_schedule
     10.90 ±  5%      -5.8        5.12 ±  3%  perf-profile.children.cycles-pp.tcp_wmem_schedule
     21.55 ±  5%      -5.6       15.90 ±  4%  perf-profile.children.cycles-pp.__release_sock
     23.41 ±  5%      -5.6       17.80 ±  4%  perf-profile.children.cycles-pp.tcp_rcv_established
     23.66 ±  5%      -5.5       18.15 ±  4%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
     11.41 ±  5%      -5.0        6.42 ±  4%  perf-profile.children.cycles-pp.refill_stock
     11.21 ±  5%      -5.0        6.26 ±  4%  perf-profile.children.cycles-pp.drain_stock
     11.19 ±  5%      -5.0        6.24 ±  4%  perf-profile.children.cycles-pp.page_counter_uncharge
     11.60 ±  5%      -4.9        6.70 ±  4%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
     16.94 ±  6%      -4.2       12.72 ±  4%  perf-profile.children.cycles-pp.release_sock
     12.91 ±  5%      -4.2        8.72 ±  4%  perf-profile.children.cycles-pp.tcp_data_queue
      7.24 ±  6%      -2.9        4.37 ±  3%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      7.56 ±  6%      -2.7        4.85 ±  3%  perf-profile.children.cycles-pp.tcp_ack
      1.46 ±  5%      -1.3        0.14 ±  8%  perf-profile.children.cycles-pp.tcp_try_rmem_schedule
      4.98 ±  4%      -1.3        3.67 ±  6%  perf-profile.children.cycles-pp.__sk_flush_backlog
      2.75 ±  8%      -1.1        1.62 ±  6%  perf-profile.children.cycles-pp.propagate_protected_usage
      0.93 ± 10%      -0.2        0.72 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.27 ±  9%      -0.2        0.08 ±  8%  perf-profile.children.cycles-pp.newidle_balance
      0.21 ±  8%      -0.2        0.03 ±102%  perf-profile.children.cycles-pp.load_balance
      0.37 ±  8%      -0.1        0.24 ±  4%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.76 ±  3%      -0.1        0.65 ±  5%  perf-profile.children.cycles-pp.tcp_stream_alloc_skb
      0.06 ±  9%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.06 ± 11%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.07 ±  9%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.10 ± 10%      +0.0        0.13 ±  5%  perf-profile.children.cycles-pp.update_rq_clock
      0.07 ± 10%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.tcp_wfree
      0.06 ±  6%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.ip_output
      0.06 ± 11%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.11 ±  8%      +0.0        0.14 ±  6%  perf-profile.children.cycles-pp.tcp_check_space
      0.07 ±  7%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.__cond_resched
      0.12 ±  6%      +0.0        0.16 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.06 ± 11%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.___perf_sw_event
      0.12 ±  5%      +0.0        0.15 ±  4%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.04 ± 44%      +0.0        0.07 ± 10%  perf-profile.children.cycles-pp.tcp_established_options
      0.08 ± 14%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.recv_data
      0.08 ±  6%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.06 ±  7%      +0.0        0.10 ±  3%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.04 ± 45%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.ip_rcv_core
      0.07 ±  8%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.07 ±  6%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.03 ± 70%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.kmem_cache_free
      0.07 ±  6%      +0.0        0.11 ±  7%  perf-profile.children.cycles-pp.lock_timer_base
      0.07 ± 10%      +0.0        0.11 ±  5%  perf-profile.children.cycles-pp.set_next_entity
      0.04 ± 45%      +0.0        0.08        perf-profile.children.cycles-pp.kfree
      0.14 ±  8%      +0.0        0.18 ±  7%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.02 ± 99%      +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.sock_recvmsg
      0.10 ±  6%      +0.0        0.14 ±  4%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.05 ± 45%      +0.0        0.09 ±  6%  perf-profile.children.cycles-pp.ip_send_check
      0.15 ±  8%      +0.0        0.19 ±  7%  perf-profile.children.cycles-pp.netif_rx_internal
      0.07 ± 10%      +0.0        0.11 ± 11%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.07 ±  9%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.__might_fault
      0.02 ± 99%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.perf_tp_event
      0.10 ±  8%      +0.0        0.14 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.09 ± 11%      +0.0        0.13 ±  7%  perf-profile.children.cycles-pp.llist_reverse_order
      0.12 ±  8%      +0.0        0.16 ±  4%  perf-profile.children.cycles-pp.__slab_free
      0.08 ± 12%      +0.0        0.12 ±  6%  perf-profile.children.cycles-pp.__ip_local_out
      0.02 ± 99%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.tcp_event_data_recv
      0.16 ±  8%      +0.0        0.20 ±  5%  perf-profile.children.cycles-pp.__netif_rx
      0.12 ±  8%      +0.0        0.17 ±  7%  perf-profile.children.cycles-pp.remove_wait_queue
      0.12 ±  9%      +0.0        0.16 ±  6%  perf-profile.children.cycles-pp.__switch_to_asm
      0.09 ±  6%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.select_idle_sibling
      0.08 ±  7%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.available_idle_cpu
      0.09 ± 11%      +0.0        0.14 ±  4%  perf-profile.children.cycles-pp.ip_local_out
      0.14 ± 22%      +0.0        0.18 ± 16%  perf-profile.children.cycles-pp.__sk_dst_check
      0.10 ±  7%      +0.0        0.15 ±  4%  perf-profile.children.cycles-pp.llist_add_batch
      0.01 ±223%      +0.0        0.06 ± 13%  perf-profile.children.cycles-pp.__ip_finish_output
      0.01 ±223%      +0.0        0.06 ±  6%  perf-profile.children.cycles-pp.reweight_entity
      0.02 ±141%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.rcu_all_qs
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.inet_ehashfn
      0.10 ±  7%      +0.1        0.16 ±  3%  perf-profile.children.cycles-pp.__smp_call_single_queue
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.copy_user_short_string
      0.02 ±141%      +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.raw_v4_input
      0.17 ±  4%      +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.01 ±223%      +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.os_xsave
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.11 ±  6%      +0.1        0.16 ±  6%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.01 ±223%      +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.alloc_pages
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.13 ±  5%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.sock_put
      0.13 ±  9%      +0.1        0.19 ±  3%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.13 ± 11%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.update_curr
      0.12 ± 11%      +0.1        0.18 ±  6%  perf-profile.children.cycles-pp.rmqueue
      0.11 ±  6%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.send_data
      0.19 ±  9%      +0.1        0.25 ±  4%  perf-profile.children.cycles-pp.update_load_avg
      0.10 ±  9%      +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.tcp_update_skb_after_send
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.tcp_ack_update_window
      0.13 ±  4%      +0.1        0.19 ±  4%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__tcp_select_window
      0.12 ±  4%      +0.1        0.18 ±  4%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.12 ±  8%      +0.1        0.18 ±  5%  perf-profile.children.cycles-pp.__ksize
      0.13 ±  4%      +0.1        0.20 ±  6%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.22 ±  8%      +0.1        0.29 ±  6%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.18 ±  8%      +0.1        0.25 ±  5%  perf-profile.children.cycles-pp.enqueue_entity
      0.15 ±  6%      +0.1        0.22 ±  4%  perf-profile.children.cycles-pp.aa_sk_perm
      0.15 ±  6%      +0.1        0.22 ±  4%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.18 ± 18%      +0.1        0.25 ± 13%  perf-profile.children.cycles-pp.ip_rcv
      0.14 ±  8%      +0.1        0.20 ±  5%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.13 ±  6%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.prepare_task_switch
      0.14 ±  7%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.23 ±  7%      +0.1        0.30 ±  5%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.16 ±  7%      +0.1        0.23 ±  4%  perf-profile.children.cycles-pp.free_pcp_prepare
      0.14 ±  4%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.22 ±  4%      +0.1        0.30 ±  4%  perf-profile.children.cycles-pp.sock_rfree
      0.37 ±  8%      +0.1        0.44 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock
      0.16 ±  7%      +0.1        0.24 ±  3%  perf-profile.children.cycles-pp.native_sched_clock
      0.19 ± 10%      +0.1        0.27 ±  5%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.15 ±  8%      +0.1        0.22 ±  2%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.17 ±  8%      +0.1        0.25 ±  4%  perf-profile.children.cycles-pp.__might_resched
      0.18 ±  3%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.19 ±  7%      +0.1        0.27 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.16 ±  5%      +0.1        0.24 ±  6%  perf-profile.children.cycles-pp.tcp_tso_segs
      0.16 ±  5%      +0.1        0.25 ±  3%  perf-profile.children.cycles-pp.select_task_rq
      0.18 ±  5%      +0.1        0.27 ±  4%  perf-profile.children.cycles-pp.__mod_timer
      0.20 ±  7%      +0.1        0.29 ±  5%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.20 ±  7%      +0.1        0.29 ±  4%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.19 ±  5%      +0.1        0.29 ±  4%  perf-profile.children.cycles-pp.sk_reset_timer
      0.20 ±  7%      +0.1        0.30 ±  5%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.19 ±  5%      +0.1        0.28 ±  5%  perf-profile.children.cycles-pp.__skb_clone
      0.30 ±  5%      +0.1        0.40        perf-profile.children.cycles-pp.menu_select
      0.21 ±  6%      +0.1        0.31 ±  4%  perf-profile.children.cycles-pp.read_tsc
      0.39 ±  7%      +0.1        0.50 ±  5%  perf-profile.children.cycles-pp.loopback_xmit
      0.19 ±  6%      +0.1        0.30 ±  3%  perf-profile.children.cycles-pp.__entry_text_start
      0.20 ±  7%      +0.1        0.31 ±  6%  perf-profile.children.cycles-pp.__fdget
      0.20 ±  3%      +0.1        0.31 ±  5%  perf-profile.children.cycles-pp.kfree_skbmem
      0.42 ±  8%      +0.1        0.53 ±  5%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.24 ±  4%      +0.1        0.36 ±  4%  perf-profile.children.cycles-pp.tcp_add_backlog
      0.27 ±  8%      +0.1        0.39 ±  6%  perf-profile.children.cycles-pp.dequeue_entity
      0.22 ±  6%      +0.1        0.34 ±  5%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.26 ±  7%      +0.1        0.38 ±  5%  perf-profile.children.cycles-pp.__alloc_pages
      0.24 ±  4%      +0.1        0.36 ±  5%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.52 ± 10%      +0.1        0.65 ±  5%  perf-profile.children.cycles-pp.ktime_get
      0.30 ±  9%      +0.1        0.43 ±  5%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.47 ±  8%      +0.1        0.61 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      0.24 ±  8%      +0.1        0.38 ±  5%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.92 ±  5%      +0.2        1.07 ±  4%  perf-profile.children.cycles-pp.lock_sock_nested
      0.33 ±  6%      +0.2        0.49 ±  5%  perf-profile.children.cycles-pp.dst_release
      0.38 ±  8%      +0.2        0.55 ±  5%  perf-profile.children.cycles-pp.skb_page_frag_refill
      0.39 ±  8%      +0.2        0.56 ±  6%  perf-profile.children.cycles-pp.sk_page_frag_refill
      0.46 ±  9%      +0.2        0.65 ±  5%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.48 ±  6%      +0.2        0.69 ±  3%  perf-profile.children.cycles-pp.napi_consume_skb
      0.96 ±  6%      +0.2        1.18 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.39 ±  6%      +0.2        0.61 ±  5%  perf-profile.children.cycles-pp.free_unref_page
      0.87 ±  7%      +0.2        1.11 ±  4%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.46 ±  7%      +0.2        0.70 ±  5%  perf-profile.children.cycles-pp.__alloc_skb
      0.45 ±  5%      +0.2        0.70 ±  7%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.54 ±  6%      +0.3        0.80 ±  4%  perf-profile.children.cycles-pp.try_to_wake_up
      0.54 ±  6%      +0.3        0.81 ±  5%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.52 ±  5%      +0.3        0.81 ±  6%  perf-profile.children.cycles-pp.check_heap_object
      0.58 ±  6%      +0.3        0.87 ±  4%  perf-profile.children.cycles-pp.__wake_up_common
      0.65 ±  5%      +0.3        0.94 ±  4%  perf-profile.children.cycles-pp.__kfree_skb
      0.71 ±  9%      +0.3        1.01 ±  5%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.68 ±  7%      +0.3        1.00 ±  4%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.61 ±  5%      +0.3        0.94 ±  6%  perf-profile.children.cycles-pp.__check_object_size
      0.81 ±  7%      +0.3        1.16 ±  4%  perf-profile.children.cycles-pp.sock_def_readable
      0.07 ± 11%      +0.5        0.52 ±  4%  perf-profile.children.cycles-pp.poll_idle
      1.10 ±  6%      +0.5        1.58 ±  4%  perf-profile.children.cycles-pp.skb_release_data
      7.24 ±  6%      +2.6        9.86 ±  6%  perf-profile.children.cycles-pp.copyout
      7.34 ±  6%      +2.7       10.00 ±  6%  perf-profile.children.cycles-pp._copy_to_iter
      8.04 ±  5%      +3.0       11.09 ±  6%  perf-profile.children.cycles-pp.__skb_datagram_iter
      8.05 ±  5%      +3.0       11.10 ±  6%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      8.92 ±  3%      +4.8       13.77 ±  4%  perf-profile.children.cycles-pp.copyin
      9.25 ±  3%      +5.0       14.28 ±  4%  perf-profile.children.cycles-pp._copy_from_iter
      9.48 ±  3%      +5.1       14.63 ±  4%  perf-profile.children.cycles-pp.skb_do_copy_data_nocache
     16.11 ±  3%      +7.4       23.54 ±  5%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
     13.98 ±  5%      -7.1        6.91 ±  4%  perf-profile.self.cycles-pp.page_counter_try_charge
      9.91 ±  5%      -4.5        5.44 ±  4%  perf-profile.self.cycles-pp.page_counter_uncharge
      6.80 ±  5%      -3.4        3.40 ±  4%  perf-profile.self.cycles-pp.try_charge_memcg
      2.74 ±  8%      -1.1        1.61 ±  6%  perf-profile.self.cycles-pp.propagate_protected_usage
      0.06 ±  6%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.05 ±  7%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.06 ±  7%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.enqueue_entity
      0.06 ±  6%      +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.__check_object_size
      0.06 ±  8%      +0.0        0.08 ±  7%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.07 ± 10%      +0.0        0.10 ±  6%  perf-profile.self.cycles-pp.tcp_wfree
      0.06 ±  6%      +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.rmqueue
      0.14 ± 10%      +0.0        0.17 ±  5%  perf-profile.self.cycles-pp.__switch_to
      0.06 ±  6%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.__mod_timer
      0.06 ± 14%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.update_curr
      0.07 ±  9%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.update_load_avg
      0.06 ±  9%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.__alloc_pages
      0.06 ± 11%      +0.0        0.09 ±  6%  perf-profile.self.cycles-pp.recv_data
      0.04 ± 44%      +0.0        0.07 ±  8%  perf-profile.self.cycles-pp.tcp_established_options
      0.07 ±  5%      +0.0        0.10 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.05 ± 45%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.recv
      0.06 ±  9%      +0.0        0.09 ±  8%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.04 ± 45%      +0.0        0.08 ±  9%  perf-profile.self.cycles-pp.do_softirq
      0.06 ±  7%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
      0.08 ±  6%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.06 ± 11%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.04 ± 44%      +0.0        0.07 ±  6%  perf-profile.self.cycles-pp.kfree
      0.07 ±  5%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.skb_do_copy_data_nocache
      0.07 ±  7%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp._copy_to_iter
      0.04 ± 45%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.ip_rcv_core
      0.07 ±  8%      +0.0        0.10 ± 16%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.06 ± 11%      +0.0        0.10 ±  9%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.05 ± 45%      +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.___perf_sw_event
      0.07 ± 10%      +0.0        0.11 ± 11%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.03 ± 70%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.process_backlog
      0.03 ± 70%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.07 ± 10%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.10 ±  9%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.menu_select
      0.08 ±  8%      +0.0        0.12        perf-profile.self.cycles-pp.__softirqentry_text_start
      0.07 ± 10%      +0.0        0.11 ±  5%  perf-profile.self.cycles-pp.__alloc_skb
      0.08 ± 10%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.07 ±  5%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.prepare_task_switch
      0.12 ±  6%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.07 ± 10%      +0.0        0.11 ± 11%  perf-profile.self.cycles-pp.tcp_rcv_space_adjust
      0.02 ± 99%      +0.0        0.06 ±  7%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.02 ± 99%      +0.0        0.06 ±  7%  perf-profile.self.cycles-pp.loopback_xmit
      0.10 ± 10%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.05 ± 44%      +0.0        0.09 ±  6%  perf-profile.self.cycles-pp.ip_send_check
      0.09 ± 10%      +0.0        0.13 ±  7%  perf-profile.self.cycles-pp.send_omni_inner
      0.07 ±  8%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.08 ±  8%      +0.0        0.12 ±  5%  perf-profile.self.cycles-pp.send_data
      0.12 ±  8%      +0.0        0.16 ±  4%  perf-profile.self.cycles-pp.__slab_free
      0.06 ± 11%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.02 ± 99%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.kmem_cache_free
      0.02 ± 99%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.tcp_event_data_recv
      0.08 ±  8%      +0.0        0.13 ±  5%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.09 ± 10%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.llist_reverse_order
      0.09 ±  7%      +0.0        0.14 ±  4%  perf-profile.self.cycles-pp.__sys_sendto
      0.02 ±141%      +0.0        0.06 ±  6%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.08 ±  7%      +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.available_idle_cpu
      0.11 ±  9%      +0.0        0.16 ±  5%  perf-profile.self.cycles-pp.__switch_to_asm
      0.10 ±  7%      +0.0        0.15 ±  4%  perf-profile.self.cycles-pp.llist_add_batch
      0.10 ±  5%      +0.0        0.15 ±  7%  perf-profile.self.cycles-pp.aa_sk_perm
      0.01 ±223%      +0.0        0.06 ±  6%  perf-profile.self.cycles-pp.reweight_entity
      0.11 ±  3%      +0.1        0.16 ±  7%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.01 ±223%      +0.1        0.06        perf-profile.self.cycles-pp.os_xsave
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.copy_user_short_string
      0.09 ±  6%      +0.1        0.14 ±  4%  perf-profile.self.cycles-pp.__entry_text_start
      0.02 ±141%      +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.raw_v4_input
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.free_unref_page
      0.13 ±  8%      +0.1        0.18 ±  4%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.ip_output
      0.01 ±223%      +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.release_sock
      0.01 ±223%      +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.select_task_rq
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.sock_sendmsg
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.perf_tp_event
      0.10 ± 10%      +0.1        0.16 ±  5%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_add_backlog
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_ack_update_window
      0.10 ±  6%      +0.1        0.16 ±  7%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.13 ±  7%      +0.1        0.19 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.13 ±  4%      +0.1        0.19 ±  5%  perf-profile.self.cycles-pp.sock_put
      0.08 ±  5%      +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.skb_attempt_defer_free
      0.16 ±  6%      +0.1        0.22 ±  5%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.12 ±  4%      +0.1        0.18 ±  3%  perf-profile.self.cycles-pp.__release_sock
      0.13 ±  9%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.ip_finish_output2
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__tcp_select_window
      0.12 ±  8%      +0.1        0.18 ±  4%  perf-profile.self.cycles-pp.__ksize
      0.15 ±  6%      +0.1        0.22 ±  5%  perf-profile.self.cycles-pp.__send
      0.36 ±  8%      +0.1        0.43 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock
      0.14 ±  7%      +0.1        0.21 ±  7%  perf-profile.self.cycles-pp.recv_omni
      0.13 ±  7%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.16 ±  6%      +0.1        0.23 ±  5%  perf-profile.self.cycles-pp.free_pcp_prepare
      0.16 ±  8%      +0.1        0.23 ±  3%  perf-profile.self.cycles-pp.native_sched_clock
      0.22 ±  6%      +0.1        0.29 ±  4%  perf-profile.self.cycles-pp.sock_rfree
      0.16 ±  6%      +0.1        0.24 ±  4%  perf-profile.self.cycles-pp.__might_resched
      0.16 ±  6%      +0.1        0.24 ±  5%  perf-profile.self.cycles-pp.tcp_tso_segs
      0.18 ±  8%      +0.1        0.27 ±  3%  perf-profile.self.cycles-pp.wait_woken
      0.16 ±  5%      +0.1        0.25 ±  5%  perf-profile.self.cycles-pp.__skb_clone
      0.19 ±  5%      +0.1        0.28 ±  7%  perf-profile.self.cycles-pp.tcp_ack
      0.36 ±  8%      +0.1        0.45 ±  5%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.20 ±  5%      +0.1        0.31 ±  4%  perf-profile.self.cycles-pp.read_tsc
      0.19 ±  5%      +0.1        0.30 ±  5%  perf-profile.self.cycles-pp.net_rx_action
      0.20 ±  6%      +0.1        0.30 ±  6%  perf-profile.self.cycles-pp.__fdget
      0.21 ±  3%      +0.1        0.32 ±  5%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.19 ±  5%      +0.1        0.30 ±  6%  perf-profile.self.cycles-pp.kfree_skbmem
      0.23 ±  5%      +0.1        0.34 ±  5%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.21 ±  5%      +0.1        0.33 ±  6%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.22 ±  7%      +0.1        0.34 ±  4%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.24 ±  5%      +0.1        0.36 ±  5%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.32 ±  6%      +0.1        0.44 ±  5%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.27 ±  3%      +0.1        0.42 ±  3%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.51 ±  6%      +0.2        0.66 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.29 ±  5%      +0.2        0.45 ±  4%  perf-profile.self.cycles-pp._copy_from_iter
      0.32 ±  7%      +0.2        0.48 ±  5%  perf-profile.self.cycles-pp.dst_release
      0.56 ±  6%      +0.2        0.75 ±  4%  perf-profile.self.cycles-pp.skb_release_data
      0.38 ±  5%      +0.2        0.59 ±  6%  perf-profile.self.cycles-pp.check_heap_object
      0.96 ±  6%      +0.2        1.18 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.51 ±  6%      +0.2        0.76 ±  5%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.74 ±  3%      +0.4        1.13 ±  5%  perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.34 ±  6%      +0.4        0.76 ±  4%  perf-profile.self.cycles-pp.__sk_mem_raise_allocated
      0.05 ± 46%      +0.4        0.50 ±  4%  perf-profile.self.cycles-pp.poll_idle
     16.01 ±  3%      +7.4       23.42 ±  5%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://01.org/lkp



View attachment "config-6.0.0-rc3-00089-g1813e51eece0" of type "text/plain" (163956 bytes)

View attachment "job-script" of type "text/plain" (8441 bytes)

View attachment "job.yaml" of type "text/plain" (5824 bytes)

View attachment "reproduce" of type "text/plain" (4955 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ