lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <202305260951.338c3bfe-oliver.sang@intel.com>
Date: Fri, 26 May 2023 10:31:08 +0800
From: kernel test robot <oliver.sang@...el.com>
To: David Howells <dhowells@...hat.com>
CC: <oe-lkp@...ts.linux.dev>, <lkp@...el.com>, Eric Dumazet
	<edumazet@...gle.com>, "David S. Miller" <davem@...emloft.net>, David Ahern
	<dsahern@...nel.org>, Jakub Kicinski <kuba@...nel.org>, Paolo Abeni
	<pabeni@...hat.com>, Jens Axboe <axboe@...nel.dk>, Matthew Wilcox
	<willy@...radead.org>, <netdev@...r.kernel.org>, <ying.huang@...el.com>,
	<feng.tang@...el.com>, <fengwei.yin@...el.com>, <oliver.sang@...el.com>
Subject: [dhowells-fs:sendpage-1] [tcp]  77f5a42b2d:  netperf.Throughput_Mbps
 -14.6% regression


Hello,

kernel test robot noticed a -14.6% regression of netperf.Throughput_Mbps on:


commit: 77f5a42b2db3e39c971a7d11f0b63745cef8b496 ("tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES")
https://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git sendpage-1

testcase: netperf
test machine: 128 threads 4 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory
parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 50%
	cluster: cs-localhost
	send_size: 10K
	test: TCP_SENDFILE
	cpufreq_governor: performance

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/



If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <oliver.sang@...el.com>
| Closes: https://lore.kernel.org/oe-lkp/202305260951.338c3bfe-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/50%/debian-11.1-x86_64-20220510.cgz/300s/10K/lkp-icl-2sp2/TCP_SENDFILE/netperf

commit: 
  b032c5b94f ("tcp: Support MSG_SPLICE_PAGES")
  77f5a42b2d ("tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES")

b032c5b94fdcfda3 77f5a42b2db3e39c971a7d11f0b 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     23176           -14.6%      19795        netperf.Throughput_Mbps
   1483319           -14.6%    1266931        netperf.Throughput_total_Mbps
    164111 ±  7%     -11.3%     145564 ±  2%  netperf.time.involuntary_context_switches
    658.78           -13.4%     570.30        netperf.time.user_time
 5.432e+09           -14.6%   4.64e+09        netperf.workload
 5.871e+09           +14.9%  6.745e+09        cpuidle..time
 7.205e+08           +13.3%  8.164e+08        cpuidle..usage
     13.69            +2.2       15.92        mpstat.cpu.all.idle%
      2.67            +0.3        2.99        mpstat.cpu.all.irq%
     30044 ± 21%     +82.6%      54861 ± 26%  numa-vmstat.node0.nr_anon_pages
    673.50 ± 39%     +74.3%       1174 ± 20%  numa-vmstat.node0.nr_page_table_pages
   5091909           +14.7%    5837888        vmstat.system.cs
   2625027           +13.7%    2983398        vmstat.system.in
    417561 ±  3%     +11.1%     463713        proc-vmstat.nr_mapped
      3036            +4.0%       3158        proc-vmstat.nr_page_table_pages
    775516            +5.3%     816466 ±  5%  proc-vmstat.pgfree
    120143 ± 21%     +82.6%     219394 ± 26%  numa-meminfo.node0.AnonPages
    156834 ± 24%     +90.2%     298365 ± 25%  numa-meminfo.node0.AnonPages.max
      2694 ± 38%     +74.2%       4694 ± 20%  numa-meminfo.node0.PageTables
     88914 ± 53%     -68.1%      28354 ± 95%  numa-meminfo.node2.AnonHugePages
  7.19e+08           +13.3%  8.147e+08        turbostat.C1
     15.08            +2.2       17.31        turbostat.C1%
     10.22           +15.5%      11.81        turbostat.CPU%c1
 7.979e+08           +13.8%  9.084e+08        turbostat.IRQ
   1479712           +13.5%    1679022        turbostat.POLL
      0.05            +0.0        0.06        turbostat.POLL%
      0.86 ± 30%     -58.1%       0.36 ± 31%  sched_debug.cfs_rq:/.load_avg.min
   7919342 ±  2%      -9.3%    7186168 ±  2%  sched_debug.cfs_rq:/.min_vruntime.min
    241910 ±  2%     -27.7%     174919 ±  2%  sched_debug.cpu.avg_idle.avg
    272049 ±  4%     -36.6%     172555 ±  5%  sched_debug.cpu.avg_idle.stddev
   6051828           +14.8%    6948552        sched_debug.cpu.nr_switches.avg
   9751445 ±  3%     +18.0%   11507180 ±  3%  sched_debug.cpu.nr_switches.max
   1258987 ±  9%     +24.1%    1562137 ±  4%  sched_debug.cpu.nr_switches.stddev
 4.431e+10            +3.4%   4.58e+10        perf-stat.i.branch-instructions
      0.89            -0.0        0.84        perf-stat.i.branch-miss-rate%
 3.896e+08            -2.3%  3.805e+08        perf-stat.i.branch-misses
      6.56 ±  5%      -1.0        5.56 ±  2%  perf-stat.i.cache-miss-rate%
 1.055e+08 ±  5%     -14.7%   90006058 ±  2%  perf-stat.i.cache-misses
   5140409           +14.8%    5900606        perf-stat.i.context-switches
      1.29            -4.8%       1.22        perf-stat.i.cpi
 3.019e+11            -1.5%  2.975e+11        perf-stat.i.cpu-cycles
     27653 ±  3%     +32.4%      36604 ±  6%  perf-stat.i.cpu-migrations
      2891 ±  5%     +15.5%       3339 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.01 ±  2%      -0.0        0.01 ± 20%  perf-stat.i.dTLB-load-miss-rate%
   7555716 ±  2%     -19.0%    6122873 ± 20%  perf-stat.i.dTLB-load-misses
 6.929e+10            +1.9%  7.062e+10        perf-stat.i.dTLB-loads
    957475 ±  4%     +10.0%    1053256 ±  6%  perf-stat.i.dTLB-store-misses
 3.778e+10            +1.4%  3.831e+10        perf-stat.i.dTLB-stores
 2.345e+11            +3.5%  2.427e+11        perf-stat.i.instructions
      0.78            +5.0%       0.82        perf-stat.i.ipc
      2.36            -1.5%       2.32        perf-stat.i.metric.GHz
    264.09 ±  5%     -10.2%     237.26 ±  2%  perf-stat.i.metric.K/sec
      1195            +2.2%       1221        perf-stat.i.metric.M/sec
  14860505 ±  7%     -15.1%   12621197 ±  3%  perf-stat.i.node-load-misses
   8950975 ±  3%     -13.9%    7706748        perf-stat.i.node-store-misses
      0.88            -0.0        0.83        perf-stat.overall.branch-miss-rate%
      6.52 ±  5%      -1.0        5.51 ±  2%  perf-stat.overall.cache-miss-rate%
      1.29            -4.8%       1.23        perf-stat.overall.cpi
      2869 ±  5%     +15.3%       3307 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.01 ±  2%      -0.0        0.01 ± 20%  perf-stat.overall.dTLB-load-miss-rate%
      0.78            +5.1%       0.82        perf-stat.overall.ipc
     12960           +21.4%      15729        perf-stat.overall.path-length
 4.416e+10            +3.4%  4.564e+10        perf-stat.ps.branch-instructions
 3.883e+08            -2.3%  3.792e+08        perf-stat.ps.branch-misses
 1.052e+08 ±  5%     -14.7%   89701078 ±  2%  perf-stat.ps.cache-misses
   5123329           +14.8%    5881511        perf-stat.ps.context-switches
 3.009e+11            -1.5%  2.965e+11        perf-stat.ps.cpu-cycles
     27520 ±  3%     +32.3%      36412 ±  6%  perf-stat.ps.cpu-migrations
 6.906e+10            +1.9%  7.038e+10        perf-stat.ps.dTLB-loads
    952596 ±  4%     +10.0%    1047647 ±  6%  perf-stat.ps.dTLB-store-misses
 3.765e+10            +1.4%  3.818e+10        perf-stat.ps.dTLB-stores
 2.337e+11            +3.5%  2.419e+11        perf-stat.ps.instructions
  14801383 ±  7%     -15.1%   12571364 ±  3%  perf-stat.ps.node-load-misses
   8918890 ±  3%     -13.9%    7679297        perf-stat.ps.node-store-misses
  7.04e+13            +3.7%  7.298e+13        perf-stat.total.instructions
     10.64           -10.6        0.00        perf-profile.calltrace.cycles-pp.tcp_build_frag.do_tcp_sendpages.tcp_sendpage.inet_sendpage.kernel_sendpage
      7.79            -7.8        0.00        perf-profile.calltrace.cycles-pp.tcp_wmem_schedule.tcp_build_frag.do_tcp_sendpages.tcp_sendpage.inet_sendpage
      7.54            -7.5        0.00        perf-profile.calltrace.cycles-pp.__sk_mem_schedule.tcp_wmem_schedule.tcp_build_frag.do_tcp_sendpages.tcp_sendpage
      7.36            -7.4        0.00        perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_build_frag.do_tcp_sendpages
     31.89            -2.7       29.22        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
     32.00            -2.7       29.34        perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     32.22            -2.7       29.56        perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     32.70            -2.7       30.05        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
     32.80            -2.6       30.16        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.process_requests
     29.44            -2.1       27.38        perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom
     11.30            -1.4        9.96        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg
     11.23            -1.3        9.88        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      7.38            -1.1        6.30        perf-profile.calltrace.cycles-pp.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
     10.00            -1.0        9.00        perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked
      6.46            -1.0        5.46        perf-profile.calltrace.cycles-pp.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile
     10.15            -1.0        9.16        perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg
     10.50            -1.0        9.53        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      5.52            -0.9        4.60        perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
      7.06            -0.9        6.17        perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      5.20            -0.9        4.32        perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage
      6.58            -0.8        5.75        perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
     10.27            -0.8        9.47        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      4.43            -0.8        3.67        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage
     15.06            -0.7       14.38        perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
      8.78            -0.7        8.10        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     13.48            -0.7       12.80        perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip
     11.77            -0.7       11.09        perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
     13.43            -0.7       12.76        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq
     12.03            -0.7       11.36        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
     15.67            -0.7       15.00        perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
     12.09            -0.7       11.42        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     13.04            -0.7       12.38        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq
     15.91            -0.7       15.26        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
     15.82            -0.7       15.17        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
      8.91            -0.6        8.28        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      3.10            -0.5        2.61        perf-profile.calltrace.cycles-pp.filemap_get_pages.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
      2.88            -0.5        2.42        perf-profile.calltrace.cycles-pp.filemap_get_read_batch.filemap_get_pages.filemap_read.generic_file_splice_read.splice_direct_to_actor
      1.91 ±  9%      -0.5        1.46 ±  2%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      1.74 ±  9%      -0.4        1.29 ±  2%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg
      1.83 ±  9%      -0.4        1.38 ±  2%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg
      2.04 ±  9%      -0.4        1.60 ±  2%  perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom
      1.39 ±  9%      -0.4        0.96 ±  3%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      1.18 ±  8%      -0.4        0.76 ±  3%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      1.47 ±  3%      -0.4        1.07 ±  3%  perf-profile.calltrace.cycles-pp.try_charge_memcg.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule
      3.22            -0.4        2.83        perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      2.96            -0.4        2.58        perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
      2.54            -0.3        2.19 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      2.51            -0.3        2.17 ±  2%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      2.05 ±  9%      -0.3        1.72 ±  2%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.02 ±  2%      -0.3        0.69 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
      2.59            -0.3        2.26        perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      1.59 ±  9%      -0.3        1.27 ±  2%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      1.28            -0.2        1.08        perf-profile.calltrace.cycles-pp.copy_page_to_iter_pipe.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
      0.76 ±  2%      -0.1        0.68        perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_release.__splice_from_pipe.generic_splice_sendpage.direct_splice_actor.splice_direct_to_actor
      0.70            -0.1        0.63        perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      0.58            +0.1        0.64        perf-profile.calltrace.cycles-pp.skb_release_data.napi_consume_skb.net_rx_action.__do_softirq.do_softirq
      0.68            +0.1        0.75        perf-profile.calltrace.cycles-pp.napi_consume_skb.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip
      1.13            +0.1        1.20        perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      0.57            +0.1        0.65 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      1.36            +0.1        1.46 ±  2%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      0.59 ±  2%      +0.1        0.73        perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.93            +0.1        1.07        perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.99            +0.1        1.14        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.79 ±  2%      +0.2        0.95 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single
      0.82            +0.2        0.98 ±  3%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single
      0.97            +0.2        1.14 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
      2.84            +0.2        3.02        perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.24            +0.2        1.46 ±  2%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt
      0.93            +0.2        1.16        perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_data_queue
      0.49 ± 46%      +0.3        0.74        perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      1.05            +0.3        1.30 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established
      1.09            +0.3        1.36        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
      0.56 ± 46%      +0.3        0.86        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken
      1.20            +0.3        1.49        perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.52 ± 46%      +0.3        0.84 ±  4%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
      1.81            +0.3        2.13 ±  2%  perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter
      0.58 ± 45%      +0.4        0.94 ±  5%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.wait_woken
      2.38            +0.4        2.81 ±  2%  perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state
      2.52            +0.5        2.99 ±  2%  perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      2.18 ±  7%      +0.5        2.68 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      2.23 ±  7%      +0.5        2.73 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
      2.00 ±  9%      +0.5        2.55 ±  2%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
      3.10            +0.6        3.66 ±  2%  perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      0.00            +0.6        0.57        perf-profile.calltrace.cycles-pp.__alloc_skb.tcp_stream_alloc_skb.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage
      3.58            +0.6        4.20        perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.00            +0.7        0.69        perf-profile.calltrace.cycles-pp.read_tsc.ktime_get.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      0.00            +0.8        0.75        perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage.inet_sendpage
      5.44 ±  3%      +0.8        6.29        perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      7.23 ±  2%      +0.9        8.11        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
      6.00 ±  2%      +0.9        6.92        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established
      0.00            +1.0        0.96        perf-profile.calltrace.cycles-pp.iov_iter_extract_pages.skb_splice_from_iter.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage
      5.67            +1.0        6.63        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage.inet_sendpage
      0.00            +1.0        0.96 ±  2%  perf-profile.calltrace.cycles-pp.tcp_current_mss.tcp_send_mss.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage
      0.00            +1.0        0.97        perf-profile.calltrace.cycles-pp.ktime_get.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.do_tcp_sendpages
      5.67            +1.0        6.64        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage
      9.54            +1.0       10.58        perf-profile.calltrace.cycles-pp.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
      0.00            +1.1        1.07        perf-profile.calltrace.cycles-pp.skb_append_pagefrags.skb_splice_from_iter.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage
      5.99            +1.1        7.06        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      6.02            +1.1        7.10        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      0.00            +1.3        1.26        perf-profile.calltrace.cycles-pp.tcp_send_mss.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage.inet_sendpage
      8.20            +1.4        9.58        perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
      8.77            +1.4       10.18        perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      9.03            +1.5       10.48        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      9.10            +1.5       10.56        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     46.25            +1.5       47.77        perf-profile.calltrace.cycles-pp.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
      9.94            +1.6       11.56        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     11.32            +1.8       13.13        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     11.36            +1.8       13.18        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     11.34            +1.8       13.16        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     11.44            +1.9       13.32        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      0.00            +1.9        1.94        perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
     12.81            +2.0       14.84        perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
      0.00            +2.0        2.05        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.do_tcp_sendpages
      0.00            +2.2        2.22        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage
      0.00            +2.5        2.46        perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage.inet_sendpage
     37.23            +2.8       39.99        perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      0.00            +2.8        2.76        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage
     37.11            +2.8       39.88        perf-profile.calltrace.cycles-pp.generic_splice_sendpage.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile
     36.69            +2.8       39.49        perf-profile.calltrace.cycles-pp.__splice_from_pipe.generic_splice_sendpage.direct_splice_actor.splice_direct_to_actor.do_splice_direct
      0.00            +2.9        2.94        perf-profile.calltrace.cycles-pp.skb_splice_from_iter.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage.inet_sendpage
     34.28            +2.9       37.22        perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.generic_splice_sendpage
     34.79            +3.0       37.74        perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.generic_splice_sendpage.direct_splice_actor.splice_direct_to_actor
      0.00            +3.0        2.95        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage.inet_sendpage
     34.47            +3.0       37.43        perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.generic_splice_sendpage.direct_splice_actor
     33.91            +3.0       36.88        perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe
     33.42            +3.0       36.44        perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage
     16.84            +3.2       20.00        perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
      0.00            +3.4        3.35 ±  2%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked
      0.00            +5.7        5.66 ±  2%  perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked.do_tcp_sendpages
      0.00            +5.8        5.82 ±  2%  perf-profile.calltrace.cycles-pp.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage
      0.00            +6.0        6.04 ±  2%  perf-profile.calltrace.cycles-pp.tcp_wmem_schedule.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage.inet_sendpage
      0.00           +18.9       18.86        perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.do_tcp_sendpages.tcp_sendpage.inet_sendpage.kernel_sendpage
     10.85           -10.8        0.00        perf-profile.children.cycles-pp.tcp_build_frag
     31.93            -2.7       29.26        perf-profile.children.cycles-pp.tcp_recvmsg
     32.01            -2.7       29.35        perf-profile.children.cycles-pp.inet_recvmsg
     32.24            -2.7       29.57        perf-profile.children.cycles-pp.sock_recvmsg
     32.72            -2.7       30.06        perf-profile.children.cycles-pp.__sys_recvfrom
     32.82            -2.6       30.17        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     34.07            -2.6       31.52        perf-profile.children.cycles-pp.recv
     34.30            -2.5       31.76        perf-profile.children.cycles-pp.process_requests
     29.47            -2.1       27.41        perf-profile.children.cycles-pp.tcp_recvmsg_locked
      6.95 ±  2%      -2.0        4.97        perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
      7.70            -1.8        5.92        perf-profile.children.cycles-pp.__sk_mem_schedule
      7.88            -1.8        6.10        perf-profile.children.cycles-pp.tcp_wmem_schedule
      7.56            -1.8        5.79 ±  2%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
     85.01            -1.8       83.26        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     84.48            -1.7       82.78        perf-profile.children.cycles-pp.do_syscall_64
     11.31            -1.4        9.96        perf-profile.children.cycles-pp.skb_copy_datagram_iter
     11.29            -1.4        9.94        perf-profile.children.cycles-pp.__skb_datagram_iter
      6.83            -1.2        5.60        perf-profile.children.cycles-pp._raw_spin_lock_bh
      7.47            -1.1        6.36        perf-profile.children.cycles-pp.generic_file_splice_read
      3.32 ±  4%      -1.0        2.27 ±  4%  perf-profile.children.cycles-pp.try_charge_memcg
      6.59            -1.0        5.56        perf-profile.children.cycles-pp.filemap_read
      5.73            -1.0        4.78        perf-profile.children.cycles-pp.lock_sock_nested
      7.10            -0.9        6.21        perf-profile.children.cycles-pp._copy_to_iter
      6.72            -0.8        5.87        perf-profile.children.cycles-pp.copyout
      4.60            -0.8        3.80        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     15.14            -0.7       14.44        perf-profile.children.cycles-pp.net_rx_action
     13.49            -0.7       12.80        perf-profile.children.cycles-pp.process_backlog
     16.41            -0.7       15.72        perf-profile.children.cycles-pp.__local_bh_enable_ip
     11.88            -0.7       11.19        perf-profile.children.cycles-pp.tcp_v4_rcv
     13.53            -0.7       12.84        perf-profile.children.cycles-pp.__napi_poll
      2.93 ±  2%      -0.7        2.24        perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
     15.80            -0.7       15.11        perf-profile.children.cycles-pp.__do_softirq
     12.10            -0.7       11.41        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     13.11            -0.7       12.42        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     12.14            -0.7       11.46        perf-profile.children.cycles-pp.ip_local_deliver_finish
     15.90            -0.7       15.22        perf-profile.children.cycles-pp.do_softirq
      1.53 ±  4%      -0.5        0.99 ±  5%  perf-profile.children.cycles-pp.page_counter_try_charge
     17.60            -0.5       17.06        perf-profile.children.cycles-pp.__dev_queue_xmit
     17.96            -0.5       17.44        perf-profile.children.cycles-pp.ip_finish_output2
      3.14            -0.5        2.64        perf-profile.children.cycles-pp.filemap_get_pages
      2.92            -0.5        2.44        perf-profile.children.cycles-pp.filemap_get_read_batch
      1.54 ±  3%      -0.4        1.10 ±  3%  perf-profile.children.cycles-pp.refill_stock
      1.27 ±  5%      -0.4        0.84 ±  4%  perf-profile.children.cycles-pp.drain_stock
      1.24 ±  5%      -0.4        0.81 ±  4%  perf-profile.children.cycles-pp.page_counter_uncharge
     18.99            -0.4       18.58        perf-profile.children.cycles-pp.__ip_queue_xmit
      3.26            -0.4        2.87        perf-profile.children.cycles-pp.simple_copy_to_iter
      3.17            -0.4        2.78        perf-profile.children.cycles-pp.__check_object_size
      4.39            -0.4        4.01        perf-profile.children.cycles-pp.tcp_data_queue
      3.92            -0.4        3.54        perf-profile.children.cycles-pp.tcp_ack
      2.89            -0.4        2.52        perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      2.66            -0.3        2.32        perf-profile.children.cycles-pp.check_heap_object
     16.45            -0.3       16.13        perf-profile.children.cycles-pp.tcp_rcv_established
     16.65            -0.3       16.36        perf-profile.children.cycles-pp.tcp_v4_do_rcv
     20.21            -0.3       19.92        perf-profile.children.cycles-pp.__tcp_transmit_skb
      1.34            -0.2        1.13        perf-profile.children.cycles-pp.copy_page_to_iter_pipe
      0.78 ±  7%      -0.2        0.62 ±  5%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      1.23            -0.1        1.08        perf-profile.children.cycles-pp.security_file_permission
      1.34            -0.1        1.20        perf-profile.children.cycles-pp.touch_atime
      1.08            -0.1        0.96        perf-profile.children.cycles-pp.atime_needs_update
      0.97            -0.1        0.86        perf-profile.children.cycles-pp.apparmor_file_permission
      0.56            -0.1        0.46 ±  2%  perf-profile.children.cycles-pp._copy_from_user
      0.79 ±  2%      -0.1        0.70        perf-profile.children.cycles-pp.page_cache_pipe_buf_release
      0.31 ±  6%      -0.1        0.22 ±  6%  perf-profile.children.cycles-pp.propagate_protected_usage
      0.74            -0.1        0.65        perf-profile.children.cycles-pp.do_splice_to
      1.38            -0.1        1.31        perf-profile.children.cycles-pp.tcp_send_mss
      0.45            -0.1        0.39 ±  3%  perf-profile.children.cycles-pp.current_time
      0.41            -0.1        0.35        perf-profile.children.cycles-pp.__virt_addr_valid
      0.76            -0.1        0.70        perf-profile.children.cycles-pp.__cond_resched
      0.61            -0.1        0.56        perf-profile.children.cycles-pp.__entry_text_start
      0.63            -0.1        0.58        perf-profile.children.cycles-pp.__fget_light
      0.51            -0.0        0.46 ±  2%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.34            -0.0        0.30 ±  2%  perf-profile.children.cycles-pp.__put_user_8
      0.27 ±  2%      -0.0        0.23 ±  3%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.42            -0.0        0.38        perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.48 ±  2%      -0.0        0.44        perf-profile.children.cycles-pp.netperf_sendfile
      0.30 ±  2%      -0.0        0.26        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.30            -0.0        0.27        perf-profile.children.cycles-pp.aa_file_perm
      0.97            -0.0        0.94        perf-profile.children.cycles-pp.napi_consume_skb
      0.24            -0.0        0.21 ±  2%  perf-profile.children.cycles-pp.sanity
      0.19            -0.0        0.16 ±  2%  perf-profile.children.cycles-pp.rw_verify_area
      0.17 ±  2%      -0.0        0.15        perf-profile.children.cycles-pp.fsnotify_perm
      0.16 ±  2%      -0.0        0.14 ±  2%  perf-profile.children.cycles-pp.skb_release_head_state
      0.14 ±  2%      -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.47            -0.0        0.45        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.28            -0.0        0.26 ±  2%  perf-profile.children.cycles-pp.rcu_all_qs
      0.16 ±  3%      -0.0        0.13 ±  3%  perf-profile.children.cycles-pp.folio_mark_accessed
      0.16 ±  2%      -0.0        0.14 ±  3%  perf-profile.children.cycles-pp.skb_clone
      0.13 ±  4%      -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.check_stack_object
      0.20 ±  2%      -0.0        0.18 ±  3%  perf-profile.children.cycles-pp._raw_spin_unlock_bh
      0.16 ±  3%      -0.0        0.14        perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.make_vfsuid
      0.22 ±  2%      -0.0        0.20 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.30            -0.0        0.29 ±  2%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.10 ±  5%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xas_start
      0.07            -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.tcp_leave_memory_pressure
      0.10            -0.0        0.09        perf-profile.children.cycles-pp.is_vmalloc_addr
      0.05            +0.0        0.06        perf-profile.children.cycles-pp.ct_nmi_exit
      0.05            +0.0        0.06        perf-profile.children.cycles-pp.x2apic_send_IPI
      0.05            +0.0        0.06        perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.11            +0.0        0.12        perf-profile.children.cycles-pp.resched_curr
      0.06            +0.0        0.07        perf-profile.children.cycles-pp.rb_next
      0.07 ±  5%      +0.0        0.08        perf-profile.children.cycles-pp.switch_fpu_return
      0.12 ±  3%      +0.0        0.13        perf-profile.children.cycles-pp.__get_task_ioprio
      0.13 ±  2%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.06 ±  6%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.irqentry_exit
      0.06 ±  7%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__tcp_cleanup_rbuf
      0.06            +0.0        0.07 ±  6%  perf-profile.children.cycles-pp.reweight_entity
      0.12 ±  3%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.finish_task_switch
      0.11 ±  3%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.18 ±  2%      +0.0        0.20 ±  2%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.13 ±  3%      +0.0        0.15        perf-profile.children.cycles-pp.check_preempt_curr
      0.08 ±  6%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.07            +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.17 ±  2%      +0.0        0.19        perf-profile.children.cycles-pp.kfree_skbmem
      0.10 ±  4%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.09 ±  5%      +0.0        0.11 ±  3%  perf-profile.children.cycles-pp.os_xsave
      0.50            +0.0        0.52        perf-profile.children.cycles-pp.kmem_cache_free
      0.22 ±  3%      +0.0        0.24 ±  2%  perf-profile.children.cycles-pp.__slab_free
      0.09            +0.0        0.11        perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.36            +0.0        0.38 ±  2%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.17 ±  2%      +0.0        0.19        perf-profile.children.cycles-pp.__update_load_avg_se
      0.11 ±  3%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.tcp_skb_entail
      0.06 ±  8%      +0.0        0.08        perf-profile.children.cycles-pp.nr_iowait_cpu
      0.46 ±  2%      +0.0        0.48        perf-profile.children.cycles-pp.irqtime_account_irq
      0.39            +0.0        0.42        perf-profile.children.cycles-pp.netif_rx_internal
      0.42            +0.0        0.44 ±  2%  perf-profile.children.cycles-pp.__netif_rx
      0.15 ±  3%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.perf_tp_event
      0.27 ±  2%      +0.0        0.30 ±  2%  perf-profile.children.cycles-pp.__switch_to
      0.17 ±  2%      +0.0        0.20 ±  2%  perf-profile.children.cycles-pp.ip_send_check
      0.11 ±  3%      +0.0        0.14        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.04 ± 45%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.tick_nohz_stop_idle
      0.16 ±  3%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.llist_add_batch
      0.17            +0.0        0.20 ±  2%  perf-profile.children.cycles-pp.__x2apic_send_IPI_dest
      0.16 ±  6%      +0.0        0.19 ±  5%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.22 ±  2%      +0.0        0.25 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.18 ±  2%      +0.0        0.21 ±  3%  perf-profile.children.cycles-pp.__smp_call_single_queue
      0.24 ±  3%      +0.0        0.27        perf-profile.children.cycles-pp.__ip_local_out
      0.20 ±  2%      +0.0        0.24        perf-profile.children.cycles-pp.select_idle_sibling
      0.18 ±  2%      +0.0        0.21 ±  3%  perf-profile.children.cycles-pp.available_idle_cpu
      1.06            +0.0        1.09        perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.16 ±  6%      +0.0        0.19 ±  5%  perf-profile.children.cycles-pp.update_blocked_averages
      0.32            +0.0        0.36 ±  2%  perf-profile.children.cycles-pp.update_curr
      0.18 ±  2%      +0.0        0.21 ±  3%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.96            +0.0        1.00        perf-profile.children.cycles-pp.loopback_xmit
      0.28 ±  2%      +0.0        0.31        perf-profile.children.cycles-pp.ip_local_out
      0.12 ±  4%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.tcp_downgrade_zcopy_pure
      0.19            +0.0        0.22 ±  2%  perf-profile.children.cycles-pp.llist_reverse_order
      0.18 ±  2%      +0.0        0.22 ±  2%  perf-profile.children.cycles-pp.remove_wait_queue
      0.19 ±  2%      +0.0        0.22 ±  2%  perf-profile.children.cycles-pp.irqentry_enter
      0.20 ±  3%      +0.0        0.24        perf-profile.children.cycles-pp.___perf_sw_event
      0.02 ±141%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.44            +0.0        0.48        perf-profile.children.cycles-pp.native_sched_clock
      0.20 ±  2%      +0.0        0.24        perf-profile.children.cycles-pp.update_rq_clock
      0.15 ±  4%      +0.0        0.19 ±  2%  perf-profile.children.cycles-pp.tick_irq_enter
      0.27 ±  2%      +0.0        0.31 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.28 ±  2%      +0.0        0.32 ±  2%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.22            +0.0        0.26        perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.20 ±  4%      +0.0        0.24 ±  5%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.22 ±  3%      +0.0        0.27 ±  5%  perf-profile.children.cycles-pp.find_busiest_group
      0.60            +0.0        0.65        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.25 ±  3%      +0.0        0.30        perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.29 ±  2%      +0.0        0.34 ±  2%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.23            +0.0        0.28 ±  2%  perf-profile.children.cycles-pp.set_next_entity
      0.09 ± 18%      +0.0        0.14 ± 25%  perf-profile.children.cycles-pp.x86_64_start_kernel
      0.09 ± 18%      +0.0        0.14 ± 25%  perf-profile.children.cycles-pp.x86_64_start_reservations
      0.09 ± 18%      +0.0        0.14 ± 25%  perf-profile.children.cycles-pp.start_kernel
      0.09 ± 18%      +0.0        0.14 ± 25%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.09 ± 18%      +0.0        0.14 ± 25%  perf-profile.children.cycles-pp.rest_init
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.update_min_vruntime
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.pipe_unlock
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.error_entry
      0.01 ±223%      +0.1        0.06        perf-profile.children.cycles-pp.perf_trace_buf_update
      0.21 ±  3%      +0.1        0.26 ±  2%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.49            +0.1        0.54        perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__calc_delta
      0.32            +0.1        0.38 ±  2%  perf-profile.children.cycles-pp.select_task_rq
      0.10            +0.1        0.16 ±  2%  perf-profile.children.cycles-pp.tcp_chrono_stop
      0.30 ±  3%      +0.1        0.36 ±  4%  perf-profile.children.cycles-pp.load_balance
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.32 ±  2%      +0.1        0.39        perf-profile.children.cycles-pp.prepare_task_switch
      0.38            +0.1        0.45        perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.61            +0.1        0.70        perf-profile.children.cycles-pp.enqueue_entity
      0.60            +0.1        0.70        perf-profile.children.cycles-pp.update_load_avg
      0.52            +0.1        0.62 ±  2%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.67            +0.1        0.77        perf-profile.children.cycles-pp.dequeue_entity
      0.74            +0.1        0.86        perf-profile.children.cycles-pp.dequeue_task_fair
      0.00            +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.iov_iter_bvec
      0.61 ±  2%      +0.1        0.74        perf-profile.children.cycles-pp.menu_select
      1.00            +0.2        1.16        perf-profile.children.cycles-pp.schedule_idle
      0.40 ±  2%      +0.2        0.56        perf-profile.children.cycles-pp.tcp_rate_check_app_limited
      0.14 ±  5%      +0.2        0.30        perf-profile.children.cycles-pp.tcp_mtu_probe
      0.86            +0.2        1.02 ±  3%  perf-profile.children.cycles-pp.activate_task
      0.83            +0.2        0.99 ±  3%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.74 ±  4%      +0.2        0.92 ±  5%  perf-profile.children.cycles-pp.newidle_balance
      1.00            +0.2        1.19 ±  2%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.12            +0.2        0.32        perf-profile.children.cycles-pp.tcp_push
      1.28            +0.2        1.51 ±  2%  perf-profile.children.cycles-pp.sched_ttwu_pending
      1.29            +0.2        1.53        perf-profile.children.cycles-pp.try_to_wake_up
      1.08 ±  2%      +0.2        1.33 ±  4%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.46            +0.3        1.72        perf-profile.children.cycles-pp.__wake_up_common
      0.24 ±  2%      +0.3        0.52        perf-profile.children.cycles-pp.tcp_tso_segs
      1.52            +0.3        1.80        perf-profile.children.cycles-pp.__wake_up_common_lock
      1.70            +0.3        2.00        perf-profile.children.cycles-pp.sock_def_readable
      1.85            +0.3        2.18 ±  2%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.00            +0.4        0.37 ±  2%  perf-profile.children.cycles-pp.iov_iter_advance
     11.96            +0.4       12.36        perf-profile.children.cycles-pp.release_sock
      2.27            +0.4        2.68 ±  2%  perf-profile.children.cycles-pp.schedule
      2.32            +0.4        2.74 ±  2%  perf-profile.children.cycles-pp.schedule_timeout
      0.70            +0.4        1.14        perf-profile.children.cycles-pp.read_tsc
      2.42            +0.5        2.87 ±  2%  perf-profile.children.cycles-pp.sysvec_call_function_single
      2.53            +0.5        3.00 ±  2%  perf-profile.children.cycles-pp.wait_woken
      3.14            +0.5        3.69 ±  2%  perf-profile.children.cycles-pp.__schedule
      3.12            +0.6        3.68 ±  2%  perf-profile.children.cycles-pp.sk_wait_data
      1.03 ±  2%      +0.6        1.62        perf-profile.children.cycles-pp.ktime_get
     10.34            +0.7       11.09        perf-profile.children.cycles-pp.__release_sock
      0.00            +1.0        1.01        perf-profile.children.cycles-pp.iov_iter_extract_pages
      0.00            +1.1        1.09        perf-profile.children.cycles-pp.skb_append_pagefrags
      7.76            +1.3        9.05        perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      8.81            +1.4       10.26        perf-profile.children.cycles-pp.acpi_safe_halt
      8.84            +1.4       10.29        perf-profile.children.cycles-pp.acpi_idle_enter
      9.11            +1.5       10.60        perf-profile.children.cycles-pp.cpuidle_enter_state
      9.17            +1.5       10.68        perf-profile.children.cycles-pp.cpuidle_enter
     46.29            +1.5       47.80        perf-profile.children.cycles-pp.splice_direct_to_actor
     10.04            +1.7       11.70        perf-profile.children.cycles-pp.cpuidle_idle_call
     11.36            +1.8       13.18        perf-profile.children.cycles-pp.start_secondary
     11.42            +1.9       13.29        perf-profile.children.cycles-pp.do_idle
     11.44            +1.9       13.32        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     11.44            +1.9       13.32        perf-profile.children.cycles-pp.cpu_startup_entry
     12.44            +2.2       14.68        perf-profile.children.cycles-pp.tcp_write_xmit
      9.81            +2.5       12.35        perf-profile.children.cycles-pp.__tcp_push_pending_frames
     37.26            +2.8       40.02        perf-profile.children.cycles-pp.direct_splice_actor
     37.16            +2.8       39.93        perf-profile.children.cycles-pp.generic_splice_sendpage
     36.79            +2.8       39.58        perf-profile.children.cycles-pp.__splice_from_pipe
     34.86            +2.9       37.79        perf-profile.children.cycles-pp.pipe_to_sendpage
     34.33            +2.9       37.27        perf-profile.children.cycles-pp.kernel_sendpage
     34.53            +2.9       37.48        perf-profile.children.cycles-pp.sock_sendpage
     34.02            +3.0       36.98        perf-profile.children.cycles-pp.inet_sendpage
     33.58            +3.0       36.57        perf-profile.children.cycles-pp.tcp_sendpage
      0.00            +3.0        3.04        perf-profile.children.cycles-pp.skb_splice_from_iter
     17.08            +3.1       20.14        perf-profile.children.cycles-pp.do_tcp_sendpages
      0.00           +19.1       19.14        perf-profile.children.cycles-pp.tcp_sendmsg_locked
      3.46 ±  5%      -0.9        2.54 ±  4%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      6.67            -0.8        5.83        perf-profile.self.cycles-pp.copyout
      4.58            -0.8        3.78        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.37 ±  4%      -0.5        0.87 ±  5%  perf-profile.self.cycles-pp.page_counter_try_charge
      1.72 ±  3%      -0.5        1.22 ±  3%  perf-profile.self.cycles-pp.try_charge_memcg
      2.73            -0.5        2.25 ±  2%  perf-profile.self.cycles-pp.__sk_mem_raise_allocated
      2.71            -0.5        2.25        perf-profile.self.cycles-pp.filemap_get_read_batch
      2.18            -0.4        1.76        perf-profile.self.cycles-pp._raw_spin_lock_bh
      1.09 ±  5%      -0.4        0.70 ±  5%  perf-profile.self.cycles-pp.page_counter_uncharge
      2.15 ±  2%      -0.3        1.88        perf-profile.self.cycles-pp.check_heap_object
      1.31            -0.2        1.11        perf-profile.self.cycles-pp.filemap_read
      1.10            -0.2        0.92 ±  2%  perf-profile.self.cycles-pp.copy_page_to_iter_pipe
      0.73 ±  7%      -0.2        0.56 ±  5%  perf-profile.self.cycles-pp.mem_cgroup_uncharge_skmem
      1.02            -0.1        0.91        perf-profile.self.cycles-pp.do_sendfile
      0.54            -0.1        0.45        perf-profile.self.cycles-pp._copy_from_user
      0.64 ±  2%      -0.1        0.55 ±  2%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.30 ±  7%      -0.1        0.22 ±  6%  perf-profile.self.cycles-pp.propagate_protected_usage
      0.76 ±  2%      -0.1        0.68        perf-profile.self.cycles-pp.page_cache_pipe_buf_release
      0.67 ±  2%      -0.1        0.60 ±  2%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.56            -0.1        0.49        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.68            -0.1        0.62        perf-profile.self.cycles-pp.tcp_v4_rcv
      0.48            -0.1        0.42 ±  2%  perf-profile.self.cycles-pp.tcp_sendpage
      0.40            -0.1        0.35 ±  2%  perf-profile.self.cycles-pp.sendfile
      0.39            -0.1        0.34 ±  2%  perf-profile.self.cycles-pp._copy_to_iter
      0.70            -0.0        0.65        perf-profile.self.cycles-pp.__splice_from_pipe
      0.38            -0.0        0.33 ±  2%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.34            -0.0        0.29        perf-profile.self.cycles-pp.__put_user_8
      0.60            -0.0        0.56        perf-profile.self.cycles-pp.__fget_light
      0.48            -0.0        0.43        perf-profile.self.cycles-pp.atime_needs_update
      0.48            -0.0        0.44 ±  2%  perf-profile.self.cycles-pp.__fsnotify_parent
      1.00            -0.0        0.95        perf-profile.self.cycles-pp.skb_release_data
      0.41            -0.0        0.37        perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.45            -0.0        0.41        perf-profile.self.cycles-pp.__cond_resched
      0.78            -0.0        0.74 ±  2%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.31 ±  2%      -0.0        0.27 ±  2%  perf-profile.self.cycles-pp.current_time
      0.24 ±  2%      -0.0        0.20 ±  2%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.48            -0.0        0.44 ±  2%  perf-profile.self.cycles-pp.sendfile_tcp_stream
      0.43            -0.0        0.39        perf-profile.self.cycles-pp.netperf_sendfile
      0.34 ±  2%      -0.0        0.31 ±  3%  perf-profile.self.cycles-pp.__x64_sys_sendfile64
      0.22 ±  3%      -0.0        0.19        perf-profile.self.cycles-pp.filemap_get_pages
      0.25 ±  2%      -0.0        0.22 ±  3%  perf-profile.self.cycles-pp.security_file_permission
      0.40 ±  2%      -0.0        0.37 ±  2%  perf-profile.self.cycles-pp.__check_object_size
      0.29 ±  2%      -0.0        0.26        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.26 ±  2%      -0.0        0.23        perf-profile.self.cycles-pp.aa_file_perm
      0.30            -0.0        0.28        perf-profile.self.cycles-pp.tcp_send_mss
      0.33 ±  2%      -0.0        0.30 ±  2%  perf-profile.self.cycles-pp.generic_splice_sendpage
      0.21            -0.0        0.18 ±  2%  perf-profile.self.cycles-pp.sanity
      0.33            -0.0        0.30 ±  2%  perf-profile.self.cycles-pp.splice_direct_to_actor
      0.15 ±  3%      -0.0        0.12 ±  3%  perf-profile.self.cycles-pp.do_splice_to
      0.42            -0.0        0.39        perf-profile.self.cycles-pp.do_syscall_64
      0.34 ±  2%      -0.0        0.31 ±  2%  perf-profile.self.cycles-pp.inet_sendpage
      0.22            -0.0        0.20        perf-profile.self.cycles-pp.generic_file_splice_read
      0.25            -0.0        0.22 ±  2%  perf-profile.self.cycles-pp.touch_atime
      0.16 ±  2%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.skb_clone
      0.16 ±  3%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.fsnotify_perm
      0.14 ±  2%      -0.0        0.12 ±  3%  perf-profile.self.cycles-pp.rw_verify_area
      0.22 ±  3%      -0.0        0.20 ±  3%  perf-profile.self.cycles-pp.__entry_text_start
      0.11 ±  4%      -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.12 ±  4%      -0.0        0.11 ±  3%  perf-profile.self.cycles-pp.folio_mark_accessed
      0.16            -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.tcp_stream_alloc_skb
      0.14 ±  2%      -0.0        0.13 ±  3%  perf-profile.self.cycles-pp.__sk_mem_schedule
      0.14 ±  3%      -0.0        0.12        perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.30            -0.0        0.29        perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.12 ±  4%      -0.0        0.10 ±  4%  perf-profile.self.cycles-pp.simple_copy_to_iter
      0.15 ±  2%      -0.0        0.14        perf-profile.self.cycles-pp.__tcp_select_window
      0.07            -0.0        0.06        perf-profile.self.cycles-pp.tcp_rate_skb_sent
      0.06            -0.0        0.05        perf-profile.self.cycles-pp.splice_from_pipe_next
      0.14            +0.0        0.15        perf-profile.self.cycles-pp.enqueue_to_backlog
      0.05            +0.0        0.06        perf-profile.self.cycles-pp.schedule_idle
      0.06            +0.0        0.07        perf-profile.self.cycles-pp.reweight_entity
      0.06            +0.0        0.07        perf-profile.self.cycles-pp.__tcp_cleanup_rbuf
      0.11            +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.__get_task_ioprio
      0.10 ±  4%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.07            +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.05            +0.0        0.06 ±  6%  perf-profile.self.cycles-pp.select_task_rq
      0.11 ±  4%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.napi_consume_skb
      0.06            +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.sysvec_call_function_single
      0.06            +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.irq_enter_rcu
      0.11 ±  4%      +0.0        0.12        perf-profile.self.cycles-pp.perf_tp_event
      0.08            +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.finish_task_switch
      0.06 ±  8%      +0.0        0.07        perf-profile.self.cycles-pp.switch_fpu_return
      0.09            +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.os_xsave
      0.06 ±  7%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.06            +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.sk_wait_data
      0.20 ±  2%      +0.0        0.21        perf-profile.self.cycles-pp.loopback_xmit
      0.08 ±  5%      +0.0        0.10        perf-profile.self.cycles-pp.schedule
      0.16 ±  3%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.kfree_skbmem
      0.09 ±  4%      +0.0        0.11 ±  3%  perf-profile.self.cycles-pp.tcp_rcv_space_adjust
      0.09 ±  4%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.08 ±  6%      +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.06 ±  7%      +0.0        0.08        perf-profile.self.cycles-pp.cpuidle_enter_state
      0.15 ±  3%      +0.0        0.17        perf-profile.self.cycles-pp.do_softirq
      0.13 ±  3%      +0.0        0.15 ±  2%  perf-profile.self.cycles-pp.do_idle
      0.15            +0.0        0.17 ±  4%  perf-profile.self.cycles-pp.update_curr
      0.08            +0.0        0.10 ±  3%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.22 ±  3%      +0.0        0.24 ±  2%  perf-profile.self.cycles-pp.__slab_free
      0.15 ±  2%      +0.0        0.17 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.11            +0.0        0.13        perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.11 ±  4%      +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.24 ±  2%      +0.0        0.26        perf-profile.self.cycles-pp.__mod_timer
      0.46            +0.0        0.48        perf-profile.self.cycles-pp.tcp_current_mss
      0.10 ±  3%      +0.0        0.12        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.09 ±  5%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.asm_sysvec_call_function_single
      0.17 ±  2%      +0.0        0.19        perf-profile.self.cycles-pp.sock_def_readable
      0.06 ±  9%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.11 ±  3%      +0.0        0.13 ±  2%  perf-profile.self.cycles-pp.dequeue_entity
      0.08 ±  6%      +0.0        0.10        perf-profile.self.cycles-pp.skb_attempt_defer_free
      0.08 ±  5%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.tcp_downgrade_zcopy_pure
      0.49            +0.0        0.51        perf-profile.self.cycles-pp.kmem_cache_free
      0.12 ±  6%      +0.0        0.15 ±  5%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.64            +0.0        0.67        perf-profile.self.cycles-pp.tcp_rcv_established
      0.16 ±  4%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.ip_send_check
      0.14 ±  4%      +0.0        0.17 ±  2%  perf-profile.self.cycles-pp.irqentry_enter
      0.10 ±  3%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.update_rq_clock
      0.16 ±  3%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.llist_add_batch
      0.18 ±  2%      +0.0        0.20 ±  2%  perf-profile.self.cycles-pp.available_idle_cpu
      0.12 ±  3%      +0.0        0.15 ±  2%  perf-profile.self.cycles-pp.prepare_task_switch
      0.22 ±  2%      +0.0        0.24 ±  3%  perf-profile.self.cycles-pp.enqueue_entity
      0.13            +0.0        0.16 ±  3%  perf-profile.self.cycles-pp.__wake_up_common
      0.26            +0.0        0.29 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.21 ±  2%      +0.0        0.24 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      0.18 ±  2%      +0.0        0.21 ±  2%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.08 ±  6%      +0.0        0.11 ±  3%  perf-profile.self.cycles-pp.tcp_chrono_stop
      0.17 ±  2%      +0.0        0.20        perf-profile.self.cycles-pp.__x2apic_send_IPI_dest
      0.17 ±  4%      +0.0        0.21 ±  2%  perf-profile.self.cycles-pp.___perf_sw_event
      0.19            +0.0        0.22        perf-profile.self.cycles-pp.llist_reverse_order
      0.20 ±  2%      +0.0        0.23 ±  3%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.22 ±  3%      +0.0        0.25 ±  3%  perf-profile.self.cycles-pp.update_load_avg
      0.15 ±  3%      +0.0        0.19 ±  3%  perf-profile.self.cycles-pp.try_to_wake_up
      0.45            +0.0        0.49 ±  2%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.42            +0.0        0.46        perf-profile.self.cycles-pp.native_sched_clock
      0.22 ±  2%      +0.0        0.26 ±  2%  perf-profile.self.cycles-pp.wait_woken
      0.34            +0.0        0.38        perf-profile.self.cycles-pp.__schedule
      0.22 ±  2%      +0.0        0.27        perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.59            +0.0        0.64        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.29 ±  2%      +0.0        0.34 ±  2%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__update_blocked_fair
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__calc_delta
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.14 ±  3%      +0.1        0.19 ±  3%  perf-profile.self.cycles-pp.__release_sock
      0.43            +0.1        0.49        perf-profile.self.cycles-pp.__ip_queue_xmit
      0.24 ±  2%      +0.1        0.30        perf-profile.self.cycles-pp.menu_select
      0.22 ±  5%      +0.1        0.28 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.26 ±  4%      +0.1        0.34 ± 10%  perf-profile.self.cycles-pp.newidle_balance
      0.08 ±  5%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.__tcp_push_pending_frames
      0.00            +0.1        0.10        perf-profile.self.cycles-pp.iov_iter_bvec
      0.36 ±  3%      +0.1        0.50        perf-profile.self.cycles-pp.tcp_rate_check_app_limited
      0.12 ±  4%      +0.1        0.26        perf-profile.self.cycles-pp.tcp_mtu_probe
      0.34 ±  7%      +0.2        0.49        perf-profile.self.cycles-pp.ktime_get
      0.75            +0.2        0.91        perf-profile.self.cycles-pp.do_tcp_sendpages
      0.10            +0.2        0.30 ±  2%  perf-profile.self.cycles-pp.tcp_push
      0.23 ±  3%      +0.3        0.49 ±  2%  perf-profile.self.cycles-pp.tcp_tso_segs
      0.00            +0.3        0.34 ±  2%  perf-profile.self.cycles-pp.iov_iter_advance
      0.67            +0.4        1.10        perf-profile.self.cycles-pp.read_tsc
      0.00            +0.6        0.64 ±  2%  perf-profile.self.cycles-pp.iov_iter_extract_pages
      0.80            +0.7        1.48        perf-profile.self.cycles-pp.tcp_write_xmit
      5.90            +0.9        6.80        perf-profile.self.cycles-pp.acpi_safe_halt
      0.00            +1.0        0.98        perf-profile.self.cycles-pp.skb_splice_from_iter
      0.00            +1.0        1.05        perf-profile.self.cycles-pp.skb_append_pagefrags
      0.00            +1.8        1.79        perf-profile.self.cycles-pp.tcp_sendmsg_locked




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



View attachment "config-6.4.0-rc1-00352-g77f5a42b2db3" of type "text/plain" (158726 bytes)

View attachment "job-script" of type "text/plain" (8855 bytes)

View attachment "job.yaml" of type "text/plain" (6211 bytes)

View attachment "reproduce" of type "text/plain" (4499 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ