lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Tue, 12 Jan 2021 22:21:09 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     "Paul E. McKenney" <paulmck@...nel.org>
Cc:     Borislav Petkov <bp@...e.de>, Jonathan Lemon <bsd@...com>,
        Tony Luck <tony.luck@...el.com>,
        LKML <linux-kernel@...r.kernel.org>, x86@...nel.org,
        lkp@...ts.01.org, lkp@...el.com, ying.huang@...el.com,
        feng.tang@...el.com, zhengjun.xing@...el.com
Subject: [x86/mce]  7bb39313cd:  netperf.Throughput_tps -4.5% regression


Greeting,

FYI, we noticed a -4.5% regression of netperf.Throughput_tps due to commit:


commit: 7bb39313cd6239e7eb95198950a02b4ad2a08316 ("x86/mce: Make mce_timed_out() identify holdout CPUs")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git ras/core


in testcase: netperf
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 16
	cluster: cs-localhost
	test: TCP_CRR
	cpufreq_governor: performance
	ucode: 0x5003003

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap3/TCP_CRR/netperf/0x5003003

commit: 
  v5.11-rc2
  7bb39313cd ("x86/mce: Make mce_timed_out() identify holdout CPUs")

       v5.11-rc2 7bb39313cd6239e7eb95198950a 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    267859            -4.5%     255695        netperf.Throughput_total_tps
     16741            -4.5%      15980        netperf.Throughput_tps
    393117            +4.9%     412186 ±  2%  netperf.time.involuntary_context_switches
     12382 ±  2%      -3.6%      11931 ±  2%  netperf.time.minor_page_faults
     50.53            -2.8%      49.14        netperf.time.user_time
 1.571e+08            -4.9%  1.495e+08        netperf.time.voluntary_context_switches
  80357764            -4.5%   76708698        netperf.workload
 1.212e+08            -9.1%  1.102e+08        cpuidle.POLL.usage
    375068 ±  4%      -4.8%     357044 ±  3%  meminfo.Mapped
   1861784            -1.1%    1840821        vmstat.system.cs
     32.17 ±  4%      -2.7%      31.32 ±  3%  boot-time.boot
      5308 ±  4%      -3.1%       5143 ±  4%  boot-time.idle
     40216 ±  3%     -26.0%      29750 ±  7%  slabinfo.ep_head.active_objs
     40216 ±  3%     -26.0%      29750 ±  7%  slabinfo.ep_head.num_objs
    988814 ±  7%     -17.6%     814384 ±  5%  numa-numastat.node0.local_node
   1075387 ±  6%     -17.5%     887013 ±  7%  numa-numastat.node0.numa_hit
   1813214 ± 46%     -33.5%    1206266 ± 61%  numa-numastat.node1.local_node
   1856724 ± 47%     -30.4%    1292904 ± 57%  numa-numastat.node1.numa_hit
     86696           -58.9%      35670 ±103%  numa-numastat.node3.other_node
    611.50           +77.7%       1086 ± 45%  numa-meminfo.node0.PageTables
    187124 ± 20%     -47.4%      98509 ± 59%  numa-meminfo.node1.Inactive
    187124 ± 20%     -47.4%      98509 ± 59%  numa-meminfo.node1.Inactive(anon)
   2310503 ± 53%     -34.8%    1506795 ± 69%  numa-meminfo.node1.MemUsed
     27633 ± 90%    +261.7%      99938 ± 34%  numa-meminfo.node2.AnonHugePages
     37838           +49.5%      56583 ± 15%  numa-meminfo.node3.KReclaimable
     37838           +49.5%      56583 ± 15%  numa-meminfo.node3.SReclaimable
    152.50           +78.7%     272.50 ± 45%  numa-vmstat.node0.nr_page_table_pages
    887968 ±  6%     -12.8%     773979 ±  4%  numa-vmstat.node0.numa_local
     46671 ± 20%     -47.2%      24622 ± 59%  numa-vmstat.node1.nr_inactive_anon
     46672 ± 20%     -47.2%      24622 ± 59%  numa-vmstat.node1.nr_zone_inactive_anon
   1420034 ± 15%     -14.2%    1217944 ± 17%  numa-vmstat.node2.numa_hit
   1290117 ± 13%     -16.2%    1081736 ± 20%  numa-vmstat.node2.numa_local
      9461           +49.5%      14147 ± 15%  numa-vmstat.node3.nr_slab_reclaimable
    857579            -2.0%     840331        proc-vmstat.nr_active_anon
   1151700            -2.0%    1128251        proc-vmstat.nr_file_pages
     95738 ±  5%      -4.8%      91134 ±  3%  proc-vmstat.nr_mapped
    909635            -2.6%     886183        proc-vmstat.nr_shmem
    857579            -2.0%     840331        proc-vmstat.nr_zone_active_anon
     87234 ±  5%     +16.3%     101415 ±  6%  proc-vmstat.numa_hint_faults_local
   6316423            -9.6%    5709525 ±  3%  proc-vmstat.numa_hit
   6056034           -10.0%    5449405 ±  3%  proc-vmstat.numa_local
    762995            -8.5%     698109 ±  5%  proc-vmstat.pgactivate
   7399790           -11.8%    6525473 ±  3%  proc-vmstat.pgalloc_normal
   5171910           -15.7%    4359981 ±  4%  proc-vmstat.pgfree
      2754 ± 22%     -41.4%       1613 ± 29%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    224619 ±  6%     -32.2%     152200 ± 33%  sched_debug.cfs_rq:/.MIN_vruntime.max
     23108 ±  7%     -37.5%      14443 ± 28%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     15019 ±  2%     -37.3%       9416 ± 35%  sched_debug.cfs_rq:/.load.avg
    755415           -60.0%     302070 ± 57%  sched_debug.cfs_rq:/.load.max
     83126           -53.8%      38400 ± 51%  sched_debug.cfs_rq:/.load.stddev
      2754 ± 22%     -41.4%       1613 ± 29%  sched_debug.cfs_rq:/.max_vruntime.avg
    224619 ±  6%     -32.2%     152200 ± 33%  sched_debug.cfs_rq:/.max_vruntime.max
     23108 ±  7%     -37.5%      14443 ± 28%  sched_debug.cfs_rq:/.max_vruntime.stddev
      0.13 ±  3%      -7.5%       0.12 ±  5%  sched_debug.cfs_rq:/.nr_running.avg
      1.67           -27.5%       1.21 ± 11%  sched_debug.cfs_rq:/.nr_running.max
      0.35            -7.7%       0.32 ±  3%  sched_debug.cfs_rq:/.nr_running.stddev
     62.46 ±  6%      -4.9%      59.40 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      1.67           -27.5%       1.21 ± 11%  sched_debug.cpu.nr_running.max
     16.14            +5.8%      17.07        perf-stat.i.MPKI
 7.537e+09            -3.7%  7.259e+09        perf-stat.i.branch-instructions
      1.92            +0.1        1.98 ±  2%  perf-stat.i.branch-miss-rate%
 1.429e+08            -3.0%  1.387e+08        perf-stat.i.branch-misses
   1881443            -1.1%    1860483        perf-stat.i.context-switches
      2.24            +4.4%       2.34        perf-stat.i.cpi
    644.64           -12.2%     565.89 ±  2%  perf-stat.i.cpu-migrations
   1262820           -17.5%    1041583 ±  7%  perf-stat.i.dTLB-load-misses
 1.122e+10            -3.6%  1.082e+10        perf-stat.i.dTLB-loads
      0.00 ±  4%      +0.0        0.00 ± 28%  perf-stat.i.dTLB-store-miss-rate%
 6.576e+09            -3.7%  6.331e+09        perf-stat.i.dTLB-stores
 1.105e+08            -3.0%  1.072e+08        perf-stat.i.iTLB-load-misses
 3.756e+10            -3.7%  3.617e+10        perf-stat.i.instructions
      0.45            -3.6%       0.44        perf-stat.i.ipc
      0.49 ±  2%     -47.0%       0.26 ± 17%  perf-stat.i.metric.K/sec
    135.38            -3.6%     130.49        perf-stat.i.metric.M/sec
  22614378            +5.1%   23778202        perf-stat.i.node-load-misses
  12909912            -5.7%   12170648        perf-stat.i.node-store-misses
    839965            -9.3%     762062 ±  3%  perf-stat.i.node-stores
     16.02            +2.7%      16.45        perf-stat.overall.MPKI
      2.20            +3.7%       2.28        perf-stat.overall.cpi
      0.01            -0.0        0.01 ±  6%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  4%      +0.0        0.00 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
      0.45            -3.5%       0.44        perf-stat.overall.ipc
 7.513e+09            -3.7%  7.235e+09        perf-stat.ps.branch-instructions
 1.425e+08            -3.0%  1.382e+08        perf-stat.ps.branch-misses
   1875148            -1.1%    1854065        perf-stat.ps.context-switches
    642.62           -12.2%     564.16 ±  2%  perf-stat.ps.cpu-migrations
   1258717           -17.5%    1038404 ±  7%  perf-stat.ps.dTLB-load-misses
 1.119e+10            -3.6%  1.078e+10        perf-stat.ps.dTLB-loads
 6.555e+09            -3.7%   6.31e+09        perf-stat.ps.dTLB-stores
 1.101e+08            -3.0%  1.068e+08        perf-stat.ps.iTLB-load-misses
 3.744e+10            -3.7%  3.605e+10        perf-stat.ps.instructions
  22540456            +5.1%   23698053        perf-stat.ps.node-load-misses
  12867845            -5.7%   12129784        perf-stat.ps.node-store-misses
    838334            -9.3%     760491 ±  3%  perf-stat.ps.node-stores
 1.145e+13            -3.7%  1.102e+13        perf-stat.total.instructions
      0.01 ± 23%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.__flush_work.lru_add_drain_all.khugepaged
      0.00           -33.3%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.__fput.task_work_run.exit_to_user_mode_prepare
      0.00           -50.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
      0.05 ± 91%     -94.7%       0.00 ± 20%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
      0.02 ± 61%     -86.1%       0.00 ± 34%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.path_openat
      0.90 ± 72%     -78.0%       0.20 ±135%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.26 ± 74%    +284.8%       1.02 ± 36%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ± 23%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.__flush_work.lru_add_drain_all.khugepaged
      0.01 ±  4%   +2347.8%       0.28 ± 96%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.__release_sock.release_sock.__inet_stream_connect
      0.02 ± 35%     -60.8%       0.01 ± 53%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
      0.01           -25.0%       0.00 ± 11%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.aa_sk_perm.security_socket_setsockopt.__sys_setsockopt
      0.00          +100.0%       0.01 ± 31%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.apparmor_socket_create.security_socket_create.__sock_create
      1.10 ± 68%     -97.6%       0.03 ±144%  perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.path_openat
      2.56 ±  3%     -22.9%       1.97 ± 23%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.__alloc_pages_nodemask.alloc_pages_vma.shmem_alloc_page
     48.75 ± 11%     -81.5%       9.03 ± 67%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.smpboot_thread_fn.kthread.ret_from_fork
      1.80 ±  2%     +18.0%       2.12 ±  3%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      6.00           -66.7%       2.00 ± 50%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.smpboot_thread_fn.kthread.ret_from_fork
    252.04 ± 13%     -95.5%      11.43 ± 85%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.smpboot_thread_fn.kthread.ret_from_fork
      1.80 ±  2%     +18.0%       2.12 ±  3%  perf-sched.wait_and_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      2.56 ±  3%     -22.9%       1.97 ± 23%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__alloc_pages_nodemask.alloc_pages_vma.shmem_alloc_page
      0.01           +31.3%       0.01 ± 15%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__release_sock.release_sock.__inet_stream_connect
      0.02 ± 17%     -32.1%       0.01 ±  9%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.apparmor_file_alloc_security.security_file_alloc.__alloc_file
      0.03 ± 28%     -41.5%       0.02 ± 41%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write.__sock_release.sock_close
      0.03 ± 47%     -40.8%       0.02 ± 53%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.sock_alloc_inode.alloc_inode
      0.72 ± 29%     -63.2%       0.27 ±133%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.mutex_lock.__fdget_pos.ksys_write
     48.74 ± 11%     -81.5%       9.03 ± 67%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.smpboot_thread_fn.kthread.ret_from_fork
      0.03 ± 17%     -42.2%       0.02 ± 19%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.13 ± 53%     -89.1%       0.01 ± 20%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
      1.79           +18.1%       2.11 ±  3%  perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      0.01           +13.9%       0.01 ±  4%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      0.02 ± 87%     -88.8%       0.00 ±113%  perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.path_openat
      1.48 ± 33%    +103.1%       3.01 ± 20%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__release_sock.release_sock.tcp_sendmsg
      1.86 ± 46%     -71.4%       0.53 ± 88%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.lock_sock_nested.inet_getname.__sys_accept4_file
      0.90 ± 43%     -65.4%       0.31 ±114%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.mutex_lock.__fdget_pos.ksys_write
    252.03 ± 13%     -95.5%      11.43 ± 85%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.smpboot_thread_fn.kthread.ret_from_fork
      0.98           -69.7%       0.30 ±128%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.72 ± 33%     -94.8%       0.04 ± 17%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
      1.79           +18.1%       2.11 ±  3%  perf-sched.wait_time.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      1.47 ± 87%     -92.7%       0.11 ±105%  perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.path_openat
   2046972 ±  2%     -11.5%    1812588 ±  6%  softirqs.CPU1.NET_RX
   1671767 ± 14%     -23.3%    1281431 ± 12%  softirqs.CPU100.NET_RX
    113468 ±  8%     -12.9%      98840 ±  8%  softirqs.CPU100.RCU
   1603739 ± 14%     -21.7%    1255037 ± 14%  softirqs.CPU101.NET_RX
   1646435 ±  3%     -27.9%    1187090 ± 10%  softirqs.CPU102.NET_RX
    110832           -17.0%      91991 ±  8%  softirqs.CPU102.RCU
   1649593 ± 12%     -25.8%    1223382 ± 12%  softirqs.CPU103.NET_RX
    112916 ±  7%     -17.9%      92746 ±  7%  softirqs.CPU103.RCU
   1655514 ± 15%     -28.8%    1179473 ± 11%  softirqs.CPU104.NET_RX
    111853 ± 10%     -18.0%      91667 ±  7%  softirqs.CPU104.RCU
   1589238 ± 17%     -24.0%    1207866 ± 11%  softirqs.CPU105.NET_RX
   1488192 ± 15%     -23.1%    1144980 ±  9%  softirqs.CPU107.NET_RX
    104900 ±  9%     -13.9%      90303 ±  5%  softirqs.CPU107.RCU
   1613291 ± 12%     -27.1%    1176018 ±  8%  softirqs.CPU108.NET_RX
    109518 ±  7%     -17.0%      90954 ±  5%  softirqs.CPU108.RCU
   1562636 ± 11%     -23.3%    1197966 ± 16%  softirqs.CPU109.NET_RX
    108400 ±  6%     -15.0%      92117 ±  9%  softirqs.CPU109.RCU
   1608148 ± 14%     -25.4%    1200448 ±  7%  softirqs.CPU110.NET_RX
    108479 ±  8%     -14.9%      92280 ±  6%  softirqs.CPU110.RCU
   1536733 ± 14%     -29.4%    1084338 ±  4%  softirqs.CPU112.NET_RX
    107319 ±  9%     -19.0%      86973 ±  4%  softirqs.CPU112.RCU
   1612392 ± 18%     -28.1%    1159150 ± 12%  softirqs.CPU114.NET_RX
    108191 ± 12%     -18.6%      88016 ±  8%  softirqs.CPU114.RCU
   1513496 ± 13%     -27.0%    1105582 ±  8%  softirqs.CPU116.NET_RX
    104391 ±  9%     -15.8%      87925 ±  6%  softirqs.CPU116.RCU
   1513988 ± 11%     -28.0%    1089380 ± 12%  softirqs.CPU117.NET_RX
    106065 ±  6%     -18.2%      86807 ±  6%  softirqs.CPU117.RCU
   1446539 ± 16%     -28.6%    1032810 ±  8%  softirqs.CPU118.NET_RX
    103092 ± 10%     -18.9%      83632 ±  6%  softirqs.CPU118.RCU
   1493229 ± 20%     -31.3%    1026209 ± 12%  softirqs.CPU119.NET_RX
    105429 ± 12%     -19.9%      84403 ±  8%  softirqs.CPU119.RCU
   1473279 ± 17%     -25.5%    1098199 ± 14%  softirqs.CPU12.NET_RX
   1462094 ± 16%     -24.0%    1111118 ± 10%  softirqs.CPU13.NET_RX
    101305 ± 11%     -15.4%      85703 ±  5%  softirqs.CPU13.RCU
    105487 ±  9%     -14.8%      89872 ±  9%  softirqs.CPU14.RCU
   1489223 ± 18%     -23.7%    1135643 ± 11%  softirqs.CPU15.NET_RX
    105366 ± 11%     -16.3%      88207 ±  8%  softirqs.CPU15.RCU
   1561209 ± 18%     -32.1%    1059715 ± 11%  softirqs.CPU16.NET_RX
    106585 ± 12%     -21.0%      84231 ±  8%  softirqs.CPU16.RCU
    105726 ± 13%     -15.3%      89564 ±  3%  softirqs.CPU17.RCU
   1732778 ± 13%     -30.9%    1197313 ± 20%  softirqs.CPU174.NET_RX
    116192 ±  8%     -22.4%      90127 ± 11%  softirqs.CPU174.RCU
     51349 ±  7%      -8.8%      46816 ±  2%  softirqs.CPU174.SCHED
   1329580 ±  6%     -22.7%    1027256 ± 24%  softirqs.CPU178.NET_RX
   1471160 ± 16%     -24.1%    1117009 ± 10%  softirqs.CPU18.NET_RX
   1485214 ± 15%     -25.5%    1106819 ±  7%  softirqs.CPU19.NET_RX
    102826 ±  8%     -15.4%      86963 ±  4%  softirqs.CPU19.RCU
   1941589 ±  5%     -15.4%    1642655 ±  9%  softirqs.CPU2.NET_RX
   1508799 ± 12%     -27.0%    1101749 ± 12%  softirqs.CPU20.NET_RX
    105120 ±  7%     -17.3%      86946 ±  8%  softirqs.CPU20.RCU
   1476181 ± 14%     -24.4%    1116092 ± 14%  softirqs.CPU21.NET_RX
    104102 ±  7%     -15.4%      88026 ±  9%  softirqs.CPU21.RCU
   1511843 ± 14%     -27.7%    1093053 ± 14%  softirqs.CPU22.NET_RX
    106348 ±  9%     -18.9%      86280 ±  9%  softirqs.CPU22.RCU
   1544533 ± 14%     -31.4%    1060181 ± 16%  softirqs.CPU23.NET_RX
    105893 ±  9%     -19.7%      85074 ± 11%  softirqs.CPU23.RCU
   1792369 ± 12%     -17.5%    1478028 ±  7%  softirqs.CPU3.NET_RX
    117778 ±  7%      -9.8%     106201 ±  5%  softirqs.CPU3.RCU
   1671681 ± 11%     -20.5%    1328518 ± 12%  softirqs.CPU4.NET_RX
    112918 ±  5%     -11.7%      99681 ±  7%  softirqs.CPU4.RCU
   1673957 ± 11%     -25.6%    1244616 ±  8%  softirqs.CPU5.NET_RX
    112336 ±  4%     -16.9%      93326 ±  6%  softirqs.CPU5.RCU
   1560853 ± 16%     -24.4%    1179814 ± 10%  softirqs.CPU7.NET_RX
    106434 ±  9%     -15.0%      90451 ±  7%  softirqs.CPU7.RCU
   1711115 ± 14%     -17.0%    1420819 ± 11%  softirqs.CPU96.NET_RX
    114442 ±  9%     -12.2%     100523 ±  7%  softirqs.CPU96.RCU
   1734032 ± 14%     -19.7%    1392592 ±  9%  softirqs.CPU98.NET_RX
    116248 ±  7%     -11.4%     102987 ±  6%  softirqs.CPU98.RCU
     10286 ± 71%     -85.9%       1449 ±117%  interrupts.32:PCI-MSI.524290-edge.eth0-TxRx-1
     28379 ± 13%     -93.7%       1794 ±130%  interrupts.33:PCI-MSI.524291-edge.eth0-TxRx-2
      4484 ± 31%     -82.4%     788.25 ±128%  interrupts.34:PCI-MSI.524292-edge.eth0-TxRx-3
   2881155 ±  4%     -11.3%    2556776        interrupts.CAL:Function_call_interrupts
     23006 ±  6%     -22.1%      17929 ± 10%  interrupts.CPU0.CAL:Function_call_interrupts
      2171 ±  6%     -22.9%       1673 ± 28%  interrupts.CPU0.NMI:Non-maskable_interrupts
      2171 ±  6%     -22.9%       1673 ± 28%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      6120 ± 10%     -19.9%       4903 ±  7%  interrupts.CPU0.RES:Rescheduling_interrupts
      2681 ±  5%     -32.6%       1807 ± 22%  interrupts.CPU1.NMI:Non-maskable_interrupts
      2681 ±  5%     -32.6%       1807 ± 22%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
     18941 ± 19%     -39.0%      11549 ± 20%  interrupts.CPU101.CAL:Function_call_interrupts
     21634 ± 10%     -37.2%      13589 ± 19%  interrupts.CPU102.CAL:Function_call_interrupts
      2916 ±  3%     -45.5%       1589 ± 23%  interrupts.CPU102.NMI:Non-maskable_interrupts
      2916 ±  3%     -45.5%       1589 ± 23%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
      4942 ±  6%     -15.2%       4192 ± 12%  interrupts.CPU102.RES:Rescheduling_interrupts
     19167 ± 20%     -39.2%      11652 ± 12%  interrupts.CPU103.CAL:Function_call_interrupts
      2396 ±  5%     -39.6%       1447 ± 41%  interrupts.CPU103.NMI:Non-maskable_interrupts
      2396 ±  5%     -39.6%       1447 ± 41%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
      5530 ± 16%     -23.5%       4230 ± 10%  interrupts.CPU103.RES:Rescheduling_interrupts
     18965 ± 29%     -41.5%      11089 ± 12%  interrupts.CPU104.CAL:Function_call_interrupts
      2286 ±  7%     -28.9%       1625 ± 13%  interrupts.CPU104.NMI:Non-maskable_interrupts
      2286 ±  7%     -28.9%       1625 ± 13%  interrupts.CPU104.PMI:Performance_monitoring_interrupts
     19747 ± 18%     -39.2%      12014 ± 20%  interrupts.CPU106.CAL:Function_call_interrupts
     18986 ± 21%     -28.8%      13512 ± 17%  interrupts.CPU107.CAL:Function_call_interrupts
     16894 ± 17%     -28.5%      12073 ± 24%  interrupts.CPU108.CAL:Function_call_interrupts
     20090 ± 21%     -42.6%      11537 ± 12%  interrupts.CPU109.CAL:Function_call_interrupts
      4991 ± 15%     -21.3%       3925 ± 11%  interrupts.CPU109.RES:Rescheduling_interrupts
     10286 ± 71%     -85.9%       1449 ±117%  interrupts.CPU11.32:PCI-MSI.524290-edge.eth0-TxRx-1
     19407 ± 19%     -36.7%      12277 ± 25%  interrupts.CPU110.CAL:Function_call_interrupts
     21568           -36.3%      13740 ±  5%  interrupts.CPU111.CAL:Function_call_interrupts
      5310 ± 17%     -22.1%       4134 ±  9%  interrupts.CPU111.RES:Rescheduling_interrupts
     20745 ± 25%     -41.8%      12072 ± 11%  interrupts.CPU112.CAL:Function_call_interrupts
      5189 ± 18%     -25.8%       3847 ±  6%  interrupts.CPU112.RES:Rescheduling_interrupts
     20852 ±  2%     -39.3%      12652 ± 24%  interrupts.CPU113.CAL:Function_call_interrupts
     20236 ± 20%     -37.2%      12701 ± 13%  interrupts.CPU114.CAL:Function_call_interrupts
     19577 ± 19%     -38.9%      11952 ± 19%  interrupts.CPU115.CAL:Function_call_interrupts
     16739 ± 23%     -30.2%      11689 ± 17%  interrupts.CPU116.CAL:Function_call_interrupts
     17891 ± 19%     -33.2%      11959 ± 17%  interrupts.CPU117.CAL:Function_call_interrupts
      4942 ± 11%     -21.2%       3895 ±  2%  interrupts.CPU117.RES:Rescheduling_interrupts
     18916 ± 19%     -38.5%      11640 ± 19%  interrupts.CPU118.CAL:Function_call_interrupts
      2292 ± 16%     -44.5%       1272 ± 34%  interrupts.CPU118.NMI:Non-maskable_interrupts
      2292 ± 16%     -44.5%       1272 ± 34%  interrupts.CPU118.PMI:Performance_monitoring_interrupts
      2182 ± 17%     -53.8%       1009 ± 49%  interrupts.CPU119.NMI:Non-maskable_interrupts
      2182 ± 17%     -53.8%       1009 ± 49%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
     28379 ± 13%     -93.7%       1794 ±130%  interrupts.CPU12.33:PCI-MSI.524291-edge.eth0-TxRx-2
     17020 ± 12%     -31.7%      11622 ± 21%  interrupts.CPU12.CAL:Function_call_interrupts
    140.50 ± 80%     -93.2%       9.50 ± 21%  interrupts.CPU125.TLB:TLB_shootdowns
    941.50 ± 31%     +85.9%       1750 ± 26%  interrupts.CPU127.NMI:Non-maskable_interrupts
    941.50 ± 31%     +85.9%       1750 ± 26%  interrupts.CPU127.PMI:Performance_monitoring_interrupts
      4484 ± 31%     -82.4%     788.25 ±128%  interrupts.CPU13.34:PCI-MSI.524292-edge.eth0-TxRx-3
     19948 ± 22%     -41.1%      11740 ± 17%  interrupts.CPU13.CAL:Function_call_interrupts
     60.50 ± 65%     -80.6%      11.75 ± 71%  interrupts.CPU133.TLB:TLB_shootdowns
    510.00 ± 29%    +190.2%       1480 ± 28%  interrupts.CPU139.NMI:Non-maskable_interrupts
    510.00 ± 29%    +190.2%       1480 ± 28%  interrupts.CPU139.PMI:Performance_monitoring_interrupts
     18645 ± 27%     -33.1%      12469 ± 33%  interrupts.CPU14.CAL:Function_call_interrupts
    602.50 ± 14%    +169.3%       1622 ± 54%  interrupts.CPU141.NMI:Non-maskable_interrupts
    602.50 ± 14%    +169.3%       1622 ± 54%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
    494.50 ±  4%    +196.9%       1468 ± 39%  interrupts.CPU142.NMI:Non-maskable_interrupts
    494.50 ±  4%    +196.9%       1468 ± 39%  interrupts.CPU142.PMI:Performance_monitoring_interrupts
     17426 ±  7%     -28.7%      12423 ± 18%  interrupts.CPU148.CAL:Function_call_interrupts
     18569 ± 16%     -26.3%      13677 ± 15%  interrupts.CPU15.CAL:Function_call_interrupts
      2169 ± 10%     -36.6%       1376 ± 28%  interrupts.CPU15.NMI:Non-maskable_interrupts
      2169 ± 10%     -36.6%       1376 ± 28%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      3408 ±  9%     +28.5%       4379 ±  9%  interrupts.CPU151.RES:Rescheduling_interrupts
     64.00 ± 62%     -83.2%      10.75 ± 70%  interrupts.CPU156.TLB:TLB_shootdowns
     19007 ± 24%     -43.0%      10828 ± 14%  interrupts.CPU16.CAL:Function_call_interrupts
      2092 ± 26%     -38.9%       1278 ± 23%  interrupts.CPU16.NMI:Non-maskable_interrupts
      2092 ± 26%     -38.9%       1278 ± 23%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
      5246 ± 23%     -30.5%       3647 ±  9%  interrupts.CPU16.RES:Rescheduling_interrupts
    482.00 ± 32%     +95.7%     943.25 ± 16%  interrupts.CPU164.NMI:Non-maskable_interrupts
    482.00 ± 32%     +95.7%     943.25 ± 16%  interrupts.CPU164.PMI:Performance_monitoring_interrupts
      2660 ± 16%     +52.6%       4060 ± 11%  interrupts.CPU167.RES:Rescheduling_interrupts
      2742 ±  9%     -33.1%       1835 ± 17%  interrupts.CPU168.NMI:Non-maskable_interrupts
      2742 ±  9%     -33.1%       1835 ± 17%  interrupts.CPU168.PMI:Performance_monitoring_interrupts
     22737 ±  4%     -43.3%      12886 ± 18%  interrupts.CPU17.CAL:Function_call_interrupts
      1780 ± 26%     -31.9%       1212 ± 34%  interrupts.CPU17.NMI:Non-maskable_interrupts
      1780 ± 26%     -31.9%       1212 ± 34%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      5224 ± 17%     -23.3%       4005        interrupts.CPU17.RES:Rescheduling_interrupts
     19639 ±  6%     -24.9%      14751 ± 21%  interrupts.CPU170.CAL:Function_call_interrupts
     18215 ± 10%     -27.4%      13233 ± 24%  interrupts.CPU172.CAL:Function_call_interrupts
      2294 ± 11%     -34.3%       1506 ± 40%  interrupts.CPU173.NMI:Non-maskable_interrupts
      2294 ± 11%     -34.3%       1506 ± 40%  interrupts.CPU173.PMI:Performance_monitoring_interrupts
     16139 ±  4%     -19.5%      12989 ± 19%  interrupts.CPU174.CAL:Function_call_interrupts
      2343 ± 13%     -36.4%       1489 ± 41%  interrupts.CPU174.NMI:Non-maskable_interrupts
      2343 ± 13%     -36.4%       1489 ± 41%  interrupts.CPU174.PMI:Performance_monitoring_interrupts
      5541 ±  2%     -21.3%       4360 ± 20%  interrupts.CPU174.RES:Rescheduling_interrupts
      1715 ± 13%     -33.0%       1148 ± 33%  interrupts.CPU175.NMI:Non-maskable_interrupts
      1715 ± 13%     -33.0%       1148 ± 33%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
     17163           -25.4%      12802 ± 17%  interrupts.CPU177.CAL:Function_call_interrupts
     16499 ±  2%     -24.5%      12463 ± 26%  interrupts.CPU179.CAL:Function_call_interrupts
     17962 ± 23%     -29.0%      12751 ±  7%  interrupts.CPU18.CAL:Function_call_interrupts
     16631 ± 10%     -27.3%      12088 ± 35%  interrupts.CPU182.CAL:Function_call_interrupts
     18734 ± 12%     -31.0%      12927 ± 16%  interrupts.CPU19.CAL:Function_call_interrupts
      4815 ± 13%     -17.3%       3983 ±  4%  interrupts.CPU19.RES:Rescheduling_interrupts
      1208 ±  7%     +47.2%       1778 ± 13%  interrupts.CPU190.NMI:Non-maskable_interrupts
      1208 ±  7%     +47.2%       1778 ± 13%  interrupts.CPU190.PMI:Performance_monitoring_interrupts
      1023 ± 29%     +80.2%       1843 ± 12%  interrupts.CPU191.NMI:Non-maskable_interrupts
      1023 ± 29%     +80.2%       1843 ± 12%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
    172.00 ± 16%     -54.4%      78.50 ± 67%  interrupts.CPU191.TLB:TLB_shootdowns
     22496 ± 20%     -27.3%      16363 ±  5%  interrupts.CPU2.CAL:Function_call_interrupts
      2131 ±  6%     -25.6%       1586 ± 18%  interrupts.CPU2.NMI:Non-maskable_interrupts
      2131 ±  6%     -25.6%       1586 ± 18%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      6794 ±  6%      -7.7%       6271 ±  9%  interrupts.CPU2.RES:Rescheduling_interrupts
      2222 ± 11%     -29.5%       1566 ± 15%  interrupts.CPU20.NMI:Non-maskable_interrupts
      2222 ± 11%     -29.5%       1566 ± 15%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      5043 ± 14%     -19.2%       4076 ± 10%  interrupts.CPU20.RES:Rescheduling_interrupts
     17075 ± 22%     -27.2%      12433 ±  6%  interrupts.CPU21.CAL:Function_call_interrupts
     20179 ± 16%     -35.1%      13086 ± 11%  interrupts.CPU22.CAL:Function_call_interrupts
      2502 ± 11%     -33.6%       1661 ± 24%  interrupts.CPU22.NMI:Non-maskable_interrupts
      2502 ± 11%     -33.6%       1661 ± 24%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      5171 ± 17%     -24.7%       3894 ± 11%  interrupts.CPU22.RES:Rescheduling_interrupts
      2418 ±  5%     -45.5%       1318 ± 36%  interrupts.CPU23.NMI:Non-maskable_interrupts
      2418 ±  5%     -45.5%       1318 ± 36%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      4920 ± 20%     -21.3%       3870 ± 16%  interrupts.CPU23.RES:Rescheduling_interrupts
      2514 ±  3%     -27.4%       1825 ± 21%  interrupts.CPU25.NMI:Non-maskable_interrupts
      2514 ±  3%     -27.4%       1825 ± 21%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
     21668 ± 19%     -32.1%      14710 ±  6%  interrupts.CPU3.CAL:Function_call_interrupts
      2404 ± 25%     -43.6%       1356 ± 27%  interrupts.CPU3.NMI:Non-maskable_interrupts
      2404 ± 25%     -43.6%       1356 ± 27%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
      1623 ± 11%     -31.6%       1110 ± 31%  interrupts.CPU30.NMI:Non-maskable_interrupts
      1623 ± 11%     -31.6%       1110 ± 31%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      1504 ± 21%     -32.7%       1012 ± 48%  interrupts.CPU33.NMI:Non-maskable_interrupts
      1504 ± 21%     -32.7%       1012 ± 48%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     19617 ± 20%     -33.5%      13050 ± 12%  interrupts.CPU4.CAL:Function_call_interrupts
      5893 ± 10%     -13.9%       5071 ±  8%  interrupts.CPU4.RES:Rescheduling_interrupts
     17281 ± 11%     -28.0%      12446 ± 17%  interrupts.CPU5.CAL:Function_call_interrupts
      1841 ± 17%     -31.0%       1271 ± 38%  interrupts.CPU5.NMI:Non-maskable_interrupts
      1841 ± 17%     -31.0%       1271 ± 38%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      5522 ± 10%     -19.4%       4451 ± 12%  interrupts.CPU5.RES:Rescheduling_interrupts
     16124 ± 10%     -17.4%      13324 ± 15%  interrupts.CPU52.CAL:Function_call_interrupts
      3789 ±  4%     +27.3%       4824 ± 18%  interrupts.CPU52.RES:Rescheduling_interrupts
      1394 ±  3%     -30.3%     971.50 ± 27%  interrupts.CPU59.NMI:Non-maskable_interrupts
      1394 ±  3%     -30.3%     971.50 ± 27%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
      2857 ±  5%     -46.0%       1543 ± 42%  interrupts.CPU6.NMI:Non-maskable_interrupts
      2857 ±  5%     -46.0%       1543 ± 42%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      9146 ± 10%     +38.9%      12703 ± 22%  interrupts.CPU66.CAL:Function_call_interrupts
     17031 ± 20%     -25.2%      12736 ±  6%  interrupts.CPU7.CAL:Function_call_interrupts
      2275 ±  3%     -48.8%       1165 ± 44%  interrupts.CPU7.NMI:Non-maskable_interrupts
      2275 ±  3%     -48.8%       1165 ± 44%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      9706 ±  6%     +36.4%      13237 ± 13%  interrupts.CPU71.CAL:Function_call_interrupts
     20840 ± 11%     -18.4%      17005 ± 10%  interrupts.CPU72.CAL:Function_call_interrupts
      3150 ±  8%     -24.6%       2375 ± 25%  interrupts.CPU72.NMI:Non-maskable_interrupts
      3150 ±  8%     -24.6%       2375 ± 25%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
      1872 ±  8%     -29.8%       1314 ± 19%  interrupts.CPU79.NMI:Non-maskable_interrupts
      1872 ±  8%     -29.8%       1314 ± 19%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      2176 ±  3%     -40.4%       1296 ± 28%  interrupts.CPU8.NMI:Non-maskable_interrupts
      2176 ±  3%     -40.4%       1296 ± 28%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
     16478 ±  9%     -21.6%      12912 ± 23%  interrupts.CPU80.CAL:Function_call_interrupts
     17876 ± 21%     -29.8%      12548 ± 22%  interrupts.CPU9.CAL:Function_call_interrupts
      1775 ±  6%     -28.2%       1274 ± 14%  interrupts.CPU90.NMI:Non-maskable_interrupts
      1775 ±  6%     -28.2%       1274 ± 14%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      1734 ±  5%     -31.9%       1181 ± 25%  interrupts.CPU91.NMI:Non-maskable_interrupts
      1734 ±  5%     -31.9%       1181 ± 25%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
      1663 ± 19%     -40.0%     998.50 ±  8%  interrupts.CPU92.NMI:Non-maskable_interrupts
      1663 ± 19%     -40.0%     998.50 ±  8%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
      1073           +54.9%       1662 ±  8%  interrupts.CPU94.NMI:Non-maskable_interrupts
      1073           +54.9%       1662 ±  8%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
     23944 ±  7%     -36.3%      15257 ±  9%  interrupts.CPU96.CAL:Function_call_interrupts
      5617 ± 17%     -25.6%       4178 ±  4%  interrupts.CPU96.RES:Rescheduling_interrupts
     22637 ± 26%     -27.3%      16459 ±  7%  interrupts.CPU97.CAL:Function_call_interrupts
      2782 ±  8%     -40.2%       1663 ± 30%  interrupts.CPU97.NMI:Non-maskable_interrupts
      2782 ±  8%     -40.2%       1663 ± 30%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
     22209 ± 14%     -34.5%      14551 ±  7%  interrupts.CPU98.CAL:Function_call_interrupts
     20426 ±  7%     -32.3%      13828 ± 18%  interrupts.CPU99.CAL:Function_call_interrupts
      1.12            -0.5        0.66 ± 36%  perf-profile.calltrace.cycles-pp.__alloc_file.alloc_empty_file.alloc_file.alloc_file_pseudo.sock_alloc_file
     14.11            -0.4       13.68        perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
     13.94            -0.4       13.52        perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
     14.27            -0.4       13.86        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
     14.35            -0.4       13.94        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      4.60            -0.3        4.25 ±  2%  perf-profile.calltrace.cycles-pp.refill_obj_stock.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start
      4.20            -0.3        3.87 ±  2%  perf-profile.calltrace.cycles-pp.drain_obj_stock.refill_obj_stock.kmem_cache_free.rcu_do_batch.rcu_core
     10.75            -0.3       10.43        perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      6.27            -0.3        5.97        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.__tcp_close.tcp_close
      0.55 ±  2%      -0.3        0.26 ±100%  perf-profile.calltrace.cycles-pp.alloc_file.alloc_file_pseudo.sock_alloc_file.__sys_socket.__x64_sys_socket
      6.03            -0.3        5.74        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.__tcp_close
      4.14            -0.3        3.86        perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     11.90            -0.3       11.62        perf-profile.calltrace.cycles-pp.inet_release.__sock_release.sock_close.__fput.task_work_run
     12.05            -0.3       11.77        perf-profile.calltrace.cycles-pp.sock_close.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      4.53            -0.3        4.25        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.04            -0.3       11.76        perf-profile.calltrace.cycles-pp.__sock_release.sock_close.__fput.task_work_run.exit_to_user_mode_prepare
     10.56            -0.3       10.29        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
      4.50            -0.3        4.24        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.08            -0.3        2.81        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.52            -0.3        0.25 ±100%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file
      1.69            -0.2        1.45 ±  2%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.asm_call_sysvec_on_stack
      1.71            -0.2        1.47        perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack
      3.42            -0.2        3.19 ±  2%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock.kmem_cache_free
      1.73            -0.2        1.49        perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu
      3.32            -0.2        3.09 ±  3%  perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock
      2.08            -0.2        1.85        perf-profile.calltrace.cycles-pp.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      2.21            -0.2        1.98        perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.08            -0.2        1.85        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt
      2.08            -0.2        1.85        perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      4.07            -0.2        3.85        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      4.08            -0.2        3.87        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.12            -0.2        6.91        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.__tcp_close.tcp_close.inet_release
      7.12            -0.2        6.93        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.__tcp_close.tcp_close.inet_release.__sock_release
      2.92 ±  2%      -0.2        2.73        perf-profile.calltrace.cycles-pp.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.50            -0.2        4.31 ±  2%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
      2.91            -0.2        2.72        perf-profile.calltrace.cycles-pp.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.88            -0.2        2.69        perf-profile.calltrace.cycles-pp.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.51            -0.2        4.33 ±  2%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      4.49            -0.2        4.31 ±  2%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
      4.51            -0.2        4.33 ±  2%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      4.43            -0.2        4.25 ±  2%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
      2.50            -0.2        2.33 ±  2%  perf-profile.calltrace.cycles-pp.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock.kmem_cache_free.rcu_do_batch
      2.17 ±  3%      -0.2        2.01 ±  2%  perf-profile.calltrace.cycles-pp.__sk_destruct.inet_release.__sock_release.sock_close.__fput
      3.35            -0.1        3.21        perf-profile.calltrace.cycles-pp.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.34            -0.1        3.20        perf-profile.calltrace.cycles-pp.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.69            -0.1        3.58        perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      5.12            -0.1        5.01        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
      0.78 ±  4%      -0.1        0.70        perf-profile.calltrace.cycles-pp.tcp_create_openreq_child.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.66 ±  3%      -0.1        0.57 ±  2%  perf-profile.calltrace.cycles-pp.inet_csk_clone_lock.tcp_create_openreq_child.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv
      1.25 ±  3%      -0.1        1.17        perf-profile.calltrace.cycles-pp.__dentry_kill.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.90 ±  2%      -0.1        0.82 ±  3%  perf-profile.calltrace.cycles-pp.alloc_file_pseudo.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept
      2.30            -0.1        2.22        perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      1.03 ±  3%      -0.1        0.95 ±  3%  perf-profile.calltrace.cycles-pp.alloc_file_pseudo.sock_alloc_file.__sys_socket.__x64_sys_socket.do_syscall_64
      0.64 ±  3%      -0.1        0.56 ±  2%  perf-profile.calltrace.cycles-pp.sk_clone_lock.inet_csk_clone_lock.tcp_create_openreq_child.tcp_v4_syn_recv_sock.tcp_check_req
      1.03 ±  3%      -0.1        0.95 ±  3%  perf-profile.calltrace.cycles-pp.sock_alloc_file.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.90 ±  2%      -0.1        0.82 ±  3%  perf-profile.calltrace.cycles-pp.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64
      0.92 ±  2%      -0.1        0.84 ±  4%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.drain_obj_stock.refill_obj_stock.kmem_cache_free.rcu_do_batch
      0.83 ±  2%      -0.1        0.76 ±  4%  perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.drain_obj_stock.refill_obj_stock.kmem_cache_free
      0.93 ±  3%      -0.1        0.86 ±  3%  perf-profile.calltrace.cycles-pp.__memcg_kmem_uncharge.drain_obj_stock.refill_obj_stock.kmem_cache_free.__sk_destruct
      2.27            -0.1        2.21        perf-profile.calltrace.cycles-pp.__sock_create.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.70 ±  2%      -0.1        0.65 ±  2%  perf-profile.calltrace.cycles-pp.sock_def_wakeup.tcp_fin.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
      0.74 ±  2%      -0.1        0.69 ±  2%  perf-profile.calltrace.cycles-pp.tcp_fin.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.59            -0.1        0.54 ±  3%  perf-profile.calltrace.cycles-pp.alloc_file.alloc_file_pseudo.sock_alloc_file.__sys_accept4_file.__sys_accept4
      0.58            -0.1        0.53 ±  2%  perf-profile.calltrace.cycles-pp.alloc_empty_file.alloc_file.alloc_file_pseudo.sock_alloc_file.__sys_accept4_file
      0.67            -0.1        0.62 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_wakeup.tcp_fin.tcp_data_queue.tcp_rcv_established
      4.94            -0.1        4.89        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      0.72 ±  2%      -0.1        0.67 ±  2%  perf-profile.calltrace.cycles-pp.sock_alloc.__sock_create.__sys_socket.__x64_sys_socket.do_syscall_64
      0.59            -0.0        0.54 ±  3%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_wakeup.tcp_fin
      1.10            -0.0        1.06        perf-profile.calltrace.cycles-pp.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64
      0.81 ±  4%      -0.0        0.76 ±  2%  perf-profile.calltrace.cycles-pp.kmem_cache_free.__dentry_kill.__fput.task_work_run.exit_to_user_mode_prepare
      0.81 ±  3%      -0.0        0.77        perf-profile.calltrace.cycles-pp.sk_alloc.inet_create.__sock_create.__sys_socket.__x64_sys_socket
      0.61            -0.0        0.57 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_wakeup.tcp_fin.tcp_data_queue
      0.71 ±  2%      -0.0        0.67 ±  2%  perf-profile.calltrace.cycles-pp.new_inode_pseudo.sock_alloc.__sock_create.__sys_socket.__x64_sys_socket
      0.69 ±  2%      -0.0        0.65 ±  2%  perf-profile.calltrace.cycles-pp.alloc_inode.new_inode_pseudo.sock_alloc.__sock_create.__sys_socket
      0.64 ±  3%      -0.0        0.59 ±  3%  perf-profile.calltrace.cycles-pp.sk_prot_alloc.sk_alloc.inet_create.__sock_create.__sys_socket
      0.91            -0.0        0.88        perf-profile.calltrace.cycles-pp.inet_csk_accept.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept
      1.54            -0.0        1.51        perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      0.60 ±  3%      -0.0        0.57 ±  2%  perf-profile.calltrace.cycles-pp.sock_alloc.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64
      0.60 ±  3%      -0.0        0.57 ±  2%  perf-profile.calltrace.cycles-pp.new_inode_pseudo.sock_alloc.__sys_accept4_file.__sys_accept4.__x64_sys_accept
      0.56            -0.0        0.53 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file.__sys_accept4
      0.71 ±  2%      -0.0        0.69 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.64            -0.0        0.61 ±  2%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
      1.33            -0.0        1.30        perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
      1.26            -0.0        1.23        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      0.66            -0.0        0.64 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
      0.81            +0.1        0.87        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.42            +0.1        1.52        perf-profile.calltrace.cycles-pp.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      1.17 ±  3%      +0.1        1.28        perf-profile.calltrace.cycles-pp.__release_sock.__tcp_close.tcp_close.inet_release.__sock_release
      1.15 ±  3%      +0.1        1.26        perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.__tcp_close.tcp_close
      1.15 ±  3%      +0.1        1.26        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.__tcp_close.tcp_close.inet_release
      0.57            +0.1        0.71 ±  2%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      1.68            +0.2        1.83        perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.86            +0.2        1.02        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
      4.02            +0.2        4.17        perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process
      3.92            +0.2        4.09        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process
      0.66            +0.2        0.82        perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      1.01            +0.2        1.18        perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      0.97            +0.2        1.15        perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
      0.93            +0.2        1.12        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
      4.13            +0.2        4.34        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv
      3.95            +0.2        4.16        perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      4.66            +0.2        4.87        perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
      4.27            +0.2        4.49        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
      8.63            +0.2        8.88        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.76            +0.3        9.01        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.72            +0.3        8.98        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.78            +0.3        8.04        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
     14.21            +0.3       14.49        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
      8.37            +0.3        8.65        perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     14.23            +0.3       14.52        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
      4.83            +0.3        5.12        perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect
      5.79            +0.3        6.09        perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
     15.11            +0.3       15.42        perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     13.71            +0.3       14.05        perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
      5.48            +0.4        5.91        perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      5.42            +0.4        5.85        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect
      5.45            +0.4        5.88        perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect
      5.30            +0.4        5.74        perf-profile.calltrace.cycles-pp.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      0.00            +0.5        0.53 ±  2%  perf-profile.calltrace.cycles-pp.tcp_finish_connect.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
     11.42            +0.7       12.14        perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
     11.61            +0.7       12.33        perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.60            +0.7       12.32        perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.47            +0.7       12.20        perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.48            -0.5        9.96        perf-profile.children.cycles-pp.kmem_cache_free
     14.12            -0.4       13.68        perf-profile.children.cycles-pp.task_work_run
      7.83            -0.4        7.41        perf-profile.children.cycles-pp.refill_obj_stock
     13.95            -0.4       13.52        perf-profile.children.cycles-pp.__fput
     14.29            -0.4       13.87        perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      7.04            -0.4        6.64 ±  2%  perf-profile.children.cycles-pp.drain_obj_stock
     14.36            -0.4       13.95        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      7.85            -0.4        7.46        perf-profile.children.cycles-pp.rcu_do_batch
      6.30            -0.4        5.91 ±  2%  perf-profile.children.cycles-pp.page_counter_uncharge
      7.88            -0.4        7.50        perf-profile.children.cycles-pp.rcu_core
      1.54            -0.4        1.18        perf-profile.children.cycles-pp.__dev_queue_xmit
      5.97            -0.4        5.62 ±  2%  perf-profile.children.cycles-pp.page_counter_cancel
     19.11            -0.3       18.81        perf-profile.children.cycles-pp.ip_output
     12.05            -0.3       11.77        perf-profile.children.cycles-pp.sock_close
     12.04            -0.3       11.76        perf-profile.children.cycles-pp.__sock_release
     11.90            -0.3       11.62        perf-profile.children.cycles-pp.inet_release
      4.53            -0.3        4.26        perf-profile.children.cycles-pp.__x64_sys_recvfrom
      4.51            -0.3        4.24        perf-profile.children.cycles-pp.__sys_recvfrom
      3.10            -0.3        2.83        perf-profile.children.cycles-pp.poll_idle
     24.72            -0.3       24.47        perf-profile.children.cycles-pp.__softirqentry_text_start
      4.27            -0.2        4.03 ±  2%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge
     13.62            -0.2       13.40        perf-profile.children.cycles-pp.__tcp_push_pending_frames
     13.60            -0.2       13.38        perf-profile.children.cycles-pp.tcp_write_xmit
      5.54            -0.2        5.33 ±  2%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      4.07            -0.2        3.86        perf-profile.children.cycles-pp.tcp_recvmsg
      4.08            -0.2        3.87        perf-profile.children.cycles-pp.inet_recvmsg
      2.58            -0.2        2.38 ±  2%  perf-profile.children.cycles-pp.__sk_destruct
      0.92            -0.2        0.72        perf-profile.children.cycles-pp.dev_hard_start_xmit
      2.92 ±  2%      -0.2        2.73        perf-profile.children.cycles-pp.__x64_sys_accept
      2.91 ±  2%      -0.2        2.72        perf-profile.children.cycles-pp.__sys_accept4
      2.88 ±  2%      -0.2        2.69        perf-profile.children.cycles-pp.__sys_accept4_file
      4.51            -0.2        4.33 ±  2%  perf-profile.children.cycles-pp.run_ksoftirqd
      3.30            -0.2        3.13        perf-profile.children.cycles-pp.irq_exit_rcu
      1.93 ±  3%      -0.2        1.77 ±  2%  perf-profile.children.cycles-pp.alloc_file_pseudo
      1.94 ±  2%      -0.2        1.78 ±  2%  perf-profile.children.cycles-pp.sock_alloc_file
      2.84 ±  4%      -0.2        2.69 ±  2%  perf-profile.children.cycles-pp.kmem_cache_alloc
      3.35            -0.1        3.21        perf-profile.children.cycles-pp.__x64_sys_socket
      3.34            -0.1        3.20        perf-profile.children.cycles-pp.__sys_socket
      0.36 ±  2%      -0.1        0.23 ± 22%  perf-profile.children.cycles-pp.start_kernel
      1.70            -0.1        1.58 ±  4%  perf-profile.children.cycles-pp.ktime_get
      0.79            -0.1        0.67        perf-profile.children.cycles-pp.loopback_xmit
      0.29            -0.1        0.17 ±  4%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      1.04            -0.1        0.93 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      3.70            -0.1        3.58        perf-profile.children.cycles-pp.tcp_recvmsg_locked
      1.15            -0.1        1.04 ±  3%  perf-profile.children.cycles-pp.alloc_file
      1.14            -0.1        1.03 ±  3%  perf-profile.children.cycles-pp.alloc_empty_file
      1.14            -0.1        1.03 ±  3%  perf-profile.children.cycles-pp.__alloc_file
      1.56            -0.1        1.46        perf-profile.children.cycles-pp._raw_spin_lock
      0.85 ±  2%      -0.1        0.76        perf-profile.children.cycles-pp.lock_sock_nested
      0.78 ±  4%      -0.1        0.70        perf-profile.children.cycles-pp.tcp_create_openreq_child
      0.15 ±  3%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.netif_skb_features
      0.63 ±  3%      -0.1        0.55 ±  4%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.25 ±  3%      -0.1        1.17        perf-profile.children.cycles-pp.__dentry_kill
      0.66 ±  3%      -0.1        0.57 ±  2%  perf-profile.children.cycles-pp.inet_csk_clone_lock
      0.64 ±  3%      -0.1        0.56 ±  2%  perf-profile.children.cycles-pp.sk_clone_lock
      0.21 ±  2%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.validate_xmit_skb
      1.31 ±  3%      -0.1        1.24        perf-profile.children.cycles-pp.new_inode_pseudo
      2.30            -0.1        2.23        perf-profile.children.cycles-pp.sk_wait_data
      1.32 ±  3%      -0.1        1.25        perf-profile.children.cycles-pp.sock_alloc
      0.83            -0.1        0.76 ±  3%  perf-profile.children.cycles-pp.aa_sk_perm
      1.27 ±  3%      -0.1        1.20        perf-profile.children.cycles-pp.alloc_inode
      1.33            -0.1        1.26 ±  2%  perf-profile.children.cycles-pp.propagate_protected_usage
      0.49 ±  3%      -0.1        0.43 ±  5%  perf-profile.children.cycles-pp.mod_timer
      0.41 ±  3%      -0.1        0.34 ±  6%  perf-profile.children.cycles-pp.security_file_free
      2.27            -0.1        2.21        perf-profile.children.cycles-pp.__sock_create
      1.06 ±  3%      -0.1        1.00        perf-profile.children.cycles-pp.obj_cgroup_charge
      0.31 ±  8%      -0.1        0.25 ±  3%  perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
      0.23 ±  2%      -0.1        0.17 ±  4%  perf-profile.children.cycles-pp.ip_finish_output
      0.70 ±  4%      -0.1        0.64 ±  2%  perf-profile.children.cycles-pp.sk_prot_alloc
      0.54            -0.1        0.48 ±  4%  perf-profile.children.cycles-pp.sk_reset_timer
      0.40 ±  3%      -0.1        0.34 ±  5%  perf-profile.children.cycles-pp.apparmor_file_free_security
      0.51            -0.1        0.45 ±  4%  perf-profile.children.cycles-pp.security_file_alloc
      1.11            -0.1        1.06        perf-profile.children.cycles-pp.inet_accept
      0.78 ±  3%      -0.1        0.73        perf-profile.children.cycles-pp.__kfree_skb
      0.45 ±  2%      -0.0        0.40 ±  4%  perf-profile.children.cycles-pp.apparmor_file_alloc_security
      0.85 ±  4%      -0.0        0.81        perf-profile.children.cycles-pp.__memcg_kmem_charge
      0.82 ±  4%      -0.0        0.78        perf-profile.children.cycles-pp.page_counter_try_charge
      0.33 ±  6%      -0.0        0.28 ±  8%  perf-profile.children.cycles-pp.__tcp_get_metrics
      0.78 ±  2%      -0.0        0.73 ±  2%  perf-profile.children.cycles-pp.sock_def_wakeup
      0.43 ±  4%      -0.0        0.38 ±  4%  perf-profile.children.cycles-pp.tcp_get_metrics
      0.36            -0.0        0.32 ±  2%  perf-profile.children.cycles-pp.ip_rcv_core
      0.32 ±  3%      -0.0        0.27 ±  4%  perf-profile.children.cycles-pp.sock_recvmsg
      0.32            -0.0        0.27 ±  6%  perf-profile.children.cycles-pp.security_socket_recvmsg
      1.89            -0.0        1.84        perf-profile.children.cycles-pp.schedule_timeout
      0.14            -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.tcp_wfree
      0.81 ±  3%      -0.0        0.77        perf-profile.children.cycles-pp.sk_alloc
      0.20 ±  2%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp.kfree
      0.68            -0.0        0.64        perf-profile.children.cycles-pp.__alloc_skb
      0.28 ±  5%      -0.0        0.24 ±  5%  perf-profile.children.cycles-pp.security_socket_post_create
      0.23            -0.0        0.19 ±  6%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.48 ±  3%      -0.0        0.45 ±  2%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      1.81            -0.0        1.77        perf-profile.children.cycles-pp.schedule
      0.26            -0.0        0.22 ±  5%  perf-profile.children.cycles-pp.apparmor_sk_clone_security
      0.23 ±  2%      -0.0        0.19 ±  6%  perf-profile.children.cycles-pp.apparmor_sk_free_security
      0.38            -0.0        0.34 ±  3%  perf-profile.children.cycles-pp.netif_rx
      0.23 ±  2%      -0.0        0.20 ±  6%  perf-profile.children.cycles-pp.security_sk_free
      0.27 ±  3%      -0.0        0.24 ±  5%  perf-profile.children.cycles-pp.apparmor_socket_post_create
      0.26            -0.0        0.22 ±  3%  perf-profile.children.cycles-pp.security_sk_clone
      0.36            -0.0        0.33 ±  3%  perf-profile.children.cycles-pp.netif_rx_internal
      0.23 ±  2%      -0.0        0.20 ±  2%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.23 ±  2%      -0.0        0.20 ±  5%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.91            -0.0        0.88        perf-profile.children.cycles-pp.inet_csk_accept
      0.65            -0.0        0.62 ±  2%  perf-profile.children.cycles-pp.update_load_avg
      0.21 ±  2%      -0.0        0.18 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.16 ±  6%      -0.0        0.13        perf-profile.children.cycles-pp.ipv4_mtu
      0.48            -0.0        0.45 ±  2%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      0.17 ±  5%      -0.0        0.14        perf-profile.children.cycles-pp.raw_local_deliver
      1.56            -0.0        1.53        perf-profile.children.cycles-pp.wait_woken
      0.28 ±  3%      -0.0        0.25 ±  4%  perf-profile.children.cycles-pp.inet_csk_reqsk_queue_hash_add
      0.57            -0.0        0.55        perf-profile.children.cycles-pp.update_rq_clock
      0.23            -0.0        0.20 ±  2%  perf-profile.children.cycles-pp.inode_init_always
      0.08 ± 12%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.36            -0.0        0.34 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.33            -0.0        0.31 ±  2%  perf-profile.children.cycles-pp.sched_clock
      0.15 ±  3%      -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.security_socket_getsockopt
      0.11 ±  9%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.inet_getname
      0.11 ±  9%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.sk_setup_caps
      0.08            -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.d_instantiate
      0.07 ± 14%      -0.0        0.05        perf-profile.children.cycles-pp.nf_hook_slow
      0.24 ±  2%      -0.0        0.23 ±  3%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.21 ±  2%      -0.0        0.20 ±  2%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.21 ±  2%      -0.0        0.19 ±  6%  perf-profile.children.cycles-pp.__ip_finish_output
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.sock_put
      0.12 ±  4%      -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.migrate_enable
      0.08            -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.__destroy_inode
      0.30            -0.0        0.29 ±  2%  perf-profile.children.cycles-pp.skb_release_all
      0.27            -0.0        0.25 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.14            -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.__ip_local_out
      0.10            -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.migrate_disable
      0.08            -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_sock_ops
      0.23 ±  2%      -0.0        0.22 ±  3%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.15 ±  3%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.ip_local_out
      0.29            -0.0        0.27        perf-profile.children.cycles-pp.skb_release_head_state
      0.14 ±  3%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
      0.12 ±  4%      -0.0        0.11 ±  3%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.12 ±  4%      -0.0        0.11 ±  3%  perf-profile.children.cycles-pp.resched_curr
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.tcp_validate_incoming
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.find_busiest_group
      0.12 ±  4%      -0.0        0.11 ±  3%  perf-profile.children.cycles-pp.___slab_alloc
      0.08            -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.destroy_inode
      0.07 ±  7%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_sock_addr
      0.11            +0.0        0.12        perf-profile.children.cycles-pp.apparmor_socket_create
      0.15            +0.0        0.16        perf-profile.children.cycles-pp.sock_rfree
      0.11            +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.security_socket_create
      0.21            +0.0        0.22        perf-profile.children.cycles-pp.__switch_to
      0.30            +0.0        0.32 ±  2%  perf-profile.children.cycles-pp.rebalance_domains
      0.12            +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.06            +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.tcp_mtup_init
      0.11 ±  9%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.secure_tcp_ts_off
      0.38 ±  2%      +0.0        0.41 ±  2%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.11 ±  4%      +0.0        0.14 ±  6%  perf-profile.children.cycles-pp.inet_twsk_alloc
      0.29 ±  5%      +0.0        0.32 ±  2%  perf-profile.children.cycles-pp.inet_csk_route_child_sock
      0.12 ±  4%      +0.0        0.16 ±  4%  perf-profile.children.cycles-pp.tcp_sync_mss
      0.05            +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.tcp_ca_openreq_child
      0.15 ±  3%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.tcp_parse_options
      0.06            +0.0        0.10        perf-profile.children.cycles-pp.tcp_stream_memory_free
      0.30 ±  4%      +0.0        0.35 ±  3%  perf-profile.children.cycles-pp.inet_csk_route_req
      0.28 ±  3%      +0.0        0.33        perf-profile.children.cycles-pp.tcp_init_metrics
      0.29            +0.0        0.34 ±  2%  perf-profile.children.cycles-pp.tcp_init_sock
      0.39            +0.0        0.43 ±  3%  perf-profile.children.cycles-pp.tcp_time_wait
      0.30 ±  3%      +0.0        0.35 ±  2%  perf-profile.children.cycles-pp.tcp_v4_init_sock
      0.34            +0.1        0.40        perf-profile.children.cycles-pp.__inet_hash_connect
      0.08 ±  5%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.tcp_synack_rtt_meas
      0.10            +0.1        0.16 ±  2%  perf-profile.children.cycles-pp.tcp_assign_congestion_control
      0.11            +0.1        0.18 ±  3%  perf-profile.children.cycles-pp.skb_entail
      0.05            +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.tcp_sndbuf_expand
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.tcp_try_fastopen
      0.16            +0.1        0.25 ±  4%  perf-profile.children.cycles-pp.tcp_openreq_init_rwin
      0.08 ±  6%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd
      1.42            +0.1        1.52        perf-profile.children.cycles-pp.tcp_child_process
      0.38 ±  2%      +0.1        0.49 ±  3%  perf-profile.children.cycles-pp.ip_rcv_finish
      0.41            +0.1        0.53        perf-profile.children.cycles-pp.tcp_finish_connect
      0.71 ±  3%      +0.1        0.82        perf-profile.children.cycles-pp.ip_route_output_flow
      0.00            +0.1        0.12 ±  5%  perf-profile.children.cycles-pp.tcp_push
      0.24 ±  4%      +0.1        0.37        perf-profile.children.cycles-pp.tcp_connect_init
      0.89 ±  2%      +0.1        1.01 ±  2%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.34            +0.1        0.47 ±  3%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.23 ±  2%      +0.1        0.36 ±  4%  perf-profile.children.cycles-pp.inet_sk_rebuild_header
      1.05 ±  2%      +0.1        1.19        perf-profile.children.cycles-pp.ip_route_output_key_hash
      0.15            +0.1        0.29 ±  3%  perf-profile.children.cycles-pp.tcp_select_initial_window
      1.00 ±  3%      +0.1        1.14        perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      1.69            +0.1        1.83        perf-profile.children.cycles-pp.tcp_conn_request
      0.15            +0.1        0.30 ±  2%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.15 ±  3%      +0.2        0.33 ±  6%  perf-profile.children.cycles-pp.__tcp_select_window
      4.66            +0.2        4.87        perf-profile.children.cycles-pp.tcp_connect
      0.27 ±  3%      +0.2        0.51 ±  2%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      8.64            +0.2        8.88        perf-profile.children.cycles-pp.sock_sendmsg
      8.76            +0.3        9.01        perf-profile.children.cycles-pp.__x64_sys_sendto
      0.68            +0.3        0.93 ±  2%  perf-profile.children.cycles-pp.tcp_init_transfer
      8.72            +0.3        8.98        perf-profile.children.cycles-pp.__sys_sendto
      8.38            +0.3        8.65        perf-profile.children.cycles-pp.tcp_sendmsg
     15.24            +0.3       15.52        perf-profile.children.cycles-pp.ip_rcv
      5.79            +0.3        6.09        perf-profile.children.cycles-pp.tcp_v4_connect
     13.85            +0.3       14.15        perf-profile.children.cycles-pp.tcp_v4_rcv
      1.88            +0.4        2.25        perf-profile.children.cycles-pp.tcp_ack
      0.69            +0.4        1.13        perf-profile.children.cycles-pp.__sk_dst_check
      5.30            +0.4        5.75        perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process
      7.04            +0.5        7.56        perf-profile.children.cycles-pp.release_sock
      0.80            +0.5        1.33        perf-profile.children.cycles-pp.ipv4_dst_check
      7.88            +0.6        8.52        perf-profile.children.cycles-pp.__release_sock
     11.47            +0.7       12.20        perf-profile.children.cycles-pp.inet_stream_connect
     11.60            +0.7       12.32        perf-profile.children.cycles-pp.__sys_connect
     11.42            +0.7       12.14        perf-profile.children.cycles-pp.__inet_stream_connect
     11.61            +0.7       12.34        perf-profile.children.cycles-pp.__x64_sys_connect
     10.50            +0.8       11.27        perf-profile.children.cycles-pp.tcp_rcv_state_process
     15.55            +0.9       16.48        perf-profile.children.cycles-pp.tcp_v4_do_rcv
      4.76            -0.3        4.46 ±  2%  perf-profile.self.cycles-pp.page_counter_cancel
      2.90            -0.3        2.64        perf-profile.self.cycles-pp.poll_idle
      0.28 ±  3%      -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      1.32 ±  2%      -0.1        1.22 ±  6%  perf-profile.self.cycles-pp.ktime_get
      1.18            -0.1        1.08        perf-profile.self.cycles-pp._raw_spin_lock
      0.14 ±  3%      -0.1        0.05        perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.62 ±  4%      -0.1        0.54 ±  4%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.86 ±  3%      -0.1        0.78        perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.34            -0.1        0.26 ±  3%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.74            -0.1        0.68 ±  3%  perf-profile.self.cycles-pp.aa_sk_perm
      1.31            -0.1        1.25 ±  3%  perf-profile.self.cycles-pp.propagate_protected_usage
      0.42 ±  3%      -0.1        0.36        perf-profile.self.cycles-pp.__sk_destruct
      0.38 ±  4%      -0.1        0.32 ±  7%  perf-profile.self.cycles-pp.apparmor_file_free_security
      0.42 ±  3%      -0.0        0.38 ±  4%  perf-profile.self.cycles-pp.apparmor_file_alloc_security
      0.32 ±  6%      -0.0        0.27 ±  8%  perf-profile.self.cycles-pp.__tcp_get_metrics
      0.76 ±  2%      -0.0        0.71 ±  3%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.14            -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.tcp_wfree
      0.36 ±  2%      -0.0        0.32 ±  3%  perf-profile.self.cycles-pp.ip_rcv_core
      0.18 ±  2%      -0.0        0.15 ±  3%  perf-profile.self.cycles-pp.kfree
      0.68 ±  2%      -0.0        0.64 ±  2%  perf-profile.self.cycles-pp.page_counter_try_charge
      0.46            -0.0        0.43 ±  2%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      0.25            -0.0        0.21 ±  5%  perf-profile.self.cycles-pp.apparmor_sk_clone_security
      0.25 ±  4%      -0.0        0.21 ±  5%  perf-profile.self.cycles-pp.ip_finish_output2
      0.21 ±  2%      -0.0        0.17 ±  4%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.15            -0.0        0.12 ± 11%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.28 ±  7%      -0.0        0.25        perf-profile.self.cycles-pp.fib_lookup_good_nhc
      0.21 ±  2%      -0.0        0.18 ±  6%  perf-profile.self.cycles-pp.apparmor_sk_free_security
      0.21 ±  2%      -0.0        0.19 ±  7%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.20 ±  2%      -0.0        0.17 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.17 ±  3%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.raw_local_deliver
      0.80            -0.0        0.78        perf-profile.self.cycles-pp.__inet_lookup_established
      0.32 ±  3%      -0.0        0.29 ±  5%  perf-profile.self.cycles-pp.page_counter_uncharge
      0.15 ±  3%      -0.0        0.12        perf-profile.self.cycles-pp.ipv4_mtu
      0.12            -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.loopback_xmit
      0.31 ±  3%      -0.0        0.29 ±  2%  perf-profile.self.cycles-pp.do_idle
      0.18 ±  2%      -0.0        0.16 ±  7%  perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.11 ±  4%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.sk_setup_caps
      0.08 ±  6%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.ip_rcv
      0.08 ±  6%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.tcp_rearm_rto
      0.30            -0.0        0.28 ±  2%  perf-profile.self.cycles-pp.native_sched_clock
      0.23 ±  2%      -0.0        0.21 ±  4%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.17 ±  3%      -0.0        0.15 ±  3%  perf-profile.self.cycles-pp.__alloc_skb
      0.41            -0.0        0.40 ±  2%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.17            -0.0        0.15 ±  2%  perf-profile.self.cycles-pp.inode_init_always
      0.15 ±  3%      -0.0        0.13 ±  3%  perf-profile.self.cycles-pp.process_backlog
      0.15 ±  6%      -0.0        0.13 ±  8%  perf-profile.self.cycles-pp.__ip_finish_output
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.sock_put
      0.12 ±  4%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.sk_clone_lock
      0.07 ± 14%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.24 ±  2%      -0.0        0.23 ±  3%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.12 ±  4%      -0.0        0.11 ±  6%  perf-profile.self.cycles-pp.resched_curr
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.tcp_validate_incoming
      0.07            -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.08 ±  6%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.12            -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.net_rx_action
      0.07 ±  7%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.18            -0.0        0.17 ±  2%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.15 ±  3%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.09            -0.0        0.08        perf-profile.self.cycles-pp.tcp_make_synack
      0.06            +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.06 ±  9%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.tcp_mtup_init
      0.07 ±  7%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.tcp_openreq_init_rwin
      0.08 ±  6%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.secure_tcp_ts_off
      0.10            +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.inet_twsk_alloc
      0.11            +0.0        0.14 ±  7%  perf-profile.self.cycles-pp.tcp_init_sock
      0.12 ±  4%      +0.0        0.16 ±  4%  perf-profile.self.cycles-pp.__inet_bind
      0.08            +0.0        0.12 ±  7%  perf-profile.self.cycles-pp.tcp_v4_syn_recv_sock
      0.06            +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_stream_memory_free
      0.11 ±  4%      +0.0        0.15 ±  5%  perf-profile.self.cycles-pp.tcp_sync_mss
      0.22 ±  4%      +0.0        0.26 ±  7%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.09            +0.0        0.13        perf-profile.self.cycles-pp.__inet_hash_connect
      0.07 ±  7%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.tcp_init_transfer
      0.14 ±  3%      +0.0        0.18 ±  4%  perf-profile.self.cycles-pp.tcp_parse_options
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.switch_fpu_return
      0.09            +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.tcp_connect_init
      0.10 ±  5%      +0.1        0.15 ±  8%  perf-profile.self.cycles-pp.tcp_update_metrics
      0.10 ±  5%      +0.1        0.16 ±  4%  perf-profile.self.cycles-pp.tcp_assign_congestion_control
      0.10            +0.1        0.17 ±  6%  perf-profile.self.cycles-pp.skb_entail
      0.07 ±  7%      +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.tcp_init_metrics
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.tcp_try_fastopen
      0.00            +0.1        0.09 ±  5%  perf-profile.self.cycles-pp.tcp_v4_send_synack
      0.03 ±100%      +0.1        0.12 ±  3%  perf-profile.self.cycles-pp.tcp_sndbuf_expand
      0.07 ±  7%      +0.1        0.17 ±  6%  perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd
      0.00            +0.1        0.11 ±  3%  perf-profile.self.cycles-pp.tcp_push
      0.34            +0.1        0.47 ±  3%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.34            +0.1        0.47 ±  3%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.15 ±  3%      +0.1        0.29 ±  3%  perf-profile.self.cycles-pp.tcp_select_initial_window
      0.15 ±  3%      +0.1        0.29 ±  2%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.15            +0.2        0.32 ±  5%  perf-profile.self.cycles-pp.__tcp_select_window
      0.47            +0.2        0.65 ±  2%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      0.22            +0.2        0.45 ±  3%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.77            +0.5        1.29        perf-profile.self.cycles-pp.ipv4_dst_check
      1.52 ± 12%      +0.7        2.20 ± 10%  perf-profile.self.cycles-pp.cpuidle_enter_state


                                                                                
                               netperf.Throughput_tps                           
                                                                                
  17400 +-------------------------------------------------------------------+   
  17200 |-+                         .+..                                    |   
        |               +. .+.. .+.+    +.+.+                               |   
  17000 |-+            :  +    +             +                              |   
  16800 |.+..+. .+.    :                      + .+.+.  .+.      .+.+. .+..  |   
  16600 |-+    +   +..+                        +     +.   +.+.+.     +    +.|   
  16400 |-+                                                                 |   
        |                                                                   |   
  16200 |-+                                 O        O                      |   
  16000 |-+                    O        O O      O O                        |   
  15800 |-+                 O        O         O                            |   
  15600 |-O    O        O        O O                                        |   
        |    O     O  O   O                                                 |   
  15400 |-+      O                                                          |   
  15200 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.11.0-rc2-00001-g7bb39313cd62" of type "text/plain" (172412 bytes)

View attachment "job-script" of type "text/plain" (7842 bytes)

View attachment "job.yaml" of type "text/plain" (5347 bytes)

View attachment "reproduce" of type "text/plain" (1091 bytes)

Powered by blists - more mailing lists