lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Mon, 12 Nov 2018 11:57:21 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     David Howells <dhowells@...hat.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        David Howells <dhowells@...hat.com>, lkp@...org
Subject: [LKP] [vfs]  38f27d0787:  netperf.Throughput_total_tps -61.6%
 regression

Greeting,

FYI, we noticed a -61.6% regression of netperf.Throughput_total_tps due to commit:


commit: 38f27d0787552bbd012ea533351435215e3cdfa6 ("vfs: Implement a filesystem superblock creation/configuration context")
https://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git mount-api-uapi

in testcase: netperf
on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 50%
	cluster: cs-localhost
	test: TCP_CRR
	cpufreq_governor: performance

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.2/50%/debian-x86_64-2018-04-03.cgz/300s/lkp-bdw-ep2/TCP_CRR/netperf

commit: 
  936d8e7dfd ("vfs: Separate changing mount flags full remount")
  38f27d0787 ("vfs: Implement a filesystem superblock creation/configuration context")

936d8e7dfd044dfa 38f27d0787552bbd012ea53335 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at_ip_tcp_fastopen_defer_connect/0x
         16:4         -262%           5:4     perf-profile.children.cycles-pp.schedule_timeout
         %stddev     %change         %stddev
             \          |                \  
    713415 ±  3%     -61.6%     274105        netperf.Throughput_total_tps
     16789           -62.9%       6229        netperf.Throughput_tps
   1976574 ± 15%     -28.5%    1412346        netperf.time.involuntary_context_switches
      3717 ±  2%     +11.3%       4136        netperf.time.percent_of_cpu_this_job_got
     10750 ±  2%     +15.5%      12419        netperf.time.system_time
    590.70 ±  2%     -65.8%     202.31        netperf.time.user_time
 4.306e+08           -62.2%   1.63e+08        netperf.time.voluntary_context_switches
  2.14e+08 ±  3%     -61.6%   82231508        netperf.workload
    576581 ± 10%     -32.2%     390687 ±  5%  numa-numastat.node0.local_node
    583764 ± 10%     -31.4%     400700 ±  6%  numa-numastat.node0.numa_hit
    483.67            -7.6%     446.76        pmeter.Average_Active_Power
     34.71           -59.8%      13.95        pmeter.performance_per_watt
    111199 ± 20%     +34.6%     149662 ± 15%  numa-meminfo.node0.AnonPages
    111075 ± 19%     -32.5%      75006 ± 31%  numa-meminfo.node1.AnonHugePages
    155532 ± 15%     -25.5%     115837 ± 18%  numa-meminfo.node1.AnonPages
 6.623e+08           -62.5%  2.485e+08        softirqs.NET_RX
  51641103 ± 13%     -44.1%   28860122        softirqs.RCU
   7781693           +38.8%   10802802        softirqs.SCHED
     54.50 ±  2%     +34.9%      73.50 ±  2%  vmstat.procs.r
   5037932 ±  2%     -59.2%    2057894        vmstat.system.cs
    196920 ±  2%      +2.9%     202632        vmstat.system.in
     38.31 ±  4%     -23.2       15.16 ± 10%  mpstat.cpu.idle%
     21.82           -14.4        7.44 ±  2%  mpstat.cpu.soft%
     36.73 ±  3%     +39.5       76.24        mpstat.cpu.sys%
      3.13 ±  3%      -2.0        1.17 ±  2%  mpstat.cpu.usr%
 7.018e+09           -63.7%  2.549e+09        cpuidle.C1.time
 7.293e+08           -60.7%  2.869e+08        cpuidle.C1.usage
 6.197e+08 ±  8%     -81.0%  1.179e+08 ±  9%  cpuidle.C1E.time
  25585732 ±  4%     -79.7%    5204543 ±  9%  cpuidle.C1E.usage
 1.241e+08 ±  3%     -64.9%   43618992 ±  2%  cpuidle.POLL.time
  18052173 ± 10%     -60.8%    7070699 ± 15%  cpuidle.POLL.usage
     15560 ± 13%     +35.6%      21100 ± 19%  numa-vmstat.node0
     27807 ± 20%     +34.6%      37421 ± 15%  numa-vmstat.node0.nr_anon_pages
     18107 ± 11%     -30.6%      12568 ± 32%  numa-vmstat.node1
     38890 ± 15%     -25.5%      28965 ± 18%  numa-vmstat.node1.nr_anon_pages
    677616 ±  7%     -18.0%     555700 ±  7%  numa-vmstat.node1.numa_hit
    510099 ± 10%     -23.4%     390637 ±  9%  numa-vmstat.node1.numa_local
      4899            +2.0%       4999        proc-vmstat.nr_inactive_anon
     53982            -1.3%      53274        proc-vmstat.nr_slab_unreclaimable
    226.75 ±133%     -78.4%      49.00        proc-vmstat.nr_written
      4899            +2.0%       4999        proc-vmstat.nr_zone_inactive_anon
   1165190 ±  4%     -30.2%     813783        proc-vmstat.numa_hit
   1148022 ±  4%     -30.6%     796626        proc-vmstat.numa_local
   1415980 ±  5%     -35.6%     911335        proc-vmstat.pgalloc_normal
   1348585 ±  5%     -37.3%     846011        proc-vmstat.pgfree
      1881 ±  2%     +29.0%       2426        turbostat.Avg_MHz
     67.58 ±  2%     +19.4       86.95        turbostat.Busy%
 7.293e+08           -60.7%  2.869e+08        turbostat.C1
     25.46 ±  2%     -16.2        9.29 ±  2%  turbostat.C1%
  25585350 ±  4%     -79.7%    5204256 ±  9%  turbostat.C1E
      2.25 ±  8%      -1.8        0.43 ±  9%  turbostat.C1E%
     29.90 ±  4%     -64.5%      10.62 ±  2%  turbostat.CPU%c1
    263.47            -9.6%     238.22        turbostat.PkgWatt
      3822 ± 10%     -23.2%       2936 ± 10%  slabinfo.eventpoll_pwq.active_objs
      3822 ± 10%     -23.2%       2936 ± 10%  slabinfo.eventpoll_pwq.num_objs
     36117 ±  3%     -29.0%      25656        slabinfo.filp.active_objs
      1130 ±  3%     -28.8%     804.75        slabinfo.filp.active_slabs
     36198 ±  3%     -28.8%      25767        slabinfo.filp.num_objs
      1130 ±  3%     -28.8%     804.75        slabinfo.filp.num_slabs
    123324            -9.0%     112286        slabinfo.kmalloc-64.active_objs
      1930            -9.0%       1756        slabinfo.kmalloc-64.active_slabs
    123575            -9.0%     112467        slabinfo.kmalloc-64.num_objs
      1930            -9.0%       1756        slabinfo.kmalloc-64.num_slabs
 5.415e+12           -10.5%  4.849e+12        perf-stat.branch-instructions
      2.60            -1.4        1.16        perf-stat.branch-miss-rate%
 1.409e+11           -60.2%  5.611e+10        perf-stat.branch-misses
      1.15            +0.7        1.83        perf-stat.cache-miss-rate%
 8.556e+09           -34.3%  5.622e+09        perf-stat.cache-misses
 7.472e+11           -58.9%  3.069e+11        perf-stat.cache-references
 1.585e+09 ±  2%     -59.3%  6.449e+08        perf-stat.context-switches
      1.92           +60.0%       3.07        perf-stat.cpi
 5.241e+13 ±  2%     +26.8%  6.646e+13        perf-stat.cpu-cycles
   1916531 ± 28%     -70.9%     558026 ±  5%  perf-stat.cpu-migrations
      0.12 ±  2%      -0.1        0.06        perf-stat.dTLB-load-miss-rate%
 9.874e+09 ±  3%     -61.5%  3.797e+09        perf-stat.dTLB-load-misses
 8.274e+12           -27.9%  5.965e+12        perf-stat.dTLB-loads
      0.02 ±  3%      -0.0        0.02 ±  2%  perf-stat.dTLB-store-miss-rate%
 9.868e+08 ±  3%     -67.9%  3.166e+08 ±  2%  perf-stat.dTLB-store-misses
 5.282e+12           -60.7%  2.077e+12        perf-stat.dTLB-stores
 8.869e+09 ±  3%     -59.8%  3.569e+09 ± 12%  perf-stat.iTLB-load-misses
 3.395e+10 ±  2%     -62.9%   1.26e+10        perf-stat.iTLB-loads
 2.732e+13           -20.8%  2.165e+13        perf-stat.instructions
      3082 ±  3%     +99.7%       6156 ± 11%  perf-stat.instructions-per-iTLB-miss
      0.52           -37.5%       0.33        perf-stat.ipc
     90.17            +5.5       95.67        perf-stat.node-load-miss-rate%
 3.261e+09           -18.2%  2.667e+09        perf-stat.node-load-misses
 3.554e+08 ±  3%     -66.0%  1.207e+08 ±  3%  perf-stat.node-loads
     82.19           -23.3       58.86        perf-stat.node-store-miss-rate%
  2.71e+09           -55.1%  1.216e+09        perf-stat.node-store-misses
 5.875e+08 ±  2%     +44.6%  8.497e+08        perf-stat.node-stores
    127712          +106.1%     263223        perf-stat.path-length
   3402504 ±  3%     +43.0%    4863968 ± 15%  sched_debug.cfs_rq:/.MIN_vruntime.max
      2.71 ± 56%    +189.2%       7.83 ± 27%  sched_debug.cfs_rq:/.load_avg.min
   3402504 ±  3%     +43.0%    4863968 ± 15%  sched_debug.cfs_rq:/.max_vruntime.max
   3323827 ±  3%     +58.7%    5275872        sched_debug.cfs_rq:/.min_vruntime.avg
   3473943 ±  2%     +55.1%    5388654        sched_debug.cfs_rq:/.min_vruntime.max
   3178116 ±  4%     +62.6%    5167570        sched_debug.cfs_rq:/.min_vruntime.min
     61982 ± 15%     -35.2%      40166 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.59           +29.7%       0.77        sched_debug.cfs_rq:/.nr_running.avg
      0.44           -27.3%       0.32 ±  2%  sched_debug.cfs_rq:/.nr_running.stddev
     11.21 ±  6%     +38.5%      15.52        sched_debug.cfs_rq:/.runnable_load_avg.avg
      9.25 ±  6%     -24.8%       6.96 ±  2%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
    222942 ± 33%     -45.6%     121355 ± 37%  sched_debug.cfs_rq:/.spread0.max
     61989 ± 15%     -35.2%      40158 ±  6%  sched_debug.cfs_rq:/.spread0.stddev
    742.06 ±  7%     +18.8%     881.73        sched_debug.cfs_rq:/.util_avg.avg
      1566 ±  3%     +12.7%       1764 ±  2%  sched_debug.cfs_rq:/.util_avg.max
    322.17 ±  7%     -15.2%     273.25 ±  4%  sched_debug.cfs_rq:/.util_avg.stddev
    427.33 ±  3%     +56.8%     670.16        sched_debug.cfs_rq:/.util_est_enqueued.avg
    987.38 ± 10%     +25.4%       1238 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.max
    341.79           -16.9%     284.18        sched_debug.cfs_rq:/.util_est_enqueued.stddev
     11.30 ±  6%     +33.5%      15.09        sched_debug.cpu.cpu_load[0].avg
      8.43 ±  4%     -19.0%       6.83 ±  5%  sched_debug.cpu.cpu_load[0].stddev
     11.34 ±  5%     +34.8%      15.28        sched_debug.cpu.cpu_load[1].avg
      0.08 ± 99%   +2400.0%       2.08 ± 37%  sched_debug.cpu.cpu_load[1].min
      6.06 ±  4%     -23.5%       4.64 ±  4%  sched_debug.cpu.cpu_load[1].stddev
     11.61 ±  4%     +34.5%      15.61        sched_debug.cpu.cpu_load[2].avg
      0.88 ± 20%    +476.2%       5.04 ± 18%  sched_debug.cpu.cpu_load[2].min
      5.21 ±  9%     -31.2%       3.59 ±  4%  sched_debug.cpu.cpu_load[2].stddev
     12.09 ±  5%     +33.3%      16.12        sched_debug.cpu.cpu_load[3].avg
      2.08 ± 25%    +268.0%       7.67 ± 11%  sched_debug.cpu.cpu_load[3].min
      5.67 ± 29%     -45.4%       3.10 ±  5%  sched_debug.cpu.cpu_load[3].stddev
     12.85 ±  4%     +29.7%      16.66        sched_debug.cpu.cpu_load[4].avg
      1.29 ± 36%    +716.1%      10.54 ±  7%  sched_debug.cpu.cpu_load[4].min
      8.46 ± 33%     -51.6%       4.09 ± 28%  sched_debug.cpu.cpu_load[4].stddev
      1130           +33.7%       1511        sched_debug.cpu.curr->pid.avg
    999.01           -25.8%     741.70 ±  2%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  6%     +45.4%       0.00 ±  2%  sched_debug.cpu.next_balance.stddev
      0.59           +29.5%       0.77        sched_debug.cpu.nr_running.avg
      0.45           -27.5%       0.32 ±  5%  sched_debug.cpu.nr_running.stddev
   8960269           -59.6%    3623716        sched_debug.cpu.nr_switches.avg
   9249479           -60.2%    3678097        sched_debug.cpu.nr_switches.max
   8415736 ±  2%     -58.1%    3528976        sched_debug.cpu.nr_switches.min
    161945 ±  8%     -81.2%      30435 ±  5%  sched_debug.cpu.nr_switches.stddev
     -6.96           +40.1%      -9.75        sched_debug.cpu.nr_uninterruptible.min
     31.20 ±  2%     -19.5       11.68        perf-profile.calltrace.cycles-pp.secondary_startup_64
     30.82 ±  2%     -19.3       11.52        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     30.81 ±  2%     -19.3       11.52        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     30.74 ±  2%     -19.2       11.50        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     24.54           -16.8        7.73        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
     24.47           -16.8        7.71        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
     24.22           -16.6        7.64        perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     24.14           -16.5        7.62        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     23.62           -16.2        7.45        perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     23.19           -15.9        7.30        perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
     22.64           -15.5        7.12        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     25.32 ±  3%     -15.5        9.83        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     22.16           -15.2        6.93        perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     24.41 ±  3%     -14.9        9.53        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     21.51           -14.8        6.74        perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
     21.30           -14.6        6.66        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     21.04           -14.5        6.58        perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     17.01           -11.9        5.06        perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
     16.66           -11.7        4.95        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     15.16           -10.8        4.40        perf-profile.calltrace.cycles-pp.sock_close.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
     15.13           -10.7        4.38        perf-profile.calltrace.cycles-pp.__sock_release.sock_close.__fput.task_work_run.exit_to_usermode_loop
     14.97           -10.6        4.33        perf-profile.calltrace.cycles-pp.inet_release.__sock_release.sock_close.__fput.task_work_run
     14.59           -10.4        4.21        perf-profile.calltrace.cycles-pp.tcp_close.inet_release.__sock_release.sock_close.__fput
     14.86            -9.7        5.17        perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.82            -9.7        5.16        perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.84            -9.4        4.41 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.79            -9.4        4.40 ±  2%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.38            -9.4        5.00        perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.27            -9.3        4.97        perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
     13.60            -9.3        4.33 ±  2%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.40            -9.1        4.27 ±  2%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     12.65            -8.8        3.87        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv
     11.85            -8.1        3.72 ±  2%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     10.78            -7.5        3.25        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_close.inet_release.__sock_release.sock_close
     10.75            -7.5        3.25        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_close.inet_release.__sock_release
     10.54            -7.2        3.29        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     10.50            -7.2        3.28        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      9.57            -6.7        2.85        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_close.inet_release
      9.17            -6.5        2.71        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_close
      8.32            -5.8        2.52        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
      8.07            -5.6        2.43        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      6.75            -4.8        1.94 ±  2%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      6.83            -4.7        2.09        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.76            -4.7        2.07        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.02            -4.6        2.40        perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      6.98            -4.6        2.38        perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect
      6.88            -4.5        2.34        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect
      6.98            -4.5        2.49        perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      6.44            -4.5        1.95        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.35            -4.4        1.91        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      6.22            -4.1        2.12        perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect
      5.82            -3.9        1.88        perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      5.87            -3.8        2.06        perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
      5.34            -3.5        1.81        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      5.19            -3.4        1.75        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
      5.08            -3.4        1.71        perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv
      4.91            -3.3        1.65        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_state_process
      4.38 ±  2%      -3.2        1.21        perf-profile.calltrace.cycles-pp.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.93            -3.2        1.76        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
      4.36 ±  2%      -3.2        1.21        perf-profile.calltrace.cycles-pp.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.65            -3.0        1.66        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect
      4.08            -2.8        1.33        perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      4.11            -2.6        1.48        perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect
      4.04            -2.6        1.45        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect
      2.94            -2.1        0.84        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.85            -2.0        0.82        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      3.06            -2.0        1.06        perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      2.81            -1.8        0.99        perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg
      2.79            -1.8        0.98        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg
      2.74            -1.8        0.96        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_woken.sk_wait_data
      2.45 ±  2%      -1.7        0.73        perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
      2.38 ±  2%      -1.7        0.70        perf-profile.calltrace.cycles-pp.tcp_fin.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
      2.14 ±  6%      -1.6        0.52        perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      2.35            -1.6        0.75 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.32            -1.6        0.74        perf-profile.calltrace.cycles-pp.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.19            -1.5        0.65        perf-profile.calltrace.cycles-pp.inet_accept.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.11            -1.4        0.67        perf-profile.calltrace.cycles-pp.tcp_child_process.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv
      2.13            -1.3        0.78        perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
      1.67            -1.3        0.39 ± 57%  perf-profile.calltrace.cycles-pp.__sock_create.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.83            -1.3        0.56 ±  2%  perf-profile.calltrace.cycles-pp.inet_csk_accept.inet_accept.__sys_accept4.__x64_sys_accept.do_syscall_64
      1.62 ±  2%      -1.2        0.38 ± 57%  perf-profile.calltrace.cycles-pp.release_sock.tcp_close.inet_release.__sock_release.sock_close
      1.84            -1.2        0.67        perf-profile.calltrace.cycles-pp.tcp_check_req.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv
      1.42 ±  2%      -1.2        0.26 ±100%  perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      1.68            -1.1        0.59 ±  2%  perf-profile.calltrace.cycles-pp.__entry_SYSCALL_64_trampoline
      1.55            -1.0        0.51        perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      0.00            +0.6        0.60 ±  8%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.mntput_no_expire
      0.00            +0.6        0.65 ±  7%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.mntput_no_expire.__fput
      0.00            +0.9        0.87        perf-profile.calltrace.cycles-pp.mnt_get_count.mntput_no_expire.__fput.task_work_run.exit_to_usermode_loop
     63.93           +22.7       86.64        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     63.62           +22.9       86.53        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.21 ±  2%     +53.6       71.84        perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.05 ±  2%     +53.7       71.80        perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
     17.79 ±  2%     +53.9       71.65        perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +65.4       65.45        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.mntput_no_expire.__fput.task_work_run
      0.00           +65.6       65.56        perf-profile.calltrace.cycles-pp._raw_spin_lock.mntput_no_expire.__fput.task_work_run.exit_to_usermode_loop
      0.00           +66.6       66.59        perf-profile.calltrace.cycles-pp.mntput_no_expire.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
     30.38           -20.7        9.64        perf-profile.children.cycles-pp.__tcp_transmit_skb
     28.87           -19.8        9.08        perf-profile.children.cycles-pp.__ip_queue_xmit
     31.20 ±  2%     -19.5       11.68        perf-profile.children.cycles-pp.secondary_startup_64
     31.20 ±  2%     -19.5       11.68        perf-profile.children.cycles-pp.cpu_startup_entry
     31.17 ±  2%     -19.5       11.67        perf-profile.children.cycles-pp.do_idle
     30.82 ±  2%     -19.3       11.52        perf-profile.children.cycles-pp.start_secondary
     28.01           -19.2        8.79        perf-profile.children.cycles-pp.ip_output
     26.80           -18.4        8.39        perf-profile.children.cycles-pp.ip_finish_output2
     24.94           -17.1        7.85        perf-profile.children.cycles-pp.__local_bh_enable_ip
     24.65           -16.9        7.75        perf-profile.children.cycles-pp.do_softirq
     24.98           -16.9        8.08        perf-profile.children.cycles-pp.__softirqentry_text_start
     24.38           -16.7        7.67        perf-profile.children.cycles-pp.do_softirq_own_stack
     23.80           -16.3        7.47        perf-profile.children.cycles-pp.net_rx_action
     23.35           -16.0        7.33        perf-profile.children.cycles-pp.process_backlog
     25.66 ±  3%     -15.7        9.97        perf-profile.children.cycles-pp.cpuidle_enter_state
     22.81           -15.7        7.15        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     22.56           -15.4        7.15        perf-profile.children.cycles-pp.tcp_v4_do_rcv
     22.31           -15.4        6.96        perf-profile.children.cycles-pp.ip_rcv
     24.72 ±  3%     -15.1        9.67        perf-profile.children.cycles-pp.intel_idle
     21.66           -14.9        6.76        perf-profile.children.cycles-pp.ip_local_deliver
     21.33           -14.8        6.55        perf-profile.children.cycles-pp.__tcp_push_pending_frames
     21.45           -14.8        6.69        perf-profile.children.cycles-pp.ip_local_deliver_finish
     21.28           -14.7        6.53        perf-profile.children.cycles-pp.tcp_write_xmit
     21.21           -14.6        6.62        perf-profile.children.cycles-pp.tcp_v4_rcv
     15.17           -10.8        4.40        perf-profile.children.cycles-pp.sock_close
     15.13           -10.7        4.39        perf-profile.children.cycles-pp.__sock_release
     14.97           -10.6        4.33        perf-profile.children.cycles-pp.inet_release
     14.60           -10.4        4.22        perf-profile.children.cycles-pp.tcp_close
     14.87            -9.7        5.17        perf-profile.children.cycles-pp.__x64_sys_connect
     14.82            -9.7        5.16        perf-profile.children.cycles-pp.__sys_connect
     14.17            -9.5        4.67        perf-profile.children.cycles-pp.tcp_rcv_state_process
     13.85            -9.4        4.42 ±  2%  perf-profile.children.cycles-pp.__x64_sys_sendto
     13.79            -9.4        4.40 ±  2%  perf-profile.children.cycles-pp.__sys_sendto
     14.39            -9.4        5.00        perf-profile.children.cycles-pp.inet_stream_connect
     14.27            -9.3        4.97        perf-profile.children.cycles-pp.__inet_stream_connect
     13.60            -9.3        4.33 ±  2%  perf-profile.children.cycles-pp.sock_sendmsg
     13.41            -9.1        4.28 ±  2%  perf-profile.children.cycles-pp.tcp_sendmsg
     11.85            -8.1        3.73 ±  2%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
     10.84            -7.3        3.58        perf-profile.children.cycles-pp.release_sock
     10.19            -6.8        3.39        perf-profile.children.cycles-pp.__release_sock
      8.21            -5.8        2.42 ±  2%  perf-profile.children.cycles-pp.tcp_rcv_established
      6.84            -4.7        2.09        perf-profile.children.cycles-pp.__x64_sys_recvfrom
      6.76            -4.7        2.07        perf-profile.children.cycles-pp.__sys_recvfrom
      6.87            -4.7        2.21        perf-profile.children.cycles-pp.__sched_text_start
      6.44            -4.5        1.95        perf-profile.children.cycles-pp.inet_recvmsg
      6.98            -4.5        2.49        perf-profile.children.cycles-pp.tcp_v4_connect
      6.36            -4.4        1.92        perf-profile.children.cycles-pp.tcp_recvmsg
      5.62            -4.0        1.65        perf-profile.children.cycles-pp.tcp_data_queue
      5.87            -3.8        2.06        perf-profile.children.cycles-pp.tcp_connect
      5.00            -3.6        1.44        perf-profile.children.cycles-pp.__wake_up_common_lock
      4.70            -3.4        1.35 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      4.46            -3.2        1.27        perf-profile.children.cycles-pp.try_to_wake_up
      4.38 ±  2%      -3.2        1.21        perf-profile.children.cycles-pp.__x64_sys_accept
      4.36 ±  2%      -3.2        1.21        perf-profile.children.cycles-pp.__sys_accept4
      4.08            -2.8        1.33        perf-profile.children.cycles-pp.sk_wait_data
      3.79            -2.7        1.09        perf-profile.children.cycles-pp.tcp_fin
      3.83            -2.7        1.14        perf-profile.children.cycles-pp.sock_def_readable
      3.97            -2.6        1.38        perf-profile.children.cycles-pp.schedule
      2.97            -2.1        0.85        perf-profile.children.cycles-pp.schedule_idle
      3.12            -2.0        1.08        perf-profile.children.cycles-pp.wait_woken
      2.68            -1.8        0.89        perf-profile.children.cycles-pp.tcp_ack
      2.52 ±  3%      -1.8        0.77 ±  5%  perf-profile.children.cycles-pp.__dev_queue_xmit
      2.38            -1.7        0.68        perf-profile.children.cycles-pp.ttwu_do_activate
      2.14 ±  6%      -1.6        0.52        perf-profile.children.cycles-pp.dput
      2.35            -1.6        0.75 ±  2%  perf-profile.children.cycles-pp.__x64_sys_socket
      2.22            -1.6        0.62 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
      2.32            -1.6        0.74        perf-profile.children.cycles-pp.__sys_socket
      2.19            -1.5        0.65 ±  2%  perf-profile.children.cycles-pp.inet_accept
      2.00 ±  7%      -1.5        0.47 ±  2%  perf-profile.children.cycles-pp.dentry_kill
      1.94 ±  7%      -1.5        0.45 ±  2%  perf-profile.children.cycles-pp.__dentry_kill
      2.25            -1.5        0.78 ±  2%  perf-profile.children.cycles-pp.__entry_SYSCALL_64_trampoline
      2.12            -1.4        0.67        perf-profile.children.cycles-pp.tcp_child_process
      1.95            -1.4        0.55 ±  2%  perf-profile.children.cycles-pp.enqueue_entity
      1.81 ±  7%      -1.4        0.43        perf-profile.children.cycles-pp.sock_alloc_file
      1.79 ±  7%      -1.4        0.43        perf-profile.children.cycles-pp.alloc_file_pseudo
      2.14            -1.4        0.78        perf-profile.children.cycles-pp.tcp_conn_request
      1.83            -1.3        0.56 ±  3%  perf-profile.children.cycles-pp.inet_csk_accept
      1.73 ±  4%      -1.2        0.51 ±  9%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      1.85            -1.2        0.67        perf-profile.children.cycles-pp.tcp_check_req
      1.68            -1.2        0.51        perf-profile.children.cycles-pp.__sock_create
      1.46            -1.2        0.30 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      1.60 ±  4%      -1.1        0.47 ± 10%  perf-profile.children.cycles-pp.loopback_xmit
      1.62            -1.1        0.51 ±  2%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      1.51            -1.1        0.40 ±  3%  perf-profile.children.cycles-pp.sock_def_wakeup
      1.29 ± 10%      -1.1        0.23 ±  3%  perf-profile.children.cycles-pp.destroy_inode
      1.62            -1.1        0.56        perf-profile.children.cycles-pp.dequeue_task_fair
      1.27 ± 10%      -1.1        0.22        perf-profile.children.cycles-pp.__destroy_inode
      1.51            -1.0        0.47 ±  2%  perf-profile.children.cycles-pp.__x64_sys_bind
      1.56            -1.0        0.52        perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
      1.32 ±  5%      -1.0        0.28 ±  2%  perf-profile.children.cycles-pp.inet_csk_destroy_sock
      1.48            -1.0        0.46        perf-profile.children.cycles-pp.__sys_bind
      1.45            -1.0        0.45 ±  2%  perf-profile.children.cycles-pp.tcp_filter
      1.40            -1.0        0.42 ±  2%  perf-profile.children.cycles-pp.update_load_avg
      1.40            -1.0        0.43 ±  2%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      1.11 ± 12%      -1.0        0.16 ±  2%  perf-profile.children.cycles-pp.security_inode_free
      1.42 ±  3%      -0.9        0.48 ±  2%  perf-profile.children.cycles-pp.__alloc_skb
      1.34            -0.9        0.41 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.39            -0.9        0.47        perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.09 ± 11%      -0.9        0.17 ±  2%  perf-profile.children.cycles-pp.d_instantiate
      1.05 ±  6%      -0.9        0.13 ±  3%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      1.31            -0.9        0.41 ±  2%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      1.30            -0.9        0.40        perf-profile.children.cycles-pp.switch_mm_irqs_off
      1.39            -0.9        0.49        perf-profile.children.cycles-pp.dequeue_entity
      1.35            -0.9        0.45        perf-profile.children.cycles-pp.nf_hook_slow
      1.24            -0.9        0.38        perf-profile.children.cycles-pp.autoremove_wake_function
      1.17            -0.8        0.32 ±  2%  perf-profile.children.cycles-pp.__kfree_skb
      1.07 ±  4%      -0.8        0.23        perf-profile.children.cycles-pp.tcp_send_fin
      1.21            -0.8        0.38 ±  2%  perf-profile.children.cycles-pp.selinux_socket_sock_rcv_skb
      0.96 ±  8%      -0.8        0.12 ±  4%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      1.06            -0.8        0.24 ±  3%  perf-profile.children.cycles-pp.lock_sock_nested
      0.92 ± 13%      -0.8        0.11 ±  4%  perf-profile.children.cycles-pp.security_d_instantiate
      1.09 ±  2%      -0.8        0.30 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.88 ±  6%      -0.8        0.12 ±  5%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.82 ± 15%      -0.7        0.10 ±  5%  perf-profile.children.cycles-pp.selinux_inode_free_security
      0.81 ± 14%      -0.7        0.10 ±  5%  perf-profile.children.cycles-pp.inode_doinit_with_dentry
      0.96            -0.7        0.28 ±  2%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.96            -0.6        0.32 ±  2%  perf-profile.children.cycles-pp.selinux_ip_postroute
      0.90            -0.6        0.26 ±  2%  perf-profile.children.cycles-pp.tcp_create_openreq_child
      0.87            -0.6        0.24 ±  2%  perf-profile.children.cycles-pp.inet_create
      0.84 ±  5%      -0.6        0.25 ±  3%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.78 ±  7%      -0.6        0.21 ± 22%  perf-profile.children.cycles-pp.netif_rx_internal
      0.82            -0.6        0.25        perf-profile.children.cycles-pp.security_socket_bind
      0.90            -0.6        0.34        perf-profile.children.cycles-pp.sk_reset_timer
      0.78            -0.5        0.25        perf-profile.children.cycles-pp.sel_netport_sid
      0.77            -0.5        0.24        perf-profile.children.cycles-pp.selinux_socket_bind
      0.72            -0.5        0.20 ±  2%  perf-profile.children.cycles-pp.inet_csk_clone_lock
      0.83 ±  2%      -0.5        0.30 ±  3%  perf-profile.children.cycles-pp.menu_select
      0.72 ±  2%      -0.5        0.21 ±  4%  perf-profile.children.cycles-pp.set_next_entity
      0.77            -0.5        0.26        perf-profile.children.cycles-pp.selinux_ip_postroute_compat
      0.78 ±  3%      -0.5        0.27        perf-profile.children.cycles-pp.tcp_v4_send_synack
      0.76            -0.5        0.24 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.82            -0.5        0.31        perf-profile.children.cycles-pp.tcp_done
      0.67            -0.5        0.17 ±  2%  perf-profile.children.cycles-pp.sk_clone_lock
      0.70            -0.5        0.22        perf-profile.children.cycles-pp.select_task_rq_fair
      0.86            -0.5        0.38        perf-profile.children.cycles-pp.mod_timer
      0.70            -0.5        0.22 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
      0.70            -0.5        0.23 ±  3%  perf-profile.children.cycles-pp.selinux_sock_rcv_skb_compat
      0.69 ±  3%      -0.5        0.23        perf-profile.children.cycles-pp.ip_route_output_key_hash
      0.67            -0.5        0.21 ±  2%  perf-profile.children.cycles-pp.sched_clock
      0.66 ±  3%      -0.4        0.21 ±  2%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      0.66            -0.4        0.21 ±  2%  perf-profile.children.cycles-pp.update_curr
      0.64            -0.4        0.21 ±  2%  perf-profile.children.cycles-pp.native_sched_clock
      0.69            -0.4        0.26 ±  2%  perf-profile.children.cycles-pp.sock_alloc
      0.58 ±  3%      -0.4        0.15 ±  3%  perf-profile.children.cycles-pp.skb_release_all
      0.64            -0.4        0.21 ±  2%  perf-profile.children.cycles-pp.switch_mm
      0.64            -0.4        0.22        perf-profile.children.cycles-pp.tcp_init_transfer
      0.60 ±  4%      -0.4        0.17 ±  2%  perf-profile.children.cycles-pp.tcp_v4_destroy_sock
      0.65            -0.4        0.23 ±  2%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.67            -0.4        0.26 ±  3%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.54 ±  3%      -0.4        0.14 ±  3%  perf-profile.children.cycles-pp.skb_release_head_state
      0.54 ±  3%      -0.4        0.14 ±  6%  perf-profile.children.cycles-pp.__slab_free
      0.57 ±  2%      -0.4        0.16 ±  9%  perf-profile.children.cycles-pp.__inet_bind
      0.62            -0.4        0.22 ±  4%  perf-profile.children.cycles-pp.ip_local_out
      0.58            -0.4        0.19 ±  4%  perf-profile.children.cycles-pp.__ip_local_out
      0.61 ±  2%      -0.4        0.22        perf-profile.children.cycles-pp.alloc_inode
      0.59 ±  3%      -0.4        0.21        perf-profile.children.cycles-pp.load_new_mm_cr3
      0.57 ±  3%      -0.4        0.19 ±  2%  perf-profile.children.cycles-pp.avc_has_perm
      0.60 ±  2%      -0.4        0.22        perf-profile.children.cycles-pp.kmem_cache_alloc
      0.50 ±  2%      -0.4        0.14 ±  3%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.58 ±  4%      -0.4        0.23 ± 14%  perf-profile.children.cycles-pp.rcu_process_callbacks
      0.53 ±  2%      -0.4        0.18 ±  7%  perf-profile.children.cycles-pp.poll_idle
      0.48 ±  3%      -0.3        0.13 ±  3%  perf-profile.children.cycles-pp.check_preempt_curr
      0.38 ±  5%      -0.3        0.04 ± 58%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.48 ±  3%      -0.3        0.15 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.40 ±  5%      -0.3        0.07 ± 20%  perf-profile.children.cycles-pp.ret_from_fork
      0.40 ±  5%      -0.3        0.07 ± 20%  perf-profile.children.cycles-pp.kthread
      0.48            -0.3        0.15 ±  2%  perf-profile.children.cycles-pp.tcp_get_metrics
      0.50            -0.3        0.18 ±  4%  perf-profile.children.cycles-pp.tcp_finish_connect
      0.49            -0.3        0.16 ±  2%  perf-profile.children.cycles-pp.__x64_sys_getsockopt
      0.47            -0.3        0.15 ±  4%  perf-profile.children.cycles-pp.__sk_destruct
      0.42 ±  2%      -0.3        0.10 ±  4%  perf-profile.children.cycles-pp.tcp_v4_init_sock
      0.48 ±  2%      -0.3        0.17 ±  2%  perf-profile.children.cycles-pp.lock_timer_base
      0.40 ±  3%      -0.3        0.09        perf-profile.children.cycles-pp.dst_release
      0.45 ±  4%      -0.3        0.14 ±  5%  perf-profile.children.cycles-pp.ip_route_output_flow
      0.46 ±  3%      -0.3        0.15 ±  4%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.40 ±  2%      -0.3        0.09        perf-profile.children.cycles-pp.tcp_init_sock
      0.49            -0.3        0.19 ±  2%  perf-profile.children.cycles-pp.tcp_set_state
      0.46            -0.3        0.15 ±  4%  perf-profile.children.cycles-pp.__sys_getsockopt
      0.45            -0.3        0.14 ±  3%  perf-profile.children.cycles-pp.alloc_file
      0.45            -0.3        0.15 ±  3%  perf-profile.children.cycles-pp.___perf_sw_event
      0.37 ± 14%      -0.3        0.08 ± 67%  perf-profile.children.cycles-pp.ktime_get_with_offset
      0.40 ±  2%      -0.3        0.11 ±  7%  perf-profile.children.cycles-pp.inet_csk_get_port
      0.41 ±  2%      -0.3        0.12        perf-profile.children.cycles-pp.__x64_sys_close
      0.42 ±  2%      -0.3        0.13 ±  5%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.40 ±  2%      -0.3        0.11 ±  4%  perf-profile.children.cycles-pp.__tcp_get_metrics
      0.45 ±  3%      -0.3        0.17 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      0.45 ±  2%      -0.3        0.17 ±  3%  perf-profile.children.cycles-pp.__call_rcu
      0.36 ±  3%      -0.3        0.09 ±  5%  perf-profile.children.cycles-pp.resched_curr
      0.43 ±  5%      -0.3        0.16 ±  2%  perf-profile.children.cycles-pp.__kmalloc_reserve
      0.40 ±  2%      -0.3        0.13 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.40 ±  2%      -0.3        0.13 ±  3%  perf-profile.children.cycles-pp.ipv4_mtu
      0.39 ±  8%      -0.3        0.12 ± 24%  perf-profile.children.cycles-pp.read_tsc
      0.30 ±  4%      -0.3        0.04 ± 57%  perf-profile.children.cycles-pp.sk_stream_kill_queues
      0.40            -0.3        0.13 ±  3%  perf-profile.children.cycles-pp.alloc_empty_file
      0.38            -0.3        0.12 ±  3%  perf-profile.children.cycles-pp.finish_task_switch
      0.40            -0.3        0.14 ±  3%  perf-profile.children.cycles-pp.__skb_clone
      0.39 ±  4%      -0.3        0.13        perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.39 ±  2%      -0.3        0.14 ±  3%  perf-profile.children.cycles-pp.selinux_parse_skb
      0.40            -0.2        0.15 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.33 ±  2%      -0.2        0.09        perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.36            -0.2        0.12 ±  3%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.31 ±  8%      -0.2        0.07        perf-profile.children.cycles-pp.tcp_write_queue_purge
      0.38 ±  5%      -0.2        0.14 ±  3%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.36            -0.2        0.12 ±  3%  perf-profile.children.cycles-pp.__alloc_file
      0.35 ±  2%      -0.2        0.11 ±  4%  perf-profile.children.cycles-pp.sk_alloc
      0.32 ±  2%      -0.2        0.08 ±  8%  perf-profile.children.cycles-pp.skb_release_data
      0.38            -0.2        0.15 ±  3%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.36            -0.2        0.12        perf-profile.children.cycles-pp.sk_stop_timer
      0.36 ±  2%      -0.2        0.13 ±  6%  perf-profile.children.cycles-pp.inode_init_always
      0.30 ±  4%      -0.2        0.06 ±  6%  perf-profile.children.cycles-pp.sock_rfree
      0.33            -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.tcp_update_metrics
      0.39 ±  4%      -0.2        0.16 ±  4%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.32            -0.2        0.09        perf-profile.children.cycles-pp.tcp_time_wait
      0.33 ±  2%      -0.2        0.10        perf-profile.children.cycles-pp.tcp_init_metrics
      0.35 ±  3%      -0.2        0.12 ±  5%  perf-profile.children.cycles-pp.security_socket_connect
      0.35 ±  2%      -0.2        0.12 ±  3%  perf-profile.children.cycles-pp.kfree
      0.34 ±  4%      -0.2        0.11 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      0.33 ±  4%      -0.2        0.10        perf-profile.children.cycles-pp.ip_finish_output
      0.31 ±  2%      -0.2        0.09 ±  5%  perf-profile.children.cycles-pp.__sk_dst_check
      0.30 ±  3%      -0.2        0.08 ±  8%  perf-profile.children.cycles-pp.account_entity_enqueue
      0.36 ±  3%      -0.2        0.14 ±  8%  perf-profile.children.cycles-pp.update_cfs_group
      0.38 ±  6%      -0.2        0.16 ± 14%  perf-profile.children.cycles-pp.start_kernel
      0.32 ±  8%      -0.2        0.11 ±  4%  perf-profile.children.cycles-pp.fib_table_lookup
      0.30            -0.2        0.09 ±  4%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.30 ±  3%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.ipv4_dst_check
      0.32            -0.2        0.11 ±  7%  perf-profile.children.cycles-pp.sk_prot_alloc
      0.29 ±  3%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.rcu_segcblist_enqueue
      0.30 ±  7%      -0.2        0.10 ±  5%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.33            -0.2        0.13 ±  9%  perf-profile.children.cycles-pp.ktime_get
      0.28            -0.2        0.09 ±  9%  perf-profile.children.cycles-pp.select_idle_sibling
      0.33 ±  3%      -0.2        0.13 ±  3%  perf-profile.children.cycles-pp.inet_csk_clear_xmit_timers
      0.29 ±  2%      -0.2        0.10 ±  5%  perf-profile.children.cycles-pp.sock_getsockopt
      0.29 ±  2%      -0.2        0.10 ±  7%  perf-profile.children.cycles-pp.selinux_socket_connect
      0.28 ±  2%      -0.2        0.10 ±  5%  perf-profile.children.cycles-pp.__inet_hash_connect
      0.27 ±  4%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.reweight_entity
      0.24 ±  2%      -0.2        0.06        perf-profile.children.cycles-pp.__list_add_valid
      0.26 ±  3%      -0.2        0.08 ±  6%  perf-profile.children.cycles-pp.filp_close
      0.25 ±  4%      -0.2        0.07 ±  6%  perf-profile.children.cycles-pp.inet_csk_bind_conflict
      0.29 ±  2%      -0.2        0.11        perf-profile.children.cycles-pp.__update_load_avg_se
      0.26            -0.2        0.08 ±  6%  perf-profile.children.cycles-pp.tcp_current_mss
      0.27            -0.2        0.09 ±  7%  perf-profile.children.cycles-pp.__fget_light
      0.27            -0.2        0.09        perf-profile.children.cycles-pp.selinux_socket_connect_helper
      0.25            -0.2        0.08 ± 10%  perf-profile.children.cycles-pp.update_rq_clock
      0.25 ±  4%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.ip_rcv_core
      0.24 ±  3%      -0.2        0.07 ±  5%  perf-profile.children.cycles-pp.security_socket_post_create
      0.24 ±  4%      -0.2        0.07        perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.25            -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.kmem_cache_alloc_trace
      0.23            -0.2        0.06        perf-profile.children.cycles-pp.ip_send_check
      0.26 ±  4%      -0.2        0.10        perf-profile.children.cycles-pp.__x64_sys_setsockopt
      0.23 ±  4%      -0.2        0.07        perf-profile.children.cycles-pp.selinux_socket_post_create
      0.24 ±  2%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.__might_fault
      0.22            -0.2        0.07 ±  7%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.24 ±  2%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.__inet_check_established
      0.24            -0.2        0.08        perf-profile.children.cycles-pp.skb_copy_datagram_iter
      0.25            -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.tcp_make_synack
      0.24 ±  3%      -0.2        0.09        perf-profile.children.cycles-pp.__sys_setsockopt
      0.22            -0.2        0.07 ±  7%  perf-profile.children.cycles-pp.sock_has_perm
      0.24 ±  2%      -0.2        0.09 ±  4%  perf-profile.children.cycles-pp.sock_alloc_inode
      0.23 ±  3%      -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.24 ±  2%      -0.1        0.09        perf-profile.children.cycles-pp.del_timer
      0.24            -0.1        0.09 ±  4%  perf-profile.children.cycles-pp._cond_resched
      0.23 ±  3%      -0.1        0.08        perf-profile.children.cycles-pp.inet_put_port
      0.23 ±  2%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.19 ±  9%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.netif_skb_features
      0.23 ±  2%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.inet_csk_route_req
      0.22 ±  3%      -0.1        0.07        perf-profile.children.cycles-pp.inet_csk_route_child_sock
      0.23            -0.1        0.09        perf-profile.children.cycles-pp.security_inet_conn_request
      0.25 ±  6%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.17 ±  3%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.pick_next_entity
      0.21 ±  5%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.security_socket_create
      0.19 ±  2%      -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.fput
      0.21 ±  2%      -0.1        0.07        perf-profile.children.cycles-pp._copy_to_iter
      0.21 ±  3%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.security_inode_alloc
      0.21 ±  2%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.inet_unhash
      0.18 ±  2%      -0.1        0.05        perf-profile.children.cycles-pp.remove_wait_queue
      0.16 ±  4%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.tcp_send_mss
      0.16            -0.1        0.03 ±100%  perf-profile.children.cycles-pp.move_addr_to_user
      0.18 ±  2%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ksize
      0.24 ±  3%      -0.1        0.11 ±  7%  perf-profile.children.cycles-pp.inet_csk_reqsk_queue_hash_add
      0.22            -0.1        0.09        perf-profile.children.cycles-pp.evict
      0.20 ±  2%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.selinux_inode_alloc_security
      0.20 ±  2%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_stage2
      0.18 ±  2%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.tcp_wfree
      0.20 ±  2%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.tcp_init_xmit_timers
      0.21            -0.1        0.09 ±  4%  perf-profile.children.cycles-pp._copy_from_iter_full
      0.17 ±  2%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.available_idle_cpu
      0.18 ±  4%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.selinux_socket_create
      0.20 ±  4%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.selinux_inet_conn_request
      0.17            -0.1        0.05        perf-profile.children.cycles-pp.inet_ehash_insert
      0.19 ±  2%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.tcp_release_cb
      0.21 ±  2%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.native_write_msr
      0.18 ±  6%      -0.1        0.06        perf-profile.children.cycles-pp.__d_alloc
      0.18 ±  2%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.memcpy_erms
      0.14            -0.1        0.03 ±100%  perf-profile.children.cycles-pp.secure_tcp_seq
      0.14 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.selinux_netlbl_socket_post_create
      0.16 ±  5%      -0.1        0.05        perf-profile.children.cycles-pp.rcu_eqs_enter
      0.22 ±  3%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.17 ±  2%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.inet_ehashfn
      0.15            -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.inet_sk_rebuild_header
      0.15 ±  2%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.inet_csk_complete_hashdance
      0.16 ±  4%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.security_sk_alloc
      0.16 ±  4%      -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.17 ±  3%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.bictcp_acked
      0.15 ±  5%      -0.1        0.05        perf-profile.children.cycles-pp.memset_erms
      0.12 ±  4%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.selinux_sk_alloc_security
      0.17 ±  6%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.ip_rcv_finish
      0.15 ±  7%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__inet_lookup_listener
      0.15 ±  2%      -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.sock_setsockopt
      0.15 ±  2%      -0.1        0.05        perf-profile.children.cycles-pp.tcp_parse_options
      0.15 ±  5%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.selinux_ipv4_output
      0.14 ±  3%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.15 ±  2%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.tcp_v4_inbound_md5_hash
      0.15 ±  3%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__inet_inherit_port
      0.12 ±  5%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.kfree_skbmem
      0.13 ±  3%      -0.1        0.05        perf-profile.children.cycles-pp.__alloc_fd
      0.13 ±  3%      -0.1        0.05        perf-profile.children.cycles-pp.tcp_cleanup_rbuf
      0.09 ±  7%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__internal_add_timer
      0.11 ±  4%      -0.1        0.05        perf-profile.children.cycles-pp.account_entity_dequeue
      0.11 ±  4%      -0.1        0.05        perf-profile.children.cycles-pp.tcp_init_buffer_space
      0.11 ±  7%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.12 ±  5%      -0.0        0.09 ± 17%  perf-profile.children.cycles-pp.clockevents_program_event
      0.09 ±  4%      +0.0        0.13 ±  8%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.08 ±  5%      +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.find_next_bit
      0.00            +0.2        0.22        perf-profile.children.cycles-pp.cpumask_next
      0.00            +0.9        0.89        perf-profile.children.cycles-pp.mnt_get_count
     63.97           +22.7       86.71        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     63.66           +22.9       86.59        perf-profile.children.cycles-pp.do_syscall_64
     18.22 ±  2%     +53.6       71.85        perf-profile.children.cycles-pp.exit_to_usermode_loop
     18.07 ±  2%     +53.7       71.81        perf-profile.children.cycles-pp.task_work_run
     17.81 ±  2%     +53.8       71.66        perf-profile.children.cycles-pp.__fput
      3.11 ±  7%     +63.2       66.28        perf-profile.children.cycles-pp._raw_spin_lock
      1.59 ± 12%     +64.1       65.65        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.00           +66.6       66.59        perf-profile.children.cycles-pp.mntput_no_expire
     24.72 ±  3%     -15.0        9.67        perf-profile.self.cycles-pp.intel_idle
      2.14            -1.4        0.75 ±  2%  perf-profile.self.cycles-pp.__entry_SYSCALL_64_trampoline
      1.79            -1.1        0.65 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      1.39            -0.9        0.47        perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.03 ±  7%      -0.9        0.12 ±  3%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      1.13            -0.9        0.27 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.94 ±  8%      -0.8        0.12 ±  6%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      1.07 ±  2%      -0.8        0.29        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.08            -0.8        0.31 ±  2%  perf-profile.self.cycles-pp.__sched_text_start
      0.87 ±  6%      -0.8        0.11 ±  4%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.83            -0.6        0.23 ±  2%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.91            -0.5        0.36        perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.80            -0.5        0.27        perf-profile.self.cycles-pp.tcp_v4_rcv
      0.70 ±  2%      -0.5        0.19 ±  2%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.68 ±  2%      -0.5        0.19 ±  2%  perf-profile.self.cycles-pp.update_load_avg
      0.67            -0.5        0.19 ±  6%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.66            -0.4        0.23 ±  3%  perf-profile.self.cycles-pp.tcp_ack
      0.61            -0.4        0.20 ±  2%  perf-profile.self.cycles-pp.native_sched_clock
      0.58            -0.4        0.17 ±  3%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.54 ±  3%      -0.4        0.14 ±  3%  perf-profile.self.cycles-pp.__slab_free
      0.59 ±  3%      -0.4        0.21        perf-profile.self.cycles-pp.load_new_mm_cr3
      0.56 ±  3%      -0.4        0.18 ±  2%  perf-profile.self.cycles-pp.avc_has_perm
      0.52 ±  2%      -0.4        0.16 ±  8%  perf-profile.self.cycles-pp.ip_finish_output2
      0.50 ±  2%      -0.4        0.14 ±  3%  perf-profile.self.cycles-pp.selinux_socket_sock_rcv_skb
      0.54 ±  2%      -0.4        0.18 ±  2%  perf-profile.self.cycles-pp.sel_netport_sid
      0.49            -0.3        0.15        perf-profile.self.cycles-pp.update_curr
      0.48 ±  2%      -0.3        0.14 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.39 ±  3%      -0.3        0.08 ±  5%  perf-profile.self.cycles-pp.dst_release
      0.42            -0.3        0.13 ±  3%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.40 ±  2%      -0.3        0.11 ±  4%  perf-profile.self.cycles-pp.__tcp_get_metrics
      0.41 ±  3%      -0.3        0.14 ±  3%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.41            -0.3        0.14 ±  3%  perf-profile.self.cycles-pp.__alloc_skb
      0.42 ±  5%      -0.3        0.14 ±  7%  perf-profile.self.cycles-pp.poll_idle
      0.43 ±  2%      -0.3        0.16 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      0.41 ±  2%      -0.3        0.14 ±  3%  perf-profile.self.cycles-pp.___perf_sw_event
      0.41            -0.3        0.14 ±  3%  perf-profile.self.cycles-pp.tcp_rcv_state_process
      0.40            -0.3        0.13        perf-profile.self.cycles-pp.__ip_queue_xmit
      0.42            -0.3        0.15 ±  2%  perf-profile.self.cycles-pp.do_syscall_64
      0.38 ±  4%      -0.3        0.11        perf-profile.self.cycles-pp.net_rx_action
      0.39 ±  2%      -0.3        0.13 ±  5%  perf-profile.self.cycles-pp.__switch_to
      0.38            -0.3        0.12 ±  3%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.34 ±  3%      -0.3        0.08 ±  5%  perf-profile.self.cycles-pp.resched_curr
      0.38            -0.3        0.12 ±  3%  perf-profile.self.cycles-pp.do_idle
      0.39 ±  2%      -0.3        0.13 ±  3%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.36 ±  8%      -0.3        0.11 ± 26%  perf-profile.self.cycles-pp.read_tsc
      0.40            -0.2        0.15 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      0.39 ±  2%      -0.2        0.15 ±  2%  perf-profile.self.cycles-pp.kmem_cache_free
      0.36 ±  2%      -0.2        0.12        perf-profile.self.cycles-pp.ipv4_mtu
      0.32 ±  2%      -0.2        0.08 ± 10%  perf-profile.self.cycles-pp.skb_release_data
      0.36            -0.2        0.13 ±  3%  perf-profile.self.cycles-pp.selinux_parse_skb
      0.29 ±  3%      -0.2        0.06        perf-profile.self.cycles-pp.sock_rfree
      0.35 ±  3%      -0.2        0.12        perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.39 ±  4%      -0.2        0.16 ±  5%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.34 ±  3%      -0.2        0.11 ±  3%  perf-profile.self.cycles-pp.process_backlog
      0.39 ±  3%      -0.2        0.17 ±  3%  perf-profile.self.cycles-pp.menu_select
      0.32 ±  2%      -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.__skb_clone
      0.30 ±  2%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.set_next_entity
      0.35 ±  2%      -0.2        0.13 ±  3%  perf-profile.self.cycles-pp.ip_output
      0.34 ±  2%      -0.2        0.12 ±  3%  perf-profile.self.cycles-pp.kfree
      0.35 ±  3%      -0.2        0.13 ±  5%  perf-profile.self.cycles-pp.update_cfs_group
      0.31 ±  2%      -0.2        0.09        perf-profile.self.cycles-pp.loopback_xmit
      0.35 ±  2%      -0.2        0.14 ±  3%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.30 ±  2%      -0.2        0.09 ±  5%  perf-profile.self.cycles-pp.wait_woken
      0.34            -0.2        0.12 ±  3%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.32 ±  7%      -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.fib_table_lookup
      0.30 ±  2%      -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.enqueue_entity
      0.29            -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.28 ±  4%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.rcu_segcblist_enqueue
      0.22            -0.2        0.03 ±100%  perf-profile.self.cycles-pp.ip_rcv
      0.28 ±  4%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.try_to_wake_up
      0.28            -0.2        0.09 ±  5%  perf-profile.self.cycles-pp.finish_task_switch
      0.27            -0.2        0.07 ±  5%  perf-profile.self.cycles-pp.ipv4_dst_check
      0.29 ±  3%      -0.2        0.10        perf-profile.self.cycles-pp.__might_sleep
      0.27 ±  3%      -0.2        0.08 ±  6%  perf-profile.self.cycles-pp.do_softirq
      0.30            -0.2        0.11 ±  4%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.27 ±  2%      -0.2        0.08        perf-profile.self.cycles-pp.enqueue_task_fair
      0.31            -0.2        0.12        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.27 ±  4%      -0.2        0.08        perf-profile.self.cycles-pp.reweight_entity
      0.28            -0.2        0.10 ± 11%  perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.28 ±  5%      -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.26 ±  3%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.25 ±  5%      -0.2        0.07 ±  6%  perf-profile.self.cycles-pp.inet_csk_bind_conflict
      0.24 ±  3%      -0.2        0.06        perf-profile.self.cycles-pp.__list_add_valid
      0.29 ±  3%      -0.2        0.11 ±  3%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.26 ±  2%      -0.2        0.09 ±  9%  perf-profile.self.cycles-pp.__fget_light
      0.25 ±  3%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.ip_rcv_core
      0.27 ±  6%      -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.23 ±  3%      -0.2        0.06        perf-profile.self.cycles-pp.ip_send_check
      0.24            -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      0.25 ±  3%      -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.selinux_ip_postroute_compat
      0.23 ±  3%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.selinux_sock_rcv_skb_compat
      0.21 ±  2%      -0.2        0.06 ±  6%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.20 ±  3%      -0.2        0.05        perf-profile.self.cycles-pp.dequeue_entity
      0.21 ±  3%      -0.1        0.06        perf-profile.self.cycles-pp.sock_def_readable
      0.24            -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.21            -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.sock_has_perm
      0.21 ±  3%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.21 ±  3%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.ip_local_deliver
      0.20            -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.account_entity_enqueue
      0.20 ±  2%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.entry_SYSCALL_64_stage2
      0.20 ±  2%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.selinux_ip_postroute
      0.18 ±  2%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.ksize
      0.19 ±  4%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__sk_destruct
      0.19            -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_close
      0.17 ±  2%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.available_idle_cpu
      0.21 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.native_write_msr
      0.17 ±  2%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.tcp_wfree
      0.24            -0.1        0.12 ±  4%  perf-profile.self.cycles-pp.mod_timer
      0.15 ±  7%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.tcp_conn_request
      0.17 ±  3%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.tcp_data_queue
      0.16 ±  5%      -0.1        0.05        perf-profile.self.cycles-pp.rcu_eqs_enter
      0.18 ±  2%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.tcp_release_cb
      0.16 ±  4%      -0.1        0.05        perf-profile.self.cycles-pp.sk_clone_lock
      0.17 ±  4%      -0.1        0.06        perf-profile.self.cycles-pp.tcp_connect
      0.17 ±  3%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.inet_ehashfn
      0.17            -0.1        0.07 ± 19%  perf-profile.self.cycles-pp.ktime_get
      0.16 ±  4%      -0.1        0.06        perf-profile.self.cycles-pp.bictcp_acked
      0.12 ±  4%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.15            -0.1        0.05        perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.15 ±  2%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.kmem_cache_alloc_trace
      0.14 ±  7%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.nf_hook_slow
      0.15 ±  7%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__inet_lookup_listener
      0.15 ±  3%      -0.1        0.05        perf-profile.self.cycles-pp.memset_erms
      0.15 ±  3%      -0.1        0.05        perf-profile.self.cycles-pp.tcp_parse_options
      0.16 ±  2%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.memcpy_erms
      0.13 ±  6%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.__release_sock
      0.12 ±  4%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.15 ±  3%      -0.1        0.06        perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.15 ±  3%      -0.1        0.07 ±  5%  perf-profile.self.cycles-pp.__call_rcu
      0.11 ±  4%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.11 ±  8%      -0.1        0.05 ±  9%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.10 ±  5%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.lock_timer_base
      0.09 ±  4%      +0.0        0.13 ±  9%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.cpumask_next
      0.00            +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.find_next_bit
      0.00            +0.1        0.13        perf-profile.self.cycles-pp.mntput_no_expire
      0.00            +0.7        0.69        perf-profile.self.cycles-pp.mnt_get_count
      1.57 ± 12%     +63.4       65.02        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                               netperf.Throughput_tps                           
                                                                                
  18000 +-+-----------------------------------------------------------------+   
        |              .+.      .+.+.                               .+.+..+.|   
  16000 +-+..+.+.+.+..+   +.+..+     +..+.+.+..+.+.+.+..+.+.+.+..+.+        |   
        |                                                                   |   
        |                                                                   |   
  14000 +-+                                                                 |   
        |                                                                   |   
  12000 +-+                                                                 |   
        |                                                                   |   
  10000 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
   8000 +-+                                                                 |   
        |                                                                   |   
   6000 O-O--O-O-O-O--O-O-O-O--O-O-O-O--O-O-O--O-O-O-O--O-O-O-O--O-O--------+   
                                                                                
                                                                                                                                                                
                            netperf.Throughput_total_tps                        
                                                                                
  750000 +-+----------------------------------------------------------------+   
  700000 +-+.+..+.+.+.+.     +.+.     +.+..+.+.+.+.+..+.+.+.+..+        +. .|   
         |                                                                + |   
  650000 +-+                                                                |   
  600000 +-+                                                                |   
         |                                                                  |   
  550000 +-+                                                                |   
  500000 +-+                                                                |   
  450000 +-+                                                                |   
         |                                                                  |   
  400000 +-+                                                                |   
  350000 +-+                                                                |   
         |                                                                  |   
  300000 O-O O  O O O O  O O O O  O O O O  O O O O O  O O O O  O O O        |   
  250000 +-+----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   netperf.workload                             
                                                                                
  2.4e+08 +-+---------------------------------------------------------------+   
          |                                                                 |   
  2.2e+08 +-+.+.. .+.+.+.+..+.+.+.+.+..+.+.+. .+..+.+.+. .+..+.+.+.+.+..+. .|   
    2e+08 +-+    +                           +          +                 + |   
          |                                                                 |   
  1.8e+08 +-+                                                               |   
          |                                                                 |   
  1.6e+08 +-+                                                               |   
          |                                                                 |   
  1.4e+08 +-+                                                               |   
  1.2e+08 +-+                                                               |   
          |                                                                 |   
    1e+08 +-+                                                               |   
          |                                                                 |   
    8e+07 O-O-O--O-O-O-O-O--O-O-O-O-O--O-O-O-O-O--O-O-O-O-O--O-O-O-O--------+   
                                                                                
                                                                                                                                                                
                            pmeter.Average_Active_Power                         
                                                                                
  500 +-+-------------------------------------------------------------------+   
  450 O-O  O   O  O   O  O   O    O O  O   O  O O    O O O  O   O  O      +.|   
      |             O      O    O        O        O           O             |   
  400 +-+                                                                   |   
  350 +-+                                                                   |   
      |                                                                     |   
  300 +-+                                                                   |   
  250 +-+                                                                   |   
  200 +-+                                                                   |   
      |                                                                     |   
  150 +-+                                                                   |   
  100 +-+                                                                   |   
      |                                                                     |   
   50 +-+                                                                   |   
    0 +-+----O--------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              netperf.time.user_time                            
                                                                                
  650 +-+-------------------------------------------------------------------+   
  600 +-+..+. .+..+. .+..+. .+.. .+.  .+.+.  .+. .+..+. .+..   .+.. .+.    .|   
      |      +      +      +    +   +.     +.   +      +    +.+    +   +..+ |   
  550 +-+                                                                   |   
  500 +-+                                                                   |   
      |                                                                     |   
  450 +-+                                                                   |   
  400 +-+                                                                   |   
  350 +-+                                                                   |   
      |                                                                     |   
  300 +-+                                                                   |   
  250 +-+                                                                   |   
      O O  O O O    O O  O O O  O O O    O    O   O    O O      O           |   
  200 +-+         O                    O   O    O    O      O O    O        |   
  150 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              netperf.time.system_time                          
                                                                                
  12500 +-+-----------------------------O--------O--------O-O-O----O--------+   
        O O  O O O O  O O O O  O O O O    O O  O   O O  O        O          |   
        |                                                                   |   
  12000 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  11500 +-+                                                                 |   
        |                                                                   |   
  11000 +-+       .+..+.+.+.+..+.                                           |   
        |.+..+.+.+               +.+.+..+.+.+..+.+.+.+..+.+.+.+..+.+.+.     |   
        |                                                              +.. :|   
  10500 +-+                                                                :|   
        |                                                                 + |   
        |                                                                   |   
  10000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                     netperf.time.percent_of_cpu_this_job_got                   
                                                                                
  4200 +-+------------------------------------------------------------------+   
       |               O O    O O  O O O  O O O O  O O O  O O O  O O        |   
  4100 O-O  O O O  O O      O                                               |   
       |                                                                    |   
  4000 +-+                                                                  |   
       |                                                                    |   
  3900 +-+                                                                  |   
       |                                                                    |   
  3800 +-+..+. .+..+.+.+.+..+.+.+..+.+.+..+. .+.+..+.+.  .+.+.+..+.+.+      |   
       |      +                             +          +.             +     |   
  3700 +-+                                                             +.. :|   
       |                                                                   :|   
  3600 +-+                                                                + |   
       |                                                                    |   
  3500 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        netperf.time.voluntary_context_switches                 
                                                                                
  4.5e+08 +-+---------------------------------------------------------------+   
          |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.|   
    4e+08 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
  3.5e+08 +-+                                                               |   
          |                                                                 |   
    3e+08 +-+                                                               |   
          |                                                                 |   
  2.5e+08 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    2e+08 +-+                                                               |   
          O O O  O O O O O  O O O O O  O O O O O  O O O O O  O O O O        |   
  1.5e+08 +-+---------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           pmeter.performance_per_watt                          
                                                                                
  40 +-+--------------------------------------------------------------------+   
     |                                                                      |   
  35 +-+..+.+.+..+.+..+.+.+..+.+.+..+.+..+.+.+..+.+.+..+.+.+..+.+..+.+.+..+.|   
  30 +-+                                                                    |   
     |                                                                      |   
  25 +-+                                                                    |   
     |                                                                      |   
  20 +-+                                                                    |   
     |                                                                      |   
  15 O-O  O   O  O O  O O O  O O O  O O  O O O  O O O  O O O  O O  O        |   
  10 +-+                                                                    |   
     |                                                                      |   
   5 +-+                                                                    |   
     |                                                                      |   
   0 +-+----O---------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen

View attachment "config-4.19.0-00013-g38f27d0" of type "text/plain" (167747 bytes)

View attachment "job-script" of type "text/plain" (7583 bytes)

View attachment "job.yaml" of type "text/plain" (5133 bytes)

View attachment "reproduce" of type "text/plain" (2521 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ