lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20220429062049.GA8992@xsang-OptiPlex-9020>
Date:   Fri, 29 Apr 2022 14:20:49 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Jens Axboe <axboe@...nel.dk>
Cc:     Ammar Faizi <ammarfaizi2@...weeb.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com, fengwei.yin@...el.com,
        LKML <linux-kernel@...r.kernel.org>
Subject: [net]  02dcea6024:  netperf.Throughput_tps -9.1% regression



Greeting,

FYI, we noticed a -9.1% regression of netperf.Throughput_tps due to commit:


commit: 02dcea6024d411cda23bd77e5e604f94a32c332d ("net: allow sk_prot->release_cb() without sock lock held")
https://github.com/ammarfaizi2/linux-block axboe/linux-block/sock-nolock.2

in testcase: netperf
on test machine: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 25%
	cluster: cs-localhost
	test: TCP_RR
	cpufreq_governor: performance
	ucode: 0xd000331

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/25%/debian-10.4-x86_64-20200603.cgz/300s/lkp-icl-2sp2/TCP_RR/netperf/0xd000331

commit: 
  5aa32dd9bd ("net: add support for fast socket release")
  02dcea6024 ("net: allow sk_prot->release_cb() without sock lock held")

5aa32dd9bdb57e4a 02dcea6024d411cda23bd77e5e6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   2666474            -9.1%    2425047        netperf.Throughput_total_tps
     83327            -9.1%      75782        netperf.Throughput_tps
     18796           -17.8%      15449 ±  2%  netperf.time.involuntary_context_switches
      1409            -9.5%       1275        netperf.time.percent_of_cpu_this_job_got
      3992            -9.5%       3614        netperf.time.system_time
    252.46            -9.8%     227.82 ±  2%  netperf.time.user_time
 7.999e+08            -9.1%  7.275e+08        netperf.time.voluntary_context_switches
 7.999e+08            -9.1%  7.275e+08        netperf.workload
    638430 ± 18%     -25.9%     473353 ± 23%  sched_debug.cfs_rq:/.spread0.max
     82314 ±  2%     -37.2%      51672        meminfo.Active
     81665 ±  2%     -37.2%      51317        meminfo.Active(anon)
      7.82            -0.8        7.06        mpstat.cpu.all.soft%
     17.06            -1.7       15.36        mpstat.cpu.all.sys%
     17958 ±  6%     -37.8%      11177 ±  9%  numa-vmstat.node1.nr_active_anon
     17958 ±  6%     -37.8%      11177 ±  9%  numa-vmstat.node1.nr_zone_active_anon
     38.50            -9.5%      34.83        vmstat.procs.r
  10515612            -9.1%    9558579        vmstat.system.cs
   5490715            -8.7%    5012829        vmstat.system.in
    351502 ±  2%     -14.5%     300576 ± 10%  numa-meminfo.node0.AnonPages.max
     71769 ±  6%     -37.7%      44711 ±  9%  numa-meminfo.node1.Active
     71538 ±  6%     -37.7%      44573 ±  9%  numa-meminfo.node1.Active(anon)
     65476 ± 33%     +89.8%     124250 ± 30%  numa-meminfo.node1.AnonPages.max
     72.60            +2.7       75.28        turbostat.C1%
   7392888 ±  2%     -11.9%    6516319        turbostat.POLL
      0.19 ±  2%      -0.0        0.17        turbostat.POLL%
    347.54            -2.8%     337.93        turbostat.PkgWatt
     20367 ±  2%     -37.1%      12807        proc-vmstat.nr_active_anon
     81436            -1.8%      79968        proc-vmstat.nr_anon_pages
    756705 ±  5%      +5.5%     798547        proc-vmstat.nr_mapped
      3231 ±  2%      +2.3%       3306        proc-vmstat.nr_page_table_pages
     20367 ±  2%     -37.1%      12807        proc-vmstat.nr_zone_active_anon
    739185 ±  4%     -11.1%     657230 ±  4%  proc-vmstat.numa_hint_faults_local
     21363 ±  3%     -48.0%      11099 ±  2%  proc-vmstat.pgactivate
   2272744            -3.8%    2185795 ±  2%  proc-vmstat.pgfault
 2.071e+10           -10.0%  1.864e+10        perf-stat.i.branch-instructions
      1.14            +0.0        1.17        perf-stat.i.branch-miss-rate%
 2.304e+08            -7.6%   2.13e+08        perf-stat.i.branch-misses
 1.272e+09            -9.5%  1.151e+09        perf-stat.i.cache-references
  10601392            -9.1%    9641359        perf-stat.i.context-switches
 1.248e+11            -9.2%  1.134e+11        perf-stat.i.cpu-cycles
    487.17           -13.9%     419.41        perf-stat.i.cpu-migrations
 3.072e+10            -9.7%  2.774e+10        perf-stat.i.dTLB-loads
 1.791e+10            -9.5%  1.621e+10        perf-stat.i.dTLB-stores
 1.034e+11            -9.7%  9.332e+10        perf-stat.i.instructions
      0.87            -1.5%       0.86        perf-stat.i.major-faults
      0.97            -9.2%       0.89        perf-stat.i.metric.GHz
    123.22 ±  5%     -11.6%     108.95 ±  4%  perf-stat.i.metric.K/sec
    551.53            -9.7%     497.95        perf-stat.i.metric.M/sec
      7152            -4.3%       6845 ±  2%  perf-stat.i.minor-faults
     49.74 ± 14%     +13.1       62.89 ± 10%  perf-stat.i.node-store-miss-rate%
      7153            -4.3%       6846 ±  2%  perf-stat.i.page-faults
      1.11            +0.0        1.14        perf-stat.overall.branch-miss-rate%
      0.02 ±  2%      +0.0        0.03        perf-stat.overall.dTLB-load-miss-rate%
 2.064e+10           -10.0%  1.857e+10        perf-stat.ps.branch-instructions
 2.296e+08            -7.5%  2.123e+08        perf-stat.ps.branch-misses
 1.268e+09            -9.5%  1.148e+09        perf-stat.ps.cache-references
  10563666            -9.0%    9609247        perf-stat.ps.context-switches
 1.244e+11            -9.2%   1.13e+11        perf-stat.ps.cpu-cycles
    485.80           -13.9%     418.22        perf-stat.ps.cpu-migrations
 3.061e+10            -9.7%  2.765e+10        perf-stat.ps.dTLB-loads
 1.784e+10            -9.5%  1.615e+10        perf-stat.ps.dTLB-stores
  1.03e+11            -9.7%  9.302e+10        perf-stat.ps.instructions
      0.87            -1.4%       0.86        perf-stat.ps.major-faults
      7123            -4.2%       6822 ±  2%  perf-stat.ps.minor-faults
      7124            -4.2%       6823 ±  2%  perf-stat.ps.page-faults
 3.112e+13            -9.6%  2.813e+13        perf-stat.total.instructions
     36.60            -3.4       33.20        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.send
     36.03            -3.3       32.69        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     35.22            -3.3       31.93        perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     34.40            -3.2       31.23        perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     28.77            -2.8       25.98        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     28.62            -2.8       25.84        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
     26.84            -2.6       24.24        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
     25.13            -2.5       22.63        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
     23.88            -2.3       21.56        perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
     21.36            -2.1       19.26        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     21.14            -2.1       19.04        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
     20.85            -2.1       18.76        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
     19.58            -2.0       17.56        perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
     19.31            -2.0       17.32        perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
     19.27            -2.0       17.28        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
     18.36            -1.9       16.49        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
     17.57            -1.8       15.78        perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
     18.36            -1.8       16.58 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.send.send_omni_inner
     17.68            -1.8       15.90        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
     17.76            -1.8       15.98        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     18.38            -1.7       16.66 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.send.send_omni_inner.send_tcp_rr
     18.61            -1.6       16.96        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.send.recv_omni.process_requests
     18.41            -1.6       16.78        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.send.recv_omni
      1.97 ± 38%      -1.3        0.67 ± 79%  perf-profile.calltrace.cycles-pp.send_tcp_rr
      1.96 ± 38%      -1.3        0.66 ± 79%  perf-profile.calltrace.cycles-pp.send_omni_inner.send_tcp_rr
     15.27 ±  2%      -1.2       14.05        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
     15.17 ±  2%      -1.2       13.96        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     16.41            -1.2       15.23        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      1.26 ± 39%      -0.9        0.36 ±103%  perf-profile.calltrace.cycles-pp.send.send_omni_inner.send_tcp_rr
     15.46            -0.9       14.55        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     15.52            -0.9       14.63        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.24 ± 39%      -0.9        0.35 ±103%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.send.send_omni_inner.send_tcp_rr
      8.98 ±  2%      -0.8        8.19        perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
     14.72            -0.7       14.04        perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      8.47            -0.7        7.79 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.send_omni_inner.send_tcp_rr
      8.29            -0.7        7.62 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.send_omni_inner
      0.90 ± 27%      -0.6        0.25 ±100%  perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      8.70            -0.6        8.15        perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      8.45            -0.6        7.90        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      8.19            -0.5        7.66        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
      5.48            -0.5        4.98 ±  2%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
      6.50            -0.5        6.01 ±  2%  perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      5.66            -0.5        5.16 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      8.17            -0.5        7.68        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.recv_omni
      5.73            -0.5        5.24 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
      8.38            -0.5        7.90        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.recv_omni.process_requests
      7.32            -0.4        6.92        perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
      5.53            -0.4        5.14        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
      3.99            -0.4        3.61 ±  2%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      5.30            -0.4        4.92        perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
      3.09            -0.3        2.80 ±  2%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      2.18            -0.2        1.94 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken
      0.55            -0.2        0.34 ± 70%  perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
      2.73            -0.2        2.54        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
      1.87            -0.2        1.68 ±  3%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      2.10            -0.2        1.94        perf-profile.calltrace.cycles-pp.sysvec_reschedule_ipi.asm_sysvec_reschedule_ipi.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state
      1.65            -0.1        1.50 ±  2%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      1.68            -0.1        1.53 ±  2%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
      2.02            -0.1        1.89        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle
      1.98 ±  2%      -0.1        1.86        perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
      1.70            -0.1        1.58 ±  2%  perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      1.62            -0.1        1.50 ±  2%  perf-profile.calltrace.cycles-pp.__alloc_skb.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      2.23            -0.1        2.12 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      2.65            -0.1        2.54        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      1.32            -0.1        1.21 ±  2%  perf-profile.calltrace.cycles-pp.kfree.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack
      2.13            -0.1        2.02 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      2.60            -0.1        2.50        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      1.55            -0.1        1.46 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common
      0.94            -0.1        0.86 ±  2%  perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
      1.21 ±  2%      -0.1        1.14 ±  3%  perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      0.90            -0.1        0.83        perf-profile.calltrace.cycles-pp.irqentry_enter.sysvec_reschedule_ipi.asm_sysvec_reschedule_ipi.acpi_idle_do_entry.acpi_idle_enter
      0.61            -0.1        0.54 ±  4%  perf-profile.calltrace.cycles-pp.nr_iowait_cpu.tick_nohz_idle_exit.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
      0.58 ±  2%      -0.1        0.52        perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.77 ±  2%      -0.1        0.72 ±  3%  perf-profile.calltrace.cycles-pp.native_apic_msr_eoi_write.sysvec_reschedule_ipi.asm_sysvec_reschedule_ipi.acpi_idle_do_entry.acpi_idle_enter
      1.14            -0.0        1.10 ±  2%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
      1.05 ±  2%      +0.2        1.21 ±  3%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.63            +0.2        0.80 ±  3%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
      1.89            +0.2        2.08 ±  2%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
      0.00            +0.5        0.54 ±  5%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     44.47            +0.8       45.32        perf-profile.calltrace.cycles-pp.asm_sysvec_reschedule_ipi.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
      0.00            +1.1        1.07 ±  8%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_idle_do_entry
      6.15            +1.1        7.30        perf-profile.calltrace.cycles-pp.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.00            +1.2        1.16 ±  7%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_idle_do_entry.acpi_idle_enter
      0.71 ±  2%      +1.2        1.96 ±  6%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state
     39.37            +4.5       43.85        perf-profile.calltrace.cycles-pp.cpu_startup_entry.secondary_startup_64_no_verify
     39.23            +4.5       43.71        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
     39.66            +4.6       44.22        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     28.60            +4.8       33.37        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     28.69            +4.8       33.47        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
     27.08            +4.8       31.91        perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     31.37            +5.0       36.35        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
      2.56 ±  2%      +7.7       10.24 ±  6%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
     54.71            -4.2       50.49        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     54.30            -4.2       50.09        perf-profile.children.cycles-pp.do_syscall_64
     38.52            -3.4       35.14        perf-profile.children.cycles-pp.send
     36.82            -3.2       33.58        perf-profile.children.cycles-pp.__x64_sys_sendto
     36.63            -3.2       33.41        perf-profile.children.cycles-pp.__sys_sendto
     36.06            -3.2       32.90        perf-profile.children.cycles-pp.sock_sendmsg
     35.28            -3.1       32.18        perf-profile.children.cycles-pp.tcp_sendmsg
     34.49            -3.0       31.49        perf-profile.children.cycles-pp.tcp_sendmsg_locked
     28.93            -2.5       26.39        perf-profile.children.cycles-pp.__tcp_push_pending_frames
     28.82            -2.5       26.29        perf-profile.children.cycles-pp.tcp_write_xmit
     29.26            -2.3       26.92 ±  2%  perf-profile.children.cycles-pp.send_tcp_rr
     29.40            -2.3       27.11        perf-profile.children.cycles-pp.recv_omni
     29.41            -2.3       27.12        perf-profile.children.cycles-pp.process_requests
     29.56            -2.3       27.30        perf-profile.children.cycles-pp.send_omni_inner
     27.13            -1.9       25.22        perf-profile.children.cycles-pp.__tcp_transmit_skb
     25.38            -1.9       23.51        perf-profile.children.cycles-pp.__ip_queue_xmit
     24.09            -1.7       22.35        perf-profile.children.cycles-pp.ip_finish_output2
     21.72            -1.7       20.03        perf-profile.children.cycles-pp.__local_bh_enable_ip
     21.32            -1.6       19.72        perf-profile.children.cycles-pp.do_softirq
     19.77            -1.5       18.27        perf-profile.children.cycles-pp.net_rx_action
     19.48            -1.5       18.00        perf-profile.children.cycles-pp.__napi_poll
     19.44            -1.5       17.97        perf-profile.children.cycles-pp.process_backlog
     21.18            -1.5       19.72        perf-profile.children.cycles-pp.__softirqentry_text_start
     18.52            -1.4       17.12        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     17.76            -1.3       16.42        perf-profile.children.cycles-pp.tcp_v4_rcv
     17.85            -1.3       16.52        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     17.91            -1.3       16.59        perf-profile.children.cycles-pp.ip_local_deliver_finish
     15.82            -1.3       14.56        perf-profile.children.cycles-pp.tcp_v4_do_rcv
     15.73            -1.2       14.49        perf-profile.children.cycles-pp.tcp_rcv_established
     19.41            -1.1       18.30        perf-profile.children.cycles-pp.recv
     11.24            -1.0       10.19        perf-profile.children.cycles-pp.__schedule
     16.57            -1.0       15.56        perf-profile.children.cycles-pp.__sys_recvfrom
      9.40            -1.0        8.40        perf-profile.children.cycles-pp.sk_wait_data
     16.62            -1.0       15.63        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     15.85            -1.0       14.87        perf-profile.children.cycles-pp.tcp_recvmsg
     15.91            -1.0       14.93        perf-profile.children.cycles-pp.inet_recvmsg
     15.29            -0.9       14.37        perf-profile.children.cycles-pp.tcp_recvmsg_locked
      9.07            -0.8        8.32        perf-profile.children.cycles-pp.sock_def_readable
      8.82            -0.7        8.08        perf-profile.children.cycles-pp.__wake_up_common_lock
      8.54            -0.7        7.82        perf-profile.children.cycles-pp.__wake_up_common
      6.86            -0.7        6.14        perf-profile.children.cycles-pp.wait_woken
      6.04            -0.7        5.36        perf-profile.children.cycles-pp.schedule_timeout
      5.96            -0.7        5.29        perf-profile.children.cycles-pp.schedule
      7.78            -0.6        7.14        perf-profile.children.cycles-pp.try_to_wake_up
      1.29            -0.4        0.88 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      5.59            -0.4        5.21        perf-profile.children.cycles-pp.schedule_idle
      4.25            -0.4        3.88        perf-profile.children.cycles-pp.tcp_ack
      3.25            -0.3        2.90        perf-profile.children.cycles-pp.pick_next_task_fair
      2.31            -0.3        2.00 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      3.31            -0.3        3.02        perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      2.03            -0.2        1.78 ±  2%  perf-profile.children.cycles-pp.dequeue_entity
      2.40 ±  2%      -0.2        2.17        perf-profile.children.cycles-pp.__kfree_skb
      2.80            -0.2        2.60        perf-profile.children.cycles-pp.skb_copy_datagram_iter
      2.76            -0.2        2.57        perf-profile.children.cycles-pp.__skb_datagram_iter
      2.36            -0.2        2.17        perf-profile.children.cycles-pp.ttwu_do_activate
      2.26            -0.2        2.08        perf-profile.children.cycles-pp.enqueue_task_fair
      1.80            -0.2        1.62        perf-profile.children.cycles-pp.tcp_stream_alloc_skb
      2.14            -0.2        1.97        perf-profile.children.cycles-pp.sysvec_reschedule_ipi
      1.86            -0.2        1.71 ±  2%  perf-profile.children.cycles-pp.update_load_avg
      1.70            -0.1        1.56        perf-profile.children.cycles-pp.enqueue_entity
      1.86            -0.1        1.72        perf-profile.children.cycles-pp.skb_release_data
      0.38 ±  9%      -0.1        0.24 ±  7%  perf-profile.children.cycles-pp.newidle_balance
      2.12            -0.1        1.99        perf-profile.children.cycles-pp.__dev_queue_xmit
      0.88            -0.1        0.75 ±  2%  perf-profile.children.cycles-pp.update_curr
      1.73            -0.1        1.60        perf-profile.children.cycles-pp.__alloc_skb
      2.10            -0.1        1.97        perf-profile.children.cycles-pp.set_next_entity
      1.13            -0.1        1.01 ±  2%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      1.07            -0.1        0.95 ±  2%  perf-profile.children.cycles-pp.check_preempt_curr
      1.28            -0.1        1.17 ±  2%  perf-profile.children.cycles-pp.select_task_rq
      0.67 ±  2%      -0.1        0.57 ±  2%  perf-profile.children.cycles-pp.release_sock
      1.40            -0.1        1.30 ±  2%  perf-profile.children.cycles-pp.kfree
      1.11            -0.1        1.02 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      1.08            -0.1        0.99        perf-profile.children.cycles-pp._copy_to_iter
      0.51 ±  9%      -0.1        0.42 ±  6%  perf-profile.children.cycles-pp.skb_release_head_state
      0.44 ± 10%      -0.1        0.35 ±  6%  perf-profile.children.cycles-pp.dst_release
      0.71            -0.1        0.62        perf-profile.children.cycles-pp.finish_task_switch
      0.97            -0.1        0.88 ±  2%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      1.07            -0.1        0.99 ±  2%  perf-profile.children.cycles-pp.__slab_free
      0.68            -0.1        0.60 ±  3%  perf-profile.children.cycles-pp.__ksize
      0.90            -0.1        0.82 ±  3%  perf-profile.children.cycles-pp.prepare_task_switch
      1.72            -0.1        1.64        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.89            -0.1        0.82 ±  2%  perf-profile.children.cycles-pp.copyout
      0.92            -0.1        0.84        perf-profile.children.cycles-pp.__check_object_size
      0.59            -0.1        0.51 ±  2%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.92 ±  2%      -0.1        0.85        perf-profile.children.cycles-pp.skb_do_copy_data_nocache
      0.84            -0.1        0.77 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      2.30            -0.1        2.23        perf-profile.children.cycles-pp.update_rq_clock
      0.56            -0.1        0.50 ±  3%  perf-profile.children.cycles-pp.sk_page_frag_refill
      1.08            -0.1        1.02 ±  2%  perf-profile.children.cycles-pp.__switch_to
      0.60 ±  4%      -0.1        0.53 ±  3%  perf-profile.children.cycles-pp.tcp_eat_recv_skb
      0.65            -0.1        0.59 ±  3%  perf-profile.children.cycles-pp.nr_iowait_cpu
      1.09            -0.1        1.02        perf-profile.children.cycles-pp.loopback_xmit
      0.54            -0.1        0.48 ±  3%  perf-profile.children.cycles-pp.skb_page_frag_refill
      1.17            -0.1        1.11        perf-profile.children.cycles-pp.xmit_one
      0.98 ±  3%      -0.1        0.92        perf-profile.children.cycles-pp.__inet_lookup_established
      0.77            -0.1        0.71 ±  3%  perf-profile.children.cycles-pp.remove_wait_queue
      0.12 ± 44%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.ipv4_dst_check
      0.82            -0.1        0.76 ±  3%  perf-profile.children.cycles-pp.tcp_queue_rcv
      0.75            -0.1        0.70        perf-profile.children.cycles-pp.simple_copy_to_iter
      1.21            -0.1        1.16        perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.58 ±  2%      -0.1        0.53        perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.93            -0.1        0.88        perf-profile.children.cycles-pp.irqentry_enter
      0.58 ±  2%      -0.1        0.53 ±  4%  perf-profile.children.cycles-pp.select_idle_sibling
      0.65            -0.1        0.60        perf-profile.children.cycles-pp.__might_resched
      0.61            -0.0        0.56 ±  3%  perf-profile.children.cycles-pp.___perf_sw_event
      0.13 ±  7%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.resched_curr
      0.12 ± 36%      -0.0        0.08 ± 13%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.46 ±  2%      -0.0        0.41 ±  5%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.67 ±  2%      -0.0        0.62        perf-profile.children.cycles-pp._copy_from_iter
      0.60            -0.0        0.56        perf-profile.children.cycles-pp.__entry_text_start
      0.53 ±  2%      -0.0        0.49 ±  3%  perf-profile.children.cycles-pp.lock_sock_nested
      0.13 ± 29%      -0.0        0.08 ± 11%  perf-profile.children.cycles-pp.__sk_dst_check
      0.50            -0.0        0.46 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.57 ±  2%      -0.0        0.53 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.35            -0.0        0.31 ±  3%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.66            -0.0        0.62        perf-profile.children.cycles-pp.__x2apic_send_IPI_dest
      0.64 ±  2%      -0.0        0.60        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.57 ±  2%      -0.0        0.53 ±  2%  perf-profile.children.cycles-pp.recv_data
      0.53 ±  3%      -0.0        0.49        perf-profile.children.cycles-pp.pick_next_entity
      0.44 ±  2%      -0.0        0.41        perf-profile.children.cycles-pp.aa_sk_perm
      0.33 ±  3%      -0.0        0.30 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.22 ±  2%      -0.0        0.18 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.20 ±  5%      -0.0        0.17 ±  8%  perf-profile.children.cycles-pp.sock_rfree
      0.12 ±  4%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.71            -0.0        0.68 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.43 ±  2%      -0.0        0.40 ±  5%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.44 ±  2%      -0.0        0.40 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
      0.40 ±  2%      -0.0        0.37 ±  4%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.37 ±  2%      -0.0        0.34 ±  3%  perf-profile.children.cycles-pp.save_fpregs_to_fpstate
      0.21 ±  3%      -0.0        0.19 ±  6%  perf-profile.children.cycles-pp.perf_tp_event
      0.20 ±  4%      -0.0        0.17 ±  6%  perf-profile.children.cycles-pp.tcp_update_skb_after_send
      0.30 ±  3%      -0.0        0.28 ±  2%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.11 ±  5%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.inet_ehashfn
      0.27            -0.0        0.24 ±  3%  perf-profile.children.cycles-pp.copyin
      0.20 ±  2%      -0.0        0.17 ±  7%  perf-profile.children.cycles-pp.tcp_cleanup_rbuf
      0.14 ±  6%      -0.0        0.12 ±  9%  perf-profile.children.cycles-pp.update_min_vruntime
      0.39            -0.0        0.36        perf-profile.children.cycles-pp.__netif_rx
      0.22 ±  2%      -0.0        0.20 ±  4%  perf-profile.children.cycles-pp.reweight_entity
      0.10 ±  3%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.import_single_range
      0.10 ±  5%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.36            -0.0        0.34        perf-profile.children.cycles-pp.netif_rx_internal
      0.30 ±  2%      -0.0        0.27        perf-profile.children.cycles-pp.__cond_resched
      0.33            -0.0        0.31        perf-profile.children.cycles-pp.enqueue_to_backlog
      0.42            -0.0        0.40 ±  2%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.08 ±  6%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.skb_clone_tx_timestamp
      0.11            -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.perf_trace_buf_update
      0.08 ±  5%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.tcp_rate_skb_delivered
      0.08 ±  6%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.tcp_release_cb
      0.97            +0.0        1.00        perf-profile.children.cycles-pp.sched_clock_cpu
      0.80            +0.0        0.83        perf-profile.children.cycles-pp.native_sched_clock
      0.04 ± 45%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.run_rebalance_domains
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.05 ±  8%      +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.05            +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.09 ±  7%      +0.1        0.17 ±  5%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.10 ±  3%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.scheduler_tick
      0.08 ± 12%      +0.1        0.19 ± 27%  perf-profile.children.cycles-pp.tick_irq_enter
      0.16 ±  3%      +0.1        0.29 ±  6%  perf-profile.children.cycles-pp.update_process_times
      0.01 ±223%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.09 ± 10%      +0.1        0.22 ± 21%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.17 ±  2%      +0.1        0.30 ±  5%  perf-profile.children.cycles-pp.tick_sched_handle
      1.09            +0.2        1.25 ±  3%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.06 ±  7%      +0.2        0.22 ± 11%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.66            +0.2        0.84 ±  3%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.22            +0.2        0.40 ±  5%  perf-profile.children.cycles-pp.tick_sched_timer
      0.19 ± 10%      +0.2        0.37 ±  5%  perf-profile.children.cycles-pp.__irq_exit_rcu
      1.94            +0.2        2.13        perf-profile.children.cycles-pp.menu_select
      0.21 ±  7%      +0.2        0.43 ± 12%  perf-profile.children.cycles-pp.clockevents_program_event
      1.49            +0.3        1.78 ±  7%  perf-profile.children.cycles-pp.ktime_get
      0.31            +0.3        0.64 ±  4%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.61 ±  3%      +0.6        1.23 ±  6%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.64 ±  2%      +0.7        1.34 ±  6%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      1.02 ±  2%      +1.2        2.20 ±  5%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      2.02 ±  2%      +4.5        6.49 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     39.56            +4.6       44.11        perf-profile.children.cycles-pp.do_idle
     39.66            +4.6       44.22        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     39.66            +4.6       44.22        perf-profile.children.cycles-pp.cpu_startup_entry
     28.82            +4.8       33.66        perf-profile.children.cycles-pp.cpuidle_enter_state
     28.89            +4.8       33.74        perf-profile.children.cycles-pp.cpuidle_enter
     27.27            +4.9       32.16        perf-profile.children.cycles-pp.acpi_idle_enter
     26.91            +4.9       31.81        perf-profile.children.cycles-pp.acpi_idle_do_entry
     31.64            +5.0       36.69        perf-profile.children.cycles-pp.cpuidle_idle_call
      1.26            -0.4        0.86 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      1.88            -0.1        1.76        perf-profile.self.cycles-pp.update_rq_clock
      1.10            -0.1        1.01 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.43 ± 11%      -0.1        0.34 ±  7%  perf-profile.self.cycles-pp.dst_release
      1.38 ±  2%      -0.1        1.30        perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.98            -0.1        0.90        perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.67            -0.1        0.59 ±  2%  perf-profile.self.cycles-pp.__ksize
      1.06            -0.1        0.98 ±  2%  perf-profile.self.cycles-pp.__slab_free
      1.68            -0.1        1.60        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.43            -0.1        1.35        perf-profile.self.cycles-pp.set_next_entity
      0.37            -0.1        0.29 ±  3%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.88 ±  2%      -0.1        0.80        perf-profile.self.cycles-pp.update_load_avg
      0.76 ±  2%      -0.1        0.68 ±  2%  perf-profile.self.cycles-pp.enqueue_entity
      0.64            -0.1        0.57 ±  2%  perf-profile.self.cycles-pp.finish_task_switch
      0.54            -0.1        0.47 ±  3%  perf-profile.self.cycles-pp.skb_page_frag_refill
      0.44 ±  2%      -0.1        0.38 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.24 ±  3%      -0.1        0.18 ±  2%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.74            -0.1        0.67 ±  3%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.64            -0.1        0.58 ±  4%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.71 ±  2%      -0.1        0.65        perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      1.04            -0.1        0.98 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.65            -0.1        0.59 ±  3%  perf-profile.self.cycles-pp.__wake_up_common
      0.12 ± 47%      -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.ipv4_dst_check
      1.48            -0.1        1.42 ±  2%  perf-profile.self.cycles-pp.__schedule
      0.73            -0.1        0.68        perf-profile.self.cycles-pp.__check_object_size
      0.13 ±  5%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.resched_curr
      0.73 ±  2%      -0.1        0.68 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.58 ±  2%      -0.1        0.53        perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.12 ± 38%      -0.0        0.07 ±  9%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.81 ±  2%      -0.0        0.76 ±  3%  perf-profile.self.cycles-pp.tcp_queue_rcv
      0.11 ± 32%      -0.0        0.07 ± 16%  perf-profile.self.cycles-pp.ipv4_mtu
      0.71            -0.0        0.66        perf-profile.self.cycles-pp.irqentry_enter
      0.60            -0.0        0.55        perf-profile.self.cycles-pp.__might_resched
      0.31            -0.0        0.26        perf-profile.self.cycles-pp.dequeue_entity
      0.51 ±  4%      -0.0        0.47 ±  4%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.50 ±  2%      -0.0        0.46 ±  3%  perf-profile.self.cycles-pp.kfree
      0.54            -0.0        0.49 ±  2%  perf-profile.self.cycles-pp.___perf_sw_event
      0.44 ±  3%      -0.0        0.40        perf-profile.self.cycles-pp.skb_release_data
      0.43            -0.0        0.39 ±  3%  perf-profile.self.cycles-pp.recv_omni
      0.45            -0.0        0.41 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.44 ±  2%      -0.0        0.40 ±  3%  perf-profile.self.cycles-pp.select_task_rq
      0.84            -0.0        0.81        perf-profile.self.cycles-pp.__skb_datagram_iter
      0.28            -0.0        0.24 ±  5%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.66            -0.0        0.62 ±  2%  perf-profile.self.cycles-pp.__x2apic_send_IPI_dest
      0.51 ±  3%      -0.0        0.48        perf-profile.self.cycles-pp.pick_next_entity
      0.63 ±  2%      -0.0        0.59        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.40 ±  3%      -0.0        0.36 ±  2%  perf-profile.self.cycles-pp.tcp_eat_recv_skb
      0.82            -0.0        0.79        perf-profile.self.cycles-pp.wait_woken
      0.57            -0.0        0.53 ±  3%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.38 ±  2%      -0.0        0.35 ±  2%  perf-profile.self.cycles-pp.sk_wait_data
      0.18 ±  4%      -0.0        0.15 ±  8%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.52 ±  2%      -0.0        0.49 ±  2%  perf-profile.self.cycles-pp.recv_data
      0.20 ±  7%      -0.0        0.16 ±  7%  perf-profile.self.cycles-pp.sock_rfree
      0.32 ±  2%      -0.0        0.30 ±  3%  perf-profile.self.cycles-pp.sysvec_reschedule_ipi
      0.36 ±  2%      -0.0        0.34 ±  2%  perf-profile.self.cycles-pp.save_fpregs_to_fpstate
      0.28            -0.0        0.25 ±  6%  perf-profile.self.cycles-pp.prepare_task_switch
      0.27 ±  3%      -0.0        0.24 ±  3%  perf-profile.self.cycles-pp.__might_sleep
      0.34 ±  2%      -0.0        0.31 ±  4%  perf-profile.self.cycles-pp.recv
      0.26 ±  2%      -0.0        0.23 ±  2%  perf-profile.self.cycles-pp.net_rx_action
      0.17 ±  2%      -0.0        0.14 ±  5%  perf-profile.self.cycles-pp.newidle_balance
      0.10 ±  5%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.inet_ehashfn
      0.45 ±  2%      -0.0        0.43 ±  2%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.19 ±  4%      -0.0        0.17 ±  7%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.22 ±  2%      -0.0        0.19 ±  3%  perf-profile.self.cycles-pp.reweight_entity
      0.25 ±  3%      -0.0        0.22 ±  2%  perf-profile.self.cycles-pp.aa_sk_perm
      0.21 ±  2%      -0.0        0.19 ±  4%  perf-profile.self.cycles-pp.process_backlog
      0.19 ±  3%      -0.0        0.17 ±  2%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.12 ±  6%      -0.0        0.10 ±  9%  perf-profile.self.cycles-pp.update_min_vruntime
      0.33 ±  2%      -0.0        0.31 ±  3%  perf-profile.self.cycles-pp.loopback_xmit
      0.10 ±  5%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.09 ±  4%      -0.0        0.07 ±  9%  perf-profile.self.cycles-pp.import_single_range
      0.11 ±  6%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.tcp_update_skb_after_send
      0.13 ±  2%      -0.0        0.11 ±  8%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.08 ±  7%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.cpus_share_cache
      0.08 ±  9%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.tcp_tso_segs
      0.18 ±  2%      -0.0        0.17 ±  2%  perf-profile.self.cycles-pp.__cond_resched
      0.13 ±  3%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.tcp_rate_skb_delivered
      0.12 ±  3%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.07 ±  5%      +0.0        0.09 ±  6%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.06 ±  7%      +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.tcp_release_cb
      0.15 ±  4%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.sock_sendmsg
      0.04 ± 45%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.77            +0.0        0.80        perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.05 ±  8%      +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.06 ±  8%      +0.2        0.22 ± 12%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.22 ±  3%      +0.2        0.42 ±  4%  perf-profile.self.cycles-pp.release_sock
      0.64 ±  2%      +0.3        0.93 ± 12%  perf-profile.self.cycles-pp.ktime_get
     22.15            +3.8       25.92        perf-profile.self.cycles-pp.acpi_idle_do_entry




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://01.org/lkp



View attachment "config-5.18.0-rc4-00072-g02dcea6024d4" of type "text/plain" (162679 bytes)

View attachment "job-script" of type "text/plain" (8366 bytes)

View attachment "job.yaml" of type "text/plain" (5713 bytes)

View attachment "reproduce" of type "text/plain" (1875 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ