lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Thu, 21 May 2020 16:45:35 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Yuqi Jin <jinyuqi@...wei.com>
Cc:     "David S. Miller" <davem@...emloft.net>,
        Peter Zijlstra <peterz@...radead.org>,
        Eric Dumazet <edumazet@...gle.com>,
        Alexey Kuznetsov <kuznet@....inr.ac.ru>,
        Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>,
        Jakub Kicinski <kuba@...nel.org>,
        Jiri Pirko <jiri@...nulli.us>,
        Arvind Sankar <nivedita@...m.mit.edu>,
        Jiong Wang <jiongwang@...wei.com>,
        Shaokun Zhang <zhangshaokun@...ilicon.com>,
        LKML <linux-kernel@...r.kernel.org>, netdev@...r.kernel.org,
        lkp@...ts.01.org
Subject: [net] a6211caa63: netperf.Throughput_total_tps 160.7% improvement

Greeting,

FYI, we noticed a 160.7% improvement of netperf.Throughput_total_tps due to commit:


commit: a6211caa634da39d861a47437ffcda8b38ef421b ("net: revert "net: get rid of an signed integer overflow in ip_idents_reserve()"")
https://git.kernel.org/cgit/linux/kernel/git/davem/net.git master

in testcase: netperf
on test machine: 104 threads Skylake with 192G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 200%
	cluster: cs-localhost
	test: UDP_RR
	cpufreq_governor: performance
	ucode: 0x2000065

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-20191114.cgz/300s/lkp-skl-fpga01/UDP_RR/netperf/0x2000065

commit: 
  61d0301e6c ("dt-bindings: net: dsa: b53: Add missing size and address cells to example")
  a6211caa63 ("net: revert "net: get rid of an signed integer overflow in ip_idents_reserve()"")

61d0301e6c05db55 a6211caa634da39d861a47437ff 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   2382389 ±  3%    +160.7%    6210158        netperf.Throughput_total_tps
     11453 ±  3%    +160.7%      29856        netperf.Throughput_tps
 2.834e+08 ± 26%     +94.4%  5.509e+08 ±  8%  netperf.time.involuntary_context_switches
     14146            -6.7%      13198        netperf.time.system_time
      1458           +63.3%       2382        netperf.time.user_time
 4.302e+08 ± 19%    +203.9%  1.308e+09 ±  3%  netperf.time.voluntary_context_switches
 7.147e+08 ±  3%    +160.7%  1.863e+09        netperf.workload
     28604 ± 30%     -70.3%       8507 ±170%  numa-numastat.node1.other_node
    765.00 ±  4%     +13.3%     867.00 ±  4%  slabinfo.mnt_cache.active_objs
    765.00 ±  4%     +13.3%     867.00 ±  4%  slabinfo.mnt_cache.num_objs
     13.15 ±  2%      +9.1       22.29        mpstat.cpu.all.soft%
     76.18           -15.1       61.06        mpstat.cpu.all.sys%
      9.10            +5.9       14.97        mpstat.cpu.all.usr%
  35582343 ±  3%     -24.5%   26879340 ± 10%  cpuidle.C1.time
   6354413 ±  3%     -18.3%    5193270 ± 10%  cpuidle.C1.usage
   6640796 ± 11%     +29.1%    8574021 ±  3%  cpuidle.POLL.time
   2047691 ± 14%     +44.1%    2949986 ±  2%  cpuidle.POLL.usage
     88.25            -6.8%      82.25        vmstat.cpu.sy
      8.25 ±  5%     +69.7%      14.00        vmstat.cpu.us
   4677547 ±  3%    +159.8%   12150848        vmstat.system.cs
    213476            +1.5%     216585        vmstat.system.in
     16116 ± 17%     -49.6%       8120 ± 25%  numa-meminfo.node0.Inactive
     15901 ± 17%     -49.8%       7986 ± 26%  numa-meminfo.node0.Inactive(anon)
     19329 ± 11%     -25.0%      14497 ± 14%  numa-meminfo.node0.Mapped
     17899 ± 14%     -48.3%       9256 ± 31%  numa-meminfo.node0.Shmem
     14756 ± 19%     +47.4%      21749 ±  9%  numa-meminfo.node1.Inactive
     14595 ± 19%     +47.2%      21486 ±  9%  numa-meminfo.node1.Inactive(anon)
      3975 ± 17%     -49.8%       1996 ± 26%  numa-vmstat.node0.nr_inactive_anon
      4864 ± 11%     -24.6%       3669 ± 14%  numa-vmstat.node0.nr_mapped
      4474 ± 14%     -48.3%       2314 ± 31%  numa-vmstat.node0.nr_shmem
      3975 ± 17%     -49.8%       1996 ± 26%  numa-vmstat.node0.nr_zone_inactive_anon
      3668 ± 18%     +45.4%       5334 ± 10%  numa-vmstat.node1.nr_inactive_anon
      3668 ± 18%     +45.4%       5334 ± 10%  numa-vmstat.node1.nr_zone_inactive_anon
      7597            -3.0%       7370        proc-vmstat.nr_inactive_anon
      8899            -2.8%       8651        proc-vmstat.nr_mapped
     26511 ±  3%      -3.9%      25486 ±  2%  proc-vmstat.nr_shmem
      7597            -3.0%       7370        proc-vmstat.nr_zone_inactive_anon
    781674            +1.5%     793716        proc-vmstat.numa_hit
    747985            +1.6%     760013        proc-vmstat.numa_local
     37078 ±  4%     -17.0%      30783 ±  7%  proc-vmstat.pgactivate
    851230            +1.8%     866712        proc-vmstat.pgfault
      3.21 ±  9%     -29.9%       2.25 ± 12%  sched_debug.cfs_rq:/.load_avg.min
  -1551214          -150.7%     786316 ±223%  sched_debug.cfs_rq:/.spread0.avg
    584.71 ±  7%     -10.6%     522.75 ±  4%  sched_debug.cfs_rq:/.util_avg.min
    135.54 ±  3%      +8.1%     146.50 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    711376 ± 41%     +41.7%    1008145 ±  6%  sched_debug.cpu.avg_idle.max
      4410 ±  4%     -16.7%       3674 ±  7%  sched_debug.cpu.avg_idle.min
   6783672 ±  3%    +161.8%   17758928        sched_debug.cpu.nr_switches.avg
   8197064 ±  4%    +130.4%   18889627        sched_debug.cpu.nr_switches.max
   4787733 ± 10%    +236.1%   16093322 ±  2%  sched_debug.cpu.nr_switches.min
    992865 ± 18%     -48.2%     514302 ± 17%  sched_debug.cpu.nr_switches.stddev
     70.67 ±  8%     -26.8%      51.75 ± 17%  sched_debug.cpu.nr_uninterruptible.max
 1.114e+10 ±  3%    +171.2%  3.021e+10        perf-stat.i.branch-instructions
      1.66            -0.1        1.60        perf-stat.i.branch-miss-rate%
 1.834e+08 ±  3%    +160.7%  4.782e+08        perf-stat.i.branch-misses
  41518884 ± 19%    +119.3%   91068934 ± 21%  perf-stat.i.cache-misses
 3.629e+08          +189.7%  1.051e+09 ±  4%  perf-stat.i.cache-references
   4716783 ±  3%    +160.3%   12278919        perf-stat.i.context-switches
      4.99 ±  3%     -64.7%       1.76        perf-stat.i.cpi
 2.855e+11            -4.3%  2.733e+11        perf-stat.i.cpu-cycles
      7245 ± 21%     -53.6%       3361 ± 28%  perf-stat.i.cycles-between-cache-misses
  92817683 ±  4%    +201.7%    2.8e+08 ±  7%  perf-stat.i.dTLB-load-misses
 1.665e+10 ±  3%    +172.1%   4.53e+10        perf-stat.i.dTLB-loads
   9731355 ±  7%    +202.8%   29466191 ±  4%  perf-stat.i.dTLB-store-misses
 9.977e+09 ±  3%    +172.2%  2.715e+10        perf-stat.i.dTLB-stores
     53.61 ±  2%      +4.4       58.02 ±  2%  perf-stat.i.iTLB-load-miss-rate%
  48591600 ±  5%    +189.8%  1.408e+08 ±  4%  perf-stat.i.iTLB-load-misses
  41279422 ±  3%    +140.2%   99161913        perf-stat.i.iTLB-loads
 5.709e+10 ±  3%    +172.7%  1.557e+11        perf-stat.i.instructions
      0.20 ±  2%    +178.6%       0.57        perf-stat.i.ipc
      2.75            -4.5%       2.63        perf-stat.i.metric.GHz
      0.57 ± 39%     -75.5%       0.14 ± 30%  perf-stat.i.metric.K/sec
    368.55 ±  3%    +171.3%     999.80        perf-stat.i.metric.M/sec
      2707            +1.8%       2755        perf-stat.i.minor-faults
   5928215 ± 27%    +184.2%   16848721 ± 23%  perf-stat.i.node-load-misses
    101013 ± 26%     +85.7%     187548 ± 14%  perf-stat.i.node-loads
   2780627 ±  8%     +96.4%    5460264 ±  8%  perf-stat.i.node-store-misses
      2707            +1.8%       2755        perf-stat.i.page-faults
      1.65            -0.1        1.58        perf-stat.overall.branch-miss-rate%
      5.01 ±  3%     -64.9%       1.76        perf-stat.overall.cpi
      7162 ± 21%     -55.8%       3165 ± 24%  perf-stat.overall.cycles-between-cache-misses
     54.05 ±  2%      +4.6       58.65 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      0.20 ±  3%    +184.9%       0.57        perf-stat.overall.ipc
     24334            +4.6%      25454        perf-stat.overall.path-length
  1.11e+10 ±  3%    +171.1%  3.011e+10        perf-stat.ps.branch-instructions
 1.828e+08 ±  3%    +160.6%  4.764e+08        perf-stat.ps.branch-misses
  41392364 ± 19%    +119.2%   90737530 ± 21%  perf-stat.ps.cache-misses
 3.618e+08          +189.6%  1.048e+09 ±  4%  perf-stat.ps.cache-references
   4701274 ±  3%    +160.2%   12234892        perf-stat.ps.context-switches
 2.845e+11            -4.3%  2.723e+11        perf-stat.ps.cpu-cycles
  92531227 ±  4%    +201.5%   2.79e+08 ±  7%  perf-stat.ps.dTLB-load-misses
 1.659e+10 ±  3%    +172.0%  4.513e+10        perf-stat.ps.dTLB-loads
   9700889 ±  7%    +202.7%   29361270 ±  4%  perf-stat.ps.dTLB-store-misses
 9.944e+09 ±  3%    +172.1%  2.705e+10        perf-stat.ps.dTLB-stores
  48437762 ±  5%    +189.7%  1.403e+08 ±  4%  perf-stat.ps.iTLB-load-misses
  41133588 ±  3%    +140.2%   98803267        perf-stat.ps.iTLB-loads
  5.69e+10 ±  3%    +172.6%  1.551e+11        perf-stat.ps.instructions
   5909971 ± 27%    +184.0%   16786827 ± 23%  perf-stat.ps.node-load-misses
    101876 ± 25%     +83.8%     187272 ± 14%  perf-stat.ps.node-loads
   2771558 ±  8%     +96.3%    5440535 ±  8%  perf-stat.ps.node-store-misses
 1.739e+13 ±  3%    +172.7%  4.742e+13        perf-stat.total.instructions
      4608 ± 18%    +186.0%      13177 ± 62%  interrupts.CPU0.RES:Rescheduling_interrupts
      4349 ± 98%    +215.2%      13708 ± 32%  interrupts.CPU1.RES:Rescheduling_interrupts
      3614 ± 31%    +101.4%       7279 ± 39%  interrupts.CPU10.RES:Rescheduling_interrupts
      4465 ± 66%    +209.0%      13796 ± 42%  interrupts.CPU101.RES:Rescheduling_interrupts
      6524 ± 45%    +201.9%      19696 ± 41%  interrupts.CPU102.RES:Rescheduling_interrupts
      5961 ± 31%    +290.9%      23304 ± 60%  interrupts.CPU103.RES:Rescheduling_interrupts
      3330 ± 44%    +346.9%      14882 ± 51%  interrupts.CPU15.RES:Rescheduling_interrupts
      2516 ± 59%   +1044.5%      28798 ±120%  interrupts.CPU18.RES:Rescheduling_interrupts
      3744 ± 42%    +364.1%      17377 ± 49%  interrupts.CPU2.RES:Rescheduling_interrupts
      4907 ± 25%    +206.7%      15051 ± 11%  interrupts.CPU21.RES:Rescheduling_interrupts
      3520 ± 55%    +411.1%      17994 ± 52%  interrupts.CPU23.RES:Rescheduling_interrupts
      2886 ± 47%    +327.9%      12351 ± 43%  interrupts.CPU27.RES:Rescheduling_interrupts
      2588 ± 81%    +399.6%      12930 ± 49%  interrupts.CPU29.RES:Rescheduling_interrupts
      3549 ±  5%    +386.1%      17255 ± 97%  interrupts.CPU3.RES:Rescheduling_interrupts
      2984 ± 44%    +286.1%      11523 ± 80%  interrupts.CPU30.RES:Rescheduling_interrupts
      3237 ± 25%    +443.6%      17599 ± 71%  interrupts.CPU31.RES:Rescheduling_interrupts
      4551 ± 75%    +168.3%      12211 ± 61%  interrupts.CPU33.RES:Rescheduling_interrupts
      3952 ± 83%    +413.6%      20300 ± 45%  interrupts.CPU35.RES:Rescheduling_interrupts
      2983 ± 86%    +632.2%      21846 ± 73%  interrupts.CPU36.RES:Rescheduling_interrupts
      4917 ± 58%    +292.2%      19282 ± 81%  interrupts.CPU40.RES:Rescheduling_interrupts
      2589 ± 70%    +614.5%      18499 ± 92%  interrupts.CPU44.RES:Rescheduling_interrupts
      4009 ±108%    +302.0%      16116 ± 58%  interrupts.CPU45.RES:Rescheduling_interrupts
      5099 ± 29%    +125.3%      11486 ± 41%  interrupts.CPU46.RES:Rescheduling_interrupts
      4820 ± 39%    +176.2%      13312 ± 55%  interrupts.CPU47.RES:Rescheduling_interrupts
      6377 ± 90%    +201.0%      19197 ± 54%  interrupts.CPU5.RES:Rescheduling_interrupts
      6547 ± 45%    +120.8%      14457 ± 32%  interrupts.CPU50.RES:Rescheduling_interrupts
      4026 ± 48%    +330.7%      17342 ± 61%  interrupts.CPU54.RES:Rescheduling_interrupts
      4342 ± 32%    +248.0%      15110 ±  7%  interrupts.CPU58.RES:Rescheduling_interrupts
      4162 ± 41%    +141.8%      10067 ± 23%  interrupts.CPU59.RES:Rescheduling_interrupts
      4439 ± 30%    +231.9%      14734 ± 31%  interrupts.CPU6.RES:Rescheduling_interrupts
      4160 ± 48%    +226.3%      13575 ± 33%  interrupts.CPU61.RES:Rescheduling_interrupts
      3257 ± 34%    +128.9%       7458 ± 23%  interrupts.CPU62.RES:Rescheduling_interrupts
      2763 ± 17%    +264.6%      10077 ± 24%  interrupts.CPU63.RES:Rescheduling_interrupts
      4463 ± 70%    +211.2%      13889 ± 73%  interrupts.CPU64.RES:Rescheduling_interrupts
      3378 ± 47%    +568.4%      22585 ±129%  interrupts.CPU66.RES:Rescheduling_interrupts
      3033 ± 36%    +285.2%      11684 ± 24%  interrupts.CPU67.RES:Rescheduling_interrupts
      4746 ± 43%    +174.6%      13035 ± 20%  interrupts.CPU68.RES:Rescheduling_interrupts
      3248 ± 23%    +222.9%      10491 ± 40%  interrupts.CPU7.RES:Rescheduling_interrupts
      2247 ± 34%    +348.9%      10086 ± 37%  interrupts.CPU70.RES:Rescheduling_interrupts
      3328 ± 46%    +466.6%      18858 ± 64%  interrupts.CPU72.RES:Rescheduling_interrupts
      4881 ± 27%    +217.3%      15486 ± 25%  interrupts.CPU73.RES:Rescheduling_interrupts
      2956 ± 60%    +384.6%      14326 ± 35%  interrupts.CPU75.RES:Rescheduling_interrupts
      3863 ± 61%    +343.6%      17140 ± 67%  interrupts.CPU79.RES:Rescheduling_interrupts
      2850 ± 68%    +367.1%      13316 ± 28%  interrupts.CPU81.RES:Rescheduling_interrupts
      3384 ± 40%    +330.3%      14565 ± 52%  interrupts.CPU83.RES:Rescheduling_interrupts
      4285 ± 42%    +123.6%       9583 ± 26%  interrupts.CPU84.RES:Rescheduling_interrupts
      5129 ± 64%    +133.5%      11976 ± 53%  interrupts.CPU85.RES:Rescheduling_interrupts
      3496 ± 87%    +517.6%      21592 ± 58%  interrupts.CPU87.RES:Rescheduling_interrupts
      2920 ±100%    +523.9%      18218 ± 70%  interrupts.CPU88.RES:Rescheduling_interrupts
      5249 ± 56%    +209.0%      16220 ± 56%  interrupts.CPU92.RES:Rescheduling_interrupts
      4098 ± 78%    +351.4%      18499 ± 75%  interrupts.CPU93.RES:Rescheduling_interrupts
      2070 ± 46%    +640.1%      15323 ± 66%  interrupts.CPU96.RES:Rescheduling_interrupts
      4232 ±109%    +249.3%      14782 ± 58%  interrupts.CPU97.RES:Rescheduling_interrupts
      4886 ± 24%    +119.6%      10731 ± 39%  interrupts.CPU98.RES:Rescheduling_interrupts
    466502 ± 32%    +231.9%    1548094 ± 30%  interrupts.RES:Rescheduling_interrupts
   7611209 ± 14%    +129.9%   17497928 ±  2%  softirqs.CPU0.NET_RX
    117644 ±  3%     -10.1%     105808        softirqs.CPU0.TIMER
   7754285 ±  6%    +125.0%   17446076        softirqs.CPU1.NET_RX
   7661563 ±  4%    +131.7%   17752614 ±  2%  softirqs.CPU10.NET_RX
   6265597 ±  6%    +196.8%   18597014        softirqs.CPU100.NET_RX
    111790           -12.1%      98285        softirqs.CPU100.TIMER
   6003920 ±  5%    +206.5%   18403172        softirqs.CPU101.NET_RX
    112387           -12.7%      98082        softirqs.CPU101.TIMER
   5914242 ±  6%    +214.9%   18622144        softirqs.CPU102.NET_RX
    113186           -13.6%      97820        softirqs.CPU102.TIMER
   6036155 ±  6%    +203.2%   18302912        softirqs.CPU103.NET_RX
    111454           -12.0%      98105        softirqs.CPU103.TIMER
   7865285 ±  6%    +125.9%   17764453        softirqs.CPU11.NET_RX
   7785952 ±  4%    +127.2%   17687980        softirqs.CPU12.NET_RX
   7679186 ± 13%    +131.1%   17745182        softirqs.CPU13.NET_RX
   7489829 ±  7%    +135.1%   17612130        softirqs.CPU14.NET_RX
   7986983 ±  5%    +115.9%   17240928 ±  4%  softirqs.CPU15.NET_RX
   7685814 ±  2%    +131.1%   17758139        softirqs.CPU16.NET_RX
   7763445 ±  2%    +127.8%   17682866        softirqs.CPU17.NET_RX
   7775718 ±  4%    +126.1%   17579079        softirqs.CPU18.NET_RX
   7692242 ±  4%    +127.7%   17513046        softirqs.CPU19.NET_RX
   7318854 ±  9%    +140.9%   17631491        softirqs.CPU2.NET_RX
   7432944 ± 16%    +139.5%   17801499        softirqs.CPU20.NET_RX
   7798319 ±  5%    +129.9%   17927670        softirqs.CPU21.NET_RX
   8016516 ±  3%    +121.0%   17715449        softirqs.CPU22.NET_RX
   7816079 ±  4%    +119.7%   17168660 ±  4%  softirqs.CPU23.NET_RX
   7813370 ±  2%    +127.0%   17735135        softirqs.CPU24.NET_RX
   7868286 ±  4%    +121.9%   17456994        softirqs.CPU25.NET_RX
   5716747 ± 10%    +211.2%   17791004        softirqs.CPU26.NET_RX
    115007 ±  2%     -12.8%     100280        softirqs.CPU26.TIMER
   6070414 ±  4%    +201.9%   18327734        softirqs.CPU27.NET_RX
    114508 ±  2%     -13.8%      98735        softirqs.CPU27.TIMER
   5619088 ± 14%    +225.1%   18267351        softirqs.CPU28.NET_RX
    116359 ±  2%     -13.6%     100482        softirqs.CPU28.TIMER
   6213794 ±  5%    +205.7%   18998636 ±  2%  softirqs.CPU29.NET_RX
    113060           -12.8%      98573        softirqs.CPU29.TIMER
   8030911 ±  4%    +120.5%   17710712        softirqs.CPU3.NET_RX
   5963093 ±  6%    +205.6%   18220250        softirqs.CPU30.NET_RX
    115032           -11.6%     101730 ±  4%  softirqs.CPU30.TIMER
   5971100 ±  9%    +201.7%   18015213        softirqs.CPU31.NET_RX
    113379           -11.2%     100672 ±  2%  softirqs.CPU31.TIMER
   6047995 ±  6%    +197.5%   17992368        softirqs.CPU32.NET_RX
    113267           -12.2%      99398        softirqs.CPU32.TIMER
   6068931 ±  7%    +203.2%   18400418        softirqs.CPU33.NET_RX
    126026 ± 16%     -21.7%      98700        softirqs.CPU33.TIMER
   6144173 ±  8%    +198.9%   18365776        softirqs.CPU34.NET_RX
    113227           -12.9%      98564        softirqs.CPU34.TIMER
   6345890 ±  6%    +191.0%   18468925        softirqs.CPU35.NET_RX
    112662           -12.1%      99063        softirqs.CPU35.TIMER
   5791449 ±  4%    +218.9%   18471601        softirqs.CPU36.NET_RX
    113723           -13.0%      98943        softirqs.CPU36.TIMER
   5811184 ±  8%    +211.4%   18093802 ±  3%  softirqs.CPU37.NET_RX
    113198           -11.5%     100188        softirqs.CPU37.TIMER
   5878689 ±  5%    +214.0%   18457807        softirqs.CPU38.NET_RX
    113697           -13.0%      98900        softirqs.CPU38.TIMER
   6036575 ±  8%    +202.9%   18281922        softirqs.CPU39.NET_RX
    113087           -12.7%      98699        softirqs.CPU39.TIMER
   7715041 ±  4%    +129.6%   17712443 ±  2%  softirqs.CPU4.NET_RX
   5753853 ± 10%    +220.0%   18412975        softirqs.CPU40.NET_RX
    113355 ±  2%     -13.1%      98467        softirqs.CPU40.TIMER
   6192776 ±  3%    +198.0%   18452898        softirqs.CPU41.NET_RX
    112945           -12.9%      98428        softirqs.CPU41.TIMER
   5803388 ±  5%    +217.7%   18435506        softirqs.CPU42.NET_RX
   5918098 ± 10%    +208.9%   18283279        softirqs.CPU43.NET_RX
    112855           -12.8%      98415        softirqs.CPU43.TIMER
   6051506 ±  8%    +201.3%   18230382        softirqs.CPU44.NET_RX
    112687           -12.1%      99002        softirqs.CPU44.TIMER
   5974980 ±  6%    +206.4%   18307269        softirqs.CPU45.NET_RX
    112919           -12.8%      98430        softirqs.CPU45.TIMER
   5725365 ± 12%    +220.8%   18365705        softirqs.CPU46.NET_RX
    113460           -13.0%      98665        softirqs.CPU46.TIMER
   6084344 ±  5%    +202.7%   18417111        softirqs.CPU47.NET_RX
    112834           -13.0%      98164        softirqs.CPU47.TIMER
   6256784 ±  5%    +198.2%   18654929        softirqs.CPU48.NET_RX
    112693 ±  2%     -12.4%      98713        softirqs.CPU48.TIMER
   5994904 ±  7%    +205.1%   18293336        softirqs.CPU49.NET_RX
    112767           -12.6%      98562        softirqs.CPU49.TIMER
   7728398 ±  5%    +129.3%   17724544        softirqs.CPU5.NET_RX
   6003663 ±  8%    +207.4%   18454182        softirqs.CPU50.NET_RX
    112681           -13.0%      98072        softirqs.CPU50.TIMER
   6143237 ±  6%    +200.3%   18445995        softirqs.CPU51.NET_RX
    112722           -12.3%      98887        softirqs.CPU51.TIMER
   7663998 ± 14%    +127.7%   17447812        softirqs.CPU52.NET_RX
   7743623 ±  5%    +126.0%   17502781        softirqs.CPU53.NET_RX
   7342228 ±  9%    +140.9%   17688208        softirqs.CPU54.NET_RX
   8032766 ±  3%    +122.7%   17885003        softirqs.CPU55.NET_RX
   7736344 ±  4%    +128.7%   17693498 ±  3%  softirqs.CPU56.NET_RX
   7878109 ±  2%    +121.7%   17468805        softirqs.CPU57.NET_RX
   7822540 ±  5%    +123.3%   17470237        softirqs.CPU58.NET_RX
   7807645 ± 10%    +123.7%   17462378        softirqs.CPU59.NET_RX
   7836732 ±  4%    +122.9%   17466230        softirqs.CPU6.NET_RX
   7818018 ±  6%    +125.8%   17653948        softirqs.CPU60.NET_RX
   8147307 ±  4%    +116.6%   17649112        softirqs.CPU61.NET_RX
   7675584 ±  4%    +133.8%   17944850 ±  2%  softirqs.CPU62.NET_RX
   7819166 ±  6%    +126.2%   17683764        softirqs.CPU63.NET_RX
   7758109 ±  4%    +126.0%   17529854        softirqs.CPU64.NET_RX
   7663761 ± 12%    +130.9%   17697627        softirqs.CPU65.NET_RX
   7524645 ±  7%    +134.3%   17633068        softirqs.CPU66.NET_RX
   8050561 ±  4%    +121.7%   17851477        softirqs.CPU67.NET_RX
   7680779 ±  2%    +130.0%   17663190        softirqs.CPU68.NET_RX
   7706030 ±  2%    +128.0%   17569195        softirqs.CPU69.NET_RX
   7854752 ± 10%    +123.0%   17518515        softirqs.CPU7.NET_RX
   7749293 ±  2%    +128.2%   17686713 ±  2%  softirqs.CPU70.NET_RX
   7762287 ±  4%    +125.9%   17536102        softirqs.CPU71.NET_RX
   7430632 ± 16%    +138.0%   17683431        softirqs.CPU72.NET_RX
   7822935 ±  5%    +127.2%   17774751        softirqs.CPU73.NET_RX
   8097380 ±  4%    +118.8%   17719244        softirqs.CPU74.NET_RX
   7834736 ±  4%    +128.0%   17859974        softirqs.CPU75.NET_RX
   7834363 ±  3%    +122.2%   17407472 ±  3%  softirqs.CPU76.NET_RX
   7882104 ±  4%    +125.3%   17757705        softirqs.CPU77.NET_RX
   5808530 ± 10%    +205.1%   17724062        softirqs.CPU78.NET_RX
    113054           -12.2%      99231        softirqs.CPU78.TIMER
   5981769 ±  4%    +206.8%   18352732        softirqs.CPU79.NET_RX
    113644 ±  3%     -13.3%      98539        softirqs.CPU79.TIMER
   8021831 ±  7%    +122.5%   17851420        softirqs.CPU8.NET_RX
   5632680 ± 14%    +224.6%   18282437        softirqs.CPU80.NET_RX
    113396 ±  3%     -12.4%      99333 ±  3%  softirqs.CPU80.TIMER
   6179103 ±  6%    +192.7%   18085354 ±  4%  softirqs.CPU81.NET_RX
    112293           -10.1%     100988 ±  2%  softirqs.CPU81.TIMER
   5945340 ±  5%    +205.6%   18171651        softirqs.CPU82.NET_RX
    113059           -12.9%      98492        softirqs.CPU82.TIMER
   5935515 ±  8%    +203.7%   18025789        softirqs.CPU83.NET_RX
    112751           -11.4%      99886 ±  2%  softirqs.CPU83.TIMER
   6033617 ±  6%    +200.8%   18150908        softirqs.CPU84.NET_RX
    112657           -12.5%      98631        softirqs.CPU84.TIMER
   6082408 ±  6%    +202.2%   18383125        softirqs.CPU85.NET_RX
    112975           -13.1%      98127        softirqs.CPU85.TIMER
   6314946 ±  6%    +191.2%   18390473        softirqs.CPU86.NET_RX
    111921           -12.3%      98111        softirqs.CPU86.TIMER
   6376108 ±  6%    +186.2%   18251399        softirqs.CPU87.NET_RX
    111983           -12.2%      98357        softirqs.CPU87.TIMER
   5802245 ±  5%    +219.4%   18531405        softirqs.CPU88.NET_RX
    114244           -13.5%      98785        softirqs.CPU88.TIMER
   5839247 ±  9%    +219.9%   18680117        softirqs.CPU89.NET_RX
    113444           -13.1%      98639 ±  2%  softirqs.CPU89.TIMER
   8177186 ±  4%    +116.6%   17708425        softirqs.CPU9.NET_RX
   5901819 ±  5%    +210.9%   18350770        softirqs.CPU90.NET_RX
    112855           -13.2%      97988        softirqs.CPU90.TIMER
   6034808 ±  8%    +203.4%   18312456        softirqs.CPU91.NET_RX
    112511           -12.7%      98172        softirqs.CPU91.TIMER
   5745445 ±  9%    +220.9%   18438900        softirqs.CPU92.NET_RX
    113273 ±  2%     -13.6%      97826        softirqs.CPU92.TIMER
   6129480 ±  3%    +201.8%   18501681        softirqs.CPU93.NET_RX
    112494           -13.2%      97681        softirqs.CPU93.TIMER
   5852642 ±  6%    +214.9%   18428832        softirqs.CPU94.NET_RX
    112812           -13.5%      97562        softirqs.CPU94.TIMER
   5890746 ±  8%    +211.0%   18321360        softirqs.CPU95.NET_RX
    113647           -13.7%      98072        softirqs.CPU95.TIMER
   6039907 ±  9%    +202.8%   18289418        softirqs.CPU96.NET_RX
    112247           -12.7%      97967        softirqs.CPU96.TIMER
   5983287 ±  6%    +205.3%   18269666        softirqs.CPU97.NET_RX
    113011 ±  2%     -13.1%      98260        softirqs.CPU97.TIMER
   5694254 ± 12%    +220.7%   18264029        softirqs.CPU98.NET_RX
    114122           -14.1%      98038        softirqs.CPU98.TIMER
   6067754 ±  4%    +202.5%   18353600        softirqs.CPU99.NET_RX
    112309           -12.8%      97877        softirqs.CPU99.TIMER
 7.155e+08 ±  3%    +161.4%  1.871e+09        softirqs.NET_RX
  11540641            -8.8%   10528497        softirqs.TIMER
     46.32           -41.2        5.16 ±  6%  perf-profile.calltrace.cycles-pp.ip_idents_reserve.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg
     46.62           -40.8        5.77 ±  6%  perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg
     46.99           -40.4        6.63 ±  5%  perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     51.10           -37.1       14.04 ±  2%  perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     74.29           -24.2       50.13        perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     74.72           -23.3       51.38        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     75.42           -22.1       53.33        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     75.54           -21.9       53.66        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     95.86            -6.2       89.64        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     96.09            -5.9       90.19        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.87            -2.6        1.25 ±  2%  perf-profile.calltrace.cycles-pp.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
      5.40            -1.8        3.60        perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
      5.54            -1.7        3.84        perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb
      5.97            -0.8        5.15        perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb
      1.22            -0.7        0.53 ±  3%  perf-profile.calltrace.cycles-pp.sock_def_write_space.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
      0.59 ±  2%      +0.2        0.82 ±  9%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule.exit_to_usermode_loop.do_syscall_64
      0.53            +0.3        0.81        perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      0.80 ±  5%      +0.3        1.13 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom
      1.17 ±  3%      +0.3        1.51        perf-profile.calltrace.cycles-pp.netif_rx_internal.netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
      1.18 ±  3%      +0.4        1.53        perf-profile.calltrace.cycles-pp.netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
      0.61            +0.4        1.00        perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data
      0.00            +0.5        0.50        perf-profile.calltrace.cycles-pp.sockfd_lookup_light.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.5        0.53        perf-profile.calltrace.cycles-pp.sockfd_lookup_light.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.5        0.54 ±  2%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.reweight_entity.enqueue_task_fair.activate_task
      0.00            +0.5        0.54        perf-profile.calltrace.cycles-pp.ktime_get_with_offset.netif_rx_internal.netif_rx.loopback_xmit.dev_hard_start_xmit
      0.00            +0.5        0.55        perf-profile.calltrace.cycles-pp._copy_from_user.move_addr_to_kernel.__sys_sendto.__x64_sys_sendto.do_syscall_64
      0.00            +0.6        0.56 ±  4%  perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.57        perf-profile.calltrace.cycles-pp.siphash_3u32.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg
      0.00            +0.6        0.57        perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.udp_recvmsg.inet_recvmsg.__sys_recvfrom
      0.00            +0.6        0.59        perf-profile.calltrace.cycles-pp.__get_user_4.move_addr_to_user.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      0.00            +0.6        0.59 ±  3%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout
      0.81 ±  3%      +0.6        1.40 ±  3%  perf-profile.calltrace.cycles-pp.available_idle_cpu.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.60 ±  2%  perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.00            +0.6        0.60 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +0.6        0.61 ±  4%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule
      0.00            +0.6        0.62 ±  2%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__sched_text_start.schedule
      0.00            +0.6        0.63        perf-profile.calltrace.cycles-pp.native_write_msr
      0.00            +0.7        0.66 ±  5%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule
      0.00            +0.7        0.67 ±  4%  perf-profile.calltrace.cycles-pp.validate_xmit_skb.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb
      0.00            +0.7        0.68        perf-profile.calltrace.cycles-pp.__check_object_size.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg
      0.00            +0.7        0.69 ±  2%  perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      0.00            +0.7        0.70        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data
      0.00            +0.7        0.70 ±  2%  perf-profile.calltrace.cycles-pp.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      0.00            +0.7        0.71        perf-profile.calltrace.cycles-pp.move_addr_to_kernel.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.74        perf-profile.calltrace.cycles-pp.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
      0.00            +0.7        0.74 ±  7%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.00            +0.8        0.76        perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.77 ±  2%  perf-profile.calltrace.cycles-pp.__switch_to_asm
      0.81 ±  2%      +0.8        1.60 ±  2%  perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.80        perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      1.19 ±  3%      +0.8        2.00 ±  2%  perf-profile.calltrace.cycles-pp.ip_route_output_key_hash_rcu.ip_route_output_key_hash.ip_route_output_flow.udp_sendmsg.sock_sendmsg
      0.00            +0.8        0.81 ±  2%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.58 ±  6%      +0.8        1.40 ±  5%  perf-profile.calltrace.cycles-pp.__udp4_lib_lookup.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      0.00            +0.8        0.83        perf-profile.calltrace.cycles-pp.__switch_to
      0.00            +0.9        0.85        perf-profile.calltrace.cycles-pp._copy_from_iter_full.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg
      0.00            +0.9        0.90 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.26 ±100%      +0.9        1.16 ±  6%  perf-profile.calltrace.cycles-pp.udp4_lib_lookup2.__udp4_lib_lookup.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      0.00            +0.9        0.90 ±  2%  perf-profile.calltrace.cycles-pp.security_socket_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
      0.00            +0.9        0.92        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule.schedule_timeout
      0.80 ±  4%      +1.0        1.75 ±  4%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout
      0.94 ±  3%      +1.0        1.90        perf-profile.calltrace.cycles-pp.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg
      0.00            +1.0        0.96        perf-profile.calltrace.cycles-pp._copy_to_iter.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.00            +1.0        1.00 ±  3%  perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout
      1.29 ±  3%      +1.0        2.31 ±  2%  perf-profile.calltrace.cycles-pp.ip_route_output_key_hash.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto
      1.33 ±  4%      +1.0        2.35 ±  2%  perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      0.99 ±  3%      +1.1        2.08        perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets
      1.50 ±  3%      +1.1        2.63        perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb
      0.25 ±100%      +1.1        1.39 ±  2%  perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      0.00            +1.1        1.14        perf-profile.calltrace.cycles-pp.__check_object_size.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.00            +1.2        1.19 ±  5%  perf-profile.calltrace.cycles-pp.fib_table_lookup.ip_route_output_key_hash_rcu.ip_route_output_key_hash.ip_route_output_flow.udp_sendmsg
      0.64 ±  3%      +1.2        1.83        perf-profile.calltrace.cycles-pp.move_addr_to_user.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.2        1.23 ±  2%  perf-profile.calltrace.cycles-pp.load_new_mm_cr3.switch_mm_irqs_off.__sched_text_start.schedule.schedule_timeout
      1.59 ±  3%      +1.2        2.83        perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg
      0.76 ±  5%      +1.3        2.08        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets
      2.28 ±  3%      +1.4        3.66        perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg
      0.77 ±  3%      +1.4        2.17 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.24            +1.8        2.99        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
      1.14 ±  3%      +2.1        3.26        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      1.88 ±  2%      +2.2        4.04 ±  2%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.91 ±  2%      +2.8        4.71 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets
      3.80 ±  2%      +3.0        6.79        perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      1.86 ±  3%      +3.5        5.34 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      1.89 ±  3%      +3.5        5.42 ±  3%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      1.91 ±  3%      +3.6        5.46 ±  3%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      8.02 ±  2%      +6.6       14.63 ±  2%  perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu
      9.29 ±  2%      +6.7       15.95        perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      9.40            +6.8       16.21        perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      5.19 ±  3%      +7.0       12.18        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp
      5.31 ±  2%      +7.0       12.36 ±  2%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      5.27 ±  3%      +7.2       12.48        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
      5.40 ±  2%      +7.3       12.67 ±  2%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb
      6.52 ±  2%      +7.3       13.87 ±  2%  perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv
      5.32 ±  3%      +7.4       12.68        perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg
      5.50 ±  2%      +7.5       12.97 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb
      5.81 ±  2%      +7.6       13.40 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb
     10.24 ±  2%      +7.8       18.03        perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
      6.03 ±  3%      +7.9       13.94        perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom
     10.35 ±  2%      +8.1       18.42        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     10.43 ±  2%      +8.2       18.64        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     10.57 ±  2%      +8.3       18.91        perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
      8.13 ±  2%      +8.6       16.71        perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
     10.97 ±  2%      +8.9       19.82        perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     12.58 ±  2%      +9.1       21.67        perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
     11.44 ±  2%      +9.3       20.69        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     13.03 ±  2%      +9.5       22.48        perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     13.30 ±  2%      +9.7       23.05        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     20.09            +9.8       29.85        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg
     13.41 ±  2%      +9.9       23.33        perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     13.59 ±  2%     +10.2       23.77        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb
     13.64 ±  2%     +10.3       23.89        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb
     20.86           +10.3       31.20        perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg
     21.05           +10.6       31.66        perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     21.35           +10.9       32.23        perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      9.86 ±  2%     +11.2       21.07        perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     10.09 ±  2%     +11.5       21.62        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.46 ±  2%     +13.9       25.36        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.59 ±  2%     +14.1       25.71        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     46.36           -41.2        5.17 ±  6%  perf-profile.children.cycles-pp.ip_idents_reserve
     46.62           -40.8        5.79 ±  6%  perf-profile.children.cycles-pp.__ip_select_ident
     47.00           -40.4        6.64 ±  5%  perf-profile.children.cycles-pp.__ip_make_skb
     51.11           -37.0       14.08 ±  2%  perf-profile.children.cycles-pp.ip_make_skb
     74.31           -24.1       50.16        perf-profile.children.cycles-pp.udp_sendmsg
     74.73           -23.3       51.39        perf-profile.children.cycles-pp.sock_sendmsg
     75.43           -22.1       53.35        perf-profile.children.cycles-pp.__sys_sendto
     75.55           -21.9       53.69        perf-profile.children.cycles-pp.__x64_sys_sendto
     95.94            -6.2       89.72        perf-profile.children.cycles-pp.do_syscall_64
     96.15            -5.9       90.25        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.87            -2.6        1.25 ±  2%  perf-profile.children.cycles-pp.sock_wfree
      5.40            -1.8        3.62        perf-profile.children.cycles-pp.loopback_xmit
      5.54            -1.7        3.85        perf-profile.children.cycles-pp.dev_hard_start_xmit
      3.13            -1.0        2.17        perf-profile.children.cycles-pp._raw_spin_lock
      5.98            -0.8        5.18        perf-profile.children.cycles-pp.__dev_queue_xmit
      1.22            -0.7        0.53 ±  3%  perf-profile.children.cycles-pp.sock_def_write_space
      0.52 ±  4%      -0.2        0.29        perf-profile.children.cycles-pp.dst_release
      0.67 ±  3%      -0.2        0.49 ±  2%  perf-profile.children.cycles-pp.ipv4_pktinfo_prepare
      0.18 ±  6%      -0.1        0.07 ± 12%  perf-profile.children.cycles-pp.resched_curr
      0.45 ±  2%      -0.1        0.39 ±  2%  perf-profile.children.cycles-pp.udp_rmem_release
      0.46 ±  3%      -0.0        0.43        perf-profile.children.cycles-pp.skb_set_owner_w
      0.06 ±  7%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.skb_pull_rcsum
      0.14 ± 13%      +0.0        0.18 ±  5%  perf-profile.children.cycles-pp.update_process_times
      0.17 ± 11%      +0.0        0.21 ±  3%  perf-profile.children.cycles-pp.tick_sched_timer
      0.14 ± 13%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.tick_sched_handle
      0.13 ±  9%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.apparmor_ipv4_postroute
      0.05            +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.udp4_hwcsum
      0.06 ±  9%      +0.0        0.10        perf-profile.children.cycles-pp.__skb_try_recv_from_queue
      0.08 ±  5%      +0.0        0.13 ±  5%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.__usecs_to_jiffies
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.__bitmap_and
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.rcu_note_context_switch
      0.21 ±  5%      +0.1        0.26 ±  3%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.08 ±  5%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.skb_release_data
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.apparmor_socket_recvmsg
      0.11 ±  9%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb
      0.23 ±  9%      +0.1        0.29 ±  5%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.54 ±  8%      +0.1        0.60 ±  2%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.__fdget
      0.06 ± 15%      +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.21 ±  2%      +0.1        0.27        perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.11 ±  4%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.kfree_skbmem
      0.08 ±  5%      +0.1        0.15 ±  5%  perf-profile.children.cycles-pp.__kfree_skb_flush
      0.06 ±  7%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.kmalloc_slab
      0.01 ±173%      +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.udp_queue_rcv_skb
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.skb_csum_hwoffload_help
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.is_cpu_allowed
      0.11 ±  3%      +0.1        0.18 ±  7%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.skb_clone_tx_timestamp
      0.08 ±  6%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.skb_consume_udp
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.apparmor_ip_postroute
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.eth_type_trans
      0.07 ±  7%      +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.xfrm_lookup_with_ifid
      0.01 ±173%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.default_wake_function
      0.08 ±  8%      +0.1        0.16 ±  2%  perf-profile.children.cycles-pp.rb_erase
      0.04 ± 57%      +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.rcu_all_qs
      0.06 ±  6%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.iov_iter_advance
      0.47 ±  7%      +0.1        0.55 ±  2%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.13            +0.1        0.21 ±  2%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.07 ±  6%      +0.1        0.15 ±  2%  perf-profile.children.cycles-pp.ip_send_check
      0.07 ±  5%      +0.1        0.17 ±  9%  perf-profile.children.cycles-pp.xfrm_lookup_route
      0.42 ±  8%      +0.1        0.52 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.06            +0.1        0.15 ±  3%  perf-profile.children.cycles-pp.__enqueue_entity
      0.00            +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.deactivate_task
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.skb_network_protocol
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.switch_ldt
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.netdev_core_pick_tx
      0.00            +0.1        0.10 ±  7%  perf-profile.children.cycles-pp.__raw_spin_unlock
      0.27 ±  3%      +0.1        0.38 ±  4%  perf-profile.children.cycles-pp.put_prev_entity
      0.08 ±  5%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.wakeup_preempt_entity
      0.11 ±  4%      +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.03 ±100%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.__raise_softirq_irqoff
      0.07 ±  6%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp.apparmor_socket_sendmsg
      0.27 ±  5%      +0.1        0.38        perf-profile.children.cycles-pp.nf_hook_slow
      0.06 ±  6%      +0.1        0.18 ±  3%  perf-profile.children.cycles-pp.native_load_tls
      0.03 ±100%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.should_failslab
      0.09 ±  4%      +0.1        0.22        perf-profile.children.cycles-pp.check_stack_object
      0.01 ±173%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.receiver_wake_function
      0.08 ±  5%      +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.ksoftirqd_running
      0.15            +0.1        0.29        perf-profile.children.cycles-pp.kmem_cache_free
      0.09 ±  7%      +0.1        0.23        perf-profile.children.cycles-pp._cond_resched
      0.00            +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.inet_send_prepare
      0.00            +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.raw_local_deliver
      0.07 ± 26%      +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.cpumask_next
      0.24            +0.1        0.39        perf-profile.children.cycles-pp.ip_rcv_finish
      0.10 ±  4%      +0.2        0.25 ± 10%  perf-profile.children.cycles-pp.cpuacct_charge
      0.16 ±  2%      +0.2        0.32 ±  2%  perf-profile.children.cycles-pp.set_next_buddy
      0.11 ±  4%      +0.2        0.28        perf-profile.children.cycles-pp.__ip_local_out
      0.10 ±  7%      +0.2        0.26 ±  3%  perf-profile.children.cycles-pp.import_single_range
      0.27 ±  4%      +0.2        0.44 ±  3%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.08 ±  5%      +0.2        0.26 ±  4%  perf-profile.children.cycles-pp.inet_sendmsg
      0.12 ±  6%      +0.2        0.30 ±  2%  perf-profile.children.cycles-pp.__ksize
      0.11 ±  6%      +0.2        0.29 ±  3%  perf-profile.children.cycles-pp.clear_buddies
      0.16 ±  2%      +0.2        0.36        perf-profile.children.cycles-pp.read_tsc
      0.15 ±  5%      +0.2        0.36        perf-profile.children.cycles-pp.kfree
      0.26 ±  8%      +0.2        0.47        perf-profile.children.cycles-pp.__ip_finish_output
      0.17 ±  4%      +0.2        0.39        perf-profile.children.cycles-pp.copyin
      0.11 ±  4%      +0.2        0.34 ±  8%  perf-profile.children.cycles-pp.netif_skb_features
      0.15 ±  3%      +0.2        0.38        perf-profile.children.cycles-pp.ip_local_out
      0.10 ±  7%      +0.2        0.34 ±  2%  perf-profile.children.cycles-pp.account_entity_enqueue
      0.18 ±  7%      +0.3        0.44        perf-profile.children.cycles-pp.ip_setup_cork
      0.28 ±  5%      +0.3        0.54        perf-profile.children.cycles-pp.ktime_get_with_offset
      0.24 ±  6%      +0.3        0.51        perf-profile.children.cycles-pp.__check_heap_object
      0.18 ±  4%      +0.3        0.45 ±  4%  perf-profile.children.cycles-pp.update_min_vruntime
      0.15 ±  4%      +0.3        0.42        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.11 ±  4%      +0.3        0.38        perf-profile.children.cycles-pp.ip_rcv_core
      0.23 ±  4%      +0.3        0.51        perf-profile.children.cycles-pp.__consume_stateless_skb
      0.20 ±  2%      +0.3        0.48        perf-profile.children.cycles-pp.finish_task_switch
      0.15 ± 14%      +0.3        0.45 ±  3%  perf-profile.children.cycles-pp._find_next_bit
      0.18 ±  5%      +0.3        0.50        perf-profile.children.cycles-pp._copy_to_user
      0.55            +0.3        0.87        perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.39 ±  2%      +0.3        0.72 ±  2%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.81 ±  5%      +0.3        1.14 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.41 ±  3%      +0.3        0.75        perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.15 ±  4%      +0.3        0.49        perf-profile.children.cycles-pp.native_sched_clock
      1.18 ±  3%      +0.3        1.52        perf-profile.children.cycles-pp.netif_rx_internal
      0.24 ±  3%      +0.3        0.58        perf-profile.children.cycles-pp.siphash_3u32
      0.40 ±  6%      +0.4        0.76        perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.20 ±  4%      +0.4        0.55 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.15 ±  2%      +0.4        0.51        perf-profile.children.cycles-pp.sched_clock
      0.20 ±  6%      +0.4        0.56        perf-profile.children.cycles-pp._copy_from_user
      1.18 ±  3%      +0.4        1.54        perf-profile.children.cycles-pp.netif_rx
      0.35            +0.4        0.72        perf-profile.children.cycles-pp.native_write_msr
      0.20 ±  4%      +0.4        0.57 ±  2%  perf-profile.children.cycles-pp.___perf_sw_event
      0.21 ±  4%      +0.4        0.58        perf-profile.children.cycles-pp.copyout
      0.20 ±  3%      +0.4        0.57 ±  3%  perf-profile.children.cycles-pp.cpumask_next_wrap
      0.16 ±  4%      +0.4        0.54 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.33 ±  4%      +0.4        0.71        perf-profile.children.cycles-pp.__virt_addr_valid
      0.17 ±  4%      +0.4        0.56        perf-profile.children.cycles-pp.sched_clock_cpu
      0.42            +0.4        0.81        perf-profile.children.cycles-pp.check_preempt_curr
      0.62            +0.4        1.02        perf-profile.children.cycles-pp.__kmalloc_reserve
      0.30 ±  3%      +0.4        0.71 ±  2%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.32 ±  2%      +0.4        0.74 ±  2%  perf-profile.children.cycles-pp.pick_next_entity
      0.20 ±  3%      +0.4        0.63        perf-profile.children.cycles-pp.__get_user_4
      0.47 ±  2%      +0.4        0.91 ±  2%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.32 ±  3%      +0.4        0.77 ±  2%  perf-profile.children.cycles-pp.sock_recvmsg
      0.27 ±  3%      +0.4        0.72        perf-profile.children.cycles-pp.move_addr_to_kernel
      0.76            +0.5        1.22        perf-profile.children.cycles-pp.__switch_to
      0.23 ±  2%      +0.5        0.69 ±  4%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.34            +0.5        0.81 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.42 ±  3%      +0.5        0.89        perf-profile.children.cycles-pp._copy_from_iter_full
      0.32 ±  5%      +0.5        0.80 ±  3%  perf-profile.children.cycles-pp.account_entity_dequeue
      0.31 ±  4%      +0.5        0.79 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.28 ±  2%      +0.5        0.76        perf-profile.children.cycles-pp.ipv4_mtu
      0.29 ±  4%      +0.5        0.78        perf-profile.children.cycles-pp.___might_sleep
      0.28 ±  3%      +0.5        0.78        perf-profile.children.cycles-pp.update_rq_clock
      0.39            +0.6        0.97        perf-profile.children.cycles-pp._copy_to_iter
      0.32 ±  3%      +0.6        0.90        perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.31 ±  7%      +0.6        0.90 ±  2%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.28 ±  4%      +0.6        0.87        perf-profile.children.cycles-pp.__fget_light
      0.81 ±  3%      +0.6        1.40 ±  3%  perf-profile.children.cycles-pp.available_idle_cpu
      0.37 ±  5%      +0.6        0.98 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      0.35 ±  5%      +0.6        0.96 ±  3%  perf-profile.children.cycles-pp.update_cfs_group
      0.38 ±  4%      +0.6        1.00        perf-profile.children.cycles-pp.__might_fault
      0.40 ±  3%      +0.6        1.03 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.36 ±  3%      +0.7        1.01        perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.52 ±  4%      +0.7        1.18        perf-profile.children.cycles-pp.set_next_entity
      0.48 ±  6%      +0.7        1.16 ±  6%  perf-profile.children.cycles-pp.udp4_lib_lookup2
      0.43 ±  4%      +0.7        1.13        perf-profile.children.cycles-pp.aa_sk_perm
      0.34 ±  4%      +0.7        1.06        perf-profile.children.cycles-pp.sockfd_lookup_light
      0.44 ±  2%      +0.8        1.19 ±  5%  perf-profile.children.cycles-pp.fib_table_lookup
      0.81 ±  2%      +0.8        1.60 ±  2%  perf-profile.children.cycles-pp.switch_fpu_return
      1.20 ±  4%      +0.8        2.02 ±  2%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      0.59 ±  6%      +0.8        1.42 ±  5%  perf-profile.children.cycles-pp.__udp4_lib_lookup
      0.94 ±  3%      +1.0        1.90        perf-profile.children.cycles-pp.ip_generic_getfrag
      0.81 ±  4%      +1.0        1.79 ±  4%  perf-profile.children.cycles-pp.dequeue_entity
      1.65 ±  2%      +1.0        2.65        perf-profile.children.cycles-pp.switch_mm_irqs_off
      1.30 ±  3%      +1.0        2.32 ±  2%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      1.33 ±  4%      +1.0        2.36 ±  2%  perf-profile.children.cycles-pp.ip_route_output_flow
      0.92 ±  3%      +1.1        2.04        perf-profile.children.cycles-pp.__check_object_size
      1.51 ±  3%      +1.1        2.65        perf-profile.children.cycles-pp.__alloc_skb
      0.67            +1.2        1.86        perf-profile.children.cycles-pp.load_new_mm_cr3
      0.65 ±  3%      +1.2        1.85        perf-profile.children.cycles-pp.move_addr_to_user
      1.60 ±  3%      +1.2        2.85        perf-profile.children.cycles-pp.alloc_skb_with_frags
      2.28 ±  3%      +1.4        3.66        perf-profile.children.cycles-pp.sock_alloc_send_pskb
      0.78 ±  3%      +1.4        2.20 ±  4%  perf-profile.children.cycles-pp.enqueue_entity
      0.88 ±  2%      +1.5        2.41 ±  2%  perf-profile.children.cycles-pp.reweight_entity
      1.41 ±  2%      +1.6        3.03        perf-profile.children.cycles-pp.pick_next_task_fair
      1.13 ±  2%      +1.7        2.83        perf-profile.children.cycles-pp.update_curr
      1.24            +1.8        3.00        perf-profile.children.cycles-pp.entry_SYSCALL_64
      1.33            +1.9        3.24 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      1.90 ±  2%      +2.2        4.09 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      1.29 ±  3%      +2.4        3.67        perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.92 ±  2%      +2.8        4.73 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      3.80 ±  2%      +3.0        6.82        perf-profile.children.cycles-pp.__ip_append_data
      1.87 ±  3%      +3.5        5.35 ±  3%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.90 ±  3%      +3.5        5.43 ±  3%  perf-profile.children.cycles-pp.activate_task
      1.91 ±  3%      +3.6        5.46 ±  3%  perf-profile.children.cycles-pp.ttwu_do_activate
      8.02 ±  2%      +6.6       14.64 ±  2%  perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
      9.30 ±  2%      +6.7       16.00        perf-profile.children.cycles-pp.udp_queue_rcv_one_skb
      9.40            +6.8       16.22        perf-profile.children.cycles-pp.udp_unicast_rcv_skb
      5.32 ±  2%      +7.1       12.39 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
      7.25            +7.2       14.42        perf-profile.children.cycles-pp.__sched_text_start
      5.40 ±  2%      +7.3       12.68 ±  2%  perf-profile.children.cycles-pp.autoremove_wake_function
      7.37            +7.3       14.70        perf-profile.children.cycles-pp.schedule
      6.52 ±  2%      +7.4       13.88 ±  2%  perf-profile.children.cycles-pp.sock_def_readable
      5.33 ±  3%      +7.4       12.69        perf-profile.children.cycles-pp.schedule_timeout
      5.50 ±  2%      +7.5       12.97 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      5.82 ±  2%      +7.6       13.42 ±  2%  perf-profile.children.cycles-pp.__wake_up_common_lock
     10.25 ±  2%      +7.8       18.04        perf-profile.children.cycles-pp.__udp4_lib_rcv
      6.03 ±  3%      +7.9       13.96        perf-profile.children.cycles-pp.__skb_wait_for_more_packets
     10.36 ±  2%      +8.1       18.45        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     10.43 ±  2%      +8.2       18.65        perf-profile.children.cycles-pp.ip_local_deliver_finish
     10.59 ±  2%      +8.4       18.95        perf-profile.children.cycles-pp.ip_local_deliver
      8.14 ±  2%      +8.6       16.76        perf-profile.children.cycles-pp.__skb_recv_udp
     10.97 ±  2%      +8.9       19.85        perf-profile.children.cycles-pp.ip_rcv
     12.59 ±  2%      +9.1       21.68        perf-profile.children.cycles-pp.process_backlog
     11.44 ±  2%      +9.3       20.70        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     13.05 ±  2%      +9.5       22.51        perf-profile.children.cycles-pp.net_rx_action
     13.34 ±  2%      +9.8       23.09        perf-profile.children.cycles-pp.__softirqentry_text_start
     20.11            +9.8       29.91        perf-profile.children.cycles-pp.ip_finish_output2
     13.42 ±  2%     +10.0       23.37        perf-profile.children.cycles-pp.do_softirq_own_stack
     13.61 ±  2%     +10.2       23.82        perf-profile.children.cycles-pp.do_softirq
     13.70 ±  2%     +10.3       24.05        perf-profile.children.cycles-pp.__local_bh_enable_ip
     20.86           +10.4       31.22        perf-profile.children.cycles-pp.ip_output
     21.06           +10.6       31.67        perf-profile.children.cycles-pp.ip_send_skb
     21.35           +10.9       32.25        perf-profile.children.cycles-pp.udp_send_skb
      9.86 ±  2%     +11.2       21.08        perf-profile.children.cycles-pp.udp_recvmsg
     10.09 ±  2%     +11.5       21.63        perf-profile.children.cycles-pp.inet_recvmsg
     11.46 ±  2%     +13.9       25.39        perf-profile.children.cycles-pp.__sys_recvfrom
     11.60 ±  2%     +14.1       25.75        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     46.11           -41.0        5.12 ±  6%  perf-profile.self.cycles-pp.ip_idents_reserve
      2.64            -1.9        0.69 ±  4%  perf-profile.self.cycles-pp.sock_wfree
      3.12            -1.0        2.12        perf-profile.self.cycles-pp._raw_spin_lock
      1.20            -0.7        0.51 ±  3%  perf-profile.self.cycles-pp.sock_def_write_space
      0.90 ±  2%      -0.3        0.55 ±  4%  perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb
      0.70 ±  3%      -0.3        0.45        perf-profile.self.cycles-pp.sock_def_readable
      0.50 ±  4%      -0.2        0.27        perf-profile.self.cycles-pp.dst_release
      0.99 ±  4%      -0.2        0.80        perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.18 ±  4%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.resched_curr
      0.44 ±  2%      -0.1        0.38 ±  2%  perf-profile.self.cycles-pp.udp_rmem_release
      0.70 ±  3%      -0.0        0.66        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.46 ±  2%      -0.0        0.43        perf-profile.self.cycles-pp.skb_set_owner_w
      0.07 ±  5%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.copyin
      0.12 ±  6%      +0.0        0.15 ±  2%  perf-profile.self.cycles-pp.apparmor_ipv4_postroute
      0.06 ±  9%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.skb_pull_rcsum
      0.11 ±  4%      +0.0        0.15 ±  2%  perf-profile.self.cycles-pp.nf_hook_slow
      0.11 ±  6%      +0.0        0.15        perf-profile.self.cycles-pp.put_prev_entity
      0.05 ±  8%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.__skb_try_recv_from_queue
      0.08 ± 10%      +0.0        0.12 ±  5%  perf-profile.self.cycles-pp.rb_next
      0.11 ±  7%      +0.0        0.15 ±  2%  perf-profile.self.cycles-pp.apparmor_socket_sock_rcv_skb
      0.13 ±  6%      +0.0        0.18 ±  3%  perf-profile.self.cycles-pp.ktime_get_with_offset
      0.05            +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.netif_rx_internal
      0.12 ±  3%      +0.0        0.17 ±  4%  perf-profile.self.cycles-pp._copy_from_iter_full
      0.05 ±  8%      +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.ttwu_do_wakeup
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__usecs_to_jiffies
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__bitmap_and
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.sock_recvmsg
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.apparmor_ip_postroute
      0.08 ±  5%      +0.1        0.13 ±  3%  perf-profile.self.cycles-pp.skb_release_data
      0.20 ±  3%      +0.1        0.26        perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.skb_csum_hwoffload_help
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__ip_select_ident
      0.06 ±  9%      +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.06 ±  9%      +0.1        0.11 ±  7%  perf-profile.self.cycles-pp.kmalloc_slab
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.skb_clone_tx_timestamp
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.apparmor_socket_recvmsg
      0.17 ±  2%      +0.1        0.24 ±  3%  perf-profile.self.cycles-pp.ipv4_pktinfo_prepare
      0.17 ± 12%      +0.1        0.23 ±  6%  perf-profile.self.cycles-pp.__ip_finish_output
      0.08 ± 10%      +0.1        0.14 ±  3%  perf-profile.self.cycles-pp.rb_erase
      0.08 ± 11%      +0.1        0.14        perf-profile.self.cycles-pp.ip_setup_cork
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.is_cpu_allowed
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.kfree_skbmem
      0.09 ±  4%      +0.1        0.16 ±  6%  perf-profile.self.cycles-pp.ktime_get
      0.01 ±173%      +0.1        0.08        perf-profile.self.cycles-pp.default_wake_function
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.activate_task
      0.08 ±  6%      +0.1        0.15 ± 15%  perf-profile.self.cycles-pp.skb_consume_udp
      0.06 ±  6%      +0.1        0.13 ±  9%  perf-profile.self.cycles-pp.xfrm_lookup_with_ifid
      0.06 ±  6%      +0.1        0.13 ±  3%  perf-profile.self.cycles-pp.iov_iter_advance
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.sock_sendmsg
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.__wake_up_common_lock
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.eth_type_trans
      0.14 ±  3%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.ip_make_skb
      0.01 ±173%      +0.1        0.09        perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.13            +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.skb_network_protocol
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.udp_queue_rcv_skb
      0.07 ± 12%      +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.udp_unicast_rcv_skb
      0.04 ± 57%      +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__ip_local_out
      0.00            +0.1        0.08        perf-profile.self.cycles-pp._copy_from_user
      0.00            +0.1        0.08        perf-profile.self.cycles-pp._copy_to_user
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.should_failslab
      0.07 ±  6%      +0.1        0.15        perf-profile.self.cycles-pp.ip_send_check
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.move_addr_to_kernel
      0.06 ±  6%      +0.1        0.15 ±  3%  perf-profile.self.cycles-pp.__kfree_skb_flush
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.netdev_core_pick_tx
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.check_preempt_curr
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__raw_spin_unlock
      0.06 ±  6%      +0.1        0.15 ±  4%  perf-profile.self.cycles-pp.__kmalloc_reserve
      0.06            +0.1        0.15 ±  2%  perf-profile.self.cycles-pp.__enqueue_entity
      0.03 ±100%      +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.inet_sendmsg
      0.00            +0.1        0.09        perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.udp4_hwcsum
      0.00            +0.1        0.09 ± 13%  perf-profile.self.cycles-pp.deactivate_task
      0.29            +0.1        0.38        perf-profile.self.cycles-pp.udp_queue_rcv_one_skb
      0.14 ±  6%      +0.1        0.23        perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.41 ±  2%      +0.1        0.51 ±  3%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.switch_ldt
      0.10 ±  4%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.04 ± 57%      +0.1        0.14        perf-profile.self.cycles-pp.sockfd_lookup_light
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.ip_local_out
      0.11 ±  4%      +0.1        0.21 ±  3%  perf-profile.self.cycles-pp._copy_to_iter
      0.08 ±  6%      +0.1        0.18        perf-profile.self.cycles-pp.check_stack_object
      0.08 ±  8%      +0.1        0.18 ±  4%  perf-profile.self.cycles-pp.wakeup_preempt_entity
      0.07 ±  7%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.apparmor_socket_sendmsg
      0.06            +0.1        0.17 ±  2%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.06 ± 11%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.11 ±  4%  perf-profile.self.cycles-pp._cond_resched
      0.05 ±  8%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.__wake_up_common
      0.06 ±  6%      +0.1        0.18 ±  3%  perf-profile.self.cycles-pp.native_load_tls
      0.00            +0.1        0.12 ±  3%  perf-profile.self.cycles-pp.ip_rcv_finish
      0.01 ±173%      +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.__raise_softirq_irqoff
      0.08 ± 10%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.alloc_skb_with_frags
      0.03 ±100%      +0.1        0.14 ±  3%  perf-profile.self.cycles-pp.ip_rcv
      0.00            +0.1        0.12 ±  6%  perf-profile.self.cycles-pp.receiver_wake_function
      0.07 ±  6%      +0.1        0.19 ±  2%  perf-profile.self.cycles-pp.dequeue_entity
      0.00            +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.raw_local_deliver
      0.07 ±  5%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.ksoftirqd_running
      0.15 ±  2%      +0.1        0.28        perf-profile.self.cycles-pp.kmem_cache_free
      0.11 ±  4%      +0.1        0.25        perf-profile.self.cycles-pp.do_softirq
      0.40 ±  3%      +0.1        0.54 ±  3%  perf-profile.self.cycles-pp.process_backlog
      0.10 ±  4%      +0.1        0.23 ±  3%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.00            +0.1        0.14 ±  3%  perf-profile.self.cycles-pp.__list_add_valid
      0.06 ±  6%      +0.1        0.20 ±  6%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.00            +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.inet_send_prepare
      0.06 ± 11%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.schedule_timeout
      0.00            +0.1        0.15 ±  3%  perf-profile.self.cycles-pp.autoremove_wake_function
      0.30 ±  7%      +0.1        0.45        perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.16 ±  2%      +0.1        0.31        perf-profile.self.cycles-pp.ip_local_deliver
      0.16 ±  2%      +0.1        0.31 ±  2%  perf-profile.self.cycles-pp.set_next_buddy
      0.09 ±  8%      +0.2        0.24 ±  4%  perf-profile.self.cycles-pp.clear_buddies
      0.25 ±  3%      +0.2        0.41        perf-profile.self.cycles-pp.__udp4_lib_rcv
      0.10 ±  5%      +0.2        0.25 ± 10%  perf-profile.self.cycles-pp.cpuacct_charge
      0.10 ±  4%      +0.2        0.26 ±  3%  perf-profile.self.cycles-pp.__udp4_lib_lookup
      0.16 ±  6%      +0.2        0.33 ±  3%  perf-profile.self.cycles-pp.set_next_entity
      0.23 ±  6%      +0.2        0.39        perf-profile.self.cycles-pp.sock_alloc_send_pskb
      0.12 ±  9%      +0.2        0.28 ±  2%  perf-profile.self.cycles-pp.schedule
      0.10 ±  4%      +0.2        0.26 ±  3%  perf-profile.self.cycles-pp.import_single_range
      0.10 ±  5%      +0.2        0.26 ±  3%  perf-profile.self.cycles-pp.cpumask_next_wrap
      0.10 ±  7%      +0.2        0.28        perf-profile.self.cycles-pp.ip_generic_getfrag
      0.10 ±  5%      +0.2        0.27 ±  2%  perf-profile.self.cycles-pp.__might_fault
      0.14 ±  5%      +0.2        0.32 ±  2%  perf-profile.self.cycles-pp.finish_task_switch
      0.12 ±  3%      +0.2        0.30 ±  3%  perf-profile.self.cycles-pp.__ksize
      0.08            +0.2        0.27 ±  9%  perf-profile.self.cycles-pp.netif_skb_features
      0.09 ±  4%      +0.2        0.28 ±  3%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.15 ±  3%      +0.2        0.35 ±  2%  perf-profile.self.cycles-pp.read_tsc
      0.10            +0.2        0.30        perf-profile.self.cycles-pp.ip_route_output_key_hash
      0.07            +0.2        0.27 ±  2%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.12 ±  3%      +0.2        0.32 ±  2%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.32 ±  5%      +0.2        0.52 ±  2%  perf-profile.self.cycles-pp.__alloc_skb
      0.15 ±  3%      +0.2        0.36 ±  2%  perf-profile.self.cycles-pp.pick_next_entity
      0.15 ±  7%      +0.2        0.36 ±  2%  perf-profile.self.cycles-pp.kfree
      0.12 ±  5%      +0.2        0.33 ±  2%  perf-profile.self.cycles-pp.do_softirq_own_stack
      0.16 ±  6%      +0.2        0.38 ±  3%  perf-profile.self.cycles-pp.update_rq_clock
      0.14 ±  5%      +0.2        0.36        perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.10 ±  5%      +0.2        0.32 ±  2%  perf-profile.self.cycles-pp.account_entity_enqueue
      0.14 ±  7%      +0.2        0.36 ±  3%  perf-profile.self.cycles-pp.try_to_wake_up
      0.22 ±  5%      +0.2        0.45        perf-profile.self.cycles-pp.ip_output
      0.25 ±  7%      +0.2        0.49 ±  7%  perf-profile.self.cycles-pp.udp_send_skb
      0.25 ±  6%      +0.2        0.49        perf-profile.self.cycles-pp.enqueue_to_backlog
      0.30 ±  3%      +0.2        0.55        perf-profile.self.cycles-pp.net_rx_action
      0.18 ±  4%      +0.3        0.44 ±  5%  perf-profile.self.cycles-pp.update_min_vruntime
      0.24 ±  5%      +0.3        0.50        perf-profile.self.cycles-pp.__check_heap_object
      0.14 ±  8%      +0.3        0.40        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.10 ±  4%      +0.3        0.37        perf-profile.self.cycles-pp.ip_rcv_core
      0.24 ±  3%      +0.3        0.51 ±  2%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.13 ±  6%      +0.3        0.41 ±  3%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.16 ±  2%      +0.3        0.45        perf-profile.self.cycles-pp.move_addr_to_user
      0.14 ±  3%      +0.3        0.42 ±  2%  perf-profile.self.cycles-pp.reweight_entity
      0.17 ±  3%      +0.3        0.46 ±  5%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.15 ± 14%      +0.3        0.45 ±  3%  perf-profile.self.cycles-pp._find_next_bit
      0.20 ±  4%      +0.3        0.49        perf-profile.self.cycles-pp.pick_next_task_fair
      0.26 ±  3%      +0.3        0.56 ±  4%  perf-profile.self.cycles-pp.__ip_make_skb
      0.23 ±  6%      +0.3        0.54 ± 10%  perf-profile.self.cycles-pp.inet_recvmsg
      0.18 ±  4%      +0.3        0.49 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      0.21 ±  2%      +0.3        0.53        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.80 ±  5%      +0.3        1.12 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.13 ±  6%      +0.3        0.45 ±  2%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.15 ±  5%      +0.3        0.48        perf-profile.self.cycles-pp.native_sched_clock
      0.41 ±  5%      +0.3        0.74        perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.17 ±  5%      +0.3        0.51        perf-profile.self.cycles-pp.___perf_sw_event
      0.49 ±  4%      +0.3        0.83 ±  3%  perf-profile.self.cycles-pp.ip_finish_output2
      0.32 ±  3%      +0.3        0.66        perf-profile.self.cycles-pp.__virt_addr_valid
      0.23 ±  4%      +0.3        0.57        perf-profile.self.cycles-pp.siphash_3u32
      0.17 ±  4%      +0.4        0.52 ± 10%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.35            +0.4        0.71        perf-profile.self.cycles-pp.native_write_msr
      0.18 ±  8%      +0.4        0.54 ±  9%  perf-profile.self.cycles-pp.enqueue_entity
      0.23 ±  4%      +0.4        0.60 ±  2%  perf-profile.self.cycles-pp.__sys_sendto
      0.29 ±  5%      +0.4        0.67 ±  2%  perf-profile.self.cycles-pp.loopback_xmit
      0.28 ±  4%      +0.4        0.68 ±  2%  perf-profile.self.cycles-pp.__check_object_size
      0.19 ±  7%      +0.4        0.59 ±  2%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.34 ±  4%      +0.4        0.75 ±  2%  perf-profile.self.cycles-pp.__skb_recv_udp
      0.73            +0.4        1.15        perf-profile.self.cycles-pp.__switch_to
      0.20 ±  4%      +0.4        0.62        perf-profile.self.cycles-pp.__get_user_4
      0.25 ±  2%      +0.4        0.69        perf-profile.self.cycles-pp.ipv4_mtu
      0.30 ±  4%      +0.5        0.75 ±  3%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.33 ±  2%      +0.5        0.79 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.17 ±  8%      +0.5        0.64 ±  2%  perf-profile.self.cycles-pp.__skb_wait_for_more_packets
      0.28 ±  4%      +0.5        0.74        perf-profile.self.cycles-pp.___might_sleep
      0.30 ±  2%      +0.5        0.78 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      5.62            +0.5        6.12        perf-profile.self.cycles-pp.do_syscall_64
      0.26 ±  2%      +0.5        0.77 ±  5%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.30 ±  6%      +0.5        0.82 ±  2%  perf-profile.self.cycles-pp.aa_sk_perm
      0.29 ±  4%      +0.5        0.83 ±  2%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.79 ±  3%      +0.6        1.36 ±  3%  perf-profile.self.cycles-pp.available_idle_cpu
      0.27 ±  3%      +0.6        0.86        perf-profile.self.cycles-pp.__fget_light
      0.36 ±  5%      +0.6        0.96 ±  2%  perf-profile.self.cycles-pp.__calc_delta
      0.34 ±  4%      +0.6        0.95 ±  3%  perf-profile.self.cycles-pp.update_cfs_group
      0.39 ±  3%      +0.6        1.01 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.34 ±  3%      +0.6        0.96        perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.56 ±  2%      +0.6        1.19        perf-profile.self.cycles-pp.__ip_append_data
      0.48 ±  6%      +0.7        1.14 ±  6%  perf-profile.self.cycles-pp.udp4_lib_lookup2
      0.46 ±  2%      +0.7        1.19 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.44 ±  3%      +0.7        1.18 ±  5%  perf-profile.self.cycles-pp.fib_table_lookup
      0.87            +0.8        1.62 ±  2%  perf-profile.self.cycles-pp.__sched_text_start
      0.40 ±  4%      +0.8        1.18 ±  3%  perf-profile.self.cycles-pp.udp_recvmsg
      0.80 ±  2%      +0.8        1.59 ±  2%  perf-profile.self.cycles-pp.switch_fpu_return
      0.60            +0.8        1.40 ±  3%  perf-profile.self.cycles-pp.update_load_avg
      0.41 ±  6%      +0.9        1.27 ±  2%  perf-profile.self.cycles-pp.udp_sendmsg
      0.77 ±  2%      +1.0        1.72 ±  2%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.66            +1.2        1.85        perf-profile.self.cycles-pp.load_new_mm_cr3
      1.24            +1.8        3.00        perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.29 ±  3%      +2.4        3.65        perf-profile.self.cycles-pp.syscall_return_via_sysret


                                                                                
                               netperf.Throughput_tps                           
                                                                                
  32000 +-------------------------------------------------------------------+   
  30000 |-+O  O  O O  O  O  O  O  O O  O  O  O  O  O O  O  O  O  O  O O     |   
        |                                                                   |   
  28000 |-+                                                                 |   
  26000 |-+                                                                 |   
  24000 |-+                                                                 |   
  22000 |-+                                                                 |   
        |                                                                   |   
  20000 |-+                                                                 |   
  18000 |-+                                                                 |   
  16000 |-+                                                                 |   
  14000 |-+                                                                 |   
        |..  .+.. .+..                                    .+..              |   
  12000 |-++.    +    +..+..+..+..+.+..+..+..+..+..+.  .+.    +..+..+.+..+..|   
  10000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             netperf.Throughput_total_tps                       
                                                                                
  6.5e+06 +-----------------------------------------------------------------+   
          |  O  O O  O  O  O O  O  O  O O  O  O  O O  O  O  O O  O  O  O    |   
    6e+06 |-+                                                               |   
  5.5e+06 |-+                                                               |   
          |                                                                 |   
    5e+06 |-+                                                               |   
  4.5e+06 |-+                                                               |   
          |                                                                 |   
    4e+06 |-+                                                               |   
  3.5e+06 |-+                                                               |   
          |                                                                 |   
    3e+06 |-+                                                               |   
  2.5e+06 |..+..+.  .+..+.. .+..  .+.. .+..+..+..+.        .+.             .|   
          |       +.       +    +.    +            +..+..+.   +..+..+..+.+. |   
    2e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   netperf.workload                             
                                                                                
    2e+09 +-----------------------------------------------------------------+   
          |  O  O O  O  O  O O  O  O  O O  O  O  O O  O  O  O O  O  O  O    |   
  1.8e+09 |-+                                                               |   
          |                                                                 |   
  1.6e+09 |-+                                                               |   
          |                                                                 |   
  1.4e+09 |-+                                                               |   
          |                                                                 |   
  1.2e+09 |-+                                                               |   
          |                                                                 |   
    1e+09 |-+                                                               |   
          |                                                                 |   
    8e+08 |..  .+.  .+..+.. .+..          .+..  .+.        .+.             .|   
          |  +.   +.       +    +..+..+.+.    +.   +..+..+.   +..+..+..+.+. |   
    6e+08 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              netperf.time.user_time                            
                                                                                
  2600 +--------------------------------------------------------------------+   
       |                O        O                                          |   
  2400 |-+O  O       O     O  O     O     O    O     O     O     O    O     |   
       |        O  O                   O    O     O     O     O    O        |   
       |                                                                    |   
  2200 |-+                                                                  |   
       |                                                                    |   
  2000 |-+                                                                  |   
       |                                                                    |   
  1800 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
  1600 |-.+..                                                               |   
       |.    +..+..+.  .+..+..+..+..+..+..+.+..+..+..  .+..+..+..+.  .+..+..|   
  1400 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              netperf.time.system_time                          
                                                                                
  14400 +-------------------------------------------------------------------+   
        |                                                                   |   
  14200 |-+           +..  .+..+..                  .+..+..        .+.      |   
        |..+..+..+. ..   +.       +.+..+..+..+..+..+       +..+..+.   +..+..|   
  14000 |-+        +                                                        |   
        |                                                                   |   
  13800 |-+                                                                 |   
        |                                                                   |   
  13600 |-+                                                                 |   
        |                                                                   |   
  13400 |-+                                                                 |   
        |          O                                                        |   
  13200 |-+O  O  O          O  O    O  O  O  O  O  O O  O  O  O  O  O O     |   
        |             O  O        O                                         |   
  13000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        netperf.time.voluntary_context_switches                 
                                                                                
  1.6e+09 +-----------------------------------------------------------------+   
          |             O  O       O                                        |   
  1.4e+09 |-+                   O     O          O O     O          O  O    |   
  1.2e+09 |-+O  O O  O       O          O  O          O     O O  O          |   
          |                                   O                             |   
    1e+09 |-+                                                               |   
          |                                                                 |   
    8e+08 |-+                                                               |   
          |                                                                 |   
    6e+08 |..  .+.+..+                            .+..                  .+..|   
    4e+08 |-++.       :    +.     .+..+.+..+..+..+    +..  .+.+..+..  .+    |   
          |            :  +  +..+.                       +.         +.      |   
    2e+08 |-+          : +                                                  |   
          |             +                                                   |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.7.0-rc5-00221-ga6211caa634da" of type "text/plain" (202345 bytes)

View attachment "job-script" of type "text/plain" (7726 bytes)

View attachment "job.yaml" of type "text/plain" (5369 bytes)

View attachment "reproduce" of type "text/plain" (10675 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ