lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20190620020956.GO7221@shao2-debian>
Date:   Thu, 20 Jun 2019 10:09:56 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Eric Dumazet <edumazet@...gle.com>
Cc:     "David S. Miller" <davem@...emloft.net>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [tcp] ede61ca474: netperf.Throughput_Mbps 36.5% improvement

Greeting,

FYI, we noticed a 36.5% improvement of netperf.Throughput_Mbps due to commit:


commit: ede61ca474a0348b975d9824565b66c7595461de ("tcp: add tcp_rx_skb_cache sysctl")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: netperf
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	ip: ipv4
	runtime: 600s
	nr_threads: 200%
	cluster: cs-localhost
	test: TCP_STREAM
	cpufreq_governor: performance

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------+
| testcase: change | netperf:                                                   |
| test machine     | 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory |
| test parameters  | cluster=cs-localhost                                       |
|                  | cpufreq_governor=performance                               |
|                  | ip=ipv4                                                    |
|                  | nr_threads=200%                                            |
|                  | runtime=600s                                               |
|                  | test=TCP_STREAM                                            |
|                  | ucode=0x400001c                                            |
+------------------+------------------------------------------------------------+


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-2018-04-03.cgz/600s/lkp-skl-2sp7/TCP_STREAM/netperf

commit: 
  a8e11e5c56 ("sysctl: define proc_do_static_key()")
  ede61ca474 ("tcp: add tcp_rx_skb_cache sysctl")

a8e11e5c5611a9f7 ede61ca474a0348b975d9824565 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          2:4          -50%            :5     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :4           50%           2:5     dmesg.WARNING:at_ip__netif_receive_skb_core/0x
           :4           25%           1:5     dmesg.WARNING:at_ip_ip_finish_output2/0x
         %stddev     %change         %stddev
             \          |                \  
      3310           +36.5%       4518        netperf.Throughput_Mbps
    476765           +36.5%     650735        netperf.Throughput_total_Mbps
 1.443e+09           +36.0%  1.962e+09        netperf.time.involuntary_context_switches
     26037            -1.9%      25552        netperf.time.system_time
    926.25           +65.4%       1532 ±  2%  netperf.time.user_time
      4705 ±  4%    +883.6%      46279 ±  5%  netperf.time.voluntary_context_switches
 2.182e+09           +36.5%  2.979e+09        netperf.workload
    317804 ±  6%      -7.3%     294661 ±  6%  meminfo.DirectMap4k
      2925            -5.2%       2774        turbostat.Avg_MHz
     22.65            -7.2       15.43 ±  3%  mpstat.cpu.all.soft%
      4.78            +2.2        6.93 ±  2%  mpstat.cpu.all.usr%
  5.39e+08            -8.6%  4.928e+08 ±  2%  numa-numastat.node0.local_node
  5.39e+08            -8.6%  4.928e+08 ±  2%  numa-numastat.node0.numa_hit
 2.691e+08            -9.2%  2.443e+08        numa-vmstat.node0.numa_hit
 2.691e+08            -9.2%  2.442e+08        numa-vmstat.node0.numa_local
     94.00            -2.1%      92.00        vmstat.cpu.sy
   4796080           +35.7%    6509475        vmstat.system.cs
    153412            +1.3%     155421        vmstat.system.in
 1.092e+09            -8.4%      1e+09        proc-vmstat.numa_hit
 1.092e+09            -8.4%      1e+09        proc-vmstat.numa_local
 8.728e+09            -8.4%  7.993e+09        proc-vmstat.pgalloc_normal
 8.727e+09            -8.4%  7.992e+09        proc-vmstat.pgfree
    811.55            -9.8%     732.06        sched_debug.cfs_rq:/.util_est_enqueued.avg
      1586 ±  3%     -10.6%       1417 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.max
    293.47 ±  2%     -15.6%     247.77 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      4.36           +10.4%       4.82 ±  2%  sched_debug.cpu.cpu_load[2].min
      7098 ±  3%     -11.8%       6262 ± 11%  sched_debug.cpu.load.min
  20026979           +35.1%   27057938        sched_debug.cpu.nr_switches.avg
  20909879           +36.3%   28506716        sched_debug.cpu.nr_switches.max
  18629543           +31.2%   24444109        sched_debug.cpu.nr_switches.min
    521083 ±  8%     +81.1%     943750 ±  8%  sched_debug.cpu.nr_switches.stddev
     24.09           -24.2%      18.26        perf-stat.i.MPKI
 1.233e+10           +27.5%  1.572e+10        perf-stat.i.branch-instructions
      1.93            +0.0        1.97        perf-stat.i.branch-miss-rate%
 2.338e+08           +31.0%  3.064e+08        perf-stat.i.branch-misses
 1.452e+09            -1.9%  1.424e+09        perf-stat.i.cache-references
   4822346           +35.7%    6545267        perf-stat.i.context-switches
      3.48           -26.7%       2.55 ±  2%  perf-stat.i.cpi
 2.099e+11            -5.3%  1.989e+11        perf-stat.i.cpu-cycles
      1.04 ±  4%      +0.1        1.16 ±  2%  perf-stat.i.dTLB-load-miss-rate%
 1.904e+08 ±  4%     +44.6%  2.753e+08 ±  2%  perf-stat.i.dTLB-load-misses
 1.818e+10           +29.5%  2.355e+10        perf-stat.i.dTLB-loads
  36389994 ±  6%     +36.7%   49728544 ±  6%  perf-stat.i.dTLB-store-misses
 1.035e+10           +32.0%  1.367e+10        perf-stat.i.dTLB-stores
     49.33            +2.6       51.92        perf-stat.i.iTLB-load-miss-rate%
  40247503 ±  2%     +43.2%   57619885 ±  2%  perf-stat.i.iTLB-load-misses
  41247559           +28.9%   53170095        perf-stat.i.iTLB-loads
  6.12e+10           +28.1%  7.842e+10        perf-stat.i.instructions
      1550 ±  2%      -8.5%       1418 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.29           +35.3%       0.40 ±  2%  perf-stat.i.ipc
    676689 ± 22%    +161.1%    1766997 ± 15%  perf-stat.i.node-loads
     23.72           -23.5%      18.15        perf-stat.overall.MPKI
      1.90            +0.1        1.95        perf-stat.overall.branch-miss-rate%
      3.43           -26.1%       2.54 ±  2%  perf-stat.overall.cpi
      1.04 ±  4%      +0.1        1.16 ±  2%  perf-stat.overall.dTLB-load-miss-rate%
     49.38            +2.6       52.00        perf-stat.overall.iTLB-load-miss-rate%
      1521 ±  2%     -10.5%       1361 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.29           +35.3%       0.39 ±  2%  perf-stat.overall.ipc
     71.87 ±  5%     -23.4       48.48 ± 25%  perf-stat.overall.node-load-miss-rate%
     16855            -6.1%      15820        perf-stat.overall.path-length
 1.231e+10           +27.5%  1.569e+10        perf-stat.ps.branch-instructions
 2.334e+08           +31.0%  3.059e+08        perf-stat.ps.branch-misses
 1.449e+09            -1.9%  1.421e+09        perf-stat.ps.cache-references
   4814104           +35.7%    6533972        perf-stat.ps.context-switches
 2.096e+11            -5.3%  1.985e+11        perf-stat.ps.cpu-cycles
 1.901e+08 ±  4%     +44.6%  2.748e+08 ±  2%  perf-stat.ps.dTLB-load-misses
 1.815e+10           +29.5%  2.351e+10        perf-stat.ps.dTLB-loads
  36327516 ±  6%     +36.7%   49642182 ±  6%  perf-stat.ps.dTLB-store-misses
 1.033e+10           +32.0%  1.364e+10        perf-stat.ps.dTLB-stores
  40178575 ±  2%     +43.2%   57520127 ±  2%  perf-stat.ps.iTLB-load-misses
  41177027           +28.9%   53078387        perf-stat.ps.iTLB-loads
 6.109e+10           +28.1%  7.829e+10        perf-stat.ps.instructions
    675658 ± 22%    +161.4%    1765986 ± 15%  perf-stat.ps.node-loads
 3.679e+13           +28.1%  4.713e+13        perf-stat.total.instructions
      1.00 ± 70%  +12880.0%     129.80 ±117%  interrupts.65:PCI-MSI.31981598-edge.i40e-eth0-TxRx-29
     57.25 ±124%     -91.3%       5.00 ±190%  interrupts.90:PCI-MSI.31981623-edge.i40e-eth0-TxRx-54
      7755           -12.9%       6754 ±  6%  interrupts.CPU0.NMI:Non-maskable_interrupts
      7755           -12.9%       6754 ±  6%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
     61810 ±  3%     +33.3%      82407 ± 18%  interrupts.CPU10.RES:Rescheduling_interrupts
     56578 ±  4%     +54.8%      87606 ± 10%  interrupts.CPU12.RES:Rescheduling_interrupts
     62386 ± 10%     +26.5%      78916 ± 11%  interrupts.CPU15.RES:Rescheduling_interrupts
     65540 ±  6%     +26.3%      82769 ±  7%  interrupts.CPU19.RES:Rescheduling_interrupts
     66735 ±  5%     +24.4%      83044 ±  6%  interrupts.CPU21.RES:Rescheduling_interrupts
     58038 ±  4%     +44.3%      83734 ±  9%  interrupts.CPU22.RES:Rescheduling_interrupts
     66281 ±  6%     +31.2%      86973 ± 13%  interrupts.CPU23.RES:Rescheduling_interrupts
     69758 ± 15%     +26.2%      88045 ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
     64356 ±  4%     +20.5%      77519 ±  9%  interrupts.CPU25.RES:Rescheduling_interrupts
     67092 ±  7%     +30.4%      87484 ± 11%  interrupts.CPU27.RES:Rescheduling_interrupts
     64721 ±  8%     +31.0%      84808 ± 15%  interrupts.CPU28.RES:Rescheduling_interrupts
     67218 ±  4%     +20.2%      80771 ±  7%  interrupts.CPU30.RES:Rescheduling_interrupts
      8009           -23.3%       6139 ± 20%  interrupts.CPU33.NMI:Non-maskable_interrupts
      8009           -23.3%       6139 ± 20%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     68432 ±  6%     +27.8%      87485 ±  6%  interrupts.CPU33.RES:Rescheduling_interrupts
     62253 ±  6%     +47.1%      91571 ±  9%  interrupts.CPU34.RES:Rescheduling_interrupts
      7961           -23.3%       6110 ± 20%  interrupts.CPU35.NMI:Non-maskable_interrupts
      7961           -23.3%       6110 ± 20%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      7775           -22.0%       6065 ± 20%  interrupts.CPU36.NMI:Non-maskable_interrupts
      7775           -22.0%       6065 ± 20%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      7791           -22.0%       6074 ± 20%  interrupts.CPU37.NMI:Non-maskable_interrupts
      7791           -22.0%       6074 ± 20%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      7750           -22.1%       6034 ± 20%  interrupts.CPU38.NMI:Non-maskable_interrupts
      7750           -22.1%       6034 ± 20%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
     60522 ±  9%     +40.2%      84834 ±  5%  interrupts.CPU38.RES:Rescheduling_interrupts
      7749           -22.1%       6035 ± 20%  interrupts.CPU39.NMI:Non-maskable_interrupts
      7749           -22.1%       6035 ± 20%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
     62257 ±  4%     +25.4%      78063 ± 13%  interrupts.CPU39.RES:Rescheduling_interrupts
     62146 ± 16%     +32.0%      82038 ± 11%  interrupts.CPU41.RES:Rescheduling_interrupts
      7783           -22.3%       6050 ± 20%  interrupts.CPU42.NMI:Non-maskable_interrupts
      7783           -22.3%       6050 ± 20%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      7793           -22.3%       6058 ± 20%  interrupts.CPU43.NMI:Non-maskable_interrupts
      7793           -22.3%       6058 ± 20%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
     58804 ± 18%     +42.2%      83622 ± 10%  interrupts.CPU43.RES:Rescheduling_interrupts
      7721           -12.1%       6785 ±  6%  interrupts.CPU44.NMI:Non-maskable_interrupts
      7721           -12.1%       6785 ±  6%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      7774           -22.5%       6026 ± 20%  interrupts.CPU45.NMI:Non-maskable_interrupts
      7774           -22.5%       6026 ± 20%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
      7771           -22.1%       6051 ± 20%  interrupts.CPU46.NMI:Non-maskable_interrupts
      7771           -22.1%       6051 ± 20%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
     59920 ± 12%     +35.0%      80893 ± 15%  interrupts.CPU46.RES:Rescheduling_interrupts
      7785           -22.4%       6040 ± 20%  interrupts.CPU47.NMI:Non-maskable_interrupts
      7785           -22.4%       6040 ± 20%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
     62454 ± 14%     +26.8%      79204 ± 11%  interrupts.CPU47.RES:Rescheduling_interrupts
      7752           -12.5%       6783 ±  6%  interrupts.CPU48.NMI:Non-maskable_interrupts
      7752           -12.5%       6783 ±  6%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      7789           -22.4%       6045 ± 20%  interrupts.CPU49.NMI:Non-maskable_interrupts
      7789           -22.4%       6045 ± 20%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
     56431 ± 14%     +45.7%      82247 ± 17%  interrupts.CPU49.RES:Rescheduling_interrupts
      7750           -22.2%       6027 ± 20%  interrupts.CPU50.NMI:Non-maskable_interrupts
      7750           -22.2%       6027 ± 20%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
     61121 ±  5%     +31.5%      80349 ± 19%  interrupts.CPU50.RES:Rescheduling_interrupts
      7812           -22.4%       6058 ± 20%  interrupts.CPU51.NMI:Non-maskable_interrupts
      7812           -22.4%       6058 ± 20%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
     61379 ± 10%     +47.1%      90306 ±  6%  interrupts.CPU51.RES:Rescheduling_interrupts
     56451 ±  8%     +23.5%      69695 ±  3%  interrupts.CPU53.RES:Rescheduling_interrupts
     62541 ±  6%     +46.5%      91595 ± 10%  interrupts.CPU54.RES:Rescheduling_interrupts
     60908 ±  9%     +55.6%      94803 ± 16%  interrupts.CPU56.RES:Rescheduling_interrupts
     65323 ± 12%     +27.5%      83269 ±  5%  interrupts.CPU57.RES:Rescheduling_interrupts
     60774 ± 10%     +33.3%      80996 ±  8%  interrupts.CPU6.RES:Rescheduling_interrupts
     62070 ±  8%     +26.8%      78725 ±  2%  interrupts.CPU60.RES:Rescheduling_interrupts
     65281 ±  3%     +30.3%      85086 ± 13%  interrupts.CPU61.RES:Rescheduling_interrupts
     61530 ±  9%     +36.1%      83722 ± 11%  interrupts.CPU62.RES:Rescheduling_interrupts
     59129 ±  5%     +37.6%      81379 ± 18%  interrupts.CPU67.RES:Rescheduling_interrupts
     67413 ±  8%     +16.6%      78602 ± 10%  interrupts.CPU69.RES:Rescheduling_interrupts
     67134 ± 11%     +20.2%      80710 ± 13%  interrupts.CPU70.RES:Rescheduling_interrupts
     57213 ±  8%     +50.6%      86168 ± 11%  interrupts.CPU71.RES:Rescheduling_interrupts
   4630804 ±  3%     +26.4%    5851310 ±  2%  interrupts.RES:Rescheduling_interrupts
  17331944           +37.9%   23900673        softirqs.CPU0.NET_RX
  17530939           +36.3%   23889797        softirqs.CPU1.NET_RX
  17136982 ±  2%     +39.0%   23820881 ±  2%  softirqs.CPU10.NET_RX
     58917           +34.3%      79149 ± 24%  softirqs.CPU10.RCU
  17571701 ±  2%     +37.4%   24147419        softirqs.CPU11.NET_RX
  17672278           +31.9%   23306041 ±  4%  softirqs.CPU12.NET_RX
  17325581           +38.4%   23975105 ±  2%  softirqs.CPU13.NET_RX
  17294123 ±  2%     +38.9%   24026366 ±  2%  softirqs.CPU14.NET_RX
  17581330           +32.6%   23306553 ±  4%  softirqs.CPU15.NET_RX
  17744380           +35.3%   24005037 ±  2%  softirqs.CPU16.NET_RX
  17366681           +36.0%   23625553 ±  3%  softirqs.CPU17.NET_RX
  17513047           +34.6%   23580015 ±  2%  softirqs.CPU18.NET_RX
     63886           +30.7%      83523 ± 22%  softirqs.CPU18.RCU
  17950167           +36.1%   24429312 ±  3%  softirqs.CPU19.NET_RX
  17280755 ±  2%     +38.0%   23846236 ±  3%  softirqs.CPU2.NET_RX
  18132078           +33.8%   24265893        softirqs.CPU20.NET_RX
  18027898           +36.2%   24548533        softirqs.CPU21.NET_RX
  17696115           +38.9%   24572590 ±  2%  softirqs.CPU22.NET_RX
  17847777           +34.1%   23940312 ±  2%  softirqs.CPU23.NET_RX
  18099147           +37.0%   24789584        softirqs.CPU24.NET_RX
  18223713           +36.3%   24845983        softirqs.CPU25.NET_RX
  17512311 ±  2%     +37.4%   24069333 ±  3%  softirqs.CPU26.NET_RX
  17841600           +33.6%   23839917 ±  2%  softirqs.CPU27.NET_RX
  17601089 ±  3%     +35.4%   23828320 ±  4%  softirqs.CPU28.NET_RX
  18169931           +36.7%   24835918        softirqs.CPU29.NET_RX
  17445850 ±  2%     +36.1%   23739675 ±  4%  softirqs.CPU3.NET_RX
  17722005 ±  2%     +29.8%   23008127        softirqs.CPU30.NET_RX
  17663859 ±  2%     +38.4%   24447008        softirqs.CPU31.NET_RX
  17928131           +34.5%   24108420 ±  2%  softirqs.CPU32.NET_RX
     62457           +29.9%      81131 ± 22%  softirqs.CPU32.RCU
  18220692           +34.7%   24536407 ±  2%  softirqs.CPU33.NET_RX
  18286650           +34.3%   24565350        softirqs.CPU34.NET_RX
  17304413 ±  2%     +40.8%   24363651        softirqs.CPU35.NET_RX
  17173191           +33.4%   22916547 ±  4%  softirqs.CPU36.NET_RX
  17563105           +34.5%   23628157 ±  3%  softirqs.CPU37.NET_RX
  17435903 ±  2%     +35.1%   23564057 ±  3%  softirqs.CPU38.NET_RX
     62230 ±  4%     +34.3%      83598 ± 24%  softirqs.CPU38.RCU
  17272182 ±  2%     +35.1%   23336729 ±  5%  softirqs.CPU39.NET_RX
  17335637           +34.0%   23236226 ±  3%  softirqs.CPU4.NET_RX
  17164703 ±  2%     +36.6%   23445940 ±  2%  softirqs.CPU40.NET_RX
     62436 ±  2%     +34.6%      84012 ± 23%  softirqs.CPU40.RCU
  17030886 ±  2%     +31.7%   22422929 ±  4%  softirqs.CPU41.NET_RX
  17349878 ±  2%     +39.2%   24157883        softirqs.CPU42.NET_RX
     61585 ±  3%     +37.2%      84468 ± 23%  softirqs.CPU42.RCU
  17775923           +36.6%   24279566 ±  2%  softirqs.CPU43.NET_RX
     62964 ±  2%     +35.3%      85174 ± 23%  softirqs.CPU43.RCU
  17268058 ±  2%     +36.6%   23592781 ±  2%  softirqs.CPU44.NET_RX
     63331 ±  2%     +33.2%      84384 ± 23%  softirqs.CPU44.RCU
  17148500 ±  2%     +38.8%   23796364 ±  2%  softirqs.CPU45.NET_RX
  17615847           +35.1%   23796420        softirqs.CPU46.NET_RX
     63792 ±  2%     +34.6%      85859 ± 23%  softirqs.CPU46.RCU
  17153040 ±  2%     +38.6%   23779686 ±  3%  softirqs.CPU47.NET_RX
  17324943 ±  2%     +38.8%   24050281 ±  2%  softirqs.CPU48.NET_RX
     63757           +34.1%      85479 ± 23%  softirqs.CPU48.RCU
  17102014           +32.3%   22627699 ±  3%  softirqs.CPU49.NET_RX
     64602 ±  2%     +30.1%      84027 ± 22%  softirqs.CPU49.RCU
  17338693           +36.7%   23705985        softirqs.CPU5.NET_RX
     59132           +33.0%      78620 ± 24%  softirqs.CPU5.RCU
  17425587 ±  2%     +32.7%   23117622 ±  4%  softirqs.CPU50.NET_RX
  17618503           +36.0%   23964794 ±  3%  softirqs.CPU51.NET_RX
  17354234           +37.8%   23913189 ±  4%  softirqs.CPU52.NET_RX
     63888           +33.3%      85175 ± 23%  softirqs.CPU52.RCU
  17115148 ±  2%     +36.2%   23311982 ±  4%  softirqs.CPU53.NET_RX
  17735102           +35.2%   23986709        softirqs.CPU54.NET_RX
  18057531           +34.2%   24239055 ±  2%  softirqs.CPU55.NET_RX
  17551567           +39.3%   24454599 ±  4%  softirqs.CPU56.NET_RX
  18150174           +34.1%   24345105        softirqs.CPU57.NET_RX
  17949707           +36.3%   24462203 ±  2%  softirqs.CPU58.NET_RX
  17612693 ±  2%     +40.8%   24794166        softirqs.CPU59.NET_RX
  17495928 ±  2%     +36.3%   23839559 ±  2%  softirqs.CPU6.NET_RX
     58659           +33.9%      78562 ± 24%  softirqs.CPU6.RCU
  18113824           +37.0%   24822642        softirqs.CPU60.NET_RX
  18138838           +35.8%   24629071 ±  4%  softirqs.CPU61.NET_RX
  17915148           +35.7%   24309691 ±  3%  softirqs.CPU62.NET_RX
  17681225           +39.1%   24596506        softirqs.CPU63.NET_RX
  17853174           +36.3%   24329595 ±  4%  softirqs.CPU64.NET_RX
  17861934 ±  2%     +35.7%   24231143 ±  2%  softirqs.CPU65.NET_RX
  17843127 ±  3%     +39.6%   24906670        softirqs.CPU66.NET_RX
  17240363 ±  2%     +38.3%   23839354        softirqs.CPU67.NET_RX
  17773093           +37.0%   24341283 ±  2%  softirqs.CPU68.NET_RX
  18018477 ±  2%     +37.9%   24840396        softirqs.CPU69.NET_RX
  17692495           +36.6%   24168650 ±  2%  softirqs.CPU7.NET_RX
  17975980 ±  2%     +35.3%   24319019 ±  2%  softirqs.CPU70.NET_RX
  17796849           +31.9%   23479744 ±  3%  softirqs.CPU71.NET_RX
  17621674           +34.0%   23613890 ±  2%  softirqs.CPU8.NET_RX
  17292292 ±  2%     +32.6%   22931589 ±  3%  softirqs.CPU9.NET_RX
 1.269e+09           +36.0%  1.726e+09        softirqs.NET_RX
     61.47            -6.8       54.67        perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     61.67            -6.7       54.96        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     48.45            -5.5       42.98        perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      8.37 ± 14%      -4.4        4.00 ±  6%  perf-profile.calltrace.cycles-pp.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      8.34 ± 14%      -4.4        3.98 ±  6%  perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      8.11 ± 15%      -4.3        3.82 ±  7%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg
      7.84 ± 16%      -4.2        3.67 ±  7%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked
      6.56 ± 18%      -4.0        2.55 ± 10%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill
      6.37 ± 19%      -4.0        2.38 ± 10%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill
     22.37 ±  2%      -3.7       18.63 ±  3%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
     28.02 ±  3%      -3.5       24.55 ±  3%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     26.62 ±  3%      -3.5       23.17 ±  3%  perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
     26.79 ±  3%      -3.4       23.35 ±  3%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     26.82 ±  3%      -3.4       23.39 ±  3%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     26.92 ±  3%      -3.4       23.49 ±  3%  perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
     27.21 ±  2%      -3.4       23.83 ±  3%  perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     27.46 ±  2%      -3.2       24.24 ±  3%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     27.77 ±  2%      -3.2       24.61 ±  3%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
     27.97 ±  2%      -3.1       24.87 ±  3%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     28.20 ±  2%      -3.1       25.13 ±  3%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     28.28 ±  2%      -3.1       25.21 ±  3%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     28.45 ±  2%      -3.0       25.43 ±  3%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
     28.49 ±  2%      -3.0       25.48 ±  3%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
      6.62 ±  7%      -2.9        3.67 ±  2%  perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      9.31            -2.9        6.38 ±  3%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established
      9.47            -2.9        6.56 ±  3%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
      9.68            -2.9        6.80 ±  2%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      9.68            -2.9        6.81 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      4.28 ± 17%      -2.9        1.41 ± 16%  perf-profile.calltrace.cycles-pp.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      4.07 ± 18%      -2.8        1.25 ± 17%  perf-profile.calltrace.cycles-pp.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg
      3.90 ± 18%      -2.8        1.14 ± 18%  perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked
      3.66 ± 20%      -2.7        0.99 ± 19%  perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb
      3.53 ± 20%      -2.6        0.91 ± 20%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim
     12.40            -1.4       10.96        perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
     12.55            -1.4       11.18 ±  2%  perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      2.19 ± 38%      -1.2        0.97 ± 25%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data
      2.28 ± 37%      -1.0        1.25 ± 28%  perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb
      6.30 ± 61%      -1.0        5.34 ±181%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.30 ± 61%      -1.0        5.35 ±181%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      4.00 ± 61%      -0.9        3.12 ±183%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.00 ± 61%      -0.9        3.12 ±183%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.95 ± 10%      -0.8        4.14 ±  6%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
      4.68 ± 13%      -0.7        4.00 ±  6%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
      0.26 ±100%      +0.4        0.69 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.send.send_omni_inner.send_tcp_stream.main
      0.00            +0.7        0.68 ± 14%  perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_recvmsg.inet_recvmsg
      2.54 ±  4%      +0.7        3.25 ±  3%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg
      3.33 ±  4%      +0.8        4.12 ±  3%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      2.95 ±  4%      +0.8        3.74 ±  3%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg
      0.00            +0.8        0.83 ± 14%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      3.62 ±  4%      +0.8        4.47 ±  3%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.00            +1.0        1.05 ± 18%  perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue
      0.00            +1.1        1.13 ± 18%  perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack
      0.00            +1.2        1.16 ± 14%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.00            +1.4        1.38 ± 16%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      9.54            +1.4       10.94        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg
     10.05            +1.4       11.47        perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      9.70            +1.4       11.13        perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      0.00            +1.4        1.45 ± 16%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
      0.00            +1.8        1.78 ± 15%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      0.14 ±173%      +2.0        2.13 ±  8%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      5.82            +2.8        8.59 ±  2%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      5.92            +2.8        8.72 ±  2%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
      6.04            +2.8        8.89 ±  2%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
      6.59            +3.0        9.60 ±  2%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      6.61            +3.0        9.62 ±  2%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
     26.70 ±  3%      +4.9       31.60        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     26.74 ±  3%      +4.9       31.65        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.98 ± 18%      -8.1        4.85 ± 10%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     61.75            -6.8       54.98        perf-profile.children.cycles-pp.tcp_sendmsg
     61.95            -6.7       55.27        perf-profile.children.cycles-pp.sock_sendmsg
     62.43            -6.5       55.90        perf-profile.children.cycles-pp.__sys_sendto
     62.53            -6.5       56.02        perf-profile.children.cycles-pp.__x64_sys_sendto
     48.75 ±  2%      -5.6       43.19        perf-profile.children.cycles-pp.tcp_sendmsg_locked
      8.40 ± 15%      -4.4        4.04 ±  7%  perf-profile.children.cycles-pp.sk_page_frag_refill
      8.37 ± 15%      -4.4        4.01 ±  7%  perf-profile.children.cycles-pp.skb_page_frag_refill
      8.06 ± 15%      -4.3        3.75 ±  7%  perf-profile.children.cycles-pp.get_page_from_freelist
      8.21 ± 15%      -4.3        3.90 ±  7%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      7.43 ± 16%      -4.1        3.29 ±  8%  perf-profile.children.cycles-pp.__free_pages_ok
      7.10 ± 17%      -4.1        2.98 ±  9%  perf-profile.children.cycles-pp.free_one_page
      6.99 ± 17%      -4.1        2.93 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      7.08 ± 17%      -4.0        3.08 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
     26.97 ±  3%      -3.5       23.46 ±  3%  perf-profile.children.cycles-pp.tcp_v4_rcv
     27.13 ±  3%      -3.5       23.64 ±  3%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     27.17 ±  3%      -3.5       23.68 ±  3%  perf-profile.children.cycles-pp.ip_local_deliver_finish
     27.26 ±  3%      -3.5       23.80 ±  3%  perf-profile.children.cycles-pp.ip_local_deliver
     27.56 ±  2%      -3.4       24.15 ±  3%  perf-profile.children.cycles-pp.ip_rcv
     27.82 ±  2%      -3.3       24.48 ±  3%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     24.03 ±  2%      -3.3       20.74 ±  2%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
     28.14 ±  2%      -3.3       24.86 ±  3%  perf-profile.children.cycles-pp.process_backlog
     28.36 ±  2%      -3.2       25.12 ±  3%  perf-profile.children.cycles-pp.net_rx_action
     28.62 ±  2%      -3.2       25.42 ±  3%  perf-profile.children.cycles-pp.__softirqentry_text_start
     28.67 ±  2%      -3.2       25.47 ±  3%  perf-profile.children.cycles-pp.do_softirq_own_stack
     28.86 ±  2%      -3.2       25.70 ±  3%  perf-profile.children.cycles-pp.do_softirq
     29.01 ±  2%      -3.1       25.88 ±  3%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      4.56 ± 13%      -3.1        1.49 ± 10%  perf-profile.children.cycles-pp.___pskb_trim
      6.88 ±  7%      -3.0        3.84 ±  2%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
     30.92 ±  2%      -2.9       28.03 ±  2%  perf-profile.children.cycles-pp.ip_finish_output2
     31.95 ±  2%      -2.7       29.24 ±  2%  perf-profile.children.cycles-pp.ip_output
     30.90 ±  2%      -2.7       28.21 ±  2%  perf-profile.children.cycles-pp.tcp_write_xmit
     32.87 ±  2%      -2.7       30.21 ±  2%  perf-profile.children.cycles-pp.__ip_queue_xmit
     33.82 ±  2%      -2.4       31.37 ±  2%  perf-profile.children.cycles-pp.__tcp_transmit_skb
     12.58            -1.3       11.24        perf-profile.children.cycles-pp.__release_sock
     12.88            -1.2       11.63        perf-profile.children.cycles-pp.release_sock
      3.44 ± 18%      -1.0        2.43 ±  5%  perf-profile.children.cycles-pp.skb_release_data
     95.46            -0.9       94.54        perf-profile.children.cycles-pp.do_syscall_64
      3.88 ± 16%      -0.9        2.97 ±  5%  perf-profile.children.cycles-pp.__kfree_skb
     95.56            -0.9       94.68        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.21 ±  5%      -0.1        0.15 ±  9%  perf-profile.children.cycles-pp.prep_new_page
      0.08 ±  8%      -0.0        0.04 ± 50%  perf-profile.children.cycles-pp.kfree_skbmem
      0.09 ± 13%      -0.0        0.05 ± 52%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.09 ± 13%      -0.0        0.05 ± 52%  perf-profile.children.cycles-pp.shmem_alloc_page
      0.15 ±  3%      -0.0        0.12 ±  9%  perf-profile.children.cycles-pp.kfree
      0.08 ±  5%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.sock_rfree
      0.07            -0.0        0.05 ±  9%  perf-profile.children.cycles-pp.tcp_cleanup_rbuf
      0.23 ±  3%      -0.0        0.21 ±  2%  perf-profile.children.cycles-pp.__list_add_valid
      0.05            +0.0        0.06        perf-profile.children.cycles-pp.bictcp_cong_avoid
      0.05            +0.0        0.06        perf-profile.children.cycles-pp.__sock_wfree
      0.09 ±  4%      +0.0        0.10 ±  3%  perf-profile.children.cycles-pp.sock_recvmsg
      0.20 ±  2%      +0.0        0.22 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.08            +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.add_wait_queue
      0.05            +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.sock_put
      0.16 ±  2%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.skb_entail
      0.10 ±  7%      +0.0        0.12 ±  6%  perf-profile.children.cycles-pp.selinux_ipv4_output
      0.06 ±  7%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.tcp_stream_memory_free
      0.05 ±  8%      +0.0        0.07 ± 10%  perf-profile.children.cycles-pp.rcu_all_qs
      0.06 ±  9%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.lock_timer_base
      0.36            +0.0        0.38 ±  3%  perf-profile.children.cycles-pp.__switch_to_asm
      0.17 ±  2%      +0.0        0.19 ±  5%  perf-profile.children.cycles-pp.__calc_delta
      0.09 ±  8%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
      0.08 ±  5%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.import_single_range
      0.06 ±  6%      +0.0        0.09 ±  8%  perf-profile.children.cycles-pp.tcp_event_data_recv
      0.09 ±  4%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.tcp_v4_send_check
      0.12 ± 11%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.ksoftirqd_running
      0.10 ±  5%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.tcp_wfree
      0.05 ±  9%      +0.0        0.08 ±  4%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.09 ±  4%      +0.0        0.12        perf-profile.children.cycles-pp.tcp_release_cb
      0.08            +0.0        0.11 ± 10%  perf-profile.children.cycles-pp.rb_erase_cached
      0.12 ±  5%      +0.0        0.15 ±  9%  perf-profile.children.cycles-pp.find_next_bit
      0.04 ± 57%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.__tcp_select_window
      0.12            +0.0        0.15 ±  5%  perf-profile.children.cycles-pp.ip_rcv_core
      0.12 ±  5%      +0.0        0.15 ±  9%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.06 ±  7%      +0.0        0.09 ±  8%  perf-profile.children.cycles-pp.tcp_queue_rcv
      0.26            +0.0        0.30 ±  4%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.16 ±  4%      +0.0        0.19 ±  4%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.08 ±  6%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.ip_send_check
      0.22            +0.0        0.26 ±  2%  perf-profile.children.cycles-pp.pick_next_entity
      0.12 ±  8%      +0.0        0.16 ±  4%  perf-profile.children.cycles-pp.__skb_clone
      0.18 ±  4%      +0.0        0.22 ±  6%  perf-profile.children.cycles-pp.ip_finish_output
      0.33 ±  4%      +0.0        0.37 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.18 ±  2%      +0.0        0.22 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.31 ±  4%      +0.0        0.35 ±  3%  perf-profile.children.cycles-pp.___perf_sw_event
      0.42 ±  4%      +0.0        0.46 ±  3%  perf-profile.children.cycles-pp.update_cfs_group
      0.19 ±  2%      +0.0        0.23 ±  3%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.22 ±  4%      +0.0        0.27 ±  5%  perf-profile.children.cycles-pp.selinux_parse_skb
      0.01 ±173%      +0.0        0.06        perf-profile.children.cycles-pp.rb_next
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.ip_queue_xmit
      0.23            +0.1        0.28 ±  4%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.__tcp_ack_snd_check
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.__put_compound_page
      0.29            +0.1        0.34 ±  5%  perf-profile.children.cycles-pp.set_next_entity
      0.33 ±  2%      +0.1        0.38 ±  3%  perf-profile.children.cycles-pp.native_write_msr
      0.23            +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.26 ±  2%      +0.1        0.32 ±  4%  perf-profile.children.cycles-pp.check_preempt_curr
      0.18            +0.1        0.24 ±  2%  perf-profile.children.cycles-pp.sock_has_perm
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.sk_stream_write_space
      0.49            +0.1        0.54 ±  3%  perf-profile.children.cycles-pp.switch_fpu_return
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.tcp_data_queue
      0.34 ±  2%      +0.1        0.40 ±  5%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.30 ±  2%      +0.1        0.36 ±  3%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.47            +0.1        0.54 ±  3%  perf-profile.children.cycles-pp.selinux_ip_postroute_compat
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.tcp_v4_fill_cb
      0.67            +0.1        0.73 ±  3%  perf-profile.children.cycles-pp.load_new_mm_cr3
      0.31 ±  2%      +0.1        0.38 ±  2%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.14            +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.28            +0.1        0.35 ±  3%  perf-profile.children.cycles-pp.sk_reset_timer
      0.56 ±  2%      +0.1        0.63 ±  3%  perf-profile.children.cycles-pp.__switch_to
      0.30 ±  3%      +0.1        0.38 ±  3%  perf-profile.children.cycles-pp.__ip_local_out
      0.47 ±  3%      +0.1        0.55 ±  3%  perf-profile.children.cycles-pp.netif_rx
      0.46 ±  4%      +0.1        0.54 ±  3%  perf-profile.children.cycles-pp.netif_rx_internal
      0.33 ±  3%      +0.1        0.41 ±  4%  perf-profile.children.cycles-pp.ip_local_out
      0.25            +0.1        0.33 ±  3%  perf-profile.children.cycles-pp.mod_timer
      0.50 ±  2%      +0.1        0.59        perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.43            +0.1        0.52 ±  2%  perf-profile.children.cycles-pp.selinux_sock_rcv_skb_compat
      0.40            +0.1        0.49 ±  2%  perf-profile.children.cycles-pp.__fget_light
      0.31            +0.1        0.40 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.58            +0.1        0.68 ±  3%  perf-profile.children.cycles-pp.selinux_ip_postroute
      0.28 ± 14%      +0.1        0.38 ± 10%  perf-profile.children.cycles-pp.skb_release_all
      0.00            +0.1        0.10 ±  3%  perf-profile.children.cycles-pp.tcp_check_space
      0.36            +0.1        0.47        perf-profile.children.cycles-pp.simple_copy_to_iter
      0.27 ± 14%      +0.1        0.37 ± 10%  perf-profile.children.cycles-pp.skb_release_head_state
      0.35            +0.1        0.45 ±  2%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.37            +0.1        0.48 ±  3%  perf-profile.children.cycles-pp.lock_sock_nested
      0.43 ±  3%      +0.1        0.54 ±  3%  perf-profile.children.cycles-pp.read_tsc
      0.44 ± 10%      +0.1        0.55 ±  8%  perf-profile.children.cycles-pp.tcp_send_mss
      0.24 ±  3%      +0.1        0.36 ±  2%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.45 ± 10%      +0.1        0.56 ±  8%  perf-profile.children.cycles-pp.tcp_current_mss
      0.46            +0.1        0.58 ±  2%  perf-profile.children.cycles-pp.sockfd_lookup_light
      1.33            +0.1        1.44 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.54 ±  2%      +0.1        0.66 ±  2%  perf-profile.children.cycles-pp.__check_object_size
      0.54 ±  4%      +0.1        0.67 ±  3%  perf-profile.children.cycles-pp.ktime_get
      0.65            +0.1        0.78 ±  2%  perf-profile.children.cycles-pp.selinux_socket_sock_rcv_skb
      0.85            +0.1        1.00 ±  3%  perf-profile.children.cycles-pp.nf_hook_slow
      0.71            +0.1        0.86        perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.77            +0.1        0.92        perf-profile.children.cycles-pp.sk_filter_trim_cap
      1.23            +0.2        1.38 ±  3%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.98 ±  2%      +0.2        1.16 ±  3%  perf-profile.children.cycles-pp.loopback_xmit
      0.00            +0.2        0.20 ±  6%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      1.05            +0.2        1.25 ±  3%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.00            +0.2        0.20 ±  7%  perf-profile.children.cycles-pp.__sk_mem_schedule
      1.63            +0.2        1.87 ±  3%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.91            +0.3        1.16 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.79            +0.3        1.06 ±  3%  perf-profile.children.cycles-pp.syscall_return_via_sysret
     10.09            +1.5       11.56 ±  2%  perf-profile.children.cycles-pp._copy_from_iter_full
      9.86            +1.5       11.35        perf-profile.children.cycles-pp.copyin
      0.47 ±  3%      +1.6        2.09 ±  8%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.81 ±  2%      +1.7        2.49 ±  7%  perf-profile.children.cycles-pp.tcp_ack
      6.04 ±  2%      +2.8        8.79 ±  2%  perf-profile.children.cycles-pp.copyout
      6.16            +2.8        8.96 ±  2%  perf-profile.children.cycles-pp._copy_to_iter
      6.73            +3.0        9.68 ±  2%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      6.71            +3.0        9.67 ±  2%  perf-profile.children.cycles-pp.__skb_datagram_iter
     15.78            +4.2       20.00 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
     26.84 ±  3%      +4.9       31.76        perf-profile.children.cycles-pp.tcp_recvmsg
     26.87 ±  3%      +4.9       31.79        perf-profile.children.cycles-pp.inet_recvmsg
     27.28 ±  3%      +5.0       32.28        perf-profile.children.cycles-pp.__sys_recvfrom
     27.35 ±  3%      +5.0       32.36        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     12.97 ± 18%      -8.1        4.85 ± 10%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.16 ±  2%      -0.1        1.07        perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.20 ±  5%      -0.1        0.12 ±  6%  perf-profile.self.cycles-pp.skb_page_frag_refill
      0.18 ±  2%      -0.1        0.12 ±  8%  perf-profile.self.cycles-pp.___pskb_trim
      0.20 ±  4%      -0.0        0.15 ±  7%  perf-profile.self.cycles-pp.prep_new_page
      0.08 ±  5%      -0.0        0.03 ± 81%  perf-profile.self.cycles-pp.kfree_skbmem
      0.30 ±  2%      -0.0        0.26 ±  9%  perf-profile.self.cycles-pp.update_rq_clock
      0.15 ±  7%      -0.0        0.11 ± 11%  perf-profile.self.cycles-pp.__sk_dst_check
      0.55 ±  2%      -0.0        0.52        perf-profile.self.cycles-pp.get_page_from_freelist
      0.15 ±  2%      -0.0        0.12 ±  9%  perf-profile.self.cycles-pp.kfree
      0.44            -0.0        0.41 ±  3%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.23 ±  3%      -0.0        0.20 ±  4%  perf-profile.self.cycles-pp.free_one_page
      0.22 ±  3%      -0.0        0.20 ±  5%  perf-profile.self.cycles-pp.enqueue_entity
      0.08 ±  5%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.sock_rfree
      0.07 ±  6%      -0.0        0.05        perf-profile.self.cycles-pp.tcp_cleanup_rbuf
      0.23 ±  2%      -0.0        0.21 ±  3%  perf-profile.self.cycles-pp.__list_add_valid
      0.05            +0.0        0.06        perf-profile.self.cycles-pp.bictcp_cong_avoid
      0.06            +0.0        0.07 ±  6%  perf-profile.self.cycles-pp.woken_wake_function
      0.06 ±  7%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.tcp_queue_rcv
      0.10 ±  5%      +0.0        0.11        perf-profile.self.cycles-pp.ip_local_deliver
      0.05 ±  8%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.tcp_stream_memory_free
      0.05            +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.sock_put
      0.06            +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.selinux_ipv4_postroute
      0.07 ±  7%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.tcp_event_new_data_sent
      0.07 ±  6%      +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.release_sock
      0.08 ±  5%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.tcp_sendmsg
      0.09 ±  5%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_v4_send_check
      0.06 ±  6%      +0.0        0.08 ±  9%  perf-profile.self.cycles-pp.import_single_range
      0.05 ±  8%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.sockfd_lookup_light
      0.15 ±  5%      +0.0        0.17 ±  2%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.09            +0.0        0.11 ±  5%  perf-profile.self.cycles-pp.sk_stream_alloc_skb
      0.06 ± 11%      +0.0        0.08        perf-profile.self.cycles-pp.security_sock_rcv_skb
      0.09 ±  5%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.10 ±  5%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.07 ±  6%      +0.0        0.09 ± 22%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.21 ±  3%      +0.0        0.23 ±  5%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.18 ±  3%      +0.0        0.20 ±  4%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.17 ±  2%      +0.0        0.19 ±  5%  perf-profile.self.cycles-pp.__calc_delta
      0.10 ±  4%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.09 ±  4%      +0.0        0.11 ±  3%  perf-profile.self.cycles-pp.tcp_wfree
      0.10 ±  5%      +0.0        0.12 ±  5%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.04 ± 58%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
      0.09 ±  4%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.pick_next_entity
      0.10 ±  4%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.sock_def_readable
      0.09 ±  9%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp._copy_to_iter
      0.12 ±  9%      +0.0        0.14 ±  2%  perf-profile.self.cycles-pp.ksoftirqd_running
      0.14 ±  3%      +0.0        0.17 ±  5%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.06            +0.0        0.09 ±  8%  perf-profile.self.cycles-pp.tcp_event_data_recv
      0.14 ±  3%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.__alloc_skb
      0.09 ±  5%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.tcp_release_cb
      0.11 ±  4%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.ip_rcv_core
      0.15 ±  2%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.10 ± 10%      +0.0        0.13 ±  5%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.10 ±  4%      +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.mod_timer
      0.04 ± 57%      +0.0        0.07 ± 17%  perf-profile.self.cycles-pp.rb_erase_cached
      0.10 ± 11%      +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.__skb_clone
      0.10 ±  4%      +0.0        0.14 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.11 ±  3%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.selinux_ip_postroute
      0.20 ±  2%      +0.0        0.23 ±  2%  perf-profile.self.cycles-pp.process_backlog
      0.15 ±  3%      +0.0        0.19 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.12 ±  4%      +0.0        0.16 ±  3%  perf-profile.self.cycles-pp.try_to_wake_up
      0.12 ±  4%      +0.0        0.16        perf-profile.self.cycles-pp.selinux_sock_rcv_skb_compat
      0.18 ±  6%      +0.0        0.22 ±  2%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.16 ±  5%      +0.0        0.20 ±  3%  perf-profile.self.cycles-pp.ip_output
      0.17 ±  2%      +0.0        0.20 ±  3%  perf-profile.self.cycles-pp.net_rx_action
      0.18 ±  4%      +0.0        0.21 ±  4%  perf-profile.self.cycles-pp.selinux_ip_postroute_compat
      0.07 ±  5%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.ip_send_check
      0.24            +0.0        0.28 ±  6%  perf-profile.self.cycles-pp.loopback_xmit
      0.28 ±  5%      +0.0        0.31 ±  5%  perf-profile.self.cycles-pp.___perf_sw_event
      0.03 ±100%      +0.0        0.06 ±  7%  perf-profile.self.cycles-pp.rcu_all_qs
      0.17            +0.0        0.21 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.22 ±  3%      +0.0        0.26 ±  4%  perf-profile.self.cycles-pp.selinux_parse_skb
      0.42 ±  5%      +0.0        0.46 ±  4%  perf-profile.self.cycles-pp.update_cfs_group
      0.16 ±  6%      +0.0        0.20 ±  5%  perf-profile.self.cycles-pp.tcp_current_mss
      0.21 ±  2%      +0.0        0.26 ±  4%  perf-profile.self.cycles-pp.selinux_socket_sock_rcv_skb
      0.15 ±  3%      +0.0        0.20 ±  6%  perf-profile.self.cycles-pp.tcp_ack
      0.14 ±  5%      +0.0        0.18 ±  4%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.21 ±  8%      +0.0        0.25 ±  3%  perf-profile.self.cycles-pp.ktime_get
      0.15 ±  5%      +0.0        0.20        perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.14 ±  5%      +0.0        0.19 ±  3%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.18 ±  8%      +0.0        0.23 ±  2%  perf-profile.self.cycles-pp.__check_object_size
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.netif_rx_internal
      0.16 ±  2%      +0.1        0.22 ±  2%  perf-profile.self.cycles-pp.__sys_sendto
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.rb_next
      0.33 ±  2%      +0.1        0.38 ±  4%  perf-profile.self.cycles-pp.native_write_msr
      0.18 ±  3%      +0.1        0.24 ±  3%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.49            +0.1        0.54 ±  3%  perf-profile.self.cycles-pp.switch_fpu_return
      0.22            +0.1        0.28 ±  2%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.17 ±  2%      +0.1        0.23 ±  2%  perf-profile.self.cycles-pp.sock_has_perm
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_v4_fill_cb
      0.31 ±  4%      +0.1        0.37 ±  2%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.67            +0.1        0.73 ±  3%  perf-profile.self.cycles-pp.load_new_mm_cr3
      0.50 ±  2%      +0.1        0.57 ±  3%  perf-profile.self.cycles-pp.__switch_to
      0.45 ±  3%      +0.1        0.51        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.26            +0.1        0.34 ±  2%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.29            +0.1        0.38 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.56 ±  2%      +0.1        0.65 ±  3%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.39 ±  2%      +0.1        0.48 ±  2%  perf-profile.self.cycles-pp.__fget_light
      0.36            +0.1        0.46 ±  3%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.32 ±  2%      +0.1        0.42 ±  2%  perf-profile.self.cycles-pp.skb_release_data
      0.41 ±  2%      +0.1        0.51 ±  3%  perf-profile.self.cycles-pp.read_tsc
      0.65 ±  2%      +0.1        0.76 ±  4%  perf-profile.self.cycles-pp.__schedule
      0.33 ±  2%      +0.1        0.44 ±  3%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.41            +0.1        0.53 ±  4%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.00            +0.2        0.19 ±  7%  perf-profile.self.cycles-pp.__sk_mem_raise_allocated
      0.81 ±  2%      +0.2        1.03        perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.79            +0.3        1.06 ±  3%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.71            +0.4        2.11 ±  3%  perf-profile.self.cycles-pp.do_syscall_64
     15.64            +4.2       19.83 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string


                                                                                
                              netperf.Throughput_Mbps                           
                                                                                
  5000 +-+------------------------------------------------------------------+   
  4500 O-+       O   O                                                      |   
       |  O   O                                                             |   
  4000 +-+                                                                  |   
  3500 +-+                                                                  |   
       |..+...+..+...+..+...+..+...+..+...+     +...+..+...+..+...+..+...+..|   
  3000 +-+                                :     :                           |   
  2500 +-+                                 :   :                            |   
  2000 +-+                                 :   :                            |   
       |                                   :   :                            |   
  1500 +-+                                  : :                             |   
  1000 +-+                                  : :                             |   
       |                                    : :                             |   
   500 +-+                                   :                              |   
     0 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            netperf.Throughput_total_Mbps                       
                                                                                
  700000 +-+----------------------------------------------------------------+   
         O  O   O  O  O                                                     |   
  600000 +-+                                                                |   
         |                                                                  |   
  500000 +-++...+..+..+...+..+..  ..+..+...      +...+..+..+...+..+..+...+..|   
         |                      +.         +     :                          |   
  400000 +-+                               :    :                           |   
         |                                  :   :                           |   
  300000 +-+                                :   :                           |   
         |                                  :   :                           |   
  200000 +-+                                 : :                            |   
         |                                   : :                            |   
  100000 +-+                                 : :                            |   
         |                                    :                             |   
       0 +-+----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   netperf.workload                             
                                                                                
  3.5e+09 +-+---------------------------------------------------------------+   
          |                                                                 |   
    3e+09 O-+O   O  O  O                                                    |   
          |                                                                 |   
  2.5e+09 +-+                                                               |   
          |..+...+..+..+...+..+..+..+...+..+      +..+..+...+..+..+..+...+..|   
    2e+09 +-+                              :      :                         |   
          |                                 :    :                          |   
  1.5e+09 +-+                               :    :                          |   
          |                                 :   :                           |   
    1e+09 +-+                                :  :                           |   
          |                                  : :                            |   
    5e+08 +-+                                : :                            |   
          |                                   :                             |   
        0 +-+---------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              netperf.time.user_time                            
                                                                                
  1600 +-+-------O---O------------------------------------------------------+   
       O  O   O                                                             |   
  1400 +-+                                                                  |   
  1200 +-+                                                                  |   
       |                                                                    |   
  1000 +-++...+..+...                                                       |   
       |             +..+...+..+...+..+...+     +...+..+...+..+...+..+...+..|   
   800 +-+                                :     :                           |   
       |                                   :   :                            |   
   600 +-+                                 :   :                            |   
   400 +-+                                  :  :                            |   
       |                                    : :                             |   
   200 +-+                                  : :                             |   
       |                                     :                              |   
     0 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       netperf.time.voluntary_context_switches                  
                                                                                
  50000 +-+----O------O-----------------------------------------------------+   
  45000 +-+       O                                                         |   
        O  O                                                                |   
  40000 +-+                                                                 |   
  35000 +-+                                                                 |   
        |                                                                   |   
  30000 +-+                                                                 |   
  25000 +-+                                                                 |   
  20000 +-+                     +                             +             |   
        |             +        : :                           : :            |   
  15000 +-+          + :      :  :                           :  :           |   
  10000 +-+         +   :     :   :                         :   :           |   
        | .+...+.. +    :    :    :                       .+     :          |   
   5000 +-+       +      +..+      +...+..+..  ..+..+...+.        +..+...+..|   
      0 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       netperf.time.involuntary_context_switches                
                                                                                
  2.5e+09 +-+---------------------------------------------------------------+   
          |                                                                 |   
          |                                                                 |   
    2e+09 O-+O   O  O  O                                                    |   
          |                                                                 |   
          |                                                                 |   
  1.5e+09 +-++...+..+..+...+..+..+..+...+..+      +..+..+...+..+..+..+...+..|   
          |                                :      :                         |   
    1e+09 +-+                               :    :                          |   
          |                                 :    :                          |   
          |                                  :  :                           |   
    5e+08 +-+                                :  :                           |   
          |                                  : :                            |   
          |                                   ::                            |   
        0 +-+---------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp4: 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-2018-04-03.cgz/600s/lkp-csl-2sp4/TCP_STREAM/netperf/0x400001c

commit: 
  a8e11e5c56 ("sysctl: define proc_do_static_key()")
  ede61ca474 ("tcp: add tcp_rx_skb_cache sysctl")

a8e11e5c5611a9f7 ede61ca474a0348b975d9824565 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :1          100%           1:1     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          1:1         -100%            :1     dmesg.WARNING:at_ip__netif_receive_skb_core/0x
           :1          100%           1:1     dmesg.WARNING:stack_recursion





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.2.0-rc3-00259-gede61ca" of type "text/plain" (196402 bytes)

View attachment "job-script" of type "text/plain" (7730 bytes)

View attachment "job.yaml" of type "text/plain" (5002 bytes)

View attachment "reproduce" of type "text/plain" (8051 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ