lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Wed, 3 Mar 2021 17:00:33 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Honglei Wang <redsky110@...il.com>
Cc:     0day robot <lkp@...el.com>, LKML <linux-kernel@...r.kernel.org>,
        lkp@...ts.01.org, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com, davem@...emloft.net, edumazet@...gle.com,
        netdev@...r.kernel.org, redsky110@...il.com
Subject: [tcp]  ff0d41306d:  stress-ng.sockmany.ops_per_sec 338.2% improvement


Greeting,

FYI, we noticed a 338.2% improvement of stress-ng.sockmany.ops_per_sec due to commit:


commit: ff0d41306d8c7b91c21707559ef67f13e55b3406 ("[PATCH] tcp: avoid unnecessary loop if even ports are used up")
url: https://github.com/0day-ci/linux/commits/Honglei-Wang/tcp-avoid-unnecessary-loop-if-even-ports-are-used-up/20210220-200541
base: https://git.kernel.org/cgit/linux/kernel/git/davem/net.git 3af409ca278d4a8d50e91f9f7c4c33b175645cf3

in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
with following parameters:

	nr_threads: 100%
	disk: 1HDD
	testtime: 60s
	class: network
	test: sockmany
	cpufreq_governor: performance
	ucode: 0x5003006


In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.sockmany.ops_per_sec 572.1% improvement         |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory |
| test parameters  | class=os                                                             |
|                  | cpufreq_governor=performance                                         |
|                  | disk=1HDD                                                            |
|                  | fs=ext4                                                              |
|                  | nr_threads=10%                                                       |
|                  | test=sockmany                                                        |
|                  | testtime=60s                                                         |
|                  | ucode=0x5003006                                                      |
+------------------+----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  network/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/sockmany/stress-ng/60s/0x5003006

commit: 
  3af409ca27 ("net: enetc: fix destroyed phylink dereference during unbind")
  ff0d41306d ("tcp: avoid unnecessary loop if even ports are used up")

3af409ca278d4a8d ff0d41306d8c7b91c21707559ef 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   1968948          +337.5%    8614180 ±  2%  stress-ng.sockmany.ops
     32487          +338.2%     142359 ±  2%  stress-ng.sockmany.ops_per_sec
   1964201          +332.8%    8501580 ±  2%  stress-ng.time.involuntary_context_switches
      9018            -4.3%       8627        stress-ng.time.percent_of_cpu_this_job_got
      5651            -4.4%       5403        stress-ng.time.system_time
   2031507          +332.9%    8794405 ±  2%  stress-ng.time.voluntary_context_switches
    128079 ± 55%     +96.4%     251510 ± 31%  cpuidle.C1.usage
    335719 ± 51%     +89.1%     634765 ± 17%  cpuidle.C1E.usage
   2651720           +11.7%    2963054        vmstat.memory.cache
     63209          +324.8%     268490 ±  2%  vmstat.system.cs
      0.99            +0.1        1.12 ±  2%  mpstat.cpu.all.irq%
      1.56            +3.6        5.13 ±  2%  mpstat.cpu.all.soft%
      0.23            +0.1        0.36 ±  2%  mpstat.cpu.all.usr%
    287823 ± 12%    +128.3%     657171 ±  5%  numa-numastat.node0.local_node
    340321 ±  2%    +102.8%     690114        numa-numastat.node0.numa_hit
    340485 ± 10%     +94.9%     663744 ±  2%  numa-numastat.node1.local_node
    374587 ±  2%     +91.5%     717404 ±  2%  numa-numastat.node1.numa_hit
   1631924           +18.7%    1937517        meminfo.KReclaimable
   8668029           +12.5%    9749676        meminfo.Memused
   1631924           +18.7%    1937517        meminfo.SReclaimable
   4430317           +18.0%    5225972        meminfo.SUnreclaim
   6062242           +18.2%    7163489        meminfo.Slab
   8724017           +33.7%   11666875        meminfo.max_used_kB
    405237           +20.1%     486839        proc-vmstat.nr_slab_reclaimable
   1099299           +19.5%    1313379        proc-vmstat.nr_slab_unreclaimable
    747588           +92.3%    1437790        proc-vmstat.numa_hit
    660960          +104.4%    1351164 ±  2%  proc-vmstat.numa_local
     27094 ±  2%      -6.7%      25291 ±  4%  proc-vmstat.pgactivate
   2303828          +155.2%    5878562 ±  2%  proc-vmstat.pgalloc_normal
   1813432 ± 19%    +171.9%    4930494 ± 14%  proc-vmstat.pgfree
    122980 ±  7%     -48.5%      63324 ± 30%  sched_debug.cfs_rq:/.spread0.avg
    401.35 ±  3%     +21.9%     489.32 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    559.81 ±  4%     +15.1%     644.44 ±  5%  sched_debug.cpu.clock_task.stddev
     22854          +291.9%      89556 ±  3%  sched_debug.cpu.nr_switches.avg
     31316 ±  3%    +313.1%     129359 ±  5%  sched_debug.cpu.nr_switches.max
      9328 ± 28%    +521.5%      57972 ± 13%  sched_debug.cpu.nr_switches.min
      2292 ± 11%    +455.8%      12742 ±  7%  sched_debug.cpu.nr_switches.stddev
    204017           +23.8%     252546 ±  5%  numa-vmstat.node0.nr_slab_reclaimable
    555391           +22.5%     680395 ±  3%  numa-vmstat.node0.nr_slab_unreclaimable
    953619 ±  4%     +20.0%    1144710 ±  4%  numa-vmstat.node0.numa_hit
    899530 ±  3%     +21.1%    1089217 ±  7%  numa-vmstat.node0.numa_local
    201468           +14.7%     231159 ±  6%  numa-vmstat.node1.nr_slab_reclaimable
    544805           +14.6%     624239 ±  4%  numa-vmstat.node1.nr_slab_unreclaimable
    927332 ±  4%     +23.4%    1144293 ±  5%  numa-vmstat.node1.numa_hit
    738221 ±  3%     +29.6%     956639 ±  8%  numa-vmstat.node1.numa_local
    817215           +24.4%    1016626 ±  5%  numa-meminfo.node0.KReclaimable
   4372400 ±  2%     +15.4%    5046004 ±  4%  numa-meminfo.node0.MemUsed
    817215           +24.4%    1016626 ±  5%  numa-meminfo.node0.SReclaimable
   2224145           +23.2%    2739496 ±  3%  numa-meminfo.node0.SUnreclaim
   3041360           +23.5%    3756123 ±  3%  numa-meminfo.node0.Slab
    807671           +15.2%     930583 ±  6%  numa-meminfo.node1.KReclaimable
   4264803           +11.2%    4740532 ±  4%  numa-meminfo.node1.MemUsed
    807671           +15.2%     930583 ±  6%  numa-meminfo.node1.SReclaimable
   2184118           +15.0%    2511438 ±  4%  numa-meminfo.node1.SUnreclaim
   2991790           +15.0%    3442022 ±  4%  numa-meminfo.node1.Slab
    100.50 ± 38%     +69.3%     170.17 ± 41%  interrupts.CPU11.RES:Rescheduling_interrupts
     81.00 ± 10%    +101.2%     163.00 ± 60%  interrupts.CPU12.RES:Rescheduling_interrupts
    514.67 ±  3%     +28.9%     663.50 ± 37%  interrupts.CPU13.CAL:Function_call_interrupts
     83.17 ± 14%     +71.3%     142.50 ± 20%  interrupts.CPU15.RES:Rescheduling_interrupts
    467.33 ± 15%     +66.4%     777.83 ± 64%  interrupts.CPU18.CAL:Function_call_interrupts
     85.83 ± 16%    +441.6%     464.83 ±132%  interrupts.CPU18.RES:Rescheduling_interrupts
     82.67 ± 16%    +303.6%     333.67 ±114%  interrupts.CPU19.RES:Rescheduling_interrupts
      1557 ± 47%     -58.5%     646.50 ± 32%  interrupts.CPU2.CAL:Function_call_interrupts
     97.17 ± 29%     +45.1%     141.00 ± 21%  interrupts.CPU20.RES:Rescheduling_interrupts
     96.50 ± 23%     +53.9%     148.50 ± 38%  interrupts.CPU29.RES:Rescheduling_interrupts
    120.17 ± 12%    +586.7%     825.17 ± 87%  interrupts.CPU3.RES:Rescheduling_interrupts
     95.17 ± 14%    +213.8%     298.67 ±113%  interrupts.CPU30.RES:Rescheduling_interrupts
     83.17 ± 13%    +238.9%     281.83 ±120%  interrupts.CPU33.RES:Rescheduling_interrupts
     81.00 ± 12%     +53.1%     124.00 ± 27%  interrupts.CPU35.RES:Rescheduling_interrupts
     83.33 ± 22%     +51.8%     126.50 ± 10%  interrupts.CPU37.RES:Rescheduling_interrupts
     86.33 ± 29%     +89.0%     163.17 ± 39%  interrupts.CPU39.RES:Rescheduling_interrupts
     80.83 ± 16%    +324.7%     343.33 ±128%  interrupts.CPU40.RES:Rescheduling_interrupts
     90.33 ± 34%     +99.8%     180.50 ± 46%  interrupts.CPU45.RES:Rescheduling_interrupts
     84.33 ± 20%     +85.2%     156.17 ± 10%  interrupts.CPU47.RES:Rescheduling_interrupts
    103.17 ±  7%     +71.6%     177.00 ± 28%  interrupts.CPU48.RES:Rescheduling_interrupts
    105.83 ± 11%     +28.0%     135.50 ± 16%  interrupts.CPU49.RES:Rescheduling_interrupts
     88.83 ± 14%     +68.5%     149.67 ± 25%  interrupts.CPU54.RES:Rescheduling_interrupts
     82.17 ± 14%     +56.0%     128.17 ± 26%  interrupts.CPU55.RES:Rescheduling_interrupts
     83.00 ± 20%    +106.4%     171.33 ± 45%  interrupts.CPU57.RES:Rescheduling_interrupts
     79.83 ± 13%     +82.5%     145.67 ± 43%  interrupts.CPU59.RES:Rescheduling_interrupts
     87.17 ± 15%     +80.1%     157.00 ± 35%  interrupts.CPU6.RES:Rescheduling_interrupts
     87.83 ± 17%    +136.2%     207.50 ± 47%  interrupts.CPU61.RES:Rescheduling_interrupts
     86.00 ± 11%     +64.3%     141.33 ± 21%  interrupts.CPU62.RES:Rescheduling_interrupts
     86.00 ± 12%     +61.4%     138.83 ± 19%  interrupts.CPU65.RES:Rescheduling_interrupts
     84.33 ± 13%     +95.5%     164.83 ± 46%  interrupts.CPU66.RES:Rescheduling_interrupts
     94.50 ± 26%    +255.2%     335.67 ±120%  interrupts.CPU7.RES:Rescheduling_interrupts
     91.67 ±  9%     +29.8%     119.00 ± 18%  interrupts.CPU73.RES:Rescheduling_interrupts
     83.17 ± 12%    +223.4%     269.00 ± 70%  interrupts.CPU79.RES:Rescheduling_interrupts
     90.00 ± 11%     +80.6%     162.50 ± 37%  interrupts.CPU8.RES:Rescheduling_interrupts
     79.33 ± 12%     +50.2%     119.17 ± 26%  interrupts.CPU80.RES:Rescheduling_interrupts
     76.17 ± 11%    +324.1%     323.00 ±139%  interrupts.CPU88.RES:Rescheduling_interrupts
     78.67 ± 16%     +61.0%     126.67 ± 32%  interrupts.CPU93.RES:Rescheduling_interrupts
     72.67 ±  5%     +41.1%     102.50 ±  8%  interrupts.CPU94.RES:Rescheduling_interrupts
    137.17 ±  6%     +49.3%     204.83 ± 19%  interrupts.CPU95.RES:Rescheduling_interrupts
     10764 ± 10%     +49.7%      16119 ±  6%  interrupts.RES:Rescheduling_interrupts
   1537929           +19.1%    1830994        slabinfo.TCP.active_objs
    109941           +20.8%     132811        slabinfo.TCP.active_slabs
   1539186           +20.8%    1859366        slabinfo.TCP.num_objs
    109941           +20.8%     132811        slabinfo.TCP.num_slabs
   1613836           +17.9%    1903094        slabinfo.dentry.active_objs
     38427           +18.1%      45397        slabinfo.dentry.active_slabs
   1613982           +18.1%    1906701        slabinfo.dentry.num_objs
     38427           +18.1%      45397        slabinfo.dentry.num_slabs
     42469 ± 17%     +64.3%      69770 ±  5%  slabinfo.dmaengine-unmap-16.active_objs
      1010 ± 17%     +64.8%       1665 ±  5%  slabinfo.dmaengine-unmap-16.active_slabs
     42469 ± 17%     +64.8%      69981 ±  5%  slabinfo.dmaengine-unmap-16.num_objs
      1010 ± 17%     +64.8%       1665 ±  5%  slabinfo.dmaengine-unmap-16.num_slabs
   1551220           +19.3%    1849910        slabinfo.filp.active_objs
     48507           +21.4%      58887        slabinfo.filp.active_slabs
   1552268           +21.4%    1884408        slabinfo.filp.num_objs
     48507           +21.4%      58887        slabinfo.filp.num_slabs
    115939           +25.6%     145663        slabinfo.kmalloc-128.active_objs
      3624           +36.7%       4956        slabinfo.kmalloc-128.active_slabs
    115994           +36.7%     158610        slabinfo.kmalloc-128.num_objs
      3624           +36.7%       4956        slabinfo.kmalloc-128.num_slabs
   1567458           +19.7%    1877025        slabinfo.kmalloc-16.active_objs
      6125           +22.2%       7486        slabinfo.kmalloc-16.active_slabs
   1568110           +22.2%    1916606        slabinfo.kmalloc-16.num_objs
      6125           +22.2%       7486        slabinfo.kmalloc-16.num_slabs
      2454 ± 11%     -17.6%       2021 ± 14%  slabinfo.kmalloc-1k.active_slabs
     78563 ± 11%     -17.6%      64710 ± 14%  slabinfo.kmalloc-1k.num_objs
      2454 ± 11%     -17.6%       2021 ± 14%  slabinfo.kmalloc-1k.num_slabs
     55291           +23.7%      68386        slabinfo.kmalloc-256.active_objs
      1728           +35.2%       2336        slabinfo.kmalloc-256.active_slabs
     55314           +35.2%      74789        slabinfo.kmalloc-256.num_objs
      1728           +35.2%       2336        slabinfo.kmalloc-256.num_slabs
     88755           +18.9%     105533        slabinfo.kmalloc-512.active_objs
      2777           +31.2%       3644        slabinfo.kmalloc-512.active_slabs
     88898           +31.2%     116650        slabinfo.kmalloc-512.num_objs
      2777           +31.2%       3644        slabinfo.kmalloc-512.num_slabs
   1553431           +18.9%    1847560        slabinfo.lsm_file_cache.active_objs
      9138           +19.5%      10919        slabinfo.lsm_file_cache.active_slabs
   1553573           +19.5%    1856425        slabinfo.lsm_file_cache.num_objs
      9138           +19.5%      10919        slabinfo.lsm_file_cache.num_slabs
     60863 ±  9%     -17.0%      50516 ± 15%  slabinfo.skbuff_fclone_cache.active_objs
      2348 ±  9%     -17.5%       1936 ± 13%  slabinfo.skbuff_fclone_cache.active_slabs
     75158 ±  9%     -17.5%      61980 ± 13%  slabinfo.skbuff_fclone_cache.num_objs
      2348 ±  9%     -17.5%       1936 ± 13%  slabinfo.skbuff_fclone_cache.num_slabs
   1547186           +19.2%    1844129        slabinfo.sock_inode_cache.active_objs
     39692           +21.3%      48161        slabinfo.sock_inode_cache.active_slabs
   1548020           +21.3%    1878297        slabinfo.sock_inode_cache.num_objs
     39692           +21.3%      48161        slabinfo.sock_inode_cache.num_slabs
     36.65            -9.4%      33.21        perf-stat.i.MPKI
      2.20            -0.2        2.05        perf-stat.i.branch-miss-rate%
  1.94e+08            -6.8%  1.808e+08        perf-stat.i.branch-misses
 9.187e+08            -6.5%  8.586e+08        perf-stat.i.cache-misses
 1.649e+09            -7.8%  1.521e+09        perf-stat.i.cache-references
     66115          +324.9%     280914 ±  2%  perf-stat.i.context-switches
      5.66            -2.3%       5.53        perf-stat.i.cpi
    779.30 ± 57%    +178.0%       2166 ± 42%  perf-stat.i.cpu-migrations
    334.97            +6.9%     358.10        perf-stat.i.cycles-between-cache-misses
 1.389e+10            +3.9%  1.443e+10        perf-stat.i.dTLB-loads
      0.02 ±  8%      +0.0        0.04 ±  5%  perf-stat.i.dTLB-store-miss-rate%
   1519777 ±  8%    +110.9%    3205350 ±  5%  perf-stat.i.dTLB-store-misses
 6.481e+09           +14.1%  7.396e+09        perf-stat.i.dTLB-stores
     92.64            -1.5       91.17        perf-stat.i.iTLB-load-miss-rate%
   6311216          +388.0%   30801112 ±  3%  perf-stat.i.iTLB-load-misses
    751765 ±  5%    +263.0%    2728968 ±  3%  perf-stat.i.iTLB-loads
 4.442e+10            +3.7%  4.607e+10        perf-stat.i.instructions
     12789           -85.1%       1906 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.20            +3.8%       0.21        perf-stat.i.ipc
      0.31           -58.8%       0.13 ± 15%  perf-stat.i.metric.K/sec
    332.55            +4.1%     346.29        perf-stat.i.metric.M/sec
     40.59            +3.3       43.89        perf-stat.i.node-load-miss-rate%
 3.012e+08            -4.0%  2.891e+08        perf-stat.i.node-load-misses
 4.622e+08           -17.2%  3.828e+08        perf-stat.i.node-loads
     94.89            -6.3       88.58        perf-stat.i.node-store-miss-rate%
 1.016e+08 ±  3%      +8.2%  1.099e+08        perf-stat.i.node-store-misses
   2293688 ±  3%    +338.0%   10046935 ±  2%  perf-stat.i.node-stores
     37.12           -11.1%      33.01        perf-stat.overall.MPKI
      2.15            -0.2        1.99        perf-stat.overall.branch-miss-rate%
      5.74            -3.8%       5.52        perf-stat.overall.cpi
    277.31            +6.9%     296.39        perf-stat.overall.cycles-between-cache-misses
      0.02 ±  8%      +0.0        0.04 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
     89.35            +2.5       91.87        perf-stat.overall.iTLB-load-miss-rate%
      7035           -78.7%       1496 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.17            +3.9%       0.18        perf-stat.overall.ipc
     39.46            +3.5       43.01        perf-stat.overall.node-load-miss-rate%
     97.79            -6.2       91.63        perf-stat.overall.node-store-miss-rate%
 1.909e+08            -6.7%   1.78e+08        perf-stat.ps.branch-misses
 9.042e+08            -6.6%  8.444e+08        perf-stat.ps.cache-misses
 1.623e+09            -7.8%  1.497e+09        perf-stat.ps.cache-references
     65106          +324.6%     276427 ±  2%  perf-stat.ps.context-switches
    767.80 ± 57%    +178.0%       2134 ± 42%  perf-stat.ps.cpu-migrations
 1.367e+10            +3.9%  1.421e+10        perf-stat.ps.dTLB-loads
   1497010 ±  8%    +110.7%    3153486 ±  5%  perf-stat.ps.dTLB-store-misses
 6.377e+09           +14.2%  7.281e+09        perf-stat.ps.dTLB-stores
   6213710          +388.2%   30337117 ±  3%  perf-stat.ps.iTLB-load-misses
    740380 ±  5%    +262.9%    2687156 ±  4%  perf-stat.ps.iTLB-loads
 4.371e+10            +3.7%  4.535e+10        perf-stat.ps.instructions
 2.965e+08            -4.2%  2.841e+08        perf-stat.ps.node-load-misses
 4.549e+08           -17.3%  3.764e+08        perf-stat.ps.node-loads
  99918589 ±  3%      +8.3%  1.082e+08        perf-stat.ps.node-store-misses
   2258701 ±  3%    +337.8%    9888880 ±  3%  perf-stat.ps.node-stores
 2.787e+12            +3.7%  2.891e+12        perf-stat.total.instructions
     75.43           -13.7       61.68 ±  3%  perf-profile.calltrace.cycles-pp.__inet_check_established.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
     23.97            -6.2       17.77        perf-profile.calltrace.cycles-pp._raw_spin_lock.__inet_check_established.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect
     99.12            -5.4       93.67        perf-profile.calltrace.cycles-pp.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
     99.47            -4.8       94.65        perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
     99.67            -3.7       95.92        perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
     99.67            -3.7       95.93        perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     99.67            -3.7       95.95        perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     99.67            -3.7       95.95        perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     99.87            -1.3       98.59        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.08            -0.4        0.70 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__inet_check_established.__inet_hash_connect.tcp_v4_connect
     99.94            -0.3       99.60        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.00            -0.2        0.76 ±  2%  perf-profile.calltrace.cycles-pp.inet_ehashfn.__inet_check_established.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect
      0.00            +0.7        0.65 ± 17%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.inet_shutdown
      0.00            +0.7        0.68 ± 15%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect
      0.00            +0.7        0.70 ± 18%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.inet_shutdown.__sys_shutdown
      0.00            +0.7        0.70 ± 15%  perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
      0.00            +0.7        0.73 ± 15%  perf-profile.calltrace.cycles-pp.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      0.00            +0.8        0.79 ± 18%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.inet_shutdown.__sys_shutdown.__x64_sys_shutdown
      0.00            +0.8        0.79 ± 18%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.inet_shutdown.__sys_shutdown.__x64_sys_shutdown.do_syscall_64
      0.00            +0.9        0.91 ± 19%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.94 ± 20%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        0.98 ± 19%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        0.99 ± 19%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        1.05 ± 15%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect
      0.00            +1.1        1.09 ± 19%  perf-profile.calltrace.cycles-pp.inet_shutdown.__sys_shutdown.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.14 ± 19%  perf-profile.calltrace.cycles-pp.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.14 ± 19%  perf-profile.calltrace.cycles-pp.__sys_shutdown.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.2        1.24 ± 15%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect
      0.00            +1.2        1.24 ± 15%  perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      5.67 ± 10%      +6.2       11.87 ±  6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect
     16.93            +8.7       25.64 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
     75.61           -13.8       61.82 ±  3%  perf-profile.children.cycles-pp.__inet_check_established
     24.09            -6.0       18.12 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock
     99.32            -5.5       93.78        perf-profile.children.cycles-pp.__inet_hash_connect
     99.47            -4.8       94.65        perf-profile.children.cycles-pp.tcp_v4_connect
     99.67            -3.7       95.92        perf-profile.children.cycles-pp.__inet_stream_connect
     99.67            -3.7       95.93        perf-profile.children.cycles-pp.inet_stream_connect
     99.67            -3.7       95.95        perf-profile.children.cycles-pp.__x64_sys_connect
     99.67            -3.7       95.95        perf-profile.children.cycles-pp.__sys_connect
     99.90            -1.3       98.63        perf-profile.children.cycles-pp.do_syscall_64
     99.97            -0.3       99.64        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.83            -0.3        0.52 ±  4%  perf-profile.children.cycles-pp.___might_sleep
      1.02            -0.2        0.79 ±  2%  perf-profile.children.cycles-pp.inet_ehashfn
      0.19 ±  2%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.rcu_all_qs
      0.10 ±  4%      -0.0        0.07 ±  9%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.14 ±  3%      -0.0        0.11 ±  6%  perf-profile.children.cycles-pp._raw_spin_unlock_bh
      0.04 ± 44%      +0.0        0.07 ±  8%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.ip_send_skb
      0.00            +0.1        0.06 ± 18%  perf-profile.children.cycles-pp.__x64_sys_close
      0.00            +0.1        0.06 ± 18%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.inode_init_always
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.__slab_free
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.update_curr
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.lock_sock_nested
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.security_sk_alloc
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.tcp_write_queue_purge
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.tcp_recvmsg
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.07 ± 19%  perf-profile.children.cycles-pp.tcp_write_timer
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.1        0.07 ± 21%  perf-profile.children.cycles-pp.inet_recvmsg
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.netif_rx
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.netif_rx_internal
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.00            +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.__kfree_skb
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.tcp_v4_send_synack
      0.00            +0.1        0.09 ± 35%  perf-profile.children.cycles-pp.drain_obj_stock
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.sk_reset_timer
      0.00            +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.preempt_schedule_common
      0.00            +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
      0.00            +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.__sys_recvfrom
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.mod_timer
      0.00            +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.tcp_data_queue
      0.00            +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.___slab_alloc
      0.00            +0.1        0.10 ± 20%  perf-profile.children.cycles-pp.sock_alloc_inode
      0.00            +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.ip_route_output_flow
      0.00            +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.tcp_set_state
      0.00            +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.__slab_alloc
      0.00            +0.1        0.11 ± 16%  perf-profile.children.cycles-pp.schedule
      0.00            +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.schedule_timeout
      0.00            +0.1        0.12 ± 17%  perf-profile.children.cycles-pp.sk_clone_lock
      0.00            +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.tcp_reset
      0.00            +0.1        0.12 ± 17%  perf-profile.children.cycles-pp.inet_csk_clone_lock
      0.00            +0.1        0.12 ± 17%  perf-profile.children.cycles-pp.tcp_send_fin
      0.02 ±141%      +0.1        0.14 ± 18%  perf-profile.children.cycles-pp.tcp_delack_timer_handler
      0.00            +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.00            +0.1        0.13 ± 15%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      0.02 ±141%      +0.1        0.15 ± 19%  perf-profile.children.cycles-pp.tcp_delack_timer
      0.00            +0.1        0.13 ± 18%  perf-profile.children.cycles-pp.tcp_done
      0.00            +0.1        0.14 ± 17%  perf-profile.children.cycles-pp.tcp_v4_send_reset
      0.00            +0.1        0.14 ± 18%  perf-profile.children.cycles-pp.tcp_create_openreq_child
      0.00            +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.refill_obj_stock
      0.00            +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      0.00            +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.ip_send_unicast_reply
      0.00            +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.alloc_empty_file
      0.00            +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.__alloc_file
      0.00            +0.1        0.14 ± 18%  perf-profile.children.cycles-pp.tcp_validate_incoming
      0.00            +0.1        0.15 ± 19%  perf-profile.children.cycles-pp.alloc_file
      0.07 ± 13%      +0.2        0.22 ± 17%  perf-profile.children.cycles-pp.call_timer_fn
      0.00            +0.2        0.15 ± 17%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.00            +0.2        0.15 ± 17%  perf-profile.children.cycles-pp.loopback_xmit
      0.00            +0.2        0.16 ± 19%  perf-profile.children.cycles-pp.sk_alloc
      0.00            +0.2        0.16 ± 17%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +0.2        0.16 ± 15%  perf-profile.children.cycles-pp.inet_csk_accept
      0.00            +0.2        0.16 ± 20%  perf-profile.children.cycles-pp.__alloc_skb
      0.00            +0.2        0.16 ± 18%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.08 ± 12%      +0.2        0.24 ± 17%  perf-profile.children.cycles-pp.run_timer_softirq
      0.00            +0.2        0.17 ± 17%  perf-profile.children.cycles-pp.alloc_inode
      0.00            +0.2        0.17 ± 15%  perf-profile.children.cycles-pp.__wake_up_common
      0.00            +0.2        0.17 ± 15%  perf-profile.children.cycles-pp.inet_accept
      0.00            +0.2        0.18 ± 17%  perf-profile.children.cycles-pp.sk_prot_alloc
      0.00            +0.2        0.18 ± 17%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.00            +0.2        0.19 ± 15%  perf-profile.children.cycles-pp.sock_def_readable
      0.00            +0.2        0.19 ± 45%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.00            +0.2        0.19 ± 16%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.00            +0.2        0.19 ± 45%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.00            +0.2        0.19 ± 17%  perf-profile.children.cycles-pp.sock_alloc
      0.00            +0.2        0.20 ± 14%  perf-profile.children.cycles-pp.__sched_text_start
      0.00            +0.2        0.20 ± 19%  perf-profile.children.cycles-pp.inet_create
      0.00            +0.2        0.20 ± 42%  perf-profile.children.cycles-pp.ret_from_fork
      0.00            +0.2        0.20 ± 42%  perf-profile.children.cycles-pp.kthread
      0.00            +0.2        0.20 ± 16%  perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
      0.00            +0.2        0.21 ± 15%  perf-profile.children.cycles-pp.tcp_conn_request
      0.00            +0.2        0.22 ± 16%  perf-profile.children.cycles-pp.tcp_ack
      0.00            +0.2        0.22 ± 26%  perf-profile.children.cycles-pp.__sk_destruct
      0.00            +0.2        0.22 ± 16%  perf-profile.children.cycles-pp.tcp_child_process
      0.00            +0.2        0.22 ± 20%  perf-profile.children.cycles-pp.__dentry_kill
      0.00            +0.2        0.23 ± 16%  perf-profile.children.cycles-pp.tcp_check_req
      0.00            +0.2        0.23 ± 18%  perf-profile.children.cycles-pp.alloc_file_pseudo
      0.00            +0.2        0.23 ± 19%  perf-profile.children.cycles-pp.sock_alloc_file
      0.00            +0.3        0.27 ± 17%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.00            +0.3        0.29 ± 18%  perf-profile.children.cycles-pp.__tcp_close
      0.00            +0.3        0.29 ± 20%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.03 ±100%      +0.3        0.32 ± 28%  perf-profile.children.cycles-pp.rcu_core
      0.77 ±  5%      +0.3        1.07 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.00            +0.3        0.30 ± 18%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
      0.69 ±  7%      +0.3        0.99 ±  9%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.03 ± 99%      +0.3        0.33 ± 17%  perf-profile.children.cycles-pp.__sys_sendto
      0.03 ± 99%      +0.3        0.33 ± 17%  perf-profile.children.cycles-pp.__x64_sys_sendto
      0.00            +0.3        0.31 ± 19%  perf-profile.children.cycles-pp.tcp_sendmsg
      0.00            +0.3        0.32 ± 29%  perf-profile.children.cycles-pp.rcu_do_batch
      0.00            +0.3        0.32 ± 16%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.00            +0.3        0.32 ± 17%  perf-profile.children.cycles-pp.tcp_close
      0.00            +0.3        0.32 ± 18%  perf-profile.children.cycles-pp.__sock_create
      0.00            +0.3        0.32 ± 18%  perf-profile.children.cycles-pp.sock_sendmsg
      0.00            +0.3        0.34 ± 18%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.06 ±  8%      +0.4        0.41 ± 17%  perf-profile.children.cycles-pp.__sys_accept4_file
      0.06 ±  8%      +0.4        0.42 ± 17%  perf-profile.children.cycles-pp.__sys_accept4
      0.06 ±  8%      +0.4        0.42 ± 16%  perf-profile.children.cycles-pp.__x64_sys_accept
      0.00            +0.4        0.39 ± 16%  perf-profile.children.cycles-pp.tcp_rcv_established
      0.05            +0.4        0.46 ± 19%  perf-profile.children.cycles-pp.__x64_sys_socket
      0.05            +0.4        0.46 ± 19%  perf-profile.children.cycles-pp.__sys_socket
      0.00            +0.4        0.45 ± 30%  perf-profile.children.cycles-pp.kmem_cache_free
      0.00            +0.5        0.49 ± 20%  perf-profile.children.cycles-pp.inet_release
      0.00            +0.5        0.53 ± 20%  perf-profile.children.cycles-pp.__sock_release
      0.00            +0.5        0.53 ± 20%  perf-profile.children.cycles-pp.sock_close
      0.12 ±  3%      +0.6        0.70 ± 15%  perf-profile.children.cycles-pp.tcp_connect
      0.12 ±  3%      +0.6        0.73 ± 15%  perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process
      0.06 ±  6%      +0.9        0.91 ± 19%  perf-profile.children.cycles-pp.__fput
      0.06 ±  7%      +0.9        0.94 ± 20%  perf-profile.children.cycles-pp.task_work_run
      0.07 ±  7%      +0.9        0.98 ± 19%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.07 ±  5%      +0.9        0.99 ± 19%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.00            +1.1        1.09 ± 19%  perf-profile.children.cycles-pp.inet_shutdown
      0.00            +1.1        1.14 ± 19%  perf-profile.children.cycles-pp.__x64_sys_shutdown
      0.00            +1.1        1.14 ± 19%  perf-profile.children.cycles-pp.__sys_shutdown
      0.02 ±141%      +1.2        1.19 ± 17%  perf-profile.children.cycles-pp.tcp_write_xmit
      0.02 ±141%      +1.2        1.19 ± 17%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
      0.22 ±  3%      +1.2        1.43 ± 16%  perf-profile.children.cycles-pp.tcp_rcv_state_process
      0.19 ±  3%      +1.2        1.40 ± 16%  perf-profile.children.cycles-pp.__release_sock
      0.19            +1.3        1.44 ± 16%  perf-profile.children.cycles-pp.release_sock
      0.22 ±  4%      +1.6        1.82 ± 16%  perf-profile.children.cycles-pp.tcp_v4_rcv
      0.23 ±  3%      +1.6        1.84 ± 16%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      0.23 ±  3%      +1.6        1.85 ± 16%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      0.23 ±  4%      +1.6        1.86 ± 16%  perf-profile.children.cycles-pp.ip_local_deliver
      0.25 ±  3%      +1.6        1.89 ± 16%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      0.24 ±  3%      +1.7        1.91 ± 16%  perf-profile.children.cycles-pp.ip_rcv
      0.91 ±  2%      +1.7        2.62 ± 13%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.24 ±  3%      +1.7        1.97 ± 16%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      0.25 ±  3%      +1.8        2.02 ± 16%  perf-profile.children.cycles-pp.process_backlog
      0.26 ±  4%      +1.8        2.06 ± 16%  perf-profile.children.cycles-pp.net_rx_action
      0.40 ±  6%      +1.9        2.26 ± 15%  perf-profile.children.cycles-pp.do_softirq
      0.23 ±  3%      +1.9        2.12 ± 16%  perf-profile.children.cycles-pp.ip_finish_output2
      0.24 ±  2%      +2.0        2.23 ± 16%  perf-profile.children.cycles-pp.ip_output
      0.25 ±  3%      +2.1        2.33 ± 16%  perf-profile.children.cycles-pp.__ip_queue_xmit
      1.07 ±  6%      +2.1        3.19 ± 12%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      0.39 ±  6%      +2.1        2.52 ± 16%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.28 ±  2%      +2.2        2.52 ± 16%  perf-profile.children.cycles-pp.__tcp_transmit_skb
      0.39 ±  6%      +2.3        2.69 ± 17%  perf-profile.children.cycles-pp.__softirqentry_text_start
      6.76 ±  8%      +5.8       12.61 ±  6%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     17.06            +8.8       25.88 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_bh
     50.21            -7.3       42.93 ±  5%  perf-profile.self.cycles-pp.__inet_check_established
     22.80            -5.6       17.24 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      0.79            -0.3        0.49 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.96            -0.2        0.74 ±  2%  perf-profile.self.cycles-pp.inet_ehashfn
      0.51 ±  2%      -0.2        0.36 ±  4%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.14 ±  4%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.rcu_all_qs
      0.18 ±  2%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp._cond_resched
      0.11 ±  4%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp._raw_spin_unlock_bh
      0.00            +0.1        0.06 ± 18%  perf-profile.self.cycles-pp.__slab_free
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
      0.00            +0.1        0.07 ± 13%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.00            +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.00            +0.1        0.10 ± 19%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.00            +0.1        0.10 ± 17%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.00            +0.2        0.23 ± 29%  perf-profile.self.cycles-pp.kmem_cache_free
      0.00            +0.3        0.31 ± 16%  perf-profile.self.cycles-pp.__inet_lookup_established
     11.30 ±  4%      +2.6       13.91 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      6.66 ±  8%      +5.8       12.48 ±  6%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     65636 ±  3%    +312.0%     270405 ±  9%  softirqs.CPU0.NET_RX
     14402 ± 14%     +63.3%      23512 ± 10%  softirqs.CPU0.RCU
      6255 ±  8%    +142.9%      15196 ±  2%  softirqs.CPU0.TIMER
     59711 ± 19%    +340.7%     263124 ± 12%  softirqs.CPU1.NET_RX
     11149 ±  7%     +72.9%      19274 ±  4%  softirqs.CPU1.RCU
      5340 ±  3%    +156.6%      13702        softirqs.CPU1.TIMER
     65266          +293.8%     256991 ±  7%  softirqs.CPU10.NET_RX
     10398           +83.4%      19067 ±  3%  softirqs.CPU10.RCU
      5416 ±  2%    +152.3%      13667        softirqs.CPU10.TIMER
     64454          +325.4%     274194 ± 14%  softirqs.CPU11.NET_RX
     10223           +81.8%      18581 ±  3%  softirqs.CPU11.RCU
      5435 ±  2%    +153.5%      13780        softirqs.CPU11.TIMER
     65141          +315.4%     270604 ± 18%  softirqs.CPU12.NET_RX
     10144           +85.0%      18772 ± 10%  softirqs.CPU12.RCU
      5399 ±  2%    +153.2%      13671 ±  6%  softirqs.CPU12.TIMER
     65486 ±  2%    +306.8%     266395 ±  8%  softirqs.CPU13.NET_RX
     10403 ±  2%     +71.3%      17824 ±  2%  softirqs.CPU13.RCU
      5418 ±  2%    +161.2%      14154 ±  4%  softirqs.CPU13.TIMER
     65166          +300.6%     261036 ± 14%  softirqs.CPU14.NET_RX
     10202 ±  2%     +86.7%      19052 ±  5%  softirqs.CPU14.RCU
      5464 ±  3%    +149.7%      13643        softirqs.CPU14.TIMER
     64647          +358.7%     296569 ± 19%  softirqs.CPU15.NET_RX
     10540 ±  4%     +78.6%      18821 ±  6%  softirqs.CPU15.RCU
      5649 ±  8%    +139.6%      13536        softirqs.CPU15.TIMER
     65031 ±  2%    +368.5%     304698 ±  7%  softirqs.CPU16.NET_RX
     10759 ±  4%     +82.6%      19645 ±  2%  softirqs.CPU16.RCU
      5573 ±  6%    +147.1%      13773 ±  2%  softirqs.CPU16.TIMER
     64827          +326.1%     276213 ±  4%  softirqs.CPU17.NET_RX
     10908 ±  6%     +79.5%      19584 ±  5%  softirqs.CPU17.RCU
      5423          +157.4%      13959        softirqs.CPU17.TIMER
     64871          +295.4%     256498 ±  9%  softirqs.CPU18.NET_RX
     10661 ±  2%     +77.0%      18870 ±  6%  softirqs.CPU18.RCU
      5394          +155.8%      13798        softirqs.CPU18.TIMER
     65669 ±  3%    +290.1%     256203 ± 17%  softirqs.CPU19.NET_RX
     10688 ±  2%     +76.4%      18855 ±  5%  softirqs.CPU19.RCU
      5441          +150.3%      13623        softirqs.CPU19.TIMER
     64660          +366.3%     301532 ± 18%  softirqs.CPU2.NET_RX
     10936           +81.0%      19792 ±  5%  softirqs.CPU2.RCU
      5415 ±  2%    +157.5%      13946        softirqs.CPU2.TIMER
     65327 ±  2%    +326.2%     278443 ± 17%  softirqs.CPU20.NET_RX
     10789 ±  3%     +81.6%      19596 ±  5%  softirqs.CPU20.RCU
      5398 ±  2%    +153.2%      13669 ±  2%  softirqs.CPU20.TIMER
     65907 ±  2%    +320.0%     276814 ±  8%  softirqs.CPU21.NET_RX
     10628           +88.7%      20055 ±  6%  softirqs.CPU21.RCU
      5509 ±  2%    +152.0%      13883 ±  3%  softirqs.CPU21.TIMER
     65040          +276.9%     245159 ± 10%  softirqs.CPU22.NET_RX
     10521           +71.6%      18057 ±  4%  softirqs.CPU22.RCU
      5403 ±  2%    +157.8%      13929 ±  8%  softirqs.CPU22.TIMER
     65297 ±  2%    +313.0%     269696 ± 13%  softirqs.CPU23.NET_RX
     11407 ± 13%     +68.1%      19180 ±  4%  softirqs.CPU23.RCU
      5622 ±  7%    +142.1%      13614 ±  2%  softirqs.CPU23.TIMER
     64721          +332.4%     279830 ± 19%  softirqs.CPU24.NET_RX
     11583 ±  3%     +70.3%      19726 ±  5%  softirqs.CPU24.RCU
      5785 ±  9%    +134.8%      13587 ±  5%  softirqs.CPU24.TIMER
     65042          +332.9%     281590 ±  7%  softirqs.CPU25.NET_RX
     11141 ±  2%     +83.3%      20423 ± 10%  softirqs.CPU25.RCU
      5353          +156.2%      13713 ±  5%  softirqs.CPU25.TIMER
     58802 ± 23%    +406.0%     297551 ± 13%  softirqs.CPU26.NET_RX
     10389 ±  8%     +94.9%      20252 ±  7%  softirqs.CPU26.RCU
      5285          +153.4%      13396 ±  4%  softirqs.CPU26.TIMER
     64806          +419.3%     336531 ± 16%  softirqs.CPU27.NET_RX
     10607           +80.6%      19153 ±  5%  softirqs.CPU27.RCU
      5328          +151.5%      13399 ±  4%  softirqs.CPU27.TIMER
     64644          +342.6%     286130 ± 13%  softirqs.CPU28.NET_RX
     10719 ±  4%     +84.7%      19799 ±  3%  softirqs.CPU28.RCU
      5355          +143.5%      13039 ±  4%  softirqs.CPU28.TIMER
     64931          +374.6%     308152 ± 18%  softirqs.CPU29.NET_RX
     10407           +85.0%      19250 ±  3%  softirqs.CPU29.RCU
      5411          +146.7%      13350 ±  4%  softirqs.CPU29.TIMER
     64538          +347.3%     288687 ±  9%  softirqs.CPU3.NET_RX
     10678           +84.6%      19712 ±  4%  softirqs.CPU3.RCU
      5408 ±  2%    +165.7%      14368 ±  5%  softirqs.CPU3.TIMER
     65007 ±  2%    +423.5%     340342 ± 17%  softirqs.CPU30.NET_RX
     10651 ±  5%     +84.5%      19657 ±  6%  softirqs.CPU30.RCU
      5495 ±  3%    +146.3%      13533 ±  4%  softirqs.CPU30.TIMER
     64984          +342.4%     287519 ± 14%  softirqs.CPU31.NET_RX
     10464           +75.7%      18390 ±  5%  softirqs.CPU31.RCU
      5387          +146.0%      13250 ±  3%  softirqs.CPU31.TIMER
     59006 ± 23%    +384.5%     285909 ± 14%  softirqs.CPU32.NET_RX
      9705 ± 11%     +94.1%      18835 ±  7%  softirqs.CPU32.RCU
      5208 ±  2%    +171.6%      14147 ± 16%  softirqs.CPU32.TIMER
     64860          +316.2%     269979 ± 10%  softirqs.CPU33.NET_RX
     10061 ±  2%     +89.4%      19058 ±  2%  softirqs.CPU33.RCU
      5300          +150.6%      13280 ±  3%  softirqs.CPU33.TIMER
     64809          +307.7%     264248 ± 17%  softirqs.CPU34.NET_RX
      9968           +87.1%      18652 ±  3%  softirqs.CPU34.RCU
      5364          +146.0%      13195 ±  2%  softirqs.CPU34.TIMER
     64861          +325.8%     276161 ± 12%  softirqs.CPU35.NET_RX
     10069           +85.1%      18640 ±  3%  softirqs.CPU35.RCU
      5389          +148.1%      13367 ±  4%  softirqs.CPU35.TIMER
     59089 ± 23%    +420.1%     307315 ± 18%  softirqs.CPU36.NET_RX
      9648 ±  9%    +100.3%      19326 ± 10%  softirqs.CPU36.RCU
      5253 ±  2%    +155.6%      13427 ±  4%  softirqs.CPU36.TIMER
     65133          +355.5%     296651 ± 11%  softirqs.CPU37.NET_RX
     10020 ±  2%     +88.4%      18873 ±  4%  softirqs.CPU37.RCU
      5439 ±  5%    +144.0%      13271 ±  5%  softirqs.CPU37.TIMER
     65044          +315.0%     269907 ± 16%  softirqs.CPU38.NET_RX
      9951 ±  2%     +86.5%      18556 ±  6%  softirqs.CPU38.RCU
      5334          +142.3%      12927 ±  7%  softirqs.CPU38.TIMER
     61324 ± 12%    +339.7%     269630 ± 15%  softirqs.CPU39.NET_RX
      9891 ±  5%     +91.4%      18927 ±  3%  softirqs.CPU39.RCU
      5253 ±  4%    +151.0%      13184 ±  2%  softirqs.CPU39.TIMER
     59536 ± 20%    +342.2%     263268 ±  3%  softirqs.CPU4.NET_RX
     10248 ± 13%     +81.1%      18560 ±  5%  softirqs.CPU4.RCU
      5306 ±  5%    +157.0%      13639 ±  2%  softirqs.CPU4.TIMER
     65051          +339.9%     286131 ± 12%  softirqs.CPU40.NET_RX
      9993 ±  2%     +88.1%      18801 ±  3%  softirqs.CPU40.RCU
      5428 ±  3%    +147.6%      13437 ±  4%  softirqs.CPU40.TIMER
     66089 ±  2%    +332.8%     286027 ± 14%  softirqs.CPU41.NET_RX
     10483 ±  6%     +82.3%      19111 ±  7%  softirqs.CPU41.RCU
      5525 ±  4%    +138.2%      13161 ±  5%  softirqs.CPU41.TIMER
     64979          +345.8%     289650 ± 11%  softirqs.CPU42.NET_RX
     10008 ±  3%     +86.5%      18661 ±  3%  softirqs.CPU42.RCU
      5337          +149.4%      13309 ±  3%  softirqs.CPU42.TIMER
     64234          +321.8%     270970 ±  9%  softirqs.CPU43.NET_RX
      9899 ±  2%     +86.3%      18441 ±  5%  softirqs.CPU43.RCU
      5326          +149.9%      13307 ±  3%  softirqs.CPU43.TIMER
     65083          +376.2%     309898 ± 18%  softirqs.CPU44.NET_RX
      9958 ±  2%     +93.8%      19296 ±  6%  softirqs.CPU44.RCU
      5346 ±  2%    +148.1%      13261 ±  4%  softirqs.CPU44.TIMER
     65076          +403.6%     327744 ± 11%  softirqs.CPU45.NET_RX
      9942 ±  3%     +91.4%      19027 ± 10%  softirqs.CPU45.RCU
      5299          +152.5%      13383 ±  6%  softirqs.CPU45.TIMER
     65682 ±  2%    +358.2%     300968 ± 18%  softirqs.CPU46.NET_RX
      9760 ±  4%     +96.3%      19161 ±  5%  softirqs.CPU46.RCU
      5350          +151.9%      13474 ±  4%  softirqs.CPU46.TIMER
     64475          +332.7%     279008 ± 13%  softirqs.CPU47.NET_RX
     10005 ±  3%     +83.6%      18368 ±  6%  softirqs.CPU47.RCU
      5329          +149.2%      13279 ±  2%  softirqs.CPU47.TIMER
     64759 ±  2%    +264.5%     236058 ± 15%  softirqs.CPU48.NET_RX
     10354 ±  2%     +67.5%      17341 ±  9%  softirqs.CPU48.RCU
      5430 ±  2%    +152.5%      13710 ±  3%  softirqs.CPU48.TIMER
     64904          +327.1%     277209 ± 13%  softirqs.CPU49.NET_RX
     10365 ±  2%     +80.1%      18667 ±  3%  softirqs.CPU49.RCU
      5416 ±  2%    +154.0%      13760        softirqs.CPU49.TIMER
     64809          +342.6%     286815 ± 11%  softirqs.CPU5.NET_RX
     10495           +81.1%      19004 ±  5%  softirqs.CPU5.RCU
      5424 ±  2%    +157.0%      13941 ±  2%  softirqs.CPU5.TIMER
     64092          +323.0%     271142 ± 12%  softirqs.CPU50.NET_RX
     10243 ±  5%     +83.8%      18830 ±  6%  softirqs.CPU50.RCU
      5394          +152.0%      13594        softirqs.CPU50.TIMER
     64623          +339.4%     283964 ± 11%  softirqs.CPU51.NET_RX
     10338           +80.4%      18654 ±  2%  softirqs.CPU51.RCU
      5699 ±  7%    +160.1%      14824 ± 14%  softirqs.CPU51.TIMER
     65332          +292.6%     256522 ± 10%  softirqs.CPU52.NET_RX
     10252           +82.1%      18671 ±  4%  softirqs.CPU52.RCU
      5513 ±  3%    +145.5%      13538        softirqs.CPU52.TIMER
     65157 ±  2%    +300.1%     260686 ± 13%  softirqs.CPU53.NET_RX
     10258 ±  2%     +76.9%      18152 ±  4%  softirqs.CPU53.RCU
      5414          +151.9%      13640 ±  2%  softirqs.CPU53.TIMER
     64777          +325.0%     275316 ± 10%  softirqs.CPU54.NET_RX
     10325 ±  4%     +72.6%      17826 ±  3%  softirqs.CPU54.RCU
      5664 ±  9%    +143.8%      13808 ±  3%  softirqs.CPU54.TIMER
     65262          +274.9%     244693 ±  9%  softirqs.CPU55.NET_RX
     10069           +85.1%      18641 ±  3%  softirqs.CPU55.RCU
      5455          +153.3%      13820 ±  2%  softirqs.CPU55.TIMER
     64669          +334.1%     280750 ± 18%  softirqs.CPU56.NET_RX
     10211 ±  6%     +82.9%      18671 ±  7%  softirqs.CPU56.RCU
      5409 ±  2%    +153.9%      13736 ±  3%  softirqs.CPU56.TIMER
     65109          +299.3%     259999 ± 10%  softirqs.CPU57.NET_RX
     10178 ±  2%     +83.7%      18697 ±  2%  softirqs.CPU57.RCU
      5421 ±  2%    +152.1%      13665        softirqs.CPU57.TIMER
     65527 ±  2%    +298.4%     261032 ±  7%  softirqs.CPU58.NET_RX
     10138           +83.3%      18578 ±  3%  softirqs.CPU58.RCU
      5444 ±  2%    +155.5%      13911 ±  3%  softirqs.CPU58.TIMER
     65194          +315.1%     270633 ±  4%  softirqs.CPU59.NET_RX
     10195           +79.5%      18297 ±  4%  softirqs.CPU59.RCU
      5392 ±  2%    +154.4%      13719        softirqs.CPU59.TIMER
     64608          +301.6%     259488 ± 16%  softirqs.CPU6.NET_RX
     10235           +79.0%      18323 ±  3%  softirqs.CPU6.RCU
      5399 ±  2%    +170.0%      14579 ± 14%  softirqs.CPU6.TIMER
     64973 ±  2%    +288.5%     252424 ± 14%  softirqs.CPU60.NET_RX
     10247 ±  2%     +75.6%      17998 ±  4%  softirqs.CPU60.RCU
      5451 ±  2%    +152.9%      13784        softirqs.CPU60.TIMER
     65180 ±  2%    +344.8%     289939 ± 11%  softirqs.CPU61.NET_RX
     11034 ± 14%     +68.1%      18551 ±  6%  softirqs.CPU61.RCU
      5625 ±  7%    +173.2%      15369 ± 14%  softirqs.CPU61.TIMER
     64849          +287.7%     251418 ±  6%  softirqs.CPU62.NET_RX
     10588 ±  4%     +69.9%      17986 ±  2%  softirqs.CPU62.RCU
      5587 ±  7%    +142.5%      13552        softirqs.CPU62.TIMER
     65191          +303.1%     262796 ± 11%  softirqs.CPU63.NET_RX
     10228 ±  3%     +79.1%      18316 ±  5%  softirqs.CPU63.RCU
      5420 ±  2%    +148.3%      13457 ±  2%  softirqs.CPU63.TIMER
     64813          +309.0%     265080 ± 15%  softirqs.CPU64.NET_RX
     10451 ±  3%     +79.2%      18730 ±  3%  softirqs.CPU64.RCU
      5488          +151.1%      13780        softirqs.CPU64.TIMER
     65284          +327.0%     278735 ±  7%  softirqs.CPU65.NET_RX
     10823 ±  2%     +80.5%      19533 ±  4%  softirqs.CPU65.RCU
      5491 ±  2%    +152.5%      13865        softirqs.CPU65.TIMER
     65208          +270.7%     241728 ±  4%  softirqs.CPU66.NET_RX
     10337           +76.5%      18248 ±  6%  softirqs.CPU66.RCU
      5429          +153.2%      13749 ±  2%  softirqs.CPU66.TIMER
     65515 ±  2%    +319.1%     274551 ±  9%  softirqs.CPU67.NET_RX
     10376 ±  2%     +78.9%      18565 ±  4%  softirqs.CPU67.RCU
      5440 ±  2%    +149.7%      13584 ±  2%  softirqs.CPU67.TIMER
     64731          +318.1%     270671 ± 11%  softirqs.CPU68.NET_RX
     10697 ±  4%     +80.0%      19257 ±  4%  softirqs.CPU68.RCU
      5399 ±  2%    +152.6%      13639 ±  2%  softirqs.CPU68.TIMER
     65174          +330.2%     280395 ± 15%  softirqs.CPU69.NET_RX
     10292           +81.9%      18723 ±  4%  softirqs.CPU69.RCU
      5460          +148.4%      13565        softirqs.CPU69.TIMER
     65120 ±  2%    +312.9%     268888 ± 14%  softirqs.CPU7.NET_RX
     10507 ±  3%     +78.2%      18719 ±  2%  softirqs.CPU7.RCU
      5408          +153.2%      13696 ±  3%  softirqs.CPU7.TIMER
     65546          +296.6%     259938 ±  8%  softirqs.CPU70.NET_RX
     10458           +73.6%      18158 ±  4%  softirqs.CPU70.RCU
      5495          +150.8%      13782 ±  5%  softirqs.CPU70.TIMER
     65388 ±  2%    +303.2%     263625 ± 15%  softirqs.CPU71.NET_RX
     11398 ± 14%     +65.5%      18859 ±  5%  softirqs.CPU71.RCU
      5670 ±  6%    +143.9%      13830 ±  3%  softirqs.CPU71.TIMER
     64605          +312.9%     266751 ± 11%  softirqs.CPU72.NET_RX
     10711 ±  3%     +78.8%      19148 ±  5%  softirqs.CPU72.RCU
      5397          +148.5%      13409 ±  4%  softirqs.CPU72.TIMER
     65247 ±  2%    +317.3%     272264 ±  9%  softirqs.CPU73.NET_RX
     10405           +75.7%      18284 ±  2%  softirqs.CPU73.RCU
      5361          +148.4%      13317 ±  4%  softirqs.CPU73.TIMER
     65370          +379.6%     313484 ± 13%  softirqs.CPU74.NET_RX
     10010 ±  5%     +95.5%      19570 ±  3%  softirqs.CPU74.RCU
      5390          +150.4%      13498 ±  5%  softirqs.CPU74.TIMER
     64987 ±  2%    +336.7%     283777 ± 13%  softirqs.CPU75.NET_RX
     10337           +83.3%      18946 ±  4%  softirqs.CPU75.RCU
      5497 ±  6%    +142.8%      13348 ±  5%  softirqs.CPU75.TIMER
     64578          +358.0%     295795 ± 11%  softirqs.CPU76.NET_RX
     10296           +88.8%      19438 ±  4%  softirqs.CPU76.RCU
      5460 ±  3%    +140.9%      13155 ±  3%  softirqs.CPU76.TIMER
     65089          +296.1%     257835 ±  7%  softirqs.CPU77.NET_RX
     10396           +80.8%      18791 ±  3%  softirqs.CPU77.RCU
      5506 ±  3%    +137.5%      13078 ±  4%  softirqs.CPU77.TIMER
     64561          +358.4%     295956 ± 11%  softirqs.CPU78.NET_RX
     10365 ±  2%     +84.8%      19151 ±  2%  softirqs.CPU78.RCU
      5399          +145.6%      13260 ±  4%  softirqs.CPU78.TIMER
     64881          +356.6%     296230 ±  7%  softirqs.CPU79.NET_RX
     10183           +83.0%      18631 ±  5%  softirqs.CPU79.RCU
      5405          +149.9%      13508 ±  2%  softirqs.CPU79.TIMER
     64802          +356.2%     295606 ± 12%  softirqs.CPU8.NET_RX
     10496 ±  3%     +82.3%      19130 ±  4%  softirqs.CPU8.RCU
      5557 ±  4%    +149.4%      13858 ±  3%  softirqs.CPU8.TIMER
     66166 ±  2%    +316.6%     275632 ± 11%  softirqs.CPU80.NET_RX
      9975 ±  4%     +81.8%      18135 ±  7%  softirqs.CPU80.RCU
      5419 ±  4%    +149.5%      13518 ± 11%  softirqs.CPU80.TIMER
     65173          +380.4%     313113 ± 16%  softirqs.CPU81.NET_RX
      9892 ±  2%     +87.6%      18559 ±  4%  softirqs.CPU81.RCU
      5315          +149.7%      13270 ±  4%  softirqs.CPU81.TIMER
     64551          +331.2%     278313 ± 14%  softirqs.CPU82.NET_RX
      9931 ±  2%     +94.6%      19330 ±  6%  softirqs.CPU82.RCU
      5349          +149.4%      13342 ±  2%  softirqs.CPU82.TIMER
     65434 ±  2%    +397.8%     325748 ±  8%  softirqs.CPU83.NET_RX
      9955           +87.9%      18703 ±  5%  softirqs.CPU83.RCU
      5400 ±  2%    +149.6%      13477 ±  3%  softirqs.CPU83.TIMER
     64894          +324.0%     275183 ± 29%  softirqs.CPU84.NET_RX
      9772 ±  4%     +92.8%      18845 ±  4%  softirqs.CPU84.RCU
      5332          +152.1%      13444 ±  4%  softirqs.CPU84.TIMER
     64992          +356.7%     296833 ± 13%  softirqs.CPU85.NET_RX
     10260 ±  8%     +84.9%      18975 ±  7%  softirqs.CPU85.RCU
      5319          +152.4%      13422 ±  4%  softirqs.CPU85.TIMER
     64954          +396.3%     322356 ± 12%  softirqs.CPU86.NET_RX
      9827           +88.4%      18517 ±  6%  softirqs.CPU86.RCU
      5285          +149.8%      13204 ±  4%  softirqs.CPU86.TIMER
     64950          +407.4%     329550 ±  7%  softirqs.CPU87.NET_RX
      9913 ±  4%     +94.8%      19311 ±  8%  softirqs.CPU87.RCU
      5417          +147.3%      13396 ±  2%  softirqs.CPU87.TIMER
     64621          +337.2%     282555 ± 17%  softirqs.CPU88.NET_RX
      9877           +84.1%      18188 ±  4%  softirqs.CPU88.RCU
      5400          +147.8%      13383 ±  5%  softirqs.CPU88.TIMER
     64549          +328.5%     276593 ± 13%  softirqs.CPU89.NET_RX
      9775 ±  3%     +82.6%      17853 ±  4%  softirqs.CPU89.RCU
      5367          +142.5%      13017 ±  4%  softirqs.CPU89.TIMER
     65373 ±  2%    +332.4%     282682 ±  8%  softirqs.CPU9.NET_RX
     10558 ±  3%     +75.1%      18485 ±  2%  softirqs.CPU9.RCU
      5394 ±  2%    +151.9%      13591 ±  2%  softirqs.CPU9.TIMER
     65230          +336.2%     284504 ±  9%  softirqs.CPU90.NET_RX
     10086 ±  2%     +85.4%      18704 ±  6%  softirqs.CPU90.RCU
      5358          +147.1%      13239 ±  5%  softirqs.CPU90.TIMER
     64791          +366.5%     302228 ±  8%  softirqs.CPU91.NET_RX
     10239 ±  2%     +82.0%      18631 ±  3%  softirqs.CPU91.RCU
      5353          +149.8%      13376 ±  3%  softirqs.CPU91.TIMER
     64608          +349.6%     290453 ± 12%  softirqs.CPU92.NET_RX
      9957 ±  3%     +88.5%      18769 ±  4%  softirqs.CPU92.RCU
      5311          +147.3%      13138 ±  2%  softirqs.CPU92.TIMER
     64576          +351.7%     291673 ± 13%  softirqs.CPU93.NET_RX
      9984 ±  5%     +90.2%      18993 ±  4%  softirqs.CPU93.RCU
      5318          +149.7%      13278 ±  4%  softirqs.CPU93.TIMER
     64951          +359.2%     298230 ± 12%  softirqs.CPU94.NET_RX
      9872 ±  4%     +87.9%      18547 ±  3%  softirqs.CPU94.RCU
      5433 ±  4%    +143.2%      13211 ±  4%  softirqs.CPU94.TIMER
     63825          +344.4%     283611 ± 13%  softirqs.CPU95.NET_RX
     10222           +92.5%      19679 ±  5%  softirqs.CPU95.RCU
      5288 ±  2%    +152.1%      13330 ±  2%  softirqs.CPU95.TIMER
   6207557          +334.2%   26955885 ±  2%  softirqs.NET_RX
    995456           +82.3%    1814562 ±  2%  softirqs.RCU
    239542 ±  2%      +7.8%     258235 ±  3%  softirqs.SCHED
    521072          +150.5%    1305253 ±  2%  softirqs.TIMER


                                                                                
                            stress-ng.time.user_time                            
                                                                                
  12 +----------------------------------------------------------------------+   
  11 |-+                 O O                                                |   
     |      O      O    O                                                   |   
  10 |-+ O    OO OO   O        O O   O  OO    O                             |   
   9 |O+O  O         O      O O   OO  O    OO                               |   
   8 |-+                                                                    |   
   7 |-+                                                                    |   
     |                                                                      |   
   6 |-+                                                                    |   
   5 |-+                                                                    |   
   4 |-+                                                                    |   
   3 |-+                                                                    |   
     |+.  .++.++.+++.++.++.++.++.+++.+   +.  .++       .+ .++.     +.  .+ .+|   
   2 |-+++                            +.+  ++   +.++.++  +    +++.+  ++  +  |   
   1 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.time.system_time                          
                                                                                
  5700 +--------------------------------------------------------------------+   
       |               .+       +.    +                                     |   
  5650 |+.++.+++.++.+++  +.++.++  ++.+ +.++.+++.++.+++.++.++.+++.++.+++.++.+|   
       |                                                                    |   
  5600 |-+                                                                  |   
       |                                                                    |   
  5550 |-+                                                                  |   
       |                                                                    |   
  5500 |-+            O       O                                             |   
       |O O  O          O                                                   |   
  5450 |-+                                                                  |   
       |   O     O       O     O      OO      O                             |   
  5400 |-+    O   O        OO   O O  O   OO  O                              |   
       |       O    OO             O        O                               |   
  5350 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    stress-ng.time.percent_of_cpu_this_job_got                  
                                                                                
  9050 +--------------------------------------------------------------------+   
  9000 |+.++.+++.++.++                 +.++.+++.++.+++.++.++.+++.++.+++.++.+|   
       |                                                                    |   
  8950 |-+                                                                  |   
  8900 |-+                                                                  |   
       |                                                                    |   
  8850 |-+                                                                  |   
  8800 |-+                                                                  |   
  8750 |-+            O       O                                             |   
       |O O  O          O                                                   |   
  8700 |-+                                                                  |   
  8650 |-+ O     O       O  O  O      OO      O                             |   
       |      O   O        O    O OO O   OO OO                              |   
  8600 |-+     O    OO                                                      |   
  8550 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      stress-ng.time.voluntary_context_switches                 
                                                                                
  1e+07 +-------------------------------------------------------------------+   
        |      O      O   OO O                                              |   
  9e+06 |-+O    O  OO            OO OO O O OOO O                            |   
  8e+06 |O+ O O   O      O    O O       O                                   |   
        |              O                                                    |   
  7e+06 |-+                                                                 |   
        |                                                                   |   
  6e+06 |-+                                                                 |   
        |                                                                   |   
  5e+06 |-+                                                                 |   
  4e+06 |-+                                                                 |   
        |                                                                   |   
  3e+06 |-+                                                                 |   
        |                                 .+  .+            .+              |   
  2e+06 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                     stress-ng.time.involuntary_context_switches                
                                                                                
  1e+07 +-------------------------------------------------------------------+   
        |      O           O                                                |   
  9e+06 |-+O    O  OO O      O   OO OO   O OOO                              |   
  8e+06 |O+ O O   O      OO   O O      OO      O                            |   
        |              O                                                    |   
  7e+06 |-+                                                                 |   
  6e+06 |-+                                                                 |   
        |                                                                   |   
  5e+06 |-+                                                                 |   
  4e+06 |-+                                                                 |   
        |                                                                   |   
  3e+06 |-+                                                                 |   
  2e+06 |-+    ++.+++.  .++   +.      .+ +.  +.  .+ +.++.     +.  +.++   +.+|   
        |+.++.+       ++   +.+  +++.++  +  ++  ++  +     +++.+  ++    +.+   |   
  1e+06 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               stress-ng.sockmany.ops                           
                                                                                
  1e+07 +-------------------------------------------------------------------+   
        |      O      O    O O                                              |   
  9e+06 |-+O    O  OO     O      OO OO O O OOO O                            |   
  8e+06 |O+ O O   O    O O    O O       O                                   |   
        |                                                                   |   
  7e+06 |-+                                                                 |   
  6e+06 |-+                                                                 |   
        |                                                                   |   
  5e+06 |-+                                                                 |   
  4e+06 |-+                                                                 |   
        |                                                                   |   
  3e+06 |-+                                                                 |   
  2e+06 |+.++.+++.+++.++.+++.++.+++. +.+        +.+ +.++.     +.+++.+++.++.+|   
        |                           +   ++.+++.+   +     +++.+              |   
  1e+06 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           stress-ng.sockmany.ops_per_sec                       
                                                                                
  160000 +------------------------------------------------------------------+   
         |      OO  O  O    O O   O O     O  O                              |   
  140000 |-+O      O O     O    O  O  OOO  O  OO                            |   
         |O  OO         OO     O                                            |   
  120000 |-+                                                                |   
         |                                                                  |   
  100000 |-+                                                                |   
         |                                                                  |   
   80000 |-+                                                                |   
         |                                                                  |   
   60000 |-+                                                                |   
         |                                                                  |   
   40000 |-+                                                                |   
         |+.+++.++.+++.+++.++.+++.+++.+++.++.+++.+++.+++.++.+++.+++.++.+++.+|   
   20000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp5: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  os/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/sockmany/stress-ng/60s/0x5003006

commit: 
  3af409ca27 ("net: enetc: fix destroyed phylink dereference during unbind")
  ff0d41306d ("tcp: avoid unnecessary loop if even ports are used up")

3af409ca278d4a8d ff0d41306d8c7b91c21707559ef 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    406932          +572.4%    2736324 ±  2%  stress-ng.sockmany.ops
      6765          +572.1%      45470 ±  2%  stress-ng.sockmany.ops_per_sec
      1903 ± 10%    +211.9%       5935 ±  4%  stress-ng.time.involuntary_context_switches
     10220            +6.7%      10904 ±  6%  stress-ng.time.minor_page_faults
    856.33            -2.5%     835.00        stress-ng.time.percent_of_cpu_this_job_got
    532.86            -2.7%     518.28        stress-ng.time.system_time
    811820          +572.8%    5461555 ±  2%  stress-ng.time.voluntary_context_switches
      9.59            +5.2%      10.09        iostat.cpu.system
    958912           +12.4%    1077693        meminfo.SUnreclaim
   1216083           +10.4%    1343126        meminfo.Slab
      0.18            +0.7        0.86 ±  2%  mpstat.cpu.all.soft%
      0.04            +0.0        0.07        mpstat.cpu.all.usr%
      0.73 ± 30%    +220.5%       2.33 ± 87%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      2.99 ±  2%    +124.8%       6.73 ±108%  perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
     90.00            -1.1%      89.00        vmstat.cpu.id
     26898          +531.6%     169904 ±  2%  vmstat.system.cs
      3951 ± 45%    +118.4%       8629 ± 57%  sched_debug.cfs_rq:/.load.avg
    -11161          +259.9%     -40170        sched_debug.cfs_rq:/.spread0.min
      2339 ± 13%     +59.6%       3734 ± 37%  sched_debug.cpu.avg_idle.min
    171001 ± 19%    +100.9%     343533 ± 11%  numa-numastat.node0.local_node
    220436 ±  6%     +72.7%     380593 ±  6%  numa-numastat.node0.numa_hit
    170511 ± 20%     +90.4%     324657 ± 10%  numa-numastat.node1.local_node
    207667 ±  7%     +80.2%     374171 ±  3%  numa-numastat.node1.numa_hit
   7118182 ±  4%    +397.1%   35384127 ±  3%  cpuidle.C1.time
    583205 ±  2%    +576.6%    3946063 ±  2%  cpuidle.C1.usage
  10115087 ± 18%     +20.6%   12200316        cpuidle.C1E.usage
    497217 ± 23%    +209.7%    1539923 ±  6%  cpuidle.POLL.time
    115595 ±  5%    +124.7%     259768 ±  7%  cpuidle.POLL.usage
     63981            +3.2%      66038        proc-vmstat.nr_slab_reclaimable
    238932           +12.5%     268761        proc-vmstat.nr_slab_unreclaimable
    353.50 ± 58%    +351.0%       1594 ± 78%  proc-vmstat.numa_hint_faults
    217.00 ± 66%    +268.0%     798.67 ± 44%  proc-vmstat.numa_hint_faults_local
    461585           +70.4%     786370 ±  2%  proc-vmstat.numa_hit
    374975           +86.6%     699774 ±  3%  proc-vmstat.numa_local
    835776          +179.0%    2331789 ±  6%  proc-vmstat.pgalloc_normal
    570397 ±  2%    +254.2%    2020484 ±  7%  proc-vmstat.pgfree
     12704 ±  2%     +16.9%      14857 ±  2%  slabinfo.TCP.active_slabs
    177868 ±  2%     +16.9%     208016 ±  2%  slabinfo.TCP.num_objs
     12704 ±  2%     +16.9%      14857 ±  2%  slabinfo.TCP.num_slabs
     26316 ±  8%    +165.7%      69913 ±  3%  slabinfo.dmaengine-unmap-16.active_objs
    626.67 ±  8%    +165.9%       1666 ±  3%  slabinfo.dmaengine-unmap-16.active_slabs
     26342 ±  8%    +165.7%      69985 ±  3%  slabinfo.dmaengine-unmap-16.num_objs
    626.67 ±  8%    +165.9%       1666 ±  3%  slabinfo.dmaengine-unmap-16.num_slabs
     18132           +43.9%      26089 ±  8%  slabinfo.kmalloc-128.active_objs
    582.50           +44.9%     844.17 ±  7%  slabinfo.kmalloc-128.active_slabs
     18654           +44.9%      27031 ±  7%  slabinfo.kmalloc-128.num_objs
    582.50           +44.9%     844.17 ±  7%  slabinfo.kmalloc-128.num_slabs
    233241           +41.0%     328829 ±  2%  slabinfo.kmalloc-16.active_objs
    915.67           +44.8%       1326 ±  2%  slabinfo.kmalloc-16.active_slabs
    234552           +44.8%     339569 ±  2%  slabinfo.kmalloc-16.num_objs
    915.67           +44.8%       1326 ±  2%  slabinfo.kmalloc-16.num_slabs
     13671 ±  2%     +40.4%      19187 ±  2%  slabinfo.kmalloc-256.active_objs
    427.67 ±  2%     +42.5%     609.50 ±  2%  slabinfo.kmalloc-256.active_slabs
     13700 ±  2%     +42.5%      19519 ±  2%  slabinfo.kmalloc-256.num_objs
    427.67 ±  2%     +42.5%     609.50 ±  2%  slabinfo.kmalloc-256.num_slabs
     22110           +17.7%      26024        slabinfo.kmalloc-512.active_objs
    692.50           +21.0%     837.67        slabinfo.kmalloc-512.active_slabs
     22175           +21.0%      26828        slabinfo.kmalloc-512.num_objs
    692.50           +21.0%     837.67        slabinfo.kmalloc-512.num_slabs
    191005           +23.4%     235765 ±  3%  slabinfo.lsm_file_cache.active_objs
      1123           +23.6%       1388 ±  3%  slabinfo.lsm_file_cache.active_slabs
    191069           +23.6%     236193 ±  3%  slabinfo.lsm_file_cache.num_objs
      1123           +23.6%       1388 ±  3%  slabinfo.lsm_file_cache.num_slabs
      2640 ± 23%     +61.7%       4270 ±  6%  slabinfo.request_sock_TCP.active_objs
      2640 ± 23%     +61.7%       4270 ±  6%  slabinfo.request_sock_TCP.num_objs
      5264 ±  6%     +55.5%       8188 ±  2%  slabinfo.skbuff_head_cache.active_objs
      5265 ±  6%     +55.5%       8189 ±  2%  slabinfo.skbuff_head_cache.num_objs
    192129           +29.4%     248569        slabinfo.tw_sock_TCP.active_objs
      5825           +30.0%       7576        slabinfo.tw_sock_TCP.active_slabs
    192257           +30.0%     250019        slabinfo.tw_sock_TCP.num_objs
      5825           +30.0%       7576        slabinfo.tw_sock_TCP.num_slabs
     25.38           -16.2%      21.28        perf-stat.i.MPKI
 2.217e+09           +23.1%   2.73e+09        perf-stat.i.branch-instructions
      1.13 ±  2%      +0.1        1.27        perf-stat.i.branch-miss-rate%
  24373526           +40.4%   34215124        perf-stat.i.branch-misses
  88152704            +5.9%   93328112        perf-stat.i.cache-misses
 2.878e+08            +6.2%  3.055e+08        perf-stat.i.cache-references
     27876          +530.4%     175723 ±  2%  perf-stat.i.context-switches
      2.46 ±  2%     -16.0%       2.07        perf-stat.i.cpi
 2.762e+10            +6.1%  2.929e+10        perf-stat.i.cpu-cycles
    117.83 ±  4%     +26.1%     148.63 ±  3%  perf-stat.i.cpu-migrations
  16153047 ± 11%     +44.7%   23371999 ± 11%  perf-stat.i.dTLB-load-misses
 3.465e+09 ±  2%     +27.8%  4.427e+09        perf-stat.i.dTLB-loads
      0.01 ± 19%      +0.0        0.01 ± 13%  perf-stat.i.dTLB-store-miss-rate%
    134988 ± 14%    +155.5%     344922 ± 13%  perf-stat.i.dTLB-store-misses
   1.7e+09 ±  2%     +39.5%  2.372e+09        perf-stat.i.dTLB-stores
     63.19           +17.4       80.64        perf-stat.i.iTLB-load-miss-rate%
   4020769          +256.0%   14313812        perf-stat.i.iTLB-load-misses
   1967939 ±  2%     +58.3%    3115314        perf-stat.i.iTLB-loads
 1.119e+10           +26.6%  1.416e+10        perf-stat.i.instructions
      3490 ±  2%     -68.7%       1092        perf-stat.i.instructions-per-iTLB-miss
      0.42 ±  2%     +18.2%       0.49        perf-stat.i.ipc
      0.29            +6.1%       0.31        perf-stat.i.metric.GHz
      0.30 ± 34%     -63.7%       0.11 ±  9%  perf-stat.i.metric.K/sec
     80.62           +28.0%     103.16        perf-stat.i.metric.M/sec
     94.91            -9.0       85.91        perf-stat.i.node-store-miss-rate%
  20243974 ±  3%     -11.1%   17987019        perf-stat.i.node-store-misses
    519871 ±  5%    +337.7%    2275278 ±  2%  perf-stat.i.node-stores
     25.72           -16.2%      21.56        perf-stat.overall.MPKI
      1.10            +0.2        1.25        perf-stat.overall.branch-miss-rate%
      2.47 ±  2%     -16.3%       2.07        perf-stat.overall.cpi
      0.01 ± 15%      +0.0        0.01 ± 13%  perf-stat.overall.dTLB-store-miss-rate%
     67.12           +15.0       82.13        perf-stat.overall.iTLB-load-miss-rate%
      2789           -64.5%     989.27        perf-stat.overall.instructions-per-iTLB-miss
      0.41 ±  2%     +19.4%       0.48        perf-stat.overall.ipc
     97.51            -8.8       88.75        perf-stat.overall.node-store-miss-rate%
 2.182e+09           +23.2%  2.688e+09        perf-stat.ps.branch-instructions
  23976684           +40.4%   33667917        perf-stat.ps.branch-misses
  86792250            +5.8%   91827397        perf-stat.ps.cache-misses
 2.833e+08            +6.1%  3.006e+08        perf-stat.ps.cache-references
     27275          +534.4%     173046 ±  2%  perf-stat.ps.context-switches
 2.719e+10            +6.0%  2.883e+10        perf-stat.ps.cpu-cycles
    115.93 ±  4%     +26.2%     146.33 ±  3%  perf-stat.ps.cpu-migrations
  15904910 ± 11%     +44.5%   22977809 ± 11%  perf-stat.ps.dTLB-load-misses
 3.411e+09           +27.8%  4.357e+09        perf-stat.ps.dTLB-loads
    132772 ± 14%    +155.5%     339281 ± 13%  perf-stat.ps.dTLB-store-misses
 1.673e+09 ±  2%     +39.5%  2.335e+09        perf-stat.ps.dTLB-stores
   3948292          +257.0%   14096088        perf-stat.ps.iTLB-load-misses
   1934787 ±  2%     +58.5%    3066788        perf-stat.ps.iTLB-loads
 1.101e+10           +26.6%  1.394e+10        perf-stat.ps.instructions
  19931573 ±  3%     -11.2%   17693059        perf-stat.ps.node-store-misses
    509785 ±  5%    +339.6%    2240990 ±  2%  perf-stat.ps.node-stores
 6.971e+11 ±  2%     +27.0%  8.855e+11        perf-stat.total.instructions
     73769 ± 10%     -24.7%      55542 ±  2%  interrupts.CAL:Function_call_interrupts
     26.33 ± 70%   +1144.3%     327.67 ± 17%  interrupts.CPU0.RES:Rescheduling_interrupts
     71.50 ± 42%    +984.8%     775.67 ± 40%  interrupts.CPU1.RES:Rescheduling_interrupts
     43.33 ±119%   +1041.2%     494.50 ± 34%  interrupts.CPU10.RES:Rescheduling_interrupts
     63.50 ± 77%    +481.9%     369.50 ± 27%  interrupts.CPU11.RES:Rescheduling_interrupts
     47.00 ± 70%    +850.4%     446.67 ± 44%  interrupts.CPU12.RES:Rescheduling_interrupts
     39.17 ±135%    +939.6%     407.17 ± 48%  interrupts.CPU13.RES:Rescheduling_interrupts
     37.67 ±103%    +748.2%     319.50 ± 49%  interrupts.CPU14.RES:Rescheduling_interrupts
     28.83 ±148%   +1181.5%     369.50 ± 31%  interrupts.CPU15.RES:Rescheduling_interrupts
     31.67 ± 91%    +887.4%     312.67 ± 44%  interrupts.CPU16.RES:Rescheduling_interrupts
     30.83 ±133%    +935.1%     319.17 ± 24%  interrupts.CPU17.RES:Rescheduling_interrupts
     61.00 ± 37%    +440.2%     329.50 ± 47%  interrupts.CPU18.RES:Rescheduling_interrupts
     54.67 ± 35%    +511.6%     334.33 ± 41%  interrupts.CPU19.RES:Rescheduling_interrupts
     67.17 ± 59%   +1071.5%     786.83 ± 38%  interrupts.CPU2.RES:Rescheduling_interrupts
     33.83 ± 77%    +707.9%     273.33 ± 36%  interrupts.CPU20.RES:Rescheduling_interrupts
     22.17 ± 76%   +1046.6%     254.17 ± 33%  interrupts.CPU21.RES:Rescheduling_interrupts
    119.00 ± 27%    +925.6%       1220 ± 79%  interrupts.CPU22.NMI:Non-maskable_interrupts
    119.00 ± 27%    +925.6%       1220 ± 79%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
     22.67 ± 66%   +1788.2%     428.00 ± 48%  interrupts.CPU22.RES:Rescheduling_interrupts
    120.83 ±  9%   +1111.7%       1464 ±107%  interrupts.CPU23.NMI:Non-maskable_interrupts
    120.83 ±  9%   +1111.7%       1464 ±107%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
     41.00 ±134%    +449.2%     225.17 ± 36%  interrupts.CPU23.RES:Rescheduling_interrupts
    269.83 ± 75%    +768.8%       2344 ± 69%  interrupts.CPU24.NMI:Non-maskable_interrupts
    269.83 ± 75%    +768.8%       2344 ± 69%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
     60.33 ± 69%    +748.3%     511.83 ± 41%  interrupts.CPU24.RES:Rescheduling_interrupts
     51.83 ± 86%   +1797.7%     983.67 ± 26%  interrupts.CPU25.RES:Rescheduling_interrupts
     44.33 ± 61%   +1635.7%     769.50 ± 23%  interrupts.CPU26.RES:Rescheduling_interrupts
     72.17 ± 83%   +1060.3%     837.33 ± 23%  interrupts.CPU27.RES:Rescheduling_interrupts
     51.67 ±147%   +1175.2%     658.83 ± 18%  interrupts.CPU28.RES:Rescheduling_interrupts
    186.67 ± 71%    +468.0%       1060 ± 42%  interrupts.CPU29.NMI:Non-maskable_interrupts
    186.67 ± 71%    +468.0%       1060 ± 42%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
     44.83 ± 66%   +1643.5%     781.67 ± 46%  interrupts.CPU29.RES:Rescheduling_interrupts
     83.17 ± 93%    +686.4%     654.00 ± 19%  interrupts.CPU3.RES:Rescheduling_interrupts
     30.83 ± 79%   +1429.2%     471.50 ± 44%  interrupts.CPU30.RES:Rescheduling_interrupts
     13.17 ± 74%   +4870.9%     654.50 ± 54%  interrupts.CPU31.RES:Rescheduling_interrupts
     22.83 ± 49%   +1778.1%     428.83 ± 59%  interrupts.CPU32.RES:Rescheduling_interrupts
    136.17 ± 18%   +1238.3%       1822 ± 85%  interrupts.CPU33.NMI:Non-maskable_interrupts
    136.17 ± 18%   +1238.3%       1822 ± 85%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     54.00 ±111%   +1000.9%     594.50 ± 44%  interrupts.CPU33.RES:Rescheduling_interrupts
     49.17 ± 63%    +713.9%     400.17 ± 66%  interrupts.CPU34.RES:Rescheduling_interrupts
     38.50 ± 67%    +955.0%     406.17 ± 51%  interrupts.CPU35.RES:Rescheduling_interrupts
     41.83 ± 83%    +918.7%     426.17 ± 17%  interrupts.CPU36.RES:Rescheduling_interrupts
     37.67 ±107%    +989.8%     410.50 ± 52%  interrupts.CPU37.RES:Rescheduling_interrupts
     50.00 ±100%    +743.0%     421.50 ± 23%  interrupts.CPU38.RES:Rescheduling_interrupts
     24.00 ±110%   +1014.6%     267.50 ± 47%  interrupts.CPU39.RES:Rescheduling_interrupts
     34.50 ± 70%   +2195.7%     792.00 ± 38%  interrupts.CPU4.RES:Rescheduling_interrupts
     50.50 ± 95%    +506.3%     306.17 ± 53%  interrupts.CPU40.RES:Rescheduling_interrupts
     42.17 ±132%    +501.2%     253.50 ± 33%  interrupts.CPU41.RES:Rescheduling_interrupts
     46.17 ± 96%    +571.5%     310.00 ± 57%  interrupts.CPU42.RES:Rescheduling_interrupts
     37.67 ±107%   +1051.3%     433.67 ± 72%  interrupts.CPU43.RES:Rescheduling_interrupts
    115.00 ± 22%    +731.9%     956.67 ± 71%  interrupts.CPU44.NMI:Non-maskable_interrupts
    115.00 ± 22%    +731.9%     956.67 ± 71%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
     11.67 ±132%   +1657.1%     205.00 ± 62%  interrupts.CPU44.RES:Rescheduling_interrupts
     22.83 ± 85%    +838.7%     214.33 ± 56%  interrupts.CPU45.RES:Rescheduling_interrupts
    510.17 ±  3%    +106.2%       1051 ± 78%  interrupts.CPU46.CAL:Function_call_interrupts
     46.50 ± 32%    +373.1%     220.00 ± 37%  interrupts.CPU46.RES:Rescheduling_interrupts
     14.33 ±119%   +1297.7%     200.33 ± 86%  interrupts.CPU47.RES:Rescheduling_interrupts
     42.67 ±103%    +515.6%     262.67 ± 43%  interrupts.CPU48.RES:Rescheduling_interrupts
     47.33 ±140%    +793.7%     423.00 ± 53%  interrupts.CPU49.RES:Rescheduling_interrupts
     65.17 ± 97%    +702.6%     523.00 ± 23%  interrupts.CPU5.RES:Rescheduling_interrupts
     65.67 ± 59%    +298.0%     261.33 ± 48%  interrupts.CPU50.RES:Rescheduling_interrupts
     18.67 ± 75%   +2134.8%     417.17 ± 41%  interrupts.CPU51.RES:Rescheduling_interrupts
     64.17 ± 61%    +490.9%     379.17 ± 34%  interrupts.CPU52.RES:Rescheduling_interrupts
     56.33 ± 97%    +670.4%     434.00 ± 51%  interrupts.CPU54.RES:Rescheduling_interrupts
     32.33 ± 77%   +1141.2%     401.33 ± 15%  interrupts.CPU55.RES:Rescheduling_interrupts
    221.83 ±105%    +366.4%       1034 ±101%  interrupts.CPU56.NMI:Non-maskable_interrupts
    221.83 ±105%    +366.4%       1034 ±101%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
     33.17 ±134%    +814.1%     303.17 ± 42%  interrupts.CPU56.RES:Rescheduling_interrupts
     36.83 ± 82%   +1127.6%     452.17 ± 47%  interrupts.CPU57.RES:Rescheduling_interrupts
     15.00 ± 61%   +3018.9%     467.83 ± 45%  interrupts.CPU58.RES:Rescheduling_interrupts
     20.67 ± 63%   +1154.0%     259.17 ± 32%  interrupts.CPU59.RES:Rescheduling_interrupts
     32.50 ± 49%   +1437.4%     499.67 ± 43%  interrupts.CPU6.RES:Rescheduling_interrupts
     44.83 ± 79%    +673.6%     346.83 ± 53%  interrupts.CPU60.RES:Rescheduling_interrupts
     15.83 ± 56%   +1555.8%     262.17 ± 51%  interrupts.CPU61.RES:Rescheduling_interrupts
     39.83 ±123%    +864.9%     384.33 ± 47%  interrupts.CPU62.RES:Rescheduling_interrupts
     51.50 ± 85%    +411.7%     263.50 ± 50%  interrupts.CPU64.RES:Rescheduling_interrupts
     28.67 ±155%    +719.8%     235.00 ± 63%  interrupts.CPU65.RES:Rescheduling_interrupts
     36.83 ±101%    +557.0%     242.00 ± 64%  interrupts.CPU66.RES:Rescheduling_interrupts
      7.83 ± 75%   +3757.4%     302.17 ± 64%  interrupts.CPU67.RES:Rescheduling_interrupts
     31.83 ±120%    +492.7%     188.67 ± 74%  interrupts.CPU68.RES:Rescheduling_interrupts
     41.67 ±100%    +466.4%     236.00 ± 47%  interrupts.CPU69.RES:Rescheduling_interrupts
     52.17 ± 33%   +1000.3%     574.00 ± 32%  interrupts.CPU7.RES:Rescheduling_interrupts
    121.17 ± 24%    +612.4%     863.17 ± 62%  interrupts.CPU70.NMI:Non-maskable_interrupts
    121.17 ± 24%    +612.4%     863.17 ± 62%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
     16.33 ± 76%   +1075.5%     192.00 ± 66%  interrupts.CPU70.RES:Rescheduling_interrupts
    110.17 ± 25%    +859.8%       1057 ± 75%  interrupts.CPU71.NMI:Non-maskable_interrupts
    110.17 ± 25%    +859.8%       1057 ± 75%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
     36.00 ±146%    +724.5%     296.83 ± 54%  interrupts.CPU71.RES:Rescheduling_interrupts
    415.83 ±101%    +208.9%       1284 ± 76%  interrupts.CPU72.NMI:Non-maskable_interrupts
    415.83 ±101%    +208.9%       1284 ± 76%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
     28.00 ±180%    +848.2%     265.50 ± 83%  interrupts.CPU72.RES:Rescheduling_interrupts
      6.67 ± 51%   +7032.5%     475.50 ± 71%  interrupts.CPU73.RES:Rescheduling_interrupts
      6.67 ±126%   +5160.0%     350.67 ± 33%  interrupts.CPU74.RES:Rescheduling_interrupts
    424.00 ±116%    +260.7%       1529 ± 60%  interrupts.CPU75.NMI:Non-maskable_interrupts
    424.00 ±116%    +260.7%       1529 ± 60%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
     23.17 ± 84%   +1324.5%     330.00 ± 30%  interrupts.CPU75.RES:Rescheduling_interrupts
     14.50 ±110%   +2033.3%     309.33 ± 37%  interrupts.CPU76.RES:Rescheduling_interrupts
    169.50 ± 89%    +400.3%     848.00 ± 58%  interrupts.CPU77.NMI:Non-maskable_interrupts
    169.50 ± 89%    +400.3%     848.00 ± 58%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
     22.17 ±172%   +1917.3%     447.17 ± 52%  interrupts.CPU77.RES:Rescheduling_interrupts
     28.67 ±152%    +804.7%     259.33 ± 40%  interrupts.CPU78.RES:Rescheduling_interrupts
     14.00 ±113%   +2435.7%     355.00 ± 35%  interrupts.CPU79.RES:Rescheduling_interrupts
    196.67 ± 46%    +281.2%     749.67 ± 70%  interrupts.CPU8.NMI:Non-maskable_interrupts
    196.67 ± 46%    +281.2%     749.67 ± 70%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
     59.50 ± 40%    +917.4%     605.33 ± 28%  interrupts.CPU8.RES:Rescheduling_interrupts
     19.17 ± 72%   +1612.2%     328.17 ± 25%  interrupts.CPU80.RES:Rescheduling_interrupts
    121.50 ± 30%   +1200.8%       1580 ± 63%  interrupts.CPU81.NMI:Non-maskable_interrupts
    121.50 ± 30%   +1200.8%       1580 ± 63%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
     52.17 ±114%    +460.7%     292.50 ± 41%  interrupts.CPU81.RES:Rescheduling_interrupts
     50.50 ±121%    +407.9%     256.50 ± 37%  interrupts.CPU82.RES:Rescheduling_interrupts
     37.67 ± 91%    +992.9%     411.67 ± 53%  interrupts.CPU83.RES:Rescheduling_interrupts
     25.83 ± 67%   +1121.9%     315.67 ± 61%  interrupts.CPU84.RES:Rescheduling_interrupts
     38.50 ± 97%    +925.5%     394.83 ± 57%  interrupts.CPU85.RES:Rescheduling_interrupts
     21.17 ± 74%   +1758.3%     393.33 ± 75%  interrupts.CPU86.RES:Rescheduling_interrupts
     47.17 ±101%    +822.6%     435.17 ± 50%  interrupts.CPU87.RES:Rescheduling_interrupts
     16.67 ± 52%   +1871.0%     328.50 ± 72%  interrupts.CPU88.RES:Rescheduling_interrupts
     64.67 ± 31%    +548.5%     419.33 ± 37%  interrupts.CPU9.RES:Rescheduling_interrupts
     40.83 ± 97%    +653.9%     307.83 ± 51%  interrupts.CPU91.RES:Rescheduling_interrupts
     53.00 ± 84%    +511.6%     324.17 ± 45%  interrupts.CPU92.RES:Rescheduling_interrupts
     17.00 ± 80%   +1996.1%     356.33 ± 44%  interrupts.CPU93.RES:Rescheduling_interrupts
     14.67 ±109%   +2461.4%     375.67 ± 49%  interrupts.CPU94.RES:Rescheduling_interrupts
     16.83 ±126%   +1366.3%     246.83 ± 55%  interrupts.CPU95.RES:Rescheduling_interrupts
      3674 ±  4%    +927.8%      37769 ±  5%  interrupts.RES:Rescheduling_interrupts
    435.67 ± 40%    +121.2%     963.50 ± 78%  interrupts.TLB:TLB_shootdowns
      8830 ± 67%   +1260.1%     120103 ± 24%  softirqs.CPU0.NET_RX
     11931 ±  4%     +30.5%      15564 ± 11%  softirqs.CPU0.SCHED
     16183 ± 52%    +757.3%     138737 ± 21%  softirqs.CPU1.NET_RX
      9241 ±  9%     +41.7%      13090 ± 16%  softirqs.CPU1.RCU
      9990 ± 13%     +26.7%      12657 ± 11%  softirqs.CPU1.SCHED
     10833 ± 82%    +587.9%      74518 ± 24%  softirqs.CPU10.NET_RX
      9309           +11.1%      10343 ±  4%  softirqs.CPU10.SCHED
     16966 ± 76%    +396.6%      84257 ± 21%  softirqs.CPU11.NET_RX
     14684 ± 55%    +373.7%      69565 ± 44%  softirqs.CPU12.NET_RX
     11864 ±112%    +528.0%      74509 ± 31%  softirqs.CPU13.NET_RX
      9913 ± 81%    +627.3%      72106 ± 12%  softirqs.CPU14.NET_RX
      8183 ±  9%     +13.6%       9292 ±  6%  softirqs.CPU14.RCU
      9255 ±  2%     +16.3%      10768 ±  5%  softirqs.CPU14.SCHED
     11106 ±121%    +442.0%      60198 ± 30%  softirqs.CPU15.NET_RX
     12926 ± 76%    +407.5%      65607 ± 28%  softirqs.CPU16.NET_RX
      8222 ±102%    +652.2%      61853 ± 38%  softirqs.CPU17.NET_RX
     16663 ± 38%    +259.4%      59894 ± 27%  softirqs.CPU18.NET_RX
      9252 ±  2%     +11.7%      10335 ±  6%  softirqs.CPU18.SCHED
     14125 ± 45%    +306.1%      57361 ± 43%  softirqs.CPU19.NET_RX
     19278 ± 49%    +501.2%     115899 ± 19%  softirqs.CPU2.NET_RX
      9140 ±  8%     +21.4%      11098 ±  4%  softirqs.CPU2.RCU
      9487 ±  3%     +19.2%      11306 ±  7%  softirqs.CPU2.SCHED
     12296 ± 93%    +336.9%      53727 ± 27%  softirqs.CPU20.NET_RX
      9869 ± 77%    +324.5%      41897 ± 31%  softirqs.CPU21.NET_RX
      8826 ± 54%    +607.0%      62404 ± 31%  softirqs.CPU22.NET_RX
      9315 ±  2%     +14.8%      10695 ±  6%  softirqs.CPU22.SCHED
     10475 ± 89%    +456.7%      58319 ± 48%  softirqs.CPU23.NET_RX
     12950 ± 35%   +1009.4%     143672 ± 30%  softirqs.CPU24.NET_RX
      9948 ±  5%     +41.8%      14102 ±  9%  softirqs.CPU24.SCHED
     18416 ± 51%    +741.8%     155031 ± 31%  softirqs.CPU25.NET_RX
      9391 ± 10%     +38.1%      12968 ± 16%  softirqs.CPU25.RCU
      9237 ± 12%     +30.6%      12061 ±  7%  softirqs.CPU25.SCHED
     14158 ± 58%    +850.3%     134537 ± 25%  softirqs.CPU26.NET_RX
      9363 ±  3%     +26.9%      11879 ± 10%  softirqs.CPU26.SCHED
     15992 ± 50%    +799.7%     143881 ± 21%  softirqs.CPU27.NET_RX
      9150 ±  8%     +28.5%      11761 ± 11%  softirqs.CPU27.RCU
      9397 ±  2%     +24.8%      11726 ±  9%  softirqs.CPU27.SCHED
     12036 ± 78%    +787.5%     106824 ± 21%  softirqs.CPU28.NET_RX
     10926 ± 85%   +1028.1%     123256 ± 32%  softirqs.CPU29.NET_RX
      8771 ±  5%     +29.1%      11324 ± 15%  softirqs.CPU29.RCU
     18088 ± 76%    +610.3%     128474 ± 31%  softirqs.CPU3.NET_RX
      9461 ±  2%     +25.1%      11832 ±  8%  softirqs.CPU3.SCHED
     14278 ± 77%    +639.0%     105521 ± 39%  softirqs.CPU30.NET_RX
      8865 ±  8%     +22.5%      10855 ± 14%  softirqs.CPU30.RCU
      6563 ± 83%   +1487.2%     104167 ± 44%  softirqs.CPU31.NET_RX
      8461 ±  5%     +27.5%      10788 ± 16%  softirqs.CPU31.RCU
      9456 ±  2%     +14.8%      10860 ± 10%  softirqs.CPU31.SCHED
      9245 ± 45%    +923.6%      94637 ± 61%  softirqs.CPU32.NET_RX
     12745 ± 96%    +703.7%     102431 ± 54%  softirqs.CPU33.NET_RX
     14430 ± 66%    +436.6%      77441 ± 49%  softirqs.CPU34.NET_RX
     17445 ± 57%    +317.8%      72879 ± 42%  softirqs.CPU35.NET_RX
     13208 ± 59%    +481.6%      76816 ± 33%  softirqs.CPU36.NET_RX
      9637 ± 64%    +640.8%      71395 ± 27%  softirqs.CPU37.NET_RX
     13024 ± 85%    +553.0%      85048 ± 22%  softirqs.CPU38.NET_RX
     11309 ± 91%    +533.3%      71623 ± 16%  softirqs.CPU39.NET_RX
      9184 ±  2%     +15.9%      10640 ±  3%  softirqs.CPU39.SCHED
      9592 ± 41%   +1125.9%     117599 ± 23%  softirqs.CPU4.NET_RX
      8361 ±  5%     +36.4%      11402 ± 12%  softirqs.CPU4.RCU
      9456           +16.2%      10987 ±  9%  softirqs.CPU4.SCHED
     10046 ± 86%    +508.8%      61156 ± 26%  softirqs.CPU40.NET_RX
     13357 ± 78%    +383.7%      64610 ± 20%  softirqs.CPU41.NET_RX
      9495 ±  4%     +11.3%      10570 ±  7%  softirqs.CPU41.SCHED
     14571 ±118%    +329.8%      62628 ± 37%  softirqs.CPU42.NET_RX
     12159 ± 59%    +375.1%      57766 ± 46%  softirqs.CPU43.NET_RX
      4342 ± 86%   +1102.7%      52221 ± 79%  softirqs.CPU44.NET_RX
     10529 ±108%    +313.3%      43516 ± 43%  softirqs.CPU45.NET_RX
     14525 ± 41%    +264.8%      52993 ± 37%  softirqs.CPU46.NET_RX
      4517 ±114%    +754.3%      38592 ± 45%  softirqs.CPU47.NET_RX
     12059 ± 86%    +446.5%      65907 ± 31%  softirqs.CPU48.NET_RX
      7573 ±  9%     +19.9%       9079 ±  6%  softirqs.CPU48.RCU
      7588 ±112%    +652.5%      57103 ± 18%  softirqs.CPU49.NET_RX
     13904 ± 62%    +751.3%     118360 ± 25%  softirqs.CPU5.NET_RX
      8563 ±  9%     +34.4%      11507 ±  5%  softirqs.CPU5.RCU
      9370           +26.5%      11854 ±  8%  softirqs.CPU5.SCHED
     14523 ± 38%    +311.5%      59763 ± 31%  softirqs.CPU50.NET_RX
      8029 ± 26%     +28.0%      10279 ±  4%  softirqs.CPU50.SCHED
      9569 ± 64%    +678.7%      74515 ± 39%  softirqs.CPU51.NET_RX
      9410 ±  2%     +10.6%      10411 ±  4%  softirqs.CPU51.SCHED
     18241 ± 66%    +303.0%      73517 ± 20%  softirqs.CPU52.NET_RX
     20138 ± 67%    +258.5%      72195 ± 28%  softirqs.CPU53.NET_RX
      9574 ±  5%     +11.8%      10699 ±  7%  softirqs.CPU53.SCHED
     14135 ± 69%    +424.0%      74076 ± 36%  softirqs.CPU54.NET_RX
      7701 ± 57%    +718.3%      63025 ±  9%  softirqs.CPU55.NET_RX
     12838 ± 94%    +461.9%      72138 ± 34%  softirqs.CPU56.NET_RX
      9297           +16.8%      10858 ±  9%  softirqs.CPU56.SCHED
      8682 ± 57%    +663.8%      66313 ± 31%  softirqs.CPU57.NET_RX
      6705 ± 78%   +1063.1%      77987 ± 26%  softirqs.CPU58.NET_RX
      7435 ± 13%     +31.5%       9780 ± 12%  softirqs.CPU58.RCU
      9715 ± 90%    +434.0%      51883 ± 48%  softirqs.CPU59.NET_RX
      9224           +16.8%      10769 ± 10%  softirqs.CPU59.SCHED
     16734 ± 49%    +355.6%      76237 ± 32%  softirqs.CPU6.NET_RX
     12885 ± 55%    +350.5%      58051 ± 53%  softirqs.CPU60.NET_RX
      7611 ± 75%    +438.9%      41017 ± 44%  softirqs.CPU61.NET_RX
     13643 ± 74%    +290.5%      53273 ± 50%  softirqs.CPU62.NET_RX
      8762 ±  5%     +10.5%       9680 ±  5%  softirqs.CPU62.SCHED
      9390 ±  2%     +12.9%      10598 ±  4%  softirqs.CPU63.SCHED
     14037 ± 67%    +301.2%      56322 ± 52%  softirqs.CPU64.NET_RX
      9442 ±  2%     +11.5%      10530 ±  8%  softirqs.CPU64.SCHED
      5997 ±132%    +593.9%      41613 ± 54%  softirqs.CPU65.NET_RX
      5753 ± 73%    +760.1%      49486 ± 36%  softirqs.CPU66.NET_RX
      9275           +11.0%      10297 ±  3%  softirqs.CPU66.SCHED
      4948 ±117%    +723.3%      40743 ± 54%  softirqs.CPU67.NET_RX
     14236 ± 82%    +207.4%      43759 ± 29%  softirqs.CPU69.NET_RX
     11767 ± 48%    +885.1%     115925 ± 20%  softirqs.CPU7.NET_RX
      7972 ± 12%     +37.5%      10958 ± 13%  softirqs.CPU7.RCU
      9355 ±  2%     +21.0%      11321 ±  6%  softirqs.CPU7.SCHED
      6165 ± 61%    +504.1%      37246 ± 30%  softirqs.CPU70.NET_RX
      7331 ±149%    +535.7%      46605 ± 41%  softirqs.CPU71.NET_RX
      4994 ±135%   +1042.4%      57059 ± 35%  softirqs.CPU72.NET_RX
      3843 ± 93%   +1959.4%      79143 ± 41%  softirqs.CPU73.NET_RX
      9112 ±  2%     +12.4%      10246 ±  6%  softirqs.CPU73.SCHED
      7262 ± 88%   +1084.9%      86049 ± 29%  softirqs.CPU74.NET_RX
      7966 ±  6%     +18.5%       9444 ± 10%  softirqs.CPU74.RCU
      9226           +16.3%      10730 ± 10%  softirqs.CPU74.SCHED
      7467 ±112%    +704.7%      60088 ± 33%  softirqs.CPU75.NET_RX
      5702 ±102%    +988.5%      62070 ± 48%  softirqs.CPU76.NET_RX
      4963 ±153%   +1267.7%      67889 ± 29%  softirqs.CPU77.NET_RX
      6960 ±111%    +883.5%      68457 ± 24%  softirqs.CPU78.NET_RX
      7773 ±  7%     +21.4%       9437 ± 12%  softirqs.CPU78.RCU
      9181 ±  3%     +18.7%      10896 ±  7%  softirqs.CPU78.SCHED
      7508 ± 83%    +599.5%      52524 ± 13%  softirqs.CPU79.NET_RX
     24309 ± 43%    +355.9%     110836 ± 11%  softirqs.CPU8.NET_RX
      9386           +18.4%      11110 ±  9%  softirqs.CPU8.SCHED
      7213 ± 36%    +581.0%      49121 ± 23%  softirqs.CPU80.NET_RX
      8464 ±114%    +638.1%      62474 ± 24%  softirqs.CPU81.NET_RX
      9082 ±  4%     +13.0%      10264 ±  4%  softirqs.CPU81.SCHED
      9317 ± 67%    +474.4%      53517 ± 34%  softirqs.CPU82.NET_RX
      9424 ±  3%     +13.3%      10681 ±  7%  softirqs.CPU82.SCHED
     14526 ± 39%    +350.9%      65496 ± 42%  softirqs.CPU83.NET_RX
      7002 ± 77%    +791.7%      62441 ± 46%  softirqs.CPU84.NET_RX
     11867 ± 61%    +381.0%      57077 ± 36%  softirqs.CPU85.NET_RX
      7553 ± 76%    +752.2%      64370 ± 31%  softirqs.CPU86.NET_RX
     15523 ± 82%    +322.8%      65638 ± 35%  softirqs.CPU87.NET_RX
      9254 ±  2%      +9.3%      10117 ±  5%  softirqs.CPU87.SCHED
      4728 ± 43%   +1256.9%      64167 ± 64%  softirqs.CPU88.NET_RX
     12545 ± 96%    +335.6%      54650 ± 46%  softirqs.CPU89.NET_RX
     23303 ± 46%    +307.1%      94858 ± 27%  softirqs.CPU9.NET_RX
      9478 ±  2%     +19.5%      11324 ±  5%  softirqs.CPU9.SCHED
     16165 ± 43%    +249.1%      56432 ± 42%  softirqs.CPU91.NET_RX
     18285 ± 64%    +249.3%      63878 ± 46%  softirqs.CPU92.NET_RX
      7378 ±105%    +634.2%      54177 ± 53%  softirqs.CPU93.NET_RX
      5036 ±145%   +1383.9%      74734 ± 52%  softirqs.CPU94.NET_RX
      9235 ±  2%     +16.4%      10747 ±  8%  softirqs.CPU94.SCHED
      3753 ±111%   +1215.1%      49360 ± 50%  softirqs.CPU95.NET_RX
   1098222          +536.8%    6992957 ±  2%  softirqs.NET_RX
    823060           +11.5%     917430 ±  4%  softirqs.RCU
    906425           +11.7%    1012124        softirqs.SCHED
     96956           +66.2%     161160 ±  5%  softirqs.TIMER
     62.15 ±  6%     -14.3       47.90 ±  6%  perf-profile.calltrace.cycles-pp.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
     62.70 ±  6%     -12.9       49.83 ±  6%  perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
     62.86 ±  6%     -11.5       51.39 ±  6%  perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
     62.87 ±  6%     -11.5       51.40 ±  6%  perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     62.87 ±  6%     -11.4       51.45 ±  6%  perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     62.87 ±  6%     -11.4       51.45 ±  6%  perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     39.79 ±  7%     -10.5       29.28 ±  6%  perf-profile.calltrace.cycles-pp.__inet_check_established.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
     17.61 ±  7%      -3.4       14.20 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
     16.14 ±  7%      -3.4       12.75 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__inet_check_established.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect
      1.26 ±  6%      -0.3        1.00 ±  6%  perf-profile.calltrace.cycles-pp.inet_ehashfn.__inet_check_established.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect
      0.69 ±  7%      -0.2        0.48 ± 45%  perf-profile.calltrace.cycles-pp.___might_sleep.__inet_hash_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
      0.00            +0.6        0.59 ±  9%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      0.00            +0.6        0.60 ±  8%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.60 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.65 ± 12%  perf-profile.calltrace.cycles-pp.__sock_create.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.69 ± 10%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.inet_shutdown
      0.00            +0.7        0.71 ± 32%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      0.00            +0.7        0.71 ±  9%  perf-profile.calltrace.cycles-pp.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.72 ±  9%  perf-profile.calltrace.cycles-pp.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.73 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.76 ± 10%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.inet_shutdown.__sys_shutdown
      0.00            +0.8        0.79 ±  9%  perf-profile.calltrace.cycles-pp.__tcp_close.tcp_close.inet_release.__sock_release.sock_close
      0.00            +0.8        0.82 ±  9%  perf-profile.calltrace.cycles-pp.tcp_close.inet_release.__sock_release.sock_close.__fput
      0.00            +0.9        0.88 ± 10%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.inet_shutdown.__sys_shutdown.__x64_sys_shutdown
      0.00            +0.9        0.88 ± 10%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.inet_shutdown.__sys_shutdown.__x64_sys_shutdown.do_syscall_64
      0.00            +0.9        0.90 ± 10%  perf-profile.calltrace.cycles-pp.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.90 ± 10%  perf-profile.calltrace.cycles-pp.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.44 ± 45%      +1.0        1.41 ± 24%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.0        1.00 ±  7%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      0.00            +1.0        1.01 ±  9%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect
      0.00            +1.0        1.03 ±  8%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect
      0.00            +1.0        1.04 ±  7%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
      0.00            +1.0        1.05 ±  9%  perf-profile.calltrace.cycles-pp.inet_release.__sock_release.sock_close.__fput.task_work_run
      0.00            +1.1        1.08 ± 10%  perf-profile.calltrace.cycles-pp.__sock_release.sock_close.__fput.task_work_run.exit_to_user_mode_prepare
      0.00            +1.1        1.09 ± 10%  perf-profile.calltrace.cycles-pp.sock_close.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00            +1.1        1.12 ±  9%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect
      0.00            +1.1        1.13 ± 10%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process
      0.00            +1.1        1.15 ± 10%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process
      0.00            +1.2        1.17 ± 10%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv
      0.00            +1.2        1.18 ± 10%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
      0.00            +1.2        1.20 ± 10%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
      0.09 ±223%      +1.2        1.30 ± 23%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.09 ±223%      +1.2        1.30 ± 23%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt
      0.09 ±223%      +1.2        1.30 ± 22%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.3        1.29 ±  8%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      0.00            +1.3        1.29 ±  8%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      0.00            +1.3        1.31 ±  9%  perf-profile.calltrace.cycles-pp.inet_shutdown.__sys_shutdown.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.3        1.35 ±  9%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect
      0.00            +1.4        1.38 ±  9%  perf-profile.calltrace.cycles-pp.__sys_shutdown.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.4        1.38 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.4        1.39 ±  9%  perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
      0.00            +1.4        1.43 ± 10%  perf-profile.calltrace.cycles-pp.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      0.00            +1.5        1.47 ±  9%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect
      0.00            +1.5        1.48 ±  9%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect
      0.00            +1.5        1.49 ±  9%  perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      0.00            +1.5        1.49 ± 16%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
      0.00            +1.5        1.51 ±  9%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      0.00            +1.5        1.55 ±  9%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00            +1.6        1.59 ±  6%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      0.00            +1.6        1.59 ±  9%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00            +1.6        1.65 ±  9%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00            +1.7        1.66 ±  9%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
      0.00            +1.7        1.66 ±  9%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00            +1.7        1.68 ±  9%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.7        1.69 ±  9%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.7        1.70 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.54 ± 11%      +1.8        4.32 ± 52%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.00            +2.7        2.75 ± 17%  perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
      0.00            +2.8        2.78 ± 17%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
      0.00            +2.8        2.79 ± 18%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
      0.00            +2.8        2.81 ± 17%  perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
      0.00            +2.9        2.86 ± 17%  perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
      0.00            +3.1        3.09 ± 20%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack
      0.00            +3.2        3.19 ± 14%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq
      0.00            +3.2        3.25 ± 17%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack
      0.00            +3.4        3.41 ± 11%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
      0.00            +3.4        3.43 ± 11%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
      0.00            +3.4        3.44 ± 11%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
      0.00            +3.5        3.47 ± 11%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
      0.00            +3.5        3.49 ± 11%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
     62.50 ±  6%     -14.4       48.10 ±  6%  perf-profile.children.cycles-pp.__inet_hash_connect
     62.70 ±  6%     -12.9       49.84 ±  6%  perf-profile.children.cycles-pp.tcp_v4_connect
     62.86 ±  6%     -11.5       51.39 ±  6%  perf-profile.children.cycles-pp.__inet_stream_connect
     62.87 ±  6%     -11.5       51.40 ±  6%  perf-profile.children.cycles-pp.inet_stream_connect
     62.87 ±  6%     -11.4       51.45 ±  6%  perf-profile.children.cycles-pp.__sys_connect
     62.87 ±  6%     -11.4       51.45 ±  6%  perf-profile.children.cycles-pp.__x64_sys_connect
     39.97 ±  7%     -10.5       29.42 ±  6%  perf-profile.children.cycles-pp.__inet_check_established
     17.71 ±  7%      -3.2       14.53 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_bh
     16.31 ±  7%      -2.8       13.49 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock
      1.26 ±  6%      -0.2        1.03 ±  6%  perf-profile.children.cycles-pp.inet_ehashfn
      0.22 ±  8%      -0.1        0.16 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_bh
      0.06 ±  9%      +0.0        0.10 ± 49%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.security_sk_free
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.memset_erms
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__ksize
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.evict
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.ipv4_mtu
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.__x64_sys_getsockopt
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.inet_ehash_insert
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.select_idle_sibling
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.kmem_cache_alloc_trace
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__inet_twsk_schedule
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__skb_datagram_iter
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.__x64_sys_setsockopt
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.__sys_setsockopt
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.__x64_sys_close
      0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.ip_send_skb
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.tcp_get_metrics
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.tcp_timewait_state_process
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.get_partial_node
      0.00            +0.1        0.07 ± 16%  perf-profile.children.cycles-pp.security_sk_clone
      0.00            +0.1        0.07 ± 16%  perf-profile.children.cycles-pp.apparmor_sk_clone_security
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.inet_csk_clear_xmit_timers
      0.07 ± 11%      +0.1        0.13 ± 24%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.skb_release_head_state
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.tcp_finish_connect
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.call_rcu
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.tcp_connect_init
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.apparmor_sk_alloc_security
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.inet_csk_reqsk_queue_hash_add
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.sk_stop_timer
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.__skb_clone
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.tcp_v4_destroy_sock
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.00            +0.1        0.07 ±  9%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.09 ± 10%      +0.1        0.16 ± 27%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.security_file_alloc
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.skb_release_all
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.00            +0.1        0.08 ± 18%  perf-profile.children.cycles-pp.inet_twsk_kill
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.__fget_light
      0.09 ± 13%      +0.1        0.17 ± 28%  perf-profile.children.cycles-pp.sched_clock
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.__sys_getsockname
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.00            +0.1        0.08 ± 15%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.fib_table_lookup
      0.00            +0.1        0.08 ±  7%  perf-profile.children.cycles-pp.__x64_sys_getsockname
      0.00            +0.1        0.08 ± 22%  perf-profile.children.cycles-pp.ktime_get_with_offset
      0.10 ± 15%      +0.1        0.18 ± 27%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.__memcg_kmem_charge
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.inet_twsk_alloc
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.__kmalloc_reserve
      0.00            +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.security_sk_alloc
      0.00            +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.00            +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.ip_finish_output
      0.00            +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.inet_csk_destroy_sock
      0.00            +0.1        0.10 ± 16%  perf-profile.children.cycles-pp.inet_twsk_deschedule_put
      0.00            +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.update_curr
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.skb_release_data
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.set_next_entity
      0.00            +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.d_alloc_pseudo
      0.00            +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.__d_alloc
      0.00            +0.1        0.10 ± 11%  perf-profile.children.cycles-pp.drain_obj_stock
      0.00            +0.1        0.11 ± 10%  perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
      0.00            +0.1        0.11 ± 13%  perf-profile.children.cycles-pp.allocate_slab
      0.00            +0.1        0.12 ± 14%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.00            +0.1        0.12 ± 19%  perf-profile.children.cycles-pp.memcpy_erms
      0.00            +0.1        0.12 ± 14%  perf-profile.children.cycles-pp.tcp_init_transfer
      0.07 ± 11%      +0.1        0.19 ± 38%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.1        0.13 ± 12%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.00            +0.1        0.13 ± 13%  perf-profile.children.cycles-pp.tcp_set_state
      0.11 ± 12%      +0.1        0.25 ± 15%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.sk_reset_timer
      0.04 ± 71%      +0.1        0.18 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.1        0.15 ± 11%  perf-profile.children.cycles-pp.poll_idle
      0.00            +0.2        0.15 ± 16%  perf-profile.children.cycles-pp.netif_rx_internal
      0.00            +0.2        0.15 ± 10%  perf-profile.children.cycles-pp.lock_timer_base
      0.00            +0.2        0.15 ± 20%  perf-profile.children.cycles-pp.tcp_v4_send_synack
      0.00            +0.2        0.16 ± 16%  perf-profile.children.cycles-pp.netif_rx
      0.00            +0.2        0.16 ± 11%  perf-profile.children.cycles-pp.__slab_free
      0.00            +0.2        0.16 ±  9%  perf-profile.children.cycles-pp.tcp_send_fin
      0.00            +0.2        0.17 ±  8%  perf-profile.children.cycles-pp.ip_send_unicast_reply
      0.00            +0.2        0.17 ±  9%  perf-profile.children.cycles-pp.refill_obj_stock
      0.00            +0.2        0.18 ± 10%  perf-profile.children.cycles-pp.tcp_done
      0.00            +0.2        0.18 ±  6%  perf-profile.children.cycles-pp.dequeue_entity
      0.00            +0.2        0.18 ±  9%  perf-profile.children.cycles-pp.tcp_v4_send_ack
      0.00            +0.2        0.18 ± 12%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.00            +0.2        0.18 ± 15%  perf-profile.children.cycles-pp.mod_timer
      0.00            +0.2        0.18 ±  7%  perf-profile.children.cycles-pp.dst_release
      0.00            +0.2        0.19 ± 10%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +0.2        0.19 ±  8%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.2        0.19 ±  8%  perf-profile.children.cycles-pp.lock_sock_nested
      0.00            +0.2        0.20 ±  6%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.00            +0.2        0.20 ± 12%  perf-profile.children.cycles-pp.tcp_time_wait
      0.00            +0.2        0.20 ±  7%  perf-profile.children.cycles-pp.ip_route_output_flow
      0.00            +0.2        0.20 ± 12%  perf-profile.children.cycles-pp.sock_alloc_inode
      0.00            +0.2        0.22 ± 10%  perf-profile.children.cycles-pp.__alloc_file
      0.00            +0.2        0.22 ± 10%  perf-profile.children.cycles-pp.alloc_empty_file
      0.00            +0.2        0.23 ± 10%  perf-profile.children.cycles-pp.alloc_file
      0.00            +0.2        0.23 ± 19%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.00            +0.2        0.23 ± 10%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.00            +0.2        0.23 ±  7%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      0.00            +0.2        0.24 ± 19%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.00            +0.2        0.24 ±  8%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.00            +0.2        0.24 ±  8%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +0.2        0.24 ± 10%  perf-profile.children.cycles-pp.tcp_delack_timer_handler
      0.00            +0.2        0.25 ± 12%  perf-profile.children.cycles-pp.__kfree_skb
      0.07 ± 20%      +0.2        0.32 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.3        0.25 ± 17%  perf-profile.children.cycles-pp.___slab_alloc
      0.00            +0.3        0.26 ±  8%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      0.03 ±100%      +0.3        0.29 ± 15%  perf-profile.children.cycles-pp.kthread
      0.03 ±100%      +0.3        0.29 ± 15%  perf-profile.children.cycles-pp.ret_from_fork
      0.00            +0.3        0.26 ± 10%  perf-profile.children.cycles-pp.alloc_inode
      0.00            +0.3        0.26 ± 11%  perf-profile.children.cycles-pp.wait_woken
      0.00            +0.3        0.26 ± 16%  perf-profile.children.cycles-pp.__slab_alloc
      0.00            +0.3        0.26 ± 10%  perf-profile.children.cycles-pp.tcp_delack_timer
      0.00            +0.3        0.27 ±  7%  perf-profile.children.cycles-pp.__dentry_kill
      0.01 ±223%      +0.3        0.28 ± 12%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.02 ±141%      +0.3        0.30 ± 10%  perf-profile.children.cycles-pp.call_timer_fn
      0.00            +0.3        0.29 ± 10%  perf-profile.children.cycles-pp.__alloc_skb
      0.00            +0.3        0.29 ± 12%  perf-profile.children.cycles-pp.loopback_xmit
      0.04 ± 71%      +0.3        0.33 ± 10%  perf-profile.children.cycles-pp.run_timer_softirq
      0.00            +0.3        0.29 ± 16%  perf-profile.children.cycles-pp.sk_clone_lock
      0.00            +0.3        0.30 ± 15%  perf-profile.children.cycles-pp.inet_csk_clone_lock
      0.00            +0.3        0.31 ±  8%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.00            +0.3        0.31 ± 12%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.00            +0.3        0.32 ±  9%  perf-profile.children.cycles-pp.sock_alloc
      0.01 ±223%      +0.3        0.33 ±  9%  perf-profile.children.cycles-pp.sk_wait_data
      0.00            +0.3        0.33 ±  7%  perf-profile.children.cycles-pp.__sk_destruct
      0.00            +0.3        0.33 ± 11%  perf-profile.children.cycles-pp.schedule_idle
      0.00            +0.3        0.33 ± 13%  perf-profile.children.cycles-pp.sk_prot_alloc
      0.00            +0.3        0.33 ± 14%  perf-profile.children.cycles-pp.tcp_create_openreq_child
      0.07 ± 18%      +0.3        0.40 ±  7%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.3        0.34 ±  9%  perf-profile.children.cycles-pp.inet_csk_accept
      0.00            +0.3        0.34 ± 13%  perf-profile.children.cycles-pp.sk_alloc
      0.00            +0.3        0.35 ±  8%  perf-profile.children.cycles-pp.rcu_do_batch
      0.01 ±223%      +0.4        0.39 ±  9%  perf-profile.children.cycles-pp.alloc_file_pseudo
      0.01 ±223%      +0.4        0.39 ±  8%  perf-profile.children.cycles-pp.sock_alloc_file
      0.05 ± 45%      +0.4        0.43 ±  7%  perf-profile.children.cycles-pp.schedule
      0.04 ± 44%      +0.4        0.43 ±  9%  perf-profile.children.cycles-pp.schedule_timeout
      0.02 ±142%      +0.4        0.41 ± 12%  perf-profile.children.cycles-pp.tcp_conn_request
      0.01 ±223%      +0.4        0.41 ± 10%  perf-profile.children.cycles-pp.tcp_ack
      0.00            +0.4        0.40 ± 12%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.00            +0.4        0.41 ± 10%  perf-profile.children.cycles-pp.inet_accept
      0.00            +0.4        0.41 ±  6%  perf-profile.children.cycles-pp.tcp_data_queue
      0.04 ± 75%      +0.4        0.45 ± 10%  perf-profile.children.cycles-pp.tcp_child_process
      0.00            +0.4        0.42 ± 13%  perf-profile.children.cycles-pp.inet_create
      0.00            +0.5        0.46 ± 12%  perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
      0.06 ± 13%      +0.5        0.55 ±  8%  perf-profile.children.cycles-pp.tcp_recvmsg_locked
      0.02 ±141%      +0.5        0.51 ± 11%  perf-profile.children.cycles-pp.tcp_check_req
      0.00            +0.5        0.49 ± 10%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.06 ± 15%      +0.5        0.56 ±  8%  perf-profile.children.cycles-pp.tcp_recvmsg
      0.06 ± 13%      +0.5        0.56 ±  8%  perf-profile.children.cycles-pp.inet_recvmsg
      0.06 ± 18%      +0.5        0.56 ±  9%  perf-profile.children.cycles-pp.try_to_wake_up
      0.06 ± 17%      +0.5        0.58 ±  9%  perf-profile.children.cycles-pp.__wake_up_common
      0.07 ± 10%      +0.5        0.60 ±  8%  perf-profile.children.cycles-pp.__sys_recvfrom
      0.07 ± 11%      +0.5        0.60 ±  9%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
      0.07 ± 17%      +0.6        0.63 ±  9%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.07 ± 21%      +0.6        0.65 ±  9%  perf-profile.children.cycles-pp.sock_def_readable
      0.06 ±  9%      +0.6        0.65 ± 11%  perf-profile.children.cycles-pp.__sock_create
      0.06 ±  6%      +0.6        0.67 ± 10%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.00            +0.6        0.62 ±  9%  perf-profile.children.cycles-pp.kmem_cache_free
      0.09 ±  8%      +0.7        0.74 ±  9%  perf-profile.children.cycles-pp.__sched_text_start
      0.04 ± 45%      +0.7        0.71 ±  9%  perf-profile.children.cycles-pp.__sys_accept4_file
      0.04 ± 45%      +0.7        0.72 ±  9%  perf-profile.children.cycles-pp.__sys_accept4
      0.04 ± 45%      +0.7        0.73 ±  9%  perf-profile.children.cycles-pp.__x64_sys_accept
      0.05 ± 47%      +0.8        0.83 ±  9%  perf-profile.children.cycles-pp.tcp_rcv_established
      0.00            +0.8        0.79 ±  9%  perf-profile.children.cycles-pp.__tcp_close
      0.09 ± 10%      +0.8        0.90 ± 10%  perf-profile.children.cycles-pp.__x64_sys_socket
      0.09 ± 10%      +0.8        0.90 ± 10%  perf-profile.children.cycles-pp.__sys_socket
      0.00            +0.8        0.82 ±  9%  perf-profile.children.cycles-pp.tcp_close
      0.00            +1.1        1.05 ± 10%  perf-profile.children.cycles-pp.inet_release
      0.54 ±  7%      +1.1        1.61 ± 21%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.01 ±223%      +1.1        1.09 ± 10%  perf-profile.children.cycles-pp.sock_close
      0.00            +1.1        1.08 ± 10%  perf-profile.children.cycles-pp.__sock_release
      0.16 ± 12%      +1.2        1.39 ±  9%  perf-profile.children.cycles-pp.tcp_connect
      0.15 ± 11%      +1.3        1.43 ± 10%  perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process
      0.00            +1.3        1.31 ±  9%  perf-profile.children.cycles-pp.inet_shutdown
      0.00            +1.4        1.38 ±  9%  perf-profile.children.cycles-pp.__sys_shutdown
      0.00            +1.4        1.38 ±  8%  perf-profile.children.cycles-pp.__x64_sys_shutdown
      0.11 ± 16%      +1.4        1.51 ±  9%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
      0.06 ±  9%      +1.5        1.55 ±  9%  perf-profile.children.cycles-pp.__fput
      0.07 ± 11%      +1.5        1.60 ±  9%  perf-profile.children.cycles-pp.task_work_run
      0.12 ± 15%      +1.5        1.66 ±  8%  perf-profile.children.cycles-pp.tcp_sendmsg
      0.12 ± 16%      +1.6        1.68 ±  9%  perf-profile.children.cycles-pp.sock_sendmsg
      0.12 ± 16%      +1.6        1.69 ±  9%  perf-profile.children.cycles-pp.__sys_sendto
      0.12 ± 16%      +1.6        1.70 ±  8%  perf-profile.children.cycles-pp.__x64_sys_sendto
      0.07 ±  8%      +1.6        1.65 ±  9%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.07 ± 12%      +1.6        1.67 ±  9%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.17 ± 10%      +1.7        1.86 ±  8%  perf-profile.children.cycles-pp.__release_sock
      0.18 ± 10%      +1.8        1.95 ±  8%  perf-profile.children.cycles-pp.release_sock
      2.99 ± 10%      +1.9        4.87 ± 46%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.24 ± 11%      +2.4        2.62 ±  9%  perf-profile.children.cycles-pp.tcp_rcv_state_process
      0.12 ± 14%      +2.5        2.60 ±  8%  perf-profile.children.cycles-pp.tcp_write_xmit
      0.12 ± 14%      +2.5        2.62 ±  8%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
      0.30 ± 11%      +3.2        3.47 ±  9%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      0.79 ±  5%      +3.6        4.34 ±  9%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.34 ± 11%      +3.6        3.91 ±  9%  perf-profile.children.cycles-pp.do_softirq
      0.29 ± 12%      +3.6        3.88 ±  9%  perf-profile.children.cycles-pp.tcp_v4_rcv
      0.30 ± 12%      +3.6        3.92 ±  9%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      0.30 ± 12%      +3.6        3.94 ±  9%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      0.30 ± 12%      +3.7        3.97 ±  8%  perf-profile.children.cycles-pp.ip_local_deliver
      0.30 ± 11%      +3.7        4.04 ±  8%  perf-profile.children.cycles-pp.ip_rcv
      0.32 ± 12%      +3.8        4.13 ±  8%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      0.32 ± 11%      +3.9        4.23 ±  9%  perf-profile.children.cycles-pp.process_backlog
      0.32 ± 12%      +3.9        4.25 ±  9%  perf-profile.children.cycles-pp.ip_finish_output2
      0.33 ± 12%      +3.9        4.28 ±  9%  perf-profile.children.cycles-pp.net_rx_action
      0.33 ± 12%      +4.1        4.41 ±  8%  perf-profile.children.cycles-pp.ip_output
      0.35 ± 11%      +4.2        4.56 ±  8%  perf-profile.children.cycles-pp.__ip_queue_xmit
      0.37 ± 10%      +4.5        4.88 ±  9%  perf-profile.children.cycles-pp.__tcp_transmit_skb
      0.79 ±  8%      +4.6        5.37 ±  7%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.79 ±  8%      +4.8        5.56 ±  7%  perf-profile.children.cycles-pp.__softirqentry_text_start
      2.82 ± 11%      +5.2        8.04 ± 21%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
     22.39 ±  8%      -6.9       15.54 ±  6%  perf-profile.self.cycles-pp.__inet_check_established
     17.18 ±  7%      -2.8       14.36 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_bh
     16.03 ±  7%      -2.8       13.23 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock
      1.17 ±  6%      -0.2        0.95 ±  6%  perf-profile.self.cycles-pp.inet_ehashfn
      0.18 ±  9%      -0.0        0.13 ± 10%  perf-profile.self.cycles-pp._raw_spin_unlock_bh
      0.00            +0.1        0.05 ± 13%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.memset_erms
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.set_next_entity
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.fib_table_lookup
      0.00            +0.1        0.06 ± 26%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.03 ±101%      +0.1        0.09 ± 13%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.page_counter_try_charge
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.refill_obj_stock
      0.00            +0.1        0.07 ± 19%  perf-profile.self.cycles-pp.apparmor_sk_clone_security
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.05 ± 45%      +0.1        0.12 ± 24%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.07 ± 13%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.1        0.07 ± 18%  perf-profile.self.cycles-pp.__alloc_skb
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.__fget_light
      0.04 ± 44%      +0.1        0.12 ± 24%  perf-profile.self.cycles-pp.do_idle
      0.08 ± 14%      +0.1        0.16 ± 28%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.1        0.08 ± 18%  perf-profile.self.cycles-pp.___slab_alloc
      0.00            +0.1        0.08 ± 11%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.sk_alloc
      0.00            +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.ip_finish_output2
      0.00            +0.1        0.09 ± 10%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      0.00            +0.1        0.10 ± 14%  perf-profile.self.cycles-pp.skb_release_data
      0.00            +0.1        0.10 ± 13%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.00            +0.1        0.11 ± 12%  perf-profile.self.cycles-pp.tcp_ack
      0.00            +0.1        0.11 ± 22%  perf-profile.self.cycles-pp.update_rq_clock
      0.00            +0.1        0.12 ± 19%  perf-profile.self.cycles-pp.memcpy_erms
      0.00            +0.1        0.12 ± 15%  perf-profile.self.cycles-pp.poll_idle
      0.11 ± 12%      +0.1        0.24 ± 14%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.2        0.15 ± 13%  perf-profile.self.cycles-pp.__sched_text_start
      0.00            +0.2        0.15 ±  8%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.00            +0.2        0.15 ± 11%  perf-profile.self.cycles-pp.__slab_free
      0.00            +0.2        0.18 ± 15%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.00            +0.2        0.18 ±  8%  perf-profile.self.cycles-pp.dst_release
      0.00            +0.2        0.20 ±  8%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.07 ± 20%      +0.2        0.32 ± 12%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.3        0.30 ±  8%  perf-profile.self.cycles-pp.kmem_cache_free
      0.00            +0.4        0.38 ± 11%  perf-profile.self.cycles-pp.__inet_lookup_established





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.11.0-rc7-00144-gff0d41306d8c" of type "text/plain" (172449 bytes)

View attachment "job-script" of type "text/plain" (8176 bytes)

View attachment "job.yaml" of type "text/plain" (5589 bytes)

View attachment "reproduce" of type "text/plain" (341 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ