lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20200512030929.GA5770@shao2-debian>
Date:   Tue, 12 May 2020 11:09:29 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Jia-Ju Bai <baijiaju1990@...il.com>
Cc:     shaggy@...nel.org, Markus.Elfring@....de,
        jfs-discussion@...ts.sourceforge.net, linux-kernel@...r.kernel.org,
        Jia-Ju Bai <baijiaju1990@...il.com>, lkp@...ts.01.org
Subject: [fs] 05c5a0273b: netperf.Throughput_total_tps -71.8% regression

Greeting,

FYI, we noticed a -71.8% regression of netperf.Throughput_total_tps due to commit:


commit: 05c5a0273bc25627b9e9bbebc858b74e88f49c5b ("[PATCH v2] fs: jfs: fix a possible data race in txBegin()")
url: https://github.com/0day-ci/linux/commits/Jia-Ju-Bai/fs-jfs-fix-a-possible-data-race-in-txBegin/20200506-064230
base: https://github.com/kleikamp/linux-shaggy jfs-next

in testcase: netperf
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 1
	cluster: cs-localhost
	test: TCP_CRR
	cpufreq_governor: performance
	ucode: 0x2000065

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/1/debian-x86_64-20191114.cgz/300s/lkp-skl-2sp7/TCP_CRR/netperf/0x2000065

commit: 
  7aba5dcc23 ("jfs: Replace zero-length array with flexible-array member")
  05c5a0273b ("fs: jfs: fix a possible data race in txBegin()")

7aba5dcc234635b4 05c5a0273bc25627b9e9bbebc85 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :4          100%           4:4     kmsg.IP-Config:Failed_to_open_erspan0
           :4          100%           4:4     kmsg.IP-Config:Failed_to_open_gretap0
           :4          100%           4:4     kmsg.kvm:already_loaded_the_other_module
         %stddev     %change         %stddev
             \          |                \  
     32033           -71.8%       9048        netperf.Throughput_total_tps
     32033           -71.8%       9048        netperf.Throughput_tps
     34270 ± 19%     -32.2%      23234 ± 11%  netperf.time.involuntary_context_switches
     87.50            -1.7%      86.00        netperf.time.percent_of_cpu_this_job_got
    237.76            +7.1%     254.71        netperf.time.system_time
     30.66           -66.6%      10.25 ±  3%  netperf.time.user_time
  17926206           -69.8%    5405643        netperf.time.voluntary_context_switches
   9610194           -71.8%    2714625        netperf.workload
      0.25            -0.1        0.12 ±  3%  mpstat.cpu.all.usr%
    181085           -60.2%      72003        vmstat.system.cs
     24.19 ±  2%      +8.7%      26.30        boot-time.boot
     15.27 ±  2%     +12.6%      17.18        boot-time.dhcp
      1449 ±  2%     +10.6%       1602        boot-time.idle
 1.589e+08           -16.1%  1.334e+08        cpuidle.C1.time
  10004352           -12.6%    8738852        cpuidle.C1.usage
  24458189 ±  3%     -76.6%    5718567 ± 15%  cpuidle.POLL.time
  18015660           -95.3%     840897 ± 17%  cpuidle.POLL.usage
     42120 ±  2%      +9.9%      46309 ±  5%  softirqs.CPU10.SCHED
     41095 ±  4%     +10.9%      45593 ±  5%  softirqs.CPU27.SCHED
     25223 ± 54%     +70.0%      42867        softirqs.CPU36.SCHED
    166155 ±  8%     -10.1%     149301        softirqs.CPU65.RCU
  28947742           -71.8%    8152772        softirqs.NET_RX
  12415372            -9.2%   11268919        softirqs.RCU
   8089600 ± 11%     -23.4%    6200320 ±  8%  meminfo.DirectMap2M
    503148 ±  8%     -41.3%     295276 ±  8%  meminfo.DirectMap4k
     75713           +34.4%     101777        meminfo.KReclaimable
     22720           +11.6%      25360        meminfo.Percpu
     75713           +34.4%     101777        meminfo.SReclaimable
    152955           +16.4%     178056        meminfo.SUnreclaim
    228668           +22.4%     279834        meminfo.Slab
    137288           +17.8%     161678        meminfo.VmallocUsed
     33315 ± 18%     +75.5%      58482 ± 12%  numa-meminfo.node0.KReclaimable
     33315 ± 18%     +75.5%      58482 ± 12%  numa-meminfo.node0.SReclaimable
     74713 ± 16%     +34.8%     100736 ± 11%  numa-meminfo.node0.SUnreclaim
    108029 ± 16%     +47.4%     159219 ± 11%  numa-meminfo.node0.Slab
    101746 ± 44%     +68.1%     171066 ± 12%  numa-meminfo.node1.Active
    101684 ± 44%     +68.1%     170963 ± 12%  numa-meminfo.node1.Active(anon)
     60012 ± 62%     +94.5%     116727 ± 18%  numa-meminfo.node1.AnonHugePages
     98606 ± 46%     +69.1%     166702 ± 12%  numa-meminfo.node1.AnonPages
      5648            +3.4%       5842        proc-vmstat.nr_shmem
     18920           +34.5%      25446        proc-vmstat.nr_slab_reclaimable
     38238           +16.4%      44513        proc-vmstat.nr_slab_unreclaimable
    895359 ±  5%     -17.9%     734777        proc-vmstat.numa_hit
    872027 ±  5%     -18.4%     711460        proc-vmstat.numa_local
      1080 ±  9%     +27.4%       1375 ±  4%  proc-vmstat.pgactivate
   1530529 ± 15%     -42.6%     878573 ±  2%  proc-vmstat.pgalloc_normal
   1505342 ± 15%     -43.5%     849856 ±  2%  proc-vmstat.pgfree
      8335 ± 18%     +75.4%      14620 ± 12%  numa-vmstat.node0.nr_slab_reclaimable
     18678 ± 16%     +34.8%      25183 ± 11%  numa-vmstat.node0.nr_slab_unreclaimable
    647500 ± 10%     -21.7%     507083 ±  8%  numa-vmstat.node0.numa_hit
    494986 ± 13%     -27.6%     358327 ± 13%  numa-vmstat.node0.numa_local
     25431 ± 44%     +68.1%      42742 ± 12%  numa-vmstat.node1.nr_active_anon
     24661 ± 46%     +69.0%      41679 ± 12%  numa-vmstat.node1.nr_anon_pages
     25431 ± 44%     +68.1%      42742 ± 12%  numa-vmstat.node1.nr_zone_active_anon
    725209 ±  8%     -16.4%     606495 ±  8%  numa-vmstat.node1.numa_hit
    702021 ±  8%     -17.9%     576047 ± 10%  numa-vmstat.node1.numa_local
     49724 ± 16%     -42.5%      28572 ± 16%  sched_debug.cfs_rq:/.load.avg
    113065 ± 11%     -24.3%      85601 ± 10%  sched_debug.cfs_rq:/.load.stddev
     57662 ±  7%     -47.5%      30301 ±  9%  sched_debug.cfs_rq:/.min_vruntime.max
      6576 ±  8%     -34.6%       4300 ± 19%  sched_debug.cfs_rq:/.min_vruntime.min
     10636 ±  5%     -39.1%       6478 ± 15%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.16 ±  8%     -29.7%       0.11 ±  8%  sched_debug.cfs_rq:/.nr_running.avg
      0.36 ±  2%     -13.7%       0.31 ±  4%  sched_debug.cfs_rq:/.nr_running.stddev
     35.45 ± 20%     -52.9%      16.69        sched_debug.cfs_rq:/.runnable_load_avg.avg
     95.13 ± 14%     -32.7%      63.99        sched_debug.cfs_rq:/.runnable_load_avg.stddev
     50290 ± 14%     -43.7%      28330 ± 15%  sched_debug.cfs_rq:/.runnable_weight.avg
    113143 ± 10%     -24.9%      84975 ± 10%  sched_debug.cfs_rq:/.runnable_weight.stddev
     33703 ± 43%     -68.0%      10796 ± 42%  sched_debug.cfs_rq:/.spread0.max
     10636 ±  5%     -39.1%       6478 ± 15%  sched_debug.cfs_rq:/.spread0.stddev
    140.66 ± 10%     -17.8%     115.68 ±  7%  sched_debug.cfs_rq:/.util_avg.avg
    247.04 ±  4%     -11.6%     218.36 ±  4%  sched_debug.cfs_rq:/.util_avg.stddev
     51.66 ± 11%     -41.4%      30.29 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    179.93 ±  3%     -26.3%     132.58 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      1.26 ±  2%     +10.1%       1.39 ±  3%  sched_debug.cpu.clock.stddev
      1.26 ±  2%     +10.2%       1.39 ±  3%  sched_debug.cpu.clock_task.stddev
    365656 ±  9%     -57.9%     154017        sched_debug.cpu.nr_switches.avg
   3644073 ± 21%     -81.6%     671977 ± 17%  sched_debug.cpu.nr_switches.max
      1315 ±  5%    +863.7%      12673 ± 91%  sched_debug.cpu.nr_switches.min
    592717 ± 15%     -75.6%     144437 ± 18%  sched_debug.cpu.nr_switches.stddev
    341.50 ± 96%    +267.0%       1253 ±  7%  interrupts.CPU12.RES:Rescheduling_interrupts
    145.00 ± 42%    +316.6%     604.00 ± 78%  interrupts.CPU15.NMI:Non-maskable_interrupts
    145.00 ± 42%    +316.6%     604.00 ± 78%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    168.25 ± 70%    +219.9%     538.25 ± 63%  interrupts.CPU16.NMI:Non-maskable_interrupts
    168.25 ± 70%    +219.9%     538.25 ± 63%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    316.25 ± 69%    +303.7%       1276 ± 83%  interrupts.CPU17.NMI:Non-maskable_interrupts
    316.25 ± 69%    +303.7%       1276 ± 83%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      3796 ± 83%     -82.0%     685.00 ±119%  interrupts.CPU18.RES:Rescheduling_interrupts
      2353 ± 49%     -84.5%     364.00 ±134%  interrupts.CPU20.RES:Rescheduling_interrupts
    107.50 ± 26%    +199.8%     322.25 ± 80%  interrupts.CPU23.NMI:Non-maskable_interrupts
    107.50 ± 26%    +199.8%     322.25 ± 80%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
     89.00 ± 14%     +90.2%     169.25 ± 23%  interrupts.CPU24.NMI:Non-maskable_interrupts
     89.00 ± 14%     +90.2%     169.25 ± 23%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
     93.25 ± 19%    +101.6%     188.00 ± 35%  interrupts.CPU31.NMI:Non-maskable_interrupts
     93.25 ± 19%    +101.6%     188.00 ± 35%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
     87.25 ± 14%     +88.3%     164.25 ± 13%  interrupts.CPU32.NMI:Non-maskable_interrupts
     87.25 ± 14%     +88.3%     164.25 ± 13%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    104.50 ± 34%    +106.9%     216.25 ± 46%  interrupts.CPU33.NMI:Non-maskable_interrupts
    104.50 ± 34%    +106.9%     216.25 ± 46%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
      1951 ± 71%     -80.2%     387.00 ±113%  interrupts.CPU33.RES:Rescheduling_interrupts
    117.25 ± 42%    +295.7%     464.00 ± 68%  interrupts.CPU51.NMI:Non-maskable_interrupts
    117.25 ± 42%    +295.7%     464.00 ± 68%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
    142.50 ± 52%    +186.7%     408.50 ± 55%  interrupts.CPU52.NMI:Non-maskable_interrupts
    142.50 ± 52%    +186.7%     408.50 ± 55%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
    320.00 ± 65%    +246.8%       1109 ± 57%  interrupts.CPU53.NMI:Non-maskable_interrupts
    320.00 ± 65%    +246.8%       1109 ± 57%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    106.75 ± 30%    +111.2%     225.50 ± 56%  interrupts.CPU67.NMI:Non-maskable_interrupts
    106.75 ± 30%    +111.2%     225.50 ± 56%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
     87.00 ± 13%    +123.6%     194.50 ± 33%  interrupts.CPU68.NMI:Non-maskable_interrupts
     87.00 ± 13%    +123.6%     194.50 ± 33%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
     23466           +23.8%      29040 ±  5%  interrupts.NMI:Non-maskable_interrupts
     23466           +23.8%      29040 ±  5%  interrupts.PMI:Performance_monitoring_interrupts
     13.69 ± 25%     -71.4%       3.92 ± 48%  perf-stat.i.MPKI
  8.78e+08           +90.0%  1.668e+09        perf-stat.i.branch-instructions
      2.30 ± 14%      -1.5        0.83 ± 33%  perf-stat.i.branch-miss-rate%
  20600243 ± 12%     -31.0%   14224189 ± 29%  perf-stat.i.branch-misses
      2.48 ± 18%      +1.7        4.22 ± 14%  perf-stat.i.cache-miss-rate%
  57749473 ± 23%     -44.2%   32231609 ± 44%  perf-stat.i.cache-references
    182988           -60.2%      72782        perf-stat.i.context-switches
      1.65 ±  5%     -39.4%       1.00 ±  5%  perf-stat.i.cpi
 7.078e+09 ±  3%     +18.6%  8.393e+09 ±  3%  perf-stat.i.cpu-cycles
     85.15            -8.1%      78.28        perf-stat.i.cpu-migrations
      0.09 ± 32%      -0.1        0.02 ±106%  perf-stat.i.dTLB-load-miss-rate%
 1.284e+09           +89.1%  2.428e+09        perf-stat.i.dTLB-loads
      0.01 ± 51%      -0.0        0.00 ±133%  perf-stat.i.dTLB-store-miss-rate%
 7.676e+08           +81.0%   1.39e+09        perf-stat.i.dTLB-stores
     28.71 ± 22%     -19.9        8.79 ± 72%  perf-stat.i.iTLB-load-miss-rate%
    661655 ± 18%     -65.2%     230579 ± 60%  perf-stat.i.iTLB-load-misses
   1672859 ± 15%     +53.1%    2560787 ± 13%  perf-stat.i.iTLB-loads
 4.377e+09           +92.6%  8.432e+09        perf-stat.i.instructions
      7111 ± 22%    +568.3%      47528 ± 35%  perf-stat.i.instructions-per-iTLB-miss
      0.61 ±  4%     +63.4%       1.00 ±  4%  perf-stat.i.ipc
      0.10 ±  3%     +18.6%       0.12 ±  3%  perf-stat.i.metric.GHz
     41.51           +84.7%      76.66        perf-stat.i.metric.M/sec
    131741 ±  7%     +48.3%     195398 ±  7%  perf-stat.i.node-load-misses
     36778 ± 31%     -55.6%      16347 ± 16%  perf-stat.i.node-stores
     13.22 ± 23%     -70.9%       3.85 ± 46%  perf-stat.overall.MPKI
      2.35 ± 12%      -1.5        0.86 ± 30%  perf-stat.overall.branch-miss-rate%
      2.34 ± 18%      +1.9        4.20 ± 14%  perf-stat.overall.cache-miss-rate%
      1.62 ±  4%     -38.4%       1.00 ±  5%  perf-stat.overall.cpi
      0.08 ± 30%      -0.1        0.02 ±103%  perf-stat.overall.dTLB-load-miss-rate%
      0.01 ± 50%      -0.0        0.00 ±132%  perf-stat.overall.dTLB-store-miss-rate%
     28.68 ± 22%     -19.9        8.73 ± 71%  perf-stat.overall.iTLB-load-miss-rate%
      6894 ± 21%    +579.7%      46861 ± 36%  perf-stat.overall.instructions-per-iTLB-miss
      0.62 ±  4%     +62.5%       1.01 ±  4%  perf-stat.overall.ipc
    138819          +581.5%     946109        perf-stat.overall.path-length
 8.751e+08           +90.0%  1.662e+09        perf-stat.ps.branch-instructions
  20537517 ± 12%     -30.9%   14182248 ± 29%  perf-stat.ps.branch-misses
  57557495 ± 23%     -44.2%   32125500 ± 44%  perf-stat.ps.cache-references
    182374           -60.2%      72538        perf-stat.ps.context-switches
 7.055e+09 ±  3%     +18.6%  8.366e+09 ±  3%  perf-stat.ps.cpu-cycles
     84.88            -8.1%      78.04        perf-stat.ps.cpu-migrations
  1.28e+09           +89.1%   2.42e+09        perf-stat.ps.dTLB-loads
 7.651e+08           +81.0%  1.385e+09        perf-stat.ps.dTLB-stores
    659450 ± 18%     -65.2%     229817 ± 60%  perf-stat.ps.iTLB-load-misses
   1667327 ± 15%     +53.1%    2552296 ± 13%  perf-stat.ps.iTLB-loads
 4.363e+09           +92.6%  8.405e+09        perf-stat.ps.instructions
    131326 ±  7%     +48.3%     194773 ±  7%  perf-stat.ps.node-load-misses
     36665 ± 31%     -55.5%      16301 ± 16%  perf-stat.ps.node-stores
 1.334e+12           +92.5%  2.568e+12        perf-stat.total.instructions
      1551 ± 13%    -100.0%       0.00        slabinfo.Acpi-Parse.active_objs
      1551 ± 13%    -100.0%       0.00        slabinfo.Acpi-Parse.num_objs
    172.50 ±  6%    -100.0%       0.00        slabinfo.UDPv6.active_objs
    172.50 ±  6%    -100.0%       0.00        slabinfo.UDPv6.num_objs
      1648 ±  9%     -42.1%     954.25 ±  2%  slabinfo.UNIX.active_objs
      1648 ±  9%     -42.1%     954.25 ±  2%  slabinfo.UNIX.num_objs
    497.25 ±  3%     -38.7%     304.75 ±  6%  slabinfo.bdev_cache.active_objs
    497.25 ±  3%     -38.7%     304.75 ±  6%  slabinfo.bdev_cache.num_objs
      3990          -100.0%       0.00        slabinfo.cred_jar.active_objs
      3990          -100.0%       0.00        slabinfo.cred_jar.num_objs
      1917           -26.9%       1400        slabinfo.dentry.active_slabs
      1917           -26.9%       1400        slabinfo.dentry.num_slabs
      2225 ±  8%    -100.0%       0.00        slabinfo.dmaengine-unmap-16.active_objs
      2225 ±  8%    -100.0%       0.00        slabinfo.dmaengine-unmap-16.num_objs
    761.25 ± 11%    -100.0%       0.00        slabinfo.file_lock_cache.active_objs
    761.25 ± 11%    -100.0%       0.00        slabinfo.file_lock_cache.num_objs
      3185 ±  2%     -12.2%       2795 ±  2%  slabinfo.files_cache.active_objs
      3185 ±  2%     -12.2%       2795 ±  2%  slabinfo.files_cache.num_objs
     19419 ±  6%     +25.2%      24310 ±  8%  slabinfo.filp.active_objs
    613.00 ±  6%     +24.4%     762.50 ±  8%  slabinfo.filp.active_slabs
     19628 ±  6%     +24.4%      24408 ±  8%  slabinfo.filp.num_objs
    613.00 ±  6%     +24.4%     762.50 ±  8%  slabinfo.filp.num_slabs
      2080 ± 11%    -100.0%       0.00        slabinfo.fsnotify_mark_connector.active_objs
      2080 ± 11%    -100.0%       0.00        slabinfo.fsnotify_mark_connector.num_objs
      1007           +58.3%       1594        slabinfo.inode_cache.active_slabs
      1007           +58.3%       1594        slabinfo.inode_cache.num_slabs
      2517           -35.4%       1625        slabinfo.kernfs_node_cache.active_slabs
      2517           -35.4%       1625        slabinfo.kernfs_node_cache.num_slabs
      4927 ±  2%     -13.8%       4248 ±  4%  slabinfo.kmalloc-1k.active_objs
      5040 ±  2%     -15.1%       4280 ±  4%  slabinfo.kmalloc-1k.num_objs
      4837 ±  4%     +28.0%       6193        slabinfo.kmalloc-2k.active_objs
      4870 ±  5%     +28.1%       6237        slabinfo.kmalloc-2k.num_objs
      1191           +19.1%       1419        slabinfo.kmalloc-4k.active_objs
      1219           +18.3%       1442        slabinfo.kmalloc-4k.num_objs
      3935 ±  4%     +14.6%       4511 ±  5%  slabinfo.kmalloc-rcl-64.active_objs
      3935 ±  4%     +14.6%       4511 ±  5%  slabinfo.kmalloc-rcl-64.num_objs
    387.00 ± 12%     -13.2%     336.00 ±  8%  slabinfo.kmem_cache.active_objs
    387.00 ± 12%     -13.2%     336.00 ±  8%  slabinfo.kmem_cache.num_objs
    759.00 ± 10%     -36.2%     484.00 ±  5%  slabinfo.kmem_cache_node.active_objs
    816.00 ± 10%     -39.2%     496.00 ±  5%  slabinfo.kmem_cache_node.num_objs
     12875          -100.0%       0.00        slabinfo.lsm_file_cache.active_objs
     12875          -100.0%       0.00        slabinfo.lsm_file_cache.num_objs
      2271           -27.9%       1639        slabinfo.mm_struct.active_objs
      2271           -27.9%       1639        slabinfo.mm_struct.num_objs
    790.50 ±  7%    -100.0%       0.00        slabinfo.mnt_cache.active_objs
    790.50 ±  7%    -100.0%       0.00        slabinfo.mnt_cache.num_objs
    123.75 ±  8%    -100.0%       0.00        slabinfo.nfs_read_data.active_objs
    123.75 ±  8%    -100.0%       0.00        slabinfo.nfs_read_data.num_objs
    294.50 ± 17%    +360.4%       1355 ± 15%  slabinfo.numa_policy.active_objs
    294.50 ± 17%    +360.4%       1355 ± 15%  slabinfo.numa_policy.num_objs
      3128           +73.2%       5418        slabinfo.pid.active_objs
      3128           +73.2%       5418        slabinfo.pid.num_objs
     10638           -13.0%       9252        slabinfo.proc_inode_cache.active_objs
     10638           -12.9%       9266        slabinfo.proc_inode_cache.num_objs
      5651           -15.6%       4771        slabinfo.shmem_inode_cache.active_objs
      5651           -15.6%       4771        slabinfo.shmem_inode_cache.num_objs
      2794           -11.6%       2469        slabinfo.signal_cache.active_objs
      2794           -11.6%       2469        slabinfo.signal_cache.num_objs
     13766 ± 10%     -46.0%       7439 ± 16%  slabinfo.sock_inode_cache.active_objs
     13865 ± 10%     -45.9%       7507 ± 16%  slabinfo.sock_inode_cache.num_objs
      4036 ±  2%     +16.8%       4713 ±  2%  slabinfo.task_delay_info.active_objs
      4036 ±  2%     +16.8%       4713 ±  2%  slabinfo.task_delay_info.num_objs
    940.25           +13.2%       1064        slabinfo.task_struct.active_objs
    947.50           -43.4%     536.00        slabinfo.task_struct.active_slabs
    947.50           +13.2%       1072        slabinfo.task_struct.num_objs
    947.50           -43.4%     536.00        slabinfo.task_struct.num_slabs
      1909 ±  2%     +34.3%       2564        slabinfo.trace_event_file.active_objs
      1909 ±  2%     +34.3%       2564        slabinfo.trace_event_file.num_objs
      1966           -41.0%       1159 ±  3%  slabinfo.tw_sock_TCP.active_slabs
      1966           -41.0%       1159 ±  3%  slabinfo.tw_sock_TCP.num_slabs
     38.42 ±  3%      -6.9       31.49 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     38.56 ±  3%      -6.9       31.66 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     13.48 ±  3%      -2.6       10.86 ±  5%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     13.07 ±  3%      -2.5       10.59 ±  5%  perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     14.11 ±  3%      -2.3       11.79 ±  5%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     12.53 ±  3%      -2.3       10.22 ±  5%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     12.39 ±  3%      -2.3       10.12 ±  5%  perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
     13.88 ±  3%      -2.2       11.64 ±  5%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
     12.77 ±  3%      -2.2       10.53 ±  5%  perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
     14.41 ±  3%      -2.2       12.18 ±  5%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
     14.41 ±  3%      -2.2       12.21 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
     14.29 ±  3%      -2.1       12.15 ±  5%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     12.58 ±  3%      -2.1       10.46 ±  5%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     14.30 ±  3%      -2.1       12.18 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
      8.47 ±  3%      -1.7        6.73        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.43 ±  3%      -1.7        6.72        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.31 ±  3%      -1.6        6.68        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.00 ±  3%      -1.6        7.39 ±  3%  perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
      9.22 ±  3%      -1.6        7.62 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.20 ±  3%      -1.6        7.62 ±  3%  perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.21 ±  3%      -1.6        6.63        perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
      9.07 ±  3%      -1.6        7.51 ±  3%  perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.81            -1.3        3.51 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.77 ±  2%      -1.3        3.50 ±  3%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.98 ±  3%      -1.2        5.73        perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      9.35 ±  3%      -1.2        8.11 ±  4%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      9.27 ±  3%      -1.2        8.06 ±  5%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
      4.46 ±  2%      -1.1        3.37 ±  3%  perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.45            -1.1        3.37 ±  4%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect
      4.49 ±  2%      -1.1        3.42 ±  4%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect
      4.05            -1.1        2.99 ±  4%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect
      4.52 ±  2%      -1.0        3.48 ±  4%  perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      4.40 ±  2%      -1.0        3.36 ±  3%  perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      6.03 ±  3%      -0.9        5.18 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      6.02 ±  3%      -0.8        5.17 ±  2%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      5.18 ±  4%      -0.8        4.39 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_close.inet_release
      7.17 ±  2%      -0.8        6.38 ±  5%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      1.03 ±  2%      -0.8        0.26 ±100%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
      4.58 ±  4%      -0.7        3.85        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
      0.98 ±  2%      -0.7        0.26 ±100%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
      3.51 ±  3%      -0.7        2.80 ±  4%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      3.77 ±  6%      -0.7        3.07 ±  3%  perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
      5.87 ±  4%      -0.7        5.21 ±  8%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_close.inet_release.__sock_release.sock_close
      5.86 ±  4%      -0.7        5.21 ±  8%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_close.inet_release.__sock_release
      4.98 ±  4%      -0.6        4.33 ±  6%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_close
      3.29 ±  3%      -0.6        2.64 ±  3%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_state_process
      3.42 ±  3%      -0.6        2.78 ±  4%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
      3.35 ±  3%      -0.6        2.71 ±  3%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv
      0.93 ±  6%      -0.6        0.28 ±100%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      4.42 ±  5%      -0.6        3.81        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      7.44 ±  4%      -0.6        6.85 ±  6%  perf-profile.calltrace.cycles-pp.tcp_close.inet_release.__sock_release.sock_close.__fput
      9.05 ±  4%      -0.6        8.46 ±  5%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.33 ±  5%      -0.6        3.74 ±  2%  perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      3.25 ±  5%      -0.6        2.70 ±  3%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
      7.74 ±  4%      -0.5        7.19 ±  5%  perf-profile.calltrace.cycles-pp.sock_close.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      7.61 ±  4%      -0.5        7.07 ±  6%  perf-profile.calltrace.cycles-pp.inet_release.__sock_release.sock_close.__fput.task_work_run
      7.72 ±  4%      -0.5        7.19 ±  5%  perf-profile.calltrace.cycles-pp.__sock_release.sock_close.__fput.task_work_run.exit_to_usermode_loop
      0.93 ±  7%      -0.5        0.40 ± 57%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.94 ±  6%      -0.5        0.42 ± 58%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process
      0.95 ±  7%      -0.5        0.43 ± 58%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv
      1.55 ±  5%      -0.5        1.03        perf-profile.calltrace.cycles-pp.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      4.02            -0.5        3.50 ±  4%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      2.68 ±  5%      -0.5        2.20 ±  3%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect
      1.10 ±  3%      -0.5        0.62 ±  5%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
      2.69 ±  5%      -0.5        2.21 ±  3%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect
      1.11 ±  7%      -0.5        0.64 ±  9%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      1.03 ±  5%      -0.5        0.56 ±  3%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.15 ±  3%      -0.4        0.72 ±  7%  perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      3.07 ±  4%      -0.4        2.64 ±  3%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect
      1.03 ±  6%      -0.4        0.60 ±  9%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.99 ±  4%      -0.4        0.63 ±  4%  perf-profile.calltrace.cycles-pp.tcp_fin.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.93 ±  6%      -0.3        0.61 ±  6%  perf-profile.calltrace.cycles-pp.sock_def_wakeup.tcp_fin.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
      0.88 ±  5%      -0.3        0.56 ±  5%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_wakeup.tcp_fin.tcp_data_queue.tcp_rcv_established
      1.40 ±  5%      -0.3        1.12 ±  3%  perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.52 ± 10%      -0.2        1.28 ±  6%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.64 ±  2%      -0.1        0.53 ±  3%  perf-profile.calltrace.cycles-pp.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file.__sys_accept4
      0.58 ±  5%      -0.1        0.53 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file
      2.08 ±  2%      +0.1        2.21 ±  3%  perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.64 ±  4%      +0.3        0.93 ±  6%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      1.15 ±  3%      +0.3        1.44 ±  7%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.96 ±  4%      +0.3        1.26 ± 10%  perf-profile.calltrace.cycles-pp.tcp_fin.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
      0.98 ±  4%      +0.3        1.30 ± 10%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.34 ±101%      +0.5        0.81 ± 18%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.35 ±101%      +0.5        0.85 ± 18%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.00            +0.6        0.57 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_get_next_event.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
      0.00            +0.6        0.59 ± 12%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
      0.13 ±173%      +0.6        0.72 ±  5%  perf-profile.calltrace.cycles-pp.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +0.6        0.61 ±  6%  perf-profile.calltrace.cycles-pp.hrtimer_next_event_without.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.13 ±173%      +0.6        0.75 ±  7%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv
      0.00            +0.6        0.62 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_established
      0.00            +0.6        0.62 ± 12%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.16 ±173%      +0.7        0.82 ± 36%  perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt
      0.89 ± 11%      +0.7        1.58 ±  4%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.13 ±173%      +0.7        0.83 ± 20%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.42 ± 18%      +0.7        2.15 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      0.58 ± 61%      +0.7        1.33 ± 35%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.05 ± 16%      +1.0        2.08 ± 25%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.12 ± 18%      +1.0        2.15 ± 27%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.00            +1.1        1.15 ± 13%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      2.12 ± 11%      +1.2        3.28 ±  2%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      1.41 ± 13%      +1.3        2.71 ± 24%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.39 ±101%      +1.4        1.77 ± 38%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +1.5        1.48 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.do_raw_spin_lock._raw_spin_lock_irqsave.rcu_core.__softirqentry_text_start
      0.00            +1.5        1.49 ± 30%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      3.14 ± 13%      +1.6        4.75 ± 11%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.35 ± 25%      +1.8        3.14 ± 15%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.00            +1.9        1.86 ±  6%  perf-profile.calltrace.cycles-pp.do_raw_spin_lock._raw_spin_lock_irqsave.rcu_core.__softirqentry_text_start.irq_exit
      1.62 ± 20%      +2.2        3.83 ± 13%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      3.29 ± 11%      +2.4        5.72 ±  4%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      3.98 ±  7%      +2.9        6.83 ±  2%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.40 ± 14%      +3.9        6.30 ± 24%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      4.14 ±  4%      +6.5       10.63 ± 11%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     48.02 ±  4%      +7.9       55.92        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     47.19 ±  4%      +8.0       55.23        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     56.36 ±  2%      +8.1       64.47        perf-profile.calltrace.cycles-pp.secondary_startup_64
     54.68 ±  3%      +8.7       63.34        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     54.66 ±  3%      +8.7       63.34        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     54.62 ±  3%      +8.7       63.31        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     10.59 ±  2%     +10.3       20.89 ±  9%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      8.93 ±  2%     +10.6       19.50 ±  9%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     38.73 ±  3%      -6.7       32.02 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     38.86 ±  3%      -6.7       32.19 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     13.52 ±  3%      -2.7       10.87 ±  5%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     13.11 ±  3%      -2.5       10.60 ±  5%  perf-profile.children.cycles-pp.ip_rcv
     17.80 ±  3%      -2.5       15.33 ±  4%  perf-profile.children.cycles-pp.__tcp_transmit_skb
     13.39 ±  2%      -2.4       10.95 ±  4%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      2.72            -2.4        0.35 ±  4%  perf-profile.children.cycles-pp.poll_idle
     14.15 ±  3%      -2.3       11.81 ±  5%  perf-profile.children.cycles-pp.net_rx_action
     12.56 ±  3%      -2.3       10.22 ±  5%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     12.46 ±  3%      -2.3       10.12 ±  5%  perf-profile.children.cycles-pp.tcp_v4_rcv
     12.80 ±  3%      -2.3       10.53 ±  5%  perf-profile.children.cycles-pp.ip_local_deliver
     13.93 ±  3%      -2.3       11.66 ±  5%  perf-profile.children.cycles-pp.process_backlog
     14.46 ±  3%      -2.2       12.24 ±  5%  perf-profile.children.cycles-pp.do_softirq
     12.61 ±  3%      -2.1       10.46 ±  5%  perf-profile.children.cycles-pp.ip_local_deliver_finish
     14.34 ±  3%      -2.1       12.20 ±  5%  perf-profile.children.cycles-pp.do_softirq_own_stack
     16.07 ±  3%      -1.8       14.25 ±  4%  perf-profile.children.cycles-pp.ip_finish_output2
     16.42 ±  3%      -1.8       14.64 ±  4%  perf-profile.children.cycles-pp.ip_output
     16.86 ±  3%      -1.8       15.08 ±  4%  perf-profile.children.cycles-pp.__ip_queue_xmit
      8.47 ±  3%      -1.7        6.73        perf-profile.children.cycles-pp.__x64_sys_sendto
     14.60 ±  3%      -1.7       12.89 ±  4%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      8.43 ±  3%      -1.7        6.72        perf-profile.children.cycles-pp.__sys_sendto
      8.32 ±  3%      -1.6        6.68        perf-profile.children.cycles-pp.sock_sendmsg
      9.00 ±  3%      -1.6        7.39 ±  3%  perf-profile.children.cycles-pp.__inet_stream_connect
      9.22 ±  3%      -1.6        7.62 ±  3%  perf-profile.children.cycles-pp.__x64_sys_connect
      8.22 ±  3%      -1.6        6.63        perf-profile.children.cycles-pp.tcp_sendmsg
      9.20 ±  3%      -1.6        7.62 ±  3%  perf-profile.children.cycles-pp.__sys_connect
      9.08 ±  3%      -1.6        7.51 ±  3%  perf-profile.children.cycles-pp.inet_stream_connect
      6.35 ±  2%      -1.5        4.81 ±  3%  perf-profile.children.cycles-pp.__release_sock
     11.90 ±  3%      -1.5       10.38 ±  5%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
     11.88 ±  3%      -1.5       10.38 ±  5%  perf-profile.children.cycles-pp.tcp_write_xmit
      4.82 ±  2%      -1.3        3.51 ±  3%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
      4.77 ±  2%      -1.3        3.50 ±  3%  perf-profile.children.cycles-pp.__sys_recvfrom
      8.17 ±  3%      -1.3        6.92 ±  5%  perf-profile.children.cycles-pp.tcp_rcv_state_process
      6.99 ±  3%      -1.3        5.73        perf-profile.children.cycles-pp.tcp_sendmsg_locked
      6.10 ±  2%      -1.2        4.88 ±  3%  perf-profile.children.cycles-pp.release_sock
      1.47 ±  5%      -1.2        0.32 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      1.45 ±  7%      -1.1        0.35 ± 10%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      5.09            -1.1        3.99 ±  4%  perf-profile.children.cycles-pp.tcp_rcv_established
      4.46 ±  2%      -1.1        3.37 ±  3%  perf-profile.children.cycles-pp.inet_recvmsg
      4.41 ±  2%      -1.0        3.36 ±  3%  perf-profile.children.cycles-pp.tcp_recvmsg
      1.83 ±  8%      -1.0        0.83 ± 14%  perf-profile.children.cycles-pp.tcp_ack
      0.91 ±  8%      -0.7        0.20 ± 18%  perf-profile.children.cycles-pp.__inet_lookup_established
      3.77 ±  6%      -0.7        3.08 ±  3%  perf-profile.children.cycles-pp.tcp_connect
      2.63 ±  2%      -0.6        2.00 ±  6%  perf-profile.children.cycles-pp.try_to_wake_up
      2.75 ±  2%      -0.6        2.13 ±  6%  perf-profile.children.cycles-pp.__wake_up_common
      1.19 ±  6%      -0.6        0.59 ± 11%  perf-profile.children.cycles-pp.read_tsc
      7.45 ±  4%      -0.6        6.86 ±  6%  perf-profile.children.cycles-pp.tcp_close
      9.06 ±  4%      -0.6        8.47 ±  5%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      4.33 ±  5%      -0.6        3.74 ±  2%  perf-profile.children.cycles-pp.tcp_v4_connect
      3.03 ±  2%      -0.6        2.45 ±  6%  perf-profile.children.cycles-pp.__wake_up_common_lock
      7.75 ±  4%      -0.6        7.19 ±  5%  perf-profile.children.cycles-pp.sock_close
      7.72 ±  4%      -0.5        7.19 ±  5%  perf-profile.children.cycles-pp.__sock_release
      7.61 ±  4%      -0.5        7.07 ±  6%  perf-profile.children.cycles-pp.inet_release
      1.56 ±  6%      -0.5        1.04        perf-profile.children.cycles-pp.tcp_child_process
      0.98 ±  6%      -0.4        0.53 ±  7%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.95 ±  6%      -0.4        0.54 ± 11%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.68 ±  5%      -0.4        0.27 ± 11%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      2.34 ±  3%      -0.4        1.94 ±  6%  perf-profile.children.cycles-pp.sock_def_readable
      0.90 ±  4%      -0.4        0.52 ±  7%  perf-profile.children.cycles-pp.__kfree_skb
      0.98 ±  5%      -0.4        0.60 ±  9%  perf-profile.children.cycles-pp.activate_task
      0.96 ±  8%      -0.4        0.58 ± 17%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.99 ±  5%      -0.4        0.64 ±  9%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.78 ±  5%      -0.3        0.44 ±  8%  perf-profile.children.cycles-pp.enqueue_entity
      1.06 ±  5%      -0.3        0.73 ±  6%  perf-profile.children.cycles-pp.sock_def_wakeup
      0.39 ± 10%      -0.3        0.11 ±  7%  perf-profile.children.cycles-pp.skb_release_all
      1.40 ±  5%      -0.3        1.12 ±  3%  perf-profile.children.cycles-pp.tcp_conn_request
      0.53 ±  9%      -0.3        0.27 ±  3%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.36 ± 10%      -0.3        0.10 ± 10%  perf-profile.children.cycles-pp.skb_release_head_state
      0.35 ± 12%      -0.3        0.09 ± 20%  perf-profile.children.cycles-pp.skb_release_data
      0.48 ± 13%      -0.2        0.24 ± 13%  perf-profile.children.cycles-pp.__skb_datagram_iter
      0.29 ± 10%      -0.2        0.06 ± 63%  perf-profile.children.cycles-pp.dst_release
      1.59 ±  9%      -0.2        1.35 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      0.34 ± 11%      -0.2        0.11 ± 17%  perf-profile.children.cycles-pp.__switch_to
      0.48 ± 12%      -0.2        0.26 ± 14%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      0.27 ±  9%      -0.2        0.04 ± 58%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.55 ±  9%      -0.2        0.32 ± 11%  perf-profile.children.cycles-pp.set_next_entity
      0.69 ±  4%      -0.2        0.47 ±  6%  perf-profile.children.cycles-pp.native_write_msr
      0.29 ± 20%      -0.2        0.08 ±  8%  perf-profile.children.cycles-pp.ipv4_mtu
      0.27 ±  6%      -0.2        0.06 ± 14%  perf-profile.children.cycles-pp.call_rcu
      0.45 ± 11%      -0.2        0.24 ±  7%  perf-profile.children.cycles-pp.sched_clock
      1.13 ±  4%      -0.2        0.93 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.51 ±  7%      -0.2        0.30 ±  9%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.71 ±  3%      -0.2        0.52 ±  3%  perf-profile.children.cycles-pp.sk_reset_timer
      0.50 ±  5%      -0.2        0.31 ±  6%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.29 ±  5%      -0.2        0.10 ± 23%  perf-profile.children.cycles-pp.__slab_free
      0.90 ±  4%      -0.2        0.71 ±  6%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.24 ± 11%      -0.2        0.06 ± 26%  perf-profile.children.cycles-pp.ipv4_dst_check
      0.67 ± 13%      -0.2        0.50 ±  2%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.22 ±  6%      -0.2        0.05 ± 58%  perf-profile.children.cycles-pp.__ksize
      0.40 ± 14%      -0.2        0.23 ±  8%  perf-profile.children.cycles-pp.native_sched_clock
      0.19 ± 15%      -0.2        0.03 ±102%  perf-profile.children.cycles-pp.nf_hook_slow
      0.20 ± 18%      -0.2        0.04 ± 60%  perf-profile.children.cycles-pp.sock_rfree
      0.22 ±  3%      -0.2        0.07 ± 19%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.23 ± 11%      -0.1        0.08 ± 23%  perf-profile.children.cycles-pp.__switch_to_asm
      0.52 ±  5%      -0.1        0.38 ±  2%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.21 ±  9%      -0.1        0.07 ± 16%  perf-profile.children.cycles-pp.__check_object_size
      0.47 ± 15%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp._find_next_bit
      0.17 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp._cond_resched
      0.25 ± 20%      -0.1        0.12 ± 23%  perf-profile.children.cycles-pp.__x64_sys_close
      0.27 ± 18%      -0.1        0.14 ± 38%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.50 ± 13%      -0.1        0.36 ± 13%  perf-profile.children.cycles-pp.tcp_create_openreq_child
      0.17 ±  9%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.48 ±  9%      -0.1        0.35 ± 14%  perf-profile.children.cycles-pp.update_rq_clock
      0.18 ± 16%      -0.1        0.05 ± 58%  perf-profile.children.cycles-pp.__might_sleep
      0.21 ± 14%      -0.1        0.08 ± 59%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.23 ± 10%      -0.1        0.10 ± 19%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.25 ± 11%      -0.1        0.13 ± 22%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.19 ± 11%      -0.1        0.07 ± 17%  perf-profile.children.cycles-pp.__fget_light
      0.21 ± 15%      -0.1        0.09 ±  9%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.20 ± 11%      -0.1        0.08 ± 23%  perf-profile.children.cycles-pp.select_idle_sibling
      0.20 ± 24%      -0.1        0.08 ± 74%  perf-profile.children.cycles-pp.wake_up_klogd_work_func
      0.15 ± 10%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.update_cfs_group
      0.21 ± 23%      -0.1        0.10 ± 17%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.36 ±  6%      -0.1        0.25 ± 16%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.23 ± 19%      -0.1        0.12 ± 10%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.44 ±  7%      -0.1        0.34 ±  4%  perf-profile.children.cycles-pp.tcp_send_fin
      1.51 ±  5%      -0.1        1.41 ±  5%  perf-profile.children.cycles-pp.wait_woken
      0.14 ± 19%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.16 ± 29%      -0.1        0.06 ± 66%  perf-profile.children.cycles-pp.rb_next
      0.36 ± 12%      -0.1        0.26 ±  3%  perf-profile.children.cycles-pp.inet_csk_destroy_sock
      0.27 ± 13%      -0.1        0.17 ± 16%  perf-profile.children.cycles-pp.__sk_dst_check
      0.76 ±  7%      -0.1        0.67 ±  6%  perf-profile.children.cycles-pp.dequeue_entity
      0.15 ± 12%      -0.1        0.06 ± 58%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      0.12 ±  9%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.inet_reqsk_alloc
      0.27 ±  6%      -0.1        0.18 ± 15%  perf-profile.children.cycles-pp.memcpy_erms
      0.37 ± 11%      -0.1        0.28 ± 15%  perf-profile.children.cycles-pp.inet_csk_clone_lock
      0.14 ± 18%      -0.1        0.05 ± 64%  perf-profile.children.cycles-pp.inet_sk_rebuild_header
      0.42 ±  8%      -0.1        0.34 ± 16%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.18 ± 10%      -0.1        0.10 ± 17%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.11 ± 24%      -0.1        0.04 ±107%  perf-profile.children.cycles-pp.native_apic_mem_write
      0.15 ± 25%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.tcp_current_mss
      0.19 ± 10%      -0.1        0.12 ± 17%  perf-profile.children.cycles-pp._copy_to_iter
      0.11 ± 14%      -0.1        0.04 ± 60%  perf-profile.children.cycles-pp.reweight_entity
      0.16 ± 23%      -0.1        0.09 ± 37%  perf-profile.children.cycles-pp.menu_reflect
      0.12 ±  8%      -0.1        0.06 ± 62%  perf-profile.children.cycles-pp.sock_recvmsg
      0.12 ± 29%      -0.1        0.05 ± 67%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.12 ± 19%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.tcp_release_cb
      0.10 ± 23%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.run_local_timers
      0.25 ± 12%      -0.1        0.19 ± 14%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.09 ± 25%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.rb_insert_color
      0.26 ±  5%      -0.1        0.20 ± 10%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.12 ±  3%      -0.1        0.06 ± 62%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.12 ± 28%      -0.1        0.07 ± 31%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.08 ± 10%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.tcp_add_backlog
      0.12 ± 13%      -0.1        0.07 ± 17%  perf-profile.children.cycles-pp.security_sk_alloc
      0.11 ± 13%      -0.1        0.06 ± 20%  perf-profile.children.cycles-pp.apparmor_sk_alloc_security
      0.19 ± 13%      -0.0        0.15 ± 10%  perf-profile.children.cycles-pp.tcp_make_synack
      0.07 ± 11%      -0.0        0.03 ±102%  perf-profile.children.cycles-pp.security_socket_getsockopt
      0.20 ±  6%      -0.0        0.16 ± 10%  perf-profile.children.cycles-pp.sk_prot_alloc
      0.10 ± 27%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.tcp_get_metrics
      0.11 ± 17%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.11 ± 14%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.pick_next_entity
      0.11 ±  9%      -0.0        0.08 ± 23%  perf-profile.children.cycles-pp.inode_init_always
      0.15 ± 12%      -0.0        0.12 ± 15%  perf-profile.children.cycles-pp.remove_wait_queue
      0.07 ± 16%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.read
      0.06 ± 14%      +0.0        0.09 ± 17%  perf-profile.children.cycles-pp.forkshell
      0.07 ± 13%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.security_sk_free
      0.06            +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.inet_csk_reqsk_queue_drop
      0.14 ±  7%      +0.0        0.18 ±  9%  perf-profile.children.cycles-pp.evict
      0.11 ±  6%      +0.0        0.15 ± 12%  perf-profile.children.cycles-pp.sock_alloc_inode
      0.06 ± 66%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.add_wait_queue
      0.03 ±100%      +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.08 ± 17%      +0.1        0.13 ± 16%  perf-profile.children.cycles-pp.try_to_del_timer_sync
      0.04 ± 58%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.down_write
      0.03 ±102%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.03 ±102%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.do_group_exit
      0.03 ±102%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.do_exit
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.alloc_set_pte
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.bpf_sk_storage_free
      0.09 ± 38%      +0.1        0.15 ± 19%  perf-profile.children.cycles-pp.execve
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.00            +0.1        0.06 ± 20%  perf-profile.children.cycles-pp.sock_owned_by_me
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.clear_inode
      0.01 ±173%      +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.__sk_free
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__fd_install
      0.18 ± 15%      +0.1        0.25 ± 12%  perf-profile.children.cycles-pp.del_timer
      0.03 ±102%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.__libc_start_main
      0.03 ±102%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.main
      0.00            +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.update_dl_rq_load_avg
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.rcu_read_lock_bh_held
      0.00            +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.lockdep_init_map
      0.00            +0.1        0.07 ± 25%  perf-profile.children.cycles-pp.netlink_has_listeners
      0.06 ± 22%      +0.1        0.12 ±  8%  perf-profile.children.cycles-pp.ksys_read
      0.06 ± 22%      +0.1        0.12 ±  8%  perf-profile.children.cycles-pp.vfs_read
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.register_lock_class
      0.00            +0.1        0.07 ± 21%  perf-profile.children.cycles-pp.inet_twsk_hashdance
      0.32 ±  9%      +0.1        0.39 ±  5%  perf-profile.children.cycles-pp.alloc_empty_file
      0.01 ±173%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.cmd_stat
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.inet_csk_init_xmit_timers
      0.32 ±  6%      +0.1        0.39 ±  9%  perf-profile.children.cycles-pp.inet_create
      0.31 ± 11%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.__alloc_file
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.do_filp_open
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.path_openat
      0.01 ±173%      +0.1        0.09 ±  7%  perf-profile.children.cycles-pp.__note_gp_changes
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.trace_hardirqs_off_caller
      0.33 ±  6%      +0.1        0.41 ±  9%  perf-profile.children.cycles-pp.tcp_done
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.inet_sk_state_store
      0.14 ±  7%      +0.1        0.22 ± 13%  perf-profile.children.cycles-pp.sk_alloc
      0.06 ± 13%      +0.1        0.14 ± 32%  perf-profile.children.cycles-pp.rcu_gp_kthread
      0.03 ±100%      +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.tcp_update_metrics
      0.04 ± 59%      +0.1        0.12 ± 27%  perf-profile.children.cycles-pp._copy_from_user
      0.03 ±100%      +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.inet_put_port
      0.16 ± 17%      +0.1        0.24 ±  4%  perf-profile.children.cycles-pp.__x64_sys_setsockopt
      0.00            +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.detach_if_pending
      0.00            +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.cgroup_sk_alloc
      0.01 ±173%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.enqueue_timer
      0.21 ±  8%      +0.1        0.30 ±  7%  perf-profile.children.cycles-pp.__inet_bind
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.do_sys_open
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.do_sys_openat2
      0.15 ± 14%      +0.1        0.24 ±  4%  perf-profile.children.cycles-pp.__sys_setsockopt
      0.03 ±100%      +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.filemap_map_pages
      0.01 ±173%      +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.__update_idle_core
      0.00            +0.1        0.09 ± 27%  perf-profile.children.cycles-pp.ret_from_intr
      0.17 ± 10%      +0.1        0.26 ±  9%  perf-profile.children.cycles-pp.newidle_balance
      0.00            +0.1        0.10 ± 11%  perf-profile.children.cycles-pp.init_timer_key
      0.03 ±100%      +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.00            +0.1        0.10 ± 21%  perf-profile.children.cycles-pp.do_raw_spin_trylock
      0.00            +0.1        0.10 ± 18%  perf-profile.children.cycles-pp._raw_write_lock_bh
      0.01 ±173%      +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.set_next_task_idle
      0.00            +0.1        0.11 ± 20%  perf-profile.children.cycles-pp.restore_regs_and_return_to_kernel
      0.15 ± 10%      +0.1        0.26 ± 16%  perf-profile.children.cycles-pp.inet_csk_route_req
      0.09 ±  8%      +0.1        0.21 ±  5%  perf-profile.children.cycles-pp.sock_setsockopt
      0.24 ± 10%      +0.1        0.36 ±  6%  perf-profile.children.cycles-pp.__kmalloc_reserve
      0.22 ±  8%      +0.1        0.33 ±  9%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.00            +0.1        0.12 ± 32%  perf-profile.children.cycles-pp.__local_bh_enable
      2.08 ±  2%      +0.1        2.21 ±  3%  perf-profile.children.cycles-pp.sk_wait_data
      0.39 ±  7%      +0.1        0.51 ±  9%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.09 ± 20%      +0.1        0.21 ± 10%  perf-profile.children.cycles-pp.tcp_set_state
      0.09 ± 26%      +0.1        0.22 ± 11%  perf-profile.children.cycles-pp.del_timer_sync
      0.03 ±100%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.cpuacct_charge
      0.07 ± 62%      +0.1        0.20 ±  6%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.06 ± 61%      +0.1        0.20 ±  2%  perf-profile.children.cycles-pp.handle_pte_fault
      0.00            +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.00            +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.trace_softirqs_on
      0.00            +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.lock_unpin_lock
      0.34 ± 19%      +0.1        0.48 ± 10%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.09 ±  8%      +0.1        0.24 ± 11%  perf-profile.children.cycles-pp.dput
      0.20 ± 13%      +0.1        0.35 ±  7%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.00            +0.1        0.15 ± 29%  perf-profile.children.cycles-pp.trace_softirqs_off
      0.00            +0.2        0.15 ± 12%  perf-profile.children.cycles-pp.lockdep_rtnl_is_held
      0.00            +0.2        0.15 ± 14%  perf-profile.children.cycles-pp.trace_hardirqs_on_thunk
      0.23 ± 12%      +0.2        0.39 ± 11%  perf-profile.children.cycles-pp.__sk_destruct
      0.07 ± 64%      +0.2        0.22 ±  4%  perf-profile.children.cycles-pp.handle_mm_fault
      0.00            +0.2        0.16 ± 11%  perf-profile.children.cycles-pp.__raw_spin_lock_init
      0.00            +0.2        0.16 ± 15%  perf-profile.children.cycles-pp._raw_spin_unlock_bh
      0.00            +0.2        0.16 ± 13%  perf-profile.children.cycles-pp.lock_pin_lock
      0.06 ± 11%      +0.2        0.22 ± 14%  perf-profile.children.cycles-pp.trigger_load_balance
      0.10 ± 28%      +0.2        0.26 ±  4%  perf-profile.children.cycles-pp.do_page_fault
      0.22 ± 21%      +0.2        0.39 ± 15%  perf-profile.children.cycles-pp.fib_table_lookup
      0.11 ± 31%      +0.2        0.28 ±  6%  perf-profile.children.cycles-pp.page_fault
      0.16 ±  7%      +0.2        0.33 ±  9%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.16 ± 49%      +0.2        0.34 ± 29%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.2        0.18 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_nested
      0.19 ± 26%      +0.2        0.38 ± 25%  perf-profile.children.cycles-pp.finish_task_switch
      0.09 ± 21%      +0.2        0.29 ±  9%  perf-profile.children.cycles-pp.___perf_sw_event
      0.19 ± 14%      +0.2        0.40 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.00            +0.2        0.21 ± 31%  perf-profile.children.cycles-pp.trace_hardirqs_off_thunk
      0.00            +0.2        0.21 ± 11%  perf-profile.children.cycles-pp.tracer_hardirqs_on
      0.11 ± 15%      +0.2        0.33 ± 12%  perf-profile.children.cycles-pp.raise_softirq
      0.18 ±  4%      +0.2        0.40 ±  8%  perf-profile.children.cycles-pp.kfree
      0.31            +0.2        0.53 ± 14%  perf-profile.children.cycles-pp.ktime_get_with_offset
      1.40 ±  5%      +0.2        1.63 ±  2%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.39 ±  9%      +0.3        0.65 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.22 ± 37%      +0.3        0.49 ±  9%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.00            +0.3        0.27 ± 12%  perf-profile.children.cycles-pp.tracer_hardirqs_off
      0.18 ± 26%      +0.3        0.46 ±  8%  perf-profile.children.cycles-pp.update_blocked_averages
      0.27 ±  8%      +0.3        0.54 ±  3%  perf-profile.children.cycles-pp.update_curr
      0.00            +0.3        0.28 ±  6%  perf-profile.children.cycles-pp.debug_check_no_locks_freed
      0.00            +0.3        0.29 ± 14%  perf-profile.children.cycles-pp.do_raw_spin_unlock
      0.00            +0.3        0.32 ± 10%  perf-profile.children.cycles-pp.reacquire_held_locks
      0.39 ± 10%      +0.4        0.74 ± 13%  perf-profile.children.cycles-pp.kmem_cache_free
      0.31 ±  7%      +0.4        0.67 ±  5%  perf-profile.children.cycles-pp.ip_route_output_flow
      0.39 ± 12%      +0.4        0.75 ±  7%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      0.48 ± 51%      +0.4        0.85 ± 18%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.4        0.38 ± 13%  perf-profile.children.cycles-pp.__local_bh_disable_ip
      0.62 ± 11%      +0.4        1.01 ±  4%  perf-profile.children.cycles-pp.netif_rx_internal
      0.00            +0.4        0.40 ± 10%  perf-profile.children.cycles-pp.trace_hardirqs_on
      2.03 ±  6%      +0.4        2.44 ±  4%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.33 ± 28%      +0.4        0.75 ± 40%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.4        0.42 ±  7%  perf-profile.children.cycles-pp.fs_reclaim_acquire
      0.20 ±  6%      +0.4        0.63 ±  6%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.4        0.43 ± 21%  perf-profile.children.cycles-pp._raw_spin_unlock_irq
      0.43 ± 11%      +0.4        0.87 ±  6%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      0.47 ± 15%      +0.4        0.91 ± 20%  perf-profile.children.cycles-pp.scheduler_tick
      2.42            +0.4        2.87 ±  7%  perf-profile.children.cycles-pp.tcp_data_queue
      0.13 ± 20%      +0.4        0.58 ± 10%  perf-profile.children.cycles-pp.__might_fault
      0.13 ± 18%      +0.5        0.58 ±  8%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.28 ± 65%      +0.5        0.74 ±  5%  perf-profile.children.cycles-pp.run_timer_softirq
      0.00            +0.6        0.58 ±  5%  perf-profile.children.cycles-pp.rcu_is_watching
      0.64 ± 10%      +0.6        1.23 ±  4%  perf-profile.children.cycles-pp.netif_rx
      0.70 ± 34%      +0.7        1.36 ± 34%  perf-profile.children.cycles-pp.irq_enter
      0.92 ± 11%      +0.7        1.60 ±  4%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.49 ± 10%      +0.8        1.25 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.00            +0.8        0.79 ±  5%  perf-profile.children.cycles-pp.trace_hardirqs_off
      0.00            +0.8        0.82 ±  5%  perf-profile.children.cycles-pp.find_held_lock
      0.22 ±  8%      +0.9        1.08 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      1.29 ± 16%      +0.9        2.23 ± 25%  perf-profile.children.cycles-pp.update_process_times
      1.35 ± 16%      +0.9        2.29 ± 26%  perf-profile.children.cycles-pp.tick_sched_handle
      0.26 ± 31%      +0.9        1.20 ± 13%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +1.0        1.03 ±  2%  perf-profile.children.cycles-pp.mark_held_locks
      0.00            +1.0        1.04 ±  6%  perf-profile.children.cycles-pp._raw_spin_unlock
      0.70 ± 14%      +1.1        1.76 ±  4%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.49 ± 11%      +1.1        1.58 ± 30%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +1.1        1.15 ± 11%  perf-profile.children.cycles-pp.lockdep_hardirqs_off
      2.52 ± 10%      +1.2        3.70 ±  2%  perf-profile.children.cycles-pp.rcu_core
      0.00            +1.2        1.19 ±  3%  perf-profile.children.cycles-pp.match_held_lock
      1.65 ± 14%      +1.3        2.90 ± 23%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +1.4        1.37 ± 11%  perf-profile.children.cycles-pp.rcu_lockdep_current_cpu_online
      3.23 ± 13%      +1.6        4.83 ± 11%  perf-profile.children.cycles-pp.menu_select
      0.00            +1.7        1.68 ±  3%  perf-profile.children.cycles-pp.rcu_read_lock_held
      2.81 ±  9%      +1.7        4.56 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +1.8        1.76 ±  4%  perf-profile.children.cycles-pp.lockdep_hardirqs_on
      0.00            +1.8        1.79 ± 12%  perf-profile.children.cycles-pp.debug_lockdep_rcu_enabled
      1.40 ± 24%      +1.8        3.21 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +2.2        2.19        perf-profile.children.cycles-pp.mark_lock
      1.68 ± 20%      +2.2        3.88 ± 13%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +2.5        2.46 ±  9%  perf-profile.children.cycles-pp.__lock_text_start
      0.00            +2.7        2.65 ±  4%  perf-profile.children.cycles-pp.rcu_read_lock_held_common
      4.25 ±  6%      +2.9        7.11 ±  2%  perf-profile.children.cycles-pp.irq_exit
      0.00            +3.0        2.97 ±  4%  perf-profile.children.cycles-pp.do_raw_spin_lock
      2.18 ± 33%      +3.4        5.57 ±  4%  perf-profile.children.cycles-pp.ktime_get
      2.75 ± 16%      +3.9        6.65 ± 25%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.00            +4.7        4.71 ±  3%  perf-profile.children.cycles-pp.lock_release
      0.00            +4.7        4.74        perf-profile.children.cycles-pp.rcu_read_lock_sched_held
      0.00            +6.2        6.18 ±  2%  perf-profile.children.cycles-pp.lock_is_held_type
      4.58 ±  8%      +6.6       11.14 ± 11%  perf-profile.children.cycles-pp.hrtimer_interrupt
     49.50 ±  3%      +7.4       56.87        perf-profile.children.cycles-pp.cpuidle_enter
     49.48 ±  3%      +7.4       56.85        perf-profile.children.cycles-pp.cpuidle_enter_state
     56.39 ±  2%      +8.1       64.48        perf-profile.children.cycles-pp.do_idle
     56.36 ±  2%      +8.1       64.47        perf-profile.children.cycles-pp.secondary_startup_64
     56.36 ±  2%      +8.1       64.47        perf-profile.children.cycles-pp.cpu_startup_entry
     54.68 ±  3%      +8.7       63.34        perf-profile.children.cycles-pp.start_secondary
     10.58 ±  4%     +10.5       21.06 ±  9%  perf-profile.children.cycles-pp.apic_timer_interrupt
      9.66 ±  4%     +10.6       20.30 ±  9%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.00           +10.8       10.77 ±  3%  perf-profile.children.cycles-pp.__lock_acquire
      0.00           +14.2       14.15 ±  2%  perf-profile.children.cycles-pp.lock_acquire
      2.65 ±  2%      -2.3        0.32 ±  8%  perf-profile.self.cycles-pp.poll_idle
      2.35 ±  8%      -1.8        0.58 ±  6%  perf-profile.self.cycles-pp.do_syscall_64
      1.62 ±  6%      -1.5        0.12 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.47 ±  6%      -1.1        0.32 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.45 ±  7%      -1.1        0.35 ± 10%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.81 ±  9%      -0.6        0.17 ± 16%  perf-profile.self.cycles-pp.__inet_lookup_established
      1.14 ±  5%      -0.6        0.56 ± 13%  perf-profile.self.cycles-pp.read_tsc
      1.26 ± 15%      -0.6        0.70 ± 21%  perf-profile.self.cycles-pp.menu_select
      0.81 ±  8%      -0.5        0.32 ±  4%  perf-profile.self.cycles-pp.__sched_text_start
      0.56 ± 10%      -0.4        0.17 ± 15%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.63 ±  4%      -0.4        0.25 ± 14%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.50 ± 10%      -0.4        0.14 ± 19%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.46 ± 10%      -0.4        0.11 ±  7%  perf-profile.self.cycles-pp.tcp_ack
      0.42 ± 12%      -0.3        0.11 ± 12%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.47 ±  6%      -0.3        0.17 ±  9%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.50 ± 14%      -0.3        0.22 ±  7%  perf-profile.self.cycles-pp.do_idle
      0.57 ±  7%      -0.3        0.29 ±  2%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.48 ± 12%      -0.3        0.21 ±  4%  perf-profile.self.cycles-pp.___might_sleep
      0.35 ± 12%      -0.3        0.09 ± 18%  perf-profile.self.cycles-pp.skb_release_data
      0.36 ±  5%      -0.3        0.10 ± 11%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.38 ± 11%      -0.3        0.12 ±  9%  perf-profile.self.cycles-pp.set_next_entity
      0.29 ± 17%      -0.2        0.04 ± 60%  perf-profile.self.cycles-pp.sock_def_readable
      0.31 ± 12%      -0.2        0.08 ± 10%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.26 ± 11%      -0.2        0.03 ±100%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.29 ± 12%      -0.2        0.06 ± 64%  perf-profile.self.cycles-pp.dst_release
      0.35 ± 10%      -0.2        0.12 ±  5%  perf-profile.self.cycles-pp.ip_finish_output2
      0.69 ±  4%      -0.2        0.47 ±  6%  perf-profile.self.cycles-pp.native_write_msr
      0.32 ±  9%      -0.2        0.10 ± 19%  perf-profile.self.cycles-pp.__switch_to
      0.35 ± 14%      -0.2        0.14 ± 21%  perf-profile.self.cycles-pp.update_rq_clock
      0.29 ±  8%      -0.2        0.09 ± 12%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.27 ± 20%      -0.2        0.07 ± 10%  perf-profile.self.cycles-pp.ipv4_mtu
      1.03 ±  7%      -0.2        0.83 ±  5%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.24 ±  8%      -0.2        0.05 ± 60%  perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.28 ±  5%      -0.2        0.10 ± 23%  perf-profile.self.cycles-pp.__slab_free
      0.27 ±  7%      -0.2        0.09 ± 21%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.22 ±  6%      -0.2        0.05 ± 58%  perf-profile.self.cycles-pp.__ksize
      0.20 ± 20%      -0.2        0.03 ±105%  perf-profile.self.cycles-pp.net_rx_action
      0.39 ± 15%      -0.2        0.22 ±  9%  perf-profile.self.cycles-pp.native_sched_clock
      0.21 ±  8%      -0.2        0.05 ± 58%  perf-profile.self.cycles-pp.aa_sk_perm
      0.24 ±  9%      -0.2        0.07 ± 15%  perf-profile.self.cycles-pp.loopback_xmit
      0.22 ± 12%      -0.2        0.06 ± 16%  perf-profile.self.cycles-pp.enqueue_entity
      0.25 ± 10%      -0.2        0.09 ± 17%  perf-profile.self.cycles-pp.process_backlog
      0.22 ± 21%      -0.2        0.06 ± 20%  perf-profile.self.cycles-pp.fib_table_lookup
      0.19 ± 16%      -0.2        0.04 ±107%  perf-profile.self.cycles-pp.ipv4_dst_check
      0.23 ± 14%      -0.2        0.08 ± 21%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.19 ± 23%      -0.1        0.04 ± 60%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.23 ± 11%      -0.1        0.08 ± 23%  perf-profile.self.cycles-pp.__switch_to_asm
      0.19 ± 16%      -0.1        0.04 ± 60%  perf-profile.self.cycles-pp.sock_rfree
      0.35 ± 14%      -0.1        0.20 ± 17%  perf-profile.self.cycles-pp.note_gp_changes
      0.46 ± 14%      -0.1        0.32 ±  6%  perf-profile.self.cycles-pp._find_next_bit
      0.21 ± 18%      -0.1        0.08 ± 27%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.21 ± 23%      -0.1        0.07 ± 17%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.21 ± 14%      -0.1        0.08 ± 59%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.19 ± 11%      -0.1        0.07 ± 13%  perf-profile.self.cycles-pp.__fget_light
      0.16 ± 16%      -0.1        0.04 ± 59%  perf-profile.self.cycles-pp.__might_sleep
      0.24 ± 12%      -0.1        0.13 ± 22%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.16 ±  2%      -0.1        0.05 ± 60%  perf-profile.self.cycles-pp.tcp_rcv_state_process
      0.14 ± 24%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.15 ± 10%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.update_cfs_group
      0.17 ±  8%      -0.1        0.07 ± 16%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.14 ± 13%      -0.1        0.04 ±100%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.19 ± 18%      -0.1        0.09 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.17 ± 11%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.mod_timer
      0.17 ± 17%      -0.1        0.08 ± 20%  perf-profile.self.cycles-pp.ip_local_deliver
      0.14 ± 15%      -0.1        0.05 ± 58%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      0.27 ±  6%      -0.1        0.18 ± 13%  perf-profile.self.cycles-pp.memcpy_erms
      0.18 ±  8%      -0.1        0.09 ± 17%  perf-profile.self.cycles-pp.ip_output
      0.14 ± 27%      -0.1        0.06 ± 66%  perf-profile.self.cycles-pp.rb_next
      0.18 ± 10%      -0.1        0.10 ± 15%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.42 ±  8%      -0.1        0.34 ± 16%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.30 ± 14%      -0.1        0.22 ± 11%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.14 ±  8%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.11 ± 14%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.reweight_entity
      0.18 ±  6%      -0.1        0.11 ± 21%  perf-profile.self.cycles-pp.kfree
      0.12 ± 20%      -0.1        0.05 ± 67%  perf-profile.self.cycles-pp.menu_reflect
      0.12 ± 12%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.dequeue_entity
      0.12 ± 13%      -0.1        0.07 ± 25%  perf-profile.self.cycles-pp.newidle_balance
      0.08 ± 23%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.run_local_timers
      0.08 ± 19%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.rb_insert_color
      0.24 ± 12%      -0.0        0.19 ±  3%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.12 ± 18%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.resched_curr
      0.11 ± 14%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.12 ± 24%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.update_curr
      0.10 ± 14%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      0.09 ± 15%      -0.0        0.04 ± 59%  perf-profile.self.cycles-pp.rcu_eqs_exit
      0.07 ± 22%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.11 ±  4%      -0.0        0.08 ± 22%  perf-profile.self.cycles-pp.finish_task_switch
      0.10 ± 15%      -0.0        0.07 ± 21%  perf-profile.self.cycles-pp.inet_csk_accept
      0.08 ±  8%      -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.release_sock
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.05 ± 62%      +0.0        0.08 ± 15%  perf-profile.self.cycles-pp.__wake_up_common
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.register_lock_class
      0.15 ± 27%      +0.1        0.22 ± 15%  perf-profile.self.cycles-pp.rcu_core
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.reacquire_held_locks
      0.03 ±100%      +0.1        0.12 ± 18%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.00            +0.1        0.10 ±  9%  perf-profile.self.cycles-pp._raw_spin_unlock
      0.00            +0.1        0.10 ± 25%  perf-profile.self.cycles-pp.__lock_text_start
      0.00            +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.do_raw_spin_trylock
      0.00            +0.1        0.11 ± 10%  perf-profile.self.cycles-pp.lock_pin_lock
      0.00            +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.lock_unpin_lock
      0.00            +0.1        0.14 ±  8%  perf-profile.self.cycles-pp.trace_softirqs_on
      0.00            +0.1        0.14 ± 31%  perf-profile.self.cycles-pp.trace_softirqs_off
      0.00            +0.2        0.18 ± 11%  perf-profile.self.cycles-pp.rcu_read_lock_held
      0.00            +0.2        0.18 ± 17%  perf-profile.self.cycles-pp.tracer_hardirqs_on
      0.00            +0.2        0.21 ±  7%  perf-profile.self.cycles-pp.trace_hardirqs_on
      0.00            +0.2        0.24 ± 10%  perf-profile.self.cycles-pp.tracer_hardirqs_off
      0.00            +0.3        0.28 ± 13%  perf-profile.self.cycles-pp.do_raw_spin_unlock
      0.00            +0.3        0.28 ±  6%  perf-profile.self.cycles-pp.debug_check_no_locks_freed
      0.00            +0.3        0.28 ±  3%  perf-profile.self.cycles-pp.__local_bh_disable_ip
      0.00            +0.4        0.39 ± 14%  perf-profile.self.cycles-pp.rcu_read_lock_sched_held
      0.00            +0.4        0.43 ±  7%  perf-profile.self.cycles-pp.trace_hardirqs_off
      0.00            +0.5        0.48 ±  6%  perf-profile.self.cycles-pp.rcu_is_watching
      0.00            +0.5        0.51 ±  9%  perf-profile.self.cycles-pp.find_held_lock
      0.00            +0.5        0.53 ±  3%  perf-profile.self.cycles-pp.mark_held_locks
      0.00            +0.7        0.68 ±  8%  perf-profile.self.cycles-pp.rcu_read_lock_held_common
      0.00            +0.8        0.75 ± 11%  perf-profile.self.cycles-pp.lockdep_hardirqs_on
      0.00            +0.9        0.88 ±  6%  perf-profile.self.cycles-pp.match_held_lock
      0.00            +1.1        1.13 ± 11%  perf-profile.self.cycles-pp.lockdep_hardirqs_off
      0.00            +1.3        1.30 ±  5%  perf-profile.self.cycles-pp.do_raw_spin_lock
      0.00            +1.3        1.31 ± 11%  perf-profile.self.cycles-pp.rcu_lockdep_current_cpu_online
      0.00            +1.5        1.55 ± 12%  perf-profile.self.cycles-pp.debug_lockdep_rcu_enabled
      0.00            +2.1        2.07        perf-profile.self.cycles-pp.mark_lock
      0.00            +3.3        3.29 ±  2%  perf-profile.self.cycles-pp.lock_release
      0.00            +3.3        3.30 ±  2%  perf-profile.self.cycles-pp.lock_acquire
      0.00            +5.4        5.41        perf-profile.self.cycles-pp.lock_is_held_type
      0.00            +9.1        9.07 ±  4%  perf-profile.self.cycles-pp.__lock_acquire


                                                                                
                               netperf.Throughput_tps                           
                                                                                
  35000 +-------------------------------------------------------------------+   
        |  +.+..+.+..+..+.+     +.+..+.+..+..+    +.+..+..+.+..+..+.+..+.+..|   
  30000 |-+:              :     :            :    :                         |   
        |  :              :     :            :    :                         |   
  25000 |-:                :   :             :   :                          |   
        | :                :   :              :  :                          |   
  20000 |-:                :   :              :  :                          |   
        | :O O    O  O  O O: O :O O           :  :                          |   
  15000 |:+                 : :               : :                           |   
        |:                  : :               : :                           |   
  10000 |:+                 : :           O  O:O: O O  O  O O               |   
        |:                  : :      O O       ::                           |   
   5000 |-+                  :                 :                            |   
        |                    :                 :                            |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            netperf.Throughput_total_tps                        
                                                                                
  35000 +-------------------------------------------------------------------+   
        |  +.+..+.+..+..+.+     +.+..+.+..+..+    +.+..+..+.+..+..+.+..+.+..|   
  30000 |-+:              :     :            :    :                         |   
        |  :              :     :            :    :                         |   
  25000 |-:                :   :             :   :                          |   
        | :                :   :              :  :                          |   
  20000 |-:                :   :              :  :                          |   
        | :O O    O  O  O O: O :O O           :  :                          |   
  15000 |:+                 : :               : :                           |   
        |:                  : :               : :                           |   
  10000 |:+                 : :           O  O:O: O O  O  O O               |   
        |:                  : :      O O       ::                           |   
   5000 |-+                  :                 :                            |   
        |                    :                 :                            |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                  netperf.workload                              
                                                                                
  1e+07 +-------------------------------------------------------------------+   
  9e+06 |-+:              :     :    +.+..+..+    +       +.+..+..+.+..+.+..|   
        |  :              :     :            :    :                         |   
  8e+06 |-:                :   :             :    :                         |   
  7e+06 |-:                :   :             :   :                          |   
        | :                :   :              :  :                          |   
  6e+06 |-:                :   :              :  :                          |   
  5e+06 |-:O O    O  O  O O :O :O O           :  :                          |   
  4e+06 |:+                 : :               : :                           |   
        |:                  : :               : :                           |   
  3e+06 |:+                 : :      O O  O  O:O: O O  O  O O               |   
  2e+06 |:+                 : :                ::                           |   
        |                    :                 :                            |   
  1e+06 |-+                  :                 :                            |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             netperf.time.user_time                             
                                                                                
  40 +----------------------------------------------------------------------+   
     |                                                                      |   
  35 |-+                                   +            .+.. .+..           |   
  30 |-++.+..+..+..+.+..+         .+..+.. +:     +.+..+.    +    +..+..+.+..|   
     |  :               :     +.+.       +  :    :                          |   
  25 |-:                 :    :             :   :                           |   
     | :                 :   :              :   :                           |   
  20 |-:  O     O    O   :   :O O           :   :                           |   
     | :O          O    O: O :               :  :                           |   
  15 |:+                  :  :               : :                            |   
  10 |:+                  : :      O  O  O O :O:   O  O  O                  |   
     |:                   : :                : : O          O               |   
   5 |:+                  : :                 ::                            |   
     |                     :                  :                             |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        netperf.time.voluntary_context_switches                 
                                                                                
    2e+07 +-----------------------------------------------------------------+   
  1.8e+07 |-+                        .+.+..+..+    +.+..+.+..+.+..+.+..+.+..|   
          |  +.+..+.+.. .+..+    +.+.         :    :                        |   
  1.6e+07 |-+:         +    :    :            :    :                        |   
  1.4e+07 |-+:              :    :            :   :                         |   
          | :                :  :              :  :                         |   
  1.2e+07 |-:                :  :              :  :                         |   
    1e+07 |-:O O    O  O O  O:O :O O           :  :                         |   
    8e+06 |-:                :  :              : :                          |   
          |:                 : :               : :                          |   
    6e+06 |:+                : :      O O  O  O:O: O O  O O  O              |   
    4e+06 |:+                 ::                ::                          |   
          |:                  ::                :                           |   
    2e+06 |-+                 :                 :                           |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc5-00002-g05c5a0273bc25" of type "text/plain" (205642 bytes)

View attachment "job-script" of type "text/plain" (7963 bytes)

View attachment "job.yaml" of type "text/plain" (5539 bytes)

View attachment "reproduce" of type "text/plain" (326 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ