lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20210513155515.GB23902@xsang-OptiPlex-9020>
Date:   Thu, 13 May 2021 23:55:15 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     "Paul E. McKenney" <paulmck@...nel.org>
Cc:     0day robot <lkp@...el.com>, Thomas Gleixner <tglx@...utronix.de>,
        John Stultz <john.stultz@...aro.org>,
        Stephen Boyd <sboyd@...nel.org>,
        Jonathan Corbet <corbet@....net>,
        Mark Rutland <Mark.Rutland@....com>,
        Marc Zyngier <maz@...nel.org>, Andi Kleen <ak@...ux.intel.com>,
        Xing Zhengjun <zhengjun.xing@...ux.intel.com>,
        Feng Tang <feng.tang@...el.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        ying.huang@...el.com, zhengjun.xing@...el.com, kernel-team@...com,
        neeraju@...eaurora.org, "Paul E. McKenney" <paulmck@...nel.org>
Subject: [clocksource]  388450c708:  netperf.Throughput_tps -65.1% regression



Greeting,

FYI, we noticed a -65.1% regression of netperf.Throughput_tps due to commit:


commit: 388450c7081ded73432e2b7148c1bb9a0b039963 ("[PATCH v12 clocksource 4/5] clocksource: Reduce clocksource-skew threshold for TSC")
url: https://github.com/0day-ci/linux/commits/Paul-E-McKenney/Do-not-mark-clocks-unstable-due-to-delays-for-v5-13/20210501-083404
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 2d036dfa5f10df9782f5278fc591d79d283c1fad

in testcase: netperf
on test machine: 96 threads 2 sockets Ice Lake with 256G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 25%
	cluster: cs-localhost
	test: UDP_RR
	cpufreq_governor: performance
	ucode: 0xb000280

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


also as Feng Tang checked, this is a "unstable clocksource" case.
attached dmesg FYI.


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/25%/debian-10.4-x86_64-20200603.cgz/300s/lkp-icl-2sp1/UDP_RR/netperf/0xb000280

commit: 
  a48ef5b9b6 ("clocksource: Limit number of CPUs checked for clock synchronization")
  388450c708 ("clocksource: Reduce clocksource-skew threshold for TSC")

a48ef5b9b65fab26 388450c7081ded73432e2b7148c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   2552064           -65.1%     891438        netperf.Throughput_total_tps
    106336           -65.1%      37143        netperf.Throughput_tps
      9493           -48.5%       4889        netperf.time.involuntary_context_switches
      8378 ±  2%      -8.5%       7666 ±  2%  netperf.time.minor_page_faults
      1113           +14.5%       1274        netperf.time.percent_of_cpu_this_job_got
      3114           +16.5%       3628        netperf.time.system_time
    238.81           -11.8%     210.71        netperf.time.user_time
 7.656e+08           -65.1%  2.674e+08        netperf.time.voluntary_context_switches
 7.656e+08           -65.1%  2.674e+08        netperf.workload
    639.13           -14.2%     548.45        pmeter.Average_Active_Power
     25129           +18.2%      29715        uptime.idle
     62530           -27.8%      45143        slabinfo.radix_tree_node.active_objs
      1116           -27.8%     805.67        slabinfo.radix_tree_node.active_slabs
     62530           -27.8%      45143        slabinfo.radix_tree_node.num_objs
      1116           -27.8%     805.67        slabinfo.radix_tree_node.num_slabs
 1.527e+09           -98.8%   17975152 ±  5%  turbostat.C1
     20.76           -20.1        0.62 ±  7%  turbostat.C1%
  18935941 ± 13%   +2701.4%  5.305e+08        turbostat.C1E
     65.67 ±  2%      +7.9%      70.83 ±  3%  turbostat.PkgTmp
 6.049e+09           -97.0%  1.799e+08 ±  7%  cpuidle.C1.time
 1.527e+09           -98.8%   17975177 ±  5%  cpuidle.C1.usage
  18936233 ± 13%   +2701.3%  5.305e+08        cpuidle.C1E.usage
  26807255           +66.2%   44545592 ± 17%  cpuidle.POLL.time
   3856520 ±  3%     -56.2%    1688057 ± 17%  cpuidle.POLL.usage
     74.02           +15.6       89.60        mpstat.cpu.all.idle%
      0.62 ±  2%      -0.6        0.00        mpstat.cpu.all.irq%
      6.57            -4.3        2.23        mpstat.cpu.all.soft%
     17.04            -9.7        7.31        mpstat.cpu.all.sys%
      1.74            -0.9        0.86        mpstat.cpu.all.usr%
     66034 ± 11%   +5978.5%    4013902 ± 49%  numa-meminfo.node1.Active
     65933 ± 11%   +5987.8%    4013878 ± 49%  numa-meminfo.node1.Active(anon)
   6475043 ± 61%     -91.9%     521267 ± 41%  numa-meminfo.node1.Inactive
   6474945 ± 61%     -91.9%     521242 ± 41%  numa-meminfo.node1.Inactive(anon)
   2795743 ± 50%     -77.9%     617984 ± 39%  numa-meminfo.node1.Mapped
     16495 ± 11%   +5996.2%    1005570 ± 49%  numa-vmstat.node1.nr_active_anon
   1619161 ± 61%     -92.0%     129300 ± 41%  numa-vmstat.node1.nr_inactive_anon
    698820 ± 50%     -78.0%     153526 ± 40%  numa-vmstat.node1.nr_mapped
     16495 ± 11%   +5996.2%    1005570 ± 49%  numa-vmstat.node1.nr_zone_active_anon
   1619160 ± 61%     -92.0%     129300 ± 41%  numa-vmstat.node1.nr_zone_inactive_anon
     72.00           +20.8%      87.00        vmstat.cpu.id
  12674271           -34.8%    8264276        vmstat.memory.cache
     28.83           -13.3%      25.00        vmstat.procs.r
  10041753           -65.0%    3517904        vmstat.system.cs
    204196            +9.3%     223115        vmstat.system.in
     79614         +8119.8%    6544149        meminfo.Active
     79459         +8135.6%    6543994        meminfo.Active(anon)
  12515837           -35.1%    8122403        meminfo.Cached
  11946542           -37.0%    7529199        meminfo.Committed_AS
  11747198           -92.6%     867161 ±  6%  meminfo.Inactive
  11747048           -92.6%     867014 ±  6%  meminfo.Inactive(anon)
   4315200 ±  6%     -77.7%     963241 ±  4%  meminfo.Mapped
  14185177           -32.1%    9630648        meminfo.Memused
     14936 ±  3%     -44.1%       8345        meminfo.PageTables
  11547928           -38.0%    7154494        meminfo.Shmem
  14306545           -30.9%    9884359        meminfo.max_used_kB
     19809         +8135.2%    1631378        proc-vmstat.nr_active_anon
     69703            -7.8%      64270        proc-vmstat.nr_anon_pages
   6203411            +1.8%    6316647        proc-vmstat.nr_dirty_background_threshold
  12421991            +1.8%   12648739        proc-vmstat.nr_dirty_threshold
   3120664           -35.0%    2026997        proc-vmstat.nr_file_pages
  62410280            +1.8%   63544296        proc-vmstat.nr_free_pages
   2928510           -92.6%     217781 ±  6%  proc-vmstat.nr_inactive_anon
   1070932 ±  7%     -77.2%     244114 ±  3%  proc-vmstat.nr_mapped
      3722 ±  3%     -43.7%       2097        proc-vmstat.nr_page_table_pages
   2878686           -38.0%    1785019        proc-vmstat.nr_shmem
     28786            -8.7%      26291        proc-vmstat.nr_slab_reclaimable
     19809         +8135.2%    1631378        proc-vmstat.nr_zone_active_anon
   2928510           -92.6%     217781 ±  6%  proc-vmstat.nr_zone_inactive_anon
    605165 ±  4%     -70.0%     181791 ±  2%  proc-vmstat.numa_hint_faults
    500428 ±  3%     -82.6%      86975 ±  6%  proc-vmstat.numa_hint_faults_local
   8324661           -34.0%    5497994        proc-vmstat.numa_hit
   8238072           -34.3%    5411430        proc-vmstat.numa_local
    835983 ±  7%     -60.4%     330746 ±  4%  proc-vmstat.numa_pte_updates
     21697         +6821.8%    1501840 ±  6%  proc-vmstat.pgactivate
   8398800           -33.8%    5558275        proc-vmstat.pgalloc_normal
   1964293           -25.0%    1474197        proc-vmstat.pgfault
    875949           +13.2%     991262        proc-vmstat.pgfree
     78682 ± 15%    +185.8%     224886 ±  7%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   1076987 ±  2%     +24.5%    1340633        sched_debug.cfs_rq:/.MIN_vruntime.max
    260325 ±  7%     +69.4%     441009 ±  2%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     78682 ± 15%    +185.8%     224886 ±  7%  sched_debug.cfs_rq:/.max_vruntime.avg
   1076987 ±  2%     +24.5%    1340633        sched_debug.cfs_rq:/.max_vruntime.max
    260325 ±  7%     +69.4%     441009 ±  2%  sched_debug.cfs_rq:/.max_vruntime.stddev
    948642           +12.9%    1070711        sched_debug.cfs_rq:/.min_vruntime.avg
    604932 ±  4%     +13.7%     687739 ±  4%  sched_debug.cfs_rq:/.min_vruntime.min
    109825 ±  7%     +45.9%     160254 ±  5%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.38 ±  5%     -11.8%       0.33 ±  6%  sched_debug.cfs_rq:/.nr_running.avg
    844.78 ±  3%     -14.2%     725.17 ±  8%  sched_debug.cfs_rq:/.runnable_avg.max
     -3091         +2594.8%     -83301        sched_debug.cfs_rq:/.spread0.min
    109825 ±  7%     +45.9%     160258 ±  5%  sched_debug.cfs_rq:/.spread0.stddev
    843.81 ±  3%     -14.8%     719.14 ±  8%  sched_debug.cfs_rq:/.util_avg.max
    189.98 ±  6%     -15.9%     159.82 ±  8%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    795.50 ±  2%     -20.5%     632.78 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.max
      2800 ±  2%    +284.1%      10757 ± 47%  sched_debug.cpu.avg_idle.min
      1206 ±  5%     -55.2%     539.86 ±  4%  sched_debug.cpu.clock_task.stddev
    711.73 ±  6%     -44.5%     395.18 ±  2%  sched_debug.cpu.curr->pid.avg
      1274 ±  2%     -15.1%       1081        sched_debug.cpu.curr->pid.stddev
      0.31 ±  3%     -12.2%       0.27 ±  3%  sched_debug.cpu.nr_running.avg
  15846045           -65.2%    5507786        sched_debug.cpu.nr_switches.avg
  20528236 ±  4%     -63.6%    7475974        sched_debug.cpu.nr_switches.max
   9220868 ±  5%     -63.1%    3406063 ±  6%  sched_debug.cpu.nr_switches.min
   1692698 ±  6%     -49.9%     848477 ±  5%  sched_debug.cpu.nr_switches.stddev
      1.00          -100.0%       0.00        sched_debug.sched_clock_stable()
     17.42            -1.3%      17.19        perf-stat.i.MPKI
 1.729e+10           -60.3%  6.871e+09        perf-stat.i.branch-instructions
      1.75            +0.2        1.92        perf-stat.i.branch-miss-rate%
 2.942e+08           -56.6%  1.276e+08        perf-stat.i.branch-misses
      0.92 ±  7%      +0.7        1.65 ±  3%  perf-stat.i.cache-miss-rate%
  12221846 ±  4%     -22.7%    9443692 ±  3%  perf-stat.i.cache-misses
 1.473e+09           -61.4%  5.683e+08        perf-stat.i.cache-references
  10129998           -65.0%    3546333        perf-stat.i.context-switches
      1.46          +133.2%       3.41        perf-stat.i.cpi
 1.243e+11            -9.3%  1.128e+11        perf-stat.i.cpu-cycles
    274.12            -7.8%     252.77 ±  2%  perf-stat.i.cpu-migrations
      0.03 ± 11%      -0.0        0.02 ±  6%  perf-stat.i.dTLB-load-miss-rate%
   7714060 ±  3%     -80.1%    1532509 ±  6%  perf-stat.i.dTLB-load-misses
 2.493e+10           -62.3%  9.411e+09        perf-stat.i.dTLB-loads
    413851 ±  7%     -53.3%     193085 ±  5%  perf-stat.i.dTLB-store-misses
 1.517e+10           -63.6%  5.519e+09        perf-stat.i.dTLB-stores
  8.66e+10           -60.7%  3.407e+10        perf-stat.i.instructions
      0.69           -56.6%       0.30        perf-stat.i.ipc
      1.30            -9.3%       1.17        perf-stat.i.metric.GHz
    613.38           -62.0%     233.08        perf-stat.i.metric.M/sec
      6360           -25.0%       4772        perf-stat.i.minor-faults
     62.26 ±  9%     +33.5       95.78        perf-stat.i.node-load-miss-rate%
    641062 ± 41%    +106.7%    1325277 ±  2%  perf-stat.i.node-load-misses
   1270854 ± 15%     -93.6%      81705 ± 21%  perf-stat.i.node-loads
     85.77           -33.6       52.21        perf-stat.i.node-store-miss-rate%
   2320340           -54.6%    1052549        perf-stat.i.node-store-misses
   1605640           +10.1%    1767572        perf-stat.i.node-stores
      6361           -25.0%       4772        perf-stat.i.page-faults
     17.01            -1.9%      16.68        perf-stat.overall.MPKI
      1.70            +0.2        1.86        perf-stat.overall.branch-miss-rate%
      0.83 ±  5%      +0.8        1.66 ±  3%  perf-stat.overall.cache-miss-rate%
      1.44          +130.5%       3.31        perf-stat.overall.cpi
     10192 ±  4%     +17.2%      11943 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.03 ±  3%      -0.0        0.02 ±  6%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  7%      +0.0        0.00 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
      0.70           -56.6%       0.30        perf-stat.overall.ipc
     33.04 ± 35%     +61.2       94.21        perf-stat.overall.node-load-miss-rate%
     59.09           -21.8       37.32        perf-stat.overall.node-store-miss-rate%
     34154           +12.3%      38365        perf-stat.overall.path-length
 1.723e+10           -60.3%  6.847e+09        perf-stat.ps.branch-instructions
 2.932e+08           -56.6%  1.272e+08        perf-stat.ps.branch-misses
  12182782 ±  4%     -22.7%    9416463 ±  3%  perf-stat.ps.cache-misses
 1.468e+09           -61.4%  5.664e+08        perf-stat.ps.cache-references
  10094934           -65.0%    3533893        perf-stat.ps.context-switches
 1.239e+11            -9.3%  1.124e+11        perf-stat.ps.cpu-cycles
    273.37            -7.8%     252.06 ±  2%  perf-stat.ps.cpu-migrations
   7683633 ±  3%     -80.1%    1527379 ±  6%  perf-stat.ps.dTLB-load-misses
 2.485e+10           -62.3%  9.379e+09        perf-stat.ps.dTLB-loads
    412328 ±  7%     -53.3%     192438 ±  5%  perf-stat.ps.dTLB-store-misses
 1.512e+10           -63.6%    5.5e+09        perf-stat.ps.dTLB-stores
 8.631e+10           -60.7%  3.395e+10        perf-stat.ps.instructions
      6338           -25.0%       4754        perf-stat.ps.minor-faults
    638936 ± 41%    +106.7%    1320628 ±  2%  perf-stat.ps.node-load-misses
   1266002 ± 15%     -93.6%      81460 ± 21%  perf-stat.ps.node-loads
   2312502           -54.6%    1049001        perf-stat.ps.node-store-misses
   1601288           +10.0%    1761928        perf-stat.ps.node-stores
      6338           -25.0%       4755        perf-stat.ps.page-faults
 2.615e+13           -60.8%  1.026e+13        perf-stat.total.instructions
   4863693 ±  3%     -63.3%    1784617 ± 13%  softirqs.CPU0.NET_RX
   8390807 ±  6%     -64.3%    2996025 ±  8%  softirqs.CPU1.NET_RX
   7739940           -62.3%    2916844 ± 10%  softirqs.CPU10.NET_RX
   7995905           -63.8%    2894634 ±  6%  softirqs.CPU11.NET_RX
   7672503 ±  6%     -64.7%    2710697 ±  5%  softirqs.CPU12.NET_RX
   7845055 ±  5%     -63.6%    2858075 ±  4%  softirqs.CPU13.NET_RX
   8007447 ±  2%     -63.6%    2914695 ±  8%  softirqs.CPU14.NET_RX
   7921355 ±  8%     -63.6%    2885268 ± 10%  softirqs.CPU15.NET_RX
   8134485 ±  5%     -64.7%    2873713 ±  3%  softirqs.CPU16.NET_RX
   8369520 ±  4%     -65.7%    2869899 ±  9%  softirqs.CPU17.NET_RX
   8431236 ±  5%     -64.6%    2983107 ±  3%  softirqs.CPU18.NET_RX
   8194774 ±  4%     -64.1%    2942680 ±  5%  softirqs.CPU19.NET_RX
   8131381 ±  4%     -61.6%    3126035 ±  4%  softirqs.CPU2.NET_RX
   8635834 ±  5%     -65.0%    3020495 ±  3%  softirqs.CPU20.NET_RX
   8660660 ±  6%     -65.5%    2991711 ±  5%  softirqs.CPU21.NET_RX
   8842820 ±  3%     -67.0%    2915579 ±  8%  softirqs.CPU22.NET_RX
   8703612 ±  5%     -66.7%    2896298 ±  6%  softirqs.CPU23.NET_RX
   7667353 ±  4%     -60.9%    2995334 ±  7%  softirqs.CPU24.NET_RX
   8202704 ±  6%     -61.3%    3176485 ±  3%  softirqs.CPU25.NET_RX
   8035819 ±  3%     -59.0%    3294811 ±  6%  softirqs.CPU26.NET_RX
   7667060 ±  2%     -56.2%    3355323 ±  7%  softirqs.CPU27.NET_RX
   7818298 ±  4%     -55.7%    3460870 ±  4%  softirqs.CPU28.NET_RX
   8209299 ±  6%     -58.9%    3372827 ±  4%  softirqs.CPU29.NET_RX
   7826514 ± 14%     -60.1%    3119356 ±  3%  softirqs.CPU3.NET_RX
   7905831 ±  5%     -56.3%    3457821 ±  5%  softirqs.CPU30.NET_RX
   7920713 ±  5%     -58.6%    3278887 ±  5%  softirqs.CPU31.NET_RX
   8192169 ±  3%     -59.5%    3316165 ±  5%  softirqs.CPU32.NET_RX
   7941276 ±  4%     -59.7%    3200765 ±  5%  softirqs.CPU33.NET_RX
   7504068 ± 14%     -57.8%    3166178 ±  4%  softirqs.CPU34.NET_RX
   7870330 ±  3%     -59.7%    3170329 ±  6%  softirqs.CPU35.NET_RX
   7943349 ±  3%     -61.4%    3064142 ±  4%  softirqs.CPU36.NET_RX
   7937130 ±  3%     -62.0%    3013094 ±  7%  softirqs.CPU37.NET_RX
   8267476 ±  3%     -62.1%    3133563 ±  4%  softirqs.CPU38.NET_RX
   8028779 ±  7%     -62.3%    3024788 ±  6%  softirqs.CPU39.NET_RX
   8219986 ±  4%     -62.4%    3093371 ±  5%  softirqs.CPU4.NET_RX
   8113648 ±  6%     -62.5%    3040010 ±  7%  softirqs.CPU40.NET_RX
   8356796 ±  4%     -63.1%    3081411 ±  5%  softirqs.CPU41.NET_RX
   8278770 ±  5%     -62.1%    3138752 ±  5%  softirqs.CPU42.NET_RX
   8260552 ±  7%     -63.0%    3058037 ±  4%  softirqs.CPU43.NET_RX
   7882157 ±  2%     -59.8%    3172162 ±  6%  softirqs.CPU44.NET_RX
   8316423 ±  2%     -60.1%    3320176 ±  8%  softirqs.CPU45.NET_RX
   8444938 ±  3%     -62.5%    3169460 ±  3%  softirqs.CPU46.NET_RX
   7083563 ± 20%     -54.6%    3216099        softirqs.CPU47.NET_RX
  10454503 ±  5%     -71.4%    2986371 ±  6%  softirqs.CPU48.NET_RX
   8629502 ±  5%     -67.8%    2778512 ±  4%  softirqs.CPU49.NET_RX
   8049381 ±  5%     -59.7%    3240792 ±  5%  softirqs.CPU5.NET_RX
   8874688 ±  4%     -70.7%    2597980 ±  5%  softirqs.CPU50.NET_RX
   7903829 ± 16%     -65.5%    2730206 ±  6%  softirqs.CPU51.NET_RX
   8468110 ±  3%     -70.3%    2512198 ±  6%  softirqs.CPU52.NET_RX
   8696922 ±  5%     -72.3%    2405739 ±  7%  softirqs.CPU53.NET_RX
   8425470           -69.4%    2574360 ±  8%  softirqs.CPU54.NET_RX
   7831673 ±  4%     -68.6%    2461209 ±  7%  softirqs.CPU55.NET_RX
   8735067 ± 11%     -71.3%    2511232 ± 10%  softirqs.CPU56.NET_RX
   8383685 ±  4%     -70.2%    2496584 ± 11%  softirqs.CPU57.NET_RX
   8306674 ±  3%     -69.9%    2501269 ±  9%  softirqs.CPU58.NET_RX
   8197548 ±  4%     -68.8%    2558582 ±  3%  softirqs.CPU59.NET_RX
   8024359 ±  2%     -60.5%    3171087 ±  8%  softirqs.CPU6.NET_RX
   8271211 ±  8%     -69.0%    2564369 ±  5%  softirqs.CPU60.NET_RX
   8082682 ±  7%     -69.7%    2446964 ±  3%  softirqs.CPU61.NET_RX
   8119390 ±  2%     -70.6%    2389317 ±  5%  softirqs.CPU62.NET_RX
   7646737 ±  8%     -67.3%    2503721 ±  8%  softirqs.CPU63.NET_RX
   7949118 ±  6%     -68.7%    2490606 ±  7%  softirqs.CPU64.NET_RX
   7744965 ±  3%     -68.7%    2421934 ±  9%  softirqs.CPU65.NET_RX
   7472381 ±  8%     -69.8%    2258480 ±  5%  softirqs.CPU66.NET_RX
   7651658 ±  9%     -69.0%    2368252 ± 11%  softirqs.CPU67.NET_RX
   7442612 ±  5%     -70.4%    2202676 ± 10%  softirqs.CPU68.NET_RX
   7015192 ±  6%     -68.1%    2239285 ± 12%  softirqs.CPU69.NET_RX
   8345794 ±  5%     -61.7%    3193281 ±  7%  softirqs.CPU7.NET_RX
   6965622 ±  5%     -67.7%    2248121 ± 10%  softirqs.CPU70.NET_RX
   6968466 ±  8%     -68.9%    2165129 ±  4%  softirqs.CPU71.NET_RX
   9305958 ±  3%     -66.3%    3138664 ±  4%  softirqs.CPU72.NET_RX
   8453497 ±  6%     -65.8%    2891626 ±  5%  softirqs.CPU73.NET_RX
   8801156 ±  4%     -69.6%    2678589 ±  5%  softirqs.CPU74.NET_RX
   8525358 ±  5%     -69.9%    2563539 ±  8%  softirqs.CPU75.NET_RX
   8427135 ±  6%     -69.6%    2562708 ±  5%  softirqs.CPU76.NET_RX
   8091793 ±  6%     -67.8%    2604572 ± 12%  softirqs.CPU77.NET_RX
   8222394 ±  4%     -69.6%    2503244 ± 14%  softirqs.CPU78.NET_RX
   8021023 ±  8%     -67.0%    2646990 ± 10%  softirqs.CPU79.NET_RX
   7123789 ±  7%     -57.4%    3034199 ±  7%  softirqs.CPU8.NET_RX
   7907282 ±  3%     -68.3%    2503273 ± 14%  softirqs.CPU80.NET_RX
   8064970 ±  8%     -68.3%    2555326 ± 12%  softirqs.CPU81.NET_RX
   7408809 ± 16%     -65.4%    2561466 ±  2%  softirqs.CPU82.NET_RX
   8144151 ±  3%     -68.4%    2575811 ±  9%  softirqs.CPU83.NET_RX
   7689141 ±  6%     -66.7%    2563415 ±  9%  softirqs.CPU84.NET_RX
   7712230 ±  4%     -65.6%    2649338 ±  8%  softirqs.CPU85.NET_RX
   7673765 ±  4%     -68.2%    2440968 ±  5%  softirqs.CPU86.NET_RX
   7847121 ±  4%     -68.7%    2455763 ± 12%  softirqs.CPU87.NET_RX
   7598116 ±  5%     -65.8%    2599653 ± 11%  softirqs.CPU88.NET_RX
   7823019 ±  3%     -66.9%    2587190 ±  2%  softirqs.CPU89.NET_RX
   7964223 ±  3%     -62.8%    2962506 ±  8%  softirqs.CPU9.NET_RX
   7170381 ±  4%     -64.6%    2539672 ±  6%  softirqs.CPU90.NET_RX
   7402249 ±  6%     -65.3%    2568897 ±  4%  softirqs.CPU91.NET_RX
   7427317 ±  3%     -69.0%    2301245 ±  8%  softirqs.CPU92.NET_RX
   7488111 ±  2%     -70.0%    2246435 ±  9%  softirqs.CPU93.NET_RX
   7161998 ±  6%     -67.9%    2301183 ±  3%  softirqs.CPU94.NET_RX
   5089290 ± 29%     -62.2%    1925573 ±  4%  softirqs.CPU95.NET_RX
 7.656e+08           -65.1%  2.675e+08        softirqs.NET_RX
    286032 ± 86%     -86.5%      38694 ± 34%  interrupts.CAL:Function_call_interrupts
      3451 ± 89%     -70.6%       1014 ± 41%  interrupts.CPU0.CAL:Function_call_interrupts
     19257 ±  5%    +206.8%      59079 ± 10%  interrupts.CPU0.RES:Rescheduling_interrupts
      3514 ± 60%     -74.1%     910.33 ± 73%  interrupts.CPU1.CAL:Function_call_interrupts
    127.67 ± 17%     -35.2%      82.67 ± 20%  interrupts.CPU1.IWI:IRQ_work_interrupts
     30556 ±  6%    +227.1%      99939 ±  8%  interrupts.CPU1.RES:Rescheduling_interrupts
      2842 ± 86%     -88.7%     322.50 ± 43%  interrupts.CPU10.CAL:Function_call_interrupts
     27500 ±  2%    +252.8%      97010 ± 10%  interrupts.CPU10.RES:Rescheduling_interrupts
      2835 ± 88%     -85.6%     408.50 ± 29%  interrupts.CPU11.CAL:Function_call_interrupts
     30707 ±  9%    +212.2%      95866 ±  6%  interrupts.CPU11.RES:Rescheduling_interrupts
      3005 ± 90%     -86.6%     401.33 ± 33%  interrupts.CPU12.CAL:Function_call_interrupts
     29040 ±  8%    +211.8%      90551 ±  6%  interrupts.CPU12.RES:Rescheduling_interrupts
      2349 ± 48%     -85.3%     344.83 ± 40%  interrupts.CPU13.CAL:Function_call_interrupts
     28673 ±  8%    +233.7%      95693 ±  5%  interrupts.CPU13.RES:Rescheduling_interrupts
      2601 ± 83%     -88.1%     309.50 ± 45%  interrupts.CPU14.CAL:Function_call_interrupts
      7782 ± 15%     -20.8%       6163 ± 22%  interrupts.CPU14.NMI:Non-maskable_interrupts
      7782 ± 15%     -20.8%       6163 ± 22%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
     28922 ±  2%    +235.7%      97099 ±  9%  interrupts.CPU14.RES:Rescheduling_interrupts
      2824 ± 69%     -88.8%     317.67 ± 43%  interrupts.CPU15.CAL:Function_call_interrupts
     28498 ±  8%    +235.2%      95519 ± 11%  interrupts.CPU15.RES:Rescheduling_interrupts
      2684 ± 74%     -88.7%     302.67 ± 50%  interrupts.CPU16.CAL:Function_call_interrupts
     29427 ±  5%    +223.2%      95105 ±  4%  interrupts.CPU16.RES:Rescheduling_interrupts
      3160 ± 82%     -90.0%     316.00 ± 45%  interrupts.CPU17.CAL:Function_call_interrupts
     30523 ±  3%    +211.1%      94965 ±  9%  interrupts.CPU17.RES:Rescheduling_interrupts
      3704 ±125%     -91.0%     333.00 ± 47%  interrupts.CPU18.CAL:Function_call_interrupts
     30460 ±  4%    +223.1%      98410 ±  4%  interrupts.CPU18.RES:Rescheduling_interrupts
      3204 ±103%     -90.0%     320.00 ± 42%  interrupts.CPU19.CAL:Function_call_interrupts
     29430 ±  5%    +229.3%      96915 ±  8%  interrupts.CPU19.RES:Rescheduling_interrupts
     29776 ±  4%    +251.9%     104776 ±  5%  interrupts.CPU2.RES:Rescheduling_interrupts
      3451 ±110%     -90.0%     345.50 ± 42%  interrupts.CPU20.CAL:Function_call_interrupts
     31388 ±  5%    +218.8%     100052 ±  4%  interrupts.CPU20.RES:Rescheduling_interrupts
      3259 ± 80%     -90.9%     298.00 ± 48%  interrupts.CPU21.CAL:Function_call_interrupts
     31581 ±  6%    +215.3%      99561 ±  6%  interrupts.CPU21.RES:Rescheduling_interrupts
      3710 ±107%     -91.7%     308.00 ± 43%  interrupts.CPU22.CAL:Function_call_interrupts
    152.83 ±  7%     -26.7%     112.00 ± 15%  interrupts.CPU22.IWI:IRQ_work_interrupts
     32670 ±  3%    +198.3%      97453 ± 10%  interrupts.CPU22.RES:Rescheduling_interrupts
      2706 ± 60%     -87.4%     340.17 ± 42%  interrupts.CPU23.CAL:Function_call_interrupts
     32641 ±  5%    +195.4%      96416 ±  7%  interrupts.CPU23.RES:Rescheduling_interrupts
      3098 ± 82%     -87.9%     374.50 ± 39%  interrupts.CPU24.CAL:Function_call_interrupts
     27419 ±  6%    +234.9%      91830 ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
      2911 ± 77%     -86.4%     397.33 ± 42%  interrupts.CPU25.CAL:Function_call_interrupts
     28869 ±  9%    +240.3%      98234 ±  3%  interrupts.CPU25.RES:Rescheduling_interrupts
      2802 ± 92%     -87.7%     345.00 ± 55%  interrupts.CPU26.CAL:Function_call_interrupts
     28342 ±  3%    +262.3%     102693 ±  6%  interrupts.CPU26.RES:Rescheduling_interrupts
      2979 ±102%     -88.2%     350.83 ± 40%  interrupts.CPU27.CAL:Function_call_interrupts
     26142 ±  2%    +298.6%     104201 ±  6%  interrupts.CPU27.RES:Rescheduling_interrupts
      2376 ± 64%     -81.2%     447.67 ± 60%  interrupts.CPU28.CAL:Function_call_interrupts
     26581 ±  3%    +301.9%     106824 ±  5%  interrupts.CPU28.RES:Rescheduling_interrupts
      3740 ±137%     -91.3%     325.00 ± 47%  interrupts.CPU29.CAL:Function_call_interrupts
      4645 ± 29%     +57.0%       7294 ± 13%  interrupts.CPU29.NMI:Non-maskable_interrupts
      4645 ± 29%     +57.0%       7294 ± 13%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
     28734 ±  6%    +259.7%     103359 ±  3%  interrupts.CPU29.RES:Rescheduling_interrupts
      3000 ± 77%     -88.3%     351.17 ± 48%  interrupts.CPU3.CAL:Function_call_interrupts
     28213 ± 14%    +267.0%     103531 ±  4%  interrupts.CPU3.RES:Rescheduling_interrupts
      3232 ±125%     -90.0%     324.83 ± 43%  interrupts.CPU30.CAL:Function_call_interrupts
     26938 ±  6%    +296.6%     106834 ±  6%  interrupts.CPU30.RES:Rescheduling_interrupts
      2962 ±100%     -89.0%     325.50 ± 45%  interrupts.CPU31.CAL:Function_call_interrupts
     28530 ± 12%    +254.2%     101045 ±  6%  interrupts.CPU31.RES:Rescheduling_interrupts
      2685 ±103%     -86.8%     354.33 ± 35%  interrupts.CPU32.CAL:Function_call_interrupts
     27297 ±  6%    +276.2%     102682 ±  3%  interrupts.CPU32.RES:Rescheduling_interrupts
      2440 ± 95%     -84.2%     385.00 ± 40%  interrupts.CPU33.CAL:Function_call_interrupts
     27354 ±  2%    +264.1%      99598 ±  5%  interrupts.CPU33.RES:Rescheduling_interrupts
      3145 ±117%     -88.4%     365.67 ± 40%  interrupts.CPU34.CAL:Function_call_interrupts
     25952 ± 15%    +274.8%      97271 ±  6%  interrupts.CPU34.RES:Rescheduling_interrupts
      3064 ±113%     -88.8%     342.17 ± 42%  interrupts.CPU35.CAL:Function_call_interrupts
     27847 ±  4%    +251.2%      97813 ±  7%  interrupts.CPU35.RES:Rescheduling_interrupts
      1981 ± 35%     -81.9%     358.67 ± 42%  interrupts.CPU36.CAL:Function_call_interrupts
     28654 ±  3%    +231.0%      94855 ±  4%  interrupts.CPU36.RES:Rescheduling_interrupts
      2818 ±101%     -87.9%     341.83 ± 43%  interrupts.CPU37.CAL:Function_call_interrupts
     28630 ±  4%    +224.5%      92893 ±  7%  interrupts.CPU37.RES:Rescheduling_interrupts
      3472 ±127%     -85.7%     497.67 ± 67%  interrupts.CPU38.CAL:Function_call_interrupts
     28809 ±  4%    +236.9%      97073 ±  7%  interrupts.CPU38.RES:Rescheduling_interrupts
     25.67 ±152%     -96.8%       0.83 ±107%  interrupts.CPU38.TLB:TLB_shootdowns
      2522 ± 93%     -85.9%     356.83 ± 39%  interrupts.CPU39.CAL:Function_call_interrupts
    132.00 ± 19%     -43.7%      74.33 ± 31%  interrupts.CPU39.IWI:IRQ_work_interrupts
     28626 ±  9%    +228.5%      94035 ±  7%  interrupts.CPU39.RES:Rescheduling_interrupts
      2024 ± 20%     -73.2%     542.17 ± 79%  interrupts.CPU4.CAL:Function_call_interrupts
     29504 ±  4%    +249.3%     103064 ±  4%  interrupts.CPU4.RES:Rescheduling_interrupts
      3371 ±111%     -85.3%     496.67 ± 65%  interrupts.CPU40.CAL:Function_call_interrupts
     29374 ±  7%    +222.0%      94573 ±  5%  interrupts.CPU40.RES:Rescheduling_interrupts
      3558 ±114%     -90.3%     345.67 ± 42%  interrupts.CPU41.CAL:Function_call_interrupts
     29469 ±  5%    +219.9%      94267 ±  6%  interrupts.CPU41.RES:Rescheduling_interrupts
      3039 ± 92%     -88.5%     349.83 ± 42%  interrupts.CPU42.CAL:Function_call_interrupts
     30316 ±  6%    +219.6%      96895 ±  6%  interrupts.CPU42.RES:Rescheduling_interrupts
      2564 ± 86%     -86.0%     359.33 ± 36%  interrupts.CPU43.CAL:Function_call_interrupts
     29890 ± 10%    +215.3%      94254 ±  6%  interrupts.CPU43.RES:Rescheduling_interrupts
      3165 ±108%     -83.5%     521.00 ± 62%  interrupts.CPU44.CAL:Function_call_interrupts
     29111 ±  2%    +240.2%      99026 ±  8%  interrupts.CPU44.RES:Rescheduling_interrupts
      3290 ±105%     -87.9%     398.83 ± 32%  interrupts.CPU45.CAL:Function_call_interrupts
    123.83 ± 13%     -44.0%      69.33 ± 39%  interrupts.CPU45.IWI:IRQ_work_interrupts
     30439 ±  3%    +241.1%     103830 ± 10%  interrupts.CPU45.RES:Rescheduling_interrupts
      2685 ± 60%     -85.7%     385.17 ± 29%  interrupts.CPU46.CAL:Function_call_interrupts
     30271 ±  4%    +226.9%      98955 ±  5%  interrupts.CPU46.RES:Rescheduling_interrupts
      2460 ± 61%     -83.6%     402.83 ± 39%  interrupts.CPU47.CAL:Function_call_interrupts
     24896 ± 17%    +309.5%     101944 ±  2%  interrupts.CPU47.RES:Rescheduling_interrupts
      4131 ± 56%     -91.7%     342.17 ± 71%  interrupts.CPU48.CAL:Function_call_interrupts
    229.33 ± 10%     -67.8%      73.83 ± 29%  interrupts.CPU48.IWI:IRQ_work_interrupts
     32291 ±  6%    +213.0%     101059 ±  8%  interrupts.CPU48.RES:Rescheduling_interrupts
      2975 ± 65%     -82.3%     527.83 ±117%  interrupts.CPU49.CAL:Function_call_interrupts
     25835 ±  3%    +256.4%      92091 ±  4%  interrupts.CPU49.RES:Rescheduling_interrupts
      4078 ±120%     -92.4%     308.50 ± 51%  interrupts.CPU5.CAL:Function_call_interrupts
     29451 ±  7%    +265.0%     107495 ±  7%  interrupts.CPU5.RES:Rescheduling_interrupts
      2088 ± 16%     -83.9%     335.67 ± 39%  interrupts.CPU50.CAL:Function_call_interrupts
    183.00 ±  8%     -44.4%     101.67 ± 31%  interrupts.CPU50.IWI:IRQ_work_interrupts
     25789 ±  4%    +233.4%      85992 ±  6%  interrupts.CPU50.RES:Rescheduling_interrupts
      2140 ± 30%     -81.7%     392.17 ± 54%  interrupts.CPU51.CAL:Function_call_interrupts
    182.17 ± 21%     -49.0%      92.83 ± 44%  interrupts.CPU51.IWI:IRQ_work_interrupts
     24006 ± 13%    +280.3%      91303 ±  7%  interrupts.CPU51.RES:Rescheduling_interrupts
      2632 ± 59%     -78.1%     577.50 ± 91%  interrupts.CPU52.CAL:Function_call_interrupts
    165.00 ± 16%     -32.7%     111.00 ± 37%  interrupts.CPU52.IWI:IRQ_work_interrupts
     25342 ±  6%    +231.4%      83975 ±  7%  interrupts.CPU52.RES:Rescheduling_interrupts
      3081 ± 83%     -89.7%     316.83 ± 51%  interrupts.CPU53.CAL:Function_call_interrupts
    214.67 ± 19%     -60.8%      84.17 ± 37%  interrupts.CPU53.IWI:IRQ_work_interrupts
     25636 ±  6%    +214.3%      80579 ±  7%  interrupts.CPU53.RES:Rescheduling_interrupts
      3515 ± 92%     -87.2%     451.33 ± 64%  interrupts.CPU54.CAL:Function_call_interrupts
    187.67 ±  8%     -49.6%      94.50 ± 35%  interrupts.CPU54.IWI:IRQ_work_interrupts
     26324 ± 12%    +225.6%      85723 ±  5%  interrupts.CPU54.RES:Rescheduling_interrupts
      2106 ± 52%     -81.7%     386.33 ± 36%  interrupts.CPU55.CAL:Function_call_interrupts
    178.33 ± 26%     -55.6%      79.17 ± 26%  interrupts.CPU55.IWI:IRQ_work_interrupts
     23170 ±  3%    +254.3%      82097 ±  8%  interrupts.CPU55.RES:Rescheduling_interrupts
      2958 ± 63%     -84.0%     473.67 ± 70%  interrupts.CPU56.CAL:Function_call_interrupts
    211.50 ± 11%     -63.4%      77.33 ± 55%  interrupts.CPU56.IWI:IRQ_work_interrupts
     25318 ± 11%    +230.2%      83588 ± 10%  interrupts.CPU56.RES:Rescheduling_interrupts
      3311 ± 78%     -90.6%     312.33 ± 42%  interrupts.CPU57.CAL:Function_call_interrupts
    177.17 ± 21%     -61.3%      68.50 ± 61%  interrupts.CPU57.IWI:IRQ_work_interrupts
     23748 ±  3%    +247.5%      82534 ± 10%  interrupts.CPU57.RES:Rescheduling_interrupts
      2040 ± 22%     -82.7%     352.17 ± 55%  interrupts.CPU58.CAL:Function_call_interrupts
    189.33 ± 12%     -54.3%      86.50 ± 36%  interrupts.CPU58.IWI:IRQ_work_interrupts
     24089 ±  5%    +246.4%      83445 ± 10%  interrupts.CPU58.RES:Rescheduling_interrupts
      3202 ± 65%     -89.4%     340.00 ± 51%  interrupts.CPU59.CAL:Function_call_interrupts
    170.83 ± 22%     -38.0%     106.00 ± 36%  interrupts.CPU59.IWI:IRQ_work_interrupts
     23171 ±  7%    +268.2%      85320 ±  3%  interrupts.CPU59.RES:Rescheduling_interrupts
      3395 ± 82%     -90.3%     330.83 ± 44%  interrupts.CPU6.CAL:Function_call_interrupts
     28920 ±  2%    +262.1%     104724 ± 10%  interrupts.CPU6.RES:Rescheduling_interrupts
      2986 ± 91%     -88.4%     345.67 ± 35%  interrupts.CPU60.CAL:Function_call_interrupts
    182.00 ± 10%     -61.6%      69.83 ± 37%  interrupts.CPU60.IWI:IRQ_work_interrupts
     24300 ± 11%    +252.2%      85587 ±  5%  interrupts.CPU60.RES:Rescheduling_interrupts
      2271 ± 53%     -85.6%     326.33 ± 44%  interrupts.CPU61.CAL:Function_call_interrupts
    173.83 ± 23%     -49.1%      88.50 ± 22%  interrupts.CPU61.IWI:IRQ_work_interrupts
     23463 ±  4%    +245.5%      81067 ±  3%  interrupts.CPU61.RES:Rescheduling_interrupts
      2977 ± 89%     -84.4%     464.67 ± 46%  interrupts.CPU62.CAL:Function_call_interrupts
    179.33 ± 13%     -49.6%      90.33 ± 31%  interrupts.CPU62.IWI:IRQ_work_interrupts
     24321 ±  5%    +226.2%      79323 ±  5%  interrupts.CPU62.RES:Rescheduling_interrupts
      2538 ± 55%     -86.0%     355.83 ± 46%  interrupts.CPU63.CAL:Function_call_interrupts
     24810 ± 14%    +235.4%      83214 ±  6%  interrupts.CPU63.RES:Rescheduling_interrupts
      2666 ± 64%     -83.0%     452.67 ± 97%  interrupts.CPU64.CAL:Function_call_interrupts
    198.33 ±  7%     -60.2%      79.00 ± 29%  interrupts.CPU64.IWI:IRQ_work_interrupts
     23899 ±  4%    +245.8%      82634 ±  8%  interrupts.CPU64.RES:Rescheduling_interrupts
      4035 ±110%     -92.0%     322.83 ± 45%  interrupts.CPU65.CAL:Function_call_interrupts
    188.17 ± 16%     -49.8%      94.50 ± 30%  interrupts.CPU65.IWI:IRQ_work_interrupts
     23838 ±  6%    +236.2%      80137 ±  9%  interrupts.CPU65.RES:Rescheduling_interrupts
      3209 ± 99%     -88.6%     364.50 ± 43%  interrupts.CPU66.CAL:Function_call_interrupts
     22598 ±  7%    +231.5%      74917 ±  6%  interrupts.CPU66.RES:Rescheduling_interrupts
      4930 ±117%     -92.0%     392.00 ± 58%  interrupts.CPU67.CAL:Function_call_interrupts
     24093 ± 13%    +224.9%      78279 ± 10%  interrupts.CPU67.RES:Rescheduling_interrupts
      4320 ±133%     -89.1%     471.33 ± 70%  interrupts.CPU68.CAL:Function_call_interrupts
    172.83 ± 12%     -60.5%      68.33 ± 59%  interrupts.CPU68.IWI:IRQ_work_interrupts
     22871 ±  8%    +221.3%      73481 ± 10%  interrupts.CPU68.RES:Rescheduling_interrupts
      2078 ± 27%     -81.9%     376.83 ± 51%  interrupts.CPU69.CAL:Function_call_interrupts
    148.33 ± 32%     -52.4%      70.67 ± 64%  interrupts.CPU69.IWI:IRQ_work_interrupts
     22078 ± 10%    +239.7%      75006 ± 11%  interrupts.CPU69.RES:Rescheduling_interrupts
      1881 ± 31%     -82.5%     329.17 ± 44%  interrupts.CPU7.CAL:Function_call_interrupts
     29726 ±  6%    +255.6%     105711 ±  7%  interrupts.CPU7.RES:Rescheduling_interrupts
      2814 ± 81%     -86.1%     391.33 ± 38%  interrupts.CPU70.CAL:Function_call_interrupts
    143.67 ± 11%     -50.3%      71.33 ± 36%  interrupts.CPU70.IWI:IRQ_work_interrupts
     22063 ±  9%    +239.2%      74844 ±  9%  interrupts.CPU70.RES:Rescheduling_interrupts
      1918 ± 14%     -79.6%     391.83 ± 57%  interrupts.CPU71.CAL:Function_call_interrupts
     23347 ± 12%    +210.1%      72393 ±  5%  interrupts.CPU71.RES:Rescheduling_interrupts
      3504 ± 88%     -90.0%     351.33 ± 44%  interrupts.CPU72.CAL:Function_call_interrupts
    200.17 ± 17%     -60.5%      79.00 ± 50%  interrupts.CPU72.IWI:IRQ_work_interrupts
     28435 ±  5%    +236.2%      95606 ±  4%  interrupts.CPU72.RES:Rescheduling_interrupts
      3326 ± 88%     -89.7%     342.33 ± 42%  interrupts.CPU73.CAL:Function_call_interrupts
    205.33 ±  6%     -58.1%      86.00 ± 24%  interrupts.CPU73.IWI:IRQ_work_interrupts
     26276 ±  8%    +236.3%      88373 ±  5%  interrupts.CPU73.RES:Rescheduling_interrupts
      3511 ± 96%     -88.8%     393.33 ± 38%  interrupts.CPU74.CAL:Function_call_interrupts
    198.00 ± 18%     -54.8%      89.50 ± 30%  interrupts.CPU74.IWI:IRQ_work_interrupts
     26063 ±  5%    +216.5%      82482 ±  4%  interrupts.CPU74.RES:Rescheduling_interrupts
      3334 ± 96%     -85.6%     479.33 ± 51%  interrupts.CPU75.CAL:Function_call_interrupts
    209.33 ± 10%     -53.0%      98.33 ± 21%  interrupts.CPU75.IWI:IRQ_work_interrupts
     24945 ±  6%    +218.2%      79367 ± 10%  interrupts.CPU75.RES:Rescheduling_interrupts
      2969 ±103%     -86.5%     401.17 ± 33%  interrupts.CPU76.CAL:Function_call_interrupts
    187.17 ± 13%     -58.6%      77.50 ± 25%  interrupts.CPU76.IWI:IRQ_work_interrupts
     24388 ±  4%    +220.9%      78254 ±  5%  interrupts.CPU76.RES:Rescheduling_interrupts
      3323 ± 97%     -87.8%     406.67 ± 40%  interrupts.CPU77.CAL:Function_call_interrupts
    189.17 ± 11%     -54.0%      87.00 ± 42%  interrupts.CPU77.IWI:IRQ_work_interrupts
      6865 ±  8%     -34.2%       4515 ± 26%  interrupts.CPU77.NMI:Non-maskable_interrupts
      6865 ±  8%     -34.2%       4515 ± 26%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
     24997 ± 16%    +222.7%      80678 ± 13%  interrupts.CPU77.RES:Rescheduling_interrupts
      2755 ± 75%     -85.4%     401.67 ± 36%  interrupts.CPU78.CAL:Function_call_interrupts
    189.83 ± 13%     -53.9%      87.50 ± 19%  interrupts.CPU78.IWI:IRQ_work_interrupts
     23071 ±  9%    +232.0%      76599 ± 15%  interrupts.CPU78.RES:Rescheduling_interrupts
      2612 ± 83%     -85.6%     376.00 ± 35%  interrupts.CPU79.CAL:Function_call_interrupts
    195.17 ± 21%     -55.0%      87.83 ± 35%  interrupts.CPU79.IWI:IRQ_work_interrupts
     23168 ±  3%    +250.4%      81181 ± 11%  interrupts.CPU79.RES:Rescheduling_interrupts
      3874 ±125%     -90.6%     362.50 ± 43%  interrupts.CPU8.CAL:Function_call_interrupts
     25471 ± 10%    +293.0%     100101 ±  7%  interrupts.CPU8.RES:Rescheduling_interrupts
      4028 ±117%     -90.9%     367.33 ± 44%  interrupts.CPU80.CAL:Function_call_interrupts
      6621 ± 13%     -35.9%       4241 ± 37%  interrupts.CPU80.NMI:Non-maskable_interrupts
      6621 ± 13%     -35.9%       4241 ± 37%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
     21879 ±  3%    +251.2%      76830 ± 15%  interrupts.CPU80.RES:Rescheduling_interrupts
      2563 ± 60%     -85.6%     370.17 ± 33%  interrupts.CPU81.CAL:Function_call_interrupts
    191.83 ± 26%     -73.4%      51.00 ± 47%  interrupts.CPU81.IWI:IRQ_work_interrupts
     22362 ±  9%    +252.6%      78849 ± 13%  interrupts.CPU81.RES:Rescheduling_interrupts
      2382 ± 75%     -84.0%     382.00 ± 40%  interrupts.CPU82.CAL:Function_call_interrupts
    172.67 ± 17%     -54.1%      79.33 ± 52%  interrupts.CPU82.IWI:IRQ_work_interrupts
      6795 ± 18%     -54.0%       3123 ± 30%  interrupts.CPU82.NMI:Non-maskable_interrupts
      6795 ± 18%     -54.0%       3123 ± 30%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
     21096 ± 19%    +273.9%      78881 ±  2%  interrupts.CPU82.RES:Rescheduling_interrupts
      3244 ± 85%     -88.3%     379.83 ± 36%  interrupts.CPU83.CAL:Function_call_interrupts
    186.83 ± 18%     -63.8%      67.67 ± 38%  interrupts.CPU83.IWI:IRQ_work_interrupts
     22935 ±  6%    +244.3%      78970 ± 11%  interrupts.CPU83.RES:Rescheduling_interrupts
      2873 ± 97%     -88.0%     344.50 ± 43%  interrupts.CPU84.CAL:Function_call_interrupts
    189.50 ± 16%     -55.1%      85.17 ± 20%  interrupts.CPU84.IWI:IRQ_work_interrupts
     22009 ±  6%    +260.3%      79292 ± 10%  interrupts.CPU84.RES:Rescheduling_interrupts
      2744 ± 73%     -84.0%     438.67 ± 41%  interrupts.CPU85.CAL:Function_call_interrupts
    198.83 ± 14%     -53.1%      93.17 ± 40%  interrupts.CPU85.IWI:IRQ_work_interrupts
     22470 ±  7%    +264.9%      81997 ± 11%  interrupts.CPU85.RES:Rescheduling_interrupts
      3106 ± 88%     -87.8%     378.67 ± 38%  interrupts.CPU86.CAL:Function_call_interrupts
    198.00 ± 12%     -45.6%     107.67 ± 26%  interrupts.CPU86.IWI:IRQ_work_interrupts
     21961 ±  6%    +243.7%      75483 ±  7%  interrupts.CPU86.RES:Rescheduling_interrupts
      2896 ± 81%     -88.1%     344.17 ± 39%  interrupts.CPU87.CAL:Function_call_interrupts
    175.67 ± 11%     -45.4%      96.00 ± 37%  interrupts.CPU87.IWI:IRQ_work_interrupts
     23199 ±  6%    +227.3%      75922 ± 13%  interrupts.CPU87.RES:Rescheduling_interrupts
      2375 ± 29%     -86.0%     332.00 ± 40%  interrupts.CPU88.CAL:Function_call_interrupts
    184.50 ± 20%     -61.7%      70.67 ± 39%  interrupts.CPU88.IWI:IRQ_work_interrupts
     23301 ±  8%    +241.3%      79534 ± 13%  interrupts.CPU88.RES:Rescheduling_interrupts
      3240 ± 98%     -89.5%     341.83 ± 45%  interrupts.CPU89.CAL:Function_call_interrupts
    180.00 ±  9%     -63.4%      65.83 ± 37%  interrupts.CPU89.IWI:IRQ_work_interrupts
     22757 ±  5%    +246.3%      78811 ±  3%  interrupts.CPU89.RES:Rescheduling_interrupts
      2188 ± 51%     -81.7%     401.00 ± 40%  interrupts.CPU9.CAL:Function_call_interrupts
     27998 ±  5%    +249.9%      97965 ±  8%  interrupts.CPU9.RES:Rescheduling_interrupts
      2449 ± 65%     -84.9%     368.83 ± 42%  interrupts.CPU90.CAL:Function_call_interrupts
    157.00 ± 26%     -52.4%      74.67 ± 21%  interrupts.CPU90.IWI:IRQ_work_interrupts
     21244 ±  8%    +264.3%      77400 ±  4%  interrupts.CPU90.RES:Rescheduling_interrupts
      3782 ±109%     -89.3%     403.33 ± 40%  interrupts.CPU91.CAL:Function_call_interrupts
    174.50 ± 15%     -53.4%      81.33 ± 19%  interrupts.CPU91.IWI:IRQ_work_interrupts
     22532 ± 14%    +249.5%      78751 ±  5%  interrupts.CPU91.RES:Rescheduling_interrupts
      2528 ± 70%     -85.5%     366.83 ± 36%  interrupts.CPU92.CAL:Function_call_interrupts
    180.17 ± 18%     -40.7%     106.83 ± 24%  interrupts.CPU92.IWI:IRQ_work_interrupts
     23823 ± 15%    +198.1%      71009 ±  8%  interrupts.CPU92.RES:Rescheduling_interrupts
      3253 ±116%     -84.2%     512.67 ± 48%  interrupts.CPU93.CAL:Function_call_interrupts
    173.33 ± 17%     -47.3%      91.33 ± 32%  interrupts.CPU93.IWI:IRQ_work_interrupts
     22272 ±  8%    +211.7%      69420 ±  9%  interrupts.CPU93.RES:Rescheduling_interrupts
      2250 ± 60%     -84.7%     343.50 ± 15%  interrupts.CPU94.CAL:Function_call_interrupts
    160.00 ± 16%     -62.6%      59.83 ± 38%  interrupts.CPU94.IWI:IRQ_work_interrupts
     21853 ± 10%    +224.7%      70960 ±  4%  interrupts.CPU94.RES:Rescheduling_interrupts
      1984 ± 75%     -82.7%     342.83 ± 42%  interrupts.CPU95.CAL:Function_call_interrupts
     15688 ± 29%    +281.0%      59770 ±  5%  interrupts.CPU95.RES:Rescheduling_interrupts
     14515           -38.0%       8992        interrupts.IWI:IRQ_work_interrupts
   2517985          +240.2%    8565028 ±  2%  interrupts.RES:Rescheduling_interrupts
     53.22           -30.5       22.69        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     51.71           -29.6       22.06        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.23           -18.6       14.60        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     32.99           -18.5       14.49        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     31.92           -17.9       14.00        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     31.31           -17.6       13.73        perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     27.57           -15.3       12.28        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     22.93           -12.6       10.29        perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     22.55           -12.4       10.14        perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     22.14           -12.2        9.97        perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg
     21.27           -11.7        9.60        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg
     18.48           -11.1        7.38        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb
     18.30           -11.0        7.30        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb
     18.06           -10.9        7.20        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     17.97           -10.8        7.22        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     17.79           -10.6        7.14        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     17.01           -10.1        6.91        perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
     16.70            -9.9        6.79        perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
     16.60            -9.9        6.75        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
     16.11            -9.7        6.43        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.03            -9.6        6.40        perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     15.05            -8.7        6.32        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
     14.75            -8.6        6.20        perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     14.40            -8.3        6.05        perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll
     14.34            -8.3        6.03        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     14.29            -8.3        6.01        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     14.19            -8.2        5.97        perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
     13.54            -7.9        5.68        perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
     13.44            -7.8        5.64        perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     12.20            -7.1        5.14        perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu
     11.60            -7.0        4.55        perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
     11.57            -6.7        4.87        perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv
     11.13            -6.4        4.70        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb
     10.27            -6.0        4.28        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb
      9.82            -5.7        4.10        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb
      9.68            -5.6        4.04        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      9.09            -5.5        3.55        perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom
      7.80            -4.5        3.28        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      7.20            -4.4        2.79        perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg
      7.55            -4.4        3.17        perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      6.73            -4.1        2.60        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
      6.50            -4.0        2.50        perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp
      6.24            -3.7        2.52        perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      3.91            -2.4        1.51        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
      3.99            -2.4        1.59        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      3.90            -2.3        1.57        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      3.78            -2.2        1.54        perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      3.35            -2.1        1.22        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets
      3.14            -1.9        1.20        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle
      2.99            -1.9        1.08        perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      2.79            -1.7        1.12        perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg
      2.82            -1.6        1.18        perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      2.27            -1.4        0.92        perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg
      2.25 ±  3%      -1.4        0.90 ±  3%  perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      2.18            -1.3        0.88        perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb
      1.94 ±  3%      -1.2        0.76 ±  3%  perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg
      2.01            -1.0        0.96 ±  3%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.70 ±  4%      -1.0        0.66 ±  4%  perf-profile.calltrace.cycles-pp.ip_idents_reserve.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg
      1.64            -1.0        0.66        perf-profile.calltrace.cycles-pp._copy_to_iter.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      1.47            -0.9        0.57 ±  2%  perf-profile.calltrace.cycles-pp.__consume_stateless_skb.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      1.34            -0.8        0.54        perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule_idle
      1.41            -0.7        0.69 ±  2%  perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.27 ±  4%      -0.7        0.57        perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      1.23 ±  4%      -0.7        0.54 ±  2%  perf-profile.calltrace.cycles-pp.ip_route_output_key_hash.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto
      2.44 ±  4%      -0.4        2.08 ±  3%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb
      0.00            +0.6        0.56 ±  4%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times
      0.00            +0.6        0.56 ±  2%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +0.6        0.58 ±  3%  perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +0.6        0.59 ±  6%  perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle
      0.00            +0.6        0.60 ±  5%  perf-profile.calltrace.cycles-pp.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
     29.82            +0.6       30.44        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +0.6        0.63 ±  2%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.tick_nohz_idle_exit
      0.00            +0.6        0.64 ±  2%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.tick_nohz_idle_exit.do_idle
      0.00            +0.6        0.64        perf-profile.calltrace.cycles-pp.sched_clock_cpu.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.7        0.66 ±  5%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.7        0.66 ±  2%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.tick_nohz_idle_exit.do_idle.cpu_startup_entry
     29.65            +0.7       30.32        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.7        0.67 ±  2%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.7        0.70 ±  5%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +0.7        0.71 ±  5%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +0.7        0.74 ±  2%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.8        0.75 ±  2%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +0.8        0.76 ±  2%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.0        1.00        perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt
      0.00            +1.0        1.01        perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.19 ±141%      +1.0        1.21 ±  5%  perf-profile.calltrace.cycles-pp.netif_rx_internal.netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
      0.20 ±141%      +1.0        1.22 ±  6%  perf-profile.calltrace.cycles-pp.netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
      0.00            +1.0        1.04 ±  6%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_with_offset.netif_rx_internal.netif_rx.loopback_xmit
      0.00            +1.0        1.04        perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.1        1.07 ±  6%  perf-profile.calltrace.cycles-pp.ktime_get_with_offset.netif_rx_internal.netif_rx.loopback_xmit.dev_hard_start_xmit
      0.00            +1.1        1.07        perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +1.1        1.07        perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.1        1.08        perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter
      0.00            +1.1        1.09        perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu
      0.00            +1.1        1.10        perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt
      0.00            +1.1        1.11        perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.1        1.12 ±  2%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.2        1.16 ±  2%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.3        1.30 ±  4%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.8        1.78 ±  2%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +2.1        2.14        perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +2.1        2.14        perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +4.0        4.05        perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +4.1        4.08        perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.60 ±  2%      +6.5        7.14        perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.67 ±  2%      +6.6        7.23        perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +8.4        8.43        perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_nohz_idle_enter.do_idle.cpu_startup_entry
      0.00            +8.5        8.53        perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_idle_enter.do_idle.cpu_startup_entry.start_secondary
      0.34 ± 70%      +8.7        9.08        perf-profile.calltrace.cycles-pp.tick_nohz_idle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +9.2        9.21        perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state
      0.00            +9.3        9.32        perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter
      0.00            +9.4        9.40        perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter.do_idle
      0.00            +9.7        9.70        perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00           +14.6       14.60        perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_nohz_idle_exit.do_idle.cpu_startup_entry
      0.00           +14.7       14.68        perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_idle_exit
      0.00           +14.7       14.69        perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary
      0.00           +14.8       14.76        perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_idle_exit.do_idle
      0.00           +14.8       14.84        perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_idle_exit.do_idle.cpu_startup_entry
      0.00           +15.1       15.06        perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary
      1.07           +29.7       30.73        perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     43.21           +32.1       75.30        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     43.16           +32.1       75.27        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     42.98           +32.2       75.20        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     43.48           +32.4       75.89        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     53.30           -30.5       22.75        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     51.78           -29.7       22.12        perf-profile.children.cycles-pp.do_syscall_64
     33.28           -18.7       14.62        perf-profile.children.cycles-pp.__x64_sys_sendto
     33.02           -18.5       14.50        perf-profile.children.cycles-pp.__sys_sendto
     31.95           -17.9       14.01        perf-profile.children.cycles-pp.sock_sendmsg
     31.39           -17.6       13.76        perf-profile.children.cycles-pp.udp_sendmsg
     27.74           -15.4       12.38        perf-profile.children.cycles-pp.intel_idle
     22.95           -12.7       10.29        perf-profile.children.cycles-pp.udp_send_skb
     22.57           -12.4       10.15        perf-profile.children.cycles-pp.ip_send_skb
     22.16           -12.2        9.98        perf-profile.children.cycles-pp.ip_output
     21.29           -11.7        9.61        perf-profile.children.cycles-pp.ip_finish_output2
     18.57           -11.2        7.42        perf-profile.children.cycles-pp.__local_bh_enable_ip
     18.33           -11.0        7.31        perf-profile.children.cycles-pp.do_softirq
     18.24           -10.9        7.36        perf-profile.children.cycles-pp.__softirqentry_text_start
     18.02           -10.8        7.25        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     17.82           -10.7        7.16        perf-profile.children.cycles-pp.__sys_recvfrom
     17.05           -10.1        6.93        perf-profile.children.cycles-pp.net_rx_action
     16.72            -9.9        6.79        perf-profile.children.cycles-pp.__napi_poll
     16.63            -9.9        6.76        perf-profile.children.cycles-pp.process_backlog
     16.11            -9.7        6.44        perf-profile.children.cycles-pp.inet_recvmsg
     16.06            -9.7        6.41        perf-profile.children.cycles-pp.udp_recvmsg
     15.07            -8.7        6.33        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     14.77            -8.6        6.21        perf-profile.children.cycles-pp.ip_rcv
     14.25            -8.5        5.77        perf-profile.children.cycles-pp.__schedule
     14.41            -8.4        6.06        perf-profile.children.cycles-pp.ip_local_deliver
     14.36            -8.3        6.04        perf-profile.children.cycles-pp.ip_local_deliver_finish
     14.31            -8.3        6.02        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     14.20            -8.2        5.97        perf-profile.children.cycles-pp.__udp4_lib_rcv
     13.55            -7.9        5.69        perf-profile.children.cycles-pp.udp_unicast_rcv_skb
     13.47            -7.8        5.65        perf-profile.children.cycles-pp.udp_queue_rcv_one_skb
     11.66            -7.1        4.58        perf-profile.children.cycles-pp.__skb_recv_udp
     12.21            -7.1        5.15        perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
     11.58            -6.7        4.88        perf-profile.children.cycles-pp.sock_def_readable
     11.16            -6.4        4.71        perf-profile.children.cycles-pp.__wake_up_common_lock
     10.28            -6.0        4.28        perf-profile.children.cycles-pp.__wake_up_common
      9.84            -5.7        4.11        perf-profile.children.cycles-pp.autoremove_wake_function
      9.74            -5.7        4.07        perf-profile.children.cycles-pp.try_to_wake_up
      9.11            -5.5        3.56        perf-profile.children.cycles-pp.__skb_wait_for_more_packets
      7.87            -4.6        3.32        perf-profile.children.cycles-pp.schedule_idle
      7.22            -4.4        2.80        perf-profile.children.cycles-pp.schedule_timeout
      6.74            -4.1        2.61        perf-profile.children.cycles-pp.schedule
      6.26            -3.7        2.52        perf-profile.children.cycles-pp.ip_make_skb
      4.24            -2.6        1.66        perf-profile.children.cycles-pp.pick_next_task_fair
      4.01            -2.4        1.60        perf-profile.children.cycles-pp.ttwu_do_activate
      3.92            -2.3        1.57        perf-profile.children.cycles-pp.enqueue_task_fair
      3.80            -2.3        1.54        perf-profile.children.cycles-pp.__ip_append_data
      3.38            -2.1        1.23 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      3.21            -2.0        1.24        perf-profile.children.cycles-pp.set_next_entity
      3.06            -1.9        1.11 ±  2%  perf-profile.children.cycles-pp.dequeue_entity
      3.23            -1.9        1.35        perf-profile.children.cycles-pp.update_load_avg
      2.89            -1.9        1.02        perf-profile.children.cycles-pp._raw_spin_lock
      2.91            -1.7        1.22        perf-profile.children.cycles-pp.enqueue_entity
      2.80            -1.7        1.13        perf-profile.children.cycles-pp.sock_alloc_send_pskb
      2.47            -1.5        1.01        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      2.28 ±  3%      -1.4        0.91 ±  3%  perf-profile.children.cycles-pp.__ip_make_skb
      2.28            -1.4        0.92        perf-profile.children.cycles-pp.alloc_skb_with_frags
      2.38            -1.3        1.05        perf-profile.children.cycles-pp.update_rq_clock
      2.22            -1.3        0.89        perf-profile.children.cycles-pp.__alloc_skb
      1.95 ±  4%      -1.2        0.76 ±  3%  perf-profile.children.cycles-pp.__ip_select_ident
      2.06            -1.1        0.98 ±  2%  perf-profile.children.cycles-pp.menu_select
      1.64            -1.1        0.58        perf-profile.children.cycles-pp.update_curr
      1.70 ±  4%      -1.0        0.66 ±  3%  perf-profile.children.cycles-pp.ip_idents_reserve
      1.66            -1.0        0.67        perf-profile.children.cycles-pp._copy_to_iter
      1.47            -0.9        0.57 ±  2%  perf-profile.children.cycles-pp.__consume_stateless_skb
      1.28 ±  4%      -0.7        0.57        perf-profile.children.cycles-pp.ip_route_output_flow
      1.20            -0.7        0.48        perf-profile.children.cycles-pp.copy_user_generic_unrolled
      1.25 ±  4%      -0.7        0.55 ±  2%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      1.09            -0.6        0.44 ±  2%  perf-profile.children.cycles-pp.copyout
      1.08            -0.6        0.43 ±  3%  perf-profile.children.cycles-pp.__switch_to
      1.08            -0.6        0.46 ±  2%  perf-profile.children.cycles-pp.sock_wfree
      0.98            -0.6        0.38 ±  2%  perf-profile.children.cycles-pp.__slab_free
      1.02            -0.6        0.42        perf-profile.children.cycles-pp.select_task_rq_fair
      1.05 ±  5%      -0.6        0.47 ±  2%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      1.07            -0.6        0.49 ±  4%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.92 ±  2%      -0.6        0.37 ±  2%  perf-profile.children.cycles-pp.__ksize
      0.88            -0.5        0.34 ±  2%  perf-profile.children.cycles-pp.prepare_to_wait_exclusive
      0.91            -0.5        0.38 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.89            -0.5        0.38 ±  3%  perf-profile.children.cycles-pp.move_addr_to_user
      0.83            -0.5        0.36 ±  3%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.78            -0.5        0.32 ±  2%  perf-profile.children.cycles-pp.sock_def_write_space
      0.75            -0.5        0.30        perf-profile.children.cycles-pp.__might_fault
      0.78            -0.4        0.33 ±  2%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.73            -0.4        0.29 ±  2%  perf-profile.children.cycles-pp.udp_rmem_release
      0.70            -0.4        0.28        perf-profile.children.cycles-pp.___might_sleep
      0.69            -0.4        0.28 ±  3%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.67            -0.4        0.26 ±  2%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.66 ±  3%      -0.4        0.25 ±  2%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.72            -0.4        0.31 ±  3%  perf-profile.children.cycles-pp.check_preempt_curr
      0.66 ±  2%      -0.4        0.26 ±  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.64 ±  2%      -0.4        0.25        perf-profile.children.cycles-pp.skb_release_data
      2.47 ±  4%      -0.4        2.08 ±  3%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.62            -0.4        0.25        perf-profile.children.cycles-pp.___perf_sw_event
      0.68 ±  8%      -0.4        0.31 ±  4%  perf-profile.children.cycles-pp.fib_table_lookup
      0.95            -0.4        0.58        perf-profile.children.cycles-pp.sched_clock
      0.66            -0.4        0.29 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.68            -0.4        0.31 ±  4%  perf-profile.children.cycles-pp.__check_object_size
      0.60 ±  3%      -0.4        0.23 ±  3%  perf-profile.children.cycles-pp.pick_next_entity
      0.66            -0.4        0.29 ±  3%  perf-profile.children.cycles-pp.ip_generic_getfrag
      0.63 ±  2%      -0.4        0.27        perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
      0.61 ±  2%      -0.4        0.26        perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.57 ±  2%      -0.3        0.22        perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.65 ±  2%      -0.3        0.30 ±  3%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.59            -0.3        0.25        perf-profile.children.cycles-pp.__switch_to_asm
      0.56            -0.3        0.23 ±  2%  perf-profile.children.cycles-pp.resched_curr
      0.57 ±  2%      -0.3        0.24 ±  3%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.54 ±  2%      -0.3        0.22        perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.85            -0.3        0.54        perf-profile.children.cycles-pp.native_sched_clock
      0.53            -0.3        0.23 ±  2%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.50 ±  2%      -0.3        0.21 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.50 ±  3%      -0.3        0.23 ±  4%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.46            -0.3        0.20 ±  3%  perf-profile.children.cycles-pp.__udp4_lib_lookup
      0.44 ±  2%      -0.3        0.18 ±  2%  perf-profile.children.cycles-pp.move_addr_to_kernel
      0.46 ±  2%      -0.3        0.20 ±  4%  perf-profile.children.cycles-pp.ip_finish_output
      0.38            -0.3        0.12 ±  3%  perf-profile.children.cycles-pp.reweight_entity
      0.43 ±  2%      -0.3        0.18 ±  4%  perf-profile.children.cycles-pp.kmem_cache_free
      0.42 ±  4%      -0.3        0.17 ±  3%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.40 ±  2%      -0.3        0.15        perf-profile.children.cycles-pp.available_idle_cpu
      0.42            -0.3        0.17 ±  3%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.42            -0.2        0.17 ±  3%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.41            -0.2        0.17 ±  2%  perf-profile.children.cycles-pp.aa_sk_perm
      0.40 ±  2%      -0.2        0.17 ±  5%  perf-profile.children.cycles-pp._copy_from_iter_full
      0.43 ±  2%      -0.2        0.20 ±  4%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.39            -0.2        0.16 ±  4%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.38 ±  2%      -0.2        0.15 ±  5%  perf-profile.children.cycles-pp.update_cfs_group
      0.39 ±  2%      -0.2        0.17 ±  3%  perf-profile.children.cycles-pp.perf_tp_event
      0.40 ±  4%      -0.2        0.17 ±  4%  perf-profile.children.cycles-pp.__fget_light
      0.37            -0.2        0.15 ±  6%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.36 ±  3%      -0.2        0.15 ±  6%  perf-profile.children.cycles-pp.cpus_share_cache
      0.34            -0.2        0.14 ±  3%  perf-profile.children.cycles-pp.__entry_text_start
      0.34            -0.2        0.14 ±  4%  perf-profile.children.cycles-pp.sock_recvmsg
      0.32 ±  2%      -0.2        0.13 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.32            -0.2        0.14 ±  3%  perf-profile.children.cycles-pp._copy_from_user
      0.30 ±  4%      -0.2        0.12 ±  6%  perf-profile.children.cycles-pp.update_min_vruntime
      0.30            -0.2        0.12 ±  3%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.32 ±  3%      -0.2        0.15 ±  5%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.32 ±  2%      -0.2        0.15 ±  3%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.31            -0.2        0.14 ±  2%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.28 ±  3%      -0.2        0.11 ±  5%  perf-profile.children.cycles-pp.rb_next
      0.30            -0.2        0.13 ±  3%  perf-profile.children.cycles-pp._copy_to_user
      0.29 ±  4%      -0.2        0.12 ±  4%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.28 ±  5%      -0.2        0.12 ±  5%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.26            -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
      0.24 ±  3%      -0.2        0.09 ±  4%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.26 ±  2%      -0.1        0.12 ±  8%  perf-profile.children.cycles-pp.udp4_lib_lookup2
      0.27 ±  4%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.23 ±  3%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.ip_local_out
      0.23 ±  2%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.21 ±  3%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.siphash_3u32
      0.39 ±  2%      -0.1        0.27 ±  2%  perf-profile.children.cycles-pp.finish_task_switch
      0.20 ±  4%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.copyin
      0.20 ±  2%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.__ip_local_out
      0.20 ±  3%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.22 ±  4%      -0.1        0.10 ±  3%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.20 ±  4%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.migrate_disable
      0.18 ±  2%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.place_entity
      0.20 ±  7%      -0.1        0.09 ± 11%  perf-profile.children.cycles-pp.__calc_delta
      0.19 ±  3%      -0.1        0.08 ± 10%  perf-profile.children.cycles-pp.migrate_enable
      0.18 ±  3%      -0.1        0.07 ±  8%  perf-profile.children.cycles-pp.call_cpuidle
      0.17 ±  3%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.18 ±  4%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.__slab_alloc
      0.17 ±  5%      -0.1        0.07 ±  8%  perf-profile.children.cycles-pp.___slab_alloc
      0.17 ±  2%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__put_user_nocheck_4
      0.15 ±  4%      -0.1        0.06 ±  8%  perf-profile.children.cycles-pp.menu_reflect
      0.16 ±  2%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.kfree
      0.17 ± 13%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.fib_lookup_good_nhc
      0.17 ±  3%      -0.1        0.07 ±  8%  perf-profile.children.cycles-pp.newidle_balance
      0.19 ±  3%      -0.1        0.10 ±  3%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.12 ±  4%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.__build_skb_around
      0.12 ±  3%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.15 ±  3%      -0.1        0.06 ±  8%  perf-profile.children.cycles-pp.ip_send_check
      0.11 ±  3%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.15 ±  3%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.ip_rcv_core
      0.15 ±  2%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.dst_release
      0.15 ±  5%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__get_user_4
      0.14 ±  2%      -0.1        0.06        perf-profile.children.cycles-pp.ipv4_pktinfo_prepare
      0.17 ±  2%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.switch_fpu_return
      0.13 ±  8%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.netif_skb_features
      0.13 ±  6%      -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.ip_rcv_finish
      0.13 ±  5%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.13            -0.1        0.06 ±  8%  perf-profile.children.cycles-pp.__ip_finish_output
      0.11 ±  4%      -0.1        0.04 ± 44%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.10 ±  3%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.__list_add_valid
      0.10 ±  4%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.__skb_try_recv_from_queue
      0.12 ±  4%      -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.14 ±  4%      -0.1        0.07 ± 12%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.11 ±  4%      -0.1        0.05 ±  7%  perf-profile.children.cycles-pp.__cond_resched
      0.13 ±  2%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.__check_heap_object
      0.11 ±  3%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.10 ±  6%      -0.1        0.05        perf-profile.children.cycles-pp.rb_erase
      0.12 ±  6%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.09 ±  5%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.import_single_range
      0.11 ±  5%      -0.0        0.09 ±  5%  perf-profile.children.cycles-pp.rebalance_domains
      0.07 ±  5%      -0.0        0.05        perf-profile.children.cycles-pp._raw_spin_trylock
      0.17 ±  4%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      1.07            +0.2        1.23        perf-profile.children.cycles-pp.sched_clock_cpu
      0.17 ±  4%      +0.2        0.38 ± 14%  perf-profile.children.cycles-pp.poll_idle
      0.26 ± 28%      +0.3        0.60 ± 24%  perf-profile.children.cycles-pp.start_kernel
      0.03 ± 70%      +0.6        0.62 ±  2%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
     30.02            +0.7       30.72        perf-profile.children.cycles-pp.cpuidle_enter
     29.95            +0.7       30.70        perf-profile.children.cycles-pp.cpuidle_enter_state
      0.46 ± 22%      +0.8        1.22 ±  6%  perf-profile.children.cycles-pp.netif_rx
      0.43 ± 24%      +0.8        1.22 ±  6%  perf-profile.children.cycles-pp.netif_rx_internal
      0.15 ±  7%      +0.8        0.96 ±  4%  perf-profile.children.cycles-pp.update_process_times
      0.09 ±  7%      +0.8        0.91 ±  5%  perf-profile.children.cycles-pp.scheduler_tick
      0.16 ±  7%      +0.8        0.97 ±  4%  perf-profile.children.cycles-pp.tick_sched_handle
      0.00            +0.9        0.86 ±  2%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.09 ±141%      +1.0        1.07 ±  6%  perf-profile.children.cycles-pp.ktime_get_with_offset
      0.00            +1.3        1.29        perf-profile.children.cycles-pp.sched_clock_local
      0.00            +1.5        1.47        perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.14 ± 12%      +1.5        1.63        perf-profile.children.cycles-pp.clockevents_program_event
      0.18 ±  7%      +1.6        1.80 ±  3%  perf-profile.children.cycles-pp.tick_sched_timer
      0.28 ±  5%      +2.2        2.49 ±  2%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.07 ± 12%      +2.3        2.36        perf-profile.children.cycles-pp.tick_irq_enter
      0.08 ± 14%      +2.3        2.37        perf-profile.children.cycles-pp.irq_enter_rcu
      0.49 ±  3%      +5.2        5.64        perf-profile.children.cycles-pp.hrtimer_interrupt
      0.53 ±  3%      +5.2        5.68        perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.83 ±  2%      +8.3        9.12        perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.95 ±  2%      +8.3        9.25        perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.52            +8.6        9.16        perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.00           +26.2       26.19        perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event
      0.00           +26.5       26.47        perf-profile.children.cycles-pp.sched_clock_tick
      1.09           +29.9       30.96        perf-profile.children.cycles-pp.tick_nohz_idle_exit
     43.21           +32.1       75.30        perf-profile.children.cycles-pp.start_secondary
     43.48           +32.4       75.89        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     43.48           +32.4       75.89        perf-profile.children.cycles-pp.cpu_startup_entry
     43.37           +32.5       75.85        perf-profile.children.cycles-pp.do_idle
      0.84 ±  2%     +53.8       54.62        perf-profile.children.cycles-pp.ktime_get
      0.00           +56.7       56.71        perf-profile.children.cycles-pp.read_hpet
     27.74           -15.4       12.37        perf-profile.self.cycles-pp.intel_idle
      2.82            -1.8        0.99        perf-profile.self.cycles-pp._raw_spin_lock
      1.88            -1.6        0.27 ±  3%  perf-profile.self.cycles-pp.update_rq_clock
      2.78            -1.5        1.26        perf-profile.self.cycles-pp.__schedule
      2.44            -1.4        1.00        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.70 ±  4%      -1.0        0.65 ±  4%  perf-profile.self.cycles-pp.ip_idents_reserve
      1.68            -1.0        0.69        perf-profile.self.cycles-pp.update_load_avg
      1.51            -1.0        0.56        perf-profile.self.cycles-pp.set_next_entity
      1.17            -0.7        0.43 ±  2%  perf-profile.self.cycles-pp.do_idle
      1.15            -0.7        0.46        perf-profile.self.cycles-pp.copy_user_generic_unrolled
      1.01 ±  2%      -0.7        0.35 ±  2%  perf-profile.self.cycles-pp.enqueue_task_fair
      1.00            -0.6        0.40 ±  4%  perf-profile.self.cycles-pp.__switch_to
      0.97            -0.6        0.37 ±  2%  perf-profile.self.cycles-pp.__slab_free
      1.04            -0.6        0.44        perf-profile.self.cycles-pp.enqueue_entity
      0.96 ±  2%      -0.6        0.39 ±  2%  perf-profile.self.cycles-pp.__skb_wait_for_more_packets
      0.91 ±  2%      -0.5        0.36 ±  2%  perf-profile.self.cycles-pp.__ksize
      0.87            -0.5        0.37        perf-profile.self.cycles-pp.__update_load_avg_se
      0.77            -0.5        0.29 ±  2%  perf-profile.self.cycles-pp.update_curr
      0.77            -0.5        0.31 ±  2%  perf-profile.self.cycles-pp.sock_def_write_space
      0.77 ±  2%      -0.5        0.31 ±  3%  perf-profile.self.cycles-pp.udp_sendmsg
      0.72            -0.4        0.28 ±  2%  perf-profile.self.cycles-pp.udp_rmem_release
      0.71 ±  2%      -0.4        0.28        perf-profile.self.cycles-pp.udp_queue_rcv_one_skb
      0.71            -0.4        0.28 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.68 ±  2%      -0.4        0.27 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.66            -0.4        0.26        perf-profile.self.cycles-pp.nr_iowait_cpu
      0.66            -0.4        0.26        perf-profile.self.cycles-pp.___might_sleep
      0.66 ±  3%      -0.4        0.25 ±  2%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.66            -0.4        0.26 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.64 ±  2%      -0.4        0.24 ±  2%  perf-profile.self.cycles-pp.skb_release_data
      0.76 ±  2%      -0.4        0.37 ±  2%  perf-profile.self.cycles-pp.menu_select
      0.57 ±  3%      -0.4        0.21 ±  3%  perf-profile.self.cycles-pp.pick_next_entity
      0.62            -0.4        0.26 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.56 ±  2%      -0.3        0.21 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.56            -0.3        0.22 ±  2%  perf-profile.self.cycles-pp.resched_curr
      0.57            -0.3        0.24        perf-profile.self.cycles-pp.__switch_to_asm
      0.54            -0.3        0.21 ±  2%  perf-profile.self.cycles-pp.___perf_sw_event
      0.52 ±  2%      -0.3        0.21 ±  4%  perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb
      0.47            -0.3        0.16 ±  2%  perf-profile.self.cycles-pp.process_backlog
      0.82            -0.3        0.52 ±  2%  perf-profile.self.cycles-pp.native_sched_clock
      0.48 ±  2%      -0.3        0.19 ±  3%  perf-profile.self.cycles-pp.schedule_timeout
      0.52 ± 10%      -0.3        0.23 ±  7%  perf-profile.self.cycles-pp.fib_table_lookup
      0.47 ±  2%      -0.3        0.19 ±  5%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.38 ±  2%      -0.3        0.13 ±  6%  perf-profile.self.cycles-pp.dequeue_entity
      0.41            -0.2        0.16 ±  4%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.41 ±  2%      -0.2        0.16 ±  2%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.41 ±  3%      -0.2        0.17 ±  3%  perf-profile.self.cycles-pp.sock_def_readable
      0.36 ±  2%      -0.2        0.12 ±  4%  perf-profile.self.cycles-pp.reweight_entity
      0.41            -0.2        0.16 ±  2%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.39            -0.2        0.14 ±  3%  perf-profile.self.cycles-pp.available_idle_cpu
      0.40 ±  3%      -0.2        0.16 ±  4%  perf-profile.self.cycles-pp.__skb_recv_udp
      0.39            -0.2        0.15 ±  3%  perf-profile.self.cycles-pp.__wake_up_common
      0.47 ±  4%      -0.2        0.25 ±  2%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.46            -0.2        0.24 ±  2%  perf-profile.self.cycles-pp.udp_recvmsg
      0.38            -0.2        0.16 ±  4%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.38 ±  4%      -0.2        0.16 ±  5%  perf-profile.self.cycles-pp.__fget_light
      0.35 ±  3%      -0.2        0.14 ±  7%  perf-profile.self.cycles-pp.update_cfs_group
      0.35 ±  2%      -0.2        0.15 ±  6%  perf-profile.self.cycles-pp.cpus_share_cache
      0.32            -0.2        0.11 ±  4%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.34            -0.2        0.14 ±  3%  perf-profile.self.cycles-pp.__entry_text_start
      0.35 ±  2%      -0.2        0.15 ±  3%  perf-profile.self.cycles-pp.__check_object_size
      0.32 ±  3%      -0.2        0.12 ±  3%  perf-profile.self.cycles-pp.udp_send_skb
      0.32 ±  3%      -0.2        0.12        perf-profile.self.cycles-pp.__ip_append_data
      0.34 ±  2%      -0.2        0.15 ±  3%  perf-profile.self.cycles-pp.__sys_sendto
      0.48            -0.2        0.29 ±  2%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.33            -0.2        0.14 ±  2%  perf-profile.self.cycles-pp.finish_task_switch
      0.33 ±  8%      -0.2        0.14 ±  5%  perf-profile.self.cycles-pp.ip_finish_output2
      0.33 ±  2%      -0.2        0.15 ±  6%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      0.30 ±  3%      -0.2        0.12 ±  4%  perf-profile.self.cycles-pp.perf_tp_event
      0.29            -0.2        0.11 ±  4%  perf-profile.self.cycles-pp.net_rx_action
      0.31 ±  2%      -0.2        0.13 ±  5%  perf-profile.self.cycles-pp.do_syscall_64
      0.30 ±  3%      -0.2        0.13 ±  5%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.28 ±  4%      -0.2        0.11 ±  6%  perf-profile.self.cycles-pp.update_min_vruntime
      0.27 ±  3%      -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.rb_next
      0.27 ±  3%      -0.2        0.11 ±  4%  perf-profile.self.cycles-pp.__might_sleep
      0.29 ±  3%      -0.2        0.13 ±  3%  perf-profile.self.cycles-pp.sock_wfree
      0.26            -0.2        0.10 ±  3%  perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
      0.26            -0.2        0.11 ±  5%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.25 ±  2%      -0.2        0.10 ±  6%  perf-profile.self.cycles-pp.aa_sk_perm
      0.24 ±  6%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.24 ±  4%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.24            -0.1        0.09        perf-profile.self.cycles-pp.ip_output
      0.26 ±  2%      -0.1        0.11 ±  3%  perf-profile.self.cycles-pp.__ip_make_skb
      0.24 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__alloc_skb
      0.26            -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.25 ±  3%      -0.1        0.11 ±  8%  perf-profile.self.cycles-pp.udp4_lib_lookup2
      0.23 ±  3%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.23 ±  9%      -0.1        0.10 ±  6%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.21            -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.siphash_3u32
      0.21            -0.1        0.09 ±  5%  perf-profile.self.cycles-pp.__udp4_lib_lookup
      0.22 ±  3%      -0.1        0.10 ±  3%  perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
      0.21 ±  3%      -0.1        0.09        perf-profile.self.cycles-pp.sock_sendmsg
      0.21 ±  9%      -0.1        0.09 ±  7%  perf-profile.self.cycles-pp.do_softirq
      0.20 ±  3%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.20 ±  2%      -0.1        0.08 ±  4%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.19 ±  3%      -0.1        0.08        perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.19 ±  8%      -0.1        0.08 ±  9%  perf-profile.self.cycles-pp.__calc_delta
      0.19 ±  4%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.migrate_disable
      0.19 ±  3%      -0.1        0.08 ±  4%  perf-profile.self.cycles-pp.ip_route_output_key_hash
      0.18 ±  3%      -0.1        0.07 ±  5%  perf-profile.self.cycles-pp.prepare_to_wait_exclusive
      0.21 ±  2%      -0.1        0.10 ±  3%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.18 ±  4%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.migrate_enable
      0.19 ±  3%      -0.1        0.08 ±  7%  perf-profile.self.cycles-pp.__udp4_lib_rcv
      0.18 ±  2%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.call_cpuidle
      0.17 ±  3%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.place_entity
      0.19 ±  3%      -0.1        0.09 ±  5%  perf-profile.self.cycles-pp.schedule_idle
      0.17 ±  2%      -0.1        0.07        perf-profile.self.cycles-pp.schedule
      0.17 ±  5%      -0.1        0.07 ±  8%  perf-profile.self.cycles-pp.___slab_alloc
      0.17 ±  3%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__put_user_nocheck_4
      0.16 ±  4%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.cpu_startup_entry
      0.17 ±  2%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.ip_send_skb
      0.15 ±  3%      -0.1        0.06        perf-profile.self.cycles-pp.move_addr_to_user
      0.15 ±  3%      -0.1        0.06        perf-profile.self.cycles-pp.kfree
      0.16 ± 15%      -0.1        0.07 ± 11%  perf-profile.self.cycles-pp.fib_lookup_good_nhc
      0.15 ±  8%      -0.1        0.06        perf-profile.self.cycles-pp.loopback_xmit
      0.14 ±  4%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.newidle_balance
      0.14 ±  2%      -0.1        0.05 ±  9%  perf-profile.self.cycles-pp.ip_send_check
      0.14 ±  4%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.ip_rcv_core
      0.16 ±  2%      -0.1        0.08 ±  4%  perf-profile.self.cycles-pp.switch_fpu_return
      0.15 ±  7%      -0.1        0.07 ±  5%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.10 ±  4%      -0.1        0.03 ± 99%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.14 ±  5%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.__get_user_4
      0.11 ±  3%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.15 ±  4%      -0.1        0.08 ±  9%  perf-profile.self.cycles-pp.check_preempt_curr
      0.14 ±  5%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.13 ±  5%      -0.1        0.05        perf-profile.self.cycles-pp._copy_to_iter
      0.14 ±  5%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.kmem_cache_free
      0.13 ±  2%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.dst_release
      0.11 ±  4%      -0.1        0.04 ± 44%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.11 ±  4%      -0.1        0.04 ± 45%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.12 ±  6%      -0.1        0.05        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.11 ±  4%      -0.1        0.05        perf-profile.self.cycles-pp.rcu_eqs_exit
      0.12 ±  5%      -0.1        0.06 ± 17%  perf-profile.self.cycles-pp.poll_idle
      0.10 ±  4%      -0.1        0.04 ± 44%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.11 ±  3%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__check_heap_object
      0.10 ±  3%      -0.1        0.05 ±  7%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.10 ±  7%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.08            -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.sockfd_lookup_light
      0.08 ±  4%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.import_single_range
      0.07 ±  5%      -0.0        0.05        perf-profile.self.cycles-pp._raw_spin_trylock
      0.09 ±  4%      -0.0        0.07        perf-profile.self.cycles-pp.tick_nohz_idle_exit
      0.15 ±  5%      +0.3        0.45        perf-profile.self.cycles-pp.sched_clock_cpu
      0.00            +0.9        0.87        perf-profile.self.cycles-pp.sched_clock_local
      0.00           +56.7       56.70        perf-profile.self.cycles-pp.read_hpet


                                                                                
                               netperf.Throughput_tps                           
                                                                                
  120000 +------------------------------------------------------------------+   
         |  .+                                                              |   
  110000 |.+  +.+.+.+.++.     .+.+   .+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.+    |   
  100000 |-+             +.+.+    +.+                                       |   
         |                                                                  |   
   90000 |-+                                                                |   
   80000 |-+                                                                |   
         |                                                                  |   
   70000 |-+                                                                |   
   60000 |-+                                                                |   
         |                                                                  |   
   50000 |-+                                                                |   
   40000 |-+                                                                |   
         | O OO O O O OO O O O O OO O O O OO O O O OO O O O O OO O O O OO O |   
   30000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             netperf.Throughput_total_tps                       
                                                                                
  2.8e+06 +-----------------------------------------------------------------+   
  2.6e+06 |-+.+                                                             |   
          |.+  +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+.+.++.+.+.+.++.+.+.+    |   
  2.4e+06 |-+                                                               |   
  2.2e+06 |-+                                                               |   
          |                                                                 |   
    2e+06 |-+                                                               |   
  1.8e+06 |-+                                                               |   
  1.6e+06 |-+                                                               |   
          |                                                                 |   
  1.4e+06 |-+                                                               |   
  1.2e+06 |-+                                                               |   
          |                                                                 |   
    1e+06 |-O OO O O OO O   O OO O O O OO O O OO O O O OO O O O OO O O OO O |   
   800000 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                  netperf.workload                              
                                                                                
  8e+08 +-------------------------------------------------------------------+   
        |.+   ++.+.+.+.+.++.+.+.+.+.++.+.+.+.+.++.+.+.+.+.++.+.+.+.+.++     |   
  7e+08 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  6e+08 |-+                                                                 |   
        |                                                                   |   
  5e+08 |-+                                                                 |   
        |                                                                   |   
  4e+08 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  3e+08 |-O O OO O O O O  O O O O O OO O O O O OO O O O O OO O O O O OO O O |   
        |                O                                                  |   
  2e+08 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            pmeter.Average_Active_Power                         
                                                                                
  660 +---------------------------------------------------------------------+   
      |.                                                            +.      |   
  640 |-+.+.+.++. .+.   .+.      .+. .+.+.+.+.+.+.+ .+. .+.+.+.+.+. : +     |   
      |          +   +.+   +.+.++   +              +   +           +        |   
  620 |-+                                                                   |   
      |                                                                     |   
  600 |-+                                                                   |   
      |                                                                     |   
  580 |-+                                                                   |   
      |                                                                     |   
  560 |-+                                         O          O              |   
      | O O O OO       O       O    O   O O O O O  O O O O O   O O OO O O O |   
  540 |-+        O O O   O O O  O O   O                                     |   
      |                                                                     |   
  520 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              netperf.time.user_time                            
                                                                                
  250 +---------------------------------------------------------------------+   
      |        +.   .+.+.   .+.++.+.+                                       |   
  240 |.+   +. : +.+     +.+         :                .+.+.+.   .+.         |   
      |  + +  +                      :            ++.+       +.+   ++.+     |   
      |   +                           +.+. .+.+. +                          |   
  230 |-+ O                               +     +                           |   
      |                                                                     |   
  220 |-+                                                                   |   
      |                                                                     |   
  210 |-+                           O       O O O  O O O O O   O O O    O O |   
      |                        O  O       O       O          O      O O     |   
      |     O OO O O O     O O  O     O O                                   |   
  200 |-O                                                                   |   
      |                O O                                                  |   
  190 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             netperf.time.system_time                           
                                                                                
  3700 +--------------------------------------------------------------------+   
       |              O     OO O O O O O OO O O   O O OO O   O O O O  O O O |   
  3600 |-+   O    O O                           O          O         O      |   
       |   O  O O         O                                                 |   
  3500 |-O              O                                                   |   
       |                                                                    |   
  3400 |-+                                                                  |   
       |                                                                    |   
  3300 |-+                                                                  |   
       |                                                                    |   
  3200 |-+                                                                  |   
       |.     +.+.+.+.                .+.++.+.+.+.                          |   
  3100 |-+. .+        +.+.+.++.+.+.+.+            +.+.++.+.+.+.+.+.+.++     |   
       |   +                                                                |   
  3000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                     netperf.time.percent_of_cpu_this_job_got                   
                                                                                
  1300 +--------------------------------------------------------------------+   
       |              O     OO O O O O O OO O O O O O OO O O O O O O  O O O |   
       |   O O    O O                                                O      |   
  1250 |-+    O O                                                           |   
       | O              O O                                                 |   
       |                                                                    |   
  1200 |-+                                                                  |   
       |                                                                    |   
  1150 |-+                                                                  |   
       |                                                                    |   
       |.+   ++.+.+.+.+.+.+.++.+.+.+.+.+.++.+.+.+.+.+.++.+.+.+. .+.+.++     |   
  1100 |-+: :                                                  +            |   
       |  : :                                                               |   
       |   +                                                                |   
  1050 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       netperf.time.voluntary_context_switches                  
                                                                                
  8e+08 +-------------------------------------------------------------------+   
        |.+.+.++.+.+.+.+.++.+.+.+.+.++.+.+.+.+.++.+.+.+.+.++.+.+.+.+.++     |   
  7e+08 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  6e+08 |-+                                                                 |   
        |                                                                   |   
  5e+08 |-+                                                                 |   
        |                                                                   |   
  4e+08 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  3e+08 |-O O OO O O O O    O O O O OO O O O O OO O O O O OO O O O O OO O O |   
        |                OO                                                 |   
  2e+08 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      netperf.time.involuntary_context_switches                 
                                                                                
  11000 +-------------------------------------------------------------------+   
        |                                                                   |   
  10000 |.+     .+.+.+         .+.+.++. .+.   .+                            |   
        | +. .++      + .++.+.+        +   +.+  +.+.+.+.+.++.+.+.+.+.++     |   
   9000 |-+ +          +                                                    |   
        |                                                                   |   
   8000 |-+                                                                 |   
        |                                                                   |   
   7000 |-+                                                                 |   
        |                                                                   |   
   6000 |-O   OO O                                                          |   
        |   O      O O O        O   OO   O O          O          O          |   
   5000 |-+              OO O O   O    O     O OO O O   O OO O O   O OO O O |   
        |                                                                   |   
   4000 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.12.0-rc3-00029-g388450c7081d" of type "text/plain" (172872 bytes)

View attachment "job-script" of type "text/plain" (8567 bytes)

View attachment "job.yaml" of type "text/plain" (5659 bytes)

View attachment "reproduce" of type "text/plain" (1474 bytes)

Download attachment "dmesg.xz" of type "application/x-xz" (30108 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ