lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20200616075832.GN5653@shao2-debian>
Date:   Tue, 16 Jun 2020 15:58:32 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Xiyu Yang <xiyuyang19@...an.edu.cn>
Cc:     Vlad Yasevich <vyasevich@...il.com>,
        Neil Horman <nhorman@...driver.com>,
        Marcelo Ricardo Leitner <marcelo.leitner@...il.com>,
        "David S. Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>, linux-sctp@...r.kernel.org,
        netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
        yuanxzhang@...an.edu.cn, kjlu@....edu,
        Xiyu Yang <xiyuyang19@...an.edu.cn>,
        Xin Tan <tanxin.ctf@...il.com>, lkp@...ts.01.org
Subject: [sctp] cf8d0e3e58: netperf.Throughput_tps -100.0% regression

Greeting,

FYI, we noticed a -100.0% regression of netperf.Throughput_tps due to commit:


commit: cf8d0e3e583cba8e7eb935f9b97ac94d84fccc2d ("[PATCH] sctp: Fix sk_buff leak when receiving a datagram")
url: https://github.com/0day-ci/linux/commits/Xiyu-Yang/sctp-Fix-sk_buff-leak-when-receiving-a-datagram/20200613-204024


in testcase: netperf
on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 25%
	cluster: cs-localhost
	test: SCTP_RR
	cpufreq_governor: performance
	ucode: 0xd6

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-7.6/25%/debian-x86_64-20191114.cgz/300s/lkp-cfl-e1/SCTP_RR/netperf/0xd6

commit: 
  df2fbf5bfa (" - Add the hwmon support on the i.MX SC (Anson Huang)")
  cf8d0e3e58 ("sctp: Fix sk_buff leak when receiving a datagram")

df2fbf5bfa0e7fff cf8d0e3e583cba8e7eb935f9b97 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4            4%           0:4     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    408957          -100.0%      12.64        netperf.Throughput_total_tps
    102239          -100.0%       3.16        netperf.Throughput_tps
      2905 ±  3%     -99.9%       2.50 ± 20%  netperf.time.involuntary_context_switches
    272.25          -100.0%       0.00        netperf.time.percent_of_cpu_this_job_got
    804.17          -100.0%       0.05 ±  9%  netperf.time.system_time
     28.03 ±  3%    -100.0%       0.00        netperf.time.user_time
 1.227e+08          -100.0%       3834        netperf.time.voluntary_context_switches
 1.227e+08          -100.0%       3792        netperf.workload
     46752           +19.4%      55800        meminfo.SUnreclaim
     27616           -36.8%      17444        meminfo.Shmem
     93045            +9.6%     101966        meminfo.Slab
     67.54           +32.2       99.78        mpstat.cpu.all.idle%
     12.25           -12.3        0.00 ± 82%  mpstat.cpu.all.soft%
     18.92           -18.9        0.05 ±  4%  mpstat.cpu.all.sys%
      1.29 ±  3%      -1.1        0.17 ±  3%  mpstat.cpu.all.usr%
     67.00           +47.8%      99.00        vmstat.cpu.id
      5.00          -100.0%       0.00        vmstat.procs.r
   1594034           -99.9%     824.00 ± 18%  vmstat.system.cs
     32889           -25.1%      24628 ± 29%  vmstat.system.in
 1.618e+08 ±  2%     -96.3%    6037417 ±107%  cpuidle.C1.time
  43353175 ±  3%     -99.9%      27615 ± 70%  cpuidle.C1.usage
 2.072e+09 ± 12%     +89.1%  3.918e+09 ± 22%  cpuidle.C6.time
   3022784 ± 36%     +65.6%    5006863 ±  2%  cpuidle.C6.usage
 4.839e+08          -100.0%      83712 ± 10%  cpuidle.POLL.time
 2.021e+08 ±  2%    -100.0%      43151 ±  3%  cpuidle.POLL.usage
     67099            -3.9%      64479        proc-vmstat.nr_active_anon
      6905           -36.9%       4360        proc-vmstat.nr_shmem
     11687           +19.4%      13949        proc-vmstat.nr_slab_unreclaimable
     67099            -3.9%      64479        proc-vmstat.nr_zone_active_anon
      5050           -95.5%     229.75 ± 36%  proc-vmstat.pgactivate
    411650            -1.3%     406257        proc-vmstat.pgfault
      2447 ±  4%    +301.6%       9828        slabinfo.kmalloc-1k.active_objs
     76.00 ±  4%    +303.3%     306.50        slabinfo.kmalloc-1k.active_slabs
      2447 ±  4%    +301.6%       9828        slabinfo.kmalloc-1k.num_objs
     76.00 ±  4%    +303.3%     306.50        slabinfo.kmalloc-1k.num_slabs
      1376 ± 20%    +500.6%       8264        slabinfo.skbuff_head_cache.active_objs
     43.50 ± 19%    +492.0%     257.50        slabinfo.skbuff_head_cache.active_slabs
      1392 ± 19%    +493.7%       8264        slabinfo.skbuff_head_cache.num_objs
     43.50 ± 19%    +492.0%     257.50        slabinfo.skbuff_head_cache.num_slabs
      3927 ±  2%     +30.4%       5121 ±  5%  slabinfo.vmap_area.active_objs
      3927 ±  2%     +30.4%       5121 ±  5%  slabinfo.vmap_area.num_objs
   6406342 ±  6%    -100.0%     266.00 ± 88%  softirqs.CPU0.NET_RX
     60072 ±  3%     -36.1%      38377 ±  5%  softirqs.CPU0.SCHED
   6564332 ±  5%    -100.0%       2160 ±141%  softirqs.CPU1.NET_RX
     60737 ±  2%     -37.1%      38197 ± 10%  softirqs.CPU1.SCHED
   9008528 ±  3%    -100.0%     234.25 ± 99%  softirqs.CPU10.NET_RX
     66252           -49.5%      33448 ± 46%  softirqs.CPU10.SCHED
   9283861 ±  8%    -100.0%     279.75 ± 90%  softirqs.CPU11.NET_RX
     66582 ±  2%     -47.5%      34958 ± 23%  softirqs.CPU11.SCHED
   8963322          -100.0%     256.75 ± 97%  softirqs.CPU12.NET_RX
     66032           -38.8%      40408 ±  6%  softirqs.CPU12.SCHED
   8746032 ±  4%    -100.0%     132.00 ±143%  softirqs.CPU13.NET_RX
     65043 ±  2%     -38.1%      40247 ±  6%  softirqs.CPU13.SCHED
   8873173 ±  5%    -100.0%     374.00 ± 54%  softirqs.CPU14.NET_RX
     65818 ±  2%     -37.1%      41404 ±  4%  softirqs.CPU14.SCHED
   8259641 ±  6%    -100.0%     142.50 ±147%  softirqs.CPU15.NET_RX
     64447           -43.7%      36289 ± 20%  softirqs.CPU15.SCHED
   6493869 ±  3%     -99.8%      12396 ±135%  softirqs.CPU2.NET_RX
     60111           -48.0%      31266 ± 26%  softirqs.CPU2.SCHED
   6125878 ± 14%    -100.0%     393.00 ± 62%  softirqs.CPU3.NET_RX
     58270 ±  3%     -44.0%      32613 ± 29%  softirqs.CPU3.SCHED
   6393208 ±  8%    -100.0%       2507 ±121%  softirqs.CPU4.NET_RX
     59705 ±  2%     -32.1%      40528 ±  4%  softirqs.CPU4.SCHED
   6754260 ±  4%    -100.0%     275.50 ± 83%  softirqs.CPU5.NET_RX
     60095           -41.2%      35324 ± 21%  softirqs.CPU5.SCHED
   6392475 ± 10%    -100.0%     365.00 ± 56%  softirqs.CPU6.NET_RX
     60106 ±  2%     -31.4%      41239 ±  5%  softirqs.CPU6.SCHED
   6753541 ± 11%    -100.0%     369.75 ± 57%  softirqs.CPU7.NET_RX
     60424 ±  3%     -46.8%      32126 ± 19%  softirqs.CPU7.SCHED
    122824 ±  2%     +25.1%     153616 ±  3%  softirqs.CPU7.TIMER
   8843455          -100.0%     134.25 ±157%  softirqs.CPU8.NET_RX
     66140           -53.9%      30474 ± 22%  softirqs.CPU8.SCHED
   8869113 ±  2%    -100.0%     372.00 ± 55%  softirqs.CPU9.NET_RX
     66031           -43.5%      37287 ±  8%  softirqs.CPU9.SCHED
 1.227e+08          -100.0%      20661 ± 70%  softirqs.NET_RX
   1005873           -41.9%     584194 ±  5%  softirqs.SCHED
     10637 ± 63%     -99.7%      37.09 ±100%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    139927 ± 56%     -99.6%     541.70 ±102%  sched_debug.cfs_rq:/.MIN_vruntime.max
     36202 ± 58%     -99.6%     136.59 ±101%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     74727 ±  7%     +52.0%     113556 ± 14%  sched_debug.cfs_rq:/.load.avg
    177162 ±  4%    +393.4%     874165 ±  5%  sched_debug.cfs_rq:/.load.max
     74477 ±  2%    +247.4%     258738 ±  6%  sched_debug.cfs_rq:/.load.stddev
    426.88 ± 14%    +138.2%       1017 ±  3%  sched_debug.cfs_rq:/.load_avg.max
    126.41 ±  8%    +132.0%     293.30 ±  6%  sched_debug.cfs_rq:/.load_avg.stddev
     10637 ± 63%     -99.7%      37.09 ±100%  sched_debug.cfs_rq:/.max_vruntime.avg
    139927 ± 56%     -99.6%     541.70 ±102%  sched_debug.cfs_rq:/.max_vruntime.max
     36202 ± 58%     -99.6%     136.59 ±101%  sched_debug.cfs_rq:/.max_vruntime.stddev
    268465 ± 10%     -97.7%       6248 ±  7%  sched_debug.cfs_rq:/.min_vruntime.avg
    334392 ± 10%     -95.1%      16374 ± 22%  sched_debug.cfs_rq:/.min_vruntime.max
    193521 ±  9%     -98.7%       2585 ± 16%  sched_debug.cfs_rq:/.min_vruntime.min
     41824 ±  8%     -90.7%       3906 ± 25%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.52 ±  5%     -53.4%       0.24 ± 19%  sched_debug.cfs_rq:/.nr_running.avg
      0.49           -21.7%       0.38 ±  2%  sched_debug.cfs_rq:/.nr_running.stddev
    986.92 ±  6%     -32.9%     662.09 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
     37277 ± 35%    -102.1%    -791.51        sched_debug.cfs_rq:/.spread0.avg
    103203 ±  7%     -91.0%       9334 ± 33%  sched_debug.cfs_rq:/.spread0.max
    -37666           -88.2%      -4454        sched_debug.cfs_rq:/.spread0.min
     41824 ±  8%     -90.7%       3907 ± 25%  sched_debug.cfs_rq:/.spread0.stddev
    456.69           -61.0%     178.00 ±  7%  sched_debug.cfs_rq:/.util_avg.avg
    800.92 ±  3%     -11.0%     712.54 ±  2%  sched_debug.cfs_rq:/.util_avg.max
    314.31 ±  3%     -39.9%     188.93 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    211.22 ±  8%     -88.7%      23.85 ± 34%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    673.52           -69.6%     204.92 ± 20%  sched_debug.cfs_rq:/.util_est_enqueued.max
    283.03           -80.3%      55.88 ± 22%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    278538 ±  7%    +197.1%     827499        sched_debug.cpu.avg_idle.avg
      6971 ± 40%   +8560.0%     603748 ± 17%  sched_debug.cpu.avg_idle.min
    382038           -73.9%      99753 ± 24%  sched_debug.cpu.avg_idle.stddev
    655.75 ±  8%     -48.2%     339.80 ± 12%  sched_debug.cpu.curr->pid.avg
      0.48 ±  9%     -54.5%       0.22 ± 22%  sched_debug.cpu.nr_running.avg
      0.50 ±  3%     -27.0%       0.36        sched_debug.cpu.nr_running.stddev
  13759930 ± 12%     -99.9%      11334 ± 11%  sched_debug.cpu.nr_switches.avg
  17527844 ± 13%     -99.8%      35376 ±  9%  sched_debug.cpu.nr_switches.max
   9687731 ±  9%    -100.0%       3502 ± 25%  sched_debug.cpu.nr_switches.min
   2306115 ±  9%     -99.6%       8405 ±  5%  sched_debug.cpu.nr_switches.stddev
     85.50 ±  3%    +205.3%     261.07        perf-stat.i.MPKI
 4.347e+09           -98.6%   59820289 ±  6%  perf-stat.i.branch-instructions
      1.86            +9.3       11.13 ±  8%  perf-stat.i.branch-miss-rate%
  75998371 ±  2%     -93.1%    5268886 ±  4%  perf-stat.i.branch-misses
      0.17 ±  5%      +2.0        2.14 ±  2%  perf-stat.i.cache-miss-rate%
    872353 ±  8%     +11.6%     973170 ± 11%  perf-stat.i.cache-misses
 1.803e+09 ±  2%     -97.9%   37399419 ± 15%  perf-stat.i.cache-references
   1609007           -99.9%     809.20 ± 19%  perf-stat.i.context-switches
      1.54          +262.5%       5.56 ±  2%  perf-stat.i.cpi
 3.219e+10           -97.2%   9.12e+08 ± 15%  perf-stat.i.cpu-cycles
     68.67           -75.5%      16.80        perf-stat.i.cpu-migrations
     61297 ± 12%     -98.0%       1226        perf-stat.i.cycles-between-cache-misses
      0.02 ±  7%      +1.3        1.32 ±  5%  perf-stat.i.dTLB-load-miss-rate%
  6.12e+09           -99.0%   62388437 ±  7%  perf-stat.i.dTLB-loads
      0.00 ± 15%      +0.3        0.32 ± 22%  perf-stat.i.dTLB-store-miss-rate%
 3.682e+09           -99.2%   29058708 ±  9%  perf-stat.i.dTLB-stores
  55056423 ±  2%     -99.6%     223398 ± 29%  perf-stat.i.iTLB-load-misses
   8777246 ±  3%     -99.7%      23051 ± 20%  perf-stat.i.iTLB-loads
 2.153e+10           -98.7%  2.862e+08 ±  6%  perf-stat.i.instructions
    460.84          +126.3%       1042 ± 10%  perf-stat.i.instructions-per-iTLB-miss
      0.67           -70.5%       0.20 ±  3%  perf-stat.i.ipc
      2.01           -97.2%       0.06 ± 15%  perf-stat.i.metric.GHz
    997.50           -98.8%      11.80 ±  9%  perf-stat.i.metric.M/sec
     12121 ±  6%     -13.9%      10438 ±  5%  perf-stat.i.node-loads
     83.80 ±  3%     +54.7%     129.68 ±  8%  perf-stat.overall.MPKI
      1.75            +7.1        8.82 ±  2%  perf-stat.overall.branch-miss-rate%
      0.05 ±  6%      +2.6        2.62 ±  5%  perf-stat.overall.cache-miss-rate%
      1.50          +111.5%       3.16 ±  9%  perf-stat.overall.cpi
     37132 ±  8%     -97.5%     932.06 ±  4%  perf-stat.overall.cycles-between-cache-misses
      0.01 ±  7%      +0.8        0.76 ± 11%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  9%      +0.2        0.22 ± 23%  perf-stat.overall.dTLB-store-miss-rate%
    391.23 ±  2%    +251.6%       1375 ± 22%  perf-stat.overall.instructions-per-iTLB-miss
      0.67           -52.3%       0.32 ±  9%  perf-stat.overall.ipc
     53544        +42950.0%   23050893 ±  6%  perf-stat.overall.path-length
 4.333e+09           -98.6%   59732810 ±  6%  perf-stat.ps.branch-instructions
  75750698 ±  2%     -93.1%    5259021 ±  4%  perf-stat.ps.branch-misses
    869952 ±  8%     +11.6%     971046 ± 11%  perf-stat.ps.cache-misses
 1.797e+09 ±  2%     -97.9%   37279828 ± 15%  perf-stat.ps.cache-references
   1603685           -99.9%     806.52 ± 19%  perf-stat.ps.context-switches
 3.209e+10           -97.2%  9.095e+08 ± 15%  perf-stat.ps.cpu-cycles
     68.45           -75.5%      16.75        perf-stat.ps.cpu-migrations
   6.1e+09           -99.0%   62279544 ±  7%  perf-stat.ps.dTLB-loads
  3.67e+09           -99.2%   28995701 ±  9%  perf-stat.ps.dTLB-stores
  54874328 ±  2%     -99.6%     222673 ± 29%  perf-stat.ps.iTLB-load-misses
   8748251 ±  3%     -99.7%      22981 ± 20%  perf-stat.ps.iTLB-loads
 2.146e+10           -98.7%  2.858e+08 ±  6%  perf-stat.ps.instructions
     12091 ±  6%     -13.8%      10422 ±  5%  perf-stat.ps.node-loads
 6.569e+12           -98.7%  8.741e+10 ±  6%  perf-stat.total.instructions
     15868 ±  8%     -63.4%       5801 ± 15%  interrupts.CAL:Function_call_interrupts
      6098 ± 12%     -98.1%     117.75 ± 34%  interrupts.CPU0.NMI:Non-maskable_interrupts
      6098 ± 12%     -98.1%     117.75 ± 34%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      2330 ± 11%     -96.2%      88.00 ±117%  interrupts.CPU0.RES:Rescheduling_interrupts
      1239 ± 33%     -85.9%     174.75 ± 42%  interrupts.CPU1.CAL:Function_call_interrupts
      6065 ± 26%     -97.7%     136.50 ± 39%  interrupts.CPU1.NMI:Non-maskable_interrupts
      6065 ± 26%     -97.7%     136.50 ± 39%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
      4163 ± 74%     -98.1%      81.00 ±100%  interrupts.CPU1.RES:Rescheduling_interrupts
    884.25 ± 20%     -70.7%     258.75 ± 62%  interrupts.CPU10.CAL:Function_call_interrupts
      7369 ± 32%     -97.9%     153.50 ± 40%  interrupts.CPU10.NMI:Non-maskable_interrupts
      7369 ± 32%     -97.9%     153.50 ± 40%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      2269 ± 18%     -92.7%     166.75 ± 47%  interrupts.CPU10.RES:Rescheduling_interrupts
      7010 ± 18%     -98.1%     132.25 ± 40%  interrupts.CPU11.NMI:Non-maskable_interrupts
      7010 ± 18%     -98.1%     132.25 ± 40%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      2341 ± 16%     -69.6%     711.50 ±109%  interrupts.CPU11.RES:Rescheduling_interrupts
      7919 ± 15%     -98.5%     120.50 ± 42%  interrupts.CPU12.NMI:Non-maskable_interrupts
      7919 ± 15%     -98.5%     120.50 ± 42%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      2386 ± 19%     -94.0%     144.00 ±100%  interrupts.CPU12.RES:Rescheduling_interrupts
    935.75 ± 21%     -83.0%     159.25 ± 18%  interrupts.CPU13.CAL:Function_call_interrupts
      7595 ± 13%     -98.0%     154.00 ± 24%  interrupts.CPU13.NMI:Non-maskable_interrupts
      7595 ± 13%     -98.0%     154.00 ± 24%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      2150 ± 13%     -91.1%     191.75 ± 77%  interrupts.CPU13.RES:Rescheduling_interrupts
      1001 ± 45%     -87.2%     128.00 ± 25%  interrupts.CPU14.CAL:Function_call_interrupts
      7903 ± 10%     -98.2%     139.00 ± 36%  interrupts.CPU14.NMI:Non-maskable_interrupts
      7903 ± 10%     -98.2%     139.00 ± 36%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      2308 ± 15%     -90.1%     228.50 ± 54%  interrupts.CPU14.RES:Rescheduling_interrupts
      1065 ± 33%     -83.5%     175.25 ± 41%  interrupts.CPU15.CAL:Function_call_interrupts
      6965 ± 10%     -98.0%     142.00 ± 38%  interrupts.CPU15.NMI:Non-maskable_interrupts
      6965 ± 10%     -98.0%     142.00 ± 38%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      2405 ± 15%     -92.5%     179.25 ± 80%  interrupts.CPU15.RES:Rescheduling_interrupts
      5867 ± 35%     -97.0%     175.50 ± 27%  interrupts.CPU2.NMI:Non-maskable_interrupts
      5867 ± 35%     -97.0%     175.50 ± 27%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      2248 ±  8%     -97.2%      63.25 ± 88%  interrupts.CPU2.RES:Rescheduling_interrupts
    981.50 ± 24%     -65.8%     335.50 ±108%  interrupts.CPU3.CAL:Function_call_interrupts
      5039 ± 18%     -97.0%     150.00 ± 39%  interrupts.CPU3.NMI:Non-maskable_interrupts
      5039 ± 18%     -97.0%     150.00 ± 39%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
      2294 ± 24%     -94.5%     127.25 ± 85%  interrupts.CPU3.RES:Rescheduling_interrupts
      1040 ± 15%     -85.9%     146.75 ± 20%  interrupts.CPU4.CAL:Function_call_interrupts
      5630 ± 29%     -97.1%     161.75 ± 19%  interrupts.CPU4.NMI:Non-maskable_interrupts
      5630 ± 29%     -97.1%     161.75 ± 19%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
      2345 ± 17%     -90.7%     218.25 ±106%  interrupts.CPU4.RES:Rescheduling_interrupts
      6569 ± 18%     -98.0%     133.00 ± 31%  interrupts.CPU5.NMI:Non-maskable_interrupts
      6569 ± 18%     -98.0%     133.00 ± 31%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      2228 ± 11%     -76.0%     535.75 ± 63%  interrupts.CPU5.RES:Rescheduling_interrupts
      6364 ±  9%     -97.8%     142.75 ± 32%  interrupts.CPU6.NMI:Non-maskable_interrupts
      6364 ±  9%     -97.8%     142.75 ± 32%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      2190 ±  8%     -94.3%     123.75 ±105%  interrupts.CPU6.RES:Rescheduling_interrupts
    986.25 ± 36%     -83.5%     162.50 ± 29%  interrupts.CPU7.CAL:Function_call_interrupts
      6177 ± 22%     -97.4%     160.00 ± 21%  interrupts.CPU7.NMI:Non-maskable_interrupts
      6177 ± 22%     -97.4%     160.00 ± 21%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      2244 ±  9%     -87.1%     290.00 ± 98%  interrupts.CPU7.RES:Rescheduling_interrupts
      8589 ±  8%     -98.2%     157.75 ± 17%  interrupts.CPU8.NMI:Non-maskable_interrupts
      8589 ±  8%     -98.2%     157.75 ± 17%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      2361 ± 16%     -97.6%      56.75 ± 74%  interrupts.CPU8.RES:Rescheduling_interrupts
    779.75 ± 18%     -59.2%     318.25 ± 48%  interrupts.CPU9.CAL:Function_call_interrupts
      7461 ± 11%     -97.9%     153.75 ± 24%  interrupts.CPU9.NMI:Non-maskable_interrupts
      7461 ± 11%     -97.9%     153.75 ± 24%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      2425 ± 14%     -93.4%     159.25 ± 64%  interrupts.CPU9.RES:Rescheduling_interrupts
    108626 ±  2%     -97.9%       2330 ± 26%  interrupts.NMI:Non-maskable_interrupts
    108626 ±  2%     -97.9%       2330 ± 26%  interrupts.PMI:Performance_monitoring_interrupts
     38694 ± 17%     -91.3%       3365 ± 12%  interrupts.RES:Rescheduling_interrupts
    363.75 ±  6%     -76.2%      86.75 ±  5%  interrupts.TLB:TLB_shootdowns
     51.55 ±  7%     -51.1        0.49 ± 61%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     51.22 ±  7%     -50.7        0.49 ± 61%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     37.59 ±  7%     -37.6        0.00        perf-profile.calltrace.cycles-pp.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     37.31 ±  7%     -37.3        0.00        perf-profile.calltrace.cycles-pp.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     36.75 ±  7%     -36.7        0.00        perf-profile.calltrace.cycles-pp.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     35.43 ±  7%     -35.4        0.00        perf-profile.calltrace.cycles-pp.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64
     35.05 ±  7%     -35.0        0.00        perf-profile.calltrace.cycles-pp.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg
     34.29 ±  7%     -34.3        0.00        perf-profile.calltrace.cycles-pp.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
     30.84 ±  7%     -30.8        0.00        perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
     30.09 ±  7%     -30.1        0.00        perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc
     25.32 ±  7%     -25.3        0.00        perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
     23.03 ±  7%     -23.0        0.00        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
     22.64 ±  7%     -22.6        0.00        perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
     22.04 ±  7%     -22.0        0.00        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
     20.53 ±  7%     -20.5        0.00        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.sctp_packet_transmit
     20.43 ±  7%     -20.4        0.00        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
     20.23 ±  7%     -20.2        0.00        perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     20.16 ±  7%     -20.2        0.00        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     19.83 ±  7%     -19.8        0.00        perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     19.53 ±  7%     -19.5        0.00        perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
     19.19 ±  7%     -19.2        0.00        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     18.76 ±  7%     -18.8        0.00        perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     18.48 ±  7%     -18.5        0.00        perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
     18.33 ±  7%     -18.3        0.00        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     18.28 ±  7%     -18.3        0.00        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     18.08 ±  7%     -18.1        0.00        perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
     17.43 ±  7%     -17.4        0.00        perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg
     16.36 ±  7%     -16.4        0.00        perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
     16.29 ±  6%     -16.3        0.00        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     15.37 ±  7%     -15.4        0.00        perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
     12.95 ±  7%     -13.0        0.00        perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.63 ±  7%     -12.6        0.00        perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.47 ±  7%     -12.5        0.00        perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     12.25 ±  7%     -12.2        0.00        perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu
     12.01 ±  7%     -12.0        0.00        perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.40 ±  7%     -11.4        0.00        perf-profile.calltrace.cycles-pp.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64
     11.34 ±  7%     -11.3        0.00        perf-profile.calltrace.cycles-pp.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg
      8.62 ±  8%      -8.6        0.00        perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
      6.90 ±  7%      -6.9        0.00        perf-profile.calltrace.cycles-pp.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      5.95 ±  8%      -5.9        0.00        perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      5.70 ±  8%      -5.7        0.00        perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm
      5.30 ±  8%      -5.3        0.00        perf-profile.calltrace.cycles-pp.skb_release_all.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter
      5.30 ±  8%      -5.3        0.00        perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.consume_skb.sctp_chunk_put.sctp_outq_sack
      5.09 ±  8%      -5.1        0.00        perf-profile.calltrace.cycles-pp.sctp_wfree.skb_release_head_state.skb_release_all.consume_skb.sctp_chunk_put
      0.00            +0.7        0.65 ± 13%  perf-profile.calltrace.cycles-pp.rcu_nmi_enter.irq_enter.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state
      0.00            +0.7        0.68 ± 16%  perf-profile.calltrace.cycles-pp.update_ts_time_stats.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.__irqentry_text_start
      0.00            +0.7        0.71 ± 20%  perf-profile.calltrace.cycles-pp._find_next_bit.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length
      0.00            +0.7        0.73 ±  8%  perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.00            +0.8        0.81 ± 26%  perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.8        0.82 ± 14%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      0.00            +0.9        0.86 ± 26%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start
      0.00            +0.9        0.94 ± 10%  perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start
      0.00            +1.0        0.99 ±  9%  perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.__irqentry_text_start
      0.00            +1.0        1.02 ± 17%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.__irqentry_text_start
      0.77 ±  8%      +1.3        2.11 ± 12%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.00            +1.4        1.41 ± 16%  perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
      0.00            +1.4        1.44 ± 10%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state
      2.80 ± 21%      +1.6        4.36 ± 26%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
      2.80 ± 21%      +1.7        4.46 ± 26%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
      0.00            +1.8        1.81 ± 25%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start
      3.13 ± 18%      +1.9        5.00 ± 29%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
      3.13 ± 18%      +1.9        5.01 ± 29%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
      3.13 ± 18%      +1.9        5.01 ± 29%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
      0.89 ±  8%      +1.9        2.83 ± 18%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.00            +2.0        2.01 ± 12%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state
      0.00            +2.1        2.12 ± 28%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.17 ±  9%      +2.5        3.72 ± 14%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.00            +2.8        2.85 ±  6%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter
      0.00            +3.7        3.71 ± 37%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state
      0.00            +4.2        4.23 ± 19%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +4.8        4.80 ± 18%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.00            +5.1        5.11 ± 30%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter
      0.00            +5.4        5.45 ± 20%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start
      1.96 ±  8%      +5.8        7.78 ± 10%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00           +10.1       10.07 ± 21%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state
      0.00           +13.7       13.72 ± 16%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter
      0.00           +23.5       23.53 ± 17%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter.do_idle
      0.00           +27.8       27.83 ± 13%  perf-profile.calltrace.cycles-pp.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     23.04 ± 23%     +31.9       54.98 ±  9%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     36.81 ± 10%     +42.5       79.31 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     36.83 ± 10%     +43.8       80.63 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     43.31 ±  7%     +47.9       91.22        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     43.38 ±  7%     +48.0       91.34        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     43.36 ±  7%     +48.0       91.34        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     46.51 ±  8%     +49.8       96.35        perf-profile.calltrace.cycles-pp.secondary_startup_64
     51.59 ±  7%     -49.7        1.91 ± 45%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     51.31 ±  7%     -49.4        1.91 ± 45%  perf-profile.children.cycles-pp.do_syscall_64
     37.59 ±  7%     -37.6        0.00        perf-profile.children.cycles-pp.__sys_sendmsg
     37.31 ±  7%     -37.3        0.00        perf-profile.children.cycles-pp.___sys_sendmsg
     36.75 ±  7%     -36.8        0.00        perf-profile.children.cycles-pp.____sys_sendmsg
     35.44 ±  7%     -35.4        0.00        perf-profile.children.cycles-pp.sock_sendmsg
     35.06 ±  7%     -35.1        0.00        perf-profile.children.cycles-pp.sctp_sendmsg
     34.31 ±  7%     -34.3        0.00        perf-profile.children.cycles-pp.sctp_sendmsg_to_asoc
     30.85 ±  7%     -30.9        0.00        perf-profile.children.cycles-pp.sctp_primitive_SEND
     30.78 ±  7%     -30.8        0.00        perf-profile.children.cycles-pp.sctp_do_sm
     30.65 ±  7%     -30.6        0.00        perf-profile.children.cycles-pp.sctp_cmd_interpreter
     30.21 ±  7%     -30.2        0.00        perf-profile.children.cycles-pp.sctp_outq_flush
     25.32 ±  7%     -25.3        0.00        perf-profile.children.cycles-pp.sctp_packet_transmit
     23.04 ±  7%     -23.0        0.00        perf-profile.children.cycles-pp.__ip_queue_xmit
     22.65 ±  7%     -22.6        0.00        perf-profile.children.cycles-pp.ip_output
     22.05 ±  7%     -22.0        0.00        perf-profile.children.cycles-pp.ip_finish_output2
     20.61 ±  7%     -20.6        0.00        perf-profile.children.cycles-pp.__local_bh_enable_ip
     20.45 ±  7%     -20.5        0.00        perf-profile.children.cycles-pp.do_softirq
     20.25 ±  7%     -20.2        0.00        perf-profile.children.cycles-pp.do_softirq_own_stack
     19.84 ±  7%     -19.7        0.17 ±153%  perf-profile.children.cycles-pp.net_rx_action
     19.54 ±  7%     -19.5        0.00        perf-profile.children.cycles-pp.process_backlog
     19.19 ±  7%     -19.2        0.00        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     18.77 ±  7%     -18.8        0.00        perf-profile.children.cycles-pp.ip_rcv
     18.48 ±  7%     -18.5        0.00        perf-profile.children.cycles-pp.ip_local_deliver
     18.33 ±  7%     -18.3        0.00        perf-profile.children.cycles-pp.ip_local_deliver_finish
     18.29 ±  7%     -18.3        0.00        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     18.10 ±  7%     -18.1        0.00        perf-profile.children.cycles-pp.sctp_rcv
     20.20 ±  7%     -16.2        3.98 ± 38%  perf-profile.children.cycles-pp.__softirqentry_text_start
     16.30 ±  6%     -16.2        0.12 ± 77%  perf-profile.children.cycles-pp.poll_idle
     15.38 ±  7%     -15.4        0.00        perf-profile.children.cycles-pp.sctp_assoc_bh_rcv
     12.95 ±  7%     -13.0        0.00        perf-profile.children.cycles-pp.__sys_recvmsg
     12.64 ±  7%     -12.6        0.00        perf-profile.children.cycles-pp.___sys_recvmsg
     12.01 ±  7%     -12.0        0.00        perf-profile.children.cycles-pp.____sys_recvmsg
     11.41 ±  7%     -11.4        0.00        perf-profile.children.cycles-pp.inet_recvmsg
     11.35 ±  7%     -11.3        0.00        perf-profile.children.cycles-pp.sctp_recvmsg
      8.62 ±  8%      -8.6        0.00        perf-profile.children.cycles-pp.sctp_outq_sack
      7.98 ±  8%      -8.0        0.00        perf-profile.children.cycles-pp.sctp_chunk_put
      6.92 ±  7%      -6.9        0.00        perf-profile.children.cycles-pp.sctp_skb_recv_datagram
      6.66 ±  8%      -6.7        0.00        perf-profile.children.cycles-pp.consume_skb
      6.11 ±  8%      -6.1        0.01 ±173%  perf-profile.children.cycles-pp.skb_release_all
      6.10 ±  8%      -6.1        0.01 ±173%  perf-profile.children.cycles-pp.skb_release_head_state
      6.08 ±  8%      -5.9        0.20 ± 71%  perf-profile.children.cycles-pp.__schedule
      5.09 ±  8%      -5.1        0.00        perf-profile.children.cycles-pp.sctp_wfree
      4.55 ±  8%      -4.5        0.05 ± 58%  perf-profile.children.cycles-pp.__wake_up_common_lock
      4.19 ±  8%      -4.2        0.04 ±100%  perf-profile.children.cycles-pp.__wake_up_common
      4.12 ±  8%      -4.1        0.04 ±100%  perf-profile.children.cycles-pp.autoremove_wake_function
      4.08 ±  8%      -3.9        0.17 ± 44%  perf-profile.children.cycles-pp.try_to_wake_up
      3.52 ±  9%      -3.4        0.10 ± 90%  perf-profile.children.cycles-pp.schedule
      2.77 ±  8%      -2.7        0.10 ± 70%  perf-profile.children.cycles-pp.schedule_idle
      1.75 ± 10%      -1.7        0.05 ±112%  perf-profile.children.cycles-pp.dequeue_task_fair
      1.73 ±  3%      -1.7        0.08 ±132%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.61 ±  9%      -1.5        0.09 ±100%  perf-profile.children.cycles-pp.ttwu_do_activate
      2.06 ±  7%      -1.5        0.56 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.59 ±  9%      -1.5        0.09 ±100%  perf-profile.children.cycles-pp.activate_task
      1.50 ± 10%      -1.4        0.05 ±112%  perf-profile.children.cycles-pp.dequeue_entity
      1.54 ±  9%      -1.4        0.11 ± 71%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.44 ±  8%      -1.4        0.06 ±101%  perf-profile.children.cycles-pp.__slab_free
      1.35 ±  9%      -1.2        0.11 ± 84%  perf-profile.children.cycles-pp.kmem_cache_free
      1.31 ± 11%      -1.2        0.08 ±102%  perf-profile.children.cycles-pp.enqueue_entity
      1.22 ± 10%      -1.1        0.08 ±104%  perf-profile.children.cycles-pp.update_load_avg
      0.70 ± 14%      -0.7        0.03 ±100%  perf-profile.children.cycles-pp.update_curr
      0.66 ±  3%      -0.6        0.03 ±100%  perf-profile.children.cycles-pp.__switch_to
      0.58 ±  7%      -0.5        0.06 ±100%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.56 ±  6%      -0.5        0.07 ± 70%  perf-profile.children.cycles-pp.___might_sleep
      0.55 ±  4%      -0.4        0.13 ± 64%  perf-profile.children.cycles-pp.update_rq_clock
      0.38 ±  6%      -0.3        0.05 ±112%  perf-profile.children.cycles-pp.__might_sleep
      0.29 ± 11%      -0.3        0.03 ±100%  perf-profile.children.cycles-pp.finish_task_switch
      0.32 ± 10%      -0.2        0.10 ± 51%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.66 ± 10%      -0.2        0.44 ± 19%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.23 ± 16%      -0.2        0.04 ±110%  perf-profile.children.cycles-pp.___perf_sw_event
      0.20 ±  9%      -0.1        0.08 ±101%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.11 ± 24%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.cpuacct_charge
      0.09 ± 17%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.irq_work_run_list
      0.11 ±  4%      +0.0        0.14 ± 15%  perf-profile.children.cycles-pp.call_cpuidle
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.cpuidle_get_cpu_driver
      0.00            +0.1        0.08 ± 33%  perf-profile.children.cycles-pp.update_dl_rq_load_avg
      0.00            +0.1        0.10 ± 28%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.00            +0.1        0.10 ± 28%  perf-profile.children.cycles-pp.wp_page_copy
      0.00            +0.1        0.11 ± 47%  perf-profile.children.cycles-pp.raise_softirq
      0.00            +0.1        0.11 ± 45%  perf-profile.children.cycles-pp.rcu_is_cpu_rrupt_from_idle
      0.00            +0.1        0.11 ± 43%  perf-profile.children.cycles-pp.pipe_read
      0.00            +0.1        0.12 ± 23%  perf-profile.children.cycles-pp._dl_addr
      0.00            +0.1        0.12 ± 36%  perf-profile.children.cycles-pp.new_sync_read
      0.00            +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.retint_kernel
      0.00            +0.1        0.13 ± 57%  perf-profile.children.cycles-pp.do_filp_open
      0.00            +0.1        0.13 ± 57%  perf-profile.children.cycles-pp.path_openat
      0.00            +0.1        0.13 ± 30%  perf-profile.children.cycles-pp.do_wp_page
      0.00            +0.1        0.14 ± 31%  perf-profile.children.cycles-pp.rcu_idle_enter
      0.00            +0.1        0.14 ± 36%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.00            +0.1        0.14 ± 32%  perf-profile.children.cycles-pp.tick_program_event
      0.00            +0.1        0.14 ± 34%  perf-profile.children.cycles-pp.idle_cpu
      0.00            +0.1        0.14 ± 46%  perf-profile.children.cycles-pp.cpumask_next_and
      0.00            +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.00            +0.1        0.15 ± 34%  perf-profile.children.cycles-pp.trigger_load_balance
      0.05 ±  8%      +0.2        0.21 ± 13%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_enter
      0.00            +0.2        0.16 ± 50%  perf-profile.children.cycles-pp.x86_pmu_disable
      0.08 ± 59%      +0.2        0.25 ± 40%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.00            +0.2        0.18 ± 26%  perf-profile.children.cycles-pp.do_sys_open
      0.00            +0.2        0.18 ± 26%  perf-profile.children.cycles-pp.do_sys_openat2
      0.00            +0.2        0.19 ± 48%  perf-profile.children.cycles-pp.check_tsc_unstable
      0.00            +0.2        0.19 ± 35%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.2        0.19 ± 26%  perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event
      0.00            +0.2        0.20 ± 15%  perf-profile.children.cycles-pp.rcu_dynticks_task_exit
      0.04 ± 59%      +0.2        0.25 ± 36%  perf-profile.children.cycles-pp.ksoftirqd_running
      0.00            +0.2        0.21 ± 67%  perf-profile.children.cycles-pp.do_fault
      0.05 ± 58%      +0.2        0.26 ± 38%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.00            +0.2        0.22 ± 50%  perf-profile.children.cycles-pp.timekeeping_advance
      0.01 ±173%      +0.2        0.23 ± 24%  perf-profile.children.cycles-pp.rcu_qs
      0.00            +0.2        0.22 ± 63%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.00            +0.2        0.23 ± 64%  perf-profile.children.cycles-pp.irq_work_tick
      0.00            +0.2        0.23 ± 41%  perf-profile.children.cycles-pp.leave_mm
      0.00            +0.2        0.23 ± 69%  perf-profile.children.cycles-pp.__libc_fork
      0.00            +0.2        0.24 ± 26%  perf-profile.children.cycles-pp.run_local_timers
      0.00            +0.2        0.24 ± 63%  perf-profile.children.cycles-pp.read
      0.00            +0.2        0.24 ± 49%  perf-profile.children.cycles-pp.ksys_read
      0.03 ±100%      +0.2        0.27 ± 31%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.00            +0.3        0.25 ± 47%  perf-profile.children.cycles-pp.vfs_read
      0.00            +0.3        0.27 ± 30%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.00            +0.3        0.28 ± 66%  perf-profile.children.cycles-pp.forkshell
      0.24 ±  9%      +0.3        0.53 ± 15%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.3        0.29 ± 70%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.00            +0.3        0.29 ± 48%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.00            +0.3        0.29 ± 48%  perf-profile.children.cycles-pp.do_group_exit
      0.00            +0.3        0.29 ± 48%  perf-profile.children.cycles-pp.do_exit
      0.00            +0.3        0.29 ± 28%  perf-profile.children.cycles-pp.rb_erase
      0.08 ± 33%      +0.3        0.38 ± 43%  perf-profile.children.cycles-pp.get_cpu_device
      0.00            +0.3        0.30 ± 54%  perf-profile.children.cycles-pp.kthread
      0.00            +0.3        0.30 ± 20%  perf-profile.children.cycles-pp.rcu_nmi_exit
      0.00            +0.3        0.30 ± 97%  perf-profile.children.cycles-pp.load_elf_binary
      0.00            +0.3        0.31 ± 24%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.3        0.31 ± 93%  perf-profile.children.cycles-pp.exec_binprm
      0.00            +0.3        0.33 ± 49%  perf-profile.children.cycles-pp.ret_from_fork
      0.00            +0.3        0.33 ± 28%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.3        0.34 ± 41%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.3        0.35 ± 22%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.00            +0.4        0.35 ± 40%  perf-profile.children.cycles-pp.account_process_tick
      0.01 ±173%      +0.4        0.39 ± 35%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.4        0.38 ± 18%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.00            +0.4        0.41 ± 47%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.00            +0.4        0.42 ± 16%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.23 ± 11%      +0.4        0.66 ± 14%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.4        0.43 ± 54%  perf-profile.children.cycles-pp.mmput
      0.00            +0.4        0.43 ± 54%  perf-profile.children.cycles-pp.exit_mmap
      0.00            +0.4        0.44 ± 19%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.25 ±  5%      +0.5        0.71 ± 21%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.00            +0.5        0.46 ± 16%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.23 ± 13%      +0.5        0.69 ± 35%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.5        0.46 ± 45%  perf-profile.children.cycles-pp.handle_mm_fault
      0.00            +0.5        0.49 ±106%  perf-profile.children.cycles-pp.__do_execve_file
      0.00            +0.5        0.50 ±103%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.00            +0.5        0.50 ±103%  perf-profile.children.cycles-pp.execve
      0.40 ± 13%      +0.5        0.91 ± 35%  perf-profile.children.cycles-pp.sched_clock
      0.39 ± 14%      +0.5        0.90 ± 36%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.5        0.51 ± 20%  perf-profile.children.cycles-pp.update_blocked_averages
      0.00            +0.5        0.54 ± 37%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.5        0.54 ± 13%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.00            +0.6        0.55 ± 38%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.00            +0.6        0.56 ± 41%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.00            +0.6        0.56 ± 55%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.6        0.56 ± 13%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.6        0.57 ± 19%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.00            +0.6        0.57 ± 11%  perf-profile.children.cycles-pp.find_busiest_group
      0.00            +0.6        0.58 ± 38%  perf-profile.children.cycles-pp.__run_timers
      0.00            +0.6        0.58 ± 37%  perf-profile.children.cycles-pp.native_read_msr
      0.00            +0.6        0.63 ± 42%  perf-profile.children.cycles-pp.page_fault
      0.00            +0.6        0.63 ± 21%  perf-profile.children.cycles-pp.interrupt_entry
      0.00            +0.7        0.66 ± 40%  perf-profile.children.cycles-pp.run_timer_softirq
      0.42 ±  9%      +0.7        1.09 ±  4%  perf-profile.children.cycles-pp.read_tsc
      0.44 ± 11%      +0.7        1.13 ± 32%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.7        0.74 ± 33%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.7        0.74 ±  9%  perf-profile.children.cycles-pp.rcu_nmi_enter
      0.00            +0.8        0.76 ±  8%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.8        0.76 ± 80%  perf-profile.children.cycles-pp.rcu_core
      0.10 ±  4%      +0.8        0.89 ± 12%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.8        0.80 ± 31%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.8        0.84 ± 35%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.23 ±  3%      +0.8        1.07 ± 15%  perf-profile.children.cycles-pp._find_next_bit
      0.30 ± 11%      +0.8        1.15 ± 24%  perf-profile.children.cycles-pp.native_write_msr
      0.00            +0.9        0.88 ± 14%  perf-profile.children.cycles-pp.load_balance
      0.00            +0.9        0.89 ± 27%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +0.9        0.94 ± 10%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.00            +1.0        0.96 ± 19%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +1.0        1.01 ± 19%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.46 ±  7%      +1.1        1.56 ± 19%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.00            +1.1        1.10 ± 16%  perf-profile.children.cycles-pp.rebalance_domains
      0.81 ±  8%      +1.4        2.22 ± 15%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +1.5        1.50 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      3.13 ± 18%      +1.9        5.01 ± 29%  perf-profile.children.cycles-pp.start_kernel
      0.48 ±  6%      +2.0        2.46 ± 18%  perf-profile.children.cycles-pp.ktime_get
      0.00            +2.0        1.98 ± 27%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +2.2        2.18 ± 15%  perf-profile.children.cycles-pp.tick_irq_enter
      0.94 ±  8%      +2.2        3.15 ± 21%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.01 ±173%      +2.2        2.25 ± 31%  perf-profile.children.cycles-pp.scheduler_tick
      1.24 ±  8%      +2.7        3.91 ± 18%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +3.1        3.09 ±  9%  perf-profile.children.cycles-pp.irq_enter
      0.06 ± 13%      +4.3        4.40 ± 19%  perf-profile.children.cycles-pp.update_process_times
      0.07 ±  7%      +4.9        4.94 ± 19%  perf-profile.children.cycles-pp.tick_sched_handle
      0.03 ±100%      +5.4        5.45 ± 30%  perf-profile.children.cycles-pp.irq_exit
      0.08 ±  5%      +5.6        5.65 ± 21%  perf-profile.children.cycles-pp.tick_sched_timer
      2.08 ±  9%      +6.2        8.25 ± 13%  perf-profile.children.cycles-pp.menu_select
      0.11 ±  9%     +10.3       10.38 ± 20%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.17 ±  9%     +13.6       13.80 ± 15%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.24 ± 10%     +23.3       23.56 ± 17%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.27 ±  9%     +25.5       25.82 ± 14%  perf-profile.children.cycles-pp.__irqentry_text_start
     23.04 ± 23%     +32.0       55.06 ±  9%  perf-profile.children.cycles-pp.intel_idle
     39.62 ± 11%     +45.4       85.06        perf-profile.children.cycles-pp.cpuidle_enter_state
     39.63 ± 11%     +45.5       85.09        perf-profile.children.cycles-pp.cpuidle_enter
     43.38 ±  7%     +48.0       91.34        perf-profile.children.cycles-pp.start_secondary
     46.51 ±  8%     +49.8       96.35        perf-profile.children.cycles-pp.secondary_startup_64
     46.51 ±  8%     +49.8       96.35        perf-profile.children.cycles-pp.cpu_startup_entry
     46.49 ±  8%     +50.0       96.46        perf-profile.children.cycles-pp.do_idle
     16.17 ±  6%     -16.0        0.12 ± 77%  perf-profile.self.cycles-pp.poll_idle
      1.72 ±  3%      -1.6        0.08 ±132%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      2.04 ±  7%      -1.6        0.45 ± 33%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.43 ±  8%      -1.4        0.06 ±101%  perf-profile.self.cycles-pp.__slab_free
      0.64 ±  2%      -0.6        0.03 ±100%  perf-profile.self.cycles-pp.__switch_to
      0.63 ± 13%      -0.6        0.05 ±112%  perf-profile.self.cycles-pp.update_load_avg
      0.55 ±  6%      -0.5        0.07 ± 70%  perf-profile.self.cycles-pp.___might_sleep
      0.47 ±  7%      -0.4        0.05 ±108%  perf-profile.self.cycles-pp.kmem_cache_free
      0.35 ±  5%      -0.3        0.03 ±100%  perf-profile.self.cycles-pp.__might_sleep
      0.35 ±  4%      -0.3        0.07 ± 62%  perf-profile.self.cycles-pp.update_rq_clock
      0.28 ±  5%      -0.2        0.03 ±100%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.31 ± 11%      -0.2        0.10 ± 51%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.61 ±  8%      -0.2        0.41 ± 22%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.11 ± 24%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.cpuacct_charge
      0.11 ±  4%      +0.0        0.14 ± 15%  perf-profile.self.cycles-pp.call_cpuidle
      0.00            +0.1        0.07 ± 22%  perf-profile.self.cycles-pp.cpuidle_get_cpu_driver
      0.00            +0.1        0.08 ± 33%  perf-profile.self.cycles-pp.update_dl_rq_load_avg
      0.01 ±173%      +0.1        0.11 ± 48%  perf-profile.self.cycles-pp.rcu_qs
      0.00            +0.1        0.10 ± 28%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.00            +0.1        0.11 ± 47%  perf-profile.self.cycles-pp.raise_softirq
      0.00            +0.1        0.12 ± 23%  perf-profile.self.cycles-pp._dl_addr
      0.00            +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.retint_kernel
      0.00            +0.1        0.13 ± 35%  perf-profile.self.cycles-pp.trigger_load_balance
      0.00            +0.1        0.14 ± 32%  perf-profile.self.cycles-pp.tick_program_event
      0.00            +0.1        0.14 ± 34%  perf-profile.self.cycles-pp.idle_cpu
      0.00            +0.1        0.14 ± 53%  perf-profile.self.cycles-pp.rcu_nmi_exit
      0.00            +0.1        0.14 ± 35%  perf-profile.self.cycles-pp.rcu_irq_enter
      0.09 ± 27%      +0.1        0.24 ± 31%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.05 ±  8%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_enter
      0.00            +0.2        0.15 ± 13%  perf-profile.self.cycles-pp.rcu_dynticks_task_exit
      0.00            +0.2        0.15 ± 36%  perf-profile.self.cycles-pp.sched_clock_idle_wakeup_event
      0.00            +0.2        0.16 ± 56%  perf-profile.self.cycles-pp.check_tsc_unstable
      0.08 ± 59%      +0.2        0.24 ± 43%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.00            +0.2        0.16 ± 50%  perf-profile.self.cycles-pp.x86_pmu_disable
      0.00            +0.2        0.17 ± 29%  perf-profile.self.cycles-pp.timerqueue_del
      0.00            +0.2        0.17 ± 37%  perf-profile.self.cycles-pp.tick_sched_timer
      0.00            +0.2        0.18 ± 53%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.01 ±173%      +0.2        0.19 ± 30%  perf-profile.self.cycles-pp.irq_work_needs_cpu
      0.00            +0.2        0.18 ± 29%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.2        0.18 ± 18%  perf-profile.self.cycles-pp.update_blocked_averages
      0.00            +0.2        0.19 ± 17%  perf-profile.self.cycles-pp.clockevents_program_event
      0.00            +0.2        0.19 ± 84%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.04 ± 59%      +0.2        0.24 ± 38%  perf-profile.self.cycles-pp.ksoftirqd_running
      0.00            +0.2        0.20 ± 32%  perf-profile.self.cycles-pp.run_local_timers
      0.04 ± 59%      +0.2        0.25 ± 41%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.00            +0.2        0.21 ± 68%  perf-profile.self.cycles-pp.irq_work_tick
      0.00            +0.2        0.21 ± 51%  perf-profile.self.cycles-pp.run_timer_softirq
      0.00            +0.2        0.22 ± 48%  perf-profile.self.cycles-pp.load_balance
      0.00            +0.2        0.22 ± 24%  perf-profile.self.cycles-pp.irq_exit
      0.00            +0.2        0.22 ± 27%  perf-profile.self.cycles-pp.rebalance_domains
      0.00            +0.2        0.23 ± 18%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +0.2        0.23 ± 24%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.2        0.24 ± 23%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.00            +0.2        0.24 ± 35%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.00            +0.2        0.24 ± 16%  perf-profile.self.cycles-pp.update_ts_time_stats
      0.00            +0.2        0.24 ± 18%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.00            +0.3        0.26 ± 35%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.24 ±  9%      +0.3        0.52 ± 17%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.3        0.28 ± 26%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.3        0.28 ± 32%  perf-profile.self.cycles-pp.rb_erase
      0.08 ± 29%      +0.3        0.38 ± 43%  perf-profile.self.cycles-pp.get_cpu_device
      0.00            +0.3        0.31 ± 44%  perf-profile.self.cycles-pp.__run_timers
      0.00            +0.3        0.32 ±  7%  perf-profile.self.cycles-pp.rcu_nmi_enter
      0.00            +0.3        0.34 ± 41%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.3        0.34 ± 36%  perf-profile.self.cycles-pp.account_process_tick
      0.20 ± 10%      +0.4        0.56 ± 31%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.00            +0.4        0.37 ± 32%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.4        0.37 ± 47%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.03 ±100%      +0.4        0.43 ± 20%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.00            +0.4        0.41 ± 18%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.4        0.42 ± 16%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.00            +0.4        0.44 ± 19%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.5        0.47 ± 65%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.6        0.55 ± 38%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.6        0.56 ± 42%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.00            +0.6        0.57 ± 19%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.00            +0.6        0.57 ± 32%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.6        0.58 ± 37%  perf-profile.self.cycles-pp.native_read_msr
      0.00            +0.6        0.62 ± 50%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.41 ±  8%      +0.6        1.04 ±  5%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.6        0.63 ± 21%  perf-profile.self.cycles-pp.interrupt_entry
      0.08 ± 10%      +0.7        0.77 ± 12%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.7        0.73 ± 49%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +0.7        0.75 ± 24%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.23 ±  3%      +0.8        1.05 ± 13%  perf-profile.self.cycles-pp._find_next_bit
      0.30 ± 11%      +0.8        1.14 ± 23%  perf-profile.self.cycles-pp.native_write_msr
      0.00            +1.0        0.96 ± 19%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.14 ± 10%      +1.2        1.38 ± 30%  perf-profile.self.cycles-pp.ktime_get
      0.57 ±  9%      +2.8        3.37 ± 14%  perf-profile.self.cycles-pp.menu_select
      0.10 ± 16%      +3.3        3.40 ±  5%  perf-profile.self.cycles-pp.cpuidle_enter_state
     23.04 ± 23%     +31.8       54.89 ±  9%  perf-profile.self.cycles-pp.intel_idle


                                                                                
                               netperf.Throughput_tps                           
                                                                                
  120000 +------------------------------------------------------------------+   
         |                                                                  |   
  100000 |.++.+.++.+.++.+.++.+.++.+.++.+.+.++.+.++.+.++.+.++.+.++.+.++.+.++.|   
         |                                                                  |   
         |                                                                  |   
   80000 |-+                                                                |   
         |                                                                  |   
   60000 |-+                                                                |   
         |                                                                  |   
   40000 |-+                                                                |   
         |                                                                  |   
         |                                                                  |   
   20000 |-+                                                                |   
         |                                                                  |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            netperf.Throughput_total_tps                        
                                                                                
  450000 +------------------------------------------------------------------+   
         |.++.+.++.+.+ .+.+ .+.++.+.++.+.+.++.+.++.+.++.+.++.+.++.+.++.+.+ .|   
  400000 |-+          +    +                                              + |   
  350000 |-+                                                                |   
         |                                                                  |   
  300000 |-+                                                                |   
  250000 |-+                                                                |   
         |                                                                  |   
  200000 |-+                                                                |   
  150000 |-+                                                                |   
         |                                                                  |   
  100000 |-+                                                                |   
   50000 |-+                                                                |   
         |                                                                  |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   netperf.workload                             
                                                                                
  1.4e+08 +-----------------------------------------------------------------+   
          |                                      .+          +.++.   +. .+  |   
  1.2e+08 |.++.+.++.+.++.+.++.++.+.++.+.++.+.++.+  +.+.++.+.+     +.+  +  +.|   
          |                                                                 |   
    1e+08 |-+                                                               |   
          |                                                                 |   
    8e+07 |-+                                                               |   
          |                                                                 |   
    6e+07 |-+                                                               |   
          |                                                                 |   
    4e+07 |-+                                                               |   
          |                                                                 |   
    2e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             netperf.time.system_time                           
                                                                                
  900 +---------------------------------------------------------------------+   
      |.+.++.+.+.++.+.+.++.+.+.++.+.+.++.+.+.++.+.      .+.++.   .++.+.+. +.|   
  800 |-+                                         +.++.+      +.+        +  |   
  700 |-+                                                                   |   
      |                                                                     |   
  600 |-+                                                                   |   
  500 |-+                                                                   |   
      |                                                                     |   
  400 |-+                                                                   |   
  300 |-+                                                                   |   
      |                                                                     |   
  200 |-+                                                                   |   
  100 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        netperf.time.voluntary_context_switches                 
                                                                                
  1.4e+08 +-----------------------------------------------------------------+   
          |                                      .+          +.++.   +. .+  |   
  1.2e+08 |.++.+.++.+.++.+.++.++.+.++.+.++.+.++.+  +.+.++.+.+     +.+  +  +.|   
          |                                                                 |   
    1e+08 |-+                                                               |   
          |                                                                 |   
    8e+07 |-+                                                               |   
          |                                                                 |   
    6e+07 |-+                                                               |   
          |                                                                 |   
    4e+07 |-+                                                               |   
          |                                                                 |   
    2e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                     netperf.time.involuntary_context_switches                  
                                                                                
  3500 +--------------------------------------------------------------------+   
       |                                                     .+.            |   
  3000 |.+     .++.+.+.       .+.      .+.           .+     +   +.  .+.    .|   
       |  +.+.+        ++.+.++   +.++.+   ++.+.++.+.+  +.+.+      ++   +.++ |   
  2500 |-+                                                                  |   
       |                                                                    |   
  2000 |-+                                                                  |   
       |                                                                    |   
  1500 |-+                                                                  |   
       |                                                                    |   
  1000 |-+                                                                  |   
       |                                                                    |   
   500 |-+                                                                  |   
       |                                                                    |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              netperf.time.user_time                            
                                                                                
  29.5 +--------------------------------------------------------------------+   
       |  ::  +                  :: +     :  +                            + |   
    29 |-+:: ::                 : : :    : :::                     +      ::|   
  28.5 |-:  :: :       +        :  : :   : :::                     :      ::|   
       | +  +  :       :       +   + :  +  +  :                    ::     : |   
    28 |-:     :      ::       :     :  :     :   +         +   +  ::  +  : |   
       |:       :     ::      :      : :      :  + :        :   :  :: :: :  |   
  27.5 |:+      ++   +  :     :       ::      : +  :   +    ::  :::  :: ::  |   
       |          :  :  :    +        +        ::   :  :    :: : ::  +  ::  |   
    27 |-+        : :   :    :                 :    +.+ :   :: : ::      :  |   
  26.5 |-+         ::   +   +                  +        :  : : :  :      +  |   
       |           +     :  :                            : :  ::  +         |   
    26 |-+               : :                             +.:  :             |   
       |                  ::                               +  +             |   
  25.5 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                     netperf.time.percent_of_cpu_this_job_got                   
                                                                                
  276 +---------------------------------------------------------------------+   
      |      :   :                  :        :                              |   
  275 |-+   : : ::  +.+    +       : :      ::  +        +                  |   
      |     : : : ::   :   :       : :      : :::        :                  |   
      |    :   :  ::   :  : :     :  :     :  :::        ::                 |   
  274 |-+.++   +  +     + : :  ++.+  : +.+.+  +  :       ::       +         |   
      |+                 :   :+       ::         :      :  :      :         |   
  273 |-+                +   +        +          :      :  +     : : +.+  +.|   
      |                                          :      :   :    : :+  :  : |   
  272 |-+                                        :      :   +   +  +    : : |   
      |                                           :     :    : :        : : |   
      |                                           :     :    : :        ::  |   
  271 |-+                                         +.++ :      +         ::  |   
      |                                               +:                 :  |   
  270 +---------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.7.0-14529-gcf8d0e3e583cb" of type "text/plain" (206221 bytes)

View attachment "job-script" of type "text/plain" (7860 bytes)

View attachment "job.yaml" of type "text/plain" (5404 bytes)

View attachment "reproduce" of type "text/plain" (493 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ