lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Sat, 28 Mar 2020 21:23:25 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Aubrey Li <aubrey.li@...el.com>
Cc:     vincent.guittot@...aro.org, mingo@...hat.com, peterz@...radead.org,
        juri.lelli@...hat.com, dietmar.eggemann@....com,
        rostedt@...dmis.org, bsegall@...gle.com, mgorman@...e.de,
        linux-kernel@...r.kernel.org, tim.c.chen@...ux.intel.com,
        vpillai@...italocean.com, joel@...lfernandes.org,
        Aubrey Li <aubrey.li@...el.com>,
        Aubrey Li <aubrey.li@...ux.intel.com>, lkp@...ts.01.org
Subject: [sched/fair] 59901cb452: netperf.Throughput_Mbps -27.3% regression

Greeting,

FYI, we noticed a -27.3% regression of netperf.Throughput_Mbps due to commit:


commit: 59901cb4520c44bfce81f523bc61e7284a931ad1 ("[PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group")
url: https://github.com/0day-ci/linux/commits/Aubrey-Li/sched-fair-Don-t-pull-task-if-local-group-is-more-loaded-than-busiest-group/20200326-080334


in testcase: netperf
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 25%
	cluster: cs-localhost
	send_size: 5K
	test: TCP_SENDFILE
	cpufreq_governor: performance
	ucode: 0x2000065

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_Mbps -31.1% regression                     |
| test machine     | 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory |
| test parameters  | cluster=cs-localhost                                                   |
|                  | cpufreq_governor=performance                                           |
|                  | ip=ipv4                                                                |
|                  | nr_threads=25%                                                         |
|                  | runtime=300s                                                           |
|                  | send_size=10K                                                          |
|                  | test=SCTP_STREAM_MANY                                                  |
|                  | ucode=0xb000038                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | unixbench: unixbench.score -4.0% regression                            |
| test machine     | 104 threads Skylake with 192G memory                                   |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=30%                                                            |
|                  | runtime=300s                                                           |
|                  | test=whetstone-double                                                  |
|                  | ucode=0x2000065                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | unixbench: unixbench.score 29.1% improvement                           |
| test machine     | 104 threads Skylake with 192G memory                                   |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=30%                                                            |
|                  | runtime=300s                                                           |
|                  | test=context1                                                          |
|                  | ucode=0x2000065                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | kbuild: kbuild.buildtime_per_iteration 7.5% regression                 |
| test machine     | 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory       |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=50%                                                            |
|                  | runtime=300s                                                           |
|                  | target=vmlinux_prereq                                                  |
|                  | ucode=0x27                                                             |
+------------------+------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/25%/debian-x86_64-20191114.cgz/300s/5K/lkp-skl-2sp7/TCP_SENDFILE/netperf/0x2000065

commit: 
  9c40365a65 ("threads: Update PID limit comment according to futex UAPI change")
  59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group")

9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :4           50%           2:4     dmesg.WARNING:at_ip__fsnotify_parent/0x
         %stddev     %change         %stddev
             \          |                \  
     18708           -27.3%      13593        netperf.Throughput_Mbps
    336757           -27.3%     244677        netperf.Throughput_total_Mbps
      8293 ±  2%     +82.7%      15156 ± 57%  netperf.time.involuntary_context_switches
      2291            +2.6%       2350        netperf.time.maximum_resident_set_size
      5103            -8.9%       4651        netperf.time.minor_page_faults
      4704            +1.4%       4771        netperf.time.system_time
    695.32            -9.7%     627.59        netperf.time.user_time
    431.25 ± 51%    +394.1%       2131 ± 59%  netperf.time.voluntary_context_switches
 2.466e+09           -27.3%  1.792e+09        netperf.workload
    526188 ±  5%     -21.3%     414060 ±  7%  meminfo.DirectMap4k
     53993 ±  3%     -13.9%      46507 ±  5%  meminfo.Shmem
   1482325 ±  2%     -24.4%    1120083        vmstat.system.cs
    150508            -1.5%     148218        vmstat.system.in
      3999 ±  8%     -10.8%       3566 ±  5%  slabinfo.kmalloc-rcl-64.active_objs
      3999 ±  8%     -10.8%       3566 ±  5%  slabinfo.kmalloc-rcl-64.num_objs
     10897           +12.6%      12274 ±  4%  slabinfo.vmap_area.active_objs
     10898           +12.7%      12277 ±  4%  slabinfo.vmap_area.num_objs
 2.208e+08           -20.9%  1.746e+08        cpuidle.C1.usage
   5894730 ± 24%   +7005.1%  4.188e+08 ± 25%  cpuidle.C6.time
     19279 ±  3%   +2594.5%     519470 ± 17%  cpuidle.C6.usage
  35085456 ±  9%     -39.3%   21307532 ±  7%  cpuidle.POLL.time
   4822644 ± 12%     -49.7%    2423791 ±  9%  cpuidle.POLL.usage
     76807            -2.7%      74740        proc-vmstat.nr_active_anon
     13500 ±  3%     -13.8%      11639 ±  5%  proc-vmstat.nr_shmem
     76807            -2.7%      74740        proc-vmstat.nr_zone_active_anon
     15013 ± 91%     -86.2%       2071 ±151%  proc-vmstat.numa_hint_faults
     13589 ±  5%     -15.5%      11476 ± 11%  proc-vmstat.pgactivate
    833769            -2.1%     815970        proc-vmstat.pgfault
    754892            +1.1%     762828        proc-vmstat.pgfree
    152278 ± 42%     -96.3%       5569 ± 23%  sched_debug.cfs_rq:/.min_vruntime.min
   1169341           +30.4%    1524450 ±  7%  sched_debug.cfs_rq:/.min_vruntime.stddev
   1169341           +30.4%    1524462 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
      1.58 ±  6%     +22.0%       1.93 ±  7%  sched_debug.cpu.clock.stddev
      1.58 ±  6%     +22.1%       1.93 ±  7%  sched_debug.cpu.clock_task.stddev
   3098435           -28.6%    2211323 ±  9%  sched_debug.cpu.nr_switches.avg
      1573 ± 16%     -21.6%       1233 ±  9%  sched_debug.cpu.nr_switches.min
     56.75 ± 11%     +18.5%      67.25 ±  4%  sched_debug.cpu.nr_uninterruptible.max
      6543 ±  8%     +16.6%       7631 ±  5%  numa-vmstat.node0.nr_kernel_stack
    574.25 ± 30%     +86.8%       1072 ± 22%  numa-vmstat.node0.nr_page_table_pages
      8042 ± 19%     +23.3%       9919 ± 13%  numa-vmstat.node0.nr_slab_reclaimable
     34409 ± 22%     -56.6%      14946 ± 54%  numa-vmstat.node1.nr_active_anon
     25847 ± 30%     -67.3%       8458 ± 93%  numa-vmstat.node1.nr_anon_pages
      6123 ±  8%     -18.0%       5022 ±  8%  numa-vmstat.node1.nr_kernel_stack
    854.50 ± 19%     -56.5%     372.00 ± 65%  numa-vmstat.node1.nr_page_table_pages
     18421 ±  7%     -17.9%      15125 ±  7%  numa-vmstat.node1.nr_slab_unreclaimable
     34409 ± 22%     -56.6%      14946 ± 54%  numa-vmstat.node1.nr_zone_active_anon
     21046 ± 47%     +55.4%      32703 ± 30%  numa-vmstat.node1.numa_other
     32169 ± 19%     +23.3%      39680 ± 13%  numa-meminfo.node0.KReclaimable
      6543 ±  8%     +16.6%       7630 ±  5%  numa-meminfo.node0.KernelStack
   1001821 ±  9%     +14.4%    1145780 ±  4%  numa-meminfo.node0.MemUsed
      2299 ± 30%     +86.6%       4290 ± 22%  numa-meminfo.node0.PageTables
     32169 ± 19%     +23.3%      39680 ± 13%  numa-meminfo.node0.SReclaimable
    102778 ±  7%     +19.9%     123222 ±  6%  numa-meminfo.node0.Slab
    137587 ± 22%     -56.6%      59688 ± 54%  numa-meminfo.node1.Active
    137493 ± 22%     -56.6%      59648 ± 54%  numa-meminfo.node1.Active(anon)
     52091 ± 42%     -68.4%      16453 ±161%  numa-meminfo.node1.AnonHugePages
    103405 ± 30%     -67.3%      33836 ± 93%  numa-meminfo.node1.AnonPages
      6116 ±  8%     -18.0%       5017 ±  8%  numa-meminfo.node1.KernelStack
   1050635 ±  8%     -13.4%     909860 ±  5%  numa-meminfo.node1.MemUsed
      3416 ± 19%     -56.6%       1483 ± 65%  numa-meminfo.node1.PageTables
     73684 ±  7%     -17.9%      60503 ±  7%  numa-meminfo.node1.SUnreclaim
    108980 ±  7%     -19.1%      88152 ± 10%  numa-meminfo.node1.Slab
     81937 ±  6%     +25.2%     102562 ±  9%  softirqs.CPU16.RCU
     21615 ± 70%    +204.7%      65858 ± 22%  softirqs.CPU16.SCHED
   5010947 ± 28%     -67.3%    1637461 ±148%  softirqs.CPU25.NET_RX
     45054 ±  5%     -21.6%      35310 ± 26%  softirqs.CPU25.SCHED
   4691793 ± 34%     -69.2%    1444457 ±170%  softirqs.CPU26.NET_RX
     48190 ±  9%     -29.8%      33825 ± 34%  softirqs.CPU26.SCHED
   5045331 ± 36%     -77.2%    1148265 ±171%  softirqs.CPU29.NET_RX
     51391 ± 23%     -69.3%      15752 ±133%  softirqs.CPU36.SCHED
     52040 ± 11%     -73.0%      14042 ±111%  softirqs.CPU46.SCHED
     37936 ± 50%     -78.3%       8234 ±104%  softirqs.CPU47.SCHED
     46988 ± 16%     -67.0%      15513 ± 99%  softirqs.CPU52.SCHED
   5967380 ± 30%     -80.4%    1169737 ±173%  softirqs.CPU54.NET_RX
   6170696 ± 42%     -76.1%    1472055 ±173%  softirqs.CPU57.NET_RX
   6718083 ± 40%     -77.3%    1528221 ±173%  softirqs.CPU58.NET_RX
   6397847 ± 27%     -74.5%    1629273 ±173%  softirqs.CPU59.NET_RX
     57352 ±  5%     -35.6%      36922 ± 10%  softirqs.CPU63.SCHED
   4947229 ± 28%     -76.6%    1159904 ±173%  softirqs.CPU64.NET_RX
   4902059 ± 41%     -72.8%    1331256 ±169%  softirqs.CPU68.NET_RX
 3.636e+08           -23.0%  2.799e+08        softirqs.NET_RX
    106.50 ± 96%     -96.7%       3.50 ±125%  interrupts.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19
     24013            -1.8%      23590        interrupts.CAL:Function_call_interrupts
     14701 ± 60%     -99.2%     123.75 ± 89%  interrupts.CPU11.RES:Rescheduling_interrupts
    106.00 ± 97%     -97.2%       3.00 ±135%  interrupts.CPU19.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19
     23189 ± 24%     -94.6%       1242 ±139%  interrupts.CPU25.RES:Rescheduling_interrupts
     23527 ± 34%     -95.2%       1129 ±152%  interrupts.CPU26.RES:Rescheduling_interrupts
     13122 ± 50%     -98.4%     207.50 ±143%  interrupts.CPU29.RES:Rescheduling_interrupts
     21974 ± 29%     -77.8%       4872 ±172%  interrupts.CPU3.RES:Rescheduling_interrupts
     28365 ± 52%     -79.1%       5941 ±171%  interrupts.CPU36.RES:Rescheduling_interrupts
     27230 ± 58%     -98.1%     518.00 ± 98%  interrupts.CPU46.RES:Rescheduling_interrupts
     21075 ± 63%     -99.0%     205.25 ± 58%  interrupts.CPU47.RES:Rescheduling_interrupts
      6718 ± 19%     -43.9%       3772 ± 63%  interrupts.CPU48.NMI:Non-maskable_interrupts
      6718 ± 19%     -43.9%       3772 ± 63%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      7806 ± 72%     -73.1%       2100 ±147%  interrupts.CPU52.RES:Rescheduling_interrupts
      7343 ± 23%     -57.7%       3106 ± 96%  interrupts.CPU54.NMI:Non-maskable_interrupts
      7343 ± 23%     -57.7%       3106 ± 96%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      7679 ± 84%     -98.4%     123.75 ±104%  interrupts.CPU54.RES:Rescheduling_interrupts
     35789 ± 57%     -75.5%       8758 ±173%  interrupts.CPU56.RES:Rescheduling_interrupts
     23882 ± 72%     -94.9%       1212 ±172%  interrupts.CPU57.RES:Rescheduling_interrupts
     21256 ± 72%     -94.7%       1134 ±170%  interrupts.CPU58.RES:Rescheduling_interrupts
     73.50 ±145%     -90.8%       6.75 ± 72%  interrupts.CPU6.TLB:TLB_shootdowns
     32631 ± 47%     -68.2%      10387 ± 72%  interrupts.CPU61.RES:Rescheduling_interrupts
     29653 ± 29%     -81.4%       5518 ±171%  interrupts.CPU62.RES:Rescheduling_interrupts
     25421 ± 21%     -91.3%       2218 ±165%  interrupts.CPU63.RES:Rescheduling_interrupts
     14735 ± 54%     -98.6%     200.50 ±155%  interrupts.CPU64.RES:Rescheduling_interrupts
     11691 ± 85%     -64.2%       4184 ±172%  interrupts.CPU66.RES:Rescheduling_interrupts
      6240 ± 32%     -68.2%       1982 ±140%  interrupts.CPU67.NMI:Non-maskable_interrupts
      6240 ± 32%     -68.2%       1982 ±140%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      6781 ±120%     -93.8%     423.50 ±145%  interrupts.CPU68.RES:Rescheduling_interrupts
     12981 ± 64%     -98.7%     174.50 ±105%  interrupts.CPU70.RES:Rescheduling_interrupts
     14238 ± 44%     -65.1%       4975 ±168%  interrupts.CPU71.RES:Rescheduling_interrupts
    107.00 ±  5%     -17.1%      88.75 ± 11%  interrupts.IWI:IRQ_work_interrupts
    389189 ±  4%     -31.5%     266460 ±  8%  interrupts.NMI:Non-maskable_interrupts
    389189 ±  4%     -31.5%     266460 ±  8%  interrupts.PMI:Performance_monitoring_interrupts
   1296392 ±  4%     -34.6%     847860 ± 22%  interrupts.RES:Rescheduling_interrupts
      1588 ± 29%     -61.8%     607.25 ± 35%  interrupts.TLB:TLB_shootdowns
     10.18            -4.0%       9.77        perf-stat.i.MPKI
 1.552e+10           -25.5%  1.157e+10        perf-stat.i.branch-instructions
  3.48e+08           -25.2%  2.605e+08        perf-stat.i.branch-misses
      1.52 ± 28%      -1.1        0.43 ± 27%  perf-stat.i.cache-miss-rate%
  11586741 ± 29%     -83.7%    1894187 ± 32%  perf-stat.i.cache-misses
 8.042e+08           -28.5%  5.749e+08        perf-stat.i.cache-references
   1492579 ±  2%     -24.4%    1127729        perf-stat.i.context-switches
      1.22           +23.5%       1.51 ±  2%  perf-stat.i.cpi
 9.616e+10            -7.9%  8.853e+10        perf-stat.i.cpu-cycles
    162.88 ±  9%     +39.8%     227.77 ±  4%  perf-stat.i.cpu-migrations
     10318 ± 43%    +758.6%      88595 ± 30%  perf-stat.i.cycles-between-cache-misses
  10338031           -26.2%    7628495        perf-stat.i.dTLB-load-misses
 2.437e+10           -25.5%  1.815e+10        perf-stat.i.dTLB-loads
      0.00 ±  5%      +0.0        0.00 ±  2%  perf-stat.i.dTLB-store-miss-rate%
     12952 ±  3%     +12.2%      14538 ±  2%  perf-stat.i.dTLB-store-misses
 1.401e+10           -25.6%  1.041e+10        perf-stat.i.dTLB-stores
     49.00           -13.4       35.64 ±  6%  perf-stat.i.iTLB-load-miss-rate%
  11130067 ±  2%     -29.9%    7798895 ±  3%  perf-stat.i.iTLB-load-misses
  11481481           +22.6%   14080265 ±  7%  perf-stat.i.iTLB-loads
 7.868e+10           -25.5%  5.862e+10        perf-stat.i.instructions
      7290 ±  2%      +6.5%       7762        perf-stat.i.instructions-per-iTLB-miss
      0.82           -18.9%       0.66 ±  2%  perf-stat.i.ipc
      2680            -2.2%       2621        perf-stat.i.minor-faults
     95.89            -3.7       92.22        perf-stat.i.node-load-miss-rate%
   1617319 ± 41%     -83.8%     262514 ± 32%  perf-stat.i.node-load-misses
     53805 ± 29%     -57.3%      22960 ± 19%  perf-stat.i.node-loads
     96.01            -7.8       88.18 ±  2%  perf-stat.i.node-store-miss-rate%
    213867 ± 31%     -86.0%      29841 ± 10%  perf-stat.i.node-store-misses
      6788 ±  7%     -22.0%       5294 ± 12%  perf-stat.i.node-stores
      2680            -2.2%       2621        perf-stat.i.page-faults
     10.22            -4.1%       9.81        perf-stat.overall.MPKI
      1.44 ± 29%      -1.1        0.33 ± 32%  perf-stat.overall.cache-miss-rate%
      1.22           +23.6%       1.51 ±  2%  perf-stat.overall.cpi
      9226 ± 34%    +470.3%      52623 ± 35%  perf-stat.overall.cycles-between-cache-misses
      0.00 ±  3%      +0.0        0.00 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     49.22           -13.5       35.72 ±  6%  perf-stat.overall.iTLB-load-miss-rate%
      7072 ±  2%      +6.3%       7520        perf-stat.overall.instructions-per-iTLB-miss
      0.82           -19.0%       0.66 ±  2%  perf-stat.overall.ipc
     96.27            -4.6       91.64        perf-stat.overall.node-load-miss-rate%
     96.53           -11.8       84.69 ±  3%  perf-stat.overall.node-store-miss-rate%
      9587            +2.7%       9844        perf-stat.overall.path-length
 1.547e+10           -25.5%  1.153e+10        perf-stat.ps.branch-instructions
 3.468e+08           -25.1%  2.596e+08        perf-stat.ps.branch-misses
  11548381 ± 29%     -83.6%    1888636 ± 32%  perf-stat.ps.cache-misses
 8.015e+08           -28.5%   5.73e+08        perf-stat.ps.cache-references
   1487561 ±  2%     -24.4%    1123929        perf-stat.ps.context-switches
 9.583e+10            -7.9%  8.823e+10        perf-stat.ps.cpu-cycles
    162.41 ±  9%     +40.1%     227.57 ±  4%  perf-stat.ps.cpu-migrations
  10302904           -26.2%    7603035        perf-stat.ps.dTLB-load-misses
 2.429e+10           -25.5%  1.809e+10        perf-stat.ps.dTLB-loads
     12914 ±  3%     +12.4%      14511 ±  2%  perf-stat.ps.dTLB-store-misses
 1.396e+10           -25.6%  1.038e+10        perf-stat.ps.dTLB-stores
  11092764 ±  2%     -29.9%    7772733 ±  3%  perf-stat.ps.iTLB-load-misses
  11443703           +22.6%   14033574 ±  7%  perf-stat.ps.iTLB-loads
 7.841e+10           -25.5%  5.843e+10        perf-stat.ps.instructions
      2671            -2.2%       2612        perf-stat.ps.minor-faults
   1611848 ± 41%     -83.8%     261634 ± 32%  perf-stat.ps.node-load-misses
     53714 ± 29%     -57.3%      22938 ± 19%  perf-stat.ps.node-loads
    213258 ± 31%     -86.0%      29755 ± 10%  perf-stat.ps.node-store-misses
      6777 ±  7%     -21.8%       5301 ± 12%  perf-stat.ps.node-stores
      2671            -2.2%       2612        perf-stat.ps.page-faults
 2.365e+13           -25.4%  1.764e+13        perf-stat.total.instructions
     37.98 ±  2%     -20.6       17.33 ±  5%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     38.08 ±  2%     -20.1       17.96 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     38.03 ±  2%     -20.1       17.91 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     40.82 ±  2%     -19.6       21.26 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     40.23 ±  2%     -19.3       20.91 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     40.24 ±  2%     -19.3       20.93 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     40.24 ±  2%     -19.3       20.94 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      0.89 ±  3%      +0.1        1.03 ±  2%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      0.62 ±  4%      +0.1        0.77 ±  3%  perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64
      0.59 ±  5%      +0.2        0.75 ±  6%  perf-profile.calltrace.cycles-pp.__inet_lookup_established.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      0.54 ±  4%      +0.2        0.73 ±  4%  perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_release.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor
      0.42 ± 57%      +0.2        0.62 ±  2%  perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_confirm.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor
      0.58 ±  5%      +0.2        0.80 ±  4%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.wait_woken
      1.74            +0.2        1.96        perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
      0.57 ±  4%      +0.2        0.79 ±  3%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      0.68 ±  2%      +0.3        0.94        perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      0.63 ±  4%      +0.3        0.89 ±  2%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.68 ±  4%      +0.3        0.93 ±  3%  perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      1.09 ±  4%      +0.3        1.37        perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
      2.04 ±  2%      +0.3        2.31 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
      0.61 ±  5%      +0.3        0.89        perf-profile.calltrace.cycles-pp.copy_page_to_iter.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
      0.63 ±  6%      +0.3        0.91 ±  3%  perf-profile.calltrace.cycles-pp.ipv4_mtu.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked
      1.91 ±  2%      +0.3        2.19 ±  2%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      0.83 ±  4%      +0.3        1.14        perf-profile.calltrace.cycles-pp.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.69 ±  2%      +0.3        1.02        perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
      2.07 ±  6%      +0.3        2.40 ±  4%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor
      0.72 ±  3%      +0.3        1.06        perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
      2.17 ±  6%      +0.4        2.53 ±  3%  perf-profile.calltrace.cycles-pp.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
      2.04            +0.4        2.41        perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
      0.84 ±  3%      +0.4        1.21 ±  4%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.87 ±  3%      +0.4        1.24 ±  4%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.15 ±  3%      +0.4        1.59 ±  3%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg
      1.03 ±  3%      +0.4        1.48 ±  3%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_woken.sk_wait_data
      1.06 ±  3%      +0.5        1.52 ±  3%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg
      0.13 ±173%      +0.5        0.60 ±  3%  perf-profile.calltrace.cycles-pp._copy_from_user.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.13 ±173%      +0.5        0.64        perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile
      1.70 ±  4%      +0.5        2.20        perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      1.25 ±  3%      +0.5        1.76 ±  3%  perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      1.38 ±  3%      +0.5        1.90 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
      0.00            +0.5        0.53        perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
      0.00            +0.5        0.53 ±  2%  perf-profile.calltrace.cycles-pp.__x86_indirect_thunk_rax
      3.01 ±  2%      +0.5        3.55        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      0.00            +0.5        0.54 ±  4%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg
      1.50 ±  3%      +0.5        2.04 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      1.25            +0.6        1.81 ±  2%  perf-profile.calltrace.cycles-pp.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
      0.00            +0.6        0.56 ±  4%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.13 ±173%      +0.6        0.69 ±  5%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout
      1.60 ±  3%      +0.6        2.16 ±  3%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.00            +0.6        0.57 ±  3%  perf-profile.calltrace.cycles-pp.fsnotify.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.15 ±  2%      +0.6        3.74        perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
      1.20 ± 21%      +0.6        1.78 ±  3%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
      1.66 ±  3%      +0.6        2.26 ±  3%  perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.00            +0.6        0.60 ±  2%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
      0.00            +0.6        0.61 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
      0.00            +0.6        0.62 ± 13%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.41            +0.6        2.03 ±  2%  perf-profile.calltrace.cycles-pp.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
      0.00            +0.6        0.64 ±  4%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      0.00            +0.7        0.68 ±  3%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg
      1.30 ±  8%      +0.7        2.00 ±  9%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage
      1.82 ±  2%      +0.7        2.52        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages
      2.09 ±  4%      +0.7        2.79        perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.33 ±  9%      +0.7        2.04 ±  9%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage.inet_sendpage
      0.98 ± 28%      +0.7        1.73        perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
      2.00            +0.8        2.77        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked
      3.62 ±  3%      +0.8        4.41        perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
      1.57 ±  9%      +0.8        2.37 ±  9%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      1.57 ±  9%      +0.8        2.38 ±  9%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      0.00            +0.9        0.90 ± 29%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit
      2.21 ±  9%      +1.1        3.30 ±  9%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
      2.56 ±  4%      +1.1        3.65        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      2.33 ±  3%      +1.1        3.43 ±  5%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established
      2.57 ±  4%      +1.1        3.68        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      4.25 ±  2%      +1.1        5.36        perf-profile.calltrace.cycles-pp.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile
      3.06 ±  2%      +1.2        4.27        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
      3.09            +1.2        4.32        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
      3.83 ±  4%      +1.4        5.18 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage
      6.54 ±  2%      +1.4        7.91        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      6.57 ±  2%      +1.4        7.96        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      2.85 ±  7%      +1.4        4.27 ±  7%  perf-profile.calltrace.cycles-pp.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
      4.88 ±  2%      +1.5        6.33        perf-profile.calltrace.cycles-pp.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      3.58            +1.5        5.04 ±  2%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
      4.26 ±  3%      +1.5        5.76 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage
      3.34            +1.5        4.88 ±  4%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
      3.74            +1.6        5.38 ±  2%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      4.67 ±  4%      +1.7        6.36 ±  2%  perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
      7.08            +2.7        9.80        perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage
      7.22 ±  2%      +2.7        9.96        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      7.26 ±  2%      +2.8       10.01        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      7.66            +2.8       10.47        perf-profile.calltrace.cycles-pp.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
      8.00 ±  2%      +3.2       11.16        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      8.20 ±  2%      +3.3       11.50        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg
      8.45 ±  2%      +3.3       11.79        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      8.32 ±  2%      +3.3       11.65        perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg
      9.22            +3.5       12.69        perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
      9.33            +3.5       12.84        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
      9.37            +3.5       12.90        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
      9.48            +3.6       13.05        perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
      9.67            +3.7       13.39        perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
      9.87            +3.9       13.76        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     10.15            +4.0       14.14        perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
     10.35            +4.1       14.43        perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     10.48            +4.2       14.65        perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     10.46            +4.2       14.64        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     10.55            +4.2       14.76        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
     10.55            +4.2       14.78        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
     15.87 ±  2%      +6.1       21.99 ±  2%  perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage
     16.60 ±  2%      +6.3       22.89 ±  2%  perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe
     16.70 ±  2%      +6.4       23.05 ±  2%  perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe
     17.05 ±  2%      +6.4       23.46 ±  2%  perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor
     17.48 ±  2%      +6.5       24.01 ±  2%  perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor
     20.28            +6.8       27.05        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     20.33            +6.8       27.13        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.76            +6.9       27.67        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.80            +6.9       27.72        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.00 ±  2%      +7.0       26.05 ±  2%  perf-profile.calltrace.cycles-pp.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct
     19.36 ±  2%      +7.1       26.49 ±  2%  perf-profile.calltrace.cycles-pp.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile
     19.67 ±  2%      +7.3       26.97 ±  2%  perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
     25.77 ±  2%      +9.3       35.08        perf-profile.calltrace.cycles-pp.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
     26.29 ±  2%      +9.5       35.75        perf-profile.calltrace.cycles-pp.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     28.65 ±  2%     +10.4       39.06        perf-profile.calltrace.cycles-pp.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.77 ±  2%     +10.7       40.51        perf-profile.calltrace.cycles-pp.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     54.03           +18.5       72.54        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     54.22           +18.5       72.75        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     37.98 ±  2%     -20.6       17.34 ±  5%  perf-profile.children.cycles-pp.intel_idle
     38.62 ±  2%     -20.4       18.26 ±  5%  perf-profile.children.cycles-pp.cpuidle_enter
     38.62 ±  2%     -20.4       18.26 ±  5%  perf-profile.children.cycles-pp.cpuidle_enter_state
     40.82 ±  2%     -19.6       21.26 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64
     40.82 ±  2%     -19.6       21.26 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
     40.82 ±  2%     -19.6       21.26 ±  4%  perf-profile.children.cycles-pp.do_idle
     40.24 ±  2%     -19.3       20.94 ±  4%  perf-profile.children.cycles-pp.start_secondary
      0.05            +0.0        0.06 ±  6%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.08            +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.tcp_update_skb_after_send
      0.11 ±  3%      +0.0        0.13 ±  5%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.07 ±  6%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.05            +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.06 ±  7%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.rb_next
      0.07 ± 10%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.10 ± 10%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.09 ±  5%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.16 ±  6%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.tcp_check_space
      0.05 ±  9%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__x86_indirect_thunk_r14
      0.11 ±  8%      +0.0        0.13 ±  3%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.06 ±  6%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.pick_next_entity
      0.09 ±  7%      +0.0        0.11 ±  3%  perf-profile.children.cycles-pp.sock_rfree
      0.05 ±  8%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.tcp_tx_timestamp
      0.12 ±  7%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp.remove_wait_queue
      0.11 ± 11%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.native_write_msr
      0.20 ±  2%      +0.0        0.22 ±  3%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.05 ±  9%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.tcp_recv_timestamp
      0.23 ±  2%      +0.0        0.25 ±  6%  perf-profile.children.cycles-pp.ip_send_check
      0.08 ±  5%      +0.0        0.11 ±  3%  perf-profile.children.cycles-pp.__x86_indirect_thunk_r10
      0.07 ±  6%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.cpus_share_cache
      0.11 ±  4%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.check_stack_object
      0.09 ±  8%      +0.0        0.12 ± 10%  perf-profile.children.cycles-pp.update_cfs_group
      0.07 ±  5%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.finish_task_switch
      0.18 ±  4%      +0.0        0.21 ±  5%  perf-profile.children.cycles-pp.generic_splice_sendpage
      0.12            +0.0        0.15 ±  2%  perf-profile.children.cycles-pp.aa_sk_perm
      0.12 ±  4%      +0.0        0.16 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.11 ±  8%      +0.0        0.14 ±  6%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.10 ±  8%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.sched_clock
      0.06            +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.tcp_event_data_recv
      0.04 ± 57%      +0.0        0.07        perf-profile.children.cycles-pp.task_tick_fair
      0.04 ± 58%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.bictcp_acked
      0.10 ±  7%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.native_sched_clock
      0.08 ±  8%      +0.0        0.12 ±  7%  perf-profile.children.cycles-pp.check_preempt_curr
      0.05 ±  8%      +0.0        0.09 ±  9%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.05            +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.kfree_skbmem
      0.06 ±  9%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.__copy_skb_header
      0.16 ±  2%      +0.0        0.20 ±  3%  perf-profile.children.cycles-pp.tcp_wfree
      0.14 ±  3%      +0.0        0.17 ±  4%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.06            +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.newidle_balance
      0.11 ±  7%      +0.0        0.15 ±  6%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.15 ±  3%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.sock_recvmsg
      0.07            +0.0        0.11        perf-profile.children.cycles-pp.inet_ehashfn
      0.07 ± 11%      +0.0        0.11 ±  3%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.tcp_chrono_stop
      0.13 ±  3%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.10 ±  5%      +0.0        0.14 ±  6%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.18 ±  4%      +0.0        0.22 ±  3%  perf-profile.children.cycles-pp.inet_send_prepare
      0.06 ±  9%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.tcp_push
      0.22 ±  3%      +0.0        0.26 ±  3%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.12 ±  4%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.09 ±  7%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp.__tcp_select_window
      0.21 ±  4%      +0.0        0.26 ±  4%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.07 ± 10%      +0.0        0.12 ±  9%  perf-profile.children.cycles-pp.netif_skb_features
      0.01 ±173%      +0.0        0.06 ± 14%  perf-profile.children.cycles-pp.raw_local_deliver
      0.21 ±  6%      +0.0        0.26 ±  6%  perf-profile.children.cycles-pp.__put_user_8
      0.05 ±  8%      +0.0        0.10        perf-profile.children.cycles-pp.___perf_sw_event
      0.32 ±  3%      +0.0        0.36        perf-profile.children.cycles-pp.tcp_rate_check_app_limited
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.default_wake_function
      0.08 ± 10%      +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.sanity
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.tcp_v4_inbound_md5_hash
      0.03 ±100%      +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.ip_queue_xmit
      0.21 ±  3%      +0.1        0.26 ±  3%  perf-profile.children.cycles-pp.aa_file_perm
      0.15 ±  5%      +0.1        0.20 ±  5%  perf-profile.children.cycles-pp.__switch_to
      0.15 ±  3%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.iov_iter_pipe
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.__usecs_to_jiffies
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.bictcp_cong_avoid
      0.07 ±  6%      +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.resched_curr
      0.15 ±  9%      +0.1        0.21 ±  3%  perf-profile.children.cycles-pp.rcu_all_qs
      0.09 ±  7%      +0.1        0.15 ± 12%  perf-profile.children.cycles-pp._find_next_bit
      0.07 ± 12%      +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.kfree
      0.07 ±  7%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.04 ± 57%      +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.apparmor_ipv4_postroute
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.06        perf-profile.children.cycles-pp._raw_spin_unlock_bh
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.tcp_options_write
      0.08 ± 10%      +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.26            +0.1        0.32 ±  4%  perf-profile.children.cycles-pp.__ip_local_out
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.eth_type_trans
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.tcp_stream_memory_free
      0.28            +0.1        0.34 ±  5%  perf-profile.children.cycles-pp.ip_local_out
      0.07 ± 10%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.ip_rcv_finish
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.ksoftirqd_running
      0.11 ±  4%      +0.1        0.17 ±  6%  perf-profile.children.cycles-pp.tcp_v4_send_check
      0.35            +0.1        0.42 ±  5%  perf-profile.children.cycles-pp.__ksize
      0.12 ±  7%      +0.1        0.18 ±  6%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.09 ±  4%      +0.1        0.16 ±  5%  perf-profile.children.cycles-pp.skb_entail
      0.24 ±  2%      +0.1        0.32 ±  4%  perf-profile.children.cycles-pp.__sk_dst_check
      0.20 ±  5%      +0.1        0.27 ±  4%  perf-profile.children.cycles-pp.sock_put
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.put_prev_task_fair
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.account_entity_dequeue
      0.09            +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.ip_rcv_core
      0.42            +0.1        0.49 ±  3%  perf-profile.children.cycles-pp.skb_release_head_state
      0.24 ±  4%      +0.1        0.31 ±  2%  perf-profile.children.cycles-pp.rw_verify_area
      0.21 ±  2%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.ipv4_dst_check
      0.44            +0.1        0.51 ±  3%  perf-profile.children.cycles-pp.skb_release_all
      0.04 ± 57%      +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.01 ±173%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.23 ±  6%      +0.1        0.30 ±  2%  perf-profile.children.cycles-pp.update_curr
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.__tcp_ack_snd_check
      0.17 ±  7%      +0.1        0.25 ±  5%  perf-profile.children.cycles-pp.skb_clone
      0.11 ± 17%      +0.1        0.19 ±  3%  perf-profile.children.cycles-pp.xas_start
      0.17 ±  4%      +0.1        0.25 ±  5%  perf-profile.children.cycles-pp.kmem_cache_free
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.tcp_rack_advance
      0.55 ±  4%      +0.1        0.64 ±  2%  perf-profile.children.cycles-pp.page_cache_pipe_buf_confirm
      0.12 ±  4%      +0.1        0.21 ±  3%  perf-profile.children.cycles-pp.timestamp_truncate
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.rb_first
      0.30 ±  8%      +0.1        0.39 ±  2%  perf-profile.children.cycles-pp._cond_resched
      0.16 ±  2%      +0.1        0.25 ±  7%  perf-profile.children.cycles-pp.tcp_add_backlog
      0.17 ±  4%      +0.1        0.26 ±  9%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.38            +0.1        0.47 ±  5%  perf-profile.children.cycles-pp.__skb_clone
      0.17 ±  2%      +0.1        0.26 ±  4%  perf-profile.children.cycles-pp.ktime_get_with_offset
      0.12 ±  4%      +0.1        0.22 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.tcp_cleanup_rbuf
      0.11 ±  6%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.10 ± 11%  perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb
      0.37            +0.1        0.47 ±  3%  perf-profile.children.cycles-pp.dst_release
      0.25 ±  3%      +0.1        0.35 ±  2%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.17 ±  6%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.__ip_finish_output
      0.11 ±  6%      +0.1        0.22 ±  8%  perf-profile.children.cycles-pp.nf_hook_slow
      0.36 ±  3%      +0.1        0.47 ±  4%  perf-profile.children.cycles-pp.set_next_entity
      0.20 ±  2%      +0.1        0.31 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.12 ±  5%      +0.1        0.23 ±  4%  perf-profile.children.cycles-pp.update_process_times
      0.00            +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.rcu_core
      0.04 ± 57%      +0.1        0.15 ± 13%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.26 ±  4%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.49 ±  3%      +0.1        0.61 ±  3%  perf-profile.children.cycles-pp._copy_from_user
      0.12 ±  5%      +0.1        0.24 ±  5%  perf-profile.children.cycles-pp.tick_sched_handle
      0.27 ±  8%      +0.1        0.39        perf-profile.children.cycles-pp.xas_load
      0.24 ±  8%      +0.1        0.37 ±  4%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.21 ±  2%      +0.1        0.34        perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.48 ±  4%      +0.1        0.60        perf-profile.children.cycles-pp.__fget_light
      0.21 ±  3%      +0.1        0.34 ±  5%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.23 ±  3%      +0.1        0.36 ±  6%  perf-profile.children.cycles-pp.__kmalloc_reserve
      0.32 ±  3%      +0.1        0.46 ±  6%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.32 ±  4%      +0.1        0.45 ±  5%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.10 ± 11%      +0.1        0.24 ±  8%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.18 ± 10%      +0.1        0.32 ±  2%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.12 ± 10%      +0.1        0.27 ± 14%  perf-profile.children.cycles-pp.clockevents_program_event
      0.53            +0.1        0.68        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.31 ±  3%      +0.1        0.45 ±  4%  perf-profile.children.cycles-pp.tcp_established_options
      0.14 ±  5%      +0.1        0.29 ±  5%  perf-profile.children.cycles-pp.tick_sched_timer
      0.10 ±  4%      +0.1        0.25 ±  5%  perf-profile.children.cycles-pp.tcp_release_cb
      0.35 ±  2%      +0.2        0.50        perf-profile.children.cycles-pp.__might_sleep
      0.30 ±  8%      +0.2        0.45 ±  7%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.34 ±  2%      +0.2        0.50 ±  6%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.41 ±  5%      +0.2        0.57 ±  4%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.29 ±  2%      +0.2        0.46        perf-profile.children.cycles-pp.__fsnotify_parent
      0.26 ±  2%      +0.2        0.43 ±  4%  perf-profile.children.cycles-pp.__slab_free
      0.34 ±  5%      +0.2        0.52 ±  2%  perf-profile.children.cycles-pp.mod_timer
      0.08 ± 13%      +0.2        0.26 ± 15%  perf-profile.children.cycles-pp.irq_exit
      0.18 ±  8%      +0.2        0.36 ±  3%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.33 ±  4%      +0.2        0.51 ±  4%  perf-profile.children.cycles-pp.current_time
      0.35 ±  5%      +0.2        0.54 ±  2%  perf-profile.children.cycles-pp.sk_reset_timer
      0.55 ±  4%      +0.2        0.73 ±  4%  perf-profile.children.cycles-pp.page_cache_pipe_buf_release
      0.31 ±  6%      +0.2        0.51 ±  2%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.78 ±  4%      +0.2        0.97 ±  3%  perf-profile.children.cycles-pp.__inet_lookup_established
      1.16            +0.2        1.37 ±  2%  perf-profile.children.cycles-pp.skb_release_data
      0.44            +0.2        0.65        perf-profile.children.cycles-pp.netif_rx_internal
      0.45 ±  2%      +0.2        0.66 ±  2%  perf-profile.children.cycles-pp.netif_rx
      0.48 ±  5%      +0.2        0.70 ±  5%  perf-profile.children.cycles-pp.dequeue_entity
      0.48 ±  3%      +0.2        0.70 ±  3%  perf-profile.children.cycles-pp.__might_fault
      0.59 ±  5%      +0.2        0.81 ±  3%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.76            +0.2        0.99 ±  4%  perf-profile.children.cycles-pp.read_tsc
      0.53 ±  2%      +0.2        0.76 ±  4%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.77            +0.2        2.00        perf-profile.children.cycles-pp.__check_object_size
      0.48 ±  4%      +0.2        0.72 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      1.24 ±  4%      +0.2        1.48        perf-profile.children.cycles-pp.common_file_perm
      0.69 ±  2%      +0.2        0.94        perf-profile.children.cycles-pp.do_splice_to
      0.65 ±  5%      +0.3        0.90 ±  2%  perf-profile.children.cycles-pp.menu_select
      0.49 ±  5%      +0.3        0.76 ±  2%  perf-profile.children.cycles-pp.fsnotify
      0.62 ±  5%      +0.3        0.90        perf-profile.children.cycles-pp.copy_page_to_iter
      0.63 ±  2%      +0.3        0.90        perf-profile.children.cycles-pp._raw_spin_lock
      2.04 ±  2%      +0.3        2.32 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.42 ± 10%      +0.3        0.71 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.44 ±  4%      +0.3        0.74 ±  4%  perf-profile.children.cycles-pp.update_load_avg
      0.89 ±  2%      +0.3        1.20        perf-profile.children.cycles-pp.___might_sleep
      0.57 ±  3%      +0.3        0.90 ±  4%  perf-profile.children.cycles-pp.enqueue_task_fair
      2.17 ±  2%      +0.3        2.49        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.58 ±  3%      +0.3        0.93 ±  3%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.58 ±  3%      +0.3        0.92 ±  3%  perf-profile.children.cycles-pp.activate_task
      0.53 ±  4%      +0.3        0.87 ±  2%  perf-profile.children.cycles-pp.atime_needs_update
      0.72 ±  2%      +0.3        1.07        perf-profile.children.cycles-pp.sk_stream_alloc_skb
      2.08 ±  6%      +0.3        2.42 ±  4%  perf-profile.children.cycles-pp.find_get_entry
      0.77 ±  6%      +0.4        1.13 ±  3%  perf-profile.children.cycles-pp.ipv4_mtu
      2.06            +0.4        2.42        perf-profile.children.cycles-pp.simple_copy_to_iter
      2.19 ±  6%      +0.4        2.55 ±  3%  perf-profile.children.cycles-pp.pagecache_get_page
      0.88 ±  3%      +0.4        1.25 ±  3%  perf-profile.children.cycles-pp.schedule_idle
      0.96 ±  2%      +0.4        1.37        perf-profile.children.cycles-pp.loopback_xmit
      1.16 ±  3%      +0.4        1.59 ±  3%  perf-profile.children.cycles-pp.schedule_timeout
      1.01 ±  2%      +0.4        1.46        perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.98            +0.4        1.43 ±  2%  perf-profile.children.cycles-pp.__alloc_skb
      0.66 ±  4%      +0.5        1.12 ±  2%  perf-profile.children.cycles-pp.touch_atime
      1.06 ±  3%      +0.5        1.52 ±  3%  perf-profile.children.cycles-pp.schedule
      1.32 ±  3%      +0.5        1.78 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      0.98 ±  2%      +0.5        1.45        perf-profile.children.cycles-pp.ktime_get
      1.84            +0.5        2.31 ±  2%  perf-profile.children.cycles-pp.__kfree_skb
      0.77 ±  2%      +0.5        1.25        perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.56 ± 11%      +0.5        1.04 ±  7%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.65 ± 12%      +0.5        1.12 ±  6%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.25 ±  3%      +0.5        1.76 ±  3%  perf-profile.children.cycles-pp.wait_woken
      1.38 ±  3%      +0.5        1.91 ±  3%  perf-profile.children.cycles-pp.__wake_up_common
      1.49 ±  3%      +0.6        2.04 ±  2%  perf-profile.children.cycles-pp.__wake_up_common_lock
      1.77 ±  3%      +0.6        2.33        perf-profile.children.cycles-pp.security_file_permission
      3.19 ±  2%      +0.6        3.78        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      1.66 ±  3%      +0.6        2.25 ±  2%  perf-profile.children.cycles-pp.sock_def_readable
      3.17 ±  2%      +0.6        3.75        perf-profile.children.cycles-pp.copyout
      1.67 ±  3%      +0.6        2.27 ±  3%  perf-profile.children.cycles-pp.sk_wait_data
      1.36            +0.6        1.97 ±  2%  perf-profile.children.cycles-pp.tcp_current_mss
      1.41            +0.6        2.05 ±  2%  perf-profile.children.cycles-pp.tcp_send_mss
      2.01 ±  2%      +0.7        2.66 ±  2%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      1.54 ±  2%      +0.7        2.21        perf-profile.children.cycles-pp.__dev_queue_xmit
      3.65 ±  3%      +0.8        4.44        perf-profile.children.cycles-pp._copy_to_iter
      1.91 ±  3%      +0.8        2.73 ±  3%  perf-profile.children.cycles-pp.__sched_text_start
      2.45 ±  2%      +0.9        3.35 ±  2%  perf-profile.children.cycles-pp.tcp_ack
      4.28 ±  2%      +1.1        5.41        perf-profile.children.cycles-pp.generic_file_read_iter
      2.78 ±  7%      +1.3        4.10 ±  7%  perf-profile.children.cycles-pp.__release_sock
      6.55 ±  2%      +1.4        7.93        perf-profile.children.cycles-pp.__skb_datagram_iter
      6.57 ±  2%      +1.4        7.96        perf-profile.children.cycles-pp.skb_copy_datagram_iter
      3.92 ±  4%      +1.4        5.33 ±  3%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      4.88 ±  2%      +1.5        6.34        perf-profile.children.cycles-pp.generic_file_splice_read
      3.58 ±  6%      +1.7        5.26 ±  6%  perf-profile.children.cycles-pp.release_sock
      4.98 ±  3%      +1.7        6.70 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      4.96 ±  3%      +1.8        6.72 ±  2%  perf-profile.children.cycles-pp.lock_sock_nested
      7.11            +2.7        9.84        perf-profile.children.cycles-pp.do_tcp_sendpages
      7.67            +2.8       10.50        perf-profile.children.cycles-pp.tcp_sendpage_locked
      7.20            +3.1       10.33 ±  2%  perf-profile.children.cycles-pp.tcp_write_xmit
      7.25            +3.1       10.39 ±  2%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
      9.24            +3.5       12.71        perf-profile.children.cycles-pp.tcp_v4_rcv
      9.33            +3.5       12.84        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      9.37            +3.5       12.90        perf-profile.children.cycles-pp.ip_local_deliver_finish
      9.49            +3.6       13.07        perf-profile.children.cycles-pp.ip_local_deliver
      8.91            +3.6       12.53 ±  2%  perf-profile.children.cycles-pp.tcp_rcv_established
      9.07            +3.7       12.73 ±  2%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      9.67            +3.7       13.41        perf-profile.children.cycles-pp.ip_rcv
      9.88            +3.9       13.77        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     10.16            +4.0       14.16        perf-profile.children.cycles-pp.process_backlog
     10.36            +4.1       14.44        perf-profile.children.cycles-pp.net_rx_action
     10.48            +4.2       14.66        perf-profile.children.cycles-pp.do_softirq_own_stack
     10.56            +4.2       14.79        perf-profile.children.cycles-pp.do_softirq
     10.52            +4.3       14.84        perf-profile.children.cycles-pp.__softirqentry_text_start
     10.80            +4.3       15.14        perf-profile.children.cycles-pp.__local_bh_enable_ip
     11.79            +4.8       16.56        perf-profile.children.cycles-pp.ip_finish_output2
     12.06            +5.0       17.04        perf-profile.children.cycles-pp.ip_output
     12.61            +5.1       17.74        perf-profile.children.cycles-pp.__ip_queue_xmit
     13.35            +5.5       18.82        perf-profile.children.cycles-pp.__tcp_transmit_skb
     15.91 ±  2%      +6.1       22.05 ±  2%  perf-profile.children.cycles-pp.tcp_sendpage
     16.62 ±  2%      +6.3       22.92 ±  2%  perf-profile.children.cycles-pp.inet_sendpage
     16.71 ±  2%      +6.4       23.07 ±  2%  perf-profile.children.cycles-pp.kernel_sendpage
     17.06 ±  2%      +6.4       23.48 ±  2%  perf-profile.children.cycles-pp.sock_sendpage
     17.48 ±  2%      +6.5       24.02 ±  2%  perf-profile.children.cycles-pp.pipe_to_sendpage
     20.29            +6.8       27.07        perf-profile.children.cycles-pp.tcp_recvmsg
     20.33            +6.8       27.13        perf-profile.children.cycles-pp.inet_recvmsg
     20.76            +6.9       27.67        perf-profile.children.cycles-pp.__sys_recvfrom
     20.81            +6.9       27.73        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     19.04 ±  2%      +7.1       26.10 ±  2%  perf-profile.children.cycles-pp.__splice_from_pipe
     19.37 ±  2%      +7.1       26.51 ±  2%  perf-profile.children.cycles-pp.splice_from_pipe
     19.68 ±  2%      +7.3       26.97 ±  2%  perf-profile.children.cycles-pp.direct_splice_actor
     25.77 ±  2%      +9.3       35.09        perf-profile.children.cycles-pp.splice_direct_to_actor
     26.31 ±  2%      +9.5       35.77        perf-profile.children.cycles-pp.do_splice_direct
     28.69 ±  2%     +10.4       39.12        perf-profile.children.cycles-pp.do_sendfile
     29.79 ±  2%     +10.7       40.54        perf-profile.children.cycles-pp.__x64_sys_sendfile64
     54.10           +18.5       72.61        perf-profile.children.cycles-pp.do_syscall_64
     54.27           +18.5       72.80        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     37.98 ±  2%     -20.6       17.34 ±  5%  perf-profile.self.cycles-pp.intel_idle
      0.10 ±  4%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.schedule_timeout
      0.09            +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.check_stack_object
      0.06            +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.finish_task_switch
      0.08 ±  5%      +0.0        0.10 ± 10%  perf-profile.self.cycles-pp.tcp_rcv_space_adjust
      0.06 ±  6%      +0.0        0.08        perf-profile.self.cycles-pp.pick_next_entity
      0.06 ±  9%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.do_softirq
      0.05 ±  9%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.tcp_recv_timestamp
      0.11 ±  6%      +0.0        0.13 ±  5%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.09 ±  7%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.sock_rfree
      0.08 ±  5%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.__x86_indirect_thunk_r10
      0.17 ±  4%      +0.0        0.19 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.10 ±  5%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.__sk_dst_check
      0.11 ± 11%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.native_write_msr
      0.10 ±  5%      +0.0        0.12 ±  5%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.05            +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.11 ±  4%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.10 ±  9%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.08 ±  6%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.do_splice_direct
      0.07 ± 10%      +0.0        0.10 ± 11%  perf-profile.self.cycles-pp.kmem_cache_free
      0.07 ±  6%      +0.0        0.10 ±  9%  perf-profile.self.cycles-pp.cpus_share_cache
      0.09 ±  9%      +0.0        0.12 ±  7%  perf-profile.self.cycles-pp.update_cfs_group
      0.22 ±  3%      +0.0        0.25 ±  4%  perf-profile.self.cycles-pp.ip_send_check
      0.17 ±  4%      +0.0        0.20 ±  3%  perf-profile.self.cycles-pp.inet_send_prepare
      0.09 ±  4%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.aa_sk_perm
      0.10 ±  4%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.native_sched_clock
      0.06            +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.do_splice_to
      0.11 ±  7%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp.pagecache_get_page
      0.15 ±  5%      +0.0        0.18        perf-profile.self.cycles-pp._cond_resched
      0.05            +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.kfree_skbmem
      0.15 ±  3%      +0.0        0.19 ±  4%  perf-profile.self.cycles-pp.tcp_wfree
      0.15 ±  4%      +0.0        0.18 ±  5%  perf-profile.self.cycles-pp.generic_splice_sendpage
      0.05 ±  9%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.ktime_get_with_offset
      0.05 ±  8%      +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.04 ± 57%      +0.0        0.07 ± 10%  perf-profile.self.cycles-pp._copy_from_user
      0.04 ± 57%      +0.0        0.07        perf-profile.self.cycles-pp.validate_xmit_skb
      0.04 ± 58%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.rb_next
      0.07 ± 10%      +0.0        0.10 ± 10%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.05 ±  8%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.16 ±  2%      +0.0        0.20 ±  5%  perf-profile.self.cycles-pp.sock_def_readable
      0.07 ±  7%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.copyout
      0.06            +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.newidle_balance
      0.04 ± 57%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.skb_release_head_state
      0.04 ± 58%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.bictcp_acked
      0.05            +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.tcp_push
      0.05 ±  8%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.tcp_event_data_recv
      0.05            +0.0        0.09 ± 14%  perf-profile.self.cycles-pp.nf_hook_slow
      0.06 ±  9%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.__copy_skb_header
      0.06 ±  6%      +0.0        0.10        perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.07 ±  6%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.inet_ehashfn
      0.11 ±  4%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.ip_local_deliver
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.raw_local_deliver
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.netif_skb_features
      0.12 ±  4%      +0.0        0.17 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      0.09 ±  7%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.__tcp_select_window
      0.20 ±  4%      +0.0        0.24 ±  2%  perf-profile.self.cycles-pp.set_next_entity
      0.08 ±  8%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.skb_entail
      0.07 ±  6%      +0.0        0.11 ±  3%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.32 ±  3%      +0.0        0.36        perf-profile.self.cycles-pp.tcp_rate_check_app_limited
      0.23 ±  4%      +0.0        0.28 ±  6%  perf-profile.self.cycles-pp.security_file_permission
      0.21 ±  6%      +0.0        0.26 ±  4%  perf-profile.self.cycles-pp.__put_user_8
      0.01 ±173%      +0.0        0.06        perf-profile.self.cycles-pp.__x86_indirect_thunk_r14
      0.24 ±  2%      +0.0        0.29 ±  6%  perf-profile.self.cycles-pp.tcp_sendpage_locked
      0.10 ±  8%      +0.0        0.15 ±  2%  perf-profile.self.cycles-pp.rcu_all_qs
      0.04 ± 58%      +0.0        0.09        perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.06 ±  9%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.21 ±  3%      +0.1        0.26 ±  2%  perf-profile.self.cycles-pp.aa_file_perm
      0.07 ±  5%      +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.tcp_event_new_data_sent
      0.09 ±  5%      +0.1        0.14 ±  9%  perf-profile.self.cycles-pp.kernel_sendpage
      0.01 ±173%      +0.1        0.07 ± 13%  perf-profile.self.cycles-pp.inet_recvmsg
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp._raw_spin_unlock_bh
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.iov_iter_pipe
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.bictcp_cong_avoid
      0.33            +0.1        0.38 ±  6%  perf-profile.self.cycles-pp.__skb_clone
      0.17            +0.1        0.23 ±  3%  perf-profile.self.cycles-pp.net_rx_action
      0.14 ±  7%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.__switch_to
      0.07 ±  6%      +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.sanity
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.tcp_rack_advance
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.tcp_chrono_stop
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__tcp_send_ack
      0.08 ±  5%      +0.1        0.13 ±  6%  perf-profile.self.cycles-pp.__ip_finish_output
      0.15 ±  4%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.xas_load
      0.15 ±  4%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.09 ±  7%      +0.1        0.15 ± 15%  perf-profile.self.cycles-pp._find_next_bit
      0.14 ± 14%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.lock_sock_nested
      0.07 ±  7%      +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.07 ±  7%      +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.kfree
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.tcp_options_write
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.resched_curr
      0.16 ±  5%      +0.1        0.22        perf-profile.self.cycles-pp.process_backlog
      0.15 ±  5%      +0.1        0.21 ±  7%  perf-profile.self.cycles-pp.release_sock
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.__ip_local_out
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.tcp_tx_timestamp
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.netif_rx_internal
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.tcp_stream_memory_free
      0.10 ±  4%      +0.1        0.16 ±  6%  perf-profile.self.cycles-pp.tcp_v4_send_check
      0.13 ±  8%      +0.1        0.19 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.eth_type_trans
      0.09 ±  7%      +0.1        0.15 ±  5%  perf-profile.self.cycles-pp.ip_output
      0.35            +0.1        0.42 ±  5%  perf-profile.self.cycles-pp.__ksize
      0.11 ± 17%      +0.1        0.17 ±  4%  perf-profile.self.cycles-pp.xas_start
      0.20 ±  2%      +0.1        0.26 ±  3%  perf-profile.self.cycles-pp.ipv4_dst_check
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.ksoftirqd_running
      0.20 ±  3%      +0.1        0.27 ±  5%  perf-profile.self.cycles-pp.sock_put
      0.17 ±  7%      +0.1        0.24 ±  5%  perf-profile.self.cycles-pp.menu_select
      0.09            +0.1        0.16 ±  5%  perf-profile.self.cycles-pp.ip_rcv_core
      0.00            +0.1        0.07 ± 16%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.14 ±  5%      +0.1        0.21 ±  6%  perf-profile.self.cycles-pp.enqueue_entity
      0.15 ±  7%      +0.1        0.22 ± 12%  perf-profile.self.cycles-pp.__might_fault
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.__tcp_push_pending_frames
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.__tcp_ack_snd_check
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.ip_queue_xmit
      0.01 ±173%      +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.53 ±  5%      +0.1        0.60 ±  2%  perf-profile.self.cycles-pp.page_cache_pipe_buf_confirm
      0.22 ±  5%      +0.1        0.29 ±  2%  perf-profile.self.cycles-pp.rw_verify_area
      0.13 ±  6%      +0.1        0.21 ±  3%  perf-profile.self.cycles-pp.mod_timer
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.rb_first
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.tcp_mstamp_refresh
      0.29            +0.1        0.36        perf-profile.self.cycles-pp.sock_sendpage
      0.28 ±  2%      +0.1        0.36 ±  4%  perf-profile.self.cycles-pp.__x64_sys_sendfile64
      0.11 ±  4%      +0.1        0.19 ±  4%  perf-profile.self.cycles-pp.timestamp_truncate
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.apparmor_ipv4_postroute
      0.16 ± 10%      +0.1        0.24 ±  3%  perf-profile.self.cycles-pp.skb_clone
      0.14 ±  3%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.10 ±  8%      +0.1        0.18 ±  8%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.___perf_sw_event
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.tcp_rearm_rto
      0.14            +0.1        0.22 ±  3%  perf-profile.self.cycles-pp.tcp_send_mss
      0.13 ±  3%      +0.1        0.22 ±  6%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.08 ±  5%      +0.1        0.17 ±  9%  perf-profile.self.cycles-pp.wait_woken
      0.11 ±  6%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.13 ±  6%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.current_time
      0.00            +0.1        0.09 ± 14%  perf-profile.self.cycles-pp.apparmor_socket_sock_rcv_skb
      0.12 ±  3%      +0.1        0.21 ± 11%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.tcp_cleanup_rbuf
      0.52            +0.1        0.61        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.15 ±  8%      +0.1        0.24 ±  4%  perf-profile.self.cycles-pp.direct_splice_actor
      0.36            +0.1        0.45 ±  4%  perf-profile.self.cycles-pp.dst_release
      0.16 ±  5%      +0.1        0.25        perf-profile.self.cycles-pp.loopback_xmit
      0.10 ±  8%      +0.1        0.19 ±  7%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.40            +0.1        0.50 ±  2%  perf-profile.self.cycles-pp.pipe_to_sendpage
      0.26 ±  4%      +0.1        0.36 ±  2%  perf-profile.self.cycles-pp.splice_from_pipe
      0.35 ±  5%      +0.1        0.46 ±  4%  perf-profile.self.cycles-pp.__sched_text_start
      0.28 ±  2%      +0.1        0.39 ±  2%  perf-profile.self.cycles-pp.simple_copy_to_iter
      0.13 ±  5%      +0.1        0.24 ±  5%  perf-profile.self.cycles-pp.touch_atime
      0.26 ±  7%      +0.1        0.37        perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.28            +0.1        0.39 ±  3%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.27 ±  7%      +0.1        0.38 ±  3%  perf-profile.self.cycles-pp.ip_finish_output2
      0.40 ±  2%      +0.1        0.52 ±  3%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.35 ±  2%      +0.1        0.47 ±  4%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.44 ±  4%      +0.1        0.56        perf-profile.self.cycles-pp.tcp_sendpage
      0.48 ±  3%      +0.1        0.60 ±  2%  perf-profile.self.cycles-pp.inet_sendpage
      0.29 ±  4%      +0.1        0.41 ±  2%  perf-profile.self.cycles-pp._copy_to_iter
      0.21 ±  6%      +0.1        0.34 ±  2%  perf-profile.self.cycles-pp.update_load_avg
      0.46 ±  4%      +0.1        0.59        perf-profile.self.cycles-pp.__fget_light
      0.38 ±  2%      +0.1        0.51 ±  3%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.41            +0.1        0.55 ±  3%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.20 ±  4%      +0.1        0.33 ±  3%  perf-profile.self.cycles-pp.__alloc_skb
      0.35 ±  3%      +0.1        0.49 ±  2%  perf-profile.self.cycles-pp.generic_file_splice_read
      0.29 ±  4%      +0.1        0.43 ±  4%  perf-profile.self.cycles-pp.tcp_established_options
      0.24 ±  3%      +0.1        0.38 ±  4%  perf-profile.self.cycles-pp.tcp_ack
      0.09 ±  4%      +0.1        0.23 ±  6%  perf-profile.self.cycles-pp.tcp_release_cb
      0.33            +0.1        0.47 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      0.38 ±  4%      +0.1        0.52 ±  4%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.17 ±  8%      +0.1        0.32 ±  2%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.60            +0.1        0.75        perf-profile.self.cycles-pp.__skb_datagram_iter
      0.28 ±  3%      +0.1        0.43 ±  2%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.28 ±  7%      +0.2        0.44 ±  7%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.15 ± 12%      +0.2        0.30 ±  5%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.70 ±  4%      +0.2        0.86 ±  4%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.20 ±  6%      +0.2        0.36 ±  4%  perf-profile.self.cycles-pp.atime_needs_update
      0.52 ±  3%      +0.2        0.69 ±  3%  perf-profile.self.cycles-pp.page_cache_pipe_buf_release
      0.25 ± 11%      +0.2        0.42 ±  3%  perf-profile.self.cycles-pp.splice_direct_to_actor
      0.26 ±  3%      +0.2        0.43 ±  4%  perf-profile.self.cycles-pp.__slab_free
      0.45 ±  9%      +0.2        0.63 ±  5%  perf-profile.self.cycles-pp.tcp_current_mss
      1.00 ±  5%      +0.2        1.20        perf-profile.self.cycles-pp.common_file_perm
      0.33 ±  3%      +0.2        0.53 ±  3%  perf-profile.self.cycles-pp.__splice_from_pipe
      0.73 ±  2%      +0.2        0.93 ±  4%  perf-profile.self.cycles-pp.read_tsc
      1.15            +0.2        1.36        perf-profile.self.cycles-pp.skb_release_data
      1.77 ±  5%      +0.2        1.98 ±  4%  perf-profile.self.cycles-pp.find_get_entry
      0.54 ±  5%      +0.2        0.77        perf-profile.self.cycles-pp.copy_page_to_iter
      0.51 ±  4%      +0.2        0.75        perf-profile.self.cycles-pp.do_sendfile
      0.59            +0.3        0.84 ±  3%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.47 ±  3%      +0.3        0.73        perf-profile.self.cycles-pp.fsnotify
      0.93 ±  3%      +0.3        1.19        perf-profile.self.cycles-pp.generic_file_read_iter
      0.59            +0.3        0.85        perf-profile.self.cycles-pp._raw_spin_lock
      2.04 ±  2%      +0.3        2.32 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.34 ±  6%      +0.3        0.63 ±  5%  perf-profile.self.cycles-pp.ktime_get
      0.86 ±  2%      +0.3        1.15        perf-profile.self.cycles-pp.___might_sleep
      2.17 ±  2%      +0.3        2.49 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.74 ±  6%      +0.3        1.07 ±  4%  perf-profile.self.cycles-pp.ipv4_mtu
      1.08            +0.3        1.42        perf-profile.self.cycles-pp._raw_spin_lock_bh
      1.67 ±  2%      +0.3        2.01 ±  2%  perf-profile.self.cycles-pp.do_tcp_sendpages
      0.81 ±  2%      +0.4        1.18 ±  2%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.72 ±  3%      +0.4        1.16        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      3.17 ±  2%      +0.6        3.75        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      3.36            +0.8        4.15        perf-profile.self.cycles-pp.do_syscall_64
      3.89 ±  4%      +1.4        5.29 ±  4%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                               netperf.Throughput_Mbps                          
                                                                                
  20000 +-------------------------------------------------------------------+   
  18000 |.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.|   
        |                                                                   |   
  16000 |-+                                                                 |   
  14000 |-O    O                          O O      O O                      |   
        |          O    O   O  O O O O  O      O O      O O                 |   
  12000 |-+                                                                 |   
  10000 |-+                                                                 |   
   8000 |-+                                                                 |   
        |                                                                   |   
   6000 |-+                                                                 |   
   4000 |-+                                                                 |   
        |                                                                   |   
   2000 |-+                                                                 |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            netperf.Throughput_total_Mbps                       
                                                                                
  350000 +------------------------------------------------------------------+   
         |.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+.+.+..+.+   +..+.+.+    +.+.|   
  300000 |-+                                                                |   
         |                                                                  |   
  250000 |-O    O                   O      O O O O O  O O O                 |   
         |          O    O   O O  O   O O                                   |   
  200000 |-+                                                                |   
         |                                                                  |   
  150000 |-+                                                                |   
         |                                                                  |   
  100000 |-+                                                                |   
         |                                                                  |   
   50000 |-+                                                                |   
         |                                                                  |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   netperf.workload                             
                                                                                
  2.5e+09 +-----------------------------------------------------------------+   
          |            +.+.                                                 |   
          |                                                                 |   
    2e+09 |-+                                                               |   
          | O    O       O          O  O O O O O  O O O O O                 |   
          |          O        O O O                                         |   
  1.5e+09 |-+                                                               |   
          |                                                                 |   
    1e+09 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    5e+08 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              netperf.time.user_time                            
                                                                                
  700 +---------------------------------------------------------------------+   
      | O    O                    +      O O    O O  O   O                  |   
  600 |-+         O   O    O O  O O O  O      O        O                    |   
      |                                                                     |   
  500 |-+                                                                   |   
      |                                                                     |   
  400 |-+                                                                   |   
      |                                                                     |   
  300 |-+                                                                   |   
      |                                                                     |   
  200 |-+                                                                   |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             netperf.time.system_time                           
                                                                                
  5000 +--------------------------------------------------------------------+   
  4500 |.+..+.+.+..+.+.+.+..+.+.+..+.+.+..+.+.+.+..+.+.+..+.+.+..+.+.+.+..+.|   
       |                                                                    |   
  4000 |-+                                                                  |   
  3500 |-+                                                                  |   
       |                                                                    |   
  3000 |-+                                                                  |   
  2500 |-+                                                                  |   
  2000 |-+                                                                  |   
       |                                                                    |   
  1500 |-+                                                                  |   
  1000 |-+                                                                  |   
       |                                                                    |   
   500 |-+                                                                  |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-bdw-ex2: 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/25%/debian-x86_64-20191114.cgz/300s/10K/lkp-bdw-ex2/SCTP_STREAM_MANY/netperf/0xb000038

commit: 
  9c40365a65 ("threads: Update PID limit comment according to futex UAPI change")
  59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group")

9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      8310           -31.1%       5723 ±  8%  netperf.Throughput_Mbps
    332426           -31.1%     228942 ±  8%  netperf.Throughput_total_Mbps
     13748 ±  2%     -15.8%      11583 ±  7%  netperf.time.minor_page_faults
    587.05           -26.0%     434.31 ±  6%  netperf.time.user_time
 1.217e+09           -31.1%  8.384e+08 ±  8%  netperf.workload
    763112 ±  5%     +19.0%     908201 ±  4%  meminfo.Committed_AS
  10662925 ± 10%     +16.3%   12404278        meminfo.DirectMap2M
      3.24            -0.8        2.48 ± 11%  mpstat.cpu.all.soft%
      2.02            -0.5        1.50 ±  6%  mpstat.cpu.all.usr%
     72.00            +2.1%      73.50        vmstat.cpu.id
   2290192           -31.3%    1573919 ±  8%  vmstat.system.cs
  6.88e+09 ±  6%     -54.1%  3.157e+09 ± 16%  cpuidle.C1.time
 3.412e+08 ±  2%     -50.4%  1.691e+08 ± 17%  cpuidle.C1.usage
  45203190 ± 48%    +165.0%  1.198e+08 ± 22%  cpuidle.C1E.usage
     25963            -4.1%      24904        proc-vmstat.nr_slab_reclaimable
     66104            -6.4%      61871        proc-vmstat.nr_slab_unreclaimable
      9019 ± 34%     -77.7%       2009 ± 72%  proc-vmstat.numa_hint_faults_local
 1.565e+09           -31.1%  1.078e+09 ±  8%  proc-vmstat.numa_hit
 1.565e+09           -31.1%  1.078e+09 ±  8%  proc-vmstat.numa_local
 9.044e+09           -31.1%  6.228e+09 ±  8%  proc-vmstat.pgalloc_normal
 9.044e+09           -31.1%  6.228e+09 ±  8%  proc-vmstat.pgfree
     39376 ± 22%     -56.0%      17319 ± 38%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   3360482 ± 13%     -57.8%    1416646 ± 30%  sched_debug.cfs_rq:/.MIN_vruntime.max
    333186 ± 18%     -56.4%     145249 ± 31%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     39376 ± 22%     -56.0%      17319 ± 38%  sched_debug.cfs_rq:/.max_vruntime.avg
   3360482 ± 13%     -57.8%    1416646 ± 30%  sched_debug.cfs_rq:/.max_vruntime.max
    333186 ± 18%     -56.4%     145249 ± 31%  sched_debug.cfs_rq:/.max_vruntime.stddev
     54158 ± 49%     -81.3%      10104 ± 32%  sched_debug.cfs_rq:/.min_vruntime.min
   1726678 ±  2%     +32.7%    2291681 ± 13%  sched_debug.cfs_rq:/.min_vruntime.stddev
      1387 ±  7%     +21.8%       1689 ± 12%  sched_debug.cfs_rq:/.runnable_avg.max
    460.02 ±  2%     +11.2%     511.65 ±  4%  sched_debug.cfs_rq:/.runnable_avg.stddev
   1726688 ±  2%     +32.7%    2291718 ± 13%  sched_debug.cfs_rq:/.spread0.stddev
     14277 ± 28%     -30.3%       9949 ±  4%  sched_debug.cpu.avg_idle.min
   1784858           -34.9%    1161189 ±  4%  sched_debug.cpu.nr_switches.avg
   6522465 ± 11%     +32.0%    8612184 ± 16%  sched_debug.cpu.nr_switches.max
      2464 ± 11%     -21.9%       1924 ±  2%  sched_debug.cpu.nr_switches.min
      7868 ±  7%     +10.8%       8717 ±  5%  numa-vmstat.node0.nr_kernel_stack
    677.50 ± 50%     +88.2%       1274 ± 29%  numa-vmstat.node0.nr_page_table_pages
      5910 ± 22%     +56.5%       9251 ±  7%  numa-vmstat.node0.nr_slab_reclaimable
      4052 ±  9%     -77.4%     917.50 ± 98%  numa-vmstat.node1.nr_inactive_anon
      6752 ±  2%     -16.2%       5660        numa-vmstat.node1.nr_kernel_stack
      2939 ± 17%     -39.0%       1793 ± 20%  numa-vmstat.node1.nr_mapped
    434.25 ± 54%     -90.8%      39.75 ± 53%  numa-vmstat.node1.nr_page_table_pages
      4467 ±  8%     -75.3%       1102 ± 82%  numa-vmstat.node1.nr_shmem
      6931 ± 22%     -49.3%       3517 ± 13%  numa-vmstat.node1.nr_slab_reclaimable
     17688 ±  8%     -37.7%      11019 ±  7%  numa-vmstat.node1.nr_slab_unreclaimable
      4052 ±  9%     -77.4%     917.50 ± 98%  numa-vmstat.node1.nr_zone_inactive_anon
 1.873e+08 ±  2%     -51.3%   91255968 ± 63%  numa-vmstat.node1.numa_hit
 1.872e+08 ±  2%     -51.3%   91208577 ± 63%  numa-vmstat.node1.numa_local
     25632 ± 87%     -75.5%       6288 ±130%  numa-vmstat.node2.nr_active_anon
     25596 ± 87%     -75.5%       6267 ±131%  numa-vmstat.node2.nr_anon_pages
     25632 ± 87%     -75.5%       6288 ±130%  numa-vmstat.node2.nr_zone_active_anon
     23643 ± 22%     +56.5%      37005 ±  7%  numa-meminfo.node0.KReclaimable
      7871 ±  7%     +10.7%       8713 ±  5%  numa-meminfo.node0.KernelStack
      7269 ± 24%     +24.6%       9060 ± 20%  numa-meminfo.node0.Mapped
    737950 ±  7%     +23.2%     909436        numa-meminfo.node0.MemUsed
      2710 ± 50%     +88.1%       5098 ± 29%  numa-meminfo.node0.PageTables
     23643 ± 22%     +56.5%      37005 ±  7%  numa-meminfo.node0.SReclaimable
     98061 ±  8%     +26.1%     123682 ±  6%  numa-meminfo.node0.Slab
     16278 ± 10%     -77.1%       3725 ± 96%  numa-meminfo.node1.Inactive
     16210 ±  9%     -77.3%       3672 ± 98%  numa-meminfo.node1.Inactive(anon)
     27728 ± 22%     -49.3%      14069 ± 13%  numa-meminfo.node1.KReclaimable
      6748 ±  2%     -16.0%       5670        numa-meminfo.node1.KernelStack
     11640 ± 18%     -39.3%       7065 ± 21%  numa-meminfo.node1.Mapped
    593633 ±  3%     -10.8%     529651 ±  5%  numa-meminfo.node1.MemUsed
      1736 ± 55%     -90.4%     167.00 ± 51%  numa-meminfo.node1.PageTables
     27728 ± 22%     -49.3%      14069 ± 13%  numa-meminfo.node1.SReclaimable
     70762 ±  8%     -37.7%      44079 ±  7%  numa-meminfo.node1.SUnreclaim
     17873 ±  8%     -75.3%       4415 ± 82%  numa-meminfo.node1.Shmem
     98490 ±  5%     -41.0%      58148 ±  8%  numa-meminfo.node1.Slab
    102626 ± 86%     -75.4%      25195 ±130%  numa-meminfo.node2.Active
    102494 ± 87%     -75.4%      25163 ±130%  numa-meminfo.node2.Active(anon)
     71781 ±102%     -84.3%      11236 ±162%  numa-meminfo.node2.AnonHugePages
    102350 ± 87%     -75.5%      25082 ±131%  numa-meminfo.node2.AnonPages
      9418 ±  4%     -25.7%       6997 ±  9%  slabinfo.Acpi-State.active_objs
      9418 ±  4%     -25.7%       6997 ±  9%  slabinfo.Acpi-State.num_objs
     10164            -9.7%       9182 ±  3%  slabinfo.cred_jar.active_objs
     10164            -9.7%       9182 ±  3%  slabinfo.cred_jar.num_objs
      7263 ±  6%     -30.8%       5027 ± 10%  slabinfo.files_cache.active_objs
      7263 ±  6%     -30.8%       5027 ± 10%  slabinfo.files_cache.num_objs
     52454 ±  5%     -16.9%      43590 ±  4%  slabinfo.filp.active_objs
    820.50 ±  5%     -16.6%     684.50 ±  4%  slabinfo.filp.active_slabs
     52535 ±  5%     -16.6%      43839 ±  4%  slabinfo.filp.num_objs
    820.50 ±  5%     -16.6%     684.50 ±  4%  slabinfo.filp.num_slabs
      5691 ±  2%     -25.7%       4226 ±  7%  slabinfo.mm_struct.active_objs
      5691 ±  2%     -25.7%       4226 ±  7%  slabinfo.mm_struct.num_objs
     18729           -13.4%      16227 ±  6%  slabinfo.pde_opener.active_objs
     18729           -13.4%      16227 ±  6%  slabinfo.pde_opener.num_objs
     11365           -23.0%       8755 ± 12%  slabinfo.pool_workqueue.active_objs
     11366           -23.0%       8756 ± 12%  slabinfo.pool_workqueue.num_objs
      3857 ±  4%     -18.9%       3126 ±  5%  slabinfo.sighand_cache.active_objs
      3859 ±  4%     -18.8%       3135 ±  4%  slabinfo.sighand_cache.num_objs
      6370 ±  5%     -22.4%       4942 ±  6%  slabinfo.signal_cache.active_objs
      6372 ±  5%     -22.3%       4950 ±  6%  slabinfo.signal_cache.num_objs
      9525 ±  4%     -24.2%       7218 ±  9%  slabinfo.task_delay_info.active_objs
      9525 ±  4%     -24.2%       7218 ±  9%  slabinfo.task_delay_info.num_objs
      2720 ±  3%     -12.5%       2379        slabinfo.task_struct.active_objs
    909.25 ±  3%     -11.9%     801.00        slabinfo.task_struct.active_slabs
      2728 ±  3%     -11.9%       2403        slabinfo.task_struct.num_objs
    909.25 ±  3%     -11.9%     801.00        slabinfo.task_struct.num_slabs
     62.27            -6.7%      58.10        perf-stat.i.MPKI
 1.506e+10           -18.1%  1.233e+10 ±  4%  perf-stat.i.branch-instructions
      1.93            -0.2        1.68 ±  4%  perf-stat.i.branch-miss-rate%
 2.872e+08           -28.3%  2.058e+08 ±  8%  perf-stat.i.branch-misses
      0.75 ± 11%      -0.4        0.38 ± 30%  perf-stat.i.cache-miss-rate%
  34087325 ± 11%     -61.2%   13229676 ± 41%  perf-stat.i.cache-misses
 4.561e+09           -25.3%  3.405e+09 ±  6%  perf-stat.i.cache-references
   2306970           -31.2%    1587675 ±  8%  perf-stat.i.context-switches
      2.02           +21.0%       2.44 ±  2%  perf-stat.i.cpi
 1.472e+11            -4.0%  1.413e+11 ±  2%  perf-stat.i.cpu-cycles
    778.21 ± 23%   +1447.9%      12045 ± 71%  perf-stat.i.cpu-migrations
      4566 ± 11%    +244.8%      15744 ± 36%  perf-stat.i.cycles-between-cache-misses
  23558623           -23.0%   18146040 ±  8%  perf-stat.i.dTLB-load-misses
 2.475e+10           -23.0%  1.905e+10 ±  5%  perf-stat.i.dTLB-loads
      0.06            +0.0        0.09 ±  9%  perf-stat.i.dTLB-store-miss-rate%
 1.639e+10           -29.7%  1.153e+10 ±  8%  perf-stat.i.dTLB-stores
     32.24 ±  6%      -7.0       25.20 ±  6%  perf-stat.i.iTLB-load-miss-rate%
  17483265 ±  6%     -24.0%   13290898 ±  7%  perf-stat.i.iTLB-load-misses
  36909400 ±  4%      +7.2%   39555285        perf-stat.i.iTLB-loads
 7.295e+10           -20.2%   5.82e+10 ±  4%  perf-stat.i.instructions
      0.50           -17.0%       0.41 ±  3%  perf-stat.i.ipc
   7892760 ± 14%     -56.1%    3466546 ± 33%  perf-stat.i.node-load-misses
     82559 ±  5%     -49.5%      41717 ± 30%  perf-stat.i.node-loads
     85.39 ±  2%     -21.2       64.20 ± 12%  perf-stat.i.node-store-miss-rate%
   4935653 ± 12%     -64.6%    1747970 ± 45%  perf-stat.i.node-store-misses
     62.52            -6.5%      58.45 ±  2%  perf-stat.overall.MPKI
      1.91            -0.2        1.67 ±  4%  perf-stat.overall.branch-miss-rate%
      0.75 ± 11%      -0.4        0.38 ± 34%  perf-stat.overall.cache-miss-rate%
      2.02           +20.6%       2.43 ±  3%  perf-stat.overall.cpi
      4380 ± 12%    +188.2%      12623 ± 39%  perf-stat.overall.cycles-between-cache-misses
      0.06            +0.0        0.09 ± 10%  perf-stat.overall.dTLB-store-miss-rate%
     32.16 ±  6%      -7.0       25.13 ±  6%  perf-stat.overall.iTLB-load-miss-rate%
      0.50           -17.0%       0.41 ±  3%  perf-stat.overall.ipc
     85.86 ±  2%     -19.6       66.27 ± 14%  perf-stat.overall.node-store-miss-rate%
     18033           +16.1%      20929 ±  3%  perf-stat.overall.path-length
 1.501e+10           -18.2%  1.228e+10 ±  4%  perf-stat.ps.branch-instructions
 2.862e+08           -28.4%  2.048e+08 ±  8%  perf-stat.ps.branch-misses
  33952691 ± 11%     -61.3%   13129750 ± 41%  perf-stat.ps.cache-misses
 4.544e+09           -25.4%   3.39e+09 ±  6%  perf-stat.ps.cache-references
   2298503           -31.3%    1580015 ±  8%  perf-stat.ps.context-switches
 1.466e+11            -4.0%  1.408e+11 ±  2%  perf-stat.ps.cpu-cycles
    794.78 ± 24%   +1419.0%      12072 ± 70%  perf-stat.ps.cpu-migrations
  23475449           -23.0%   18074421 ±  8%  perf-stat.ps.dTLB-load-misses
 2.466e+10           -23.1%  1.896e+10 ±  5%  perf-stat.ps.dTLB-loads
 1.633e+10           -29.8%  1.147e+10 ±  8%  perf-stat.ps.dTLB-stores
  17419953 ±  6%     -24.0%   13232122 ±  7%  perf-stat.ps.iTLB-load-misses
  36786011 ±  4%      +7.1%   39405794 ±  2%  perf-stat.ps.iTLB-loads
 7.268e+10           -20.3%  5.795e+10 ±  4%  perf-stat.ps.instructions
   7861951 ± 14%     -56.2%    3443289 ± 33%  perf-stat.ps.node-load-misses
     82432 ±  5%     -49.5%      41606 ± 30%  perf-stat.ps.node-loads
   4916204 ± 12%     -64.7%    1734707 ± 45%  perf-stat.ps.node-store-misses
 2.195e+13           -20.3%   1.75e+13 ±  4%  perf-stat.total.instructions
      3037 ± 66%     +68.9%       5130 ± 41%  interrupts.CPU10.NMI:Non-maskable_interrupts
      3037 ± 66%     +68.9%       5130 ± 41%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      2687 ± 97%    +145.3%       6592 ± 13%  interrupts.CPU103.NMI:Non-maskable_interrupts
      2687 ± 97%    +145.3%       6592 ± 13%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
      2179 ± 35%    +216.1%       6887 ±  2%  interrupts.CPU106.NMI:Non-maskable_interrupts
      2179 ± 35%    +216.1%       6887 ±  2%  interrupts.CPU106.PMI:Performance_monitoring_interrupts
      3006 ± 56%    +126.1%       6798 ±  9%  interrupts.CPU107.RES:Rescheduling_interrupts
      1996 ± 27%    +197.3%       5935 ± 31%  interrupts.CPU108.NMI:Non-maskable_interrupts
      1996 ± 27%    +197.3%       5935 ± 31%  interrupts.CPU108.PMI:Performance_monitoring_interrupts
      2101 ± 92%    +151.9%       5294 ± 49%  interrupts.CPU109.RES:Rescheduling_interrupts
      1580 ± 57%    +330.4%       6801 ± 38%  interrupts.CPU118.RES:Rescheduling_interrupts
      1362 ± 14%    +274.5%       5101 ± 39%  interrupts.CPU119.RES:Rescheduling_interrupts
      3530 ± 35%     -83.7%     574.75 ±115%  interrupts.CPU122.RES:Rescheduling_interrupts
      1343 ± 43%     -74.1%     348.00 ±147%  interrupts.CPU137.RES:Rescheduling_interrupts
      2221 ± 39%     -60.7%     874.25 ±131%  interrupts.CPU141.NMI:Non-maskable_interrupts
      2221 ± 39%     -60.7%     874.25 ±131%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
      3578 ± 57%     -80.8%     688.00 ±134%  interrupts.CPU142.NMI:Non-maskable_interrupts
      3578 ± 57%     -80.8%     688.00 ±134%  interrupts.CPU142.PMI:Performance_monitoring_interrupts
      2935 ± 38%     -73.7%     772.00 ±171%  interrupts.CPU146.RES:Rescheduling_interrupts
      3052 ± 84%   +1035.4%      34656 ±135%  interrupts.CPU15.RES:Rescheduling_interrupts
      2687 ± 13%     -59.0%       1102 ± 98%  interrupts.CPU157.RES:Rescheduling_interrupts
      4192 ± 70%     -65.3%       1455 ±141%  interrupts.CPU166.NMI:Non-maskable_interrupts
      4192 ± 70%     -65.3%       1455 ±141%  interrupts.CPU166.PMI:Performance_monitoring_interrupts
      3234 ± 75%     -72.2%     898.25 ±139%  interrupts.CPU167.NMI:Non-maskable_interrupts
      3234 ± 75%     -72.2%     898.25 ±139%  interrupts.CPU167.PMI:Performance_monitoring_interrupts
      3496 ± 55%     -74.3%     897.00 ±133%  interrupts.CPU172.NMI:Non-maskable_interrupts
      3496 ± 55%     -74.3%     897.00 ±133%  interrupts.CPU172.PMI:Performance_monitoring_interrupts
      3794 ± 51%     -88.5%     435.00 ± 68%  interrupts.CPU173.NMI:Non-maskable_interrupts
      3794 ± 51%     -88.5%     435.00 ± 68%  interrupts.CPU173.PMI:Performance_monitoring_interrupts
      4717 ± 56%     -79.4%     972.75 ±136%  interrupts.CPU174.NMI:Non-maskable_interrupts
      4717 ± 56%     -79.4%     972.75 ±136%  interrupts.CPU174.PMI:Performance_monitoring_interrupts
      4786 ± 13%     -95.5%     213.25 ± 87%  interrupts.CPU175.RES:Rescheduling_interrupts
      3500 ± 64%     -76.2%     832.50 ±107%  interrupts.CPU184.NMI:Non-maskable_interrupts
      3500 ± 64%     -76.2%     832.50 ±107%  interrupts.CPU184.PMI:Performance_monitoring_interrupts
      2748 ±  4%     -71.9%     772.25 ± 95%  interrupts.CPU185.NMI:Non-maskable_interrupts
      2748 ±  4%     -71.9%     772.25 ± 95%  interrupts.CPU185.PMI:Performance_monitoring_interrupts
      3896 ± 50%     -84.2%     616.50 ±115%  interrupts.CPU186.NMI:Non-maskable_interrupts
      3896 ± 50%     -84.2%     616.50 ±115%  interrupts.CPU186.PMI:Performance_monitoring_interrupts
      6332 ± 22%     -90.6%     597.50 ±110%  interrupts.CPU187.NMI:Non-maskable_interrupts
      6332 ± 22%     -90.6%     597.50 ±110%  interrupts.CPU187.PMI:Performance_monitoring_interrupts
      2798 ± 62%     -91.2%     246.25 ±163%  interrupts.CPU188.RES:Rescheduling_interrupts
      7445 ± 30%     -89.6%     776.00 ±126%  interrupts.CPU33.RES:Rescheduling_interrupts
      2500 ± 31%     -88.3%     293.75 ±158%  interrupts.CPU35.RES:Rescheduling_interrupts
      3157 ± 15%     -81.9%     572.25 ±164%  interrupts.CPU36.RES:Rescheduling_interrupts
      4402 ± 61%     -80.8%     844.50 ±168%  interrupts.CPU39.RES:Rescheduling_interrupts
      6686 ± 71%     -91.7%     553.00 ±169%  interrupts.CPU41.RES:Rescheduling_interrupts
      5143 ± 37%     -64.2%       1841 ±149%  interrupts.CPU42.NMI:Non-maskable_interrupts
      5143 ± 37%     -64.2%       1841 ±149%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      5837 ± 32%     -68.2%       1855 ±159%  interrupts.CPU43.NMI:Non-maskable_interrupts
      5837 ± 32%     -68.2%       1855 ±159%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      3557 ± 47%     -97.9%      75.25 ±126%  interrupts.CPU44.RES:Rescheduling_interrupts
      3370 ± 37%    +130.1%       7754 ± 39%  interrupts.CPU5.RES:Rescheduling_interrupts
      4376 ± 37%     -78.9%     924.75 ± 65%  interrupts.CPU58.RES:Rescheduling_interrupts
      2807 ± 64%    +108.3%       5847 ± 36%  interrupts.CPU6.RES:Rescheduling_interrupts
      7502 ± 52%     -83.6%       1226 ±112%  interrupts.CPU61.RES:Rescheduling_interrupts
      3688 ± 67%     -88.3%     430.75 ±167%  interrupts.CPU65.RES:Rescheduling_interrupts
      2911 ± 16%     -51.7%       1405 ± 62%  interrupts.CPU75.NMI:Non-maskable_interrupts
      2911 ± 16%     -51.7%       1405 ± 62%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
      2318 ± 21%     -66.1%     786.25 ±117%  interrupts.CPU76.NMI:Non-maskable_interrupts
      2318 ± 21%     -66.1%     786.25 ±117%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      2279 ± 27%     -86.4%     310.75 ± 34%  interrupts.CPU77.NMI:Non-maskable_interrupts
      2279 ± 27%     -86.4%     310.75 ± 34%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      5185 ± 35%     -91.0%     465.00 ± 34%  interrupts.CPU77.RES:Rescheduling_interrupts
      4118 ± 34%     -74.6%       1047 ± 87%  interrupts.CPU79.RES:Rescheduling_interrupts
      5165 ± 22%     -63.7%       1875 ±107%  interrupts.CPU86.RES:Rescheduling_interrupts
      3514 ± 65%     -70.4%       1041 ±106%  interrupts.CPU88.NMI:Non-maskable_interrupts
      3514 ± 65%     -70.4%       1041 ±106%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
      4784 ± 50%     -78.3%       1037 ± 96%  interrupts.CPU89.NMI:Non-maskable_interrupts
      4784 ± 50%     -78.3%       1037 ± 96%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
      2389 ± 60%    +193.3%       7009 ± 21%  interrupts.CPU9.RES:Rescheduling_interrupts
      3639 ± 19%     -78.3%     789.75 ±128%  interrupts.CPU90.NMI:Non-maskable_interrupts
      3639 ± 19%     -78.3%     789.75 ±128%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      2414 ± 45%     -68.2%     768.50 ±126%  interrupts.CPU91.NMI:Non-maskable_interrupts
      2414 ± 45%     -68.2%     768.50 ±126%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
      3940 ± 28%     -99.7%      11.00 ± 94%  interrupts.CPU93.RES:Rescheduling_interrupts
      4768 ± 51%     -82.2%     849.75 ±136%  interrupts.CPU94.NMI:Non-maskable_interrupts
      4768 ± 51%     -82.2%     849.75 ±136%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      3071 ± 51%     -96.3%     114.25 ±129%  interrupts.CPU94.RES:Rescheduling_interrupts
      2672 ± 40%     -73.5%     707.25 ±171%  interrupts.CPU95.RES:Rescheduling_interrupts
    644992 ± 20%     +28.1%     826353 ±  9%  interrupts.RES:Rescheduling_interrupts
    141316 ±  2%     -10.7%     126249 ±  8%  softirqs.CPU10.RCU
     38660 ±  4%     -39.7%      23314 ± 40%  softirqs.CPU10.SCHED
     40648           -24.3%      30768 ± 18%  softirqs.CPU101.SCHED
    135335 ±  2%     -10.6%     120927 ±  7%  softirqs.CPU102.RCU
     39708 ± 24%     -48.1%      20612 ± 50%  softirqs.CPU102.SCHED
    357756 ± 92%    +233.0%    1191209 ± 45%  softirqs.CPU108.NET_RX
    140033 ±  4%     -11.5%     123936 ± 10%  softirqs.CPU108.RCU
     39682 ±  2%     -40.4%      23651 ± 53%  softirqs.CPU108.SCHED
   1162992 ± 22%     -62.5%     436694 ± 91%  softirqs.CPU113.NET_RX
     36929 ±  7%     -37.5%      23090 ± 43%  softirqs.CPU114.SCHED
     39419 ± 15%     -29.6%      27760 ± 23%  softirqs.CPU116.SCHED
     37471 ± 14%     -20.4%      29821 ±  8%  softirqs.CPU117.SCHED
     38725 ±  4%     -29.5%      27287 ± 24%  softirqs.CPU122.SCHED
    380987 ±112%    -100.0%       4.75 ± 60%  softirqs.CPU124.NET_RX
    684082 ±108%    -100.0%       6.50 ± 82%  softirqs.CPU126.NET_RX
     35779 ± 14%     +17.7%      42122 ±  5%  softirqs.CPU126.SCHED
    136179 ±  4%      -7.4%     126126 ±  5%  softirqs.CPU136.RCU
    136482 ±  3%     -10.7%     121892 ±  3%  softirqs.CPU137.RCU
     37701 ± 13%     -19.4%      30389 ± 19%  softirqs.CPU14.SCHED
    138037 ±  5%      -7.9%     127161 ±  6%  softirqs.CPU143.RCU
    135813 ±  4%     -11.1%     120675 ±  2%  softirqs.CPU144.RCU
    136752 ±  3%      -9.5%     123693 ±  2%  softirqs.CPU146.RCU
    140989 ±  2%      -7.0%     131118 ±  4%  softirqs.CPU147.RCU
    140477 ±  4%      -7.0%     130636 ±  5%  softirqs.CPU151.RCU
    140183 ±  3%      -9.2%     127229 ±  4%  softirqs.CPU157.RCU
     41010 ±  3%     -41.3%      24083 ± 51%  softirqs.CPU157.SCHED
     34115 ± 11%     +19.5%      40766 ± 12%  softirqs.CPU159.SCHED
   1225282 ± 94%     -94.6%      66471 ±173%  softirqs.CPU160.NET_RX
     38414 ±  9%     -38.0%      23813 ± 45%  softirqs.CPU161.SCHED
    137423 ±  4%      -7.0%     127861 ±  3%  softirqs.CPU162.RCU
     30810 ± 31%     +27.3%      39228 ±  6%  softirqs.CPU164.SCHED
    138560 ±  3%     -13.0%     120523 ±  5%  softirqs.CPU165.RCU
     42022 ±  5%     -24.5%      31709 ± 38%  softirqs.CPU165.SCHED
    973700 ± 95%     -96.8%      30927 ±173%  softirqs.CPU172.NET_RX
    977062 ± 77%    -100.0%       2.75 ± 69%  softirqs.CPU173.NET_RX
    547399 ±162%    -100.0%       2.75 ± 86%  softirqs.CPU179.NET_RX
     64145 ±114%    -100.0%       7.00 ± 66%  softirqs.CPU183.NET_RX
     39496           +10.4%      43614 ±  6%  softirqs.CPU183.SCHED
     26478 ± 28%     +53.6%      40670        softirqs.CPU186.SCHED
   1601031 ± 85%    -100.0%     477.00 ±173%  softirqs.CPU187.NET_RX
    133259 ±  2%      -6.6%     124422 ±  4%  softirqs.CPU189.RCU
    137106 ±  2%      -8.4%     125547 ±  2%  softirqs.CPU190.RCU
    102176 ± 68%     -72.1%      28546 ±173%  softirqs.CPU191.NET_RX
   1500188 ± 32%     -71.1%     433553 ± 99%  softirqs.CPU23.NET_RX
   1456825 ± 48%     -78.7%     310211 ±145%  softirqs.CPU27.NET_RX
     30211 ± 21%     +32.5%      40018 ± 16%  softirqs.CPU27.SCHED
   1249314 ± 45%     -83.4%     207506 ±173%  softirqs.CPU28.NET_RX
     32688 ± 12%     +22.9%      40167 ± 10%  softirqs.CPU28.SCHED
    614013 ± 33%     -91.8%      50330 ±172%  softirqs.CPU31.NET_RX
    856129 ± 54%     -92.1%      67308 ±172%  softirqs.CPU32.NET_RX
    826453 ± 88%     -92.6%      61272 ±153%  softirqs.CPU33.NET_RX
    572160 ± 58%     -92.0%      45490 ±172%  softirqs.CPU34.NET_RX
   1198932 ± 26%     -80.2%     237504 ±173%  softirqs.CPU35.NET_RX
    764186 ± 42%     -98.9%       8411 ±170%  softirqs.CPU36.NET_RX
   1957332 ± 39%     -97.6%      47130 ±172%  softirqs.CPU38.NET_RX
     25563 ± 24%     +51.2%      38644 ± 12%  softirqs.CPU38.SCHED
   1904800 ± 68%     -69.5%     581766 ±173%  softirqs.CPU39.NET_RX
    136554           -10.5%     122273 ±  3%  softirqs.CPU41.RCU
   1937742 ± 65%     -83.7%     316430 ±173%  softirqs.CPU42.NET_RX
   1965972 ± 41%     -84.5%     305614 ±173%  softirqs.CPU43.NET_RX
    138112 ±  5%      -8.5%     126429 ±  4%  softirqs.CPU44.RCU
    938187 ± 79%     -88.1%     111510 ±172%  softirqs.CPU45.NET_RX
   1859081 ± 63%    -100.0%     142.50 ±  6%  softirqs.CPU47.NET_RX
   1418766 ± 55%     -96.3%      51993 ±172%  softirqs.CPU56.NET_RX
     31415 ± 16%     +33.4%      41922 ±  8%  softirqs.CPU56.SCHED
   1608741 ± 92%     -93.1%     110943 ±172%  softirqs.CPU57.NET_RX
   1185884 ± 49%     -52.3%     565791 ±113%  softirqs.CPU58.NET_RX
   2280687 ± 64%     -86.8%     301429 ±153%  softirqs.CPU60.NET_RX
    989103 ± 66%     -84.0%     158376 ±173%  softirqs.CPU63.NET_RX
   1357266 ± 62%     -87.3%     172102 ±173%  softirqs.CPU64.NET_RX
   1353050 ± 14%     -77.5%     305024 ±170%  softirqs.CPU67.NET_RX
   1061562 ± 62%     -70.1%     317101 ±173%  softirqs.CPU68.NET_RX
    135927 ±  4%      -9.8%     122670 ±  7%  softirqs.CPU70.RCU
   1219305 ± 63%     -66.8%     404442 ±163%  softirqs.CPU71.NET_RX
    138945 ±  3%     -12.6%     121504 ±  7%  softirqs.CPU71.RCU
   1342914 ± 22%     -57.5%     570854 ±101%  softirqs.CPU75.NET_RX
   1370885 ± 17%     -60.3%     543996 ±107%  softirqs.CPU76.NET_RX
     38700 ±  6%     -20.6%      30718 ± 21%  softirqs.CPU77.SCHED
     37200 ± 11%     -32.2%      25215 ± 53%  softirqs.CPU8.SCHED
   2042132 ± 26%     -76.3%     484633 ±140%  softirqs.CPU84.NET_RX
   1162850 ± 36%     -69.4%     355800 ±116%  softirqs.CPU85.NET_RX
   1454966 ± 30%     -92.4%     109880 ±173%  softirqs.CPU87.NET_RX
     29191 ± 28%     +48.1%      43243 ± 10%  softirqs.CPU87.SCHED
   1982705 ± 30%     -90.8%     181832 ± 87%  softirqs.CPU88.NET_RX
     26016 ± 21%     +60.9%      41872 ± 10%  softirqs.CPU88.SCHED
   2230453 ±  9%     -92.5%     166989 ±173%  softirqs.CPU89.NET_RX
     23850 ± 15%     +65.7%      39517 ±  6%  softirqs.CPU89.SCHED
   1324547 ± 34%    -100.0%       3.25 ± 70%  softirqs.CPU90.NET_RX
     30624 ± 11%     +34.9%      41296 ± 10%  softirqs.CPU90.SCHED
    392004 ±131%    -100.0%       3.75 ± 87%  softirqs.CPU91.NET_RX
    716600 ± 98%     -97.2%      20120 ±173%  softirqs.CPU92.NET_RX
 1.739e+08           -31.1%  1.198e+08 ±  8%  softirqs.NET_RX
     35.60            -2.7       32.88 ±  4%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     38.93            -2.7       36.26 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     38.93            -2.7       36.26 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     38.91            -2.7       36.25 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     39.06            -2.6       36.42 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     36.62            -2.5       34.09 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     36.69            -2.5       34.17 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      7.72 ±  2%      -1.7        5.99 ±  4%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      7.62 ±  2%      -1.7        5.91 ±  4%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
      7.53 ±  2%      -1.7        5.83 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
      7.26 ±  2%      -1.7        5.60 ±  4%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.sctp_packet_transmit
      7.24 ±  2%      -1.7        5.58 ±  4%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
      7.16 ±  2%      -1.6        5.52 ±  4%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
      7.18 ±  2%      -1.6        5.54 ±  4%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
      7.06 ±  2%      -1.6        5.45 ±  4%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
      6.98 ±  2%      -1.6        5.36 ±  4%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
      6.87 ±  2%      -1.6        5.27 ±  4%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
      6.71 ±  2%      -1.6        5.14 ±  4%  perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
      6.57 ±  2%      -1.6        5.01 ±  4%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
      6.51 ±  2%      -1.6        4.96 ±  4%  perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
      6.62 ±  2%      -1.6        5.07 ±  4%  perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
      6.58 ±  2%      -1.5        5.03 ±  4%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     10.09 ±  2%      -1.4        8.72 ±  3%  perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      8.06            -1.3        6.75 ±  4%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      5.28 ±  2%      -1.2        4.05 ±  4%  perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      5.26            -1.2        4.09 ±  5%  perf-profile.calltrace.cycles-pp.memcpy_erms.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      4.95 ±  2%      -1.2        3.78 ±  4%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      8.95 ±  2%      -1.1        7.82 ±  3%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc
      4.45 ±  2%      -1.1        3.36 ±  4%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu
      4.45            -1.0        3.45 ±  5%  perf-profile.calltrace.cycles-pp.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      4.31            -1.0        3.33 ±  5%  perf-profile.calltrace.cycles-pp._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      4.20            -1.0        3.23 ±  5%  perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      4.17            -0.9        3.22 ±  5%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user
      8.12            -0.9        7.23 ±  4%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
      7.75            -0.9        6.87 ±  3%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
      3.15 ±  2%      -0.8        2.36 ±  5%  perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
      7.22 ±  2%      -0.7        6.56 ±  3%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg
      7.06 ±  2%      -0.6        6.44 ±  3%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      4.18 ±  3%      -0.6        3.59 ±  7%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      4.17 ±  3%      -0.6        3.57 ±  7%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      1.42            -0.6        0.86 ± 57%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event
      1.39            -0.6        0.84 ± 57%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready
      3.76 ±  3%      -0.5        3.21 ±  7%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg
      2.45            -0.5        1.90 ±  3%  perf-profile.calltrace.cycles-pp.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      3.66 ±  3%      -0.5        3.12 ±  7%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg
      3.63 ±  3%      -0.5        3.10 ±  7%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      1.92            -0.5        1.44 ±  5%  perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      1.75            -0.4        1.33 ±  5%  perf-profile.calltrace.cycles-pp.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm
      1.08            -0.4        0.72 ±  5%  perf-profile.calltrace.cycles-pp.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      1.56            -0.4        1.21 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter
      1.46            -0.3        1.14 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data
      0.96            -0.3        0.64 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg
      0.72 ±  2%      -0.3        0.40 ± 57%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
      0.93            -0.3        0.62 ±  6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg
      0.65 ±  2%      -0.2        0.41 ± 57%  perf-profile.calltrace.cycles-pp.copy_msghdr_from_user.sendmsg_copy_msghdr.___sys_sendmsg.__sys_sendmsg.do_syscall_64
      0.65 ±  2%      -0.2        0.41 ± 57%  perf-profile.calltrace.cycles-pp.sendmsg_copy_msghdr.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.92 ±  2%      -0.2        0.70 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
      0.87 ±  2%      -0.2        0.67        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      0.76 ±  4%      -0.2        0.57 ±  4%  perf-profile.calltrace.cycles-pp.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      0.87 ±  3%      -0.1        0.74 ±  6%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.85 ±  3%      -0.1        0.72 ±  7%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      1.00 ±  3%      -0.1        0.89 ±  3%  perf-profile.calltrace.cycles-pp.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      0.98 ±  3%      -0.1        0.88 ±  3%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg
      0.96 ±  3%      -0.1        0.86 ±  3%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg
      2.75 ±  5%      +0.9        3.69 ± 13%  perf-profile.calltrace.cycles-pp.kfree_skb.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      2.07 ±  5%      +1.1        3.14 ± 14%  perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.kfree_skb.sctp_recvmsg.inet_recvmsg
      2.21 ±  6%      +1.1        3.27 ± 14%  perf-profile.calltrace.cycles-pp.__free_pages_ok.kfree_skb.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      1.90 ±  6%      +1.1        3.02 ± 14%  perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.kfree_skb.sctp_recvmsg
      1.75 ±  6%      +1.2        2.92 ± 15%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.kfree_skb
      1.99            +1.3        3.29 ±  3%  perf-profile.calltrace.cycles-pp.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      1.85            +1.3        3.19 ±  3%  perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
      1.82 ±  2%      +1.4        3.17 ±  3%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush
      1.81            +1.4        3.17 ±  3%  perf-profile.calltrace.cycles-pp.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit
     23.40            +2.0       25.38 ±  2%  perf-profile.calltrace.cycles-pp.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
     58.47            +3.2       61.64 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     58.27            +3.2       61.47 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.29            +3.2       19.53 ±  2%  perf-profile.calltrace.cycles-pp.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
     16.19            +3.3       19.45 ±  2%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
     16.10            +3.3       19.40 ±  2%  perf-profile.calltrace.cycles-pp.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg
     12.86            +3.4       16.27 ±  2%  perf-profile.calltrace.cycles-pp.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
     14.14            +3.7       17.89 ±  2%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock
     13.93            +3.8       17.70 ±  2%  perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg
     13.93            +3.8       17.70 ±  2%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock
      7.65 ±  2%      +4.2       11.90        perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
      6.70 ±  2%      +4.5       11.22        perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      6.50 ±  2%      +4.6       11.08        perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm
     42.95            +4.6       47.55 ±  2%  perf-profile.calltrace.cycles-pp.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.73            +4.6       12.34 ±  2%  perf-profile.calltrace.cycles-pp.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
     42.69            +4.7       47.35 ±  2%  perf-profile.calltrace.cycles-pp.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.90 ±  2%      +4.7       10.56 ±  2%  perf-profile.calltrace.cycles-pp.__free_pages_ok.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter
      5.67 ±  2%      +4.7       10.33 ±  2%  perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.consume_skb.sctp_chunk_put.sctp_outq_sack
      7.36            +4.7       12.04 ±  2%  perf-profile.calltrace.cycles-pp._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
     41.99            +4.8       46.77 ±  2%  perf-profile.calltrace.cycles-pp.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.82            +4.8       11.62 ±  2%  perf-profile.calltrace.cycles-pp.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      5.18 ±  2%      +4.8        9.99 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.consume_skb.sctp_chunk_put
     41.20            +4.9       46.11 ±  2%  perf-profile.calltrace.cycles-pp.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64
      4.89 ±  2%      +4.9        9.80 ±  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.consume_skb
     40.83            +5.0       45.80 ±  2%  perf-profile.calltrace.cycles-pp.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg
      5.99            +5.1       11.06 ±  2%  perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user
      5.96            +5.1       11.03 ±  2%  perf-profile.calltrace.cycles-pp.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb._sctp_make_chunk
      5.97            +5.1       11.05 ±  2%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty
      7.63            +6.5       14.09 ±  3%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb
      7.33            +6.5       13.86 ±  3%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve
      5.58 ±  2%      +6.9       12.47 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller
      5.26 ±  2%      +7.0       12.24 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node
     35.73            -2.7       33.02 ±  5%  perf-profile.children.cycles-pp.intel_idle
     38.93            -2.7       36.26 ±  4%  perf-profile.children.cycles-pp.start_secondary
     39.06            -2.6       36.42 ±  4%  perf-profile.children.cycles-pp.do_idle
     39.06            -2.6       36.42 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64
     39.06            -2.6       36.42 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
     36.82            -2.5       34.32 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter
     36.81            -2.5       34.32 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter_state
     17.84 ±  2%      -2.2       15.63 ±  4%  perf-profile.children.cycles-pp.sctp_outq_flush
      7.99 ±  2%      -1.8        6.19 ±  4%  perf-profile.children.cycles-pp.__ip_queue_xmit
     15.83            -1.8       14.03 ±  4%  perf-profile.children.cycles-pp.sctp_packet_transmit
      7.83 ±  2%      -1.8        6.07 ±  4%  perf-profile.children.cycles-pp.ip_output
      7.69 ±  2%      -1.7        5.95 ±  4%  perf-profile.children.cycles-pp.ip_finish_output2
      7.33 ±  2%      -1.7        5.66 ±  4%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      7.25 ±  2%      -1.7        5.59 ±  4%  perf-profile.children.cycles-pp.do_softirq
      7.19 ±  2%      -1.6        5.54 ±  4%  perf-profile.children.cycles-pp.do_softirq_own_stack
      7.07 ±  2%      -1.6        5.45 ±  4%  perf-profile.children.cycles-pp.net_rx_action
      6.98 ±  2%      -1.6        5.37 ±  4%  perf-profile.children.cycles-pp.process_backlog
      6.87 ±  2%      -1.6        5.28 ±  4%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      7.43 ±  2%      -1.6        5.85 ±  5%  perf-profile.children.cycles-pp.__softirqentry_text_start
      6.71 ±  2%      -1.6        5.15 ±  4%  perf-profile.children.cycles-pp.ip_rcv
      6.63 ±  2%      -1.6        5.07 ±  4%  perf-profile.children.cycles-pp.ip_local_deliver
      6.59 ±  2%      -1.6        5.04 ±  4%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      6.52 ±  2%      -1.6        4.97 ±  4%  perf-profile.children.cycles-pp.sctp_rcv
      6.57 ±  2%      -1.5        5.02 ±  4%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      8.00            -1.5        6.48 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
     10.09 ±  2%      -1.4        8.72 ±  3%  perf-profile.children.cycles-pp.sctp_primitive_SEND
      5.45            -1.2        4.23 ±  5%  perf-profile.children.cycles-pp.memcpy_erms
      4.45            -1.0        3.46 ±  5%  perf-profile.children.cycles-pp.sctp_user_addto_chunk
      4.32            -1.0        3.33 ±  5%  perf-profile.children.cycles-pp._copy_from_iter_full
      4.21            -1.0        3.24 ±  5%  perf-profile.children.cycles-pp.copyin
      3.16 ±  2%      -0.8        2.37 ±  5%  perf-profile.children.cycles-pp.sctp_ulpq_tail_data
      4.19 ±  3%      -0.6        3.59 ±  7%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      4.17 ±  3%      -0.6        3.58 ±  7%  perf-profile.children.cycles-pp.__skb_datagram_iter
      3.77 ±  3%      -0.5        3.22 ±  7%  perf-profile.children.cycles-pp._copy_to_iter
      2.45            -0.5        1.91 ±  3%  perf-profile.children.cycles-pp.sctp_skb_recv_datagram
      3.66 ±  3%      -0.5        3.12 ±  7%  perf-profile.children.cycles-pp.copyout
      1.93            -0.5        1.46 ±  5%  perf-profile.children.cycles-pp.sctp_ulpq_tail_event
      1.75            -0.4        1.33 ±  5%  perf-profile.children.cycles-pp.sctp_data_ready
      1.33            -0.4        0.93 ±  4%  perf-profile.children.cycles-pp.lock_sock_nested
      1.22            -0.4        0.85 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      1.57            -0.4        1.21 ±  6%  perf-profile.children.cycles-pp.__wake_up_common_lock
      1.46            -0.3        1.14 ±  6%  perf-profile.children.cycles-pp.__wake_up_common
      1.42            -0.3        1.10 ±  6%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.39            -0.3        1.08 ±  6%  perf-profile.children.cycles-pp.try_to_wake_up
      1.07 ±  6%      -0.3        0.77 ±  5%  perf-profile.children.cycles-pp.sctp_ulpevent_make_rcvmsg
      0.83 ±  2%      -0.2        0.58 ±  5%  perf-profile.children.cycles-pp.__list_del_entry_valid
      1.84 ±  3%      -0.2        1.61 ±  4%  perf-profile.children.cycles-pp.__sched_text_start
      0.56 ±  3%      -0.2        0.33 ±  7%  perf-profile.children.cycles-pp.sctp_datamsg_put
      1.00            -0.2        0.77        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.53 ±  3%      -0.2        0.31 ±  7%  perf-profile.children.cycles-pp.sctp_chunk_free
      0.74 ±  8%      -0.2        0.52 ±  4%  perf-profile.children.cycles-pp.sctp_outq_select_transport
      0.92 ±  2%      -0.2        0.70 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.67 ±  9%      -0.2        0.46 ±  2%  perf-profile.children.cycles-pp.sctp_packet_config
      0.77 ±  4%      -0.2        0.57 ±  4%  perf-profile.children.cycles-pp.sctp_ulpevent_free
      0.73            -0.2        0.54 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
      0.85            -0.2        0.67        perf-profile.children.cycles-pp.skb_release_all
      0.84            -0.2        0.66        perf-profile.children.cycles-pp.skb_release_head_state
      1.10            -0.2        0.92 ±  4%  perf-profile.children.cycles-pp.copy_msghdr_from_user
      0.54 ±  8%      -0.2        0.37 ±  5%  perf-profile.children.cycles-pp.__skb_clone
      0.89            -0.1        0.74 ±  3%  perf-profile.children.cycles-pp._copy_from_user
      0.55 ±  2%      -0.1        0.41 ±  6%  perf-profile.children.cycles-pp.enqueue_entity
      0.65 ±  2%      -0.1        0.51 ±  7%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.64            -0.1        0.50 ±  7%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.87 ±  3%      -0.1        0.74 ±  6%  perf-profile.children.cycles-pp.schedule_idle
      0.64            -0.1        0.51 ±  8%  perf-profile.children.cycles-pp.activate_task
      0.58            -0.1        0.45 ±  5%  perf-profile.children.cycles-pp.sctp_chunkify
      0.40            -0.1        0.28 ±  5%  perf-profile.children.cycles-pp.__list_add_valid
      0.66 ±  2%      -0.1        0.54 ±  5%  perf-profile.children.cycles-pp.sendmsg_copy_msghdr
      0.41 ±  2%      -0.1        0.29 ±  5%  perf-profile.children.cycles-pp.__slab_free
      0.31 ± 14%      -0.1        0.20 ±  4%  perf-profile.children.cycles-pp.ipv4_dst_check
      1.00 ±  3%      -0.1        0.89 ±  3%  perf-profile.children.cycles-pp.schedule_timeout
      0.41 ±  7%      -0.1        0.30 ±  5%  perf-profile.children.cycles-pp.sctp_hash_cmp
      0.99 ±  3%      -0.1        0.88 ±  3%  perf-profile.children.cycles-pp.schedule
      0.28 ± 15%      -0.1        0.17 ±  4%  perf-profile.children.cycles-pp.__copy_skb_header
      0.39 ±  2%      -0.1        0.28 ±  4%  perf-profile.children.cycles-pp.sctp_addrs_lookup_transport
      0.42 ±  2%      -0.1        0.32 ±  3%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.22 ± 12%      -0.1        0.13 ±  5%  perf-profile.children.cycles-pp.sctp_transport_hold
      0.41 ±  8%      -0.1        0.33 ±  6%  perf-profile.children.cycles-pp.sctp_endpoint_lookup_assoc
      0.39 ±  7%      -0.1        0.30 ±  8%  perf-profile.children.cycles-pp.sctp_epaddr_lookup_transport
      0.45            -0.1        0.36 ±  4%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.48            -0.1        0.40 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.41            -0.1        0.34 ±  5%  perf-profile.children.cycles-pp.___might_sleep
      0.32 ±  2%      -0.1        0.24 ±  6%  perf-profile.children.cycles-pp.sctp_addto_chunk
      0.32 ±  4%      -0.1        0.24 ±  4%  perf-profile.children.cycles-pp.sctp_eat_data
      0.47 ±  2%      -0.1        0.40 ±  4%  perf-profile.children.cycles-pp.import_iovec
      0.36            -0.1        0.29 ±  4%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.36 ±  2%      -0.1        0.29 ±  3%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.32            -0.1        0.25 ±  4%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.48            -0.1        0.41 ±  4%  perf-profile.children.cycles-pp.update_load_avg
      0.43            -0.1        0.36 ±  3%  perf-profile.children.cycles-pp.sctp_sf_eat_data_6_2
      0.23 ±  4%      -0.1        0.16 ±  6%  perf-profile.children.cycles-pp.skb_release_data
      0.35 ±  3%      -0.1        0.28 ±  4%  perf-profile.children.cycles-pp.sctp_wfree
      0.41 ±  3%      -0.1        0.35 ±  5%  perf-profile.children.cycles-pp.rw_copy_check_uvector
      0.23 ±  3%      -0.1        0.17 ±  5%  perf-profile.children.cycles-pp.memset_erms
      0.25 ±  2%      -0.1        0.19 ±  9%  perf-profile.children.cycles-pp.sctp_sock_rfree
      0.21 ±  3%      -0.1        0.15 ±  7%  perf-profile.children.cycles-pp.__switch_to
      0.15 ±  9%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.ip_local_out
      0.22 ±  5%      -0.1        0.16 ±  6%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.46 ±  5%      -0.1        0.40 ±  8%  perf-profile.children.cycles-pp.recvmsg_copy_msghdr
      0.52 ±  3%      -0.1        0.46 ±  3%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.29 ±  2%      -0.1        0.23 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.36 ±  5%      -0.1        0.31 ±  3%  perf-profile.children.cycles-pp.aa_sk_perm
      0.29            -0.1        0.24 ±  5%  perf-profile.children.cycles-pp.__fget_light
      0.16 ±  2%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.36 ±  3%      -0.1        0.30 ±  3%  perf-profile.children.cycles-pp.sctp_sched_fcfs_dequeue
      0.15 ±  5%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.16 ±  2%      -0.1        0.11 ±  6%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.60 ±  2%      -0.1        0.55 ±  2%  perf-profile.children.cycles-pp.__check_object_size
      0.30            -0.1        0.24 ±  3%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.26 ±  3%      -0.1        0.21 ±  5%  perf-profile.children.cycles-pp.sctp_make_sack
      0.27 ±  4%      -0.1        0.22        perf-profile.children.cycles-pp.loopback_xmit
      0.16 ±  9%      -0.1        0.11 ±  7%  perf-profile.children.cycles-pp.ipv4_mtu
      0.13 ±  6%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.22            -0.0        0.17 ±  6%  perf-profile.children.cycles-pp.sctp_inq_pop
      0.15 ±  3%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.check_preempt_curr
      0.12 ±  5%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.ip_send_check
      0.17 ±  4%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.sctp_outq_flush_ctrl
      0.11 ±  3%      -0.0        0.07 ± 17%  perf-profile.children.cycles-pp.resched_curr
      0.22 ±  3%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.sock_kmalloc
      0.14 ±  5%      -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.sock_wfree
      0.14 ±  8%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.__ip_local_out
      0.25 ±  3%      -0.0        0.20 ±  7%  perf-profile.children.cycles-pp.sctp_check_transmitted
      0.15 ±  3%      -0.0        0.11 ±  3%  perf-profile.children.cycles-pp.sctp_transport_put
      0.22 ±  3%      -0.0        0.18 ±  9%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.44 ±  5%      -0.0        0.40 ±  3%  perf-profile.children.cycles-pp.dequeue_entity
      0.18 ±  6%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.12 ±  8%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.sctp_chunk_hold
      0.11 ± 15%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__ip_finish_output
      0.07 ±  7%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.sctp_hash_key
      0.14 ± 11%      -0.0        0.10 ± 11%  perf-profile.children.cycles-pp.dst_release
      0.23 ±  2%      -0.0        0.20 ±  4%  perf-profile.children.cycles-pp.move_addr_to_kernel
      0.19 ±  5%      -0.0        0.15 ±  4%  perf-profile.children.cycles-pp.sctp_ulpevent_receive_data
      0.24 ±  2%      -0.0        0.21 ±  7%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.17 ±  2%      -0.0        0.14 ±  6%  perf-profile.children.cycles-pp.sctp_v4_addr_valid
      0.08 ±  6%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.06 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.sctp_bind_addr_match
      0.17 ±  5%      -0.0        0.13        perf-profile.children.cycles-pp.sctp_association_put
      0.16 ±  2%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.sock_kfree_s
      0.19            -0.0        0.16 ±  7%  perf-profile.children.cycles-pp.update_rq_clock
      0.17            -0.0        0.14 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.18 ±  2%      -0.0        0.15 ±  8%  perf-profile.children.cycles-pp.__zone_watermark_ok
      0.13 ±  9%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.36            -0.0        0.33 ±  5%  perf-profile.children.cycles-pp.kfree
      0.20 ±  4%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.sctp_sched_dequeue_common
      0.11 ±  4%      -0.0        0.08 ± 14%  perf-profile.children.cycles-pp.sctp_association_hold
      0.07 ±  5%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.sctp_v4_from_skb
      0.07 ±  6%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.pick_next_entity
      0.07 ±  7%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.30            -0.0        0.28 ±  5%  perf-profile.children.cycles-pp.sctp_packet_transmit_chunk
      0.20 ±  4%      -0.0        0.18 ±  6%  perf-profile.children.cycles-pp.update_curr
      0.14            -0.0        0.12 ±  7%  perf-profile.children.cycles-pp.sctp_sockaddr_af
      0.14 ±  3%      -0.0        0.11        perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.10 ±  7%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.del_timer
      0.09 ±  9%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.sctp_transport_reset_t3_rtx
      0.12 ±  8%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.09 ±  4%      -0.0        0.07 ± 14%  perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase
      0.13 ±  6%      -0.0        0.11 ± 10%  perf-profile.children.cycles-pp.skb_clone
      0.08 ± 10%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.sctp_v4_ecn_capable
      0.12 ±  5%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.mod_timer
      0.14 ±  6%      -0.0        0.12 ±  5%  perf-profile.children.cycles-pp.__kmalloc
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.account_entity_dequeue
      0.08 ±  5%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.09 ±  4%      -0.0        0.07        perf-profile.children.cycles-pp.lock_timer_base
      0.14 ±  6%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.mod_node_page_state
      0.10 ±  4%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.sctp_validate_data
      0.08 ±  5%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.11 ±  3%      -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.__switch_to_asm
      0.09 ±  4%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.netif_rx
      0.09            -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.netif_rx_internal
      0.08            -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.07            -0.0        0.06        perf-profile.children.cycles-pp.__x64_sys_sendmsg
      0.06            -0.0        0.05        perf-profile.children.cycles-pp.__free_pages
      0.07 ± 10%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.___perf_sw_event
      0.05 ± 60%      +0.0        0.09 ± 15%  perf-profile.children.cycles-pp.irq_enter
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.load_balance
      2.75 ±  5%      +0.9        3.69 ± 13%  perf-profile.children.cycles-pp.kfree_skb
     25.85            +2.0       27.81 ±  2%  perf-profile.children.cycles-pp.sctp_cmd_interpreter
     23.41            +2.0       25.38 ±  2%  perf-profile.children.cycles-pp.sctp_sendmsg_to_asoc
     25.96            +2.0       27.94 ±  2%  perf-profile.children.cycles-pp.sctp_do_sm
     19.22            +2.5       21.76 ±  2%  perf-profile.children.cycles-pp.sctp_assoc_bh_rcv
     58.51            +3.2       61.68 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     58.31            +3.2       61.52 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     16.37            +3.2       19.60 ±  2%  perf-profile.children.cycles-pp.release_sock
     16.19            +3.3       19.46 ±  2%  perf-profile.children.cycles-pp.__release_sock
     16.11            +3.3       19.40 ±  2%  perf-profile.children.cycles-pp.sctp_backlog_rcv
     12.87            +3.4       16.28 ±  2%  perf-profile.children.cycles-pp.sctp_datamsg_from_user
      7.65 ±  2%      +4.3       11.90        perf-profile.children.cycles-pp.sctp_outq_sack
      7.37 ±  2%      +4.4       11.72        perf-profile.children.cycles-pp.sctp_chunk_put
      6.81 ±  2%      +4.5       11.31        perf-profile.children.cycles-pp.consume_skb
     42.95            +4.6       47.55 ±  2%  perf-profile.children.cycles-pp.__sys_sendmsg
      7.74            +4.6       12.34 ±  2%  perf-profile.children.cycles-pp.sctp_make_datafrag_empty
     42.70            +4.6       47.35 ±  2%  perf-profile.children.cycles-pp.___sys_sendmsg
      7.52            +4.7       12.17 ±  2%  perf-profile.children.cycles-pp._sctp_make_chunk
     42.00            +4.8       46.78 ±  2%  perf-profile.children.cycles-pp.____sys_sendmsg
     41.21            +4.9       46.12 ±  2%  perf-profile.children.cycles-pp.sock_sendmsg
     40.84            +5.0       45.82 ±  2%  perf-profile.children.cycles-pp.sctp_sendmsg
      7.75 ±  2%      +5.7       13.48 ±  4%  perf-profile.children.cycles-pp.free_one_page
      8.12 ±  2%      +5.7       13.86 ±  4%  perf-profile.children.cycles-pp.__free_pages_ok
      7.47 ±  2%      +5.8       13.30 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      8.98            +6.1       15.05 ±  2%  perf-profile.children.cycles-pp.__alloc_skb
      7.92            +6.4       14.31 ±  3%  perf-profile.children.cycles-pp.__kmalloc_reserve
      7.87            +6.4       14.29 ±  2%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      7.77            +6.4       14.21 ±  2%  perf-profile.children.cycles-pp.kmalloc_large_node
      7.64            +6.5       14.10 ±  3%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      7.35            +6.5       13.88 ±  3%  perf-profile.children.cycles-pp.get_page_from_freelist
      6.01 ±  2%      +6.8       12.84 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     12.88 ±  2%     +12.8       25.66 ±  4%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     35.73            -2.7       33.02 ±  5%  perf-profile.self.cycles-pp.intel_idle
      7.95            -1.5        6.43 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      5.40            -1.2        4.20 ±  5%  perf-profile.self.cycles-pp.memcpy_erms
      2.65            -0.6        2.05        perf-profile.self.cycles-pp.do_syscall_64
      0.78            -0.3        0.50 ±  4%  perf-profile.self.cycles-pp.__alloc_skb
      0.81            -0.3        0.56        perf-profile.self.cycles-pp._raw_spin_lock
      0.81 ±  2%      -0.2        0.56 ±  5%  perf-profile.self.cycles-pp.__list_del_entry_valid
      1.00            -0.2        0.77        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.92 ±  2%      -0.2        0.70 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.51 ±  4%      -0.2        0.29 ±  7%  perf-profile.self.cycles-pp.sctp_datamsg_put
      0.57            -0.2        0.38 ±  2%  perf-profile.self.cycles-pp.sctp_datamsg_from_user
      0.70 ±  3%      -0.2        0.53 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.75 ±  2%      -0.1        0.61 ±  5%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.38            -0.1        0.26 ±  7%  perf-profile.self.cycles-pp.__list_add_valid
      0.41            -0.1        0.30 ±  6%  perf-profile.self.cycles-pp.sctp_recvmsg
      0.30 ± 14%      -0.1        0.19 ±  6%  perf-profile.self.cycles-pp.ipv4_dst_check
      0.28 ± 15%      -0.1        0.17 ±  7%  perf-profile.self.cycles-pp.__copy_skb_header
      0.40 ±  2%      -0.1        0.29 ±  5%  perf-profile.self.cycles-pp.__slab_free
      0.33 ±  2%      -0.1        0.24 ±  7%  perf-profile.self.cycles-pp.free_one_page
      0.39 ±  2%      -0.1        0.30 ±  3%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.34 ± 13%      -0.1        0.25        perf-profile.self.cycles-pp.sctp_cmd_interpreter
      0.21 ± 13%      -0.1        0.12 ±  4%  perf-profile.self.cycles-pp.sctp_transport_hold
      0.39 ±  5%      -0.1        0.31 ±  5%  perf-profile.self.cycles-pp.sctp_packet_transmit
      0.46 ±  2%      -0.1        0.38 ±  5%  perf-profile.self.cycles-pp.sctp_outq_flush
      0.30 ±  2%      -0.1        0.22 ±  3%  perf-profile.self.cycles-pp.sctp_chunk_put
      0.33            -0.1        0.25 ±  7%  perf-profile.self.cycles-pp.kmem_cache_free
      0.40            -0.1        0.33 ±  6%  perf-profile.self.cycles-pp.___might_sleep
      0.27 ±  5%      -0.1        0.20 ±  6%  perf-profile.self.cycles-pp.sctp_rcv
      0.26 ±  7%      -0.1        0.18 ±  8%  perf-profile.self.cycles-pp.sctp_packet_config
      0.18 ±  3%      -0.1        0.11 ± 11%  perf-profile.self.cycles-pp.sctp_ulpq_tail_event
      0.18 ±  2%      -0.1        0.11        perf-profile.self.cycles-pp.sctp_chunkify
      0.23 ±  4%      -0.1        0.16 ±  6%  perf-profile.self.cycles-pp.skb_release_data
      0.23 ±  6%      -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.33 ±  3%      -0.1        0.27 ±  7%  perf-profile.self.cycles-pp.__sched_text_start
      0.26 ±  2%      -0.1        0.20 ±  7%  perf-profile.self.cycles-pp.__skb_clone
      0.22 ±  3%      -0.1        0.16 ±  4%  perf-profile.self.cycles-pp.memset_erms
      0.22 ±  5%      -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.20 ±  4%      -0.1        0.14 ±  7%  perf-profile.self.cycles-pp.__switch_to
      0.15 ±  3%      -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.24            -0.1        0.18 ±  9%  perf-profile.self.cycles-pp.sctp_sock_rfree
      0.28            -0.1        0.22 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.31 ±  2%      -0.1        0.26 ±  3%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.18 ± 10%      -0.1        0.12 ±  6%  perf-profile.self.cycles-pp.sctp_data_ready
      0.27 ±  2%      -0.1        0.21 ±  4%  perf-profile.self.cycles-pp.__might_sleep
      0.29 ±  2%      -0.1        0.24 ±  5%  perf-profile.self.cycles-pp.__fget_light
      0.12 ±  7%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.ip_send_check
      0.29 ±  6%      -0.0        0.24 ±  4%  perf-profile.self.cycles-pp.__check_object_size
      0.27 ±  4%      -0.0        0.22        perf-profile.self.cycles-pp.sctp_wfree
      0.26 ±  7%      -0.0        0.21 ±  5%  perf-profile.self.cycles-pp.sctp_sendmsg
      0.15 ± 10%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.ipv4_mtu
      0.11 ±  3%      -0.0        0.07 ± 17%  perf-profile.self.cycles-pp.resched_curr
      0.22 ±  9%      -0.0        0.17 ±  4%  perf-profile.self.cycles-pp.sctp_skb_recv_datagram
      0.14 ±  5%      -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.sock_wfree
      0.15 ±  3%      -0.0        0.10 ±  4%  perf-profile.self.cycles-pp.sctp_eat_data
      0.13 ± 13%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.dst_release
      0.15 ±  4%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.sctp_transport_put
      0.08 ± 12%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.sctp_ulpevent_free
      0.07 ±  7%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.pick_next_entity
      0.07 ±  7%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.sctp_hash_key
      0.33 ±  2%      -0.0        0.29 ±  6%  perf-profile.self.cycles-pp.__free_pages_ok
      0.12 ±  5%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.sctp_chunk_hold
      0.08 ±  6%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.14 ±  3%      -0.0        0.10 ±  8%  perf-profile.self.cycles-pp.enqueue_entity
      0.18 ±  7%      -0.0        0.14 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.17 ±  5%      -0.0        0.13        perf-profile.self.cycles-pp.sctp_association_put
      0.21 ±  5%      -0.0        0.18 ±  7%  perf-profile.self.cycles-pp.____sys_recvmsg
      0.20 ±  2%      -0.0        0.17 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.15 ±  3%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.sctp_v4_addr_valid
      0.14 ±  8%      -0.0        0.10 ±  4%  perf-profile.self.cycles-pp.sctp_epaddr_lookup_transport
      0.14 ±  3%      -0.0        0.11 ±  3%  perf-profile.self.cycles-pp.set_next_entity
      0.16 ±  4%      -0.0        0.13        perf-profile.self.cycles-pp.do_idle
      0.16 ±  2%      -0.0        0.13 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.13 ±  9%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.18 ±  2%      -0.0        0.15 ±  8%  perf-profile.self.cycles-pp.__zone_watermark_ok
      0.20 ±  4%      -0.0        0.17 ±  7%  perf-profile.self.cycles-pp.sctp_sendmsg_to_asoc
      0.09 ±  7%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.sctp_assoc_rwnd_increase
      0.15 ±  4%      -0.0        0.12 ± 12%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.12 ±  7%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.sctp_check_transmitted
      0.10 ±  7%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.sctp_ulpevent_make_rcvmsg
      0.14 ±  3%      -0.0        0.11 ±  6%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.16 ±  2%      -0.0        0.13 ±  3%  perf-profile.self.cycles-pp.sctp_sched_fcfs_dequeue
      0.12 ±  4%      -0.0        0.10        perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.12 ±  5%      -0.0        0.10 ± 11%  perf-profile.self.cycles-pp.ip_finish_output2
      0.08 ±  5%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.sock_kmalloc
      0.06 ±  6%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.sockfd_lookup_light
      0.10 ±  7%      -0.0        0.08 ± 14%  perf-profile.self.cycles-pp.sctp_association_hold
      0.09 ±  4%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.11 ±  4%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.sctp_ulpevent_receive_data
      0.11 ±  4%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.sctp_make_datafrag_empty
      0.09 ±  4%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.consume_skb
      0.09 ±  4%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.sctp_inq_pop
      0.08 ±  5%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.14 ±  3%      -0.0        0.12 ±  3%  perf-profile.self.cycles-pp.mod_node_page_state
      0.07 ±  5%      -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.sctp_v4_ecn_capable
      0.09 ± 10%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.rw_copy_check_uvector
      0.11 ±  3%      -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.__switch_to_asm
      0.11 ±  4%      -0.0        0.10 ±  8%  perf-profile.self.cycles-pp.sctp_sched_dequeue_common
      0.08 ±  5%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.update_curr
      0.09 ±  4%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.copy_msghdr_from_user
      0.08            -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.__sctp_packet_append_chunk
      0.08 ±  5%      -0.0        0.07        perf-profile.self.cycles-pp.sctp_addrs_lookup_transport
      0.06            -0.0        0.05        perf-profile.self.cycles-pp.__free_pages
      0.06 ±  7%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.___perf_sw_event
     12.88 ±  2%     +12.8       25.66 ±  4%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/30%/debian-x86_64-20191114.cgz/300s/lkp-skl-fpga01/whetstone-double/unixbench/0x2000065

commit: 
  9c40365a65 ("threads: Update PID limit comment according to futex UAPI change")
  59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group")

9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     23098            -4.0%      22179        unixbench.score
    474.19            +2.0%     483.52        unixbench.time.elapsed_time
    474.19            +2.0%     483.52        unixbench.time.elapsed_time.max
     20407           -15.6%      17220        unixbench.time.involuntary_context_switches
     11653            +2.5%      11940        unixbench.time.user_time
      6674            -2.7%       6494        unixbench.time.voluntary_context_switches
  60218701            -2.2%   58918844        unixbench.workload
     51504 ±  2%     +61.4%      83147 ± 16%  cpuidle.C1.usage
     45407 ±  4%      +9.0%      49503 ±  6%  meminfo.Shmem
      0.00 ±117%      +0.0        0.03 ± 78%  mpstat.cpu.all.soft%
      1505            +5.3%       1584        vmstat.system.cs
      9163            -9.2%       8320 ±  6%  numa-meminfo.node0.KernelStack
    992.75 ± 12%     -49.9%     497.00 ± 65%  numa-meminfo.node1.Mlocked
    535099 ±  2%      -7.3%     496178 ±  3%  numa-meminfo.node1.Unevictable
      9165            -9.1%       8327 ±  6%  numa-vmstat.node0.nr_kernel_stack
    133774 ±  2%      -7.3%     124044 ±  3%  numa-vmstat.node1.nr_unevictable
    133774 ±  2%      -7.3%     124044 ±  3%  numa-vmstat.node1.nr_zone_unevictable
      1.24 ± 16%      -0.7        0.49 ± 32%  perf-profile.children.cycles-pp.ktime_get
      0.94 ± 18%      -0.5        0.43 ± 29%  perf-profile.children.cycles-pp.clockevents_program_event
      0.11 ± 36%      +0.1        0.20 ± 32%  perf-profile.children.cycles-pp.tick_irq_enter
      1.12 ± 17%      -0.7        0.38 ± 49%  perf-profile.self.cycles-pp.ktime_get
    842912 ± 13%     -42.0%     489022 ± 36%  numa-numastat.node0.local_node
    851323 ± 14%     -39.6%     514237 ± 31%  numa-numastat.node0.numa_hit
      8420 ±172%    +199.5%      25221 ± 57%  numa-numastat.node0.other_node
    780061 ± 13%     +47.3%    1149237 ± 15%  numa-numastat.node1.local_node
    805331 ± 14%     +43.8%    1157736 ± 14%  numa-numastat.node1.numa_hit
     92.00 ±  6%      -5.4%      87.00        proc-vmstat.nr_dirtied
     11349 ±  4%      +9.0%      12374 ±  6%  proc-vmstat.nr_shmem
     19448            -1.5%      19147        proc-vmstat.nr_slab_reclaimable
     44561            -3.1%      43181        proc-vmstat.nr_slab_unreclaimable
     13330 ± 44%     -84.1%       2121 ± 82%  proc-vmstat.numa_hint_faults
      5558 ±107%     -99.3%      37.25 ±148%  proc-vmstat.numa_hint_faults_local
     51552 ± 33%     -54.6%      23426 ± 53%  proc-vmstat.numa_pte_updates
      5345 ± 12%     +24.6%       6662 ± 16%  proc-vmstat.pgactivate
   1739692            +1.0%    1756902        proc-vmstat.pgfree
      1018 ±  7%     -16.4%     850.75 ±  5%  slabinfo.file_lock_cache.active_objs
      1018 ±  7%     -16.4%     850.75 ±  5%  slabinfo.file_lock_cache.num_objs
      4826           -11.2%       4285 ±  3%  slabinfo.files_cache.active_objs
      4826           -11.2%       4285 ±  3%  slabinfo.files_cache.num_objs
      2826 ±  2%     -11.0%       2515        slabinfo.sighand_cache.active_objs
      2828 ±  2%     -11.0%       2517        slabinfo.sighand_cache.num_objs
      1151 ±  2%     -24.1%     873.25 ± 22%  slabinfo.task_group.active_objs
      1151 ±  2%     -24.1%     873.25 ± 22%  slabinfo.task_group.num_objs
     59534 ±  2%      -9.2%      54079 ±  3%  slabinfo.vm_area_struct.active_objs
      1490 ±  2%      -9.1%       1354 ±  3%  slabinfo.vm_area_struct.active_slabs
     59640 ±  2%      -9.1%      54195 ±  3%  slabinfo.vm_area_struct.num_objs
      1490 ±  2%      -9.1%       1354 ±  3%  slabinfo.vm_area_struct.num_slabs
 7.107e+09            -2.7%  6.915e+09        perf-stat.i.branch-instructions
     12.39 ± 15%      -6.0        6.37 ± 23%  perf-stat.i.cache-miss-rate%
      1477            +5.5%       1558        perf-stat.i.context-switches
    111.88            -1.3%     110.46        perf-stat.i.cpu-migrations
 1.099e+10            -2.5%  1.071e+10        perf-stat.i.dTLB-loads
 3.823e+09            -2.6%  3.724e+09        perf-stat.i.dTLB-stores
   2132180           -20.1%    1703908 ± 14%  perf-stat.i.iTLB-loads
 6.802e+10            -2.7%  6.618e+10        perf-stat.i.instructions
      0.88            -4.5%       0.84 ±  2%  perf-stat.i.ipc
      4076            -1.5%       4014        perf-stat.i.minor-faults
    107204 ±  2%     -31.1%      73884 ± 11%  perf-stat.i.node-load-misses
     14357 ±  5%      -9.6%      12974 ±  3%  perf-stat.i.node-loads
     26949 ±  4%     -51.0%      13212 ± 18%  perf-stat.i.node-store-misses
      4076            -1.5%       4014        perf-stat.i.page-faults
     12.57 ± 16%      -6.1        6.42 ± 24%  perf-stat.overall.cache-miss-rate%
      1.04            +3.8%       1.08        perf-stat.overall.cpi
      0.96            -3.6%       0.93        perf-stat.overall.ipc
     88.19            -3.3       84.92        perf-stat.overall.node-load-miss-rate%
     78.98           -11.9       67.04 ±  8%  perf-stat.overall.node-store-miss-rate%
    536500            +1.3%     543249        perf-stat.overall.path-length
 7.106e+09            -2.8%  6.907e+09        perf-stat.ps.branch-instructions
      1474            +5.5%       1554        perf-stat.ps.context-switches
    111.66            -1.3%     110.22        perf-stat.ps.cpu-migrations
 1.098e+10            -2.6%   1.07e+10        perf-stat.ps.dTLB-loads
 3.823e+09            -2.7%  3.719e+09        perf-stat.ps.dTLB-stores
   2128273           -20.1%    1700626 ± 14%  perf-stat.ps.iTLB-loads
 6.801e+10            -2.8%   6.61e+10        perf-stat.ps.instructions
      4061            -1.4%       4004        perf-stat.ps.minor-faults
    107032 ±  2%     -31.1%      73785 ± 11%  perf-stat.ps.node-load-misses
     14320 ±  5%      -9.5%      12960 ±  3%  perf-stat.ps.node-loads
     26906 ±  4%     -51.0%      13193 ± 18%  perf-stat.ps.node-store-misses
      4061            -1.4%       4004        perf-stat.ps.page-faults
     51615 ±  9%     +38.7%      71599 ± 14%  sched_debug.cfs_rq:/.exec_clock.avg
     89952 ±  7%     +79.0%     160981 ± 14%  sched_debug.cfs_rq:/.exec_clock.max
      6590 ± 39%     -88.3%     770.93 ±168%  sched_debug.cfs_rq:/.exec_clock.min
     21438 ±  5%    +140.7%      51593 ± 19%  sched_debug.cfs_rq:/.exec_clock.stddev
   1649313 ±  8%     +39.8%    2305637 ± 14%  sched_debug.cfs_rq:/.min_vruntime.avg
   2846100 ±  7%     +82.5%    5195365 ± 14%  sched_debug.cfs_rq:/.min_vruntime.max
    225248 ± 36%     -87.1%      29096 ±143%  sched_debug.cfs_rq:/.min_vruntime.min
    677866 ±  5%    +145.1%    1661129 ± 19%  sched_debug.cfs_rq:/.min_vruntime.stddev
    130.54 ±  7%     -54.9%      58.93 ±100%  sched_debug.cfs_rq:/.removed.load_avg.max
     19.52 ± 32%     -67.2%       6.41 ±100%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    130.54 ±  7%     -54.9%      58.93 ±100%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     19.52 ± 32%     -67.2%       6.41 ±100%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
  -1075371          -254.5%    1660977 ± 63%  sched_debug.cfs_rq:/.spread0.avg
    121453 ± 95%   +3646.9%    4550704 ± 34%  sched_debug.cfs_rq:/.spread0.max
  -2499411           -75.4%    -615566        sched_debug.cfs_rq:/.spread0.min
    677861 ±  5%    +145.1%    1661130 ± 19%  sched_debug.cfs_rq:/.spread0.stddev
      6.00 ±  7%     -31.6%       4.10 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     25.29 ±  7%     -22.8%      19.52 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    241338 ±  5%     +12.8%     272122 ±  4%  sched_debug.cpu.clock.avg
    241342 ±  5%     +12.8%     272127 ±  4%  sched_debug.cpu.clock.max
    241333 ±  5%     +12.8%     272117 ±  4%  sched_debug.cpu.clock.min
      2.47 ±  2%     +12.0%       2.76 ±  4%  sched_debug.cpu.clock.stddev
    241338 ±  5%     +12.8%     272122 ±  4%  sched_debug.cpu.clock_task.avg
    241342 ±  5%     +12.8%     272127 ±  4%  sched_debug.cpu.clock_task.max
    241333 ±  5%     +12.8%     272117 ±  4%  sched_debug.cpu.clock_task.min
      2.47 ±  2%     +12.0%       2.76 ±  4%  sched_debug.cpu.clock_task.stddev
      4932 ±  3%     +12.7%       5557 ±  3%  sched_debug.cpu.nr_switches.avg
      3525 ±  5%     +43.7%       5065 ± 17%  sched_debug.cpu.nr_switches.stddev
      2933 ±  5%     +20.6%       3536 ±  5%  sched_debug.cpu.sched_count.avg
     19139 ± 27%     +56.8%      30011 ± 48%  sched_debug.cpu.sched_count.max
      2773 ± 10%     +55.8%       4320 ± 18%  sched_debug.cpu.sched_count.stddev
      1280 ±  5%     +22.6%       1568 ±  6%  sched_debug.cpu.sched_goidle.avg
      9370 ± 27%     +57.8%      14789 ± 48%  sched_debug.cpu.sched_goidle.max
      1374 ± 10%     +56.0%       2144 ± 19%  sched_debug.cpu.sched_goidle.stddev
      1302 ±  5%     +21.4%       1581 ±  6%  sched_debug.cpu.ttwu_count.avg
      1455 ±  4%     +56.4%       2276 ± 19%  sched_debug.cpu.ttwu_count.stddev
    594.81 ±  4%     +28.4%     763.77 ±  7%  sched_debug.cpu.ttwu_local.avg
      2311 ± 24%     +79.3%       4143 ± 10%  sched_debug.cpu.ttwu_local.max
    345.75 ±  7%    +105.0%     708.72 ±  4%  sched_debug.cpu.ttwu_local.stddev
    241334 ±  5%     +12.8%     272118 ±  4%  sched_debug.cpu_clk
    240831 ±  5%     +12.8%     271615 ±  4%  sched_debug.ktime
    241752 ±  5%     +12.7%     272537 ±  4%  sched_debug.sched_clk
     74.25 ±159%     -97.0%       2.25 ±101%  interrupts.CPU1.TLB:TLB_shootdowns
    105.75 ± 40%   +3409.2%       3711 ± 64%  interrupts.CPU100.NMI:Non-maskable_interrupts
    105.75 ± 40%   +3409.2%       3711 ± 64%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
    159.00 ± 10%     +64.2%     261.00 ± 22%  interrupts.CPU100.RES:Rescheduling_interrupts
      1558 ±163%    +190.8%       4532 ± 53%  interrupts.CPU101.NMI:Non-maskable_interrupts
      1558 ±163%    +190.8%       4532 ± 53%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
    179.25 ±  6%     +48.5%     266.25 ± 12%  interrupts.CPU101.RES:Rescheduling_interrupts
    143.00 ± 82%   +2066.4%       3098 ± 83%  interrupts.CPU102.NMI:Non-maskable_interrupts
    143.00 ± 82%   +2066.4%       3098 ± 83%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
    173.50 ± 19%     +66.9%     289.50 ±  9%  interrupts.CPU102.RES:Rescheduling_interrupts
    193.75 ±  6%     +68.5%     326.50 ± 17%  interrupts.CPU103.RES:Rescheduling_interrupts
      3261 ±140%     -96.0%     129.00 ±128%  interrupts.CPU12.RES:Rescheduling_interrupts
    398.25 ±110%     -99.4%       2.50 ±103%  interrupts.CPU16.TLB:TLB_shootdowns
    146.25 ±160%     -97.9%       3.00 ± 62%  interrupts.CPU26.TLB:TLB_shootdowns
    372.25 ± 79%     -99.0%       3.75 ± 51%  interrupts.CPU29.TLB:TLB_shootdowns
    101.50 ±107%     -97.0%       3.00 ± 84%  interrupts.CPU3.TLB:TLB_shootdowns
     98.75 ± 21%   +6523.3%       6540 ±168%  interrupts.CPU31.RES:Rescheduling_interrupts
     94.50 ± 38%    +110.3%     198.75 ± 22%  interrupts.CPU33.RES:Rescheduling_interrupts
    100.25 ± 38%    +114.2%     214.75 ± 31%  interrupts.CPU34.RES:Rescheduling_interrupts
     81.50 ± 18%    +140.8%     196.25 ± 49%  interrupts.CPU35.RES:Rescheduling_interrupts
     88.75 ± 45%   +5932.7%       5354 ±167%  interrupts.CPU36.RES:Rescheduling_interrupts
     71.25 ± 26%    +111.2%     150.50 ± 16%  interrupts.CPU38.RES:Rescheduling_interrupts
     65.25 ± 28%    +136.8%     154.50 ± 20%  interrupts.CPU39.RES:Rescheduling_interrupts
     45.75 ± 12%    +288.0%     177.50 ± 51%  interrupts.CPU40.RES:Rescheduling_interrupts
     75.00 ± 21%     +69.0%     126.75 ±  7%  interrupts.CPU41.RES:Rescheduling_interrupts
    618.25 ±144%    +554.8%       4048 ± 51%  interrupts.CPU42.NMI:Non-maskable_interrupts
    618.25 ±144%    +554.8%       4048 ± 51%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
     80.25 ± 16%    +132.1%     186.25 ± 23%  interrupts.CPU42.RES:Rescheduling_interrupts
    861.75 ± 99%    +596.3%       6000 ± 30%  interrupts.CPU43.NMI:Non-maskable_interrupts
    861.75 ± 99%    +596.3%       6000 ± 30%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
     69.25 ± 17%    +173.6%     189.50 ± 10%  interrupts.CPU43.RES:Rescheduling_interrupts
     96.25 ± 50%     +99.0%     191.50 ± 13%  interrupts.CPU44.RES:Rescheduling_interrupts
      1133 ± 91%    +246.8%       3931 ± 27%  interrupts.CPU45.NMI:Non-maskable_interrupts
      1133 ± 91%    +246.8%       3931 ± 27%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    617.00 ±146%    +614.5%       4408 ± 38%  interrupts.CPU47.NMI:Non-maskable_interrupts
    617.00 ±146%    +614.5%       4408 ± 38%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
    617.00 ±146%    +945.3%       6449 ± 22%  interrupts.CPU49.NMI:Non-maskable_interrupts
    617.00 ±146%    +945.3%       6449 ± 22%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
     81.00 ± 55%   +2323.5%       1963 ±159%  interrupts.CPU49.RES:Rescheduling_interrupts
      3679 ± 63%     -96.8%     116.25 ± 40%  interrupts.CPU5.NMI:Non-maskable_interrupts
      3679 ± 63%     -96.8%     116.25 ± 40%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      3501 ± 37%     -96.4%     124.50 ± 43%  interrupts.CPU57.NMI:Non-maskable_interrupts
      3501 ± 37%     -96.4%     124.50 ± 43%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      4911 ±  6%     -72.3%       1362 ±157%  interrupts.CPU62.NMI:Non-maskable_interrupts
      4911 ±  6%     -72.3%       1362 ±157%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      4086 ± 36%     -67.1%       1342 ±157%  interrupts.CPU64.NMI:Non-maskable_interrupts
      4086 ± 36%     -67.1%       1342 ±157%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
    245.25 ± 44%     -85.4%      35.75 ± 97%  interrupts.CPU7.RES:Rescheduling_interrupts
     63.00 ± 71%     -94.4%       3.50 ± 95%  interrupts.CPU77.RES:Rescheduling_interrupts
      2334 ± 51%    +217.0%       7399 ±  6%  interrupts.CPU78.NMI:Non-maskable_interrupts
      2334 ± 51%    +217.0%       7399 ±  6%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
    132.75 ± 21%    +157.6%     342.00 ± 22%  interrupts.CPU78.RES:Rescheduling_interrupts
    128.25 ± 27%    +103.5%     261.00 ± 14%  interrupts.CPU79.RES:Rescheduling_interrupts
      3236 ±143%     -99.0%      33.00 ±101%  interrupts.CPU8.RES:Rescheduling_interrupts
      2481 ± 43%    +179.1%       6924 ± 17%  interrupts.CPU80.NMI:Non-maskable_interrupts
      2481 ± 43%    +179.1%       6924 ± 17%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
    132.50 ±  9%    +100.9%     266.25 ± 14%  interrupts.CPU80.RES:Rescheduling_interrupts
    131.50 ± 33%     +90.3%     250.25 ± 20%  interrupts.CPU81.RES:Rescheduling_interrupts
      1570 ± 63%    +371.1%       7400 ±  6%  interrupts.CPU82.NMI:Non-maskable_interrupts
      1570 ± 63%    +371.1%       7400 ±  6%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
    131.75 ± 13%   +2020.3%       2793 ±159%  interrupts.CPU82.RES:Rescheduling_interrupts
    769.50 ±130%     -87.3%      97.50 ±105%  interrupts.CPU9.RES:Rescheduling_interrupts
    147.50 ± 19%     +79.3%     264.50 ± 21%  interrupts.CPU90.RES:Rescheduling_interrupts
    180.50 ±  9%   +1677.8%       3209 ±158%  interrupts.CPU91.RES:Rescheduling_interrupts
    164.50 ± 20%    +119.1%     360.50 ± 26%  interrupts.CPU93.RES:Rescheduling_interrupts
      1573 ±162%    +307.6%       6411 ± 23%  interrupts.CPU94.NMI:Non-maskable_interrupts
      1573 ±162%    +307.6%       6411 ± 23%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
    148.25 ± 20%     +76.9%     262.25 ± 17%  interrupts.CPU94.RES:Rescheduling_interrupts
    143.75 ±  8%     +68.5%     242.25 ± 16%  interrupts.CPU95.RES:Rescheduling_interrupts
    168.75 ±  6%    +127.0%     383.00 ± 48%  interrupts.CPU96.RES:Rescheduling_interrupts
      2281 ±106%    +181.2%       6415 ± 23%  interrupts.CPU97.NMI:Non-maskable_interrupts
      2281 ±106%    +181.2%       6415 ± 23%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
    163.75 ± 17%     +71.3%     280.50 ± 16%  interrupts.CPU97.RES:Rescheduling_interrupts
     84.75 ± 27%   +5459.6%       4711 ± 64%  interrupts.CPU98.NMI:Non-maskable_interrupts
     84.75 ± 27%   +5459.6%       4711 ± 64%  interrupts.CPU98.PMI:Performance_monitoring_interrupts
    176.25 ±  8%     +91.2%     337.00 ± 23%  interrupts.CPU98.RES:Rescheduling_interrupts
    164.25 ±  8%     +61.5%     265.25 ± 20%  interrupts.CPU99.RES:Rescheduling_interrupts
    242630 ± 11%     +27.8%     310009 ± 13%  interrupts.NMI:Non-maskable_interrupts
    242630 ± 11%     +27.8%     310009 ± 13%  interrupts.PMI:Performance_monitoring_interrupts
      3842 ± 14%     -81.4%     714.00 ± 62%  interrupts.TLB:TLB_shootdowns
     39213 ±  2%     +59.4%      62496 ±  8%  softirqs.CPU1.SCHED
    112479 ±  3%     +11.6%     125580 ±  6%  softirqs.CPU10.RCU
     47544 ±  5%     +34.8%      64105 ±  6%  softirqs.CPU10.SCHED
    115525 ±  5%     +34.5%     155325 ±  5%  softirqs.CPU100.RCU
     42079 ±  4%     -34.4%      27593 ± 14%  softirqs.CPU100.SCHED
    115645 ±  4%     +33.4%     154231 ±  4%  softirqs.CPU101.RCU
     42390 ±  5%     -32.4%      28661 ±  9%  softirqs.CPU101.SCHED
    113237 ±  5%     +37.9%     156145 ±  4%  softirqs.CPU102.RCU
     43877 ±  3%     -42.3%      25316 ± 19%  softirqs.CPU102.SCHED
    115048 ±  4%     +35.3%     155692 ±  6%  softirqs.CPU103.RCU
     42057 ±  2%     -34.8%      27428 ± 19%  softirqs.CPU103.SCHED
     50287 ±  2%     +26.8%      63739 ±  2%  softirqs.CPU12.SCHED
     51055 ±  3%     +24.5%      63567 ±  2%  softirqs.CPU13.SCHED
    111557 ±  4%     +15.1%     128441 ±  6%  softirqs.CPU14.RCU
     51201 ±  8%     +21.9%      62437 ±  6%  softirqs.CPU14.SCHED
    106509 ±  4%     +31.4%     139927 ±  7%  softirqs.CPU15.RCU
    106554 ±  5%     +31.4%     140038 ± 10%  softirqs.CPU16.RCU
    102888 ±  3%     +22.9%     126467 ±  7%  softirqs.CPU17.RCU
    102645 ±  4%     +23.8%     127097 ±  8%  softirqs.CPU18.RCU
    103074 ±  3%     +27.2%     131067 ±  8%  softirqs.CPU19.RCU
     40123 ±  2%     +64.7%      66094 ± 16%  softirqs.CPU2.SCHED
    105947 ±  4%     +31.3%     139080 ± 11%  softirqs.CPU20.RCU
     56698 ±  3%     +10.3%      62534 ±  4%  softirqs.CPU20.SCHED
    107845 ±  4%     +25.5%     135380 ±  7%  softirqs.CPU21.RCU
    104635 ±  2%     +22.9%     128563 ±  7%  softirqs.CPU22.RCU
     57459 ±  2%      +9.0%      62636 ±  4%  softirqs.CPU22.SCHED
    107558 ±  4%     +26.9%     136472 ±  9%  softirqs.CPU23.RCU
    103404 ±  2%     +21.9%     126077 ±  9%  softirqs.CPU24.RCU
     57845            +8.9%      63018 ±  3%  softirqs.CPU24.SCHED
    103795 ±  3%     +27.4%     132281 ± 10%  softirqs.CPU25.RCU
    116855 ±  3%     +39.2%     162714 ±  3%  softirqs.CPU26.RCU
     40401 ±  3%     +14.5%      46275 ± 10%  softirqs.CPU26.SCHED
    117905 ±  5%     +38.2%     163001 ±  3%  softirqs.CPU27.RCU
    119161           +36.5%     162677 ±  2%  softirqs.CPU28.RCU
    155833 ±  4%     +31.8%     205398 ± 11%  softirqs.CPU28.TIMER
    120298 ±  6%     +39.4%     167637 ±  3%  softirqs.CPU29.RCU
    165459 ±  5%     +28.9%     213346 ±  9%  softirqs.CPU29.TIMER
     41776 ±  2%     +47.3%      61528 ± 12%  softirqs.CPU3.SCHED
    111332 ±  3%     +27.8%     142261 ±  3%  softirqs.CPU30.RCU
    109782 ±  4%     +19.3%     130961 ±  4%  softirqs.CPU31.RCU
    163967 ±  3%     +27.8%     209524 ± 10%  softirqs.CPU31.TIMER
    110345 ±  4%     +25.3%     138304 ± 12%  softirqs.CPU32.RCU
    110912 ±  4%     +27.2%     141121 ±  4%  softirqs.CPU33.RCU
    111899 ±  5%     +27.5%     142697 ±  8%  softirqs.CPU34.RCU
    164801 ±  4%     +25.8%     207277 ± 10%  softirqs.CPU34.TIMER
    110069 ±  3%     +27.6%     140414 ± 11%  softirqs.CPU35.RCU
    166154 ±  2%     +26.4%     210021 ±  9%  softirqs.CPU35.TIMER
    107927 ±  3%     +26.8%     136882 ±  4%  softirqs.CPU36.RCU
    167316 ±  2%     +23.4%     206530 ± 11%  softirqs.CPU36.TIMER
    107149 ±  3%     +28.0%     137185 ±  7%  softirqs.CPU37.RCU
     52823 ±  5%     -24.0%      40128 ± 16%  softirqs.CPU37.SCHED
    172148 ±  2%     +20.5%     207427 ± 11%  softirqs.CPU37.TIMER
    106831 ±  4%     +18.7%     126777 ±  6%  softirqs.CPU38.RCU
    169088 ±  3%     +24.9%     211164 ±  9%  softirqs.CPU38.TIMER
    109008 ±  4%     +29.2%     140872 ± 10%  softirqs.CPU39.RCU
     52842 ±  4%     -12.9%      46051 ±  3%  softirqs.CPU39.SCHED
    168352 ±  2%     +25.8%     211848 ±  8%  softirqs.CPU39.TIMER
     43344 ±  5%     +40.9%      61065 ±  9%  softirqs.CPU4.SCHED
    109019 ±  4%     +29.5%     141157 ±  8%  softirqs.CPU40.RCU
     53829 ±  2%     -23.8%      41002 ± 25%  softirqs.CPU40.SCHED
    107752 ±  9%     +19.7%     128985 ±  5%  softirqs.CPU41.RCU
     57665           -16.3%      48293 ±  5%  softirqs.CPU41.SCHED
    104372 ±  4%     +29.1%     134762 ±  6%  softirqs.CPU42.RCU
     56657           -22.2%      44093 ±  8%  softirqs.CPU42.SCHED
    176519 ±  2%     +18.9%     209864 ±  9%  softirqs.CPU42.TIMER
    102140 ±  4%     +30.8%     133570 ±  3%  softirqs.CPU43.RCU
     57875 ±  2%     -28.5%      41392 ±  6%  softirqs.CPU43.SCHED
    177283 ±  3%     +17.1%     207557 ±  9%  softirqs.CPU43.TIMER
    102750 ±  4%     +27.6%     131155 ±  6%  softirqs.CPU44.RCU
     57039 ±  5%     -22.0%      44481 ±  7%  softirqs.CPU44.SCHED
    180258 ±  3%     +17.0%     210978 ±  8%  softirqs.CPU44.TIMER
    103951 ±  3%     +47.1%     152906 ±  2%  softirqs.CPU45.RCU
     57950 ±  2%     -19.4%      46698 ±  7%  softirqs.CPU45.SCHED
    180059 ±  2%     +17.5%     211546 ±  8%  softirqs.CPU45.TIMER
    106543 ±  4%     +46.6%     156185 ±  3%  softirqs.CPU46.RCU
     58936 ±  3%     -24.8%      44309 ±  6%  softirqs.CPU46.SCHED
    175875 ±  2%     +16.8%     205464 ± 11%  softirqs.CPU46.TIMER
    107599 ±  4%     +45.6%     156681 ±  2%  softirqs.CPU47.RCU
     61135 ± 11%     -26.8%      44767 ±  9%  softirqs.CPU47.SCHED
    175037 ±  3%     +18.6%     207548 ± 10%  softirqs.CPU47.TIMER
    104196 ±  4%     +47.8%     154014 ±  3%  softirqs.CPU48.RCU
     56551 ±  6%     -19.3%      45611 ± 16%  softirqs.CPU48.SCHED
    178334 ±  5%     +18.4%     211119 ±  8%  softirqs.CPU48.TIMER
    105321 ±  4%     +46.8%     154628 ±  3%  softirqs.CPU49.RCU
    176284 ±  2%     +18.9%     209524 ±  9%  softirqs.CPU49.TIMER
     43643 ±  3%     +40.7%      61423 ± 11%  softirqs.CPU5.SCHED
    104189 ±  4%     +46.2%     152326 ±  3%  softirqs.CPU50.RCU
     58720 ±  2%     -15.3%      49718 ± 11%  softirqs.CPU50.SCHED
    180182 ±  3%     +16.7%     210254 ±  9%  softirqs.CPU50.TIMER
    105511 ±  3%     +46.2%     154243 ±  2%  softirqs.CPU51.RCU
    106737 ±  4%     +39.1%     148420 ± 10%  softirqs.CPU52.RCU
    108068 ±  4%     +35.5%     146416 ± 11%  softirqs.CPU53.RCU
     50659 ±  3%     +21.0%      61284 ±  9%  softirqs.CPU53.SCHED
    110776 ±  5%     +29.5%     143423 ±  8%  softirqs.CPU54.RCU
     49820 ±  3%     +24.6%      62062 ±  6%  softirqs.CPU54.SCHED
    108892 ±  4%     +29.6%     141149 ±  9%  softirqs.CPU55.RCU
     48807 ±  3%     +28.1%      62502 ±  6%  softirqs.CPU55.SCHED
    109303 ±  4%     +30.2%     142335 ± 10%  softirqs.CPU56.RCU
     48744 ±  4%     +29.7%      63244 ±  4%  softirqs.CPU56.SCHED
    107665 ±  4%     +32.4%     142500 ±  8%  softirqs.CPU57.RCU
     46939 ±  4%     +32.8%      62320 ±  6%  softirqs.CPU57.SCHED
    107899 ±  5%     +32.1%     142541 ±  9%  softirqs.CPU58.RCU
     47890 ±  6%     +29.7%      62112 ± 10%  softirqs.CPU58.SCHED
    112252 ±  5%     +31.6%     147720 ±  9%  softirqs.CPU59.RCU
     45201 ±  5%     +33.9%      60536 ± 11%  softirqs.CPU59.SCHED
     43271 ±  3%     +44.9%      62698 ±  5%  softirqs.CPU6.SCHED
     42947 ±  4%     +41.6%      60815 ± 10%  softirqs.CPU60.SCHED
     41118 ±  3%     +48.2%      60919 ± 10%  softirqs.CPU61.SCHED
     43558 ±  9%     +41.0%      61422 ±  8%  softirqs.CPU62.SCHED
     42404 ±  5%     +45.0%      61473 ±  8%  softirqs.CPU63.SCHED
     44782 ±  6%     +35.2%      60539 ± 11%  softirqs.CPU65.SCHED
     45270 ±  8%     +34.9%      61049 ±  9%  softirqs.CPU67.SCHED
     46135 ±  6%     +32.4%      61099 ± 12%  softirqs.CPU69.SCHED
    114120 ±  4%     +13.0%     128950 ±  9%  softirqs.CPU7.RCU
     43859 ±  4%     +42.6%      62558 ±  4%  softirqs.CPU7.SCHED
     45007 ±  4%     +42.1%      63939 ±  2%  softirqs.CPU70.SCHED
     46279 ±  4%     +31.2%      60706 ± 10%  softirqs.CPU71.SCHED
     47194 ±  7%     +29.5%      61097 ±  9%  softirqs.CPU72.SCHED
     45037 ±  4%     +37.5%      61916 ±  7%  softirqs.CPU73.SCHED
     44473 ±  7%     +38.4%      61533 ±  8%  softirqs.CPU74.SCHED
    116513 ±  5%     +32.7%     154642 ±  7%  softirqs.CPU75.RCU
     43065 ±  8%     +41.5%      60952 ± 10%  softirqs.CPU75.SCHED
    114917 ±  4%     +33.8%     153786 ±  7%  softirqs.CPU76.RCU
     44462           +40.3%      62390 ± 14%  softirqs.CPU76.SCHED
    114722 ±  3%     +34.1%     153870 ±  7%  softirqs.CPU77.RCU
     46507 ±  2%     +31.3%      61066 ±  8%  softirqs.CPU77.SCHED
    109126 ±  4%     +39.8%     152515 ±  3%  softirqs.CPU78.RCU
     52943 ±  4%     -44.1%      29615 ± 22%  softirqs.CPU78.SCHED
    113274 ±  2%     +34.9%     152829 ±  3%  softirqs.CPU79.RCU
     49330 ±  4%     -39.3%      29950 ± 19%  softirqs.CPU79.SCHED
    114525 ±  4%     +10.7%     126743 ±  6%  softirqs.CPU8.RCU
     45993 ±  5%     +35.6%      62367 ±  5%  softirqs.CPU8.SCHED
    107326 ±  4%     +40.3%     150622 ±  3%  softirqs.CPU80.RCU
     51636 ±  4%     -40.0%      30997 ± 16%  softirqs.CPU80.SCHED
    103564 ±  4%     +48.3%     153578 ±  3%  softirqs.CPU81.RCU
     54621 ±  3%     -45.0%      30058 ± 18%  softirqs.CPU81.SCHED
    106692 ±  5%     +38.1%     147383 ±  6%  softirqs.CPU82.RCU
    107429 ±  4%     +38.3%     148549 ±  3%  softirqs.CPU83.RCU
     48429 ±  3%     -29.7%      34026 ± 15%  softirqs.CPU83.SCHED
    109037 ±  3%     +30.6%     142391 ± 11%  softirqs.CPU84.RCU
    111642 ±  5%     +30.6%     145830 ±  7%  softirqs.CPU85.RCU
    110949 ±  5%     +33.3%     147882 ±  3%  softirqs.CPU86.RCU
    110815 ±  5%     +31.2%     145429 ±  7%  softirqs.CPU87.RCU
    111916 ±  5%     +34.2%     150213 ±  3%  softirqs.CPU88.RCU
     42154 ±  5%     -22.1%      32856 ± 15%  softirqs.CPU88.SCHED
    111872 ±  4%     +33.9%     149766 ±  3%  softirqs.CPU89.RCU
     47628 ±  3%     +32.5%      63122 ±  3%  softirqs.CPU9.SCHED
    112803 ±  5%     +38.3%     155970 ±  5%  softirqs.CPU90.RCU
     45236 ±  5%     -37.4%      28321 ± 23%  softirqs.CPU90.SCHED
    117379 ±  5%     +32.4%     155399 ±  4%  softirqs.CPU91.RCU
    116981 ±  3%     +26.6%     148101 ±  6%  softirqs.CPU92.RCU
    113561 ±  5%     +39.4%     158341 ±  4%  softirqs.CPU93.RCU
     45469 ±  3%     -43.2%      25823 ± 27%  softirqs.CPU93.SCHED
    112230 ±  5%     +38.5%     155456 ±  3%  softirqs.CPU94.RCU
    113604 ±  4%     +34.2%     152470 ±  5%  softirqs.CPU95.RCU
     45422 ±  4%     -33.9%      30021 ± 17%  softirqs.CPU95.SCHED
    112933 ±  4%     +38.3%     156199 ±  4%  softirqs.CPU96.RCU
     44320 ±  6%     -40.9%      26191 ± 20%  softirqs.CPU96.SCHED
    113319 ±  5%     +36.9%     155146 ±  6%  softirqs.CPU97.RCU
     43834 ±  2%     -40.0%      26290 ± 19%  softirqs.CPU97.SCHED
    116386 ±  6%     +32.5%     154262 ±  7%  softirqs.CPU98.RCU
     41001 ±  8%     -31.1%      28239 ± 22%  softirqs.CPU98.SCHED
    116165 ±  5%     +33.7%     155277 ±  5%  softirqs.CPU99.RCU
     41299 ±  5%     -33.7%      27372 ± 20%  softirqs.CPU99.SCHED
  11518008 ±  4%     +25.8%   14489368 ±  4%  softirqs.RCU
  17583486 ±  3%     +16.4%   20469522 ±  4%  softirqs.TIMER



***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/30%/debian-x86_64-20191114.cgz/300s/lkp-skl-fpga01/context1/unixbench/0x2000065

commit: 
  9c40365a65 ("threads: Update PID limit comment according to futex UAPI change")
  59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group")

9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     kmsg.ipmi_si_dmi-ipmi-si.#:IRQ_index#not_found
         %stddev     %change         %stddev
             \          |                \  
      3115           +29.1%       4022 ±  3%  unixbench.score
     15433 ±  2%     -36.2%       9850 ± 12%  unixbench.time.involuntary_context_switches
      2114           -15.0%       1797 ±  3%  unixbench.time.percent_of_cpu_this_job_got
      7797           -17.5%       6433 ±  3%  unixbench.time.system_time
    477.52           +24.2%     593.20        unixbench.time.user_time
  3.69e+08           +23.6%  4.561e+08 ±  2%  unixbench.time.voluntary_context_switches
 4.873e+08           +29.0%  6.288e+08 ±  3%  unixbench.workload
    747856 ±  6%     -13.1%     649552 ±  6%  meminfo.DirectMap4k
      1.40            +0.6        1.99 ±  3%  mpstat.cpu.all.usr%
    181.50 ± 14%    +205.5%     554.50 ± 61%  numa-meminfo.node0.Mlocked
     45.00 ± 14%    +207.2%     138.25 ± 61%  numa-vmstat.node0.nr_mlock
     76.00            +1.3%      77.00        vmstat.cpu.id
   3747208           +20.0%    4498355 ±  2%  vmstat.system.cs
    237644            -4.2%     227776        vmstat.system.in
      2932 ±  2%     -11.5%       2596 ±  5%  slabinfo.khugepaged_mm_slot.active_objs
      2932 ±  2%     -11.5%       2596 ±  5%  slabinfo.khugepaged_mm_slot.num_objs
      1160 ±  8%     -22.8%     895.75 ± 10%  slabinfo.task_group.active_objs
      1160 ±  8%     -22.8%     895.75 ± 10%  slabinfo.task_group.num_objs
 7.874e+09           -24.7%  5.932e+09 ±  5%  cpuidle.C1.time
 8.948e+09 ± 82%     +99.3%  1.783e+10 ± 30%  cpuidle.C1E.time
  28927686 ± 31%     +37.6%   39802094 ± 20%  cpuidle.C1E.usage
  32968312         +1040.8%  3.761e+08 ± 16%  cpuidle.POLL.time
   3321066         +3571.2%  1.219e+08 ± 17%  cpuidle.POLL.usage
     77470            -1.2%      76570        proc-vmstat.nr_active_anon
     44723            -1.6%      44026        proc-vmstat.nr_slab_unreclaimable
     77470            -1.2%      76570        proc-vmstat.nr_zone_active_anon
     11008 ± 26%     -56.0%       4844 ± 56%  proc-vmstat.numa_hint_faults
      1806 ± 10%     -87.9%     218.75 ± 52%  proc-vmstat.numa_hint_faults_local
     10143 ±  5%     -11.1%       9016 ±  3%  proc-vmstat.pgactivate
   3687542 ±  3%     -16.9%    3066128 ± 10%  sched_debug.cfs_rq:/.MIN_vruntime.max
   1247933 ±  4%     -15.6%    1053151 ± 11%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     73039           -14.6%      62394 ±  5%  sched_debug.cfs_rq:/.exec_clock.avg
     46502           -38.8%      28463 ± 28%  sched_debug.cfs_rq:/.exec_clock.min
     10133 ±  6%     +94.8%      19737 ± 27%  sched_debug.cfs_rq:/.exec_clock.stddev
     12925 ±  8%     +37.1%      17715 ± 27%  sched_debug.cfs_rq:/.load.avg
   3687542 ±  3%     -16.9%    3066129 ± 10%  sched_debug.cfs_rq:/.max_vruntime.max
   1247933 ±  4%     -15.6%    1053152 ± 11%  sched_debug.cfs_rq:/.max_vruntime.stddev
   3425325           -21.7%    2680459 ±  5%  sched_debug.cfs_rq:/.min_vruntime.avg
   2191093           -44.6%    1213462 ± 28%  sched_debug.cfs_rq:/.min_vruntime.min
    471746 ±  6%     +77.8%     838686 ± 25%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.24 ± 26%     +90.8%       0.46 ± 19%  sched_debug.cfs_rq:/.nr_spread_over.avg
      2.64 ± 36%    +151.8%       6.65 ± 55%  sched_debug.cfs_rq:/.nr_spread_over.max
      0.51 ± 26%    +137.0%       1.20 ± 44%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      1437 ±  3%     +37.1%       1970 ±  6%  sched_debug.cfs_rq:/.runnable_avg.max
    375.43 ±  2%     +15.1%     432.07 ±  4%  sched_debug.cfs_rq:/.runnable_avg.stddev
    -96957         +1058.8%   -1123518        sched_debug.cfs_rq:/.spread0.min
    471746 ±  6%     +77.8%     838694 ± 25%  sched_debug.cfs_rq:/.spread0.stddev
    858.93 ±  5%     +45.9%       1253 ± 13%  sched_debug.cfs_rq:/.util_avg.max
    649.39 ±  6%     +56.7%       1017 ± 15%  sched_debug.cfs_rq:/.util_est_enqueued.max
    442164 ±  3%     +19.3%     527581 ±  3%  sched_debug.cpu.avg_idle.avg
    514800 ±  4%     +15.5%     594740 ± 13%  sched_debug.cpu.max_idle_balance_cost.max
      1444 ±148%    +613.0%      10298 ± 72%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ±  3%     -16.2%       0.00 ±  8%  sched_debug.cpu.next_balance.stddev
      1.04 ±  5%     +81.0%       1.88 ± 22%  sched_debug.cpu.nr_running.max
      0.41           +12.1%       0.46 ±  6%  sched_debug.cpu.nr_running.stddev
   8064604 ±  2%     +33.5%   10769101 ± 11%  sched_debug.cpu.nr_switches.max
    915034 ±  6%    +206.7%    2806624 ± 32%  sched_debug.cpu.nr_switches.stddev
   8061279 ±  2%     +33.5%   10761752 ± 11%  sched_debug.cpu.sched_count.max
    914923 ±  6%    +206.7%    2806145 ± 32%  sched_debug.cpu.sched_count.stddev
   4029443 ±  2%     +29.7%    5227316 ± 11%  sched_debug.cpu.sched_goidle.max
    457354 ±  6%    +196.6%    1356319 ± 31%  sched_debug.cpu.sched_goidle.stddev
   3230031           +17.6%    3799652 ±  8%  sched_debug.cpu.ttwu_count.avg
   4032123           +37.2%    5533499 ± 11%  sched_debug.cpu.ttwu_count.max
    457261 ±  6%    +217.1%    1449903 ± 33%  sched_debug.cpu.ttwu_count.stddev
    497.05          +952.2%       5229 ± 17%  sched_debug.cpu.ttwu_local.avg
      1512          +449.7%       8311 ± 24%  sched_debug.cpu.ttwu_local.max
    354.64 ±  4%    +688.8%       2797 ± 29%  sched_debug.cpu.ttwu_local.min
    130.66 ±  6%   +1584.8%       2201 ± 49%  sched_debug.cpu.ttwu_local.stddev
     97364 ±  2%     -15.3%      82438 ± 10%  softirqs.CPU100.SCHED
     95732           -13.2%      83091 ± 10%  softirqs.CPU101.SCHED
     98020           -15.3%      83044 ± 11%  softirqs.CPU102.SCHED
    134854 ±  2%      +9.5%     147696 ±  3%  softirqs.CPU33.TIMER
    100296 ±  4%      -9.8%      90488 ±  8%  softirqs.CPU34.SCHED
     98665 ±  4%     -10.0%      88752 ±  8%  softirqs.CPU36.SCHED
    101562 ±  2%     -12.5%      88834 ±  8%  softirqs.CPU37.SCHED
    103241           -14.5%      88256 ± 10%  softirqs.CPU38.SCHED
    102016           -15.1%      86641 ±  9%  softirqs.CPU39.SCHED
    103072 ±  2%     -15.9%      86642 ± 11%  softirqs.CPU40.SCHED
    103642           -16.7%      86299 ± 10%  softirqs.CPU41.SCHED
    136281            +8.1%     147364 ±  3%  softirqs.CPU41.TIMER
    104080           -18.4%      84916 ± 11%  softirqs.CPU42.SCHED
    104161           -25.6%      77445 ± 19%  softirqs.CPU43.SCHED
    103794 ±  2%     -18.3%      84779 ± 11%  softirqs.CPU44.SCHED
    104881           -19.0%      84947 ± 12%  softirqs.CPU45.SCHED
    106249           -21.7%      83209 ± 11%  softirqs.CPU46.SCHED
    105901           -21.9%      82708 ± 12%  softirqs.CPU47.SCHED
    106109           -21.0%      83838 ± 10%  softirqs.CPU48.SCHED
    104208 ±  3%     -20.5%      82800 ± 11%  softirqs.CPU49.SCHED
    105449           -20.8%      83521 ± 10%  softirqs.CPU50.SCHED
    106650           -12.8%      92988 ±  6%  softirqs.CPU52.SCHED
    110131           -23.6%      84129 ± 11%  softirqs.CPU78.SCHED
    107950 ±  3%     -22.3%      83897 ± 11%  softirqs.CPU79.SCHED
    110414           -23.5%      84412 ± 10%  softirqs.CPU80.SCHED
    107954 ±  2%     -21.3%      84966 ± 10%  softirqs.CPU81.SCHED
    107635           -22.6%      83318 ± 13%  softirqs.CPU82.SCHED
    107434 ±  2%     -22.4%      83344 ± 11%  softirqs.CPU83.SCHED
    106151 ±  2%     -21.5%      83345 ± 12%  softirqs.CPU84.SCHED
    104935 ±  5%     -20.0%      83933 ± 11%  softirqs.CPU85.SCHED
    103760 ±  2%     -19.3%      83770 ± 11%  softirqs.CPU86.SCHED
    103250 ±  4%     -18.6%      84093 ± 11%  softirqs.CPU87.SCHED
    104851 ±  2%     -20.2%      83629 ± 12%  softirqs.CPU88.SCHED
    104007           -19.5%      83676 ± 12%  softirqs.CPU89.SCHED
    101221           -17.8%      83192 ± 11%  softirqs.CPU90.SCHED
    103382 ±  2%     -19.4%      83322 ± 11%  softirqs.CPU91.SCHED
     99759 ±  3%     -16.9%      82889 ± 12%  softirqs.CPU92.SCHED
     99723           -15.8%      84012 ± 11%  softirqs.CPU93.SCHED
     99706 ±  2%     -15.8%      83927 ± 11%  softirqs.CPU94.SCHED
    100802           -17.3%      83351 ± 11%  softirqs.CPU95.SCHED
     99622 ±  2%     -16.3%      83351 ± 10%  softirqs.CPU96.SCHED
     98714 ±  2%     -16.4%      82483 ± 11%  softirqs.CPU97.SCHED
     99272           -16.5%      82919 ± 11%  softirqs.CPU98.SCHED
     97699 ±  2%     -15.7%      82320 ± 11%  softirqs.CPU99.SCHED
     30.70 ± 17%     -41.8%      17.86 ±  2%  perf-stat.i.MPKI
 6.229e+09           +21.0%  7.537e+09 ±  2%  perf-stat.i.branch-instructions
  87521494 ±  2%     +26.1%  1.103e+08 ±  2%  perf-stat.i.branch-misses
  26161990           -61.0%   10190442 ± 20%  perf-stat.i.cache-misses
 7.094e+08            -3.2%  6.866e+08        perf-stat.i.cache-references
   3766575           +20.0%    4520959 ±  2%  perf-stat.i.context-switches
      3.51 ±  8%     -22.6%       2.72 ±  3%  perf-stat.i.cpi
 8.561e+10            +2.8%    8.8e+10        perf-stat.i.cpu-cycles
      1199 ± 10%  +12547.8%     151679 ± 18%  perf-stat.i.cpu-migrations
      3326 ±  8%   +1896.7%      66426 ± 59%  perf-stat.i.cycles-between-cache-misses
   4368880 ±  5%    +260.9%   15765930 ± 11%  perf-stat.i.dTLB-load-misses
 7.644e+09           +21.3%  9.275e+09 ±  2%  perf-stat.i.dTLB-loads
     83528 ± 41%   +1345.4%    1207309 ± 15%  perf-stat.i.dTLB-store-misses
   4.5e+09           +22.5%  5.512e+09 ±  2%  perf-stat.i.dTLB-stores
   4412580           +49.5%    6597934 ±  4%  perf-stat.i.iTLB-load-misses
  29081328           +25.9%   36603347        perf-stat.i.iTLB-loads
 2.835e+10           +21.5%  3.444e+10 ±  2%  perf-stat.i.instructions
      0.31 ±  4%     +21.5%       0.38 ±  3%  perf-stat.i.ipc
     94.83            -2.4       92.45        perf-stat.i.node-load-miss-rate%
   4078999           -58.8%    1678711 ± 18%  perf-stat.i.node-load-misses
    146125           -60.1%      58372 ± 13%  perf-stat.i.node-loads
     95.95           -10.6       85.36        perf-stat.i.node-store-miss-rate%
   3822185           -59.7%    1539813 ± 20%  perf-stat.i.node-store-misses
     11551 ±  3%      -8.5%      10571 ±  5%  perf-stat.i.node-stores
     25.02           -20.3%      19.94 ±  2%  perf-stat.overall.MPKI
      1.41            +0.1        1.46        perf-stat.overall.branch-miss-rate%
      3.69            -2.2        1.49 ± 21%  perf-stat.overall.cache-miss-rate%
      3.02           -15.4%       2.56 ±  2%  perf-stat.overall.cpi
      3272          +178.9%       9127 ± 25%  perf-stat.overall.cycles-between-cache-misses
      0.06 ±  5%      +0.1        0.17 ±  9%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 41%      +0.0        0.02 ± 13%  perf-stat.overall.dTLB-store-miss-rate%
     13.18            +2.1       15.27 ±  3%  perf-stat.overall.iTLB-load-miss-rate%
      6425           -18.7%       5224 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.33           +18.2%       0.39 ±  2%  perf-stat.overall.ipc
     22735            -5.8%      21419        perf-stat.overall.path-length
  6.21e+09           +21.1%  7.518e+09 ±  2%  perf-stat.ps.branch-instructions
  87263324 ±  2%     +26.1%  1.101e+08 ±  2%  perf-stat.ps.branch-misses
  26081704           -61.1%   10158408 ± 20%  perf-stat.ps.cache-misses
 7.072e+08            -3.2%  6.848e+08        perf-stat.ps.cache-references
   3754979           +20.1%    4509479 ±  2%  perf-stat.ps.context-switches
 8.535e+10            +2.8%  8.777e+10        perf-stat.ps.cpu-cycles
      1197 ± 10%  +12540.5%     151373 ± 18%  perf-stat.ps.cpu-migrations
   4356788 ±  5%    +261.1%   15731108 ± 11%  perf-stat.ps.dTLB-load-misses
 7.621e+09           +21.4%  9.251e+09 ±  2%  perf-stat.ps.dTLB-loads
     83463 ± 41%   +1343.5%    1204761 ± 15%  perf-stat.ps.dTLB-store-misses
 4.487e+09           +22.5%  5.498e+09 ±  2%  perf-stat.ps.dTLB-stores
   4399472           +49.6%    6581866 ±  4%  perf-stat.ps.iTLB-load-misses
  28993919           +25.9%   36508403        perf-stat.ps.iTLB-loads
 2.826e+10           +21.6%  3.436e+10 ±  2%  perf-stat.ps.instructions
   4066345           -58.8%    1673367 ± 18%  perf-stat.ps.node-load-misses
    145705           -60.0%      58229 ± 13%  perf-stat.ps.node-loads
   3810301           -59.7%    1534753 ± 20%  perf-stat.ps.node-store-misses
     11533 ±  3%      -8.5%      10555 ±  5%  perf-stat.ps.node-stores
 1.108e+13           +21.5%  1.346e+13 ±  2%  perf-stat.total.instructions
     49518            -1.5%      48784        interrupts.CAL:Function_call_interrupts
     75573 ±  2%     -21.1%      59624 ± 19%  interrupts.CPU0.RES:Rescheduling_interrupts
    293.50 ± 46%     -83.7%      47.75 ± 39%  interrupts.CPU0.TLB:TLB_shootdowns
      1651 ± 43%     -90.6%     155.75 ± 89%  interrupts.CPU1.TLB:TLB_shootdowns
    107991 ±  2%     -25.4%      80516 ± 13%  interrupts.CPU10.RES:Rescheduling_interrupts
     53.00 ± 95%     -70.3%      15.75 ± 34%  interrupts.CPU10.TLB:TLB_shootdowns
    104860 ±  3%     -45.3%      57407 ± 29%  interrupts.CPU100.RES:Rescheduling_interrupts
    101815 ±  3%     -42.4%      58608 ± 28%  interrupts.CPU101.RES:Rescheduling_interrupts
    104224 ±  3%     -45.2%      57146 ± 32%  interrupts.CPU102.RES:Rescheduling_interrupts
      3026 ±  8%     -70.6%     891.00 ±149%  interrupts.CPU103.NMI:Non-maskable_interrupts
      3026 ±  8%     -70.6%     891.00 ±149%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
     97748 ±  3%     -43.9%      54866 ± 25%  interrupts.CPU103.RES:Rescheduling_interrupts
    106274 ±  2%     -25.2%      79442 ± 18%  interrupts.CPU11.RES:Rescheduling_interrupts
    109794 ±  3%     -26.2%      81077 ± 13%  interrupts.CPU12.RES:Rescheduling_interrupts
    110641 ±  4%     -28.4%      79170 ± 17%  interrupts.CPU13.RES:Rescheduling_interrupts
    112578 ±  4%     -28.9%      80064 ± 13%  interrupts.CPU14.RES:Rescheduling_interrupts
    203.50 ±101%     -92.1%      16.00 ± 76%  interrupts.CPU14.TLB:TLB_shootdowns
    115653 ±  3%     -29.9%      81035 ± 16%  interrupts.CPU15.RES:Rescheduling_interrupts
      2336 ± 30%     +56.1%       3646 ± 24%  interrupts.CPU16.NMI:Non-maskable_interrupts
      2336 ± 30%     +56.1%       3646 ± 24%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    113166 ±  4%     -31.0%      78040 ± 14%  interrupts.CPU16.RES:Rescheduling_interrupts
    117545 ±  6%     -34.2%      77376 ± 15%  interrupts.CPU17.RES:Rescheduling_interrupts
    119816 ±  5%     -35.9%      76821 ± 18%  interrupts.CPU18.RES:Rescheduling_interrupts
    208.00 ±115%     -93.0%      14.50 ± 51%  interrupts.CPU18.TLB:TLB_shootdowns
    116298 ±  5%     -33.1%      77798 ± 18%  interrupts.CPU19.RES:Rescheduling_interrupts
    118.50 ± 93%     -87.8%      14.50 ± 51%  interrupts.CPU19.TLB:TLB_shootdowns
    471.50 ± 54%     -97.1%      13.75 ± 37%  interrupts.CPU2.TLB:TLB_shootdowns
    119931 ±  4%     -34.2%      78873 ± 16%  interrupts.CPU20.RES:Rescheduling_interrupts
    119193 ±  3%     -33.7%      79035 ± 10%  interrupts.CPU21.RES:Rescheduling_interrupts
    119705 ±  4%     -33.6%      79531 ± 21%  interrupts.CPU22.RES:Rescheduling_interrupts
    119224           -33.6%      79145 ± 15%  interrupts.CPU23.RES:Rescheduling_interrupts
    239.50 ± 82%     -94.8%      12.50 ± 47%  interrupts.CPU23.TLB:TLB_shootdowns
    114268 ±  5%     -32.0%      77747 ± 18%  interrupts.CPU24.RES:Rescheduling_interrupts
    130.75 ± 54%     -87.2%      16.75 ± 80%  interrupts.CPU24.TLB:TLB_shootdowns
    114570 ±  2%     -27.4%      83205 ± 15%  interrupts.CPU25.RES:Rescheduling_interrupts
    401.00 ± 34%     -90.8%      36.75 ± 37%  interrupts.CPU26.TLB:TLB_shootdowns
      2215 ± 26%     -81.2%     417.25 ± 96%  interrupts.CPU27.TLB:TLB_shootdowns
      1056 ± 71%     -76.4%     249.75 ± 54%  interrupts.CPU29.TLB:TLB_shootdowns
     94273 ±  4%     -14.7%      80458 ± 17%  interrupts.CPU3.RES:Rescheduling_interrupts
    438.75 ± 54%     -80.6%      85.25 ±138%  interrupts.CPU3.TLB:TLB_shootdowns
    297.75 ± 52%     -69.4%      91.00 ±123%  interrupts.CPU30.TLB:TLB_shootdowns
     99323 ±  4%     -17.3%      82125 ± 18%  interrupts.CPU31.RES:Rescheduling_interrupts
    101813 ±  4%     -22.1%      79310 ± 22%  interrupts.CPU32.RES:Rescheduling_interrupts
    104108 ±  4%     -23.9%      79216 ± 21%  interrupts.CPU33.RES:Rescheduling_interrupts
    108408 ±  6%     -30.2%      75649 ± 19%  interrupts.CPU34.RES:Rescheduling_interrupts
    107851 ±  7%     -30.2%      75255 ± 22%  interrupts.CPU35.RES:Rescheduling_interrupts
      3050 ±  5%     -38.5%       1875 ± 56%  interrupts.CPU36.NMI:Non-maskable_interrupts
      3050 ±  5%     -38.5%       1875 ± 56%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    110860 ±  5%     -36.0%      70961 ± 21%  interrupts.CPU36.RES:Rescheduling_interrupts
      2994 ±  5%     -47.5%       1573 ± 56%  interrupts.CPU37.NMI:Non-maskable_interrupts
      2994 ±  5%     -47.5%       1573 ± 56%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
    110701 ±  6%     -35.9%      70912 ± 21%  interrupts.CPU37.RES:Rescheduling_interrupts
      2950 ±  3%     -60.7%       1158 ± 70%  interrupts.CPU38.NMI:Non-maskable_interrupts
      2950 ±  3%     -60.7%       1158 ± 70%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    114683 ±  2%     -39.3%      69605 ± 27%  interrupts.CPU38.RES:Rescheduling_interrupts
      2959 ±  3%     -53.1%       1387 ± 64%  interrupts.CPU39.NMI:Non-maskable_interrupts
      2959 ±  3%     -53.1%       1387 ± 64%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
    112794 ±  3%     -41.0%      66597 ± 21%  interrupts.CPU39.RES:Rescheduling_interrupts
     96299 ±  6%     -19.1%      77895 ± 15%  interrupts.CPU4.RES:Rescheduling_interrupts
      3084 ±  6%     -62.4%       1158 ± 67%  interrupts.CPU40.NMI:Non-maskable_interrupts
      3084 ±  6%     -62.4%       1158 ± 67%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    115740 ±  6%     -42.7%      66275 ± 28%  interrupts.CPU40.RES:Rescheduling_interrupts
    336.75 ± 73%     -94.3%      19.25 ± 40%  interrupts.CPU40.TLB:TLB_shootdowns
      3109 ±  5%     -59.0%       1273 ± 57%  interrupts.CPU41.NMI:Non-maskable_interrupts
      3109 ±  5%     -59.0%       1273 ± 57%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
    117515 ±  3%     -45.4%      64133 ± 26%  interrupts.CPU41.RES:Rescheduling_interrupts
      3074 ± 12%     -64.3%       1097 ± 89%  interrupts.CPU42.NMI:Non-maskable_interrupts
      3074 ± 12%     -64.3%       1097 ± 89%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
    118729 ±  3%     -47.7%      62131 ± 30%  interrupts.CPU42.RES:Rescheduling_interrupts
      3026 ±  7%     -70.3%     898.50 ± 83%  interrupts.CPU43.NMI:Non-maskable_interrupts
      3026 ±  7%     -70.3%     898.50 ± 83%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
    116750 ±  3%     -48.2%      60473 ± 29%  interrupts.CPU43.RES:Rescheduling_interrupts
      2999 ± 13%     -71.5%     853.50 ±116%  interrupts.CPU44.NMI:Non-maskable_interrupts
      2999 ± 13%     -71.5%     853.50 ±116%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
    119184           -48.6%      61233 ± 30%  interrupts.CPU44.RES:Rescheduling_interrupts
    119035 ±  2%     -48.7%      61050 ± 32%  interrupts.CPU45.RES:Rescheduling_interrupts
    122760 ±  3%     -52.1%      58842 ± 30%  interrupts.CPU46.RES:Rescheduling_interrupts
    122662 ±  2%     -53.5%      57035 ± 33%  interrupts.CPU47.RES:Rescheduling_interrupts
    123349 ±  2%     -51.7%      59540 ± 29%  interrupts.CPU48.RES:Rescheduling_interrupts
    123.00 ±107%     -88.6%      14.00 ± 36%  interrupts.CPU48.TLB:TLB_shootdowns
    123038 ±  2%     -53.4%      57374 ± 32%  interrupts.CPU49.RES:Rescheduling_interrupts
    174.50 ± 70%     -92.3%      13.50 ± 24%  interrupts.CPU49.TLB:TLB_shootdowns
     94491 ±  3%     -16.2%      79161 ± 17%  interrupts.CPU5.RES:Rescheduling_interrupts
    120948 ±  2%     -51.3%      58853 ± 28%  interrupts.CPU50.RES:Rescheduling_interrupts
    121202 ±  3%     -53.5%      56355 ± 35%  interrupts.CPU51.RES:Rescheduling_interrupts
    127597 ±  3%     -32.8%      85784 ± 14%  interrupts.CPU52.RES:Rescheduling_interrupts
    122288 ±  4%     -33.4%      81461 ± 14%  interrupts.CPU53.RES:Rescheduling_interrupts
    119757 ±  3%     -32.0%      81484 ± 19%  interrupts.CPU54.RES:Rescheduling_interrupts
    232.50 ± 97%     -94.3%      13.25 ± 86%  interrupts.CPU54.TLB:TLB_shootdowns
    114657 ±  4%     -27.8%      82728 ± 16%  interrupts.CPU55.RES:Rescheduling_interrupts
     91.50 ±132%     -84.7%      14.00 ± 59%  interrupts.CPU55.TLB:TLB_shootdowns
    112537 ±  6%     -28.0%      80980 ± 19%  interrupts.CPU56.RES:Rescheduling_interrupts
    180.00 ± 94%     -95.1%       8.75 ± 74%  interrupts.CPU56.TLB:TLB_shootdowns
    110747 ±  4%     -26.4%      81457 ± 17%  interrupts.CPU57.RES:Rescheduling_interrupts
    309.25 ± 99%     -97.1%       9.00 ± 73%  interrupts.CPU57.TLB:TLB_shootdowns
    108297           -26.4%      79689 ± 20%  interrupts.CPU58.RES:Rescheduling_interrupts
     96801 ±  2%     -18.7%      78664 ± 16%  interrupts.CPU6.RES:Rescheduling_interrupts
    104449 ±  5%     -21.7%      81825 ± 18%  interrupts.CPU60.RES:Rescheduling_interrupts
    150.50 ± 85%     -93.7%       9.50 ± 28%  interrupts.CPU60.TLB:TLB_shootdowns
    103863 ±  5%     -21.9%      81091 ± 19%  interrupts.CPU61.RES:Rescheduling_interrupts
     99849 ±  5%     -18.8%      81035 ± 19%  interrupts.CPU62.RES:Rescheduling_interrupts
     99399 ±  4%     -18.8%      80732 ± 16%  interrupts.CPU63.RES:Rescheduling_interrupts
    134.50 ± 88%     -92.8%       9.75 ± 38%  interrupts.CPU64.TLB:TLB_shootdowns
     94400 ±  3%     -21.2%      74427 ± 22%  interrupts.CPU66.RES:Rescheduling_interrupts
     96.75 ±136%     -89.1%      10.50 ± 86%  interrupts.CPU66.TLB:TLB_shootdowns
     93400 ±  6%     -21.2%      73562 ± 23%  interrupts.CPU68.RES:Rescheduling_interrupts
    147.25 ±147%     -94.2%       8.50 ± 55%  interrupts.CPU69.TLB:TLB_shootdowns
    101127 ±  6%     -21.5%      79428 ± 17%  interrupts.CPU7.RES:Rescheduling_interrupts
     43.25 ±101%     -76.9%      10.00 ± 86%  interrupts.CPU71.TLB:TLB_shootdowns
     93.00 ±134%     -90.6%       8.75 ± 45%  interrupts.CPU73.TLB:TLB_shootdowns
     73.50 ±129%     -86.4%      10.00 ± 57%  interrupts.CPU74.TLB:TLB_shootdowns
     78700 ±  4%     -18.1%      64461 ± 26%  interrupts.CPU75.RES:Rescheduling_interrupts
     82332 ±  5%     -23.2%      63213 ± 19%  interrupts.CPU76.RES:Rescheduling_interrupts
     80750 ±  3%     -20.5%      64217 ± 21%  interrupts.CPU77.RES:Rescheduling_interrupts
    167.00 ± 86%     -53.9%      77.00 ±158%  interrupts.CPU77.TLB:TLB_shootdowns
    132925 ±  2%     -55.5%      59127 ± 32%  interrupts.CPU78.RES:Rescheduling_interrupts
    130753 ±  3%     -52.1%      62630 ± 28%  interrupts.CPU79.RES:Rescheduling_interrupts
      2129 ± 28%    +106.7%       4402 ± 20%  interrupts.CPU8.NMI:Non-maskable_interrupts
      2129 ± 28%    +106.7%       4402 ± 20%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    102071 ±  3%     -20.1%      81537 ± 15%  interrupts.CPU8.RES:Rescheduling_interrupts
    130307 ±  5%     -54.5%      59229 ± 29%  interrupts.CPU80.RES:Rescheduling_interrupts
    130176 ±  3%     -52.9%      61349 ± 29%  interrupts.CPU81.RES:Rescheduling_interrupts
    125592 ±  2%     -52.8%      59330 ± 35%  interrupts.CPU82.RES:Rescheduling_interrupts
    126379 ±  2%     -53.4%      58871 ± 30%  interrupts.CPU83.RES:Rescheduling_interrupts
    122387 ±  4%     -52.3%      58375 ± 32%  interrupts.CPU84.RES:Rescheduling_interrupts
    123520 ±  5%     -51.6%      59756 ± 29%  interrupts.CPU85.RES:Rescheduling_interrupts
    122586 ±  6%     -50.6%      60556 ± 30%  interrupts.CPU86.RES:Rescheduling_interrupts
    119406 ±  5%     -49.8%      59963 ± 29%  interrupts.CPU87.RES:Rescheduling_interrupts
    119861 ±  4%     -51.1%      58631 ± 33%  interrupts.CPU88.RES:Rescheduling_interrupts
    117115 ±  4%     -50.0%      58574 ± 32%  interrupts.CPU89.RES:Rescheduling_interrupts
    104723 ±  2%     -25.1%      78456 ± 20%  interrupts.CPU9.RES:Rescheduling_interrupts
    113197 ±  3%     -49.1%      57667 ± 31%  interrupts.CPU90.RES:Rescheduling_interrupts
    117511 ±  3%     -49.1%      59777 ± 31%  interrupts.CPU91.RES:Rescheduling_interrupts
    113502 ±  2%     -48.6%      58311 ± 35%  interrupts.CPU92.RES:Rescheduling_interrupts
    111247           -46.3%      59729 ± 29%  interrupts.CPU93.RES:Rescheduling_interrupts
    109067 ±  4%     -45.0%      60027 ± 29%  interrupts.CPU94.RES:Rescheduling_interrupts
    109639 ±  2%     -46.4%      58724 ± 29%  interrupts.CPU95.RES:Rescheduling_interrupts
    107661 ±  3%     -45.2%      59031 ± 29%  interrupts.CPU96.RES:Rescheduling_interrupts
      2987 ±  7%     -69.8%     900.75 ±117%  interrupts.CPU97.NMI:Non-maskable_interrupts
      2987 ±  7%     -69.8%     900.75 ±117%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
    107034 ±  2%     -47.6%      56109 ± 29%  interrupts.CPU97.RES:Rescheduling_interrupts
     96.00 ±125%     -86.5%      13.00 ± 12%  interrupts.CPU97.TLB:TLB_shootdowns
    107585 ±  2%     -46.3%      57788 ± 31%  interrupts.CPU98.RES:Rescheduling_interrupts
    104668 ±  5%     -46.5%      55991 ± 33%  interrupts.CPU99.RES:Rescheduling_interrupts
     93.25 ±108%     -84.2%      14.75 ± 14%  interrupts.CPU99.TLB:TLB_shootdowns
  11221741           -34.2%    7378335 ±  9%  interrupts.RES:Rescheduling_interrupts
     19833 ± 39%     -56.4%       8648 ± 38%  interrupts.TLB:TLB_shootdowns
     30.31            -8.5       21.77 ± 28%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
     31.04            -8.5       22.53 ± 27%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
     34.59            -6.9       27.65 ± 18%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
     36.44            -6.3       30.10 ± 15%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
     36.85            -6.1       30.70 ± 14%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
     36.88            -6.1       30.77 ± 14%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
     36.89            -6.1       30.79 ± 14%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     39.76            -5.9       33.83 ± 14%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     40.65            -4.6       36.05 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     40.72            -4.6       36.15 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     38.95            -4.4       34.52 ±  8%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
     39.08            -4.4       34.68 ±  8%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
     39.31            -4.4       34.93 ±  8%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
     39.59            -4.3       35.27 ±  8%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
     40.70            -3.9       36.76 ±  7%  perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     40.77            -3.9       36.87 ±  7%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     46.74            -1.8       44.95 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     46.77            -1.8       45.00 ±  2%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     46.78            -1.8       45.02 ±  2%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     47.29            -1.5       45.81 ±  2%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      6.01 ±  7%      -1.3        4.68 ±  9%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      6.03 ±  7%      -1.3        4.73 ±  9%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      6.17 ±  7%      -1.2        4.99 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      6.16 ±  6%      -1.2        4.98 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      6.33 ±  6%      -1.1        5.26 ±  9%  perf-profile.calltrace.cycles-pp.write
      0.53 ±  2%      +0.1        0.63 ± 14%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read
      0.55 ±  2%      +0.2        0.74 ± 23%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.70 ±  2%      +0.2        0.91 ± 19%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      0.70 ±  2%      +0.2        0.93 ± 21%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
      0.54            +0.2        0.79 ± 24%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule
      1.16            +0.3        1.43 ± 11%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle
      0.64 ±  4%      +0.3        0.95 ± 22%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.78 ±  3%      +0.3        1.12 ± 21%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      1.53            +0.4        1.94 ± 10%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      1.02 ±  2%      +0.4        1.43 ± 21%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      1.27 ±  2%      +0.5        1.73 ± 22%  perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
      0.26 ±100%      +0.6        0.85 ± 46%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.60 ±  3%      +0.7        2.28 ± 21%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.02 ±  6%      +0.8        1.83 ±  9%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.05 ±  6%      +0.8        1.87 ±  9%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.24            +0.8        2.08 ± 38%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.pipe_read
      1.22 ±  7%      +0.9        2.16 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.23 ±  7%      +0.9        2.18 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      1.44 ±  7%      +1.1        2.56 ±  9%  perf-profile.calltrace.cycles-pp.read
      1.46            +1.4        2.90 ± 25%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.pipe_read.new_sync_read
      2.73            +1.5        4.22 ± 28%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      2.78            +1.5        4.30 ± 27%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +1.6        1.55 ± 85%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      3.23 ±  6%      +1.6        4.84 ± 21%  perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
      2.29 ±  2%      +1.8        4.12 ± 23%  perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      3.08 ±  7%      +2.1        5.15 ± 26%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.pipe_read.new_sync_read.vfs_read
      3.17 ±  6%      +2.1        5.27 ± 26%  perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
      5.35            +2.2        7.56 ± 25%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.50            +2.3        7.76 ± 25%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.79            +2.7        8.48 ± 21%  perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      5.90            +2.8        8.67 ± 22%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     31.89            -8.3       23.59 ± 25%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     34.59            -6.9       27.66 ± 18%  perf-profile.children.cycles-pp.__account_scheduler_latency
     30.51            -6.9       23.60 ± 19%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     36.58            -6.3       30.26 ± 15%  perf-profile.children.cycles-pp.enqueue_entity
     36.85            -6.1       30.71 ± 14%  perf-profile.children.cycles-pp.enqueue_task_fair
     36.88            -6.1       30.77 ± 14%  perf-profile.children.cycles-pp.activate_task
     36.90            -6.1       30.80 ± 14%  perf-profile.children.cycles-pp.ttwu_do_activate
     39.93            -6.0       33.95 ± 13%  perf-profile.children.cycles-pp.intel_idle
     38.96            -4.4       34.52 ±  8%  perf-profile.children.cycles-pp.try_to_wake_up
     39.09            -4.4       34.69 ±  8%  perf-profile.children.cycles-pp.autoremove_wake_function
     41.17            -4.4       36.79 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter_state
     39.31            -4.4       34.93 ±  8%  perf-profile.children.cycles-pp.__wake_up_common
     41.18            -4.4       36.81 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter
     39.59            -4.3       35.28 ±  8%  perf-profile.children.cycles-pp.__wake_up_common_lock
     40.71            -3.9       36.77 ±  7%  perf-profile.children.cycles-pp.pipe_write
     40.79            -3.9       36.89 ±  7%  perf-profile.children.cycles-pp.new_sync_write
     41.10            -3.8       37.30 ±  6%  perf-profile.children.cycles-pp.vfs_write
     41.23            -3.7       37.53 ±  6%  perf-profile.children.cycles-pp.ksys_write
     46.78            -1.8       45.02 ±  2%  perf-profile.children.cycles-pp.start_secondary
     47.29            -1.5       45.81 ±  2%  perf-profile.children.cycles-pp.secondary_startup_64
     47.29            -1.5       45.81 ±  2%  perf-profile.children.cycles-pp.cpu_startup_entry
     47.29            -1.5       45.82 ±  2%  perf-profile.children.cycles-pp.do_idle
      6.34 ±  6%      -1.1        5.29 ±  9%  perf-profile.children.cycles-pp.write
      0.34 ±  5%      -0.1        0.29 ± 12%  perf-profile.children.cycles-pp.tick_sched_handle
      0.33 ±  5%      -0.0        0.29 ± 11%  perf-profile.children.cycles-pp.update_process_times
      0.24 ±  3%      -0.0        0.20 ± 14%  perf-profile.children.cycles-pp.scheduler_tick
      0.05 ±  8%      +0.0        0.08 ± 16%  perf-profile.children.cycles-pp.generic_pipe_buf_confirm
      0.23            +0.0        0.26 ±  4%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.12 ±  4%      +0.0        0.16 ±  7%  perf-profile.children.cycles-pp.cpus_share_cache
      0.06            +0.0        0.09 ± 26%  perf-profile.children.cycles-pp.atime_needs_update
      0.26 ±  4%      +0.0        0.30 ±  9%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.08 ±  5%      +0.0        0.12 ± 26%  perf-profile.children.cycles-pp.copyin
      0.09 ±  7%      +0.0        0.13 ± 25%  perf-profile.children.cycles-pp.stack_access_ok
      0.09            +0.0        0.13 ± 20%  perf-profile.children.cycles-pp.__calc_delta
      0.04 ± 57%      +0.0        0.08 ± 20%  perf-profile.children.cycles-pp.is_bpf_text_address
      0.23            +0.0        0.27 ±  6%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.12 ±  7%      +0.0        0.16 ± 19%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.09 ±  4%      +0.0        0.13 ± 25%  perf-profile.children.cycles-pp.touch_atime
      0.06 ±  6%      +0.0        0.10 ± 31%  perf-profile.children.cycles-pp.__might_fault
      0.03 ±100%      +0.0        0.07 ±  6%  perf-profile.children.cycles-pp.rcu_core
      0.12 ±  6%      +0.0        0.17 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.21 ±  3%      +0.0        0.26 ± 10%  perf-profile.children.cycles-pp.newidle_balance
      0.11            +0.0        0.16 ± 23%  perf-profile.children.cycles-pp.in_sched_functions
      0.09 ±  5%      +0.0        0.13 ± 26%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.01 ±173%      +0.1        0.06 ± 26%  perf-profile.children.cycles-pp.__x64_sys_write
      0.04 ± 57%      +0.1        0.09 ± 40%  perf-profile.children.cycles-pp.set_next_task_idle
      0.06 ±  6%      +0.1        0.12 ± 39%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.09 ±  7%      +0.1        0.14 ± 26%  perf-profile.children.cycles-pp.menu_reflect
      0.06 ±  7%      +0.1        0.11 ± 25%  perf-profile.children.cycles-pp.find_next_bit
      0.01 ±173%      +0.1        0.07 ± 19%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.bpf_prog_kallsyms_find
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.get_cpu_device
      0.29 ±  4%      +0.1        0.35 ±  8%  perf-profile.children.cycles-pp.copyout
      0.00            +0.1        0.06 ± 28%  perf-profile.children.cycles-pp.in_lock_functions
      0.09 ±  5%      +0.1        0.15 ± 26%  perf-profile.children.cycles-pp.current_time
      0.23            +0.1        0.29 ± 10%  perf-profile.children.cycles-pp.mutex_unlock
      0.12 ±  5%      +0.1        0.18 ± 25%  perf-profile.children.cycles-pp.fsnotify
      0.08 ±  5%      +0.1        0.15 ± 30%  perf-profile.children.cycles-pp.file_update_time
      0.20 ±  2%      +0.1        0.26 ±  3%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.07            +0.1        0.13 ± 38%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rbx
      0.04 ± 57%      +0.1        0.11 ± 24%  perf-profile.children.cycles-pp.put_prev_entity
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.14 ±  3%      +0.1        0.21 ± 21%  perf-profile.children.cycles-pp.resched_curr
      0.00            +0.1        0.07 ± 31%  perf-profile.children.cycles-pp.timestamp_truncate
      0.00            +0.1        0.07 ± 30%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.11 ±  6%      +0.1        0.18 ± 21%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.32 ±  6%      +0.1        0.39 ± 11%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.11 ±  4%      +0.1        0.19 ± 23%  perf-profile.children.cycles-pp.account_entity_dequeue
      0.00            +0.1        0.07 ± 43%  perf-profile.children.cycles-pp.__update_idle_core
      0.28 ±  3%      +0.1        0.35 ±  6%  perf-profile.children.cycles-pp.irq_exit
      0.17 ±  4%      +0.1        0.25 ± 24%  perf-profile.children.cycles-pp.__fget_light
      0.23 ±  3%      +0.1        0.31 ± 20%  perf-profile.children.cycles-pp.reweight_entity
      0.19 ±  4%      +0.1        0.27 ± 13%  perf-profile.children.cycles-pp.pick_next_entity
      0.23 ±  2%      +0.1        0.30 ± 16%  perf-profile.children.cycles-pp.check_preempt_curr
      0.27 ±  4%      +0.1        0.35 ± 14%  perf-profile.children.cycles-pp.native_write_msr
      0.11 ± 11%      +0.1        0.19 ± 21%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.11 ±  3%      +0.1        0.20 ± 25%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.38            +0.1        0.46 ±  9%  perf-profile.children.cycles-pp.___might_sleep
      0.19 ±  3%      +0.1        0.28 ± 22%  perf-profile.children.cycles-pp.__fdget_pos
      0.13 ±  9%      +0.1        0.22 ± 15%  perf-profile.children.cycles-pp.clockevents_program_event
      0.13 ±  6%      +0.1        0.23 ± 25%  perf-profile.children.cycles-pp.__might_sleep
      0.53 ±  3%      +0.1        0.63 ± 11%  perf-profile.children.cycles-pp.update_rq_clock
      0.19 ±  5%      +0.1        0.29 ± 17%  perf-profile.children.cycles-pp.read_tsc
      0.26 ±  2%      +0.1        0.36 ± 17%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.26 ±  3%      +0.1        0.36 ± 22%  perf-profile.children.cycles-pp.common_file_perm
      0.18 ±  2%      +0.1        0.28 ± 16%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.09 ±  8%      +0.1        0.19 ± 28%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.16 ±  4%      +0.1        0.26 ± 32%  perf-profile.children.cycles-pp.switch_fpu_return
      0.16 ±  2%      +0.1        0.28 ± 30%  perf-profile.children.cycles-pp.___perf_sw_event
      0.25            +0.1        0.38 ± 20%  perf-profile.children.cycles-pp.native_sched_clock
      0.19 ±  4%      +0.1        0.32 ± 31%  perf-profile.children.cycles-pp.orc_find
      0.34            +0.1        0.47 ± 16%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.29 ±  2%      +0.1        0.43 ± 21%  perf-profile.children.cycles-pp.__orc_find
      0.26            +0.1        0.40 ± 20%  perf-profile.children.cycles-pp.sched_clock
      0.24 ±  3%      +0.1        0.38 ± 27%  perf-profile.children.cycles-pp.finish_task_switch
      0.23 ±  7%      +0.1        0.37 ± 22%  perf-profile.children.cycles-pp.kernel_text_address
      0.28 ±  3%      +0.1        0.42 ± 22%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.74            +0.1        0.89 ±  6%  perf-profile.children.cycles-pp.mutex_lock
      0.23            +0.1        0.38 ± 26%  perf-profile.children.cycles-pp.copy_page_from_iter
      0.56 ±  5%      +0.2        0.71 ± 11%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.28            +0.2        0.43 ± 21%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.45 ±  3%      +0.2        0.60 ± 13%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.63            +0.2        0.79 ± 13%  perf-profile.children.cycles-pp.prepare_to_wait_event
      0.52 ±  4%      +0.2        0.69 ± 19%  perf-profile.children.cycles-pp.__switch_to
      0.39 ±  2%      +0.2        0.55 ± 18%  perf-profile.children.cycles-pp.__switch_to_asm
      0.35 ±  3%      +0.2        0.52 ± 23%  perf-profile.children.cycles-pp.security_file_permission
      0.24            +0.2        0.41 ± 32%  perf-profile.children.cycles-pp._find_next_bit
      1.16 ±  5%      +0.2        1.33 ±  5%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.35 ±  4%      +0.2        0.53 ± 28%  perf-profile.children.cycles-pp.update_cfs_group
      0.28 ±  5%      +0.2        0.45 ± 24%  perf-profile.children.cycles-pp.__kernel_text_address
      0.44            +0.2        0.65 ± 23%  perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
      0.16 ±  2%      +0.2        0.38 ± 51%  perf-profile.children.cycles-pp.available_idle_cpu
      0.33 ±  3%      +0.2        0.55 ± 26%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.43 ±  4%      +0.2        0.67 ± 24%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.35 ±  4%      +0.3        0.61 ± 13%  perf-profile.children.cycles-pp.ktime_get
      0.60 ±  2%      +0.3        0.88 ± 16%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.40 ±  2%      +0.3        0.69 ± 28%  perf-profile.children.cycles-pp.__unwind_start
      0.85            +0.3        1.17 ± 19%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.65 ±  3%      +0.3        0.97 ± 22%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      1.18            +0.3        1.50 ± 12%  perf-profile.children.cycles-pp.set_next_entity
      0.94            +0.3        1.28 ± 17%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.79 ±  3%      +0.4        1.15 ± 21%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.03 ±  2%      +0.4        1.45 ± 21%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.74 ±  2%      +0.4        1.17 ± 24%  perf-profile.children.cycles-pp.update_curr
      0.59 ±  2%      +0.5        1.13 ± 37%  perf-profile.children.cycles-pp.select_task_rq_fair
      1.77            +0.6        2.33 ± 13%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.62 ±  3%      +0.7        2.33 ± 21%  perf-profile.children.cycles-pp.menu_select
      1.73            +0.8        2.57 ± 22%  perf-profile.children.cycles-pp.unwind_next_frame
      1.52 ±  2%      +0.9        2.44 ± 24%  perf-profile.children.cycles-pp.update_load_avg
      1.51            +0.9        2.46 ± 25%  perf-profile.children.cycles-pp.dequeue_entity
      1.46 ±  6%      +1.1        2.59 ±  9%  perf-profile.children.cycles-pp.read
      1.76            +1.1        2.91 ± 25%  perf-profile.children.cycles-pp.dequeue_task_fair
      2.76            +1.4        4.17 ± 23%  perf-profile.children.cycles-pp.arch_stack_walk
      0.14 ±  5%      +1.4        1.59 ± 85%  perf-profile.children.cycles-pp.poll_idle
      3.35            +1.5        4.84 ± 21%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      2.80            +1.6        4.37 ± 28%  perf-profile.children.cycles-pp.schedule_idle
      0.55 ±  2%      +1.8        2.33 ± 71%  perf-profile.children.cycles-pp._raw_spin_lock
      3.29            +2.0        5.28 ± 26%  perf-profile.children.cycles-pp.schedule
      5.81            +2.7        8.51 ± 21%  perf-profile.children.cycles-pp.pipe_read
      5.91            +2.8        8.68 ± 22%  perf-profile.children.cycles-pp.new_sync_read
      6.38            +3.0        9.40 ± 21%  perf-profile.children.cycles-pp.vfs_read
      6.56            +3.1        9.63 ± 21%  perf-profile.children.cycles-pp.ksys_read
      5.99            +3.5        9.51 ± 27%  perf-profile.children.cycles-pp.__sched_text_start
     30.51            -6.9       23.59 ± 19%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     39.93            -6.0       33.94 ± 13%  perf-profile.self.cycles-pp.intel_idle
      0.05            +0.0        0.07 ± 13%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.08 ±  5%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.check_preempt_curr
      0.23            +0.0        0.26 ±  4%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.08            +0.0        0.11 ± 19%  perf-profile.self.cycles-pp.in_sched_functions
      0.06 ±  6%      +0.0        0.09 ± 16%  perf-profile.self.cycles-pp.schedule
      0.05            +0.0        0.08 ± 20%  perf-profile.self.cycles-pp.generic_pipe_buf_confirm
      0.05            +0.0        0.08 ± 30%  perf-profile.self.cycles-pp.__kernel_text_address
      0.05 ±  8%      +0.0        0.08 ± 27%  perf-profile.self.cycles-pp.put_task_stack
      0.12 ±  4%      +0.0        0.16 ±  7%  perf-profile.self.cycles-pp.cpus_share_cache
      0.08 ±  5%      +0.0        0.11 ± 23%  perf-profile.self.cycles-pp.stack_access_ok
      0.08 ±  5%      +0.0        0.11 ± 17%  perf-profile.self.cycles-pp.ksys_read
      0.06 ± 11%      +0.0        0.09 ± 23%  perf-profile.self.cycles-pp.kernel_text_address
      0.06 ±  6%      +0.0        0.10 ± 21%  perf-profile.self.cycles-pp.menu_reflect
      0.04 ± 58%      +0.0        0.08 ± 19%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.20 ±  4%      +0.0        0.24 ±  9%  perf-profile.self.cycles-pp.newidle_balance
      0.12 ±  6%      +0.0        0.16 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.05 ±  8%      +0.0        0.10 ± 39%  perf-profile.self.cycles-pp.unwind_get_return_address
      0.10 ±  4%      +0.0        0.15 ± 27%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.08 ±  8%      +0.0        0.13 ± 24%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.01 ±173%      +0.0        0.06 ± 20%  perf-profile.self.cycles-pp.security_file_permission
      0.05 ±  9%      +0.1        0.10 ± 35%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rbx
      0.18 ±  2%      +0.1        0.24 ± 15%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.04 ± 57%      +0.1        0.09 ± 27%  perf-profile.self.cycles-pp.put_prev_entity
      0.09 ±  5%      +0.1        0.14 ± 19%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.05 ±  8%      +0.1        0.11 ± 40%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.01 ±173%      +0.1        0.07 ± 19%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.bpf_prog_kallsyms_find
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.get_cpu_device
      0.23 ±  3%      +0.1        0.29 ± 14%  perf-profile.self.cycles-pp.reweight_entity
      0.20 ±  2%      +0.1        0.25 ±  3%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.18 ±  3%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.pick_next_entity
      0.00            +0.1        0.06 ± 22%  perf-profile.self.cycles-pp.activate_task
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.__is_insn_slot_addr
      0.08 ±  8%      +0.1        0.14 ± 32%  perf-profile.self.cycles-pp.arch_stack_walk
      0.00            +0.1        0.06 ± 28%  perf-profile.self.cycles-pp.__x64_sys_write
      0.23            +0.1        0.29 ± 10%  perf-profile.self.cycles-pp.mutex_unlock
      0.11 ±  7%      +0.1        0.18 ± 25%  perf-profile.self.cycles-pp.fsnotify
      0.30 ±  2%      +0.1        0.36 ± 16%  perf-profile.self.cycles-pp.try_to_wake_up
      0.11 ±  4%      +0.1        0.17 ± 23%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.17 ±  2%      +0.1        0.24 ± 24%  perf-profile.self.cycles-pp.__unwind_start
      0.14 ±  3%      +0.1        0.21 ± 21%  perf-profile.self.cycles-pp.resched_curr
      0.09 ±  7%      +0.1        0.15 ± 33%  perf-profile.self.cycles-pp.new_sync_read
      0.00            +0.1        0.07 ± 31%  perf-profile.self.cycles-pp.timestamp_truncate
      0.17 ±  2%      +0.1        0.24 ± 17%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.ksys_write
      0.11 ±  6%      +0.1        0.18 ± 24%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.31 ±  6%      +0.1        0.38 ± 11%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.00            +0.1        0.07 ± 39%  perf-profile.self.cycles-pp.__update_idle_core
      0.39 ±  4%      +0.1        0.46 ±  8%  perf-profile.self.cycles-pp.mutex_lock
      0.17 ±  2%      +0.1        0.24 ± 23%  perf-profile.self.cycles-pp.__fget_light
      0.11 ±  4%      +0.1        0.18 ± 24%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.12 ± 10%      +0.1        0.20 ± 25%  perf-profile.self.cycles-pp.__might_sleep
      0.08 ±  8%      +0.1        0.16 ± 28%  perf-profile.self.cycles-pp.dequeue_entity
      0.00            +0.1        0.08 ± 34%  perf-profile.self.cycles-pp.put_prev_task_fair
      0.37            +0.1        0.46 ±  9%  perf-profile.self.cycles-pp.___might_sleep
      0.15 ±  3%      +0.1        0.24 ± 17%  perf-profile.self.cycles-pp.vfs_read
      0.26 ±  4%      +0.1        0.35 ± 14%  perf-profile.self.cycles-pp.native_write_msr
      0.23 ±  2%      +0.1        0.31 ± 21%  perf-profile.self.cycles-pp.common_file_perm
      0.20 ±  2%      +0.1        0.28 ± 24%  perf-profile.self.cycles-pp.finish_task_switch
      0.18 ±  4%      +0.1        0.27 ± 16%  perf-profile.self.cycles-pp.read_tsc
      0.17 ±  8%      +0.1        0.26 ± 22%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.15 ±  4%      +0.1        0.25 ± 31%  perf-profile.self.cycles-pp.switch_fpu_return
      0.42            +0.1        0.53 ± 17%  perf-profile.self.cycles-pp.do_idle
      0.12 ±  4%      +0.1        0.24 ± 31%  perf-profile.self.cycles-pp.___perf_sw_event
      0.38 ±  2%      +0.1        0.49 ± 16%  perf-profile.self.cycles-pp.pipe_read
      0.27 ±  3%      +0.1        0.39 ± 19%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.18 ±  4%      +0.1        0.31 ± 29%  perf-profile.self.cycles-pp.orc_find
      0.24            +0.1        0.37 ± 19%  perf-profile.self.cycles-pp.native_sched_clock
      0.28 ±  2%      +0.1        0.41 ± 21%  perf-profile.self.cycles-pp.__orc_find
      0.30            +0.1        0.44 ± 24%  perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
      0.21 ±  2%      +0.1        0.34 ± 22%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.38 ±  3%      +0.1        0.52 ± 17%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.27 ±  3%      +0.1        0.41 ± 22%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.48 ±  4%      +0.2        0.64 ± 18%  perf-profile.self.cycles-pp.__switch_to
      0.74            +0.2        0.91 ± 11%  perf-profile.self.cycles-pp.enqueue_entity
      0.39 ±  2%      +0.2        0.55 ± 19%  perf-profile.self.cycles-pp.__switch_to_asm
      0.23            +0.2        0.40 ± 32%  perf-profile.self.cycles-pp._find_next_bit
      0.16 ± 14%      +0.2        0.34 ± 13%  perf-profile.self.cycles-pp.ktime_get
      0.37 ±  3%      +0.2        0.55 ± 20%  perf-profile.self.cycles-pp._raw_spin_lock
      0.44 ±  6%      +0.2        0.62 ± 22%  perf-profile.self.cycles-pp.menu_select
      0.28 ±  4%      +0.2        0.46 ± 37%  perf-profile.self.cycles-pp.update_cfs_group
      0.16 ±  2%      +0.2        0.37 ± 51%  perf-profile.self.cycles-pp.available_idle_cpu
      1.55            +0.2        1.76 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.30 ±  3%      +0.2        0.53 ± 28%  perf-profile.self.cycles-pp.select_task_rq_fair
      1.09 ±  2%      +0.2        1.32 ± 15%  perf-profile.self.cycles-pp.__sched_text_start
      0.59 ±  3%      +0.3        0.86 ± 17%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.83            +0.3        1.14 ± 19%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.94 ±  2%      +0.3        1.27 ± 17%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.55 ±  2%      +0.3        0.90 ± 24%  perf-profile.self.cycles-pp.update_curr
      0.62 ±  2%      +0.4        1.07 ± 29%  perf-profile.self.cycles-pp.update_load_avg
      1.15            +0.5        1.68 ± 21%  perf-profile.self.cycles-pp.unwind_next_frame
      1.58            +0.8        2.35 ± 22%  perf-profile.self.cycles-pp.do_syscall_64
      0.09 ±  5%      +1.4        1.50 ± 89%  perf-profile.self.cycles-pp.poll_idle



***************************************************************************************************
lkp-hsw-d01: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/target/tbox_group/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/50%/debian-x86_64-20191114.cgz/300s/vmlinux_prereq/lkp-hsw-d01/kbuild/0x27

commit: 
  9c40365a65 ("threads: Update PID limit comment according to futex UAPI change")
  59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group")

9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.BUG:Bad_page_map_in_process
          1:4          -25%            :4     dmesg.BUG:Bad_rss-counter_state_mm:#type:MM_ANONPAGES_val
          1:4          -25%            :4     dmesg.BUG:Bad_rss-counter_state_mm:#type:MM_SWAPENTS_val
           :4           25%           1:4     dmesg.Kernel_panic-not_syncing:Fatal_exception
           :4           25%           1:4     dmesg.RIP:__d_lookup_rcu
           :4           25%           1:4     dmesg.canonical_address#:#[##]
          1:4          -25%            :4     kmsg.file:libuuid.so.#fault:filemap_fault_mmap:generic_file_mmap_readpage:simple_readpage
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a07fffffa0ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a17fffffa1ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a27fffffa2ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a37fffffa3ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a47fffffa4ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a57fffffa5ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a67fffffa6ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a77fffffa7ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a7fffff8aff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a7fffff9aff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a87fffffa8ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#a97fffffa9ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#aa7fffffaaff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ab7fffffabff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ac7fffffacff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ad7fffffadff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ae7fffffaeff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#af7fffffafff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b07fffffb0ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b17fffffb1ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b27fffffb2ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b37fffffb3ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b47fffffb4ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b57fffffb5ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b67fffffb6ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b77fffffb7ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b7fffff8bff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b7fffff9bff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b87fffffb8ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#b97fffffb9ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ba7fffffbaff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#bb7fffffbbff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#bc7fffffbcff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#bd7fffffbdff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#be7fffffbeff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#bf7fffffbfff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c07fffffc0ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c17fffffc1ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c27fffffc2ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c37fffffc3ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c47fffffc4ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c57fffffc5ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c67fffffc6ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c77fffffc7ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c7fffff8cff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c7fffff9cff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c87fffffc8ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#c97fffffc9ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ca7fffffcaff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#cb7fffffcbff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#cc7fffffccff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#cd7fffffcdff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ce7fffffceff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#cf7fffffcfff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d07fffffd0ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d17fffffd1ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d27fffffd2ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d37fffffd3ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d47fffffd4ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d57fffffd5ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d67fffffd6ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d77fffffd7ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d7fffff8dff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d7fffff9dff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d87fffffd8ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#d97fffffd9ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#da7fffffdaff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#db7fffffdbff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#dc7fffffdcff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#dd7fffffddff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#de7fffffdeff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#df7fffffdfff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e07fffffe0ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e17fffffe1ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e27fffffe2ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e37fffffe3ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e47fffffe4ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e57fffffe5ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e67fffffe6ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e77fffffe7ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e7fffff8eff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e7fffff9eff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e87fffffe8ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#e97fffffe9ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ea7fffffeaff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#eb7fffffebff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ec7fffffecff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ed7fffffedff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ee7fffffeeff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ef7fffffefff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f07ffffff0ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f17ffffff1ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f27ffffff2ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f37ffffff3ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f47ffffff4ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f57ffffff5ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f67ffffff6ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f77ffffff7ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f7fffff8fff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f7fffff9fff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f87ffffff8ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#f97ffffff9ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fa7ffffffaff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fb7ffffffbff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fc7ffffffcff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fd7ffffffdff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fe7ffffffeff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#ff7fffffffff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff80ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff81ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff82ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff83ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff84ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff85ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff86ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff87ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff88ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff89ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff90ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff91ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff92ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff93ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff94ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff95ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff96ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff97ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff98ff
          1:4          -25%            :4     kmsg.swap_info_get:Bad_swap_file_entry#fffff99ff
          0:4            0%           0:4     perf-profile.children.cycles-pp.warning_at
          0:4            3%           0:4     perf-profile.children.cycles-pp.warning
          0:4            4%           1:4     perf-profile.children.cycles-pp.error_entry
          0:4            4%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    173.17            +7.5%     186.17        kbuild.buildtime_per_iteration
    346.13            +7.5%     372.13        kbuild.time.elapsed_time
    346.13            +7.5%     372.13        kbuild.time.elapsed_time.max
     62328            -4.4%      59613        kbuild.time.involuntary_context_switches
    119.99            +3.7%     124.42        kbuild.time.system_time
      1258            +7.8%       1357        kbuild.time.user_time
      3976 ±  2%      -5.2%       3768 ±  2%  proc-vmstat.pgactivate
      8581            -7.6%       7925        vmstat.system.cs
     29.08 ±  6%     +27.4%      37.06 ± 21%  boot-time.boot
    190.33 ±  5%     +31.2%     249.80 ± 20%  boot-time.idle
 2.474e+08 ± 62%     +85.2%   4.58e+08 ± 21%  cpuidle.C6.time
    361421 ± 60%     +70.5%     616268 ±  5%  cpuidle.C6.usage
      2265 ±  6%     -16.4%       1895 ± 10%  slabinfo.cred_jar.active_objs
      2301 ±  5%     -15.3%       1950 ±  9%  slabinfo.cred_jar.num_objs
    793.33           +14.3%     907.00 ±  4%  slabinfo.task_delay_info.active_objs
    793.33           +14.3%     907.00 ±  4%  slabinfo.task_delay_info.num_objs
    249422           +10.7%     276098 ±  4%  softirqs.CPU0.RCU
    138168           +30.3%     180101 ± 12%  softirqs.CPU0.TIMER
    139666            +9.2%     152539 ±  2%  softirqs.CPU2.TIMER
    138194 ±  2%     +11.1%     153534 ±  2%  softirqs.CPU4.TIMER
    139323 ±  2%     +11.9%     155966        softirqs.CPU7.TIMER
   1153221           +10.4%    1273097        softirqs.TIMER
    267.67 ± 15%     -37.9%     166.33 ±  9%  interrupts.CPU0.CAL:Function_call_interrupts
      3697 ± 14%     -17.8%       3040 ±  9%  interrupts.CPU0.RES:Rescheduling_interrupts
    245.00 ± 19%     -48.6%     126.00 ±  8%  interrupts.CPU0.TLB:TLB_shootdowns
    188.67 ± 19%     -39.2%     114.67 ± 22%  interrupts.CPU1.TLB:TLB_shootdowns
    292.33 ±  9%     -13.1%     254.00 ± 12%  interrupts.CPU2.CAL:Function_call_interrupts
    241.67 ± 23%     -31.6%     165.33 ±  7%  interrupts.CPU2.TLB:TLB_shootdowns
      7195 ± 19%     -37.5%       4497 ± 29%  interrupts.CPU4.NMI:Non-maskable_interrupts
      7195 ± 19%     -37.5%       4497 ± 29%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
      1361 ±  5%     -16.8%       1132 ± 10%  interrupts.TLB:TLB_shootdowns
      0.00        +1.4e+11%       1382 ±114%  sched_debug.cfs_rq:/.MIN_vruntime.avg
      0.00        +1.1e+12%      11058 ±114%  sched_debug.cfs_rq:/.MIN_vruntime.max
     77852           +20.1%      93480        sched_debug.cfs_rq:/.exec_clock.avg
     82443           +20.3%      99210        sched_debug.cfs_rq:/.exec_clock.max
     72993           +18.4%      86426        sched_debug.cfs_rq:/.exec_clock.min
      2882 ±  6%     +33.9%       3858 ±  9%  sched_debug.cfs_rq:/.exec_clock.stddev
    119503 ±  2%     +20.2%     143687 ±  8%  sched_debug.cfs_rq:/.load.avg
      0.00        +1.4e+11%       1382 ±114%  sched_debug.cfs_rq:/.max_vruntime.avg
      0.00        +1.1e+12%      11058 ±114%  sched_debug.cfs_rq:/.max_vruntime.max
    463036           +18.8%     549986        sched_debug.cfs_rq:/.min_vruntime.avg
    489987           +18.7%     581626        sched_debug.cfs_rq:/.min_vruntime.max
    434924           +17.3%     510306        sched_debug.cfs_rq:/.min_vruntime.min
     17242           +24.4%      21446 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.68 ±  6%     +18.1%       0.80 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
      0.44 ±  3%     -18.2%       0.36 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
     17242           +24.4%      21446 ±  6%  sched_debug.cfs_rq:/.spread0.stddev
    180483           +21.0%     218362 ±  3%  sched_debug.cpu.clock.avg
    180483           +21.0%     218363 ±  3%  sched_debug.cpu.clock.max
    180482           +21.0%     218361 ±  3%  sched_debug.cpu.clock.min
    180483           +21.0%     218362 ±  3%  sched_debug.cpu.clock_task.avg
    180483           +21.0%     218363 ±  3%  sched_debug.cpu.clock_task.max
    180482           +21.0%     218361 ±  3%  sched_debug.cpu.clock_task.min
      9075 ±  6%     +37.6%      12486        sched_debug.cpu.curr->pid.avg
     13910           +37.2%      19088        sched_debug.cpu.curr->pid.max
      6416 ±  2%     +37.6%       8831        sched_debug.cpu.curr->pid.stddev
      1.83 ± 19%     -24.7%       1.38 ± 12%  sched_debug.cpu.nr_running.max
      0.66 ± 18%     -30.6%       0.46 ±  4%  sched_debug.cpu.nr_running.stddev
     22.33 ± 28%     -53.9%      10.29 ± 11%  sched_debug.cpu.nr_uninterruptible.max
     10.96 ±  9%     -35.3%       7.10 ±  7%  sched_debug.cpu.nr_uninterruptible.stddev
    180482           +21.0%     218362 ±  3%  sched_debug.cpu_clk
    179957           +21.0%     217836 ±  3%  sched_debug.ktime
    181437           +20.8%     219109 ±  3%  sched_debug.sched_clk
     13.61            +6.4%      14.47        perf-stat.i.MPKI
 3.762e+09            -7.5%  3.478e+09        perf-stat.i.branch-instructions
      3.00            +0.1        3.08        perf-stat.i.branch-miss-rate%
 1.126e+08            -5.5%  1.064e+08        perf-stat.i.branch-misses
      8.73            -0.6        8.17 ±  2%  perf-stat.i.cache-miss-rate%
  20492578           -10.3%   18373371        perf-stat.i.cache-misses
 2.365e+08            -2.6%  2.303e+08        perf-stat.i.cache-references
      8639            -7.7%       7977        perf-stat.i.context-switches
      0.92            +9.1%       1.00 ±  2%  perf-stat.i.cpi
 1.582e+10            -1.2%  1.562e+10        perf-stat.i.cpu-cycles
    182.40           -11.9%     160.76        perf-stat.i.cpu-migrations
    799.50            +9.7%     877.23        perf-stat.i.cycles-between-cache-misses
      0.31            +0.0        0.33        perf-stat.i.dTLB-load-miss-rate%
  14810335            -4.5%   14149852        perf-stat.i.dTLB-load-misses
 4.694e+09            -7.5%  4.341e+09        perf-stat.i.dTLB-loads
  2.55e+09            -7.0%  2.373e+09        perf-stat.i.dTLB-stores
   4617509            +8.3%    4998953 ±  2%  perf-stat.i.iTLB-load-misses
  17720319            +7.6%   19060873        perf-stat.i.iTLB-loads
 1.731e+10            -7.5%  1.601e+10        perf-stat.i.instructions
      3841           -13.1%       3339 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      1.09            -6.6%       1.02        perf-stat.i.ipc
    164253            -7.3%     152206        perf-stat.i.minor-faults
  13452948           -11.3%   11934691        perf-stat.i.node-loads
   1875909 ±  3%     -11.5%    1659475 ±  3%  perf-stat.i.node-stores
    164253            -7.3%     152206        perf-stat.i.page-faults
     13.66            +5.3%      14.38        perf-stat.overall.MPKI
      2.99            +0.1        3.06        perf-stat.overall.branch-miss-rate%
      8.67            -0.7        7.98        perf-stat.overall.cache-miss-rate%
      0.91            +6.8%       0.98        perf-stat.overall.cpi
    771.90           +10.2%     850.28        perf-stat.overall.cycles-between-cache-misses
      0.31            +0.0        0.32        perf-stat.overall.dTLB-load-miss-rate%
      3749           -14.5%       3205 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      1.09            -6.4%       1.02        perf-stat.overall.ipc
 3.751e+09            -7.5%  3.469e+09        perf-stat.ps.branch-instructions
 1.123e+08            -5.5%  1.062e+08        perf-stat.ps.branch-misses
  20432412           -10.3%   18323590        perf-stat.ps.cache-misses
 2.358e+08            -2.6%  2.297e+08        perf-stat.ps.cache-references
      8613            -7.6%       7955        perf-stat.ps.context-switches
 1.577e+10            -1.2%  1.558e+10        perf-stat.ps.cpu-cycles
    181.89           -11.9%     160.32        perf-stat.ps.cpu-migrations
  14766724            -4.4%   14111258        perf-stat.ps.dTLB-load-misses
  4.68e+09            -7.5%   4.33e+09        perf-stat.ps.dTLB-loads
 2.543e+09            -6.9%  2.366e+09        perf-stat.ps.dTLB-stores
   4603933            +8.3%    4985424 ±  2%  perf-stat.ps.iTLB-load-misses
  17668225            +7.6%   19009302        perf-stat.ps.iTLB-loads
 1.726e+10            -7.5%  1.597e+10        perf-stat.ps.instructions
    163774            -7.3%     151790        perf-stat.ps.minor-faults
  13413384           -11.3%   11902353        perf-stat.ps.node-loads
   1870377 ±  3%     -11.5%    1654965 ±  3%  perf-stat.ps.node-stores
    163774            -7.3%     151790        perf-stat.ps.page-faults
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.write._fini
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp._fini
      4.07 ± 13%      -1.8        2.30 ± 21%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write._fini
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.devkmsg_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write.ksys_write
      3.46 ± 14%      -1.4        2.03 ± 21%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write
      3.27 ± 13%      -1.4        1.90 ± 20%  perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit
      3.20 ± 14%      -1.1        2.08 ± 29%  perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
      3.20 ± 14%      -1.1        2.08 ± 29%  perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
      2.16 ± 14%      -0.9        1.27 ± 18%  perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
      1.02 ± 14%      -0.6        0.45 ± 72%  perf-profile.calltrace.cycles-pp.delay_tsc.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
      1.24 ±  2%      -0.3        0.95 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.24 ±  2%      -0.3        0.95 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.56 ±  3%      -0.2        0.36 ± 70%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.84 ±  8%      -0.2        0.66 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.84 ±  8%      -0.2        0.66 ±  3%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.84 ±  8%      -0.2        0.66 ±  3%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.80 ±  8%      -0.2        0.63 ±  3%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.80 ±  8%      -0.2        0.63 ±  3%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      0.79 ±  3%      -0.1        0.71 ±  6%  perf-profile.calltrace.cycles-pp.read
      0.73 ±  3%      -0.1        0.65 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      0.73 ±  3%      -0.1        0.65 ±  8%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.63 ±  4%      -0.1        0.56 ±  6%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.98 ±  3%      -0.1        0.91 ±  6%  perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.65 ±  4%      -0.1        0.58 ±  4%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.58 ±  2%      +0.1        0.67 ±  7%  perf-profile.calltrace.cycles-pp.wide_int_to_tree
      0.66 ± 11%      +0.1        0.78 ± 11%  perf-profile.calltrace.cycles-pp.c_lex_with_flags
      0.65 ±  6%      +0.2        0.81 ±  7%  perf-profile.calltrace.cycles-pp.linemap_lookup
      0.34 ± 70%      +0.2        0.54 ±  2%  perf-profile.calltrace.cycles-pp.pop_scope
      2.11 ±  5%      +0.2        2.36 ±  3%  perf-profile.calltrace.cycles-pp._cpp_lex_direct
      1.14 ±  2%      +0.3        1.44 ± 11%  perf-profile.calltrace.cycles-pp.memchr
      8.11 ±  5%      -2.4        5.75 ± 11%  perf-profile.children.cycles-pp.do_syscall_64
      8.13 ±  5%      -2.4        5.77 ± 11%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      4.19 ± 12%      -1.8        2.36 ± 20%  perf-profile.children.cycles-pp.ksys_write
      4.21 ± 12%      -1.8        2.38 ± 20%  perf-profile.children.cycles-pp.write
      4.18 ± 12%      -1.8        2.36 ± 20%  perf-profile.children.cycles-pp.vfs_write
      4.17 ± 13%      -1.8        2.36 ± 20%  perf-profile.children.cycles-pp.new_sync_write
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.children.cycles-pp._fini
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.children.cycles-pp.devkmsg_write
      4.07 ± 13%      -1.8        2.30 ± 20%  perf-profile.children.cycles-pp.devkmsg_emit
      4.65 ± 13%      -1.5        3.13 ± 20%  perf-profile.children.cycles-pp.vprintk_emit
      4.73 ± 14%      -1.5        3.25 ± 16%  perf-profile.children.cycles-pp.console_unlock
      4.09 ± 15%      -1.2        2.94 ± 16%  perf-profile.children.cycles-pp.serial8250_console_write
      3.99 ± 15%      -1.1        2.88 ± 16%  perf-profile.children.cycles-pp.wait_for_xmitr
      3.89 ± 15%      -1.1        2.80 ± 16%  perf-profile.children.cycles-pp.uart_console_write
      3.80 ± 15%      -1.0        2.75 ± 15%  perf-profile.children.cycles-pp.serial8250_console_putchar
      2.71 ± 15%      -0.8        1.94 ± 15%  perf-profile.children.cycles-pp.io_serial_in
      1.27 ± 15%      -0.3        0.94 ± 18%  perf-profile.children.cycles-pp.delay_tsc
      0.63 ±  7%      -0.3        0.31 ± 18%  perf-profile.children.cycles-pp.vt_console_print
      0.62 ±  9%      -0.3        0.31 ± 18%  perf-profile.children.cycles-pp.lf
      0.62 ±  9%      -0.3        0.31 ± 18%  perf-profile.children.cycles-pp.con_scroll
      0.62 ±  9%      -0.3        0.31 ± 18%  perf-profile.children.cycles-pp.fbcon_scroll
      0.61 ±  8%      -0.3        0.30 ± 17%  perf-profile.children.cycles-pp.fbcon_redraw
      0.59 ±  7%      -0.3        0.29 ± 18%  perf-profile.children.cycles-pp.fbcon_putcs
      0.59 ±  7%      -0.3        0.29 ± 18%  perf-profile.children.cycles-pp.bit_putcs
      0.50 ±  4%      -0.3        0.24 ± 22%  perf-profile.children.cycles-pp.drm_fb_helper_cfb_imageblit
      0.49 ±  3%      -0.2        0.24 ± 22%  perf-profile.children.cycles-pp.cfb_imageblit
      0.86 ±  7%      -0.2        0.66 ±  3%  perf-profile.children.cycles-pp.mmput
      0.84 ±  8%      -0.2        0.66 ±  3%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.84 ±  8%      -0.2        0.66 ±  3%  perf-profile.children.cycles-pp.do_group_exit
      0.84 ±  8%      -0.2        0.66 ±  3%  perf-profile.children.cycles-pp.do_exit
      0.85 ±  7%      -0.2        0.66 ±  3%  perf-profile.children.cycles-pp.exit_mmap
      0.62 ±  6%      -0.2        0.47 ±  3%  perf-profile.children.cycles-pp.unmap_vmas
      0.61 ±  7%      -0.1        0.47 ±  3%  perf-profile.children.cycles-pp.unmap_page_range
      0.52 ± 11%      -0.1        0.41 ±  7%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.84 ±  3%      -0.1        0.73 ±  6%  perf-profile.children.cycles-pp.read
      0.92 ±  7%      -0.1        0.82 ±  5%  perf-profile.children.cycles-pp.do_sys_open
      0.70 ±  5%      -0.1        0.61 ±  4%  perf-profile.children.cycles-pp.ksys_read
      0.90 ±  7%      -0.1        0.81 ±  5%  perf-profile.children.cycles-pp.do_sys_openat2
      0.69 ±  7%      -0.1        0.59 ±  5%  perf-profile.children.cycles-pp.vfs_read
      1.25            -0.1        1.16 ±  4%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.49 ±  9%      -0.1        0.41 ±  7%  perf-profile.children.cycles-pp.release_pages
      0.45            -0.1        0.37 ± 11%  perf-profile.children.cycles-pp.bitmap_clear_bit
      0.22 ±  9%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.execve
      0.24 ±  5%      -0.1        0.17 ± 12%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.24 ±  5%      -0.1        0.17 ± 12%  perf-profile.children.cycles-pp.__do_execve_file
      0.20 ±  8%      -0.1        0.13 ± 24%  perf-profile.children.cycles-pp.statistics_fini_pass
      0.10 ± 16%      -0.1        0.04 ± 70%  perf-profile.children.cycles-pp.__libc_fork
      0.20 ±  8%      -0.1        0.14 ± 17%  perf-profile.children.cycles-pp.filemap_map_pages
      0.32 ±  7%      -0.1        0.26 ±  9%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.29 ±  9%      -0.1        0.24 ±  5%  perf-profile.children.cycles-pp.irq_exit
      0.27 ±  7%      -0.1        0.21 ± 17%  perf-profile.children.cycles-pp.bitmap_bit_p
      0.12 ±  6%      -0.1        0.07 ± 25%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.20 ±  8%      -0.0        0.16 ± 13%  perf-profile.children.cycles-pp.side_effects_p
      0.12 ± 25%      -0.0        0.08 ± 30%  perf-profile.children.cycles-pp.alloc_empty_file
      0.09 ± 13%      -0.0        0.05 ± 72%  perf-profile.children.cycles-pp.io_serial_out
      0.17 ± 10%      -0.0        0.13 ±  9%  perf-profile.children.cycles-pp.note_stores
      0.18 ± 10%      -0.0        0.13 ± 16%  perf-profile.children.cycles-pp.bitmap_copy
      0.14 ±  8%      -0.0        0.10 ± 24%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.24 ±  7%      -0.0        0.19 ± 14%  perf-profile.children.cycles-pp.rcu_core
      0.14 ± 14%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.load_elf_binary
      0.09 ± 24%      -0.0        0.05 ± 78%  perf-profile.children.cycles-pp.copy_page
      0.25 ±  5%      -0.0        0.20 ±  6%  perf-profile.children.cycles-pp.init_alias_analysis
      0.12 ± 14%      -0.0        0.07 ± 28%  perf-profile.children.cycles-pp.find_unreachable_blocks
      0.16 ± 10%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.14 ± 14%      -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.search_binary_handler
      0.11 ± 25%      -0.0        0.07 ± 17%  perf-profile.children.cycles-pp.ret_from_fork
      0.11 ± 27%      -0.0        0.07 ± 35%  perf-profile.children.cycles-pp.__alloc_file
      0.09 ± 14%      -0.0        0.05 ± 72%  perf-profile.children.cycles-pp.lra_inheritance
      0.09 ±  5%      -0.0        0.05 ± 72%  perf-profile.children.cycles-pp.page_remove_rmap
      0.19 ± 11%      -0.0        0.15 ± 16%  perf-profile.children.cycles-pp.get_attribute_name
      0.12 ± 10%      -0.0        0.08 ± 32%  perf-profile.children.cycles-pp.for_each_inc_dec
      0.15 ±  8%      -0.0        0.11 ± 22%  perf-profile.children.cycles-pp.update_stmt_operands
      0.07 ±  6%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp.wrapup_global_declaration_2
      0.18 ±  9%      -0.0        0.15 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.12 ± 14%      -0.0        0.08 ± 20%  perf-profile.children.cycles-pp.gimple_simplify
      0.10 ± 25%      -0.0        0.06 ± 19%  perf-profile.children.cycles-pp.kthread
      0.16 ± 10%      -0.0        0.13 ± 10%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.17 ±  9%      -0.0        0.14 ± 17%  perf-profile.children.cycles-pp.walk_stmt_load_store_addr_ops
      0.11 ±  7%      -0.0        0.08 ± 22%  perf-profile.children.cycles-pp.execute_update_addresses_taken
      0.09 ± 14%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.bitmap_and
      0.08 ±  5%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.alloc_set_pte
      0.10 ± 14%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.et_set_father
      0.12 ±  7%      -0.0        0.10 ± 14%  perf-profile.children.cycles-pp.hash_rtx_cb
      0.09 ± 15%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__do_munmap
      0.13 ±  6%      -0.0        0.11 ± 11%  perf-profile.children.cycles-pp.is_gimple_reg
      0.09 ± 10%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.find_get_entry
      0.10 ±  8%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.pagecache_get_page
      0.10 ±  4%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.substitute_and_fold_dom_walker::before_dom_children
      0.09 ±  5%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.realloc
      0.09 ± 10%      +0.0        0.11 ± 12%  perf-profile.children.cycles-pp.rtx_equal_p
      0.07 ±  6%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.c_fully_fold
      0.05 ±  8%      +0.0        0.08 ± 12%  perf-profile.children.cycles-pp._cond_resched
      0.09 ±  5%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.build_unary_op
      0.04 ± 71%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.cpp_interpret_integer
      0.13 ±  6%      +0.0        0.16 ±  5%  perf-profile.children.cycles-pp.declspecs_add_type
      0.10 ± 17%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp.task_work_run
      0.09 ±  9%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.pushdecl
      0.08 ± 16%      +0.0        0.11 ± 12%  perf-profile.children.cycles-pp.force_fit_type
      0.04 ± 73%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.__fput
      0.09 ± 13%      +0.0        0.13 ± 13%  perf-profile.children.cycles-pp.is_typedef_decl
      0.11 ± 14%      +0.0        0.15 ± 11%  perf-profile.children.cycles-pp.convert
      0.03 ± 70%      +0.0        0.07 ± 20%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.03 ± 70%      +0.0        0.07        perf-profile.children.cycles-pp.do_dentry_open
      0.04 ± 73%      +0.0        0.08 ± 16%  perf-profile.children.cycles-pp.page_add_new_anon_rmap
      0.04 ± 70%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.start_function
      0.16 ± 11%      +0.0        0.21 ±  6%  perf-profile.children.cycles-pp.decl_attributes
      0.04 ± 71%      +0.0        0.08        perf-profile.children.cycles-pp.alloc_stmt_list
      0.02 ±141%      +0.0        0.06 ± 14%  perf-profile.children.cycles-pp.tree_int_cst_sgn
      0.02 ±141%      +0.0        0.06 ± 13%  perf-profile.children.cycles-pp.gimple_call_flags
      0.02 ±141%      +0.0        0.06 ± 13%  perf-profile.children.cycles-pp.build_asm_expr
      0.08 ± 11%      +0.0        0.13 ± 12%  perf-profile.children.cycles-pp.wi::divmod_internal
      0.49 ±  4%      +0.0        0.54 ±  2%  perf-profile.children.cycles-pp.pop_scope
      0.03 ± 70%      +0.1        0.09 ± 19%  perf-profile.children.cycles-pp.pop_stmt_list
      0.10 ±  4%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.get_range_from_loc
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.build_component_ref
      0.16 ± 10%      +0.1        0.23 ±  6%  perf-profile.children.cycles-pp.build_null_declspecs
      0.00            +0.1        0.07 ± 35%  perf-profile.children.cycles-pp.vma_merge
      0.34 ± 13%      +0.1        0.41 ±  6%  perf-profile.children.cycles-pp.fold_binary_loc
      0.59 ±  3%      +0.1        0.69 ±  7%  perf-profile.children.cycles-pp.wide_int_to_tree
      0.17 ± 14%      +0.1        0.28 ± 12%  perf-profile.children.cycles-pp.linemap_add_macro_token
      0.72 ± 10%      +0.1        0.85 ± 11%  perf-profile.children.cycles-pp.c_lex_with_flags
      0.58 ±  5%      +0.2        0.74 ± 16%  perf-profile.children.cycles-pp.htab_find_slot_with_hash
      0.76 ±  4%      +0.2        0.96 ± 10%  perf-profile.children.cycles-pp.linemap_lookup
      0.58 ± 11%      +0.2        0.83 ± 22%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.58 ± 11%      +0.2        0.83 ± 22%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.58 ± 11%      +0.2        0.83 ± 22%  perf-profile.children.cycles-pp.irq_work_run
      0.58 ± 11%      +0.2        0.83 ± 22%  perf-profile.children.cycles-pp.printk
      2.21 ±  5%      +0.3        2.47 ±  3%  perf-profile.children.cycles-pp._cpp_lex_direct
      1.16 ±  3%      +0.3        1.46 ± 12%  perf-profile.children.cycles-pp.memchr
      0.66 ± 26%      +0.3        0.97 ±  9%  perf-profile.children.cycles-pp.irq_work_run_list
      2.71 ± 15%      -0.8        1.94 ± 15%  perf-profile.self.cycles-pp.io_serial_in
      1.27 ± 15%      -0.3        0.94 ± 18%  perf-profile.self.cycles-pp.delay_tsc
      0.49 ±  3%      -0.2        0.24 ± 22%  perf-profile.self.cycles-pp.cfb_imageblit
      0.43 ±  3%      -0.1        0.35 ± 12%  perf-profile.self.cycles-pp.bitmap_clear_bit
      0.20 ±  8%      -0.1        0.13 ± 24%  perf-profile.self.cycles-pp.statistics_fini_pass
      0.22 ±  5%      -0.1        0.16 ±  8%  perf-profile.self.cycles-pp.unmap_page_range
      0.32 ±  7%      -0.1        0.26 ± 11%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.26 ±  8%      -0.1        0.21 ± 19%  perf-profile.self.cycles-pp.bitmap_bit_p
      0.09 ± 18%      -0.1        0.04 ± 73%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.18 ± 10%      -0.1        0.12 ± 19%  perf-profile.self.cycles-pp.bitmap_copy
      0.20 ±  8%      -0.1        0.15 ± 13%  perf-profile.self.cycles-pp.side_effects_p
      0.09 ± 13%      -0.0        0.05 ± 72%  perf-profile.self.cycles-pp.io_serial_out
      0.17 ±  5%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.note_stores
      0.16 ± 10%      -0.0        0.12 ± 10%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.09 ± 24%      -0.0        0.05 ± 78%  perf-profile.self.cycles-pp.copy_page
      0.17 ±  9%      -0.0        0.13 ±  9%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.18 ± 12%      -0.0        0.14 ±  9%  perf-profile.self.cycles-pp.get_attribute_name
      0.09 ± 14%      -0.0        0.04 ± 71%  perf-profile.self.cycles-pp.lra_inheritance
      0.24 ±  5%      -0.0        0.20 ±  8%  perf-profile.self.cycles-pp.init_alias_analysis
      0.11 ± 18%      -0.0        0.07 ± 28%  perf-profile.self.cycles-pp.find_unreachable_blocks
      0.12 ± 10%      -0.0        0.08 ± 32%  perf-profile.self.cycles-pp.for_each_inc_dec
      0.07 ±  6%      -0.0        0.04 ± 71%  perf-profile.self.cycles-pp.wrapup_global_declaration_2
      0.13 ± 12%      -0.0        0.10 ± 14%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.09 ± 14%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.bitmap_and
      0.07 ±  7%      -0.0        0.04 ± 71%  perf-profile.self.cycles-pp.page_remove_rmap
      0.10 ±  8%      -0.0        0.07 ± 23%  perf-profile.self.cycles-pp.update_stmt_operands
      0.12 ±  7%      -0.0        0.10 ± 14%  perf-profile.self.cycles-pp.hash_rtx_cb
      0.10 ±  9%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.et_set_father
      0.10 ±  4%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.substitute_and_fold_dom_walker::before_dom_children
      0.08            +0.0        0.09        perf-profile.self.cycles-pp.realloc
      0.07 ±  6%      +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.htab_traverse
      0.08 ±  5%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.build_unary_op
      0.08 ±  5%      +0.0        0.11 ± 11%  perf-profile.self.cycles-pp.rtx_equal_p
      0.07 ± 11%      +0.0        0.10 ±  9%  perf-profile.self.cycles-pp.force_fit_type
      0.07 ±  7%      +0.0        0.09 ± 10%  perf-profile.self.cycles-pp.c_fully_fold
      0.13 ±  6%      +0.0        0.16 ±  5%  perf-profile.self.cycles-pp.declspecs_add_type
      0.10 ± 12%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.convert
      0.08 ± 10%      +0.0        0.11 ± 11%  perf-profile.self.cycles-pp.pushdecl
      0.04 ± 71%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.start_function
      0.15 ± 11%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.decl_attributes
      0.03 ± 70%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.malloc@plt
      0.09 ± 10%      +0.0        0.13 ± 13%  perf-profile.self.cycles-pp.is_typedef_decl
      0.04 ± 71%      +0.0        0.08        perf-profile.self.cycles-pp.alloc_stmt_list
      0.02 ±141%      +0.0        0.06 ± 13%  perf-profile.self.cycles-pp.mark_irreducible_loops
      0.08 ± 17%      +0.1        0.13 ± 12%  perf-profile.self.cycles-pp.wi::divmod_internal
      0.02 ±141%      +0.1        0.07 ± 25%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.49 ±  3%      +0.1        0.54 ±  2%  perf-profile.self.cycles-pp.pop_scope
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.build_component_ref
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.gimple_call_flags
      0.09 ±  5%      +0.1        0.15 ± 12%  perf-profile.self.cycles-pp.get_range_from_loc
      0.02 ±141%      +0.1        0.08 ± 12%  perf-profile.self.cycles-pp.pop_stmt_list
      0.16 ± 10%      +0.1        0.23 ±  4%  perf-profile.self.cycles-pp.build_null_declspecs
      0.29 ±  9%      +0.1        0.35 ±  9%  perf-profile.self.cycles-pp.fold_binary_loc
      0.13 ±  6%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.linemap_add_macro_token
      0.57 ±  4%      +0.1        0.67 ±  8%  perf-profile.self.cycles-pp.wide_int_to_tree
      0.70 ±  3%      +0.2        0.89 ± 10%  perf-profile.self.cycles-pp.linemap_lookup
      2.14 ±  6%      +0.2        2.39 ±  3%  perf-profile.self.cycles-pp._cpp_lex_direct
      1.12 ±  2%      +0.3        1.40 ± 12%  perf-profile.self.cycles-pp.memchr





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc4-00106-g59901cb4520c4" of type "text/plain" (203613 bytes)

View attachment "job-script" of type "text/plain" (7927 bytes)

View attachment "job.yaml" of type "text/plain" (5493 bytes)

View attachment "reproduce" of type "text/plain" (1445 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ