lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <202303091155.672f546a-yujie.liu@intel.com>
Date:   Thu, 9 Mar 2023 11:46:50 +0800
From:   kernel test robot <yujie.liu@...el.com>
To:     Zhang Qiao <zhangqiao22@...wei.com>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        <linux-kernel@...r.kernel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Roman Kagan <rkagan@...zon.de>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <zhengjun.xing@...ux.intel.com>,
        <fengwei.yin@...el.com>
Subject: [linus:master] [sched/fair] 829c1651e9: hackbench.throughput -7.9%
 regression

Greeting,

FYI, we noticed a -7.9% regression of hackbench.throughput due to commit:

commit: 829c1651e9c4a6f78398d3e67651cef9bb6b42cc ("sched/fair: sanitize vruntime of entity being placed")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: hackbench
on test machine: 104 threads 2 sockets (Skylake) with 192G memory
with following parameters:

	nr_threads: 100%
	iterations: 4
	mode: process
	ipc: socket
	cpufreq_governor: performance

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.rawudp.ops_per_sec 74.0% improvement                                      |
| test machine     | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz (Cascade Lake) with 128G memory |
| test parameters  | class=network                                                                                  |
|                  | cpufreq_governor=performance                                                                   |
|                  | nr_threads=100%                                                                                |
|                  | test=rawudp                                                                                    |
|                  | testtime=60s                                                                                   |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput 125.5% improvement                                             |
| test machine     | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz (Cascade Lake) with 128G memory |
| test parameters  | cpufreq_governor=performance                                                                   |
|                  | ipc=pipe                                                                                       |
|                  | iterations=8                                                                                   |
|                  | mode=process                                                                                   |
|                  | nr_threads=50%                                                                                 |
+------------------+------------------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <yujie.liu@...el.com>
| Link: https://lore.kernel.org/oe-lkp/202303091155.672f546a-yujie.liu@intel.com


Details are as below:

=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase:
  gcc-11/performance/socket/4/x86_64-rhel-8.3/process/100%/debian-11.1-x86_64-20220510.cgz/lkp-skl-fpga01/hackbench

commit: 
  a2e90611b9 ("sched/fair: Remove capacity inversion detection")
  829c1651e9 ("sched/fair: sanitize vruntime of entity being placed")

a2e90611b9f425ad 829c1651e9c4a6f78398d3e6765 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    332.56           +19.0%     395.75        uptime.boot
    591284 ± 13%     +45.9%     862691 ± 13%  numa-numastat.node0.local_node
    633561 ± 15%     +43.8%     911278 ± 13%  numa-numastat.node0.numa_hit
      1.25 ±  6%      -0.2        1.06 ± 10%  mpstat.cpu.all.idle%
      0.03            -0.0        0.03        mpstat.cpu.all.soft%
     22.68            -2.8       19.86        mpstat.cpu.all.usr%
   3746600 ± 13%     -15.3%    3174871        vmstat.memory.cache
      1751           +30.0%       2277        vmstat.procs.r
   1628685          +277.9%    6154443 ±  3%  vmstat.system.cs
    320461           +58.9%     509072 ±  4%  vmstat.system.in
   3618671 ± 13%     -16.1%    3035929        meminfo.Cached
   2906667 ± 18%     -23.1%    2234788 ±  2%  meminfo.Committed_AS
   1295992 ± 39%     -47.9%     675180 ±  7%  meminfo.Inactive
   1295802 ± 39%     -47.9%     674982 ±  7%  meminfo.Inactive(anon)
    535486 ± 32%     -55.2%     240147 ± 23%  meminfo.Mapped
   6007002 ±  9%     -13.2%    5213349        meminfo.Memused
    888389 ± 56%     -65.6%     305644 ± 13%  meminfo.Shmem
    468144 ±  9%     -20.5%     372290 ± 11%  numa-meminfo.node1.AnonPages.max
   1789452 ± 60%     -80.3%     353145 ± 19%  numa-meminfo.node1.FilePages
   1090426 ± 46%     -59.8%     438087 ± 19%  numa-meminfo.node1.Inactive
   1090417 ± 46%     -59.8%     437944 ± 19%  numa-meminfo.node1.Inactive(anon)
    422719 ± 33%     -64.5%     150054 ± 29%  numa-meminfo.node1.Mapped
   3077746 ± 32%     -55.8%    1361807 ± 10%  numa-meminfo.node1.MemUsed
    877875 ± 56%     -66.4%     295052 ± 13%  numa-meminfo.node1.Shmem
    633532 ± 15%     +43.9%     911355 ± 13%  numa-vmstat.node0.numa_hit
    591255 ± 13%     +45.9%     862767 ± 13%  numa-vmstat.node0.numa_local
    446087 ± 61%     -79.9%      89778 ± 18%  numa-vmstat.node1.nr_file_pages
    271096 ± 46%     -59.1%     110864 ± 20%  numa-vmstat.node1.nr_inactive_anon
    105177 ± 33%     -63.0%      38966 ± 29%  numa-vmstat.node1.nr_mapped
    218193 ± 57%     -65.5%      75255 ± 13%  numa-vmstat.node1.nr_shmem
    271096 ± 46%     -59.1%     110864 ± 20%  numa-vmstat.node1.nr_zone_inactive_anon
   1087461 ±  8%     +19.9%    1304147 ±  4%  turbostat.C1
    238405 ±  3%     +48.4%     353679 ±  6%  turbostat.C1E
      1.17 ±  4%      -0.2        0.99 ±  5%  turbostat.C6%
      0.26 ±  7%     -22.8%       0.20 ±  7%  turbostat.CPU%c6
  91428380           +93.8%  1.772e+08 ±  5%  turbostat.IRQ
    446543 ±  6%     -40.4%     266049 ±  8%  turbostat.POLL
     34.72            -6.8%      32.36        turbostat.RAMWatt
    176568            -7.9%     162531        hackbench.throughput
    173811           -18.4%     141887        hackbench.throughput_avg
    176568            -7.9%     162531        hackbench.throughput_best
    167465           -36.9%     105686        hackbench.throughput_worst
    282.47           +22.4%     345.62        hackbench.time.elapsed_time
    282.47           +22.4%     345.62        hackbench.time.elapsed_time.max
  1.08e+08 ±  2%    +648.7%  8.084e+08 ±  5%  hackbench.time.involuntary_context_switches
     22102           +26.9%      28058        hackbench.time.system_time
      6651            +7.9%       7177        hackbench.time.user_time
 3.542e+08          +276.0%  1.332e+09 ±  4%  hackbench.time.voluntary_context_switches
    121493 ±  4%      -8.7%     110874 ±  2%  proc-vmstat.nr_anon_pages
    905193 ± 14%     -16.0%     760560        proc-vmstat.nr_file_pages
    324344 ± 39%     -47.5%     170335 ±  7%  proc-vmstat.nr_inactive_anon
     82632            -2.0%      80963        proc-vmstat.nr_kernel_stack
    133808 ± 32%     -54.1%      61447 ± 23%  proc-vmstat.nr_mapped
    222621 ± 57%     -65.0%      77988 ± 13%  proc-vmstat.nr_shmem
     31327            -1.5%      30854        proc-vmstat.nr_slab_reclaimable
    118181            -7.8%     108941        proc-vmstat.nr_slab_unreclaimable
    324344 ± 39%     -47.5%     170335 ±  7%  proc-vmstat.nr_zone_inactive_anon
    146649 ±  4%     -23.3%     112510 ± 17%  proc-vmstat.numa_hint_faults
     95646 ±  6%     -42.3%      55232 ± 23%  proc-vmstat.numa_hint_faults_local
   2501175 ±  6%     +34.1%    3353611 ±  4%  proc-vmstat.pgalloc_normal
   2282129 ± 10%     +44.0%    3287348 ±  4%  proc-vmstat.pgfree
    111663 ±  2%      +9.6%     122351 ±  6%  proc-vmstat.pgreuse
   2184832           +21.6%    2657792        proc-vmstat.unevictable_pgs_scanned
     14.29 ±  5%     +21.1%      17.31 ±  5%  sched_debug.cfs_rq:/.h_nr_running.avg
      8.90 ±  3%     -16.9%       7.40 ±  5%  sched_debug.cfs_rq:/.h_nr_running.stddev
     59684 ±126%     -61.8%      22786 ±  7%  sched_debug.cfs_rq:/.load.max
      7944 ± 90%     -49.3%       4026 ±  9%  sched_debug.cfs_rq:/.load.stddev
  11840155           +39.0%   16456271        sched_debug.cfs_rq:/.min_vruntime.avg
  13870482 ±  3%    +115.5%   29897645 ± 16%  sched_debug.cfs_rq:/.min_vruntime.max
  11419937           +32.2%   15100337        sched_debug.cfs_rq:/.min_vruntime.min
    302622 ± 14%    +434.1%    1616191 ± 21%  sched_debug.cfs_rq:/.min_vruntime.stddev
     14521 ±  3%     +23.1%      17873 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
     24482 ±  2%     +16.6%      28534 ±  8%  sched_debug.cfs_rq:/.runnable_avg.max
      3781 ±  4%     +15.6%       4372 ±  8%  sched_debug.cfs_rq:/.runnable_avg.stddev
  -1538686          +365.4%   -7160429        sched_debug.cfs_rq:/.spread0.min
    307224 ± 14%    +426.4%    1617267 ± 22%  sched_debug.cfs_rq:/.spread0.stddev
      1557 ±  4%     -11.5%       1379 ±  3%  sched_debug.cfs_rq:/.util_avg.max
    308.20 ± 16%     +63.4%     503.64 ± 11%  sched_debug.cfs_rq:/.util_avg.min
    227.94 ±  3%     -29.5%     160.63 ±  4%  sched_debug.cfs_rq:/.util_avg.stddev
    506.05 ±  5%     +31.3%     664.53 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     10.67 ± 46%    +452.6%      58.94 ± 48%  sched_debug.cfs_rq:/.util_est_enqueued.min
    340.86 ±  3%      -9.9%     307.11 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    173106           +20.4%     208437        sched_debug.cpu.clock.avg
    173428           +20.7%     209392        sched_debug.cpu.clock.max
    172807           +20.0%     207372        sched_debug.cpu.clock.min
    188.14 ± 45%    +211.4%     585.91 ± 18%  sched_debug.cpu.clock.stddev
    171366           +20.4%     206279        sched_debug.cpu.clock_task.avg
    171999           +20.7%     207633        sched_debug.cpu.clock_task.max
    155591           +21.8%     189434        sched_debug.cpu.clock_task.min
      1596 ±  3%     +17.2%       1871 ±  4%  sched_debug.cpu.clock_task.stddev
      6694 ± 16%     -32.3%       4534 ± 24%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ± 43%    +205.0%       0.00 ± 18%  sched_debug.cpu.next_balance.stddev
     14.30 ±  5%     +20.6%      17.26 ±  5%  sched_debug.cpu.nr_running.avg
      8.91 ±  3%     -17.0%       7.40 ±  5%  sched_debug.cpu.nr_running.stddev
   1834484 ±  2%    +437.1%    9853555 ±  5%  sched_debug.cpu.nr_switches.avg
   2261734 ±  5%    +516.3%   13939145 ±  6%  sched_debug.cpu.nr_switches.max
   1565403          +370.7%    7368423 ±  6%  sched_debug.cpu.nr_switches.min
    133984 ± 18%    +870.5%    1300270 ±  4%  sched_debug.cpu.nr_switches.stddev
    172800           +20.0%     207351        sched_debug.cpu_clk
    169631           +20.4%     204181        sched_debug.ktime
    173545           +19.9%     208099        sched_debug.sched_clk
      7.31           +21.4%       8.87        perf-stat.i.MPKI
 1.916e+10           +15.5%  2.213e+10        perf-stat.i.branch-instructions
      2.17            -0.1        2.07        perf-stat.i.branch-miss-rate%
 4.135e+08            +7.2%  4.435e+08        perf-stat.i.branch-misses
     23.00            -9.2       13.78 ±  3%  perf-stat.i.cache-miss-rate%
 1.699e+08           -36.4%  1.081e+08 ±  2%  perf-stat.i.cache-misses
 7.463e+08           +34.9%  1.007e+09        perf-stat.i.cache-references
   1631170          +385.1%    7912003 ±  4%  perf-stat.i.context-switches
      2.75            -7.7%       2.54        perf-stat.i.cpi
    104173            +1.1%     105339        perf-stat.i.cpu-clock
    128043 ±  2%    +289.0%     498025 ±  6%  perf-stat.i.cpu-migrations
      1686          +129.8%       3875 ±  9%  perf-stat.i.cycles-between-cache-misses
      0.50 ±  2%      +0.6        1.13        perf-stat.i.dTLB-load-miss-rate%
 1.456e+08 ±  2%    +156.8%  3.738e+08        perf-stat.i.dTLB-load-misses
 2.903e+10           +10.0%  3.194e+10        perf-stat.i.dTLB-loads
      0.13 ±  7%      +0.1        0.21 ±  2%  perf-stat.i.dTLB-store-miss-rate%
  22070920 ±  7%     +84.1%   40627527 ±  2%  perf-stat.i.dTLB-store-misses
 1.752e+10            +9.5%   1.92e+10        perf-stat.i.dTLB-stores
     29.21            +3.0       32.23        perf-stat.i.iTLB-load-miss-rate%
  57279315           +57.3%   90075362 ±  3%  perf-stat.i.iTLB-load-misses
 1.383e+08           +31.1%  1.813e+08        perf-stat.i.iTLB-loads
 1.029e+11           +11.6%  1.148e+11        perf-stat.i.instructions
      1821           -19.0%       1475 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.36           +11.6%       0.41        perf-stat.i.ipc
      1096            -4.7%       1044        perf-stat.i.metric.K/sec
    639.86           +11.1%     710.59        perf-stat.i.metric.M/sec
      4992            -3.8%       4802 ±  2%  perf-stat.i.minor-faults
     22.97           +22.1       45.10 ±  5%  perf-stat.i.node-load-miss-rate%
  11795656            +9.0%   12859890 ±  9%  perf-stat.i.node-load-misses
  40616847           -46.0%   21938686 ±  4%  perf-stat.i.node-loads
      8.56 ±  3%     +29.5       38.06 ±  9%  perf-stat.i.node-store-miss-rate%
   2902019 ±  2%     +60.8%    4666181 ± 13%  perf-stat.i.node-store-misses
  33981660           -50.6%   16788672 ±  5%  perf-stat.i.node-stores
      5022            -3.7%       4834 ±  2%  perf-stat.i.page-faults
    104173            +1.1%     105340        perf-stat.i.task-clock
      7.21           +11.6%       8.05        perf-stat.overall.MPKI
      2.15            -0.1        2.05        perf-stat.overall.branch-miss-rate%
     22.82            -9.3       13.53 ±  3%  perf-stat.overall.cache-miss-rate%
      2.74            -6.3%       2.56        perf-stat.overall.cpi
      1661           +41.7%       2354        perf-stat.overall.cycles-between-cache-misses
      0.50 ±  2%      +0.5        0.99        perf-stat.overall.dTLB-load-miss-rate%
      0.13 ±  7%      +0.1        0.19 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     29.28            +2.2       31.47        perf-stat.overall.iTLB-load-miss-rate%
      1804           -23.8%       1374 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.37            +6.8%       0.39        perf-stat.overall.ipc
     22.25            +7.3       29.59 ±  6%  perf-stat.overall.node-load-miss-rate%
      7.63 ±  2%      +7.9       15.50 ± 12%  perf-stat.overall.node-store-miss-rate%
 1.914e+10            +9.2%  2.089e+10        perf-stat.ps.branch-instructions
 4.122e+08            +4.0%  4.286e+08        perf-stat.ps.branch-misses
 1.692e+08           -29.7%   1.19e+08        perf-stat.ps.cache-misses
 7.412e+08           +18.7%    8.8e+08        perf-stat.ps.cache-references
   1614353          +282.0%    6167224 ±  4%  perf-stat.ps.context-switches
    124528 ±  2%    +181.4%     350438 ±  6%  perf-stat.ps.cpu-migrations
 1.448e+08 ±  2%    +111.2%   3.06e+08        perf-stat.ps.dTLB-load-misses
 2.898e+10            +5.6%  3.059e+10        perf-stat.ps.dTLB-loads
  21927790 ±  7%     +60.0%   35093075 ±  2%  perf-stat.ps.dTLB-store-misses
 1.748e+10            +5.3%   1.84e+10        perf-stat.ps.dTLB-stores
  56957377           +39.7%   79589638 ±  2%  perf-stat.ps.iTLB-load-misses
 1.376e+08           +25.9%  1.732e+08        perf-stat.ps.iTLB-loads
 1.027e+11            +6.4%  1.093e+11        perf-stat.ps.instructions
     23.77 ±  9%     -14.8%      20.26 ±  3%  perf-stat.ps.major-faults
      4672           -20.7%       3705 ±  2%  perf-stat.ps.minor-faults
  40554146           -33.5%   26953905 ±  3%  perf-stat.ps.node-loads
   2797533 ±  2%     +36.8%    3825728 ± 11%  perf-stat.ps.node-store-misses
  33847416           -38.3%   20890250 ±  3%  perf-stat.ps.node-stores
      4696           -20.7%       3725 ±  2%  perf-stat.ps.page-faults
 2.906e+13           +30.2%  3.782e+13        perf-stat.total.instructions
     15.22 ±  2%      -8.6        6.66 ±  3%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
     13.83 ±  2%      -7.8        5.98 ±  4%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
     13.60 ±  2%      -7.8        5.85 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
      8.99 ±  2%      -5.4        3.56 ±  4%  perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
      6.45 ±  3%      -4.3        2.19 ±  5%  perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      5.88 ±  4%      -3.4        2.48 ±  3%  perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      5.76 ±  3%      -3.3        2.43 ±  3%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      5.57 ±  5%      -3.3        2.30 ±  3%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      5.14 ±  5%      -3.1        2.03 ±  4%  perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
      6.24 ±  2%      -2.9        3.31 ±  2%  perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
      6.18 ±  2%      -2.9        3.29 ±  2%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      6.12 ±  2%      -2.9        3.23 ±  2%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg
     51.05            -2.4       48.66        perf-profile.calltrace.cycles-pp.__libc_read
      5.65            -2.3        3.37        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_read
      5.51 ±  2%      -2.2        3.27 ±  2%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_write
      3.19 ±  3%      -1.8        1.41 ±  3%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
      2.79 ±  3%      -1.7        1.14 ±  3%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
      2.66 ±  4%      -1.6        1.06 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      2.68 ±  2%      -1.3        1.36 ±  6%  perf-profile.calltrace.cycles-pp.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
      2.98 ±  2%      -1.2        1.82        perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
      2.25 ±  2%      -1.1        1.15 ±  2%  perf-profile.calltrace.cycles-pp.skb_release_head_state.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      2.72 ±  2%      -1.1        1.63 ±  2%  perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
      2.79 ±  2%      -1.1        1.71 ±  2%  perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
      2.18 ±  2%      -1.1        1.11 ±  2%  perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      1.98 ±  3%      -1.0        0.95 ±  3%  perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.consume_skb.unix_stream_read_generic
     41.54            -1.0       40.55        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
      2.06            -1.0        1.08 ±  3%  perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      1.18 ±  5%      -0.9        0.28 ±100%  perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
      2.06            -0.8        1.23        perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_write
      2.16 ±  2%      -0.7        1.46        perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_read
      2.19 ±  2%      -0.6        1.54 ±  4%  perf-profile.calltrace.cycles-pp.__kmem_cache_free.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      1.45 ±  2%      -0.6        0.83        perf-profile.calltrace.cycles-pp._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      4.82            -0.5        4.27        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      1.32 ±  2%      -0.5        0.86 ±  2%  perf-profile.calltrace.cycles-pp.__check_object_size.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      0.94 ±  3%      -0.4        0.54        perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg
      1.14 ±  2%      -0.4        0.76 ±  4%  perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      1.00 ±  3%      -0.4        0.65 ±  6%  perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.__kmem_cache_alloc_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
      1.30 ±  3%      -0.3        0.99 ±  3%  perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.vfs_read.ksys_read.do_syscall_64
      0.98            -0.3        0.69 ±  2%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.90 ±  3%      -0.2        0.68        perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      1.15 ±  3%      -0.2        0.93 ±  3%  perf-profile.calltrace.cycles-pp.security_socket_recvmsg.sock_recvmsg.sock_read_iter.vfs_read.ksys_read
      0.85            -0.2        0.64 ±  2%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
      0.56 ±  3%      -0.2        0.36 ± 70%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      0.84 ±  3%      -0.2        0.67 ±  4%  perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter.vfs_read
      1.06            -0.2        0.90 ±  3%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.22 ±  2%      -0.2        1.07        perf-profile.calltrace.cycles-pp.security_socket_sendmsg.sock_sendmsg.sock_write_iter.vfs_write.ksys_write
      0.88            -0.1        0.80 ±  5%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.93 ±  2%      -0.1        0.88 ±  2%  perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
      0.63            +0.3        0.90        perf-profile.calltrace.cycles-pp.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
      0.00            +0.5        0.54        perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     30.43            +0.6       31.02        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      0.00            +0.6        0.60 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg
      0.00            +0.6        0.61 ±  3%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.6        0.61 ±  3%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop
      0.00            +0.7        0.66 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      0.00            +0.7        0.66 ±  4%  perf-profile.calltrace.cycles-pp.__switch_to_asm.__libc_write
      0.00            +0.8        0.76 ±  3%  perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      0.00            +0.8        0.76 ±  5%  perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop
      0.00            +0.8        0.77 ±  2%  perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.schedule_timeout
     32.79            +0.8       33.56        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      0.00            +0.8        0.80 ±  3%  perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.8        0.81 ±  3%  perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +0.8        0.81 ±  3%  perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.8        0.83        perf-profile.calltrace.cycles-pp.__switch_to_asm.__libc_read
      0.00            +0.8        0.84 ±  3%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +0.9        0.88 ±  2%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     29.52            +0.9       30.42        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     31.98            +0.9       32.88        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     29.91            +0.9       30.82        perf-profile.calltrace.cycles-pp.sock_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.94 ± 22%  perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function
      0.00            +1.0        0.97 ±  3%  perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +1.0        0.98        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.schedule_timeout
      0.00            +1.0        1.00        perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +1.1        1.07 ±  3%  perf-profile.calltrace.cycles-pp.prepare_to_wait.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     27.84            +1.4       29.27        perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_read_iter.vfs_read.ksys_read.do_syscall_64
      0.00            +1.4        1.43 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common
     27.39            +1.5       28.87        perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read.ksys_read
      0.00            +1.5        1.50 ±  3%  perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.schedule_timeout.unix_stream_data_wait
      0.00            +1.6        1.58 ± 16%  perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +1.6        1.62 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     27.43            +1.6       29.06        perf-profile.calltrace.cycles-pp.sock_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.6        1.63 ±  6%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +1.7        1.68 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      0.00            +1.8        1.77 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +1.9        1.88        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.unix_stream_data_wait
      0.00            +1.9        1.93 ±  2%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.unix_stream_data_wait
      0.00            +2.0        1.98 ± 11%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +2.0        1.99 ±  4%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
     26.48            +2.0       28.48        perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.vfs_write.ksys_write.do_syscall_64
      0.00            +2.1        2.12 ±  4%  perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     25.07            +2.2       27.30        perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write.ksys_write
      0.00            +2.3        2.30 ± 10%  perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     47.96            +2.5       50.45        perf-profile.calltrace.cycles-pp.__libc_write
      0.00            +2.9        2.92 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      0.00            +3.4        3.44        perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
     39.02            +4.6       43.61        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
      0.70 ± 13%      +5.0        5.67 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.unix_stream_data_wait
      4.80            +5.1        9.86 ±  3%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      0.70 ± 14%      +5.2        5.94        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
     35.89            +5.6       41.48        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      0.00            +6.0        5.98 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00            +6.1        6.14 ±  5%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.83 ± 14%      +6.3        7.10 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +6.3        6.32 ±  5%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.18 ±141%      +7.0        7.14 ±  5%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      1.98 ± 13%     +12.1       14.07 ±  2%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic
      1.78 ± 16%     +12.1       13.90 ±  2%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      2.02 ± 13%     +12.3       14.34 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg
      1.82 ± 16%     +12.4       14.22 ±  2%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg
      2.06 ± 13%     +12.5       14.58 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      1.87 ± 15%     +12.7       14.60 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg
      2.61 ± 10%     +13.2       15.79 ±  2%  perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
      2.08 ± 14%     +13.2       15.29 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      2.40 ± 13%     +14.6       17.02 ±  2%  perf-profile.calltrace.cycles-pp.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
     15.24 ±  2%      -8.6        6.67 ±  3%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
     13.84 ±  2%      -7.8        5.99 ±  4%  perf-profile.children.cycles-pp.alloc_skb_with_frags
     13.68 ±  2%      -7.8        5.90 ±  4%  perf-profile.children.cycles-pp.__alloc_skb
      9.01 ±  2%      -5.4        3.56 ±  4%  perf-profile.children.cycles-pp.consume_skb
      6.76 ± 13%      -4.6        2.18 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     11.24            -4.5        6.70        perf-profile.children.cycles-pp.syscall_return_via_sysret
      6.46 ±  3%      -4.3        2.20 ±  5%  perf-profile.children.cycles-pp.skb_release_data
      4.16 ± 12%      -4.0        0.14 ± 59%  perf-profile.children.cycles-pp.___slab_alloc
      5.74 ± 16%      -3.9        1.89 ±  3%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      5.92 ±  4%      -3.4        2.50 ±  3%  perf-profile.children.cycles-pp.kmalloc_reserve
      5.91 ±  3%      -3.4        2.51 ±  3%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      5.70 ±  5%      -3.3        2.37 ±  4%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      5.26 ±  5%      -3.2        2.11 ±  3%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      6.24 ±  2%      -2.9        3.32 ±  2%  perf-profile.children.cycles-pp.unix_stream_read_actor
      6.19 ±  2%      -2.9        3.29 ±  2%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      6.12 ±  2%      -2.9        3.24 ±  2%  perf-profile.children.cycles-pp.__skb_datagram_iter
      3.08 ±  2%      -2.4        0.64 ± 12%  perf-profile.children.cycles-pp.__slab_free
     51.44            -2.3       49.10        perf-profile.children.cycles-pp.__libc_read
      5.34 ±  2%      -2.1        3.20        perf-profile.children.cycles-pp.__entry_text_start
      3.61 ±  3%      -2.0        1.60 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      3.21 ±  3%      -1.8        1.42 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
      2.80 ±  3%      -1.7        1.15 ±  3%  perf-profile.children.cycles-pp.copyout
      4.12 ±  2%      -1.5        2.58 ±  2%  perf-profile.children.cycles-pp.__check_object_size
      2.70 ±  2%      -1.3        1.37 ±  6%  perf-profile.children.cycles-pp.kmem_cache_free
      2.80            -1.2        1.60 ±  3%  perf-profile.children.cycles-pp.check_heap_object
      3.00 ±  2%      -1.2        1.83        perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
      2.26 ±  2%      -1.1        1.16 ±  2%  perf-profile.children.cycles-pp.skb_release_head_state
      2.80 ±  2%      -1.1        1.72 ±  2%  perf-profile.children.cycles-pp.simple_copy_to_iter
      2.20 ±  2%      -1.1        1.13 ±  2%  perf-profile.children.cycles-pp.unix_destruct_scm
      1.99 ±  3%      -1.0        0.96 ±  2%  perf-profile.children.cycles-pp.sock_wfree
      2.17 ±  2%      -0.7        1.43 ±  5%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      1.45            -0.7        0.72 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      2.20 ±  2%      -0.6        1.55 ±  4%  perf-profile.children.cycles-pp.__kmem_cache_free
      1.48 ±  3%      -0.6        0.84        perf-profile.children.cycles-pp._copy_from_iter
      1.48 ±  3%      -0.5        0.98        perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      1.41 ±  3%      -0.5        0.91 ±  8%  perf-profile.children.cycles-pp.mod_objcg_state
      0.72            -0.5        0.25 ± 15%  perf-profile.children.cycles-pp.__build_skb_around
      2.06            -0.5        1.60 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      0.96 ±  3%      -0.4        0.54        perf-profile.children.cycles-pp.copyin
      1.08 ±  2%      -0.4        0.72 ±  2%  perf-profile.children.cycles-pp.__might_resched
      1.32 ±  3%      -0.3        1.00 ±  2%  perf-profile.children.cycles-pp.sock_recvmsg
      1.75            -0.3        1.44 ±  3%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.54 ±  6%      -0.3        0.25 ±  6%  perf-profile.children.cycles-pp.skb_unlink
      0.68            -0.3        0.41 ±  3%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.68            -0.3        0.40 ±  3%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.53 ±  3%      -0.3        0.27        perf-profile.children.cycles-pp.unix_write_space
      1.84 ±  3%      -0.3        1.58 ±  3%  perf-profile.children.cycles-pp.aa_sk_perm
      0.85 ±  3%      -0.2        0.60 ±  4%  perf-profile.children.cycles-pp.__check_heap_object
      0.71 ±  2%      -0.2        0.46 ±  3%  perf-profile.children.cycles-pp.__get_obj_cgroup_from_memcg
      0.71 ±  6%      -0.2        0.48 ±  4%  perf-profile.children.cycles-pp.skb_set_owner_w
      1.16 ±  3%      -0.2        0.93 ±  3%  perf-profile.children.cycles-pp.security_socket_recvmsg
      1.22 ±  2%      -0.2        0.99 ±  4%  perf-profile.children.cycles-pp.__fdget_pos
      0.48 ±  3%      -0.2        0.29 ±  3%  perf-profile.children.cycles-pp.__might_fault
      1.11 ±  2%      -0.2        0.92 ±  5%  perf-profile.children.cycles-pp.__fget_light
      0.23 ±  5%      -0.2        0.06 ± 15%  perf-profile.children.cycles-pp.fsnotify_perm
      0.99 ±  3%      -0.2        0.82 ±  5%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.62 ±  8%      -0.2        0.46 ±  4%  perf-profile.children.cycles-pp.skb_queue_tail
      0.61 ±  2%      -0.2        0.45        perf-profile.children.cycles-pp.__might_sleep
      1.23 ±  2%      -0.2        1.07        perf-profile.children.cycles-pp.security_socket_sendmsg
      0.33 ±  2%      -0.2        0.17 ±  2%  perf-profile.children.cycles-pp.refill_obj_stock
      0.41 ±  2%      -0.2        0.25 ±  4%  perf-profile.children.cycles-pp.kmalloc_slab
      0.57 ±  2%      -0.1        0.43 ±  3%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.42            -0.1        0.29 ±  4%  perf-profile.children.cycles-pp.is_vmalloc_addr
      0.52 ±  2%      -0.1        0.40 ±  3%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.47 ±  2%      -0.1        0.35 ±  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.54 ±  2%      -0.1        0.42        perf-profile.children.cycles-pp.wait_for_unix_gc
      0.47 ±  3%      -0.1        0.35 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.32 ±  5%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.tick_sched_handle
      0.21 ±  4%      -0.1        0.10 ± 13%  perf-profile.children.cycles-pp.task_tick_fair
      0.41 ±  3%      -0.1        0.30 ±  5%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.35 ±  6%      -0.1        0.24 ±  6%  perf-profile.children.cycles-pp.tick_sched_timer
      0.31 ±  4%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.update_process_times
      0.26 ±  5%      -0.1        0.16 ±  5%  perf-profile.children.cycles-pp.kmalloc_size_roundup
      0.24 ±  3%      -0.1        0.15 ±  6%  perf-profile.children.cycles-pp.scheduler_tick
      0.30 ±  4%      -0.1        0.21 ±  4%  perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
      0.20 ±  2%      -0.1        0.12 ±  9%  perf-profile.children.cycles-pp.should_failslab
      0.50            -0.1        0.43 ±  2%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.24 ±  3%      -0.1        0.18 ±  4%  perf-profile.children.cycles-pp.rcu_all_qs
      0.19 ±  5%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.apparmor_socket_sendmsg
      0.20 ±  3%      -0.1        0.13 ±  5%  perf-profile.children.cycles-pp.aa_file_perm
      0.14 ±  6%      -0.1        0.08 ± 19%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.19 ±  2%      -0.1        0.12 ±  4%  perf-profile.children.cycles-pp.apparmor_socket_recvmsg
      0.36            -0.1        0.30        perf-profile.children.cycles-pp.security_socket_getpeersec_dgram
      0.18 ± 11%      -0.1        0.12 ±  5%  perf-profile.children.cycles-pp.memcg_account_kmem
      0.11 ± 18%      -0.0        0.08 ±  7%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.18 ±  2%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp.apparmor_socket_getpeersec_dgram
      0.08 ±  4%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.put_pid
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.default_send_IPI_single_phys
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.invalidate_user_asid
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.__bitmap_and
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.read@plt
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.sysvec_reschedule_ipi
      0.00            +0.1        0.08 ±  4%  perf-profile.children.cycles-pp.schedule_debug
      0.00            +0.1        0.09 ±  5%  perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
      0.00            +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.place_entity
      0.02 ± 99%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
      0.07 ± 12%      +0.1        0.19 ±  4%  perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.1        0.12 ±  5%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.12 ±  8%  perf-profile.children.cycles-pp.perf_trace_buf_alloc
      0.00            +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.switch_ldt
      0.00            +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.00            +0.2        0.15 ± 16%  perf-profile.children.cycles-pp.migrate_task_rq_fair
      0.00            +0.2        0.16 ±  6%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.00            +0.2        0.16 ±  6%  perf-profile.children.cycles-pp.save_fpregs_to_fpstate
      0.00            +0.2        0.16 ±  6%  perf-profile.children.cycles-pp.perf_trace_buf_update
      0.00            +0.2        0.16 ±  9%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.00            +0.2        0.18 ± 10%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.01 ±223%      +0.2        0.19 ±  3%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.2        0.18 ±  3%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.2        0.19 ±  8%  perf-profile.children.cycles-pp.select_idle_cpu
      0.00            +0.2        0.20 ±  4%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.00            +0.2        0.21 ± 16%  perf-profile.children.cycles-pp.set_task_cpu
      0.00            +0.2        0.22 ±  4%  perf-profile.children.cycles-pp.resched_curr
      0.08 ± 11%      +0.2        0.32 ±  5%  perf-profile.children.cycles-pp.task_h_load
      0.00            +0.2        0.24 ±  3%  perf-profile.children.cycles-pp.finish_wait
      0.04 ± 45%      +0.3        0.30 ±  4%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.3        0.26 ± 17%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.66            +0.3        0.93        perf-profile.children.cycles-pp.mutex_lock
      0.04 ± 45%      +0.3        0.32 ±  7%  perf-profile.children.cycles-pp.cpuacct_charge
      0.00            +0.3        0.28 ±  4%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.20 ±  3%      +0.3        0.47 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.20 ±  5%      +0.3        0.48 ±  4%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.02 ± 99%      +0.3        0.31 ±  3%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.3        0.29 ±  3%  perf-profile.children.cycles-pp.clear_buddies
      0.06 ±  9%      +0.3        0.35 ±  3%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.3        0.30 ±  8%  perf-profile.children.cycles-pp.cr4_update_irqsoff
      0.00            +0.3        0.33 ± 20%  perf-profile.children.cycles-pp.available_idle_cpu
      0.08 ± 12%      +0.4        0.43 ±  4%  perf-profile.children.cycles-pp.perf_tp_event
      0.07 ± 14%      +0.4        0.43 ±  5%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.12 ± 13%      +0.4        0.49 ±  6%  perf-profile.children.cycles-pp.wake_affine
      0.11 ±  4%      +0.4        0.48 ±  4%  perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
      0.06 ± 19%      +0.4        0.44 ±  5%  perf-profile.children.cycles-pp.finish_task_switch
      0.08 ± 14%      +0.4        0.48 ±  3%  perf-profile.children.cycles-pp.__calc_delta
      0.04 ± 45%      +0.4        0.48 ±  2%  perf-profile.children.cycles-pp.pick_next_entity
      0.01 ±223%      +0.5        0.48 ±  5%  perf-profile.children.cycles-pp.set_next_buddy
      0.09 ± 12%      +0.6        0.67 ±  2%  perf-profile.children.cycles-pp.update_min_vruntime
     30.45            +0.6       31.04        perf-profile.children.cycles-pp.ksys_write
      0.08 ± 12%      +0.6        0.68 ±  3%  perf-profile.children.cycles-pp.os_xsave
      0.29 ±  6%      +0.7        0.98 ±  3%  perf-profile.children.cycles-pp.update_cfs_group
      0.14 ±  6%      +0.7        0.86 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.16 ±  9%      +0.7        0.89 ±  2%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.08 ± 14%      +0.8        0.84 ±  3%  perf-profile.children.cycles-pp.check_preempt_wakeup
     32.81            +0.8       33.57        perf-profile.children.cycles-pp.ksys_read
      0.14 ± 13%      +0.8        0.93        perf-profile.children.cycles-pp.update_rq_clock
      0.14 ± 10%      +0.8        0.94 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.11 ± 12%      +0.9        0.97 ±  3%  perf-profile.children.cycles-pp.check_preempt_curr
      0.22 ±  7%      +0.9        1.08 ±  4%  perf-profile.children.cycles-pp.prepare_to_wait
     29.57            +0.9       30.44        perf-profile.children.cycles-pp.vfs_write
      0.08 ± 17%      +0.9        0.96 ± 23%  perf-profile.children.cycles-pp.select_idle_sibling
     32.01            +0.9       32.90        perf-profile.children.cycles-pp.vfs_read
      0.14 ± 13%      +0.9        1.04 ±  4%  perf-profile.children.cycles-pp.__switch_to
     29.92            +0.9       30.83        perf-profile.children.cycles-pp.sock_read_iter
      0.07 ± 19%      +0.9        0.99 ±  4%  perf-profile.children.cycles-pp.put_prev_entity
      0.14 ± 17%      +1.0        1.13 ±  3%  perf-profile.children.cycles-pp.___perf_sw_event
      0.21 ± 13%      +1.4        1.58 ±  3%  perf-profile.children.cycles-pp.reweight_entity
      0.22 ± 12%      +1.4        1.63 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
     27.85            +1.4       29.28        perf-profile.children.cycles-pp.unix_stream_recvmsg
      0.24 ± 11%      +1.4        1.66 ±  5%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
     27.44            +1.5       28.90        perf-profile.children.cycles-pp.unix_stream_read_generic
      0.21 ± 10%      +1.5        1.71 ±  2%  perf-profile.children.cycles-pp.set_next_entity
     27.44            +1.6       29.07        perf-profile.children.cycles-pp.sock_write_iter
      0.28 ± 14%      +1.7        1.99 ± 11%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.30 ± 12%      +1.8        2.14 ±  4%  perf-profile.children.cycles-pp.switch_fpu_return
      0.33 ± 14%      +2.0        2.32 ± 10%  perf-profile.children.cycles-pp.select_task_rq
     26.50            +2.0       28.50        perf-profile.children.cycles-pp.sock_sendmsg
      0.32 ± 13%      +2.1        2.40 ±  3%  perf-profile.children.cycles-pp.prepare_task_switch
     25.13            +2.2       27.34        perf-profile.children.cycles-pp.unix_stream_sendmsg
      2.10            +2.3        4.42        perf-profile.children.cycles-pp._raw_spin_lock
     48.33            +2.5       50.78        perf-profile.children.cycles-pp.__libc_write
      0.42 ± 10%      +2.6        2.97 ±  2%  perf-profile.children.cycles-pp.dequeue_entity
      0.47 ± 10%      +3.0        3.50        perf-profile.children.cycles-pp.enqueue_entity
      0.52 ± 10%      +3.1        3.61 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      0.53 ± 11%      +3.4        3.94 ±  4%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.54 ± 10%      +3.5        4.00 ±  2%  perf-profile.children.cycles-pp.update_curr
     80.77            +3.5       84.30        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.56 ±  8%      +3.8        4.40 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      9.67            +4.5       14.16 ±  2%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.82 ± 10%      +4.9        5.69 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.81 ± 11%      +5.2        6.00        perf-profile.children.cycles-pp.enqueue_task_fair
     74.28            +5.7       79.93        perf-profile.children.cycles-pp.do_syscall_64
      0.36 ± 23%      +6.2        6.58 ±  5%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.95 ± 12%      +6.2        7.17        perf-profile.children.cycles-pp.ttwu_do_activate
      0.86 ± 13%      +8.0        8.90 ±  4%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      2.01 ± 13%     +11.9       13.94 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
      2.07 ± 13%     +12.2       14.25 ±  2%  perf-profile.children.cycles-pp.autoremove_wake_function
      2.45 ± 10%     +12.3       14.71 ±  2%  perf-profile.children.cycles-pp.schedule_timeout
      2.12 ± 13%     +12.5       14.62 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      2.33 ± 12%     +13.0       15.31 ±  2%  perf-profile.children.cycles-pp.__wake_up_common_lock
      2.61 ± 10%     +13.2       15.80 ±  2%  perf-profile.children.cycles-pp.sock_def_readable
      2.40 ± 13%     +14.7       17.08 ±  2%  perf-profile.children.cycles-pp.unix_stream_data_wait
      2.73 ± 11%     +18.0       20.73 ±  2%  perf-profile.children.cycles-pp.schedule
      2.78 ± 11%     +18.1       20.85 ±  2%  perf-profile.children.cycles-pp.__schedule
     11.23            -4.5        6.69        perf-profile.self.cycles-pp.syscall_return_via_sysret
      5.73 ± 16%      -3.9        1.88 ±  3%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      8.71            -3.4        5.30 ±  2%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      3.05 ±  2%      -2.4        0.62 ± 12%  perf-profile.self.cycles-pp.__slab_free
      6.64            -2.2        4.45        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.56 ±  3%      -2.0        1.57 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      4.77            -1.9        2.84        perf-profile.self.cycles-pp.__entry_text_start
      2.03 ±  2%      -1.1        0.96 ±  7%  perf-profile.self.cycles-pp.kmem_cache_free
      1.40            -0.9        0.48 ±  6%  perf-profile.self.cycles-pp.check_heap_object
      0.99 ±  3%      -0.8        0.21 ±  8%  perf-profile.self.cycles-pp.skb_release_data
      1.43 ±  3%      -0.8        0.67 ±  3%  perf-profile.self.cycles-pp.sock_wfree
      0.85 ±  5%      -0.8        0.09 ± 62%  perf-profile.self.cycles-pp.___slab_alloc
      1.98            -0.6        1.34 ±  3%  perf-profile.self.cycles-pp.unix_stream_read_generic
      1.60 ±  2%      -0.5        1.09 ±  4%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      1.24 ±  2%      -0.5        0.78 ±  6%  perf-profile.self.cycles-pp.mod_objcg_state
      0.68            -0.4        0.24 ± 15%  perf-profile.self.cycles-pp.__build_skb_around
      0.94            -0.4        0.50 ±  4%  perf-profile.self.cycles-pp.__alloc_skb
      0.99 ±  3%      -0.4        0.57 ±  3%  perf-profile.self.cycles-pp.vfs_write
      0.97            -0.4        0.56 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      1.37 ±  2%      -0.4        0.98 ±  4%  perf-profile.self.cycles-pp.__kmem_cache_free
      0.93            -0.4        0.56 ±  2%  perf-profile.self.cycles-pp.sock_write_iter
      0.87            -0.4        0.50 ±  4%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      1.06 ±  2%      -0.3        0.71 ±  2%  perf-profile.self.cycles-pp.__might_resched
      0.74 ±  2%      -0.3        0.43 ±  3%  perf-profile.self.cycles-pp.__cond_resched
      0.73 ±  6%      -0.3        0.48 ±  3%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.84 ±  2%      -0.2        0.59 ±  4%  perf-profile.self.cycles-pp.__check_heap_object
      1.54            -0.2        1.30 ±  3%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.60 ±  2%      -0.2        0.36 ±  2%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      1.13 ±  3%      -0.2        0.91 ±  5%  perf-profile.self.cycles-pp.aa_sk_perm
      0.70 ±  6%      -0.2        0.47 ±  4%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.66 ±  2%      -0.2        0.45 ±  3%  perf-profile.self.cycles-pp.__get_obj_cgroup_from_memcg
      1.43 ±  3%      -0.2        1.22        perf-profile.self.cycles-pp.unix_stream_sendmsg
      0.76            -0.2        0.56        perf-profile.self.cycles-pp.sock_read_iter
      1.09 ±  2%      -0.2        0.92 ±  5%  perf-profile.self.cycles-pp.__fget_light
      0.43 ±  2%      -0.2        0.26 ±  3%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.23 ±  5%      -0.2        0.06 ± 15%  perf-profile.self.cycles-pp.fsnotify_perm
      0.33 ±  3%      -0.2        0.17 ±  2%  perf-profile.self.cycles-pp.refill_obj_stock
      0.31 ±  4%      -0.2        0.15 ± 15%  perf-profile.self.cycles-pp.security_file_permission
      0.94 ±  3%      -0.1        0.79 ±  5%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.29            -0.1        0.14 ±  9%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.54            -0.1        0.40        perf-profile.self.cycles-pp.__might_sleep
      0.41            -0.1        0.28 ±  4%  perf-profile.self.cycles-pp.is_vmalloc_addr
      0.34 ±  2%      -0.1        0.22 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.27 ±  4%      -0.1        0.15 ±  3%  perf-profile.self.cycles-pp._copy_from_iter
      0.24 ±  2%      -0.1        0.12 ±  5%  perf-profile.self.cycles-pp.ksys_write
      0.15 ±  4%      -0.1        0.06 ± 16%  perf-profile.self.cycles-pp.sock_recvmsg
      0.27 ±  4%      -0.1        0.19 ±  4%  perf-profile.self.cycles-pp.kmalloc_slab
      0.35            -0.1        0.27        perf-profile.self.cycles-pp.__check_object_size
      0.17 ±  2%      -0.1        0.09 ±  6%  perf-profile.self.cycles-pp.skb_copy_datagram_from_iter
      0.22 ±  3%      -0.1        0.14 ±  3%  perf-profile.self.cycles-pp.ksys_read
      0.28 ±  2%      -0.1        0.21 ±  3%  perf-profile.self.cycles-pp.consume_skb
      0.20 ±  2%      -0.1        0.13 ±  6%  perf-profile.self.cycles-pp.kmalloc_reserve
      0.18 ±  5%      -0.1        0.11 ±  3%  perf-profile.self.cycles-pp.apparmor_socket_sendmsg
      0.26 ±  3%      -0.1        0.19 ±  2%  perf-profile.self.cycles-pp.sock_alloc_send_pskb
      0.40 ±  2%      -0.1        0.34 ±  3%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.16 ±  3%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.alloc_skb_with_frags
      0.15 ±  3%      -0.1        0.09 ± 11%  perf-profile.self.cycles-pp.sock_sendmsg
      0.18 ±  4%      -0.1        0.12 ±  7%  perf-profile.self.cycles-pp.aa_file_perm
      0.18 ±  2%      -0.1        0.12 ±  5%  perf-profile.self.cycles-pp.apparmor_socket_recvmsg
      0.34 ±  2%      -0.1        0.29 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
      0.20            -0.1        0.14 ±  3%  perf-profile.self.cycles-pp._copy_to_iter
      0.12 ±  7%      -0.1        0.07 ± 24%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.20 ±  2%      -0.1        0.15 ±  4%  perf-profile.self.cycles-pp.rcu_all_qs
      0.12 ±  7%      -0.0        0.08 ±  9%  perf-profile.self.cycles-pp.kmalloc_size_roundup
      0.11 ±  4%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.__fdget_pos
      0.42 ±  2%      -0.0        0.37 ±  2%  perf-profile.self.cycles-pp.unix_stream_recvmsg
      0.64 ±  3%      -0.0        0.60 ±  2%  perf-profile.self.cycles-pp.__libc_write
      0.08            -0.0        0.04 ± 44%  perf-profile.self.cycles-pp.copyout
      0.18 ±  2%      -0.0        0.15        perf-profile.self.cycles-pp.security_socket_getpeersec_dgram
      0.09 ±  5%      -0.0        0.06 ± 20%  perf-profile.self.cycles-pp.should_failslab
      0.10 ±  7%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__might_fault
      0.17 ±  2%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.apparmor_socket_getpeersec_dgram
      0.16 ±  3%      -0.0        0.13 ±  6%  perf-profile.self.cycles-pp.obj_cgroup_uncharge_pages
      0.08 ±  6%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.28 ±  2%      -0.0        0.26 ±  2%  perf-profile.self.cycles-pp.unix_write_space
      0.17 ±  2%      -0.0        0.15 ±  3%  perf-profile.self.cycles-pp.wait_for_unix_gc
      0.12 ±  4%      -0.0        0.10 ±  6%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.07            -0.0        0.06        perf-profile.self.cycles-pp.skb_copy_datagram_iter
      0.08 ±  5%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.scm_recv
      0.17 ±  2%      +0.0        0.19 ±  3%  perf-profile.self.cycles-pp.__x64_sys_read
      0.19 ±  3%      +0.0        0.21 ±  4%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.finish_wait
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.cr4_update_irqsoff
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.perf_trace_buf_update
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.wake_affine
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.invalidate_user_asid
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.__bitmap_and
      0.00            +0.1        0.07 ±  9%  perf-profile.self.cycles-pp.asm_sysvec_reschedule_ipi
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.read@plt
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.schedule_debug
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.perf_trace_buf_alloc
      0.00            +0.1        0.09 ± 18%  perf-profile.self.cycles-pp.migrate_task_rq_fair
      0.00            +0.1        0.09 ±  5%  perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
      0.00            +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.place_entity
      0.00            +0.1        0.10 ±  6%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.07 ± 10%      +0.1        0.18 ±  4%  perf-profile.self.cycles-pp.__list_add_valid
      0.00            +0.1        0.12 ± 13%  perf-profile.self.cycles-pp.select_idle_cpu
      0.00            +0.1        0.12 ±  5%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.12 ± 15%  perf-profile.self.cycles-pp.__cgroup_account_cputime
      0.00            +0.1        0.13 ± 11%  perf-profile.self.cycles-pp.switch_ldt
      0.00            +0.1        0.14 ±  4%  perf-profile.self.cycles-pp.check_preempt_curr
      0.00            +0.1        0.15 ±  7%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.14 ±  3%      +0.1        0.28 ±  2%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.15 ±  6%  perf-profile.self.cycles-pp.save_fpregs_to_fpstate
      0.00            +0.2        0.15 ±  6%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.02 ± 99%      +0.2        0.19 ±  4%  perf-profile.self.cycles-pp.ttwu_do_activate
      0.01 ±223%      +0.2        0.18 ±  4%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.2        0.17 ±  5%  perf-profile.self.cycles-pp.autoremove_wake_function
      0.00            +0.2        0.18 ±  2%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.2        0.20 ±  4%  perf-profile.self.cycles-pp.rcu_note_context_switch
      0.28 ±  2%      +0.2        0.48 ±  2%  perf-profile.self.cycles-pp.mutex_lock
      0.00            +0.2        0.20 ±  6%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.00            +0.2        0.20 ±  4%  perf-profile.self.cycles-pp.resched_curr
      0.07 ± 10%      +0.2        0.30 ±  4%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.19 ±  4%      +0.2        0.42 ±  5%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.04 ± 45%      +0.2        0.28 ±  6%  perf-profile.self.cycles-pp.perf_tp_event
      0.00            +0.2        0.23 ±  6%  perf-profile.self.cycles-pp.finish_task_switch
      0.04 ± 45%      +0.2        0.28 ±  4%  perf-profile.self.cycles-pp.rb_erase
      0.08 ± 11%      +0.2        0.32 ±  5%  perf-profile.self.cycles-pp.task_h_load
      0.00            +0.3        0.26 ± 18%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.00            +0.3        0.27 ±  3%  perf-profile.self.cycles-pp.clear_buddies
      0.04 ± 71%      +0.3        0.31 ±  7%  perf-profile.self.cycles-pp.cpuacct_charge
      0.02 ± 99%      +0.3        0.30 ±  4%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.3        0.28 ±  4%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.05 ± 45%      +0.3        0.33 ±  6%  perf-profile.self.cycles-pp.select_task_rq
      0.67 ±  3%      +0.3        0.96 ±  7%  perf-profile.self.cycles-pp.vfs_read
      0.07 ± 11%      +0.3        0.36 ±  3%  perf-profile.self.cycles-pp.schedule_timeout
      0.06 ± 16%      +0.3        0.36 ±  6%  perf-profile.self.cycles-pp.dequeue_entity
      0.06 ±  8%      +0.3        0.37 ±  3%  perf-profile.self.cycles-pp.__wake_up_common
      0.02 ±141%      +0.3        0.33 ±  2%  perf-profile.self.cycles-pp.schedule
      0.00            +0.3        0.32 ± 20%  perf-profile.self.cycles-pp.available_idle_cpu
      0.01 ±223%      +0.3        0.36 ±  6%  perf-profile.self.cycles-pp.set_next_entity
      0.04 ± 45%      +0.4        0.40 ±  4%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.07 ± 14%      +0.4        0.43 ±  5%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.08 ± 11%      +0.4        0.44 ±  4%  perf-profile.self.cycles-pp.prepare_to_wait
      0.08 ± 14%      +0.4        0.48 ±  4%  perf-profile.self.cycles-pp.__calc_delta
      0.05 ± 46%      +0.4        0.45 ±  3%  perf-profile.self.cycles-pp.switch_fpu_return
      0.02 ±141%      +0.4        0.42 ±  2%  perf-profile.self.cycles-pp.pick_next_entity
      0.77            +0.4        1.17 ±  5%  perf-profile.self.cycles-pp.__libc_read
      0.07 ± 15%      +0.4        0.48 ±  3%  perf-profile.self.cycles-pp.unix_stream_data_wait
      0.02 ±142%      +0.4        0.44 ± 20%  perf-profile.self.cycles-pp.select_idle_sibling
      0.01 ±223%      +0.5        0.46 ±  5%  perf-profile.self.cycles-pp.set_next_buddy
      0.07 ± 17%      +0.5        0.53 ±  3%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.09 ± 13%      +0.5        0.55 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.09 ± 14%      +0.5        0.57 ±  4%  perf-profile.self.cycles-pp.reweight_entity
      0.08 ± 16%      +0.5        0.57        perf-profile.self.cycles-pp.update_rq_clock
      0.09 ± 12%      +0.5        0.59 ±  2%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.09 ± 10%      +0.6        0.64        perf-profile.self.cycles-pp.update_min_vruntime
      0.02 ±141%      +0.6        0.59 ±  3%  perf-profile.self.cycles-pp.put_prev_entity
      0.07 ± 16%      +0.6        0.66 ±  2%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.08 ± 12%      +0.6        0.68 ±  3%  perf-profile.self.cycles-pp.os_xsave
      0.29 ±  6%      +0.7        0.96 ±  3%  perf-profile.self.cycles-pp.update_cfs_group
      0.14 ±  9%      +0.7        0.83 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.12 ±  8%      +0.7        0.80 ±  5%  perf-profile.self.cycles-pp.prepare_task_switch
      1.33 ±  2%      +0.8        2.09 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.13 ± 11%      +0.8        0.91 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      1.74            +0.8        2.52        perf-profile.self.cycles-pp._raw_spin_lock
      0.14 ± 10%      +0.8        0.93 ±  2%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.14 ± 15%      +0.8        0.98 ±  4%  perf-profile.self.cycles-pp.__switch_to
      0.13 ± 16%      +0.9        1.06 ±  3%  perf-profile.self.cycles-pp.___perf_sw_event
      0.17 ± 12%      +1.2        1.34        perf-profile.self.cycles-pp.enqueue_entity
      0.22 ± 12%      +1.4        1.61 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      0.24 ± 11%      +1.4        1.66 ±  5%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.26 ± 12%      +1.5        1.76 ±  3%  perf-profile.self.cycles-pp.__schedule
      0.23 ±  9%      +1.5        1.76 ±  4%  perf-profile.self.cycles-pp.update_load_avg
      0.23 ±  9%      +1.8        1.99 ±  2%  perf-profile.self.cycles-pp.update_curr
      0.52 ± 11%      +3.3        3.81 ±  4%  perf-profile.self.cycles-pp.switch_mm_irqs_off



To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests

View attachment "config-6.2.0-rc6-00109-g829c1651e9c4" of type "text/plain" (167113 bytes)

View attachment "job-script" of type "text/plain" (7733 bytes)

View attachment "job.yaml" of type "text/plain" (5422 bytes)

View attachment "reproduce" of type "text/plain" (570 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ