lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20210319140742.GC30349@xsang-OptiPlex-9020>
Date:   Fri, 19 Mar 2021 22:07:42 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Kees Cook <keescook@...omium.org>
Cc:     0day robot <lkp@...el.com>, LKML <linux-kernel@...r.kernel.org>,
        lkp@...ts.01.org, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com, Andrew Morton <akpm@...ux-foundation.org>,
        Kees Cook <keescook@...omium.org>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Michal Hocko <mhocko@...e.com>,
        Alexey Dobriyan <adobriyan@...il.com>,
        Lee Duncan <lduncan@...e.com>, Chris Leech <cleech@...hat.com>,
        Adam Nichols <adam@...mm-co.com>,
        linux-fsdevel@...r.kernel.org, linux-hardening@...r.kernel.org
Subject: [seq_file]  5fd6060e50:  stress-ng.eventfd.ops_per_sec -49.1%
 regression



Greeting,

FYI, we noticed a -49.1% regression of stress-ng.eventfd.ops_per_sec due to commit:


commit: 5fd6060e506cc226f2575b01baef7af9ca76aa44 ("[PATCH v2] seq_file: Unconditionally use vmalloc for buffer")
url: https://github.com/0day-ci/linux/commits/Kees-Cook/seq_file-Unconditionally-use-vmalloc-for-buffer/20210316-015127
base: https://git.kernel.org/cgit/linux/kernel/git/kees/linux.git for-next/pstore

in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
with following parameters:

	nr_threads: 10%
	disk: 1HDD
	testtime: 60s
	fs: ext4
	class: os
	test: eventfd
	cpufreq_governor: performance
	ucode: 0x5003006




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  os/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/eventfd/stress-ng/60s/0x5003006

commit: 
  7db688e99c ("pstore/ram: Rate-limit "uncorrectable error in header" message")
  5fd6060e50 ("seq_file: Unconditionally use vmalloc for buffer")

7db688e99c0f770a 5fd6060e506cc226f2575b01bae 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
  32516754           -49.1%   16547637        stress-ng.eventfd.ops
    541944           -49.1%     275793        stress-ng.eventfd.ops_per_sec
     55102 ±  2%      +7.2%      59091 ±  2%  stress-ng.time.involuntary_context_switches
    929.67            -3.9%     893.67        stress-ng.time.percent_of_cpu_this_job_got
     30.90           -49.0%      15.77 ±  2%  stress-ng.time.user_time
  65019233           -49.1%   33087762        stress-ng.time.voluntary_context_switches
      8.90            +2.7%       9.14        iostat.cpu.system
     38002 ±  5%     -11.0%      33835 ± 11%  sched_debug.cfs_rq:/.min_vruntime.max
   2004321           -49.0%    1021836        vmstat.system.cs
      0.11 ±  2%      -0.0        0.09        mpstat.cpu.all.soft%
      0.46            -0.2        0.26 ±  2%  mpstat.cpu.all.usr%
    261727 ±  6%   +2257.7%    6170837 ±  9%  numa-numastat.node0.local_node
    309032 ±  9%   +1909.5%    6209968 ±  9%  numa-numastat.node0.numa_hit
    317574 ±  8%   +3280.6%   10735840 ±  7%  numa-numastat.node1.local_node
    356782 ± 10%   +2922.4%   10783284 ±  7%  numa-numastat.node1.numa_hit
   1090175 ± 18%    +235.6%    3658217 ± 10%  numa-vmstat.node0.numa_hit
    999598 ± 15%    +262.0%    3618161 ± 10%  numa-vmstat.node0.numa_local
    949368 ± 21%    +567.3%    6335563 ±  8%  numa-vmstat.node1.numa_hit
    787502 ± 19%    +677.6%    6123276 ±  8%  numa-vmstat.node1.numa_local
 4.116e+08           -24.7%  3.099e+08 ±  3%  cpuidle.C1.time
  35296067 ±  2%     -63.7%   12799762 ±  8%  cpuidle.C1.usage
   8130367 ± 24%     +76.7%   14367098 ± 16%  cpuidle.C1E.usage
  61265170 ±  2%     -50.7%   30225048        cpuidle.POLL.time
  30192926 ±  3%     -49.4%   15263734 ±  3%  cpuidle.POLL.usage
    585.67 ±103%     -77.6%     131.33 ± 47%  proc-vmstat.numa_hint_faults
    454.67 ±131%     -86.3%      62.17 ± 98%  proc-vmstat.numa_hint_faults_local
    696009         +2325.3%   16880339        proc-vmstat.numa_hit
    609418         +2655.7%   16793735        proc-vmstat.numa_local
   1040066 ±  2%   +1546.5%   17125024        proc-vmstat.pgalloc_normal
    911730 ±  2%   +1764.3%   16996996        proc-vmstat.pgfree
     37931 ±  3%     -11.8%      33457        slabinfo.filp.active_objs
      1204 ±  3%     -12.3%       1057        slabinfo.filp.active_slabs
     38571 ±  3%     -12.3%      33837        slabinfo.filp.num_objs
      1204 ±  3%     -12.3%       1057        slabinfo.filp.num_slabs
     14031 ±  2%     -24.8%      10556        slabinfo.kmalloc-256.active_objs
    442.33 ±  2%     -25.5%     329.50        slabinfo.kmalloc-256.active_slabs
     14165 ±  2%     -25.4%      10562        slabinfo.kmalloc-256.num_objs
    442.33 ±  2%     -25.5%     329.50        slabinfo.kmalloc-256.num_slabs
      9871          +309.4%      40411 ±  5%  slabinfo.vmap_area.active_objs
    154.00          +310.3%     631.83 ±  4%  slabinfo.vmap_area.active_slabs
      9887          +309.2%      40463 ±  4%  slabinfo.vmap_area.num_objs
    154.00          +310.3%     631.83 ±  4%  slabinfo.vmap_area.num_slabs
     19298 ± 13%     -52.4%       9195 ± 10%  softirqs.CPU15.RCU
     10810 ±  5%     -18.3%       8826 ± 14%  softirqs.CPU15.SCHED
     20339 ± 27%     -43.2%      11558 ± 19%  softirqs.CPU16.RCU
     19915 ± 25%     -38.9%      12165 ± 15%  softirqs.CPU17.RCU
     43084 ± 16%     -45.4%      23524 ± 34%  softirqs.CPU2.RCU
     31777 ± 48%     -43.6%      17935 ± 17%  softirqs.CPU26.RCU
     39011 ± 23%     -66.2%      13199 ± 24%  softirqs.CPU32.RCU
     11700 ±  5%     -13.2%      10158 ±  6%  softirqs.CPU32.SCHED
     10293 ±  5%     +12.4%      11570 ±  2%  softirqs.CPU44.SCHED
     10704 ±  3%     +14.0%      12201 ±  9%  softirqs.CPU45.SCHED
     33336 ± 17%     -50.4%      16549 ± 39%  softirqs.CPU5.RCU
     24875 ± 15%     -38.3%      15340 ± 29%  softirqs.CPU53.RCU
     26395 ± 33%     -54.3%      12055 ± 19%  softirqs.CPU55.RCU
     28254 ± 33%     -52.7%      13377 ± 24%  softirqs.CPU61.RCU
     29803 ± 37%     -63.0%      11033 ± 25%  softirqs.CPU63.RCU
     22122 ± 17%     -37.6%      13802 ± 23%  softirqs.CPU7.RCU
     32270 ± 25%     -47.7%      16871 ± 20%  softirqs.CPU78.RCU
     31018 ± 26%     -49.1%      15784 ± 45%  softirqs.CPU81.RCU
     10264 ±  3%     +16.5%      11957 ± 11%  softirqs.CPU84.SCHED
     10086 ±  6%     +17.2%      11819 ±  8%  softirqs.CPU86.SCHED
   2302045           -28.8%    1638717        softirqs.RCU
      0.01 ± 42%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.01 ±100%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
      0.97 ± 99%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      0.01 ± 14%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
      0.28 ±142%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
      0.01 ± 65%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
      0.82 ± 46%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      0.00 ± 22%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
      0.01 ± 17%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.58 ±182%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
      5.89 ± 99%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      0.01 ± 14%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
      1.34 ±140%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
      0.21 ±211%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
      3.21 ± 15%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      0.13 ±188%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
      0.10 ±128%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
      3.21 ± 25%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      3.03 ± 26%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
      0.07 ±140%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
      0.82 ± 46%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      7.17 ± 32%    -100.0%       0.00        perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
    172.33 ± 96%    -100.0%       0.00        perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
     96.17          -100.0%       0.00        perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.preempt_schedule_common._cond_resched.stop_one_cpu
      6.67 ± 20%    -100.0%       0.00        perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      4.67 ± 42%    -100.0%       0.00        perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
     96.83          -100.0%       0.00        perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
      4.83 ± 45%    -100.0%       0.00        perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      0.61 ±194%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
      1.12 ± 83%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
     11.52 ± 22%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      4.32 ± 10%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
      5.92 ±152%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
      3.21 ± 15%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
      0.09 ±142%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
      2.24 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      2.74 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
      0.70 ± 76%    -100.0%       0.00        perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
      9.24 ± 24%    -100.0%       0.00        perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      4.31 ± 10%    -100.0%       0.00        perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
     17.80 ±  3%     -25.3%      13.29 ±  4%  perf-stat.i.MPKI
  3.91e+09           -36.6%  2.479e+09        perf-stat.i.branch-instructions
      1.16            +0.1        1.24 ±  3%  perf-stat.i.branch-miss-rate%
  44593160           -32.9%   29918950 ±  2%  perf-stat.i.branch-misses
  45190201 ±  6%     -51.9%   21750001 ±  6%  perf-stat.i.cache-misses
 3.369e+08 ±  3%     -54.6%  1.529e+08 ±  3%  perf-stat.i.cache-references
   2069040           -49.0%    1054803        perf-stat.i.context-switches
      1.90           +42.6%       2.71        perf-stat.i.cpi
 3.524e+10           -11.4%  3.123e+10        perf-stat.i.cpu-cycles
    171.19 ±  4%     -16.7%     142.54 ±  2%  perf-stat.i.cpu-migrations
    897.91 ±  6%     +72.0%       1544 ±  4%  perf-stat.i.cycles-between-cache-misses
      0.00 ± 55%      +0.0        0.02 ± 26%  perf-stat.i.dTLB-load-miss-rate%
     90639 ± 20%    +379.4%     434539 ± 10%  perf-stat.i.dTLB-load-misses
 5.159e+09           -39.3%  3.133e+09        perf-stat.i.dTLB-loads
      0.00 ± 24%      +0.0        0.00 ± 25%  perf-stat.i.dTLB-store-miss-rate%
     25252 ± 10%    +104.3%      51578 ± 12%  perf-stat.i.dTLB-store-misses
 3.225e+09           -44.8%  1.781e+09        perf-stat.i.dTLB-stores
     76.00            -3.5       72.47        perf-stat.i.iTLB-load-miss-rate%
  34565197           -48.9%   17646961        perf-stat.i.iTLB-load-misses
  10478017 ±  4%     -37.8%    6515713 ±  5%  perf-stat.i.iTLB-loads
 1.851e+10           -38.1%  1.146e+10        perf-stat.i.instructions
    583.76           +20.3%     702.31        perf-stat.i.instructions-per-iTLB-miss
      0.53           -29.0%       0.38        perf-stat.i.ipc
      0.37           -11.4%       0.33        perf-stat.i.metric.GHz
      0.59 ±  7%     +31.5%       0.78        perf-stat.i.metric.K/sec
    131.90           -40.2%      78.82        perf-stat.i.metric.M/sec
     87.72            +8.1       95.81        perf-stat.i.node-load-miss-rate%
   1334578 ±  3%     -79.2%     277059 ±  8%  perf-stat.i.node-loads
   7293953 ±  4%     -22.5%    5655933 ±  4%  perf-stat.i.node-store-misses
     18.20 ±  3%     -26.7%      13.34 ±  3%  perf-stat.overall.MPKI
      1.14            +0.1        1.21        perf-stat.overall.branch-miss-rate%
      1.90           +43.1%       2.73        perf-stat.overall.cpi
    783.04 ±  6%     +84.2%       1441 ±  6%  perf-stat.overall.cycles-between-cache-misses
      0.00 ± 20%      +0.0        0.01 ±  9%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 10%      +0.0        0.00 ± 12%  perf-stat.overall.dTLB-store-miss-rate%
     76.74            -3.7       73.05        perf-stat.overall.iTLB-load-miss-rate%
    535.61           +21.3%     649.61        perf-stat.overall.instructions-per-iTLB-miss
      0.53           -30.1%       0.37        perf-stat.overall.ipc
     88.60            +8.3       96.86        perf-stat.overall.node-load-miss-rate%
 3.847e+09           -36.6%  2.439e+09        perf-stat.ps.branch-instructions
  43872091           -32.9%   29436754 ±  2%  perf-stat.ps.branch-misses
  44454773 ±  6%     -51.9%   21395194 ±  6%  perf-stat.ps.cache-misses
 3.314e+08 ±  3%     -54.6%  1.504e+08 ±  3%  perf-stat.ps.cache-references
   2035394           -49.0%    1037584        perf-stat.ps.context-switches
 3.467e+10           -11.4%  3.073e+10        perf-stat.ps.cpu-cycles
    168.47 ±  4%     -16.7%     140.28 ±  2%  perf-stat.ps.cpu-migrations
     89350 ± 20%    +378.8%     427772 ± 10%  perf-stat.ps.dTLB-load-misses
 5.075e+09           -39.3%  3.082e+09        perf-stat.ps.dTLB-loads
     24898 ± 10%    +104.2%      50844 ± 12%  perf-stat.ps.dTLB-store-misses
 3.173e+09           -44.8%  1.752e+09        perf-stat.ps.dTLB-stores
  34003945           -48.9%   17359689        perf-stat.ps.iTLB-load-misses
  10308325 ±  4%     -37.8%    6410071 ±  5%  perf-stat.ps.iTLB-loads
 1.821e+10           -38.1%  1.128e+10        perf-stat.ps.instructions
   1312891 ±  3%     -79.2%     272597 ±  8%  perf-stat.ps.node-loads
   7175119 ±  4%     -22.5%    5563364 ±  4%  perf-stat.ps.node-store-misses
 1.151e+12           -38.1%   7.13e+11        perf-stat.total.instructions
     79831 ±  5%     +55.8%     124388 ±  3%  interrupts.CAL:Function_call_interrupts
    911.17 ± 40%     -74.2%     235.33 ± 29%  interrupts.CPU0.RES:Rescheduling_interrupts
      1922 ± 31%     -77.2%     437.33 ± 51%  interrupts.CPU1.RES:Rescheduling_interrupts
    747.83 ± 18%     +66.5%       1244 ±  9%  interrupts.CPU10.CAL:Function_call_interrupts
    736.67 ± 44%     -70.5%     217.17 ± 44%  interrupts.CPU10.RES:Rescheduling_interrupts
    757.17 ± 20%    +122.0%       1680 ± 58%  interrupts.CPU11.CAL:Function_call_interrupts
    800.00 ± 38%     -68.5%     252.17 ± 41%  interrupts.CPU11.RES:Rescheduling_interrupts
    718.50 ± 19%     +95.8%       1407 ± 26%  interrupts.CPU12.CAL:Function_call_interrupts
    651.00 ± 35%     -72.1%     181.83 ± 78%  interrupts.CPU12.RES:Rescheduling_interrupts
    695.33 ± 42%     -74.4%     178.17 ± 55%  interrupts.CPU13.RES:Rescheduling_interrupts
    676.67 ± 25%    +219.5%       2162 ± 65%  interrupts.CPU14.CAL:Function_call_interrupts
    810.17 ± 45%     -79.9%     162.50 ± 91%  interrupts.CPU14.RES:Rescheduling_interrupts
      3349 ± 62%     -89.8%     340.50 ± 52%  interrupts.CPU15.NMI:Non-maskable_interrupts
      3349 ± 62%     -89.8%     340.50 ± 52%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    477.33 ± 17%     -44.1%     266.67 ± 64%  interrupts.CPU15.RES:Rescheduling_interrupts
    655.83 ± 26%     -70.1%     196.00 ± 63%  interrupts.CPU16.RES:Rescheduling_interrupts
    611.33 ± 35%     -69.8%     184.83 ± 71%  interrupts.CPU17.RES:Rescheduling_interrupts
    724.83 ± 40%     -71.2%     209.00 ± 45%  interrupts.CPU18.RES:Rescheduling_interrupts
    608.67 ± 34%     -68.2%     193.50 ±110%  interrupts.CPU19.RES:Rescheduling_interrupts
      1701 ± 32%     -74.2%     438.83 ± 42%  interrupts.CPU2.RES:Rescheduling_interrupts
    599.33 ± 19%     -64.3%     213.83 ± 76%  interrupts.CPU21.RES:Rescheduling_interrupts
    926.50 ± 41%     -75.2%     230.17 ± 79%  interrupts.CPU22.RES:Rescheduling_interrupts
    630.00 ± 19%     -71.4%     180.17 ± 55%  interrupts.CPU23.RES:Rescheduling_interrupts
      1787 ± 28%     -60.1%     712.50 ± 31%  interrupts.CPU24.RES:Rescheduling_interrupts
      1343 ± 21%     -59.3%     547.33 ± 16%  interrupts.CPU25.RES:Rescheduling_interrupts
    775.17 ± 17%     +55.6%       1206 ±  4%  interrupts.CPU26.CAL:Function_call_interrupts
      1566 ± 34%     -63.9%     565.83 ± 31%  interrupts.CPU26.RES:Rescheduling_interrupts
    679.67 ± 23%     +84.6%       1254 ± 10%  interrupts.CPU27.CAL:Function_call_interrupts
      1631 ± 23%     -60.4%     646.33 ± 35%  interrupts.CPU27.RES:Rescheduling_interrupts
    731.17 ± 24%     +74.2%       1274 ±  8%  interrupts.CPU28.CAL:Function_call_interrupts
      1339 ± 31%     -59.0%     549.50 ± 49%  interrupts.CPU29.RES:Rescheduling_interrupts
    883.50 ± 15%     +81.9%       1607 ± 41%  interrupts.CPU3.CAL:Function_call_interrupts
      1237 ± 37%     -78.0%     272.67 ± 46%  interrupts.CPU3.RES:Rescheduling_interrupts
    736.33 ± 20%     +81.3%       1334 ± 10%  interrupts.CPU30.CAL:Function_call_interrupts
      1310 ± 26%     -62.1%     497.00 ± 34%  interrupts.CPU30.RES:Rescheduling_interrupts
    840.83 ± 25%     +52.4%       1281 ± 16%  interrupts.CPU32.CAL:Function_call_interrupts
    893.00 ± 35%     -67.6%     289.67 ± 39%  interrupts.CPU32.RES:Rescheduling_interrupts
      2714 ± 73%     -76.0%     650.83 ±107%  interrupts.CPU33.NMI:Non-maskable_interrupts
      2714 ± 73%     -76.0%     650.83 ±107%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
      1023 ± 20%     -52.0%     490.67 ± 34%  interrupts.CPU33.RES:Rescheduling_interrupts
    754.33 ± 15%     +62.8%       1228 ± 13%  interrupts.CPU34.CAL:Function_call_interrupts
      1069 ± 39%     -64.5%     379.33 ± 43%  interrupts.CPU34.RES:Rescheduling_interrupts
    762.50 ± 22%     +53.0%       1166 ±  4%  interrupts.CPU35.CAL:Function_call_interrupts
      1232 ± 11%     -57.9%     518.33 ± 44%  interrupts.CPU35.RES:Rescheduling_interrupts
    812.50 ± 20%     +56.7%       1273 ± 15%  interrupts.CPU36.CAL:Function_call_interrupts
    721.33 ± 28%     +71.3%       1235 ±  6%  interrupts.CPU38.CAL:Function_call_interrupts
    784.17 ± 21%     +83.4%       1438 ± 19%  interrupts.CPU4.CAL:Function_call_interrupts
      1249 ± 42%     -83.2%     210.33 ± 33%  interrupts.CPU4.RES:Rescheduling_interrupts
    727.17 ± 11%    +121.1%       1608 ± 61%  interrupts.CPU40.CAL:Function_call_interrupts
    661.67 ± 14%     +75.0%       1157 ±  5%  interrupts.CPU41.CAL:Function_call_interrupts
    936.50 ± 37%     -67.1%     308.17 ± 45%  interrupts.CPU41.RES:Rescheduling_interrupts
    673.33 ± 17%     +88.8%       1271 ±  6%  interrupts.CPU43.CAL:Function_call_interrupts
      1008 ± 23%     -48.2%     522.83 ± 35%  interrupts.CPU43.RES:Rescheduling_interrupts
    647.00 ± 24%     +80.2%       1165 ±  3%  interrupts.CPU44.CAL:Function_call_interrupts
    690.67 ± 20%     +68.5%       1163 ±  3%  interrupts.CPU47.CAL:Function_call_interrupts
      1016 ± 32%     -58.8%     418.83 ± 38%  interrupts.CPU47.RES:Rescheduling_interrupts
    591.83 ± 49%     -74.1%     153.00 ± 52%  interrupts.CPU48.RES:Rescheduling_interrupts
    781.17 ± 23%     +54.2%       1204 ±  9%  interrupts.CPU49.CAL:Function_call_interrupts
    826.17 ± 49%    +250.2%       2892 ± 69%  interrupts.CPU49.NMI:Non-maskable_interrupts
    826.17 ± 49%    +250.2%       2892 ± 69%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
    946.83 ± 40%     -68.9%     294.17 ± 52%  interrupts.CPU5.RES:Rescheduling_interrupts
    760.83 ± 20%     +55.8%       1185 ±  8%  interrupts.CPU50.CAL:Function_call_interrupts
    729.33 ± 34%     -69.5%     222.33 ± 68%  interrupts.CPU50.RES:Rescheduling_interrupts
    756.83 ± 13%     +59.1%       1204 ±  6%  interrupts.CPU51.CAL:Function_call_interrupts
    669.83 ± 13%     +78.1%       1193 ± 11%  interrupts.CPU52.CAL:Function_call_interrupts
    665.50 ± 33%     -59.5%     269.67 ± 49%  interrupts.CPU52.RES:Rescheduling_interrupts
    704.50 ± 12%     +73.0%       1218 ±  5%  interrupts.CPU53.CAL:Function_call_interrupts
    691.00 ± 12%     +67.5%       1157 ±  3%  interrupts.CPU54.CAL:Function_call_interrupts
    663.00 ± 10%     +78.8%       1185 ±  7%  interrupts.CPU55.CAL:Function_call_interrupts
    647.50 ± 15%     +80.8%       1170 ±  5%  interrupts.CPU56.CAL:Function_call_interrupts
    644.83 ± 15%     +82.3%       1175 ±  6%  interrupts.CPU57.CAL:Function_call_interrupts
    689.17 ± 12%     +70.0%       1171 ±  4%  interrupts.CPU58.CAL:Function_call_interrupts
    667.67 ± 39%     -77.1%     152.67 ± 42%  interrupts.CPU58.RES:Rescheduling_interrupts
    638.50 ± 17%     +82.7%       1166        interrupts.CPU59.CAL:Function_call_interrupts
    525.17 ± 27%     -70.4%     155.67 ± 55%  interrupts.CPU59.RES:Rescheduling_interrupts
    936.83 ± 36%     -65.7%     321.33 ± 52%  interrupts.CPU6.RES:Rescheduling_interrupts
    738.50 ± 18%    +121.2%       1633 ± 60%  interrupts.CPU60.CAL:Function_call_interrupts
    606.33 ± 48%     -61.6%     233.00 ± 35%  interrupts.CPU60.RES:Rescheduling_interrupts
    694.17 ± 18%     +74.7%       1212 ±  8%  interrupts.CPU61.CAL:Function_call_interrupts
    654.33 ± 22%     +85.2%       1211 ± 13%  interrupts.CPU62.CAL:Function_call_interrupts
    502.00 ± 60%     -70.9%     146.17 ± 66%  interrupts.CPU62.RES:Rescheduling_interrupts
    712.33 ± 10%     +84.3%       1313 ± 23%  interrupts.CPU63.CAL:Function_call_interrupts
      3704 ± 60%     -89.1%     402.83 ± 51%  interrupts.CPU63.NMI:Non-maskable_interrupts
      3704 ± 60%     -89.1%     402.83 ± 51%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
    617.33 ± 29%     -60.2%     245.50 ± 57%  interrupts.CPU63.RES:Rescheduling_interrupts
    650.33 ± 14%     +80.7%       1174 ±  6%  interrupts.CPU64.CAL:Function_call_interrupts
    669.00 ± 21%     +75.9%       1176 ±  8%  interrupts.CPU65.CAL:Function_call_interrupts
    718.67 ± 31%     +77.6%       1276 ± 14%  interrupts.CPU66.CAL:Function_call_interrupts
    715.33 ± 15%     +58.9%       1136 ±  4%  interrupts.CPU67.CAL:Function_call_interrupts
    602.50 ± 39%     -71.3%     172.67 ± 62%  interrupts.CPU67.RES:Rescheduling_interrupts
    578.17 ±  6%    +175.3%       1591 ± 60%  interrupts.CPU68.CAL:Function_call_interrupts
    660.33 ± 13%     +76.9%       1168 ±  5%  interrupts.CPU69.CAL:Function_call_interrupts
    728.67 ± 25%     +58.2%       1152 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
    815.00 ± 47%     -77.9%     180.50 ± 60%  interrupts.CPU7.RES:Rescheduling_interrupts
    671.50 ± 17%     +78.2%       1196 ± 12%  interrupts.CPU70.CAL:Function_call_interrupts
    594.33 ± 60%     -77.8%     132.17 ± 51%  interrupts.CPU70.RES:Rescheduling_interrupts
    730.00 ± 30%     +62.5%       1186 ±  4%  interrupts.CPU71.CAL:Function_call_interrupts
    660.67 ± 32%     -82.1%     118.50 ± 40%  interrupts.CPU71.RES:Rescheduling_interrupts
    859.50 ± 22%     +43.0%       1229 ±  6%  interrupts.CPU72.CAL:Function_call_interrupts
      1160 ± 41%     -68.4%     366.33 ± 22%  interrupts.CPU72.RES:Rescheduling_interrupts
    811.83 ± 18%     +59.2%       1292 ± 15%  interrupts.CPU73.CAL:Function_call_interrupts
    851.33 ± 20%     +50.1%       1277 ± 10%  interrupts.CPU74.CAL:Function_call_interrupts
      1067 ± 22%     -53.7%     493.83 ± 52%  interrupts.CPU74.RES:Rescheduling_interrupts
    858.83 ± 10%     +44.4%       1240 ±  9%  interrupts.CPU75.CAL:Function_call_interrupts
    747.50 ± 13%     +60.0%       1196 ±  5%  interrupts.CPU76.CAL:Function_call_interrupts
    757.83 ± 15%     +57.1%       1190 ±  7%  interrupts.CPU77.CAL:Function_call_interrupts
    965.83 ± 22%     -61.7%     369.50 ± 32%  interrupts.CPU79.RES:Rescheduling_interrupts
    654.83 ± 16%     +84.7%       1209 ± 13%  interrupts.CPU8.CAL:Function_call_interrupts
    667.33 ± 53%     -70.2%     199.17 ± 79%  interrupts.CPU8.RES:Rescheduling_interrupts
    773.33 ± 17%     +65.6%       1281 ± 13%  interrupts.CPU80.CAL:Function_call_interrupts
      2552 ± 31%     -60.6%       1004 ± 73%  interrupts.CPU80.NMI:Non-maskable_interrupts
      2552 ± 31%     -60.6%       1004 ± 73%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
    962.17 ± 21%     -60.4%     381.00 ± 14%  interrupts.CPU80.RES:Rescheduling_interrupts
    769.33 ± 20%     +53.9%       1184 ± 10%  interrupts.CPU81.CAL:Function_call_interrupts
      2283 ± 56%     -71.1%     658.83 ± 73%  interrupts.CPU81.NMI:Non-maskable_interrupts
      2283 ± 56%     -71.1%     658.83 ± 73%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
    717.83 ± 18%     -40.2%     429.50 ± 31%  interrupts.CPU81.RES:Rescheduling_interrupts
    635.33 ±  6%     +88.8%       1199 ±  8%  interrupts.CPU82.CAL:Function_call_interrupts
    673.00 ± 16%     +79.9%       1210 ±  5%  interrupts.CPU83.CAL:Function_call_interrupts
    778.67 ± 24%     +69.0%       1316 ± 12%  interrupts.CPU85.CAL:Function_call_interrupts
    703.83 ± 19%     +81.6%       1277 ± 11%  interrupts.CPU86.CAL:Function_call_interrupts
    674.00 ± 17%     +68.9%       1138 ±  3%  interrupts.CPU87.CAL:Function_call_interrupts
    673.17 ± 10%     +89.9%       1278 ± 17%  interrupts.CPU88.CAL:Function_call_interrupts
    700.17 ± 17%     +66.1%       1162 ± 10%  interrupts.CPU89.CAL:Function_call_interrupts
    909.17 ± 37%     -79.0%     190.67 ± 44%  interrupts.CPU9.RES:Rescheduling_interrupts
    671.67 ± 16%     +78.8%       1200 ± 10%  interrupts.CPU90.CAL:Function_call_interrupts
    869.17 ± 28%     -53.2%     407.17 ± 34%  interrupts.CPU90.RES:Rescheduling_interrupts
    714.33 ± 12%     +71.6%       1226 ± 10%  interrupts.CPU91.CAL:Function_call_interrupts
    694.50 ± 11%     +77.0%       1229 ±  7%  interrupts.CPU92.CAL:Function_call_interrupts
    694.67 ± 22%     +70.0%       1181 ±  5%  interrupts.CPU93.CAL:Function_call_interrupts
    780.67 ± 26%     -59.3%     318.00 ± 31%  interrupts.CPU93.RES:Rescheduling_interrupts
    629.50 ± 11%     +81.7%       1143 ±  3%  interrupts.CPU94.CAL:Function_call_interrupts
    589.00           +99.0%       1172 ±  6%  interrupts.CPU95.CAL:Function_call_interrupts
    671.50 ± 30%     -61.0%     261.67 ± 61%  interrupts.CPU95.RES:Rescheduling_interrupts
     83175 ±  6%     -60.3%      33000 ±  8%  interrupts.RES:Rescheduling_interrupts
     10.72 ±  9%      -7.4        3.31 ± 10%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.70 ±  9%      -7.4        3.30 ± 10%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.08 ±  9%      -7.2        2.92 ± 10%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.04 ±  9%      -7.2        2.88 ± 10%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      6.46 ±  8%      -6.5        0.00        perf-profile.calltrace.cycles-pp.__kmalloc_node.seq_read_iter.seq_read.vfs_read.ksys_read
      5.52 ±  7%      -5.5        0.00        perf-profile.calltrace.cycles-pp.kfree.single_release.__fput.task_work_run.exit_to_user_mode_prepare
      5.45 ±  8%      -5.5        0.00        perf-profile.calltrace.cycles-pp.obj_cgroup_charge.__kmalloc_node.seq_read_iter.seq_read.vfs_read
     32.13 ±  7%      -4.7       27.43 ±  8%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.51 ±  7%      -2.9        0.65 ±  7%  perf-profile.calltrace.cycles-pp.alloc_empty_file.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      3.49 ±  7%      -2.8        0.64 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_file.alloc_empty_file.path_openat.do_filp_open.do_sys_openat2
      6.34 ±  7%      -2.7        3.69 ± 11%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.98 ±  7%      -2.5        3.45 ± 10%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.15 ±  9%      -2.4        2.73 ± 10%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      5.49 ±  7%      -2.2        3.30 ± 11%  perf-profile.calltrace.cycles-pp.eventfd_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.98 ±  7%      -2.0        2.98 ± 10%  perf-profile.calltrace.cycles-pp.__wake_up_common.eventfd_write.vfs_write.ksys_write.do_syscall_64
      4.58 ±  7%      -1.8        2.74 ±  9%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.eventfd_write.vfs_write.ksys_write
      2.45 ±  7%      -1.8        0.62 ± 13%  perf-profile.calltrace.cycles-pp.do_dentry_open.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      4.47 ±  7%      -1.8        2.68 ± 10%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.28 ±  7%      -1.7        2.54 ± 10%  perf-profile.calltrace.cycles-pp.eventfd_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      3.01 ±  7%      -1.4        1.59 ±  8%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.10 ±  7%      -1.3        1.76 ±  8%  perf-profile.calltrace.cycles-pp.schedule.eventfd_read.new_sync_read.vfs_read.ksys_read
      2.06 ± 13%      -1.1        0.97 ± 10%  perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      1.81 ± 13%      -1.0        0.82 ± 10%  perf-profile.calltrace.cycles-pp.walk_component.link_path_walk.path_openat.do_filp_open.do_sys_openat2
      2.04 ±  6%      -1.0        1.08 ±  8%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.eventfd_write.vfs_write
      1.96 ±  6%      -0.9        1.04 ±  9%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.eventfd_write
      1.49 ±  6%      -0.7        0.80 ±  9%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common
      1.30 ± 15%      -0.7        0.64 ±  9%  perf-profile.calltrace.cycles-pp.lookup_fast.walk_component.link_path_walk.path_openat.do_filp_open
      1.51 ±  8%      -0.5        0.99 ± 10%  perf-profile.calltrace.cycles-pp.seq_show.seq_read_iter.seq_read.vfs_read.ksys_read
      0.00            +0.7        0.73 ±  8%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.00            +0.8        0.78 ±  6%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.eventfd_read
      0.00            +0.9        0.88 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.__vmalloc_node_range.__vmalloc_node.seq_read_iter.seq_read
      0.00            +0.9        0.88 ±  6%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.eventfd_read.new_sync_read
      0.00            +1.1        1.09 ±  7%  perf-profile.calltrace.cycles-pp.insert_vmap_area.alloc_vmap_area.__get_vm_area_node.__vmalloc_node_range.__vmalloc_node
      0.00            +1.5        1.53 ±  8%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.00            +1.7        1.68 ±  7%  perf-profile.calltrace.cycles-pp.__schedule.schedule.eventfd_read.new_sync_read.vfs_read
      0.00            +5.0        4.98 ±  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.find_vmap_area.__vunmap.single_release
      0.00            +5.1        5.13 ±  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_vmap_area_noflush.remove_vm_area.__vunmap
      0.00            +5.3        5.32 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock.find_vmap_area.__vunmap.single_release.__fput
      0.00            +5.4        5.35 ±  7%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.remove_vm_area.__vunmap.single_release
      0.00            +5.4        5.42 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock.free_vmap_area_noflush.remove_vm_area.__vunmap.single_release
      0.00            +5.4        5.45 ±  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__get_vm_area_node.__vmalloc_node_range.__vmalloc_node
      0.00            +5.6        5.57 ±  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.alloc_vmap_area.__get_vm_area_node.__vmalloc_node_range
     14.60 ±  7%      +5.6       20.17 ±  8%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +5.6        5.58 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock.remove_vm_area.__vunmap.single_release.__fput
     14.40 ±  7%      +5.7       20.07 ±  8%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +5.7        5.75 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__get_vm_area_node.__vmalloc_node_range.__vmalloc_node.seq_read_iter
      0.00            +6.0        6.05 ±  7%  perf-profile.calltrace.cycles-pp.find_vmap_area.__vunmap.single_release.__fput.task_work_run
      0.00            +6.1        6.12 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock.alloc_vmap_area.__get_vm_area_node.__vmalloc_node_range.__vmalloc_node
      0.00            +6.9        6.93 ±  8%  perf-profile.calltrace.cycles-pp.free_vmap_area_noflush.remove_vm_area.__vunmap.single_release.__fput
      0.00            +8.3        8.28 ±  8%  perf-profile.calltrace.cycles-pp.alloc_vmap_area.__get_vm_area_node.__vmalloc_node_range.__vmalloc_node.seq_read_iter
      8.44 ±  7%      +8.3       16.79 ±  8%  perf-profile.calltrace.cycles-pp.seq_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.38 ±  7%      +8.4       16.74 ±  8%  perf-profile.calltrace.cycles-pp.seq_read_iter.seq_read.vfs_read.ksys_read.do_syscall_64
     40.76 ±  7%      +8.5       49.21 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      8.41 ±  7%     +13.3       21.66 ±  7%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      8.34 ±  7%     +13.3       21.61 ±  7%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      7.90 ±  7%     +13.5       21.36 ±  7%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.00           +13.5       13.54 ±  8%  perf-profile.calltrace.cycles-pp.remove_vm_area.__vunmap.single_release.__fput.task_work_run
      7.70 ±  7%     +13.5       21.25 ±  7%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      6.94 ±  7%     +13.9       20.82 ±  7%  perf-profile.calltrace.cycles-pp.single_release.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00           +14.2       14.19 ±  8%  perf-profile.calltrace.cycles-pp.__get_vm_area_node.__vmalloc_node_range.__vmalloc_node.seq_read_iter.seq_read
      0.00           +15.5       15.55 ±  8%  perf-profile.calltrace.cycles-pp.__vmalloc_node_range.__vmalloc_node.seq_read_iter.seq_read.vfs_read
      0.00           +15.6       15.55 ±  8%  perf-profile.calltrace.cycles-pp.__vmalloc_node.seq_read_iter.seq_read.vfs_read.ksys_read
      0.00           +20.5       20.47 ±  7%  perf-profile.calltrace.cycles-pp.__vunmap.single_release.__fput.task_work_run.exit_to_user_mode_prepare
      8.82 ±  7%      -8.6        0.20 ±  9%  perf-profile.children.cycles-pp.refill_obj_stock
     10.72 ±  9%      -7.4        3.31 ± 10%  perf-profile.children.cycles-pp.do_sys_open
     10.71 ±  9%      -7.4        3.30 ± 10%  perf-profile.children.cycles-pp.do_sys_openat2
     10.09 ±  9%      -7.2        2.92 ± 10%  perf-profile.children.cycles-pp.do_filp_open
     10.05 ±  9%      -7.2        2.89 ± 10%  perf-profile.children.cycles-pp.path_openat
      7.18 ±  8%      -7.0        0.13 ± 19%  perf-profile.children.cycles-pp.obj_cgroup_charge
      6.47 ±  8%      -6.4        0.08 ± 20%  perf-profile.children.cycles-pp.__kmalloc_node
      6.24 ±  8%      -6.1        0.10 ± 10%  perf-profile.children.cycles-pp.drain_obj_stock
      5.99 ±  7%      -6.0        0.00        perf-profile.children.cycles-pp.__sched_text_start
      5.53 ±  7%      -5.4        0.16 ± 14%  perf-profile.children.cycles-pp.kfree
     32.20 ±  7%      -4.7       27.51 ±  8%  perf-profile.children.cycles-pp.do_syscall_64
      4.35 ±  9%      -4.2        0.18 ± 10%  perf-profile.children.cycles-pp.page_counter_uncharge
      4.33 ±  9%      -4.1        0.18 ± 11%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge
      4.31 ±  9%      -4.1        0.17 ± 11%  perf-profile.children.cycles-pp.page_counter_cancel
      3.93 ±  8%      -3.5        0.48 ±  9%  perf-profile.children.cycles-pp.__memcg_kmem_charge
      3.83 ±  8%      -3.4        0.44 ±  9%  perf-profile.children.cycles-pp.page_counter_try_charge
      3.52 ±  7%      -3.0        0.57 ± 10%  perf-profile.children.cycles-pp.kmem_cache_alloc
      3.51 ±  8%      -2.9        0.65 ±  7%  perf-profile.children.cycles-pp.alloc_empty_file
      3.49 ±  8%      -2.8        0.64 ±  8%  perf-profile.children.cycles-pp.__alloc_file
      6.35 ±  7%      -2.7        3.70 ± 11%  perf-profile.children.cycles-pp.ksys_write
      6.00 ±  7%      -2.5        3.47 ± 10%  perf-profile.children.cycles-pp.vfs_write
      5.17 ±  9%      -2.4        2.78 ±  9%  perf-profile.children.cycles-pp.poll_idle
      5.49 ±  7%      -2.2        3.31 ± 11%  perf-profile.children.cycles-pp.eventfd_write
      4.99 ±  7%      -2.0        2.99 ± 10%  perf-profile.children.cycles-pp.__wake_up_common
      1.90 ±  9%      -1.9        0.05 ± 45%  perf-profile.children.cycles-pp.propagate_protected_usage
      4.60 ±  7%      -1.8        2.75 ± 10%  perf-profile.children.cycles-pp.try_to_wake_up
      2.45 ±  7%      -1.8        0.62 ± 13%  perf-profile.children.cycles-pp.do_dentry_open
      4.48 ±  7%      -1.8        2.69 ± 10%  perf-profile.children.cycles-pp.new_sync_read
      2.33 ±  5%      -1.8        0.55 ± 11%  perf-profile.children.cycles-pp.kmem_cache_free
      4.32 ±  7%      -1.7        2.57 ± 10%  perf-profile.children.cycles-pp.eventfd_read
      3.04 ±  7%      -1.4        1.61 ±  8%  perf-profile.children.cycles-pp.schedule_idle
      3.12 ±  7%      -1.3        1.78 ±  8%  perf-profile.children.cycles-pp.schedule
      1.35 ±  7%      -1.2        0.11 ± 15%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      1.53 ±  5%      -1.2        0.36 ± 13%  perf-profile.children.cycles-pp.single_open
      2.06 ± 13%      -1.1        0.97 ± 10%  perf-profile.children.cycles-pp.link_path_walk
      1.81 ± 13%      -1.0        0.82 ± 11%  perf-profile.children.cycles-pp.walk_component
      1.28 ±  7%      -1.0        0.31 ± 11%  perf-profile.children.cycles-pp.security_file_permission
      1.13 ±  8%      -0.9        0.21 ± 15%  perf-profile.children.cycles-pp.common_file_perm
      2.05 ±  7%      -0.9        1.17 ±  8%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.98 ±  7%      -0.8        1.13 ±  8%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.62 ± 13%      -0.8        0.85 ±  9%  perf-profile.children.cycles-pp.lookup_fast
      1.60 ±  6%      -0.8        0.83 ±  7%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.92 ±  7%      -0.8        0.17 ± 15%  perf-profile.children.cycles-pp.kmem_cache_alloc_trace
      1.62 ±  6%      -0.7        0.88 ±  5%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.81 ± 10%      -0.7        0.12 ± 13%  perf-profile.children.cycles-pp.ima_file_check
      0.80 ± 10%      -0.7        0.11 ± 14%  perf-profile.children.cycles-pp.security_task_getsecid
      0.78 ± 10%      -0.7        0.11 ± 14%  perf-profile.children.cycles-pp.apparmor_task_getsecid
      1.45 ±  7%      -0.7        0.79 ±  6%  perf-profile.children.cycles-pp.dequeue_entity
      1.52 ±  7%      -0.6        0.87 ± 10%  perf-profile.children.cycles-pp.enqueue_entity
      0.94 ± 13%      -0.6        0.31 ± 17%  perf-profile.children.cycles-pp.dput
      0.99 ±  6%      -0.6        0.37 ± 10%  perf-profile.children.cycles-pp.rcu_do_batch
      1.04 ±  6%      -0.6        0.42 ± 10%  perf-profile.children.cycles-pp.rcu_core
      1.34 ±  6%      -0.6        0.76 ± 10%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.81 ±  7%      -0.6        0.24 ± 10%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.80 ±  7%      -0.6        0.23 ± 10%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.74 ± 10%      -0.6        0.18 ± 14%  perf-profile.children.cycles-pp.security_file_open
      0.84 ±  6%      -0.6        0.28 ± 11%  perf-profile.children.cycles-pp.ret_from_fork
      0.83 ±  7%      -0.6        0.28 ± 11%  perf-profile.children.cycles-pp.kthread
      0.72 ± 10%      -0.5        0.18 ± 13%  perf-profile.children.cycles-pp.apparmor_file_open
      1.52 ±  8%      -0.5        0.99 ± 10%  perf-profile.children.cycles-pp.seq_show
      1.06 ±  7%      -0.5        0.54 ±  8%  perf-profile.children.cycles-pp.set_next_entity
      1.13 ±  6%      -0.5        0.68 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.78 ± 20%      -0.4        0.34 ± 13%  perf-profile.children.cycles-pp.unlazy_child
      0.60 ± 16%      -0.4        0.17 ± 19%  perf-profile.children.cycles-pp.terminate_walk
      0.51 ± 10%      -0.4        0.08 ± 20%  perf-profile.children.cycles-pp.aa_get_task_label
      1.07 ±  9%      -0.4        0.64 ±  8%  perf-profile.children.cycles-pp.seq_printf
      0.60 ±  4%      -0.4        0.18 ± 12%  perf-profile.children.cycles-pp.seq_open
      1.03 ±  9%      -0.4        0.62 ±  8%  perf-profile.children.cycles-pp.seq_vprintf
      0.53 ± 12%      -0.4        0.12 ± 26%  perf-profile.children.cycles-pp.lockref_put_return
      1.01 ±  9%      -0.4        0.61 ±  8%  perf-profile.children.cycles-pp.vsnprintf
      0.81 ±  7%      -0.4        0.43 ±  7%  perf-profile.children.cycles-pp.update_curr
      0.67 ±  9%      -0.4        0.30 ±  7%  perf-profile.children.cycles-pp.security_file_alloc
      0.90 ±  8%      -0.4        0.53 ±  7%  perf-profile.children.cycles-pp.update_rq_clock
      0.57 ±  8%      -0.3        0.24 ±  6%  perf-profile.children.cycles-pp.apparmor_file_alloc_security
      0.52 ± 25%      -0.3        0.19 ± 20%  perf-profile.children.cycles-pp.step_into
      0.58 ± 25%      -0.3        0.26 ± 16%  perf-profile.children.cycles-pp.lockref_get_not_dead
      0.90 ±  9%      -0.3        0.61 ± 21%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.37 ± 18%      -0.3        0.09 ± 15%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.33 ± 30%      -0.3        0.08 ± 17%  perf-profile.children.cycles-pp.__legitimize_path
      0.29 ± 26%      -0.3        0.04 ± 71%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.49 ±  8%      -0.2        0.25 ±  9%  perf-profile.children.cycles-pp.security_file_free
      0.48 ±  8%      -0.2        0.25 ±  9%  perf-profile.children.cycles-pp.apparmor_file_free_security
      0.48 ±  8%      -0.2        0.27 ±  9%  perf-profile.children.cycles-pp.__switch_to
      0.57 ±  7%      -0.2        0.36 ± 14%  perf-profile.children.cycles-pp._copy_to_iter
      0.47 ±  8%      -0.2        0.27 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.67 ±  9%      -0.2        0.47 ± 20%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.47 ±  8%      -0.2        0.28 ± 15%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.43 ±  9%      -0.2        0.25 ±  8%  perf-profile.children.cycles-pp.__switch_to_asm
      0.35 ±  8%      -0.2        0.17 ± 12%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.45 ±  6%      -0.2        0.27 ± 16%  perf-profile.children.cycles-pp.__fdget_pos
      0.33 ±  5%      -0.2        0.16 ±  8%  perf-profile.children.cycles-pp.pick_next_entity
      0.40 ±  7%      -0.2        0.24 ± 18%  perf-profile.children.cycles-pp.__fget_light
      0.30 ± 11%      -0.2        0.15 ± 13%  perf-profile.children.cycles-pp.update_cfs_group
      0.32 ±  9%      -0.1        0.18 ± 11%  perf-profile.children.cycles-pp.switch_fpu_return
      0.35 ±  9%      -0.1        0.21 ± 10%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.30 ±  6%      -0.1        0.16 ± 11%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.34 ± 10%      -0.1        0.20 ± 10%  perf-profile.children.cycles-pp.check_preempt_curr
      0.34 ±  9%      -0.1        0.20 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.41 ±  5%      -0.1        0.27 ± 11%  perf-profile.children.cycles-pp.eventfd_show_fdinfo
      0.39 ±  8%      -0.1        0.25 ± 17%  perf-profile.children.cycles-pp.__might_fault
      0.47 ±  8%      -0.1        0.33 ± 13%  perf-profile.children.cycles-pp.___might_sleep
      0.31 ±  7%      -0.1        0.18 ± 10%  perf-profile.children.cycles-pp.available_idle_cpu
      0.32 ± 10%      -0.1        0.18 ± 11%  perf-profile.children.cycles-pp.pid_revalidate
      0.41 ±  6%      -0.1        0.27 ± 11%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.27 ±  9%      -0.1        0.14 ± 12%  perf-profile.children.cycles-pp.__x64_sys_close
      0.37 ±  8%      -0.1        0.24 ± 15%  perf-profile.children.cycles-pp.getname_flags
      0.28 ±  9%      -0.1        0.15 ±  9%  perf-profile.children.cycles-pp.reweight_entity
      0.19 ± 12%      -0.1        0.06 ± 51%  perf-profile.children.cycles-pp.cpuacct_charge
      0.21 ±  5%      -0.1        0.09 ± 10%  perf-profile.children.cycles-pp.__check_object_size
      0.37 ±  6%      -0.1        0.25 ± 10%  perf-profile.children.cycles-pp.sched_clock
      0.46 ±  6%      -0.1        0.34 ± 15%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.22 ±  9%      -0.1        0.10 ± 12%  perf-profile.children.cycles-pp.perf_tp_event
      0.32 ±  9%      -0.1        0.20 ± 21%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.29 ±  7%      -0.1        0.18 ±  6%  perf-profile.children.cycles-pp.format_decode
      0.35 ±  6%      -0.1        0.23 ± 12%  perf-profile.children.cycles-pp.native_sched_clock
      0.29 ± 11%      -0.1        0.17 ±  9%  perf-profile.children.cycles-pp.number
      0.32 ±  9%      -0.1        0.21 ± 20%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.20 ± 10%      -0.1        0.10 ± 12%  perf-profile.children.cycles-pp.filp_close
      0.25 ± 11%      -0.1        0.16 ±  9%  perf-profile.children.cycles-pp.__entry_text_start
      0.21 ± 13%      -0.1        0.12 ± 16%  perf-profile.children.cycles-pp.finish_task_switch
      0.25 ± 11%      -0.1        0.15 ± 12%  perf-profile.children.cycles-pp.tid_fd_revalidate
      0.15 ± 14%      -0.1        0.06 ± 47%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.19 ±  9%      -0.1        0.10 ± 10%  perf-profile.children.cycles-pp.fput_many
      0.23 ±  8%      -0.1        0.14 ± 12%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.19 ±  9%      -0.1        0.11 ± 18%  perf-profile.children.cycles-pp.task_dump_owner
      0.20 ±  7%      -0.1        0.11 ± 13%  perf-profile.children.cycles-pp.strncpy_from_user
      0.10 ± 14%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.alloc_fd
      0.23 ± 12%      -0.1        0.15 ± 12%  perf-profile.children.cycles-pp.resched_curr
      0.20 ± 10%      -0.1        0.13 ± 14%  perf-profile.children.cycles-pp.memcpy_erms
      0.18 ±  8%      -0.1        0.10 ± 14%  perf-profile.children.cycles-pp.inode_permission
      0.18 ±  4%      -0.1        0.11 ± 17%  perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
      0.10 ± 27%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.legitimize_mnt
      0.13 ± 24%      -0.1        0.06 ± 14%  perf-profile.children.cycles-pp.__legitimize_mnt
      0.15 ±  9%      -0.1        0.08 ± 19%  perf-profile.children.cycles-pp.pid_update_inode
      0.20 ±  9%      -0.1        0.13 ± 10%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.30 ±  5%      -0.1        0.23 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.15 ± 15%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.12 ± 11%      -0.1        0.06 ± 14%  perf-profile.children.cycles-pp.task_work_add
      0.16 ±  8%      -0.1        0.10 ± 12%  perf-profile.children.cycles-pp.copyout
      0.27 ±  5%      -0.1        0.21 ± 16%  perf-profile.children.cycles-pp.read_tsc
      0.19 ±  7%      -0.1        0.14 ± 10%  perf-profile.children.cycles-pp.place_entity
      0.10 ±  4%      -0.1        0.05 ± 45%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.09 ±  5%      -0.1        0.04 ± 71%  perf-profile.children.cycles-pp.aa_file_perm
      0.09 ±  7%      -0.1        0.04 ± 45%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.13 ±  7%      -0.0        0.08 ± 21%  perf-profile.children.cycles-pp.___perf_sw_event
      0.14 ±  3%      -0.0        0.09 ± 12%  perf-profile.children.cycles-pp._copy_from_user
      0.11 ± 13%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.call_rcu
      0.10 ±  7%      -0.0        0.05 ± 47%  perf-profile.children.cycles-pp.update_min_vruntime
      0.13 ±  7%      -0.0        0.09 ± 18%  perf-profile.children.cycles-pp.fsnotify
      0.15 ±  9%      -0.0        0.11 ± 11%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.09 ±  7%      -0.0        0.05 ± 46%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.14 ± 15%      -0.0        0.09 ± 14%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.11 ± 10%      -0.0        0.07 ± 14%  perf-profile.children.cycles-pp.__d_lookup
      0.11 ±  6%      -0.0        0.07 ± 17%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.12 ±  7%      -0.0        0.08 ± 13%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.10 ± 12%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.lockref_put_or_lock
      0.07 ±  6%      -0.0        0.04 ± 73%  perf-profile.children.cycles-pp.proc_pid_permission
      0.11 ±  9%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.09 ± 13%      -0.0        0.05 ± 46%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.09 ± 13%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.10 ± 10%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__calc_delta
      0.09 ±  8%      -0.0        0.05 ±  9%  perf-profile.children.cycles-pp.call_cpuidle
      0.08 ±  5%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.10 ± 13%      -0.0        0.07 ± 17%  perf-profile.children.cycles-pp.tid_fd_mode
      0.09 ± 10%      -0.0        0.07 ± 16%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.03 ±100%      +0.0        0.07 ± 16%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.kmem_cache_alloc_node_trace
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.map_kernel_range_noflush
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.09 ± 23%      +0.1        0.17 ± 28%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.rb_insert_color
      0.15 ± 26%      +0.1        0.29 ± 55%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.00            +0.2        0.16 ± 14%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge_page
      0.00            +0.2        0.17 ± 12%  perf-profile.children.cycles-pp.free_pcp_prepare
      0.00            +0.2        0.21 ± 15%  perf-profile.children.cycles-pp.free_unref_page
      0.00            +0.3        0.31 ±  8%  perf-profile.children.cycles-pp.__free_pages
      0.00            +0.4        0.39 ±  9%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.4        0.40 ± 11%  perf-profile.children.cycles-pp.unmap_kernel_range_noflush
      0.00            +0.5        0.49 ±  8%  perf-profile.children.cycles-pp.__memcg_kmem_charge_page
      0.00            +0.9        0.88 ±  8%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.00            +1.1        1.09 ±  6%  perf-profile.children.cycles-pp.insert_vmap_area
      0.00            +3.3        3.29 ±  7%  perf-profile.children.cycles-pp.__schedule
     14.61 ±  7%      +5.6       20.19 ±  8%  perf-profile.children.cycles-pp.ksys_read
     14.42 ±  7%      +5.7       20.09 ±  8%  perf-profile.children.cycles-pp.vfs_read
      0.00            +6.0        6.05 ±  7%  perf-profile.children.cycles-pp.find_vmap_area
      0.00            +6.9        6.93 ±  8%  perf-profile.children.cycles-pp.free_vmap_area_noflush
      0.00            +8.3        8.29 ±  8%  perf-profile.children.cycles-pp.alloc_vmap_area
      8.44 ±  7%      +8.4       16.79 ±  8%  perf-profile.children.cycles-pp.seq_read
      8.39 ±  7%      +8.4       16.75 ±  8%  perf-profile.children.cycles-pp.seq_read_iter
     40.84 ±  7%      +8.5       49.29 ±  8%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      8.43 ±  7%     +13.2       21.67 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      8.35 ±  7%     +13.3       21.62 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      7.91 ±  7%     +13.5       21.37 ±  7%  perf-profile.children.cycles-pp.task_work_run
      7.72 ±  7%     +13.5       21.26 ±  7%  perf-profile.children.cycles-pp.__fput
      0.00           +13.6       13.55 ±  8%  perf-profile.children.cycles-pp.remove_vm_area
      6.94 ±  7%     +13.9       20.82 ±  7%  perf-profile.children.cycles-pp.single_release
      0.00           +14.2       14.20 ±  8%  perf-profile.children.cycles-pp.__get_vm_area_node
      0.00           +15.6       15.56 ±  8%  perf-profile.children.cycles-pp.__vmalloc_node
      0.00           +15.6       15.56 ±  8%  perf-profile.children.cycles-pp.__vmalloc_node_range
      0.00           +20.5       20.48 ±  7%  perf-profile.children.cycles-pp.__vunmap
      0.00           +26.5       26.50 ±  8%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.71 ±  7%     +27.9       28.63 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
      2.98 ±  9%      -2.8        0.14 ± 14%  perf-profile.self.cycles-pp.page_counter_cancel
      3.22 ±  8%      -2.8        0.42 ±  9%  perf-profile.self.cycles-pp.page_counter_try_charge
      2.51 ±  7%      -2.4        0.10 ± 12%  perf-profile.self.cycles-pp.refill_obj_stock
      5.07 ±  9%      -2.3        2.72 ±  9%  perf-profile.self.cycles-pp.poll_idle
      1.88 ±  9%      -1.9        0.03 ± 70%  perf-profile.self.cycles-pp.propagate_protected_usage
      1.30 ±  7%      -1.2        0.09 ± 16%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      1.04 ±  8%      -0.9        0.16 ± 16%  perf-profile.self.cycles-pp.common_file_perm
      0.87 ±  6%      -0.8        0.11 ± 15%  perf-profile.self.cycles-pp.kfree
      0.96 ±  7%      -0.7        0.23 ± 10%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.86 ±  4%      -0.6        0.30 ± 11%  perf-profile.self.cycles-pp.kmem_cache_free
      0.72 ± 10%      -0.5        0.17 ± 12%  perf-profile.self.cycles-pp.apparmor_file_open
      0.50 ± 10%      -0.4        0.08 ± 16%  perf-profile.self.cycles-pp.aa_get_task_label
      0.52 ± 13%      -0.4        0.12 ± 26%  perf-profile.self.cycles-pp.lockref_put_return
      0.55 ±  8%      -0.3        0.23 ±  8%  perf-profile.self.cycles-pp.apparmor_file_alloc_security
      0.72 ±  8%      -0.3        0.40 ±  8%  perf-profile.self.cycles-pp.update_rq_clock
      0.57 ± 26%      -0.3        0.26 ± 15%  perf-profile.self.cycles-pp.lockref_get_not_dead
      0.64 ±  7%      -0.3        0.33 ±  8%  perf-profile.self.cycles-pp.set_next_entity
      0.86 ± 10%      -0.3        0.59 ± 21%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.35 ±  9%      -0.3        0.08 ± 19%  perf-profile.self.cycles-pp.kmem_cache_alloc_trace
      0.33 ± 15%      -0.3        0.07 ± 24%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.53 ±  7%      -0.3        0.28 ±  7%  perf-profile.self.cycles-pp.update_load_avg
      0.29 ± 26%      -0.3        0.04 ± 71%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.48 ±  8%      -0.2        0.24 ±  9%  perf-profile.self.cycles-pp.apparmor_file_free_security
      0.53 ±  8%      -0.2        0.31 ± 10%  perf-profile.self.cycles-pp.do_idle
      0.46 ±  8%      -0.2        0.26 ±  4%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.41 ±  5%      -0.2        0.20 ± 16%  perf-profile.self.cycles-pp.enqueue_entity
      0.47 ±  8%      -0.2        0.27 ±  7%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.44 ±  8%      -0.2        0.25 ±  9%  perf-profile.self.cycles-pp.__switch_to
      0.43 ±  9%      -0.2        0.25 ±  8%  perf-profile.self.cycles-pp.__switch_to_asm
      0.38 ±  7%      -0.2        0.20 ±  6%  perf-profile.self.cycles-pp.update_curr
      0.50 ±  8%      -0.2        0.31 ± 19%  perf-profile.self.cycles-pp.vfs_read
      0.32 ±  5%      -0.2        0.15 ±  8%  perf-profile.self.cycles-pp.pick_next_entity
      0.39 ±  7%      -0.2        0.24 ± 16%  perf-profile.self.cycles-pp.__fget_light
      0.30 ± 12%      -0.2        0.15 ± 14%  perf-profile.self.cycles-pp.update_cfs_group
      0.18 ± 57%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.dput
      0.32 ±  8%      -0.1        0.17 ± 12%  perf-profile.self.cycles-pp.switch_fpu_return
      0.36 ±  9%      -0.1        0.22 ± 17%  perf-profile.self.cycles-pp.__wake_up_common
      0.30 ±  6%      -0.1        0.16 ± 11%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.31 ±  7%      -0.1        0.17 ±  9%  perf-profile.self.cycles-pp.available_idle_cpu
      0.28 ±  9%      -0.1        0.15 ±  8%  perf-profile.self.cycles-pp.reweight_entity
      0.33 ± 10%      -0.1        0.19 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.46 ±  8%      -0.1        0.33 ± 14%  perf-profile.self.cycles-pp.___might_sleep
      0.19 ± 12%      -0.1        0.06 ± 51%  perf-profile.self.cycles-pp.cpuacct_charge
      0.34 ±  6%      -0.1        0.22 ± 11%  perf-profile.self.cycles-pp.native_sched_clock
      0.19 ± 10%      -0.1        0.08 ± 15%  perf-profile.self.cycles-pp.perf_tp_event
      0.32 ±  9%      -0.1        0.21 ± 21%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.27 ±  8%      -0.1        0.17 ±  9%  perf-profile.self.cycles-pp.try_to_wake_up
      0.27 ±  6%      -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.format_decode
      0.26 ± 10%      -0.1        0.16 ±  9%  perf-profile.self.cycles-pp.number
      0.21 ±  9%      -0.1        0.11 ± 12%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.24 ± 13%      -0.1        0.14 ± 11%  perf-profile.self.cycles-pp.vsnprintf
      0.25 ± 11%      -0.1        0.16 ±  9%  perf-profile.self.cycles-pp.__entry_text_start
      0.17 ±  4%      -0.1        0.08 ±  7%  perf-profile.self.cycles-pp.dequeue_entity
      0.13 ± 26%      -0.1        0.06 ± 46%  perf-profile.self.cycles-pp.__legitimize_mnt
      0.23 ± 12%      -0.1        0.15 ± 12%  perf-profile.self.cycles-pp.resched_curr
      0.18 ±  4%      -0.1        0.11 ± 14%  perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
      0.10 ± 17%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.19 ± 11%      -0.1        0.12 ± 12%  perf-profile.self.cycles-pp.memcpy_erms
      0.10 ± 14%      -0.1        0.03 ±101%  perf-profile.self.cycles-pp.check_preempt_curr
      0.31 ±  7%      -0.1        0.24 ±  9%  perf-profile.self.cycles-pp.eventfd_read
      0.14 ±  6%      -0.1        0.07 ± 15%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.16 ± 13%      -0.1        0.09 ± 19%  perf-profile.self.cycles-pp.finish_task_switch
      0.10 ± 14%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.15 ± 15%      -0.1        0.09 ±  8%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.27 ±  6%      -0.1        0.21 ±  9%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.12 ±  9%      -0.1        0.06 ± 19%  perf-profile.self.cycles-pp.task_work_add
      0.26 ±  5%      -0.1        0.20 ± 16%  perf-profile.self.cycles-pp.read_tsc
      0.09 ±  7%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.lookup_fast
      0.08 ± 11%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.strncpy_from_user
      0.09 ±  5%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.08 ± 14%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.19 ±  8%      -0.1        0.14 ± 10%  perf-profile.self.cycles-pp.place_entity
      0.10 ± 13%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.path_openat
      0.08 ±  7%      -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.pid_revalidate
      0.15 ±  8%      -0.0        0.11 ± 10%  perf-profile.self.cycles-pp.new_sync_read
      0.10 ±  6%      -0.0        0.05 ± 46%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.11 ±  4%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.seq_show
      0.09 ±  5%      -0.0        0.05 ± 47%  perf-profile.self.cycles-pp.update_min_vruntime
      0.10 ±  8%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.link_path_walk
      0.09 ± 14%      -0.0        0.05 ± 45%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.13 ±  8%      -0.0        0.09 ± 17%  perf-profile.self.cycles-pp.fsnotify
      0.12 ±  8%      -0.0        0.08 ± 11%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.08 ±  8%      -0.0        0.05 ± 45%  perf-profile.self.cycles-pp.call_cpuidle
      0.10 ±  8%      -0.0        0.06 ± 19%  perf-profile.self.cycles-pp.___perf_sw_event
      0.09 ± 12%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.10 ± 12%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.__calc_delta
      0.08 ±  5%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.09 ±  5%      -0.0        0.06 ± 19%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.map_kernel_range_noflush
      0.07 ± 27%      +0.1        0.15 ± 28%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.11 ± 12%  perf-profile.self.cycles-pp.__get_vm_area_node
      0.00            +0.1        0.13 ±  9%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.3        0.31 ±  8%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.00            +0.3        0.31 ±  8%  perf-profile.self.cycles-pp.__free_pages
      0.00            +0.3        0.31 ±  9%  perf-profile.self.cycles-pp.__vmalloc_node_range
      0.00            +0.3        0.34 ± 11%  perf-profile.self.cycles-pp.__vunmap
      0.00            +0.4        0.38 ± 10%  perf-profile.self.cycles-pp.unmap_kernel_range_noflush
      0.00            +0.4        0.39 ±  7%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.6        0.61 ±  8%  perf-profile.self.cycles-pp.remove_vm_area
      0.00            +0.7        0.68 ± 11%  perf-profile.self.cycles-pp.__schedule
      0.00            +0.7        0.72 ±  6%  perf-profile.self.cycles-pp.find_vmap_area
      0.00            +1.0        0.96 ±  6%  perf-profile.self.cycles-pp.insert_vmap_area
      0.00            +1.0        1.02 ± 13%  perf-profile.self.cycles-pp.alloc_vmap_area
      0.00            +1.0        1.02 ±  8%  perf-profile.self.cycles-pp.free_vmap_area_noflush
      0.69 ±  7%      +1.4        2.10 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock
      0.00           +26.3       26.26 ±  8%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                            stress-ng.time.user_time                            
                                                                                
  45 +----------------------------------------------------------------------+   
     |                                                                      |   
  40 |-+                     +.                                             |   
     |              .+.+.+. +  +.+.+.+                                      |   
     |          .+.+       +          :                                     |   
  35 |.+.+.+.+.+                      :                                     |   
     |                                 +..+.+.+.+.+.+.+.+.+.           .+.+.|   
  30 |-+                                                    +.+.+.+.+.+     |   
     |                                                                      |   
  25 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
  20 |-+       O       O O         O          O   O                         |   
     |   O O     O O O     O O O O   O O  O O   O   O   O     O O   O     O |   
  15 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       stress-ng.time.voluntary_context_switches                
                                                                                
  7.5e+07 +-----------------------------------------------------------------+   
          |  .+.+.  .+. .+. .+. .+.                                         |   
    7e+07 |.+     ++   +   +   +   ++.+.+.                                  |   
  6.5e+07 |-+                             +.+.+. .++.+.+. .+.+. .+.++.+.+.+.|   
          |                                     +        +     +            |   
    6e+07 |-+                                                               |   
  5.5e+07 |-+                                                               |   
          |                                                                 |   
    5e+07 |-+                                                               |   
  4.5e+07 |-+                                                               |   
          |                                                                 |   
    4e+07 |-+                                                               |   
  3.5e+07 |-+      O         O                                              |   
          | O O O O  O O O O   O O OO O O O O O O OO O O O O O O O OO O O O |   
    3e+07 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 stress-ng.eventfd.ops                          
                                                                                
  3.6e+07 +-----------------------------------------------------------------+   
  3.4e+07 |.+     ++   +.+.+.+.+.+.++.+.+.                                  |   
          |                               +. .+. .++. .+. .+.+.   .+ .+.+.+.|   
  3.2e+07 |-+                               +   +    +   +     +.+  +       |   
    3e+07 |-+                                                               |   
          |                                                                 |   
  2.8e+07 |-+                                                               |   
  2.6e+07 |-+                                                               |   
  2.4e+07 |-+                                                               |   
          |                                                                 |   
  2.2e+07 |-+                                                               |   
    2e+07 |-+                                                               |   
          |                  O                                              |   
  1.8e+07 |-+   O OO O O O          O         O    O   O       O   O      O |   
  1.6e+07 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.eventfd.ops_per_sec                       
                                                                                
  600000 +------------------------------------------------------------------+   
         |.+     +.+  +.+.+.+.+.+.+.+.+.+.                                  |   
  550000 |-+                              ++.+. .+.+.+.+. .+.+.   +. .+.+.+.|   
         |                                     +         +     +.+  +       |   
  500000 |-+                                                                |   
         |                                                                  |   
  450000 |-+                                                                |   
         |                                                                  |   
  400000 |-+                                                                |   
         |                                                                  |   
  350000 |-+                                                                |   
         |                                                                  |   
  300000 |-+       O        O                                               |   
         | O O O O  O O O O   O O O O O O OO O O O O O O O O O O OO O O O O |   
  250000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.11.0-00004-g5fd6060e506c" of type "text/plain" (172445 bytes)

View attachment "job-script" of type "text/plain" (8226 bytes)

View attachment "job.yaml" of type "text/plain" (5617 bytes)

View attachment "reproduce" of type "text/plain" (535 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ