lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <202307192135.203ac24e-oliver.sang@intel.com>
Date:   Wed, 19 Jul 2023 22:07:52 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Thomas Gleixner <tglx@...utronix.de>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        <linux-kernel@...r.kernel.org>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <fengwei.yin@...el.com>,
        <oliver.sang@...el.com>
Subject: [linus:master] [x86/fpu]  b81fac906a:  stress-ng.af-alg.ops_per_sec
 -34.6% regression



Hello,

kernel test robot noticed a -34.6% regression of stress-ng.af-alg.ops_per_sec on:


commit: b81fac906a8f9e682e513ddd95697ec7a20878d4 ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

still regression on:
fix commit: fe3e0a13e597c1c8617814bf9b42ab732db5c26e
linus/master: ccff6d117d8dc8d8d86e8695a75e5f8b01e573bf

testcase: stress-ng
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:

	nr_threads: 10%
	disk: 1HDD
	testtime: 60s
	fs: ext4
	class: os
	test: af-alg
	cpufreq_governor: performance


besides, we noticed there are below information in dmesg (attached kmsg.xz),
which we didn't see in dmesg for parent:

kern  :info  : [   65.660392] AVX or AES-NI instructions are not detected.
user  :notice: [   65.661743] stress-ng: info:  [4738] af-alg: 9 cryptographic algorithms are internal and may be unused

kern  :info  : [   65.720205] AVX or AES-NI instructions are not detected.
kern  :info  : [   65.820334] AVX2 instructions are not detected.
kern  :info  : [   66.130644] AVX or AES-NI instructions are not detected.
kern  :info  : [   66.167749] AVX or AES-NI instructions are not detected.
kern  :info  : [   66.213178] AVX2 instructions are not detected.
kern  :info  : [   66.318235] AVX or AES-NI instructions are not detected.
kern  :info  : [   66.360766] AVX or AES-NI instructions are not detected.
kern  :info  : [   66.437318] AVX2 instructions are not detected.
kern  :info  : [   66.683666] AVX or AES-NI instructions are not detected.
user  :notice: [   66.713976] BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled

kern  :info  : [   66.738790] AVX or AES-NI instructions are not detected.
kern  :info  : [   66.822332] AVX2 instructions are not detected.
kern  :info  : [   67.083360] AVX or AES-NI instructions are not detected.
kern  :info  : [   67.122804] AVX or AES-NI instructions are not detected.



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@...el.com>
| Closes: https://lore.kernel.org/oe-lkp/202307192135.203ac24e-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  os/gcc-12/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp7/af-alg/stress-ng/60s

commit: 
  1703db2b90 ("x86/fpu: Mark init functions __init")
  b81fac906a ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()")

1703db2b90c91b2e b81fac906a8f9e682e513ddd956 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   4453936           +17.2%    5218922        cpuidle..usage
    475.17 ±  9%     -39.4%     288.00 ±  4%  perf-c2c.HITM.local
     68421 ±  6%     -68.2%      21789 ±115%  numa-meminfo.node1.AnonHugePages
    397537 ± 11%     -30.9%     274577 ± 25%  numa-meminfo.node1.AnonPages.max
  11275439 ±  8%     -40.6%    6702239 ± 12%  numa-numastat.node0.local_node
  11307159 ±  8%     -40.5%    6726387 ± 12%  numa-numastat.node0.numa_hit
  11307295 ±  8%     -40.5%    6726432 ± 12%  numa-vmstat.node0.numa_hit
  11275576 ±  8%     -40.6%    6702284 ± 12%  numa-vmstat.node0.numa_local
     90.28            +2.6%      92.63        iostat.cpu.idle
      6.74           -23.4%       5.16        iostat.cpu.system
      2.95           -26.4%       2.17 ±  2%  iostat.cpu.user
      0.12            -0.0        0.10        mpstat.cpu.all.soft%
      5.78            -1.7        4.12        mpstat.cpu.all.sys%
      3.02            -0.8        2.22 ±  2%  mpstat.cpu.all.usr%
      5.67 ±  8%     -29.4%       4.00        vmstat.procs.r
     27980           -29.5%      19716        vmstat.system.cs
     69859           +18.8%      83015        vmstat.system.in
    137449           -21.7%     107685        meminfo.Active
    135863           -21.9%     106091        meminfo.Active(anon)
    538488 ±  5%     +52.9%     823250 ±  4%  meminfo.DirectMap4k
     78913 ±  2%     -14.6%      67370        meminfo.Mapped
    181011           -22.4%     140469        meminfo.Shmem
     90999           -34.6%      59547        stress-ng.af-alg.ops
      1516           -34.6%     992.18        stress-ng.af-alg.ops_per_sec
     87.83           -32.8%      59.00        stress-ng.time.percent_of_cpu_this_job_got
     52.14           -32.7%      35.10        stress-ng.time.system_time
    175731           -34.6%     114964        stress-ng.time.voluntary_context_switches
    317.50           -22.4%     246.33        turbostat.Avg_MHz
     10.35            -2.1        8.23        turbostat.Busy%
      3073            -2.4%       2999        turbostat.Bzy_MHz
     14183           +20.3%      17060 ±  4%  turbostat.C1
    783741 ±  2%    +124.3%    1758028        turbostat.C1E
      7.98 ±  2%      +8.2       16.21        turbostat.C1E%
   4538585           +19.2%    5411831        turbostat.IRQ
     30452 ±  2%     -62.3%      11490 ±  9%  turbostat.POLL
    157.35            -3.5%     151.83        turbostat.PkgWatt
     33967           -21.9%      26523        proc-vmstat.nr_active_anon
     89639            -1.3%      88437        proc-vmstat.nr_anon_pages
    748728            -1.3%     738629        proc-vmstat.nr_file_pages
    101000            -4.0%      96980        proc-vmstat.nr_inactive_anon
     19752 ±  2%     -14.7%      16856        proc-vmstat.nr_mapped
     45267           -22.4%      35122        proc-vmstat.nr_shmem
     37642            -1.1%      37241        proc-vmstat.nr_slab_unreclaimable
     33967           -21.9%      26523        proc-vmstat.nr_zone_active_anon
    101000            -4.0%      96980        proc-vmstat.nr_zone_inactive_anon
  21330465           -30.8%   14764467        proc-vmstat.numa_hit
  21264274           -30.9%   14698273        proc-vmstat.numa_local
     74767 ±  8%     -26.1%      55256 ± 23%  proc-vmstat.numa_pte_updates
     77297           -21.4%      60741        proc-vmstat.pgactivate
  21716615           -30.7%   15050540        proc-vmstat.pgalloc_normal
  24651421           -31.0%   17003260        proc-vmstat.pgfault
  21608074           -30.8%   14960607        proc-vmstat.pgfree
    576512          +309.8%    2362624        proc-vmstat.unevictable_pgs_scanned
 2.657e+09           -28.1%   1.91e+09        perf-stat.i.branch-instructions
  55105518           -27.9%   39728424        perf-stat.i.branch-misses
      5.80 ±  2%      +1.3        7.13 ±  2%  perf-stat.i.cache-miss-rate%
   6856942 ±  3%      -9.4%    6209194 ±  2%  perf-stat.i.cache-misses
 1.314e+08           -28.1%   94523874        perf-stat.i.cache-references
     28932           -29.8%      20310        perf-stat.i.context-switches
      1.16            +7.1%       1.24        perf-stat.i.cpi
 1.969e+10           -22.9%  1.519e+10        perf-stat.i.cpu-cycles
    649.36 ±  2%     -37.1%     408.14 ±  2%  perf-stat.i.cpu-migrations
      3018 ±  2%     -15.6%       2548        perf-stat.i.cycles-between-cache-misses
      0.04 ±  2%      +0.0        0.06 ±  3%  perf-stat.i.dTLB-load-miss-rate%
   1941644            +8.1%    2099200 ±  2%  perf-stat.i.dTLB-load-misses
  4.69e+09           -29.1%  3.326e+09        perf-stat.i.dTLB-loads
      0.12            +0.0        0.13        perf-stat.i.dTLB-store-miss-rate%
   2651571           -26.7%    1942815        perf-stat.i.dTLB-store-misses
 2.113e+09           -29.5%   1.49e+09        perf-stat.i.dTLB-stores
 1.698e+10           -27.7%  1.227e+10        perf-stat.i.instructions
      0.87            -6.4%       0.81        perf-stat.i.ipc
      2035 ±  3%     -61.7%     778.70 ±  9%  perf-stat.i.major-faults
      0.31           -22.9%       0.24        perf-stat.i.metric.GHz
    167.15 ±  2%     -13.2%     145.14 ±  3%  perf-stat.i.metric.K/sec
    149.81           -28.9%     106.53        perf-stat.i.metric.M/sec
    388617           -31.0%     268024        perf-stat.i.minor-faults
   3246190 ±  7%     -23.5%    2483951 ±  4%  perf-stat.i.node-load-misses
     32.23 ±  2%      +6.0       38.26 ±  2%  perf-stat.i.node-store-miss-rate%
    886521 ±  3%     +24.8%    1106439 ±  3%  perf-stat.i.node-store-misses
   1968660 ±  2%      -8.4%    1802894        perf-stat.i.node-stores
    390652           -31.2%     268802        perf-stat.i.page-faults
      5.24 ±  2%      +1.3        6.59        perf-stat.overall.cache-miss-rate%
      1.16            +6.7%       1.24        perf-stat.overall.cpi
      2863 ±  2%     -14.8%       2440        perf-stat.overall.cycles-between-cache-misses
      0.04 ±  2%      +0.0        0.06 ±  3%  perf-stat.overall.dTLB-load-miss-rate%
      0.13            +0.0        0.13        perf-stat.overall.dTLB-store-miss-rate%
      0.86            -6.3%       0.81        perf-stat.overall.ipc
     30.85 ±  2%      +7.0       37.89 ±  2%  perf-stat.overall.node-store-miss-rate%
 2.617e+09           -28.1%  1.881e+09        perf-stat.ps.branch-instructions
  54225753           -27.9%   39076523        perf-stat.ps.branch-misses
   6779427 ±  3%      -9.6%    6127594 ±  2%  perf-stat.ps.cache-misses
 1.294e+08           -28.1%   93009393        perf-stat.ps.cache-references
     28483           -29.8%      19996        perf-stat.ps.context-switches
  1.94e+10           -22.9%  1.495e+10        perf-stat.ps.cpu-cycles
    638.69 ±  2%     -37.1%     401.75 ±  2%  perf-stat.ps.cpu-migrations
   1913060            +8.0%    2066590 ±  2%  perf-stat.ps.dTLB-load-misses
  4.62e+09           -29.1%  3.275e+09        perf-stat.ps.dTLB-loads
   2610836           -26.8%    1912030        perf-stat.ps.dTLB-store-misses
 2.081e+09           -29.5%  1.468e+09        perf-stat.ps.dTLB-stores
 1.673e+10           -27.8%  1.208e+10        perf-stat.ps.instructions
      2005 ±  3%     -61.8%     766.73 ±  9%  perf-stat.ps.major-faults
    382594           -31.1%     263761        perf-stat.ps.minor-faults
   3194487 ±  7%     -23.5%    2443082 ±  4%  perf-stat.ps.node-load-misses
    872822 ±  3%     +24.7%    1088214 ±  2%  perf-stat.ps.node-store-misses
   1956457 ±  2%      -8.8%    1783576        perf-stat.ps.node-stores
    384599           -31.2%     264527        perf-stat.ps.page-faults
 1.046e+12           -27.3%  7.603e+11        perf-stat.total.instructions
      0.00 ± 20%     -72.7%       0.00 ±141%  perf-sched.sch_delay.avg.ms.__cond_resched.async_synchronize_cookie_domain.call_usermodehelper_exec_async.ret_from_fork
      0.00 ± 19%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.dput.step_into.path_lookupat.filename_lookup
      0.00 ± 17%     +84.6%       0.00 ± 20%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.01 ±  7%    +375.3%       0.07 ± 11%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 20%     +52.0%       0.01 ±  5%  perf-sched.sch_delay.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read
      0.01 ±  7%     +29.3%       0.01 ±  3%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.00 ± 20%     -72.7%       0.00 ±141%  perf-sched.sch_delay.max.ms.__cond_resched.async_synchronize_cookie_domain.call_usermodehelper_exec_async.ret_from_fork
      0.00 ± 11%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.dput.step_into.path_lookupat.filename_lookup
      0.02 ± 29%     -78.2%       0.00 ± 89%  perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.prepare_kernel_cred.call_usermodehelper_exec_async.ret_from_fork
      0.01 ± 38%     -55.6%       0.00 ± 14%  perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock_killable.pcpu_alloc.__percpu_counter_init.mm_init
      0.02 ± 82%     -64.2%       0.01 ± 60%  perf-sched.sch_delay.max.ms.__cond_resched.unmap_vmas.unmap_region.do_vmi_align_munmap.do_vmi_munmap
      0.02 ± 27%     +67.6%       0.04 ± 19%  perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.03 ± 44%     -58.0%       0.01 ±  8%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.00 ±223%    +900.0%       0.02 ± 29%  perf-sched.sch_delay.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
      0.02 ± 23%     +72.0%       0.03 ± 20%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 31%     -83.9%       0.00 ± 99%  perf-sched.sch_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma
      4.04 ± 46%    +382.2%      19.47 ±  2%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 35%    +223.4%       0.03 ± 13%  perf-sched.sch_delay.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
      4.20 ± 17%     +60.4%       6.74 ± 18%  perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.01 ±  6%    +108.1%       0.01 ±  6%  perf-sched.total_sch_delay.average.ms
      5.58 ± 10%    +249.1%      19.47 ±  2%  perf-sched.total_sch_delay.max.ms
     13.94           +47.6%      20.58        perf-sched.total_wait_and_delay.average.ms
     79638           -29.4%      56254        perf-sched.total_wait_and_delay.count.ms
     13.94           +47.6%      20.57        perf-sched.total_wait_time.average.ms
      0.00          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      2.08 ±  2%     +51.4%       3.15 ±  2%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    384.28           -30.6%     266.53 ±  2%  perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
    298.54 ± 13%     +63.3%     487.41 ± 18%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    368.03 ±  5%     -80.2%      72.93        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      1.48 ±  8%     -19.7%       1.19 ±  6%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
      1.82           +61.9%       2.95 ±  2%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
     78.21 ±  3%     +32.4%     103.56        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
     14.85 ±  2%     +37.7%      20.44        perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
    726.00 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.count.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
     12628 ±  2%     -29.5%       8908 ±  2%  perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     12520 ±  2%     -29.7%       8797 ±  2%  perf-sched.wait_and_delay.count.do_wait.kernel_wait.call_usermodehelper_exec_work.process_one_work
    618.00 ±  3%     -41.8%     359.83 ±  8%  perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      5408 ±  4%     -62.3%       2039 ±  7%  perf-sched.wait_and_delay.count.io_schedule.folio_wait_bit_common.filemap_fault.__do_fault
    167.33           +55.0%     259.33 ±  2%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
     52.67 ±  8%     -39.9%      31.67 ±  8%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
     13.50 ±  8%    +860.5%     129.67        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      1478 ± 23%     -69.8%     446.50 ± 18%  perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
     12518 ±  2%     -29.7%       8800 ±  2%  perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
      7535           -23.4%       5772        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     22238           -25.8%      16493        perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
      0.06 ±112%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      3.92 ± 29%     -40.6%       2.33 ±  4%  perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
      5.35 ± 11%   +3637.0%     200.02 ± 11%  perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
      5.03           +68.3%       8.46 ± 14%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      1.88 ± 12%   +2136.1%      42.02        perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      2.08 ±  2%     +51.4%       3.14 ±  2%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 39%     -70.6%       0.00 ± 56%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.56 ± 17%     +48.9%       0.84 ±  9%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.10 ±223%  +51536.6%      51.98 ±  5%  perf-sched.wait_time.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
    384.27           -30.6%     266.52 ±  2%  perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
    298.54 ± 13%     +63.3%     487.40 ± 18%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    368.02 ±  5%     -80.2%      72.92        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 37%     -79.4%       0.00 ±159%  perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma
      1.48 ±  8%     -19.8%       1.18 ±  6%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
      1.82           +62.1%       2.94 ±  2%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
      4.08           +18.6%       4.84 ± 13%  perf-sched.wait_time.avg.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
     78.20 ±  3%     +32.3%     103.49        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      1.89 ± 12%   +2117.5%      42.02        perf-sched.wait_time.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read
     14.84 ±  2%     +37.7%      20.43        perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
      0.02 ± 29%     -68.5%       0.01 ± 31%  perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.perf_event_mmap_event.perf_event_mmap
      0.05 ±146%     -87.6%       0.01 ± 21%  perf-sched.wait_time.max.ms.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.[unknown]
      0.01 ± 56%     -69.8%       0.00 ± 70%  perf-sched.wait_time.max.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open
      0.01 ± 57%     -84.4%       0.00 ±141%  perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma.do_vmi_align_munmap
      3.76 ± 12%   +2857.8%     111.15 ± 11%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.28 ±223%  +41327.3%     116.13 ± 12%  perf-sched.wait_time.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
      0.02 ± 38%     -92.1%       0.00 ±141%  perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.mmap_region
      0.03 ± 57%     -94.4%       0.00 ±142%  perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma
      3.92 ± 29%     -40.7%       2.32 ±  4%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
      5.12 ± 10%   +3808.0%     200.02 ± 11%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
      4.09          +126.9%       9.29 ± 65%  perf-sched.wait_time.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
      5.02           +68.5%       8.45 ± 14%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      3.79 ± 12%   +2833.2%     111.15 ± 11%  perf-sched.wait_time.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
     24.78 ±  2%      -6.4       18.42 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     24.75 ±  2%      -6.4       18.39 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.72 ±  3%      -3.2        7.54 ±  4%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.46 ±  3%      -3.2        7.31 ±  4%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      9.77 ±  3%      -3.0        6.75 ±  5%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.81 ±  3%      -3.0        6.79 ±  4%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.15            -2.5        7.63 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.15            -2.5        7.63 ±  3%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.15            -2.5        7.63 ±  3%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     17.43 ±  2%      -2.4       14.98        perf-profile.calltrace.cycles-pp.asm_exc_page_fault
     15.02 ±  3%      -2.4       12.59        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      9.32            -2.4        6.94 ±  4%  perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      9.27            -2.4        6.89 ±  4%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
      9.31            -2.4        6.93 ±  4%  perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
     16.35 ±  2%      -2.4       13.98        perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
     16.27 ±  2%      -2.3       13.93        perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     12.31 ±  4%      -2.2       10.08        perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
     15.14 ±  2%      -2.2       12.93        perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      9.96 ±  8%      -2.1        7.90 ±  3%  perf-profile.calltrace.cycles-pp.jent_kcapi_random._rng_recvmsg.sock_recvmsg.sock_read_iter.vfs_read
      9.96 ±  8%      -2.1        7.90 ±  3%  perf-profile.calltrace.cycles-pp.jent_read_entropy.jent_kcapi_random._rng_recvmsg.sock_recvmsg.sock_read_iter
      9.96 ±  8%      -2.1        7.90 ±  3%  perf-profile.calltrace.cycles-pp.jent_gen_entropy.jent_read_entropy.jent_kcapi_random._rng_recvmsg.sock_recvmsg
      9.95 ±  8%      -2.1        7.89 ±  3%  perf-profile.calltrace.cycles-pp.jent_measure_jitter.jent_gen_entropy.jent_read_entropy.jent_kcapi_random._rng_recvmsg
      9.96 ±  8%      -2.1        7.90 ±  3%  perf-profile.calltrace.cycles-pp._rng_recvmsg.sock_recvmsg.sock_read_iter.vfs_read.ksys_read
      6.08 ±  4%      -1.9        4.14 ±  5%  perf-profile.calltrace.cycles-pp.do_vmi_munmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      6.01 ±  4%      -1.9        4.08 ±  5%  perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap.vm_mmap_pgoff
      9.37 ±  5%      -1.8        7.52 ±  3%  perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      8.37 ±  8%      -1.7        6.63 ±  3%  perf-profile.calltrace.cycles-pp.jent_lfsr_time.jent_measure_jitter.jent_gen_entropy.jent_read_entropy.jent_kcapi_random
      9.11 ±  5%      -1.7        7.38 ±  3%  perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
      4.25 ±  3%      -1.3        2.94 ±  5%  perf-profile.calltrace.cycles-pp.__split_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap
      7.37 ±  3%      -1.3        6.12 ±  3%  perf-profile.calltrace.cycles-pp.next_uptodate_page.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
      6.49            -1.2        5.32 ±  3%  perf-profile.calltrace.cycles-pp.call_usermodehelper_exec_async.ret_from_fork
      6.36            -1.2        5.20 ±  3%  perf-profile.calltrace.cycles-pp.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
      2.67 ±  5%      -1.0        1.68 ± 10%  perf-profile.calltrace.cycles-pp.vma_prepare.__split_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region
      1.25 ± 12%      -1.0        0.30 ±101%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.vma_prepare.__split_vma.do_vmi_align_munmap
      1.35 ± 11%      -0.9        0.43 ± 72%  perf-profile.calltrace.cycles-pp.down_write.vma_prepare.__split_vma.do_vmi_align_munmap.do_vmi_munmap
      4.68 ±  3%      -0.8        3.83 ±  3%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
      2.26 ±  3%      -0.8        1.44 ±  7%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.__mmput.exit_mm.do_exit
      4.44 ±  3%      -0.8        3.63 ±  3%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      4.79 ±  2%      -0.8        3.98 ±  3%  perf-profile.calltrace.cycles-pp.bprm_execve.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
      4.40 ±  3%      -0.8        3.60 ±  3%  perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
      4.32 ±  3%      -0.8        3.54 ±  3%  perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
      4.22 ±  2%      -0.7        3.48 ±  3%  perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.kernel_execve.call_usermodehelper_exec_async
      4.24 ±  2%      -0.7        3.50 ±  3%  perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
      4.14 ±  2%      -0.7        3.42 ±  3%  perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.kernel_execve
      1.93 ±  3%      -0.6        1.28 ±  6%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.__mmput.exit_mm.do_exit
      1.89 ±  3%      -0.6        1.25 ±  6%  perf-profile.calltrace.cycles-pp.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput.exit_mm
      1.41 ±  5%      -0.6        0.78 ± 10%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.__mmput.exit_mm
      1.32 ±  9%      -0.6        0.74 ± 11%  perf-profile.calltrace.cycles-pp.unmap_region.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap
      1.50 ±  4%      -0.5        0.98 ±  7%  perf-profile.calltrace.cycles-pp.release_pages.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput
      1.57 ± 18%      -0.5        1.10 ±  3%  perf-profile.calltrace.cycles-pp.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.55 ± 18%      -0.5        1.08 ±  3%  perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      1.78 ±  2%      -0.4        1.34 ±  3%  perf-profile.calltrace.cycles-pp.call_usermodehelper_exec_work.process_one_work.worker_thread.kthread.ret_from_fork
      2.92            -0.4        2.53 ±  2%  perf-profile.calltrace.cycles-pp.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.64 ±  4%      -0.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.vma_prepare.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect
      0.72 ±  5%      -0.4        0.36 ± 70%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.elf_map.load_elf_interp
      0.64 ±  7%      -0.4        0.27 ±100%  perf-profile.calltrace.cycles-pp.sendmsg
      2.02 ±  2%      -0.3        1.68 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.02 ±  3%      -0.3        1.68 ±  3%  perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.41 ±  3%      -0.3        1.08 ±  3%  perf-profile.calltrace.cycles-pp.kernel_clone.user_mode_thread.call_usermodehelper_exec_work.process_one_work.worker_thread
      1.41 ±  3%      -0.3        1.08 ±  3%  perf-profile.calltrace.cycles-pp.user_mode_thread.call_usermodehelper_exec_work.process_one_work.worker_thread.kthread
      0.59 ±  4%      -0.3        0.27 ±100%  perf-profile.calltrace.cycles-pp.getenv
      1.49            -0.3        1.17 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
      1.50            -0.3        1.18 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
      1.51            -0.3        1.19 ±  3%  perf-profile.calltrace.cycles-pp.__munmap
      0.68 ±  5%      -0.3        0.37 ± 71%  perf-profile.calltrace.cycles-pp.vma_interval_tree_insert.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      1.54 ±  7%      -0.3        1.23 ±  5%  perf-profile.calltrace.cycles-pp.jent_memaccess.jent_measure_jitter.jent_gen_entropy.jent_read_entropy.jent_kcapi_random
      1.47 ±  2%      -0.3        1.16 ± 16%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
      1.49 ±  2%      -0.3        1.18 ± 16%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.75 ±  3%      -0.3        1.45 ±  4%  perf-profile.calltrace.cycles-pp.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.41 ±  2%      -0.3        1.11 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
      1.40 ±  2%      -0.3        1.11 ±  3%  perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
      1.16 ±  4%      -0.3        0.87 ±  5%  perf-profile.calltrace.cycles-pp.load_elf_interp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
      0.76 ±  5%      -0.3        0.47 ± 44%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.elf_map.load_elf_interp.load_elf_binary.search_binary_handler
      1.11 ±  5%      -0.3        0.82 ±  9%  perf-profile.calltrace.cycles-pp.alloc_bprm.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
      0.74 ±  6%      -0.3        0.46 ± 44%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.elf_map.load_elf_interp.load_elf_binary
      1.07 ±  4%      -0.3        0.78 ±  5%  perf-profile.calltrace.cycles-pp.elf_map.load_elf_interp.load_elf_binary.search_binary_handler.exec_binprm
      1.03 ±  5%      -0.3        0.76 ± 10%  perf-profile.calltrace.cycles-pp.mm_init.alloc_bprm.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
      1.39 ±  4%      -0.3        1.13 ±  6%  perf-profile.calltrace.cycles-pp.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64
      1.12 ±  3%      -0.3        0.86 ±  4%  perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.user_mode_thread.call_usermodehelper_exec_work.process_one_work
      2.00 ±  2%      -0.2        1.77 ±  2%  perf-profile.calltrace.cycles-pp._dl_addr
      0.93 ±  4%      -0.2        0.71 ±  4%  perf-profile.calltrace.cycles-pp.unmap_region.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
      0.68 ±  6%      -0.2        0.47 ± 45%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
      1.48 ±  5%      -0.2        1.26 ±  6%  perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      0.75 ±  3%      -0.2        0.55 ±  3%  perf-profile.calltrace.cycles-pp.dup_task_struct.copy_process.kernel_clone.user_mode_thread.call_usermodehelper_exec_work
      1.09 ±  2%      -0.2        0.90 ±  4%  perf-profile.calltrace.cycles-pp.__mmap
      1.07 ±  2%      -0.2        0.88 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
      1.06 ±  2%      -0.2        0.88 ±  4%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      1.07 ±  2%      -0.2        0.88 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      1.04 ±  3%      -0.2        0.86 ±  4%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      0.86 ±  6%      -0.2        0.68 ±  9%  perf-profile.calltrace.cycles-pp.elf_map.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
      1.08 ±  2%      -0.2        0.90 ±  6%  perf-profile.calltrace.cycles-pp.copy_page.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault
      0.92 ±  2%      -0.2        0.75 ±  4%  perf-profile.calltrace.cycles-pp.vma_complete.__split_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region
      0.65 ±  5%      -0.2        0.49 ± 45%  perf-profile.calltrace.cycles-pp.filemap_fault.__do_fault.do_cow_fault.do_fault.__handle_mm_fault
      0.72 ±  5%      -0.1        0.58 ±  8%  perf-profile.calltrace.cycles-pp.do_set_pte.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
      0.76 ±  4%      -0.1        0.62 ±  6%  perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
      0.94 ±  3%      -0.1        0.80 ±  3%  perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      0.95 ±  3%      -0.1        0.81 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      1.00 ±  5%      -0.1        0.86 ±  8%  perf-profile.calltrace.cycles-pp.vma_interval_tree_insert.vma_prepare.__split_vma.do_vmi_align_munmap.do_vmi_munmap
      0.80 ±  3%      -0.1        0.67 ±  5%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.96 ±  3%      -0.1        0.82 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64
      0.78 ±  4%      -0.1        0.65 ±  4%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.98 ±  3%      -0.1        0.85 ±  3%  perf-profile.calltrace.cycles-pp.open64
      0.74 ±  4%      -0.1        0.61 ±  5%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.95 ±  3%      -0.1        0.82 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      0.81 ±  4%      -0.1        0.69 ±  4%  perf-profile.calltrace.cycles-pp.__sys_bind.__x64_sys_bind.do_syscall_64.entry_SYSCALL_64_after_hwframe.bind
      0.81 ±  4%      -0.1        0.69 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_bind.do_syscall_64.entry_SYSCALL_64_after_hwframe.bind
      0.82 ±  4%      -0.1        0.70 ±  4%  perf-profile.calltrace.cycles-pp.bind
      0.74 ±  5%      -0.1        0.62 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.81 ±  5%      -0.1        0.70 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.bind
      0.81 ±  4%      -0.1        0.70 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.bind
      0.80 ±  5%      -0.1        0.68 ±  5%  perf-profile.calltrace.cycles-pp.alg_bind.__sys_bind.__x64_sys_bind.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.73 ±  4%      -0.1        0.62 ±  8%  perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.41 ±  3%      -0.1        1.31 ±  3%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.67 ±  5%      -0.1        0.58 ±  7%  perf-profile.calltrace.cycles-pp.__do_fault.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault
      0.67 ±  3%      -0.1        0.60 ±  6%  perf-profile.calltrace.cycles-pp.mas_store_prealloc.vma_complete.__split_vma.do_vmi_align_munmap.do_vmi_munmap
      0.71 ±  5%      -0.1        0.66 ±  3%  perf-profile.calltrace.cycles-pp.finish_fault.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault
      0.69 ± 10%      +0.1        0.84 ±  3%  perf-profile.calltrace.cycles-pp.__intel_pmu_enable_all.perf_adjust_freq_unthr_context.perf_event_task_tick.scheduler_tick.update_process_times
      1.28 ±  7%      +0.2        1.50 ±  3%  perf-profile.calltrace.cycles-pp.perf_adjust_freq_unthr_context.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle
      1.32 ±  7%      +0.2        1.54 ±  3%  perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      2.00 ±  5%      +0.3        2.28 ±  3%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      2.38 ±  5%      +0.3        2.71 ±  4%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      2.41 ±  5%      +0.3        2.74 ±  4%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      2.73 ±  5%      +0.4        3.13 ±  3%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      3.18 ±  6%      +0.5        3.67 ±  2%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.6        0.61 ±  6%  perf-profile.calltrace.cycles-pp.verify_pkcs7_message_sig.verify_pkcs7_signature.mod_verify_sig.module_sig_check.load_module
      0.00            +0.6        0.62 ±  6%  perf-profile.calltrace.cycles-pp.module_sig_check.load_module.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.62 ±  6%  perf-profile.calltrace.cycles-pp.mod_verify_sig.module_sig_check.load_module.__do_sys_finit_module.do_syscall_64
      0.00            +0.6        0.62 ±  6%  perf-profile.calltrace.cycles-pp.verify_pkcs7_signature.mod_verify_sig.module_sig_check.load_module.__do_sys_finit_module
      0.00            +0.6        0.65 ±  3%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      3.58 ±  5%      +0.7        4.23 ±  2%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.60 ±  5%      +0.7        4.25 ±  2%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      5.61 ±  5%      +0.9        6.50 ±  4%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      5.88 ±  5%      +1.0        6.83 ±  4%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +1.3        1.33 ±  7%  perf-profile.calltrace.cycles-pp.load_module.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
      0.00            +1.9        1.90 ± 11%  perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock
      0.00            +2.5        2.51 ±  4%  perf-profile.calltrace.cycles-pp.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit
      6.02 ±  2%      +2.8        8.78 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +3.2        3.18 ±  7%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit._printk
      9.98            +3.4       13.35        perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +3.6        3.56 ±  8%  perf-profile.calltrace.cycles-pp.do_one_initcall.do_init_module.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.8        3.82 ±  8%  perf-profile.calltrace.cycles-pp.do_init_module.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     12.24 ±  3%      +4.3       16.55        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     12.51 ±  3%      +4.4       16.96        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      3.43 ±  3%      +4.6        7.98 ±  3%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
     13.69 ±  3%      +4.7       18.35        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     13.69 ±  3%      +4.7       18.35        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     13.67 ±  3%      +4.7       18.33        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.68 ±  2%      +4.7        7.34 ±  3%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      2.38 ±  3%      +4.7        7.05 ±  3%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +4.7        4.67 ±  3%  perf-profile.calltrace.cycles-pp.memcpy_toio.drm_fb_memcpy.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm
     13.10 ±  3%      +4.7       17.80        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     13.93 ±  3%      +4.7       18.65        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      0.00            +4.7        4.73 ±  3%  perf-profile.calltrace.cycles-pp.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail
      0.00            +4.7        4.73 ±  3%  perf-profile.calltrace.cycles-pp.drm_fb_memcpy.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail
      0.00            +4.7        4.73 ±  3%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit
      0.00            +4.7        4.74 ±  3%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_commit
      0.00            +4.8        4.76 ±  3%  perf-profile.calltrace.cycles-pp.commit_tail.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty
      0.00            +4.8        4.76 ±  3%  perf-profile.calltrace.cycles-pp.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb
      0.00            +4.8        4.79 ±  3%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work
      0.00            +4.8        4.80 ±  3%  perf-profile.calltrace.cycles-pp.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work
      0.00            +4.8        4.80 ±  3%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread
      0.00            +5.2        5.18 ±  7%  perf-profile.calltrace.cycles-pp.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
      0.00            +5.2        5.19 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.syscall
      0.00            +5.2        5.19 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
      0.00            +5.2        5.19 ±  7%  perf-profile.calltrace.cycles-pp.syscall
      0.00            +5.4        5.38 ±  3%  perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +5.4        5.38 ±  3%  perf-profile.calltrace.cycles-pp.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread
     49.25 ±  2%      -4.4       44.89        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     49.19 ±  2%      -4.3       44.84        perf-profile.children.cycles-pp.do_syscall_64
     12.99 ±  2%      -3.6        9.34 ±  4%  perf-profile.children.cycles-pp.vm_mmap_pgoff
     12.84 ±  2%      -3.6        9.21 ±  4%  perf-profile.children.cycles-pp.do_mmap
     12.50 ±  2%      -3.6        8.91 ±  4%  perf-profile.children.cycles-pp.mmap_region
     10.87 ±  3%      -3.2        7.67 ±  4%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
     21.46 ±  2%      -2.9       18.51        perf-profile.children.cycles-pp.asm_exc_page_fault
     20.13 ±  2%      -2.8       17.30        perf-profile.children.cycles-pp.exc_page_fault
     20.06 ±  2%      -2.8       17.25        perf-profile.children.cycles-pp.do_user_addr_fault
     18.96 ±  2%      -2.7       16.28 ±  2%  perf-profile.children.cycles-pp.handle_mm_fault
     18.50 ±  2%      -2.6       15.88 ±  2%  perf-profile.children.cycles-pp.__handle_mm_fault
      8.98 ±  3%      -2.6        6.42 ±  3%  perf-profile.children.cycles-pp.do_vmi_munmap
     10.39            -2.6        7.84 ±  3%  perf-profile.children.cycles-pp.do_group_exit
     10.39            -2.5        7.84 ±  3%  perf-profile.children.cycles-pp.__x64_sys_exit_group
     10.37            -2.5        7.83 ±  3%  perf-profile.children.cycles-pp.do_exit
      8.82 ±  3%      -2.5        6.30 ±  3%  perf-profile.children.cycles-pp.do_vmi_align_munmap
      4.66 ±  7%      -2.5        2.19 ± 16%  perf-profile.children.cycles-pp.down_write
     15.37 ±  2%      -2.4       12.93 ±  2%  perf-profile.children.cycles-pp.do_fault
      9.35            -2.4        6.96 ±  4%  perf-profile.children.cycles-pp.exit_mm
      9.32            -2.4        6.94 ±  4%  perf-profile.children.cycles-pp.__mmput
      9.29            -2.4        6.90 ±  3%  perf-profile.children.cycles-pp.exit_mmap
      3.93 ±  9%      -2.3        1.64 ± 18%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      3.74 ±  9%      -2.2        1.55 ± 19%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
     10.28 ±  8%      -2.1        8.16 ±  3%  perf-profile.children.cycles-pp.jent_gen_entropy
     10.27 ±  8%      -2.1        8.16 ±  3%  perf-profile.children.cycles-pp.jent_measure_jitter
      9.96 ±  8%      -2.1        7.90 ±  3%  perf-profile.children.cycles-pp.jent_kcapi_random
      9.96 ±  8%      -2.1        7.90 ±  3%  perf-profile.children.cycles-pp.jent_read_entropy
      9.96 ±  8%      -2.1        7.90 ±  3%  perf-profile.children.cycles-pp._rng_recvmsg
     12.31 ±  3%      -2.0       10.28 ±  3%  perf-profile.children.cycles-pp.do_read_fault
     12.18 ±  3%      -2.0       10.16 ±  3%  perf-profile.children.cycles-pp.filemap_map_pages
      8.64 ±  8%      -1.8        6.85 ±  3%  perf-profile.children.cycles-pp.jent_lfsr_time
     10.05 ±  3%      -1.6        8.42 ±  3%  perf-profile.children.cycles-pp.next_uptodate_page
      5.94 ±  2%      -1.6        4.30 ±  5%  perf-profile.children.cycles-pp.__split_vma
      3.62 ±  4%      -1.5        2.14 ±  9%  perf-profile.children.cycles-pp.free_pgtables
      3.81 ±  4%      -1.3        2.49 ±  9%  perf-profile.children.cycles-pp.vma_prepare
      2.60 ±  6%      -1.2        1.37 ± 13%  perf-profile.children.cycles-pp.unlink_file_vma
      2.24 ±  5%      -1.2        1.03 ± 17%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      6.49            -1.2        5.32 ±  3%  perf-profile.children.cycles-pp.call_usermodehelper_exec_async
      6.36            -1.2        5.20 ±  3%  perf-profile.children.cycles-pp.kernel_execve
      2.98 ±  4%      -1.0        2.00 ±  4%  perf-profile.children.cycles-pp.unmap_region
      5.35 ±  2%      -1.0        4.39 ±  2%  perf-profile.children.cycles-pp.unmap_vmas
      5.05 ±  2%      -0.9        4.15 ±  2%  perf-profile.children.cycles-pp.unmap_page_range
      4.97 ±  3%      -0.9        4.08 ±  2%  perf-profile.children.cycles-pp.zap_pmd_range
      4.92 ±  2%      -0.9        4.04 ±  2%  perf-profile.children.cycles-pp.zap_pte_range
      1.33 ± 19%      -0.8        0.48 ± 28%  perf-profile.children.cycles-pp.osq_lock
      4.88 ±  2%      -0.8        4.07 ±  3%  perf-profile.children.cycles-pp.bprm_execve
      2.60            -0.8        1.82 ±  4%  perf-profile.children.cycles-pp.tlb_finish_mmu
      4.28 ±  2%      -0.7        3.56 ±  4%  perf-profile.children.cycles-pp.exec_binprm
      4.26 ±  2%      -0.7        3.54 ±  4%  perf-profile.children.cycles-pp.search_binary_handler
      4.18 ±  2%      -0.7        3.48 ±  4%  perf-profile.children.cycles-pp.load_elf_binary
      2.02 ±  2%      -0.7        1.36 ±  6%  perf-profile.children.cycles-pp.tlb_batch_pages_flush
      2.78 ±  4%      -0.5        2.24 ±  7%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      1.62 ±  4%      -0.5        1.09 ±  7%  perf-profile.children.cycles-pp.release_pages
      2.45 ±  3%      -0.5        1.94 ±  3%  perf-profile.children.cycles-pp.__vm_munmap
      1.94 ±  5%      -0.5        1.48 ±  6%  perf-profile.children.cycles-pp.elf_map
      1.78 ±  2%      -0.4        1.34 ±  3%  perf-profile.children.cycles-pp.call_usermodehelper_exec_work
      3.03            -0.4        2.64        perf-profile.children.cycles-pp.do_cow_fault
      1.91 ±  3%      -0.4        1.53 ±  2%  perf-profile.children.cycles-pp.__x64_sys_munmap
      1.17 ±  3%      -0.4        0.79 ±  7%  perf-profile.children.cycles-pp.up_write
      0.66 ± 13%      -0.4        0.31 ± 12%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      2.02 ±  2%      -0.3        1.68 ±  3%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      2.02 ±  3%      -0.3        1.68 ±  3%  perf-profile.children.cycles-pp.do_mprotect_pkey
      1.41 ±  3%      -0.3        1.08 ±  3%  perf-profile.children.cycles-pp.user_mode_thread
      1.58 ±  4%      -0.3        1.26 ±  3%  perf-profile.children.cycles-pp.vma_complete
      1.46 ±  2%      -0.3        1.14 ±  3%  perf-profile.children.cycles-pp.kernel_clone
      1.52 ±  2%      -0.3        1.20 ±  3%  perf-profile.children.cycles-pp.__munmap
      1.59 ±  7%      -0.3        1.27 ±  5%  perf-profile.children.cycles-pp.jent_memaccess
      1.76 ±  3%      -0.3        1.46 ±  4%  perf-profile.children.cycles-pp.mprotect_fixup
      1.12 ±  5%      -0.3        0.83 ±  9%  perf-profile.children.cycles-pp.alloc_bprm
      1.17 ±  4%      -0.3        0.88 ±  5%  perf-profile.children.cycles-pp.load_elf_interp
      1.96 ±  3%      -0.3        1.68        perf-profile.children.cycles-pp.__x64_sys_openat
      1.94 ±  3%      -0.3        1.67        perf-profile.children.cycles-pp.do_sys_openat2
      1.04 ±  5%      -0.3        0.77 ± 10%  perf-profile.children.cycles-pp.mm_init
      2.01 ±  2%      -0.3        1.74        perf-profile.children.cycles-pp.do_filp_open
      1.99 ±  2%      -0.3        1.72        perf-profile.children.cycles-pp.path_openat
      1.16 ±  2%      -0.3        0.90 ±  4%  perf-profile.children.cycles-pp.copy_process
      2.08            -0.2        1.84 ±  4%  perf-profile.children.cycles-pp.mas_store_prealloc
      2.02 ±  2%      -0.2        1.78 ±  2%  perf-profile.children.cycles-pp._dl_addr
      0.82 ±  5%      -0.2        0.57 ± 12%  perf-profile.children.cycles-pp.pcpu_alloc
      0.85            -0.2        0.61 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.29 ±  2%      -0.2        1.05 ±  6%  perf-profile.children.cycles-pp.do_set_pte
      2.10 ±  2%      -0.2        1.87 ±  2%  perf-profile.children.cycles-pp.__do_softirq
      1.63 ±  4%      -0.2        1.40 ±  5%  perf-profile.children.cycles-pp.page_remove_rmap
      1.18 ±  4%      -0.2        0.95        perf-profile.children.cycles-pp.mas_alloc_nodes
      0.65 ±  4%      -0.2        0.43 ± 11%  perf-profile.children.cycles-pp.__percpu_counter_init
      0.68 ±  6%      -0.2        0.46 ±  2%  perf-profile.children.cycles-pp.vma_expand
      1.08 ±  4%      -0.2        0.86        perf-profile.children.cycles-pp.mas_preallocate
      2.14 ±  2%      -0.2        1.93        perf-profile.children.cycles-pp._raw_spin_lock
      0.95 ±  2%      -0.2        0.74 ±  6%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.65 ±  3%      -0.2        0.45 ±  2%  perf-profile.children.cycles-pp.alloc_thread_stack_node
      0.76 ±  3%      -0.2        0.56 ±  2%  perf-profile.children.cycles-pp.dup_task_struct
      1.09 ±  2%      -0.2        0.90 ±  5%  perf-profile.children.cycles-pp.__mmap
      0.41 ±  5%      -0.2        0.22 ±  5%  perf-profile.children.cycles-pp.folio_lruvec_lock_irqsave
      1.11 ±  2%      -0.2        0.93 ±  6%  perf-profile.children.cycles-pp.copy_page
      1.38 ±  2%      -0.2        1.22 ±  2%  perf-profile.children.cycles-pp.rcu_core
      0.36 ±  7%      -0.2        0.20 ± 10%  perf-profile.children.cycles-pp.__get_vm_area_node
      0.60 ±  3%      -0.2        0.44 ±  3%  perf-profile.children.cycles-pp.__vmalloc_node_range
      1.26 ±  2%      -0.2        1.10 ±  2%  perf-profile.children.cycles-pp.rcu_do_batch
      0.65 ±  4%      -0.2        0.50 ±  4%  perf-profile.children.cycles-pp.stress_mwc8
      0.94 ±  5%      -0.1        0.80 ±  3%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.31 ±  7%      -0.1        0.17 ± 13%  perf-profile.children.cycles-pp.alloc_vmap_area
      0.37 ±  8%      -0.1        0.23 ±  8%  perf-profile.children.cycles-pp.delayed_vfree_work
      1.79 ±  3%      -0.1        1.65 ±  2%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.68 ±  6%      -0.1        0.55 ±  8%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.99 ±  3%      -0.1        0.85 ±  2%  perf-profile.children.cycles-pp.open64
      0.77 ±  4%      -0.1        0.64 ±  6%  perf-profile.children.cycles-pp.begin_new_exec
      0.56 ±  5%      -0.1        0.43 ±  8%  perf-profile.children.cycles-pp.exec_mmap
      0.51 ±  5%      -0.1        0.38 ±  5%  perf-profile.children.cycles-pp.__fput
      0.66 ±  3%      -0.1        0.53 ±  2%  perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.41 ±  5%      -0.1        0.28 ± 10%  perf-profile.children.cycles-pp.free_swap_cache
      0.64 ±  7%      -0.1        0.52 ±  9%  perf-profile.children.cycles-pp.sendmsg
      0.58 ±  4%      -0.1        0.45 ±  4%  perf-profile.children.cycles-pp.task_work_run
      0.53 ±  5%      -0.1        0.41 ±  6%  perf-profile.children.cycles-pp.folio_batch_move_lru
      0.53 ±  8%      -0.1        0.41 ±  5%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.56 ±  7%      -0.1        0.44 ±  8%  perf-profile.children.cycles-pp.___sys_sendmsg
      0.42 ±  4%      -0.1        0.30 ±  8%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.81 ±  4%      -0.1        0.69 ±  5%  perf-profile.children.cycles-pp.__x64_sys_bind
      0.81 ±  4%      -0.1        0.69 ±  5%  perf-profile.children.cycles-pp.__sys_bind
      0.52 ±  7%      -0.1        0.40 ±  5%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      1.06 ±  2%      -0.1        0.94 ±  4%  perf-profile.children.cycles-pp.perf_event_mmap
      0.80 ±  5%      -0.1        0.68 ±  5%  perf-profile.children.cycles-pp.alg_bind
      1.02 ±  2%      -0.1        0.91 ±  4%  perf-profile.children.cycles-pp.perf_event_mmap_event
      0.57 ±  5%      -0.1        0.46 ±  4%  perf-profile.children.cycles-pp.flush_tlb_func
      0.82 ±  4%      -0.1        0.71 ±  4%  perf-profile.children.cycles-pp.bind
      0.48 ±  9%      -0.1        0.36 ±  9%  perf-profile.children.cycles-pp.stress_rndbuf
      0.32 ±  8%      -0.1        0.21 ±  8%  perf-profile.children.cycles-pp.remove_vm_area
      0.75 ±  5%      -0.1        0.64 ±  2%  perf-profile.children.cycles-pp.kmem_cache_alloc_bulk
      0.32 ±  5%      -0.1        0.21 ± 21%  perf-profile.children.cycles-pp.__rb_insert_augmented
      0.36 ±  5%      -0.1        0.25 ±  9%  perf-profile.children.cycles-pp.kernel_wait
      0.63 ±  4%      -0.1        0.53 ±  5%  perf-profile.children.cycles-pp.open_last_lookups
      0.36 ±  5%      -0.1        0.26 ±  8%  perf-profile.children.cycles-pp.do_wait
      0.78 ±  5%      -0.1        0.68 ±  3%  perf-profile.children.cycles-pp.mas_store_gfp
      0.68 ±  6%      -0.1        0.58 ±  4%  perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
      0.47 ±  7%      -0.1        0.37 ±  6%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.43 ±  7%      -0.1        0.33 ±  5%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.18 ± 15%      -0.1        0.08 ± 26%  perf-profile.children.cycles-pp.__rb_erase_color
      0.74 ±  5%      -0.1        0.65 ±  6%  perf-profile.children.cycles-pp.__do_fault
      0.65 ±  3%      -0.1        0.55 ±  5%  perf-profile.children.cycles-pp.__slab_free
      0.63 ±  3%      -0.1        0.54 ±  5%  perf-profile.children.cycles-pp.mas_destroy
      0.69 ±  4%      -0.1        0.59 ±  9%  perf-profile.children.cycles-pp.filemap_fault
      0.72 ±  3%      -0.1        0.63 ±  2%  perf-profile.children.cycles-pp.mas_wr_bnode
      1.06 ±  3%      -0.1        0.97 ±  4%  perf-profile.children.cycles-pp.__schedule
      0.46 ±  5%      -0.1        0.36 ±  7%  perf-profile.children.cycles-pp.filename_lookup
      0.62 ±  3%      -0.1        0.53 ±  6%  perf-profile.children.cycles-pp.getenv
      0.38 ±  3%      -0.1        0.29 ±  5%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.45 ±  5%      -0.1        0.36 ±  6%  perf-profile.children.cycles-pp.path_lookupat
      0.51 ±  4%      -0.1        0.42 ± 10%  perf-profile.children.cycles-pp.perf_event_mmap_output
      0.49 ±  5%      -0.1        0.40 ± 10%  perf-profile.children.cycles-pp.__mmdrop
      0.38 ±  3%      -0.1        0.29 ±  5%  perf-profile.children.cycles-pp.lru_add_drain
      0.51 ±  4%      -0.1        0.42 ±  3%  perf-profile.children.cycles-pp.lock_vma_under_rcu
      0.69 ±  7%      -0.1        0.60 ±  5%  perf-profile.children.cycles-pp.mtree_range_walk
      0.69 ±  5%      -0.1        0.60 ±  6%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.62 ±  2%      -0.1        0.53 ±  7%  perf-profile.children.cycles-pp.do_open
      0.50 ±  3%      -0.1        0.42 ±  4%  perf-profile.children.cycles-pp.vfs_fstatat
      0.48 ±  3%      -0.1        0.40 ±  9%  perf-profile.children.cycles-pp.do_dentry_open
      0.41 ±  8%      -0.1        0.33 ± 12%  perf-profile.children.cycles-pp.___slab_alloc
      0.47 ±  6%      -0.1        0.39 ±  8%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.57 ±  7%      -0.1        0.49 ±  6%  perf-profile.children.cycles-pp.mas_walk
      0.70 ±  5%      -0.1        0.63 ±  5%  perf-profile.children.cycles-pp.crypto_alloc_tfm_node
      0.32 ±  4%      -0.1        0.24 ± 11%  perf-profile.children.cycles-pp.lru_add_fn
      0.50 ±  4%      -0.1        0.43 ± 11%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.42 ±  5%      -0.1        0.35 ±  5%  perf-profile.children.cycles-pp.vfs_statx
      0.52 ±  5%      -0.1        0.44 ±  4%  perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.45            -0.1        0.38 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
      0.11 ± 13%      -0.1        0.04 ± 45%  perf-profile.children.cycles-pp.free_vmap_area_noflush
      0.30 ±  7%      -0.1        0.23 ± 13%  perf-profile.children.cycles-pp.getopt_long
      0.27            -0.1        0.20 ±  7%  perf-profile.children.cycles-pp.dput
      0.28 ± 11%      -0.1        0.22 ±  4%  perf-profile.children.cycles-pp.getdents64
      0.41 ±  3%      -0.1        0.34 ±  6%  perf-profile.children.cycles-pp.__xstat64
      0.27 ± 12%      -0.1        0.20 ±  4%  perf-profile.children.cycles-pp.__x64_sys_getdents64
      0.64 ±  6%      -0.1        0.57 ±  6%  perf-profile.children.cycles-pp.__filemap_get_folio
      0.33 ±  9%      -0.1        0.27 ±  4%  perf-profile.children.cycles-pp.mas_find
      0.25 ± 12%      -0.1        0.18 ±  4%  perf-profile.children.cycles-pp.__close_nocancel
      0.44 ±  4%      -0.1        0.38 ± 10%  perf-profile.children.cycles-pp.find_idlest_cpu
      0.39 ±  6%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp.walk_component
      0.40 ±  6%      -0.1        0.34 ±  8%  perf-profile.children.cycles-pp.kmem_cache_free_bulk
      0.25 ± 10%      -0.1        0.19 ±  3%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.26 ± 11%      -0.1        0.20 ±  2%  perf-profile.children.cycles-pp.iterate_dir
      0.16 ±  7%      -0.1        0.10 ± 14%  perf-profile.children.cycles-pp.vma_interval_tree_augment_rotate
      0.28 ±  6%      -0.1        0.22 ± 11%  perf-profile.children.cycles-pp.wake_up_new_task
      0.46 ±  3%      -0.1        0.40 ±  3%  perf-profile.children.cycles-pp.mas_split
      0.38 ±  4%      -0.1        0.33 ±  5%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.34 ±  6%      -0.1        0.28 ±  5%  perf-profile.children.cycles-pp.crypto_create_tfm_node
      0.22 ±  8%      -0.1        0.16 ± 18%  perf-profile.children.cycles-pp.__do_sys_newfstat
      0.30 ±  6%      -0.1        0.24 ±  4%  perf-profile.children.cycles-pp.try_to_wake_up
      0.42 ±  4%      -0.1        0.36 ± 11%  perf-profile.children.cycles-pp.find_idlest_group
      0.36 ±  5%      -0.1        0.31 ±  5%  perf-profile.children.cycles-pp.step_into
      0.17 ± 12%      -0.1        0.12 ± 18%  perf-profile.children.cycles-pp.vfs_fstat
      0.36 ±  6%      -0.1        0.31 ±  3%  perf-profile.children.cycles-pp.__getrlimit
      0.32 ±  6%      -0.1        0.26 ±  6%  perf-profile.children.cycles-pp.jent_kcapi_init
      0.32 ±  6%      -0.1        0.26 ±  6%  perf-profile.children.cycles-pp.jent_entropy_collector_alloc
      0.27 ±  5%      -0.1        0.22 ±  8%  perf-profile.children.cycles-pp.touch_atime
      0.15 ±  8%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.security_file_alloc
      0.32 ±  6%      -0.1        0.27 ±  6%  perf-profile.children.cycles-pp.rng_bind
      0.20 ± 10%      -0.1        0.15 ±  8%  perf-profile.children.cycles-pp.wait_task_zombie
      0.22 ±  5%      -0.1        0.17 ± 10%  perf-profile.children.cycles-pp.folio_add_lru_vma
      0.16 ±  8%      -0.0        0.11 ± 12%  perf-profile.children.cycles-pp.lockref_put_or_lock
      0.74 ±  4%      -0.0        0.69 ±  3%  perf-profile.children.cycles-pp.finish_fault
      0.26 ±  8%      -0.0        0.22 ±  7%  perf-profile.children.cycles-pp.xas_find
      0.18 ± 12%      -0.0        0.13 ±  8%  perf-profile.children.cycles-pp.find_unlink_vmap_area
      0.16 ±  3%      -0.0        0.11 ± 17%  perf-profile.children.cycles-pp.__wake_up_common
      0.26 ±  8%      -0.0        0.21 ± 11%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.18 ±  8%      -0.0        0.14 ± 14%  perf-profile.children.cycles-pp._compound_head
      0.49 ±  3%      -0.0        0.44 ±  4%  perf-profile.children.cycles-pp.vma_alloc_folio
      0.20 ±  9%      -0.0        0.15 ±  8%  perf-profile.children.cycles-pp.__list_add_valid
      0.23 ±  6%      -0.0        0.18 ±  8%  perf-profile.children.cycles-pp.unmap_single_vma
      0.18 ± 12%      -0.0        0.13 ±  6%  perf-profile.children.cycles-pp.put_cred_rcu
      0.10 ±  8%      -0.0        0.06 ± 52%  perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
      0.32 ±  2%      -0.0        0.28 ±  3%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.18 ±  8%      -0.0        0.13 ± 14%  perf-profile.children.cycles-pp.do_wp_page
      0.15 ±  3%      -0.0        0.10 ± 13%  perf-profile.children.cycles-pp.percpu_counter_destroy
      0.12 ± 11%      -0.0        0.08 ± 12%  perf-profile.children.cycles-pp.apparmor_file_alloc_security
      0.11 ±  9%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.__unfreeze_partials
      0.08 ± 14%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp.security_inode_getattr
      0.17 ± 10%      -0.0        0.13 ±  9%  perf-profile.children.cycles-pp.release_task
      0.30 ±  6%      -0.0        0.26 ±  9%  perf-profile.children.cycles-pp.wmemchr
      0.37 ±  3%      -0.0        0.33 ±  5%  perf-profile.children.cycles-pp.___perf_sw_event
      0.28 ±  7%      -0.0        0.24 ±  5%  perf-profile.children.cycles-pp.vm_area_dup
      0.17 ± 12%      -0.0        0.13 ± 10%  perf-profile.children.cycles-pp.try_to_unlazy
      0.15 ± 11%      -0.0        0.11 ± 12%  perf-profile.children.cycles-pp.free_percpu
      0.31 ±  4%      -0.0        0.27 ±  5%  perf-profile.children.cycles-pp.mas_next_entry
      0.16 ±  5%      -0.0        0.12 ± 12%  perf-profile.children.cycles-pp.__fxstatat64
      0.12 ± 15%      -0.0        0.08 ± 12%  perf-profile.children.cycles-pp.atime_needs_update
      0.15 ±  7%      -0.0        0.12 ±  9%  perf-profile.children.cycles-pp.userfaultfd_unmap_prep
      0.23 ±  7%      -0.0        0.20 ±  7%  perf-profile.children.cycles-pp.pick_link
      0.16 ±  8%      -0.0        0.13 ±  5%  perf-profile.children.cycles-pp.dcache_readdir
      0.16 ± 10%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.__crypto_xor
      0.17 ±  8%      -0.0        0.14 ±  9%  perf-profile.children.cycles-pp.__pthread_initialize_minimal_internal
      0.12 ±  8%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.__lookup_slow
      0.22 ±  5%      -0.0        0.20 ±  7%  perf-profile.children.cycles-pp.d_path
      0.14 ±  4%      -0.0        0.11 ± 12%  perf-profile.children.cycles-pp.__do_sys_newfstatat
      0.11 ± 12%      -0.0        0.08 ± 12%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.22 ±  6%      -0.0        0.19 ±  6%  perf-profile.children.cycles-pp.vm_area_alloc
      0.09 ±  8%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.12 ±  8%      -0.0        0.09 ± 15%  perf-profile.children.cycles-pp.af_alg_get_rsgl
      0.14 ±  4%      -0.0        0.12 ± 10%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.12 ±  8%      -0.0        0.09 ± 11%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.18 ±  4%      -0.0        0.16 ±  5%  perf-profile.children.cycles-pp.folio_add_new_anon_rmap
      0.14 ±  6%      -0.0        0.12 ± 11%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.13 ±  5%      -0.0        0.11 ±  5%  perf-profile.children.cycles-pp.folio_test_hugetlb
      0.10 ±  4%      -0.0        0.08 ± 14%  perf-profile.children.cycles-pp.folio_mark_accessed
      0.07 ± 11%      -0.0        0.05 ±  7%  perf-profile.children.cycles-pp.dentry_kill
      0.08 ±  6%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.check_heap_object
      0.10 ±  6%      +0.0        0.12 ±  9%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.09 ±  9%      +0.0        0.12 ±  9%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.06 ±  9%      +0.0        0.09 ± 12%  perf-profile.children.cycles-pp.ct_idle_exit
      0.08 ± 12%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.finish_task_switch
      0.12 ± 12%      +0.0        0.15 ±  7%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.16 ±  4%      +0.0        0.20 ± 10%  perf-profile.children.cycles-pp.native_sched_clock
      0.17 ± 10%      +0.0        0.20 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      0.11 ±  9%      +0.0        0.15 ±  8%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.16 ± 10%      +0.1        0.22 ±  5%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.17 ± 45%      +0.1        0.23 ±  8%  perf-profile.children.cycles-pp.xts_decrypt
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.camellia_decrypt_cbc_2way
      0.11 ± 10%      +0.1        0.18 ±  7%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.trace_module_notify
      0.01 ±223%      +0.1        0.09 ± 16%  perf-profile.children.cycles-pp.shmem_read_folio_gfp
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.__serpent_enc_blk8_avx
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.serpent_ecb_enc_8way_avx
      0.00            +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.serpent_ecb_dec_8way_avx
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.shash_digest_unaligned
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.shash_update_unaligned
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp._sha256_update
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.sha256_base_do_update
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.sha256_ni_transform
      0.01 ±223%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
      0.01 ±223%      +0.1        0.10 ± 16%  perf-profile.children.cycles-pp.drm_gem_get_pages
      0.01 ±223%      +0.1        0.10 ± 16%  perf-profile.children.cycles-pp.drm_gem_shmem_get_pages
      0.01 ±223%      +0.1        0.10 ± 16%  perf-profile.children.cycles-pp.drm_gem_shmem_get_pages_locked
      0.00            +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.drm_client_buffer_vmap
      0.01 ±223%      +0.1        0.11 ± 14%  perf-profile.children.cycles-pp.shmem_get_folio_gfp
      0.00            +0.1        0.10 ± 11%  perf-profile.children.cycles-pp.llist_reverse_order
      0.00            +0.1        0.10 ± 11%  perf-profile.children.cycles-pp.mpihelp_addmul_1
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.mpih_sqr_n_basecase
      0.00            +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.blocking_notifier_call_chain
      0.00            +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.notifier_call_chain
      0.00            +0.1        0.11 ± 19%  perf-profile.children.cycles-pp.early_mod_check
      0.01 ±223%      +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.drm_gem_shmem_object_vmap
      0.01 ±223%      +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.drm_gem_shmem_vmap_locked
      0.00            +0.1        0.11 ± 13%  perf-profile.children.cycles-pp.pkcs7_verify
      0.00            +0.1        0.11 ± 13%  perf-profile.children.cycles-pp.pkcs7_digest
      0.01 ±223%      +0.1        0.12 ± 15%  perf-profile.children.cycles-pp.drm_gem_vmap_unlocked
      0.01 ±223%      +0.1        0.12 ± 15%  perf-profile.children.cycles-pp.drm_gem_vmap
      0.00            +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.__camellia_enc_blk_2way
      0.00            +0.1        0.12 ± 19%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.38 ± 45%      +0.1        0.50 ±  3%  perf-profile.children.cycles-pp.ecb_encrypt
      0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.__serpent_dec_blk8_avx
      0.39 ± 44%      +0.1        0.52 ±  5%  perf-profile.children.cycles-pp.ecb_decrypt
      0.00            +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.free_module
      0.00            +0.1        0.13 ± 13%  perf-profile.children.cycles-pp.io_serial_out
      0.03 ±103%      +0.1        0.16 ± 12%  perf-profile.children.cycles-pp.__mutex_lock
      0.00            +0.1        0.13 ± 12%  perf-profile.children.cycles-pp.mpih_sqr_n
      0.00            +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.native_flush_tlb_global
      0.00            +0.1        0.15 ± 15%  perf-profile.children.cycles-pp.__flush_tlb_all
      0.00            +0.2        0.15 ± 15%  perf-profile.children.cycles-pp.fast_imageblit
      0.00            +0.2        0.16 ± 15%  perf-profile.children.cycles-pp.sys_imageblit
      0.88 ±  8%      +0.2        1.04 ±  4%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.00            +0.2        0.18 ±  7%  perf-profile.children.cycles-pp.camellia_dec_blk_2way
      1.59 ±  7%      +0.2        1.77 ±  3%  perf-profile.children.cycles-pp.perf_event_task_tick
      1.57 ±  7%      +0.2        1.75 ±  3%  perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context
      0.46 ±  6%      +0.2        0.66 ±  3%  perf-profile.children.cycles-pp.menu_select
      0.00            +0.2        0.20 ± 15%  perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit
      0.00            +0.2        0.20 ± 17%  perf-profile.children.cycles-pp.flush_tlb_kernel_range
      0.00            +0.2        0.22 ± 17%  perf-profile.children.cycles-pp.fbcon_putcs
      0.00            +0.2        0.22 ± 17%  perf-profile.children.cycles-pp.bit_putcs
      0.00            +0.2        0.24 ± 16%  perf-profile.children.cycles-pp.fbcon_redraw
      0.00            +0.2        0.24 ± 14%  perf-profile.children.cycles-pp.lf
      0.00            +0.2        0.24 ± 14%  perf-profile.children.cycles-pp.con_scroll
      0.00            +0.2        0.24 ± 14%  perf-profile.children.cycles-pp.fbcon_scroll
      0.00            +0.2        0.25 ± 15%  perf-profile.children.cycles-pp.vt_console_print
      0.00            +0.3        0.30 ± 10%  perf-profile.children.cycles-pp.mpihelp_submul_1
      0.00            +0.3        0.33 ±  8%  perf-profile.children.cycles-pp.__sysvec_call_function
      0.00            +0.3        0.33 ±  8%  perf-profile.children.cycles-pp.mpihelp_divrem
      0.00            +0.4        0.36 ±  8%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      3.94 ±  6%      +0.4        4.32 ±  2%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.00            +0.4        0.39 ± 30%  perf-profile.children.cycles-pp.set_memory_ro
      0.00            +0.4        0.40 ± 31%  perf-profile.children.cycles-pp.module_enable_ro
      0.00            +0.4        0.42 ±  9%  perf-profile.children.cycles-pp.sysvec_call_function
      0.00            +0.4        0.42 ± 28%  perf-profile.children.cycles-pp.change_page_attr_set_clr
      0.25 ±  5%      +0.4        0.69 ±  4%  perf-profile.children.cycles-pp.__memcpy
      0.00            +0.4        0.45 ±  5%  perf-profile.children.cycles-pp.drm_fbdev_generic_damage_blit_real
      0.00            +0.5        0.46 ± 23%  perf-profile.children.cycles-pp.__purge_vmap_area_lazy
      0.00            +0.5        0.48 ±  6%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.00            +0.5        0.48 ±  6%  perf-profile.children.cycles-pp.mpi_powm
      0.00            +0.5        0.48 ±  6%  perf-profile.children.cycles-pp.pkcs1pad_verify
      0.00            +0.5        0.48 ±  6%  perf-profile.children.cycles-pp.rsa_enc
      0.00            +0.5        0.49 ± 24%  perf-profile.children.cycles-pp.on_each_cpu_cond_mask
      0.00            +0.5        0.50 ±  5%  perf-profile.children.cycles-pp.public_key_verify_signature
      0.00            +0.5        0.50 ± 23%  perf-profile.children.cycles-pp.smp_call_function_many_cond
      0.00            +0.5        0.50 ±  5%  perf-profile.children.cycles-pp.pkcs7_validate_trust
      0.00            +0.5        0.50 ±  5%  perf-profile.children.cycles-pp.pkcs7_validate_trust_one
      0.00            +0.5        0.51 ± 18%  perf-profile.children.cycles-pp._vm_unmap_aliases
      0.00            +0.5        0.52 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_call_function
      4.44 ±  5%      +0.5        4.97 ±  2%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      4.42 ±  5%      +0.5        4.95 ±  2%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +0.6        0.61 ±  6%  perf-profile.children.cycles-pp.verify_pkcs7_message_sig
      0.00            +0.6        0.62 ±  6%  perf-profile.children.cycles-pp.module_sig_check
      0.00            +0.6        0.62 ±  6%  perf-profile.children.cycles-pp.mod_verify_sig
      0.00            +0.6        0.62 ±  6%  perf-profile.children.cycles-pp.verify_pkcs7_signature
      6.73 ±  5%      +0.7        7.45 ±  4%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.01 ±223%      +0.8        0.76 ± 21%  perf-profile.children.cycles-pp.delay_tsc
      7.07 ±  5%      +0.8        7.89 ±  3%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.00            +1.3        1.33 ±  7%  perf-profile.children.cycles-pp.load_module
      0.14 ± 23%      +2.4        2.50 ± 11%  perf-profile.children.cycles-pp.io_serial_in
      6.13 ±  2%      +2.8        8.90 ±  2%  perf-profile.children.cycles-pp.intel_idle
      0.18 ± 18%      +3.1        3.26 ±  7%  perf-profile.children.cycles-pp.wait_for_lsr
      0.18 ± 18%      +3.2        3.40 ±  7%  perf-profile.children.cycles-pp.serial8250_console_write
      9.98            +3.4       13.36        perf-profile.children.cycles-pp.ret_from_fork
      0.19 ± 18%      +3.5        3.66 ±  7%  perf-profile.children.cycles-pp.console_flush_all
      0.19 ± 18%      +3.5        3.66 ±  7%  perf-profile.children.cycles-pp.console_unlock
      0.00            +3.6        3.56 ±  8%  perf-profile.children.cycles-pp.do_one_initcall
      0.19 ± 18%      +3.6        3.80 ±  7%  perf-profile.children.cycles-pp._printk
      0.19 ± 18%      +3.6        3.80 ±  7%  perf-profile.children.cycles-pp.vprintk_emit
      0.00            +3.8        3.82 ±  8%  perf-profile.children.cycles-pp.do_init_module
     12.69 ±  3%      +4.5       17.19        perf-profile.children.cycles-pp.cpuidle_enter_state
     12.72 ±  3%      +4.5       17.24        perf-profile.children.cycles-pp.cpuidle_enter
      3.43 ±  3%      +4.6        7.98 ±  3%  perf-profile.children.cycles-pp.kthread
     13.69 ±  3%      +4.7       18.35        perf-profile.children.cycles-pp.start_secondary
      2.68 ±  2%      +4.7        7.34 ±  3%  perf-profile.children.cycles-pp.worker_thread
      2.38 ±  3%      +4.7        7.05 ±  3%  perf-profile.children.cycles-pp.process_one_work
      0.05 ± 48%      +4.7        4.73 ±  3%  perf-profile.children.cycles-pp.ast_primary_plane_helper_atomic_update
      0.05 ± 48%      +4.7        4.73 ±  3%  perf-profile.children.cycles-pp.drm_fb_memcpy
      0.05 ± 48%      +4.7        4.73 ±  3%  perf-profile.children.cycles-pp.memcpy_toio
      0.05 ± 48%      +4.7        4.73 ±  3%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_planes
      0.06 ± 47%      +4.7        4.74 ±  3%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_tail_rpm
      0.06 ± 21%      +4.7        4.76 ±  3%  perf-profile.children.cycles-pp.commit_tail
      0.06 ± 21%      +4.7        4.76 ±  3%  perf-profile.children.cycles-pp.ast_mode_config_helper_atomic_commit_tail
      0.08 ± 32%      +4.7        4.80 ±  3%  perf-profile.children.cycles-pp.drm_atomic_commit
      0.08 ± 32%      +4.7        4.79 ±  3%  perf-profile.children.cycles-pp.drm_atomic_helper_commit
     13.93 ±  3%      +4.7       18.65        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     13.93 ±  3%      +4.7       18.65        perf-profile.children.cycles-pp.cpu_startup_entry
      0.08 ± 32%      +4.7        4.80 ±  3%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
     13.92 ±  3%      +4.7       18.65        perf-profile.children.cycles-pp.do_idle
     13.34 ±  3%      +4.8       18.10        perf-profile.children.cycles-pp.cpuidle_idle_call
      0.00            +5.2        5.18 ±  7%  perf-profile.children.cycles-pp.__do_sys_finit_module
      0.00            +5.2        5.19 ±  7%  perf-profile.children.cycles-pp.syscall
      0.11 ± 20%      +5.3        5.38 ±  3%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.11 ± 20%      +5.3        5.38 ±  3%  perf-profile.children.cycles-pp.drm_fbdev_generic_helper_fb_dirty
      8.53 ±  8%      -1.8        6.76 ±  3%  perf-profile.self.cycles-pp.jent_lfsr_time
      9.90 ±  3%      -1.6        8.32 ±  3%  perf-profile.self.cycles-pp.next_uptodate_page
      2.19 ±  5%      -1.2        1.02 ± 17%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      1.30 ± 18%      -0.8        0.48 ± 29%  perf-profile.self.cycles-pp.osq_lock
      2.99 ±  2%      -0.6        2.37        perf-profile.self.cycles-pp.zap_pte_range
      2.75 ±  4%      -0.5        2.20 ±  6%  perf-profile.self.cycles-pp.vma_interval_tree_insert
      1.14 ±  3%      -0.4        0.78 ±  6%  perf-profile.self.cycles-pp.up_write
      0.66 ± 13%      -0.4        0.30 ± 11%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.54 ±  8%      -0.3        1.23 ±  6%  perf-profile.self.cycles-pp.jent_memaccess
      0.95 ±  4%      -0.3        0.67 ±  7%  perf-profile.self.cycles-pp.release_pages
      1.72 ±  2%      -0.2        1.50 ±  2%  perf-profile.self.cycles-pp._dl_addr
      1.47 ±  5%      -0.2        1.25 ±  6%  perf-profile.self.cycles-pp.page_remove_rmap
      0.94 ±  2%      -0.2        0.73 ±  5%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.68            -0.2        0.49 ±  9%  perf-profile.self.cycles-pp.down_write
      1.09 ±  4%      -0.2        0.92 ±  6%  perf-profile.self.cycles-pp.copy_page
      0.82 ±  6%      -0.2        0.67 ±  4%  perf-profile.self.cycles-pp.filemap_map_pages
      0.37 ±  4%      -0.1        0.25 ± 11%  perf-profile.self.cycles-pp.free_swap_cache
      0.50 ±  4%      -0.1        0.39        perf-profile.self.cycles-pp.stress_mwc8
      0.18 ± 10%      -0.1        0.08 ± 18%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.29 ±  4%      -0.1        0.20 ± 20%  perf-profile.self.cycles-pp.__rb_insert_augmented
      0.28 ±  5%      -0.1        0.18 ± 17%  perf-profile.self.cycles-pp.pcpu_alloc
      0.67 ±  7%      -0.1        0.58 ±  3%  perf-profile.self.cycles-pp.mtree_range_walk
      0.63 ±  3%      -0.1        0.54 ±  5%  perf-profile.self.cycles-pp.__slab_free
      0.46 ±  6%      -0.1        0.37 ±  8%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.41 ±  6%      -0.1        0.32 ±  6%  perf-profile.self.cycles-pp.mmap_region
      0.16 ± 14%      -0.1        0.08 ± 26%  perf-profile.self.cycles-pp.__rb_erase_color
      0.40 ±  4%      -0.1        0.32 ±  7%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.44 ±  4%      -0.1        0.37 ±  7%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.52 ±  5%      -0.1        0.44 ±  4%  perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.51 ±  4%      -0.1        0.44 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.30 ± 12%      -0.1        0.24 ± 14%  perf-profile.self.cycles-pp.stress_rndbuf
      0.16 ±  7%      -0.1        0.10 ± 14%  perf-profile.self.cycles-pp.vma_interval_tree_augment_rotate
      0.22 ± 10%      -0.1        0.16 ±  8%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.27 ±  7%      -0.1        0.22 ±  9%  perf-profile.self.cycles-pp.___slab_alloc
      0.27 ±  5%      -0.1        0.22 ± 12%  perf-profile.self.cycles-pp.perf_event_mmap_output
      0.20 ± 10%      -0.1        0.14 ±  5%  perf-profile.self.cycles-pp.__list_add_valid
      0.12 ± 11%      -0.0        0.07 ± 13%  perf-profile.self.cycles-pp.apparmor_file_alloc_security
      0.22 ±  7%      -0.0        0.17 ±  8%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.07 ± 17%      -0.0        0.02 ± 99%  perf-profile.self.cycles-pp.apparmor_file_open
      0.43 ±  3%      -0.0        0.38 ±  5%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.15 ±  9%      -0.0        0.10 ± 17%  perf-profile.self.cycles-pp.lockref_put_or_lock
      0.22 ± 10%      -0.0        0.17 ±  8%  perf-profile.self.cycles-pp.lock_vma_under_rcu
      0.22 ±  9%      -0.0        0.17 ±  9%  perf-profile.self.cycles-pp.do_dentry_open
      0.10 ±  8%      -0.0        0.05 ± 50%  perf-profile.self.cycles-pp.mem_cgroup_update_lru_size
      0.30 ±  6%      -0.0        0.26 ±  5%  perf-profile.self.cycles-pp.kmem_cache_free_bulk
      0.12 ±  7%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.filemap_read
      0.16 ±  9%      -0.0        0.12 ± 16%  perf-profile.self.cycles-pp._compound_head
      0.32 ±  5%      -0.0        0.28 ±  6%  perf-profile.self.cycles-pp.___perf_sw_event
      0.22 ±  8%      -0.0        0.18 ± 12%  perf-profile.self.cycles-pp.unmap_single_vma
      0.13 ± 14%      -0.0        0.09 ± 19%  perf-profile.self.cycles-pp.__fput
      0.12 ±  8%      -0.0        0.08 ± 20%  perf-profile.self.cycles-pp.alloc_vmap_area
      0.28 ±  6%      -0.0        0.24 ±  4%  perf-profile.self.cycles-pp.mas_wr_node_store
      0.09 ± 12%      -0.0        0.06 ± 46%  perf-profile.self.cycles-pp.xas_start
      0.10 ± 11%      -0.0        0.07 ± 17%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.10 ± 13%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.__schedule
      0.14 ±  9%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.__crypto_xor
      0.07 ± 14%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.mas_spanning_rebalance
      0.10 ±  9%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.folio_test_hugetlb
      0.07 ± 16%      +0.0        0.10 ± 11%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.16 ±  6%      +0.0        0.19 ±  8%  perf-profile.self.cycles-pp.native_sched_clock
      0.11 ± 12%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.1        0.08 ±  9%  perf-profile.self.cycles-pp.trace_module_notify
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.__serpent_enc_blk8_avx
      0.00            +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.sha256_ni_transform
      0.00            +0.1        0.09 ± 15%  perf-profile.self.cycles-pp.llist_reverse_order
      0.00            +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.mpihelp_addmul_1
      0.20 ±  6%      +0.1        0.30 ±  3%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.__camellia_enc_blk_2way
      0.00            +0.1        0.12 ± 20%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.00            +0.1        0.12 ± 11%  perf-profile.self.cycles-pp.__serpent_dec_blk8_avx
      0.00            +0.1        0.13 ± 13%  perf-profile.self.cycles-pp.io_serial_out
      0.00            +0.1        0.14 ± 23%  perf-profile.self.cycles-pp.vprintk_emit
      0.00            +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.native_flush_tlb_global
      0.00            +0.2        0.15 ± 15%  perf-profile.self.cycles-pp.fast_imageblit
      0.88 ±  8%      +0.2        1.04 ±  4%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.00            +0.2        0.18 ±  5%  perf-profile.self.cycles-pp.camellia_dec_blk_2way
      0.34 ±  5%      +0.2        0.56 ±  3%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +0.3        0.29 ± 10%  perf-profile.self.cycles-pp.mpihelp_submul_1
      0.24 ±  6%      +0.4        0.68 ±  4%  perf-profile.self.cycles-pp.__memcpy
      0.00            +0.5        0.46 ± 25%  perf-profile.self.cycles-pp.smp_call_function_many_cond
      0.01 ±223%      +0.8        0.76 ± 21%  perf-profile.self.cycles-pp.delay_tsc
      0.14 ± 23%      +2.4        2.49 ± 11%  perf-profile.self.cycles-pp.io_serial_in
      6.13 ±  2%      +2.8        8.90 ±  2%  perf-profile.self.cycles-pp.intel_idle
      0.05 ± 48%      +4.7        4.71 ±  3%  perf-profile.self.cycles-pp.memcpy_toio




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



View attachment "config-6.4.0-rc2-00017-gb81fac906a8f" of type "text/plain" (158776 bytes)

View attachment "job-script" of type "text/plain" (8858 bytes)

View attachment "job.yaml" of type "text/plain" (6295 bytes)

View attachment "reproduce" of type "text/plain" (533 bytes)

Download attachment "kmsg.xz" of type "application/x-xz" (39464 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ