lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <202307181434.89dc200f-oliver.sang@intel.com>
Date:   Tue, 18 Jul 2023 14:11:16 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Lorenzo Stoakes <lstoakes@...il.com>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        <linux-kernel@...r.kernel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        David Hildenbrand <david@...hat.com>,
        "Kirill A . Shutemov" <kirill@...temov.name>,
        "Peter Zijlstra" <peterz@...radead.org>, Jan Kara <jack@...e.cz>,
        Jason Gunthorpe <jgg@...dia.com>,
        John Hubbard <jhubbard@...dia.com>,
        Mika Penttilä <mpenttil@...hat.com>,
        <linux-mm@...ck.org>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <fengwei.yin@...el.com>,
        <oliver.sang@...el.com>
Subject: [linus:master] [mm/gup]  a6e79df92e:
 stress-ng.vm-splice.vm-splice_calls_per_sec -3.9% regression



Hello,

kernel test robot noticed a -3.9% regression of stress-ng.vm-splice.vm-splice_calls_per_sec on:


commit: a6e79df92e4a371ba7b751a20e3a31fde0f528b8 ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

testcase: stress-ng
test machine: 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory
parameters:

	nr_threads: 1
	disk: 1HDD
	testtime: 60s
	fs: ext4
	class: os
	test: vm-splice
	cpufreq_governor: performance




If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@...el.com>
| Closes: https://lore.kernel.org/oe-lkp/202307181434.89dc200f-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  os/gcc-12/performance/1HDD/ext4/x86_64-rhel-8.3/1/debian-11.1-x86_64-20220510.cgz/lkp-csl-d02/vm-splice/stress-ng/60s

commit: 
  8ac268436e ("mm/gup: disallow FOLL_LONGTERM GUP-nonfast writing to file-backed mappings")
  a6e79df92e ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings")

8ac268436e6d52f3 a6e79df92e4a371ba7b751a20e3 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
 2.215e+09            -0.1%  2.211e+09        cpuidle..time
   3152181            -0.0%    3151772        cpuidle..usage
     94.92 ±  2%      +1.7%      96.52 ±  5%  uptime.boot
      3250            +1.2%       3289 ±  3%  uptime.idle
     22.65 ± 10%      +8.1%      24.48 ± 20%  boot-time.boot
     13.18 ± 16%     +14.4%      15.08 ± 33%  boot-time.dhcp
    750.93 ±  8%      +6.2%     797.50 ± 16%  boot-time.idle
      2.67 ± 41%     -12.5%       2.33 ± 76%  perf-c2c.DRAM.local
     35.83 ± 20%      -3.3%      34.67 ± 18%  perf-c2c.HITM.local
     35.83 ± 20%      -3.3%      34.67 ± 18%  perf-c2c.HITM.total
     96.27            +0.0%      96.30        iostat.cpu.idle
      0.20 ± 25%     -25.7%       0.15 ± 19%  iostat.cpu.iowait
      2.11            +1.5%       2.14        iostat.cpu.system
      1.42            -0.9%       1.41        iostat.cpu.user
     96.28            +0.0       96.33        mpstat.cpu.all.idle%
      0.19 ± 27%      -0.1        0.14 ± 21%  mpstat.cpu.all.iowait%
      0.49            -0.0        0.49        mpstat.cpu.all.irq%
      0.02 ±  4%      -0.0        0.02 ± 12%  mpstat.cpu.all.soft%
      1.59            +0.0        1.61        mpstat.cpu.all.sys%
      1.43            -0.0        1.42        mpstat.cpu.all.usr%
     62.13            -0.2%      62.03        time.elapsed_time
     62.13            -0.2%      62.03        time.elapsed_time.max
      8.00            +0.0%       8.00        time.file_system_inputs
    305.33 ± 40%      +1.6%     310.17 ± 41%  time.involuntary_context_switches
      0.50 ±223%     -33.3%       0.33 ±141%  time.major_page_faults
      7008            +1.4%       7104        time.maximum_resident_set_size
      9342            +0.1%       9348        time.minor_page_faults
      4096            +0.0%       4096        time.page_size
     95.83            +0.2%      96.00        time.percent_of_cpu_this_job_got
     34.39            +0.9%      34.69        time.system_time
     25.34            -1.1%      25.05        time.user_time
    172.50            -1.1%     170.67        time.voluntary_context_switches
     96.00            +0.0%      96.00        vmstat.cpu.id
      1.67 ± 28%     +20.0%       2.00        vmstat.cpu.sy
      1.00            +0.0%       1.00        vmstat.cpu.us
      0.00          -100.0%       0.00        vmstat.cpu.wa
     15.00            +0.0%      15.00        vmstat.io.bi
    540.17 ± 34%     -32.1%     367.00 ±  3%  vmstat.io.bo
      2071            -1.1%       2048        vmstat.memory.buff
   2964356            -0.0%    2964040        vmstat.memory.cache
  1.28e+08            +0.0%   1.28e+08        vmstat.memory.free
      0.00          -100.0%       0.00        vmstat.procs.b
      1.00            +0.0%       1.00        vmstat.procs.r
     30277            +0.3%      30363        vmstat.system.cs
     50967            +0.0%      50971        vmstat.system.in
     62.13            -0.2%      62.03        stress-ng.time.elapsed_time
     62.13            -0.2%      62.03        stress-ng.time.elapsed_time.max
      8.00            +0.0%       8.00        stress-ng.time.file_system_inputs
    305.33 ± 40%      +1.6%     310.17 ± 41%  stress-ng.time.involuntary_context_switches
      0.50 ±223%     -33.3%       0.33 ±141%  stress-ng.time.major_page_faults
      7008            +1.4%       7104        stress-ng.time.maximum_resident_set_size
      9342            +0.1%       9348        stress-ng.time.minor_page_faults
      4096            +0.0%       4096        stress-ng.time.page_size
     95.83            +0.2%      96.00        stress-ng.time.percent_of_cpu_this_job_got
     34.39            +0.9%      34.69        stress-ng.time.system_time
     25.34            -1.1%      25.05        stress-ng.time.user_time
    172.50            -1.1%     170.67        stress-ng.time.voluntary_context_switches
     78220            -3.9%      75148        stress-ng.vm-splice.MB_per_sec_vm-splice_rate
  23767939            -1.2%   23479853        stress-ng.vm-splice.ops
    396132            -1.2%     391330        stress-ng.vm-splice.ops_per_sec
   2355809            -3.9%    2263294        stress-ng.vm-splice.vm-splice_calls_per_sec
    148.00            +0.2%     148.33        turbostat.Avg_MHz
      3.97            +0.0        3.98        turbostat.Busy%
      3730            -0.0%       3729        turbostat.Bzy_MHz
     11627 ± 37%     -33.2%       7769 ± 35%  turbostat.C1
      0.05 ± 23%      -0.0        0.04 ± 21%  turbostat.C1%
    964449            +0.5%     969373        turbostat.C1E
      2.79 ±  2%      +0.0        2.82 ±  3%  turbostat.C1E%
   2161154            -0.2%    2157633        turbostat.C6
     93.46            -0.0       93.43        turbostat.C6%
     95.20            -0.0%      95.16        turbostat.CPU%c1
      0.84 ±  5%      +3.6%       0.87 ±  4%  turbostat.CPU%c6
     40.83 ±  2%      +2.0%      41.67 ±  2%  turbostat.CoreTmp
      0.27 ±  3%      -0.6%       0.27 ±  2%  turbostat.IPC
   3311296            -0.3%    3301795        turbostat.IRQ
     12220 ±  2%      +2.2%      12487 ±  7%  turbostat.POLL
      0.44 ±  8%      -1.9%       0.44 ±  3%  turbostat.Pkg%pc2
     40.17            +0.8%      40.50        turbostat.PkgTmp
     79.53            +0.2%      79.70        turbostat.PkgWatt
      9.92            +0.4%       9.96        turbostat.RAMWatt
      3000            +0.0%       3000        turbostat.TSC_MHz
     55887 ±  2%      -0.6%      55570        meminfo.Active
     53780 ±  2%      -0.6%      53481        meminfo.Active(anon)
      2106            -0.9%       2088        meminfo.Active(file)
     58324 ± 19%      -7.3%      54090 ± 16%  meminfo.AnonHugePages
    305866            +0.3%     306782        meminfo.AnonPages
      2071            -1.2%       2046        meminfo.Buffers
   2872642            -0.0%    2872316        meminfo.Cached
  65801604            +0.0%   65801604        meminfo.CommitLimit
    541659            -0.1%     541187        meminfo.Committed_AS
 1.274e+08            -0.1%  1.272e+08        meminfo.DirectMap1G
   6348117 ±  8%      +2.7%    6519125 ± 15%  meminfo.DirectMap2M
    159306 ±  8%      +2.4%     163061 ± 10%  meminfo.DirectMap4k
      3.33 ± 91%     -40.0%       2.00 ± 28%  meminfo.Dirty
      2048            +0.0%       2048        meminfo.Hugepagesize
    312658            +0.4%     313874        meminfo.Inactive
    312485            +0.4%     313699        meminfo.Inactive(anon)
    172.33            +1.1%     174.17 ±  3%  meminfo.Inactive(file)
     91290            +0.0%      91297        meminfo.KReclaimable
      8731            +0.2%       8750        meminfo.KernelStack
     39762            +0.5%      39970        meminfo.Mapped
 1.272e+08            +0.0%  1.272e+08        meminfo.MemAvailable
  1.28e+08            +0.0%   1.28e+08        meminfo.MemFree
 1.316e+08            +0.0%  1.316e+08        meminfo.MemTotal
   3639982            -0.1%    3636363        meminfo.Memused
     11.00            +0.0%      11.00        meminfo.Mlocked
      3940            -0.1%       3937        meminfo.PageTables
     16434            -0.2%      16402        meminfo.Percpu
     91290            +0.0%      91297        meminfo.SReclaimable
     83130            -0.2%      82932        meminfo.SUnreclaim
     60659 ±  2%      -0.5%      60327        meminfo.Shmem
    174421            -0.1%     174230        meminfo.Slab
   2811768            -0.0%    2811768        meminfo.Unevictable
 3.436e+10            +0.0%  3.436e+10        meminfo.VmallocTotal
     20254            -0.1%      20242        meminfo.VmallocUsed
   3821383            -0.1%    3815898        meminfo.max_used_kB
     62.67 ± 11%      +3.7%      65.00 ± 13%  proc-vmstat.direct_map_level2_splits
      2.50 ± 20%      +6.7%       2.67 ± 35%  proc-vmstat.direct_map_level3_splits
     13444 ±  2%      -0.6%      13370        proc-vmstat.nr_active_anon
    526.33            -0.9%     521.67        proc-vmstat.nr_active_file
     76467            +0.3%      76697        proc-vmstat.nr_anon_pages
     28.00 ± 20%      -7.1%      26.00 ± 16%  proc-vmstat.nr_anon_transparent_hugepages
      2.50 ± 30%     +13.3%       2.83 ± 42%  proc-vmstat.nr_dirtied
      0.33 ±223%    -100.0%       0.00        proc-vmstat.nr_dirty
   3173243            +0.0%    3173331        proc-vmstat.nr_dirty_background_threshold
   6354245            +0.0%    6354422        proc-vmstat.nr_dirty_threshold
    718681            -0.0%     718591        proc-vmstat.nr_file_pages
  31990830            +0.0%   31991721        proc-vmstat.nr_free_pages
     78124            +0.4%      78428        proc-vmstat.nr_inactive_anon
     42.83            +1.6%      43.50 ±  3%  proc-vmstat.nr_inactive_file
      8731            +0.2%       8750        proc-vmstat.nr_kernel_stack
      9942            +0.5%       9993        proc-vmstat.nr_mapped
      2.00            +0.0%       2.00        proc-vmstat.nr_mlock
    984.67            -0.1%     984.00        proc-vmstat.nr_page_table_pages
     15167 ±  2%      -0.6%      15082        proc-vmstat.nr_shmem
     22822            +0.0%      22824        proc-vmstat.nr_slab_reclaimable
     20782            -0.2%      20732        proc-vmstat.nr_slab_unreclaimable
    702941            -0.0%     702941        proc-vmstat.nr_unevictable
      1.33 ± 55%     +50.0%       2.00 ± 70%  proc-vmstat.nr_written
     13444 ±  2%      -0.6%      13370        proc-vmstat.nr_zone_active_anon
    526.33            -0.9%     521.67        proc-vmstat.nr_zone_active_file
     78124            +0.4%      78428        proc-vmstat.nr_zone_inactive_anon
     42.83            +1.6%      43.50 ±  3%  proc-vmstat.nr_zone_inactive_file
    702941            -0.0%     702941        proc-vmstat.nr_zone_unevictable
      0.00          -100.0%       0.00        proc-vmstat.nr_zone_write_pending
    240485 ±  2%      -0.8%     238484        proc-vmstat.numa_hit
      0.00          -100.0%       0.00        proc-vmstat.numa_interleave
    237924            +0.2%     238423        proc-vmstat.numa_local
     20711            -0.7%      20574        proc-vmstat.pgactivate
      0.00          -100.0%       0.00        proc-vmstat.pgalloc_dma32
    269051            +0.2%     269719        proc-vmstat.pgalloc_normal
    186850            +0.6%     187979        proc-vmstat.pgfault
    235433            +0.4%     236396        proc-vmstat.pgfree
      1008            +0.0%       1008        proc-vmstat.pgpgin
     33545 ± 33%     -31.6%      22950 ±  3%  proc-vmstat.pgpgout
      7555 ±  3%      -0.2%       7541        proc-vmstat.pgreuse
     46.33 ±  5%      -2.2%      45.33 ±  5%  proc-vmstat.thp_collapse_alloc
      0.00       +1.7e+101%       0.17 ±223%  proc-vmstat.thp_deferred_split_page
      1.67 ± 89%     +80.0%       3.00 ± 54%  proc-vmstat.thp_fault_alloc
      0.00       +1.7e+101%       0.17 ±223%  proc-vmstat.thp_split_pmd
      0.00          -100.0%       0.00        proc-vmstat.thp_zero_page_alloc
    145.50           +14.8%     167.00 ± 28%  proc-vmstat.unevictable_pgs_culled
      3.00            +0.0%       3.00        proc-vmstat.unevictable_pgs_mlocked
      2.00 ± 70%     +25.0%       2.50 ± 44%  proc-vmstat.unevictable_pgs_munlocked
      0.56 ±  6%      -0.5%       0.56 ±  2%  perf-stat.i.MPKI
 7.615e+08            +2.4%  7.796e+08        perf-stat.i.branch-instructions
      1.60            -0.1        1.54        perf-stat.i.branch-miss-rate%
  13526797            -1.2%   13370260        perf-stat.i.branch-misses
      7.09 ±  6%      -0.2        6.84 ±  5%  perf-stat.i.cache-miss-rate%
    391404 ±  3%      +0.7%     394008 ±  2%  perf-stat.i.cache-misses
   2549946 ±  5%      +2.2%    2605110 ±  2%  perf-stat.i.cache-references
     31277            +0.0%      31282        perf-stat.i.context-switches
      1.23            -1.0%       1.22        perf-stat.i.cpi
     36001            +0.0%      36001        perf-stat.i.cpu-clock
 4.932e+09            +0.5%  4.958e+09        perf-stat.i.cpu-cycles
     51.47            +0.0%      51.49 ±  2%  perf-stat.i.cpu-migrations
    287546 ± 19%      -0.4%     286416 ± 18%  perf-stat.i.cycles-between-cache-misses
      0.24 ± 24%      +0.0        0.24 ± 28%  perf-stat.i.dTLB-load-miss-rate%
   2615899 ± 24%      +3.8%    2714646 ± 28%  perf-stat.i.dTLB-load-misses
 1.134e+09            +0.1%  1.135e+09        perf-stat.i.dTLB-loads
      0.39 ± 20%      -0.1        0.26 ± 14%  perf-stat.i.dTLB-store-miss-rate%
   2504597 ± 20%     -32.7%    1685000 ± 15%  perf-stat.i.dTLB-store-misses
 6.341e+08            +1.0%  6.406e+08        perf-stat.i.dTLB-stores
     93.04            -0.3       92.79        perf-stat.i.iTLB-load-miss-rate%
   8340558 ±  2%      -2.6%    8125806        perf-stat.i.iTLB-load-misses
    521392            +2.1%     532578        perf-stat.i.iTLB-loads
 4.094e+09            +1.4%  4.151e+09        perf-stat.i.instructions
    675.38            +4.3%     704.63        perf-stat.i.instructions-per-iTLB-miss
      0.83            +0.9%       0.83        perf-stat.i.ipc
      0.03 ± 91%     +16.7%       0.04 ± 80%  perf-stat.i.major-faults
      0.14            +0.5%       0.14        perf-stat.i.metric.GHz
    156.53 ± 11%     -13.3%     135.70 ±  5%  perf-stat.i.metric.K/sec
     70.25            +1.0%      70.97        perf-stat.i.metric.M/sec
      1496            +0.4%       1502        perf-stat.i.minor-faults
     45238 ±  8%      -0.5%      45001 ±  8%  perf-stat.i.node-loads
     21419 ± 11%      +7.3%      22978 ±  9%  perf-stat.i.node-stores
      1496            +0.4%       1502        perf-stat.i.page-faults
     36001            +0.0%      36001        perf-stat.i.task-clock
      0.62 ±  5%      +0.8%       0.63 ±  2%  perf-stat.overall.MPKI
      1.78            -0.1        1.72        perf-stat.overall.branch-miss-rate%
     15.39 ±  4%      -0.3       15.13 ±  2%  perf-stat.overall.cache-miss-rate%
      1.20            -0.8%       1.19        perf-stat.overall.cpi
     12612 ±  3%      -0.2%      12582 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.23 ± 24%      +0.0        0.24 ± 28%  perf-stat.overall.dTLB-load-miss-rate%
      0.39 ± 20%      -0.1        0.26 ± 14%  perf-stat.overall.dTLB-store-miss-rate%
     94.11            -0.3       93.85        perf-stat.overall.iTLB-load-miss-rate%
    491.33 ±  3%      +4.0%     510.96        perf-stat.overall.instructions-per-iTLB-miss
      0.83            +0.9%       0.84        perf-stat.overall.ipc
 7.495e+08            +2.4%  7.673e+08        perf-stat.ps.branch-instructions
  13315576            -1.2%   13160202        perf-stat.ps.branch-misses
    385407 ±  3%      +0.7%     387953 ±  2%  perf-stat.ps.cache-misses
   2510302 ±  5%      +2.2%    2564508 ±  2%  perf-stat.ps.cache-references
     30781            +0.0%      30785        perf-stat.ps.context-switches
     35430            +0.0%      35430        perf-stat.ps.cpu-clock
 4.854e+09            +0.5%  4.879e+09        perf-stat.ps.cpu-cycles
     50.66            +0.0%      50.67 ±  2%  perf-stat.ps.cpu-migrations
   2574365 ± 24%      +3.8%    2671514 ± 28%  perf-stat.ps.dTLB-load-misses
 1.116e+09            +0.1%  1.117e+09        perf-stat.ps.dTLB-loads
   2464819 ± 20%     -32.7%    1658231 ± 15%  perf-stat.ps.dTLB-store-misses
  6.24e+08            +1.0%  6.305e+08        perf-stat.ps.dTLB-stores
   8208113 ±  2%      -2.6%    7996727        perf-stat.ps.iTLB-load-misses
    513124            +2.1%     524127        perf-stat.ps.iTLB-loads
 4.029e+09            +1.4%  4.085e+09        perf-stat.ps.instructions
      0.03 ± 91%     +16.7%       0.04 ± 80%  perf-stat.ps.major-faults
      1472            +0.4%       1478        perf-stat.ps.minor-faults
     44541 ±  8%      -0.5%      44305 ±  8%  perf-stat.ps.node-loads
     21084 ± 10%      +7.3%      22620 ±  9%  perf-stat.ps.node-stores
      1472            +0.4%       1479        perf-stat.ps.page-faults
     35430            +0.0%      35430        perf-stat.ps.task-clock
 2.543e+11            +1.4%  2.577e+11        perf-stat.total.instructions
     12.88 ±223%      -6.3%      12.07 ±186%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    463.73 ±223%      -6.3%     434.60 ±186%  sched_debug.cfs_rq:/.MIN_vruntime.max
      0.00            +0.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.min
     76.21 ±223%      -6.3%      71.42 ±186%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.38 ±  8%      -3.6%       0.37 ± 13%  sched_debug.cfs_rq:/.h_nr_running.avg
      1.33 ± 55%     -12.5%       1.17 ± 31%  sched_debug.cfs_rq:/.h_nr_running.max
      0.51 ± 11%      -4.0%       0.49 ±  5%  sched_debug.cfs_rq:/.h_nr_running.stddev
     27457 ± 79%     -40.7%      16276 ± 56%  sched_debug.cfs_rq:/.load.avg
    576320 ±134%     -61.8%     220048 ±168%  sched_debug.cfs_rq:/.load.max
     99666 ±123%     -56.8%      43065 ±133%  sched_debug.cfs_rq:/.load.stddev
    559.87 ± 18%      -2.5%     545.69 ± 18%  sched_debug.cfs_rq:/.load_avg.avg
     14785 ± 20%      -6.7%      13800 ± 26%  sched_debug.cfs_rq:/.load_avg.max
      0.17 ±223%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.min
      2423 ± 20%      -6.5%       2267 ± 26%  sched_debug.cfs_rq:/.load_avg.stddev
     12.94 ±223%      -6.7%      12.07 ±186%  sched_debug.cfs_rq:/.max_vruntime.avg
    465.73 ±223%      -6.7%     434.60 ±186%  sched_debug.cfs_rq:/.max_vruntime.max
      0.00            +0.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.min
     76.54 ±223%      -6.7%      71.42 ±186%  sched_debug.cfs_rq:/.max_vruntime.stddev
      3264 ±  2%      -1.7%       3209 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
     35040 ± 10%      +2.2%      35818 ± 10%  sched_debug.cfs_rq:/.min_vruntime.max
    354.23 ± 15%      +3.0%     364.73 ± 12%  sched_debug.cfs_rq:/.min_vruntime.min
      6667 ±  2%      +0.0%       6670 ±  3%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.38 ±  8%      -4.8%       0.37 ± 13%  sched_debug.cfs_rq:/.nr_running.avg
      1.33 ± 55%     -25.0%       1.00        sched_debug.cfs_rq:/.nr_running.max
      0.51 ± 11%      -6.0%       0.48 ±  3%  sched_debug.cfs_rq:/.nr_running.stddev
     61.00 ± 49%     +22.8%      74.89 ± 27%  sched_debug.cfs_rq:/.removed.load_avg.avg
      1024            +0.0%       1024        sched_debug.cfs_rq:/.removed.load_avg.max
    232.88 ± 23%     +12.5%     261.99 ± 12%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     19.46 ± 78%     +11.2%      21.63 ± 25%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    385.33 ± 45%     +11.9%     431.17 ± 20%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     78.12 ± 53%     +10.9%      86.66 ± 22%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     19.45 ± 78%     +10.9%      21.56 ± 24%  sched_debug.cfs_rq:/.removed.util_avg.avg
    385.33 ± 45%     +11.9%     431.00 ± 20%  sched_debug.cfs_rq:/.removed.util_avg.max
     78.11 ± 53%     +10.7%      86.47 ± 22%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    627.54 ±  4%      -3.1%     607.93 ±  7%  sched_debug.cfs_rq:/.runnable_avg.avg
      1724 ± 10%     -15.4%       1458 ±  6%  sched_debug.cfs_rq:/.runnable_avg.max
      2.67 ± 91%     +12.5%       3.00 ±103%  sched_debug.cfs_rq:/.runnable_avg.min
    378.19 ±  8%      -7.5%     349.81 ±  4%  sched_debug.cfs_rq:/.runnable_avg.stddev
      0.06 ±223%    -100.0%       0.00        sched_debug.cfs_rq:/.spread.avg
      2.00 ±223%    -100.0%       0.00        sched_debug.cfs_rq:/.spread.max
      0.33 ±223%    -100.0%       0.00        sched_debug.cfs_rq:/.spread.stddev
    -23087           +19.8%     -27668        sched_debug.cfs_rq:/.spread0.avg
      8724 ± 90%     -43.0%       4972 ±139%  sched_debug.cfs_rq:/.spread0.max
    -26014           +17.4%     -30531        sched_debug.cfs_rq:/.spread0.min
      6681 ±  2%      -0.0%       6680 ±  3%  sched_debug.cfs_rq:/.spread0.stddev
    624.48 ±  4%      -3.0%     605.63 ±  7%  sched_debug.cfs_rq:/.util_avg.avg
      1719 ± 10%     -15.5%       1452 ±  6%  sched_debug.cfs_rq:/.util_avg.max
      2.17 ± 62%     +38.5%       3.00 ±103%  sched_debug.cfs_rq:/.util_avg.min
    377.21 ±  8%      -7.4%     349.32 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
     48.43 ± 43%      +1.9%      49.36 ± 21%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    493.83 ±  3%      -0.6%     491.00 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.max
    133.78 ± 19%      +4.7%     140.10 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    448176 ±  5%      -3.8%     431289 ±  3%  sched_debug.cpu.avg_idle.avg
    938898 ±  9%      -0.4%     935316 ±  9%  sched_debug.cpu.avg_idle.max
      3006 ± 47%     +38.5%       4163 ± 50%  sched_debug.cpu.avg_idle.min
    197844 ± 11%      -0.5%     196827 ±  4%  sched_debug.cpu.avg_idle.stddev
     31086 ±  7%      +5.4%      32777 ± 15%  sched_debug.cpu.clock.avg
     31087 ±  7%      +5.4%      32778 ± 15%  sched_debug.cpu.clock.max
     31084 ±  7%      +5.4%      32776 ± 15%  sched_debug.cpu.clock.min
      0.78 ± 12%     -11.1%       0.69 ± 10%  sched_debug.cpu.clock.stddev
     30776 ±  6%      +4.5%      32170 ± 13%  sched_debug.cpu.clock_task.avg
     30893 ±  6%      +4.9%      32391 ± 13%  sched_debug.cpu.clock_task.max
     28933            -0.8%      28692        sched_debug.cpu.clock_task.min
    314.73 ± 99%     +87.7%     590.72 ±124%  sched_debug.cpu.clock_task.stddev
      1108 ± 10%      +2.7%       1138 ± 11%  sched_debug.cpu.curr->pid.avg
      3289            -0.1%       3287        sched_debug.cpu.curr->pid.max
      1513 ±  2%      +0.6%       1522 ±  3%  sched_debug.cpu.curr->pid.stddev
    500000            +0.0%     500000        sched_debug.cpu.max_idle_balance_cost.avg
    500000            +0.0%     500000        sched_debug.cpu.max_idle_balance_cost.max
    500000            +0.0%     500000        sched_debug.cpu.max_idle_balance_cost.min
      4294            +0.0%       4294        sched_debug.cpu.next_balance.avg
      4294            +0.0%       4294        sched_debug.cpu.next_balance.max
      4294            +0.0%       4294        sched_debug.cpu.next_balance.min
      0.00 ± 16%     +22.1%       0.00 ± 30%  sched_debug.cpu.next_balance.stddev
      0.38 ±  7%      -3.6%       0.37 ±  7%  sched_debug.cpu.nr_running.avg
      1.50 ± 33%      +0.0%       1.50 ± 50%  sched_debug.cpu.nr_running.max
      0.51 ±  5%      +0.8%       0.52 ± 10%  sched_debug.cpu.nr_running.stddev
      5027            -0.2%       5019        sched_debug.cpu.nr_switches.avg
     37965 ± 21%     -19.3%      30629 ± 16%  sched_debug.cpu.nr_switches.max
    275.33 ± 26%      -3.4%     266.00 ± 16%  sched_debug.cpu.nr_switches.min
      7864 ±  7%      -7.2%       7295 ±  7%  sched_debug.cpu.nr_switches.stddev
 2.346e+09 ±  8%      -9.3%  2.128e+09 ±  8%  sched_debug.cpu.nr_uninterruptible.avg
 4.295e+09            +0.0%  4.295e+09        sched_debug.cpu.nr_uninterruptible.max
      0.17 ±223%    -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.min
  2.13e+09            +0.5%   2.14e+09        sched_debug.cpu.nr_uninterruptible.stddev
     31085 ±  7%      +5.4%      32776 ± 15%  sched_debug.cpu_clk
    996147            +0.0%     996147        sched_debug.dl_rq:.dl_bw->bw.avg
    996147            +0.0%     996147        sched_debug.dl_rq:.dl_bw->bw.max
    996147            +0.0%     996147        sched_debug.dl_rq:.dl_bw->bw.min
 4.295e+09            +0.0%  4.295e+09        sched_debug.jiffies
     31034 ±  7%      +5.5%      32725 ± 15%  sched_debug.ktime
    950.00            +0.0%     950.00        sched_debug.rt_rq:.rt_runtime.avg
    950.00            +0.0%     950.00        sched_debug.rt_rq:.rt_runtime.max
    950.00            +0.0%     950.00        sched_debug.rt_rq:.rt_runtime.min
     29328            -0.8%      29103        sched_debug.sched_clk
      1.00            +0.0%       1.00        sched_debug.sched_clock_stable()
  58611259            +0.0%   58611259        sched_debug.sysctl_sched.sysctl_sched_features
      0.75            +0.0%       0.75        sched_debug.sysctl_sched.sysctl_sched_idle_min_granularity
     24.00            +0.0%      24.00        sched_debug.sysctl_sched.sysctl_sched_latency
      3.00            +0.0%       3.00        sched_debug.sysctl_sched.sysctl_sched_min_granularity
      1.00            +0.0%       1.00        sched_debug.sysctl_sched.sysctl_sched_tunable_scaling
      4.00            +0.0%       4.00        sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages.pte_alloc_one.__do_fault.do_cow_fault
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_node_trace.__get_vm_area_node.__vmalloc_node_range
      0.00 ±223%     -33.3%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.copy_strings.isra.0.do_execveat_common
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.[unknown]
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.do_brk_flags.__do_sys_brk.do_syscall_64
      0.00 ±223%      +0.0%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.mmap_region.do_mmap.vm_mmap_pgoff
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vma_prepare.__split_vma.mprotect_fixup
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.dput.step_into.link_path_walk.part
      0.00 ±223%      +0.0%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.jbd2_journal_commit_transaction.kjournald2.kthread.ret_from_fork
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.getname_flags.part.0
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.mmap_region
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region.do_mmap
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.mmput.exit_mm.do_exit.do_group_exit
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.futex_exec_release.exec_mm_release.exec_mmap
      0.00        +1.5e+99%       0.00 ±152%  perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock_killable.pcpu_alloc.__percpu_counter_init.mm_init
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      0.00          +5e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
      0.00 ±223%     -80.0%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput
      0.00          +5e+98%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.wp_page_copy.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      0.00           +11.1%       0.00 ± 14%  perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 13%      +0.0%       0.01 ± 17%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.00            +0.0%       0.00        perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      0.00            +0.0%       0.00        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00            +0.0%       0.00        perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.00 ±223%    +500.0%       0.00 ±100%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.00 ± 99%      +0.0%       0.00 ± 99%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.00 ± 66%     +27.3%       0.00 ± 53%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.00 ±223%    +100.0%       0.00 ±141%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00 ± 73%     -68.8%       0.00 ±223%  perf-sched.sch_delay.avg.ms.io_schedule.bit_wait_io.__wait_on_bit.out_of_line_wait_on_bit
      0.00 ±142%     -55.6%       0.00 ±223%  perf-sched.sch_delay.avg.ms.kjournald2.kthread.ret_from_fork
      0.00 ± 19%     -13.3%       0.00 ± 17%  perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.00            +0.0%       0.00        perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.00            +0.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.00 ±  9%     +21.7%       0.00 ± 23%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select
      0.00 ± 14%      -4.8%       0.00 ± 14%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.00 ± 72%     +23.1%       0.00 ± 46%  perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__flush_work.isra.0
      0.00            +0.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.00 ± 79%     +26.7%       0.00 ± 46%  perf-sched.sch_delay.avg.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
      0.00 ±107%     +66.7%       0.00 ± 72%  perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.submit_bio_wait
      0.00 ± 13%      +0.0%       0.00 ± 24%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.00 ±223%    +100.0%       0.00 ±141%  perf-sched.sch_delay.avg.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
      0.00 ± 17%      +0.0%       0.00 ± 17%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.00 ± 12%     +18.2%       0.00 ± 28%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 19%      +0.0%       0.01 ± 19%  perf-sched.sch_delay.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read
      0.00            +0.0%       0.00        perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
      0.00 ±223%     +20.0%       0.00 ±223%  perf-sched.sch_delay.avg.ms.wait_transaction_locked.add_transaction_credits.start_this_handle.jbd2__journal_start
      0.00            +0.0%       0.00        perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.pte_alloc_one.__do_fault.do_cow_fault
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_node_trace.__get_vm_area_node.__vmalloc_node_range
      0.00 ±  7%     -10.3%       0.00 ± 10%  perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      0.00 ±223%     +33.3%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.copy_strings.isra.0.do_execveat_common
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.[unknown]
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.do_brk_flags.__do_sys_brk.do_syscall_64
      0.00 ±223%      +0.0%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.mmap_region.do_mmap.vm_mmap_pgoff
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.down_write.vma_prepare.__split_vma.mprotect_fixup
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.dput.step_into.link_path_walk.part
      0.00 ±223%      +0.0%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.jbd2_journal_commit_transaction.kjournald2.kthread.ret_from_fork
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.getname_flags.part.0
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.mmap_region
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region.do_mmap
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.mmput.exit_mm.do_exit.do_group_exit
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.futex_exec_release.exec_mm_release.exec_mmap
      0.00        +1.5e+99%       0.00 ±152%  perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock_killable.pcpu_alloc.__percpu_counter_init.mm_init
      0.00        +3.3e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      0.00          +5e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
      0.00 ±223%     -80.0%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput
      0.00          +5e+98%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.wp_page_copy.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.00 ±223%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      0.00 ± 16%      +8.0%       0.00 ± 16%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 17%     -10.8%       0.01 ± 17%  perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.15 ±115%     -70.3%       0.05 ± 77%  perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      0.01 ± 29%      -5.1%       0.01 ± 30%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ±  6%      +2.9%       0.01        perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.00 ±223%    +500.0%       0.00 ±100%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.00 ± 82%     -20.0%       0.00 ± 70%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.00 ± 67%     +69.2%       0.00 ± 78%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.00 ±104%     -14.3%       0.00 ±152%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00 ± 71%     -68.4%       0.00 ±223%  perf-sched.sch_delay.max.ms.io_schedule.bit_wait_io.__wait_on_bit.out_of_line_wait_on_bit
      0.00 ±142%     -55.6%       0.00 ±223%  perf-sched.sch_delay.max.ms.kjournald2.kthread.ret_from_fork
      0.01 ± 27%      +4.1%       0.01 ± 39%  perf-sched.sch_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.00 ± 11%      +0.0%       0.00 ± 11%  perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.00 ± 16%      +7.4%       0.00 ± 14%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.01           +36.7%       0.01 ± 43%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select
      0.01 ± 47%     -31.1%       0.01 ± 26%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.00 ± 72%     +23.1%       0.00 ± 46%  perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__flush_work.isra.0
      0.00 ± 50%     -35.7%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.00 ± 79%     +26.7%       0.00 ± 46%  perf-sched.sch_delay.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
      0.00 ±107%     +66.7%       0.00 ± 72%  perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.submit_bio_wait
      0.00 ± 12%      +4.5%       0.00 ± 17%  perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.00 ±223%    +100.0%       0.00 ±141%  perf-sched.sch_delay.max.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
      0.00 ± 23%      +7.1%       0.01 ± 16%  perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.01 ± 18%   +1600.0%       0.19 ±211%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 16%      -2.9%       0.01 ± 20%  perf-sched.sch_delay.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
      0.00 ± 17%      -7.7%       0.00        perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
      0.00 ±223%     +20.0%       0.00 ±223%  perf-sched.sch_delay.max.ms.wait_transaction_locked.add_transaction_credits.start_this_handle.jbd2__journal_start
      0.01 ± 56%     +57.3%       0.02 ± 35%  perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.00            +0.0%       0.00        perf-sched.total_sch_delay.average.ms
      0.15 ±115%     +38.3%       0.21 ±183%  perf-sched.total_sch_delay.max.ms
      4.96 ±  5%      -0.2%       4.95 ±  4%  perf-sched.total_wait_and_delay.average.ms
     78048            +0.1%      78092        perf-sched.total_wait_and_delay.count.ms
      3689 ± 15%     -12.7%       3222 ± 10%  perf-sched.total_wait_and_delay.max.ms
      4.96 ±  5%      -0.2%       4.95 ±  4%  perf-sched.total_wait_time.average.ms
      3689 ± 15%     -12.7%       3222 ± 10%  perf-sched.total_wait_time.max.ms
     24.63 ±223%     +54.1%      37.96 ±223%  perf-sched.wait_and_delay.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
     18.22 ±  4%      +2.1%      18.61 ±  3%  perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
    127.99 ±223%     -83.3%      21.41 ±223%  perf-sched.wait_and_delay.avg.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats
     91.96 ± 71%     -21.3%      72.38 ±100%  perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      0.00       +1.7e+104%     166.83 ±223%  perf-sched.wait_and_delay.avg.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
      0.00       +1.7e+104%     166.69 ±223%  perf-sched.wait_and_delay.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
    799.93            -0.0%     799.93        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.11 ±  4%      +1.7%       0.11        perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
    289.17 ±  7%      -1.1%     285.86 ± 10%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00       +1.7e+104%     166.69 ±223%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
     10.42 ±223%    +300.2%      41.69 ±223%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
    106.44 ±181%     -89.3%      11.36 ±223%  perf-sched.wait_and_delay.avg.ms.kjournald2.kthread.ret_from_fork
    326.27 ±  5%      -3.9%     313.39 ±  7%  perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.00       +8.3e+101%       0.83 ±223%  perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    307.78 ±  2%      -1.7%     302.70 ±  2%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    390.51 ±  4%      -3.5%     376.68        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.83 ±223%    +650.5%       6.20 ±118%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
    468.47 ±144%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.submit_bio_wait
    453.60            -0.0%     453.60        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
     84.17 ±  5%      -2.4%      82.13 ±  9%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    663.27 ±  3%      -1.3%     654.87 ±  4%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    494.15 ±  4%      -5.1%     469.05 ±  7%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.17 ±223%      +0.0%       0.17 ±223%  perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
    204.00 ±  8%      +2.9%     209.83 ±  6%  perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      0.17 ±223%    +100.0%       0.33 ±223%  perf-sched.wait_and_delay.count.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats
      1.00 ± 81%     -33.3%       0.67 ±111%  perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      0.00       +1.7e+101%       0.17 ±223%  perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
      0.00       +1.7e+101%       0.17 ±223%  perf-sched.wait_and_delay.count.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
     10.00            +0.0%      10.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     76912            +0.0%      76930        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     49.00            +0.0%      49.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00       +1.7e+101%       0.17 ±223%  perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      2.67 ±223%     -50.0%       1.33 ±223%  perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.33 ±141%     -50.0%       0.17 ±223%  perf-sched.wait_and_delay.count.kjournald2.kthread.ret_from_fork
    122.00 ±  5%      +4.0%     126.83 ±  7%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.00       +1.7e+102%       1.67 ±223%  perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
     54.50            -0.3%      54.33        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
     12.33 ±  7%      +5.4%      13.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.17 ±223%    +300.0%       0.67 ± 70%  perf-sched.wait_and_delay.count.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
      0.67 ±141%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.__wait_for_common.submit_bio_wait
     10.00            +0.0%      10.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
     58.00 ±  4%      +2.9%      59.67 ±  7%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    283.17 ±  9%      +3.4%     292.83 ±  6%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
    188.67 ±  4%      +0.4%     189.33 ±  7%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
     24.63 ±223%     +54.1%      37.96 ±223%  perf-sched.wait_and_delay.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      1000            +0.0%       1000        perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
    127.99 ±223%     -66.5%      42.82 ±223%  perf-sched.wait_and_delay.max.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats
     95.47 ± 70%     -24.2%      72.39 ±100%  perf-sched.wait_and_delay.max.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      0.00       +1.7e+104%     166.83 ±223%  perf-sched.wait_and_delay.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
      0.00       +1.7e+104%     166.69 ±223%  perf-sched.wait_and_delay.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
    999.92            -0.0%     999.92        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1000            -0.0%       1000        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      1668 ± 56%     -20.0%       1335 ± 35%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00       +1.7e+104%     166.69 ±223%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
    166.68 ±223%      +0.1%     166.83 ±223%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
    106.44 ±181%     -89.3%      11.36 ±223%  perf-sched.wait_and_delay.max.ms.kjournald2.kthread.ret_from_fork
      1001            +0.0%       1001        perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.00       +8.3e+101%       0.83 ±223%  perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      1000            +0.0%       1000        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    527.77 ±  7%      -5.3%     499.99        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.83 ±223%    +650.5%       6.20 ±118%  perf-sched.wait_and_delay.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
    936.93 ±144%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.submit_bio_wait
    504.50            -0.0%     504.34        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    948.17            -0.1%     947.17        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      2667 ± 27%      +6.2%       2831 ± 23%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      3647 ± 17%     -16.6%       3042 ±  2%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
     24.63 ±223%     +54.1%      37.96 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      0.00       +8.4e+100%       0.08 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_node_trace.__get_vm_area_node.__vmalloc_node_range
      0.01 ±223%     +16.1%       0.01 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.__splice_from_pipe.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.22 ±  4%      +2.1%      18.61 ±  3%  perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
    127.99 ±223%     -83.3%      21.41 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats
     91.96 ± 71%     -21.3%      72.38 ±100%  perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      0.00 ±102%     +40.0%       0.00 ±141%  perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00       +1.7e+104%     166.83 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
      0.01 ±136%     -38.3%       0.00 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write
      0.00 ±182%     +77.8%       0.01 ±199%  perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.splice_from_pipe.do_splice.__do_splice
      0.00       +1.7e+104%     166.69 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
      0.08 ±223%     -88.0%       0.01 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
    799.93            -0.0%     799.92        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.42 ±  9%     +35.2%       0.57 ± 37%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.11 ±  4%      +1.7%       0.11        perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
    289.17 ±  7%      -1.1%     285.86 ± 10%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.20 ±  2%      -1.3%       0.20 ±  2%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.00       +8.5e+100%       0.09 ±223%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.03 ± 54%     +13.8%       0.03 ± 69%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.02 ±107%  +9.2e+05%     166.71 ±223%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
     10.43 ±223%    +299.8%      41.70 ±223%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.75 ±134%     -97.6%       0.02 ±223%  perf-sched.wait_time.avg.ms.io_schedule.bit_wait_io.__wait_on_bit.out_of_line_wait_on_bit
    106.43 ±181%     -89.3%      11.36 ±223%  perf-sched.wait_time.avg.ms.kjournald2.kthread.ret_from_fork
    326.26 ±  5%      -3.9%     313.39 ±  7%  perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      4.03 ±  9%      +6.4%       4.29 ± 14%  perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    307.78 ±  2%      -1.7%     302.70 ±  2%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    390.51 ±  4%      -3.5%     376.68        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.00 ±223%     +83.3%       0.00 ±142%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr
      0.28            -0.7%       0.28 ±  2%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      3.27 ± 70%    +115.5%       7.05 ± 97%  perf-sched.wait_time.avg.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
    468.47 ±144%    -100.0%       0.00 ±223%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.submit_bio_wait
    453.60            -0.0%     453.60        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.00 ± 76%     +25.0%       0.00 ± 82%  perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
     84.16 ±  5%      -2.4%      82.13 ±  9%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    663.27 ±  3%      -1.3%     654.86 ±  4%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.42 ±  9%     +35.1%       0.57 ± 36%  perf-sched.wait_time.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read
      0.00        +1.7e+98%       0.00 ±223%  perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
    494.15 ±  4%      -5.1%     469.04 ±  7%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
     24.63 ±223%     +54.1%      37.96 ±223%  perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      0.00       +8.4e+100%       0.08 ±223%  perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_node_trace.__get_vm_area_node.__vmalloc_node_range
      0.01 ±223%     +16.1%       0.01 ±223%  perf-sched.wait_time.max.ms.__cond_resched.__splice_from_pipe.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1000            +0.0%       1000        perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
    127.99 ±223%     -66.5%      42.82 ±223%  perf-sched.wait_time.max.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats
     95.47 ± 70%     -24.2%      72.39 ±100%  perf-sched.wait_time.max.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      0.00 ±102%     +53.3%       0.00 ±131%  perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00       +1.7e+104%     166.83 ±223%  perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
      0.01 ±136%     -38.3%       0.00 ±223%  perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write
      0.01 ±197%      +6.7%       0.01 ±199%  perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.splice_from_pipe.do_splice.__do_splice
      0.00       +1.7e+104%     166.69 ±223%  perf-sched.wait_time.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
      0.08 ±223%     -88.0%       0.01 ±223%  perf-sched.wait_time.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
    999.92            -0.0%     999.92        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.84 ±  9%     +35.2%       1.13 ± 37%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000            -0.0%       1000        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      1668 ± 56%     -20.0%       1335 ± 35%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.56 ±  2%      +3.7%       0.58 ±  3%  perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.00       +8.5e+100%       0.09 ±223%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.05 ± 28%      +0.0%       0.05 ± 29%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.02 ±110%  +7.5e+05%     166.72 ±223%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
    166.70 ±223%      +0.1%     166.86 ±223%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      1.51 ±134%     -97.6%       0.04 ±223%  perf-sched.wait_time.max.ms.io_schedule.bit_wait_io.__wait_on_bit.out_of_line_wait_on_bit
    106.43 ±181%     -89.3%      11.36 ±223%  perf-sched.wait_time.max.ms.kjournald2.kthread.ret_from_fork
      1001            +0.0%       1001        perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      4.83 ±  7%      +0.0%       4.83 ±  7%  perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      1000            +0.0%       1000        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    527.77 ±  7%      -5.3%     499.99        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.00 ±223%     +83.3%       0.00 ±142%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr
      0.62 ±  2%      +1.5%       0.63 ±  2%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      3.32 ± 70%    +112.3%       7.05 ± 97%  perf-sched.wait_time.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
    936.93 ±144%    -100.0%       0.00 ±223%  perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.submit_bio_wait
    504.50            -0.0%     504.33        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.00 ± 76%     +25.0%       0.00 ± 82%  perf-sched.wait_time.max.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
    948.17            -0.1%     947.17        perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      2667 ± 27%      +6.2%       2831 ± 23%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.85 ±  9%     +35.1%       1.15 ± 36%  perf-sched.wait_time.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
      0.00        +2.8e+99%       0.00 ±100%  perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
      3647 ± 17%     -16.6%       3042 ±  2%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
     12.97 ±  9%      -2.2       10.76 ± 30%  perf-profile.calltrace.cycles-pp.write
     11.29 ±  8%      -1.8        9.47 ± 30%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     11.06 ±  8%      -1.8        9.27 ± 30%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     10.29 ±  8%      -1.8        8.53 ± 30%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     10.68 ±  9%      -1.7        9.00 ± 30%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     27.88 ± 12%      -1.7       26.23 ± 21%  perf-profile.calltrace.cycles-pp.vmsplice
      9.11 ± 10%      -1.6        7.46 ± 30%  perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.95 ± 13%      -1.5       23.42 ± 20%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.vmsplice
     24.52 ± 13%      -1.5       23.06 ± 20%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.vmsplice
     23.77 ± 13%      -1.4       22.35 ± 20%  perf-profile.calltrace.cycles-pp.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe.vmsplice
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
      2.61 ± 45%      -1.2        1.38 ±100%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
      6.63 ± 10%      -1.2        5.48 ± 31%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.vfs_write.ksys_write.do_syscall_64
      6.50 ±  9%      -1.1        5.38 ± 31%  perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write.ksys_write
      6.05 ±  9%      -1.1        4.95 ± 31%  perf-profile.calltrace.cycles-pp.rep_movs_alternative.copyin._copy_from_iter.copy_page_from_iter.pipe_write
      6.26 ±  9%      -1.1        5.18 ± 31%  perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write
      2.30 ± 45%      -1.1        1.23 ±101%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      2.19 ± 46%      -1.0        1.16 ±101%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      6.59 ±  7%      -0.7        5.91 ± 23%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      5.24 ±  5%      -0.5        4.70 ± 25%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      5.21 ±  6%      -0.5        4.66 ± 25%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.92 ± 14%      -0.5        3.46 ± 25%  perf-profile.calltrace.cycles-pp.__splice_from_pipe.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe.vmsplice
      2.70 ± 12%      -0.3        2.39 ± 26%  perf-profile.calltrace.cycles-pp.add_to_pipe.iter_to_pipe.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.27 ± 16%      -0.3        1.96 ± 24%  perf-profile.calltrace.cycles-pp.pipe_to_user.__splice_from_pipe.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.46 ± 14%      -0.3       14.17 ± 20%  perf-profile.calltrace.cycles-pp.iter_to_pipe.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe.vmsplice
      2.12 ± 16%      -0.3        1.83 ± 25%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_to_user.__splice_from_pipe.__do_sys_vmsplice.do_syscall_64
      7.34 ±  8%      -0.3        7.06 ± 24%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.82 ± 10%      -0.3        0.56 ± 48%  perf-profile.calltrace.cycles-pp.mutex_lock.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe.vmsplice
      4.04 ±  3%      -0.2        3.79 ± 25%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.30 ±100%      -0.2        0.09 ±223%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.39 ± 71%      -0.2        0.20 ±142%  perf-profile.calltrace.cycles-pp.__intel_pmu_enable_all.perf_adjust_freq_unthr_context.perf_event_task_tick.scheduler_tick.update_process_times
      0.28 ±100%      -0.2        0.08 ±223%  perf-profile.calltrace.cycles-pp.anon_pipe_buf_release.__splice_from_pipe.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.62 ± 12%      -0.2        0.43 ± 72%  perf-profile.calltrace.cycles-pp.__entry_text_start.write
      0.64 ± 10%      -0.2        0.48 ± 74%  perf-profile.calltrace.cycles-pp.pipe_to_null.__splice_from_pipe.splice_from_pipe.do_splice.__do_splice
      1.82 ± 13%      -0.2        1.67 ± 25%  perf-profile.calltrace.cycles-pp._copy_to_iter.copy_page_to_iter.pipe_to_user.__splice_from_pipe.__do_sys_vmsplice
      1.26 ± 14%      -0.1        1.12 ± 26%  perf-profile.calltrace.cycles-pp.rep_movs_alternative.copyout._copy_to_iter.copy_page_to_iter.pipe_to_user
      2.95 ±  8%      -0.1        2.82 ± 24%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.41 ± 14%      -0.1        1.29 ± 27%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.copy_page_to_iter.pipe_to_user.__splice_from_pipe
      1.24 ± 12%      -0.1        1.16 ± 20%  perf-profile.calltrace.cycles-pp.__import_iovec.import_iovec.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.08 ±223%      -0.1        0.00        perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.08 ±223%      -0.1        0.00        perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
      1.76 ±  8%      -0.1        1.68 ± 21%  perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.86 ± 18%      -0.1        0.78 ± 25%  perf-profile.calltrace.cycles-pp.__fget_light.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe.vmsplice
      2.34 ±  7%      -0.1        2.27 ± 22%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.71 ±  8%      -0.1        1.64 ± 21%  perf-profile.calltrace.cycles-pp.perf_adjust_freq_unthr_context.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle
      2.73 ±  7%      -0.1        2.66 ± 23%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.29 ± 12%      -0.1        1.22 ± 19%  perf-profile.calltrace.cycles-pp.import_iovec.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe.vmsplice
      0.47 ± 74%      -0.1        0.41 ±104%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
     12.46 ±  5%      -0.1       12.40 ± 24%  perf-profile.calltrace.cycles-pp.splice
      2.65 ±  7%      -0.1        2.59 ± 23%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      4.51 ±  6%      -0.0        4.47 ± 24%  perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_release.__splice_from_pipe.splice_from_pipe.do_splice.__do_splice
      0.76 ± 21%      -0.0        0.72 ± 48%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.do_splice.__do_splice.__x64_sys_splice
      7.70 ±  5%      -0.0        7.66 ± 24%  perf-profile.calltrace.cycles-pp.__splice_from_pipe.splice_from_pipe.do_splice.__do_splice.__x64_sys_splice
      0.80 ±  8%      -0.0        0.78 ± 20%  perf-profile.calltrace.cycles-pp.stress_time_now
     11.26 ±  5%      -0.0       11.24 ± 24%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.splice
     11.08 ±  5%      -0.0       11.06 ± 24%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.splice
      8.57 ±  5%      -0.0        8.56 ± 24%  perf-profile.calltrace.cycles-pp.splice_from_pipe.do_splice.__do_splice.__x64_sys_splice.do_syscall_64
      0.79 ± 16%      -0.0        0.78 ± 47%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.19 ±141%      -0.0        0.18 ±141%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.80 ± 30%      -0.0        0.79 ± 48%  perf-profile.calltrace.cycles-pp.clock_nanosleep
      1.22 ± 31%      -0.0        1.22 ± 23%  perf-profile.calltrace.cycles-pp.__entry_text_start.vmsplice
      0.30 ±101%      +0.0        0.30 ±100%  perf-profile.calltrace.cycles-pp.stress_time_now_timespec
      0.18 ±141%      +0.0        0.19 ±141%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.09 ±223%      +0.0        0.11 ±223%  perf-profile.calltrace.cycles-pp.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.39 ± 71%      +0.0        0.41 ± 74%  perf-profile.calltrace.cycles-pp.__entry_text_start.splice
     10.04 ±  5%      +0.0       10.08 ± 23%  perf-profile.calltrace.cycles-pp.do_splice.__do_splice.__x64_sys_splice.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.24 ±  5%      +0.0       10.28 ± 24%  perf-profile.calltrace.cycles-pp.__do_splice.__x64_sys_splice.do_syscall_64.entry_SYSCALL_64_after_hwframe.splice
     10.81 ±  5%      +0.1       10.86 ± 24%  perf-profile.calltrace.cycles-pp.__x64_sys_splice.do_syscall_64.entry_SYSCALL_64_after_hwframe.splice
      0.60 ± 54%      +0.1        0.66 ± 47%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.58 ± 54%      +0.1        0.64 ± 47%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.84 ± 20%      +0.1        0.92 ± 24%  perf-profile.calltrace.cycles-pp.security_file_permission.do_splice.__do_splice.__x64_sys_splice.do_syscall_64
      0.00            +0.1        0.08 ±223%  perf-profile.calltrace.cycles-pp.mutex_unlock.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe.vmsplice
      0.00            +0.1        0.08 ±223%  perf-profile.calltrace.cycles-pp.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     10.59 ± 16%      +0.1       10.68 ± 19%  perf-profile.calltrace.cycles-pp.iov_iter_get_pages2.iter_to_pipe.__do_sys_vmsplice.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.54 ± 16%      +0.1       10.63 ± 19%  perf-profile.calltrace.cycles-pp.__iov_iter_get_pages_alloc.iov_iter_get_pages2.iter_to_pipe.__do_sys_vmsplice.do_syscall_64
      0.22 ±144%      +0.1        0.32 ±101%  perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64
      0.00            +0.1        0.10 ±223%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.58 ± 75%      +0.1        0.69 ± 60%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     10.30 ± 16%      +0.1       10.44 ± 18%  perf-profile.calltrace.cycles-pp.get_user_pages_fast.__iov_iter_get_pages_alloc.iov_iter_get_pages2.iter_to_pipe.__do_sys_vmsplice
      0.45 ± 77%      +0.1        0.59 ± 48%  perf-profile.calltrace.cycles-pp.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      4.00 ± 16%      +0.2        4.16 ± 17%  perf-profile.calltrace.cycles-pp.try_grab_folio.gup_pte_range.gup_pgd_range.lockless_pages_from_mm.internal_get_user_pages_fast
     10.15 ± 16%      +0.2       10.34 ± 19%  perf-profile.calltrace.cycles-pp.internal_get_user_pages_fast.get_user_pages_fast.__iov_iter_get_pages_alloc.iov_iter_get_pages2.iter_to_pipe
      0.24 ±143%      +0.2        0.44 ± 72%  perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.24 ±143%      +0.2        0.44 ± 72%  perf-profile.calltrace.cycles-pp.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      9.86 ± 16%      +0.2       10.07 ± 19%  perf-profile.calltrace.cycles-pp.gup_pgd_range.lockless_pages_from_mm.internal_get_user_pages_fast.get_user_pages_fast.__iov_iter_get_pages_alloc
     10.02 ± 16%      +0.2       10.25 ± 19%  perf-profile.calltrace.cycles-pp.lockless_pages_from_mm.internal_get_user_pages_fast.get_user_pages_fast.__iov_iter_get_pages_alloc.iov_iter_get_pages2
      8.89 ± 17%      +0.2        9.14 ± 18%  perf-profile.calltrace.cycles-pp.gup_pte_range.gup_pgd_range.lockless_pages_from_mm.internal_get_user_pages_fast.get_user_pages_fast
     41.12 ± 10%      +3.8       44.97 ± 24%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     43.49 ± 10%      +3.9       47.35 ± 24%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     33.51 ± 13%      +4.5       38.00 ± 25%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     39.54 ±  9%      +4.9       44.45 ± 24%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     40.60 ±  8%      +4.9       45.54 ± 24%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     40.84 ±  8%      +5.1       45.93 ± 23%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     40.87 ±  8%      +5.1       45.96 ± 24%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     40.87 ±  8%      +5.1       45.96 ± 24%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     48.50 ±  8%      -3.3       45.23 ± 22%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     47.68 ±  8%      -3.2       44.51 ± 22%  perf-profile.children.cycles-pp.do_syscall_64
     13.24 ±  8%      -2.1       11.09 ± 30%  perf-profile.children.cycles-pp.write
     10.33 ±  8%      -1.8        8.58 ± 30%  perf-profile.children.cycles-pp.vfs_write
     10.71 ±  8%      -1.7        9.02 ± 30%  perf-profile.children.cycles-pp.ksys_write
     28.00 ± 12%      -1.7       26.31 ± 20%  perf-profile.children.cycles-pp.vmsplice
      9.15 ± 10%      -1.7        7.49 ± 30%  perf-profile.children.cycles-pp.pipe_write
     23.92 ± 13%      -1.4       22.48 ± 20%  perf-profile.children.cycles-pp.__do_sys_vmsplice
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.children.cycles-pp.x86_64_start_kernel
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.children.cycles-pp.x86_64_start_reservations
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.children.cycles-pp.start_kernel
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.children.cycles-pp.arch_call_rest_init
      2.62 ± 45%      -1.2        1.39 ±100%  perf-profile.children.cycles-pp.rest_init
      7.32 ±  9%      -1.2        6.09 ± 29%  perf-profile.children.cycles-pp.rep_movs_alternative
      6.65 ± 10%      -1.2        5.50 ± 31%  perf-profile.children.cycles-pp.copy_page_from_iter
      6.51 ±  9%      -1.1        5.39 ± 31%  perf-profile.children.cycles-pp._copy_from_iter
      6.37 ±  9%      -1.1        5.28 ± 31%  perf-profile.children.cycles-pp.copyin
      6.76 ±  6%      -0.7        6.07 ± 22%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      7.47 ±  7%      -0.6        6.83 ± 21%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     11.82 ±  5%      -0.5       11.30 ± 24%  perf-profile.children.cycles-pp.__splice_from_pipe
      5.39 ±  5%      -0.5        4.90 ± 21%  perf-profile.children.cycles-pp.hrtimer_interrupt
      5.41 ±  5%      -0.5        4.93 ± 20%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      2.70 ± 12%      -0.3        2.39 ± 26%  perf-profile.children.cycles-pp.add_to_pipe
      2.28 ± 16%      -0.3        1.97 ± 24%  perf-profile.children.cycles-pp.pipe_to_user
      2.16 ± 16%      -0.3        1.85 ± 25%  perf-profile.children.cycles-pp.copy_page_to_iter
     14.52 ± 14%      -0.3       14.22 ± 20%  perf-profile.children.cycles-pp.iter_to_pipe
      4.37 ±  5%      -0.3        4.08 ± 20%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.38 ± 11%      -0.2        1.17 ± 27%  perf-profile.children.cycles-pp.mutex_lock
      0.33 ± 41%      -0.2        0.14 ± 50%  perf-profile.children.cycles-pp.tick_irq_enter
      0.34 ± 41%      -0.2        0.15 ± 46%  perf-profile.children.cycles-pp.irq_enter_rcu
      1.84 ± 13%      -0.2        1.68 ± 25%  perf-profile.children.cycles-pp._copy_to_iter
      3.15 ±  6%      -0.1        3.00 ± 22%  perf-profile.children.cycles-pp.tick_sched_timer
      0.59 ± 24%      -0.1        0.44 ± 11%  perf-profile.children.cycles-pp.ktime_get
      0.64 ±  9%      -0.1        0.49 ± 25%  perf-profile.children.cycles-pp.clockevents_program_event
      1.62 ± 13%      -0.1        1.48 ± 27%  perf-profile.children.cycles-pp.copyout
      0.53 ±  9%      -0.1        0.40 ± 31%  perf-profile.children.cycles-pp.lapic_next_deadline
      1.52 ±  7%      -0.1        1.41 ± 27%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.35 ± 13%      -0.1        0.24 ± 11%  perf-profile.children.cycles-pp.read_tsc
      0.76 ± 11%      -0.1        0.66 ± 23%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.52 ± 17%      -0.1        0.43 ± 22%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.47 ± 18%      -0.1        0.38 ± 28%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.50 ± 10%      -0.1        0.40 ± 17%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.37 ± 27%      -0.1        0.28 ± 37%  perf-profile.children.cycles-pp.hrtimer_wakeup
      0.38 ± 24%      -0.1        0.29 ± 33%  perf-profile.children.cycles-pp.try_to_wake_up
      0.24 ± 37%      -0.1        0.15 ± 58%  perf-profile.children.cycles-pp.pipe_lock
      2.37 ± 10%      -0.1        2.29 ± 20%  perf-profile.children.cycles-pp.__entry_text_start
      1.27 ± 12%      -0.1        1.19 ± 20%  perf-profile.children.cycles-pp.__import_iovec
      0.28 ± 16%      -0.1        0.19 ± 27%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.62 ±  8%      -0.1        0.54 ± 21%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.29 ± 34%      -0.1        0.22 ± 42%  perf-profile.children.cycles-pp.clock_gettime
      0.47 ± 17%      -0.1        0.40 ± 17%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      1.32 ± 13%      -0.1        1.25 ± 20%  perf-profile.children.cycles-pp.import_iovec
      0.13 ± 31%      -0.1        0.07 ± 86%  perf-profile.children.cycles-pp.tick_sched_do_timer
      1.83 ±  6%      -0.1        1.76 ± 20%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.62 ± 13%      -0.1        0.55 ± 36%  perf-profile.children.cycles-pp.stress_time_now_timespec
      0.13 ± 27%      -0.1        0.07 ± 57%  perf-profile.children.cycles-pp.write@plt
      0.12 ± 11%      -0.1        0.06 ± 76%  perf-profile.children.cycles-pp.kill_fasync
      0.22 ± 17%      -0.1        0.16 ± 30%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.18 ± 11%      -0.1        0.12 ± 30%  perf-profile.children.cycles-pp.update_irq_load_avg
      2.47 ±  6%      -0.1        2.41 ± 21%  perf-profile.children.cycles-pp.scheduler_tick
      0.11 ± 21%      -0.1        0.05 ± 73%  perf-profile.children.cycles-pp.__x64_sys_write
      0.22 ± 15%      -0.1        0.16 ± 30%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.21 ± 23%      -0.1        0.15 ± 35%  perf-profile.children.cycles-pp.enqueue_entity
      1.80 ±  7%      -0.1        1.75 ± 20%  perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context
      2.88 ±  5%      -0.1        2.82 ± 21%  perf-profile.children.cycles-pp.tick_sched_handle
      0.23 ± 26%      -0.1        0.18 ± 34%  perf-profile.children.cycles-pp.activate_task
      0.23 ± 26%      -0.1        0.18 ± 35%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.41 ± 14%      -0.1        0.36 ± 12%  perf-profile.children.cycles-pp.__cond_resched
      2.81 ±  5%      -0.1        2.76 ± 21%  perf-profile.children.cycles-pp.update_process_times
      0.25 ± 29%      -0.1        0.20 ± 34%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.08 ± 21%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__wake_up_common
      0.52 ± 15%      -0.0        0.47 ± 17%  perf-profile.children.cycles-pp.stress_vm_splice
      0.31 ± 10%      -0.0        0.26 ± 29%  perf-profile.children.cycles-pp.perf_rotate_context
      0.09 ± 39%      -0.0        0.04 ±120%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.32 ± 15%      -0.0        0.27 ± 35%  perf-profile.children.cycles-pp.file_update_time
      0.16 ± 25%      -0.0        0.12 ± 48%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.23 ± 20%      -0.0        0.19 ± 16%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.51 ±  6%      -0.0        4.47 ± 24%  perf-profile.children.cycles-pp.page_cache_pipe_buf_release
      0.32 ± 11%      -0.0        0.28 ± 28%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.82 ± 13%      -0.0        0.78 ± 25%  perf-profile.children.cycles-pp.mutex_unlock
      0.09 ± 24%      -0.0        0.05 ± 77%  perf-profile.children.cycles-pp.timerqueue_del
      0.15 ± 40%      -0.0        0.12 ± 35%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.27 ±  7%      -0.0        0.24 ± 30%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.78 ±  9%      -0.0        0.75 ± 30%  perf-profile.children.cycles-pp.pipe_to_null
      0.16 ± 24%      -0.0        0.13 ± 49%  perf-profile.children.cycles-pp.current_time
      0.08 ± 54%      -0.0        0.06 ± 84%  perf-profile.children.cycles-pp.prepare_task_switch
      1.38 ± 16%      -0.0        1.36 ± 21%  perf-profile.children.cycles-pp.__fget_light
      0.04 ± 74%      -0.0        0.02 ±142%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.05 ± 73%      -0.0        0.02 ±146%  perf-profile.children.cycles-pp.rb_erase
      0.02 ±152%      -0.0        0.00        perf-profile.children.cycles-pp.__x64_sys_openat
      0.02 ±152%      -0.0        0.00        perf-profile.children.cycles-pp.do_sys_openat2
      0.06 ± 49%      -0.0        0.03 ±106%  perf-profile.children.cycles-pp.is_valid_gup_args
      0.02 ±149%      -0.0        0.00        perf-profile.children.cycles-pp.worker_thread
      0.06 ± 72%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.ct_idle_exit
      0.17 ± 34%      -0.0        0.14 ± 29%  perf-profile.children.cycles-pp.update_load_avg
      0.02 ±146%      -0.0        0.00        perf-profile.children.cycles-pp.cpuidle_reflect
      0.18 ± 27%      -0.0        0.16 ± 24%  perf-profile.children.cycles-pp.update_blocked_averages
      0.02 ±141%      -0.0        0.00        perf-profile.children.cycles-pp.vmsplice@plt
      0.04 ± 77%      -0.0        0.02 ±143%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.02 ±141%      -0.0        0.00        perf-profile.children.cycles-pp.poll_idle
      0.08 ± 31%      -0.0        0.06 ± 17%  perf-profile.children.cycles-pp.__switch_to_asm
      0.07 ± 80%      -0.0        0.05 ± 76%  perf-profile.children.cycles-pp.__switch_to
      0.03 ±100%      -0.0        0.01 ±223%  perf-profile.children.cycles-pp.reweight_entity
      0.09 ± 25%      -0.0        0.07 ± 19%  perf-profile.children.cycles-pp.x86_pmu_disable
      0.28 ± 49%      -0.0        0.26 ± 22%  perf-profile.children.cycles-pp.schedule_idle
      0.18 ± 28%      -0.0        0.16 ± 29%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.30 ± 15%      -0.0        0.28 ± 24%  perf-profile.children.cycles-pp.native_sched_clock
      0.21 ± 31%      -0.0        0.19 ± 33%  perf-profile.children.cycles-pp.get_pipe_info
      0.02 ±141%      -0.0        0.00        perf-profile.children.cycles-pp.update_dl_rq_load_avg
      0.02 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.do_filp_open
      0.02 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.path_openat
      0.12 ± 31%      -0.0        0.10 ± 44%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.04 ± 73%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.39 ± 19%      -0.0        0.38 ± 25%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.83 ±  8%      -0.0        0.82 ± 19%  perf-profile.children.cycles-pp.stress_time_now
      0.11 ± 49%      -0.0        0.10 ± 22%  perf-profile.children.cycles-pp.__x64_sys_vmsplice
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.menu_reflect
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.link_path_walk
      0.05 ± 82%      -0.0        0.04 ± 75%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.04 ± 73%      -0.0        0.03 ±105%  perf-profile.children.cycles-pp.ct_kernel_exit
      0.16 ± 15%      -0.0        0.15 ± 47%  perf-profile.children.cycles-pp.__get_task_ioprio
      0.08 ± 50%      -0.0        0.07 ± 50%  perf-profile.children.cycles-pp.__fdget
      0.17 ± 17%      -0.0        0.16 ± 27%  perf-profile.children.cycles-pp.rcu_all_qs
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.pmd_huge
      0.04 ± 72%      -0.0        0.03 ±101%  perf-profile.children.cycles-pp.update_cfs_group
      0.02 ±141%      -0.0        0.01 ±223%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.cpuidle_not_available
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.account_process_tick
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.first_iovec_segment
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.timekeeping_advance
      0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.update_wall_time
      0.12 ± 17%      -0.0        0.12 ± 26%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.03 ±101%      -0.0        0.02 ±146%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.24 ± 39%      -0.0        0.23 ± 39%  perf-profile.children.cycles-pp.find_busiest_group
      0.31 ± 32%      -0.0        0.30 ± 21%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.14 ± 23%      -0.0        0.14 ± 38%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.09 ± 26%      -0.0        0.08 ± 38%  perf-profile.children.cycles-pp.timerqueue_add
      0.06 ± 95%      -0.0        0.06 ± 46%  perf-profile.children.cycles-pp.set_next_entity
      0.04 ± 72%      -0.0        0.03 ±101%  perf-profile.children.cycles-pp.vfs_read
      0.06 ± 73%      -0.0        0.06 ± 47%  perf-profile.children.cycles-pp.pipe_clear_nowait
      0.02 ±223%      -0.0        0.01 ±223%  perf-profile.children.cycles-pp.sched_idle_set_state
      0.05 ± 48%      -0.0        0.04 ± 73%  perf-profile.children.cycles-pp.update_curr
      0.22 ± 40%      -0.0        0.22 ± 37%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.22 ± 10%      -0.0        0.21 ± 19%  perf-profile.children.cycles-pp.import_ubuf
      0.14 ± 53%      -0.0        0.13 ± 36%  perf-profile.children.cycles-pp.hrtimer_start_range_ns
      0.08 ± 17%      -0.0        0.08 ± 47%  perf-profile.children.cycles-pp.rw_verify_area
      0.03 ±144%      -0.0        0.03 ±147%  perf-profile.children.cycles-pp.call_cpuidle
      0.02 ±142%      -0.0        0.02 ±141%  perf-profile.children.cycles-pp.iov_iter_advance
      8.58 ±  5%      -0.0        8.58 ± 24%  perf-profile.children.cycles-pp.splice_from_pipe
      0.23 ± 16%      -0.0        0.23 ± 27%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.09 ± 50%      -0.0        0.09 ± 67%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.08 ± 23%      -0.0        0.08 ± 63%  perf-profile.children.cycles-pp.clock_gettime@plt
      0.01 ±223%      -0.0        0.01 ±223%  perf-profile.children.cycles-pp.rcu_preempt_deferred_qs
      0.04 ± 72%      -0.0        0.04 ±102%  perf-profile.children.cycles-pp.ksys_read
      0.05 ±108%      -0.0        0.04 ±110%  perf-profile.children.cycles-pp.___perf_sw_event
      0.01 ±223%      -0.0        0.01 ±223%  perf-profile.children.cycles-pp.__nanosleep
      0.03 ±103%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.02 ±142%      -0.0        0.02 ±141%  perf-profile.children.cycles-pp.select_task_rq
      0.10 ± 18%      -0.0        0.10 ± 44%  perf-profile.children.cycles-pp.update_rq_clock
      0.05 ± 74%      +0.0        0.05 ±113%  perf-profile.children.cycles-pp.rcu_pending
      0.01 ±223%      +0.0        0.01 ±223%  perf-profile.children.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.02 ±223%      +0.0        0.02 ±142%  perf-profile.children.cycles-pp.__hrtimer_start_range_ns
      0.38 ± 14%      +0.0        0.38 ± 19%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.01 ±223%      +0.0        0.01 ±223%  perf-profile.children.cycles-pp.rb_next
      0.01 ±223%      +0.0        0.01 ±223%  perf-profile.children.cycles-pp.rcu_core
      0.04 ± 71%      +0.0        0.04 ± 73%  perf-profile.children.cycles-pp.read
      0.04 ±105%      +0.0        0.04 ±107%  perf-profile.children.cycles-pp.trigger_load_balance
      0.09 ± 50%      +0.0        0.09 ± 16%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.06 ± 54%      +0.0        0.07 ± 48%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.04 ±112%      +0.0        0.04 ± 75%  perf-profile.children.cycles-pp.wait_for_space
      0.02 ±141%      +0.0        0.02 ±144%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.04 ±101%      +0.0        0.05 ± 45%  perf-profile.children.cycles-pp.inode_needs_update_time
      0.08 ± 16%      +0.0        0.08 ± 54%  perf-profile.children.cycles-pp.hrtimer_update_next_event
     12.79 ±  5%      +0.0       12.80 ± 24%  perf-profile.children.cycles-pp.splice
      0.05 ± 81%      +0.0        0.05 ± 47%  perf-profile.children.cycles-pp.idle_cpu
      0.02 ±141%      +0.0        0.02 ±142%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.05 ± 52%      +0.0        0.06 ± 49%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.03 ±103%      +0.0        0.04 ± 72%  perf-profile.children.cycles-pp.handle_mm_fault
      0.02 ±141%      +0.0        0.03 ±100%  perf-profile.children.cycles-pp.error_entry
      0.03 ±100%      +0.0        0.04 ± 72%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.05 ± 52%      +0.0        0.06 ± 48%  perf-profile.children.cycles-pp.exc_page_fault
      0.05 ±145%      +0.0        0.06 ± 80%  perf-profile.children.cycles-pp.kthread
      0.21 ± 21%      +0.0        0.22 ± 22%  perf-profile.children.cycles-pp._raw_spin_lock
      0.03 ±141%      +0.0        0.04 ±141%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.05 ±145%      +0.0        0.06 ± 81%  perf-profile.children.cycles-pp.ret_from_fork
      0.01 ±223%      +0.0        0.02 ±141%  perf-profile.children.cycles-pp.do_fault
      0.01 ±223%      +0.0        0.02 ±141%  perf-profile.children.cycles-pp.usleep
      0.01 ±223%      +0.0        0.02 ±141%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.01 ±223%      +0.0        0.02 ±142%  perf-profile.children.cycles-pp.rb_insert_color
      0.01 ±223%      +0.0        0.02 ±142%  perf-profile.children.cycles-pp.__collapse_huge_page_copy
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.hrtimer_forward
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.unmap_page_range
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.zap_pmd_range
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.zap_pte_range
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.copy_mc_enhanced_fast_string
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.os_xsave
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.splice_write_null
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.check_cpu_stall
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.intel_idle_irq
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.do_read_fault
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.filemap_map_pages
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.cmd_record
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.__cmd_record
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.hrtimer_init_sleeper
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.__hrtimer_init
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.irqentry_enter
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.perf_swevent_start
      0.02 ±144%      +0.0        0.03 ±144%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.11 ± 14%      +0.0        0.12 ± 27%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.unmap_vmas
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.02 ±141%      +0.0        0.03 ±100%  perf-profile.children.cycles-pp.newidle_balance
      0.01 ±223%      +0.0        0.02 ±144%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.01 ±223%      +0.0        0.02 ±142%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.01 ±223%      +0.0        0.02 ±142%  perf-profile.children.cycles-pp.sched_setaffinity
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.do_group_exit
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.do_exit
      0.00            +0.0        0.01 ±223%  perf-profile.children.cycles-pp.exit_mm
      0.10 ± 19%      +0.0        0.11 ± 73%  perf-profile.children.cycles-pp.aa_file_perm
      0.08 ± 55%      +0.0        0.09 ± 31%  perf-profile.children.cycles-pp.local_clock
      0.58 ± 30%      +0.0        0.60 ± 24%  perf-profile.children.cycles-pp.__schedule
      0.48 ± 17%      +0.0        0.50 ± 13%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.33 ± 45%      +0.0        0.34 ± 44%  perf-profile.children.cycles-pp.load_balance
      0.09 ± 24%      +0.0        0.10 ± 48%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.02 ±144%      +0.0        0.03 ±106%  perf-profile.children.cycles-pp._find_next_and_bit
      0.01 ±223%      +0.0        0.02 ±149%  perf-profile.children.cycles-pp.arch_cpu_idle_exit
      0.13 ± 12%      +0.0        0.14 ± 29%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.11 ± 15%      +0.0        0.13 ± 31%  perf-profile.children.cycles-pp.dequeue_entity
      0.79 ± 32%      +0.0        0.80 ± 37%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.01 ±223%      +0.0        0.03 ±100%  perf-profile.children.cycles-pp.collapse_huge_page
      0.00            +0.0        0.02 ±141%  perf-profile.children.cycles-pp.evlist_cpu_iterator__next
      0.04 ± 77%      +0.0        0.06 ± 48%  perf-profile.children.cycles-pp.tick_program_event
      0.01 ±223%      +0.0        0.03 ±147%  perf-profile.children.cycles-pp.get_cpu_device
      0.01 ±223%      +0.0        0.03 ±101%  perf-profile.children.cycles-pp.khugepaged
      0.01 ±223%      +0.0        0.03 ±101%  perf-profile.children.cycles-pp.khugepaged_scan_mm_slot
      0.01 ±223%      +0.0        0.03 ±101%  perf-profile.children.cycles-pp.hpage_collapse_scan_pmd
      0.00            +0.0        0.02 ±142%  perf-profile.children.cycles-pp.irq_work_tick
      0.00            +0.0        0.02 ±142%  perf-profile.children.cycles-pp.mas_store_prealloc
      0.03 ±100%      +0.0        0.05 ± 47%  perf-profile.children.cycles-pp.cmd_stat
      0.03 ±100%      +0.0        0.05 ± 47%  perf-profile.children.cycles-pp.dispatch_events
      0.03 ±100%      +0.0        0.05 ± 47%  perf-profile.children.cycles-pp.process_interval
      0.13 ± 32%      +0.0        0.15 ± 24%  perf-profile.children.cycles-pp.splice_from_pipe_next
      0.00            +0.0        0.02 ±141%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.0        0.02 ±141%  perf-profile.children.cycles-pp.tick_nohz_stop_idle
      0.00            +0.0        0.02 ±144%  perf-profile.children.cycles-pp._copy_from_user
      0.00            +0.0        0.02 ±146%  perf-profile.children.cycles-pp.exit_mmap
      0.00            +0.0        0.02 ±149%  perf-profile.children.cycles-pp.__mmput
      0.00            +0.0        0.02 ±145%  perf-profile.children.cycles-pp.get_timespec64
      0.00            +0.0        0.02 ±141%  perf-profile.children.cycles-pp.ct_nmi_exit
      0.15 ± 50%      +0.0        0.18 ± 39%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.34 ± 46%      +0.0        0.37 ± 42%  perf-profile.children.cycles-pp.rebalance_domains
      0.04 ± 77%      +0.0        0.07 ± 54%  perf-profile.children.cycles-pp.pipe_unlock
      0.62 ± 35%      +0.0        0.65 ± 41%  perf-profile.children.cycles-pp.__do_softirq
      0.13 ± 13%      +0.0        0.16 ± 26%  perf-profile.children.cycles-pp.perf_pmu_nop_void
      0.03 ±103%      +0.0        0.06 ± 77%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.10 ± 23%      +0.0        0.13 ± 35%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.02 ±142%      +0.0        0.05 ± 47%  perf-profile.children.cycles-pp.read_counters
      0.00            +0.0        0.03 ±102%  perf-profile.children.cycles-pp.mmap_region
      0.04 ±104%      +0.0        0.07 ± 48%  perf-profile.children.cycles-pp.__libc_start_main
      0.04 ±104%      +0.0        0.07 ± 48%  perf-profile.children.cycles-pp.main
      0.04 ±104%      +0.0        0.07 ± 48%  perf-profile.children.cycles-pp.run_builtin
      0.01 ±223%      +0.0        0.04 ±107%  perf-profile.children.cycles-pp.irqentry_exit
      0.00            +0.0        0.03 ±223%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.00            +0.0        0.03 ±100%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.00            +0.0        0.03 ±100%  perf-profile.children.cycles-pp.do_mmap
      0.22 ± 38%      +0.0        0.26 ± 46%  perf-profile.children.cycles-pp.__fdget_pos
      0.01 ±223%      +0.0        0.04 ±120%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.00            +0.0        0.04 ±223%  perf-profile.children.cycles-pp._nohz_idle_balance
      0.32 ± 19%      +0.0        0.36 ± 29%  perf-profile.children.cycles-pp.schedule
      0.04 ±109%      +0.0        0.07 ± 23%  perf-profile.children.cycles-pp.kfree
      0.54 ± 30%      +0.0        0.58 ± 28%  perf-profile.children.cycles-pp.common_nsleep
      0.52 ± 30%      +0.0        0.56 ± 28%  perf-profile.children.cycles-pp.hrtimer_nanosleep
      0.48 ± 33%      +0.0        0.52 ± 29%  perf-profile.children.cycles-pp.do_nanosleep
      0.05 ± 73%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.execve
      0.05 ± 73%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.05 ± 73%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.do_execveat_common
      0.01 ±223%      +0.0        0.06 ± 46%  perf-profile.children.cycles-pp.exec_binprm
      0.01 ±223%      +0.0        0.06 ± 46%  perf-profile.children.cycles-pp.search_binary_handler
      0.01 ±223%      +0.0        0.06 ± 46%  perf-profile.children.cycles-pp.load_elf_binary
     10.07 ±  5%      +0.0       10.12 ± 23%  perf-profile.children.cycles-pp.do_splice
     10.82 ±  5%      +0.0       10.88 ± 24%  perf-profile.children.cycles-pp.__x64_sys_splice
     10.26 ±  5%      +0.0       10.30 ± 23%  perf-profile.children.cycles-pp.__do_splice
      0.02 ±141%      +0.1        0.07 ± 16%  perf-profile.children.cycles-pp.bprm_execve
      0.86 ± 13%      +0.1        0.91 ± 20%  perf-profile.children.cycles-pp.menu_select
      1.12 ± 17%      +0.1        1.18 ± 24%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.59 ± 31%      +0.1        0.65 ± 28%  perf-profile.children.cycles-pp.__x64_sys_clock_nanosleep
      0.22 ± 25%      +0.1        0.28 ± 40%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.33 ± 18%      +0.1        0.40 ± 25%  perf-profile.children.cycles-pp.native_irq_return_iret
      1.29 ± 14%      +0.1        1.36 ± 24%  perf-profile.children.cycles-pp.security_file_permission
     10.56 ± 16%      +0.1       10.64 ± 19%  perf-profile.children.cycles-pp.__iov_iter_get_pages_alloc
      0.80 ± 31%      +0.1        0.89 ± 26%  perf-profile.children.cycles-pp.clock_nanosleep
     10.59 ± 16%      +0.1       10.69 ± 19%  perf-profile.children.cycles-pp.iov_iter_get_pages2
     10.32 ± 16%      +0.1       10.45 ± 18%  perf-profile.children.cycles-pp.get_user_pages_fast
      4.00 ± 16%      +0.2        4.16 ± 17%  perf-profile.children.cycles-pp.try_grab_folio
     10.17 ± 16%      +0.2       10.35 ± 19%  perf-profile.children.cycles-pp.internal_get_user_pages_fast
      9.88 ± 16%      +0.2       10.08 ± 19%  perf-profile.children.cycles-pp.gup_pgd_range
     10.04 ± 16%      +0.2       10.26 ± 19%  perf-profile.children.cycles-pp.lockless_pages_from_mm
      0.00            +0.3        0.26 ± 40%  perf-profile.children.cycles-pp.folio_fast_pin_allowed
      8.90 ± 16%      +0.4        9.26 ± 19%  perf-profile.children.cycles-pp.gup_pte_range
     43.49 ± 10%      +3.9       47.35 ± 24%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     43.49 ± 10%      +3.9       47.35 ± 24%  perf-profile.children.cycles-pp.cpu_startup_entry
     43.48 ± 10%      +3.9       47.34 ± 24%  perf-profile.children.cycles-pp.do_idle
     42.90 ± 10%      +3.9       46.77 ± 24%  perf-profile.children.cycles-pp.cpuidle_idle_call
     41.73 ± 10%      +3.9       45.61 ± 24%  perf-profile.children.cycles-pp.cpuidle_enter
     41.72 ± 10%      +3.9       45.60 ± 24%  perf-profile.children.cycles-pp.cpuidle_enter_state
     33.51 ± 13%      +4.5       38.00 ± 25%  perf-profile.children.cycles-pp.intel_idle
     40.87 ±  8%      +5.1       45.96 ± 24%  perf-profile.children.cycles-pp.start_secondary
      7.16 ± 10%      -1.2        5.94 ± 30%  perf-profile.self.cycles-pp.rep_movs_alternative
      2.64 ± 12%      -0.3        2.32 ± 26%  perf-profile.self.cycles-pp.add_to_pipe
      1.67 ±  7%      -0.3        1.40 ± 22%  perf-profile.self.cycles-pp.__do_sys_vmsplice
      0.85 ± 21%      -0.2        0.64 ± 25%  perf-profile.self.cycles-pp.pipe_write
      1.06 ± 14%      -0.2        0.90 ± 29%  perf-profile.self.cycles-pp.mutex_lock
      0.30 ± 40%      -0.1        0.17 ± 33%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.53 ±  9%      -0.1        0.40 ± 31%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.91 ± 14%      -0.1        0.78 ± 19%  perf-profile.self.cycles-pp.gup_pgd_range
      1.48 ±  7%      -0.1        1.36 ± 27%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      1.09 ± 12%      -0.1        0.98 ± 18%  perf-profile.self.cycles-pp.vmsplice
      0.76 ± 11%      -0.1        0.66 ± 23%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.34 ± 14%      -0.1        0.23 ± 10%  perf-profile.self.cycles-pp.read_tsc
      0.82 ±  4%      -0.1        0.72 ± 30%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.50 ± 10%      -0.1        0.40 ± 17%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.47 ± 18%      -0.1        0.37 ± 28%  perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.25 ± 35%      -0.1        0.16 ± 58%  perf-profile.self.cycles-pp.clock_gettime
      0.27 ± 17%      -0.1        0.18 ± 23%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      1.25 ±  9%      -0.1        1.18 ± 21%  perf-profile.self.cycles-pp.iter_to_pipe
      0.22 ± 39%      -0.1        0.15 ± 58%  perf-profile.self.cycles-pp.pipe_lock
      3.68 ±  3%      -0.1        3.60 ± 26%  perf-profile.self.cycles-pp.__splice_from_pipe
      1.06 ± 14%      -0.1        0.99 ± 22%  perf-profile.self.cycles-pp.__import_iovec
      0.57 ±  9%      -0.1        0.50 ± 36%  perf-profile.self.cycles-pp.vfs_write
      0.12 ± 33%      -0.1        0.05 ± 79%  perf-profile.self.cycles-pp.write@plt
      0.57 ± 13%      -0.1        0.51 ± 36%  perf-profile.self.cycles-pp.stress_time_now_timespec
      0.31 ± 25%      -0.1        0.25 ± 26%  perf-profile.self.cycles-pp.ktime_get
      0.18 ± 11%      -0.1        0.12 ± 30%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.09 ± 54%      -0.1        0.03 ±150%  perf-profile.self.cycles-pp.tick_sched_do_timer
      0.10 ± 22%      -0.1        0.04 ± 73%  perf-profile.self.cycles-pp.__x64_sys_write
      0.07 ± 24%      -0.1        0.02 ±141%  perf-profile.self.cycles-pp.__wake_up_common
      0.41 ± 18%      -0.1        0.36 ± 13%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.14 ± 18%      -0.1        0.08 ± 26%  perf-profile.self.cycles-pp.internal_get_user_pages_fast
      0.19 ± 15%      -0.1        0.14 ± 51%  perf-profile.self.cycles-pp.__iov_iter_get_pages_alloc
      0.52 ± 10%      -0.1        0.47 ± 38%  perf-profile.self.cycles-pp.write
      0.31 ± 19%      -0.1        0.26 ± 27%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.11 ±  6%      -0.0        0.06 ± 74%  perf-profile.self.cycles-pp.kill_fasync
      0.09 ± 39%      -0.0        0.04 ±120%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.14 ± 37%      -0.0        0.09 ± 66%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.10 ± 41%      -0.0        0.06 ± 78%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.27 ±  8%      -0.0        0.22 ± 27%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      1.34 ± 15%      -0.0        1.30 ± 21%  perf-profile.self.cycles-pp.__fget_light
      0.05 ± 74%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.31 ±  9%      -0.0        0.27 ± 24%  perf-profile.self.cycles-pp.do_syscall_64
      0.24 ± 13%      -0.0        0.21 ± 13%  perf-profile.self.cycles-pp.__cond_resched
      0.96 ± 11%      -0.0        0.92 ± 23%  perf-profile.self.cycles-pp.__entry_text_start
      0.05 ± 71%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.rb_erase
      0.79 ± 14%      -0.0        0.76 ± 25%  perf-profile.self.cycles-pp.mutex_unlock
      0.15 ± 27%      -0.0        0.12 ± 23%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.67 ±  8%      -0.0        0.64 ± 31%  perf-profile.self.cycles-pp.pipe_to_null
      0.10 ± 28%      -0.0        0.06 ± 50%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.09 ± 48%      -0.0        0.06 ± 81%  perf-profile.self.cycles-pp.get_user_pages_fast
      0.08 ± 31%      -0.0        0.05 ± 48%  perf-profile.self.cycles-pp.__switch_to_asm
      0.44 ± 13%      -0.0        0.41 ± 18%  perf-profile.self.cycles-pp.stress_vm_splice
      0.12 ± 37%      -0.0        0.09 ± 54%  perf-profile.self.cycles-pp.current_time
      0.08 ± 26%      -0.0        0.06 ± 49%  perf-profile.self.cycles-pp.x86_pmu_disable
      0.06 ± 48%      -0.0        0.04 ±102%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.06 ± 76%      -0.0        0.04 ±102%  perf-profile.self.cycles-pp.__switch_to
      0.04 ± 77%      -0.0        0.02 ±142%  perf-profile.self.cycles-pp.is_valid_gup_args
      0.04 ± 73%      -0.0        0.02 ±142%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.15 ± 13%      -0.0        0.12 ± 64%  perf-profile.self.cycles-pp.__get_task_ioprio
      0.44 ± 13%      -0.0        0.42 ± 33%  perf-profile.self.cycles-pp.copyout
      0.11 ± 18%      -0.0        0.08 ± 56%  perf-profile.self.cycles-pp.file_update_time
      4.44 ±  6%      -0.0        4.42 ± 24%  perf-profile.self.cycles-pp.page_cache_pipe_buf_release
      0.14 ±  8%      -0.0        0.11 ± 28%  perf-profile.self.cycles-pp.pipe_to_user
      0.03 ±101%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.do_nanosleep
      0.22 ± 27%      -0.0        0.20 ± 16%  perf-profile.self.cycles-pp._copy_to_iter
      0.11 ± 20%      -0.0        0.09 ± 24%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.06 ± 21%      -0.0        0.04 ± 73%  perf-profile.self.cycles-pp.rw_verify_area
      0.02 ±141%      -0.0        0.00        perf-profile.self.cycles-pp.poll_idle
      0.03 ±147%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.tick_sched_timer
      0.03 ±100%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.enqueue_entity
      0.03 ±100%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.reweight_entity
      0.12 ± 31%      -0.0        0.10 ± 60%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.36 ± 18%      -0.0        0.34 ± 23%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.02 ±141%      -0.0        0.00        perf-profile.self.cycles-pp.update_dl_rq_load_avg
      0.06 ± 52%      -0.0        0.05 ± 76%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.21 ± 27%      -0.0        0.19 ± 46%  perf-profile.self.cycles-pp.__x64_sys_splice
      0.05 ±104%      -0.0        0.03 ±103%  perf-profile.self.cycles-pp.timerqueue_add
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.29 ± 14%      -0.0        0.28 ± 25%  perf-profile.self.cycles-pp.native_sched_clock
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.tick_nohz_idle_retain_tick
      0.03 ±106%      -0.0        0.02 ±146%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.14 ± 16%      -0.0        0.13 ± 29%  perf-profile.self.cycles-pp.__do_splice
      0.30 ± 30%      -0.0        0.29 ± 20%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.14 ± 19%      -0.0        0.13 ± 33%  perf-profile.self.cycles-pp.rcu_all_qs
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.cpuidle_reflect
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.try_to_wake_up
      0.25 ± 14%      -0.0        0.24 ± 23%  perf-profile.self.cycles-pp.do_splice
      0.04 ± 72%      -0.0        0.03 ±101%  perf-profile.self.cycles-pp.update_cfs_group
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.menu_reflect
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.02 ±141%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.update_rt_rq_load_avg
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.cpuidle_not_available
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.account_process_tick
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.pick_next_task_fair
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.update_curr
      0.01 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.20 ±  5%      -0.0        0.19 ± 19%  perf-profile.self.cycles-pp.import_ubuf
      0.03 ±101%      -0.0        0.02 ±146%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.14 ± 15%      -0.0        0.13 ± 34%  perf-profile.self.cycles-pp._copy_from_iter
      0.06 ± 52%      -0.0        0.05 ± 55%  perf-profile.self.cycles-pp.scheduler_tick
      0.07 ± 50%      -0.0        0.06 ± 50%  perf-profile.self.cycles-pp.update_process_times
      0.37 ± 12%      -0.0        0.36 ± 35%  perf-profile.self.cycles-pp.copyin
      0.11 ± 33%      -0.0        0.10 ± 36%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.06 ± 56%      -0.0        0.06 ± 50%  perf-profile.self.cycles-pp.perf_rotate_context
      0.02 ±142%      -0.0        0.02 ±141%  perf-profile.self.cycles-pp.iov_iter_advance
      0.01 ±223%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.sched_idle_set_state
      0.02 ±141%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.clockevents_program_event
      0.09 ± 60%      -0.0        0.09 ± 28%  perf-profile.self.cycles-pp.__x64_sys_vmsplice
      0.12 ± 25%      -0.0        0.12 ± 53%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.01 ±223%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.01 ±223%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.rcu_preempt_deferred_qs
      0.02 ±144%      -0.0        0.02 ±141%  perf-profile.self.cycles-pp.wait_for_space
      0.01 ±223%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.__nanosleep
      0.01 ±223%      -0.0        0.01 ±223%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.03 ±141%      -0.0        0.03 ±147%  perf-profile.self.cycles-pp.call_cpuidle
      0.03 ±100%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.03 ±102%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.pipe_clear_nowait
      0.01 ±223%      +0.0        0.01 ±223%  perf-profile.self.cycles-pp.hrtimer_nanosleep
      0.04 ±104%      +0.0        0.04 ±111%  perf-profile.self.cycles-pp.___perf_sw_event
      0.98 ± 14%      +0.0        0.98 ± 21%  perf-profile.self.cycles-pp.perf_adjust_freq_unthr_context
      0.02 ±141%      +0.0        0.02 ±142%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.01 ±223%      +0.0        0.01 ±223%  perf-profile.self.cycles-pp.rb_next
      0.01 ±223%      +0.0        0.01 ±223%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.01 ±223%      +0.0        0.01 ±223%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.16 ± 31%      +0.0        0.17 ± 40%  perf-profile.self.cycles-pp.get_pipe_info
      0.21 ± 22%      +0.0        0.21 ± 26%  perf-profile.self.cycles-pp._raw_spin_lock
      0.05 ± 81%      +0.0        0.05 ± 48%  perf-profile.self.cycles-pp.idle_cpu
      0.04 ±101%      +0.0        0.05 ± 45%  perf-profile.self.cycles-pp.inode_needs_update_time
      0.09 ± 22%      +0.0        0.09 ± 74%  perf-profile.self.cycles-pp.aa_file_perm
      0.03 ±100%      +0.0        0.04 ± 72%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.11 ± 32%      +0.0        0.12 ± 39%  perf-profile.self.cycles-pp.do_idle
      0.03 ±141%      +0.0        0.04 ±141%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.01 ±223%      +0.0        0.02 ±141%  perf-profile.self.cycles-pp.rebalance_domains
      0.01 ±223%      +0.0        0.02 ±141%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.01 ±223%      +0.0        0.02 ±142%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp._nohz_idle_balance
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.copy_mc_enhanced_fast_string
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.os_xsave
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.tick_nohz_idle_exit
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.check_cpu_stall
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.intel_idle_irq
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.__hrtimer_init
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.newidle_balance
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.set_next_entity
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.perf_swevent_start
      0.02 ±142%      +0.0        0.03 ±100%  perf-profile.self.cycles-pp.error_entry
      0.03 ±100%      +0.0        0.04 ± 71%  perf-profile.self.cycles-pp.__fdget
      0.03 ±101%      +0.0        0.04 ±106%  perf-profile.self.cycles-pp.trigger_load_balance
      0.70 ±  8%      +0.0        0.70 ± 18%  perf-profile.self.cycles-pp.stress_time_now
      0.01 ±223%      +0.0        0.02 ±142%  perf-profile.self.cycles-pp.usleep
      0.01 ±223%      +0.0        0.02 ±141%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.__update_blocked_fair
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.00            +0.0        0.01 ±223%  perf-profile.self.cycles-pp._copy_from_user
      0.03 ±142%      +0.0        0.04 ±107%  perf-profile.self.cycles-pp.load_balance
      0.02 ±144%      +0.0        0.03 ±106%  perf-profile.self.cycles-pp._find_next_and_bit
      0.09 ± 10%      +0.0        0.11 ± 27%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.03 ±103%      +0.0        0.04 ± 75%  perf-profile.self.cycles-pp.update_load_avg
      0.07 ± 50%      +0.0        0.09 ± 18%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.06 ± 80%      +0.0        0.08 ± 22%  perf-profile.self.cycles-pp.__schedule
      0.01 ±223%      +0.0        0.02 ±141%  perf-profile.self.cycles-pp.clock_gettime@plt
      0.12 ± 56%      +0.0        0.14 ± 44%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.04 ± 77%      +0.0        0.06 ± 46%  perf-profile.self.cycles-pp.tick_program_event
      0.01 ±223%      +0.0        0.03 ±102%  perf-profile.self.cycles-pp.__x64_sys_clock_nanosleep
      0.03 ±100%      +0.0        0.05 ± 75%  perf-profile.self.cycles-pp.__fdget_pos
      0.00            +0.0        0.02 ±142%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.0        0.02 ±142%  perf-profile.self.cycles-pp.irq_work_tick
      0.03 ±100%      +0.0        0.05 ± 76%  perf-profile.self.cycles-pp.import_iovec
      0.00            +0.0        0.02 ±144%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.13 ± 33%      +0.0        0.15 ± 24%  perf-profile.self.cycles-pp.splice_from_pipe_next
      0.05 ± 82%      +0.0        0.07 ± 23%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.01 ±223%      +0.0        0.03 ±101%  perf-profile.self.cycles-pp.clock_nanosleep
      0.01 ±223%      +0.0        0.04 ±109%  perf-profile.self.cycles-pp.rcu_pending
      0.00            +0.0        0.02 ±142%  perf-profile.self.cycles-pp.ct_nmi_exit
      0.12 ± 13%      +0.0        0.14 ± 28%  perf-profile.self.cycles-pp.perf_pmu_nop_void
      0.01 ±223%      +0.0        0.04 ±168%  perf-profile.self.cycles-pp.__do_softirq
      0.00            +0.0        0.03 ±147%  perf-profile.self.cycles-pp.get_cpu_device
      0.16 ± 28%      +0.0        0.19 ± 32%  perf-profile.self.cycles-pp.security_file_permission
      0.02 ±142%      +0.0        0.05 ± 46%  perf-profile.self.cycles-pp.pipe_unlock
      0.01 ±223%      +0.0        0.04 ± 73%  perf-profile.self.cycles-pp.kfree
      0.35 ± 16%      +0.0        0.38 ± 21%  perf-profile.self.cycles-pp.menu_select
      0.16 ± 10%      +0.0        0.18 ± 42%  perf-profile.self.cycles-pp.ksys_write
      0.14 ± 23%      +0.0        0.17 ± 39%  perf-profile.self.cycles-pp.lockless_pages_from_mm
      0.86 ± 15%      +0.0        0.89 ± 28%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.01 ±223%      +0.0        0.04 ±117%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      1.03 ± 19%      +0.0        1.06 ± 22%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.51 ± 12%      +0.0        0.54 ± 16%  perf-profile.self.cycles-pp.splice
      0.00            +0.0        0.04 ± 72%  perf-profile.self.cycles-pp.iov_iter_get_pages2
      4.91 ± 18%      +0.0        4.95 ± 21%  perf-profile.self.cycles-pp.gup_pte_range
      0.27 ± 13%      +0.0        0.32 ± 29%  perf-profile.self.cycles-pp.splice_from_pipe
      0.22 ± 25%      +0.1        0.28 ± 40%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.33 ± 18%      +0.1        0.40 ± 25%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.14 ± 51%  perf-profile.self.cycles-pp.folio_fast_pin_allowed
      4.00 ± 16%      +0.2        4.16 ± 17%  perf-profile.self.cycles-pp.try_grab_folio
     33.51 ± 13%      +4.5       38.00 ± 25%  perf-profile.self.cycles-pp.intel_idle



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



View attachment "config-6.4.0-rc4-00185-ga6e79df92e4a" of type "text/plain" (158760 bytes)

View attachment "job-script" of type "text/plain" (8907 bytes)

View attachment "job.yaml" of type "text/plain" (6105 bytes)

View attachment "reproduce" of type "text/plain" (537 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ