lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Tue, 10 Aug 2021 14:42:39 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Bart Van Assche <bvanassche@....org>
Cc:     Jens Axboe <axboe@...nel.dk>, Christoph Hellwig <hch@....de>,
        Ming Lei <ming.lei@...hat.com>,
        Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>,
        Martijn Coenen <maco@...roid.com>,
        Jaegeuk Kim <jaegeuk@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com
Subject: [loop]  2112f5c133: fxmark.hdd_ext4_no_jnl_DWTL_1_directio.works/sec
 -49.6% regression



Greeting,

FYI, we noticed a -49.6% regression of fxmark.hdd_ext4_no_jnl_DWTL_1_directio.works/sec due to commit:


commit: 2112f5c1330a671fa852051d85cb9eadc05d7eb7 ("loop: Select I/O scheduler 'none' from inside add_disk()")
https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git for-5.15/block


in testcase: fxmark
on test machine: 272 threads 2 sockets Intel(R) Xeon Phi(TM) CPU 7255 @ 1.10GHz with 112G memory
with following parameters:

	disk: 1HDD
	media: hdd
	test: DWTL
	fstype: ext4_no_jnl
	directio: directio
	cpufreq_governor: performance
	ucode: 0x11




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/directio/1HDD/ext4_no_jnl/x86_64-rhel-8.3/hdd/debian-10.4-x86_64-20200603.cgz/lkp-knm02/DWTL/fxmark/0x11

commit: 
  90b7198001 ("blk-mq: Introduce the BLK_MQ_F_NO_SCHED_BY_DEFAULT flag")
  2112f5c133 ("loop: Select I/O scheduler 'none' from inside add_disk()")

90b7198001f23ea3 2112f5c1330a671fa852051d85c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      4.86 ±  7%     -14.8%       4.14 ±  2%  fxmark.hdd_ext4_no_jnl_DWTL_18_directio.idle_sec
      0.25 ±  9%     -41.3%       0.15 ±  3%  fxmark.hdd_ext4_no_jnl_DWTL_18_directio.irq_sec
      3.46 ±  3%     -31.2%       2.38 ±  3%  fxmark.hdd_ext4_no_jnl_DWTL_18_directio.irq_util
      0.40 ±  6%     -14.7%       0.34        fxmark.hdd_ext4_no_jnl_DWTL_18_directio.real_sec
      0.06 ±  6%     +68.6%       0.10 ± 15%  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.irq_sec
      4.65 ±  5%     -13.7%       4.02 ±  4%  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.irq_util
      1.25 ±  2%     +95.6%       2.45 ± 14%  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.real_sec
      0.87 ±  3%    +100.3%       1.75 ±  9%  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.secs
     72.74           -40.9%      42.98 ± 11%  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.sys_util
      0.27 ±  3%    +371.3%       1.29 ± 20%  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.user_sec
     21.81 ±  2%    +138.9%      52.10 ±  9%  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.user_util
     27378 ±  3%     -49.6%      13786 ± 10%  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.works/sec
      0.06 ±  8%    +105.9%       0.12 ± 18%  fxmark.hdd_ext4_no_jnl_DWTL_2_directio.irq_sec
      4.15 ±  9%     +57.7%       6.54 ±  7%  fxmark.hdd_ext4_no_jnl_DWTL_2_directio.irq_util
      0.68 ±  4%     +32.5%       0.90 ± 20%  fxmark.hdd_ext4_no_jnl_DWTL_2_directio.real_sec
      0.40 ±  4%     +52.8%       0.61 ± 54%  fxmark.hdd_ext4_no_jnl_DWTL_2_directio.secs
      0.82 ±  4%     +24.1%       1.02 ± 17%  fxmark.hdd_ext4_no_jnl_DWTL_2_directio.sys_sec
      0.29           +58.0%       0.46 ± 61%  fxmark.hdd_ext4_no_jnl_DWTL_2_directio.user_sec
      0.38 ± 11%     +67.0%       0.63 ± 21%  fxmark.hdd_ext4_no_jnl_DWTL_4_directio.idle_sec
     18.84 ±  9%     +61.4%      30.40 ± 17%  fxmark.hdd_ext4_no_jnl_DWTL_4_directio.idle_util
      3.50 ±  2%     +30.6%       4.57 ± 17%  fxmark.hdd_ext4_no_jnl_DWTL_4_directio.irq_util
      0.83           +11.6%       0.93 ±  4%  fxmark.hdd_ext4_no_jnl_DWTL_4_directio.sys_sec
      0.72           -44.9%       0.40 ± 14%  fxmark.hdd_ext4_no_jnl_DWTL_4_directio.user_sec
     35.80 ±  3%     -46.4%      19.18 ± 16%  fxmark.hdd_ext4_no_jnl_DWTL_4_directio.user_util
    202.25            -5.9%     190.41        fxmark.time.elapsed_time
    202.25            -5.9%     190.41        fxmark.time.elapsed_time.max
     24.67 ±  3%     -31.1%      17.00        fxmark.time.percent_of_cpu_this_job_got
     44.77 ±  2%     -40.1%      26.82        fxmark.time.system_time
     46090 ±  2%      +7.6%      49608        fxmark.time.voluntary_context_switches
      1301 ±  2%     +68.1%       2187 ± 46%  meminfo.Active(anon)
      1297 ±  2%     +67.3%       2170 ± 47%  numa-meminfo.node0.Active(anon)
    140.09            -2.2%     137.01        boot-time.boot
     35285            -2.5%      34396        boot-time.idle
    324.33 ±  3%     +65.5%     536.83 ± 47%  numa-vmstat.node0.nr_active_anon
    324.33 ±  3%     +65.5%     536.83 ± 47%  numa-vmstat.node0.nr_zone_active_anon
     18298 ±  8%     -11.7%      16165 ±  2%  softirqs.CPU1.SCHED
     62712           -11.9%      55253 ±  4%  softirqs.TIMER
     43.71            +8.1%      47.24        iostat.cpu.idle
     18.54 ±  4%     -39.9%      11.14 ±  3%  iostat.cpu.system
      4.05 ±  4%     +49.1%       6.04 ±  5%  iostat.cpu.user
      0.56 ±  7%      +0.2        0.81 ±  4%  mpstat.cpu.all.soft%
     18.13 ±  3%      -7.7       10.46 ±  2%  mpstat.cpu.all.sys%
      4.31 ±  4%      +2.1        6.37 ±  5%  mpstat.cpu.all.usr%
    245.33 ± 10%     +43.3%     351.50 ± 41%  vmstat.io.bi
     13969 ±  2%      +7.7%      15048 ±  2%  vmstat.io.bo
      5.50 ± 13%     -51.5%       2.67 ± 27%  vmstat.procs.r
      5398            +7.7%       5815        vmstat.system.cs
    176232 ± 11%      +8.6%     191382        vmstat.system.in
    466.17 ±  2%     -32.8%     313.33 ±  2%  slabinfo.ext4_allocation_context.active_objs
    637.67 ±  2%     -24.7%     480.33        slabinfo.ext4_allocation_context.num_objs
      1106 ±  8%     -26.2%     816.17 ±  7%  slabinfo.ext4_extent_status.active_objs
      1428 ±  5%     -19.8%       1146 ±  5%  slabinfo.ext4_extent_status.num_objs
    673.50 ±  2%     -50.0%     336.67 ± 10%  slabinfo.mbcache.active_objs
    887.50 ±  2%     -41.2%     521.83 ±  4%  slabinfo.mbcache.num_objs
      7695           +22.8%       9449 ± 14%  slabinfo.proc_inode_cache.active_objs
      8031           +21.2%       9736 ± 13%  slabinfo.proc_inode_cache.num_objs
    318.83           +70.8%     544.50 ± 47%  proc-vmstat.nr_active_anon
     28780            +5.1%      30244        proc-vmstat.nr_inactive_file
     11263            -4.7%      10733        proc-vmstat.nr_mapped
     35545            +2.2%      36324        proc-vmstat.nr_slab_reclaimable
     69562            +2.3%      71136        proc-vmstat.nr_slab_unreclaimable
    318.83           +70.8%     544.50 ± 47%  proc-vmstat.nr_zone_active_anon
     28780            +5.1%      30244        proc-vmstat.nr_zone_inactive_file
   1680638            -1.2%    1660900        proc-vmstat.numa_hit
   1680859            -1.2%    1660903        proc-vmstat.numa_local
   1694164            -1.2%    1674369        proc-vmstat.pgalloc_normal
    986109            -5.4%     932819        proc-vmstat.pgfault
   1679614            -1.2%    1659352        proc-vmstat.pgfree
     88729            -3.2%      85929        proc-vmstat.pgreuse
      2.52            +4.9%       2.65 ±  2%  perf-stat.i.MPKI
  70070899 ±  6%      +8.0%   75671172        perf-stat.i.branch-instructions
      0.68 ±  3%      +0.0        0.73        perf-stat.i.branch-miss-rate%
      1.39 ±  3%      +0.1        1.48        perf-stat.i.cache-miss-rate%
   1705485 ±  8%      +8.3%    1847096        perf-stat.i.cache-misses
  10121014 ±  5%      +7.6%   10890953        perf-stat.i.cache-references
      4037           +10.1%       4446        perf-stat.i.context-switches
      0.45 ±  6%      +7.2%       0.48        perf-stat.i.cpi
     85941            +3.3%      88775        perf-stat.i.cpu-clock
 1.778e+09 ± 11%      +9.5%  1.948e+09        perf-stat.i.cpu-cycles
    176.45           -26.0%     130.60        perf-stat.i.cpu-migrations
     88.54 ±  4%      +6.7%      94.50        perf-stat.i.cycles-between-cache-misses
      0.12 ±  7%      +0.0        0.13 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   4737699 ± 12%     +10.7%    5242941        perf-stat.i.iTLB-load-misses
 3.377e+08 ±  6%      +8.0%  3.646e+08        perf-stat.i.iTLB-loads
  3.38e+08 ±  6%      +8.0%  3.651e+08        perf-stat.i.instructions
      0.01 ± 11%      +9.6%       0.01        perf-stat.i.metric.GHz
     38.70 ±  5%      +8.3%      41.89        perf-stat.i.metric.K/sec
      1.49 ±  6%      +8.0%       1.61        perf-stat.i.metric.M/sec
     85941            +3.3%      88775        perf-stat.i.task-clock
  69731811 ±  6%      +8.9%   75910537        perf-stat.ps.branch-instructions
   5670583 ±  9%     +10.3%    6252936        perf-stat.ps.branch-misses
   1699265 ±  7%      +9.2%    1854993 ±  2%  perf-stat.ps.cache-misses
  10086387 ±  5%      +8.5%   10939307        perf-stat.ps.cache-references
      4111            +7.2%       4405        perf-stat.ps.context-switches
     85596            +3.4%      88519        perf-stat.ps.cpu-clock
 1.772e+09 ± 11%     +10.5%  1.957e+09        perf-stat.ps.cpu-cycles
    179.07           -27.5%     129.88        perf-stat.ps.cpu-migrations
   4720160 ± 12%     +11.6%    5265696        perf-stat.ps.iTLB-load-misses
 3.361e+08 ±  6%      +8.8%  3.658e+08        perf-stat.ps.iTLB-loads
 3.364e+08 ±  6%      +8.9%  3.663e+08        perf-stat.ps.instructions
     85596            +3.4%      88519        perf-stat.ps.task-clock
      8494 ± 11%     -38.5%       5226 ± 20%  interrupts.CPU0.CAL:Function_call_interrupts
     13520 ± 11%     -41.6%       7891 ±  8%  interrupts.CPU1.CAL:Function_call_interrupts
      3424 ± 35%     -49.5%       1730 ± 31%  interrupts.CPU10.CAL:Function_call_interrupts
      1616 ± 38%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
      1616 ± 38%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      1239 ± 26%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
      1239 ± 26%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      3649 ± 22%     -48.6%       1876 ± 34%  interrupts.CPU12.CAL:Function_call_interrupts
      1639 ± 40%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
      1639 ± 40%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
      1479 ± 44%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
      1479 ± 44%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
      1680 ± 40%    -100.0%       0.00        interrupts.CPU14.NMI:Non-maskable_interrupts
      1680 ± 40%    -100.0%       0.00        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      4349 ± 28%     -54.0%       2002 ± 31%  interrupts.CPU15.CAL:Function_call_interrupts
      1351 ± 52%    -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
      1351 ± 52%    -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      3473 ± 23%     -47.1%       1839 ± 25%  interrupts.CPU16.CAL:Function_call_interrupts
      1476 ± 24%    -100.0%       0.00        interrupts.CPU16.NMI:Non-maskable_interrupts
      1476 ± 24%    -100.0%       0.00        interrupts.CPU16.PMI:Performance_monitoring_interrupts
      3924 ± 31%     -50.8%       1928 ± 31%  interrupts.CPU17.CAL:Function_call_interrupts
      1570 ± 51%    -100.0%       0.00        interrupts.CPU17.NMI:Non-maskable_interrupts
      1570 ± 51%    -100.0%       0.00        interrupts.CPU17.PMI:Performance_monitoring_interrupts
     10071 ± 12%     -50.8%       4952 ± 26%  interrupts.CPU2.CAL:Function_call_interrupts
      2300 ± 32%     -95.1%     112.83 ±223%  interrupts.CPU2.NMI:Non-maskable_interrupts
      2300 ± 32%     -95.1%     112.83 ±223%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1106 ± 41%     -57.3%     472.50 ± 40%  interrupts.CPU2.RES:Rescheduling_interrupts
      2871 ± 24%     -40.0%       1724 ± 30%  interrupts.CPU24.CAL:Function_call_interrupts
      8191 ± 11%     -43.2%       4651 ± 20%  interrupts.CPU3.CAL:Function_call_interrupts
      2376 ± 30%     -95.8%     100.33 ±223%  interrupts.CPU3.NMI:Non-maskable_interrupts
      2376 ± 30%     -95.8%     100.33 ±223%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    774.33 ± 12%     -25.0%     580.67 ± 18%  interrupts.CPU3.RES:Rescheduling_interrupts
      1494 ± 48%    -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
      1494 ± 48%    -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
     79.50 ± 33%     -74.0%      20.67 ± 32%  interrupts.CPU4.RES:Rescheduling_interrupts
      3652 ± 25%     -44.4%       2030 ± 25%  interrupts.CPU5.CAL:Function_call_interrupts
      1404 ± 28%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
      1404 ± 28%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      3481 ± 24%     -45.6%       1894 ± 26%  interrupts.CPU6.CAL:Function_call_interrupts
      1655 ± 39%    -100.0%       0.00        interrupts.CPU6.NMI:Non-maskable_interrupts
      1655 ± 39%    -100.0%       0.00        interrupts.CPU6.PMI:Performance_monitoring_interrupts
    113.17 ± 74%     -82.3%      20.00 ± 35%  interrupts.CPU6.RES:Rescheduling_interrupts
      1670 ± 39%     -99.9%       1.17 ±223%  interrupts.CPU7.NMI:Non-maskable_interrupts
      1670 ± 39%     -99.9%       1.17 ±223%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
     38787 ± 11%     +14.2%      44280 ±  2%  interrupts.CPU72.LOC:Local_timer_interrupts
      1551 ± 56%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
      1551 ± 56%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      3462 ± 22%     -48.3%       1790 ± 29%  interrupts.CPU9.CAL:Function_call_interrupts
      1528 ± 49%    -100.0%       0.00        interrupts.CPU9.NMI:Non-maskable_interrupts
      1528 ± 49%    -100.0%       0.00        interrupts.CPU9.PMI:Performance_monitoring_interrupts
     31739 ± 16%     -86.0%       4442 ± 15%  interrupts.NMI:Non-maskable_interrupts
     31739 ± 16%     -86.0%       4442 ± 15%  interrupts.PMI:Performance_monitoring_interrupts
      1.52 ± 55%    +634.6%      11.16 ± 66%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.31 ± 44%     -80.0%       0.06 ±137%  perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.01 ± 40%    +698.8%       0.11 ± 51%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.06 ± 15%     -90.8%       0.01 ±178%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.04 ± 11%    +150.0%       0.09 ± 52%  perf-sched.sch_delay.avg.ms.io_schedule.put_and_wait_on_page_locked.filemap_get_pages.filemap_read
      0.21 ± 65%    +236.3%       0.69 ± 55%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.wp_page_copy
      0.00 ±223%   +1768.8%       0.05 ±125%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.page_cache_ra_unbounded.filemap_get_pages
      0.03 ± 35%    +170.9%       0.08 ± 53%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__ext4_journal_get_write_access.ext4_ext_insert_extent.ext4_ext_map_blocks
      0.05 ± 76%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__getblk_gfp.__ext4_sb_bread_gfp.ext4_fill_super
      0.10 ±106%    +240.5%       0.36 ± 71%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.ext4_handle_inode_extension.ext4_dio_write_iter
      0.41 ±  7%     -87.8%       0.05 ± 21%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write
      0.02 ± 42%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.ext4_read_bh.__ext4_sb_bread_gfp.ext4_fill_super
      0.05 ± 43%     -92.0%       0.00 ±223%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.ext4_read_bh.ext4_read_inode_bitmap.__ext4_new_inode
      0.03 ± 18%     +85.4%       0.06 ± 29%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.mempool_alloc.bio_alloc_bioset.iomap_dio_bio_actor
      1.27 ± 57%    +280.4%       4.82 ± 42%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.02 ± 14%    +102.4%       0.04 ± 17%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.iomap_dio_submit_bio
      0.02 ± 55%   +1389.1%       0.34 ±187%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      0.17 ± 76%    +416.5%       0.87 ± 32%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.wp_page_copy.__handle_mm_fault.handle_mm_fault
      0.01 ± 40%    +794.5%       0.11 ± 73%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
      0.23 ± 22%     -65.4%       0.08 ± 25%  perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.25 ± 55%    +193.9%       0.73 ± 44%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.13 ± 83%     -79.1%       0.03 ± 32%  perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
      0.30 ± 10%     -59.6%       0.12 ± 18%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.05 ± 18%     -48.1%       0.03 ± 70%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.sync_rcu_exp_select_cpus
      0.03 ± 11%    +455.4%       0.16 ± 69%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
      0.02 ± 15%     -73.8%       0.01 ±141%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.kthread_stop.destroy_workqueue
      0.03 ± 17%     -75.8%       0.01 ±142%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
      0.16 ±  4%     -53.3%       0.07 ±  8%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.31 ± 59%    +175.0%       0.85 ± 45%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.22 ±  4%     -76.2%       0.05 ±  4%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      3.69 ± 52%    +656.7%      27.91 ± 48%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
      1.60 ± 90%   +1184.8%      20.60 ± 84%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      3.49 ± 22%     -95.1%       0.17 ±197%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.20 ± 55%  +13022.6%      26.40 ±105%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.07 ± 30%    +117.4%       0.16 ± 73%  perf-sched.sch_delay.max.ms.jbd2_log_wait_commit.ext4_sync_file.loop_process_work.process_one_work
      1.51 ± 77%    +400.7%       7.56 ± 41%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.wp_page_copy
      0.01 ±223%   +6246.9%       0.34 ±112%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.page_cache_ra_unbounded.filemap_get_pages
      0.07 ± 98%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__getblk_gfp.__ext4_sb_bread_gfp.ext4_fill_super
      0.51 ± 33%    +390.0%       2.52 ± 62%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__getblk_gfp.ext4_read_block_bitmap_nowait.ext4_read_block_bitmap
      0.02 ±109%   +3500.0%       0.76 ±125%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.security_task_alloc.copy_process
      1.57 ± 54%    +489.0%       9.25 ± 35%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.copy_strings.isra.0
      4.31 ±123%    +390.3%      21.14 ± 51%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.43 ± 97%    +489.0%       2.53 ± 81%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.ext4_handle_inode_extension.ext4_dio_write_iter
      1.08 ±154%    +579.7%       7.37 ± 40%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.unlink_file_vma.free_pgtables
      2.11 ±145%    +520.9%      13.10 ± 89%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.terminate_walk.path_openat
      0.23 ± 30%     -77.7%       0.05 ±118%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.ext4_mb_regular_allocator.ext4_mb_new_blocks.ext4_ext_map_blocks
      0.02 ± 15%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.ext4_read_bh.__ext4_sb_bread_gfp.ext4_fill_super
      0.18 ±192%   +1446.4%       2.78 ±106%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file
      0.66 ±219%    +437.2%       3.57 ±101%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region
      0.62 ± 56%    +298.4%       2.49 ± 62%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.mempool_alloc.bio_alloc_bioset.iomap_dio_bio_actor
      0.40 ±133%   +2056.0%       8.60 ± 58%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.remove_vma.exit_mmap.mmput
      0.01 ±223%   +2655.3%       0.22 ±171%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.ext4_io_submit
      0.43 ±138%   +1118.9%       5.20 ± 48%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.unmap_vmas.exit_mmap.mmput
      1.34 ± 67%    +955.4%      14.18 ± 56%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.wp_page_copy.__handle_mm_fault.handle_mm_fault
      0.08 ±136%   +8653.5%       7.19 ± 55%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
      0.04 ± 14%     -82.9%       0.01 ±141%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      1.92 ± 72%     -90.8%       0.18 ± 58%  perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.submit_bio_wait
      0.05 ± 17%     -47.8%       0.03 ± 70%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.sync_rcu_exp_select_cpus
      0.03 ± 21%    +765.9%       0.30 ± 74%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__synchronize_srcu.part.0
      0.02 ± 15%     -73.2%       0.01 ±141%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.kthread_stop.destroy_workqueue
      0.05 ± 22%     -86.1%       0.01 ±142%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
      0.14 ±  4%     -48.1%       0.07 ± 11%  perf-sched.total_sch_delay.average.ms
     24.72 ±  6%     -73.3%       6.61 ± 10%  perf-sched.total_wait_and_delay.average.ms
     44689 ±  4%     +92.5%      86019 ± 13%  perf-sched.total_wait_and_delay.count.ms
     24.58 ±  6%     -73.4%       6.53 ± 10%  perf-sched.total_wait_time.average.ms
    754.46 ±  9%     -51.8%     363.67 ± 16%  perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    612.74 ± 17%     -38.6%     376.52 ± 43%  perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    253.26 ±  8%     -27.9%     182.65 ± 11%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     40.41 ±  9%     +83.7%      74.23 ±  8%  perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     11.29 ± 21%     -54.1%       5.18 ± 31%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2253 ± 63%     -78.1%     493.54 ±223%  perf-sched.wait_and_delay.avg.ms.kthreadd.ret_from_fork
    292.41 ± 15%     -93.9%      17.77 ± 80%  perf-sched.wait_and_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
     76.88 ± 40%     -62.7%      28.65 ± 17%  perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.89 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_handle_inode_extension
    284.05 ± 47%     -98.7%       3.66 ±139%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
    303.25 ± 21%     -62.8%     112.82 ± 39%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.drm_gem_shmem_vunmap.drm_gem_vunmap
     40.29 ± 10%    +735.1%     336.47 ± 25%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
    249.10 ± 14%     -33.9%     164.76 ± 17%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      1.22 ± 12%     -74.5%       0.31 ±  6%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.iomap_dio_rw
      6.27 ±  8%     -30.2%       4.38 ± 18%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
     74.37 ± 10%     -96.5%       2.62 ± 17%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
     23.77 ±  7%     -84.7%       3.64 ± 16%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     33.17 ± 13%     -53.8%      15.33 ± 19%  perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     32.33 ± 12%     -52.6%      15.33 ± 20%  perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
     16.33 ±159%  +19108.2%       3137 ± 32%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
    609.83 ± 32%    +678.6%       4748 ± 24%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     78.33 ±  5%     -19.6%      63.00 ± 13%  perf-sched.wait_and_delay.count.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      1005 ± 19%     -52.9%     473.33 ± 40%  perf-sched.wait_and_delay.count.io_schedule.bit_wait_io.__wait_on_bit.out_of_line_wait_on_bit
      2.67 ± 27%     -87.5%       0.33 ±223%  perf-sched.wait_and_delay.count.kthreadd.ret_from_fork
     10.67 ± 33%     -90.6%       1.00 ±100%  perf-sched.wait_and_delay.count.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
      2465 ± 30%     +98.1%       4882 ± 17%  perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
      0.50 ±223%   +1466.7%       7.83 ± 66%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__kmalloc.security_task_alloc.copy_process
      0.67 ±223%   +2250.0%      15.67 ± 41%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.copy_pte_range.copy_page_range.dup_mmap
    429.50 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_handle_inode_extension
      0.33 ±223%   +2550.0%       8.83 ± 15%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc.prepare_creds.copy_creds
      2.00 ±141%  +86075.0%       1723 ± 98%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.loop_process_work.process_one_work.worker_thread
      4.67 ± 36%    +135.7%      11.00 ± 15%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.mutex_lock.drm_gem_shmem_vunmap.drm_gem_vunmap
     19.17 ± 70%   +1964.3%     395.67 ± 11%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.mutex_lock.perf_poll.do_sys_poll
      1.17 ±223%   +5385.7%      64.00 ± 15%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.remove_vma.exit_mmap.mmput
      0.50 ±100%    +566.7%       3.33 ± 37%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.shrink_dentry_list.shrink_dcache_parent.d_invalidate
    221.67 ± 16%     -89.8%      22.67 ± 32%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      7.50 ±122%   +1435.6%     115.17 ± 12%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
    315.00 ± 10%     -36.5%     200.00 ± 17%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     16274 ±  5%     +62.0%      26366 ± 18%  perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.iomap_dio_rw
      1600 ±  9%     +32.8%       2124 ± 13%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
     28.00 ± 27%     -80.4%       5.50 ± 70%  perf-sched.wait_and_delay.count.schedule_timeout.wait_for_completion.__wait_rcu_gp.synchronize_rcu
      3945 ±  6%    +305.0%      15978 ±  7%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     12389 ±  4%     +31.1%      16246 ±  8%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
      5957 ±  3%     -43.5%       3364 ± 51%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1008          +478.4%       5834 ± 16%  perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    333.84 ±141%    +213.1%       1045 ±  2%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      5774 ± 60%     -82.9%     987.07 ±223%  perf-sched.wait_and_delay.max.ms.kthreadd.ret_from_fork
    796.54 ± 13%     -97.6%      19.29 ± 74%  perf-sched.wait_and_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
      5960 ±  3%     -53.3%       2782 ± 36%  perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     55.75 ±223%    +647.1%     416.53 ±104%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.security_task_alloc.copy_process
     18.40 ± 52%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_handle_inode_extension
      3040 ± 44%     -86.5%     409.51 ±104%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
    166.68 ±223%    +500.1%       1000        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.remove_vma.exit_mmap.mmput
      6154 ±  9%     -30.7%       4266 ± 39%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    671.45 ± 27%     -71.5%     191.60 ± 36%  perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_for_completion.__wait_rcu_gp.synchronize_rcu
      2765 ±  5%     -35.9%       1772 ± 36%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
    754.23 ±  9%     -51.8%     363.30 ± 16%  perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      7.68 ± 14%     -41.3%       4.50 ± 54%  perf-sched.wait_time.avg.ms.blk_mq_freeze_queue_wait.lo_release.blkdev_put.blkdev_close
    612.42 ± 17%     -38.5%     376.46 ± 43%  perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    253.17 ±  8%     -27.9%     182.47 ± 11%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     40.21 ±  9%     +84.2%      74.07 ±  7%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     11.21 ± 21%     -54.1%       5.15 ± 31%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2252 ± 63%     -78.1%     493.53 ±223%  perf-sched.wait_time.avg.ms.kthreadd.ret_from_fork
    292.06 ± 15%     -93.5%      18.98 ± 65%  perf-sched.wait_time.avg.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
     76.81 ± 40%     -62.8%      28.59 ± 17%  perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.02 ±223%   +2140.4%       0.43 ± 41%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.__pmd_alloc.__handle_mm_fault
    130.32 ±135%     -98.6%       1.84 ±108%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.shmem_alloc_page
      0.27 ± 91%   +7486.5%      20.55 ±132%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.wp_page_copy
      0.03 ±223%   +1184.5%       0.42 ± 56%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.page_cache_ra_unbounded.filemap_get_pages
      0.08 ±193%    +553.1%       0.54 ± 45%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault
      0.02 ±223%   +2297.2%       0.42 ± 93%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__anon_vma_prepare.do_fault.__handle_mm_fault
      0.02 ±104%   +5056.0%       1.21 ±101%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__ext4_handle_dirty_metadata.ext4_free_blocks.ext4_remove_blocks
      0.31 ± 14%     -22.4%       0.24 ± 14%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__ext4_handle_dirty_metadata.ext4_mb_mark_diskspace_used.ext4_mb_new_blocks
      0.51 ± 17%     -33.1%       0.34 ± 22%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc
      0.38 ± 20%     -33.5%       0.25 ± 16%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__getblk_gfp.ext4_read_block_bitmap_nowait.ext4_read_block_bitmap
      0.05 ±168%   +2659.9%       1.30 ±120%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__invalidate_mapping_pages.drop_pagecache_sb.iterate_supers
      0.48 ± 28%     -49.4%       0.24 ± 30%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.ext4_find_extent.ext4_ext_map_blocks
      0.02 ±143%   +3488.2%       0.66 ±167%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.filemap_read.blkdev_read_iter
      0.01 ±146%   +2977.0%       0.45 ±108%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.do_sys_ftruncate.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.02 ±223%   +2993.0%       0.66 ± 77%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.anon_vma_clone.__split_vma
      0.55 ±191%   +3156.2%      17.97 ± 98%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.unlink_anon_vmas.free_pgtables
      0.11 ± 87%  +67368.9%      73.88 ± 80%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.step_into.path_openat
     13.55 ± 81%     -64.5%       4.81 ±127%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.terminate_walk.path_openat
      0.11 ±223%    +891.1%       1.12 ±131%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.walk_component.link_path_walk
      0.05 ±144%    +501.3%       0.31 ± 25%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.exit_mmap.mmput.begin_new_exec
      0.12 ±172%  +53070.5%      61.86 ±103%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.exit_mmap.mmput.do_exit
      0.47 ± 10%     -41.4%       0.28 ± 34%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_dio_write_iter
      0.47 ±  6%     -30.6%       0.33 ± 19%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_iomap_begin
      0.32 ± 26%     -55.6%       0.14 ± 77%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.ext4_mb_regular_allocator.ext4_mb_new_blocks.ext4_ext_map_blocks
      0.19 ± 72%    +202.1%       0.58 ± 71%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.filemap_read.new_sync_read.vfs_read
    284.05 ± 47%     -98.0%       5.55 ± 74%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      0.02 ±108%  +7.5e+05%     169.09 ±220%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.dup_mm.copy_process
      0.10 ±131%    +548.8%       0.66 ± 77%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma
      0.00 ±223%   +4311.5%       0.19 ± 49%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mmput.begin_new_exec.load_elf_binary
      0.01 ±223%   +1388.1%       0.15 ±  8%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.__fdget_pos.ksys_write
      0.01 ±223%   +6190.0%       0.52 ± 89%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.__perf_event_read_value.perf_read
    299.44 ± 21%     -63.5%     109.18 ± 40%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.drm_gem_shmem_vunmap.drm_gem_vunmap
      0.07 ±129%  +16966.2%      11.60 ±155%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.perf_event_ctx_lock_nested.isra
      0.07 ± 21%   +6160.4%       4.66 ±200%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.slub_cpu_dead.cpuhp_invoke_callback
      1.07 ±  8%   +1239.0%      14.36 ± 89%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.zone_pcp_update.page_alloc_cpu_dead
      0.02 ±223%   +9766.7%       2.47 ±142%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_write_begin
      0.16 ±128%    +171.4%       0.43 ± 34%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
      0.89 ± 11%     -77.9%       0.20 ± 42%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.29 ±  8%     -22.1%       0.22 ±  8%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.submit_bio_checks.submit_bio_noacct.iomap_dio_submit_bio
     40.27 ± 10%    +734.7%     336.12 ± 24%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      4.15 ±  5%     -55.9%       1.83 ± 31%  perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    248.93 ± 14%     -33.9%     164.46 ± 17%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      1.16 ± 12%     -78.3%       0.25 ±  6%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.iomap_dio_rw
      5.98 ±  8%     -28.7%       4.26 ± 19%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.44 ± 75%     -92.0%       0.04 ±115%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.__flush_work._cpu_down
     21.62 ± 23%     -66.9%       7.15 ± 85%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.cpuhp_kick_ap.cpuhp_kick_ap_work
      6.61 ± 11%     -29.0%       4.69 ± 20%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.devtmpfs_submit_req.devtmpfs_delete_node
      0.16 ± 16%     -64.1%       0.06 ±141%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.kthread_stop.destroy_workqueue
     74.21 ± 10%     -96.6%       2.55 ± 17%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
     23.55 ±  7%     -84.8%       3.59 ± 16%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
      5957 ±  3%     -43.6%       3362 ± 51%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1008          +478.4%       5834 ± 16%  perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.89 ± 90%   +1187.8%      11.46 ± 43%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_common_interrupt.[unknown]
    192.98 ±189%    +338.3%     845.79 ± 43%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
    489.10 ± 86%    +113.7%       1045 ±  2%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     13.04 ±  6%     -65.5%       4.49 ±109%  perf-sched.wait_time.max.ms.io_schedule.__lock_page.truncate_inode_pages_range.blkdev_flush_mapping
      1.05 ± 30%     -91.2%       0.09 ±148%  perf-sched.wait_time.max.ms.io_schedule.__lock_page_killable.filemap_fault.__do_fault
      5774 ± 60%     -82.9%     987.06 ±223%  perf-sched.wait_time.max.ms.kthreadd.ret_from_fork
    796.19 ± 13%     -97.5%      20.19 ± 60%  perf-sched.wait_time.max.ms.percpu_rwsem_wait.__percpu_down_read.cpus_read_lock.rebuild_sched_domains
      5960 ±  3%     -53.4%       2779 ± 36%  perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.12 ±135%  +1.4e+05%     167.33 ±223%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.__get_free_pages.pgd_alloc
      0.04 ±223%   +1803.9%       0.72 ± 47%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.__pmd_alloc.__handle_mm_fault
      1.31 ± 94%  +28723.2%     376.91 ±119%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.wp_page_copy
      0.06 ±223%   +4565.0%       3.02 ±137%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.page_cache_ra_unbounded.filemap_get_pages
      0.13 ±196%   +1741.6%       2.42 ± 71%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault
      0.04 ±223%   +3487.7%       1.27 ±115%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__anon_vma_prepare.do_fault.__handle_mm_fault
      0.02 ±104%  +13173.8%       3.12 ± 77%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__ext4_handle_dirty_metadata.ext4_free_blocks.ext4_remove_blocks
      0.27 ± 83%     -84.9%       0.04 ±223%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__ext4_mark_inode_dirty.ext4_ext_insert_extent.ext4_ext_map_blocks
      0.15 ±172%   +3234.6%       5.16 ±105%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__invalidate_mapping_pages.drop_pagecache_sb.iterate_supers
      1.57 ±208%  +22486.7%     354.23 ±130%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.security_prepare_creds.prepare_creds
     57.49 ±215%    +623.7%     416.08 ±104%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.security_task_alloc.copy_process
      0.58 ±177%   +1283.8%       8.03 ± 38%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.change_p4d_range.change_protection.mprotect_fixup
      0.02 ±143%  +27957.3%       5.14 ±207%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.filemap_read.blkdev_read_iter
      0.02 ±148%  +13329.7%       2.04 ±129%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.do_sys_ftruncate.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.05 ±223%   +1013.5%       0.54 ± 66%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read_killable.create_elf_tables.isra
      0.37 ± 60%    +231.6%       1.23 ± 59%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.__vma_adjust.shift_arg_pages
      0.02 ±223%   +7813.3%       1.69 ± 86%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.anon_vma_clone.__split_vma
      8.69 ±146%   +2648.3%     238.94 ±181%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.dup_mmap.dup_mm
      0.63 ±167%  +79960.2%     500.51 ± 99%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.unlink_anon_vmas.free_pgtables
     18.45 ±218%   +1476.9%     290.95 ±122%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.unlink_file_vma.free_pgtables
      0.14 ±223%    +389.7%       0.71 ± 61%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_brk_flags.load_elf_binary
      0.70 ±113%    +276.2%       2.65 ± 44%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_mmap_pgoff.elf_map
      1.17 ±140%    +207.3%       3.58 ± 52%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
      0.24 ±118%  +2.8e+05%     668.00 ± 70%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.step_into.path_openat
      0.11 ±223%   +1250.7%       1.52 ± 97%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.walk_component.link_path_walk
      0.11 ±176%    +755.4%       0.95 ± 39%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.exit_mmap.mmput.begin_new_exec
      0.14 ±153%  +3.7e+05%     500.23 ± 99%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.exit_mmap.mmput.do_exit
      2.60 ± 35%     -62.7%       0.97 ± 69%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_dio_write_iter
      0.31 ±142%    +496.3%       1.87 ±116%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_ext_remove_space
     16.02 ± 68%     -74.4%       4.10 ± 74%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_handle_inode_extension
      1.98 ± 19%     -90.2%       0.19 ± 95%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.ext4_mb_regular_allocator.ext4_mb_new_blocks.ext4_ext_map_blocks
      0.29 ± 79%    +529.9%       1.81 ± 79%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.filemap_read.new_sync_read.vfs_read
      3040 ± 44%     -76.9%     702.86 ± 40%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      9.26 ±164%   +6517.0%     612.75 ± 58%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file
      0.24 ±201%    +880.9%       2.33 ±116%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__d_alloc.d_alloc
      0.02 ±108%  +7.6e+05%     172.87 ±214%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.dup_mm.copy_process
      7.98 ±203%   +3034.4%     250.17 ±142%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.getname_flags.do_sys_openat2
      0.44 ±138%    +281.0%       1.67 ± 72%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.getname_flags.user_path_at_empty
      0.27 ±150%   +2339.7%       6.58 ± 80%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma
      0.00 ±223%   +6596.2%       0.29 ± 52%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mmput.begin_new_exec.load_elf_binary
      0.10 ±223%   +1457.8%       1.48 ±148%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mnt_want_write.filename_create.do_mkdirat
      0.01 ±223%   +4871.2%       0.49 ± 60%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.__fdget_pos.ksys_write
      0.01 ±223%  +16114.0%       1.35 ±124%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.__perf_event_read_value.perf_read
      0.22 ±146%  +1.5e+05%     334.67 ±140%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.perf_event_ctx_lock_nested.isra
      2.66 ± 16%    +612.1%      18.93 ± 85%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.zone_pcp_update.page_alloc_cpu_dead
      0.02 ±223%  +1.4e+06%     341.22 ±136%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_write_begin
      0.24 ±138%    +673.7%       1.89 ± 45%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
    179.86 ±204%    +456.1%       1000        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.remove_vma.exit_mmap.mmput
      4.57 ± 27%     -68.0%       1.46 ± 47%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.40 ±188%   +2661.7%      11.02 ±159%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      6154 ±  9%     -30.7%       4266 ± 39%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      5.13 ± 85%     -99.0%       0.05 ±144%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.__flush_work._cpu_down
    671.12 ± 27%     -71.5%     191.20 ± 36%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.__wait_rcu_gp.synchronize_rcu
     47.93 ± 29%     -79.9%       9.62 ± 79%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.cpuhp_kick_ap.cpuhp_kick_ap_work
     25.76 ± 56%     -64.0%       9.28 ± 31%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.devtmpfs_submit_req.devtmpfs_delete_node
      2765 ±  6%     -35.9%       1772 ± 36%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     72.36 ±  5%     -68.1        4.23 ± 10%  perf-profile.calltrace.cycles-pp.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list.blk_finish_plug.__iomap_dio_rw
     72.38 ±  5%     -68.1        4.32 ±  9%  perf-profile.calltrace.cycles-pp.blk_mq_flush_plug_list.blk_flush_plug_list.blk_finish_plug.__iomap_dio_rw.iomap_dio_rw
     72.39 ±  5%     -68.0        4.36 ±  9%  perf-profile.calltrace.cycles-pp.blk_flush_plug_list.blk_finish_plug.__iomap_dio_rw.iomap_dio_rw.ext4_dio_write_iter
     72.40 ±  5%     -68.0        4.37 ±  9%  perf-profile.calltrace.cycles-pp.blk_finish_plug.__iomap_dio_rw.iomap_dio_rw.ext4_dio_write_iter.new_sync_write
     76.11 ±  5%     -58.2       17.92 ±  6%  perf-profile.calltrace.cycles-pp.iomap_dio_rw.ext4_dio_write_iter.new_sync_write.vfs_write.ksys_write
     76.11 ±  5%     -58.2       17.93 ±  6%  perf-profile.calltrace.cycles-pp.__iomap_dio_rw.iomap_dio_rw.ext4_dio_write_iter.new_sync_write.vfs_write
     76.68 ±  5%     -56.9       19.81 ±  5%  perf-profile.calltrace.cycles-pp.ext4_dio_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     52.61 ±  5%     -52.6        0.00        perf-profile.calltrace.cycles-pp.__blk_mq_delay_run_hw_queue.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list.blk_finish_plug
     52.60 ±  5%     -52.6        0.00        perf-profile.calltrace.cycles-pp.__blk_mq_run_hw_queue.__blk_mq_delay_run_hw_queue.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list
     52.60 ±  5%     -52.6        0.00        perf-profile.calltrace.cycles-pp.blk_mq_sched_dispatch_requests.__blk_mq_run_hw_queue.__blk_mq_delay_run_hw_queue.blk_mq_sched_insert_requests.blk_mq_flush_plug_list
     52.59 ±  5%     -52.6        0.00        perf-profile.calltrace.cycles-pp.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.__blk_mq_run_hw_queue.__blk_mq_delay_run_hw_queue.blk_mq_sched_insert_requests
     52.58 ±  5%     -52.6        0.00        perf-profile.calltrace.cycles-pp.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.__blk_mq_run_hw_queue.__blk_mq_delay_run_hw_queue
     51.92 ±  5%     -51.6        0.35 ±100%  perf-profile.calltrace.cycles-pp.dd_dispatch_request.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.__blk_mq_run_hw_queue
     78.17 ±  5%     -50.8       27.37 ±  2%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     44.70 ±  7%     -44.7        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock.dd_dispatch_request.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests
     44.67 ±  7%     -44.7        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.dd_dispatch_request.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests
     19.69 ±  6%     -19.7        0.00        perf-profile.calltrace.cycles-pp.dd_insert_requests.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list.blk_finish_plug
     19.59 ±  6%     -19.6        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock.dd_insert_requests.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list
     19.58 ±  6%     -19.6        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.dd_insert_requests.blk_mq_sched_insert_requests.blk_mq_flush_plug_list
      7.01 ± 20%      -6.7        0.32 ±100%  perf-profile.calltrace.cycles-pp.dd_queued.dd_dispatch_request.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests
      2.89 ±  8%      -2.4        0.50 ± 73%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      2.38 ± 16%      -2.0        0.35 ±102%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      3.05 ±  7%      -1.8        1.23 ± 63%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.05 ±  7%      -1.8        1.23 ± 63%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
      3.04 ±  8%      -1.8        1.23 ± 63%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.42 ±  9%      -1.2        2.22 ± 33%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      1.59 ±  7%      -1.0        0.56 ± 77%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.54 ±  7%      -1.0        0.55 ± 78%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.6        0.62 ± 11%  perf-profile.calltrace.cycles-pp.proc_pid_status.proc_single_show.seq_read_iter.seq_read.vfs_read
      0.00            +0.6        0.63 ± 12%  perf-profile.calltrace.cycles-pp.proc_single_show.seq_read_iter.seq_read.vfs_read.ksys_read
      0.00            +0.7        0.68 ± 19%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      0.00            +0.7        0.71 ± 10%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write
      0.00            +0.7        0.73 ± 20%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.20 ±141%      +0.7        0.93 ± 32%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
      0.20 ±141%      +0.7        0.94 ± 33%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
      0.20 ±141%      +0.7        0.94 ± 33%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify
      0.00            +0.7        0.74 ± 17%  perf-profile.calltrace.cycles-pp.__ext4_mark_inode_dirty.ext4_handle_inode_extension.ext4_dio_write_iter.new_sync_write.vfs_write
      0.00            +0.8        0.77 ± 19%  perf-profile.calltrace.cycles-pp.read_counters.process_interval.dispatch_events.cmd_stat.run_builtin
      0.00            +0.8        0.79 ± 18%  perf-profile.calltrace.cycles-pp.process_interval.dispatch_events.cmd_stat.run_builtin.main
      0.00            +0.8        0.79 ± 17%  perf-profile.calltrace.cycles-pp.cmd_stat.run_builtin.main.__libc_start_main
      0.00            +0.8        0.79 ± 17%  perf-profile.calltrace.cycles-pp.dispatch_events.cmd_stat.run_builtin.main.__libc_start_main
      0.00            +0.8        0.79 ± 21%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.smpboot_thread_fn
      0.00            +0.8        0.80 ± 11%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00            +0.8        0.80 ± 11%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      0.00            +0.8        0.80 ± 18%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.81 ± 18%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.82 ± 25%  perf-profile.calltrace.cycles-pp.xas_find.find_lock_entries.__invalidate_mapping_pages.drop_pagecache_sb.iterate_supers
      0.00            +0.8        0.83 ± 20%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.smpboot_thread_fn.kthread
      0.00            +0.8        0.84 ± 15%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.raise_softirq
      0.00            +0.8        0.85 ± 26%  perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.00            +0.8        0.85 ± 26%  perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.00            +0.9        0.85 ± 25%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.00            +0.9        0.85 ± 25%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.00            +0.9        0.87 ± 14%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.raise_softirq
      0.00            +0.9        0.87 ± 30%  perf-profile.calltrace.cycles-pp.rcu_gp_kthread.kthread.ret_from_fork
      0.00            +0.9        0.88 ± 24%  perf-profile.calltrace.cycles-pp.setlocale
      0.00            +0.9        0.89 ± 16%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.raise_softirq.blk_mq_complete_request_remote
      0.00            +0.9        0.91 ± 35%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.iomap_dio_bio_end_io
      0.00            +0.9        0.92 ± 20%  perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64
      0.00            +0.9        0.92 ± 11%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.raise_softirq.blk_mq_complete_request_remote
      0.00            +0.9        0.93 ± 16%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.raise_softirq.blk_mq_complete_request_remote.blk_mq_complete_request
      0.00            +0.9        0.95 ± 22%  perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        0.95 ± 33%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.iomap_dio_bio_end_io.blk_update_request
      0.00            +1.0        0.95 ± 41%  perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.new_sync_write.vfs_write
      0.00            +1.0        0.96 ±  9%  perf-profile.calltrace.cycles-pp.seq_read_iter.seq_read.vfs_read.ksys_read.do_syscall_64
      0.00            +1.0        0.96 ± 12%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.raise_softirq.blk_mq_complete_request_remote.blk_mq_complete_request
      0.00            +1.0        0.97 ± 15%  perf-profile.calltrace.cycles-pp.ext4_da_write_end.generic_perform_write.ext4_buffered_write_iter.do_iter_readv_writev.do_iter_write
      0.00            +1.0        0.97 ±  8%  perf-profile.calltrace.cycles-pp.seq_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        0.97 ± 15%  perf-profile.calltrace.cycles-pp.ext4_handle_inode_extension.ext4_dio_write_iter.new_sync_write.vfs_write.ksys_write
      0.00            +1.0        0.98 ± 30%  perf-profile.calltrace.cycles-pp.ext4_mb_mark_diskspace_used.ext4_mb_new_blocks.ext4_ext_map_blocks.ext4_map_blocks.ext4_iomap_begin
      0.00            +1.0        0.98 ± 33%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.iomap_dio_bio_end_io.blk_update_request.blk_mq_end_request
      0.00            +1.0        0.98 ± 39%  perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.new_sync_write.vfs_write.ksys_write
      0.46 ± 72%      +1.0        1.44 ± 46%  perf-profile.calltrace.cycles-pp.machines__deliver_event.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event.process_simple
      0.00            +1.0        0.98 ± 26%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.worker_thread
      0.27 ±100%      +1.0        1.26 ± 35%  perf-profile.calltrace.cycles-pp.__dso__load_kallsyms.dso__load.map__load.thread__find_map.build_id__mark_dso_hit
      0.00            +1.0        0.99 ± 31%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.iomap_dio_bio_end_io.blk_update_request
      0.28 ±100%      +1.0        1.28 ± 36%  perf-profile.calltrace.cycles-pp.map__load.thread__find_map.build_id__mark_dso_hit.machines__deliver_event.perf_session__deliver_event
      0.28 ±100%      +1.0        1.27 ± 36%  perf-profile.calltrace.cycles-pp.dso__load.map__load.thread__find_map.build_id__mark_dso_hit.machines__deliver_event
      0.00            +1.0        1.00 ± 39%  perf-profile.calltrace.cycles-pp.generic_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.29 ±100%      +1.0        1.30 ± 37%  perf-profile.calltrace.cycles-pp.thread__find_map.build_id__mark_dso_hit.machines__deliver_event.perf_session__deliver_event.__ordered_events__flush
      0.00            +1.0        1.01 ± 21%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      0.08 ±223%      +1.0        1.10 ± 37%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      0.00            +1.0        1.01 ± 31%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.iomap_dio_bio_end_io.blk_update_request.blk_mq_end_request
      0.00            +1.0        1.02 ± 23%  perf-profile.calltrace.cycles-pp.__libc_fork
      0.00            +1.0        1.02 ± 20%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
      0.33 ±100%      +1.0        1.36 ± 41%  perf-profile.calltrace.cycles-pp.build_id__mark_dso_hit.machines__deliver_event.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event
      0.00            +1.0        1.03 ± 21%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
      0.00            +1.0        1.03 ± 20%  perf-profile.calltrace.cycles-pp.perf_poll.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        1.03 ± 27%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.worker_thread.kthread
      0.00            +1.1        1.06 ± 19%  perf-profile.calltrace.cycles-pp.kallsyms__parse.__dso__load_kallsyms.dso__load.map__load.thread__find_map
      0.26 ±100%      +1.1        1.33 ± 33%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      0.26 ±100%      +1.1        1.34 ± 32%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      0.00            +1.1        1.09 ± 33%  perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate
      0.26 ±100%      +1.1        1.36 ± 32%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
      0.26 ±100%      +1.1        1.36 ± 32%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      0.00            +1.1        1.09 ± 24%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
      0.27 ±100%      +1.1        1.38 ± 30%  perf-profile.calltrace.cycles-pp.__libc_read
      0.00            +1.1        1.14 ±  7%  perf-profile.calltrace.cycles-pp.bio_iov_iter_get_pages.iomap_dio_bio_actor.iomap_apply.__iomap_dio_rw.iomap_dio_rw
      0.08 ±223%      +1.2        1.27 ± 42%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      0.00            +1.2        1.19 ± 10%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.2        1.19 ± 10%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.2        1.19 ± 10%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.2        1.20 ± 23%  perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.new_sync_read.vfs_read
      0.00            +1.2        1.21 ± 23%  perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
      0.08 ±223%      +1.2        1.30 ± 43%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      0.08 ±223%      +1.2        1.31 ± 42%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
      0.00            +1.2        1.24 ± 20%  perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64
      0.00            +1.2        1.24 ± 21%  perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.00            +1.2        1.25 ± 33%  perf-profile.calltrace.cycles-pp.seq_read_iter.seq_read.proc_reg_read.vfs_read.ksys_read
      0.41 ± 70%      +1.3        1.66 ± 16%  perf-profile.calltrace.cycles-pp.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.do_iter_readv_writev
      0.00            +1.3        1.25 ± 32%  perf-profile.calltrace.cycles-pp.seq_read.proc_reg_read.vfs_read.ksys_read.do_syscall_64
      0.00            +1.3        1.26 ± 33%  perf-profile.calltrace.cycles-pp.proc_reg_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.3        1.28 ± 20%  perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64.nsinfo__new
      0.00            +1.3        1.28 ± 26%  perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.09 ±223%      +1.3        1.39 ± 43%  perf-profile.calltrace.cycles-pp.__libc_write
      0.00            +1.3        1.32 ± 20%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64.nsinfo__new.thread__new
      0.00            +1.3        1.32 ± 20%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__xstat64.nsinfo__new.thread__new.____machine__findnew_thread
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.calltrace.cycles-pp.drm_fb_memcpy_dstclip.mgag200_simple_display_pipe_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail.commit_tail
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_helper_dirtyfb.drm_fb_helper_damage_work
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_helper_dirtyfb
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.calltrace.cycles-pp.mgag200_simple_display_pipe_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail.commit_tail.drm_atomic_helper_commit
      0.00            +1.4        1.36 ± 19%  perf-profile.calltrace.cycles-pp.__xstat64.nsinfo__new.thread__new.____machine__findnew_thread.machine__process_fork_event
      0.89 ± 13%      +1.4        2.25 ± 17%  perf-profile.calltrace.cycles-pp.commit_tail.drm_atomic_helper_commit.drm_atomic_helper_dirtyfb.drm_fb_helper_damage_work.process_one_work
      0.89 ± 13%      +1.4        2.26 ± 17%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit.drm_atomic_helper_dirtyfb.drm_fb_helper_damage_work.process_one_work.worker_thread
      0.00            +1.4        1.38 ± 17%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.90 ± 13%      +1.4        2.29 ± 17%  perf-profile.calltrace.cycles-pp.drm_atomic_helper_dirtyfb.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread
      0.00            +1.4        1.40 ± 18%  perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      0.93 ± 13%      +1.4        2.33 ± 17%  perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.10 ±223%      +1.5        1.55 ± 16%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +1.5        1.46 ± 22%  perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      0.00            +1.5        1.47 ± 17%  perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      0.00            +1.5        1.50 ± 30%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.__queue_work
      0.00            +1.5        1.50 ± 23%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.schedule_timeout
      0.08 ±223%      +1.5        1.58 ±  8%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.do_iter_readv_writev
      0.00            +1.5        1.53 ± 20%  perf-profile.calltrace.cycles-pp.nsinfo__new.thread__new.____machine__findnew_thread.machine__process_fork_event.machines__deliver_event
      0.00            +1.5        1.54 ± 19%  perf-profile.calltrace.cycles-pp.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
      0.00            +1.5        1.54 ± 19%  perf-profile.calltrace.cycles-pp.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
      0.00            +1.5        1.54 ± 18%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__poll
      0.00            +1.5        1.54 ± 18%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
      0.00            +1.5        1.55 ±  6%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter
      0.00            +1.6        1.55 ± 18%  perf-profile.calltrace.cycles-pp.__poll
      0.00            +1.6        1.57 ± 22%  perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.schedule_timeout.io_schedule_timeout
      0.00            +1.6        1.57 ± 30%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.__queue_work.queue_work_on
      0.00            +1.6        1.60 ± 19%  perf-profile.calltrace.cycles-pp.thread__new.____machine__findnew_thread.machine__process_fork_event.machines__deliver_event.perf_session__deliver_event
      0.00            +1.6        1.60 ± 30%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.__queue_work.queue_work_on.loop_queue_rq
      0.00            +1.6        1.60 ± 17%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.6        1.64 ± 19%  perf-profile.calltrace.cycles-pp.____machine__findnew_thread.machine__process_fork_event.machines__deliver_event.perf_session__deliver_event.process_simple
      0.00            +1.7        1.68 ± 19%  perf-profile.calltrace.cycles-pp.machine__process_fork_event.machines__deliver_event.perf_session__deliver_event.process_simple.perf_session__process_events
      0.00            +1.7        1.70 ± 16%  perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +1.7        1.71 ± 16%  perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork
      0.10 ±223%      +1.7        1.84 ± 14%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.11 ±223%      +1.8        1.90 ± 15%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      0.00            +1.8        1.80 ± 55%  perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.cmd_record.cmd_sched.run_builtin
      0.00            +1.8        1.84 ± 18%  perf-profile.calltrace.cycles-pp.machines__deliver_event.perf_session__deliver_event.process_simple.perf_session__process_events.record__finish_output
      0.00            +1.9        1.86 ± 24%  perf-profile.calltrace.cycles-pp.find_lock_entries.__invalidate_mapping_pages.drop_pagecache_sb.iterate_supers.drop_caches_sysctl_handler
      0.00            +1.9        1.87 ± 19%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.9        1.87 ± 18%  perf-profile.calltrace.cycles-pp.perf_session__deliver_event.process_simple.perf_session__process_events.record__finish_output.cmd_record
      0.40 ± 71%      +1.9        2.32 ± 26%  perf-profile.calltrace.cycles-pp.ext4_free_blocks.ext4_remove_blocks.ext4_ext_rm_leaf.ext4_ext_remove_space.ext4_ext_truncate
      0.08 ±223%      +1.9        2.01 ± 16%  perf-profile.calltrace.cycles-pp.execve
      0.11 ±223%      +1.9        2.04 ± 14%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      0.72 ± 30%      +1.9        2.66 ± 13%  perf-profile.calltrace.cycles-pp.ext4_mb_new_blocks.ext4_ext_map_blocks.ext4_map_blocks.ext4_iomap_begin.iomap_apply
      0.00            +2.0        1.96 ± 38%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch
      0.00            +2.0        1.99 ± 15%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.00            +2.0        2.00 ± 16%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      0.00            +2.0        2.00 ± 16%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.00            +2.0        2.00 ± 16%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.00            +2.0        2.00 ± 38%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
      0.00            +2.0        2.00 ± 38%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
      0.40 ± 71%      +2.0        2.42 ± 25%  perf-profile.calltrace.cycles-pp.ext4_remove_blocks.ext4_ext_rm_leaf.ext4_ext_remove_space.ext4_ext_truncate.ext4_truncate
      0.00            +2.1        2.06 ±  8%  perf-profile.calltrace.cycles-pp.try_to_wake_up.raise_softirq.blk_mq_complete_request_remote.blk_mq_complete_request.loop_process_work
      0.00            +2.1        2.11 ± 55%  perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.cmd_record.cmd_sched.run_builtin.main
      0.00            +2.1        2.13 ±  8%  perf-profile.calltrace.cycles-pp.raise_softirq.blk_mq_complete_request_remote.blk_mq_complete_request.loop_process_work.process_one_work
      0.08 ±223%      +2.2        2.28 ± 17%  perf-profile.calltrace.cycles-pp.__schedule.schedule.worker_thread.kthread.ret_from_fork
      0.00            +2.2        2.21 ±  7%  perf-profile.calltrace.cycles-pp.blk_mq_complete_request_remote.blk_mq_complete_request.loop_process_work.process_one_work.worker_thread
      0.00            +2.2        2.22 ±  7%  perf-profile.calltrace.cycles-pp.blk_mq_complete_request.loop_process_work.process_one_work.worker_thread.kthread
      0.00            +2.2        2.23 ± 31%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      0.08 ±223%      +2.3        2.34 ± 16%  perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork
      1.19 ± 51%      +2.3        3.47 ± 26%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.memcpy_toio.drm_fb_memcpy_dstclip.mgag200_simple_display_pipe_update.drm_atomic_helper_commit_planes
      0.54 ±  6%      +2.3        2.83 ± 10%  perf-profile.calltrace.cycles-pp.iomap_dio_bio_actor.iomap_apply.__iomap_dio_rw.iomap_dio_rw.ext4_dio_write_iter
      0.00            +2.4        2.36 ± 30%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.io_schedule_timeout
      1.36 ± 24%      +2.4        3.74 ± 24%  perf-profile.calltrace.cycles-pp.process_simple.perf_session__process_events.record__finish_output.cmd_record.cmd_sched
      1.50 ± 24%      +2.4        3.91 ± 28%  perf-profile.calltrace.cycles-pp.perf_session__process_events.record__finish_output.cmd_record.cmd_sched.run_builtin
      1.52 ± 24%      +2.4        3.94 ± 28%  perf-profile.calltrace.cycles-pp.record__finish_output.cmd_record.cmd_sched.run_builtin.main
      0.47 ± 71%      +2.5        2.98 ± 24%  perf-profile.calltrace.cycles-pp.ext4_ext_rm_leaf.ext4_ext_remove_space.ext4_ext_truncate.ext4_truncate.ext4_setattr
      1.33 ± 24%      +2.5        3.85 ± 14%  perf-profile.calltrace.cycles-pp.ext4_ext_map_blocks.ext4_map_blocks.ext4_iomap_begin.iomap_apply.__iomap_dio_rw
      0.00            +2.5        2.54 ± 18%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__queue_work.queue_work_on.loop_queue_rq.__blk_mq_try_issue_directly
      0.00            +2.6        2.58 ± 32%  perf-profile.calltrace.cycles-pp.try_to_wake_up.iomap_dio_bio_end_io.blk_update_request.blk_mq_end_request.blk_complete_reqs
      0.72 ± 18%      +2.7        3.38 ± 23%  perf-profile.calltrace.cycles-pp.ext4_ext_remove_space.ext4_ext_truncate.ext4_truncate.ext4_setattr.notify_change
      0.00            +2.7        2.67 ± 30%  perf-profile.calltrace.cycles-pp.iomap_dio_bio_end_io.blk_update_request.blk_mq_end_request.blk_complete_reqs.__softirqentry_text_start
      1.23 ± 24%      +2.7        3.96 ± 32%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime
      0.00            +2.8        2.75 ± 13%  perf-profile.calltrace.cycles-pp.__queue_work.queue_work_on.loop_queue_rq.__blk_mq_try_issue_directly.blk_mq_request_issue_directly
      0.00            +2.8        2.77 ± 13%  perf-profile.calltrace.cycles-pp.queue_work_on.loop_queue_rq.__blk_mq_try_issue_directly.blk_mq_request_issue_directly.blk_mq_try_issue_list_directly
      1.24 ± 24%      +2.8        4.04 ± 32%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
      0.79 ± 15%      +2.9        3.74 ± 23%  perf-profile.calltrace.cycles-pp.__invalidate_mapping_pages.drop_pagecache_sb.iterate_supers.drop_caches_sysctl_handler.proc_sys_call_handler
      0.30 ±100%      +3.0        3.31 ± 15%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.84 ± 18%      +3.0        3.85 ± 21%  perf-profile.calltrace.cycles-pp.ext4_ext_truncate.ext4_truncate.ext4_setattr.notify_change.do_truncate
      1.06 ± 22%      +3.0        4.07 ± 12%  perf-profile.calltrace.cycles-pp.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.do_iter_readv_writev.do_iter_write
      0.31 ±100%      +3.0        3.36 ± 15%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.48 ± 24%      +3.1        4.53 ± 15%  perf-profile.calltrace.cycles-pp.ext4_map_blocks.ext4_iomap_begin.iomap_apply.__iomap_dio_rw.iomap_dio_rw
      0.00            +3.1        3.06 ± 26%  perf-profile.calltrace.cycles-pp.blk_update_request.blk_mq_end_request.blk_complete_reqs.__softirqentry_text_start.run_ksoftirqd
      0.32 ±100%      +3.1        3.42 ± 16%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.32 ±100%      +3.1        3.43 ± 16%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      1.54 ± 23%      +3.2        4.75 ± 15%  perf-profile.calltrace.cycles-pp.ext4_iomap_begin.iomap_apply.__iomap_dio_rw.iomap_dio_rw.ext4_dio_write_iter
      0.00            +3.2        3.23 ± 29%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
      0.00            +3.2        3.24 ± 25%  perf-profile.calltrace.cycles-pp.blk_mq_end_request.blk_complete_reqs.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
      0.34 ±100%      +3.3        3.59 ± 16%  perf-profile.calltrace.cycles-pp.read
      0.00            +3.4        3.35 ± 29%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template
      0.37 ± 70%      +3.4        3.74 ± 56%  perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
      0.00            +3.4        3.39 ± 29%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
      0.00            +3.5        3.46 ± 24%  perf-profile.calltrace.cycles-pp.blk_complete_reqs.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
      0.00            +3.5        3.49 ± 28%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up
      0.00            +3.5        3.52 ± 28%  perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
      1.00 ± 19%      +3.7        4.66 ± 22%  perf-profile.calltrace.cycles-pp.ext4_truncate.ext4_setattr.notify_change.do_truncate.do_sys_ftruncate
      1.04 ± 14%      +3.7        4.79 ± 24%  perf-profile.calltrace.cycles-pp.drop_pagecache_sb.iterate_supers.drop_caches_sysctl_handler.proc_sys_call_handler.new_sync_write
      1.04 ± 14%      +3.8        4.80 ± 24%  perf-profile.calltrace.cycles-pp.iterate_supers.drop_caches_sysctl_handler.proc_sys_call_handler.new_sync_write.vfs_write
      1.07 ± 12%      +3.8        4.92 ± 24%  perf-profile.calltrace.cycles-pp.drop_caches_sysctl_handler.proc_sys_call_handler.new_sync_write.vfs_write.ksys_write
      1.07 ± 12%      +3.8        4.92 ± 24%  perf-profile.calltrace.cycles-pp.proc_sys_call_handler.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.00            +3.9        3.88 ± 26%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      0.89 ± 10%      +3.9        4.77 ± 21%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.io_schedule_timeout.__iomap_dio_rw
      0.89 ± 10%      +3.9        4.80 ± 21%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.iomap_dio_rw
      0.00            +3.9        3.92 ± 21%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +3.9        3.95 ± 21%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +4.1        4.06 ± 11%  perf-profile.calltrace.cycles-pp.loop_queue_rq.__blk_mq_try_issue_directly.blk_mq_request_issue_directly.blk_mq_try_issue_list_directly.blk_mq_sched_insert_requests
      0.00            +4.1        4.13 ± 11%  perf-profile.calltrace.cycles-pp.__blk_mq_try_issue_directly.blk_mq_request_issue_directly.blk_mq_try_issue_list_directly.blk_mq_sched_insert_requests.blk_mq_flush_plug_list
      0.00            +4.2        4.16 ± 11%  perf-profile.calltrace.cycles-pp.blk_mq_request_issue_directly.blk_mq_try_issue_list_directly.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list
      1.37 ± 22%      +4.2        5.56 ±  9%  perf-profile.calltrace.cycles-pp.generic_perform_write.ext4_buffered_write_iter.do_iter_readv_writev.do_iter_write.lo_write_bvec
      0.94 ± 10%      +4.2        5.13 ± 20%  perf-profile.calltrace.cycles-pp.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.iomap_dio_rw.ext4_dio_write_iter
      0.00            +4.2        4.20 ± 11%  perf-profile.calltrace.cycles-pp.blk_mq_try_issue_list_directly.blk_mq_sched_insert_requests.blk_mq_flush_plug_list.blk_flush_plug_list.blk_finish_plug
      0.95 ± 10%      +4.2        5.16 ± 20%  perf-profile.calltrace.cycles-pp.io_schedule_timeout.__iomap_dio_rw.iomap_dio_rw.ext4_dio_write_iter.new_sync_write
      0.77 ± 24%      +4.5        5.27 ± 44%  perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
      0.77 ± 24%      +4.5        5.30 ± 44%  perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow
      1.63 ± 20%      +4.5        6.16 ± 36%  perf-profile.calltrace.cycles-pp.cmd_sched.run_builtin.main.__libc_start_main
      1.62 ± 20%      +4.5        6.16 ± 36%  perf-profile.calltrace.cycles-pp.cmd_record.cmd_sched.run_builtin.main.__libc_start_main
      1.57 ± 23%      +4.6        6.21 ±  7%  perf-profile.calltrace.cycles-pp.ext4_buffered_write_iter.do_iter_readv_writev.do_iter_write.lo_write_bvec.loop_process_work
      1.60 ± 23%      +4.7        6.27 ±  7%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.lo_write_bvec.loop_process_work.process_one_work
      1.26 ± 18%      +4.7        6.00 ± 20%  perf-profile.calltrace.cycles-pp.ext4_setattr.notify_change.do_truncate.do_sys_ftruncate.do_syscall_64
      1.65 ± 23%      +4.8        6.44 ±  7%  perf-profile.calltrace.cycles-pp.do_iter_write.lo_write_bvec.loop_process_work.process_one_work.worker_thread
      1.30 ± 18%      +4.9        6.19 ± 21%  perf-profile.calltrace.cycles-pp.notify_change.do_truncate.do_sys_ftruncate.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.68 ± 23%      +4.9        6.58 ±  7%  perf-profile.calltrace.cycles-pp.lo_write_bvec.loop_process_work.process_one_work.worker_thread.kthread
      0.74 ± 15%      +5.0        5.79 ± 16%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
      2.39 ± 23%      +5.4        7.80 ± 10%  perf-profile.calltrace.cycles-pp.iomap_apply.__iomap_dio_rw.iomap_dio_rw.ext4_dio_write_iter.new_sync_write
      1.09 ± 23%      +5.5        6.60 ± 38%  perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event
      1.80 ± 20%      +5.5        7.33 ± 30%  perf-profile.calltrace.cycles-pp.__libc_start_main
      1.80 ± 20%      +5.5        7.33 ± 30%  perf-profile.calltrace.cycles-pp.main.__libc_start_main
      1.80 ± 20%      +5.5        7.33 ± 30%  perf-profile.calltrace.cycles-pp.run_builtin.main.__libc_start_main
      2.09 ± 24%      +7.5        9.63 ±  4%  perf-profile.calltrace.cycles-pp.loop_process_work.process_one_work.worker_thread.kthread.ret_from_fork
      3.99 ± 22%      +9.7       13.65 ±  3%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      4.46 ± 21%     +11.7       16.16 ±  3%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      5.33 ± 20%     +17.6       22.95 ±  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      5.33 ± 20%     +17.6       22.96 ±  5%  perf-profile.calltrace.cycles-pp.ret_from_fork
     72.64 ±  5%     -68.1        4.57 ±  8%  perf-profile.children.cycles-pp.blk_mq_sched_insert_requests
     72.66 ±  5%     -68.0        4.68 ±  7%  perf-profile.children.cycles-pp.blk_mq_flush_plug_list
     72.67 ±  5%     -68.0        4.70 ±  8%  perf-profile.children.cycles-pp.blk_flush_plug_list
     72.43 ±  5%     -68.0        4.47 ±  9%  perf-profile.children.cycles-pp.blk_finish_plug
     64.68 ±  6%     -63.8        0.86 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock
     64.47 ±  6%     -63.7        0.75 ± 37%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     76.11 ±  5%     -58.2       17.93 ±  6%  perf-profile.children.cycles-pp.__iomap_dio_rw
     76.11 ±  5%     -58.2       17.94 ±  6%  perf-profile.children.cycles-pp.iomap_dio_rw
     76.68 ±  5%     -56.9       19.83 ±  5%  perf-profile.children.cycles-pp.ext4_dio_write_iter
     52.89 ±  5%     -52.6        0.29 ± 31%  perf-profile.children.cycles-pp.__blk_mq_delay_run_hw_queue
     53.18 ±  4%     -52.3        0.88 ± 34%  perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched
     53.20 ±  4%     -52.3        0.91 ± 31%  perf-profile.children.cycles-pp.__blk_mq_run_hw_queue
     53.20 ±  4%     -52.3        0.91 ± 31%  perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
     53.19 ±  4%     -52.3        0.91 ± 31%  perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
     52.42 ±  5%     -51.7        0.70 ± 33%  perf-profile.children.cycles-pp.dd_dispatch_request
     78.58 ±  5%     -51.1       27.52 ±  2%  perf-profile.children.cycles-pp.new_sync_write
     78.70 ±  5%     -50.8       27.90 ±  2%  perf-profile.children.cycles-pp.vfs_write
     78.69 ±  5%     -50.6       28.04 ±  2%  perf-profile.children.cycles-pp.ksys_write
     84.60 ±  4%     -27.6       57.02 ±  5%  perf-profile.children.cycles-pp.do_syscall_64
     84.65 ±  4%     -27.4       57.20 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     19.69 ±  6%     -19.7        0.00        perf-profile.children.cycles-pp.dd_insert_requests
      7.55 ± 20%      -6.9        0.67 ± 33%  perf-profile.children.cycles-pp.dd_queued
      3.05 ±  7%      -1.8        1.28 ± 56%  perf-profile.children.cycles-pp.start_secondary
      3.42 ±  9%      -1.2        2.21 ± 32%  perf-profile.children.cycles-pp.do_idle
      3.42 ±  9%      -1.2        2.22 ± 33%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
      3.42 ±  9%      -1.2        2.22 ± 33%  perf-profile.children.cycles-pp.cpu_startup_entry
      1.03 ± 19%      -0.9        0.14 ± 21%  perf-profile.children.cycles-pp.cpumask_next
      3.05 ±  7%      -0.8        2.25 ± 25%  perf-profile.children.cycles-pp.scheduler_tick
      1.78 ±  9%      -0.8        0.99 ± 38%  perf-profile.children.cycles-pp.cpuidle_enter
      1.77 ±  9%      -0.8        0.99 ± 38%  perf-profile.children.cycles-pp.cpuidle_enter_state
      2.74 ±  6%      -0.7        2.01 ± 25%  perf-profile.children.cycles-pp.task_tick_fair
      0.73 ± 18%      -0.5        0.20 ± 15%  perf-profile.children.cycles-pp._find_next_bit
      0.59 ± 13%      -0.4        0.20 ± 52%  perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list
      0.50 ±  6%      -0.4        0.12 ± 34%  perf-profile.children.cycles-pp.load_balance
      0.40 ±  9%      -0.3        0.07 ± 57%  perf-profile.children.cycles-pp.newidle_balance
      0.25 ±  9%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.find_busiest_group
      0.24 ± 11%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.45 ± 13%      -0.2        0.30 ± 27%  perf-profile.children.cycles-pp.menu_select
      0.22 ± 41%      -0.1        0.07 ± 79%  perf-profile.children.cycles-pp.mntput_no_expire
      0.19 ± 52%      -0.1        0.05 ± 49%  perf-profile.children.cycles-pp.__block_write_full_page
      0.19 ± 52%      -0.1        0.05 ± 59%  perf-profile.children.cycles-pp.__writepage
      0.23 ± 13%      -0.1        0.10 ± 49%  perf-profile.children.cycles-pp.intel_idle
      0.20 ± 10%      -0.1        0.08 ± 80%  perf-profile.children.cycles-pp.rebalance_domains
      0.13 ± 13%      -0.1        0.06 ± 75%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.01 ±223%      +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.bio_endio
      0.01 ±223%      +0.1        0.07 ± 30%  perf-profile.children.cycles-pp.blk_mq_rq_ctx_init
      0.15 ±  4%      +0.1        0.21 ± 23%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.02 ±142%      +0.1        0.08 ± 29%  perf-profile.children.cycles-pp.sbitmap_get
      0.05 ± 49%      +0.1        0.12 ± 21%  perf-profile.children.cycles-pp.update_cfs_group
      0.02 ±142%      +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.ext4_mb_use_preallocated
      0.00            +0.1        0.07 ± 21%  perf-profile.children.cycles-pp.free_swap_cache
      0.00            +0.1        0.07 ± 43%  perf-profile.children.cycles-pp.PyParser_AddToken
      0.00            +0.1        0.07 ± 37%  perf-profile.children.cycles-pp.blkdev_read_iter
      0.00            +0.1        0.08 ± 35%  perf-profile.children.cycles-pp.page_add_new_anon_rmap
      0.00            +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.__x64_sys_readlink
      0.03 ±100%      +0.1        0.10 ± 25%  perf-profile.children.cycles-pp.get_stack_info_noinstr
      0.00            +0.1        0.08 ± 36%  perf-profile.children.cycles-pp.unlink_file_vma
      0.00            +0.1        0.08 ± 35%  perf-profile.children.cycles-pp.show_stat
      0.01 ±223%      +0.1        0.09 ± 53%  perf-profile.children.cycles-pp.atime_needs_update
      0.02 ±141%      +0.1        0.10 ± 38%  perf-profile.children.cycles-pp.proc_pident_instantiate
      0.04 ± 71%      +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.touch_atime
      0.00            +0.1        0.08 ± 21%  perf-profile.children.cycles-pp.page_cache_ra_unbounded
      0.00            +0.1        0.08 ± 31%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.__netlink_dump_start
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.netlink_rcv_skb
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.rtnetlink_rcv_msg
      0.00            +0.1        0.08 ± 28%  perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.sendto
      0.01 ±223%      +0.1        0.09 ± 41%  perf-profile.children.cycles-pp.sync_regs
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.09 ± 30%  perf-profile.children.cycles-pp.__xa_set_mark
      0.00            +0.1        0.09 ± 44%  perf-profile.children.cycles-pp.____sys_sendmsg
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.snmp_get_cpu_field
      0.06 ± 52%      +0.1        0.15 ± 40%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.netlink_unicast
      0.01 ±223%      +0.1        0.10 ± 34%  perf-profile.children.cycles-pp.__dquot_alloc_space
      0.01 ±223%      +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.ext4_da_reserve_space
      0.00            +0.1        0.09 ± 52%  perf-profile.children.cycles-pp.___sys_sendmsg
      0.00            +0.1        0.09 ± 52%  perf-profile.children.cycles-pp.__sys_sendmsg
      0.00            +0.1        0.10 ± 31%  perf-profile.children.cycles-pp.alloc_bprm
      0.00            +0.1        0.10 ± 38%  perf-profile.children.cycles-pp.internal_add_timer
      0.01 ±223%      +0.1        0.11 ± 22%  perf-profile.children.cycles-pp.try_charge_memcg
      0.00            +0.1        0.10 ± 39%  perf-profile.children.cycles-pp.___slab_alloc
      0.00            +0.1        0.10 ± 37%  perf-profile.children.cycles-pp.bitmap_list_string
      0.01 ±223%      +0.1        0.11 ± 35%  perf-profile.children.cycles-pp.timestamp_truncate
      0.00            +0.1        0.10 ± 47%  perf-profile.children.cycles-pp.find_extend_vma
      0.02 ±141%      +0.1        0.12 ± 40%  perf-profile.children.cycles-pp.blk_mq_start_request
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.__get_user_pages_remote
      0.03 ±100%      +0.1        0.13 ± 40%  perf-profile.children.cycles-pp.get_callchain_entry
      0.00            +0.1        0.10 ± 32%  perf-profile.children.cycles-pp.refill_obj_stock
      0.02 ±142%      +0.1        0.12 ± 37%  perf-profile.children.cycles-pp.cpuacct_charge
      0.01 ±223%      +0.1        0.11 ± 31%  perf-profile.children.cycles-pp.netlink_sendmsg
      0.00            +0.1        0.10 ± 39%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.10 ± 43%  perf-profile.children.cycles-pp.anon_vma_fork
      0.00            +0.1        0.10 ± 32%  perf-profile.children.cycles-pp.get_arg_page
      0.00            +0.1        0.11 ± 32%  perf-profile.children.cycles-pp.vmacache_find
      0.00            +0.1        0.11 ± 24%  perf-profile.children.cycles-pp.lock_page_memcg
      0.01 ±223%      +0.1        0.12 ± 37%  perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
      0.00            +0.1        0.11 ± 32%  perf-profile.children.cycles-pp.d_path
      0.02 ±141%      +0.1        0.12 ± 40%  perf-profile.children.cycles-pp.do_open_execat
      0.00            +0.1        0.11 ± 49%  perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
      0.02 ±141%      +0.1        0.13 ± 45%  perf-profile.children.cycles-pp.path_init
      0.10 ± 34%      +0.1        0.21 ± 28%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.11 ± 32%  perf-profile.children.cycles-pp.__slab_alloc
      0.00            +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.__snmp6_fill_stats64
      0.01 ±223%      +0.1        0.12 ± 35%  perf-profile.children.cycles-pp.perf_output_end
      0.00            +0.1        0.11 ± 35%  perf-profile.children.cycles-pp.filp_close
      0.01 ±223%      +0.1        0.12 ± 46%  perf-profile.children.cycles-pp.proc_alloc_inode
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.page_remove_rmap
      0.00            +0.1        0.12 ± 43%  perf-profile.children.cycles-pp.terminate_walk
      0.00            +0.1        0.12 ± 19%  perf-profile.children.cycles-pp.inet6_fill_link_af
      0.00            +0.1        0.12 ± 19%  perf-profile.children.cycles-pp.inet6_fill_ifla6_attrs
      0.03 ±100%      +0.1        0.14 ± 39%  perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
      0.00            +0.1        0.12 ± 32%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.01 ±223%      +0.1        0.12 ± 44%  perf-profile.children.cycles-pp.security_path_truncate
      0.05 ± 75%      +0.1        0.16 ± 29%  perf-profile.children.cycles-pp.iomap_dio_complete
      0.03 ±100%      +0.1        0.14 ± 28%  perf-profile.children.cycles-pp.ext4_mb_release_context
      0.00            +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.____sys_recvmsg
      0.00            +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.__irqentry_text_end
      0.01 ±223%      +0.1        0.13 ± 35%  perf-profile.children.cycles-pp.perf_tp_event_match
      0.04 ± 73%      +0.1        0.16 ± 24%  perf-profile.children.cycles-pp.blk_mq_get_tag
      0.05 ± 81%      +0.1        0.16 ± 45%  perf-profile.children.cycles-pp.__perf_sw_event
      0.00            +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.count_free
      0.02 ± 99%      +0.1        0.14 ± 28%  perf-profile.children.cycles-pp.rcu_all_qs
      0.02 ±142%      +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.__might_fault
      0.01 ±223%      +0.1        0.13 ± 48%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.02 ±142%      +0.1        0.14 ± 36%  perf-profile.children.cycles-pp.ftrace_graph_ret_addr
      0.00            +0.1        0.12 ± 47%  perf-profile.children.cycles-pp.pte_alloc_one
      0.00            +0.1        0.12 ± 43%  perf-profile.children.cycles-pp.perf_event_mmap_output
      0.02 ±141%      +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.blkdev_write_iter
      0.02 ±143%      +0.1        0.14 ± 33%  perf-profile.children.cycles-pp.__libc_pwrite
      0.03 ±102%      +0.1        0.15 ± 27%  perf-profile.children.cycles-pp.gup_pgd_range
      0.02 ±142%      +0.1        0.14 ± 33%  perf-profile.children.cycles-pp.ksys_pwrite64
      0.00            +0.1        0.12 ± 42%  perf-profile.children.cycles-pp.__delete_from_page_cache
      0.06 ± 47%      +0.1        0.19 ± 42%  perf-profile.children.cycles-pp.kallsyms_expand_symbol
      0.01 ±223%      +0.1        0.13 ± 39%  perf-profile.children.cycles-pp.ext4_es_can_be_merged
      0.01 ±223%      +0.1        0.13 ± 30%  perf-profile.children.cycles-pp.recvmsg
      0.02 ±142%      +0.1        0.14 ± 38%  perf-profile.children.cycles-pp.__get_free_pages
      0.01 ±223%      +0.1        0.13 ± 42%  perf-profile.children.cycles-pp.aa_file_perm
      0.01 ±223%      +0.1        0.14 ± 43%  perf-profile.children.cycles-pp.user_path_at_empty
      0.01 ±223%      +0.1        0.14 ± 15%  perf-profile.children.cycles-pp.__clear_user
      0.00            +0.1        0.13 ± 24%  perf-profile.children.cycles-pp.up_write
      0.04 ± 73%      +0.1        0.17 ± 37%  perf-profile.children.cycles-pp.alloc_buffer_head
      0.00            +0.1        0.13 ± 47%  perf-profile.children.cycles-pp.vm_area_alloc
      0.01 ±223%      +0.1        0.14 ± 32%  perf-profile.children.cycles-pp.__sys_recvmsg
      0.01 ±223%      +0.1        0.14 ± 55%  perf-profile.children.cycles-pp.common_perm_cond
      0.00            +0.1        0.13 ± 44%  perf-profile.children.cycles-pp.diskstats_show
      0.00            +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.ext4_inode_block_valid
      0.03 ±100%      +0.1        0.16 ± 35%  perf-profile.children.cycles-pp.__calc_delta
      0.00            +0.1        0.13 ± 32%  perf-profile.children.cycles-pp.___sys_recvmsg
      0.01 ±223%      +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.string_nocheck
      0.07 ± 52%      +0.1        0.21 ± 22%  perf-profile.children.cycles-pp.__blk_mq_alloc_request
      0.00            +0.1        0.14 ± 38%  perf-profile.children.cycles-pp.copyout
      0.05 ± 45%      +0.1        0.18 ± 46%  perf-profile.children.cycles-pp.ext4_mb_load_buddy_gfp
      0.06 ± 11%      +0.1        0.20 ± 24%  perf-profile.children.cycles-pp.lru_add_drain
      0.02 ±141%      +0.1        0.15 ± 33%  perf-profile.children.cycles-pp.crypto_shash_update
      0.00            +0.1        0.14 ± 27%  perf-profile.children.cycles-pp.__perf_event_account_interrupt
      0.00            +0.1        0.14 ± 42%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.01 ±223%      +0.1        0.15 ± 27%  perf-profile.children.cycles-pp.alloc_pages
      0.00            +0.1        0.14 ± 31%  perf-profile.children.cycles-pp.remove_wait_queue
      0.01 ±223%      +0.1        0.15 ± 34%  perf-profile.children.cycles-pp.obj_cgroup_charge_pages
      0.01 ±223%      +0.1        0.15 ± 15%  perf-profile.children.cycles-pp.reweight_entity
      0.00            +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.count_partial
      0.00            +0.1        0.14 ± 29%  perf-profile.children.cycles-pp._copy_to_user
      0.01 ±223%      +0.1        0.15 ± 29%  perf-profile.children.cycles-pp.map_id_range_down
      0.03 ±101%      +0.1        0.17 ± 35%  perf-profile.children.cycles-pp.shmem_alloc_page
      0.01 ±223%      +0.1        0.15 ± 46%  perf-profile.children.cycles-pp.del_timer_sync
      0.00            +0.1        0.14 ± 42%  perf-profile.children.cycles-pp.vm_area_dup
      0.01 ±223%      +0.1        0.15 ± 43%  perf-profile.children.cycles-pp.proc_pid_lookup
      0.02 ±144%      +0.1        0.16 ± 57%  perf-profile.children.cycles-pp.ext4_es_remove_extent
      0.00            +0.1        0.15 ± 46%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.00            +0.1        0.15 ± 57%  perf-profile.children.cycles-pp.io_serial_out
      0.00            +0.1        0.15 ± 34%  perf-profile.children.cycles-pp.rcu_gp_cleanup
      0.00            +0.1        0.15 ±115%  perf-profile.children.cycles-pp.__kernel_read
      0.01 ±223%      +0.1        0.16 ± 14%  perf-profile.children.cycles-pp.rtnl_dump_ifinfo
      0.01 ±223%      +0.1        0.16 ± 14%  perf-profile.children.cycles-pp.rtnl_fill_ifinfo
      0.15 ± 21%      +0.1        0.30 ± 23%  perf-profile.children.cycles-pp.native_sched_clock
      0.06 ± 45%      +0.1        0.21 ± 16%  perf-profile.children.cycles-pp.get_stack_info
      0.00            +0.1        0.15 ± 27%  perf-profile.children.cycles-pp.vfs_fstat
      0.00            +0.2        0.15 ± 36%  perf-profile.children.cycles-pp.__rblist__findnew
      0.00            +0.2        0.15 ± 12%  perf-profile.children.cycles-pp.get_slabinfo
      0.01 ±223%      +0.2        0.16 ± 38%  perf-profile.children.cycles-pp.proc_root_lookup
      0.05 ± 50%      +0.2        0.20 ± 45%  perf-profile.children.cycles-pp.free_unref_page_list
      0.01 ±223%      +0.2        0.16 ± 38%  perf-profile.children.cycles-pp.path_put
      0.03 ±102%      +0.2        0.18 ± 41%  perf-profile.children.cycles-pp.perf_read
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.__x64_sys_sendto
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.__sys_sendto
      0.00            +0.2        0.15 ± 19%  perf-profile.children.cycles-pp.fput_many
      0.00            +0.2        0.15 ± 44%  perf-profile.children.cycles-pp.task_state
      0.01 ±223%      +0.2        0.16 ± 16%  perf-profile.children.cycles-pp.netlink_dump
      0.03 ±100%      +0.2        0.18 ± 39%  perf-profile.children.cycles-pp.proc_pident_lookup
      0.05 ± 48%      +0.2        0.20 ± 23%  perf-profile.children.cycles-pp.__read_extent_tree_block
      0.01 ±223%      +0.2        0.16 ± 45%  perf-profile.children.cycles-pp.__vsnprintf_chk
      0.02 ±141%      +0.2        0.17 ± 41%  perf-profile.children.cycles-pp.map_id_up
      0.02 ±142%      +0.2        0.18 ± 33%  perf-profile.children.cycles-pp.new_inode
      0.01 ±223%      +0.2        0.16 ± 27%  perf-profile.children.cycles-pp.vma_link
      0.01 ±223%      +0.2        0.16 ± 31%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.05 ± 50%      +0.2        0.21 ± 37%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.01 ±223%      +0.2        0.17 ± 30%  perf-profile.children.cycles-pp.remove_mapping
      0.01 ±223%      +0.2        0.17 ± 30%  perf-profile.children.cycles-pp.__remove_mapping
      0.05 ± 81%      +0.2        0.20 ± 32%  perf-profile.children.cycles-pp.__do_sys_wait4
      0.05 ± 81%      +0.2        0.20 ± 32%  perf-profile.children.cycles-pp.kernel_wait4
      0.00            +0.2        0.16 ± 34%  perf-profile.children.cycles-pp.proc_reg_read_iter
      0.01 ±223%      +0.2        0.17 ± 36%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.03 ±101%      +0.2        0.20 ± 48%  perf-profile.children.cycles-pp.ext4_fc_track_inode
      0.03 ±101%      +0.2        0.19 ± 34%  perf-profile.children.cycles-pp.__x64_sys_munmap
      0.07 ± 30%      +0.2        0.23 ± 31%  perf-profile.children.cycles-pp.perf_trace_buf_alloc
      0.01 ±223%      +0.2        0.17 ± 43%  perf-profile.children.cycles-pp.vfprintf
      0.01 ±223%      +0.2        0.17 ± 22%  perf-profile.children.cycles-pp.slab_show
      0.03 ±100%      +0.2        0.19 ± 20%  perf-profile.children.cycles-pp.getifaddrs
      0.01 ±223%      +0.2        0.17 ± 34%  perf-profile.children.cycles-pp.remove_vma
      0.00            +0.2        0.16 ± 31%  perf-profile.children.cycles-pp.__switch_to
      0.04 ± 75%      +0.2        0.20 ± 73%  perf-profile.children.cycles-pp.start_this_handle
      0.04 ± 72%      +0.2        0.20 ± 37%  perf-profile.children.cycles-pp.string
      0.06 ± 50%      +0.2        0.23 ± 34%  perf-profile.children.cycles-pp.alloc_page_buffers
      0.04 ±104%      +0.2        0.20 ± 32%  perf-profile.children.cycles-pp.do_wait
      0.05 ± 45%      +0.2        0.22 ± 15%  perf-profile.children.cycles-pp.xas_store
      0.05 ± 80%      +0.2        0.22 ± 40%  perf-profile.children.cycles-pp.do_set_pte
      0.01 ±223%      +0.2        0.18 ± 21%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.08 ± 24%      +0.2        0.24 ± 22%  perf-profile.children.cycles-pp.shmem_add_to_page_cache
      0.01 ±223%      +0.2        0.18 ± 31%  perf-profile.children.cycles-pp.down_read
      0.01 ±223%      +0.2        0.18 ± 41%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.04 ± 76%      +0.2        0.21 ± 42%  perf-profile.children.cycles-pp.strncpy_from_user
      0.23 ± 23%      +0.2        0.40 ± 22%  perf-profile.children.cycles-pp.update_rq_clock
      0.17 ± 20%      +0.2        0.34 ± 18%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.01 ±223%      +0.2        0.18 ± 15%  perf-profile.children.cycles-pp.make_kuid
      0.04 ± 72%      +0.2        0.21 ± 56%  perf-profile.children.cycles-pp.end_page_writeback
      0.01 ±223%      +0.2        0.18 ± 22%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.01 ±223%      +0.2        0.18 ± 13%  perf-profile.children.cycles-pp.mod_objcg_state
      0.07 ± 25%      +0.2        0.24 ± 39%  perf-profile.children.cycles-pp.ext4_es_insert_delayed_block
      0.06 ± 76%      +0.2        0.23 ± 38%  perf-profile.children.cycles-pp.current_time
      0.01 ±223%      +0.2        0.18 ± 39%  perf-profile.children.cycles-pp.clear_page_erms
      0.01 ±223%      +0.2        0.19 ± 42%  perf-profile.children.cycles-pp.unit_file_find_dropin_paths
      0.10 ± 33%      +0.2        0.28 ± 41%  perf-profile.children.cycles-pp.update_iter
      0.01 ±223%      +0.2        0.19 ± 41%  perf-profile.children.cycles-pp.kernel_init_free_pages
      0.14 ± 38%      +0.2        0.32 ± 45%  perf-profile.children.cycles-pp.wb_workfn
      0.14 ± 38%      +0.2        0.32 ± 45%  perf-profile.children.cycles-pp.wb_writeback
      0.05 ± 72%      +0.2        0.23 ± 37%  perf-profile.children.cycles-pp.ext4_group_desc_csum
      0.02 ±146%      +0.2        0.20 ± 51%  perf-profile.children.cycles-pp.copy_page_range
      0.12 ±  9%      +0.2        0.30 ± 34%  perf-profile.children.cycles-pp.bsearch
      0.04 ± 71%      +0.2        0.22 ± 28%  perf-profile.children.cycles-pp.inode_permission
      0.02 ±142%      +0.2        0.20 ± 63%  perf-profile.children.cycles-pp.__module_text_address
      0.04 ±102%      +0.2        0.22 ± 31%  perf-profile.children.cycles-pp.__check_object_size
      0.04 ± 75%      +0.2        0.23 ± 28%  perf-profile.children.cycles-pp.shmem_alloc_and_acct_page
      0.00            +0.2        0.19 ± 62%  perf-profile.children.cycles-pp.__module_address
      0.05 ± 80%      +0.2        0.24 ± 32%  perf-profile.children.cycles-pp.wait4
      0.04 ± 71%      +0.2        0.22 ± 19%  perf-profile.children.cycles-pp.perf_trace_buf_update
      0.02 ±141%      +0.2        0.21 ± 69%  perf-profile.children.cycles-pp.mempool_alloc
      0.05 ± 74%      +0.2        0.24 ± 32%  perf-profile.children.cycles-pp.kernel_fpu_begin_mask
      0.04 ± 80%      +0.2        0.23 ± 40%  perf-profile.children.cycles-pp.d_alloc
      0.01 ±223%      +0.2        0.20 ± 38%  perf-profile.children.cycles-pp.copy_strings
      0.06 ± 64%      +0.2        0.26 ± 37%  perf-profile.children.cycles-pp.ext4_io_submit
      0.05 ± 45%      +0.2        0.24 ± 55%  perf-profile.children.cycles-pp.is_module_text_address
      0.10 ± 36%      +0.2        0.30 ± 41%  perf-profile.children.cycles-pp.s_next
      0.07 ± 47%      +0.2        0.26 ± 28%  perf-profile.children.cycles-pp.common_file_perm
      0.01 ±223%      +0.2        0.21 ± 42%  perf-profile.children.cycles-pp.timekeeping_update
      0.01 ±223%      +0.2        0.21 ± 38%  perf-profile.children.cycles-pp.select_idle_cpu
      0.06 ± 53%      +0.2        0.25 ± 36%  perf-profile.children.cycles-pp.__es_insert_extent
      0.02 ±141%      +0.2        0.22 ± 35%  perf-profile.children.cycles-pp.proc_pid_make_inode
      0.07 ± 54%      +0.2        0.27 ± 31%  perf-profile.children.cycles-pp.create_empty_buffers
      0.05 ± 47%      +0.2        0.25 ± 27%  perf-profile.children.cycles-pp.do_dentry_open
      0.03 ±100%      +0.2        0.23 ± 43%  perf-profile.children.cycles-pp.deactivate_file_page
      0.02 ±141%      +0.2        0.22 ± 25%  perf-profile.children.cycles-pp.malloc
      0.06 ± 46%      +0.2        0.26 ± 14%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.02 ±142%      +0.2        0.22 ± 93%  perf-profile.children.cycles-pp.filemap_get_pages
      0.03 ±100%      +0.2        0.24 ± 25%  perf-profile.children.cycles-pp.perf_misc_flags
      0.04 ± 75%      +0.2        0.25 ± 47%  perf-profile.children.cycles-pp.unlock_page
      0.02 ±142%      +0.2        0.22 ± 30%  perf-profile.children.cycles-pp.stop_this_handle
      0.06 ± 19%      +0.2        0.27 ± 55%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.03 ±102%      +0.2        0.24 ± 31%  perf-profile.children.cycles-pp.sock_sendmsg
      0.04 ± 83%      +0.2        0.25 ± 22%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.01 ±223%      +0.2        0.22 ± 29%  perf-profile.children.cycles-pp.poll_freewait
      0.03 ±100%      +0.2        0.24 ± 27%  perf-profile.children.cycles-pp.proc_ns_get_link
      0.09 ± 28%      +0.2        0.30 ± 29%  perf-profile.children.cycles-pp.next_uptodate_page
      0.03 ±141%      +0.2        0.24 ± 33%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.02 ±141%      +0.2        0.23 ± 13%  perf-profile.children.cycles-pp.__restore_fpregs_from_fpstate
      0.03 ±141%      +0.2        0.24 ± 33%  perf-profile.children.cycles-pp.alloc_inode
      0.06 ± 72%      +0.2        0.27 ± 41%  perf-profile.children.cycles-pp.ext4_group_desc_csum_set
      0.07 ± 20%      +0.2        0.29 ± 40%  perf-profile.children.cycles-pp.perf_output_begin_forward
      0.12 ±  7%      +0.2        0.34 ± 29%  perf-profile.children.cycles-pp.search_extable
      0.06 ± 51%      +0.2        0.28 ± 24%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.06 ± 46%      +0.2        0.28 ± 26%  perf-profile.children.cycles-pp.__vm_munmap
      0.01 ±223%      +0.2        0.23 ± 36%  perf-profile.children.cycles-pp.kvm_is_in_guest
      0.14 ± 10%      +0.2        0.36 ± 31%  perf-profile.children.cycles-pp.search_exception_tables
      0.10 ± 39%      +0.2        0.32 ± 28%  perf-profile.children.cycles-pp.___perf_sw_event
      0.01 ±223%      +0.2        0.24 ± 57%  perf-profile.children.cycles-pp.mb_free_blocks
      0.02 ±142%      +0.2        0.24 ± 33%  perf-profile.children.cycles-pp.prep_new_page
      0.06 ± 33%      +0.2        0.29 ± 39%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.04 ± 79%      +0.2        0.27 ± 19%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.04 ± 71%      +0.2        0.26 ± 39%  perf-profile.children.cycles-pp.mprotect_fixup
      0.12 ± 20%      +0.2        0.35 ± 22%  perf-profile.children.cycles-pp.error_entry
      0.04 ± 73%      +0.2        0.27 ± 86%  perf-profile.children.cycles-pp.kmem_cache_alloc_trace
      0.06 ± 47%      +0.2        0.29 ± 29%  perf-profile.children.cycles-pp.memset_erms
      0.02 ±142%      +0.2        0.25 ± 17%  perf-profile.children.cycles-pp.__fget_files
      0.08 ± 20%      +0.2        0.31 ± 18%  perf-profile.children.cycles-pp.unmap_region
      0.08 ± 25%      +0.2        0.31 ± 33%  perf-profile.children.cycles-pp.__entry_text_start
      0.06 ± 29%      +0.2        0.30 ± 46%  perf-profile.children.cycles-pp.ext4_es_insert_extent
      0.10 ± 29%      +0.2        0.33 ± 30%  perf-profile.children.cycles-pp.mem_cgroup_charge
      0.05 ± 72%      +0.2        0.29 ± 22%  perf-profile.children.cycles-pp.free_pgtables
      0.05 ± 57%      +0.2        0.29 ± 23%  perf-profile.children.cycles-pp.select_idle_sibling
      0.09 ± 27%      +0.2        0.33 ± 24%  perf-profile.children.cycles-pp.__pagevec_lru_add
      0.07 ± 28%      +0.2        0.30 ± 37%  perf-profile.children.cycles-pp.__task_pid_nr_ns
      0.02 ±223%      +0.2        0.26 ± 34%  perf-profile.children.cycles-pp.save_fpregs_to_fpstate
      0.06 ± 46%      +0.2        0.30 ± 26%  perf-profile.children.cycles-pp.__kmalloc
      0.04 ± 72%      +0.2        0.29 ± 20%  perf-profile.children.cycles-pp.switch_fpu_return
      0.03 ±111%      +0.2        0.28 ± 33%  perf-profile.children.cycles-pp.__d_alloc
      0.09 ± 31%      +0.2        0.33 ± 29%  perf-profile.children.cycles-pp.lru_cache_add
      0.04 ±102%      +0.2        0.28 ± 21%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.07 ± 18%      +0.2        0.32 ± 32%  perf-profile.children.cycles-pp.mark_page_accessed
      0.04 ±105%      +0.2        0.28 ± 16%  perf-profile.children.cycles-pp.__set_page_dirty
      0.19 ± 22%      +0.2        0.44 ± 20%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.03 ±103%      +0.2        0.28 ± 27%  perf-profile.children.cycles-pp.xas_start
      0.08 ± 51%      +0.2        0.33 ± 34%  perf-profile.children.cycles-pp.ext4_read_block_bitmap_nowait
      0.01 ±223%      +0.2        0.26 ± 37%  perf-profile.children.cycles-pp.__vma_adjust
      0.02 ±141%      +0.2        0.27 ± 26%  perf-profile.children.cycles-pp.__switch_to_asm
      0.28 ± 58%      +0.2        0.53 ± 31%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.11 ± 21%      +0.2        0.36 ± 24%  perf-profile.children.cycles-pp.ext4_find_extent
      0.28 ± 60%      +0.3        0.53 ± 30%  perf-profile.children.cycles-pp.file_write_and_wait_range
      0.05 ± 45%      +0.3        0.30 ± 36%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      0.05 ± 45%      +0.3        0.30 ± 36%  perf-profile.children.cycles-pp.do_mprotect_pkey
      0.03 ±100%      +0.3        0.28 ± 27%  perf-profile.children.cycles-pp.devkmsg_read
      0.01 ±223%      +0.3        0.26 ± 31%  perf-profile.children.cycles-pp.__do_sys_newfstat
      0.02 ±142%      +0.3        0.28 ± 62%  perf-profile.children.cycles-pp.do_faccessat
      0.03 ±103%      +0.3        0.29 ± 35%  perf-profile.children.cycles-pp.__split_vma
      0.04 ±105%      +0.3        0.29 ± 75%  perf-profile.children.cycles-pp.find_vma
      0.08 ± 23%      +0.3        0.34 ± 23%  perf-profile.children.cycles-pp.kfree
      0.04 ± 89%      +0.3        0.30 ± 43%  perf-profile.children.cycles-pp.timekeeping_advance
      0.06 ± 17%      +0.3        0.32 ± 37%  perf-profile.children.cycles-pp.page_mapping
      0.06 ± 87%      +0.3        0.33 ± 23%  perf-profile.children.cycles-pp.__close
      0.08 ± 21%      +0.3        0.35 ± 29%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.05 ± 83%      +0.3        0.32 ± 47%  perf-profile.children.cycles-pp.__es_remove_extent
      0.16 ±  7%      +0.3        0.43 ± 29%  perf-profile.children.cycles-pp.fixup_exception
      0.06 ± 52%      +0.3        0.33 ± 15%  perf-profile.children.cycles-pp.set_next_entity
      0.06 ± 50%      +0.3        0.33 ± 14%  perf-profile.children.cycles-pp.number
      0.07 ± 83%      +0.3        0.34 ± 28%  perf-profile.children.cycles-pp.symbol__is_idle
      0.09 ± 28%      +0.3        0.36 ± 37%  perf-profile.children.cycles-pp.perf_event_pid_type
      0.04 ±115%      +0.3        0.31 ± 31%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.06 ± 72%      +0.3        0.33 ± 33%  perf-profile.children.cycles-pp.format_decode
      0.10 ± 25%      +0.3        0.37 ± 23%  perf-profile.children.cycles-pp.ext4_journal_check_start
      0.08 ± 24%      +0.3        0.35 ± 20%  perf-profile.children.cycles-pp.jbd2_journal_stop
      0.07 ± 29%      +0.3        0.35 ± 24%  perf-profile.children.cycles-pp.__mmap
      0.09 ± 30%      +0.3        0.37 ± 20%  perf-profile.children.cycles-pp.getname_flags
      0.10 ± 25%      +0.3        0.38 ± 26%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.11 ± 34%      +0.3        0.40 ± 12%  perf-profile.children.cycles-pp.wp_page_copy
      0.10 ± 12%      +0.3        0.38 ± 26%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.12 ± 34%      +0.3        0.41 ± 28%  perf-profile.children.cycles-pp.generic_update_time
      0.08 ± 14%      +0.3        0.37 ± 33%  perf-profile.children.cycles-pp.__dentry_kill
      0.00            +0.3        0.29 ± 80%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.18 ± 12%      +0.3        0.47 ± 31%  perf-profile.children.cycles-pp.kernelmode_fixup_or_oops
      0.08 ± 25%      +0.3        0.38 ± 36%  perf-profile.children.cycles-pp.__alloc_file
      0.06 ± 77%      +0.3        0.35 ± 53%  perf-profile.children.cycles-pp.ext4_finish_bio
      0.10 ± 24%      +0.3        0.40 ± 36%  perf-profile.children.cycles-pp.ext4_read_block_bitmap
      0.05 ± 48%      +0.3        0.35 ± 23%  perf-profile.children.cycles-pp.__fxstat64
      0.04 ± 73%      +0.3        0.34 ± 10%  perf-profile.children.cycles-pp.elf_map
      0.08 ± 51%      +0.3        0.38 ± 16%  perf-profile.children.cycles-pp.rmqueue
      0.07 ± 49%      +0.3        0.38 ± 27%  perf-profile.children.cycles-pp.down_write
      0.10 ± 15%      +0.3        0.40 ± 33%  perf-profile.children.cycles-pp.submit_bio_checks
      0.09 ± 26%      +0.3        0.39 ± 17%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.07 ± 18%      +0.3        0.38 ± 23%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.05 ±101%      +0.3        0.35 ± 53%  perf-profile.children.cycles-pp.ext4_release_io_end
      0.12 ± 35%      +0.3        0.43 ± 27%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      0.07 ± 57%      +0.3        0.38 ± 51%  perf-profile.children.cycles-pp.ext4_end_io_rsv_work
      0.09 ± 27%      +0.3        0.40 ± 14%  perf-profile.children.cycles-pp.__ext4_journal_get_write_access
      0.02 ±141%      +0.3        0.33 ± 79%  perf-profile.children.cycles-pp.__fdget_pos
      0.08 ± 65%      +0.3        0.39 ± 42%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.10 ± 25%      +0.3        0.42 ± 38%  perf-profile.children.cycles-pp.alloc_empty_file
      0.09 ± 43%      +0.3        0.41 ± 31%  perf-profile.children.cycles-pp.ext4_bio_write_page
      0.11 ± 26%      +0.3        0.43 ± 23%  perf-profile.children.cycles-pp.security_file_permission
      0.05 ± 76%      +0.3        0.37 ± 28%  perf-profile.children.cycles-pp._PyEval_EvalFrameDefault
      0.08 ± 64%      +0.3        0.40 ± 44%  perf-profile.children.cycles-pp.sys_imageblit
      0.08 ± 64%      +0.3        0.40 ± 44%  perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit
      0.08 ± 18%      +0.3        0.40 ± 24%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.11 ± 25%      +0.3        0.44 ± 18%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.06 ± 50%      +0.3        0.39 ± 54%  perf-profile.children.cycles-pp.filemap_read
      0.08 ± 48%      +0.3        0.41 ± 34%  perf-profile.children.cycles-pp.ext4_extent_block_csum
      0.09 ± 51%      +0.3        0.42 ± 24%  perf-profile.children.cycles-pp.pick_link
      0.17 ± 22%      +0.3        0.50 ± 16%  perf-profile.children.cycles-pp.ext4_ext_insert_extent
      0.15 ± 19%      +0.3        0.48 ± 21%  perf-profile.children.cycles-pp.__find_get_block
      0.10 ± 31%      +0.3        0.43 ± 31%  perf-profile.children.cycles-pp.ext4_extent_block_csum_set
      0.05 ± 73%      +0.3        0.39 ± 16%  perf-profile.children.cycles-pp.__fget_light
      0.01 ±223%      +0.3        0.35 ± 26%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.11 ± 48%      +0.3        0.45 ± 23%  perf-profile.children.cycles-pp.mpage_submit_page
      0.11 ± 32%      +0.3        0.46 ± 31%  perf-profile.children.cycles-pp.__ext4_ext_dirty
      0.06 ± 52%      +0.3        0.40 ± 30%  perf-profile.children.cycles-pp.check_preempt_curr
      0.14 ± 26%      +0.4        0.50 ± 15%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      0.16 ± 32%      +0.4        0.52 ± 23%  perf-profile.children.cycles-pp.ext4_write_checks
      0.11 ± 34%      +0.4        0.47 ± 36%  perf-profile.children.cycles-pp.dput
      0.08 ± 29%      +0.4        0.45 ± 22%  perf-profile.children.cycles-pp.perf_event_mmap
      0.09 ± 22%      +0.4        0.46 ± 23%  perf-profile.children.cycles-pp.__ext4_journal_stop
      0.12 ± 20%      +0.4        0.48 ±  5%  perf-profile.children.cycles-pp.xas_load
      0.12 ± 50%      +0.4        0.49 ± 29%  perf-profile.children.cycles-pp.rcu_do_batch
      0.15 ± 24%      +0.4        0.53 ± 23%  perf-profile.children.cycles-pp.do_anonymous_page
      0.08 ±140%      +0.4        0.46 ± 35%  perf-profile.children.cycles-pp.ext4_sync_file
      0.12 ± 51%      +0.4        0.51 ± 21%  perf-profile.children.cycles-pp.mpage_map_and_submit_buffers
      0.12 ± 20%      +0.4        0.51 ± 40%  perf-profile.children.cycles-pp.__ext4_handle_dirty_metadata
      0.12 ± 33%      +0.4        0.52 ± 19%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.00            +0.4        0.40 ± 48%  perf-profile.children.cycles-pp.perf_mmap__read_init
      0.06 ± 76%      +0.4        0.45 ± 23%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.09 ± 16%      +0.4        0.50 ± 25%  perf-profile.children.cycles-pp.do_open
      0.14 ± 14%      +0.4        0.55 ± 25%  perf-profile.children.cycles-pp.__pagevec_release
      0.10 ± 15%      +0.4        0.51 ± 31%  perf-profile.children.cycles-pp.dup_mmap
      0.10 ± 30%      +0.4        0.52 ±  7%  perf-profile.children.cycles-pp.__block_commit_write
      0.10 ± 20%      +0.4        0.52 ± 22%  perf-profile.children.cycles-pp.invalidate_inode_page
      0.06 ± 72%      +0.4        0.48 ± 36%  perf-profile.children.cycles-pp.openat64
      0.42 ± 34%      +0.4        0.84 ± 18%  perf-profile.children.cycles-pp.do_writepages
      0.09 ± 23%      +0.4        0.51 ± 23%  perf-profile.children.cycles-pp.__close_nocancel
      0.08 ± 48%      +0.4        0.50 ± 44%  perf-profile.children.cycles-pp.lookup_fast
      0.08 ± 84%      +0.4        0.51 ± 57%  perf-profile.children.cycles-pp.common_interrupt
      0.10 ± 28%      +0.4        0.53 ± 17%  perf-profile.children.cycles-pp.perf_evsel__read
      0.13 ± 20%      +0.4        0.56 ± 20%  perf-profile.children.cycles-pp.zap_pte_range
      0.10 ± 33%      +0.4        0.54 ± 33%  perf-profile.children.cycles-pp.jbd2__journal_start
      0.11 ± 17%      +0.4        0.55 ± 31%  perf-profile.children.cycles-pp.dup_mm
      0.18 ± 29%      +0.4        0.62 ± 13%  perf-profile.children.cycles-pp.ext4_da_map_blocks
      0.12 ± 41%      +0.4        0.56 ±  7%  perf-profile.children.cycles-pp.__fput
      0.10 ± 29%      +0.4        0.55 ± 15%  perf-profile.children.cycles-pp.mark_buffer_dirty
      0.11 ± 30%      +0.4        0.55 ± 11%  perf-profile.children.cycles-pp.block_write_end
      0.18 ± 36%      +0.4        0.62 ± 25%  perf-profile.children.cycles-pp.file_update_time
      0.08 ± 85%      +0.4        0.53 ± 55%  perf-profile.children.cycles-pp.asm_common_interrupt
      0.12 ± 15%      +0.4        0.57 ± 19%  perf-profile.children.cycles-pp.__do_munmap
      0.18 ± 27%      +0.5        0.64 ± 14%  perf-profile.children.cycles-pp.ext4_da_get_block_prep
      0.16 ± 43%      +0.5        0.62 ± 17%  perf-profile.children.cycles-pp.map__process_kallsym_symbol
      0.12 ± 26%      +0.5        0.57 ± 11%  perf-profile.children.cycles-pp.generic_write_end
      0.22 ± 34%      +0.5        0.68 ±  5%  perf-profile.children.cycles-pp.__get_user_pages
      0.13 ± 21%      +0.5        0.59 ± 27%  perf-profile.children.cycles-pp.__lookup_slow
      0.09 ± 28%      +0.5        0.56 ± 28%  perf-profile.children.cycles-pp.put_prev_entity
      0.18 ± 26%      +0.5        0.66 ± 27%  perf-profile.children.cycles-pp.shmem_write_begin
      0.19 ± 26%      +0.5        0.67 ± 29%  perf-profile.children.cycles-pp.shmem_getpage_gfp
      0.19 ± 17%      +0.5        0.68 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.15 ± 27%      +0.5        0.64 ± 22%  perf-profile.children.cycles-pp.ext4_inode_csum
      0.14 ± 17%      +0.5        0.64 ± 22%  perf-profile.children.cycles-pp.unmap_page_range
      0.22 ± 34%      +0.5        0.72 ± 11%  perf-profile.children.cycles-pp.get_user_pages_unlocked
      0.12 ± 33%      +0.5        0.62 ± 11%  perf-profile.children.cycles-pp.proc_pid_status
      0.13 ± 33%      +0.5        0.64 ± 12%  perf-profile.children.cycles-pp.proc_single_show
      0.17 ± 26%      +0.5        0.69 ± 19%  perf-profile.children.cycles-pp.ext4_inode_csum_set
      0.15 ± 34%      +0.5        0.67 ± 18%  perf-profile.children.cycles-pp.task_work_run
      0.01 ±223%      +0.5        0.54 ± 32%  perf-profile.children.cycles-pp.swake_up_locked
      0.19 ±  9%      +0.5        0.73 ± 12%  perf-profile.children.cycles-pp.stack_access_ok
      0.45 ± 18%      +0.5        1.00 ± 10%  perf-profile.children.cycles-pp.update_load_avg
      0.15 ± 16%      +0.5        0.70 ± 21%  perf-profile.children.cycles-pp.unmap_vmas
      0.21 ± 45%      +0.5        0.76 ± 21%  perf-profile.children.cycles-pp.ext4_writepages
      0.18 ± 23%      +0.6        0.73 ± 23%  perf-profile.children.cycles-pp.filemap_map_pages
      0.20 ± 21%      +0.6        0.76 ± 22%  perf-profile.children.cycles-pp.__getblk_gfp
      0.21 ± 36%      +0.6        0.77 ± 33%  perf-profile.children.cycles-pp.open64
      0.16 ± 15%      +0.6        0.72 ± 24%  perf-profile.children.cycles-pp.begin_new_exec
      0.18 ±  6%      +0.6        0.75 ± 23%  perf-profile.children.cycles-pp.release_pages
      0.01 ±223%      +0.6        0.57 ± 35%  perf-profile.children.cycles-pp.swake_up_one
      0.19 ± 24%      +0.6        0.75 ± 10%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      0.16 ± 25%      +0.6        0.72 ± 19%  perf-profile.children.cycles-pp.__might_sleep
      0.09 ± 15%      +0.6        0.66 ± 46%  perf-profile.children.cycles-pp.mutex_lock
      0.37 ± 45%      +0.6        0.94 ± 33%  perf-profile.children.cycles-pp.start_kernel
      0.18 ± 27%      +0.6        0.75 ± 11%  perf-profile.children.cycles-pp.ext4_block_bitmap_csum_set
      0.20 ± 14%      +0.6        0.77 ± 29%  perf-profile.children.cycles-pp.kmem_cache_free
      0.16 ± 18%      +0.6        0.75 ± 16%  perf-profile.children.cycles-pp.__perf_event_header__init_id
      0.10 ± 26%      +0.6        0.68 ± 95%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.6        0.59 ± 54%  perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
      0.17 ± 33%      +0.6        0.77 ± 22%  perf-profile.children.cycles-pp.step_into
      0.16 ± 22%      +0.6        0.77 ± 19%  perf-profile.children.cycles-pp.read_counters
      0.25 ± 14%      +0.6        0.86 ± 24%  perf-profile.children.cycles-pp.kernel_text_address
      0.15 ± 24%      +0.6        0.78 ± 16%  perf-profile.children.cycles-pp.__open64_nocancel
      0.16 ± 22%      +0.6        0.79 ± 18%  perf-profile.children.cycles-pp.process_interval
      0.25 ± 31%      +0.6        0.89 ± 34%  perf-profile.children.cycles-pp.s_show
      0.38 ± 25%      +0.6        1.02 ± 39%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.16 ± 22%      +0.6        0.79 ± 17%  perf-profile.children.cycles-pp.cmd_stat
      0.16 ± 22%      +0.6        0.79 ± 17%  perf-profile.children.cycles-pp.dispatch_events
      0.22 ± 20%      +0.6        0.87 ± 18%  perf-profile.children.cycles-pp.__ext4_get_inode_loc
      0.58 ± 19%      +0.7        1.24 ± 15%  perf-profile.children.cycles-pp.blk_mq_submit_bio
      0.24 ± 20%      +0.7        0.92 ± 17%  perf-profile.children.cycles-pp.ext4_get_inode_loc
      0.18 ± 29%      +0.7        0.86 ± 23%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.09 ± 25%      +0.7        0.79 ± 28%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.26 ± 14%      +0.7        0.97 ± 20%  perf-profile.children.cycles-pp.memcpy_erms
      0.04 ±102%      +0.7        0.75 ± 48%  perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
      0.19 ± 21%      +0.7        0.90 ± 26%  perf-profile.children.cycles-pp.copy_process
      0.16 ± 20%      +0.7        0.88 ± 24%  perf-profile.children.cycles-pp.setlocale
      0.41 ± 26%      +0.7        1.14 ± 34%  perf-profile.children.cycles-pp.__generic_file_write_iter
      0.18 ± 19%      +0.7        0.91 ± 21%  perf-profile.children.cycles-pp.xas_find
      0.28 ± 11%      +0.8        1.04 ± 24%  perf-profile.children.cycles-pp.__kernel_text_address
      0.30 ± 22%      +0.8        1.05 ± 19%  perf-profile.children.cycles-pp.do_fault
      0.34 ± 19%      +0.8        1.10 ± 15%  perf-profile.children.cycles-pp.iomap_dio_submit_bio
      0.23 ± 25%      +0.8        0.99 ± 20%  perf-profile.children.cycles-pp.__ext4_journal_start_sb
      0.22 ± 22%      +0.8        1.00 ± 25%  perf-profile.children.cycles-pp.__do_sys_clone
      0.22 ± 22%      +0.8        1.00 ± 25%  perf-profile.children.cycles-pp.kernel_clone
      0.25 ± 23%      +0.8        1.02 ± 23%  perf-profile.children.cycles-pp.__libc_fork
      0.10 ± 29%      +0.8        0.88 ± 29%  perf-profile.children.cycles-pp.rcu_gp_kthread
      0.20 ± 26%      +0.8        0.98 ± 14%  perf-profile.children.cycles-pp.ext4_da_write_end
      0.29 ± 27%      +0.8        1.08 ± 19%  perf-profile.children.cycles-pp.internal_get_user_pages_fast
      0.24 ± 10%      +0.8        1.02 ± 18%  perf-profile.children.cycles-pp.perf_output_copy
      0.15 ± 38%      +0.8        0.94 ± 32%  perf-profile.children.cycles-pp.delay_tsc
      0.27 ± 48%      +0.8        1.06 ± 19%  perf-profile.children.cycles-pp.kallsyms__parse
      0.47 ±  9%      +0.8        1.27 ± 23%  perf-profile.children.cycles-pp.__get_user_nocheck_8
      0.32 ± 24%      +0.8        1.13 ± 18%  perf-profile.children.cycles-pp.iov_iter_get_pages
      0.61 ±  5%      +0.8        1.44 ± 11%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.37 ± 21%      +0.8        1.21 ± 15%  perf-profile.children.cycles-pp.ext4_handle_inode_extension
      0.38 ± 27%      +0.9        1.24 ± 14%  perf-profile.children.cycles-pp.ext4_mb_mark_diskspace_used
      0.49 ± 36%      +0.9        1.36 ± 41%  perf-profile.children.cycles-pp.build_id__mark_dso_hit
      0.39 ± 42%      +0.9        1.26 ± 35%  perf-profile.children.cycles-pp.__dso__load_kallsyms
      0.43 ± 39%      +0.9        1.30 ± 37%  perf-profile.children.cycles-pp.thread__find_map
      0.26 ± 15%      +0.9        1.14 ± 23%  perf-profile.children.cycles-pp.orc_find
      0.40 ± 41%      +0.9        1.28 ± 36%  perf-profile.children.cycles-pp.map__load
      0.16 ± 24%      +0.9        1.04 ± 20%  perf-profile.children.cycles-pp.perf_poll
      0.22 ± 32%      +0.9        1.10 ± 14%  perf-profile.children.cycles-pp.mmap_region
      0.39 ± 42%      +0.9        1.27 ± 36%  perf-profile.children.cycles-pp.dso__load
      0.33 ± 12%      +0.9        1.21 ± 21%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.37 ± 30%      +0.9        1.26 ± 33%  perf-profile.children.cycles-pp.proc_reg_read
      0.23 ± 30%      +0.9        1.12 ± 18%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.29 ± 30%      +0.9        1.19 ± 25%  perf-profile.children.cycles-pp.seq_vprintf
      0.54 ±  9%      +0.9        1.44 ± 20%  perf-profile.children.cycles-pp.perf_callchain_user
      0.30 ± 30%      +0.9        1.21 ± 24%  perf-profile.children.cycles-pp.seq_printf
      0.36 ± 24%      +0.9        1.27 ± 16%  perf-profile.children.cycles-pp.bio_iov_iter_get_pages
      0.24 ± 24%      +0.9        1.16 ± 47%  perf-profile.children.cycles-pp.finish_task_switch
      0.26 ± 23%      +0.9        1.18 ± 18%  perf-profile.children.cycles-pp.__alloc_pages
      0.26 ± 29%      +1.0        1.22 ± 12%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.26 ± 29%      +1.0        1.22 ± 12%  perf-profile.children.cycles-pp.do_group_exit
      0.26 ± 29%      +1.0        1.22 ± 12%  perf-profile.children.cycles-pp.do_exit
      0.25 ± 29%      +1.0        1.23 ± 15%  perf-profile.children.cycles-pp.do_mmap
      0.31 ± 19%      +1.0        1.30 ± 14%  perf-profile.children.cycles-pp.ext4_reserve_inode_write
      0.16 ± 24%      +1.0        1.16 ± 37%  perf-profile.children.cycles-pp.preempt_schedule_common
      0.40 ± 26%      +1.0        1.40 ± 42%  perf-profile.children.cycles-pp.__libc_write
      0.26 ± 11%      +1.0        1.27 ±  4%  perf-profile.children.cycles-pp.___might_sleep
      0.15 ± 33%      +1.0        1.18 ± 21%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.32 ± 27%      +1.0        1.36 ± 24%  perf-profile.children.cycles-pp.vsnprintf
      0.70 ± 17%      +1.0        1.75 ± 17%  perf-profile.children.cycles-pp.submit_bio_noacct
      0.20 ± 23%      +1.1        1.28 ± 15%  perf-profile.children.cycles-pp.rcu_core
      0.27 ± 29%      +1.1        1.36 ± 15%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.29 ± 23%      +1.1        1.40 ± 15%  perf-profile.children.cycles-pp.exit_mmap
      0.32 ± 21%      +1.1        1.43 ± 11%  perf-profile.children.cycles-pp.link_path_walk
      0.54 ± 19%      +1.1        1.66 ± 16%  perf-profile.children.cycles-pp.ext4_block_write_begin
      0.13 ± 23%      +1.1        1.25 ± 20%  perf-profile.children.cycles-pp.pipe_write
      0.30 ± 23%      +1.1        1.42 ± 14%  perf-profile.children.cycles-pp.mmput
      0.18 ± 32%      +1.1        1.30 ± 20%  perf-profile.children.cycles-pp.__wake_up_common
      1.09 ± 13%      +1.2        2.25 ± 30%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.39 ± 27%      +1.2        1.63 ±  8%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      0.37 ± 26%      +1.2        1.61 ± 13%  perf-profile.children.cycles-pp.crc32c_pcl_intel_update
      0.30 ± 23%      +1.3        1.56 ± 17%  perf-profile.children.cycles-pp.filename_lookup
      0.28 ± 29%      +1.3        1.55 ± 18%  perf-profile.children.cycles-pp.nsinfo__new
      0.25 ± 22%      +1.3        1.55 ± 19%  perf-profile.children.cycles-pp.__x64_sys_poll
      0.25 ± 22%      +1.3        1.56 ± 19%  perf-profile.children.cycles-pp.do_sys_poll
      0.21 ± 28%      +1.3        1.52 ± 23%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.25 ± 22%      +1.3        1.56 ± 18%  perf-profile.children.cycles-pp.__poll
      0.29 ± 28%      +1.3        1.62 ± 18%  perf-profile.children.cycles-pp.thread__new
      0.25 ± 21%      +1.3        1.58 ± 29%  perf-profile.children.cycles-pp.__cond_resched
      0.34 ± 11%      +1.3        1.66 ± 17%  perf-profile.children.cycles-pp.load_elf_binary
      0.15 ± 25%      +1.3        1.48 ± 24%  perf-profile.children.cycles-pp.pipe_read
      0.32 ± 25%      +1.3        1.65 ± 12%  perf-profile.children.cycles-pp.vfs_statx
      0.31 ± 20%      +1.4        1.66 ± 16%  perf-profile.children.cycles-pp.path_lookupat
      0.45 ± 24%      +1.4        1.80 ±  6%  perf-profile.children.cycles-pp.pagecache_get_page
      0.52 ± 28%      +1.4        1.88 ± 21%  perf-profile.children.cycles-pp.__libc_read
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.children.cycles-pp.memcpy_toio
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.children.cycles-pp.drm_fb_memcpy_dstclip
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_tail
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_planes
      0.88 ± 13%      +1.4        2.24 ± 16%  perf-profile.children.cycles-pp.mgag200_simple_display_pipe_update
      0.89 ± 13%      +1.4        2.25 ± 17%  perf-profile.children.cycles-pp.commit_tail
      0.89 ± 13%      +1.4        2.26 ± 17%  perf-profile.children.cycles-pp.drm_atomic_helper_commit
      0.30 ± 28%      +1.4        1.67 ± 19%  perf-profile.children.cycles-pp.____machine__findnew_thread
      0.33 ± 26%      +1.4        1.71 ± 12%  perf-profile.children.cycles-pp.__xstat64
      0.35 ± 11%      +1.4        1.73 ± 16%  perf-profile.children.cycles-pp.exec_binprm
      0.32 ± 20%      +1.4        1.71 ± 19%  perf-profile.children.cycles-pp.io_serial_in
      0.33 ± 26%      +1.4        1.72 ± 11%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.90 ± 13%      +1.4        2.29 ± 17%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
      0.30 ± 27%      +1.4        1.70 ± 18%  perf-profile.children.cycles-pp.machine__process_fork_event
      0.36 ± 21%      +1.4        1.76 ± 16%  perf-profile.children.cycles-pp.walk_component
      0.93 ± 13%      +1.4        2.33 ± 17%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.49 ± 23%      +1.4        1.92 ± 14%  perf-profile.children.cycles-pp.ext4_do_update_inode
      0.37 ± 20%      +1.4        1.81 ± 14%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.01 ±223%      +1.5        1.47 ± 55%  perf-profile.children.cycles-pp.perf_mmap__read_head
      0.46 ± 22%      +1.5        1.92 ± 12%  perf-profile.children.cycles-pp.ext4_dirty_inode
      0.40 ± 17%      +1.5        1.87 ± 24%  perf-profile.children.cycles-pp.find_lock_entries
      0.42 ± 10%      +1.5        1.89 ± 20%  perf-profile.children.cycles-pp.perf_output_sample
      0.39 ± 12%      +1.5        1.88 ± 17%  perf-profile.children.cycles-pp.bprm_execve
      0.41 ± 14%      +1.6        2.01 ± 16%  perf-profile.children.cycles-pp.execve
      0.58 ± 22%      +1.6        2.21 ± 16%  perf-profile.children.cycles-pp.ext4_mark_iloc_dirty
      0.66 ± 22%      +1.6        2.30 ± 15%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.30 ± 12%      +1.7        1.96 ± 14%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.51 ± 21%      +1.7        2.16 ± 12%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.56 ± 29%      +1.7        2.24 ± 20%  perf-profile.children.cycles-pp.seq_read
      0.96 ± 23%      +1.7        2.71 ± 12%  perf-profile.children.cycles-pp.ext4_mb_new_blocks
      0.27 ± 26%      +1.8        2.03 ± 19%  perf-profile.children.cycles-pp.new_sync_read
      0.74 ± 23%      +1.8        2.53 ± 13%  perf-profile.children.cycles-pp.handle_mm_fault
      6.65 ± 10%      +1.8        8.46 ± 17%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.52 ± 19%      +1.8        2.33 ± 26%  perf-profile.children.cycles-pp.ext4_free_blocks
      0.46 ± 13%      +1.8        2.28 ± 15%  perf-profile.children.cycles-pp.do_execveat_common
      0.46 ± 13%      +1.8        2.28 ± 15%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.50 ± 19%      +1.8        2.33 ± 24%  perf-profile.children.cycles-pp.path_openat
      0.51 ± 20%      +1.9        2.38 ± 23%  perf-profile.children.cycles-pp.do_filp_open
      0.62 ± 27%      +1.9        2.51 ± 17%  perf-profile.children.cycles-pp.seq_read_iter
      0.54 ± 20%      +1.9        2.43 ± 24%  perf-profile.children.cycles-pp.ext4_remove_blocks
      0.03 ±100%      +1.9        1.93 ± 52%  perf-profile.children.cycles-pp.perf_mmap__push
      0.43 ± 21%      +1.9        2.35 ± 20%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.58 ±  8%      +1.9        2.52 ± 14%  perf-profile.children.cycles-pp.__orc_find
      0.43 ± 22%      +2.0        2.38 ± 20%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.14 ± 32%      +2.0        2.14 ±  8%  perf-profile.children.cycles-pp.raise_softirq
      0.20 ± 28%      +2.0        2.22 ±  7%  perf-profile.children.cycles-pp.blk_mq_complete_request_remote
      0.20 ± 28%      +2.0        2.22 ±  7%  perf-profile.children.cycles-pp.blk_mq_complete_request
      0.84 ± 19%      +2.0        2.87 ± 10%  perf-profile.children.cycles-pp.iomap_dio_bio_actor
      0.44 ± 22%      +2.0        2.49 ± 19%  perf-profile.children.cycles-pp.uart_console_write
      0.45 ± 23%      +2.1        2.53 ± 20%  perf-profile.children.cycles-pp.serial8250_console_write
      0.58 ± 19%      +2.1        2.69 ± 22%  perf-profile.children.cycles-pp.do_sys_openat2
      0.59 ± 19%      +2.1        2.72 ± 22%  perf-profile.children.cycles-pp.do_sys_open
      0.31 ± 17%      +2.1        2.46 ± 12%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.04 ±100%      +2.2        2.24 ± 53%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.91 ± 24%      +2.2        3.16 ± 12%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.52 ± 15%      +2.3        2.77 ± 25%  perf-profile.children.cycles-pp.iomap_dio_bio_end_io
      0.63 ± 19%      +2.4        2.98 ± 24%  perf-profile.children.cycles-pp.ext4_ext_rm_leaf
      0.39 ± 23%      +2.4        2.78 ± 27%  perf-profile.children.cycles-pp.irq_work_run
      1.36 ± 25%      +2.4        3.74 ± 24%  perf-profile.children.cycles-pp.process_simple
      0.30 ± 25%      +2.4        2.69 ± 11%  perf-profile.children.cycles-pp.enqueue_entity
      0.55 ± 25%      +2.4        2.94 ± 23%  perf-profile.children.cycles-pp.printk
      0.41 ± 23%      +2.4        2.80 ± 28%  perf-profile.children.cycles-pp.irq_work_run_list
      0.39 ± 23%      +2.4        2.79 ± 28%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.39 ± 23%      +2.4        2.79 ± 28%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.40 ± 22%      +2.4        2.79 ± 28%  perf-profile.children.cycles-pp.irq_work_single
      0.39 ± 23%      +2.4        2.79 ± 28%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.88 ± 31%      +2.4        3.28 ± 13%  perf-profile.children.cycles-pp.machines__deliver_event
      1.52 ± 24%      +2.4        3.93 ± 29%  perf-profile.children.cycles-pp.perf_session__process_events
      1.52 ± 24%      +2.4        3.94 ± 28%  perf-profile.children.cycles-pp.record__finish_output
      0.40 ±  9%      +2.5        2.89 ± 12%  perf-profile.children.cycles-pp.queue_work_on
      1.02 ± 27%      +2.5        3.52 ± 19%  perf-profile.children.cycles-pp.perf_session__deliver_event
      0.56 ± 25%      +2.5        3.07 ± 23%  perf-profile.children.cycles-pp.console_unlock
      0.56 ± 25%      +2.5        3.08 ± 23%  perf-profile.children.cycles-pp.vprintk_emit
      1.38 ± 21%      +2.6        3.96 ± 12%  perf-profile.children.cycles-pp.ext4_ext_map_blocks
      0.49 ± 10%      +2.6        3.09 ± 14%  perf-profile.children.cycles-pp.__queue_work
      0.68 ± 17%      +2.6        3.28 ± 21%  perf-profile.children.cycles-pp.blk_update_request
      0.35 ± 26%      +2.6        2.96 ± 10%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.18 ± 19%      +2.7        3.85 ± 11%  perf-profile.children.cycles-pp.exc_page_fault
      0.68 ± 17%      +2.7        3.36 ± 19%  perf-profile.children.cycles-pp.blk_mq_end_request
      0.72 ± 18%      +2.7        3.41 ± 23%  perf-profile.children.cycles-pp.ext4_ext_remove_space
      0.35 ± 27%      +2.7        3.04 ± 10%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.93 ± 21%      +2.9        3.80 ± 13%  perf-profile.children.cycles-pp.__ext4_mark_inode_dirty
      0.90 ± 18%      +2.9        3.80 ± 20%  perf-profile.children.cycles-pp.blk_complete_reqs
      0.80 ± 15%      +3.0        3.81 ± 23%  perf-profile.children.cycles-pp.__invalidate_mapping_pages
      0.84 ± 18%      +3.0        3.85 ± 21%  perf-profile.children.cycles-pp.ext4_ext_truncate
      1.06 ± 22%      +3.0        4.08 ± 12%  perf-profile.children.cycles-pp.ext4_da_write_begin
      0.54 ± 27%      +3.1        3.60 ± 16%  perf-profile.children.cycles-pp.read
      1.53 ± 20%      +3.1        4.62 ± 13%  perf-profile.children.cycles-pp.ext4_map_blocks
      1.36 ± 17%      +3.1        4.47 ±  9%  perf-profile.children.cycles-pp.asm_exc_page_fault
      1.58 ± 20%      +3.2        4.82 ± 13%  perf-profile.children.cycles-pp.ext4_iomap_begin
      0.93 ±  7%      +3.4        4.29 ± 13%  perf-profile.children.cycles-pp.__unwind_start
      0.29 ± 21%      +3.7        3.95 ± 21%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.48 ±  8%      +3.7        4.15 ± 11%  perf-profile.children.cycles-pp.loop_queue_rq
      1.01 ± 19%      +3.7        4.67 ± 22%  perf-profile.children.cycles-pp.ext4_truncate
      1.04 ± 14%      +3.8        4.80 ± 24%  perf-profile.children.cycles-pp.drop_pagecache_sb
      1.05 ± 14%      +3.8        4.81 ± 24%  perf-profile.children.cycles-pp.iterate_supers
      1.07 ± 12%      +3.8        4.92 ± 24%  perf-profile.children.cycles-pp.drop_caches_sysctl_handler
      1.07 ± 12%      +3.9        4.92 ± 24%  perf-profile.children.cycles-pp.proc_sys_call_handler
      0.70 ± 11%      +3.9        4.63 ± 16%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.70 ±  8%      +4.1        4.76 ± 14%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      1.00 ± 26%      +4.1        5.06 ± 12%  perf-profile.children.cycles-pp.vfs_read
      1.16 ± 12%      +4.1        5.25 ± 18%  perf-profile.children.cycles-pp.io_schedule_timeout
      1.02 ± 26%      +4.2        5.20 ± 12%  perf-profile.children.cycles-pp.ksys_read
      0.00            +4.2        4.19 ± 11%  perf-profile.children.cycles-pp.__blk_mq_try_issue_directly
      0.00            +4.2        4.21 ± 11%  perf-profile.children.cycles-pp.blk_mq_request_issue_directly
      0.00            +4.2        4.25 ± 10%  perf-profile.children.cycles-pp.blk_mq_try_issue_list_directly
      0.75 ± 15%      +4.3        5.09 ± 17%  perf-profile.children.cycles-pp.dequeue_entity
      1.70 ± 11%      +4.4        6.08 ± 13%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.22 ± 12%      +4.4        5.66 ± 16%  perf-profile.children.cycles-pp.schedule_timeout
      1.63 ± 20%      +4.5        6.16 ± 36%  perf-profile.children.cycles-pp.cmd_sched
      0.79 ± 14%      +4.6        5.42 ± 16%  perf-profile.children.cycles-pp.dequeue_task_fair
      1.58 ± 23%      +4.6        6.21 ±  7%  perf-profile.children.cycles-pp.ext4_buffered_write_iter
      1.61 ± 23%      +4.7        6.33 ±  7%  perf-profile.children.cycles-pp.do_iter_readv_writev
      1.26 ± 18%      +4.8        6.02 ± 20%  perf-profile.children.cycles-pp.ext4_setattr
      1.65 ± 23%      +4.8        6.44 ±  7%  perf-profile.children.cycles-pp.do_iter_write
      1.79 ± 22%      +4.9        6.67 ± 10%  perf-profile.children.cycles-pp.generic_perform_write
      1.64 ± 20%      +4.9        6.53 ± 34%  perf-profile.children.cycles-pp.cmd_record
      1.30 ± 18%      +4.9        6.20 ± 21%  perf-profile.children.cycles-pp.notify_change
      1.68 ± 23%      +4.9        6.58 ±  7%  perf-profile.children.cycles-pp.lo_write_bvec
      1.32 ± 19%      +5.0        6.29 ± 20%  perf-profile.children.cycles-pp.do_truncate
      0.74 ± 15%      +5.0        5.79 ± 16%  perf-profile.children.cycles-pp.smpboot_thread_fn
      1.36 ± 18%      +5.2        6.52 ± 20%  perf-profile.children.cycles-pp.do_sys_ftruncate
      2.98 ±  6%      +5.2        8.14 ±  8%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      2.45 ± 20%      +5.3        7.79 ± 10%  perf-profile.children.cycles-pp.iomap_apply
      1.80 ± 20%      +5.5        7.33 ± 30%  perf-profile.children.cycles-pp.__libc_start_main
      1.80 ± 20%      +5.5        7.33 ± 30%  perf-profile.children.cycles-pp.main
      1.80 ± 20%      +5.5        7.33 ± 30%  perf-profile.children.cycles-pp.run_builtin
      3.24 ±  7%      +5.6        8.86 ±  8%  perf-profile.children.cycles-pp.update_curr
      1.85 ±  4%      +5.7        7.56 ± 13%  perf-profile.children.cycles-pp.unwind_next_frame
      2.49 ±  5%      +7.5       10.00 ± 12%  perf-profile.children.cycles-pp.perf_callchain_kernel
      2.09 ± 24%      +7.5        9.63 ±  4%  perf-profile.children.cycles-pp.loop_process_work
      1.33 ± 17%      +8.3        9.61 ± 12%  perf-profile.children.cycles-pp.try_to_wake_up
      3.12 ±  5%      +8.7       11.85 ± 12%  perf-profile.children.cycles-pp.get_perf_callchain
      3.15 ±  5%      +8.8       11.96 ± 12%  perf-profile.children.cycles-pp.perf_callchain
      3.99 ± 22%      +9.7       13.65 ±  3%  perf-profile.children.cycles-pp.process_one_work
      3.49 ±  6%      +9.9       13.41 ± 11%  perf-profile.children.cycles-pp.perf_prepare_sample
      2.02 ± 12%     +10.2       12.25 ±  9%  perf-profile.children.cycles-pp.schedule
      2.62 ± 12%     +11.4       14.01 ± 10%  perf-profile.children.cycles-pp.__schedule
      4.46 ± 21%     +11.7       16.16 ±  3%  perf-profile.children.cycles-pp.worker_thread
      4.04 ±  6%     +11.8       15.82 ± 12%  perf-profile.children.cycles-pp.perf_event_output_forward
      4.09 ±  6%     +12.0       16.11 ± 12%  perf-profile.children.cycles-pp.__perf_event_overflow
      4.11 ±  6%     +12.1       16.23 ± 12%  perf-profile.children.cycles-pp.perf_swevent_overflow
      4.29 ±  7%     +12.6       16.90 ± 12%  perf-profile.children.cycles-pp.perf_tp_event
      5.33 ± 20%     +17.6       22.95 ±  5%  perf-profile.children.cycles-pp.kthread
      5.35 ± 20%     +17.6       22.98 ±  5%  perf-profile.children.cycles-pp.ret_from_fork
     61.60 ±  7%     -60.9        0.75 ± 37%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      6.31 ± 21%      -5.7        0.58 ± 35%  perf-profile.self.cycles-pp.dd_queued
      0.67 ± 17%      -0.5        0.19 ± 17%  perf-profile.self.cycles-pp._find_next_bit
      0.32 ± 27%      -0.3        0.03 ±101%  perf-profile.self.cycles-pp.cpumask_next
      0.23 ± 13%      -0.1        0.10 ± 49%  perf-profile.self.cycles-pp.intel_idle
      0.21 ± 14%      -0.1        0.13 ± 34%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.getname_flags
      0.05 ± 47%      +0.1        0.11 ± 22%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.1        0.07 ± 43%  perf-profile.self.cycles-pp.ext4_read_block_bitmap_nowait
      0.00            +0.1        0.07 ± 28%  perf-profile.self.cycles-pp.stop_this_handle
      0.00            +0.1        0.07 ± 18%  perf-profile.self.cycles-pp.asm_exc_page_fault
      0.00            +0.1        0.08 ± 41%  perf-profile.self.cycles-pp.kernel_fpu_begin_mask
      0.00            +0.1        0.08 ± 22%  perf-profile.self.cycles-pp.down_read
      0.02 ±144%      +0.1        0.10 ± 37%  perf-profile.self.cycles-pp.ext4_mark_iloc_dirty
      0.00            +0.1        0.08 ± 37%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.01 ±223%      +0.1        0.09 ± 18%  perf-profile.self.cycles-pp.blk_update_request
      0.01 ±223%      +0.1        0.09 ± 47%  perf-profile.self.cycles-pp.sync_regs
      0.00            +0.1        0.08 ± 26%  perf-profile.self.cycles-pp.try_charge_memcg
      0.00            +0.1        0.08 ± 50%  perf-profile.self.cycles-pp.start_this_handle
      0.01 ±223%      +0.1        0.09 ± 40%  perf-profile.self.cycles-pp.timestamp_truncate
      0.01 ±223%      +0.1        0.09 ± 23%  perf-profile.self.cycles-pp.find_vma
      0.00            +0.1        0.08 ± 35%  perf-profile.self.cycles-pp.__es_insert_extent
      0.02 ±142%      +0.1        0.10 ± 29%  perf-profile.self.cycles-pp.vfs_write
      0.00            +0.1        0.08 ± 30%  perf-profile.self.cycles-pp.snmp_get_cpu_field
      0.00            +0.1        0.09 ± 33%  perf-profile.self.cycles-pp.__ext4_journal_start_sb
      0.02 ±142%      +0.1        0.10 ± 16%  perf-profile.self.cycles-pp.mark_buffer_dirty
      0.00            +0.1        0.09 ± 35%  perf-profile.self.cycles-pp.refill_obj_stock
      0.00            +0.1        0.09 ± 32%  perf-profile.self.cycles-pp.reweight_entity
      0.00            +0.1        0.09 ± 24%  perf-profile.self.cycles-pp.perf_misc_flags
      0.00            +0.1        0.09 ± 32%  perf-profile.self.cycles-pp.vmacache_find
      0.03 ±102%      +0.1        0.12 ± 48%  perf-profile.self.cycles-pp.perf_callchain_user
      0.00            +0.1        0.09 ± 47%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.10 ± 19%  perf-profile.self.cycles-pp.ext4_reserve_inode_write
      0.05 ± 75%      +0.1        0.14 ± 43%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.10 ± 45%  perf-profile.self.cycles-pp.notify_change
      0.01 ±223%      +0.1        0.11 ± 41%  perf-profile.self.cycles-pp.ext4_mb_release_context
      0.00            +0.1        0.10 ± 34%  perf-profile.self.cycles-pp.ext4_setattr
      0.01 ±223%      +0.1        0.11 ± 35%  perf-profile.self.cycles-pp.perf_callchain
      0.00            +0.1        0.10 ± 27%  perf-profile.self.cycles-pp.lock_page_memcg
      0.00            +0.1        0.10 ± 37%  perf-profile.self.cycles-pp.generic_perform_write
      0.01 ±223%      +0.1        0.11 ± 32%  perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
      0.03 ±100%      +0.1        0.13 ± 40%  perf-profile.self.cycles-pp.get_callchain_entry
      0.02 ±142%      +0.1        0.12 ± 37%  perf-profile.self.cycles-pp.cpuacct_charge
      0.00            +0.1        0.10 ± 52%  perf-profile.self.cycles-pp.io_serial_out
      0.00            +0.1        0.10 ± 51%  perf-profile.self.cycles-pp.blk_mq_flush_plug_list
      0.03 ±100%      +0.1        0.13 ± 31%  perf-profile.self.cycles-pp.gup_pgd_range
      0.00            +0.1        0.11 ± 39%  perf-profile.self.cycles-pp.select_idle_cpu
      0.06 ± 49%      +0.1        0.17 ± 45%  perf-profile.self.cycles-pp.kallsyms_expand_symbol
      0.01 ±223%      +0.1        0.12 ± 33%  perf-profile.self.cycles-pp.perf_tp_event_match
      0.03 ±101%      +0.1        0.14 ± 46%  perf-profile.self.cycles-pp.ext4_es_lookup_extent
      0.04 ±105%      +0.1        0.15 ± 45%  perf-profile.self.cycles-pp.loop_process_work
      0.03 ±100%      +0.1        0.14 ± 34%  perf-profile.self.cycles-pp.kmem_cache_alloc_trace
      0.07 ± 26%      +0.1        0.18 ± 22%  perf-profile.self.cycles-pp.pagecache_get_page
      0.00            +0.1        0.11 ± 30%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.01 ±223%      +0.1        0.12 ± 27%  perf-profile.self.cycles-pp.common_file_perm
      0.10 ± 34%      +0.1        0.21 ± 28%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.02 ±141%      +0.1        0.13 ± 41%  perf-profile.self.cycles-pp.ftrace_graph_ret_addr
      0.02 ±141%      +0.1        0.14 ± 34%  perf-profile.self.cycles-pp.finish_task_switch
      0.01 ±223%      +0.1        0.12 ± 47%  perf-profile.self.cycles-pp.aa_file_perm
      0.00            +0.1        0.12 ± 31%  perf-profile.self.cycles-pp.ext4_map_blocks
      0.00            +0.1        0.12 ± 20%  perf-profile.self.cycles-pp.__irqentry_text_end
      0.14 ± 18%      +0.1        0.26 ± 21%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.1        0.12 ± 35%  perf-profile.self.cycles-pp.__rblist__findnew
      0.00            +0.1        0.12 ± 55%  perf-profile.self.cycles-pp.ext4_ext_remove_space
      0.02 ±142%      +0.1        0.14 ± 45%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.00            +0.1        0.12 ± 13%  perf-profile.self.cycles-pp.count_free
      0.01 ±223%      +0.1        0.13 ± 34%  perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.12 ± 48%  perf-profile.self.cycles-pp.set_next_entity
      0.04 ± 75%      +0.1        0.17 ± 23%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.02 ±141%      +0.1        0.14 ± 30%  perf-profile.self.cycles-pp.__kmalloc
      0.01 ±223%      +0.1        0.14 ± 41%  perf-profile.self.cycles-pp.dequeue_entity
      0.00            +0.1        0.12 ± 32%  perf-profile.self.cycles-pp.perf_trace_buf_update
      0.03 ±103%      +0.1        0.16 ± 23%  perf-profile.self.cycles-pp.ext4_block_write_begin
      0.00            +0.1        0.13 ± 55%  perf-profile.self.cycles-pp.ext4_da_write_begin
      0.06 ± 17%      +0.1        0.19 ± 31%  perf-profile.self.cycles-pp.ext4_ext_map_blocks
      0.01 ±223%      +0.1        0.14 ± 35%  perf-profile.self.cycles-pp.map_id_range_down
      0.00            +0.1        0.13 ± 60%  perf-profile.self.cycles-pp.mb_free_blocks
      0.01 ±223%      +0.1        0.14 ± 38%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.03 ±100%      +0.1        0.16 ± 27%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.01 ±223%      +0.1        0.14 ± 36%  perf-profile.self.cycles-pp.string_nocheck
      0.00            +0.1        0.13 ± 39%  perf-profile.self.cycles-pp.ext4_es_can_be_merged
      0.00            +0.1        0.13 ± 17%  perf-profile.self.cycles-pp.xas_store
      0.03 ±100%      +0.1        0.16 ± 35%  perf-profile.self.cycles-pp.__calc_delta
      0.00            +0.1        0.13 ± 27%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.04 ±102%      +0.1        0.17 ± 26%  perf-profile.self.cycles-pp.map__process_kallsym_symbol
      0.02 ±141%      +0.1        0.16 ± 50%  perf-profile.self.cycles-pp.symbol__is_idle
      0.01 ±223%      +0.1        0.14 ± 22%  perf-profile.self.cycles-pp.invalidate_inode_page
      0.00            +0.1        0.14 ± 27%  perf-profile.self.cycles-pp.__perf_event_account_interrupt
      0.00            +0.1        0.14 ± 38%  perf-profile.self.cycles-pp.__fget_light
      0.01 ±223%      +0.1        0.15 ± 40%  perf-profile.self.cycles-pp.__pagevec_lru_add
      0.01 ±223%      +0.1        0.15 ± 23%  perf-profile.self.cycles-pp.rmqueue
      0.03 ±100%      +0.1        0.17 ± 15%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.00            +0.1        0.14 ± 42%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.00            +0.1        0.14 ± 43%  perf-profile.self.cycles-pp.__ext4_mark_inode_dirty
      0.01 ±223%      +0.1        0.15 ± 45%  perf-profile.self.cycles-pp.ext4_free_blocks
      0.00            +0.1        0.14 ± 25%  perf-profile.self.cycles-pp.mod_objcg_state
      0.00            +0.1        0.14 ± 57%  perf-profile.self.cycles-pp.vfprintf
      0.01 ±223%      +0.1        0.15 ± 27%  perf-profile.self.cycles-pp.__alloc_pages
      0.03 ±144%      +0.1        0.18 ± 28%  perf-profile.self.cycles-pp.__get_user_pages
      0.00            +0.1        0.15 ± 14%  perf-profile.self.cycles-pp.filemap_map_pages
      0.00            +0.1        0.15 ± 61%  perf-profile.self.cycles-pp.__perf_event_overflow
      0.00            +0.1        0.15 ± 32%  perf-profile.self.cycles-pp.malloc
      0.07 ± 50%      +0.2        0.22 ± 12%  perf-profile.self.cycles-pp.do_user_addr_fault
      0.03 ±111%      +0.2        0.18 ± 46%  perf-profile.self.cycles-pp.__es_remove_extent
      0.02 ±143%      +0.2        0.18 ± 21%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.01 ±223%      +0.2        0.16 ± 43%  perf-profile.self.cycles-pp.security_file_permission
      0.01 ±223%      +0.2        0.17 ± 42%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.01 ±223%      +0.2        0.16 ± 29%  perf-profile.self.cycles-pp.down_write
      0.02 ±144%      +0.2        0.18 ± 23%  perf-profile.self.cycles-pp.__ext4_journal_get_write_access
      0.03 ±100%      +0.2        0.18 ± 40%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +0.2        0.16 ± 32%  perf-profile.self.cycles-pp.clear_page_erms
      0.01 ±223%      +0.2        0.17 ± 25%  perf-profile.self.cycles-pp.unwind_get_return_address
      0.00            +0.2        0.16 ± 41%  perf-profile.self.cycles-pp.map_id_up
      0.00            +0.2        0.16 ± 33%  perf-profile.self.cycles-pp.__switch_to
      0.06 ± 55%      +0.2        0.22 ± 37%  perf-profile.self.cycles-pp.vsnprintf
      0.02 ±141%      +0.2        0.18 ± 30%  perf-profile.self.cycles-pp.__kernel_text_address
      0.16 ± 18%      +0.2        0.32 ± 19%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.00            +0.2        0.16 ± 50%  perf-profile.self.cycles-pp.ext4_ext_rm_leaf
      0.05 ± 72%      +0.2        0.22 ± 32%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.09 ± 20%      +0.2        0.26 ± 30%  perf-profile.self.cycles-pp.error_entry
      0.03 ±101%      +0.2        0.20 ± 38%  perf-profile.self.cycles-pp.__iomap_dio_rw
      0.08 ± 31%      +0.2        0.25 ± 21%  perf-profile.self.cycles-pp.next_uptodate_page
      0.04 ± 77%      +0.2        0.22 ± 42%  perf-profile.self.cycles-pp.___perf_sw_event
      0.03 ±100%      +0.2        0.20 ± 28%  perf-profile.self.cycles-pp.sys_imageblit
      0.11 ± 19%      +0.2        0.28 ± 17%  perf-profile.self.cycles-pp.__find_get_block
      0.06 ± 47%      +0.2        0.24 ± 27%  perf-profile.self.cycles-pp.xas_load
      0.02 ±223%      +0.2        0.19 ± 29%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.03 ±100%      +0.2        0.21 ± 41%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.00            +0.2        0.18 ± 26%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.00            +0.2        0.18 ± 66%  perf-profile.self.cycles-pp.__module_address
      0.01 ±223%      +0.2        0.19 ± 34%  perf-profile.self.cycles-pp.kvm_is_in_guest
      0.01 ±223%      +0.2        0.19 ± 40%  perf-profile.self.cycles-pp.__invalidate_mapping_pages
      0.03 ±103%      +0.2        0.22 ± 23%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.00            +0.2        0.19 ± 39%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.00            +0.2        0.20 ± 68%  perf-profile.self.cycles-pp.record__mmap_read_evlist
      0.00            +0.2        0.20 ± 39%  perf-profile.self.cycles-pp.__perf_event_header__init_id
      0.03 ±101%      +0.2        0.23 ± 26%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.13 ± 29%      +0.2        0.33 ± 28%  perf-profile.self.cycles-pp.update_curr
      0.06 ± 46%      +0.2        0.26 ± 25%  perf-profile.self.cycles-pp.memset_erms
      0.07 ± 17%      +0.2        0.27 ± 33%  perf-profile.self.cycles-pp.__entry_text_start
      0.04 ± 75%      +0.2        0.25 ± 36%  perf-profile.self.cycles-pp.try_to_wake_up
      0.04 ± 71%      +0.2        0.24 ± 31%  perf-profile.self.cycles-pp.__cond_resched
      0.02 ±142%      +0.2        0.23 ± 26%  perf-profile.self.cycles-pp.__fget_files
      0.07 ± 49%      +0.2        0.28 ± 27%  perf-profile.self.cycles-pp.kfree
      0.02 ±141%      +0.2        0.23 ± 13%  perf-profile.self.cycles-pp.__restore_fpregs_from_fpstate
      0.08 ± 31%      +0.2        0.30 ± 19%  perf-profile.self.cycles-pp.copy_page_from_iter_atomic
      0.05 ± 45%      +0.2        0.26 ± 30%  perf-profile.self.cycles-pp.mark_page_accessed
      0.05 ± 46%      +0.2        0.27 ± 17%  perf-profile.self.cycles-pp.zap_pte_range
      0.07 ± 20%      +0.2        0.29 ± 40%  perf-profile.self.cycles-pp.perf_output_begin_forward
      0.06 ± 17%      +0.2        0.28 ± 46%  perf-profile.self.cycles-pp.page_mapping
      0.06 ± 48%      +0.2        0.28 ± 27%  perf-profile.self.cycles-pp.get_perf_callchain
      0.02 ±141%      +0.2        0.24 ± 33%  perf-profile.self.cycles-pp.xas_start
      0.08 ± 10%      +0.2        0.30 ± 18%  perf-profile.self.cycles-pp.perf_tp_event
      0.05 ± 49%      +0.2        0.28 ± 25%  perf-profile.self.cycles-pp.mutex_lock
      0.13 ± 17%      +0.2        0.36 ± 29%  perf-profile.self.cycles-pp.__get_user_nocheck_8
      0.01 ±223%      +0.2        0.24 ± 33%  perf-profile.self.cycles-pp.save_fpregs_to_fpstate
      0.09 ± 18%      +0.2        0.33 ± 18%  perf-profile.self.cycles-pp.kernel_text_address
      0.07 ± 50%      +0.2        0.31 ± 21%  perf-profile.self.cycles-pp.__ext4_get_inode_loc
      0.05 ± 74%      +0.2        0.30 ± 39%  perf-profile.self.cycles-pp.__task_pid_nr_ns
      0.05 ± 74%      +0.2        0.30 ± 15%  perf-profile.self.cycles-pp.number
      0.02 ±141%      +0.2        0.27 ± 26%  perf-profile.self.cycles-pp.__switch_to_asm
      0.14 ± 25%      +0.3        0.39 ± 21%  perf-profile.self.cycles-pp.update_load_avg
      0.08 ± 22%      +0.3        0.34 ± 30%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.06 ± 71%      +0.3        0.32 ± 32%  perf-profile.self.cycles-pp.format_decode
      0.10 ± 24%      +0.3        0.37 ± 18%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.01 ±223%      +0.3        0.28 ± 29%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.10 ± 14%      +0.3        0.37 ± 28%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.00            +0.3        0.28 ± 45%  perf-profile.self.cycles-pp.perf_mmap__read_init
      0.11 ± 20%      +0.3        0.40 ± 34%  perf-profile.self.cycles-pp.perf_prepare_sample
      0.04 ± 74%      +0.3        0.34 ± 23%  perf-profile.self.cycles-pp._PyEval_EvalFrameDefault
      0.08 ± 17%      +0.3        0.39 ± 30%  perf-profile.self.cycles-pp.release_pages
      0.08 ± 77%      +0.3        0.40 ± 34%  perf-profile.self.cycles-pp.kallsyms__parse
      0.10 ± 19%      +0.3        0.41 ± 21%  perf-profile.self.cycles-pp.perf_output_copy
      0.09 ± 19%      +0.3        0.40 ± 22%  perf-profile.self.cycles-pp.__unwind_start
      0.12 ± 15%      +0.3        0.46 ± 37%  perf-profile.self.cycles-pp.kmem_cache_free
      0.08 ± 23%      +0.4        0.44 ± 14%  perf-profile.self.cycles-pp.perf_poll
      0.14 ± 14%      +0.4        0.55 ± 44%  perf-profile.self.cycles-pp.drop_pagecache_sb
      0.24 ± 12%      +0.4        0.66 ± 15%  perf-profile.self.cycles-pp._raw_spin_lock
      0.13 ± 29%      +0.4        0.56 ± 24%  perf-profile.self.cycles-pp.crc32c_pcl_intel_update
      0.18 ± 19%      +0.5        0.65 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.12 ± 33%      +0.5        0.62 ± 29%  perf-profile.self.cycles-pp.__schedule
      0.14 ± 29%      +0.5        0.64 ± 24%  perf-profile.self.cycles-pp.ext4_do_update_inode
      0.18 ± 18%      +0.5        0.70 ± 11%  perf-profile.self.cycles-pp.perf_callchain_kernel
      0.13 ± 28%      +0.5        0.65 ± 21%  perf-profile.self.cycles-pp.__might_sleep
      0.17 ± 10%      +0.5        0.70 ± 11%  perf-profile.self.cycles-pp.stack_access_ok
      0.16 ± 18%      +0.5        0.69 ± 22%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.14 ± 18%      +0.5        0.69 ± 27%  perf-profile.self.cycles-pp.xas_find
      0.14 ± 24%      +0.6        0.76 ± 23%  perf-profile.self.cycles-pp.perf_output_sample
      0.24 ± 13%      +0.7        0.93 ± 18%  perf-profile.self.cycles-pp.memcpy_erms
      0.22 ± 14%      +0.8        1.00 ± 29%  perf-profile.self.cycles-pp.find_lock_entries
      0.15 ± 38%      +0.8        0.94 ± 32%  perf-profile.self.cycles-pp.delay_tsc
      0.25 ± 16%      +0.8        1.04 ± 23%  perf-profile.self.cycles-pp.orc_find
      0.24 ± 11%      +0.9        1.16 ±  6%  perf-profile.self.cycles-pp.___might_sleep
      0.00            +1.1        1.08 ± 55%  perf-profile.self.cycles-pp.perf_mmap__read_head
      0.86 ± 15%      +1.2        2.07 ± 18%  perf-profile.self.cycles-pp.memcpy_toio
      0.32 ± 20%      +1.4        1.70 ± 19%  perf-profile.self.cycles-pp.io_serial_in
      0.58 ±  8%      +1.9        2.51 ± 14%  perf-profile.self.cycles-pp.__orc_find
      0.78 ±  9%      +2.3        3.06 ± 15%  perf-profile.self.cycles-pp.unwind_next_frame


                                                                                
                    fxmark.hdd_ext4_no_jnl_DWTL_1_directio.secs                 
                                                                                
  2.2 +---------------------------------------------------------------------+   
      | O O O     O     O OOO  OO    O                                      |   
    2 |O+        O                                                          |   
      |    O        O O               O      O                              |   
  1.8 |-+            O O           O   O  O O  O O O                        |   
      |      O OO            O   OO     O       O                           |   
  1.6 |-+                                                                   |   
      |                                                                     |   
  1.4 |-+                                  O      O                         |   
      |                                                                     |   
  1.2 |-+                                                     +             |   
      |                                                       ::            |   
    1 |-+                                                     ::            |   
      |++.++++.++++.+++++.++++.+++++. +++.++++.++++ .++++.+++:  ++++.++  .++|   
  0.8 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                  fxmark.hdd_ext4_no_jnl_DWTL_1_directio.works_sec              
                                                                                
  30000 +-------------------------------------------------------------------+   
  28000 |-+                           +             +        .+        ++.  |   
        |++.+++++.+ +++.++++++.+++++.+ +++.+++++.+++ :.++++++ : +++.  +   ++|   
  26000 |-+        +                                 +        : :   ++      |   
  24000 |-+                                                    ::           |   
        |                                                      :            |   
  22000 |-+                                                    :            |   
  20000 |-+                                                    +            |   
  18000 |-+                                                                 |   
        |                                   O      O                        |   
  16000 |-+                                                                 |   
  14000 |-+    OO O            O  OO     O       O                          |   
        |    O       OO OO           O OO  O OOO  O O                       |   
  12000 |OO O O    OO     OOOO  OO    O                                     |   
  10000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fxmark.time.system_time                            
                                                                                
  55 +----------------------------------------------------------------------+   
     |                                                                      |   
  50 |-+                 +     +     +   +     +        +  +      +         |   
     | ::       ++       :     :     ::  :     ::       : ::      :         |   
     |: :       : +     : :   : :  +: : : :   : :  +   : :: :    : :.+   .++|   
  45 |++ ++++.++   + + .+ ++. + +.+ +  :+ +.+ +  ++ +. + +  +++.++ +  :++   |   
     |              + +      +         +     +        +               +     |   
  40 |-+                                                                    |   
     |                                                                      |   
  35 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
  30 |-+                                                                    |   
     |OO OOOO OOOO OOOO OOOO OOOO OOOO OOOO OOOO OOO                        |   
  25 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                     fxmark.time.percent_of_cpu_this_job_got                    
                                                                                
  28 +----------------------------------------------------------------------+   
     |                               +                  +         +         |   
  26 |-+.       ++.      +     +     ::  +     +.       ::.+      ::        |   
     |+  ++ +.+ :  + +  + ++  + +.+++ : + +.+ +  ++++  + +  ++  ++ +.+ ++.++|   
     |:    ::  :    :::+    + :        ::    ::      : :      :+     : :    |   
  24 |-+   +   +    + +      +         +     +       ::       +       :     |   
     |                                                +               +     |   
  22 |-+                                                                    |   
     |                                                                      |   
  20 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
  18 |-+                                                                    |   
     |OO OOOO OOOO OOOO OOOO OOOO OOOO OOOO OOOO OOO                        |   
  16 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fxmark.time.elapsed_time                           
                                                                                
  215 +---------------------------------------------------------------------+   
      | :                 :     :         :     :                           |   
  210 |-::               ::     :        ::     :                           |   
      | ::               : :    ::    +  : :    ::      +  +      +         |   
      |: :  +            : :   : :    :  : :   : :    + :+ :      :         |   
  205 |:+ : :    ++     :  :   : :   : ::  :   : :    : : + :     ::  ++    |   
      |+  +: :   : +  + +  +  .+ + +.+ :+  +  .+ + +.: :    +    : +.+  :.++|   
  200 |-+  + +.++   ++ +    ++    +    +    ++    +  + +     ++.++      +   |   
      |                                                                     |   
  195 |-+                                                                   |   
      |                                    O      O                         |   
      |                     OO  OO O   O  O O  O O O                        |   
  190 |OO OOO       OOO O OO   O  O  OO      O                              |   
      |      O OOOO    O                O       O                           |   
  185 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           fxmark.time.elapsed_time.max                         
                                                                                
  215 +---------------------------------------------------------------------+   
      | :                 :     :         :     :                           |   
  210 |-::               ::     :        ::     :                           |   
      | ::               : :    ::    +  : :    ::      +  +      +         |   
      |: :  +            : :   : :    :  : :   : :    + :+ :      :         |   
  205 |:+ : :    ++     :  :   : :   : ::  :   : :    : : + :     ::  ++    |   
      |+  +: :   : +  + +  +  .+ + +.+ :+  +  .+ + +.: :    +    : +.+  :.++|   
  200 |-+  + +.++   ++ +    ++    +    +    ++    +  + +     ++.++      +   |   
      |                                                                     |   
  195 |-+                                                                   |   
      |                                    O      O                         |   
      |                     OO  OO O   O  O O  O O O                        |   
  190 |OO OOO       OOO O OO   O  O  OO      O                              |   
      |      O OOOO    O                O       O                           |   
  185 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       perf-sched.total_sch_delay.average.ms                    
                                                                                
  0.24 +--------------------------------------------------------------------+   
       | :                :     :         :     :                           |   
  0.22 |-:                :     :         :     :                           |   
   0.2 |-::               ::    ::        :     :                           |   
       | ::               ::    ::        :     :                           |   
  0.18 |:+:              : :   : :       : :   : :       +                  |   
  0.16 |:+:              : :   : :       : :   : :       :        +         |   
       |:  : +    +      :  :  :  :   +  : :   : :      : :+      :+       +|   
  0.14 |++ :+ :   :+ .++ +  :+ +  :+++ + + :.+ + :.++++ : + :+. +:  ++++ .+ |   
  0.12 |-+ +  ++.+  +   +   + +   +     +  +  +  +     :    +  + +      +   |   
       |                                               +                    |   
   0.1 |-+                                                                  |   
  0.08 |O+ OOOOO OOO    OOO        O  O       O                             |   
       | O          O OO    OOOOO O OO  OO   O OO   O                       |   
  0.06 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
         perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork        
                                                                                
  45 +----------------------------------------------------------------------+   
     | :                 :     :         :     :                            |   
  40 |-:                 :     :         :     :        +                   |   
  35 |-::                :     :         :     ::       :  +                |   
     | ::  +             :     :     +   :     ::       :  :      +         |   
  30 |:+:  :            : :   : :    :  : :   : :     + ::: :     :         |   
  25 |:+:  :    +    +  : :   : :  + :: : :   : :  + ::: :: :     ::.+++  ++|   
     |:  :: :   :+   :  : :   : :  :: : : :   :  : ::: : +  +    : +    :+  |   
  20 |++ :+ :  +  + : :.+ :+. + :.+ +  :+ :.+ +  :+ +  :     : .+:      +   |   
  15 |-+ +  +.+    ++ +   +  +  +      +  +  +   +     +     ++  +          |   
     |                                                                      |   
  10 |-+                                                                    |   
   5 |-+                                                                    |   
     |OO OOOO OOOO OOOO OOOO OOOO OOOO OOOO OOOO OOO                        |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
          perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork       
                                                                                
   0.5 +--------------------------------------------------------------------+   
  0.45 |-+                +     +         +     +                           |   
       | :                :     :         :     :                           |   
   0.4 |-::               ::    ::        :     :                           |   
  0.35 |-::               ::    ::        :     :                           |   
       |: :              : :   : :       : :   : :                          |   
   0.3 |:+:              : :   : :       : :   : :       + +                |   
  0.25 |:+ : +           :  :  :  :   +. : :   : :  +   : :::     +.        |   
   0.2 |++ :+ ++.++++.++++  :+++  :+++  ++ :.+++ :.+ ++ : + ++.+++  +++++.++|   
       |   +                +     +        +     +     +                    |   
  0.15 |-+                                                                  |   
   0.1 |-+                                                                  |   
       | O  OO    O       O O OO  O  O  O  O   O   O                        |   
  0.05 |O+ O  OO O OO OOOO   O  O  OO O  OO  OO OO  O                       |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
      perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork      
                                                                                
  50 +----------------------------------------------------------------------+   
  45 |-+                 +     +         +     +                            |   
     | :                 :     :         :     :                            |   
  40 |-:                 :     :         :     :        +                   |   
  35 |-::                :     :         :     ::       :  +                |   
     |: :  +            ::    ::     +  ::    : :     + :  :      +         |   
  30 |:+:  :            : :   : :    :: : :   : :  + :: ::: :     :         |   
  25 |:+:  ::   +    +  : :   : :  + :: : :   : :  ::::: :: :     ::.+++ .++|   
  20 |++ :: :   :+.  ::.+ :   + :  ::  :+ :   +  :: +  : +  +   +: +    +   |   
     |   :+ +. +   +: +   :+.+  :.+ +  +  :.++   :+    :     + + +          |   
  15 |-+ +    +     +     +     +         +      +     +      +             |   
  10 |-+                                                                    |   
     |                                                                      |   
   5 |OO OOOO OOOO OOOO OOOO OOOO OOOO OOOO OOOO OOO                        |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork     
                                                                                
  0.22 +--------------------------------------------------------------------+   
       |+                +     +         +     +                            |   
   0.2 |:+               :     :         :     :                            |   
  0.18 |::               ::    ::        ::    ::                           |   
       |:+               :+    :+        :+    :+     +                     |   
  0.16 |-+:            +:  :  :  :      : :   : :     :   +          ++  .+ |   
       |  :  +    +    ::  :  :  :   +  :  :  :  :   : :  : +        : ++  +|   
  0.14 |-+ ++ + .+:    :+   +++   +++ +.+  +.++  +.+++ :.+ :::      +       |   
       |       +   :+.:                                +   : +.+++ +        |   
  0.12 |-+         +  +                                    +      +         |   
   0.1 |-+                                                                  |   
       |                                                                    |   
  0.08 |O+  O     OOO    O  O  O     O    O  O   O  O                       |   
       |   O OOO O    OO  O  OO O OOO O O  O  OO   O                        |   
  0.06 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
       perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork  
                                                                                
  18000 +-------------------------------------------------------------------+   
        |                                O       O                          |   
  16000 |-+ OOO O  O     OO  O O OOO OO O  O   O  O                         |   
  14000 |O+    O  O OOO O  OO   O      O     OO     O                       |   
        |                                                                   |   
  12000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
        |                                                                   |   
   8000 |-+           +                                                     |   
   6000 |-+ +   +.    :: +  + .+  +     +   + +   +     +      + +          |   
        |: : + +  +  + : :: :+  : :+. + ::  :+ :  :+ +. ::   .+ :::         |   
   4000 |++:  +    ++   + +:    +:   + : +.:   +.:  +  + ++++   + +.+ +++.+ |   
        | +                +     +     +   +     +                   +     +|   
   2000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  0.6 +---------------------------------------------------------------------+   
      | +                 +     +     :   +     +       +  +      +         |   
  0.5 |-:        +       ::     :     :  ::     :       :: ::     :         |   
      |: :       :+.     ::    ::    ::  ::    ::       ::: +     ::        |   
      |+ :  + .+:   +   +  :   + : +.+ :+  :   + : +.   : +  :    :+.+   .++|   
  0.4 |:+: + +  +    :+ :  :+  : :+    ::  :+  : :+  + :     + .+:    + +   |   
      |:  ::         +: :  :: :  ::    ::  :: :  ::   +:      + ::     +    |   
  0.3 |-+ +            :   + ::  +     :   + ::  +     :         :          |   
      |                +     +         +     +         +         +          |   
  0.2 |-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
  0.1 |-+                                                                   |   
      |OO OOOO OOOO OOOOO OOOO OOOOO OOOO OOOO OOOOO                        |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  450 +---------------------------------------------------------------------+   
      | +                 +     +         +     +                           |   
  400 |-:                 :     :         :     :                           |   
  350 |-::               ::     :        ::     :                           |   
      | ::               ::     :     +  ::     :       +  +      +         |   
  300 |:+:  +     +      : :   : :    :  : :   : :      :+ :      :    +    |   
  250 |:+:  :    + :     : :   : :    :  : :   : :    + : + :     ::   :  + |   
      |:  :: :   : :    :  :   : : + : ::  :   : : +. ::    +    : :.+: :+ :|   
  200 |++ :+ +. +   : + +  :+ .+ :+ +: :+  :+ .+ :+  + :     : .+: +  + +  +|   
  150 |-+ +    +    ++ +   + +   +   + +   + +   +     +     ++  +          |   
      |                                                                     |   
  100 |-+                                                                   |   
   50 |-+                                                                   |   
      |   O O       OO    OO O O O O OOO  OOOO O OOO                        |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  800 +---------------------------------------------------------------------+   
      |       +:       +     +         +     +                   +          |   
  700 |-+.++ +  :   ++ :   ++ :  ++  + :   ++ :  ++    +     ++  :          |   
  600 |:+  : :  +  : : :: +   : + :  : :: +   : + :    :     : : ::         |   
      |::   ::  :  : : ::+     :: : : : :+     :: :    ::    : :: :         |   
  500 |++   :    :+   : +      +   :: : +      +   : + :+   :   + ++.   +   |   
      |     +    +    :            :  +            :+::  + +:        +  :+ +|   
  400 |-+             :            +               +  :   + +         ++  + |   
      |               +                               +                     |   
  300 |-+                                                                   |   
  200 |-+                                                                   |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                                                                
                                                                                
  12 +----------------------------------------------------------------------+   
     |                                                                      |   
  10 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
   8 |-+                                                                    |   
     |                                                                      |   
   6 |-+                                                                    |   
     |                                                                      |   
   4 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
   2 |-+.                +     +         +     +.    .+        .+           |   
     |+  ++++.++++.++++.+ ++.++ +.++++.++ +.+++  ++++  +++.++++  +++.++++.++|   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.14.0-rc4-00039-g2112f5c1330a" of type "text/plain" (175326 bytes)

View attachment "job-script" of type "text/plain" (8174 bytes)

View attachment "job.yaml" of type "text/plain" (5544 bytes)

View attachment "reproduce" of type "text/plain" (254 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ