lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Thu, 3 Mar 2022 23:31:08 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Mel Gorman <mgorman@...hsingularity.net>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        "Gautham R. Shenoy" <gautham.shenoy@....com>,
        LKML <linux-kernel@...r.kernel.org>, x86@...nel.org,
        lkp@...ts.01.org, lkp@...el.com, ying.huang@...el.com,
        feng.tang@...el.com, zhengjun.xing@...ux.intel.com,
        fengwei.yin@...el.com, aubrey.li@...ux.intel.com,
        yu.c.chen@...el.com
Subject: [sched/fair]  2cfb7a1b03:  fsmark.files_per_sec -26.2% regression



Greeting,

FYI, we noticed a -26.2% regression of fsmark.files_per_sec due to commit:


commit: 2cfb7a1b031b0e816af7a6ee0c6ab83b0acdf05a ("sched/fair: Improve consistency of allowed NUMA balance calculations")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core

in testcase: fsmark
on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	iterations: 1x
	nr_threads: 64t
	disk: 1BRD_48G
	fs: ext4
	filesize: 4M
	test_size: 24G
	sync_method: NoSync
	cpufreq_governor: performance
	ucode: 0x500320a

test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-9/performance/1BRD_48G/4M/ext4/1x/x86_64-rhel-8.3/64t/debian-10.4-x86_64-20200603.cgz/NoSync/lkp-csl-2ap4/24G/fsmark/0x500320a

commit: 
  889c5d60fb ("selftests/rseq: Change type of rseq_offset to ptrdiff_t")
  2cfb7a1b03 ("sched/fair: Improve consistency of allowed NUMA balance calculations")

889c5d60fbcf332c 2cfb7a1b031b0e816af7a6ee0c6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    122957 ±  7%     -41.3%      72222        fsmark.app_overhead
      2440 ±  2%     -26.2%       1801 ±  4%  fsmark.files_per_sec
    869.83           +25.5%       1091 ±  4%  fsmark.time.percent_of_cpu_this_job_got
    169.06 ±  2%     +29.9%     219.63 ±  4%  fsmark.time.system_time
      5.73 ±  4%     +12.9%       6.46 ±  5%  iostat.cpu.system
   1120075            -7.5%    1036423 ±  9%  vmstat.io.bo
    364503 ±  7%      +5.7%     385342        vmstat.system.in
     10457 ±  6%     +52.1%      15903 ±  9%  meminfo.Active
      6997 ±  9%     +81.0%      12666 ± 13%  meminfo.Active(anon)
     28931 ±  3%     +28.8%      37262 ±  4%  meminfo.Shmem
    844341 ± 13%    +342.6%    3737198 ± 74%  numa-numastat.node2.local_node
    911032 ± 12%    +317.3%    3801896 ± 73%  numa-numastat.node2.numa_hit
    379577 ±146%     -81.7%      69374 ± 27%  numa-numastat.node2.other_node
    849951 ± 20%    +237.0%    2864139 ± 80%  numa-numastat.node3.local_node
    927632 ± 18%    +215.0%    2922048 ± 79%  numa-numastat.node3.numa_hit
 2.262e+09            -3.7%  2.178e+09 ±  4%  perf-stat.i.branch-instructions
  50059502 ±  2%      +9.9%   55031386 ±  5%  perf-stat.i.cache-misses
    267.03            -2.2%     261.23 ±  2%  perf-stat.i.cpu-migrations
 1.206e+09            -4.6%   1.15e+09 ±  5%  perf-stat.i.dTLB-stores
 1.119e+10            -3.6%  1.079e+10 ±  4%  perf-stat.i.instructions
      9.90 ±  5%     -13.4%       8.58 ±  8%  perf-stat.i.major-faults
     68.64 ±  2%     +12.8       81.46        perf-stat.i.node-load-miss-rate%
  12916037 ±  3%     +15.1%   14866227 ±  6%  perf-stat.i.node-load-misses
   5586107 ±  6%     -40.0%    3349633 ±  8%  perf-stat.i.node-loads
      3.50 ±  3%     +14.4%       4.00 ±  3%  perf-stat.overall.cpi
      3328 ±  3%     -10.9%       2966 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.29 ±  3%     -12.5%       0.25 ±  3%  perf-stat.overall.ipc
     69.90 ±  2%     +11.8       81.72        perf-stat.overall.node-load-miss-rate%
  48090872 ±  2%     +11.2%   53476431 ±  4%  perf-stat.ps.cache-misses
 3.747e+10 ±  2%     +10.9%  4.157e+10 ±  5%  perf-stat.ps.cpu-cycles
  12291100 ±  3%     +15.9%   14246666 ±  5%  perf-stat.ps.node-load-misses
   5292116 ±  6%     -39.7%    3191213 ±  8%  perf-stat.ps.node-loads
      1733 ±  8%     +82.8%       3167 ± 13%  proc-vmstat.nr_active_anon
   2334357            -8.8%    2128372 ±  9%  proc-vmstat.nr_dirty
   5347564            -5.9%    5029652 ±  8%  proc-vmstat.nr_file_pages
     79430            +1.0%      80254        proc-vmstat.nr_inactive_anon
   4713828            -6.8%    4395202 ±  9%  proc-vmstat.nr_inactive_file
     31588            +1.6%      32107        proc-vmstat.nr_kernel_stack
     10415            +6.3%      11069        proc-vmstat.nr_mapped
      7220 ±  3%     +29.1%       9318 ±  4%  proc-vmstat.nr_shmem
     87414            +4.2%      91099 ±  2%  proc-vmstat.nr_slab_unreclaimable
      1733 ±  8%     +82.8%       3167 ± 13%  proc-vmstat.nr_zone_active_anon
     79430            +1.0%      80254        proc-vmstat.nr_zone_inactive_anon
   4713828            -6.8%    4395202 ±  9%  proc-vmstat.nr_zone_inactive_file
   2334942            -8.8%    2128919 ±  9%  proc-vmstat.nr_zone_write_pending
   1465471 ±  8%    -100.0%       0.00        proc-vmstat.numa_foreign
  12106397           +11.9%   13543781        proc-vmstat.numa_hit
  11937052           +11.3%   13287929        proc-vmstat.numa_local
   1459228 ±  8%    -100.0%       0.00        proc-vmstat.numa_miss
   1716007 ±  6%     -84.8%     260872        proc-vmstat.numa_other
      2675 ± 14%    +116.9%       5802 ±  9%  proc-vmstat.pgactivate
     50209 ± 31%     -63.8%      18155 ± 74%  numa-meminfo.node0.AnonHugePages
   8252135 ± 17%     -37.3%    5174756 ± 24%  numa-meminfo.node0.FilePages
   7185000 ±  5%     -31.1%    4948415 ± 19%  numa-meminfo.node1.FilePages
   7146622 ±  3%     -36.6%    4533737 ±  9%  numa-meminfo.node1.Inactive
   7091176 ±  4%     -36.9%    4476760 ±  9%  numa-meminfo.node1.Inactive(file)
    307.00 ± 41%    +333.4%       1330 ± 48%  numa-meminfo.node2.Active
    219.50 ± 12%    +280.4%     835.00 ± 71%  numa-meminfo.node2.Active(file)
      3261 ±198%   +1417.1%      49473 ±155%  numa-meminfo.node2.AnonPages
      6083 ±188%    +855.5%      58123 ±132%  numa-meminfo.node2.AnonPages.max
   1477008 ± 24%     +43.5%    2118995 ±  8%  numa-meminfo.node2.Dirty
      3557 ±178%   +1420.0%      54067 ±145%  numa-meminfo.node2.Inactive(anon)
     75574 ±  8%     +98.3%     149829 ±  6%  numa-meminfo.node2.KReclaimable
    551.67 ±  5%    +733.2%       4596 ± 45%  numa-meminfo.node2.Mapped
     85.33 ±128%    +852.9%     813.17 ± 22%  numa-meminfo.node2.PageTables
     75574 ±  8%     +98.3%     149829 ±  6%  numa-meminfo.node2.SReclaimable
     61158 ±  4%     +28.9%      78837 ±  5%  numa-meminfo.node2.SUnreclaim
    432.17 ± 66%   +1046.9%       4956 ± 80%  numa-meminfo.node2.Shmem
    136733 ±  5%     +67.2%     228667 ±  4%  numa-meminfo.node2.Slab
     41.67 ±164%  +1.6e+05%      66511 ±113%  numa-meminfo.node2.Unevictable
      5151 ± 13%    +127.5%      11719 ± 14%  numa-meminfo.node3.Active
      4938 ± 14%    +122.3%      10980 ± 15%  numa-meminfo.node3.Active(anon)
    211.67 ±  9%    +248.5%     737.67 ± 92%  numa-meminfo.node3.Active(file)
   1303369 ± 20%     +64.1%    2138250 ± 10%  numa-meminfo.node3.Dirty
   2739749 ± 37%    +103.7%    5579919 ± 16%  numa-meminfo.node3.FilePages
   2390623 ± 14%     +87.9%    4492696 ± 10%  numa-meminfo.node3.Inactive
   2310089 ± 16%     +91.5%    4424939 ± 10%  numa-meminfo.node3.Inactive(file)
     88743 ± 31%     +92.3%     170619 ±  8%  numa-meminfo.node3.KReclaimable
   3580239 ± 30%    +149.2%    8920534 ± 49%  numa-meminfo.node3.MemUsed
    567.83 ± 36%     +85.3%       1052 ± 35%  numa-meminfo.node3.PageTables
     88743 ± 31%     +92.3%     170619 ±  8%  numa-meminfo.node3.SReclaimable
    162480 ± 22%     +56.0%     253448 ±  4%  numa-meminfo.node3.Slab
   2080535 ± 18%     -37.8%    1293625 ± 23%  numa-vmstat.node0.nr_file_pages
     27818 ± 91%    +165.0%      73718 ± 28%  numa-vmstat.node0.numa_other
   1985397 ±  6%     -29.4%    1402170 ±  4%  numa-vmstat.node1.nr_dirtied
   1815794 ±  5%     -31.9%    1237101 ± 18%  numa-vmstat.node1.nr_file_pages
   1792343 ±  5%     -37.6%    1119190 ±  8%  numa-vmstat.node1.nr_inactive_file
   1282190 ±  9%     -32.4%     866377 ±  9%  numa-vmstat.node1.nr_written
   1792321 ±  5%     -37.6%    1119187 ±  8%  numa-vmstat.node1.nr_zone_inactive_file
     21.33 ±153%    +478.1%     123.33 ± 31%  numa-vmstat.node2.nr_active_anon
     55.17 ± 12%    +282.5%     211.00 ± 71%  numa-vmstat.node2.nr_active_file
    371312 ± 22%     +42.7%     529941 ±  7%  numa-vmstat.node2.nr_dirty
    920.17 ±180%   +1372.5%      13549 ±145%  numa-vmstat.node2.nr_inactive_anon
    140.67 ±  6%    +718.2%       1151 ± 45%  numa-vmstat.node2.nr_mapped
     21.33 ±131%    +843.8%     201.33 ± 22%  numa-vmstat.node2.nr_page_table_pages
    107.67 ± 67%   +1050.5%       1238 ± 80%  numa-vmstat.node2.nr_shmem
     19029 ±  7%     +96.8%      37452 ±  6%  numa-vmstat.node2.nr_slab_reclaimable
     15301 ±  4%     +28.9%      19720 ±  5%  numa-vmstat.node2.nr_slab_unreclaimable
     21.33 ±153%    +478.1%     123.33 ± 31%  numa-vmstat.node2.nr_zone_active_anon
     55.17 ± 12%    +282.5%     211.00 ± 71%  numa-vmstat.node2.nr_zone_active_file
    920.17 ±180%   +1372.5%      13549 ±145%  numa-vmstat.node2.nr_zone_inactive_anon
    371434 ± 22%     +42.7%     530074 ±  7%  numa-vmstat.node2.nr_zone_write_pending
   1090475 ±  9%    +179.5%    3048218 ± 49%  numa-vmstat.node2.numa_hit
    999961 ±  9%    +195.7%    2957320 ± 51%  numa-vmstat.node2.numa_local
    343707 ±130%     -72.9%      93030 ± 19%  numa-vmstat.node2.numa_other
      1205 ± 14%    +126.5%       2730 ± 14%  numa-vmstat.node3.nr_active_anon
     53.00 ±  9%    +245.3%     183.00 ± 93%  numa-vmstat.node3.nr_active_file
    672301 ± 15%    +104.4%    1374415 ±  4%  numa-vmstat.node3.nr_dirtied
    328259 ± 19%     +62.8%     534500 ±  9%  numa-vmstat.node3.nr_dirty
    689847 ± 36%    +102.3%    1395219 ± 16%  numa-vmstat.node3.nr_file_pages
    582451 ± 14%     +90.0%    1106487 ±  9%  numa-vmstat.node3.nr_inactive_file
    142.33 ± 37%     +84.1%     262.00 ± 35%  numa-vmstat.node3.nr_page_table_pages
     22316 ± 31%     +91.2%      42672 ±  7%  numa-vmstat.node3.nr_slab_reclaimable
    343968 ± 12%    +144.1%     839703 ± 10%  numa-vmstat.node3.nr_written
      1205 ± 14%    +126.5%       2730 ± 14%  numa-vmstat.node3.nr_zone_active_anon
     53.00 ±  9%    +245.3%     183.00 ± 93%  numa-vmstat.node3.nr_zone_active_file
    582443 ± 14%     +90.0%    1106485 ±  9%  numa-vmstat.node3.nr_zone_inactive_file
    328324 ± 19%     +62.9%     534716 ±  9%  numa-vmstat.node3.nr_zone_write_pending
   1341212 ± 22%    +110.8%    2827390 ± 37%  numa-vmstat.node3.numa_hit
   1242142 ± 24%    +121.1%    2746075 ± 38%  numa-vmstat.node3.numa_local
     12.09 ± 15%      -6.6        5.54 ±  7%  perf-profile.calltrace.cycles-pp.ret_from_fork
     12.09 ± 15%      -6.6        5.54 ±  7%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
     11.97 ± 15%      -6.5        5.48 ±  7%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
     11.96 ± 15%      -6.5        5.48 ±  7%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
     11.61 ± 15%      -6.2        5.41 ±  8%  perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
     11.61 ± 15%      -6.2        5.41 ±  8%  perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
     10.88 ± 15%      -6.0        4.92 ±  9%  perf-profile.calltrace.cycles-pp.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work.worker_thread
     10.87 ± 15%      -6.0        4.92 ±  9%  perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work
     10.87 ± 15%      -5.9        4.92 ±  9%  perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback
     10.87 ± 15%      -5.9        4.92 ±  9%  perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn
      5.95 ± 27%      -4.9        1.09 ± 11%  perf-profile.calltrace.cycles-pp.copy_page_from_iter_atomic.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write
      5.73 ± 28%      -4.7        1.06 ± 11%  perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter_atomic.generic_perform_write.ext4_buffered_write_iter.new_sync_write
      5.66 ± 27%      -4.6        1.05 ± 10%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter_atomic.generic_perform_write.ext4_buffered_write_iter
      8.20 ± 17%      -4.6        3.62 ±  9%  perf-profile.calltrace.cycles-pp.submit_bio_noacct.ext4_io_submit.ext4_writepages.do_writepages.__writeback_single_inode
      8.20 ± 17%      -4.6        3.62 ±  9%  perf-profile.calltrace.cycles-pp.__submit_bio.submit_bio_noacct.ext4_io_submit.ext4_writepages.do_writepages
      8.20 ± 17%      -4.6        3.62 ±  9%  perf-profile.calltrace.cycles-pp.ext4_io_submit.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
      8.19 ± 17%      -4.6        3.62 ±  9%  perf-profile.calltrace.cycles-pp.brd_submit_bio.__submit_bio.submit_bio_noacct.ext4_io_submit.ext4_writepages
      8.17 ± 17%      -4.6        3.61 ±  9%  perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_submit_bio.__submit_bio.submit_bio_noacct.ext4_io_submit
      8.07 ± 12%      -4.5        3.55 ± 13%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      5.53 ± 11%      -3.5        2.00 ± 32%  perf-profile.calltrace.cycles-pp.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write
      6.69 ± 13%      -3.5        3.17 ± 13%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      4.29 ± 15%      -2.4        1.93 ±  9%  perf-profile.calltrace.cycles-pp.brd_insert_page.brd_do_bvec.brd_submit_bio.__submit_bio.submit_bio_noacct
      4.31 ± 13%      -2.3        2.02 ± 16%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      4.23 ± 13%      -2.3        1.98 ± 16%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.20 ± 13%      -2.2        1.02 ± 10%  perf-profile.calltrace.cycles-pp.pagecache_get_page.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write
      3.19 ± 13%      -2.2        1.01 ± 11%  perf-profile.calltrace.cycles-pp.__filemap_get_folio.pagecache_get_page.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter
      3.08 ± 16%      -1.7        1.36 ± 11%  perf-profile.calltrace.cycles-pp._raw_spin_lock.brd_insert_page.brd_do_bvec.brd_submit_bio.__submit_bio
      2.84 ± 25%      -1.7        1.12 ± 22%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      2.50 ± 22%      -1.7        0.81 ± 11%  perf-profile.calltrace.cycles-pp.__block_commit_write.block_write_end.generic_write_end.generic_perform_write.ext4_buffered_write_iter
      2.50 ± 24%      -1.5        0.96 ± 23%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.22 ± 14%      -1.4        0.82 ± 11%  perf-profile.calltrace.cycles-pp.block_write_end.generic_write_end.generic_perform_write.ext4_buffered_write_iter.new_sync_write
      2.11 ± 13%      -1.4        0.74 ±  9%  perf-profile.calltrace.cycles-pp.filemap_add_folio.__filemap_get_folio.pagecache_get_page.ext4_da_write_begin.generic_perform_write
      1.78 ± 22%      -1.2        0.59 ± 49%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.96 ± 21%      -1.2        0.76 ± 21%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.69 ± 20%      -1.1        0.56 ± 48%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.98 ± 11%      -1.1        0.90 ± 54%  perf-profile.calltrace.cycles-pp.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write
      1.10 ± 15%      -0.8        0.27 ±100%  perf-profile.calltrace.cycles-pp.mpage_map_and_submit_buffers.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
      1.30 ± 10%      -0.8        0.48 ± 45%  perf-profile.calltrace.cycles-pp.__filemap_add_folio.filemap_add_folio.__filemap_get_folio.pagecache_get_page.ext4_da_write_begin
      1.08 ± 16%      -0.8        0.28 ±100%  perf-profile.calltrace.cycles-pp.ext4_put_io_end.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
      1.14 ± 27%      -0.7        0.40 ± 75%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.20 ± 23%      -0.6        0.59 ± 12%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.90 ± 14%      -0.5        1.38 ± 10%  perf-profile.calltrace.cycles-pp.stop_this_handle.jbd2_journal_stop.__ext4_journal_stop.__mark_inode_dirty.generic_write_end
      0.73 ± 16%      -0.4        0.36 ± 70%  perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work
      0.73 ± 16%      -0.4        0.36 ± 70%  perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn
      0.73 ± 16%      -0.4        0.36 ± 70%  perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
      0.73 ± 16%      -0.4        0.36 ± 70%  perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
      0.73 ± 16%      -0.4        0.36 ± 70%  perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread
      0.32 ±103%      +0.9        1.25 ± 11%  perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.generic_update_time.file_update_time
      0.22 ±143%      +0.9        1.17 ± 11%  perf-profile.calltrace.cycles-pp.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.generic_update_time
      0.46 ± 75%      +1.0        1.48 ± 12%  perf-profile.calltrace.cycles-pp.ext4_dirty_inode.__mark_inode_dirty.generic_update_time.file_update_time.ext4_write_checks
      0.66 ± 50%      +1.0        1.70 ± 13%  perf-profile.calltrace.cycles-pp.ext4_write_checks.ext4_buffered_write_iter.new_sync_write.vfs_write.ksys_write
      0.64 ± 50%      +1.0        1.69 ± 12%  perf-profile.calltrace.cycles-pp.file_update_time.ext4_write_checks.ext4_buffered_write_iter.new_sync_write.vfs_write
      0.62 ± 50%      +1.0        1.67 ± 11%  perf-profile.calltrace.cycles-pp.generic_update_time.file_update_time.ext4_write_checks.ext4_buffered_write_iter.new_sync_write
      0.62 ± 50%      +1.1        1.67 ± 11%  perf-profile.calltrace.cycles-pp.__mark_inode_dirty.generic_update_time.file_update_time.ext4_write_checks.ext4_buffered_write_iter
      2.42 ± 10%      +1.1        3.54 ± 12%  perf-profile.calltrace.cycles-pp.ext4_mark_iloc_dirty.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_write_end
      1.79 ±  6%      +1.2        2.98 ± 11%  perf-profile.calltrace.cycles-pp.ext4_reserve_inode_write.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_write_end
      0.90 ± 19%      +1.4        2.28 ±  8%  perf-profile.calltrace.cycles-pp.jbd2_journal_dirty_metadata.__ext4_handle_dirty_metadata.ext4_mark_iloc_dirty.__ext4_mark_inode_dirty.ext4_dirty_inode
      1.18 ± 10%      +1.4        2.55 ±  9%  perf-profile.calltrace.cycles-pp.__ext4_journal_get_write_access.ext4_reserve_inode_write.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty
      3.82 ± 12%      +1.4        5.20 ±  7%  perf-profile.calltrace.cycles-pp.jbd2_journal_stop.__ext4_journal_stop.__mark_inode_dirty.generic_write_end.generic_perform_write
      0.97 ± 16%      +1.4        2.37 ± 11%  perf-profile.calltrace.cycles-pp.ext4_journal_check_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.generic_write_end
      1.07 ± 10%      +1.4        2.47 ±  8%  perf-profile.calltrace.cycles-pp.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.__ext4_mark_inode_dirty.ext4_dirty_inode
      3.96 ± 12%      +1.4        5.40 ±  8%  perf-profile.calltrace.cycles-pp.__ext4_journal_stop.__mark_inode_dirty.generic_write_end.generic_perform_write.ext4_buffered_write_iter
      3.08 ± 13%      +1.6        4.64 ±  9%  perf-profile.calltrace.cycles-pp.add_transaction_credits.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode
      1.02 ± 18%      +1.7        2.68 ±  8%  perf-profile.calltrace.cycles-pp.__ext4_handle_dirty_metadata.ext4_mark_iloc_dirty.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty
      4.26 ±  8%      +2.3        6.56 ± 12%  perf-profile.calltrace.cycles-pp.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.generic_perform_write
      3.82 ± 15%      +2.4        6.22 ±  8%  perf-profile.calltrace.cycles-pp._raw_read_lock.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode
     43.99 ±  8%      +7.8       51.83 ±  9%  perf-profile.calltrace.cycles-pp.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write.ksys_write
     45.33 ±  9%      +8.4       53.74 ±  9%  perf-profile.calltrace.cycles-pp.write
     45.28 ±  9%      +8.4       53.72 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     45.27 ±  9%      +8.5       53.72 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     45.24 ±  9%      +8.5       53.71 ±  9%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     45.20 ±  9%      +8.5       53.69 ±  9%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     45.12 ±  9%      +8.5       53.64 ±  9%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     44.78 ±  9%      +8.8       53.56 ±  9%  perf-profile.calltrace.cycles-pp.ext4_buffered_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     20.02 ± 15%     +13.3       33.28 ±  9%  perf-profile.calltrace.cycles-pp.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.generic_write_end
     20.05 ± 16%     +14.2       34.28 ±  9%  perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty
     21.00 ± 15%     +14.7       35.68 ±  9%  perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.generic_perform_write
     32.04 ± 12%     +16.5       48.56 ±  8%  perf-profile.calltrace.cycles-pp.generic_write_end.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write
     25.28 ± 14%     +17.0       42.26 ±  9%  perf-profile.calltrace.cycles-pp.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.generic_perform_write.ext4_buffered_write_iter
     29.40 ± 13%     +18.3       47.71 ±  8%  perf-profile.calltrace.cycles-pp.__mark_inode_dirty.generic_write_end.generic_perform_write.ext4_buffered_write_iter.new_sync_write
     12.10 ± 15%      -6.6        5.54 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
     12.09 ± 15%      -6.6        5.54 ±  7%  perf-profile.children.cycles-pp.kthread
     11.97 ± 15%      -6.5        5.48 ±  7%  perf-profile.children.cycles-pp.worker_thread
     11.96 ± 15%      -6.5        5.48 ±  7%  perf-profile.children.cycles-pp.process_one_work
     11.61 ± 15%      -6.2        5.41 ±  8%  perf-profile.children.cycles-pp.writeback_sb_inodes
     11.61 ± 15%      -6.2        5.41 ±  8%  perf-profile.children.cycles-pp.wb_workfn
     11.61 ± 15%      -6.2        5.41 ±  8%  perf-profile.children.cycles-pp.wb_writeback
     11.61 ± 15%      -6.2        5.40 ±  8%  perf-profile.children.cycles-pp.__writeback_single_inode
     11.61 ± 15%      -6.2        5.40 ±  8%  perf-profile.children.cycles-pp.do_writepages
     11.61 ± 15%      -6.2        5.40 ±  8%  perf-profile.children.cycles-pp.ext4_writepages
      9.40 ± 24%      -5.4        3.98 ± 12%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      5.95 ± 27%      -4.9        1.09 ± 11%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      8.68 ± 15%      -4.7        3.98 ±  7%  perf-profile.children.cycles-pp.submit_bio_noacct
      8.68 ± 15%      -4.7        3.98 ±  7%  perf-profile.children.cycles-pp.__submit_bio
      8.68 ± 15%      -4.7        3.98 ±  7%  perf-profile.children.cycles-pp.brd_submit_bio
      8.65 ± 15%      -4.7        3.96 ±  8%  perf-profile.children.cycles-pp.brd_do_bvec
      5.73 ± 28%      -4.7        1.06 ± 11%  perf-profile.children.cycles-pp.copyin
      5.72 ± 28%      -4.7        1.06 ± 11%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      8.60 ± 15%      -4.6        3.95 ±  7%  perf-profile.children.cycles-pp.ext4_io_submit
      7.86 ± 12%      -4.3        3.52 ± 13%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      5.54 ± 12%      -3.5        2.00 ± 32%  perf-profile.children.cycles-pp.ext4_da_write_begin
      5.17 ± 14%      -2.8        2.33 ± 15%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      5.10 ± 14%      -2.8        2.30 ± 15%  perf-profile.children.cycles-pp.hrtimer_interrupt
      4.67 ± 16%      -2.5        2.14 ±  8%  perf-profile.children.cycles-pp.brd_insert_page
      3.89 ± 15%      -2.2        1.66 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock
      3.22 ± 13%      -2.2        1.02 ± 10%  perf-profile.children.cycles-pp.pagecache_get_page
      3.20 ± 13%      -2.2        1.02 ± 10%  perf-profile.children.cycles-pp.__filemap_get_folio
      3.13 ± 24%      -1.8        1.30 ± 20%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      2.50 ± 17%      -1.7        0.82 ± 12%  perf-profile.children.cycles-pp.block_write_end
      2.42 ± 16%      -1.6        0.81 ± 11%  perf-profile.children.cycles-pp.__block_commit_write
      2.53 ± 24%      -1.6        0.97 ± 23%  perf-profile.children.cycles-pp.menu_select
      2.11 ± 13%      -1.4        0.74 ±  9%  perf-profile.children.cycles-pp.filemap_add_folio
      2.19 ± 19%      -1.3        0.92 ± 18%  perf-profile.children.cycles-pp.tick_sched_timer
      1.91 ± 13%      -1.2        0.66 ±  9%  perf-profile.children.cycles-pp.__alloc_pages
      2.00 ± 20%      -1.2        0.80 ± 20%  perf-profile.children.cycles-pp.tick_sched_handle
      1.91 ± 18%      -1.1        0.77 ± 18%  perf-profile.children.cycles-pp.update_process_times
      1.69 ± 14%      -1.1        0.59 ± 10%  perf-profile.children.cycles-pp.get_page_from_freelist
      1.63 ± 21%      -1.1        0.54 ± 32%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.98 ± 11%      -1.1        0.90 ± 54%  perf-profile.children.cycles-pp.ext4_block_write_begin
      1.43 ± 22%      -1.0        0.44 ± 41%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.72 ± 27%      -0.9        0.79 ± 38%  perf-profile.children.cycles-pp.ktime_get
      1.45 ± 11%      -0.8        0.63 ± 13%  perf-profile.children.cycles-pp.irq_exit_rcu
      1.31 ± 10%      -0.7        0.56 ±  9%  perf-profile.children.cycles-pp.__filemap_add_folio
      0.94 ± 17%      -0.7        0.22 ± 16%  perf-profile.children.cycles-pp.folio_alloc
      1.31 ± 21%      -0.7        0.61 ± 28%  perf-profile.children.cycles-pp.clockevents_program_event
      1.26 ± 16%      -0.7        0.58 ±  8%  perf-profile.children.cycles-pp.mpage_map_and_submit_buffers
      1.13 ± 11%      -0.7        0.46 ± 15%  perf-profile.children.cycles-pp.mark_buffer_dirty
      1.00 ± 14%      -0.6        0.36 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.10 ± 11%      -0.6        0.48 ± 15%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.02 ± 18%      -0.6        0.40 ± 16%  perf-profile.children.cycles-pp.scheduler_tick
      0.78 ± 20%      -0.6        0.17 ± 11%  perf-profile.children.cycles-pp.folio_add_lru
      0.99 ± 10%      -0.6        0.39 ± 11%  perf-profile.children.cycles-pp.create_empty_buffers
      1.17 ± 15%      -0.6        0.59 ±  9%  perf-profile.children.cycles-pp.ext4_put_io_end
      1.15 ± 16%      -0.6        0.58 ±  9%  perf-profile.children.cycles-pp.ext4_release_io_end
      1.14 ± 16%      -0.6        0.58 ±  9%  perf-profile.children.cycles-pp.ext4_finish_bio
      0.78 ± 21%      -0.6        0.22 ± 65%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      1.03 ± 17%      -0.6        0.48 ±  8%  perf-profile.children.cycles-pp.mpage_submit_page
      0.68 ± 21%      -0.5        0.16 ± 12%  perf-profile.children.cycles-pp.__pagevec_lru_add
      1.94 ± 14%      -0.5        1.44 ± 10%  perf-profile.children.cycles-pp.stop_this_handle
      0.65 ± 13%      -0.4        0.21 ± 13%  perf-profile.children.cycles-pp.rmqueue
      0.80 ±  8%      -0.4        0.37 ± 10%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.76 ± 18%      -0.4        0.34 ± 10%  perf-profile.children.cycles-pp.ext4_bio_write_page
      0.81 ± 10%      -0.4        0.40 ±  9%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.74 ± 15%      -0.4        0.34 ± 18%  perf-profile.children.cycles-pp.__folio_mark_dirty
      0.70 ± 11%      -0.4        0.32 ± 11%  perf-profile.children.cycles-pp.alloc_page_buffers
      0.50 ± 24%      -0.4        0.14 ± 45%  perf-profile.children.cycles-pp.vprintk_emit
      0.58 ± 20%      -0.4        0.22 ± 52%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.66 ± 10%      -0.4        0.32 ± 10%  perf-profile.children.cycles-pp.alloc_buffer_head
      0.65 ± 10%      -0.3        0.30 ± 18%  perf-profile.children.cycles-pp.rebalance_domains
      0.66 ± 15%      -0.3        0.32 ±  9%  perf-profile.children.cycles-pp.folio_end_writeback
      0.48 ± 21%      -0.3        0.14 ± 45%  perf-profile.children.cycles-pp.console_unlock
      0.64 ± 22%      -0.3        0.30 ± 27%  perf-profile.children.cycles-pp.prep_new_page
      0.61 ± 24%      -0.3        0.29 ± 28%  perf-profile.children.cycles-pp.kernel_init_free_pages
      0.60 ± 24%      -0.3        0.28 ± 28%  perf-profile.children.cycles-pp.clear_page_erms
      0.64 ± 15%      -0.3        0.34 ± 30%  perf-profile.children.cycles-pp.ext4_fill_raw_inode
      0.58 ± 15%      -0.3        0.28 ± 10%  perf-profile.children.cycles-pp.__folio_end_writeback
      0.38 ± 21%      -0.3        0.08 ± 78%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.54 ± 19%      -0.3        0.24 ± 18%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.42 ± 23%      -0.3        0.13 ± 45%  perf-profile.children.cycles-pp.serial8250_console_write
      0.60 ± 10%      -0.3        0.33 ±  9%  perf-profile.children.cycles-pp.__mem_cgroup_charge
      0.52 ± 15%      -0.3        0.24 ± 12%  perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
      0.39 ± 24%      -0.3        0.12 ± 42%  perf-profile.children.cycles-pp.wait_for_lsr
      0.45 ± 28%      -0.3        0.19 ± 11%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.73 ± 16%      -0.2        0.49 ± 17%  perf-profile.children.cycles-pp.__writeback_inodes_wb
      0.37 ± 12%      -0.2        0.13 ± 13%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.26 ± 26%      -0.2        0.02 ± 99%  perf-profile.children.cycles-pp.node_dirty_ok
      0.45 ± 11%      -0.2        0.23 ± 17%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.42 ± 18%      -0.2        0.20 ± 12%  perf-profile.children.cycles-pp.__folio_start_writeback
      0.30 ± 14%      -0.2        0.08 ± 33%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.30 ± 13%      -0.2        0.08 ± 33%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.47 ± 11%      -0.2        0.25 ± 16%  perf-profile.children.cycles-pp.__getblk_gfp
      0.42 ±  6%      -0.2        0.21 ± 11%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.31 ± 23%      -0.2        0.10 ± 47%  perf-profile.children.cycles-pp.io_serial_in
      0.43 ± 11%      -0.2        0.23 ± 18%  perf-profile.children.cycles-pp.__find_get_block
      0.39 ± 15%      -0.2        0.20 ± 50%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.34 ± 17%      -0.2        0.15 ± 16%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.43 ±  9%      -0.2        0.24 ± 11%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.37 ± 16%      -0.2        0.19 ± 51%  perf-profile.children.cycles-pp.tick_irq_enter
      0.30 ± 16%      -0.2        0.13 ± 20%  perf-profile.children.cycles-pp.load_balance
      0.32 ± 23%      -0.2        0.16 ± 12%  perf-profile.children.cycles-pp.ext4_inode_csum_set
      0.24 ± 10%      -0.2        0.07 ±  9%  perf-profile.children.cycles-pp.jbd2_write_access_granted
      0.33 ± 29%      -0.2        0.17 ± 34%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.29 ± 16%      -0.2        0.13 ± 17%  perf-profile.children.cycles-pp.native_sched_clock
      0.26 ± 14%      -0.2        0.11 ± 10%  perf-profile.children.cycles-pp.xas_load
      0.20 ± 18%      -0.2        0.04 ± 72%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.18 ± 20%      -0.2        0.02 ± 99%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.29 ± 23%      -0.1        0.14 ±  8%  perf-profile.children.cycles-pp.ext4_inode_csum
      0.22 ±  8%      -0.1        0.08 ± 11%  perf-profile.children.cycles-pp.__might_resched
      0.29 ± 10%      -0.1        0.14 ±  9%  perf-profile.children.cycles-pp.read_tsc
      0.32 ± 13%      -0.1        0.17 ± 12%  perf-profile.children.cycles-pp.charge_memcg
      0.24 ± 17%      -0.1        0.10 ± 21%  perf-profile.children.cycles-pp.update_rq_clock
      0.20 ± 10%      -0.1        0.06 ± 26%  perf-profile.children.cycles-pp.ext4_es_insert_delayed_block
      0.22 ± 19%      -0.1        0.08 ± 24%  perf-profile.children.cycles-pp.update_blocked_averages
      0.26 ± 15%      -0.1        0.12 ±  7%  perf-profile.children.cycles-pp.folio_clear_dirty_for_io
      0.21 ± 33%      -0.1        0.08 ± 51%  perf-profile.children.cycles-pp.irq_work_run_list
      0.22 ± 15%      -0.1        0.09 ± 21%  perf-profile.children.cycles-pp.find_busiest_group
      0.20 ± 32%      -0.1        0.07 ± 50%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.20 ± 32%      -0.1        0.07 ± 50%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.20 ± 32%      -0.1        0.07 ± 50%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.20 ± 32%      -0.1        0.07 ± 50%  perf-profile.children.cycles-pp.irq_work_run
      0.20 ± 32%      -0.1        0.07 ± 50%  perf-profile.children.cycles-pp.irq_work_single
      0.20 ± 32%      -0.1        0.07 ± 50%  perf-profile.children.cycles-pp._printk
      0.22 ± 20%      -0.1        0.10 ±  8%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.23 ± 29%      -0.1        0.10 ± 32%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.21 ± 15%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.24 ±  9%      -0.1        0.11 ±  6%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.26 ± 14%      -0.1        0.14 ± 10%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.20 ± 21%      -0.1        0.08 ±  7%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.20 ± 38%      -0.1        0.09 ± 65%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.19 ± 23%      -0.1        0.08 ± 12%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.20 ± 14%      -0.1        0.09 ± 13%  perf-profile.children.cycles-pp.__radix_tree_preload
      0.19 ± 26%      -0.1        0.09 ± 36%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.18 ± 14%      -0.1        0.09 ± 22%  perf-profile.children.cycles-pp.___slab_alloc
      0.18 ± 14%      -0.1        0.09 ± 21%  perf-profile.children.cycles-pp.__slab_alloc
      0.19 ± 19%      -0.1        0.10 ± 14%  perf-profile.children.cycles-pp.mpage_process_page_bufs
      0.16 ± 14%      -0.1        0.07 ± 14%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.16 ± 14%      -0.1        0.07 ± 14%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.13 ± 17%      -0.1        0.05 ± 47%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.15 ± 20%      -0.1        0.07 ± 25%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.14 ± 23%      -0.1        0.05 ± 51%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.15 ± 18%      -0.1        0.06 ± 17%  perf-profile.children.cycles-pp.brd_lookup_page
      0.15 ± 21%      -0.1        0.07 ± 31%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.15 ± 14%      -0.1        0.07 ± 14%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.15 ± 35%      -0.1        0.07 ± 41%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.18 ± 21%      -0.1        0.10 ±  6%  perf-profile.children.cycles-pp.crc32c_pcl_intel_update
      0.14 ± 11%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.release_pages
      0.15 ± 27%      -0.1        0.07 ± 24%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.16 ± 14%      -0.1        0.08 ± 22%  perf-profile.children.cycles-pp.allocate_slab
      0.11 ± 26%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.11 ± 12%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.fault_in_iov_iter_readable
      0.11 ± 15%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.__cond_resched
      0.13 ± 17%      -0.1        0.06 ± 14%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.09 ± 14%      -0.1        0.02 ± 99%  perf-profile.children.cycles-pp.__pagevec_release
      0.11 ± 26%      -0.1        0.05 ± 47%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.12 ± 22%      -0.1        0.06 ± 13%  perf-profile.children.cycles-pp.__xa_clear_mark
      0.14 ± 20%      -0.1        0.08 ± 21%  perf-profile.children.cycles-pp.try_charge_memcg
      0.12 ± 16%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__count_memcg_events
      0.16 ±  7%      -0.1        0.11 ± 12%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.10 ± 11%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.folio_memcg_lock
      0.09 ± 20%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.16 ±  7%      -0.0        0.12 ± 12%  perf-profile.children.cycles-pp.do_sys_open
      0.16 ±  7%      -0.0        0.12 ± 12%  perf-profile.children.cycles-pp.do_sys_openat2
      0.15 ±  5%      -0.0        0.11 ± 14%  perf-profile.children.cycles-pp.do_filp_open
      0.15 ±  6%      -0.0        0.11 ± 14%  perf-profile.children.cycles-pp.path_openat
      0.10 ± 20%      -0.0        0.06 ± 17%  perf-profile.children.cycles-pp.mem_cgroup_track_foreign_dirty_slowpath
      0.72 ± 28%      +1.0        1.70 ± 13%  perf-profile.children.cycles-pp.ext4_write_checks
      0.70 ± 29%      +1.0        1.69 ± 12%  perf-profile.children.cycles-pp.file_update_time
      0.68 ± 29%      +1.0        1.67 ± 11%  perf-profile.children.cycles-pp.generic_update_time
      2.49 ± 10%      +1.2        3.67 ± 12%  perf-profile.children.cycles-pp.ext4_mark_iloc_dirty
      1.85 ±  6%      +1.3        3.10 ± 11%  perf-profile.children.cycles-pp.ext4_reserve_inode_write
      0.92 ± 19%      +1.4        2.37 ±  8%  perf-profile.children.cycles-pp.jbd2_journal_dirty_metadata
      1.22 ± 10%      +1.4        2.66 ±  9%  perf-profile.children.cycles-pp.__ext4_journal_get_write_access
      1.00 ± 16%      +1.5        2.46 ± 11%  perf-profile.children.cycles-pp.ext4_journal_check_start
      1.11 ± 10%      +1.5        2.59 ±  8%  perf-profile.children.cycles-pp.jbd2_journal_get_write_access
      3.91 ± 13%      +1.5        5.40 ±  7%  perf-profile.children.cycles-pp.jbd2_journal_stop
      4.06 ± 13%      +1.5        5.60 ±  8%  perf-profile.children.cycles-pp.__ext4_journal_stop
      3.16 ± 13%      +1.6        4.80 ±  9%  perf-profile.children.cycles-pp.add_transaction_credits
      1.06 ± 18%      +1.7        2.78 ±  8%  perf-profile.children.cycles-pp.__ext4_handle_dirty_metadata
      4.39 ±  8%      +2.4        6.82 ± 12%  perf-profile.children.cycles-pp.__ext4_mark_inode_dirty
      3.99 ± 15%      +2.5        6.47 ±  8%  perf-profile.children.cycles-pp._raw_read_lock
     44.00 ±  8%      +7.8       51.84 ±  9%  perf-profile.children.cycles-pp.generic_perform_write
     45.75 ±  9%      +8.2       53.97 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     45.73 ±  9%      +8.2       53.96 ±  9%  perf-profile.children.cycles-pp.do_syscall_64
     45.34 ±  9%      +8.4       53.76 ±  9%  perf-profile.children.cycles-pp.write
     45.24 ±  9%      +8.5       53.72 ±  9%  perf-profile.children.cycles-pp.ksys_write
     45.21 ±  9%      +8.5       53.70 ±  9%  perf-profile.children.cycles-pp.vfs_write
     45.13 ±  9%      +8.5       53.66 ±  9%  perf-profile.children.cycles-pp.new_sync_write
     44.78 ±  9%      +8.8       53.56 ±  9%  perf-profile.children.cycles-pp.ext4_buffered_write_iter
     20.52 ± 15%     +14.0       34.49 ±  9%  perf-profile.children.cycles-pp.jbd2__journal_start
     20.34 ± 16%     +14.0       34.34 ±  9%  perf-profile.children.cycles-pp.start_this_handle
     21.53 ± 15%     +15.4       36.97 ±  9%  perf-profile.children.cycles-pp.__ext4_journal_start_sb
     32.08 ± 12%     +16.5       48.56 ±  8%  perf-profile.children.cycles-pp.generic_write_end
     25.87 ± 14%     +17.9       43.73 ±  9%  perf-profile.children.cycles-pp.ext4_dirty_inode
     30.18 ± 13%     +19.2       49.41 ±  8%  perf-profile.children.cycles-pp.__mark_inode_dirty
      5.68 ± 28%      -4.6        1.06 ± 11%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      3.78 ± 15%      -2.1        1.64 ± 10%  perf-profile.self.cycles-pp._raw_spin_lock
      3.81 ± 15%      -2.0        1.76 ±  7%  perf-profile.self.cycles-pp.brd_do_bvec
      1.49 ± 31%      -0.8        0.68 ± 45%  perf-profile.self.cycles-pp.ktime_get
      0.90 ± 21%      -0.6        0.32 ± 10%  perf-profile.self.cycles-pp.__block_commit_write
      0.78 ± 20%      -0.6        0.22 ± 66%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.99 ± 14%      -0.5        0.50 ± 17%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.92 ± 14%      -0.5        1.43 ± 10%  perf-profile.self.cycles-pp.stop_this_handle
      0.77 ± 12%      -0.4        0.35 ± 13%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.44 ± 27%      -0.3        0.12 ± 48%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.59 ± 23%      -0.3        0.28 ± 28%  perf-profile.self.cycles-pp.clear_page_erms
      0.38 ± 21%      -0.3        0.08 ± 78%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.42 ± 29%      -0.3        0.14 ± 50%  perf-profile.self.cycles-pp.update_process_times
      0.45 ± 11%      -0.2        0.23 ± 17%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.29 ± 13%      -0.2        0.07 ± 33%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.40 ± 26%      -0.2        0.18 ± 27%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.42 ±  6%      -0.2        0.21 ± 11%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.31 ± 23%      -0.2        0.10 ± 47%  perf-profile.self.cycles-pp.io_serial_in
      0.23 ± 18%      -0.2        0.04 ± 72%  perf-profile.self.cycles-pp.__pagevec_lru_add
      0.41 ± 12%      -0.2        0.22 ± 19%  perf-profile.self.cycles-pp.__find_get_block
      0.28 ± 13%      -0.2        0.10 ± 12%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.23 ± 11%      -0.2        0.07 ±  9%  perf-profile.self.cycles-pp.jbd2_write_access_granted
      0.33 ± 30%      -0.2        0.17 ± 34%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.38 ±  9%      -0.2        0.22 ±  9%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.28 ± 16%      -0.2        0.13 ± 15%  perf-profile.self.cycles-pp.native_sched_clock
      0.20 ± 18%      -0.2        0.05 ± 47%  perf-profile.self.cycles-pp.rmqueue
      0.19 ± 16%      -0.2        0.04 ± 73%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.19 ± 13%      -0.1        0.05 ± 46%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.22 ±  9%      -0.1        0.08 ±  9%  perf-profile.self.cycles-pp.__might_resched
      0.28 ± 11%      -0.1        0.14 ±  9%  perf-profile.self.cycles-pp.read_tsc
      0.26 ± 12%      -0.1        0.12 ±  9%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.21 ± 13%      -0.1        0.09 ± 10%  perf-profile.self.cycles-pp.xas_load
      0.20 ± 23%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.19 ± 23%      -0.1        0.08 ± 12%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.14 ±  6%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__alloc_pages
      0.20 ± 11%      -0.1        0.09 ±  7%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.20 ± 15%      -0.1        0.09 ± 14%  perf-profile.self.cycles-pp.__radix_tree_preload
      0.19 ± 11%      -0.1        0.09 ± 11%  perf-profile.self.cycles-pp.ext4_bio_write_page
      0.18 ± 37%      -0.1        0.08 ± 61%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.17 ± 12%      -0.1        0.07 ± 14%  perf-profile.self.cycles-pp.__folio_end_writeback
      0.23 ± 13%      -0.1        0.14 ± 11%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.18 ± 20%      -0.1        0.10 ± 14%  perf-profile.self.cycles-pp.mpage_process_page_bufs
      0.17 ± 31%      -0.1        0.08 ± 38%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.16 ± 18%      -0.1        0.08 ± 17%  perf-profile.self.cycles-pp.do_idle
      0.14 ± 23%      -0.1        0.05 ± 51%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.15 ± 19%      -0.1        0.07 ± 16%  perf-profile.self.cycles-pp.__folio_start_writeback
      0.15 ± 14%      -0.1        0.07 ± 14%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.13 ± 18%      -0.1        0.05 ± 47%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.14 ± 12%      -0.1        0.06 ± 25%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.13 ±  9%      -0.1        0.06 ±  9%  perf-profile.self.cycles-pp.release_pages
      0.12 ± 12%      -0.1        0.05 ± 46%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.10 ± 22%      -0.1        0.03 ±103%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.13 ± 15%      -0.1        0.06 ±  8%  perf-profile.self.cycles-pp.folio_clear_dirty_for_io
      0.10 ± 23%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.mpage_map_and_submit_buffers
      0.14 ± 15%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.ext4_finish_bio
      0.16 ± 16%      -0.1        0.10 ±  6%  perf-profile.self.cycles-pp.crc32c_pcl_intel_update
      0.11 ± 18%      -0.1        0.05 ± 51%  perf-profile.self.cycles-pp.allocate_slab
      0.10 ± 13%      -0.1        0.04 ± 45%  perf-profile.self.cycles-pp.folio_memcg_lock
      0.11 ± 11%      -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.__count_memcg_events
      0.16 ±  5%      -0.0        0.11 ± 10%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.10 ± 20%      -0.0        0.06 ± 17%  perf-profile.self.cycles-pp.mem_cgroup_track_foreign_dirty_slowpath
      0.08 ± 14%      +0.0        0.12 ± 19%  perf-profile.self.cycles-pp.jbd2__journal_start
      0.10 ± 22%      +0.3        0.40 ±  9%  perf-profile.self.cycles-pp.__ext4_handle_dirty_metadata
      0.92 ± 19%      +1.4        2.35 ±  8%  perf-profile.self.cycles-pp.jbd2_journal_dirty_metadata
      0.95 ± 17%      +1.5        2.43 ± 12%  perf-profile.self.cycles-pp.ext4_journal_check_start
      0.87 ± 13%      +1.6        2.50 ±  8%  perf-profile.self.cycles-pp.jbd2_journal_get_write_access
      3.14 ± 13%      +1.6        4.78 ±  9%  perf-profile.self.cycles-pp.add_transaction_credits
      1.93 ± 13%      +2.0        3.93 ±  7%  perf-profile.self.cycles-pp.jbd2_journal_stop
      3.94 ± 15%      +2.5        6.44 ±  8%  perf-profile.self.cycles-pp._raw_read_lock
     13.14 ± 17%      +9.8       22.93 ±  9%  perf-profile.self.cycles-pp.start_this_handle




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.17.0-rc1-00026-g2cfb7a1b031b" of type "text/plain" (162056 bytes)

View attachment "job-script" of type "text/plain" (8158 bytes)

View attachment "job.yaml" of type "text/plain" (5585 bytes)

View attachment "reproduce" of type "text/plain" (1437 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ