lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Fri, 23 Sep 2016 09:22:46 +0800
From:   kernel test robot <xiaolong.ye@...el.com>
To:     Jaegeuk Kim <jaegeuk@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [lkp] [f2fs]  b873b798af:  fsmark.files_per_sec 73.0% improvement


FYI, we noticed a 73.0% improvement of fsmark.files_per_sec due to commit:

commit b873b798af6386f3c7ca1636d4989e9b8f9d1794 ("Revert "f2fs: use percpu_rw_semaphore"")
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

in testcase: fsmark
on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 128G memory
with following parameters:

	iterations: 8
	disk: 1SSD
	nr_threads: 4
	fs: f2fs
	filesize: 9B
	test_size: 40G
	sync_method: fsyncBeforeClose
	nr_directories: 16d
	nr_files_per_directory: 256fpd
	cpufreq_governor: performance

The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
  gcc-6/performance/1SSD/9B/f2fs/8/x86_64-rhel-7.2/16d/256fpd/4/debian-x86_64-2016-08-31.cgz/fsyncBeforeClose/lkp-hsw-ep4/40G/fsmark

commit: 
  3408fef744 ("Merge branch 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")
  b873b798af ("Revert "f2fs: use percpu_rw_semaphore"")

3408fef7448ce7d3 b873b798af6386f3c7ca1636d4 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
         %stddev     %change         %stddev
             \          |                \  
     22427 ±  2%     +73.0%      38798 ±  1%  fsmark.files_per_sec
    469.31 ±  2%     -41.7%     273.52 ±  1%  fsmark.time.elapsed_time
    469.31 ±  2%     -41.7%     273.52 ±  1%  fsmark.time.elapsed_time.max
     2e+08 ±  0%     +16.5%   2.33e+08 ±  0%  fsmark.time.file_system_outputs
    324.25 ±  0%      +4.3%     338.25 ±  0%  fsmark.time.percent_of_cpu_this_job_got
      1472 ±  1%     -40.2%     879.79 ±  1%  fsmark.time.system_time
     52.26 ±  0%     -11.3%      46.36 ±  3%  fsmark.time.user_time
  37520477 ±  2%     -64.8%   13224338 ±  0%  fsmark.time.voluntary_context_switches
    345437 ±  2%     -39.4%     209409 ±  1%  interrupts.CAL:Function_call_interrupts
   4201472 ± 10%     +19.1%    5002240 ±  0%  meminfo.DirectMap2M
     93747 ±  1%     -78.2%      20431 ±  3%  meminfo.Dirty
    538708 ± 13%     +17.3%     631889 ±  3%  numa-meminfo.node0.SUnreclaim
     48936 ± 66%     -85.8%       6931 ± 47%  numa-meminfo.node1.Dirty
    301268 ±  2%     -42.0%     174711 ±  5%  softirqs.RCU
    726268 ±  0%     -31.2%     499340 ±  1%  softirqs.SCHED
   1247621 ±  0%     -25.8%     925631 ±  0%  softirqs.TIMER
      8.64 ±  1%      +2.3%       8.83 ±  1%  turbostat.%Busy
    244.00 ±  1%      +3.4%     252.25 ±  1%  turbostat.Avg_MHz
      0.80 ±  3%     +29.1%       1.03 ±  5%  turbostat.CPU%c3
      0.01 ±  0%    -100.0%       0.00 ± -1%  turbostat.Pkg%pc3
    194.75 ±  2%     +71.4%     333.75 ±  1%  vmstat.io.bi
    122798 ±  2%    +120.2%     270434 ±  1%  vmstat.io.bo
    162637 ±  0%     -39.0%      99205 ±  1%  vmstat.system.cs
     94048 ±  0%     +20.1%     112911 ±  0%  vmstat.system.in
   7078528 ± 24%     +45.5%   10295956 ±  1%  numa-vmstat.node0.nr_dirtied
    134688 ± 13%     +17.3%     157981 ±  3%  numa-vmstat.node0.nr_slab_unreclaimable
   3932913 ± 27%     +64.7%    6477432 ±  2%  numa-vmstat.node0.nr_written
     12243 ± 66%     -86.0%       1708 ± 45%  numa-vmstat.node1.nr_dirty
     12372 ± 66%     -85.7%       1772 ± 43%  numa-vmstat.node1.nr_zone_write_pending
    356019 ± 45%     +54.0%     548372 ± 21%  numa-vmstat.node1.numa_miss
  24986528 ±  0%     +16.4%   29089616 ±  0%  proc-vmstat.nr_dirtied
     23466 ±  1%     -78.2%       5104 ±  3%  proc-vmstat.nr_dirty
  14460939 ±  0%     +28.7%   18611323 ±  0%  proc-vmstat.nr_written
     23740 ±  1%     -77.8%       5265 ±  3%  proc-vmstat.nr_zone_write_pending
      2852 ±  4%     -21.8%       2230 ±  6%  proc-vmstat.numa_hint_faults
      2325 ±  9%     -24.6%       1752 ±  5%  proc-vmstat.numa_hint_faults_local
    943958 ±  3%     -37.8%     587185 ±  1%  proc-vmstat.pgfault
   1651946 ±  1%     -18.4%    1347438 ±  0%  proc-vmstat.pgfree
  57844058 ±  0%     +28.7%   74445459 ±  0%  proc-vmstat.pgpgout
 4.069e+08 ±  5%     -73.0%  1.098e+08 ±  2%  cpuidle.C1-HSW.time
  37792253 ±  3%     -65.8%   12910843 ±  0%  cpuidle.C1-HSW.usage
 2.018e+08 ±  9%     -90.7%   18847626 ±  4%  cpuidle.C1E-HSW.time
   1985790 ±  9%     -94.3%     112882 ±  1%  cpuidle.C1E-HSW.usage
 4.711e+08 ±  3%     -50.7%  2.324e+08 ±  8%  cpuidle.C3-HSW.time
   2312993 ±  5%     -74.6%     587232 ±  9%  cpuidle.C3-HSW.usage
 2.995e+10 ±  2%     -40.9%  1.771e+10 ±  1%  cpuidle.C6-HSW.time
  31182059 ±  2%     -40.6%   18519022 ±  1%  cpuidle.C6-HSW.usage
 5.744e+08 ±  6%     -50.5%  2.844e+08 ±  9%  cpuidle.POLL.time
    386829 ±  2%     -30.6%     268320 ±  0%  cpuidle.POLL.usage
 1.189e+12 ±  4%     -38.3%  7.335e+11 ± 13%  perf-stat.branch-instructions
 8.976e+09 ±  3%     -38.8%  5.491e+09 ± 13%  perf-stat.branch-misses
      2.67 ±  1%     +20.1%       3.20 ±  6%  perf-stat.cache-miss-rate%
 3.896e+09 ±  5%     -16.4%  3.257e+09 ±  9%  perf-stat.cache-misses
 1.461e+11 ±  5%     -29.7%  1.028e+11 ± 16%  perf-stat.cache-references
  76682921 ±  2%     -64.3%   27344845 ±  0%  perf-stat.context-switches
 1.008e+13 ±  2%     -37.7%  6.275e+12 ± 13%  perf-stat.cpu-cycles
    340726 ±  4%     -50.3%     169411 ±  2%  perf-stat.cpu-migrations
      2.17 ±  2%      +4.2%       2.26 ±  0%  perf-stat.dTLB-load-miss-rate%
 1.902e+10 ±  2%     -28.3%  1.363e+10 ±  5%  perf-stat.dTLB-load-misses
 8.576e+11 ±  2%     -31.3%  5.888e+11 ±  5%  perf-stat.dTLB-loads
      0.19 ±  4%     -51.3%       0.09 ±  2%  perf-stat.dTLB-store-miss-rate%
 9.342e+08 ±  4%     -66.5%  3.132e+08 ±  9%  perf-stat.dTLB-store-misses
  4.95e+11 ±  5%     -31.1%  3.409e+11 ± 10%  perf-stat.dTLB-stores
 1.662e+09 ±  1%     -34.0%  1.097e+09 ±  3%  perf-stat.iTLB-loads
 5.754e+12 ±  4%     -36.1%  3.679e+12 ± 12%  perf-stat.instructions
      0.57 ±  1%      +2.8%       0.59 ±  0%  perf-stat.ipc
    911944 ±  3%     -38.1%     564696 ±  2%  perf-stat.minor-faults
 1.254e+09 ±  1%     -24.3%  9.495e+08 ± 14%  perf-stat.node-load-misses
 1.039e+09 ±  4%     -15.5%  8.778e+08 ± 10%  perf-stat.node-loads
  3.43e+08 ±  7%     -15.3%  2.905e+08 ±  3%  perf-stat.node-stores
    911944 ±  3%     -38.1%     564694 ±  2%  perf-stat.page-faults
      5256 ±  0%    -100.0%       0.00 ± -1%  slabinfo.Acpi-Parse.active_objs
      5256 ±  0%    -100.0%       0.00 ± -1%  slabinfo.Acpi-Parse.num_objs
     11523 ±  2%      -8.6%      10532 ±  4%  slabinfo.anon_vma.active_objs
     11523 ±  2%      -8.6%      10532 ±  4%  slabinfo.anon_vma.num_objs
    703.50 ±  6%    -100.0%       0.00 ± -1%  slabinfo.blkdev_ioc.active_objs
    703.50 ±  6%    -100.0%       0.00 ± -1%  slabinfo.blkdev_ioc.num_objs
     24860 ±  0%     -74.8%       6261 ±  8%  slabinfo.cred_jar.active_objs
    591.25 ±  0%     -74.9%     148.25 ±  8%  slabinfo.cred_jar.active_slabs
     24860 ±  0%     -74.8%       6261 ±  8%  slabinfo.cred_jar.num_objs
    591.25 ±  0%     -74.9%     148.25 ±  8%  slabinfo.cred_jar.num_slabs
      3378 ±  1%     +25.2%       4228 ±  0%  slabinfo.ftrace_event_field.active_objs
      3378 ±  1%     +25.2%       4228 ±  0%  slabinfo.ftrace_event_field.num_objs
     59330 ±  0%     +14.8%      68136 ±  0%  slabinfo.kernfs_node_cache.active_objs
    872.50 ±  0%     +14.8%       1002 ±  0%  slabinfo.kernfs_node_cache.active_slabs
     59330 ±  0%     +14.8%      68136 ±  0%  slabinfo.kernfs_node_cache.num_objs
    872.50 ±  0%     +14.8%       1002 ±  0%  slabinfo.kernfs_node_cache.num_slabs
      6676 ±  1%     +16.2%       7754 ±  3%  slabinfo.kmalloc-1024.active_objs
      7034 ±  1%     +13.8%       8005 ±  3%  slabinfo.kmalloc-1024.num_objs
      7888 ±  2%     -16.9%       6554 ±  1%  slabinfo.kmalloc-128.num_objs
    140664 ±  1%    -100.0%       0.00 ± -1%  slabinfo.nat_entry.active_objs
      1105 ±  1%    -100.0%       0.00 ± -1%  slabinfo.nat_entry.active_slabs
    141508 ±  1%    -100.0%       0.00 ± -1%  slabinfo.nat_entry.num_objs
      1105 ±  1%    -100.0%       0.00 ± -1%  slabinfo.nat_entry.num_slabs
      6617 ±  1%    -100.0%       0.00 ± -1%  slabinfo.nat_entry_set.active_objs
      6617 ±  1%    -100.0%       0.00 ± -1%  slabinfo.nat_entry_set.num_objs
      2058 ±  1%     +21.8%       2507 ±  3%  slabinfo.trace_event_file.active_objs
      2058 ±  1%     +21.8%       2507 ±  3%  slabinfo.trace_event_file.num_objs
     10569 ±  6%     -43.8%       5944 ±  0%  sched_debug.cfs_rq:/.exec_clock.avg
     41631 ± 12%     -34.6%      27235 ±  1%  sched_debug.cfs_rq:/.exec_clock.max
     31.35 ±  3%     -38.3%      19.36 ±  7%  sched_debug.cfs_rq:/.exec_clock.min
     11878 ± 11%     -38.1%       7355 ±  3%  sched_debug.cfs_rq:/.exec_clock.stddev
    176.51 ±  7%     +64.9%     291.01 ±  5%  sched_debug.cfs_rq:/.load_avg.avg
    423.29 ± 14%     +64.1%     694.45 ± 27%  sched_debug.cfs_rq:/.load_avg.max
    154.03 ±  9%     +58.1%     243.55 ±  3%  sched_debug.cfs_rq:/.load_avg.min
     56938 ±  3%     -39.7%      34341 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
    203752 ± 14%     -36.3%     129884 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
     53503 ± 11%     -39.0%      32655 ±  3%  sched_debug.cfs_rq:/.min_vruntime.stddev
      1.18 ±  9%     -23.2%       0.91 ±  3%  sched_debug.cfs_rq:/.nr_spread_over.avg
      5.35 ± 57%     -46.7%       2.85 ± 22%  sched_debug.cfs_rq:/.nr_spread_over.max
      0.88 ±  0%      -9.3%       0.80 ±  0%  sched_debug.cfs_rq:/.nr_spread_over.min
      0.74 ± 45%     -46.3%       0.40 ± 29%  sched_debug.cfs_rq:/.nr_spread_over.stddev
   -133253 ±-27%     -30.0%     -93231 ± -9%  sched_debug.cfs_rq:/.spread0.avg
   -185913 ±-19%     -33.2%    -124114 ± -7%  sched_debug.cfs_rq:/.spread0.min
     53510 ± 11%     -39.0%      32659 ±  3%  sched_debug.cfs_rq:/.spread0.stddev
    121.15 ±  5%     +37.4%     166.47 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
     60.38 ±  7%     +64.8%      99.50 ±  5%  sched_debug.cfs_rq:/.util_avg.min
    268435 ±  5%     -37.7%     167150 ±  6%  sched_debug.cpu.clock.avg
    268443 ±  5%     -37.7%     167155 ±  6%  sched_debug.cpu.clock.max
    268426 ±  5%     -37.7%     167141 ±  6%  sched_debug.cpu.clock.min
    268435 ±  5%     -37.7%     167150 ±  6%  sched_debug.cpu.clock_task.avg
    268443 ±  5%     -37.7%     167155 ±  6%  sched_debug.cpu.clock_task.max
    268426 ±  5%     -37.7%     167141 ±  6%  sched_debug.cpu.clock_task.min
    427.52 ±  4%     -39.5%     258.79 ± 18%  sched_debug.cpu.curr->pid.avg
      7133 ±  5%     -38.5%       4388 ±  2%  sched_debug.cpu.curr->pid.max
      1597 ±  3%     -39.4%     968.49 ± 10%  sched_debug.cpu.curr->pid.stddev
     69143 ±  5%     -43.4%      39168 ±  0%  sched_debug.cpu.nr_load_updates.avg
    104414 ±  7%     -41.2%      61358 ±  0%  sched_debug.cpu.nr_load_updates.max
     40699 ± 12%     -35.7%      26180 ± 13%  sched_debug.cpu.nr_load_updates.min
     16173 ± 11%     -45.1%       8874 ±  2%  sched_debug.cpu.nr_load_updates.stddev
    572452 ±  6%     -68.8%     178351 ±  0%  sched_debug.cpu.nr_switches.avg
   2286232 ± 16%     -62.2%     863098 ±  1%  sched_debug.cpu.nr_switches.max
      1022 ±  9%     -31.2%     703.45 ±  7%  sched_debug.cpu.nr_switches.min
    670927 ± 13%     -66.2%     226751 ±  2%  sched_debug.cpu.nr_switches.stddev
     27.81 ±  8%     -30.2%      19.40 ± 22%  sched_debug.cpu.nr_uninterruptible.max
    -84.97 ±-15%     -60.2%     -33.80 ± -8%  sched_debug.cpu.nr_uninterruptible.min
     18.80 ±  9%     -57.2%       8.06 ±  6%  sched_debug.cpu.nr_uninterruptible.stddev
    572289 ±  6%     -68.9%     178115 ±  0%  sched_debug.cpu.sched_count.avg
   2287959 ± 16%     -62.3%     861574 ±  1%  sched_debug.cpu.sched_count.max
    683.79 ± 11%     -38.5%     420.60 ±  9%  sched_debug.cpu.sched_count.min
    671226 ± 13%     -66.2%     226646 ±  2%  sched_debug.cpu.sched_count.stddev
    285499 ±  6%     -69.0%      88575 ±  0%  sched_debug.cpu.sched_goidle.avg
   1139461 ± 16%     -62.3%     429764 ±  1%  sched_debug.cpu.sched_goidle.max
    277.76 ± 13%     -37.3%     174.25 ± 10%  sched_debug.cpu.sched_goidle.min
    334870 ± 13%     -66.3%     112944 ±  2%  sched_debug.cpu.sched_goidle.stddev
    286808 ±  6%     -68.9%      89323 ±  0%  sched_debug.cpu.ttwu_count.avg
   1292490 ± 15%     -55.4%     576405 ±  0%  sched_debug.cpu.ttwu_count.max
    300.28 ±  5%     -45.5%     163.75 ±  4%  sched_debug.cpu.ttwu_count.min
    348803 ± 13%     -62.2%     131973 ±  1%  sched_debug.cpu.ttwu_count.stddev
    255200 ± 11%     +21.7%     310625 ±  0%  sched_debug.cpu.ttwu_local.max
    162.35 ±  5%     -50.8%      79.90 ±  3%  sched_debug.cpu.ttwu_local.min
    268435 ±  5%     -37.7%     167147 ±  6%  sched_debug.cpu_clk
    267057 ±  5%     -37.9%     165797 ±  6%  sched_debug.ktime
    268435 ±  5%     -37.7%     167147 ±  6%  sched_debug.sched_clk
    204.00 ± 27%    -100.0%       0.00 ± -1%  sched_debug.tailf2fs_flush-259
      1.80 ± 21%     -49.3%       0.91 ±  3%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      2.37 ±  5%     +43.4%       3.40 ±  6%  perf-profile.calltrace.cycles-pp.__alloc_percpu_gfp.__percpu_counter_init.f2fs_alloc_inode.alloc_inode.new_inode_pseudo
      1.14 ±  4%     +43.6%       1.63 ±  2%  perf-profile.calltrace.cycles-pp.__blk_mq_complete_request.blk_mq_complete_request.__nvme_process_cq.nvme_irq.__handle_irq_event_percpu
      7.85 ±  7%     -36.7%       4.97 ± 12%  perf-profile.calltrace.cycles-pp.__f2fs_add_link.f2fs_create.path_openat.do_filp_open.do_sys_open
      0.62 ±  3%     +39.8%       0.87 ± 12%  perf-profile.calltrace.cycles-pp.__f2fs_submit_merged_bio.f2fs_submit_merged_bio_cond.f2fs_wait_on_page_writeback.wait_on_node_pages_writeback.f2fs_do_sync_file
      1.99 ±  7%     +51.6%       3.02 ±  9%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.filemap_write_and_wait_range.f2fs_do_sync_file.f2fs_sync_file.vfs_fsync_range
      0.97 ±  3%     +42.3%       1.39 ±  6%  perf-profile.calltrace.cycles-pp.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write.vfs_write.sys_write
      1.19 ±  4%     +45.1%       1.72 ±  1%  perf-profile.calltrace.cycles-pp.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event.handle_edge_irq.handle_irq
      1.17 ±  4%     +43.8%       1.67 ±  1%  perf-profile.calltrace.cycles-pp.__nvme_process_cq.nvme_irq.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event
      2.42 ±  5%     +44.2%       3.49 ±  6%  perf-profile.calltrace.cycles-pp.__percpu_counter_init.f2fs_alloc_inode.alloc_inode.new_inode_pseudo.new_inode
      1.16 ±  7%     +37.9%       1.59 ±  8%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.f2fs_balance_fs.f2fs_create.path_openat.do_filp_open
      0.93 ±  9%     +29.6%       1.21 ± 10%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.f2fs_balance_fs.f2fs_write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages
      0.40 ± 57%    +332.7%       1.72 ± 33%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.f2fs_write_data_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes
      1.08 ±  2%     +38.9%       1.51 ±  5%  perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      0.79 ±  4%     +47.9%       1.17 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up.__wake_up_bit.end_page_writeback.f2fs_write_end_io.bio_endio
      0.79 ±  4%     +48.6%       1.18 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_bit.end_page_writeback.f2fs_write_end_io.bio_endio.blk_update_request
      0.78 ±  4%     +48.9%       1.17 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up.__wake_up_bit.end_page_writeback.f2fs_write_end_io
      0.87 ±  8%    +248.6%       3.05 ± 31%  perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread
      0.59 ±  5%    +259.9%       2.13 ± 32%  perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn
      0.71 ±  4%     +44.4%       1.02 ±  2%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.wake_bit_function
      1.72 ± 27%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.wake_up_q.rwsem_wake
      2.69 ±  4%     +42.8%       3.84 ±  7%  perf-profile.calltrace.cycles-pp.alloc_inode.new_inode_pseudo.new_inode.f2fs_new_inode.f2fs_create
      0.99 ±  3%     +45.3%       1.43 ±  2%  perf-profile.calltrace.cycles-pp.bio_endio.blk_update_request.blk_mq_end_request.nvme_complete_rq.__blk_mq_complete_request
      1.14 ±  4%     +44.1%       1.64 ±  2%  perf-profile.calltrace.cycles-pp.blk_mq_complete_request.__nvme_process_cq.nvme_irq.__handle_irq_event_percpu.handle_irq_event_percpu
      1.09 ±  4%     +45.5%       1.58 ±  2%  perf-profile.calltrace.cycles-pp.blk_mq_end_request.nvme_complete_rq.__blk_mq_complete_request.blk_mq_complete_request.__nvme_process_cq
      1.02 ±  3%     +46.1%       1.49 ±  2%  perf-profile.calltrace.cycles-pp.blk_update_request.blk_mq_end_request.nvme_complete_rq.__blk_mq_complete_request.blk_mq_complete_request
      1.00 ± 14%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.call_rwsem_wake.up_write.percpu_up_write.set_node_addr.new_node_page
      0.78 ±  4%     +47.9%       1.15 ±  3%  perf-profile.calltrace.cycles-pp.default_wake_function.wake_bit_function.__wake_up_common.__wake_up.__wake_up_bit
      1.40 ±  4%     +42.1%       1.98 ±  1%  perf-profile.calltrace.cycles-pp.do_IRQ.ret_from_intr.cpuidle_enter.call_cpuidle.cpu_startup_entry
     16.62 ±  5%     -24.3%      12.58 ± 10%  perf-profile.calltrace.cycles-pp.do_fsync.sys_fsync.entry_SYSCALL_64_fastpath
      1.98 ±  7%     +51.4%       3.00 ±  8%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range.f2fs_do_sync_file.f2fs_sync_file
      0.55 ±  4%    +233.8%       1.85 ± 33%  perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
      0.91 ±  4%     +48.8%       1.36 ±  3%  perf-profile.calltrace.cycles-pp.end_page_writeback.f2fs_write_end_io.bio_endio.blk_update_request.blk_mq_end_request
      2.23 ± 16%     -55.4%       1.00 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      0.70 ±  4%     +43.8%       1.01 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function
      1.56 ± 23%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.wake_up_q
      7.83 ±  7%     -36.9%       4.94 ± 12%  perf-profile.calltrace.cycles-pp.f2fs_add_regular_entry.__f2fs_add_link.f2fs_create.path_openat.do_filp_open
      2.58 ±  5%     +43.4%       3.71 ±  6%  perf-profile.calltrace.cycles-pp.f2fs_alloc_inode.alloc_inode.new_inode_pseudo.new_inode.f2fs_new_inode
      1.33 ±  8%     +50.5%       2.00 ±  7%  perf-profile.calltrace.cycles-pp.f2fs_balance_fs.f2fs_create.path_openat.do_filp_open.do_sys_open
      1.15 ±  8%     +59.0%       1.83 ±  9%  perf-profile.calltrace.cycles-pp.f2fs_balance_fs.f2fs_write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
      0.77 ±  2%     +52.9%       1.17 ±  8%  perf-profile.calltrace.cycles-pp.f2fs_dentry_hash.find_in_level.f2fs_find_entry.f2fs_lookup.path_openat
     16.57 ±  5%     -24.4%      12.52 ± 10%  perf-profile.calltrace.cycles-pp.f2fs_do_sync_file.f2fs_sync_file.vfs_fsync_range.do_fsync.sys_fsync
      1.02 ±  3%     +42.2%       1.45 ±  6%  perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      4.88 ±  7%     +59.7%       7.79 ±  6%  perf-profile.calltrace.cycles-pp.f2fs_find_entry.f2fs_lookup.path_openat.do_filp_open.do_sys_open
      4.96 ±  7%     +58.6%       7.87 ±  6%  perf-profile.calltrace.cycles-pp.f2fs_lookup.path_openat.do_filp_open.do_sys_open.sys_open
      3.11 ±  4%     +41.6%       4.40 ±  6%  perf-profile.calltrace.cycles-pp.f2fs_new_inode.f2fs_create.path_openat.do_filp_open.do_sys_open
      0.68 ±  3%     +41.1%       0.95 ± 13%  perf-profile.calltrace.cycles-pp.f2fs_submit_merged_bio_cond.f2fs_wait_on_page_writeback.wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_sync_file
     16.59 ±  5%     -24.4%      12.55 ± 10%  perf-profile.calltrace.cycles-pp.f2fs_sync_file.vfs_fsync_range.do_fsync.sys_fsync.entry_SYSCALL_64_fastpath
      1.03 ±  4%     +59.1%       1.64 ± 13%  perf-profile.calltrace.cycles-pp.f2fs_wait_on_page_writeback.wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_sync_file.vfs_fsync_range
      0.67 ±  2%     +35.6%       0.91 ±  7%  perf-profile.calltrace.cycles-pp.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write
      1.65 ±  7%     +43.8%       2.37 ± 10%  perf-profile.calltrace.cycles-pp.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range
      1.32 ±  7%     +45.0%       1.92 ±  9%  perf-profile.calltrace.cycles-pp.f2fs_write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range
      1.98 ±  7%     +51.1%       2.98 ±  8%  perf-profile.calltrace.cycles-pp.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range.f2fs_do_sync_file
      0.55 ±  4%    +232.9%       1.85 ± 33%  perf-profile.calltrace.cycles-pp.f2fs_write_data_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
      0.98 ±  3%     +45.8%       1.43 ±  2%  perf-profile.calltrace.cycles-pp.f2fs_write_end_io.bio_endio.blk_update_request.blk_mq_end_request.nvme_complete_rq
      3.10 ±  4%     -65.8%       1.06 ± 13%  perf-profile.calltrace.cycles-pp.f2fs_write_node_page.fsync_node_pages.f2fs_do_sync_file.f2fs_sync_file.vfs_fsync_range
      2.08 ±  6%     +63.2%       3.40 ±  9%  perf-profile.calltrace.cycles-pp.filemap_write_and_wait_range.f2fs_do_sync_file.f2fs_sync_file.vfs_fsync_range.do_fsync
      6.71 ± 14%     -46.1%       3.62 ± 18%  perf-profile.calltrace.cycles-pp.find_get_pages_tag.pagevec_lookup_tag.fsync_node_pages.f2fs_do_sync_file.f2fs_sync_file
      4.83 ±  7%     +59.5%       7.71 ±  6%  perf-profile.calltrace.cycles-pp.find_in_level.f2fs_find_entry.f2fs_lookup.path_openat.do_filp_open
      2.81 ±  8%     +76.4%       4.95 ±  5%  perf-profile.calltrace.cycles-pp.find_target_dentry.find_in_level.f2fs_find_entry.f2fs_lookup.path_openat
     11.21 ±  7%     -52.2%       5.37 ± 15%  perf-profile.calltrace.cycles-pp.fsync_node_pages.f2fs_do_sync_file.f2fs_sync_file.vfs_fsync_range.do_fsync
      0.85 ±  3%     +38.6%       1.18 ±  6%  perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write.vfs_write
      3.32 ±  9%     -78.2%       0.72 ± 12%  perf-profile.calltrace.cycles-pp.get_node_info.new_node_page.new_inode_page.init_inode_metadata.f2fs_add_regular_entry
      1.27 ±  4%     +41.7%       1.80 ±  1%  perf-profile.calltrace.cycles-pp.handle_edge_irq.handle_irq.do_IRQ.ret_from_intr.cpuidle_enter
      1.29 ±  4%     +42.5%       1.85 ±  1%  perf-profile.calltrace.cycles-pp.handle_irq.do_IRQ.ret_from_intr.cpuidle_enter.call_cpuidle
      1.22 ±  4%     +43.6%       1.75 ±  1%  perf-profile.calltrace.cycles-pp.handle_irq_event.handle_edge_irq.handle_irq.do_IRQ.ret_from_intr
      1.21 ±  3%     +44.6%       1.74 ±  1%  perf-profile.calltrace.cycles-pp.handle_irq_event_percpu.handle_irq_event.handle_edge_irq.handle_irq.do_IRQ
      7.20 ±  8%     -46.5%       3.85 ± 15%  perf-profile.calltrace.cycles-pp.init_inode_metadata.f2fs_add_regular_entry.__f2fs_add_link.f2fs_create.path_openat
      1.28 ±  8%    +187.7%       3.68 ± 26%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      1.27 ±  5%     +45.4%       1.85 ± 10%  perf-profile.calltrace.cycles-pp.memset_erms.__alloc_percpu_gfp.__percpu_counter_init.f2fs_alloc_inode.alloc_inode
      2.75 ±  4%     +42.4%       3.91 ±  6%  perf-profile.calltrace.cycles-pp.new_inode.f2fs_new_inode.f2fs_create.path_openat.do_filp_open
      6.96 ±  8%     -48.7%       3.57 ± 14%  perf-profile.calltrace.cycles-pp.new_inode_page.init_inode_metadata.f2fs_add_regular_entry.__f2fs_add_link.f2fs_create
      2.70 ±  4%     +42.9%       3.85 ±  7%  perf-profile.calltrace.cycles-pp.new_inode_pseudo.new_inode.f2fs_new_inode.f2fs_create.path_openat
      6.95 ±  8%     -49.0%       3.55 ± 15%  perf-profile.calltrace.cycles-pp.new_node_page.new_inode_page.init_inode_metadata.f2fs_add_regular_entry.__f2fs_add_link
      1.12 ±  4%     +44.3%       1.61 ±  2%  perf-profile.calltrace.cycles-pp.nvme_complete_rq.__blk_mq_complete_request.blk_mq_complete_request.__nvme_process_cq.nvme_irq
      1.17 ±  4%     +44.5%       1.70 ±  1%  perf-profile.calltrace.cycles-pp.nvme_irq.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event.handle_edge_irq
      6.83 ± 14%     -45.8%       3.70 ± 18%  perf-profile.calltrace.cycles-pp.pagevec_lookup_tag.fsync_node_pages.f2fs_do_sync_file.f2fs_sync_file.vfs_fsync_range
      1.00 ±  6%     +37.6%       1.38 ±  7%  perf-profile.calltrace.cycles-pp.pcpu_alloc.__alloc_percpu_gfp.__percpu_counter_init.f2fs_alloc_inode.alloc_inode
      1.56 ± 10%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.percpu_down_write.get_node_info.new_node_page.new_inode_page.init_inode_metadata
      1.58 ±  5%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.percpu_down_write.set_node_addr.f2fs_write_node_page.fsync_node_pages.f2fs_do_sync_file
      1.15 ±  8%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.percpu_down_write.set_node_addr.new_node_page.new_inode_page.init_inode_metadata
      0.87 ± 11%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.percpu_up_write.get_node_info.new_node_page.new_inode_page.init_inode_metadata
      1.03 ± 14%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.percpu_up_write.set_node_addr.new_node_page.new_inode_page.init_inode_metadata
      1.12 ±  8%    +213.3%       3.53 ± 27%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      3.52 ± 17%     -41.6%       2.05 ± 17%  perf-profile.calltrace.cycles-pp.radix_tree_next_chunk.find_get_pages_tag.pagevec_lookup_tag.fsync_node_pages.f2fs_do_sync_file
      1.28 ±  8%    +187.7%       3.68 ± 26%  perf-profile.calltrace.cycles-pp.ret_from_fork
      1.43 ±  3%     +41.6%       2.02 ±  1%  perf-profile.calltrace.cycles-pp.ret_from_intr.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
      1.63 ± 13%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.rwsem_wake.call_rwsem_wake.up_write.percpu_up_write.set_node_addr
      2.35 ±  6%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.set_node_addr.f2fs_write_node_page.fsync_node_pages.f2fs_do_sync_file.f2fs_sync_file
      2.33 ± 11%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.set_node_addr.new_node_page.new_inode_page.init_inode_metadata.f2fs_add_regular_entry
     16.63 ±  5%     -24.3%      12.59 ± 10%  perf-profile.calltrace.cycles-pp.sys_fsync.entry_SYSCALL_64_fastpath
      1.16 ±  1%     +41.0%       1.64 ±  5%  perf-profile.calltrace.cycles-pp.sys_write.entry_SYSCALL_64_fastpath
      0.77 ±  4%     +47.6%       1.14 ±  3%  perf-profile.calltrace.cycles-pp.try_to_wake_up.default_wake_function.wake_bit_function.__wake_up_common.__wake_up
      2.31 ± 12%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.call_rwsem_wake.up_write
      0.73 ±  3%     +45.2%       1.05 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.default_wake_function.wake_bit_function.__wake_up_common
      1.81 ± 26%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.wake_up_q.rwsem_wake.call_rwsem_wake
      1.02 ± 14%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.up_write.percpu_up_write.set_node_addr.new_node_page.new_inode_page
     16.60 ±  5%     -24.4%      12.56 ± 10%  perf-profile.calltrace.cycles-pp.vfs_fsync_range.do_fsync.sys_fsync.entry_SYSCALL_64_fastpath
      1.14 ±  2%     +41.3%       1.61 ±  5%  perf-profile.calltrace.cycles-pp.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.21 ±  5%     +59.6%       1.94 ± 12%  perf-profile.calltrace.cycles-pp.wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_sync_file.vfs_fsync_range.do_fsync
      0.78 ±  4%     +48.7%       1.16 ±  3%  perf-profile.calltrace.cycles-pp.wake_bit_function.__wake_up_common.__wake_up.__wake_up_bit.end_page_writeback
      2.37 ± 12%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.call_rwsem_wake.up_write.percpu_up_write
      0.96 ±  9%    +253.0%       3.38 ± 28%  perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      0.96 ±  9%    +253.0%       3.38 ± 28%  perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
      1.12 ±  8%    +213.6%       3.53 ± 27%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.85 ±  8%    +249.6%       2.96 ± 31%  perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work
      3.22 ±  9%     -50.4%       1.59 ±  6%  perf-profile.children.cycles-pp.__account_scheduler_latency
      2.38 ±  5%     +43.0%       3.40 ±  6%  perf-profile.children.cycles-pp.__alloc_percpu_gfp
      1.38 ±  6%     +57.5%       2.17 ±  8%  perf-profile.children.cycles-pp.__blk_mq_complete_request
      7.85 ±  7%     -36.7%       4.97 ± 12%  perf-profile.children.cycles-pp.__f2fs_add_link
      0.67 ±  4%     +46.8%       0.98 ±  8%  perf-profile.children.cycles-pp.__f2fs_submit_merged_bio
      2.08 ±  7%     +69.9%       3.53 ±  8%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.97 ±  3%     +43.6%       1.40 ±  6%  perf-profile.children.cycles-pp.__generic_file_write_iter
      1.43 ±  6%     +58.8%       2.28 ±  7%  perf-profile.children.cycles-pp.__handle_irq_event_percpu
      1.82 ± 10%     -47.3%       0.96 ±  6%  perf-profile.children.cycles-pp.__kernel_text_address
      1.41 ±  6%     +58.7%       2.24 ±  7%  perf-profile.children.cycles-pp.__nvme_process_cq
      2.42 ±  5%     +44.5%       3.50 ±  6%  perf-profile.children.cycles-pp.__percpu_counter_init
      3.59 ±  6%     +64.2%       5.89 ± 15%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.86 ±  7%     +53.4%       1.31 ± 10%  perf-profile.children.cycles-pp.__radix_tree_lookup
      1.46 ±  6%     -32.9%       0.98 ±  9%  perf-profile.children.cycles-pp.__schedule
      1.34 ±  3%     -11.0%       1.20 ±  5%  perf-profile.children.cycles-pp.__tick_nohz_idle_enter
      1.09 ±  1%     +40.1%       1.53 ±  5%  perf-profile.children.cycles-pp.__vfs_write
      1.04 ±  3%     +49.4%       1.56 ±  8%  perf-profile.children.cycles-pp.__wake_up
      1.08 ±  5%     +59.9%       1.72 ±  9%  perf-profile.children.cycles-pp.__wake_up_bit
      1.01 ±  4%     +54.8%       1.56 ±  8%  perf-profile.children.cycles-pp.__wake_up_common
      0.87 ±  8%    +248.6%       3.05 ± 31%  perf-profile.children.cycles-pp.__writeback_inodes_wb
      0.59 ±  5%    +259.9%       2.13 ± 32%  perf-profile.children.cycles-pp.__writeback_single_inode
      1.10 ±  7%     +81.0%       2.00 ± 15%  perf-profile.children.cycles-pp._raw_spin_lock
      3.67 ±  8%     -50.3%       1.82 ±  7%  perf-profile.children.cycles-pp.activate_task
      2.69 ±  4%     +42.6%       3.84 ±  6%  perf-profile.children.cycles-pp.alloc_inode
      1.23 ±  5%     +58.8%       1.95 ±  8%  perf-profile.children.cycles-pp.bio_endio
      1.39 ±  6%     +58.2%       2.19 ±  8%  perf-profile.children.cycles-pp.blk_mq_complete_request
      1.32 ±  5%     +59.1%       2.10 ±  8%  perf-profile.children.cycles-pp.blk_mq_end_request
      1.24 ±  5%     +59.6%       1.98 ±  8%  perf-profile.children.cycles-pp.blk_update_request
      3.44 ± 10%     -82.2%       0.61 ± 14%  perf-profile.children.cycles-pp.call_rwsem_wake
      0.99 ±  4%     +55.9%       1.54 ±  8%  perf-profile.children.cycles-pp.default_wake_function
      1.67 ±  6%     +56.1%       2.61 ±  7%  perf-profile.children.cycles-pp.do_IRQ
     16.62 ±  5%     -24.3%      12.58 ± 10%  perf-profile.children.cycles-pp.do_fsync
      2.68 ±  6%    +106.4%       5.53 ± 16%  perf-profile.children.cycles-pp.do_writepages
      2.90 ±  9%     -49.2%       1.47 ±  6%  perf-profile.children.cycles-pp.dump_trace
      1.11 ±  5%     +61.1%       1.79 ±  9%  perf-profile.children.cycles-pp.end_page_writeback
      3.56 ±  8%     -50.4%       1.77 ±  7%  perf-profile.children.cycles-pp.enqueue_entity
      3.60 ±  8%     -50.3%       1.79 ±  7%  perf-profile.children.cycles-pp.enqueue_task_fair
      7.83 ±  7%     -36.9%       4.94 ± 12%  perf-profile.children.cycles-pp.f2fs_add_regular_entry
      2.59 ±  5%     +43.7%       3.72 ±  6%  perf-profile.children.cycles-pp.f2fs_alloc_inode
      2.49 ±  8%     +54.4%       3.84 ±  8%  perf-profile.children.cycles-pp.f2fs_balance_fs
      0.86 ±  2%     +49.1%       1.28 ±  9%  perf-profile.children.cycles-pp.f2fs_dentry_hash
     16.57 ±  5%     -24.4%      12.53 ± 10%  perf-profile.children.cycles-pp.f2fs_do_sync_file
      1.02 ±  3%     +42.8%       1.46 ±  6%  perf-profile.children.cycles-pp.f2fs_file_write_iter
      4.88 ±  7%     +59.8%       7.80 ±  6%  perf-profile.children.cycles-pp.f2fs_find_entry
      4.97 ±  7%     +58.5%       7.88 ±  6%  perf-profile.children.cycles-pp.f2fs_lookup
      3.11 ±  4%     +41.6%       4.41 ±  6%  perf-profile.children.cycles-pp.f2fs_new_inode
      0.68 ±  4%     +40.7%       0.96 ± 13%  perf-profile.children.cycles-pp.f2fs_submit_merged_bio_cond
     16.60 ±  5%     -24.4%      12.55 ± 10%  perf-profile.children.cycles-pp.f2fs_sync_file
      1.04 ±  4%     +58.6%       1.65 ± 13%  perf-profile.children.cycles-pp.f2fs_wait_on_page_writeback
      0.67 ±  2%     +35.6%       0.91 ±  7%  perf-profile.children.cycles-pp.f2fs_write_begin
      1.73 ±  7%     +65.8%       2.87 ±  9%  perf-profile.children.cycles-pp.f2fs_write_cache_pages
      1.39 ±  7%     +68.6%       2.33 ±  9%  perf-profile.children.cycles-pp.f2fs_write_data_page
      2.62 ±  6%    +104.0%       5.35 ± 16%  perf-profile.children.cycles-pp.f2fs_write_data_pages
      1.20 ±  5%     +58.7%       1.91 ±  8%  perf-profile.children.cycles-pp.f2fs_write_end_io
      3.11 ±  4%     -65.5%       1.07 ± 13%  perf-profile.children.cycles-pp.f2fs_write_node_page
      2.08 ±  6%     +63.1%       3.40 ±  9%  perf-profile.children.cycles-pp.filemap_write_and_wait_range
      0.95 ± 11%     +47.9%       1.40 ±  8%  perf-profile.children.cycles-pp.find_get_entry
      6.83 ± 13%     -43.6%       3.85 ± 17%  perf-profile.children.cycles-pp.find_get_pages_tag
      4.84 ±  7%     +59.4%       7.71 ±  6%  perf-profile.children.cycles-pp.find_in_level
      2.82 ±  8%     +76.0%       4.96 ±  5%  perf-profile.children.cycles-pp.find_target_dentry
     11.21 ±  7%     -52.2%       5.37 ± 15%  perf-profile.children.cycles-pp.fsync_node_pages
      0.85 ±  2%     +39.0%       1.18 ±  7%  perf-profile.children.cycles-pp.generic_perform_write
      3.51 ±  8%     -75.7%       0.85 ± 10%  perf-profile.children.cycles-pp.get_node_info
      0.69 ± 10%     +52.7%       1.05 ±  7%  perf-profile.children.cycles-pp.get_node_page
      1.54 ±  6%     +55.7%       2.39 ±  8%  perf-profile.children.cycles-pp.handle_edge_irq
      1.56 ±  5%     +56.2%       2.44 ±  8%  perf-profile.children.cycles-pp.handle_irq
      1.47 ±  5%     +57.7%       2.32 ±  7%  perf-profile.children.cycles-pp.handle_irq_event
      1.46 ±  5%     +58.0%       2.30 ±  7%  perf-profile.children.cycles-pp.handle_irq_event_percpu
      7.20 ±  8%     -46.4%       3.86 ± 15%  perf-profile.children.cycles-pp.init_inode_metadata
      1.28 ±  8%    +187.7%       3.68 ± 26%  perf-profile.children.cycles-pp.kthread
      1.35 ±  5%     +44.7%       1.96 ± 10%  perf-profile.children.cycles-pp.memset_erms
      0.65 ± 14%    +107.7%       1.35 ± 20%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      2.75 ±  4%     +42.3%       3.91 ±  6%  perf-profile.children.cycles-pp.new_inode
      6.97 ±  8%     -48.8%       3.57 ± 14%  perf-profile.children.cycles-pp.new_inode_page
      2.70 ±  4%     +42.8%       3.86 ±  7%  perf-profile.children.cycles-pp.new_inode_pseudo
      6.96 ±  8%     -48.9%       3.55 ± 15%  perf-profile.children.cycles-pp.new_node_page
      1.35 ±  6%     +58.9%       2.15 ±  8%  perf-profile.children.cycles-pp.nvme_complete_rq
      1.42 ±  6%     +58.2%       2.24 ±  7%  perf-profile.children.cycles-pp.nvme_irq
      1.88 ±  6%     +44.5%       2.72 ±  8%  perf-profile.children.cycles-pp.pagecache_get_page
      6.98 ± 13%     -43.3%       3.96 ± 17%  perf-profile.children.cycles-pp.pagevec_lookup_tag
      1.01 ±  6%     +38.5%       1.40 ±  7%  perf-profile.children.cycles-pp.pcpu_alloc
      1.91 ±  4%    -100.0%       0.00 ± -1%  perf-profile.children.cycles-pp.percpu_down_read
      4.30 ±  8%    -100.0%       0.00 ± -1%  perf-profile.children.cycles-pp.percpu_down_write
      2.56 ± 12%    -100.0%       0.00 ± -1%  perf-profile.children.cycles-pp.percpu_up_write
      2.42 ± 10%     -48.6%       1.24 ±  6%  perf-profile.children.cycles-pp.print_context_stack
      1.12 ±  8%    +213.3%       3.53 ± 27%  perf-profile.children.cycles-pp.process_one_work
      3.68 ± 16%     -38.9%       2.25 ± 16%  perf-profile.children.cycles-pp.radix_tree_next_chunk
      1.28 ±  8%    +186.5%       3.68 ± 26%  perf-profile.children.cycles-pp.ret_from_fork
      1.70 ±  5%     +55.6%       2.65 ±  7%  perf-profile.children.cycles-pp.ret_from_intr
      3.42 ± 10%     -82.1%       0.61 ± 13%  perf-profile.children.cycles-pp.rwsem_wake
      2.90 ±  9%     -49.1%       1.48 ±  6%  perf-profile.children.cycles-pp.save_stack_trace_tsk
      1.50 ±  6%     -31.7%       1.03 ±  9%  perf-profile.children.cycles-pp.schedule
      4.69 ±  8%     -88.3%       0.55 ±  9%  perf-profile.children.cycles-pp.set_node_addr
     16.64 ±  5%     -24.3%      12.60 ± 10%  perf-profile.children.cycles-pp.sys_fsync
      1.17 ±  2%     +40.7%       1.65 ±  5%  perf-profile.children.cycles-pp.sys_write
      1.16 ±  2%     -13.2%       1.00 ±  6%  perf-profile.children.cycles-pp.tick_nohz_stop_sched_tick
      4.48 ±  9%     -50.9%       2.20 ±  9%  perf-profile.children.cycles-pp.try_to_wake_up
      3.82 ±  8%     -50.5%       1.89 ±  7%  perf-profile.children.cycles-pp.ttwu_do_activate
      2.73 ± 11%     -75.1%       0.68 ±  6%  perf-profile.children.cycles-pp.up_write
     16.61 ±  5%     -24.3%      12.56 ± 10%  perf-profile.children.cycles-pp.vfs_fsync_range
      1.14 ±  2%     +41.8%       1.62 ±  5%  perf-profile.children.cycles-pp.vfs_write
      1.21 ±  5%     +59.8%       1.94 ± 12%  perf-profile.children.cycles-pp.wait_on_node_pages_writeback
      0.94 ±  5%     +61.3%       1.51 ±  8%  perf-profile.children.cycles-pp.wake_bit_function
      3.57 ± 10%     -81.4%       0.66 ± 10%  perf-profile.children.cycles-pp.wake_up_q
      0.96 ±  9%    +253.0%       3.38 ± 28%  perf-profile.children.cycles-pp.wb_workfn
      0.96 ±  9%    +253.0%       3.38 ± 28%  perf-profile.children.cycles-pp.wb_writeback
      1.12 ±  8%    +213.6%       3.53 ± 27%  perf-profile.children.cycles-pp.worker_thread
      0.85 ±  8%    +249.6%       2.96 ± 31%  perf-profile.children.cycles-pp.writeback_sb_inodes
      2.58 ±  7%     +59.9%       4.12 ± 14%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.86 ±  7%     +53.4%       1.31 ± 10%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.86 ±  2%     +49.1%       1.28 ±  9%  perf-profile.self.cycles-pp.f2fs_dentry_hash
      3.19 ± 10%     -48.2%       1.65 ± 18%  perf-profile.self.cycles-pp.find_get_pages_tag
      2.82 ±  8%     +76.0%       4.96 ±  5%  perf-profile.self.cycles-pp.find_target_dentry
      1.35 ±  5%     +44.7%       1.96 ± 10%  perf-profile.self.cycles-pp.memset_erms
      0.65 ± 14%    +107.7%       1.35 ± 20%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.74 ±  8%     +37.9%       1.03 ±  9%  perf-profile.self.cycles-pp.pcpu_alloc
      3.20 ±  8%    -100.0%       0.00 ± -1%  perf-profile.self.cycles-pp.percpu_down_write
      3.68 ± 16%     -38.9%       2.25 ± 16%  perf-profile.self.cycles-pp.radix_tree_next_chunk



                                 perf-stat.dTLB-loads

  9.5e+11 *+---------------------------*------------------------------------+
          |  .  ..*...              ..  +                                   |
    9e+11 ++  *.      *....*...*...*     +       ..*...*....*...            |
  8.5e+11 ++                              +  ..*.               *...*...*...|
          |                                *.                               *
    8e+11 ++                                                                |
          |                                                                 |
  7.5e+11 ++                                                                |
          |                                                                 |
    7e+11 ++                                                                |
  6.5e+11 ++                                                                |
          O                                                                 |
    6e+11 ++                                                                |
          |   O       O                                                     |
  5.5e+11 ++------O---------------------------------------------------------+


                              perf-stat.dTLB-load-misses

  2.2e+10 ++----------------------------------------------------------------+
  2.1e+10 *+                           *                                    |
          | ..   .*...*....*...*..   .. +                                   |
    2e+10 ++   ..                 . .    +    .*...  ..*....*...       .*.. |
  1.9e+10 ++  *                    *      + ..     *.           *... ..    .|
          |                                *                        *       *
  1.8e+10 ++                                                                |
  1.7e+10 ++                                                                |
  1.6e+10 ++                                                                |
          |                                                                 |
  1.5e+10 O+                                                                |
  1.4e+10 ++                                                                |
          |   O       O                                                     |
  1.3e+10 ++      O                                                         |
  1.2e+10 ++----------------------------------------------------------------+


                              perf-stat.dTLB-store-misses

  1.2e+09 ++----------------------------------------------------------------+
  1.1e+09 *+                                                                |
          | ..           ..*...                                             |
    1e+09 ++        ..*..      *...*...               .*....*...      ..*...|
    9e+08 ++  *...*.                   *...*...*... ..          *...*.      *
          |                                        *                        |
    8e+08 ++                                                                |
    7e+08 ++                                                                |
    6e+08 ++                                                                |
          |                                                                 |
    5e+08 ++                                                                |
    4e+08 ++                                                                |
          |           O                                                     |
    3e+08 O+  O   O                                                         |
    2e+08 ++----------------------------------------------------------------+


                                 perf-stat.iTLB-loads

  1.8e+09 ++----------------------------------------------------------------+
          *..                 .*...  ..*...*...             *..             |
  1.7e+09 ++ .  ..*...*.... ..     *.          *...*.      .   .  ..*...*...|
  1.6e+09 ++  *.           *                         ..  ..     *.          *
          |                                             .                   |
  1.5e+09 ++                                           *                    |
          |                                                                 |
  1.4e+09 ++                                                                |
          |                                                                 |
  1.3e+09 ++                                                                |
  1.2e+09 ++                                                                |
          O                                                                 |
  1.1e+09 ++      O                                                         |
          |   O       O                                                     |
    1e+09 ++----------------------------------------------------------------+


                                 perf-stat.page-faults

  1.05e+06 ++---------------------------------------------------------------+
     1e+06 ++                                  .*.                          |
           |                                 ..   ..          ..*..         |
    950000 *+                        ..*....*              .*.     .        |
    900000 ++..   .*...  ..*...  ..*.               *... ..         *...    |
           |    ..     *.      *.                       *               *...*
    850000 ++  *                                                            |
    800000 ++                                                               |
    750000 ++                                                               |
           |                                                                |
    700000 ++                                                               |
    650000 ++                                                               |
           |                                                                |
    600000 O+  O                                                            |
    550000 ++------O---O----------------------------------------------------+


                             perf-stat.context-switches

  8e+07 *+-------*---*---*---*----*---*---*---*----*-------*---*----*-------+
        |   *..                                        *.               *...*
  7e+07 ++                                                                  |
        |                                                                   |
        |                                                                   |
  6e+07 ++                                                                  |
        |                                                                   |
  5e+07 ++                                                                  |
        |                                                                   |
  4e+07 ++                                                                  |
        |                                                                   |
        |                                                                   |
  3e+07 O+  O    O   O                                                      |
        |                                                                   |
  2e+07 ++------------------------------------------------------------------+


                              perf-stat.cpu-migrations

  400000 ++-----------------------------------------------------------------+
         |                                   ..*...                         |
         *..                        ..*....*.      *..       ..*...         |
  350000 ++ .        .*...*...  ..*.                  .  ..*.      .      ..*
         |   *...  ..         *.                       *.           *...*.  |
         |       *.                                                         |
  300000 ++                                                                 |
         |                                                                  |
  250000 ++                                                                 |
         |                                                                  |
         |                                                                  |
  200000 ++                                                                 |
         |                                                                  |
         O   O   O    O                                                     |
  150000 ++-----------------------------------------------------------------+


                                perf-stat.minor-faults

  1.05e+06 ++---------------------------------------------------------------+
     1e+06 ++                                  .*.                          |
           |                                 ..   ..          ..*..         |
    950000 *+                        ..*....*              .*.     .        |
    900000 ++..   .*...  ..*...  ..*.               *... ..         *...    |
           |    ..     *.      *.                       *               *...*
    850000 ++  *                                                            |
    800000 ++                                                               |
    750000 ++                                                               |
           |                                                                |
    700000 ++                                                               |
    650000 ++                                                               |
           |                                                                |
    600000 O+  O                                                            |
    550000 ++------O---O----------------------------------------------------+


                          perf-stat.dTLB-store-miss-rate_

  0.22 ++-------------------------------------------------------------------+
       |                                                                    |
   0.2 *+.                                                                ..*
       |  .                                           *...       ..*....*.  |
  0.18 ++  *....*...*...*..                         ..    .  ..*.           |
       |                   ..   .*...*....*...     .       *.               |
  0.16 ++                     ..              *...*                         |
       |                     *                                              |
  0.14 ++                                                                   |
       |                                                                    |
  0.12 ++                                                                   |
       |                                                                    |
   0.1 ++                                                                   |
       O   O    O   O                                                       |
  0.08 ++-------------------------------------------------------------------+



                              fsmark.time.system_time

  1600 ++-------------------------------------------------------------------+
       |                                                                    |
  1500 *+..   ..*...*...*....*...*...*....*...*...*...   ..*...*...*....    |
  1400 ++  *..                                        *..               *...*
       |                                                                    |
  1300 ++                                                                   |
       |                                                                    |
  1200 ++                                                                   |
       |                                                                    |
  1100 ++                                                                   |
  1000 ++                                                                   |
       |                                                                    |
   900 O+                                                                   |
       |   O    O   O                                                       |
   800 ++-------------------------------------------------------------------+


                      fsmark.time.percent_of_cpu_this_job_got

  340 ++--O-----------------------------------------------------------------+
      O                                                                     |
      |        O   O                                                        |
  335 ++                                                                    |
      |                                                                     |
      |                                                                     |
      |                                                                     |
  330 ++                                                                    |
      |                                                                     |
      |                                                                     |
  325 ++  *..                                         *..              .*...*
      | ..   ..                                      +   ..          ..     |
      |.                                .*..        +            ..*.       |
      *        *...*....*...*...*.... ..    .      +       *...*.           |
  320 ++-----------------------------*-------*----*-------------------------+


                             fsmark.time.elapsed_time

  500 ++--------------------------------------------------------------------+
      *..             ..*...  ..*....*...*...*....*...   ..*...*...*....    |
      |  .   ..*...*..      *.                        *..               *...*
  450 ++  *..                                                               |
      |                                                                     |
      |                                                                     |
  400 ++                                                                    |
      |                                                                     |
  350 ++                                                                    |
      |                                                                     |
      |                                                                     |
  300 ++                                                                    |
      O                                                                     |
      |   O    O   O                                                        |
  250 ++--------------------------------------------------------------------+


                           fsmark.time.elapsed_time.max

  500 ++--------------------------------------------------------------------+
      *..             ..*...  ..*....*...*...*....*...   ..*...*...*....    |
      |  .   ..*...*..      *.                        *..               *...*
  450 ++  *..                                                               |
      |                                                                     |
      |                                                                     |
  400 ++                                                                    |
      |                                                                     |
  350 ++                                                                    |
      |                                                                     |
      |                                                                     |
  300 ++                                                                    |
      O                                                                     |
      |   O    O   O                                                        |
  250 ++--------------------------------------------------------------------+


                        fsmark.time.voluntary_context_switches

    4e+07 ++-----------------------*---*------------------------------------+
          *...  ..*...*....*...*.          *...*...*...   ..*...*...*...    |
  3.5e+07 ++  *.                                       *..              *...*
          |                                                                 |
          |                                                                 |
    3e+07 ++                                                                |
          |                                                                 |
  2.5e+07 ++                                                                |
          |                                                                 |
    2e+07 ++                                                                |
          |                                                                 |
          |                                                                 |
  1.5e+07 O+      O   O                                                     |
          |   O                                                             |
    1e+07 ++----------------------------------------------------------------+


                            fsmark.time.file_system_outputs

  2.35e+08 ++---------------------------------------------------------------+
           O   O   O   O                                                    |
   2.3e+08 ++                                                               |
  2.25e+08 ++                                                               |
           |                                                                |
   2.2e+08 ++                                                               |
           |                                                                |
  2.15e+08 ++                                                               |
           |                                                                |
   2.1e+08 ++                                                               |
  2.05e+08 ++                                                               |
           |                                                                |
     2e+08 *+..*...*...*...*...*...*...*....*...*...*...*...*...*...*...*...*
           |                                                                |
  1.95e+08 ++---------------------------------------------------------------+


                                fsmark.files_per_sec

  40000 ++-----------O------------------------------------------------------+
  38000 O+  O    O                                                          |
        |                                                                   |
  36000 ++                                                                  |
  34000 ++                                                                  |
        |                                                                   |
  32000 ++                                                                  |
  30000 ++                                                                  |
  28000 ++                                                                  |
        |                                                                   |
  26000 ++                                                                  |
  24000 ++                                                                  |
        *...*....*...*...  ..*....                   ..*...           ..*...*
  22000 ++               *.       *...*...*...*....*.      *...*....*.      |
  20000 ++------------------------------------------------------------------+


                       interrupts.CAL:Function_call_interrupts

  380000 ++-----------------------------------------------------------------+
         |                                                .*...             |
  360000 *+.            ..*...*...*...*....*...*...*... ..     *....*..     |
  340000 ++ .  ..*....*.                               *               .  ..*
         |   *.                                                         *.  |
  320000 ++                                                                 |
  300000 ++                                                                 |
         |                                                                  |
  280000 ++                                                                 |
  260000 ++                                                                 |
         |                                                                  |
  240000 ++                                                                 |
  220000 ++                                                                 |
         O   O   O                                                          |
  200000 ++-----------O-----------------------------------------------------+


                                   vmstat.io.bi

  340 ++--O--------O--------------------------------------------------------+
      O        O                                                            |
  320 ++                                                                    |
  300 ++                                                                    |
      |                                                                     |
  280 ++                                                                    |
      |                                                                     |
  260 ++                                                                    |
      |                                                                     |
  240 ++                                                                    |
  220 ++                                                                    |
      |                                                                     |
  200 ++..*....                                     ..*....           ..*...*
      *.       *...*....*...*...*....*...*...*....*.       *...*...*..      |
  180 ++--------------------------------------------------------------------+


                                    vmstat.io.bo

  280000 ++-----------O-----------------------------------------------------+
         O   O   O                                                          |
  260000 ++                                                                 |
  240000 ++                                                                 |
         |                                                                  |
  220000 ++                                                                 |
  200000 ++                                                                 |
         |                                                                  |
  180000 ++                                                                 |
  160000 ++                                                                 |
         |                                                                  |
  140000 ++                                                                 |
  120000 *+..*...*....*...*...*...*...*....*...*...*...*...  ..*....*...*...*
         |                                                 *.               |
  100000 ++-----------------------------------------------------------------+


                                  vmstat.system.in

  115000 ++-----------------------------------------------------------------+
         |   O   O    O                                                     |
         O                                                                  |
  110000 ++                                                                 |
         |                                                                  |
         |                                                                  |
  105000 ++                                                                 |
         |                                                                  |
  100000 ++                                                                 |
         |                                                                  |
         |                                                                  |
   95000 ++..*...                                    ..*...           ..*...*
         *.      *....*...*...*...*...*....*...*...*.      *...*....*.      |
         |                                                                  |
   90000 ++-----------------------------------------------------------------+


                                  vmstat.system.cs

  170000 ++-----------------------------------------------------------------+
         *...*...*....*...*...*...*...*....*...*...*...*...*...*....*...*...*
  160000 ++                                                                 |
  150000 ++                                                                 |
         |                                                                  |
  140000 ++                                                                 |
         |                                                                  |
  130000 ++                                                                 |
         |                                                                  |
  120000 ++                                                                 |
  110000 ++                                                                 |
         |                                                                  |
  100000 ++  O   O    O                                                     |
         O                                                                  |
   90000 ++-----------------------------------------------------------------+

	[*] bisect-good sample
	[O] bisect-bad  sample





Thanks,
Xiaolong

View attachment "config-4.8.0-rc2-00233-gb873b79" of type "text/plain" (152771 bytes)

View attachment "job-script" of type "text/plain" (7072 bytes)

View attachment "job.yaml" of type "text/plain" (4638 bytes)

View attachment "reproduce" of type "text/plain" (371 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ