lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20181119091440.GO18977@shao2-debian>
Date:   Mon, 19 Nov 2018 17:14:40 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Jaegeuk Kim <jaegeuk@...nel.org>
Cc:     Sahitya Tummala <stummala@...eaurora.org>,
        Chao Yu <yuchao0@...wei.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [LKP] [f2fs]  853137cef4:  fsmark.files_per_sec 40.1% improvement

Greeting,

FYI, we noticed a 40.1% improvement of fsmark.files_per_sec due to commit:


commit: 853137cef46ccc490e6fd4b160a1c252d6459842 ("f2fs: fix performance issue observed with multi-thread sequential read")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: fsmark
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
with following parameters:

	iterations: 1x
	nr_threads: 64t
	disk: 1BRD_48G
	fs: f2fs
	filesize: 4M
	test_size: 40G
	sync_method: fsyncBeforeClose
	ucode: 0x42d
	cpufreq_governor: performance

test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-7/performance/1BRD_48G/4M/f2fs/1x/x86_64-rhel-7.2/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/ivb44/40G/fsmark/0x42d

commit: 
  dda9f4b9ca ("f2fs: fix to skip verifying block address for non-regular inode")
  853137cef4 ("f2fs: fix performance issue observed with multi-thread sequential read")

dda9f4b9cac6bdd2 853137cef46ccc490e6fd4b160 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_ret_from_intr/0x
          4:4         -100%            :4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=rwsem_down_write_failed/0x
          1:4          -25%            :4     kmsg.pstore:crypto_comp_decompress_failed,ret=
          1:4          -25%            :4     kmsg.pstore:decompression_failed
         %stddev     %change         %stddev
             \          |                \  
   1292197           -19.9%    1034902        fsmark.app_overhead
    265.67           +40.1%     372.30 ±  3%  fsmark.files_per_sec
     41.39           -24.2%      31.38 ±  2%  fsmark.time.elapsed_time
     41.39           -24.2%      31.38 ±  2%  fsmark.time.elapsed_time.max
     27468           -99.3%     205.75 ±  9%  fsmark.time.involuntary_context_switches
     79094            -6.3%      74088        fsmark.time.minor_page_faults
      3837           -91.5%     324.50 ±  5%  fsmark.time.percent_of_cpu_this_job_got
      1587           -93.6%     101.14 ±  8%  fsmark.time.system_time
   1235762 ±  2%     -98.9%      13888 ±  6%  fsmark.time.voluntary_context_switches
     59558            -6.5%      55713        interrupts.CAL:Function_call_interrupts
     23.54 ±  9%     +67.9       91.48        mpstat.cpu.idle%
      0.04 ±  3%      -0.0        0.00 ± 23%  mpstat.cpu.iowait%
     74.31 ±  2%     -67.9        6.39 ±  7%  mpstat.cpu.sys%
    277783 ±  3%     -35.8%     178384        softirqs.RCU
    283065 ±  4%      -7.4%     262224 ±  6%  softirqs.SCHED
   1134400 ±  7%     -14.5%     969409 ±  5%  softirqs.TIMER
    920993 ±  2%     +28.3%    1181660 ±  5%  vmstat.io.bo
     38.50 ±  7%     -89.0%       4.25 ± 10%  vmstat.procs.r
     52008 ±  2%     -95.5%       2337 ±  4%  vmstat.system.cs
     57273           -11.9%      50430        vmstat.system.in
    276.98 ± 14%     -18.5%     225.62 ± 19%  sched_debug.cfs_rq:/.load_avg.stddev
      5303 ± 16%     -25.6%       3943 ± 26%  sched_debug.cfs_rq:/.min_vruntime.avg
      9599 ±105%     -66.0%       3268 ±  8%  sched_debug.cfs_rq:/.runnable_weight.avg
     42652 ±144%     -84.1%       6788 ±  4%  sched_debug.cfs_rq:/.runnable_weight.stddev
     49.18 ± 26%     -41.0%      29.03 ± 21%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    191284 ± 10%     +17.9%     225588 ±  5%  meminfo.DirectMap4k
    158559 ±  2%     +51.5%     240138 ±  3%  meminfo.Dirty
    225168 ±159%     -93.2%      15267        meminfo.Inactive(anon)
     27200           -10.4%      24381        meminfo.Mapped
    412653 ±  2%     -59.4%     167408 ±  2%  meminfo.SReclaimable
    305084 ±147%     -93.6%      19584        meminfo.Shmem
    523258           -48.4%     270220        meminfo.Slab
    151.00 ± 79%    +830.1%       1404 ±  2%  meminfo.Writeback
  52991739 ±  3%     -95.7%    2255645 ± 11%  cpuidle.C1.time
    533066 ±  3%     -94.7%      28389 ± 17%  cpuidle.C1.usage
  32907668 ±  2%     -98.4%     537881 ±  4%  cpuidle.C1E.time
    243722 ±  2%     -98.5%       3607        cpuidle.C1E.usage
  54974737 ±  2%     -97.7%    1264852 ±  2%  cpuidle.C3.time
    316006 ±  2%     -98.0%       6351        cpuidle.C3.usage
 3.284e+08 ± 15%    +348.6%  1.473e+09 ±  5%  cpuidle.C6.time
    475703 ± 12%    +229.0%    1565046 ±  6%  cpuidle.C6.usage
     53413 ±  5%     -96.2%       2007 ± 22%  cpuidle.POLL.time
      1051 ±  4%     -95.0%      52.75 ± 15%  cpuidle.POLL.usage
    217102 ±  2%     -26.3%     159989 ± 22%  numa-meminfo.node0.Active
    196552 ±  2%     -31.7%     134228 ± 28%  numa-meminfo.node0.Active(anon)
     20549           +25.4%      25761 ± 11%  numa-meminfo.node0.Active(file)
     78355 ±  2%     +49.7%     117324 ±  3%  numa-meminfo.node0.Dirty
      6462 ±  9%     +16.6%       7532 ±  3%  numa-meminfo.node0.KernelStack
      2154 ± 17%     +63.8%       3529 ± 19%  numa-meminfo.node0.PageTables
    206265           -54.4%      94031 ± 10%  numa-meminfo.node0.SReclaimable
    260986           -40.5%     155389 ±  7%  numa-meminfo.node0.Slab
     80164 ±  2%     +52.9%     122541 ±  7%  numa-meminfo.node1.Dirty
    106012 ±168%     -99.1%     943.50 ± 81%  numa-meminfo.node1.Inactive(anon)
      5978 ± 10%     -19.4%       4815 ±  5%  numa-meminfo.node1.KernelStack
     11366           -13.2%       9870        numa-meminfo.node1.Mapped
      3275 ± 12%     -44.5%       1816 ± 35%  numa-meminfo.node1.PageTables
    207093 ±  3%     -64.6%      73366 ± 11%  numa-meminfo.node1.SReclaimable
     55885 ±  4%     -25.8%      41453 ±  8%  numa-meminfo.node1.SUnreclaim
    262979           -56.3%     114819 ±  9%  numa-meminfo.node1.Slab
      2332 ±  2%     -86.7%     309.25 ±  4%  turbostat.Avg_MHz
     77.98 ±  2%     -68.1        9.89 ±  4%  turbostat.Busy%
    526510 ±  3%     -95.7%      22547 ± 17%  turbostat.C1
      2.50 ±  4%      -2.4        0.12 ±  8%  turbostat.C1%
    243631 ±  2%     -98.6%       3530 ±  2%  turbostat.C1E
      1.55 ±  2%      -1.5        0.03 ± 13%  turbostat.C1E%
    315854 ±  2%     -98.0%       6228        turbostat.C3
      2.60 ±  2%      -2.5        0.08 ±  5%  turbostat.C3%
    473755 ± 12%    +230.0%    1563267 ±  6%  turbostat.C6
     15.40 ± 13%     +74.4       89.83        turbostat.C6%
     13.39 ±  4%    +146.9%      33.08        turbostat.CPU%c1
      0.49 ±  5%     -95.9%       0.02        turbostat.CPU%c3
      8.13 ± 18%    +601.5%      57.02        turbostat.CPU%c6
    152.73           -40.7%      90.60        turbostat.CorWatt
   2578350 ±  2%     -31.6%    1764094 ±  5%  turbostat.IRQ
    182.65           -34.2%     120.19        turbostat.PkgWatt
     10.95            -6.3%      10.26 ±  2%  turbostat.RAMWatt
     49108 ±  2%     -31.7%      33562 ± 28%  numa-vmstat.node0.nr_active_anon
      5168 ±  2%     +24.4%       6429 ± 11%  numa-vmstat.node0.nr_active_file
     19426 ±  3%     +50.2%      29174 ±  3%  numa-vmstat.node0.nr_dirty
      6462 ±  9%     +16.6%       7534 ±  3%  numa-vmstat.node0.nr_kernel_stack
    538.25 ± 17%     +64.1%     883.50 ± 18%  numa-vmstat.node0.nr_page_table_pages
     51684 ±  2%     -54.4%      23550 ± 10%  numa-vmstat.node0.nr_slab_reclaimable
     17.75 ± 59%    +752.1%     151.25 ± 14%  numa-vmstat.node0.nr_writeback
     49109 ±  2%     -31.7%      33563 ± 28%  numa-vmstat.node0.nr_zone_active_anon
      5164 ±  2%     +24.6%       6432 ± 11%  numa-vmstat.node0.nr_zone_active_file
     18336 ±  3%     +59.9%      29314 ±  3%  numa-vmstat.node0.nr_zone_write_pending
     20180 ±  2%     +51.6%      30586 ±  6%  numa-vmstat.node1.nr_dirty
     26503 ±168%     -99.1%     235.25 ± 82%  numa-vmstat.node1.nr_inactive_anon
      5987 ± 11%     -19.4%       4825 ±  5%  numa-vmstat.node1.nr_kernel_stack
      2970 ±  2%     -15.9%       2498 ±  2%  numa-vmstat.node1.nr_mapped
    820.75 ± 12%     -44.8%     452.75 ± 35%  numa-vmstat.node1.nr_page_table_pages
     51894 ±  3%     -64.6%      18375 ± 11%  numa-vmstat.node1.nr_slab_reclaimable
     13978 ±  4%     -25.9%      10363 ±  8%  numa-vmstat.node1.nr_slab_unreclaimable
     13.75 ±110%   +1201.8%     179.00 ±  9%  numa-vmstat.node1.nr_writeback
     26503 ±168%     -99.1%     235.25 ± 82%  numa-vmstat.node1.nr_zone_inactive_anon
     19082 ±  2%     +61.0%      30724 ±  6%  numa-vmstat.node1.nr_zone_write_pending
      1363 ±  2%     -28.5%     975.25 ± 20%  slabinfo.UNIX.active_objs
      1363 ±  2%     -28.5%     975.25 ± 20%  slabinfo.UNIX.num_objs
      6207 ±  4%      -9.2%       5639        slabinfo.cred_jar.active_objs
      6207 ±  4%      -9.2%       5639        slabinfo.cred_jar.num_objs
   4071714           -99.9%       5808 ±  6%  slabinfo.ext4_io_end.active_objs
     63623           -99.9%      91.50 ±  6%  slabinfo.ext4_io_end.active_slabs
   4071916           -99.9%       5886 ±  6%  slabinfo.ext4_io_end.num_objs
     63623           -99.9%      91.50 ±  6%  slabinfo.ext4_io_end.num_slabs
    600.00 ± 10%     -18.3%     490.00 ±  3%  slabinfo.file_lock_cache.active_objs
    600.00 ± 10%     -18.3%     490.00 ±  3%  slabinfo.file_lock_cache.num_objs
    850.50 ±  8%     -21.0%     672.00 ±  6%  slabinfo.mnt_cache.active_objs
    850.50 ±  8%     -21.0%     672.00 ±  6%  slabinfo.mnt_cache.num_objs
      1139 ±  2%     -22.1%     887.75 ± 14%  slabinfo.nsproxy.active_objs
      1139 ±  2%     -22.1%     887.75 ± 14%  slabinfo.nsproxy.num_objs
      2990           -21.8%       2339 ± 16%  slabinfo.pool_workqueue.active_objs
      2991           -21.8%       2339 ± 16%  slabinfo.pool_workqueue.num_objs
    404.25 ± 14%     -24.0%     307.25 ± 11%  slabinfo.secpath_cache.active_objs
    404.25 ± 14%     -24.0%     307.25 ± 11%  slabinfo.secpath_cache.num_objs
    890.00 ±  3%     -21.1%     702.00 ± 12%  slabinfo.skbuff_fclone_cache.active_objs
    890.00 ±  3%     -21.1%     702.00 ± 12%  slabinfo.skbuff_fclone_cache.num_objs
      2565 ±  2%     -21.6%       2011 ± 12%  slabinfo.sock_inode_cache.active_objs
      2565 ±  2%     -21.6%       2011 ± 12%  slabinfo.sock_inode_cache.num_objs
    343.00 ±  2%     +12.6%     386.25 ±  4%  proc-vmstat.kswapd_low_wmark_hit_quickly
     95521 ± 25%     -18.9%      77469 ±  2%  proc-vmstat.nr_active_anon
     39667           +51.1%      59943 ±  3%  proc-vmstat.nr_dirty
     56291 ±159%     -93.2%       3816        proc-vmstat.nr_inactive_anon
      6927           -10.1%       6227        proc-vmstat.nr_mapped
      1356            -1.5%       1336        proc-vmstat.nr_page_table_pages
     76271 ±147%     -93.6%       4895        proc-vmstat.nr_shmem
    103211 ±  2%     -59.4%      41881 ±  2%  proc-vmstat.nr_slab_reclaimable
     27650 ±  6%      -7.0%      25702        proc-vmstat.nr_slab_unreclaimable
     25.75 ±115%   +1242.7%     345.75 ±  4%  proc-vmstat.nr_writeback
     95523 ± 25%     -18.9%      77470 ±  2%  proc-vmstat.nr_zone_active_anon
     56291 ±159%     -93.2%       3816        proc-vmstat.nr_zone_inactive_anon
     37490           +60.6%      60220 ±  3%  proc-vmstat.nr_zone_write_pending
    565177 ±  9%   +1112.8%    6854385 ± 37%  proc-vmstat.numa_foreign
  20889636           -31.0%   14423625 ± 17%  proc-vmstat.numa_hit
  20880137           -31.0%   14414150 ± 17%  proc-vmstat.numa_local
    565177 ±  9%   +1112.8%    6854385 ± 37%  proc-vmstat.numa_miss
    574675 ±  9%   +1094.4%    6863860 ± 37%  proc-vmstat.numa_other
     21071 ±  3%     -43.4%      11934        proc-vmstat.pgactivate
    239238           -12.6%     209196 ±  3%  proc-vmstat.pgfault
  10891256            -1.6%   10721631        proc-vmstat.pgfree
   5349612            -1.4%    5272402        proc-vmstat.pgscan_kswapd
   5339340            -1.4%    5262072        proc-vmstat.pgsteal_kswapd
  11982464           -99.7%      34464 ±  8%  proc-vmstat.slabs_scanned
 5.343e+11           -85.4%  7.817e+10 ± 25%  perf-stat.branch-instructions
      0.59 ±  8%      +2.3        2.93 ± 48%  perf-stat.branch-miss-rate%
     51.17           +11.5       62.62 ±  2%  perf-stat.cache-miss-rate%
 2.179e+09           -20.2%  1.739e+09 ±  8%  perf-stat.cache-misses
 4.259e+09           -34.9%  2.774e+09 ±  6%  perf-stat.cache-references
   2370742 ±  2%     -96.8%      77021 ±  3%  perf-stat.context-switches
      1.75           -20.5%       1.39 ± 14%  perf-stat.cpi
 4.909e+12           -89.9%  4.971e+11 ± 12%  perf-stat.cpu-cycles
    183130           -99.7%     629.75 ± 15%  perf-stat.cpu-migrations
      0.10 ± 20%      +0.2        0.29 ± 21%  perf-stat.dTLB-load-miss-rate%
 8.137e+08 ± 20%     -66.1%  2.762e+08 ±  2%  perf-stat.dTLB-load-misses
 8.484e+11           -88.3%  9.894e+10 ± 21%  perf-stat.dTLB-loads
      0.10 ± 20%      -0.0        0.05 ± 12%  perf-stat.dTLB-store-miss-rate%
  46384489 ± 20%     -50.9%   22757656 ±  7%  perf-stat.dTLB-store-misses
     56.39           +26.7       83.13        perf-stat.iTLB-load-miss-rate%
  51241211 ±  3%     -41.0%   30213199 ±  5%  perf-stat.iTLB-load-misses
  39625929 ±  3%     -84.5%    6141009 ±  8%  perf-stat.iTLB-loads
   2.8e+12           -86.8%  3.707e+11 ± 25%  perf-stat.instructions
     54709 ±  3%     -77.7%      12227 ± 24%  perf-stat.instructions-per-iTLB-miss
      0.57           +28.6%       0.73 ± 14%  perf-stat.ipc
    229749           -12.8%     200436 ±  3%  perf-stat.minor-faults
     39.56           -14.9       24.69 ± 22%  perf-stat.node-load-miss-rate%
   6.3e+08           -77.3%  1.431e+08 ± 27%  perf-stat.node-load-misses
 9.628e+08           -55.3%    4.3e+08 ±  7%  perf-stat.node-loads
    229759           -12.8%     200443 ±  3%  perf-stat.page-faults
     90.78           -85.3        5.44 ± 10%  perf-profile.calltrace.cycles-pp.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages
     91.40           -85.3        6.13 ± 10%  perf-profile.calltrace.cycles-pp.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages
     92.24           -84.5        7.76 ± 10%  perf-profile.calltrace.cycles-pp.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
     92.49           -84.3        8.21 ± 10%  perf-profile.calltrace.cycles-pp.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range
     92.80           -80.0       12.78 ±  9%  perf-profile.calltrace.cycles-pp.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
     92.80           -55.5       37.32 ±  6%  perf-profile.calltrace.cycles-pp.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file
     92.80           -55.5       37.32 ±  6%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.f2fs_do_sync_file.do_fsync.__x64_sys_fsync.do_syscall_64
     92.80           -55.5       37.32 ±  6%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.do_fsync.__x64_sys_fsync
     92.80           -55.5       37.32 ±  6%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.do_fsync
     92.90           -55.4       37.47 ±  5%  perf-profile.calltrace.cycles-pp.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
     92.90           -55.4       37.47 ±  5%  perf-profile.calltrace.cycles-pp.f2fs_do_sync_file.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
     92.90           -55.4       37.47 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
     55.19           -55.2        0.00        perf-profile.calltrace.cycles-pp.down_write.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page
     55.16           -55.2        0.00        perf-profile.calltrace.cycles-pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.f2fs_submit_page_write.do_write_page
     55.16           -55.2        0.00        perf-profile.calltrace.cycles-pp.call_rwsem_down_write_failed.down_write.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
     53.73           -53.7        0.00        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.f2fs_submit_page_write
     56.61           -52.2        4.45 ±  9%  perf-profile.calltrace.cycles-pp.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page
     94.91           -47.8       47.13 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     94.91           -47.7       47.17 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     33.80           -32.9        0.88 ± 17%  perf-profile.calltrace.cycles-pp.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page
     32.79           -32.8        0.00        perf-profile.calltrace.cycles-pp.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page
     31.91           -31.9        0.00        perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data
      0.00            +0.6        0.59 ±  9%  perf-profile.calltrace.cycles-pp.__test_set_page_writeback.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages
      0.00            +0.6        0.60 ± 11%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      0.00            +0.7        0.68 ± 15%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.f2fs_write_begin.generic_perform_write.__generic_file_write_iter
      0.00            +0.8        0.79 ± 16%  perf-profile.calltrace.cycles-pp.f2fs_map_blocks.f2fs_preallocate_blocks.f2fs_file_write_iter.__vfs_write.vfs_write
      0.00            +0.8        0.80 ±  8%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +0.8        0.81 ± 15%  perf-profile.calltrace.cycles-pp.f2fs_preallocate_blocks.f2fs_file_write_iter.__vfs_write.vfs_write.ksys_write
      0.00            +0.9        0.89 ±  7%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.00            +0.9        0.91 ± 17%  perf-profile.calltrace.cycles-pp.f2fs_set_data_page_dirty.f2fs_write_end.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
      0.00            +0.9        0.95 ±  9%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.f2fs_write_begin.generic_perform_write.__generic_file_write_iter
      0.00            +1.0        1.02 ± 20%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +1.1        1.13 ±  7%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.00            +1.2        1.17 ± 19%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.64 ± 20%      +1.3        1.92 ± 15%  perf-profile.calltrace.cycles-pp.f2fs_write_end.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write
      0.00            +1.3        1.30 ± 18%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +1.8        1.81 ± 11%  perf-profile.calltrace.cycles-pp.pagecache_get_page.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
      0.00            +1.9        1.89 ± 10%  perf-profile.calltrace.cycles-pp.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write
      0.00            +2.0        2.03 ± 10%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.00            +2.3        2.31 ± 14%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.00            +2.3        2.33 ± 10%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.00            +2.3        2.33 ±  6%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.09 ±  2%      +2.8        3.94 ±  9%  perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.__submit_merged_bio.f2fs_submit_page_write.do_write_page
      1.09 ±  2%      +2.8        3.94 ±  9%  perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.__submit_merged_bio.f2fs_submit_page_write
      1.09 ±  2%      +2.9        3.94 ±  9%  perf-profile.calltrace.cycles-pp.submit_bio.__submit_merged_bio.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
      1.09 ±  2%      +2.9        3.94 ±  9%  perf-profile.calltrace.cycles-pp.__submit_merged_bio.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page
      0.00            +3.0        3.01 ± 12%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.00            +3.1        3.11 ± 10%  perf-profile.calltrace.cycles-pp.clear_page_erms.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec
      0.00            +3.5        3.51 ±  8%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request
      0.00            +3.6        3.58 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request
      0.13 ±173%      +3.7        3.82 ±  8%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write
      0.00            +3.8        3.77 ±  8%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter
      0.00            +3.8        3.78 ±  8%  perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
      0.00            +3.9        3.87 ±  7%  perf-profile.calltrace.cycles-pp.submit_bio.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages
      0.00            +3.9        3.87 ±  7%  perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages
      0.00            +3.9        3.87 ±  7%  perf-profile.calltrace.cycles-pp.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
      0.00            +3.9        3.87 ±  7%  perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.__submit_merged_bio.__submit_merged_write_cond
      0.00            +3.9        3.88 ±  7%  perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range
      0.55 ±  6%      +4.3        4.81 ±  9%  perf-profile.calltrace.cycles-pp.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request.submit_bio
      0.00            +5.9        5.94 ± 10%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      0.00            +6.0        5.96 ± 10%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.92 ±  3%      +6.2        7.14 ±  8%  perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.__submit_merged_bio
      1.50 ± 10%      +6.4        7.90 ± 10%  perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write.vfs_write
      1.51 ± 10%      +6.4        7.95 ± 10%  perf-profile.calltrace.cycles-pp.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write.vfs_write.ksys_write
      1.84 ± 11%      +7.1        8.93 ± 10%  perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64
      1.85 ± 11%      +7.1        8.95 ± 10%  perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.87 ± 11%      +7.2        9.08 ± 10%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.88 ± 11%      +7.3        9.13 ± 10%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +12.0       12.04 ± 28%  perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range
      0.00           +12.4       12.45 ±  4%  perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range
      0.00           +24.5       24.50 ± 12%  perf-profile.calltrace.cycles-pp.__mutex_lock.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      3.82 ±  8%     +32.8       36.61 ±  4%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      3.95 ± 11%     +39.3       43.22 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      4.22 ± 11%     +42.2       46.47 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      4.22 ± 11%     +42.2       46.47 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      4.22 ± 11%     +42.2       46.47 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      4.22 ± 11%     +43.1       47.28 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     90.80           -85.3        5.46 ± 10%  perf-profile.children.cycles-pp.do_write_page
     91.41           -85.3        6.14 ± 10%  perf-profile.children.cycles-pp.f2fs_outplace_write_data
     92.24           -84.5        7.78 ± 10%  perf-profile.children.cycles-pp.f2fs_do_write_data_page
     92.50           -84.3        8.23 ± 10%  perf-profile.children.cycles-pp.__write_data_page
     92.80           -80.0       12.79 ±  9%  perf-profile.children.cycles-pp.f2fs_write_cache_pages
     85.66           -73.6       12.05 ± 28%  perf-profile.children.cycles-pp.osq_lock
     92.80           -55.5       37.32 ±  6%  perf-profile.children.cycles-pp.f2fs_write_data_pages
     92.80           -55.5       37.32 ±  6%  perf-profile.children.cycles-pp.file_write_and_wait_range
     92.80           -55.5       37.32 ±  6%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
     92.80           -55.5       37.32 ±  6%  perf-profile.children.cycles-pp.do_writepages
     92.91           -55.4       37.47 ±  5%  perf-profile.children.cycles-pp.f2fs_do_sync_file
     92.90           -55.4       37.47 ±  5%  perf-profile.children.cycles-pp.do_fsync
     92.90           -55.4       37.47 ±  5%  perf-profile.children.cycles-pp.__x64_sys_fsync
     55.18           -55.2        0.00        perf-profile.children.cycles-pp.rwsem_down_write_failed
     55.18           -55.2        0.00        perf-profile.children.cycles-pp.call_rwsem_down_write_failed
     55.32           -55.1        0.23 ± 11%  perf-profile.children.cycles-pp.down_write
     56.61           -52.1        4.47 ±  9%  perf-profile.children.cycles-pp.f2fs_submit_page_write
     95.03           -47.5       47.57 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
     95.03           -47.4       47.61 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     33.81           -32.9        0.91 ± 16%  perf-profile.children.cycles-pp.f2fs_allocate_data_block
     32.79            -8.3       24.51 ± 12%  perf-profile.children.cycles-pp.__mutex_lock
      0.38 ±  9%      -0.3        0.09 ± 11%  perf-profile.children.cycles-pp.try_to_wake_up
      0.49 ±  6%      -0.2        0.26 ± 16%  perf-profile.children.cycles-pp.up_write
      0.22 ± 11%      -0.2        0.07 ± 22%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.22 ±  9%      -0.1        0.07 ± 24%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.21 ± 11%      -0.1        0.07 ± 24%  perf-profile.children.cycles-pp.enqueue_entity
      0.17 ± 13%      -0.1        0.03 ±105%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.14 ± 15%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.save_stack_trace_tsk
      0.18 ± 12%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.__schedule
      0.32 ±  5%      -0.1        0.22 ±  9%  perf-profile.children.cycles-pp.f2fs_get_node_info
      0.30 ± 10%      -0.1        0.22 ± 14%  perf-profile.children.cycles-pp.down_read
      0.10 ±  8%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.f2fs_lookup_rb_tree_ret
      0.05            +0.0        0.07 ± 34%  perf-profile.children.cycles-pp.f2fs_create
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.read_node_page
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.hrtimer_forward
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.call_cpuidle
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.sched_avg_update
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.find_next_and_bit
      0.07 ± 17%      +0.1        0.14 ± 16%  perf-profile.children.cycles-pp.up_read
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.00            +0.1        0.07 ± 21%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.f2fs_set_node_page_dirty
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.menu_reflect
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.can_stop_idle_tick
      0.00            +0.1        0.07 ± 29%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.cpu_load_update_active
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.cpumask_next_and
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.update_group_capacity
      0.00            +0.1        0.08 ± 34%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.00            +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.__lookup_nat_cache
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.tag_pages_for_writeback
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.pm_qos_request
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.__intel_pmu_disable_all
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.08 ± 21%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.09 ± 19%  perf-profile.children.cycles-pp.alloc_pages_current
      0.00            +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.note_gp_changes
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.00            +0.1        0.09 ± 21%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.update_rq_clock
      0.06 ±  6%      +0.1        0.15 ± 11%  perf-profile.children.cycles-pp.do_sys_open
      0.06 ± 11%      +0.1        0.15 ±  9%  perf-profile.children.cycles-pp.do_filp_open
      0.06 ± 11%      +0.1        0.15 ±  9%  perf-profile.children.cycles-pp.path_openat
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.release_pages
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.00            +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.rcu_needs_cpu
      0.01 ±173%      +0.1        0.11 ± 27%  perf-profile.children.cycles-pp.handle_mm_fault
      0.01 ±173%      +0.1        0.11 ± 25%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.00            +0.1        0.10 ± 28%  perf-profile.children.cycles-pp.account_process_tick
      0.00            +0.1        0.10 ± 13%  perf-profile.children.cycles-pp.rcu_irq_exit
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.__radix_tree_replace
      0.00            +0.1        0.10 ± 39%  perf-profile.children.cycles-pp.__vfs_read
      0.00            +0.1        0.10 ± 20%  perf-profile.children.cycles-pp.f2fs_update_data_blkaddr
      0.00            +0.1        0.10 ± 20%  perf-profile.children.cycles-pp.f2fs_set_data_blkaddr
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.00            +0.1        0.10 ± 12%  perf-profile.children.cycles-pp._cond_resched
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.intel_pmu_disable_all
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.ksys_read
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.vfs_read
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.mem_cgroup_try_charge
      0.01 ±173%      +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.__do_page_fault
      0.01 ±173%      +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.page_fault
      0.01 ±173%      +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.do_page_fault
      0.00            +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.dec_zone_page_state
      0.06 ± 34%      +0.1        0.17 ± 32%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.00            +0.1        0.11 ± 22%  perf-profile.children.cycles-pp.set_page_dirty
      0.00            +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.00            +0.1        0.12 ±  5%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.ktime_get_real_ts64
      0.05 ±  8%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.1        0.13 ± 11%  perf-profile.children.cycles-pp.cpu_load_update
      0.01 ±173%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.radix_tree_tag_set
      0.00            +0.1        0.13        perf-profile.children.cycles-pp.node_dirty_ok
      0.07 ± 14%      +0.1        0.21 ± 21%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.1        0.14 ± 15%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.07 ±  7%      +0.1        0.20 ±  6%  perf-profile.children.cycles-pp.radix_tree_tag_clear
      0.00            +0.1        0.14 ± 16%  perf-profile.children.cycles-pp.get_cpu_device
      0.00            +0.1        0.14 ± 15%  perf-profile.children.cycles-pp.do_gettimeofday
      0.00            +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.__might_sleep
      0.00            +0.2        0.15 ± 14%  perf-profile.children.cycles-pp.native_sched_clock
      0.06 ±  6%      +0.2        0.22 ± 18%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.01 ±173%      +0.2        0.18 ±  8%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.01 ±173%      +0.2        0.18 ± 11%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.00            +0.2        0.17 ± 13%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.2        0.17 ± 12%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.2        0.17 ±  9%  perf-profile.children.cycles-pp.sched_clock
      0.15 ± 33%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.f2fs_reserve_new_blocks
      0.01 ±173%      +0.2        0.19 ± 20%  perf-profile.children.cycles-pp.account_page_dirtied
      0.00            +0.2        0.17 ±  2%  perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      0.00            +0.2        0.18 ± 26%  perf-profile.children.cycles-pp.__radix_tree_insert
      0.15 ± 63%      +0.2        0.33 ± 23%  perf-profile.children.cycles-pp.f2fs_mark_inode_dirty_sync
      0.00            +0.2        0.18 ± 14%  perf-profile.children.cycles-pp.__radix_tree_create
      0.00            +0.2        0.18 ± 17%  perf-profile.children.cycles-pp.page_mapping
      0.00            +0.2        0.18 ± 13%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.2        0.19 ± 11%  perf-profile.children.cycles-pp.__indirect_thunk_start
      0.00            +0.2        0.19 ±  9%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.19 ± 62%      +0.2        0.38 ± 23%  perf-profile.children.cycles-pp.f2fs_inode_dirtied
      0.00            +0.2        0.19 ± 11%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.2        0.20 ± 15%  perf-profile.children.cycles-pp.rcu_process_callbacks
      0.00            +0.2        0.20 ± 11%  perf-profile.children.cycles-pp.page_cache_tree_insert
      0.00            +0.2        0.20 ± 69%  perf-profile.children.cycles-pp.kthread
      0.00            +0.2        0.20 ± 65%  perf-profile.children.cycles-pp.ret_from_fork
      0.00            +0.2        0.21 ±  9%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.2        0.21 ±  8%  perf-profile.children.cycles-pp.read_tsc
      0.06 ±  9%      +0.2        0.27 ± 11%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.00            +0.2        0.22 ± 10%  perf-profile.children.cycles-pp.__x64_sys_gettimeofday
      0.00            +0.2        0.22 ± 11%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.2        0.22 ± 10%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.01 ±173%      +0.2        0.23 ± 11%  perf-profile.children.cycles-pp.brd_lookup_page
      0.03 ±173%      +0.2        0.25 ± 39%  perf-profile.children.cycles-pp.delay_tsc
      0.01 ±173%      +0.2        0.24 ± 21%  perf-profile.children.cycles-pp.radix_tree_lookup_slot
      0.04 ± 57%      +0.2        0.27 ± 11%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.07 ± 11%      +0.2        0.31 ±  9%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.00            +0.2        0.25 ± 11%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +0.3        0.25 ± 12%  perf-profile.children.cycles-pp.find_next_bit
      0.09            +0.3        0.35 ± 15%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      0.08 ± 10%      +0.3        0.34 ±  9%  perf-profile.children.cycles-pp.__lru_cache_add
      0.04 ± 57%      +0.3        0.31 ±  6%  perf-profile.children.cycles-pp.___might_sleep
      0.00            +0.3        0.29 ±  9%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +0.3        0.29 ± 10%  perf-profile.children.cycles-pp.native_write_msr
      0.00            +0.3        0.30 ± 25%  perf-profile.children.cycles-pp.rcu_check_callbacks
      0.06 ± 11%      +0.3        0.37 ± 16%  perf-profile.children.cycles-pp.find_get_entry
      0.03 ±100%      +0.3        0.37 ± 15%  perf-profile.children.cycles-pp.__entry_trampoline_start
      0.06 ± 14%      +0.3        0.40 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.00            +0.4        0.35 ±  9%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.4        0.37 ± 13%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.4        0.37 ±  5%  perf-profile.children.cycles-pp.run_timer_softirq
      0.12 ±  6%      +0.4        0.49 ±  5%  perf-profile.children.cycles-pp.test_clear_page_writeback
      0.12 ±  7%      +0.4        0.49 ±  5%  perf-profile.children.cycles-pp.end_page_writeback
      0.00            +0.4        0.38 ±  7%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.4        0.38 ±  9%  perf-profile.children.cycles-pp.irq_enter
      0.12 ± 35%      +0.4        0.51 ± 38%  perf-profile.children.cycles-pp.io_serial_in
      0.10 ± 25%      +0.4        0.50 ± 19%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.10 ± 25%      +0.4        0.50 ± 19%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.10 ± 25%      +0.4        0.50 ± 19%  perf-profile.children.cycles-pp.irq_work_run
      0.10 ± 25%      +0.4        0.50 ± 19%  perf-profile.children.cycles-pp.printk
      0.00            +0.4        0.41 ± 13%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.10 ± 11%      +0.4        0.53 ± 13%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.00            +0.4        0.44 ± 11%  perf-profile.children.cycles-pp.find_busiest_group
      0.15 ±  5%      +0.5        0.61 ± 15%  perf-profile.children.cycles-pp.__get_node_page
      0.18 ±  3%      +0.5        0.64 ±  7%  perf-profile.children.cycles-pp.f2fs_write_end_io
      0.10 ±  8%      +0.5        0.56 ±  9%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      0.12 ± 16%      +0.5        0.58 ± 18%  perf-profile.children.cycles-pp.irq_work_run_list
      0.12 ±  4%      +0.5        0.61 ± 11%  perf-profile.children.cycles-pp.scheduler_tick
      0.15 ±  3%      +0.5        0.64 ±  9%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.30 ± 15%      +0.5        0.79 ± 16%  perf-profile.children.cycles-pp.f2fs_map_blocks
      0.22 ±  3%      +0.5        0.71 ± 12%  perf-profile.children.cycles-pp.f2fs_get_dnode_of_data
      0.03 ±100%      +0.5        0.52 ± 14%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.10 ± 12%      +0.5        0.60 ± 13%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.00            +0.5        0.50 ± 16%  perf-profile.children.cycles-pp.update_blocked_averages
      0.30 ± 15%      +0.5        0.81 ± 15%  perf-profile.children.cycles-pp.f2fs_preallocate_blocks
      0.40 ± 10%      +0.5        0.91 ± 17%  perf-profile.children.cycles-pp.f2fs_set_data_page_dirty
      0.00            +0.5        0.52 ± 18%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.17 ± 44%      +0.5        0.70 ± 38%  perf-profile.children.cycles-pp.vprintk_emit
      0.16 ± 39%      +0.5        0.70 ± 34%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.16 ± 39%      +0.5        0.70 ± 34%  perf-profile.children.cycles-pp.uart_console_write
      0.16 ± 41%      +0.6        0.71 ± 35%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.16 ± 41%      +0.6        0.72 ± 36%  perf-profile.children.cycles-pp.serial8250_console_write
      0.00            +0.6        0.56 ± 10%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.18 ± 45%      +0.6        0.76 ± 37%  perf-profile.children.cycles-pp.console_unlock
      0.00            +0.6        0.62 ±  8%  perf-profile.children.cycles-pp.load_balance
      0.18 ±  6%      +0.8        0.97 ±  9%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      0.00            +0.8        0.83 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.9        0.94 ±  5%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.90 ± 15%      +1.0        1.93 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock
      0.17 ±  2%      +1.1        1.23 ± 18%  perf-profile.children.cycles-pp.update_process_times
      0.01 ±173%      +1.2        1.17 ±  6%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.17 ±  2%      +1.2        1.37 ± 17%  perf-profile.children.cycles-pp.tick_sched_handle
      0.64 ± 20%      +1.3        1.95 ± 15%  perf-profile.children.cycles-pp.f2fs_write_end
      0.19 ±  3%      +1.4        1.54 ± 16%  perf-profile.children.cycles-pp.tick_sched_timer
      0.33 ±  6%      +1.6        1.94 ± 10%  perf-profile.children.cycles-pp.f2fs_write_begin
      0.40 ±  4%      +1.9        2.34 ± 11%  perf-profile.children.cycles-pp.pagecache_get_page
      0.09 ± 39%      +2.0        2.12 ±  8%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.10 ± 35%      +2.3        2.42 ±  8%  perf-profile.children.cycles-pp.irq_exit
      0.04 ±110%      +2.4        2.40 ±  6%  perf-profile.children.cycles-pp.menu_select
      0.27 ±  3%      +2.4        2.62 ± 13%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.40 ±  5%      +2.8        3.18 ± 10%  perf-profile.children.cycles-pp.clear_page_erms
      0.33 ±  5%      +3.0        3.36 ± 11%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.48 ±  2%      +3.3        3.79 ±  8%  perf-profile.children.cycles-pp.copyin
      0.49 ±  2%      +3.3        3.82 ±  8%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      0.48 ±  2%      +3.3        3.81 ±  8%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.58 ±  4%      +3.6        4.18 ±  9%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.61 ±  5%      +3.8        4.37 ±  8%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.13 ±  6%      +3.8        3.91 ±  7%  perf-profile.children.cycles-pp.__submit_merged_write_cond
      0.61 ±  5%      +4.2        4.84 ±  9%  perf-profile.children.cycles-pp.brd_insert_page
      0.45 ± 11%      +5.9        6.37 ±  8%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.46 ± 11%      +5.9        6.40 ±  8%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.02 ±  3%      +6.2        7.18 ±  8%  perf-profile.children.cycles-pp.brd_do_bvec
      1.53 ±  9%      +6.4        7.93 ± 10%  perf-profile.children.cycles-pp.generic_perform_write
      1.52 ±  9%      +6.4        7.95 ± 10%  perf-profile.children.cycles-pp.__generic_file_write_iter
      1.20 ±  2%      +6.6        7.83 ±  8%  perf-profile.children.cycles-pp.brd_make_request
      1.21 ±  2%      +6.6        7.85 ±  8%  perf-profile.children.cycles-pp.submit_bio
      1.21 ±  2%      +6.6        7.85 ±  8%  perf-profile.children.cycles-pp.generic_make_request
      1.21 ±  2%      +6.6        7.85 ±  8%  perf-profile.children.cycles-pp.__submit_merged_bio
      1.84 ± 11%      +7.1        8.94 ± 10%  perf-profile.children.cycles-pp.f2fs_file_write_iter
      1.92 ± 10%      +7.2        9.16 ± 12%  perf-profile.children.cycles-pp.__vfs_write
      1.94 ± 10%      +7.4        9.29 ± 12%  perf-profile.children.cycles-pp.vfs_write
      1.95 ± 10%      +7.4        9.35 ± 12%  perf-profile.children.cycles-pp.ksys_write
      0.64 ±  3%     +11.8       12.46 ±  4%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      3.82 ±  8%     +33.5       37.34 ±  4%  perf-profile.children.cycles-pp.intel_idle
      3.96 ± 11%     +40.2       44.20 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter_state
      4.22 ± 11%     +42.2       46.47 ±  5%  perf-profile.children.cycles-pp.start_secondary
      4.22 ± 11%     +43.1       47.28 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64
      4.22 ± 11%     +43.1       47.28 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
      4.23 ± 11%     +43.1       47.35 ±  4%  perf-profile.children.cycles-pp.do_idle
     85.30           -73.3       12.02 ± 28%  perf-profile.self.cycles-pp.osq_lock
      0.14 ±  6%      -0.1        0.08 ± 21%  perf-profile.self.cycles-pp.mutex_lock
      0.08 ± 10%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.f2fs_lookup_rb_tree_ret
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.f2fs_inode_dirtied
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.menu_reflect
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.read_node_page
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.hrtimer_forward
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.rcu_process_callbacks
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.call_cpuidle
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.sched_avg_update
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.find_next_and_bit
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
      0.08 ± 10%      +0.1        0.15 ± 15%  perf-profile.self.cycles-pp.f2fs_update_dirty_page
      0.07 ± 12%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.up_read
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.07 ± 26%  perf-profile.self.cycles-pp.account_page_dirtied
      0.00            +0.1        0.07 ± 22%  perf-profile.self.cycles-pp.can_stop_idle_tick
      0.00            +0.1        0.07 ± 28%  perf-profile.self.cycles-pp.ktime_get_real_ts64
      0.00            +0.1        0.07 ± 29%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.00            +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.update_group_capacity
      0.00            +0.1        0.08 ± 24%  perf-profile.self.cycles-pp._cond_resched
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.node_dirty_ok
      0.00            +0.1        0.08 ± 34%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.clockevents_program_event
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.00            +0.1        0.08 ± 31%  perf-profile.self.cycles-pp.add_to_page_cache_lru
      0.00            +0.1        0.08 ± 23%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.pm_qos_request
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.release_pages
      0.00            +0.1        0.09 ± 30%  perf-profile.self.cycles-pp.f2fs_file_write_iter
      0.00            +0.1        0.09 ± 21%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.04 ± 57%      +0.1        0.13 ± 23%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.1        0.09 ± 24%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.00            +0.1        0.09 ± 22%  perf-profile.self.cycles-pp.f2fs_get_dnode_of_data
      0.00            +0.1        0.09 ± 15%  perf-profile.self.cycles-pp.rcu_needs_cpu
      0.00            +0.1        0.09 ± 14%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.00            +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.f2fs_write_end_io
      0.00            +0.1        0.10 ± 28%  perf-profile.self.cycles-pp.account_process_tick
      0.00            +0.1        0.10 ± 33%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.00            +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.08 ± 10%      +0.1        0.18 ± 10%  perf-profile.self.cycles-pp.__write_data_page
      0.00            +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.__set_page_dirty_nobuffers
      0.00            +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.f2fs_write_cache_pages
      0.00            +0.1        0.11 ± 17%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.1        0.11 ±  6%  perf-profile.self.cycles-pp.dec_zone_page_state
      0.00            +0.1        0.11 ± 15%  perf-profile.self.cycles-pp.f2fs_map_blocks
      0.00            +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.04 ± 57%      +0.1        0.15 ± 10%  perf-profile.self.cycles-pp.radix_tree_tag_clear
      0.00            +0.1        0.12 ±  7%  perf-profile.self.cycles-pp.f2fs_update_extent_tree_range
      0.00            +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.00            +0.1        0.12 ±  5%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.12 ± 18%  perf-profile.self.cycles-pp.f2fs_write_begin
      0.06 ±  7%      +0.1        0.18 ± 18%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.06 ± 11%      +0.1        0.18 ± 19%  perf-profile.self.cycles-pp.f2fs_do_write_data_page
      0.00            +0.1        0.13 ± 11%  perf-profile.self.cycles-pp.cpu_load_update
      0.01 ±173%      +0.1        0.14 ±  8%  perf-profile.self.cycles-pp.radix_tree_tag_set
      0.00            +0.1        0.13 ± 13%  perf-profile.self.cycles-pp.load_balance
      0.00            +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.1        0.14 ± 16%  perf-profile.self.cycles-pp.get_cpu_device
      0.00            +0.1        0.14 ±  9%  perf-profile.self.cycles-pp.find_get_entry
      0.00            +0.1        0.14 ± 19%  perf-profile.self.cycles-pp.rebalance_domains
      0.00            +0.1        0.14 ± 10%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.00            +0.1        0.14 ± 12%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.1        0.14 ± 18%  perf-profile.self.cycles-pp.__radix_tree_create
      0.00            +0.1        0.15 ±  7%  perf-profile.self.cycles-pp.__might_sleep
      0.07 ±  7%      +0.1        0.21 ± 19%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.00            +0.1        0.15 ± 20%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.2        0.15 ± 15%  perf-profile.self.cycles-pp.do_syscall_64
      0.00            +0.2        0.15 ± 14%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.2        0.16 ±  5%  perf-profile.self.cycles-pp.irq_exit
      0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.07 ±  5%      +0.2        0.23 ± 20%  perf-profile.self.cycles-pp.f2fs_submit_page_write
      0.00            +0.2        0.17 ± 21%  perf-profile.self.cycles-pp.pagecache_get_page
      0.00            +0.2        0.17 ±  2%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      0.00            +0.2        0.18 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.18 ±  9%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +0.2        0.18 ± 13%  perf-profile.self.cycles-pp.test_clear_page_writeback
      0.06 ±  6%      +0.2        0.24 ± 17%  perf-profile.self.cycles-pp.up_write
      0.03 ±100%      +0.2        0.21 ±  8%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.00            +0.2        0.18 ± 10%  perf-profile.self.cycles-pp.__indirect_thunk_start
      0.00            +0.2        0.18 ± 17%  perf-profile.self.cycles-pp.page_mapping
      0.04 ± 58%      +0.2        0.23 ± 13%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.00            +0.2        0.21 ±  8%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.2        0.21 ±  9%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.2        0.22 ± 13%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.00            +0.2        0.22 ±  3%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.00            +0.2        0.22 ± 10%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.00            +0.2        0.22 ± 19%  perf-profile.self.cycles-pp.ktime_get
      0.03 ±173%      +0.2        0.25 ± 39%  perf-profile.self.cycles-pp.delay_tsc
      0.10 ±  4%      +0.2        0.34 ± 13%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.2        0.25 ± 11%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.3        0.25 ± 12%  perf-profile.self.cycles-pp.find_next_bit
      0.00            +0.3        0.26 ±  4%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.3        0.26 ± 25%  perf-profile.self.cycles-pp.rcu_check_callbacks
      0.00            +0.3        0.27 ±  6%  perf-profile.self.cycles-pp.run_timer_softirq
      0.00            +0.3        0.27 ± 12%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.04 ± 57%      +0.3        0.31 ±  6%  perf-profile.self.cycles-pp.___might_sleep
      0.00            +0.3        0.29 ± 10%  perf-profile.self.cycles-pp.native_write_msr
      0.00            +0.3        0.29 ± 17%  perf-profile.self.cycles-pp.update_blocked_averages
      0.00            +0.3        0.30 ± 11%  perf-profile.self.cycles-pp.find_busiest_group
      0.03 ±100%      +0.3        0.37 ± 15%  perf-profile.self.cycles-pp.__entry_trampoline_start
      0.06 ± 14%      +0.3        0.40 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.12 ± 35%      +0.4        0.51 ± 38%  perf-profile.self.cycles-pp.io_serial_in
      0.10 ± 11%      +0.4        0.52 ± 12%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.00            +0.4        0.44 ± 10%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.11 ±  4%      +0.5        0.66 ±  7%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.10 ±  8%      +0.8        0.91 ± 14%  perf-profile.self.cycles-pp.f2fs_write_end
      0.01 ±173%      +0.9        0.87 ±  8%  perf-profile.self.cycles-pp.menu_select
      0.39 ±  3%      +1.5        1.90 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock
      0.37            +1.7        2.09 ±  6%  perf-profile.self.cycles-pp.brd_do_bvec
      0.40 ±  5%      +2.7        3.14 ± 11%  perf-profile.self.cycles-pp.clear_page_erms
      0.48 ±  2%      +3.3        3.79 ±  8%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.64 ±  3%     +11.7       12.36 ±  5%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      3.82 ±  8%     +33.5       37.29 ±  4%  perf-profile.self.cycles-pp.intel_idle


                                                                                
                              fsmark.time.system_time                           
                                                                                
  1800 +-+------------------------------------------------------------------+   
       |.+.+                                                                |   
  1600 +-+ :     +.++.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.|   
  1400 +-+ :     :                                                          |   
       |   :     :                                                          |   
  1200 +-+  :    :                                                          |   
  1000 +-+  :   :                                                           |   
       |    :   :                                                           |   
   800 +-+  :   :                                                           |   
   600 +-+  :   :                                                           |   
       |    :   :                                                           |   
   400 +-+   :  :                                                           |   
   200 +-+   : :                                                            |   
       O O O : : O OO O O O O O O O O O O OO O O O                          |   
     0 +-+---O-O------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      fsmark.time.percent_of_cpu_this_job_got                   
                                                                                
  4000 +-+------------------------------------------------------------------+   
       |.+ :     +  +.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+          +.+   + |   
  3500 +-+ :     :                                                          |   
  3000 +-+ :     :                                                          |   
       |    :    :                                                          |   
  2500 +-+  :   :                                                           |   
       |    :   :                                                           |   
  2000 +-+  :   :                                                           |   
       |    :   :                                                           |   
  1500 +-+  :   :                                                           |   
  1000 +-+   :  :                                                           |   
       |     : :                                                            |   
   500 +-+   : :                                                            |   
       O O O O.O O OO O O O O O O O O O O OO O O O                          |   
     0 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            fsmark.time.elapsed_time                            
                                                                                
  45 +-+--------------------------------------------------------------------+   
     |.+.+        .+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.|   
  40 +-+ :     +.+                                                          |   
  35 +-+ :     :     O O O                                                  |   
     O O O     O O O       O O O O O O O OO O O O                           |   
  30 +-+  :    :                                                            |   
  25 +-+  :   :                                                             |   
     |    :   :                                                             |   
  20 +-+  :   :                                                             |   
  15 +-+  :   :                                                             |   
     |    :   :                                                             |   
  10 +-+   :  :                                                             |   
   5 +-+   : :                                                              |   
     |     : :                                                              |   
   0 +-+---O-O--------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          fsmark.time.elapsed_time.max                          
                                                                                
  45 +-+--------------------------------------------------------------------+   
     |.+.+        .+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.|   
  40 +-+ :     +.+                                                          |   
  35 +-+ :     :     O O O                                                  |   
     O O O     O O O       O O O O O O O OO O O O                           |   
  30 +-+  :    :                                                            |   
  25 +-+  :   :                                                             |   
     |    :   :                                                             |   
  20 +-+  :   :                                                             |   
  15 +-+  :   :                                                             |   
     |    :   :                                                             |   
  10 +-+   :  :                                                             |   
   5 +-+   : :                                                              |   
     |     : :                                                              |   
   0 +-+---O-O--------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            fsmark.time.minor_page_faults                       
                                                                                
  92000 +-+-----------------------------------------------------------------+   
  90000 +-+.+                                                               |   
        |   :    +.+                                                        |   
  88000 +-+ :    : :                                                        |   
  86000 +-+  :   :  :                                                       |   
  84000 +-+  :   :  :                                                       |   
  82000 +-+  :   :  :                .+                                     |   
        |    :   :   :   +.   .+.++.+  + .+.          .+.           +   +.  |   
  80000 +-+  :  :    +. +  +.+          +   +.+.+.++.+   +.+.+.+. .+ + +  +.|   
  78000 +-+  :  :      +                                         +    +     |   
  76000 O-+   : :                                                           |   
  74000 +-O O : :O O   O   O O O  O O O O O O   O                           |   
        |     : :    O   O       O            O   O                         |   
  72000 +-+   O.O                                                           |   
  70000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        fsmark.time.voluntary_context_switches                  
                                                                                
  1.6e+06 +-+---------------------------------------------------------------+   
          |                 +.+.+.+.   +                                    |   
  1.4e+06 +-+              +        +.+ + .+.+. .+                  .+.     |   
          |               +              +     +  +.+. .+. .++. .+.+   ++.+.|   
  1.2e+06 +-+.+    +.+.+.+                            +   +    +            |   
    1e+06 +-+ :    :                                                        |   
          |   :    :                                                        |   
   800000 +-+  :  :                                                         |   
          |    :  :                                                         |   
   600000 +-+  :  :                                                         |   
   400000 +-+  :  :                                                         |   
          |    :  :                                                         |   
   200000 +-+   ::                                                          |   
          |     ::                                                          |   
        0 O-O-O-OO-O-O-O-O-OO-O-O-O-O-OO-O-O-O-O-OO-------------------------+   
                                                                                
                                                                                                                                                                
                      fsmark.time.involuntary_context_switches                  
                                                                                
  30000 +-+-----------------------------------------------------------------+   
        |                                          +.     .+.+.+.+.++.+.+.+.|   
  25000 +-+.+    +.+.+.+                           : +.+.+                  |   
        |   :    :      +               +.    +   +                         |   
        |   :    :       +              : +. + + +                          |   
  20000 +-+  :   :        +       +.+  :    +   +                           |   
        |    :   :         +.+.+.+   + :                                    |   
  15000 +-+  :   :                    +                                     |   
        |    :   :                                                          |   
  10000 +-+  :  :                                                           |   
        |    :  :                                                           |   
        |     : :                                                           |   
   5000 +-+   : :                                                           |   
        |     : :                                                           |   
      0 O-O-O-O-OO-O-O-O-O-O-O-O-OO-O-O-O-O-O-O-O-O-------------------------+   
                                                                                
                                                                                                                                                                
                               fsmark.files_per_sec                             
                                                                                
  400 +-+-------------------------------------------------------------------+   
      O O O     O O          O   O O O O O O O O                            |   
  350 +-+           O O  O O   O                 O                          |   
  300 +-+               O                                                   |   
      |         +.+.                                                        |   
  250 +-+.+     :   +.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.|   
      |   :     :                                                           |   
  200 +-+ :    :                                                            |   
      |    :   :                                                            |   
  150 +-+  :   :                                                            |   
  100 +-+  :   :                                                            |   
      |    :   :                                                            |   
   50 +-+   : :                                                             |   
      |     : :                                                             |   
    0 +-+---O-O-------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen

View attachment "config-4.18.0-rc3-00276-g853137c" of type "text/plain" (166063 bytes)

View attachment "job-script" of type "text/plain" (7643 bytes)

View attachment "job.yaml" of type "text/plain" (5234 bytes)

View attachment "reproduce" of type "text/plain" (1707 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ