lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200616074222.GJ5653@shao2-debian>
Date:   Tue, 16 Jun 2020 15:42:22 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Chao Yu <yuchao0@...wei.com>
Cc:     Jaegeuk Kim <jaegeuk@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [f2fs] ca7f76e680: aim7.jobs-per-min 41.5% improvement

Greeting,

FYI, we noticed a 41.5% improvement of aim7.jobs-per-min due to commit:


commit: ca7f76e680745d3b8a386638045f85dac1c4b2f4 ("f2fs: fix wrong discard space")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: aim7
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: f2fs
	test: sync_disk_rw
	load: 100
	cpufreq_governor: performance
	ucode: 0x42e

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/4BRD_12G/f2fs/x86_64-rhel-7.6/100/RAID0/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e

commit: 
  dc35d73a42 ("f2fs: compress: don't compress any datas after cp stop")
  ca7f76e680 ("f2fs: fix wrong discard space")

dc35d73a42291b2a ca7f76e680745d3b8a386638045 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          2:4          -50%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      7998           +41.5%      11320        aim7.jobs-per-min
     75.06           -29.3%      53.04        aim7.time.elapsed_time
     75.06           -29.3%      53.04        aim7.time.elapsed_time.max
  85958910            -6.6%   80295428        aim7.time.file_system_outputs
    602222            +1.6%     611895        aim7.time.involuntary_context_switches
     10270 ±  3%     -14.9%       8743 ±  2%  aim7.time.minor_page_faults
    890.41           -35.4%     575.08        aim7.time.system_time
     10.52           -18.2%       8.61        aim7.time.user_time
  20399073           -59.1%    8346573        aim7.time.voluntary_context_switches
      0.58 ±  3%      +0.2        0.75        mpstat.cpu.all.usr%
     23.66            +2.5%      24.25        boot-time.boot
    950.90            +2.6%     975.47        boot-time.idle
   5501204            -9.7%    4966671        numa-numastat.node0.local_node
   5503559            -9.7%    4970571        numa-numastat.node0.numa_hit
    548367           +30.7%     716543        vmstat.io.bo
    586187           -25.6%     435987        vmstat.system.cs
     23.21            -3.6%      22.38        iostat.cpu.system
     90248           +77.3%     159991        iostat.md0.w/s
    577606           +25.9%     727269        iostat.md0.wkB/s
     71559 ±  3%     -21.0%      56538        meminfo.AnonHugePages
    116092            -9.0%     105645        meminfo.Inactive
     99685           -10.8%      88920        meminfo.Inactive(file)
  17955986           -12.3%   15739104        meminfo.Memused
     32249 ±  2%     -11.4%      28568        meminfo.Shmem
    510766           +29.6%     661837        meminfo.max_used_kB
     48979 ±  4%     -13.6%      42299 ±  3%  numa-meminfo.node0.Inactive(file)
     52129 ±  2%     -14.3%      44685 ±  2%  numa-meminfo.node0.KReclaimable
   8981460           -13.4%    7780127        numa-meminfo.node0.MemUsed
     52129 ±  2%     -14.3%      44685 ±  2%  numa-meminfo.node0.SReclaimable
    106708 ±  5%      -7.5%      98756 ±  2%  numa-meminfo.node0.Slab
   8924236           -11.7%    7884151 ±  2%  numa-meminfo.node1.MemUsed
 1.134e+09 ±  6%     -55.8%  5.013e+08        cpuidle.C1.time
  16571069 ±  6%     -51.2%    8080515        cpuidle.C1.usage
 9.877e+08 ± 13%     -31.4%  6.775e+08 ±  5%  cpuidle.C1E.time
   8259195 ± 12%     -59.7%    3332136 ± 10%  cpuidle.C1E.usage
 5.364e+08 ± 19%     +38.8%  7.447e+08 ±  5%  cpuidle.C6.time
   1073451 ± 41%     +54.6%    1659694 ± 24%  cpuidle.C6.usage
    415787           -54.4%     189765        cpuidle.POLL.time
     67117 ±  2%     -54.8%      30351        cpuidle.POLL.usage
      1536 ± 13%     +29.2%       1984 ± 10%  slabinfo.fsnotify_mark_connector.active_objs
      1536 ± 13%     +29.2%       1984 ± 10%  slabinfo.fsnotify_mark_connector.num_objs
    464.00 ±  5%     +37.9%     640.00 ±  7%  slabinfo.kmalloc-rcl-128.active_objs
    464.00 ±  5%     +37.9%     640.00 ±  7%  slabinfo.kmalloc-rcl-128.num_objs
      1092 ±  4%     +14.4%       1249 ±  6%  slabinfo.kmalloc-rcl-96.active_objs
      1092 ±  4%     +14.4%       1249 ±  6%  slabinfo.kmalloc-rcl-96.num_objs
     78622           -11.2%      69811        slabinfo.radix_tree_node.active_objs
      1403           -11.2%       1246        slabinfo.radix_tree_node.active_slabs
     78622           -11.2%      69811        slabinfo.radix_tree_node.num_objs
      1403           -11.2%       1246        slabinfo.radix_tree_node.num_slabs
   2285879           -14.4%    1955601        numa-vmstat.node0.nr_dirtied
     12271 ±  5%     -13.1%      10660 ±  3%  numa-vmstat.node0.nr_inactive_file
     13032 ±  2%     -14.4%      11156        numa-vmstat.node0.nr_slab_reclaimable
   2272917           -14.5%    1943710        numa-vmstat.node0.nr_written
     12271 ±  5%     -13.1%      10661 ±  3%  numa-vmstat.node0.nr_zone_inactive_file
   2758058           -12.5%    2413771        numa-vmstat.node0.numa_hit
   2601941           -13.3%    2256178 ±  2%  numa-vmstat.node0.numa_local
   2300478           -13.1%    1999423        numa-vmstat.node1.nr_dirtied
   2287494           -13.1%    1987272        numa-vmstat.node1.nr_written
   2793838           -11.4%    2475375        numa-vmstat.node1.numa_hit
   2765728           -11.5%    2448996        numa-vmstat.node1.numa_local
     72.97 ± 40%     +45.1%     105.91 ± 33%  sched_debug.cfs_rq:/.load_avg.avg
    780.25 ± 21%     +88.7%       1472 ± 30%  sched_debug.cfs_rq:/.load_avg.max
    182.33 ± 35%     +62.1%     295.54 ± 19%  sched_debug.cfs_rq:/.load_avg.stddev
     91231 ± 54%     -95.6%       4001 ± 18%  sched_debug.cfs_rq:/.min_vruntime.min
    992.05 ± 21%     +43.8%       1426 ±  6%  sched_debug.cfs_rq:/.runnable_avg.avg
      3045 ± 15%     +46.2%       4452 ± 13%  sched_debug.cfs_rq:/.runnable_avg.max
    702.16 ± 20%     +50.0%       1053 ± 13%  sched_debug.cfs_rq:/.runnable_avg.stddev
    425.23 ±  8%     +15.8%     492.35 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
      1031           +34.0%       1382 ± 18%  sched_debug.cfs_rq:/.util_avg.max
    262.75 ±  9%     +34.0%     351.99 ±  6%  sched_debug.cfs_rq:/.util_avg.stddev
     53.53 ± 15%     -22.0%      41.78 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     17200 ± 44%     -89.1%       1868 ± 17%  sched_debug.cpu.avg_idle.min
      0.00 ± 15%     +82.9%       0.00 ± 48%  sched_debug.cpu.next_balance.stddev
     83.50 ± 23%     -42.5%      48.00 ± 21%  sched_debug.cpu.nr_uninterruptible.max
     21.46 ± 19%     -35.0%      13.95 ± 13%  sched_debug.cpu.nr_uninterruptible.stddev
     67893            -1.7%      66731        proc-vmstat.nr_active_anon
  10691167            -7.2%    9925885        proc-vmstat.nr_dirtied
     12978            -7.5%      12003 ±  3%  proc-vmstat.nr_dirty
   1190397            +4.7%    1246050        proc-vmstat.nr_dirty_background_threshold
   2383706            +4.7%    2495147        proc-vmstat.nr_dirty_threshold
    332216            -1.1%     328636        proc-vmstat.nr_file_pages
  11982444            +4.7%   12542388        proc-vmstat.nr_free_pages
      4102            +2.2%       4191        proc-vmstat.nr_inactive_anon
     24952           -10.4%      22354        proc-vmstat.nr_inactive_file
      7052            +3.0%       7263        proc-vmstat.nr_mapped
      2025 ±  2%      -3.0%       1964        proc-vmstat.nr_page_table_pages
      8055 ±  2%     -11.4%       7139        proc-vmstat.nr_shmem
     25569            -5.7%      24121        proc-vmstat.nr_slab_reclaimable
     60.75 ± 23%    +100.0%     121.50 ± 33%  proc-vmstat.nr_writeback
  10635151            -7.2%    9871766        proc-vmstat.nr_written
     67893            -1.7%      66731        proc-vmstat.nr_zone_active_anon
      4102            +2.2%       4191        proc-vmstat.nr_zone_inactive_anon
     24952           -10.4%      22354        proc-vmstat.nr_zone_inactive_file
  10793514            -7.3%   10004830        proc-vmstat.numa_hit
  10777815            -7.3%    9989143        proc-vmstat.numa_local
      3895 ± 74%     -68.1%       1244 ± 32%  proc-vmstat.numa_pages_migrated
      7023 ±  2%     -30.8%       4861        proc-vmstat.pgactivate
  10928413            -7.5%   10104541        proc-vmstat.pgalloc_normal
      9654           -10.4%       8651        proc-vmstat.pgdeactivate
    237080 ±  2%     -25.6%     176491 ±  3%  proc-vmstat.pgfault
   1640892            -3.9%    1576599        proc-vmstat.pgfree
      3895 ± 74%     -68.1%       1244 ± 32%  proc-vmstat.pgmigrate_success
  42552033            -7.2%   39495741        proc-vmstat.pgpgout
     19298           -10.8%      17219        proc-vmstat.pgrotated
     13.90 ±  4%      +7.8%      14.99 ±  3%  perf-stat.i.MPKI
 4.317e+09            -6.7%  4.029e+09        perf-stat.i.branch-instructions
      1.95 ±  3%      +0.6        2.52 ±  2%  perf-stat.i.branch-miss-rate%
  45918409 ±  2%      +7.0%   49118114 ±  3%  perf-stat.i.branch-misses
     17.13            +2.5       19.60        perf-stat.i.cache-miss-rate%
  39926130            +9.6%   43744236        perf-stat.i.cache-misses
 2.523e+08            -5.2%  2.392e+08        perf-stat.i.cache-references
    599901           -25.7%     445546        perf-stat.i.context-switches
      2.26 ±  3%      +6.2%       2.40        perf-stat.i.cpi
 3.977e+10            -5.9%  3.741e+10        perf-stat.i.cpu-cycles
     48176           -72.7%      13133        perf-stat.i.cpu-migrations
      1009           -12.7%     880.94 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.64            -0.0        0.61 ±  3%  perf-stat.i.dTLB-load-miss-rate%
  33152560           -10.8%   29565873 ±  3%  perf-stat.i.dTLB-load-misses
 4.966e+09            -5.5%  4.693e+09        perf-stat.i.dTLB-loads
      0.18 ±  4%      -0.0        0.15 ±  2%  perf-stat.i.dTLB-store-miss-rate%
   2378797 ±  4%     -12.1%    2091893 ±  3%  perf-stat.i.dTLB-store-misses
 1.317e+09            +3.0%  1.357e+09        perf-stat.i.dTLB-stores
     68.47           -15.1       53.42        perf-stat.i.iTLB-load-miss-rate%
   8216629           -42.9%    4691920        perf-stat.i.iTLB-load-misses
   4301064           +30.6%    5615809 ±  3%  perf-stat.i.iTLB-loads
 1.981e+10            -6.2%  1.858e+10        perf-stat.i.instructions
      2788 ±  3%     +46.3%       4078 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.83            -5.9%       0.78        perf-stat.i.metric.GHz
      0.24 ± 10%    +131.1%       0.56 ±  6%  perf-stat.i.metric.K/sec
    227.35            -4.8%     216.32        perf-stat.i.metric.M/sec
     45.27            -1.5       43.80        perf-stat.i.node-load-miss-rate%
  18589033            -1.5%   18304945        perf-stat.i.node-load-misses
  12517155           +13.7%   14232925        perf-stat.i.node-store-misses
  18575324           +11.3%   20676441        perf-stat.i.node-stores
      1.06 ±  2%      +0.2        1.22 ±  3%  perf-stat.overall.branch-miss-rate%
     15.83            +2.5       18.29        perf-stat.overall.cache-miss-rate%
    996.16           -14.1%     855.28        perf-stat.overall.cycles-between-cache-misses
      0.66            -0.0        0.63 ±  3%  perf-stat.overall.dTLB-load-miss-rate%
      0.18 ±  4%      -0.0        0.15 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     65.64           -20.1       45.54 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      2411           +64.2%       3960        perf-stat.overall.instructions-per-iTLB-miss
     47.60            -0.6       47.03        perf-stat.overall.node-load-miss-rate%
     40.26            +0.5       40.77        perf-stat.overall.node-store-miss-rate%
 4.257e+09            -7.2%  3.951e+09        perf-stat.ps.branch-instructions
  45324442 ±  2%      +6.4%   48223476 ±  3%  perf-stat.ps.branch-misses
  39373167            +8.9%   42887411        perf-stat.ps.cache-misses
 2.488e+08            -5.8%  2.345e+08        perf-stat.ps.cache-references
    591509           -26.2%     436737        perf-stat.ps.context-switches
 3.922e+10            -6.5%  3.668e+10        perf-stat.ps.cpu-cycles
     47502           -72.9%      12874        perf-stat.ps.cpu-migrations
  32689942           -11.3%   28983395 ±  3%  perf-stat.ps.dTLB-load-misses
 4.897e+09            -6.0%  4.601e+09        perf-stat.ps.dTLB-loads
   2345925 ±  4%     -12.6%    2051157 ±  3%  perf-stat.ps.dTLB-store-misses
 1.299e+09            +2.4%  1.331e+09        perf-stat.ps.dTLB-stores
   8102419           -43.2%    4600177        perf-stat.ps.iTLB-load-misses
   4240977           +29.8%    5504884 ±  3%  perf-stat.ps.iTLB-loads
 1.954e+10            -6.8%  1.822e+10        perf-stat.ps.instructions
  18329122            -2.1%   17943105        perf-stat.ps.node-load-misses
  12342162           +13.0%   13951584        perf-stat.ps.node-store-misses
  18315870           +10.7%   20267915        perf-stat.ps.node-stores
  1.48e+12           -34.2%  9.735e+11        perf-stat.total.instructions
    122.75 ± 89%     -74.5%      31.25 ±  7%  interrupts.42:PCI-MSI.2621448-edge.eth0-TxRx-7
    152826           -29.0%     108455        interrupts.CPU0.LOC:Local_timer_interrupts
      6665 ± 15%     -36.3%       4246 ± 14%  interrupts.CPU0.RES:Rescheduling_interrupts
    152828           -29.2%     108194        interrupts.CPU1.LOC:Local_timer_interrupts
      5599 ±  8%     -37.9%       3474 ±  3%  interrupts.CPU1.RES:Rescheduling_interrupts
    152281           -28.8%     108431        interrupts.CPU10.LOC:Local_timer_interrupts
      3939 ±  4%     -44.8%       2172 ± 30%  interrupts.CPU10.NMI:Non-maskable_interrupts
      3939 ±  4%     -44.8%       2172 ± 30%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      5445 ±  3%     -37.8%       3388        interrupts.CPU10.RES:Rescheduling_interrupts
    152253           -28.8%     108417        interrupts.CPU11.LOC:Local_timer_interrupts
      3187 ± 28%     -25.5%       2373 ± 36%  interrupts.CPU11.NMI:Non-maskable_interrupts
      3187 ± 28%     -25.5%       2373 ± 36%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    152655           -28.9%     108471        interrupts.CPU12.LOC:Local_timer_interrupts
      6315 ± 13%     -43.3%       3579 ±  2%  interrupts.CPU12.RES:Rescheduling_interrupts
    152850           -29.0%     108475        interrupts.CPU13.LOC:Local_timer_interrupts
      3349 ± 21%     -27.4%       2431 ± 36%  interrupts.CPU13.NMI:Non-maskable_interrupts
      3349 ± 21%     -27.4%       2431 ± 36%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      5711 ±  2%     -32.8%       3836 ± 19%  interrupts.CPU13.RES:Rescheduling_interrupts
    152819           -29.1%     108317        interrupts.CPU14.LOC:Local_timer_interrupts
      3019 ± 32%     -38.3%       1862 ±  2%  interrupts.CPU14.NMI:Non-maskable_interrupts
      3019 ± 32%     -38.3%       1862 ±  2%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      5738           -38.2%       3546 ±  2%  interrupts.CPU14.RES:Rescheduling_interrupts
    152859           -29.2%     108233        interrupts.CPU15.LOC:Local_timer_interrupts
      2611 ± 23%     -18.2%       2135 ± 21%  interrupts.CPU15.NMI:Non-maskable_interrupts
      2611 ± 23%     -18.2%       2135 ± 21%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      5669           -39.5%       3430        interrupts.CPU15.RES:Rescheduling_interrupts
    152721           -29.0%     108431        interrupts.CPU16.LOC:Local_timer_interrupts
      5619           -39.6%       3396        interrupts.CPU16.RES:Rescheduling_interrupts
    152667           -29.0%     108459        interrupts.CPU17.LOC:Local_timer_interrupts
      5614           -37.8%       3490 ±  2%  interrupts.CPU17.RES:Rescheduling_interrupts
    152713           -29.0%     108449        interrupts.CPU18.LOC:Local_timer_interrupts
      5683 ±  2%     -28.5%       4066 ± 18%  interrupts.CPU18.RES:Rescheduling_interrupts
    152924           -29.1%     108456        interrupts.CPU19.LOC:Local_timer_interrupts
      5601           -37.9%       3478 ±  3%  interrupts.CPU19.RES:Rescheduling_interrupts
    152728           -29.0%     108459        interrupts.CPU2.LOC:Local_timer_interrupts
      2882 ± 30%     -24.7%       2171 ± 24%  interrupts.CPU2.NMI:Non-maskable_interrupts
      2882 ± 30%     -24.7%       2171 ± 24%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      5413 ±  2%     -31.5%       3706 ± 10%  interrupts.CPU2.RES:Rescheduling_interrupts
    152541           -28.9%     108435        interrupts.CPU20.LOC:Local_timer_interrupts
      5860           -37.0%       3691        interrupts.CPU20.RES:Rescheduling_interrupts
    152329           -28.8%     108387        interrupts.CPU21.LOC:Local_timer_interrupts
      5582           -37.7%       3477 ±  4%  interrupts.CPU21.RES:Rescheduling_interrupts
    152212           -28.6%     108706        interrupts.CPU22.LOC:Local_timer_interrupts
      5723           -38.2%       3536 ±  2%  interrupts.CPU22.RES:Rescheduling_interrupts
    152451           -28.9%     108455        interrupts.CPU23.LOC:Local_timer_interrupts
      5645           -37.5%       3531        interrupts.CPU23.RES:Rescheduling_interrupts
    152813           -29.0%     108450        interrupts.CPU24.LOC:Local_timer_interrupts
      3192 ± 28%     -28.4%       2286 ± 34%  interrupts.CPU24.NMI:Non-maskable_interrupts
      3192 ± 28%     -28.4%       2286 ± 34%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      5587 ±  2%     -37.1%       3511        interrupts.CPU24.RES:Rescheduling_interrupts
    152870           -29.1%     108438        interrupts.CPU25.LOC:Local_timer_interrupts
      5629 ±  2%     -36.0%       3603 ±  3%  interrupts.CPU25.RES:Rescheduling_interrupts
    152727           -29.0%     108410        interrupts.CPU26.LOC:Local_timer_interrupts
      5516           -36.4%       3511        interrupts.CPU26.RES:Rescheduling_interrupts
    152618           -29.1%     108171        interrupts.CPU27.LOC:Local_timer_interrupts
      2562 ± 33%     -27.9%       1848 ±  4%  interrupts.CPU27.NMI:Non-maskable_interrupts
      2562 ± 33%     -27.9%       1848 ±  4%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      5501           -37.6%       3435 ±  2%  interrupts.CPU27.RES:Rescheduling_interrupts
    152552           -28.9%     108426        interrupts.CPU28.LOC:Local_timer_interrupts
      5527 ±  2%     -35.3%       3575 ±  2%  interrupts.CPU28.RES:Rescheduling_interrupts
    152168           -28.7%     108462        interrupts.CPU29.LOC:Local_timer_interrupts
      5429 ±  2%     -38.1%       3358 ±  3%  interrupts.CPU29.RES:Rescheduling_interrupts
    152691           -29.2%     108172        interrupts.CPU3.LOC:Local_timer_interrupts
    152496           -28.9%     108442        interrupts.CPU30.LOC:Local_timer_interrupts
      5439           -37.2%       3417 ±  3%  interrupts.CPU30.RES:Rescheduling_interrupts
    152408           -28.9%     108434        interrupts.CPU31.LOC:Local_timer_interrupts
      1996           +72.7%       3448 ±  7%  interrupts.CPU31.NMI:Non-maskable_interrupts
      1996           +72.7%       3448 ±  7%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      5405           -37.9%       3358        interrupts.CPU31.RES:Rescheduling_interrupts
    152368           -29.0%     108213        interrupts.CPU32.LOC:Local_timer_interrupts
      5487 ±  4%     -34.9%       3570 ±  5%  interrupts.CPU32.RES:Rescheduling_interrupts
    122.75 ± 89%     -74.5%      31.25 ±  7%  interrupts.CPU33.42:PCI-MSI.2621448-edge.eth0-TxRx-7
    152190           -28.8%     108434        interrupts.CPU33.LOC:Local_timer_interrupts
      5397 ±  3%     -37.0%       3398 ±  2%  interrupts.CPU33.RES:Rescheduling_interrupts
    152274           -28.8%     108410        interrupts.CPU34.LOC:Local_timer_interrupts
      5644 ±  2%     -37.5%       3526        interrupts.CPU34.RES:Rescheduling_interrupts
    152204           -28.8%     108394        interrupts.CPU35.LOC:Local_timer_interrupts
      5440           -36.9%       3434        interrupts.CPU35.RES:Rescheduling_interrupts
    152846           -29.0%     108452        interrupts.CPU36.LOC:Local_timer_interrupts
      5751 ±  2%     -38.7%       3526 ±  2%  interrupts.CPU36.RES:Rescheduling_interrupts
    152906           -29.1%     108450        interrupts.CPU37.LOC:Local_timer_interrupts
      5655           -38.3%       3489 ±  2%  interrupts.CPU37.RES:Rescheduling_interrupts
    152894           -29.1%     108466        interrupts.CPU38.LOC:Local_timer_interrupts
      5718           -40.1%       3426        interrupts.CPU38.RES:Rescheduling_interrupts
    152859           -29.1%     108396        interrupts.CPU39.LOC:Local_timer_interrupts
      5705 ±  2%     -39.1%       3477 ±  2%  interrupts.CPU39.RES:Rescheduling_interrupts
    152552           -28.9%     108451        interrupts.CPU4.LOC:Local_timer_interrupts
      5646 ± 10%     -37.0%       3558 ± 10%  interrupts.CPU4.RES:Rescheduling_interrupts
    152750           -29.0%     108474        interrupts.CPU40.LOC:Local_timer_interrupts
      3276 ± 24%     -26.8%       2399 ± 35%  interrupts.CPU40.NMI:Non-maskable_interrupts
      3276 ± 24%     -26.8%       2399 ± 35%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      5576           -38.9%       3406 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
    152315           -28.8%     108444        interrupts.CPU41.LOC:Local_timer_interrupts
      5628           -39.3%       3415 ±  2%  interrupts.CPU41.RES:Rescheduling_interrupts
    152653           -29.0%     108402        interrupts.CPU42.LOC:Local_timer_interrupts
      5601           -38.8%       3429        interrupts.CPU42.RES:Rescheduling_interrupts
    152608           -28.9%     108444        interrupts.CPU43.LOC:Local_timer_interrupts
      5697           -37.9%       3539 ±  3%  interrupts.CPU43.RES:Rescheduling_interrupts
    152559           -28.7%     108808        interrupts.CPU44.LOC:Local_timer_interrupts
      4041           -41.9%       2349 ± 35%  interrupts.CPU44.NMI:Non-maskable_interrupts
      4041           -41.9%       2349 ± 35%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      5733           -38.3%       3535        interrupts.CPU44.RES:Rescheduling_interrupts
    152335           -28.7%     108551        interrupts.CPU45.LOC:Local_timer_interrupts
      5763           -39.1%       3510 ±  3%  interrupts.CPU45.RES:Rescheduling_interrupts
    152247           -28.4%     109025        interrupts.CPU46.LOC:Local_timer_interrupts
      5638           -38.9%       3446 ±  3%  interrupts.CPU46.RES:Rescheduling_interrupts
    152218           -28.7%     108515        interrupts.CPU47.LOC:Local_timer_interrupts
      5622           -38.3%       3467        interrupts.CPU47.RES:Rescheduling_interrupts
    152536           -28.9%     108438        interrupts.CPU5.LOC:Local_timer_interrupts
      5818 ± 13%     -32.9%       3902 ± 24%  interrupts.CPU5.RES:Rescheduling_interrupts
    152446           -28.8%     108470        interrupts.CPU6.LOC:Local_timer_interrupts
      5329 ±  2%     -34.8%       3477        interrupts.CPU6.RES:Rescheduling_interrupts
    152385           -28.8%     108460        interrupts.CPU7.LOC:Local_timer_interrupts
      3418 ± 24%     -32.0%       2325 ± 36%  interrupts.CPU7.NMI:Non-maskable_interrupts
      3418 ± 24%     -32.0%       2325 ± 36%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      6075 ± 15%     -39.9%       3652 ±  3%  interrupts.CPU7.RES:Rescheduling_interrupts
    152777           -29.1%     108340        interrupts.CPU8.LOC:Local_timer_interrupts
      3062 ± 31%     -39.2%       1862 ±  2%  interrupts.CPU8.NMI:Non-maskable_interrupts
      3062 ± 31%     -39.2%       1862 ±  2%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      5715           -37.9%       3547 ±  2%  interrupts.CPU8.RES:Rescheduling_interrupts
    152182           -28.8%     108424        interrupts.CPU9.LOC:Local_timer_interrupts
      3430 ± 24%     -46.5%       1833 ±  2%  interrupts.CPU9.NMI:Non-maskable_interrupts
      3430 ± 24%     -46.5%       1833 ±  2%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      5429           -31.8%       3703 ± 12%  interrupts.CPU9.RES:Rescheduling_interrupts
   7323124           -28.9%    5205113        interrupts.LOC:Local_timer_interrupts
    146059 ±  4%     -10.9%     130082        interrupts.NMI:Non-maskable_interrupts
    146059 ±  4%     -10.9%     130082        interrupts.PMI:Performance_monitoring_interrupts
    271567           -36.9%     171453        interrupts.RES:Rescheduling_interrupts
     17140 ±  3%     -34.8%      11168        softirqs.CPU0.RCU
     17986 ±  5%     -19.6%      14464 ±  7%  softirqs.CPU0.SCHED
     28979 ±  4%     -20.9%      22922 ±  5%  softirqs.CPU0.TIMER
     17619           -38.8%      10778 ±  4%  softirqs.CPU1.RCU
     15605 ±  7%     -27.1%      11383 ±  4%  softirqs.CPU1.SCHED
     16585 ±  3%     -33.9%      10961 ±  2%  softirqs.CPU10.RCU
     13799 ±  2%     -28.2%       9904        softirqs.CPU10.SCHED
     27653 ±  2%     -14.0%      23789 ±  6%  softirqs.CPU10.TIMER
     16548 ±  2%     -31.8%      11278 ±  4%  softirqs.CPU11.RCU
     13273 ±  3%     -22.9%      10231 ±  6%  softirqs.CPU11.SCHED
     16266 ±  6%     -29.5%      11463 ±  2%  softirqs.CPU12.RCU
     14858 ±  6%     -28.1%      10682 ±  6%  softirqs.CPU12.SCHED
     33722 ± 18%     -24.9%      25329 ±  5%  softirqs.CPU12.TIMER
     15708 ±  2%     -32.1%      10673 ±  2%  softirqs.CPU13.RCU
     13743 ±  5%     -26.4%      10119 ±  4%  softirqs.CPU13.SCHED
     29036 ±  3%     -20.1%      23189 ±  4%  softirqs.CPU13.TIMER
     16055 ±  2%     -31.6%      10986 ±  3%  softirqs.CPU14.RCU
     13325 ±  5%     -25.8%       9883 ±  8%  softirqs.CPU14.SCHED
     28544 ±  5%     -19.1%      23081 ±  7%  softirqs.CPU14.TIMER
     15921 ±  2%     -33.5%      10587 ±  2%  softirqs.CPU15.RCU
     13359 ±  3%     -27.2%       9729 ±  4%  softirqs.CPU15.SCHED
     29422 ±  5%     -23.2%      22599 ±  5%  softirqs.CPU15.TIMER
     17193           -35.9%      11025        softirqs.CPU16.RCU
     13323 ±  4%     -26.9%       9736 ±  7%  softirqs.CPU16.SCHED
     28613 ±  5%     -20.4%      22775 ±  4%  softirqs.CPU16.TIMER
     17197           -34.9%      11191        softirqs.CPU17.RCU
     13120 ±  4%     -22.6%      10151        softirqs.CPU17.SCHED
     28246 ±  2%     -19.0%      22875 ±  4%  softirqs.CPU17.TIMER
     17045           -35.2%      11052        softirqs.CPU18.RCU
     13658           -25.2%      10221 ±  2%  softirqs.CPU18.SCHED
     28477           -20.0%      22794 ±  4%  softirqs.CPU18.TIMER
     16917           -34.6%      11063        softirqs.CPU19.RCU
     13777           -27.5%       9992 ±  3%  softirqs.CPU19.SCHED
     28454 ±  3%     -20.0%      22775 ±  6%  softirqs.CPU19.TIMER
     17271 ±  2%     -35.6%      11124        softirqs.CPU2.RCU
     13807 ±  5%     -25.7%      10262 ±  4%  softirqs.CPU2.SCHED
     28069 ±  3%     -17.6%      23118 ±  6%  softirqs.CPU2.TIMER
     16727           -34.9%      10886        softirqs.CPU20.RCU
     13544 ±  2%     -27.3%       9849 ±  3%  softirqs.CPU20.SCHED
     29083 ±  4%     -21.0%      22987 ±  6%  softirqs.CPU20.TIMER
     16432           -33.7%      10899 ±  2%  softirqs.CPU21.RCU
     13384 ±  2%     -25.1%      10027 ±  3%  softirqs.CPU21.SCHED
     28221 ±  2%     -17.1%      23402 ±  6%  softirqs.CPU21.TIMER
     16478           -33.8%      10916 ±  2%  softirqs.CPU22.RCU
     13285 ±  2%     -24.9%       9976        softirqs.CPU22.SCHED
     29361 ±  6%     -21.8%      22961 ±  5%  softirqs.CPU22.TIMER
     16217 ±  2%     -33.8%      10734 ±  2%  softirqs.CPU23.RCU
     13344 ±  4%     -26.6%       9798 ±  5%  softirqs.CPU23.SCHED
     28138 ±  2%     -18.2%      23016 ±  5%  softirqs.CPU23.TIMER
     17081 ±  4%     -36.5%      10849 ±  2%  softirqs.CPU24.RCU
     13721           -28.8%       9765 ±  2%  softirqs.CPU24.SCHED
     17285 ±  3%     -38.9%      10567 ±  4%  softirqs.CPU25.RCU
     13652           -26.3%      10065        softirqs.CPU25.SCHED
     28148           -19.8%      22575 ±  7%  softirqs.CPU25.TIMER
     17536 ±  4%     -36.7%      11092        softirqs.CPU26.RCU
     13498 ±  3%     -25.8%      10009 ±  3%  softirqs.CPU26.SCHED
     28293 ±  5%     -19.4%      22798 ±  7%  softirqs.CPU26.TIMER
     17772 ±  3%     -37.7%      11066        softirqs.CPU27.RCU
     13915 ±  4%     -32.9%       9342 ±  7%  softirqs.CPU27.SCHED
     28883 ±  5%     -20.9%      22856 ±  6%  softirqs.CPU27.TIMER
     17120 ±  2%     -36.1%      10944 ±  4%  softirqs.CPU28.RCU
     13447 ±  3%     -25.5%      10016        softirqs.CPU28.SCHED
     28693 ±  2%     -22.8%      22153 ±  9%  softirqs.CPU28.TIMER
     17768 ±  3%     -36.7%      11247        softirqs.CPU29.RCU
     13008 ±  6%     -21.2%      10256 ±  2%  softirqs.CPU29.SCHED
     17261 ±  2%     -35.6%      11109        softirqs.CPU3.RCU
     13617 ±  9%     -22.1%      10603 ±  5%  softirqs.CPU3.SCHED
     27993 ±  2%     -16.6%      23351 ±  7%  softirqs.CPU3.TIMER
     17527 ±  2%     -35.6%      11282        softirqs.CPU30.RCU
     13802 ±  3%     -28.8%       9831 ±  6%  softirqs.CPU30.SCHED
     29335 ±  4%     -19.1%      23743 ±  7%  softirqs.CPU30.TIMER
     17177 ±  2%     -36.7%      10880        softirqs.CPU31.RCU
     13970           -27.7%      10107        softirqs.CPU31.SCHED
     27907 ±  4%     -17.1%      23143 ±  7%  softirqs.CPU31.TIMER
     15513 ±  6%     -31.1%      10683        softirqs.CPU32.RCU
     13604 ±  4%     -24.9%      10219        softirqs.CPU32.SCHED
     28401 ±  3%     -16.8%      23636 ±  8%  softirqs.CPU32.TIMER
     16438           -35.5%      10602        softirqs.CPU33.RCU
     13884 ±  2%     -29.3%       9817 ±  5%  softirqs.CPU33.SCHED
     28035 ±  4%     -16.0%      23546 ±  8%  softirqs.CPU33.TIMER
     16511 ±  3%     -35.0%      10725 ±  2%  softirqs.CPU34.RCU
     13845           -28.6%       9883 ±  7%  softirqs.CPU34.SCHED
     27630 ±  4%     -13.9%      23784 ±  8%  softirqs.CPU34.TIMER
     16437 ±  4%     -34.5%      10759        softirqs.CPU35.RCU
     13764 ±  2%     -27.9%       9925 ±  5%  softirqs.CPU35.SCHED
     16989 ±  4%     -35.6%      10948        softirqs.CPU36.RCU
     12964 ±  5%     -24.0%       9857        softirqs.CPU36.SCHED
     27393           -18.1%      22447 ±  5%  softirqs.CPU36.TIMER
     17454           -35.7%      11223 ±  2%  softirqs.CPU37.RCU
     13500 ±  4%     -26.8%       9882 ±  4%  softirqs.CPU37.SCHED
     28336 ±  3%     -19.4%      22833 ±  3%  softirqs.CPU37.TIMER
     20134 ± 14%     -31.1%      13870 ± 16%  softirqs.CPU38.RCU
     13271 ±  4%     -25.0%       9947 ±  3%  softirqs.CPU38.SCHED
     29935 ±  6%     -17.0%      24854 ±  2%  softirqs.CPU38.TIMER
     19850 ± 11%     -37.6%      12384 ± 16%  softirqs.CPU39.RCU
     13570           -27.8%       9803 ±  5%  softirqs.CPU39.SCHED
     30276 ±  8%     -24.1%      22994 ± 11%  softirqs.CPU39.TIMER
     16670 ±  3%     -34.2%      10963 ±  4%  softirqs.CPU4.RCU
     13283 ±  5%     -22.7%      10262 ±  2%  softirqs.CPU4.SCHED
     17482           -29.6%      12315 ± 15%  softirqs.CPU40.RCU
     13702 ±  3%     -27.5%       9935 ±  3%  softirqs.CPU40.SCHED
     28521 ±  3%     -16.9%      23697 ±  8%  softirqs.CPU40.TIMER
     17356 ±  2%     -35.5%      11192 ±  2%  softirqs.CPU41.RCU
     13463 ±  2%     -27.6%       9746 ±  6%  softirqs.CPU41.SCHED
     28259 ±  2%     -20.0%      22603 ±  5%  softirqs.CPU41.TIMER
     17727 ±  2%     -36.2%      11301 ±  2%  softirqs.CPU42.RCU
     13690           -28.3%       9820 ±  3%  softirqs.CPU42.SCHED
     17241 ±  2%     -35.4%      11134        softirqs.CPU43.RCU
     13661           -25.7%      10155 ±  2%  softirqs.CPU43.SCHED
     28137 ±  3%     -19.3%      22717 ±  6%  softirqs.CPU43.TIMER
     16832           -35.4%      10877        softirqs.CPU44.RCU
     13715           -26.8%      10036 ±  3%  softirqs.CPU44.SCHED
     29149 ±  7%     -22.2%      22690 ±  5%  softirqs.CPU44.TIMER
     16717           -35.1%      10851        softirqs.CPU45.RCU
     13690 ±  2%     -26.4%      10072        softirqs.CPU45.SCHED
     28192 ±  2%     -19.7%      22641 ±  5%  softirqs.CPU45.TIMER
     16561           -35.1%      10755        softirqs.CPU46.RCU
     13475 ±  2%     -26.6%       9886        softirqs.CPU46.SCHED
     29227 ±  8%     -21.8%      22861 ±  5%  softirqs.CPU46.TIMER
     16988 ±  2%     -35.7%      10928        softirqs.CPU47.RCU
     13693           -26.3%      10094        softirqs.CPU47.SCHED
     28344 ±  2%     -19.5%      22803 ±  4%  softirqs.CPU47.TIMER
     17152 ±  2%     -34.3%      11277 ±  2%  softirqs.CPU5.RCU
     13954 ±  6%     -27.4%      10125 ±  5%  softirqs.CPU5.SCHED
     28016 ±  3%     -12.9%      24406 ± 10%  softirqs.CPU5.TIMER
     17496 ±  3%     -35.9%      11212        softirqs.CPU6.RCU
     13667           -28.0%       9843 ±  9%  softirqs.CPU6.SCHED
     29313 ±  6%     -17.1%      24297 ± 10%  softirqs.CPU6.TIMER
     16871 ±  2%     -34.0%      11134 ±  2%  softirqs.CPU7.RCU
     14093 ±  3%     -27.2%      10263        softirqs.CPU7.SCHED
     27786 ±  2%     -11.1%      24697 ±  9%  softirqs.CPU7.TIMER
     15933 ±  4%     -32.9%      10697        softirqs.CPU8.RCU
     13506 ±  4%     -26.3%       9949 ±  3%  softirqs.CPU8.SCHED
     35708 ± 22%     -35.1%      23168 ±  7%  softirqs.CPU8.TIMER
     16502 ±  3%     -34.5%      10801        softirqs.CPU9.RCU
     13399 ±  4%     -27.0%       9786 ±  4%  softirqs.CPU9.SCHED
     27553 ±  3%     -14.8%      23461 ±  7%  softirqs.CPU9.TIMER
    816730 ±  2%     -34.8%     532482        softirqs.RCU
    659242           -26.3%     485737        softirqs.SCHED
   1382277 ±  2%     -18.2%    1131336 ±  5%  softirqs.TIMER
     15.54 ±  2%     -15.5        0.00        perf-profile.calltrace.cycles-pp.set_node_addr.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
     15.31 ±  2%     -15.3        0.00        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.set_node_addr.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file
     14.76 ±  2%     -14.8        0.00        perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.set_node_addr.__write_node_page.f2fs_fsync_node_pages
     13.15 ±  2%     -13.2        0.00        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.set_node_addr.__write_node_page
     33.69            -9.1       24.59 ±  2%  perf-profile.calltrace.cycles-pp.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
     22.33            -6.6       15.70 ±  3%  perf-profile.calltrace.cycles-pp.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
     44.27            -6.0       38.26 ±  2%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     44.25            -6.0       38.24 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     44.27            -6.0       38.27 ±  2%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     44.90            -6.0       38.94        perf-profile.calltrace.cycles-pp.secondary_startup_64
     38.50            -4.9       33.63        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     39.04            -4.8       34.24 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     39.11            -4.8       34.31 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.73            -2.7        2.99        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_fsync_node_pages
      6.22            -2.6        3.59        perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file
      6.25            -2.6        3.62        perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      2.77            -2.5        0.26 ±100%  perf-profile.calltrace.cycles-pp.rwsem_wake.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      7.53            -1.7        5.83        perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      2.42 ±  2%      -1.2        1.21 ± 20%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      1.99            -1.0        0.95 ±  2%  perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
      1.87            -1.0        0.91 ±  2%  perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      1.19 ±  3%      -0.6        0.59        perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      2.25            -0.5        1.72        perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.96            -0.5        1.48 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      1.96            -0.5        1.48 ±  2%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      1.95            -0.5        1.47 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
      1.84            -0.5        1.38 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      1.12 ±  2%      -0.4        0.71 ±  4%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.10 ±  2%      -0.4        0.69 ±  5%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.61 ±  2%      -0.4        0.25 ±100%  perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
      1.04            -0.1        0.90 ±  5%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.78            +0.0        0.82 ±  4%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.47 ±  4%      +0.2        1.67 ±  3%  perf-profile.calltrace.cycles-pp.pagevec_lookup_range_tag.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      1.46 ±  4%      +0.2        1.66 ±  3%  perf-profile.calltrace.cycles-pp.find_get_pages_range_tag.pagevec_lookup_range_tag.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      0.59 ±  8%      +0.3        0.86 ±  4%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.58 ±  4%      +0.4        1.01        perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.issue_flush_thread.kthread
      0.60 ±  4%      +0.4        1.05        perf-profile.calltrace.cycles-pp.swake_up_locked.complete.issue_flush_thread.kthread.ret_from_fork
      0.61 ±  4%      +0.5        1.06        perf-profile.calltrace.cycles-pp.complete.issue_flush_thread.kthread.ret_from_fork
      0.74 ±  4%      +0.5        1.25        perf-profile.calltrace.cycles-pp.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write
      0.13 ±173%      +0.5        0.65 ±  4%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +0.5        0.52 ±  2%  perf-profile.calltrace.cycles-pp.f2fs_update_data_blkaddr.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages
      0.00            +0.5        0.54 ±  5%  perf-profile.calltrace.cycles-pp.submit_bio_wait.__submit_flush_wait.issue_flush_thread.kthread.ret_from_fork
      0.00            +0.5        0.54 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page
      0.00            +0.6        0.55        perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_fsync_node_pages
      0.00            +0.6        0.56        perf-profile.calltrace.cycles-pp.f2fs_submit_merged_ipu_write.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range
      0.00            +0.6        0.56        perf-profile.calltrace.cycles-pp.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page.__write_node_page
      0.77 ±  3%      +0.6        1.35        perf-profile.calltrace.cycles-pp.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages
      0.00            +0.6        0.59 ±  5%  perf-profile.calltrace.cycles-pp.__submit_flush_wait.issue_flush_thread.kthread.ret_from_fork
      0.00            +0.6        0.60 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_write_cache_pages
      0.96 ±  3%      +0.6        1.56        perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages
      0.76            +0.6        1.37        perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.__submit_merged_bio.__submit_merged_write_cond.f2fs_fsync_node_pages
      1.00 ±  3%      +0.6        1.61        perf-profile.calltrace.cycles-pp.submit_bio.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages
      1.02 ±  3%      +0.6        1.64        perf-profile.calltrace.cycles-pp.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
      0.00            +0.6        0.62        perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked
      0.78            +0.6        1.40        perf-profile.calltrace.cycles-pp.submit_bio.__submit_merged_bio.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file
      0.00            +0.6        0.63        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.issue_flush_thread
      0.00            +0.6        0.63        perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete
      2.33 ± 17%      +0.6        2.97 ±  7%  perf-profile.calltrace.cycles-pp.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page
      0.78            +0.6        1.42        perf-profile.calltrace.cycles-pp.__submit_merged_bio.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      0.00            +0.6        0.64 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_allocate_data_block.do_write_page
      0.00            +0.7        0.67 ±  6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.f2fs_space_for_roll_forward.f2fs_do_sync_file
      1.19 ±  8%      +0.7        1.86 ±  3%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_write_cache_pages
      0.00            +0.7        0.73 ±  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter
      0.72 ±  7%      +0.7        1.47 ±  5%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      0.75 ±  7%      +0.8        1.51 ±  5%  perf-profile.calltrace.cycles-pp.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      0.89 ±  4%      +0.8        1.69 ±  2%  perf-profile.calltrace.cycles-pp.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      0.00            +0.8        0.80        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.f2fs_issue_flush
      1.29 ±  4%      +0.8        2.10        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write
      0.14 ±173%      +0.8        0.98        perf-profile.calltrace.cycles-pp.wait_for_completion.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      0.00            +0.9        0.86        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.f2fs_issue_flush.f2fs_do_sync_file
      1.44 ±  4%      +0.9        2.31        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
      0.00            +0.9        0.89        perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter
      0.98 ±  2%      +0.9        1.89 ±  2%  perf-profile.calltrace.cycles-pp.issue_flush_thread.kthread.ret_from_fork
      1.54 ±  5%      +1.0        2.51 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages
      1.57 ±  5%      +1.0        2.54 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
      1.50 ±  2%      +1.0        2.51        perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.__submit_merged_bio.__submit_merged_write_cond
      0.53 ±  2%      +1.1        1.62        perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.__submit_merged_bio
      0.00            +1.1        1.10        perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.f2fs_allocate_data_block.do_write_page
      0.00            +1.1        1.13        perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data
      1.58 ±  4%      +1.2        2.76 ±  2%  perf-profile.calltrace.cycles-pp.ret_from_fork
      1.58 ±  4%      +1.2        2.76 ±  2%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.00            +1.3        1.28        perf-profile.calltrace.cycles-pp.rwsem_wake.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page
      3.06 ±  3%      +2.0        5.06 ±  2%  perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range
     49.92            +3.1       53.02        perf-profile.calltrace.cycles-pp.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
      2.81 ± 14%      +3.9        6.73 ±  2%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page
      1.31            +4.0        5.32        perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page
      1.33            +4.0        5.35        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page.__write_node_page
      1.58            +4.2        5.76        perf-profile.calltrace.cycles-pp.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages
     51.62            +4.2       55.84        perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     51.67            +4.2       55.90        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     51.89            +4.3       56.23        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     51.93            +4.4       56.29        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     52.14            +4.5       56.63        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     52.16            +4.5       56.66        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     52.57            +4.6       57.19        perf-profile.calltrace.cycles-pp.write
      2.44            +4.8        7.23        perf-profile.calltrace.cycles-pp.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      2.43            +4.8        7.22        perf-profile.calltrace.cycles-pp.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file
      0.00            +5.3        5.34 ± 11%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_read_slowpath.__write_node_page.f2fs_fsync_node_pages
      0.00            +6.0        5.98 ±  9%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file
      0.00            +6.4        6.44 ±  8%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      0.00            +8.2        8.21 ±  6%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_allocate_data_block.do_write_page
      0.00            +9.8        9.79 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data
      0.00           +10.3       10.34 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page
      0.70 ±  2%     +11.8       12.47 ±  4%  perf-profile.calltrace.cycles-pp.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page
      5.36 ±  8%     +11.9       17.27 ±  4%  perf-profile.calltrace.cycles-pp.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
      5.51 ±  8%     +12.0       17.53 ±  3%  perf-profile.calltrace.cycles-pp.f2fs_write_single_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range
      3.10 ± 13%     +12.5       15.57 ±  4%  perf-profile.calltrace.cycles-pp.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages
      3.40 ± 11%     +12.7       16.13 ±  4%  perf-profile.calltrace.cycles-pp.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.f2fs_write_data_pages
      9.22 ±  5%     +14.3       23.56 ±  3%  perf-profile.calltrace.cycles-pp.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      9.36 ±  5%     +14.4       23.75 ±  3%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter
      9.34 ±  5%     +14.4       23.73 ±  3%  perf-profile.calltrace.cycles-pp.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file
      9.39 ±  5%     +14.4       23.79 ±  3%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      9.48 ±  5%     +14.4       23.93 ±  3%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
     15.57 ±  2%     -15.1        0.46 ±  3%  perf-profile.children.cycles-pp.set_node_addr
     33.70            -9.1       24.60 ±  2%  perf-profile.children.cycles-pp.f2fs_fsync_node_pages
     22.33            -6.6       15.70 ±  3%  perf-profile.children.cycles-pp.__write_node_page
     44.27            -6.0       38.27 ±  2%  perf-profile.children.cycles-pp.start_secondary
     44.90            -6.0       38.94        perf-profile.children.cycles-pp.secondary_startup_64
     44.90            -6.0       38.94        perf-profile.children.cycles-pp.cpu_startup_entry
     44.90            -6.0       38.94        perf-profile.children.cycles-pp.do_idle
     38.62            -4.8       33.87        perf-profile.children.cycles-pp.intel_idle
     39.65            -4.7       34.92        perf-profile.children.cycles-pp.cpuidle_enter_state
     39.66            -4.7       34.93        perf-profile.children.cycles-pp.cpuidle_enter
      2.34 ± 15%      -2.0        0.30 ±  3%  perf-profile.children.cycles-pp.f2fs_get_node_info
      2.34 ± 35%      -2.0        0.30 ±  4%  perf-profile.children.cycles-pp.f2fs_is_checkpointed_node
      3.66 ±  2%      -2.0        1.69        perf-profile.children.cycles-pp.wake_up_q
      3.81 ±  2%      -2.0        1.85        perf-profile.children.cycles-pp.rwsem_wake
      2.14 ± 12%      -1.9        0.22 ±  6%  perf-profile.children.cycles-pp.f2fs_need_inode_block_update
      4.53 ±  2%      -1.3        3.24        perf-profile.children.cycles-pp.try_to_wake_up
      1.29 ±  4%      -1.1        0.23 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      2.67 ±  3%      -1.0        1.65        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.11 ± 10%      -1.0        0.12 ±  5%  perf-profile.children.cycles-pp.f2fs_need_dentry_mark
      4.19            -1.0        3.21        perf-profile.children.cycles-pp.ttwu_do_activate
      4.19            -1.0        3.21        perf-profile.children.cycles-pp.activate_task
      4.16            -1.0        3.18        perf-profile.children.cycles-pp.enqueue_task_fair
      3.99            -1.0        3.04        perf-profile.children.cycles-pp.enqueue_entity
      3.22            -0.8        2.45        perf-profile.children.cycles-pp.__account_scheduler_latency
      2.64            -0.5        2.10        perf-profile.children.cycles-pp.stack_trace_save_tsk
      2.33            -0.5        1.80        perf-profile.children.cycles-pp.sched_ttwu_pending
      2.49 ±  2%      -0.5        1.98        perf-profile.children.cycles-pp.arch_stack_walk
      2.79 ±  2%      -0.4        2.35        perf-profile.children.cycles-pp.__sched_text_start
      1.14 ±  2%      -0.4        0.73 ±  5%  perf-profile.children.cycles-pp.schedule_idle
      1.61 ±  2%      -0.3        1.28 ±  2%  perf-profile.children.cycles-pp.unwind_next_frame
      0.34 ±  7%      -0.2        0.13 ±  5%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.71 ±  3%      -0.2        0.53        perf-profile.children.cycles-pp.dequeue_entity
      0.81 ±  2%      -0.2        0.63 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.67 ±  3%      -0.1        0.52        perf-profile.children.cycles-pp.update_load_avg
      0.49 ±  4%      -0.1        0.35 ±  3%  perf-profile.children.cycles-pp.select_task_rq_fair
      1.06            -0.1        0.93 ±  4%  perf-profile.children.cycles-pp.menu_select
      1.68            -0.1        1.56        perf-profile.children.cycles-pp.schedule
      0.24 ±  2%      -0.1        0.12 ±  8%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.50 ±  6%      -0.1        0.42 ±  2%  perf-profile.children.cycles-pp.orc_find
      0.36 ±  3%      -0.1        0.28 ±  2%  perf-profile.children.cycles-pp.__lookup_nat_cache
      0.42 ±  5%      -0.1        0.35 ±  5%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.40 ±  6%      -0.1        0.33 ±  3%  perf-profile.children.cycles-pp.__kernel_text_address
      0.37 ±  9%      -0.1        0.30 ±  3%  perf-profile.children.cycles-pp.kernel_text_address
      0.14 ±  8%      -0.1        0.08 ± 10%  perf-profile.children.cycles-pp.update_cfs_rq_h_load
      0.32 ±  5%      -0.1        0.25 ±  4%  perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
      0.24 ±  3%      -0.1        0.18 ±  4%  perf-profile.children.cycles-pp.f2fs_update_inode_page
      0.41 ±  5%      -0.1        0.35 ±  2%  perf-profile.children.cycles-pp.__orc_find
      0.15 ±  7%      -0.1        0.10 ± 13%  perf-profile.children.cycles-pp.update_cfs_group
      0.27 ±  2%      -0.1        0.22 ±  3%  perf-profile.children.cycles-pp.f2fs_write_inode
      0.31 ±  4%      -0.0        0.26 ±  2%  perf-profile.children.cycles-pp.update_curr
      0.30 ±  4%      -0.0        0.26 ±  6%  perf-profile.children.cycles-pp.set_next_entity
      0.12 ±  5%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.wake_q_add
      0.20 ±  7%      -0.0        0.16 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.08 ±  5%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.cpuacct_charge
      0.18 ±  7%      -0.0        0.14 ±  7%  perf-profile.children.cycles-pp.__list_add_valid
      0.13 ±  9%      -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.12 ±  4%      -0.0        0.09 ±  9%  perf-profile.children.cycles-pp.f2fs_update_inode
      0.14 ±  7%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.native_write_msr
      0.52 ±  2%      -0.0        0.48 ±  3%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.14 ±  5%      -0.0        0.11 ±  6%  perf-profile.children.cycles-pp.finish_task_switch
      0.29 ±  2%      -0.0        0.26 ±  3%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.18 ±  8%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.select_idle_sibling
      0.18 ±  4%      -0.0        0.15 ±  8%  perf-profile.children.cycles-pp.__switch_to_asm
      0.09 ± 14%      -0.0        0.07 ± 13%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.38 ±  4%      -0.0        0.35 ±  3%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.20 ±  4%      -0.0        0.18 ±  4%  perf-profile.children.cycles-pp.native_sched_clock
      0.24 ±  3%      -0.0        0.22 ±  6%  perf-profile.children.cycles-pp.update_rq_clock
      0.18 ±  6%      -0.0        0.16 ±  2%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.17 ±  2%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.___perf_sw_event
      0.09 ±  5%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.08 ±  5%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.in_sched_functions
      0.07 ±  7%      -0.0        0.05        perf-profile.children.cycles-pp.rcu_idle_exit
      0.08 ±  5%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.__set_nat_cache_dirty
      0.06 ± 15%      +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.__get_segment_type
      0.05 ±  9%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__fdget_pos
      0.10 ±  4%      +0.0        0.12 ±  5%  perf-profile.children.cycles-pp.ksys_lseek
      0.06 ±  7%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.f2fs_inode_chksum_verify
      0.07 ±  6%      +0.0        0.09 ±  8%  perf-profile.children.cycles-pp.common_file_perm
      0.68 ±  2%      +0.0        0.71        perf-profile.children.cycles-pp.hrtimer_interrupt
      0.13            +0.0        0.16 ±  6%  perf-profile.children.cycles-pp.read_tsc
      0.06 ± 13%      +0.0        0.09        perf-profile.children.cycles-pp.xas_set_mark
      0.06 ± 13%      +0.0        0.09        perf-profile.children.cycles-pp.__xa_set_mark
      0.05 ±  8%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.tag_pages_for_writeback
      0.04 ± 57%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.put_prev_entity
      0.10 ± 14%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.file_update_time
      0.06 ± 13%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.bio_associate_blkg_from_css
      0.06 ± 13%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.08 ±  5%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.cpumask_next
      0.09 ±  4%      +0.0        0.12 ±  9%  perf-profile.children.cycles-pp.security_file_permission
      0.25 ±  4%      +0.0        0.28 ±  6%  perf-profile.children.cycles-pp.up_read
      0.08 ± 10%      +0.0        0.11        perf-profile.children.cycles-pp.rmqueue
      0.07 ±  5%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.unlock_page
      0.04 ± 58%      +0.0        0.07 ± 15%  perf-profile.children.cycles-pp.update_blocked_averages
      0.07 ± 11%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.bio_associate_blkg
      0.18 ±  7%      +0.0        0.22 ±  7%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.10 ±  8%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.account_page_dirtied
      0.11 ±  4%      +0.0        0.15 ±  8%  perf-profile.children.cycles-pp.kmem_cache_free
      0.04 ± 58%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      0.15 ±  8%      +0.0        0.19 ±  9%  perf-profile.children.cycles-pp.dput
      0.15 ±  8%      +0.0        0.19 ±  7%  perf-profile.children.cycles-pp.__dentry_kill
      0.15 ±  5%      +0.0        0.19 ±  7%  perf-profile.children.cycles-pp.evict
      0.15 ±  5%      +0.0        0.19 ±  7%  perf-profile.children.cycles-pp.f2fs_evict_inode
      0.05 ±  8%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.sb_clear_inode_writeback
      0.15 ±  5%      +0.0        0.20 ±  7%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.15 ±  5%      +0.0        0.20 ±  7%  perf-profile.children.cycles-pp.__fput
      0.12 ± 12%      +0.0        0.17 ±  4%  perf-profile.children.cycles-pp.___might_sleep
      0.10 ± 11%      +0.0        0.14 ±  8%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.06 ± 11%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.f2fs_truncate_data_blocks_range
      0.07 ± 10%      +0.0        0.12 ±  7%  perf-profile.children.cycles-pp.blk_throtl_bio
      0.08 ±  8%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp.f2fs_merge_page_bio
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.f2fs_llseek
      0.15 ±  5%      +0.0        0.20 ±  7%  perf-profile.children.cycles-pp.close
      0.15 ±  5%      +0.0        0.20 ±  7%  perf-profile.children.cycles-pp.task_work_run
      0.13 ±  3%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.update_stats_enqueue_sleeper
      0.11 ±  4%      +0.0        0.16 ±  6%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.06 ±  6%      +0.0        0.11 ± 15%  perf-profile.children.cycles-pp.f2fs_truncate
      0.06 ±  6%      +0.0        0.11 ± 15%  perf-profile.children.cycles-pp.f2fs_do_truncate_blocks
      0.03 ±100%      +0.0        0.07 ± 14%  perf-profile.children.cycles-pp.__xa_clear_mark
      0.12 ±  7%      +0.1        0.17 ±  2%  perf-profile.children.cycles-pp.locate_dirty_segment
      0.82            +0.1        0.87 ±  3%  perf-profile.children.cycles-pp.release_pages
      0.07 ± 13%      +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.f2fs_update_dirty_page
      0.01 ±173%      +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.bvec_alloc
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__bio_associate_blkg
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.bio_free
      0.01 ±173%      +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      0.01 ±173%      +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.31 ±  7%      +0.1        0.37 ±  3%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.07 ± 10%      +0.1        0.13 ± 12%  perf-profile.children.cycles-pp.f2fs_wait_on_node_pages_writeback
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.set_page_dirty
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.down_read_trylock
      0.04 ± 58%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.f2fs_get_block
      0.04 ± 60%      +0.1        0.10 ±  7%  perf-profile.children.cycles-pp.mutex_unlock
      0.03 ±100%      +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.f2fs_invalidate_blocks
      0.08 ± 10%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.f2fs_del_fsync_node_entry
      0.09 ±  7%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.__has_merged_page
      0.07 ±  5%      +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.sb_mark_inode_writeback
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.bio_endio
      0.00            +0.1        0.06 ± 20%  perf-profile.children.cycles-pp.__sb_start_write
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.rcu_all_qs
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.xas_clear_mark
      0.13 ± 12%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.copyin
      0.12 ± 12%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp.f2fs_balance_fs
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.f2fs_target_device
      0.01 ±173%      +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.f2fs_reserve_new_blocks
      0.11 ±  7%      +0.1        0.18 ±  6%  perf-profile.children.cycles-pp.__remove_ino_entry
      0.12 ± 13%      +0.1        0.19 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.01 ±173%      +0.1        0.08 ± 15%  perf-profile.children.cycles-pp.f2fs_reserve_block
      0.11 ±  7%      +0.1        0.18 ±  4%  perf-profile.children.cycles-pp.f2fs_inplace_write_data
      0.10 ± 10%      +0.1        0.17 ±  8%  perf-profile.children.cycles-pp.wait_for_completion_io
      0.17 ±  8%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.xas_load
      0.08 ± 14%      +0.1        0.15 ±  4%  perf-profile.children.cycles-pp.__might_sleep
      0.10 ±  9%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.__is_cp_guaranteed
      1.05 ±  2%      +0.1        1.13 ±  3%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.14 ± 10%      +0.1        0.22        perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      0.14 ±  8%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.03 ±100%      +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.f2fs_lookup_extent_cache
      0.14 ±  3%      +0.1        0.23 ±  4%  perf-profile.children.cycles-pp.read_node_page
      0.57 ±  2%      +0.1        0.65 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.13 ± 11%      +0.1        0.22 ±  5%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.12 ± 14%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.mempool_alloc
      0.00            +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.md_submit_flush_data
      1.15 ±  2%      +0.1        1.24 ±  3%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.13 ±  3%      +0.1        0.22 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.15 ±  4%      +0.1        0.24 ±  9%  perf-profile.children.cycles-pp.mutex_lock
      0.14 ±  7%      +0.1        0.23 ±  3%  perf-profile.children.cycles-pp.__bio_alloc
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.f2fs_wait_on_page_writeback
      0.16 ±  7%      +0.1        0.27 ±  3%  perf-profile.children.cycles-pp.f2fs_get_dnode_of_data
      0.15 ±  8%      +0.1        0.27 ±  6%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.09 ±  8%      +0.1        0.21 ± 23%  perf-profile.children.cycles-pp.f2fs_preallocate_blocks
      0.22 ±  3%      +0.1        0.34 ±  6%  perf-profile.children.cycles-pp.find_get_entry
      0.23 ±  7%      +0.1        0.35        perf-profile.children.cycles-pp.f2fs_set_data_page_dirty
      0.27            +0.1        0.39 ±  4%  perf-profile.children.cycles-pp.f2fs_is_valid_blkaddr
      0.01 ±173%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.07 ±  7%      +0.1        0.19 ±  8%  perf-profile.children.cycles-pp.load_balance
      0.21 ± 12%      +0.1        0.34 ±  5%  perf-profile.children.cycles-pp.xas_find_marked
      0.16 ±  7%      +0.1        0.29 ±  3%  perf-profile.children.cycles-pp.brd_lookup_page
      0.08 ±  8%      +0.1        0.21 ±  4%  perf-profile.children.cycles-pp.newidle_balance
      0.01 ±173%      +0.1        0.15 ±  5%  perf-profile.children.cycles-pp.find_busiest_group
      0.07 ± 11%      +0.1        0.21 ±  3%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.29 ±  4%      +0.1        0.43 ±  4%  perf-profile.children.cycles-pp.lseek64
      0.00            +0.1        0.14 ± 32%  perf-profile.children.cycles-pp.f2fs_convert_inline_inode
      0.12 ±  4%      +0.1        0.27 ±  5%  perf-profile.children.cycles-pp.up_write
      0.27 ±  4%      +0.1        0.41 ±  5%  perf-profile.children.cycles-pp.update_sit_entry
      0.08 ± 10%      +0.1        0.23 ±  6%  perf-profile.children.cycles-pp.__wake_up_common
      0.01 ±173%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.preempt_schedule_common
      2.22 ±  2%      +0.2        2.38        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.48 ±  9%      +0.2        0.65 ±  4%  perf-profile.children.cycles-pp.process_one_work
      0.12 ±  7%      +0.2        0.28 ±  8%  perf-profile.children.cycles-pp._cond_resched
      0.21 ±  5%      +0.2        0.39 ±  4%  perf-profile.children.cycles-pp.down_write
      0.39 ±  4%      +0.2        0.56        perf-profile.children.cycles-pp.__mutex_lock
      0.39 ± 10%      +0.2        0.56        perf-profile.children.cycles-pp.f2fs_submit_merged_ipu_write
      0.32 ±  8%      +0.2        0.50        perf-profile.children.cycles-pp.f2fs_write_end
      0.34 ±  2%      +0.2        0.52 ±  5%  perf-profile.children.cycles-pp.pagecache_get_page
      0.21 ± 10%      +0.2        0.40        perf-profile.children.cycles-pp.generic_make_request_checks
      0.16 ±  4%      +0.2        0.35 ±  8%  perf-profile.children.cycles-pp.submit_flushes
      0.35 ±  3%      +0.2        0.55 ±  3%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.29 ±  4%      +0.2        0.50 ±  2%  perf-profile.children.cycles-pp.test_clear_page_writeback
      0.38 ±  4%      +0.2        0.58        perf-profile.children.cycles-pp.__get_node_page
      0.30 ±  3%      +0.2        0.51 ±  2%  perf-profile.children.cycles-pp.end_page_writeback
      0.15            +0.2        0.36        perf-profile.children.cycles-pp.__wake_up_common_lock
      0.30 ±  3%      +0.2        0.51 ±  2%  perf-profile.children.cycles-pp.f2fs_set_node_page_dirty
      0.20 ±  7%      +0.2        0.42 ±  3%  perf-profile.children.cycles-pp.__queue_work
      0.37 ±  5%      +0.2        0.60 ±  3%  perf-profile.children.cycles-pp.clear_page_erms
      0.38 ±  4%      +0.2        0.61 ±  3%  perf-profile.children.cycles-pp.prep_new_page
      0.36 ±  5%      +0.2        0.59        perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.51 ±  2%      +0.2        0.74        perf-profile.children.cycles-pp.down_read
      0.28            +0.2        0.52 ±  2%  perf-profile.children.cycles-pp.f2fs_update_data_blkaddr
      0.28 ±  2%      +0.2        0.53 ±  2%  perf-profile.children.cycles-pp.f2fs_set_data_blkaddr
      0.71 ±  2%      +0.2        0.96 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
      0.21 ±  7%      +0.2        0.45 ±  2%  perf-profile.children.cycles-pp.queue_work_on
      1.52 ±  4%      +0.3        1.78 ±  3%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      1.54 ±  4%      +0.3        1.79 ±  3%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.59 ±  8%      +0.3        0.86 ±  4%  perf-profile.children.cycles-pp.worker_thread
      0.22 ±  7%      +0.3        0.49 ±  2%  perf-profile.children.cycles-pp.md_flush_request
      0.47 ±  3%      +0.3        0.75 ±  2%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.75 ±  4%      +0.3        1.05        perf-profile.children.cycles-pp.swake_up_locked
      0.52 ±  3%      +0.3        0.82 ±  2%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.77 ±  5%      +0.3        1.07        perf-profile.children.cycles-pp.complete
     10.60            +0.4       10.96        perf-profile.children.cycles-pp.__submit_merged_write_cond
      0.63 ±  2%      +0.4        1.00 ±  2%  perf-profile.children.cycles-pp.brd_insert_page
      0.37 ±  9%      +0.4        0.75 ±  2%  perf-profile.children.cycles-pp.raid0_make_request
      0.51 ±  2%      +0.4        0.91        perf-profile.children.cycles-pp.f2fs_write_end_io
      0.41 ±  4%      +0.4        0.81 ±  4%  perf-profile.children.cycles-pp.submit_bio_wait
      0.41 ±  8%      +0.4        0.82 ±  2%  perf-profile.children.cycles-pp.md_handle_request
      0.48 ±  3%      +0.4        0.89        perf-profile.children.cycles-pp.schedule_timeout
      0.45 ±  3%      +0.4        0.88 ±  3%  perf-profile.children.cycles-pp.__submit_flush_wait
      0.49 ±  6%      +0.5        0.96        perf-profile.children.cycles-pp.md_make_request
      0.49 ±  6%      +0.5        0.98        perf-profile.children.cycles-pp.wait_for_completion
      0.74 ±  4%      +0.5        1.25        perf-profile.children.cycles-pp.f2fs_write_begin
      1.14 ±  3%      +0.7        1.84        perf-profile.children.cycles-pp.brd_do_bvec
      0.73 ±  6%      +0.7        1.47 ±  5%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.75 ±  7%      +0.8        1.51 ±  5%  perf-profile.children.cycles-pp.f2fs_space_for_roll_forward
      0.89 ±  4%      +0.8        1.69 ±  2%  perf-profile.children.cycles-pp.f2fs_issue_flush
      1.30 ±  4%      +0.8        2.14        perf-profile.children.cycles-pp.generic_perform_write
      1.45 ±  4%      +0.9        2.32        perf-profile.children.cycles-pp.__generic_file_write_iter
      0.98 ±  2%      +0.9        1.89 ±  2%  perf-profile.children.cycles-pp.issue_flush_thread
      1.74 ±  2%      +1.2        2.89        perf-profile.children.cycles-pp.brd_make_request
      1.58 ±  4%      +1.2        2.76 ±  2%  perf-profile.children.cycles-pp.ret_from_fork
      1.58 ±  4%      +1.2        2.76 ±  2%  perf-profile.children.cycles-pp.kthread
      2.38            +1.2        3.58 ±  2%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      1.81 ±  2%      +1.3        3.08        perf-profile.children.cycles-pp.__submit_merged_bio
      2.46 ±  3%      +1.8        4.27        perf-profile.children.cycles-pp.generic_make_request
      2.47 ±  3%      +1.8        4.29        perf-profile.children.cycles-pp.submit_bio
     49.92            +3.1       53.02        perf-profile.children.cycles-pp.f2fs_do_sync_file
     51.63            +4.2       55.84        perf-profile.children.cycles-pp.f2fs_file_write_iter
     51.68            +4.2       55.91        perf-profile.children.cycles-pp.new_sync_write
     51.91            +4.3       56.24        perf-profile.children.cycles-pp.vfs_write
     51.94            +4.4       56.30        perf-profile.children.cycles-pp.ksys_write
     52.61            +4.6       57.24        perf-profile.children.cycles-pp.write
     52.63            +4.6       57.27        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     52.61            +4.6       57.25        perf-profile.children.cycles-pp.do_syscall_64
      2.44            +4.8        7.23        perf-profile.children.cycles-pp.f2fs_do_write_node_page
      3.92 ± 10%      +4.8        8.73 ±  2%  perf-profile.children.cycles-pp.f2fs_submit_page_write
      5.36 ±  8%     +11.9       17.27 ±  4%  perf-profile.children.cycles-pp.f2fs_do_write_data_page
      5.51 ±  8%     +12.0       17.53 ±  3%  perf-profile.children.cycles-pp.f2fs_write_single_data_page
      1.47 ±  2%     +12.4       13.92 ±  3%  perf-profile.children.cycles-pp.f2fs_allocate_data_block
      3.41 ± 11%     +12.8       16.22 ±  4%  perf-profile.children.cycles-pp.f2fs_outplace_write_data
      9.23 ±  5%     +14.3       23.56 ±  3%  perf-profile.children.cycles-pp.f2fs_write_cache_pages
      9.36 ±  5%     +14.4       23.75 ±  3%  perf-profile.children.cycles-pp.do_writepages
      9.35 ±  5%     +14.4       23.73 ±  3%  perf-profile.children.cycles-pp.f2fs_write_data_pages
      9.39 ±  5%     +14.4       23.79 ±  3%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      9.48 ±  5%     +14.4       23.93 ±  3%  perf-profile.children.cycles-pp.file_write_and_wait_range
      5.53 ±  7%     +17.4       22.89 ±  2%  perf-profile.children.cycles-pp.do_write_page
     38.61            -4.8       33.85        perf-profile.self.cycles-pp.intel_idle
      2.66 ±  3%      -1.0        1.65        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.92            -0.2        0.68 ±  4%  perf-profile.self.cycles-pp.unwind_next_frame
      0.42 ±  6%      -0.2        0.19 ±  2%  perf-profile.self.cycles-pp.rwsem_down_read_slowpath
      0.39 ±  2%      -0.2        0.22 ±  5%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.45 ±  2%      -0.1        0.32 ±  6%  perf-profile.self.cycles-pp.__sched_text_start
      0.11 ±  3%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.rwsem_mark_wake
      0.30 ±  4%      -0.1        0.22 ±  3%  perf-profile.self.cycles-pp.update_load_avg
      0.11 ±  7%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.wake_up_q
      0.41 ±  3%      -0.1        0.33 ±  6%  perf-profile.self.cycles-pp.menu_select
      0.14 ±  8%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.update_cfs_rq_h_load
      0.23 ±  7%      -0.1        0.18 ±  8%  perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
      0.15 ±  6%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.update_cfs_group
      0.40 ±  4%      -0.1        0.35        perf-profile.self.cycles-pp.__orc_find
      0.13 ±  5%      -0.1        0.08 ± 15%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.44 ±  3%      -0.0        0.39 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.16 ±  7%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.12 ±  5%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.wake_q_add
      0.08 ±  5%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.cpuacct_charge
      0.17 ±  6%      -0.0        0.14 ±  8%  perf-profile.self.cycles-pp.do_idle
      0.20 ±  7%      -0.0        0.16 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.14 ±  7%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.native_write_msr
      0.17 ±  6%      -0.0        0.14 ±  9%  perf-profile.self.cycles-pp.__list_add_valid
      0.20 ±  9%      -0.0        0.17 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.16 ±  5%      -0.0        0.13 ±  7%  perf-profile.self.cycles-pp.set_next_entity
      0.12 ±  6%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.enqueue_entity
      0.09 ± 14%      -0.0        0.07 ± 13%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.18 ±  6%      -0.0        0.15 ±  8%  perf-profile.self.cycles-pp.__switch_to_asm
      0.15 ±  2%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.___perf_sw_event
      0.11 ±  7%      -0.0        0.09 ±  9%  perf-profile.self.cycles-pp.stack_trace_save_tsk
      0.19 ±  2%      -0.0        0.17 ±  4%  perf-profile.self.cycles-pp.update_rq_clock
      0.18 ±  3%      -0.0        0.15 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.19 ±  4%      -0.0        0.17 ±  4%  perf-profile.self.cycles-pp.native_sched_clock
      0.10 ±  7%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.finish_task_switch
      0.09 ± 11%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.kernel_text_address
      0.10 ±  5%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.update_stats_enqueue_sleeper
      0.07 ± 10%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.schedule
      0.06 ±  7%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.write
      0.10 ±  8%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.10 ±  7%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.05 ±  8%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.f2fs_get_node_info
      0.12 ±  3%      +0.0        0.15 ±  7%  perf-profile.self.cycles-pp.read_tsc
      0.07 ± 14%      +0.0        0.10        perf-profile.self.cycles-pp.f2fs_write_cache_pages
      0.06 ± 13%      +0.0        0.09        perf-profile.self.cycles-pp.xas_set_mark
      0.05 ±  9%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.locate_dirty_segment
      0.05 ±  8%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.test_clear_page_writeback
      0.08 ±  5%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.kmem_cache_free
      0.04 ± 58%      +0.0        0.07 ± 17%  perf-profile.self.cycles-pp.generic_make_request
      0.04 ± 60%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.__get_segment_type
      0.07 ±  5%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.unlock_page
      0.07 ±  6%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.f2fs_do_write_data_page
      0.18 ±  7%      +0.0        0.22 ±  7%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.10 ± 12%      +0.0        0.14 ±  7%  perf-profile.self.cycles-pp.__write_node_page
      0.08 ±  6%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.07            +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.f2fs_set_node_page_dirty
      0.05 ±  8%      +0.0        0.09 ±  8%  perf-profile.self.cycles-pp.raid0_make_request
      0.07 ± 12%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.f2fs_get_dnode_of_data
      0.11 ±  4%      +0.0        0.15 ±  4%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.07 ±  7%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.md_make_request
      0.12 ±  8%      +0.0        0.17 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.06 ± 11%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.f2fs_write_single_data_page
      0.08 ± 11%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.f2fs_issue_flush
      0.03 ±100%      +0.0        0.07 ± 17%  perf-profile.self.cycles-pp.__submit_merged_write_cond
      0.03 ±100%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.md_handle_request
      0.08 ±  8%      +0.1        0.13 ±  9%  perf-profile.self.cycles-pp.__get_node_page
      0.01 ±173%      +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.f2fs_write_end
      0.01 ±173%      +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.account_page_dirtied
      0.01 ±173%      +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.blk_throtl_bio
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.f2fs_write_inode
      0.09 ±  7%      +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.f2fs_write_data_pages
      0.08 ±  5%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.read_node_page
      0.10 ±  7%      +0.1        0.15 ±  7%  perf-profile.self.cycles-pp.xas_load
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__bio_associate_blkg
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.bio_alloc_bioset
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.generic_perform_write
      0.05 ± 58%      +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.generic_make_request_checks
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.f2fs_llseek
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.down_read_trylock
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.__set_page_dirty_nobuffers
      0.09 ±  7%      +0.1        0.15 ±  8%  perf-profile.self.cycles-pp.__has_merged_page
      0.09 ±  4%      +0.1        0.15 ± 12%  perf-profile.self.cycles-pp.f2fs_submit_page_write
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.issue_flush_thread
      0.15 ± 12%      +0.1        0.21 ±  4%  perf-profile.self.cycles-pp.f2fs_do_sync_file
      0.30 ±  7%      +0.1        0.36 ±  3%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.do_write_page
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.xas_clear_mark
      0.11 ± 10%      +0.1        0.17 ±  8%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.07 ± 10%      +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.03 ±100%      +0.1        0.09 ±  8%  perf-profile.self.cycles-pp.mutex_unlock
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp._cond_resched
      0.13 ±  5%      +0.1        0.20 ±  5%  perf-profile.self.cycles-pp.find_get_entry
      0.12 ± 15%      +0.1        0.18 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.01 ±173%      +0.1        0.08        perf-profile.self.cycles-pp.f2fs_lookup_extent_cache
      0.10 ±  9%      +0.1        0.17 ±  3%  perf-profile.self.cycles-pp.__is_cp_guaranteed
      0.11 ±  7%      +0.1        0.18 ±  2%  perf-profile.self.cycles-pp.f2fs_balance_fs
      0.13 ±  5%      +0.1        0.21 ±  8%  perf-profile.self.cycles-pp.mutex_lock
      0.08 ± 12%      +0.1        0.16 ±  6%  perf-profile.self.cycles-pp.f2fs_write_end_io
      0.08 ±  8%      +0.1        0.16 ±  7%  perf-profile.self.cycles-pp.f2fs_file_write_iter
      0.12 ±  8%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.f2fs_write_begin
      0.12 ±  3%      +0.1        0.20 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.13 ±  9%      +0.1        0.22 ±  5%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.03 ±100%      +0.1        0.11 ± 14%  perf-profile.self.cycles-pp.submit_flushes
      0.15 ±  5%      +0.1        0.24 ±  6%  perf-profile.self.cycles-pp.f2fs_allocate_data_block
      0.00            +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.27            +0.1        0.39 ±  3%  perf-profile.self.cycles-pp.f2fs_is_valid_blkaddr
      0.21 ± 10%      +0.1        0.34 ±  5%  perf-profile.self.cycles-pp.xas_find_marked
      0.18 ±  6%      +0.1        0.32 ±  4%  perf-profile.self.cycles-pp.down_write
      0.12 ±  3%      +0.1        0.26 ±  4%  perf-profile.self.cycles-pp.up_write
      0.26 ±  5%      +0.1        0.39 ±  3%  perf-profile.self.cycles-pp.update_sit_entry
      0.70 ±  4%      +0.1        0.84 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.29 ±  2%      +0.2        1.44 ±  2%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.24 ±  4%      +0.2        0.40 ±  6%  perf-profile.self.cycles-pp.do_syscall_64
      0.35 ±  6%      +0.2        0.55        perf-profile.self.cycles-pp.brd_do_bvec
      0.44            +0.2        0.65        perf-profile.self.cycles-pp.down_read
      0.37 ±  4%      +0.2        0.58 ±  3%  perf-profile.self.cycles-pp.clear_page_erms
      0.62            +0.3        0.87 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.34 ± 11%      +0.3        0.63 ±  6%  perf-profile.self.cycles-pp.__percpu_counter_sum
      1.31 ±  2%      +0.3        1.65 ±  6%  perf-profile.self.cycles-pp.rwsem_optimistic_spin
      2.17 ±  2%      +1.1        3.31        perf-profile.self.cycles-pp.rwsem_spin_on_owner


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  12500 +-------------------------------------------------------------------+   
  12000 |-+                 OO            OO     O                          |   
        |          O     OO    OO  OO OO O   OO O    O  O                   |   
  11500 |O+      OO  OO           O                OO  O  OOO               |   
  11000 |-+OO  O        O                                     O             |   
        |                                                                   |   
  10500 |-+   O                                                             |   
  10000 |-+                                                                 |   
   9500 |-+                                                                 |   
        |                                                                   |   
   9000 |-+                                                                 |   
   8500 |-+                                                                 |   
        |+. +.++.  +.++. ++.+ .++. ++.+                                  +.+|   
   8000 |-++     ++     +    +    +    +.+++.++.++.+++.++.+++.++.++.+++.+   |   
   7500 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               aim7.time.system_time                            
                                                                                
  950 +---------------------------------------------------------------------+   
      |                                  +   +.++.++.+ .+    +.       +.    |   
  900 |+.++.++.++.++.++.+++.++.++.++.++.+ +.+         +  +.++  ++.++.+  ++.+|   
  850 |-+                                                                   |   
      |                                                                     |   
  800 |-+                                                                   |   
      |                                                                     |   
  750 |-+                                                                   |   
      |                                                                     |   
  700 |-+                                                                   |   
  650 |-+                                                                   |   
      |                                                                     |   
  600 |-+ O O                                                               |   
      |O O   O OO OO OO OOO  O OO OO OO O   O  O  OO  O  O OOO              |   
  550 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             aim7.time.elapsed_time                             
                                                                                
  80 +----------------------------------------------------------------------+   
     |                                           +. +.                      |   
  75 |+.++.+ .++.++.++.++. +.++.++.+ .++.+++.++.+  +  ++.++.++.++.++.++.++.+|   
     |      +             +         +                                       |   
  70 |-+                                                                    |   
     |                                                                      |   
  65 |-+                                                                    |   
     |                                                                      |   
  60 |-+                                                                    |   
     |     O                                                                |   
  55 |-+ O  O                                                O              |   
     |O O     OO OO OO O        O  O  OO         O O  O  OO O               |   
  50 |-+                O OO OO  O  O    OOO OO O   O  O                    |   
     |                                                                      |   
  45 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           aim7.time.elapsed_time.max                           
                                                                                
  80 +----------------------------------------------------------------------+   
     |                                           +. +.                      |   
  75 |+.++.+ .++.++.++.++. +.++.++.+ .++.+++.++.+  +  ++.++.++.++.++.++.++.+|   
     |      +             +         +                                       |   
  70 |-+                                                                    |   
     |                                                                      |   
  65 |-+                                                                    |   
     |                                                                      |   
  60 |-+                                                                    |   
     |     O                                                                |   
  55 |-+ O  O                                                O              |   
     |O O     OO OO OO O        O  O  OO         O O  O  OO O               |   
  50 |-+                O OO OO  O  O    OOO OO O   O  O                    |   
     |                                                                      |   
  45 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         aim7.time.voluntary_context_switches                   
                                                                                
  2.2e+07 +-----------------------------------------------------------------+   
          |+.++.+ +.+++.++.++ .+++.++.+ +.+++.++.+ +. + .+   ++.+++. +.++   |   
    2e+07 |-+    +           +         +          +  + +  +.+       +    +.+|   
          |                                                                 |   
  1.8e+07 |-+                                                               |   
          |                                                                 |   
  1.6e+07 |-+                                                               |   
          |                                                                 |   
  1.4e+07 |-+                                                               |   
          |                                                                 |   
  1.2e+07 |-+                                                               |   
          |                                                                 |   
    1e+07 |-+                                                               |   
          |                                                                 |   
    8e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             aim7.time.file_system_outputs                      
                                                                                
  8.7e+07 +-----------------------------------------------------------------+   
          |                                                                 |   
  8.6e+07 |+.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+|   
          |                                                                 |   
  8.5e+07 |-+                                                               |   
          |                                                                 |   
  8.4e+07 |-+                                                               |   
          |                                                                 |   
  8.3e+07 |-+                                                               |   
          |                                                                 |   
  8.2e+07 |-+                                                               |   
          |                                                                 |   
  8.1e+07 |-+                                                               |   
          |  O   OO OOO O  OOO OOO OO OOO OOO OO O O OOO OO OOO             |   
    8e+07 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-11903-gca7f76e680745" of type "text/plain" (202296 bytes)

View attachment "job-script" of type "text/plain" (8036 bytes)

View attachment "job.yaml" of type "text/plain" (5643 bytes)

View attachment "reproduce" of type "text/plain" (1009 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ