lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20220207081731.GD387@xsang-OptiPlex-9020>
Date:   Mon, 7 Feb 2022 16:17:31 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Jaegeuk Kim <jaegeuk@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com, fengwei.yin@...el.com
Subject: [f2fs]  26ababe972:  aim7.jobs-per-min -92.2% regression



Greeting,

FYI, we noticed a -92.2% regression of aim7.jobs-per-min due to commit:


commit: 26ababe972240454bf3eb3ac282a6867d7428de5 ("f2fs: add a way to limit roll forward recovery time")
url: https://github.com/0day-ci/linux/commits/Chao-Yu/f2fs-introduce-F2FS_IPU_DEFRAG-to-allow-file-defragment-in-IPU-mode/20220204-000519

in testcase: aim7
on test machine: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: f2fs
	test: sync_disk_rw
	load: 100
	cpufreq_governor: performance
	ucode: 0x5003006

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/4BRD_12G/f2fs/x86_64-rhel-8.3/100/RAID0/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/sync_disk_rw/aim7/0x5003006

commit: 
  f26f32510f ("f2fs: expose discard related parameters in sysfs")
  26ababe972 ("f2fs: add a way to limit roll forward recovery time")

f26f32510fcfa76a 26ababe972240454bf3eb3ac282 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      7072 ±  2%     -92.2%     554.83        aim7.jobs-per-min
     84.91 ±  2%   +1173.7%       1081        aim7.time.elapsed_time
     84.91 ±  2%   +1173.7%       1081        aim7.time.elapsed_time.max
  83270128           +37.5%  1.145e+08        aim7.time.file_system_outputs
    773116 ±  2%     +98.2%    1532098 ±  8%  aim7.time.involuntary_context_switches
     16072 ±  2%     -45.1%       8831        aim7.time.minor_page_faults
      5103 ±  3%     -91.3%     446.26 ± 20%  aim7.time.system_time
  36844705           -59.2%   15043600 ±  6%  aim7.time.voluntary_context_switches
   3673593 ±  3%     -13.9%    3163404 ± 14%  numa-numastat.node0.numa_hit
 2.177e+09         +4165.3%  9.287e+10        cpuidle..time
  28265775          +609.8%  2.006e+08        cpuidle..usage
    125.93          +791.6%       1122        uptime.boot
      5228 ±  2%    +543.3%      33634 ±  2%  uptime.idle
     30.34            +6.6%      32.35 ±  2%  iostat.cpu.idle
      0.09 ±  7%  +73334.1%      65.79        iostat.cpu.iowait
     69.41           -97.3%       1.84 ±  8%  iostat.cpu.system
     28.81            +3.4       32.23 ±  2%  mpstat.cpu.all.idle%
      0.09 ±  7%     +65.8       65.91        mpstat.cpu.all.iowait%
      1.12 ±  4%      +0.3        1.40 ±  6%  mpstat.cpu.all.irq%
     69.74           -69.4        0.33 ± 25%  mpstat.cpu.all.sys%
      0.15 ± 10%      -0.1        0.02 ± 11%  mpstat.cpu.all.usr%
     68.83           -97.7%       1.60 ± 30%  vmstat.cpu.sy
      0.00       +6.5e+103%      65.40        vmstat.cpu.wa
    467151 ±  2%     -91.9%      37999        vmstat.io.bo
      0.00       +8.7e+103%      86.60        vmstat.procs.b
     63.33          -100.0%       0.00        vmstat.procs.r
    832364           -95.3%      38937 ±  3%  vmstat.system.cs
    312405           -41.8%     181666        vmstat.system.in
     80445 ±  8%     -77.7%      17975        meminfo.Active
     71750 ±  9%     -79.1%      14995        meminfo.Active(anon)
      8694           -65.7%       2979        meminfo.Active(file)
     98227 ±  4%    +100.5%     196961        meminfo.AnonHugePages
    531554           -20.2%     423925        meminfo.Committed_AS
     37075 ±  6%     -43.2%      21065        meminfo.Dirty
     18090           -10.5%      16194        meminfo.KernelStack
     39877           -17.9%      32756        meminfo.Mapped
  16674813           -37.9%   10354875        meminfo.Memused
    322.00 ± 72%    +516.3%       1984 ± 12%  meminfo.Mlocked
     87183 ±  8%     -70.5%      25757        meminfo.Shmem
      1415 ± 47%     -96.4%      50.60 ±  6%  meminfo.Writeback
  16676304           -37.8%   10369960        meminfo.max_used_kB
      2008           -97.7%      46.60 ±  5%  turbostat.Avg_MHz
     72.33           -69.1        3.25 ±  7%  turbostat.Busy%
      2784           -48.2%       1442 ±  2%  turbostat.Bzy_MHz
  14047565           -52.1%    6734949 ± 36%  turbostat.C1
      7.30 ±  3%      -5.7        1.56 ± 45%  turbostat.C1%
  13207840 ±  5%   +1246.9%  1.779e+08 ±  8%  turbostat.C1E
     11.03 ± 38%     +76.9       87.96 ± 11%  turbostat.C1E%
     24.88 ±  4%    +288.1%      96.57        turbostat.CPU%c1
      2.79 ± 46%     -93.7%       0.18 ±188%  turbostat.CPU%c6
     58.00           -21.7%      45.40 ±  2%  turbostat.CoreTmp
  27718656 ±  2%    +611.5%  1.972e+08        turbostat.IRQ
     57.83           -20.5%      46.00        turbostat.PkgTmp
    203.22           -51.1%      99.45 ±  8%  turbostat.PkgWatt
     57.41           -11.8%      50.62        turbostat.RAMWatt
      4352 ± 24%     -41.3%       2555        numa-meminfo.node0.Active(file)
     19124 ±  6%     -45.8%      10373 ±  5%  numa-meminfo.node0.Dirty
   8858697 ± 12%     -27.2%    6451575 ±  2%  numa-meminfo.node0.MemUsed
    235.00 ± 92%    +744.4%       1984 ± 12%  numa-meminfo.node0.Mlocked
    920.17 ± 17%     -89.6%      95.80 ± 12%  numa-meminfo.node0.Writeback
     74723 ±  8%     -87.6%       9251 ± 38%  numa-meminfo.node1.Active
     70405 ±  9%     -87.5%       8827 ± 39%  numa-meminfo.node1.Active(anon)
      4317 ± 21%     -90.2%     423.80 ±  2%  numa-meminfo.node1.Active(file)
     18633 ±  6%     -42.8%      10659 ±  7%  numa-meminfo.node1.Dirty
    981253 ±112%     -94.9%      50077 ±  9%  numa-meminfo.node1.FilePages
     55036 ± 40%     -36.6%      34910 ±  7%  numa-meminfo.node1.KReclaimable
      7834 ± 14%     -23.0%       6029 ±  6%  numa-meminfo.node1.KernelStack
   7805195 ± 14%     -50.0%    3903381 ±  4%  numa-meminfo.node1.MemUsed
     55036 ± 40%     -36.6%      34910 ±  7%  numa-meminfo.node1.SReclaimable
     76345 ±  9%     -84.2%      12082 ± 44%  numa-meminfo.node1.Shmem
    747.00 ± 39%     -89.9%      75.80 ± 12%  numa-meminfo.node1.Writeback
      1085 ± 24%     -41.2%     638.60        numa-vmstat.node0.nr_active_file
      4966 ±  4%     -47.6%       2600 ±  5%  numa-vmstat.node0.nr_dirty
     58.50 ± 91%    +747.2%     495.60 ± 12%  numa-vmstat.node0.nr_mlock
    215.00 ±  9%     -88.8%      24.00 ± 16%  numa-vmstat.node0.nr_writeback
      1085 ± 24%     -41.2%     638.60        numa-vmstat.node0.nr_zone_active_file
      4156 ±  6%     -37.3%       2604 ±  5%  numa-vmstat.node0.nr_zone_write_pending
     17584 ±  9%     -87.5%       2206 ± 39%  numa-vmstat.node1.nr_active_anon
      1076 ± 21%     -90.2%     105.60 ±  2%  numa-vmstat.node1.nr_active_file
   2286775 ±  3%     +15.2%    2634474 ±  4%  numa-vmstat.node1.nr_dirtied
      4821 ±  6%     -44.7%       2667 ±  7%  numa-vmstat.node1.nr_dirty
    245329 ±112%     -94.9%      12529 ±  9%  numa-vmstat.node1.nr_file_pages
      7835 ± 14%     -23.1%       6027 ±  6%  numa-vmstat.node1.nr_kernel_stack
     19089 ±  9%     -84.1%       3027 ± 44%  numa-vmstat.node1.nr_shmem
     13753 ± 40%     -36.5%       8727 ±  7%  numa-vmstat.node1.nr_slab_reclaimable
    246.33 ± 28%     -92.5%      18.40 ± 10%  numa-vmstat.node1.nr_writeback
   2275787 ±  3%     +15.2%    2622530 ±  4%  numa-vmstat.node1.nr_written
     17584 ±  9%     -87.5%       2206 ± 39%  numa-vmstat.node1.nr_zone_active_anon
      1076 ± 21%     -90.2%     105.60 ±  2%  numa-vmstat.node1.nr_zone_active_file
      4169 ±  4%     -36.2%       2661 ±  7%  numa-vmstat.node1.nr_zone_write_pending
   2792580 ± 14%     -15.8%    2350375 ± 10%  numa-vmstat.node1.numa_hit
   2728014 ± 15%     -15.5%    2305184 ± 11%  numa-vmstat.node1.numa_local
     17918 ±  9%     -79.1%       3748        proc-vmstat.nr_active_anon
      2167           -65.6%     744.80        proc-vmstat.nr_active_file
     65311            -4.4%      62416        proc-vmstat.nr_anon_pages
      9540 ±  6%     -44.8%       5268        proc-vmstat.nr_dirty
   2857588            +5.5%    3015038        proc-vmstat.nr_dirty_background_threshold
   5722164            +5.5%    6037448        proc-vmstat.nr_dirty_threshold
    681038            -2.5%     663687        proc-vmstat.nr_file_pages
  28750952            +5.5%   30329739        proc-vmstat.nr_free_pages
     69127            -5.7%      65189        proc-vmstat.nr_inactive_anon
     18887            -3.0%      18316        proc-vmstat.nr_inactive_file
     18093           -10.5%      16191        proc-vmstat.nr_kernel_stack
     10227           -17.4%       8448        proc-vmstat.nr_mapped
     81.00 ± 72%    +511.9%     495.60 ± 12%  proc-vmstat.nr_mlock
      1580 ±  2%      -7.2%       1466        proc-vmstat.nr_page_table_pages
     21781 ±  8%     -70.4%       6442        proc-vmstat.nr_shmem
     33505            -8.8%      30558        proc-vmstat.nr_slab_reclaimable
     46901            -1.7%      46101        proc-vmstat.nr_slab_unreclaimable
    339.17 ± 38%     -96.3%      12.60 ± 10%  proc-vmstat.nr_writeback
     17918 ±  9%     -79.1%       3748        proc-vmstat.nr_zone_active_anon
      2167           -65.6%     744.80        proc-vmstat.nr_zone_active_file
     69127            -5.7%      65189        proc-vmstat.nr_zone_inactive_anon
     18887            -3.0%      18316        proc-vmstat.nr_zone_inactive_file
      8278           -36.4%       5268        proc-vmstat.nr_zone_write_pending
     10958 ± 25%     -81.7%       2002 ± 12%  proc-vmstat.numa_hint_faults
      7596 ± 29%     -87.0%     990.80 ± 19%  proc-vmstat.numa_hint_faults_local
   7398687            -6.4%    6925087        proc-vmstat.numa_hit
   7319007            -6.5%    6845367        proc-vmstat.numa_local
     20920 ± 68%    +188.4%      60327 ± 33%  proc-vmstat.numa_pte_updates
     30957 ±  7%     +39.1%      43059 ±  3%  proc-vmstat.pgactivate
   7406644            -6.4%    6932257        proc-vmstat.pgalloc_normal
    431922          +650.0%    3239543        proc-vmstat.pgfault
   1772503          +114.9%    3808790        proc-vmstat.pgfree
     25835 ±  2%    +719.6%     211757        proc-vmstat.pgreuse
     13069 ±  3%     -10.9%      11643 ±  8%  proc-vmstat.pgrotated
 5.548e+09           -95.3%  2.628e+08        perf-stat.i.branch-instructions
  55918924           -92.9%    3997072 ±  8%  perf-stat.i.branch-misses
  66195997           -95.4%    3047429 ±  9%  perf-stat.i.cache-misses
 3.163e+08           -94.7%   16660905 ±  7%  perf-stat.i.cache-references
    861052           -95.5%      39002 ±  3%  perf-stat.i.context-switches
      6.78           -57.1%       2.91 ±  3%  perf-stat.i.cpi
 1.793e+11           -97.9%  3.738e+09 ±  3%  perf-stat.i.cpu-cycles
    104239           -99.8%     162.98 ±  4%  perf-stat.i.cpu-migrations
      2764 ±  7%     -54.0%       1270 ±  6%  perf-stat.i.cycles-between-cache-misses
      0.12 ± 27%      -0.1        0.03 ± 61%  perf-stat.i.dTLB-load-miss-rate%
   3833742 ±  5%     -97.6%      92195 ± 57%  perf-stat.i.dTLB-load-misses
 7.077e+09           -94.8%  3.649e+08        perf-stat.i.dTLB-loads
      0.03 ± 32%      -0.0        0.01 ± 36%  perf-stat.i.dTLB-store-miss-rate%
    174720 ± 13%     -90.0%      17449 ± 33%  perf-stat.i.dTLB-store-misses
 1.662e+09           -89.0%  1.832e+08        perf-stat.i.dTLB-stores
   5944634           -82.7%    1025857 ±  2%  perf-stat.i.iTLB-load-misses
  10113965           -83.5%    1663970 ±  4%  perf-stat.i.iTLB-loads
 2.571e+10           -94.9%    1.3e+09        perf-stat.i.instructions
      4235 ±  2%     -70.0%       1271 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.17 ±  6%    +106.3%       0.35 ±  2%  perf-stat.i.ipc
      1.38 ± 11%     -83.1%       0.23 ± 12%  perf-stat.i.major-faults
      2.04           -97.9%       0.04 ±  3%  perf-stat.i.metric.GHz
    712.31 ±  4%     -68.2%     226.18 ±  6%  perf-stat.i.metric.K/sec
    165.84           -94.4%       9.21        perf-stat.i.metric.M/sec
      3631           -21.8%       2838        perf-stat.i.minor-faults
     90.38            -3.9       86.46        perf-stat.i.node-load-miss-rate%
  29619314           -96.7%     968983 ±  9%  perf-stat.i.node-load-misses
   2373725           -93.7%     149616 ±  4%  perf-stat.i.node-loads
     88.58            -4.0       84.56        perf-stat.i.node-store-miss-rate%
  12276582           -96.4%     444622 ± 10%  perf-stat.i.node-store-misses
    963137           -91.8%      78504 ±  3%  perf-stat.i.node-stores
      3632           -21.9%       2838        perf-stat.i.page-faults
      1.01            +0.5        1.52 ±  9%  perf-stat.overall.branch-miss-rate%
     20.93            -2.6       18.35 ±  9%  perf-stat.overall.cache-miss-rate%
      6.98           -58.8%       2.88 ±  3%  perf-stat.overall.cpi
      2709           -54.5%       1233 ±  6%  perf-stat.overall.cycles-between-cache-misses
      0.05 ±  4%      -0.0        0.03 ± 58%  perf-stat.overall.dTLB-load-miss-rate%
      4325           -70.7%       1267 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.14          +142.8%       0.35 ±  3%  perf-stat.overall.ipc
     92.58            -6.0       86.56        perf-stat.overall.node-load-miss-rate%
     92.73            -7.9       84.87        perf-stat.overall.node-store-miss-rate%
 5.484e+09           -95.2%  2.625e+08        perf-stat.ps.branch-instructions
  55264217           -92.8%    3993550 ±  8%  perf-stat.ps.branch-misses
  65436484           -95.3%    3044597 ±  9%  perf-stat.ps.cache-misses
 3.127e+08           -94.7%   16645820 ±  7%  perf-stat.ps.cache-references
    851195           -95.4%      38965 ±  3%  perf-stat.ps.context-switches
     86980            +1.1%      87919        perf-stat.ps.cpu-clock
 1.773e+11           -97.9%  3.735e+09 ±  3%  perf-stat.ps.cpu-cycles
    103047           -99.8%     162.83 ±  4%  perf-stat.ps.cpu-migrations
   3790109 ±  5%     -97.6%      92136 ± 57%  perf-stat.ps.dTLB-load-misses
 6.996e+09           -94.8%  3.646e+08        perf-stat.ps.dTLB-loads
    172721 ± 13%     -89.9%      17436 ± 33%  perf-stat.ps.dTLB-store-misses
 1.643e+09           -88.9%   1.83e+08        perf-stat.ps.dTLB-stores
   5876380           -82.6%    1024904 ±  2%  perf-stat.ps.iTLB-load-misses
   9997900           -83.4%    1662418 ±  4%  perf-stat.ps.iTLB-loads
 2.541e+10           -94.9%  1.298e+09        perf-stat.ps.instructions
      1.36 ± 11%     -82.8%       0.23 ± 12%  perf-stat.ps.major-faults
      3584           -20.9%       2835        perf-stat.ps.minor-faults
  29279822           -96.7%     968074 ±  9%  perf-stat.ps.node-load-misses
   2346886           -93.6%     149475 ±  4%  perf-stat.ps.node-loads
  12135036           -96.3%     444204 ± 10%  perf-stat.ps.node-store-misses
    951718           -91.8%      78431 ±  3%  perf-stat.ps.node-stores
      3585           -20.9%       2836        perf-stat.ps.page-faults
     86980            +1.1%      87919        perf-stat.ps.task-clock
 2.202e+12 ±  2%     -36.1%  1.408e+12        perf-stat.total.instructions
     20238 ± 12%    +545.4%     130631 ±  5%  softirqs.CPU0.RCU
     34891 ±  4%    +303.1%     140655        softirqs.CPU0.SCHED
    744.17 ± 33%   +2961.8%      22785 ±  9%  softirqs.CPU0.TIMER
     18727 ± 10%    +611.6%     133257 ±  3%  softirqs.CPU1.RCU
     30977 ±  2%    +336.3%     135155        softirqs.CPU1.SCHED
    258.67 ± 73%  +10937.8%      28551 ±  7%  softirqs.CPU1.TIMER
     18167 ± 12%    +625.7%     131849 ±  3%  softirqs.CPU10.RCU
     30191 ±  2%    +353.2%     136817        softirqs.CPU10.SCHED
     86.67 ± 15%  +34545.8%      30026 ±  3%  softirqs.CPU10.TIMER
     18410 ± 10%    +624.0%     133302 ±  3%  softirqs.CPU11.RCU
     30610 ±  2%    +344.6%     136107        softirqs.CPU11.SCHED
    556.00 ±126%   +5207.8%      29511 ±  5%  softirqs.CPU11.TIMER
     17781 ± 11%    +637.2%     131093 ±  3%  softirqs.CPU12.RCU
     30308 ±  2%    +346.8%     135412        softirqs.CPU12.SCHED
     97.00 ± 21%  +29784.3%      28987 ±  6%  softirqs.CPU12.TIMER
     17954 ± 12%    +631.4%     131307 ±  3%  softirqs.CPU13.RCU
     30696 ±  3%    +343.2%     136053        softirqs.CPU13.SCHED
    217.67 ±104%  +13463.7%      29523 ±  5%  softirqs.CPU13.TIMER
     18246 ± 13%    +622.9%     131897 ±  3%  softirqs.CPU14.RCU
     30213 ±  2%    +352.5%     136719        softirqs.CPU14.SCHED
    122.67 ± 40%  +23727.8%      29228 ±  5%  softirqs.CPU14.TIMER
     19208 ± 18%    +572.6%     129185 ±  4%  softirqs.CPU15.RCU
     30368 ±  3%    +352.7%     137473        softirqs.CPU15.SCHED
    415.33 ±108%   +7127.1%      30016 ±  6%  softirqs.CPU15.TIMER
     18826 ± 17%    +577.4%     127529 ±  4%  softirqs.CPU16.RCU
     30350 ±  3%    +347.9%     135938        softirqs.CPU16.SCHED
    301.33 ±161%   +9351.3%      28479 ±  6%  softirqs.CPU16.TIMER
     17786 ± 11%    +623.9%     128750 ±  5%  softirqs.CPU17.RCU
     30476 ±  2%    +346.8%     136173        softirqs.CPU17.SCHED
     97.33 ± 18%  +29807.7%      29110 ± 11%  softirqs.CPU17.TIMER
     17818 ± 12%    +623.0%     128820 ±  4%  softirqs.CPU18.RCU
     30410 ±  2%    +352.0%     137438        softirqs.CPU18.SCHED
    170.67 ±113%  +16756.4%      28768 ± 10%  softirqs.CPU18.TIMER
     17903 ± 13%    +615.3%     128069 ±  4%  softirqs.CPU19.RCU
     30433 ±  2%    +350.2%     137014        softirqs.CPU19.SCHED
    122.83 ± 67%  +23461.8%      28941 ±  9%  softirqs.CPU19.TIMER
     18212 ± 12%    +616.6%     130498 ±  5%  softirqs.CPU2.RCU
     30954 ±  2%    +341.0%     136497        softirqs.CPU2.SCHED
    148.33 ± 38%  +19549.0%      29146 ±  6%  softirqs.CPU2.TIMER
     17784 ± 12%    +624.8%     128899 ±  4%  softirqs.CPU20.RCU
     30344          +355.3%     138155        softirqs.CPU20.SCHED
     93.00 ± 11%  +29803.7%      27810 ±  6%  softirqs.CPU20.TIMER
     17742 ± 12%    +599.9%     124179 ±  4%  softirqs.CPU21.RCU
     30098 ±  2%    +362.7%     139272 ±  3%  softirqs.CPU21.SCHED
     93.50 ± 18%  +30266.2%      28392 ± 16%  softirqs.CPU21.TIMER
     18003 ± 14%    +596.2%     125336 ±  7%  softirqs.CPU22.RCU
     30976 ±  4%    +350.0%     139378        softirqs.CPU22.SCHED
    696.33 ±174%   +3544.6%      25378 ±  7%  softirqs.CPU22.TIMER
     17581 ± 14%    +605.7%     124078 ±  7%  softirqs.CPU23.RCU
     30312 ±  2%    +350.0%     136410        softirqs.CPU23.SCHED
    117.17 ± 55%  +22294.4%      26238 ±  7%  softirqs.CPU23.TIMER
     17613 ± 14%    +604.8%     124145 ±  7%  softirqs.CPU24.RCU
     30397 ±  2%    +350.6%     136973        softirqs.CPU24.SCHED
     84.33 ±  8%  +30148.5%      25509 ±  7%  softirqs.CPU24.TIMER
     17759 ± 16%    +604.1%     125052 ±  6%  softirqs.CPU25.RCU
     30682 ±  4%    +344.6%     136416        softirqs.CPU25.SCHED
     95.67 ± 24%  +29101.2%      27935 ± 12%  softirqs.CPU25.TIMER
     17599 ± 14%    +574.3%     118670 ± 13%  softirqs.CPU26.RCU
     30492 ±  4%    +361.6%     140770 ±  2%  softirqs.CPU26.SCHED
    260.17 ±155%  +11910.0%      31246 ± 16%  softirqs.CPU26.TIMER
     17593 ± 13%    +606.9%     124362 ±  6%  softirqs.CPU27.RCU
     30522 ±  3%    +350.8%     137587        softirqs.CPU27.SCHED
    152.33 ± 90%  +18049.4%      27647 ± 17%  softirqs.CPU27.TIMER
     17394 ± 13%    +605.1%     122648 ±  5%  softirqs.CPU28.RCU
     30372 ±  2%    +341.2%     134008        softirqs.CPU28.SCHED
     89.00 ± 21%  +32958.0%      29421 ±  9%  softirqs.CPU28.TIMER
     17543 ± 13%    +604.4%     123578 ±  5%  softirqs.CPU29.RCU
     30382 ±  2%    +338.8%     133321 ±  2%  softirqs.CPU29.SCHED
     79.50 ±  4%  +37446.7%      29849 ± 17%  softirqs.CPU29.TIMER
     18044 ± 10%    +627.1%     131204 ±  4%  softirqs.CPU3.RCU
     30669 ±  2%    +346.3%     136873        softirqs.CPU3.SCHED
    126.50 ± 42%  +23344.6%      29657 ±  4%  softirqs.CPU3.TIMER
     17540 ± 12%    +646.4%     130918 ±  5%  softirqs.CPU30.RCU
     30367 ±  2%    +347.2%     135815        softirqs.CPU30.SCHED
     80.17 ± 10%  +36352.1%      29222 ± 11%  softirqs.CPU30.TIMER
     17648 ± 12%    +627.0%     128308 ±  4%  softirqs.CPU31.RCU
     30396 ±  3%    +343.2%     134723        softirqs.CPU31.SCHED
     80.83 ±  5%  +37019.3%      30004 ± 13%  softirqs.CPU31.TIMER
     17410 ± 12%    +650.5%     130659 ±  4%  softirqs.CPU32.RCU
     30339 ±  3%    +351.1%     136869        softirqs.CPU32.SCHED
    111.17 ± 49%  +26499.2%      29569 ± 16%  softirqs.CPU32.TIMER
     17513 ± 13%    +648.5%     131085 ±  4%  softirqs.CPU33.RCU
     30400 ±  2%    +343.5%     134832        softirqs.CPU33.SCHED
     89.50 ± 30%  +33473.2%      30048 ± 14%  softirqs.CPU33.TIMER
     17512 ± 14%    +656.6%     132505 ±  5%  softirqs.CPU34.RCU
     30439 ±  2%    +348.1%     136385        softirqs.CPU34.SCHED
     88.67 ± 20%  +32096.1%      28547 ± 14%  softirqs.CPU34.TIMER
     17654 ± 15%    +643.1%     131185 ±  4%  softirqs.CPU35.RCU
     30294 ±  2%    +349.7%     136246        softirqs.CPU35.SCHED
     76.67 ±  8%  +37856.5%      29100 ± 17%  softirqs.CPU35.TIMER
     17513 ± 12%    +652.8%     131837 ±  4%  softirqs.CPU36.RCU
     30381 ±  2%    +346.1%     135539        softirqs.CPU36.SCHED
     82.00 ±  7%  +36895.9%      30336 ± 18%  softirqs.CPU36.TIMER
     18848 ± 14%    +597.3%     131436 ±  4%  softirqs.CPU37.RCU
     30850 ±  4%    +346.6%     137783        softirqs.CPU37.SCHED
    744.33 ±130%   +3989.7%      30441 ± 17%  softirqs.CPU37.TIMER
     18747 ± 22%    +582.1%     127886 ±  7%  softirqs.CPU38.RCU
     30610 ±  2%    +350.2%     137803 ±  3%  softirqs.CPU38.SCHED
    289.17 ±160%   +9916.6%      28964 ± 14%  softirqs.CPU38.TIMER
     17555 ± 13%    +649.0%     131495 ±  5%  softirqs.CPU39.RCU
     30534 ±  3%    +347.6%     136677        softirqs.CPU39.SCHED
     78.83 ± 10%  +35106.7%      27754 ±  9%  softirqs.CPU39.TIMER
     18050 ± 11%    +632.5%     132216 ±  3%  softirqs.CPU4.RCU
     30528 ±  2%    +349.1%     137111        softirqs.CPU4.SCHED
    146.00 ± 79%  +20060.5%      29434 ±  3%  softirqs.CPU4.TIMER
     17520 ± 14%    +647.0%     130880 ±  4%  softirqs.CPU40.RCU
     30212 ±  4%    +353.0%     136874        softirqs.CPU40.SCHED
     82.00 ± 11%  +33788.0%      27788 ±  9%  softirqs.CPU40.TIMER
     17510 ± 13%    +644.9%     130432 ±  5%  softirqs.CPU41.RCU
     30481 ±  3%    +346.5%     136088        softirqs.CPU41.SCHED
     79.83 ± 13%  +35440.5%      28373 ± 12%  softirqs.CPU41.TIMER
     17686 ± 15%    +638.9%     130693 ±  5%  softirqs.CPU42.RCU
     30324 ±  2%    +353.3%     137460        softirqs.CPU42.SCHED
     93.00 ± 46%  +29485.6%      27514 ± 13%  softirqs.CPU42.TIMER
     17657 ± 15%    +636.4%     130030 ±  4%  softirqs.CPU43.RCU
     29786 ±  5%    +359.3%     136820 ±  2%  softirqs.CPU43.SCHED
    116.67 ± 58%  +23356.2%      27365 ± 15%  softirqs.CPU43.TIMER
     17110 ± 12%    +495.1%     101831 ±  8%  softirqs.CPU44.RCU
     30487 ±  3%    +335.3%     132722        softirqs.CPU44.SCHED
    304.33 ±153%   +9464.4%      29107 ± 13%  softirqs.CPU44.TIMER
     17690 ± 12%    +660.3%     134495 ±  3%  softirqs.CPU45.RCU
     29884 ±  3%    +343.0%     132383        softirqs.CPU45.SCHED
    263.33 ±128%  +10173.7%      27054 ± 17%  softirqs.CPU45.TIMER
     17707 ± 12%    +644.0%     131735 ±  4%  softirqs.CPU46.RCU
     30169 ±  3%    +342.4%     133476        softirqs.CPU46.SCHED
    125.83 ± 38%  +19918.9%      25190 ± 14%  softirqs.CPU46.TIMER
     17568 ± 12%    +649.9%     131752 ±  2%  softirqs.CPU47.RCU
     30099          +345.5%     134105        softirqs.CPU47.SCHED
    111.00 ± 28%  +21257.7%      23707 ± 13%  softirqs.CPU47.TIMER
     17803 ± 12%    +656.2%     134619 ±  3%  softirqs.CPU48.RCU
     30443 ±  2%    +343.9%     135132        softirqs.CPU48.SCHED
    109.83 ± 28%  +20174.7%      22268 ± 13%  softirqs.CPU48.TIMER
     17775 ± 11%    +663.9%     135787 ±  2%  softirqs.CPU49.RCU
     30209 ±  2%    +345.0%     134435        softirqs.CPU49.SCHED
    120.17 ± 59%  +19445.5%      23487 ± 11%  softirqs.CPU49.TIMER
     18117 ± 11%    +623.1%     131006 ±  5%  softirqs.CPU5.RCU
     30571 ±  2%    +346.8%     136607        softirqs.CPU5.SCHED
    526.17 ±122%   +5143.9%      27591 ±  7%  softirqs.CPU5.TIMER
     17518 ± 12%    +642.5%     130082 ±  4%  softirqs.CPU50.RCU
     30502 ±  2%    +330.8%     131405        softirqs.CPU50.SCHED
    146.17 ± 82%  +15640.4%      23007 ± 15%  softirqs.CPU50.TIMER
     17853 ± 10%    +625.6%     129543 ±  3%  softirqs.CPU51.RCU
     30707 ±  2%    +331.8%     132590        softirqs.CPU51.SCHED
    383.83 ±149%   +6097.6%      23788 ± 16%  softirqs.CPU51.TIMER
     18124 ± 11%    +640.5%     134202 ±  2%  softirqs.CPU52.RCU
     30588 ±  3%    +340.5%     134747        softirqs.CPU52.SCHED
    103.50 ± 28%  +21598.2%      22457 ±  6%  softirqs.CPU52.TIMER
     18375 ± 12%    +614.4%     131272 ±  8%  softirqs.CPU53.RCU
     30840 ±  3%    +333.7%     133741        softirqs.CPU53.SCHED
    666.33 ±122%   +3388.1%      23242 ± 15%  softirqs.CPU53.TIMER
     17727 ± 11%    +664.5%     135519 ±  2%  softirqs.CPU54.RCU
     30644 ±  3%    +336.7%     133832        softirqs.CPU54.SCHED
    186.67 ± 74%  +11417.6%      21499 ± 11%  softirqs.CPU54.TIMER
     17935 ± 12%    +654.2%     135279 ±  3%  softirqs.CPU55.RCU
     30621 ±  2%    +339.2%     134483        softirqs.CPU55.SCHED
    127.83 ± 47%  +18067.6%      23224 ±  7%  softirqs.CPU55.TIMER
     18049 ± 13%    +655.7%     136406 ±  4%  softirqs.CPU56.RCU
     30798 ±  2%    +335.9%     134239        softirqs.CPU56.SCHED
    374.33 ±116%   +6396.7%      24319 ± 13%  softirqs.CPU56.TIMER
     18010 ± 11%    +632.6%     131948 ±  2%  softirqs.CPU57.RCU
     30647 ±  2%    +337.0%     133914        softirqs.CPU57.SCHED
    104.83 ± 23%  +22630.2%      23828 ±  5%  softirqs.CPU57.TIMER
     19441 ± 12%    +601.9%     136458 ±  3%  softirqs.CPU58.RCU
     30778 ±  2%    +337.6%     134691        softirqs.CPU58.SCHED
    395.17 ±128%   +6047.2%      24291 ±  6%  softirqs.CPU58.TIMER
     17935 ± 12%    +647.6%     134088 ±  3%  softirqs.CPU59.RCU
     30782 ±  2%    +338.5%     134983        softirqs.CPU59.SCHED
    147.67 ± 25%  +14605.6%      21715 ±  5%  softirqs.CPU59.TIMER
     17639 ± 11%    +643.2%     131096 ±  3%  softirqs.CPU6.RCU
     30173 ±  2%    +344.1%     133996        softirqs.CPU6.SCHED
     88.50 ± 10%  +35630.4%      31621 ±  5%  softirqs.CPU6.TIMER
     18067 ± 11%    +573.5%     121692 ±  6%  softirqs.CPU60.RCU
     30532 ±  2%    +335.9%     133087        softirqs.CPU60.SCHED
     91.33 ±  6%  +25680.5%      23546 ±  7%  softirqs.CPU60.TIMER
     18280 ± 12%    +580.6%     124409 ±  5%  softirqs.CPU61.RCU
     30765 ±  2%    +335.0%     133846        softirqs.CPU61.SCHED
    284.50 ± 95%   +8407.8%      24204 ±  8%  softirqs.CPU61.TIMER
     18341 ±  9%    +571.2%     123108 ±  5%  softirqs.CPU62.RCU
     30576 ±  2%    +343.6%     135640        softirqs.CPU62.SCHED
    114.50 ± 38%  +18464.0%      21255 ± 13%  softirqs.CPU62.TIMER
     18376 ± 14%    +570.5%     123207 ±  5%  softirqs.CPU63.RCU
     30646 ±  3%    +340.8%     135097        softirqs.CPU63.SCHED
    198.67 ± 96%  +11210.2%      22469 ± 11%  softirqs.CPU63.TIMER
     18271 ± 11%    +578.0%     123874 ±  5%  softirqs.CPU64.RCU
     31230          +333.6%     135404        softirqs.CPU64.SCHED
    126.33 ± 33%  +17348.8%      22043 ± 10%  softirqs.CPU64.TIMER
     17853 ± 11%    +572.2%     120017 ±  4%  softirqs.CPU65.RCU
     30803 ±  2%    +335.4%     134127        softirqs.CPU65.SCHED
    114.33 ± 40%  +21017.7%      24144 ± 10%  softirqs.CPU65.TIMER
     17368 ± 13%    +633.7%     127435 ±  5%  softirqs.CPU66.RCU
     30481 ±  2%    +346.6%     136116        softirqs.CPU66.SCHED
     83.67 ±  7%  +29218.2%      24529 ± 14%  softirqs.CPU66.TIMER
     17690 ± 14%    +614.2%     126341 ±  4%  softirqs.CPU67.RCU
     30430 ±  2%    +339.6%     133776        softirqs.CPU67.SCHED
    112.50 ± 53%  +23369.9%      26403 ± 10%  softirqs.CPU67.TIMER
     17622 ± 14%    +618.4%     126602 ±  4%  softirqs.CPU68.RCU
     30553 ±  3%    +342.1%     135073        softirqs.CPU68.SCHED
     80.67 ± 13%  +30845.6%      24962 ± 22%  softirqs.CPU68.TIMER
     17497 ± 13%    +617.1%     125481 ±  5%  softirqs.CPU69.RCU
     30564 ±  2%    +339.1%     134218        softirqs.CPU69.SCHED
    166.33 ±112%  +14096.7%      23613 ± 10%  softirqs.CPU69.TIMER
     19231 ± 12%    +577.2%     130245 ±  4%  softirqs.CPU7.RCU
     30555 ±  3%    +341.0%     134753        softirqs.CPU7.SCHED
    706.00 ±130%   +4218.3%      30487 ±  7%  softirqs.CPU7.TIMER
     17595 ± 14%    +583.5%     120260 ± 10%  softirqs.CPU70.RCU
     30568 ±  3%    +344.2%     135776        softirqs.CPU70.SCHED
    111.33 ± 63%  +20961.6%      23448 ± 13%  softirqs.CPU70.TIMER
     17488 ± 14%    +617.3%     125442 ±  5%  softirqs.CPU71.RCU
     30501 ±  2%    +342.3%     134913        softirqs.CPU71.SCHED
     78.67 ±  6%  +28574.4%      22557 ± 13%  softirqs.CPU71.TIMER
     17531 ± 13%    +595.3%     121905 ±  4%  softirqs.CPU72.RCU
     30560 ±  2%    +323.9%     129551        softirqs.CPU72.SCHED
    139.33 ± 90%  +17958.9%      25162 ± 10%  softirqs.CPU72.TIMER
     17496 ± 13%    +607.2%     123738 ±  5%  softirqs.CPU73.RCU
     30551 ±  2%    +326.9%     130417        softirqs.CPU73.SCHED
     82.33 ±  4%  +30216.3%      24960 ±  6%  softirqs.CPU73.TIMER
     17489 ± 13%    +614.7%     125003 ±  5%  softirqs.CPU74.RCU
     30685 ±  3%    +331.0%     132254        softirqs.CPU74.SCHED
     74.17 ±  7%  +32411.4%      24112 ± 18%  softirqs.CPU74.TIMER
     17374 ± 13%    +611.0%     123538 ±  4%  softirqs.CPU75.RCU
     30588 ±  2%    +325.7%     130226        softirqs.CPU75.SCHED
    101.50 ± 34%  +24139.4%      24603 ±  9%  softirqs.CPU75.TIMER
     17735 ± 12%    +620.7%     127817 ±  5%  softirqs.CPU76.RCU
     31071 ±  3%    +327.4%     132804        softirqs.CPU76.SCHED
    469.17 ±178%   +4796.7%      22973 ±  9%  softirqs.CPU76.TIMER
     17540 ± 13%    +609.0%     124353 ±  5%  softirqs.CPU77.RCU
     30672 ±  2%    +327.0%     130966        softirqs.CPU77.SCHED
     78.17 ±  5%  +28156.8%      22087 ± 12%  softirqs.CPU77.TIMER
     17426 ± 12%    +624.4%     126238 ±  5%  softirqs.CPU78.RCU
     30570 ±  2%    +336.0%     133271        softirqs.CPU78.SCHED
    262.67 ± 98%   +8249.1%      21930 ± 11%  softirqs.CPU78.TIMER
     17694 ± 14%    +608.9%     125436 ±  5%  softirqs.CPU79.RCU
     30661 ±  2%    +332.9%     132719        softirqs.CPU79.SCHED
     82.50 ±  7%  +26567.9%      22001 ±  6%  softirqs.CPU79.TIMER
     17717 ± 13%    +646.5%     132257 ±  4%  softirqs.CPU8.RCU
     30416 ±  3%    +352.1%     137523        softirqs.CPU8.SCHED
     85.50 ± 10%  +35103.5%      30099 ±  7%  softirqs.CPU8.TIMER
     17570 ± 13%    +613.0%     125283 ±  5%  softirqs.CPU80.RCU
     30674 ±  2%    +329.1%     131619        softirqs.CPU80.SCHED
     86.17 ± 14%  +24558.7%      21247 ±  8%  softirqs.CPU80.TIMER
     17781 ± 12%    +614.4%     127023 ±  5%  softirqs.CPU81.RCU
     30783 ±  3%    +335.9%     134196        softirqs.CPU81.SCHED
    252.17 ±146%   +7964.8%      20336 ± 12%  softirqs.CPU81.TIMER
     17761 ± 14%    +586.9%     122001 ±  8%  softirqs.CPU82.RCU
     30562 ±  2%    +328.7%     131022        softirqs.CPU82.SCHED
    118.17 ± 77%  +18662.0%      22170 ± 11%  softirqs.CPU82.TIMER
     17329 ± 13%    +621.6%     125057 ±  6%  softirqs.CPU83.RCU
     30721 ±  2%    +334.7%     133546        softirqs.CPU83.SCHED
     86.67 ±  3%  +25490.2%      22178 ± 17%  softirqs.CPU83.TIMER
     17421 ± 12%    +612.0%     124047 ±  6%  softirqs.CPU84.RCU
     30551 ±  2%    +338.9%     134083        softirqs.CPU84.SCHED
     80.17 ±  8%  +24465.1%      19693 ± 18%  softirqs.CPU84.TIMER
     17583 ± 13%    +612.8%     125339 ±  6%  softirqs.CPU85.RCU
     30745 ±  3%    +334.1%     133470        softirqs.CPU85.SCHED
     88.50 ± 10%  +23394.7%      20792 ± 15%  softirqs.CPU85.TIMER
     17534 ± 13%    +621.7%     126550 ±  6%  softirqs.CPU86.RCU
     30617 ±  2%    +340.2%     134765        softirqs.CPU86.SCHED
     88.33 ± 17%  +23353.0%      20716 ± 18%  softirqs.CPU86.TIMER
     17651 ± 13%    +615.2%     126236 ±  6%  softirqs.CPU87.RCU
     30061 ±  2%    +346.4%     134190        softirqs.CPU87.SCHED
    160.67 ±107%  +12279.4%      19889 ± 17%  softirqs.CPU87.TIMER
     18070 ± 13%    +607.2%     127794 ±  9%  softirqs.CPU9.RCU
     30552 ±  4%    +351.2%     137840 ±  2%  softirqs.CPU9.SCHED
    104.17 ± 13%  +29787.1%      31132 ± 13%  softirqs.CPU9.TIMER
      2501 ± 64%  +12459.9%     314163 ±111%  softirqs.NET_RX
   1573110 ± 12%    +616.7%   11274807 ±  4%  softirqs.RCU
   2690306 ±  2%    +342.0%   11891468        softirqs.SCHED
     16482 ±  2%  +13780.1%    2287807        softirqs.TIMER
     86.51           -66.3       20.26 ± 15%  perf-profile.calltrace.cycles-pp.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
     86.93           -65.6       21.37 ± 15%  perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     86.94           -65.5       21.43 ± 15%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     86.97           -65.5       21.51 ± 15%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     86.97           -65.4       21.52 ± 14%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     87.00           -65.4       21.57 ± 14%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     87.00           -65.4       21.57 ± 15%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     87.10           -65.4       21.72 ± 14%  perf-profile.calltrace.cycles-pp.write
     48.48           -41.4        7.05 ± 16%  perf-profile.calltrace.cycles-pp.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
     22.65 ±  7%     -20.0        2.69 ± 16%  perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
     17.22 ±  8%     -17.0        0.25 ±123%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file
     17.17 ±  8%     -16.9        0.24 ±124%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.__submit_merged_write_cond.f2fs_fsync_node_pages
     17.39 ±  8%     -16.9        0.50 ± 53%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
     17.78 ±  2%     -14.1        3.68 ± 15%  perf-profile.calltrace.cycles-pp.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
     21.55 ±  2%     -13.5        8.08 ± 16%  perf-profile.calltrace.cycles-pp.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc.file_write_and_wait_range
     21.58 ±  2%     -13.4        8.21 ± 16%  perf-profile.calltrace.cycles-pp.f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc.file_write_and_wait_range.f2fs_do_sync_file
     21.59 ±  2%     -13.4        8.22 ± 16%  perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
     21.59 ±  2%     -13.4        8.22 ± 16%  perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter
     21.62 ±  2%     -13.4        8.25 ± 16%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
     11.96 ±  3%      -8.5        3.50 ± 19%  perf-profile.calltrace.cycles-pp.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
      8.61 ±  9%      -8.4        0.25 ±122%  perf-profile.calltrace.cycles-pp.f2fs_is_checkpointed_node.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
     12.01 ±  3%      -8.3        3.70 ± 19%  perf-profile.calltrace.cycles-pp.f2fs_write_single_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
      7.29 ±  9%      -7.3        0.00        perf-profile.calltrace.cycles-pp.f2fs_get_node_info.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.f2fs_write_data_pages
      7.12 ±  9%      -7.1        0.00        perf-profile.calltrace.cycles-pp.f2fs_need_dentry_mark.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      6.28 ±  9%      -6.3        0.00        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.f2fs_is_checkpointed_node.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      6.22 ±  9%      -6.2        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.f2fs_is_checkpointed_node.f2fs_do_sync_file.f2fs_file_write_iter
      6.20 ±  9%      -6.2        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.f2fs_is_checkpointed_node.f2fs_do_sync_file
      9.37 ±  9%      -6.1        3.23 ± 20%  perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
      6.39 ± 10%      -5.9        0.51 ± 89%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages
      6.35 ± 10%      -5.9        0.49 ± 89%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.__submit_merged_write_cond.f2fs_write_cache_pages
      6.48 ± 10%      -5.8        0.69 ± 58%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
      5.76 ±  9%      -5.8        0.00        perf-profile.calltrace.cycles-pp.f2fs_need_inode_block_update.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      5.41 ± 10%      -5.4        0.00        perf-profile.calltrace.cycles-pp.f2fs_get_node_info.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      5.34 ±  9%      -5.3        0.00        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.f2fs_get_node_info.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages
      5.29 ±  9%      -5.3        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.f2fs_get_node_info.f2fs_do_write_data_page.f2fs_write_single_data_page
      5.28 ±  9%      -5.3        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.f2fs_get_node_info.f2fs_do_write_data_page
      5.15 ±  9%      -5.2        0.00        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.f2fs_need_dentry_mark.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      5.10 ±  9%      -5.1        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.f2fs_need_dentry_mark.f2fs_fsync_node_pages.f2fs_do_sync_file
      5.08 ±  9%      -5.1        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.f2fs_need_dentry_mark.f2fs_fsync_node_pages
      7.27 ±  6%      -4.7        2.54 ± 13%  perf-profile.calltrace.cycles-pp.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file
      7.27 ±  6%      -4.7        2.55 ± 13%  perf-profile.calltrace.cycles-pp.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      4.37 ±  6%      -2.0        2.35 ± 21%  perf-profile.calltrace.cycles-pp.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages
      4.48 ±  6%      -1.9        2.60 ± 21%  perf-profile.calltrace.cycles-pp.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.f2fs_write_data_pages
      1.59 ±  6%      -1.0        0.57 ± 90%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.__submit_merged_write_cond
      1.86 ±  6%      -1.0        0.87 ± 10%  perf-profile.calltrace.cycles-pp.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages
      1.48 ± 13%      -0.9        0.60 ± 51%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page
      1.31 ±  6%      -0.8        0.50 ± 52%  perf-profile.calltrace.cycles-pp.schedule.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      1.31 ±  6%      -0.8        0.50 ± 52%  perf-profile.calltrace.cycles-pp.__schedule.schedule.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file
      1.72 ±  4%      -0.7        1.00 ± 29%  perf-profile.calltrace.cycles-pp.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page
      1.78 ± 12%      -0.7        1.10 ± 13%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page.__write_node_page
      1.96 ± 11%      -0.6        1.34 ± 14%  perf-profile.calltrace.cycles-pp.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages
      1.82 ± 17%      -0.6        1.22 ± 23%  perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
      1.20 ± 10%      -0.4        0.77 ±  9%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page
      1.33 ± 10%      -0.3        1.03 ±  9%  perf-profile.calltrace.cycles-pp.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page
      0.79 ±  7%      +0.3        1.05 ± 22%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
      0.42 ± 44%      +0.3        0.76 ± 11%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.58 ±  3%      +0.5        1.08 ±  5%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
      0.58 ±  3%      +0.5        1.09 ±  5%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.wait_for_completion
      0.71 ±  2%      +0.6        1.26 ±  5%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.f2fs_issue_flush.f2fs_do_sync_file
      0.70 ±  2%      +0.6        1.26 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_for_completion.f2fs_issue_flush
      0.72 ±  2%      +0.6        1.28 ±  5%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter
      0.17 ±141%      +0.6        0.75 ± 12%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.74 ±  2%      +0.6        1.33 ±  4%  perf-profile.calltrace.cycles-pp.wait_for_completion.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      0.00            +0.6        0.63 ±  9%  perf-profile.calltrace.cycles-pp.submit_bio_noacct.__submit_merged_bio.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file
      0.00            +0.6        0.63 ±  9%  perf-profile.calltrace.cycles-pp.__submit_bio.submit_bio_noacct.__submit_merged_bio.__submit_merged_write_cond.f2fs_fsync_node_pages
      0.00            +0.6        0.63 ± 12%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.00            +0.6        0.64 ± 10%  perf-profile.calltrace.cycles-pp.__submit_merged_bio.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      0.00            +0.6        0.64 ±  6%  perf-profile.calltrace.cycles-pp.issue_flush_thread.kthread.ret_from_fork
      0.00            +0.7        0.65 ±  8%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.7        0.72 ± 14%  perf-profile.calltrace.cycles-pp.congestion_wait.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
      0.00            +0.7        0.74 ±  7%  perf-profile.calltrace.cycles-pp.submit_bio_noacct.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages
      0.00            +0.7        0.74 ±  7%  perf-profile.calltrace.cycles-pp.__submit_bio.submit_bio_noacct.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages
      0.00            +0.8        0.76 ±  6%  perf-profile.calltrace.cycles-pp.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages
      0.00            +0.8        0.77 ± 11%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +0.8        0.79 ± 13%  perf-profile.calltrace.cycles-pp.calc_global_load_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.84 ±  2%      +0.8        1.63        perf-profile.calltrace.cycles-pp.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      0.00            +0.9        0.86 ±  7%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.9        0.95 ± 22%  perf-profile.calltrace.cycles-pp.generic_perform_write.f2fs_buffered_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write
      0.00            +1.0        0.96 ± 22%  perf-profile.calltrace.cycles-pp.f2fs_buffered_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
      0.00            +1.0        1.03 ±  7%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.1        1.06 ±  9%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.1        1.10 ±  9%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.93 ± 18%      +1.2        2.14 ± 17%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      0.94 ± 18%      +1.2        2.17 ± 17%  perf-profile.calltrace.cycles-pp.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      0.00            +1.3        1.31 ± 16%  perf-profile.calltrace.cycles-pp.timekeeping_max_deferment.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.00            +1.7        1.71 ±  7%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +1.7        1.71 ±  7%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.00            +1.7        1.74 ±  7%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.8        1.77 ±  6%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +2.1        2.13 ±  3%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +2.4        2.39 ± 14%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.00            +2.6        2.62 ±  3%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +2.8        2.84 ± 10%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.00            +2.9        2.89 ±  4%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +3.0        3.02 ±  7%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +3.1        3.08 ±  7%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +3.5        3.54 ±  5%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +3.9        3.94 ± 11%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +4.9        4.92 ± 14%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +5.0        5.03 ±  5%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +9.4        9.44 ±  3%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +9.6        9.56 ±  3%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00           +14.5       14.47 ±  3%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.00           +18.0       17.97 ±  5%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     10.30           +33.0       43.28 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     11.03           +55.5       66.51 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     11.04           +56.2       67.20 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     12.16           +62.2       74.38 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     12.16           +62.3       74.42 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     12.16           +62.3       74.42 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     12.31           +62.8       75.08 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     86.51           -66.2       20.26 ± 15%  perf-profile.children.cycles-pp.f2fs_do_sync_file
     86.93           -65.6       21.37 ± 15%  perf-profile.children.cycles-pp.f2fs_file_write_iter
     86.95           -65.5       21.43 ± 15%  perf-profile.children.cycles-pp.new_sync_write
     69.63           -65.5        4.16 ± 34%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     86.97           -65.5       21.51 ± 15%  perf-profile.children.cycles-pp.vfs_write
     86.98           -65.5       21.53 ± 14%  perf-profile.children.cycles-pp.ksys_write
     87.11           -65.4       21.73 ± 14%  perf-profile.children.cycles-pp.write
     87.12           -64.7       22.45 ± 14%  perf-profile.children.cycles-pp.do_syscall_64
     87.12           -64.7       22.46 ± 14%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     69.98           -64.6        5.35 ± 27%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     48.77           -47.0        1.72 ± 36%  perf-profile.children.cycles-pp.prepare_to_wait_event
     48.48           -41.4        7.05 ± 16%  perf-profile.children.cycles-pp.f2fs_fsync_node_pages
     32.09 ±  8%     -26.1        6.00 ± 18%  perf-profile.children.cycles-pp.__submit_merged_write_cond
     17.78 ±  2%     -14.1        3.73 ± 15%  perf-profile.children.cycles-pp.__write_node_page
     21.55 ±  2%     -13.5        8.09 ± 16%  perf-profile.children.cycles-pp.f2fs_write_cache_pages
     21.58 ±  2%     -13.4        8.22 ± 16%  perf-profile.children.cycles-pp.f2fs_write_data_pages
     21.59 ±  2%     -13.4        8.22 ± 16%  perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
     21.62 ±  2%     -13.4        8.25 ± 16%  perf-profile.children.cycles-pp.file_write_and_wait_range
     21.59 ±  2%     -13.4        8.22 ± 16%  perf-profile.children.cycles-pp.do_writepages
     12.72 ±  9%     -12.3        0.43 ± 36%  perf-profile.children.cycles-pp.f2fs_get_node_info
     11.53 ±  3%     -10.6        0.90 ± 37%  perf-profile.children.cycles-pp.finish_wait
     10.06            -8.7        1.34 ± 21%  perf-profile.children.cycles-pp.__wake_up_common_lock
     11.96 ±  3%      -8.5        3.51 ± 19%  perf-profile.children.cycles-pp.f2fs_do_write_data_page
     12.02 ±  3%      -8.3        3.71 ± 18%  perf-profile.children.cycles-pp.f2fs_write_single_data_page
      8.62 ±  9%      -8.2        0.45 ± 36%  perf-profile.children.cycles-pp.f2fs_is_checkpointed_node
      7.13 ±  9%      -6.9        0.20 ± 45%  perf-profile.children.cycles-pp.f2fs_need_dentry_mark
     11.64 ±  6%      -6.7        4.91 ± 16%  perf-profile.children.cycles-pp.do_write_page
      5.77 ±  9%      -5.5        0.27 ± 34%  perf-profile.children.cycles-pp.f2fs_need_inode_block_update
      7.27 ±  6%      -4.7        2.56 ± 13%  perf-profile.children.cycles-pp.f2fs_do_write_node_page
      5.61 ±  6%      -3.5        2.08 ± 19%  perf-profile.children.cycles-pp.osq_lock
      3.02 ±  5%      -2.3        0.76 ± 36%  perf-profile.children.cycles-pp.__mutex_lock
      4.48 ±  6%      -1.9        2.61 ± 21%  perf-profile.children.cycles-pp.f2fs_outplace_write_data
      3.58 ±  5%      -1.7        1.89 ± 20%  perf-profile.children.cycles-pp.f2fs_allocate_data_block
      4.16 ±  8%      -1.2        2.95 ± 13%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      4.62 ±  3%      -1.1        3.53 ±  9%  perf-profile.children.cycles-pp.schedule
      3.61 ±  3%      -1.0        2.63 ± 10%  perf-profile.children.cycles-pp.newidle_balance
      3.76 ±  3%      -0.9        2.85 ±  9%  perf-profile.children.cycles-pp.pick_next_task_fair
      3.29 ± 10%      -0.9        2.38 ± 11%  perf-profile.children.cycles-pp.f2fs_submit_page_write
      5.12 ±  3%      -0.8        4.35 ±  7%  perf-profile.children.cycles-pp.__schedule
      3.32 ±  4%      -0.5        2.80 ±  8%  perf-profile.children.cycles-pp.load_balance
      0.97            -0.4        0.52 ± 15%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.01            -0.4        0.60 ± 13%  perf-profile.children.cycles-pp.__wake_up_common
      0.63            -0.3        0.30 ± 16%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.63            -0.3        0.30 ± 16%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.69 ±  4%      -0.3        0.36 ± 18%  perf-profile.children.cycles-pp.set_node_addr
      0.32 ±  2%      -0.2        0.10 ± 30%  perf-profile.children.cycles-pp.select_task_rq_fair
      1.24            -0.2        1.04 ± 12%  perf-profile.children.cycles-pp.try_to_wake_up
      0.20 ±  4%      -0.2        0.04 ± 82%  perf-profile.children.cycles-pp.rwsem_wake
      0.56            -0.1        0.41 ± 10%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.47            -0.1        0.35 ±  7%  perf-profile.children.cycles-pp.dequeue_entity
      0.18 ±  5%      -0.1        0.07 ± 60%  perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
      0.22 ±  3%      -0.1        0.12 ± 40%  perf-profile.children.cycles-pp.__pagevec_release
      0.36 ±  3%      -0.1        0.26 ± 12%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.25 ±  2%      -0.1        0.16 ± 10%  perf-profile.children.cycles-pp.update_curr
      0.21 ±  3%      -0.1        0.13 ± 32%  perf-profile.children.cycles-pp.release_pages
      0.30 ±  2%      -0.1        0.24 ± 15%  perf-profile.children.cycles-pp.flush_smp_call_function_from_idle
      0.08 ±  4%      -0.0        0.04 ± 83%  perf-profile.children.cycles-pp.f2fs_write_inode
      0.06 ±  9%      +0.0        0.10 ± 28%  perf-profile.children.cycles-pp.evict
      0.06 ±  9%      +0.1        0.11 ± 29%  perf-profile.children.cycles-pp.__fput
      0.06 ±  9%      +0.1        0.11 ± 29%  perf-profile.children.cycles-pp.__dentry_kill
      0.05 ± 45%      +0.1        0.10 ± 28%  perf-profile.children.cycles-pp.f2fs_evict_inode
      0.06 ±  9%      +0.1        0.11 ± 32%  perf-profile.children.cycles-pp.task_work_run
      0.06 ±  9%      +0.1        0.11 ± 20%  perf-profile.children.cycles-pp.__close
      0.10 ±  4%      +0.1        0.15 ± 16%  perf-profile.children.cycles-pp.set_next_entity
      0.02 ± 99%      +0.1        0.08 ± 28%  perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.1        0.06 ± 12%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.09 ±  5%      +0.1        0.15 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.06 ± 25%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.06 ±  9%      +0.1        0.12 ± 27%  perf-profile.children.cycles-pp.dput
      0.04 ± 71%      +0.1        0.10 ± 13%  perf-profile.children.cycles-pp.xas_load
      0.00            +0.1        0.06 ± 23%  perf-profile.children.cycles-pp.__is_cp_guaranteed
      0.15 ±  3%      +0.1        0.22 ± 15%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.07 ± 19%  perf-profile.children.cycles-pp.kmem_cache_free
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.read_counters
      0.07 ±  7%      +0.1        0.14 ± 41%  perf-profile.children.cycles-pp.up_write
      0.08 ±  7%      +0.1        0.15 ± 27%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.__libc_start_main
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.main
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.run_builtin
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.cmd_stat
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.dispatch_events
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.process_interval
      0.02 ±141%      +0.1        0.09 ± 35%  perf-profile.children.cycles-pp.__has_merged_page
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.__bio_alloc
      0.00            +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.00            +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.preempt_schedule_common
      0.00            +0.1        0.08 ± 31%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.1        0.08 ± 34%  perf-profile.children.cycles-pp.tick_program_event
      0.00            +0.1        0.08 ± 29%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.08 ±  4%      +0.1        0.16 ± 21%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.00            +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.f2fs_flush_sit_entries
      0.00            +0.1        0.08 ± 38%  perf-profile.children.cycles-pp.f2fs_sync_node_pages
      0.00            +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.wait_for_completion_io_timeout
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.seq_read_iter
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.f2fs_update_dirty_page
      0.00            +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.1        0.09 ± 23%  perf-profile.children.cycles-pp.mempool_alloc
      0.01 ±223%      +0.1        0.10 ± 32%  perf-profile.children.cycles-pp.locate_dirty_segment
      0.00            +0.1        0.09 ± 38%  perf-profile.children.cycles-pp.menu_reflect
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.timekeeping_advance
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.update_wall_time
      0.00            +0.1        0.09 ± 33%  perf-profile.children.cycles-pp.exec_binprm
      0.00            +0.1        0.09 ± 33%  perf-profile.children.cycles-pp.load_elf_binary
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.f2fs_del_fsync_node_entry
      0.00            +0.1        0.09 ± 19%  perf-profile.children.cycles-pp.sb_mark_inode_writeback
      0.00            +0.1        0.09 ± 29%  perf-profile.children.cycles-pp.copyin
      0.05            +0.1        0.14 ± 21%  perf-profile.children.cycles-pp.mutex_lock
      0.00            +0.1        0.09 ± 21%  perf-profile.children.cycles-pp.__libc_fork
      0.00            +0.1        0.10 ± 50%  perf-profile.children.cycles-pp.do_fault
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.__might_resched
      0.00            +0.1        0.10 ± 33%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.00            +0.1        0.10 ± 26%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.08 ±  6%      +0.1        0.18 ± 17%  perf-profile.children.cycles-pp.update_sit_entry
      0.00            +0.1        0.10 ± 28%  perf-profile.children.cycles-pp.f2fs_issue_checkpoint
      0.00            +0.1        0.10 ± 20%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.1        0.10 ± 28%  perf-profile.children.cycles-pp.bprm_execve
      0.00            +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.submit_bio_checks
      0.00            +0.1        0.10 ± 22%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.00            +0.1        0.10 ± 39%  perf-profile.children.cycles-pp.exit_mmap
      0.00            +0.1        0.11 ± 39%  perf-profile.children.cycles-pp.mmput
      0.00            +0.1        0.11 ± 29%  perf-profile.children.cycles-pp.do_filp_open
      0.00            +0.1        0.11 ± 29%  perf-profile.children.cycles-pp.path_openat
      0.08 ± 11%      +0.1        0.20 ± 31%  perf-profile.children.cycles-pp.__folio_mark_dirty
      0.00            +0.1        0.11 ± 14%  perf-profile.children.cycles-pp.f2fs_get_dnode_of_data
      0.00            +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.new_sync_read
      0.10 ± 10%      +0.1        0.21 ± 18%  perf-profile.children.cycles-pp.__filemap_get_folio
      0.10 ±  8%      +0.1        0.22 ± 20%  perf-profile.children.cycles-pp.pagecache_get_page
      0.00            +0.1        0.12 ±  8%  perf-profile.children.cycles-pp.__cond_resched
      0.00            +0.1        0.12 ± 23%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.f2fs_balance_fs
      0.00            +0.1        0.12 ± 21%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      0.00            +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.block_operations
      0.00            +0.1        0.12 ± 33%  perf-profile.children.cycles-pp.profile_tick
      0.00            +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.do_sys_open
      0.00            +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.do_sys_openat2
      0.07 ±  7%      +0.1        0.19 ± 21%  perf-profile.children.cycles-pp.f2fs_set_data_page_dirty
      0.00            +0.1        0.13 ± 77%  perf-profile.children.cycles-pp.f2fs_inplace_write_data
      0.00            +0.1        0.13 ± 23%  perf-profile.children.cycles-pp.brd_lookup_page
      0.08 ±  5%      +0.1        0.21 ±  9%  perf-profile.children.cycles-pp.down_write
      0.11 ±  3%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.f2fs_submit_merged_ipu_write
      0.13 ±  6%      +0.1        0.26 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.1        0.13 ± 24%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.09 ±  5%      +0.1        0.23 ± 18%  perf-profile.children.cycles-pp.f2fs_is_valid_blkaddr
      0.00            +0.1        0.14 ± 44%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.00            +0.1        0.14 ± 38%  perf-profile.children.cycles-pp.note_gp_changes
      0.14 ± 14%      +0.1        0.28 ± 15%  perf-profile.children.cycles-pp.__folio_start_writeback
      0.00            +0.1        0.14 ± 30%  perf-profile.children.cycles-pp.execve
      0.00            +0.1        0.14 ± 30%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.00            +0.1        0.14 ± 30%  perf-profile.children.cycles-pp.do_execveat_common
      0.00            +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.memcpy_toio
      0.00            +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.00            +0.1        0.14 ± 24%  perf-profile.children.cycles-pp.__queue_work
      0.10 ± 13%      +0.1        0.25 ± 34%  perf-profile.children.cycles-pp.f2fs_update_data_blkaddr
      0.00            +0.1        0.14 ± 18%  perf-profile.children.cycles-pp.read
      0.10 ± 11%      +0.1        0.25 ± 34%  perf-profile.children.cycles-pp.f2fs_set_data_blkaddr
      0.00            +0.1        0.15 ± 25%  perf-profile.children.cycles-pp.queue_work_on
      0.00            +0.1        0.15 ± 19%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.2        0.15 ±  4%  perf-profile.children.cycles-pp.call_cpuidle
      0.00            +0.2        0.15 ± 20%  perf-profile.children.cycles-pp.ksys_read
      0.00            +0.2        0.15 ± 20%  perf-profile.children.cycles-pp.vfs_read
      0.00            +0.2        0.16 ± 20%  perf-profile.children.cycles-pp.error_entry
      0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.timerqueue_del
      0.12 ±  5%      +0.2        0.28 ± 17%  perf-profile.children.cycles-pp.__get_node_page
      0.13 ±  8%      +0.2        0.29 ± 24%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.00            +0.2        0.17 ± 12%  perf-profile.children.cycles-pp.delay_tsc
      0.00            +0.2        0.17 ± 36%  perf-profile.children.cycles-pp.timerqueue_add
      0.11 ±  9%      +0.2        0.29 ± 50%  perf-profile.children.cycles-pp.f2fs_set_node_page_dirty
      0.00            +0.2        0.18 ± 35%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.00            +0.2        0.18 ± 19%  perf-profile.children.cycles-pp.md_flush_request
      0.09 ±  5%      +0.2        0.27 ± 21%  perf-profile.children.cycles-pp.f2fs_write_end
      0.01 ±223%      +0.2        0.19 ± 18%  perf-profile.children.cycles-pp.cpumask_next
      0.22 ±  2%      +0.2        0.40 ± 11%  perf-profile.children.cycles-pp._find_next_bit
      0.00            +0.2        0.19 ± 36%  perf-profile.children.cycles-pp.handle_mm_fault
      0.13 ±  8%      +0.2        0.32 ± 20%  perf-profile.children.cycles-pp.filemap_dirty_folio
      0.00            +0.2        0.20 ± 22%  perf-profile.children.cycles-pp.trigger_load_balance
      0.09 ±  4%      +0.2        0.29 ± 21%  perf-profile.children.cycles-pp.__folio_end_writeback
      0.00            +0.2        0.20 ± 19%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.00            +0.2        0.20 ± 30%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.2        0.20 ± 18%  perf-profile.children.cycles-pp.rcu_dynticks_inc
      0.00            +0.2        0.20 ± 19%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.00            +0.2        0.20 ± 40%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.00            +0.2        0.21 ± 11%  perf-profile.children.cycles-pp.prepare_to_wait
      0.00            +0.2        0.21 ± 40%  perf-profile.children.cycles-pp.exc_page_fault
      0.00            +0.2        0.21 ± 34%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.2        0.21 ± 21%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.2        0.21 ± 22%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.05 ±  7%      +0.2        0.27 ± 21%  perf-profile.children.cycles-pp.raid0_make_request
      0.10 ±  5%      +0.2        0.32 ± 18%  perf-profile.children.cycles-pp.folio_end_writeback
      0.00            +0.2        0.23 ± 34%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.00            +0.2        0.23 ± 16%  perf-profile.children.cycles-pp.submit_flushes
      0.06 ±  7%      +0.3        0.31 ± 15%  perf-profile.children.cycles-pp.submit_bio_wait
      0.00            +0.3        0.25 ± 14%  perf-profile.children.cycles-pp.__sched_text_start
      0.06 ±  7%      +0.3        0.32 ± 13%  perf-profile.children.cycles-pp.blkdev_issue_flush
      0.00            +0.3        0.26 ± 21%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.51            +0.3        0.77 ± 12%  perf-profile.children.cycles-pp.schedule_idle
      0.06 ±  7%      +0.3        0.33 ± 13%  perf-profile.children.cycles-pp.__submit_flush_wait
      0.06 ±  6%      +0.3        0.32 ± 19%  perf-profile.children.cycles-pp.md_handle_request
      0.00            +0.3        0.27 ± 23%  perf-profile.children.cycles-pp.call_timer_fn
      0.00            +0.3        0.29 ± 38%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.3        0.29 ± 38%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.00            +0.3        0.31 ± 19%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.00            +0.3        0.31 ± 20%  perf-profile.children.cycles-pp.__write_checkpoint_sync
      0.00            +0.3        0.31 ± 20%  perf-profile.children.cycles-pp.f2fs_write_checkpoint
      0.18 ±  7%      +0.3        0.49 ± 35%  perf-profile.children.cycles-pp.folio_mark_dirty
      0.00            +0.3        0.32 ± 21%  perf-profile.children.cycles-pp.__checkpoint_and_complete_reqs
      0.28 ±  3%      +0.3        0.61 ± 23%  perf-profile.children.cycles-pp.down_read_trylock
      0.00            +0.3        0.32 ± 21%  perf-profile.children.cycles-pp.issue_checkpoint_thread
      0.07 ±  5%      +0.3        0.40 ± 15%  perf-profile.children.cycles-pp.md_submit_bio
      0.18 ±  4%      +0.3        0.51 ± 29%  perf-profile.children.cycles-pp.f2fs_write_begin
      0.00            +0.3        0.34 ± 12%  perf-profile.children.cycles-pp.run_timer_softirq
      0.00            +0.4        0.35 ± 19%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.17 ±  2%      +0.4        0.54 ± 15%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.4        0.37 ±  6%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.4        0.38 ± 13%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.00            +0.4        0.38 ±  5%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.26 ±  2%      +0.4        0.64 ±  6%  perf-profile.children.cycles-pp.issue_flush_thread
      0.00            +0.4        0.39 ±  9%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.4        0.39 ± 33%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.14 ±  3%      +0.4        0.55 ±  6%  perf-profile.children.cycles-pp.update_blocked_averages
      0.17 ±  4%      +0.4        0.58 ± 16%  perf-profile.children.cycles-pp.f2fs_write_end_io
      0.00            +0.4        0.41 ±  6%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.06 ±  6%      +0.4        0.49 ± 15%  perf-profile.children.cycles-pp.process_one_work
      0.00            +0.4        0.44 ± 10%  perf-profile.children.cycles-pp.io_serial_in
      0.51 ±  4%      +0.4        0.95 ± 10%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.00            +0.4        0.45 ± 10%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.28            +0.5        0.75 ±  7%  perf-profile.children.cycles-pp.brd_do_bvec
      0.28 ±  2%      +0.5        0.76 ±  6%  perf-profile.children.cycles-pp.brd_submit_bio
      0.00            +0.5        0.51 ±  8%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.15 ± 25%      +0.5        0.66 ± 33%  perf-profile.children.cycles-pp.start_kernel
      0.00            +0.5        0.55 ± 14%  perf-profile.children.cycles-pp.tick_irq_enter
      0.08 ±  5%      +0.5        0.63 ± 12%  perf-profile.children.cycles-pp.worker_thread
      0.58 ±  3%      +0.6        1.15 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock
      0.00            +0.6        0.59 ± 14%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.00            +0.6        0.61 ±  6%  perf-profile.children.cycles-pp.wait_for_lsr
      0.00            +0.6        0.62 ±  6%  perf-profile.children.cycles-pp.serial8250_console_write
      0.33 ±  5%      +0.6        0.95 ± 22%  perf-profile.children.cycles-pp.generic_perform_write
      0.33 ±  4%      +0.6        0.96 ± 22%  perf-profile.children.cycles-pp.f2fs_buffered_write_iter
      0.00            +0.7        0.65 ±  6%  perf-profile.children.cycles-pp.irq_work_run
      0.00            +0.7        0.65 ±  6%  perf-profile.children.cycles-pp._printk
      0.00            +0.7        0.65 ±  6%  perf-profile.children.cycles-pp.vprintk_emit
      0.00            +0.7        0.65 ±  6%  perf-profile.children.cycles-pp.console_unlock
      0.00            +0.7        0.66 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.00            +0.7        0.66 ±  7%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.00            +0.7        0.66 ±  7%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.00            +0.7        0.66 ±  7%  perf-profile.children.cycles-pp.irq_work_single
      0.00            +0.7        0.66 ± 10%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.7        0.66 ±  7%  perf-profile.children.cycles-pp.irq_work_run_list
      0.00            +0.7        0.67 ±  7%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.74 ±  2%      +0.7        1.43 ±  5%  perf-profile.children.cycles-pp.wait_for_completion
      0.00            +0.7        0.72 ± 14%  perf-profile.children.cycles-pp.congestion_wait
      0.08 ±  6%      +0.7        0.81 ± 14%  perf-profile.children.cycles-pp.native_sched_clock
      0.84 ±  2%      +0.8        1.63        perf-profile.children.cycles-pp.f2fs_issue_flush
      0.00            +0.8        0.82 ± 13%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.00            +0.8        0.83 ± 11%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.09 ±  5%      +0.8        0.93 ± 13%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.03 ± 70%      +0.9        0.90 ±  9%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.9        0.89 ±  5%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.73 ±  2%      +0.9        1.63 ±  5%  perf-profile.children.cycles-pp.schedule_timeout
      0.44 ±  2%      +1.0        1.46 ±  6%  perf-profile.children.cycles-pp.__submit_merged_bio
      0.00            +1.1        1.08 ±  4%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +1.1        1.10 ±  7%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +1.1        1.13 ±  9%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.93 ± 18%      +1.2        2.15 ± 17%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.94 ± 18%      +1.2        2.17 ± 17%  perf-profile.children.cycles-pp.f2fs_space_for_roll_forward
      0.00            +1.3        1.31 ± 16%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.57 ±  2%      +1.3        1.91 ±  6%  perf-profile.children.cycles-pp.submit_bio_noacct
      0.35 ±  2%      +1.4        1.71 ±  7%  perf-profile.children.cycles-pp.kthread
      0.56            +1.4        1.93 ±  5%  perf-profile.children.cycles-pp.__submit_bio
      0.35 ±  2%      +1.4        1.71 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
      0.12 ±  3%      +1.8        1.87 ±  4%  perf-profile.children.cycles-pp.scheduler_tick
      0.10 ±  3%      +2.2        2.30 ±  4%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.01 ±223%      +2.4        2.41 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.13 ±  4%      +2.7        2.82 ±  5%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.07 ± 10%      +2.8        2.88 ± 10%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.02 ± 99%      +3.0        3.02 ±  4%  perf-profile.children.cycles-pp.clockevents_program_event
      0.16 ±  2%      +3.0        3.17 ±  7%  perf-profile.children.cycles-pp.update_process_times
      0.17            +3.1        3.24 ±  7%  perf-profile.children.cycles-pp.tick_sched_handle
      0.04 ± 44%      +3.5        3.51 ±  7%  perf-profile.children.cycles-pp.ktime_get
      0.18 ±  2%      +3.5        3.71 ±  5%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +4.0        3.97 ± 11%  perf-profile.children.cycles-pp.poll_idle
      0.18 ±  4%      +4.8        4.99 ± 14%  perf-profile.children.cycles-pp.menu_select
      0.23            +5.1        5.30 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.31 ±  2%      +9.6        9.90 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.32 ±  2%      +9.7       10.02 ±  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.46           +14.7       15.20 ±  5%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      1.16 ±  2%     +16.9       18.07 ±  6%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     11.06           +33.3       44.35 ±  3%  perf-profile.children.cycles-pp.intel_idle
     11.17           +56.6       67.73 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter_state
     11.17           +56.6       67.80 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter
     12.16           +62.3       74.42 ±  3%  perf-profile.children.cycles-pp.start_secondary
     12.31           +62.8       75.07 ±  4%  perf-profile.children.cycles-pp.do_idle
     12.31           +62.8       75.08 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     12.31           +62.8       75.08 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
     69.63           -65.5        4.15 ± 35%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      5.58 ±  6%      -3.5        2.08 ± 19%  perf-profile.self.cycles-pp.osq_lock
      0.50            -0.3        0.18 ± 30%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.20 ±  3%      -0.1        0.10 ± 38%  perf-profile.self.cycles-pp.release_pages
      0.24 ±  2%      -0.1        0.16 ± 26%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.13 ±  3%      -0.1        0.05 ± 14%  perf-profile.self.cycles-pp.update_curr
      0.20 ±  2%      -0.1        0.12 ± 22%  perf-profile.self.cycles-pp.update_load_avg
      0.05 ±  7%      +0.1        0.10 ± 28%  perf-profile.self.cycles-pp.__filemap_get_folio
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.irq_enter_rcu
      0.09 ±  4%      +0.1        0.15 ± 10%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.15 ±  4%      +0.1        0.21 ± 15%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.06 ± 23%  perf-profile.self.cycles-pp.__is_cp_guaranteed
      0.00            +0.1        0.07 ± 28%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.1        0.07 ± 18%  perf-profile.self.cycles-pp.md_submit_bio
      0.06 ±  7%      +0.1        0.13 ± 30%  perf-profile.self.cycles-pp.f2fs_do_sync_file
      0.06 ±  7%      +0.1        0.14 ± 43%  perf-profile.self.cycles-pp.up_write
      0.00            +0.1        0.07 ± 25%  perf-profile.self.cycles-pp.__wake_up_common
      0.00            +0.1        0.07 ± 33%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.08 ± 35%  perf-profile.self.cycles-pp.tick_program_event
      0.07 ±  7%      +0.1        0.15 ± 13%  perf-profile.self.cycles-pp.f2fs_allocate_data_block
      0.00            +0.1        0.08 ± 31%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.01 ±223%      +0.1        0.09 ± 35%  perf-profile.self.cycles-pp.__has_merged_page
      0.00            +0.1        0.08 ± 31%  perf-profile.self.cycles-pp.__get_node_page
      0.00            +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.hrtimer_update_next_event
      0.00            +0.1        0.08 ± 30%  perf-profile.self.cycles-pp.f2fs_write_begin
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.1        0.08 ± 26%  perf-profile.self.cycles-pp.f2fs_file_write_iter
      0.00            +0.1        0.09 ± 22%  perf-profile.self.cycles-pp.__write_node_page
      0.00            +0.1        0.09 ± 15%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.1        0.09 ± 39%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.00            +0.1        0.09 ± 33%  perf-profile.self.cycles-pp.cpumask_next
      0.00            +0.1        0.09 ± 19%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.__might_resched
      0.00            +0.1        0.10 ± 26%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.04 ± 44%      +0.1        0.14 ± 21%  perf-profile.self.cycles-pp.mutex_lock
      0.08 ±  6%      +0.1        0.17 ± 15%  perf-profile.self.cycles-pp.update_sit_entry
      0.00            +0.1        0.10 ± 32%  perf-profile.self.cycles-pp.rebalance_domains
      0.02 ±141%      +0.1        0.12 ± 32%  perf-profile.self.cycles-pp.f2fs_submit_page_write
      0.00            +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.f2fs_write_end_io
      0.00            +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.1        0.10 ± 17%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.00            +0.1        0.10 ± 25%  perf-profile.self.cycles-pp.rcu_eqs_exit
      0.00            +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.clockevents_program_event
      0.00            +0.1        0.11 ± 39%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.1        0.11 ± 34%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.08 ±  6%      +0.1        0.18 ±  8%  perf-profile.self.cycles-pp.down_write
      0.00            +0.1        0.11 ± 20%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.23 ±  3%      +0.1        0.34 ± 10%  perf-profile.self.cycles-pp.__schedule
      0.00            +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.submit_flushes
      0.00            +0.1        0.12 ± 43%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.1        0.12 ± 25%  perf-profile.self.cycles-pp.f2fs_balance_fs
      0.00            +0.1        0.12 ± 32%  perf-profile.self.cycles-pp.f2fs_write_data_pages
      0.00            +0.1        0.12 ± 33%  perf-profile.self.cycles-pp.profile_tick
      0.00            +0.1        0.12 ± 25%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.09 ±  5%      +0.1        0.22 ± 18%  perf-profile.self.cycles-pp.f2fs_is_valid_blkaddr
      0.00            +0.1        0.13 ± 26%  perf-profile.self.cycles-pp.error_entry
      0.00            +0.1        0.13 ± 17%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.00            +0.1        0.14 ± 10%  perf-profile.self.cycles-pp.memcpy_toio
      0.00            +0.1        0.14 ±  9%  perf-profile.self.cycles-pp.call_cpuidle
      0.00            +0.1        0.15 ± 15%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.20 ±  2%      +0.2        0.36 ±  8%  perf-profile.self.cycles-pp._find_next_bit
      0.00            +0.2        0.17 ± 12%  perf-profile.self.cycles-pp.delay_tsc
      0.00            +0.2        0.19 ± 21%  perf-profile.self.cycles-pp.rcu_dynticks_inc
      0.00            +0.2        0.19 ± 18%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +0.2        0.19 ± 24%  perf-profile.self.cycles-pp.trigger_load_balance
      0.02 ±141%      +0.2        0.23 ± 15%  perf-profile.self.cycles-pp.update_blocked_averages
      0.01 ±223%      +0.2        0.22 ± 12%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.23 ± 14%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.00            +0.2        0.23 ± 15%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.3        0.26 ± 21%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.28 ±  2%      +0.3        0.55 ± 12%  perf-profile.self.cycles-pp.down_read_trylock
      0.00            +0.3        0.27 ± 41%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.10 ±  3%      +0.3        0.41 ±  8%  perf-profile.self.cycles-pp.brd_do_bvec
      0.00            +0.3        0.31 ±  6%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.4        0.36 ± 34%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.4        0.38 ± 13%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +0.4        0.38 ±  5%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.00            +0.4        0.39 ±  9%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.00            +0.4        0.42 ± 13%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.4        0.44 ± 10%  perf-profile.self.cycles-pp.io_serial_in
      0.50 ±  5%      +0.4        0.95 ± 10%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.00            +0.5        0.53 ± 10%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.00            +0.5        0.55 ± 29%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.00            +0.5        0.55 ± 19%  perf-profile.self.cycles-pp.update_process_times
      0.00            +0.5        0.55 ±  7%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.44 ±  2%      +0.6        1.08 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock
      0.00            +0.6        0.65 ± 10%  perf-profile.self.cycles-pp.read_tsc
      0.58            +0.7        1.27 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.08 ±  6%      +0.7        0.79 ± 15%  perf-profile.self.cycles-pp.native_sched_clock
      0.10 ± 23%      +0.8        0.85 ± 20%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.00            +0.8        0.82 ± 13%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.00            +1.1        1.08 ±  4%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +1.1        1.10 ±  7%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +1.3        1.30 ± 17%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.10 ±  4%      +1.8        1.86 ± 31%  perf-profile.self.cycles-pp.menu_select
      0.00            +2.4        2.36 ± 13%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +3.0        2.98 ±  8%  perf-profile.self.cycles-pp.ktime_get
      0.00            +3.9        3.91 ± 11%  perf-profile.self.cycles-pp.poll_idle
     10.98           +32.9       43.88 ±  3%  perf-profile.self.cycles-pp.intel_idle




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.17.0-rc1-00037-g26ababe97224" of type "text/plain" (174625 bytes)

View attachment "job-script" of type "text/plain" (8205 bytes)

View attachment "job.yaml" of type "text/plain" (5561 bytes)

View attachment "reproduce" of type "text/plain" (1009 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ