lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <YsL4FPFyYnxgIP4D@xsang-OptiPlex-9020>
Date:   Mon, 4 Jul 2022 22:24:20 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Tianchen Ding <dtcccc@...ux.alibaba.com>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Valentin Schneider <vschneid@...hat.com>,
        LKML <linux-kernel@...r.kernel.org>, x86@...nel.org,
        lkp@...ts.01.org, lkp@...el.com, ying.huang@...el.com,
        feng.tang@...el.com, zhengjun.xing@...ux.intel.com,
        fengwei.yin@...el.com, aubrey.li@...ux.intel.com,
        yu.c.chen@...el.com
Subject: [sched]  f3dd3f6745:
 fxmark.ssd_ext4_no_jnl_DWSL_72_directio.works/sec -33.0% regression



Greeting,

FYI, we noticed a -33.0% regression of fxmark.ssd_ext4_no_jnl_DWSL_72_directio.works/sec due to commit:


commit: f3dd3f674555bd9455c5ae7fafce0696bd9931b3 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core

in testcase: fxmark
on test machine: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with 128G memory
with following parameters:

	disk: 1SSD
	media: ssd
	test: DWSL
	fstype: ext4_no_jnl
	directio: directio
	cpufreq_governor: performance
	ucode: 0xd000331

test-description: FxMark is a filesystem benchmark that test multicore scalability.
test-url: https://github.com/sslab-gatech/fxmark

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_tps 1.3% improvement                                |
| test machine     | 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G memory |
| test parameters  | cluster=cs-localhost                                                            |
|                  | cpufreq_governor=performance                                                    |
|                  | ip=ipv4                                                                         |
|                  | nr_threads=25%                                                                  |
|                  | runtime=300s                                                                    |
|                  | test=UDP_RR                                                                     |
|                  | ucode=0xd000331                                                                 |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min 246.4% improvement                                      |
| test machine     | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | disk=4BRD_12G                                                                   |
|                  | fs=f2fs                                                                         |
|                  | load=100                                                                        |
|                  | md=RAID1                                                                        |
|                  | test=sync_disk_rw                                                               |
|                  | ucode=0x500320a                                                                 |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min 3.9% improvement                                        |
| test machine     | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | disk=4BRD_12G                                                                   |
|                  | fs=xfs                                                                          |
|                  | load=300                                                                        |
|                  | md=RAID1                                                                        |
|                  | test=sync_disk_rw                                                               |
|                  | ucode=0x500320a                                                                 |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput 7.2% improvement                                |
| test machine     | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | ipc=pipe                                                                        |
|                  | iterations=8                                                                    |
|                  | mode=process                                                                    |
|                  | nr_threads=50%                                                                  |
|                  | ucode=0x500320a                                                                 |
+------------------+---------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-11/performance/directio/1SSD/ext4_no_jnl/x86_64-rhel-8.3/ssd/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp5/DWSL/fxmark/0xd000331

commit: 
  28156108fe ("sched: Fix the check of nr_running at queue wakelist")
  f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    518.88 ±  5%     -99.9%       0.48        fxmark.ssd_ext4_no_jnl_DWSL_18_directio.idle_sec
     57.91 ±  5%     -99.9%       0.05        fxmark.ssd_ext4_no_jnl_DWSL_18_directio.idle_util
    348.18 ±  8%    +148.6%     865.70        fxmark.ssd_ext4_no_jnl_DWSL_18_directio.iowait_sec
     38.86 ±  8%    +148.6%      96.59        fxmark.ssd_ext4_no_jnl_DWSL_18_directio.iowait_util
      3.62           -25.6%       2.70 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_18_directio.softirq_sec
      0.40           -25.6%       0.30 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_18_directio.softirq_util
     37.96 ±  2%     -99.9%       0.03        fxmark.ssd_ext4_no_jnl_DWSL_2_directio.idle_sec
     38.38 ±  2%     -99.9%       0.03        fxmark.ssd_ext4_no_jnl_DWSL_2_directio.idle_util
     53.08 ±  2%     +72.0%      91.31        fxmark.ssd_ext4_no_jnl_DWSL_2_directio.iowait_sec
     53.66 ±  2%     +71.6%      92.06        fxmark.ssd_ext4_no_jnl_DWSL_2_directio.iowait_util
      0.98 ±  5%     -15.1%       0.83 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_2_directio.softirq_sec
      0.99 ±  5%     -15.4%       0.84 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_2_directio.softirq_util
    865.64 ±  5%     -99.9%       0.98        fxmark.ssd_ext4_no_jnl_DWSL_36_directio.idle_sec
     48.76 ±  5%     -99.9%       0.06        fxmark.ssd_ext4_no_jnl_DWSL_36_directio.idle_util
    854.04 ±  5%    +100.5%       1712        fxmark.ssd_ext4_no_jnl_DWSL_36_directio.iowait_sec
     48.11 ±  5%    +100.5%      96.47        fxmark.ssd_ext4_no_jnl_DWSL_36_directio.iowait_util
      8.44           -46.7%       4.50 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_36_directio.softirq_sec
      0.48           -46.7%       0.25 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_36_directio.softirq_util
     28.81           +31.1%      37.76 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_36_directio.sys_sec
      1.62           +31.1%       2.13 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_36_directio.sys_util
    975762           -23.4%     747169 ±  3%  fxmark.ssd_ext4_no_jnl_DWSL_36_directio.works
     19514           -23.4%      14942 ±  3%  fxmark.ssd_ext4_no_jnl_DWSL_36_directio.works/sec
    112.89           -99.9%       0.08 ±  5%  fxmark.ssd_ext4_no_jnl_DWSL_4_directio.idle_sec
     56.95           -99.9%       0.04 ±  5%  fxmark.ssd_ext4_no_jnl_DWSL_4_directio.idle_util
     75.46 ±  2%    +150.0%     188.64        fxmark.ssd_ext4_no_jnl_DWSL_4_directio.iowait_sec
     38.07          +149.7%      95.07        fxmark.ssd_ext4_no_jnl_DWSL_4_directio.iowait_util
      1.34 ±  3%     -20.2%       1.07 ±  4%  fxmark.ssd_ext4_no_jnl_DWSL_4_directio.softirq_sec
      0.68 ±  3%     -20.3%       0.54 ±  4%  fxmark.ssd_ext4_no_jnl_DWSL_4_directio.softirq_util
    508.25 ±  9%     -99.7%       1.46        fxmark.ssd_ext4_no_jnl_DWSL_54_directio.idle_sec
     19.07 ±  9%     -99.7%       0.05        fxmark.ssd_ext4_no_jnl_DWSL_54_directio.idle_util
      2075 ±  2%     +25.1%       2595        fxmark.ssd_ext4_no_jnl_DWSL_54_directio.iowait_sec
     77.86 ±  2%     +24.5%      96.91        fxmark.ssd_ext4_no_jnl_DWSL_54_directio.iowait_util
      9.07 ±  4%     -31.1%       6.26 ±  4%  fxmark.ssd_ext4_no_jnl_DWSL_54_directio.softirq_sec
      0.34 ±  4%     -31.4%       0.23 ±  4%  fxmark.ssd_ext4_no_jnl_DWSL_54_directio.softirq_util
    816890           -29.5%     575621 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_54_directio.works
     16336           -29.5%      11511 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_54_directio.works/sec
    507.19 ± 10%     -99.6%       1.97        fxmark.ssd_ext4_no_jnl_DWSL_72_directio.idle_sec
     14.21 ± 10%     -99.6%       0.06        fxmark.ssd_ext4_no_jnl_DWSL_72_directio.idle_util
      2974           +17.2%       3486        fxmark.ssd_ext4_no_jnl_DWSL_72_directio.iowait_sec
     83.34           +16.9%      97.44        fxmark.ssd_ext4_no_jnl_DWSL_72_directio.iowait_util
     31.28           +23.0%      38.50 ±  8%  fxmark.ssd_ext4_no_jnl_DWSL_72_directio.irq_sec
      0.88           +22.7%       1.08 ±  8%  fxmark.ssd_ext4_no_jnl_DWSL_72_directio.irq_util
      9.45           -29.6%       6.66 ±  3%  fxmark.ssd_ext4_no_jnl_DWSL_72_directio.softirq_sec
      0.26           -29.7%       0.19 ±  3%  fxmark.ssd_ext4_no_jnl_DWSL_72_directio.softirq_util
    826258           -33.0%     553194 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_72_directio.works
     16523           -33.0%      11062 ±  2%  fxmark.ssd_ext4_no_jnl_DWSL_72_directio.works/sec
   1930279 ±  3%      -6.2%    1811239 ±  2%  fxmark.time.voluntary_context_switches
  55807773           -11.6%   49357289 ±  4%  cpuidle..usage
  53054978           -11.7%   46862001        proc-vmstat.pgpgout
    830.64 ±  3%     -21.2%     654.40 ±  5%  uptime.idle
     46.17 ±  2%     -46.7%      24.59 ±  2%  iostat.cpu.idle
     48.27 ±  2%     +44.6%      69.80        iostat.cpu.iowait
     47.91 ±  2%     -22.1       25.77 ±  2%  mpstat.cpu.all.idle%
     47.73 ±  2%     +22.1       69.87        mpstat.cpu.all.iowait%
      0.39 ±  3%      -0.1        0.31 ±  2%  mpstat.cpu.all.soft%
     98555           -11.8%      86971 ±  3%  vmstat.io.bo
    102932           -19.4%      82934 ±  2%  vmstat.system.cs
     76098            -7.5%      70394 ±  6%  vmstat.system.in
    102783           -19.4%      82894 ±  2%  perf-stat.i.context-switches
     54.23            -5.3%      51.38        perf-stat.i.cpu-migrations
    102570           -19.4%      82720 ±  2%  perf-stat.ps.context-switches
     54.15            -5.2%      51.31        perf-stat.ps.cpu-migrations
      1.39 ±  5%     -13.0%       1.21 ±  7%  sched_debug.cfs_rq:/.h_nr_running.max
   1095631 ±166%     -90.9%     100111 ± 17%  sched_debug.cfs_rq:/.load.avg
      1990 ±  9%     -22.9%       1535 ± 17%  sched_debug.cfs_rq:/.load_avg.max
    306.85 ± 10%     -17.4%     253.49 ± 15%  sched_debug.cfs_rq:/.load_avg.stddev
      6.20 ±  9%     -21.5%       4.86 ± 25%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      2873 ± 10%     -29.5%       2026 ± 10%  sched_debug.cpu.clock_task.stddev
   3837026 ±  7%     -23.4%    2940395 ±  8%  sched_debug.cpu.nr_switches.max
     43.97 ± 42%     -13.1       30.88 ± 10%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     50.19 ± 33%     -12.7       37.50 ± 14%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     49.35 ± 33%     -12.4       36.96 ± 14%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     49.34 ± 33%     -12.4       36.96 ± 14%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.41 ± 60%      -0.6        0.78 ± 15%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
      0.00            +0.8        0.75 ±  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.dd_insert_requests.blk_mq_sched_insert_request.blk_mq_submit_bio
      0.00            +0.8        0.80 ± 10%  perf-profile.calltrace.cycles-pp._raw_spin_lock.dd_insert_requests.blk_mq_sched_insert_request.blk_mq_submit_bio.submit_bio_noacct_nocheck
      0.08 ±223%      +0.8        0.93 ± 13%  perf-profile.calltrace.cycles-pp.dd_insert_requests.blk_mq_sched_insert_request.blk_mq_submit_bio.submit_bio_noacct_nocheck.submit_bh_wbc
      0.08 ±223%      +0.9        0.94 ± 13%  perf-profile.calltrace.cycles-pp.blk_mq_sched_insert_request.blk_mq_submit_bio.submit_bio_noacct_nocheck.submit_bh_wbc.__sync_dirty_buffer
     44.82 ± 42%     -13.4       31.39 ± 10%  perf-profile.children.cycles-pp.cpuidle_idle_call
     50.17 ± 33%     -12.7       37.48 ± 14%  perf-profile.children.cycles-pp.do_idle
     50.19 ± 33%     -12.7       37.50 ± 14%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     50.19 ± 33%     -12.7       37.50 ± 14%  perf-profile.children.cycles-pp.cpu_startup_entry
     49.35 ± 33%     -12.4       36.96 ± 14%  perf-profile.children.cycles-pp.start_secondary
     24.38 ± 33%      -6.7       17.72 ± 14%  perf-profile.children.cycles-pp.mwait_idle_with_hints
      1.78 ± 59%      -0.8        1.01 ± 11%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.45 ± 59%      -0.6        0.81 ± 15%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.70 ± 77%      -0.4        0.32 ± 31%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.15 ± 13%      -0.1        0.09 ± 46%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.38 ± 45%      +0.6        0.94 ± 13%  perf-profile.children.cycles-pp.blk_mq_sched_insert_request
      0.66 ± 45%      +0.8        1.47 ± 16%  perf-profile.children.cycles-pp.dd_insert_requests
      0.70 ± 77%      -0.4        0.32 ± 30%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.26 ± 63%      -0.1        0.14 ± 10%  perf-profile.self.cycles-pp.tick_nohz_next_event


***************************************************************************************************
lkp-icl-2sp2: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/25%/debian-11.1-x86_64-20220510.cgz/300s/lkp-icl-2sp2/UDP_RR/netperf/0xd000331

commit: 
  28156108fe ("sched: Fix the check of nr_running at queue wakelist")
  f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   2897681            +1.3%    2934351        netperf.Throughput_total_tps
     90552            +1.3%      91698        netperf.Throughput_tps
     10378            +4.8%      10881        netperf.time.minor_page_faults
      1511           -39.2%     918.83        netperf.time.percent_of_cpu_this_job_got
      4215           -39.9%       2532        netperf.time.system_time
    337.04           -30.3%     235.04        netperf.time.user_time
 8.693e+08            +1.3%  8.803e+08        netperf.time.voluntary_context_switches
 8.693e+08            +1.3%  8.803e+08        netperf.workload
     20979 ±  4%      -9.0%      19095 ±  5%  proc-vmstat.pgactivate
     73.20            +3.1       76.26        turbostat.C1%
      0.78            +2.9        3.64        mpstat.cpu.all.irq%
      6.06            -1.6        4.47        mpstat.cpu.all.soft%
     16.74            -2.0       14.74        mpstat.cpu.all.sys%
     38.00           -28.1%      27.33        vmstat.procs.r
  11439523            +1.3%   11584593        vmstat.system.cs
   5951961            +1.2%    6025940        vmstat.system.in
    119282 ±  8%     -75.4%      29291 ± 20%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   1541193 ± 12%     -63.0%     570028 ±  9%  sched_debug.cfs_rq:/.MIN_vruntime.max
    354476 ±  7%     -68.6%     111435 ± 14%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.32 ±  3%     -21.1%       0.26 ±  5%  sched_debug.cfs_rq:/.h_nr_running.avg
      9525 ±  3%     +20.0%      11432 ±  4%  sched_debug.cfs_rq:/.load.avg
     33675 ± 10%    +114.1%      72104 ± 87%  sched_debug.cfs_rq:/.load.max
     12820 ±  2%     +57.4%      20178 ± 26%  sched_debug.cfs_rq:/.load.stddev
    119282 ±  8%     -75.4%      29291 ± 20%  sched_debug.cfs_rq:/.max_vruntime.avg
   1541193 ± 12%     -63.0%     570028 ±  9%  sched_debug.cfs_rq:/.max_vruntime.max
    354476 ±  7%     -68.6%     111435 ± 14%  sched_debug.cfs_rq:/.max_vruntime.stddev
   1104703 ± 10%     -59.1%     451566 ± 10%  sched_debug.cfs_rq:/.min_vruntime.avg
   1722807 ± 12%     -54.9%     776144 ± 20%  sched_debug.cfs_rq:/.min_vruntime.max
    459283 ± 14%     -50.0%     229644 ± 22%  sched_debug.cfs_rq:/.min_vruntime.min
    288286 ± 12%     -67.7%      93188 ±  9%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.32 ±  2%     -21.1%       0.26 ±  5%  sched_debug.cfs_rq:/.nr_running.avg
    296.34           -30.8%     204.95 ±  2%  sched_debug.cfs_rq:/.runnable_avg.avg
    263.94           -27.6%     191.18        sched_debug.cfs_rq:/.runnable_avg.stddev
    810487 ± 23%     -51.7%     391282 ± 33%  sched_debug.cfs_rq:/.spread0.max
   -453070           -65.7%    -155225        sched_debug.cfs_rq:/.spread0.min
    288295 ± 12%     -67.7%      93193 ±  9%  sched_debug.cfs_rq:/.spread0.stddev
    295.88           -30.8%     204.74 ±  2%  sched_debug.cfs_rq:/.util_avg.avg
    263.59           -27.5%     191.03        sched_debug.cfs_rq:/.util_avg.stddev
    156.07 ±  3%     -50.1%      77.80 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    735.13 ±  5%     -11.2%     652.93 ±  8%  sched_debug.cfs_rq:/.util_est_enqueued.max
    224.80           -36.1%     143.75 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      1094 ±  3%     -13.0%     951.53 ±  7%  sched_debug.cpu.curr->pid.avg
      0.28 ±  2%     -24.9%       0.21 ±  6%  sched_debug.cpu.nr_running.avg
      9.67           -26.9%       7.07        perf-stat.i.MPKI
 1.954e+10            +3.9%  2.031e+10        perf-stat.i.branch-instructions
      1.33            -0.1        1.27        perf-stat.i.branch-miss-rate%
      1.25 ±  2%      +0.3        1.57        perf-stat.i.cache-miss-rate%
 9.408e+08           -24.0%  7.152e+08        perf-stat.i.cache-references
  11524667            +1.3%   11671113        perf-stat.i.context-switches
      1.29            -4.6%       1.23        perf-stat.i.cpi
    366.03            +7.3%     392.62        perf-stat.i.cpu-migrations
 2.865e+10            +4.1%  2.984e+10        perf-stat.i.dTLB-loads
 1.744e+10            +3.7%  1.809e+10        perf-stat.i.dTLB-stores
 9.917e+10            +3.9%   1.03e+11        perf-stat.i.instructions
      0.79            +4.7%       0.82        perf-stat.i.ipc
    520.11            +3.6%     538.63        perf-stat.i.metric.M/sec
   2630922            -4.1%    2523585        perf-stat.i.node-store-misses
      9.49           -26.8%       6.94        perf-stat.overall.MPKI
      1.29            -0.1        1.23        perf-stat.overall.branch-miss-rate%
      1.18 ±  3%      +0.3        1.50        perf-stat.overall.cache-miss-rate%
      1.27            -4.5%       1.21        perf-stat.overall.cpi
      0.79            +4.7%       0.82        perf-stat.overall.ipc
     34314            +2.6%      35206        perf-stat.overall.path-length
 1.948e+10            +3.9%  2.024e+10        perf-stat.ps.branch-instructions
 9.376e+08           -24.0%  7.128e+08        perf-stat.ps.cache-references
  11486058            +1.3%   11632058        perf-stat.ps.context-switches
    365.07            +7.3%     391.61        perf-stat.ps.cpu-migrations
 2.856e+10            +4.1%  2.974e+10        perf-stat.ps.dTLB-loads
 1.738e+10            +3.7%  1.803e+10        perf-stat.ps.dTLB-stores
 9.884e+10            +3.9%  1.027e+11        perf-stat.ps.instructions
   2622155            -4.1%    2515203        perf-stat.ps.node-store-misses
 2.983e+13            +3.9%  3.099e+13        perf-stat.total.instructions
     48.27           -48.3        0.00        perf-profile.calltrace.cycles-pp.asm_sysvec_reschedule_ipi.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
     18.85            -4.4       14.41        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg
     19.51            -4.4       15.08        perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     19.97            -4.4       15.55        perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     15.81            -4.4       11.39        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_send_skb.udp_send_skb
     16.00            -4.4       11.60        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg
     30.37            -4.4       25.98        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     31.58            -4.4       27.21        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
     15.53            -4.4       11.17        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_send_skb
     29.41            -4.3       25.08        perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     13.59            -4.0        9.59 ±  2%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
     14.18            -4.0       10.18 ±  2%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
     13.68            -4.0        9.70 ±  2%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
     10.94            -3.7        7.20 ±  2%  perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
     10.86            -3.7        7.13 ±  2%  perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      9.47            -3.7        5.82 ±  2%  perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv
      9.76            -3.6        6.12 ±  2%  perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu
     11.76            -3.6        8.12        perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      9.20            -3.6        5.56 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb
     11.88            -3.6        8.25 ±  2%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
      8.12            -3.6        4.49 ±  2%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
     11.94            -3.6        8.31 ±  2%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      8.94            -3.6        5.32 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb
      8.28            -3.6        4.68 ±  2%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb
     12.54            -3.4        9.13 ±  2%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
     26.78 ±  3%      -3.1       23.71 ±  3%  perf-profile.calltrace.cycles-pp.send_udp_rr.main.__libc_start_main
     26.50 ±  3%      -3.1       23.44 ±  3%  perf-profile.calltrace.cycles-pp.send_omni_inner.send_udp_rr.main.__libc_start_main
     15.83 ±  3%      -2.6       13.24 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner.send_udp_rr.main
     15.98 ±  3%      -2.5       13.48 ±  3%  perf-profile.calltrace.cycles-pp.sendto.send_omni_inner.send_udp_rr.main.__libc_start_main
     15.88            -2.4       13.50        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner
     16.10            -2.4       13.73        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner.send_udp_rr
      5.76            -2.3        3.44        perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      6.01            -2.3        3.69        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     15.89            -2.0       13.90        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.recv_omni
     16.11            -2.0       14.12        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.recv_omni.process_requests
     16.27            -2.0       14.29        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto.recv_omni.process_requests.spawn_child
     16.51            -2.0       14.54        perf-profile.calltrace.cycles-pp.sendto.recv_omni.process_requests.spawn_child.accept_connection
      2.97            -1.8        1.19        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
      2.16            -1.3        0.84        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle
      6.30            -1.3        4.99 ±  2%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp
      6.46            -1.2        5.24 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
      6.54            -1.1        5.47 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg
      8.34            -0.9        7.48 ±  2%  perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom
     10.27            -0.8        9.44        perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
     14.52            -0.6       13.91        perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     14.60            -0.6       14.00        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.43            -0.6       15.88        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom
      8.58 ±  2%      -0.4        8.18 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom.send_omni_inner.send_udp_rr
      2.41            -0.2        2.19        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets
      8.36            -0.2        8.16        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni
      2.08            -0.2        1.89        perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      8.65            -0.2        8.46        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni.process_requests
      8.79            -0.2        8.61        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni.process_requests.spawn_child
      9.72            -0.1        9.58        perf-profile.calltrace.cycles-pp.recvfrom.recv_omni.process_requests.spawn_child.accept_connection
      2.13            +0.0        2.18        perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.62            +0.3        0.96        perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.00            +0.5        0.51        perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule_idle.do_idle
      0.00            +0.5        0.52 ±  2%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending
      0.00            +0.8        0.77        perf-profile.calltrace.cycles-pp.resched_curr.check_preempt_curr.ttwu_do_wakeup.sched_ttwu_pending.__sysvec_call_function_single
      0.00            +0.8        0.82        perf-profile.calltrace.cycles-pp.update_rq_clock.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
      0.00            +0.9        0.86 ±  2%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry
      0.00            +0.9        0.88        perf-profile.calltrace.cycles-pp.irqentry_enter.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter
      0.00            +0.9        0.88        perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single
      0.00            +0.9        0.94        perf-profile.calltrace.cycles-pp.llist_reverse_order.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
      0.00            +0.9        0.94        perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
      0.00            +1.2        1.20 ±  2%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter
      0.00            +1.6        1.61        perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single
      0.00            +1.7        1.71 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +1.8        1.81        perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry
      0.00            +2.0        1.97        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single
      0.00            +2.1        2.06        perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
      0.00            +4.3        4.35        perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry
     42.79            +4.4       47.19        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     43.00            +4.4       47.41        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     43.06            +4.4       47.46        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     43.31            +4.4       47.73        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      0.00            +6.8        6.81        perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter
      6.60            +7.3       13.93        perf-profile.calltrace.cycles-pp.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     31.15            +7.5       38.70        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     29.32            +7.6       36.88        perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     30.97            +7.6       38.54        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     34.13            +7.6       41.74        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +9.7        9.67        perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state
      0.00           +56.3       56.35        perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
     25.79           -25.8        0.00        perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
     49.81            -4.6       45.22        perf-profile.children.cycles-pp.do_syscall_64
     50.35            -4.6       45.78        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     15.98            -4.5       11.51        perf-profile.children.cycles-pp.__softirqentry_text_start
     16.04            -4.4       11.59        perf-profile.children.cycles-pp.do_softirq
     16.31            -4.4       11.90        perf-profile.children.cycles-pp.__local_bh_enable_ip
     19.02            -4.4       14.66        perf-profile.children.cycles-pp.ip_finish_output2
     19.69            -4.4       15.34        perf-profile.children.cycles-pp.ip_send_skb
     20.14            -4.3       15.81        perf-profile.children.cycles-pp.udp_send_skb
     29.63            -4.2       25.43        perf-profile.children.cycles-pp.udp_sendmsg
     30.46            -4.2       26.29        perf-profile.children.cycles-pp.sock_sendmsg
     31.69            -4.1       27.54        perf-profile.children.cycles-pp.__sys_sendto
     31.90            -4.1       27.75        perf-profile.children.cycles-pp.__x64_sys_sendto
     33.74            -4.1       29.64        perf-profile.children.cycles-pp.sendto
     14.01            -4.1        9.95        perf-profile.children.cycles-pp.__napi_poll
     14.48            -4.0       10.45        perf-profile.children.cycles-pp.net_rx_action
     13.86            -4.0        9.83        perf-profile.children.cycles-pp.process_backlog
      9.73            -3.7        6.03        perf-profile.children.cycles-pp.sock_def_readable
      8.39            -3.7        4.69        perf-profile.children.cycles-pp.try_to_wake_up
     10.04            -3.7        6.35        perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
     11.14            -3.7        7.46        perf-profile.children.cycles-pp.udp_unicast_rcv_skb
      9.47            -3.7        5.78        perf-profile.children.cycles-pp.__wake_up_common_lock
     11.09            -3.7        7.41        perf-profile.children.cycles-pp.udp_queue_rcv_one_skb
      9.19            -3.7        5.51        perf-profile.children.cycles-pp.__wake_up_common
     12.00            -3.7        8.33        perf-profile.children.cycles-pp.__udp4_lib_rcv
      8.52            -3.7        4.86        perf-profile.children.cycles-pp.autoremove_wake_function
     12.42            -3.7        8.77        perf-profile.children.cycles-pp.__schedule
     12.16            -3.6        8.52        perf-profile.children.cycles-pp.ip_local_deliver_finish
     12.11            -3.6        8.46        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     12.78            -3.4        9.36        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     27.78            -2.6       25.14        perf-profile.children.cycles-pp.send_udp_rr
     27.69            -2.4       25.32        perf-profile.children.cycles-pp.send_omni_inner
      6.07            -2.3        3.74        perf-profile.children.cycles-pp.schedule_idle
     27.42            -2.1       25.31        perf-profile.children.cycles-pp.accept_connection
     27.42            -2.1       25.31        perf-profile.children.cycles-pp.spawn_child
     27.42            -2.1       25.31        perf-profile.children.cycles-pp.process_requests
     27.40            -2.1       25.30        perf-profile.children.cycles-pp.recv_omni
      3.50            -1.6        1.94 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      2.25            -1.3        0.93        perf-profile.children.cycles-pp.set_next_entity
      6.65            -1.2        5.43        perf-profile.children.cycles-pp.schedule
      2.41            -1.1        1.29        perf-profile.children.cycles-pp.update_rq_clock
      1.08            -1.0        0.12 ±  4%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      7.25            -0.9        6.37        perf-profile.children.cycles-pp.schedule_timeout
      8.59            -0.8        7.75        perf-profile.children.cycles-pp.__skb_wait_for_more_packets
     10.53            -0.8        9.74        perf-profile.children.cycles-pp.__skb_recv_udp
     14.82            -0.7       14.16        perf-profile.children.cycles-pp.udp_recvmsg
     14.88            -0.7       14.23        perf-profile.children.cycles-pp.inet_recvmsg
      2.66            -0.6        2.10        perf-profile.children.cycles-pp.ttwu_do_activate
      2.55            -0.5        2.03        perf-profile.children.cycles-pp.enqueue_task_fair
     16.68            -0.5       16.16        perf-profile.children.cycles-pp.__sys_recvfrom
     16.84            -0.5       16.33        perf-profile.children.cycles-pp.__x64_sys_recvfrom
      0.74 ±  2%      -0.5        0.28 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.58            -0.4        0.14 ±  7%  perf-profile.children.cycles-pp.pick_next_entity
      0.79 ±  3%      -0.4        0.36        perf-profile.children.cycles-pp.finish_task_switch
      0.64 ±  2%      -0.4        0.22 ±  4%  perf-profile.children.cycles-pp.switch_mm_irqs_off
     20.21            -0.4       19.80        perf-profile.children.cycles-pp.recvfrom
      1.03            -0.4        0.63        perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      2.02            -0.3        1.71        perf-profile.children.cycles-pp.update_load_avg
      1.22            -0.3        0.96        perf-profile.children.cycles-pp.ttwu_do_wakeup
      1.14 ±  2%      -0.2        0.90        perf-profile.children.cycles-pp.check_preempt_curr
      2.50            -0.2        2.28        perf-profile.children.cycles-pp.dequeue_task_fair
      1.94            -0.2        1.72        perf-profile.children.cycles-pp.enqueue_entity
      0.71            -0.2        0.52 ±  4%  perf-profile.children.cycles-pp.nr_iowait_cpu
      2.21            -0.2        2.03        perf-profile.children.cycles-pp.dequeue_entity
      1.02            -0.1        0.91        perf-profile.children.cycles-pp.irqentry_enter
      0.94            -0.0        0.90 ±  2%  perf-profile.children.cycles-pp.update_curr
      0.12 ±  6%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.57 ±  2%      -0.0        0.53        perf-profile.children.cycles-pp.__update_load_avg_se
      1.39            -0.0        1.35        perf-profile.children.cycles-pp.native_irq_return_iret
      0.19 ±  3%      -0.0        0.16 ±  9%  perf-profile.children.cycles-pp.fib_lookup_good_nhc
      0.21            -0.0        0.18 ±  4%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.24 ±  2%      -0.0        0.22 ±  5%  perf-profile.children.cycles-pp.rcu_nmi_enter
      0.14 ±  4%      -0.0        0.12        perf-profile.children.cycles-pp.error_entry
      0.22 ±  2%      -0.0        0.20 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.06 ±  8%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.47            +0.0        0.48        perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.37            +0.0        0.39 ±  2%  perf-profile.children.cycles-pp.netif_rx_internal
      0.76            +0.0        0.78        perf-profile.children.cycles-pp.ip_generic_getfrag
      0.40            +0.0        0.42 ±  2%  perf-profile.children.cycles-pp.__netif_rx
      0.34            +0.0        0.36 ±  2%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.09 ±  4%      +0.0        0.11 ±  3%  perf-profile.children.cycles-pp.place_entity
      0.16 ±  5%      +0.0        0.18 ±  5%  perf-profile.children.cycles-pp.__get_user_4
      0.08 ±  4%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.08 ±  6%      +0.0        0.10 ±  3%  perf-profile.children.cycles-pp.clear_buddies
      0.06 ±  6%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.menu_reflect
      0.53            +0.0        0.56        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.47            +0.0        0.50        perf-profile.children.cycles-pp.aa_sk_perm
      0.44 ±  2%      +0.0        0.47        perf-profile.children.cycles-pp.__might_fault
      0.94            +0.0        0.97        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.74            +0.0        0.78        perf-profile.children.cycles-pp.check_heap_object
      0.06 ±  7%      +0.0        0.10 ±  3%  perf-profile.children.cycles-pp.rb_erase
      0.09 ±  5%      +0.0        0.13 ±  5%  perf-profile.children.cycles-pp.__list_add_valid
      1.68            +0.0        1.72        perf-profile.children.cycles-pp.loopback_xmit
      0.20 ±  3%      +0.0        0.24 ±  2%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.74            +0.0        0.79 ±  3%  perf-profile.children.cycles-pp.select_idle_sibling
      1.10            +0.0        1.15        perf-profile.children.cycles-pp.__consume_stateless_skb
      0.74            +0.0        0.79        perf-profile.children.cycles-pp.___perf_sw_event
      2.19            +0.0        2.24        perf-profile.children.cycles-pp.menu_select
      1.09            +0.0        1.14        perf-profile.children.cycles-pp.skb_release_data
      0.38 ±  2%      +0.0        0.43 ± 11%  perf-profile.children.cycles-pp._copy_to_user
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.idle_cpu
      1.30            +0.1        1.35        perf-profile.children.cycles-pp.__check_object_size
      0.21 ±  3%      +0.1        0.26 ±  4%  perf-profile.children.cycles-pp.update_cfs_group
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      1.01            +0.1        1.06 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.tick_nohz_idle_got_tick
      0.10 ±  4%      +0.1        0.16 ±  2%  perf-profile.children.cycles-pp.sched_clock
      0.65            +0.1        0.71 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.09 ±  7%      +0.1        0.15 ± 13%  perf-profile.children.cycles-pp.update_blocked_averages
      4.40            +0.1        4.47        perf-profile.children.cycles-pp.__ip_append_data
      0.16 ±  4%      +0.1        0.24 ±  2%  perf-profile.children.cycles-pp.x2apic_send_IPI
      0.08 ±  7%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.99            +0.1        1.07 ±  5%  perf-profile.children.cycles-pp.move_addr_to_user
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.04 ± 45%      +0.1        0.13 ± 14%  perf-profile.children.cycles-pp.cpuacct_charge
      0.17 ±  2%      +0.1        0.26 ±  2%  perf-profile.children.cycles-pp.ip_rcv_core
      0.10 ±  5%      +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.10 ±  6%      +0.1        0.21 ± 10%  perf-profile.children.cycles-pp.find_busiest_group
      0.00            +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.send_call_function_single_ipi
      0.71            +0.1        0.83        perf-profile.children.cycles-pp.__x2apic_send_IPI_dest
      0.12 ±  7%      +0.1        0.25 ±  9%  perf-profile.children.cycles-pp.load_balance
      0.82            +0.2        0.98        perf-profile.children.cycles-pp._raw_spin_lock
      0.56 ±  3%      +0.2        0.71 ±  2%  perf-profile.children.cycles-pp.available_idle_cpu
      7.00            +0.2        7.15        perf-profile.children.cycles-pp.ip_make_skb
      0.33            +0.2        0.50 ±  2%  perf-profile.children.cycles-pp.ip_rcv
      1.10 ±  2%      +0.2        1.32        perf-profile.children.cycles-pp.prepare_task_switch
      0.00            +0.2        0.23        perf-profile.children.cycles-pp.llist_add_batch
      0.38 ±  5%      +0.2        0.62 ±  8%  perf-profile.children.cycles-pp.newidle_balance
      0.00            +0.3        0.26        perf-profile.children.cycles-pp.__smp_call_single_queue
      0.18 ±  4%      +0.3        0.51        perf-profile.children.cycles-pp.__irq_exit_rcu
      0.97            +0.5        1.44        perf-profile.children.cycles-pp.native_sched_clock
      1.16            +0.6        1.73        perf-profile.children.cycles-pp.sched_clock_cpu
      0.13 ±  8%      +0.6        0.78        perf-profile.children.cycles-pp.resched_curr
      0.06 ± 13%      +0.9        0.95 ±  2%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +1.0        0.95        perf-profile.children.cycles-pp.llist_reverse_order
      0.07 ± 12%      +1.2        1.27 ±  2%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.07 ±  5%      +1.7        1.79        perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.00            +1.8        1.84        perf-profile.children.cycles-pp.__flush_smp_call_function_queue
     43.06            +4.4       47.46        perf-profile.children.cycles-pp.start_secondary
     43.19            +4.4       47.61        perf-profile.children.cycles-pp.do_idle
      0.00            +4.4        4.42        perf-profile.children.cycles-pp.sched_ttwu_pending
     43.31            +4.4       47.73        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     43.31            +4.4       47.73        perf-profile.children.cycles-pp.cpu_startup_entry
      0.00            +6.9        6.87        perf-profile.children.cycles-pp.__sysvec_call_function_single
     31.36            +7.6       38.95        perf-profile.children.cycles-pp.cpuidle_enter
     29.52            +7.6       37.11        perf-profile.children.cycles-pp.acpi_idle_enter
     31.21            +7.6       38.82        perf-profile.children.cycles-pp.cpuidle_enter_state
     34.43            +7.7       42.08        perf-profile.children.cycles-pp.cpuidle_idle_call
     29.00            +7.8       36.80        perf-profile.children.cycles-pp.acpi_idle_do_entry
      0.00            +9.8        9.77        perf-profile.children.cycles-pp.sysvec_call_function_single
      0.00           +33.6       33.56        perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      1.56            -1.3        0.23 ±  3%  perf-profile.self.cycles-pp.set_next_entity
      1.92            -1.2        0.76        perf-profile.self.cycles-pp.update_rq_clock
      1.65            -0.6        1.05        perf-profile.self.cycles-pp.__schedule
      0.73 ±  2%      -0.5        0.28 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.57            -0.4        0.12 ±  7%  perf-profile.self.cycles-pp.pick_next_entity
      0.64 ±  3%      -0.4        0.21 ±  3%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.72 ±  3%      -0.4        0.30        perf-profile.self.cycles-pp.finish_task_switch
      1.02            -0.4        0.62        perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.61            -0.4        0.23 ±  3%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.63 ±  2%      -0.3        0.33 ±  4%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.88            -0.3        0.60        perf-profile.self.cycles-pp.update_load_avg
      0.52 ±  2%      -0.2        0.32 ±  3%  perf-profile.self.cycles-pp.acpi_idle_enter
      0.70            -0.2        0.50 ±  4%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.86            -0.1        0.74 ±  2%  perf-profile.self.cycles-pp.enqueue_entity
      0.46 ±  2%      -0.1        0.35 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.21 ±  3%      -0.1        0.12 ±  5%  perf-profile.self.cycles-pp.__napi_poll
      0.29 ±  3%      -0.1        0.20 ±  3%  perf-profile.self.cycles-pp.process_backlog
      0.77            -0.1        0.68        perf-profile.self.cycles-pp.irqentry_enter
      0.43 ±  3%      -0.1        0.35 ±  3%  perf-profile.self.cycles-pp.select_task_rq
      0.97            -0.1        0.91        perf-profile.self.cycles-pp.udp_sendmsg
      0.36 ±  4%      -0.1        0.30 ±  2%  perf-profile.self.cycles-pp.dequeue_entity
      0.79 ±  2%      -0.0        0.74 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.51 ±  2%      -0.0        0.47        perf-profile.self.cycles-pp.__update_load_avg_se
      0.25 ±  2%      -0.0        0.21 ±  6%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.12 ±  5%      -0.0        0.08 ± 14%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.14 ±  3%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.tick_nohz_idle_exit
      0.10 ±  4%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.ttwu_do_activate
      1.38            -0.0        1.35        perf-profile.self.cycles-pp.native_irq_return_iret
      0.18 ±  3%      -0.0        0.15 ±  9%  perf-profile.self.cycles-pp.fib_lookup_good_nhc
      0.13 ±  3%      -0.0        0.11        perf-profile.self.cycles-pp.error_entry
      0.19            -0.0        0.16 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.08 ±  5%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.ttwu_do_wakeup
      0.22 ±  3%      -0.0        0.20 ±  3%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.62            -0.0        0.60        perf-profile.self.cycles-pp.skb_set_owner_w
      0.07 ±  5%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.place_entity
      0.45            +0.0        0.46        perf-profile.self.cycles-pp.__might_resched
      0.56            +0.0        0.58        perf-profile.self.cycles-pp.check_heap_object
      0.08            +0.0        0.10 ±  6%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.12 ±  4%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.53            +0.0        0.55        perf-profile.self.cycles-pp.udp_recvmsg
      0.16 ±  2%      +0.0        0.18 ±  4%  perf-profile.self.cycles-pp.skb_release_data
      0.16 ±  3%      +0.0        0.18 ±  3%  perf-profile.self.cycles-pp.__get_user_4
      0.20 ±  3%      +0.0        0.23 ±  5%  perf-profile.self.cycles-pp.do_softirq
      0.91            +0.0        0.93        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.55            +0.0        0.57        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.44 ±  2%      +0.0        0.46 ±  2%  perf-profile.self.cycles-pp.net_rx_action
      0.04 ± 44%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.clear_buddies
      0.08 ±  6%      +0.0        0.11 ±  3%  perf-profile.self.cycles-pp.__list_add_valid
      0.26 ±  2%      +0.0        0.29        perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.62 ±  2%      +0.0        0.65        perf-profile.self.cycles-pp.__skb_recv_udp
      0.20 ±  4%      +0.0        0.24 ±  4%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.20 ±  4%      +0.0        0.24        perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.66 ±  2%      +0.0        0.70        perf-profile.self.cycles-pp.___perf_sw_event
      0.03 ± 70%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__irq_exit_rcu
      0.18 ±  4%      +0.1        0.23 ±  6%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.sched_clock
      0.06 ±  6%      +0.1        0.12 ±  9%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.__sysvec_call_function_single
      0.63 ±  2%      +0.1        0.70 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      0.08 ±  4%      +0.1        0.14 ±  3%  perf-profile.self.cycles-pp.ip_rcv
      0.15 ±  5%      +0.1        0.22 ±  2%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.16 ±  3%      +0.1        0.23 ±  3%  perf-profile.self.cycles-pp.x2apic_send_IPI
      0.00            +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.tick_irq_enter
      0.04 ± 45%      +0.1        0.13 ± 15%  perf-profile.self.cycles-pp.cpuacct_charge
      0.17 ±  2%      +0.1        0.26 ±  2%  perf-profile.self.cycles-pp.ip_rcv_core
      0.09 ±  6%      +0.1        0.18 ±  2%  perf-profile.self.cycles-pp.schedule
      0.71            +0.1        0.82        perf-profile.self.cycles-pp.__x2apic_send_IPI_dest
      0.00            +0.1        0.12 ±  5%  perf-profile.self.cycles-pp.send_call_function_single_ipi
      0.54 ±  3%      +0.2        0.70 ±  3%  perf-profile.self.cycles-pp.available_idle_cpu
      0.77            +0.2        0.94 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      0.35 ±  7%      +0.2        0.52        perf-profile.self.cycles-pp.prepare_task_switch
      0.07 ±  5%      +0.2        0.25        perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.00            +0.2        0.21 ±  3%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.00            +0.2        0.22        perf-profile.self.cycles-pp.llist_add_batch
      0.00            +0.3        0.29        perf-profile.self.cycles-pp.sysvec_call_function_single
      0.00            +0.3        0.32        perf-profile.self.cycles-pp.irq_enter_rcu
      0.60 ±  4%      +0.3        0.94        perf-profile.self.cycles-pp.schedule_timeout
      0.00            +0.4        0.38 ±  3%  perf-profile.self.cycles-pp.asm_sysvec_call_function_single
     23.90            +0.4       24.31        perf-profile.self.cycles-pp.acpi_idle_do_entry
      0.93            +0.4        1.38        perf-profile.self.cycles-pp.native_sched_clock
      0.13 ±  8%      +0.6        0.78        perf-profile.self.cycles-pp.resched_curr
      0.00            +0.9        0.88        perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.00            +1.0        0.95        perf-profile.self.cycles-pp.llist_reverse_order



***************************************************************************************************
lkp-csl-2sp9: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-11/performance/4BRD_12G/f2fs/x86_64-rhel-8.3/100/RAID1/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp9/sync_disk_rw/aim7/0x500320a

commit: 
  28156108fe ("sched: Fix the check of nr_running at queue wakelist")
  f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      1526 ± 10%    +246.4%       5285 ± 25%  aim7.jobs-per-min
    398.30 ± 11%     -69.4%     121.76 ± 26%  aim7.time.elapsed_time
    398.30 ± 11%     -69.4%     121.76 ± 26%  aim7.time.elapsed_time.max
  86549701            -7.5%   80055785        aim7.time.file_system_outputs
   4061406 ± 10%     -60.0%    1624315 ± 22%  aim7.time.involuntary_context_switches
     13445 ±  3%     -10.5%      12039 ±  4%  aim7.time.minor_page_faults
      1767 ±  9%     -32.9%       1186 ± 15%  aim7.time.system_time
 3.832e+08 ± 12%     -67.8%  1.232e+08 ± 30%  aim7.time.voluntary_context_switches
 3.208e+10 ± 12%     -72.0%  8.998e+09 ± 27%  cpuidle..time
 5.072e+08 ± 13%     -69.4%   1.55e+08 ± 31%  cpuidle..usage
     43.08           -24.9%      32.37 ± 21%  iostat.cpu.idle
      6.13 ±  3%    +119.1%      13.43 ± 11%  iostat.cpu.system
    430.38 ± 11%     -64.2%     153.88 ± 20%  uptime.boot
     17079 ± 11%     -67.9%       5479 ±  2%  uptime.idle
     42.81           -11.5       31.28 ± 22%  mpstat.cpu.all.idle%
      0.76 ±  2%      +0.2        0.93 ±  7%  mpstat.cpu.all.irq%
      0.05            +0.0        0.06 ±  7%  mpstat.cpu.all.soft%
      5.34 ±  3%      +7.3       12.64 ± 12%  mpstat.cpu.all.sys%
      0.09 ±  8%      +0.1        0.20 ± 10%  mpstat.cpu.all.usr%
     42.67           -25.4%      31.83 ± 22%  vmstat.cpu.id
    108692 ±  9%    +212.7%     339895 ± 24%  vmstat.io.bo
 1.162e+08 ±  2%     -16.3%   97235935 ±  3%  vmstat.memory.free
     88.33 ±  2%     -33.6%      58.67 ± 14%  vmstat.procs.b
      5.17 ±  7%    +100.0%      10.33 ± 14%  vmstat.procs.r
    185561           +14.2%     211999 ±  3%  vmstat.system.in
     10476 ±  8%     +61.0%      16866 ± 12%  numa-meminfo.node0.Dirty
     31580 ±  9%     +18.1%      37306 ±  8%  numa-meminfo.node0.Inactive(file)
  54824112 ±  7%     -19.4%   44208669 ±  8%  numa-meminfo.node0.MemFree
  10835242 ± 36%     +98.0%   21450685 ± 17%  numa-meminfo.node0.MemUsed
      9855 ± 12%     +67.2%      16479 ± 13%  numa-meminfo.node1.Dirty
     29343 ±  9%     +26.5%      37127 ±  6%  numa-meminfo.node1.Inactive(file)
  61393162 ±  4%     -13.5%   53116009 ± 10%  numa-meminfo.node1.MemFree
   4621337 ± 56%    +179.1%   12898489 ± 41%  numa-meminfo.node1.MemUsed
    180156 ±  2%     -34.3%     118377 ± 13%  meminfo.AnonHugePages
     20545 ±  3%     +58.5%      32563 ± 12%  meminfo.Dirty
     60890           +22.1%      74342 ±  6%  meminfo.Inactive(file)
    138527 ±  3%     +29.7%     179674 ±  4%  meminfo.KReclaimable
 1.157e+08 ±  2%     -16.3%   96822390 ±  3%  meminfo.MemAvailable
 1.162e+08 ±  2%     -16.3%   97308937 ±  3%  meminfo.MemFree
  15456872 ± 15%    +122.3%   34364917 ± 10%  meminfo.Memused
    138527 ±  3%     +29.7%     179674 ±  4%  meminfo.SReclaimable
    322884           +13.2%     365463 ±  2%  meminfo.Slab
  17500146 ± 16%     +99.0%   34825108 ± 11%  meminfo.max_used_kB
      2613 ±  8%     +61.2%       4213 ± 13%  numa-vmstat.node0.nr_dirty
  13706147 ±  7%     -19.4%   11052112 ±  8%  numa-vmstat.node0.nr_free_pages
      7894 ±  9%     +18.1%       9326 ±  8%  numa-vmstat.node0.nr_inactive_file
     97.83 ± 67%    -100.0%       0.00        numa-vmstat.node0.nr_mlock
      7894 ±  9%     +18.1%       9326 ±  8%  numa-vmstat.node0.nr_zone_inactive_file
    621.67 ± 18%     -63.2%     228.50 ± 72%  numa-vmstat.node0.nr_zone_write_pending
      2462 ± 12%     +67.5%       4126 ± 13%  numa-vmstat.node1.nr_dirty
  15348666 ±  4%     -13.5%   13279101 ± 10%  numa-vmstat.node1.nr_free_pages
      7336 ±  9%     +26.6%       9285 ±  6%  numa-vmstat.node1.nr_inactive_file
      7336 ±  9%     +26.6%       9285 ±  6%  numa-vmstat.node1.nr_zone_inactive_file
    568.00 ± 11%     -68.1%     181.17 ± 64%  numa-vmstat.node1.nr_zone_write_pending
    444.83           +52.0%     676.33 ±  4%  turbostat.Avg_MHz
     15.96            +8.4       24.38 ±  4%  turbostat.Busy%
 2.476e+08 ± 12%     -67.0%   81669432 ± 35%  turbostat.C1
     46.80            -6.9       39.93 ± 12%  turbostat.C1%
 1.495e+08 ± 12%     -74.3%   38375703 ± 18%  turbostat.C1E
     84.02           -11.2%      74.64 ±  2%  turbostat.CPU%c1
      0.02         +4791.7%       0.98 ±133%  turbostat.CPU%c6
  74404370 ± 11%     -64.8%   26214041 ± 22%  turbostat.IRQ
 1.096e+08 ± 18%     -68.6%   34360239 ± 35%  turbostat.POLL
     53.17            +3.4%      55.00        turbostat.PkgTmp
    163.95            +6.6%     174.70        turbostat.PkgWatt
     14.14            +6.7%      15.08        turbostat.RAMWatt
     63785            +3.3%      65914        proc-vmstat.nr_anon_pages
  10809810            -7.6%    9991919        proc-vmstat.nr_dirtied
      5118 ±  3%     +59.4%       8158 ± 12%  proc-vmstat.nr_dirty
   2887461 ±  2%     -16.3%    2415794 ±  3%  proc-vmstat.nr_dirty_background_threshold
   5781983 ±  2%     -16.3%    4837496 ±  3%  proc-vmstat.nr_dirty_threshold
  29054604 ±  2%     -16.3%   24327424 ±  3%  proc-vmstat.nr_free_pages
     15227           +22.0%      18583 ±  6%  proc-vmstat.nr_inactive_file
    114.67 ± 44%    -100.0%       0.00        proc-vmstat.nr_mlock
     34630 ±  3%     +29.7%      44918 ±  4%  proc-vmstat.nr_slab_reclaimable
  10774150            -7.6%    9955315        proc-vmstat.nr_written
     15227           +22.0%      18583 ±  6%  proc-vmstat.nr_zone_inactive_file
      1097 ± 12%     -62.0%     417.00 ± 80%  proc-vmstat.nr_zone_write_pending
   6341509 ±  8%     +89.3%   12003005 ±  6%  proc-vmstat.numa_hit
   6264928 ±  8%     +90.3%   11922778 ±  6%  proc-vmstat.numa_local
   6341353 ±  8%     +89.3%   12002253 ±  6%  proc-vmstat.pgalloc_normal
      3894 ± 22%     +97.2%       7678 ±  6%  proc-vmstat.pgdeactivate
   1283415 ±  6%     -54.3%     586146 ± 37%  proc-vmstat.pgfault
   3193580           -26.2%    2357922 ± 26%  proc-vmstat.pgfree
  43111608            -7.6%   39849599        proc-vmstat.pgpgout
     46760 ± 10%     -57.3%      19952 ± 15%  proc-vmstat.pgreuse
      7619 ± 23%    +100.4%      15269 ±  6%  proc-vmstat.pgrotated
      0.09 ± 15%     +81.1%       0.16 ± 16%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.28 ±  6%     +31.3%       0.37 ±  7%  sched_debug.cfs_rq:/.h_nr_running.stddev
      0.05 ±142%   +1084.6%       0.61 ± 46%  sched_debug.cfs_rq:/.load_avg.min
      8133 ± 17%     -31.6%       5564 ± 16%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.09 ± 14%     +80.4%       0.16 ± 16%  sched_debug.cfs_rq:/.nr_running.avg
      0.28 ±  6%     +31.0%       0.37 ±  7%  sched_debug.cfs_rq:/.nr_running.stddev
      8.27 ± 44%    +243.8%      28.42 ± 47%  sched_debug.cfs_rq:/.removed.load_avg.avg
    289.86 ± 31%    +125.7%     654.22 ± 43%  sched_debug.cfs_rq:/.removed.load_avg.max
     44.17 ± 30%    +178.7%     123.10 ± 35%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      3.51 ± 40%    +254.2%      12.43 ± 39%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    144.11 ± 30%    +121.1%     318.69 ± 41%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     19.97 ± 28%    +179.9%      55.88 ± 32%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      3.51 ± 40%    +254.2%      12.43 ± 39%  sched_debug.cfs_rq:/.removed.util_avg.avg
    144.11 ± 30%    +121.1%     318.69 ± 41%  sched_debug.cfs_rq:/.removed.util_avg.max
     19.97 ± 28%    +179.9%      55.88 ± 32%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    101.10 ±  4%    +109.3%     211.65 ± 11%  sched_debug.cfs_rq:/.runnable_avg.avg
      1066 ±  3%      +8.4%       1155 ±  7%  sched_debug.cfs_rq:/.runnable_avg.max
      0.12 ±170%   +6854.8%       8.56 ±140%  sched_debug.cfs_rq:/.runnable_avg.min
    155.43 ±  3%     +40.0%     217.67 ±  7%  sched_debug.cfs_rq:/.runnable_avg.stddev
      8133 ± 17%     -31.6%       5564 ± 16%  sched_debug.cfs_rq:/.spread0.stddev
    100.71 ±  4%    +109.5%     211.01 ± 11%  sched_debug.cfs_rq:/.util_avg.avg
      1065 ±  3%      +8.4%       1155 ±  7%  sched_debug.cfs_rq:/.util_avg.max
      0.17 ±154%   +4752.3%       8.47 ±141%  sched_debug.cfs_rq:/.util_avg.min
    155.41 ±  3%     +40.1%     217.75 ±  7%  sched_debug.cfs_rq:/.util_avg.stddev
     17.21 ±  4%     +71.8%      29.56 ±  8%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    104.99 ±  3%     +17.1%     122.98 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    203519 ±  6%     +86.0%     378456 ± 15%  sched_debug.cpu.avg_idle.avg
     23207 ± 10%     -34.1%      15298 ± 15%  sched_debug.cpu.avg_idle.min
    215927 ± 12%     -64.8%      75933 ± 19%  sched_debug.cpu.clock.avg
    215930 ± 12%     -64.8%      75937 ± 19%  sched_debug.cpu.clock.max
    215922 ± 12%     -64.8%      75928 ± 19%  sched_debug.cpu.clock.min
    214420 ± 12%     -64.8%      75461 ± 19%  sched_debug.cpu.clock_task.avg
    214557 ± 12%     -64.8%      75513 ± 19%  sched_debug.cpu.clock_task.max
    213787 ± 12%     -65.0%      74884 ± 19%  sched_debug.cpu.clock_task.min
    489.12 ± 10%     +77.2%     866.67 ± 22%  sched_debug.cpu.curr->pid.avg
      9698 ±  7%     -39.1%       5905 ±  7%  sched_debug.cpu.curr->pid.max
    538311 ±  3%     +87.0%    1006375 ± 51%  sched_debug.cpu.max_idle_balance_cost.max
      5365 ± 62%   +1667.9%      94854 ±105%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.09 ±  8%     +87.6%       0.17 ± 21%  sched_debug.cpu.nr_running.avg
      0.28 ±  3%     +30.1%       0.37 ±  8%  sched_debug.cpu.nr_running.stddev
   3886268 ± 16%     -74.8%     979579 ± 36%  sched_debug.cpu.nr_switches.avg
   4753123 ± 19%     -76.3%    1127185 ± 40%  sched_debug.cpu.nr_switches.max
   2510015 ± 10%     -69.0%     778502 ± 32%  sched_debug.cpu.nr_switches.min
    513442 ± 44%     -85.2%      75784 ± 69%  sched_debug.cpu.nr_switches.stddev
    215923 ± 12%     -64.8%      75929 ± 19%  sched_debug.cpu_clk
    215350 ± 12%     -65.0%      75355 ± 19%  sched_debug.ktime
    216859 ± 12%     -64.6%      76708 ± 19%  sched_debug.sched_clk
 3.027e+09           +37.7%  4.168e+09        perf-stat.i.branch-instructions
      1.19            +0.5        1.68 ± 20%  perf-stat.i.branch-miss-rate%
  35249374 ±  2%     +67.7%   59107652 ±  8%  perf-stat.i.branch-misses
     18.14 ±  4%      +5.0       23.10 ±  5%  perf-stat.i.cache-miss-rate%
  51539072 ±  2%     +75.3%   90324359 ±  6%  perf-stat.i.cache-misses
 2.857e+08 ±  4%     +33.4%  3.811e+08 ±  8%  perf-stat.i.cache-references
   1945960            +5.2%    2046993 ±  3%  perf-stat.i.context-switches
      2.88            +8.2%       3.12 ±  4%  perf-stat.i.cpi
 3.961e+10           +52.3%   6.03e+10 ±  4%  perf-stat.i.cpu-cycles
      6686 ± 11%    +327.7%      28595 ± 20%  perf-stat.i.cpu-migrations
      0.01 ± 23%      +0.0        0.06 ± 59%  perf-stat.i.dTLB-load-miss-rate%
    565205 ± 16%    +332.6%    2444922 ± 28%  perf-stat.i.dTLB-load-misses
 3.747e+09           +45.1%  5.438e+09 ±  2%  perf-stat.i.dTLB-loads
      0.00 ± 24%      +0.0        0.01 ± 68%  perf-stat.i.dTLB-store-miss-rate%
     51172 ± 14%    +216.7%     162069 ± 29%  perf-stat.i.dTLB-store-misses
 1.864e+09           +36.4%  2.544e+09        perf-stat.i.dTLB-stores
     33.72            +1.2       34.89 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   7385612           +21.5%    8974148        perf-stat.i.iTLB-load-misses
  14725202           +20.9%   17808328        perf-stat.i.iTLB-loads
 1.387e+10           +42.1%  1.972e+10        perf-stat.i.instructions
      1941           +24.1%       2408 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.60 ± 10%    +181.3%       1.68 ±  6%  perf-stat.i.major-faults
      0.45           +52.2%       0.69 ±  4%  perf-stat.i.metric.GHz
    558.11           +57.7%     880.04 ±  8%  perf-stat.i.metric.K/sec
    101.40           +40.4%     142.35        perf-stat.i.metric.M/sec
     94.97            -3.4       91.55        perf-stat.i.node-load-miss-rate%
  22231769           +56.6%   34809028 ±  5%  perf-stat.i.node-load-misses
   1074012 ±  7%    +143.7%    2617508 ± 13%  perf-stat.i.node-loads
     92.47            -4.1       88.35 ±  2%  perf-stat.i.node-store-miss-rate%
   8136157           +87.1%   15224518 ±  8%  perf-stat.i.node-store-misses
    631409 ± 13%    +160.4%    1644054 ± 20%  perf-stat.i.node-stores
      1.16            +0.3        1.42 ±  7%  perf-stat.overall.branch-miss-rate%
     18.07 ±  4%      +5.7       23.74 ±  3%  perf-stat.overall.cache-miss-rate%
      2.86 ±  2%      +7.1%       3.06 ±  2%  perf-stat.overall.cpi
    768.67           -13.0%     668.99 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.02 ± 15%      +0.0        0.04 ± 26%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 12%      +0.0        0.01 ± 28%  perf-stat.overall.dTLB-store-miss-rate%
      1878           +16.9%       2197        perf-stat.overall.instructions-per-iTLB-miss
      0.35 ±  2%      -6.6%       0.33 ±  2%  perf-stat.overall.ipc
     95.39            -2.4       93.04        perf-stat.overall.node-load-miss-rate%
     92.80            -2.5       90.35        perf-stat.overall.node-store-miss-rate%
 3.019e+09           +36.9%  4.132e+09        perf-stat.ps.branch-instructions
  35157825 ±  2%     +66.6%   58574778 ±  7%  perf-stat.ps.branch-misses
  51407192 ±  2%     +74.1%   89524967 ±  6%  perf-stat.ps.cache-misses
  2.85e+08 ±  4%     +32.5%  3.777e+08 ±  8%  perf-stat.ps.cache-references
   1940981            +4.5%    2029274 ±  3%  perf-stat.ps.context-switches
 3.951e+10           +51.3%  5.977e+10 ±  4%  perf-stat.ps.cpu-cycles
      6668 ± 11%    +324.9%      28333 ± 20%  perf-stat.ps.cpu-migrations
    563754 ± 16%    +329.7%    2422257 ± 28%  perf-stat.ps.dTLB-load-misses
 3.738e+09           +44.2%   5.39e+09 ±  2%  perf-stat.ps.dTLB-loads
     51038 ± 14%    +214.6%     160565 ± 28%  perf-stat.ps.dTLB-store-misses
  1.86e+09           +35.6%  2.521e+09        perf-stat.ps.dTLB-stores
   7366790           +20.8%    8895533        perf-stat.ps.iTLB-load-misses
  14687561           +20.2%   17652009        perf-stat.ps.iTLB-loads
 1.384e+10           +41.2%  1.954e+10        perf-stat.ps.instructions
      0.60 ± 10%    +179.5%       1.66 ±  6%  perf-stat.ps.major-faults
  22174931           +55.6%   34501460 ±  5%  perf-stat.ps.node-load-misses
   1071229 ±  7%    +142.1%    2593745 ± 13%  perf-stat.ps.node-loads
   8115368           +85.9%   15089754 ±  7%  perf-stat.ps.node-store-misses
    629775 ± 13%    +158.6%    1628862 ± 20%  perf-stat.ps.node-stores
  5.52e+12 ± 10%     -56.7%  2.392e+12 ± 24%  perf-stat.total.instructions
     73.04 ±  5%     -19.4       53.65 ±  7%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     72.28 ±  5%     -19.2       53.04 ±  7%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     72.26 ±  5%     -19.2       53.03 ±  7%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     72.23 ±  5%     -19.2       53.01 ±  7%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     67.43 ±  5%     -18.3       49.10 ±  7%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     66.16 ±  5%     -17.8       48.37 ±  7%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     65.59 ±  5%     -17.6       48.02 ±  7%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     15.97 ± 16%      -8.8        7.15 ± 34%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     25.00 ±  7%      -7.9       17.06 ± 23%  perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      8.52 ± 23%      -3.5        5.00 ± 13%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter
      3.83 ± 28%      -2.9        0.92 ± 38%  perf-profile.calltrace.cycles-pp.folio_wake_bit.folio_end_writeback.f2fs_write_end_io.raid_end_bio_io.raid1_end_write_request
      3.75 ± 28%      -2.9        0.86 ± 40%  perf-profile.calltrace.cycles-pp.__wake_up_common.folio_wake_bit.folio_end_writeback.f2fs_write_end_io.raid_end_bio_io
      3.68 ± 28%      -2.8        0.83 ± 40%  perf-profile.calltrace.cycles-pp.wake_page_function.__wake_up_common.folio_wake_bit.folio_end_writeback.f2fs_write_end_io
      3.88 ± 27%      -2.8        1.04 ± 32%  perf-profile.calltrace.cycles-pp.folio_end_writeback.f2fs_write_end_io.raid_end_bio_io.raid1_end_write_request.__submit_bio
      3.56 ± 28%      -2.8        0.80 ± 41%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_page_function.__wake_up_common.folio_wake_bit.folio_end_writeback
      3.97 ± 26%      -2.7        1.26 ± 24%  perf-profile.calltrace.cycles-pp.f2fs_write_end_io.raid_end_bio_io.raid1_end_write_request.__submit_bio.__submit_bio_noacct
      4.08 ± 25%      -2.7        1.37 ± 21%  perf-profile.calltrace.cycles-pp.raid1_end_write_request.__submit_bio.__submit_bio_noacct.flush_bio_list.flush_pending_writes
      5.24 ±  6%      -2.4        2.80 ± 22%  perf-profile.calltrace.cycles-pp.io_schedule.folio_wait_bit_common.folio_wait_writeback.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback
      5.19 ±  6%      -2.4        2.78 ± 22%  perf-profile.calltrace.cycles-pp.schedule.io_schedule.folio_wait_bit_common.folio_wait_writeback.f2fs_wait_on_page_writeback
      5.13 ±  6%      -2.4        2.76 ± 22%  perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.folio_wait_bit_common.folio_wait_writeback
      4.01 ± 26%      -2.3        1.67 ±  7%  perf-profile.calltrace.cycles-pp.raid_end_bio_io.raid1_end_write_request.__submit_bio.__submit_bio_noacct.flush_bio_list
      4.55 ± 21%      -2.2        2.31 ±  6%  perf-profile.calltrace.cycles-pp.flush_pending_writes.raid1d.md_thread.kthread.ret_from_fork
      4.55 ± 20%      -2.2        2.31 ±  6%  perf-profile.calltrace.cycles-pp.raid1d.md_thread.kthread.ret_from_fork
      4.55 ± 21%      -2.2        2.31 ±  6%  perf-profile.calltrace.cycles-pp.flush_bio_list.flush_pending_writes.raid1d.md_thread.kthread
      4.51 ± 21%      -2.2        2.27 ±  7%  perf-profile.calltrace.cycles-pp.__submit_bio_noacct.flush_bio_list.flush_pending_writes.raid1d.md_thread
      4.50 ± 21%      -2.2        2.27 ±  7%  perf-profile.calltrace.cycles-pp.__submit_bio.__submit_bio_noacct.flush_bio_list.flush_pending_writes.raid1d
      4.56 ± 20%      -2.2        2.33 ±  6%  perf-profile.calltrace.cycles-pp.md_thread.kthread.ret_from_fork
      4.94 ± 15%      -1.7        3.20 ±  3%  perf-profile.calltrace.cycles-pp.ret_from_fork
      4.94 ± 15%      -1.7        3.20 ±  3%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      3.31 ±  4%      -1.3        2.04 ± 20%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.io_schedule.folio_wait_bit_common
      3.28 ±  4%      -1.3        2.03 ± 20%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.io_schedule
      1.98 ±  5%      -0.8        1.14 ±  3%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.93 ±  4%      -0.8        1.10 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      1.26 ± 19%      -0.7        0.54 ± 52%  perf-profile.calltrace.cycles-pp.down_read.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file
      1.55 ±  9%      -0.7        0.88 ±  9%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      2.07 ±  2%      +0.3        2.36 ± 10%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.77 ±  7%      +0.4        1.15 ± 10%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry
      0.63 ±  7%      +0.4        1.03 ±  7%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle
      1.28 ±  4%      +0.4        1.69 ± 10%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      0.63 ± 12%      +0.5        1.12 ± 11%  perf-profile.calltrace.cycles-pp.prepare_write_begin.f2fs_write_begin.generic_perform_write.f2fs_buffered_write_iter.f2fs_file_write_iter
      2.95 ±  6%      +0.5        3.48 ±  6%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
      3.00 ±  6%      +0.5        3.52 ±  6%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
      0.26 ±100%      +0.6        0.82 ±  8%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue
      0.69 ± 15%      +0.6        1.30 ± 11%  perf-profile.calltrace.cycles-pp.f2fs_write_begin.generic_perform_write.f2fs_buffered_write_iter.f2fs_file_write_iter.new_sync_write
      0.27 ±100%      +0.6        0.88 ± 17%  perf-profile.calltrace.cycles-pp.brd_submit_bio.__submit_bio.__submit_bio_noacct.flush_bio_list.flush_pending_writes
      0.00            +0.6        0.62 ± 10%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback
      0.00            +0.7        0.66 ±  9%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback
      0.10 ±223%      +0.7        0.84 ± 17%  perf-profile.calltrace.cycles-pp.copy_to_brd.brd_do_bvec.brd_submit_bio.__submit_bio.__submit_bio_noacct
      0.10 ±223%      +0.8        0.86 ± 17%  perf-profile.calltrace.cycles-pp.raid1_end_write_request.__submit_bio.__submit_bio_noacct.flush_bio_list.raid1_unplug
      2.10 ±  7%      +0.8        2.88 ±  9%  perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_file_write_iter
      0.00            +0.8        0.78 ± 14%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.rwsem_down_read_slowpath.__submit_merged_write_cond
      0.70 ±  5%      +0.8        1.51 ± 14%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      0.08 ±223%      +0.8        0.93 ± 18%  perf-profile.calltrace.cycles-pp.f2fs_merge_page_bio.f2fs_inplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages
      0.00            +0.9        0.95 ± 39%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      0.10 ±223%      +1.0        1.05 ± 17%  perf-profile.calltrace.cycles-pp.f2fs_inplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.__f2fs_write_data_pages
      0.00            +1.0        0.95 ± 13%  perf-profile.calltrace.cycles-pp.__schedule.schedule.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback
      0.00            +1.0        0.96 ± 13%  perf-profile.calltrace.cycles-pp.schedule.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback
      0.00            +1.0        0.96 ± 20%  perf-profile.calltrace.cycles-pp.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.__f2fs_write_data_pages
      0.94 ± 22%      +1.0        1.91 ± 13%  perf-profile.calltrace.cycles-pp.generic_perform_write.f2fs_buffered_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write
      0.00            +1.0        0.98 ± 37%  perf-profile.calltrace.cycles-pp.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      0.00            +1.0        0.98 ± 16%  perf-profile.calltrace.cycles-pp.f2fs_submit_merged_ipu_write.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_file_write_iter
      0.95 ± 23%      +1.0        1.94 ± 13%  perf-profile.calltrace.cycles-pp.f2fs_buffered_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
      0.12 ±223%      +1.1        1.18 ± 11%  perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      3.13 ±  7%      +1.1        4.20 ± 13%  perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
      0.00            +1.1        1.08 ± 41%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.rwsem_down_read_slowpath
      0.39 ±105%      +1.1        1.47 ± 17%  perf-profile.calltrace.cycles-pp.brd_submit_bio.__submit_bio.__submit_bio_noacct.flush_bio_list.raid1_unplug
      0.09 ±223%      +1.1        1.19 ± 16%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page
      0.13 ±223%      +1.1        1.28 ± 30%  perf-profile.calltrace.cycles-pp.f2fs_is_checkpointed_node.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      0.13 ±223%      +1.2        1.37 ± 18%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
      0.13 ±223%      +1.2        1.37 ± 18%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.__wait_for_common
      0.00            +1.3        1.26 ± 12%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_file_write_iter
      0.39 ±104%      +1.3        1.74 ± 17%  perf-profile.calltrace.cycles-pp.f2fs_submit_merged_ipu_write.f2fs_write_cache_pages.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
      0.12 ±223%      +1.4        1.54 ± 15%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page
      0.15 ±223%      +1.4        1.59 ± 18%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__wait_for_common.f2fs_issue_flush
      0.15 ±223%      +1.4        1.60 ± 18%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__wait_for_common.f2fs_issue_flush.f2fs_do_sync_file
      0.15 ±223%      +1.4        1.60 ± 18%  perf-profile.calltrace.cycles-pp.schedule_timeout.__wait_for_common.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter
      0.16 ±223%      +1.5        1.64 ± 18%  perf-profile.calltrace.cycles-pp.__wait_for_common.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      0.39 ±110%      +1.5        1.88 ± 17%  perf-profile.calltrace.cycles-pp.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      0.27 ±155%      +1.5        1.79 ± 33%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.set_node_addr.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file
      0.30 ±154%      +1.6        1.93 ± 32%  perf-profile.calltrace.cycles-pp.set_node_addr.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      0.65 ±102%      +1.7        2.33 ± 17%  perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_submit_bio.__submit_bio.__submit_bio_noacct.flush_bio_list
      0.10 ±223%      +1.7        1.84 ± 10%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file
      0.61 ±106%      +1.8        2.40 ± 17%  perf-profile.calltrace.cycles-pp.__submit_bio.__submit_bio_noacct.flush_bio_list.raid1_unplug.__blk_flush_plug
      0.61 ±106%      +1.8        2.40 ± 17%  perf-profile.calltrace.cycles-pp.__submit_bio_noacct.flush_bio_list.raid1_unplug.__blk_flush_plug.blk_finish_plug
      0.73 ± 86%      +1.8        2.56 ± 17%  perf-profile.calltrace.cycles-pp.blk_finish_plug.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
      0.64 ±106%      +1.9        2.53 ± 17%  perf-profile.calltrace.cycles-pp.flush_bio_list.raid1_unplug.__blk_flush_plug.blk_finish_plug.__f2fs_write_data_pages
      0.15 ±223%      +1.9        2.04 ± 15%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page.__write_node_page
      0.64 ±106%      +1.9        2.54 ± 17%  perf-profile.calltrace.cycles-pp.raid1_unplug.__blk_flush_plug.blk_finish_plug.__f2fs_write_data_pages.do_writepages
      0.65 ±106%      +1.9        2.56 ± 17%  perf-profile.calltrace.cycles-pp.__blk_flush_plug.blk_finish_plug.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
      0.36 ±111%      +1.9        2.29 ± 15%  perf-profile.calltrace.cycles-pp.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages
      0.14 ±223%      +2.1        2.26 ± 24%  perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page
      0.36 ±109%      +2.3        2.63 ± 23%  perf-profile.calltrace.cycles-pp.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page.__write_node_page
      0.58 ±107%      +2.6        3.21 ± 19%  perf-profile.calltrace.cycles-pp.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.__f2fs_write_data_pages.do_writepages
      0.48 ±109%      +2.7        3.20 ± 22%  perf-profile.calltrace.cycles-pp.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages
      0.62 ±107%      +2.8        3.43 ± 19%  perf-profile.calltrace.cycles-pp.f2fs_write_single_data_page.f2fs_write_cache_pages.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
      1.39 ± 69%      +4.2        5.61 ± 18%  perf-profile.calltrace.cycles-pp.f2fs_write_cache_pages.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
      1.02 ± 91%      +4.9        5.89 ± 18%  perf-profile.calltrace.cycles-pp.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file
      1.02 ± 91%      +4.9        5.89 ± 18%  perf-profile.calltrace.cycles-pp.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
      2.34 ± 60%      +6.0        8.30 ± 17%  perf-profile.calltrace.cycles-pp.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range
      2.35 ± 60%      +6.0        8.30 ± 17%  perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file
      2.36 ± 60%      +6.0        8.32 ± 17%  perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter
      2.36 ± 60%      +6.0        8.33 ± 17%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      2.39 ± 60%      +6.1        8.47 ± 17%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
      2.00 ± 86%      +8.0        9.96 ± 22%  perf-profile.calltrace.cycles-pp.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
      2.89 ± 85%     +10.6       13.51 ± 22%  perf-profile.calltrace.cycles-pp.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
     19.95 ± 22%     +19.9       39.80 ±  9%  perf-profile.calltrace.cycles-pp.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
     20.97 ± 22%     +21.0       41.94 ±  9%  perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     20.97 ± 22%     +21.0       41.96 ±  9%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     21.37 ± 21%     +21.0       42.39 ±  9%  perf-profile.calltrace.cycles-pp.write
     21.00 ± 22%     +21.0       42.02 ±  9%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     21.00 ± 22%     +21.0       42.04 ±  9%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     21.02 ± 22%     +21.1       42.09 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     21.02 ± 22%     +21.1       42.10 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     73.04 ±  5%     -19.4       53.65 ±  7%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     73.04 ±  5%     -19.4       53.65 ±  7%  perf-profile.children.cycles-pp.cpu_startup_entry
     73.01 ±  5%     -19.4       53.63 ±  7%  perf-profile.children.cycles-pp.do_idle
     72.28 ±  5%     -19.2       53.04 ±  7%  perf-profile.children.cycles-pp.start_secondary
     68.17 ±  5%     -18.5       49.67 ±  7%  perf-profile.children.cycles-pp.cpuidle_idle_call
     66.30 ±  5%     -17.7       48.57 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter
     66.28 ±  5%     -17.7       48.56 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter_state
     16.17 ± 16%      -8.9        7.24 ± 34%  perf-profile.children.cycles-pp.poll_idle
     49.10 ±  2%      -8.4       40.65 ±  3%  perf-profile.children.cycles-pp.mwait_idle_with_hints
     25.27 ±  7%      -8.0       17.24 ± 23%  perf-profile.children.cycles-pp.intel_idle_irq
      4.32 ± 26%      -3.2        1.12 ± 39%  perf-profile.children.cycles-pp.folio_wake_bit
      3.82 ± 27%      -2.9        0.94 ± 37%  perf-profile.children.cycles-pp.__wake_up_common
      3.69 ± 28%      -2.8        0.85 ± 39%  perf-profile.children.cycles-pp.wake_page_function
      3.96 ± 25%      -2.7        1.27 ± 23%  perf-profile.children.cycles-pp.folio_end_writeback
      4.09 ± 23%      -2.5        1.64 ± 14%  perf-profile.children.cycles-pp.f2fs_write_end_io
      4.17 ± 23%      -2.4        1.80 ± 12%  perf-profile.children.cycles-pp.raid_end_bio_io
      5.26 ±  6%      -2.4        2.90 ± 21%  perf-profile.children.cycles-pp.io_schedule
      4.55 ± 21%      -2.2        2.31 ±  6%  perf-profile.children.cycles-pp.flush_pending_writes
      4.55 ± 20%      -2.2        2.31 ±  6%  perf-profile.children.cycles-pp.raid1d
      4.56 ± 20%      -2.2        2.33 ±  6%  perf-profile.children.cycles-pp.md_thread
      5.54 ± 17%      -2.1        3.43 ±  9%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      4.33 ± 20%      -2.1        2.23 ±  6%  perf-profile.children.cycles-pp.raid1_end_write_request
      4.01 ± 17%      -2.0        1.97 ±  6%  perf-profile.children.cycles-pp.try_to_wake_up
      4.94 ± 15%      -1.7        3.20 ±  3%  perf-profile.children.cycles-pp.ret_from_fork
      4.94 ± 15%      -1.7        3.20 ±  3%  perf-profile.children.cycles-pp.kthread
      2.01 ±  5%      -0.9        1.15 ±  3%  perf-profile.children.cycles-pp.schedule_idle
      1.58 ±  9%      -0.7        0.90 ±  9%  perf-profile.children.cycles-pp.menu_select
      1.84 ±  9%      -0.6        1.29 ±  7%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.51 ±  7%      -0.5        1.00 ±  4%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      1.61 ±  9%      -0.4        1.17 ±  5%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.76 ±  6%      -0.4        0.38 ±  4%  perf-profile.children.cycles-pp.update_rq_clock
      1.05 ±  8%      -0.3        0.71 ±  5%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      1.38 ± 12%      -0.3        1.05 ±  8%  perf-profile.children.cycles-pp.down_read
      1.04 ±  8%      -0.3        0.70 ±  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.54 ±  4%      -0.3        0.22 ±  9%  perf-profile.children.cycles-pp.set_next_entity
      1.26 ±  8%      -0.3        0.94 ±  5%  perf-profile.children.cycles-pp.enqueue_entity
      0.51 ± 20%      -0.3        0.23 ± 32%  perf-profile.children.cycles-pp.f2fs_put_page
      0.57 ± 14%      -0.3        0.30 ±  9%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.36 ± 11%      -0.2        0.12 ± 13%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.41 ± 16%      -0.2        0.18 ± 10%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.56 ±  9%      -0.2        0.35 ±  8%  perf-profile.children.cycles-pp.ktime_get
      0.55 ±  8%      -0.2        0.40 ±  4%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.18 ± 13%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.check_preempt_curr
      0.23 ± 12%      -0.1        0.09 ± 12%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.20 ± 14%      -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.37 ± 10%      -0.1        0.24 ±  9%  perf-profile.children.cycles-pp.clockevents_program_event
      0.42 ±  7%      -0.1        0.29 ± 10%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.40 ±  5%      -0.1        0.28 ± 11%  perf-profile.children.cycles-pp.__switch_to_asm
      0.96 ±  3%      -0.1        0.84 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      0.40 ±  9%      -0.1        0.29 ±  5%  perf-profile.children.cycles-pp.tick_sched_timer
      0.36 ±  6%      -0.1        0.26 ±  8%  perf-profile.children.cycles-pp.native_sched_clock
      0.37 ±  5%      -0.1        0.26 ±  8%  perf-profile.children.cycles-pp.__switch_to
      1.02 ±  2%      -0.1        0.93 ±  4%  perf-profile.children.cycles-pp.dequeue_entity
      0.18 ±  9%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.reweight_entity
      0.32 ± 10%      -0.1        0.23 ±  5%  perf-profile.children.cycles-pp.tick_sched_handle
      0.31 ± 12%      -0.1        0.22 ±  6%  perf-profile.children.cycles-pp.update_process_times
      0.26 ±  3%      -0.1        0.20 ±  6%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.45 ±  3%      -0.1        0.39 ±  3%  perf-profile.children.cycles-pp.update_curr
      0.14 ± 14%      -0.1        0.08 ± 13%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.24 ±  5%      -0.1        0.18 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.15 ±  4%      -0.1        0.10 ± 14%  perf-profile.children.cycles-pp.read_tsc
      0.14 ± 13%      -0.1        0.09 ± 10%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      1.14 ±  2%      -0.0        1.09 ±  3%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.12 ±  6%      -0.0        0.07 ±  9%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.20 ±  3%      -0.0        0.16 ±  4%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.08 ±  7%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.07 ± 12%      -0.0        0.03 ±102%  perf-profile.children.cycles-pp.call_cpuidle
      0.20 ± 11%      -0.0        0.15 ±  8%  perf-profile.children.cycles-pp.scheduler_tick
      0.07 ± 15%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.08 ± 14%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.06 ±  7%      -0.0        0.02 ± 99%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.12 ±  6%      -0.0        0.08 ±  7%  perf-profile.children.cycles-pp.rebalance_domains
      0.09 ± 16%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.rcu_dynticks_inc
      0.11 ±  6%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.09 ±  5%      -0.0        0.05 ±  7%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.10 ±  8%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__calc_delta
      0.07 ± 15%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.08 ±  5%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.08 ± 11%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.update_min_vruntime
      0.26            -0.0        0.24        perf-profile.children.cycles-pp.prepare_task_switch
      0.05 ± 47%      +0.0        0.08 ± 16%  perf-profile.children.cycles-pp.__slab_free
      0.09 ± 14%      +0.0        0.12 ±  7%  perf-profile.children.cycles-pp.__might_resched
      0.06 ± 21%      +0.0        0.11 ± 12%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.02 ±144%      +0.1        0.07 ± 16%  perf-profile.children.cycles-pp.bio_associate_blkg_from_css
      0.04 ± 75%      +0.1        0.09 ± 19%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.04 ± 75%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.queue_work_on
      0.03 ±103%      +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.__queue_work
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.children.cycles-pp._wait_barrier
      0.10 ± 14%      +0.1        0.16 ± 14%  perf-profile.children.cycles-pp.kmem_cache_free
      0.01 ±223%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.f2fs_lookup_rb_tree
      0.01 ±223%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.folio_clear_dirty_for_io
      0.01 ±223%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.update_sd_pick_busiest
      0.01 ±223%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.sb_mark_inode_writeback
      0.06 ± 56%      +0.1        0.13 ± 15%  perf-profile.children.cycles-pp.md_flush_request
      0.03 ±100%      +0.1        0.09 ± 14%  perf-profile.children.cycles-pp.__bio_alloc
      0.01 ±223%      +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.mutex_lock
      0.02 ±141%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.06 ± 51%      +0.1        0.13 ± 15%  perf-profile.children.cycles-pp.mempool_alloc
      0.02 ±142%      +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.copyin
      0.02 ±142%      +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.01 ±223%      +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.cpumask_next
      0.40 ±  3%      +0.1        0.46 ±  5%  perf-profile.children.cycles-pp.up_read
      0.01 ±223%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.rmqueue
      0.02 ±152%      +0.1        0.10 ± 23%  perf-profile.children.cycles-pp.xas_load
      0.01 ±223%      +0.1        0.08 ± 22%  perf-profile.children.cycles-pp.f2fs_update_dirty_folio
      0.04 ±104%      +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.submit_bio_noacct
      0.02 ±144%      +0.1        0.10 ± 19%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      0.01 ±223%      +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.read_node_page
      0.01 ±223%      +0.1        0.08 ± 22%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.05 ± 73%      +0.1        0.13 ± 14%  perf-profile.children.cycles-pp.bio_alloc_clone
      0.01 ±223%      +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.f2fs_update_data_blkaddr
      0.03 ±102%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.01 ±223%      +0.1        0.09 ± 23%  perf-profile.children.cycles-pp.f2fs_get_node_page
      0.01 ±223%      +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.wake_q_add
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.osq_unlock
      0.04 ± 78%      +0.1        0.13 ± 18%  perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush
      0.16 ± 17%      +0.1        0.25 ±  9%  perf-profile.children.cycles-pp.update_blocked_averages
      0.10 ± 34%      +0.1        0.19 ± 14%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.16 ± 32%      +0.1        0.25 ± 14%  perf-profile.children.cycles-pp.process_one_work
      0.04 ±105%      +0.1        0.13 ± 31%  perf-profile.children.cycles-pp.f2fs_invalidate_blocks
      0.01 ±223%      +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.wake_affine
      0.01 ±223%      +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.__is_cp_guaranteed
      0.07 ± 61%      +0.1        0.17 ± 18%  perf-profile.children.cycles-pp.mem_cgroup_wb_stats
      0.07 ± 58%      +0.1        0.17 ± 18%  perf-profile.children.cycles-pp.cgroup_rstat_flush_locked
      0.01 ±223%      +0.1        0.12 ± 10%  perf-profile.children.cycles-pp.__remove_ino_entry
      0.07 ± 58%      +0.1        0.17 ± 18%  perf-profile.children.cycles-pp.__mem_cgroup_flush_stats
      0.07 ± 58%      +0.1        0.17 ± 18%  perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe
      0.04 ±104%      +0.1        0.14 ± 30%  perf-profile.children.cycles-pp.f2fs_truncate
      0.04 ±104%      +0.1        0.14 ± 30%  perf-profile.children.cycles-pp.f2fs_do_truncate_blocks
      0.04 ±104%      +0.1        0.14 ± 30%  perf-profile.children.cycles-pp.f2fs_truncate_data_blocks_range
      0.07 ± 61%      +0.1        0.17 ± 17%  perf-profile.children.cycles-pp.balance_dirty_pages
      0.06 ± 83%      +0.1        0.16 ± 19%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.04 ± 76%      +0.1        0.15 ± 16%  perf-profile.children.cycles-pp.f2fs_del_fsync_node_entry
      0.01 ±223%      +0.1        0.13 ± 22%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      0.00            +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.__insert_extent_tree
      0.15 ± 31%      +0.1        0.27 ± 10%  perf-profile.children.cycles-pp.__submit_merged_bio
      0.03 ±106%      +0.1        0.15 ± 18%  perf-profile.children.cycles-pp.up_write
      0.04 ±109%      +0.1        0.15 ± 19%  perf-profile.children.cycles-pp.update_sit_entry
      0.14 ± 36%      +0.1        0.26 ± 16%  perf-profile.children.cycles-pp.submit_bio_wait
      0.14 ± 33%      +0.1        0.26 ± 16%  perf-profile.children.cycles-pp.blkdev_issue_flush
      0.14 ± 35%      +0.1        0.26 ± 17%  perf-profile.children.cycles-pp.__submit_flush_wait
      0.08 ± 60%      +0.1        0.21 ± 14%  perf-profile.children.cycles-pp.__has_merged_page
      0.08 ± 59%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.00            +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.f2fs_update_extent_tree_range
      0.06 ± 82%      +0.1        0.19 ± 14%  perf-profile.children.cycles-pp.f2fs_update_inode
      0.04 ±103%      +0.1        0.17 ± 17%  perf-profile.children.cycles-pp.__folio_mark_dirty
      0.04 ±115%      +0.1        0.17 ± 20%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.08 ± 38%      +0.1        0.22 ± 21%  perf-profile.children.cycles-pp.clear_page_erms
      0.16 ± 28%      +0.1        0.30 ± 12%  perf-profile.children.cycles-pp.f2fs_write_inode
      0.10 ± 52%      +0.1        0.24 ± 17%  perf-profile.children.cycles-pp.__filemap_get_folio
      0.10 ± 52%      +0.1        0.24 ± 16%  perf-profile.children.cycles-pp.pagecache_get_page
      0.04 ±112%      +0.2        0.19 ± 20%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.05 ±106%      +0.2        0.20 ± 14%  perf-profile.children.cycles-pp.down_write
      0.05 ±104%      +0.2        0.20 ± 20%  perf-profile.children.cycles-pp.f2fs_dirty_data_folio
      0.03 ±155%      +0.2        0.18 ± 18%  perf-profile.children.cycles-pp.f2fs_balance_fs
      0.05 ±117%      +0.2        0.21 ± 27%  perf-profile.children.cycles-pp.xas_find_marked
      0.05 ±109%      +0.2        0.21 ± 18%  perf-profile.children.cycles-pp.complete
      0.08 ± 72%      +0.2        0.25 ± 14%  perf-profile.children.cycles-pp.__get_node_page
      0.07 ± 82%      +0.2        0.24 ± 25%  perf-profile.children.cycles-pp.__dentry_kill
      0.07 ± 82%      +0.2        0.24 ± 25%  perf-profile.children.cycles-pp.evict
      0.07 ± 82%      +0.2        0.24 ± 25%  perf-profile.children.cycles-pp.f2fs_evict_inode
      0.22 ± 33%      +0.2        0.38 ± 14%  perf-profile.children.cycles-pp.worker_thread
      0.11 ± 54%      +0.2        0.28 ± 19%  perf-profile.children.cycles-pp.f2fs_write_end
      0.08 ± 65%      +0.2        0.25 ± 14%  perf-profile.children.cycles-pp.f2fs_update_inode_page
      0.07 ± 81%      +0.2        0.24 ± 25%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.07 ± 82%      +0.2        0.24 ± 26%  perf-profile.children.cycles-pp.dput
      0.07 ± 82%      +0.2        0.24 ± 26%  perf-profile.children.cycles-pp.__fput
      0.07 ± 82%      +0.2        0.24 ± 26%  perf-profile.children.cycles-pp.dentry_kill
      0.07 ± 82%      +0.2        0.24 ± 25%  perf-profile.children.cycles-pp.task_work_run
      0.07 ± 82%      +0.2        0.24 ± 25%  perf-profile.children.cycles-pp.__close
      0.05 ±109%      +0.2        0.23 ± 15%  perf-profile.children.cycles-pp.f2fs_dirty_node_folio
      0.21 ± 26%      +0.2        0.40 ± 15%  perf-profile.children.cycles-pp.finish_task_switch
      0.05 ±110%      +0.2        0.24 ± 19%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.08 ± 82%      +0.2        0.28 ± 24%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.09 ± 66%      +0.2        0.28 ± 16%  perf-profile.children.cycles-pp.filemap_dirty_folio
      0.10 ± 63%      +0.2        0.30 ± 18%  perf-profile.children.cycles-pp.__folio_end_writeback
      0.12 ± 39%      +0.2        0.32 ± 20%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.08 ± 83%      +0.2        0.29 ± 23%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.12 ± 44%      +0.2        0.32 ± 13%  perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
      0.05 ±111%      +0.2        0.26 ± 20%  perf-profile.children.cycles-pp.f2fs_is_valid_blkaddr
      0.13 ± 41%      +0.2        0.34 ± 21%  perf-profile.children.cycles-pp.__alloc_pages
      0.26 ± 25%      +0.2        0.48 ± 10%  perf-profile.children.cycles-pp._find_next_bit
      0.04 ±114%      +0.2        0.28 ± 26%  perf-profile.children.cycles-pp.__lookup_nat_cache
      0.08 ± 74%      +0.2        0.33 ± 14%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.16 ± 40%      +0.2        0.41 ± 16%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.34 ± 21%      +0.2        0.58 ± 10%  perf-profile.children.cycles-pp.cpumask_next_and
      0.06 ±117%      +0.2        0.31 ± 29%  perf-profile.children.cycles-pp.release_pages
      0.09 ± 88%      +0.2        0.34 ± 15%  perf-profile.children.cycles-pp.find_busiest_queue
      0.07 ±115%      +0.2        0.32 ± 29%  perf-profile.children.cycles-pp.__pagevec_release
      0.09 ± 70%      +0.3        0.34 ± 16%  perf-profile.children.cycles-pp.__folio_start_writeback
      0.70 ± 14%      +0.3        0.96 ±  8%  perf-profile.children.cycles-pp.idle_cpu
      0.24 ± 38%      +0.3        0.55 ± 12%  perf-profile.children.cycles-pp.raid1_write_request
      2.08 ±  2%      +0.3        2.40 ± 10%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.06 ±114%      +0.3        0.40 ± 21%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.13 ± 89%      +0.3        0.47 ± 19%  perf-profile.children.cycles-pp.issue_flush_thread
      0.34 ± 40%      +0.4        0.77 ± 13%  perf-profile.children.cycles-pp.raid1_make_request
      0.37 ± 40%      +0.5        0.85 ± 14%  perf-profile.children.cycles-pp.md_handle_request
      0.63 ± 12%      +0.5        1.12 ± 11%  perf-profile.children.cycles-pp.prepare_write_begin
      0.40 ± 40%      +0.5        0.91 ± 14%  perf-profile.children.cycles-pp.md_submit_bio
      1.32 ±  3%      +0.5        1.83 ±  8%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.14 ± 79%      +0.5        0.66 ± 18%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.41 ± 37%      +0.5        0.95 ± 18%  perf-profile.children.cycles-pp.brd_insert_page
      0.35 ± 78%      +0.6        0.94 ± 17%  perf-profile.children.cycles-pp.wake_up_q
      0.69 ± 15%      +0.6        1.30 ± 11%  perf-profile.children.cycles-pp.f2fs_write_begin
      0.21 ±108%      +0.6        0.84 ± 30%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.21 ±108%      +0.6        0.84 ± 29%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.36 ± 51%      +0.6        0.99 ± 17%  perf-profile.children.cycles-pp.copy_to_brd
      0.24 ± 69%      +0.7        0.93 ± 18%  perf-profile.children.cycles-pp.f2fs_merge_page_bio
      0.20 ± 69%      +0.8        0.95 ± 38%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.19 ±109%      +0.8        0.96 ± 30%  perf-profile.children.cycles-pp.f2fs_need_inode_block_update
      0.27 ± 69%      +0.8        1.05 ± 17%  perf-profile.children.cycles-pp.f2fs_inplace_write_data
      0.20 ± 69%      +0.8        0.98 ± 37%  perf-profile.children.cycles-pp.f2fs_space_for_roll_forward
      0.18 ± 73%      +0.8        0.97 ± 20%  perf-profile.children.cycles-pp.f2fs_outplace_write_data
      0.44 ± 75%      +0.9        1.32 ± 17%  perf-profile.children.cycles-pp.rwsem_wake
      0.17 ± 78%      +0.9        1.12 ± 21%  perf-profile.children.cycles-pp.f2fs_lookup_extent_tree
      0.95 ± 22%      +1.0        1.92 ± 13%  perf-profile.children.cycles-pp.generic_perform_write
      0.95 ± 23%      +1.0        1.94 ± 13%  perf-profile.children.cycles-pp.f2fs_buffered_write_iter
      0.27 ±102%      +1.0        1.28 ± 30%  perf-profile.children.cycles-pp.f2fs_is_checkpointed_node
      0.25 ±108%      +1.1        1.31 ± 34%  perf-profile.children.cycles-pp.f2fs_get_node_info
      0.34 ± 93%      +1.3        1.62 ± 18%  perf-profile.children.cycles-pp.schedule_timeout
      0.38 ± 89%      +1.3        1.71 ± 18%  perf-profile.children.cycles-pp.__wait_for_common
      1.22 ± 16%      +1.4        2.60 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock
      0.48 ± 73%      +1.4        1.88 ± 17%  perf-profile.children.cycles-pp.f2fs_issue_flush
      0.93 ± 42%      +1.4        2.33 ± 17%  perf-profile.children.cycles-pp.brd_do_bvec
      0.94 ± 42%      +1.4        2.35 ± 17%  perf-profile.children.cycles-pp.brd_submit_bio
      0.42 ±100%      +1.5        1.94 ± 32%  perf-profile.children.cycles-pp.set_node_addr
      2.08 ±  5%      +1.6        3.66 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.86 ± 57%      +1.7        2.54 ± 17%  perf-profile.children.cycles-pp.raid1_unplug
      0.87 ± 56%      +1.7        2.56 ± 17%  perf-profile.children.cycles-pp.blk_finish_plug
      0.87 ± 56%      +1.7        2.56 ± 17%  perf-profile.children.cycles-pp.__blk_flush_plug
      2.59 ±  6%      +1.8        4.37 ±  5%  perf-profile.children.cycles-pp.__submit_merged_write_cond
      0.78 ± 55%      +1.9        2.73 ± 14%  perf-profile.children.cycles-pp.f2fs_submit_merged_ipu_write
      0.46 ± 81%      +2.0        2.50 ± 15%  perf-profile.children.cycles-pp.f2fs_submit_page_write
      3.67 ± 20%      +2.1        5.78 ±  8%  perf-profile.children.cycles-pp.update_sg_lb_stats
      8.63 ± 10%      +2.1       10.74 ±  6%  perf-profile.children.cycles-pp.__schedule
      3.82 ± 20%      +2.2        6.05 ±  8%  perf-profile.children.cycles-pp.update_sd_lb_stats
      3.87 ± 20%      +2.3        6.14 ±  8%  perf-profile.children.cycles-pp.find_busiest_group
      0.47 ± 75%      +2.4        2.91 ± 23%  perf-profile.children.cycles-pp.__mutex_lock
      0.75 ± 64%      +2.5        3.21 ± 19%  perf-profile.children.cycles-pp.f2fs_do_write_data_page
      0.80 ± 65%      +2.6        3.43 ± 19%  perf-profile.children.cycles-pp.f2fs_write_single_data_page
      5.18 ± 18%      +2.7        7.84 ±  8%  perf-profile.children.cycles-pp.pick_next_task_fair
      4.12 ± 22%      +2.8        6.91 ±  9%  perf-profile.children.cycles-pp.load_balance
      6.70 ± 15%      +2.9        9.64 ±  7%  perf-profile.children.cycles-pp.schedule
      0.71 ± 82%      +2.9        3.65 ± 18%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.69 ± 72%      +3.0        3.72 ± 22%  perf-profile.children.cycles-pp.f2fs_allocate_data_block
      4.44 ± 22%      +3.1        7.50 ±  9%  perf-profile.children.cycles-pp.newidle_balance
      2.65 ± 24%      +3.4        6.02 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.86 ± 78%      +4.2        5.04 ± 21%  perf-profile.children.cycles-pp.osq_lock
      1.44 ± 62%      +4.2        5.62 ± 18%  perf-profile.children.cycles-pp.f2fs_write_cache_pages
      1.14 ± 84%      +4.6        5.72 ± 19%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      1.12 ± 75%      +4.8        5.89 ± 18%  perf-profile.children.cycles-pp.f2fs_do_write_node_page
      1.26 ± 74%      +5.4        6.64 ± 18%  perf-profile.children.cycles-pp.do_write_page
      1.35 ± 85%      +5.8        7.16 ± 19%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      3.33 ± 24%      +5.9        9.20 ±  5%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      2.34 ± 60%      +6.0        8.30 ± 17%  perf-profile.children.cycles-pp.__f2fs_write_data_pages
      2.35 ± 60%      +6.0        8.30 ± 17%  perf-profile.children.cycles-pp.do_writepages
      2.36 ± 60%      +6.0        8.32 ± 17%  perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
      2.36 ± 60%      +6.0        8.33 ± 17%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      2.39 ± 60%      +6.1        8.47 ± 17%  perf-profile.children.cycles-pp.file_write_and_wait_range
      2.03 ± 84%      +7.9        9.96 ± 22%  perf-profile.children.cycles-pp.__write_node_page
      2.95 ± 81%     +10.6       13.52 ± 22%  perf-profile.children.cycles-pp.f2fs_fsync_node_pages
     19.95 ± 22%     +19.9       39.80 ±  9%  perf-profile.children.cycles-pp.f2fs_do_sync_file
     20.97 ± 22%     +21.0       41.94 ±  9%  perf-profile.children.cycles-pp.f2fs_file_write_iter
     20.98 ± 22%     +21.0       41.97 ±  9%  perf-profile.children.cycles-pp.new_sync_write
     21.00 ± 22%     +21.0       42.04 ±  9%  perf-profile.children.cycles-pp.vfs_write
     21.01 ± 22%     +21.0       42.05 ±  9%  perf-profile.children.cycles-pp.ksys_write
     21.37 ± 21%     +21.0       42.41 ±  9%  perf-profile.children.cycles-pp.write
     21.18 ± 22%     +21.3       42.45 ±  9%  perf-profile.children.cycles-pp.do_syscall_64
     21.19 ± 22%     +21.3       42.46 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     15.91 ± 16%      -8.8        7.11 ± 34%  perf-profile.self.cycles-pp.poll_idle
     48.31 ±  2%      -8.0       40.27 ±  2%  perf-profile.self.cycles-pp.mwait_idle_with_hints
      1.05 ±  8%      -0.5        0.52 ± 31%  perf-profile.self.cycles-pp.folio_wait_bit_common
      0.87 ± 11%      -0.4        0.46 ± 24%  perf-profile.self.cycles-pp.f2fs_wait_on_node_pages_writeback
      0.91 ±  8%      -0.4        0.53 ± 10%  perf-profile.self.cycles-pp.menu_select
      0.60 ±  7%      -0.3        0.27 ±  4%  perf-profile.self.cycles-pp.update_rq_clock
      1.27 ± 13%      -0.3        0.96 ±  7%  perf-profile.self.cycles-pp.down_read
      0.35 ± 11%      -0.2        0.12 ± 13%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.69            -0.2        0.47 ±  2%  perf-profile.self.cycles-pp.__schedule
      0.27 ±  8%      -0.2        0.05 ±  8%  perf-profile.self.cycles-pp.set_next_entity
      0.31 ± 16%      -0.2        0.12 ± 39%  perf-profile.self.cycles-pp.folio_wait_writeback
      0.52 ±  9%      -0.2        0.35 ±  6%  perf-profile.self.cycles-pp.enqueue_entity
      0.42 ± 12%      -0.2        0.25 ± 10%  perf-profile.self.cycles-pp.ktime_get
      0.36 ± 12%      -0.1        0.23 ±  4%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.40 ±  5%      -0.1        0.28 ± 11%  perf-profile.self.cycles-pp.__switch_to_asm
      0.22 ±  9%      -0.1        0.10 ±  9%  perf-profile.self.cycles-pp.do_idle
      0.36 ±  5%      -0.1        0.26 ±  8%  perf-profile.self.cycles-pp.__switch_to
      0.23 ± 16%      -0.1        0.12 ± 32%  perf-profile.self.cycles-pp.ttwu_do_activate
      0.35 ±  6%      -0.1        0.24 ±  7%  perf-profile.self.cycles-pp.native_sched_clock
      0.36 ±  5%      -0.1        0.27 ± 11%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.18 ±  9%      -0.1        0.10 ± 14%  perf-profile.self.cycles-pp.reweight_entity
      1.12 ±  2%      -0.1        1.04 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.24 ±  5%      -0.1        0.18 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.12 ±  8%      -0.1        0.06 ± 49%  perf-profile.self.cycles-pp.f2fs_wait_on_page_writeback
      0.15 ±  4%      -0.1        0.10 ± 10%  perf-profile.self.cycles-pp.read_tsc
      0.31 ± 12%      -0.1        0.26 ±  4%  perf-profile.self.cycles-pp.llist_reverse_order
      0.13 ± 10%      -0.1        0.08 ± 14%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.07 ± 12%      -0.0        0.03 ±102%  perf-profile.self.cycles-pp.call_cpuidle
      0.09 ± 17%      -0.0        0.05 ± 47%  perf-profile.self.cycles-pp.rcu_dynticks_inc
      0.13 ± 11%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.11 ±  6%      -0.0        0.07 ± 12%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.09 ±  5%      -0.0        0.05 ±  7%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.27 ±  5%      -0.0        0.23 ±  2%  perf-profile.self.cycles-pp.__list_add_valid
      0.09 ± 11%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.__calc_delta
      0.07 ±  6%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.update_min_vruntime
      0.08 ±  5%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.12 ±  6%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.dequeue_entity
      0.15 ±  4%      +0.0        0.18 ±  6%  perf-profile.self.cycles-pp.update_curr
      0.13 ± 12%      +0.0        0.16 ±  5%  perf-profile.self.cycles-pp.finish_task_switch
      0.08 ± 11%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.__might_resched
      0.10 ±  9%      +0.0        0.14 ±  4%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.04 ± 71%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.find_busiest_group
      0.42 ±  4%      +0.0        0.46 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.10 ± 13%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.02 ±144%      +0.1        0.07 ± 16%  perf-profile.self.cycles-pp.bio_associate_blkg_from_css
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.self.cycles-pp._wait_barrier
      0.15 ±  6%      +0.1        0.20 ±  9%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.01 ±223%      +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.mutex_lock
      0.10 ± 23%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.cpumask_next_and
      0.40 ±  3%      +0.1        0.46 ±  4%  perf-profile.self.cycles-pp.up_read
      0.01 ±223%      +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.f2fs_lookup_rb_tree
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.update_sd_pick_busiest
      0.01 ±223%      +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.__folio_start_writeback
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.f2fs_file_write_iter
      0.02 ±142%      +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.01 ±223%      +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.08 ± 24%      +0.1        0.15 ±  7%  perf-profile.self.cycles-pp.load_balance
      0.01 ±223%      +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.raid_end_bio_io
      0.01 ±223%      +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.md_handle_request
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.f2fs_issue_flush
      0.03 ±102%      +0.1        0.11 ± 21%  perf-profile.self.cycles-pp.brd_do_bvec
      0.01 ±223%      +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.f2fs_submit_page_write
      0.04 ± 76%      +0.1        0.12 ± 18%  perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush
      0.01 ±223%      +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.__write_node_page
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.osq_unlock
      0.01 ±223%      +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.wake_q_add
      0.17 ± 15%      +0.1        0.26 ±  4%  perf-profile.self.cycles-pp.newidle_balance
      0.01 ±223%      +0.1        0.10 ± 13%  perf-profile.self.cycles-pp.f2fs_inplace_write_data
      0.01 ±223%      +0.1        0.10 ± 22%  perf-profile.self.cycles-pp.f2fs_write_end_io
      0.00            +0.1        0.09 ± 23%  perf-profile.self.cycles-pp.f2fs_get_node_page
      0.12 ± 12%      +0.1        0.21 ± 14%  perf-profile.self.cycles-pp.f2fs_submit_merged_ipu_write
      0.02 ±144%      +0.1        0.11 ± 20%  perf-profile.self.cycles-pp.__f2fs_write_data_pages
      0.12 ± 23%      +0.1        0.21 ±  9%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.03 ±105%      +0.1        0.13 ± 17%  perf-profile.self.cycles-pp.__filemap_get_folio
      0.01 ±223%      +0.1        0.11 ± 17%  perf-profile.self.cycles-pp.__is_cp_guaranteed
      0.01 ±223%      +0.1        0.12 ± 17%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.03 ±106%      +0.1        0.15 ± 18%  perf-profile.self.cycles-pp.update_sit_entry
      0.03 ±106%      +0.1        0.15 ± 20%  perf-profile.self.cycles-pp.up_write
      0.08 ± 63%      +0.1        0.20 ± 14%  perf-profile.self.cycles-pp.__has_merged_page
      0.03 ±111%      +0.1        0.16 ± 13%  perf-profile.self.cycles-pp.f2fs_do_sync_file
      0.04 ±115%      +0.1        0.18 ± 25%  perf-profile.self.cycles-pp.f2fs_fsync_node_pages
      0.04 ±110%      +0.1        0.18 ± 14%  perf-profile.self.cycles-pp.down_write
      0.03 ±155%      +0.1        0.16 ± 19%  perf-profile.self.cycles-pp.f2fs_allocate_data_block
      0.07 ± 59%      +0.1        0.21 ± 21%  perf-profile.self.cycles-pp.clear_page_erms
      0.05 ±117%      +0.2        0.21 ± 27%  perf-profile.self.cycles-pp.xas_find_marked
      0.04 ±112%      +0.2        0.20 ± 13%  perf-profile.self.cycles-pp.rwsem_mark_wake
      0.10 ± 45%      +0.2        0.27 ± 12%  perf-profile.self.cycles-pp.raid1_write_request
      0.02 ±223%      +0.2        0.18 ± 17%  perf-profile.self.cycles-pp.f2fs_balance_fs
      0.08 ± 81%      +0.2        0.27 ± 28%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.24 ± 26%      +0.2        0.45 ± 10%  perf-profile.self.cycles-pp._find_next_bit
      0.05 ±111%      +0.2        0.26 ± 20%  perf-profile.self.cycles-pp.f2fs_is_valid_blkaddr
      0.08 ± 87%      +0.2        0.29 ± 16%  perf-profile.self.cycles-pp.find_busiest_queue
      0.05 ±112%      +0.2        0.26 ± 19%  perf-profile.self.cycles-pp.f2fs_lookup_extent_tree
      0.13 ± 46%      +0.2        0.36 ± 17%  perf-profile.self.cycles-pp.raid1_end_write_request
      0.06 ±120%      +0.2        0.29 ± 29%  perf-profile.self.cycles-pp.release_pages
      0.16 ± 40%      +0.2        0.40 ± 16%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.68 ± 14%      +0.2        0.93 ±  7%  perf-profile.self.cycles-pp.idle_cpu
      0.08 ± 94%      +0.3        0.37 ± 19%  perf-profile.self.cycles-pp.rwsem_optimistic_spin
      0.06 ±114%      +0.3        0.40 ± 21%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.09 ± 77%      +0.4        0.46 ± 17%  perf-profile.self.cycles-pp.rwsem_down_read_slowpath
      0.15 ±117%      +0.5        0.63 ± 31%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.13 ± 78%      +0.5        0.61 ± 18%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      1.07 ± 10%      +0.5        1.56 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock
      0.34 ± 50%      +0.6        0.95 ± 17%  perf-profile.self.cycles-pp.copy_to_brd
      2.70 ± 22%      +1.6        4.35 ±  9%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.86 ± 78%      +4.1        5.01 ± 21%  perf-profile.self.cycles-pp.osq_lock
      3.32 ± 24%      +5.9        9.19 ±  5%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
lkp-csl-2sp9: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-11/performance/4BRD_12G/xfs/x86_64-rhel-8.3/300/RAID1/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp9/sync_disk_rw/aim7/0x500320a

commit: 
  28156108fe ("sched: Fix the check of nr_running at queue wakelist")
  f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      6574            +3.9%       6832        aim7.jobs-per-min
    273.98            -3.8%     263.61        aim7.time.elapsed_time
    273.98            -3.8%     263.61        aim7.time.elapsed_time.max
    980796            +6.2%    1041949        aim7.time.involuntary_context_switches
     47635 ±  2%      -7.0%      44310        aim7.time.minor_page_faults
      6110 ±  2%      -8.0%       5624        aim7.time.system_time
  69966039            +2.2%   71483752        aim7.time.voluntary_context_switches
     34031 ±  6%     +18.8%      40422 ± 10%  meminfo.KernelStack
     16884 ± 10%     +26.3%      21319 ± 14%  numa-meminfo.node1.KernelStack
     16885 ± 10%     +26.2%      21316 ± 14%  numa-vmstat.node1.nr_kernel_stack
      5.77 ±  9%     +27.8%       7.38 ± 14%  sched_debug.cpu.clock.stddev
      9750           -14.0%       8382        uptime.idle
     31.73 ±  2%     -15.0%      26.97        iostat.cpu.idle
     40.12           +13.8%      45.67        iostat.cpu.iowait
     31.25 ±  3%      -4.8       26.44        mpstat.cpu.all.idle%
     40.41            +5.6       46.02        mpstat.cpu.all.iowait%
     34031 ±  6%     +18.8%      40418 ± 10%  proc-vmstat.nr_kernel_stack
     56829            +3.4%      58750 ±  2%  proc-vmstat.nr_slab_unreclaimable
    465583            +3.6%     482389        proc-vmstat.pgactivate
     34967            -3.0%      33923        proc-vmstat.pgreuse
     31.00 ±  3%     -15.1%      26.33        vmstat.cpu.id
     39.50           +14.8%      45.33        vmstat.cpu.wa
    273156            +4.0%     284008        vmstat.io.bo
    509661            +5.3%     536464        vmstat.system.cs
    200604            +4.7%     210117        vmstat.system.in
  38505571            +7.8%   41516807        perf-stat.i.branch-misses
     18.74            +0.9       19.61 ±  2%  perf-stat.i.cache-miss-rate%
  45792562            +6.5%   48766141        perf-stat.i.cache-misses
    514616            +5.2%     541341        perf-stat.i.context-switches
      4.99            -3.4%       4.81        perf-stat.i.cpi
     89102            +7.5%      95767        perf-stat.i.cpu-migrations
 1.291e+09            +6.3%  1.373e+09        perf-stat.i.dTLB-stores
   4605702        +6.9e+06%    3.2e+11 ±223%  perf-stat.i.iTLB-load-misses
      3333            +1.5%       3385        perf-stat.i.minor-faults
  16366919            +7.2%   17545996        perf-stat.i.node-load-misses
   2107545            +5.4%    2222279        perf-stat.i.node-loads
     73.73            +2.4       76.10        perf-stat.i.node-store-miss-rate%
   8621292 ±  2%      +9.0%    9396473        perf-stat.i.node-store-misses
      3335            +1.5%       3386        perf-stat.i.page-faults
      1.30            +0.1        1.40        perf-stat.overall.branch-miss-rate%
     18.82            +0.9       19.72        perf-stat.overall.cache-miss-rate%
      4.97            -3.0%       4.82        perf-stat.overall.cpi
      1533            -8.5%       1402        perf-stat.overall.cycles-between-cache-misses
      0.20            +3.1%       0.21        perf-stat.overall.ipc
     74.18 ±  2%      +2.2       76.38        perf-stat.overall.node-store-miss-rate%
  38363874            +7.8%   41358628        perf-stat.ps.branch-misses
  45621719            +6.5%   48579204        perf-stat.ps.cache-misses
    512684            +5.2%     539255        perf-stat.ps.context-switches
     88768            +7.5%      95397        perf-stat.ps.cpu-migrations
 1.286e+09            +6.3%  1.368e+09        perf-stat.ps.dTLB-stores
   4588811          +7e+06%   3.19e+11 ±223%  perf-stat.ps.iTLB-load-misses
  16305621            +7.2%   17478491        perf-stat.ps.node-load-misses
   2099675            +5.4%    2213764        perf-stat.ps.node-loads
   8589098 ±  2%      +9.0%    9360368        perf-stat.ps.node-store-misses
 3.869e+12            -3.3%  3.743e+12        perf-stat.total.instructions
      5.91 ±  2%      -1.2        4.68 ±  3%  perf-profile.calltrace.cycles-pp.__mutex_lock.__flush_workqueue.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq
      5.52 ±  2%      -1.2        4.31 ±  3%  perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.__flush_workqueue.xlog_cil_push_now.xlog_cil_force_seq
      7.38 ±  2%      -1.0        6.42 ±  3%  perf-profile.calltrace.cycles-pp.__flush_workqueue.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync
      2.32 ±  2%      -1.0        1.37 ±  2%  perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.xfs_file_fsync.xfs_file_buffered_write.new_sync_write.vfs_write
      2.01            -0.9        1.14        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.xfs_file_fsync
      2.01            -0.9        1.15 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.xfs_file_fsync.xfs_file_buffered_write
      2.01            -0.9        1.15 ±  2%  perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.xfs_file_fsync.xfs_file_buffered_write.new_sync_write
      8.08            -0.8        7.28 ±  3%  perf-profile.calltrace.cycles-pp.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write
      1.07 ±  2%      -0.5        0.59 ±  4%  perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write
      1.07 ±  2%      -0.5        0.59 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync
      1.06 ±  3%      -0.5        0.58 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_force_lsn.xfs_log_force_seq
      1.50            -0.4        1.11        perf-profile.calltrace.cycles-pp.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write.new_sync_write
      0.72 ±  4%      -0.1        0.63 ±  5%  perf-profile.calltrace.cycles-pp.xlog_ioend_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.59 ±  3%      +0.1        0.65 ±  2%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write
      0.84 ±  7%      +0.1        0.93 ±  4%  perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_vn_update_time.file_update_time
      1.15 ±  4%      +0.1        1.25 ±  4%  perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_write_checks
      1.22 ±  4%      +0.1        1.32 ±  4%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_write_checks.xfs_file_buffered_write
      0.65 ±  6%      +0.2        0.81 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq
      0.68 ±  6%      +0.2        0.84 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync
      0.72 ±  4%      +0.6        1.32 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_seq.xfs_log_force_seq
      0.72 ±  4%      +0.6        1.34 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync
      0.73 ±  4%      +0.6        1.34 ±  4%  perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write
     10.11            +0.6       10.74        perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write.new_sync_write
      0.00            +0.7        0.66 ±  3%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
     14.74 ±  2%      +0.9       15.61 ±  2%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_write.new_sync_write.vfs_write
      0.00            +0.9        0.87 ±  2%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      5.91 ±  2%      -1.2        4.68 ±  3%  perf-profile.children.cycles-pp.__mutex_lock
      5.56 ±  2%      -1.2        4.35 ±  3%  perf-profile.children.cycles-pp.osq_lock
      7.38 ±  2%      -1.0        6.42 ±  3%  perf-profile.children.cycles-pp.__flush_workqueue
      8.08            -0.8        7.28 ±  3%  perf-profile.children.cycles-pp.xlog_cil_push_now
      2.02 ±  2%      -0.5        1.51        perf-profile.children.cycles-pp.xlog_force_lsn
      1.46            -0.2        1.27 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
      1.16 ±  2%      -0.2        1.02 ±  3%  perf-profile.children.cycles-pp.__wake_up_common
      0.72 ±  4%      -0.1        0.63 ±  5%  perf-profile.children.cycles-pp.xlog_ioend_work
      1.16 ±  2%      -0.1        1.10 ±  4%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.56            -0.1        0.50 ±  2%  perf-profile.children.cycles-pp.schedule_idle
      0.28 ±  4%      -0.1        0.22 ±  5%  perf-profile.children.cycles-pp.folio_wake_bit
      0.23 ±  4%      -0.1        0.17 ±  6%  perf-profile.children.cycles-pp.wake_page_function
      0.47 ±  4%      -0.1        0.41 ±  5%  perf-profile.children.cycles-pp.iomap_finish_ioend
      0.40 ±  4%      -0.1        0.34 ±  4%  perf-profile.children.cycles-pp.folio_end_writeback
      0.14 ±  6%      -0.1        0.09 ± 10%  perf-profile.children.cycles-pp.set_next_entity
      0.45 ±  3%      -0.0        0.40 ±  6%  perf-profile.children.cycles-pp.xlog_state_clean_iclog
      0.45 ±  4%      -0.0        0.41 ±  3%  perf-profile.children.cycles-pp.raid_end_bio_io
      0.12 ±  6%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xlog_state_done_syncing
      0.36 ±  3%      -0.0        0.32 ±  2%  perf-profile.children.cycles-pp.complete
      0.08 ±  5%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      0.14 ±  4%      +0.0        0.16 ±  3%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.08 ±  7%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.15 ±  2%      +0.0        0.16 ±  4%  perf-profile.children.cycles-pp.iomap_write_end
      0.11 ±  8%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.__switch_to
      0.10 ±  4%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.prepare_task_switch
      0.18 ±  5%      +0.0        0.20 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.18 ±  3%      +0.0        0.20 ±  4%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.15 ±  5%      +0.0        0.17 ±  6%  perf-profile.children.cycles-pp.finish_task_switch
      0.06 ±  8%      +0.0        0.08 ±  4%  perf-profile.children.cycles-pp.llist_reverse_order
      0.12 ±  7%      +0.0        0.14 ±  4%  perf-profile.children.cycles-pp.wake_affine
      0.03 ± 70%      +0.0        0.06 ±  6%  perf-profile.children.cycles-pp.___perf_sw_event
      0.26 ±  3%      +0.0        0.29 ±  4%  perf-profile.children.cycles-pp.update_curr
      0.15 ±  7%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.33 ±  4%      +0.0        0.36 ±  4%  perf-profile.children.cycles-pp.xlog_cil_push_work
      0.51 ±  2%      +0.0        0.55 ±  3%  perf-profile.children.cycles-pp.dequeue_entity
      0.46 ±  4%      +0.0        0.51 ±  3%  perf-profile.children.cycles-pp.select_task_rq
      0.15 ±  5%      +0.0        0.20 ±  4%  perf-profile.children.cycles-pp.xlog_write
      0.09 ±  5%      +0.0        0.14 ±  6%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.48 ±  3%      +0.1        0.53 ±  4%  perf-profile.children.cycles-pp.iomap_write_iter
      0.08 ±  9%      +0.1        0.13 ±  4%  perf-profile.children.cycles-pp.llist_add_batch
      0.08 ± 11%      +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.__smp_call_single_queue
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.xlog_cil_write_commit_record
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.xlog_cil_set_ctx_write_state
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.xlog_write_get_more_iclog_space
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.xlog_write_partial
      0.59 ±  3%      +0.1        0.65 ±  2%  perf-profile.children.cycles-pp.iomap_file_buffered_write
      0.64            +0.1        0.70 ±  3%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.47 ±  2%      +0.1        0.54 ±  2%  perf-profile.children.cycles-pp.enqueue_entity
      0.60 ±  2%      +0.1        0.67 ±  3%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.14 ±  7%      +0.1        0.23 ±  3%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.62            +0.1        0.71 ±  3%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.89 ±  7%      +0.1        0.99 ±  4%  perf-profile.children.cycles-pp.xlog_cil_insert_items
      1.22 ±  3%      +0.1        1.34 ±  5%  perf-profile.children.cycles-pp.xlog_cil_commit
      1.30 ±  3%      +0.1        1.42 ±  4%  perf-profile.children.cycles-pp.__xfs_trans_commit
      3.34 ±  3%      +0.4        3.72 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock
      0.37 ±  2%      +0.4        0.78 ±  3%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.41            +0.5        0.88 ±  2%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
     14.75 ±  2%      +0.9       15.61 ±  2%  perf-profile.children.cycles-pp.file_write_and_wait_range
      5.52 ±  2%      -1.2        4.32 ±  3%  perf-profile.self.cycles-pp.osq_lock
      0.11 ±  6%      +0.0        0.13 ±  6%  perf-profile.self.cycles-pp.enqueue_entity
      0.17 ±  4%      +0.0        0.19 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.05 ±  7%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.06 ±  9%      +0.0        0.08 ±  4%  perf-profile.self.cycles-pp.llist_reverse_order
      0.01 ±223%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.___perf_sw_event
      0.08 ±  9%      +0.1        0.13 ±  4%  perf-profile.self.cycles-pp.llist_add_batch
      0.66 ±  2%      +0.1        0.72 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue



***************************************************************************************************
lkp-csl-2sp9: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
  gcc-11/performance/pipe/8/x86_64-rhel-8.3/process/50%/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp9/hackbench/0x500320a

commit: 
  28156108fe ("sched: Fix the check of nr_running at queue wakelist")
  f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    317933            +7.2%     340823 ±  2%  hackbench.throughput
    281.33            -5.6%     265.66        hackbench.time.elapsed_time
    281.33            -5.6%     265.66        hackbench.time.elapsed_time.max
 9.332e+08 ±  2%     -10.1%  8.387e+08 ±  2%  hackbench.time.involuntary_context_switches
     22332            -6.2%      20950        hackbench.time.system_time
      1668            -3.3%       1613        hackbench.time.user_time
 2.484e+09            -7.0%  2.311e+09        hackbench.time.voluntary_context_switches
      0.99 ±  2%      +0.3        1.33        mpstat.cpu.all.irq%
 4.607e+08 ±  3%     -14.8%  3.927e+08        cpuidle..time
  59750049           -14.4%   51160896        cpuidle..usage
    107526 ± 46%     -44.8%      59395 ± 13%  numa-meminfo.node0.Mapped
    886278 ± 86%     -81.7%     162163 ± 69%  numa-meminfo.node1.Mapped
     26816 ± 46%     -44.4%      14905 ± 14%  numa-vmstat.node0.nr_mapped
    220693 ± 86%     -81.7%      40319 ± 67%  numa-vmstat.node1.nr_mapped
      1851 ±  4%     +30.6%       2418 ±  3%  sched_debug.cpu.avg_idle.min
     86.08 ±  9%     +29.5%     111.46 ± 11%  sched_debug.cpu.clock_task.stddev
  20680126 ±  3%     -49.3%   10478464 ±  2%  turbostat.C1
      0.43 ± 22%      -0.2        0.22 ±  3%  turbostat.C1%
      0.38 ±  2%      -0.0        0.35 ±  3%  turbostat.POLL%
   4953449 ± 39%     -34.7%    3235753 ±  7%  vmstat.memory.cache
  12228000            -2.2%   11965069        vmstat.system.cs
   1391657            +4.2%    1450409        vmstat.system.in
    359385 ±  9%     -10.0%     323464 ±  2%  meminfo.AnonPages
   4834275 ± 40%     -35.4%    3122465 ±  8%  meminfo.Cached
   3260439 ± 61%     -53.9%    1501744 ± 17%  meminfo.Committed_AS
   2453612 ± 81%     -70.9%     713687 ± 36%  meminfo.Inactive
   2453432 ± 81%     -70.9%     713334 ± 36%  meminfo.Inactive(anon)
    993573 ± 79%     -77.7%     221100 ± 53%  meminfo.Mapped
   6637365 ± 31%     -27.4%    4821818 ±  5%  meminfo.Memused
   2183740 ± 90%     -78.4%     471571 ± 53%  meminfo.Shmem
   7821764 ± 17%     -21.1%    6170585 ±  6%  meminfo.max_used_kB
     89894 ±  9%     -10.0%      80918 ±  2%  proc-vmstat.nr_anon_pages
   1209129 ± 40%     -35.5%     779734 ±  8%  proc-vmstat.nr_file_pages
    613996 ± 81%     -71.0%     178154 ± 36%  proc-vmstat.nr_inactive_anon
    248662 ± 79%     -77.7%      55432 ± 52%  proc-vmstat.nr_mapped
    546495 ± 90%     -78.6%     117009 ± 53%  proc-vmstat.nr_shmem
    613996 ± 81%     -71.0%     178154 ± 36%  proc-vmstat.nr_zone_inactive_anon
    303380 ± 16%     -39.0%     185031 ± 21%  proc-vmstat.numa_hint_faults
    200464 ± 20%     -42.8%     114585 ± 22%  proc-vmstat.numa_hint_faults_local
  89782752            +3.6%   93042546        proc-vmstat.numa_hit
  89705243            +3.6%   92968283        proc-vmstat.numa_local
    672643 ± 37%     -45.7%     365098 ±  9%  proc-vmstat.numa_pte_updates
  89776545            +3.6%   93036044        proc-vmstat.pgalloc_normal
   1626079 ±  4%     -10.8%    1450258 ±  3%  proc-vmstat.pgfault
  89378790            +4.1%   93037516        proc-vmstat.pgfree
     15.75            -2.9%      15.30        perf-stat.i.MPKI
 2.723e+10            +2.2%  2.782e+10        perf-stat.i.branch-instructions
      1.15            -0.0        1.13        perf-stat.i.branch-miss-rate%
      2.15 ±  5%      +0.2        2.34 ±  4%  perf-stat.i.cache-miss-rate%
  42514112 ±  4%      +8.4%   46069933 ±  5%  perf-stat.i.cache-misses
  12338031            -2.2%   12068469        perf-stat.i.context-switches
      1.80            -1.9%       1.76        perf-stat.i.cpi
 3.993e+10            +2.1%  4.076e+10        perf-stat.i.dTLB-loads
 2.294e+10            +1.8%  2.336e+10        perf-stat.i.dTLB-stores
     66.43            +1.1       67.51        perf-stat.i.iTLB-load-miss-rate%
  76738049            -3.1%   74356040        perf-stat.i.iTLB-loads
 1.353e+11            +2.1%  1.382e+11        perf-stat.i.instructions
      0.56            +2.1%       0.57        perf-stat.i.ipc
      1047            +2.0%       1068        perf-stat.i.metric.M/sec
      5305 ±  3%      -5.9%       4990 ±  3%  perf-stat.i.minor-faults
   8761133 ±  3%      +9.8%    9619722 ±  4%  perf-stat.i.node-load-misses
   5459816 ±  2%      +7.2%    5852880 ±  4%  perf-stat.i.node-store-misses
      5307 ±  3%      -5.9%       4993 ±  3%  perf-stat.i.page-faults
     15.51            -3.0%      15.05        perf-stat.overall.MPKI
      1.13            -0.0        1.12        perf-stat.overall.branch-miss-rate%
      2.03 ±  4%      +0.2        2.22 ±  4%  perf-stat.overall.cache-miss-rate%
      1.78            -2.0%       1.75        perf-stat.overall.cpi
      0.12 ±  6%      -0.0        0.11 ±  2%  perf-stat.overall.dTLB-load-miss-rate%
     66.43            +1.1       67.51        perf-stat.overall.iTLB-load-miss-rate%
      0.56            +2.1%       0.57        perf-stat.overall.ipc
 2.714e+10            +2.1%  2.771e+10        perf-stat.ps.branch-instructions
  42404117 ±  4%      +8.3%   45914737 ±  5%  perf-stat.ps.cache-misses
  12297697            -2.2%   12031391        perf-stat.ps.context-switches
 3.978e+10            +2.0%   4.06e+10        perf-stat.ps.dTLB-loads
 2.286e+10            +1.8%  2.327e+10        perf-stat.ps.dTLB-stores
  76489212            -3.1%   74128795        perf-stat.ps.iTLB-loads
 1.349e+11            +2.0%  1.376e+11        perf-stat.ps.instructions
      5314 ±  3%      -6.6%       4961 ±  3%  perf-stat.ps.minor-faults
   8734611 ±  3%      +9.7%    9580673 ±  4%  perf-stat.ps.node-load-misses
   5441379 ±  2%      +7.1%    5827671 ±  4%  perf-stat.ps.node-store-misses
      5317 ±  3%      -6.6%       4964 ±  3%  perf-stat.ps.page-faults
 3.807e+13            -3.7%  3.667e+13        perf-stat.total.instructions
     13.75            +0.3       14.09        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.new_sync_read
     15.17 ±  3%      +0.5       15.70        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
     15.28 ±  3%      +0.5       15.82        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      5.21 ±  4%      +0.7        5.91 ±  5%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.exit_to_user_mode_loop
      5.42 ±  4%      +0.7        6.14 ±  5%  perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      6.62 ±  9%      +0.8        7.45 ±  6%  perf-profile.calltrace.cycles-pp.__get_user_nocheck_8.perf_callchain_user.get_perf_callchain.perf_callchain.perf_prepare_sample
      3.28 ± 16%      +1.0        4.24 ±  5%  perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task
      3.38 ± 16%      +1.0        4.34 ±  5%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up
      3.36 ± 16%      +1.0        4.33 ±  5%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu
      3.44 ± 16%      +1.0        4.44 ±  5%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up.autoremove_wake_function
      3.49 ± 16%      +1.0        4.49 ±  5%  perf-profile.calltrace.cycles-pp.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common
      4.06 ± 15%      +1.1        5.14 ±  2%  perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      9.21 ±  4%      +1.1       10.30        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      9.10 ±  4%      +1.1       10.19        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     57.10            +1.1       58.22        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
     56.86            +1.1       57.98        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      8.61 ±  4%      +1.1        9.74 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      8.67 ±  4%      +1.1        9.81 ±  3%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
     58.57            +1.2       59.74        perf-profile.calltrace.cycles-pp.__libc_write
      8.76 ±  4%      +1.2        9.99 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.65 ± 69%      -0.6        0.05 ± 77%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
      0.65 ± 69%      -0.6        0.05 ± 77%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.65 ± 69%      -0.6        0.05 ± 77%  perf-profile.children.cycles-pp.do_idle
      0.64 ± 69%      -0.6        0.05 ± 77%  perf-profile.children.cycles-pp.start_secondary
      1.24 ±  8%      -0.2        1.04 ±  4%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.28 ± 50%      -0.2        0.09 ± 27%  perf-profile.children.cycles-pp.__mutex_lock
      0.23 ± 41%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.23 ± 41%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.perf_mmap__push
      0.23 ± 41%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.record__pushfn
      0.23 ± 40%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.23 ± 40%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.__generic_file_write_iter
      0.23 ± 40%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.generic_perform_write
      0.53 ± 11%      -0.1        0.42 ±  5%  perf-profile.children.cycles-pp.copyin
      0.24 ± 37%      -0.1        0.13 ± 28%  perf-profile.children.cycles-pp.__mutex_unlock_slowpath
      0.16 ± 55%      -0.1        0.06 ± 48%  perf-profile.children.cycles-pp.wake_up_q
     15.64            +0.4       16.08        perf-profile.children.cycles-pp.enqueue_task_fair
     15.75            +0.4       16.20        perf-profile.children.cycles-pp.ttwu_do_activate
      3.51 ± 16%      +1.0        4.50 ±  5%  perf-profile.children.cycles-pp.perf_trace_sched_migrate_task
     59.07            +1.0       60.08        perf-profile.children.cycles-pp.__libc_write
      4.08 ± 15%      +1.1        5.15 ±  2%  perf-profile.children.cycles-pp.set_task_cpu
     39.22            +1.1       40.33        perf-profile.children.cycles-pp.schedule
      9.11 ±  5%      +1.2       10.27        perf-profile.children.cycles-pp.exit_to_user_mode_loop
     10.37 ±  5%      +1.2       11.60        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     10.18 ±  5%      +1.2       11.41        perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.16 ±  3%      -0.0        0.14 ±  4%  perf-profile.self.cycles-pp.finish_task_switch
      0.06 ±  6%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.21 ±  8%      +0.0        0.25 ±  6%  perf-profile.self.cycles-pp.set_next_buddy





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://01.org/lkp



View attachment "config-5.19.0-rc2-00010-gf3dd3f674555" of type "text/plain" (163940 bytes)

View attachment "job-script" of type "text/plain" (8244 bytes)

View attachment "job.yaml" of type "text/plain" (5672 bytes)

View attachment "reproduce" of type "text/plain" (254 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ