lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Sun, 31 Jul 2022 16:22:53 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Shakeel Butt <shakeelb@...gle.com>
CC:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Johannes Weiner <hannes@...xchg.org>,
        Michal Hocko <mhocko@...nel.org>,
        Michal Koutný <mkoutny@...e.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        LKML <linux-kernel@...r.kernel.org>, <cgroups@...r.kernel.org>,
        <linux-mm@...ck.org>, <lkp@...ts.01.org>, <lkp@...el.com>,
        <ying.huang@...el.com>, <feng.tang@...el.com>,
        <zhengjun.xing@...ux.intel.com>, <fengwei.yin@...el.com>
Subject: [memcg]  fd25a9e0e2:  fio.write_iops 47.8% improvement



Greeting,

FYI, we noticed a 47.8% improvement of fio.write_iops due to commit:


commit: fd25a9e0e23b995fd0ba5e2f00a1099452cbc3cf ("memcg: unify memcg stat flushing")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: fio-basic
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	disk: 2pmem
	fs: ext4
	runtime: 200s
	nr_task: 50%
	time_based: tb
	rw: randrw
	bs: 4k
	ioengine: sync
	test_size: 200G
	cpufreq_governor: performance
	ucode: 0x500320a

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio

In addition to that, the commit also has significant impact on the following tests:

+------------------+--------------------------------------------------------------------------------+
| testcase: change | fio-basic: fio.write_iops 23.1% improvement                                    |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory |
| test parameters  | bs=4k                                                                          |
|                  | cpufreq_governor=performance                                                   |
|                  | disk=2pmem                                                                     |
|                  | fs=xfs                                                                         |
|                  | ioengine=mmap                                                                  |
|                  | nr_task=50%                                                                    |
|                  | runtime=200s                                                                   |
|                  | rw=rw                                                                          |
|                  | test_size=200G                                                                 |
|                  | time_based=tb                                                                  |
|                  | ucode=0x500320a                                                                |
+------------------+--------------------------------------------------------------------------------+
| testcase: change | fio-basic: fio.write_iops 14.0% improvement                                    |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory |
| test parameters  | bs=2M                                                                          |
|                  | cpufreq_governor=performance                                                   |
|                  | disk=2pmem                                                                     |
|                  | fs=ext4                                                                        |
|                  | ioengine=libaio                                                                |
|                  | nr_task=50%                                                                    |
|                  | runtime=200s                                                                   |
|                  | rw=rw                                                                          |
|                  | test_size=200G                                                                 |
|                  | time_based=tb                                                                  |
|                  | ucode=0x500320a                                                                |
+------------------+--------------------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  4k/gcc-11/performance/2pmem/ext4/sync/x86_64-rhel-8.3/50%/debian-11.1-x86_64-20220510.cgz/200s/randrw/lkp-csl-2sp7/200G/fio-basic/tb/0x500320a

commit: 
  11192d9c12 ("memcg: flush stats only if updated")
  fd25a9e0e2 ("memcg: unify memcg stat flushing")

11192d9c124d58d6 fd25a9e0e23b995fd0ba5e2f00a 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.53 ± 27%      -0.5        0.01        fio.latency_1000us%
      0.39 ± 87%      +2.5        2.85 ±  7%  fio.latency_100us%
     21.94 ±  4%      +3.3       25.24 ±  7%  fio.latency_10us%
      0.15 ± 66%      +0.9        1.04 ± 14%  fio.latency_250us%
      1.01 ± 19%      -1.0        0.01        fio.latency_2ms%
     72.48            -6.0       66.44 ±  3%  fio.latency_4us%
      0.05 ± 87%      +0.2        0.24 ± 17%  fio.latency_500us%
      0.94 ± 16%      +1.0        1.94 ± 15%  fio.latency_50us%
      0.05 ± 32%      -0.0        0.01 ±  5%  fio.latency_750us%
      3556 ±  2%     +47.8%       5257 ±  2%  fio.read_bw_MBps
      5592           +45.9%       8156 ±  8%  fio.read_clat_90%_us
      8960          +391.4%      44032 ±  9%  fio.read_clat_95%_us
     32560 ± 49%    +280.0%     123712 ± 12%  fio.read_clat_99%_us
      4827 ±  7%     +84.1%       8887 ±  3%  fio.read_clat_mean_us
    910504 ±  2%     +47.8%    1345904 ±  2%  fio.read_iops
  1.47e+09 ±  2%     +48.4%  2.181e+09 ±  2%  fio.time.file_system_inputs
 1.462e+09 ±  2%     +49.0%  2.178e+09 ±  2%  fio.time.file_system_outputs
     38246 ±  4%     +37.3%      52515        fio.time.involuntary_context_switches
      4744            -4.0%       4553        fio.time.percent_of_cpu_this_job_got
      9279            -5.0%       8818        fio.time.system_time
    263.86 ±  3%     +57.3%     415.07 ± 14%  fio.time.user_time
   1465507 ± 23%    +231.0%    4850690 ± 19%  fio.time.voluntary_context_switches
 3.654e+08 ±  2%     +49.0%  5.446e+08 ±  2%  fio.workload
      3556 ±  2%     +47.8%       5257 ±  2%  fio.write_bw_MBps
      6760           +17.9%       7972 ±  6%  fio.write_clat_90%_us
     17440 ± 23%    +116.5%      37760 ± 13%  fio.write_clat_95%_us
   1068032 ±  2%     -87.5%     133952 ±  8%  fio.write_clat_99%_us
     37376 ±  5%     -72.6%      10227 ±  2%  fio.write_clat_mean_us
    910464 ±  2%     +47.8%    1345940 ±  2%  fio.write_iops
    219862 ±  6%     +13.7%     249928 ±  6%  numa-meminfo.node1.SUnreclaim
      0.02 ± 29%   +6915.7%       1.53 ± 28%  iostat.cpu.iowait
     48.50            -4.7%      46.24        iostat.cpu.system
      1.36 ±  3%     +55.6%       2.12 ± 14%  iostat.cpu.user
   7243408 ±  2%     +14.3%    8281985 ±  3%  meminfo.Active
   7196323 ±  2%     +14.4%    8235503 ±  3%  meminfo.Active(file)
 2.035e+08           -14.4%  1.741e+08 ±  3%  meminfo.max_used_kB
      0.02 ± 30%      +1.5        1.55 ± 28%  mpstat.cpu.all.iowait%
      0.61            +0.1        0.75 ±  3%  mpstat.cpu.all.irq%
      1.37 ±  3%      +0.8        2.14 ± 14%  mpstat.cpu.all.usr%
 1.175e+08 ±  9%     +38.4%  1.626e+08 ±  4%  numa-numastat.node0.local_node
 1.174e+08 ±  9%     +38.0%  1.619e+08 ±  4%  numa-numastat.node0.numa_hit
 1.858e+08 ±  4%     +67.9%  3.121e+08 ±  3%  numa-numastat.node1.local_node
 1.856e+08 ±  4%     +67.5%  3.109e+08 ±  3%  numa-numastat.node1.numa_hit
   3583655 ±  2%     +47.3%    5278691        vmstat.io.bi
   3450488 ±  2%     +50.6%    5197844        vmstat.io.bo
      0.00       +1.4e+102%       1.38 ± 35%  vmstat.procs.b
     31151 ± 21%    +211.7%      97094 ± 21%  vmstat.system.cs
    424562 ± 13%     +20.4%     511014 ± 10%  sched_debug.cfs_rq:/.load.avg
    362.49 ± 13%     +17.1%     424.53 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     21255 ± 25%    +266.5%      77891 ± 28%  sched_debug.cpu.nr_switches.avg
      1307 ±  8%     +50.2%       1964 ± 22%  sched_debug.cpu.nr_switches.min
     39602 ± 26%    +156.4%     101541 ± 25%  sched_debug.cpu.nr_switches.stddev
      0.00 ±173%      +0.2        0.18 ±198%  turbostat.C1%
      0.08           +32.8%       0.11 ±  4%  turbostat.IPC
   2758150 ± 23%    +235.1%    9243143 ± 21%  turbostat.POLL
      0.04 ± 19%      +0.1        0.19 ± 30%  turbostat.POLL%
    266.30            +3.2%     274.72        turbostat.PkgWatt
     47.33            +9.3%      51.73        turbostat.RAMWatt
  41422157 ± 10%     +50.3%   62254549 ±  7%  numa-vmstat.node0.nr_dirtied
  39572106 ± 10%     +54.6%   61195242 ±  7%  numa-vmstat.node0.nr_written
 1.174e+08 ±  9%     +38.0%  1.619e+08 ±  4%  numa-vmstat.node0.numa_hit
 1.175e+08 ±  9%     +38.4%  1.626e+08 ±  4%  numa-vmstat.node0.numa_local
 1.413e+08           +48.6%    2.1e+08        numa-vmstat.node1.nr_dirtied
     54958 ±  6%     +13.7%      62481 ±  6%  numa-vmstat.node1.nr_slab_unreclaimable
 1.369e+08           +51.3%  2.072e+08        numa-vmstat.node1.nr_written
 1.856e+08 ±  4%     +67.5%  3.109e+08 ±  3%  numa-vmstat.node1.numa_hit
 1.858e+08 ±  4%     +67.9%  3.121e+08 ±  3%  numa-vmstat.node1.numa_local
   1798939 ±  2%     +14.4%    2058578 ±  3%  proc-vmstat.nr_active_file
 1.827e+08 ±  2%     +49.0%  2.723e+08 ±  2%  proc-vmstat.nr_dirtied
  26632755            +4.2%   27739758        proc-vmstat.nr_file_pages
  49856263            -2.3%   48708764        proc-vmstat.nr_free_pages
  24130513            +3.5%   24977194        proc-vmstat.nr_inactive_file
    526765            +5.2%     554092        proc-vmstat.nr_slab_reclaimable
    106917 ±  3%      +5.8%     113170 ±  3%  proc-vmstat.nr_slab_unreclaimable
 1.764e+08 ±  2%     +52.1%  2.683e+08        proc-vmstat.nr_written
   1798912 ±  2%     +14.4%    2058707 ±  3%  proc-vmstat.nr_zone_active_file
  24130862            +3.5%   24977575        proc-vmstat.nr_zone_inactive_file
 3.031e+08 ±  2%     +56.0%  4.729e+08 ±  3%  proc-vmstat.numa_hit
 3.033e+08 ±  2%     +56.5%  4.747e+08 ±  3%  proc-vmstat.numa_local
  35134068 ±  3%     +49.1%   52378797 ±  2%  proc-vmstat.pgactivate
 3.654e+08 ±  2%     +48.7%  5.434e+08 ±  2%  proc-vmstat.pgalloc_normal
 3.493e+08 ±  2%     +54.5%  5.397e+08 ±  2%  proc-vmstat.pgfree
 7.351e+08 ±  2%     +48.4%  1.091e+09 ±  2%  proc-vmstat.pgpgin
 7.057e+08 ±  2%     +52.1%  1.073e+09        proc-vmstat.pgpgout
    245935 ± 27%     -60.2%      97869 ± 84%  proc-vmstat.workingset_refault_file
     10.03           +10.4%      11.08        perf-stat.i.MPKI
 8.051e+09           +24.5%  1.003e+10        perf-stat.i.branch-instructions
      0.51            +0.0        0.54        perf-stat.i.branch-miss-rate%
  39726140 ±  2%     +32.4%   52583047 ±  2%  perf-stat.i.branch-misses
  2.78e+08 ±  3%     +41.0%  3.918e+08        perf-stat.i.cache-misses
 3.942e+08 ±  2%     +42.7%  5.626e+08 ±  2%  perf-stat.i.cache-references
     31675 ± 21%    +212.7%      99052 ± 21%  perf-stat.i.context-switches
      3.60           -24.6%       2.72        perf-stat.i.cpi
 1.389e+11            -3.7%  1.338e+11        perf-stat.i.cpu-cycles
    121.30            +5.2%     127.62        perf-stat.i.cpu-migrations
    545.94 ±  2%     -30.4%     379.99 ±  2%  perf-stat.i.cycles-between-cache-misses
   8673578 ±  7%     +40.2%   12162840 ±  9%  perf-stat.i.dTLB-load-misses
 1.042e+10           +29.4%  1.349e+10        perf-stat.i.dTLB-loads
   1200725 ±  8%     +53.6%    1844381 ±  5%  perf-stat.i.dTLB-store-misses
 4.745e+09 ±  2%     +46.2%  6.937e+09        perf-stat.i.dTLB-stores
     85.85            +3.4       89.29 ±  2%  perf-stat.i.iTLB-load-miss-rate%
  16340901 ±  3%     +35.2%   22091169 ±  5%  perf-stat.i.iTLB-load-misses
 3.904e+10           +28.5%  5.016e+10        perf-stat.i.instructions
      2813 ±  3%      -9.2%       2554 ±  5%  perf-stat.i.instructions-per-iTLB-miss
      0.28           +36.0%       0.38 ±  2%  perf-stat.i.ipc
      1.45            -3.7%       1.39        perf-stat.i.metric.GHz
      1082 ±  2%     +34.2%       1452        perf-stat.i.metric.K/sec
    245.99           +31.3%     323.05        perf-stat.i.metric.M/sec
     52.07            -6.5       45.60 ±  3%  perf-stat.i.node-load-miss-rate%
  29492421 ±  2%     +18.4%   34909755 ±  2%  perf-stat.i.node-load-misses
  27411662 ±  3%     +53.6%   42106608 ±  4%  perf-stat.i.node-loads
     45.38 ±  2%     -11.0       34.37 ±  4%  perf-stat.i.node-store-miss-rate%
  23596478 ±  3%     +63.2%   38509129 ±  3%  perf-stat.i.node-stores
     10.09           +11.0%      11.20        perf-stat.overall.MPKI
      0.49            +0.0        0.52        perf-stat.overall.branch-miss-rate%
      3.56           -25.0%       2.67        perf-stat.overall.cpi
    500.17 ±  2%     -31.6%     342.14 ±  2%  perf-stat.overall.cycles-between-cache-misses
     87.02            +3.2       90.23 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      0.28           +33.3%       0.37        perf-stat.overall.ipc
     51.82            -6.5       45.32 ±  3%  perf-stat.overall.node-load-miss-rate%
     44.95 ±  2%     -11.5       33.46 ±  3%  perf-stat.overall.node-store-miss-rate%
     21580           -13.1%      18743        perf-stat.overall.path-length
 8.012e+09           +24.5%  9.974e+09        perf-stat.ps.branch-instructions
  39519404 ±  2%     +32.2%   52256061 ±  2%  perf-stat.ps.branch-misses
 2.766e+08 ±  3%     +40.8%  3.894e+08        perf-stat.ps.cache-misses
 3.922e+08 ±  2%     +42.5%  5.591e+08        perf-stat.ps.cache-references
     31343 ± 21%    +212.5%      97946 ± 21%  perf-stat.ps.context-switches
 1.382e+11            -3.6%  1.332e+11        perf-stat.ps.cpu-cycles
    120.76            +5.1%     126.96        perf-stat.ps.cpu-migrations
   8632327 ±  7%     +40.1%   12090064 ±  9%  perf-stat.ps.dTLB-load-misses
 1.037e+10           +29.4%  1.342e+10        perf-stat.ps.dTLB-loads
   1195744 ±  8%     +53.4%    1834079 ±  5%  perf-stat.ps.dTLB-store-misses
 4.721e+09 ±  2%     +46.0%  6.895e+09        perf-stat.ps.dTLB-stores
  16269398 ±  3%     +34.9%   21954037 ±  5%  perf-stat.ps.iTLB-load-misses
 3.885e+10           +28.4%  4.989e+10        perf-stat.ps.instructions
  29347557 ±  2%     +18.3%   34718132 ±  2%  perf-stat.ps.node-load-misses
  27282181 ±  2%     +53.7%   41931623 ±  4%  perf-stat.ps.node-loads
  23505686 ±  3%     +63.1%   38330434 ±  3%  perf-stat.ps.node-stores
 7.885e+12           +29.4%  1.021e+13        perf-stat.total.instructions
     41.86 ± 30%     -41.9        0.00        perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write
     41.98 ± 30%     -41.3        0.69 ± 11%  perf-profile.calltrace.cycles-pp.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write.ext4_buffered_write_iter
     42.00 ± 30%     -41.3        0.70 ± 11%  perf-profile.calltrace.cycles-pp.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write.ext4_buffered_write_iter.new_sync_write
     42.11 ± 30%     -41.2        0.93 ± 12%  perf-profile.calltrace.cycles-pp.balance_dirty_pages_ratelimited.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write
     40.46 ± 31%     -40.5        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited
     40.46 ± 31%     -40.5        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages
     48.70 ± 25%     -34.9       13.76 ± 23%  perf-profile.calltrace.cycles-pp.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write.ksys_write
     48.86 ± 25%     -34.9       13.93 ± 23%  perf-profile.calltrace.cycles-pp.ext4_buffered_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     48.88 ± 25%     -34.9       13.96 ± 23%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     48.96 ± 25%     -34.9       14.06 ± 23%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     48.98 ± 25%     -34.9       14.09 ± 23%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     49.02 ± 25%     -34.9       14.13 ± 23%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
     49.00 ± 25%     -34.9       14.12 ± 23%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     49.12 ± 25%     -34.9       14.27 ± 23%  perf-profile.calltrace.cycles-pp.__libc_write
      0.00            +0.7        0.68 ± 11%  perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited
      0.00            +0.7        0.68 ± 11%  perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.mem_cgroup_wb_stats.balance_dirty_pages
      0.00            +0.7        0.68 ± 11%  perf-profile.calltrace.cycles-pp.mem_cgroup_flush_stats.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write
      0.57 ± 42%      +0.8        1.35 ± 14%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin
      0.60 ± 42%      +0.8        1.38 ± 13%  perf-profile.calltrace.cycles-pp.__alloc_pages.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write
      0.39 ± 81%      +0.8        1.22 ± 17%  perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages.pagecache_get_page.grab_cache_page_write_begin
      0.18 ±173%      +0.9        1.10 ± 25%  perf-profile.calltrace.cycles-pp.rmqueue_bulk.rmqueue.get_page_from_freelist.__alloc_pages.page_cache_ra_unbounded
      0.39 ±143%      +0.9        1.32 ± 26%  perf-profile.calltrace.cycles-pp.try_to_free_buffers.invalidate_inode_page.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64
      0.18 ±173%      +1.0        1.15 ± 20%  perf-profile.calltrace.cycles-pp.rmqueue_bulk.rmqueue.get_page_from_freelist.__alloc_pages.pagecache_get_page
      0.19 ±173%      +1.0        1.16 ± 23%  perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages.page_cache_ra_unbounded.force_page_cache_ra
      0.26 ±133%      +1.0        1.26 ± 20%  perf-profile.calltrace.cycles-pp.__alloc_pages.page_cache_ra_unbounded.force_page_cache_ra.filemap_get_pages.filemap_read
      0.20 ±173%      +1.0        1.21 ± 21%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.page_cache_ra_unbounded.force_page_cache_ra.filemap_get_pages
      0.68 ± 84%      +1.1        1.74 ± 23%  perf-profile.calltrace.cycles-pp.invalidate_inode_page.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      0.40 ± 79%      +1.3        1.72 ± 23%  perf-profile.calltrace.cycles-pp.ext4_end_bio.pmem_submit_bio.__submit_bio.__submit_bio_noacct.ext4_bio_write_page
      0.12 ±264%      +1.5        1.57 ± 23%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.pagevec_lru_move_fn.mark_page_accessed
      0.12 ±264%      +1.5        1.58 ± 23%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.pagevec_lru_move_fn.mark_page_accessed.filemap_read
      0.12 ±264%      +1.5        1.58 ± 23%  perf-profile.calltrace.cycles-pp.lock_page_lruvec_irqsave.pagevec_lru_move_fn.mark_page_accessed.filemap_read.new_sync_read
      0.28 ±173%      +1.5        1.82 ± 34%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.rmqueue_bulk.rmqueue.get_page_from_freelist
      0.28 ±173%      +1.5        1.83 ± 34%  perf-profile.calltrace.cycles-pp._raw_spin_lock.rmqueue_bulk.rmqueue.get_page_from_freelist.__alloc_pages
      0.13 ±264%      +1.6        1.74 ± 22%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.mark_page_accessed.filemap_read.new_sync_read.vfs_read
      0.15 ±264%      +1.8        1.94 ± 22%  perf-profile.calltrace.cycles-pp.mark_page_accessed.filemap_read.new_sync_read.vfs_read.ksys_read
      0.35 ±101%      +1.9        2.23 ± 30%  perf-profile.calltrace.cycles-pp.__memcpy_flushcache.write_pmem.pmem_do_write.pmem_submit_bio.__submit_bio
      0.35 ±101%      +1.9        2.25 ± 30%  perf-profile.calltrace.cycles-pp.write_pmem.pmem_do_write.pmem_submit_bio.__submit_bio.__submit_bio_noacct
      0.36 ±101%      +1.9        2.26 ± 30%  perf-profile.calltrace.cycles-pp.pmem_do_write.pmem_submit_bio.__submit_bio.__submit_bio_noacct.ext4_bio_write_page
      1.24 ± 44%      +3.1        4.36 ± 22%  perf-profile.calltrace.cycles-pp.pmem_submit_bio.__submit_bio.__submit_bio_noacct.ext4_bio_write_page.mpage_submit_page
      1.28 ± 44%      +3.2        4.48 ± 22%  perf-profile.calltrace.cycles-pp.__submit_bio.__submit_bio_noacct.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs
      1.29 ± 44%      +3.2        4.48 ± 22%  perf-profile.calltrace.cycles-pp.__submit_bio_noacct.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
      1.70 ± 42%      +3.7        5.44 ± 21%  perf-profile.calltrace.cycles-pp.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages
      1.96 ± 43%      +3.9        5.89 ± 21%  perf-profile.calltrace.cycles-pp.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages
      0.46 ± 85%      +4.2        4.63 ± 26%  perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.filemap_fdatawrite_wbc
      1.38 ± 87%      +4.3        5.70 ± 37%  perf-profile.calltrace.cycles-pp.lru_cache_add.add_to_page_cache_lru.page_cache_ra_unbounded.force_page_cache_ra.filemap_get_pages
      1.35 ± 90%      +4.3        5.67 ± 37%  perf-profile.calltrace.cycles-pp.__pagevec_lru_add.lru_cache_add.add_to_page_cache_lru.page_cache_ra_unbounded.force_page_cache_ra
      2.08 ± 55%      +4.4        6.44 ± 35%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.page_cache_ra_unbounded.force_page_cache_ra.filemap_get_pages.filemap_read
      0.96 ±134%      +4.4        5.33 ± 38%  perf-profile.calltrace.cycles-pp.lock_page_lruvec_irqsave.__pagevec_lru_add.lru_cache_add.add_to_page_cache_lru.page_cache_ra_unbounded
      1.41 ± 84%      +4.4        5.86 ± 39%  perf-profile.calltrace.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin
      1.38 ± 86%      +4.4        5.83 ± 39%  perf-profile.calltrace.cycles-pp.__pagevec_lru_add.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin
      1.04 ±119%      +4.5        5.50 ± 40%  perf-profile.calltrace.cycles-pp.lock_page_lruvec_irqsave.__pagevec_lru_add.lru_cache_add.add_to_page_cache_lru.pagecache_get_page
      0.52 ± 85%      +4.5        4.99 ± 26%  perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
      0.52 ± 85%      +4.5        5.00 ± 26%  perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64
      0.52 ± 85%      +4.5        5.00 ± 26%  perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.generic_fadvise
      0.52 ± 85%      +4.5        5.00 ± 26%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64
      0.52 ± 85%      +4.5        5.00 ± 26%  perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      2.12 ± 53%      +4.5        6.61 ± 36%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write
      3.36 ± 34%      +5.4        8.80 ± 27%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter
      3.36 ± 33%      +5.4        8.81 ± 27%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write
      4.55 ± 23%      +5.5       10.06 ± 26%  perf-profile.calltrace.cycles-pp.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write
      4.72 ± 24%      +6.2       10.97 ± 24%  perf-profile.calltrace.cycles-pp.page_cache_ra_unbounded.force_page_cache_ra.filemap_get_pages.filemap_read.new_sync_read
      4.74 ± 24%      +6.3       10.99 ± 24%  perf-profile.calltrace.cycles-pp.force_page_cache_ra.filemap_get_pages.filemap_read.new_sync_read.vfs_read
      5.46 ± 20%      +6.6       12.11 ± 23%  perf-profile.calltrace.cycles-pp.filemap_get_pages.filemap_read.new_sync_read.vfs_read.ksys_read
      6.19 ± 22%      +8.3       14.46 ± 22%  perf-profile.calltrace.cycles-pp.filemap_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      6.23 ± 22%      +8.3       14.50 ± 22%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.30 ± 21%      +8.3       14.59 ± 22%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      6.32 ± 21%      +8.3       14.62 ± 22%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      6.34 ± 21%      +8.3       14.65 ± 22%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      6.35 ± 21%      +8.3       14.66 ± 22%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
      6.47 ± 21%      +8.3       14.82 ± 22%  perf-profile.calltrace.cycles-pp.__libc_read
      1.98 ±127%      +8.8       10.82 ± 39%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.__pagevec_lru_add.lru_cache_add.add_to_page_cache_lru
      1.88 ±136%      +8.9       10.76 ± 39%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.__pagevec_lru_add.lru_cache_add
     41.98 ± 30%     -41.3        0.69 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_wb_stats
     42.00 ± 30%     -41.3        0.71 ± 11%  perf-profile.children.cycles-pp.balance_dirty_pages
     42.11 ± 30%     -41.2        0.93 ± 12%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
     41.87 ± 30%     -41.2        0.70 ± 11%  perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe
     48.72 ± 25%     -34.9       13.79 ± 23%  perf-profile.children.cycles-pp.generic_perform_write
     48.86 ± 25%     -34.9       13.94 ± 23%  perf-profile.children.cycles-pp.ext4_buffered_write_iter
     48.90 ± 25%     -34.9       13.99 ± 23%  perf-profile.children.cycles-pp.new_sync_write
     48.97 ± 25%     -34.9       14.09 ± 23%  perf-profile.children.cycles-pp.vfs_write
     49.00 ± 25%     -34.9       14.12 ± 23%  perf-profile.children.cycles-pp.ksys_write
     49.16 ± 25%     -34.8       14.32 ± 23%  perf-profile.children.cycles-pp.__libc_write
     50.76 ± 10%     -11.1       39.64 ± 11%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     51.00 ± 10%      -9.9       41.11 ± 11%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.44 ± 13%      -0.7        0.72 ± 16%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      1.40 ±  8%      -0.7        0.70 ± 11%  perf-profile.children.cycles-pp.cgroup_rstat_flush_locked
      0.83 ± 12%      -0.3        0.56 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush
      0.42 ± 14%      -0.2        0.22 ± 21%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.44 ± 12%      -0.2        0.24 ± 19%  perf-profile.children.cycles-pp.__count_memcg_events
      0.02 ±129%      +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.__list_add_valid
      0.07 ± 17%      +0.1        0.13 ± 10%  perf-profile.children.cycles-pp.page_mapping
      0.01 ±174%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.uncharge_page
      0.04 ± 58%      +0.1        0.11 ± 16%  perf-profile.children.cycles-pp.unlock_page
      0.01 ±264%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.drop_buffers
      0.00            +0.1        0.08 ± 44%  perf-profile.children.cycles-pp.xas_init_marks
      0.08 ± 16%      +0.1        0.17 ± 27%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.06 ± 19%      +0.1        0.15 ± 22%  perf-profile.children.cycles-pp.workingset_activation
      0.04 ± 79%      +0.1        0.13 ± 29%  perf-profile.children.cycles-pp.xas_clear_mark
      0.01 ±174%      +0.1        0.11 ± 20%  perf-profile.children.cycles-pp.page_counter_cancel
      0.08 ± 44%      +0.1        0.18 ± 22%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.06 ± 58%      +0.1        0.17 ± 19%  perf-profile.children.cycles-pp.uncharge_batch
      0.01 ±174%      +0.1        0.12 ± 21%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.03 ±102%      +0.1        0.14 ± 23%  perf-profile.children.cycles-pp.workingset_age_nonresident
      0.00            +0.1        0.13 ± 18%  perf-profile.children.cycles-pp.mem_cgroup_wb_domain
      0.12 ± 11%      +0.1        0.25 ± 12%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.11 ± 38%      +0.1        0.24 ± 18%  perf-profile.children.cycles-pp.__mem_cgroup_uncharge_list
      0.15 ± 11%      +0.2        0.30 ± 13%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.15 ± 30%      +0.2        0.33 ± 19%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.15 ± 30%      +0.2        0.33 ± 19%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.06 ±141%      +0.2        0.26 ± 54%  perf-profile.children.cycles-pp.poll_idle
      0.23 ± 36%      +0.2        0.44 ± 16%  perf-profile.children.cycles-pp.memcg_slab_free_hook
      0.22 ± 33%      +0.2        0.43 ± 21%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      0.16 ± 30%      +0.2        0.37 ± 15%  perf-profile.children.cycles-pp.jbd2_journal_grab_journal_head
      0.16 ± 28%      +0.2        0.38 ± 15%  perf-profile.children.cycles-pp.jbd2_journal_try_to_free_buffers
      0.24 ± 42%      +0.2        0.48 ± 27%  perf-profile.children.cycles-pp.__free_one_page
      0.22 ± 38%      +0.2        0.46 ± 20%  perf-profile.children.cycles-pp.find_lock_entries
      0.22 ± 29%      +0.3        0.47 ± 20%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.24 ± 10%      +0.3        0.55 ± 25%  perf-profile.children.cycles-pp.xas_store
      0.24 ± 35%      +0.4        0.66 ± 23%  perf-profile.children.cycles-pp.ext4_put_io_end_defer
      0.46 ± 37%      +0.4        0.88 ± 23%  perf-profile.children.cycles-pp.__delete_from_page_cache
      0.37 ± 47%      +0.4        0.80 ± 29%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.64 ± 24%      +0.5        1.14 ± 14%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.42 ± 45%      +0.5        0.93 ± 28%  perf-profile.children.cycles-pp.free_unref_page_list
      0.37 ± 31%      +0.6        0.92 ± 20%  perf-profile.children.cycles-pp.test_clear_page_writeback
      0.52 ± 37%      +0.6        1.09 ± 23%  perf-profile.children.cycles-pp.__remove_mapping
      0.38 ± 31%      +0.6        1.01 ± 22%  perf-profile.children.cycles-pp.end_page_writeback
      0.57 ± 69%      +0.7        1.27 ± 26%  perf-profile.children.cycles-pp.free_buffer_head
      0.00            +0.7        0.70 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_flush_stats
      0.63 ± 62%      +0.7        1.38 ± 24%  perf-profile.children.cycles-pp.kmem_cache_free
      0.45 ± 31%      +0.8        1.20 ± 21%  perf-profile.children.cycles-pp.ext4_finish_bio
      0.61 ± 66%      +0.8        1.38 ± 25%  perf-profile.children.cycles-pp.try_to_free_buffers
      0.80 ± 57%      +0.9        1.75 ± 23%  perf-profile.children.cycles-pp.invalidate_inode_page
      1.16 ± 23%      +1.2        2.31 ± 24%  perf-profile.children.cycles-pp._raw_spin_lock
      0.70 ± 32%      +1.2        1.87 ± 21%  perf-profile.children.cycles-pp.ext4_end_bio
      0.91 ± 35%      +1.4        2.26 ± 22%  perf-profile.children.cycles-pp.rmqueue_bulk
      1.02 ± 31%      +1.4        2.40 ± 20%  perf-profile.children.cycles-pp.rmqueue
      1.12 ± 27%      +1.4        2.57 ± 17%  perf-profile.children.cycles-pp.get_page_from_freelist
      1.18 ± 25%      +1.5        2.66 ± 16%  perf-profile.children.cycles-pp.__alloc_pages
      0.30 ± 93%      +1.5        1.78 ± 22%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.40 ± 72%      +1.5        1.94 ± 22%  perf-profile.children.cycles-pp.mark_page_accessed
      0.70 ± 33%      +1.7        2.35 ± 25%  perf-profile.children.cycles-pp.__memcpy_flushcache
      0.70 ± 33%      +1.7        2.37 ± 24%  perf-profile.children.cycles-pp.write_pmem
      0.70 ± 33%      +1.7        2.38 ± 25%  perf-profile.children.cycles-pp.pmem_do_write
      1.92 ± 31%      +3.5        5.45 ± 21%  perf-profile.children.cycles-pp.ext4_bio_write_page
      2.14 ± 32%      +3.8        5.88 ± 21%  perf-profile.children.cycles-pp.mpage_submit_page
      2.98 ± 19%      +3.9        6.88 ± 20%  perf-profile.children.cycles-pp.pmem_submit_bio
      2.25 ± 32%      +4.0        6.25 ± 21%  perf-profile.children.cycles-pp.mpage_process_page_bufs
      3.15 ± 19%      +4.0        7.15 ± 20%  perf-profile.children.cycles-pp.__submit_bio
      3.16 ± 19%      +4.0        7.16 ± 20%  perf-profile.children.cycles-pp.__submit_bio_noacct
      2.48 ± 32%      +4.3        6.80 ± 20%  perf-profile.children.cycles-pp.do_writepages
      2.48 ± 32%      +4.3        6.80 ± 20%  perf-profile.children.cycles-pp.ext4_writepages
      2.48 ± 32%      +4.3        6.80 ± 20%  perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
      0.65 ± 50%      +4.3        5.00 ± 26%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.65 ± 50%      +4.3        5.00 ± 26%  perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
      3.36 ± 34%      +5.4        8.80 ± 27%  perf-profile.children.cycles-pp.pagecache_get_page
      3.37 ± 33%      +5.4        8.81 ± 27%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      4.55 ± 23%      +5.5       10.06 ± 26%  perf-profile.children.cycles-pp.ext4_da_write_begin
      4.74 ± 24%      +6.3       10.99 ± 24%  perf-profile.children.cycles-pp.force_page_cache_ra
      4.73 ± 24%      +6.3       11.07 ± 24%  perf-profile.children.cycles-pp.page_cache_ra_unbounded
      5.46 ± 20%      +6.7       12.12 ± 23%  perf-profile.children.cycles-pp.filemap_get_pages
      6.20 ± 22%      +8.3       14.46 ± 22%  perf-profile.children.cycles-pp.filemap_read
      6.24 ± 21%      +8.3       14.51 ± 22%  perf-profile.children.cycles-pp.new_sync_read
      6.34 ± 21%      +8.3       14.63 ± 22%  perf-profile.children.cycles-pp.vfs_read
      6.36 ± 21%      +8.3       14.66 ± 22%  perf-profile.children.cycles-pp.ksys_read
      6.49 ± 21%      +8.4       14.84 ± 22%  perf-profile.children.cycles-pp.__libc_read
      2.80 ± 86%      +8.8       11.62 ± 38%  perf-profile.children.cycles-pp.lru_cache_add
      2.74 ± 88%      +8.8       11.57 ± 38%  perf-profile.children.cycles-pp.__pagevec_lru_add
      4.20 ± 54%      +8.9       13.13 ± 36%  perf-profile.children.cycles-pp.add_to_page_cache_lru
     51.00 ± 10%      -9.9       41.11 ± 11%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.12 ± 15%      -0.6        0.55 ± 17%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.82 ± 12%      -0.3        0.55 ± 12%  perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush
      0.51 ± 17%      -0.2        0.30 ± 15%  perf-profile.self.cycles-pp._raw_spin_lock
      0.19 ± 14%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.cgroup_rstat_flush_locked
      0.06 ± 16%      +0.1        0.11 ± 14%  perf-profile.self.cycles-pp.xas_store
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.submit_bio_checks
      0.01 ±174%      +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.__slab_free
      0.06 ± 15%      +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.page_mapping
      0.04 ± 58%      +0.1        0.10 ± 16%  perf-profile.self.cycles-pp.unlock_page
      0.02 ±129%      +0.1        0.08 ± 21%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.01 ±173%      +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.uncharge_page
      0.01 ±174%      +0.1        0.08 ± 21%  perf-profile.self.cycles-pp.__remove_mapping
      0.01 ±264%      +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.drop_buffers
      0.00            +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.__list_add_valid
      0.01 ±173%      +0.1        0.09 ± 20%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.01 ±264%      +0.1        0.09 ± 23%  perf-profile.self.cycles-pp.mpage_prepare_extent_to_map
      0.04 ± 79%      +0.1        0.12 ± 30%  perf-profile.self.cycles-pp.xas_clear_mark
      0.04 ± 79%      +0.1        0.13 ± 22%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.04 ± 79%      +0.1        0.13 ± 23%  perf-profile.self.cycles-pp.test_clear_page_writeback
      0.10 ± 24%      +0.1        0.20 ± 18%  perf-profile.self.cycles-pp.kmem_cache_free
      0.01 ±174%      +0.1        0.11 ± 21%  perf-profile.self.cycles-pp.page_counter_cancel
      0.06 ± 62%      +0.1        0.17 ± 17%  perf-profile.self.cycles-pp.ext4_bio_write_page
      0.03 ±102%      +0.1        0.14 ± 23%  perf-profile.self.cycles-pp.workingset_age_nonresident
      0.03 ±104%      +0.1        0.15 ± 40%  perf-profile.self.cycles-pp.___slab_alloc
      0.12 ± 12%      +0.1        0.24 ± 13%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.00            +0.1        0.13 ± 18%  perf-profile.self.cycles-pp.mem_cgroup_wb_domain
      0.12 ± 29%      +0.2        0.28 ± 19%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.06 ±141%      +0.2        0.25 ± 54%  perf-profile.self.cycles-pp.poll_idle
      0.16 ± 25%      +0.2        0.36 ± 14%  perf-profile.self.cycles-pp.memcg_slab_free_hook
      0.16 ± 30%      +0.2        0.37 ± 15%  perf-profile.self.cycles-pp.jbd2_journal_grab_journal_head
      0.18 ± 35%      +0.2        0.40 ± 22%  perf-profile.self.cycles-pp.__free_one_page
      0.20 ± 37%      +0.2        0.42 ± 20%  perf-profile.self.cycles-pp.find_lock_entries
      0.12 ± 35%      +0.2        0.36 ± 23%  perf-profile.self.cycles-pp.mpage_process_page_bufs
      0.24 ± 35%      +0.4        0.65 ± 23%  perf-profile.self.cycles-pp.ext4_put_io_end_defer
      0.40 ± 14%      +0.4        0.83 ± 18%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.64 ± 24%      +0.5        1.14 ± 14%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.69 ± 33%      +1.6        2.33 ± 25%  perf-profile.self.cycles-pp.__memcpy_flushcache


***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  4k/gcc-11/performance/2pmem/xfs/mmap/x86_64-rhel-8.3/50%/debian-11.1-x86_64-20220510.cgz/200s/rw/lkp-csl-2sp7/200G/fio-basic/tb/0x500320a

commit: 
  11192d9c12 ("memcg: flush stats only if updated")
  fd25a9e0e2 ("memcg: unify memcg stat flushing")

11192d9c124d58d6 fd25a9e0e23b995fd0ba5e2f00a 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.09 ± 23%      -0.1        0.01        fio.latency_1000us%
      0.14 ± 12%      -0.1        0.03 ±  5%  fio.latency_100us%
      2.90 ±  5%      +1.2        4.14 ±  9%  fio.latency_10us%
      0.39 ±  8%      +0.2        0.63 ± 10%  fio.latency_20us%
      0.29 ±  6%      -0.2        0.09 ±  4%  fio.latency_250us%
      0.02 ± 30%      -0.0        0.01        fio.latency_2ms%
     24.32 ±  2%      +9.1       33.40 ±  2%  fio.latency_2us%
     25.64 ±  2%      -6.1       19.51 ±  4%  fio.latency_4us%
      0.33 ± 30%      -0.3        0.03 ±  9%  fio.latency_500us%
      0.33 ±  9%      -0.2        0.14 ±  9%  fio.latency_50us%
      0.30 ± 18%      -0.3        0.01 ±  6%  fio.latency_750us%
      4684           +23.1%       5766 ±  3%  fio.read_bw_MBps
    982.67 ±  2%     +43.8%       1413 ±  6%  fio.read_clat_90%_us
      1533 ±  3%     +64.5%       2522        fio.read_clat_95%_us
      3637           +46.0%       5312 ±  4%  fio.read_clat_99%_us
    883.59           +29.1%       1140        fio.read_clat_mean_us
      6383          +103.3%      12981 ±  4%  fio.read_clat_stddev
   1199278           +23.1%    1476197 ±  3%  fio.read_iops
 1.927e+09           +23.1%  2.371e+09 ±  3%  fio.time.file_system_outputs
     10231 ±  4%     -51.0%       5013 ± 10%  fio.time.involuntary_context_switches
 2.413e+08           +23.0%  2.968e+08 ±  3%  fio.time.major_page_faults
  22094002 ±  5%     +27.1%   28086792        fio.time.maximum_resident_set_size
   7057976 ±  3%     +23.6%    8725928 ±  3%  fio.time.minor_page_faults
      1747 ±  4%     -51.8%     842.83 ±  5%  fio.time.percent_of_cpu_this_job_got
      3064 ±  4%     -66.2%       1036 ±  6%  fio.time.system_time
    467.32 ±  2%     +43.4%     669.93 ±  5%  fio.time.user_time
 4.818e+08           +23.1%  5.928e+08 ±  3%  fio.workload
      4685           +23.1%       5767 ±  3%  fio.write_bw_MBps
      6112 ±  3%      -9.2%       5546 ±  4%  fio.write_clat_95%_us
    444757 ± 21%     -97.1%      12800 ±  5%  fio.write_clat_99%_us
     37030           -19.7%      29718 ±  4%  fio.write_clat_mean_us
   1199468           +23.1%    1476390 ±  3%  fio.write_iops
 1.549e+10           +11.8%  1.732e+10        cpuidle..time
  32060523           +11.9%   35866344        cpuidle..usage
     49.50            +2.3%      50.64        iostat.cpu.idle
     30.40 ±  2%     +26.3%      38.40        iostat.cpu.iowait
     17.77 ±  4%     -56.1%       7.80 ±  4%  iostat.cpu.system
      2.33 ±  2%     +35.9%       3.17 ±  5%  iostat.cpu.user
     29.83 ±  2%     +26.8%      37.83        vmstat.cpu.wa
   4519957           +23.7%    5592929 ±  3%  vmstat.io.bo
     29.00 ±  4%     +30.5%      37.83        vmstat.procs.b
     19.00 ±  7%     -47.4%      10.00 ±  5%  vmstat.procs.r
     47466 ± 49%     -46.7%      25314 ± 12%  meminfo.Active
     45586 ± 51%     -48.6%      23440 ± 13%  meminfo.Active(anon)
    929526           +26.9%    1179873 ±  3%  meminfo.PageTables
     66821 ± 40%     -38.3%      41242 ±  8%  meminfo.Shmem
   1420637 ±  2%     +19.0%    1691263 ±  5%  meminfo.Writeback
     30.69 ±  2%      +8.1       38.77        mpstat.cpu.all.iowait%
      0.75 ±  3%      +0.1        0.87 ±  4%  mpstat.cpu.all.irq%
      0.05 ±  3%      +0.0        0.05        mpstat.cpu.all.soft%
     17.13 ±  4%     -10.2        6.93 ±  5%  mpstat.cpu.all.sys%
      2.35 ±  2%      +0.8        3.20 ±  5%  mpstat.cpu.all.usr%
    595.00 ±  3%     -42.6%     341.33 ±  4%  turbostat.Avg_MHz
     21.33 ±  3%      -9.1       12.27 ±  4%  turbostat.Busy%
     78.60           +11.5%      87.64        turbostat.CPU%c1
      0.13 ±  4%     +87.2%       0.24        turbostat.IPC
    209.63            -5.7%     197.72        turbostat.PkgWatt
     42.66            +8.1%      46.14        turbostat.RAMWatt
  11378737 ±  2%     +18.3%   13462900 ±  3%  numa-meminfo.node0.Dirty
    308669 ±  4%     +32.9%     410318 ±  6%  numa-meminfo.node0.Writeback
     44240 ± 53%     -48.0%      22983 ± 16%  numa-meminfo.node1.Active
     44240 ± 53%     -49.4%      22365 ± 14%  numa-meminfo.node1.Active(anon)
    374833 ±  5%      +7.8%     403918 ±  4%  numa-meminfo.node1.KReclaimable
    757587 ±  2%     +32.1%    1000984 ±  3%  numa-meminfo.node1.PageTables
    374833 ±  5%      +7.8%     403918 ±  4%  numa-meminfo.node1.SReclaimable
   1088573 ±  5%     +16.3%    1265938 ±  5%  numa-meminfo.node1.Writeback
  52202136 ±  4%     +22.7%   64049942 ±  3%  numa-vmstat.node0.nr_dirtied
   2844482 ±  2%     +18.4%    3367535 ±  3%  numa-vmstat.node0.nr_dirty
    165.83 ± 56%     -79.6%      33.83 ±199%  numa-vmstat.node0.nr_mlock
     78282 ±  3%     +30.9%     102434 ±  4%  numa-vmstat.node0.nr_writeback
  49430955 ±  6%     +27.2%   62887052 ±  3%  numa-vmstat.node0.nr_written
   2922973 ±  2%     +18.7%    3470209 ±  3%  numa-vmstat.node0.nr_zone_write_pending
     11061 ± 53%     -49.6%       5575 ± 15%  numa-vmstat.node1.nr_active_anon
 1.887e+08           +23.1%  2.324e+08 ±  4%  numa-vmstat.node1.nr_dirtied
     66.17 ±141%    +156.2%     169.50 ± 43%  numa-vmstat.node1.nr_mlock
    188820 ±  2%     +32.7%     250515 ±  3%  numa-vmstat.node1.nr_page_table_pages
     93715 ±  5%      +7.7%     100970 ±  4%  numa-vmstat.node1.nr_slab_reclaimable
    271839 ±  5%     +16.6%     316969 ±  6%  numa-vmstat.node1.nr_writeback
 1.809e+08 ±  2%     +23.9%  2.241e+08 ±  4%  numa-vmstat.node1.nr_written
     11061 ± 53%     -49.6%       5575 ± 15%  numa-vmstat.node1.nr_zone_active_anon
      0.27 ± 27%     -49.2%       0.14 ± 10%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.40 ±  5%     -14.2%       0.34 ±  5%  sched_debug.cfs_rq:/.h_nr_running.stddev
    239463 ± 32%     -57.7%     101381 ± 10%  sched_debug.cfs_rq:/.load.avg
    335444 ±  6%     -20.0%     268223 ±  5%  sched_debug.cfs_rq:/.load.stddev
      0.27 ± 27%     -49.3%       0.14 ± 10%  sched_debug.cfs_rq:/.nr_running.avg
      0.40 ±  5%     -14.3%       0.34 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
    317.95 ± 24%     -48.2%     164.84 ±  5%  sched_debug.cfs_rq:/.runnable_avg.avg
    315.63 ±  7%     -29.5%     222.58 ±  3%  sched_debug.cfs_rq:/.runnable_avg.stddev
    317.60 ± 24%     -48.3%     164.20 ±  6%  sched_debug.cfs_rq:/.util_avg.avg
    315.18 ±  7%     -29.5%     222.09 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    136.90 ± 40%     -76.8%      31.76 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    221.23 ± 12%     -41.1%     130.35 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      1060 ± 27%     -51.7%     512.30 ±  8%  sched_debug.cpu.curr->pid.avg
      1940 ±  9%     -18.8%       1575 ±  4%  sched_debug.cpu.curr->pid.stddev
      0.21 ± 24%     -46.7%       0.11 ±  8%  sched_debug.cpu.nr_running.avg
      0.38 ±  8%     -17.0%       0.31 ±  4%  sched_debug.cpu.nr_running.stddev
     11396 ± 51%     -48.6%       5859 ± 13%  proc-vmstat.nr_active_anon
 2.409e+08           +23.1%  2.964e+08 ±  3%  proc-vmstat.nr_dirtied
  12452137            +2.0%   12702886        proc-vmstat.nr_dirty
  47083061            +2.5%   48263837        proc-vmstat.nr_file_pages
  29628750            -4.2%   28385431        proc-vmstat.nr_free_pages
  46383516            +2.6%   47570490        proc-vmstat.nr_inactive_file
  45891563            +2.4%   47011143        proc-vmstat.nr_mapped
    232389           +26.9%     294974 ±  3%  proc-vmstat.nr_page_table_pages
     16705 ± 40%     -38.3%      10310 ±  8%  proc-vmstat.nr_shmem
    132773            +2.0%     135451        proc-vmstat.nr_slab_reclaimable
    355004 ±  2%     +19.0%     422599 ±  5%  proc-vmstat.nr_writeback
 2.305e+08           +24.5%   2.87e+08 ±  4%  proc-vmstat.nr_written
     11396 ± 51%     -48.6%       5859 ± 13%  proc-vmstat.nr_zone_active_anon
  46383514            +2.6%   47570490        proc-vmstat.nr_zone_inactive_file
  12808434            +2.5%   13126679        proc-vmstat.nr_zone_write_pending
 1.265e+08 ±  9%     -28.2%   90860448 ±  4%  proc-vmstat.numa_pte_updates
 4.903e+08           +23.0%   6.03e+08 ±  3%  proc-vmstat.pgfault
   1127515            +6.8%    1204554        proc-vmstat.pgfree
 9.227e+08           +24.5%  1.149e+09 ±  4%  proc-vmstat.pgpgout
 1.112e+08           +23.7%  1.375e+08 ±  3%  proc-vmstat.pgreuse
     13.87            -1.6%      13.66        perf-stat.i.MPKI
      0.35            +0.0        0.37 ±  2%  perf-stat.i.branch-miss-rate%
  17443526            +9.6%   19118078 ±  5%  perf-stat.i.branch-misses
     76.66            +4.3       80.98        perf-stat.i.cache-miss-rate%
 2.696e+08           +12.4%  3.029e+08 ±  3%  perf-stat.i.cache-misses
 3.554e+08 ±  2%      +6.8%  3.797e+08 ±  3%  perf-stat.i.cache-references
      1.94 ±  2%     -43.1%       1.10        perf-stat.i.cpi
 5.592e+10 ±  3%     -43.3%  3.171e+10 ±  3%  perf-stat.i.cpu-cycles
    212.99 ±  3%     -44.8%     117.47        perf-stat.i.cycles-between-cache-misses
      0.01 ± 14%      +0.0        0.02 ± 14%  perf-stat.i.dTLB-load-miss-rate%
    861200 ± 16%     +58.0%    1360994 ± 14%  perf-stat.i.dTLB-load-misses
  7.16e+09            +7.8%  7.719e+09 ±  3%  perf-stat.i.dTLB-loads
   4970815           +21.8%    6053776 ±  3%  perf-stat.i.dTLB-store-misses
 3.702e+09           +20.6%  4.466e+09 ±  3%  perf-stat.i.dTLB-stores
      0.63 ±  2%     +53.0%       0.97        perf-stat.i.ipc
   1192159           +22.0%    1454052 ±  3%  perf-stat.i.major-faults
      0.58 ±  3%     -43.4%       0.33 ±  3%  perf-stat.i.metric.GHz
      1271            +8.5%       1380 ±  3%  perf-stat.i.metric.K/sec
    170.68            +9.0%     186.00 ±  3%  perf-stat.i.metric.M/sec
     37395 ±  3%     +20.8%      45168 ±  3%  perf-stat.i.minor-faults
  34087239 ±  3%      +8.9%   37130893        perf-stat.i.node-load-misses
  25769456 ±  4%     +20.3%   30992977 ± 10%  perf-stat.i.node-loads
   1229555           +21.9%    1499220 ±  3%  perf-stat.i.page-faults
      0.33            +0.0        0.36 ±  2%  perf-stat.overall.branch-miss-rate%
     75.82            +3.9       79.69        perf-stat.overall.cache-miss-rate%
      2.17 ±  2%     -46.6%       1.16        perf-stat.overall.cpi
    209.90 ±  3%     -49.8%     105.47        perf-stat.overall.cycles-between-cache-misses
      0.01 ± 15%      +0.0        0.02 ± 13%  perf-stat.overall.dTLB-load-miss-rate%
      0.46 ±  2%     +87.1%       0.86        perf-stat.overall.ipc
     11007           -13.6%       9506        perf-stat.overall.path-length
  17381310            +9.7%   19064131 ±  5%  perf-stat.ps.branch-misses
 2.689e+08           +12.5%  3.025e+08 ±  3%  perf-stat.ps.cache-misses
 3.547e+08 ±  2%      +7.1%  3.797e+08 ±  3%  perf-stat.ps.cache-references
 5.643e+10 ±  3%     -43.4%  3.192e+10 ±  4%  perf-stat.ps.cpu-cycles
    856351 ± 16%     +58.2%    1354654 ± 14%  perf-stat.ps.dTLB-load-misses
 7.162e+09            +7.8%  7.719e+09 ±  4%  perf-stat.ps.dTLB-loads
   4936410           +22.4%    6041559 ±  3%  perf-stat.ps.dTLB-store-misses
 3.696e+09           +20.7%  4.461e+09 ±  3%  perf-stat.ps.dTLB-stores
   1183837           +22.6%    1451485 ±  3%  perf-stat.ps.major-faults
     37115 ±  3%     +21.5%      45089 ±  3%  perf-stat.ps.minor-faults
  33988476 ±  3%      +9.0%   37042889        perf-stat.ps.node-load-misses
  25570492 ±  4%     +21.0%   30930343 ± 10%  perf-stat.ps.node-loads
   1220952           +22.6%    1496575 ±  3%  perf-stat.ps.page-faults
     30.04 ± 29%     -30.0        0.00        perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.fault_dirty_shared_page
     26.77 ± 37%     -26.8        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited
     26.75 ± 37%     -26.8        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages
     37.94 ± 16%     -24.0       13.94 ±  8%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     36.20 ± 18%     -23.9       12.33 ±  8%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     38.68 ± 15%     -23.1       15.58 ±  9%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     38.76 ± 15%     -23.0       15.74 ±  9%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
     38.95 ± 15%     -22.9       16.05 ±  9%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
     15.58 ± 29%     -15.4        0.17 ±141%  perf-profile.calltrace.cycles-pp.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.fault_dirty_shared_page.do_fault
     15.64 ± 29%     -15.3        0.39 ± 71%  perf-profile.calltrace.cycles-pp.balance_dirty_pages.balance_dirty_pages_ratelimited.fault_dirty_shared_page.do_fault.__handle_mm_fault
     15.78 ± 28%     -14.9        0.86 ± 12%  perf-profile.calltrace.cycles-pp.balance_dirty_pages_ratelimited.fault_dirty_shared_page.do_fault.__handle_mm_fault.handle_mm_fault
     16.00 ± 27%     -14.7        1.33 ± 12%  perf-profile.calltrace.cycles-pp.fault_dirty_shared_page.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
     14.63 ± 29%     -14.6        0.00        perf-profile.calltrace.cycles-pp.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.fault_dirty_shared_page.do_wp_page
     14.69 ± 29%     -14.4        0.27 ±100%  perf-profile.calltrace.cycles-pp.balance_dirty_pages.balance_dirty_pages_ratelimited.fault_dirty_shared_page.do_wp_page.__handle_mm_fault
     14.81 ± 28%     -14.1        0.75 ± 12%  perf-profile.calltrace.cycles-pp.balance_dirty_pages_ratelimited.fault_dirty_shared_page.do_wp_page.__handle_mm_fault.handle_mm_fault
     14.99 ± 28%     -13.9        1.13 ± 11%  perf-profile.calltrace.cycles-pp.fault_dirty_shared_page.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
     16.86 ± 20%     -12.4        4.41 ± 11%  perf-profile.calltrace.cycles-pp.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
     18.72 ± 18%     -12.1        6.61 ±  7%  perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      1.30 ± 28%      -0.6        0.68 ± 14%  perf-profile.calltrace.cycles-pp.__count_memcg_events.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.18 ±141%      +0.6        0.74 ± 12%  perf-profile.calltrace.cycles-pp.filemap_fault.__do_fault.do_fault.__handle_mm_fault.handle_mm_fault
      0.18 ±142%      +0.6        0.80 ± 12%  perf-profile.calltrace.cycles-pp.iomap_iter.iomap_page_mkwrite.__xfs_filemap_fault.do_page_mkwrite.do_wp_page
      0.22 ±141%      +0.7        0.89 ± 11%  perf-profile.calltrace.cycles-pp.__do_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.23 ±141%      +0.7        0.90 ± 25%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.70 ± 72%      +0.7        1.39 ±  6%  perf-profile.calltrace.cycles-pp.__set_page_dirty_nobuffers.iomap_page_mkwrite.__xfs_filemap_fault.do_page_mkwrite.do_fault
      0.20 ±142%      +0.7        0.95 ±  8%  perf-profile.calltrace.cycles-pp.iomap_iter.iomap_page_mkwrite.__xfs_filemap_fault.do_page_mkwrite.do_fault
      0.10 ±223%      +0.8        0.87 ± 15%  perf-profile.calltrace.cycles-pp.td_io_queue
      0.51 ± 75%      +0.8        1.28 ±  7%  perf-profile.calltrace.cycles-pp.sync_regs.error_entry
      0.82 ± 74%      +0.8        1.63 ±  9%  perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.page_mkclean_one.rmap_walk_file.page_mkclean.clear_page_dirty_for_io
      0.55 ± 74%      +0.9        1.43 ± 10%  perf-profile.calltrace.cycles-pp.get_io_u
      0.60 ± 75%      +0.9        1.50 ±  8%  perf-profile.calltrace.cycles-pp.error_entry
      0.73 ± 76%      +0.9        1.64 ± 24%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.74 ± 76%      +0.9        1.66 ± 24%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.24 ± 76%      +1.0        2.22 ± 16%  perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.pmem_submit_bio.__submit_bio.__submit_bio_noacct
      1.37 ± 41%      +1.0        2.34 ± 13%  perf-profile.calltrace.cycles-pp.iomap_page_mkwrite.__xfs_filemap_fault.do_page_mkwrite.do_wp_page.__handle_mm_fault
      0.55 ± 73%      +1.0        1.55 ±  8%  perf-profile.calltrace.cycles-pp.io_completed
      1.29 ± 76%      +1.0        2.31 ± 16%  perf-profile.calltrace.cycles-pp.iomap_finish_ioend.pmem_submit_bio.__submit_bio.__submit_bio_noacct.iomap_submit_ioend
      1.13 ± 41%      +1.0        2.17 ±  6%  perf-profile.calltrace.cycles-pp.fio_gettime
      0.08 ±223%      +1.2        1.29 ±  9%  perf-profile.calltrace.cycles-pp.xfs_buffered_write_iomap_begin.iomap_iter.iomap_page_mkwrite.__xfs_filemap_fault.do_page_mkwrite
      1.65 ± 40%      +1.2        2.89 ± 11%  perf-profile.calltrace.cycles-pp.__xfs_filemap_fault.do_page_mkwrite.do_wp_page.__handle_mm_fault.handle_mm_fault
      1.67 ± 40%      +1.3        2.93 ± 11%  perf-profile.calltrace.cycles-pp.do_page_mkwrite.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      1.16 ± 78%      +1.3        2.44 ± 24%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      1.47 ± 37%      +1.3        2.77 ±  5%  perf-profile.calltrace.cycles-pp.iomap_page_mkwrite.__xfs_filemap_fault.do_page_mkwrite.do_fault.__handle_mm_fault
      1.30 ± 78%      +1.4        2.73 ± 25%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      1.78 ± 37%      +1.6        3.38 ±  5%  perf-profile.calltrace.cycles-pp.__xfs_filemap_fault.do_page_mkwrite.do_fault.__handle_mm_fault.handle_mm_fault
      1.79 ± 37%      +1.6        3.41 ±  5%  perf-profile.calltrace.cycles-pp.do_page_mkwrite.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      3.66 ± 34%      +2.2        5.84 ± 11%  perf-profile.calltrace.cycles-pp.rmap_walk_file.page_mkclean.clear_page_dirty_for_io.write_cache_pages.iomap_writepages
      3.82 ± 35%      +2.4        6.18 ± 12%  perf-profile.calltrace.cycles-pp.page_mkclean.clear_page_dirty_for_io.write_cache_pages.iomap_writepages.xfs_vm_writepages
      4.97 ± 31%      +2.5        7.51 ± 12%  perf-profile.calltrace.cycles-pp.clear_page_dirty_for_io.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
      0.51 ±146%      +3.2        3.69 ± 47%  perf-profile.calltrace.cycles-pp.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64
      7.34 ± 43%      +4.9       12.24 ± 13%  perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
      1.47 ±143%      +7.9        9.40 ± 49%  perf-profile.calltrace.cycles-pp.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      1.47 ±143%      +7.9        9.40 ± 49%  perf-profile.calltrace.cycles-pp.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      1.47 ±143%      +7.9        9.40 ± 49%  perf-profile.calltrace.cycles-pp.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.47 ±143%      +8.0        9.42 ± 49%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.posix_fadvise
      1.47 ±143%      +8.0        9.42 ± 49%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      1.47 ±143%      +8.0        9.42 ± 49%  perf-profile.calltrace.cycles-pp.posix_fadvise
     30.21 ± 29%     -29.4        0.80 ± 14%  perf-profile.children.cycles-pp.mem_cgroup_wb_stats
     30.33 ± 29%     -29.3        1.04 ± 13%  perf-profile.children.cycles-pp.balance_dirty_pages
     30.04 ± 29%     -29.3        0.77 ± 14%  perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe
     30.59 ± 28%     -29.0        1.62 ± 11%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
     31.00 ± 27%     -28.5        2.48 ± 10%  perf-profile.children.cycles-pp.fault_dirty_shared_page
     27.21 ± 36%     -26.7        0.52 ± 68%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     27.89 ± 34%     -26.4        1.53 ± 31%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     37.97 ± 16%     -24.0       14.00 ±  8%  perf-profile.children.cycles-pp.handle_mm_fault
     36.21 ± 18%     -23.8       12.36 ±  8%  perf-profile.children.cycles-pp.__handle_mm_fault
     38.71 ± 15%     -23.1       15.65 ±  9%  perf-profile.children.cycles-pp.do_user_addr_fault
     38.77 ± 15%     -23.0       15.76 ±  9%  perf-profile.children.cycles-pp.exc_page_fault
     38.98 ± 15%     -22.9       16.12 ±  9%  perf-profile.children.cycles-pp.asm_exc_page_fault
     16.87 ± 20%     -12.4        4.42 ± 11%  perf-profile.children.cycles-pp.do_wp_page
     18.73 ± 18%     -12.1        6.63 ±  7%  perf-profile.children.cycles-pp.do_fault
      3.27 ± 36%      -2.5        0.76 ± 14%  perf-profile.children.cycles-pp.cgroup_rstat_flush_locked
      2.14 ± 28%      -1.5        0.68 ± 16%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      1.51 ± 30%      -0.7        0.78 ± 14%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      1.20 ± 34%      -0.7        0.49 ± 18%  perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush
      1.33 ± 28%      -0.6        0.71 ± 15%  perf-profile.children.cycles-pp.__count_memcg_events
      0.03 ±100%      +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.inc_node_page_state
      0.04 ± 71%      +0.0        0.09 ± 18%  perf-profile.children.cycles-pp.task_tick_fair
      0.01 ±223%      +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.perf_exclude_event
      0.00            +0.1        0.05 ± 13%  perf-profile.children.cycles-pp.iomap_do_writepage
      0.02 ±143%      +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.vma_interval_tree_iter_next
      0.02 ±141%      +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.__irqentry_text_end
      0.05 ± 76%      +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.fput_many
      0.01 ±223%      +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.02 ±144%      +0.1        0.08 ± 32%  perf-profile.children.cycles-pp.get_start_offset
      0.02 ±141%      +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.io_bytes_exceeded
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.xfs_filemap_fault
      0.05 ± 74%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.in_ramp_time
      0.01 ±223%      +0.1        0.08 ± 31%  perf-profile.children.cycles-pp.rcu_pending
      0.04 ± 72%      +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.ntime_since
      0.07 ± 77%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.xas_find_marked
      0.06 ± 75%      +0.1        0.13 ± 21%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.07 ± 73%      +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.__inc_zone_page_state
      0.02 ±142%      +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.utime_since
      0.11 ± 17%      +0.1        0.19 ± 25%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.05 ± 73%      +0.1        0.13 ± 15%  perf-profile.children.cycles-pp.up_write
      0.06 ± 78%      +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.xas_set_mark
      0.02 ±142%      +0.1        0.10 ± 37%  perf-profile.children.cycles-pp.__schedule
      0.01 ±223%      +0.1        0.09 ± 12%  perf-profile.children.cycles-pp.zbd_unaligned_write
      0.10 ± 40%      +0.1        0.18 ± 16%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.03 ±100%      +0.1        0.11 ± 20%  perf-profile.children.cycles-pp.log_io_u
      0.04 ± 73%      +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      0.06 ± 74%      +0.1        0.15 ±  9%  perf-profile.children.cycles-pp.finish_mkwrite_fault
      0.06 ± 78%      +0.1        0.16 ± 31%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.08 ± 56%      +0.1        0.18 ±  9%  perf-profile.children.cycles-pp.unlock_page_memcg
      0.06 ± 74%      +0.1        0.16 ± 13%  perf-profile.children.cycles-pp.down_read_trylock
      0.06 ± 74%      +0.1        0.17 ± 12%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.07 ± 78%      +0.1        0.17 ± 27%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.06 ± 75%      +0.1        0.16 ± 11%  perf-profile.children.cycles-pp.utime_since_now
      0.02 ±145%      +0.1        0.13 ± 24%  perf-profile.children.cycles-pp.llist_reverse_order
      0.09 ± 44%      +0.1        0.20 ± 19%  perf-profile.children.cycles-pp.init_icd
      0.14 ± 43%      +0.1        0.25 ± 10%  perf-profile.children.cycles-pp.next_uptodate_page
      0.10 ± 57%      +0.1        0.21 ± 11%  perf-profile.children.cycles-pp.rcu_all_qs
      0.07 ± 77%      +0.1        0.18 ± 10%  perf-profile.children.cycles-pp.xas_start
      0.06 ± 76%      +0.1        0.18 ±  9%  perf-profile.children.cycles-pp.vmacache_find
      0.10 ± 77%      +0.1        0.22 ± 16%  perf-profile.children.cycles-pp.check_pte
      0.09 ± 64%      +0.1        0.21 ± 11%  perf-profile.children.cycles-pp.put_io_u
      0.09 ± 58%      +0.1        0.21 ± 11%  perf-profile.children.cycles-pp.td_io_prep
      0.06 ± 75%      +0.1        0.18 ± 15%  perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
      0.12 ± 42%      +0.1        0.24 ±  5%  perf-profile.children.cycles-pp.__get_io_u
      0.08 ± 59%      +0.1        0.21 ±  8%  perf-profile.children.cycles-pp.find_vma
      0.12 ± 43%      +0.1        0.25 ± 13%  perf-profile.children.cycles-pp.__fprop_inc_percpu
      0.09 ± 75%      +0.1        0.22 ± 22%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.14 ± 49%      +0.1        0.27 ± 11%  perf-profile.children.cycles-pp.down_write
      0.14 ± 62%      +0.1        0.27 ±  6%  perf-profile.children.cycles-pp.PageHuge
      0.04 ±105%      +0.1        0.17 ± 11%  perf-profile.children.cycles-pp.io_u_mark_depth
      0.23 ± 27%      +0.1        0.37 ± 10%  perf-profile.children.cycles-pp.file_update_time
      0.14 ± 43%      +0.2        0.30 ± 10%  perf-profile.children.cycles-pp.rand_between
      0.18 ± 48%      +0.2        0.34 ± 11%  perf-profile.children.cycles-pp.__might_sleep
      0.18 ± 48%      +0.2        0.34 ± 10%  perf-profile.children.cycles-pp.__xa_clear_mark
      0.07 ± 75%      +0.2        0.23 ± 19%  perf-profile.children.cycles-pp.io_u_mark_submit
      0.14 ± 39%      +0.2        0.30 ±  9%  perf-profile.children.cycles-pp.do_set_pte
      0.08 ± 83%      +0.2        0.23 ± 31%  perf-profile.children.cycles-pp.get_next_seq_offset
      0.13 ± 47%      +0.2        0.30 ± 10%  perf-profile.children.cycles-pp.finish_fault
      0.21 ± 46%      +0.2        0.38 ±  9%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.22 ± 47%      +0.2        0.39 ±  8%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.16 ± 43%      +0.2        0.34 ± 20%  perf-profile.children.cycles-pp.lock_page_memcg
      0.11 ± 40%      +0.2        0.30 ± 13%  perf-profile.children.cycles-pp.io_u_sync_complete
      0.16 ± 43%      +0.2        0.35 ± 13%  perf-profile.children.cycles-pp.set_page_dirty
      0.16 ± 49%      +0.2        0.36 ±  9%  perf-profile.children.cycles-pp.___perf_sw_event
      0.02 ±141%      +0.2        0.23 ± 42%  perf-profile.children.cycles-pp.xas_find
      0.13 ± 48%      +0.2        0.34 ± 15%  perf-profile.children.cycles-pp.io_queue_event
      0.15 ± 45%      +0.2        0.36 ± 22%  perf-profile.children.cycles-pp.fio_mmapio_queue
      0.16 ± 37%      +0.2        0.38 ± 10%  perf-profile.children.cycles-pp.xfs_iunlock
      0.23 ± 44%      +0.2        0.44 ±  9%  perf-profile.children.cycles-pp.__xa_set_mark
      0.22 ± 49%      +0.2        0.43 ± 18%  perf-profile.children.cycles-pp.scheduler_tick
      0.10 ± 59%      +0.2        0.32 ±  4%  perf-profile.children.cycles-pp.io_u_mark_complete
      0.26 ± 47%      +0.2        0.48 ±  7%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.23 ± 42%      +0.2        0.46 ± 12%  perf-profile.children.cycles-pp.handle_pte_fault
      0.22 ± 44%      +0.2        0.46 ±  7%  perf-profile.children.cycles-pp.account_io_completion
      0.22 ± 44%      +0.2        0.46 ±  7%  perf-profile.children.cycles-pp.__cond_resched
      0.31 ± 47%      +0.2        0.56 ±  6%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.28 ± 43%      +0.2        0.53 ± 10%  perf-profile.children.cycles-pp.filemap_map_pages
      0.29 ± 43%      +0.2        0.54 ± 10%  perf-profile.children.cycles-pp.xfs_filemap_map_pages
      0.27 ± 41%      +0.3        0.53 ± 12%  perf-profile.children.cycles-pp.pagecache_get_page
      0.32 ± 42%      +0.3        0.59 ± 10%  perf-profile.children.cycles-pp.do_read_fault
      0.26 ± 46%      +0.3        0.53 ±  5%  perf-profile.children.cycles-pp.add_lat_sample
      0.24 ± 36%      +0.3        0.54 ± 14%  perf-profile.children.cycles-pp.up_read
      0.10 ± 27%      +0.3        0.40 ± 33%  perf-profile.children.cycles-pp.__pagevec_release
      0.28 ± 45%      +0.3        0.58 ± 12%  perf-profile.children.cycles-pp.add_clat_sample
      0.32 ± 45%      +0.3        0.63 ± 11%  perf-profile.children.cycles-pp.xfs_ilock
      0.32 ± 59%      +0.3        0.65 ± 16%  perf-profile.children.cycles-pp.clockevents_program_event
      0.36 ± 43%      +0.3        0.70 ± 10%  perf-profile.children.cycles-pp.down_read
      0.26 ± 38%      +0.4        0.62 ± 16%  perf-profile.children.cycles-pp.page_mapping
      0.27 ± 48%      +0.4        0.62 ±  8%  perf-profile.children.cycles-pp.__perf_sw_event
      0.22 ± 41%      +0.4        0.58 ± 29%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.38 ± 41%      +0.4        0.74 ± 12%  perf-profile.children.cycles-pp.filemap_fault
      0.22 ± 42%      +0.4        0.58 ± 29%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.19 ± 21%      +0.4        0.56 ± 20%  perf-profile.children.cycles-pp.unlock_page
      0.29 ± 45%      +0.4        0.65 ±  8%  perf-profile.children.cycles-pp.___might_sleep
      0.24 ± 40%      +0.4        0.64 ± 28%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.38 ± 48%      +0.4        0.78 ± 18%  perf-profile.children.cycles-pp.update_process_times
      0.32 ± 46%      +0.4        0.73 ±  8%  perf-profile.children.cycles-pp.thread_main
      0.38 ± 47%      +0.4        0.80 ± 18%  perf-profile.children.cycles-pp.tick_sched_handle
      0.45 ± 41%      +0.4        0.89 ± 11%  perf-profile.children.cycles-pp.__do_fault
      0.42 ± 48%      +0.4        0.86 ± 19%  perf-profile.children.cycles-pp.tick_sched_timer
      0.33 ± 33%      +0.5        0.84 ± 31%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.13 ± 52%      +0.6        0.68 ± 39%  perf-profile.children.cycles-pp.release_pages
      1.09 ± 41%      +0.6        1.65 ±  9%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.45 ± 43%      +0.6        1.02 ± 13%  perf-profile.children.cycles-pp.td_io_queue
      0.56 ± 50%      +0.6        1.17 ± 20%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.59 ± 42%      +0.6        1.20 ±  9%  perf-profile.children.cycles-pp.xas_load
      0.45 ± 46%      +0.6        1.10 ± 14%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.63 ± 43%      +0.7        1.34 ±  7%  perf-profile.children.cycles-pp.sync_regs
      0.59 ± 41%      +0.7        1.31 ±  9%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      0.00            +0.8        0.78 ± 13%  perf-profile.children.cycles-pp.mem_cgroup_flush_stats
      0.68 ± 42%      +0.8        1.48 ±  9%  perf-profile.children.cycles-pp.get_io_u
      0.78 ± 42%      +0.9        1.66 ±  8%  perf-profile.children.cycles-pp.error_entry
      0.66 ± 41%      +0.9        1.56 ±  8%  perf-profile.children.cycles-pp.io_completed
      0.13 ±141%      +0.9        1.07 ± 44%  perf-profile.children.cycles-pp.find_lock_entries
      0.80 ± 40%      +1.0        1.76 ±  8%  perf-profile.children.cycles-pp.iomap_iter
      0.96 ± 51%      +1.0        1.96 ±  6%  perf-profile.children.cycles-pp.page_vma_mapped_walk
      1.01 ± 54%      +1.0        2.03 ± 20%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.02 ± 55%      +1.0        2.05 ± 20%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      1.36 ± 41%      +1.0        2.40 ± 14%  perf-profile.children.cycles-pp.test_clear_page_writeback
      1.14 ± 41%      +1.0        2.18 ±  6%  perf-profile.children.cycles-pp.fio_gettime
      1.92 ± 38%      +1.1        3.04 ±  8%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      1.51 ± 41%      +1.1        2.65 ± 12%  perf-profile.children.cycles-pp.end_page_writeback
      0.99 ± 41%      +1.2        2.16 ±  8%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.25 ±157%      +1.2        1.44 ± 58%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      1.56 ± 41%      +1.2        2.76 ± 12%  perf-profile.children.cycles-pp.iomap_finish_ioend
      1.51 ± 58%      +1.4        2.88 ± 20%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.30 ±154%      +1.6        1.86 ± 54%  perf-profile.children.cycles-pp.deactivate_file_page
      1.72 ± 58%      +1.6        3.29 ± 20%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      1.84 ± 36%      +1.7        3.54 ± 21%  perf-profile.children.cycles-pp.iomap_submit_ioend
      3.82 ± 28%      +2.1        5.92 ± 11%  perf-profile.children.cycles-pp.rmap_walk_file
      3.97 ± 29%      +2.2        6.20 ± 11%  perf-profile.children.cycles-pp.page_mkclean
      2.85 ± 39%      +2.3        5.14 ±  8%  perf-profile.children.cycles-pp.iomap_page_mkwrite
      4.98 ± 31%      +2.6        7.55 ± 11%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      3.45 ± 38%      +2.9        6.31 ±  7%  perf-profile.children.cycles-pp.__xfs_filemap_fault
      3.47 ± 38%      +2.9        6.34 ±  7%  perf-profile.children.cycles-pp.do_page_mkwrite
      0.52 ±146%      +3.2        3.72 ± 47%  perf-profile.children.cycles-pp.__invalidate_mapping_pages
      0.95 ±142%      +4.7        5.69 ± 52%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.95 ±142%      +4.7        5.69 ± 52%  perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
      7.34 ± 43%      +4.9       12.29 ± 13%  perf-profile.children.cycles-pp.iomap_writepage_map
      1.47 ±143%      +7.9        9.40 ± 49%  perf-profile.children.cycles-pp.__x64_sys_fadvise64
      1.47 ±143%      +7.9        9.40 ± 49%  perf-profile.children.cycles-pp.ksys_fadvise64_64
      1.47 ±143%      +7.9        9.40 ± 49%  perf-profile.children.cycles-pp.generic_fadvise
      1.47 ±143%      +8.0        9.42 ± 49%  perf-profile.children.cycles-pp.posix_fadvise
     12.85 ± 38%      +8.0       20.81 ± 11%  perf-profile.children.cycles-pp.iomap_writepages
     12.85 ± 38%      +8.0       20.81 ± 11%  perf-profile.children.cycles-pp.write_cache_pages
      1.62 ±129%      +8.0        9.63 ± 48%  perf-profile.children.cycles-pp.do_syscall_64
      1.62 ±129%      +8.0        9.63 ± 48%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     14.58 ± 39%      +8.9       23.44 ± 10%  perf-profile.children.cycles-pp.do_writepages
     14.58 ± 39%      +8.9       23.44 ± 10%  perf-profile.children.cycles-pp.xfs_vm_writepages
     27.20 ± 36%     -26.7        0.51 ± 68%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.58 ± 27%      -1.2        0.42 ± 17%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      1.17 ± 34%      -0.7        0.47 ± 19%  perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush
      0.42 ± 31%      -0.3        0.09 ± 12%  perf-profile.self.cycles-pp.cgroup_rstat_flush_locked
      0.02 ±141%      +0.0        0.06 ± 13%  perf-profile.self.cycles-pp.finish_fault
      0.04 ± 76%      +0.0        0.09 ± 12%  perf-profile.self.cycles-pp.write_pmem
      0.04 ± 76%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.in_ramp_time
      0.02 ±142%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.pmem_do_write
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.do_fault
      0.04 ± 73%      +0.1        0.09 ±  9%  perf-profile.self.cycles-pp.xfs_ilock
      0.05 ± 73%      +0.1        0.11 ± 15%  perf-profile.self.cycles-pp.iomap_finish_ioend
      0.05 ± 76%      +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.fput_many
      0.02 ±141%      +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.__irqentry_text_end
      0.04 ± 73%      +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.03 ±102%      +0.1        0.08 ± 21%  perf-profile.self.cycles-pp.do_set_pte
      0.04 ± 72%      +0.1        0.09 ± 14%  perf-profile.self.cycles-pp.file_update_time
      0.01 ±223%      +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.memset_erms
      0.02 ±142%      +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.vma_interval_tree_iter_next
      0.05 ± 76%      +0.1        0.11 ± 16%  perf-profile.self.cycles-pp.balance_dirty_pages
      0.04 ± 73%      +0.1        0.10 ± 14%  perf-profile.self.cycles-pp.exc_page_fault
      0.02 ±141%      +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.io_bytes_exceeded
      0.03 ±100%      +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.ntime_since
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.xfs_filemap_fault
      0.02 ±144%      +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.utime_since
      0.06 ± 74%      +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.__bio_try_merge_page
      0.01 ±223%      +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.05 ± 72%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.up_write
      0.03 ±100%      +0.1        0.10 ± 18%  perf-profile.self.cycles-pp.pmem_submit_bio
      0.07 ± 77%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.xas_find_marked
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.__inc_zone_page_state
      0.06 ± 76%      +0.1        0.13 ± 14%  perf-profile.self.cycles-pp.iomap_add_to_ioend
      0.04 ± 75%      +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.04 ± 73%      +0.1        0.11 ± 12%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.00            +0.1        0.07 ±  9%  perf-profile.self.cycles-pp.__set_page_dirty
      0.06 ± 74%      +0.1        0.13 ± 11%  perf-profile.self.cycles-pp.set_page_dirty
      0.06 ± 76%      +0.1        0.13 ± 11%  perf-profile.self.cycles-pp.xas_set_mark
      0.06 ± 81%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.fio_mmapio_prep
      0.13 ± 39%      +0.1        0.22 ± 10%  perf-profile.self.cycles-pp.pagecache_get_page
      0.02 ±142%      +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.log_io_u
      0.07 ± 74%      +0.1        0.15 ± 13%  perf-profile.self.cycles-pp.rcu_all_qs
      0.06 ± 74%      +0.1        0.15 ± 11%  perf-profile.self.cycles-pp.unlock_page_memcg
      0.03 ±105%      +0.1        0.12 ± 18%  perf-profile.self.cycles-pp.xfs_iunlock
      0.09 ± 74%      +0.1        0.17 ± 13%  perf-profile.self.cycles-pp.flush_tlb_mm_range
      0.09 ± 56%      +0.1        0.18 ± 15%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.05 ± 74%      +0.1        0.14 ±  8%  perf-profile.self.cycles-pp.down_read_trylock
      0.07 ± 81%      +0.1        0.16 ± 11%  perf-profile.self.cycles-pp.down_write
      0.03 ±101%      +0.1        0.12 ± 11%  perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
      0.09 ± 42%      +0.1        0.18 ± 15%  perf-profile.self.cycles-pp.init_icd
      0.09 ± 76%      +0.1        0.19 ± 18%  perf-profile.self.cycles-pp.check_pte
      0.09 ± 55%      +0.1        0.19 ± 13%  perf-profile.self.cycles-pp.asm_exc_page_fault
      0.06 ± 75%      +0.1        0.16 ± 13%  perf-profile.self.cycles-pp.xas_start
      0.06 ± 73%      +0.1        0.16 ±  9%  perf-profile.self.cycles-pp.utime_since_now
      0.07 ± 78%      +0.1        0.17 ± 27%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.10 ± 40%      +0.1        0.21 ± 11%  perf-profile.self.cycles-pp.filemap_fault
      0.06 ± 78%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.vmacache_find
      0.05 ± 77%      +0.1        0.16 ± 18%  perf-profile.self.cycles-pp.account_page_dirtied
      0.02 ±145%      +0.1        0.13 ± 24%  perf-profile.self.cycles-pp.llist_reverse_order
      0.14 ± 43%      +0.1        0.25 ± 11%  perf-profile.self.cycles-pp.next_uptodate_page
      0.07 ± 78%      +0.1        0.18 ± 13%  perf-profile.self.cycles-pp.fault_dirty_shared_page
      0.14 ± 44%      +0.1        0.25 ± 12%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.14 ± 47%      +0.1        0.26 ±  5%  perf-profile.self.cycles-pp.end_page_writeback
      0.08 ± 75%      +0.1        0.20 ± 13%  perf-profile.self.cycles-pp.td_io_prep
      0.08 ± 80%      +0.1        0.19 ± 11%  perf-profile.self.cycles-pp.put_io_u
      0.07 ± 82%      +0.1        0.18 ± 19%  perf-profile.self.cycles-pp.update_process_times
      0.11 ± 40%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.__cond_resched
      0.10 ± 36%      +0.1        0.23 ±  9%  perf-profile.self.cycles-pp.__xfs_filemap_fault
      0.10 ± 76%      +0.1        0.22 ±  6%  perf-profile.self.cycles-pp.PageHuge
      0.04 ±105%      +0.1        0.16 ± 12%  perf-profile.self.cycles-pp.io_u_mark_depth
      0.12 ± 60%      +0.1        0.25 ±  5%  perf-profile.self.cycles-pp.rmap_walk_file
      0.10 ± 59%      +0.1        0.24 ±  7%  perf-profile.self.cycles-pp.__get_io_u
      0.11 ± 44%      +0.1        0.24 ± 10%  perf-profile.self.cycles-pp.handle_pte_fault
      0.16 ± 48%      +0.1        0.29 ± 13%  perf-profile.self.cycles-pp.write_cache_pages
      0.17 ± 48%      +0.1        0.32 ± 10%  perf-profile.self.cycles-pp.__might_sleep
      0.07 ± 74%      +0.1        0.21 ± 21%  perf-profile.self.cycles-pp.io_u_mark_submit
      0.15 ± 43%      +0.2        0.30 ± 20%  perf-profile.self.cycles-pp.lock_page_memcg
      0.14 ± 45%      +0.2        0.29 ±  8%  perf-profile.self.cycles-pp.rand_between
      0.07 ± 85%      +0.2        0.22 ± 30%  perf-profile.self.cycles-pp.get_next_seq_offset
      0.10 ± 59%      +0.2        0.26 ± 12%  perf-profile.self.cycles-pp.__perf_sw_event
      0.10 ± 60%      +0.2        0.26 ± 11%  perf-profile.self.cycles-pp.___perf_sw_event
      0.19 ± 48%      +0.2        0.35 ±  6%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.08 ± 58%      +0.2        0.24 ± 32%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.14 ± 37%      +0.2        0.30 ± 11%  perf-profile.self.cycles-pp.error_entry
      0.24 ± 44%      +0.2        0.41 ± 10%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.10 ± 61%      +0.2        0.27 ± 16%  perf-profile.self.cycles-pp.io_queue_event
      0.10 ± 59%      +0.2        0.28 ± 14%  perf-profile.self.cycles-pp.io_u_sync_complete
      0.17 ± 42%      +0.2        0.36 ± 10%  perf-profile.self.cycles-pp.do_user_addr_fault
      0.08 ± 74%      +0.2        0.28 ±  5%  perf-profile.self.cycles-pp.io_u_mark_complete
      0.17 ± 37%      +0.2        0.37 ±  8%  perf-profile.self.cycles-pp.iomap_iter
      0.15 ± 46%      +0.2        0.35 ± 22%  perf-profile.self.cycles-pp.fio_mmapio_queue
      0.25 ± 46%      +0.2        0.46 ±  8%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.22 ± 42%      +0.2        0.44 ±  9%  perf-profile.self.cycles-pp.down_read
      0.20 ± 47%      +0.2        0.42 ± 11%  perf-profile.self.cycles-pp.iomap_page_mkwrite
      0.15 ± 37%      +0.2        0.38 ±  7%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
      0.25 ± 44%      +0.2        0.48 ± 12%  perf-profile.self.cycles-pp.__set_page_dirty_nobuffers
      0.22 ± 45%      +0.2        0.45 ±  7%  perf-profile.self.cycles-pp.account_io_completion
      0.26 ± 55%      +0.3        0.51 ±  6%  perf-profile.self.cycles-pp.page_mkclean_one
      0.25 ± 46%      +0.3        0.51 ±  5%  perf-profile.self.cycles-pp.add_lat_sample
      0.26 ± 46%      +0.3        0.52 ± 14%  perf-profile.self.cycles-pp.test_clear_page_writeback
      0.21 ± 51%      +0.3        0.48 ± 10%  perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
      0.24 ± 35%      +0.3        0.52 ± 14%  perf-profile.self.cycles-pp.up_read
      0.24 ± 42%      +0.3        0.54 ± 13%  perf-profile.self.cycles-pp.handle_mm_fault
      0.26 ± 44%      +0.3        0.56 ± 13%  perf-profile.self.cycles-pp.add_clat_sample
      0.34 ± 67%      +0.3        0.67 ± 14%  perf-profile.self.cycles-pp.ktime_get
      0.24 ± 37%      +0.3        0.57 ± 16%  perf-profile.self.cycles-pp.page_mapping
      0.18 ± 20%      +0.3        0.52 ± 20%  perf-profile.self.cycles-pp.unlock_page
      0.27 ± 44%      +0.3        0.62 ±  7%  perf-profile.self.cycles-pp.___might_sleep
      0.26 ± 46%      +0.4        0.63 ±  9%  perf-profile.self.cycles-pp.thread_main
      0.04 ±142%      +0.4        0.41 ± 40%  perf-profile.self.cycles-pp.deactivate_file_page
      0.36 ± 44%      +0.4        0.81 ± 11%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.31 ± 43%      +0.5        0.78 ± 14%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.51 ± 42%      +0.5        1.02 ±  8%  perf-profile.self.cycles-pp.xas_load
      0.12 ± 53%      +0.5        0.64 ± 39%  perf-profile.self.cycles-pp.release_pages
      0.43 ± 43%      +0.5        0.98 ± 13%  perf-profile.self.cycles-pp.td_io_queue
      0.08 ±142%      +0.6        0.66 ± 43%  perf-profile.self.cycles-pp.pagevec_lru_move_fn
      0.62 ± 51%      +0.6        1.22 ±  5%  perf-profile.self.cycles-pp.page_vma_mapped_walk
      0.62 ± 44%      +0.7        1.31 ±  7%  perf-profile.self.cycles-pp.sync_regs
      0.66 ± 42%      +0.8        1.43 ±  9%  perf-profile.self.cycles-pp.get_io_u
      0.10 ±141%      +0.8        0.88 ± 43%  perf-profile.self.cycles-pp.find_lock_entries
      0.64 ± 41%      +0.9        1.52 ±  8%  perf-profile.self.cycles-pp.io_completed
      1.09 ± 42%      +1.0        2.07 ±  6%  perf-profile.self.cycles-pp.fio_gettime
      0.99 ± 41%      +1.2        2.16 ±  8%  perf-profile.self.cycles-pp.native_irq_return_iret



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  2M/gcc-11/performance/2pmem/ext4/libaio/x86_64-rhel-8.3/50%/debian-11.1-x86_64-20220510.cgz/200s/rw/lkp-csl-2sp7/200G/fio-basic/tb/0x500320a

commit: 
  11192d9c12 ("memcg: flush stats only if updated")
  fd25a9e0e2 ("memcg: unify memcg stat flushing")

11192d9c124d58d6 fd25a9e0e23b995fd0ba5e2f00a 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.39 ± 88%      +7.0        7.36 ± 29%  fio.latency_100ms%
      0.01            +0.0        0.02 ± 19%  fio.latency_10ms%
      0.02 ± 22%      +0.0        0.02 ± 28%  fio.latency_20ms%
     35.65 ±  6%      +7.2       42.90 ± 11%  fio.latency_250ms%
      0.01            +0.0        0.01 ±  9%  fio.latency_4ms%
     59.44 ±  4%     -15.7       43.78 ± 18%  fio.latency_500ms%
      0.13 ± 46%      +0.7        0.84 ± 30%  fio.latency_50ms%
      4734 ±  2%     +14.0%       5398 ±  2%  fio.read_bw_MBps
 2.916e+08 ±  2%     -13.8%  2.512e+08 ±  3%  fio.read_clat_mean_us
  98714844 ±  7%     +32.8%  1.311e+08 ± 13%  fio.read_clat_stddev
      2367 ±  2%     +14.0%       2699 ±  2%  fio.read_iops
   1290688 ±  2%     +40.6%    1814909 ±  9%  fio.read_slat_mean_us
    755337 ±  5%    +128.4%    1724856 ±  7%  fio.read_slat_stddev
 9.728e+08 ±  2%     +15.0%  1.119e+09 ±  2%  fio.time.file_system_inputs
 1.942e+09 ±  2%     +14.1%  2.217e+09 ±  2%  fio.time.file_system_outputs
     22091 ±  6%     -47.7%      11558 ± 12%  fio.time.involuntary_context_switches
      3556 ±  7%     -56.4%       1549 ± 11%  fio.time.percent_of_cpu_this_job_got
      7057 ±  7%     -58.3%       2943 ± 11%  fio.time.system_time
    121.36 ±  4%     +56.0%     189.33 ± 13%  fio.time.user_time
    948915 ±  2%     +14.2%    1083277 ±  2%  fio.workload
      4729 ±  2%     +14.0%       5391 ±  2%  fio.write_bw_MBps
 2.917e+08 ±  2%     -13.2%  2.531e+08 ±  3%  fio.write_clat_mean_us
  99135535 ±  7%     +32.5%  1.313e+08 ± 14%  fio.write_clat_stddev
      2364 ±  2%     +14.0%       2695 ±  2%  fio.write_iops
  17604710 ±  2%     -17.6%   14505732 ±  4%  fio.write_slat_mean_us
   9434086 ±  9%     +61.3%   15215638        fio.write_slat_stddev
 1.186e+10 ±  4%     +33.7%  1.585e+10 ±  2%  cpuidle..time
  25720587 ±  3%     +33.6%   34354696 ±  3%  cpuidle..usage
     12.02 ± 23%    +165.9%      31.95 ±  6%  iostat.cpu.iowait
     37.83 ±  7%     -54.1%      17.37 ± 10%  iostat.cpu.system
      0.66 ±  4%     +48.9%       0.98 ± 13%  iostat.cpu.user
  67142307 ±  4%     +16.5%   78254209 ±  6%  numa-numastat.node0.local_node
  67142230 ±  4%     +16.6%   78263094 ±  6%  numa-numastat.node0.numa_hit
     65625 ±101%     -95.1%       3227 ±221%  numa-numastat.node0.numa_miss
     65750 ±101%     -95.1%       3227 ±221%  numa-numastat.node1.numa_foreign
     65497 ±  6%     -35.7%      42093 ± 17%  meminfo.Active
     41592 ±  9%     -55.0%      18731 ± 39%  meminfo.Active(anon)
  42484071           +17.8%   50038961        meminfo.Dirty
     55262 ± 11%     -16.9%      45947        meminfo.Mapped
     65949 ± 15%     -49.1%      33545 ± 22%  meminfo.Shmem
     12.13 ± 23%     +20.1       32.26 ±  6%  mpstat.cpu.all.iowait%
      0.65 ±  2%      +0.2        0.83 ±  5%  mpstat.cpu.all.irq%
      0.04 ±  4%      +0.0        0.06 ±  4%  mpstat.cpu.all.soft%
     37.49 ±  7%     -20.9       16.63 ± 10%  mpstat.cpu.all.sys%
      0.66 ±  4%      +0.3        0.99 ± 13%  mpstat.cpu.all.usr%
  10357776 ±  5%     +21.8%   12610998 ±  7%  numa-meminfo.node0.Dirty
     40985 ±  7%     -56.9%      17672 ± 42%  numa-meminfo.node1.Active
     40225 ±  8%     -57.6%      17072 ± 44%  numa-meminfo.node1.Active(anon)
  32140823 ±  3%     +16.5%   37431157        numa-meminfo.node1.Dirty
     55579 ± 16%     -52.5%      26413 ± 27%  numa-meminfo.node1.Shmem
     11.50 ± 24%    +172.5%      31.33 ±  6%  vmstat.cpu.wa
   2365172 ±  2%     +15.0%    2720925 ±  2%  vmstat.io.bi
   4547031 ±  2%     +14.7%    5217062 ±  2%  vmstat.io.bo
     11.17 ± 20%    +177.6%      31.00 ±  6%  vmstat.procs.b
     36.67 ±  6%     -53.6%      17.00 ± 11%  vmstat.procs.r
    177032           +10.0%     194738        vmstat.system.in
      1110 ±  7%     -50.8%     546.33 ±  9%  turbostat.Avg_MHz
     39.76 ±  7%     -20.2       19.59 ±  9%  turbostat.Busy%
  15959953 ± 17%     +43.8%   22954567 ± 11%  turbostat.C1E
     60.04 ±  4%     +33.7%      80.26 ±  2%  turbostat.CPU%c1
      0.06 ±  7%     +74.4%       0.11 ±  6%  turbostat.IPC
     66.17 ±  2%      -7.8%      61.00        turbostat.PkgTmp
    246.81 ±  2%     -13.0%     214.63 ±  2%  turbostat.PkgWatt
     50.88            +2.7%      52.24        turbostat.RAMWatt
  63017097 ±  4%     +16.6%   73450071 ±  6%  numa-vmstat.node0.nr_dirtied
   2589553 ±  5%     +21.8%    3153493 ±  7%  numa-vmstat.node0.nr_dirty
   2594037 ±  5%     +21.7%    3157643 ±  7%  numa-vmstat.node0.nr_zone_write_pending
  67142279 ±  4%     +16.6%   78263002 ±  6%  numa-vmstat.node0.numa_hit
  67142356 ±  4%     +16.5%   78254117 ±  6%  numa-vmstat.node0.numa_local
     65625 ±101%     -95.1%       3227 ±221%  numa-vmstat.node0.numa_miss
     10020 ±  8%     -57.4%       4266 ± 44%  numa-vmstat.node1.nr_active_anon
 1.798e+08 ±  2%     +13.3%  2.037e+08 ±  2%  numa-vmstat.node1.nr_dirtied
   8033999 ±  3%     +16.5%    9358569        numa-vmstat.node1.nr_dirty
     13850 ± 16%     -52.2%       6617 ± 27%  numa-vmstat.node1.nr_shmem
 1.718e+08 ±  2%     +14.0%  1.958e+08 ±  2%  numa-vmstat.node1.nr_written
     10020 ±  8%     -57.4%       4265 ± 44%  numa-vmstat.node1.nr_zone_active_anon
   8043027 ±  3%     +16.5%    9368655        numa-vmstat.node1.nr_zone_write_pending
     65750 ±101%     -95.1%       3227 ±221%  numa-vmstat.node1.numa_foreign
     10398 ±  9%     -55.0%       4682 ± 39%  proc-vmstat.nr_active_anon
 2.428e+08 ±  2%     +14.1%  2.771e+08 ±  2%  proc-vmstat.nr_dirtied
  10620936           +17.8%   12509591        proc-vmstat.nr_dirty
     13815 ± 11%     -16.9%      11486        proc-vmstat.nr_mapped
     16487 ± 15%     -49.1%       8386 ± 22%  proc-vmstat.nr_shmem
    153539            +3.2%     158400        proc-vmstat.nr_slab_unreclaimable
 2.326e+08 ±  2%     +15.0%  2.675e+08 ±  2%  proc-vmstat.nr_written
     10398 ±  9%     -55.0%       4682 ± 39%  proc-vmstat.nr_zone_active_anon
  10634810           +17.8%   12523439        proc-vmstat.nr_zone_write_pending
     34272 ±  9%     +14.0%      39081 ±  9%  proc-vmstat.numa_hint_faults_local
 1.853e+08 ±  3%     +10.8%  2.054e+08 ±  6%  proc-vmstat.numa_hit
     22095 ±  2%     -21.4%      17362 ±  7%  proc-vmstat.numa_huge_pte_updates
 1.854e+08 ±  3%     +10.9%  2.056e+08 ±  6%  proc-vmstat.numa_local
  11378151 ±  2%     -21.5%    8937282 ±  7%  proc-vmstat.numa_pte_updates
     72660 ± 42%     -62.3%      27417 ± 14%  proc-vmstat.pgactivate
 2.505e+08 ±  2%     +14.0%  2.857e+08 ±  2%  proc-vmstat.pgalloc_normal
    714616            -3.8%     687146        proc-vmstat.pgfault
 2.265e+08 ±  3%     +15.6%   2.62e+08 ±  3%  proc-vmstat.pgfree
 4.864e+08 ±  2%     +15.0%  5.596e+08 ±  2%  proc-vmstat.pgpgin
 9.305e+08 ±  2%     +15.0%   1.07e+09 ±  2%  proc-vmstat.pgpgout
      0.41 ± 17%     -58.0%       0.17 ± 16%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.42 ±  5%     -11.8%       0.37 ±  5%  sched_debug.cfs_rq:/.h_nr_running.stddev
    387722 ± 19%     -61.8%     148125 ± 22%  sched_debug.cfs_rq:/.load.avg
    365.68 ± 12%     -60.0%     146.17 ± 24%  sched_debug.cfs_rq:/.load_avg.avg
     20212 ± 10%     -26.3%      14891 ±  5%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.41 ± 17%     -58.0%       0.17 ± 16%  sched_debug.cfs_rq:/.nr_running.avg
      0.42 ±  5%     -11.8%       0.37 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
    432.73 ± 10%     -52.9%     203.67 ±  6%  sched_debug.cfs_rq:/.runnable_avg.avg
    352.26 ±  6%     -29.4%     248.64 ±  7%  sched_debug.cfs_rq:/.runnable_avg.stddev
   -115117           -22.1%     -89637        sched_debug.cfs_rq:/.spread0.min
     20214 ± 10%     -26.3%      14892 ±  5%  sched_debug.cfs_rq:/.spread0.stddev
    432.38 ± 11%     -53.0%     203.20 ±  6%  sched_debug.cfs_rq:/.util_avg.avg
    352.08 ±  6%     -29.6%     248.00 ±  7%  sched_debug.cfs_rq:/.util_avg.stddev
    253.21 ± 15%     -81.2%      47.55 ± 35%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    256.67 ± 10%     -51.8%     123.79 ± 19%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      2.73 ±  4%     -12.3%       2.40 ±  3%  sched_debug.cpu.clock.stddev
      1785 ± 12%     -63.7%     647.66 ± 19%  sched_debug.cpu.curr->pid.avg
      2199 ±  7%     -22.7%       1700 ±  8%  sched_debug.cpu.curr->pid.stddev
      0.34 ± 11%     -59.9%       0.14 ± 15%  sched_debug.cpu.nr_running.avg
      0.42 ±  6%     -20.1%       0.34 ±  6%  sched_debug.cpu.nr_running.stddev
      1585 ± 15%     +36.7%       2166 ± 12%  sched_debug.cpu.nr_switches.min
     23.07 ±  3%     +25.6%      28.98        perf-stat.i.MPKI
 4.901e+09 ±  4%     -24.1%  3.718e+09 ±  2%  perf-stat.i.branch-instructions
      0.28 ±  3%      +0.1        0.35        perf-stat.i.branch-miss-rate%
     82.75            +1.8       84.52        perf-stat.i.cache-miss-rate%
 4.468e+08 ±  2%      +8.9%  4.865e+08 ±  2%  perf-stat.i.cache-misses
  5.38e+08 ±  2%      +7.3%  5.774e+08 ±  3%  perf-stat.i.cache-references
      4.19 ±  4%     -41.7%       2.44 ±  8%  perf-stat.i.cpi
 1.063e+11 ±  7%     -51.6%  5.147e+10 ± 10%  perf-stat.i.cpu-cycles
    252.03 ±  6%     -52.7%     119.14 ±  5%  perf-stat.i.cycles-between-cache-misses
      0.02 ±  9%      +0.0        0.03 ± 17%  perf-stat.i.dTLB-load-miss-rate%
  6.43e+09 ±  4%     -15.1%   5.46e+09 ±  2%  perf-stat.i.dTLB-loads
      0.01 ±  6%      +0.0        0.01 ± 12%  perf-stat.i.dTLB-store-miss-rate%
    197331 ±  7%     +38.3%     272925 ± 14%  perf-stat.i.dTLB-store-misses
 2.947e+09 ±  2%     +12.6%  3.319e+09 ±  2%  perf-stat.i.dTLB-stores
     57.78            -1.8       56.01        perf-stat.i.iTLB-load-miss-rate%
   1859795 ±  4%      +9.6%    2038933 ±  3%  perf-stat.i.iTLB-load-misses
   1306943           +16.1%    1517987        perf-stat.i.iTLB-loads
 2.433e+10 ±  4%     -16.7%  2.027e+10 ±  2%  perf-stat.i.instructions
     13333 ±  2%     -23.5%      10195        perf-stat.i.instructions-per-iTLB-miss
      0.27 ±  6%     +75.7%       0.47 ±  7%  perf-stat.i.ipc
      1.11 ±  7%     -51.6%       0.54 ± 10%  perf-stat.i.metric.GHz
      1654 ±  2%      +6.7%       1765 ±  2%  perf-stat.i.metric.K/sec
    154.33 ±  3%     -11.7%     136.24 ±  2%  perf-stat.i.metric.M/sec
      2892 ±  2%      -4.9%       2749        perf-stat.i.minor-faults
  51092281 ±  3%     +10.9%   56656127 ±  6%  perf-stat.i.node-loads
  56739346 ±  2%     +15.3%   65443819 ±  3%  perf-stat.i.node-stores
      2906 ±  2%      -4.9%       2764        perf-stat.i.page-faults
     22.11 ±  2%     +28.8%      28.48        perf-stat.overall.MPKI
      0.26 ±  2%      +0.1        0.34        perf-stat.overall.branch-miss-rate%
     83.02            +1.2       84.20        perf-stat.overall.cache-miss-rate%
      4.37 ±  3%     -41.6%       2.55 ±  8%  perf-stat.overall.cpi
    238.26 ±  5%     -55.3%     106.40 ±  7%  perf-stat.overall.cycles-between-cache-misses
      0.02 ± 10%      +0.0        0.03 ± 18%  perf-stat.overall.dTLB-load-miss-rate%
      0.01 ±  6%      +0.0        0.01 ± 12%  perf-stat.overall.dTLB-store-miss-rate%
     13066           -24.2%       9902        perf-stat.overall.instructions-per-iTLB-miss
      0.23 ±  3%     +71.9%       0.39 ±  7%  perf-stat.overall.ipc
   5198153 ±  2%     -26.9%    3798948        perf-stat.overall.path-length
 4.884e+09 ±  4%     -24.1%  3.704e+09 ±  2%  perf-stat.ps.branch-instructions
 4.447e+08 ±  2%      +8.9%  4.844e+08 ±  2%  perf-stat.ps.cache-misses
 5.356e+08 ±  2%      +7.4%  5.754e+08 ±  3%  perf-stat.ps.cache-references
  1.06e+11 ±  7%     -51.3%  5.162e+10 ± 10%  perf-stat.ps.cpu-cycles
   1154804 ± 11%     +33.7%    1543751 ± 20%  perf-stat.ps.dTLB-load-misses
 6.409e+09 ±  4%     -15.1%  5.441e+09 ±  2%  perf-stat.ps.dTLB-loads
    196129 ±  7%     +38.3%     271181 ± 14%  perf-stat.ps.dTLB-store-misses
 2.937e+09 ±  2%     +12.6%  3.306e+09 ±  2%  perf-stat.ps.dTLB-stores
   1856169 ±  4%      +9.9%    2040184 ±  2%  perf-stat.ps.iTLB-load-misses
   1299290           +16.1%    1509095        perf-stat.ps.iTLB-loads
 2.425e+10 ±  4%     -16.7%   2.02e+10 ±  2%  perf-stat.ps.instructions
      2869 ±  2%      -4.8%       2731        perf-stat.ps.minor-faults
  50805618 ±  3%     +11.0%   56380675 ±  6%  perf-stat.ps.node-loads
  56364613 ±  2%     +15.5%   65091234 ±  3%  perf-stat.ps.node-stores
      2883 ±  2%      -4.8%       2745        perf-stat.ps.page-faults
 4.935e+12 ±  4%     -16.6%  4.115e+12 ±  2%  perf-stat.total.instructions
     55.92 ±  8%     -55.9        0.00        perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write
     56.10 ±  8%     -55.2        0.88 ± 11%  perf-profile.calltrace.cycles-pp.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write.ext4_buffered_write_iter
     56.13 ±  8%     -55.1        1.03 ±  9%  perf-profile.calltrace.cycles-pp.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write.ext4_buffered_write_iter.aio_write
     54.48 ±  8%     -54.5        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited
     54.47 ±  8%     -54.5        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.cgroup_rstat_flush_irqsafe.mem_cgroup_wb_stats.balance_dirty_pages
     56.29 ±  8%     -54.3        2.02 ± 39%  perf-profile.calltrace.cycles-pp.balance_dirty_pages_ratelimited.generic_perform_write.ext4_buffered_write_iter.aio_write.io_submit_one
     62.95 ±  8%     -39.1       23.82 ±  3%  perf-profile.calltrace.cycles-pp.generic_perform_write.ext4_buffered_write_iter.aio_write.io_submit_one.__x64_sys_io_submit
     62.97 ±  8%     -39.1       23.88 ±  3%  perf-profile.calltrace.cycles-pp.ext4_buffered_write_iter.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64
     62.97 ±  8%     -39.1       23.89 ±  3%  perf-profile.calltrace.cycles-pp.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe
     68.26 ±  8%     -24.6       43.68 ±  5%  perf-profile.calltrace.cycles-pp.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     68.26 ±  8%     -24.6       43.69 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     68.26 ±  8%     -24.5       43.72 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.syscall
     68.26 ±  8%     -24.5       43.72 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
     68.27 ±  8%     -24.5       43.74 ±  5%  perf-profile.calltrace.cycles-pp.syscall
      0.00            +0.8        0.83 ± 10%  perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.mem_cgroup_wb_stats.balance_dirty_pages
      0.53 ± 46%      +0.8        1.36 ± 57%  perf-profile.calltrace.cycles-pp.account_page_dirtied.__set_page_dirty.mark_buffer_dirty.__block_commit_write.generic_write_end
      0.00            +0.8        0.83 ± 10%  perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited
      0.00            +0.8        0.85 ±  9%  perf-profile.calltrace.cycles-pp.mem_cgroup_flush_stats.mem_cgroup_wb_stats.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write
      0.00            +0.9        0.86 ± 32%  perf-profile.calltrace.cycles-pp.__test_set_page_writeback.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
      0.00            +1.0        0.96 ± 22%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin
      0.71 ± 13%      +1.0        1.69 ± 42%  perf-profile.calltrace.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.generic_write_end.generic_perform_write
      0.51 ± 45%      +1.0        1.52 ± 54%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin
      0.00            +1.0        1.00 ± 14%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.alloc_buffer_head.alloc_page_buffers.create_empty_buffers.ext4_block_write_begin
      0.00            +1.0        1.03 ± 22%  perf-profile.calltrace.cycles-pp.__alloc_pages.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write
      0.00            +1.1        1.07 ± 15%  perf-profile.calltrace.cycles-pp.alloc_buffer_head.alloc_page_buffers.create_empty_buffers.ext4_block_write_begin.ext4_da_write_begin
      0.00            +1.1        1.11 ± 15%  perf-profile.calltrace.cycles-pp.alloc_page_buffers.create_empty_buffers.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write
      0.00            +1.2        1.20 ± 72%  perf-profile.calltrace.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin
      0.00            +1.2        1.24 ± 22%  perf-profile.calltrace.cycles-pp.invalidate_inode_page.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      0.00            +1.4        1.36 ± 65%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.page_cache_ra_unbounded.filemap_get_pages.filemap_read
      0.65 ± 10%      +1.4        2.02 ± 19%  perf-profile.calltrace.cycles-pp.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.aio_write
      0.09 ±223%      +1.4        1.47 ± 16%  perf-profile.calltrace.cycles-pp.create_empty_buffers.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter
      0.00            +1.4        1.39 ± 75%  perf-profile.calltrace.cycles-pp.__remove_mapping.remove_mapping.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64
      0.00            +1.4        1.40 ± 75%  perf-profile.calltrace.cycles-pp.remove_mapping.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      0.08 ±223%      +1.4        1.52 ± 17%  perf-profile.calltrace.cycles-pp.iov_iter_fault_in_readable.generic_perform_write.ext4_buffered_write_iter.aio_write.io_submit_one
      0.00            +1.5        1.46 ± 17%  perf-profile.calltrace.cycles-pp.__get_user_nocheck_1.iov_iter_fault_in_readable.generic_perform_write.ext4_buffered_write_iter.aio_write
      0.09 ±223%      +1.5        1.63 ± 16%  perf-profile.calltrace.cycles-pp.mark_page_accessed.filemap_read.aio_read.io_submit_one.__x64_sys_io_submit
      0.97 ± 11%      +1.7        2.68 ± 24%  perf-profile.calltrace.cycles-pp.mark_buffer_dirty.__block_commit_write.generic_write_end.generic_perform_write.ext4_buffered_write_iter
      0.47 ± 45%      +1.7        2.22 ± 67%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.page_cache_ra_unbounded.filemap_get_pages.filemap_read.aio_read
      0.87 ± 13%      +1.9        2.73 ± 60%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write
      0.10 ±223%      +2.0        2.12 ± 51%  perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.ext4_finish_bio.ext4_end_bio.pmem_submit_bio
      0.10 ±223%      +2.2        2.34 ± 45%  perf-profile.calltrace.cycles-pp.end_page_writeback.ext4_finish_bio.ext4_end_bio.pmem_submit_bio.__submit_bio
      0.98 ± 10%      +2.5        3.51 ± 17%  perf-profile.calltrace.cycles-pp.get_io_u
      1.40 ± 11%      +2.7        4.07 ±  7%  perf-profile.calltrace.cycles-pp.__block_commit_write.generic_write_end.generic_perform_write.ext4_buffered_write_iter.aio_write
      0.64 ± 19%      +2.8        3.41 ± 24%  perf-profile.calltrace.cycles-pp.ext4_end_bio.pmem_submit_bio.__submit_bio.__submit_bio_noacct.ext4_bio_write_page
      1.46 ± 10%      +2.8        4.24 ±  6%  perf-profile.calltrace.cycles-pp.generic_write_end.generic_perform_write.ext4_buffered_write_iter.aio_write.io_submit_one
      0.56 ± 48%      +2.8        3.39 ± 25%  perf-profile.calltrace.cycles-pp.ext4_finish_bio.ext4_end_bio.pmem_submit_bio.__submit_bio.__submit_bio_noacct
      1.33 ± 12%      +3.1        4.38 ± 30%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter
      1.34 ± 12%      +3.1        4.42 ± 30%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.aio_write
      0.00            +3.1        3.12 ±111%  perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64
      0.00            +3.1        3.13 ±111%  perf-profile.calltrace.cycles-pp.__pagevec_release.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      1.26 ±  9%      +3.9        5.14 ± 12%  perf-profile.calltrace.cycles-pp.copy_mc_fragile.pmem_do_read.pmem_submit_bio.__submit_bio.__submit_bio_noacct
      1.27 ±  8%      +3.9        5.18 ± 12%  perf-profile.calltrace.cycles-pp.pmem_do_read.pmem_submit_bio.__submit_bio.__submit_bio_noacct.ext4_mpage_readpages
      1.30 ±  9%      +4.0        5.34 ± 12%  perf-profile.calltrace.cycles-pp.pmem_submit_bio.__submit_bio.__submit_bio_noacct.ext4_mpage_readpages.read_pages
      1.31 ±  8%      +4.0        5.35 ± 12%  perf-profile.calltrace.cycles-pp.__submit_bio_noacct.ext4_mpage_readpages.read_pages.page_cache_ra_unbounded.filemap_get_pages
      1.31 ±  8%      +4.0        5.35 ± 12%  perf-profile.calltrace.cycles-pp.__submit_bio.__submit_bio_noacct.ext4_mpage_readpages.read_pages.page_cache_ra_unbounded
      1.37 ±  9%      +4.2        5.59 ± 13%  perf-profile.calltrace.cycles-pp.ext4_mpage_readpages.read_pages.page_cache_ra_unbounded.filemap_get_pages.filemap_read
      1.37 ±  9%      +4.2        5.60 ± 13%  perf-profile.calltrace.cycles-pp.read_pages.page_cache_ra_unbounded.filemap_get_pages.filemap_read.aio_read
      2.04 ± 11%      +4.5        6.57 ± 14%  perf-profile.calltrace.cycles-pp.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.aio_write.io_submit_one
      0.00            +5.7        5.73 ± 48%  perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.filemap_fdatawrite_wbc
      0.00            +6.1        6.06 ± 45%  perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
      0.00            +6.1        6.07 ± 45%  perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64
      0.00            +6.1        6.07 ± 45%  perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.generic_fadvise
      0.00            +6.1        6.07 ± 45%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64
      0.00            +6.1        6.07 ± 45%  perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      2.48 ± 11%      +6.2        8.65 ± 22%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.filemap_read.aio_read
      2.49 ± 11%      +6.2        8.70 ± 22%  perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.filemap_read.aio_read.io_submit_one
      0.00            +6.3        6.25 ± 75%  perf-profile.calltrace.cycles-pp.__invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64
      2.54 ± 10%      +6.3        8.86 ± 22%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.filemap_read.aio_read.io_submit_one.__x64_sys_io_submit
      2.09 ±  9%      +6.4        8.54 ± 11%  perf-profile.calltrace.cycles-pp.page_cache_ra_unbounded.filemap_get_pages.filemap_read.aio_read.io_submit_one
      2.60 ± 10%      +6.5        9.08 ± 21%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter_atomic.generic_perform_write.ext4_buffered_write_iter
      2.62 ± 10%      +6.5        9.14 ± 21%  perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter_atomic.generic_perform_write.ext4_buffered_write_iter.aio_write
      2.66 ± 10%      +6.6        9.25 ± 21%  perf-profile.calltrace.cycles-pp.copy_page_from_iter_atomic.generic_perform_write.ext4_buffered_write_iter.aio_write.io_submit_one
      2.76 ± 26%      +6.7        9.43 ± 32%  perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode
      2.20 ±  9%      +6.8        8.95 ± 10%  perf-profile.calltrace.cycles-pp.filemap_get_pages.filemap_read.aio_read.io_submit_one.__x64_sys_io_submit
      1.39 ± 33%      +6.9        8.27 ± 22%  perf-profile.calltrace.cycles-pp.__memcpy_flushcache.write_pmem.pmem_do_write.pmem_submit_bio.__submit_bio
      1.40 ± 33%      +6.9        8.32 ± 22%  perf-profile.calltrace.cycles-pp.write_pmem.pmem_do_write.pmem_submit_bio.__submit_bio.__submit_bio_noacct
      1.41 ± 33%      +7.0        8.36 ± 22%  perf-profile.calltrace.cycles-pp.pmem_do_write.pmem_submit_bio.__submit_bio.__submit_bio_noacct.ext4_bio_write_page
      2.93 ± 27%      +7.1       10.02 ± 32%  perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.calltrace.cycles-pp.wb_do_writeback.wb_workfn.process_one_work.worker_thread.kthread
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.calltrace.cycles-pp.wb_writeback.wb_do_writeback.wb_workfn.process_one_work.worker_thread
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_do_writeback.wb_workfn.process_one_work
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_do_writeback.wb_workfn
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_do_writeback
      2.96 ± 27%      +7.1       10.08 ± 32%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      2.96 ± 27%      +7.1       10.09 ± 32%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      3.00 ± 25%      +7.1       10.13 ± 32%  perf-profile.calltrace.cycles-pp.ret_from_fork
      3.00 ± 25%      +7.1       10.13 ± 32%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      2.07 ± 29%      +9.8       11.83 ± 13%  perf-profile.calltrace.cycles-pp.pmem_submit_bio.__submit_bio.__submit_bio_noacct.ext4_bio_write_page.mpage_submit_page
      2.07 ± 29%      +9.8       11.84 ± 13%  perf-profile.calltrace.cycles-pp.__submit_bio.__submit_bio_noacct.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs
      2.07 ± 29%      +9.8       11.84 ± 13%  perf-profile.calltrace.cycles-pp.__submit_bio_noacct.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
      2.45 ± 27%     +11.3       13.71 ± 11%  perf-profile.calltrace.cycles-pp.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages
      2.66 ± 27%     +12.0       14.66 ± 10%  perf-profile.calltrace.cycles-pp.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages
      0.00           +12.3       12.33 ± 58%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.posix_fadvise
      0.00           +12.3       12.33 ± 58%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      0.00           +12.3       12.33 ± 58%  perf-profile.calltrace.cycles-pp.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      0.00           +12.3       12.33 ± 58%  perf-profile.calltrace.cycles-pp.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      0.00           +12.3       12.33 ± 58%  perf-profile.calltrace.cycles-pp.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +12.3       12.33 ± 58%  perf-profile.calltrace.cycles-pp.posix_fadvise
      5.26 ± 10%     +14.5       19.74 ±  9%  perf-profile.calltrace.cycles-pp.filemap_read.aio_read.io_submit_one.__x64_sys_io_submit.do_syscall_64
      5.27 ±  9%     +14.5       19.76 ±  9%  perf-profile.calltrace.cycles-pp.aio_read.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe
     56.10 ±  8%     -55.2        0.88 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_wb_stats
     56.13 ±  8%     -55.1        1.03 ±  9%  perf-profile.children.cycles-pp.balance_dirty_pages
     55.92 ±  8%     -55.1        0.83 ± 10%  perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe
     56.29 ±  8%     -54.3        2.02 ± 39%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
     54.78 ±  8%     -51.7        3.06 ±134%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     55.13 ±  8%     -51.2        3.93 ±100%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     62.97 ±  8%     -39.1       23.86 ±  3%  perf-profile.children.cycles-pp.generic_perform_write
     62.97 ±  8%     -39.1       23.88 ±  3%  perf-profile.children.cycles-pp.ext4_buffered_write_iter
     62.97 ±  8%     -39.1       23.89 ±  3%  perf-profile.children.cycles-pp.aio_write
     68.26 ±  8%     -24.6       43.68 ±  5%  perf-profile.children.cycles-pp.io_submit_one
     68.26 ±  8%     -24.6       43.69 ±  5%  perf-profile.children.cycles-pp.__x64_sys_io_submit
     68.27 ±  8%     -24.5       43.75 ±  5%  perf-profile.children.cycles-pp.syscall
     68.43 ±  8%     -12.2       56.19 ± 10%  perf-profile.children.cycles-pp.do_syscall_64
     68.43 ±  8%     -12.2       56.19 ± 10%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.44 ±  9%      -0.6        0.83 ± 10%  perf-profile.children.cycles-pp.cgroup_rstat_flush_locked
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.__wake_up_common
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.wait_on_page_bit_common
      0.00            +0.1        0.06 ± 19%  perf-profile.children.cycles-pp.wake_up_page_bit
      0.04 ± 71%      +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.task_tick_fair
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.schedule
      0.00            +0.1        0.06 ± 19%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.xas_init_marks
      0.00            +0.1        0.07 ± 33%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.ext4_da_write_end
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
      0.00            +0.1        0.08 ± 16%  perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.xas_find_marked
      0.00            +0.1        0.09 ± 23%  perf-profile.children.cycles-pp.xas_create
      0.00            +0.1        0.09 ± 23%  perf-profile.children.cycles-pp.__cond_resched
      0.04 ± 45%      +0.1        0.14 ± 21%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.01 ±223%      +0.1        0.10 ± 23%  perf-profile.children.cycles-pp.node_page_state
      0.00            +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.__schedule
      0.00            +0.1        0.11 ± 25%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.00            +0.1        0.11 ± 24%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.00            +0.1        0.11 ± 87%  perf-profile.children.cycles-pp.unlock_page_memcg
      0.00            +0.1        0.12 ± 18%  perf-profile.children.cycles-pp.__read_end_io
      0.00            +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.00            +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.06 ± 13%      +0.1        0.18 ± 21%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.00            +0.1        0.12 ± 26%  perf-profile.children.cycles-pp.serial8250_console_write
      0.00            +0.1        0.12 ± 26%  perf-profile.children.cycles-pp.uart_console_write
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.irq_work_run
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp._printk
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.vprintk_emit
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.console_unlock
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.call_console_drivers
      0.00            +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.irq_work_run_list
      0.00            +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.00            +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.00            +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.00            +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.irq_work_single
      0.04 ± 71%      +0.1        0.16 ± 27%  perf-profile.children.cycles-pp.___might_sleep
      0.04 ± 71%      +0.1        0.16 ± 25%  perf-profile.children.cycles-pp.xa_load
      0.04 ± 45%      +0.1        0.17 ± 28%  perf-profile.children.cycles-pp.xa_get_order
      0.04 ± 79%      +0.1        0.17 ± 16%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.06 ± 13%      +0.1        0.20 ± 26%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.00            +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.__slab_free
      0.06 ± 14%      +0.1        0.21 ± 25%  perf-profile.children.cycles-pp.__xa_set_mark
      0.06 ±  8%      +0.1        0.20 ± 16%  perf-profile.children.cycles-pp.try_charge_memcg
      0.06 ± 11%      +0.2        0.21 ± 25%  perf-profile.children.cycles-pp.xas_start
      0.00            +0.2        0.16 ± 31%  perf-profile.children.cycles-pp.mod_objcg_state
      0.00            +0.2        0.16 ± 56%  perf-profile.children.cycles-pp.lock_page_memcg
      0.07 ±  9%      +0.2        0.24 ± 26%  perf-profile.children.cycles-pp.ext4_da_map_blocks
      0.00            +0.2        0.17 ± 19%  perf-profile.children.cycles-pp.xas_clear_mark
      0.06 ± 11%      +0.2        0.24 ± 26%  perf-profile.children.cycles-pp.page_mapping
      0.00            +0.2        0.19 ± 60%  perf-profile.children.cycles-pp.page_counter_cancel
      0.08 ± 14%      +0.2        0.28 ± 25%  perf-profile.children.cycles-pp.node_dirty_ok
      0.01 ±223%      +0.2        0.20 ± 22%  perf-profile.children.cycles-pp.unlock_page
      0.09 ± 10%      +0.2        0.28 ± 30%  perf-profile.children.cycles-pp.___slab_alloc
      0.01 ±223%      +0.2        0.21 ± 49%  perf-profile.children.cycles-pp.__fprop_inc_percpu
      0.03 ±223%      +0.2        0.24 ± 36%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.11 ± 14%      +0.2        0.32 ± 23%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.00            +0.2        0.21 ± 20%  perf-profile.children.cycles-pp.drop_buffers
      0.04 ± 75%      +0.2        0.26 ± 27%  perf-profile.children.cycles-pp.__xa_clear_mark
      0.00            +0.2        0.22 ± 29%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.07 ± 14%      +0.2        0.29 ± 25%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.2        0.23 ±106%  perf-profile.children.cycles-pp.mem_cgroup_wb_domain
      0.00            +0.2        0.23 ± 80%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.00            +0.3        0.26 ± 42%  perf-profile.children.cycles-pp.memcg_slab_free_hook
      0.10 ± 13%      +0.3        0.40 ± 21%  perf-profile.children.cycles-pp.filemap_get_read_batch
      0.14 ± 12%      +0.3        0.45 ± 27%  perf-profile.children.cycles-pp.ext4_da_get_block_prep
      0.02 ±223%      +0.3        0.34 ± 40%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.3        0.32 ± 16%  perf-profile.children.cycles-pp.__free_one_page
      0.00            +0.3        0.33 ± 97%  perf-profile.children.cycles-pp.uncharge_batch
      0.07 ± 17%      +0.4        0.42 ± 17%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.06 ± 13%      +0.4        0.42 ± 13%  perf-profile.children.cycles-pp.xas_store
      0.09 ± 12%      +0.4        0.46 ± 13%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.00            +0.4        0.38 ± 14%  perf-profile.children.cycles-pp.poll_idle
      0.00            +0.4        0.40 ± 22%  perf-profile.children.cycles-pp.jbd2_journal_grab_journal_head
      0.36 ± 14%      +0.4        0.76 ± 68%  perf-profile.children.cycles-pp.charge_memcg
      0.11 ± 13%      +0.4        0.51 ± 28%  perf-profile.children.cycles-pp.update_process_times
      0.00            +0.4        0.40 ± 17%  perf-profile.children.cycles-pp.find_lock_entries
      0.11 ± 14%      +0.4        0.52 ± 28%  perf-profile.children.cycles-pp.tick_sched_handle
      0.08 ±223%      +0.4        0.50 ± 39%  perf-profile.children.cycles-pp.menu_select
      0.13 ± 29%      +0.4        0.55 ± 27%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.00            +0.4        0.42 ± 22%  perf-profile.children.cycles-pp.jbd2_journal_try_to_free_buffers
      0.13 ± 29%      +0.4        0.55 ± 27%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.00            +0.4        0.43 ± 17%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.12 ± 13%      +0.4        0.55 ± 28%  perf-profile.children.cycles-pp.tick_sched_timer
      0.11 ± 14%      +0.4        0.54 ± 14%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.06 ± 21%      +0.4        0.51 ±113%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.31 ± 11%      +0.4        0.76 ± 15%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.00            +0.5        0.47 ± 28%  perf-profile.children.cycles-pp.free_buffer_head
      0.00            +0.5        0.49 ± 28%  perf-profile.children.cycles-pp.kmem_cache_free
      0.00            +0.5        0.49 ±107%  perf-profile.children.cycles-pp.__mem_cgroup_uncharge_list
      0.13 ± 18%      +0.5        0.67 ± 24%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.01 ±223%      +0.5        0.55 ± 16%  perf-profile.children.cycles-pp.free_unref_page_list
      0.16 ± 17%      +0.7        0.83 ± 17%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.22 ± 17%      +0.7        0.92 ± 43%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      0.34 ± 10%      +0.7        1.07 ± 15%  perf-profile.children.cycles-pp.alloc_buffer_head
      0.35 ± 10%      +0.7        1.08 ± 15%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.01 ±223%      +0.7        0.75 ± 24%  perf-profile.children.cycles-pp.try_to_free_buffers
      0.36 ±  9%      +0.8        1.11 ± 15%  perf-profile.children.cycles-pp.alloc_page_buffers
      0.61 ± 14%      +0.8        1.37 ± 57%  perf-profile.children.cycles-pp.account_page_dirtied
      0.25 ± 15%      +0.8        1.09 ± 18%  perf-profile.children.cycles-pp.rmqueue
      0.00            +0.8        0.85 ±  9%  perf-profile.children.cycles-pp.mem_cgroup_flush_stats
      0.01 ±223%      +0.9        0.87 ±121%  perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.31 ± 15%      +0.9        1.20 ± 24%  perf-profile.children.cycles-pp.xas_load
      0.28 ± 19%      +0.9        1.20 ± 26%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.71 ± 13%      +1.0        1.70 ± 41%  perf-profile.children.cycles-pp.__set_page_dirty
      0.48 ±  9%      +1.0        1.48 ± 16%  perf-profile.children.cycles-pp.create_empty_buffers
      0.44 ± 12%      +1.0        1.48 ± 96%  perf-profile.children.cycles-pp.__mem_cgroup_charge
      0.42 ±  9%      +1.1        1.49 ± 17%  perf-profile.children.cycles-pp.__get_user_nocheck_1
      0.44 ±  9%      +1.1        1.53 ± 17%  perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      0.38 ± 13%      +1.1        1.49 ± 19%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.44 ± 11%      +1.2        1.63 ± 16%  perf-profile.children.cycles-pp.mark_page_accessed
      0.02 ±223%      +1.2        1.22 ± 85%  perf-profile.children.cycles-pp.__delete_from_page_cache
      0.42 ± 13%      +1.2        1.63 ± 19%  perf-profile.children.cycles-pp.__alloc_pages
      0.02 ±223%      +1.2        1.25 ± 23%  perf-profile.children.cycles-pp.invalidate_inode_page
      0.66 ±  9%      +1.4        2.03 ± 19%  perf-profile.children.cycles-pp.ext4_block_write_begin
      0.02 ±223%      +1.4        1.39 ± 75%  perf-profile.children.cycles-pp.__remove_mapping
      0.02 ±223%      +1.4        1.40 ± 74%  perf-profile.children.cycles-pp.remove_mapping
      0.42 ± 11%      +1.5        1.94 ± 77%  perf-profile.children.cycles-pp.__pagevec_lru_add
      0.46 ± 11%      +1.6        2.04 ± 72%  perf-profile.children.cycles-pp.lru_cache_add
      0.97 ± 11%      +1.7        2.68 ± 24%  perf-profile.children.cycles-pp.mark_buffer_dirty
      0.44 ± 17%      +1.9        2.29 ± 43%  perf-profile.children.cycles-pp.test_clear_page_writeback
      0.95 ± 11%      +1.9        2.88 ± 58%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      0.47 ± 17%      +2.0        2.43 ± 40%  perf-profile.children.cycles-pp.end_page_writeback
      1.09 ± 14%      +2.0        3.13 ±102%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.08 ± 16%      +2.5        2.55 ±158%  perf-profile.children.cycles-pp.lock_page_lruvec_irqsave
      0.98 ± 10%      +2.5        3.52 ± 17%  perf-profile.children.cycles-pp.get_io_u
      1.40 ± 11%      +2.7        4.07 ±  7%  perf-profile.children.cycles-pp.__block_commit_write
      0.66 ± 19%      +2.8        3.41 ± 24%  perf-profile.children.cycles-pp.ext4_finish_bio
      0.66 ± 19%      +2.8        3.42 ± 24%  perf-profile.children.cycles-pp.ext4_end_bio
      1.46 ± 10%      +2.8        4.24 ±  6%  perf-profile.children.cycles-pp.generic_write_end
      1.23 ± 12%      +3.0        4.18 ± 86%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      1.33 ± 12%      +3.1        4.39 ± 30%  perf-profile.children.cycles-pp.pagecache_get_page
      1.34 ± 12%      +3.1        4.42 ± 30%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      0.03 ±161%      +3.2        3.21 ±107%  perf-profile.children.cycles-pp.__pagevec_release
      0.04 ±120%      +3.2        3.30 ±104%  perf-profile.children.cycles-pp.release_pages
      1.42 ± 11%      +3.5        4.95 ± 63%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      1.26 ±  8%      +3.9        5.14 ± 12%  perf-profile.children.cycles-pp.copy_mc_fragile
      1.27 ±  8%      +3.9        5.18 ± 12%  perf-profile.children.cycles-pp.pmem_do_read
      1.37 ±  9%      +4.2        5.60 ± 13%  perf-profile.children.cycles-pp.ext4_mpage_readpages
      1.37 ±  9%      +4.2        5.60 ± 13%  perf-profile.children.cycles-pp.read_pages
      2.04 ± 11%      +4.5        6.58 ± 14%  perf-profile.children.cycles-pp.ext4_da_write_begin
      0.03 ±157%      +6.0        6.07 ± 45%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.03 ±157%      +6.0        6.07 ± 45%  perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
      0.08 ±162%      +6.2        6.25 ± 75%  perf-profile.children.cycles-pp.__invalidate_mapping_pages
      2.49 ± 11%      +6.2        8.70 ± 22%  perf-profile.children.cycles-pp.copyout
      2.54 ± 11%      +6.3        8.86 ± 22%  perf-profile.children.cycles-pp.copy_page_to_iter
      2.10 ±  9%      +6.4        8.54 ± 11%  perf-profile.children.cycles-pp.page_cache_ra_unbounded
      2.62 ± 10%      +6.5        9.14 ± 21%  perf-profile.children.cycles-pp.copyin
      2.66 ± 10%      +6.6        9.26 ± 21%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      2.20 ±  9%      +6.8        8.95 ± 10%  perf-profile.children.cycles-pp.filemap_get_pages
      1.42 ± 32%      +6.9        8.30 ± 22%  perf-profile.children.cycles-pp.__memcpy_flushcache
      1.43 ± 32%      +6.9        8.35 ± 22%  perf-profile.children.cycles-pp.write_pmem
      1.44 ± 32%      +6.9        8.38 ± 22%  perf-profile.children.cycles-pp.pmem_do_write
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.children.cycles-pp.wb_workfn
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.children.cycles-pp.wb_do_writeback
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.children.cycles-pp.wb_writeback
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.children.cycles-pp.__writeback_inodes_wb
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.children.cycles-pp.writeback_sb_inodes
      2.95 ± 27%      +7.1       10.06 ± 32%  perf-profile.children.cycles-pp.__writeback_single_inode
      2.96 ± 27%      +7.1       10.08 ± 32%  perf-profile.children.cycles-pp.process_one_work
      2.96 ± 27%      +7.1       10.09 ± 32%  perf-profile.children.cycles-pp.worker_thread
      3.00 ± 25%      +7.1       10.13 ± 32%  perf-profile.children.cycles-pp.ret_from_fork
      3.00 ± 25%      +7.1       10.13 ± 32%  perf-profile.children.cycles-pp.kthread
      2.47 ± 26%     +11.2       13.72 ± 11%  perf-profile.children.cycles-pp.ext4_bio_write_page
      2.69 ± 26%     +12.0       14.66 ± 10%  perf-profile.children.cycles-pp.mpage_submit_page
      0.11 ±160%     +12.2       12.33 ± 58%  perf-profile.children.cycles-pp.__x64_sys_fadvise64
      0.11 ±160%     +12.2       12.33 ± 58%  perf-profile.children.cycles-pp.ksys_fadvise64_64
      0.11 ±160%     +12.2       12.33 ± 58%  perf-profile.children.cycles-pp.generic_fadvise
      0.11 ±160%     +12.2       12.33 ± 58%  perf-profile.children.cycles-pp.posix_fadvise
      2.79 ± 25%     +12.4       15.16 ± 10%  perf-profile.children.cycles-pp.mpage_process_page_bufs
      5.11 ± 10%     +12.7       17.83 ± 21%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      2.97 ± 25%     +13.1       16.10 ± 10%  perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
      2.98 ± 25%     +13.1       16.13 ± 10%  perf-profile.children.cycles-pp.do_writepages
      2.98 ± 25%     +13.1       16.13 ± 10%  perf-profile.children.cycles-pp.ext4_writepages
      3.42 ± 20%     +13.8       17.21 ± 11%  perf-profile.children.cycles-pp.pmem_submit_bio
      3.42 ± 20%     +13.8       17.22 ± 11%  perf-profile.children.cycles-pp.__submit_bio_noacct
      3.42 ± 20%     +13.8       17.22 ± 11%  perf-profile.children.cycles-pp.__submit_bio
      5.26 ±  9%     +14.5       19.75 ±  9%  perf-profile.children.cycles-pp.filemap_read
      5.27 ±  9%     +14.5       19.76 ±  9%  perf-profile.children.cycles-pp.aio_read
     54.78 ±  8%     -51.7        3.06 ±134%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.73 ±  4%      -0.3        0.44 ± 19%  perf-profile.self.cycles-pp._raw_spin_lock
      0.01 ±223%      +0.1        0.07 ± 15%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.08 ± 24%  perf-profile.self.cycles-pp.ext4_da_write_end
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.__list_add_valid
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.mem_cgroup_update_lru_size
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.account_page_dirtied
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.__read_end_io
      0.00            +0.1        0.08 ± 27%  perf-profile.self.cycles-pp.xas_find_marked
      0.00            +0.1        0.09 ± 18%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.00            +0.1        0.09 ± 21%  perf-profile.self.cycles-pp.ext4_da_write_begin
      0.00            +0.1        0.09 ± 18%  perf-profile.self.cycles-pp.try_charge_memcg
      0.00            +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.node_page_state
      0.00            +0.1        0.10 ± 16%  perf-profile.self.cycles-pp.page_counter_try_charge
      0.00            +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.__remove_mapping
      0.00            +0.1        0.10 ± 22%  perf-profile.self.cycles-pp.lru_cache_add
      0.00            +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.00            +0.1        0.10 ± 17%  perf-profile.self.cycles-pp.mod_objcg_state
      0.00            +0.1        0.10 ± 93%  perf-profile.self.cycles-pp.unlock_page_memcg
      0.01 ±223%      +0.1        0.11 ± 30%  perf-profile.self.cycles-pp.generic_write_end
      0.00            +0.1        0.11 ± 27%  perf-profile.self.cycles-pp.ext4_block_write_begin
      0.02 ±141%      +0.1        0.12 ± 30%  perf-profile.self.cycles-pp.ext4_da_get_block_prep
      0.06 ± 13%      +0.1        0.17 ± 24%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.04 ± 71%      +0.1        0.15 ± 24%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.01 ±223%      +0.1        0.12 ± 28%  perf-profile.self.cycles-pp.copy_page_from_iter_atomic
      0.06 ± 13%      +0.1        0.17 ± 28%  perf-profile.self.cycles-pp.rmqueue
      0.04 ± 71%      +0.1        0.15 ± 85%  perf-profile.self.cycles-pp.__count_memcg_events
      0.04 ± 71%      +0.1        0.16 ± 26%  perf-profile.self.cycles-pp.___might_sleep
      0.00            +0.1        0.12 ± 31%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.06 ± 11%      +0.1        0.19 ± 26%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.02 ±141%      +0.1        0.14 ± 24%  perf-profile.self.cycles-pp.ext4_mpage_readpages
      0.02 ±146%      +0.1        0.15 ± 18%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.1        0.13 ± 31%  perf-profile.self.cycles-pp.end_page_writeback
      0.05 ± 45%      +0.1        0.18 ± 27%  perf-profile.self.cycles-pp.create_empty_buffers
      0.05 ± 45%      +0.1        0.18 ± 28%  perf-profile.self.cycles-pp.node_dirty_ok
      0.00            +0.1        0.14 ± 19%  perf-profile.self.cycles-pp.__slab_free
      0.00            +0.1        0.14 ± 25%  perf-profile.self.cycles-pp.memcg_slab_free_hook
      0.05 ± 45%      +0.1        0.19 ± 25%  perf-profile.self.cycles-pp.xas_start
      0.00            +0.2        0.16 ± 58%  perf-profile.self.cycles-pp.lock_page_memcg
      0.00            +0.2        0.16 ± 19%  perf-profile.self.cycles-pp.xas_clear_mark
      0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.xas_store
      0.03 ±100%      +0.2        0.20 ± 26%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.06 ± 13%      +0.2        0.23 ± 26%  perf-profile.self.cycles-pp.page_mapping
      0.00            +0.2        0.18 ± 30%  perf-profile.self.cycles-pp.mpage_prepare_extent_to_map
      0.01 ±223%      +0.2        0.19 ± 21%  perf-profile.self.cycles-pp.unlock_page
      0.00            +0.2        0.19 ± 60%  perf-profile.self.cycles-pp.page_counter_cancel
      0.10 ± 14%      +0.2        0.29 ± 24%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.08 ± 12%      +0.2        0.28 ± 21%  perf-profile.self.cycles-pp.filemap_read
      0.04 ± 75%      +0.2        0.24 ± 28%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.00            +0.2        0.21 ± 21%  perf-profile.self.cycles-pp.drop_buffers
      0.11 ± 12%      +0.2        0.32 ± 16%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.07 ± 18%      +0.2        0.29 ± 22%  perf-profile.self.cycles-pp.pagecache_get_page
      0.00            +0.2        0.23 ±106%  perf-profile.self.cycles-pp.mem_cgroup_wb_domain
      0.08 ± 16%      +0.3        0.34 ± 22%  perf-profile.self.cycles-pp.filemap_get_read_batch
      0.07 ± 29%      +0.3        0.33 ± 29%  perf-profile.self.cycles-pp.ext4_bio_write_page
      0.00            +0.3        0.27 ±135%  perf-profile.self.cycles-pp.charge_memcg
      0.06 ± 15%      +0.3        0.33 ± 20%  perf-profile.self.cycles-pp.test_clear_page_writeback
      0.02 ±223%      +0.3        0.29 ± 36%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.3        0.28 ± 14%  perf-profile.self.cycles-pp.__free_one_page
      0.08 ± 34%      +0.3        0.40 ± 28%  perf-profile.self.cycles-pp.ext4_finish_bio
      0.02 ±141%      +0.3        0.34 ± 15%  perf-profile.self.cycles-pp.release_pages
      0.07 ± 17%      +0.3        0.42 ± 17%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.09 ± 12%      +0.3        0.44 ± 13%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.11 ± 33%      +0.4        0.46 ± 27%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.00            +0.4        0.36 ± 17%  perf-profile.self.cycles-pp.find_lock_entries
      0.00            +0.4        0.37 ± 14%  perf-profile.self.cycles-pp.poll_idle
      0.00            +0.4        0.40 ± 22%  perf-profile.self.cycles-pp.jbd2_journal_grab_journal_head
      0.10 ± 18%      +0.4        0.50 ± 15%  perf-profile.self.cycles-pp.mpage_process_page_bufs
      0.09 ± 23%      +0.4        0.49 ± 27%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.06 ± 19%      +0.4        0.49 ±118%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.20 ±  8%      +0.6        0.77 ± 21%  perf-profile.self.cycles-pp.mark_buffer_dirty
      0.26 ± 15%      +0.8        1.01 ± 24%  perf-profile.self.cycles-pp.xas_load
      0.14 ±  6%      +0.8        0.94 ± 76%  perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
      0.36 ± 10%      +0.8        1.18 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.40 ± 14%      +0.9        1.29 ± 25%  perf-profile.self.cycles-pp.__block_commit_write
      0.19 ±  8%      +0.9        1.12 ± 84%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.42 ± 10%      +1.1        1.47 ± 17%  perf-profile.self.cycles-pp.__get_user_nocheck_1
      0.44 ± 10%      +1.2        1.62 ± 15%  perf-profile.self.cycles-pp.mark_page_accessed
      0.29 ±  7%      +2.0        2.32 ±120%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.97 ±  9%      +2.5        3.47 ± 17%  perf-profile.self.cycles-pp.get_io_u
      1.25 ±  8%      +3.8        5.09 ± 12%  perf-profile.self.cycles-pp.copy_mc_fragile
      1.41 ± 32%      +6.8        8.24 ± 22%  perf-profile.self.cycles-pp.__memcpy_flushcache
      5.08 ± 10%     +12.6       17.68 ± 21%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://01.org/lkp



View attachment "config-5.15.0-00056-gfd25a9e0e23b" of type "text/plain" (159178 bytes)

View attachment "job-script" of type "text/plain" (8628 bytes)

View attachment "job.yaml" of type "text/plain" (6014 bytes)

View attachment "reproduce" of type "text/plain" (914 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ