lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Fri, 28 Feb 2020 15:42:00 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     js1304@...il.com
Cc:     Andrew Morton <akpm@...ux-foundation.org>, linux-mm@...ck.org,
        linux-kernel@...r.kernel.org, Johannes Weiner <hannes@...xchg.org>,
        Michal Hocko <mhocko@...nel.org>,
        Hugh Dickins <hughd@...gle.com>,
        Minchan Kim <minchan@...nel.org>,
        Vlastimil Babka <vbabka@...e.cz>,
        Mel Gorman <mgorman@...hsingularity.net>, kernel-team@....com,
        Joonsoo Kim <iamjoonsoo.kim@....com>, lkp@...ts.01.org
Subject: [mm/workingset] 323c95f095: fio.read_bw_MBps 19.5% improvement

Greeting,

FYI, we noticed a 19.5% improvement of fio.read_bw_MBps due to commit:


commit: 323c95f0957c8a7d59cd9d05f43f19dfc9bbc584 ("[PATCH 3/9] mm/workingset: extend the workingset detection for anon LRU")
url: https://github.com/0day-ci/linux/commits/js1304-gmail-com/workingset-protection-detection-on-the-anonymous-LRU-list/20200213-111547


in testcase: fio-basic
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	disk: 2pmem
	fs: ext4
	runtime: 200s
	nr_task: 50%
	time_based: tb
	rw: randrw
	bs: 2M
	ioengine: mmap
	test_size: 200G
	cpufreq_governor: performance
	ucode: 0x500002c

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  2M/gcc-7/performance/2pmem/ext4/mmap/x86_64-rhel-7.6/50%/debian-x86_64-20191114.cgz/200s/randrw/lkp-csl-2sp6/200G/fio-basic/tb/0x500002c

commit: 
  fffae40d0e ("mm/vmscan: protect the workingset on anonymous LRU")
  323c95f095 ("mm/workingset: extend the workingset detection for anon LRU")

fffae40d0ee3fd1e 323c95f0957c8a7d59cd9d05f43 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at_ip___perf_sw_event/0x
           :4           25%           1:4     dmesg.WARNING:at_ip_generic_make_request/0x
          2:4           45%           4:4     perf-profile.calltrace.cycles-pp.sync_regs.error_entry
          2:4           50%           4:4     perf-profile.calltrace.cycles-pp.error_entry
          2:4           53%           4:4     perf-profile.children.cycles-pp.error_entry
          0:4            6%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      0.28 ± 11%      -0.1        0.16 ± 28%  fio.latency_100ms%
      8.36 ±  8%      -5.9        2.50 ±  5%  fio.latency_10ms%
      6.10 ± 17%      -5.3        0.82 ±  6%  fio.latency_20ms%
      0.00 ±173%      +0.0        0.01 ± 10%  fio.latency_250ms%
      0.71 ± 16%      +2.8        3.56 ±  4%  fio.latency_250us%
     13.45 ±  5%      +9.4       22.85 ±  2%  fio.latency_2ms%
     29.90 ±  2%      -7.3       22.58 ±  2%  fio.latency_4ms%
      8.43 ±  7%      +9.4       17.84 ±  4%  fio.latency_500us%
      7.43 ±  3%      -3.3        4.13 ± 12%  fio.latency_750us%
      4336 ±  2%     +19.5%       5182 ±  2%  fio.read_bw_MBps
   7430144 ± 16%     -64.3%    2654208        fio.read_clat_90%_us
  11501568 ±  9%     -70.8%    3358720        fio.read_clat_95%_us
  17170432 ±  8%     -57.7%    7258112 ±  4%  fio.read_clat_99%_us
   2960419 ±  7%     -53.3%    1381107        fio.read_clat_mean_us
   4105907 ± 14%     -45.6%    2232352 ± 11%  fio.read_clat_stddev
      2168 ±  2%     +19.5%       2591 ±  2%  fio.read_iops
 1.942e+09 ±  2%     -22.0%  1.514e+09 ±  2%  fio.time.file_system_inputs
 1.783e+09 ±  2%     +19.5%  2.131e+09 ±  2%  fio.time.file_system_outputs
    216950 ± 12%     -48.8%     111051 ± 16%  fio.time.involuntary_context_switches
 2.427e+08 ±  2%     -22.0%  1.892e+08 ±  2%  fio.time.major_page_faults
 1.271e+08 ±  5%     +66.8%   2.12e+08 ±  2%  fio.time.minor_page_faults
      1619 ±  9%     -33.6%       1074 ±  2%  fio.time.percent_of_cpu_this_job_got
      2851 ±  9%     -40.1%       1708 ±  2%  fio.time.system_time
    404.54 ±  6%     +11.4%     450.81 ±  2%  fio.time.user_time
    869238 ±  2%     +19.5%    1038826 ±  2%  fio.workload
      4350 ±  2%     +19.5%       5199 ±  2%  fio.write_bw_MBps
  38404096           -12.9%   33456128        fio.write_clat_90%_us
  39452672           -14.0%   33947648        fio.write_clat_95%_us
  45744128           -19.5%   36831232 ±  3%  fio.write_clat_99%_us
  17993869 ±  4%      -9.2%   16330675 ±  2%  fio.write_clat_mean_us
  16656042 ±  5%      -6.2%   15615331        fio.write_clat_stddev
      2175 ±  2%     +19.5%       2599 ±  2%  fio.write_iops
     31.58 ±  4%      +5.8       37.39        mpstat.cpu.all.iowait%
     18.54 ±  7%      -6.5       12.04 ±  2%  mpstat.cpu.all.sys%
 1.008e+08 ± 10%     -37.9%   62633861 ± 18%  numa-numastat.node0.local_node
 1.008e+08 ±  9%     -37.9%   62658517 ± 18%  numa-numastat.node0.numa_hit
     48.29            +1.1%      48.80        iostat.cpu.idle
     31.27 ±  4%     +18.4%      37.03        iostat.cpu.iowait
     18.38 ±  7%     -35.0%      11.94 ±  2%  iostat.cpu.system
    618.00 ±  6%     -27.0%     451.25        turbostat.Avg_MHz
     22.14 ±  6%      -6.0       16.17        turbostat.Busy%
 1.361e+08 ±  7%     -26.6%   99961074 ±  3%  turbostat.IRQ
    224.76            -4.7%     214.16        turbostat.PkgWatt
   4580495           +85.7%    8505839        slabinfo.buffer_head.active_objs
    119518           +82.7%     218335        slabinfo.buffer_head.active_slabs
   4661221           +82.7%    8515085        slabinfo.buffer_head.num_objs
    119518           +82.7%     218335        slabinfo.buffer_head.num_slabs
     19148 ±  3%      +9.2%      20917 ±  3%  slabinfo.kmalloc-512.num_objs
     30.75 ±  4%     +19.5%      36.75        vmstat.cpu.wa
   4753706 ±  2%     -21.9%    3710468 ±  2%  vmstat.io.bi
   4361808 ±  2%     +19.7%    5220006 ±  2%  vmstat.io.bo
    619.75 ±  8%     +86.0%       1153 ±  5%  vmstat.memory.buff
     31.25 ±  4%     +15.2%      36.00        vmstat.procs.b
     19.00 ±  9%     -28.9%      13.50 ±  3%  vmstat.procs.r
    409038 ±  5%     -18.7%     332463 ±  2%  vmstat.system.in
   9785202          +221.1%   31421653        meminfo.Active
     60190 ±  8%     -34.3%      39569 ±  3%  meminfo.Active(anon)
   9725011          +222.7%   31382083        meminfo.Active(file)
    619.25 ±  7%     +86.7%       1156 ±  5%  meminfo.Buffers
  32198569           -68.1%   10269890        meminfo.Inactive
  28407610           -77.2%    6464247        meminfo.Inactive(file)
    650015           +60.3%    1041779        meminfo.KReclaimable
    650015           +60.3%    1041779        meminfo.SReclaimable
    829409           +47.3%    1221631        meminfo.Slab
      2564 ±  4%     +42.5%       3654 ±  4%  meminfo.Writeback
   4673366 ±  9%    +204.3%   14223125        numa-meminfo.node0.Active
   4670584 ±  9%    +204.4%   14218202        numa-meminfo.node0.Active(file)
   3328794 ±  5%     -10.7%    2972288        numa-meminfo.node0.Dirty
  16103425 ±  3%     -60.3%    6392546 ±  2%  numa-meminfo.node0.Inactive
  13636450 ±  5%     -77.2%    3114750 ±  3%  numa-meminfo.node0.Inactive(file)
    361437 ±  2%     +64.5%     594674 ±  2%  numa-meminfo.node0.KReclaimable
    361437 ±  2%     +64.5%     594674 ±  2%  numa-meminfo.node0.SReclaimable
    454217           +51.0%     686089 ±  2%  numa-meminfo.node0.Slab
   5117666 ±  8%    +236.1%   17201387        numa-meminfo.node1.Active
     57417 ±  8%     -39.6%      34652 ±  7%  numa-meminfo.node1.Active(anon)
   5060248 ±  8%    +239.2%   17166734        numa-meminfo.node1.Active(file)
  16086677 ±  3%     -76.0%    3867738 ±  5%  numa-meminfo.node1.Inactive
  14767296 ±  5%     -77.4%    3343591 ±  2%  numa-meminfo.node1.Inactive(file)
    288589 ±  3%     +54.9%     446992 ±  3%  numa-meminfo.node1.KReclaimable
    288589 ±  3%     +54.9%     446992 ±  3%  numa-meminfo.node1.SReclaimable
    375202 ±  2%     +42.7%     535430 ±  2%  numa-meminfo.node1.Slab
      1310 ±  9%     +57.4%       2062 ±  5%  numa-meminfo.node1.Writeback
      0.02 ± 38%    +151.1%       0.04 ±  4%  sched_debug.cfs_rq:/.nr_spread_over.avg
      0.14 ± 30%     +62.3%       0.22 ± 11%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    702.00 ±  3%     +20.3%     844.48 ± 11%  sched_debug.cfs_rq:/.runnable_load_avg.max
      8658 ± 41%     -88.5%     994.62 ±366%  sched_debug.cfs_rq:/.spread0.avg
     -5693          +147.0%     -14059        sched_debug.cfs_rq:/.spread0.min
    280.56 ± 11%     -20.9%     221.88 ±  5%  sched_debug.cfs_rq:/.util_avg.avg
     16889 ±  8%     +31.2%      22166 ± 13%  sched_debug.cpu.nr_switches.avg
      0.21 ± 23%     +44.0%       0.30 ±  7%  sched_debug.cpu.nr_uninterruptible.avg
     15025 ±  9%     +35.5%      20353 ± 14%  sched_debug.cpu.sched_count.avg
     16991 ± 21%     +31.0%      22263 ± 16%  sched_debug.cpu.sched_count.stddev
      6755 ± 11%     +39.0%       9388 ± 14%  sched_debug.cpu.sched_goidle.avg
    478.75 ± 16%     +37.8%     659.79 ± 19%  sched_debug.cpu.sched_goidle.min
      8464 ± 21%     +31.2%      11108 ± 16%  sched_debug.cpu.sched_goidle.stddev
      7465 ±  9%     +35.5%      10114 ± 14%  sched_debug.cpu.ttwu_count.avg
      3184 ± 15%     +79.5%       5715 ± 14%  sched_debug.cpu.ttwu_local.max
    349.50 ± 15%     +32.8%     464.02 ± 16%  sched_debug.cpu.ttwu_local.min
    516.10 ±  9%     +58.8%     819.50 ± 16%  sched_debug.cpu.ttwu_local.stddev
    135179           +16.5%     157474 ±  8%  sched_debug.ktime
     20801 ±  9%     +15.2%      23973 ±  3%  softirqs.CPU13.SCHED
     21834 ±  5%     +11.4%      24317 ±  4%  softirqs.CPU15.SCHED
     20068 ± 13%     +26.6%      25416 ±  2%  softirqs.CPU18.SCHED
     21227 ±  8%     +18.3%      25107 ±  2%  softirqs.CPU19.SCHED
     21493 ±  6%     +14.2%      24542 ±  4%  softirqs.CPU21.SCHED
     21659 ±  5%     +17.5%      25448 ±  3%  softirqs.CPU22.SCHED
     21989 ±  4%     +17.2%      25773 ±  2%  softirqs.CPU23.SCHED
     20132 ± 14%     +23.2%      24812 ±  2%  softirqs.CPU25.SCHED
     20294 ± 16%     +23.7%      25112 ±  5%  softirqs.CPU26.SCHED
     21996 ±  2%     +14.2%      25115 ±  2%  softirqs.CPU3.SCHED
     21985 ±  8%     +17.2%      25765 ±  3%  softirqs.CPU31.SCHED
     21974 ±  6%     +14.6%      25184        softirqs.CPU37.SCHED
     21943           +12.4%      24665 ±  3%  softirqs.CPU40.SCHED
     21177 ±  9%     +17.8%      24954 ±  2%  softirqs.CPU41.SCHED
     21555 ±  4%     +16.1%      25018        softirqs.CPU43.SCHED
     21565 ±  2%     +15.0%      24804        softirqs.CPU45.SCHED
     22158 ±  6%     +14.9%      25466 ±  3%  softirqs.CPU50.SCHED
     22571 ±  7%     +13.4%      25597        softirqs.CPU51.SCHED
     21477 ±  7%     +17.8%      25310        softirqs.CPU53.SCHED
     21648 ±  4%     +15.7%      25037 ±  2%  softirqs.CPU54.SCHED
     22303 ±  4%     +10.0%      24524 ±  5%  softirqs.CPU57.SCHED
     22250 ±  8%     +13.7%      25302 ±  2%  softirqs.CPU6.SCHED
     21945 ±  6%     +15.1%      25253        softirqs.CPU62.SCHED
     20860 ±  9%     +23.0%      25655 ±  3%  softirqs.CPU64.SCHED
     22491 ±  4%      +7.5%      24180 ±  5%  softirqs.CPU81.SCHED
     19570 ± 21%     +28.5%      25157 ±  2%  softirqs.CPU83.SCHED
   1167199 ±  9%    +204.5%    3554443        numa-vmstat.node0.nr_active_file
    833733 ±  5%     -11.3%     739589        numa-vmstat.node0.nr_dirty
   3409188 ±  5%     -77.1%     779009 ±  3%  numa-vmstat.node0.nr_inactive_file
     83.50 ± 22%     -60.8%      32.75 ± 16%  numa-vmstat.node0.nr_isolated_file
     90376 ±  2%     +64.5%     148639 ±  2%  numa-vmstat.node0.nr_slab_reclaimable
   1394052 ±  7%    +282.6%    5333857 ± 17%  numa-vmstat.node0.nr_vmscan_immediate_reclaim
    315.25 ±  3%     +22.5%     386.25 ± 13%  numa-vmstat.node0.nr_writeback
   1167965 ±  9%    +204.3%    3554646        numa-vmstat.node0.nr_zone_active_file
   3409621 ±  5%     -77.1%     780149 ±  3%  numa-vmstat.node0.nr_zone_inactive_file
    835951 ±  5%     -11.3%     741848        numa-vmstat.node0.nr_zone_write_pending
  62116218 ±  6%     -31.7%   42426214 ± 23%  numa-vmstat.node0.numa_hit
  62089335 ±  6%     -31.8%   42365222 ± 23%  numa-vmstat.node0.numa_local
  18221559 ± 10%    -100.0%       0.00        numa-vmstat.node0.workingset_activate
     13281 ±  7%     -22.3%      10326 ±  8%  numa-vmstat.node0.workingset_nodes
  33316100 ±  6%    -100.0%       0.00        numa-vmstat.node0.workingset_refault
  10372861 ± 12%    -100.0%       1.25 ± 66%  numa-vmstat.node0.workingset_restore
     14346 ±  8%     -39.6%       8669 ±  7%  numa-vmstat.node1.nr_active_anon
   1263972 ±  8%    +239.6%    4292221        numa-vmstat.node1.nr_active_file
  61372170 ±  9%     +31.3%   80554020 ±  8%  numa-vmstat.node1.nr_dirtied
   3692694 ±  5%     -77.4%     835792        numa-vmstat.node1.nr_inactive_file
     72150 ±  3%     +54.8%     111693 ±  3%  numa-vmstat.node1.nr_slab_reclaimable
   1341628 ± 12%    +750.8%   11415031 ±  6%  numa-vmstat.node1.nr_vmscan_immediate_reclaim
    331.25 ± 10%     +66.0%     549.75 ±  9%  numa-vmstat.node1.nr_writeback
  60469655 ±  9%     +31.7%   79613669 ±  8%  numa-vmstat.node1.nr_written
     14347 ±  8%     -39.6%       8670 ±  7%  numa-vmstat.node1.nr_zone_active_anon
   1264780 ±  8%    +239.4%    4292348        numa-vmstat.node1.nr_zone_active_file
   3693150 ±  5%     -77.3%     837009        numa-vmstat.node1.nr_zone_inactive_file
  19519796 ±  8%    -100.0%       0.00        numa-vmstat.node1.workingset_activate
     12632 ±  5%     -20.7%      10012 ±  4%  numa-vmstat.node1.workingset_nodes
  36195648 ±  4%    -100.0%       0.00        numa-vmstat.node1.workingset_refault
  11279550 ±  9%    -100.0%       0.25 ±173%  numa-vmstat.node1.workingset_restore
     15.72           +11.1%      17.46        perf-stat.i.MPKI
 7.221e+09 ±  3%     -14.9%  6.144e+09 ±  2%  perf-stat.i.branch-instructions
  29447507 ±  3%     -13.8%   25382627 ±  2%  perf-stat.i.branch-misses
     67.78            +2.0       69.77        perf-stat.i.cache-miss-rate%
      1.55 ±  4%     -14.7%       1.33        perf-stat.i.cpi
 5.857e+10 ±  7%     -27.4%  4.252e+10        perf-stat.i.cpu-cycles
    148.26 ±  2%     -12.2%     130.23        perf-stat.i.cpu-migrations
    154.03 ±  3%     -24.7%     115.95        perf-stat.i.cycles-between-cache-misses
      0.20 ±  2%      -0.0        0.15 ±  3%  perf-stat.i.dTLB-load-miss-rate%
  20033658 ±  4%     -34.6%   13108913        perf-stat.i.dTLB-load-misses
 9.948e+09 ±  3%     -13.4%  8.617e+09 ±  3%  perf-stat.i.dTLB-loads
      0.37 ±  3%      -0.1        0.28 ±  3%  perf-stat.i.dTLB-store-miss-rate%
  20362529 ±  4%     -30.4%   14162666 ±  5%  perf-stat.i.dTLB-store-misses
 5.532e+09 ±  2%     -10.6%  4.945e+09 ±  2%  perf-stat.i.dTLB-stores
     73.19            -1.4       71.76        perf-stat.i.iTLB-load-miss-rate%
  12785996 ±  2%     -10.5%   11440556 ±  3%  perf-stat.i.iTLB-load-misses
   4812663 ±  2%      -6.0%    4526122        perf-stat.i.iTLB-loads
 3.712e+10 ±  3%     -14.2%  3.185e+10 ±  2%  perf-stat.i.instructions
      2895            -4.1%       2777        perf-stat.i.instructions-per-iTLB-miss
      0.66 ±  4%     +16.5%       0.77        perf-stat.i.ipc
   1203000 ±  2%     -22.0%     938451 ±  2%  perf-stat.i.major-faults
    632441 ±  5%     +66.7%    1054315 ±  2%  perf-stat.i.minor-faults
   1835441 ±  3%      +8.6%    1992767 ±  2%  perf-stat.i.page-faults
     15.80           +10.3%      17.43        perf-stat.overall.MPKI
     67.75            +2.1       69.87        perf-stat.overall.cache-miss-rate%
      1.58 ±  4%     -15.3%       1.34        perf-stat.overall.cpi
    147.21 ±  3%     -25.5%     109.71        perf-stat.overall.cycles-between-cache-misses
      0.20 ±  2%      -0.0        0.15 ±  3%  perf-stat.overall.dTLB-load-miss-rate%
      0.37 ±  3%      -0.1        0.29 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     72.65            -1.0       71.64        perf-stat.overall.iTLB-load-miss-rate%
      2902            -4.1%       2784        perf-stat.overall.instructions-per-iTLB-miss
      0.64 ±  4%     +17.8%       0.75        perf-stat.overall.ipc
   8598809           -28.3%    6163102        perf-stat.overall.path-length
 7.185e+09 ±  3%     -14.9%  6.114e+09 ±  2%  perf-stat.ps.branch-instructions
  29306041 ±  3%     -13.8%   25261462 ±  2%  perf-stat.ps.branch-misses
 5.829e+10 ±  7%     -27.4%  4.232e+10        perf-stat.ps.cpu-cycles
    147.55 ±  2%     -12.2%     129.61        perf-stat.ps.cpu-migrations
  19935636 ±  4%     -34.6%   13047562        perf-stat.ps.dTLB-load-misses
 9.899e+09 ±  3%     -13.4%  8.575e+09 ±  2%  perf-stat.ps.dTLB-loads
  20263384 ±  4%     -30.4%   14096250 ±  5%  perf-stat.ps.dTLB-store-misses
 5.505e+09 ±  2%     -10.6%   4.92e+09 ±  2%  perf-stat.ps.dTLB-stores
  12723152 ±  2%     -10.5%   11384126 ±  3%  perf-stat.ps.iTLB-load-misses
   4789226 ±  2%      -6.0%    4503341        perf-stat.ps.iTLB-loads
 3.694e+10 ±  3%     -14.2%  3.169e+10 ±  2%  perf-stat.ps.instructions
   1197099 ±  2%     -22.0%     933922 ±  2%  perf-stat.ps.major-faults
    629469 ±  5%     +66.7%    1049045 ±  2%  perf-stat.ps.minor-faults
   1826569 ±  3%      +8.6%    1982969 ±  2%  perf-stat.ps.page-faults
 7.476e+12 ±  3%     -14.4%  6.403e+12 ±  2%  perf-stat.total.instructions
    104876 ± 15%     -55.2%      46978 ±  8%  proc-vmstat.allocstall_movable
      6670 ± 13%     +90.1%      12681 ± 23%  proc-vmstat.allocstall_normal
    270.00 ± 34%     -62.4%     101.50 ± 33%  proc-vmstat.compact_fail
    277.50 ± 33%     -61.0%     108.25 ± 36%  proc-vmstat.compact_stall
    629.25 ± 35%    +596.1%       4380 ±  8%  proc-vmstat.kswapd_low_wmark_hit_quickly
     15034 ±  8%     -34.2%       9894 ±  3%  proc-vmstat.nr_active_anon
   2432763          +222.5%    7846521        proc-vmstat.nr_active_file
 2.229e+08 ±  2%     +19.5%  2.664e+08 ±  2%  proc-vmstat.nr_dirtied
   1732719            -3.1%    1679210        proc-vmstat.nr_dirty
    280680            -4.7%     267623        proc-vmstat.nr_free_pages
   7100824           -77.3%    1613637        proc-vmstat.nr_inactive_file
    164.75 ±  9%     -54.9%      74.25 ±  8%  proc-vmstat.nr_isolated_file
   9582341            -2.0%    9389351        proc-vmstat.nr_mapped
    121263            -4.0%     116373        proc-vmstat.nr_shmem
    162508           +60.3%     260428        proc-vmstat.nr_slab_reclaimable
   5464481 ±  4%    +578.9%   37099437 ±  2%  proc-vmstat.nr_vmscan_immediate_reclaim
    639.25 ±  3%     +43.5%     917.50 ±  4%  proc-vmstat.nr_writeback
 2.217e+08 ±  2%     +19.4%  2.646e+08 ±  2%  proc-vmstat.nr_written
     15036 ±  8%     -34.2%       9895 ±  3%  proc-vmstat.nr_zone_active_anon
   2434338          +222.3%    7846813        proc-vmstat.nr_zone_active_file
   7101719           -77.2%    1615989        proc-vmstat.nr_zone_inactive_file
   1737241            -3.1%    1683939        proc-vmstat.nr_zone_write_pending
  40016131 ±  6%     -16.6%   33361886 ±  6%  proc-vmstat.numa_foreign
  25745506 ± 14%     +56.4%   40263559 ±  5%  proc-vmstat.numa_hint_faults
  19974893 ± 14%     +49.8%   29926746 ±  8%  proc-vmstat.numa_hint_faults_local
 2.039e+08 ±  2%     -22.5%  1.581e+08 ±  2%  proc-vmstat.numa_hit
      1808 ±  2%      -3.3%       1749        proc-vmstat.numa_huge_pte_updates
 2.039e+08 ±  2%     -22.5%   1.58e+08 ±  2%  proc-vmstat.numa_local
  40016131 ±  6%     -16.6%   33361886 ±  6%  proc-vmstat.numa_miss
  40047410 ±  6%     -16.6%   33393063 ±  6%  proc-vmstat.numa_other
    251563 ± 29%    +420.2%    1308599 ± 10%  proc-vmstat.numa_pages_migrated
    635.00 ± 35%    +596.9%       4425 ±  8%  proc-vmstat.pageoutrun
 1.127e+08 ±  2%     -55.0%   50732767        proc-vmstat.pgactivate
   9619324 ±  4%     -37.5%    6013199 ±  7%  proc-vmstat.pgalloc_dma32
 2.354e+08 ±  2%     -20.7%  1.868e+08 ±  2%  proc-vmstat.pgalloc_normal
 1.829e+08 ±  2%     -97.1%    5298999 ±  4%  proc-vmstat.pgdeactivate
 7.145e+08 ±  3%      +6.8%  7.627e+08 ±  2%  proc-vmstat.pgfault
 2.353e+08 ±  2%     -22.4%  1.827e+08 ±  3%  proc-vmstat.pgfree
 2.427e+08 ±  2%     -22.0%  1.892e+08 ±  2%  proc-vmstat.pgmajfault
    427089 ± 38%    +278.2%    1615323 ± 11%  proc-vmstat.pgmigrate_success
  9.71e+08 ±  2%     -22.0%  7.569e+08 ±  2%  proc-vmstat.pgpgin
 8.867e+08 ±  2%     +19.4%  1.059e+09 ±  2%  proc-vmstat.pgpgout
 1.829e+08 ±  2%     -97.1%    5298820 ±  4%  proc-vmstat.pgrefill
   5485949 ±  3%    +574.1%   36983260 ±  2%  proc-vmstat.pgrotated
 1.984e+08 ± 12%     -78.9%   41928600 ± 14%  proc-vmstat.pgscan_direct
 3.395e+08 ±  3%     +11.0%  3.768e+08        proc-vmstat.pgscan_kswapd
  89080449 ± 12%     -87.6%   11029910 ±  8%  proc-vmstat.pgsteal_direct
 1.439e+08 ±  3%     +17.0%  1.684e+08 ±  2%  proc-vmstat.pgsteal_kswapd
  37738291 ±  5%    -100.0%       0.00        proc-vmstat.workingset_activate
     25988 ±  2%     -22.6%      20107 ±  2%  proc-vmstat.workingset_nodes
  69506838 ±  2%    -100.0%       0.00        proc-vmstat.workingset_refault
  21650634 ±  5%    -100.0%       1.50 ± 33%  proc-vmstat.workingset_restore
     87.50 ±168%    -100.0%       0.00        interrupts.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78
      2.75 ±110%   +5090.9%     142.75 ±103%  interrupts.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25
     50.25 ±153%     -98.5%       0.75 ±110%  interrupts.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47
      0.25 ±173%  +28300.0%      71.00 ±159%  interrupts.86:PCI-MSI.31981619-edge.i40e-eth0-TxRx-50
  43979343 ± 10%     -35.3%   28436814 ±  6%  interrupts.CAL:Function_call_interrupts
    309869 ± 12%     -48.7%     158885 ± 43%  interrupts.CPU0.CAL:Function_call_interrupts
      1496 ± 20%     -59.7%     603.25 ± 30%  interrupts.CPU0.NMI:Non-maskable_interrupts
      1496 ± 20%     -59.7%     603.25 ± 30%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
    711.25 ± 39%    +120.0%       1564 ± 46%  interrupts.CPU0.RES:Rescheduling_interrupts
    378389 ± 11%     -40.7%     224400 ± 36%  interrupts.CPU0.TLB:TLB_shootdowns
    564422 ±  6%     -51.1%     276198 ± 18%  interrupts.CPU11.CAL:Function_call_interrupts
    675725 ± 13%     -47.0%     358389 ± 15%  interrupts.CPU11.TLB:TLB_shootdowns
      1834 ± 31%     -58.9%     754.25 ± 16%  interrupts.CPU12.NMI:Non-maskable_interrupts
      1834 ± 31%     -58.9%     754.25 ± 16%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
    483343 ± 18%     -43.3%     273844 ± 35%  interrupts.CPU13.CAL:Function_call_interrupts
      4095 ± 34%     -72.2%       1137 ± 55%  interrupts.CPU13.NMI:Non-maskable_interrupts
      4095 ± 34%     -72.2%       1137 ± 55%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    662959 ± 25%     -52.6%     313982 ± 30%  interrupts.CPU13.TLB:TLB_shootdowns
    560563 ± 24%     -49.1%     285449 ±  9%  interrupts.CPU14.CAL:Function_call_interrupts
    693290 ± 34%     -47.1%     366618 ±  5%  interrupts.CPU14.TLB:TLB_shootdowns
    558990 ± 21%     -61.4%     215573 ± 24%  interrupts.CPU15.CAL:Function_call_interrupts
      2204 ± 31%     -67.0%     727.50 ± 35%  interrupts.CPU15.NMI:Non-maskable_interrupts
      2204 ± 31%     -67.0%     727.50 ± 35%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    655709 ± 30%     -64.3%     233774 ± 18%  interrupts.CPU15.TLB:TLB_shootdowns
    531963 ± 30%     -51.7%     257180 ± 61%  interrupts.CPU16.CAL:Function_call_interrupts
      3104 ± 49%     -66.2%       1050 ± 71%  interrupts.CPU16.NMI:Non-maskable_interrupts
      3104 ± 49%     -66.2%       1050 ± 71%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    758.50 ± 52%     +58.1%       1199 ± 18%  interrupts.CPU16.RES:Rescheduling_interrupts
    624333 ± 30%     -53.8%     288559 ± 54%  interrupts.CPU16.TLB:TLB_shootdowns
    414942 ± 20%     -51.1%     203078 ± 32%  interrupts.CPU18.CAL:Function_call_interrupts
    480458 ± 21%     -52.4%     228645 ± 25%  interrupts.CPU18.TLB:TLB_shootdowns
      1109 ± 44%    +205.5%       3390 ± 41%  interrupts.CPU2.RES:Rescheduling_interrupts
    461027 ± 25%     -57.0%     198359 ± 30%  interrupts.CPU20.CAL:Function_call_interrupts
      3798 ± 63%     -81.0%     720.75 ± 86%  interrupts.CPU20.NMI:Non-maskable_interrupts
      3798 ± 63%     -81.0%     720.75 ± 86%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
    564227 ± 37%     -62.5%     211819 ± 31%  interrupts.CPU20.TLB:TLB_shootdowns
    513743 ± 29%     -48.3%     265392 ± 32%  interrupts.CPU21.CAL:Function_call_interrupts
    644566 ± 43%     -57.1%     276789 ± 32%  interrupts.CPU21.TLB:TLB_shootdowns
    503293 ± 25%     -63.1%     185627 ± 52%  interrupts.CPU22.CAL:Function_call_interrupts
    645000 ± 13%     -67.5%     209945 ± 56%  interrupts.CPU22.TLB:TLB_shootdowns
    553939 ± 27%     -61.8%     211825 ± 28%  interrupts.CPU23.CAL:Function_call_interrupts
    687580 ± 25%     -67.1%     225909 ± 29%  interrupts.CPU23.TLB:TLB_shootdowns
    474.00 ± 52%    +258.5%       1699 ± 57%  interrupts.CPU24.RES:Rescheduling_interrupts
      2.25 ±123%   +6233.3%     142.50 ±103%  interrupts.CPU25.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25
      4043 ± 63%     -71.3%       1160 ± 31%  interrupts.CPU25.NMI:Non-maskable_interrupts
      4043 ± 63%     -71.3%       1160 ± 31%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    451692 ± 11%     -39.6%     272701 ± 25%  interrupts.CPU26.CAL:Function_call_interrupts
    815.25 ± 39%    +110.4%       1715 ± 29%  interrupts.CPU26.RES:Rescheduling_interrupts
    584283 ± 11%     -41.2%     343585 ± 33%  interrupts.CPU26.TLB:TLB_shootdowns
    517137 ± 11%     -41.0%     305137 ± 22%  interrupts.CPU27.CAL:Function_call_interrupts
      2414 ± 26%     -45.2%       1323 ± 26%  interrupts.CPU27.NMI:Non-maskable_interrupts
      2414 ± 26%     -45.2%       1323 ± 26%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
    614104 ±  7%     -39.7%     370314 ± 19%  interrupts.CPU27.TLB:TLB_shootdowns
    441205 ± 12%     -32.1%     299453 ± 20%  interrupts.CPU28.CAL:Function_call_interrupts
      2273 ± 24%     -39.7%       1369 ± 19%  interrupts.CPU28.NMI:Non-maskable_interrupts
      2273 ± 24%     -39.7%       1369 ± 19%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
    544289 ±  4%     -31.6%     372065 ± 21%  interrupts.CPU28.TLB:TLB_shootdowns
    520217 ± 28%     -31.2%     357653 ± 14%  interrupts.CPU29.CAL:Function_call_interrupts
      3744 ± 63%     -55.3%       1674 ± 27%  interrupts.CPU29.NMI:Non-maskable_interrupts
      3744 ± 63%     -55.3%       1674 ± 27%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    814.25 ± 53%    +128.8%       1863 ± 39%  interrupts.CPU29.RES:Rescheduling_interrupts
    613340 ± 29%     -31.8%     418352 ± 11%  interrupts.CPU29.TLB:TLB_shootdowns
    516303 ± 28%     -48.8%     264227 ± 29%  interrupts.CPU3.CAL:Function_call_interrupts
      1951 ± 32%     -71.4%     558.25 ± 28%  interrupts.CPU3.NMI:Non-maskable_interrupts
      1951 ± 32%     -71.4%     558.25 ± 28%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    649962 ± 17%     -54.0%     298822 ± 40%  interrupts.CPU3.TLB:TLB_shootdowns
    518481 ± 11%     -22.2%     403431 ± 14%  interrupts.CPU30.CAL:Function_call_interrupts
    678.50 ± 47%    +154.1%       1724 ± 16%  interrupts.CPU30.RES:Rescheduling_interrupts
    740472 ± 16%     -40.8%     438014 ± 16%  interrupts.CPU30.TLB:TLB_shootdowns
    497599 ± 13%     -26.9%     363979 ±  2%  interrupts.CPU31.CAL:Function_call_interrupts
    560143 ± 11%     -21.8%     438031 ± 10%  interrupts.CPU31.TLB:TLB_shootdowns
    919.75 ± 37%     +68.1%       1546 ± 16%  interrupts.CPU32.RES:Rescheduling_interrupts
    526266 ± 22%     -40.5%     312935 ± 27%  interrupts.CPU33.CAL:Function_call_interrupts
      2250 ± 15%     -38.3%       1387 ±  3%  interrupts.CPU33.NMI:Non-maskable_interrupts
      2250 ± 15%     -38.3%       1387 ±  3%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
    959.25 ± 31%    +197.4%       2852 ± 59%  interrupts.CPU33.RES:Rescheduling_interrupts
    589843 ± 17%     -40.4%     351483 ±  9%  interrupts.CPU34.CAL:Function_call_interrupts
    724580 ± 19%     -43.0%     412731 ±  9%  interrupts.CPU34.TLB:TLB_shootdowns
      3061 ± 51%     -66.5%       1026 ± 54%  interrupts.CPU35.NMI:Non-maskable_interrupts
      3061 ± 51%     -66.5%       1026 ± 54%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
    527518 ± 26%     -41.3%     309832 ± 20%  interrupts.CPU36.CAL:Function_call_interrupts
      1935 ± 29%     -41.1%       1139 ± 25%  interrupts.CPU36.NMI:Non-maskable_interrupts
      1935 ± 29%     -41.1%       1139 ± 25%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    605919 ± 27%     -33.5%     402973 ± 36%  interrupts.CPU36.TLB:TLB_shootdowns
      1968 ± 19%     -43.1%       1120 ± 33%  interrupts.CPU37.NMI:Non-maskable_interrupts
      1968 ± 19%     -43.1%       1120 ± 33%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
    552091 ± 17%     -37.9%     343088 ± 20%  interrupts.CPU38.CAL:Function_call_interrupts
    741448 ± 25%     -42.9%     423500 ± 21%  interrupts.CPU38.TLB:TLB_shootdowns
    893.75 ± 22%     +66.8%       1490 ± 34%  interrupts.CPU39.RES:Rescheduling_interrupts
    651520 ± 10%     -32.9%     437091 ± 33%  interrupts.CPU39.TLB:TLB_shootdowns
    443156 ± 27%     -74.1%     114892 ± 21%  interrupts.CPU4.CAL:Function_call_interrupts
    536804 ± 32%     -73.9%     139975 ± 23%  interrupts.CPU4.TLB:TLB_shootdowns
      2376 ±  9%     -31.2%       1635 ± 25%  interrupts.CPU40.NMI:Non-maskable_interrupts
      2376 ±  9%     -31.2%       1635 ± 25%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    634311 ± 13%     -30.5%     441067 ± 19%  interrupts.CPU42.TLB:TLB_shootdowns
    605811 ± 24%     -44.1%     338572 ±  5%  interrupts.CPU45.CAL:Function_call_interrupts
    824.00 ± 43%    +169.8%       2223 ± 39%  interrupts.CPU45.RES:Rescheduling_interrupts
    789721 ± 28%     -45.9%     427130 ± 12%  interrupts.CPU45.TLB:TLB_shootdowns
    527252 ± 26%     -41.4%     309202 ± 18%  interrupts.CPU46.CAL:Function_call_interrupts
    747063 ± 34%     -55.4%     333517 ± 18%  interrupts.CPU46.TLB:TLB_shootdowns
     50.00 ±154%     -99.5%       0.25 ±173%  interrupts.CPU47.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47
    552975 ± 11%     -35.1%     358953 ± 23%  interrupts.CPU47.CAL:Function_call_interrupts
    698382 ±  5%     -34.2%     459438 ± 31%  interrupts.CPU47.TLB:TLB_shootdowns
    668725 ± 25%     -71.0%     193871 ± 59%  interrupts.CPU48.CAL:Function_call_interrupts
      2351 ± 23%     -63.0%     869.00 ± 74%  interrupts.CPU48.NMI:Non-maskable_interrupts
      2351 ± 23%     -63.0%     869.00 ± 74%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      1036 ± 39%    +143.7%       2524 ± 59%  interrupts.CPU48.RES:Rescheduling_interrupts
    723105 ± 28%     -72.2%     200997 ± 57%  interrupts.CPU48.TLB:TLB_shootdowns
    681246 ± 37%     -69.7%     206357 ± 66%  interrupts.CPU49.CAL:Function_call_interrupts
    734750 ± 36%     -70.8%     214575 ± 68%  interrupts.CPU49.TLB:TLB_shootdowns
    532908 ± 33%     -54.9%     240375 ± 53%  interrupts.CPU50.CAL:Function_call_interrupts
    622804 ± 27%     -59.6%     251480 ± 56%  interrupts.CPU50.TLB:TLB_shootdowns
      2612 ± 56%     -64.4%     929.00 ± 37%  interrupts.CPU54.RES:Rescheduling_interrupts
      1525 ± 19%     -45.9%     825.25 ± 38%  interrupts.CPU6.NMI:Non-maskable_interrupts
      1525 ± 19%     -45.9%     825.25 ± 38%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    658.25 ± 61%    +183.8%       1868 ± 51%  interrupts.CPU60.RES:Rescheduling_interrupts
      1826 ± 29%     -66.0%     620.75 ± 53%  interrupts.CPU61.NMI:Non-maskable_interrupts
      1826 ± 29%     -66.0%     620.75 ± 53%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      2182 ± 35%     -53.4%       1017 ± 20%  interrupts.CPU62.NMI:Non-maskable_interrupts
      2182 ± 35%     -53.4%       1017 ± 20%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
    479189 ± 47%     -50.5%     237071 ± 37%  interrupts.CPU64.TLB:TLB_shootdowns
    417282 ± 22%     -36.7%     263997 ± 22%  interrupts.CPU7.CAL:Function_call_interrupts
    826.50 ± 48%     +73.7%       1435 ± 27%  interrupts.CPU7.RES:Rescheduling_interrupts
    533501 ± 13%     -45.4%     291545 ± 28%  interrupts.CPU7.TLB:TLB_shootdowns
    618059 ± 18%     -53.0%     290711 ± 27%  interrupts.CPU72.CAL:Function_call_interrupts
    680178 ± 18%     -53.6%     315576 ± 36%  interrupts.CPU72.TLB:TLB_shootdowns
      2597 ± 16%     -43.7%       1463 ± 24%  interrupts.CPU73.NMI:Non-maskable_interrupts
      2597 ± 16%     -43.7%       1463 ± 24%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
    459.50 ± 41%    +431.0%       2440 ± 26%  interrupts.CPU73.RES:Rescheduling_interrupts
    521.75 ± 55%    +281.9%       1992 ± 21%  interrupts.CPU74.RES:Rescheduling_interrupts
      2656 ± 21%     -45.7%       1441 ± 33%  interrupts.CPU75.NMI:Non-maskable_interrupts
      2656 ± 21%     -45.7%       1441 ± 33%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
    524.75 ± 54%    +141.9%       1269 ± 32%  interrupts.CPU75.RES:Rescheduling_interrupts
     87.00 ±169%    -100.0%       0.00        interrupts.CPU78.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78
    650.00 ± 35%    +102.7%       1317 ± 31%  interrupts.CPU78.RES:Rescheduling_interrupts
    903.25 ± 25%     +76.1%       1590 ± 14%  interrupts.CPU84.NMI:Non-maskable_interrupts
    903.25 ± 25%     +76.1%       1590 ± 14%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
    421.00 ± 38%    +147.6%       1042 ± 39%  interrupts.CPU85.RES:Rescheduling_interrupts
    807.25 ± 29%     +56.5%       1263 ± 24%  interrupts.CPU89.RES:Rescheduling_interrupts
    521370 ±  3%     -46.6%     278562 ± 38%  interrupts.CPU9.CAL:Function_call_interrupts
    613194 ± 20%     -46.4%     328688 ± 32%  interrupts.CPU9.TLB:TLB_shootdowns
    440.00 ± 33%    +163.4%       1158 ± 26%  interrupts.CPU93.RES:Rescheduling_interrupts
    208.75 ±  9%     -40.4%     124.50 ±  3%  interrupts.IWI:IRQ_work_interrupts
    204364 ±  7%     -31.0%     140939 ±  2%  interrupts.NMI:Non-maskable_interrupts
    204364 ±  7%     -31.0%     140939 ±  2%  interrupts.PMI:Performance_monitoring_interrupts
    111435 ± 10%     +35.2%     150649 ±  6%  interrupts.RES:Rescheduling_interrupts
  52728394 ± 10%     -38.7%   32297958 ±  4%  interrupts.TLB:TLB_shootdowns
     22.74 ± 16%     -18.0        4.75 ±  6%  perf-profile.calltrace.cycles-pp.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault
     29.75 ± 13%     -16.9       12.83 ±  2%  perf-profile.calltrace.cycles-pp.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault.__handle_mm_fault
     30.06 ± 13%     -16.9       13.17 ±  2%  perf-profile.calltrace.cycles-pp.__do_fault.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault
     30.04 ± 13%     -16.9       13.15 ±  2%  perf-profile.calltrace.cycles-pp.ext4_filemap_fault.__do_fault.handle_pte_fault.__handle_mm_fault.handle_mm_fault
     17.51 ± 18%     -16.4        1.14 ± 28%  perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault
     18.25 ± 17%     -16.1        2.16 ± 12%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault
     15.88 ± 19%     -15.2        0.66 ± 61%  perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page.filemap_fault
     15.86 ± 19%     -15.2        0.66 ± 61%  perf-profile.calltrace.cycles-pp.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask
     15.86 ± 19%     -15.2        0.66 ± 61%  perf-profile.calltrace.cycles-pp.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page
     15.78 ± 19%     -15.1        0.64 ± 61%  perf-profile.calltrace.cycles-pp.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath
     37.45 ± 11%     -14.3       23.12 ±  2%  perf-profile.calltrace.cycles-pp.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
     38.00 ± 11%     -14.0       24.01 ±  2%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
     38.59 ± 11%     -13.7       24.93 ±  2%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault
     39.53 ± 11%     -13.0       26.49 ±  2%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
     39.80 ± 11%     -12.8       26.95 ±  2%  perf-profile.calltrace.cycles-pp.page_fault
     12.67 ± 19%     -12.0        0.63 ± 61%  perf-profile.calltrace.cycles-pp.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages
      6.30 ± 14%      -6.2        0.13 ±173%  perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages
      6.03 ± 27%      -6.0        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node
      5.53 ± 25%      -5.5        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages
      4.11 ± 12%      -2.1        2.05 ±  3%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault
      2.75 ± 28%      -2.0        0.72 ± 12%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages
      2.85 ± 27%      -2.0        0.83 ± 10%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages.generic_fadvise
      3.32 ±  9%      -2.0        1.35 ± 14%  perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      3.19 ± 23%      -1.8        1.35 ±  7%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64
      3.36 ± 22%      -1.7        1.70 ±  5%  perf-profile.calltrace.cycles-pp.deactivate_file_page.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      2.33 ± 17%      -1.3        0.98 ±  2%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.filemap_fault
      2.39 ± 16%      -1.3        1.07 ±  3%  perf-profile.calltrace.cycles-pp.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.filemap_fault.ext4_filemap_fault
      2.16 ±  8%      -0.7        1.50 ± 11%  perf-profile.calltrace.cycles-pp.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      1.90 ±  9%      -0.6        1.29 ± 12%  perf-profile.calltrace.cycles-pp.rmap_walk_file.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec
      1.69 ±  8%      -0.4        1.33 ±  4%  perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one.rmap_walk_file
      1.75 ±  7%      -0.3        1.44 ±  3%  perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one.rmap_walk_file.page_mkclean
      1.49 ±  8%      -0.3        1.19 ±  6%  perf-profile.calltrace.cycles-pp.smp_call_function_single.on_each_cpu_cond_mask.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one
      1.82 ±  7%      -0.3        1.54 ±  4%  perf-profile.calltrace.cycles-pp.ptep_clear_flush.page_mkclean_one.rmap_walk_file.page_mkclean.clear_page_dirty_for_io
      0.61 ±  8%      +0.3        0.89 ±  8%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault
      2.60 ±  8%      +0.4        3.01 ±  3%  perf-profile.calltrace.cycles-pp.filemap_map_pages.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault
      2.08 ±  6%      +0.4        2.51 ±  4%  perf-profile.calltrace.cycles-pp.page_mkclean_one.rmap_walk_file.page_mkclean.clear_page_dirty_for_io.mpage_submit_page
      0.73 ± 11%      +0.4        1.16 ± 16%  perf-profile.calltrace.cycles-pp.isolate_lru_pages.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat
      0.93 ±  9%      +0.5        1.38 ±  4%  perf-profile.calltrace.cycles-pp.pagevec_lookup_entries.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      0.92 ±  9%      +0.5        1.38 ±  3%  perf-profile.calltrace.cycles-pp.find_get_entries.pagevec_lookup_entries.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64
      0.77 ±  7%      +0.5        1.24 ±  5%  perf-profile.calltrace.cycles-pp.get_io_u
      0.00            +0.6        0.61 ±  5%  perf-profile.calltrace.cycles-pp.fault_dirty_shared_page.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
      2.46 ± 10%      +0.6        3.08 ±  4%  perf-profile.calltrace.cycles-pp.page_mkclean.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
      2.23 ±  6%      +0.6        2.85 ±  4%  perf-profile.calltrace.cycles-pp.rmap_walk_file.page_mkclean.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs
      0.00            +0.6        0.63 ±  9%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.22 ± 10%      +0.7        1.87 ±  4%  perf-profile.calltrace.cycles-pp.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite.do_wp_page.handle_pte_fault
      4.32 ±  6%      +0.7        4.98 ±  4%  perf-profile.calltrace.cycles-pp.__memcpy_mcsafe.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio
      4.41 ±  6%      +0.7        5.08 ±  4%  perf-profile.calltrace.cycles-pp.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio.ext4_mpage_readpages
      3.50 ±  5%      +0.7        4.20 ±  3%  perf-profile.calltrace.cycles-pp.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages
      0.00            +0.8        0.77 ±  8%  perf-profile.calltrace.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite
      5.00 ±  6%      +0.8        5.78 ±  4%  perf-profile.calltrace.cycles-pp.pmem_make_request.generic_make_request.submit_bio.ext4_mpage_readpages.filemap_fault
      0.39 ± 57%      +0.8        1.20 ±  5%  perf-profile.calltrace.cycles-pp.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite.do_wp_page
      0.00            +0.9        0.90 ±  4%  perf-profile.calltrace.cycles-pp.__test_set_page_writeback.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
      5.42 ±  6%      +0.9        6.33 ±  4%  perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault
      5.46 ±  6%      +0.9        6.38 ±  4%  perf-profile.calltrace.cycles-pp.submit_bio.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault.__do_fault
      0.00            +0.9        0.94 ± 23%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.75 ±  9%      +1.0        1.78 ± 19%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      1.42 ± 10%      +1.0        2.46 ±  3%  perf-profile.calltrace.cycles-pp.ext4_page_mkwrite.do_page_mkwrite.do_wp_page.handle_pte_fault.__handle_mm_fault
      1.44 ± 10%      +1.1        2.50 ±  3%  perf-profile.calltrace.cycles-pp.do_page_mkwrite.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
      6.32 ±  5%      +1.1        7.40 ±  4%  perf-profile.calltrace.cycles-pp.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault
      0.42 ± 57%      +1.1        1.53 ±  4%  perf-profile.calltrace.cycles-pp.mark_buffer_dirty.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite
      0.83 ±  8%      +1.1        1.95 ± 19%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.65 ±  4%      +1.2        1.87 ± 11%  perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.ext4_finish_bio.ext4_end_bio.pmem_make_request
      0.71 ±  5%      +1.7        2.40 ±  3%  perf-profile.calltrace.cycles-pp.end_page_writeback.ext4_finish_bio.ext4_end_bio.pmem_make_request.generic_make_request
      1.92 ±  5%      +1.7        3.61        perf-profile.calltrace.cycles-pp.ext4_finish_bio.ext4_end_bio.pmem_make_request.generic_make_request.submit_bio
      1.93 ±  5%      +1.7        3.63        perf-profile.calltrace.cycles-pp.ext4_end_bio.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit
      1.93 ±  8%      +1.8        3.68 ±  3%  perf-profile.calltrace.cycles-pp.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault
      8.26 ±  9%      +1.8       10.04 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.posix_fadvise
      8.26 ±  9%      +1.8       10.04 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      8.24 ±  9%      +1.8       10.02 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      8.24 ±  9%      +1.8       10.02 ±  5%  perf-profile.calltrace.cycles-pp.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
      8.24 ±  9%      +1.8       10.02 ±  5%  perf-profile.calltrace.cycles-pp.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.28 ±  9%      +1.8       10.07 ±  5%  perf-profile.calltrace.cycles-pp.posix_fadvise
      3.07 ±  9%      +2.1        5.21 ±  8%  perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__filemap_fdatawrite_range
      3.29 ±  9%      +2.3        5.56 ±  8%  perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__filemap_fdatawrite_range.generic_fadvise
      3.30 ±  9%      +2.3        5.59 ±  8%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
      3.30 ±  9%      +2.3        5.59 ±  8%  perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64
      3.30 ±  9%      +2.3        5.59 ±  8%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64
      4.45 ± 11%      +2.4        6.80 ± 13%  perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat
      3.45 ± 16%      +3.3        6.74 ±  3%  perf-profile.calltrace.cycles-pp.__memcpy_flushcache.write_pmem.pmem_do_bvec.pmem_make_request.generic_make_request
      3.49 ± 16%      +3.3        6.81 ±  3%  perf-profile.calltrace.cycles-pp.write_pmem.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio
      3.51 ± 16%      +3.3        6.86 ±  3%  perf-profile.calltrace.cycles-pp.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit
      7.42 ±  7%      +4.7       12.16 ±  2%  perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode
      7.77 ±  6%      +5.0       12.77        perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
      7.94 ±  7%      +5.0       12.96        perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      7.96 ±  7%      +5.0       13.00        perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      7.79 ±  6%      +5.1       12.85        perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
      7.79 ±  6%      +5.1       12.85        perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
      7.79 ±  6%      +5.1       12.85        perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      7.79 ±  6%      +5.1       12.85        perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
      7.79 ±  6%      +5.1       12.85        perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread
      7.79 ±  6%      +5.1       12.85        perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work
      7.79 ±  6%      +5.1       12.85        perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn
      5.48 ± 12%      +5.1       10.57 ±  2%  perf-profile.calltrace.cycles-pp.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit.ext4_bio_write_page
      5.49 ± 12%      +5.1       10.58 ±  2%  perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.ext4_io_submit.ext4_bio_write_page.mpage_submit_page
      5.50 ± 12%      +5.1       10.58 ±  2%  perf-profile.calltrace.cycles-pp.submit_bio.ext4_io_submit.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs
      5.50 ± 12%      +5.1       10.58 ±  2%  perf-profile.calltrace.cycles-pp.ext4_io_submit.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
      6.60 ± 11%      +6.0       12.63 ±  2%  perf-profile.calltrace.cycles-pp.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages
     15.73 ±  8%      +6.1       21.88 ±  5%  perf-profile.calltrace.cycles-pp.ret_from_fork
     15.73 ±  8%      +6.1       21.88 ±  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
     10.14 ±  7%      +6.8       16.91 ±  2%  perf-profile.calltrace.cycles-pp.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages
     22.84 ± 16%     -18.1        4.78 ±  6%  perf-profile.children.cycles-pp.pagecache_get_page
     29.77 ± 13%     -16.9       12.85 ±  2%  perf-profile.children.cycles-pp.filemap_fault
     30.04 ± 13%     -16.9       13.15 ±  2%  perf-profile.children.cycles-pp.ext4_filemap_fault
     30.07 ± 13%     -16.9       13.18 ±  2%  perf-profile.children.cycles-pp.__do_fault
     18.08 ± 18%     -16.8        1.31 ± 26%  perf-profile.children.cycles-pp.__alloc_pages_slowpath
     18.83 ± 17%     -16.5        2.35 ± 14%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
     16.42 ± 19%     -15.5        0.89 ± 37%  perf-profile.children.cycles-pp.try_to_free_pages
     16.40 ± 19%     -15.5        0.89 ± 37%  perf-profile.children.cycles-pp.do_try_to_free_pages
     24.16 ± 14%     -14.4        9.73 ± 11%  perf-profile.children.cycles-pp.shrink_node
     24.07 ± 14%     -14.4        9.67 ± 11%  perf-profile.children.cycles-pp.shrink_lruvec
     37.48 ± 11%     -14.3       23.17 ±  2%  perf-profile.children.cycles-pp.handle_pte_fault
     38.05 ± 11%     -14.0       24.07 ±  2%  perf-profile.children.cycles-pp.__handle_mm_fault
     38.64 ± 11%     -13.6       25.00 ±  2%  perf-profile.children.cycles-pp.handle_mm_fault
     39.58 ± 11%     -13.0       26.58 ±  2%  perf-profile.children.cycles-pp.do_page_fault
     39.84 ± 11%     -12.8       27.01 ±  2%  perf-profile.children.cycles-pp.page_fault
     15.30 ± 22%     -12.8        2.48 ± 12%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     19.34 ± 14%      -9.8        9.52 ± 10%  perf-profile.children.cycles-pp.shrink_inactive_list
      8.85 ± 23%      -8.1        0.72 ± 23%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      4.72 ± 13%      -4.6        0.11 ± 68%  perf-profile.children.cycles-pp.shrink_active_list
      6.92 ± 18%      -4.0        2.88 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     11.04 ± 10%      -3.8        7.26 ± 11%  perf-profile.children.cycles-pp.shrink_page_list
      5.65 ± 20%      -3.0        2.67 ±  3%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      4.12 ± 12%      -2.1        2.06 ±  3%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      3.42 ± 10%      -1.9        1.51 ± 13%  perf-profile.children.cycles-pp.__remove_mapping
      3.37 ± 22%      -1.7        1.70 ±  5%  perf-profile.children.cycles-pp.deactivate_file_page
      3.11 ± 15%      -1.5        1.57 ±  5%  perf-profile.children.cycles-pp.on_each_cpu_cond_mask
      2.41 ± 16%      -1.3        1.08 ±  2%  perf-profile.children.cycles-pp.__lru_cache_add
      2.96 ± 10%      -1.3        1.66 ±  9%  perf-profile.children.cycles-pp.page_referenced
      1.43 ± 25%      -1.2        0.23 ± 11%  perf-profile.children.cycles-pp.try_to_unmap_flush
      1.43 ± 25%      -1.2        0.23 ± 11%  perf-profile.children.cycles-pp.arch_tlbbatch_flush
      1.29 ± 24%      -1.0        0.31 ± 15%  perf-profile.children.cycles-pp.smp_call_function_many_cond
      1.64 ± 10%      -0.7        0.92 ±  8%  perf-profile.children.cycles-pp.page_referenced_one
      0.80 ±  9%      -0.7        0.14 ± 23%  perf-profile.children.cycles-pp.workingset_refault
      0.80 ± 13%      -0.6        0.18 ± 15%  perf-profile.children.cycles-pp.wake_all_kswapds
      0.78 ± 14%      -0.6        0.17 ± 16%  perf-profile.children.cycles-pp.wakeup_kswapd
      1.80 ± 10%      -0.6        1.23 ±  5%  perf-profile.children.cycles-pp.smp_call_function_single
      1.73 ± 11%      -0.5        1.22 ± 15%  perf-profile.children.cycles-pp.isolate_lru_pages
      1.19 ± 20%      -0.5        0.68 ± 29%  perf-profile.children.cycles-pp.create_empty_buffers
      1.21 ± 19%      -0.5        0.71 ± 28%  perf-profile.children.cycles-pp.create_page_buffers
      1.74 ± 10%      -0.5        1.25 ±  5%  perf-profile.children.cycles-pp.page_vma_mapped_walk
      0.85 ± 26%      -0.4        0.44 ± 46%  perf-profile.children.cycles-pp.alloc_page_buffers
      1.61 ± 10%      -0.4        1.22 ±  9%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.80 ± 28%      -0.4        0.40 ± 50%  perf-profile.children.cycles-pp.alloc_buffer_head
      0.64 ± 32%      -0.4        0.27 ± 71%  perf-profile.children.cycles-pp.__slab_alloc
      1.54 ± 17%      -0.4        1.17 ± 16%  perf-profile.children.cycles-pp.__block_write_begin_int
      0.86 ± 26%      -0.4        0.49 ± 40%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.63 ± 32%      -0.4        0.27 ± 73%  perf-profile.children.cycles-pp.___slab_alloc
      0.81 ± 10%      -0.4        0.46 ± 16%  perf-profile.children.cycles-pp.move_pages_to_lru
      0.48 ± 43%      -0.3        0.15 ±138%  perf-profile.children.cycles-pp.new_slab
      0.35 ± 10%      -0.3        0.09 ± 15%  perf-profile.children.cycles-pp.advance_inactive_age
      0.40 ±  8%      -0.2        0.19 ±  9%  perf-profile.children.cycles-pp.workingset_eviction
      0.49 ±  8%      -0.2        0.30 ± 13%  perf-profile.children.cycles-pp.try_to_free_buffers
      0.35 ± 10%      -0.2        0.19 ± 21%  perf-profile.children.cycles-pp.jbd2_journal_try_to_free_buffers
      0.33 ±  8%      -0.2        0.18 ± 22%  perf-profile.children.cycles-pp.jbd2_journal_grab_journal_head
      0.37 ±  9%      -0.1        0.26 ±  4%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.32 ± 13%      -0.1        0.20 ± 18%  perf-profile.children.cycles-pp.page_evictable
      0.28 ± 12%      -0.1        0.19 ± 19%  perf-profile.children.cycles-pp.page_remove_rmap
      0.14 ±  6%      -0.1        0.07 ± 12%  perf-profile.children.cycles-pp.vma_interval_tree_iter_first
      0.09 ± 11%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.get_partial_node
      0.17 ± 10%      -0.1        0.11 ± 15%  perf-profile.children.cycles-pp.drop_buffers
      0.19 ± 13%      -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.free_buffer_head
      0.24 ± 10%      -0.0        0.19 ±  3%  perf-profile.children.cycles-pp.ext4_da_get_block_prep
      0.15 ± 10%      -0.0        0.10 ± 12%  perf-profile.children.cycles-pp.mem_cgroup_page_lruvec
      0.12 ± 14%      -0.0        0.08 ± 15%  perf-profile.children.cycles-pp.__slab_free
      0.10 ± 11%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.ptep_clear_flush_young
      0.06 ±  9%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.__indirect_thunk_start
      0.06 ± 11%      +0.0        0.09 ± 12%  perf-profile.children.cycles-pp.generic_update_time
      0.08 ± 11%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.fput_many
      0.14 ±  7%      +0.0        0.17 ±  7%  perf-profile.children.cycles-pp.mempool_alloc
      0.13 ±  6%      +0.0        0.17 ±  6%  perf-profile.children.cycles-pp._find_next_bit
      0.09 ± 24%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.try_to_wake_up
      0.08 ±  5%      +0.0        0.11 ± 14%  perf-profile.children.cycles-pp.task_tick_fair
      0.07 ± 19%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.io_serial_in
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.ext4_dirty_inode
      0.06 ±  9%      +0.0        0.09 ±  8%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.09 ± 19%      +0.0        0.13 ± 18%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.09 ± 19%      +0.0        0.13 ± 18%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.06 ±  7%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.__unlock_page_memcg
      0.07 ±  6%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.pmd_devmap_trans_unstable
      0.09 ±  5%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp.bio_add_page
      0.11 ±  7%      +0.0        0.15 ± 10%  perf-profile.children.cycles-pp.PageHuge
      0.10 ± 21%      +0.0        0.14 ± 16%  perf-profile.children.cycles-pp.serial8250_console_write
      0.07 ±  7%      +0.0        0.11 ± 13%  perf-profile.children.cycles-pp.schedule
      0.07 ±  5%      +0.0        0.12 ± 11%  perf-profile.children.cycles-pp.ext4_convert_inline_data
      0.13 ± 14%      +0.0        0.18 ± 10%  perf-profile.children.cycles-pp.blk_throtl_bio
      0.06 ± 20%      +0.0        0.11 ±  7%  perf-profile.children.cycles-pp.native_write_msr
      0.17 ±  6%      +0.0        0.22 ±  8%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.10 ± 12%      +0.0        0.15 ±  8%  perf-profile.children.cycles-pp.read_tsc
      0.01 ±173%      +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.update_load_avg
      0.04 ± 57%      +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.11 ± 17%      +0.1        0.16 ±  6%  perf-profile.children.cycles-pp.irq_work_run_list
      0.01 ±173%      +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__bio_try_merge_page
      0.10 ± 17%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.console_unlock
      0.06 ±  9%      +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.page_mapped
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.22 ±  9%      +0.1        0.28        perf-profile.children.cycles-pp.bio_alloc_bioset
      0.43 ±  9%      +0.1        0.49 ±  3%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.10 ±  7%      +0.1        0.16 ±  5%  perf-profile.children.cycles-pp.vmacache_find
      0.08 ± 16%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.lru_deactivate_file_fn
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.mem_cgroup_wb_domain
      0.12 ± 13%      +0.1        0.18 ±  6%  perf-profile.children.cycles-pp.xas_clear_mark
      0.08 ± 10%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.down_read_trylock
      0.03 ±100%      +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.note_gp_changes
      0.06 ± 13%      +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.__sb_start_write
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.ext4_inode_journal_mode
      0.32 ± 11%      +0.1        0.39 ±  3%  perf-profile.children.cycles-pp.find_get_entry
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.migrate_pages
      0.04 ± 57%      +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.load_balance
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.current_time
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.run_timer_softirq
      0.08            +0.1        0.15 ± 10%  perf-profile.children.cycles-pp.balance_dirty_pages
      0.07 ± 13%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.04 ± 58%      +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.inc_zone_page_state
      0.01 ±173%      +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.11 ±  3%      +0.1        0.19 ±  4%  perf-profile.children.cycles-pp.find_vma
      0.01 ±173%      +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.xas_set_mark
      0.23 ±  7%      +0.1        0.31 ±  5%  perf-profile.children.cycles-pp.xas_find
      0.07 ± 12%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.__fprop_inc_percpu_max
      0.21 ±  7%      +0.1        0.29 ±  5%  perf-profile.children.cycles-pp.__might_sleep
      0.29 ± 11%      +0.1        0.36 ±  9%  perf-profile.children.cycles-pp._cond_resched
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.find_busiest_group
      0.18 ±  6%      +0.1        0.26 ±  4%  perf-profile.children.cycles-pp.xas_start
      0.13 ±  6%      +0.1        0.22 ± 14%  perf-profile.children.cycles-pp.scheduler_tick
      0.12 ± 15%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.13 ± 12%      +0.1        0.21 ±  6%  perf-profile.children.cycles-pp.file_update_time
      0.00            +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.migrate_misplaced_page
      0.04 ± 57%      +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.rebalance_domains
      0.08 ±  5%      +0.1        0.18 ±  8%  perf-profile.children.cycles-pp.invalidate_inode_page
      0.04 ± 58%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.finish_mkwrite_fault
      0.35 ±  8%      +0.1        0.45 ±  4%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.12 ± 16%      +0.1        0.23 ± 22%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.26 ±  7%      +0.1        0.38 ±  8%  perf-profile.children.cycles-pp.flush_tlb_func_common
      0.12 ±  6%      +0.1        0.24 ±  3%  perf-profile.children.cycles-pp.set_page_dirty
      0.35 ±  6%      +0.1        0.47 ±  5%  perf-profile.children.cycles-pp.generic_make_request_checks
      0.20 ±  7%      +0.1        0.31 ±  6%  perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.14 ± 13%      +0.1        0.26 ±  8%  perf-profile.children.cycles-pp.dec_zone_page_state
      0.45 ±  6%      +0.1        0.57 ±  7%  perf-profile.children.cycles-pp.___might_sleep
      0.14 ± 17%      +0.1        0.27 ± 22%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.21 ±  2%      +0.1        0.33 ±  9%  perf-profile.children.cycles-pp.___perf_sw_event
      0.14 ± 10%      +0.1        0.27 ±  5%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.11 ±  7%      +0.1        0.24 ±  6%  perf-profile.children.cycles-pp.__xa_set_mark
      0.14 ± 23%      +0.1        0.27 ± 11%  perf-profile.children.cycles-pp.rcu_core
      0.11 ±  6%      +0.1        0.25 ±  6%  perf-profile.children.cycles-pp.__xa_clear_mark
      0.03 ±100%      +0.1        0.17 ± 42%  perf-profile.children.cycles-pp.tick_irq_enter
      0.04 ± 58%      +0.2        0.20 ± 35%  perf-profile.children.cycles-pp.irq_enter
      0.00            +0.2        0.17 ±  6%  perf-profile.children.cycles-pp.pagevec_move_tail_fn
      0.18 ± 15%      +0.2        0.36 ±  5%  perf-profile.children.cycles-pp.__pagevec_release
      0.31 ±  2%      +0.2        0.51 ±  7%  perf-profile.children.cycles-pp.__perf_sw_event
      0.24 ± 11%      +0.2        0.43 ± 15%  perf-profile.children.cycles-pp.update_process_times
      0.52 ±  7%      +0.2        0.72 ±  6%  perf-profile.children.cycles-pp.account_page_dirtied
      0.24 ± 12%      +0.2        0.45 ± 16%  perf-profile.children.cycles-pp.tick_sched_handle
      0.33 ±  2%      +0.2        0.55 ±  7%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.33 ±  2%      +0.2        0.55 ±  6%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.94 ±  7%      +0.2        1.17 ±  2%  perf-profile.children.cycles-pp.unlock_page
      0.05 ± 59%      +0.2        0.29 ±  8%  perf-profile.children.cycles-pp.pagevec_move_tail
      0.53 ±  9%      +0.2        0.76 ±  5%  perf-profile.children.cycles-pp.page_mapping
      0.29 ±  7%      +0.3        0.56 ± 18%  perf-profile.children.cycles-pp.tick_sched_timer
      0.22 ± 13%      +0.3        0.49 ± 22%  perf-profile.children.cycles-pp.clockevents_program_event
      0.05 ± 59%      +0.3        0.33 ±  5%  perf-profile.children.cycles-pp.rotate_reclaimable_page
      0.29 ± 12%      +0.3        0.57 ± 14%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.42 ± 13%      +0.3        0.72 ±  4%  perf-profile.children.cycles-pp.release_pages
      2.77 ±  5%      +0.3        3.08 ±  3%  perf-profile.children.cycles-pp.page_mkclean
      0.43 ±  6%      +0.3        0.78 ±  4%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.39 ±  6%      +0.3        0.74 ± 15%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.39 ±  8%      +0.4        0.75 ± 18%  perf-profile.children.cycles-pp.irq_exit
      0.28 ± 15%      +0.4        0.64 ±  8%  perf-profile.children.cycles-pp.menu_select
      0.75 ±  7%      +0.4        1.16 ±  5%  perf-profile.children.cycles-pp.__set_page_dirty
      2.63 ±  8%      +0.4        3.04 ±  3%  perf-profile.children.cycles-pp.filemap_map_pages
      0.72 ±  8%      +0.4        1.14 ±  3%  perf-profile.children.cycles-pp.xas_load
      0.42 ±  5%      +0.4        0.86 ±  5%  perf-profile.children.cycles-pp.fault_dirty_shared_page
      0.39 ±  7%      +0.4        0.84 ± 15%  perf-profile.children.cycles-pp.ktime_get
      0.93 ±  9%      +0.5        1.38 ±  4%  perf-profile.children.cycles-pp.find_get_entries
      0.61 ±  7%      +0.5        1.07 ±  4%  perf-profile.children.cycles-pp.sync_regs
      0.93 ±  9%      +0.5        1.38 ±  3%  perf-profile.children.cycles-pp.pagevec_lookup_entries
      0.78 ±  7%      +0.5        1.26 ±  5%  perf-profile.children.cycles-pp.get_io_u
      0.46 ±  7%      +0.5        0.97 ±  4%  perf-profile.children.cycles-pp.__set_page_dirty_buffers
      0.96 ±  8%      +0.6        1.54 ±  3%  perf-profile.children.cycles-pp.mark_buffer_dirty
      2.93 ± 12%      +0.6        3.55 ±  4%  perf-profile.children.cycles-pp.block_page_mkwrite
      0.70 ±  4%      +0.6        1.33 ±  2%  perf-profile.children.cycles-pp.__test_set_page_writeback
      4.37 ±  6%      +0.7        5.02 ±  4%  perf-profile.children.cycles-pp.__memcpy_mcsafe
      1.09 ±  9%      +0.7        1.76 ±  5%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.69 ±  8%      +0.7        1.39 ± 15%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.11 ±  8%      +0.7        1.81 ±  3%  perf-profile.children.cycles-pp.__block_commit_write
      3.53 ±  5%      +0.7        4.27 ±  3%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      3.40 ± 12%      +1.0        4.42 ±  3%  perf-profile.children.cycles-pp.ext4_page_mkwrite
      3.42 ± 12%      +1.0        4.46 ±  3%  perf-profile.children.cycles-pp.do_page_mkwrite
      1.00 ±  6%      +1.1        2.06 ±  3%  perf-profile.children.cycles-pp.test_clear_page_writeback
      6.32 ±  5%      +1.1        7.41 ±  4%  perf-profile.children.cycles-pp.ext4_mpage_readpages
      1.15 ±  8%      +1.2        2.36 ± 14%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.26 ±  7%      +1.3        2.59 ± 13%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.07 ±  7%      +1.3        2.42 ±  3%  perf-profile.children.cycles-pp.end_page_writeback
      1.94 ±  5%      +1.7        3.66        perf-profile.children.cycles-pp.ext4_end_bio
      1.94 ±  5%      +1.7        3.66        perf-profile.children.cycles-pp.ext4_finish_bio
      8.51 ±  9%      +1.7       10.24 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      8.51 ±  9%      +1.7       10.24 ±  5%  perf-profile.children.cycles-pp.do_syscall_64
      1.94 ±  8%      +1.8        3.69 ±  4%  perf-profile.children.cycles-pp.do_wp_page
      8.24 ±  9%      +1.8       10.02 ±  5%  perf-profile.children.cycles-pp.__x64_sys_fadvise64
      8.24 ±  9%      +1.8       10.02 ±  5%  perf-profile.children.cycles-pp.ksys_fadvise64_64
      8.24 ±  9%      +1.8       10.02 ±  5%  perf-profile.children.cycles-pp.generic_fadvise
      8.28 ±  9%      +1.8       10.07 ±  5%  perf-profile.children.cycles-pp.posix_fadvise
      3.30 ±  9%      +2.3        5.59 ±  8%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      3.47 ± 16%      +3.3        6.81 ±  3%  perf-profile.children.cycles-pp.__memcpy_flushcache
      3.50 ± 16%      +3.4        6.88 ±  3%  perf-profile.children.cycles-pp.write_pmem
      7.95 ±  7%      +4.1       12.00        perf-profile.children.cycles-pp.pmem_do_bvec
      7.94 ±  7%      +5.0       12.96        perf-profile.children.cycles-pp.process_one_work
      7.96 ±  7%      +5.0       13.00        perf-profile.children.cycles-pp.worker_thread
      7.79 ±  6%      +5.1       12.85        perf-profile.children.cycles-pp.wb_workfn
      7.79 ±  6%      +5.1       12.85        perf-profile.children.cycles-pp.wb_writeback
      7.79 ±  6%      +5.1       12.85        perf-profile.children.cycles-pp.__writeback_inodes_wb
      7.79 ±  6%      +5.1       12.85        perf-profile.children.cycles-pp.writeback_sb_inodes
      7.79 ±  6%      +5.1       12.85        perf-profile.children.cycles-pp.__writeback_single_inode
      5.52 ± 12%      +5.2       10.67 ±  2%  perf-profile.children.cycles-pp.ext4_io_submit
     10.52 ±  7%      +5.9       16.44        perf-profile.children.cycles-pp.pmem_make_request
      6.60 ± 11%      +6.0       12.63 ±  2%  perf-profile.children.cycles-pp.ext4_bio_write_page
     10.95 ±  7%      +6.1       17.01        perf-profile.children.cycles-pp.generic_make_request
     10.98 ±  6%      +6.1       17.05        perf-profile.children.cycles-pp.submit_bio
     15.73 ±  8%      +6.1       21.88 ±  5%  perf-profile.children.cycles-pp.ret_from_fork
     15.73 ±  8%      +6.1       21.88 ±  5%  perf-profile.children.cycles-pp.kthread
     10.14 ±  7%      +6.8       16.92 ±  2%  perf-profile.children.cycles-pp.mpage_submit_page
     10.50 ±  7%      +6.9       17.36 ±  2%  perf-profile.children.cycles-pp.mpage_process_page_bufs
     11.06 ±  7%      +7.3       18.36 ±  2%  perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
     11.09 ±  7%      +7.4       18.45 ±  2%  perf-profile.children.cycles-pp.do_writepages
     11.09 ±  7%      +7.4       18.45 ±  2%  perf-profile.children.cycles-pp.ext4_writepages
     15.29 ± 22%     -12.8        2.47 ± 12%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.16 ± 26%      -1.0        0.20 ± 21%  perf-profile.self.cycles-pp.smp_call_function_many_cond
      0.76 ± 13%      -0.6        0.17 ± 17%  perf-profile.self.cycles-pp.wakeup_kswapd
      1.66 ± 10%      -0.6        1.09 ±  5%  perf-profile.self.cycles-pp.smp_call_function_single
      1.60 ± 10%      -0.4        1.21 ±  9%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.42 ±  8%      -0.3        0.10 ± 23%  perf-profile.self.cycles-pp.workingset_refault
      0.34 ± 10%      -0.2        0.09 ± 15%  perf-profile.self.cycles-pp.advance_inactive_age
      0.99 ±  9%      -0.2        0.79 ±  7%  perf-profile.self.cycles-pp.page_vma_mapped_walk
      0.74 ± 10%      -0.2        0.55 ±  5%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.43 ±  9%      -0.2        0.25 ± 16%  perf-profile.self.cycles-pp.move_pages_to_lru
      0.62 ±  9%      -0.2        0.47 ±  7%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.32 ±  9%      -0.1        0.17 ± 25%  perf-profile.self.cycles-pp.jbd2_journal_grab_journal_head
      0.37 ±  9%      -0.1        0.24 ± 18%  perf-profile.self.cycles-pp.__remove_mapping
      0.37 ± 10%      -0.1        0.26 ± 10%  perf-profile.self.cycles-pp.page_referenced_one
      0.23 ± 12%      -0.1        0.13 ± 17%  perf-profile.self.cycles-pp.page_evictable
      0.17 ±  6%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.workingset_eviction
      0.16 ±  9%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.14 ± 15%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.pagecache_get_page
      0.17 ± 12%      -0.1        0.11 ± 15%  perf-profile.self.cycles-pp.drop_buffers
      0.12 ±  5%      -0.1        0.06 ± 14%  perf-profile.self.cycles-pp.vma_interval_tree_iter_first
      0.26 ± 11%      -0.1        0.20 ±  9%  perf-profile.self.cycles-pp.isolate_lru_pages
      0.12 ± 11%      -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.page_referenced
      0.14 ±  8%      -0.1        0.09 ± 13%  perf-profile.self.cycles-pp.mem_cgroup_page_lruvec
      0.12 ± 11%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.page_remove_rmap
      0.12 ± 14%      -0.0        0.08 ± 15%  perf-profile.self.cycles-pp.__slab_free
      0.10 ± 11%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.ptep_clear_flush_young
      0.08 ± 10%      -0.0        0.06        perf-profile.self.cycles-pp.ext4_da_get_block_prep
      0.10 ±  7%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.rcu_all_qs
      0.06 ±  6%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.__lru_cache_add
      0.07 ± 17%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.xas_find
      0.07 ± 10%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.fput_many
      0.12 ±  8%      +0.0        0.16 ±  7%  perf-profile.self.cycles-pp._find_next_bit
      0.15 ±  8%      +0.0        0.18 ±  7%  perf-profile.self.cycles-pp.alloc_set_pte
      0.08 ± 10%      +0.0        0.12 ±  7%  perf-profile.self.cycles-pp.generic_make_request_checks
      0.08 ± 14%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.pmem_do_bvec
      0.06 ± 14%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.lru_deactivate_file_fn
      0.07 ± 19%      +0.0        0.10 ± 10%  perf-profile.self.cycles-pp.io_serial_in
      0.08 ± 19%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.free_unref_page_list
      0.09 ± 11%      +0.0        0.13 ± 11%  perf-profile.self.cycles-pp.PageHuge
      0.06 ±  6%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.page_mkclean_one
      0.07 ±  6%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.41 ±  6%      +0.0        0.45 ±  8%  perf-profile.self.cycles-pp.filemap_fault
      0.12 ± 13%      +0.0        0.15 ±  7%  perf-profile.self.cycles-pp.xas_store
      0.10 ±  4%      +0.0        0.14 ± 10%  perf-profile.self.cycles-pp.ptep_clear_flush
      0.06 ±  7%      +0.0        0.10 ± 11%  perf-profile.self.cycles-pp.pmd_devmap_trans_unstable
      0.04 ± 57%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.__unlock_page_memcg
      0.10 ± 12%      +0.0        0.14 ±  9%  perf-profile.self.cycles-pp.read_tsc
      0.08 ± 14%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.down_read_trylock
      0.07            +0.0        0.12 ±  9%  perf-profile.self.cycles-pp.ext4_convert_inline_data
      0.07 ± 11%      +0.0        0.12 ±  5%  perf-profile.self.cycles-pp.flush_tlb_mm_range
      0.06 ±  6%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.set_page_dirty
      0.07 ± 17%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.pagevec_lru_move_fn
      0.06 ± 13%      +0.0        0.11 ± 11%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.06 ± 20%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.native_write_msr
      0.42 ±  8%      +0.1        0.47 ±  3%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__indirect_thunk_start
      0.12 ± 13%      +0.1        0.18 ± 14%  perf-profile.self.cycles-pp._cond_resched
      0.10 ±  9%      +0.1        0.15 ±  4%  perf-profile.self.cycles-pp.vmacache_find
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.note_gp_changes
      0.11 ± 11%      +0.1        0.17 ± 10%  perf-profile.self.cycles-pp.xas_clear_mark
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.write_pmem
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.mem_cgroup_wb_domain
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.pagevec_move_tail_fn
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.__sb_start_write
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.balance_dirty_pages
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.ext4_inode_journal_mode
      0.18 ±  8%      +0.1        0.25 ±  5%  perf-profile.self.cycles-pp.__might_sleep
      0.04 ± 59%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
      0.13 ±  8%      +0.1        0.20 ± 10%  perf-profile.self.cycles-pp.pmem_make_request
      0.01 ±173%      +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.xas_set_mark
      0.03 ±100%      +0.1        0.10 ± 11%  perf-profile.self.cycles-pp.inc_zone_page_state
      0.03 ±100%      +0.1        0.10 ± 11%  perf-profile.self.cycles-pp.block_page_mkwrite
      0.01 ±173%      +0.1        0.09 ± 10%  perf-profile.self.cycles-pp.page_mapped
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.invalidate_inode_page
      0.10 ± 10%      +0.1        0.17 ±  6%  perf-profile.self.cycles-pp.__perf_sw_event
      0.10 ± 10%      +0.1        0.17 ± 10%  perf-profile.self.cycles-pp.page_fault
      0.00            +0.1        0.07 ± 22%  perf-profile.self.cycles-pp.fault_dirty_shared_page
      0.08 ±  8%      +0.1        0.16 ±  7%  perf-profile.self.cycles-pp.__block_commit_write
      0.00            +0.1        0.08 ± 11%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.09 ± 13%      +0.1        0.16 ± 11%  perf-profile.self.cycles-pp.mark_buffer_dirty
      0.16 ±  5%      +0.1        0.24 ±  4%  perf-profile.self.cycles-pp.xas_start
      0.01 ±173%      +0.1        0.09 ±  8%  perf-profile.self.cycles-pp.ext4_page_mkwrite
      0.35 ±  6%      +0.1        0.44 ±  7%  perf-profile.self.cycles-pp.mpage_process_page_bufs
      0.17 ±  2%      +0.1        0.27 ± 11%  perf-profile.self.cycles-pp.___perf_sw_event
      0.11 ±  9%      +0.1        0.22 ±  5%  perf-profile.self.cycles-pp.mpage_prepare_extent_to_map
      0.12 ± 13%      +0.1        0.23 ±  8%  perf-profile.self.cycles-pp.dec_zone_page_state
      0.44 ±  7%      +0.1        0.55 ±  6%  perf-profile.self.cycles-pp.___might_sleep
      0.20 ±  7%      +0.1        0.31 ±  6%  perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.28 ±  7%      +0.1        0.41 ±  4%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.23 ±  6%      +0.1        0.36 ±  4%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.22 ±  4%      +0.1        0.36 ±  8%  perf-profile.self.cycles-pp.do_wp_page
      0.20 ±  6%      +0.1        0.35 ±  4%  perf-profile.self.cycles-pp.do_page_fault
      0.10 ± 12%      +0.1        0.25 ±  5%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.17 ± 13%      +0.2        0.32 ±  8%  perf-profile.self.cycles-pp.deactivate_file_page
      0.26 ±  4%      +0.2        0.43 ±  7%  perf-profile.self.cycles-pp.handle_pte_fault
      0.22 ± 14%      +0.2        0.39 ±  7%  perf-profile.self.cycles-pp.invalidate_mapping_pages
      0.09 ±  9%      +0.2        0.27 ±  3%  perf-profile.self.cycles-pp.__block_write_begin_int
      0.29 ±  3%      +0.2        0.48 ±  6%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.21 ±  6%      +0.2        0.41 ±  3%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.49 ±  8%      +0.2        0.71 ±  5%  perf-profile.self.cycles-pp.page_mapping
      0.12 ± 27%      +0.2        0.35 ±  3%  perf-profile.self.cycles-pp.menu_select
      0.90 ±  7%      +0.2        1.14 ±  2%  perf-profile.self.cycles-pp.unlock_page
      0.25 ±  5%      +0.2        0.49 ±  4%  perf-profile.self.cycles-pp.ext4_bio_write_page
      0.33 ±  8%      +0.2        0.58 ±  6%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.17 ± 11%      +0.3        0.43 ±  8%  perf-profile.self.cycles-pp.__set_page_dirty_buffers
      0.36 ± 12%      +0.3        0.63 ±  4%  perf-profile.self.cycles-pp.release_pages
      0.35 ±  9%      +0.3        0.64 ±  4%  perf-profile.self.cycles-pp.handle_mm_fault
      0.51 ±  6%      +0.3        0.81 ±  8%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.22 ±  6%      +0.3        0.53 ±  7%  perf-profile.self.cycles-pp.test_clear_page_writeback
      1.61 ±  7%      +0.3        1.94 ±  3%  perf-profile.self.cycles-pp.filemap_map_pages
      0.54 ±  9%      +0.3        0.87 ±  5%  perf-profile.self.cycles-pp.xas_load
      0.86 ±  5%      +0.4        1.23 ±  4%  perf-profile.self.cycles-pp.ext4_finish_bio
      0.69 ±  9%      +0.4        1.06 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.86 ±  9%      +0.4        1.25 ±  4%  perf-profile.self.cycles-pp.find_get_entries
      0.30 ±  8%      +0.4        0.70 ± 17%  perf-profile.self.cycles-pp.ktime_get
      0.61 ±  6%      +0.4        1.06 ±  4%  perf-profile.self.cycles-pp.sync_regs
      0.76 ±  7%      +0.5        1.23 ±  5%  perf-profile.self.cycles-pp.get_io_u
      4.28 ±  6%      +0.6        4.92 ±  4%  perf-profile.self.cycles-pp.__memcpy_mcsafe
      1.09 ±  9%      +0.7        1.76 ±  5%  perf-profile.self.cycles-pp.native_irq_return_iret
      3.43 ± 16%      +3.3        6.72 ±  3%  perf-profile.self.cycles-pp.__memcpy_flushcache


                                                                                
                                 fio.write_bw_MBps                              
                                                                                
  6000 +--------------------------------------------------------------------+   
       |       O                                            O               |   
  5000 |-+ O O     O   O O O O O O O O O  O O O O O O O O O   O O           |   
       |         O    .+. .+.                                               |   
       |.+.+.+.+.+.+.+   +   +.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.|   
  4000 |-+                                                                  |   
       |                                                                    |   
  3000 |-+                                                                  |   
       |                                                                    |   
  2000 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
  1000 |-+                                                                  |   
       |                                                                    |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.read_clat_mean_us                          
                                                                                
  3.5e+06 +-----------------------------------------------------------------+   
          |.+                           .+        .+           .+.      +.  |   
    3e+06 |-++         .+.+   .+.+.+.+.+  +     .+  +     .++.+   +.+. +  +.|   
          |   +.+.+.+.+    +.+             +.+.+     +.+.+            +     |   
  2.5e+06 |-+                                                               |   
          |                                                                 |   
    2e+06 |-+                                                               |   
          |                                                                 |   
  1.5e+06 |-+       O        O O   O       O   O   O   O   OO O O           |   
          |   O O O   O   OO     O   O O O   O   O   O   O                  |   
    1e+06 |-+                                                               |   
          |                                                                 |   
   500000 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.read_clat_90__us                            
                                                                                
  9e+06 +-------------------------------------------------------------------+   
        |               +         +.                                    : + |   
  8e+06 |.+      .+.   + :       +  +.+.+      .+.+           +. .+.+  :   :|   
  7e+06 |-+:  +.+   +.+  : .+.+.+        :  +.+    :  +.+.+. +  +    + :   :|   
        |  : +            +              : +       : +      +         +     |   
  6e+06 |-+ +                             +         +                       |   
  5e+06 |-+                                                                 |   
        |                                                                   |   
  4e+06 |-+                                                                 |   
  3e+06 |-+                                                                 |   
        |   O O O O O   O O O O O O O O O O O O O O O O O O O O O           |   
  2e+06 |-+                                                                 |   
  1e+06 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.read_clat_95__us                           
                                                                                
  1.4e+07 +-----------------------------------------------------------------+   
          |.+                           .+        .+            +.+.    +.  |   
  1.2e+07 |-++          +.    .+.+.+.+.+  +     .+  +     .++. +    +. +  + |   
          |   +.+.+.+. +  ++.+             +.+.+     +.+.+    +       +    +|   
    1e+07 |-+         +                                                     |   
          |                                                                 |   
    8e+06 |-+                                                               |   
          |                                                                 |   
    6e+06 |-+                                                               |   
          |                                                                 |   
    4e+06 |-+                  O                                            |   
          |   O O O O O   OO O   O O O O O O O O O O O O O OO O O           |   
    2e+06 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.read_clat_99__us                           
                                                                                
  2.5e+07 +-----------------------------------------------------------------+   
          |                                                                 |   
          |                                                     +.          |   
    2e+07 |.+.          +      +.       .+.       .+.      ++. +  +.+   +.  |   
          |   +.       : +    +  +.+.+.+   +.   .+   +.   +   +      + +  +.|   
          |     +.+.+. :  ++.+               +.+       +.+            +     |   
  1.5e+07 |-+         +                                                     |   
          |                                                                 |   
    1e+07 |-+                                                               |   
          |                  O                                              |   
          |     O          O   O O   O O O     O O     O O OO O O           |   
    5e+06 |-+ O   O O O   O        O       O O     O O                      |   
          |                                                                 |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.latency_500us_                               
                                                                                
  25 +----------------------------------------------------------------------+   
     |                                                                      |   
     |                                                                      |   
  20 |-+ O   O                  O O O    O O   O O O   O       O            |   
     |     O   O  O   O O O   O       O                    O O              |   
     |                      O                O       O   O                  |   
  15 |-+                                                                    |   
     |                                                                      |   
  10 |-+                                                                    |   
     |      .+.+..+. .+.+.+.                .+.       .+.+. .+.      .+.   .|   
     |.+.+.+        +       +.+.+.+.+.+..+.+   +.+.+.+     +   +..+.+   +.+ |   
   5 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_2ms_                                
                                                                                
  25 +----------------------------------------------------------------------+   
     |     O O              O O       O        O       O O O O              |   
     |   O        O   O O O     O O O    O O     O O           O            |   
  20 |-+       O                                                            |   
     |                                                                      |   
     |                                                                      |   
  15 |.+. .+.+.+..+.+.+.+.+.+.+.+.+.+.+.. .+.+.+.+. .+.+.+. .+.+..+.+. .+.+.|   
     |   +                               +         +       +          +     |   
  10 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
   5 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_4ms_                                
                                                                                
  35 +----------------------------------------------------------------------+   
     |                                                                      |   
  30 |.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+. .+.+.    .+.+.+.+.|   
     |                                                   +     +..+         |   
  25 |-+ O   O O            O   O        O O     O O                        |   
     |     O      O   O O O   O   O O O      O O     O O O O O O            |   
  20 |-+                                                                    |   
     |                                                                      |   
  15 |-+                                                                    |   
     |                                                                      |   
  10 |-+                                                                    |   
     |                                                                      |   
   5 |-+                                                                    |   
     |                                                                      |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_10ms_                               
                                                                                
  10 +----------------------------------------------------------------------+   
   9 |-+             .+         +                                       +.+ |   
     |             .+  +   .+. + +                          .+.+..     +   :|   
   8 |.+. .+.+.+..+     +.+   +   +.+.+.. .+.+.+.+. .+.+.+.+      +.+.+    :|   
   7 |-+ +                               +         +                        |   
     |                                                                      |   
   6 |-+                                                                    |   
   5 |-+                                                                    |   
   4 |-+                                                                    |   
     |                                                                      |   
   3 |-+                  O O       O O      O       O     O O              |   
   2 |-+ O O O O  O   O O     O O O      O O   O O O   O O     O            |   
     |                                                                      |   
   1 |-+                                                                    |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_20ms_                               
                                                                                
  8 +-----------------------------------------------------------------------+   
    |                                                                   +.  |   
  7 |.+             .+          +.   .+        .+                      :  + |   
  6 |-+:           +  :    +. ..  +.+  :     .+  +      .+. .+.+.+..+. :   :|   
    |  :   .+..+  :   :   +  +         :   .+     +   .+   +          +    :|   
  5 |-+ +.+     + :    +.+              +.+        +.+                      |   
    |            +                                                          |   
  4 |-+                                                                     |   
    |                                                                       |   
  3 |-+                                                                     |   
  2 |-+                                                                     |   
    |                                                                       |   
  1 |-+                                                        O            |   
    |   O O O  O O   O O O O O  O O O O O O O O O  O O O O O O              |   
  0 +-----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.time.system_time                             
                                                                                
  3500 +--------------------------------------------------------------------+   
       |               +                                                +.  |   
  3000 |-+            + +       .+.                            .+. .+  :  + |   
       |.+. .+.+.+. .+   +.+.+.+   +.+.+.. .+.+.+.+. .+.+.+.+.+   +  + :   +|   
  2500 |-+ +       +                      +         +                 +     |   
       |                                                                    |   
  2000 |-+                                                                  |   
       |   O   O O O   O O O O O O O O O  O O O O O O O O O O O O           |   
  1500 |-+   O                                                              |   
       |                                                                    |   
  1000 |-+                                                                  |   
       |                                                                    |   
   500 |-+                                                                  |   
       |                                                                    |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       fio.time.percent_of_cpu_this_job_got                     
                                                                                
  2000 +--------------------------------------------------------------------+   
  1800 |-+                                                              +.  |   
       |              .+.       .+.                            .+. .+  :  + |   
  1600 |.+. .+.+.+.+.+   +.+.+.+   +.+.+.. .+.+.+.+. .+.+.+.+.+   +  + :   +|   
  1400 |-+ +                              +         +                 +     |   
       |                                                                    |   
  1200 |-+ O   O   O       O O   O   O    O O     O O       O   O           |   
  1000 |-+   O   O     O O     O   O   O      O O     O O O   O             |   
   800 |-+                                                                  |   
       |                                                                    |   
   600 |-+                                                                  |   
   400 |-+                                                                  |   
       |                                                                    |   
   200 |-+                                                                  |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              fio.time.major_page_faults                        
                                                                                
    3e+08 +-----------------------------------------------------------------+   
          |                                                                 |   
  2.5e+08 |-+            .+       .+.                                  .+.+.|   
          |.+.+.+.+.+.+.+  +.+.+.+   +.+.+.+.+.+.+.+.+.+.+.++.+. .+.+.+     |   
          |                                                     +           |   
    2e+08 |-+ O O O   O   OO O O O O O O O O O O O O O O O OO O O           |   
          |         O                                                       |   
  1.5e+08 |-+                                                               |   
          |                                                                 |   
    1e+08 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    5e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              fio.time.minor_page_faults                        
                                                                                
  2.5e+08 +-----------------------------------------------------------------+   
          |   O   O            O           O       O                        |   
          |     O     O   OO O   O O O O O   O O O   O O O OO O O           |   
    2e+08 |-+       O                                                       |   
          |                                                                 |   
          |                                                                 |   
  1.5e+08 |-+            .+ .+.                              .+.+.     .+.  |   
          |.+.+.+.+.+.+.+  +   +.+.+.+.+.+.+.+.+.+.+.+.+.+.++     +.+.+   +.|   
    1e+08 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    5e+07 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              fio.time.file_system_inputs                       
                                                                                
  2.5e+09 +-----------------------------------------------------------------+   
          |                                                                 |   
          |               +                                                 |   
    2e+09 |.+  .+.+.+.+. + +.+.   .+.+.+.   .+.+.+.   .+.+.  .+.   .+. .+.+.|   
          | +.+         +      +.+       +.+       +.+     ++   +.+   +     |   
          |   O O O   O   OO   O O O O O   O O     O O      O   O           |   
  1.5e+09 |-+       O        O           O     O O     O O O  O             |   
          |                                                                 |   
    1e+09 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    5e+08 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc1-00019-g323c95f0957c8a" of type "text/plain" (203610 bytes)

View attachment "job-script" of type "text/plain" (8332 bytes)

View attachment "job.yaml" of type "text/plain" (5854 bytes)

View attachment "reproduce" of type "text/plain" (892 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ