lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Sat, 20 Jun 2020 22:36:57 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Kent Overstreet <kent.overstreet@...il.com>
Cc:     linux-kernel@...r.kernel.org, akpm@...ux-foundation.org,
        viro@...iv.linux.org.uk, linux-mm@...ck.org,
        linux-fsdevel@...r.kernel.org,
        Kent Overstreet <kent.overstreet@...il.com>, lkp@...ts.01.org
Subject: [fs] 420772c473: vm-scalability.throughput 41.7% improvement

Greeting,

FYI, we noticed a 41.7% improvement of vm-scalability.throughput due to commit:


commit: 420772c4738950463682fcfb1c52aa946c3bd537 ("[PATCH 2/2] fs: generic_file_buffered_read() now uses find_get_pages_contig")
url: https://github.com/0day-ci/linux/commits/Kent-Overstreet/generic_file_buffered_read-refactoring-optimization/20200610-081203


in testcase: vm-scalability
on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
with following parameters:

	runtime: 300s
	test: lru-file-readtwice
	cpufreq_governor: performance
	ucode: 0xd6

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/lkp-cfl-e1/lru-file-readtwice/vm-scalability/0xd6

commit: 
  b987172564 ("fs: Break generic_file_buffered_read up into multiple functions")
  420772c473 ("fs: generic_file_buffered_read() now uses find_get_pages_contig")

b98717256411f1a9 420772c4738950463682fcfb1c5 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    496998           +44.2%     716762        vm-scalability.median
  15886734           +41.7%   22513243        vm-scalability.throughput
   1164694 ±  4%     -40.8%     689654        vm-scalability.time.involuntary_context_switches
    116.08           +51.3%     175.59        vm-scalability.time.user_time
 4.766e+09           +41.7%  6.754e+09        vm-scalability.workload
      2.61            +1.2        3.85        mpstat.cpu.all.usr%
    343.25 ± 15%     -28.0%     247.25 ± 19%  slabinfo.kmalloc-rcl-128.active_objs
    343.25 ± 15%     -28.0%     247.25 ± 19%  slabinfo.kmalloc-rcl-128.num_objs
      1244 ± 12%     -34.0%     821.00 ±  3%  slabinfo.numa_policy.active_objs
      1336 ±  8%     +16.6%       1557        slabinfo.numa_policy.num_objs
     94.00            -1.1%      93.00        vmstat.cpu.sy
      6.00          +862.5%      57.75        vmstat.memory.buff
      8119 ±  5%     -43.9%       4556        vmstat.system.cs
     34132            -2.1%      33401        vmstat.system.in
  19100320 ±  3%     -98.5%     280481        meminfo.Active
  18823639 ±  3%    -100.0%       5150 ± 60%  meminfo.Active(file)
     23156 ± 33%     +66.1%      38454 ± 27%  meminfo.AnonHugePages
  10989916 ±  5%    +170.9%   29770220        meminfo.Inactive
  10961062 ±  5%    +171.3%   29741482        meminfo.Inactive(file)
     80369           -67.3%      26311 ± 24%  cpuidle.C1E.usage
   4042024 ± 12%    +452.9%   22347960 ± 47%  cpuidle.C3.time
     19096 ± 18%    +542.1%     122618 ± 52%  cpuidle.C3.usage
     69171 ± 30%     +72.4%     119250 ± 30%  cpuidle.C6.usage
  33916657 ± 57%     -99.0%     347090 ± 23%  cpuidle.C8.time
     38605 ± 57%     -99.1%     365.00 ± 22%  cpuidle.C8.usage
      3128 ± 11%     +21.6%       3803 ± 12%  cpuidle.POLL.usage
     56362 ±  3%     +20.6%      67998        softirqs.CPU0.RCU
     55215 ±  5%     +30.1%      71853 ±  3%  softirqs.CPU10.RCU
     56575 ±  4%     +23.8%      70026 ±  4%  softirqs.CPU12.RCU
     56725 ±  7%     +17.1%      66442 ±  7%  softirqs.CPU14.RCU
     52704 ± 14%     +34.8%      71062 ±  9%  softirqs.CPU3.RCU
     56448 ±  3%     +21.1%      68384 ±  4%  softirqs.CPU5.RCU
     54867 ±  3%     +15.8%      63542 ±  6%  softirqs.CPU7.RCU
     54642 ±  6%     +23.9%      67714        softirqs.CPU8.RCU
     57982 ±  3%     +23.0%      71314 ±  3%  softirqs.CPU9.RCU
    879397 ±  3%     +21.1%    1064887        softirqs.RCU
     45.58 ±  7%     -28.9%      32.42 ±  7%  sched_debug.cfs_rq:/.load_avg.min
   1108728 ± 21%     +35.4%    1501386 ± 24%  sched_debug.cfs_rq:/.min_vruntime.min
      1050 ±  7%     -16.5%     877.29 ±  8%  sched_debug.cfs_rq:/.runnable_avg.min
    132410 ± 14%    +104.9%     271253 ±  7%  sched_debug.cpu.avg_idle.avg
    829517 ±  6%     +13.9%     944799 ±  4%  sched_debug.cpu.avg_idle.max
    223109 ±  8%     +45.7%     325043 ±  4%  sched_debug.cpu.avg_idle.stddev
     82912 ±  4%     -41.8%      48227 ±  2%  sched_debug.cpu.nr_switches.avg
    117117 ±  4%     -40.8%      69362 ± 11%  sched_debug.cpu.nr_switches.max
     59928 ± 17%     -33.5%      39840 ±  6%  sched_debug.cpu.nr_switches.min
     11450 ± 18%     -38.6%       7033 ± 25%  sched_debug.cpu.nr_switches.stddev
     79329 ±  4%     -43.5%      44804 ±  2%  sched_debug.cpu.sched_count.avg
    113053 ±  4%     -43.1%      64357 ± 10%  sched_debug.cpu.sched_count.max
     56349 ± 19%     -33.9%      37235 ±  6%  sched_debug.cpu.sched_count.min
     11110 ± 22%     -42.5%       6388 ± 25%  sched_debug.cpu.sched_count.stddev
     38653 ±  4%     -48.8%      19776 ±  3%  sched_debug.cpu.ttwu_count.avg
     71016 ±  9%     -50.9%      34855 ± 17%  sched_debug.cpu.ttwu_count.max
     26009 ± 21%     -39.3%      15777 ±  6%  sched_debug.cpu.ttwu_count.min
      9400 ± 16%     -50.7%       4631 ± 29%  sched_debug.cpu.ttwu_count.stddev
     25269           -46.0%      13649        sched_debug.cpu.ttwu_local.avg
     41880 ±  7%     -43.4%      23709 ± 13%  sched_debug.cpu.ttwu_local.max
      5338 ± 21%     -40.9%       3154 ± 22%  sched_debug.cpu.ttwu_local.stddev
    741.50 ± 71%    +616.2%       5310 ±134%  interrupts.133:IR-PCI-MSI.2097154-edge.eth1-TxRx-1
     22457 ± 14%     -60.4%       8898 ±  7%  interrupts.CPU0.RES:Rescheduling_interrupts
     23691 ± 11%     -60.8%       9280 ± 10%  interrupts.CPU1.RES:Rescheduling_interrupts
     23162 ± 15%     -59.9%       9296 ±  4%  interrupts.CPU10.RES:Rescheduling_interrupts
     23589 ± 12%     -62.2%       8923 ±  9%  interrupts.CPU11.RES:Rescheduling_interrupts
     23050 ± 13%     -59.0%       9448 ± 18%  interrupts.CPU12.RES:Rescheduling_interrupts
     23024 ± 14%     -61.2%       8939 ± 13%  interrupts.CPU13.RES:Rescheduling_interrupts
     23480 ± 15%     -62.7%       8753 ±  6%  interrupts.CPU14.RES:Rescheduling_interrupts
     22921 ± 14%     -63.1%       8464 ±  8%  interrupts.CPU15.RES:Rescheduling_interrupts
    741.50 ± 71%    +616.2%       5310 ±134%  interrupts.CPU2.133:IR-PCI-MSI.2097154-edge.eth1-TxRx-1
     23039 ± 16%     -59.1%       9417 ±  9%  interrupts.CPU2.RES:Rescheduling_interrupts
     23216 ± 12%     -58.2%       9709 ± 11%  interrupts.CPU3.RES:Rescheduling_interrupts
     22805 ± 15%     -60.0%       9118 ±  8%  interrupts.CPU4.RES:Rescheduling_interrupts
     22900 ± 14%     -62.4%       8599 ±  8%  interrupts.CPU5.RES:Rescheduling_interrupts
     22699 ± 15%     -61.6%       8712 ±  7%  interrupts.CPU6.RES:Rescheduling_interrupts
     22349 ± 14%     -63.3%       8207 ±  8%  interrupts.CPU7.RES:Rescheduling_interrupts
     11476           -38.8%       7029 ± 34%  interrupts.CPU8.NMI:Non-maskable_interrupts
     11476           -38.8%       7029 ± 34%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
     22696 ± 13%     -60.4%       8991 ± 17%  interrupts.CPU8.RES:Rescheduling_interrupts
     23216 ± 14%     -60.5%       9170 ± 12%  interrupts.CPU9.RES:Rescheduling_interrupts
    368298 ± 14%     -60.9%     143930 ±  7%  interrupts.RES:Rescheduling_interrupts
    287.75 ±  9%     +28.1%     368.50 ±  3%  interrupts.TLB:TLB_shootdowns
   1243480 ± 12%     -77.6%     278443 ±  3%  proc-vmstat.allocstall_movable
     90453 ± 17%     -50.2%      45084 ±  6%  proc-vmstat.allocstall_normal
     95390 ± 12%     -43.4%      53979 ± 24%  proc-vmstat.compact_daemon_migrate_scanned
     14702 ± 32%     -84.7%       2249 ± 30%  proc-vmstat.compact_fail
    155958 ± 15%     -33.3%     104069 ± 19%  proc-vmstat.compact_isolated
     98711 ± 13%     -33.3%      65866 ± 17%  proc-vmstat.compact_migrate_scanned
     14708 ± 32%     -84.6%       2259 ± 30%  proc-vmstat.compact_stall
     67.00 ± 13%   +2743.7%       1905 ±  6%  proc-vmstat.kswapd_low_wmark_hit_quickly
   4703321 ±  3%    -100.0%       1287 ± 60%  proc-vmstat.nr_active_file
   2740966 ±  5%    +171.3%    7437227        proc-vmstat.nr_inactive_file
    298.75           -32.3%     202.25 ±  5%  proc-vmstat.nr_isolated_file
     12617            -3.7%      12154        proc-vmstat.nr_shmem
    306.00 ±  4%     -89.0%      33.75 ±  7%  proc-vmstat.nr_vmscan_immediate_reclaim
   4703315 ±  3%    -100.0%       1287 ± 60%  proc-vmstat.nr_zone_active_file
   2740872 ±  5%    +171.3%    7437151        proc-vmstat.nr_zone_inactive_file
 8.985e+08            +6.1%  9.536e+08        proc-vmstat.numa_hit
 8.985e+08            +6.1%  9.536e+08        proc-vmstat.numa_local
     68.50 ± 13%   +2683.9%       1907 ±  6%  proc-vmstat.pageoutrun
 2.945e+08 ±  6%    -100.0%       8238        proc-vmstat.pgactivate
  49475350           +24.2%   61466989        proc-vmstat.pgalloc_dma32
 8.491e+08 ±  2%      +5.1%  8.924e+08        proc-vmstat.pgalloc_normal
 3.077e+08 ±  7%    -100.0%       2.25 ±173%  proc-vmstat.pgdeactivate
 8.985e+08            +6.1%  9.533e+08        proc-vmstat.pgfree
     76852 ± 15%     -34.2%      50555 ± 20%  proc-vmstat.pgmigrate_success
 3.077e+08 ±  7%    -100.0%       2.25 ±173%  proc-vmstat.pgrefill
 6.643e+08 ±  2%      -9.8%  5.992e+08 ±  2%  proc-vmstat.pgscan_direct
 2.234e+08           +52.1%  3.399e+08        proc-vmstat.pgscan_kswapd
 6.643e+08 ±  2%      -9.8%  5.992e+08 ±  2%  proc-vmstat.pgsteal_direct
 2.234e+08           +52.1%  3.399e+08        proc-vmstat.pgsteal_kswapd
  13919902           +23.8%   17230370        proc-vmstat.slabs_scanned
  11712173 ±  4%    -100.0%     607.00 ±128%  proc-vmstat.workingset_activate
   3901977           +48.5%    5796193        proc-vmstat.workingset_nodereclaim
    923196            +1.2%     933876        proc-vmstat.workingset_nodes
 1.129e+08 ±  4%     -62.5%   42308674 ± 21%  proc-vmstat.workingset_refault
     40.46            -3.9%      38.89        perf-stat.i.MPKI
 5.173e+09            +9.5%  5.665e+09        perf-stat.i.branch-instructions
      0.53            +0.0        0.56        perf-stat.i.branch-miss-rate%
  26762450           +16.4%   31164299        perf-stat.i.branch-misses
     35.57            +7.7       43.25        perf-stat.i.cache-miss-rate%
 3.785e+08           +28.7%  4.872e+08        perf-stat.i.cache-misses
 1.066e+09            +5.2%  1.122e+09        perf-stat.i.cache-references
      8174 ±  5%     -44.0%       4577        perf-stat.i.context-switches
      2.48           -10.5%       2.22        perf-stat.i.cpi
 6.507e+10            -1.8%  6.391e+10        perf-stat.i.cpu-cycles
    174.51           -22.9%     134.58        perf-stat.i.cycles-between-cache-misses
      0.06 ±  4%      -0.0        0.04 ±  9%  perf-stat.i.dTLB-load-miss-rate%
   4137765 ±  4%     -28.2%    2972441 ±  9%  perf-stat.i.dTLB-load-misses
 7.157e+09           +12.2%  8.028e+09        perf-stat.i.dTLB-loads
 3.869e+09           +16.8%  4.519e+09        perf-stat.i.dTLB-stores
  15448634           +36.0%   21014900        perf-stat.i.iTLB-load-misses
     23320 ±  2%     +24.1%      28940 ±  6%  perf-stat.i.iTLB-loads
 2.615e+10            +9.4%   2.86e+10        perf-stat.i.instructions
      2287           -24.2%       1735 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.42           +10.1%       0.46        perf-stat.i.ipc
      4.07            -1.8%       4.00        perf-stat.i.metric.GHz
      0.80 ±  5%     -27.7%       0.58 ±  4%  perf-stat.i.metric.K/sec
      1095           +12.2%       1228        perf-stat.i.metric.M/sec
      0.00 ±  8%      -0.0        0.00 ± 12%  perf-stat.i.node-load-miss-rate%
    170.06           +53.7%     261.36 ± 14%  perf-stat.i.node-load-misses
  47528763 ±  4%    +125.2%  1.071e+08 ±  4%  perf-stat.i.node-loads
      0.00 ±  2%      +0.0        0.00 ± 14%  perf-stat.i.node-store-miss-rate%
    172.96           +51.2%     261.43 ± 14%  perf-stat.i.node-store-misses
 2.047e+08            +4.9%  2.148e+08        perf-stat.i.node-stores
     40.77            -3.8%      39.22        perf-stat.overall.MPKI
      0.52            +0.0        0.55        perf-stat.overall.branch-miss-rate%
     35.50            +7.9       43.43        perf-stat.overall.cache-miss-rate%
      2.49           -10.2%       2.23        perf-stat.overall.cpi
    171.91           -23.7%     131.20        perf-stat.overall.cycles-between-cache-misses
      0.06 ±  4%      -0.0        0.04 ±  9%  perf-stat.overall.dTLB-load-miss-rate%
      1692           -19.6%       1361        perf-stat.overall.instructions-per-iTLB-miss
      0.40           +11.3%       0.45        perf-stat.overall.ipc
      0.00 ±  5%      -0.0        0.00 ± 14%  perf-stat.overall.node-load-miss-rate%
      0.00            +0.0        0.00 ± 14%  perf-stat.overall.node-store-miss-rate%
      1679           -22.9%       1295        perf-stat.overall.path-length
 5.156e+09            +9.5%  5.647e+09        perf-stat.ps.branch-instructions
  26677747           +16.4%   31063524        perf-stat.ps.branch-misses
 3.773e+08           +28.7%  4.856e+08        perf-stat.ps.cache-misses
 1.063e+09            +5.2%  1.118e+09        perf-stat.ps.cache-references
      8146 ±  5%     -44.0%       4562        perf-stat.ps.context-switches
 6.485e+10            -1.8%  6.371e+10        perf-stat.ps.cpu-cycles
   4124690 ±  4%     -28.2%    2962937 ±  9%  perf-stat.ps.dTLB-load-misses
 7.134e+09           +12.2%  8.002e+09        perf-stat.ps.dTLB-loads
 3.857e+09           +16.8%  4.505e+09        perf-stat.ps.dTLB-stores
  15398712           +36.0%   20947207        perf-stat.ps.iTLB-load-misses
     23241 ±  2%     +24.1%      28847 ±  6%  perf-stat.ps.iTLB-loads
 2.607e+10            +9.4%  2.851e+10        perf-stat.ps.instructions
    169.52           +53.7%     260.52 ± 14%  perf-stat.ps.node-load-misses
  47371107 ±  4%    +125.3%  1.067e+08 ±  4%  perf-stat.ps.node-loads
    172.43           +51.1%     260.58 ± 14%  perf-stat.ps.node-store-misses
 2.041e+08            +4.9%  2.141e+08        perf-stat.ps.node-stores
 8.004e+12            +9.3%   8.75e+12        perf-stat.total.instructions
     37.73 ±  4%     -20.8       16.90 ±  7%  perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.__do_page_cache_readahead.ondemand_readahead.generic_file_buffered_read
     39.79 ±  4%     -18.3       21.44 ±  7%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.__do_page_cache_readahead.ondemand_readahead.generic_file_buffered_read.xfs_file_buffered_aio_read
     32.37 ±  5%     -17.5       14.85 ±  7%  perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.__do_page_cache_readahead.ondemand_readahead
     32.33 ±  5%     -17.5       14.84 ±  7%  perf-profile.calltrace.cycles-pp.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.__do_page_cache_readahead
     31.78 ±  5%     -17.0       14.74 ±  7%  perf-profile.calltrace.cycles-pp.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath
     32.31 ±  5%     -16.8       15.55 ±  7%  perf-profile.calltrace.cycles-pp.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask
     65.31 ±  3%     -16.7       48.57 ±  5%  perf-profile.calltrace.cycles-pp.__do_page_cache_readahead.ondemand_readahead.generic_file_buffered_read.xfs_file_buffered_aio_read.xfs_file_read_iter
     65.33 ±  3%     -16.7       48.59 ±  5%  perf-profile.calltrace.cycles-pp.ondemand_readahead.generic_file_buffered_read.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
     23.98 ±  4%      -9.3       14.72 ±  7%  perf-profile.calltrace.cycles-pp.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages
     15.55 ±  5%      -8.3        7.21 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node
      7.75 ± 30%      -7.8        0.00        perf-profile.calltrace.cycles-pp.shrink_active_list.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages
     13.91 ±  6%      -7.3        6.56 ±  9%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages
      7.16 ± 30%      -7.2        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.shrink_active_list.shrink_lruvec.shrink_node
      6.86 ± 30%      -6.9        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.shrink_active_list.shrink_lruvec.shrink_node.do_try_to_free_pages
     84.18 ±  2%      -4.2       80.01 ±  2%  perf-profile.calltrace.cycles-pp.generic_file_buffered_read.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read
     85.19            -3.7       81.50 ±  2%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read.ksys_read
      5.99 ±  6%      -3.3        2.73 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru
      6.04 ±  6%      -3.2        2.79 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru.iomap_readpages_actor
      4.79 ± 14%      -3.2        1.59 ± 13%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_slowpath.__alloc_pages_nodemask.__do_page_cache_readahead.ondemand_readahead
      7.72 ±  4%      -3.1        4.63 ±  6%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru.iomap_readpages_actor.iomap_apply
      4.59 ± 14%      -3.1        1.51 ± 13%  perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages_slowpath.__alloc_pages_nodemask.__do_page_cache_readahead
      7.92 ±  3%      -3.1        4.84 ±  6%  perf-profile.calltrace.cycles-pp.__lru_cache_add.add_to_page_cache_lru.iomap_readpages_actor.iomap_apply.iomap_readpages
      3.60 ± 11%      -2.5        1.12 ± 20%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec
      3.35 ± 11%      -2.4        0.95 ± 21%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__remove_mapping.shrink_page_list.shrink_inactive_list
     11.26 ±  2%      -2.4        8.86 ±  5%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.iomap_readpages_actor.iomap_apply.iomap_readpages.read_pages
      7.24 ±  4%      -2.3        4.90 ±  6%  perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      3.19 ± 13%      -2.2        1.03 ± 10%  perf-profile.calltrace.cycles-pp.rmqueue_bulk.rmqueue.get_page_from_freelist.__alloc_pages_slowpath.__alloc_pages_nodemask
      8.22 ±  3%      -2.0        6.21 ±  7%  perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages
      1.68 ± 26%      -1.0        0.68 ± 57%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat
      1.58 ± 20%      -0.7        0.93 ± 16%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.rmqueue_bulk.rmqueue.get_page_from_freelist
      1.12 ± 19%      -0.3        0.86 ± 14%  perf-profile.calltrace.cycles-pp.isolate_lru_pages.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages
      0.00            +0.5        0.55 ±  6%  perf-profile.calltrace.cycles-pp.down_read.xfs_ilock.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
      0.00            +0.6        0.56 ±  6%  perf-profile.calltrace.cycles-pp.touch_atime.generic_file_buffered_read.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
      0.00            +0.6        0.61 ± 12%  perf-profile.calltrace.cycles-pp.do_shrink_slab.shrink_slab.shrink_node.do_try_to_free_pages.try_to_free_pages
      0.00            +0.6        0.61 ± 12%  perf-profile.calltrace.cycles-pp.shrink_slab.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath
      0.00            +0.6        0.63 ±  6%  perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read
      0.00            +0.6        0.63 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      0.14 ±173%      +0.6        0.78 ±  4%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.65 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read
      0.00            +0.7        0.68 ±  6%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.02 ± 14%      +0.7        1.71 ±  4%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +0.7        0.72 ± 11%  perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_slab_page.new_slab
      0.00            +0.7        0.72 ± 11%  perf-profile.calltrace.cycles-pp.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_slab_page
      0.00            +0.7        0.73 ± 10%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_slab_page.new_slab.___slab_alloc.__slab_alloc
      0.00            +0.7        0.73 ± 10%  perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_slab_page.new_slab.___slab_alloc
      0.00            +0.7        0.73 ± 10%  perf-profile.calltrace.cycles-pp.new_slab.___slab_alloc.__slab_alloc.kmem_cache_alloc.xas_nomem
      0.00            +0.7        0.73 ± 10%  perf-profile.calltrace.cycles-pp.alloc_slab_page.new_slab.___slab_alloc.__slab_alloc.kmem_cache_alloc
      0.00            +0.7        0.74 ± 11%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.xas_nomem.__add_to_page_cache_locked.add_to_page_cache_lru.iomap_readpages_actor
      0.00            +0.7        0.74 ± 11%  perf-profile.calltrace.cycles-pp.__slab_alloc.kmem_cache_alloc.xas_nomem.__add_to_page_cache_locked.add_to_page_cache_lru
      0.00            +0.7        0.74 ± 11%  perf-profile.calltrace.cycles-pp.___slab_alloc.__slab_alloc.kmem_cache_alloc.xas_nomem.__add_to_page_cache_locked
      0.00            +0.8        0.75 ± 10%  perf-profile.calltrace.cycles-pp.xas_nomem.__add_to_page_cache_locked.add_to_page_cache_lru.iomap_readpages_actor.iomap_apply
      0.00            +0.9        0.93 ± 16%  perf-profile.calltrace.cycles-pp._raw_spin_lock.rmqueue_bulk.rmqueue.get_page_from_freelist.__alloc_pages_nodemask
      1.37 ± 15%      +1.0        2.33 ±  4%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      3.35 ± 12%      +1.0        4.31        perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_readpage_actor.iomap_readpages_actor.iomap_apply.iomap_readpages
      0.00            +1.0        1.01 ± 14%  perf-profile.calltrace.cycles-pp.__list_del_entry_valid.rmqueue_bulk.rmqueue.get_page_from_freelist.__alloc_pages_nodemask
      2.62 ± 12%      +1.0        3.65 ±  4%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.iomap_readpages_actor.iomap_apply.iomap_readpages
      1.60 ± 15%      +1.1        2.70 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      1.71 ± 14%      +1.2        2.89 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      0.00            +1.2        1.20 ±  7%  perf-profile.calltrace.cycles-pp.xas_load.find_get_pages_contig.generic_file_buffered_read.xfs_file_buffered_aio_read.xfs_file_read_iter
      2.09 ± 21%      +1.2        3.34 ± 10%  perf-profile.calltrace.cycles-pp.free_unref_page_list.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      1.39 ± 34%      +1.6        2.94 ± 11%  perf-profile.calltrace.cycles-pp.free_pcppages_bulk.free_unref_page_list.shrink_page_list.shrink_inactive_list.shrink_lruvec
      2.52 ± 15%      +1.7        4.23 ±  5%  perf-profile.calltrace.cycles-pp.write
      0.00            +2.4        2.38 ±  9%  perf-profile.calltrace.cycles-pp.rmqueue_bulk.rmqueue.get_page_from_freelist.__alloc_pages_nodemask.__do_page_cache_readahead
      1.36 ± 17%      +2.6        3.93 ±  3%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.__do_page_cache_readahead.ondemand_readahead.generic_file_buffered_read
      8.62 ± 10%      +2.7       11.36 ±  4%  perf-profile.calltrace.cycles-pp.memset_erms.iomap_readpage_actor.iomap_readpages_actor.iomap_apply.iomap_readpages
      0.13 ±173%      +3.2        3.28 ±  3%  perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages_nodemask.__do_page_cache_readahead.ondemand_readahead
     12.58 ± 10%      +3.8       16.39 ±  4%  perf-profile.calltrace.cycles-pp.iomap_readpage_actor.iomap_readpages_actor.iomap_apply.iomap_readpages.read_pages
      0.00            +4.5        4.50 ±  4%  perf-profile.calltrace.cycles-pp.find_get_pages_contig.generic_file_buffered_read.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
     10.27 ±  2%     +13.1       23.39 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.generic_file_buffered_read.xfs_file_buffered_aio_read
     10.38 ±  2%     +13.2       23.55 ±  3%  perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.generic_file_buffered_read.xfs_file_buffered_aio_read.xfs_file_read_iter
     11.01 ±  2%     +13.6       24.62 ±  3%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.generic_file_buffered_read.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
     39.16 ±  5%     -23.9       15.22 ±  5%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     38.34 ±  4%     -20.4       17.98 ±  6%  perf-profile.children.cycles-pp.__alloc_pages_slowpath
     40.43 ±  3%     -17.9       22.54 ±  6%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
     38.29 ±  2%     -17.2       21.06 ±  3%  perf-profile.children.cycles-pp.shrink_lruvec
     32.95 ±  5%     -17.1       15.88 ±  6%  perf-profile.children.cycles-pp.try_to_free_pages
     32.91 ±  5%     -17.0       15.87 ±  6%  perf-profile.children.cycles-pp.do_try_to_free_pages
     38.84 ±  2%     -16.9       21.89 ±  2%  perf-profile.children.cycles-pp.shrink_node
     65.32 ±  3%     -16.7       48.58 ±  5%  perf-profile.children.cycles-pp.__do_page_cache_readahead
     65.33 ±  3%     -16.7       48.59 ±  5%  perf-profile.children.cycles-pp.ondemand_readahead
     23.74 ±  6%     -15.1        8.64 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_irq
     29.71 ±  5%      -8.7       21.04 ±  3%  perf-profile.children.cycles-pp.shrink_inactive_list
     13.32 ±  8%      -8.6        4.72 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      8.54 ± 28%      -8.5        0.00        perf-profile.children.cycles-pp.shrink_active_list
     11.64 ±  9%      -6.9        4.71 ±  6%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
     84.19 ±  2%      -4.2       80.04 ±  2%  perf-profile.children.cycles-pp.generic_file_buffered_read
     85.21            -3.7       81.54 ±  2%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_read
      7.93 ±  3%      -3.1        4.85 ±  6%  perf-profile.children.cycles-pp.__lru_cache_add
     11.27 ±  2%      -2.4        8.88 ±  5%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      7.35 ±  4%      -2.1        5.26 ±  3%  perf-profile.children.cycles-pp.__remove_mapping
      0.85 ± 19%      -0.6        0.24 ± 25%  perf-profile.children.cycles-pp.worker_thread
      0.79 ± 20%      -0.6        0.20 ± 29%  perf-profile.children.cycles-pp.drain_local_pages_wq
      0.79 ± 20%      -0.6        0.20 ± 29%  perf-profile.children.cycles-pp.drain_pages
      0.81 ± 20%      -0.6        0.22 ± 25%  perf-profile.children.cycles-pp.process_one_work
      0.78 ± 21%      -0.6        0.20 ± 29%  perf-profile.children.cycles-pp.drain_pages_zone
      0.40 ± 21%      -0.3        0.08 ± 11%  perf-profile.children.cycles-pp.lru_add_drain
      0.36 ± 22%      -0.3        0.07 ± 11%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.43 ± 15%      -0.3        0.15 ± 29%  perf-profile.children.cycles-pp.workingset_refault
      0.79 ±  8%      -0.2        0.59 ±  5%  perf-profile.children.cycles-pp.advance_inactive_age
      0.49            -0.2        0.30 ±  5%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.30 ± 10%      -0.2        0.14 ±  3%  perf-profile.children.cycles-pp.__zone_watermark_ok
      0.49 ±  8%      -0.1        0.35 ±  5%  perf-profile.children.cycles-pp.__isolate_lru_page
      0.19 ± 54%      -0.1        0.08 ± 91%  perf-profile.children.cycles-pp.__get_free_pages
      0.21 ±  3%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.wake_all_kswapds
      0.22 ± 41%      -0.1        0.12 ± 35%  perf-profile.children.cycles-pp.forkshell
      0.20 ± 45%      -0.1        0.11 ± 40%  perf-profile.children.cycles-pp.__libc_fork
      0.26 ±  7%      -0.1        0.19 ±  6%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.09 ± 13%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.pte_alloc_one
      0.12 ± 13%      -0.0        0.07 ± 24%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.12 ± 11%      -0.0        0.08 ± 19%  perf-profile.children.cycles-pp.page_fault
      0.12 ± 10%      -0.0        0.07 ± 15%  perf-profile.children.cycles-pp.handle_mm_fault
      0.12 ± 10%      -0.0        0.07 ± 15%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.07 ± 12%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.super_cache_count
      0.22            -0.0        0.18 ±  7%  perf-profile.children.cycles-pp.release_pages
      0.09 ±  9%      -0.0        0.05 ± 62%  perf-profile.children.cycles-pp.do_fault
      0.28 ±  2%      -0.0        0.24 ±  7%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.27 ±  4%      -0.0        0.24 ±  4%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.17 ±  3%      -0.0        0.14 ±  3%  perf-profile.children.cycles-pp.__mod_zone_page_state
      0.09 ±  7%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
      0.08 ±  5%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_page_lruvec
      0.07 ±  7%      +0.0        0.08        perf-profile.children.cycles-pp.PageHuge
      0.07 ± 17%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.generic_file_read_iter
      0.11 ±  7%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.07 ±  7%      +0.0        0.09 ±  8%  perf-profile.children.cycles-pp.__x64_sys_read
      0.29 ±  7%      +0.0        0.32 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.17 ± 10%      +0.0        0.20 ±  6%  perf-profile.children.cycles-pp.free_unref_page_commit
      0.08 ± 10%      +0.0        0.12 ±  7%  perf-profile.children.cycles-pp.__x64_sys_write
      0.05            +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xas_alloc
      0.14 ±  5%      +0.0        0.18 ±  3%  perf-profile.children.cycles-pp.task_tick_fair
      0.01 ±173%      +0.0        0.05 ±  9%  perf-profile.children.cycles-pp.list_lru_add
      0.06 ±  7%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.timestamp_truncate
      0.10 ±  7%      +0.0        0.15 ±  7%  perf-profile.children.cycles-pp.write_null
      0.07 ± 12%      +0.0        0.12 ± 17%  perf-profile.children.cycles-pp.__vfs_read
      0.20 ±  3%      +0.0        0.25 ±  5%  perf-profile.children.cycles-pp.scheduler_tick
      0.01 ±173%      +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.xas_free_nodes
      0.12 ± 13%      +0.1        0.18 ±  6%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.45 ±  6%      +0.1        0.51 ±  2%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.08 ±  6%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.rcu_do_batch
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.iov_iter_init
      0.01 ±173%      +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.kmem_cache_free
      0.15 ±  5%      +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.rcu_all_qs
      0.08 ±  5%      +0.1        0.16 ±  5%  perf-profile.children.cycles-pp.rcu_core
      0.32 ±  4%      +0.1        0.40 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
      0.30 ±  4%      +0.1        0.38 ±  3%  perf-profile.children.cycles-pp.tick_sched_handle
      0.29 ±  3%      +0.1        0.37 ±  3%  perf-profile.children.cycles-pp.update_process_times
      0.12 ±  8%      +0.1        0.21 ±  8%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.00            +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.00            +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.12 ±  6%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.29 ±  4%      +0.1        0.39 ±  3%  perf-profile.children.cycles-pp._cond_resched
      0.14 ±  5%      +0.1        0.24 ±  3%  perf-profile.children.cycles-pp.current_time
      0.20 ±  2%      +0.1        0.30 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      0.41 ±  2%      +0.1        0.53 ±  4%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.26 ±  4%      +0.1        0.39 ±  4%  perf-profile.children.cycles-pp.xas_start
      0.22 ±  5%      +0.1        0.35 ±  6%  perf-profile.children.cycles-pp.rw_verify_area
      0.53 ±  3%      +0.1        0.68 ±  2%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.37 ±  3%      +0.2        0.52        perf-profile.children.cycles-pp.fsnotify_parent
      0.16 ±  2%      +0.2        0.32 ±  3%  perf-profile.children.cycles-pp.aa_file_perm
      0.27 ± 10%      +0.2        0.42 ±  2%  perf-profile.children.cycles-pp.up_read
      0.97 ± 10%      +0.2        1.13 ±  6%  perf-profile.children.cycles-pp.xas_create
      0.56 ±  2%      +0.2        0.72 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      0.22 ±  3%      +0.2        0.39 ±  4%  perf-profile.children.cycles-pp.__might_fault
      0.69 ±  3%      +0.2        0.88        perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.35 ±  9%      +0.2        0.54 ±  2%  perf-profile.children.cycles-pp.xfs_iunlock
      0.75 ±  3%      +0.2        0.96        perf-profile.children.cycles-pp.apic_timer_interrupt
      0.51            +0.2        0.72 ±  2%  perf-profile.children.cycles-pp.down_read
      1.50 ±  9%      +0.2        1.75 ±  4%  perf-profile.children.cycles-pp.xas_store
      0.33 ±  5%      +0.2        0.57 ±  8%  perf-profile.children.cycles-pp.atime_needs_update
      0.55 ±  2%      +0.2        0.80 ±  2%  perf-profile.children.cycles-pp.xfs_ilock
      0.46 ±  5%      +0.3        0.74 ±  2%  perf-profile.children.cycles-pp.__fget_light
      0.39 ±  4%      +0.3        0.69 ±  7%  perf-profile.children.cycles-pp.touch_atime
      0.49 ±  4%      +0.3        0.80 ±  2%  perf-profile.children.cycles-pp.shrink_slab
      0.48 ±  3%      +0.3        0.79 ±  3%  perf-profile.children.cycles-pp.do_shrink_slab
      0.54 ±  5%      +0.3        0.86 ±  2%  perf-profile.children.cycles-pp.__fdget_pos
      0.59 ±  3%      +0.3        0.93 ±  3%  perf-profile.children.cycles-pp.fsnotify
      0.32 ±  8%      +0.4        0.69 ±  3%  perf-profile.children.cycles-pp.shadow_lru_isolate
      0.33 ±  8%      +0.4        0.70 ±  3%  perf-profile.children.cycles-pp.__list_lru_walk_one
      0.33 ±  8%      +0.4        0.70 ±  3%  perf-profile.children.cycles-pp.list_lru_walk_one_irq
      0.91 ±  2%      +0.5        1.38 ±  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.76            +0.5        1.24 ±  3%  perf-profile.children.cycles-pp.common_file_perm
      0.89 ±  5%      +0.5        1.42 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.26 ± 38%      +0.7        0.92 ± 13%  perf-profile.children.cycles-pp.xas_nomem
      0.24 ± 41%      +0.7        0.91 ± 14%  perf-profile.children.cycles-pp.alloc_slab_page
      0.27 ± 39%      +0.7        0.94 ± 14%  perf-profile.children.cycles-pp.___slab_alloc
      0.24 ± 42%      +0.7        0.92 ± 14%  perf-profile.children.cycles-pp.new_slab
      0.27 ± 39%      +0.7        0.94 ± 13%  perf-profile.children.cycles-pp.__slab_alloc
      1.16 ±  3%      +0.7        1.84 ±  3%  perf-profile.children.cycles-pp.security_file_permission
      0.29 ± 35%      +0.7        0.98 ± 13%  perf-profile.children.cycles-pp.kmem_cache_alloc
      1.44 ±  4%      +0.8        2.20        perf-profile.children.cycles-pp.vfs_write
      1.90 ±  5%      +0.8        2.73 ±  2%  perf-profile.children.cycles-pp.xas_load
      2.83 ±  5%      +1.0        3.79 ±  3%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      3.43 ±  9%      +1.0        4.43 ±  5%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      1.88 ±  4%      +1.0        2.93        perf-profile.children.cycles-pp.ksys_write
      2.43 ± 19%      +1.5        3.92 ± 11%  perf-profile.children.cycles-pp.free_unref_page_list
      2.73 ± 14%      +1.8        4.57 ±  5%  perf-profile.children.cycles-pp.write
      8.64 ± 10%      +2.7       11.38 ±  4%  perf-profile.children.cycles-pp.memset_erms
     12.59 ± 10%      +3.8       16.41 ±  4%  perf-profile.children.cycles-pp.iomap_readpage_actor
      0.00            +4.5        4.51 ±  4%  perf-profile.children.cycles-pp.find_get_pages_contig
     10.34 ±  2%     +13.2       23.50 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
     10.38 ±  2%     +13.2       23.56 ±  3%  perf-profile.children.cycles-pp.copyout
     11.03 ±  2%     +13.6       24.66 ±  3%  perf-profile.children.cycles-pp.copy_page_to_iter
     39.15 ±  5%     -23.9       15.21 ±  5%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.01 ± 10%      -0.4        0.60 ±  2%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.78 ±  8%      -0.2        0.58 ±  5%  perf-profile.self.cycles-pp.advance_inactive_age
      0.28 ± 13%      -0.2        0.10 ± 34%  perf-profile.self.cycles-pp.workingset_refault
      0.49 ±  8%      -0.1        0.35 ±  5%  perf-profile.self.cycles-pp.__isolate_lru_page
      0.27 ± 11%      -0.1        0.12 ±  4%  perf-profile.self.cycles-pp.__zone_watermark_ok
      0.23 ±  8%      -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.__alloc_pages_slowpath
      0.19 ±  7%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.52 ±  5%      -0.1        0.45        perf-profile.self.cycles-pp.__remove_mapping
      0.33 ±  4%      -0.1        0.27 ±  6%  perf-profile.self.cycles-pp.isolate_lru_pages
      0.24 ±  6%      -0.1        0.18 ±  6%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.13 ±  6%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.wake_all_kswapds
      0.19 ±  3%      -0.0        0.15 ±  7%  perf-profile.self.cycles-pp.release_pages
      0.27 ±  4%      -0.0        0.23 ±  3%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.09 ± 12%      -0.0        0.06        perf-profile.self.cycles-pp.mem_cgroup_update_lru_size
      0.17 ±  3%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.__mod_zone_page_state
      0.08 ±  5%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.mem_cgroup_page_lruvec
      0.08 ±  8%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.xas_init_marks
      0.10 ± 15%      +0.0        0.12 ± 11%  perf-profile.self.cycles-pp.iomap_page_create
      0.06 ± 14%      +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.__x64_sys_read
      0.05            +0.0        0.07 ± 14%  perf-profile.self.cycles-pp.copyout
      0.08 ±  5%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_read
      0.04 ± 57%      +0.0        0.07 ±  6%  perf-profile.self.cycles-pp.PageHuge
      0.04 ± 58%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.generic_file_read_iter
      0.06 ±  6%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.xfs_ilock
      0.04 ± 58%      +0.0        0.07 ± 14%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.05 ±  9%      +0.0        0.09 ± 13%  perf-profile.self.cycles-pp.timestamp_truncate
      0.10 ± 10%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp._cond_resched
      0.06 ± 14%      +0.0        0.10 ± 24%  perf-profile.self.cycles-pp.__vfs_read
      0.09 ±  8%      +0.0        0.14 ±  8%  perf-profile.self.cycles-pp.__fdget_pos
      0.09 ± 12%      +0.0        0.13 ±  5%  perf-profile.self.cycles-pp.xfs_iunlock
      0.01 ±173%      +0.0        0.06 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.09 ±  7%      +0.0        0.14 ±  8%  perf-profile.self.cycles-pp.write_null
      0.11 ± 12%      +0.0        0.16 ±  5%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.__vfs_write
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.iov_iter_init
      0.06 ±  6%      +0.1        0.12 ±  5%  perf-profile.self.cycles-pp.touch_atime
      0.03 ±100%      +0.1        0.09 ±  5%  perf-profile.self.cycles-pp.current_time
      0.11 ±  3%      +0.1        0.17 ±  6%  perf-profile.self.cycles-pp.rcu_all_qs
      0.10 ±  5%      +0.1        0.16 ±  9%  perf-profile.self.cycles-pp.__might_fault
      0.31 ±  8%      +0.1        0.38 ±  2%  perf-profile.self.cycles-pp.xas_store
      0.12 ±  6%      +0.1        0.20 ±  5%  perf-profile.self.cycles-pp.ksys_write
      0.11 ±  7%      +0.1        0.18 ±  5%  perf-profile.self.cycles-pp.ksys_read
      0.18 ±  2%      +0.1        0.26 ±  4%  perf-profile.self.cycles-pp.__might_sleep
      0.12 ±  6%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.17 ±  8%      +0.1        0.27 ±  6%  perf-profile.self.cycles-pp.security_file_permission
      0.30 ±  2%      +0.1        0.41        perf-profile.self.cycles-pp.down_read
      0.24 ±  3%      +0.1        0.36 ±  5%  perf-profile.self.cycles-pp.xas_start
      0.30 ±  3%      +0.1        0.43 ±  4%  perf-profile.self.cycles-pp.new_sync_read
      0.19 ±  8%      +0.1        0.33 ± 14%  perf-profile.self.cycles-pp.atime_needs_update
      0.34 ±  2%      +0.1        0.48        perf-profile.self.cycles-pp.fsnotify_parent
      0.21 ±  6%      +0.1        0.35 ±  7%  perf-profile.self.cycles-pp.rw_verify_area
      0.23            +0.1        0.37 ±  4%  perf-profile.self.cycles-pp.vfs_read
      0.25 ± 10%      +0.1        0.39 ±  2%  perf-profile.self.cycles-pp.up_read
      0.23 ±  4%      +0.2        0.39 ±  2%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.15 ±  5%      +0.2        0.30 ±  4%  perf-profile.self.cycles-pp.aa_file_perm
      0.32 ±  6%      +0.2        0.47 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.54 ±  3%      +0.2        0.69 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      0.31 ±  5%      +0.2        0.47        perf-profile.self.cycles-pp.vfs_write
      0.30 ±  3%      +0.2        0.46 ±  3%  perf-profile.self.cycles-pp.do_syscall_64
      0.30 ± 16%      +0.2        0.50 ±  7%  perf-profile.self.cycles-pp.write
      0.32 ± 14%      +0.2        0.52 ±  6%  perf-profile.self.cycles-pp.read
      0.10 ± 10%      +0.2        0.34 ±  4%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.44 ±  5%      +0.3        0.70 ±  3%  perf-profile.self.cycles-pp.__fget_light
      0.45 ±  9%      +0.3        0.73 ±  4%  perf-profile.self.cycles-pp.xfs_file_read_iter
      0.49 ±  4%      +0.3        0.78 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.60 ±  2%      +0.3        0.92 ±  3%  perf-profile.self.cycles-pp.common_file_perm
      0.57 ±  2%      +0.3        0.90 ±  4%  perf-profile.self.cycles-pp.fsnotify
      0.90            +0.5        1.37 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.98 ± 17%      +0.6        1.61 ± 12%  perf-profile.self.cycles-pp.free_pcppages_bulk
      1.63 ±  6%      +0.7        2.32 ±  2%  perf-profile.self.cycles-pp.xas_load
      0.64 ±  4%      +0.7        1.35 ±  3%  perf-profile.self.cycles-pp.generic_file_buffered_read
      3.39 ±  9%      +1.0        4.35 ±  5%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      8.55 ± 10%      +2.7       11.23 ±  4%  perf-profile.self.cycles-pp.memset_erms
      0.00            +3.0        2.96 ±  5%  perf-profile.self.cycles-pp.find_get_pages_contig
     10.22 ±  2%     +13.0       23.26 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string


                                                                                
                           vm-scalability.time.user_time                        
                                                                                
  210 +---------------------------------------------------------------------+   
  200 |-+            O  O                         O                         |   
      |     O             O  O     O  O  O  O  O     O                      |   
  190 |-+      O  O             O                                           |   
  180 |-+O                                                                  |   
      |                                                 O  O O  O  O  O  O  |   
  170 |-+                                                                   |   
  160 |-+                                                                   |   
  150 |-+                                                                   |   
      |                                                                     |   
  140 |-+                                                                   |   
  130 |-+                                                                   |   
      |..  .+..  .+..+..+.  .+..+..+..                                      |   
  120 |-++.    +.         +.          +..+..  .+..+..  .+..+.+..+           |   
  110 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                   vm-scalability.time.involuntary_context_switches             
                                                                                
  1.4e+06 +-----------------------------------------------------------------+   
          |                            .+..      +.                         |   
  1.3e+06 |-+          .+..+.+..+..+..+    +.. ..  +..                      |   
          | .+..    .+.                       +       +..     +..           |   
  1.2e+06 |.+   +.+.                                     +.. :   +          |   
  1.1e+06 |-+                                                :              |   
          |                                                 +               |   
    1e+06 |-+                                                               |   
          |                                                                 |   
   900000 |-+                                                               |   
   800000 |-+                                                               |   
          |  O  O O  O  O  O O  O  O  O O  O  O  O O  O                     |   
   700000 |-+                                               O O  O     O O  |   
          |                                              O          O       |   
   600000 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               vm-scalability.throughput                        
                                                                                
  2.4e+07 +-----------------------------------------------------------------+   
          |     O O  O  O  O O        O O  O     O O  O                     |   
  2.3e+07 |-+O                  O  O          O          O  O O     O  O O  |   
  2.2e+07 |-+                                                    O          |   
          |                                                                 |   
  2.1e+07 |-+                                                               |   
    2e+07 |-+                                                               |   
          |                                                                 |   
  1.9e+07 |-+                                                               |   
  1.8e+07 |-+                                                               |   
          |                                                                 |   
  1.7e+07 |-+                                                               |   
  1.6e+07 |-.+.. .+..+..                                                    |   
          |.    +       +..+.+..+..+..+.+..+..+..+.+..+..+..+.+..+          |   
  1.5e+07 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.median                           
                                                                                
  750000 +------------------------------------------------------------------+   
         |  O  O O  O  O  O  O O  O  O  O  O    O  O  O O  O        O O     |   
  700000 |-+                                 O                O  O       O  |   
         |                                                                  |   
         |                                                                  |   
  650000 |-+                                                                |   
         |                                                                  |   
  600000 |-+                                                                |   
         |                                                                  |   
  550000 |-+                                                                |   
         |                                                                  |   
         | .+..    .+..                                                     |   
  500000 |.+   +.+.    +..+..+.+..+..+..+..+.+..+..+..+.+..+..+..+          |   
         |                                                                  |   
  450000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.workload                         
                                                                                
  7.5e+09 +-----------------------------------------------------------------+   
          |                                                                 |   
    7e+09 |-+   O    O  O  O               O     O O  O                     |   
          |  O    O          O  O  O  O O     O          O  O O     O  O O  |   
          |                                                      O          |   
  6.5e+09 |-+                                                               |   
          |                                                                 |   
    6e+09 |-+                                                               |   
          |                                                                 |   
  5.5e+09 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    5e+09 |-+                                                               |   
          |..+..+.+..+..+..+.+..+..+..+.+..+..+..+.+..+..+..+.+..+          |   
  4.5e+09 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.7.0-00002-g420772c4738950" of type "text/plain" (202612 bytes)

View attachment "job-script" of type "text/plain" (7658 bytes)

View attachment "job.yaml" of type "text/plain" (5298 bytes)

View attachment "reproduce" of type "text/plain" (2229 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ