lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200609032743.GB12456@shao2-debian>
Date:   Tue, 9 Jun 2020 11:27:43 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Kirill A. Shutemov" <kirill.shutemov@...ux.intel.com>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Zi Yan <ziy@...dia.com>,
        William Kucharski <william.kucharski@...cle.com>,
        Yang Shi <yang.shi@...ux.alibaba.com>,
        Andrea Arcangeli <aarcange@...hat.com>,
        John Hubbard <jhubbard@...dia.com>,
        Mike Kravetz <mike.kravetz@...cle.com>,
        Ralph Campbell <rcampbell@...dia.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [thp] 3917c80280: vm-scalability.throughput 10.8% improvement

Greeting,

FYI, we noticed a 10.8% improvement of vm-scalability.throughput due to commit:


commit: 3917c80280c93a7123f1a3a6dcdb10a3ea19737d ("thp: change CoW semantics for anon-THP")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: vm-scalability
on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
with following parameters:

	runtime: 300s
	size: 8T
	test: anon-cow-seq
	cpufreq_governor: performance
	ucode: 0xca

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/8T/lkp-cfl-e1/anon-cow-seq/vm-scalability/0xca

commit: 
  5503fbf2b0 ("khugepaged: allow to collapse PTE-mapped compound pages")
  3917c80280 ("thp: change CoW semantics for anon-THP")

5503fbf2b0b80c1a 3917c80280c93a7123f1a3a6dcd 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4          121%           4:4     perf-profile.calltrace.cycles-pp.sync_regs.error_entry.do_access
           :4          134%           5:4     perf-profile.calltrace.cycles-pp.error_entry.do_access
           :4            2%           0:4     perf-profile.children.cycles-pp.error_exit
           :4          146%           5:4     perf-profile.children.cycles-pp.error_entry
           :4           16%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    660126           +10.3%     728435        vm-scalability.median
  10518749           +10.8%   11649998        vm-scalability.throughput
   3172496        +21031.8%  6.704e+08        vm-scalability.time.minor_page_faults
      1464            -8.9%       1334        vm-scalability.time.percent_of_cpu_this_job_got
      1899           +18.4%       2249        vm-scalability.time.system_time
      2519           -29.5%       1775        vm-scalability.time.user_time
    139011        +37201.6%   51853599        vm-scalability.time.voluntary_context_switches
   7358853           -84.9%    1108967        meminfo.AnonHugePages
     61335           -10.8%      54711        meminfo.max_used_kB
      6.96            +9.1       16.06        mpstat.cpu.all.idle%
      0.00 ± 47%      +0.1        0.05        mpstat.cpu.all.iowait%
      0.02 ± 14%      +0.0        0.03 ±  8%  mpstat.cpu.all.soft%
     40.37            +6.7       47.06        mpstat.cpu.all.sys%
     52.65           -15.9       36.80        mpstat.cpu.all.usr%
      7.00          +128.6%      16.00        vmstat.cpu.id
     52.00           -30.8%      36.00        vmstat.cpu.us
     15.00           -13.3%      13.00        vmstat.procs.r
      2144 ±  2%  +15837.4%     341738        vmstat.system.cs
     32969            +6.2%      35011        vmstat.system.in
      8448 ±  9%     +23.9%      10471 ±  2%  slabinfo.anon_vma.active_objs
      8493 ±  9%     +23.8%      10513 ±  2%  slabinfo.anon_vma.num_objs
     13668           +11.9%      15297 ±  2%  slabinfo.anon_vma_chain.active_objs
     13788           +11.6%      15386 ±  2%  slabinfo.anon_vma_chain.num_objs
      1646 ±  3%     -11.4%       1459 ±  6%  slabinfo.cred_jar.active_objs
      1646 ±  3%     -11.4%       1459 ±  6%  slabinfo.cred_jar.num_objs
   2999045 ±  3%    +835.8%   28065296 ±  3%  cpuidle.C1.time
     88840         +3462.6%    3165020        cpuidle.C1.usage
   1988346 ± 18%   +2065.1%   43049885        cpuidle.C1E.time
     33980 ±  4%   +8298.3%    2853778        cpuidle.C1E.usage
 3.247e+08 ±  2%     +61.0%  5.229e+08 ±  3%  cpuidle.C6.time
    679395 ±  2%     +55.7%    1057712 ±  7%  cpuidle.C6.usage
    482.50 ± 38%     -37.5%     301.75 ± 15%  cpuidle.C8.usage
    388799 ±  2%  +31015.1%   1.21e+08        cpuidle.POLL.time
     58697 ±  3%  +81355.5%   47811937        cpuidle.POLL.usage
   2003883            -4.0%    1923005        proc-vmstat.nr_active_anon
   1976686            -3.1%    1915984        proc-vmstat.nr_anon_pages
      3537           -84.7%     539.75        proc-vmstat.nr_anon_transparent_hugepages
    577086            +1.6%     586102        proc-vmstat.nr_dirty_background_threshold
   1155583            +1.6%    1173639        proc-vmstat.nr_dirty_threshold
   5852604            +1.4%    5935736        proc-vmstat.nr_free_pages
   2003882            -4.0%    1923005        proc-vmstat.nr_zone_active_anon
   4864478        +13702.2%  6.714e+08        proc-vmstat.numa_hit
   4864478        +13702.2%  6.714e+08        proc-vmstat.numa_local
   3561377        +18728.3%  6.705e+08        proc-vmstat.pgfault
   1386571           -94.1%      81605        proc-vmstat.thp_fault_alloc
     10113 ±  8%    +519.1%      62610 ±  3%  softirqs.CPU0.SCHED
      9331 ±  5%    +527.6%      58565 ±  4%  softirqs.CPU1.SCHED
      9189 ±  7%    +526.1%      57535 ±  2%  softirqs.CPU10.SCHED
      8631 ±  2%    +557.5%      56756 ±  4%  softirqs.CPU11.SCHED
      8663 ±  3%    +548.7%      56196 ±  4%  softirqs.CPU12.SCHED
      8565 ±  2%    +575.5%      57862 ±  2%  softirqs.CPU13.SCHED
      8399 ±  4%    +595.1%      58386 ±  3%  softirqs.CPU14.SCHED
    130879           +10.8%     145009 ±  6%  softirqs.CPU14.TIMER
      8188 ±  2%    +618.8%      58859 ±  5%  softirqs.CPU15.SCHED
      8752 ±  4%    +588.2%      60233 ±  2%  softirqs.CPU2.SCHED
      8697 ±  5%    +566.7%      57990 ±  2%  softirqs.CPU3.SCHED
      8712 ±  4%    +567.8%      58182 ±  2%  softirqs.CPU4.SCHED
      8632 ±  4%    +586.5%      59260 ±  4%  softirqs.CPU5.SCHED
      8580 ±  2%    +582.6%      58569 ±  4%  softirqs.CPU6.SCHED
      8390 ±  5%    +587.8%      57707 ±  2%  softirqs.CPU7.SCHED
      9177 ±  7%    +552.0%      59839 ±  6%  softirqs.CPU8.SCHED
      8740 ±  5%    +594.0%      60661 ±  3%  softirqs.CPU9.SCHED
    140768          +567.2%     939218 ±  2%  softirqs.SCHED
      5662 ± 34%    +641.8%      42006 ± 15%  interrupts.CPU0.RES:Rescheduling_interrupts
     27.00 ±132%   +4233.3%       1170 ± 13%  interrupts.CPU0.TLB:TLB_shootdowns
      4210 ± 29%   +1093.2%      50235 ± 16%  interrupts.CPU1.RES:Rescheduling_interrupts
    107.50 ± 72%   +2426.7%       2716 ± 72%  interrupts.CPU1.TLB:TLB_shootdowns
      3334 ± 20%   +1551.3%      55065 ± 12%  interrupts.CPU10.RES:Rescheduling_interrupts
      3.50 ±111%  +1.5e+05%       5308 ± 90%  interrupts.CPU10.TLB:TLB_shootdowns
      5289 ± 44%    +793.8%      47278 ± 10%  interrupts.CPU11.RES:Rescheduling_interrupts
      6.50 ±164%  +21888.5%       1429 ± 48%  interrupts.CPU11.TLB:TLB_shootdowns
      4912 ± 36%    +725.2%      40536 ± 19%  interrupts.CPU12.RES:Rescheduling_interrupts
     47.50 ±167%   +5521.6%       2670 ±136%  interrupts.CPU12.TLB:TLB_shootdowns
      4654 ± 66%    +935.0%      48169 ± 14%  interrupts.CPU13.RES:Rescheduling_interrupts
      2.75 ± 78%  +64881.8%       1787 ±105%  interrupts.CPU13.TLB:TLB_shootdowns
      5537 ± 29%    +544.7%      35698 ±  7%  interrupts.CPU14.RES:Rescheduling_interrupts
      8.00 ±144%   +5481.2%     446.50 ± 23%  interrupts.CPU14.TLB:TLB_shootdowns
      2685 ± 17%   +1199.2%      34887 ± 10%  interrupts.CPU15.RES:Rescheduling_interrupts
     14.75 ±120%  +40433.9%       5978 ±107%  interrupts.CPU15.TLB:TLB_shootdowns
     10815            -9.4%       9796        interrupts.CPU2.NMI:Non-maskable_interrupts
     10815            -9.4%       9796        interrupts.CPU2.PMI:Performance_monitoring_interrupts
      7321 ± 18%    +613.6%      52249 ± 14%  interrupts.CPU2.RES:Rescheduling_interrupts
     81.25 ±151%   +1413.5%       1229 ± 32%  interrupts.CPU2.TLB:TLB_shootdowns
      5644 ± 48%    +694.2%      44828 ±  8%  interrupts.CPU3.RES:Rescheduling_interrupts
     66.50 ±151%   +1816.2%       1274 ± 28%  interrupts.CPU3.TLB:TLB_shootdowns
      3892 ± 43%   +1062.4%      45239 ± 17%  interrupts.CPU4.RES:Rescheduling_interrupts
    138.75 ± 95%   +3391.5%       4844 ±125%  interrupts.CPU4.TLB:TLB_shootdowns
      3676 ± 52%   +1230.1%      48904 ± 15%  interrupts.CPU5.RES:Rescheduling_interrupts
     89.50 ± 95%   +3545.5%       3262 ±112%  interrupts.CPU5.TLB:TLB_shootdowns
      4037 ± 10%    +931.2%      41636 ± 11%  interrupts.CPU6.RES:Rescheduling_interrupts
     54.50 ± 64%   +2661.9%       1505 ± 80%  interrupts.CPU6.TLB:TLB_shootdowns
      4766 ± 26%    +568.3%      31851 ±  3%  interrupts.CPU7.RES:Rescheduling_interrupts
    183.25 ±134%    +683.6%       1436 ± 52%  interrupts.CPU7.TLB:TLB_shootdowns
      3159 ± 12%   +1246.4%      42532 ± 18%  interrupts.CPU8.RES:Rescheduling_interrupts
     14.00 ± 86%  +19114.3%       2690 ± 86%  interrupts.CPU8.TLB:TLB_shootdowns
      4874 ± 57%    +821.6%      44921 ± 14%  interrupts.CPU9.RES:Rescheduling_interrupts
      3.50 ± 82%  +13557.1%     478.00 ± 29%  interrupts.CPU9.TLB:TLB_shootdowns
     73659 ±  5%    +858.5%     706041 ±  3%  interrupts.RES:Rescheduling_interrupts
    848.75 ± 29%   +4403.9%      38227 ± 31%  interrupts.TLB:TLB_shootdowns
    139259           -13.5%     120406 ±  9%  sched_debug.cfs_rq:/.exec_clock.avg
    140360           -13.5%     121442 ±  9%  sched_debug.cfs_rq:/.exec_clock.max
    138472           -13.9%     119222 ±  9%  sched_debug.cfs_rq:/.exec_clock.min
    313.46 ± 14%     +30.1%     407.95 ± 15%  sched_debug.cfs_rq:/.load_avg.max
     79.30 ± 17%     +29.8%     102.96 ±  9%  sched_debug.cfs_rq:/.load_avg.stddev
   2128119           -22.2%    1655733 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
   2140917           -22.1%    1668089 ±  8%  sched_debug.cfs_rq:/.min_vruntime.max
   2118085           -22.5%    1641599 ±  9%  sched_debug.cfs_rq:/.min_vruntime.min
     16.05 ± 15%     -77.8%       3.56 ±  7%  sched_debug.cfs_rq:/.nr_spread_over.avg
     49.25 ± 17%     -71.2%      14.18 ± 29%  sched_debug.cfs_rq:/.nr_spread_over.max
      4.50 ± 18%    -100.0%       0.00        sched_debug.cfs_rq:/.nr_spread_over.min
     11.14 ± 19%     -67.1%       3.66 ± 27%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      1186 ±  7%     +27.5%       1511 ±  4%  sched_debug.cfs_rq:/.util_avg.max
    630.50 ± 15%     -37.2%     396.17 ± 29%  sched_debug.cfs_rq:/.util_avg.min
    145.43 ±  8%     +78.5%     259.64 ± 17%  sched_debug.cfs_rq:/.util_avg.stddev
    770.84 ±  8%     -23.3%     591.44 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      1106 ± 14%     +17.9%       1304 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.max
    361.00 ± 31%     -89.9%      36.62 ±133%  sched_debug.cfs_rq:/.util_est_enqueued.min
    180.68 ± 23%     +65.7%     299.36 ±  9%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    201418 ± 10%     -67.9%      64655 ± 57%  sched_debug.cpu.avg_idle.avg
    730391 ± 12%     -49.7%     367051 ± 32%  sched_debug.cpu.avg_idle.max
     33542 ± 27%     -90.1%       3332 ± 31%  sched_debug.cpu.avg_idle.min
    190922 ± 15%     -48.2%      98900 ± 42%  sched_debug.cpu.avg_idle.stddev
      3665 ± 20%     -61.5%       1411 ± 44%  sched_debug.cpu.curr->pid.min
      0.32 ±  2%     +16.9%       0.37 ±  3%  sched_debug.cpu.nr_running.stddev
     22710 ±  2%  +13593.8%    3109859 ±  9%  sched_debug.cpu.nr_switches.avg
     31861 ±  3%  +11041.0%    3549706 ± 10%  sched_debug.cpu.nr_switches.max
     16774        +15584.7%    2631018 ± 10%  sched_debug.cpu.nr_switches.min
      4003 ±  6%   +6148.1%     250147 ± 15%  sched_debug.cpu.nr_switches.stddev
      9.11 ± 17%     +39.1%      12.66 ± 17%  sched_debug.cpu.nr_uninterruptible.stddev
     19756 ±  2%  +15625.1%    3106743 ±  9%  sched_debug.cpu.sched_count.avg
     27572 ±  7%  +12756.5%    3544868 ± 10%  sched_debug.cpu.sched_count.max
     15164        +17235.3%    2628827 ± 10%  sched_debug.cpu.sched_count.min
      3260 ± 10%   +7550.5%     249440 ± 15%  sched_debug.cpu.sched_count.stddev
      5498        +28055.7%    1548079 ±  9%  sched_debug.cpu.sched_goidle.avg
      7600 ±  6%  +23146.2%    1766874 ± 10%  sched_debug.cpu.sched_goidle.max
      4526 ±  3%  +28840.9%    1309937 ± 10%  sched_debug.cpu.sched_goidle.min
    805.82 ± 20%  +15342.8%     124441 ± 15%  sched_debug.cpu.sched_goidle.stddev
      9480 ±  2%  +16295.9%    1554415 ±  9%  sched_debug.cpu.ttwu_count.avg
     13441 ±  4%  +13065.1%    1769643 ± 10%  sched_debug.cpu.ttwu_count.max
      7232        +18299.0%    1330789 ± 10%  sched_debug.cpu.ttwu_count.min
      1772 ±  8%   +6474.7%     116549 ± 12%  sched_debug.cpu.ttwu_count.stddev
      3129 ±  8%     +21.3%       3796 ± 11%  sched_debug.cpu.ttwu_local.avg
      2118 ±  5%     +18.0%       2498 ±  8%  sched_debug.cpu.ttwu_local.min
     21.99            +3.9%      22.86        perf-stat.i.MPKI
 1.036e+10           +43.4%  1.486e+10        perf-stat.i.branch-instructions
      0.05 ± 21%      +0.1        0.14        perf-stat.i.branch-miss-rate%
   3055590 ± 19%    +469.6%   17405489        perf-stat.i.branch-misses
     60.68           -32.4       28.33        perf-stat.i.cache-miss-rate%
 4.339e+08           -15.4%  3.669e+08        perf-stat.i.cache-misses
 7.157e+08           +85.0%  1.324e+09        perf-stat.i.cache-references
      2138 ±  2%  +15996.1%     344180        perf-stat.i.context-switches
      1.90           -50.2%       0.95        perf-stat.i.cpi
 6.198e+10           -11.8%  5.465e+10        perf-stat.i.cpu-cycles
     82.19 ±  2%    +210.9%     255.53 ±  2%  perf-stat.i.cpu-migrations
    143.90            +4.3%     150.08        perf-stat.i.cycles-between-cache-misses
      0.00 ± 11%      +0.0        0.01 ± 11%  perf-stat.i.dTLB-load-miss-rate%
    119619 ± 12%    +377.9%     571700 ± 12%  perf-stat.i.dTLB-load-misses
 8.141e+09           +51.2%  1.231e+10        perf-stat.i.dTLB-loads
      0.00 ±  9%      +0.2        0.17        perf-stat.i.dTLB-store-miss-rate%
     45856 ± 10%  +14804.4%    6834622        perf-stat.i.dTLB-store-misses
  2.38e+09           +71.5%  4.081e+09        perf-stat.i.dTLB-stores
     78.49           -60.0       18.49        perf-stat.i.iTLB-load-miss-rate%
    179172         +1477.5%    2826391        perf-stat.i.iTLB-load-misses
     55866        +22136.0%   12422534        perf-stat.i.iTLB-loads
 3.257e+10           +77.4%  5.778e+10        perf-stat.i.instructions
    192416           -87.8%      23561 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.53          +100.3%       1.06        perf-stat.i.ipc
      3.87           -11.8%       3.41        perf-stat.i.metric.GHz
      1.31 ±  2%     -98.4%       0.02 ± 10%  perf-stat.i.metric.K/sec
      1367           +49.8%       2049        perf-stat.i.metric.M/sec
     11146        +19798.8%    2218071        perf-stat.i.minor-faults
      0.93 ± 77%     -80.0%       0.19 ±101%  perf-stat.i.node-load-misses
  29861059           +87.6%   56014604        perf-stat.i.node-loads
  2.61e+08           -41.4%  1.531e+08        perf-stat.i.node-stores
     11146        +19798.8%    2218071        perf-stat.i.page-faults
     21.98            +4.3%      22.92        perf-stat.overall.MPKI
      0.03 ± 19%      +0.1        0.12        perf-stat.overall.branch-miss-rate%
     60.63           -32.9       27.71        perf-stat.overall.cache-miss-rate%
      1.90           -50.3%       0.95        perf-stat.overall.cpi
    142.83            +4.3%     148.96        perf-stat.overall.cycles-between-cache-misses
      0.00 ± 12%      +0.0        0.00 ± 12%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 10%      +0.2        0.17        perf-stat.overall.dTLB-store-miss-rate%
     76.23           -57.7       18.54        perf-stat.overall.iTLB-load-miss-rate%
    181715           -88.8%      20430        perf-stat.overall.instructions-per-iTLB-miss
      0.53          +101.2%       1.06        perf-stat.overall.ipc
      0.00 ± 77%      -0.0        0.00 ±100%  perf-stat.overall.node-load-miss-rate%
      3261           +77.0%       5772        perf-stat.overall.path-length
 1.033e+10           +43.4%  1.481e+10        perf-stat.ps.branch-instructions
   3046784 ± 19%    +469.6%   17353801        perf-stat.ps.branch-misses
 4.325e+08           -15.5%  3.656e+08        perf-stat.ps.cache-misses
 7.133e+08           +85.0%  1.319e+09        perf-stat.ps.cache-references
      2131 ±  2%  +16007.5%     343275        perf-stat.ps.context-switches
 6.177e+10           -11.8%  5.446e+10        perf-stat.ps.cpu-cycles
     81.93 ±  2%    +210.9%     254.73 ±  2%  perf-stat.ps.cpu-migrations
    119268 ± 12%    +377.8%     569864 ± 12%  perf-stat.ps.dTLB-load-misses
 8.114e+09           +51.2%  1.227e+10        perf-stat.ps.dTLB-loads
     45706 ± 10%  +14801.0%    6810741        perf-stat.ps.dTLB-store-misses
 2.372e+09           +71.4%  4.067e+09        perf-stat.ps.dTLB-stores
    178615         +1477.9%    2818423        perf-stat.ps.iTLB-load-misses
     55684        +22132.1%   12379802        perf-stat.ps.iTLB-loads
 3.246e+10           +77.4%  5.758e+10        perf-stat.ps.instructions
     11109        +19796.1%    2210317        perf-stat.ps.minor-faults
      0.93 ± 77%     -80.1%       0.19 ±101%  perf-stat.ps.node-load-misses
  29757247           +87.6%   55810387        perf-stat.ps.node-loads
 2.601e+08           -41.4%  1.525e+08        perf-stat.ps.node-stores
     11109        +19796.1%    2210317        perf-stat.ps.page-faults
 9.833e+12           +77.0%   1.74e+13        perf-stat.total.instructions
     43.09           -43.1        0.00        perf-profile.calltrace.cycles-pp.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.page_fault
     41.25           -41.2        0.00        perf-profile.calltrace.cycles-pp.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
     41.08           -41.1        0.00        perf-profile.calltrace.cycles-pp.copy_subpage.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault
     40.70           -40.7        0.00        perf-profile.calltrace.cycles-pp.copy_page.copy_subpage.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault
     93.71           -29.7       64.02        perf-profile.calltrace.cycles-pp.do_access
      6.22 ± 10%      -4.9        1.34 ± 29%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.page_fault
      6.22 ± 11%      -4.9        1.36 ± 29%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.page_fault
      6.23 ± 10%      -4.8        1.42 ± 29%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.page_fault
      6.23 ± 10%      -4.8        1.43 ± 29%  perf-profile.calltrace.cycles-pp.page_fault
      0.00            +0.5        0.52        perf-profile.calltrace.cycles-pp.__perf_sw_event.do_user_addr_fault.page_fault.do_access
      0.00            +0.7        0.67        perf-profile.calltrace.cycles-pp.rmqueue_bulk.rmqueue.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma
      0.00            +0.7        0.67 ±  6%  perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      0.00            +0.7        0.74 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.__lock_page.do_wp_page
      0.00            +0.8        0.76 ±  6%  perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
      0.00            +0.8        0.76 ±  5%  perf-profile.calltrace.cycles-pp.schedule.io_schedule.__lock_page.do_wp_page.__handle_mm_fault
      0.00            +0.8        0.78 ±  5%  perf-profile.calltrace.cycles-pp.io_schedule.__lock_page.do_wp_page.__handle_mm_fault.handle_mm_fault
      0.00            +0.9        0.87 ±  3%  perf-profile.calltrace.cycles-pp.__pagevec_lru_add_fn.pagevec_lru_move_fn.__lru_cache_add.wp_page_copy.do_wp_page
      0.00            +0.9        0.95 ±  6%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.00            +1.1        1.08        perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.ptep_clear_flush.wp_page_copy
      0.00            +1.1        1.12 ±  5%  perf-profile.calltrace.cycles-pp.__lock_page.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
     37.02            +1.1       38.15        perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.page_fault.do_access
      0.00            +1.2        1.18 ± 11%  perf-profile.calltrace.cycles-pp.free_pcppages_bulk.free_unref_page_list.release_pages.tlb_flush_mmu.unmap_page_range
      0.00            +1.2        1.21        perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.ptep_clear_flush.wp_page_copy.do_wp_page
      0.00            +1.2        1.24 ±  8%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      0.00            +1.3        1.28        perf-profile.calltrace.cycles-pp.mem_cgroup_try_charge_delay.wp_page_copy.do_wp_page.__handle_mm_fault.handle_mm_fault
      0.00            +1.3        1.28 ±  2%  perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.wp_page_copy
      0.00            +1.3        1.31 ±  8%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +1.3        1.31 ±  8%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +1.3        1.32 ±  7%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
      0.00            +1.5        1.46 ±  2%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.wp_page_copy.do_wp_page
      0.00            +1.5        1.48 ±  8%  perf-profile.calltrace.cycles-pp.free_unref_page_list.release_pages.tlb_flush_mmu.unmap_page_range.unmap_vmas
      0.00            +1.6        1.63 ±  2%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.wp_page_copy.do_wp_page.__handle_mm_fault
      0.00            +1.7        1.67 ±  2%  perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.ptep_clear_flush.wp_page_copy.do_wp_page.__handle_mm_fault
      0.00            +1.8        1.81 ±  2%  perf-profile.calltrace.cycles-pp.ptep_clear_flush.wp_page_copy.do_wp_page.__handle_mm_fault.handle_mm_fault
      0.00            +1.8        1.83        perf-profile.calltrace.cycles-pp.__lru_cache_add.wp_page_copy.do_wp_page.__handle_mm_fault.handle_mm_fault
      0.00            +1.9        1.94 ±  2%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.wp_page_copy.do_wp_page.__handle_mm_fault
      0.00            +2.1        2.08 ± 25%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu.unmap_page_range
      0.00            +2.1        2.10 ± 25%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu.unmap_page_range.unmap_vmas
      0.00            +2.1        2.12 ±  6%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.do_wp_page
      0.00            +2.1        2.14 ±  6%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.do_wp_page.__handle_mm_fault
      0.00            +2.2        2.16 ±  2%  perf-profile.calltrace.cycles-pp.alloc_pages_vma.wp_page_copy.do_wp_page.__handle_mm_fault.handle_mm_fault
      0.00            +2.2        2.21 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.do_wp_page.__handle_mm_fault.handle_mm_fault
      0.86 ±  8%      +2.2        3.09 ±  5%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     20.49            +2.2       22.73        perf-profile.calltrace.cycles-pp.do_rw_once
      0.00            +2.6        2.56 ±  6%  perf-profile.calltrace.cycles-pp.wake_up_page_bit.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
     37.07            +2.6       39.64        perf-profile.calltrace.cycles-pp.do_user_addr_fault.page_fault.do_access
      0.00            +2.6        2.59 ±  9%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +4.6        4.58 ± 14%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.unmap_page_range.unmap_vmas.exit_mmap
      0.00            +4.8        4.79 ± 13%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.unmap_page_range.unmap_vmas.exit_mmap.mmput
      0.92 ±  8%      +4.9        5.77 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      0.92 ±  8%      +4.9        5.78 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.75 ±  8%      +5.0        5.73 ± 11%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.75 ±  8%      +5.0        5.73 ± 11%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.73 ±  9%      +5.0        5.72 ± 11%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.73 ±  9%      +5.0        5.72 ± 11%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.73 ±  9%      +5.0        5.72 ± 11%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.73 ±  8%      +5.0        5.72 ± 11%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.72 ±  8%      +5.0        5.72 ± 11%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      0.00            +5.5        5.48 ± 12%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      0.00            +5.5        5.49 ± 12%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      0.95 ±  8%      +6.2        7.14 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.95 ±  8%      +6.2        7.15 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      0.95 ±  8%      +6.2        7.16 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      1.01 ±  7%      +6.7        7.71 ±  5%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     37.12            +8.5       45.58        perf-profile.calltrace.cycles-pp.page_fault.do_access
      0.00            +9.9        9.87        perf-profile.calltrace.cycles-pp.copy_page.wp_page_copy.do_wp_page.__handle_mm_fault.handle_mm_fault
      0.00           +12.4       12.45        perf-profile.calltrace.cycles-pp.reuse_swap_page.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.00           +19.7       19.71        perf-profile.calltrace.cycles-pp.wp_page_copy.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.00           +37.5       37.46        perf-profile.calltrace.cycles-pp.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.page_fault
     43.09           -42.7        0.40 ±  2%  perf-profile.children.cycles-pp.do_huge_pmd_wp_page
     41.27           -41.3        0.00        perf-profile.children.cycles-pp.copy_user_huge_page
     41.09           -41.1        0.00        perf-profile.children.cycles-pp.copy_subpage
     40.93           -30.8       10.18        perf-profile.children.cycles-pp.copy_page
     42.97           -16.9       26.03        perf-profile.children.cycles-pp.do_rw_once
     71.30            -9.6       61.74        perf-profile.children.cycles-pp.do_access
     43.44            -4.3       39.18        perf-profile.children.cycles-pp.__handle_mm_fault
     43.48            -3.7       39.78        perf-profile.children.cycles-pp.handle_mm_fault
     43.33            -2.2       41.09        perf-profile.children.cycles-pp.do_user_addr_fault
      1.00 ±  2%      -0.5        0.49 ±  3%  perf-profile.children.cycles-pp.__irqentry_text_start
      0.68 ±  8%      -0.5        0.22 ±  9%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.77 ±  2%      -0.3        0.43 ±  4%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.65 ±  2%      -0.3        0.33 ±  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.52 ±  4%      -0.3        0.24 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.42 ±  5%      -0.2        0.17 ±  8%  perf-profile.children.cycles-pp.tick_sched_timer
      0.39 ±  4%      -0.2        0.15 ±  7%  perf-profile.children.cycles-pp.tick_sched_handle
      0.37 ±  5%      -0.2        0.15 ±  7%  perf-profile.children.cycles-pp.update_process_times
      0.25 ±  7%      -0.1        0.10 ±  7%  perf-profile.children.cycles-pp.scheduler_tick
      0.18 ±  4%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.task_tick_fair
      0.11 ± 13%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.io_serial_out
      0.09 ±  7%      -0.0        0.06        perf-profile.children.cycles-pp.irq_exit
      0.07 ±  6%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.12 ±  6%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.12 ±  6%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.12 ±  6%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.irq_work_run
      0.12 ±  6%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.printk
      0.08 ± 15%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp._cond_resched
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.try_charge
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.finish_task_switch
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.delayacct_end
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.pmd_pfn
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.00            +0.1        0.06 ± 20%  perf-profile.children.cycles-pp.update_cfs_rq_h_load
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.pmd_devmap_trans_unstable
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.update_cfs_group
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.kernel_text_address
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.get_vma_policy
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.uncharge_page
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.free_unref_page_prepare
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.check_preempt_curr
      0.01 ±173%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.native_write_msr
      0.00            +0.1        0.08        perf-profile.children.cycles-pp.__kernel_text_address
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.mem_cgroup_page_lruvec
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.vm_normal_page
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.delay_tsc
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.sched_clock
      0.00            +0.1        0.09 ±  5%  perf-profile.children.cycles-pp.perf_exclude_event
      0.00            +0.1        0.09        perf-profile.children.cycles-pp.page_mapping
      0.00            +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.mem_cgroup_from_task
      0.00            +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.07 ±  5%      +0.1        0.17 ±  8%  perf-profile.children.cycles-pp.update_curr
      0.00            +0.1        0.10 ±  5%  perf-profile.children.cycles-pp.orc_find
      0.00            +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.do_page_fault
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.__switch_to_asm
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.1        0.11 ±  3%  perf-profile.children.cycles-pp.lru_cache_add_active_or_unevictable
      0.00            +0.1        0.11 ±  3%  perf-profile.children.cycles-pp.__orc_find
      0.00            +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.00            +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.__switch_to
      0.10 ±  4%      +0.1        0.23 ±  6%  perf-profile.children.cycles-pp.___might_sleep
      0.00            +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.00            +0.1        0.13 ± 14%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.down_read_trylock
      0.00            +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.free_unref_page_commit
      0.00            +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.00            +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.__might_sleep
      0.00            +0.2        0.16 ±  5%  perf-profile.children.cycles-pp.up_read
      0.00            +0.2        0.16 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.17 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +0.2        0.18 ±  3%  perf-profile.children.cycles-pp.__mod_zone_page_state
      0.00            +0.2        0.18 ±  8%  perf-profile.children.cycles-pp.__count_memcg_events
      0.00            +0.2        0.18 ±  2%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.2        0.19        perf-profile.children.cycles-pp.unlock_page
      0.08 ± 10%      +0.2        0.28 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.2        0.21 ±  5%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      1.33 ±  7%      +0.2        1.54        perf-profile.children.cycles-pp.get_page_from_freelist
      0.00            +0.2        0.21 ±  3%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.2        0.22        perf-profile.children.cycles-pp.__do_huge_pmd_anonymous_page
      0.00            +0.2        0.23 ±  2%  perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.2        0.23 ±  7%  perf-profile.children.cycles-pp.set_next_entity
      0.00            +0.2        0.23 ±  5%  perf-profile.children.cycles-pp._find_next_bit
      0.00            +0.2        0.24 ±  4%  perf-profile.children.cycles-pp.vmacache_find
      0.00            +0.2        0.24 ±  7%  perf-profile.children.cycles-pp.cpumask_any_but
      0.00            +0.3        0.26 ±  4%  perf-profile.children.cycles-pp.find_vma
      0.00            +0.3        0.26 ±  4%  perf-profile.children.cycles-pp.__split_huge_pmd
      0.00            +0.3        0.26 ±  4%  perf-profile.children.cycles-pp.__split_huge_pmd_locked
      0.00            +0.3        0.26 ±  3%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.00            +0.3        0.27 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.00            +0.3        0.28 ±  7%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.3        0.29 ±  4%  perf-profile.children.cycles-pp.put_page
      0.00            +0.3        0.30 ±  2%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.00            +0.3        0.32 ±  4%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.00            +0.3        0.32 ±  6%  perf-profile.children.cycles-pp.page_add_new_anon_rmap
      0.00            +0.3        0.33 ±  6%  perf-profile.children.cycles-pp.dequeue_entity
      0.00            +0.3        0.33 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_throttle_swaprate
      0.00            +0.4        0.36 ±  2%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.00            +0.4        0.37 ±  4%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.00            +0.4        0.38 ±  5%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.00            +0.4        0.39 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.00            +0.4        0.39 ±  4%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.4        0.45 ±  4%  perf-profile.children.cycles-pp.unwind_next_frame
      0.00            +0.5        0.47 ±  3%  perf-profile.children.cycles-pp.page_remove_rmap
      0.00            +0.5        0.49 ±  5%  perf-profile.children.cycles-pp.menu_select
      0.00            +0.5        0.51 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_try_charge
      0.05 ±106%      +0.5        0.56 ± 11%  perf-profile.children.cycles-pp.start_kernel
      0.00            +0.6        0.58 ±  7%  perf-profile.children.cycles-pp.schedule_idle
      0.07 ±  7%      +0.7        0.72 ±  3%  perf-profile.children.cycles-pp.___perf_sw_event
      0.00            +0.7        0.69 ±  6%  perf-profile.children.cycles-pp.arch_stack_walk
      1.33 ±  7%      +0.7        2.04        perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.00            +0.7        0.70        perf-profile.children.cycles-pp.rmqueue_bulk
      0.00            +0.8        0.75 ±  3%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.8        0.78 ±  6%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.00            +0.8        0.78 ±  5%  perf-profile.children.cycles-pp.schedule
      0.00            +0.8        0.79 ±  5%  perf-profile.children.cycles-pp.io_schedule
      0.07 ±  7%      +0.8        0.91        perf-profile.children.cycles-pp.__perf_sw_event
      0.00            +0.9        0.90 ±  3%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.04 ± 58%      +0.9        0.96 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock
      1.32 ±  7%      +0.9        2.26        perf-profile.children.cycles-pp.alloc_pages_vma
      0.00            +1.0        0.98 ±  6%  perf-profile.children.cycles-pp.__account_scheduler_latency
     43.39            +1.0       44.44        perf-profile.children.cycles-pp.page_fault
      0.00            +1.1        1.12        perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.00            +1.1        1.15 ±  5%  perf-profile.children.cycles-pp.__lock_page
      0.00            +1.3        1.26        perf-profile.children.cycles-pp.flush_tlb_func_common
      0.00            +1.3        1.28 ± 10%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.00            +1.3        1.28 ±  7%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +1.3        1.28        perf-profile.children.cycles-pp.sync_regs
      0.05            +1.3        1.33 ±  4%  perf-profile.children.cycles-pp.__schedule
      0.00            +1.3        1.34 ±  7%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.00            +1.3        1.34 ±  7%  perf-profile.children.cycles-pp.activate_task
      0.00            +1.3        1.35 ±  7%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +1.4        1.39        perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay
      0.00            +1.6        1.61 ±  7%  perf-profile.children.cycles-pp.free_unref_page_list
      0.08 ±  5%      +1.6        1.72 ±  2%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.06            +1.7        1.74 ±  3%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.00            +1.7        1.74        perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.07            +1.9        1.95 ±  2%  perf-profile.children.cycles-pp.__lru_cache_add
      0.00            +1.9        1.88        perf-profile.children.cycles-pp.ptep_clear_flush
      0.07 ±  7%      +2.1        2.18 ±  5%  perf-profile.children.cycles-pp.try_to_wake_up
      0.03 ±100%      +2.2        2.18 ±  5%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.05 ±  8%      +2.2        2.26 ±  5%  perf-profile.children.cycles-pp.__wake_up_common
      0.92 ±  7%      +2.4        3.32 ±  6%  perf-profile.children.cycles-pp.intel_idle
      0.63 ± 14%      +2.5        3.14 ± 17%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.06 ± 11%      +2.6        2.61 ±  5%  perf-profile.children.cycles-pp.wake_up_page_bit
      0.00            +2.8        2.80 ±  9%  perf-profile.children.cycles-pp.poll_idle
      0.58 ± 15%      +2.9        3.49 ± 18%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.68 ±  8%      +4.3        4.94 ± 13%  perf-profile.children.cycles-pp.release_pages
      0.68 ±  8%      +4.3        5.01 ± 13%  perf-profile.children.cycles-pp.tlb_flush_mmu
      1.70 ±  7%      +5.0        6.67 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.70 ±  7%      +5.0        6.67 ±  9%  perf-profile.children.cycles-pp.do_syscall_64
      0.74 ±  8%      +5.0        5.72 ± 11%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.74 ±  8%      +5.0        5.72 ± 11%  perf-profile.children.cycles-pp.do_group_exit
      0.74 ±  8%      +5.0        5.72 ± 11%  perf-profile.children.cycles-pp.do_exit
      0.73 ±  7%      +5.0        5.72 ± 11%  perf-profile.children.cycles-pp.mmput
      0.73 ±  7%      +5.0        5.72 ± 11%  perf-profile.children.cycles-pp.exit_mmap
      0.98 ±  8%      +5.2        6.22 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter
      0.98 ±  8%      +5.2        6.22 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter_state
      0.03 ±100%      +5.5        5.49 ± 12%  perf-profile.children.cycles-pp.unmap_vmas
      0.03 ±100%      +5.5        5.49 ± 12%  perf-profile.children.cycles-pp.unmap_page_range
      0.95 ±  8%      +6.2        7.16 ±  5%  perf-profile.children.cycles-pp.start_secondary
      1.01 ±  7%      +6.7        7.71 ±  5%  perf-profile.children.cycles-pp.secondary_startup_64
      1.01 ±  7%      +6.7        7.71 ±  5%  perf-profile.children.cycles-pp.cpu_startup_entry
      1.01 ±  7%      +6.7        7.72 ±  5%  perf-profile.children.cycles-pp.do_idle
      0.21 ±  5%     +12.6       12.84        perf-profile.children.cycles-pp.reuse_swap_page
      0.08 ±  8%     +19.8       19.86        perf-profile.children.cycles-pp.wp_page_copy
      0.09 ± 11%     +37.4       37.48        perf-profile.children.cycles-pp.do_wp_page
     40.52           -30.4       10.12 ±  2%  perf-profile.self.cycles-pp.copy_page
     41.99           -16.8       25.24        perf-profile.self.cycles-pp.do_rw_once
      0.70 ±  3%      -0.3        0.44 ±  4%  perf-profile.self.cycles-pp.rmqueue
      0.11 ± 13%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.io_serial_out
      0.07 ± 20%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.sys_imageblit
      0.03 ±100%      +0.0        0.07 ± 12%  perf-profile.self.cycles-pp.rcu_all_qs
      0.06            +0.0        0.11 ± 13%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__next_timer_interrupt
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.try_charge
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.update_rq_clock
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__wake_up_common
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.update_cfs_rq_h_load
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.06 ±  6%      +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.pmd_devmap_trans_unstable
      0.00            +0.1        0.07 ± 13%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.free_unref_page_prepare
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.get_vma_policy
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.uncharge_page
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.stack_trace_save_tsk
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.cpumask_any_but
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.mem_cgroup_page_lruvec
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.vm_normal_page
      0.00            +0.1        0.08 ± 11%  perf-profile.self.cycles-pp.mem_cgroup_update_lru_size
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.native_sched_clock
      0.01 ±173%      +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.native_write_msr
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.delay_tsc
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.perf_exclude_event
      0.00            +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.mem_cgroup_charge_statistics
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.alloc_pages_vma
      0.00            +0.1        0.09 ±  9%  perf-profile.self.cycles-pp.mem_cgroup_try_charge
      0.00            +0.1        0.09 ±  9%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.orc_find
      0.00            +0.1        0.09 ± 13%  perf-profile.self.cycles-pp.mem_cgroup_from_task
      0.00            +0.1        0.09        perf-profile.self.cycles-pp.page_mapping
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.do_page_fault
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.free_unref_page_list
      0.00            +0.1        0.10 ± 11%  perf-profile.self.cycles-pp.free_unref_page_commit
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__switch_to_asm
      0.00            +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.1        0.10 ±  7%  perf-profile.self.cycles-pp.mem_cgroup_commit_charge
      0.00            +0.1        0.11        perf-profile.self.cycles-pp.lru_cache_add_active_or_unevictable
      0.00            +0.1        0.11 ±  3%  perf-profile.self.cycles-pp.__orc_find
      0.00            +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__switch_to
      0.00            +0.1        0.12 ±  5%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.00            +0.1        0.12 ±  6%  perf-profile.self.cycles-pp.pagevec_lru_move_fn
      0.09 ±  9%      +0.1        0.21 ±  6%  perf-profile.self.cycles-pp.___might_sleep
      0.00            +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.__might_sleep
      0.00            +0.1        0.13        perf-profile.self.cycles-pp.page_add_new_anon_rmap
      0.00            +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.flush_tlb_func_common
      0.00            +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.down_read_trylock
      0.00            +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.__lock_page
      0.00            +0.1        0.14 ± 10%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.ptep_clear_flush
      0.00            +0.2        0.15 ±  8%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.00            +0.2        0.15 ±  5%  perf-profile.self.cycles-pp.up_read
      0.00            +0.2        0.16 ±  7%  perf-profile.self.cycles-pp.set_next_entity
      0.00            +0.2        0.16 ±  4%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.00            +0.2        0.17 ±  4%  perf-profile.self.cycles-pp.__mod_zone_page_state
      0.00            +0.2        0.17 ±  4%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.00            +0.2        0.18 ±  6%  perf-profile.self.cycles-pp.__count_memcg_events
      0.00            +0.2        0.19 ±  2%  perf-profile.self.cycles-pp.unlock_page
      0.00            +0.2        0.20 ±  3%  perf-profile.self.cycles-pp.unwind_next_frame
      0.00            +0.2        0.20 ±  3%  perf-profile.self.cycles-pp.__lru_cache_add
      0.00            +0.2        0.20 ±  2%  perf-profile.self.cycles-pp.page_fault
      0.00            +0.2        0.21 ±  5%  perf-profile.self.cycles-pp.free_pages_and_swap_cache
      0.00            +0.2        0.21 ±  3%  perf-profile.self.cycles-pp.__list_add_valid
      0.00            +0.2        0.21 ±  2%  perf-profile.self.cycles-pp.__schedule
      0.00            +0.2        0.23 ±  4%  perf-profile.self.cycles-pp._find_next_bit
      0.00            +0.2        0.23 ±  3%  perf-profile.self.cycles-pp.__perf_sw_event
      0.00            +0.2        0.23 ±  4%  perf-profile.self.cycles-pp.vmacache_find
      0.00            +0.2        0.24 ±  3%  perf-profile.self.cycles-pp.mem_cgroup_throttle_swaprate
      0.00            +0.2        0.24 ±  2%  perf-profile.self.cycles-pp.flush_tlb_mm_range
      0.04 ± 58%      +0.2        0.29 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.2        0.24 ±  4%  perf-profile.self.cycles-pp.__split_huge_pmd_locked
      0.00            +0.3        0.25 ±  3%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.00            +0.3        0.27 ±  4%  perf-profile.self.cycles-pp.do_user_addr_fault
      0.00            +0.3        0.29 ±  3%  perf-profile.self.cycles-pp.put_page
      0.00            +0.3        0.30 ±  2%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.00            +0.3        0.30 ±  6%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.00            +0.4        0.35        perf-profile.self.cycles-pp.handle_mm_fault
      0.00            +0.4        0.36 ±  5%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.00            +0.4        0.39 ±  8%  perf-profile.self.cycles-pp.try_to_wake_up
      0.00            +0.4        0.42 ±  3%  perf-profile.self.cycles-pp.page_remove_rmap
      0.00            +0.4        0.43        perf-profile.self.cycles-pp._raw_spin_lock
      0.00            +0.4        0.44 ±  3%  perf-profile.self.cycles-pp.unmap_page_range
      0.00            +0.5        0.47 ±  3%  perf-profile.self.cycles-pp.release_pages
      0.00            +0.5        0.48 ±  4%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.00            +0.5        0.49 ±  2%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.00            +0.5        0.54 ±  3%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.04 ± 57%      +0.6        0.59 ±  3%  perf-profile.self.cycles-pp.___perf_sw_event
      0.00            +0.6        0.58        perf-profile.self.cycles-pp.mem_cgroup_try_charge_delay
      0.00            +0.6        0.60 ±  3%  perf-profile.self.cycles-pp.wp_page_copy
      0.00            +0.7        0.74 ±  3%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.00            +0.8        0.81 ±  2%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.00            +0.8        0.83 ±  3%  perf-profile.self.cycles-pp.do_wp_page
      0.00            +1.1        1.12        perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.00            +1.3        1.28        perf-profile.self.cycles-pp.sync_regs
      0.08 ±  5%      +1.6        1.72 ±  2%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.92 ±  7%      +2.4        3.32 ±  6%  perf-profile.self.cycles-pp.intel_idle
      0.00            +2.8        2.76 ±  9%  perf-profile.self.cycles-pp.poll_idle
      0.58 ± 15%      +2.9        3.49 ± 18%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     10.62            +3.3       13.91        perf-profile.self.cycles-pp.do_access
      0.21 ±  5%     +12.5       12.75        perf-profile.self.cycles-pp.reuse_swap_page


                                                                                
                           vm-scalability.time.user_time                        
                                                                                
  2600 +--------------------------------------------------------------------+   
       |.++.++.++.+.++.++.++.+.++.++.++.+.++.++.++.++.+.++.++. +.+.++.++.++.|   
  2500 |-+                                                    +             |   
  2400 |-+                                                                  |   
       |                                                                    |   
  2300 |-+                                                                  |   
  2200 |-+                                                                  |   
       |                                                                    |   
  2100 |-+                                                                  |   
  2000 |-+                                                                  |   
       |                                                                    |   
  1900 |-+                                                                  |   
  1800 |-+                                                                  |   
       | OO OO OO O OO OO OO O OO OO OO O OO OO  O OO O OO OO OO O OO OO O  |   
  1700 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          vm-scalability.time.system_time                       
                                                                                
  2300 +--------------------------------------------------------------------+   
       |                                                OO OO OO   O   O    |   
  2250 |-OO OO OO    O O     O                O OO OO            O  O O  O  |   
  2200 |-+        O O   O OO   OO OO OO O OO O        O                     |   
       |                                                                    |   
  2150 |-+                                                                  |   
  2100 |-+                                                                  |   
       |                                                                    |   
  2050 |-+                                                                  |   
  2000 |-+                                                                  |   
       |                                                                    |   
  1950 |-+                                                                  |   
  1900 |.+ .++.++.+.+ .++.++. .++.     .+.++.+ .++.++. .++.  .++.+.++.++.+ .|   
       |  +          +       +    ++.++       +       +    ++             + |   
  1850 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                  vm-scalability.time.percent_of_cpu_this_job_got               
                                                                                
  1480 +--------------------------------------------------------------------+   
       |.++.++.++.+.++.++.++.+.++.++.++.+.++.++.++.++.+.++.++.++.+.++.++.++.|   
  1460 |-+                                                                  |   
  1440 |-+                                                                  |   
       |                                                                    |   
  1420 |-+                                                                  |   
  1400 |-+                                                                  |   
       |                                                                    |   
  1380 |-+                                                                  |   
  1360 |-+                                                                  |   
       |                                                                    |   
  1340 |-+                                              OO OO OO   O  OO    |   
  1320 |-+     OO   O  O  O  O  O     O   O      O O             O  O    O  |   
       | OO OO    O  O  O  O   O  OO O  O  O OO O   O O                     |   
  1300 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        vm-scalability.time.minor_page_faults                   
                                                                                
  7e+08 +-------------------------------------------------------------------+   
        | OO OO OO OO OO O OO OO OO OO OO O OO OO OO OO OO O OO OO  O OO O  |   
  6e+08 |-+                                                                 |   
        |                                                                   |   
  5e+08 |-+                                                                 |   
        |                                                                   |   
  4e+08 |-+                                                                 |   
        |                                                                   |   
  3e+08 |-+                                                                 |   
        |                                                                   |   
  2e+08 |-+                                                                 |   
        |                                                                   |   
  1e+08 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                   vm-scalability.time.voluntary_context_switches               
                                                                                
  7e+07 +-------------------------------------------------------------------+   
        |                                                                   |   
  6e+07 |-OO O  OO OO O  O OO OO OO OO OO O OO OO    OO                     |   
        |     O        O                          OO    OO O O   O O   O    |   
  5e+07 |-+                                                   O O   O O  O  |   
        |                                                                   |   
  4e+07 |-+                                                                 |   
        |                                                                   |   
  3e+07 |-+                                                                 |   
        |                                                                   |   
  2e+07 |-+                                                                 |   
        |                                                                   |   
  1e+07 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               vm-scalability.throughput                        
                                                                                
  1.18e+07 +----------------------------------------------------------------+   
           |                                             OO OO    OO OO  O  |   
  1.16e+07 |-+    O OO O   O OO OO OO OO OO OOO                         O   |   
           |                                                                |   
  1.14e+07 |-+          O O                        O   O       OO           |   
           |O OO O                              OO  O O                     |   
  1.12e+07 |-+                                                              |   
           |                                                                |   
   1.1e+07 |-+                                                              |   
           |                                                                |   
  1.08e+07 |-+                                                              |   
           |                                                                |   
  1.06e+07 |-+                                                              |   
           |+.++.++.+ .++.++. +.++.++.++.++.+ +.++.+ .++.++.++.++.++.++.++.+|   
  1.04e+07 +----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.median                           
                                                                                
  740000 +------------------------------------------------------------------+   
         |                                                       O          |   
  730000 |-+     OO OO    OO OO OO OO OO O OO O         O OO O     OO OO O  |   
  720000 |-+                                                                |   
         |             OO                           O  O      O O           |   
  710000 |-OO OO                               O OO  O                      |   
  700000 |-+                                                                |   
         |                                                                  |   
  690000 |-+                                                                |   
  680000 |-+                                                                |   
         |                                                                  |   
  670000 |-+                                                                |   
  660000 |-+   +. +.     .+      +.+ .++.+.+      +.     .++.  .+      +. +.|   
         |.++.+  +  ++.++  +.++.+   +       +.++.+  ++.++    ++  +.++.+  +  |   
  650000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.7.0-03894-g3917c80280c93" of type "text/plain" (203101 bytes)

View attachment "job-script" of type "text/plain" (7689 bytes)

View attachment "job.yaml" of type "text/plain" (5247 bytes)

View attachment "reproduce" of type "text/plain" (15066 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ