lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Tue, 21 Apr 2020 08:48:53 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Kirill A. Shutemov" <kirill.shutemov@...ux.intel.com>
Cc:     akpm@...ux-foundation.org, Andrea Arcangeli <aarcange@...hat.com>,
        Zi Yan <ziy@...dia.com>, Yang Shi <yang.shi@...ux.alibaba.com>,
        Ralph Campbell <rcampbell@...dia.com>,
        John Hubbard <jhubbard@...dia.com>,
        William Kucharski <william.kucharski@...cle.com>,
        linux-mm@...ck.org, linux-kernel@...r.kernel.org,
        "Kirill A. Shutemov" <kirill.shutemov@...ux.intel.com>,
        lkp@...ts.01.org
Subject: [thp] fd80f22b29: vm-scalability.median 10.3% improvement

Greeting,

FYI, we noticed a 10.3% improvement of vm-scalability.median due to commit:


commit: fd80f22b2954c87d04f4ad5574821948856fb382 ("[PATCHv4 7/8] thp: Change CoW semantics for anon-THP")
url: https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/thp-khugepaged-improvements-and-CoW-semantics/20200417-033525


in testcase: vm-scalability
on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
with following parameters:

	runtime: 300s
	size: 8T
	test: anon-cow-seq
	cpufreq_governor: performance
	ucode: 0xca

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | pxz: pxz.throughput -2.7% regression                                  |
| test machine     | 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 48G memory      |
| test parameters  | cpufreq_governor=performance                                          |
|                  | nr_threads=25%                                                        |
|                  | ucode=0x7000019                                                       |
+------------------+-----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.daemon.ops_per_sec 50.6% improvement             |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
| test parameters  | class=scheduler                                                       |
|                  | cpufreq_governor=performance                                          |
|                  | disk=1HDD                                                             |
|                  | nr_threads=100%                                                       |
|                  | sc_pid_max=4194304                                                    |
|                  | testtime=1s                                                           |
|                  | ucode=0xb000038                                                       |
+------------------+-----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/8T/lkp-cfl-e1/anon-cow-seq/vm-scalability/0xca

commit: 
  7ad886e462 ("khugepaged: Allow to collapse PTE-mapped compound pages")
  fd80f22b29 ("thp: Change CoW semantics for anon-THP")

7ad886e4626dd5ac fd80f22b2954c87d04f4ad55748 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at_ip___perf_sw_event/0x
           :4          117%           4:4     perf-profile.calltrace.cycles-pp.sync_regs.error_entry.do_access
           :4          128%           5:4     perf-profile.calltrace.cycles-pp.error_entry.do_access
           :4            4%           0:4     perf-profile.children.cycles-pp.error_exit
           :4          149%           5:4     perf-profile.children.cycles-pp.error_entry
           :4           15%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    657425           +10.3%     725413        vm-scalability.median
  10474275           +10.7%   11597453        vm-scalability.throughput
     91994 ±  5%     -10.4%      82468        vm-scalability.time.involuntary_context_switches
   3132185        +21242.1%  6.685e+08        vm-scalability.time.minor_page_faults
      1463            -8.6%       1337        vm-scalability.time.percent_of_cpu_this_job_got
      1905           +18.4%       2255        vm-scalability.time.system_time
      2519           -29.6%       1773        vm-scalability.time.user_time
    138015        +36980.2%   51176421        vm-scalability.time.voluntary_context_switches
   7576848           -85.4%    1107289        meminfo.AnonHugePages
     61139           -10.5%      54724 ±  2%  meminfo.max_used_kB
      6.99            +9.0       15.99        mpstat.cpu.all.idle%
      0.00 ± 47%      +0.1        0.05 ±  2%  mpstat.cpu.all.iowait%
     40.41            +6.8       47.17        mpstat.cpu.all.sys%
     52.58           -15.8       36.76        mpstat.cpu.all.usr%
      7.00          +128.6%      16.00        vmstat.cpu.id
     52.00           -30.8%      36.00        vmstat.cpu.us
      2094        +16003.0%     337236        vmstat.system.cs
     32964            +8.4%      35721        vmstat.system.in
      8943 ±  3%     +12.8%      10087 ±  2%  slabinfo.anon_vma.active_objs
      8973 ±  3%     +12.8%      10122 ±  2%  slabinfo.anon_vma.num_objs
    422.25 ± 17%     -23.4%     323.25 ±  7%  slabinfo.kmem_cache.active_objs
    422.25 ± 17%     -23.4%     323.25 ±  7%  slabinfo.kmem_cache.num_objs
    623.25 ± 20%     -28.2%     447.25 ± 10%  slabinfo.kmem_cache_node.active_objs
    623.25 ± 20%     -28.2%     447.25 ± 10%  slabinfo.kmem_cache_node.num_objs
   2897794          +872.6%   28184286        cpuidle.C1.time
     88459         +3320.9%    3026139        cpuidle.C1.usage
   1485069         +2731.6%   42050896        cpuidle.C1E.time
     29809 ±  3%   +9168.6%    2762901        cpuidle.C1E.usage
 3.286e+08           +66.0%  5.455e+08        cpuidle.C6.time
    690913           +64.2%    1134537        cpuidle.C6.usage
    328.00 ± 21%     +38.3%     453.75 ± 14%  cpuidle.C8.usage
    360076 ±  3%  +32227.8%  1.164e+08        cpuidle.POLL.time
     52985 ±  4%  +89121.1%   47274034        cpuidle.POLL.usage
   2053561            -6.3%    1924443        proc-vmstat.nr_active_anon
   2032025            -5.6%    1917788        proc-vmstat.nr_anon_pages
      3629           -85.1%     541.25        proc-vmstat.nr_anon_transparent_hugepages
    574497            +2.4%     588240        proc-vmstat.nr_dirty_background_threshold
   1150400            +2.4%    1177920        proc-vmstat.nr_dirty_threshold
   5824806            +2.3%    5956287        proc-vmstat.nr_free_pages
     12360            +1.1%      12490        proc-vmstat.nr_slab_unreclaimable
   2053560            -6.3%    1924443        proc-vmstat.nr_zone_active_anon
   4825398        +13780.9%  6.698e+08        proc-vmstat.numa_hit
   4825398        +13780.9%  6.698e+08        proc-vmstat.numa_local
   3523532        +18878.0%  6.687e+08        proc-vmstat.pgfault
   1382836           -94.1%      81374        proc-vmstat.thp_fault_alloc
     11160 ±  3%    +481.0%      64844 ±  3%  softirqs.CPU0.SCHED
     25100 ±  4%     -17.8%      20643 ±  3%  softirqs.CPU1.RCU
      8351          +628.1%      60808 ±  3%  softirqs.CPU1.SCHED
     25894 ±  5%     -18.0%      21220 ±  3%  softirqs.CPU10.RCU
      8798 ±  4%    +586.6%      60414        softirqs.CPU10.SCHED
     25206 ±  6%     -19.8%      20208 ±  8%  softirqs.CPU11.RCU
      8626          +572.3%      57998        softirqs.CPU11.SCHED
     25581 ±  6%     -17.7%      21050 ±  5%  softirqs.CPU12.RCU
      8650 ±  7%    +541.4%      55485 ±  2%  softirqs.CPU12.SCHED
     25630 ±  6%     -19.2%      20715 ±  4%  softirqs.CPU13.RCU
      8169 ±  2%    +594.2%      56708 ±  4%  softirqs.CPU13.SCHED
      8342          +613.7%      59538 ±  3%  softirqs.CPU14.SCHED
     24958 ±  6%     -17.4%      20620 ±  3%  softirqs.CPU15.RCU
      8548 ±  2%    +584.5%      58519 ±  2%  softirqs.CPU15.SCHED
     25792 ±  6%     -20.7%      20442 ±  3%  softirqs.CPU2.RCU
      9116 ±  8%    +554.0%      59618 ±  2%  softirqs.CPU2.SCHED
     26037 ±  6%     -19.4%      20996 ±  5%  softirqs.CPU3.RCU
      8592 ±  2%    +554.5%      56237 ±  2%  softirqs.CPU3.SCHED
     25375 ±  5%     -17.4%      20963 ±  4%  softirqs.CPU4.RCU
      8550 ±  3%    +591.2%      59097        softirqs.CPU4.SCHED
     25440 ±  6%     -22.2%      19805 ± 10%  softirqs.CPU5.RCU
      8541          +579.5%      58036        softirqs.CPU5.SCHED
     25279 ±  5%     -21.7%      19802 ± 11%  softirqs.CPU6.RCU
      8468 ±  4%    +598.4%      59137 ±  3%  softirqs.CPU6.SCHED
      8195 ±  4%    +602.8%      57594 ±  4%  softirqs.CPU7.SCHED
     26522 ±  5%     -21.3%      20880 ±  4%  softirqs.CPU8.RCU
      8806 ±  2%    +599.3%      61586 ±  5%  softirqs.CPU8.SCHED
     25375 ±  4%     -21.3%      19973 ±  6%  softirqs.CPU9.RCU
      9085 ±  6%    +545.8%      58674 ±  3%  softirqs.CPU9.SCHED
    405497 ±  5%     -17.9%     332793 ±  4%  softirqs.RCU
    140006          +574.5%     944303        softirqs.SCHED
    355.00 ± 38%    +106.8%     734.25 ± 31%  interrupts.134:IR-PCI-MSI.2097155-edge.eth1-TxRx-2
      3987 ± 19%   +1221.4%      52685 ± 16%  interrupts.CPU0.RES:Rescheduling_interrupts
      5.50 ± 39%  +55604.5%       3063 ±130%  interrupts.CPU0.TLB:TLB_shootdowns
     10809            -9.5%       9780        interrupts.CPU1.NMI:Non-maskable_interrupts
     10809            -9.5%       9780        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      5311 ± 19%    +953.5%      55953 ±  7%  interrupts.CPU1.RES:Rescheduling_interrupts
      9.75 ±108%  +40171.8%       3926 ±100%  interrupts.CPU1.TLB:TLB_shootdowns
      4771 ± 24%   +1208.3%      62430 ±  5%  interrupts.CPU10.RES:Rescheduling_interrupts
      1.25 ± 66%  +71380.0%     893.50 ± 42%  interrupts.CPU10.TLB:TLB_shootdowns
      3592 ± 25%   +1557.8%      59561 ± 10%  interrupts.CPU11.RES:Rescheduling_interrupts
      1.50 ± 74%  +1.3e+05%       2004 ±129%  interrupts.CPU11.TLB:TLB_shootdowns
     10759           -20.8%       8522 ± 25%  interrupts.CPU12.NMI:Non-maskable_interrupts
     10759           -20.8%       8522 ± 25%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      5489 ± 29%   +1107.6%      66291 ±  9%  interrupts.CPU12.RES:Rescheduling_interrupts
     17.50 ± 95%  +23080.0%       4056 ± 93%  interrupts.CPU12.TLB:TLB_shootdowns
      4058 ± 30%   +1451.1%      62946 ± 11%  interrupts.CPU13.RES:Rescheduling_interrupts
     19.25 ±146%  +21658.4%       4188 ± 42%  interrupts.CPU13.TLB:TLB_shootdowns
      4658 ± 46%   +1025.6%      52432 ± 10%  interrupts.CPU14.RES:Rescheduling_interrupts
      3.25 ± 85%  +23853.8%     778.50 ± 41%  interrupts.CPU14.TLB:TLB_shootdowns
      4756 ± 33%    +895.0%      47324 ±  8%  interrupts.CPU15.RES:Rescheduling_interrupts
      9.75 ± 26%  +24607.7%       2409 ± 85%  interrupts.CPU15.TLB:TLB_shootdowns
    355.00 ± 38%    +106.8%     734.25 ± 31%  interrupts.CPU2.134:IR-PCI-MSI.2097155-edge.eth1-TxRx-2
      4524 ± 34%   +1181.7%      57992 ±  5%  interrupts.CPU2.RES:Rescheduling_interrupts
    133.50 ±126%    +497.8%     798.00 ± 15%  interrupts.CPU2.TLB:TLB_shootdowns
      3744 ± 39%   +1436.9%      57547 ±  6%  interrupts.CPU3.RES:Rescheduling_interrupts
    149.25 ±105%    +972.7%       1601 ± 77%  interrupts.CPU3.TLB:TLB_shootdowns
      4999 ± 39%   +1317.1%      70851 ± 15%  interrupts.CPU4.RES:Rescheduling_interrupts
     54.50 ±136%   +2525.2%       1430 ± 36%  interrupts.CPU4.TLB:TLB_shootdowns
      3414 ± 27%   +1845.1%      66409 ±  5%  interrupts.CPU5.RES:Rescheduling_interrupts
     17.00 ± 58%  +18811.8%       3215 ±134%  interrupts.CPU5.TLB:TLB_shootdowns
      4719 ± 22%   +1084.9%      55918 ± 12%  interrupts.CPU6.RES:Rescheduling_interrupts
     12.00 ± 92%  +18422.9%       2222 ± 77%  interrupts.CPU6.TLB:TLB_shootdowns
      3885 ± 21%   +1184.5%      49909 ± 14%  interrupts.CPU7.RES:Rescheduling_interrupts
      9.50 ±138%  +37178.9%       3541 ± 71%  interrupts.CPU7.TLB:TLB_shootdowns
      4211 ± 36%   +1142.7%      52341 ± 16%  interrupts.CPU8.RES:Rescheduling_interrupts
      6.50 ±146%  +17103.8%       1118 ± 98%  interrupts.CPU8.TLB:TLB_shootdowns
     10781           -32.0%       7334 ± 33%  interrupts.CPU9.NMI:Non-maskable_interrupts
     10781           -32.0%       7334 ± 33%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      4599 ± 36%    +983.1%      49810 ± 10%  interrupts.CPU9.RES:Rescheduling_interrupts
      8.25 ±138%  +78321.2%       6469 ±138%  interrupts.CPU9.TLB:TLB_shootdowns
     70723 ±  5%   +1201.4%     920403 ±  3%  interrupts.RES:Rescheduling_interrupts
    458.25 ± 42%   +9003.8%      41718 ± 19%  interrupts.TLB:TLB_shootdowns
    510.80 ±  4%     +31.1%     669.47 ± 10%  sched_debug.cfs_rq:/.exec_clock.stddev
     49788 ±  8%     -32.8%      33468        sched_debug.cfs_rq:/.load.min
     96.38 ±  8%     +10.7%     106.71 ±  6%  sched_debug.cfs_rq:/.load_avg.avg
     41.75 ± 18%     -36.3%      26.58 ± 30%  sched_debug.cfs_rq:/.load_avg.min
     88.58 ± 14%     +21.5%     107.59 ±  4%  sched_debug.cfs_rq:/.load_avg.stddev
   2127529           -18.0%    1745558        sched_debug.cfs_rq:/.min_vruntime.avg
   2142032           -17.6%    1764531        sched_debug.cfs_rq:/.min_vruntime.max
   2114546           -18.2%    1729664        sched_debug.cfs_rq:/.min_vruntime.min
      0.79 ±  9%     -42.1%       0.46 ± 15%  sched_debug.cfs_rq:/.nr_running.min
      0.11 ± 40%     +97.6%       0.21 ± 15%  sched_debug.cfs_rq:/.nr_running.stddev
     15.78 ±  7%     -77.7%       3.52 ±  5%  sched_debug.cfs_rq:/.nr_spread_over.avg
     37.33 ± 17%     -70.1%      11.17 ± 22%  sched_debug.cfs_rq:/.nr_spread_over.max
      4.83 ± 17%    -100.0%       0.00        sched_debug.cfs_rq:/.nr_spread_over.min
      8.95 ± 23%     -65.5%       3.09 ± 16%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    921.34 ±  6%     -10.9%     821.24 ±  7%  sched_debug.cfs_rq:/.util_avg.avg
      1294 ±  4%     +11.6%       1445 ±  8%  sched_debug.cfs_rq:/.util_avg.max
    571.88 ± 26%     -50.2%     284.79 ± 25%  sched_debug.cfs_rq:/.util_avg.min
    170.63 ± 28%     +78.2%     303.99 ± 11%  sched_debug.cfs_rq:/.util_avg.stddev
    724.84 ±  5%     -21.3%     570.71 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    193.79 ± 31%     -91.0%      17.38 ±167%  sched_debug.cfs_rq:/.util_est_enqueued.min
    239.07 ± 12%     +25.2%     299.28 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    295277 ± 15%     -61.4%     114049 ± 54%  sched_debug.cpu.avg_idle.avg
    901251 ± 11%     -39.6%     544406 ± 15%  sched_debug.cpu.avg_idle.max
     52248 ± 45%     -86.3%       7181 ± 46%  sched_debug.cpu.avg_idle.min
    247994 ± 13%     -35.8%     159217 ± 21%  sched_debug.cpu.avg_idle.stddev
      2.11 ± 17%     -61.6%       0.81 ± 13%  sched_debug.cpu.clock.stddev
      2.11 ± 17%     -61.6%       0.81 ± 13%  sched_debug.cpu.clock_task.stddev
      4282 ± 15%     -57.5%       1821 ± 28%  sched_debug.cpu.curr->pid.min
    322.27 ± 52%    +290.6%       1258 ± 24%  sched_debug.cpu.curr->pid.stddev
      0.79 ±  9%     -42.1%       0.46 ± 30%  sched_debug.cpu.nr_running.min
     22588        +14213.6%    3233285        sched_debug.cpu.nr_switches.avg
     32090 ±  4%  +11546.6%    3737476 ±  4%  sched_debug.cpu.nr_switches.max
     16606 ±  5%  +16298.2%    2723162 ±  2%  sched_debug.cpu.nr_switches.min
      4019 ± 15%   +6856.4%     279625 ±  9%  sched_debug.cpu.nr_switches.stddev
     17.62 ± 27%     +52.0%      26.79 ± 13%  sched_debug.cpu.nr_uninterruptible.max
     19256        +16673.2%    3229873        sched_debug.cpu.sched_count.avg
     25845 ±  5%  +14339.0%    3731897 ±  4%  sched_debug.cpu.sched_count.max
     15525 ±  5%  +17417.3%    2719596 ±  2%  sched_debug.cpu.sched_count.min
      2826 ± 17%   +9770.4%     278969 ±  9%  sched_debug.cpu.sched_count.stddev
      5313 ±  2%  +30190.7%    1609376        sched_debug.cpu.sched_goidle.avg
      7353 ±  4%  +25179.8%    1858939 ±  4%  sched_debug.cpu.sched_goidle.max
      4490 ±  3%  +30063.0%    1354606 ±  2%  sched_debug.cpu.sched_goidle.min
    723.16 ±  8%  +19141.0%     139143 ±  9%  sched_debug.cpu.sched_goidle.stddev
      9230        +17408.0%    1616139        sched_debug.cpu.ttwu_count.avg
     12809 ±  5%  +14347.6%    1850726 ±  4%  sched_debug.cpu.ttwu_count.max
      7279 ±  2%  +18964.6%    1387806 ±  2%  sched_debug.cpu.ttwu_count.min
      1540 ±  4%   +8361.6%     130355 ± 10%  sched_debug.cpu.ttwu_count.stddev
      3066 ±  3%     +27.5%       3911 ±  4%  sched_debug.cpu.ttwu_local.avg
      5215 ± 19%     +33.0%       6936 ±  6%  sched_debug.cpu.ttwu_local.max
      2114 ±  3%     +27.2%       2689 ±  5%  sched_debug.cpu.ttwu_local.min
     40.44           -40.4        0.00        perf-profile.calltrace.cycles-pp.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
     38.85           -38.9        0.00        perf-profile.calltrace.cycles-pp.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault.do_page_fault
     38.72           -38.7        0.00        perf-profile.calltrace.cycles-pp.copy_subpage.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault
     38.33           -38.3        0.00        perf-profile.calltrace.cycles-pp.copy_page.copy_subpage.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault
     99.21           -37.0       62.20        perf-profile.calltrace.cycles-pp.do_access
     38.92            -2.3       36.60        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault.do_access
     38.95            -1.7       37.24        perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault.do_access
      0.41 ± 58%      +0.3        0.68 ±  5%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write._fini
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.write._fini
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp._fini
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.devkmsg_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.41 ± 58%      +0.3        0.69 ±  4%  perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write.ksys_write
      0.13 ±173%      +0.4        0.58 ±  7%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write
      0.00            +0.7        0.67 ± 10%  perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      0.00            +0.7        0.72 ± 10%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.io_schedule.__lock_page.do_wp_page
      0.00            +0.7        0.74 ± 10%  perf-profile.calltrace.cycles-pp.schedule.io_schedule.__lock_page.do_wp_page.handle_pte_fault
      0.00            +0.7        0.74 ± 11%  perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
      0.00            +0.8        0.75 ± 10%  perf-profile.calltrace.cycles-pp.io_schedule.__lock_page.do_wp_page.handle_pte_fault.__handle_mm_fault
      0.00            +0.8        0.85 ±  2%  perf-profile.calltrace.cycles-pp.__pagevec_lru_add_fn.pagevec_lru_move_fn.__lru_cache_add.wp_page_copy.do_wp_page
      0.00            +0.9        0.90 ± 10%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.00            +1.0        1.04        perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.ptep_clear_flush.wp_page_copy
      0.00            +1.1        1.07 ±  9%  perf-profile.calltrace.cycles-pp.__lock_page.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
      0.00            +1.1        1.14 ±  5%  perf-profile.calltrace.cycles-pp.free_pcppages_bulk.free_unref_page_list.release_pages.tlb_flush_mmu.unmap_page_range
     20.97            +1.2       22.12        perf-profile.calltrace.cycles-pp.do_rw_once
      0.00            +1.2        1.17 ± 10%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      0.00            +1.2        1.18        perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.ptep_clear_flush.wp_page_copy.do_wp_page
      0.00            +1.2        1.20        perf-profile.calltrace.cycles-pp.mem_cgroup_try_charge_delay.wp_page_copy.do_wp_page.handle_pte_fault.__handle_mm_fault
      0.00            +1.2        1.23 ± 10%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +1.2        1.24 ± 10%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
      0.00            +1.2        1.24 ± 10%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +1.4        1.44 ±  3%  perf-profile.calltrace.cycles-pp.free_unref_page_list.release_pages.tlb_flush_mmu.unmap_page_range.unmap_vmas
      0.00            +1.5        1.49 ±  7%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu.unmap_page_range
      0.00            +1.5        1.50 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu.unmap_page_range.unmap_vmas
      0.00            +1.6        1.55        perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.wp_page_copy.do_wp_page
      0.00            +1.6        1.56        perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.wp_page_copy.do_wp_page.handle_pte_fault
      0.00            +1.7        1.65        perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.ptep_clear_flush.wp_page_copy.do_wp_page.handle_pte_fault
      0.00            +1.7        1.73 ±  2%  perf-profile.calltrace.cycles-pp.__lru_cache_add.wp_page_copy.do_wp_page.handle_pte_fault.__handle_mm_fault
      0.00            +1.8        1.79        perf-profile.calltrace.cycles-pp.ptep_clear_flush.wp_page_copy.do_wp_page.handle_pte_fault.__handle_mm_fault
      0.00            +2.0        2.00 ± 10%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.do_wp_page
      0.00            +2.0        2.02        perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.wp_page_copy.do_wp_page.handle_pte_fault
      0.00            +2.0        2.02 ± 10%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.do_wp_page.handle_pte_fault
      1.66 ± 66%      +2.1        3.72 ± 28%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
      0.00            +2.1        2.09 ± 10%  perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.do_wp_page.handle_pte_fault.__handle_mm_fault
      1.66 ± 66%      +2.1        3.78 ± 27%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault
      0.00            +2.2        2.23        perf-profile.calltrace.cycles-pp.alloc_pages_vma.wp_page_copy.do_wp_page.handle_pte_fault.__handle_mm_fault
      1.66 ± 66%      +2.3        3.93 ± 28%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
      0.93 ±  2%      +2.3        3.22 ±  5%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.67 ± 66%      +2.3        3.99 ± 28%  perf-profile.calltrace.cycles-pp.page_fault
      0.00            +2.4        2.44 ± 11%  perf-profile.calltrace.cycles-pp.wake_up_page_bit.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
      0.00            +2.5        2.46 ±  6%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +3.9        3.91 ±  3%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.unmap_page_range.unmap_vmas.exit_mmap
      0.00            +4.2        4.16 ±  3%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.unmap_page_range.unmap_vmas.exit_mmap.mmput
      0.74 ±  4%      +4.4        5.11 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.74 ±  4%      +4.4        5.11 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72 ±  4%      +4.4        5.10 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72 ±  4%      +4.4        5.10 ±  2%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72 ±  4%      +4.4        5.10 ±  2%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.71 ±  5%      +4.4        5.09 ±  3%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.71 ±  5%      +4.4        5.09 ±  3%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      0.99 ±  3%      +4.8        5.77 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      0.99 ±  3%      +4.8        5.78 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +4.9        4.89 ±  3%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      0.00            +4.9        4.90 ±  3%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
     39.07            +5.2       44.31        perf-profile.calltrace.cycles-pp.page_fault.do_access
      1.02 ±  3%      +6.2        7.19 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.02 ±  3%      +6.2        7.20 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      1.02 ±  3%      +6.2        7.20 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      1.05 ±  3%      +6.7        7.76 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      0.00           +10.6       10.58 ±  4%  perf-profile.calltrace.cycles-pp.copy_page.copy_user_highpage.wp_page_copy.do_wp_page.handle_pte_fault
      0.00           +10.8       10.75 ±  4%  perf-profile.calltrace.cycles-pp.copy_user_highpage.wp_page_copy.do_wp_page.handle_pte_fault.__handle_mm_fault
      0.00           +13.0       13.05 ±  2%  perf-profile.calltrace.cycles-pp.reuse_swap_page.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
      0.00           +20.7       20.67 ±  3%  perf-profile.calltrace.cycles-pp.wp_page_copy.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
      0.00           +38.7       38.73        perf-profile.calltrace.cycles-pp.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault
      0.00           +39.3       39.27        perf-profile.calltrace.cycles-pp.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
     40.54           -40.1        0.40 ±  2%  perf-profile.children.cycles-pp.do_huge_pmd_wp_page
     38.96           -39.0        0.00        perf-profile.children.cycles-pp.copy_user_huge_page
     38.81           -38.8        0.00        perf-profile.children.cycles-pp.copy_subpage
     38.64           -28.0       10.61 ±  4%  perf-profile.children.cycles-pp.copy_page
     44.93           -19.6       25.29        perf-profile.children.cycles-pp.do_rw_once
     75.31           -15.4       59.96        perf-profile.children.cycles-pp.do_access
      0.67 ±  5%      -0.5        0.19 ± 12%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.97 ±  2%      -0.4        0.57 ±  5%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.74 ±  2%      -0.3        0.49 ±  6%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.63 ±  3%      -0.3        0.38 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.50 ±  2%      -0.2        0.29 ±  7%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.40            -0.2        0.21 ±  9%  perf-profile.children.cycles-pp.tick_sched_timer
      0.38            -0.2        0.19 ±  8%  perf-profile.children.cycles-pp.tick_sched_handle
      0.36            -0.2        0.19 ± 11%  perf-profile.children.cycles-pp.update_process_times
      0.23 ±  2%      -0.1        0.10 ±  8%  perf-profile.children.cycles-pp.scheduler_tick
      0.17 ±  4%      -0.1        0.06 ± 13%  perf-profile.children.cycles-pp.task_tick_fair
      0.10 ± 17%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.io_serial_out
      0.13 ±  6%      -0.0        0.10 ± 14%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.13 ±  6%      -0.0        0.10 ± 14%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.13 ±  6%      -0.0        0.10 ± 14%  perf-profile.children.cycles-pp.irq_work_run
      0.13 ±  6%      -0.0        0.10 ± 14%  perf-profile.children.cycles-pp.printk
      0.07 ± 16%      +0.0        0.11 ± 14%  perf-profile.children.cycles-pp.lf
      0.07 ± 13%      +0.0        0.11 ± 13%  perf-profile.children.cycles-pp.fbcon_redraw
      0.07 ± 17%      +0.0        0.11 ± 13%  perf-profile.children.cycles-pp.fbcon_putcs
      0.06 ± 13%      +0.0        0.11 ± 17%  perf-profile.children.cycles-pp.bit_putcs
      0.07 ± 10%      +0.0        0.11 ± 15%  perf-profile.children.cycles-pp.vt_console_print
      0.07 ± 13%      +0.0        0.11 ± 14%  perf-profile.children.cycles-pp.con_scroll
      0.07 ± 13%      +0.0        0.11 ± 14%  perf-profile.children.cycles-pp.fbcon_scroll
      0.04 ± 57%      +0.0        0.09 ± 15%  perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit
      0.04 ± 57%      +0.0        0.09 ± 15%  perf-profile.children.cycles-pp.sys_imageblit
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.rcu_eqs_enter
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.delayacct_end
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.pmd_pfn
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.update_cfs_group
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.kernel_text_address
      0.06 ± 20%      +0.1        0.13        perf-profile.children.cycles-pp._cond_resched
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.free_unref_page_prepare
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.vm_normal_page
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.mem_cgroup_page_lruvec
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.finish_task_switch
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.orc_find
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.uncharge_page
      0.00            +0.1        0.08 ± 15%  perf-profile.children.cycles-pp.mem_cgroup_from_task
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.get_vma_policy
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.check_preempt_curr
      0.00            +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.rcu_all_qs
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.page_mapping
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.00            +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.__kernel_text_address
      0.00            +0.1        0.09 ± 12%  perf-profile.children.cycles-pp.pmd_devmap_trans_unstable
      0.00            +0.1        0.09 ±  7%  perf-profile.children.cycles-pp.perf_exclude_event
      0.01 ±173%      +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.native_write_msr
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.sched_clock
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
      0.07 ±  6%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.update_curr
      0.00            +0.1        0.10 ±  5%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.delay_tsc
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.00            +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.__unwind_start
      0.00            +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.1        0.11 ± 14%  perf-profile.children.cycles-pp.__switch_to_asm
      0.00            +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.42 ± 12%      +0.1        0.53 ±  8%  perf-profile.children.cycles-pp.io_serial_in
      0.00            +0.1        0.12 ±  9%  perf-profile.children.cycles-pp.__switch_to
      0.00            +0.1        0.12 ± 10%  perf-profile.children.cycles-pp.lru_cache_add_active_or_unevictable
      0.00            +0.1        0.12 ± 13%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.00            +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +0.1        0.13 ± 14%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.00            +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.53 ± 12%      +0.1        0.66 ±  7%  perf-profile.children.cycles-pp.uart_console_write
      0.10 ± 10%      +0.1        0.24 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.00            +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.down_read_trylock
      0.00            +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.__orc_find
      0.00            +0.1        0.15 ±  4%  perf-profile.children.cycles-pp.free_unref_page_commit
      0.55 ± 12%      +0.2        0.71 ±  7%  perf-profile.children.cycles-pp.serial8250_console_write
      0.64 ± 12%      +0.2        0.79 ±  6%  perf-profile.children.cycles-pp.vprintk_emit
      0.00            +0.2        0.16 ±  5%  perf-profile.children.cycles-pp.__might_sleep
      0.00            +0.2        0.16 ±  9%  perf-profile.children.cycles-pp.__mod_zone_page_state
      0.42 ± 12%      +0.2        0.59 ±  7%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.00            +0.2        0.17 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.17 ±  4%  perf-profile.children.cycles-pp.up_read
      0.53 ± 14%      +0.2        0.71 ±  4%  perf-profile.children.cycles-pp.ksys_write
      0.53 ± 15%      +0.2        0.71 ±  4%  perf-profile.children.cycles-pp.vfs_write
      0.53 ± 15%      +0.2        0.71 ±  4%  perf-profile.children.cycles-pp.new_sync_write
      0.45 ± 12%      +0.2        0.63 ±  7%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.52 ± 14%      +0.2        0.70 ±  5%  perf-profile.children.cycles-pp.write
      0.51 ± 15%      +0.2        0.69 ±  4%  perf-profile.children.cycles-pp._fini
      0.51 ± 15%      +0.2        0.69 ±  4%  perf-profile.children.cycles-pp.devkmsg_write
      0.51 ± 15%      +0.2        0.69 ±  4%  perf-profile.children.cycles-pp.devkmsg_emit
      0.00            +0.2        0.19 ±  5%  perf-profile.children.cycles-pp.__count_memcg_events
      0.00            +0.2        0.20 ± 11%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.62 ± 12%      +0.2        0.82 ±  6%  perf-profile.children.cycles-pp.console_unlock
      0.00            +0.2        0.21 ±  7%  perf-profile.children.cycles-pp.unlock_page
      0.00            +0.2        0.23 ±  3%  perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.2        0.23 ±  3%  perf-profile.children.cycles-pp.vmacache_find
      0.00            +0.2        0.24 ± 15%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.2        0.24 ±  5%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.00            +0.2        0.25 ± 10%  perf-profile.children.cycles-pp.set_next_entity
      0.00            +0.2        0.25 ±  2%  perf-profile.children.cycles-pp.find_vma
      0.00            +0.2        0.25 ±  7%  perf-profile.children.cycles-pp._find_next_bit
      0.00            +0.3        0.26 ±  3%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.00            +0.3        0.27        perf-profile.children.cycles-pp.__split_huge_pmd
      0.00            +0.3        0.27        perf-profile.children.cycles-pp.__split_huge_pmd_locked
      0.00            +0.3        0.28 ±  3%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.00            +0.3        0.29 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.00            +0.3        0.29 ±  4%  perf-profile.children.cycles-pp.cpumask_any_but
      0.00            +0.3        0.29 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.3        0.29 ±  3%  perf-profile.children.cycles-pp.put_page
      0.00            +0.3        0.29 ±  2%  perf-profile.children.cycles-pp.mem_cgroup_throttle_swaprate
      0.00            +0.3        0.31 ±  7%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.00            +0.3        0.31 ±  6%  perf-profile.children.cycles-pp.page_add_new_anon_rmap
      0.00            +0.3        0.31 ± 14%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.3        0.34 ±  9%  perf-profile.children.cycles-pp.dequeue_entity
      0.00            +0.4        0.35 ±  7%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.00            +0.4        0.37 ±  9%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.4        0.38        perf-profile.children.cycles-pp.__mod_lruvec_state
      0.00            +0.4        0.39 ±  6%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.00            +0.4        0.43 ±  4%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.00            +0.5        0.48 ±  9%  perf-profile.children.cycles-pp.unwind_next_frame
      0.00            +0.5        0.48 ±  7%  perf-profile.children.cycles-pp.page_remove_rmap
      0.00            +0.5        0.49 ±  4%  perf-profile.children.cycles-pp.mem_cgroup_try_charge
      0.00            +0.5        0.51 ± 13%  perf-profile.children.cycles-pp.menu_select
      0.03 ±100%      +0.5        0.55 ± 13%  perf-profile.children.cycles-pp.start_kernel
      0.00            +0.6        0.61 ±  9%  perf-profile.children.cycles-pp.schedule_idle
      1.13 ±  3%      +0.6        1.76 ±  2%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.07 ± 17%      +0.7        0.75 ±  2%  perf-profile.children.cycles-pp.___perf_sw_event
      0.00            +0.7        0.74 ±  8%  perf-profile.children.cycles-pp.arch_stack_walk
      0.01 ±173%      +0.8        0.81 ±  3%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.8        0.81 ±  7%  perf-profile.children.cycles-pp.schedule
      0.00            +0.8        0.81 ±  9%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.00            +0.8        0.82 ±  7%  perf-profile.children.cycles-pp.io_schedule
      0.07 ± 17%      +0.8        0.90 ±  2%  perf-profile.children.cycles-pp.__perf_sw_event
      0.01 ±173%      +0.9        0.89 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.00            +0.9        0.94        perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.00            +1.0        0.98 ±  8%  perf-profile.children.cycles-pp.__account_scheduler_latency
      1.13 ±  3%      +1.1        2.28        perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.00            +1.2        1.16 ±  6%  perf-profile.children.cycles-pp.__lock_page
      0.00            +1.2        1.17 ±  3%  perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.00            +1.2        1.23 ±  4%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.00            +1.3        1.28 ±  9%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +1.3        1.30 ±  2%  perf-profile.children.cycles-pp.flush_tlb_func_common
      0.00            +1.3        1.31 ±  3%  perf-profile.children.cycles-pp.sync_regs
      0.00            +1.3        1.33 ±  8%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.04 ± 57%      +1.3        1.38 ±  9%  perf-profile.children.cycles-pp.__sched_text_start
      0.00            +1.3        1.35 ±  8%  perf-profile.children.cycles-pp.activate_task
      0.00            +1.4        1.35 ±  8%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +1.4        1.38 ±  2%  perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay
      1.12 ±  3%      +1.4        2.50        perf-profile.children.cycles-pp.alloc_pages_vma
      0.00            +1.6        1.57 ±  3%  perf-profile.children.cycles-pp.free_unref_page_list
      0.04 ± 58%      +1.7        1.75 ±  3%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.08 ±  6%      +1.7        1.80 ±  2%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +1.8        1.84        perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.06 ± 14%      +1.9        1.92 ±  3%  perf-profile.children.cycles-pp.__lru_cache_add
     40.80            +1.9       42.66        perf-profile.children.cycles-pp.do_page_fault
      0.00            +2.0        1.98        perf-profile.children.cycles-pp.ptep_clear_flush
      0.47 ±  2%      +2.1        2.53 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.05 ±  9%      +2.1        2.19 ±  8%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +2.2        2.21 ±  8%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.03 ±100%      +2.2        2.27 ±  8%  perf-profile.children.cycles-pp.__wake_up_common
      0.43 ±  3%      +2.4        2.83 ±  4%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.96 ±  2%      +2.5        3.46 ±  5%  perf-profile.children.cycles-pp.intel_idle
      0.04 ± 57%      +2.6        2.64 ±  8%  perf-profile.children.cycles-pp.wake_up_page_bit
      0.00            +2.7        2.65 ±  7%  perf-profile.children.cycles-pp.poll_idle
      0.67 ±  5%      +3.6        4.25 ±  3%  perf-profile.children.cycles-pp.release_pages
      0.67 ±  5%      +3.7        4.35 ±  3%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.72 ±  5%      +4.4        5.09 ±  2%  perf-profile.children.cycles-pp.mmput
      0.72 ±  5%      +4.4        5.09 ±  2%  perf-profile.children.cycles-pp.exit_mmap
      0.72 ±  4%      +4.4        5.10 ±  2%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.72 ±  4%      +4.4        5.10 ±  2%  perf-profile.children.cycles-pp.do_group_exit
      0.72 ±  4%      +4.4        5.10 ±  2%  perf-profile.children.cycles-pp.do_exit
      1.67 ±  3%      +4.5        6.13 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.67 ±  3%      +4.5        6.13 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
      0.01 ±173%      +4.9        4.90 ±  3%  perf-profile.children.cycles-pp.unmap_vmas
      0.01 ±173%      +4.9        4.90 ±  3%  perf-profile.children.cycles-pp.unmap_page_range
     40.86            +5.0       45.83        perf-profile.children.cycles-pp.page_fault
      1.02 ±  3%      +5.2        6.21 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter_state
      1.02 ±  3%      +5.2        6.22 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter
      1.02 ±  3%      +6.2        7.20 ±  4%  perf-profile.children.cycles-pp.start_secondary
      1.05 ±  3%      +6.7        7.75 ±  4%  perf-profile.children.cycles-pp.do_idle
      1.05 ±  3%      +6.7        7.76 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64
      1.05 ±  3%      +6.7        7.76 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.05 ± 58%     +10.7       10.76 ±  4%  perf-profile.children.cycles-pp.copy_user_highpage
      0.23 ±  8%     +13.0       13.18 ±  2%  perf-profile.children.cycles-pp.reuse_swap_page
      0.10 ± 14%     +20.6       20.72 ±  3%  perf-profile.children.cycles-pp.wp_page_copy
      0.11 ± 12%     +38.6       38.76        perf-profile.children.cycles-pp.do_wp_page
      0.13 ± 10%     +39.2       39.29        perf-profile.children.cycles-pp.handle_pte_fault
     38.32           -27.8       10.54 ±  4%  perf-profile.self.cycles-pp.copy_page
     43.91           -19.4       24.51        perf-profile.self.cycles-pp.do_rw_once
      0.10 ± 17%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.io_serial_out
      0.06 ±  7%      +0.0        0.10 ± 12%  perf-profile.self.cycles-pp.update_curr
      0.04 ± 57%      +0.0        0.09 ± 15%  perf-profile.self.cycles-pp.sys_imageblit
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.pmd_pfn
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.update_rq_clock
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.free_unref_page_prepare
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.uncharge_page
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.mem_cgroup_page_lruvec
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.vm_normal_page
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.get_vma_policy
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.orc_find
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.stack_trace_save_tsk
      0.00            +0.1        0.08 ± 11%  perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.08 ± 16%  perf-profile.self.cycles-pp.mem_cgroup_from_task
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.perf_exclude_event
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.mem_cgroup_charge_statistics
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.mem_cgroup_update_lru_size
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.page_mapping
      0.00            +0.1        0.09 ± 13%  perf-profile.self.cycles-pp.pmd_devmap_trans_unstable
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.free_unref_page_list
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.01 ±173%      +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.native_write_msr
      0.00            +0.1        0.09 ± 11%  perf-profile.self.cycles-pp.alloc_pages_vma
      0.00            +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.mem_cgroup_try_charge
      0.00            +0.1        0.10 ± 14%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.1        0.10 ± 17%  perf-profile.self.cycles-pp.enqueue_entity
      0.00            +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.delay_tsc
      0.00            +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.1        0.11 ± 14%  perf-profile.self.cycles-pp.__switch_to_asm
      0.00            +0.1        0.11 ± 12%  perf-profile.self.cycles-pp.mem_cgroup_commit_charge
      0.00            +0.1        0.11 ±  7%  perf-profile.self.cycles-pp.free_unref_page_commit
      0.00            +0.1        0.11 ±  6%  perf-profile.self.cycles-pp.pagevec_lru_move_fn
      0.42 ± 12%      +0.1        0.53 ±  8%  perf-profile.self.cycles-pp.io_serial_in
      0.00            +0.1        0.11 ±  3%  perf-profile.self.cycles-pp.cpumask_any_but
      0.00            +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.__switch_to
      0.00            +0.1        0.12 ± 11%  perf-profile.self.cycles-pp.lru_cache_add_active_or_unevictable
      0.00            +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.13 ±  6%  perf-profile.self.cycles-pp.flush_tlb_func_common
      0.00            +0.1        0.13 ± 12%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.00            +0.1        0.13 ±  9%  perf-profile.self.cycles-pp.down_read_trylock
      0.00            +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.page_add_new_anon_rmap
      0.00            +0.1        0.13 ± 12%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.__orc_find
      0.00            +0.1        0.13 ± 11%  perf-profile.self.cycles-pp.set_next_entity
      0.09 ±  8%      +0.1        0.23 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.00            +0.1        0.14 ± 13%  perf-profile.self.cycles-pp.__lock_page
      0.00            +0.1        0.15 ±  7%  perf-profile.self.cycles-pp.copy_user_highpage
      0.00            +0.1        0.15 ±  3%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.00            +0.1        0.15 ±  5%  perf-profile.self.cycles-pp.__might_sleep
      0.00            +0.1        0.15 ±  4%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.00            +0.2        0.15 ±  9%  perf-profile.self.cycles-pp.ptep_clear_flush
      0.00            +0.2        0.16 ± 10%  perf-profile.self.cycles-pp.__mod_zone_page_state
      0.00            +0.2        0.17 ±  6%  perf-profile.self.cycles-pp.__lru_cache_add
      0.00            +0.2        0.17 ±  5%  perf-profile.self.cycles-pp.up_read
      0.00            +0.2        0.19 ±  4%  perf-profile.self.cycles-pp.__count_memcg_events
      0.00            +0.2        0.20 ±  7%  perf-profile.self.cycles-pp.__perf_sw_event
      0.00            +0.2        0.20 ±  8%  perf-profile.self.cycles-pp.unlock_page
      0.00            +0.2        0.21 ±  5%  perf-profile.self.cycles-pp.mem_cgroup_throttle_swaprate
      0.00            +0.2        0.21 ±  7%  perf-profile.self.cycles-pp.page_fault
      0.00            +0.2        0.22 ±  3%  perf-profile.self.cycles-pp.__list_add_valid
      0.00            +0.2        0.22 ± 13%  perf-profile.self.cycles-pp.__sched_text_start
      0.00            +0.2        0.23 ±  3%  perf-profile.self.cycles-pp.vmacache_find
      0.00            +0.2        0.23 ±  3%  perf-profile.self.cycles-pp.handle_pte_fault
      0.00            +0.2        0.23 ±  4%  perf-profile.self.cycles-pp.free_pages_and_swap_cache
      0.00            +0.2        0.24 ±  9%  perf-profile.self.cycles-pp.unwind_next_frame
      0.00            +0.2        0.24 ±  5%  perf-profile.self.cycles-pp.flush_tlb_mm_range
      0.00            +0.2        0.24 ±  2%  perf-profile.self.cycles-pp.__split_huge_pmd_locked
      0.00            +0.2        0.24 ±  7%  perf-profile.self.cycles-pp._find_next_bit
      0.04 ± 57%      +0.2        0.28 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.2        0.24 ±  4%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.00            +0.3        0.28 ±  3%  perf-profile.self.cycles-pp.put_page
      0.00            +0.3        0.30 ±  5%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.00            +0.3        0.30 ±  4%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.00            +0.3        0.34 ±  6%  perf-profile.self.cycles-pp.do_page_fault
      0.00            +0.4        0.35 ±  7%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.00            +0.4        0.40 ± 13%  perf-profile.self.cycles-pp.try_to_wake_up
      0.00            +0.4        0.41 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      0.00            +0.4        0.42 ±  7%  perf-profile.self.cycles-pp.page_remove_rmap
      0.00            +0.4        0.45 ±  5%  perf-profile.self.cycles-pp.release_pages
      0.00            +0.5        0.45        perf-profile.self.cycles-pp.wp_page_copy
      0.00            +0.5        0.46 ±  4%  perf-profile.self.cycles-pp.unmap_page_range
      0.00            +0.5        0.48 ±  2%  perf-profile.self.cycles-pp.handle_mm_fault
      0.00            +0.5        0.49 ±  2%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.00            +0.5        0.54 ±  2%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.04 ± 58%      +0.6        0.62        perf-profile.self.cycles-pp.___perf_sw_event
      0.65 ±  3%      +0.6        1.26        perf-profile.self.cycles-pp.get_page_from_freelist
      0.00            +0.6        0.64 ±  4%  perf-profile.self.cycles-pp.mem_cgroup_try_charge_delay
      0.00            +0.6        0.64 ±  2%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.01 ±173%      +0.8        0.80 ±  2%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.00            +0.9        0.86 ±  5%  perf-profile.self.cycles-pp.do_wp_page
      0.00            +1.2        1.16 ±  3%  perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.00            +1.3        1.31 ±  3%  perf-profile.self.cycles-pp.sync_regs
      0.08 ±  6%      +1.7        1.80 ±  2%  perf-profile.self.cycles-pp.native_irq_return_iret
     11.15            +2.3       13.43        perf-profile.self.cycles-pp.do_access
      0.43 ±  3%      +2.4        2.82 ±  4%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.96 ±  2%      +2.5        3.46 ±  5%  perf-profile.self.cycles-pp.intel_idle
      0.00            +2.6        2.62 ±  7%  perf-profile.self.cycles-pp.poll_idle
      0.22 ±  8%     +12.9       13.09        perf-profile.self.cycles-pp.reuse_swap_page


                                                                                
                           vm-scalability.time.user_time                        
                                                                                
  2600 +--------------------------------------------------------------------+   
       |.+.+.+..+.+.+.+.+.+.. .+.+.+.+.+..+.+.+.+.+.+.+..+.+. .+.+.+..+.+.+.|   
  2500 |-+                   +                               +              |   
  2400 |-+                                                                  |   
       |                                                                    |   
  2300 |-+                                                                  |   
  2200 |-+                                                                  |   
       |                                                                    |   
  2100 |-+                                                                  |   
  2000 |-+                                                                  |   
       |                                                                    |   
  1900 |-+                                                                  |   
  1800 |-+                                                                  |   
       | O O O  O O O O O O  O O O O O O  O O O O O O O  O O O O O O        |   
  1700 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          vm-scalability.time.system_time                       
                                                                                
  2300 +--------------------------------------------------------------------+   
       | O O    O       O O  O O O O O O  O O O O O O O  O O O O O O        |   
  2250 |-+            O                                                     |   
  2200 |-+                                                                  |   
       |                                                                    |   
  2150 |-+                                                                  |   
  2100 |-+                                                                  |   
       |                                                                    |   
  2050 |-+                                                                  |   
  2000 |-+                                                                  |   
       |                                                                    |   
  1950 |-+                                                                  |   
  1900 |.+.+.+..+. .+.+.+.+..+.+.+.+.+.+..+.+.+.+.+.+.+..+.+.+.+.+.+.. .+.+.|   
       |          +                                                   +     |   
  1850 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                  vm-scalability.time.percent_of_cpu_this_job_got               
                                                                                
  1480 +--------------------------------------------------------------------+   
       |.+.+.+..+.+.+.+.+.+..+.+.+.+.+.+..+.+.+.+.+.+.+..+.+. .+.+.+..+.+. .|   
  1460 |-+                                                   +            + |   
  1440 |-+                                                                  |   
       |                                                                    |   
  1420 |-+                                                                  |   
       |                                                                    |   
  1400 |-+                                                                  |   
       |                                                                    |   
  1380 |-+                                                                  |   
  1360 |-+                                                                  |   
       |                                                                    |   
  1340 |-O O O  O O O   O O      O     O  O     O        O     O   O        |   
       |              O      O O   O O      O O   O O O    O O   O          |   
  1320 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        vm-scalability.time.minor_page_faults                   
                                                                                
  7e+08 +-------------------------------------------------------------------+   
        | O O O  O O O O O O O O  O O O O O O O O  O O O O O O O O  O       |   
  6e+08 |-+                                                                 |   
        |                                                                   |   
  5e+08 |-+                                                                 |   
        |                                                                   |   
  4e+08 |-+                                                                 |   
        |                                                                   |   
  3e+08 |-+                                                                 |   
        |                                                                   |   
  2e+08 |-+                                                                 |   
        |                                                                   |   
  1e+08 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                   vm-scalability.time.voluntary_context_switches               
                                                                                
  6e+07 +-------------------------------------------------------------------+   
        |                                                                   |   
  5e+07 |-+   O    O       O O O  O O O O O O O O  O O O O O O O O  O       |   
        | O O    O   O O O                                                  |   
        |                                                                   |   
  4e+07 |-+                                                                 |   
        |                                                                   |   
  3e+07 |-+                                                                 |   
        |                                                                   |   
  2e+07 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  1e+07 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               vm-scalability.throughput                        
                                                                                
  1.18e+07 +----------------------------------------------------------------+   
           |                                                                |   
  1.16e+07 |-+           O O O O O O O   O  O O O O O O O O O O O O O       |   
           |         O O               O                                    |   
  1.14e+07 |-O O O O                                                        |   
           |                                                                |   
  1.12e+07 |-+                                                              |   
           |                                                                |   
   1.1e+07 |-+                                                              |   
           |                                                                |   
  1.08e+07 |-+                                                              |   
           |                                                                |   
  1.06e+07 |-+                                                              |   
           |.+.+.+.+.+.+.+.+.+.+. .+.+.+.+..+.+.+.+.+.+.+.+. .+.+.+.+.+.+.+.|   
  1.04e+07 +----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.median                           
                                                                                
  730000 +------------------------------------------------------------------+   
         |              O     O O O O   O    O O O O O O O O O O  O O       |   
  720000 |-+       O  O   O O         O    O                                |   
  710000 |-O O O O                                                          |   
         |                                                                  |   
  700000 |-+                                                                |   
         |                                                                  |   
  690000 |-+                                                                |   
         |                                                                  |   
  680000 |-+                                                                |   
  670000 |-+                                                                |   
         |                                                                  |   
  660000 |-+  .+.+.  .+.+. .+.            .+.+.     .+.+.                   |   
         |.+.+     +.     +   +.+.+.+.+.+.     +.+.+     +.+.+.+..+.+.+.+.+.|   
  650000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 48G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/25%/debian-x86_64-20191114.cgz/lkp-bdw-de1/pxz/0x7000019

commit: 
  7ad886e462 ("khugepaged: Allow to collapse PTE-mapped compound pages")
  fd80f22b29 ("thp: Change CoW semantics for anon-THP")

7ad886e4626dd5ac fd80f22b2954c87d04f4ad55748 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
  24422927            -2.7%   23775553        pxz.throughput
    304.46            +2.5%     311.94        pxz.time.elapsed_time
    304.46            +2.5%     311.94        pxz.time.elapsed_time.max
    231182           +64.2%     379668        pxz.time.minor_page_faults
      1186            +2.0%       1209        pxz.time.user_time
     15150            -3.0%      14702        pxz.time.voluntary_context_switches
   3811331           +51.3%    5765084 ± 21%  cpuidle.C1E.usage
     32297            +3.2%      33323        vmstat.system.in
    871.50 ±  5%     +27.7%       1113 ±  5%  slabinfo.kmalloc-rcl-96.active_objs
    871.50 ±  5%     +27.7%       1113 ±  5%  slabinfo.kmalloc-rcl-96.num_objs
   3155553            +4.7%    3303159        proc-vmstat.numa_hit
   3155553            +4.7%    3303159        proc-vmstat.numa_local
    637123           +25.2%     797595        proc-vmstat.pgfault
     13278            -2.4%      12953        proc-vmstat.thp_fault_alloc
     11502 ± 11%     +17.8%      13545 ±  4%  softirqs.CPU1.RCU
     12311 ±  8%     +15.4%      14208 ±  3%  softirqs.CPU12.RCU
    139551 ±  7%     -14.6%     119214 ± 12%  softirqs.CPU13.TIMER
     12395 ±  3%     +13.1%      14013 ±  3%  softirqs.CPU6.RCU
     27539 ± 19%     +40.0%      38542 ± 14%  softirqs.CPU7.SCHED
     12972 ±  2%     +18.5%      15366 ±  9%  softirqs.CPU8.RCU
    113302 ± 17%     -20.4%      90201 ± 10%  sched_debug.cfs_rq:/.exec_clock.max
    493419 ± 17%     -19.8%     395713 ±  9%  sched_debug.cfs_rq:/.min_vruntime.max
      2.67 ±173%    +339.9%      11.73 ±  9%  sched_debug.cfs_rq:/.removed.load_avg.avg
     42.67 ±173%    +339.9%     187.69 ±  9%  sched_debug.cfs_rq:/.removed.load_avg.max
     10.33 ±173%    +339.9%      45.43 ±  9%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      2.67 ±173%    +339.9%      11.73 ±  9%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
     42.67 ±173%    +339.9%     187.69 ±  9%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     10.33 ±173%    +339.9%      45.43 ±  9%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      0.34 ±173%    +495.3%       2.05 ± 56%  sched_debug.cfs_rq:/.removed.util_avg.avg
      5.50 ±173%    +495.3%      32.74 ± 56%  sched_debug.cfs_rq:/.removed.util_avg.max
      1.33 ±173%    +495.3%       7.93 ± 56%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      1217 ±  6%     -13.3%       1055 ±  5%  sched_debug.cfs_rq:/.util_avg.max
    399.75 ± 38%     -51.3%     194.75 ±  4%  interrupts.37:IR-PCI-MSI.2621444-edge.eth0-TxRx-3
     58590 ±  4%    +545.8%     378380        interrupts.CAL:Function_call_interrupts
      2752 ± 65%    +792.8%      24573 ± 29%  interrupts.CPU1.CAL:Function_call_interrupts
      4048 ± 67%   +1034.4%      45918 ± 30%  interrupts.CPU1.TLB:TLB_shootdowns
      2120 ± 94%    +820.7%      19526 ± 34%  interrupts.CPU10.CAL:Function_call_interrupts
      2745 ±105%   +1116.5%      33395 ± 35%  interrupts.CPU10.TLB:TLB_shootdowns
    399.75 ± 38%     -51.3%     194.75 ±  4%  interrupts.CPU11.37:IR-PCI-MSI.2621444-edge.eth0-TxRx-3
      3938 ± 99%    +760.3%      33884 ± 45%  interrupts.CPU12.CAL:Function_call_interrupts
      5410 ±104%    +979.4%      58401 ± 47%  interrupts.CPU12.TLB:TLB_shootdowns
      7037 ± 43%    +317.6%      29386 ± 34%  interrupts.CPU14.CAL:Function_call_interrupts
      9626 ± 43%    +420.9%      50142 ± 33%  interrupts.CPU14.TLB:TLB_shootdowns
      5027 ± 73%    +541.9%      32274 ± 50%  interrupts.CPU15.CAL:Function_call_interrupts
      6938 ± 77%    +680.5%      54148 ± 50%  interrupts.CPU15.TLB:TLB_shootdowns
      4511 ± 74%    +758.0%      38706 ± 32%  interrupts.CPU3.CAL:Function_call_interrupts
      6429 ± 77%   +1003.4%      70939 ± 34%  interrupts.CPU3.TLB:TLB_shootdowns
      1704 ±152%   +1783.2%      32099 ± 24%  interrupts.CPU6.CAL:Function_call_interrupts
      2278 ±169%   +2461.8%      58376 ± 24%  interrupts.CPU6.TLB:TLB_shootdowns
      2680 ± 72%   +1546.6%      44137 ± 27%  interrupts.CPU8.CAL:Function_call_interrupts
      3948 ± 76%   +1908.1%      79279 ± 27%  interrupts.CPU8.TLB:TLB_shootdowns
     81570 ±  4%    +718.5%     667673        interrupts.TLB:TLB_shootdowns
      0.12 ± 37%      -0.1        0.05 ± 58%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.14 ± 19%      -0.1        0.09 ± 38%  perf-profile.children.cycles-pp.tick_irq_enter
      0.06 ± 60%      +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.sys_imageblit
      0.06 ± 60%      +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit
      0.09 ± 32%      +0.1        0.15 ± 21%  perf-profile.children.cycles-pp.lf
      0.09 ± 32%      +0.1        0.15 ± 21%  perf-profile.children.cycles-pp.con_scroll
      0.09 ± 32%      +0.1        0.15 ± 21%  perf-profile.children.cycles-pp.fbcon_scroll
      0.01 ±173%      +0.1        0.08 ± 19%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.10 ± 33%      +0.1        0.16 ± 22%  perf-profile.children.cycles-pp.vt_console_print
      0.09 ± 35%      +0.1        0.15 ± 21%  perf-profile.children.cycles-pp.fbcon_redraw
      0.15 ± 36%      +0.1        0.22 ± 21%  perf-profile.children.cycles-pp.copyout
      0.20 ± 23%      +0.1        0.28 ±  7%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.21 ± 44%      +0.1        0.33 ± 30%  perf-profile.children.cycles-pp.clockevents_program_event
      0.37 ± 11%      +0.1        0.51 ± 13%  perf-profile.children.cycles-pp.memcpy_erms
      0.37 ± 11%      +0.1        0.51 ± 13%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      0.90 ±  9%      +0.2        1.05 ± 10%  perf-profile.children.cycles-pp.uart_console_write
      0.37 ± 10%      +0.2        0.52 ± 14%  perf-profile.children.cycles-pp.process_one_work
      0.38 ±  9%      +0.2        0.53 ± 13%  perf-profile.children.cycles-pp.worker_thread
      0.41 ±  6%      +0.2        0.58 ± 11%  perf-profile.children.cycles-pp.ret_from_fork
      0.41 ±  5%      +0.2        0.58 ± 11%  perf-profile.children.cycles-pp.kthread
      0.83 ±  8%      +0.2        1.04 ± 16%  perf-profile.children.cycles-pp.irq_work_run
      0.83 ±  8%      +0.2        1.04 ± 16%  perf-profile.children.cycles-pp.printk
      0.83 ±  8%      +0.2        1.04 ± 16%  perf-profile.children.cycles-pp.vprintk_emit
      1.01 ± 10%      +0.2        1.23 ±  6%  perf-profile.children.cycles-pp.console_unlock
      0.83 ±  8%      +0.2        1.05 ± 16%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.83 ±  8%      +0.2        1.05 ± 16%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      1.02 ± 10%      +0.2        1.27 ±  2%  perf-profile.children.cycles-pp.irq_work_run_list
      0.06 ± 60%      +0.1        0.12 ± 25%  perf-profile.self.cycles-pp.sys_imageblit
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.10 ± 32%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.33 ± 14%      +0.2        0.51 ± 13%  perf-profile.self.cycles-pp.memcpy_erms



***************************************************************************************************
lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/sc_pid_max/tbox_group/testcase/testtime/ucode:
  scheduler/gcc-7/performance/1HDD/x86_64-rhel-7.6/100%/debian-x86_64-20191114.cgz/4194304/lkp-bdw-ep6/stress-ng/1s/0xb000038

commit: 
  7ad886e462 ("khugepaged: Allow to collapse PTE-mapped compound pages")
  fd80f22b29 ("thp: Change CoW semantics for anon-THP")

7ad886e4626dd5ac fd80f22b2954c87d04f4ad55748 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          2:4          -50%            :4     dmesg.WARNING:at_ip___perf_sw_event/0x
          2:4          -50%            :4     dmesg.WARNING:at_ip__slab_free/0x
           :4           25%           1:4     dmesg.WARNING:at_ip_fsnotify_parent/0x
          0:4            2%           0:4     perf-profile.children.cycles-pp.error_entry
          0:4            2%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     50095 ± 30%     +50.6%      75423 ±  4%  stress-ng.daemon.ops
     50028 ± 30%     +50.6%      75318 ±  4%  stress-ng.daemon.ops_per_sec
     19885 ±  5%    +169.7%      53639 ±  9%  stress-ng.fork.ops
     19886 ±  5%    +169.7%      53641 ±  9%  stress-ng.fork.ops_per_sec
  21792204 ±  4%     +10.5%   24083953        stress-ng.time.minor_page_faults
      3317            +4.3%       3460 ±  2%  boot-time.idle
    509206 ±  8%     +22.5%     623865 ± 15%  cpuidle.POLL.usage
     28124 ± 10%     +23.4%      34709 ±  8%  numa-meminfo.node0.KernelStack
     13.07 ±  8%     -16.4%      10.92 ±  9%  sched_debug.cpu.nr_uninterruptible.stddev
      8461 ±  8%     +16.5%       9856 ±  9%  softirqs.CPU18.RCU
      7663 ±  5%     +20.4%       9224 ± 12%  softirqs.CPU24.SCHED
      8458 ±  4%     +16.9%       9891 ±  5%  softirqs.CPU4.SCHED
      8136 ±  6%     +18.9%       9677 ±  5%  softirqs.CPU59.RCU
     28.11            +2.8%      28.90        iostat.cpu.idle
     62.92            -1.3%      62.10        iostat.cpu.system
      2.92 ±173%    -100.0%       0.00        iostat.sda.await.max
      2.92 ±173%    -100.0%       0.00        iostat.sda.r_await.max
      1.08 ±173%    -100.0%       0.00        iostat.sda.svctm.max
    510.50 ± 15%     -23.4%     391.25 ±  7%  proc-vmstat.nr_anon_transparent_hugepages
     11012 ± 14%     -56.7%       4771 ± 72%  proc-vmstat.numa_hint_faults_local
  39339177 ±  2%      +8.1%   42532079        proc-vmstat.numa_hit
  39209394 ±  2%      +8.1%   42398288        proc-vmstat.numa_local
  74529257 ± 20%     -30.2%   52008946        proc-vmstat.pgalloc_normal
  28830882 ±  4%     +13.8%   32816759        proc-vmstat.pgfault
  74292998 ± 20%     -30.2%   51819940        proc-vmstat.pgfree
     64139 ± 48%     -80.9%      12257        proc-vmstat.thp_fault_alloc
      2730 ±  3%     +34.5%       3673 ±  3%  proc-vmstat.thp_split_pmd
    824608 ± 13%     +16.2%     958502 ±  3%  proc-vmstat.unevictable_pgs_culled
    347634 ± 10%     +16.1%     403681 ±  4%  slabinfo.anon_vma_chain.active_objs
      5463 ±  9%     +15.9%       6333 ±  4%  slabinfo.anon_vma_chain.active_slabs
    349684 ±  9%     +15.9%     405400 ±  4%  slabinfo.anon_vma_chain.num_objs
      5463 ±  9%     +15.9%       6333 ±  4%  slabinfo.anon_vma_chain.num_slabs
     31629 ± 19%     +26.4%      39989 ±  4%  slabinfo.cred_jar.active_objs
    759.50 ± 19%     +29.2%     981.25 ±  4%  slabinfo.cred_jar.active_slabs
     31923 ± 19%     +29.2%      41240 ±  4%  slabinfo.cred_jar.num_objs
    759.50 ± 19%     +29.2%     981.25 ±  4%  slabinfo.cred_jar.num_slabs
     98068           +11.5%     109343 ±  2%  slabinfo.kmalloc-32.active_objs
     99981 ±  2%     +11.4%     111416 ±  2%  slabinfo.kmalloc-32.num_objs
     84986 ± 11%     +26.3%     107337 ±  4%  slabinfo.kmalloc-8.active_objs
     84986 ± 11%     +26.3%     107337 ±  4%  slabinfo.kmalloc-8.num_objs
     84292 ± 18%     +27.2%     107190 ±  4%  slabinfo.kmalloc-rcl-64.active_objs
      1324 ± 17%     +26.6%       1677 ±  4%  slabinfo.kmalloc-rcl-64.active_slabs
     84795 ± 17%     +26.6%     107363 ±  4%  slabinfo.kmalloc-rcl-64.num_objs
      1324 ± 17%     +26.6%       1677 ±  4%  slabinfo.kmalloc-rcl-64.num_slabs
     25273 ±  8%     +22.0%      30834 ±  2%  slabinfo.pid.active_objs
    800.00 ±  8%     +22.9%     983.25        slabinfo.pid.active_slabs
     25609 ±  8%     +22.9%      31478 ±  2%  slabinfo.pid.num_objs
    800.00 ±  8%     +22.9%     983.25        slabinfo.pid.num_slabs
      6048 ± 16%     +23.8%       7489 ±  4%  slabinfo.sighand_cache.active_objs
      6276 ± 16%     +24.8%       7833 ±  5%  slabinfo.sighand_cache.num_objs
     33366 ±  6%     +26.0%      42048 ±  2%  slabinfo.task_delay_info.active_objs
    659.75 ±  6%     +26.4%     833.75 ±  2%  slabinfo.task_delay_info.active_slabs
     33674 ±  6%     +26.4%      42550 ±  2%  slabinfo.task_delay_info.num_objs
    659.75 ±  6%     +26.4%     833.75 ±  2%  slabinfo.task_delay_info.num_slabs
      7415 ± 19%     -25.1%       5551 ±  6%  interrupts.CPU0.CAL:Function_call_interrupts
     47772 ± 35%     -67.2%      15649 ± 29%  interrupts.CPU13.RES:Rescheduling_interrupts
      6241 ± 22%     -25.7%       4636 ± 13%  interrupts.CPU15.CAL:Function_call_interrupts
      6458 ± 20%     -34.2%       4252 ± 14%  interrupts.CPU21.CAL:Function_call_interrupts
    218.25 ± 68%     -80.3%      43.00 ±113%  interrupts.CPU22.IWI:IRQ_work_interrupts
      1070 ± 91%     -97.3%      29.25 ±122%  interrupts.CPU23.IWI:IRQ_work_interrupts
      4404 ± 10%     +34.3%       5913 ± 21%  interrupts.CPU24.CAL:Function_call_interrupts
      5889 ± 20%     -25.2%       4408 ± 17%  interrupts.CPU28.CAL:Function_call_interrupts
      4589 ± 33%     +42.1%       6521 ±  3%  interrupts.CPU3.NMI:Non-maskable_interrupts
      4589 ± 33%     +42.1%       6521 ±  3%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
     66716 ± 35%     -49.5%      33712 ± 77%  interrupts.CPU3.RES:Rescheduling_interrupts
      1544 ± 27%     +71.2%       2644 ± 27%  interrupts.CPU31.TLB:TLB_shootdowns
      3669 ± 24%     +96.9%       7222 ± 30%  interrupts.CPU33.CAL:Function_call_interrupts
    434603 ± 13%     -40.6%     258166 ± 28%  interrupts.CPU33.LOC:Local_timer_interrupts
     11467 ± 16%    +181.4%      32269 ± 78%  interrupts.CPU33.RES:Rescheduling_interrupts
      3916 ±  3%     +43.2%       5609 ± 24%  interrupts.CPU34.CAL:Function_call_interrupts
     11596 ± 29%    +273.9%      43352 ± 49%  interrupts.CPU35.RES:Rescheduling_interrupts
      1989 ± 24%     +39.4%       2773 ± 10%  interrupts.CPU35.TLB:TLB_shootdowns
      1976 ± 14%     +36.0%       2687 ± 18%  interrupts.CPU40.TLB:TLB_shootdowns
      4323 ± 30%     +58.9%       6869 ± 28%  interrupts.CPU41.CAL:Function_call_interrupts
     14966 ± 42%    +259.9%      53863 ± 45%  interrupts.CPU41.RES:Rescheduling_interrupts
      1832 ± 34%     +43.5%       2629 ± 19%  interrupts.CPU75.TLB:TLB_shootdowns
      4043 ± 24%     +37.5%       5561 ± 14%  interrupts.CPU77.CAL:Function_call_interrupts
      1926 ± 24%     +51.7%       2922 ± 20%  interrupts.CPU77.TLB:TLB_shootdowns
      3515 ±  7%     +43.9%       5059 ± 12%  interrupts.CPU78.CAL:Function_call_interrupts
      1669 ± 18%     +67.0%       2787 ± 24%  interrupts.CPU78.TLB:TLB_shootdowns
      1917 ± 16%     +42.9%       2739 ± 24%  interrupts.CPU80.TLB:TLB_shootdowns
     29285 ± 68%     +79.5%      52568 ± 46%  interrupts.CPU82.RES:Rescheduling_interrupts
      1700 ± 34%     +82.2%       3097 ± 35%  interrupts.CPU82.TLB:TLB_shootdowns
     20701 ± 60%     -68.4%       6533 ±107%  interrupts.IWI:IRQ_work_interrupts
    485140 ±  3%     +10.3%     535050 ±  2%  interrupts.NMI:Non-maskable_interrupts
    485140 ±  3%     +10.3%     535050 ±  2%  interrupts.PMI:Performance_monitoring_interrupts
      3.65 ±100%      -3.7        0.00        perf-profile.calltrace.cycles-pp.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
      3.55 ±100%      -3.6        0.00        perf-profile.calltrace.cycles-pp.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault.do_page_fault
      3.52 ±100%      -3.5        0.00        perf-profile.calltrace.cycles-pp.copy_subpage.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault
      3.45 ±100%      -3.5        0.00        perf-profile.calltrace.cycles-pp.copy_page.copy_subpage.copy_user_huge_page.do_huge_pmd_wp_page.__handle_mm_fault
     14.07 ±  3%      -0.9       13.20        perf-profile.calltrace.cycles-pp.secondary_startup_64
     13.90 ±  3%      -0.9       13.04        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     13.90 ±  3%      -0.9       13.05        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     13.89 ±  3%      -0.9       13.04        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     11.66 ±  3%      -0.8       10.81 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     11.61 ±  4%      -0.8       10.77 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     10.64 ±  2%      -0.5       10.16        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_close
     10.62 ±  2%      -0.5       10.14        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close
      7.54 ±  2%      -0.4        7.11        perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.27 ±  5%      -0.4        1.85 ±  4%  perf-profile.calltrace.cycles-pp.__dentry_kill.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      2.21 ±  6%      -0.4        1.79 ±  4%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.__fput.task_work_run.exit_to_usermode_loop
      4.89 ±  5%      -0.4        4.48        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.munmap
      4.96 ±  5%      -0.4        4.56        perf-profile.calltrace.cycles-pp.munmap
      4.88 ±  5%      -0.4        4.48        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      3.83 ±  3%      -0.4        3.46 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
      3.84 ±  3%      -0.4        3.47 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.mmap64
      3.72 ±  3%      -0.4        3.35 ±  2%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
      3.73 ±  3%      -0.4        3.36 ±  2%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
      1.72 ±  4%      -0.4        1.35 ±  2%  perf-profile.calltrace.cycles-pp.inode_sb_list_add.new_inode.shmem_get_inode.__shmem_file_setup.shmem_zero_setup
      3.91 ±  2%      -0.4        3.55 ±  2%  perf-profile.calltrace.cycles-pp.mmap64
      1.67 ±  4%      -0.4        1.31 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.inode_sb_list_add.new_inode.shmem_get_inode
      1.69 ±  3%      -0.4        1.33 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock.inode_sb_list_add.new_inode.shmem_get_inode.__shmem_file_setup
      1.98 ±  3%      -0.4        1.62 ±  2%  perf-profile.calltrace.cycles-pp.__shmem_file_setup.shmem_zero_setup.mmap_region.do_mmap.vm_mmap_pgoff
      1.79 ±  3%      -0.4        1.44 ±  2%  perf-profile.calltrace.cycles-pp.new_inode.shmem_get_inode.__shmem_file_setup.shmem_zero_setup.mmap_region
      1.98 ±  3%      -0.4        1.63 ±  2%  perf-profile.calltrace.cycles-pp.shmem_zero_setup.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      1.83 ±  4%      -0.4        1.48 ±  2%  perf-profile.calltrace.cycles-pp.shmem_get_inode.__shmem_file_setup.shmem_zero_setup.mmap_region.do_mmap
      1.63 ±  4%      -0.3        1.28 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.evict.__dentry_kill.__fput
      2.24 ±  8%      -0.3        1.90 ±  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      2.22 ±  3%      -0.3        1.87 ±  2%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      2.31 ±  3%      -0.3        1.97 ±  2%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.65 ±  4%      -0.3        1.31 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.evict.__dentry_kill.__fput.task_work_run
      2.00 ± 13%      -0.3        1.68 ±  8%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      5.49 ±  3%      -0.3        5.18        perf-profile.calltrace.cycles-pp.do_fcntl
      5.46 ±  3%      -0.3        5.15        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.do_fcntl
      5.46 ±  3%      -0.3        5.15        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.do_fcntl
      5.41 ±  3%      -0.3        5.10        perf-profile.calltrace.cycles-pp.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe.do_fcntl
      1.94 ± 13%      -0.3        1.63 ±  8%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      5.42 ±  3%      -0.3        5.11        perf-profile.calltrace.cycles-pp.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe.do_fcntl
      5.75 ±  2%      -0.3        5.44        perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close
      5.41 ±  3%      -0.3        5.10        perf-profile.calltrace.cycles-pp.fcntl_dirnotify.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.34 ±  2%      -0.3        5.04        perf-profile.calltrace.cycles-pp.dnotify_flush.filp_close.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.28 ±  3%      -0.3        4.98        perf-profile.calltrace.cycles-pp.__mutex_lock.fcntl_dirnotify.do_fcntl.__x64_sys_fcntl.do_syscall_64
      5.63 ±  2%      -0.3        5.33        perf-profile.calltrace.cycles-pp.filp_close.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close
      5.25 ±  2%      -0.3        4.95        perf-profile.calltrace.cycles-pp.__mutex_lock.dnotify_flush.filp_close.__x64_sys_close.do_syscall_64
      5.20 ±  3%      -0.3        4.90        perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.fcntl_dirnotify.do_fcntl.__x64_sys_fcntl
      3.51 ±  3%      -0.3        3.22        perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      3.50 ±  3%      -0.3        3.21        perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
      5.17 ±  2%      -0.3        4.87        perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.dnotify_flush.filp_close.__x64_sys_close
      1.42 ±  9%      -0.3        1.17 ±  4%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      1.39 ± 10%      -0.2        1.14 ±  4%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      1.66 ±  5%      -0.2        1.47 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unshare
      1.66 ±  5%      -0.2        1.47 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unshare
      1.65 ±  5%      -0.2        1.47 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_unshare.do_syscall_64.entry_SYSCALL_64_after_hwframe.unshare
      1.65 ±  5%      -0.2        1.47 ±  4%  perf-profile.calltrace.cycles-pp.ksys_unshare.__x64_sys_unshare.do_syscall_64.entry_SYSCALL_64_after_hwframe.unshare
      1.65 ±  5%      -0.2        1.46 ±  5%  perf-profile.calltrace.cycles-pp.unshare_nsproxy_namespaces.ksys_unshare.__x64_sys_unshare.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.67 ±  5%      -0.2        1.48 ±  4%  perf-profile.calltrace.cycles-pp.unshare
      1.65 ±  5%      -0.2        1.46 ±  5%  perf-profile.calltrace.cycles-pp.create_new_namespaces.unshare_nsproxy_namespaces.ksys_unshare.__x64_sys_unshare.do_syscall_64
      1.25 ±  6%      -0.2        1.07 ±  6%  perf-profile.calltrace.cycles-pp.setup_net.copy_net_ns.create_new_namespaces.unshare_nsproxy_namespaces.ksys_unshare
      1.25 ±  6%      -0.2        1.07 ±  6%  perf-profile.calltrace.cycles-pp.ops_init.setup_net.copy_net_ns.create_new_namespaces.unshare_nsproxy_namespaces
      1.25 ±  6%      -0.2        1.08 ±  6%  perf-profile.calltrace.cycles-pp.copy_net_ns.create_new_namespaces.unshare_nsproxy_namespaces.ksys_unshare.__x64_sys_unshare
      1.32 ±  4%      -0.1        1.20 ±  5%  perf-profile.calltrace.cycles-pp.__nanosleep
      1.09 ±  5%      -0.1        0.97 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__nanosleep
      1.10 ±  5%      -0.1        0.99 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__nanosleep
      0.89 ±  7%      -0.1        0.79 ±  5%  perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.__nanosleep
      0.87 ±  7%      -0.1        0.77 ±  5%  perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.71 ± 10%      -0.1        0.62 ±  6%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
      1.31 ±  2%      -0.1        1.24 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
      1.31 ±  2%      -0.1        1.24 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.31 ±  2%      -0.1        1.25 ±  3%  perf-profile.calltrace.cycles-pp.creat
      1.29 ±  2%      -0.1        1.23 ±  3%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.29 ±  2%      -0.1        1.23 ±  3%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      0.61 ±  6%      +0.1        0.69 ±  4%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit
      0.61 ±  6%      +0.1        0.70 ±  4%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit
      1.74 ±  3%      +0.1        1.85 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
      1.80 ±  3%      +0.1        1.91 ±  2%  perf-profile.calltrace.cycles-pp.unlink
      1.74 ±  3%      +0.1        1.85 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.16            +0.1        1.28 ±  3%  perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.09 ±  2%      +0.1        1.21 ±  4%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
      1.59 ±  3%      +0.1        1.71 ±  2%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.04 ±  2%      +0.1        1.17 ±  4%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      1.11 ±  2%      +0.1        1.24 ±  3%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      0.78 ±  7%      +0.2        0.93 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock.evict.__dentry_kill.dput.__fput
      0.77 ±  5%      +0.2        0.93 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.evict.do_unlinkat.do_syscall_64
      0.76 ±  7%      +0.2        0.92 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.evict.__dentry_kill.dput
      0.78 ±  5%      +0.2        0.95 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.79 ± 14%      +0.2        1.01 ±  4%  perf-profile.calltrace.cycles-pp.filemap_map_pages.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault
      1.52 ±  6%      +0.2        1.74        perf-profile.calltrace.cycles-pp.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
      1.51 ±  6%      +0.2        1.76 ±  5%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      1.55 ±  6%      +0.3        1.81 ±  4%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      1.53 ±  6%      +0.3        1.85 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.inode_sb_list_add.new_inode.ramfs_get_inode
      1.73 ±  6%      +0.3        2.06 ±  3%  perf-profile.calltrace.cycles-pp.ramfs_get_inode.ramfs_mknod.path_openat.do_filp_open.do_sys_openat2
      1.70 ±  6%      +0.3        2.03 ±  3%  perf-profile.calltrace.cycles-pp.new_inode.ramfs_get_inode.ramfs_mknod.path_openat.do_filp_open
      1.56 ±  7%      +0.3        1.88 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.inode_sb_list_add.new_inode.ramfs_get_inode.ramfs_mknod
      3.07 ±  4%      +0.3        3.40        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      3.08 ±  4%      +0.3        3.41        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.58 ±  7%      +0.3        1.91 ±  3%  perf-profile.calltrace.cycles-pp.inode_sb_list_add.new_inode.ramfs_get_inode.ramfs_mknod.path_openat
      1.75 ±  6%      +0.3        2.08 ±  3%  perf-profile.calltrace.cycles-pp.ramfs_mknod.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.00            +0.5        0.54 ±  5%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput
      0.95 ± 66%      +0.9        1.83 ±  4%  perf-profile.calltrace.cycles-pp.do_select.core_sys_select.kern_select.__x64_sys_select.do_syscall_64
      1.09 ± 67%      +1.0        2.07 ±  5%  perf-profile.calltrace.cycles-pp.core_sys_select.kern_select.__x64_sys_select.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.26 ± 67%      +1.1        2.34 ±  7%  perf-profile.calltrace.cycles-pp.kern_select.__x64_sys_select.do_syscall_64.entry_SYSCALL_64_after_hwframe.__select
      1.27 ± 67%      +1.1        2.35 ±  7%  perf-profile.calltrace.cycles-pp.__x64_sys_select.do_syscall_64.entry_SYSCALL_64_after_hwframe.__select
      1.66 ± 67%      +1.1        2.81 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__select
      1.67 ± 67%      +1.2        2.82 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__select
      1.90 ± 67%      +1.3        3.18 ± 11%  perf-profile.calltrace.cycles-pp.__select
      9.60 ± 19%      +3.1       12.67 ±  9%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      9.65 ± 19%      +3.1       12.73 ±  8%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      2.92 ± 64%      +3.2        6.10 ± 22%  perf-profile.calltrace.cycles-pp.try_to_wake_up.pollwake.__wake_up_common.__wake_up_common_lock.pipe_write
      2.93 ± 64%      +3.2        6.11 ± 22%  perf-profile.calltrace.cycles-pp.pollwake.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
     10.16 ± 21%      +3.2       13.35 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     10.18 ± 21%      +3.2       13.37 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      2.96 ± 64%      +3.2        6.17 ± 21%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
      2.98 ± 64%      +3.2        6.21 ± 21%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
     10.46 ± 21%      +3.3       13.79 ±  6%  perf-profile.calltrace.cycles-pp.__GI___libc_write
      3.25 ± 64%      +3.4        6.61 ± 19%  perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      3.33 ± 64%      +3.4        6.73 ± 18%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.66 ±100%      -3.7        0.00        perf-profile.children.cycles-pp.do_huge_pmd_wp_page
      3.56 ±100%      -3.6        0.00        perf-profile.children.cycles-pp.copy_user_huge_page
      3.53 ±100%      -3.5        0.00        perf-profile.children.cycles-pp.copy_subpage
     16.72 ±  6%      -1.0       15.76        perf-profile.children.cycles-pp._raw_spin_lock
     14.07 ±  3%      -0.9       13.20        perf-profile.children.cycles-pp.do_idle
     14.07 ±  3%      -0.9       13.20        perf-profile.children.cycles-pp.secondary_startup_64
     14.07 ±  3%      -0.9       13.20        perf-profile.children.cycles-pp.cpu_startup_entry
     13.90 ±  3%      -0.9       13.05        perf-profile.children.cycles-pp.start_secondary
     11.79 ±  4%      -0.9       10.94 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
     11.80 ±  4%      -0.9       10.95 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
     11.12            -0.8       10.37        perf-profile.children.cycles-pp.__mutex_lock
     10.91 ±  3%      -0.6       10.29        perf-profile.children.cycles-pp.do_fcntl
      8.44 ±  2%      -0.6        7.85        perf-profile.children.cycles-pp.task_work_run
      7.64 ±  2%      -0.4        7.20        perf-profile.children.cycles-pp.__fput
      4.17 ±  4%      -0.4        3.75 ±  4%  perf-profile.children.cycles-pp.apic_timer_interrupt
      5.11 ±  5%      -0.4        4.72        perf-profile.children.cycles-pp.munmap
      3.99 ±  3%      -0.4        3.63 ±  2%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      4.08 ±  3%      -0.4        3.72 ±  2%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      3.99 ±  2%      -0.4        3.64 ±  2%  perf-profile.children.cycles-pp.mmap64
      1.98 ±  3%      -0.4        1.62 ±  2%  perf-profile.children.cycles-pp.__shmem_file_setup
      3.96 ±  5%      -0.4        3.60 ±  4%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.98 ±  3%      -0.4        1.63 ±  2%  perf-profile.children.cycles-pp.shmem_zero_setup
      1.85 ±  3%      -0.3        1.50 ±  2%  perf-profile.children.cycles-pp.shmem_get_inode
      2.24 ±  8%      -0.3        1.90 ±  5%  perf-profile.children.cycles-pp.kthread
      2.76 ±  8%      -0.3        2.42 ±  6%  perf-profile.children.cycles-pp.hrtimer_wakeup
      2.54 ±  3%      -0.3        2.21 ±  2%  perf-profile.children.cycles-pp.mmap_region
      3.39 ±  6%      -0.3        3.06 ±  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      2.65 ±  3%      -0.3        2.32 ±  2%  perf-profile.children.cycles-pp.do_mmap
      3.21 ±  6%      -0.3        2.88 ±  5%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      5.43 ±  3%      -0.3        5.12        perf-profile.children.cycles-pp.__x64_sys_fcntl
      5.41 ±  3%      -0.3        5.10        perf-profile.children.cycles-pp.fcntl_dirnotify
      5.35 ±  2%      -0.3        5.04        perf-profile.children.cycles-pp.dnotify_flush
      3.47 ±  3%      -0.3        3.17        perf-profile.children.cycles-pp.__dentry_kill
      5.76 ±  2%      -0.3        5.46        perf-profile.children.cycles-pp.__x64_sys_close
      8.00            -0.3        7.70        perf-profile.children.cycles-pp.exit_to_usermode_loop
      5.64 ±  2%      -0.3        5.35        perf-profile.children.cycles-pp.filp_close
      1.42 ±  9%      -0.3        1.17 ±  4%  perf-profile.children.cycles-pp.worker_thread
      1.39 ± 10%      -0.2        1.14 ±  4%  perf-profile.children.cycles-pp.process_one_work
      1.65 ±  5%      -0.2        1.46 ±  5%  perf-profile.children.cycles-pp.create_new_namespaces
      1.65 ±  5%      -0.2        1.47 ±  4%  perf-profile.children.cycles-pp.__x64_sys_unshare
      1.65 ±  5%      -0.2        1.47 ±  4%  perf-profile.children.cycles-pp.ksys_unshare
      1.65 ±  5%      -0.2        1.46 ±  5%  perf-profile.children.cycles-pp.unshare_nsproxy_namespaces
      1.67 ±  5%      -0.2        1.48 ±  4%  perf-profile.children.cycles-pp.unshare
      1.25 ±  6%      -0.2        1.07 ±  6%  perf-profile.children.cycles-pp.setup_net
      1.25 ±  6%      -0.2        1.07 ±  6%  perf-profile.children.cycles-pp.ops_init
      1.25 ±  6%      -0.2        1.08 ±  6%  perf-profile.children.cycles-pp.copy_net_ns
      0.61 ± 18%      -0.1        0.46 ±  7%  perf-profile.children.cycles-pp.loopback_net_init
      0.60 ± 18%      -0.1        0.46 ±  7%  perf-profile.children.cycles-pp.register_netdev
      0.48 ± 15%      -0.1        0.36 ±  9%  perf-profile.children.cycles-pp.memcpy_erms
      1.32 ±  4%      -0.1        1.20 ±  4%  perf-profile.children.cycles-pp.__nanosleep
      0.60 ± 11%      -0.1        0.51 ±  6%  perf-profile.children.cycles-pp.finish_task_switch
      0.57 ± 13%      -0.1        0.50 ±  2%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      1.31 ±  2%      -0.1        1.25 ±  3%  perf-profile.children.cycles-pp.creat
      0.32 ±  8%      -0.0        0.28 ±  2%  perf-profile.children.cycles-pp.___slab_alloc
      0.26 ±  8%      -0.0        0.22 ±  4%  perf-profile.children.cycles-pp.new_slab
      0.32 ±  8%      -0.0        0.29 ±  2%  perf-profile.children.cycles-pp.__slab_alloc
      0.08 ± 10%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.stop_one_cpu
      0.10 ±  7%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.put_cred_rcu
      0.50            +0.0        0.52        perf-profile.children.cycles-pp.link_path_walk
      0.15 ±  6%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.remove_vma
      0.04 ± 57%      +0.0        0.06        perf-profile.children.cycles-pp.__nptl_set_robust
      0.04 ± 57%      +0.0        0.06        perf-profile.children.cycles-pp.vmacache_find
      0.19 ±  6%      +0.0        0.22        perf-profile.children.cycles-pp.wake_up_new_task
      0.25 ±  5%      +0.0        0.28        perf-profile.children.cycles-pp.open64
      0.18 ±  2%      +0.0        0.21 ±  5%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.23 ±  4%      +0.0        0.26 ±  3%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.21 ±  2%      +0.0        0.24 ±  8%  perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.10 ± 18%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp.reweight_entity
      0.14 ± 10%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.12 ±  8%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.read
      0.10 ± 14%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.unlock_page
      0.13 ±  8%      +0.0        0.17 ±  3%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.06            +0.0        0.10 ± 35%  perf-profile.children.cycles-pp.vm_normal_page
      0.36 ±  3%      +0.0        0.41 ±  3%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.40 ±  5%      +0.0        0.44 ±  7%  perf-profile.children.cycles-pp.___might_sleep
      0.25 ± 12%      +0.0        0.29        perf-profile.children.cycles-pp.fput_many
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.lock_page_memcg
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.free_uid
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.refcount_dec_not_one
      0.26 ±  7%      +0.1        0.32 ±  2%  perf-profile.children.cycles-pp.alloc_set_pte
      0.32 ±  6%      +0.1        0.38 ±  3%  perf-profile.children.cycles-pp.down_write
      0.51 ±  7%      +0.1        0.62 ±  4%  perf-profile.children.cycles-pp.page_remove_rmap
      1.80 ±  3%      +0.1        1.91 ±  2%  perf-profile.children.cycles-pp.unlink
      0.85 ±  3%      +0.1        0.97 ±  5%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.50 ±  6%      +0.1        0.62 ± 13%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      1.59 ±  2%      +0.1        1.71 ±  2%  perf-profile.children.cycles-pp.do_unlinkat
      1.05 ±  3%      +0.1        1.18 ±  4%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.65 ± 17%      +0.1        0.79 ±  6%  perf-profile.children.cycles-pp.select_task_rq_fair
      1.74 ±  2%      +0.1        1.88 ±  2%  perf-profile.children.cycles-pp.dput
      0.89 ± 16%      +0.2        1.07 ±  7%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      2.02 ±  4%      +0.3        2.31 ±  4%  perf-profile.children.cycles-pp.unmap_page_range
      2.08 ±  4%      +0.3        2.37 ±  3%  perf-profile.children.cycles-pp.unmap_vmas
      2.08 ±  6%      +0.3        2.37 ±  2%  perf-profile.children.cycles-pp.handle_pte_fault
      1.15 ± 13%      +0.3        1.46 ±  4%  perf-profile.children.cycles-pp.filemap_map_pages
      1.74 ±  6%      +0.3        2.07 ±  3%  perf-profile.children.cycles-pp.ramfs_get_inode
      1.77 ±  6%      +0.3        2.10 ±  3%  perf-profile.children.cycles-pp.ramfs_mknod
      3.47 ±  4%      +0.3        3.82        perf-profile.children.cycles-pp.do_filp_open
      0.22 ± 59%      +0.3        0.56 ± 27%  perf-profile.children.cycles-pp.remove_wait_queue
      3.46 ±  3%      +0.4        3.81        perf-profile.children.cycles-pp.path_openat
      0.26 ± 59%      +0.4        0.63 ± 22%  perf-profile.children.cycles-pp.poll_freewait
      0.99 ± 59%      +0.8        1.83 ±  4%  perf-profile.children.cycles-pp.do_select
      1.15 ± 58%      +0.9        2.08 ±  5%  perf-profile.children.cycles-pp.core_sys_select
      1.31 ± 59%      +1.0        2.34 ±  7%  perf-profile.children.cycles-pp.kern_select
      1.32 ± 59%      +1.0        2.35 ±  7%  perf-profile.children.cycles-pp.__x64_sys_select
      1.99 ± 59%      +1.2        3.21 ± 11%  perf-profile.children.cycles-pp.__select
      8.40 ± 20%      +2.9       11.35 ± 11%  perf-profile.children.cycles-pp.__wake_up_common
     10.04 ± 19%      +3.0       13.06 ±  8%  perf-profile.children.cycles-pp.vfs_write
      9.86 ± 19%      +3.0       12.88 ±  8%  perf-profile.children.cycles-pp.ksys_write
      3.01 ± 61%      +3.2        6.17 ± 21%  perf-profile.children.cycles-pp.pollwake
      3.68 ± 50%      +3.2        6.87 ± 19%  perf-profile.children.cycles-pp.__wake_up_common_lock
     10.62 ± 21%      +3.3       13.91 ±  6%  perf-profile.children.cycles-pp.__GI___libc_write
      3.69 ± 55%      +3.3        7.03 ± 17%  perf-profile.children.cycles-pp.new_sync_write
      3.29 ± 61%      +3.3        6.62 ± 18%  perf-profile.children.cycles-pp.pipe_write
     72.84 ±  5%      +3.9       76.74        perf-profile.children.cycles-pp.do_syscall_64
     72.97 ±  5%      +3.9       76.89        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.47 ± 15%      -0.1        0.36 ±  9%  perf-profile.self.cycles-pp.memcpy_erms
      0.15 ±  5%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.new_slab
      0.08            -0.0        0.07        perf-profile.self.cycles-pp.menu_select
      0.05            +0.0        0.06 ±  6%  perf-profile.self.cycles-pp.handle_mm_fault
      0.07 ± 10%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.alloc_set_pte
      0.15 ±  3%      +0.0        0.17 ±  2%  perf-profile.self.cycles-pp.update_cfs_group
      0.10 ±  8%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp._cond_resched
      0.07 ± 16%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.orc_find
      0.22 ±  5%      +0.0        0.24 ±  3%  perf-profile.self.cycles-pp.free_pages_and_swap_cache
      0.10 ±  7%      +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.11 ±  6%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.18 ±  2%      +0.0        0.21 ±  5%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.10 ± 14%      +0.0        0.13 ±  3%  perf-profile.self.cycles-pp.unlock_page
      0.14 ±  8%      +0.0        0.17 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.17 ± 12%      +0.0        0.20 ±  2%  perf-profile.self.cycles-pp.fput_many
      0.37 ±  6%      +0.0        0.42 ±  8%  perf-profile.self.cycles-pp.___might_sleep
      0.26 ±  7%      +0.1        0.31 ±  2%  perf-profile.self.cycles-pp.down_write
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.refcount_dec_not_one
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.mark_page_accessed
      0.54 ±  3%      +0.1        0.63 ±  7%  perf-profile.self.cycles-pp.copy_p4d_range
      0.46 ±  8%      +0.1        0.56 ±  5%  perf-profile.self.cycles-pp.page_remove_rmap
      0.98 ±  4%      +0.1        1.10 ±  3%  perf-profile.self.cycles-pp.unmap_page_range
      0.57 ±  8%      +0.1        0.70 ±  4%  perf-profile.self.cycles-pp.release_pages
      0.89 ± 16%      +0.2        1.07 ±  7%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.76 ± 14%      +0.2        0.97 ±  5%  perf-profile.self.cycles-pp.filemap_map_pages





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.7.0-rc1-00053-gfd80f22b2954c" of type "text/plain" (206159 bytes)

View attachment "job-script" of type "text/plain" (7638 bytes)

View attachment "job.yaml" of type "text/plain" (5205 bytes)

View attachment "reproduce" of type "text/plain" (15066 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ