Message-ID: <5d2daec8-985c-c0a4-4832-8e5493316306@amazon.com>
Date:   Thu, 23 Jan 2020 15:52:48 +0100
From:   Alexander Graf <graf@...zon.com>
To:     David Hildenbrand <david@...hat.com>,
        Alexander Duyck <alexander.duyck@...il.com>,
        <kvm@...r.kernel.org>, <mst@...hat.com>,
        <linux-kernel@...r.kernel.org>, <willy@...radead.org>,
        <mhocko@...nel.org>, <linux-mm@...ck.org>,
        <akpm@...ux-foundation.org>, <mgorman@...hsingularity.net>,
        <vbabka@...e.cz>
CC:     <yang.zhang.wz@...il.com>, <nitesh@...hat.com>,
        <konrad.wilk@...cle.com>, <pagupta@...hat.com>, <riel@...riel.com>,
        <lcapitulino@...hat.com>, <dave.hansen@...el.com>,
        <wei.w.wang@...el.com>, <aarcange@...hat.com>,
        <pbonzini@...hat.com>, <dan.j.williams@...el.com>,
        <alexander.h.duyck@...ux.intel.com>, <osalvador@...e.de>,
        "Paterson-Jones, Roland" <rolandp@...zon.com>,
        <hannes@...xchg.org>, <hare@...e.com>
Subject: Re: [PATCH v16.1 0/9] mm / virtio: Provide support for free page
 reporting



On 23.01.20 15:05, David Hildenbrand wrote:
> On 23.01.20 11:20, Alexander Graf wrote:
>> Hi Alex,
>>
>> On 22.01.20 18:43, Alexander Duyck wrote:
>>> This series provides an asynchronous means of reporting free guest pages
>>> to a hypervisor so that the memory associated with those pages can be
>>> dropped and reused by other processes and/or guests on the host. Using
>>> this it is possible to avoid unnecessary I/O to disk and greatly improve
>>> performance in the case of memory overcommit on the host.
>>>
>>> When enabled we will be performing a scan of free memory every 2 seconds
>>> while pages of sufficiently high order are being freed. In each pass at
>>> least one sixteenth of each free list will be reported. By doing this we
>>> avoid racing against other threads that may be causing a high amount of
>>> memory churn.
>>>
>>> The lowest page order currently scanned when reporting pages is
>>> pageblock_order so that this feature will not interfere with the use of
>>> Transparent Huge Pages in the case of virtualization.
>>>
>>> Currently this is only in use by virtio-balloon however there is the hope
>>> that at some point in the future other hypervisors might be able to make
>>> use of it. In the virtio-balloon/QEMU implementation the hypervisor is
>>> currently using MADV_DONTNEED to indicate to the host kernel that the page
>>> is currently free. It will be zeroed and faulted back into the guest the
>>> next time the page is accessed.
>>>
>>> To track if a page is reported or not the Uptodate flag was repurposed and
>>> used as a Reported flag for Buddy pages. We walk though the free list
>>> isolating pages and adding them to the scatterlist until we either
>>> encounter the end of the list, processed as many pages as were listed in
>>> nr_free prior to us starting, or have filled the scatterlist with pages to
>>> be reported. If we fill the scatterlist before we reach the end of the
>>> list we rotate the list so that the first unreported page we encounter is
>>> moved to the head of the list as that is where we will resume after we
>>> have freed the reported pages back into the tail of the list.
>>>
>>> Below are the results from various benchmarks. I primarily focused on two
>>> tests. The first is the will-it-scale/page_fault2 test, and the other is
>>> a modified version of will-it-scale/page_fault1 that was enabled to use
>>> THP. I did this as it allows for better visibility into different parts
>>> of the memory subsystem. The guest is running with 32G for RAM on one
>>> node of a E5-2630 v3. The host has had some features such as CPU turbo
>>> disabled in the BIOS.
>>>
>>> Test                   page_fault1 (THP)    page_fault2
>>> Name            tasks  Process Iter  STDEV  Process Iter  STDEV
>>> Baseline            1    1012402.50  0.14%     361855.25  0.81%
>>>                      16    8827457.25  0.09%    3282347.00  0.34%
>>>
>>> Patches Applied     1    1007897.00  0.23%     361887.00  0.26%
>>>                      16    8784741.75  0.39%    3240669.25  0.48%
>>>
>>> Patches Enabled     1    1010227.50  0.39%     359749.25  0.56%
>>>                      16    8756219.00  0.24%    3226608.75  0.97%
>>>
>>> Patches Enabled     1    1050982.00  4.26%     357966.25  0.14%
>>>    page shuffle      16    8672601.25  0.49%    3223177.75  0.40%
>>>
>>> Patches enabled     1    1003238.00  0.22%     360211.00  0.22%
>>>    shuffle w/ RFC    16    8767010.50  0.32%    3199874.00  0.71%
>>>
>>> The results above are for a baseline with a linux-next-20191219 kernel,
>>> that kernel with this patch set applied but page reporting disabled in
>>> virtio-balloon, the patches applied and page reporting fully enabled, the
>>> patches enabled with page shuffling enabled, and the patches applied with
>>> page shuffling enabled and an RFC patch that makes used of MADV_FREE in
>>> QEMU. These results include the deviation seen between the average value
>>> reported here versus the high and/or low value. I observed that during the
>>> test memory usage for the first three tests never dropped whereas with the
>>> patches fully enabled the VM would drop to using only a few GB of the
>>> host's memory when switching from memhog to page fault tests.
>>>
>>> Any of the overhead visible with this patch set enabled seems due to page
>>> faults caused by accessing the reported pages and the host zeroing the page
>>> before giving it back to the guest. This overhead is much more visible when
>>> using THP than with standard 4K pages. In addition page shuffling seemed to
>>> increase the amount of faults generated due to an increase in memory churn.
>>> The overhead is reduced when using MADV_FREE as we can avoid the extra
>>> zeroing of the pages when they are reintroduced to the host, as can be seen
>>> when the RFC is applied with shuffling enabled.
>>>
>>> The overall guest size is kept fairly small to only a few GB while the test
>>> is running. If the host memory were oversubscribed this patch set should
>>> result in a performance improvement as swapping memory in the host can be
>>> avoided.
>>
>>
>> I really like the approach overall. Voluntarily propagating free memory
>> from a guest to the host has been a sore point ever since KVM was
>> around. This solution looks like a very elegant way to do so.
>>
>> The big piece I'm missing is the page cache. Linux will by default try
>> to keep the free list as small as it can in favor of page cache, so most
>> of the benefit of this patch set will be void in real world scenarios.
> 
> One approach is to move (parts of) the page cache from the guest to the
> hypervisor - e.g., using emulated NVDIMM or virtio-pmem.

Whether you can do that depends heavily on your virtualization 
environment. On a host with single tenant VMs, that's definitely 
feasible. In a Kubernetes environment, it might also be feasible.

But when you have VMs that assume that the host is interfering with them 
as little as possible, it becomes harder:

How do you ensure fairness across different VMs' page cache that is 
munged into a single big host one?

Do you even have host page cache or are you using SR-IOV / mdev for 
storage for performance reasons?


The puzzle is still incomplete, even with NVDIMM exposure to the guest 
as an option unfortunately :).


Alex



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

Powered by blists - more mailing lists