Message-ID: <3A6D54CF-76F4-4401-A434-84BEB813A65A@nvidia.com>
Date:   Thu, 06 May 2021 14:49:09 -0400
From:   "Zi Yan" <ziy@...dia.com>
To:     "David Hildenbrand" <david@...hat.com>
Cc:     "Oscar Salvador" <osalvador@...e.de>,
        "Michael Ellerman" <mpe@...erman.id.au>,
        "Benjamin Herrenschmidt" <benh@...nel.crashing.org>,
        "Thomas Gleixner" <tglx@...utronix.de>, x86@...nel.org,
        "Andy Lutomirski" <luto@...nel.org>,
        "Rafael J . Wysocki" <rafael@...nel.org>,
        "Andrew Morton" <akpm@...ux-foundation.org>,
        "Mike Rapoport" <rppt@...nel.org>,
        "Anshuman Khandual" <anshuman.khandual@....com>,
        "Michal Hocko" <mhocko@...e.com>,
        "Dan Williams" <dan.j.williams@...el.com>,
        "Wei Yang" <richard.weiyang@...ux.alibaba.com>,
        linux-ia64@...r.kernel.org, linux-kernel@...r.kernel.org,
        linuxppc-dev@...ts.ozlabs.org, linux-mm@...ck.org
Subject: Re: [RFC PATCH 0/7] Memory hotplug/hotremove at subsection size

On 6 May 2021, at 12:28, David Hildenbrand wrote:

> On 06.05.21 17:50, Zi Yan wrote:
>> On 6 May 2021, at 11:40, David Hildenbrand wrote:
>>
>>>>>> The last patch increases SECTION_SIZE_BITS to demonstrate the use of memory
>>>>>> hotplug/hotremove subsection, but is not intended to be merged as is. It is
>>>>>> there in case one wants to try this out and will be removed during the final
>>>>>> submission.
>>>>>>
>>>>>> Feel free to give suggestions and comments. I am looking forward to your
>>>>>> feedback.
>>>>>
>>>>> Please not like this.
>>>>
>>>> Do you mind sharing more useful feedback instead of just saying a lot of No?
>>>
>>> I remember reasoning about this already in another thread, no? Either you're ignoring my previous feedback or my mind is messing with me.
>>
>> I definitely remember all your suggestions:
>>
>> 1. do not use CMA allocation for 1GB THP.
>> 2. section size defines the minimum size in which we can add_memory(), so we cannot increase it.
>>
>> I am trying an alternative here. I am not using CMA allocation and not increasing the minimum size of add_memory() by decoupling the memory block size from section size, so that add_memory() can add a memory block smaller (as small as 2MB, the subsection size) than section size. In this way, section size can be increased freely. I do not see the strong tie between add_memory() and section size, especially we have subsection bitmap support.
>
> Okay, let me express my thoughts, I could have sworn I explained back then why I am not a friend of messing with the existing pageblock size:

Thanks for writing down your thoughts in detail. I will clarify my high-level plan below too.

>
> 1. Pageblock size
>
> There are a couple of features that rely on the pageblock size to be reasonably small to work as expected. One example is virtio-balloon free page reporting, then there is virtio-mem (still also glued MAX_ORDER) and we have CMA (still also glued to MAX_ORDER). Most probably there are more. We track movability/ page isolation per pageblock; it's the smallest granularity you can effectively isolate pages or mark them as CMA (MIGRATE_ISOLATE, MIGRATE_CMA). Well, and there are "ordinary" THP / huge pages most of our applications use and will use, especially on smallish systems.
>
> Assume you bump up the pageblock order to 1G. Small VMs won't be able to report any free pages to the hypervisor. You'll take the "fine-grained" out of virtio-mem. Each CMA area will have to be at least 1G big, which turns CMA essentially useless on smallish systems (like we have on arm64 with 64k base pages -- pageblock_size is 512MB and I hate it).

I understand the issue of having large pageblock in small systems. My plan for this issue is to make MAX_ORDER a variable (pageblock size would be set according to MAX_ORDER) that can be adjusted based on total memory and via boot time parameter. My apology since I did not state this clearly in my cover letter and it confused you. When we have a boot time adjustable MAX_ORDER, large pageblock like 1GB would only appear for systems with large memory. For small VMs, pageblock size would stay at 2MB, so all your concerns on smallish systems should go away.

>
> Then, imagine systems that have like 4G of main memory. By stopping grouping at 2M and instead grouping at 1G you can very easily find yourself in the system where all your 4 pageblocks are unmovable and you essentially don't optimize for huge pages in that environment any more.
>
> Long story short: we need a different mechanism on top and shall leave the pageblock size untouched, it's too tightly integrated with page isolation, ordinary THP, and CMA.

I think it is better to make pageblock size adjustable based on total memory of a system. It is not reasonable to have the same pageblock size across systems with memory sizes from <1GB to several TBs. Do you agree?

>
> 2. Section size
>
> I assume the only reason you want to touch that is because pageblock_size <= section_size, and I guess that's one of the reasons I dislike it so much. Messing with the section size really only makes sense when we want to manage metadata for larger granularity within a section.

Perhaps it is worth checking if it is feasible to make pageblock_size > section_size, so we can still have small sections when pageblock_size are large. One potential issue for that is when PFN discontinues at section boundary, we might have partial pageblock when pageblock_size is big. I guess supporting partial pageblock (or different pageblock sizes like you mentioned below ) would be the right solution.

>
> We allocate metadata per section. We mark whole sections early/online/present/.... Yes, in case of vmemmap, we manage the memmap in smaller granularity using the sub-section map, some kind of hack to support some ZONE_DEVICE cases better.
>
> Let's assume we introduce something new "gigapage_order", corresponding to 1G. We could either decide to squeeze the metadata into sections, having to increase the section size, or manage that metadata differently.
>
> Managing it differently certainly makes the necessary changes easier. Instead of adding more hacks into sections, rather manage that metadata at differently place / in a different way.

Can you elaborate on managing it differently?

>
> See [1] for an alternative. Not necessarily what I would dream off, but just to showcase that there might be alternative to group pages.

I saw this patch too. It is an interesting idea to separate different allocation orders into different regions, but it would not work for gigantic page allocations unless we have large pageblock size to utilize existing anti-fragmentation mechanisms.


>
> 3. Grouping pages > pageblock_order
>
> There are other approaches that would benefit from grouping at > pageblock_order and having bigger MAX_ORDER. And that doesn't necessarily mean to form gigantic pages only, we might want to group in multiple granularity on a single system. Memory hot(un)plug is one example, but also optimizing memory consumption by powering down DIMM banks. Also, some architectures support differing huge page sizes (aarch64) that could be improved without CMA. Why not have more than 2 THP sizes on these systems?
>
> Ideally, we'd have a mechanism that tries grouping on different granularity, like for every order in pageblock_order ... max_pageblock_order (e.g., 1 GiB), and not only add one new level of grouping (or increase the single grouping size).

I agree. In some sense, supporting partial pageblock and increasing pageblock size (e.g., to 1GB) is, at the high level, quite similar to having multiple pageblock sizes. But I am not sure if we really want to support multiple pageblock sizes, since it creates pageblock fragmentation when we want to change migratetype for part of a pageblock. This means we would break a large pageblock into small ones if we just want to steal a subset of pages from MOVEABLE for UNMOVABLE allocations. Then pageblock loses its most useful anti-fragmentation feature. Also it seems to be a replication of buddy allocator functionalities when it comes to pageblock split and merge.


The above is really a nice discussion with you on pageblock, section, memory hotplug/hotremove, which also helps me understand more on the issues with increasing MAX_ORDER to enable 1GB page allocation.

In sum, if I get it correctly, the issues I need to address are:

1. large pageblock size (which is needed when we bump MAX_ORDER for gigantic page allocation from buddy allocator) is not good for machines with small memory.

2. pageblock size is currently tied with section size (which made me want to bump section size).


For 1, I think making MAX_ORDER a variable that can be set based on total memory size and adjustable via boot time parameter should solve the problem. For small machines, we will keep MAX_ORDER as small as we have now like 4MB, whereas for large machines, we can increase MAX_ORDER to utilize gigantic pages.

For 2, supporting partial pageblock and allow a pageblock to cross multiple sections would break the tie between pageblock size and section to solve the issue.

I am going to look into them. What do you think?

—
Best Regards,
Yan Zi

Download attachment "signature.asc" of type "application/pgp-signature" (855 bytes)

Powered by blists - more mailing lists