Message-ID: <401175cf-e205-4e5c-939f-60b7ab948355@arm.com>
Date: Wed, 16 Oct 2024 15:31:28 +0100
From: Ryan Roberts <ryan.roberts@....com>
To: Zi Yan <ziy@...dia.com>, David Hildenbrand <david@...hat.com>
Cc: Andrew Morton <akpm@...ux-foundation.org>,
 Anshuman Khandual <anshuman.khandual@....com>,
 Ard Biesheuvel <ardb@...nel.org>, Catalin Marinas <catalin.marinas@....com>,
 Greg Marsden <greg.marsden@...cle.com>, Ivan Ivanov <ivan.ivanov@...e.com>,
 Kalesh Singh <kaleshsingh@...gle.com>, Marc Zyngier <maz@...nel.org>,
 Mark Rutland <mark.rutland@....com>, Matthias Brugger <mbrugger@...e.com>,
 Miroslav Benes <mbenes@...e.cz>, Oliver Upton <oliver.upton@...ux.dev>,
 Will Deacon <will@...nel.org>, kvmarm@...ts.linux.dev,
 linux-arm-kernel@...ts.infradead.org, linux-efi@...r.kernel.org,
 linux-kernel@...r.kernel.org, linux-mm@...ck.org
Subject: Re: [RFC PATCH v1 57/57] arm64: Enable boot-time page size selection

On 16/10/2024 15:21, Zi Yan wrote:
> On 16 Oct 2024, at 4:14, Ryan Roberts wrote:
> 
>> On 15/10/2024 18:42, Zi Yan wrote:
>>> On 14 Oct 2024, at 6:59, Ryan Roberts wrote:
>>>
>>>> Introduce a new Kconfig, ARM64_BOOT_TIME_PAGE_SIZE, which can be
>>>> selected instead of a page size. When selected, the resulting kernel's
>>>> page size can be configured at boot via the command line.
>>>>
>>>> For now, boot-time page size kernels are limited to 48-bit VA, since
>>>> more work is required to support LPA2. Additionally MMAP_RND_BITS and
>>>> SECTION_SIZE_BITS are configured for the worst case (64K pages). Future
>>>> work could be implemented to be able to configure these at boot time for
>>>> optimial page size-specific values.
>>>>
>>>> Signed-off-by: Ryan Roberts <ryan.roberts@....com>
>>>> ---
>>>
>>> <snip>
>>>
>>>>
>>>> @@ -1588,9 +1601,10 @@ config XEN
>>>>  # 4K  |       27          |      12      |       15             |         10              |
>>>>  # 16K |       27          |      14      |       13             |         11              |
>>>>  # 64K |       29          |      16      |       13             |         13              |
>>>> +# BOOT|       29          |    16 (max)  |       13             |         13              |
>>>>  config ARCH_FORCE_MAX_ORDER
>>>>  	int
>>>> -	default "13" if ARM64_64K_PAGES
>>>> +	default "13" if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE
>>>>  	default "11" if ARM64_16K_PAGES
>>>>  	default "10"
>>>>  	help
>>>
>>> So boot-time page size kernel always has the highest MAX_PAGE_ORDER, which
>>> means the section size increases for 4KB and 16KB page sizes. Any downside
>>> for this?
>>
>> I guess there is some cost to the buddy when MAX_PAGE_ORDER is larger than it
>> needs to be - I expect you can explain those details much better than I can. I'm
>> just setting it to the worst case for now as it was the easiest solution for the
>> initial series.
> 
> From my past experience (around 5.19), the perf impact (using vm-scalability)
> seems very small due to MAX_PAGE_ORDER increases [1] (I made MAX_PAGE_ORDER
> a boot time variable and increased it to 20 for my 1GB THP experiments).
> 
> Larger MAX_PAGE_ORDER means larger section size and larger mem_block size,
> so the granularity of memory hotplug also increases. In this case:
> 1. ARM64 4KB: mem_block size increases from 4MB to 32MB,
> 2. ARM64 16KB: mem_block size increases from 32MB to 128MB,
> 3. ARM64 64KB: mem_block size keeps the same, 512MB.
> 
> DavidH was concerned about large mem_block size before. He might have some
> opinion on this.
> 
> 
>>
>>>
>>> Is there any plan (not in this patchset) to support boot-time MAX_PAGE_ORDER
>>> to keep section size the same?
>>
>> Yes absolutely. I should have documented MAX_PAGE_ORDER in the commit log along
>> with the comments for MMAP_RND_BITS and SECTION_SIZE_BITS - that was an
>> oversight and I'll fix it in the next version. I plan to look at making all 3
>> values boot-time configurable in future (although I have no idea at this point
>> how involved that will be).
> 
> In [1], I tried to make MAX_PAGE_ORDER a boot time variable,
> but for a different purpose, allocating 1GB THP. I needed some additional
> changes in my patchset, since I assumed MAX_PAGE_ORDER can go beyond
> section size, which makes things a little bit complicated. For your case,
> I assume you are not planning to make MAX_PAGE_ORDER bigger than section
> size, then I should be able to revive my patchset with fewer changes.

Yes correct; no need to make it bigger than section size. Thanks for the patch,
I'll certainly use it as a base when I get there or if you're interested in
doing it then even better ;-)

But I don't think this is urgent. For now, boot-time page size is a new Kconfig
for arm64. It still supports the compile-time page size options. So having a
larger MAX_PAGE_ORDER than strictly necessary doesn't represent a regression,
just a limitation of boot-time page size config - something we can optimize later.

> 
> In terms of SECTION_SIZE_BITS, why do you want to make it a boot time variable?
> Since it decides the minimum memory hotplug size, I assume we should keep
> it unchanged or as small as possible to make virtual machine memory usage
> efficient.

When I say "boot-time variable" I just mean something that the arch can
configure at boot based on the selected page size. I'm not proposing to allow
the user to set it via the command line. That means we need to rid the code of
any assumptions that it is compile time constant (e.g. c preprocessor usage of
the value, etc). The same goes for MAX_PAGE_ORDER and the MMAP_RND_BITS stuff.

> 
> 
> [1] https://lore.kernel.org/linux-mm/20220811231643.1012912-1-zi.yan@sent.com/
> 
> 
> Best Regards,
> Yan, Zi

Powered by blists - more mailing lists