Message-ID: <8E3EC5A4-4387-4839-926F-3655188C20F4@nvidia.com>
Date: Wed, 07 May 2025 11:57:08 -0400
From: Zi Yan <ziy@...dia.com>
To: Usama Arif <usamaarif642@...il.com>
Cc: Andrew Morton <akpm@...ux-foundation.org>, david@...hat.com,
 linux-mm@...ck.org, hannes@...xchg.org, shakeel.butt@...ux.dev,
 riel@...riel.com, baolin.wang@...ux.alibaba.com, lorenzo.stoakes@...cle.com,
 Liam.Howlett@...cle.com, npache@...hat.com, ryan.roberts@....com,
 linux-kernel@...r.kernel.org, kernel-team@...a.com,
 Yafang Shao <laoar.shao@...il.com>
Subject: Re: [PATCH 0/1] prctl: allow overriding system THP policy to always

On 7 May 2025, at 11:12, Usama Arif wrote:

> On 07/05/2025 15:57, Zi Yan wrote:
>> +Yafang, who is also looking at changing THP config at cgroup/container level.
>>
>> On 7 May 2025, at 10:00, Usama Arif wrote:
>>
>>> Allowing override of global THP policy per process allows workloads
>>> that have shown to benefit from hugepages to do so, without regressing
>>> workloads that wouldn't benefit. This will allow such types of
>>> workloads to be run/stacked on the same machine.
>>>
>>> It also helps in rolling out hugepages in hyperscaler configurations
>>> for workloads that benefit from them, where a single THP policy is
>>> likely to be used across the entire fleet, and prctl will help override it.
>>>
>>> An advantage of doing it via prctl vs creating a cgroup specific
>>> option (like /sys/fs/cgroup/test/memory.transparent_hugepage.enabled) is
>>> that this will work even when there are no cgroups present, and my
>>> understanding is there is a strong preference of cgroups controls being
>>> hierarchical which usually means them having a numerical value.
>>
>> Hi Usama,
>>
>> Do you mind giving an example on how to change THP policy for a set of
>> processes running in a container (under a cgroup)?
>
> Hi Zi,
>
> In our case, we create the processes in the cgroup via systemd. The way we will enable THP=always
> for processes in a cgroup is in the same way we enable KSM for the cgroup.
> The change in systemd would be very similar to the line in [1], where we would set prctl PR_SET_THP_ALWAYS
> in exec-invoke.
> This is at the start of the process, but you would already know at the start of the process
> whether you want THP=always for it or not.
>
> [1] https://github.com/systemd/systemd/blob/2e72d3efafa88c1cb4d9b28dd4ade7c6ab7be29a/src/core/exec-invoke.c#L5045

You also need to add a new systemd.directives, e.g., MemoryTHP, to
pass the THP enablement or disablement info from a systemd config file.
And if you find those processes do not benefit from using THPs,
you can just change the new "MemoryTHP" config and restart the processes.

Am I getting it? Thanks.

>>
>> Yafang mentioned that the prctl approach would require restarting all running
>> services[1] and other inflexiblities, so he proposed to use BPF to change THP
>> policy[2]. I wonder if Yafang's issues also apply to your case and if you
>> have a solution to them.
>>
>> Thanks.
>>
>> [1] https://lore.kernel.org/linux-mm/CALOAHbCXMi2GaZdHJaNLXxGsJf-hkDTrztsQiceaBcJ8d8p3cA@mail.gmail.com/
>> [2] https://lore.kernel.org/linux-mm/20250429024139.34365-1-laoar.shao@gmail.com/
>>>
>>>
>>> The output and code of test program is below:
>>>
>>> [root@vm4 vmuser]# echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
>>> [root@vm4 vmuser]# echo inherit > /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled
>>> [root@vm4 vmuser]# ./a.out
>>> Default THP setting:
>>> THP is not set to 'always'.
>>> PR_SET_THP_ALWAYS = 1
>>> THP is set to 'always'.
>>> PR_SET_THP_ALWAYS = 0
>>> THP is not set to 'always'.
>>>
>>>
>>> #include <stdio.h>
>>> #include <stdlib.h>
>>> #include <string.h>
>>> #include <unistd.h>
>>> #include <sys/mman.h>
>>> #include <sys/prctl.h>
>>>
>>> #define PR_SET_THP_ALWAYS 78
>>> #define SIZE 12 * (2 * 1024 * 1024) // 24 MB
>>>
>>> void check_smaps(void) {
>>>     FILE *file = fopen("/proc/self/smaps", "r");
>>>     if (!file) {
>>>         perror("fopen");
>>>         return;
>>>     }
>>>
>>>     char line[256];
>>>     int is_hugepage = 0;
>>>     while (fgets(line, sizeof(line), file)) {
>>>         // if (strstr(line, "AnonHugePages:"))
>>>         //     printf("%s\n", line);
>>>         if (strstr(line, "AnonHugePages:") && strstr(line, "24576 kB"))
>>> {
>>>             // printf("%s\n", line);
>>>             is_hugepage = 1;
>>>             break;
>>>         }
>>>     }
>>>     fclose(file);
>>>     if (is_hugepage) {
>>>         printf("THP is set to 'always'.\n");
>>>     } else {
>>>         printf("THP is not set to 'always'.\n");
>>>     }
>>> }
>>>
>>> void test_mmap_thp(void) {
>>>     char *buffer = (char *)mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
>>>                                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>>>     if (buffer == MAP_FAILED) {
>>>         perror("mmap");
>>>         return;
>>>     }
>>>     // Touch the memory to ensure it's allocated
>>>     memset(buffer, 0, SIZE);
>>>     check_smaps();
>>>     munmap(buffer, SIZE);
>>> }
>>>
>>> int main() {
>>>     printf("Default THP setting: \n");
>>>     test_mmap_thp();
>>>     printf("PR_SET_THP_ALWAYS = 1 \n");
>>>     prctl(PR_SET_THP_ALWAYS, 1, NULL, NULL, NULL);
>>>     test_mmap_thp();
>>>     printf("PR_SET_THP_ALWAYS = 0 \n");
>>>     prctl(PR_SET_THP_ALWAYS, 0, NULL, NULL, NULL);
>>>     test_mmap_thp();
>>>
>>>     return 0;
>>> }
>>>
>>>
>>> Usama Arif (1):
>>>   prctl: allow overriding system THP policy to always per process
>>>
>>>  include/linux/huge_mm.h                          |  3 ++-
>>>  include/linux/mm_types.h                         |  7 ++-----
>>>  include/uapi/linux/prctl.h                       |  3 +++
>>>  kernel/sys.c                                     | 16 ++++++++++++++++
>>>  tools/include/uapi/linux/prctl.h                 |  3 +++
>>>  .../perf/trace/beauty/include/uapi/linux/prctl.h |  3 +++
>>>  6 files changed, 29 insertions(+), 6 deletions(-)
>>>
>>> -- 
>>> 2.47.1
>>
>>
>> --
>> Best Regards,
>> Yan, Zi


--
Best Regards,
Yan, Zi

Powered by blists - more mailing lists