linux-kernel - Re: [PATCH v3 2/2] mm: shmem: disallow hugepages if the system-wide shmem THP sysfs settings are disabled

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <924f82b1-0eb7-47e0-b7d4-5dd314df18a0@redhat.com>
Date: Mon, 23 Jun 2025 15:59:07 +0200
From: David Hildenbrand <david@...hat.com>
To: Baolin Wang <baolin.wang@...ux.alibaba.com>, akpm@...ux-foundation.org,
 hughd@...gle.com
Cc: ziy@...dia.com, lorenzo.stoakes@...cle.com, Liam.Howlett@...cle.com,
 npache@...hat.com, ryan.roberts@....com, dev.jain@....com,
 baohua@...nel.org, linux-mm@...ck.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v3 2/2] mm: shmem: disallow hugepages if the system-wide
 shmem THP sysfs settings are disabled

On 23.06.25 10:28, Baolin Wang wrote:
> When invoking thp_vma_allowable_orders(), if the TVA_ENFORCE_SYSFS flag is not
> specified, we will ignore the THP sysfs settings. And the MADV_COLLAPSE is an
> example of such a case.
> 
> The MADV_COLLAPSE will ignore the system-wide shmem THP sysfs settings, which
> means that even though we have disabled the shmem THP configuration, MADV_COLLAPSE
> will still attempt to collapse into a shmem THP. This violates the rule we have
> agreed upon: never means never.
> 
> Another rule for madvise, referring to David's suggestion: “allowing for collapsing
> in a VM without VM_HUGEPAGE in the "madvise" mode would be fine".
> 
> To fix the MADV_COLLAPSE issue for shmem, then the current strategy should be:
> 
> For shmem, if none of always, madvise, within_size, and inherit have enabled
> PMD-sized THP, then MADV_COLLAPSE will be prohibited from collapsing PMD-sized THP.

I assume we could rephrase that to "For shmem, if "shmem_enabled" is set 
to either "none" or "deny", then MADV_COLLAPSE will be prohibited from 
collapsing."

Or am I missing a case?

[...]

> @@ -672,7 +672,7 @@ static unsigned int shmem_huge_global_enabled(struct inode *inode, pgoff_t index
>   
>   		fallthrough;
>   	case SHMEM_HUGE_ADVISE:
> -		if (vm_flags & VM_HUGEPAGE)
> +		if (shmem_huge_force || (vm_flags & VM_HUGEPAGE))
>   			return maybe_pmd_order;
>   		fallthrough;
>   	default:
> @@ -1806,7 +1806,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
>   	/* Allow mTHP that will be fully within i_size. */
>   	mask |= shmem_get_orders_within_size(inode, within_size_orders, index, 0);
>   
> -	if (vm_flags & VM_HUGEPAGE)
> +	if (shmem_huge_force || (vm_flags & VM_HUGEPAGE))
>   		mask |= READ_ONCE(huge_shmem_orders_madvise);
>   
>   	if (global_orders > 0)
> diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c
> index 85bfff53dba6..9517ed99c382 100644
> --- a/tools/testing/selftests/mm/khugepaged.c
> +++ b/tools/testing/selftests/mm/khugepaged.c
> @@ -502,7 +502,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,
>   	printf("%s...", msg);
>   
>   	settings.thp_enabled = THP_ALWAYS;
> -	settings.shmem_enabled = SHMEM_NEVER;
> +	settings.shmem_enabled = SHMEM_ALWAYS;
>   	thp_push_settings(&settings);

Same question as for the other case.

-- 
Cheers,

David / dhildenb