Message-ID: <20251205194351.1646318-1-kas@kernel.org>
Date: Fri,  5 Dec 2025 19:43:36 +0000
From: Kiryl Shutsemau <kas@...nel.org>
To: Andrew Morton <akpm@...ux-foundation.org>,
	Muchun Song <muchun.song@...ux.dev>
Cc: David Hildenbrand <david@...nel.org>,
	Oscar Salvador <osalvador@...e.de>,
	Mike Rapoport <rppt@...nel.org>,
	Vlastimil Babka <vbabka@...e.cz>,
	Lorenzo Stoakes <lorenzo.stoakes@...cle.com>,
	Matthew Wilcox <willy@...radead.org>,
	Zi Yan <ziy@...dia.com>,
	Baoquan He <bhe@...hat.com>,
	Michal Hocko <mhocko@...e.com>,
	Johannes Weiner <hannes@...xchg.org>,
	Jonathan Corbet <corbet@....net>,
	Usama Arif <usamaarif642@...il.com>,
	kernel-team@...a.com,
	linux-mm@...ck.org,
	linux-kernel@...r.kernel.org,
	linux-doc@...r.kernel.org,
	Kiryl Shutsemau <kas@...nel.org>
Subject: [PATCH 00/11] mm/hugetlb: Eliminate fake head pages from vmemmap optimization

This series removes "fake head pages" from the HugeTLB vmemmap
optimization (HVO) by changing how tail pages encode their relationship
to the head page.

It simplifies compound_head() and page_ref_add_unless(). Both are in the
hot path.

Background
==========

HVO reduces memory overhead by freeing vmemmap pages for HugeTLB pages
and remapping the freed virtual addresses to a single physical page.
Previously, all tail page vmemmap entries were remapped to the first
vmemmap page (containing the head struct page), creating "fake heads" -
tail pages that appear to have PG_head set when accessed through the
deduplicated vmemmap.

This required special handling in compound_head() to detect and work
around fake heads, adding complexity and overhead to a very hot path.

New Approach
============

For architectures/configs where sizeof(struct page) is a power of 2 (the
common case), this series changes how position of the head page is encoded
in the tail pages.

Instead of storing a pointer to the head page, the ->compound_info
(renamed from ->compound_head) now stores a mask.

The mask can be applied to any tail page's virtual address to compute
the head page address. Critically, all tail pages of the same order now
have identical compound_info values, regardless of which compound page
they belong to.

This enables a key optimization: instead of remapping tail vmemmap
entries to the head page (creating fake heads), we remap them to a
shared, pre-initialized vmemmap_tail page per hstate. The head page
gets its own dedicated vmemmap page, eliminating fake heads entirely.

Benefits
========

1. Smaller generated code. On defconfig, I see ~15K reduction of text
   in vmlinux:

   add/remove: 6/33 grow/shrink: 54/262 up/down: 6130/-21922 (-15792)

2. Simplified compound_head(): No fake head detection needed. The
   function is now branchless for power-of-2 struct page sizes.

3. Eliminated race condition: The old scheme required synchronize_rcu()
   to coordinate between HVO remapping and speculative PFN walkers that
   might write to fake heads. With the head page always in writable
   memory, this synchronization is unnecessary.

4. Removed static key: hugetlb_optimize_vmemmap_key is no longer needed
   since compound_head() no longer has HVO-specific branches.

5. Cleaner architecture: The vmemmap layout is now straightforward -
   head page has its own vmemmap, tails share a read-only template.

I had hoped to see performance improvement, but my testing thus far has
shown either no change or only a slight improvement within the noise.

Series Organization
===================

Patches 1-3: Preparatory refactoring
  - Change prep_compound_tail() interface to take order
  - Rename compound_head field to compound_info
  - Move set/clear_compound_head() near compound_head()

Patch 4: Core encoding change
  - Implement mask-based encoding for power-of-2 struct page

Patches 5-6: HVO restructuring
  - Refactor vmemmap_walk to support separate head/tail pages
  - Introduce per-hstate vmemmap_tail, eliminate fake heads

Patches 7-9: Cleanup
  - Remove fake head checks from compound_head(), PageTail(), etc.
  - Remove VMEMMAP_SYNCHRONIZE_RCU and synchronize_rcu() calls
  - Remove hugetlb_optimize_vmemmap_key static key

Patch 10: Optimization
  - Implement branchless compound_head() for power-of-2 case

Patch 11: Documentation
  - Update vmemmap_dedup.rst to reflect new architecture

Kiryl Shutsemau (11):
  mm: Change the interface of prep_compound_tail()
  mm: Rename the 'compound_head' field in the 'struct page' to
    'compound_info'
  mm: Move set/clear_compound_head() to compound_head()
  mm: Rework compound_head() for power-of-2 sizeof(struct page)
  mm/hugetlb: Refactor code around vmemmap_walk
  mm/hugetlb: Remove fake head pages
  mm: Drop fake head checks and fix a race condition
  hugetlb: Remove VMEMMAP_SYNCHRONIZE_RCU
  mm/hugetlb: Remove hugetlb_optimize_vmemmap_key static key
  mm: Remove the branch from compound_head()
  hugetlb: Update vmemmap_dedup.rst

 .../admin-guide/kdump/vmcoreinfo.rst          |   2 +-
 Documentation/mm/vmemmap_dedup.rst            |  62 ++---
 include/linux/hugetlb.h                       |   3 +
 include/linux/mm_types.h                      |  20 +-
 include/linux/page-flags.h                    | 163 +++++-------
 include/linux/page_ref.h                      |   8 +-
 include/linux/types.h                         |   2 +-
 kernel/vmcore_info.c                          |   2 +-
 mm/hugetlb.c                                  |   8 +-
 mm/hugetlb_vmemmap.c                          | 245 ++++++++----------
 mm/hugetlb_vmemmap.h                          |   4 +-
 mm/internal.h                                 |  11 +-
 mm/mm_init.c                                  |   2 +-
 mm/page_alloc.c                               |   4 +-
 mm/slab.h                                     |   2 +-
 mm/util.c                                     |  15 +-
 16 files changed, 242 insertions(+), 311 deletions(-)

-- 
2.51.2

Powered by blists - more mailing lists