| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 14 Jul 2021 17:17:55 +0800
From: Muchun Song <songmuchun@...edance.com>
To: mike.kravetz@...cle.com, akpm@...ux-foundation.org,
osalvador@...e.de, mhocko@...e.com, song.bao.hua@...ilicon.com,
david@...hat.com, chenhuang5@...wei.com, bodeddub@...zon.com,
corbet@....net
Cc: duanxiongchun@...edance.com, fam.zheng@...edance.com,
linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
linux-mm@...ck.org, zhengqi.arch@...edance.com,
Muchun Song <songmuchun@...edance.com>
Subject: [PATCH 0/5] Free the 2nd vmemmap page associated with each HugeTLB page
After the feature of "Free sonme vmemmap pages of HugeTLB page" is enabled,
the mapping of the vmemmap addresses associated with a 2MB HugeTLB page
becomes the figure below.
HugeTLB struct pages(8 pages) page frame(8 pages)
+-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+---> PG_head
| | | 0 | -------------> | 0 |
| | +-----------+ +-----------+
| | | 1 | -------------> | 1 |
| | +-----------+ +-----------+
| | | 2 | ----------------^ ^ ^ ^ ^ ^
| | +-----------+ | | | | |
| | | 3 | ------------------+ | | | |
| | +-----------+ | | | |
| | | 4 | --------------------+ | | |
| 2MB | +-----------+ | | |
| | | 5 | ----------------------+ | |
| | +-----------+ | |
| | | 6 | ------------------------+ |
| | +-----------+ |
| | | 7 | --------------------------+
| | +-----------+
| |
| |
| |
+-----------+
As we can see, the 2nd vmemmap page frame (indexed by 1) is reused and remaped.
However, the 2nd vmemmap page frame is also can be freed to the buddy allocator,
then we can change the mapping from the figure above to the figure below.
HugeTLB struct pages(8 pages) page frame(8 pages)
+-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+---> PG_head
| | | 0 | -------------> | 0 |
| | +-----------+ +-----------+
| | | 1 | ---------------^ ^ ^ ^ ^ ^ ^
| | +-----------+ | | | | | |
| | | 2 | -----------------+ | | | | |
| | +-----------+ | | | | |
| | | 3 | -------------------+ | | | |
| | +-----------+ | | | |
| | | 4 | ---------------------+ | | |
| 2MB | +-----------+ | | |
| | | 5 | -----------------------+ | |
| | +-----------+ | |
| | | 6 | -------------------------+ |
| | +-----------+ |
| | | 7 | ---------------------------+
| | +-----------+
| |
| |
| |
+-----------+
After we do this, all tail vmemmap pages (1-7) are mapped to the head vmemmap
page frame (0). In other words, there are more than one page struct with
PG_head associated with each HugeTLB page. We __know__ that there is only one
head page struct, the tail page structs with PG_head are fake head page structs.
We need an approach to distinguish between those two different types of page
structs so that compound_head(), PageHead() and PageTail() can work properly
if the parameter is the tail page struct but with PG_head.
The following code snippet describes how to distinguish between real and fake
head page struct.
if (test_bit(PG_head, &page->flags)) {
unsigned long head = READ_ONCE(page[1].compound_head);
if (head & 1) {
if (head == (unsigned long)page + 1)
==> head page struct
else
==> tail page struct
} else
==> head page struct
}
We can safely access the field of the @page[1] with PG_head because the @page
is a compound page composed with at least two contiguous pages. The main
implementation is in the patch 3.
In our server, we can save extra 2GB memory with this patchset applied if there
are 1 TB HugeTLB (2 MB) pages. If the size of the HugeTLB page is 1 GB, it only
can save 4MB. For 2 MB HugeTLB page, it is a nice gain.
Muchun Song (5):
mm: introduce PAGEFLAGS_MASK to replace ((1UL << NR_PAGEFLAGS) - 1)
mm: introduce save_page_flags to cooperate with show_page_flags
mm: hugetlb: free the 2nd vmemmap page associated with each HugeTLB
page
mm: hugetlb: replace hugetlb_free_vmemmap_enabled with a static_key
mm: sparsemem: use page table lock to protect kernel pmd operations
Documentation/admin-guide/kernel-parameters.txt | 2 +-
include/linux/hugetlb.h | 6 +-
include/linux/page-flags.h | 103 ++++++++++++++++++++++--
include/trace/events/mmflags.h | 4 +
include/trace/events/page_ref.h | 8 +-
lib/test_printf.c | 2 +-
lib/vsprintf.c | 2 +-
mm/hugetlb_vmemmap.c | 67 ++++++++-------
mm/ptdump.c | 16 +++-
mm/sparse-vmemmap.c | 70 ++++++++++++----
10 files changed, 218 insertions(+), 62 deletions(-)
--
2.11.0
Powered by blists - more mailing lists