linux-kernel - [PATCH] [RFC]hugetlbfs: Get unmapped area below TASK_UNMAPPED

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 27 Apr 2020 19:10:36 +0800
From:   Shijie Hu <hushijie3@...wei.com>
To:     <mike.kravetz@...cle.com>
CC:     <linux-mm@...ck.org>, <linux-kernel@...r.kernel.org>,
        <nixiaoming@...wei.com>, <wangxu72@...wei.com>,
        <wangkefeng.wang@...wei.com>, <yangerkun@...wei.com>,
        <wangle6@...wei.com>, <cg.chen@...wei.com>
Subject: [PATCH] [RFC]hugetlbfs: Get unmapped area below TASK_UNMAPPED_BASE for hugetlbfs

In 32-bit programs, the address space is limited. When the normal mmap
consumes the space above TASK_UNMAPPED_BASE on legacy mode, it can still
successfully obtain unmapped area below TASK_UNMAPPED_BASE, but mmap or
shmat for huge pages will fail. This seems "not fair".

When the request for huge pages fails, fall back to reuse mmap_min_addr
~ TASK_UNMAPPED_BASE for hugetlbfs.

Signed-off-by: Shijie Hu <hushijie3@...wei.com>
---
 fs/hugetlbfs/inode.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index aff8642f0c2e..0f5997394aaa 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -224,7 +224,21 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
 	info.high_limit = TASK_SIZE;
 	info.align_mask = PAGE_MASK & ~huge_page_mask(h);
 	info.align_offset = 0;
-	return vm_unmapped_area(&info);
+	addr = vm_unmapped_area(&info);
+
+	/*
+	 * A failed request for huge pages very likely causes application
+	 * failure, so fall back to the top-down function here.
+	 */
+	if (unlikely(offset_in_page(addr))) {
+		VM_BUG_ON(addr != -ENOMEM);
+		info.flags = VM_UNMAPPED_AREA_TOPDOWN;
+		info.low_limit = max(PAGE_SIZE, mmap_min_addr);
+		info.high_limit = TASK_UNMAPPED_BASE;
+		addr = vm_unmapped_area(&info);
+	}
+
+	return addr;
 }
 #endif
 
-- 
2.12.3