linux-kernel - [PATCH RESEND] x86/mm: lower MAP_32BIT begin to reduce heap collisions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <20251107100004.66240-1-simonlie@amazon.de>
Date: Fri, 7 Nov 2025 10:00:04 +0000
From: Simon Liebold <simonlie@...zon.de>
To: Thomas Gleixner <tglx@...utronix.de>, Ingo Molnar <mingo@...hat.com>,
	Borislav Petkov <bp@...en8.de>, Dave Hansen <dave.hansen@...ux.intel.com>,
	<x86@...nel.org>, "H . Peter Anvin" <hpa@...or.com>, Simon Liebold
	<lieboldsimonpaul@...il.com>, <linux-kernel@...r.kernel.org>
Subject: [PATCH RESEND] x86/mm: lower MAP_32BIT begin to reduce heap collisions

Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
randomization, a heap starting high enough and being big enough can use
up all the area that MAP_32BIT looks at, leading to allocation failures.

For example, if the heap starts at 800 MiB and is 1.2 GiB large,
allocations with MAP_32BIT will always fail despite unused addresses
below 800 MiB.

Lower the begin of the address space which is available to MAP32_BIT
from 0x40000000 to 0x10000000 to give mmap more room if the randomly
allocated brk address turns out to be unfavourable high. This allows
mmap to allocate up to 75% more space.

Signed-off-by: Simon Liebold <simonlie@...zon.de>
---

Notes:
    Re-sending after rebasing and re-testing on v6.18-rc4
    
    Background: LuaJIT v2.0 uses MAP32_BIT for allocating memory. Because of
    the restriction of MAP32_BIT to limit all allocation of mmap to the
    address space from 1 GiB to 2 GiB, LuaJIT v2.0 can fail to work,
    depending on the random location of brk.
    
    I tested this change using the following reproducer:
    
    int main() {
        uintptr_t mmap_end = 0x80000000;
        uintptr_t heap_start = (uintptr_t)sbrk(0);
        printf("heap start: %p\n", heap_start);
        uintptr_t alloc_size = mmap_end - heap_start;
        uintptr_t heap_end = (uintptr_t)sbrk(alloc_size);
        printf("heap allocated until: %p\n", heap_end);
        void* addr = mmap(NULL,
            8,
            PROT_READ | PROT_WRITE,
            MAP_PRIVATE | MAP_ANONYMOUS | MAP_32BIT,
            -1,
            0);
    
        if(addr == MAP_FAILED)
            printf("mmap allocation failed\n");
        else
            printf("mmap allocation at %p\n", addr);
    
        return 0;
    }
    
    Before the change, the allocation failed:
        [root@...alhost ~]# ./repro
        heap start: 0x24bce000
        heap allocated until: 0x24bef000
        mmap allocation failed
    
    After the change, it succeeded:
        [root@...alhost ~]# ./repro
        heap start: 0x38f24000
        heap allocated until: 0x38f45000
        mmap allocation at 0x11962000
    
    Note that this does not guarantee to succeed. If the randomized heap
    start is below 0x10000000, it still fails.
    
    This is an excerpt from the output of one of the failure cases on real
    workloads:
    
        00400000-00566000 r-xp 00000000 103:01 4476567               [...]/bin/httpd.orig
        00765000-0076b000 r--p 00165000 103:01 4476567               [...]/bin/httpd.orig
        0076b000-00772000 rw-p 0016b000 103:01 4476567               [...]/bin/httpd.orig
        00772000-00778000 rw-p 00000000 00:00 0
        34a21000-35021000 rw-p 00000000 00:00 0                      [heap]
        35021000-82ea7000 rw-p 00000000 00:00 0                      [heap]
        7fee7c0ba000-7fee7c11f000 r-xp 00000000 103:01 3836609       [...]/lib/libluajit-5.1.so
        7fee7c11f000-7fee7c31f000 ---p 00065000 103:01 3836609       [...]/lib/libluajit-5.1.so
        7fee7c31f000-7fee7c321000 r--p 00065000 103:01 3836609       [...]/lib/libluajit-5.1.so
        7fee7c321000-7fee7c322000 rw-p 00067000 103:01 3836609       [...]/lib/libluajit-5.1.so
        [Other maps at high addresses...]

 arch/x86/kernel/sys_x86_64.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c
index 776ae6fa7f2d6..29c6277aa31e6 100644
--- a/arch/x86/kernel/sys_x86_64.c
+++ b/arch/x86/kernel/sys_x86_64.c
@@ -100,7 +100,7 @@ static void find_start_end(unsigned long addr, unsigned long flags,
 		   conflicts with the heap, but we assume that glibc
 		   malloc knows how to fall back to mmap. Give it 1GB
 		   of playground for now. -AK */
-		*begin = 0x40000000;
+		*begin = 0x10000000;
 		*end = 0x80000000;
 		if (current->flags & PF_RANDOMIZE) {
 			*begin = randomize_page(*begin, 0x02000000);

base-commit: 4a0c9b3391999818e2c5b93719699b255be1f682
-- 
2.47.3




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Christof Hellmis
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597