linux-kernel - [PATCH] x86/mm: lower MAP_32BIT begin to reduce heap collisions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250825104847.36669-1-simonlie@amazon.de>
Date: Mon, 25 Aug 2025 10:48:44 +0000
From: Simon Liebold <simonlie@...zon.de>
To: Thomas Gleixner <tglx@...utronix.de>, Ingo Molnar <mingo@...hat.com>,
	Borislav Petkov <bp@...en8.de>, Dave Hansen <dave.hansen@...ux.intel.com>
CC: <x86@...nel.org>, "H. Peter Anvin" <hpa@...or.com>, Andrew Morton
	<akpm@...ux-foundation.org>, Mark Brown <broonie@...nel.org>, Lorenzo Stoakes
	<lorenzo.stoakes@...cle.com>, Helge Deller <deller@....de>, "Liam R. Howlett"
	<Liam.Howlett@...cle.com>, Simon Liebold <lieboldsimonpaul@...il.com>,
	<linux-kernel@...r.kernel.org>
Subject: [PATCH] x86/mm: lower MAP_32BIT begin to reduce heap collisions

Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
randomization, a heap starting high enough and being big enough can use
up all the area that MAP_32BIT looks at, leading to allocation failures.

For example, if the heap starts at 800 MiB and is 1.2 GiB large,
allocations with MAP_32BIT will always fail despite unused addresses
below 800 MiB.

Lower the begin of the address space which is available to MAP32_BIT
from 0x40000000 to 0x10000000 to give mmap more room if the randomly
allocated brk address turns out to be unfavourable high. This allows
mmap to allocate up to 75% more space.

Signed-off-by: Simon Liebold <simonlie@...zon.de>
---

Notes:
    Background: LuaJIT v2.0 uses MAP32_BIT for allocating memory. Because of
    the restriction of MAP32_BIT to limit all allocation of mmap to the
    address space from 1 GiB to 2 GiB, LuaJIT v2.0 can fail to work,
    depending on the random location of brk.
    
    I tested this change using the following reproducer:
    
    int main() {
        uintptr_t mmap_end = 0x80000000;
        uintptr_t heap_start = (uintptr_t)sbrk(0);
        printf("heap start: %p\n", heap_start);
        uintptr_t alloc_size = mmap_end - heap_start;
        uintptr_t heap_end = (uintptr_t)sbrk(alloc_size);
        printf("heap allocated until: %p\n", heap_end);
        void* addr = mmap(NULL,
            8,
            PROT_READ | PROT_WRITE,
            MAP_PRIVATE | MAP_ANONYMOUS | MAP_32BIT,
            -1,
            0);
    
        if(addr == MAP_FAILED)
            printf("mmap allocation failed\n");
        else
            printf("mmap allocation at %p\n", addr);
    
        return 0;
    }
    
    Before the change, the allocation failed:
        [root@...alhost ~]# ./repro
        heap start: 0x24bce000
        heap allocated until: 0x24bef000
        mmap allocation failed
    
    After the change, it succeeded:
        [root@...alhost ~]# ./repro
        heap start: 0x38f24000
        heap allocated until: 0x38f45000
        mmap allocation at 0x11962000
    
    Note that this does not guarantee to succeed. If the randomized heap
    start is below 0x10000000, it still fails.

 arch/x86/kernel/sys_x86_64.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c
index 776ae6fa7f2d..29c6277aa31e 100644
--- a/arch/x86/kernel/sys_x86_64.c
+++ b/arch/x86/kernel/sys_x86_64.c
@@ -100,7 +100,7 @@ static void find_start_end(unsigned long addr, unsigned long flags,
 		   conflicts with the heap, but we assume that glibc
 		   malloc knows how to fall back to mmap. Give it 1GB
 		   of playground for now. -AK */
-		*begin = 0x40000000;
+		*begin = 0x10000000;
 		*end = 0x80000000;
 		if (current->flags & PF_RANDOMIZE) {
 			*begin = randomize_page(*begin, 0x02000000);

base-commit: 260f6f4fda93c8485c8037865c941b42b9cba5d2
-- 
2.47.3




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597