linux-kernel - flexible mmap layout and vdso mapping

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:	Wed, 27 Jun 2007 16:24:40 -0500
From:	Daniel Forrest <forrest@...g.wisc.edu>
To:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: flexible mmap layout and vdso mapping

I am having a problem with memory allocation in a program when run
under Linux 2.6.18.

The program is statically linked and uses its own malloc library based
on brk().  Under previous versions of Linux it was able to allocate up
to ~3GB of memory before failing, now it fails at ~1GB.

The problem is that the vdso is mapped at 0x40000000 which limits the
heap to addresses less than 1GB.

I have tracked this down to the point that I see it is dependent on
the function arch_pick_mmap_layout(), in particular:

	if (!(2 & exec_shield) && (sysctl_legacy_va_layout ||
			(current->personality & ADDR_COMPAT_LAYOUT) ||
			current->signal->rlim[RLIMIT_STACK].rlim_cur == RLIM_INFINITY)) {
		mm->mmap_base = TASK_UNMAPPED_BASE;
		mm->get_unmapped_area = arch_get_unmapped_area;
		mm->unmap_area = arch_unmap_area;
	} else { ...

And the vdso is apparently mapped at "mm->mmap_base".

The program I have is a compiled binary and it is running through a
queueing system so modifying "/proc/sys/kernel/exec-shield" or
changing the stack limit are problematical.

Everything worked fine under Linux 2.6.9, except I notice in this case
that "cat /proc/sys/kernel/vdso" returns 0, whereas under 2.6.18 it
returns 1.

Are there any options on a per-program basis to disable vdso mapping?
Or any per-program options to change the location?  This queueing
system supports checkpointing and process migration, but that requires
that the binary not be changed so recompiling is not desirable.

-- 
Daniel K. Forrest	Laboratory for Molecular and
forrest@...g.wisc.edu	Computational Genomics
(608) 262 - 9479	University of Wisconsin, Madison
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/