lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20070627162440.A12430@yoda.lmcg.wisc.edu>
Date:	Wed, 27 Jun 2007 16:24:40 -0500
From:	Daniel Forrest <forrest@...g.wisc.edu>
To:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: flexible mmap layout and vdso mapping

I am having a problem with memory allocation in a program when run
under Linux 2.6.18.

The program is statically linked and uses its own malloc library based
on brk().  Under previous versions of Linux it was able to allocate up
to ~3GB of memory before failing, now it fails at ~1GB.

The problem is that the vdso is mapped at 0x40000000 which limits the
heap to addresses less than 1GB.

I have tracked this down to the point that I see it is dependent on
the function arch_pick_mmap_layout(), in particular:

	if (!(2 & exec_shield) && (sysctl_legacy_va_layout ||
			(current->personality & ADDR_COMPAT_LAYOUT) ||
			current->signal->rlim[RLIMIT_STACK].rlim_cur == RLIM_INFINITY)) {
		mm->mmap_base = TASK_UNMAPPED_BASE;
		mm->get_unmapped_area = arch_get_unmapped_area;
		mm->unmap_area = arch_unmap_area;
	} else { ...


And the vdso is apparently mapped at "mm->mmap_base".


The program I have is a compiled binary and it is running through a
queueing system so modifying "/proc/sys/kernel/exec-shield" or
changing the stack limit are problematical.

Everything worked fine under Linux 2.6.9, except I notice in this case
that "cat /proc/sys/kernel/vdso" returns 0, whereas under 2.6.18 it
returns 1.


Are there any options on a per-program basis to disable vdso mapping?
Or any per-program options to change the location?  This queueing
system supports checkpointing and process migration, but that requires
that the binary not be changed so recompiling is not desirable.

-- 
Daniel K. Forrest	Laboratory for Molecular and
forrest@...g.wisc.edu	Computational Genomics
(608) 262 - 9479	University of Wisconsin, Madison
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ