| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20070531133917.a5338876.randy.dunlap@oracle.com>
Date: Thu, 31 May 2007 13:39:17 -0700
From: Randy Dunlap <randy.dunlap@...cle.com>
To: Herbert van den Bergh <Herbert.van.den.Bergh@...cle.com>
Cc: linux-kernel@...r.kernel.org
Subject: Re: [PATCH] mm: include private data mappings in RLIMIT_DATA limit
On Thu, 31 May 2007 12:43:16 -0700 Herbert van den Bergh wrote:
> Below is a patch I would like to submit for limiting process private
> memory allocations via setrlimit(RLIMIT_DATA).
>
> Thanks,
> Herbert van den Bergh.
>
>
> This patch changes how the kernel limits memory allocations that are
> controlled by setrlimit() using the RLIMIT_DATA resource. Currently the
> kernel can limit the size of a process data, bss, and heap, but does not
> limit the amount of process private memory that can be allocated with
> the mmap() call. Since malloc() calls mmap() to allocate process memory
> once brk() calls can no longer grow the heap, process private memory
> allocations cannot be limited with RLIMIT_DATA.
>
> With this patch, not only memory in the data segment of a process, but
> also private data mappings, both file-based and anonymous, are counted
> toward the RLIMIT_DATA resource limit. Executable mappings, such as
> text segments of shared objects, are not counted toward the private data
> limit. The result is that malloc() will fail once the combined size of
> the data segment and private data mappings reaches this limit.
>
> This brings the Linux behavior in line with what is documented in the
> POSIX man page for setrlimit(3p).
>
> Signed-off-by: Herbert van den Bergh (herbert.van.den.bergh@...cle.com)
> Signed-off-by: Dave McCracken (dave.mccracken@...cle.com)
>
Howdy,
The patch seems to have all tabs converted to spaces.
This could be caused by thunderbird or by copy-and-pasting.
Tbird needs help for sending patches without munging them.
See http://mbligh.org/linuxdocs/Email/Clients/Thunderbird
or google for "external editor" and use that with a decent
text editor.
Maybe it doesn't matter for this patch, but in general you
should send patches made against the latest mainline kernel,
such as 2.6.22-rc3 or 2.6.22-rc3-git4 or current git tree.
There are a few lines that are > 80 characters. Please try
to avoid that.
----------------------------------------------------------------------------------
> --- linux-2.6.21/fs/proc/task_mmu.c.orig 2007-05-29 21:05:31.000000000 -0700
> +++ linux-2.6.21/fs/proc/task_mmu.c 2007-05-30 08:17:44.000000000 -0700
> @@ -31,7 +31,7 @@
> if (hiwater_rss < mm->hiwater_rss)
> hiwater_rss = mm->hiwater_rss;
>
> - data = mm->total_vm - mm->shared_vm - mm->stack_vm;
> + data = mm->total_vm - mm->shared_vm - mm->stack_vm - mm->exec_vm;
> text = (PAGE_ALIGN(mm->end_code) - (mm->start_code & PAGE_MASK)) >> 10;
> lib = (mm->exec_vm << (PAGE_SHIFT-10)) - text;
> buffer += sprintf(buffer,
> --- linux-2.6.21/mm/mprotect.c.orig 2007-04-25 20:08:32.000000000 -0700
> +++ linux-2.6.21/mm/mprotect.c 2007-05-30 08:34:36.000000000 -0700
> @@ -162,6 +162,21 @@
> }
> }
>
> + /*
> + * Check against private data size limit. When execute permission
> + * is removed, the mapping is counted toward the private data size.
> + */
> + if (!(newflags & VM_EXEC) && (vma->vm_flags & VM_EXEC)) {
> + unsigned long rlim = current->signal->rlim[RLIMIT_DATA].rlim_cur;
> + if (rlim < RLIM_INFINITY) {
> + unsigned long priv_vm;
> + priv_vm = mm->total_vm - mm->shared_vm - mm->exec_vm;
> + priv_vm <<= PAGE_SHIFT;
> + if (priv_vm + (end - start) > rlim)
> + return -ENOMEM;
> + }
> + }
> +
> /*
> * First try to merge with previous and/or next vma.
> */
> --- linux-2.6.21/mm/mremap.c.orig 2007-04-25 20:08:32.000000000 -0700
> +++ linux-2.6.21/mm/mremap.c 2007-05-30 08:38:28.000000000 -0700
> @@ -343,6 +343,22 @@
> goto out;
> }
>
> + /*
> + * Check against private data size limit. Count memory allocations
> + * which are not shared and not executable code as private data.
> + */
> + if (!(vma->vm_flags & (VM_EXEC|VM_SHARED))) {
> + unsigned long rlim = current->signal->rlim[RLIMIT_DATA].rlim_cur;
> + if (rlim < RLIM_INFINITY) {
> + unsigned long priv_vm;
> + struct mm_struct *mm = current->mm;
> + priv_vm = mm->total_vm - mm->shared_vm - mm->exec_vm;
> + priv_vm <<= PAGE_SHIFT;
> + if (priv_vm + (new_len - old_len) > rlim)
> + goto out;
> + }
> + }
> +
> if (vma->vm_flags & VM_ACCOUNT) {
> charged = (new_len - old_len) >> PAGE_SHIFT;
> if (security_vm_enough_memory(charged))
> --- linux-2.6.21/mm/mmap.c.orig 2007-05-29 21:05:58.000000000 -0700
> +++ linux-2.6.21/mm/mmap.c 2007-05-30 08:27:55.000000000 -0700
> @@ -245,16 +245,6 @@
> if (brk < mm->end_code)
> goto out;
>
> - /*
> - * Check against rlimit here. If this check is done later after the test
> - * of oldbrk with newbrk then it can escape the test and let the data
> - * segment grow beyond its set limit the in case where the limit is
> - * not page aligned -Ram Gupta
> - */
> - rlim = current->signal->rlim[RLIMIT_DATA].rlim_cur;
> - if (rlim < RLIM_INFINITY && brk - mm->start_data > rlim)
> - goto out;
> -
> newbrk = PAGE_ALIGN(brk);
> oldbrk = PAGE_ALIGN(mm->brk);
> if (oldbrk == newbrk)
> @@ -267,6 +257,18 @@
> goto out;
> }
>
> + /*
> + * Check against private data size limit. Count memory allocations
> + * which are not shared and not executable code as private data.
> + */
> + rlim = current->signal->rlim[RLIMIT_DATA].rlim_cur;
> + if (rlim < RLIM_INFINITY) {
> + unsigned long priv_vm;
> + priv_vm = (mm->total_vm - mm->shared_vm - mm->exec_vm) << PAGE_SHIFT;
> + if (priv_vm + (newbrk-oldbrk) > rlim)
> + goto out;
> + }
> +
> /* Check against existing mmap mappings. */
> if (find_vma_intersection(mm, oldbrk, newbrk+PAGE_SIZE))
> goto out;
> @@ -875,7 +877,8 @@
> = VM_STACK_FLAGS & (VM_GROWSUP|VM_GROWSDOWN);
>
> if (file) {
> - mm->shared_vm += pages;
> + if (flags & VM_SHARED)
> + mm->shared_vm += pages;
> if ((flags & (VM_EXEC|VM_WRITE)) == VM_EXEC)
> mm->exec_vm += pages;
> } else if (flags & stack_flags)
> @@ -1041,6 +1044,20 @@
> if (!may_expand_vm(mm, len >> PAGE_SHIFT))
> return -ENOMEM;
>
> + /*
> + * Check against private data size limit. Count memory allocations
> + * which are not shared and not executable code as private data.
> + */
> + if (!(vm_flags & (VM_SHARED|VM_EXEC))) {
> + unsigned long rlim, priv_vm;
> + rlim = current->signal->rlim[RLIMIT_DATA].rlim_cur;
> + if (rlim < RLIM_INFINITY) {
> + priv_vm = (mm->total_vm - mm->shared_vm - mm->exec_vm) << PAGE_SHIFT;
> + if (priv_vm + len > rlim)
> + return -ENOMEM;
> + }
> + }
> +
> if (accountable && (!(flags & MAP_NORESERVE) ||
> sysctl_overcommit_memory == OVERCOMMIT_NEVER)) {
> if (vm_flags & VM_SHARED) {
>
>
> -
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists