linux-kernel - Re: mmap/clone returns ENOMEM with lots of free memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.DEB.2.00.1205071514040.6029@router.home>
Date:	Mon, 7 May 2012 15:15:46 -0500 (CDT)
From:	Christoph Lameter <cl@...ux.com>
To:	Robert Święcki <robert@...ecki.net>
cc:	linux-kernel@...r.kernel.org, linux-mm@...ck.org
Subject: Re: mmap/clone returns ENOMEM with lots of free memory

On Mon, 7 May 2012, Robert Święcki wrote:

> root@...-test:~/kern-fuz# ./cont.sh
> su: Cannot fork user shell
> su: Cannot fork user shell
> su: Cannot fork user shell
>
> root@...-test:~/kern-fuz# strace -e mmap,clone su test -c 'kill -CONT
> -1' 2>&1 | grep "= \-1"
> clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7fadf334f9f0) = -1 ENOMEM (Cannot allocate memory)
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = -1 ENOMEM (Cannot allocate memory)

Hmmm... That looks like some maximum virtual memory limit was violated.

Check ulimit and the overcommit settings (see /proc/meminfo's commitlimit
etc)