linux-kernel - mmap/clone returns ENOMEM with lots of free memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 7 May 2012 17:54:29 +0200
From:	Robert Święcki <robert@...ecki.net>
To:	linux-kernel@...r.kernel.org, linux-mm@...ck.org
Subject: mmap/clone returns ENOMEM with lots of free memory

Hi,

I'm fuzzing 3.4.0-rc5 (mostly with
http://code.google.com/p/iknowthis/), and got to the point that new
fuzzing threads/processes don't want to run any more. I have script
that periodically sends SIGCONT to all processes.

while [ 1 ]; do su test -c 'kill -CONT -1'; su test2 -c 'kill -CONT
-1'; su nobody -c 'kill -CONT -1'; sleep 300; done

It doesn't work:

root@...-test:~/kern-fuz# ./cont.sh
su: Cannot fork user shell
su: Cannot fork user shell
su: Cannot fork user shell

root@...-test:~/kern-fuz# strace -e mmap,clone su test -c 'kill -CONT
-1' 2>&1 | grep "= \-1"
clone(child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7fadf334f9f0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = -1 ENOMEM (Cannot allocate memory)

It happens with every user (even newly created one), so it doesn't
seem to be any per-id rlimit.

root@...-test:~/kern-fuz# su - test3
su: Cannot fork user shell
root@...-test:~/kern-fuz# su - test3
$  <-- success
root@...-test:~/kern-fuz# su - test3
su: Cannot fork user shell
root@...-test:~/kern-fuz# su - test3
su: Cannot fork user shell
root@...-test:~/kern-fuz# su - test3
su: Cannot fork user shell

Not sure how to debug it yet. I can run kdb/kgdb on it, but before I
dive into mm structures, I though I attach some proc files from your
entertainment, maybe you can spot anything interesting there.

Also, whatever happened on this machine, i.e. any syscall during
fuzzing, was invoked from non-root user.

-- 
Robert Święcki

View attachment "_proc_buddyinfo.txt" of type "text/plain" (300 bytes)

View attachment "_proc_config.txt" of type "text/plain" (109242 bytes)

View attachment "_proc_meminfo.txt" of type "text/plain" (1170 bytes)

View attachment "_proc_slabinfo.txt" of type "text/plain" (32864 bytes)

View attachment "_proc_vmallocinfo.txt" of type "text/plain" (7662 bytes)

View attachment "_proc_vmstat.txt" of type "text/plain" (2059 bytes)

Download attachment "ps_waux" of type "application/octet-stream" (10892 bytes)

View attachment "top.txt" of type "text/plain" (4355 bytes)

View attachment "ulimit_a.txt" of type "text/plain" (692 bytes)