linux-kernel - 32bit process on a 64bit arch: ENOMEM on fork, what is the exact cause?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:	Thu, 9 Dec 2010 17:33:09 +0100
From:	Francis Galiegue <fgaliegue@...il.com>
To:	linux-kernel@...r.kernel.org
Subject: 32bit process on a 64bit arch: ENOMEM on fork, what is the exact cause?

[Please Cc: me on replies, I am not on the list]

Hello everyone,

This is the situation:

* the machine is an RHEL 5.x, x86_64 with all updates applied;
* one user, tomcat, runs, well, Tomcat, with a 32bit JVM (Sun 1.6.0.16
FWIW) and it has to for now (a 32bit native Oracle driver has to be
used);
* this JVM has, as an argument -Xmx3072M:
* the application uses Runtime.exec() to fork processes on a regular basis;
* and this is the output of 'ulimit -a' for the user:
----
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30720
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
----

In the logs, I sometimes see that it fails to execute an external
program with errno=12, which means ENOMEM. In spite of that, when this
error occurs, the machine still has 250 MB RAM free.

As all occurrences of such an error I could google out are RAM/swap
exhaustion and this is clearly not the case here, I suspect this is
due to the process being 32bit. But I cannot prove it. Am I at least
right in suspecting that? If yes, what is the real cause? What memory
region was exhausted? An SNMP JVM probe showed that an
OutOfMemoryError was far, far away...

Have fun,
-- 
Francis Galiegue, fgaliegue@...il.com
"It seems obvious [...] that at least some 'business intelligence'
tools invest so much intelligence on the business side that they have
nothing left for generating SQL queries" (Stéphane Faroult, in "The
Art of SQL", ISBN 0-596-00894-5)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/