linux-kernel - 2.6.20, Java, unresponsive userspace

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-Id: <200703040407.07234.s0348365@sms.ed.ac.uk>
Date:	Sun, 4 Mar 2007 04:07:07 +0000
From:	Alistair John Strachan <s0348365@....ed.ac.uk>
To:	linux-kernel@...r.kernel.org
Subject: 2.6.20, Java, unresponsive userspace

Hi,

I've had some problems with 2.6.20 (but also with earlier kernels, like 
Debian's 2.6.18) where one of the users on our system, running a Java6 
application, is somehow able to make the userspace on the machine totally 
unresponsive, denying all remote _and_ local access to the machine.

- The AMD64 port of Linux and Java are in use
- The process runs as an unprivileged user
- The OOM killer doesn't kick in
- No swap is enabled on the machine
- I'm told that the program doesn't fork bomb
  (but I've not got a reproducible test case yet)
- The program is able to deny service even to a getty login.

If I SAK the machine, init respawns getty and everything is fine again.

alistair@...rent:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) 1000
max memory size         (kbytes, -m) 1000
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) 0
stack size              (kbytes, -s) 2048
cpu time               (seconds, -t) unlimited
max user processes              (-u) 100
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Any ideas with how to proceed for debugging this issue?

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/