linux-kernel - Re: odd userspace lockups.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20131126185833.GA4121@redhat.com>
Date:	Tue, 26 Nov 2013 13:58:33 -0500
From:	Dave Jones <davej@...hat.com>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	Linux Kernel <linux-kernel@...r.kernel.org>
Subject: Re: odd userspace lockups.

On Tue, Nov 26, 2013 at 10:44:16AM -0800, Linus Torvalds wrote:

 > > At first I thought it was just being really slow, (especially after seeing
 > > all those congestion related symbols in the stack traces), but even after
 > > being left for hours, it doesn't seem to progress.
 > 
 > It *does* seem to be hung in some congestion thing. Possibly brought
 > on by being low on memory due to a memory leak.
 > 
 > I'm not seeing any suspicious locks: the sshd that holds the mmap_sem
 > locks for a fork() seems to be in congestion-wait for the disk due to
 > trying to allocate memory, exactly like all the trinity children are.
 > 
 > You don't have memory information in your dump, but to me it looks
 > like you're basically out of memory (there's a *lot* of
 > trinity-children trying to allocate pages), and the oom killer isn't
 > triggering for whatever reason.
 > 
 > Of course, it could easily be a kernel memory leak too (rather than
 > trinity), triggered by your odd workload.

So, completely different machine: Yesterday I got back from my vacation to
find my router running 3.12+ from a week earlier had hung completely.
On the screen was some oom killer output.  After bringing it back up
I looked at the mrtg graphs of it for the last week:
http://codemonkey.org.uk/junk/mem.free-week.png
Those stair-steps are around the time of morning that cron runs.

Normally I reboot that machine every day to update to the latest kernel,
but because I was on vacation, it had enough uptime to accumulate a leak.

That machine is just routing packets (and making pretty mrtg graphs) though,
not doing any fuzz testing or anything weird.

Who knows, maybe it's something totally unrelated. I'll keep poking at things,
and try and find an easier way to reproduce it.

	Dave

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/