linux-kernel - Re: frequent lockups in 3.18rc4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20141219151210.GD13404@redhat.com>
Date:	Fri, 19 Dec 2014 10:12:10 -0500
From:	Dave Jones <davej@...hat.com>
To:	Chris Mason <clm@...com>
Cc:	Linus Torvalds <torvalds@...ux-foundation.org>,
	Mike Galbraith <umgwanakikbuti@...il.com>,
	Ingo Molnar <mingo@...nel.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Dâniel Fraga <fragabr@...il.com>,
	Sasha Levin <sasha.levin@...cle.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Suresh Siddha <sbsiddha@...il.com>,
	Oleg Nesterov <oleg@...hat.com>,
	Peter Anvin <hpa@...ux.intel.com>
Subject: Re: frequent lockups in 3.18rc4

On Fri, Dec 19, 2014 at 09:30:37AM -0500, Chris Mason wrote:

 > > in more recent builds. I've been running kitchen-sink debug kernels
 > > for my trinity runs for the last three years, and it's only this
 > > last few months that this has got to be enough of a problem that I'm
 > > not seeing the more interesting bugs. (Or perhaps we're just getting
 > > better at fixing them in -next now, so my runs are lasting longer..)
 > 
 > I think we're also adding more and more debugging.  It's definitely a 
 > good thing, but I think a lot of them are expected to stay off until 
 > you're trying to track down a specific problem.  I do always run with 
 > CONFIG_DEBUG_PAGEALLOC here and lock debugging/lockdep, and aside from 
 > being slow haven't hit trouble.

I think in the new year I'll hack up something I run on each kernel
build that picks a random subset of the debug options.  It's been on
my whiteboard for a while anyway, to try and get more 'real world'
looking kernel testing.  If I can get enough machines to test on,
it should still mean we get enough testing that we'll catch stuff early on.

It does seem like things have gotten so 'heavy' that a lot of what I've
been seeing have been ghosts. That said, there's also been several
real problems that have been shaken out during this thread over the last
two months, so I don't feel like we've wasted our time entirely.

 > I know it's 3.16 instead of 3.17, but 16K stacks are probably 
 > increasing the pressure on everything in these runs.  It's my favorite 
 > kernel feature this year, but it's likely to make trinity hurt more on 
 > memory constrained boxes.

That's actually a good point. Even just the forking/exiting overhead
is now much higher when we're starting & tearing down hundreds of child
processes every few seconds. Couple that with some children 'stuck'
in VM functions, and I could see the kernel struggling to find order
2 pages for a while. (Though never to the point where it fails).

 > I know you have traces with a ton more output, but I'm still 
 > wondering if usb-serial and printk from NMI really get along well.  I'd 
 > try with debugging back on and serial consoles off.  We carry patches 
 > to make oom print less, just because the time spent on our slow 
 > emulated serial console is enough to back the box up into a death 
 > spiral.

So I'm running out of time on this, and will realistically only
have this machine over the weekend.  I can give that a try, hopefully
if it fails, it'll fail early so we can try something else.

	Dave

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/