linux-kernel - Re: frequent lockups in 3.18rc4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Sat, 13 Dec 2014 20:04:59 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Dave Jones <davej@...hat.com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Chris Mason <clm@...com>,
	Mike Galbraith <umgwanakikbuti@...il.com>,
	Ingo Molnar <mingo@...nel.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Dâniel Fraga <fragabr@...il.com>,
	Sasha Levin <sasha.levin@...cle.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: frequent lockups in 3.18rc4

On Sat, Dec 13, 2014 at 03:41:52PM -0500, Dave Jones wrote:
> On Sat, Dec 13, 2014 at 10:04:08AM -0800, Paul E. McKenney wrote:
>  > On Sat, Dec 13, 2014 at 11:59:15AM -0500, Dave Jones wrote:
>  > > On Fri, Dec 12, 2014 at 11:14:06AM -0800, Linus Torvalds wrote:
>  > >  > On Fri, Dec 12, 2014 at 10:54 AM, Dave Jones <davej@...hat.com> wrote:
>  > >  > >
>  > >  > > Something that's still making me wonder if it's some kind of hardware
>  > >  > > problem is the non-deterministic nature of this bug.
>  > >  > 
>  > >  > I'd expect it to be a race condition, though. Which can easily cause
>  > >  > these kinds of issues, and the timing will be pretty random even if
>  > >  > the load is very regular.
>  > >  > 
>  > >  > And we know that the scheduler has an integer overflow under Sasha's
>  > >  > loads, although I didn't hear anything from Ingo and friends about it.
>  > >  > Ingo/Peter, you were cc'd on that report, where at least one of the
>  > >  > multiplcations in wake_affine() ended up overflowing..
>  > >  > 
>  > >  > Some scheduler thing that overflows only under heavy load, and screws
>  > >  > up scheduling could easily account for the RCU thread thing. I see it
>  > >  > *less* easily accounting for DaveJ's case, though, because the
>  > >  > watchdog is running at RT priority,  and the scheduler would have to
>  > >  > screw up much more to then not schedule an RT task, but..
>  > >  > 
>  > >  > I'm also not sure if the bug ever happens with preemption disabled.
>  > > 
>  > > Bah, so I see some watchdog traces with preemption off, and that then
>  > > taints the kernel, and the fuzzing stops.  I'll hack something up
>  > > so it ignores the taint and keeps going. All I really care about here
>  > > is the "machine hangs completely" case, which the trace below didn't
>  > > hit..
>  > > 
>  > > (back to fuzzing almost everything, not just lsetxattr btw)
>  > 
>  > Hmmm...  This one looks like the RCU grace-period kthread is getting
>  > starved: "idle=b4c/0/0".  Is this running with the "dangerous" patch
>  > that sets these kthreads to RT priority?
> 
> sorry, no. Ran out of time yesterday. I'll try and get to applying that
> later this evening if I get chance.

Whew!!!  You had me worried there for a bit!  ;-)

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/