linux-kernel - Re: frequent lockups in 3.18rc4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20141202184202.GM25340@linux.vnet.ibm.com>
Date:	Tue, 2 Dec 2014 10:42:02 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Dâniel Fraga <fragabr@...il.com>
Cc:	Linus Torvalds <torvalds@...ux-foundation.org>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: frequent lockups in 3.18rc4

On Tue, Dec 02, 2014 at 03:14:08PM -0200, Dâniel Fraga wrote:
> On Tue, 2 Dec 2014 09:04:07 -0800
> "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com> wrote:
> 
> > Is it harder to reproduce with CONFIG_PREEMPT=y and CONFIG_TREE_PREEMPT_RCU=n?
> 
> 	Yes, it's much harder! :)
> 
> > If it is a -lot- harder to reproduce, it might be worth bisecting among
> > the RCU read-side critical sections.  If making a few of them be
> > non-preemptible greatly reduces the probability of the bug occuring,
> > that might provide a clue about root cause.
> > 
> > On the other hand, if it is just a little harder to reproduce, this
> > RCU read-side bisection would likely be an exercise in futility.
> 
> 	Ok, I want to bisect it. Since it could be painful to bisect,
> could you suggest 2 commits between 3.16.0 and 3.17.0 so we can narrow
> the bisect? I could just bisect between 3.16.0 and 3.17.0 but it would
> take many days :).
> 
> 	Ps: if you prefer I bisect between 3.16.0 and 3.17.0, no
> problem, but you'll have to be patient ;).

I was actually suggesting something a bit different.  Instead of bisecting
by release, bisect by code.  The procedure is as follows:

1.	I figure out some reliable way of making RCU allow preemption to
	be disabled for some RCU read-side critical sections, but not for
	others.  I send you the patch, which has rcu_read_lock_test()
	as well as rcu_read_lock().

2.	You build a kernel without my Kconfig hack, with my patch from
	#1 above, and build a kernel with CONFIG_PREEMPT=y (which of
	course implies CONFIG_TREE_PREEMPT_RCU=y, given that you are
	building without my Kconfig hack).

3.	You make a list of all the rcu_read_lock() uses in the kernel
	(or ask me to provide it).  You change the rcu_read_lock()
	calls in the first half of this list to rcu_read_lock_test().

	If the kernel locks up as easily with this change as it did
	in a stock CONFIG_PREEMPT=y CONFIG_TREE_PREEMPT_RCU=y kernel,
	change half of the remaining rcu_read_lock() calls to
	rcu_read_lock_test().  If the kernel is much more resistant
	to lockup, change half of the rcu_read_lock_test() calls
	back to rcu_read_lock().

4.	It is quite possible that several of the RCU read-side critical
	sections contribute to the unreliability, in which case the
	bisection will get a bit more complicated.

Other thoughts on how to attack this?

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/