linux-kernel - Re: RCU lockup? (was: Re: [PATCH v2 tip/core/rcu 10/14] rcu: Don't redundantly disable irqs in rcu_irq

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20160124020121.GD4503@linux.vnet.ibm.com>
Date:	Sat, 23 Jan 2016 18:01:21 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Geert Uytterhoeven <geert@...ux-m68k.org>
Cc:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	Ingo Molnar <mingo@...nel.org>, jiangshanlai@...il.com,
	dipankar@...ibm.com, Andrew Morton <akpm@...ux-foundation.org>,
	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
	Josh Triplett <josh@...htriplett.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Peter Zijlstra <peterz@...radead.org>,
	Steven Rostedt <rostedt@...dmis.org>,
	David Howells <dhowells@...hat.com>,
	Eric Dumazet <edumazet@...gle.com>,
	Darren Hart <dvhart@...ux.intel.com>,
	Frédéric Weisbecker <fweisbec@...il.com>,
	Oleg Nesterov <oleg@...hat.com>,
	pranith kumar <bobby.prani@...il.com>,
	"linux-arm-kernel@...ts.infradead.org" 
	<linux-arm-kernel@...ts.infradead.org>,
	linux-renesas-soc@...r.kernel.org, arnd@...db.de, olof@...om.net
Subject: Re: RCU lockup? (was: Re: [PATCH v2 tip/core/rcu 10/14] rcu: Don't
 redundantly disable irqs in rcu_irq_{enter,exit}())

On Sat, Jan 23, 2016 at 10:43:19AM +0100, Geert Uytterhoeven wrote:
> Hi Paul,
> 
> On Fri, Jan 22, 2016 at 9:44 PM, Paul E. McKenney
> <paulmck@...ux.vnet.ibm.com> wrote:
> > On Fri, Jan 22, 2016 at 09:55:44AM +0100, Geert Uytterhoeven wrote:
> >> On Thu, Jan 21, 2016 at 5:06 PM, Paul E. McKenney
> >> <paulmck@...ux.vnet.ibm.com> wrote:
> >> > On Thu, Jan 21, 2016 at 02:22:56PM +0100, Geert Uytterhoeven wrote:
> >> >> On Thu, Dec 10, 2015 at 12:10 AM, Paul E. McKenney
> >> >> <paulmck@...ux.vnet.ibm.com> wrote:
> >> >> > This commit replaces a local_irq_save()/local_irq_restore() pair with
> >> >> > a lockdep assertion that interrupts are already disabled.  This should
> >> >> > remove the corresponding overhead from the interrupt entry/exit fastpaths.
> >> >> >
> >> >> > This change was inspired by the fact that Iftekhar Ahmed's mutation
> >> >> > testing showed that removing rcu_irq_enter()'s call to local_ird_restore()
> >> >> > had no effect, which might indicate that interrupts were always enabled
> >> >> > anyway.
> >> >> >
> >> >> > Signed-off-by: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
> >> >> > ---
> >> >> >  include/linux/rcupdate.h   |  4 ++--
> >> >> >  include/linux/rcutiny.h    |  8 ++++++++
> >> >> >  include/linux/rcutree.h    |  2 ++
> >> >> >  include/linux/tracepoint.h |  4 ++--
> >> >> >  kernel/rcu/tree.c          | 32 ++++++++++++++++++++++++++------
> >> >> >  5 files changed, 40 insertions(+), 10 deletions(-)
> >> >>
> >> >> This commit (7c9906ca5e582a773fff696975e312cef58a7386) is triggering lock ups
> >> >> during boot on r8a7791/koelsch (dual Cortex A15). Probably this commit does not
> >> >> contain the real bug, but a symptom.
> >> >
> >> > On the off-chance that it is related, here is Ding Tianhong's patch
> >> > that addressed some lockups:
> >> >
> >> > http://www.eenyhelp.com/patch-rfc-locking-mutexes-dont-spin-owner-when-wait-list-not-null-help-215929641.html
> >> >
> >> > Does that help in your case?
> >>
> >> Unfortunately not.
> >
> > We could revert the RCU patch without any real problems -- it is after
> > all just an optimization.
> 
> I replaced the calls to rcu_irq_{enter,exit}() in irq_{enter,exit}() by their
> _irqson counterparts, which should be equivalent to the old code, but the issue
> persisted. Strange...

Indeed...

> Does it matter that arm has
> #define __ARCH_IRQ_EXIT_IRQS_DISABLED   1
> ?

No idea.  I added Arnd and Olof on CC in case they can tell us more.

> I tried JTAG, but enabling JTAG on r8a7791/koelsch requires changing a switch
> on the board, which also disables the second CPU core, and thus makes the issue
> disappear...

:-(

> > Hmmm...  One issue that we have seen before is that the irq-disabled
> > indication is a software flag that is not always in sync with
> > hardware conditions.  Might it be that we are hitting a situation where
> > irqs_disabled() is giving the wrong answer, thus suppressing the lockdep
> > warning?
> 
> Possible. I tried adding 'if(!irqs_disabled) printk("something")' just before
> the RCU_LOCKDEP_WARN(), but it never triggered. Worse, the issue went away by
> doing that :-(

That would be "if (!irqs_disabled())..." with the "()", correct?

But if you had lockdep enabled, and if lockdep didn't complain, I would
not expect the "if" to complain either.  The fact that the problem was
suppressed by the extra check is a bit annoying, I will grant you that!

							Thanx, Paul

> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@...ux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds
> 

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.