linux-kernel - Re: Can RCU stall lead to hard lockups?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180207025504.GA10892@mail.hallyn.com>
Date:   Tue, 6 Feb 2018 20:55:04 -0600
From:   "Serge E. Hallyn" <serge@...lyn.com>
To:     "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
Cc:     "Serge E. Hallyn" <serge@...lyn.com>, Tejun Heo <tj@...nel.org>,
        linux-kernel@...r.kernel.org, kernel-team@...com
Subject: Re: Can RCU stall lead to hard lockups?

On Tue, Feb 06, 2018 at 06:53:37PM -0800, Paul E. McKenney wrote:
> On Tue, Feb 06, 2018 at 08:33:03PM -0600, Serge E. Hallyn wrote:
> > On Sat, Feb 03, 2018 at 12:50:32PM -0800, Paul E. McKenney wrote:
> > > On Fri, Feb 02, 2018 at 05:44:30PM -0600, Serge E. Hallyn wrote:
> > > > Quoting Paul E. McKenney (paulmck@...ux.vnet.ibm.com):
> > > > > On Tue, Jan 09, 2018 at 06:11:14AM -0800, Tejun Heo wrote:
> > > > > > Hello, Paul.
> > > > > > 
> > > > > > On Mon, Jan 08, 2018 at 08:24:25PM -0800, Paul E. McKenney wrote:
> > > > > > > > I don't know the RCU code at all but it *looks* like the first CPU is
> > > > > > > > taking a sweet while flushing printk buffer while holding a lock (the
> > > > > > > > console is IPMI serial console, which faithfully emulates 115200 baud
> > > > > > > > rate), and everyone else seems stuck waiting for that spinlock in
> > > > > > > > rcu_check_callbacks().
> > > > > > > > 
> > > > > > > > Does this sound possible?
> > > > > > > 
> > > > > > > 115200 baud?  Ouch!!!  That -will- result in trouble from console
> > > > > > > printing, and often also in RCU CPU stall warnings.
> > > > > > 
> > > > > > It could even be slower than 115200, and we occassionally see RCU
> > > > > > stall warnings caused by printk storms, for example, while the kernel
> > > > > > is trying to dump a lot of info after an OOM.  That's an issue we
> > > > > > probably want to improve from printk side; however, they don't usually
> > > > > > lead to NMI hard lockup detector kicking in and crashing the machine,
> > > > > > which is the peculiarity here.
> > > > > > 
> > > > > > Hmmm... show_state_filter(), the function which dumps all task
> > > > > > backtraces, share a similar problem and it avoids it by explicitly
> > > > > > calling touch_nmi_watchdog().  Maybe we can do something like the
> > > > > > following from RCU too?
> > > > > 
> > > > > If this fixes things for you, I would welcome such a patch.
> > > > 
> > > > Hi - would this also be relevant to 4.9-stable and 4.4-stable, or
> > > > has something elsewhere changed after 4.9 that actually triggers this?
> > > 
> > > As far as I can tell, slow console lines have been prone to RCU CPU stall
> > > warnings for a very long time.
> > 
> > Ok, thanks Paul.
> > 
> > Tejun were you going to push this?
> 
> I have it queued for the next merge window.  3eea9623926f ("rcu: Call
> touch_nmi_watchdog() while printing stall warnings") in -rcu.

D'oh - thanks!

-serge