linux-kernel - Re: printk: preempt_disable with long time resulting in softlockup/RCU stall issues

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20150709141042.GH2900@quack.suse.cz>
Date:	Thu, 9 Jul 2015 16:10:42 +0200
From:	Jan Kara <jack@...e.cz>
To:	"Luis R. Rodriguez" <mcgrof@...e.com>
Cc:	Gavin Hu <gavin.hu.2010@...il.com>, Petr Mladek <pmladek@...e.com>,
	linux-kernel@...r.kernel.org, Alex Elder <elder@...aro.org>,
	"Steven Rostedt (Red Hat)" <rostedt@...dmis.org>,
	Joe Perches <joe@...ches.com>,
	Peter Hurley <peter@...leysoftware.com>,
	Tejun Heo <tj@...nel.org>, cxie4@...vell.com, cldu@...vell.com,
	xjian@...vell.com, fswu@...vell.com, Jan Kara <jack@...e.cz>
Subject: Re: printk: preempt_disable with long time resulting in
 softlockup/RCU stall issues

On Wed 08-07-15 19:04:02, Luis R. Rodriguez wrote:
> On Wed, Jul 08, 2015 at 05:08:35PM +0800, Gavin Hu wrote:
> > Hi,
> > 
> > Yes. We should disable the printk_limit feature when panic to avoid missing
> > messages.
> 
> Sounds like you have been looking into it and have a good idea of what you
> want to do, why not try it and send some RFC patches ?

Well, there are patches which try to fix this problem. Petr has referenced
them (at least one posting of them) and we do carry them in SLES kernels
(since without them some big machines aren't able to boot with serial
console attached). The biggest obstacle for them to get merged was that:

a) Andrew Morton didn't like the additional complexity in the printk code.

b) Alan Cox was objecting that the problem isn't in the printk code but
rather in serial console drivers which are too slow to print and thus we
should instead implement some buffering there.

If people care, I can refresh my memory where we ended, refresh the patches
and repost them. Probably we should deal with this one way or another...

> While at it, then we could consider doing different things depending on the
> message type. KERN_EMERG would disable preemption, whereas KERN_INFO may not be
> so critical to require it.

I don't think keying on message type is a good idea. You want to keep
message ordering in the first place so KERN_EMERG message would have to
print everything before it in the printk ring buffer but with other levels
being async this may take even longer than before... So I don't think we
should do that unless really necessary. Printk code currently keys on
'oops_in_progress' to do some special hacks making messages more likely to
appear on console and my patches use the same to switch to a sync mode
where we write out everything we have regardless of how long it takes (at
that point we are doomed anyway so some additional softlockup won't make
matters much worse).

								Honza
-- 
Jan Kara <jack@...e.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/