linux-kernel - Re: [PATCH v2 00/17] [request for stable 3.10 inclusion] x86/nmi: Print all cpu stacks from NMI safely

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150520132259.GE2728@pathway.suse.cz>
Date:	Wed, 20 May 2015 15:22:59 +0200
From:	Petr Mladek <pmladek@...e.cz>
To:	Wang Long <long.wanglong@...wei.com>
Cc:	rostedt@...dmis.org, jkosina@...e.cz, gregkh@...uxfoundation.org,
	stable@...r.kernel.org, peifeiyue@...wei.com,
	linux-kernel@...r.kernel.org, paulmck@...ux.vnet.ibm.com,
	dzickus@...hat.com, x86@...nel.org, morgan.wang@...wei.com,
	sasha.levin@...cle.com
Subject: Re: [PATCH v2 00/17] [request for stable 3.10 inclusion] x86/nmi:
 Print all cpu stacks from NMI safely

On Tue 2015-05-19 14:57:46, Petr Mladek wrote:
> On Tue 2015-05-19 09:08:45, Wang Long wrote:
> > This is my backport patch series to Fix the problem(backport to 3.10):
> > "
> > When trigger_all_cpu_backtrace() is called on x86, it will trigger an
> > NMI on each CPU and call show_regs(). But this can lead to a hard lock
> > up if the NMI comes in on another printk().
> > "
> > The solution is described in commit "a9edc88093287183ac934be44f295f183b2c62dd":
> > when the NMI triggers, it switches the printk routine for that CPU to call 
> > a NMI safe printk function that records the printk in a per_cpu seq_buf 
> > descriptor. After all NMIs have finished recording its data, the trace_
> > seqs are printed in a safe context.
> > 
> > The solution use "switch printk routine" and "seq_buf" infrastructures, but the
> > 3.10 stable have no both of them.
> > 
> > The patch 1-13 backport the "seq_buf" infrastructures. in detail, patch 1, 2
> > and 6 only backport "seq_buf" related code.
> > 
> > The patch 14-15 backport the "switch printk routine".
> > 
> > The patch 16-17 is the patch to print all cpu stacks from NMI safely
> > 
> > as discussed in https://lkml.org/lkml/2015/5/13/497, in 3.10 stable, this is 
> > the only way to solve the problem and the backport code is a bit more.
> > 
> > v1 -> v2:
> >  * fix the indent error.
> >  * rebase on 3.10.79
> > 
> > Any thoughts?
> 
> Please, wait with the integration. I am testing it with a storm of
> sysrq requests:
> 
>     $> while true ; do echo l >/proc/sysrq-trigger ; done
> 
> with iptables enabled:
> 
>     $> iptables -A INPUT -j LOG --log-prefix "incomming packet:"
> 
> and storm of pings from other machine:
> 
>     $> ping -f <patched-host>
> 
> 
> The machine somehow freezes. It does not make sense. I am trying to investigate.

OK, it seems that the machine freezes because there are still few
messages printed in the NMI context, e.g.:

[ 3080.286277] Uhhuh. NMI received for unknown reason 3d on CPU 12.
[ 3637.939276] Uhhuh. NMI received for unknown reason 2d on CPU 13.

I am not exactly sure why I get them on the test machine. But I get
such messages from time to time when hammering it by the pings and
sysrq-l requests.

I modified vprintk_emit() to do raw_spin_trylock(&logbuf_lock)
and do not try to lock console in NMI context. The trylock fails
from time to time but it does not longer freeze.

I am going to clean up the vprintk_emit() modification and send it for
review.

Anyway, this patch set seems to work as expected. It heavily reduces
the risk of NMI/printk-related deadlocks => it is worth having.

Feel free to use the following for the whole patchset (backport):

Reviewed-by: Petr Mladek <pmladek@...e.cz>
Tested-by: Petr Mladek <pmladek@...e.cz>


Best Regards,
Petr
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/