linux-kernel - Re: Re: [V2 PATCH 1/3] x86/panic: Fix re-entrance problem due to panic on NMI

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150730122747.GA3954@dhcp22.suse.cz>
Date:	Thu, 30 Jul 2015 14:27:47 +0200
From:	Michal Hocko <mhocko@...nel.org>
To:	河合英宏 / KAWAI，HIDEHIRO 
	<hidehiro.kawai.ez@...achi.com>
Cc:	Jonathan Corbet <corbet@....net>,
	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...nel.org>,
	"Eric W. Biederman" <ebiederm@...ssion.com>,
	"H. Peter Anvin" <hpa@...or.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Vivek Goyal <vgoyal@...hat.com>,
	"linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>,
	"x86@...nel.org" <x86@...nel.org>,
	"kexec@...ts.infradead.org" <kexec@...ts.infradead.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	Ingo Molnar <mingo@...hat.com>,
	平松雅巳 / HIRAMATU，MASAMI 
	<masami.hiramatsu.pt@...achi.com>
Subject: Re: Re: [V2 PATCH 1/3] x86/panic: Fix re-entrance problem due to
 panic on NMI

On Thu 30-07-15 11:55:52, 河合英宏 / KAWAI，HIDEHIRO wrote:
> > From: Michal Hocko [mailto:mhocko@...nel.org]
[...]
> > Could you point me to the code which does that, please? Maybe we are
> > missing that in our 3.0 kernel. I was quite surprised to see this
> > behavior as well.
> 
> Please see the snippet below.
> 
> void setup_local_APIC(void)
> {
> ...
>         /*
>          * only the BP should see the LINT1 NMI signal, obviously.
>          */
>         if (!cpu)
>                 value = APIC_DM_NMI;
>         else
>                 value = APIC_DM_NMI | APIC_LVT_MASKED;
>         if (!lapic_is_integrated())             /* 82489DX */
>                 value |= APIC_LVT_LEVEL_TRIGGER;
>         apic_write(APIC_LVT1, value);
> 
> 
> LINT1 pins of cpus other than CPU 0 are masked here.
> However, at least on some of Hitachi servers, NMI caused by NMI
> button doesn't seem to be delivered through LINT1.  So, my `external NMI'
> word may not be correct.

I am not familiar with details here but I can tell you that this
particular code snippet is the same in our 3.0 based kernel so it seems
that the HW is indeed doing something differently.

> > You might still get a panic on hardlockup which will happen on all CPUs
> > from the NMI context so we have to be able to handle panic in NMI on
> > many CPUs.
> 
> Do you say about the case of a kerne panic while other cpus locks up
> in NMI context?  In that case, there is no way to do things needed by
> kdump procedure including saving registeres...

I am saying that watchdog_overflow_callback might trigger on more CPUs
and panic from NMI context as well. So this is not reduced to the NMI
button sends NMI to more CPUs.

Why cannot the panic() context save all the registers if we are going to
loop in NMI context? This would be imho preferable to returning from
panic IMO.

[...]
> > I can provide the full log but it is quite mangled. I guess the
> > CPU130 was the only one allowed to proceed with the panic while others
> > returned from the unknown NMI handling. It took a lot of time until
> > CPU130 managed to boot the crash kernel with soft lockups and RCU stalls
> > reports. CPU0 is most probably locked up waiting for CPU130 to
> > acknowledge the IPI which will not happen apparently.
> 
> There is a timeout of 1000ms in nmi_shootdown_cpus(), so I don't know
> why CPU 130 waits so long.  I'll try to consider for a while.

Yes, I do not understand the timing here either and the fact that the
log is a complete mess in the important parts doesn't help a wee bit.
 
[...]

-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/