linux-kernel - Re: kexec crash kernel running with watchdog enabled

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 24 Jun 2015 16:31:41 +0000
From:	dwalker@...o99.com
To:	Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>
Cc:	Ralf Baechle <ralf@...ux-mips.org>, linux-mips@...ux-mips.org,
	david.daney@...ium.com, d.hatayama@...fujitsu.com,
	vgoyal@...hat.com, hidehiro.kawai.ez@...achi.com,
	ebiederm@...ssion.com, linux-kernel@...r.kernel.org
Subject: Re: kexec crash kernel running with watchdog enabled

On Wed, Jun 24, 2015 at 03:52:48PM +0900, Masami Hiramatsu wrote:
> Hi,
> 
> On 2015/06/23 23:05, dwalker@...o99.com wrote:
> > 
> > Hi,
> > 
> > There was a commit in kernel/panic.c which altered when the kexec crash kernel is executed,
> > 
> > commit f06e5153f4ae2e2f3b0300f0e260e40cb7fefd45
> > Author: Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>
> > Date:   Fri Jun 6 14:37:07 2014 -0700
> > 
> >     kernel/panic.c: add "crash_kexec_post_notifiers" option for kdump after panic_notifers
> > 
> > 
> > This made it possible for smp_send_stop() to be executed prior to calling the kexec crash
> > kernel.
> > 
> > The issue is that smp_send_stop() offlines the cores, and other code depend on the cores being online.
> > 
> > In my case on Octeon here's an example,
> > 
> > panic()
> >  crash_kexec()
> >   machine_crash_shutdown()
> >    octeon_generic_shutdown()
> > 
> > Inside octeon_generic_shutdown() the Octeon watchdog is shutdown for_each_online_cpu(), but since
> > most of the cpu's already got offlined in smp_send_stop() it means the watchdog is still alive on
> > those cores. This results in a reboot during the crash kernel execution.
> 
> Ah, I see.
> 
> > Another example seem to be in default_machine_crash_shutdown() where crash_kexec_prepare_cpus() depends
> > on an IPI for saving the registers on different cores. However, the cpu's are all offlined with
> > interrupts disabled so they won't be running those IPI's in this case.
> > 
> > I'm looking for any advice on how this should be fixed, or if it's already fixed. I'm not going to be
> > submitting a patch so if anyone wants to submit one feel free to do so.
> 
> Hmm, IMHO, when the cpu goes to offline in appropriate way(smp_send_stop), it should stop
> watchdog timer on the offlined cpu too.
> Or, you can also register crash handler which stops all watchdogs, but it's a bit tricky.
> 

That doesn't really fix all the issue tho. As I was explaining generic MIPS code depends on the cpu's
effectively being online for crash data collection (with an IPI). This issue may effect other architectures also,
because smp_send_stop() offlines the cpu on other architectures also. I haven't surveyed the other architectures
enough to know what issue could happen from this tho.

Is it possible to move the smp_send_stop() below the notifiers ? I'm just throwing out ideas.

Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/