Date:	Tue, 12 Jul 2016 07:12:34 +0000
From:	河合英宏 / KAWAI，HIDEHIRO 
	<hidehiro.kawai.ez@...achi.com>
To:	"'xlpang@...hat.com'" <xlpang@...hat.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Dave Young <dyoung@...hat.com>,
	"Eric W. Biederman" <ebiederm@...ssion.com>,
	Baoquan He <bhe@...hat.com>
CC:	Michal Hocko <mhocko@...e.com>, Toshi Kani <toshi.kani@....com>,
	"Peter Zijlstra (Intel)" <peterz@...radead.org>,
	Minfei Huang <mnfhuang@...il.com>,
	"H. Peter Anvin" <hpa@...or.com>,
	Daniel Walker <dwalker@...o99.com>,
	"Ingo Molnar" <mingo@...nel.org>,
	Takao Indoh <indou.takao@...fujitsu.com>,
	"x86@...nel.org" <x86@...nel.org>,
	"Lee, Chun-Yi" <joeyli.kernel@...il.com>,
	Borislav Petkov <bp@...e.de>, Vivek Goyal <vgoyal@...hat.com>,
	Petr Mladek <pmladek@...e.com>,
	Josh Poimboeuf <jpoimboe@...hat.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...hat.com>,
	"kexec@...ts.infradead.org" <kexec@...ts.infradead.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"HATAYAMA Daisuke" <d.hatayama@...fujitsu.com>,
	Masami Hiramatsu <mhiramat@...nel.org>,
	Tejun Heo <tj@...nel.org>,
	Vitaly Kuznetsov <vkuznets@...hat.com>
Subject: RE: Re: [V3 PATCH 1/2] x86/panic: Replace smp_send_stop() with
 kdump friendly version

> From: linux-kernel-owner@...r.kernel.org [mailto:linux-kernel-owner@...r.kernel.org] On Behalf Of Xunlei Pang
> Sent: Tuesday, July 12, 2016 3:57 PM
> On 2016/07/12 at 11:56, 河合英宏 / KAWAI，HIDEHIRO wrote:
> > Hi Xunlei,
> >
> > Thanks for the review.
> >
> >> From: Xunlei Pang [mailto:xpang@...hat.com]
> >> Sent: Tuesday, July 12, 2016 12:12 PM
> >> On 2016/07/05 at 19:33, Hidehiro Kawai wrote:
> >>> This patch fixes one of the problems reported by Daniel Walker
> >>> (https://lkml.org/lkml/2015/6/24/44).
> >>>
> >>> If crash_kexec_post_notifiers boot option is specified, other CPUs
> >>> are stopped by smp_send_stop() instead of machine_crash_shutdown()
> >>> in crash_kexec() path.  This behavior change leads two problems.
> >>>
> >>>  Problem 1:
> >>>  octeon_generic_shutdown() for MIPS OCTEON assumes that other CPUs are
> >>>  still online and try to stop their watchdog timer.  If
> >>>  smp_send_stop() is called before octeon_generic_shutdown(), stopping
> >>>  watchdog timer will fail because other CPUs have been offlined by
> >>>  smp_send_stop().
> >>>
> >>>    panic()
> >>>      if crash_kexec_post_notifiers == 1
> >>>        smp_send_stop()
> >>>        atomic_notifier_call_chain()
> >>>        kmsg_dump()
> >>>      crash_kexec()
> >>>        machine_crash_shutdown()
> >>>          octeon_generic_shutdown() // shutdown watchdog for ONLINE CPUs
> >>>
> >>>  Problem 2:
> >>>  Most of architectures stop other CPUs in machine_crash_shutdown()
> >>>  path, and they also do something needed for kdump.  For example,
> >>>  they save registers, disable virtualization extensions, and so on.
> >>>  However, if smp_send_stop() stops other CPUs before
> >>>  machine_crash_shutdown(), we miss those operations.
> >>>
> >>> How do we fix these problems?  In the first place, we should stop
> >>> other CPUs as soon as possible when panic() was called, otherwise
> >>> other CPUs may wipe out a clue to the cause of the failure.  So, we
> >>> replace smp_send_stop() with more suitable one for kdump.
> >>>
> >>> This patch solves Problem 2 by replacing smp_send_stop() in panic()
> >>> with panic_smp_send_stop().  This is a weak function which calls
> >>> smp_send_stop(), and architecture dependent code may override this
> >>> with appropriate one.  This patch only provides x86-specific version.
> >>>
> >>> Changes in V3:
> >>> - Revise comments, description, and symbol names
> >>>
> >>> Changes in V2:
> >>> - Replace smp_send_stop() call with crash_kexec version which
> >>>   saves cpu states and cleans up VMX/SVM
> >>> - Drop a fix for Problem 1 at this moment
> >>>
> >>> Reported-by: Daniel Walker <dwalker@...o99.com>
> >>> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> >>> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@...achi.com>
> >>> Cc: Dave Young <dyoung@...hat.com>
> >>> Cc: Baoquan He <bhe@...hat.com>
> >>> Cc: Vivek Goyal <vgoyal@...hat.com>
> >>> Cc: Eric Biederman <ebiederm@...ssion.com>
> >>> Cc: Masami Hiramatsu <mhiramat@...nel.org>
> >>> Cc: Thomas Gleixner <tglx@...utronix.de>
> >>> Cc: Ingo Molnar <mingo@...hat.com>
> >>> Cc: "H. Peter Anvin" <hpa@...or.com>
> >>> Cc: Borislav Petkov <bp@...e.de>
> >>> Cc: Toshi Kani <toshi.kani@....com>
> >>> Cc: "Peter Zijlstra (Intel)" <peterz@...radead.org>
> >>> Cc: Takao Indoh <indou.takao@...fujitsu.com>
> >>> Cc: "Lee, Chun-Yi" <joeyli.kernel@...il.com>
> >>> Cc: Minfei Huang <mnfhuang@...il.com>
> >>> Cc: Andrew Morton <akpm@...ux-foundation.org>
> >>> Cc: Michal Hocko <mhocko@...e.com>
> >>> Cc: Vitaly Kuznetsov <vkuznets@...hat.com>
> >>> Cc: Petr Mladek <pmladek@...e.com>
> >>> Cc: Tejun Heo <tj@...nel.org>
> >>> Cc: Josh Poimboeuf <jpoimboe@...hat.com>
> >>> ---
> >>>  arch/x86/kernel/crash.c |   14 ++++++++++----
> >>>  kernel/panic.c          |   43 ++++++++++++++++++++++++++++++++-----------
> >>>  2 files changed, 42 insertions(+), 15 deletions(-)
> >>>
> >>> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> >>> index 9ef978d..3305433 100644
> >>> --- a/arch/x86/kernel/crash.c
> >>> +++ b/arch/x86/kernel/crash.c
> >>> @@ -133,15 +133,21 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
> >>>  	disable_local_APIC();
> >>>  }
> >>>
> >>> -static void kdump_nmi_shootdown_cpus(void)
> >>> +/* Override the weak function in kernel/panic.c */
> >>> +void panic_smp_send_stop(void)
> >>>  {
> >>> -	nmi_shootdown_cpus(kdump_nmi_callback);
> >>> +	static int cpus_stopped;
> >> Should be atomic_t type?
> > panic_smp_send_stop() can be called by only one panicking CPU
> > (but can be called twice). It is sufficient to be normal variable.
> 
> There are other call sites of __crash_kexec() for oops cases, which can
> call panic_smp_send_stop() concurrently on a different cpu.

In oops cases, crash_kexec() is called first, then __crash_kexec() is
called.  crash_kexec() excludes concurrent execution of panic() and
crash_kexec() via panic_cpu, so panic_smp_send_stop() shouldn't be
called concurrently.

Best regards,

Hidehiro Kawai
Hitachi, Ltd. Research & Development Group

Powered by blists - more mailing lists