[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20130819094623.GA30389@gmail.com>
Date: Mon, 19 Aug 2013 11:46:23 +0200
From: Ingo Molnar <mingo@...nel.org>
To: Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@...achi.com>
Cc: linux-kernel@...r.kernel.org, Andi Kleen <ak@...ux.intel.com>,
"H. Peter Anvin" <hpa@...or.com>, Gleb Natapov <gleb@...hat.com>,
Konrad Rzeszutek Wilk <konrad.wilk@...cle.com>,
Joerg Roedel <joro@...tes.org>, x86@...nel.org,
stable@...r.kernel.org, Marcelo Tosatti <mtosatti@...hat.com>,
Hidehiro Kawai <hidehiro.kawai.ez@...achi.com>,
Sebastian Andrzej Siewior <sebastian@...akpoint.cc>,
Ingo Molnar <mingo@...hat.com>,
Zhang Yanfei <zhangyanfei@...fujitsu.com>,
"Eric W. Biederman" <ebiederm@...ssion.com>,
yrl.pp-manager.tt@...achi.com,
Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>,
Thomas Gleixner <tglx@...utronix.de>,
Seiji Aguchi <seiji.aguchi@....com>,
Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: [PATCH] [BUGFIX] crash/ioapic: Prevent crash_kexec() from
deadlocking of ioapic_lock
* Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@...achi.com> wrote:
> Prevent crash_kexec() from deadlocking of ioapic_lock.
s/of/on
> When crash_kexec() is executed on a cpu, the cpu will get
> ioapic_lock in disable_IO_APIC(). So if the cpu gets NMI
> while locking ioapic_lock, a deadlock wiil happen. In
s/will
> this patch, ioapic_lock is initialized before
> disable_IO_APIC().
>
> To confirm this deadlocking, you'll set up as follows:
s/deadlocking/deadlock
> 1. Add mdelay(1000) after raw_spin_lock_irqsave() in
> native_ioapic_set_affinity()@arch/x86/kernel/apic/io_apic.c
>
> Although the deadlocking can occur without this modification, it will
> increase the potential of the deadlocking problem.
s/deadlocking/deadlock
>
> 2. Build and install the kernel
>
> 3. Set up the OS which will run panic() and kexec when NMI is injected
> # echo "kernel.unknown_nmi_panic=1" >> /etc/sysctl.conf
> # vim /etc/default/grub
> add "nmi_watchdog=0 crashkernel=256M" in GRUB_CMDLINE_LINUX line
> # grub2-mkconfig
>
> 4. Reboot the OS
>
> 5. Run following command for each vcpu on the guest
> # while true; do echo <CPU num> > /proc/irq/<IO-APIC-edge or IO-APIC-fasteoi>/smp_affinitity; done;
> By running this command, cpus will get ioapic_lock for setting affinity.
>
> 6. Inject NMI (push a dump button or execute 'virsh inject-nmi <domain>' if you
> use VM)
> After injecting NMI, panic() is called in an nmi-handler context.
> Then, kexec will normally run in panic(), but the operation will be stopped
> by deadlock of ioapic_lock in crash_kexec()->machine_crash_shutdown()->
s/of/on
> native_machine_crash_shutdown()->disable_IO_APIC()->clear_IO_APIC()->
> clear_IO_APIC_pin()->ioapic_read_entry().
I suppose we could do this patch if it's a common
occurance.
A few minor details need fixing:
>
> Signed-off-by: Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@...achi.com>
> Cc: Thomas Gleixner <tglx@...utronix.de>
> Cc: Ingo Molnar <mingo@...hat.com>
> Cc: "H. Peter Anvin" <hpa@...or.com>
> Cc: x86@...nel.org
> Cc: Andrew Morton <akpm@...ux-foundation.org>
> Cc: Andi Kleen <ak@...ux.intel.com>
> Cc: Seiji Aguchi <seiji.aguchi@....com>
> Cc: Konrad Rzeszutek Wilk <konrad.wilk@...cle.com>
> Cc: Sebastian Andrzej Siewior <sebastian@...akpoint.cc>
> Cc: Joerg Roedel <joro@...tes.org>
> Cc: Zhang Yanfei <zhangyanfei@...fujitsu.com>
> Cc: "Eric W. Biederman" <ebiederm@...ssion.com>
> Cc: Gleb Natapov <gleb@...hat.com>
> Cc: Marcelo Tosatti <mtosatti@...hat.com>
> Cc: linux-kernel@...r.kernel.org
> Cc: stable@...r.kernel.org
> ---
> arch/x86/include/asm/apic.h | 2 ++
> arch/x86/kernel/apic/io_apic.c | 5 +++++
> arch/x86/kernel/crash.c | 4 ++++
> 3 files changed, 11 insertions(+)
>
> diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h
> index f8119b5..ddb06af 100644
> --- a/arch/x86/include/asm/apic.h
> +++ b/arch/x86/include/asm/apic.h
> @@ -715,4 +715,6 @@ static inline void exiting_ack_irq(void)
> ack_APIC_irq();
> }
>
> +extern void ioapic_lock_init(void);
> +
> #endif /* _ASM_X86_APIC_H */
> diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
> index 9ed796c..2816c07 100644
> --- a/arch/x86/kernel/apic/io_apic.c
> +++ b/arch/x86/kernel/apic/io_apic.c
> @@ -1534,6 +1534,11 @@ void intel_ir_io_apic_print_entries(unsigned int apic,
> }
> }
>
> +void ioapic_lock_init(void)
> +{
> + raw_spin_lock_init(&ioapic_lock);
> +}
Please name this ioapic_zap_locks() to make clear that this
is crash handling related.
> +
> __apicdebuginit(void) print_IO_APIC(int ioapic_idx)
> {
> union IO_APIC_reg_00 reg_00;
> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> index 74467fe..ea039d5 100644
> --- a/arch/x86/kernel/crash.c
> +++ b/arch/x86/kernel/crash.c
> @@ -129,6 +129,10 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
>
> lapic_shutdown();
> #if defined(CONFIG_X86_IO_APIC)
Please enhance this #ifdef while at it.
> + /*
> + * Prevent crash_kexec() from deadlocking of ioapic_lock.
> + */
s/of/on.
Also, single-line comment can go /* here */.
> + ioapic_lock_init();
> disable_IO_APIC();
> #endif
> #ifdef CONFIG_HPET_TIMER
>
Thanks,
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists