linux-kernel - Re: [PATCH] x86: avoid calling arch_trigger_all_cpu

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20101018180312.GA2935@redhat.com>
Date:	Mon, 18 Oct 2010 14:03:12 -0400
From:	Don Zickus <dzickus@...hat.com>
To:	DDD <dongdong.deng@...driver.com>
Cc:	tglx@...utronix.de, hpa@...or.com, mingo@...e.hu, x86@...nel.org,
	linux-kernel@...r.kernel.org, bruce.ashfield@...driver.com
Subject: Re: [PATCH] x86: avoid calling arch_trigger_all_cpu_backtrace() at
 the same time on SMP

On Mon, Oct 18, 2010 at 07:00:15PM +0800, DDD wrote:
> CC to Ingo's mingo@...e.hu and add some code explains for this patch.
> 
> Dongdong
> 
> 
> Dongdong Deng wrote:
> >The spin_lock_debug/rcu_cpu_stall detector uses
> >trigger_all_cpu_backtrace() to dump cpu backtrace.
> >Therefore it is possible that trigger_all_cpu_backtrace()
> >could be called at the same time on different CPUs, which
> >triggers and 'unknown reason NMI' warning. The following case
> >illustrates the problem:

The patch seems reasonable.  I can queue it up.  Ingo wrote the original
code and knows which spin locks to use better than me, perhaps he can find
a moment to comment.

Cheers,
Don

> >
> >      CPU1                    CPU2                     ...   CPU N
> >                       trigger_all_cpu_backtrace()
> >                       set "backtrace_mask" to cpu mask
> >                               |
> >generate NMI interrupts  generate NMI interrupts       ...
> >    \                          |                               /
> >     \                         |                              /
> >   The "backtrace_mask" will be cleaned by the first NMI interrupt
> >  at nmi_watchdog_tick(), then the following NMI interrupts generated
> >by other cpus's arch_trigger_all_cpu_backtrace() will be took as
> >unknown reason NMI interrupts.
> >
> >This patch uses a lock to avoid the problem, and stop the
> >arch_trigger_all_cpu_backtrace() calling to avoid dumping double cpu
> >backtrace info when there is already a trigger_all_cpu_backtrace()
> >in progress.
> >
> >Signed-off-by: Dongdong Deng <dongdong.deng@...driver.com>
> >Reviewed-by: Bruce Ashfield <bruce.ashfield@...driver.com>
> >CC: Thomas Gleixner <tglx@...utronix.de>
> >CC: Ingo Molnar <mingo@...hat.com>
> >CC: "H. Peter Anvin" <hpa@...or.com>
> >CC: x86@...nel.org
> >CC: linux-kernel@...r.kernel.org
> >---
> > arch/x86/kernel/apic/hw_nmi.c |   14 ++++++++++++++
> > arch/x86/kernel/apic/nmi.c    |   14 ++++++++++++++
> > 2 files changed, 28 insertions(+), 0 deletions(-)
> >
> >diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c
> >index cefd694..3aea0a5 100644
> >--- a/arch/x86/kernel/apic/hw_nmi.c
> >+++ b/arch/x86/kernel/apic/hw_nmi.c
> >@@ -29,6 +29,16 @@ u64 hw_nmi_get_sample_period(void)
> > void arch_trigger_all_cpu_backtrace(void)
> > {
> > 	int i;
> >+	static arch_spinlock_t lock = __ARCH_SPIN_LOCK_UNLOCKED;
> 
> Why an arch spin lock vs just using a raw spin lock?
> for example. static DEFINE_RAW_SPINLOCK(lock);
> 
> The spin_lock_debug detector was used in raw_spinlock too.
> 
> arch_trigger_all_cpu_backtrace() -->
>   raw_spin_lock(lock)      -->
>   _raw_spin_lock(lock)     -->
>   __raw_spin_lock(lock)    -->
>   do_raw_spin_lock(lock)   -->
>   __spin_lock_debug(lock)  -->
>   trigger_all_cpu_backtrace()
> 
> Therefor, we have to use arch spin lock here.
> 
> 
> 
> >+	unsigned long flags;
> >+
> >+	local_irq_save(flags);
> 
> Why have to save the irq's here?
> 
> When the arch_trigger_all_cpu_backtrace() was triggered by
> "spin_lock()"'s spin_lock_debug detector, it is possible that
> the irq is enabled, thus we have to save and disable it here.
> 
> 
> 
> >+	if (!arch_spin_trylock(&lock))
> >+		/*
> >+		 * If there is already a trigger_all_cpu_backtrace()
> >+		 * in progress, don't output double cpu dump infos.
> >+		 */
> >+		goto out_restore_irq;
> > 	cpumask_copy(to_cpumask(backtrace_mask), cpu_online_mask);
> >@@ -41,6 +51,10 @@ void arch_trigger_all_cpu_backtrace(void)
> > 			break;
> > 		mdelay(1);
> > 	}
> >+
> >+	arch_spin_unlock(&lock);
> >+out_restore_irq:
> >+	local_irq_restore(flags);
> > }
> > static int __kprobes
> >diff --git a/arch/x86/kernel/apic/nmi.c b/arch/x86/kernel/apic/nmi.c
> >index a43f71c..5fa8a13 100644
> >--- a/arch/x86/kernel/apic/nmi.c
> >+++ b/arch/x86/kernel/apic/nmi.c
> >@@ -552,6 +552,16 @@ int do_nmi_callback(struct pt_regs *regs, int cpu)
> > void arch_trigger_all_cpu_backtrace(void)
> > {
> > 	int i;
> >+	static arch_spinlock_t lock = __ARCH_SPIN_LOCK_UNLOCKED;
> >+	unsigned long flags;
> >+
> >+	local_irq_save(flags);
> >+	if (!arch_spin_trylock(&lock))
> >+		/*
> >+		 * If there is already a trigger_all_cpu_backtrace()
> >+		 * in progress, don't output double cpu dump infos.
> >+		 */
> >+		goto out_restore_irq;
> > 	cpumask_copy(to_cpumask(backtrace_mask), cpu_online_mask);
> >@@ -564,4 +574,8 @@ void arch_trigger_all_cpu_backtrace(void)
> > 			break;
> > 		mdelay(1);
> > 	}
> >+
> >+	arch_spin_unlock(&lock);
> >+out_restore_irq:
> >+	local_irq_restore(flags);
> > }
> 
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/