linux-kernel - Re: [PATCH 0/3 v2] nmi perf fixes

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20100910151735.GC13563@erda.amd.com>
Date:	Fri, 10 Sep 2010 17:17:35 +0200
From:	Robert Richter <robert.richter@....com>
To:	Ingo Molnar <mingo@...e.hu>
CC:	Don Zickus <dzickus@...hat.com>,
	Peter Zijlstra <peterz@...radead.org>,
	"gorcunov@...il.com" <gorcunov@...il.com>,
	"fweisbec@...il.com" <fweisbec@...il.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"ying.huang@...el.com" <ying.huang@...el.com>,
	"ming.m.lin@...el.com" <ming.m.lin@...el.com>,
	"yinghai@...nel.org" <yinghai@...nel.org>,
	"andi@...stfloor.org" <andi@...stfloor.org>,
	"eranian@...gle.com" <eranian@...gle.com>
Subject: Re: [PATCH 0/3 v2] nmi perf fixes

On 10.09.10 10:46:34, Ingo Molnar wrote:
> 
> * Don Zickus <dzickus@...hat.com> wrote:
> 
> > On Fri, Sep 10, 2010 at 01:41:40PM +0200, Peter Zijlstra wrote:
> > > On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
> > > > Fixes to allow unknown nmis to pass through the perf nmi handler instead
> > > > of being swallowed.  Contains patches that are already in Ingo's tree.  Added
> > > > here for completeness.  Based on ingo/tip
> > > > 
> > > > Tested on intel/amd
> > > > 
> > > > v2: patch cleanups and consolidation, no code changes
> > > > 
> > > > Don Zickus (1):
> > > >   perf, x86: Fix accidentally ack'ing a second event on intel perf
> > > >     counter
> > > > 
> > > > Peter Zijlstra (1):
> > > >   perf, x86: Fix handle_irq return values
> > > > 
> > > > Robert Richter (1):
> > > >   perf, x86: Try to handle unknown nmis with an enabled PMU
> > > > 
> > > >  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
> > > >  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
> > > >  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
> > > >  3 files changed, 56 insertions(+), 20 deletions(-)
> > > 
> > > Both Ingo and I are getting Dazed and confused on our AMD machines, it
> > > started before yesterday (that is, after backing out all my recent
> > > changes it still gets dazed), so I suspect this set.
> > > 
> > > I'll look at getting a trace of the thing, but if any of you has a
> > > bright idea...
> > 
> > What are you running to create the problem?  I can try and duplicate 
> > it here.
> 
> It happens easily here - just running something like:
> 
>    perf record -g ./hackbench 10

I try to reproduce it, which systems are affected?

> 
> a couple of times triggers it. Note, unlike with the earlier bug, the 
> NMIs are not permanently 'stuck' - and everything continues working. 
> Obviously the messages are nasty looking so this is a regression we need 
> to fix.

The patch below adds ratelimits.

-Robert

--

>From 1747710d684302b806b145e5acb590ab2088e5ca Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@....com>
Date: Thu, 5 Aug 2010 18:04:31 +0200
Subject: [PATCH] x86: ratelimit NMI messages

In case of a storm of unknown NMIs the cpu get stucked. This patch
adds ratelimits to avoid this.

Signed-off-by: Robert Richter <robert.richter@....com>
---
 arch/x86/kernel/traps.c |   16 +++++++++-------
 1 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 60788de..97a492d 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -366,15 +366,17 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)
 		return;
 	}
 #endif
-	printk(KERN_EMERG
-		"Uhhuh. NMI received for unknown reason %02x on CPU %d.\n",
-			reason, smp_processor_id());
-
-	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
+	if (printk_ratelimit()) {
+		printk(KERN_EMERG "Uhhuh. NMI received for unknown reason"
+		       " %02x on CPU %d.\n", reason, smp_processor_id());
+		printk(KERN_EMERG
+		       "Do you have a strange power saving mode enabled?\n");
+	}
 	if (panic_on_unrecovered_nmi)
 		panic("NMI: Not continuing");
-
-	printk(KERN_EMERG "Dazed and confused, but trying to continue\n");
+	if (printk_ratelimit())
+		printk(KERN_EMERG
+		       "Dazed and confused, but trying to continue\n");
 }
 
 static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
-- 
1.7.1.1



-- 
Advanced Micro Devices, Inc.
Operating System Research Center

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/