linux-kernel - Re: [RFC] x86, perf: high volume of events produces a flood of unknown NMIs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4DAF2D2C.6060509@gmail.com>
Date:	Wed, 20 Apr 2011 22:59:56 +0400
From:	Cyrill Gorcunov <gorcunov@...il.com>
To:	Don Zickus <dzickus@...hat.com>
CC:	linux-kernel@...r.kernel.org, mingo@...e.hu, peterz@...radead.org,
	robert.richter@....com, andi@...stfloor.org, ming.m.lin@...el.com,
	eranian@...gle.com, ying.huang@...el.com,
	mathieu.desnoyers@...icios.com, acme@...hat.com
Subject: Re: [RFC] x86, perf: high volume of events produces a flood of unknown
 NMIs

On 04/20/2011 10:47 PM, Cyrill Gorcunov wrote:
> On 04/20/2011 10:26 PM, Don Zickus wrote:
>> Hi,
>>
>> Arnaldo pointed me at an NMI problem that happens when he tries to
>> generate a high volume of perf events.  He receives a flood of unknown
>> NMIs.
>>
>> I have been poking at the problem and came up with a patch, but it doesn't
>> always work.  I was hoping people who understood how the NMI works at a
>> low level might be able to help me.
>>
>> I was able to duplicate this on an AMD Phenom, Pentium 4, Xeon Core2quad,
>> and Nehalem.  The problem I think is the large generation of back-to-back
>> NMIs.  The perf nmi handler may accidentally handle some of those
>> extra/in-flight NMIs in its first pass, leaving the next NMI to be
>> unhandled and generating an unknown NMI message.
>>
>> Robert included logic to check for two back-to-back NMIs, but that falls
>> short when more then three are generated.  I modified his logic to account
>> for three back-to-back NMIs, but that didn't completely solve the problem.
>>
>> I took another approach at catching back-to-back NMIs that seemed to work
>> on all my machines except for the Xeon core2quad, but I am not entirely
>> sure if my approach is valid.
>>
>> The approach I took was based on the idea that if an NMI is being
>> generated while currently in an NMI handler, the current NMI when finished
>> won't continue executing the next instruction before the exception but
>> instead jump back into another NMI exception frame.
>>
>> As a result, the args passed in to the NMI handler should have the same ip
>> and sp as the previous NMI interrupt.  Otherwise one could assume that
>> some amount of time passed between interrupts (enough to return from the
>> exception and execute code).
>>
>> I thought this would allow me to trap an infinite number of back-to-back
>> NMIs.  Like I said it seemed to work in a number of machines, except for
>> my Xeon core2quad.
>>
>> Does anyone know if my approach is a valid one?  Or is there a better way
>> to catch this condition?  Or maybe some other tips or tricks I can use to
>> help come up with a solution for this?
>>
>> Or perhaps we don't care about this because in the end perf can't even
>> capture the data without spitting out a CPU Overload message.
>>
>> Thoughts?
>>
> 
> Hi Don, just a thought -- since pmi masks lvtpc we could read it and check if it's
> masked or no, though I fear it is quite time consuming operation in compare with
> frames :( (hmm, intel spec mentions only p4 and xeon as masking lvtpc)

Something like

	if (apic_read(APIC_LVTPC) & APIC_LVT_MASKED))
		handle-perf

-- 
    Cyrill
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/