linux-kernel - Re: Yet more softlockups.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 12 Jul 2013 08:38:52 -0700
From:	Dave Hansen <dave.hansen@...el.com>
To:	Ingo Molnar <mingo@...nel.org>
CC:	Dave Jones <davej@...hat.com>,
	Markus Trippelsdorf <markus@...ppelsdorf.de>,
	Thomas Gleixner <tglx@...utronix.de>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	Peter Anvin <hpa@...or.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Dave Hansen <dave.hansen@...ux.intel.com>
Subject: Re: Yet more softlockups.

On 07/12/2013 03:31 AM, Ingo Molnar wrote:
> * Dave Jones <davej@...hat.com> wrote:
>> On Wed, Jul 10, 2013 at 05:20:15PM +0200, Markus Trippelsdorf wrote:
>>  > On 2013.07.10 at 11:13 -0400, Dave Jones wrote:
>>  > > I get this right after booting..
>>  > > 
>>  > > [  114.516619] perf samples too long (4262 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
>>  > 
>>  > You can disable this warning by:
>>  > 
>>  > echo 0 > /proc/sys/kernel/perf_cpu_time_max_percent
>>
>> Yes, but why is this even being run when I'm not running perf ?
>>
>> The only NMI source running should be the watchdog.
> 
> The NMI watchdog is a perf event.
> 
> I've Cc:-ed Dave Hansen, the author of those changes - is this a false 
> positive or some real problem?

The warning comes from calling perf_sample_event_took(), which is only
called from one place: perf_event_nmi_handler().

So we can be pretty sure that the perf NMI is firing, or at least that
this handler code is running.

nmi_handle() says:
        /*
         * NMIs are edge-triggered, which means if you have enough
         * of them concurrently, you can lose some because only one
         * can be latched at any given time.  Walk the whole list
         * to handle those situations.
         */

perf_event_nmi_handler() probably gets _called_ when the watchdog NMI
goes off.  But, it should hit this check:

        if (!atomic_read(&active_events))
                return NMI_DONE;

and return quickly. This is before it has a chance to call
perf_sample_event_took().

Dave, for your case, my suspicion would be that it got turned on
inadvertently, or that we somehow have a bug which bumped up
perf_event.c's 'active_events' and we're running some perf code that we
don't have to.

But, I'm suspicious.  I was having all kinds of issues with perf and
NMIs taking hundreds of milliseconds.  I never isolated it to having a
real, single, cause.  I attributed it to my large NUMA system just being
slow.  Your description makes me wonder what I missed, though.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/