linux-kernel - Re: Yet more softlockups.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <51E026F3.6010505@sr71.net>
Date:	Fri, 12 Jul 2013 08:55:31 -0700
From:	Dave Hansen <dave@...1.net>
To:	Dave Jones <davej@...hat.com>, Dave Hansen <dave.hansen@...el.com>,
	Ingo Molnar <mingo@...nel.org>,
	Markus Trippelsdorf <markus@...ppelsdorf.de>,
	Thomas Gleixner <tglx@...utronix.de>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	Peter Anvin <hpa@...or.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Dave Hansen <dave.hansen@...ux.intel.com>
Subject: Re: Yet more softlockups.

On 07/12/2013 08:45 AM, Dave Jones wrote:
> On Fri, Jul 12, 2013 at 08:38:52AM -0700, Dave Hansen wrote:
>  > Dave, for your case, my suspicion would be that it got turned on
>  > inadvertently, or that we somehow have a bug which bumped up
>  > perf_event.c's 'active_events' and we're running some perf code that we
>  > don't have to.
>  
> What do you 'inadvertantly' ? I see this during bootup every time.
> Unless systemd or something has started playing with perf, (which afaik it isn't)

I mean that somebody turned 'active_events' on without actually wanting
perf to be on.  I'd be curious how it got set to something nonzero.
Could you stick a WARN_ONCE() or printk_ratelimit() on the three sites
that modify it?

>  > But, I'm suspicious.  I was having all kinds of issues with perf and
>  > NMIs taking hundreds of milliseconds.  I never isolated it to having a
>  > real, single, cause.  I attributed it to my large NUMA system just being
>  > slow.  Your description makes me wonder what I missed, though.
> 
> Here's a fun trick:
> 
> trinity -c perf_event_open -C4 -q -l off
> 
> Within about a minute, that brings any of my boxes to its knees.
> The softlockup detector starts going nuts, and then the box wedges solid.

On my box, the same happens with 'perf top'. ;)

*But* dropping the perf sample rate has been really effective at keeping
me from hitting it, and I've had to use _lots_ of CPUs (60-160) doing
those NMIs at once to trigger the lockups.

Being able to trigger it with so few CPUs is interesting though.  I'll
try on my hardware.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/