linux-kernel - Re: Yet more softlockups.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130712170018.GA1537@redhat.com>
Date:	Fri, 12 Jul 2013 13:00:18 -0400
From:	Dave Jones <davej@...hat.com>
To:	Dave Hansen <dave@...1.net>
Cc:	Dave Hansen <dave.hansen@...el.com>,
	Ingo Molnar <mingo@...nel.org>,
	Markus Trippelsdorf <markus@...ppelsdorf.de>,
	Thomas Gleixner <tglx@...utronix.de>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	Peter Anvin <hpa@...or.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Dave Hansen <dave.hansen@...ux.intel.com>
Subject: Re: Yet more softlockups.

On Fri, Jul 12, 2013 at 08:55:31AM -0700, Dave Hansen wrote:

 > I mean that somebody turned 'active_events' on without actually wanting
 > perf to be on.  I'd be curious how it got set to something nonzero.
 > Could you stick a WARN_ONCE() or printk_ratelimit() on the three sites
 > that modify it?
 
I'll add to my list of things to get to, but it probably won't be until post-weekend.

 > > Here's a fun trick:
 > > 
 > > trinity -c perf_event_open -C4 -q -l off
 > > 
 > > Within about a minute, that brings any of my boxes to its knees.
 > > The softlockup detector starts going nuts, and then the box wedges solid.
 > 
 > *But* dropping the perf sample rate has been really effective at keeping
 > me from hitting it, and I've had to use _lots_ of CPUs (60-160) doing
 > those NMIs at once to trigger the lockups.
 > 
 > Being able to trigger it with so few CPUs is interesting though.  I'll
 > try on my hardware.

I hacked trinity to pause for 24s after each call, and changed the kernel
to taint on lockup (so that trinity would immediatly stop).
My hope was to find the combination of perf_event_open calls that triggered this.
After 12 hours, it's still ticking along. It's now done about a thousand
more calls than is usually necessary to hit the bug. Which makes me wonder
if this is timing related somehow.

Perhaps also worth noting that the majority of calls trinity makes will -EINVAL

	Dave

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/