linux-kernel - Re: Yet more softlockups.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130712171808.GD1537@redhat.com>
Date:	Fri, 12 Jul 2013 13:18:08 -0400
From:	Dave Jones <davej@...hat.com>
To:	David Ahern <dsahern@...il.com>
Cc:	Dave Hansen <dave.hansen@...el.com>,
	Ingo Molnar <mingo@...nel.org>,
	Markus Trippelsdorf <markus@...ppelsdorf.de>,
	Thomas Gleixner <tglx@...utronix.de>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	Peter Anvin <hpa@...or.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Dave Hansen <dave.hansen@...ux.intel.com>
Subject: Re: Yet more softlockups.

On Fri, Jul 12, 2013 at 11:12:13AM -0600, David Ahern wrote:
 > On 7/12/13 9:45 AM, Dave Jones wrote:
 > > Here's a fun trick:
 > >
 > > trinity -c perf_event_open -C4 -q -l off
 > >
 > > Within about a minute, that brings any of my boxes to its knees.
 > > The softlockup detector starts going nuts, and then the box wedges solid.
 > 
 > I tried that in a VM running latest Linus tree. I see trinity children 
 > getting nuked regularly from oom.

Weird. I'm curious what the backtrace looks like in those cases.
Where is it trying to allocate memory ?
(Though that isn't usually too helpful in most cases, but in absense of
 anything else..)

 > I was dumping Vm elements using:
 > 
 > while [ 1 ]; do echo $(date) $(egrep Vm /proc/$pid/status); sleep 1; done
 > 
 > And right before the process is killed was the line:
 > 
 > Fri Jul 12 11:00:19 MDT 2013 VmPeak: 2867472 kB VmSize: 2867472 kB 
 > VmLck: 0 kB VmPin: 0 kB VmHWM: 1493092 kB VmRSS: 1493092 kB VmData: 
 > 2857944 kB VmStk: 136 kB VmExe: 100 kB VmLib: 1844 kB VmPTE: 5628 kB 
 > VmSwap: 0 kB
 > 
 > The VmData is growing fairly steadily and strace shows a lot of brk 
 > calls. Is that normal for trinity - or this command line?
 
Hmm, there are a few known leaks in trinity that I still haven't got
around to fixing, but none that should be triggered from this.

I wonder if something about being virtualised is perturbing its behaviour
somehow. Though I can't think what would matter. 

 > Looking at the perf_event_open calls I see a lot of E2BIG errors in 
 > addition to EINVAL. e.g,
 > 
 > ...
 > perf_event_open(0xba9000, 0, 0x4c, 0xcc, 0) = -1 EINVAL (Invalid argument)
 > alarm(0)                                = 1
 > getppid()                               = 9031
 > alarm(1)                                = 0
 > perf_event_open(0xba9000, 0x2a6e, 0xe, 0xfd, 0) = -1 E2BIG (Argument 
 > list too long)
 > alarm(0)                                = 1
 > getppid()                               = 9031
 > alarm(1)                                = 0

Yep, that'll happen.

Perhaps try CONFIG_MEMLEAK ?

	Dave

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/