linux-kernel - Re: [PATCH tip/core/rcu 1/2] rcu: Parallelize and economize NOCB kthread wakeups

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20140812053321.GK4184@grmbl.mre>
Date:	Tue, 12 Aug 2014 11:03:21 +0530
From:	Amit Shah <amit.shah@...hat.com>
To:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
Cc:	linux-kernel@...r.kernel.org, riel@...hat.com, mingo@...nel.org,
	laijs@...fujitsu.com, dipankar@...ibm.com,
	akpm@...ux-foundation.org, mathieu.desnoyers@...icios.com,
	josh@...htriplett.org, niv@...ibm.com, tglx@...utronix.de,
	peterz@...radead.org, rostedt@...dmis.org, dhowells@...hat.com,
	edumazet@...gle.com, dvhart@...ux.intel.com, fweisbec@...il.com,
	oleg@...hat.com, sbw@....edu
Subject: Re: [PATCH tip/core/rcu 1/2] rcu: Parallelize and economize NOCB
 kthread wakeups

On (Mon) 11 Aug 2014 [20:45:31], Paul E. McKenney wrote:
> On Mon, Aug 11, 2014 at 01:34:21PM -0700, Paul E. McKenney wrote:
> > On Tue, Aug 12, 2014 at 01:48:45AM +0530, Amit Shah wrote:
> > > On (Mon) 11 Aug 2014 [13:11:02], Paul E. McKenney wrote:
> > > > On Tue, Aug 12, 2014 at 01:11:26AM +0530, Amit Shah wrote:
> > > > > On (Mon) 11 Aug 2014 [09:28:07], Paul E. McKenney wrote:
> > > > > > On Mon, Aug 11, 2014 at 12:43:08PM +0530, Amit Shah wrote:
> > > > > > > On (Fri) 08 Aug 2014 [14:46:48], Paul E. McKenney wrote:
> > > > > > > > On Fri, Aug 08, 2014 at 02:43:47PM -0700, Paul E. McKenney wrote:
> > > > > > > > > On Sat, Aug 09, 2014 at 12:04:24AM +0530, Amit Shah wrote:
> > > > > > > > > > On (Fri) 08 Aug 2014 [11:18:35], Paul E. McKenney wrote:
> > > > > > > > > 
> > > > > > > > > [ . . . ]
> > > > > > > > > 
> > > > > > > > > > > Hmmm... What happens if you boot a7d7a143d0b4cb1914705884ca5c25e322dba693
> > > > > > > > > > > with the kernel parameter "acpi=off"?
> > > > > > > > > > 
> > > > > > > > > > That doesn't change anything - still hangs.
> > > > > > > > > > 
> > > > > > > > > > I intend to look at this more on Monday, though - turning in for
> > > > > > > > > > today.  In the meantime, if there's anything else you'd like me to
> > > > > > > > > > try, please let me know.
> > > > > > > > > 
> > > > > > > > > OK, given that I still cannot reproduce it, I do need your help with
> > > > > > > > > the diagnostics.  And so what sorts of diagnostics work for you in
> > > > > > > > > the hung state?  Are you able to dump ftrace buffers?
> > > > > > > > > 
> > > > > > > > > If you are able to dump ftrace buffers, please enable rcu:rcu_nocb_wake
> > > > > > > > > and send me the resulting trace.
> > > > > > > > 
> > > > > > > > And another random kernel boot parameter to try is rcu_nocb_poll.
> > > > > > > 
> > > > > > > Right, this gets the boot going again:
> > > > > > 
> > > > > > OK, that likely indicates a lost wakeup.  The event tracing enabled by
> > > > > > "rcu:rcu_nocb_wake" should help track those down.  Last time, it was qemu
> > > > > > losing the wakeups, but maybe it is RCU this time.  ;-)
> > > > > 
> > > > > The guest goes dead pretty early; is there a trick to enabling and
> > > > > getting these traces out of the guest that I don't know of that
> > > > > doesn't involve being booted into userspace?  I can perhaps try
> > > > > getting the trace output out from a virtio-serial channel; but even
> > > > > that driver isn't probed yet when the lockup happens.
> > > > 
> > > > First boot with the kernel parameter "trace_event=rcu:rcu_nocb_wake".
> > > > Then when the system hangs, do "sendkey alt-sysrq-z" at the "(qemu)"
> > > > prompt to dump the ftrace buffer.  This hopefully dumps the trace buffer
> > > > to dmesg.
> > > > 
> > > > In addition "sendkey alt-sysrq-t" at the "(qemu)" prompt dumps all tasks'
> > > > stacks, which would also likely be useful information.
> > > 
> > > Nah, this doesn't work -- the guest's totally locked up.  I need a way
> > > to continuously dump buffers till the lockup happens, I suppose.
> > 
> > That is a bit surprising.  Is it possible that the system is OOMing
> > quickly due to grace periods not proceeding?  If so, maybe giving the
> > VM more memory would help.
> 
> Oh, and it is necessary to build the kernel with CONFIG_RCU_TRACE=y
> for the rcu_nocb_wake trace events to be enabled in the first place.
> I am assuming that your kernel was built with CONFIG_MAGIC_SYSRQ=y.

Yes, it is :-)  I checked the rcu_nocb_poll cmdline option does indeed
dump all the ftrace buffers to dmesg.

> If all of that is in place and no joy, is it possible to extract the
> ftrace buffer from the running/hung guest?  It should be in there
> somewhere!  ;-)

I know of only virtio-console doing this (via userspace only,
though).

		Amit
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/