linux-kernel - Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20160328162954.GK4287@linux.vnet.ibm.com>
Date:	Mon, 28 Mar 2016 09:29:54 -0700
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
Cc:	Thomas Gleixner <tglx@...utronix.de>,
	Peter Zijlstra <peterz@...radead.org>,
	Jacob Pan <jacob.jun.pan@...ux.intel.com>,
	Josh Triplett <josh@...htriplett.org>,
	Ross Green <rgkernel@...il.com>,
	John Stultz <john.stultz@...aro.org>,
	lkml <linux-kernel@...r.kernel.org>,
	Ingo Molnar <mingo@...nel.org>,
	Lai Jiangshan <jiangshanlai@...il.com>, dipankar@...ibm.com,
	Andrew Morton <akpm@...ux-foundation.org>,
	rostedt <rostedt@...dmis.org>,
	David Howells <dhowells@...hat.com>,
	Eric Dumazet <edumazet@...gle.com>,
	Darren Hart <dvhart@...ux.intel.com>,
	Frédéric Weisbecker <fweisbec@...il.com>,
	Oleg Nesterov <oleg@...hat.com>,
	pranith kumar <bobby.prani@...il.com>,
	"Chatre, Reinette" <reinette.chatre@...el.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

On Mon, Mar 28, 2016 at 04:12:19PM +0000, Mathieu Desnoyers wrote:
> ----- On Mar 28, 2016, at 11:56 AM, Paul E. McKenney paulmck@...ux.vnet.ibm.com wrote:
> 
> > On Mon, Mar 28, 2016 at 03:07:36PM +0000, Mathieu Desnoyers wrote:
> >> ----- On Mar 28, 2016, at 9:29 AM, Paul E. McKenney paulmck@...ux.vnet.ibm.com
> >> wrote:
> >> 
> >> > On Mon, Mar 28, 2016 at 08:28:51AM +0200, Peter Zijlstra wrote:
> >> >> On Sun, Mar 27, 2016 at 02:09:14PM -0700, Paul E. McKenney wrote:
> >> >> 
> >> >> > > Does that system have MONITOR/MWAIT errata?
> >> >> > 
> >> >> > On the off-chance that this question was also directed at me,
> >> >> 
> >> >> Hehe, it wasn't, however, since we're here..
> >> >> 
> >> >> > here is
> >> >> > what I am running on.  I am running in a qemu/KVM virtual machine, in
> >> >> > case that matters.
> >> >> 
> >> >> Have you actually tried on real proper hardware? Does it still reproduce
> >> >> there?
> >> > 
> >> > Ross has, but I have not, given that I have a shared system on the one
> >> > hand and a single-socket (four core, eight hardware thread) laptop on
> >> > the other that has even longer reproduction times.  The repeat-by is
> >> > as follows:
> >> > 
> >> > o	Build a kernel with the following Kconfigs:
> >> > 
> >> >	CONFIG_SMP=y
> >> >	CONFIG_NR_CPUS=16
> >> >	CONFIG_PREEMPT_NONE=n
> >> >	CONFIG_PREEMPT_VOLUNTARY=n
> >> >	CONFIG_PREEMPT=y
> >> >	# This should result in CONFIG_PREEMPT_RCU=y
> >> >	CONFIG_HZ_PERIODIC=y
> >> >	CONFIG_NO_HZ_IDLE=n
> >> >	CONFIG_NO_HZ_FULL=n
> >> >	CONFIG_RCU_TRACE=y
> >> >	CONFIG_HOTPLUG_CPU=y
> >> >	CONFIG_RCU_FANOUT=2
> >> >	CONFIG_RCU_FANOUT_LEAF=2
> >> >	CONFIG_RCU_NOCB_CPU=n
> >> >	CONFIG_DEBUG_LOCK_ALLOC=n
> >> >	CONFIG_RCU_BOOST=y
> >> >	CONFIG_RCU_KTHREAD_PRIO=2
> >> >	CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
> >> >	CONFIG_RCU_EXPERT=y
> >> >	CONFIG_RCU_TORTURE_TEST=y
> >> >	CONFIG_PRINTK_TIME=y
> >> >	CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
> >> >	CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
> >> >	CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT=y
> >> > 
> >> >	If desired, you can instead build with CONFIG_RCU_TORTURE_TEST=m
> >> >	and modprobe/insmod the module manually.
> >> > 
> >> > o	Find a two-socket x86 system or larger, with at least 16 CPUs.
> >> > 
> >> > o	Boot the kernel with the following kernel boot parameters:
> >> > 
> >> >	rcutorture.onoff_interval=1 rcutorture.onoff_holdoff=30
> >> > 
> >> >	The onoff_holdoff is only needed for CONFIG_RCU_TORTURE_TEST=y.
> >> >	When manually setting up the module, you get the holdoff for
> >> >	free, courtesy of human timescales.
> >> > 
> >> > In the absence of instrumentation, I get failures usually within a
> >> > couple of hours, though sometimes much longer.  With instrumentation,
> >> > the sky appears to be the limit.  :-/
> >> > 
> >> > Ross is running on bare metal with no CPU hotplug, so perhaps his setup
> >> > is of more immediate interest.  He is seeing the same symptoms that I am,
> >> > namely a task being repeatedly awakened without actually coming out of
> >> > TASK_INTERRUPTIBLE state, let alone running.  As you pointed out earlier,
> >> > he cannot be seeing the same bug that my crude patch suppresses, but
> >> > given that I still see a few failures with that crude patch, it is quite
> >> > possible that there is still a common bug.
> >> 
> >> With respect to bare metal vs KVM guest, I've reported an issue with
> >> inaccurate detection of TSC as being an unreliable time source on a
> >> KVM guest. The basic setup is to overcommit the CPU use across the
> >> entire host, thus leading to preemption of the guest. The guest TSC
> >> watchdog then falsely assume that TSC is unreliable, because it gets
> >> preempted for a long time (e.g. 0.5 second) between reading the HPET
> >> and the TSC.
> >> 
> >> Ref. http://lkml.iu.edu/hypermail/linux/kernel/1509.1/00379.html
> >> 
> >> I'm wondering if what Paul is observing in the KVM setup might be
> >> caused by long preemption by the host. One way to stress test this
> >> is to run parallel kernel builds on the host (or in another guest)
> >> while the guest is running, thus over-committing the CPU use.
> >> 
> >> Thoughts ?
> > 
> > If I run NO_HZ_FULL, I do get warnings about unstable timesources.
> > 
> > And certainly guest VCPUs can be preempted.  However, if they were
> > preempted for the lengths of time I am seeing, I should also see
> > softlockup warnings on the host, which I do not see.
> 
> Why would you see softlockup warning on the host ?
> 
> I expect the priority at which the kvm vcpu runs is much lower than
> the priority of the rcu worker threads on the host. Therefore, you
> might very well have long preemption delays for kvm vpus while the
> rcu worker threads run fine on the host kernel because they have
> a higher priority.
> 
> Am I missing something ?

Right, host/guest confusion on my part.  I should expect softlockups
on the -guest- because rcutorture runs almost entirely in kernel
mode.  I don't see them there, either.

Another reason I do not expect preemption is the problem is because
I don't see it with waketorture, which has a large number of tasks
periodically waking up.

Yet another reason is that I can get things moving again by doing periodic
wakeups from the scheduling-clock interrupt.  (Why those wakeups make
things go but those from the timers don't is a mystery to me!)  Please see
commit 56ef96ac25c3 (DIAGS: Another horrible exploratory hack) for one
piece of this.  This would presumably have no effect on preemption at
the host level.

							Thanx, Paul

> Thanks,
> 
> Mathieu
> 
> > 
> > That said, perhaps I should cobble together something to force short
> > repeated preemptions at the host level.  Maybe that would get the
> > reproduction rate sufficiently high to enable less-dainty debugging.
> > 
> > 							Thanx, Paul
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>