linux-kernel - Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160304151824.GR3577@linux.vnet.ibm.com>
Date:	Fri, 4 Mar 2016 07:18:24 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Ross Green <rgkernel@...il.com>
Cc:	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
	John Stultz <john.stultz@...aro.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Peter Zijlstra <peterz@...radead.org>,
	lkml <linux-kernel@...r.kernel.org>,
	Ingo Molnar <mingo@...nel.org>,
	Lai Jiangshan <jiangshanlai@...il.com>, dipankar@...ibm.com,
	Andrew Morton <akpm@...ux-foundation.org>,
	Josh Triplett <josh@...htriplett.org>,
	rostedt <rostedt@...dmis.org>,
	David Howells <dhowells@...hat.com>,
	Eric Dumazet <edumazet@...gle.com>,
	Darren Hart <dvhart@...ux.intel.com>,
	Frédéric Weisbecker <fweisbec@...il.com>,
	Oleg Nesterov <oleg@...hat.com>,
	pranith kumar <bobby.prani@...il.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

On Fri, Mar 04, 2016 at 04:30:12PM +1100, Ross Green wrote:
> On Fri, Feb 26, 2016 at 12:35 PM, Paul E. McKenney <paulmck@...ux.vnet.ibm.com> wrote:

[ . . . ]

> >> OK, so what wakeup path omits the sched_wakeup event?
> >>
> >> The sched_waking event looks to occur once in try_to_wake_up() and
> >> once in try_to_wake_up_local().  Starting with try_to_wake_up():
> >>
> >> o     If the task is ->on_rq, ttwu_remote() is invoked:
> >>
> >>       o       This acquires the runqueue lock, then if
> >>               task_on_rq_queued() invokes ttwu_do_wakeup().  This
> >>               unconditionally does sched_wakeup, so we didn't go that
> >>               way.  (And this path skips the bulk of try_to_wake_up()
> >>               on return.)
> >>
> >>       o       Otherwise, we release the runqueu lock and returns zero.
> >>
> >> o     There is some ordering checking, runqueue selection, and then
> >>       p->state is set to TASK_WAKING.  And we apparently are not getting
> >>       here, either.  But I don't see any other way out.
> >>
> >>       Ignoring this for the moment...
> >>
> >>       We eventually reach to the call to ttwu_queue().
> >>
> >>       o       Here the TTWU_QUEUE path seems to avoid doing a
> >>               sched_wakeup event -- and since we are trying to wake
> >>               CPU 0 from CPU 4, so they don't share cache (x86).
> >>
> >>       o       This invokes ttwu_queue_remote(), which sends an IPI
> >>               unless polling is in effect.  I would need to enable
> >>               trace_sched_wake_idle_without_ipi() to see whether or
> >>               not the IPI was actually sent.
> >>
> >>               If the target CPU was offline, we should have seen the
> >>               cpu_is_offline() WARN_ON().  I suppose that the CPU might
> >>               go offline between the check and the ->send_IPI_mask(),
> >>               but only once.  And we are trying to wakeup on CPU 0
> >>               quite a few times.
> >>
> >>       Any thoughts on what to look for?
> >>
> >> Next, try_to_wake_up_local():
> >>
> >> o     After doing several checks, it does the sched_waking event.
> >>
> >> o     If the task is already queued, it calls ttwu_activate().
> >>
> >> o     It then invokes ttwu_do_wakeup(), which unconditionally
> >>       does the sched_wakeup() event.
> >>
> >>       So this path looks unlikely, even ignoring the fact that
> >>       the waking CPU in the traces above is always different than
> >>       the CPU to be awakened on.
> >>
> >> Any thoughts?
> >>
> >>                                                       Thanx, Paul
> G'day,
> 
> 
> Here is a series of rcu_preempt stall events(5) from linux-4.5-rc6 release.
> 
> Again some testing procedure. boot, run series of brief benchmarks and
> then leave idle.
> The first stall event appeared quite quickly - within hours, the rest
> at what appears to be random intervals after that.
> 
> 
> I thought I might give Daniels patch set a try and see how that goes!

Looks like the same issue from dmesg.

For my part, I added more tracing, which seems to have further decreased
the probability of occurrence.  The sched_wake_idle_without_ipi event
did not appear.

My next step is to try writing a torture test focused specifically on
this issue.  We need a faster reproducer to make decent progress.

							Thanx, Paul