Date:	Sat, 29 Dec 2012 19:56:38 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Frederic Weisbecker <fweisbec@...il.com>
Cc:	Steven Rostedt <rostedt@...dmis.org>,
	LKML <linux-kernel@...r.kernel.org>,
	Alessio Igor Bogani <abogani@...nel.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Avi Kivity <avi@...hat.com>,
	Chris Metcalf <cmetcalf@...era.com>,
	Christoph Lameter <cl@...ux.com>,
	Geoff Levand <geoff@...radead.org>,
	Gilad Ben Yossef <gilad@...yossef.com>,
	Hakan Akkan <hakanakkan@...il.com>,
	Ingo Molnar <mingo@...nel.org>,
	Paul Gortmaker <paul.gortmaker@...driver.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Li Zhong <zhong@...ux.vnet.ibm.com>
Subject: Re: [ANNOUNCE] 3.7-nohz1

On Mon, Dec 24, 2012 at 12:43:25AM +0100, Frederic Weisbecker wrote:
> 2012/12/21 Steven Rostedt <rostedt@...dmis.org>:
> > On Thu, 2012-12-20 at 19:32 +0100, Frederic Weisbecker wrote:
> >> Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU callbacks there and to
> >> handle the timekeeping. We set the rest as full dynticks. So you need the following kernel
> >> parameters:
> >>
> >>       rcu_nocbs=1-3 full_nohz=1-3
> >>
> >> (Note rcu_nocbs value must always be the same as full_nohz).
> >
> > Why? You can't have: rcu_nocbs=1-4 full_nohz=1-3
> 
> That should be allowed.
> 
> >   or: rcu_nocbs=1-3 full_nohz=1-4 ?
> 
> But that not.
> 
> You need to have: rcu_nocbs & full_nohz == full_nohz. This is because
> the tick is not there to maintain the local RCU callbacks anymore. So
> this must be offloaded to the rcu_nocb threads.
> 
> I just have a doubt with rcu_nocb. Do we still need the tick to
> complete the grace period for local rcu callbacks? I need to discuss
> that with Paul.

The tick is only needed if rcu_needs_cpu() returns false.  Of course,
this means that if you don't invoke rcu_needs_cpu() before returning to
adaptive-idle usermode execution, you are correct that a full_nohz CPU
would also have to be a rcu_nocbs CPU.

That said, I am getting close to having an rcu_needs_cpu() that only
returns false if there are callbacks immediately ready to invoke, at
least if RCU_FAST_NO_HZ=y.

							Thanx, Paul

> > That needs to be fixed. Either with a warning, and/or to force the two
> > to be the same. That is, if they specify:
> >
> >   rcu_nocbs=1-3 full_nohz=1-4
> >
> > Then set rcu_nocbs=1-4 with a warning about it. Or simply set
> >  full_nohz=1-3.
> 
> Yep, will do.
> 
> Thanks!
> 
> >
> > -- Steve
> >
> >>
> >> Now if you want proper isolation you need to:
> >>
> >> * Migrate your processes adequately
> >> * Migrate your irqs to CPU 0
> >> * Migrate the RCU nocb threads to CPU 0. Example with the above configuration:
> >>
> >>       for p in $(ps -o pid= -C rcuo1,rcuo2,rcuo3)
> >>       do
> >>               taskset -cp 0 $p
> >>       done
> >>
> >> Then run what you want on the full dynticks CPUs. For best results, run 1 task
> >> per CPU, mostly in userspace and mostly CPU bound (otherwise more IO = more kernel
> >> mode execution = more chances to get IPIs, tick restarted, workqueues, kthreads, etc...)
> >>
> >> This page contains a good reminder for those interested in CPU isolation: https://github.com/gby/linux/wiki
> >>
> >> But keep in mind that my tree is not yet ready for serious production.
> >>
> >
> >
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists