linux-kernel - Re: RCU vs NOHZ

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20220929152044.GE4196@paulmck-ThinkPad-P17-Gen-1>
Date:   Thu, 29 Sep 2022 08:20:44 -0700
From:   "Paul E. McKenney" <paulmck@...nel.org>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     Joel Fernandes <joel@...lfernandes.org>,
        Frederic Weisbecker <fweisbec@...il.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        linux-kernel@...r.kernel.org, Boqun Feng <boqun.feng@...il.com>,
        "Rafael J. Wysocki" <rjw@...ysocki.net>
Subject: Re: RCU vs NOHZ

On Thu, Sep 29, 2022 at 12:55:58PM +0200, Peter Zijlstra wrote:
> On Sat, Sep 17, 2022 at 07:25:08AM -0700, Paul E. McKenney wrote:
> > On Fri, Sep 16, 2022 at 11:20:14AM +0200, Peter Zijlstra wrote:
> > > On Fri, Sep 16, 2022 at 12:58:17AM -0700, Paul E. McKenney wrote:
> > > 
> > > > To the best of my knowledge at this point in time, agreed.  Who knows
> > > > what someone will come up with next week?  But for people running certain
> > > > types of real-time and HPC workloads, context tracking really does handle
> > > > both idle and userspace transitions.
> > > 
> > > Sure, but idle != nohz. Nohz is where we disable the tick, and currently
> > > RCU can inhibit this -- rcu_needs_cpu().
> > 
> > Exactly.  For non-nohz userspace execution, the tick is still running
> > anyway, so RCU of course won't be inhibiting its disabling.  And in that
> > case, RCU's hook is the tick interrupt itself.  RCU's hook is passed a
> > flag saying whether the interrupt came from userspace or from kernel.
> 
> I'm not sure how we ended up here; this is completely irrelevant and I'm
> not disagreeing with it.
> 
> > > AFAICT there really isn't an RCU hook for this, not through context
> > > tracking not through anything else.
> > 
> > There is a directly invoked RCU hook for any transition that enables or
> > disables the tick, namely the ct_*_enter() and ct_*_exit() functions,
> > that is, those functions formerly known as rcu_*_enter() and rcu_*_exit().
> 
> Context tracking doesn't know about NOHZ, therefore RCU can't either.
> Context tracking knows about IDLE, but not all IDLE is NOHZ-IDLE.
> 
> Specifically we have:
> 
> 	ct_{idle,irq,nmi,user,kernel}_enter()
> 
> And none of them are related to NOHZ in the slightest. So no, RCU does
> not have a NOHZ callback.
> 
> I'm still thikning you're conflating NOHZ_FULL (stopping the tick when
> in userspace) and regular NOHZ (stopping the tick when idle).
> 
> > And this of course means that any additional schemes to reduce RCU's
> > power consumption must be compared (with real measurements on real
> > hardware!) to Joel et al.'s work, whether in combination or as an
> > alternative.  And either way, the power savings must of course justify
> > the added code and complexity.
> 
> Well, Joel's lazy scheme has the difficulty that you can wreck things by
> improperly marking the callback as lazy when there's an explicit
> dependency on it. The talk even called that out.
> 
> I was hoping to construct a scheme that doesn't need the whole lazy
> approach.
> 
> 
> To recap; we want the CPU to go into deeper idle states, no?
> 
> RCU can currently inhibit this by having callbacks pending for this CPU
> -- in this case RCU inhibits NOHZ-IDLE and deep power states are not
> selected or less effective.
> 
> Now, deep idle states actually purge the caches, so cache locality
> cannot be an argument to keep the callbacks local.
> 
> We know when we're doing deep idle we stop the tick.
> 
> So why not, when stopping the tick, move the RCU pending crud elsewhere
> and let the CPU get on with going idle instead of inhibiting the
> stopping of the tick and wrecking deep idle?

Because doing so in the past has cost more energy than is saved.

							Thanx, Paul