Message-ID: <YqjGTFEWSJGGOjNA@FVFF77S0Q05N>
Date:   Tue, 14 Jun 2022 18:33:00 +0100
From:   Mark Rutland <mark.rutland@....com>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     rth@...ddle.net, ink@...assic.park.msu.ru, mattst88@...il.com,
        vgupta@...nel.org, linux@...linux.org.uk,
        ulli.kroll@...glemail.com, linus.walleij@...aro.org,
        shawnguo@...nel.org, Sascha Hauer <s.hauer@...gutronix.de>,
        kernel@...gutronix.de, festevam@...il.com, linux-imx@....com,
        tony@...mide.com, khilman@...nel.org, catalin.marinas@....com,
        will@...nel.org, guoren@...nel.org, bcain@...cinc.com,
        chenhuacai@...nel.org, kernel@...0n.name, geert@...ux-m68k.org,
        sammy@...my.net, monstr@...str.eu, tsbogend@...ha.franken.de,
        dinguyen@...nel.org, jonas@...thpole.se,
        stefan.kristiansson@...nalahti.fi, shorne@...il.com,
        James.Bottomley@...senpartnership.com, deller@....de,
        mpe@...erman.id.au, benh@...nel.crashing.org, paulus@...ba.org,
        paul.walmsley@...ive.com, palmer@...belt.com,
        aou@...s.berkeley.edu, hca@...ux.ibm.com, gor@...ux.ibm.com,
        agordeev@...ux.ibm.com, borntraeger@...ux.ibm.com,
        svens@...ux.ibm.com, ysato@...rs.sourceforge.jp, dalias@...c.org,
        davem@...emloft.net, richard@....at,
        anton.ivanov@...bridgegreys.com, johannes@...solutions.net,
        tglx@...utronix.de, mingo@...hat.com, bp@...en8.de,
        dave.hansen@...ux.intel.com, x86@...nel.org, hpa@...or.com,
        acme@...nel.org, alexander.shishkin@...ux.intel.com,
        jolsa@...nel.org, namhyung@...nel.org, jgross@...e.com,
        srivatsa@...il.mit.edu, amakhalov@...are.com,
        pv-drivers@...are.com, boris.ostrovsky@...cle.com,
        chris@...kel.net, jcmvbkbc@...il.com, rafael@...nel.org,
        lenb@...nel.org, pavel@....cz, gregkh@...uxfoundation.org,
        mturquette@...libre.com, sboyd@...nel.org,
        daniel.lezcano@...aro.org, lpieralisi@...nel.org,
        sudeep.holla@....com, agross@...nel.org,
        bjorn.andersson@...aro.org, anup@...infault.org,
        thierry.reding@...il.com, jonathanh@...dia.com,
        jacob.jun.pan@...ux.intel.com, Arnd Bergmann <arnd@...db.de>,
        yury.norov@...il.com, andriy.shevchenko@...ux.intel.com,
        linux@...musvillemoes.dk, rostedt@...dmis.org, pmladek@...e.com,
        senozhatsky@...omium.org, john.ogness@...utronix.de,
        paulmck@...nel.org, frederic@...nel.org, quic_neeraju@...cinc.com,
        josh@...htriplett.org, mathieu.desnoyers@...icios.com,
        jiangshanlai@...il.com, joel@...lfernandes.org,
        juri.lelli@...hat.com, vincent.guittot@...aro.org,
        dietmar.eggemann@....com, bsegall@...gle.com, mgorman@...e.de,
        bristot@...hat.com, vschneid@...hat.com, jpoimboe@...nel.org,
        linux-alpha@...r.kernel.org, linux-kernel@...r.kernel.org,
        linux-snps-arc@...ts.infradead.org,
        linux-arm-kernel@...ts.infradead.org, linux-omap@...r.kernel.org,
        linux-csky@...r.kernel.org, linux-hexagon@...r.kernel.org,
        linux-ia64@...r.kernel.org, linux-m68k@...ts.linux-m68k.org,
        linux-mips@...r.kernel.org, openrisc@...ts.librecores.org,
        linux-parisc@...r.kernel.org, linuxppc-dev@...ts.ozlabs.org,
        linux-riscv@...ts.infradead.org, linux-s390@...r.kernel.org,
        linux-sh@...r.kernel.org, sparclinux@...r.kernel.org,
        linux-um@...ts.infradead.org, linux-perf-users@...r.kernel.org,
        virtualization@...ts.linux-foundation.org,
        xen-devel@...ts.xenproject.org, linux-xtensa@...ux-xtensa.org,
        linux-acpi@...r.kernel.org, linux-pm@...r.kernel.org,
        linux-clk@...r.kernel.org, linux-arm-msm@...r.kernel.org,
        linux-tegra@...r.kernel.org, linux-arch@...r.kernel.org,
        rcu@...r.kernel.org
Subject: Re: [PATCH 00/36] cpuidle,rcu: Cleanup the mess

On Tue, Jun 14, 2022 at 06:58:30PM +0200, Peter Zijlstra wrote:
> On Tue, Jun 14, 2022 at 12:19:29PM +0100, Mark Rutland wrote:
> > On Wed, Jun 08, 2022 at 04:27:23PM +0200, Peter Zijlstra wrote:
> > > Hi All! (omg so many)
> > 
> > Hi Peter,
> > 
> > Sorry for the delay; my plate has also been rather full recently. I'm beginning
> > to page this in now.
> 
> No worries; we all have too much to do ;-)
> 
> > > These here few patches mostly clear out the utter mess that is cpuidle vs rcuidle.
> > > 
> > > At the end of the ride there's only 2 real RCU_NONIDLE() users left
> > > 
> > >   arch/arm64/kernel/suspend.c:            RCU_NONIDLE(__cpu_suspend_exit());
> > >   drivers/perf/arm_pmu.c:                 RCU_NONIDLE(armpmu_start(event, PERF_EF_RELOAD));
> > 
> > The latter of these is necessary because apparently PM notifiers are called
> > with RCU not watching. Is that still the case today (or at the end of this
> > series)? If so, that feels like fertile land for more issues (yaey...). If not,
> > we should be able to drop this.
> 
> That should be fixed; fingers crossed :-)

Cool; I'll try to give that a spin when I'm sat next to some relevant hardware. :)

> > >   kernel/cfi.c:   RCU_NONIDLE({
> > > 
> > > (the CFI one is likely dead in the kCFI rewrite) and there's only a hand full
> > > of trace_.*_rcuidle() left:
> > > 
> > >   kernel/trace/trace_preemptirq.c:                        trace_irq_enable_rcuidle(CALLER_ADDR0, CALLER_ADDR1);
> > >   kernel/trace/trace_preemptirq.c:                        trace_irq_disable_rcuidle(CALLER_ADDR0, CALLER_ADDR1);
> > >   kernel/trace/trace_preemptirq.c:                        trace_irq_enable_rcuidle(CALLER_ADDR0, caller_addr);
> > >   kernel/trace/trace_preemptirq.c:                        trace_irq_disable_rcuidle(CALLER_ADDR0, caller_addr);
> > >   kernel/trace/trace_preemptirq.c:                trace_preempt_enable_rcuidle(a0, a1);
> > >   kernel/trace/trace_preemptirq.c:                trace_preempt_disable_rcuidle(a0, a1);
> > > 
> > > All of them are in 'deprecated' code that is unused for GENERIC_ENTRY.
> > I think those are also unused on arm64 too?
> > 
> > If not, I can go attack that.
> 
> My grep spots:
> 
> arch/arm64/kernel/entry-common.c:               trace_hardirqs_on();
> arch/arm64/include/asm/daifflags.h:     trace_hardirqs_off();
> arch/arm64/include/asm/daifflags.h:             trace_hardirqs_off();

Ah; I hadn't realised those used trace_.*_rcuidle() behind the scenes.

That affects local_irq_{enable,disable,restore}() too (which is what the
daifflags.h bits are emulating), and also the generic entry code's
irqentry_exit().

So it feels to me like we should be fixing those more generally? e.g. say that
with a new STRICT_ENTRY[_RCU], we can only call trace_hardirqs_{on,off}() with
RCU watching, and alter the definition of those?

> The _on thing should be replaced with something like:
> 
> 	trace_hardirqs_on_prepare();
> 	lockdep_hardirqs_on_prepare();
> 	instrumentation_end();
> 	rcu_irq_exit();
> 	lockdep_hardirqs_on(CALLER_ADDR0);
> 
> (as I think you know, since you have some of that already). And
> something similar for the _off thing, but with _off_finish().

Sure; I knew that was necessary for the outermost parts of entry (and I think
that's all handled), I just hadn't realised that trace_hardirqs_{on,off} did
the rcuidle thing in the middle.

It'd be nice to not have to open-code the whole sequence everywhere for the
portions which run after entry and are instrumentable, so (as above) I reckon
we want to make trace_hardirqs_{on,off}() not do the rcuidle part
unnecessarily (which IIUC is an end-goal anyway)?

> > > I've touched a _lot_ of code that I can't test and likely broken some of it :/
> > > In particular, the whole ARM cpuidle stuff was quite involved with OMAP being
> > > the absolute 'winner'.
> > > 
> > > I'm hoping Mark can help me sort the remaining ARM64 bits as he moves that to
> > > GENERIC_ENTRY.
> > 
> > Moving to GENERIC_ENTRY as a whole is going to take a tonne of work
> > (refactoring both arm64 and the generic portion to be more amenable to each
> > other), but we can certainly move closer to that for the bits that matter here.
> 
> I know ... been there etc.. :-)
> 
> > Maybe we want a STRICT_ENTRY option to get rid of all the deprecated stuff that
> > we can select regardless of GENERIC_ENTRY to make that easier.
> 
> Possible yeah.
> 
> > > I've also got a note that says ARM64 can probably do a WFE based
> > > idle state and employ TIF_POLLING_NRFLAG to avoid some IPIs.
> > 
> > Possibly; I'm not sure how much of a win that'll be given that by default we'll
> > have a ~10KHz WFE wakeup from the timer, but we could take a peek.
> 
> Ohh.. I didn't know it woke up *that* often. I just know Will made use
> of it in things like smp_cond_load_relaxed() which would be somewhat
> similar to a very shallow idle state that looks at the TIF word.

We'll get some saving, I'm just not sure where that falls on the curve of idle
states. FWIW the wakeup *can* be disabled (and it'd be nice to when we have
WFxT instructions which take a timeout), it jsut happens to be on by default
for reasons.

Thanks,
Mark.

Powered by blists - more mailing lists