Message-ID: <CAFA6WYOU8HW2JVBfCeFEkn-5cd81TM-x=ArUKeaSi3NzxgKaGQ@mail.gmail.com>
Date:   Tue, 16 May 2023 15:39:21 +0530
From:   Sumit Garg <sumit.garg@...aro.org>
To:     Doug Anderson <dianders@...omium.org>
Cc:     Mark Rutland <mark.rutland@....com>,
        Catalin Marinas <catalin.marinas@....com>,
        Will Deacon <will@...nel.org>,
        Daniel Thompson <daniel.thompson@...aro.org>,
        Marc Zyngier <maz@...nel.org>, ito-yuichi@...itsu.com,
        kgdb-bugreport@...ts.sourceforge.net, Chen-Yu Tsai <wens@...e.org>,
        Masayoshi Mizuma <msys.mizuma@...il.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Ard Biesheuvel <ardb@...nel.org>,
        "Rafael J . Wysocki" <rafael.j.wysocki@...el.com>,
        linux-arm-kernel@...ts.infradead.org,
        Stephen Boyd <swboyd@...omium.org>,
        Lecopzer Chen <lecopzer.chen@...iatek.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        linux-perf-users@...r.kernel.org,
        Alexandru Elisei <alexandru.elisei@....com>,
        Andrey Konovalov <andreyknvl@...il.com>,
        Ben Dooks <ben-linux@...ff.org>,
        Borislav Petkov <bp@...en8.de>,
        Christophe Leroy <christophe.leroy@...roup.eu>,
        "Darrick J. Wong" <djwong@...nel.org>,
        Dave Hansen <dave.hansen@...ux.intel.com>,
        "David S. Miller" <davem@...emloft.net>,
        "Eric W. Biederman" <ebiederm@...ssion.com>,
        Frederic Weisbecker <frederic@...nel.org>,
        Gaosheng Cui <cuigaosheng1@...wei.com>,
        "Gautham R. Shenoy" <gautham.shenoy@....com>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        "Guilherme G. Piccoli" <gpiccoli@...lia.com>,
        Guo Ren <guoren@...nel.org>, "H. Peter Anvin" <hpa@...or.com>,
        Huacai Chen <chenhuacai@...nel.org>,
        Ingo Molnar <mingo@...nel.org>, Ingo Molnar <mingo@...hat.com>,
        "Jason A. Donenfeld" <Jason@...c4.com>,
        Jason Wessel <jason.wessel@...driver.com>,
        Jianmin Lv <lvjianmin@...ngson.cn>,
        Jiaxun Yang <jiaxun.yang@...goat.com>,
        Jinyang He <hejinyang@...ngson.cn>,
        Joey Gouly <joey.gouly@....com>,
        Kees Cook <keescook@...omium.org>,
        Laurent Dufour <ldufour@...ux.ibm.com>,
        Masahiro Yamada <masahiroy@...nel.org>,
        Masayoshi Mizuma <m.mizuma@...fujitsu.com>,
        Michael Ellerman <mpe@...erman.id.au>,
        Nicholas Piggin <npiggin@...il.com>,
        "Paul E. McKenney" <paulmck@...nel.org>,
        Philippe Mathieu-Daudé <f4bug@...at.org>,
        Pierre Gondois <Pierre.Gondois@....com>,
        Qing Zhang <zhangqing@...ngson.cn>,
        "Russell King (Oracle)" <rmk+kernel@...linux.org.uk>,
        Russell King <linux@...linux.org.uk>,
        Thomas Bogendoerfer <tsbogend@...ha.franken.de>,
        Ulf Hansson <ulf.hansson@...aro.org>,
        WANG Xuerui <kernel@...0n.name>, linux-kernel@...r.kernel.org,
        linux-mips@...r.kernel.org, linuxppc-dev@...ts.ozlabs.org,
        loongarch@...ts.linux.dev, sparclinux@...r.kernel.org,
        x86@...nel.org
Subject: Re: [PATCH v8 00/10] arm64: Add framework to turn an IPI as NMI

On Wed, 10 May 2023 at 22:20, Doug Anderson <dianders@...omium.org> wrote:
>
> Hi,
>
> On Wed, May 10, 2023 at 9:30 AM Mark Rutland <mark.rutland@....com> wrote:
> >
> > On Wed, May 10, 2023 at 08:28:17AM -0700, Doug Anderson wrote:
> > > Hi,
> >
> > Hi Doug,
> >
> > > On Wed, Apr 19, 2023 at 3:57 PM Douglas Anderson <dianders@...omium.org> wrote:
> > > > This is an attempt to resurrect Sumit's old patch series [1] that
> > > > allowed us to use the arm64 pseudo-NMI to get backtraces of CPUs and
> > > > also to round up CPUs in kdb/kgdb. The last post from Sumit that I
> > > > could find was v7, so I called this series v8. I haven't copied all of
> > > > his old changelongs here, but you can find them from the link.
> > > >

Thanks Doug for picking up this work and for all your additions/improvements.

> > > > Since v7, I have:
> > > > * Addressed the small amount of feedback that was there for v7.
> > > > * Rebased.
> > > > * Added a new patch that prevents us from spamming the logs with idle
> > > >   tasks.
> > > > * Added an extra patch to gracefully fall back to regular IPIs if
> > > >   pseudo-NMIs aren't there.
> > > >
> > > > Since there appear to be a few different patches series related to
> > > > being able to use NMIs to get stack traces of crashed systems, let me
> > > > try to organize them to the best of my understanding:
> > > >
> > > > a) This series. On its own, a) will (among other things) enable stack
> > > >    traces of all running processes with the soft lockup detector if
> > > >    you've enabled the sysctl "kernel.softlockup_all_cpu_backtrace". On
> > > >    its own, a) doesn't give a hard lockup detector.
> > > >
> > > > b) A different recently-posted series [2] that adds a hard lockup
> > > >    detector based on perf. On its own, b) gives a stack crawl of the
> > > >    locked up CPU but no stack crawls of other CPUs (even if they're
> > > >    locked too). Together with a) + b) we get everything (full lockup
> > > >    detect, full ability to get stack crawls).
> > > >
> > > > c) The old Android "buddy" hard lockup detector [3] that I'm
> > > >    considering trying to upstream. If b) lands then I believe c) would
> > > >    be redundant (at least for arm64). c) on its own is really only
> > > >    useful on arm64 for platforms that can print CPU_DBGPCSR somehow
> > > >    (see [4]). a) + c) is roughly as good as a) + b).
> >
> > > It's been 3 weeks and I haven't heard a peep on this series. That
> > > means nobody has any objections and it's all good to land, right?
> > > Right? :-P

For me it was months waiting without any feedback. So I think you are
lucky :) or atleast better than me at poking arm64 maintainers.

> >
> > FWIW, there are still longstanding soundness issues in the arm64 pseudo-NMI
> > support (and fixing that requires an overhaul of our DAIF / IRQ flag
> > management, which I've been chipping away at for a number of releases), so I
> > hadn't looked at this in detail yet because the foundations are still somewhat
> > dodgy.
> >
> > I appreciate that this has been around for a while, and it's on my queue to
> > look at.
>
> Ah, thanks for the heads up! We've been thinking about turning this on
> in production in ChromeOS because it will help us track down a whole
> class of field-generated crash reports that are otherwise opaque to
> us. It sounds as if maybe that's not a good idea quite yet? Do you
> have any idea of how much farther along this needs to go? ...of
> course, we've also run into issues with Mediatek devices because they
> don't save/restore GICR registers properly [1]. In theory, we might be
> able to work around that in the kernel.
>
> In any case, even if there are bugs that would prevent turning this on
> for production, it still seems like we could still land this series.
> It simply wouldn't do anything until someone turned on pseudo NMIs,
> which wouldn't happen till the kinks are worked out.

I agree here. We should be able to make the foundations robust later
on. IMHO, until we turn on features surrounding pseudo NMIs, I am not
sure how we can have true confidence in the underlying robustness.

-Sumit

>
> ...actually, I guess I should say that if all the patches of the
> current series do land then it actually _would_ still do something,
> even without pseudo-NMI. Assuming the last patch looks OK, it would at
> least start falling back to using regular IPIs to do backtraces. That
> wouldn't get backtraces on hard locked up CPUs but it would be better
> than what we have today where we don't get any backtraces. This would
> get arm64 on par with arm32...
>
> [1] https://issuetracker.google.com/281831288

Powered by blists - more mailing lists