Message-ID: <xhsmhr0179w1i.mognet@vschneid-thinkpadt14sgen2i.remote.csb>
Date: Fri, 02 May 2025 11:55:21 +0200
From: Valentin Schneider <vschneid@...hat.com>
To: Dave Hansen <dave.hansen@...el.com>, Steven Rostedt <rostedt@...dmis.org>
Cc: linux-kernel@...r.kernel.org, virtualization@...ts.linux.dev,
 linux-arm-kernel@...ts.infradead.org, loongarch@...ts.linux.dev,
 linux-riscv@...ts.infradead.org, linux-perf-users@...r.kernel.org,
 kvm@...r.kernel.org, linux-arch@...r.kernel.org,
 linux-modules@...r.kernel.org, linux-trace-kernel@...r.kernel.org,
 rcu@...r.kernel.org, linux-hardening@...r.kernel.org,
 linux-kselftest@...r.kernel.org, bpf@...r.kernel.org, Juri Lelli
 <juri.lelli@...hat.com>, Marcelo Tosatti <mtosatti@...hat.com>, Yair
 Podemsky <ypodemsk@...hat.com>, Josh Poimboeuf <jpoimboe@...nel.org>,
 Daniel Wagner <dwagner@...e.de>, Petr Tesarik <ptesarik@...e.com>, Nicolas
 Saenz Julienne <nsaenz@...zon.com>, Frederic Weisbecker
 <frederic@...nel.org>, "Paul E. McKenney" <paulmck@...nel.org>, Dave
 Hansen <dave.hansen@...ux.intel.com>, Sean Christopherson
 <seanjc@...gle.com>, Juergen Gross <jgross@...e.com>, Ajay Kaher
 <ajay.kaher@...adcom.com>, Alexey Makhalov
 <alexey.amakhalov@...adcom.com>, Broadcom internal kernel review list
 <bcm-kernel-feedback-list@...adcom.com>, Russell King
 <linux@...linux.org.uk>, Catalin Marinas <catalin.marinas@....com>, Will
 Deacon <will@...nel.org>, Huacai Chen <chenhuacai@...nel.org>, WANG Xuerui
 <kernel@...0n.name>, Paul Walmsley <paul.walmsley@...ive.com>, Palmer
 Dabbelt <palmer@...belt.com>, Albert Ou <aou@...s.berkeley.edu>, Alexandre
 Ghiti <alex@...ti.fr>, Thomas Gleixner <tglx@...utronix.de>, Ingo Molnar
 <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>, x86@...nel.org, "H.
 Peter Anvin" <hpa@...or.com>, Peter Zijlstra <peterz@...radead.org>,
 Arnaldo Carvalho de Melo <acme@...nel.org>, Namhyung Kim
 <namhyung@...nel.org>, Mark Rutland <mark.rutland@....com>, Alexander
 Shishkin <alexander.shishkin@...ux.intel.com>, Jiri Olsa
 <jolsa@...nel.org>, Ian Rogers <irogers@...gle.com>, Adrian Hunter
 <adrian.hunter@...el.com>, "Liang, Kan" <kan.liang@...ux.intel.com>, Pawan
 Gupta <pawan.kumar.gupta@...ux.intel.com>, Paolo Bonzini
 <pbonzini@...hat.com>, Arnd Bergmann <arnd@...db.de>, Jason Baron
 <jbaron@...mai.com>, Ard Biesheuvel <ardb@...nel.org>, Luis Chamberlain
 <mcgrof@...nel.org>, Petr Pavlu <petr.pavlu@...e.com>, Sami Tolvanen
 <samitolvanen@...gle.com>, Daniel Gomez <da.gomez@...sung.com>, Naveen N
 Rao <naveen@...nel.org>, Anil S Keshavamurthy
 <anil.s.keshavamurthy@...el.com>, "David S. Miller" <davem@...emloft.net>,
 Masami Hiramatsu <mhiramat@...nel.org>, Neeraj Upadhyay
 <neeraj.upadhyay@...nel.org>, Joel Fernandes <joel@...lfernandes.org>,
 Josh Triplett <josh@...htriplett.org>, Boqun Feng <boqun.feng@...il.com>,
 Uladzislau Rezki <urezki@...il.com>, Mathieu Desnoyers
 <mathieu.desnoyers@...icios.com>, Lai Jiangshan <jiangshanlai@...il.com>,
 Zqiang <qiang.zhang1211@...il.com>, Vincent Guittot
 <vincent.guittot@...aro.org>, Dietmar Eggemann <dietmar.eggemann@....com>,
 Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>, Kees Cook
 <kees@...nel.org>, Shuah Khan <shuah@...nel.org>, Masahiro Yamada
 <masahiroy@...nel.org>, Alice Ryhl <aliceryhl@...gle.com>, Miguel Ojeda
 <ojeda@...nel.org>, "Mike Rapoport (Microsoft)" <rppt@...nel.org>, Rong Xu
 <xur@...gle.com>, Rafael Aquini <aquini@...hat.com>, Song Liu
 <song@...nel.org>, Andrii Nakryiko <andrii@...nel.org>, Dan Carpenter
 <dan.carpenter@...aro.org>, Brian Gerst <brgerst@...il.com>, "Kirill A.
 Shutemov" <kirill.shutemov@...ux.intel.com>, Benjamin Berg
 <benjamin.berg@...el.com>, Vishal Annapurve <vannapurve@...gle.com>, Randy
 Dunlap <rdunlap@...radead.org>, John Stultz <jstultz@...gle.com>, Tiezhu
 Yang <yangtiezhu@...ngson.cn>
Subject: Re: [PATCH v5 00/25] context_tracking,x86: Defer some IPIs until a
 user->kernel transition

On 30/04/25 13:00, Dave Hansen wrote:
> On 4/30/25 12:42, Steven Rostedt wrote:
>>> Look at the syscall code for instance:
>>>
>>>> SYM_CODE_START(entry_SYSCALL_64)
>>>>         swapgs
>>>>         movq    %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
>>>>         SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
>>> You can _trivially_ audit this and know that swapgs doesn't touch memory
>>> and that as long as PER_CPU_VAR()s and the process stack don't have
>>> their mappings munged and flushes deferred that this would be correct.
>> Hmm, so there is still a path for this?
>>
>> At least if it added more ways to debug it, and some other changes to make
>> the locations where vmalloc is dangerous smaller?
>
> Being able to debug it would be a good start. But, more generally, what
> we need is for more people to be able to run the code in the first
> place. Would a _normal_ system (without setups that are trying to do
> NOHZ_FULL) ever be able to defer TLB flush IPIs?
>
> If the answer is no, then, yeah, I'll settle for some debugging options.
>
> But if you shrink the window as small as I'm talking about, it would
> look very different from this series.
>
> For instance, imagine when a CPU goes into the NOHZ mode. Could it just
> unconditionally flush the TLB on the way back into the kernel (in the
> same SWITCH_TO_KERNEL_CR3 spot)? Yeah, it'll make entry into the kernel
> expensive for NOHZ tasks, but it's not *THAT* bad. And if the entire
> point of a NOHZ_FULL task is to minimize the number of kernel entries
> then a little extra overhead there doesn't sound too bad.
>

Right, so my thought per your previous comments was to special case the
TLB flush, depend on kPTI and do it uncondtionally in SWITCH_TO_KERNEL_CR3
just like you've described - but keep the context tracking mechanism for
other deferrable operations.

My gripe with that was having two separate mechanisms
- super early entry around SWITCH_TO_KERNEL_CR3)
- later entry at context tracking

Shifting everything to SWITCH_TO_KERNEL_CR3 means we lose the
context_tracking infra to dynamically defer operations (atomically reading
and writing to context_tracking.state), which means we unconditionally run
all possible deferrable operations. This doesn't scream scalable, even
though as you say NOHZ_FULL kernel entry is already a "you lose" situation.

Yet another option is to duplicate the context tracking state specifically
for IPI deferral and have it driven in/by SWITCH_TO_KERNEL_CR3, which is
also not super savoury.

I suppose I can start poking around running deferred ops in that
SWITCH_TO_KERNEL_CR3 region, and add state/infra on top. Let's see where
this gets me :-)

Again, thanks for the insight and the suggestions Dave!

> Also, about the new hardware, I suspect there's some mystery customer
> lurking in the shadows asking folks for this functionality. Could you at
> least go _talk_ to the mystery customer(s) and see which hardware they
> care about? They might already even have the magic CPUs they need for
> this, or have them on the roadmap. If they've got Intel CPUs, I'd be
> happy to help figure it out.

Powered by blists - more mailing lists