linux-kernel - Re: [External] [PATCH] RISC-V: store percpu offset in CSR

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAEEQ3wmxJ50PZHVpdexeyy1ELqKw+5mrb+8gRCA4KNj9zsrykA@mail.gmail.com>
Date: Thu, 10 Jul 2025 19:47:27 +0800
From: yunhui cui <cuiyunhui@...edance.com>
To: Radim Krčmář <rkrcmar@...tanamicro.com>
Cc: masahiroy@...nel.org, nathan@...nel.org, nicolas.schier@...ux.dev, 
	dennis@...nel.org, tj@...nel.org, cl@...two.org, paul.walmsley@...ive.com, 
	palmer@...belt.com, aou@...s.berkeley.edu, alex@...ti.fr, andybnac@...il.com, 
	bjorn@...osinc.com, cyrilbur@...storrent.com, rostedt@...dmis.org, 
	puranjay@...nel.org, ben.dooks@...ethink.co.uk, zhangchunyan@...as.ac.cn, 
	ruanjinjie@...wei.com, jszhang@...nel.org, charlie@...osinc.com, 
	cleger@...osinc.com, antonb@...storrent.com, ajones@...tanamicro.com, 
	debug@...osinc.com, haibo1.xu@...el.com, samuel.holland@...ive.com, 
	linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org, 
	linux-mm@...ck.org, linux-riscv@...ts.infradead.org, 
	linux-riscv <linux-riscv-bounces@...ts.infradead.org>, wangziang.ok@...edance.com
Subject: Re: [External] [PATCH] RISC-V: store percpu offset in CSR_SCRATCH

Hi Radim,

On Thu, Jul 10, 2025 at 2:35 PM Radim Krčmář <rkrcmar@...tanamicro.com> wrote:
>
> 2025-07-10T11:45:06+08:00, yunhui cui <cuiyunhui@...edance.com>:
> > On Wed, Jul 9, 2025 at 10:20 PM Radim Krčmář <rkrcmar@...tanamicro.com> wrote:
> >> Is the overhead above with this patch?  And when we then use the
> >> CSR_SCRATCH for percpu, does it degrade even further?
> >
> > We can see that the percpu optimization is around 2.5% through the
> > method of fixing registers, and we can consider that the percpu
> > optimization can bring a 2.5% gain. Is there no need to add the percpu
> > optimization logic on the basis of the scratch patch for testing?
> >
> > Reference: https://lists.riscv.org/g/tech-privileged/message/2485
>
> That is when the value is in a GPR, though, and we don't know the
> performance of a CSR_SCRATCH access.
> We can hope that it's not much worse than a GPR, but an implementation
> might choose to be very slow with CSR_SCRATCH.
>
> I have in mind another method where we can use the current CSR_SCRATCH
> without changing CSR_TVAL, but I don't really want to spend time on it
> if reading the CSR doesn't give any benefit.
>
> It would be to store the percpu offset in CSR_SCRATCH permanently, do
> the early exception register shuffling with a percpu area storage, and
> load the thread pointer from there as well.
> That method would also eliminate writing CSR_SCRATCH on every exception
> entry+exit, so maybe it makes sense to try it even if CSRs are slow...
>
> Thanks.


Based on the patch, optimizations for percpu offset have been added,
with the following data:
6.989 7.046 6.976 6.986 7.001 7.017 7.007 7.064 7.008 7.039
Geometric mean: 7.013248303
Compared to reusing the scratch register, the performance has improved
by approximately 0.7%.

If more optimizations can be made to the scratch register, there
should be further performance improvements.

Patch:
---
 arch/riscv/include/asm/percpu.h | 14 ++++++++++++++
 arch/riscv/kernel/asm-offsets.c |  1 +
 arch/riscv/kernel/entry.S       |  7 +++++++
 arch/riscv/kernel/smpboot.c     |  3 +++
 4 files changed, 25 insertions(+)
 create mode 100644 arch/riscv/include/asm/percpu.h

diff --git a/arch/riscv/include/asm/percpu.h b/arch/riscv/include/asm/percpu.h
new file mode 100644
index 000000000000..1fbfcb108f84
--- /dev/null
+++ b/arch/riscv/include/asm/percpu.h
@@ -0,0 +1,14 @@
+#ifndef __ASM_PERCPU_H
+#define __ASM_PERCPU_H
+
+static inline void set_my_cpu_offset(unsigned long off)
+{
+        csr_write(CSR_SCRATCH, off);
+}
+
+#define __my_cpu_offset csr_read(CSR_SCRATCH)
+
+#include <asm-generic/percpu.h>
+
+#endif
+
diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c
index a03129f40c46..0ce96f30bf32 100644
--- a/arch/riscv/kernel/asm-offsets.c
+++ b/arch/riscv/kernel/asm-offsets.c
@@ -35,6 +35,7 @@ void asm_offsets(void)
  OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]);
  OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]);
  OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]);
+ OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu);
  OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags);
  OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count);
  OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp);
diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
index cc2fd4cd54a0..82caeee91c15 100644
--- a/arch/riscv/kernel/entry.S
+++ b/arch/riscv/kernel/entry.S
@@ -75,6 +75,13 @@ SYM_CODE_START_NOALIGN(handle_exception)
  REG_S s4, PT_CAUSE(sp)
  REG_S s5, PT_TP(sp)

+ REG_L s0, TASK_TI_CPU(tp)
+ slli s0, s0, 3
+ la s1, __per_cpu_offset
+ add s1, s1, s0
+ REG_L s1, 0(s1)
+ csrw CSR_SCRATCH, s1
+
  la s1, handle_kernel_exception
  csrw CSR_TVEC, s1

diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
index fb6ab7f8bfbd..6fa12cc84523 100644
--- a/arch/riscv/kernel/smpboot.c
+++ b/arch/riscv/kernel/smpboot.c
@@ -43,6 +43,7 @@ static DECLARE_COMPLETION(cpu_running);

 void __init smp_prepare_boot_cpu(void)
 {
+ set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
 }

 void __init smp_prepare_cpus(unsigned int max_cpus)
@@ -240,6 +241,8 @@ asmlinkage __visible void smp_callin(void)
  mmgrab(mm);
  current->active_mm = mm;

+ set_my_cpu_offset(per_cpu_offset(curr_cpuid));
+
  store_cpu_topology(curr_cpuid);
  notify_cpu_starting(curr_cpuid);

--
2.43.0

Thanks,
Yunhui