linux-kernel - [PATCH] arm64: Add read_mostly declaration/definition to irq stack ptr

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <8798a3cc-6e64-6966-d5ad-fadca79d92ba@synaptics.com>
Date:   Mon, 7 Feb 2022 16:46:42 +0800
From:   Jisheng Zhang <Jisheng.Zhang@...aptics.com>
To:     Catalin Marinas <catalin.marinas@....com>,
        Will Deacon <will@...nel.org>
Cc:     linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org
Subject: [PATCH] arm64: Add read_mostly declaration/definition to irq stack
 ptr

Add "read-mostly" qualifier to irq_stack_ptr and
irq_shadow_call_stack_ptr. This is to prevent the false sharing.

Before the patch, I got below percpu layout with one defconfig:
ffffffc008723050 <mde_ref_count>:
ffffffc008723050:       00 00 00 00
....

ffffffc008723054 <kde_ref_count>:
ffffffc008723054:       00 00 00 00
....

ffffffc008723058 <irq_stack_ptr>:
        ...

ffffffc008723060 <nmi_contexts>:
        ...

ffffffc008723070 <fpsimd_last_state>:

As can be seen, the irq_stack_ptr sits with the heavy read/write percpu
vars such as fpsimd_last_state etc. at the same cacheline.

After the patch:

ffffffc008723000 <irq_stack_ptr>:
        ...

ffffffc008723008 <cpu_number>:
        ...

ffffffc008723010 <arm64_ssbd_callback_required>:
        ...

ffffffc008723018 <bp_hardening_data>:
        ...

Now, the irq_stack_ptr sits with read mostly percpu vars such as
cpu_number etc. at the same cacheline.

Signed-off-by: Jisheng Zhang <Jisheng.Zhang@...aptics.com>
---
 arch/arm64/include/asm/stacktrace.h | 2 +-
 arch/arm64/kernel/irq.c             | 7 +++----
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h
index e77cdef9ca29..75c142bfdffe 100644
--- a/arch/arm64/include/asm/stacktrace.h
+++ b/arch/arm64/include/asm/stacktrace.h
@@ -66,7 +66,7 @@ struct stackframe {
 extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk,
 			   const char *loglvl);
 
-DECLARE_PER_CPU(unsigned long *, irq_stack_ptr);
+DECLARE_PER_CPU_READ_MOSTLY(unsigned long *, irq_stack_ptr);
 
 static inline bool on_stack(unsigned long sp, unsigned long size,
 			    unsigned long low, unsigned long high,
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index bda49430c9ea..d2e75e9bb826 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -26,13 +26,12 @@
 /* Only access this in an NMI enter/exit */
 DEFINE_PER_CPU(struct nmi_ctx, nmi_contexts);
 
-DEFINE_PER_CPU(unsigned long *, irq_stack_ptr);
+DEFINE_PER_CPU_READ_MOSTLY(unsigned long *, irq_stack_ptr);
 
-
-DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr);
+DECLARE_PER_CPU_READ_MOSTLY(unsigned long *, irq_shadow_call_stack_ptr);
 
 #ifdef CONFIG_SHADOW_CALL_STACK
-DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr);
+DEFINE_PER_CPU_READ_MOSTLY(unsigned long *, irq_shadow_call_stack_ptr);
 #endif
 
 static void init_irq_scs(void)
-- 
2.34.1