[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <20250627044805.945491-6-ankur.a.arora@oracle.com>
Date: Thu, 26 Jun 2025 21:48:05 -0700
From: Ankur Arora <ankur.a.arora@...cle.com>
To: linux-kernel@...r.kernel.org, linux-arch@...r.kernel.org,
linux-arm-kernel@...ts.infradead.org, bpf@...r.kernel.org
Cc: arnd@...db.de, catalin.marinas@....com, will@...nel.org,
peterz@...radead.org, akpm@...ux-foundation.org, mark.rutland@....com,
harisokn@...zon.com, cl@...two.org, ast@...nel.org, memxor@...il.com,
zhenglifeng1@...wei.com, xueshuai@...ux.alibaba.com,
joao.m.martins@...cle.com, boris.ostrovsky@...cle.com,
konrad.wilk@...cle.com
Subject: [PATCH v3 5/5] arm64: barrier: Handle waiting in smp_cond_load_relaxed_timewait()
smp_cond_load_{relaxed,acquire}_timewait() wait on a condition variable
until a timeout expires. This waiting is some mix of spinning while
dereferencing an address, and waiting in a WFE for a store event or
periodic events from the event-stream.
Handle the waiting part of the policy in ___smp_cond_timewait() while
offloading the spinning to the generic ___smp_cond_spinwait().
To minimize time spent spinning when the user can tolerate a large
overshoot, choose SMP_TIMEWAIT_DEFAULT_US to be the event-stream
period.
This would result in a worst case delay of ARCH_TIMER_EVT_STREAM_PERIOD_US.
Cc: Will Deacon <will@...nel.org>
Cc: Catalin Marinas <catalin.marinas@....com>
Signed-off-by: Ankur Arora <ankur.a.arora@...cle.com>
---
arch/arm64/include/asm/barrier.h | 48 ++++++++++++++++++++++++++++++++
1 file changed, 48 insertions(+)
diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
index 7c56e2621c7d..a1367f2901f0 100644
--- a/arch/arm64/include/asm/barrier.h
+++ b/arch/arm64/include/asm/barrier.h
@@ -10,6 +10,7 @@
#ifndef __ASSEMBLY__
#include <linux/kasan-checks.h>
+#include <linux/minmax.h>
#include <asm/alternative-macros.h>
@@ -222,6 +223,53 @@ do { \
#define __smp_timewait_store(ptr, val) \
__cmpwait_relaxed(ptr, val)
+/*
+ * Redefine ARCH_TIMER_EVT_STREAM_PERIOD_US locally to avoid include hell.
+ */
+#define __ARCH_TIMER_EVT_STREAM_PERIOD_US 100UL
+extern bool arch_timer_evtstrm_available(void);
+
+static inline u64 ___smp_cond_spinwait(u64 now, u64 prev, u64 end,
+ u32 *spin, bool *wait, u64 slack);
+/*
+ * To minimize time spent spinning, we want to allow a large overshoot.
+ * So, choose a default slack value of the event-stream period.
+ */
+#define SMP_TIMEWAIT_DEFAULT_US __ARCH_TIMER_EVT_STREAM_PERIOD_US
+
+static inline u64 ___smp_cond_timewait(u64 now, u64 prev, u64 end,
+ u32 *spin, bool *wait, u64 slack)
+{
+ bool wfet = alternative_has_cap_unlikely(ARM64_HAS_WFXT);
+ bool wfe, ev = arch_timer_evtstrm_available();
+ u64 evt_period = __ARCH_TIMER_EVT_STREAM_PERIOD_US;
+ u64 remaining = end - now;
+
+ if (now >= end)
+ return 0;
+ /*
+ * Use WFE if there's enough slack to get an event-stream wakeup even
+ * if we don't come out of the WFE due to natural causes.
+ */
+ wfe = ev && ((remaining + slack) > evt_period);
+
+ if (wfe || wfet) {
+ *wait = true;
+ *spin = 0;
+ return now;
+ }
+
+ /*
+ * The time remaining is shorter than our wait granularity. Let
+ * the generic spinwait policy determine how to spin.
+ */
+ return ___smp_cond_spinwait(now, prev, end, spin, wait, slack);
+}
+
+#ifndef __smp_cond_policy
+#define __smp_cond_policy ___smp_cond_timewait
+#endif
+
#include <asm-generic/barrier.h>
#endif /* __ASSEMBLY__ */
--
2.43.5
Powered by blists - more mailing lists