Message-Id: <20251030-mte-tighten-tco-v1-2-88c92e7529d9@os.amperecomputing.com>
Date: Thu, 30 Oct 2025 20:49:32 -0700
From: Carl Worth <carl@...amperecomputing.com>
To: Catalin Marinas <catalin.marinas@....com>, 
 Will Deacon <will@...nel.org>
Cc: linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org, 
 Taehyun Noh <taehyun@...xas.edu>, Carl Worth <carl@...amperecomputing.com>
Subject: [PATCH 2/2] arm64: mte: Defer disabling of TCO until
 user_access_begin/end

The PSTATE.TCO (Tag Checking Override) register, when set causes MTE
tag checking to be disabled. The TCO bit is automatically set by the
hardware when an exception is taken.

Prior to this commit, mte_disable_tco_entry would clear TCO (enable
tag checking) for either of two cases: 1. When the kernel wants tag
checking (KASAN) or 2. when userspace wants tag checking (via
SCTLR.TCF0).

In the case of userspace desired tag checking, (that is, when KASAN is
off), clearing TCO on entry to the kernel has negative performance
implications. This results in excess kernel space tag checking that
has not been requested.

For this case, move the clearing of TCO to user_space_access_begin,
and set it again in user_access_end. This restricts the tag checking
to only the duration of the userspace accesses as desired.

This patch has been measured to eliminate over 97% of kernel-side tag
checking during "perf bench futex hash"

Reported-by: Taehyun Noh <taehyun@...xas.edu>
Signed-off-by: Carl Worth <carl@...amperecomputing.com>
---
 arch/arm64/include/asm/mte.h     | 21 +++++++++++++--------
 arch/arm64/include/asm/uaccess.h | 32 +++++++++++++++++++++++++++++++-
 2 files changed, 44 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
index 70dabc884616..3608ba452da5 100644
--- a/arch/arm64/include/asm/mte.h
+++ b/arch/arm64/include/asm/mte.h
@@ -258,15 +258,20 @@ static inline void set_kernel_mte_policy(struct task_struct *task)
 		return;
 
 	/*
-	 * Re-enable tag checking (TCO set on exception entry). This is only
-	 * necessary if MTE is enabled in either the kernel or the userspace
-	 * task. With MTE disabled in the kernel and disabled or asynchronous
-	 * in userspace, tag check faults (including in uaccesses) are not
-	 * reported, therefore there is no need to re-enable checking.
-	 * This is beneficial on microarchitectures where re-enabling TCO is
-	 * expensive.
+	 * TCO is set on exception entry, (which overrides either of TCF
+	 * or TCF0 and disables tag checking).
+	 *
+	 * If KASAN is enabled and using MTE/(aka "hw_tags") we clear
+	 * TCO so that the kernel gets the tag-checking it needs for
+	 * KASAN_HW_TAGS.
+	 *
+	 * When the kernel needs to enable tag-checking temporarily,
+	 * (such as before accessing userspace memory in the case that
+	 * userspace has requested tag checking), the kernel can
+	 * temporarily change the state of TCO. See
+	 * user_access_begin().
 	 */
-	if (kasan_hw_tags_enabled() || user_uses_tagcheck())
+	if (kasan_hw_tags_enabled())
 		asm volatile(SET_PSTATE_TCO(0));
 }
 
diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
index 1aa4ecb73429..248741a66c91 100644
--- a/arch/arm64/include/asm/uaccess.h
+++ b/arch/arm64/include/asm/uaccess.h
@@ -417,11 +417,41 @@ static __must_check __always_inline bool user_access_begin(const void __user *pt
 {
 	if (unlikely(!access_ok(ptr,len)))
 		return 0;
+
+	/*
+	 * Enable tag checking for the user access if MTE is enabled
+	 * in the userspace task.
+	 *
+	 * Note: We don't need to do anything if KASAN is enabled,
+	 * since that means the tag checking override (TCO) will
+	 * already be disabled. In turn, the TCF0 bits will control
+	 * whether user-space tag checking happens .
+	 */
+	if (!kasan_hw_tags_enabled() && user_uses_tagcheck())
+		asm volatile(SET_PSTATE_TCO(0));
+
 	uaccess_ttbr0_enable();
 	return 1;
 }
+
+static __always_inline void user_access_end(void)
+{
+	/*
+	 * Restore TCO to disable tag checking now that user access is done.
+	 *
+	 * This logic uses the identical condition as in user_access_begin
+	 * to avoid writing PSTATE.TCO with a value identical to what it
+	 * already has (which would needlessly introduce a pipeline flush
+	 * and could impact performance).
+	 */
+	if (!kasan_hw_tags_enabled() && user_uses_tagcheck())
+		asm volatile(SET_PSTATE_TCO(1));
+
+	uaccess_ttbr0_disable();
+}
+
 #define user_access_begin(a,b)	user_access_begin(a,b)
-#define user_access_end()	uaccess_ttbr0_disable()
+#define user_access_end()	user_access_end()
 #define unsafe_put_user(x, ptr, label) \
 	__raw_put_mem("sttr", x, uaccess_mask_ptr(ptr), label, U)
 #define unsafe_get_user(x, ptr, label) \

-- 
2.39.5

Powered by blists - more mailing lists