linux-kernel - Re: [PATCH v14 05/13] x86/mm: use INVLPGB in flush_tlb

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20250228222032.GHZ8I2sNGMZGTjNPxI@fat_crate.local>
Date: Fri, 28 Feb 2025 23:20:32 +0100
From: Borislav Petkov <bp@...en8.de>
To: Rik van Riel <riel@...riel.com>
Cc: x86@...nel.org, linux-kernel@...r.kernel.org, peterz@...radead.org,
	dave.hansen@...ux.intel.com, zhengqi.arch@...edance.com,
	nadav.amit@...il.com, thomas.lendacky@....com, kernel-team@...a.com,
	linux-mm@...ck.org, akpm@...ux-foundation.org, jackmanb@...gle.com,
	jannh@...gle.com, mhklinux@...look.com, andrew.cooper3@...rix.com,
	Manali.Shukla@....com, mingo@...nel.org
Subject: Re: [PATCH v14 05/13] x86/mm: use INVLPGB in flush_tlb_all

On Tue, Feb 25, 2025 at 10:00:40PM -0500, Rik van Riel wrote:
> The flush_tlb_all() function is not used a whole lot, but we might
> as well use broadcast TLB flushing there, too.
> 
> Signed-off-by: Rik van Riel <riel@...riel.com>
> Tested-by: Manali Shukla <Manali.Shukla@....com>
> Tested-by: Brendan Jackman <jackmanb@...gle.com>
> Tested-by: Michael Kelley <mhklinux@...look.com>
> ---
>  arch/x86/mm/tlb.c | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)

Edits ontop:

--- /tmp/current.patch	2025-02-28 23:18:51.670490799 +0100
+++ /tmp/0001-x86-mm-Use-INVLPGB-in-flush_tlb_all.patch	2025-02-28 23:17:48.590844991 +0100
@@ -1,22 +1,23 @@
+From 5bdf59c0589b71328bd340ea48a00917def62dc0 Mon Sep 17 00:00:00 2001
 From: Rik van Riel <riel@...riel.com>
 Date: Tue, 25 Feb 2025 22:00:40 -0500
-Subject: x86/mm: Use INVLPGB in flush_tlb_all
+Subject: [PATCH] x86/mm: Use INVLPGB in flush_tlb_all()
 
-The flush_tlb_all() function is not used a whole lot, but we might
-as well use broadcast TLB flushing there, too.
+The flush_tlb_all() function is not used a whole lot, but it might as
+well use broadcast TLB flushing there, too.
+
+  [ bp: Massage, restore balanced if-else branches in the function,
+    comment some. ]
 
 Signed-off-by: Rik van Riel <riel@...riel.com>
 Signed-off-by: Borislav Petkov (AMD) <bp@...en8.de>
-Tested-by: Manali Shukla <Manali.Shukla@....com>
-Tested-by: Brendan Jackman <jackmanb@...gle.com>
-Tested-by: Michael Kelley <mhklinux@...look.com>
 Link: https://lore.kernel.org/r/20250226030129.530345-6-riel@surriel.com
 ---
- arch/x86/mm/tlb.c | 10 +++++++++-
- 1 file changed, 9 insertions(+), 1 deletion(-)
+ arch/x86/mm/tlb.c | 17 +++++++++++++++--
+ 1 file changed, 15 insertions(+), 2 deletions(-)
 
 diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
-index f44a03bca41c..a6cd61d5f423 100644
+index 5c44b94ad5af..f49627e02311 100644
 --- a/arch/x86/mm/tlb.c
 +++ b/arch/x86/mm/tlb.c
 @@ -1064,7 +1064,6 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
@@ -27,22 +28,29 @@ index f44a03bca41c..a6cd61d5f423 100644
  static void do_flush_tlb_all(void *info)
  {
  	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED);
-@@ -1074,6 +1073,15 @@ static void do_flush_tlb_all(void *info)
+@@ -1074,7 +1073,21 @@ static void do_flush_tlb_all(void *info)
  void flush_tlb_all(void)
  {
  	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
+-	on_each_cpu(do_flush_tlb_all, NULL, 1);
 +
 +	/* First try (faster) hardware-assisted TLB invalidation. */
 +	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
++		/*
++		 * TLBSYNC at the end needs to make sure all flushes done
++		 * on the current CPU have been executed system-wide.
++		 * Therefore, make sure nothing gets migrated
++		 * in-between but disable preemption as it is cheaper.
++		 */
 +		guard(preempt)();
 +		invlpgb_flush_all();
-+		return;
++	} else {
++		/* Fall back to the IPI-based invalidation. */
++		on_each_cpu(do_flush_tlb_all, NULL, 1);
 +	}
-+
-+	/* Fall back to the IPI-based invalidation. */
- 	on_each_cpu(do_flush_tlb_all, NULL, 1);
  }
  
+ /* Flush an arbitrarily large range of memory with INVLPGB. */
 -- 

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette