linux-kernel - [PATCH 3/3] group_cpus: optimize grp_spread_init

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20250910210850.404834-4-yury.norov@gmail.com>
Date: Wed, 10 Sep 2025 17:08:48 -0400
From: Yury Norov <yury.norov@...il.com>
To: Thomas Gleixner <tglx@...utronix.de>,
	Yury Norov <yury.norov@...il.com>,
	Rasmus Villemoes <linux@...musvillemoes.dk>,
	linux-kernel@...r.kernel.org
Subject: [PATCH 3/3] group_cpus: optimize grp_spread_init_one()

From: "Yury Norov (NVIDIA)" <yury.norov@...il.com>

Optimizations for grp_spread_init_one():

1. Drop most of housekeeping code in grp_spread_init_one() by using
   for_each_cpu_and_andnot_from().

2. Fix Shlemiel the Painter's algorithm by adding 'sibl = cpu' line. This
   improves the outer loop complexity from quadratic to linear.

3. Don't clear the nmsk because it's ignored in the caller code anyways,
   and switch to non-atomic set_cpu() for irqmsk as the mask is local and
   implies no concurrency.

Signed-off-by: Yury Norov (NVIDIA) <yury.norov@...il.com>
---
 lib/group_cpus.c | 25 ++++++-------------------
 1 file changed, 6 insertions(+), 19 deletions(-)

diff --git a/lib/group_cpus.c b/lib/group_cpus.c
index 6aae1560b796..35aba99d8cd0 100644
--- a/lib/group_cpus.c
+++ b/lib/group_cpus.c
@@ -17,27 +17,14 @@ static void grp_spread_init_one(struct cpumask *irqmsk, struct cpumask *nmsk,
 	const struct cpumask *siblmsk;
 	int cpu, sibl;
 
-	for ( ; cpus_per_grp > 0; ) {
-		cpu = cpumask_first(nmsk);
-
-		/* Should not happen, but I'm too lazy to think about it */
-		if (cpu >= nr_cpu_ids)
-			return;
-
-		cpumask_clear_cpu(cpu, nmsk);
-		cpumask_set_cpu(cpu, irqmsk);
-		cpus_per_grp--;
-
+	for_each_cpu(cpu, nmsk) {
 		/* If the cpu has siblings, use them first */
 		siblmsk = topology_sibling_cpumask(cpu);
-		for (sibl = -1; cpus_per_grp > 0; ) {
-			sibl = cpumask_next(sibl, siblmsk);
-			if (sibl >= nr_cpu_ids)
-				break;
-			if (!cpumask_test_and_clear_cpu(sibl, nmsk))
-				continue;
-			cpumask_set_cpu(sibl, irqmsk);
-			cpus_per_grp--;
+		sibl = cpu;
+		for_each_cpu_and_andnot_from(sibl, nmsk, siblmsk, irqmsk) {
+			__cpumask_set_cpu(sibl, irqmsk);
+			if (--cpus_per_grp)
+				return;
 		}
 	}
 }
-- 
2.43.0