linux-kernel - [PATCH 05/11] smp: Enable preemption early in smp_call_function_many

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20260203112401.3889029-6-zhouchuyi@bytedance.com>
Date: Tue,  3 Feb 2026 19:23:55 +0800
From: "Chuyi Zhou" <zhouchuyi@...edance.com>
To: <tglx@...utronix.de>, <mingo@...hat.com>, <luto@...nel.org>, 
	<peterz@...radead.org>, <paulmck@...nel.org>, <muchun.song@...ux.dev>, 
	<bp@...en8.de>, <dave.hansen@...ux.intel.com>
Cc: <linux-kernel@...r.kernel.org>, "Chuyi Zhou" <zhouchuyi@...edance.com>
Subject: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

Now smp_call_function_many_cond() disables preemption mainly for the
following reasons:

- To prevent the remote online CPU from going offline. Specifically, we
want to ensure that no new csds are queued after smpcfd_dying_cpu() has
finished. Therefore, preemption must be disabled until all necessary IPIs
are sent.

- To prevent migration to another CPU, which also implicitly prevents the
current CPU from going offline (since stop_machine requires preempting the
current task to execute offline callbacks). This can be achieved equally
using migrate_disable(), as tasks must be migrated to other CPUs before
takedown_cpu().

- To protect the per-cpu cfd_data from concurrent modification by other
smp_call_*() on the current CPU. cfd_data contains cpumasks and per-cpu
csds. Before enqueueing a csd, we block on the csd_lock() to ensure the
previous asyc csd->func() has completed, and then initialize csd->func and
csd->info. After sending the IPI, we spin-wait for the remote CPU to call
csd_unlock(). Actually the csd_lock mechanism already guarantees csd
serialization. If preemption occurs during csd_lock_wait, other concurrent
smp_call_function_many_cond calls will simply block until the previous
csd->func() completes:

task A                    task B

sd->func = fun_a
send ipis

                preempted by B
               --------------->
                        csd_lock(csd); // block until last
                                       // fun_a finished

                        csd->func = func_b;
                        csd->info = info;
                            ...
                        send ipis

                switch back to A
                <---------------

csd_lock_wait(csd); // block until remote finish func_*

This patch use migrate_disable() to protect the scope of
smp_call_function_many_cond() and enables preemption before csd_lock_wait.
This makes the potentially unpredictable csd_lock_wait preemptible. Using
cpumask_stack can avoid concurrency modification issues, and we can
fall back to the default logic if alloc_cpumask_var() fails.

Signed-off-by: Chuyi Zhou <zhouchuyi@...edance.com>
---
 kernel/smp.c | 37 ++++++++++++++++++++++++++++++++-----
 1 file changed, 32 insertions(+), 5 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index 35948afced2e..af9cee7d4939 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -802,7 +802,7 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 					unsigned int scf_flags,
 					smp_cond_func_t cond_func)
 {
-	int cpu, last_cpu, this_cpu = smp_processor_id();
+	int cpu, last_cpu, this_cpu;
 	struct call_function_data *cfd;
 	bool wait = scf_flags & SCF_WAIT;
 	bool preemptible_wait = true;
@@ -811,11 +811,18 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 	int nr_cpus = 0;
 	bool run_remote = false;
 
-	lockdep_assert_preemption_disabled();
-
-	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
+	if (!wait || !alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
 		preemptible_wait = false;
 
+	/*
+	 * Prevent the current CPU from going offline.
+	 * Being migrated to another CPU and calling csd_lock_wait() may cause
+	 * UAF due to smpcfd_dead_cpu() during the current CPU offline process.
+	 */
+	migrate_disable();
+
+	this_cpu = get_cpu();
+
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -898,6 +905,22 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 		local_irq_restore(flags);
 	}
 
+	/*
+	 * We may block in csd_lock_wait() for a significant amount of time, especially
+	 * when interrupts are disabled or with a large number of remote CPUs.
+	 * Try to enable preemption before csd_lock_wait().
+	 *
+	 * - If @wait is true, we try to use the cpumask_stack instead of cfd->cpumask to
+	 * avoid concurrency modification from tasks on the same cpu. If alloc_cpumask_var()
+	 * return false, fallback to the default logic.
+	 *
+	 * - If preemption occurs during csd_lock_wait, other concurrent
+	 * smp_call_function_many_cond() calls will simply block until the previous csd->func()
+	 * complete.
+	 */
+	if (preemptible_wait)
+		put_cpu();
+
 	if (run_remote && wait) {
 		for_each_cpu(cpu, cpumask) {
 			call_single_data_t *csd;
@@ -907,8 +930,12 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 		}
 	}
 
-	if (preemptible_wait)
+	if (!preemptible_wait)
+		put_cpu();
+	else
 		free_cpumask_var(cpumask_stack);
+
+	migrate_enable();
 }
 
 /**
-- 
2.20.1