linux-kernel - Re: [PATCH v2 0/6] futex: Use RCU-based per-CPU reference counting

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20251106092929.GR4067720@noisy.programming.kicks-ass.net>
Date: Thu, 6 Nov 2025 10:29:29 +0100
From: Peter Zijlstra <peterz@...radead.org>
To: Shrikanth Hegde <sshegde@...ux.ibm.com>
Cc: Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
	André Almeida <andrealmeid@...lia.com>,
	Darren Hart <dvhart@...radead.org>,
	Davidlohr Bueso <dave@...olabs.net>, Ingo Molnar <mingo@...hat.com>,
	Juri Lelli <juri.lelli@...hat.com>,
	Thomas Gleixner <tglx@...utronix.de>, linux-kernel@...r.kernel.org,
	Valentin Schneider <vschneid@...hat.com>,
	Waiman Long <longman@...hat.com>
Subject: Re: [PATCH v2 0/6] futex: Use RCU-based per-CPU reference counting

On Wed, Jul 16, 2025 at 11:51:46PM +0530, Shrikanth Hegde wrote:

> > Anyway, I think we can improve both. Does the below help?
> > 
> > 
> > ---
> > diff --git a/kernel/futex/core.c b/kernel/futex/core.c
> > index d9bb5567af0c..8c41d050bd1f 100644
> > --- a/kernel/futex/core.c
> > +++ b/kernel/futex/core.c
> > @@ -1680,10 +1680,10 @@ static bool futex_ref_get(struct futex_private_hash *fph)
> >   {
> >   	struct mm_struct *mm = fph->mm;
> > -	guard(rcu)();
> > +	guard(preempt)();
> > -	if (smp_load_acquire(&fph->state) == FR_PERCPU) {
> > -		this_cpu_inc(*mm->futex_ref);
> > +	if (READ_ONCE(fph->state) == FR_PERCPU) {
> > +		__this_cpu_inc(*mm->futex_ref);
> >   		return true;
> >   	}
> > @@ -1694,10 +1694,10 @@ static bool futex_ref_put(struct futex_private_hash *fph)
> >   {
> >   	struct mm_struct *mm = fph->mm;
> > -	guard(rcu)();
> > +	guard(preempt)();
> > -	if (smp_load_acquire(&fph->state) == FR_PERCPU) {
> > -		this_cpu_dec(*mm->futex_ref);
> > +	if (READ_ONCE(fph->state) == FR_PERCPU) {
> > +		__this_cpu_dec(*mm->futex_ref);
> >   		return false;
> >   	}
> 
> Yes. It helps. It improves "-b 512" numbers by at-least 5%.

While talking with Sebastian about this work, I realized this patch was
never committed. So I've written it up like so, and will commit to
tip/locking/urgent soonish.

---
Subject: futex: Optimize per-cpu reference counting
From: Peter Zijlstra <peterz@...radead.org>
Date: Wed, 16 Jul 2025 16:29:46 +0200

Shrikanth noted that the per-cpu reference counter was still some 10%
slower than the old immutable option (which removes the reference
counting entirely).

Further optimize the per-cpu reference counter by:

 - switching from RCU to preempt;
 - using __this_cpu_*() since we now have preempt disabled;
 - switching from smp_load_acquire() to READ_ONCE().

This is all safe because disabling preemption inhibits the RCU grace
period exactly like rcu_read_lock().

Having preemption disabled allows using __this_cpu_*() provided the
only access to the variable is in task context -- which is the case
here.

Furthermore, since we know changing fph->state to FR_ATOMIC demands a
full RCU grace period we can rely on the implied smp_mb() from that to
replace the acquire barrier().

This is very similar to the percpu_down_read_internal() fast-path.

The reason this is significant for PowerPC is that it uses the generic
this_cpu_*() implementation which relies on local_irq_disable() (the
x86 implementation relies on it being a single memop instruction to be
IRQ-safe). Switching to preempt_disable() and __this_cpu*() avoids
this IRQ state swizzling. Also, PowerPC needs LWSYNC for the ACQUIRE
barrier, not having to use explicit barriers safes a bunch.

Combined this reduces the performance gap by half, down to some 5%.

Reported-by: Shrikanth Hegde <sshegde@...ux.ibm.com>
Tested-by: Shrikanth Hegde <sshegde@...ux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
---
 kernel/futex/core.c |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -1680,10 +1680,10 @@ static bool futex_ref_get(struct futex_p
 {
 	struct mm_struct *mm = fph->mm;

-	guard(rcu)();
+	guard(preempt)();

-	if (smp_load_acquire(&fph->state) == FR_PERCPU) {
-		this_cpu_inc(*mm->futex_ref);
+	if (READ_ONCE(fph->state) == FR_PERCPU) {
+		__this_cpu_inc(*mm->futex_ref);
 		return true;
 	}

@@ -1694,10 +1694,10 @@ static bool futex_ref_put(struct futex_p
 {
 	struct mm_struct *mm = fph->mm;

-	guard(rcu)();
+	guard(preempt)();

-	if (smp_load_acquire(&fph->state) == FR_PERCPU) {
-		this_cpu_dec(*mm->futex_ref);
+	if (READ_ONCE(fph->state) == FR_PERCPU) {
+		__this_cpu_dec(*mm->futex_ref);
 		return false;
 	}