linux-kernel - [PATCH 3/4] rcu: Perform early sequence fetch for polling locklessly

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20220316144255.336021-4-frederic@kernel.org>
Date:   Wed, 16 Mar 2022 15:42:54 +0100
From:   Frederic Weisbecker <frederic@...nel.org>
To:     "Paul E . McKenney" <paulmck@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        Frederic Weisbecker <frederic@...nel.org>,
        Uladzislau Rezki <uladzislau.rezki@...y.com>,
        Boqun Feng <boqun.feng@...il.com>,
        Neeraj Upadhyay <quic_neeraju@...cinc.com>,
        Joel Fernandes <joel@...lfernandes.org>
Subject: [PATCH 3/4] rcu: Perform early sequence fetch for polling locklessly

The workqueue ordering guarantees that the work sees all the accesses
of the task prior to its call to the corresponding queue_work().

Therefore the sequence to poll can be retrieved locklessly.

The only downside is that it is then possible to miss the 0x1 flag set
by a prior work. But this could already happen concurrently anyway after
the exp_poll_lock is unlocked. In the worst case the slow path involving
synchronize_rcu_expedited() takes care of the situation.

Signed-off-by: Frederic Weisbecker <frederic@...nel.org>
Cc: Neeraj Upadhyay <quic_neeraju@...cinc.com>
Cc: Boqun Feng <boqun.feng@...il.com>
Cc: Uladzislau Rezki <uladzislau.rezki@...y.com>
Cc: Joel Fernandes <joel@...lfernandes.org>
---
 kernel/rcu/tree_exp.h | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 763ec35546ed..c4a19c6a83cf 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -909,9 +909,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp)
 	struct rcu_node *rnp = container_of(wp, struct rcu_node, exp_poll_wq);
 	unsigned long s;
 
-	raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
-	s = rnp->exp_seq_poll_rq;
-	raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags);
+	s = READ_ONCE(rnp->exp_seq_poll_rq);
 	if (s & 0x1)
 		return;
 	while (!sync_exp_work_done(s))
@@ -919,7 +917,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp)
 	raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
 	s = rnp->exp_seq_poll_rq;
 	if (!(s & 0x1) && sync_exp_work_done(s))
-		rnp->exp_seq_poll_rq |= 0x1;
+		WRITE_ONCE(rnp->exp_seq_poll_rq, s | 0x1);
 	raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags);
 }
 
@@ -949,7 +947,7 @@ unsigned long start_poll_synchronize_rcu_expedited(void)
 	if (rcu_init_invoked())
 		raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
 	if ((rnp->exp_seq_poll_rq & 0x1) || ULONG_CMP_LT(rnp->exp_seq_poll_rq, s)) {
-		rnp->exp_seq_poll_rq = s;
+		WRITE_ONCE(rnp->exp_seq_poll_rq, s);
 		if (rcu_init_invoked())
 			queue_work(rcu_gp_wq, &rnp->exp_poll_wq);
 	}
-- 
2.25.1