linux-kernel - [PATCH v2 7/8] rcu/nocb: Rewrite deferred wake up logic to be more clean

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20220622225102.2112026-9-joel@joelfernandes.org>
Date:   Wed, 22 Jun 2022 22:51:01 +0000
From:   "Joel Fernandes (Google)" <joel@...lfernandes.org>
To:     rcu@...r.kernel.org
Cc:     linux-kernel@...r.kernel.org, rushikesh.s.kadam@...el.com,
        urezki@...il.com, neeraj.iitr10@...il.com, frederic@...nel.org,
        paulmck@...nel.org, rostedt@...dmis.org, vineeth@...byteword.org,
        "Joel Fernandes (Google)" <joel@...lfernandes.org>
Subject: [PATCH v2 7/8] rcu/nocb: Rewrite deferred wake up logic to be more clean

There are 2 things this function does:
1. modify the gp wake timer.
2. save the value of the strongest requested wake up so far.

The strongest is "wake force" and the weakest is "lazy".

The existing logic already does the following:
1. if the existing deferred wake is stronger than the requested one
   (requested in waketype), modify the gp timer to be more in the
   future. For example, if the existing one is WAKE and the new waketype
   requested is BYPASS, then the timer is made to expire later than
   earlier.

2. even though the timer is modified in #1, a weaker waketype does not
   end up changing rdp->nocb_gp_defer to be weaker. In other words,
   ->nocb_gp_defer records the strongest waketype requested so far,
   even though the timer may or may not be the soonest expiry possible.

For simplicity, we write this logic using switch statements and
consolidate some of the timer modification operations.

Signed-off-by: Joel Fernandes (Google) <joel@...lfernandes.org>
---
 kernel/rcu/tree_nocb.h | 35 ++++++++++++++++++++++++-----------
 1 file changed, 24 insertions(+), 11 deletions(-)

diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index 255f2945b0fc..67b0bd5d233a 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -282,6 +282,7 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype,
 {
 	unsigned long flags;
 	struct rcu_data *rdp_gp = rdp->nocb_gp_rdp;
+	unsigned long mod_jif = 0;
 
 	raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags);
 
@@ -289,19 +290,31 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype,
 	 * Bypass wakeup overrides previous deferments. In case
 	 * of callback storm, no need to wake up too early.
 	 */
-	if (waketype == RCU_NOCB_WAKE_LAZY) {
-		mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush);
-		WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
-	} else if (waketype == RCU_NOCB_WAKE_BYPASS) {
-		mod_timer(&rdp_gp->nocb_timer, jiffies + 2);
-		WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
-	} else {
-		if (rdp_gp->nocb_defer_wakeup < RCU_NOCB_WAKE)
-			mod_timer(&rdp_gp->nocb_timer, jiffies + 1);
-		if (rdp_gp->nocb_defer_wakeup < waketype)
-			WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
+	switch (waketype) {
+		case RCU_NOCB_WAKE_LAZY:
+			mod_jif = jiffies_till_flush;
+			break;
+
+		case RCU_NOCB_WAKE_BYPASS:
+			mod_jif = 2;
+			break;
+
+		case RCU_NOCB_WAKE:
+		case RCU_NOCB_WAKE_FORCE:
+			// If the type of deferred wake is "stronger"
+			// than it was before, make it wake up the soonest.
+			if (rdp_gp->nocb_defer_wakeup < RCU_NOCB_WAKE)
+				mod_jif = 1;
+			break;
 	}
 
+	if (mod_jif)
+		mod_timer(&rdp_gp->nocb_timer, jiffies + mod_jif);
+
+	// If new type of wake up is strong than before, promote.
+	if (rdp_gp->nocb_defer_wakeup < waketype)
+		WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
+
 	raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags);
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, reason);
-- 
2.37.0.rc0.104.g0611611a94-goog