linux-kernel - [PATCH 6/9] cpuidle/poll_state: poll via smp_cond_load

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20240430183730.561960-7-ankur.a.arora@oracle.com>
Date: Tue, 30 Apr 2024 11:37:27 -0700
From: Ankur Arora <ankur.a.arora@...cle.com>
To: linux-pm@...r.kernel.org, kvm@...r.kernel.org,
        linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org
Cc: catalin.marinas@....com, will@...nel.org, tglx@...utronix.de,
        mingo@...hat.com, bp@...en8.de, x86@...nel.org, hpa@...or.com,
        pbonzini@...hat.com, wanpengli@...cent.com, vkuznets@...hat.com,
        rafael@...nel.org, daniel.lezcano@...aro.org, peterz@...radead.org,
        arnd@...db.de, lenb@...nel.org, mark.rutland@....com,
        harisokn@...zon.com, joao.m.martins@...cle.com,
        boris.ostrovsky@...cle.com, konrad.wilk@...cle.com,
        ankur.a.arora@...cle.com
Subject: [PATCH 6/9] cpuidle/poll_state: poll via smp_cond_load_relaxed()

From: Mihai Carabas <mihai.carabas@...cle.com>

The inner loop in poll_idle() polls up to POLL_IDLE_RELAX_COUNT times,
checking to see if the thread has the TIF_NEED_RESCHED bit set. The
loop exits once the condition is met, or if the poll time limit has
been exceeded.

The time check is done only infrequently (once in POLL_IDLE_RELAX_COUNT
iterations) so as to minimize the number of instructions executed in
each iteration. In addition, each loop iteration executes cpu_relax()
which on certain platforms provides a hint to the pipeline that the
loop is busy-waiting, thus allowing the processor to reduce power
consumption.

However, cpu_relax() is not defined optimally everywhere. In particular,
on arm64, it is implemented as a YIELD which merely serves as a hint to
prefer a different hardware thread if one is available.

arm64 exposes a better mechanism via smp_cond_load_relaxed() which uses
LDXR, WFE where the LDXR loads a memory region in exclusive state and
the WFE waits for any stores to the region.

So restructure the loop and fold both checks in smp_cond_load_relaxed().
Also, move the time check to the head of the loop so, once
TIF_NEED_RESCHED is set, we exit straight-away without doing an
unnecessary time check.

Suggested-by: Peter Zijlstra <peterz@...radead.org>
Signed-off-by: Mihai Carabas <mihai.carabas@...cle.com>
Signed-off-by: Ankur Arora <ankur.a.arora@...cle.com>
---
Changelog:

   - reorganized the loop to keep the original poll_idle() structure.

---
 drivers/cpuidle/poll_state.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c
index 9b6d90a72601..532e4ed19e0f 100644
--- a/drivers/cpuidle/poll_state.c
+++ b/drivers/cpuidle/poll_state.c
@@ -21,21 +21,21 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,

 	raw_local_irq_enable();
 	if (!current_set_polling_and_test()) {
-		unsigned int loop_count = 0;
+		unsigned int loop_count;
 		u64 limit;

 		limit = cpuidle_poll_time(drv, dev);

 		while (!need_resched()) {
-			cpu_relax();
-			if (loop_count++ < POLL_IDLE_RELAX_COUNT)
-				continue;
-
 			loop_count = 0;
 			if (local_clock_noinstr() - time_start > limit) {
 				dev->poll_time_limit = true;
 				break;
 			}
+
+			smp_cond_load_relaxed(&current_thread_info()->flags,
+					      VAL & _TIF_NEED_RESCHED ||
+					      loop_count++ >= POLL_IDLE_RELAX_COUNT);
 		}
 	}
 	raw_local_irq_disable();
-- 
2.39.3