| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20240613181613.4329-3-kprateek.nayak@amd.com>
Date: Thu, 13 Jun 2024 18:16:01 +0000
From: K Prateek Nayak <kprateek.nayak@....com>
To: <linux-kernel@...r.kernel.org>
CC: "Gautham R. Shenoy" <gautham.shenoy@....com>, K Prateek Nayak
<kprateek.nayak@....com>, Richard Henderson <richard.henderson@...aro.org>,
Ivan Kokshaysky <ink@...assic.park.msu.ru>, Matt Turner <mattst88@...il.com>,
Russell King <linux@...linux.org.uk>, Guo Ren <guoren@...nel.org>, "Michal
Simek" <monstr@...str.eu>, Dinh Nguyen <dinguyen@...nel.org>, Jonas Bonn
<jonas@...thpole.se>, Stefan Kristiansson
<stefan.kristiansson@...nalahti.fi>, Stafford Horne <shorne@...il.com>,
"James E.J. Bottomley" <James.Bottomley@...senPartnership.com>, Helge Deller
<deller@....de>, Michael Ellerman <mpe@...erman.id.au>, Nicholas Piggin
<npiggin@...il.com>, Christophe Leroy <christophe.leroy@...roup.eu>, "Naveen
N. Rao" <naveen.n.rao@...ux.ibm.com>, Yoshinori Sato
<ysato@...rs.sourceforge.jp>, Rich Felker <dalias@...c.org>, "John Paul
Adrian Glaubitz" <glaubitz@...sik.fu-berlin.de>, "David S. Miller"
<davem@...emloft.net>, Andreas Larsson <andreas@...sler.com>, Thomas Gleixner
<tglx@...utronix.de>, Ingo Molnar <mingo@...hat.com>, Borislav Petkov
<bp@...en8.de>, Dave Hansen <dave.hansen@...ux.intel.com>, "H. Peter Anvin"
<hpa@...or.com>, "Rafael J. Wysocki" <rafael@...nel.org>, Daniel Lezcano
<daniel.lezcano@...aro.org>, Peter Zijlstra <peterz@...radead.org>, "Juri
Lelli" <juri.lelli@...hat.com>, Vincent Guittot <vincent.guittot@...aro.org>,
Dietmar Eggemann <dietmar.eggemann@....com>, Steven Rostedt
<rostedt@...dmis.org>, Ben Segall <bsegall@...gle.com>, Mel Gorman
<mgorman@...e.de>, Daniel Bristot de Oliveira <bristot@...hat.com>, "Valentin
Schneider" <vschneid@...hat.com>, Andrew Donnellan <ajd@...ux.ibm.com>,
Benjamin Gray <bgray@...ux.ibm.com>, Frederic Weisbecker
<frederic@...nel.org>, Xin Li <xin3.li@...el.com>, Kees Cook
<keescook@...omium.org>, Rick Edgecombe <rick.p.edgecombe@...el.com>, "Tony
Battersby" <tonyb@...ernetics.com>, Bjorn Helgaas <bhelgaas@...gle.com>,
"Brian Gerst" <brgerst@...il.com>, Leonardo Bras <leobras@...hat.com>, Imran
Khan <imran.f.khan@...cle.com>, "Paul E. McKenney" <paulmck@...nel.org>, "Rik
van Riel" <riel@...riel.com>, Tim Chen <tim.c.chen@...ux.intel.com>, David
Vernet <void@...ifault.com>, Julia Lawall <julia.lawall@...ia.fr>,
<linux-alpha@...r.kernel.org>, <linux-arm-kernel@...ts.infradead.org>,
<linux-csky@...r.kernel.org>, <linux-openrisc@...r.kernel.org>,
<linux-parisc@...r.kernel.org>, <linuxppc-dev@...ts.ozlabs.org>,
<linux-sh@...r.kernel.org>, <sparclinux@...r.kernel.org>,
<linux-pm@...r.kernel.org>, <x86@...nel.org>
Subject: [PATCH v2 02/14] sched: Define a need_resched_or_ipi() helper and use it treewide
From: "Gautham R. Shenoy" <gautham.shenoy@....com>
Currently TIF_NEED_RESCHED is being overloaded, to wakeup an idle CPU in
TIF_POLLING mode to service an IPI even if there are no new tasks being
woken up on the said CPU.
In preparation of a proper fix, introduce a new helper
"need_resched_or_ipi()" which is intended to return true if either
the TIF_NEED_RESCHED flag or if TIF_NOTIFY_IPI flag is set. Use this
helper function in place of need_resched() in idle loops where
TIF_POLLING_NRFLAG is set.
To preserve bisectibility and avoid unbreakable idle loops, all the
need_resched() checks within TIF_POLLING_NRFLAGS sections, have been
replaced tree-wide with the need_resched_or_ipi() check.
[ prateek: Replaced some of the missed out occurrences of
need_resched() within a TIF_POLLING sections with
need_resched_or_ipi() ]
Cc: Richard Henderson <richard.henderson@...aro.org>
Cc: Ivan Kokshaysky <ink@...assic.park.msu.ru>
Cc: Matt Turner <mattst88@...il.com>
Cc: Russell King <linux@...linux.org.uk>
Cc: Guo Ren <guoren@...nel.org>
Cc: Michal Simek <monstr@...str.eu>
Cc: Dinh Nguyen <dinguyen@...nel.org>
Cc: Jonas Bonn <jonas@...thpole.se>
Cc: Stefan Kristiansson <stefan.kristiansson@...nalahti.fi>
Cc: Stafford Horne <shorne@...il.com>
Cc: "James E.J. Bottomley" <James.Bottomley@...senPartnership.com>
Cc: Helge Deller <deller@....de>
Cc: Michael Ellerman <mpe@...erman.id.au>
Cc: Nicholas Piggin <npiggin@...il.com>
Cc: Christophe Leroy <christophe.leroy@...roup.eu>
Cc: "Naveen N. Rao" <naveen.n.rao@...ux.ibm.com>
Cc: Yoshinori Sato <ysato@...rs.sourceforge.jp>
Cc: Rich Felker <dalias@...c.org>
Cc: John Paul Adrian Glaubitz <glaubitz@...sik.fu-berlin.de>
Cc: "David S. Miller" <davem@...emloft.net>
Cc: Andreas Larsson <andreas@...sler.com>
Cc: Thomas Gleixner <tglx@...utronix.de>
Cc: Ingo Molnar <mingo@...hat.com>
Cc: Borislav Petkov <bp@...en8.de>
Cc: Dave Hansen <dave.hansen@...ux.intel.com>
Cc: "H. Peter Anvin" <hpa@...or.com>
Cc: "Rafael J. Wysocki" <rafael@...nel.org>
Cc: Daniel Lezcano <daniel.lezcano@...aro.org>
Cc: Peter Zijlstra <peterz@...radead.org>
Cc: Juri Lelli <juri.lelli@...hat.com>
Cc: Vincent Guittot <vincent.guittot@...aro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@....com>
Cc: Steven Rostedt <rostedt@...dmis.org>
Cc: Ben Segall <bsegall@...gle.com>
Cc: Mel Gorman <mgorman@...e.de>
Cc: Daniel Bristot de Oliveira <bristot@...hat.com>
Cc: Valentin Schneider <vschneid@...hat.com>
Cc: Andrew Donnellan <ajd@...ux.ibm.com>
Cc: Benjamin Gray <bgray@...ux.ibm.com>
Cc: Frederic Weisbecker <frederic@...nel.org>
Cc: Xin Li <xin3.li@...el.com>
Cc: Kees Cook <keescook@...omium.org>
Cc: Rick Edgecombe <rick.p.edgecombe@...el.com>
Cc: Tony Battersby <tonyb@...ernetics.com>
Cc: Bjorn Helgaas <bhelgaas@...gle.com>
Cc: Brian Gerst <brgerst@...il.com>
Cc: Leonardo Bras <leobras@...hat.com>
Cc: Imran Khan <imran.f.khan@...cle.com>
Cc: "Paul E. McKenney" <paulmck@...nel.org>
Cc: Rik van Riel <riel@...riel.com>
Cc: Tim Chen <tim.c.chen@...ux.intel.com>
Cc: David Vernet <void@...ifault.com>
Cc: Julia Lawall <julia.lawall@...ia.fr>
Cc: linux-alpha@...r.kernel.org
Cc: linux-kernel@...r.kernel.org
Cc: linux-arm-kernel@...ts.infradead.org
Cc: linux-csky@...r.kernel.org
Cc: linux-openrisc@...r.kernel.org
Cc: linux-parisc@...r.kernel.org
Cc: linuxppc-dev@...ts.ozlabs.org
Cc: linux-sh@...r.kernel.org
Cc: sparclinux@...r.kernel.org
Cc: linux-pm@...r.kernel.org
Cc: x86@...nel.org
Signed-off-by: Gautham R. Shenoy <gautham.shenoy@....com>
Co-developed-by: K Prateek Nayak <kprateek.nayak@....com>
Signed-off-by: K Prateek Nayak <kprateek.nayak@....com>
---
v1..v2:
o Fixed a conflict with commit edc8fc01f608 ("x86: Fix
CPUIDLE_FLAG_IRQ_ENABLE leaking timer reprogram") that touched
mwait_idle_with_hints() in arch/x86/include/asm/mwait.h
---
arch/x86/include/asm/mwait.h | 2 +-
arch/x86/kernel/process.c | 2 +-
drivers/cpuidle/cpuidle-powernv.c | 2 +-
drivers/cpuidle/cpuidle-pseries.c | 2 +-
drivers/cpuidle/poll_state.c | 2 +-
include/linux/sched.h | 5 +++++
include/linux/sched/idle.h | 4 ++--
kernel/sched/idle.c | 7 ++++---
8 files changed, 16 insertions(+), 10 deletions(-)
diff --git a/arch/x86/include/asm/mwait.h b/arch/x86/include/asm/mwait.h
index 920426d691ce..3fa6f0bbd74f 100644
--- a/arch/x86/include/asm/mwait.h
+++ b/arch/x86/include/asm/mwait.h
@@ -125,7 +125,7 @@ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned lo
__monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (!need_resched()) {
+ if (!need_resched_or_ipi()) {
if (ecx & 1) {
__mwait(eax, ecx);
} else {
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index b8441147eb5e..dd73cd6f735c 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -901,7 +901,7 @@ static __cpuidle void mwait_idle(void)
}
__monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (!need_resched()) {
+ if (!need_resched_or_ipi()) {
__sti_mwait(0, 0);
raw_local_irq_disable();
}
diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c
index 9ebedd972df0..77c3bb371f56 100644
--- a/drivers/cpuidle/cpuidle-powernv.c
+++ b/drivers/cpuidle/cpuidle-powernv.c
@@ -79,7 +79,7 @@ static int snooze_loop(struct cpuidle_device *dev,
dev->poll_time_limit = false;
ppc64_runlatch_off();
HMT_very_low();
- while (!need_resched()) {
+ while (!need_resched_or_ipi()) {
if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time) {
/*
* Task has not woken up but we are exiting the polling
diff --git a/drivers/cpuidle/cpuidle-pseries.c b/drivers/cpuidle/cpuidle-pseries.c
index 14db9b7d985d..4f2b490f8b73 100644
--- a/drivers/cpuidle/cpuidle-pseries.c
+++ b/drivers/cpuidle/cpuidle-pseries.c
@@ -46,7 +46,7 @@ int snooze_loop(struct cpuidle_device *dev, struct cpuidle_driver *drv,
snooze_exit_time = get_tb() + snooze_timeout;
dev->poll_time_limit = false;
- while (!need_resched()) {
+ while (!need_resched_or_ipi()) {
HMT_low();
HMT_very_low();
if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time) {
diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c
index 9b6d90a72601..225f37897e45 100644
--- a/drivers/cpuidle/poll_state.c
+++ b/drivers/cpuidle/poll_state.c
@@ -26,7 +26,7 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,
limit = cpuidle_poll_time(drv, dev);
- while (!need_resched()) {
+ while (!need_resched_or_ipi()) {
cpu_relax();
if (loop_count++ < POLL_IDLE_RELAX_COUNT)
continue;
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 90691d99027e..e52cdd1298bf 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2069,6 +2069,11 @@ static __always_inline bool need_resched(void)
return unlikely(tif_need_resched());
}
+static __always_inline bool need_resched_or_ipi(void)
+{
+ return unlikely(tif_need_resched() || tif_notify_ipi());
+}
+
/*
* Wrappers for p->thread_info->cpu access. No-op on UP.
*/
diff --git a/include/linux/sched/idle.h b/include/linux/sched/idle.h
index e670ac282333..497518b84e8d 100644
--- a/include/linux/sched/idle.h
+++ b/include/linux/sched/idle.h
@@ -63,7 +63,7 @@ static __always_inline bool __must_check current_set_polling_and_test(void)
*/
smp_mb__after_atomic();
- return unlikely(tif_need_resched());
+ return unlikely(need_resched_or_ipi());
}
static __always_inline bool __must_check current_clr_polling_and_test(void)
@@ -76,7 +76,7 @@ static __always_inline bool __must_check current_clr_polling_and_test(void)
*/
smp_mb__after_atomic();
- return unlikely(tif_need_resched());
+ return unlikely(need_resched_or_ipi());
}
#else
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index 6e78d071beb5..7de94df5d477 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -57,7 +57,7 @@ static noinline int __cpuidle cpu_idle_poll(void)
ct_cpuidle_enter();
raw_local_irq_enable();
- while (!tif_need_resched() &&
+ while (!need_resched_or_ipi() &&
(cpu_idle_force_poll || tick_check_broadcast_expired()))
cpu_relax();
raw_local_irq_disable();
@@ -174,7 +174,7 @@ static void cpuidle_idle_call(void)
* Check if the idle task must be rescheduled. If it is the
* case, exit the function after re-enabling the local IRQ.
*/
- if (need_resched()) {
+ if (need_resched_or_ipi()) {
local_irq_enable();
return;
}
@@ -270,7 +270,7 @@ static void do_idle(void)
__current_set_polling();
tick_nohz_idle_enter();
- while (!need_resched()) {
+ while (!need_resched_or_ipi()) {
rmb();
/*
@@ -350,6 +350,7 @@ static void do_idle(void)
* RCU relies on this call to be done outside of an RCU read-side
* critical section.
*/
+ current_clr_notify_ipi();
flush_smp_call_function_queue();
schedule_idle();
--
2.34.1
Powered by blists - more mailing lists