| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20250320154807.982435-1-colin.i.king@gmail.com>
Date: Thu, 20 Mar 2025 15:48:07 +0000
From: Colin Ian King <colin.i.king@...il.com>
To: Ingo Molnar <mingo@...hat.com>,
Peter Zijlstra <peterz@...radead.org>,
Juri Lelli <juri.lelli@...hat.com>,
Vincent Guittot <vincent.guittot@...aro.org>,
Dietmar Eggemann <dietmar.eggemann@....com>,
Steven Rostedt <rostedt@...dmis.org>,
Ben Segall <bsegall@...gle.com>,
Mel Gorman <mgorman@...e.de>,
Valentin Schneider <vschneid@...hat.com>
Cc: linux-kernel@...r.kernel.org
Subject: [PATCH][next] sched: add some likely/unlikey branch hints in scheduler core
There are several places in the scheduler core where branch hinting
can improve performance. The added hints in this commit are based
on a 1.5 day run of all the 360+ stress-ng test cases and various
kernel builds on a gcov coverage instrumented kernel. The most
heavily exercised if statements were examined for very likely or
unlikely paths and branch hints added.
Benchmarked on a 24 thread Intel(R) Core(TM) Ultra 9 285K, running
Debian Sid. Built with default Debian kernel build flags with
gcc 14.2.0 (Debian 14.2.0-19)
Testing: Turbo was disabled to reduce clock frequency jitter for stable
results. Several stress-ng scheduler stressing stress-ng tests were
used to benchmark the system. Each test was run for 20 seconds with a
20 second cool-down wait between each test. Tests were run 10 times
and performance metrics based on the geometic mean of the bogo-op
throughput rates from stress-ng. Tested with 24 stressor instances
and also 1 stressor instance.
24 stressor instance results:
bogo-ops/sec metrics
original optimized
cpu-sched 1,405,594 1,476,842
fifo 10,867,835 7,220,092
flipflop 370,684 388,464
futex 886,307 1,038,800
mq 9,874,889 10,286,373
msg 14,988,007 15,251,091
mtx 1,183,226,726 1,116,962,074
mutex 7,636,566 7,624,284
poll 1,875,193 1,849,430
prio-inv 43,605,926 48,446,085
pthread 153,030 193,861
race-sched 7,509 8,029
schedmix 18,858 18,617
schedpolicy 5,273,112 5,395,978
switch 10,078,698 10,077,338
tee 219,771 224,791
yield 38,654,965 38,441,461
Goemean: 2,488,153 2,527,566
Improvement: ~1.5%
1 stressor instance results:
bogo-ops/sec metrics
original optimized
cpu-sched 146,092 146,357
fifo 609,243 595,224
flipflop 27,288 32,939
futex 259,095 273,300
mq 971,830 988,218
msg 3,630,096 3,646,838
mtx 10,829,939 11,285,273
mutex 1,179,245 1,168,837
poll 70,430 70,369
prio-inv 450,512 457,393
pthread 39,407 39,642
race-sched 1,453 1,470
schedmix 4,373 4,376
schedpolicy 522,030 523,113
switch 247,287 241,640
tee 16,685 16,771
yield 37,409,707 37,064,076
233,441 237,328
Improvement: ~1.6%
Signed-off-by: Colin Ian King <colin.i.king@...il.com>
---
kernel/sched/core.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 87540217fc09..90a08eb571ca 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5929,10 +5929,10 @@ static noinline void __schedule_bug(struct task_struct *prev)
static inline void schedule_debug(struct task_struct *prev, bool preempt)
{
#ifdef CONFIG_SCHED_STACK_END_CHECK
- if (task_stack_end_corrupted(prev))
+ if (unlikely(task_stack_end_corrupted(prev)))
panic("corrupted stack end detected inside scheduler\n");
- if (task_scs_end_corrupted(prev))
+ if (unlikely(task_scs_end_corrupted(prev)))
panic("corrupted shadow stack detected inside scheduler\n");
#endif
@@ -10402,7 +10402,7 @@ void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t)
if (!mm)
return;
src_cpu = t->migrate_from_cpu;
- if (src_cpu == -1) {
+ if (unlikely(src_cpu == -1)) {
t->last_mm_cid = -1;
return;
}
@@ -10432,7 +10432,7 @@ void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t)
return;
src_cid = __sched_mm_cid_migrate_from_try_steal_cid(src_rq, t, src_pcpu_cid,
src_cid);
- if (src_cid == -1)
+ if (unlikely(src_cid == -1))
return;
if (dst_cid_is_set) {
__mm_cid_put(mm, src_cid);
@@ -10559,7 +10559,7 @@ static void task_mm_cid_work(struct callback_head *work)
if (t->flags & PF_EXITING)
return;
mm = t->mm;
- if (!mm)
+ if (unlikely(!mm))
return;
old_scan = READ_ONCE(mm->mm_cid_next_scan);
next_scan = now + msecs_to_jiffies(MM_CID_SCAN_DELAY);
@@ -10594,7 +10594,7 @@ void init_sched_mm_cid(struct task_struct *t)
struct mm_struct *mm = t->mm;
int mm_users = 0;
- if (mm) {
+ if (likely(mm)) {
mm_users = atomic_read(&mm->mm_users);
if (mm_users == 1)
mm->mm_cid_next_scan = jiffies + msecs_to_jiffies(MM_CID_SCAN_DELAY);
@@ -10623,7 +10623,7 @@ void sched_mm_cid_exit_signals(struct task_struct *t)
struct mm_struct *mm = t->mm;
struct rq *rq;
- if (!mm)
+ if (unlikely(!mm))
return;
preempt_disable();
@@ -10645,7 +10645,7 @@ void sched_mm_cid_before_execve(struct task_struct *t)
struct mm_struct *mm = t->mm;
struct rq *rq;
- if (!mm)
+ if (unlikely(!mm))
return;
preempt_disable();
--
2.49.0
Powered by blists - more mailing lists