linux-kernel - [PATCH] sched/fair: prefer prev cpu in asymmetric wakeup path

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20201022134354.3485-1-vincent.guittot@linaro.org>
Date:   Thu, 22 Oct 2020 15:43:54 +0200
From:   Vincent Guittot <vincent.guittot@...aro.org>
To:     mingo@...hat.com, peterz@...radead.org, juri.lelli@...hat.com,
        dietmar.eggemann@....com, rostedt@...dmis.org, bsegall@...gle.com,
        mgorman@...e.de, linux-kernel@...r.kernel.org,
        valentin.schneider@....com, morten.rasmussen@....com
Cc:     Vincent Guittot <vincent.guittot@...aro.org>
Subject: [PATCH] sched/fair: prefer prev cpu in asymmetric wakeup path

During fast wakeup path, scheduler always check whether local or prev cpus
are good candidates for the task before looking for other cpus in the
domain. With
  commit b7a331615d25 ("sched/fair: Add asymmetric CPU capacity wakeup scan")
the heterogenous system gains a dedicated path but doesn't try to keep
reusing prev cpu whenever possible. If the previous cpu is idle and belong to the
asymmetric domain, we should check it 1st before looking for another cpu
because it stays one of the best candidate and it stabilizes task placement
on the system.

This change aligns asymmetric path behavior with symmetric one and reduces
cases where the task migrates across all cpus of the sd_asym_cpucapacity
domains at wakeup.

This change does not impact normal EAS mode but only the overloaded case or
when EAS is not used.

On hikey960 with performance governor (EAS disable)

./perf bench sched pipe -T -l 150000
             mainline           w/ patch
# migrations   299811                  3
ops/sec        154535(+/-0.13%)   181754(+/- 0.29) +17%

Fixes: b7a331615d25 ("sched/fair: Add asymmetric CPU capacity wakeup scan")
Signed-off-by: Vincent Guittot <vincent.guittot@...aro.org>
---
 kernel/sched/fair.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index aa4c6227cd6d..f39638fe6b94 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6170,7 +6170,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
  * maximize capacity.
  */
 static int
-select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
+select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int prev, int target)
 {
 	unsigned long best_cap = 0;
 	int cpu, best_cpu = -1;
@@ -6178,9 +6178,22 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 
 	sync_entity_load_avg(&p->se);
 
+	if ((available_idle_cpu(target) || sched_idle_cpu(target)) &&
+	    task_fits_capacity(p, capacity_of(target)))
+		return target;
+
 	cpus = this_cpu_cpumask_var_ptr(select_idle_mask);
 	cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);
 
+	/*
+	 * If the previous CPU belongs to this asymmetric domain and is idle,
+	 * check it 1st as it's the best candidate.
+	 */
+	if (prev != target && cpumask_test_cpu(prev, cpus) &&
+	    (available_idle_cpu(prev) || sched_idle_cpu(prev)) &&
+	    task_fits_capacity(p, capacity_of(prev)))
+		return prev;
+
 	for_each_cpu_wrap(cpu, cpus, target) {
 		unsigned long cpu_cap = capacity_of(cpu);
 
@@ -6223,7 +6236,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		if (!sd)
 			goto symmetric;
 
-		i = select_idle_capacity(p, sd, target);
+		i = select_idle_capacity(p, sd, prev, target);
 		return ((unsigned)i < nr_cpumask_bits) ? i : target;
 	}
 
-- 
2.17.1