linux-kernel - Trade-off between load_balance frequency and CPU utilization under high load

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <f10ea4ddee5d4ad588ca5aba3c5afac6@huawei.com>
Date: Thu, 26 Dec 2024 02:09:33 +0000
From: "liukai (Y)" <liukai284@...wei.com>
To: "mingo@...hat.com" <mingo@...hat.com>, "peterz@...radead.org"
	<peterz@...radead.org>, "juri.lelli@...hat.com" <juri.lelli@...hat.com>,
	"vincent.guittot@...aro.org" <vincent.guittot@...aro.org>
CC: "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>, "tanghui
 (C)" <tanghui20@...wei.com>, "Zhangqiao (2012 lab)" <zhangqiao22@...wei.com>,
	"Chenhui (Judy)" <judy.chenhui@...wei.com>, "'weiliang.qwl@...group.com'"
	<weiliang.qwl@...group.com>, "'henry.hj@...group.com'"
	<henry.hj@...group.com>, "'yanyan.yan@...group.com'"
	<yanyan.yan@...group.com>, "'libang.li@...group.com'"
	<libang.li@...group.com>, "liwei (JK)" <liwei728@...wei.com>
Subject: Trade-off between load_balance frequency and CPU utilization under
 high load

In our performance experiments, by gradually increasing the CPU load, we
observed that under high load, the CPU utilization (node CPU) in kernel 6.6
is higher than in 4.19, reaching up to 4% higher. By capturing flame graph
data, we found that the total execution time of load_balance in kernel 6.6
is 18% longer than in 4.19.

Benchmark: specjbb // 6.6
index   target QPS   actual QPS           RT      pod CPU     node CPU
    1        60000        60004         0.73         5.06        14.97
    2       120000       120074         0.79        10.22        29.67
    3       180000       180866         0.87        16.00        45.91
    4       240000       240091         0.92        21.69        62.94

Benchmark: specjbb // 4.19
index   target QPS   actual QPS           RT      pod CPU     node CPU
    1        60000        60004         0.72         4.86        14.81
    2       120000       120074         0.79         9.69        29.52
    3       180000       180870         0.83        14.57        42.72
    4       240000       240074         0.90        19.55        58.59

we found that in kernel 6.6, the execution of load_balance is less costly.
Even under high load, the condition this_rq->avg_idle <
sd->max_newidle_lb_cost is still easily satisfied. As a result, compared to
kernel 4.19, load_balance is executed more frequently in 6.6, leading to
higher CPU utilization.

if (!READ_ONCE(this_rq->rd->overload) ||
    (sd && this_rq->avg_idle < sd->max_newidle_lb_cost)) {

    if (sd)
        update_next_balance(sd, &next_balance);
    rcu_read_unlock();

    goto out;
}

We identified that the changes introduced by this patch
(https://lore.kernel.org/all/20211021095219.GG3891@suse.de/).

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10895,8 +10895,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf)
     rcu_read_lock();
     sd = rcu_dereference_check_sched_domain(this_rq->sd);

-     if (this_rq->avg_idle < sysctl_sched_migration_cost ||
-         !READ_ONCE(this_rq->rd->overload) ||
+     if (!READ_ONCE(this_rq->rd->overload) ||
         (sd && this_rq->avg_idle < sd->max_newidle_lb_cost)) {

             if (sd)

this_rq->avg_idle < sysctl_sched_migration_cost can reduce the frequency of
load_balance under high load, is there any way to dynamically adjust the
execution of load_balance in high-load scenarios, in order to strike a
balance between maintaining good CPU utilization and avoiding unnecessary
load_balance executions?