linux-kernel - Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <7438bb3a-96d6-485a-9ecc-63829db74b39@os.amperecomputing.com>
Date: Wed, 20 Aug 2025 11:35:07 +0800
From: Adam Li <adamli@...amperecomputing.com>
To: Valentin Schneider <vschneid@...hat.com>, mingo@...hat.com,
 peterz@...radead.org, juri.lelli@...hat.com, vincent.guittot@...aro.org
Cc: dietmar.eggemann@....com, rostedt@...dmis.org, bsegall@...gle.com,
 mgorman@...e.de, cl@...ux.com, frederic@...nel.org,
 linux-kernel@...r.kernel.org, patches@...erecomputing.com
Subject: Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB
 CPU

Hi Valentin,

Thanks for your comments.
On 8/19/2025 10:00 PM, Valentin Schneider wrote:
> On 19/08/25 02:57, Adam Li wrote:
>> A qualified CPU to run NOHZ idle load balancing (ILB) has to be:
>> 1) housekeeping CPU in housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
>> 2) and in nohz.idle_cpus_mask
>> 3) and idle
>> 4) and not current CPU
>>
>> If most CPUs are in nohz_full CPU list there is few housekeeping CPU left.
>> In the worst case if all CPUs are in nohz_full only the boot CPU is used
>> for housekeeping. And the housekeeping CPU is usually busier so it will
>> be unlikely added to nohz.idle_cpus_mask.
>>
>> Therefore if there is few housekeeping CPUs, find_new_ilb() may likely
>> failed to find any CPU to do NOHZ idle load balancing. Some NOHZ CPUs may
>> stay idle while other CPUs are busy.
>>
>> This patch adds fallback options when looking for ILB CPU if there is
>> no CPU meeting above requirements. Then it searches in bellow order:
>> 1) Try looking for the first idle housekeeping CPU
>> 2) Try looking for the first idle CPU in nohz.idle_cpus_mask if no SMT.
>> 3) Select the first housekeeping CPU even if it is busy.
>>
>> With this patch the NOHZ idle balancing happens more frequently.
>>
> 
> I'm not understanding why, in the scenarios outlined above, more NOHZ idle
> balancing is a good thing.
> 
> Considering only housekeeping CPUs, they're all covered by wakeup, periodic
> and idle balancing (on top of NOHZ idle balancing when relevant). So if
> find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs
> are either always busy or never stopping the tick when going idle, IOW they
> always have some work to do within a jiffy boundary.
> > Am I missing something?
>

I agree with your description about the housekeeping CPUs. In the worst case,
the system only has one housekeeping CPU and this housekeeping CPU is so busy
that:
1) This housekeeping CPU is unlikely idle;
2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick
is not stopped.
Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected
to do NOHZ idle load balancing.

This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask).
Here is more background:

When running llama on arm64 server, some CPUs *keep* idle while others
are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL
is set.

The problem is caused by two issues:
1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask',
this bug is fixed by another patch:
https://lore.kernel.org/all/20250815065115.289337-2-adamli@os.amperecomputing.com/

2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to
do NOHZ idle load balancing because conditions in find_new_ilb() is too strict.
This patch tries to solve this issue.

Hope this information helps.

Thanks,
-adam