| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <91ab8136-64f3-45e3-9fec-567aab193353@amd.com>
Date: Fri, 12 Sep 2025 10:38:12 +0530
From: K Prateek Nayak <kprateek.nayak@....com>
To: Tim Chen <tim.c.chen@...ux.intel.com>, Peter Zijlstra
<peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>
CC: Juri Lelli <juri.lelli@...hat.com>, Dietmar Eggemann
<dietmar.eggemann@....com>, Ben Segall <bsegall@...gle.com>, Mel Gorman
<mgorman@...e.de>, Valentin Schneider <vschneid@...hat.com>, Tim Chen
<tim.c.chen@...el.com>, Vincent Guittot <vincent.guittot@...aro.org>, "Libo
Chen" <libo.chen@...cle.com>, Abel Wu <wuyun.abel@...edance.com>, Len Brown
<len.brown@...el.com>, <linux-kernel@...r.kernel.org>, Chen Yu
<yu.c.chen@...el.com>, "Gautham R . Shenoy" <gautham.shenoy@....com>, "Zhao
Liu" <zhao1.liu@...el.com>, Vinicius Costa Gomes <vinicius.gomes@...el.com>,
Arjan Van De Ven <arjan.van.de.ven@...el.com>
Subject: Re: [PATCH v3 2/2] sched: Fix sched domain build error for GNR, CWF
in SNC-3 mode
Hello Tim,
On 9/12/2025 12:00 AM, Tim Chen wrote:
> It is possible for Granite Rapids (GNR) and Clearwater Forest
> (CWF) to have up to 3 dies per package. When sub-numa cluster (SNC-3)
> is enabled, each die will become a separate NUMA node in the package
> with different distances between dies within the same package.
>
> For example, on GNR, we see the following numa distances for a 2 socket
> system with 3 dies per socket:
>
> package 1 package2
> ----------------
> | |
> --------- ---------
> | 0 | | 3 |
> --------- ---------
> | |
> --------- ---------
> | 1 | | 4 |
> --------- ---------
> | |
> --------- ---------
> | 2 | | 5 |
> --------- ---------
> | |
> ----------------
>
> node distances:
> node 0 1 2 3 4 5
> 0: 10 15 17 21 28 26
> 1: 15 10 15 23 26 23
> 2: 17 15 10 26 23 21
> 3: 21 28 26 10 15 17
> 4: 23 26 23 15 10 15
> 5: 26 23 21 17 15 10
>
> The node distances above led to 2 problems:
>
> 1. Asymmetric routes taken between nodes in different packages led to
> asymmetric scheduler domain perspective depending on which node you
> are on. Current scheduler code failed to build domains properly with
> asymmetric distances.
>
> 2. Multiple remote distances to respective tiles on remote package create
> too many levels of domain hierarchies grouping different nodes between
> remote packages.
>
> For example, the above GNR-X topology lead to NUMA domains below:
>
> Sched domains from the perspective of a CPU in node 0, where the number
> in bracket represent node number.
>
> NUMA-level 1 [0,1] [2]
> NUMA-level 2 [0,1,2] [3]
> NUMA-level 3 [0,1,2,3] [5]
> NUMA-level 4 [0,1,2,3,5] [4]
>
> Sched domains from the perspective of a CPU in node 4
> NUMA-level 1 [4] [3,5]
> NUMA-level 2 [3,4,5] [0,2]
> NUMA-level 3 [0,2,3,4,5] [1]
>
> Scheduler group peers for load balancing from the perspective of CPU 0
> and 4 are different. Improper task could be chosen for load balancing
> between groups such as [0,2,3,4,5] [1]. Ideally you should choose nodes
> in 0 or 2 that are in same package as node 1 first. But instead tasks
> in the remote package node 3, 4, 5 could be chosen with an equal chance
> and could lead to excessive remote package migrations and imbalance of
> load between packages. We should not group partial remote nodes and
> local nodes together.
> Simplify the remote distances for CWF-X and GNR-X for the purpose of
> sched domains building, which maintains symmetry and leads to a more
> reasonable load balance hierarchy.
>
> The sched domains from the perspective of a CPU in node 0 NUMA-level 1
> is now
> NUMA-level 1 [0,1] [2]
> NUMA-level 2 [0,1,2] [3,4,5]
>
> The sched domains from the perspective of a CPU in node 4 NUMA-level 1
> is now
> NUMA-level 1 [4] [3,5]
> NUMA-level 2 [3,4,5] [0,1,2]
>
> We have the same balancing perspective from node 0 or node 4. Loads are
> now balanced equally between packages.
>
> Tested-by: Zhao Liu <zhao1.liu@...el.com>
> Co-developed-by: Vinicius Costa Gomes <vinicius.gomes@...el.com>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@...el.com>
> Signed-off-by: Tim Chen <tim.c.chen@...ux.intel.com>
Feel free to include:
Reviewed-and-tested-by: K Prateek Nayak <kprateek.nayak@....com>
--
Thanks and Regards,
Prateek
> ---
> arch/x86/kernel/smpboot.c | 28 ++++++++++++++++++++++++++++
> 1 file changed, 28 insertions(+)
>
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index 33e166f6ab12..3f894c525e49 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -515,6 +515,34 @@ static void __init build_sched_topology(void)
> set_sched_topology(topology);
> }
>
> +int arch_sched_node_distance(int from, int to)
> +{
> + int d = node_distance(from, to);
> +
> + if (!x86_has_numa_in_package)
> + return d;
> +
> + switch (boot_cpu_data.x86_vfm) {
> + case INTEL_GRANITERAPIDS_X:
> + case INTEL_ATOM_DARKMONT_X:
> + if (d < REMOTE_DISTANCE)
> + return d;
> +
> + /*
> + * Trim finer distance tuning for nodes in remote package
> + * for the purpose of building sched domains. Put NUMA nodes
> + * in each remote package in the same sched group.
> + * Simplify NUMA domains and avoid extra NUMA levels including
> + * different NUMA nodes in remote packages.
> + *
> + * GNR and CWF don't expect systmes with more than 2 packages
> + * and more than 2 hops between packages.
> + */
> + d = sched_avg_remote_numa_distance;
> + }
> + return d;
> +}
> +
> void set_cpu_sibling_map(int cpu)
> {
> bool has_smt = __max_threads_per_core > 1;
Powered by blists - more mailing lists