| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <2026020701-ether-wieldable-f250@gregkh>
Date: Sat, 7 Feb 2026 16:29:25 +0100
From: Greg KH <gregkh@...uxfoundation.org>
To: Tim Chen <tim.c.chen@...ux.intel.com>
Cc: stable@...r.kernel.org, Peter Zijlstra <peterz@...radead.org>,
Ingo Molnar <mingo@...nel.org>, Juri Lelli <juri.lelli@...hat.com>,
Dietmar Eggemann <dietmar.eggemann@....com>,
Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
Valentin Schneider <vschneid@...hat.com>,
Tim Chen <tim.c.chen@...el.com>,
Vincent Guittot <vincent.guittot@...aro.org>,
Len Brown <len.brown@...el.com>, linux-kernel@...r.kernel.org,
Chen Yu <yu.c.chen@...el.com>,
K Prateek Nayak <kprateek.nayak@....com>,
"Gautham R . Shenoy" <gautham.shenoy@....com>,
Zhao Liu <zhao1.liu@...el.com>,
Vinicius Costa Gomes <vinicius.gomes@...el.com>,
Arjan Van De Ven <arjan.van.de.ven@...el.com>
Subject: Re: [PATCH 6.18 2/2] sched/topology: Fix sched domain build error
for GNR, CWF in SNC-3 mode
On Thu, Feb 05, 2026 at 01:33:34PM -0800, Tim Chen wrote:
> [ Upstream commit 4d6dd05d07d00bc3bd91183dab4d75caa8018db9 ]
>
> It is possible for Granite Rapids (GNR) and Clearwater Forest
> (CWF) to have up to 3 dies per package. When sub-numa cluster (SNC-3)
> is enabled, each die will become a separate NUMA node in the package
> with different distances between dies within the same package.
>
> For example, on GNR, we see the following numa distances for a 2 socket
> system with 3 dies per socket:
>
> package 1 package2
> ----------------
> | |
> --------- ---------
> | 0 | | 3 |
> --------- ---------
> | |
> --------- ---------
> | 1 | | 4 |
> --------- ---------
> | |
> --------- ---------
> | 2 | | 5 |
> --------- ---------
> | |
> ----------------
>
> node distances:
> node 0 1 2 3 4 5
> 0: 10 15 17 21 28 26
> 1: 15 10 15 23 26 23
> 2: 17 15 10 26 23 21
> 3: 21 28 26 10 15 17
> 4: 23 26 23 15 10 15
> 5: 26 23 21 17 15 10
>
> The node distances above led to 2 problems:
>
> 1. Asymmetric routes taken between nodes in different packages led to
> asymmetric scheduler domain perspective depending on which node you
> are on. Current scheduler code failed to build domains properly with
> asymmetric distances.
>
> 2. Multiple remote distances to respective tiles on remote package create
> too many levels of domain hierarchies grouping different nodes between
> remote packages.
>
> For example, the above GNR topology lead to NUMA domains below:
>
> Sched domains from the perspective of a CPU in node 0, where the number
> in bracket represent node number.
>
> NUMA-level 1 [0,1] [2]
> NUMA-level 2 [0,1,2] [3]
> NUMA-level 3 [0,1,2,3] [5]
> NUMA-level 4 [0,1,2,3,5] [4]
>
> Sched domains from the perspective of a CPU in node 4
> NUMA-level 1 [4] [3,5]
> NUMA-level 2 [3,4,5] [0,2]
> NUMA-level 3 [0,2,3,4,5] [1]
>
> Scheduler group peers for load balancing from the perspective of CPU 0
> and 4 are different. Improper task could be chosen for load balancing
> between groups such as [0,2,3,4,5] [1]. Ideally you should choose nodes
> in 0 or 2 that are in same package as node 1 first. But instead tasks
> in the remote package node 3, 4, 5 could be chosen with an equal chance
> and could lead to excessive remote package migrations and imbalance of
> load between packages. We should not group partial remote nodes and
> local nodes together.
> Simplify the remote distances for CWF and GNR for the purpose of
> sched domains building, which maintains symmetry and leads to a more
> reasonable load balance hierarchy.
>
> The sched domains from the perspective of a CPU in node 0 NUMA-level 1
> is now
> NUMA-level 1 [0,1] [2]
> NUMA-level 2 [0,1,2] [3,4,5]
>
> The sched domains from the perspective of a CPU in node 4 NUMA-level 1
> is now
> NUMA-level 1 [4] [3,5]
> NUMA-level 2 [3,4,5] [0,1,2]
>
> We have the same balancing perspective from node 0 or node 4. Loads are
> now balanced equally between packages.
>
> Co-developed-by: Vinicius Costa Gomes <vinicius.gomes@...el.com>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@...el.com>
> Signed-off-by: Tim Chen <tim.c.chen@...ux.intel.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
> Reviewed-by: Chen Yu <yu.c.chen@...el.com>
> Tested-by: Zhao Liu <zhao1.liu@...el.com>
> ---
> arch/x86/kernel/smpboot.c | 70 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 70 insertions(+)
>
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index eb289abece23..5709c9cab195 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -515,6 +515,76 @@ static void __init build_sched_topology(void)
> set_sched_topology(topology);
> }
>
> +#ifdef CONFIG_NUMA
> +static int sched_avg_remote_distance;
> +static int avg_remote_numa_distance(void)
> +{
> + int i, j;
> + int distance, nr_remote, total_distance;
> +
> + if (sched_avg_remote_distance > 0)
> + return sched_avg_remote_distance;
> +
> + nr_remote = 0;
> + total_distance = 0;
> + for_each_node_state(i, N_CPU) {
> + for_each_node_state(j, N_CPU) {
> + distance = node_distance(i, j);
> +
> + if (distance >= REMOTE_DISTANCE) {
> + nr_remote++;
> + total_distance += distance;
> + }
> + }
> + }
> + if (nr_remote)
> + sched_avg_remote_distance = total_distance / nr_remote;
> + else
> + sched_avg_remote_distance = REMOTE_DISTANCE;
> +
> + return sched_avg_remote_distance;
> +}
> +
> +int arch_sched_node_distance(int from, int to)
> +{
> + int d = node_distance(from, to);
> +
> + switch (boot_cpu_data.x86_vfm) {
> + case INTEL_GRANITERAPIDS_X:
> + case INTEL_ATOM_DARKMONT_X:
> +
> + if (!x86_has_numa_in_package || topology_max_packages() == 1 ||
> + d < REMOTE_DISTANCE)
> + return d;
> +
> + /*
> + * With SNC enabled, there could be too many levels of remote
> + * NUMA node distances, creating NUMA domain levels
> + * including local nodes and partial remote nodes.
> + *
> + * Trim finer distance tuning for NUMA nodes in remote package
> + * for the purpose of building sched domains. Group NUMA nodes
> + * in the remote package in the same sched group.
> + * Simplify NUMA domains and avoid extra NUMA levels including
> + * different remote NUMA nodes and local nodes.
> + *
> + * GNR and CWF don't expect systems with more than 2 packages
> + * and more than 2 hops between packages. Single average remote
> + * distance won't be appropriate if there are more than 2
> + * packages as average distance to different remote packages
> + * could be different.
> + */
> + WARN_ONCE(topology_max_packages() > 2,
> + "sched: Expect only up to 2 packages for GNR or CWF, "
> + "but saw %d packages when building sched domains.",
> + topology_max_packages());
> +
> + d = avg_remote_numa_distance();
> + }
> + return d;
> +}
> +#endif /* CONFIG_NUMA */
> +
> void set_cpu_sibling_map(int cpu)
> {
> bool has_smt = __max_threads_per_core > 1;
> --
> 2.32.0
>
>
This breaks the build:
CC arch/x86/kernel/smpboot.o
arch/x86/kernel/smpboot.c:548:5: error: no previous prototype for ‘arch_sched_node_distance’ [-Werror=missing-prototypes]
548 | int arch_sched_node_distance(int from, int to)
| ^~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
How was it tested?
thanks,
greg k-h
Powered by blists - more mailing lists