Message-ID: <20250818-mysterious-aromatic-wasp-cdbaae@sudeepholla>
Date: Mon, 18 Aug 2025 10:33:45 +0100
From: Sudeep Holla <sudeep.holla@....com>
To: Jeremy Linton <jeremy.linton@....com>
Cc: "Christoph Lameter (Ampere)" <cl@...two.org>,
	Huang Shijie <shijie@...amperecomputing.com>,
	Sudeep Holla <sudeep.holla@....com>, <catalin.marinas@....com>,
	<will@...nel.org>, <patches@...erecomputing.com>,
	<Shubhang@...amperecomputing.com>, <krzysztof.kozlowski@...aro.org>,
	<bjorn.andersson@....qualcomm.com>, <geert+renesas@...der.be>,
	<arnd@...db.de>, <nm@...com>, <ebiggers@...nel.org>,
	<nfraprado@...labora.com>,
	<prabhakar.mahadev-lad.rj@...renesas.com>,
	<linux-arm-kernel@...ts.infradead.org>,
	<linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] arm64: defconfig: enable CONFIG_SCHED_CLUSTER

On Fri, Aug 15, 2025 at 11:46:35AM -0500, Jeremy Linton wrote:
> Hi,
> 
> 
> On 8/15/25 5:48 AM, Sudeep Holla wrote:
> > On Thu, Aug 14, 2025 at 09:30:06AM -0700, Christoph Lameter (Ampere) wrote:
> > > On Thu, 14 Aug 2025, Sudeep Holla wrote:
> > > 
> > > >    |  Different architectures use different terminology to denominate logically
> > > >    |  associated processors, but terms such as package, cluster, module, and
> > > >    |  socket are typical examples.
> > > > 
> > > > So how can one use these across architectures ? Package/Socket is quite
> > > > standard. Cluster can be group of processors or it can also be group of
> > > > processor clusters. One of the Arm vendors call it super cluster or something.
> > > > All these makes it super hard for a generic OS to interpret that information.
> > > > Just CONFIG_SCHED_CLUSTER was added with one notion of cluster which was soon
> > > > realised doesn't match with some other notion of it.
> > > 
> > > What the cluster actually is used for is up to the hardware. The linux
> > > scheduler provides this functionality. How and when this feature is used
> > > by firmware is a vendor issue. There was never a clear definition.
> > > 
> > 
> > Sure, since it is left to architecture to define what it means, it could
> > work. But what happens if we have multiple chiplet inside a socket and
> > each chiplet has multiple cluster. Do you envision using this SCHED_CLUSTER
> > at chiplet level if that works best on the platform ?
> > 
> > That could work, but we need to document all these with the best of our
> > knowledge now so that it is easy to revisit in the future.
> > 
> > > > We can enable it and I am sure someone will report a regression on their
> > > > platform and we need to disable it again. The benchmark doesn't purely
> > > > depend on just the "notion" of cluster but it is often related to the
> > > > private resource and how they are shared in the system. So even if you
> > > > strictly follow the notion of cluster as supported by CONFIG_SCHED_CLUSTER
> > > > it will fail on systems where the private resources are shared across the
> > > > "cluster" boundaries or some variant configuration.
> > > 
> > > That is not our problem. If the vendor provides clustering information and
> > > the scheduler uses that then the vendor can modify the firmware to not
> > > enable clustering.
> > > 
> > 
> > That is pure wrong. ACPI is describing the hardware. Deciding to put
> > clustering information in these tables only if it provides performance or
> > not hinder performance seem complete non-sense to me. That covering policy
> > in ACPI hardware description. Does ACPI spec mention anything about it ?
> > I mean remove some hardware description even if it is 100% accurate if it
> > hinders performance on one of the OSPM ? Doesn't sound correct at all.
> > 
> > > As mentioned before: We could create a blacklist to override the ACPI info
> > > from the vendor to ensure that clustering is off.
> > > 
> > 
> > Not a bad idea. We can see if allow or blocklist works as we start with one.
> 
> From a distro perspective it makes more sense to me to change it from a
> compile time option to a runtime kernel command line option with the default
> on/off set by this SCHED_CLUSTER flag rather than try to maintain a
> blocklist.
>

Right, that makes complete sense to me.

> 
> I agree the firmware needs a much clearer way to signal that these nodes
> represent something other than just side effects of the way the table is
> built. If the working group is hesitant to declare additional topological
> flags, maybe this idea of deriving additional topological information from
> nodes without caches is a reasonable spec clarification. That way some
> future NODE_IS_A_CLUSTER/DSU/CHIPLET/SUPERCLUSTER/RING/SLICE/WHATEVER
> doesn't turn the existing code into technical debt.
> 

100% agreed.

> But returning to the original point, its not clear to me that the HW
> 'cluster' information is really causing the performance boost vs, just
> having a medium size scheduling domain (aka just picking an arbitrary size
> 4-16 cores) under MC, or simply 'slicing' a L3 in the PPTT such that the MC
> domains are smaller, yields the same effect. I've seen a number of cases
> where 'lying' about the topology yields a better result in a benchmark. This
> is largely what is happening with these Firmware toggles that move/remove
> the NUMA domains too. Being able to manually reconfigure some of these
> scheduling levels at runtime might be useful...
> 

I share your concern and hence completely again representation of any fake
data in the ACPI topology just to get improved performance. Yes we have seen
that in the past.

-- 
Regards,
Sudeep

Powered by blists - more mailing lists