Message-ID: <8ff19556-a656-4f11-a10c-6f9b92ec9cea@arm.com>
Date: Fri, 7 Feb 2025 15:55:09 +0000
From: Christian Loehle <christian.loehle@....com>
To: Jon Hunter <jonathanh@...dia.com>,
 Dietmar Eggemann <dietmar.eggemann@....com>,
 Juri Lelli <juri.lelli@...hat.com>
Cc: Thierry Reding <treding@...dia.com>, Waiman Long <longman@...hat.com>,
 Tejun Heo <tj@...nel.org>, Johannes Weiner <hannes@...xchg.org>,
 Michal Koutny <mkoutny@...e.com>, Ingo Molnar <mingo@...hat.com>,
 Peter Zijlstra <peterz@...radead.org>,
 Vincent Guittot <vincent.guittot@...aro.org>,
 Steven Rostedt <rostedt@...dmis.org>, Ben Segall <bsegall@...gle.com>,
 Mel Gorman <mgorman@...e.de>, Valentin Schneider <vschneid@...hat.com>,
 Phil Auld <pauld@...hat.com>, Qais Yousef <qyousef@...alina.io>,
 Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
 "Joel Fernandes (Google)" <joel@...lfernandes.org>,
 Suleiman Souhlal <suleiman@...gle.com>, Aashish Sharma <shraash@...gle.com>,
 Shin Kawamura <kawasin@...gle.com>,
 Vineeth Remanan Pillai <vineeth@...byteword.org>,
 linux-kernel@...r.kernel.org, cgroups@...r.kernel.org,
 "linux-tegra@...r.kernel.org" <linux-tegra@...r.kernel.org>
Subject: Re: [PATCH v2 3/2] sched/deadline: Check bandwidth overflow earlier
 for hotplug

On 2/7/25 14:04, Jon Hunter wrote:
> 
> 
> On 07/02/2025 13:38, Dietmar Eggemann wrote:
>> On 07/02/2025 11:38, Jon Hunter wrote:
>>>
>>> On 06/02/2025 09:29, Juri Lelli wrote:
>>>> On 05/02/25 16:56, Jon Hunter wrote:
>>>>
>>>> ...
>>>>
>>>>> Thanks! That did make it easier :-)
>>>>>
>>>>> Here is what I see ...
>>>>
>>>> Thanks!
>>>>
>>>> Still different from what I can repro over here, so, unfortunately, I
>>>> had to add additional debug printks. Pushed to the same branch/repo.
>>>>
>>>> Could I ask for another run with it? Please also share the complete
>>>> dmesg from boot, as I would need to check debug output when CPUs are
>>>> first onlined.
>>
>> So you have a system with 2 big and 4 LITTLE CPUs (Denver0 Denver1 A57_0
>> A57_1 A57_2 A57_3) in one MC sched domain and (Denver1 and A57_0) are
>> isol CPUs?
> 
> I believe that 1-2 are the denvers (even thought they are listed as 0-1 in device-tree).

Interesting, I have yet to reproduce this with equal capacities in isolcpus.
Maybe I didn't try hard enough yet.

> 
>> This should be easy to set up for me on my Juno-r0 [A53 A57 A57 A53 A53 A53]
> 
> Yes I think it is similar to this.
> 
> Thanks!
> Jon
> 

I could reproduce that on a different LLLLbb with isolcpus=3,4 (Lb) and
the offlining order:
echo 0 > /sys/devices/system/cpu/cpu5/online
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 0 > /sys/devices/system/cpu/cpu3/online
echo 0 > /sys/devices/system/cpu/cpu2/online
echo 0 > /sys/devices/system/cpu/cpu4/online

while the following offlining order succeeds:
echo 0 > /sys/devices/system/cpu/cpu5/online
echo 0 > /sys/devices/system/cpu/cpu4/online
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 0 > /sys/devices/system/cpu/cpu2/online
echo 0 > /sys/devices/system/cpu/cpu3/online
(Both offline an isolcpus last, both have CPU0 online)

The issue only triggers with sugov DL threads (I guess that's obvious, but
just to mention it).

I'll investigate some more later but wanted to share for now.
A log just to ensure we're looking at the same thing (this is just 6.14-rc1
with Juri's printk):

Successful offlining:
# echo 0 > /sys/devices/system/cpu/cpu5/online
[   37.063862] dl_bw_manage: cpu=5 cap=1143 fair_server_bw=52428 total_bw=314569 dl_bw_cpus=4
[   37.070925] CPU0 attaching NULL sched-domain.
[   37.071323] CPU1 attaching NULL sched-domain.
[   37.071743] CPU2 attaching NULL sched-domain.
[   37.072135] CPU5 attaching NULL sched-domain.
[   37.072618] CPU0 attaching sched-domain(s):
[   37.073008]  domain-0: span=0-2 level=MC
[   37.073370]   groups: 0:{ span=0 cap=379 }, 1:{ span=1 cap=380 }, 2:{ span=2 cap=381 }
[   37.074131] CPU1 attaching sched-domain(s):
[   37.074503]  domain-0: span=0-2 level=MC
[   37.074871]   groups: 1:{ span=1 cap=380 }, 2:{ span=2 cap=381 }, 0:{ span=0 cap=379 }
[   37.075614] CPU2 attaching sched-domain(s):
[   37.075998]  domain-0: span=0-2 level=MC
[   37.076354]   groups: 2:{ span=2 cap=381 }, 0:{ span=0 cap=379 }, 1:{ span=1 cap=380 }
[   37.077108] root domain span: 0-2
[   37.077425] rd 0-2: Checking EAS, CPUs do not have asymmetric capacities
[   37.078028] sched_energy_set: stopping EAS
[   37.086645] psci: CPU5 killed (polled 1 ms)
# echo 0 > /sys/devices/system/cpu/cpu4/online
[   40.357879] dl_bw_manage: cpu=4 cap=1024 fair_server_bw=52428 total_bw=209713 dl_bw_cpus=2
[   40.367705] rd 0-2: Checking EAS, CPUs do not have asymmetric capacities
[   40.379707] psci: CPU4 killed (polled 1 ms)
[   40.380449] rd 0-2: Checking EAS, CPUs do not have asymmetric capacities
# echo 0 > /sys/devices/system/cpu/cpu1/online
[   43.285829] dl_bw_manage: cpu=1 cap=762 fair_server_bw=52428 total_bw=262141 dl_bw_cpus=3
[   43.295728] CPU0 attaching NULL sched-domain.
[   43.296139] CPU1 attaching NULL sched-domain.
[   43.296535] CPU2 attaching NULL sched-domain.
[   43.297116] CPU0 attaching sched-domain(s):
[   43.297496]  domain-0: span=0,2 level=MC
[   43.297893]   groups: 0:{ span=0 cap=380 }, 2:{ span=2 cap=381 }
[   43.298491] CPU2 attaching sched-domain(s):
[   43.298891]  domain-0: span=0,2 level=MC
[   43.299257]   groups: 2:{ span=2 cap=381 }, 0:{ span=0 cap=380 }
[   43.299866] root domain span: 0,2
[   43.300203] rd 0,2: Checking EAS, CPUs do not have asymmetric capacities
[   43.315715] psci: CPU1 killed (polled 1 ms)
# echo 0 > /sys/devices/system/cpu/cpu2/online
[   46.975824] dl_bw_manage: cpu=2 cap=381 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=2
[   46.981728] CPU0 attaching NULL sched-domain.
[   46.982138] CPU2 attaching NULL sched-domain.
[   46.982649] CPU0 attaching NULL sched-domain.
[   46.983078] root domain span: 0
[   46.983399] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[   46.993699] psci: CPU2 killed (polled 1 ms)
# echo 0 > /sys/devices/system/cpu/cpu3/online
[   49.858842] dl_bw_manage: cpu=3 cap=0 fair_server_bw=52428 total_bw=52428 dl_bw_cpus=1
[   49.865730] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[   49.877688] psci: CPU3 killed (polled 2 ms)
# 

Failed offlining:
# echo 0 > /sys/devices/system/cpu/cpu5/online
[   29.596992] dl_bw_manage: cpu=5 cap=1143 fair_server_bw=52428 total_bw=314569 dl_bw_cpus=4
[   29.607022] CPU0 attaching NULL sched-domain.
[   29.607419] CPU1 attaching NULL sched-domain.
[   29.607838] CPU2 attaching NULL sched-domain.
[   29.608228] CPU5 attaching NULL sched-domain.
[   29.608732] CPU0 attaching sched-domain(s):
[   29.609124]  domain-0: span=0-2 level=MC
[   29.609485]   groups: 0:{ span=0 cap=380 }, 1:{ span=1 cap=381 }, 2:{ span=2 cap=381 }
[   29.610245] CPU1 attaching sched-domain(s):
[   29.610617]  domain-0: span=0-2 level=MC
[   29.610986]   groups: 1:{ span=1 cap=381 }, 2:{ span=2 cap=381 }, 0:{ span=0 cap=380 }
[   29.611731] CPU2 attaching sched-domain(s):
[   29.612122]  domain-0: span=0-2 level=MC
[   29.612478]   groups: 2:{ span=2 cap=381 }, 0:{ span=0 cap=380 }, 1:{ span=1 cap=381 }
[   29.613230] root domain span: 0-2
[   29.613547] rd 0-2: Checking EAS, CPUs do not have asymmetric capacities
[   29.614152] sched_energy_set: stopping EAS
[   29.629987] psci: CPU5 killed (polled 0 ms)
# echo 0 > /sys/devices/system/cpu/cpu1/online
[   32.945954] dl_bw_manage: cpu=1 cap=762 fair_server_bw=52428 total_bw=262141 dl_bw_cpus=3
[   32.955858] CPU0 attaching NULL sched-domain.
[   32.956269] CPU1 attaching NULL sched-domain.
[   32.956662] CPU2 attaching NULL sched-domain.
[   32.957244] CPU0 attaching sched-domain(s):
[   32.957624]  domain-0: span=0,2 level=MC
[   32.958021]   groups: 0:{ span=0 cap=380 }, 2:{ span=2 cap=381 }
[   32.958617] CPU2 attaching sched-domain(s):
[   32.959015]  domain-0: span=0,2 level=MC
[   32.959381]   groups: 2:{ span=2 cap=381 }, 0:{ span=0 cap=380 }
[   32.959993] root domain span: 0,2
[   32.960330] rd 0,2: Checking EAS, CPUs do not have asymmetric capacities
[   32.972841] psci: CPU1 killed (polled 1 ms)
# echo 0 > /sys/devices/system/cpu/cpu3/online
[   35.921962] dl_bw_manage: cpu=3 cap=1024 fair_server_bw=52428 total_bw=314570 dl_bw_cpus=2
[   35.927945] rd 0,2: Checking EAS, CPUs do not have asymmetric capacities
[   35.935828] psci: CPU3 killed (polled 1 ms)
# echo 0 > /sys/devices/system/cpu/cpu2/online
[   38.999952] dl_bw_manage: cpu=2 cap=381 fair_server_bw=52428 total_bw=209713 dl_bw_cpus=2
[   39.005003] CPU0 attaching NULL sched-domain.
[   39.005412] CPU2 attaching NULL sched-domain.
[   39.005969] CPU0 attaching NULL sched-domain.
[   39.006370] root domain span: 0
[   39.006687] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[   39.016825] psci: CPU2 killed (polled 1 ms)
# echo 0 > /sys/devices/system/cpu/cpu4/online
sh: write error: Device or resource busy
# [   42.936031] dl_bw_manage: cpu=4 cap=0 fair_server_bw=52428 total_bw=157285 dl_bw_cpus=1


cat /sys/devices/system/cpu/cpu*/cpu_capacity 
381
381
381
381
1024
1024

Powered by blists - more mailing lists