Message-ID: <8572b3bc-46ec-4180-ba55-aa6b9ab7502b@nvidia.com>
Date: Wed, 5 Feb 2025 16:56:30 +0000
From: Jon Hunter <jonathanh@...dia.com>
To: Juri Lelli <juri.lelli@...hat.com>
Cc: Thierry Reding <treding@...dia.com>, Waiman Long <longman@...hat.com>,
 Tejun Heo <tj@...nel.org>, Johannes Weiner <hannes@...xchg.org>,
 Michal Koutny <mkoutny@...e.com>, Ingo Molnar <mingo@...hat.com>,
 Peter Zijlstra <peterz@...radead.org>,
 Vincent Guittot <vincent.guittot@...aro.org>,
 Dietmar Eggemann <dietmar.eggemann@....com>,
 Steven Rostedt <rostedt@...dmis.org>, Ben Segall <bsegall@...gle.com>,
 Mel Gorman <mgorman@...e.de>, Valentin Schneider <vschneid@...hat.com>,
 Phil Auld <pauld@...hat.com>, Qais Yousef <qyousef@...alina.io>,
 Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
 "Joel Fernandes (Google)" <joel@...lfernandes.org>,
 Suleiman Souhlal <suleiman@...gle.com>, Aashish Sharma <shraash@...gle.com>,
 Shin Kawamura <kawasin@...gle.com>,
 Vineeth Remanan Pillai <vineeth@...byteword.org>,
 linux-kernel@...r.kernel.org, cgroups@...r.kernel.org,
 "linux-tegra@...r.kernel.org" <linux-tegra@...r.kernel.org>
Subject: Re: [PATCH v2 3/2] sched/deadline: Check bandwidth overflow earlier
 for hotplug

Hi Juri,

On 05/02/2025 10:12, Juri Lelli wrote:
> On 05/02/25 07:53, Juri Lelli wrote:
>> On 03/02/25 11:01, Jon Hunter wrote:
>>> Hi Juri,
>>>
>>> On 16/01/2025 15:55, Juri Lelli wrote:
>>>> On 16/01/25 13:14, Jon Hunter wrote:
>>
>> ...
>>
>>>>> [  210.595431] dl_bw_manage: cpu=5 cap=3072 fair_server_bw=52428 total_bw=209712 dl_bw_cpus=4
>>>>> [  210.606269] dl_bw_manage: cpu=4 cap=2048 fair_server_bw=52428 total_bw=157284 dl_bw_cpus=3
>>>>> [  210.617281] dl_bw_manage: cpu=3 cap=1024 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=2
>>>>> [  210.627205] dl_bw_manage: cpu=2 cap=1024 fair_server_bw=52428 total_bw=262140 dl_bw_cpus=2
>>>>> [  210.637752] dl_bw_manage: cpu=1 cap=0 fair_server_bw=52428 total_bw=262140 dl_bw_cpus=1
>>>>                                                                             ^
>>>> Different than before but still not what I expected. Looks like there
>>>> are conditions/path I currently cannot replicate on my setup, so more
>>>> thinking. Unfortunately I will be out traveling next week, so this
>>>> might required a bit of time.
>>>
>>>
>>> I see that this is now in the mainline and our board is still failing to
>>> suspend. Let me know if there is anything else you need me to test.
>>
>> Ah, can you actually add 'sched_verbose' and to your kernel cmdline? It
>> should print our additional debug info on the console when domains get
>> reconfigured by hotplug/suspends, e.g.
>>
>>   dl_bw_manage: cpu=3 cap=3072 fair_server_bw=52428 total_bw=209712 dl_bw_cpus=4
>>   CPU0 attaching NULL sched-domain.
>>   CPU3 attaching NULL sched-domain.
>>   CPU4 attaching NULL sched-domain.
>>   CPU5 attaching NULL sched-domain.
>>   CPU0 attaching sched-domain(s):
>>    domain-0: span=0,4-5 level=MC
>>     groups: 0:{ span=0 cap=766 }, 4:{ span=4 cap=908 }, 5:{ span=5 cap=989 }
>>   CPU4 attaching sched-domain(s):
>>    domain-0: span=0,4-5 level=MC
>>     groups: 4:{ span=4 cap=908 }, 5:{ span=5 cap=989 }, 0:{ span=0 cap=766 }
>>   CPU5 attaching sched-domain(s):
>>    domain-0: span=0,4-5 level=MC
>>     groups: 5:{ span=5 cap=989 }, 0:{ span=0 cap=766 }, 4:{ span=4 cap=908 }
>>   root domain span: 0,4-5
>>   rd 0,4-5: Checking EAS, CPUs do not have asymmetric capacities
>>   psci: CPU3 killed (polled 0 ms)
>>
>> Can you please share this information as well if you are able to collect
>> it (while still running with my last proposed fix)?
> 
> Also, if you don't mind, add the following on top of the existing
> changes.
> 
> Just to be sure we don't get out of sync, I pushed current set to
> 
> https://github.com/jlelli/linux.git experimental/dl-debug


Thanks! That did make it easier :-)

Here is what I see ...

[   53.823979] PM: suspend entry (deep)
[   53.827715] Filesystems sync: 0.000 seconds
[   53.832859] Freezing user space processes
[   53.838132] Freezing user space processes completed (elapsed 0.001 seconds)
[   53.845118] OOM killer disabled.
[   53.848348] Freezing remaining freezable tasks
[   53.853884] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[   53.900686] tegra-xusb 3530000.usb: Firmware timestamp: 2020-07-06 13:39:28 UTC
[   53.918492] dwc-eth-dwmac 2490000.ethernet eth0: Link is Down
[   53.962316] Disabling non-boot CPUs ...
[   53.966192] dl_bw_manage: cpu=5 cap=3072 fair_server_bw=52428 total_bw=209712 dl_bw_cpus=4 type=DYN span=0,3-5
[   53.966231] CPU0 attaching NULL sched-domain.
[   53.980574] span=1-2
[   53.982767] CPU3 attaching NULL sched-domain.
[   53.987119] span=0-2
[   53.989309] CPU4 attaching NULL sched-domain.
[   53.993662] span=0-3
[   53.995853] CPU5 attaching NULL sched-domain.
[   54.000206] span=0-4
[   54.002433] CPU0 attaching sched-domain(s):
[   54.006614]  domain-0: span=0,3-4 level=MC
[   54.010711]   groups: 0:{ span=0 cap=1022 }, 3:{ span=3 cap=1022 }, 4:{ span=4 }
[   54.018126] CPU3 attaching sched-domain(s):
[   54.022307]  domain-0: span=0,3-4 level=MC
[   54.026404]   groups: 3:{ span=3 cap=1022 }, 4:{ span=4 }, 0:{ span=0 cap=1023 }
[   54.033821] CPU4 attaching sched-domain(s):
[   54.038001]  domain-0: span=0,3-4 level=MC
[   54.042098]   groups: 4:{ span=4 }, 0:{ span=0 cap=1023 }, 3:{ span=3 cap=1022 }
[   54.049508] root domain span: 0,3-4
[   54.052997] default domain span: 1-2,5
[   54.056756] rd 0,3-4: Checking EAS, CPUs do not have asymmetric capacities
[   54.064688] psci: CPU5 killed (polled 0 ms)
[   54.069495] dl_bw_manage: cpu=4 cap=2048 fair_server_bw=52428 total_bw=157284 dl_bw_cpus=3 type=DYN span=0,3-4
[   54.069547] CPU0 attaching NULL sched-domain.
[   54.083910] span=1-2,5
[   54.086277] CPU3 attaching NULL sched-domain.
[   54.090633] span=0-2,5
[   54.092999] CPU4 attaching NULL sched-domain.
[   54.097351] span=0-3,5
[   54.099756] CPU0 attaching sched-domain(s):
[   54.103941]  domain-0: span=0,3 level=MC
[   54.107865]   groups: 0:{ span=0 }, 3:{ span=3 cap=1023 }
[   54.113279] CPU3 attaching sched-domain(s):
[   54.117459]  domain-0: span=0,3 level=MC
[   54.121382]   groups: 3:{ span=3 cap=1023 }, 0:{ span=0 }
[   54.126793] root domain span: 0,3
[   54.130109] default domain span: 1-2,4-5
[   54.134040] rd 0,3: Checking EAS, CPUs do not have asymmetric capacities
[   54.141597] psci: CPU4 killed (polled 0 ms)
[   54.146819] dl_bw_manage: cpu=3 cap=1024 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=2 type=DYN span=0,3
[   54.156727] CPU0 attaching NULL sched-domain.
[   54.161086] span=1-2,4-5
[   54.163632] CPU3 attaching NULL sched-domain.
[   54.167988] span=0-2,4-5
[   54.170553] CPU0 attaching NULL sched-domain.
[   54.174909] span=0-5
[   54.177096] root domain span: 0
[   54.180239] default domain span: 1-5
[   54.183821] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[   54.191728] psci: CPU3 killed (polled 4 ms)
[   54.196389] dl_bw_manage: cpu=2 cap=1024 fair_server_bw=52428 total_bw=262140 dl_bw_cpus=2 type=DEF span=1-5
[   54.196518] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[   54.214816] psci: CPU2 killed (polled 0 ms)
[   54.219411] dl_bw_manage: cpu=1 cap=0 fair_server_bw=52428 total_bw=262140 dl_bw_cpus=1 type=DEF span=1-5
[   54.219493] Error taking CPU1 down: -16
[   54.232948] Non-boot CPUs are not disabled
[   54.237046] Enabling non-boot CPUs ...
[   54.241216] Detected PIPT I-cache on CPU2
[   54.245258] CPU features: SANITY CHECK: Unexpected variation in SYS_CTR_EL0. Boot CPU: 0x0000008444c004, CPU2: 0x0000009444c004
[   54.256744] CPU features: SANITY CHECK: Unexpected variation in SYS_ID_AA64DFR0_EL1. Boot CPU: 0x00000010305106, CPU2: 0x00000010305116
[   54.268954] CPU features: SANITY CHECK: Unexpected variation in SYS_ID_DFR0_EL1. Boot CPU: 0x00000003010066, CPU2: 0x00000003001066
[   54.280865] CPU2: Booted secondary processor 0x0000000001 [0x4e0f0030]
[   54.288270] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[   54.295061] CPU2 is up
[   54.297599] Detected PIPT I-cache on CPU3
[   54.301642] CPU3: Booted secondary processor 0x0000000101 [0x411fd073]
[   54.308419] CPU0 attaching NULL sched-domain.
[   54.312786] span=1-5
[   54.315031] CPU0 attaching sched-domain(s):
[   54.319220]  domain-0: span=0,3 level=MC
[   54.323145]   groups: 0:{ span=0 }, 3:{ span=3 cap=1016 }
[   54.328564] CPU3 attaching sched-domain(s):
[   54.332746]  domain-0: span=0,3 level=MC
[   54.336671]   groups: 3:{ span=3 cap=1016 }, 0:{ span=0 }
[   54.342080] root domain span: 0,3
[   54.345405] default domain span: 1-2,4-5
[   54.349338] rd 0,3: Checking EAS, CPUs do not have asymmetric capacities
[   54.356122] CPU3 is up
[   54.358649] Detected PIPT I-cache on CPU4
[   54.362677] CPU4: Booted secondary processor 0x0000000102 [0x411fd073]
[   54.369399] CPU0 attaching NULL sched-domain.
[   54.373767] span=1-2,4-5
[   54.376310] CPU3 attaching NULL sched-domain.
[   54.380667] span=0-2,4-5
[   54.383251] CPU0 attaching sched-domain(s):
[   54.387439]  domain-0: span=0,3-4 level=MC
[   54.391538]   groups: 0:{ span=0 }, 3:{ span=3 cap=1021 }, 4:{ span=4 }
[   54.398173] CPU3 attaching sched-domain(s):
[   54.402356]  domain-0: span=0,3-4 level=MC
[   54.406456]   groups: 3:{ span=3 cap=1021 }, 4:{ span=4 }, 0:{ span=0 }
[   54.413090] CPU4 attaching sched-domain(s):
[   54.417271]  domain-0: span=0,3-4 level=MC
[   54.421373]   groups: 4:{ span=4 }, 0:{ span=0 }, 3:{ span=3 cap=1021 }
[   54.428005] root domain span: 0,3-4
[   54.431503] default domain span: 1-2,5
[   54.435259] rd 0,3-4: Checking EAS, CPUs do not have asymmetric capacities
[   54.442287] CPU4 is up
[   54.444821] Detected PIPT I-cache on CPU5
[   54.448848] CPU5: Booted secondary processor 0x0000000103 [0x411fd073]
[   54.455574] CPU0 attaching NULL sched-domain.
[   54.459950] span=1-2,5
[   54.462315] CPU3 attaching NULL sched-domain.
[   54.466674] span=0-2,5
[   54.469042] CPU4 attaching NULL sched-domain.
[   54.473401] span=0-3,5
[   54.475812] CPU0 attaching sched-domain(s):
[   54.480000]  domain-0: span=0,3-5 level=MC
[   54.484099]   groups: 0:{ span=0 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
[   54.491171] CPU3 attaching sched-domain(s):
[   54.495352]  domain-0: span=0,3-5 level=MC
[   54.499452]   groups: 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }
[   54.506519] CPU4 attaching sched-domain(s):
[   54.510703]  domain-0: span=0,3-5 level=MC
[   54.514800]   groups: 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }, 3:{ span=3 }
[   54.521869] CPU5 attaching sched-domain(s):
[   54.526050]  domain-0: span=0,3-5 level=MC
[   54.530150]   groups: 5:{ span=5 }, 0:{ span=0 }, 3:{ span=3 }, 4:{ span=4 }
[   54.537217] root domain span: 0,3-5
[   54.540716] default domain span: 1-2
[   54.544303] rd 0,3-5: Checking EAS, CPUs do not have asymmetric capacities
[   54.551393] rd 0,3-5: Checking EAS, CPUs do not have asymmetric capacities
[   54.558281] CPU5 is up
[   54.568000] dwc-eth-dwmac 2490000.ethernet eth0: configuring for phy/rgmii link mode
[   55.585391] dwc-eth-dwmac 2490000.ethernet: Failed to reset the dma
[   55.591664] dwc-eth-dwmac 2490000.ethernet eth0: stmmac_hw_setup: DMA engine initialization failed
[   55.600905] dwc-eth-dwmac 2490000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   55.615015] usb-conn-gpio 3520000.padctl:ports:usb2-0:connector: repeated role: device
[   55.617967] tegra-xusb 3530000.usb: Firmware timestamp: 2020-07-06 13:39:28 UTC
[   55.655665] OOM killer enabled.
[   55.658813] Restarting tasks ... done.
[   55.664082] random: crng reseeded on system resumption
[   55.664403] VDDIO_SDMMC3_AP: voltage operation not allowed
[   55.674862] PM: suspend exit

-- 
nvpublic

Powered by blists - more mailing lists