Message-ID: <3cfa5690-644b-ba80-3fc3-7c5a3f292e70@arm.com>
Date:   Thu, 10 Jun 2021 12:07:02 +0200
From:   Dietmar Eggemann <dietmar.eggemann@....com>
To:     Lukasz Luba <lukasz.luba@....com>
Cc:     linux-kernel@...r.kernel.org, linux-pm@...r.kernel.org,
        peterz@...radead.org, rjw@...ysocki.net, viresh.kumar@...aro.org,
        vincent.guittot@...aro.org, qperret@...gle.com,
        vincent.donnefort@....com, Beata.Michalska@....com,
        mingo@...hat.com, juri.lelli@...hat.com, rostedt@...dmis.org,
        segall@...gle.com, mgorman@...e.de, bristot@...hat.com
Subject: Re: [PATCH v2 1/2] sched/fair: Take thermal pressure into account
 while estimating energy

On 10/06/2021 11:04, Lukasz Luba wrote:
> 
> 
> On 6/10/21 9:42 AM, Dietmar Eggemann wrote:
> 
> [snip]
> 
>>
>> So essentially what you want to do is:
>>
>> Make EAS aware of the frequency clamping schedutil can be faced with:
>>
>>    get_next_freq() -> cpufreq_driver_resolve_freq() ->
>> clamp_val(target_freq, policy->min, policy->max) (1)
>>
>> by subtracting CPU's Thermal Pressure (ThPr) signal from the original
>> CPU capacity `arch_scale_cpu_capacity()` (2).
>>
>> ---
>>
>> Isn't there a conceptional flaw in this design? Let's say we have a
>> big.Little system with two cpufreq cooling devices and a thermal zone
>> (something like Hikey 960). To create a ThPr scenario we have to run
>> stuff on the CPUs (e.g. hackbench (3)).
>> Eventually cpufreq_set_cur_state() [drivers/thermal/cpufreq_cooling.c]
>> will set thermal_pressure to `(2) - (2)*freq/policy->cpuinfo.max_freq`
>> and PELT will provide the ThPr signal via thermal_load_avg().
>> But to create this scenario, the system will become overutilized
>> (system-wide data, if one CPU is overutilized, the whole system is) so
>> EAS is disabled (i.e. find_energy_efficient_cpu() and compute_emergy()
>> are not executed).
> 
> Not always, it depends on thermal governor decision, workload and
> 'power actors' (in IPA naming convention). Then it depends when and how
> hard you clamp the CPUs. They (CPUs) don't have to be always
> overutilized, they might be even 50-70% utilized but the GPU reduced
> power budget by 2 Watts, so CPUs left with only 1W. Which is still OK
> for the CPUs, since they are only 'feeding' the GPU with new 'jobs'.

All this pretty much confines the usefulness of you proposed change. A
precise description of it with the patches is necessary to allow people
to start from there while exploring your patches.

>> I can see that there are episodes in which EAS is running and
>> thermal_load_avg() != 0 but those have to be when (3) has stopped and
>> you see the ThPr signal just decaying (no accruing of new ThPr). The
>> cpufreq cooling device can still issue cpufreq_set_cur_state() but only
>> with decreasing states.
> 
> It is true for some CPU heavy workloads, when no other SoC components
> are involved like: GPU, DSP, ISP, encoders, etc. For other workloads
> when CPUs don't have to do a lot, but thermal pressure might be seen on
> them, this patch help.
> 
>>
>> ---
>>
>> IMHO, a precise description of how you envision the system setup,
>> incorporating all participating subsystems, would be helpful here.
> 
> True, I hope this description above would help to understand the
> scenario.

This description belongs in the patch header. The scenario in which your
functionality would improve things has to be clear.
I'm sure that not everybody looking at this patches is immediately aware
on how IPA setups work and which specific setup you have in mind here.

>>> Signed-off-by: Lukasz Luba <lukasz.luba@....com>
>>> ---
>>>   kernel/sched/fair.c | 17 ++++++++++++++---
>>>   1 file changed, 14 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index 161b92aa1c79..1aeddecabc20 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>> @@ -6527,6 +6527,7 @@ compute_energy(struct task_struct *p, int
>>> dst_cpu, struct perf_domain *pd)
>>>       struct cpumask *pd_mask = perf_domain_span(pd);
>>>       unsigned long cpu_cap =
>>> arch_scale_cpu_capacity(cpumask_first(pd_mask));
>>>       unsigned long max_util = 0, sum_util = 0;
>>> +    unsigned long _cpu_cap = cpu_cap;
>>>       int cpu;
>>>         /*
>>> @@ -6558,14 +6559,24 @@ compute_energy(struct task_struct *p, int
>>> dst_cpu, struct perf_domain *pd)
>>>                   cpu_util_next(cpu, p, -1) + task_util_est(p);
>>>           }
>>>   +        /*
>>> +         * Take the thermal pressure from non-idle CPUs. They have
>>> +         * most up-to-date information. For idle CPUs thermal pressure
>>> +         * signal is not updated so often.
>>> +         */
>>> +        if (!idle_cpu(cpu))
>>> +            _cpu_cap = cpu_cap - thermal_load_avg(cpu_rq(cpu));
>>> +
>>
>> This one is probably the result of the fact that cpufreq cooling device
>> sets the ThPr for all CPUs of the policy (Frequency Domain (FD) or
>> Performance Domain (PD)) but PELT updates are happening per-CPU. And
>> only !idle CPUs get the update in scheduler_tick().
>>
>> Looks like thermal_pressure [per_cpu(thermal_pressure, cpu),
>> drivers/base/arch_topology.c] set by cpufreq_set_cur_state() is always
>> in sync with policy->max/cpuinfo_max_freq).
>> So for your use case this instantaneous `signal` is better than the PELT
>> one. It's precise (no decaying when frequency clamping is already gone)
>> and you avoid the per-cpu update issue.
> 
> Yes, this code implementation tries to address those issues.

The point I was making here is: why using the PELT signal
thermal_load_avg() and not per_cpu(thermal_pressure, cpu) directly,
given the fact that the latter perfectly represents the frequency clamping?

Powered by blists - more mailing lists