Message-ID: <CAKfTPtAxu5yac6teiFcPkrR-6Ui=J1Q1q7+-PQ6iNjEZP_yuyg@mail.gmail.com>
Date: Mon, 24 Nov 2025 17:30:39 +0100
From: Vincent Guittot <vincent.guittot@...aro.org>
To: Christian Loehle <christian.loehle@....com>
Cc: Yu-Che Cheng <giver@...omium.org>, Sergey Senozhatsky <senozhatsky@...omium.org>, 
	"Rafael J. Wysocki" <rafael@...nel.org>, Viresh Kumar <viresh.kumar@...aro.org>, 
	Greg Kroah-Hartman <gregkh@...uxfoundation.org>, Tomasz Figa <tfiga@...omium.org>, stable@...r.kernel.org, 
	linux-pm@...r.kernel.org, linux-kernel@...r.kernel.org, 
	Lukasz Luba <lukasz.luba@....com>, Dietmar Eggemann <dietmar.eggemann@....com>
Subject: Re: stable 6.6: commit "sched/cpufreq: Rework schedutil governor
 performance estimation' causes a regression

On Fri, 21 Nov 2025 at 17:43, Christian Loehle <christian.loehle@....com> wrote:
>
> On 11/21/25 16:35, Christian Loehle wrote:
> > On 11/21/25 15:37, Yu-Che Cheng wrote:
> >> Hi Vincent,
> >>
> >> On Fri, Nov 21, 2025 at 10:00 PM Vincent Guittot <vincent.guittot@...aro.org>
> >> wrote:
> >>>
> >>> On Fri, 21 Nov 2025 at 04:55, Sergey Senozhatsky
> >>> <senozhatsky@...omium.org> wrote:
> >>>>
> >>>> Hi Christian,
> >>>>
> >>>> On (25/11/20 10:15), Christian Loehle wrote:
> >>>>> On 11/20/25 04:45, Sergey Senozhatsky wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> We are observing a performance regression on one of our arm64
> >> boards.
> >>>>>> We tracked it down to the linux-6.6.y commit ada8d7fa0ad4
> >> ("sched/cpufreq:
> >>>
> >>> You mentioned that you tracked down to linux-6.6.y but which kernel
> >>> are you using ?
> >>>
> >>
> >> We're using ChromeOS 6.6 kernel, which is currently on top of linux-v6.6.99.
> >> But we've tested that the performance regression still happens on exactly
> >> the same scheduler codes (`kernel/sched`) as upstream v6.6.99, compared to
> >> those on v6.6.88.
> >>
> >>>>>> Rework schedutil governor performance estimation").
> >>>>>>
> >>>>>> UI speedometer benchmark:
> >>>>>> w/commit:   395  +/-38
> >>>>>> w/o commit: 439  +/-14
> >>>>>>
> >>>>>
> >>>>> Hi Sergey,
> >>>>> Would be nice to get some details. What board?
> >>>>
> >>>> It's an MT8196 chromebook.
> >>>>
> >>>>> What do the OPPs look like?
> >>>>
> >>>> How do I find that out?
> >>>
> >>> In /sys/kernel/debug/opp/cpu*/
> >>> or
> >>> /sys/devices/system/cpu/cpufreq/policy*/scaling_available_frequencies
> >>> with related_cpus
> >>>
> >>
> >> The energy model on the device is:
> >>
> >> CPU0-3:
> >> +------------+------------+
> >> | freq (khz) | power (uw) |
> >> +============+============+
> >> |     339000 |      34362 |
> >> |     400000 |      42099 |
> >> |     500000 |      52907 |
> >> |     600000 |      63795 |
> >> |     700000 |      74747 |
> >> |     800000 |      88445 |
> >> |     900000 |     101444 |
> >> |    1000000 |     120377 |
> >> |    1100000 |     136859 |
> >> |    1200000 |     154162 |
> >> |    1300000 |     174843 |
> >> |    1400000 |     196833 |
> >> |    1500000 |     217052 |
> >> |    1600000 |     247844 |
> >> |    1700000 |     281464 |
> >> |    1800000 |     321764 |
> >> |    1900000 |     352114 |
> >> |    2000000 |     383791 |
> >> |    2100000 |     421809 |
> >> |    2200000 |     461767 |
> >> |    2300000 |     503648 |
> >> |    2400000 |     540731 |
> >> +------------+------------+
> >>
> >> CPU4-6:
> >> +------------+------------+
> >> | freq (khz) | power (uw) |
> >> +============+============+
> >> |     622000 |     131738 |
> >> |     700000 |     147102 |
> >> |     800000 |     172219 |
> >> |     900000 |     205455 |
> >> |    1000000 |     233632 |
> >> |    1100000 |     254313 |
> >> |    1200000 |     288843 |
> >> |    1300000 |     330863 |
> >> |    1400000 |     358947 |
> >> |    1500000 |     400589 |
> >> |    1600000 |     444247 |
> >> |    1700000 |     497941 |
> >> |    1800000 |     539959 |
> >> |    1900000 |     584011 |
> >> |    2000000 |     657172 |
> >> |    2100000 |     746489 |
> >> |    2200000 |     822854 |
> >> |    2300000 |     904913 |
> >> |    2400000 |    1006581 |
> >> |    2500000 |    1115458 |
> >> |    2600000 |    1205167 |
> >> |    2700000 |    1330751 |
> >> |    2800000 |    1450661 |
> >> |    2900000 |    1596740 |
> >> |    3000000 |    1736568 |
> >> |    3100000 |    1887001 |
> >> |    3200000 |    2048877 |
> >> |    3300000 |    2201141 |
> >> +------------+------------+
> >>
> >> CPU7:
> >>
> >> +------------+------------+
> >> | freq (khz) | power (uw) |
> >> +============+============+
> >> |     798000 |     320028 |
> >> |     900000 |     330714 |
> >> |    1000000 |     358108 |
> >> |    1100000 |     384730 |
> >> |    1200000 |     410669 |
> >> |    1300000 |     438355 |
> >> |    1400000 |     469865 |
> >> |    1500000 |     502740 |
> >> |    1600000 |     531645 |
> >> |    1700000 |     560380 |
> >> |    1800000 |     588902 |
> >> |    1900000 |     617278 |
> >> |    2000000 |     645584 |
> >> |    2100000 |     698653 |
> >> |    2200000 |     744179 |
> >> |    2300000 |     810471 |
> >> |    2400000 |     895816 |
> >> |    2500000 |     985234 |
> >> |    2600000 |    1097802 |
> >> |    2700000 |    1201162 |
> >> |    2800000 |    1332076 |
> >> |    2900000 |    1439847 |
> >> |    3000000 |    1575917 |
> >> |    3100000 |    1741987 |
> >> |    3200000 |    1877346 |
> >> |    3300000 |    2161512 |
> >> |    3400000 |    2437879 |
> >> |    3500000 |    2933742 |
> >> |    3600000 |    3322959 |
> >> |    3626000 |    3486345 |
> >> +------------+------------+
> >>
> >>>>
> >>>>> Does this system use uclamp during the benchmark? How?
> >>>>
> >>>> How do I find that out?
> >>>
> >>> it can be set per cgroup
> >>> /sys/fs/cgroup/system.slice/<name>/cpu.uclam.min|max
> >>> or per task with sched_setattr()
> >>>
> >>> You most probably use it because it's the main reason for ada8d7fa0ad4
> >>> to remove wrong overestimate of OPP
> >>>
> >>
> >> For the speedometer case, yes, we set the uclamp.min to 20 for the whole
> >> browser and UI (chrome).
> >> There's no system-wide uclamp settings though.
> >
> > (From Sergey's traces)
> > Per-cluster time‑weighted average frequency base => revert:
> > little (cpu0–3, max 2.4 GHz): 0.746 GHz => 1.132 GHz (+51.6%)
> > mid (cpu4–6, max 3.3 GHz): 1.043 GHz => 1.303 GHz (+24.9%)
> > big (cpu7, max 3.626 GHz): 2.563 GHz => 3.116 GHz (+21.6%)
> >
> > And in particular time spent at OPPs (base => revert):
> > Big core at upper 10%: 29.6% => 61.5%
> > little cluster at 339 MHz: 50.1% => 1.0%
>
> Sorry, should be 1.0% => 50.1%

Having in mind that we have uclamp min at 20% ~204, this means that
the tasks are not put in little cluster after the revert so the little
goes back to low freq but 204 is less than half of little capacity


>
> >
> > Interesting that a uclamp.min of 20 (which shouldn't really have
> > much affect on big CPU at all, with or without headroom AFAICS?)
> > makes such a big difference here?
>
> Can we get a sched_switch / sched_migrate / sched_wakeup trace for this?
> Perfetto would also do if that is better for you.
>
> >
> >>
> >> But we also found other performance regressions in an Android guest VM,
> >> where there's no uclamp for the VM and vCPU processes from the host side.
> >> Particularly, the RAR extraction throughput reduces about 20% in the RAR
> >> app (from RARLAB).
> >> Although it's hard to tell if this is some sort of a side-effect of the UI
> >> regression as the UI is also running at the same time.
> >>
> > I'd be inclined to say that is because of the vastly different DVFS from the
> > UI workload, yes.
> >
>

Powered by blists - more mailing lists