| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAKfTPtCafw+VzSbfGtC+hxjxy=ioN9CmbsJMiRum777ds6GhSw@mail.gmail.com>
Date: Mon, 24 Nov 2025 17:40:49 +0100
From: Vincent Guittot <vincent.guittot@...aro.org>
To: Christian Loehle <christian.loehle@....com>
Cc: Yu-Che Cheng <giver@...omium.org>, Sergey Senozhatsky <senozhatsky@...omium.org>,
"Rafael J. Wysocki" <rafael@...nel.org>, Viresh Kumar <viresh.kumar@...aro.org>,
Greg Kroah-Hartman <gregkh@...uxfoundation.org>, Tomasz Figa <tfiga@...omium.org>, stable@...r.kernel.org,
linux-pm@...r.kernel.org, linux-kernel@...r.kernel.org,
Lukasz Luba <lukasz.luba@....com>, Dietmar Eggemann <dietmar.eggemann@....com>
Subject: Re: stable 6.6: commit "sched/cpufreq: Rework schedutil governor
performance estimation' causes a regression
On Mon, 24 Nov 2025 at 17:30, Vincent Guittot
<vincent.guittot@...aro.org> wrote:
>
> On Fri, 21 Nov 2025 at 17:43, Christian Loehle <christian.loehle@....com> wrote:
> >
> > On 11/21/25 16:35, Christian Loehle wrote:
> > > On 11/21/25 15:37, Yu-Che Cheng wrote:
> > >> Hi Vincent,
> > >>
> > >> On Fri, Nov 21, 2025 at 10:00 PM Vincent Guittot <vincent.guittot@...aro.org>
> > >> wrote:
> > >>>
> > >>> On Fri, 21 Nov 2025 at 04:55, Sergey Senozhatsky
> > >>> <senozhatsky@...omium.org> wrote:
> > >>>>
> > >>>> Hi Christian,
> > >>>>
> > >>>> On (25/11/20 10:15), Christian Loehle wrote:
> > >>>>> On 11/20/25 04:45, Sergey Senozhatsky wrote:
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> We are observing a performance regression on one of our arm64
> > >> boards.
> > >>>>>> We tracked it down to the linux-6.6.y commit ada8d7fa0ad4
> > >> ("sched/cpufreq:
> > >>>
> > >>> You mentioned that you tracked down to linux-6.6.y but which kernel
> > >>> are you using ?
> > >>>
> > >>
> > >> We're using ChromeOS 6.6 kernel, which is currently on top of linux-v6.6.99.
> > >> But we've tested that the performance regression still happens on exactly
> > >> the same scheduler codes (`kernel/sched`) as upstream v6.6.99, compared to
> > >> those on v6.6.88.
> > >>
> > >>>>>> Rework schedutil governor performance estimation").
> > >>>>>>
> > >>>>>> UI speedometer benchmark:
> > >>>>>> w/commit: 395 +/-38
> > >>>>>> w/o commit: 439 +/-14
> > >>>>>>
> > >>>>>
> > >>>>> Hi Sergey,
> > >>>>> Would be nice to get some details. What board?
> > >>>>
> > >>>> It's an MT8196 chromebook.
> > >>>>
> > >>>>> What do the OPPs look like?
> > >>>>
> > >>>> How do I find that out?
> > >>>
> > >>> In /sys/kernel/debug/opp/cpu*/
> > >>> or
> > >>> /sys/devices/system/cpu/cpufreq/policy*/scaling_available_frequencies
> > >>> with related_cpus
> > >>>
> > >>
> > >> The energy model on the device is:
> > >>
> > >> CPU0-3:
> > >> +------------+------------+
> > >> | freq (khz) | power (uw) |
> > >> +============+============+
> > >> | 339000 | 34362 |
> > >> | 400000 | 42099 |
> > >> | 500000 | 52907 |
> > >> | 600000 | 63795 |
> > >> | 700000 | 74747 |
> > >> | 800000 | 88445 |
> > >> | 900000 | 101444 |
> > >> | 1000000 | 120377 |
> > >> | 1100000 | 136859 |
> > >> | 1200000 | 154162 |
> > >> | 1300000 | 174843 |
> > >> | 1400000 | 196833 |
> > >> | 1500000 | 217052 |
> > >> | 1600000 | 247844 |
> > >> | 1700000 | 281464 |
> > >> | 1800000 | 321764 |
> > >> | 1900000 | 352114 |
> > >> | 2000000 | 383791 |
> > >> | 2100000 | 421809 |
> > >> | 2200000 | 461767 |
> > >> | 2300000 | 503648 |
> > >> | 2400000 | 540731 |
> > >> +------------+------------+
> > >>
> > >> CPU4-6:
> > >> +------------+------------+
> > >> | freq (khz) | power (uw) |
> > >> +============+============+
> > >> | 622000 | 131738 |
> > >> | 700000 | 147102 |
> > >> | 800000 | 172219 |
> > >> | 900000 | 205455 |
> > >> | 1000000 | 233632 |
> > >> | 1100000 | 254313 |
> > >> | 1200000 | 288843 |
> > >> | 1300000 | 330863 |
> > >> | 1400000 | 358947 |
> > >> | 1500000 | 400589 |
> > >> | 1600000 | 444247 |
> > >> | 1700000 | 497941 |
> > >> | 1800000 | 539959 |
> > >> | 1900000 | 584011 |
> > >> | 2000000 | 657172 |
> > >> | 2100000 | 746489 |
> > >> | 2200000 | 822854 |
> > >> | 2300000 | 904913 |
> > >> | 2400000 | 1006581 |
> > >> | 2500000 | 1115458 |
> > >> | 2600000 | 1205167 |
> > >> | 2700000 | 1330751 |
> > >> | 2800000 | 1450661 |
> > >> | 2900000 | 1596740 |
> > >> | 3000000 | 1736568 |
> > >> | 3100000 | 1887001 |
> > >> | 3200000 | 2048877 |
> > >> | 3300000 | 2201141 |
> > >> +------------+------------+
> > >>
> > >> CPU7:
> > >>
> > >> +------------+------------+
> > >> | freq (khz) | power (uw) |
> > >> +============+============+
> > >> | 798000 | 320028 |
> > >> | 900000 | 330714 |
> > >> | 1000000 | 358108 |
> > >> | 1100000 | 384730 |
> > >> | 1200000 | 410669 |
> > >> | 1300000 | 438355 |
> > >> | 1400000 | 469865 |
> > >> | 1500000 | 502740 |
> > >> | 1600000 | 531645 |
> > >> | 1700000 | 560380 |
> > >> | 1800000 | 588902 |
> > >> | 1900000 | 617278 |
> > >> | 2000000 | 645584 |
> > >> | 2100000 | 698653 |
> > >> | 2200000 | 744179 |
> > >> | 2300000 | 810471 |
> > >> | 2400000 | 895816 |
> > >> | 2500000 | 985234 |
> > >> | 2600000 | 1097802 |
> > >> | 2700000 | 1201162 |
> > >> | 2800000 | 1332076 |
> > >> | 2900000 | 1439847 |
> > >> | 3000000 | 1575917 |
> > >> | 3100000 | 1741987 |
> > >> | 3200000 | 1877346 |
> > >> | 3300000 | 2161512 |
> > >> | 3400000 | 2437879 |
> > >> | 3500000 | 2933742 |
> > >> | 3600000 | 3322959 |
> > >> | 3626000 | 3486345 |
> > >> +------------+------------+
> > >>
> > >>>>
> > >>>>> Does this system use uclamp during the benchmark? How?
> > >>>>
> > >>>> How do I find that out?
> > >>>
> > >>> it can be set per cgroup
> > >>> /sys/fs/cgroup/system.slice/<name>/cpu.uclam.min|max
> > >>> or per task with sched_setattr()
> > >>>
> > >>> You most probably use it because it's the main reason for ada8d7fa0ad4
> > >>> to remove wrong overestimate of OPP
> > >>>
> > >>
> > >> For the speedometer case, yes, we set the uclamp.min to 20 for the whole
> > >> browser and UI (chrome).
> > >> There's no system-wide uclamp settings though.
> > >
> > > (From Sergey's traces)
> > > Per-cluster time‑weighted average frequency base => revert:
> > > little (cpu0–3, max 2.4 GHz): 0.746 GHz => 1.132 GHz (+51.6%)
> > > mid (cpu4–6, max 3.3 GHz): 1.043 GHz => 1.303 GHz (+24.9%)
> > > big (cpu7, max 3.626 GHz): 2.563 GHz => 3.116 GHz (+21.6%)
> > >
> > > And in particular time spent at OPPs (base => revert):
> > > Big core at upper 10%: 29.6% => 61.5%
> > > little cluster at 339 MHz: 50.1% => 1.0%
> >
> > Sorry, should be 1.0% => 50.1%
>
> Having in mind that we have uclamp min at 20% ~204, this means that
> the tasks are not put in little cluster after the revert so the little
> goes back to low freq but 204 is less than half of little capacity
As Christian said, it would be good to have a trace with scheduler
events. Having task and cpu util would be interesting too: perfetto
should record all that for you
>
>
> >
> > >
> > > Interesting that a uclamp.min of 20 (which shouldn't really have
> > > much affect on big CPU at all, with or without headroom AFAICS?)
> > > makes such a big difference here?
> >
> > Can we get a sched_switch / sched_migrate / sched_wakeup trace for this?
> > Perfetto would also do if that is better for you.
> >
> > >
> > >>
> > >> But we also found other performance regressions in an Android guest VM,
> > >> where there's no uclamp for the VM and vCPU processes from the host side.
> > >> Particularly, the RAR extraction throughput reduces about 20% in the RAR
> > >> app (from RARLAB).
> > >> Although it's hard to tell if this is some sort of a side-effect of the UI
> > >> regression as the UI is also running at the same time.
> > >>
> > > I'd be inclined to say that is because of the vastly different DVFS from the
> > > UI workload, yes.
> > >
> >
Powered by blists - more mailing lists