linux-kernel - RE: [report] Performance regressions introduced via "cpuidle: menu: Remove iowait influence" on 6.12.y

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <003101dc78de$a74ee2b0$f5eca810$@telus.net>
Date: Mon, 29 Dec 2025 08:17:39 -0800
From: "Doug Smythies" <dsmythies@...us.net>
To: "'Christian Loehle'" <christian.loehle@....com>,
	"'ALOK TIWARI'" <alok.a.tiwari@...cle.com>
Cc: <gregkh@...uxfoundation.org>,
	<sashal@...nel.org>,
	<linux-kernel@...r.kernel.org>,
	<stable@...r.kernel.org>,
	<linux-pm@...r.kernel.org>,
	<rafael.j.wysocki@...el.com>,
	<daniel.lezcano@...aro.org>,
	"Doug Smythies" <dsmythies@...us.net>
Subject: RE: [report] Performance regressions introduced via "cpuidle: menu: Remove iowait influence" on 6.12.y

On 2025.12.24 05:26 Christian Loehle wrote:
> On 12/24/25 12:18, ALOK TIWARI wrote: 
>> On 12/3/2025 11:01 PM, ALOK TIWARI wrote:
>>> Hi,
>>>
>>> I’m reporting a performance regression of up to 6% sequential I/O
>>> vdbench regression observed on 6.12.y kernel.
>>> While running performance benchmarks on v6.12.60 kernel the sequential I/O vdbench metrics are showing a 5-6% performance regression when compared to v6.12.48
>>>
>>> Bisect root cause commit
>>> ========================
>>> - commit b39b62075ab4 ("cpuidle: menu: Remove iowait influence")
>>>
>>> Things work fine again when the previously removed performance- multiplier code is added back.
>>>
>>> Test details
>>> ============
>>> The system is connected to a number of disks in disk array using multipathing and directio configuration in the vdbench profile.
>>>
>>> wd=wd1,sd=sd*,rdpct=0,seekpct=sequential,xfersize=128k
>>> rd=128k64T,wd=wd1,iorate=max,elapsed=600,interval=1,warmup=300,threads=64
>>>
>>>
>>> Thanks,
>>> Alok
>>>
>> 
>> Just a gentle ping in case this was missed.
>> Please let us know if we are missing anything or if there are additional things to consider.
>> 
>
> Hi Alok,
> indeed it was missed, sorry!
> The cpuidle sysfs dumps pre and post test would be interesting like so:
> cat /sys/devices/system/cpu/cpu*/cpuidle/state*/*
> for both would be helpful so I can see what actually changed.
> Or a trace with cpu_idle events.
> Additionally a latency distribution of the IO requests would be helpful to relate
> them to the cpuidle wakeups.
>
> Thanks,
> Christian

Hi All,

For what its worth:

I have tried to recreate the issue with some of my high I/O wait type tests.
I was not successful.
The differences in idle state usage is clearly visible but has no noticeable effect on the performance numbers.

Workflow 1: My version of "critical-jobs", an attempt to do similar to the non-free SPECjbb (that Artem sometimes reports on).
Sweep from 4000 to 4600 jobs per second, and 10 disk lookups per job,
and "500" (arbitrary) units of work per lookup. 500 Gigabyte data file:

No consistent job latency differences were observed. There were some differences to experimental error in managing disk caching.
The attached graphs detail the biggest differences in idle stats.

Workflow 2: Just do a sequential read the 500 Gigabyte data file (cpu affinity forced):
No difference for the kernel 6.12 cases, but an improvement for kernel 6.19-rc2.
Graph attached (the sometimes teo poor read rates is unrelated to these idle investigations (or so I claim without proof)).

Processor: Intel(R) Core(TM) i5-10600K CPU @ 4.10GHz, 6 cores 12 CPUs.
HWP: Enabled.

Legend:
revert = kernel 6.12-rc1, 9852d85ec9d4
with = kernel 6.12-rc1 + 38f83090f515 (the patch of concern)
menu = kernel 6.19-rc2 menu gov
teo = kernel 6.19-rc2 teo gov

... Doug


Download attachment "0_residency.png" of type "image/png" (48359 bytes)

Download attachment "2_above.png" of type "image/png" (86417 bytes)

Download attachment "2_usage.png" of type "image/png" (75270 bytes)

Download attachment "read-rates.png" of type "image/png" (71242 bytes)