linux-kernel - Re: scheduler performance regression since v6.11

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d3c8527f-ffaf-4463-a305-17ca21a06ce8@meta.com>
Date: Mon, 12 May 2025 18:35:24 -0400
From: Chris Mason <clm@...a.com>
To: Peter Zijlstra <peterz@...radead.org>
Cc: linux-kernel@...r.kernel.org, Ingo Molnar <mingo@...nel.org>,
        dietmar.eggemann@....com, vschneid@...hat.com
Subject: Re: scheduler performance regression since v6.11



On 5/12/25 3:39 PM, Chris Mason wrote:
> 
> 
> On 5/12/25 2:08 PM, Peter Zijlstra wrote:
>> On Fri, May 09, 2025 at 09:49:55PM +0200, Peter Zijlstra wrote:
>>> 152 729288bc6856 kernel/sched: Fix util_est accounting for DELAY_DEQUEUE
>>> 	average rps: 1936829.47
>>> 	average rps: 1950715.10
>>>
>>> 153 84d265281d6c sched/pelt: Use rq_clock_task() for hw_pressure
>>> 	average rps: 2176857.32
>>> 	average rps: 2223004.23
>>
>> So, a little more data on this.
>>
>> The result appears stable, but reverting 729288bc6856 on master does not
>> seem to cure things.
> 
> 729288bc6856 is in that section of commits where the regression was ~1/2
> as bad.  (bad: 1.8M, good: 2M, middle ground: 1.9M) I get the same RPS
> with and without it.
> 
> I called it "good" in my bisect run, but I'll take a second pass through
> those potential suspects and see if the bad really starts there.

This commit drops us from 2M RPS down to 1.95M.  I did a bunch of runs
on the surrounding commits and the drop was very reliable:

commit 152e11f6df293e816a6a37c69757033cdc72667d
Author: Peter Zijlstra <peterz@...radead.org>
Date:   Thu May 23 12:25:32 2024 +0200

    sched/fair: Implement delayed dequeue


e1459a50ba31, which was the next earliest commit is consistently fast.


If I take 54a58a787791 or 54a58a787791 and turn off the
DELAY_DEQUEUE/ZERO features at run time, I don't get the performance
back.  But, if I patch them such that DELAY_DEQUEUE/ZERO default off

diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 7fdeb5576188c..94409e9831e97 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -37,8 +37,8 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)
  *
  * DELAY_ZERO clips the lag on dequeue (or wakeup) to 0.
  */
-SCHED_FEAT(DELAY_DEQUEUE, true)
-SCHED_FEAT(DELAY_ZERO, true)
+SCHED_FEAT(DELAY_DEQUEUE, false)
+SCHED_FEAT(DELAY_ZERO, false)

It runs at 2M QPS again.  If I enable DELAY_QUEUE/ZERO, I go back to 1.95M

This is roughly 1/2 of the total regression, v6.15 runs at 1.88M maybe
1.85M on a bad run.

Peter sent me a patch to try improving my ttwu_queue rate, I'll give
that a shot in the morning.

-chris