Message-ID: <20250212054113.19938-1-cpru@amazon.com>
Date: Tue, 11 Feb 2025 23:41:13 -0600
From: Cristian Prundeanu <cpru@...zon.com>
To: K Prateek Nayak <kprateek.nayak@....com>
CC: Cristian Prundeanu <cpru@...zon.com>, Hazem Mohamed Abuelfotoh
	<abuehaze@...zon.com>, Ali Saidi <alisaidi@...zon.com>, "Benjamin
 Herrenschmidt" <benh@...nel.crashing.org>, Geoff Blake <blakgeof@...zon.com>,
	Csaba Csoma <csabac@...zon.com>, Bjoern Doebel <doebel@...zon.com>, "Gautham
 Shenoy" <gautham.shenoy@....com>, Joseph Salisbury
	<joseph.salisbury@...cle.com>, Dietmar Eggemann <dietmar.eggemann@....com>,
	Ingo Molnar <mingo@...hat.com>, Peter Zijlstra <peterz@...radead.org>, "Linus
 Torvalds" <torvalds@...ux-foundation.org>, Borislav Petkov <bp@...en8.de>,
	<linux-arm-kernel@...ts.infradead.org>, <linux-kernel@...r.kernel.org>,
	<linux-tip-commits@...r.kernel.org>, <x86@...nel.org>
Subject: Re: [PATCH 0/2] [tip: sched/core] sched: Disable PLACE_LAG and RUN_TO_PARITY and move them to sysctl

Hi Prateek,

Thank you for the analysis details!

> Thank you for the reproducer. I haven't tried it yet (in part due
> to the slightly scary "Assumptions" section)

It wasn't meant to be scary, my apologies. It is meant to say that the 
reproducer will only perform testing-related tasks (which you'd normally 
do manually), without touching the infrastructure (firewall, networking, 
instance mangement, etc). As long as you set all that up the same way you 
do when you test manually, you will be fine. I'll clarify the README.

Should you run into any questions, please do not hesitate to contact me 
directly, and I'll help clear the path.

> v6.14-rc1                   baseline
> v6.5.0 (pre-EEVDF)          -0.95%
> v6.14-rc1 + NO_PL + NO_RTP  +6.06%

This is interesting. While you do reproduce the benefits of NO_PL+NO_RTP, 
your result shows no regression compared to the baseline CFS. I'm only 
speculating, but running both SUT and loadgen on the same machine is a 
large variation of the test setup, and can lead to result differences like 
this one.

> Digging through the scripts, I found that SCHED_BATCH setting is done
> via systemd in [3] via the "CPUSchedulingPolicy" parameter.
> [3] https://github.com/aws/repro-collection/blob/main/workloads/mysql/files/mysqld.service.tmpl

That is correct, the reproducer uses systemd to set the scheduler policy 
for mysqld.

> interestingly, if I do (version 1): [...]
> I more or less get the same results as baseline v6.14-rc1 (Weird!)
> But then if I do (version 2): [...]
> I see the performance reach to the same level as that with NO_PL +
> NO_RTP.

That's a good find. I will compare on my setup if performance changes when 
manually setting all mysqld tasks to SCHED_BATCH. And I haven't yet run 
perf sched stats on the reproducer, but it may hold useful insight. 
I'll follow up with more details as I gather them.

Your find also helps to point out that even when it works, SCHED_BATCH is 
a more complex and error prone mitigation than just disabling PL and RTP. 
The same reproducer setup that uses systemd to set SCHED_BATCH does show 
improvement in 6.12, but not in 6.13+. There may not even be a single 
approach that works well on both.

Conversely, setting NO_PLACE_LAG + NO_RUN_TO_PARITY is simply done at boot 
time, and does not require further user effort. It's even simpler if those 
two features are exposed via sysctl, making it trivial to pesist and query 
with standard Linux commands as needed. 

Peter, I've renewed my initial patch so it applies to the current 
sched/core, and removed the dependency on changing the default values 
first. I'd appreciate you considering it for merging [1].

[1] https://lore.kernel.org/20250212053644.14787-1-cpru@amazon.com

-Cristian

Powered by blists - more mailing lists