linux-kernel - Performance Degradation After Upgrading to Kernel 6.8

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CACodVevaOp4f=Gg467_m-FAdQFceGQYr7_Ahtt6CfpDVQhAsjA@mail.gmail.com>
Date: Tue, 19 Nov 2024 16:30:02 -0800
From: Chenbo Lu <chenbo.lu@...yaviation.com>
To: stable@...r.kernel.org
Cc: regressions@...ts.linux.dev, mingo@...hat.com, peterz@...radead.org, 
	juri.lelli@...hat.com, linux-kernel@...r.kernel.org
Subject: Performance Degradation After Upgrading to Kernel 6.8

Hello,

I am experiencing a significant performance degradation after
upgrading my kernel from version 6.6 to 6.8 and would appreciate any
insights or suggestions.

I am running a high-load simulation system that spawns more than 1000
threads and the overall CPU usage is 30%+ . Most of the threads are
using real-time
scheduling (SCHED_RR), and the threads of a model are using
SCHED_DEADLINE. After upgrading the kernel, I noticed that the
execution time of my model has increased from 4.5ms to 6ms.

What I Have Done So Far:
1. I found this [bug
report](https://bugzilla.kernel.org/show_bug.cgi?id=219366#c7) and
reverted the commit efa7df3e3bb5da8e6abbe37727417f32a37fba47 mentioned
in the post. Unfortunately, this did not resolve the issue.
2. I performed a git bisect and found that after these two commits
related to scheduling (RT and deadline) were merged, the problem
happened. They are 612f769edd06a6e42f7cd72425488e68ddaeef0a,
5fe7765997b139e2d922b58359dea181efe618f9
After reverting these two commits, the model execution time improved
to around 5 ms.
3. I revert two more commits, and the execution time is back to 4.7ms:
63ba8422f876e32ee564ea95da9a7313b13ff0a1,
efa7df3e3bb5da8e6abbe37727417f32a37fba47

My questions are:
1.Has anyone else experienced similar performance degradation after
upgrading to kernel 6.8?
2.Can anyone explain why these two commits are causing the problem? I
am not very familiar with the kernel code and would appreciate any
insights.
3.Are there any additional settings or configurations I need to apply
when using kernel 6.8 to avoid this issue?

My CPU is AMD Ryzen Threadripper 3970X, and my desktop uses Ubuntu 24.04.

Thanks again for any insights or suggestions. Please let me know if
any other information is needed.

Best,
Chenbo

-- 
This email and any relevant attachments may include confidential and/or 
proprietary information.  Any distribution or use by anyone other than the 
intended recipient(s) or other than for the intended purpose(s) is 
prohibited and may be unlawful.  If you are not the intended recipient of 
this message, please notify the sender by replying to this message and then 
delete it from your system.