linux-kernel - Re: [PATCH 0/2] [tip: sched/core] sched: Disable PLACE_LAG and RUN_TO

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ca5c6d15-7205-45e0-96a9-e78ab2a52b45@oracle.com>
Date: Thu, 14 Nov 2024 15:10:57 -0500
From: Joseph Salisbury <joseph.salisbury@...cle.com>
To: Cristian Prundeanu <cpru@...zon.com>, linux-tip-commits@...r.kernel.org
Cc: linux-kernel@...r.kernel.org, Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>, x86@...nel.org,
        linux-arm-kernel@...ts.infradead.org,
        Bjoern Doebel <doebel@...zon.com>,
        Hazem Mohamed Abuelfotoh <abuehaze@...zon.com>,
        Geoff Blake <blakgeof@...zon.com>, Ali Saidi <alisaidi@...zon.com>,
        Csaba Csoma <csabac@...zon.com>
Subject: Re: [PATCH 0/2] [tip: sched/core] sched: Disable PLACE_LAG and
 RUN_TO_PARITY and move them to sysctl




On 10/17/24 01:19, Cristian Prundeanu wrote:
> This patchset disables the scheduler features PLACE_LAG and RUN_TO_PARITY
> and moves them to sysctl.
>
> Replacing CFS with the EEVDF scheduler in kernel 6.6 introduced
> significant performance degradation in multiple database-oriented
> workloads. This degradation manifests in all kernel versions using EEVDF,
> across multiple Linux distributions, hardware architectures (x86_64,
> aarm64, amd64), and CPU generations.
>
> For example, running mysql+hammerdb results in a 12-17% throughput
> reduction and 12-18% latency increase compared to kernel 6.5 (using
> default scheduler settings everywhere). The magnitude of this performance
> impact is comparable to the average performance difference of a CPU
> generation over its predecessor.
>
> Testing combinations of available scheduler features showed that the
> largest improvement (short of disabling all EEVDF features) came from
> disabling both PLACE_LAG and RUN_TO_PARITY:
>
> Kernel   | default  | NO_PLACE_LAG and
> aarm64   | config   | NO_RUN_TO_PARITY
> ---------+----------+-----------------
> 6.5      | baseline |  N/A
> 6.6      | -13.2%   | -6.8%
> 6.7      | -13.1%   | -6.0%
> 6.8      | -12.3%   | -6.5%
> 6.9      | -12.7%   | -6.9%
> 6.10     | -13.5%   | -5.8%
> 6.11     | -12.6%   | -5.8%
> 6.12-rc2 | -12.2%   | -8.9%
> ---------+----------+-----------------
>
> Kernel   | default  | NO_PLACE_LAG and
> x86_64   | config   | NO_RUN_TO_PARITY
> ---------+----------+-----------------
> 6.5      | baseline |  N/A
> 6.6      | -16.8%   | -10.8%
> 6.7      | -16.4%   |  -9.9%
> 6.8      | -17.2%   |  -9.5%
> 6.9      | -17.4%   |  -9.7%
> 6.10     | -16.5%   |  -9.0%
> 6.11     | -15.0%   |  -8.5%
> 6.12-rc2 | -12.7%   | -10.9%
> ---------+----------+-----------------
>
> While the long term approach is debugging and fixing the scheduler
> behavior, algorithm changes to address performance issues of this nature
> are specialized (and likely prolonged or open-ended) research. Until a
> change is identified which fixes the performance degradation, in the
> interest of a better out-of-the-box performance: (1) disable these
> features by default, and (2) expose these values in sysctl instead of
> debugfs, so they can be more easily persisted across reboots.
>
> Cristian Prundeanu (2):
>    sched: Disable PLACE_LAG and RUN_TO_PARITY
>    sched: Move PLACE_LAG and RUN_TO_PARITY to sysctl
>
>   include/linux/sched/sysctl.h |  8 ++++++++
>   kernel/sched/core.c          | 13 +++++++++++++
>   kernel/sched/fair.c          |  5 +++--
>   kernel/sched/features.h      | 10 ----------
>   kernel/sysctl.c              | 20 ++++++++++++++++++++
>   5 files changed, 44 insertions(+), 12 deletions(-)
>
Hi Cristian,

This is a confirmation that we are also seeing a 9% performance 
regression with the TPCC benchmark after v6.6-rc1.  We narrowed down the 
regression was caused due to commit:
86bfbb7ce4f6 ("sched/fair: Add lag based placement")

This regression was reported via this thread:
https://lore.kernel.org/lkml/1c447727-92ed-416c-bca1-a7ca0974f0df@oracle.com/

Phil Auld suggested to try turning off the PLACE_LAG sched feature. We 
tested with NO_PLACE_LAG and can confirm it brought back 5% of the 
performance loss.  We do not yet know what effect NO_PLACE_LAG will have 
on other benchmarks, but it indeed helps TPCC.

Thanks for the work to move PLACE_LAG and RUN_TO_PARITY to sysctl!

Joe