linux-kernel - Re: BUG Report: Fork benchmark drop by 30% on aarch64

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20250207110754.GA10452@amazon.com>
Date: Fri, 7 Feb 2025 11:07:54 +0000
From: Hagar Hemdan <hagarhem@...zon.com>
To: Dietmar Eggemann <dietmar.eggemann@....com>
CC: Ingo Molnar <mingo@...hat.com>, Peter Zijlstra <peterz@...radead.org>,
	Juri Lelli <juri.lelli@...hat.com>, Vincent Guittot
	<vincent.guittot@...aro.org>, wuchi <wuchi.zero@...il.com>,
	<linux-kernel@...r.kernel.org>, <Mohamed@...zon.com>,
	<Abuelfotoh@...zon.com>, Hazem <abuehaze@...zon.com>
Subject: Re: BUG Report: Fork benchmark drop by 30% on aarch64

On Fri, Feb 07, 2025 at 10:14:54AM +0100, Dietmar Eggemann wrote:
> Hi Hagar,
> 
> On 05/02/2025 16:10, Hagar Hemdan wrote:
> > Hi,
> > 
> > There is about a 30% drop in fork benchmark [1] on aarch64 and a 10%
> > drop on x86_64 using kernel v6.13.1.
> > 
> > Git bisect pointed to commit eff6c8ce8d4d ("sched/core: Reduce cost
> > of sched_move_task when config autogroup") which merged starting
> > v6.4-rc1.
> > 
> > The regression only happens when number of CPUs is equal to number
> > of threads [2] that fork test is creating which means it's only visible
> > under CPU contention.
> > 
> > I used m6g.xlarge AWS EC2 Instance with 4 vCPUs and 16 GiB RAM for ARM64
> > and m6a.xlarge with also 4 vCPUs and 16 GiB RAM for x86_64.
> > 
> > I noticed this regression exists only when autogroup config is enabled.
> 
> So '# CONFIG_SCHED_AUTOGROUP is not set' in .config so we have: 
> 
> static inline void sched_autogroup_exit_task(struct task_struct *p) { }
> 
> I.e. doing a 'echo 0 > /proc/sys/kernel/sched_autogroup_enabled' still
> shows this issue?
yes, when I do 'echo 0 | sudo tee /proc/sys/kernel/sched_autogroup_enabled',
It behaves like the disable 'CONFIG_SCHED_AUTOGROUP'.
> 
> > 
> > Run the fork test with these combinations and autogroup is enabled:
> > 
> > Arch      | commit eff6c8ce8d4d | Fork Result (lps)  |  %Cpu(s)
> > ----------+---------------------+--------------------+------------------
> > aarch64   | without             | 28677.0            |  3.2 us, 96.7 sy
> > aarch64   | with                | 19860.7 (30% drop) |  2.7 us, 79.4 sy
> > x86_64    | without             | 27776.2            |  3.1 us, 96.9 sy
> > x86_64    | with                | 25020.6 (10% drop) |  4.1 us, 93.2 sy
> > ----------+---------------------+--------------------+------------------
> 
> Can you rerun with:
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 3e5a6bf587f9..62cc50c79a78 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -9057,7 +9057,7 @@ void sched_move_task(struct task_struct *tsk)
>          * group changes.
>          */
>         group = sched_get_task_group(tsk);
> -       if (group == tsk->sched_task_group)
> +       if ((group == tsk->sched_task_group) && !(tsk->flags & PF_EXITING))
>                 return;
I tried that and I see it fixed the regression and the cpu utilization
is 100% with it.

I'd like to ask if this like reverting the patch in case of exit path
and also means the enqueue/dequeue are needed in case of task exiting,
right?

Thanks for replying :)
> 
> > 
> > It seems that the commit is capping the amount of CPU resources that can
> > be utilized leaving around 18% idle in case of aarch64 and 3% idle in
> > x86_64 case which is likely the main reason behind the reported fork
> > regression.
> > 
> > When autogroup is disabled:
> > 
> > Arch      | commit eff6c8ce8d4d | Fork Result (lps)  |  %Cpu(s)
> > ----------+---------------------+--------------------+------------------
> > aarch64   | without             | 19877.8            |  2.2 us, 80.1 sy  
> > aarch64   | with                | 20086.3 (~same)    |  1.9 us, 80.2 sy
> > x86_64    | without             | 24974.2            |  4.9 us, 92.5 sy 
> > x86_64    | with                | 24921.5 (~same)    |  4.9 us, 92.4 sy
> > ----------+---------------------+--------------------+------------------
> > 
> > So when autogroup disabled, I still see the amount of idle CPU resources 
> > 18%, 3% on aarch64 and x86_64 regardless of commit.
> > 
> > Is this performance drop an expected of this commit when autogroup is
> > enabled?
> > 
> > Thanks,
> > Hagar
> > 
> > [1] https://github.com/kdlucas/byte-unixbench/blob/master/UnixBench
> > [2] Used command: ./Run -c 4 spawn
> > 
>