linux-kernel - Re: [PATCH v3 0/5] Defer throttle when task exits to user

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250804075204.GA496@bytedance>
Date: Mon, 4 Aug 2025 15:52:04 +0800
From: Aaron Lu <ziqianlu@...edance.com>
To: Matteo Martelli <matteo.martelli@...ethink.co.uk>
Cc: Valentin Schneider <vschneid@...hat.com>,
	Ben Segall <bsegall@...gle.com>,
	K Prateek Nayak <kprateek.nayak@....com>,
	Peter Zijlstra <peterz@...radead.org>,
	Chengming Zhou <chengming.zhou@...ux.dev>,
	Josh Don <joshdon@...gle.com>, Ingo Molnar <mingo@...hat.com>,
	Vincent Guittot <vincent.guittot@...aro.org>,
	Xi Wang <xii@...gle.com>, linux-kernel@...r.kernel.org,
	Juri Lelli <juri.lelli@...hat.com>,
	Dietmar Eggemann <dietmar.eggemann@....com>,
	Steven Rostedt <rostedt@...dmis.org>, Mel Gorman <mgorman@...e.de>,
	Chuyi Zhou <zhouchuyi@...edance.com>,
	Jan Kiszka <jan.kiszka@...mens.com>,
	Florian Bezdeka <florian.bezdeka@...mens.com>,
	Songtang Liu <liusongtang@...edance.com>
Subject: Re: [PATCH v3 0/5] Defer throttle when task exits to user

Hi Matteo,

On Fri, Aug 01, 2025 at 04:31:25PM +0200, Matteo Martelli wrote:
... ... 
> I encountered this issue on a test image with both PREEMPT_RT and
> CFS_BANDWIDTH kernel options enabled. The test image is based on
> freedesktop-sdk (v24.08.10) [1] with custom system configurations on
> top, and it was being run on qemu x86_64 with 4 virtual CPU cores. One
> notable system configuration is having most of system services running
> on a systemd slice, restricted on a single CPU core (with AllowedCPUs
> systemd option) and using CFS throttling (with CPUQuota systemd option).
> With this configuration I encountered RCU stalls during boots, I think
> because of the increased probability given by multiple processes being
> spawned simultaneously on the same core. After the first RCU stall, the
> system becomes unresponsive and successive RCU stalls are detected
> periodically. This seems to match with the deadlock situation described
> in your cover letter. I could only reproduce RCU stalls with the
> combination of both PREEMPT_RT and CFS_BANDWIDTH enabled.
> 
> I previously already tested this patch set at v2 (RFC) [2] on top of
> kernel v6.14 and v6.15. I've now retested it at v3 on top of kernel
> v6.16-rc7. I could no longer reproduce RCU stalls in all cases with the
> patch set applied. More specifically, in the last test I ran, without
> patch set applied, I could reproduce 32 RCU stalls in 24 hours, about 1
> or 2 every hour. In this test the system was rebooting just after the
> first RCU stall occurrence (through panic_on_rcu_stall=1 and panic=5
> kernel cmdline arguments) or after 100 seconds if no RCU stall occurred.
> This means the system rebooted 854 times in 24 hours (about 3.7%
> reproducibility). You can see below two RCU stall instances. I could not
> reproduce any RCU stall with the same test after applying the patch set.
> I obtained similar results while testing the patch set at v2 (RFC)[1].
> Another possibly interesting note is that the original custom
> configuration was with the slice CPUQuota=150%, then I retested it with
> CPUQuota=80%. The issue was reproducible in both configurations, notably
> even with CPUQuota=150% that to my understanding should correspond to no
> CFS throttling due to the CPU affinity set to 1 core only.

Agree. With cpu affinity set to 1 core, 150% quota should never hit. But
from the test results, it seems quota is hit somehow because if quota is
not hit, this series should make no difference.

Maybe fire a bpftrace script and see if quota is actually hit? A
reference script is here:
https://lore.kernel.org/lkml/20250521115115.GB24746@bytedance/

> 
> I also ran some quick tests with stress-ng and systemd CPUQuota parameter to
> verify that CFS throttling was behaving as expected. See details below after
> RCU stall logs.

Thanks for all these tests. If I read them correctly, in all these
tests, CFS throttling worked as expected. Right?

> 
> I hope this is helpful information and I can provide additional details if
> needed.
> 

Yes it's very helpful.

> Tested-by: Matteo Martelli <matteo.martelli@...ethink.co.uk>
> 

Thanks!

> [1]: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/releases/freedesktop-sdk-24.08.10
> [2]: https://lore.kernel.org/all/20250409120746.635476-1-ziqianlu@bytedance.com/
> 

I'll rebase this series after merge window for v6.17 is closed and
hopefully it's in good shape and maintainer will pick it up :)

Best regards,
Aaron