Message-ID: <55687bb5-7e8a-4d7d-a597-6f97087cab32@siemens.com>
Date: Tue, 15 Apr 2025 08:09:57 +0200
From: Jan Kiszka <jan.kiszka@...mens.com>
To: K Prateek Nayak <kprateek.nayak@....com>,
 Aaron Lu <ziqianlu@...edance.com>,
 Florian Bezdeka <florian.bezdeka@...mens.com>
Cc: Valentin Schneider <vschneid@...hat.com>, Ben Segall
 <bsegall@...gle.com>, Peter Zijlstra <peterz@...radead.org>,
 Josh Don <joshdon@...gle.com>, Ingo Molnar <mingo@...hat.com>,
 Vincent Guittot <vincent.guittot@...aro.org>, Xi Wang <xii@...gle.com>,
 linux-kernel@...r.kernel.org, Juri Lelli <juri.lelli@...hat.com>,
 Dietmar Eggemann <dietmar.eggemann@....com>,
 Steven Rostedt <rostedt@...dmis.org>, Mel Gorman <mgorman@...e.de>,
 Chengming Zhou <chengming.zhou@...ux.dev>,
 Chuyi Zhou <zhouchuyi@...edance.com>
Subject: Re: [RFC PATCH v2 0/7] Defer throttle when task exits to user

On 15.04.25 08:05, K Prateek Nayak wrote:
> Hello Jan,
> 
> On 4/15/2025 10:59 AM, Jan Kiszka wrote:
>> On 14.04.25 14:04, Aaron Lu wrote:
>>> Hi Florian,
>>>
>>> On Mon, Apr 14, 2025 at 10:54:48AM +0200, Florian Bezdeka wrote:
>>>> Hi Aaron, Hi Valentin,
>>>>
>>>> On Wed, 2025-04-09 at 20:07 +0800, Aaron Lu wrote:
>>>>> This is a continuous work based on Valentin Schneider's posting here:
>>>>> Subject: [RFC PATCH v3 00/10] sched/fair: Defer CFS throttle to
>>>>> user entry
>>>>> https://lore.kernel.org/lkml/20240711130004.2157737-1-
>>>>> vschneid@...hat.com/
>>>>>
>>>>> Valentin has described the problem very well in the above link. We
>>>>> also
>>>>> have task hung problem from time to time in our environment due to
>>>>> cfs quota.
>>>>> It is mostly visible with rwsem: when a reader is throttled, writer
>>>>> comes in
>>>>> and has to wait, the writer also makes all subsequent readers wait,
>>>>> causing problems of priority inversion or even whole system hung.
>>>>
>>>> for testing purposes I backported this series to 6.14. We're currently
>>>> hunting for a sporadic bug with PREEMPT_RT enabled. We see RCU stalls
>>>> and complete system freezes after a couple of days with some container
>>>> workload deployed. See [1].
>>>
>>> I tried to make a setup last week to reproduce the RT/cfs throttle
>>> deadlock issue Valentin described but haven't succeeded yet...
>>>
>>
>> Attached the bits with which we succeeded, sometimes. Setup: Debian 12,
>> RT kernel, 2-4 cores VM, 1-5 instances of the test, 2 min - 2 h
>> patience. As we have to succeed with at least 3 race conditions in a
>> row, that is still not bad... But maybe someone has an idea how to
>> increase probabilities further.
> 
> Looking at run.sh, there are only fair tasks with one of them being run
> with cfs bandwidth constraints. Are you saying something goes wrong on
> PREEMPT_RT as a result of using bandwidth control on fair tasks?

Yes, exactly. Also our in-field workload that triggers (most likely)
this issue is not using RT tasks itself. Only kernel threads are RT here.

> 
> What exactly is the symptom you are observing? Does one of the assert()
> trip during the run? Do you see a stall logged on dmesg? Can you provide
> more information on what to expect in this 2min - 2hr window?

I've just lost my traces from yesterday ("you have 0 minutes to find a
power adapter"), but I got nice RCU stall warnings in the VM, including
backtraces from the involved tasks (minus the read-lock holder IIRC).
Maybe Florian can drop one of his dumps.

> 
> Additionally, do you have RT throttling enabled in your setup? Can long
> running RT tasks starve fair tasks on your setup?

RT throttling is enabled (default settings) but was not kicking in - why
should it in that scenario? The only RT thread, ktimers, ran into the
held lock and stopped.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Powered by blists - more mailing lists