linux-kernel - Re: [RT BUG] Stall caused by eventpoll, rwlocks and CFS bandwidth controller

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <3f7b7ce1-6dd4-4a4e-9789-4c0cbde057bd@siemens.com>
Date: Wed, 9 Apr 2025 08:41:44 +0200
From: Jan Kiszka <jan.kiszka@...mens.com>
To: Valentin Schneider <vschneid@...hat.com>, linux-rt-users@...r.kernel.org,
 Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
 linux-kernel@...r.kernel.org, kprateek.nayak@....com
Cc: Thomas Gleixner <tglx@...utronix.de>, Juri Lelli <juri.lelli@...hat.com>,
 Clark Williams <williams@...hat.com>,
 "Luis Claudio R. Goncalves" <lgoncalv@...hat.com>,
 Andreas Ziegler <ziegler.andreas@...mens.com>,
 Felix Moessbauer <felix.moessbauer@...mens.com>,
 Florian Bezdeka <florian.bezdeka@...mens.com>
Subject: Re: [RT BUG] Stall caused by eventpoll, rwlocks and CFS bandwidth
 controller

On 12.10.23 17:07, Valentin Schneider wrote:
> Hi folks,
> 
> We've had reports of stalls happening on our v6.0-ish frankenkernels, and while
> we haven't been able to come out with a reproducer (yet), I don't see anything
> upstream that would prevent them from happening. 
> 
> The setup involves eventpoll, CFS bandwidth controller and timer
> expiry, and the sequence looks as follows (time-ordered):
> 
> p_read (on CPUn, CFS with bandwidth controller active)
> ======
> 
> ep_poll_callback()
>   read_lock_irqsave()
>   ...
>   try_to_wake_up() <- enqueue causes an update_curr() + sets need_resched
>                       due to having no more runtime
>     preempt_enable()
>       preempt_schedule() <- switch out due to p_read being now throttled
> 
> p_write
> =======
> 
> ep_poll()
>   write_lock_irq() <- blocks due to having active readers (p_read)
> 
> ktimers/n
> =========
> 
> timerfd_tmrproc()
> `\
>   ep_poll_callback()
>   `\
>     read_lock_irqsave() <- blocks due to having active writer (p_write)
> 
> 
> From this point we have a circular dependency:
> 
>   p_read -> ktimers/n (to replenish runtime of p_read)
>   ktimers/n -> p_write (to let ktimers/n acquire the readlock)
>   p_write -> p_read (to let p_write acquire the writelock)
> 
> IIUC reverting
>   286deb7ec03d ("locking/rwbase: Mitigate indefinite writer starvation")
> should unblock this as the ktimers/n thread wouldn't block, but then we're back
> to having the indefinite starvation so I wouldn't necessarily call this a win.
> 
> Two options I'm seeing:
> - Prevent p_read from being preempted when it's doing the wakeups under the
>   readlock (icky)
> - Prevent ktimers / ksoftirqd (*) from running the wakeups that have
>   ep_poll_callback() as a wait_queue_entry callback. Punting that to e.g. a
>   kworker /should/ do.
> 
> (*) It's not just timerfd, I've also seen it via net::sock_def_readable -
> it should be anything that's pollable.
> 
> I'm still scratching my head on this, so any suggestions/comments welcome!
> 

We are hunting for quite some time sporadic lock-ups or RT systems, 
first only in the field (sigh), now finally also in the lab. Those have 
a fairly high overlap with what was described here. Our baselines so 
far: 6.1-rt, Debian and vanilla. We are currently preparing experiments 
with latest mainline.

While this thread remained silent afterwards, we have found [1][2][3] as 
apparently related. But this means we are still with this RT bug, even 
in latest 6.15-rc1?

Jan

[1] https://lore.kernel.org/lkml/20231030145104.4107573-1-vschneid@redhat.com/
[2] https://lore.kernel.org/lkml/20240202080920.3337862-1-vschneid@redhat.com/
[3] https://lore.kernel.org/lkml/20250220093257.9380-1-kprateek.nayak@amd.com/

-- 
Siemens AG, Foundational Technologies
Linux Expert Center