linux-kernel - Re: [RFC PATCH v2 0/7] Defer throttle when task exits to user

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250421120648.GA3357499@bytedance>
Date: Tue, 22 Apr 2025 10:10:54 +0800
From: Aaron Lu <ziqianlu@...edance.com>
To: K Prateek Nayak <kprateek.nayak@....com>
Cc: Jan Kiszka <jan.kiszka@...mens.com>,
	Florian Bezdeka <florian.bezdeka@...mens.com>,
	Valentin Schneider <vschneid@...hat.com>,
	Ben Segall <bsegall@...gle.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Josh Don <joshdon@...gle.com>, Ingo Molnar <mingo@...hat.com>,
	Vincent Guittot <vincent.guittot@...aro.org>,
	Xi Wang <xii@...gle.com>, linux-kernel@...r.kernel.org,
	Juri Lelli <juri.lelli@...hat.com>,
	Dietmar Eggemann <dietmar.eggemann@....com>,
	Steven Rostedt <rostedt@...dmis.org>, Mel Gorman <mgorman@...e.de>,
	Chengming Zhou <chengming.zhou@...ux.dev>,
	Chuyi Zhou <zhouchuyi@...edance.com>,
	"Sebastian Andrzej Siewior," <bigeasy@...utronix.de>
Subject: Re: [RFC PATCH v2 0/7] Defer throttle when task exits to user

Hi Prateek,

On Tue, Apr 15, 2025 at 09:19:20PM +0530, K Prateek Nayak wrote:
> Hello Jan,
> 
> Sorry for the noise.
> 
> On 4/15/2025 4:46 PM, K Prateek Nayak wrote:
> > Hello Jan,
> > 
> > On 4/15/2025 3:51 PM, Jan Kiszka wrote:
> > > > Is this in line with what you are seeing?
> > > > 
> > > 
> > > Yes, and if you wait a bit longer for the second reporting round, you
> > > should get more task backtraces as well.
> > 
> > So looking at the backtrace [1], Aaron's patch should help with the
> > stalls you are seeing.
> > 
> > timerfd that queues a hrtimer also uses ep_poll_callback() to wakeup
> > the epoll waiter which queues ahead of the bandwidth timer and
> > requires the read lock but now since the writer tried to grab the
> > lock pushing readers on the slowpath. if epoll-stall-writer is now
> > throttled, it needs ktimer to replenish its bandwidth which cannot
> > happen without it grabbing the read lock first.
> > 
> > # epoll-stall-writer
> 
> So I got confused between "epoll-stall" and "epoll-stall-writer" here.
> Turns out the actual series of events (based on traces, and hopefully
> correct this time) are slightly longer. The correct series of events
> are:
> 
> # epoll-stall-writer
> 
> anon_pipe_write()
>   __wake_up_common()
>     ep_poll_callback() {
>       read_lock_irq(&ep->lock)		/* Read lock acquired here */

I was confused by this function's name. I had thought irq is off but
then realized under PREEMPT_RT, read_lock_irq() doesn't disable irq...

>       __wake_up_common()
>         ep_autoremove_wake_function()
>           try_to_wake_up()		/* Wakes up "epoll-stall" */
>             preempt_schedule()
>             ...
> 
> # "epoll-stall-writer" has run out of bandwidth, needs replenish to run

Luckily in this "only throttle when ret2user" model, epoll-stall-writer
does not need replenish to run again(and then unblock the others).

> # sched_switch: "epoll-stall-writer" => "epoll-stall"
> 
>     ... /* Resumes from epoll_wait() */
>     epoll_wait() => 1			/* Write to FIFO */
>     read() 				/* Reads one byte of data */
>     epoll_wait()
>       write_lock_irq()			/* Tries to grab write lock; "epoll-stall-writer" still has read lock */
>         schedule_rtlock()		/* Sleeps but put next readers on slowpath */
>         ...
> 
> # sched_switch: "epoll-stall" => "swapper"
> # CPU is idle
> 
> ...
> 
> # Timer interrupt schedules ktimers
> # sched_switch: "swapper" => "ktimers"
> 
> hrtimer_run_softirq()
>   timerfd_tmrproc()
>     __wake_up_common()
>       ep_poll_callback() {
>         read_lock_irq(&ep->lock)	/* Blocks since we are in rwlock slowpath */
>           schedule_rtlock()
>           ...
> 
> # sched_switch: "ktimers" => "swapper"
> # Bandwidth replenish never happens
> # Stall
> 
> From a second look at trace, this should be the right series of
> events since "epoll-stall-writer" with bandwidth control seems
> to have cut off during while doing the wakeup and hasn't run
> again.
> 
> Sorry for the noise.
> 

Thanks for the analysis.

I'm testing this reprod with this series and didn't notice any issue
yet, I'll report if anything unexpected happened.

Best wishes,
Aaron