linux-kernel - Re: [BUG almost bisected] Splat in dequeue_rt

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20240824065434.GA26474@noisy.programming.kicks-ass.net>
Date: Sat, 24 Aug 2024 08:54:34 +0200
From: Peter Zijlstra <peterz@...radead.org>
To: "Paul E. McKenney" <paulmck@...nel.org>
Cc: vschneid@...hat.com, linux-kernel@...r.kernel.org, sfr@...b.auug.org.au,
	linux-next@...r.kernel.org, kernel-team@...a.com
Subject: Re: [BUG almost bisected] Splat in dequeue_rt_stack() and build error

On Fri, Aug 23, 2024 at 02:51:03PM -0700, Paul E. McKenney wrote:

> > > Does the below help any? That's more or less what it was before Valentin
> > > asked me why it was weird like that :-)
> > > 
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 6be618110885..5757dd50b02f 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -13107,7 +13107,6 @@ static void switched_from_fair(struct rq *rq, struct task_struct *p)
> > >  	 * and we cannot use DEQUEUE_DELAYED.
> > >  	 */
> > >  	if (p->se.sched_delayed) {
> > > -		dequeue_task(rq, p, DEQUEUE_NOCLOCK | DEQUEUE_SLEEP);
> > >  		p->se.sched_delayed = 0;
> > >  		p->se.rel_deadline = 0;
> > >  		if (sched_feat(DELAY_ZERO) && p->se.vlag > 0)
> > 
> > Removing that line from 2e0199df252a still gets me the complaint about
> > __SCHED_FEAT_DELAY_ZERO being undefined.  To my naive eyes, it appears
> > that this commit:
> > 
> > 54a58a787791 ("sched/fair: Implement DELAY_ZERO")
> > 
> > Need to be placed before 2e0199df252a.  Of course, when I try it, I
> > get conflicts.  So I took just this hunk:
> > 
> > ------------------------------------------------------------------------
> > 
> > diff --git a/kernel/sched/features.h b/kernel/sched/features.h
> > index 97fb2d4920898..6c5f5424614d4 100644
> > --- a/kernel/sched/features.h
> > +++ b/kernel/sched/features.h
> > @@ -28,6 +28,11 @@ SCHED_FEAT(NEXT_BUDDY, false)
> >   */
> >  SCHED_FEAT(CACHE_HOT_BUDDY, true)
> >  
> > +/*
> > + * DELAY_ZERO clips the lag on dequeue (or wakeup) to 0.
> > + */
> > +SCHED_FEAT(DELAY_ZERO, true)
> > +
> >  /*
> >   * Allow wakeup-time preemption of the current task:
> >   */
> > 
> > ------------------------------------------------------------------------
> > 
> > That makes the build error go away.  Maybe even legitimately?

Yep.

> > Just to pick on the easy one, I took a look at the complaint about
> > cfs_rq being unused and the complaint about __SCHED_FEAT_DELAY_ZERO
> > being undefined.  This variable was added here:
> > 
> > 781773e3b680 ("sched/fair: Implement ENQUEUE_DELAYED")
> > 
> > And its first use was added here:
> > 
> > 54a58a787791 ("sched/fair: Implement DELAY_ZERO")
> > 
> > Which matches my experience.
> > 
> > So left to myself, I would run on these commits with the above hunk:
> > 
> > 54a58a7877916 sched/fair: Implement DELAY_ZERO
> > 152e11f6df293 sched/fair: Implement delayed dequeue
> > e1459a50ba318 sched: Teach dequeue_task() about special task states
> > a1c446611e31c sched,freezer: Mark TASK_FROZEN special
> > 781773e3b6803 sched/fair: Implement ENQUEUE_DELAYED
> > f12e148892ede sched/fair: Prepare pick_next_task() for delayed dequeue
> > 2e0199df252a5 sched/fair: Prepare exit/cleanup paths for delayed_dequeue
> > e28b5f8bda017 sched/fair: Assert {set_next,put_prev}_entity() are properly balanced
> > 
> > And where needed, remove the unused cfs_rq local variable.
> > 
> > Would that likely work?

Sounds about right.

> > 
> > In the meantime, SIGFOOD!
> 
> Hearing no objections...

Yeah, sorry, I'm on holidays with the kids and not glued to the screen
as per usual :-)

> Given two patches each of which might or might not need to be applied to a
> given commit, I chose to rebase as follows:
> 
> e28b5f8bda017 sched/fair: Assert {set_next,put_prev}_entity() are properly balanced
> 8aed87410a695 EXP sched/fair: Provide DELAY_ZERO definition
> 	I took this from 54a58a7877916 sched/fair: Implement DELAY_ZERO.
> 49575c0087bc0 sched/fair: Prepare exit/cleanup paths for delayed_dequeue
> 14c3207fd2456 sched/fair: Prepare pick_next_task() for delayed dequeue
> be567af45dd04 sched/fair: Implement ENQUEUE_DELAYED
> 	I dropped the unused cfs_rq local variable from requeue_delayed_entity()
> ed28f7b3ac3f4 sched,freezer: Mark TASK_FROZEN special
> 48d541847b4a6 sched: Teach dequeue_task() about special task states
> ef3b9c5d038dc sched/fair: Implement delayed dequeue
> 	--- First bad commit with dequeue_rt_stack() failures.
> 876c99c058219 sched/fair: Implement DELAY_ZERO
> 	I added the cfs_rq local variable to requeue_delayed_entity()
> 
> This is on -rcu branch peterz.2024.08.23b.
> 
> I ran 50*TREE05 in a bisection, which converged on be567af45dd04, but only
> one run of the 50 had a complaint, and that was in enqueue_dl_entry(),

Hmm, I have one other report about that. Hasn't made much sense yet --
then again, as per the above mentioned reason, I'm not able to put real
time in atm.

> not the dequeue_rt_stack() that I have been chasing.  I ran three
> additional 50*TREE05 runs on its predecessor (14c3207fd2456) with no
> failures.  I then ran 50*TREE03 on each of ed28f7b3ac3f4, 48d541847b4a6,
> and ef3b9c5d038dc.  Only this last ("ef3b9c5d038dc sched/fair: Implement
> delayed dequeue") had failure, and they were all the dequeue_rt_stack()
> failures I am chasing.  One of the runs also hung.

I'm a little confused now though; this is with the dequeue removed from
switched_from_fair() ?

Looking at your tree, 49575c0087bc0 still has that dequeue. Does the
dequeue_rt_stack() issue go away with that line removed?