linux-kernel - Re: [bisected][mainline]Kernel warnings at kernel/sched/cpudeadline.c:219

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20251008111314.GI3289052@noisy.programming.kicks-ass.net>
Date: Wed, 8 Oct 2025 13:13:14 +0200
From: Peter Zijlstra <peterz@...radead.org>
To: Shrikanth Hegde <sshegde@...ux.ibm.com>
Cc: Venkat Rao Bagalkote <venkat88@...ux.ibm.com>,
	LKML <linux-kernel@...r.kernel.org>,
	linuxppc-dev <linuxppc-dev@...ts.ozlabs.org>,
	Madhavan Srinivasan <maddy@...ux.ibm.com>, jstultz@...gle.com,
	stultz@...gle.com
Subject: Re: [bisected][mainline]Kernel warnings at
 kernel/sched/cpudeadline.c:219

On Wed, Oct 08, 2025 at 03:47:16PM +0530, Shrikanth Hegde wrote:
> 
> 
> On 10/8/25 3:20 PM, Peter Zijlstra wrote:
> > On Wed, Oct 08, 2025 at 07:41:10AM +0530, Venkat Rao Bagalkote wrote:
> > > Greetings!!!
> > > 
> > > 
> > > IBM CI has reported a kernel warnings while running CPU hot plug operation
> > > on IBM Power9 system.
> > > 
> > > 
> > > Command to reproduce the issue:
> > > 
> > > drmgr -c cpu -r -q 1
> > > 
> > > 
> > > Git Bisect is pointing to below commit as the first bad commit.
> > 
> > Does something like this help?
> > 
> > (also, for future reference, please don't line wrap logs, it makes them
> > very hard to read)
> > 
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 198d2dd45f59..65f37bfcd661 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -8328,6 +8328,7 @@ static inline void sched_set_rq_offline(struct rq *rq, int cpu)
> >   		BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
> >   		set_rq_offline(rq);
> >   	}
> > +	dl_server_stop(&rq->fair_server);
> >   	rq_unlock_irqrestore(rq, &rf);
> >   }
> 
> 
> Hi Peter. Thanks for looking into it.
> 
> I was able to repro this issue on my system. This above diff didn't help. I still see the warning.
> 
> I have to understand this dl server stuff still.
> So not sure if my understanding is completely correct.
> 
> Looks like the hrtimer is firing after the cpu was removed. The warn on hit only with
> drmgr. Regular hotplug with chcpu doesn;t hit. That's because drmgr changes the cpu_present mask.
> and warning is hit with it.

I do not know what drmgr is. I am not familiar with PowerPC tools.
AFAICT x86 never modifies cpu_present_mask after boot.

> maybe during drmgr, the dl server gets started again? Maybe that's why above patch it didn't work.
> Will see and understand this bit more.

dl_server is per cpu and is started on enqueue of a fair task when:

  - the runqueue was empty; and
  - the dl_server wasn't already active

Once the dl_server is active it has this timer (you already found this),
this timer is set for the 0-laxity moment (the last possible moment in
time where it can still run its budget and not be late), during this
time any fair runtime is accounted against it budget (subtracted from).

Once the timer fires and it still has budget left; it will enqueue the
deadline entity. However the more common case is that its budget will be
depleted, in which case the timer is reset to its period end for
replenish (where it gets new runtime budget), after which its back to
the 0-laxity.

If the deadline entity gets scheduled, it will try and pick a fair task
and run that. In the case where there is no fair task, it will
deactivate itself.

The patch I sent earlier would force stop the deadline timer on CPU
offline.

> Also, i tried this below diff which fixes it. Just ignore the hrtimer if the cpu is offline.
> Does this makes sense?
> ---
> 
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index 615411a0a881..a342cf5e4624 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -1160,6 +1160,9 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
>  	scoped_guard (rq_lock, rq) {
>  		struct rq_flags *rf = &scope.rf;
> +		if (!cpu_online(rq->cpu))
> +			return HRTIMER_NORESTART;
> +
>  		if (!dl_se->dl_throttled || !dl_se->dl_runtime)
>  			return HRTIMER_NORESTART;

This could leave the dl_server in inconsistent state. It would have to
call dl_server_stop() or something along those lines.

Also, this really should not happen; per my previous patch we should be
stopping the timer when we go offline.

Since you can readily reproduce this; perhaps you could stick something
like this in dl_server_start():

	WARN_ON_ONCE(!cpu_online(rq->cpu))

See if anybody is (re)starting the thing ?