linux-kernel - Re: [PATCH 0/10 v2] sched/fair: Fix statistics with delayed dequeue

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAKfTPtB4_gPdgSWMfk+ZZ0H8x4T8_-CUZaaUe_ZiWJTK6mAe4Q@mail.gmail.com>
Date: Mon, 2 Dec 2024 15:24:26 +0100
From: Vincent Guittot <vincent.guittot@...aro.org>
To: K Prateek Nayak <kprateek.nayak@....com>
Cc: Mike Galbraith <efault@....de>, mingo@...hat.com, peterz@...radead.org, 
	juri.lelli@...hat.com, dietmar.eggemann@....com, rostedt@...dmis.org, 
	bsegall@...gle.com, mgorman@...e.de, vschneid@...hat.com, 
	linux-kernel@...r.kernel.org, pauld@...hat.com, luis.machado@....com
Subject: Re: [PATCH 0/10 v2] sched/fair: Fix statistics with delayed dequeue

On Mon, 2 Dec 2024 at 12:42, Vincent Guittot <vincent.guittot@...aro.org> wrote:
>
> On Mon, 2 Dec 2024 at 10:58, K Prateek Nayak <kprateek.nayak@....com> wrote:
> >
> > Hello Vincent, Mike,
> >
> > On 12/2/2024 2:47 PM, Vincent Guittot wrote:
> > > On Sun, 1 Dec 2024 at 14:30, Mike Galbraith <efault@....de> wrote:
> > >>
> > >> Greetings,
> > >>
> > >> On Fri, 2024-11-29 at 17:17 +0100, Vincent Guittot wrote:
> > >>> Delayed dequeued feature keeps a sleeping sched_entitiy enqueued until its
> > >>> lag has elapsed. As a result, it stays also visible in the statistics that
> > >>> are used to balance the system and in particular the field h_nr_running.
> > >>>
> > >>> This serie fixes those metrics by creating a new h_nr_queued that tracks
> > >>> all queued tasks. It renames h_nr_running into h_nr_runnable and restores
> > >>> the behavior of h_nr_running i.e. tracking the number of fair tasks that
> > >>>   want to run.
> > >>>
> > >>> h_nr_runnable is used in several places to make decision on load balance:
> > >>>    - PELT runnable_avg
> > >>>    - deciding if a group is overloaded or has spare capacity
> > >>>    - numa stats
> > >>>    - reduced capacity management
> > >>>    - load balance between groups
> > >>
> > >> I took the series for a spin in tip v6.12-10334-gb1b238fba309, but
> > >> runnable seems to have an off-by-one issue, causing it to wander ever
> > >> further south.
> > >>
> > >> patches 1-3 applied.
> > >>    .h_nr_runnable                 : -3046
> > >>    .runnable_avg                  : 450189777126
> > >
> > > Yeah, I messed up something around finish_delayed_dequeue_entity().
> > > I'm' going to prepare a v3
> >
> > I was looking into this and I have the below diff so far that seems to
> > solve the post boot negative values of h_nr_runnable on my setup; it is
> > only lightly tested so far:
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 87552870958c..423981e65aba 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -5464,6 +5464,10 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
> >   static void set_delayed(struct sched_entity *se)
> >   {
> >         se->sched_delayed = 1;
> > +
> > +       if (!entity_is_task(se))
> > +               return;
> > +
> >         for_each_sched_entity(se) {
> >                 struct cfs_rq *cfs_rq = cfs_rq_of(se);
> >
> > @@ -5476,6 +5480,10 @@ static void set_delayed(struct sched_entity *se)
> >   static void clear_delayed(struct sched_entity *se)
> >   {
> >         se->sched_delayed = 0;
> > +
> > +       if (!entity_is_task(se))
> > +               return;
> > +
> >         for_each_sched_entity(se) {
> >                 struct cfs_rq *cfs_rq = cfs_rq_of(se);
> >
> > @@ -6977,7 +6985,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
> >         struct cfs_rq *cfs_rq;
> >         struct sched_entity *se = &p->se;
> >         int h_nr_idle = task_has_idle_policy(p);
> > -       int h_nr_runnable = 0;
> > +       int h_nr_runnable = 1;
>
> I miss to invert default value when moving from h_nr_delayed to h_nr_runnable
>
> >         int task_new = !(flags & ENQUEUE_WAKEUP);
> >         int rq_h_nr_queued = rq->cfs.h_nr_queued;
> >         u64 slice = 0;
> > @@ -7124,8 +7132,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
> >                 p = task_of(se);
> >                 h_nr_queued = 1;
> >                 h_nr_idle = task_has_idle_policy(p);
> > -               if (!task_sleep && !task_delayed)
> > -                       h_nr_runnable = !se->sched_delayed;
> > +               h_nr_runnable = !se->sched_delayed;

And I screwed up h_nr_runnable here as well

> >         } else {
> >                 cfs_rq = group_cfs_rq(se);
> >                 slice = cfs_rq_min_slice(cfs_rq);
> > diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
> > index ab911d1335ba..f4ef5aaa4674 100644
> > --- a/kernel/sched/idle.c
> > +++ b/kernel/sched/idle.c
> > @@ -457,6 +457,7 @@ static void put_prev_task_idle(struct rq *rq, struct task_struct *prev, struct t
> >
> >   static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool first)
> >   {
> > +       SCHED_WARN_ON(rq->cfs.h_nr_runnable);
> >         update_idle_core(rq);
> >         scx_update_idle(rq, true);
> >         schedstat_inc(rq->sched_goidle);
> > --
> >
> > I'm not sure if the change in dequeue_entities() is completely necessary
> > but I added it after seeing the (DEQUEUE_SLEEP | DEQUEUE_DELAYED) in
> > throttle_cfs_rq() but there the entity cannot possibly be a task so
> > perhaps that part is unnecessary ¯\_(ツ)_/¯
> >
> > Still testing! Will keep an eye out for v3.
> >
> > >
> > >>
> > >> full set applied.
> > >>    .h_nr_runnable                 : -5707
> > >>    .runnable_avg                  : 4391793519526
> > >>
> > >>          -Mike
> >
> > --
> > Thanks and Regards,
> > Prateek
> >