linux-kernel - Re: 5.6-rc3: WARNING: CPU: 48 PID: 17435 at kernel/sched/fair.c:380 enqueue_task

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Tue, 3 Mar 2020 08:55:40 +0100
From:   Vincent Guittot <vincent.guittot@...aro.org>
To:     Christian Borntraeger <borntraeger@...ibm.com>
Cc:     Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: 5.6-rc3: WARNING: CPU: 48 PID: 17435 at kernel/sched/fair.c:380 enqueue_task_fair+0x328/0x440

On Tue, 3 Mar 2020 at 08:37, Christian Borntraeger
<borntraeger@...ibm.com> wrote:
>
>
>
> On 02.03.20 19:17, Christian Borntraeger wrote:
> > On 02.03.20 12:16, Christian Borntraeger wrote:
> >>
> >>
> >> On 28.02.20 17:35, Vincent Guittot wrote:
> >>> Le vendredi 28 févr. 2020 à 16:42:27 (+0100), Christian Borntraeger a écrit :
> >>>>
> >>>>
> >>>> On 28.02.20 16:37, Vincent Guittot wrote:
> >>>>> On Fri, 28 Feb 2020 at 16:08, Christian Borntraeger
> >>>>> <borntraeger@...ibm.com> wrote:
> >>>>>>
> >>>>>> Also happened with 5.4:
> >>>>>> Seems that I just happen to have an interesting test workload/system size interaction
> >>>>>> on a newly installed system that triggers this.
> >>>>>
> >>>>> you will probably go back to 5.1 which is the version where we put
> >>>>> back the deletion of unused cfs_rq from the list which can trigger the
> >>>>> warning:
> >>>>> commit 039ae8bcf7a5 : (Fix O(nr_cgroups) in the load balancing path)
> >>>>>
> >>>>> AFAICT, we haven't changed this since
> >>>>
> >>>> So you do know what is the problem? If not is there any debug option or
> >>>> patch that I could apply to give you more information?
> >>>
> >>> No I don't know what is happening. Your test probably goes through an unexpected path
> >>>
> >>> Would it be difficult for me to reproduce your test env ?
> >>
> >> Not sure. Its a 32CPU (SMT2 -> 64) host. I have about 10 KVM guests running doing different
> >> things.
> >>
> >>>
> >>> There is an optimization in the code which could generate problem if assumption is not
> >>> true. Could you try the patch below ?
> >>>
> >>> ---
> >>>  kernel/sched/fair.c | 2 +-
> >>>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>>
> >>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>> index 3c8a379c357e..beb773c23e7d 100644
> >>> --- a/kernel/sched/fair.c
> >>> +++ b/kernel/sched/fair.c
> >>> @@ -4035,8 +4035,8 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>             __enqueue_entity(cfs_rq, se);
> >>>     se->on_rq = 1;
> >>>
> >>> +   list_add_leaf_cfs_rq(cfs_rq);
> >>>     if (cfs_rq->nr_running == 1) {
> >>> -           list_add_leaf_cfs_rq(cfs_rq);
> >>>             check_enqueue_throttle(cfs_rq);
> >>>     }
> >>>  }
> >>
> >> Now running for 3 hours. I have not seen the issue yet. I can tell tomorrow if this fixes
> >> the issue.
> >
> >
> > Still running fine. I can tell for sure tomorrow, but I have the impression that this makes the
> > WARN_ON go away.
>
> So I guess this change "fixed" the issue. If you want me to test additional patches, let me know.

Thanks for the test. For now, I don't have any other patch to test. I
have to look more deeply how the situation happens.
I will let you know if I have other patch to test

>