Message-ID: <D5F4086A-AD87-4CCB-B926-25713BBF00ED@vmware.com>
Date:   Fri, 15 Jul 2022 04:49:36 +0000
From:   Ankit Jain <ankitja@...are.com>
To:     Juri Lelli <juri.lelli@...hat.com>
CC:     "srivatsa@...il.mit.edu" <srivatsa@...il.mit.edu>,
        LKML <linux-kernel@...r.kernel.org>,
        linux-rt-users <linux-rt-users@...r.kernel.org>,
        Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
        Daniel Bristot de Oliveira <bristot@...hat.com>,
        Valentin Schneider <vschneid@...hat.com>,
        Sharan Turlapati <sturlapati@...are.com>,
        Him Kalyan Bordoloi <bordoloih@...are.com>,
        Keerthana Kalyanasundaram <keerthanak@...are.com>,
        Anish Swaminathan <anishs@...are.com>,
        Srivatsa Bhat <srivatsab@...are.com>
Subject: Re: [PATCH] sched/deadline: Fix BUG_ON condition for deboosted tasks

[Resending my previous email in plaintext]

> On 14-Jul-2022, at 12:58 PM, Juri Lelli <juri.lelli@...hat.com> wrote:
> 
> ⚠ External Email
> 
> Hi,
> 
> On 13/07/22 14:31, Srivatsa S. Bhat wrote:
>> 
>> Hi Juri,
>> 
>> On 7/13/22 12:50 AM, Juri Lelli wrote:
>>> Tasks the are being deboosted from SCHED_DEADLINE might enter
>>> enqueue_task_dl() one last time and hit an erroneous BUG_ON condition:
>>> since they are not boosted anymore, the if (is_dl_boosted()) branch is
>>> not taken, but the else if (!dl_prio) is and inside this one we
>>> BUG_ON(!is_dl_boosted), which is of course false (BUG_ON triggered)
>>> otherwise we had entered the if branch above. Long story short, the
>>> current condition doesn't make sense and always leads to triggering of a
>>> BUG.
>>> 
>>> Fix this by only checking enqueue flags, properly: ENQUEUE_REPLENISH has
>>> to be present, but additional flags are not a problem.
>>> 
>>> Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes")
>> 
>> It looks like this problem goes further back than the above commit
>> (which was merged in v5.10).
>> 
>> Even the oldest LTS kernel (4.9) has code like this:
>> 
>> if (... && p->dl.dl_boosted && ...)) {
>>      /* code */
>> 
>> } else if (!dl_prio(p->normal_prio)) {
>> 
>>      BUG_ON(!p->dl.dl_boosted || flags != ENQUEUE_REPLENISH);
>>      return;
>> }
>> 
>> And we have observed crashes in the 4.19 kernel series too (CC'ed
>> Ankit Jain and Him Kalyan who have reproduced this issue).
>> 
>> I believe commit 64be6f1f5f71 ("sched/deadline: Don't replenish from a
>> !SCHED_DEADLINE entity") introduced the problem, which dates back to
>> v3.18.
>> 
>> Would you mind updating the Fixes: tag and adding a CC: stable tag as
>> well, when you respin the patch, please?
> 
> I think you are right. Will do.
> 
> Thanks for taking a look!
> 
> Best,
> Juri
> 

Hi Juri,

I tried the patch but it still hit the BUG_ON.

[  163.094094] ------------[ cut here ]------------
[  163.094095] kernel BUG at kernel/sched/deadline.c:1525!
[  163.094103] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[  163.094105] CPU: 0 PID: 5494 Comm: stalld/34 Not tainted 4.19.247-rt108-10.ph3-rt #1-photon
[  163.094107] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
[  163.094113] RIP: 0010:enqueue_task_dl+0x35d/0x9d0
[  163.094115] Code: 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8b 56 74 85 d2 0f 88 91 fd ff ff 80 a6 0c 03 00 00 fe 41 83 e5 20 0f 85
[  163.094116] RSP: 0018:ffff9b9286537e40 EFLAGS: 00010046
[  163.094118] RAX: ffffffff840bded0 RBX: ffff8dda07c48000 RCX: 0000000000002000
[  163.094119] RDX: 0000000000000078 RSI: ffff8dda07c48000 RDI: ffff8dddb79a87c0
[  163.094120] RBP: ffff9b9286537e78 R08: 0000000000000000 R09: 000000000000007f
[  163.094121] R10: ffff9b9286537e68 R11: 0000000000000000 R12: ffff9b9286537ef0
[  163.094121] R13: 0000000000000000 R14: ffff8dddb79a87c0 R15: ffff8dda07c482b8
[  163.094123] FS:  00007f81a27e4700(0000) GS:ffff8ddbb7600000(0000) knlGS:0000000000000000
[  163.094124] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  163.094125] CR2: 00007f85e6cfc4c0 CR3: 0000000233744004 CR4: 00000000007606b0
[  163.094176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  163.094177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  163.094177] PKRU: 55555554
[  163.094178] Call Trace:
[  163.094183]  ? dequeue_task_dl+0x38/0x1d0
[  163.094188]  __sched_setscheduler+0x2e2/0x8e0
[  163.094191]  __x64_sys_sched_setattr+0x74/0xb0
[  163.094194]  do_syscall_64+0x60/0x1b0
[  163.094200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  163.094201] RIP: 0033:0x7f81b6ffe319
[  163.094202] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c
[  163.094203] RSP: 002b:00007f81a27e3e28 EFLAGS: 00000206 ORIG_RAX: 000000000000013a
[  163.094205] RAX: ffffffffffffffda RBX: 000000000000150a RCX: 00007f81b6ffe319
[  163.094205] RDX: 0000000000000000 RSI: 00007f81a27e3e50 RDI: 000000000000150a
[  163.094206] RBP: 000000000000150a R08: 0000000000000000 R09: 0000000000000030
[  163.094207] R10: 00007f80f0002090 R11: 0000000000000206 R12: 0000556306de7a20
[  163.094208] R13: 0000000000000002 R14: 0000556306ba3570 R15: 00007f81a27e4700
[  163.094210] Modules linked in: ipt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp
[  163.099512] ---[ end trace 0000000000000002 ]--- 


In enqueue_task_dl():

} else if (!dl_prio(p->normal_prio)) {
  …
  BUG_ON(!(flags & ENQUEUE_REPLENISH));
  return;
}

I observe flags value as (ENQUEUE_RESTORE |  ENQUEUE_NOCLOCK)


Thanks,
Ankit

Powered by blists - more mailing lists