linux-kernel - Re: [PATCH v2 02/12] sched/deadline: Less agressive dl

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANDhNCreD8f6pPjUa--UzXicJr=xnEGGbKdZhmJCeVPgkEV-Ag@mail.gmail.com>
Date: Mon, 15 Sep 2025 21:18:07 -0700
From: John Stultz <jstultz@...gle.com>
To: Peter Zijlstra <peterz@...radead.org>
Cc: mingo@...hat.com, juri.lelli@...hat.com, vincent.guittot@...aro.org, 
	dietmar.eggemann@....com, rostedt@...dmis.org, bsegall@...gle.com, 
	mgorman@...e.de, vschneid@...hat.com, clm@...a.com, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH v2 02/12] sched/deadline: Less agressive dl_server handling

On Mon, Sep 15, 2025 at 3:29 PM John Stultz <jstultz@...gle.com> wrote:
> From my initial debugging, it seems like the dl_se->dl_server_active
> is set, but the dl_se isn't enqueued, and because active is set we're
> short cutting out of dl_server_start(). I suspect the revert re-adding
> dl_server_stop() helps because it forces dl_server_active to be
> cleared and so we keep the dl_server_active and enqueued state in
> sync.   Maybe we are dequeueing the dl_server from update_curr_dl_se()
> due to dl_runtime_exceeded() but somehow not re-enqueueing it via
> dl_server_timer()?
>

So further digging, it looks like when the problem occurs, we call
start_dl_timer() to set the hrtimer, then when the hrtimer later
fires, we call dl_server_timer(), which catches on the `if
(!dl_se->server_has_tasks(dl_se))` case, which then calls
replenish_dl_entity() and dl_server_stopped() and finally
HRTIMER_NORESTART.  With dl_server_stopped() having set
dl_se->dl_server_idle before returning false (and notably not calling
dl_server_stop() which would clear dl_server_active).

However, dl_server_idle doesn't persist for very long, as we
dl_server_update is shortly called setting it back to 0 before calling
update_curr_dl_se() which I suspect is dequeueing the dl_server.

>From this point it seems we get into a situation where the timer
doesn't fire again. And nothing enqueues the dl_server entity back
onto the runqueue, so it never picks from the fair sched and we see
the starvation on that core.

After we get into the problematic state, the rq->fair_server looks like:
$3 = {rb_node = {__rb_parent_color = 18446612689482602296, rb_right =
0x0, rb_left = 0x0}, dl_runtime = 50000000, dl_deadline = 1000000000,
  dl_period = 1000000000, dl_bw = 52428, dl_density = 52428, runtime =
49615739, deadline = 217139555275, flags = 0, dl_throttled = 0,
  dl_yielded = 0, dl_non_contending = 0, dl_overrun = 0, dl_server =
1, dl_server_active = 1, dl_defer = 1, dl_defer_armed = 0,
  dl_defer_running = 0, dl_server_idle = 0, dl_timer = {node = {node =
{__rb_parent_color = 18446612689482602384,
        rb_right = 0xffff8881b9d1cbf8, rb_left = 0x0}, expires =
215349181777}, _softexpires = 215349181777,
    function = 0xffffffff813849d0 <dl_task_timer>, base =
0xffff8881b9d1c300, state = 0 '\000', is_rel = 0 '\000', is_soft = 0
'\000',
    is_hard = 1 '\001'}, inactive_timer = {node = {node =
{__rb_parent_color = 18446612689482602448, rb_right = 0x0, rb_left =
0x0},
      expires = 0}, _softexpires = 0, function = 0xffffffff8137faa0
<inactive_task_timer>, base = 0xffff8881b9c1c300, state = 0 '\000',
    is_rel = 0 '\000', is_soft = 0 '\000', is_hard = 1 '\001'}, rq =
0xffff8881b9d2cc00,
  server_has_tasks = 0xffffffff81363620 <fair_server_has_tasks>,
server_pick_task = 0xffffffff8136c2d0 <fair_server_pick_task>,
  pi_se = 0xffff8881b9d2d738}

I'm still getting a bit lost in the dl_server state machine, but it's
clear we're falling through a crack.

My current theory is that in dl_server_timer() we probably should be
calling dl_server_stop() instead of dl_server_stopped(), as that seems
to avoid the problem for me, but I'm also worried it might cause
problems with not setting dl_server_idle, though again my sense of the
state machine there is foggy.

thanks
-john