linux-kernel - Re: pick_next_task() picking the wrong task [v4.9.163]

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <182b993b34ee0dc80dab150984f08d33efaf5eda.camel@gmail.com>
Date:   Wed, 27 Mar 2019 20:56:06 -0400
From:   Radu Rendec <radu.rendec@...il.com>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     linux-kernel@...r.kernel.org, Ingo Molnar <mingo@...hat.com>
Subject: Re: pick_next_task() picking the wrong task [v4.9.163]

On Sat, 2019-03-23 at 11:15 +0100, Peter Zijlstra wrote:
> On Fri, Mar 22, 2019 at 05:57:59PM -0400, Radu Rendec wrote:
> > Hi Everyone,
> > 
> > I believe I'm seeing a weird behavior of pick_next_task() where it
> > chooses a lower priority task over a higher priority one. The scheduling
> > class of the two tasks is also different ('fair' vs. 'rt'). The culprit
> > seems to be the optimization at the beginning of the function, where
> > fair_sched_class.pick_next_task() is called directly.  I'm running
> > v4.9.163, but that piece of code is very similar in recent kernels.
> > 
> > [...]
> > 
> > I instrumented pick_next_task() with trace_printk() and I am sure that
> > every time the wrong task is picked, flow goes through the optimization
> 
> That's weird, because when you wake a RT task, the:
> 
>   rq->nr_running == rq->cfs.h_nr_running
> 
> condition should not be true. Maybe try adding trace_printk() to all
> rq->nr_running manipulation to see what goes wobbly?

The answer is in enqueue_top_rt_rq(): it returns before touching the run
queue counters because rt_rq_throttled(rt_rq) is true. So basically this
is RT throttling kicking in. I confirmed by disabling RT throttling and
testing again.

So there's nothing wrong with the scheduler. The "sched_wakeup: comm=.."
trace was a bit misleading. What happens when RT throttling kicks in is
that the task is woken (and probably changes state to TASK_RUNNING) but
not actually added to the run queue.

Thanks again for looking into this and sorry about the noise!

Best regards,
Radu Rendec