linux-kernel - Re: [BUG] Caused by: sched/deadline: Move CPU frequency selection triggering points

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180703150711.GA23426@localhost.localdomain>
Date:   Tue, 3 Jul 2018 17:07:11 +0200
From:   Juri Lelli <juri.lelli@...hat.com>
To:     Steven Rostedt <rostedt@...dmis.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        Ingo Molnar <mingo@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Claudio Scordino <claudio@...dence.eu.com>,
        Viresh Kumar <viresh.kumar@...aro.org>,
        Clark Williams <williams@...hat.com>
Subject: Re: [BUG] Caused by: sched/deadline: Move CPU frequency selection
 triggering points

Hi Steve,

On 03/07/18 10:54, Steven Rostedt wrote:
> When looking to test SCHED_DEADLINE, I triggered a lockup. The lockup
> appears to be caused by  WARN_ON() done inside the scheduling path, and
> I'm guessing it tried to grab the rq lock and caused a deadlock (all I
> would get would be the "--- cut here ---" line, and then nothing after
> that. But a bit of playing with the printks() I figured out it was
> happening at:
> 
> assert_clock_updated() {
> 	SCHED_WARN_ON(rq->clock_update_flags < RQCF_ACT_SKIP);
> }
> 
> I bisected it down to commit e0367b126 ("sched/deadline: Move CPU
> frequency selection triggering points"). Reverting it indeed makes the
> deadlock go away.
> 
> I commented out the WARN_ON that was being triggered, and that let the
> system continue to run. I did the following change:
> 
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -914,7 +914,12 @@ static inline void assert_clock_updated(struct rq *rq)
>          * The only reason for not seeing a clock update since the
>          * last rq_pin_lock() is if we're currently skipping updates.
>          */
> -       SCHED_WARN_ON(rq->clock_update_flags < RQCF_ACT_SKIP);
> +       if (rq->clock_update_flags < RQCF_ACT_SKIP)
> +               trace_printk("WARN_ON: [%d] rq->clock_update_flags (%d) < %d)\n",
> +                       rq->cpu, rq->clock_update_flags, RQCF_ACT_SKIP);
> +       else
> +               trace_printk("GOOD: [%d] rq->clock_update_flags (%d) >= %d)\n",
> +                       rq->cpu, rq->clock_update_flags, RQCF_ACT_SKIP);
>  }
> 
> To see what was happening. I also, added trace_printk()s to all the
> updates to clock_update_flags, and ran my test again.
> 
> Here's what I got:
> 
>    deadline_test-1393  [002]   162.127132: bprint:               push_dl_task.part.40: WARN_ON: [1] rq->clock_update_flags (0) < 2)
>    deadline_test-1393  [002]   162.127133: bprint:               update_rq_clock: [1] clock was 0 now 4
>    deadline_test-1393  [002]   162.127134: bprint:               rq_clock: GOOD: [1] rq->clock_update_flags (4) >= 2)
> 
> It appears that we hit this in this path:
> 
> push_dl_task {
> 	add_running_bw() {
> 		__add_running_bw() {
> 			cpufreq_update_util() {
> 				data->func(data, rq_clock(rq), flags);
> 				rq_clock() {
> 					assert_clock_updated()
> 
> And here the clock isn't updated and we get the splat.
> 
> Reverting the stated patch works because it added the call to
> cpufreq_update_util() that does the rq_clock() at an inappropriate
> time, which causes the splat.
> 
> I'm not sure what the right answer to this is. Reverting obviously
> works, but I'm also guessing a proper placement of update_rq_clock()
> may also work. I just don't know where that placement is, as I don't
> understand the rq_clock() updates enough.
> 
> Help?

This got into tip quite recently

ecda2b66e263 ("sched/deadline: Fix missing clock update")

could you please double check that you have that in your stack?

Thanks,

- Juri