linux-kernel - Re: [PATCH] timers/nohz: Update nohz load even if tick already stopped

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <7b782bc880a29eb7d37f2c2aff73c43e7f7d032f.camel@redhat.com>
Date:   Tue, 05 Nov 2019 01:30:58 -0600
From:   Scott Wood <swood@...hat.com>
To:     Thomas Gleixner <tglx@...utronix.de>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Frederic Weisbecker <frederic@...nel.org>,
        Ingo Molnar <mingo@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] timers/nohz: Update nohz load even if tick already
 stopped

On Tue, 2019-11-05 at 00:43 +0100, Thomas Gleixner wrote:
> On Mon, 4 Nov 2019, Thomas Gleixner wrote:
> > On Fri, 1 Nov 2019, Scott Wood wrote:
> > > On Wed, 2019-10-30 at 14:31 +0100, Peter Zijlstra wrote:
> > > > Oh argh! that's a bit radical of the remote tick. The normal tick
> > > > runs
> > > > just fine on idle CPUs, so lets mirror that.
> > > > 
> > > > How's this then?
> > 
> > ....
> >  
> > > Needs to be tick_nohz_tick_stopped_cpu(cpu)
> > > 
> > > After fixing that, I get:
> > > 
> > > [    7.439068] WARNING: CPU: 20 PID: 7 at
> > > /home/root/linux/kernel/sched/core.c:3681
> > > sched_tick_remote+0x132/0x150
> > 
> > So I'm going to apply Scotts patch if nobody comes up with a better idea
> > until tomorrow.
> 
> As Peter pointed out to me privately we should rather go and analyze the
> real thing instead of just applying duct tape.
> 
> /me drops the patch again.

The warning is due to kernel/sched/idle.c not updating curr->se.exec_start.

While debugging I noticed an issue with a particular load pattern.  The CPU
goes non-nohz for a brief time at an interval very close to twice 
tick_period.  When the tick is started, the timer expiration is more than
tick_period in the past, so hrtimer_forward() tries to catch up by adding
2*tick_period to the expiration.  Then the tick is stopped before that new
expiration, and when the tick is woken up the expiry is again advanced by
2*tick_period with the timer never actually running.  sched_tick_remote()
does fire every second, but there are streaks of several seconds where it
keeps catching the CPU in a non-nohz state, so neither the normal nor remote
ticks are calling calc_load_nohz_remote().

Is there a reason to not just remove the hrtimer_forward() from
tick_nohz_restart(), letting the timer fire if it's in the past, which will
take care of doing hrtimer_forward()?

As for the warning in sched_tick_remote(), it seems like a test for time
since the last tick on this cpu (remote or otherwise) would be better than
relying on curr->se.exec_start, in order to detect things like this.

-Scott