linux-kernel - Re: [PATCH] kernel/watchdog: fix spurious hard lockups

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20170628142952.GI5225@dhcp22.suse.cz>
Date:   Wed, 28 Jun 2017 16:29:52 +0200
From:   Michal Hocko <mhocko@...nel.org>
To:     "Liang, Kan" <kan.liang@...el.com>
Cc:     "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "dzickus@...hat.com" <dzickus@...hat.com>,
        "mingo@...nel.org" <mingo@...nel.org>,
        "akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
        "babu.moger@...cle.com" <babu.moger@...cle.com>,
        "atomlin@...hat.com" <atomlin@...hat.com>,
        "prarit@...hat.com" <prarit@...hat.com>,
        "torvalds@...ux-foundation.org" <torvalds@...ux-foundation.org>,
        "peterz@...radead.org" <peterz@...radead.org>,
        "tglx@...utronix.de" <tglx@...utronix.de>,
        "eranian@...gle.com" <eranian@...gle.com>,
        "acme@...hat.com" <acme@...hat.com>,
        "ak@...ux.intel.com" <ak@...ux.intel.com>,
        "stable@...r.kernel.org" <stable@...r.kernel.org>
Subject: Re: [PATCH] kernel/watchdog: fix spurious hard lockups

On Wed 28-06-17 13:24:08, Liang, Kan wrote:
> 
> > > From: Kan Liang <Kan.liang@...el.com>
> > >
> > > Some users reported spurious NMI watchdog timeouts.
> > >
> > > We now have more and more systems where the Turbo range is wide
> > enough
> > > that the NMI watchdog expires faster than the soft watchdog timer that
> > > updates the interrupt tick the NMI watchdog relies on.
> > 
> > AFAIR the watchdog doesn't rely on deferred timers so this would suggest
> > that a standard hrtimer can expire much later than programmed, right?
> 
> The softlockup watchdog relies on hrtimers.
> The hardlockup watchdog (NMI watchdog) relies on perf subsystem and
> using unhalted CPU cycles.
> When the softlockup watchdog expires, it updates the hrtimer_interrupts.
> When the NMI watchdog expires, it will check the hrtimer_interrupts, and
> determine if it's a hardlockup.
> The design was to make the softlockup watchdog runs with 2.5 times the
> rate of NMI watchdog. So it guarantees that the hrtimer_interrupts is
> updated before the NMI watchdog expires.
> That works well if Turbo-Mode is disabled.
> However, when Turbo-Mode is enabled, unhalted CPU cycles might run
> much faster than expected, even faster than softlockup watchdog.
> So the softlockup watchdog will not get a chance to update the
> hrtimer_interrupts, which will trigger false positives.

So it is not the hrtimer which doesn't fire but rather the NMI events
fire too quickly, right?
-- 
Michal Hocko
SUSE Labs