linux-kernel - Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20170629154406.44xo7dnw7btn4gpx@redhat.com>
Date:   Thu, 29 Jun 2017 11:44:06 -0400
From:   Don Zickus <dzickus@...hat.com>
To:     Andi Kleen <ak@...ux.intel.com>
Cc:     "Liang, Kan" <kan.liang@...el.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "mingo@...nel.org" <mingo@...nel.org>,
        "akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
        "babu.moger@...cle.com" <babu.moger@...cle.com>,
        "atomlin@...hat.com" <atomlin@...hat.com>,
        "prarit@...hat.com" <prarit@...hat.com>,
        "torvalds@...ux-foundation.org" <torvalds@...ux-foundation.org>,
        "peterz@...radead.org" <peterz@...radead.org>,
        "eranian@...gle.com" <eranian@...gle.com>,
        "acme@...hat.com" <acme@...hat.com>,
        "stable@...r.kernel.org" <stable@...r.kernel.org>
Subject: Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups

On Wed, Jun 28, 2017 at 01:14:04PM -0700, Andi Kleen wrote:
> It can be a useful debugging tool for a specific class of bugs: 
> when kernel software is looping forever.
> 
> But if that happens does it really matter how many iterations the
> loop does before it is stopped?
> 
> Even the current timeout is essentially eternity in CPU time, and 3x
> eternity is still eternity.

That isn't true.  We have customers that test the accuracy and file bugs.  I
had to write a RHEL whitepaper a number of years ago explaining why the
softlockup took 62 seconds to fire instead of 60.

Customers were changing the watchdog_thresh when the system slowed down to
purposely trigger a panic (which exposed race conditions leading Uli to
redesign the sysctl interface).

I don't feel like explaining to our customers how we regressed on our
watchdog accuracy.  It is exhausting, especially if it is a debug feature.

> 
> > The hrtimer increase maintains that and just adds a few more
> > interrupts/second.
> 
> Interruptions are a big deal for many people.

Yes, and we probably have customers that will complain on that too.

Either solution is a lose/lose.  And yes, we will probably get bit by the
false NMI problems on those Intel boxes.  This is why I was preferring a
real solution.

The question is, if the real solution is going to take a while, what is the
least sucky solution for now?  Or how do we minimize it to a specific class
of Intel boxes.

Cheers,
Don