linux-kernel - Re: frequent lockups in 3.18rc4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20141120200603.GA19499@redhat.com>
Date:	Thu, 20 Nov 2014 15:06:03 -0500
From:	Dave Jones <davej@...hat.com>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	Andy Lutomirski <luto@...capital.net>,
	Don Zickus <dzickus@...hat.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	the arch/x86 maintainers <x86@...nel.org>,
	Peter Zijlstra <peterz@...radead.org>
Subject: Re: frequent lockups in 3.18rc4

On Thu, Nov 20, 2014 at 11:43:07AM -0800, Linus Torvalds wrote:
 
 > You know what? I'm seriously starting to think that these bugs aren't
 > actually real. Or rather, I don't think it's really a true softlockup,
 > because most of them seem to happen in totally harmless code.
 > 
 > So I'm wondering whether the real issue might not be just this:
 > 
 >    [loadavg: 164.79 157.30 155.90 37/409 11893]
 > 
 > together with possibly a scheduler issue and/or a bug in the smpboot
 > thread logic (that the watchdog uses) or similar.
 > 
 > That's *especially* true if it turns out that the 3.17 problem you saw
 > was actually a perf bug that has already been fixed and is in stable.
 > We've been looking at kernel/smp.c changes, and looking for x86 IPI or
 > APIC changes, and found some harmlessly (at least on x86) suspicious
 > code and this exercise might be worth it for that reason, but what if
 > it's really just a scheduler regression.

I started a run against 3.17 with the perf fixes. If that survives
today, I'll start a bisection tomorrow.

 > There's been a *lot* more scheduler changes since 3.17 than the small
 > things we've looked at for x86 entry or IPI handling. And the
 > scheduler changes have been about things like overloaded scheduling
 > groups etc, and I could easily imaging that some bug *there* ends up
 > causing the watchdog process not to schedule.

One other data point: I put another box into service for testing,
but it's considerably slower (a ~6 year old Xeon vs the Haswell).
Maybe it's just because it's so much slower that it'll take longer,
(or slow enough that the bug is masked) but that machine hasn't had
a problem yet in almost a day of runtime.

	Dave

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/