linux-kernel - Re: frequent lockups in 3.18rc4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20141203193029.GA1122@redhat.com>
Date:	Wed, 3 Dec 2014 14:30:29 -0500
From:	Dave Jones <davej@...hat.com>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	Chris Mason <clm@...com>,
	Mike Galbraith <umgwanakikbuti@...il.com>,
	Ingo Molnar <mingo@...nel.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Dâniel Fraga <fragabr@...il.com>,
	Sasha Levin <sasha.levin@...cle.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	John Stultz <john.stultz@...aro.org>
Subject: Re: frequent lockups in 3.18rc4

On Wed, Dec 03, 2014 at 11:25:29AM -0800, Linus Torvalds wrote:
 > On Wed, Dec 3, 2014 at 11:00 AM, Dave Jones <davej@...hat.com> wrote:
 > >
 > > So right after sending my last mail, I rebooted, and restarted the run
 > > on the same kernel again.
 > >
 > > As I was writing this mail, this happened.
 > >
 > > [  524.420897] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [trinity-c178:20182]
 > >
 > > and that's all that made it over the console. I couldn't log in via ssh,
 > > and thought "ah-ha, so it IS bad".  I walked over to reboot it, and
 > > found I could actually log in on the console. check out this dmesg..
 > >
 > > [  503.683055] Clocksource tsc unstable (delta = -95946009388 ns)
 > > [  503.692038] Switched to clocksource hpet
 > > [  524.420897] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [trinity-c178:20182]
 > 
 > Interesting. That whole NMI watchdog thing happens pretty much 22s
 > after the "TSC unstable" message.
 > 
 > Have you ever seen that TSC issue before? The watchdog relies on
 > comparing get_timestamp() differences, so if the timestamp was
 > incorrect...
 
yeah, quite a lot.

# grep tsc\ unstable /var/log/messages* | wc -l
71

Usually happens pretty soon after boot, once I start the fuzzing run.
It sometimes occurs quite some time before the NMI issue though.

eg:

Dec  3 11:50:24 binary kernel: [ 4253.432642] Clocksource tsc unstable (delta = -243666538341 ns)
...
Dec  3 13:24:28 binary kernel: [ 9862.915562] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [trinity-c29:13237]


	Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/