linux-kernel - Re: [PATCH 8/9] clocksource: Improve unstable clocksource detection

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150831211233.GA1413758@devbig257.prn2.facebook.com>
Date:	Mon, 31 Aug 2015 14:12:33 -0700
From:	Shaohua Li <shli@...com>
To:	Thomas Gleixner <tglx@...utronix.de>
CC:	John Stultz <john.stultz@...aro.org>,
	lkml <linux-kernel@...r.kernel.org>,
	Prarit Bhargava <prarit@...hat.com>,
	Richard Cochran <richardcochran@...il.com>,
	Daniel Lezcano <daniel.lezcano@...aro.org>,
	Ingo Molnar <mingo@...nel.org>,
	Clark Williams <williams@...hat.com>,
	Steven Rostedt <rostedt@...dmis.org>
Subject: Re: [PATCH 8/9] clocksource: Improve unstable clocksource detection

On Wed, Aug 26, 2015 at 10:15:33AM -0700, Shaohua Li wrote:
> On Tue, Aug 18, 2015 at 10:18:09PM +0200, Thomas Gleixner wrote:
> > On Tue, 18 Aug 2015, John Stultz wrote:
> > > On Tue, Aug 18, 2015 at 12:28 PM, Thomas Gleixner <tglx@...utronix.de> wrote:
> > > > On Tue, 18 Aug 2015, John Stultz wrote:
> > > >> On Tue, Aug 18, 2015 at 1:38 AM, Thomas Gleixner <tglx@...utronix.de> wrote:
> > > >> > On Mon, 17 Aug 2015, John Stultz wrote:
> > > >> >> On Mon, Aug 17, 2015 at 3:04 PM, Thomas Gleixner <tglx@...utronix.de> wrote:
> > > >> >> > On Mon, 17 Aug 2015, John Stultz wrote:
> > > >> >> >
> > > >> >> >> From: Shaohua Li <shli@...com>
> > > >> >> >>
> > > >> >> >> >From time to time we saw TSC is marked as unstable in our systems, while
> > > >> >> >
> > > >> >> > Stray '>'
> > > >> >> >
> > > >> >> >> the CPUs declare to have stable TSC. Looking at the clocksource unstable
> > > >> >> >> detection, there are two problems:
> > > >> >> >> - watchdog clock source wrap. HPET is the most common watchdog clock
> > > >> >> >>   source. It's 32-bit and runs in 14.3Mhz. That means the hpet counter
> > > >> >> >>   can wrap in about 5 minutes.
> > > >> >> >> - threshold isn't scaled against interval. The threshold is 0.0625s in
> > > >> >> >>   0.5s interval. What if the actual interval is bigger than 0.5s?
> > > >> >> >>
> > > >> >> >> The watchdog runs in a timer bh, so hard/soft irq can defer its running.
> > > >> >> >> Heavy network stack softirq can hog a cpu. IPMI driver can disable
> > > >> >> >> interrupt for a very long time.
> > > >> >> >
> > > >> >> > And they hold off the timer softirq for more than a second? Don't you
> > > >> >> > think that's the problem which needs to be fixed?
> > > >> >>
> > > >> >> Though this is an issue I've experienced (and tried unsuccessfully to
> > > >> >> fix in a more complicated way) with the RT kernel, where high priority
> > > >> >> tasks blocked the watchdog long enough that we'd disqualify the TSC.
> > > >> >
> > > >> > Did it disqualify the watchdog due to HPET wraparounds (5 minutes) or
> > > >> > due to the fixed threshold being applied?
> > > >>
> > > >> This was years ago, but in my experience, the watchdog false positives
> > > >> were due to HPET wraparounds.
> > > >
> > > > Blocking stuff for 5 minutes is insane ....
> > > 
> > > Yea. It was usually due to -RT stress testing, which keept the
> > > machines busy for quite awhile. But again, if you have machines being
> > > maxed out with networking load, etc, even for long amounts of time, we
> > > still want to avoid false positives. Because after the watchdog
> > 
> > The networking softirq does not hog the other softirqs. It has a limit
> > on processing loops and then goes back to let the other softirqs be
> > handled. So no, I doubt that heavy networking can cause this. If it
> > does then we have some other way more serious problems.
> > 
> > I can see the issue with RT stress testing, but not with networking in
> > mainline.
> 
> Ok, the issue is triggerd in my kvm guest, I guess it's easier to
> trigger in kvm because hpet is 100Mhz.
> 
> [  135.930067] clocksource: timekeeping watchdog: Marking clocksource 'tsc' as unstable because the skew is too large:
> [  135.930095] clocksource:                       'hpet' wd_now: 2bc19ea0 wd_last: 6c4e5570 mask: ffffffff
> [  135.930105] clocksource:                       'tsc' cs_now: 481250b45b cs_last: 219e6efb50 mask: ffffffffffffffff
> [  135.938750] clocksource: Switched to clocksource hpet
> 
> The HPET clock is 100MHz, CPU speed is 2200MHz, kvm is passed correct cpu
> info, so guest cpuinfo shows TSC is stable.
> 
> hpet interval is ((0x2bc19ea0 - 0x6c4e5570) & 0xffffffff) / 100000000 = 32.1s.
> 
> The HPET wraps interval is 0xffffffff / 100000000 = 42.9s
> 
> tsc interval is (0x481250b45b - 0x219e6efb50) / 2200000000 = 75s
> 
> 32.1 + 42.9 = 75
> 
> The example shows hpet wraps, while tsc is marked unstable

Thomas & John,
Is this data enough to prove TSC unstable issue can be triggered by HPET
wrap? I can resend the patch with the data included.

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/