Date:	Wed, 18 Apr 2012 17:18:42 -0700
From:	John Stultz <john.stultz@...aro.org>
To:	Prarit Bhargava <prarit@...hat.com>
CC:	linux-kernel@...r.kernel.org, Thomas Gleixner <tglx@...utronix.de>,
	Salman Qazi <sqazi@...gle.com>, stable@...nel.org
Subject: Re: [PATCH] clocksource, prevent overflow in clocksource_cyc2ns

On 04/18/2012 04:59 PM, Prarit Bhargava wrote:
>
> Hey John,
>
> Thanks for continuing to work on this.  Coincidentally that exact patch was my
> first attempt at resolving the problem as well.  The problem is that even after
> touching the clocksource watchdog and restoring irqs the printk buffer can take
> a LONG time to flush -- and that still will cause an overflow comparison.  So
> fixing it with just a touch_clocksource_watchdog() isn't the right thing to do
> IMO.  Maybe a combination of the printk() patch you suggested earlier and the
> touch_clocksource_watchdog() is the right way to go but I'll leave that up to
> tglx and yourself to decide on a correct fix.
:( That's a bummer. Something similar may be useful on the printk side 
as well.


> There's also some additional information that I've been gathering on this issue;
> I have seen *idle* systems switch to the hpet because the clocksource watchdog
> hits the overflow comparison.  As expected it happens much less frequently on
> newer kernels (linux.git top of tree) than older stable kernels (2.6.32 based)
> due to the difference in shift values but it is happening in both cases.

Some of the recent adjustments for more robust shift calculations may 
partially be responsible for the improvement. Although I'm not sure why 
idle systems (that don't halt the TSC in idle) would trip this.  Do let 
me know if you find any particular way of reproducing this.

> The odd thing about this behaviour is that I would expect it to occur with the
> same frequency on small systems as it does on large systems with linux.git as
> the watchdog fires once/second.  AFAICT I do not see this on small systems but
> see it only on systems with greater than 24 cpus (both Intel and AMD).
Hrm.

> Using debug code similar to the dump code I previously provided, I can see that
> every so often these large systems can hit a case where the tsc wraps and the
> hpet is still monotonically increasing.  When the unstable calculation is
> performed the result is obviously affected by the overflow.  Sometimes this
> comparison overflow happens within 18 minutes, other times it can take hours or
> days.
TSC wraps? Are you sure that's what you see? Or do you have that 
switched? With the HPET wrapping?


> The other part of this puzzle is that if switch between the tsc and hpet every
> 10 seconds, and run a gettimeofday() comparison program, the gettimeofday()
> program will return a backwards time[1] event usually within half-an-hour.  [I'm
> just including this info here to point out that switching between clocksources
> seems to cause some momentary instability.  Before anyone points this out I will
> say that this not a "real world" bug.  I'm trying to find out if anyone actually
> does switch from the tsc to hpet (and back) on multi-purposed systems.  I'm
> hoping the answer to that is "no" :) ].
So, there were some recent fixes for 3.4 to address an issue 
specifically around inconsistencies at clocksource switch time:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a939e817aa7e199d2fff05a67cb745be32dd5c2d
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f695cf94837de53864180400cbac42cfa370426f

I definitely want to make sure any sort of inconsistencies like that are 
resolved. So let me know if you can still trigger anything like that 
with the latest 3.4 kernel.

thanks
-john

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists