linux-kernel - Re: [PATCH] clocksource, prevent overflow in clocksource

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 18 Apr 2012 19:59:27 -0400
From:	Prarit Bhargava <prarit@...hat.com>
To:	John Stultz <john.stultz@...aro.org>
CC:	linux-kernel@...r.kernel.org, Thomas Gleixner <tglx@...utronix.de>,
	Salman Qazi <sqazi@...gle.com>, stable@...nel.org
Subject: Re: [PATCH] clocksource, prevent overflow in clocksource_cyc2ns

> 
> Hey Prarit,
>     So at tglx's prodding I took a look at the sysrq code, and the problem is
> the entire sysrq path runs with irqs disabled. As you note,with many cores and
> many processes, it can take a while to spit all that data out.
> 
> Instead of the earlier hack I suggested, would you try the following simpler
> one? I suspect we just need to touch the clocksource watchdog before returning. 
> This should avoid the TSC disqualification you're seeing. On systems using
> clocksources that wrap, we'll still lose time, since no time accumulation
> occurred during the long irq off period, but I think that's acceptable given
> this is not normal operation.
> 
> Let me know if this helps.

Hey John,

Thanks for continuing to work on this.  Coincidentally that exact patch was my
first attempt at resolving the problem as well.  The problem is that even after
touching the clocksource watchdog and restoring irqs the printk buffer can take
a LONG time to flush -- and that still will cause an overflow comparison.  So
fixing it with just a touch_clocksource_watchdog() isn't the right thing to do
IMO.  Maybe a combination of the printk() patch you suggested earlier and the
touch_clocksource_watchdog() is the right way to go but I'll leave that up to
tglx and yourself to decide on a correct fix.

There's also some additional information that I've been gathering on this issue;
I have seen *idle* systems switch to the hpet because the clocksource watchdog
hits the overflow comparison.  As expected it happens much less frequently on
newer kernels (linux.git top of tree) than older stable kernels (2.6.32 based)
due to the difference in shift values but it is happening in both cases.

The odd thing about this behaviour is that I would expect it to occur with the
same frequency on small systems as it does on large systems with linux.git as
the watchdog fires once/second.  AFAICT I do not see this on small systems but
see it only on systems with greater than 24 cpus (both Intel and AMD).

Using debug code similar to the dump code I previously provided, I can see that
every so often these large systems can hit a case where the tsc wraps and the
hpet is still monotonically increasing.  When the unstable calculation is
performed the result is obviously affected by the overflow.  Sometimes this
comparison overflow happens within 18 minutes, other times it can take hours or
days.

The other part of this puzzle is that if switch between the tsc and hpet every
10 seconds, and run a gettimeofday() comparison program, the gettimeofday()
program will return a backwards time[1] event usually within half-an-hour.  [I'm
just including this info here to point out that switching between clocksources
seems to cause some momentary instability.  Before anyone points this out I will
say that this not a "real world" bug.  I'm trying to find out if anyone actually
does switch from the tsc to hpet (and back) on multi-purposed systems.  I'm
hoping the answer to that is "no" :) ].

P.

[1] is not really a backwards time event.  The program does a comparison of
gettimeofday() values and returns an error if the new value is less than the
older value.  The gettimeofday() value jumps *forward* slightly for one returned
value when the clocksource switches from the tsc to the hpet (I have not see a
failure the other way around), and that causes the comparison to fail on the
next comparison of gettimeofday().  If I remove the code that returns an error
gettimeofday() returns to returning incrementally increasing values.  Typical
jumps are in the tens of microseconds range.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/