| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20221129192915.GD4001@paulmck-ThinkPad-P17-Gen-1>
Date: Tue, 29 Nov 2022 11:29:15 -0800
From: "Paul E. McKenney" <paulmck@...nel.org>
To: Feng Tang <feng.tang@...el.com>
Cc: Thomas Gleixner <tglx@...utronix.de>, linux-kernel@...r.kernel.org,
john.stultz@...aro.org, sboyd@...nel.org, corbet@....net,
Mark.Rutland@....com, maz@...nel.org, kernel-team@...a.com,
neeraju@...eaurora.org, ak@...ux.intel.com,
zhengjun.xing@...el.com, Chris Mason <clm@...a.com>,
John Stultz <jstultz@...gle.com>,
Waiman Long <longman@...hat.com>
Subject: Re: [PATCH clocksource 1/3] clocksource: Reject bogus watchdog
clocksource measurements
On Mon, Nov 28, 2022 at 10:15:43AM +0800, Feng Tang wrote:
> On Wed, Nov 23, 2022 at 01:23:48PM -0800, Paul E. McKenney wrote:
> > On Wed, Nov 23, 2022 at 10:36:04AM +0800, Feng Tang wrote:
> > > On Tue, Nov 22, 2022 at 02:07:12PM -0800, Paul E. McKenney wrote:
> > > [...]
> > > > > > If PM_TIMER was involved, I would expect 'acpi_pm' instead of
> > > > > > refined-jiffies. Or am I misinterpreting the output and/or code?
> > > > >
> > > > > It's about timing. On a typical server platform, the clocksources
> > > > > init order could be:
> > > > > refined-jiffies --> hpet --> tsc-early --> acpi_pm --> tsc
> > > > >
> > > > > >From your log, TSC('tsc-early') is disabled before 'acpi_pm' get
> > > > > initialized, so 'acpi_pm' timer (if exist) had no chance to watchdog
> > > > > the tsc.
> > > > >
> > > > > > Either way, would it make sense to add CLOCK_SOURCE_MUST_VERIFY to
> > > > > > clocksource_hpet.flags?
> > > > >
> > > > > Maybe try below patch, which will skip watchdog for 'tsc-early',
> > > > > while giving 'acpi_pm' timer a chance to watchdog 'tsc'.
> > > > >
> > > > > diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
> > > > > index cafacb2e58cc..9840f0131764 100644
> > > > > --- a/arch/x86/kernel/tsc.c
> > > > > +++ b/arch/x86/kernel/tsc.c
> > > > > @@ -1131,8 +1131,7 @@ static struct clocksource clocksource_tsc_early = {
> > > > > .uncertainty_margin = 32 * NSEC_PER_MSEC,
> > > > > .read = read_tsc,
> > > > > .mask = CLOCKSOURCE_MASK(64),
> > > > > - .flags = CLOCK_SOURCE_IS_CONTINUOUS |
> > > > > - CLOCK_SOURCE_MUST_VERIFY,
> > > > > + .flags = CLOCK_SOURCE_IS_CONTINUOUS,
> > > > > .vdso_clock_mode = VDSO_CLOCKMODE_TSC,
> > > > > .enable = tsc_cs_enable,
> > > > > .resume = tsc_resume,
> > > >
> > > > Your mainline patch b50db7095fe0 ("x86/tsc: Disable clocksource watchdog
> > > > for TSC on qualified platorms") mitigates the issue so we are good for
> > > > the immediate future, at least assuming reliable TSC.
> > > >
> > > > But it also disables checking against HPET, hence my question about
> > > > marking clocksource_hpet.flags with CLOCK_SOURCE_MUST_VERIFY at boot time
> > > > on systems whose CPUs have constant_tsc, nonstop_tsc, and tsc_adjust.
> > >
> > > IIUC, this will make TSC to watchdog HPET every 500 ms. We have got
> > > report that the 500ms watchdog timer had big impact on some parallel
> > > workload on big servers, that was another factor for us to seek
> > > stopping the timer.
> >
> > Another approach would be to slow it down. Given the tighter bounds
> > on skew, it could be done every (say) 10 seconds while allowing
> > 2 milliseconds skew instead of the current 100 microseconds.
>
> Yes, this can reduce the OS noise much. One problem is if we make it
> a general interface, there is some clocksource whose warp time is
> less than 10 seconds, like ACPI PM_TIMER (3-4 seconds), and I don't
> know if other ARCHs have similar cases.
Maybe a simpler approach is for systems with such high sensitivity to
OS noise to simply disable the clocksource watchdog. ;-)
> > > Is this about the concern of possible TSC frequency calibration
> > > issue, as the 40 ms per second drift between HPET and TSC? With
> > > b50db7095fe0 backported, we also have another patch to force TSC
> > > calibration for those platforms which get the TSC freq directly
> > > from CPUID or MSR and don't have such info in dmesg:
> > > "tsc: Refined TSC clocksource calibration: 2693.509 MHz"
> > >
> > > https://lore.kernel.org/lkml/20220509144110.9242-1-feng.tang@intel.com/
> > >
> > > We did met tsc calibration issue due to some firmware issue, and
> > > this can help to catch it. You can try it if you think it's relevant.
> >
> > I am giving this a go, thank you!
>
> Thanks for spending time testing it!
And here are the results from setting tsc_force_recalibrate to 1:
$ dmesg | grep -E 'calibrat|clocksource'
[ 5.272939] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[ 16.830644] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417870 ns
[ 17.938020] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x36a8d32ce31, max_idle_ns: 881590731004 ns
[ 24.548583] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[ 49.762432] clocksource: Switched to clocksource tsc-early
[ 50.076769] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 55.615946] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x36a8d32ce31, max_idle_ns: 881590731004 ns
[ 55.640270] clocksource: Switched to clocksource tsc
[ 56.694371] tsc: Warning: TSC freq calibrated by CPUID/MSR differs from what is calibrated by HW timer, please check with vendor!!
[ 56.724550] tsc: Previous calibrated TSC freq: 1896.000 MHz
[ 56.737646] tsc: TSC freq recalibrated by [HPET]: 1975.000 MHz
Apologies for the delay, but reconfigurations put the system off the
net for some time.
Thanx, Paul
Powered by blists - more mailing lists