Date:   Wed, 23 Jan 2019 23:33:51 +0000
From:   "Lendacky, Thomas" <Thomas.Lendacky@....com>
To:     Paul Menzel <pmenzel@...gen.mpg.de>
CC:     Thomas Gleixner <tglx@...utronix.de>,
        "x86@...nel.org" <x86@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: tsc: Fast TSC calibration failed with sever AMD Ryzen processor
 (2200G, 2400G, Ryzen 7 1700)

On 1/23/19 6:56 AM, Paul Menzel wrote:
> Dear Tom,
> 
> 
> On 01/22/19 21:24, Lendacky, Thomas wrote:
>> On 1/22/19 10:53 AM, Paul Menzel wrote:
>>> [Adding Tom to CC]
> 
>>> On 01/14/19 11:09, Paul Menzel wrote:
>>>
>>>> On 01/11/19 21:43, Thomas Gleixner wrote:
>>>>
>>>>> On Mon, 7 Jan 2019, Paul Menzel wrote:
>>>>>> On 01/07/19 16:24, Thomas Gleixner wrote:
>>>>>>>> Linux 4.19.13 from Debian Sid/unstable logs the message below on the board MSI
>>>>>>>> MS-7A37/B350M MORTAR with the processor AMD Ryzen 3 2200G.
>>>>>>>>
>>>>>>>> As a result, the early time stamps do not seem to be working.
>>>>>>>
>>>>>>>>> [    0.000000] DMI: Micro-Star International Co., Ltd. MS-7A37/B350M MORTAR (MS-7A37), BIOS 1.I0 11/06/2018
>>>>>>>>> [    0.000000] tsc: Fast TSC calibration failed
>>>>>>>
>>>>>>> And the further boot log says:
>>>>>>>
>>>>>>> [    0.036000] tsc: Unable to calibrate against PIT
>>>>>>> [    0.036000] tsc: using HPET reference calibration
>>>>>>> [    0.036000] tsc: Detected 3500.117 MHz processor
>>>>>>>
>>>>>>> So the quick calibration in early boot fails because the PIT seems not to
>>>>>>> do what the kernel expects. Nothing we can cure :(
>>>>>>
>>>>>> I see. Can AMD confirm that this is the expected behavior? If yes, should
>>>>>> the fast TSC calibration be skipped on these devices?
> 
>> It's not expected behavior. All of the systems that I have access to do
>> not exhibit this issue. Having said that, I have a limited number of
>> systems available to me.
> 
> But as a data point, what Ryzen systems did you test with? Just to know, if
> there are configurations where the same processor behaves inconsistently.
> 
> Can you request one of the failing systems mentioned below to reproduce the
> problem?
> 
>> I don't have much experience in this area, but if it is something that
>> consistently occurs, you might try to see if you can better identify why
>> it fails. The message is issued in pit_hpet_ptimer_calibrate_cpu() in file
>> arch/x86/kernel/tsc.c.
> 
> With the attached patch applied, I get:
> 
>     [    0.000000] tsc: quick_pit_calibrate: break in if !pit_expect_msb, i = 42
>     [    0.000000] tsc: Fast TSC calibration failed, i = 42
>     [    0.000000] tsc: Using PIT calibration value
> 
> The functions `pit_verify_msb()` and `pit_expect_msb()` are:
> 
> ```
> static inline int pit_verify_msb(unsigned char val)
> {
>         /* Ignore LSB */
>         inb(0x42);
>         return inb(0x42) == val;
> }
> 
> static inline int pit_expect_msb(unsigned char val, u64 *tscp, unsigned long *deltap)
> {
>         int count;
>         u64 tsc = 0, prev_tsc = 0;
> 
>         for (count = 0; count < 50000; count++) {
>                 if (!pit_verify_msb(val))
>                         break;
>                 prev_tsc = tsc;
>                 tsc = get_cycles();
>         }
>         *deltap = get_cycles() - prev_tsc;
>         *tscp = tsc;
> 
>         /*
>          * We require _some_ success, but the quality control
>          * will be based on the error terms on the TSC values.
>          */
>         return count > 5;
> }
> ```
> 
> So count is smaller than or equal to 5, and `pit_verify_msb(val)` failed early,
> right?

Depends on what you mean by "failed early." The loop iterated 42 times
as it tried to reach the acceptable error rate, so pit_expect_msb() had
succeeded a number of times before failing.

One of the things I noticed while searching for info on the PIT is that
one of the specs [1] mentioned that just reading the counter values (as
opposed to using the counter latch command or the read-back command) could
return undefined values if the counters are being updated while being
read. I'm not sure if that is what is occurring or if that matters in
this day and age, I'm not familiar with this area.

There's also talk of SMIs messing with this calibration.  Could a long
running SMI (something close to 214us) run that results in keeping "count"
under 6? Again, just speculation on my part.

Thanks,
Tom

[1] http://www.scs.stanford.edu/10wi-cs140/pintos/specs/8254.pdf

> 
>>>>> It should work and we really don't want to add cpu family/model based
>>>>> decisions whether we invoke something or not. Those tables are stale before
>>>>> they hit mainline.
>>>>
>>>> Understood. If it’s supposed to work, any hints on how to debug this?
>>>>
>>>> Does some Linux kernel developers have an AMD Ryzen system, and can reproduce
>>>> the issue?
>>>>
>>>> It seems to fail with an AMD Ryzen 2400G too [1].
>>>
>>> We now have an HP EliteDesk 705 G4 MT with that processsor, showing the same
>>> problem.
>>>
>>> ```
>>> [    0.000000] Linux version 4.20.0.mx64.238 (root@...attivo.molgen.mpg.de) (gcc version 7.3.0 (GCC)) #1 SMP Mon Dec 24 14:50:00 CET 2018
>>> […]
>>> [    0.000000] NX (Execute Disable) protection: active
>>> [    0.000000] SMBIOS 3.1 present.
>>> [    0.000000] DMI: HP HP EliteDesk 705 G4 MT/83E7, BIOS Q06 Ver. 02.04.01 09/14/2018
>>> [    0.000000] tsc: Fast TSC calibration failed
>>> [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
>>> […]
>>> [    0.017860] smpboot: CPU0: AMD Ryzen 5 PRO 2400G with Radeon Vega Graphics (family: 0x17, model: 0x11, stepping: 0x0)
>>> […]
>>> ```
>>>
>>>> It also fails on an AMD Ryzen 7 1700 [2].
>>>>
>>>> ```
>>>> [ 0.000000] Linux version 4.15.0-kali3-amd64 (devel@...i.org) (gcc version 7.3.0 (Debian 7.3.0-16)) #1 SMP Debian 4.15.17-1kali1 (2018-04-25)
>>>> […]
>>>> [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>>>> [ 0.028000] tsc: Fast TSC calibration failed
>>>> [ 0.032000] tsc: PIT calibration matches HPET. 1 loops
>>>> [ 0.032000] tsc: Detected 2994.246 MHz processor
>>>> […]
>>>> [ 0.044000] smpboot: CPU0: AMD Ryzen 7 1700 Eight-Core Processor (family: 0x17, model: 0x1, stepping: 0x1)
>>>> ```
>>>>
>>>> It *works* here on one system with AMD Ryzen 5 PRO 1500 and Linux 4.14.87.
>>>>
>>>> ```
>>>> [    0.000000] Linux version 4.14.87.mx64.236 (root@...earollingstone.molgen.mpg.de) (gcc version 7.3.0 (GCC)) #1 SMP Mon Dec 10 09:48:57 CET 2018
>>>> […]
>>>> [    0.000000] tsc: Fast TSC calibration using PIT
>>>> […]
>>>> [    0.035000] smpboot: CPU0: AMD Ryzen 5 PRO 1500 Quad-Core Processor (family: 0x17, model: 0x1, stepping: 0x1)
>>>> ```
>>>
>>> How to continue from here? Is documentation for that available from AMD?
>>> I didn’t find a BKDG (Bios Kernel Developer Guide) at [3].
> 
> 
> Kind regards,
> 
> Paul
> 
> 
>>>> [1]: https://bbs.archlinux.org/viewtopic.php?pid=1781282#p1781282
>>>> [2]: https://forums.kali.org/showthread.php?40444-error-loading-amdgpu-drivers-AMD-RX580-driver[3]: https://developer.amd.com/resources/developer-guides-manuals/

Powered by blists - more mailing lists