| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <4C9007F2.9020205@redhat.com>
Date: Tue, 14 Sep 2010 13:40:34 -1000
From: Zachary Amsden <zamsden@...hat.com>
To: Jan Kiszka <jan.kiszka@....de>
CC: Avi Kivity <avi@...hat.com>,
"kvm@...r.kernel.org" <kvm@...r.kernel.org>,
Marcelo Tosatti <mtosatti@...hat.com>,
Glauber Costa <glommer@...hat.com>,
Thomas Gleixner <tglx@...utronix.de>,
John Stultz <johnstul@...ibm.com>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [KVM timekeeping 10/35] Fix deep C-state TSC desynchronization
On 09/14/2010 12:26 PM, Jan Kiszka wrote:
> Am 14.09.2010 21:32, Zachary Amsden wrote:
>
>> On 09/14/2010 12:40 AM, Jan Kiszka wrote:
>>
>>> Am 14.09.2010 11:27, Avi Kivity wrote:
>>>
>>>
>>>> On 09/14/2010 11:10 AM, Jan Kiszka wrote:
>>>>
>>>>
>>>>> Am 20.08.2010 10:07, Zachary Amsden wrote:
>>>>>
>>>>>
>>>>>> When CPUs with unstable TSCs enter deep C-state, TSC may stop
>>>>>> running. This causes us to require resynchronization. Since
>>>>>> we can't tell when this may potentially happen, we assume the
>>>>>> worst by forcing re-compensation for it at every point the VCPU
>>>>>> task is descheduled.
>>>>>>
>>>>>> Signed-off-by: Zachary Amsden<zamsden@...hat.com>
>>>>>> ---
>>>>>> arch/x86/kvm/x86.c | 2 +-
>>>>>> 1 files changed, 1 insertions(+), 1 deletions(-)
>>>>>>
>>>>>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>>>>>> index 7fc4a55..52b6c21 100644
>>>>>> --- a/arch/x86/kvm/x86.c
>>>>>> +++ b/arch/x86/kvm/x86.c
>>>>>> @@ -1866,7 +1866,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu
>>>>>> *vcpu, int cpu)
>>>>>> }
>>>>>>
>>>>>> kvm_x86_ops->vcpu_load(vcpu, cpu);
>>>>>> - if (unlikely(vcpu->cpu != cpu)) {
>>>>>> + if (unlikely(vcpu->cpu != cpu) || check_tsc_unstable()) {
>>>>>> /* Make sure TSC doesn't go backwards */
>>>>>> s64 tsc_delta = !vcpu->arch.last_host_tsc ? 0 :
>>>>>> native_read_tsc() - vcpu->arch.last_host_tsc;
>>>>>>
>>>>>>
>>>>> For yet unknown reason, this commit breaks Linux guests here if they
>>>>> are
>>>>> started with only a single VCPU. They hang during boot, obviously no
>>>>> longer receiving interrupts.
>>>>>
>>>>> I'm using kvm-kmod against a 2.6.34 host kernel, so this may be a side
>>>>> effect of the wrapping, though I cannot imagine how.
>>>>>
>>>>> Anyone any ideas?
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Most likely, time went backwards, and some 'future - past' calculation
>>>> resulted in a negative sleep value which was then interpreted as
>>>> unsigned and resulted in a 2342525634 year sleep.
>>>>
>>>>
>>> Looks like that's the case on first glance at the apic state.
>>>
>>>
>> This compensation effectively nulls the delta between current and last TSC:
>>
>> if (unlikely(vcpu->cpu != cpu) || check_tsc_unstable()) {
>> /* Make sure TSC doesn't go backwards */
>> s64 tsc_delta = !vcpu->arch.last_host_tsc ? 0 :
>> native_read_tsc() -
>> vcpu->arch.last_host_tsc;
>> if (tsc_delta< 0)
>> mark_tsc_unstable("KVM discovered backwards TSC");
>> if (check_tsc_unstable())
>> kvm_x86_ops->adjust_tsc_offset(vcpu, -tsc_delta);
>> kvm_migrate_timers(vcpu);
>> vcpu->cpu = cpu;
>>
>> If TSC has advanced quite a bit due to a TSC jump during sleep(*), it
>> will adjust the offset backwards to compensate; similarly, if it has
>> gone backwards, it will advance the offset.
>>
>> In neither case should the visible TSC go backwards, assuming
>> last_host_tsc is recorded properly, and so kvmclock should be similarly
>> unaffected.
>>
>> Perhaps the guest is more intelligent than we hope, and is comparing two
>> different clocks: kvmclock or TSC with the rate of PIT interrupts. This
>> could result in negative arithmetic begin interpreted as unsigned. Are
>> you using PIT interrupt reinjection on this guest or passing
>> -no-kvm-pit-reinjection?
>>
>>
>>>
>>>
>>>> Does your guest use kvmclock, tsc, or some other time source?
>>>>
>>>>
>>> A kernel that has kvmclock support even hangs in SMP mode. The others
>>> pick hpet or acpi_pm. TSC is considered unstable.
>>>
>>>
>> SMP mode here has always and will always be unreliable. Are you running
>> on an Intel or AMD CPU? The origin of this code comes from a workaround
>> for (*) in vendor-specific code, and perhaps it is inappropriate for both.
>>
> I'm on a fairly new Intel i7 (M 620). And I accidentally rebooted my box
> a few hours ago. Well, the issue is gone now...
>
> So I looked into the system logs and found this:
>
> [18446744053.434939] PM: resume of devices complete after 4379.595 msecs
> [18446744053.457133] PM: Finishing wakeup.
> [18446744053.457135] Restarting tasks ...
> [ 0.000999] Marking TSC unstable due to KVM discovered backwards TSC
> [270103.974668] done.
>
> From that point on the box was on hpet, including the time I did the
> failing tests this morning. The kvm-kmod version loaded at this point
> was based on kvm.git df549cfc.
>
> But my /proc/cpuinfo claims "constant_tsc", and Linux is generally happy
> with using it as clock source. Does this tell you anything?
>
Yes, quite a bit.
It's possible that marking the TSC unstable with an actively running VM
causes a boundary condition that I had not accounted for. It's also
possible that the clocksource switch triggered some bad behavior.
This suggests two debugging techniques: I can manually switch the
clocksource, and I can also load a module which does nothing other than
mark the TSC unstable. Failing that, we can investigate PM suspend /
resume for possible issues.
I'll try this on my Intel boxes to see what happens.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists