linux-kernel - Re: [KVM timekeeping 10/35] Fix deep C-state TSC desynchronization

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Sat, 18 Sep 2010 13:53:50 -1000
From:	Zachary Amsden <zamsden@...hat.com>
To:	Jan Kiszka <jan.kiszka@....de>
CC:	Glauber Costa <glommer@...hat.com>, kvm@...r.kernel.org,
	Avi Kivity <avi@...hat.com>,
	Marcelo Tosatti <mtosatti@...hat.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	John Stultz <johnstul@...ibm.com>, linux-kernel@...r.kernel.org
Subject: Re: [KVM timekeeping 10/35] Fix deep C-state TSC desynchronization

On 09/17/2010 12:31 PM, Zachary Amsden wrote:
> On 09/17/2010 12:09 PM, Zachary Amsden wrote:
>> On 09/15/2010 08:27 AM, Jan Kiszka wrote:
>>> Am 15.09.2010 14:32, Glauber Costa wrote:
>>>> On Wed, Sep 15, 2010 at 10:09:33AM +0200, Jan Kiszka wrote:
>>>>>> In any case, I'll proceed with the forcing of unstable TSC and HPET
>>>>>> clocksource and see what happens.
>>>>> I tried that before, but it did not trigger the issue that kvm-clock
>>>>> guests no longer boot properly. This only happens if the TSC is 
>>>>> marked
>>>>> unstable.
>>>> even artificially marked unstable ?
>>>>
>>> Yes. As soon as I hack tsc_unstable to 1, things go wrong. When I hack
>>> it back to 0, guest that wants kvm-clock boots again and seem to run 
>>> fine.
>>>
>>> This is issue #2, I guess. Issue #2 remains that the TSC is marked
>>> unstable. I have the feeling that this is bogus, maybe due to lacking
>>> suspend/resume awareness? The tsc clocksource does
>>>
>>>     clocksource_tsc.cycle_last = 0;
>>>
>>> on resume...
>>>
>>> Jan
>>>
>>
>> I have now reproduced this exactly.  Shouldn't be long before I have 
>> a solution.
>

Wow, bug was subtle.  Here's what happens:

kvmclock enabled
unstable TSC compensation erases TSC gain; requests kvmclock update
we get preempted after writing new kvmclock value, so HV is never entered
unstable TSC compensation erases TSC gain again; requests kvmclock update
kvmclock overflow compensation underflows because vcpu->last_guest_tsc
kvmclock advances randomly
HV entered
HV exit sets last_guest_tsc, making bug invisible

The solution is to set vcpu->last_guest_tsc always when updating 
kvmclock.  Note the bug can occur independently of the unstable TSC 
compensation, which makes it more likely only because it requests more 
kvmclock updates.  The fundamental issue is the lack of complete state 
update caused by preemption before HV is entered, but after kvmclock is 
written.

I have a patch series which fixes this now, but it has some debug gunk 
in it I need to get rid of.

I'll try to send it in the next few hours, as I'm on vacation (parents 
visiting) next week.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/