linux-kernel - Re: [RFC PATCH] KVM: x86: Fix APIC page invalidation race

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <7B9024C7-98D0-4940-91AE-40BCDE555C8F@nutanix.com>
Date:   Tue, 9 Jun 2020 01:04:03 +0000
From:   Eiichi Tsukata <eiichi.tsukata@...anix.com>
To:     Paolo Bonzini <pbonzini@...hat.com>
CC:     "sean.j.christopherson@...el.com" <sean.j.christopherson@...el.com>,
        "vkuznets@...hat.com" <vkuznets@...hat.com>,
        "wanpengli@...cent.com" <wanpengli@...cent.com>,
        "jmattson@...gle.com" <jmattson@...gle.com>,
        "joro@...tes.org" <joro@...tes.org>,
        "tglx@...utronix.de" <tglx@...utronix.de>,
        "mingo@...hat.com" <mingo@...hat.com>,
        "bp@...en8.de" <bp@...en8.de>, "x86@...nel.org" <x86@...nel.org>,
        "hpa@...or.com" <hpa@...or.com>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Felipe Franciosi <felipe@...anix.com>,
        "rkrcmar@...hat.com" <rkrcmar@...hat.com>,
        Andrea Arcangeli <aarcange@...hat.com>
Subject: Re: [RFC PATCH] KVM: x86: Fix APIC page invalidation race



> On Jun 8, 2020, at 22:13, Paolo Bonzini <pbonzini@...hat.com> wrote:
> 
> On 06/06/20 06:26, Eiichi Tsukata wrote:
>> Commit b1394e745b94 ("KVM: x86: fix APIC page invalidation") tried to
>> fix inappropriate APIC page invalidation by re-introducing arch specific
>> kvm_arch_mmu_notifier_invalidate_range() and calling it from
>> kvm_mmu_notifier_invalidate_range_start. But threre could be the
>> following race because VMCS APIC address cache can be updated
>> *before* it is unmapped.
>> 
>> Race:
>>  (Invalidator) kvm_mmu_notifier_invalidate_range_start()
>>  (Invalidator) kvm_make_all_cpus_request(kvm, KVM_REQ_APIC_PAGE_RELOAD)
>>  (KVM VCPU) vcpu_enter_guest()
>>  (KVM VCPU) kvm_vcpu_reload_apic_access_page()
>>  (Invalidator) actually unmap page
>> 
>> Symptom:
>>  The above race can make Guest OS see already freed page and Guest OS
>> will see broken APIC register values.
> 
> This is not exactly the issue.  The values in the APIC-access page do
> not really matter, the problem is that the host physical address values
> won't match between the page tables and the APIC-access page address.
> Then the processor will not trap APIC accesses, and will instead show
> the raw contents of the APIC-access page (zeroes), and cause the crash
> as you mention below.
> 
> Still, the race explains the symptoms and the patch matches this text in
> include/linux/mmu_notifier.h:
> 
> 	 * If the subsystem
>         * can't guarantee that no additional references are taken to
>         * the pages in the range, it has to implement the
>         * invalidate_range() notifier to remove any references taken
>         * after invalidate_range_start().
> 
> where the "additional reference" is in the VMCS: because we have to
> account for kvm_vcpu_reload_apic_access_page running between
> invalidate_range_start() and invalidate_range_end(), we need to
> implement invalidate_range().
> 
> The patch seems good, but I'd like Andrea Arcangeli to take a look as
> well so I've CCed him.
> 
> Thank you very much!
> 
> Paolo
> 

Hello Paolo

Thanks for detailed explanation!
I’ll fix the commit message like this:

```
Symptom:
  The above race can cause mismatch between the page tables and the
APIC-access page address in VMCS.Then the processor will not trap APIC
accesses, and will instead show the raw contents of the APIC-access page
(zeroes). Especially, Windows OS checks LAPIC modification so it can cause
BSOD crash with BugCheck CRITICAL_STRUCTURE_CORRUPTION (109). These symptoms
are the same as we previously saw in
https://bugzilla.kernel.org/show_bug.cgi?id=197951
and we are currently seeing in
https://bugzilla.redhat.com/show_bug.cgi?id=1751017.

To prevent mismatch between page tables and APIC-access page address,
this patch calls kvm_arch_mmu_notifier_invalidate_range() from
kvm_mmu_notifier_invalidate_range() instead of ..._range_start().
We need to implement invalidate_range() because we have to
account for kvm_vcpu_reload_apic_access_page() running between
invalidate_range_start() and invalidate_range_end().
```


Best

Eiichi