linux-kernel - Re: [FYI PATCH] Revert "KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Yj0FYSC2sT4k/ELl@google.com>
Date:   Thu, 24 Mar 2022 23:57:21 +0000
From:   Sean Christopherson <seanjc@...gle.com>
To:     Paolo Bonzini <pbonzini@...hat.com>
Cc:     linux-kernel@...r.kernel.org, kvm@...r.kernel.org,
        Vitaly Kuznetsov <vkuznets@...hat.com>,
        Sean Christopherson <sean.j.christopherson@...el.com>
Subject: Re: [FYI PATCH] Revert "KVM: x86/mmu: Zap only TDP MMU leafs in
 kvm_zap_gfn_range()"

On Mon, Mar 21, 2022, Paolo Bonzini wrote:
> On 3/18/22 17:48, Paolo Bonzini wrote:
> > This reverts commit cf3e26427c08ad9015956293ab389004ac6a338e.
> > 
> > Multi-vCPU Hyper-V guests started crashing randomly on boot with the
> > latest kvm/queue and the problem can be bisected the problem to this
> > particular patch. Basically, I'm not able to boot e.g. 16-vCPU guest
> > successfully anymore. Both Intel and AMD seem to be affected. Reverting
> > the commit saves the day.
> > 
> > Reported-by: Vitaly Kuznetsov <vkuznets@...hat.com>
> > Signed-off-by: Paolo Bonzini <pbonzini@...hat.com>
> 
> This is not enough, the following is also needed to account
> for "KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow
> pages":
> 
> ------------------- 8< ----------------
> From: Paolo Bonzini <pbonzini@...hat.com>
> Subject: [PATCH] kvm: x86/mmu: Flush TLB before zap_gfn_range releases RCU
> 
> Since "KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()"
> is going to be reverted, it's not going to be true anymore that
> the zap-page flow does not free any 'struct kvm_mmu_page'.  Introduce
> an early flush before tdp_mmu_zap_leafs() returns, to preserve
> bisectability.

Can I have 1-2 weeks to try and root cause and fix the underlying issue before
sending reverts to Linus?  I really don't want to paper over a TLB flushing bug
or an off-by-one bug, and I really, really don't want to end up with another
scenario where KVM zaps everything just because.

Vitaly, can you provide repro instructions?  A nearly-complete QEMU command line
would be wonderful :-)  Is the issue unique to any particular guest kernel?  I've
been unable to repro with a 112 vCPU Linux guest with these Hyper-V enlightenments:

$ : dm | grep -i hyper-v
[    0.000000] Hypervisor detected: Microsoft Hyper-V
[    0.000000] Hyper-V: privilege flags low 0x2aff, high 0x830, hints 0x4e2c, misc 0x80d12
[    0.000000] Hyper-V Host Build:14393-10.0-0-0.0
[    0.000000] Hyper-V: Nested features: 0x80101
[    0.000000] Hyper-V: LAPIC Timer Frequency: 0x3d0900
[    0.000000] Hyper-V: Using hypercall for remote TLB flush
[    0.000004] tsc: Marking TSC unstable due to running on Hyper-V
[    0.129376] Booting paravirtualized kernel on Hyper-V
[    0.140419] Hyper-V: PV spinlocks disabled
[    0.247500] Hyper-V: Using IPI hypercalls
[    0.247502] Hyper-V: Using enlightened APIC (x2apic mode)

Actually, since this is apparently specific to kvm_zap_gfn_range(), can you add
printk "tracing" in update_mtrr(), kvm_post_set_cr0(), and __kvm_request_apicv_update()
to see what is actually triggering zaps?  Capturing the start and end GFNs would be very
helpful for the MTRR case.

The APICv update seems unlikely to affect only Hyper-V guests, though there is the auto
EOI crud.  And the other two only come into play with non-coherent DMA.  In other words,
figuring out exactly what sequence leads to failure should be straightforward.