linux-kernel - Re: kvm guests crash when running "perf kvm top"

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Z_aovIbwdKIIBMuq@google.com>
Date: Wed, 9 Apr 2025 10:05:00 -0700
From: Sean Christopherson <seanjc@...gle.com>
To: Seth Forshee <sforshee@...nel.org>
Cc: Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>, 
	Arnaldo Carvalho de Melo <acme@...nel.org>, Namhyung Kim <namhyung@...nel.org>, 
	Thomas Gleixner <tglx@...utronix.de>, Borislav Petkov <bp@...en8.de>, 
	Dave Hansen <dave.hansen@...ux.intel.com>, Paolo Bonzini <pbonzini@...hat.com>, x86@...nel.org, 
	linux-perf-users@...r.kernel.org, kvm@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: kvm guests crash when running "perf kvm top"

On Tue, Apr 08, 2025, Seth Forshee wrote:
> A colleague of mine reported kvm guest hangs when running "perf kvm top"
> with a 6.1 kernel. Initially it looked like the problem might be fixed
> in newer kernels, but it turned out to be perf changes which must avoid
> triggering the issue. I was able to reproduce the guest crashes with
> 6.15-rc1 in both the host and the guest when using an older version of
> perf. A bisect of perf landed on 7b100989b4f6 "perf evlist: Remove
> __evlist__add_default", but this doesn't look to be fixing any kind of
> issue like this.
> 
> This box has an Ice Lake CPU, and we can reproduce on other Ice Lakes
> but could not reproduce on another box with Broadwell. On Broadwell
> guests would crash with older kernels in the host, but this was fixed by
> 971079464001 "KVM: x86/pmu: fix masking logic for
> MSR_CORE_PERF_GLOBAL_CTRL". That does not fix the issues we see on Ice
> Lake.
> 
> When the guests crash we aren't getting any output on the serial
> console, but I got this from a memory dump:

...

> Oops: 0000 [#1] PREEMPT SMP NOPTI
> BUG: kernel NULL pointer dereference, address: 000000000000002828

FWIW, this is probably slightly corrupted.  When I run with EPT disabled, to force
KVM to intercept #PFs, the reported CR2 is 0x28.  Which is consistent with the
guest having DS_AREA=0.  I.e. the CPU is attempting to store into the DS/PEBS
buffer.

As suspected, the issue is PEBS.  After adding a tracepoint to capture the MSRs
that KVM loads as part of the perf transition, it's easy to see that PEBS_ENABLE
gets loaded with a non-zero value immediate before death, doom, and destruction.

  CPU 0: kvm_entry: vcpu 0, rip 0xffffffff81000aa0 intr_info 0x80000b0e error_code 0x00000000
  CPU 0: kvm_perf_msr: MSR 38f: host 1000f000000fe guest 1000f000000ff
  CPU 0: kvm_perf_msr: MSR 600: host fffffe57186af000 guest 0
  CPU 0: kvm_perf_msr: MSR 3f2: host 0 guest 0
  CPU 0: kvm_perf_msr: MSR 3f1: host 0 guest 1
  CPU 0: kvm_exit: vcpu 0 reason EXCEPTION_NMI rip 0xffffffff81000aa0 info1 0x0000000000000028 intr_info 0x80000b0e error_code 0x00000000

The underlying issue is that KVM's current PMU virtualization uses perf_events
to proxy guest events, i.e. piggybacks intel_ctrl_guest_mask, which is also used
by host userspace to communicate exclude_host/exclude_guest.  And so perf's
intel_guest_get_msrs() allows using PEBS for guest events, but only if perf isn't
using PEBS for host events.

I didn't actually verify that "perf kvm top" generates for events, but I assuming
it's creating a precise, a.k.a. PEBS, event that measures _only_ guest, i.e.
excludes host.  That causes a false positive of sorts in intel_guest_get_msrs(),
and ultimately results in KVM running the guest with a PEBS event enabled, even
though the guest isn't using the (virtual) PMU.

Pre-ICX CPUs don't isolate PEBS events across the guest/host boundary, and so
perf/KVM hard disable PEBS on VM-Enter.  And a simple (well, simple for perf)
precise event doesn't cause problems, because perf/KVM will disable PEBS events
that are counting the host.  I.e. if a PEBS event counts host *and* guest, it's
"fine".

Long story short, masking PEBS_ENABLE with the guest's value (in addition to
what perf allows) fixes the issue on my end.  Assuming testing goes well, I'll
post this as a proper patch.

--
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index cdb19e3ba3aa..1d01fb43a337 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4336,7 +4336,7 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data)
        arr[pebs_enable] = (struct perf_guest_switch_msr){
                .msr = MSR_IA32_PEBS_ENABLE,
                .host = cpuc->pebs_enabled & ~cpuc->intel_ctrl_guest_mask,
-               .guest = pebs_mask & ~cpuc->intel_ctrl_host_mask,
+               .guest = pebs_mask & ~cpuc->intel_ctrl_host_mask & kvm_pmu->pebs_enable,
        };

        if (arr[pebs_enable].host) {
--

> Let me know if I can provide any additional information or testing.

Uber nit: in the future, explicitly state whether a command is being run in the
guest or host.  I had a brain fart and it took me an embarrasingly long time to
grok that running "perf kvm top" in the guest would be nonsensical.