linux-kernel - Re: [PATCH] KVM: Documentation: document how KVM is tested

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <aHA-AVQQkUYmcsSP@google.com>
Date: Thu, 10 Jul 2025 15:26:09 -0700
From: Sean Christopherson <seanjc@...gle.com>
To: Paolo Bonzini <pbonzini@...hat.com>
Cc: linux-kernel@...r.kernel.org, kvm@...r.kernel.org, 
	Rick Edgecombe <rick.p.edgecombe@...el.com>
Subject: Re: [PATCH] KVM: Documentation: document how KVM is tested

On Thu, Jul 10, 2025, Paolo Bonzini wrote:
> +Testing of KVM code
> +-------------------
> +
> +All features contributed to KVM, and in many cases bugfixes too, should be
> +accompanied by some kind of tests and/or enablement in open source guests
> +and VMMs.  KVM is covered by multiple test suites:
> +
> +*Selftests*
> +  These are low level tests included in the kernel tree.  While relatively
> +  challenging to write, they allow granular testing of kernel APIs.  This
> +  includes API failure scenarios, invoking APIs after specific guest
> +  instructions, and testing multiple calls to ``KVM_CREATE_VM`` within
> +  a single test.
> +
> +``kvm-unit-tests``
> +  A collection of small guests that test CPU and emulated device features
> +  from a guest's perspective.  They run under QEMU or ``kvmtool``,
> +  are relatively easy to write.  

IMO, selftests aren't meaningfully harder to write than KVM-Unit-Tests.  They
require a bit more boilerplate to get started, but in many ways I would argue
they're easier to write because the C code has full control over the VM.

E.g. doing multi-threaded testing in KUT gets painful because it often requires
using IPIs to coordinate between vCPUs, trying to test different vCPU models
requires writing multiple config entries, etc.

And I think the boilerplate "problem" is mostly solvable, by expanding the KVM
selftests harness to handle the bulk of the work for simple tests, e.g. see
https://lore.kernel.org/all/ZjUwqEXPA5QVItyX@google.com.

The reason I bring this up is that I'd prefer to steer people in the direction
of KVM selftests over KVM-Unit-Tests.  Or at the very least, not steer them too
hard towards KVM-Unit-Tests by suggesting that KVM-Unit-Tests will require less
effort.

Because selftests can define the vCPU model and VM capabilities, they can provide
significantly more coverage than KUT in many cases.  E.g. one of the things on my
wish list is to rewrite KUT's MSR test as a KVM selftest that tests both guest and
host userspace behavior, and that permutes over various vCPU models.

And from a maintenance perspective, having selftests in-tree is a huge benefit for
me.  That alone is honestly reason enough for me to push for selftests over KUT.

All that said, this looks awesome!  Thanks for writing this up!

> + `kvm-`unit-tests`` are generally not
> +  KVM-specific; they can be run with any accelerator that QEMU support
> +  or even on bare metal, making it possible to compare behavior across
> +  hypervisors and processor families.