Message-ID: <20231108111806.92604-34-nsaenz@amazon.com>
Date:   Wed, 8 Nov 2023 11:18:06 +0000
From:   Nicolas Saenz Julienne <nsaenz@...zon.com>
To:     <kvm@...r.kernel.org>
CC:     <linux-kernel@...r.kernel.org>, <linux-hyperv@...r.kernel.org>,
        <pbonzini@...hat.com>, <seanjc@...gle.com>, <vkuznets@...hat.com>,
        <anelkz@...zon.com>, <graf@...zon.com>, <dwmw@...zon.co.uk>,
        <jgowans@...zon.com>, <corbert@....net>, <kys@...rosoft.com>,
        <haiyangz@...rosoft.com>, <decui@...rosoft.com>, <x86@...nel.org>,
        <linux-doc@...r.kernel.org>,
        Nicolas Saenz Julienne <nsaenz@...zon.com>
Subject: [RFC 33/33] Documentation: KVM: Introduce "Emulating Hyper-V VSM with KVM"

Introduce "Emulating Hyper-V VSM with KVM", which describes the KVM APIs
made available to a VMM that wants to emulate Hyper-V's VSM.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@...zon.com>
---
 .../virt/kvm/x86/emulating-hyperv-vsm.rst     | 136 ++++++++++++++++++
 1 file changed, 136 insertions(+)
 create mode 100644 Documentation/virt/kvm/x86/emulating-hyperv-vsm.rst

diff --git a/Documentation/virt/kvm/x86/emulating-hyperv-vsm.rst b/Documentation/virt/kvm/x86/emulating-hyperv-vsm.rst
new file mode 100644
index 000000000000..8f76bf09c530
--- /dev/null
+++ b/Documentation/virt/kvm/x86/emulating-hyperv-vsm.rst
@@ -0,0 +1,136 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==============================
+Emulating Hyper-V VSM with KVM
+==============================
+
+Hyper-V's Virtual Secure Mode (VSM) is a virtualisation security feature
+that leverages the hypervisor to create secure execution environments
+within a guest. VSM is documented as part of Microsoft's Hypervisor Top
+Level Functional Specification[1].
+
+Emulating Hyper-V's Virtual Secure Mode with KVM requires coordination
+between KVM and the VMM. Most of the VSM state and configuration is left
+to be handled by user-space, but some has made its way into KVM. This
+document describes the mechanisms through which a VMM can implement VSM
+support.
+
+Virtual Trust Levels
+--------------------
+
+The main concept VSM introduces are Virtual Trust Levels or VTLs. Each
+VTL is a CPU mode, with its own private CPU architectural state,
+interrupt subsystem (limited to a local APIC), and memory access
+permissions. VTLs are hierarchical, where VTL0 corresponds to normal
+guest execution and VTL > 0 to privileged execution contexts. In
+practice, when virtualising Windows on top of KVM, we only see VTL0 and
+VTL1. Although the spec allows going all the way to VTL15. VTLs are
+orthogonal to ring levels, so each VTL is capable of runnig its own
+operating system and user-space[2].
+
+  ┌──────────────────────────────┐ ┌──────────────────────────────┐
+  │ Normal Mode (VTL0)           │ │ Secure Mode (VTL1)           │
+  │ ┌──────────────────────────┐ │ │ ┌──────────────────────────┐ │
+  │ │   User-mode Processes    │ │ │ │Secure User-mode Processes│ │
+  │ └──────────────────────────┘ │ │ └──────────────────────────┘ │
+  │ ┌──────────────────────────┐ │ │ ┌──────────────────────────┐ │
+  │ │         Kernel           │ │ │ │      Secure Kernel       │ │
+  │ └──────────────────────────┘ │ │ └──────────────────────────┘ │
+  └──────────────────────────────┘ └──────────────────────────────┘
+  ┌───────────────────────────────────────────────────────────────┐
+  │                         Hypervisor/KVM                        │
+  └───────────────────────────────────────────────────────────────┘
+  ┌───────────────────────────────────────────────────────────────┐
+  │                           Hardware                            │
+  └───────────────────────────────────────────────────────────────┘
+
+VTLs break the core assumption that a vCPU has a single architectural
+state, lAPIC state, SynIC state, etc. As such, each VTL is modeled as a
+distinct KVM vCPU, with the restriction that only one is allowed to run
+at any moment in time. Having multiple KVM vCPUs tracking a single guest
+CPU complicates vCPU numbering. VMs that enable VSM are expected to use
+CAP_APIC_ID_GROUPS to segregate vCPUs (and their lAPICs) into different
+groups. For example, a 4 CPU VSM VM will setup the APIC ID groups feature
+so only the first two bits of the APIC ID are exposed to the guest. The
+remaining bits represent the vCPU's VTL. The 'sibling' vCPU to VTL0's
+vCPU2 at VTL3 will have an APIC ID of 0xE. Using this approach a VMM and
+KVM are capable of querying a vCPU's VTL, or finding the vCPU associated
+to a specific VTL.
+
+KVM's lAPIC implementation is aware of groups, and takes note of the
+source vCPU's group when delivering IPIs. As such, it shouldn't be
+possible to target a different VTL through the APIC. Interrupts are
+delivered to the vCPU's lAPIC subsystem regardless of the VTL's runstate,
+this also includes timers. Ultimately, any interrupt incoming from an
+outside source (IOAPIC/MSIs) is routed to VTL0.
+
+Moving Between VTLs
+-------------------
+
+All VSM configuration and VTL handling hypercalls are passed through to
+user-space. Notably the two primitives that allow switching between VTLs.
+All shared state synchronization and KVM vCPU scheduling is left to the
+VMM to manage. For example, upon receiving a VTL call, the VMM stops the
+vCPU that issued the hypercall, and schedules the vCPU corresponding to
+the next privileged VTL. When that privileged vCPU is done executing, it
+issues a VTL return hypercall, so the opposite operation happens. All
+this is transparent to KVM, which limits itself to running vCPUs.
+
+An interrupt directed at a privileged VTL always has precedence over the
+execution of lower VTLs. To honor this, the VMM can monitor events
+targeted at privileged vCPUs with poll(), and should trigger an
+asynchronous VTL switch whenever events become available. Additionally,
+the target VTL's vCPU VP assist overlay page is used to notify the target
+VTL with the reason for the switch. The VMM can keep track of the VP
+assist page by installing an MSR filter for HV_X64_MSR_VP_ASSIST_PAGE.
+
+Hyper-V VP registers
+--------------------
+
+VP register hypercalls are passed through to user-space. All requests can
+be fulfilled either by using already existing KVM state ioctls, or are
+related to VSM's configuration, which is already handled by the VMM. Note
+that HV_REGISTER_VSM_CODE_PAGE_OFFSETS is the only VSM specific VP
+register the kernel controls, as such it is made available through the
+KVM_HV_GET_VSM_STATE ioctl.
+
+Per-VTL Memory Protections
+--------------------------
+
+A privileged VTL can change the memory access restrictions of lower VTLs.
+It does so to hide secrets from them, or to control what they are allowed
+to execute. The list of memory protections allowed is[3]:
+ - No access
+ - Read-only, no execute
+ - Read-only, execute
+ - Read/write, no execute
+ - Read/write, execute
+
+VTL memory protection hypercalls are passed through to user-space, but
+KVM provides an interface that allows changing memory protections on a
+per-VTL basis. This is made possible by the KVM VTL device. VMMs can
+create one per VTL and it exposes a ioctl, KVM_SET_MEMORY_ATTRIBUTES,
+that controls the memory protections applied to that VTL. The KVM TDP MMU
+is VTL aware and page faults are resolved taking into account the
+corresponding VTL device's memory attributes.
+
+When a memory access violates VTL memory protections, KVM issues a secure
+memory intercept, which is passed as a SynIC message into the next
+privileged VTL. This happens transparently for the VMM. Additionally, KVM
+exits with a user-space memory fault. This allows the VMM to stop the
+vCPU while the secure intercept is handled by the privileged VTL. In the
+good case, the instruction that triggered the fault is emulated and
+control is returned to the lower VTL, in the bad case, Windows crashes
+gracefully.
+
+Hyper-V's TLFS also states that DMA should follow VTL0's memory access
+restrictions. This is out of scope for this document, as IOMMU mappings
+are not handled by KVM.
+
+[1] https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/tlfs/Hypervisor%20Top%20Level%20Functional%20Specification%20v6.0b.pdf
+
+[2] Conceptually this design is similar to arm's TrustZone: The
+hypervisor plays the role of EL3. Windows (VTL0) runs in Non-Secure
+(EL0/EL1) and the secure kernel (VTL1) in Secure World (EL1s/EL0s).
+
+[3] TLFS 15.9.3
-- 
2.40.1

Powered by blists - more mailing lists