Message-ID: <20241121201448.36170-1-adrian.hunter@intel.com>
Date: Thu, 21 Nov 2024 22:14:39 +0200
From: Adrian Hunter <adrian.hunter@...el.com>
To: pbonzini@...hat.com,
	seanjc@...gle.com,
	kvm@...r.kernel.org,
	dave.hansen@...ux.intel.com
Cc: rick.p.edgecombe@...el.com,
	kai.huang@...el.com,
	adrian.hunter@...el.com,
	reinette.chatre@...el.com,
	xiaoyao.li@...el.com,
	tony.lindgren@...ux.intel.com,
	binbin.wu@...ux.intel.com,
	dmatlack@...gle.com,
	isaku.yamahata@...el.com,
	nik.borisov@...e.com,
	linux-kernel@...r.kernel.org,
	x86@...nel.org,
	yan.y.zhao@...el.com,
	chao.gao@...el.com,
	weijiang.yang@...el.com
Subject: [PATCH 0/7] KVM: TDX: TD vcpu enter/exit

Hi

This patch series introduces callbacks to facilitate the entry of a TD VCPU
and the corresponding save/restore of host state.

There are some outstanding things still to do (see below), so we expect to
post future revisions of this patch set, but please do review the current
patches so that they can be made ready for hand off to Paolo.  Also
direction is needed for "x86/virt/tdx: Add SEAMCALL wrapper to enter/exit
TDX guest" because it will affect KVM.

This patch set is one of several patch sets that are all needed to provide
the ability to run a functioning TD VM.  They have been split from the
"marker" sections of patch set "[PATCH v19 000/130] KVM TDX basic feature
support":

  https://lore.kernel.org/all/cover.1708933498.git.isaku.yamahata@intel.com/

The recent patch sets are:

  TDX host: metadata reading
  TDX vCPU/VM creation
  TDX KVM MMU part 2
  TD vcpu enter/exit			<- this one
  TD vcpu exits/interrupts/hypercalls   <- still to come

Notably, a later patch sets deal with VCPU exits, interrupts and
hypercalls.

For x86 maintainers

This series has 1 commit that is an RFC that needs input from x86
maintainers:

  x86/virt/tdx: Add SEAMCALL wrapper to enter/exit TDX guest

This is because wrapping TDH.VP.ENTER means dealing with multiple input and
output formats for the data in argument registers. We would like
maintainers to comment on the discussion that we will have on it.

Overview

A TD VCPU is entered via the SEAMCALL TDH.VP.ENTER. The TDX Module manages
the save/restore of guest state and, in conjunction with the SEAMCALL
interface, handles certain aspects of host state. However, there are
specific elements of the host state that require additional attention, as
detailed in the Intel TDX ABI documentation for TDH.VP.ENTER.

TDX is quite different from VMX in this regard.  For VMX, the host VMM is
heavily involved in restoring, managing and saving guest CPU state, whereas
for TDX this is handled by the TDX Module.  In that way, the TDX Module can
protect the confidentiality and integrity of TD CPU state.

The TDX Module does not save/restore all host CPU state because the host
VMM can do it more efficiently and selectively.  CPU state referred to
below is host CPU state.  Often values are already held in memory so no
explicit save is needed, and restoration may not be needed if the kernel
is not using a feature.

Outstanding things still to do:

  - how to wrap TDH.VP.ENTER SEAMCALL, refer to patch "x86/virt/tdx:
    Add SEAMCALL wrapper to enter/exit TDX guest"
  - tdx_vcpu_enter_exit() calls guest_state_enter_irqoff()
    and guest_state_exit_irqoff() which comments say should be
    called from non-instrumentable code but noinst was removed
    at Sean's suggestion:
  	https://lore.kernel.org/all/Zg8tJspL9uBmMZFO@google.com/
    noinstr is also needed to retain NMI-blocking by avoiding
    instrumented code that leads to an IRET which unblocks NMIs.
    A later patch set will deal with NMI VM-exits.
  - disallow TDX guest to use Intel PT
  	I think Tony will fix tdx_get_supported_xfam()
  - disallow PERFMON (TD attribute bit 63)
  - save/restore MSR IA32_UMWAIT_CONTROL or disallow guest
    CPUID(7,0).ECX.WAITPKG[5]
  - save/restore IA32_DEBUGCTL
  	VMX does:
  		vmx_vcpu_load() -> get_debugctlmsr()
  		vmx_vcpu_run() -> update_debugctlmsr()
  	TDX Module only preserves bits 1, 12 and 14

Key Details

Argument Passing: Similar to other SEAMCALLs, TDH.VP.ENTER passes
arguments through General Purpose Registers (GPRs). For the special case
of the TD guest invoking TDG.VP.VMCALL, nearly any GPR can be used,
as well as XMM0 to XMM15. Notably, RBP is not used, and Linux mandates the
TDX Module feature NO_RBP_MOD, which is enforced elsewhere. Additionally,
XMM registers are not required for the existing Guest Hypervisor
Communication Interface and are handled by existing KVM code should they be
modified by the guest.

Debug Register Handling: After TDH.VP.ENTER returns, registers DR0, DR1,
DR2, DR3, DR6, and DR7 are set to their architectural INIT values. Existing
KVM code already handles the restoration of host values as needed, refer
vcpu_enter_guest() which calls hw_breakpoint_restore().

MSR Restoration: Certain Model-Specific Registers (MSRs) need to be
restored post TDH.VP.ENTER. The Intel TDX ABI documentation provides a
detailed list in the msr_preservation.json file. Most MSRs do not require
restoration if the guest is not utilizing the corresponding feature. The
following features are currently assumed to be unsupported, and their MSRs
are not restored:
  PERFMON            (TD ATTRIBUTES[63])
  LBRs               (XFAM[15])
  User Interrupts    (XFAM[14])
  Intel PT           (XFAM[8])
The one feature that is supported:
  CET                (XFAM[11-12])	is restored via kvm_put_guest_fpu()

Other host MSR/Register Handling:

MSR IA32_XFD is already restored by KVM, refer to kvm_put_guest_fpu().
The TDX Module sets MSR IA32_XFD_ERR to its RESET value (0) which is
fine for the kernel.

MSR IA32_DEBUGCTL appears to have been overlooked.  According to
msr_preservation.json, the TDX Module preserves only bits 1, 12 and 14.
For VMX there is code to save and restore in vmx_vcpu_load() and
vmx_vcpu_run() respectively, but TDX does not use those functions.

MSR IA32_UARCH_MISC_CTL is not utilized by the kernel, so it is fine if
the TDX Module sets it to it's RESET value.

MSR IA32_KERNEL_GS_BASE is addressed in patch "KVM: TDX: vcpu_run:
save/restore host state (host kernel gs)".

MSRs IA32_XSS and XCRO are handled in patch "KVM: TDX: restore host xsave
state when exiting from the guest TD".

MSRs IA32_STAR, IA32_LSTAR, IA32_FMASK, and IA32_TSC_AUX are handled in
patch "KVM: TDX: restore user ret MSRs".

MSR IA32_TSX_CTRL is handled in patch "KVM: TDX: Add TSX_CTRL msr into
uret_msrs list".

MSR IA32_UMWAIT_CONTROL appears to have been overlooked.  The host value
needs to be restored if guest CPUID(7,0).ECX.WAITPKG[5] is 1, otherwise
that guest CPUID value needs to be disallowed.

Additional Notes

The patch "KVM: TDX: Implement TDX vcpu enter/exit path" highlights that
TDX does not support "PAUSE-loop exiting".  According to the TDX Module
Base arch. spec., hypercalls are expected to be used instead.  Note that
the Linux TDX guest supports existing hypercalls via TDG.VP.VMCALL.

Base

This series is based off of a kvm-coco-queue commit and some pre-req
series:
1. commit ee69eb746754 ("KVM: x86/mmu: Prevent aliased memslot GFNs") (in
   kvm-coco-queue).
2. v7 of "TDX host: metadata reading tweaks, bug fix and info dump" [1].
3. v1 of "KVM: VMX: Initialize TDX when loading KVM module" [2], with some
   new feedback from Sean.
4. v2 of “TDX vCPU/VM creation” [3]
5. v2 of "TDX KVM MMU part 2" [4]

It requires TDX module 1.5.06.00.0744[5], or later. This is due to removal
of the workarounds for the lack of the NO_RBP_MOD feature required by the
kernel. Now NO_RBP_MOD is enabled (in VM/vCPU creation patches), and this
particular version of the TDX module has a required NO_RBP_MOD related bug
fix.
A working edk2 commit is 95d8a1c ("UnitTestFrameworkPkg: Use TianoCore
mirror of subhook submodule").

Testing

The series has been tested as part of the development branch for the TDX
base series. The testing consisted of TDX kvm-unit-tests and booting a
Linux TD, and TDX enhanced KVM selftests.

The full KVM branch is here:
https://github.com/intel/tdx/tree/tdx_kvm_dev-2024-11-20

Matching QEMU:
https://github.com/intel-staging/qemu-tdx/commits/tdx-qemu-upstream-v6.1/

[0] https://lore.kernel.org/kvm/20240904030751.117579-1-rick.p.edgecombe@intel.com/
[1] https://lore.kernel.org/kvm/cover.1731318868.git.kai.huang@intel.com/#t
[2] https://lore.kernel.org/kvm/cover.1730120881.git.kai.huang@intel.com/
[3] https://lore.kernel.org/kvm/20241030190039.77971-1-rick.p.edgecombe@intel.com/
[4] https://lore.kernel.org/kvm/20241112073327.21979-1-yan.y.zhao@intel.com/
[5] https://github.com/intel/tdx-module/releases/tag/TDX_1.5.06

Chao Gao (1):
      KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o
      wrmsr

Isaku Yamahata (4):
      KVM: TDX: Implement TDX vcpu enter/exit path
      KVM: TDX: vcpu_run: save/restore host state(host kernel gs)
      KVM: TDX: restore host xsave state when exit from the guest TD
      KVM: TDX: restore user ret MSRs

Kai Huang (1):
      x86/virt/tdx: Add SEAMCALL wrapper to enter/exit TDX guest

Yang Weijiang (1):
      KVM: TDX: Add TSX_CTRL msr into uret_msrs list

 arch/x86/include/asm/kvm_host.h |   1 +
 arch/x86/include/asm/tdx.h      |   1 +
 arch/x86/kvm/vmx/main.c         |  45 ++++++++-
 arch/x86/kvm/vmx/tdx.c          | 212 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx.h          |  14 +++
 arch/x86/kvm/vmx/x86_ops.h      |   9 ++
 arch/x86/kvm/x86.c              |  24 ++++-
 arch/x86/virt/vmx/tdx/tdx.c     |   8 ++
 arch/x86/virt/vmx/tdx/tdx.h     |   1 +
 9 files changed, 306 insertions(+), 9 deletions(-)

Regards
Adrian

Powered by blists - more mailing lists