Message-ID: <20260129212510.967611-12-dmatlack@google.com>
Date: Thu, 29 Jan 2026 21:24:58 +0000
From: David Matlack <dmatlack@...gle.com>
To: Alex Williamson <alex@...zbot.org>
Cc: Adithya Jayachandran <ajayachandra@...dia.com>, Alexander Graf <graf@...zon.com>, 
	Alex Mastro <amastro@...com>, Alistair Popple <apopple@...dia.com>, 
	Andrew Morton <akpm@...ux-foundation.org>, Ankit Agrawal <ankita@...dia.com>, 
	Bjorn Helgaas <bhelgaas@...gle.com>, Chris Li <chrisl@...nel.org>, 
	David Matlack <dmatlack@...gle.com>, David Rientjes <rientjes@...gle.com>, 
	Jacob Pan <jacob.pan@...ux.microsoft.com>, Jason Gunthorpe <jgg@...dia.com>, 
	Jason Gunthorpe <jgg@...pe.ca>, Jonathan Corbet <corbet@....net>, Josh Hilke <jrhilke@...gle.com>, 
	Kevin Tian <kevin.tian@...el.com>, kexec@...ts.infradead.org, kvm@...r.kernel.org, 
	Leon Romanovsky <leon@...nel.org>, Leon Romanovsky <leonro@...dia.com>, linux-doc@...r.kernel.org, 
	linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org, 
	linux-mm@...ck.org, linux-pci@...r.kernel.org, Lukas Wunner <lukas@...ner.de>, 
	"Michał Winiarski" <michal.winiarski@...el.com>, Mike Rapoport <rppt@...nel.org>, 
	Parav Pandit <parav@...dia.com>, Pasha Tatashin <pasha.tatashin@...een.com>, 
	Pranjal Shrivastava <praan@...gle.com>, Pratyush Yadav <pratyush@...nel.org>, 
	Raghavendra Rao Ananta <rananta@...gle.com>, Rodrigo Vivi <rodrigo.vivi@...el.com>, 
	Saeed Mahameed <saeedm@...dia.com>, Samiullah Khawaja <skhawaja@...gle.com>, 
	Shuah Khan <skhan@...uxfoundation.org>, 
	"Thomas Hellström" <thomas.hellstrom@...ux.intel.com>, Tomita Moeko <tomitamoeko@...il.com>, 
	Vipin Sharma <vipinsh@...gle.com>, Vivek Kasireddy <vivek.kasireddy@...el.com>, 
	William Tu <witu@...dia.com>, Yi Liu <yi.l.liu@...el.com>, Zhu Yanjun <yanjun.zhu@...ux.dev>
Subject: [PATCH v2 11/22] docs: liveupdate: Document VFIO device file preservation

Add documentation for preserving VFIO device files across a Live Update,
as well as some generic file preservation documentation. This
documentation will be extended in the future as new types of files are
supported and new dependency/ordering requirements are added.

Signed-off-by: David Matlack <dmatlack@...gle.com>
---
 Documentation/userspace-api/liveupdate.rst | 144 +++++++++++++++++++++
 1 file changed, 144 insertions(+)

diff --git a/Documentation/userspace-api/liveupdate.rst b/Documentation/userspace-api/liveupdate.rst
index 41c0473e4f16..dbf1e4aeddd7 100644
--- a/Documentation/userspace-api/liveupdate.rst
+++ b/Documentation/userspace-api/liveupdate.rst
@@ -14,6 +14,150 @@ ioctl uAPI
 ===========
 .. kernel-doc:: include/uapi/linux/liveupdate.h
 
+File Preservation
+=================
+
+Files can be preserved across Live Update in sessions. Since only one process
+can open /dev/liveupdate, sessions must be created by a centralized process
+(e.g. "luod") and then passed via UDS to lower privilege processes (e.g. VMMs)
+for them to preserve their own files.
+
+luod::
+
+  luo_fd = open("/dev/liveupdate", ...);
+
+  ...
+
+  // Create a new session with the given name.
+  struct liveupdate_ioctl_create_session arg = {
+          .size = sizeof(arg),
+	  .name = SESSION_NAME,
+  };
+  ioctl(luo_fd, LIVEUPDATE_IOCTL_CREATE_SESSION, &arg);
+
+  // Send session_fd to the VMM over UDS.
+  send_session_fd(..., arg.fd);
+
+VMM::
+
+  // Receive the newly created session from luod over UDS
+  session_fd = create_session(SESSION_NAME);
+
+  ...
+
+  // Preserve a file with a unique token value in the session.
+  struct liveupdate_session_preserve_fd arg = {
+          .size = sizeof(arg),
+          .fd = fd,
+          .token = TOKEN,
+  }
+  ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &arg);
+
+Files can be unpreserved with the LIVEUPDATE_SESSION_UNPRESERVE_FD ioctl. They
+are also unpreserved once the last reference to the session is dropped.  To
+carry preserved files across a Live Update, references must be kept on the
+session files through the reboot(LINUX_REBOOT_CMD_KEXEC) syscall.
+
+While a file is preserved in a session, the kernel holds an extra reference
+to it to prevent it from being destroyed.
+
+Only the following types of files support LIVEUPDATE_SESSION_PRESERVE_FD. More
+types of files are expected to be added in the future.
+
+ - memfd
+ - VFIO character device files (vfio-pci only)
+
+File Retrieval
+==============
+
+Files that are preserved in a session retrieved after
+reboot(LINUX_REBOOT_CMD_KEXEC).
+
+luod::
+
+  luo_fd = open("/dev/liveupdate", ...);
+
+  ...
+
+  struct liveupdate_ioctl_retrieve_session arg = {
+          .size = sizeof(arg),
+	  .name = SESSION_NAME,
+  };
+  ioctl(luo_fd, LIVEUPDATE_IOCTL_RETRIEVE_SESSION, &arg);
+
+  // Send session_fd to VMM over UDS.
+  send_session_fd(..., arg.fd);
+
+VMM::
+
+  // Receive the retrieved session from luod over UDS
+  session_fd = retrieve_session(SESSION_NAME);
+
+  ...
+
+  // Retrieve the file associated with the token from the session.
+  struct liveupdate_session_retrieve_fd arg = {
+          .size = sizeof(arg),
+          .token = TOKEN,
+  };
+  ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &arg);
+
+  ...
+
+  ioctl(session_fd, LIVEUPDATE_SESSION_FINISH, ...);
+
+A session can only be finished once all of the files within it have been
+retrieved, and are fully restored from the kernel's perspective. The exact
+requirements will vary by file type.
+
+VFIO Character Device (cdev) Files
+==================================
+
+The kernel supports preserving VFIO character device files across Live Update
+within a session::
+
+  device_fd = open("/dev/vfio/devices/X");
+
+  ...
+
+  ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, { ..., device_fd, ...});
+
+Attempting to preserve files acquired via VFIO_GROUP_GET_DEVICE_FD will fail.
+
+Since the kernel holds an extra reference to files preserved in sessions, there
+is no way for the underlying PCI device to be unbound from vfio-pci while it
+is being preserved.
+
+When a VFIO device file is preserved in a session, interrupts must be disabled
+on the device prior to reboot(LINUX_REBOOT_CMD_KEXEC), or the kexec will fail.
+
+Preserved VFIO device files can be retrieved after a Live Update just like any
+other preserved file::
+
+  ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &arg);
+  device_fd = arg.fd;
+
+  ...
+
+  ioctl(session_fd, LIVEUPDATE_SESSION_FINISH, ...);
+
+Prior to LIVEUPDATE_SESSION_FINISH, preserved devices must be retrieved from
+the session and bound to an iommufd. Attempting to open the device through
+its character device (/dev/vfio/devices/X) or VFIO_GROUP_GET_DEVICE_FD will
+fail with -EBUSY.
+
+The eventual goal of these support is to preserve devices running uninterrupted
+across a Live Update. However there are many steps still needed to achieve this
+(see Future Work below). So for now, VFIO will reset and restore the device
+back into an idle state during reboot(LINUX_REBOOT_CMD_KEXEC).
+
+Future work:
+
+ - Preservation of iommufd files
+ - Preservation of IOMMU driver state
+ - Preservation of PCI state (BAR resources, device state, bridge state, ...)
+ - Preservation of vfio-pci driver state
+
 See Also
 ========
 
-- 
2.53.0.rc1.225.gd81095ad13-goog

Powered by blists - more mailing lists