Message-Id: <20250728-luo-pci-v1-0-955b078dd653@kernel.org>
Date: Mon, 28 Jul 2025 01:24:30 -0700
From: Chris Li <chrisl@...nel.org>
To: Bjorn Helgaas <bhelgaas@...gle.com>, 
 Greg Kroah-Hartman <gregkh@...uxfoundation.org>, 
 "Rafael J. Wysocki" <rafael@...nel.org>, Danilo Krummrich <dakr@...nel.org>, 
 Len Brown <lenb@...nel.org>
Cc: linux-kernel@...r.kernel.org, linux-pci@...r.kernel.org, 
 linux-acpi@...r.kernel.org, David Matlack <dmatlack@...gle.com>, 
 Pasha Tatashin <tatashin@...gle.com>, Jason Miu <jasonmiu@...gle.com>, 
 Vipin Sharma <vipinsh@...gle.com>, Saeed Mahameed <saeedm@...dia.com>, 
 Adithya Jayachandran <ajayachandra@...dia.com>, 
 Parav Pandit <parav@...dia.com>, William Tu <witu@...dia.com>, 
 Mike Rapoport <rppt@...nel.org>, Chris Li <chrisl@...nel.org>, 
 Jason Gunthorpe <jgg@...pe.ca>, Leon Romanovsky <leon@...nel.org>
Subject: [RFC PATCH 00/25] Live Update Orchestrator: PCI subsystem

The LUO PCI subsystem is based on the LUO V2 series.
https://lore.kernel.org/lkml/20250515182322.117840-1-pasha.tatashin@soleen.com/

It registers the PCI as a LUO subsystem and forwards the liveupdate
callback to the device. The struct dev_liveupdate has been add to struct
device to keep track of the liveupdate related context.

A device can be marked as requested for liveupdate during the normal
state.

In the prepare() callback. The PCI core will build a list of the PCI device
for liveupdate based on the PCI device dependency:
1) The VF device is dependent on the PF device for SR-IOV function.
   The PF device needs to restore the number of VF.
2) The requested device is dependent on the PCI bridge it is on to preserve
   the bridge bus master. All the way to the root bridge. If the bus master
   has been disabled on the bridge, the DMA on the children devices will
   get impacted.

The list of liveupdate devices is used for prepare(), cancel(), freeze()
and finalized() callback.

The PCI subsystem will preserve the driver name for each liveupdate PCI
device and only probe that driver after kexec boot up.

It also saves the number of VF for the live updated PF device. The PF
driver will be responsible for restoring the number of VF.

Preserving the PCI device state during kexec boot up will need to change
the device probing logic significantly. After liveupdate kexec, the device
can't just be initialized as a fresh start any more. It needs to adopt the
already initialized state from the previous kernel.

Currently it is using pci_lu_adopt() function to detect if the device is
under liveupdate, then skip the device initialization write if needed.
That part of the code is pretty invasive and spread into many PCI device
initialization code paths. I am open to suggestion how it can be done
cleaner.

After kexec boot up, the PF device will probe before VF. Inside the PF
driver probe(), the PF driver will restore the number of VF and create the
VF device. Then the VF driver's probing will be called.

Disclaimer:
The data preservation format is not final. It currently uses C struct
directly. It does not deal with version change on the data format yet. I
do have some idea how to address the versioning of data layout. Those
will be outside the scope of this series.

Testing:
Testing was done with Intel diorite NVMe VF device 8086:1457. Bind the PF
with pci-lu-stub-pf driver and VF with pci-lu-stub driver.  The VF is mark
mark as requested
[  317.393914] pci-lu-stub 0000:09:00.0: Marking device as live update requested

Now perform luo prepare, the PCI subsystem builds the liveupdate device
list from the PCI root bridge. The PF device and PCI bridge will be mark
depended.
[  330.870750] pci-lu-stub 0000:09:00.0: PCI liveupdate: collect liveupdate device: [requested]
[  330.879214] pci_bus 0000:09: PCI liveupdate: collect liveupdate bus 0000:09
[  330.886219] pci-lu-stub-pf 0000:05:00.1: PCI liveupdate: collect liveupdate device: [depended]
[  330.894845] pci_bus 0000:05: PCI liveupdate: collect liveupdate bus 0000:05
[  330.901829] pcieport 0000:04:01.0: PCI liveupdate: collect liveupdate device: [depended]
[  330.909944] pci_bus 0000:04: PCI liveupdate: collect liveupdate bus 0000:04
[  330.916933] pci-lu-stub 0000:09:00.0: pci_lu_stub_prepare(): data: 0x1eaf1c000
[  330.924174] pci-lu-stub-pf 0000:05:00.1: pci_lu_stub_prepare(): data: 0x1a2abe000
[  330.931678] PCI liveupdate: prepare data[23654a000]
[  330.936587] luo_core: Switched from [normal] to [prepared] state

After kexec reboot. The liveupdate devices are probed and restores the live
update context.
[    3.628261] pci 0000:04:01.0: PCI liveupdate: liveupdate restore [depended] driver: pcieport data: [0] num_vfs: 0
[    4.769292] pci 0000:05:00.1: PCI liveupdate: liveupdate restore [depended] driver: pci-lu-stub-pf data: [1a2abe000] num_vfs: 4
[   16.811848] pci 0000:09:00.0: PCI liveupdate: liveupdate restore [requested] driver: pci-lu-stub data: [1eaf1c000] num_vfs: 0

Perform luo finish to convert from update state to normal state. The
reserved folio will be freed.
[  287.836486] PCI liveupdate: finish data[23654a000]
[  287.841309] pci-lu-stub-pf 0000:05:00.1: pci_lu_stub_finish(): data: 0x1a2abe000
[  287.848733] pci-lu-stub 0000:09:00.0: pci_lu_stub_finish(): data: 0x1eaf1c000
[  287.855897] luo_core: Switched from [updated] to [normal] state

Signed-off-by: Chris Li <chrisl@...nel.org>
---
Chris Li (14):
      PCI/LUO: Register with Liveupdate Orchestrator
      PCI/LUO: Add struct dev_liveupdate
      PCI/LUO: Create requested liveupdate device list
      PCI/LUO: Forward prepare()/freeze()/cancel() callbacks to driver
      PCI/LUO: Restore state at PCI enumeration
      PCI/LUO: Forward finish callbacks to drivers
      PCI/LUO: Save and restore driver name
      PCI/LUO: Add liveupdate to pcieport driver
      PCI/LUO: Save SR-IOV number of VF
      PCI/LUO: Add pci_liveupdate_get_driver_data()
      PCI: pci-lu-stub: Add a stub driver for Live Update testing
      PCI/LUO: Track liveupdate buses
      PCI/LUO: Avoid write to liveupdate devices at boot
      PCI: pci-lu-pf-stub: Add a PF stub driver for Live Update testing

David Matlack (1):
      PCI/LUO: Clean up PCI_SER_GET()

Jason Miu (10):
      PCI/LUO: Save struct pci_dev info during prepare phase
      PCI/LUO: Check the device function numbers in restoration
      PCI/LUO: Restore power state of a PCI device
      PCI/LUO: Restore PM related fields
      PCI/LUO: Restore the pme_poll flag
      PCI/LUO: Restore the no_d3cold flag
      PCI/LUO: Restore pci_dev fields during probe
      PCI/LUO: Save and restore the PCI resource
      PCI/LUO: Save PCI bus and host bridge states
      PCI/LUO: Check the PCI bus state after restoration

 drivers/pci/Kconfig            |  10 +
 drivers/pci/Makefile           |   2 +
 drivers/pci/ats.c              |   7 +-
 drivers/pci/bus.c              |   5 +
 drivers/pci/iov.c              |  58 ++--
 drivers/pci/liveupdate.c       | 707 +++++++++++++++++++++++++++++++++++++++++
 drivers/pci/msi/msi.c          |  32 +-
 drivers/pci/msi/pcidev_msi.c   |   4 +-
 drivers/pci/pci-acpi.c         |   3 +
 drivers/pci/pci-lu-stub.c      | 216 +++++++++++++
 drivers/pci/pci.c              | 105 ++++--
 drivers/pci/pci.h              |  70 ++++
 drivers/pci/pcie/aspm.c        |   7 +-
 drivers/pci/pcie/pme.c         |  11 +-
 drivers/pci/pcie/portdrv.c     |  13 +
 drivers/pci/probe.c            |  92 ++++--
 drivers/pci/setup-bus.c        |  10 +-
 include/linux/dev_liveupdate.h |  64 ++++
 include/linux/device.h         |  15 +
 include/linux/device/driver.h  |   6 +
 include/linux/pci.h            |   9 +
 21 files changed, 1352 insertions(+), 94 deletions(-)
---
base-commit: 57fb5d5e70ca837e0cf3e38c59112cce460b643d
change-id: 20250724-luo-pci-1291890b710f

Best regards,
-- 
Chris Li <chrisl@...nel.org>

Powered by blists - more mailing lists