| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20210309083357.65467-1-mgurtovoy@nvidia.com>
Date: Tue, 9 Mar 2021 08:33:48 +0000
From: Max Gurtovoy <mgurtovoy@...dia.com>
To: <jgg@...dia.com>, <alex.williamson@...hat.com>,
<cohuck@...hat.com>, <kvm@...r.kernel.org>,
<linux-kernel@...r.kernel.org>
CC: <liranl@...dia.com>, <oren@...dia.com>, <tzahio@...dia.com>,
<leonro@...dia.com>, <yarong@...dia.com>, <aviadye@...dia.com>,
<shahafs@...dia.com>, <artemp@...dia.com>, <kwankhede@...dia.com>,
<ACurrid@...dia.com>, <cjia@...dia.com>, <yishaih@...dia.com>,
<mjrosato@...ux.ibm.com>, <aik@...abs.ru>, <hch@....de>,
Max Gurtovoy <mgurtovoy@...dia.com>
Subject: [PATCH v3 0/9] Introduce vfio-pci-core subsystem
Hi Alex and Cornelia,
This series split the vfio_pci driver into 2 parts: pci drivers and a
subsystem driver that will also be library of code. The main pci driver,
vfio_pci.ko will be used as before and it will bind to the subsystem
driver vfio_pci_core.ko to register to the VFIO subsystem.
New vendor vfio pci drivers were introduced in this series are:
- igd_vfio_pci.ko
- nvlink2gpu_vfio_pci.ko
- npu2_vfio_pci.ko
These drivers also will bind to the subsystem driver vfio_pci_core.ko to
register to the VFIO subsystem. These drivers will also add vendor
specific extensions that are relevant only for the devices that will be
driven by them. This is a typical Linux subsystem framework behaviour.
This series is coming to solve some of the issues that were raised in the
previous attempt for extending vfio-pci for vendor specific
functionality: https://lkml.org/lkml/2020/5/17/376 by Yan Zhao.
This solution is also deterministic in a sense that when a user will
bind to a vendor specific vfio-pci driver, it will get all the special
goodies of the HW. Non-common code will be pushed only to the specific
vfio_pci driver.
This subsystem framework will also ease on adding new vendor specific
functionality to VFIO devices in the future by allowing another module
to provide the pci_driver that can setup number of details before
registering to VFIO subsystem (such as inject its own operations).
Below we can see the proposed changes:
+-------------------------------------------------------------------+
| |
| VFIO |
| |
+-------------------------------------------------------------------+
+-------------------------------------------------------------------+
| |
| VFIO_PCI_CORE |
| |
+-------------------------------------------------------------------+
+--------------+ +------------+ +-------------+ +-------------------+
| | | | | | | |
| | | | | | | |
| VFIO_PCI | |IGD_VFIO_PCI| |NPU2_VFIO_PCI| |NVLINK2GPU_VFIO_PCI|
| | | | | | | |
| | | | | | | |
+--------------+ +------------+ +-------------+ +-------------------+
Patches (1/9) - (4/9) introduce the above changes for vfio_pci and
vfio_pci_core.
Patches (6/9) and (7/9) are a preparation for adding nvlink2 related vfio
pci drivers.
Patch (8/9) introduce new npu2_vfio_pci and nvlink2gpu_vfio_pci drivers
that will drive NVLINK2 capable devices exist today (IBMs emulated PCI
managment device and NVIDIAs NVLINK2 capable GPUs). These drivers add
vendor specific functionality that is related to the IBM NPU2 unit which
is an NVLink2 host bus adapter and to capable NVIDIA GPUs.
Patch (9/9) introduce new igd_vfio_pci driver for adding special extensions
for INTELs Graphics card (GVT-d).
All new 3 vendor specific vfio_pci drivers can be extended easily and
new vendor specific functionality might be added to the needed vendor
drivers. In case there is a vendor specific vfio_pci driver for the
device that a user would like to use, this driver should be bounded to
that device. Otherwise, vfio_pci.ko should be bounded to it (backward
compatability is also supported).
This framework will allow adding more HW specific features such as Live
Migration in the future by extending existing vendor vfio_pci drivers or
creating new ones (e.g. mlx5_vfio_pci.ko that will drive live migration
for mlx5 PCI devices and mlx5_snap_vfio_pci.ko that will drive live
migration for mlx5_snap devices such as mlx5 NVMe and Virtio-BLK SNAP
devices).
Testing:
1. vfio_pci.ko was tested by using 2 VirtIO-BLK physical functions based
on NVIDIAs Bluefield-2 SNAP technology. These 2 PCI functions were
passed to a QEMU based VM after binding to vfio_pci.ko and basic FIO
read/write was issued in the VM to each exposed block device (vda, vdb).
2. igd_vfio_pci.ko was only compiled and loaded/unloaded successfully on x86 server.
3. npu2_vfio_pci.ko and nvlink2gpu_vfio_pci.KO were successfully
compiled and loaded/unloaded on P9 server + vfio probe/remove of devices (without
QEMU/VM).
Note: After this series will be approved, a new discovery/matching mechanism
should be introduced in order to help users to decide which driver should
be bounded to which device.
changes from v2:
- Use container_of approach between core and vfio_pci drivers.
- Comment on private/public attributes for the core structure to ease on
code maintanance.
- rename core structure to vfio_pci_core_device.
- create 3 new vendor drivers (igd_vfio_pci, npu2_vfio_pci,
nvlink2gpu_vfio_pci) and preserve backward compatibility.
- rebase on top of Linux 5.11 tag.
- remove patches that were accepted as stand-alone.
- removed mlx5_vfio_pci driver from this series (3 vendor drivers will be enough
for emphasizing the approch).
changes from v1:
- Create a private and public vfio-pci structures (From Alex)
- register to vfio core directly from vfio-pci-core (for now, expose
minimal public funcionality to vfio pci drivers. This will remove the
implicit behaviour from v1. More power to the drivers can be added in
the future)
- Add patch 3/9 to emphasize the needed extension for LM feature (From
Cornelia)
- take/release refcount for the pci module during core open/release
- update nvlink, igd and zdev to PowerNV, X86 and s390 extensions for
vfio-pci core
- fix segfault bugs in current vfio-pci zdev code
Max Gurtovoy (9):
vfio-pci: rename vfio_pci.c to vfio_pci_core.c
vfio-pci: rename vfio_pci_private.h to vfio_pci_core.h
vfio-pci: rename vfio_pci_device to vfio_pci_core_device
vfio-pci: introduce vfio_pci_core subsystem driver
vfio/pci: introduce vfio_pci_device structure
vfio-pci-core: export vfio_pci_register_dev_region function
vfio/pci_core: split nvlink2 to nvlink2gpu and npu2
vfio/pci: export nvlink2 support into vendor vfio_pci drivers
vfio/pci: export igd support into vendor vfio_pci driver
drivers/vfio/pci/Kconfig | 53 +-
drivers/vfio/pci/Makefile | 20 +-
.../pci/{vfio_pci_igd.c => igd_vfio_pci.c} | 159 +-
drivers/vfio/pci/igd_vfio_pci.h | 24 +
drivers/vfio/pci/npu2_trace.h | 50 +
drivers/vfio/pci/npu2_vfio_pci.c | 364 +++
drivers/vfio/pci/npu2_vfio_pci.h | 24 +
.../vfio/pci/{trace.h => nvlink2gpu_trace.h} | 27 +-
...io_pci_nvlink2.c => nvlink2gpu_vfio_pci.c} | 296 +-
drivers/vfio/pci/nvlink2gpu_vfio_pci.h | 24 +
drivers/vfio/pci/vfio_pci.c | 2433 ++---------------
drivers/vfio/pci/vfio_pci_config.c | 70 +-
drivers/vfio/pci/vfio_pci_core.c | 2245 +++++++++++++++
drivers/vfio/pci/vfio_pci_core.h | 242 ++
drivers/vfio/pci/vfio_pci_intrs.c | 42 +-
drivers/vfio/pci/vfio_pci_private.h | 228 --
drivers/vfio/pci/vfio_pci_rdwr.c | 18 +-
drivers/vfio/pci/vfio_pci_zdev.c | 12 +-
18 files changed, 3528 insertions(+), 2803 deletions(-)
rename drivers/vfio/pci/{vfio_pci_igd.c => igd_vfio_pci.c} (58%)
create mode 100644 drivers/vfio/pci/igd_vfio_pci.h
create mode 100644 drivers/vfio/pci/npu2_trace.h
create mode 100644 drivers/vfio/pci/npu2_vfio_pci.c
create mode 100644 drivers/vfio/pci/npu2_vfio_pci.h
rename drivers/vfio/pci/{trace.h => nvlink2gpu_trace.h} (72%)
rename drivers/vfio/pci/{vfio_pci_nvlink2.c => nvlink2gpu_vfio_pci.c} (57%)
create mode 100644 drivers/vfio/pci/nvlink2gpu_vfio_pci.h
create mode 100644 drivers/vfio/pci/vfio_pci_core.c
create mode 100644 drivers/vfio/pci/vfio_pci_core.h
delete mode 100644 drivers/vfio/pci/vfio_pci_private.h
--
2.25.4
Powered by blists - more mailing lists