Message-Id: <20240719115741.3694893-1-rick.wertenbroek@gmail.com>
Date: Fri, 19 Jul 2024 13:57:37 +0200
From: Rick Wertenbroek <rick.wertenbroek@...il.com>
To: rick.wertenbroek@...g-vd.ch
Cc: damien.lemoal@...nel.org,
	alberto.dassatti@...g-vd.ch,
	Rick Wertenbroek <rick.wertenbroek@...il.com>,
	Manivannan Sadhasivam <manivannan.sadhasivam@...aro.org>,
	Krzysztof Wilczyński <kw@...ux.com>,
	Kishon Vijay Abraham I <kishon@...nel.org>,
	Bjorn Helgaas <bhelgaas@...gle.com>,
	Niklas Cassel <cassel@...nel.org>,
	Frank Li <Frank.Li@....com>,
	Damien Le Moal <dlemoal@...nel.org>,
	Lars-Peter Clausen <lars@...afoo.de>,
	linux-pci@...r.kernel.org,
	linux-kernel@...r.kernel.org
Subject: [PATCH 0/2] PCI: endpoint: Introduce 'get_bar' to map fixed address BARs in EPC

Hello,

I have been exploring the possibility to add FPGA based PCI endpoint controller
support within the current PCI Endpoint Framework [1]. The FPGA IP based
controllers are mostly similar to the ones found in SoCs with one major
difference. As they reside in the FPGA fabric and do not have direct access to
the SoC main memory space they usually come with pre-assigned BAR mappings
within their own memory space (memory space that can be accessed by the SoC CPU
or softcore within the FPGA running Linux, however the physical addresses may be
different). Therefore it is not possible to allocate memory from the CPU running
linux with 'pci_epf_alloc_space' and set the endpoint controller to reroute the
PCI read/write TLPs to that address with 'pci_epc_set_bar'.

Therefore, I suggest to introduce 'pci_epc_get_bar' and the 'fixed_addr'
endpoint controller bar feature to indicate that the BAR resides at a given
address (usually set during the FPGA IP configuration before synthesis). As for
providing this information from the FPGA IDE to Linux, the device tree can be
used. This choice is at the discretion of the specific endpoint controller driver.

The endpoint functions can query the endpoint controller driver on their BAR
features, as is already the case with the features 'fixed_size' or 'only_64bit'.
If the BAR is 'fixed_addr' (new feature) they call 'pci_epc_get_bar' instead of
'pci_epf_alloc_space' followed by 'pci_epc_set_bar'. 'pci_epc_get_bar' will
fill the 'pci_epf_bar' structure with physical address and do an ioremap() to
provide the 'pci_epf_bar.addr' virtual address.

This change it would allow writing PCI endpoint controller drivers for
Intel/Altera and AMD/Xilinx PCI endpoint IPs.

The patches are for linux-next tag : next-20240719

I have successfully demonstrated that it is possible to write such a driver
thanks to the new 'pci_epc_get_bar' function for the AMD/Xilinx PCI endpoint IP
for ZynqUS+ devices [2,3]. And have successfully run the PCI Test Function [4]
as well as NVMe endpoint functions I am working on. (Note: the function is called
'pci_epc_get_fixed_bar' in that example code and drivers).

The new 'pci_epc_get_bar' function would also allow to write virtual / emulated
PCI endpoint controllers that run on the host machine where both the PCI
endpoint function and the driver for the PCI device on the virtual bus run on
the same machine. For now this is not possible because the BARs backing memory
allocated with 'pci_epf_alloc_space' are in RAM (non reserved part of RAM) and
when a PCI driver will need to be loaded e.g., '/drivers/misc/pci_endpoint_test'
for the PCI Test Function [4]. It will fail to request the regions for the BARs
in '__pci_request_region' (drivers/pci/pci.c) because the memory space allocated
is in RAM and will cause a conflict. Normally this problem does not appear
because the endpoint function driver and the PCI driver are not running
on the same machine (the function driver is running in the endpoint, e.g., an
SoC connected to the host PC via PCIe and the PCI driver is running on the host
PC). With the new 'pci_epc_get_bar' function one could mark part of the RAM as
reserved memory and use it for the BARs, and then because it is reserved memory
when loading the PCI driver, there will be no conflict in '__pci_request_region'.
This would allow to run the PCI endpoint functions on the host. This is useful
for research and development of new functions and very similar to the NVMeVirt
project [5], where they emulate an NVMe PCI device on the host PC, in RAM. With
the proposed changes I could introduce a "virtual PCI endpoint controller" driver
that would allow to run the endpoint functions available in the Linux PCI
Endpoint Framework [1]. This would allow for a much lighter setup to develop new
functions without the need of an SoC board for most of the development.


The suggested change does not break any compatibility with the existing endpoint
controllers drivers or endpoint function drivers and could be introduced without
further changes. I provided the changes in the 'pci-epf-test' driver to show how
endpoint function drivers would take advantage of this new function to be able to
become compatible with endpoint controller drivers with fixed address BARs.

Cheers.
Rick

[1] https://www.kernel.org/doc/html/latest/PCI/endpoint/index.html
[2] https://github.com/rick-heig/nvme_csd/tree/main/platforms/zcu106
[3] driver (Linux 6.5) https://github.com/rick-heig/linux-xlnx/blob/csd_20231212/drivers/pci/controller/pcie-xilinx-ep.c
[4] https://www.kernel.org/doc/html/latest/PCI/endpoint/pci-test-function.html
[5] https://github.com/snu-csl/nvmevirt

Rick Wertenbroek (2):
  PCI: endpoint: Introduce 'get_bar' to map fixed address BARs in EPC
  PCI: endpoint: pci-epf-test: Add support for controller with fixed
    addr BARs

 drivers/pci/endpoint/functions/pci-epf-test.c | 38 +++++++++++++++++--
 drivers/pci/endpoint/pci-epc-core.c           | 37 ++++++++++++++++++
 include/linux/pci-epc.h                       |  7 ++++
 3 files changed, 78 insertions(+), 4 deletions(-)

-- 
2.25.1

Powered by blists - more mailing lists