| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20260118135440.1958279-35-den@valinux.co.jp> Date: Sun, 18 Jan 2026 22:54:36 +0900 From: Koichiro Den <den@...inux.co.jp> To: Frank.Li@....com, dave.jiang@...el.com, cassel@...nel.org, mani@...nel.org, kwilczynski@...nel.org, kishon@...nel.org, bhelgaas@...gle.com, geert+renesas@...der.be, robh@...nel.org, vkoul@...nel.org, jdmason@...zu.us, allenbh@...il.com, jingoohan1@...il.com, lpieralisi@...nel.org Cc: linux-pci@...r.kernel.org, linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org, linux-renesas-soc@...r.kernel.org, devicetree@...r.kernel.org, dmaengine@...r.kernel.org, iommu@...ts.linux.dev, ntb@...ts.linux.dev, netdev@...r.kernel.org, linux-kselftest@...r.kernel.org, arnd@...db.de, gregkh@...uxfoundation.org, joro@...tes.org, will@...nel.org, robin.murphy@....com, magnus.damm@...il.com, krzk+dt@...nel.org, conor+dt@...nel.org, corbet@....net, skhan@...uxfoundation.org, andriy.shevchenko@...ux.intel.com, jbrunet@...libre.com, utkarsh02t@...il.com Subject: [RFC PATCH v4 34/38] Documentation: driver-api: ntb: Document remote embedded-DMA transport The NTB transport code is split into a common library (ntb_transport_core) and NTB client modules. Document the two transport variants: - ntb_transport: legacy shared-memory rings (CPU/local DMA memcpy) - ntb_transport_edma: remote embedded-DMA data plane Also describe how to select the desired driver (module load order or driver_override binding) and add data-flow diagrams for both directions. Signed-off-by: Koichiro Den <den@...inux.co.jp> --- Documentation/driver-api/ntb.rst | 193 +++++++++++++++++++++++++++++++ 1 file changed, 193 insertions(+) diff --git a/Documentation/driver-api/ntb.rst b/Documentation/driver-api/ntb.rst index a49c41383779..75f96726c373 100644 --- a/Documentation/driver-api/ntb.rst +++ b/Documentation/driver-api/ntb.rst @@ -132,6 +132,199 @@ Transport queue pair. Network data is copied between socket buffers and the Transport queue pair buffer. The Transport client may be used for other things besides Netdev, however no other applications have yet been written. +Transport variants +~~~~~~~~~~~~~~~~~~ + +The ``ntb_transport`` module is a thin NTB client driver. Most of its +functionality is implemented in the ``ntb_transport_core`` library module, +which provides a "queue pair" abstraction to transport clients such as +``ntb_netdev``. Another transport variant, ``ntb_transport_edma``, relies +on an endpoint embedded DMA engine for the data plane. When +``ntb_transport_edma`` is loaded before ``ntb_transport``, or when an NTB +device is explicitly bound to ``ntb_transport_edma`` via sysfs, it will be +selected. Only one transport driver can bind to a given NTB device, and the +upper layer does not need to care which variant is active:: + + +--------------------+ + | ntb_transport_core | + +--------------------+ + ^ ^ + | | + ntb_transport -----+ +----- ntb_transport_edma + (cpu/dma memcpy) (remote embedded DMA transfer) + | + v + +-----------+ + | ntb_edma | + +-----------+ + ^ + | + +----------------+ + | | + ntb_dw_edma [...] + + +Legacy shared-memory backend (``ntb_transport``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default backend uses the NTB memory windows as the data plane. For a TX, +the payload is copied into a window-backed ring buffer and the receiver copies +it back out. Copying is performed by the CPU or by a local DMA engine when the +``use_dma`` module parameter is set. + +This mode is widely applicable but is sensitive to memory window size, as +one descriptor can hold the entire MTU-sized packet data. It also requires +one extra memcpy on both ends, as opposed to the Remote embedded DMA +backend, described below. + +Remote embedded DMA backend (``ntb_transport_edma``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The remote embedded DMA backend moves payload data directly between the two +systems' memories, and uses the NTB memory windows only for control +structures and for exposing the endpoint PCI embedded DMA engine to the +host. It is provided by the ``ntb_transport_edma`` module. + +The current implementation supports Synopsys DesignWare PCIe embedded DMA +(eDMA) via the ``ntb_dw_edma``, which is the only remote embedded DMA +backend option at the moment, and the ``dw-edma`` DMA-engine driver +(``drivers/dma/dw-edma``). The transport is not inherently tied to +DesignWare: additional vendor-specific backends can be added by registering +an ``ntb_edma_backend`` implementation (See ``[...]`` in the above figure.) + +At a high level: + +* One memory window is reserved as the "eDMA window". The endpoint maps its DMA + register block and linked-list descriptor memory into that window so the + host can ioremap it. + +* The remaining memory windows contain small per-QP control rings used to + exchange receive-buffer addresses and completion information. + +* For RC->EP traffic the RC controls the endpoint DMA read channels through the + eDMA window and the DMA engine pulls from RC memory into an EP RX buffer. + +* For EP->RC traffic the endpoint uses its local DMA write channels to push into + an RC RX buffer. + +Because the data plane no longer uses window-backed payload rings, this mode +scales better when window space is scarce (for example, when using many queue +pairs). + +The following figures illustrate the data flow when ``ntb_netdev`` sits on top +of the transport: + +:: + + Figure 1. RC->EP traffic via ntb_netdev + ntb_transport_edma + backed by ntb_dw_edma + + EP RC + phys addr phys addr + space space + +-+ +-+ + | | | | + | | || | | + +-+-----. || | | + EDMA REG | | \\ [A] || | | + +-+----. '---+-+ || | | + | | \\ | |<---------[0-a]---------- + +-+-----------| |<----------[2]----------. + EDMA LL | | | | || | | : + | | | | || | | : + +-+-----------+-+ || [B] | | : + | | || ++ | | : + ---------[0-b]----------->||----------------' + | | ++ || || | | + | | || || ++ | | + | | ||<----------[4]----------- + | | ++ || | | + | | [C] || | | + .--|#|<------------------------[3]------|#|<-. + : |#| || |#| : + [5] | | || | | [1] + : | | || | | : + '->|#| |#|--' + |#| |#| + | | | | + + Figure 2. EP->RC traffic via ntb_netdev + ntb_transport_edma + backed by ntb_dw_edma + + EP RC + phys addr phys addr + space space + +-+ +-+ + | | | | + | | || | | + +-+ || | | + EDMA REG | | || | | + +-+ || | | + ^ | | || | | + : +-+ || | | + : EDMA LL| | || | | + : | | || | | + : +-+ || [C] | | + : | | || ++ | | + : -----------[4]----------->|| | | + : | | ++ || || | | + : | | || || ++ | | + '----------------[2]-----||<--------[0-b]----------- + | | ++ || | | + | | [B] || | | + .->|#|--------[3]---------------------->|#|--. + : |#| || |#| : + [1] | | || | | [5] + : | | || | | : + '--|#| |#|<-' + |#| |#| + | | | | + + 0-a. configure remote embedded DMA (e.g. program endpoint DMA registers) + 0-b. DMA-map and publish destination address (DAR) + 1. network stack builds skb (copy from application/user memory) + 2. consume DAR, DMA-map source address (SAR) and start the DMA transfer + 3. DMA transfer (payload moves between RC/EP memory) + 4. consume completion (commit) + 5. network stack delivers data to application/user memory + + [A]: Dedicated MW that aggregates DMA regs and LL (peer ioremaps it) + [B]: Control-plane ring buffer for "produce" + [C]: Control-plane ring buffer for "consume" + +Enabling the remote embedded DMA transport requires: + +* ``CONFIG_NTB_TRANSPORT`` and ``CONFIG_NTB_TRANSPORT_EDMA``, + +* a matching embedded-DMA backend enabled and/or loaded (e.g. + ``CONFIG_NTB_DW_EDMA``), + +* an endpoint configuration exposing an extra Memory Window, which, + in the ``ntb_dw_edma`` case, exposes eDMA registers and LL region. That + means at least the two Memory Windows (MW1 and MW2) need to be present. + +* loading ``ntb_transport_edma`` (instead of ``ntb_transport``) on both sides, + or explicitly bind to ``ntb_transport_edma`` when both are loaded. See + the following:: + + dev=<ntb device> # pick one from: /sys/bus/ntb/devices/ + + # switch from ntb_transport -> ntb_transport_edma + echo $dev > /sys/bus/ntb/drivers/ntb_transport/unbind + echo ntb_transport_edma > /sys/bus/ntb/devices/$dev/driver_override + echo $dev > /sys/bus/ntb/drivers/ntb_transport_edma/bind + + # switch back (optional) + echo $dev > /sys/bus/ntb/drivers/ntb_transport_edma/unbind + echo ntb_transport > /sys/bus/ntb/devices/$dev/driver_override + echo $dev > /sys/bus/ntb/drivers/ntb_transport/bind + +The remote embedded DMA mode uses a different memory window layout from the +legacy shared-memory transport. There is no automatic fallback at runtime: +if the endpoint does not expose a compatible eDMA window, +``ntb_transport_edma`` will fail to attach. In that case, users need to +manually switch back to ``ntb_transport``. + NTB Ping Pong Test Client (ntb\_pingpong) ----------------------------------------- -- 2.51.0
Powered by blists - more mailing lists