Message-ID: <71518468-e877-4b78-ac22-bb50e40915f3@intel.com>
Date: Mon, 12 Jan 2026 08:43:28 -0700
From: Dave Jiang <dave.jiang@...el.com>
To: Koichiro Den <den@...inux.co.jp>
Cc: Frank.Li@....com, ntb@...ts.linux.dev, linux-pci@...r.kernel.org,
 dmaengine@...r.kernel.org, linux-renesas-soc@...r.kernel.org,
 netdev@...r.kernel.org, linux-kernel@...r.kernel.org, mani@...nel.org,
 kwilczynski@...nel.org, kishon@...nel.org, bhelgaas@...gle.com,
 corbet@....net, geert+renesas@...der.be, magnus.damm@...il.com,
 robh@...nel.org, krzk+dt@...nel.org, conor+dt@...nel.org, vkoul@...nel.org,
 joro@...tes.org, will@...nel.org, robin.murphy@....com, jdmason@...zu.us,
 allenbh@...il.com, andrew+netdev@...n.ch, davem@...emloft.net,
 edumazet@...gle.com, kuba@...nel.org, pabeni@...hat.com,
 Basavaraj.Natikar@....com, Shyam-sundar.S-k@....com,
 kurt.schwemmer@...rosemi.com, logang@...tatee.com, jingoohan1@...il.com,
 lpieralisi@...nel.org, utkarsh02t@...il.com, jbrunet@...libre.com,
 dlemoal@...nel.org, arnd@...db.de, elfring@...rs.sourceforge.net
Subject: Re: [RFC PATCH v3 26/35] NTB: ntb_transport: Introduce DW eDMA backed
 transport mode



On 1/10/26 6:43 AM, Koichiro Den wrote:
> On Thu, Jan 08, 2026 at 10:55:46AM -0700, Dave Jiang wrote:
>>
>>
>> On 1/7/26 6:25 PM, Koichiro Den wrote:
>>> On Wed, Jan 07, 2026 at 12:02:15PM -0700, Dave Jiang wrote:
>>>>
>>>>
>>>> On 1/7/26 7:54 AM, Koichiro Den wrote:
>>>>> On Tue, Jan 06, 2026 at 11:51:03AM -0700, Dave Jiang wrote:
>>>>>>
>>>>>>
>>>>>> On 12/17/25 8:16 AM, Koichiro Den wrote:
>>>>>>> Add a new ntb_transport backend that uses a DesignWare eDMA engine
>>>>>>> located on the endpoint, to be driven by both host and endpoint.
>>>>>>>
>>>>>>> The endpoint exposes a dedicated memory window which contains the eDMA
>>>>>>> register block, a small control structure (struct ntb_edma_info) and
>>>>>>> per-channel linked-list (LL) rings for read channels. Endpoint drives
>>>>>>> its local eDMA write channels for its transmission, while host side
>>>>>>> uses the remote eDMA read channels for its transmission.
>>>>>>>
>>>>>>> A key benefit of this backend is that the memory window no longer needs
>>>>>>> to carry data-plane payload. This makes the design less sensitive to
>>>>>>> limited memory window space and allows scaling to multiple queue pairs.
>>>>>>> The memory window layout is specific to the eDMA-backed backend, so
>>>>>>> there is no automatic fallback to the memcpy-based default transport
>>>>>>> that requires the different layout.
>>>>>>>
>>>>>>> Signed-off-by: Koichiro Den <den@...inux.co.jp>
>>>>>>> ---
>>>>>>>  drivers/ntb/Kconfig                  |  12 +
>>>>>>>  drivers/ntb/Makefile                 |   2 +
>>>>>>>  drivers/ntb/ntb_transport_core.c     |  15 +-
>>>>>>>  drivers/ntb/ntb_transport_edma.c     | 987 +++++++++++++++++++++++++++
>>>>>>>  drivers/ntb/ntb_transport_internal.h |  15 +
>>>>>>>  5 files changed, 1029 insertions(+), 2 deletions(-)
>>>>>>>  create mode 100644 drivers/ntb/ntb_transport_edma.c
>>>>>>>
>>>>>>> diff --git a/drivers/ntb/Kconfig b/drivers/ntb/Kconfig
>>>>>>> index df16c755b4da..5ba6d0b7f5ba 100644
>>>>>>> --- a/drivers/ntb/Kconfig
>>>>>>> +++ b/drivers/ntb/Kconfig
>>>>>>> @@ -37,4 +37,16 @@ config NTB_TRANSPORT
>>>>>>>  
>>>>>>>  	 If unsure, say N.
>>>>>>>  
>>>>>>> +config NTB_TRANSPORT_EDMA
>>>>>>> +	bool "NTB Transport backed by remote eDMA"
>>>>>>> +	depends on NTB_TRANSPORT
>>>>>>> +	depends on PCI
>>>>>>> +	select DMA_ENGINE
>>>>>>> +	select NTB_EDMA
>>>>>>> +	help
>>>>>>> +	  Enable a transport backend that uses a remote DesignWare eDMA engine
>>>>>>> +	  exposed through a dedicated NTB memory window. The host uses the
>>>>>>> +	  endpoint's eDMA engine to move data in both directions.
>>>>>>> +	  Say Y here if you intend to use the 'use_remote_edma' module parameter.
>>>>>>> +
>>>>>>>  endif # NTB
>>>>>>> diff --git a/drivers/ntb/Makefile b/drivers/ntb/Makefile
>>>>>>> index 9b66e5fafbc0..b9086b32ecde 100644
>>>>>>> --- a/drivers/ntb/Makefile
>>>>>>> +++ b/drivers/ntb/Makefile
>>>>>>> @@ -6,3 +6,5 @@ ntb-y			:= core.o
>>>>>>>  ntb-$(CONFIG_NTB_MSI)	+= msi.o
>>>>>>>  
>>>>>>>  ntb_transport-y		:= ntb_transport_core.o
>>>>>>> +ntb_transport-$(CONFIG_NTB_TRANSPORT_EDMA)	+= ntb_transport_edma.o
>>>>>>> +ntb_transport-$(CONFIG_NTB_TRANSPORT_EDMA)	+= hw/edma/ntb_hw_edma.o
>>>>>>> diff --git a/drivers/ntb/ntb_transport_core.c b/drivers/ntb/ntb_transport_core.c
>>>>>>> index 40c2548f5930..bd21232f26fe 100644
>>>>>>> --- a/drivers/ntb/ntb_transport_core.c
>>>>>>> +++ b/drivers/ntb/ntb_transport_core.c
>>>>>>> @@ -104,6 +104,12 @@ module_param(use_msi, bool, 0644);
>>>>>>>  MODULE_PARM_DESC(use_msi, "Use MSI interrupts instead of doorbells");
>>>>>>>  #endif
>>>>>>>  
>>>>>>> +bool use_remote_edma;
>>>>>>> +#ifdef CONFIG_NTB_TRANSPORT_EDMA
>>>>>>> +module_param(use_remote_edma, bool, 0644);
>>>>>>> +MODULE_PARM_DESC(use_remote_edma, "Use remote eDMA mode (when enabled, use_msi is ignored)");
>>>>>>> +#endif
>>>>>>
>>>>>> This seems clunky. Can the ntb_transport_core determine this when the things are called through ntb_transport_edma? Or maybe a set_transport_type can be introduced by the transport itself during allocation?
>>>>>
>>>>> Agreed. I plan to drop 'use_remote_edma' and instead,
>>>>> - add a module parameter: transport_type={"default","edma"} (defaulting to "default"),
>>>>> - introduce ntb_transport_backend_register() for transports to self-register via
>>>>>   struct ntb_transport_backend { .name, .ops }, and
>>>>> - have the core select the backend whose .name matches transport_type.
>>>>>
>>>>> I think this should keep any non-default transport-specific logic out of
>>>>> ntb_transport_core, or at least keep it to a minimum, while still allowing
>>>>> non-defualt transports (*ntb_transport_edma is the only choice for now
>>>>> though) to plug in cleanly.
>>>>>
>>>>> If you see a cleaner approach, I would appreciate it if you could elaborate
>>>>> a bit more on your idea.
>>>>
>>>
>>> Thank you for the comment, let me respond inline below.
>>>
>>>> Do you think it's flexible enough that we can determine a transport type per 'ntb_transport_mw' or is this an all or nothing type of thing?
>>>
>>> At least in the current implementation, the remote eDMA use is an
>>> all-or-nothing type rather than something that can be selected per
>>> ntb_transport_mw.
>>>
>>> The way remote eDMA consumes MW is quite similar to how ntb_msi uses them
>>> today. Assuming multiple MWs are available, the last MW is reserved to
>>> expose the remote eDMA info/register/LL regions to the host by packing all
>>> of them into a single MW. In that sense, it does not map naturally to a
>>> per-MW selection model.
>>>
>>>> I'm trying to see if we can do away with the module param.
>>>
>>> I think it is useful to keep an explicit way for an administrator to choose
>>> the transport type (default vs edma). Even on platforms where dw-edma is
>>> available, there can potentially be platform-specific or hard-to-reproduce
>>> issues (e.g. problems that only show up with certain transfer patterns),
>>> and having a way to fall back the long-existing traditional transport can
>>> be valuable.
>>>
>>> That said, I am not opposed to making the default behavior an automatic
>>> selection, where edma is chosen when it's available and the parameter is
>>> left unset.
>>>
>>>> Or I guess when you probe ntb_netdev, the selection would happen there and thus transport_type would be in ntb_netdev module?
>>>
>>> I'm not sure how selecting the transport type at ntb_netdev probe time
>>> would work in practice, and what additional benefit that would provide.
>>
>> So currently ntb_netdev or ntb_transport are not auto-loaded right? They are manually probed by the user. So with the new transport, the user would modprobe ntb_transport_edma.ko. And that would trigger the eDMA transport setup right? With the ntb_transport_core library existing, we should be able to load both the ntb_transport_host and ntb_transport_edma at the same time theoretically. And ntb_netdev should be able to select one or the other transport. This is the most versatile scenario. An alternative is there can be only 1 transport ever loaded, and when ntb_transport_edma is loaded, it just looks like the default transport and netdev functions as it always has without knowing what the underneath transport is. On the platform if there are multiple NTB ports, it would be nice to have the flexibility of allowing each port choose the usage of the current transport and the edma transport if the user desires.
> 
> I was assuming manual load in my previous response. Also in this RFC v3,
> ntb_transport_edma is not even a standalone module yet (although I do think
> it should be). At this point, I feel the RFC v3 implementation is still a
> bit too rough to use as a basis for discussing the ideal long-term design,
> so I'd like to set it aside for a moment and focus on what the ideal shape
> could look like.
> 
> My current thoughts on the ideal structure, after reading your last
> comment, are as follows:
> 
> * The existing cpu/dma memcpy-based transport becomes "ntb_transport_host",
>   and the new eDMA-based transport becomes "ntb_transport_edma".
> * Each transport is a separate kernel module, and each provides its own
>   ntb_client implementation (i.e. each registers independently with the
>   NTB core). In this model, it should be perfectly fine for both modules to
>   be loaded at the same time.
> * Common pieces (e.g. ntb_transport_bus registration, shared helpers, and
>   the boundary/API exposed to ntb_transport_clients such as ntb_netdev)
>   should live in a shared library module, such as "ntb_transport_core" (or
>   "ntb_transport", naming TBD).
> 
> Then, for transport type selection:
> 
> * If we want to switch the transport type (host vs edma) on a per-NTB-port
>   (device) basis, we can rely on the standard driver override framework
>   (ie. driver_override, unbind/bind). To make that work, at first we need
>   to add driver_override support to ntb_bus.
> * In the case that ntb_netdev wants to explicitly select a transport type,
>   I think it should still be handled via the per-NTB-port driver_override
>   rather than building transport-selection logic into ntb_netdev itself
>   (perhaps with some extension to the boundary API for
>   ntb_transport_clients).
> * If ntb_transport_host / ntb_transport_edma are built-in modules, a
>   post-boot rebind might be sufficient in most cases. If that's not
>   sufficient, we could also consider providing a kernel parameter to define
>   a boot-time policy. For example, something like:
>     ntb_transport.policy=edma@...0:01:00.0,host@...0:5f:00.0
> 
> How doe that sound? In any case, I am planning to submit RFC v4.

Yup that sounds about what I was thinking. If you are submitting RFC v4 w/o the changes mentioned about, just mention that the progress is moving towards that in the cover letter to remind people. Thanks!

Any additional thoughts Jon?

> 
> Thanks for the review,
> Koichiro

Powered by blists - more mailing lists