Message-ID: <3wtr4dllqg2ijbzwb4eklmcwwuzgt4my4bdcw2ivslgj3aoix4@kckvh7fpna6y>
Date: Wed, 4 Feb 2026 16:27:46 +0900
From: Koichiro Den <den@...inux.co.jp>
To: Frank Li <Frank.li@....com>
Cc: vkoul@...nel.org, mani@...nel.org, jingoohan1@...il.com, 
	lpieralisi@...nel.org, kwilczynski@...nel.org, robh@...nel.org, bhelgaas@...gle.com, 
	dmaengine@...r.kernel.org, linux-pci@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v2 0/7] dmaengine: dw-edma, PCI: dwc: Enable remote use
 of integrated DesignWare eDMA

On Tue, Feb 03, 2026 at 12:26:34PM -0500, Frank Li wrote:
> On Tue, Feb 03, 2026 at 10:59:10AM +0900, Koichiro Den wrote:
> > On Mon, Feb 02, 2026 at 11:59:05AM -0500, Frank Li wrote:
> > > On Sun, Feb 01, 2026 at 11:32:23AM +0900, Koichiro Den wrote:
> > > > On Tue, Jan 27, 2026 at 12:34:13PM +0900, Koichiro Den wrote:
> > > > > Hi,
> > > > >
> > > > > Per Frank Li's suggestion [1], this revision combines the previously posted
> > > > > PCI/dwc helper series and the dmaengine/dw-edma series into a single
> > > > > 7-patch set. This series therefore supersedes the two earlier postings:
> > > > >
> > > > >   - [PATCH 0/5] dmaengine: dw-edma: Add helpers for remote eDMA use scenarios
> > > > >     https://lore.kernel.org/dmaengine/20260126073652.3293564-1-den@valinux.co.jp/
> > > > >   - [PATCH 0/2] PCI: dwc: Expose integrated DesignWare eDMA windows
> > > > >     https://lore.kernel.org/linux-pci/20260126071550.3233631-1-den@valinux.co.jp/
> > > > >
> > > > > [1] https://lore.kernel.org/linux-pci/aXeoxxG+9cFML1sx@lizhi-Precision-Tower-5810/
> > > > >
> > > > > Some DesignWare PCIe endpoint platforms integrate a DesignWare eDMA
> > > > > instance alongside the PCIe controller. In remote eDMA use cases, the host
> > > > > needs access to the eDMA register block and the per-channel linked-list
> > > > > (LL) regions via PCIe BARs, while the endpoint may still boot with a
> > > > > standard EP configuration (and may also use dw-edma locally).
> > > > >
> > > > > This series provides the following building blocks:
> > > > >
> > > > >   * dmaengine: Add an optional dma_slave_caps.hw_id so DMA providers can expose
> > > > >     a provider-defined hardware channel identifier to clients, and report it
> > > > >     from dw-edma. This allows users to correlate a DMA channel with
> > > > >     hardware-specific resources such as per-channel LL regions.
> > > > >
> > > > >   * dmaengine/dw-edma: Add features useful for remote-controlled EP eDMA usage:
> > > > >       - per-channel interrupt routing control (configured via dmaengine_slave_config(),
> > > > >         passing a small dw-edma-specific structure through
> > > > >         dma_slave_config.peripheral_config / dma_slave_config.peripheral_size),
> > > > >       - optional completion polling when local IRQ handling is disabled, and
> > > > >       - notify-only channels for cases where the local side needs interrupt
> > > > > 	notification without cookie-based accounting (i.e. its peer
> > > > > 	prepares and submits the descriptors), useful when host-to-endpoint
> > > > > 	interrupt delivery is difficult or unavailable without it.
> > > > >
> > > > >   * PCI: dwc: Add query-only helper APIs to expose resources of an integrated
> > > > >     DesignWare eDMA instance:
> > > > >       - the physical base/size of the eDMA register window, and
> > > > >       - the per-channel LL region base/size, keyed by transfer direction and
> > > > >         the hardware channel identifier (hw_id).
> > > > >
> > > > > The first real user will likely be the DesignWare backend in the NTB transport work:
> > > > >
> > > > >   [RFC PATCH v4 25/38] NTB: hw: Add remote eDMA backend registry and DesignWare backend
> > > > >   https://lore.kernel.org/linux-pci/20260118135440.1958279-26-den@valinux.co.jp/
> > > > >
> > > > >     (Note: the implementation in this series has been updated since that
> > > > >     RFC v4, so the RFC series will also need some adjustments. I have an
> > > > >     updated RFC series locally and can post an RFC v5 if that would help
> > > > >     review/testing.)
> > > > >
> > > > > Apply/merge notes:
> > > > >   - Patches 1-5 apply cleanly on dmaengine.git next.
> > > > >   - Patches 6-7 apply cleanly on pci.git controller/dwc.
> > > > >
> > > > > Changes in v2:
> > > > >   - Combine the two previously posted series into a single set (per Frank's
> > > > >     suggestion). Order dmaengine/dw-edma patches first so hw_id support
> > > > >     lands before the PCI LL-region helper, which assumes
> > > > >     dma_slave_caps.hw_id availability.
> > > > >
> > > > > Thanks for reviewing,
> > > > >
> > > > >
> > > > > Koichiro Den (7):
> > > > >   dmaengine: Add hw_id to dma_slave_caps
> > > > >   dmaengine: dw-edma: Report channel hw_id in dma_slave_caps
> > > > >   dmaengine: dw-edma: Add per-channel interrupt routing control
> > > > >   dmaengine: dw-edma: Poll completion when local IRQ handling is
> > > > >     disabled
> > > > >   dmaengine: dw-edma: Add notify-only channels support
> > > > >   PCI: dwc: Add helper to query integrated dw-edma register window
> > > > >   PCI: dwc: Add helper to query integrated dw-edma linked-list region
> > > >
> > > >
> > > > Hi Mani, Vinod (and others),
> > > >
> > > > I'd appreciate your thoughts on the overall design of patches 3–5/7 when
> > > > you have a moment.
> > > >
> > > >   - [PATCH v2 3/7] dmaengine: dw-edma: Add per-channel interrupt routing control
> > > >     https://lore.kernel.org/dmaengine/20260127033420.3460579-4-den@valinux.co.jp/
> > > >   - [PATCH v2 4/7] dmaengine: dw-edma: Poll completion when local IRQ handling is disabled
> > > >     https://lore.kernel.org/dmaengine/20260127033420.3460579-5-den@valinux.co.jp/
> > > >   - [PATCH v2 5/7] dmaengine: dw-edma: Add notify-only channels support
> > > >     https://lore.kernel.org/dmaengine/20260127033420.3460579-6-den@valinux.co.jp/
> > > >
> > > > My cover letter might have been too terse, so let me add a bit more context
> > > > and two questions at the end.
> > > >
> > > > (Frank already provided helpful feedback on v1/RFC for Patch 3/7. Thanks, Frank.)
> > > >
> > > >
> > > > Motivation for these three patches
> > > > ----------------------------------
> > > >
> > > >   For remote use of a DMA channel (i.e. the host drives a channel that
> > > >   resides in the endpoint (EP)):
> > > >
> > > >   * The EP initializes its DMA channels during the normal SoC glue
> > > >     driver probe.
> > > >   * It obtains a dma_chan to delegate to the host via the standard
> > > >     dma_request_channel().
> > > >   * It exposes the underlying HW resources to the host ("delegation").
> > > >   * The host also acquires a channel via the standard
> > > >     dma_request_channel(). The underlying HW resource is the same as on the
> > > >     EP side, but it's driven remotely from the host.
> > > >
> > > >   and I'm targeting two operating modes:
> > > >
> > > >   1). Standard use of the remote channel
> > > >
> > > >     * The host submits a transaction and handles its completion, just
> > > >       like a local dmaengine client would. Under the hood, the HW channel
> > > >       resides in the remote EP.
> > > >     * In this scenario, we need to ensure:
> > > >       (a). completion interrupts are delivered only to the host. Or,
> > > >       (b). even if (a) is not possible (i.e. the EP also gets interrupted),
> > > >            the EP must not acknowledge/clear the interrupt status in a way
> > > >            that would race with host.
> > > >
> > > >                                   dmaengine_submit()
> > > >                                           :
> > > >                                           v
> > > >        +----------+                 +----------+
> > > >        | dma_chan |--(delegate)--->: dma_chan :
> > > >        +----------+                 +----------+
> > > >       EP (delegator)              Host (delegatee)
> > > >                                           ^
> > > >                                           :
> > > >                                 completion interrupt
> > > >
> > > >   2). Notify-only channel
> > > >
> > > >     * The host submits a transaction, and the EP gets the interrupt
> > > >       and runs any registered callbacks.
> > > >     * In this scenario, we need to ensure:
> > > >       (a). completion interrupts are delivered only to the EP. Or,
> > > >       (b). even if (a) is not possible (i.e. the host also gets
> > > >            interrupted), the host must not acknowledge/clear the interrupt
> > > >            status in a way that would race with the EP.
> > > >       (c). repeated dmaengine_submit() calls must not get stuck, even
> > > >            though it cannot rely on interrupt-driven transaction status
> > > >            management.
> > >
> > > According to RM:
> > >
> > > WR_DONE_INT_STATUS
> > > Done Interrupt Status. The DMA write channel has successfully completed the DMA transfer. For
> > > more information, see "Interrupts and Error Handling". Each bit corresponds to a DMA channel. Bit
> > > [0] corresponds to channel 0.
> > > - Enabling: For more information, see "Interrupts and Error Handling".
> > > - Masking: The DMA write interrupt Mask register has no effect on this register.
> > > - Clearing: You must write a 1'b1 to the corresponding channel bit in the DMA write interrupt Clear register
> > > to clear this interrupt bit.
> > >
> > > Note: You can write to this register to emulate interrupt generation, during software or hardware testing. A
> > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > write to the address triggers an interrupt, but the DMA does not set the Done or Abort bits in this register.
> > >
> > >
> > > So you just need write this register to trigger a fake irq. needn't do
> > > data transfer.
> >
> > Thanks for the comment.
> >
> > One concern I have with using the fake interrupt mechanism is that it is
> > effectively channel-less, while the only documented acknowledgment path is
> > via {WR,RD}_DONE_INT_CLEAR[x], which is strictly channel-based. The RM does
> > not describe how a channel-less, emulated interrupt is associated with (or
> > cleared by) a specific channel's INT_CLEAR bit.
> 
> According to spec, it should only generate a irq, but no bit set at status.
> so needn't clean.
> 
> >
> > Because of that, I don't think there is a spec-backed guarantee that
> > writing INT_CLEAR for an arbitrary (even if reserved) channel will reliably
> > clear a fake interrupt, though I might be missing something. In the very
> > first RFC v1 series [1], I ended up writing 1 to all enabled channels
> > simply as the most defensive approach. However, that clearly does not
> > compose well once the same eDMA instance is used for real data transfers,
> > since it risks clearing real completion events.
> 
> Transfer a dummy data is not big issue. Only have to write at least 2
> register to finish a data transfer to trigger remote doorbell.
> 
> If write INT_STATUS work, which will most likely push doorbell. I have not
> did test, dose write INT_STATUS work?

In the RM, WR_DONE_INT_CLEAR has the description:

    Done Interrupt Clear. You must write a 1'b1 to clear the
    corresponding bit in the Done interrupt status field of the
    DMA write interrupt status register. Each bit corresponds to a
    DMA channel. Bit [0] corresponds to channel 0.
    Note: Reading from this self-clearing register field always
    returns a '0'.

Based on this, I originally assumed that writing a 1 to at least one
channel bit was required. However, after reading your comment, I tested
this on R-Car S4 Spider (eDMA) and verified that writing all 0s to
INT_CLEAR is sufficient to deassert a raised fake irq. This means the fake
irq can be acked without interfering with ongoing real DMA transfers on any
channel.

For HDMA, I do not currently have access to suitable hardware, so I have
not been able to validate the same behavior. I guess something like this
might work:

    SET_BOTH_CH_32(dw, 0, int_clear, 0) // channel 0 chosen arbitrarily

Anyway, based on the above, I have prepared v3 of the series and also
locally updated my RFC series ("[RFC PATCH v4 00/38] NTB transport backed
by PCI EP embedded DMA") accordingly. This fake irq approach has been
working well in my testing, so I plan to send v3 shortly.

Thanks for the review, it helped a lot,
Koichiro

> 
> Frank
> 
> >
> > What's your view on this?
> >
> > [1] https://lore.kernel.org/all/20251023071916.901355-16-den@valinux.co.jp/
> >
> > Thanks again for taking a close look at this.
> >
> > Koichiro
> >
> > >
> > > Frank
> > >
> > > >       (d). callback can be registered for the dma_chan on the EP.
> > > >
> > > >                                   dmaengine_submit()
> > > >                                           :
> > > >                                           v
> > > >        +----------+                 +----------+
> > > >        | dma_chan |--(delegate)--->: dma_chan :
> > > >        +----------+                 +----------+
> > > >       EP (delegator)              Host (delegatee)
> > > >              ^
> > > >              :
> > > >       completion interrupt
> > > >
> > > >
> > > >   Patch mapping:
> > > >     - (a)(b) are addressed by [PATCH v2 3/7].
> > > >     - (c) is addressed by [PATCH v2 4/7].
> > > >     - (d) is addressed by [PATCH v2 5/7].
> > > >
> > > >
> > > > Questions
> > > > ---------
> > > >
> > > >   1. Are these two use cases (1) and (2) acceptable?
> > > >   2. To support (1) and (2), is the current implementation direction acceptable?
> > > >      Or should this be generalized into a dmaengine API rather than being a
> > > >      dw-edma-core-specific extension?
> > > >
> > > >
> > > > Any feedback would be appreciated.
> > > >
> > > > Kind regards,
> > > > Koichiro
> > > >
> > > >
> > > > >
> > > > >  MAINTAINERS                                  |   2 +-
> > > > >  drivers/dma/dmaengine.c                      |   1 +
> > > > >  drivers/dma/dw-edma/dw-edma-core.c           | 167 +++++++++++++++++--
> > > > >  drivers/dma/dw-edma/dw-edma-core.h           |  21 +++
> > > > >  drivers/dma/dw-edma/dw-edma-v0-core.c        |  26 ++-
> > > > >  drivers/pci/controller/dwc/pcie-designware.c |  74 ++++++++
> > > > >  drivers/pci/controller/dwc/pcie-designware.h |   2 +
> > > > >  include/linux/dma/edma.h                     |  57 +++++++
> > > > >  include/linux/dmaengine.h                    |   2 +
> > > > >  include/linux/pcie-dwc-edma.h                |  72 ++++++++
> > > > >  10 files changed, 398 insertions(+), 26 deletions(-)
> > > > >  create mode 100644 include/linux/pcie-dwc-edma.h
> > > > >
> > > > > --
> > > > > 2.51.0
> > > > >

Powered by blists - more mailing lists