Message-ID: <ym3evivt3co5ic7p5co624me3hmvoa53it22a6ddcu7dacl3wk@le233yriaip5>
Date: Mon, 22 Dec 2025 15:51:06 +0530
From: Manivannan Sadhasivam <mani@...nel.org>
To: Krishna Chaitanya Chundru <krishna.chundru@....qualcomm.com>
Cc: Niklas Cassel <cassel@...nel.org>, Koichiro Den <den@...inux.co.jp>, 
	ntb@...ts.linux.dev, linux-pci@...r.kernel.org, dmaengine@...r.kernel.org, 
	linux-kernel@...r.kernel.org, Frank.Li@....com, kwilczynski@...nel.org, kishon@...nel.org, 
	bhelgaas@...gle.com, corbet@....net, vkoul@...nel.org, jdmason@...zu.us, 
	dave.jiang@...el.com, allenbh@...il.com, Basavaraj.Natikar@....com, 
	Shyam-sundar.S-k@....com, kurt.schwemmer@...rosemi.com, logang@...tatee.com, 
	jingoohan1@...il.com, lpieralisi@...nel.org, robh@...nel.org, jbrunet@...libre.com, 
	fancer.lancer@...il.com, arnd@...db.de, pstanner@...hat.com, 
	elfring@...rs.sourceforge.net
Subject: Re: [RFC PATCH v2 19/27] PCI: dwc: ep: Cache MSI outbound iATU
 mapping

On Mon, Dec 22, 2025 at 01:44:02PM +0530, Krishna Chaitanya Chundru wrote:
> 
> 
> On 12/22/2025 1:20 PM, Niklas Cassel wrote:
> > On Mon, Dec 22, 2025 at 10:40:12AM +0530, Krishna Chaitanya Chundru wrote:
> > > On 12/8/2025 1:27 PM, Niklas Cassel wrote:
> > > > On Sun, Nov 30, 2025 at 01:03:57AM +0900, Koichiro Den wrote:
> > > > 
> > > > I guess the problem is that some EPF drivers, even if only
> > > > one capability can be enabled (MSI/MSI-X), call both
> > > > pci_epc_set_msi() and pci_epc_set_msix(), e.g.:
> > > > https://github.com/torvalds/linux/blob/v6.18/drivers/pci/endpoint/functions/pci-epf-test.c#L969-L987
> > > > 
> > > > To fill in the number of MSI/MSI-X irqs.
> > > > 
> > > > While other EPF drivers only call either pci_epc_set_msi() or
> > > > pci_epc_set_msix(), depending on the IRQ type that will actually
> > > > be used:
> > > > https://github.com/torvalds/linux/blob/v6.18/drivers/nvme/target/pci-epf.c#L2247-L2262
> > > > 
> > > > I think both versions is okay, just because the number of IRQs
> > > > is filled in for both MSI/MSI-X, AFAICT, only one of them will
> > > > get enabled.
> > > > 
> > > > 
> > > > I guess it might be hard for an EPC driver to know which capability
> > > > that is currently enabled, as to enable a capability is only a config
> > > > space write by the host side.
> > > As the host is the one which enables MSI/MSIX, it should be better the
> > > controller
> > > driver takes this decision and the EPF driver just sends only raise_irq.
> > > Because technically, host can disable MSI and enable MSIX at runtime also.
> > > 
> > > In the controller driver,  it can check which is enabled and chose b/w
> > > MSIX/MSI/Legacy.
> > I'm not sure if I'm following, but if by "the controller driver", you
> > mean the EPC driver, and not the host side driver, how can the EPC
> > driver know how many interrupts a specific EPF driver wants to use?
> I meant the dwc drivers here.
> Set msi & set msix still need to called from the EPF driver only to tell how
> many
> interrupts they want to configure etc.

Please leave a newline before and after your reply to make it readable in text
based clients, which some of the poor folks like me still use.

> > 
> >  From the kdoc to pci_epc_set_msi(), the nr_irqs parameter is defined as:
> > @nr_irqs: number of MSI interrupts required by the EPF
> > https://github.com/torvalds/linux/blob/v6.19-rc2/drivers/pci/endpoint/pci-epc-core.c#L305
> > 
> > 
> > Anyway, I posted Koichiro's patch here:
> > https://lore.kernel.org/linux-pci/20251210071358.2267494-2-cassel@kernel.org/
> I will comment on that patch.
> > 
> > See my comment:
> >    pci-epf-test does change between MSI and MSI-X without calling
> >    dw_pcie_ep_stop(), however, the msg_addr address written by the host
> >    will be the same address, at least when using a Linux host using a DWC
> >    based controller. If another host ends up using different msg_addr for
> >    MSI and MSI-X, then I think that we will need to modify pci-epf-test to
> >    call a function when changing IRQ type, such that pcie-designware-ep.c
> >    can tear down the MSI/MSI-X mapping.
> Maybe for arm based systems we are getting same address but for x86 based
> systems
> it is not guarantee that you will get same address.
> > So if we want to improve things, I think we need to modify the EPF drivers
> > to call a function when changing the IRQ type. The EPF driver should know
> > which IRQ type that is currently in use (see e.g. nvme_epf->irq_type in
> > drivers/nvme/target/pci-epf.c).
> My suggestion is let EPF driver call raise_irq with the vector number then
> the dwc driver
> can raise IRQ based on which IRQ host enables it.
> > Additionally, I don't think that the host side should be allowed to change
> > the IRQ type (using e.g. setpci) when the EPF driver is in a "running state".
> In the host driver itelf they can choose to change it by using
> pci_alloc_irq_vectors
> <https://elixir.bootlin.com/linux/v6.18.2/C/ident/pci_alloc_irq_vectors>,
> Currently it is not present but in future someone can change it, as spec
> didn't say you
> can't update it.

The spec has some wording on it (though not very clear) in r6.0, sec 6.1.4

"System software initializes the message address and message data (from here on
referred to as the “vector”) during device configuration, allocating one or more
vectors to each MSI-capable Function."

This *sounds* like the MSI/MSI-X initialization should happen during device
configuration.

> > I think things will break badly if you e.g. try to do this on an PCIe
> > connected network card while the network card is in use.
> I agree on this.
> 
> I just want to highlight there is possibility of this in future, if someone
> comes up with a
> clean logic.
> 

I don't know if this is even possible. For example, I don't think a host is
allowed to reattach a device which was using MSI to a VM which only uses MSI-X
during live device migration in virtualization world. I bet the device might not
perform as expected if that happens.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Powered by blists - more mailing lists