Message-ID: <Zv7t9HnRsfTxb2Xs@lizhi-Precision-Tower-5810>
Date: Thu, 3 Oct 2024 15:18:12 -0400
From: Frank Li <Frank.li@....com>
To: Manivannan Sadhasivam <manivannan.sadhasivam@...aro.org>
Cc: Bjorn Helgaas <bhelgaas@...gle.com>, Richard Zhu <hongxing.zhu@....com>,
	Lucas Stach <l.stach@...gutronix.de>,
	Lorenzo Pieralisi <lpieralisi@...nel.org>,
	Krzysztof Wilczyński <kw@...ux.com>,
	Rob Herring <robh@...nel.org>, Shawn Guo <shawnguo@...nel.org>,
	Sascha Hauer <s.hauer@...gutronix.de>,
	Pengutronix Kernel Team <kernel@...gutronix.de>,
	Fabio Estevam <festevam@...il.com>, linux-pci@...r.kernel.org,
	linux-kernel@...r.kernel.org, linux-arm-kernel@...ts.infradead.org,
	imx@...ts.linux.dev, alyssa@...enzweig.io, bpf@...r.kernel.org,
	broonie@...nel.org, jgg@...pe.ca, joro@...tes.org,
	lgirdwood@...il.com, maz@...nel.org, p.zabel@...gutronix.de,
	robin.murphy@....com, will@...nel.org
Subject: Re: [PATCH v2 0/2] PCI: add enabe(disable)_device() hook for bridge

On Thu, Oct 03, 2024 at 10:45:28AM +0530, Manivannan Sadhasivam wrote:
> On Mon, Sep 30, 2024 at 03:42:20PM -0400, Frank Li wrote:
> > Some system's IOMMU stream(master) ID bits(such as 6bits) less than
> > pci_device_id (16bit). It needs add hardware configuration to enable
> > pci_device_id to stream ID convert.
> >
> > https://lore.kernel.org/imx/20240622173849.GA1432357@bhelgaas/
> > This ways use pcie bus notifier (like apple pci controller), when new PCIe
> > device added, bus notifier will call register specific callback to handle
> > look up table (LUT) configuration.
> >
> > https://lore.kernel.org/imx/20240429150842.GC1709920-robh@kernel.org/
> > which parse dt's 'msi-map' and 'iommu-map' property to static config LUT
> > table (qcom use this way). This way is rejected by DT maintainer Rob.
> >
>
> What is the issue in doing this during the probe() stage? It looks like you are
> working with the static info in the devicetree, which is already available
> during the controller probe().

There are problems.
One: It is not good to manually parser this property in pci host bridge
drivers.

Two: of_map default is bypass map. For example: if in dts only 2 sid, 0xA
and 0xB. If try to enable 3rd function RID(103), there are no error report.
of_map will return RID 103 as stream ID.  DMA will write to wrong
possition possibly.

https://elixir.bootlin.com/linux/v6.12-rc1/source/drivers/of/base.c#L2070

Three: LUT resource is limited, if DT provide 16 entry, but LUT have only 8
items, if more device enable, not LUT avaible and can't return error. of
course, it may fix dts, but It'd better that driver can handle error
properly when meet wrong dtb file.

>
> > Above ways can resolve LUT take or stream id out of usage the problem. If
> > there are not enough stream id resource, not error return, EP hardware
> > still issue DMA to do transfer, which may transfer to wrong possition.
> >
> > Add enable(disable)_device() hook for bridge can return error when not
> > enough resource, and PCI device can't enabled.
> >
>
> {enable/disable}_device() doesn't convey the fact you are mapping BDF to SID in
> the hardware. Maybe something like, {map/unmap}_bdf2sid() or similar would make
> sense.

It is called in PCI common code do_pci_enable_device(), hook functin name
should be similar with caller. {map/unmap}_bdf2sid() is just implementation
in dwc.

stream id is only ARM platform concept.

May other host bridge do difference thing at enable/disable_device().

So I am still perfer use {enable/disable}_device().


Frank

>
> - Mani
>
> > Basicallly this version can match Bjorn's requirement:
> > 1: simple, because it is rare that there are no LUT resource.
> > 2: EP driver probe failure when no LUT, but lspci can see such device.
> >
> > [    2.164415] nvme nvme0: pci function 0000:01:00.0
> > [    2.169142] pci 0000:00:00.0: Error enabling bridge (-1), continuing
> > [    2.175654] nvme 0000:01:00.0: probe with driver nvme failed with error -12
> >
> > > lspci
> > 0000:00:00.0 PCI bridge: Philips Semiconductors Device 0000
> > 0000:01:00.0 Non-Volatile memory controller: Micron Technology Inc 2100AI NVMe SSD [Nitro] (rev 03)
> >
> > To: Bjorn Helgaas <bhelgaas@...gle.com>
> > To: Richard Zhu <hongxing.zhu@....com>
> > To: Lucas Stach <l.stach@...gutronix.de>
> > To: Lorenzo Pieralisi <lpieralisi@...nel.org>
> > To: Krzysztof Wilczyński <kw@...ux.com>
> > To: Manivannan Sadhasivam <manivannan.sadhasivam@...aro.org>
> > To: Rob Herring <robh@...nel.org>
> > To: Shawn Guo <shawnguo@...nel.org>
> > To: Sascha Hauer <s.hauer@...gutronix.de>
> > To: Pengutronix Kernel Team <kernel@...gutronix.de>
> > To: Fabio Estevam <festevam@...il.com>
> > Cc: linux-pci@...r.kernel.org
> > Cc: linux-kernel@...r.kernel.org
> > Cc: linux-arm-kernel@...ts.infradead.org
> > Cc: imx@...ts.linux.dev
> > Cc: Frank.li@....com \
> > Cc: alyssa@...enzweig.io \
> > Cc: bpf@...r.kernel.org \
> > Cc: broonie@...nel.org \
> > Cc: jgg@...pe.ca \
> > Cc: joro@...tes.org \
> > Cc: l.stach@...gutronix.de \
> > Cc: lgirdwood@...il.com \
> > Cc: maz@...nel.org \
> > Cc: p.zabel@...gutronix.de \
> > Cc: robin.murphy@....com \
> > Cc: will@...nel.org \
> >
> > Signed-off-by: Frank Li <Frank.Li@....com>
> > ---
> > Changes in v2:
> > - see each patch
> > - Link to v1: https://lore.kernel.org/r/20240926-imx95_lut-v1-0-d0c62087dbab@nxp.com
> >
> > ---
> > Frank Li (2):
> >       PCI: Add enable_device() and disable_device() callbacks for bridges
> >       PCI: imx6: Add IOMMU and ITS MSI support for i.MX95
> >
> >  drivers/pci/controller/dwc/pci-imx6.c | 133 +++++++++++++++++++++++++++++++++-
> >  drivers/pci/pci.c                     |  14 ++++
> >  include/linux/pci.h                   |   2 +
> >  3 files changed, 148 insertions(+), 1 deletion(-)
> > ---
> > base-commit: 2849622e7b01d5aea1b060ba3955054798c1e0bb
> > change-id: 20240926-imx95_lut-1c68222e0944
> >
> > Best regards,
> > ---
> > Frank Li <Frank.Li@....com>
> >
> >
>
> --
> மணிவண்ணன் சதாசிவம்

Powered by blists - more mailing lists