linux-kernel - Re: [PATCH v4 0/5] PCI: EP: Add RC-to-EP doorbell with platform MSI controller

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <ZywN2GosYG4ERGW2@lizhi-Precision-Tower-5810>
Date: Wed, 6 Nov 2024 19:46:16 -0500
From: Frank Li <Frank.li@....com>
To: Niklas Cassel <cassel@...nel.org>
Cc: Manivannan Sadhasivam <manivannan.sadhasivam@...aro.org>,
	Krzysztof Wilczyński <kw@...ux.com>,
	Kishon Vijay Abraham I <kishon@...nel.org>,
	Bjorn Helgaas <bhelgaas@...gle.com>, Arnd Bergmann <arnd@...db.de>,
	Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
	linux-kernel@...r.kernel.org, linux-pci@...r.kernel.org,
	imx@...ts.linux.dev, dlemoal@...nel.org, maz@...nel.org,
	tglx@...utronix.de, jdmason@...zu.us
Subject: Re: [PATCH v4 0/5] PCI: EP: Add RC-to-EP doorbell with platform MSI
 controller

On Thu, Nov 07, 2024 at 12:57:16AM +0100, Niklas Cassel wrote:
> On Wed, Nov 06, 2024 at 12:13:09PM -0500, Frank Li wrote:
> > On Wed, Nov 06, 2024 at 10:36:42AM +0100, Niklas Cassel wrote:
> > > On Wed, Nov 06, 2024 at 10:15:27AM +0100, Niklas Cassel wrote:
> > > >
> > > > I do get a domain, but I do not get any IRQ on the EP side when the RC side is
> > > > writing the doorbell (as part of pcitest -B),
> > > >
> > > > [    7.978984] pci_epc_alloc_doorbell: num_db: 1
> > > > [    7.979545] pci_epf_test_bind: doorbell_addr: 0x40
> > > > [    7.979978] pci_epf_test_bind: doorbell_data: 0x0
> > > > [    7.980397] pci_epf_test_bind: doorbell_bar: 0x1
> > > > [   21.114613] pci_epf_enable_doorbell db_bar: 1
> > > > [   21.115001] pci_epf_enable_doorbell: doorbell_addr: 0xfe650040
> > > > [   21.115512] pci_epf_enable_doorbell: phys_addr: 0xfe650000
> > > > [   21.115994] pci_epf_enable_doorbell: success
> > > >
> > > > # cat /proc/interrupts | grep epc
> > > > 117:          0          0          0          0          0          0          0          0  ITS-pMSI-a40000000.pcie-ep   0 Edge      pci-epc-doorbell0
> > > >
> > > > Even if I try to write the doorbell manually from EP side using devmem:
> > > >
> > > > # devmem 0xfe670040 32 1
> > >
> > > Sorry, this should of course have been:
> > > # devmem 0xfe650040 32 1
> >
> > Thank you test it. You can't write it at EP side. ITS identify the bus
> > master. master ID (streamid) of CPU is the diffference with PCI master's
> > ID (streamid). You set msi-parent = <&its0 0x0000>, not sure if 0x0000 is
> > validate stream.
>
> I see, this makes sense since the ITS converts BDF to an MSI specifier.
>
>
> >
> > You have to run at RC side, "devmem (Bar1+0x40) 32 0".  So PCIe EP
> > controller can use EP streamid.
> >
> > some system need special register to config stream id, you can refer host
> > side's settings.
>
> > <&its0 0x0000>,  second argument is your PCIe controller's stream ID. You
> > can ref RC side.
>
> The RC node looks like this:
> msi-map = <0x0000 &its1 0x0000 0x1000>;
> So it does indeed use 0x0 as the MSI specifier.
>
>
> >
> > >
> > > Considering that the RC node is using &its1, that is probably
> > > also what should be used in the EP node when running the controller
> > > in EP mode instead of RC mode.
> >
> > Generally,  RC node should use smmu-map, instead &its1. Or your PCI
> > controller direct use 16bit RID as streamid.
>
> smmu-map? Do you mean iommu-map?
>
> I don't see why we would need to have the SMMU enabled to use ITS.
> The iommu is currently disabled on my platform.
>
> I did enable the iommu, and all BAR tests, read tests, write tests,
> and copy tests pass. However I get an iommu error when the RC is
> writing the doorbell. Perhaps you need to do dma_map_single() on
> the address that you are setting the inbound address translation to?
>
>
>
> Without the IOMMU, if I modify pci_endpoint_test.c to not send the
> DISABLE_DOORBELL command on error (so that EP side still has DB enabled),
> I can read all BARs except BAR1 (which was used for the doorbell):

If you enable IOMMU, please double check pci_epc_write_msi_msg() pass
down iommu mapped address. I have not test iommu enabled's case yet.

Frank

> [   21.077417] pci 0000:01:00.0: BAR 0 [mem 0xf0300000-0xf03fffff]: assigned
> [   21.078029] pci 0000:01:00.0: BAR 1 [mem 0xf0400000-0xf04fffff]: assigned
> [   21.078640] pci 0000:01:00.0: BAR 2 [mem 0xf0500000-0xf05fffff]: assigned
> [   21.079250] pci 0000:01:00.0: BAR 3 [mem 0xf0600000-0xf06fffff]: assigned
> [   21.079860] pci 0000:01:00.0: BAR 5 [mem 0xf0700000-0xf07fffff]: assigned
> # pcitest -B
> [   25.156623] COMMAND_ENABLE_DOORBELL complete - status: 0x440
> [   25.157131] db_bar: 1 addr: 0x40 data: 0x0
> [   25.157501] setting irq_type to: 1
> [   25.157802] writing: 0x0 to offset: 0x40 in BAR: 1
> [   35.300676] we wrote to the BAR, status is now: 0x0
>
> status is not updated after writing to DB,
> and /proc/interrupts on EP side is not incrementing.
>
> # devmem 0xf0300000
> 0x00000000
> # devmem 0xf0400040
> 0xFFFFFFFF
> # devmem 0xf0500000
> 0x00000000
> # devmem 0xf0600000
> 0x00000000
> # devmem 0xf0700000
> 0x00000000
>
> So there does seem to be something wrong with the inbound translation,
> at least when testing on rk3588 which only uses 1MB fixed size BARs:
> https://github.com/torvalds/linux/blob/v6.12-rc6/drivers/pci/controller/dwc/pcie-dw-rockchip.c#L276-L281
>
>
>
> You also didn't answer which imx platform that you are using to test this,
> I don't see a single imx platform that specifies "msi-parent".
>
>
> Kind regards,
> Niklas