Message-ID: <20250116-removed-evoke-1908811ab92a@spud>
Date: Thu, 16 Jan 2025 17:45:33 +0000
From: Conor Dooley <conor@...nel.org>
To: Bjorn Helgaas <helgaas@...nel.org>
Cc: daire.mcnamara@...rochip.com, linux-pci@...r.kernel.org,
	devicetree@...r.kernel.org, conor.dooley@...rochip.com,
	lpieralisi@...nel.org, kw@...ux.com, robh@...nel.org,
	bhelgaas@...gle.com, linux-kernel@...r.kernel.org,
	linux-riscv@...ts.infradead.org, krzk+dt@...nel.org,
	conor+dt@...nel.org, ilpo.jarvinen@...ux.intel.com,
	kevin.xie@...rfivetech.com, Frank Li <Frank.Li@....com>
Subject: Re: [PATCH v10 1/3] PCI: microchip: Fix outbound address translation
 tables

On Thu, Jan 16, 2025 at 11:07:22AM -0600, Bjorn Helgaas wrote:
> [+cc Frank, original patch at
> https://lore.kernel.org/r/20241011140043.1250030-2-daire.mcnamara@microchip.com]
> 
> On Thu, Jan 16, 2025 at 04:46:19PM +0000, Conor Dooley wrote:
> > On Thu, Jan 16, 2025 at 09:42:53AM -0600, Bjorn Helgaas wrote:
> > > On Tue, Jan 14, 2025 at 06:13:10PM -0600, Bjorn Helgaas wrote:
> > > > On Fri, Oct 11, 2024 at 03:00:41PM +0100, daire.mcnamara@...rochip.com wrote:
> > > > > From: Daire McNamara <daire.mcnamara@...rochip.com>
> > > > > 
> > > > > On Microchip PolarFire SoC (MPFS) the PCIe Root Port can be behind one of
> > > > > three general-purpose Fabric Interface Controller (FIC) buses that
> > > > > encapsulate an AXI-M interface. That FIC is responsible for managing
> > > > > the translations of the upper 32-bits of the AXI-M address. On MPFS,
> > > > > the Root Port driver needs to take account of that outbound address
> > > > > translation done by the parent FIC bus before setting up its own
> > > > > outbound address translation tables.  In all cases on MPFS,
> > > > > the remaining outbound address translation tables are 32-bit only.
> > > > > 
> > > > > Limit the outbound address translation tables to 32-bit only.
> > > > 
> > > > I don't quite understand what this is saying.  It seems like the code
> > > > keeps only the low 32 bits of a PCI address and throws away any
> > > > address bits above the low 32.
> > > > 
> > > > If that's what the FIC does, I wouldn't describe the FIC as
> > > > "translating the upper 32 bits" since it sounds like the translation
> > > > is just truncation.
> > > > 
> > > > I guess it must be more complicated than that?  I assume you can still
> > > > reach BARs that have PCI addresses above 4GB using CPU loads/stores?
> > > > 
> > > > The apertures through the host bridge for MMIO access are described by
> > > > DT ranges properties, so this must be something that can't be
> > > > described that way?
> > > 
> > > Ping?  I'd really like to understand this before the v6.14 merge
> > > window opens on Sunday.
> > 
> > Daire's been having some issues getting onto the corporate VPN to send
> > his reply, I've pasted it below on his behalf:
> > 
> > There are 3 Fabric Inter Connect (FIC) buses on PolarFire SoC - each of
> > these FIC buses contain an AXI master bus and are 64-bits wide. These
> > AXI-Masters (each with an individual 64-bit AXI base address – for example
> > FIC1’s AXI Master has a base address of 0x2000000000) are connected to
> > general purpose FPGA logic. This FPGA logic is, in turn, connected to a
> > 2nd 32-bit AXI master which is attached to the PCIe block in RootPort mode.
> > Conceptually, on the other side of this configurable logic, there is a
> > 32-bit bus to a hard PCIe rootport.  So, again conceptually, outbound address
> > translation looks like this:
> > 
> >                  Processor Complex à FIC (64-bit AXI-M) à Configurable Logic à 32-bit AXI-M à PCIe Rootport
> > 		 (This how it came to me from Daire, I think the á is meant to
> > 		 be an arrow)
> > 
> >  This allows a designer two broad choices:
> > 
> >     Choice of FIC (effectively choice of AXI bus)
> >     Ability to offset the AXI address of any peripherals they add in the
> >     Fabric.
> > 
> > So, for the case of an outbound AXI address, from the processors’ point
> > of view (or Linux’ point of view if you prefer), the processor uses a
> > 64-bit AXI address, then – in a very general way of viewing the process
> > and thinking only about accessing the PCIe device – the FPGA logic can
> > be configured to adjust that AXI-M address to any arbitrary “address”
> > before it passes that new “address” to the Root Port over a second 32-bit
> > AXI bus (the main constraint is that the FPGA logic can only use a 32-bit
> > address on that AXI-M interface to the Root Port).
> > 
> > To manage this complexity, Microchip have design rules for customers
> > building their FPGA logic where we strongly recommend that they only
> > interact with  the upper 32 bits of the 64-bit address in the FPGA logic
> > and pass the lower 32 bits through (unmodified) to the AXI-M side of the
> > PCIe Root Port. This allows them to “move” a 64-bit AXI-M window for their
> > PCIe Root Port (as viewed by the processor) for their particular design –
> > if they need to - so that they can also access any other AXI-M windows
> > associated with any other peripherals they might add to their design.
> > 
> > In practise, so far, all customers, and our own internal boards have all
> > started by using one of two major reference designs from us (one using FIC1
> > where the AXI-M window destined for the PCIe Root Port starts at 0x2000000000
> > and one using FIC2 where its AXI-M window, again destined for the PCIe Root
> > Port starts at 0x3000000000).
> 
> Is there something special about this that cannot be described by a DT
> 'ranges' property?  This sounds conceptually similar to Frank's nice
> picture at
> https://lore.kernel.org/r/20241119-pci_fixup_addr-v8-2-c4bfa5193288@nxp.com


Aye, it is similar, it is described using ranges properties, will end
up looking something like:

	fabric-pcie-bus@...0000000 {
		compatible = "simple-bus";
		#address-cells = <2>;
		#size-cells = <2>;
		ranges = <0x0 0x40000000 0x0 0x40000000 0x0 0x20000000>,
			 <0x30 0x0 0x30 0x0 0x10 0x0>;
		dma-ranges = <0x0 0x0 0x0 0x80000000 0x0 0x4000000>,
			     <0x0 0x4000000 0x0 0xc4000000 0x0 0x6000000>,
			     <0x0 0xa000000 0x0 0x8a000000 0x0 0x8000000>,
			     <0x0 0x12000000 0x14 0x12000000 0x0 0x10000000>,
			     <0x0 0x22000000 0x10 0x22000000 0x0 0x5e000000>;

		pcie: pcie@...0000000 {
			compatible = "microchip,pcie-host-1.0";
			#address-cells = <0x3>;
			#interrupt-cells = <0x1>;
			#size-cells = <0x2>;
			device_type = "pci";

			dma-noncoherent;
			reg = <0x30 0x0 0x0 0x8000000>, <0x0 0x43008000 0x0 0x2000>, <0x0 0x4300a000 0x0 0x2000>;

			ranges = <0x43000000 0x0 0x9000000 0x30 0x9000000 0x0 0xf000000>,
				 <0x1000000 0x0 0x8000000 0x30 0x8000000 0x0 0x1000000>,
				 <0x3000000 0x0 0x18000000 0x30 0x18000000 0x0 0x70000000>;
			dma-ranges = <0x3000000 0x0 0x80000000 0x0 0x0 0x0 0x4000000>,
				     <0x3000000 0x0 0x84000000 0x0 0x4000000 0x0 0x6000000>,
				     <0x3000000 0x0 0x8a000000 0x0 0xa000000 0x0 0x8000000>,
				     <0x3000000 0x0 0x92000000 0x0 0x12000000 0x0 0x10000000>,
				     <0x3000000 0x0 0xa2000000 0x0 0x22000000 0x0 0x5e000000>;

		};
	}

Download attachment "signature.asc" of type "application/pgp-signature" (229 bytes)

Powered by blists - more mailing lists