Message-ID: <20250116-debatable-hazelnut-6501986373fa@spud>
Date: Thu, 16 Jan 2025 16:46:19 +0000
From: Conor Dooley <conor@...nel.org>
To: Bjorn Helgaas <helgaas@...nel.org>
Cc: daire.mcnamara@...rochip.com, linux-pci@...r.kernel.org,
	devicetree@...r.kernel.org, conor.dooley@...rochip.com,
	lpieralisi@...nel.org, kw@...ux.com, robh@...nel.org,
	bhelgaas@...gle.com, linux-kernel@...r.kernel.org,
	linux-riscv@...ts.infradead.org, krzk+dt@...nel.org,
	conor+dt@...nel.org, ilpo.jarvinen@...ux.intel.com,
	kevin.xie@...rfivetech.com
Subject: Re: [PATCH v10 1/3] PCI: microchip: Fix outbound address translation
 tables

On Thu, Jan 16, 2025 at 09:42:53AM -0600, Bjorn Helgaas wrote:
> On Tue, Jan 14, 2025 at 06:13:10PM -0600, Bjorn Helgaas wrote:
> > On Fri, Oct 11, 2024 at 03:00:41PM +0100, daire.mcnamara@...rochip.com wrote:
> > > From: Daire McNamara <daire.mcnamara@...rochip.com>
> > > 
> > > On Microchip PolarFire SoC (MPFS) the PCIe Root Port can be behind one of
> > > three general-purpose Fabric Interface Controller (FIC) buses that
> > > encapsulate an AXI-M interface. That FIC is responsible for managing
> > > the translations of the upper 32-bits of the AXI-M address. On MPFS,
> > > the Root Port driver needs to take account of that outbound address
> > > translation done by the parent FIC bus before setting up its own
> > > outbound address translation tables.  In all cases on MPFS,
> > > the remaining outbound address translation tables are 32-bit only.
> > > 
> > > Limit the outbound address translation tables to 32-bit only.
> > 
> > I don't quite understand what this is saying.  It seems like the code
> > keeps only the low 32 bits of a PCI address and throws away any
> > address bits above the low 32.
> > 
> > If that's what the FIC does, I wouldn't describe the FIC as
> > "translating the upper 32 bits" since it sounds like the translation
> > is just truncation.
> > 
> > I guess it must be more complicated than that?  I assume you can still
> > reach BARs that have PCI addresses above 4GB using CPU loads/stores?
> > 
> > The apertures through the host bridge for MMIO access are described by
> > DT ranges properties, so this must be something that can't be
> > described that way?
> 
> Ping?  I'd really like to understand this before the v6.14 merge
> window opens on Sunday.

Daire's been having some issues getting onto the corporate VPN to send
his reply, I've pasted it below on his behalf:

There are 3 Fabric Inter Connect (FIC) buses on PolarFire SoC - each of
these FIC buses contain an AXI master bus and are 64-bits wide. These
AXI-Masters (each with an individual 64-bit AXI base address – for example
FIC1’s AXI Master has a base address of 0x2000000000) are connected to
general purpose FPGA logic. This FPGA logic is, in turn, connected to a
2nd 32-bit AXI master which is attached to the PCIe block in RootPort mode.
Conceptually, on the other side of this configurable logic, there is a
32-bit bus to a hard PCIe rootport.  So, again conceptually, outbound address
translation looks like this:

                 Processor Complex à FIC (64-bit AXI-M) à Configurable Logic à 32-bit AXI-M à PCIe Rootport
		 (This how it came to me from Daire, I think the á is meant to
		 be an arrow)

 This allows a designer two broad choices:

    Choice of FIC (effectively choice of AXI bus)
    Ability to offset the AXI address of any peripherals they add in the
    Fabric.

 

So, for the case of an outbound AXI address, from the processors’ point
of view (or Linux’ point of view if you prefer), the processor uses a
64-bit AXI address, then – in a very general way of viewing the process
and thinking only about accessing the PCIe device – the FPGA logic can
be configured to adjust that AXI-M address to any arbitrary “address”
before it passes that new “address” to the Root Port over a second 32-bit
AXI bus (the main constraint is that the FPGA logic can only use a 32-bit
address on that AXI-M interface to the Root Port).

 

To manage this complexity, Microchip have design rules for customers
building their FPGA logic where we strongly recommend that they only
interact with  the upper 32 bits of the 64-bit address in the FPGA logic
and pass the lower 32 bits through (unmodified) to the AXI-M side of the
PCIe Root Port. This allows them to “move” a 64-bit AXI-M window for their
PCIe Root Port (as viewed by the processor) for their particular design –
if they need to - so that they can also access any other AXI-M windows
associated with any other peripherals they might add to their design.

 

In practise, so far, all customers, and our own internal boards have all
started by using one of two major reference designs from us (one using FIC1
where the AXI-M window destined for the PCIe Root Port starts at 0x2000000000
and one using FIC2 where its AXI-M window, again destined for the PCIe Root
Port starts at 0x3000000000).

best,

daire

Download attachment "signature.asc" of type "application/pgp-signature" (229 bytes)

Powered by blists - more mailing lists