Message-ID: <Z8sRl1c//SZXKhB+@lizhi-Precision-Tower-5810>
Date: Fri, 7 Mar 2025 10:32:39 -0500
From: Frank Li <Frank.li@....com>
To: Bjorn Helgaas <helgaas@...nel.org>
Cc: Rob Herring <robh@...nel.org>, Saravana Kannan <saravanak@...gle.com>,
	Jingoo Han <jingoohan1@...il.com>,
	Manivannan Sadhasivam <manivannan.sadhasivam@...aro.org>,
	Lorenzo Pieralisi <lpieralisi@...nel.org>,
	Krzysztof Wilczyński <kw@...ux.com>,
	Bjorn Helgaas <bhelgaas@...gle.com>,
	Richard Zhu <hongxing.zhu@....com>,
	Lucas Stach <l.stach@...gutronix.de>,
	Shawn Guo <shawnguo@...nel.org>,
	Sascha Hauer <s.hauer@...gutronix.de>,
	Pengutronix Kernel Team <kernel@...gutronix.de>,
	Fabio Estevam <festevam@...il.com>, devicetree@...r.kernel.org,
	linux-kernel@...r.kernel.org, linux-pci@...r.kernel.org,
	linux-arm-kernel@...ts.infradead.org, imx@...ts.linux.dev,
	Niklas Cassel <cassel@...nel.org>
Subject: Re: [PATCH v9 3/7] PCI: Add parent_bus_offset to resource_entry

On Tue, Mar 04, 2025 at 05:25:45PM -0500, Frank Li wrote:
> On Tue, Mar 04, 2025 at 05:11:54PM -0500, Frank Li wrote:
> > On Tue, Mar 04, 2025 at 11:50:10AM -0600, Bjorn Helgaas wrote:
> > > On Mon, Mar 03, 2025 at 04:57:29PM -0500, Frank Li wrote:
> > > > On Wed, Feb 26, 2025 at 06:23:26PM -0600, Bjorn Helgaas wrote:
> > > > > On Tue, Jan 28, 2025 at 05:07:36PM -0500, Frank Li wrote:
> > > > > > Introduce `parent_bus_offset` in `resource_entry` and a new API,
> > > > > > `pci_add_resource_parent_bus_offset()`, to provide necessary information
> > > > > > for PCI controllers with address translation units.
> > > > > >
> > > > > > Typical PCI data flow involves:
> > > > > >   CPU (CPU address) -> Bus Fabric (Intermediate address) ->
> > > > > >   PCI Controller (PCI bus address) -> PCI Bus.
> > > > > >
> > > > > > While most bus fabrics preserve address consistency, some modify addresses
> > > > > > to intermediate values. The `parent_bus_offset` enables PCI controllers to
> > > > > > translate these intermediate addresses correctly to PCI bus addresses.
> > > > > >
> > > > > > Pave the road to remove hardcoded cpu_addr_fixup() and similar patterns in
> > > > > > PCI controller drivers.
> > > > > > ...
> > > > >
> > > > > > +++ b/drivers/pci/of.c
> > > > > > @@ -402,7 +402,17 @@ static int devm_of_pci_get_host_bridge_resources(struct device *dev,
> > > > > >  			res->flags &= ~IORESOURCE_MEM_64;
> > > > > >  		}
> > > > > >
> > > > > > -		pci_add_resource_offset(resources, res,	res->start - range.pci_addr);
> > > > > > +		/*
> > > > > > +		 * IORESOURCE_IO res->start is io space start address.
> > > > > > +		 * IORESOURCE_MEM res->start is cpu start address, which is the
> > > > > > +		 * same as range.cpu_addr.
> > > > > > +		 *
> > > > > > +		 * Use (range.cpu_addr - range.parent_bus_addr) to align both
> > > > > > +		 * IO and MEM's parent_bus_offset always offset to cpu address.
> > > > > > +		 */
> > > > > > +
> > > > > > +		pci_add_resource_parent_bus_offset(resources, res, res->start - range.pci_addr,
> > > > > > +						   range.cpu_addr - range.parent_bus_addr);
> > > > >
> > > > > I don't know exactly where it needs to go, but I think we can call
> > > > > .cpu_addr_fixup() once at startup on the base of the region.  This
> > > > > will tell us the offset that applies to the entire region, i.e.,
> > > > > parent_bus_offset.
> > > > >
> > > > > Then we can remove all the .cpu_addr_fixup() calls in
> > > > > cdns_pcie_host_init_address_translation(),
> > > > > cdns_pcie_set_outbound_region(), and dw_pcie_prog_outbound_atu().
> > > > >
> > > > > Until we can get rid of all the .cpu_addr_fixup() implementations,
> > > > > We'll still have that single call at startup (I guess once for cadence
> > > > > and another for designware), but it should simplify the current
> > > > > callers quite a bit.
> > > >
> > > > I don't think it can simple code. cdns_pcie_set_outbound_region() and
> > > > dw_pcie_prog_outbound_atu() are called by EP functions, which have not use
> > > > "resource" to manage outbound windows.
> > >
> > > Let's ignore cadence for now.  I don't think we need to solve that
> > > until later.
> > >
> > > dw_pcie_prog_outbound_atu() is called by:
> > >
> > >   - dw_pcie_other_conf_map_bus(): atu.parent_bus_addr = pp->cfg0_base
> > >
> > >     I think dw_pcie_host_init() can set pp->cfg0_base with the correct
> > >     intermediate address, either via the the of_property_read_reg() or
> > >     .cpu_addr_fixup().
>
> And chicken and egg problem here for artpec6_pcie_cpu_addr_fixup(), which
> need cfg0_base. But try to use .cpu_addr_fixup() to get cfg0_base's
> intermediate address.

Bjorn:
	Do you have chance to check my reply? some dwc platform driver
.cpu_addr_fixup() implement have dependence with old initilize sequency.

	Can I use original method? If change each driver's .cpu_addr_fixup()
implement, it will involve more risk, even more than directly clean it as
my RFC patch.

Frank
>
> Frank
>
> > >
> > >     If dw_pcie_host_init() does this, then we don't need
> > >     .cpu_addr_fixup() in dw_pcie_prog_outbound_atu().
> > >
> > >   - dw_pcie_rd_other_conf(): atu.parent_bus_addr = pp->io_base
> > >
> > >     Similarly, dw_pcie_host_init() should be able to set pp->io_base
> > >     to the intermediate address, so we don't need .cpu_addr_fixup() in
> > >     dw_pcie_prog_outbound_atu().
> >
> > I found some driver's cpu_addr_fixup()'s implement depend on the
> > initilize sequence.
> >
> > for example:
> > 	pcie-artpec6.c
> >
> > static u64 artpec6_pcie_cpu_addr_fixup(struct dw_pcie *pci, u64 cpu_addr)
> > {
> >         struct artpec6_pcie *artpec6_pcie = to_artpec6_pcie(pci);
> >         struct dw_pcie_rp *pp = &pci->pp;
> >         struct dw_pcie_ep *ep = &pci->ep;
> >
> >         switch (artpec6_pcie->mode) {
> >         case DW_PCIE_RC_TYPE:
> >                 return cpu_addr - pp->cfg0_base;
> >         case DW_PCIE_EP_TYPE:
> >                 return cpu_addr - ep->phys_base;
> >         default:
> >                 dev_err(pci->dev, "UNKNOWN device type\n");
> >         }
> >         return cpu_addr;
> > }
> >
> > This implement require *cfg0_base* and *phys_base*, pp/ep, need set before
> > call artpec6_pcie_cpu_addr_fixup().
> >
> > static u64 visconti_pcie_cpu_addr_fixup(struct dw_pcie *pci, u64 cpu_addr)
> > {
> >         struct dw_pcie_rp *pp = &pci->pp;
> >
> >         return cpu_addr & ~pp->io_base;
> > }
> >
> > this one require *io_base* and *pp* need be set before call
> > visconti_pcie_cpu_addr_fixup()
> >
> > Because I have not such hardware platform, it is not trivial change and
> > it is hard to involve bugs.
> >
> > If move .cpu_addr_fixup() too early, it will cause kernel dump.
> >
> > I suggest keep current overall sequent and try to clean up these driver's
> > cpu_addr_fixup() firstly.
> >
> > Frank
> >
> > >
> > >   - dw_pcie_iatu_setup(): atu.parent_bus_addr = entry->res->start
> > >
> > >     Here "entry" iterates through bridge->windows, and we should be
> > >     able to set entry->parent_bus_offset at init-time, using
> > >     .cpu_addr_fixup() if necessary, so we can apply that offset
> > >     unconditionally, regardless of use_parent_dt_ranges, and we won't
> > >     need .cpu_addr_fixup() in dw_pcie_prog_outbound_atu().
> > >
> > >   - dw_pcie_pme_turn_off:
> > >     atu.parent_bus_addr = pci->pp.msg_res->start - pci->pp.msg_parent_bus_offset
> > >
> > >     This should be the same as dw_pcie_iatu_setup() since
> > >     msg_parent_bus_offset comes from the window iteration in
> > >     dw_pcie_host_request_msg_tlp_res().  As long as the windows have
> > >     the correct parent_bus_offset at init-time, we should be all set.

Powered by blists - more mailing lists