Message-ID: <91246a61c5e89c87528e6421b01ec4fddc133901.camel@linux.ibm.com>
Date: Tue, 12 Nov 2024 15:58:57 +0100
From: Niklas Schnelle <schnelle@...ux.ibm.com>
To: Bjorn Helgaas <bhelgaas@...gle.com>,
        Gerald Schaefer
 <gerald.schaefer@...ux.ibm.com>,
        Heiko Carstens <hca@...ux.ibm.com>, Vasily
 Gorbik <gor@...ux.ibm.com>,
        Alexander Gordeev <agordeev@...ux.ibm.com>,
        Sven Schnelle <svens@...ux.ibm.com>,
        Christian Borntraeger
 <borntraeger@...ux.ibm.com>,
        Gerd Bayer <gbayer@...ux.ibm.com>
Cc: linux-s390@...r.kernel.org, linux-kernel@...r.kernel.org,
        linux-pci@...r.kernel.org
Subject: Re: [PATCH v2] PCI: s390: Handle ARI on bus without associated
 struct pci_dev

On Tue, 2024-11-05 at 16:34 +0100, Niklas Schnelle wrote:
> On Wed, 2024-09-18 at 11:04 +0200, Niklas Schnelle wrote:
> > On s390 PCI busses are virtualized and the downstream ports are
> > invisible to the OS and struct pci_bus::self is NULL. This associated
> > struct pci_dev is however relied upon in pci_ari_enabled() to check
> > whether ARI is enabled for the bus. ARI is therefor always detected as
> > disabled.
> > 
> > At the same time firmware on s390 always enables and relies upon ARI
> > thus causing a mismatch. Moreover with per-PCI function pass-through
> > there may exist busses with no function with devfn 0. For example
> > a SR-IOV capable device with two PFs may have separate function
> > dependency link chains for each of the PFs and their child VFs. In this
> > case the OS may only see the second PF and its child VFs on a bus
> > without a devfn 0 function. A situation which is also not supported by
> > the common pci_configure_ari() code.
> > 
> > Dispite simply being a mismatch this causes problems as some PCI devices
> > present a different SR-IOV topology depending on PCI_SRIOV_CTRL_ARI.
> > 
> > A similar mismatch may occur with SR-IOV when virtfn_add_bus() creates new
> > busses with no associated struct pci_dev. Here too pci_ari_enabled()
> > on these busses would return false even if ARI is actually used.
> > 
> > Prevent both mismatches by moving the ari_enabled flag from struct
> > pci_dev to struct pci_bus making it independent from struct pci_bus::
> > self. Let the bus inherit the ari_enabled state from its parent bus when
> > there is no bridge device such that busses added by virtfn_add_bus()
> > match their parent. For s390 set ari_enabled when the device supports
> > ARI in the awareness that all PCIe ports on s390 systems are ARI
> > capable.
> > 
> > Signed-off-by: Niklas Schnelle <schnelle@...ux.ibm.com>
> > ---
> > Note: In a comment of the v1 thread I discussed an alternative idea for
> > a generic solution that would include the Jailhouse hypervisor via
> > hypervisor_isolated_pci_functions(). As Bjorn correctly pointed out this
> > more generic solution however lacks a way to indicate that ARI is really
> > enabled in the hardware. So instead for now I propose to stick with this
> > patch which only enables this unconditionally on s390x and SR-IOV
> > virtual busses where the ARI is inherited.
> > ---
> > Changes in v2:
> > - Rebased on v6.11
> > - Link to v1: https://lore.kernel.org/r/20240730-ari_no_bus_dev-v1-1-7de17676f9fe@linux.ibm.com
> > 
> > 
> 
> @Bjorn, this seems to have dropped off your radar or would you like me
> to see if I can find a different approach?
> 
> Thanks,
> Niklas
> 

Note I'll send a v3 with some small changes for this. We've found in
internal testing that this patch set the ARI enabled on s390 too late
thus not using it it during probing. My previous testing only looked at
the ARI enabled flag later and so didn't detect this. Sorry for any
annoyance.

Thanks,
Niklas

Powered by blists - more mailing lists