linux-kernel - Re: [PATCH] PCI: Mark NVIDIA T4 GPUs to avoid bus reset

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ZPqMCDWvITlOLHgJ@wuzongyong-alibaba>
Date:   Fri, 8 Sep 2023 10:50:48 +0800
From:   Wu Zongyong <wuzongyong@...ux.alibaba.com>
To:     Bjorn Helgaas <helgaas@...nel.org>
Cc:     Wu Zongyong <wuzongyong@...ux.alibaba.com>, lukas@...ner.de,
        sdonthineni@...dia.com, bhelgaas@...gle.com,
        linux-pci@...r.kernel.org, linux-kernel@...r.kernel.org,
        wllenyj@...ux.alibaba.com, wutu.xq2@...ux.alibaba.com,
        gerry@...ux.alibaba.com, pjaroszynski@...dia.com
Subject: Re: [PATCH] PCI: Mark NVIDIA T4 GPUs to avoid bus reset

On Wed, Aug 09, 2023 at 06:05:18PM -0500, Bjorn Helgaas wrote:
> On Mon, Apr 10, 2023 at 08:34:11PM +0800, Wu Zongyong wrote:
> > NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4
> > card is direct attached to a Root Port only. So avoid bus reset by
> > marking T4 GPUs PCI_DEV_FLAGS_NO_BUS_RESET.
> > 
> > Fixes: 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset")
> > Signed-off-by: Wu Zongyong <wuzongyong@...ux.alibaba.com>
> 
> Applied to pci/virtualization for v6.6, thanks!

I talk about the issue with NVIDIA, and they think the issue is probably related
the pci link instead of the T4 GPU card.

I will try to describe the issue I met in detail.

The T4 card which is direct attached to a Root Port and I rebind it to
vfio-pci driver. Then I try to use to call some vfio-related api and the
ioctl VFIO_GROUP_GET_DEVICE_FD failed.

The stack is (base on kernel v5.10):
    vfio_group_fops_unl_ioctl
         vfio_group_get_device_fd
            vfio_pci_open
                vfio_pci_enable // return value is -19
                    pci_try_reset_function
                        __pci_reset_function_locked

After the __pci_reset_function_locked(), the dmesg shows:
   [12207494.508467] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Down
   [12207494.508535] vfio-pci 0000:40:00.0: No device request channel registered, blocked until released by user
   [12207494.518426] pci 0000:40:00.0: Removing from iommu group 84
   [12207495.532365] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Card present
   [12207495.532367] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Up

NVIDIA people thinks this root port is not going through this reset logic and getting the
link down/hot plug interrupts[1].

Can you revert the patch I sent and maybe we should dig it deeply.

> 
> > ---
> >  drivers/pci/quirks.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index 44cab813bf95..be86b93b9e38 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)
> >   */
> >  static void quirk_nvidia_no_bus_reset(struct pci_dev *dev)
> >  {
> > -	if ((dev->device & 0xffc0) == 0x2340)
> > +	if ((dev->device & 0xffc0) == 0x2340 || dev->device == 0x1eb8)
> >  		quirk_no_bus_reset(dev);
> >  }
> >  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
> > -- 
> > 2.34.3
> >