linux-kernel - Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for grace hopper

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20231024082854.0b767d74.alex.williamson@redhat.com>
Date:   Tue, 24 Oct 2023 08:28:54 -0600
From:   Alex Williamson <alex.williamson@...hat.com>
To:     Ankit Agrawal <ankita@...dia.com>
Cc:     Jason Gunthorpe <jgg@...dia.com>,
        Yishai Hadas <yishaih@...dia.com>,
        "shameerali.kolothum.thodi@...wei.com" 
        <shameerali.kolothum.thodi@...wei.com>,
        "kevin.tian@...el.com" <kevin.tian@...el.com>,
        Aniket Agashe <aniketa@...dia.com>, Neo Jia <cjia@...dia.com>,
        Kirti Wankhede <kwankhede@...dia.com>,
        "Tarun Gupta (SW-GPU)" <targupta@...dia.com>,
        Vikram Sethi <vsethi@...dia.com>,
        Andy Currid <acurrid@...dia.com>,
        Alistair Popple <apopple@...dia.com>,
        John Hubbard <jhubbard@...dia.com>,
        Dan Williams <danw@...dia.com>,
        "Anuj Aggarwal (SW-GPU)" <anuaggarwal@...dia.com>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for
 grace hopper

On Tue, 24 Oct 2023 14:03:25 +0000
Ankit Agrawal <ankita@...dia.com> wrote:

> >> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated
> >> > as an IO Port BAR, it occurs to me that there's no config space
> >> > emulation of BAR2 (or BAR3) here.  Doesn't this mean that QEMU registers
> >> > the BAR as 32-bit, non-prefetchable?  ie. VFIOBAR.type & .mem64 are
> >> > wrong?  
> >>
> >> Maybe I didn't understand the question, but the PCI config space read/write
> >> would still be handled by vfio_pci_core_read/write() which returns the
> >> appropriate flags. I have checked that the device BARs are 64b and
> >> prefetchable in the VM.  
> >
> > vfio_pci_core_read/write() accesses the physical device, which doesn't
> > implement BAR2.  Why would an unimplemented BAR2 on the physical device
> > report 64-bit, prefetchable?
> >
> > QEMU records VFIOBAR.type and .mem64 from reading the BAR register in
> > vfio_bar_prepare() and passes this type to pci_register_bar() in
> > vfio_bar_register().  Without an implementation of a config space read
> > op in the variant driver and with no physical implementation of BAR2 on
> > the device, I don't see how we get correct values in these fields.  
> 
> I think I see the cause of confusion. There are real PCIe compliant BARs
> present on the device, just that it isn't being used once the C2C
> interconnect is active. The BARs are 64b prefetchable. Here it the lspci
> snippet of the device on the host.
> # lspci -v -s 9:1:0.0
> 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1)
>         Subsystem: NVIDIA Corporation Device 16eb
>         Physical Slot: 0-5
>         Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0, IOMMU group 19
>         Memory at 661002000000 (64-bit, prefetchable) [size=16M]
>         Memory at 662000000000 (64-bit, prefetchable) [size=128G]
>         Memory at 661000000000 (64-bit, prefetchable) [size=32M]
> 
> I suppose this answers the BAR sizing question as well?

Does this BAR2 size match the size we're reporting for the region?  Now
I'm confused why we need to intercept the BAR2 region info if there's
physically a real BAR behind it.  Thanks,

Alex