linux-kernel - Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for grace hopper

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <BY5PR12MB37636C06DED20856CF604A86B0DFA@BY5PR12MB3763.namprd12.prod.outlook.com>
Date:   Tue, 24 Oct 2023 14:03:25 +0000
From:   Ankit Agrawal <ankita@...dia.com>
To:     Alex Williamson <alex.williamson@...hat.com>
CC:     Jason Gunthorpe <jgg@...dia.com>,
        Yishai Hadas <yishaih@...dia.com>,
        "shameerali.kolothum.thodi@...wei.com" 
        <shameerali.kolothum.thodi@...wei.com>,
        "kevin.tian@...el.com" <kevin.tian@...el.com>,
        Aniket Agashe <aniketa@...dia.com>, Neo Jia <cjia@...dia.com>,
        Kirti Wankhede <kwankhede@...dia.com>,
        "Tarun Gupta (SW-GPU)" <targupta@...dia.com>,
        Vikram Sethi <vsethi@...dia.com>,
        Andy Currid <acurrid@...dia.com>,
        Alistair Popple <apopple@...dia.com>,
        John Hubbard <jhubbard@...dia.com>,
        Dan Williams <danw@...dia.com>,
        "Anuj Aggarwal (SW-GPU)" <anuaggarwal@...dia.com>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for grace
 hopper

>> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated
>> > as an IO Port BAR, it occurs to me that there's no config space
>> > emulation of BAR2 (or BAR3) here.  Doesn't this mean that QEMU registers
>> > the BAR as 32-bit, non-prefetchable?  ie. VFIOBAR.type & .mem64 are
>> > wrong?
>>
>> Maybe I didn't understand the question, but the PCI config space read/write
>> would still be handled by vfio_pci_core_read/write() which returns the
>> appropriate flags. I have checked that the device BARs are 64b and
>> prefetchable in the VM.
>
> vfio_pci_core_read/write() accesses the physical device, which doesn't
> implement BAR2.  Why would an unimplemented BAR2 on the physical device
> report 64-bit, prefetchable?
>
> QEMU records VFIOBAR.type and .mem64 from reading the BAR register in
> vfio_bar_prepare() and passes this type to pci_register_bar() in
> vfio_bar_register().  Without an implementation of a config space read
> op in the variant driver and with no physical implementation of BAR2 on
> the device, I don't see how we get correct values in these fields.

I think I see the cause of confusion. There are real PCIe compliant BARs
present on the device, just that it isn't being used once the C2C
interconnect is active. The BARs are 64b prefetchable. Here it the lspci
snippet of the device on the host.
# lspci -v -s 9:1:0.0
0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1)
        Subsystem: NVIDIA Corporation Device 16eb
        Physical Slot: 0-5
        Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0, IOMMU group 19
        Memory at 661002000000 (64-bit, prefetchable) [size=16M]
        Memory at 662000000000 (64-bit, prefetchable) [size=128G]
        Memory at 661000000000 (64-bit, prefetchable) [size=32M]

I suppose this answers the BAR sizing question as well?