linux-kernel - Re: drivers/pci: (and/or KVM): Slow PCI initialization during VM boot with passthrough of large BAR Nvidia GPUs on DGX H100

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20241126154145.638dba46.alex.williamson@redhat.com>
Date: Tue, 26 Nov 2024 15:41:45 -0700
From: Alex Williamson <alex.williamson@...hat.com>
To: Mitchell Augustin <mitchell.augustin@...onical.com>
Cc: linux-pci@...r.kernel.org, kvm@...r.kernel.org, Bjorn Helgaas
 <bhelgaas@...gle.com>, linux-kernel@...r.kernel.org
Subject: Re: drivers/pci: (and/or KVM): Slow PCI initialization during VM
 boot with passthrough of large BAR Nvidia GPUs on DGX H100

On Tue, 26 Nov 2024 16:18:26 -0600
Mitchell Augustin <mitchell.augustin@...onical.com> wrote:
> > The BAR space is walked, faulted, and mapped.  I'm sure you're at
> > least experiencing scaling issues of that with 128GB BARs.  
> 
> The part that is strange to me is that I don't see the initialization
> slowdown at all when the GPUs are hotplugged after boot completes.
> Isn't what you describe here also happening during the hotplugging
> process, or is it different in some way?

The only thing that comes to mind would be if you're using a vIOMMU and
it's configured in non-passthrough mode so the BARs aren't added to the
DMA address space after the vIOMMU is enabled during boot.  But your
virt-install command doesn't show a vIOMMU configuration for the VM.

If the slowness is confined to the guest kernel boot, can you share the
log of that boot with timestamps?  Thanks,

Alex