linux-kernel - Re: [PATCH] PCI: hide mysterious 8MB 64-bit pref BAR on Intel Arc PCIe Switch

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <dca5e72c-ab77-4bcc-af3b-8ac504f3629e@iscas.ac.cn>
Date: Tue, 22 Jul 2025 05:52:19 +0800
From: Vivian Wang <wangruikang@...as.ac.cn>
To: Icenowy Zheng <uwu@...nowy.me>, Bjorn Helgaas <bhelgaas@...gle.com>,
 Lucas De Marchi <lucas.demarchi@...el.com>,
 Thomas Hellström <thomas.hellstrom@...ux.intel.com>,
 Rodrigo Vivi <rodrigo.vivi@...el.com>
Cc: linux-pci@...r.kernel.org, intel-xe@...ts.freedesktop.org,
 linux-kernel@...r.kernel.org, Han Gao <rabenda.cn@...il.com>
Subject: Re: [PATCH] PCI: hide mysterious 8MB 64-bit pref BAR on Intel Arc
 PCIe Switch

On 7/22/25 01:30, Icenowy Zheng wrote:

> The upstream port device of Intel Arc series dGPUs' internal PCIe switch
> contains a mysterious 8MB 64-bit prefetchable BAR. All reads to memory
> mapped to that BAR returns 0xFFFFFFFF and writes have no effect.

Reading all-ones is certainly not a good indication that this BAR does
nothing. My guess would be that maybe it's some write-only registers
like MSI doorbells.

AMD has docs on how their GPUs have a tiny doorbell default-2M 64bit
pref BAR [1], which I find interesting, even though very indirect as
evidence.

[1]: https://rocm.docs.amd.com/en/latest/how-to/Bar-Memory.html#bar-configuration-for-amd-gpus

> When the PCI bus isn't configured by any firmware (e.g. a PCIe
> controller solely initialized by Linux kernel), the PCI space allocation
> algorithm of Linux will allocate the main VRAM BAR of Arc dGPU device at
> base+0, and then the 8MB BAR at base+256M, which prevents the main VRAM
> BAR gets resized. As the functionality and performance of Arc dGPU will
> get severely restricted with small BAR, this makes a problem.

While trying to look around for if any documentation is available on the
"mysterious BAR", I found that this problem has indeed been discovered
before with no satisfactory resolution. Linking for posterity:

[2]: https://issues.redhat.com/browse/RHEL-3672
[3]: https://github.com/raspberrypi/linux/issues/6621

If my doorbell theory was correct, and that this is analogous for
amdgpu, there is this interesting comment in
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c since commit d6895ad39f3b
("drm/amdgpu: resize VRAM BAR for CPU access v6"):

  /* Free the VRAM and doorbell BAR, we most likely need to move both. */

That seems to be the right way *if* it makes sense that the "mysterious
BAR" is supposed to be managed by the GPU driver. Of course, only
someone from Intel can confirm for sure what's going on...

Vivian "dramforever" Wang

> Hide the mysterious 8MB BAR to Linux PCI subsystem, to allow resizing
> the VRAM BAR to VRAM size with the Linux PCI space allocation algorithm.
>
> Signed-off-by: Icenowy Zheng <uwu@...nowy.me>
> ---
>  drivers/pci/quirks.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index d7f4ee634263..df304bfec6e9 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -3650,6 +3650,22 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x37d0, quirk_broken_intx_masking);
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x37d1, quirk_broken_intx_masking);
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x37d2, quirk_broken_intx_masking);
>  
> +/*
> + * Intel Arc dGPUs' internal switch upstream port contains a mysterious 8MB
> + * 64-bit prefetchable BAR that blocks resize of main dGPU VRAM BAR with
> + * Linux's PCI space allocation algorithm.
> + */
> +static void quirk_intel_xe_upstream(struct pci_dev *pdev)
> +{
> +	memset(&pdev->resource[0], 0, sizeof(pdev->resource[0]));
> +}
> +/* Intel Arc A380 PCI Express Switch Upstream Port */
> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x4fa1, quirk_intel_xe_upstream);
> +/* Intel Arc A770 PCI Express Switch Upstream Port */
> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x4fa0, quirk_intel_xe_upstream);
> +/* Intel Arc B580 PCI Express Switch Upstream Port */
> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0xe2ff, quirk_intel_xe_upstream);
> +
>  static u16 mellanox_broken_intx_devs[] = {
>  	PCI_DEVICE_ID_MELLANOX_HERMON_SDR,
>  	PCI_DEVICE_ID_MELLANOX_HERMON_DDR,