linux-kernel - Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20231026205319.GA1832508@bhelgaas>
Date:   Thu, 26 Oct 2023 15:53:19 -0500
From:   Bjorn Helgaas <helgaas@...nel.org>
To:     Tomasz Pala <gotar@...anet.pl>
Cc:     linux-pci@...r.kernel.org,
        Dan J Williams <dan.j.williams@...el.com>,
        Kan Liang <kan.liang@...ux.intel.com>,
        Tony Luck <tony.luck@...el.com>,
        David E Box <david.e.box@...el.com>,
        Yunying Sun <yunying.sun@...el.com>,
        Dave Jiang <dave.jiang@...el.com>,
        Mika Westerberg <mika.westerberg@...ux.intel.com>,
        Giovanni Cabiddu <giovanni.cabiddu@...el.com>,
        Herbert Xu <herbert@...dor.apana.org.au>,
        Hans de Goede <hdegoede@...hat.com>,
        Florent DELAHAYE <linuxkernelml@...ead.fr>,
        Konrad J Hambrick <kjhambrick@...il.com>,
        Matt Hansen <2lprbe78@...k.com>,
        Nicholas Johnson <nicholas.johnson-opensource@...look.com.au>,
        Benoit Grégoire <benoitg@...us.ca>,
        Werner Sembach <wse@...edocomputers.com>,
        mumblingdrunkard@...tonmail.com, linux-kernel@...r.kernel.org,
        Bjorn Helgaas <bhelgaas@...gle.com>
Subject: Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of
 ECAM space

On Thu, Oct 12, 2023 at 05:33:47PM +0200, Tomasz Pala wrote:
> On Tue, Jan 10, 2023 at 12:02:43 -0600, Bjorn Helgaas wrote:
> > Normally we reject ECAM space unless it is reported as reserved in the E820
> > table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2).  This
> > means PCI extended config space (offsets 0x100-0xfff) may not be accessible.
> > 
> > Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does
> > mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is
> > normally converted to an E820 entry by a bootloader or EFI stub.
> > 
> > 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes
> > E820 entries that correspond to EfiMemoryMappedIO regions because some
> > other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the
> > E820 entries prevent Linux from allocating BAR space for hot-added devices.
> > 
> > Allow use of ECAM for extended config space when the region is covered by
> > an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
> > _CRS.
> 
> I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.

Thanks very much for the report, and sorry for the inconvenience and
my delay in looking at it.

> efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
> [mem 0x7f800000-0xfed1bfff] available for PCI devices
> PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
> [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
> PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
> ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
> ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
> ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
> ixgbe: probe of 0000:02:00.0 failed with error -16

Something is wrong with our allocation scheme.  Both the MMCONFIG
region and the ixgbe BAR 0 are at 0x80000000, which obviously cannot
work.  Maybe the full dmesg log will have a clue about why we didn't
move ixgbe out of the way.

> After disabling the code causing this (using always-false condition:
> 		if (size >= 256*1024 && 0) {
> ) in the chunk:
> 
> https://lore.kernel.org/lkml/20221208190341.1560157-2-helgaas@kernel.org/
> 
> the BAR starts at 0x90000000 (not 0x80000000):
> 
> efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map
> [...]
> [mem 0x90000000-0xfed1bfff] available for PCI devices
> [...]
> PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
> PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry
> 
> and everything seems to work again.
> 
> 
> I've got full system bootup logs from the upstream and worked around,
> but I'm not sure if this is OK to attach them (the CC list is long).

Would you mind opening a new report at https://bugzilla.kernel.org,
attaching those logs, and responding here with the URL?

I looked at the proxmox thread you mentioned, but sometimes people
strip out parts of the log they think are irrelevant, and in this
case, the stripped-out parts *are* relevant.

Bjorn