linux-kernel - Re: PCIe 32-bit MMIO exhaustion

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAErSpo4t94BXYhhE+bh5-3_PsdKTSa113beqqz14W+_emdCGMQ@mail.gmail.com>
Date:	Thu, 29 Jan 2015 09:23:29 -0600
From:	Bjorn Helgaas <bhelgaas@...gle.com>
To:	Daniel J Blueman <daniel@...ascale.com>
Cc:	Ingo Molnar <mingo@...hat.com>,
	Jiang Liu <jiang.liu@...ux.intel.com>,
	H Peter Anvin <hpa@...or.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	Steffen Persvold <sp@...ascale.com>,
	"x86@...nel.org" <x86@...nel.org>, Yinghai Lu <yinghai@...nel.org>
Subject: Re: PCIe 32-bit MMIO exhaustion

[+cc Yinghai]

Hi Daniel,

On Wed, Jan 28, 2015 at 2:42 AM, Daniel J Blueman <daniel@...ascale.com> wrote:
> With systems with a large number of PCI devices, we're seeing lack of 32-bit
> MMIO space, eg one quad-port NetXtreme-2 adapter takes 128MB of space [1].
>
> An errata to the PCIe 2.1 spec provides guidance on limitations with 64-bit
> non-prefetchable BARs (since bridges have only 32-bit non-prefetchable
> ranges) stating that vendors can enable the prefetchable bit in BARs under
> certain circumstances to allow 64-bit allocation [2].
>
> The problem with that, is that vendors can't know apriori what hosts their
> products will be in, so can't just advertise prefetchable 64-bit BARs. What
> can be done, is system firmware can use the 64-bit prefetchable BAR in
> bridges, and assign a 64-bit non-prefetchable device BAR into that area,
> where it is safe to do so (following the guidance).
>
> At present, linux denies such allocations [3] and disables the BARs. It
> seems a practical solution to allow them if the firmware believes it is
> safe.

This particular message ([3]):

> pci 0002:01:00.0: BAR 0: [mem size 0x00002000 64bit] conflicts with PCI Bus
> 0002:00 [mem 0x10020000000-0x10027ffffff pref]

is misleading at best and likely a symptom of a bug.  We printed the
*size* of BAR 0, not an address, which means we haven't assigned space
for the BAR.  That means it should not conflict with anything.

We already do revert to firmware assignments in some situations when
Linux can't figure out how to assign things itself.  But apparently
not in *this* situation.

Without seeing the whole picture, it's hard for me to figure out
what's going on here.  Could you open a bug report at
http://bugzilla.kernel.org (category drivers/PCI) and attach a
complete dmesg and "lspci -vv" output?  Then we can look at what
firmware did and what Linux thought was wrong with it.

Bjorn

> --- [1]
>
> 0000:01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
>         Subsystem: Dell Device 1f26
>         Flags: bus master, fast devsel, latency 0, IRQ 24
>         Memory at e6000000 (64-bit, non-prefetchable) [size=32M]
>         Capabilities: [48] Power Management version 3
>         Capabilities: [50] Vital Product Data
>         Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
>         Capabilities: [a0] MSI-X: Enable+ Count=9 Masked-
>         Capabilities: [ac] Express Endpoint, MSI 00
>         Capabilities: [100] Device Serial Number d4-ae-52-ff-fe-ea-5c-e8
>         Capabilities: [110] Advanced Error Reporting
>         Capabilities: [150] Power Budgeting <?>
>         Capabilities: [160] Virtual Channel
>         Kernel driver in use: bnx2
>
> 0000:01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
>         Subsystem: Dell Device 1f26
>         Flags: bus master, fast devsel, latency 0, IRQ 25
>         Memory at e8000000 (64-bit, non-prefetchable) [size=32M]
>         Capabilities: [48] Power Management version 3
>         Capabilities: [50] Vital Product Data
>         Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
>         Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
>         Capabilities: [ac] Express Endpoint, MSI 00
>         Capabilities: [100] Device Serial Number d4-ae-52-ff-fe-ea-5c-ea
>         Capabilities: [110] Advanced Error Reporting
>         Capabilities: [150] Power Budgeting <?>
>         Capabilities: [160] Virtual Channel
>         Kernel driver in use: bnx2
>
> 0000:02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
>         Subsystem: Dell Device 1f26
>         Flags: bus master, fast devsel, latency 0, IRQ 28
>         Memory at ea000000 (64-bit, non-prefetchable) [size=32M]
>         Capabilities: [48] Power Management version 3
>         Capabilities: [50] Vital Product Data
>         Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
>         Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
>         Capabilities: [ac] Express Endpoint, MSI 00
>         Capabilities: [100] Device Serial Number d4-ae-52-ff-fe-ea-5c-ec
>         Capabilities: [110] Advanced Error Reporting
>         Capabilities: [150] Power Budgeting <?>
>         Capabilities: [160] Virtual Channel
>         Kernel driver in use: bnx2
>
> 0000:02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
>         Subsystem: Dell Device 1f26
>         Flags: bus master, fast devsel, latency 0, IRQ 29
>         Memory at ec000000 (64-bit, non-prefetchable) [size=32M]
>         Capabilities: [48] Power Management version 3
>         Capabilities: [50] Vital Product Data
>         Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
>         Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
>         Capabilities: [ac] Express Endpoint, MSI 00
>         Capabilities: [100] Device Serial Number d4-ae-52-ff-fe-ea-5c-ee
>         Capabilities: [110] Advanced Error Reporting
>         Capabilities: [150] Power Budgeting <?>
>         Capabilities: [160] Virtual Channel
>         Kernel driver in use: bnx2
>
> -- [2] p13
>
> https://www.pcisig.com/specifications/pciexpress/base2/PCIe_Base_r2.1_Errata_08Jun10.pdf
>
> -- [3]
>
> pci 0002:01:00.0: BAR 0: [mem size 0x00002000 64bit] conflicts with PCI Bus
> 0002:00 [mem 0x10020000000-0x10027ffffff pref]
> --
> Daniel J Blueman
> Principal Software Engineer, Numascale
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/