linux-kernel - PCIe 32-bit MMIO exhaustion

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <54C8A10B.3070207@numascale.com>
Date:	Wed, 28 Jan 2015 16:42:51 +0800
From:	Daniel J Blueman <daniel@...ascale.com>
To:	Bjorn Helgaas <bhelgaas@...gle.com>,
	Ingo Molnar <mingo@...hat.com>,
	Jiang Liu <jiang.liu@...ux.intel.com>,
	H Peter Anvin <hpa@...or.com>,
	Thomas Gleixner <tglx@...utronix.de>
CC:	Linux Kernel <linux-kernel@...r.kernel.org>,
	Steffen Persvold <sp@...ascale.com>,
	"x86@...nel.org" <x86@...nel.org>
Subject: PCIe 32-bit MMIO exhaustion

With systems with a large number of PCI devices, we're seeing lack of 
32-bit MMIO space, eg one quad-port NetXtreme-2 adapter takes 128MB of 
space [1].

An errata to the PCIe 2.1 spec provides guidance on limitations with 
64-bit non-prefetchable BARs (since bridges have only 32-bit 
non-prefetchable ranges) stating that vendors can enable the 
prefetchable bit in BARs under certain circumstances to allow 64-bit 
allocation [2].

The problem with that, is that vendors can't know apriori what hosts 
their products will be in, so can't just advertise prefetchable 64-bit 
BARs. What can be done, is system firmware can use the 64-bit 
prefetchable BAR in bridges, and assign a 64-bit non-prefetchable device 
BAR into that area, where it is safe to do so (following the guidance).

At present, linux denies such allocations [3] and disables the BARs. It 
seems a practical solution to allow them if the firmware believes it is 
safe.

Is this plausible?

Thanks,
   Daniel

--- [1]

0000:01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II 
BCM5709 Gigabit Ethernet (rev 20)
	Subsystem: Dell Device 1f26
	Flags: bus master, fast devsel, latency 0, IRQ 24
	Memory at e6000000 (64-bit, non-prefetchable) [size=32M]
	Capabilities: [48] Power Management version 3
	Capabilities: [50] Vital Product Data
	Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
	Capabilities: [a0] MSI-X: Enable+ Count=9 Masked-
	Capabilities: [ac] Express Endpoint, MSI 00
	Capabilities: [100] Device Serial Number d4-ae-52-ff-fe-ea-5c-e8
	Capabilities: [110] Advanced Error Reporting
	Capabilities: [150] Power Budgeting <?>
	Capabilities: [160] Virtual Channel
	Kernel driver in use: bnx2

0000:01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II 
BCM5709 Gigabit Ethernet (rev 20)
	Subsystem: Dell Device 1f26
	Flags: bus master, fast devsel, latency 0, IRQ 25
	Memory at e8000000 (64-bit, non-prefetchable) [size=32M]
	Capabilities: [48] Power Management version 3
	Capabilities: [50] Vital Product Data
	Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
	Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
	Capabilities: [ac] Express Endpoint, MSI 00
	Capabilities: [100] Device Serial Number d4-ae-52-ff-fe-ea-5c-ea
	Capabilities: [110] Advanced Error Reporting
	Capabilities: [150] Power Budgeting <?>
	Capabilities: [160] Virtual Channel
	Kernel driver in use: bnx2

0000:02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II 
BCM5709 Gigabit Ethernet (rev 20)
	Subsystem: Dell Device 1f26
	Flags: bus master, fast devsel, latency 0, IRQ 28
	Memory at ea000000 (64-bit, non-prefetchable) [size=32M]
	Capabilities: [48] Power Management version 3
	Capabilities: [50] Vital Product Data
	Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
	Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
	Capabilities: [ac] Express Endpoint, MSI 00
	Capabilities: [100] Device Serial Number d4-ae-52-ff-fe-ea-5c-ec
	Capabilities: [110] Advanced Error Reporting
	Capabilities: [150] Power Budgeting <?>
	Capabilities: [160] Virtual Channel
	Kernel driver in use: bnx2

0000:02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II 
BCM5709 Gigabit Ethernet (rev 20)
	Subsystem: Dell Device 1f26
	Flags: bus master, fast devsel, latency 0, IRQ 29
	Memory at ec000000 (64-bit, non-prefetchable) [size=32M]
	Capabilities: [48] Power Management version 3
	Capabilities: [50] Vital Product Data
	Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
	Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
	Capabilities: [ac] Express Endpoint, MSI 00
	Capabilities: [100] Device Serial Number d4-ae-52-ff-fe-ea-5c-ee
	Capabilities: [110] Advanced Error Reporting
	Capabilities: [150] Power Budgeting <?>
	Capabilities: [160] Virtual Channel
	Kernel driver in use: bnx2

-- [2] p13

https://www.pcisig.com/specifications/pciexpress/base2/PCIe_Base_r2.1_Errata_08Jun10.pdf

-- [3]

pci 0002:01:00.0: BAR 0: [mem size 0x00002000 64bit] conflicts with PCI 
Bus 0002:00 [mem 0x10020000000-0x10027ffffff pref]
-- 
Daniel J Blueman
Principal Software Engineer, Numascale
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/