linux-kernel - Mapping more than 3GB PCIe BAR memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAN6MAiPzqpOhfUxoJzerHBKz57J4ha3nUBtujYNpwoYbUKiQBA@mail.gmail.com>
Date:	Wed, 16 Sep 2015 16:18:47 +0530
From:	Anil Abraham <anil.abraham.in@...il.com>
To:	linux-kernel@...r.kernel.org
Subject: Mapping more than 3GB PCIe BAR memory

Hi, I have a use case where there are two PCIe devices connected to
two CPU sockets. Each of the device is exposing 1GB device memory over
their one of the BARs(BAR5/6). Since other two BARs(BAR 1/2/3/4) also
expose some amount of memory, the total memory requirement on the
upstream PCI-bridge goes upto 1.5GB(some round-offs happen at the
bridge) per socket. With this the linux kernel fails to allocate
region in the physical address space. I see the following errors in
the kernel bootup:

[    6.589964] pci 0000:84:06.0: BAR 14: can't assign mem (size 0x60000000)
[    6.611992] pci 0000:84:00.0: BAR 14: can't assign mem (size 0x100000)
[    6.633446] pci 0000:84:01.0: BAR 14: can't assign mem (size 0x100000)
[    6.654898] pci 0000:84:02.0: BAR 14: can't assign mem (size 0x100000)
[    6.676351] pci 0000:84:03.0: BAR 14: can't assign mem (size 0x100000)
[    6.697806] pci 0000:84:04.0: BAR 14: can't assign mem (size 0x100000)
[    6.719261] pci 0000:84:05.0: BAR 14: can't assign mem (size 0x100000)
[    6.740716] pci 0000:85:00.0: BAR 0: can't assign mem (size 0x10000)
...
[    7.087688] pci 0000:8b:00.0: BAR 4: can't assign mem (size 0x40000000)
[    7.109427] pci 0000:8b:00.0: BAR 0: can't assign mem (size 0x100000)
[    7.130599] pci 0000:8b:00.0: BAR 2: can't assign mem (size 0x2000)

I looked at the kernel code specially the files
drivers/pci/setup-res.c and pci_bus_alloc_resource() functions,
further down to e820.c. There I found this macro

#define MAX_GAP_END 0x100000000ull

Does this mean that PCI devices can be allocated addresses in 32-bit
space only even though the kernel is 64 bit and the device supports
64-bit registers. I can see that one of the reason could be to be
backward compatible with 32-bit devices. Is there a way we can
overcome this if at all, it is a limitation.

kernel version: 3.10.62

Kernel boot logs attached

Thanks in Advance for any responses.

-Anil

Download attachment "kernel-pci-boot.rtf" of type "application/rtf" (46500 bytes)