Message-ID: <c4393deec4c14cb79ddf89daca5f78cc@svr-chch-ex1.atlnz.lc>
Date:   Thu, 4 Jul 2019 05:11:18 +0000
From:   Chris Packham <Chris.Packham@...iedtelesis.co.nz>
To:     "linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linuxppc-dev@...ts.ozlabs.org" <linuxppc-dev@...ts.ozlabs.org>
Subject: pci memory resources not being assigned to bridge

Hi,

I'm in the process of updating our products from a 4.4 based kernel to 
5.1 (and probably 5.2 since that release is imminent).

On one product which uses a Freescale/NXP P2041 CPU, IDT pcie bridge and 
Marvell switch chip[1].

Annoyingly the hardware has a reset line that holds the switch chip in 
reset but the bridge chip has a separate reset line that is not held 
high by default. So when the system starts up the initial scan sees the 
bridge and assigns resource to its end points. Later once the switch 
chip reset is released (either via a gpio-hog or manually) and the bus 
rescanned the switch devices are detected but the host side of the pci 
bridge isn't assigned any memory space so pci accesses trigger a master 
abort which on that SoC leads to a machine check exception.

[root@...uxbox ~]# uname -a
Linux linuxbox 5.2.0-rc7-at1+ #8 SMP Thu Jul 4 04:26:18 UTC 2019 ppc 
GNU/Linu[root@...uxbox ~]# lspci -t
-+-[2000:00]---00.0-[01-06]----00.0-[02-06]--+-02.0-[03]--
  |                                           +-03.0-[04]--
  |                                           +-04.0-[05]--
  |                                           \-05.0-[06]--
  \-[0000:00]-
[root@...uxbox ~]# lspci -v -s 2000:01:00
2000:01:00.0 Class 0604: Device 111d:803e (rev 0e)
         Flags: bus master, fast devsel, latency 0
         Bus: primary=01, secondary=02, subordinate=06, sec-latency=0
         I/O behind bridge: None
         Memory behind bridge: None
         Prefetchable memory behind bridge: None
         Capabilities: [40] Express Upstream Port, MSI 00
         Capabilities: [c0] Power Management version 3
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [200] Virtual Channel
         Kernel driver in use: pcieport

[root@...uxbox ~]# echo 472 >/sys/class/gpio/export
[root@...uxbox ~]# echo out >/sys/class/gpio/gpio472/direction
[root@...uxbox ~]# echo 0 >/sys/class/gpio/gpio472/value
[root@...uxbox ~]# echo 1 >/sys/bus/pci/rescan
pci 2000:03:00.0: [11ab:e097] type 00 class 0x058000
pci 2000:03:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 2000:03:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
pci 2000:03:00.0: Adding to iommu group 15
pci_bus 2000:03: busn_res: [bus 03] end is updated to 03
pci 2000:04:00.0: [11ab:e097] type 00 class 0x058000
pci 2000:04:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 2000:04:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
pci 2000:04:00.0: Adding to iommu group 15
pci_bus 2000:04: busn_res: [bus 04] end is updated to 04
pci 2000:05:00.0: [11ab:e097] type 00 class 0x058000
pci 2000:05:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 2000:05:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
pci 2000:05:00.0: Adding to iommu group 15
pci_bus 2000:05: busn_res: [bus 05] end is updated to 05
pci 2000:06:00.0: [11ab:e097] type 00 class 0x058000
pci 2000:06:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 2000:06:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
pci 2000:06:00.0: Adding to iommu group 15
pci_bus 2000:06: busn_res: [bus 06] end is updated to 06
pci_bus 2000:02: busn_res: [bus 02-06] end is updated to 06
pci_bus 2000:01: busn_res: [bus 01-06] end is updated to 06
pcieport 2000:00:00.0: BAR 9: no space for [mem size 0x00400000 64bit pref]
pcieport 2000:00:00.0: BAR 9: failed to assign [mem size 0x00400000 
64bit pref]
pcieport 2000:01:00.0: BAR 8: assigned [mem 0xc40000000-0xc4fffffff]
pcieport 2000:01:00.0: BAR 9: assigned [mem 0xc50000000-0xc503fffff 
64bit pref]
pci 2000:02:02.0: BAR 8: assigned [mem 0xc40000000-0xc43ffffff]
pci 2000:02:03.0: BAR 8: assigned [mem 0xc44000000-0xc47ffffff]
pci 2000:02:04.0: BAR 8: assigned [mem 0xc48000000-0xc4bffffff]
pci 2000:02:05.0: BAR 8: assigned [mem 0xc4c000000-0xc4fffffff]
pci 2000:02:02.0: BAR 9: assigned [mem 0xc50000000-0xc500fffff 64bit pref]
pci 2000:02:03.0: BAR 9: assigned [mem 0xc50100000-0xc501fffff 64bit pref]
pci 2000:02:04.0: BAR 9: assigned [mem 0xc50200000-0xc502fffff 64bit pref]
pci 2000:02:05.0: BAR 9: assigned [mem 0xc50300000-0xc503fffff 64bit pref]
pci 2000:03:00.0: BAR 2: assigned [mem 0xc40000000-0xc43ffffff 64bit]
pci 2000:03:00.0: BAR 0: assigned [mem 0xc50000000-0xc500fffff 64bit pref]
pci 2000:02:02.0: PCI bridge to [bus 03]
pci 2000:02:02.0:   bridge window [mem 0xc40000000-0xc43ffffff]
pci 2000:02:02.0:   bridge window [mem 0xc50000000-0xc500fffff 64bit pref]
pci 2000:04:00.0: BAR 2: assigned [mem 0xc44000000-0xc47ffffff 64bit]
pci 2000:04:00.0: BAR 0: assigned [mem 0xc50100000-0xc501fffff 64bit pref]
pci 2000:02:03.0: PCI bridge to [bus 04]
pci 2000:02:03.0:   bridge window [mem 0xc44000000-0xc47ffffff]
pci 2000:02:03.0:   bridge window [mem 0xc50100000-0xc501fffff 64bit pref]
pci 2000:05:00.0: BAR 2: assigned [mem 0xc48000000-0xc4bffffff 64bit]
pci 2000:05:00.0: BAR 0: assigned [mem 0xc50200000-0xc502fffff 64bit pref]
pci 2000:02:04.0: PCI bridge to [bus 05]
pci 2000:02:04.0:   bridge window [mem 0xc48000000-0xc4bffffff]
pci 2000:02:04.0:   bridge window [mem 0xc50200000-0xc502fffff 64bit pref]
pci 2000:06:00.0: BAR 2: assigned [mem 0xc4c000000-0xc4fffffff 64bit]
pci 2000:06:00.0: BAR 0: assigned [mem 0xc50300000-0xc503fffff 64bit pref]
pci 2000:02:05.0: PCI bridge to [bus 06]
pci 2000:02:05.0:   bridge window [mem 0xc4c000000-0xc4fffffff]
pci 2000:02:05.0:   bridge window [mem 0xc50300000-0xc503fffff 64bit pref]
[root@...uxbox ~]# lspci -t
-+-[2000:00]---00.0-[01-06]----00.0-[02-06]--+-02.0-[03]----00.0
  |                                           +-03.0-[04]----00.0
  |                                           +-04.0-[05]----00.0
  |                                           \-05.0-[06]----00.0
  \-[0000:00]-
[root@...uxbox ~]# lspci -v -s 2000:01:00
2000:01:00.0 Class 0604: Device 111d:803e (rev 0e)
         Flags: bus master, fast devsel, latency 0
         Bus: primary=01, secondary=02, subordinate=06, sec-latency=0
         I/O behind bridge: None
         Memory behind bridge: None
         Prefetchable memory behind bridge: None
         Capabilities: [40] Express Upstream Port, MSI 00
         Capabilities: [c0] Power Management version 3
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [200] Virtual Channel
         Kernel driver in use: pcieport

[root@...uxbox ~]# modprobe pptest
pptest: loading out-of-tree module taints kernel.
pptest: module license 'unspecified' taints kernel.
Disabling lock debugging due to kernel taint
pptest 2000:03:00.0: enabling device (0000 -> 0002)
pptest 2000:03:00.0: resource0 [mem 0xc50000000-0xc500fffff 64bit pref] 
mapped to c1980000
pptest 2000:03:00.0: resource2 [mem 0xc40000000-0xc43ffffff 64bit] 
mapped to c1b00000
Oops: Machine check, sig: 7 [#1]
BE PAGE_SIZE=4K SMP NR_CPUS=8 CoreNet Generic
Modules linked in: pptest(PO+)
CPU: 0 PID: 417 Comm: modprobe Tainted: P   M       O      5.2.0-rc7-at1+ #8
NIP:  b96fd10c LR: b96fd100 CTR: 80419df0
REGS: af19bf08 TRAP: 0204   Tainted: P   M       O       (5.2.0-rc7-at1+)
MSR:  00029002 <CE,EE,ME>  CR: 28228422  XER: 20000000
DEAR: c1b0004c ESR: 00000000
GPR00: b96fd100 ad107c68 ada1dd00 00000055 ae7303c4 ae739ba8 2dd8e000 
00021002
GPR08: 00000007 c1b0004c 2dd8e000 000001f8 000001f8 10030838 0000001d 
00000003
GPR16: ad107eb0 b96fd68e b96fd61c 0000fff1 ad107e00 00000398 b96bd000 
00000017
GPR24: b96fd488 00000002 b96fd308 b96fd308 b96fd280 adb2cd10 a9e55800 
a9e55878
NIP [b96fd10c] 0xb96fd10c
LR [b96fd100] 0xb96fd100
Call Trace:
[ad107c68] [b96fd100] 0xb96fd100 (unreliable)
[ad107c88] [803d0344] pci_device_probe+0xc4/0x150
[ad107ca8] [8043898c] really_probe+0x1ec/0x340
[ad107cd8] [8043909c] device_driver_attach+0x7c/0x90
[ad107cf8] [8043916c] __driver_attach+0xbc/0x120
[ad107d18] [804364b8] bus_for_each_dev+0x68/0xc0
[ad107d48] [80437ad8] bus_add_driver+0x178/0x250
[ad107d68] [80439888] driver_register+0x88/0x160
[ad107d78] [80002c00] do_one_initcall+0x40/0x210
[ad107dd8] [800c1218] do_init_module+0x60/0x1e8
[ad107df8] [800c0198] load_module+0x1c68/0x2230
[ad107ea8] [800c0940] sys_finit_module+0xa0/0xd0
[ad107f38] [800111d8] ret_from_syscall+0x0/0x38
--- interrupt: c00 at 0xff1ca78
     LR = 0x1000edf4
Instruction dump:
7f84e378 38a00000 480000f5 80fd0008 7fe3fb78 7f84e378 38a00002 38de01e0
480000dd 813d0008 3929004c 7c0004ac <7ca04c2c> 0c050000 4c00012c 3c80b970
---[ end trace baaeae7553646d27 ]---

Machine check in kernel mode.
Caused by (from MCSR=a000): Load Error Report
Guarded Load Error Report
Bus error

If I remove the bridge, release the switch reset then rescan the pci bus 
the memory space is assigned and things appear to work.

[root@...uxbox ~]# echo 1 
 >/sys/bus/pci/devices/2000:01:00.0/removepci_bus 2000:03: busn_res: 
[bus 03] is released
pci 2000:02:02.0: Removing from iommu group 15
pci_bus 2000:04: busn_res: [bus 04] is released
pci 2000:02:03.0: Removing from iommu group 15
pci_bus 2000:05: busn_res: [bus 05] is released
pci 2000:02:04.0: Removing from iommu group 15
pci_bus 2000:06: busn_res: [bus 06] is released
pci 2000:02:05.0: Removing from iommu group 15
pci_bus 2000:02: busn_res: [bus 02-06] is released
pci 2000:01:00.0: Removing from iommu group 15
[root@...uxbox ~]# echo 472 >/sys/class/gpio/export
[root@...uxbox ~]# echo out >/sys/class/gpio/gpio472/direction
[root@...uxbox ~]# echo 0 >/sys/class/gpio/gpio472/value
[root@...uxbox ~]# echo 1 >/sys/bus/pci/rescan

snip

[root@...uxbox ~]# lspci -t
-+-[2000:00]---00.0-[01-06]----00.0-[02-06]--+-02.0-[03]----00.0
  |                                           +-03.0-[04]----00.0
  |                                           +-04.0-[05]----00.0
  |                                           \-05.0-[06]----00.0
  \-[0000:00]-
[root@...uxbox ~]# lspci -v -s 2000:01:00
2000:01:00.0 Class 0604: Device 111d:803e (rev 0e)
         Flags: bus master, fast devsel, latency 0
         Bus: primary=01, secondary=02, subordinate=06, sec-latency=0
         I/O behind bridge: None
         Memory behind bridge: e0000000-efffffff [size=256M]
         Prefetchable memory behind bridge: 
00000000f0000000-00000000f03fffff [size=4M]
         Capabilities: [40] Express Upstream Port, MSI 00
         Capabilities: [c0] Power Management version 3
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [200] Virtual Channel
         Kernel driver in use: pcieport

[root@...uxbox ~]# modprobe pptest
pptest: loading out-of-tree module taints kernel.
pptest: module license 'unspecified' taints kernel.
Disabling lock debugging due to kernel taint
pptest 2000:03:00.0: enabling device (0000 -> 0002)
pptest 2000:03:00.0: resource0 [mem 0xc50000000-0xc500fffff 64bit pref] 
mapped to c1980000
pptest 2000:03:00.0: resource2 [mem 0xc40000000-0xc43ffffff 64bit] 
mapped to c1b00000
pptest 2000:03:00.0:  0x4c = e0973
pptest 2000:03:00.0:  0x50 = 11ab
pptest 2000:04:00.0: enabling device (0000 -> 0002)
pptest 2000:04:00.0: resource0 [mem 0xc50100000-0xc501fffff 64bit pref] 
mapped to c5b80000
pptest 2000:04:00.0: resource2 [mem 0xc44000000-0xc47ffffff 64bit] 
mapped to c5d00000
pptest 2000:04:00.0:  0x4c = e0973
pptest 2000:04:00.0:  0x50 = 11ab
pptest 2000:05:00.0: enabling device (0000 -> 0002)
pptest 2000:05:00.0: resource0 [mem 0xc50200000-0xc502fffff 64bit pref] 
mapped to c9d80000
pptest 2000:05:00.0: resource2 [mem 0xc48000000-0xc4bffffff 64bit] 
mapped to c9f00000
pptest 2000:05:00.0:  0x4c = e0973
pptest 2000:05:00.0:  0x50 = 11ab
pptest 2000:06:00.0: enabling device (0000 -> 0002)
pptest 2000:06:00.0: resource0 [mem 0xc50300000-0xc503fffff 64bit pref] 
mapped to cdf80000
pptest 2000:06:00.0: resource2 [mem 0xc4c000000-0xc4fffffff 64bit] 
mapped to ce100000
pptest 2000:06:00.0:  0x4c = e0973
pptest 2000:06:00.0:  0x50 = 11ab

Previously (under 4.4.x) we were able to get the memory space assigned 
by triggering a rescan (i.e. the first case above) but I've always seen 
the code that did that as a bit of a hack.

Why would the rescan in the second case assign memory that wasn't 
assigned by the first case? Is there any way I can avoid the removal 
dance, ideally by assigning reset gpios in the device tree?

Thanks,
Chris

--
[1] - Similar hardware to 
https://lore.kernel.org/linux-pci/403548ec3a7543b08ca32e47a1465e70@svr-chch-ex1.atlnz.lc/ 
but a different CPU and architecture.

Powered by blists - more mailing lists