Message-ID: <5283004.YemU4lhtKD@pcbe13110.cern.ch>
Date:	Tue, 8 Jul 2014 09:15:36 +0200
From:	Federico Vaga <federico.vaga@...n.ch>
To:	Bjorn Helgaas <bhelgaas@...gle.com>
CC:	"linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	<michel.arruat@...n.ch>
Subject: Re: PCIe bus enumeration

(I'm changing my email address to the work one. Initially it was just 
my personal curiosity but now you are helping me with my work, so I 
think is correct in this way)

> > So, It looks like that some BIOS disable the bridge when there is
> > nothing behind it. Why? Power save? :/
> 
> Could be power savings, or possibly to conserve bus numbers, which
> are a limited resource.

what is the maximum number of buses?

> >> If you can get to an EFI shell on this box, you might be able to
> >> confirm this with the "pci" command.  Booting Linux with
> >> "pci=earlydump" is similar in that it dumps PCI config space
> >> before
> >> we change anything.
> > 
> > yes I confirm, the bridge are not there if I don't plug the card.
> > 
> >> To solve this problem, I think you need slot information even
> >> when
> >> there's no hotplug.  This has been raised before [1, 2], and I
> >> think it's a good idea, but nobody has implemented it yet.
> > 
> > Yes, but if the BIOS disable the bridge there is nothing we can
> > do.
> 
> Well, it's true that it's hard to get constant *bus numbers*, but
> it's never really been a good idea to rely on those, because
> they're assigned at the discretion of the OS, and there are reasons
> why the OS might want to reallocate them, e.g., to accommodate a
> deep hot-plugged hierarchy.  If you shift focus to *slot numbers*,
> then I think there's a lot more we can do.

At this point I'm a little bit confused about the definition "slot 
numbers" :) You mean the 22, 25, ...

> >> Another curious thing is that you refer to "slot 10", but there's
> >> no obvious connection between that and the "slot 21" in the PCIe
> >> capability of the Root Port leading to that slot.  But I guess
> >> you said the slots are in a backplane (they're not an integral
> >> part of the motherboard).  In that case, there's no way for the
> >> motherboard to know what the labels on the backplane are.
> > 
> > It is written on the backplane. I said slot 10 because I'm
> > counting
> > the available slot, but on the backplane they are 22, 25, and
> > other
> > no-consecutive numbers.
> 
> The 22, 25, etc., are in the same range as the slot numbers in the
> PCIe Slot Capabilities registers, so maybe the backplane is
> constructed to make this possible.  The external PCIe chassis I'm
> familiar with have one fast link on a cable leading to the box, with
> a PCIe switch inside the box.  The upstream port is connected to
> the incoming link, and there's a downstream port connected to each
> slot. In this case, the slot numbers in the downstream ports' Slot
> Capabilities registers can be made to match the silkscreen labels
> on the board since everything is fixed by the hardware.
> 
> Your backplane sounds a little different (you have Ports on the root
> bus leading directly to slots in the backplane, so I assume those
> Ports are on the motherboard, not the backplane), but maybe the
> motherboard & backplane are designed as a unit so the Port slot
> numbers could match the backplane.

Yes, the backplane is almost "empty". Except for the 9 PCIe backplane 
which has PCI bridges on it. At the moment I cannot check physically 
this kind of backplane, but from the lspci output I understand that 
there is a bridge on the backplane because the motherboard is the 
same.

> 
> > If I use `biosdecode` I can get that information, but only for the
> > "first level" of bridges. On some backplane I have PCI bridges
> > behind bridges, and in this case biosdecode doesn't help: it just
> > tell me about the bridge on the motherboard.
> 
> What specific biosdecode information are you using? 

I was looking at the "PCI interrupt routing", but it seems that it 
returns only information about the last bridge in the interrupt's 
routing. Here an example with a different backplane (9 PCIe). 

It seems fine for backplane without PCI Bridge on the backplane.

I attached two files, one for each type of backplane.


Maybe I'm just misunderstanding the output of biosdecode. I didn't 
find an explanation of its output: I'm guessing the meaning.

> There's a fair
> amount of stuff in the PCI-to-PCI bridge spec about slot and chassis
> numbering, including some about expansion chassis.  I doubt that
> Linux implements all that, so there's probably room for a lot of
> improvement.  I attached your lspci output to the bugzilla
> (https://bugzilla.kernel.org/show_bug.cgi?id=72681).  Maybe you
> could attach the biosdecode info there, too, and we could see if
> there's a way we can make this easier.

ok

-- 
Federico Vaga
View attachment "biosdecode-1-level-bridges" of type "text/plain" (1871 bytes)

View attachment "biosdecode-n-level-bridges" of type "text/plain" (1998 bytes)

Powered by blists - more mailing lists