Message-ID: <4EA6D76C.5050505@numascale.com>
Date:	Tue, 25 Oct 2011 17:36:12 +0200
From:	Steffen Persvold <sp@...ascale.com>
To:	Bjorn Helgaas <bhelgaas@...gle.com>
CC:	Daniel J Blueman <daniel@...ascale-asia.com>,
	Ingo Molnar <mingo@...hat.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	H Peter Anvin <hpa@...or.com>, linux-kernel@...r.kernel.org,
	x86@...nel.org, Jesse Barnes <jbarnes@...tuousgeek.org>,
	'Arne Georg Gleditsch' <arne.gleditsch@...ascale.com>
Subject: Re: [PATCH 3/3] Add NumaChip quirk

On 10/25/2011 16:38, Bjorn Helgaas wrote:
> On Tue, Oct 18, 2011 at 2:22 AM, Daniel J Blueman
> <daniel@...ascale-asia.com>  wrote:
>> Add quirk for Numascale's NumaChip to prevent resource conflicts.
[]
> This feels like a band-aid ... what's the background here?  I can see
> that you're disabling PCI resources, and I can read that this avoids
> conflicts, but what's the underlying cause of the conflict?
>
> I wonder if there's a more generic problem that should be fixed
> differently.  Presumably the NumaChip designers put those BARs there
> for a reason, and often when we report "conflicts," it's really a clue
> that we're doing something wrong in host bridge discovery or in
> generic PCI.
>
> Can you post a complete dmesg log showing the conflict?
>

Hi Bjorn,

The issue is a bit complicated, but here's the story; NumaChip is a 
coherent NorthBridge device on AMD systems (i.e part of the coherent 
fabric) but the BIOS does *not* assign any resources to it, in fact the 
BIOS skips our device entirely leaving our BAR registers at HW init 
values (0x00000000). This is in fact by design (AMD AGESA code). This 
isn't really a big issue for us anyway because we have other means of 
reaching our CSR logic (not going into detail, but the other patches in 
this patchset would reveal how we do that).

Linux however, when scanning the PCI buses finds our device (because it 
is responding to config space requests) it thinks that we got a BAR0 
that starts at 0x00000000 which obviously isn't correct. In addition, in 
the bootloader that we've written for NumaChip systems (to bring them 
all together as a huge coherent system) we had to use the expansion rom 
config space register (F0x030) as kind of a "scratch register".

The end result is :

[    4.636297] pci 0000:00:1a.0: reg 10: [mem 0x00000000-0x000fffff]
[    4.640317] pci 0000:00:1a.0: reg 30: [mem 0x3fff0000-0x3fffffff pref]
[    4.644369] pci 0000:00:1a.1: [1b47:0602] type 0 class 0x000600

Neither of these resources are real, our device will not respond to 
requests to any of those windows.

The conflict we refer to in the patch is that since Linux thinks we have 
those windows assigned to us, we get conflicts later on with real devices :

[    5.887856] pnp 00:0e: disabling [mem 0x00000000-0x0009ffff] because 
it overlaps 0000:00:1a.0 BAR 0 [mem 0x00000000-0x000fffff]
[    5.899525] pnp 00:0e: disabling [mem 0x000c0000-0x000cffff] because 
it overlaps 0000:00:1a.0 BAR 0 [mem 0x00000000-0x000fffff]
[    5.911002] pnp 00:0e: disabling [mem 0x000e0000-0x000fffff] because 
it overlaps 0000:00:1a.0 BAR 0 [mem 0x00000000-0x000fffff]


I guess technically, the Linux PCI bus probing code should check the 
Command register (offset 0x4) to see if MemorySpace is enabled (which in 
our case it won't be) before checking the BAR registers.

 From my perspective however, it seemed easier and less intrusive to 
just hook in a quirk for our device since we do not use those BAR 
settings anyway in any form once Linux is booted.

I hope this clear things up a bit. If you have any other questions don't 
hesitate to ask.

Kind regards,
-- 
Steffen Persvold, Chief Architect NumaChip
Numascale AS - www.numascale.com
Tel: +47 92 49 25 54 Skype: spersvold
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists