Message-ID: <CAB=NE6X0ySW=1udE3LCbo_eCjMfdrOMrceGsPvBZbDW+N-pLTw@mail.gmail.com>
Date:	Wed, 24 Jun 2015 15:29:22 -0700
From:	"Luis R. Rodriguez" <mcgrof@...e.com>
To:	Benjamin Herrenschmidt <benh@...nel.crashing.org>
Cc:	"Michael S. Tsirkin" <mst@...hat.com>,
	Bjorn Helgaas <bhelgaas@...gle.com>,
	Toshi Kani <toshi.kani@...com>,
	Andy Lutomirski <luto@...capital.net>,
	Juergen Gross <jgross@...e.com>,
	Tomi Valkeinen <tomi.valkeinen@...com>,
	Arnd Bergmann <arnd@...db.de>,
	"linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"xen-devel@...ts.xensource.com" <xen-devel@...ts.xensource.com>,
	linux-fbdev <linux-fbdev@...r.kernel.org>,
	Suresh Siddha <sbsiddha@...il.com>,
	Ingo Molnar <mingo@...e.hu>,
	Thomas Gleixner <tglx@...utronix.de>,
	Daniel Vetter <daniel.vetter@...ll.ch>,
	Dave Airlie <airlied@...hat.com>,
	Antonino Daplas <adaplas@...il.com>,
	Jean-Christophe Plagniol-Villard <plagnioj@...osoft.com>,
	Dave Hansen <dave.hansen@...ux.intel.com>,
	venkatesh.pallipadi@...el.com,
	Stefan Bader <stefan.bader@...onical.com>,
	Ville Syrjälä <syrjala@....fi>,
	Mel Gorman <mgorman@...e.de>, Vlastimil Babka <vbabka@...e.cz>,
	Borislav Petkov <bp@...e.de>, Davidlohr Bueso <dbueso@...e.de>,
	Konrad Rzeszutek Wilk <konrad.wilk@...cle.com>,
	Ville Syrjälä <ville.syrjala@...ux.intel.com>,
	David Vrabel <david.vrabel@...rix.com>,
	Jan Beulich <jbeulich@...e.com>,
	Roger Pau Monné <roger.pau@...rix.com>
Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants

On Wed, Jun 24, 2015 at 3:05 PM, Benjamin Herrenschmidt
<benh@...nel.crashing.org> wrote:
> On Wed, 2015-06-24 at 18:38 +0200, Luis R. Rodriguez wrote:
>> On Wed, Jun 24, 2015 at 08:42:23AM +1000, Benjamin Herrenschmidt wrote:
>> > On Fri, 2015-06-19 at 15:08 -0700, Luis R. Rodriguez wrote:
>> > > From: "Luis R. Rodriguez" <mcgrof@...e.com>
>> > >
>> > > PCI BARs tell us whether prefetching is safe, but they don't say anything
>> > > about write combining (WC).  WC changes ordering rules and allows writes to
>> > > be collapsed, so it's not safe in general to use it on a prefetchable
>> > > region.
>> >
>> > Well, the PCIe spec at least specifies that a prefetchable BAR also
>> > tolerates write merging...
>>
>> How can that be determined and can that be used as a full bullet proof hint
>> to enable wc ? And are you sure? :)
>
> Well, I"m sure the spec says that ;-) But it could be new to PCIe, I
> haven't checked legacy PCI.

OK cool so to be clear from what I gather you are suggesting (or not
and letting me make it) is that we might be able to enforce
write-merging on prefetchable areas, and if we can *ensure* we do this
then automatically enable write-combining behind the scenes?

>> Reason all this was stated was to be
>> apologetic over why we can't automate this behind the scenes. Otherwise
>> we could amend what you stated into the commit log to elaborate on our
>> technical apology. Let me know!
>
> At least on powerpc, for mmap of resource to userspace, we take off the
> garded bit in the PTE for prefetchable BARs. This has the effect
> architecturally of enabling both prefetch and write combine (ie. side
> effect)

That's pretty darn sexy.

> though afaik, the implementations probably don't actually
> prefetch. We've done that for years.

Neat!

> In fact we don't have a way to split the notions, it's either G or no G,
> which carries both meanings.

Interesting.

> Do you have example/case of a device having problems ?

Nope but at least what made me squint at this being a possible
"feature" was that in practice when reviewing all of the kernels
pending device drivers using MTRR (potential write-combine candidates)
I encountered a slew of them which had the architectural unfortunate
practice of combining PCI bars for MMIO and their respective
write-combined desirable area (framebuffer for video, PIO buffers for
infiniband, etc). Now, to me that read more as a practice for old
school devices when such things were likely still being evaluated,
more modern devices seem to adhere to sticking a full PCI bar with
write-combining or not. Did you not encounter such mismatch splits on
powerpc ? Was such possibility addressed?

If what you are implying here is applicable to the x86 world I'm all
for enabling this as we'd have less code to maintain but I'll note
that getting a clarification alone on that prefetchable !=
write-combining was in and of itself hard, I'd be surprised if we
could get full architectural buy-in to this as an immediate automatic
feature. Because of this and because PAT did have some errata as well,
I would not be surprised if some PCI bridges / devices would end up
finding corner cases, as such if we can really do what you're saying
and unless we can get some super sane certainty over it across the
board, I'd be inclined to leave such things as a part of a new API.
Maybe have some folks test using the new API for all calls and after
some sanity of testing / releases consider a full switch.

That is, unless of course you're sure all this is sane and would wager
all-in on it from the get-go.

 Luis
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists