linux-kernel - Re: [PATCH] intel-iommu: Default to non-coherent for domains unattached to iommus

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20111112010850.GT14486@sequoia.sous-sol.org>
Date:	Fri, 11 Nov 2011 17:08:50 -0800
From:	Chris Wright <chrisw@...s-sol.org>
To:	David Woodhouse <dwmw2@...radead.org>
Cc:	Chris Wright <chrisw@...s-sol.org>,
	Roland Dreier <roland@...estorage.com>,
	Alex Williamson <alex.williamson@...hat.com>,
	iommu@...ts.linux-foundation.org, linux-pci@...r.kernel.org,
	linux-kernel@...r.kernel.org, ddutile@...hat.com
Subject: Re: [PATCH] intel-iommu: Default to non-coherent for domains
 unattached to iommus

* David Woodhouse (dwmw2@...radead.org) wrote:
> On Fri, 2011-11-11 at 16:51 -0800, Chris Wright wrote:
> > * Roland Dreier (roland@...estorage.com) wrote:
> > > On Fri, Nov 11, 2011 at 4:37 PM, David Woodhouse <dwmw2@...radead.org> wrote:
> > > > This brain-damage only affects the first chipsets
> > > > from before we worked out that cache incoherency was a *really* f*cking
> > > > stupid idea, doesn't it?
> > > 
> > > As we talked about at KS, I have some Westmere EP (ie latest
> > > and greatest server platform) systems where the BIOS exposes
> > > an option that allows choosing VT-d coherency on or off, and
> > > defaults it to "off".
> > 
> > That's just more brain damage AFAICT.  Esp. if you do performance
> > testing (and choose not to use passthrough mode)... have and it's
> > quite measurable.  I switched default to on long time ago, w/out
> > issue.
> > 
> > > What is the "official" Intel line on coherency with Westmere and
> > > Tylersburg -- because as I also mentioned, I was seeing some
> > > problems with VT-d and the default "coherency off" setting that
> > > looked like the IOMMU HW is getting stale PTEs (ie a missing
> > > or not working cache flush).
> > 
> > That sounds like sw bugs more than official recommendation issue.
> 
> Although the cache-flushing has been tested on the original chipsets
> fairly well, and it's one of the parts I've mostly rewritten when doing
> performance work since I inherited the code, so that might not be my
> first suspicion.

All the stale PTE issues I've encountered in the past have turned into
fixed sw bugs (perhaps it's since been fixed?).  Also, I thought with
Coherency On/Off it's only effecting the use of clflush, not IOTLB or
Context Entry cache flushing (invalidations).

On a slightly separate, but performance related note...have you ever
tried using the hw queue?  Currently we only have a sw queue, but the
submission path for invalidations doesn't really queue (unless I missed
it).  It seems to pull from the software queue and submit/wait,
submit/wait...Seems simple enough to submit the whole queue and then
issue the wait.

This would be a huge win if we ever have an emulated IOMMU.  We could
make the sw queue bigger, and allocate more than a single page for the
hw queue.  We'd only exit on running the queue rather than every
invalidation.

> I would be more inclined to suspect that there's some
> chipset buffering that we aren't correctly flushing (which might in
> itself be a hardware issue, since the way to flush the cache is supposed
> to be well-defined).

Roland, have you tried switching BIOS to Coherency On and can do you
ever see stale PTEs?

thanks,
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/