linux-kernel - Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in non-coherent domains

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20240522122939.GT20229@nvidia.com>
Date: Wed, 22 May 2024 09:29:39 -0300
From: Jason Gunthorpe <jgg@...dia.com>
To: "Tian, Kevin" <kevin.tian@...el.com>
Cc: Alex Williamson <alex.williamson@...hat.com>,
	"Vetter, Daniel" <daniel.vetter@...el.com>,
	"Zhao, Yan Y" <yan.y.zhao@...el.com>,
	"kvm@...r.kernel.org" <kvm@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"x86@...nel.org" <x86@...nel.org>,
	"iommu@...ts.linux.dev" <iommu@...ts.linux.dev>,
	"pbonzini@...hat.com" <pbonzini@...hat.com>,
	"seanjc@...gle.com" <seanjc@...gle.com>,
	"dave.hansen@...ux.intel.com" <dave.hansen@...ux.intel.com>,
	"luto@...nel.org" <luto@...nel.org>,
	"peterz@...radead.org" <peterz@...radead.org>,
	"tglx@...utronix.de" <tglx@...utronix.de>,
	"mingo@...hat.com" <mingo@...hat.com>,
	"bp@...en8.de" <bp@...en8.de>, "hpa@...or.com" <hpa@...or.com>,
	"corbet@....net" <corbet@....net>,
	"joro@...tes.org" <joro@...tes.org>,
	"will@...nel.org" <will@...nel.org>,
	"robin.murphy@....com" <robin.murphy@....com>,
	"baolu.lu@...ux.intel.com" <baolu.lu@...ux.intel.com>,
	"Liu, Yi L" <yi.l.liu@...el.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains

On Wed, May 22, 2024 at 06:24:14AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@...dia.com>
> > Sent: Wednesday, May 22, 2024 2:38 AM
> > 
> > On Tue, May 21, 2024 at 12:19:45PM -0600, Alex Williamson wrote:
> > > > I'm OK with this. If devices are insecure then they need quirks in
> > > > vfio to disclose their problems, we shouldn't punish everyone who
> > > > followed the spec because of some bad actors.
> > > >
> > > > But more broadly in a security engineered environment we can trust the
> > > > no-snoop bit to work properly.
> > >
> > >  The spec has an interesting requirement on devices sending no-snoop
> > >  transactions anyway (regarding PCI_EXP_DEVCTL_NOSNOOP_EN):
> > >
> > >  "Even when this bit is Set, a Function is only permitted to Set the No
> > >   Snoop attribute on a transaction when it can guarantee that the
> > >   address of the transaction is not stored in any cache in the system."
> > >
> > > I wouldn't think the function itself has such visibility and it would
> > > leave the problem of reestablishing coherency to the driver, but am I
> > > overlooking something that implicitly makes this safe?
> > 
> > I think it is just bad spec language! People are clearly using
> > no-snoop on cachable memory today. The authors must have had some
> > other usage in mind than what the industry actually did.
> 
> sure no-snoop can be used on cacheable memory but then the driver
> needs to flush the cache before triggering the no-snoop DMA so it
> still meets the spec "the address of the transaction is not stored
> in any cache in the system".

Flush does not mean evict.. The way I read the above it is trying to
say the driver must map all the memory non-cachable to ensure it never
gets pulled into a cache in the first place.

> > Maybe not entire, but as an additional step to reduce the cost of
> > this. ARM would like this for instance.
> 
> I searched PCI_EXP_DEVCTL_NOSNOOP_EN but surprisingly it's not
> touched by i915 driver. sort of suggesting that Intel GPU doesn't follow
> the spec to honor that bit...

Or the BIOS turns it on and the OS just leaves it..

> I'm fine to do a special check in the attach path to enable the flush
> only for Intel GPU.

We already effectively do this already by checking the domain
capabilities. Only the Intel GPU will have a non-coherent domain.

> or alternatively could ARM SMMU driver implement
> @enforce_cache_coherency by disabling PCI nosnoop cap when
> the SMMU itself cannot force snoop? Then VFIO/IOMMUFD could
> still check enforce_cache_coherency generally to apply the cache
> flush trick... 😊

I like this a lot less than having vfio understand it..

Jason