Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D15F717951@SHSMSX101.ccr.corp.intel.com>
Date:	Fri, 20 Nov 2015 08:10:21 +0000
From:	"Tian, Kevin" <kevin.tian@...el.com>
To:	Alex Williamson <alex.williamson@...hat.com>
CC:	"Song, Jike" <jike.song@...el.com>,
	"xen-devel@...ts.xen.org" <xen-devel@...ts.xen.org>,
	"igvt-g@...1.01.org" <igvt-g@...1.01.org>,
	"intel-gfx@...ts.freedesktop.org" <intel-gfx@...ts.freedesktop.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"White, Michael L" <michael.l.white@...el.com>,
	"Dong, Eddie" <eddie.dong@...el.com>,
	"Li, Susie" <susie.li@...el.com>,
	"Cowperthwaite, David J" <david.j.cowperthwaite@...el.com>,
	"Reddy, Raghuveer" <raghuveer.reddy@...el.com>,
	"Zhu, Libo" <libo.zhu@...el.com>,
	"Zhou, Chao" <chao.zhou@...el.com>,
	"Wang, Hongbo" <hongbo.wang@...el.com>,
	"Lv, Zhiyuan" <zhiyuan.lv@...el.com>,
	qemu-devel <qemu-devel@...gnu.org>,
	Paolo Bonzini <pbonzini@...hat.com>,
	Gerd Hoffmann <kraxel@...hat.com>
Subject: RE: [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a
 Mediated Graphics Passthrough Solution from Intel

> From: Tian, Kevin
> Sent: Friday, November 20, 2015 3:10 PM

> > > >
> > > > The proposal is therefore that GPU vendors can expose vGPUs to
> > > > userspace, and thus to QEMU, using the VFIO API.  For instance, vfio
> > > > supports modular bus drivers and IOMMU drivers.  An intel-vfio-gvt-d
> > > > module (or extension of i915) can register as a vfio bus driver, create
> > > > a struct device per vGPU, create an IOMMU group for that device, and
> > > > register that device with the vfio-core.  Since we don't rely on the
> > > > system IOMMU for GVT-d vGPU assignment, another vGPU vendor driver (or
> > > > extension of the same module) can register a "type1" compliant IOMMU
> > > > driver into vfio-core.  From the perspective of QEMU then, all of the
> > > > existing vfio-pci code is re-used, QEMU remains largely unaware of any
> > > > specifics of the vGPU being assigned, and the only necessary change so
> > > > far is how QEMU traverses sysfs to find the device and thus the IOMMU
> > > > group leading to the vfio group.
> > >
> > > GVT-g requires to pin guest memory and query GPA->HPA information,
> > > upon which shadow GTTs will be updated accordingly from (GMA->GPA)
> > > to (GMA->HPA). So yes, here a dummy or simple "type1" compliant IOMMU
> > > can be introduced just for this requirement.
> > >
> > > However there's one tricky point which I'm not sure whether overall
> > > VFIO concept will be violated. GVT-g doesn't require system IOMMU
> > > to function, however host system may enable system IOMMU just for
> > > hardening purpose. This means two-level translations existing (GMA->
> > > IOVA->HPA), so the dummy IOMMU driver has to request system IOMMU
> > > driver to allocate IOVA for VMs and then setup IOVA->HPA mapping
> > > in IOMMU page table. In this case, multiple VM's translations are
> > > multiplexed in one IOMMU page table.
> > >
> > > We might need create some group/sub-group or parent/child concepts
> > > among those IOMMUs for thorough permission control.
> >
> > My thought here is that this is all abstracted through the vGPU IOMMU
> > and device vfio backends.  It's the GPU driver itself, or some vfio
> > extension of that driver, mediating access to the device and deciding
> > when to configure GPU MMU mappings.  That driver has access to the GPA
> > to HVA translations thanks to the type1 complaint IOMMU it implements
> > and can pin pages as needed to create GPA to HPA mappings.  That should
> > give it all the pieces it needs to fully setup mappings for the vGPU.
> > Whether or not there's a system IOMMU is simply an exercise for that
> > driver.  It needs to do a DMA mapping operation through the system IOMMU
> > the same for a vGPU as if it was doing it for itself, because they are
> > in fact one in the same.  The GMA to IOVA mapping seems like an internal
> > detail.  I assume the IOVA is some sort of GPA, and the GMA is managed
> > through mediation of the device.
> 
> Sorry I'm not familiar with VFIO internal. My original worry is that system
> IOMMU for GPU may be already claimed by another vfio driver (e.g. host kernel
> wants to harden gfx driver from rest sub-systems, regardless of whether vGPU
> is created or not). In that case vGPU IOMMU driver shouldn't manage system
> IOMMU directly.
> 
> btw, curious today how VFIO coordinates with system IOMMU driver regarding
> to whether a IOMMU is used to control device assignment, or used for kernel
> hardening. Somehow two are conflicting since different address spaces are
> concerned (GPA vs. IOVA)...
> 

Here is a more concrete example:

KVMGT doesn't require IOMMU. All DMA targets are already replaced with 
HPA thru shadow GTT. So DMA requests from GPU all contain HPAs.

When IOMMU is enabled, one simple approach is to have vGPU IOMMU
driver configure system IOMMU with identity mapping (HPA->HPA). We 
can't use (GPA->HPA) since GPAs from multiple VMs are conflicting. 

However, we still have host gfx driver running. When IOMMU is enabled, 
dma_alloc_*** will return IOVA (drvers/iommu/iova.c) in host gfx driver,
which will have IOVA->HPA programmed to system IOMMU.

One IOMMU device entry can only translate one address space, so here
comes a conflict (HPA->HPA vs. IOVA->HPA). To solve this, vGPU IOMMU
driver needs to allocate IOVA from iova.c for each VM w/ vGPU assigned,
and then KVMGT will program IOVA in shadow GTT accordingly. It adds
one additional mapping layer (GPA->IOVA->HPA). In this way two 
requirements can be unified together since only IOVA->HPA mapping 
needs to be built.

So unlike existing type1 IOMMU driver which controls IOMMU alone, vGPU 
IOMMU driver needs to cooperate with other agent (iova.c here) to
co-manage system IOMMU. This may not impact existing VFIO framework.
Just want to highlight additional work here when implementing the vGPU
IOMMU driver.

Thanks
Kevin
 


Thanks
Kevin

Powered by blists - more mailing lists