Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D15F717855@SHSMSX101.ccr.corp.intel.com>
Date:	Fri, 20 Nov 2015 07:09:38 +0000
From:	"Tian, Kevin" <kevin.tian@...el.com>
To:	Alex Williamson <alex.williamson@...hat.com>
CC:	"Song, Jike" <jike.song@...el.com>,
	"xen-devel@...ts.xen.org" <xen-devel@...ts.xen.org>,
	"igvt-g@...1.01.org" <igvt-g@...1.01.org>,
	"intel-gfx@...ts.freedesktop.org" <intel-gfx@...ts.freedesktop.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"White, Michael L" <michael.l.white@...el.com>,
	"Dong, Eddie" <eddie.dong@...el.com>,
	"Li, Susie" <susie.li@...el.com>,
	"Cowperthwaite, David J" <david.j.cowperthwaite@...el.com>,
	"Reddy, Raghuveer" <raghuveer.reddy@...el.com>,
	"Zhu, Libo" <libo.zhu@...el.com>,
	"Zhou, Chao" <chao.zhou@...el.com>,
	"Wang, Hongbo" <hongbo.wang@...el.com>,
	"Lv, Zhiyuan" <zhiyuan.lv@...el.com>,
	qemu-devel <qemu-devel@...gnu.org>,
	Paolo Bonzini <pbonzini@...hat.com>,
	Gerd Hoffmann <kraxel@...hat.com>
Subject: RE: [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a
 Mediated Graphics Passthrough Solution from Intel

> From: Alex Williamson [mailto:alex.williamson@...hat.com]
> Sent: Friday, November 20, 2015 4:03 AM
> 
> > >
> > > The proposal is therefore that GPU vendors can expose vGPUs to
> > > userspace, and thus to QEMU, using the VFIO API.  For instance, vfio
> > > supports modular bus drivers and IOMMU drivers.  An intel-vfio-gvt-d
> > > module (or extension of i915) can register as a vfio bus driver, create
> > > a struct device per vGPU, create an IOMMU group for that device, and
> > > register that device with the vfio-core.  Since we don't rely on the
> > > system IOMMU for GVT-d vGPU assignment, another vGPU vendor driver (or
> > > extension of the same module) can register a "type1" compliant IOMMU
> > > driver into vfio-core.  From the perspective of QEMU then, all of the
> > > existing vfio-pci code is re-used, QEMU remains largely unaware of any
> > > specifics of the vGPU being assigned, and the only necessary change so
> > > far is how QEMU traverses sysfs to find the device and thus the IOMMU
> > > group leading to the vfio group.
> >
> > GVT-g requires to pin guest memory and query GPA->HPA information,
> > upon which shadow GTTs will be updated accordingly from (GMA->GPA)
> > to (GMA->HPA). So yes, here a dummy or simple "type1" compliant IOMMU
> > can be introduced just for this requirement.
> >
> > However there's one tricky point which I'm not sure whether overall
> > VFIO concept will be violated. GVT-g doesn't require system IOMMU
> > to function, however host system may enable system IOMMU just for
> > hardening purpose. This means two-level translations existing (GMA->
> > IOVA->HPA), so the dummy IOMMU driver has to request system IOMMU
> > driver to allocate IOVA for VMs and then setup IOVA->HPA mapping
> > in IOMMU page table. In this case, multiple VM's translations are
> > multiplexed in one IOMMU page table.
> >
> > We might need create some group/sub-group or parent/child concepts
> > among those IOMMUs for thorough permission control.
> 
> My thought here is that this is all abstracted through the vGPU IOMMU
> and device vfio backends.  It's the GPU driver itself, or some vfio
> extension of that driver, mediating access to the device and deciding
> when to configure GPU MMU mappings.  That driver has access to the GPA
> to HVA translations thanks to the type1 complaint IOMMU it implements
> and can pin pages as needed to create GPA to HPA mappings.  That should
> give it all the pieces it needs to fully setup mappings for the vGPU.
> Whether or not there's a system IOMMU is simply an exercise for that
> driver.  It needs to do a DMA mapping operation through the system IOMMU
> the same for a vGPU as if it was doing it for itself, because they are
> in fact one in the same.  The GMA to IOVA mapping seems like an internal
> detail.  I assume the IOVA is some sort of GPA, and the GMA is managed
> through mediation of the device.

Sorry I'm not familiar with VFIO internal. My original worry is that system 
IOMMU for GPU may be already claimed by another vfio driver (e.g. host kernel
wants to harden gfx driver from rest sub-systems, regardless of whether vGPU 
is created or not). In that case vGPU IOMMU driver shouldn't manage system
IOMMU directly.

btw, curious today how VFIO coordinates with system IOMMU driver regarding
to whether a IOMMU is used to control device assignment, or used for kernel 
hardening. Somehow two are conflicting since different address spaces are
concerned (GPA vs. IOVA)...

> 
> 
> > > There are a few areas where we know we'll need to extend the VFIO API to
> > > make this work, but it seems like they can all be done generically.  One
> > > is that PCI BARs are described through the VFIO API as regions and each
> > > region has a single flag describing whether mmap (ie. direct mapping) of
> > > that region is possible.  We expect that vGPUs likely need finer
> > > granularity, enabling some areas within a BAR to be trapped and fowarded
> > > as a read or write access for the vGPU-vfio-device module to emulate,
> > > while other regions, like framebuffers or texture regions, are directly
> > > mapped.  I have prototype code to enable this already.
> >
> > Yes in GVT-g one BAR resource might be partitioned among multiple vGPUs.
> > If VFIO can support such partial resource assignment, it'd be great. Similar
> > parent/child concept might also be required here, so any resource enumerated
> > on a vGPU shouldn't break limitations enforced on the physical device.
> 
> To be clear, I'm talking about partitioning of the BAR exposed to the
> guest.  Partitioning of the physical BAR would be managed by the vGPU
> vfio device driver.  For instance when the guest mmap's a section of the
> virtual BAR, the vGPU device driver would map that to a portion of the
> physical device BAR.
> 
> > One unique requirement for GVT-g here, though, is that vGPU device model
> > need to know guest BAR configuration for proper emulation (e.g. register
> > IO emulation handler to KVM). Similar is about guest MSI vector for virtual
> > interrupt injection. Not sure how this can be fit into common VFIO model.
> > Does VFIO allow vendor specific extension today?
> 
> As a vfio device driver all config accesses and interrupt configuration
> would be forwarded to you, so I don't see this being a problem.

Sure, nice to know that.

Thanks
Kevin

Powered by blists - more mailing lists