linux-kernel - RE: [RFC 0/2] VFIO: Add virtual MSI doorbell support.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CY1PR0301MB127660D71378F009918DDDD5908D0@CY1PR0301MB1276.namprd03.prod.outlook.com>
Date:	Tue, 28 Jul 2015 17:23:45 +0000
From:	Bhushan Bharat <Bharat.Bhushan@...escale.com>
To:	Alex Williamson <alex.williamson@...hat.com>,
	Pranavkumar Sawargaonkar <pranavkumar@...aro.org>
CC:	"kvm@...r.kernel.org" <kvm@...r.kernel.org>,
	"kvmarm@...ts.cs.columbia.edu" <kvmarm@...ts.cs.columbia.edu>,
	"linux-arm-kernel@...ts.infradead.org" 
	<linux-arm-kernel@...ts.infradead.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"christoffer.dall@...aro.org" <christoffer.dall@...aro.org>,
	"marc.zyngier@....com" <marc.zyngier@....com>,
	"will.deacon@....com" <will.deacon@....com>,
	"bhelgaas@...gle.com" <bhelgaas@...gle.com>,
	"arnd@...db.de" <arnd@...db.de>,
	"rob.herring@...aro.org" <rob.herring@...aro.org>,
	"eric.auger@...aro.org" <eric.auger@...aro.org>,
	"patches@....com" <patches@....com>,
	Stuart Yoder <stuart.yoder@...escale.com>
Subject: RE: [RFC 0/2] VFIO: Add virtual MSI doorbell support.

Hi Alex,

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@...hat.com]
> Sent: Tuesday, July 28, 2015 9:52 PM
> To: Pranavkumar Sawargaonkar
> Cc: kvm@...r.kernel.org; kvmarm@...ts.cs.columbia.edu; linux-arm-
> kernel@...ts.infradead.org; linux-kernel@...r.kernel.org;
> christoffer.dall@...aro.org; marc.zyngier@....com; will.deacon@....com;
> bhelgaas@...gle.com; arnd@...db.de; rob.herring@...aro.org;
> eric.auger@...aro.org; patches@....com; Bhushan Bharat-R65777; Yoder
> Stuart-B08248
> Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support.
> 
> On Fri, 2015-07-24 at 14:33 +0530, Pranavkumar Sawargaonkar wrote:
> > In current VFIO MSI/MSI-X implementation, linux host kernel allocates
> > MSI/MSI-X vectors when userspace requests through vfio ioctls.
> > Vfio creates irqfd mappings to notify MSI/MSI-X interrupts to the
> > userspace when raised.
> > Guest OS will see emulated MSI/MSI-X controller and receives an
> > interrupt when kernel notifies the same via irqfd.
> >
> > Host kernel allocates MSI/MSI-X using standard linux routines like
> > pci_enable_msix_range() and pci_enable_msi_range().
> > These routines along with requset_irq() in host kernel sets up
> > MSI/MSI-X vectors with Physical MSI/MSI-X addresses provided by
> > interrupt controller driver in host kernel.
> >
> > This means when a device is assigned with the guest OS, MSI/MSI-X
> > addresses present in PCIe EP are the PAs programmed by the host linux
> kernel.
> >
> > In x86 MSI/MSI-X physical address range is reserved and iommu is aware
> > about these addreses and transalation is bypassed for these address range.
> >
> > Unlike x86, ARM/ARM64 does not reserve MSI/MSI-X Physical address
> > range and all the transactions including MSI go through iommu/smmu
> without bypass.
> > This requires extending current vfio MSI layer with additional
> > functionality for ARM/ARM64 by 1. Programing IOVA (referred as a MSI
> > virtual doorbell address)
> >    in device's MSI vector as a MSI address.
> >    This IOVA will be provided by the userspace based on the
> >    MSI/MSI-X addresses reserved for the guest.
> > 2. Create an IOMMU mapping between this IOVA and
> >    Physical address (PA) assigned to the MSI vector.
> >
> > This RFC is proposing a solution for MSI/MSI-X passthrough for
> ARM/ARM64.
> 
> 
> Hi Pranavkumar,
> 
> Freescale has the same, or very similar, need, so any solution in this space
> will need to work for both ARM and powerpc.  I'm not a big fan of this
> approach as it seems to require the user to configure MSI/X via ioctl and then
> call a separate ioctl mapping the doorbells.  That's more code for the user,
> more code to get wrong and potentially a gap between configuring MSI/X
> and enabling mappings where we could see IOMMU faults.
> 
> If we know that doorbell mappings are required, why can't we set aside a
> bank of IOVA space and have them mapped automatically as MSI/X is being
> configured?  Then the user's need for special knowledge and handling of this
> case is limited to setup.  The IOVA space will be mapped and used as needed,
> we only need the user to specify the IOVA space reserved for this.  Thanks,

We probably need a mix of both to support Freescale PowerPC and ARM based machines.
In this mix mode kernel vfio driver will reserve some IOVA for mapping MSI page/s. If any other iova mapping will overlap with this then it will return error and user-space. Ideally this should be choosen in such a way that it never overlap, which is easy on some systems but can be tricky on some other system like Freescale PowerPC. This is not sufficient for at-least Freescale PowerPC based SOC. This is because of hardware limitation, where we need to fit this reserved iova address within aperture decided by user-space. So if we allow user-space to change this reserved iova address to a value decided by user-spece itself then we can support both ARM/PowerPC based solutions.

I have some implementation ready/tested with this approach and if this approach looks good then I can submit a RFC patch.

Thanks
-Bharat
> 
> Alex