Message-ID: <CAB995MtCYAj-aWjf79+C8MiMM5U-PATn7DRoqsVW6V3G_ihO_A@mail.gmail.com>
Date:	Sat, 7 May 2016 21:20:19 +0530
From:	Nitin Saxena <nitin.lnx@...il.com>
To:	Alex Williamson <alex.williamson@...hat.com>,
	linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org
Subject: Fwd: VFIO MSIX Query

Hi,

I have specific questions related to the problem I am facing in using
VFIO for MSIX interrupt.

My userspace program running on x86 talks to NIC connected via PCIe
interface. Current program works fine in poll mode with no VF and
SRIOV requirement. I am changing my app from poll mode to interrupt
mode using VFIO NOIOMMU feature as I am interested in getting MSIX
interrupts only (For this I have ported VFIO-NOIOMMU feature patch
from kernel 4.5 to 3.10 SL7).  After applying the patch I can see
/dev/vfio/noiommu-X and I could assign my PCI device to
/dev/vfio/noiommu-0. Seems like kernel patch applied from 4.5 to 3.10
working fine. Also MSIX interrupts works fine with another kernel
application but I am facing issues in using MSIX interrupts with my
userspace program integrated with VFIO. I am running single process
only  with one MSIX interrupt requirement

Few Questions:

Q1) VFIO_GROUP_GET_DEVICE_FD ioctl call stops poll mode data traffic. Why?
I used my poll mode data path loop (where I am not waiting for MSIX)
with my vfio_init() API I found that if I call follow VFIO sequence in
vfio_init():
a) create container fd (b) get group_fd (c) VFIO_SET_IOMMU as NOIOMMU
==> then my poll mode data path works fine
but if I call a) create container fd (b) get group_fd (c)
VFIO_SET_IOMMU as NOIOMMU (d) VFIO_GROUP_GET_DEVICE_FD  ==> then my
poll mode data path does not work. I don't know why


Q2)Does MSIX bar/table mapping required for my case. Why?
Even if VFIO is not used in my userspace program, BAR0 and BAR1 was
already mmapped to process but not MSIX table. Do I need to mmap MSIX
table in my interrupt based data path as I saw this mmaping of MSIX
table in QEMU and dpdk both but I think these software must be using
VFIO_IOMMU_TYPE1 and not VFIO_NOIOMMU which is my case. So can you
confirm is it needed for me to mmap MSIX table.

Q3) Does VFIO_DEVICE_RESET required, when and why ?
My userspace program implicitly resets PCI hardware and then I call my
vfio_init() API and then start data path. So at what point I should
can VFIO_DEVICE_RESET?

Q4) Is there any Do's and Dont's document for VFIO so that I can
follow proper sequence.


Attaching kernel patch that I created for VFIO-NOIOMMU support from
kernel 4.5 to 3.10

Thanks in advance,
Nitin


On Wed, May 4, 2016 at 1:32 AM, Alex Williamson
<alex.williamson@...hat.com> wrote:
>
> Hi,
>
> On Tue, 3 May 2016 23:56:38 +0530
> Nitin Saxena <nitin.lnx@...il.com> wrote:
>
> > Hi,
> >
> > I am a newbie to VFIO framework and trying to use it for MSIX interrupt
> > handling in my userspace application. My userspace application is like
> > intel's dpdk.
> >
> > My query is why vfio kernel code does not support msix masking/ unmasking
> > i.e VFIO_SET_ACTION_TRIGGER is not implemented in kernel vfio code for MSIX
> > but its present for legacy INTX?
>
> I think you mean VFIO_IRQ_SET_ACTION_MASK and
> VFIO_IRQ_SET_ACTION_UNMASK are not implemented for MSI/X,
> VFIO_IRQ_SET_ACTION_TRIGGER is the only one that is implemented.  It
> mostly comes down to it hasn't been needed.  We would just need to
> verify that disable_irq() and enable_irq() do the right thing and
> enable the features through the existing VFIO API.  The reason we have
> these for INTx is that INTx is level triggered and we must mask the
> interrupt on the host while it's serviced by the user because the
> interrupt will continue to assert and we have no service guarantees
> from the user.  MSI/X is of course edge triggered, so doesn't have this
> problem.  The device would need to continue to re-assert the
> interrupt.  In cases like a QEMU user, we might want to continue to
> receive guest masked interrupts to QEMU for emulating the pending bits
> array, so even if we had VFIO masking support for these interrupts, QEMU
> probably would not use it.  If MSI/X masking is useful for you vs simply
> ignoring the eventfd, patches welcome.
>
> > Another question since MSIX interrupt masking is absent in kernel who is
> > supposed to mask/unmask msix interrupt? does userspace needs to mask/unmask
> > interrupt after getting notified by eventfd. Is there any usecase where
> > MSIX handling is done via VFIO framework?
>
> MSI/X is edge triggered and VFIO does not set the AUTOMASKED flag on
> them, so there is simply no masking of them at all aside from fully
> disabling them.  QEMU has full support for MSI/X through VFIO that you
> can use for reference.  Thanks,
>
> Alex

View attachment "vfio_noiommu_changes-3.10-123.0-SL7.0.patch" of type "text/x-patch" (14785 bytes)

Powered by blists - more mailing lists