Date:   Thu, 12 Sep 2019 18:32:01 -0700
From:   Megha Dey <megha.dey@...ux.intel.com>
To:     linux-kernel@...r.kernel.org, x86@...nel.org,
        linux-pci@...r.kernel.org, maz@...nel.org, bhelgaas@...gle.com,
        rafael@...nel.org, gregkh@...uxfoundation.org, tglx@...utronix.de,
        hpa@...or.com, alex.williamson@...hat.com, jgg@...lanox.com
Cc:     ashok.raj@...el.com, megha.dey@...el.com, jacob.jun.pan@...el.com,
        Megha Dey <megha.dey@...ux.intel.com>
Subject: [RFC V1 0/7] Add support for a new IMS interrupt mechanism

Currently, MSI (Message signaled interrupts) and MSI-X are the de facto
standard for device interrupt mechanism. MSI-X supports up to 2048
interrupts per device while MSI supports 32, which seems more than enough
for current devices. However, the introduction of SIOV (Scalable IO
virtualization) shifts the creation of assignable virtual devices from
hardware to a more software assisted approach. This flexible composition
of direct assignable devices, a.k.a. assignable device interfaces (ADIs)
unchains hardware from costly PCI standard. Under SIOV, device resource
can now be mapped directly to a guest or other user space drivers for
near native DMA performance. To complete functionality of ADIs, a matching
interrupt resource must also be introduced which will be scalable.

Interrupt message storage (IMS) is conceived  as a scalable albeit device
specific interrupt mechanism to meet such a demand. With IMS, there is
theoretically no upper bound on the number of interrupts which a device
can support. The size and location of IMS is device-specific; some devices
may implement IMS as on-device storage which are memory-mapped, others may
opt to implement IMS in system memory. IMS stores each interrupt message as
a DWORD size data payload and a 64-bit address(same as MSI-X). Access to
the IMS is through the host driver due to the non-architectural nature of
device IMS unlike the architectural MSI-X table which are accessed through
PCI drivers.

In this patchset, we introduce generic IMS APIs that fits the Linux IRQ
subsystem, supports IMS IRQ chip and domains that can be used by drivers
which are capable of generating IMS interrupts.

The IMS has been introduced as part of Intel's Scalable I/O virtualization
specification:
https://software.intel.com/en-us/download/intel-scalable-io-virtualization-technical-specification

This patchset is based on Linux 5.3-rc8.

Currently there is no device out in the market which supports SIOV (Hence no
device supports IMS).

This series is a basic patchset to get the ball rolling and receive some
inital comments. As per my discussion with Marc Zyngier and Thomas Gleixner
at the Linux Plumbers, I need to do the following:
1. Since a device can support MSI-X and IMS simultaneously, ensure proper
   locking mechanism for the 'msi_list' in the device structure.
2. Introduce dynamic allocation of IMS vectors perhaps by using a group ID
3. IMS support of a device needs to be discoverable. A bit in the vendor
   specific capability in the PCI config is to be added rather than getting
   this information from each device driver.

Jason Gunthorpe of Mellanox technologies is looking to do something similar
on ARM platforms and was wondering why IMS is x86 sepcific. Perhaps we can
use this thread to discuss further on this. 

Megha Dey (7):
  genirq/msi: Differentiate between various MSI based interrupts
  drivers/base: Introduce callbacks for IMS interrupt domain
  x86/ims: Add support for a new IMS irq domain
  irq_remapping: New interfaces to support IMS irqdomain
  x86/ims: Introduce x86_ims_ops
  ims-msi: Add APIs to allocate/free IMS interrupts
  ims: Add the set_desc callback

 arch/mips/pci/msi-xlp.c              |   2 +-
 arch/s390/pci/pci_irq.c              |   2 +-
 arch/x86/include/asm/irq_remapping.h |  13 ++
 arch/x86/include/asm/msi.h           |   4 +
 arch/x86/include/asm/pci.h           |   4 +
 arch/x86/include/asm/x86_init.h      |  10 +
 arch/x86/kernel/apic/Makefile        |   1 +
 arch/x86/kernel/apic/ims.c           | 118 ++++++++++++
 arch/x86/kernel/apic/msi.c           |   6 +-
 arch/x86/kernel/x86_init.c           |  23 +++
 arch/x86/pci/xen.c                   |   2 +-
 drivers/base/Kconfig                 |   7 +
 drivers/base/Makefile                |   1 +
 drivers/base/ims-msi.c               | 353 +++++++++++++++++++++++++++++++++++
 drivers/iommu/intel_irq_remapping.c  |  30 +++
 drivers/iommu/irq_remapping.c        |   9 +
 drivers/iommu/irq_remapping.h        |   3 +
 drivers/pci/msi.c                    |  19 +-
 drivers/vfio/mdev/mdev_core.c        |   6 +
 drivers/vfio/mdev/mdev_private.h     |   1 -
 include/linux/intel-iommu.h          |   1 +
 include/linux/mdev.h                 |   2 +
 include/linux/msi.h                  |  55 +++++-
 kernel/irq/msi.c                     |   2 +-
 24 files changed, 655 insertions(+), 19 deletions(-)
 create mode 100644 arch/x86/kernel/apic/ims.c
 create mode 100644 drivers/base/ims-msi.c

-- 
2.7.4

Powered by blists - more mailing lists