| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <cover.1722512548.git.leon@kernel.org>
Date: Thu, 1 Aug 2024 15:05:09 +0300
From: Leon Romanovsky <leon@...nel.org>
To: Jason Gunthorpe <jgg@...dia.com>
Cc: Leon Romanovsky <leonro@...dia.com>,
Christian König <christian.koenig@....com>,
dri-devel@...ts.freedesktop.org,
linaro-mm-sig@...ts.linaro.org,
linux-kernel@...r.kernel.org,
linux-media@...r.kernel.org,
linux-rdma@...r.kernel.org,
Michael Margolin <mrgolin@...zon.com>,
Mustafa Ismail <mustafa.ismail@...el.com>,
netdev@...r.kernel.org,
Saeed Mahameed <saeedm@...dia.com>,
Selvin Xavier <selvin.xavier@...adcom.com>,
Sumit Semwal <sumit.semwal@...aro.org>,
Tariq Toukan <tariqt@...dia.com>,
Tatyana Nikolova <tatyana.e.nikolova@...el.com>,
Yishai Hadas <yishaih@...dia.com>
Subject: [PATCH rdma-next 0/8] Introducing Multi-Path DMA Support for mlx5 RDMA Driver
From: Leon Romanovsky <leonro@...dia.com>
>From Yishai,
Overview
--------
This patch series aims to enable multi-path DMA support, allowing an
mlx5 RDMA device to issue DMA commands through multiple paths. This
feature is critical for improving performance and reaching line rate
in certain environments where issuing PCI transactions over one path
may be significantly faster than over another. These differences can
arise from various PCI generations in the system or the specific system
topology.
To achieve this functionality, we introduced a data direct DMA device
that can serve the RDMA device by issuing DMA transactions on its behalf.
The main key features and changes are described below.
Multi-Path Discovery
--------------------
API Implementation:
* Introduced an API to discover multiple paths for a given mlx5 RDMA device.
IOCTL Command:
* Added a new IOCTL command, MLX5_IB_METHOD_GET_DATA_DIRECT_SYSFS_PATH, to
the DEVICE object. When an affiliated Data-Direct/DMA device is present,
its sysfs path is returned.
Feature Activation by mlx5 RDMA Application
-------------------------------------------
UVERBS Extension:
* Extended UVERBS_METHOD_REG_DMABUF_MR over UVERBS_OBJECT_MR to include
mlx5 extended flags.
Access Flag:
* Introduced the MLX5_IB_UAPI_REG_DMABUF_ACCESS_DATA_DIRECT flag, allowing
applications to request the use of the affiliated DMA device for DMABUF
registration.
Data-Direct/DMA Device
----------------------
New Driver:
* Introduced a new driver to manage the new DMA PF device ID (0x2100).
Its registration/un-registration is handled as part of the mlx5_ib init/exit
flows, with mlx5 IB devices as its clients.
Functionality:
* The driver does not interface directly with the firmware (no command interface,
no caps, etc.) but works over PCI to activate its DMA functionality. It serves
as the DMA device for efficiently accessing other PCI devices (e.g., GPU PF) and
reads its VUID over PCI to handle NICs registrations with the same VUID.
mlx5 IB RDMA Device
---------------------------
VUID Query:
* Reads its affiliated DMA PF VUID via the QUERY_VUID command with the data_direct
bit set.
Driver Registration:
* Registers with the DMA PF driver to be notified upon bind/unbind.
Application Request Handling:
* Uses the DMA PF device upon application request as described above.
DMABUF over Umem
----------------
Introduced an option to obtain a DMABUF UMEM using a different DMA
device instead of the IB device, allowing the device to register over
IOMMU with the expected DMA device for a given buffer registration.
Further details are provided in the commit logs of the patches in this
series.
Thanks
Yishai Hadas (8):
net/mlx5: Add IFC related stuff for data direct
RDMA/mlx5: Introduce the 'data direct' driver
RDMA/mlx5: Add the initialization flow to utilize the 'data direct'
device
RDMA/umem: Add support for creating pinned DMABUF umem with a given
dma device
RDMA/umem: Introduce an option to revoke DMABUF umem
RDMA: Pass uverbs_attr_bundle as part of '.reg_user_mr_dmabuf' API
RDMA/mlx5: Add support for DMABUF MR registrations with Data-direct
RDMA/mlx5: Introduce GET_DATA_DIRECT_SYSFS_PATH ioctl
drivers/infiniband/core/umem_dmabuf.c | 66 +++-
drivers/infiniband/core/uverbs_std_types_mr.c | 2 +-
drivers/infiniband/hw/bnxt_re/ib_verbs.c | 3 +-
drivers/infiniband/hw/bnxt_re/ib_verbs.h | 2 +-
drivers/infiniband/hw/efa/efa.h | 2 +-
drivers/infiniband/hw/efa/efa_verbs.c | 4 +-
drivers/infiniband/hw/irdma/verbs.c | 2 +-
drivers/infiniband/hw/mlx5/Makefile | 1 +
drivers/infiniband/hw/mlx5/cmd.c | 21 ++
drivers/infiniband/hw/mlx5/cmd.h | 2 +
drivers/infiniband/hw/mlx5/data_direct.c | 227 +++++++++++++
drivers/infiniband/hw/mlx5/data_direct.h | 23 ++
drivers/infiniband/hw/mlx5/main.c | 125 +++++++
drivers/infiniband/hw/mlx5/mlx5_ib.h | 22 +-
drivers/infiniband/hw/mlx5/mr.c | 304 +++++++++++++++---
drivers/infiniband/hw/mlx5/odp.c | 5 +-
drivers/infiniband/hw/mlx5/std_types.c | 55 +++-
drivers/infiniband/hw/mlx5/umr.c | 93 ++++--
drivers/infiniband/hw/mlx5/umr.h | 1 +
include/linux/mlx5/mlx5_ifc.h | 51 ++-
include/rdma/ib_umem.h | 18 ++
include/rdma/ib_verbs.h | 2 +-
include/uapi/rdma/mlx5_user_ioctl_cmds.h | 9 +
include/uapi/rdma/mlx5_user_ioctl_verbs.h | 4 +
24 files changed, 944 insertions(+), 100 deletions(-)
create mode 100644 drivers/infiniband/hw/mlx5/data_direct.c
create mode 100644 drivers/infiniband/hw/mlx5/data_direct.h
--
2.45.2
Powered by blists - more mailing lists