[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <afc34f07-ff4c-4947-a203-ef244dcf43e8@linux.dev>
Date: Thu, 7 Mar 2024 07:01:10 +0100
From: Zhu Yanjun <yanjun.zhu@...ux.dev>
To: Leon Romanovsky <leon@...nel.org>, Christoph Hellwig <hch@....de>,
Robin Murphy <robin.murphy@....com>,
Marek Szyprowski <m.szyprowski@...sung.com>, Joerg Roedel <joro@...tes.org>,
Will Deacon <will@...nel.org>, Jason Gunthorpe <jgg@...pe.ca>,
Chaitanya Kulkarni <chaitanyak@...dia.com>
Cc: Jonathan Corbet <corbet@....net>, Jens Axboe <axboe@...nel.dk>,
Keith Busch <kbusch@...nel.org>, Sagi Grimberg <sagi@...mberg.me>,
Yishai Hadas <yishaih@...dia.com>,
Shameer Kolothum <shameerali.kolothum.thodi@...wei.com>,
Kevin Tian <kevin.tian@...el.com>,
Alex Williamson <alex.williamson@...hat.com>,
Jérôme Glisse <jglisse@...hat.com>,
Andrew Morton <akpm@...ux-foundation.org>, linux-doc@...r.kernel.org,
linux-kernel@...r.kernel.org, linux-block@...r.kernel.org,
linux-rdma@...r.kernel.org, iommu@...ts.linux.dev,
linux-nvme@...ts.infradead.org, kvm@...r.kernel.org, linux-mm@...ck.org,
Bart Van Assche <bvanassche@....org>,
Damien Le Moal <damien.lemoal@...nsource.wdc.com>,
Amir Goldstein <amir73il@...il.com>,
"josef@...icpanda.com" <josef@...icpanda.com>,
"Martin K. Petersen" <martin.petersen@...cle.com>,
"daniel@...earbox.net" <daniel@...earbox.net>,
Dan Williams <dan.j.williams@...el.com>, "jack@...e.com" <jack@...e.com>,
Leon Romanovsky <leonro@...dia.com>, Zhu Yanjun <zyjzyj2000@...il.com>
Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps
在 2024/3/5 12:18, Leon Romanovsky 写道:
> This is complimentary part to the proposed LSF/MM topic.
> https://lore.kernel.org/linux-rdma/22df55f8-cf64-4aa8-8c0b-b556c867b926@linux.dev/T/#m85672c860539fdbbc8fe0f5ccabdc05b40269057
I am interested in this topic. Hope I can join the meeting to discuss
this topic.
Zhu Yanjun
>
> This is posted as RFC to get a feedback on proposed split, but RDMA, VFIO and
> DMA patches are ready for review and inclusion, the NVMe patches are still in
> progress as they require agreement on API first.
>
> Thanks
>
> -------------------------------------------------------------------------------
> The DMA mapping operation performs two steps at one same time: allocates
> IOVA space and actually maps DMA pages to that space. This one shot
> operation works perfectly for non-complex scenarios, where callers use
> that DMA API in control path when they setup hardware.
>
> However in more complex scenarios, when DMA mapping is needed in data
> path and especially when some sort of specific datatype is involved,
> such one shot approach has its drawbacks.
>
> That approach pushes developers to introduce new DMA APIs for specific
> datatype. For example existing scatter-gather mapping functions, or
> latest Chuck's RFC series to add biovec related DMA mapping [1] and
> probably struct folio will need it too.
>
> These advanced DMA mapping APIs are needed to calculate IOVA size to
> allocate it as one chunk and some sort of offset calculations to know
> which part of IOVA to map.
>
> Instead of teaching DMA to know these specific datatypes, let's separate
> existing DMA mapping routine to two steps and give an option to advanced
> callers (subsystems) perform all calculations internally in advance and
> map pages later when it is needed.
>
> In this series, three users are converted and each of such conversion
> presents different positive gain:
> 1. RDMA simplifies and speeds up its pagefault handling for
> on-demand-paging (ODP) mode.
> 2. VFIO PCI live migration code saves huge chunk of memory.
> 3. NVMe PCI avoids intermediate SG table manipulation and operates
> directly on BIOs.
>
> Thanks
>
> [1] https://lore.kernel.org/all/169772852492.5232.17148564580779995849.stgit@klimt.1015granger.net
>
> Chaitanya Kulkarni (2):
> block: add dma_link_range() based API
> nvme-pci: use blk_rq_dma_map() for NVMe SGL
>
> Leon Romanovsky (14):
> mm/hmm: let users to tag specific PFNs
> dma-mapping: provide an interface to allocate IOVA
> dma-mapping: provide callbacks to link/unlink pages to specific IOVA
> iommu/dma: Provide an interface to allow preallocate IOVA
> iommu/dma: Prepare map/unmap page functions to receive IOVA
> iommu/dma: Implement link/unlink page callbacks
> RDMA/umem: Preallocate and cache IOVA for UMEM ODP
> RDMA/umem: Store ODP access mask information in PFN
> RDMA/core: Separate DMA mapping to caching IOVA and page linkage
> RDMA/umem: Prevent UMEM ODP creation with SWIOTLB
> vfio/mlx5: Explicitly use number of pages instead of allocated length
> vfio/mlx5: Rewrite create mkey flow to allow better code reuse
> vfio/mlx5: Explicitly store page list
> vfio/mlx5: Convert vfio to use DMA link API
>
> Documentation/core-api/dma-attributes.rst | 7 +
> block/blk-merge.c | 156 ++++++++++++++
> drivers/infiniband/core/umem_odp.c | 219 +++++++------------
> drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 +
> drivers/infiniband/hw/mlx5/odp.c | 59 +++--
> drivers/iommu/dma-iommu.c | 129 ++++++++---
> drivers/nvme/host/pci.c | 220 +++++--------------
> drivers/vfio/pci/mlx5/cmd.c | 252 ++++++++++++----------
> drivers/vfio/pci/mlx5/cmd.h | 22 +-
> drivers/vfio/pci/mlx5/main.c | 136 +++++-------
> include/linux/blk-mq.h | 9 +
> include/linux/dma-map-ops.h | 13 ++
> include/linux/dma-mapping.h | 39 ++++
> include/linux/hmm.h | 3 +
> include/rdma/ib_umem_odp.h | 22 +-
> include/rdma/ib_verbs.h | 54 +++++
> kernel/dma/debug.h | 2 +
> kernel/dma/direct.h | 7 +-
> kernel/dma/mapping.c | 91 ++++++++
> mm/hmm.c | 34 +--
> 20 files changed, 870 insertions(+), 605 deletions(-)
>
Powered by blists - more mailing lists