Message-ID: <MWHPR21MB1593093B61DC506B64986B14D7619@MWHPR21MB1593.namprd21.prod.outlook.com>
Date:   Wed, 24 Nov 2021 17:03:26 +0000
From:   "Michael Kelley (LINUX)" <mikelley@...rosoft.com>
To:     Tianyu Lan <ltykernel@...il.com>,
        "tglx@...utronix.de" <tglx@...utronix.de>,
        "mingo@...hat.com" <mingo@...hat.com>,
        "bp@...en8.de" <bp@...en8.de>,
        "dave.hansen@...ux.intel.com" <dave.hansen@...ux.intel.com>,
        "x86@...nel.org" <x86@...nel.org>, "hpa@...or.com" <hpa@...or.com>,
        "luto@...nel.org" <luto@...nel.org>,
        "peterz@...radead.org" <peterz@...radead.org>,
        "jgross@...e.com" <jgross@...e.com>,
        "sstabellini@...nel.org" <sstabellini@...nel.org>,
        "boris.ostrovsky@...cle.com" <boris.ostrovsky@...cle.com>,
        KY Srinivasan <kys@...rosoft.com>,
        Haiyang Zhang <haiyangz@...rosoft.com>,
        Stephen Hemminger <sthemmin@...rosoft.com>,
        "wei.liu@...nel.org" <wei.liu@...nel.org>,
        Dexuan Cui <decui@...rosoft.com>,
        "joro@...tes.org" <joro@...tes.org>,
        "will@...nel.org" <will@...nel.org>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "kuba@...nel.org" <kuba@...nel.org>,
        "jejb@...ux.ibm.com" <jejb@...ux.ibm.com>,
        "martin.petersen@...cle.com" <martin.petersen@...cle.com>,
        "hch@....de" <hch@....de>,
        "m.szyprowski@...sung.com" <m.szyprowski@...sung.com>,
        "robin.murphy@....com" <robin.murphy@....com>,
        Tianyu Lan <Tianyu.Lan@...rosoft.com>,
        "thomas.lendacky@....com" <thomas.lendacky@....com>,
        "xen-devel@...ts.xenproject.org" <xen-devel@...ts.xenproject.org>
CC:     "iommu@...ts.linux-foundation.org" <iommu@...ts.linux-foundation.org>,
        "linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-scsi@...r.kernel.org" <linux-scsi@...r.kernel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        vkuznets <vkuznets@...hat.com>,
        "brijesh.singh@....com" <brijesh.singh@....com>,
        "konrad.wilk@...cle.com" <konrad.wilk@...cle.com>,
        "parri.andrea@...il.com" <parri.andrea@...il.com>,
        "dave.hansen@...el.com" <dave.hansen@...el.com>
Subject: RE: [PATCH V2 5/6] net: netvsc: Add Isolation VM support for netvsc
 driver

From: Tianyu Lan <ltykernel@...il.com> Sent: Tuesday, November 23, 2021 6:31 AM
> 
> In Isolation VM, all shared memory with host needs to mark visible
> to host via hvcall. vmbus_establish_gpadl() has already done it for
> netvsc rx/tx ring buffer. The page buffer used by vmbus_sendpacket_
> pagebuffer() stills need to be handled. Use DMA API to map/umap
> these memory during sending/receiving packet and Hyper-V swiotlb
> bounce buffer dma address will be returned. The swiotlb bounce buffer
> has been masked to be visible to host during boot up.
> 
> Allocate rx/tx ring buffer via dma_alloc_noncontiguous() in Isolation
> VM. After calling vmbus_establish_gpadl() which marks these pages visible
> to host, map these pages unencrypted addes space via dma_vmap_noncontiguous().
> 

The big unresolved topic is how best to do the allocation and mapping of the big
netvsc send and receive buffers.  Let me summarize and make a recommendation.

Background
==========
1.  Each Hyper-V synthetic network device requires a large pre-allocated receive
     buffer (defaults to 16 Mbytes) and a similar send buffer (defaults to 1 Mbyte).
2.  The buffers are allocated in guest memory and shared with the Hyper-V host.
     As such, in the Hyper-V SNP environment, the memory must be unencrypted
     and accessed in the Hyper-V guest with shared_gpa_boundary (i.e., VTOM)
     added to the physical memory address.
3.  The buffers need *not* be contiguous in guest physical memory, but must be
     contiguously mapped in guest kernel virtual space.
4.  Network devices may come and go during the life of the VM, so allocation of
     these buffers and their mappings may be done after Linux has been running for
     a long time.
5.  Performance of the allocation and mapping process is not an issue since it is
     done only on synthetic network device add/remove.
6.  So the primary goals are an appropriate logical abstraction, code that is
     simple and straightforward, and efficient memory usage.

Approaches
==========
During the development of these patches, four approaches have been
implemented:

1.  Two virtual mappings:  One from vmalloc() to allocate the guest memory, and
     the second from vmap_pfns() after adding the shared_gpa_boundary.   This is
     implemented in Hyper-V or netvsc specific code, with no use of DMA APIs.
     No separate list of physical pages is maintained, so for creating the second
     mapping, the PFN list is assembled temporarily by doing virt-to-phys()
     page-by-page on the vmalloc mapping, and then discarded because it is no
     longer needed.  [v4 of the original patch series.]

2.  Two virtual mappings as in (1) above, but implemented via new DMA calls
     dma_map_decrypted() and dma_unmap_encrypted().  [v3 of the original
     patch series.]

3.  Two virtual mappings as in (1) above, but implemented via DMA noncontiguous
      allocation and mapping calls, as enhanced to allow for custom map/unmap
      implementations.  A list of physical pages is maintained in the dma_sgt_handle
      as expected by the DMA noncontiguous API.  [New split-off patch series v1 & v2]

4.   Single virtual mapping from vmap_pfns().  The netvsc driver allocates physical
      memory via alloc_pages() with as much contiguity as possible, and maintains a
      list of physical pages and ranges.   Single virtual map is setup with vmap_pfns()
      after adding shared_gpa_boundary.  [v5 of the original patch series.]

Both implementations using DMA APIs use very little of the existing DMA
machinery.  Both require extensions to the DMA APIs, and custom ops functions.
While in some sense the netvsc send and receive buffers involve DMA, they
do not require any DMA actions on a per-I/O basis.  It seems better to me to
not try to fit these two buffers into the DMA model as a one-off.  Let's just
use Hyper-V specific code to allocate and map them, as is done with the
Hyper-V VMbus channel ring buffers.

That leaves approaches (1) and (4) above.  Between those two, (1) is
simpler even though there are two virtual mappings.  Using alloc_pages() as
in (4) is messy and there's no real benefit to using higher order allocations.
(4) also requires maintaining a separate list of PFNs and ranges, which offsets
some of the benefits to having only one virtual mapping active at any point in
time.

I don't think there's a clear "right" answer, so it's a judgment call.  We've
explored what other approaches would look like, and I'd say let's go with
(1) as the simpler approach.  Thoughts?

Michael

Powered by blists - more mailing lists