| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <a96626db-4ac9-3ce4-64e9-92568e4f827a@gmail.com>
Date: Fri, 20 Aug 2021 23:20:21 +0800
From: Tianyu Lan <ltykernel@...il.com>
To: Michael Kelley <mikelley@...rosoft.com>,
KY Srinivasan <kys@...rosoft.com>,
Haiyang Zhang <haiyangz@...rosoft.com>,
Stephen Hemminger <sthemmin@...rosoft.com>,
"wei.liu@...nel.org" <wei.liu@...nel.org>,
Dexuan Cui <decui@...rosoft.com>,
"tglx@...utronix.de" <tglx@...utronix.de>,
"mingo@...hat.com" <mingo@...hat.com>,
"bp@...en8.de" <bp@...en8.de>, "x86@...nel.org" <x86@...nel.org>,
"hpa@...or.com" <hpa@...or.com>,
"dave.hansen@...ux.intel.com" <dave.hansen@...ux.intel.com>,
"luto@...nel.org" <luto@...nel.org>,
"peterz@...radead.org" <peterz@...radead.org>,
"konrad.wilk@...cle.com" <konrad.wilk@...cle.com>,
"boris.ostrovsky@...cle.com" <boris.ostrovsky@...cle.com>,
"jgross@...e.com" <jgross@...e.com>,
"sstabellini@...nel.org" <sstabellini@...nel.org>,
"joro@...tes.org" <joro@...tes.org>,
"will@...nel.org" <will@...nel.org>,
"davem@...emloft.net" <davem@...emloft.net>,
"kuba@...nel.org" <kuba@...nel.org>,
"jejb@...ux.ibm.com" <jejb@...ux.ibm.com>,
"martin.petersen@...cle.com" <martin.petersen@...cle.com>,
"arnd@...db.de" <arnd@...db.de>, "hch@....de" <hch@....de>,
"m.szyprowski@...sung.com" <m.szyprowski@...sung.com>,
"robin.murphy@....com" <robin.murphy@....com>,
"thomas.lendacky@....com" <thomas.lendacky@....com>,
"brijesh.singh@....com" <brijesh.singh@....com>,
"ardb@...nel.org" <ardb@...nel.org>,
Tianyu Lan <Tianyu.Lan@...rosoft.com>,
"pgonda@...gle.com" <pgonda@...gle.com>,
"martin.b.radev@...il.com" <martin.b.radev@...il.com>,
"akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
"kirill.shutemov@...ux.intel.com" <kirill.shutemov@...ux.intel.com>,
"rppt@...nel.org" <rppt@...nel.org>,
"sfr@...b.auug.org.au" <sfr@...b.auug.org.au>,
"saravanand@...com" <saravanand@...com>,
"krish.sadhukhan@...cle.com" <krish.sadhukhan@...cle.com>,
"aneesh.kumar@...ux.ibm.com" <aneesh.kumar@...ux.ibm.com>,
"xen-devel@...ts.xenproject.org" <xen-devel@...ts.xenproject.org>,
"rientjes@...gle.com" <rientjes@...gle.com>,
"hannes@...xchg.org" <hannes@...xchg.org>,
"tj@...nel.org" <tj@...nel.org>
Cc: "iommu@...ts.linux-foundation.org" <iommu@...ts.linux-foundation.org>,
"linux-arch@...r.kernel.org" <linux-arch@...r.kernel.org>,
"linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"linux-scsi@...r.kernel.org" <linux-scsi@...r.kernel.org>,
"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
vkuznets <vkuznets@...hat.com>,
"parri.andrea@...il.com" <parri.andrea@...il.com>,
"dave.hansen@...el.com" <dave.hansen@...el.com>
Subject: Re: [PATCH V3 13/13] HV/Storvsc: Add Isolation VM support for storvsc
driver
On 8/20/2021 2:17 AM, Michael Kelley wrote:
> From: Tianyu Lan <ltykernel@...il.com> Sent: Monday, August 9, 2021 10:56 AM
>>
>
> Subject line tag should be "scsi: storvsc:"
>
>> In Isolation VM, all shared memory with host needs to mark visible
>> to host via hvcall. vmbus_establish_gpadl() has already done it for
>> storvsc rx/tx ring buffer. The page buffer used by vmbus_sendpacket_
>> mpb_desc() still need to handle. Use DMA API to map/umap these
>
> s/need to handle/needs to be handled/
>
>> memory during sending/receiving packet and Hyper-V DMA ops callback
>> will use swiotlb function to allocate bounce buffer and copy data
>> from/to bounce buffer.
>>
>> Signed-off-by: Tianyu Lan <Tianyu.Lan@...rosoft.com>
>> ---
>> drivers/scsi/storvsc_drv.c | 68 +++++++++++++++++++++++++++++++++++---
>> 1 file changed, 63 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
>> index 328bb961c281..78320719bdd8 100644
>> --- a/drivers/scsi/storvsc_drv.c
>> +++ b/drivers/scsi/storvsc_drv.c
>> @@ -21,6 +21,8 @@
>> #include <linux/device.h>
>> #include <linux/hyperv.h>
>> #include <linux/blkdev.h>
>> +#include <linux/io.h>
>> +#include <linux/dma-mapping.h>
>> #include <scsi/scsi.h>
>> #include <scsi/scsi_cmnd.h>
>> #include <scsi/scsi_host.h>
>> @@ -427,6 +429,8 @@ struct storvsc_cmd_request {
>> u32 payload_sz;
>>
>> struct vstor_packet vstor_packet;
>> + u32 hvpg_count;
>
> This count is really the number of entries in the dma_range
> array, right? If so, perhaps "dma_range_count" would be
> a better name so that it is more tightly associated.
Yes, will update.
>
>> + struct hv_dma_range *dma_range;
>> };
>>
>>
>> @@ -509,6 +513,14 @@ struct storvsc_scan_work {
>> u8 tgt_id;
>> };
>>
>> +#define storvsc_dma_map(dev, page, offset, size, dir) \
>> + dma_map_page(dev, page, offset, size, dir)
>> +
>> +#define storvsc_dma_unmap(dev, dma_range, dir) \
>> + dma_unmap_page(dev, dma_range.dma, \
>> + dma_range.mapping_size, \
>> + dir ? DMA_FROM_DEVICE : DMA_TO_DEVICE)
>> +
>
> Each of these macros is used only once. IMHO, they don't
> add a lot of value. Just coding dma_map/unmap_page()
> inline would be fine and eliminate these lines of code.
OK. Will update.
>
>> static void storvsc_device_scan(struct work_struct *work)
>> {
>> struct storvsc_scan_work *wrk;
>> @@ -1260,6 +1272,7 @@ static void storvsc_on_channel_callback(void *context)
>> struct hv_device *device;
>> struct storvsc_device *stor_device;
>> struct Scsi_Host *shost;
>> + int i;
>>
>> if (channel->primary_channel != NULL)
>> device = channel->primary_channel->device_obj;
>> @@ -1314,6 +1327,15 @@ static void storvsc_on_channel_callback(void *context)
>> request = (struct storvsc_cmd_request *)scsi_cmd_priv(scmnd);
>> }
>>
>> + if (request->dma_range) {
>> + for (i = 0; i < request->hvpg_count; i++)
>> + storvsc_dma_unmap(&device->device,
>> + request->dma_range[i],
>> + request->vstor_packet.vm_srb.data_in == READ_TYPE);
>
> I think you can directly get the DMA direction as request->cmd->sc_data_direction.
>
>> +
>> + kfree(request->dma_range);
>> + }
>> +
>> storvsc_on_receive(stor_device, packet, request);
>> continue;
>> }
>> @@ -1810,7 +1832,9 @@ static int storvsc_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *scmnd)
>> unsigned int hvpgoff, hvpfns_to_add;
>> unsigned long offset_in_hvpg = offset_in_hvpage(sgl->offset);
>> unsigned int hvpg_count = HVPFN_UP(offset_in_hvpg + length);
>> + dma_addr_t dma;
>> u64 hvpfn;
>> + u32 size;
>>
>> if (hvpg_count > MAX_PAGE_BUFFER_COUNT) {
>>
>> @@ -1824,6 +1848,13 @@ static int storvsc_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *scmnd)
>> payload->range.len = length;
>> payload->range.offset = offset_in_hvpg;
>>
>> + cmd_request->dma_range = kcalloc(hvpg_count,
>> + sizeof(*cmd_request->dma_range),
>> + GFP_ATOMIC);
>
> With this patch, it appears that storvsc_queuecommand() is always
> doing bounce buffering, even when running in a non-isolated VM.
In the non-isolated VM, SWIOTLB_FORCE mode isn't enabled and so
the swiotlb bounce buffer will not work.
> The dma_range is always allocated, and the inner loop below does
> the dma mapping for every I/O page. The corresponding code in
> storvsc_on_channel_callback() that does the dma unmap allows for
> the dma_range to be NULL, but that never happens.
Yes, dma mapping function will return PA directly in non-isolated VM.
>
>> + if (!cmd_request->dma_range) {
>> + ret = -ENOMEM;
>
> The other memory allocation failure in this function returns
> SCSI_MLQUEUE_DEVICE_BUSY. It may be debatable as to whether
> that's the best approach, but that's a topic for a different patch. I
> would suggest being consistent and using the same return code
> here.
OK. I will keep to return SCSI_MLQUEUE_DEVICE_BUSY here.
>
>> + goto free_payload;
>> + }
>>
>> for (i = 0; sgl != NULL; sgl = sg_next(sgl)) {
>> /*
>> @@ -1847,9 +1878,29 @@ static int storvsc_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *scmnd)
>> * last sgl should be reached at the same time that
>> * the PFN array is filled.
>> */
>> - while (hvpfns_to_add--)
>> - payload->range.pfn_array[i++] = hvpfn++;
>> + while (hvpfns_to_add--) {
>> + size = min(HV_HYP_PAGE_SIZE - offset_in_hvpg,
>> + (unsigned long)length);
>> + dma = storvsc_dma_map(&dev->device, pfn_to_page(hvpfn++),
>> + offset_in_hvpg, size,
>> + scmnd->sc_data_direction);
>> + if (dma_mapping_error(&dev->device, dma)) {
>> + ret = -ENOMEM;
>
> The typical error from dma_map_page() will be running out of
> bounce buffer memory. This is a transient condition that should be
> retried at the higher levels. So make sure to return an error code
> that indicates the I/O should be resubmitted.
OK. It looks like error code should be SCSI_MLQUEUE_DEVICE_BUSY here.
>
>> + goto free_dma_range;
>> + }
>> +
>> + if (offset_in_hvpg) {
>> + payload->range.offset = dma & ~HV_HYP_PAGE_MASK;
>> + offset_in_hvpg = 0;
>> + }
>
> I'm not clear on why payload->range.offset needs to be set again.
> Even after the dma mapping is done, doesn't the offset in the first
> page have to be the same? If it wasn't the same, Hyper-V wouldn't
> be able to process the PFN list correctly. In fact, couldn't the above
> code just always set offset_in_hvpg = 0?
The offset will be changed. The swiotlb bounce buffer is allocated with
IO_TLB_SIZE(2K) as unit. So the offset here may be changed.
>
>> +
>> + cmd_request->dma_range[i].dma = dma;
>> + cmd_request->dma_range[i].mapping_size = size;
>> + payload->range.pfn_array[i++] = dma >> HV_HYP_PAGE_SHIFT;
>> + length -= size;
>> + }
>> }
>> + cmd_request->hvpg_count = hvpg_count;
>
> This line just saves the size of the dma_range array. Could
> it be moved up with the code that allocates the dma_range
> array? To me, it would make more sense to have all that
> code together in one place.
Sure. Will update.
>
>> }
>
> The whole approach here is to do dma remapping on each individual page
> of the I/O buffer. But wouldn't it be possible to use dma_map_sg() to map
> each scatterlist entry as a unit? Each scatterlist entry describes a range of
> physically contiguous memory. After dma_map_sg(), the resulting dma
> address must also refer to a physically contiguous range in the swiotlb
> bounce buffer memory. So at the top of the "for" loop over the scatterlist
> entries, do dma_map_sg() if we're in an isolated VM. Then compute the
> hvpfn value based on the dma address instead of sg_page(). But everything
> else is the same, and the inner loop for populating the pfn_arry is unmodified.
> Furthermore, the dma_range array that you've added is not needed, since
> scatterlist entries already have a dma_address field for saving the mapped
> address, and dma_unmap_sg() uses that field.
I don't use dma_map_sg() here in order to avoid introducing one more
loop(e,g dma_map_sg()). We already have a loop to populate
cmd_request->dma_range[] and so do the dma map in the same loop.
>
> One thing: There's a maximum swiotlb mapping size, which I think works
> out to be 256 Kbytes. See swiotlb_max_mapping_size(). We need to make
> sure that we don't get a scatterlist entry bigger than this size. But I think
> this already happens because you set the device->dma_mask field in
> Patch 11 of this series. __scsi_init_queue checks for this setting and
> sets max_sectors to limits transfers to the max mapping size.
I will double check.
>
>>
>> cmd_request->payload = payload;
>> @@ -1860,13 +1911,20 @@ static int storvsc_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *scmnd)
>> put_cpu();
>>
>> if (ret == -EAGAIN) {
>> - if (payload_sz > sizeof(cmd_request->mpb))
>> - kfree(payload);
>> /* no more space */
>> - return SCSI_MLQUEUE_DEVICE_BUSY;
>> + ret = SCSI_MLQUEUE_DEVICE_BUSY;
>> + goto free_dma_range;
>> }
>>
>> return 0;
>> +
>> +free_dma_range:
>> + kfree(cmd_request->dma_range);
>> +
>> +free_payload:
>> + if (payload_sz > sizeof(cmd_request->mpb))
>> + kfree(payload);
>> + return ret;
>> }
>>
>> static struct scsi_host_template scsi_driver = {
>> --
>> 2.25.1
>
Powered by blists - more mailing lists