netdev - Re: [PATCH v8 00/10] Introduce VDUSE

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Tue, 29 Jun 2021 12:10:51 +0800
From:   Jason Wang <jasowang@...hat.com>
To:     "Liu, Xiaodong" <xiaodong.liu@...el.com>,
        Xie Yongji <xieyongji@...edance.com>,
        "mst@...hat.com" <mst@...hat.com>,
        "stefanha@...hat.com" <stefanha@...hat.com>,
        "sgarzare@...hat.com" <sgarzare@...hat.com>,
        "parav@...dia.com" <parav@...dia.com>,
        "hch@...radead.org" <hch@...radead.org>,
        "christian.brauner@...onical.com" <christian.brauner@...onical.com>,
        "rdunlap@...radead.org" <rdunlap@...radead.org>,
        "willy@...radead.org" <willy@...radead.org>,
        "viro@...iv.linux.org.uk" <viro@...iv.linux.org.uk>,
        "axboe@...nel.dk" <axboe@...nel.dk>,
        "bcrl@...ck.org" <bcrl@...ck.org>,
        "corbet@....net" <corbet@....net>,
        "mika.penttila@...tfour.com" <mika.penttila@...tfour.com>,
        "dan.carpenter@...cle.com" <dan.carpenter@...cle.com>,
        "joro@...tes.org" <joro@...tes.org>,
        "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>
Cc:     "songmuchun@...edance.com" <songmuchun@...edance.com>,
        "virtualization@...ts.linux-foundation.org" 
        <virtualization@...ts.linux-foundation.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
        "iommu@...ts.linux-foundation.org" <iommu@...ts.linux-foundation.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v8 00/10] Introduce VDUSE - vDPA Device in Userspace


在 2021/6/28 下午1:54, Liu, Xiaodong 写道:
>> Several issues:
>>
>> - VDUSE needs to limit the total size of the bounce buffers (64M if I was not
>> wrong). Does it work for SPDK?
> Yes, Jason. It is enough and works for SPDK.
> Since it's a kind of bounce buffer mainly for in-flight IO, so limited size like
> 64MB is enough.


Ok.


>
>> - VDUSE can use hugepages but I'm not sure we can mandate hugepages (or we
>> need introduce new flags for supporting this)
> Same with your worry, I'm afraid too that it is a hard for a kernel module
> to directly preallocate hugepage internal.
> What I tried is that:
> 1. A simple agent daemon (represents for one device)  `preallocates` and maps
>      dozens of 2MB hugepages (like 64MB) for one device.
> 2. The daemon passes its mapping addr&len and hugepage fd to kernel
>      module through created IOCTL.
> 3. Kernel module remaps the hugepages inside kernel.


Such model should work, but the main "issue" is that it introduce  
overheads in the case of vhost-vDPA.

Note that in the case of vhost-vDPA, we don't use bounce buffer, the  
userspace pages were shared directly.

And since DMA is not done per page, it prevents us from using tricks  
like vm_insert_page() in those cases.


> 4. Vhost user target gets and maps hugepage fd from kernel module
>      in vhost-user msg through Unix Domain Socket cmsg.
> Then kernel module and target map on the same hugepage based
> bounce buffer for in-flight IO.
>
> If there is one option in VDUSE to map userspace preallocated memory, then
> VDUSE should be able to mandate it even it is hugepage based.
>

As above, this requires some kind of re-design since VDUSE depends on  
the model of mmap(MAP_SHARED) instead of umem registering.

Thanks