lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Fri, 21 Oct 2022 14:28:15 +0800 From: Jason Wang <jasowang@...hat.com> To: Ming Lei <tom.leiming@...il.com>, Yongji Xie <xieyongji@...edance.com> Cc: Stefan Hajnoczi <stefanha@...il.com>, Ziyang Zhang <ZiyangZhang@...ux.alibaba.com>, Stefan Hajnoczi <stefanha@...hat.com>, io-uring@...r.kernel.org, linux-block@...r.kernel.org, linux-kernel <linux-kernel@...r.kernel.org>, "Denis V. Lunev" <den@...nvz.org>, Xiaoguang Wang <xiaoguang.wang@...ux.alibaba.com> Subject: Re: ublk-qcow2: ublk-qcow2 is available 在 2022/10/18 14:59, Ming Lei 写道: > On Mon, Oct 17, 2022 at 07:11:59PM +0800, Yongji Xie wrote: >> On Fri, Oct 14, 2022 at 8:57 PM Ming Lei <tom.leiming@...il.com> wrote: >>> On Thu, Oct 13, 2022 at 02:48:04PM +0800, Yongji Xie wrote: >>>> On Wed, Oct 12, 2022 at 10:22 PM Stefan Hajnoczi <stefanha@...il.com> wrote: >>>>> On Sat, 8 Oct 2022 at 04:43, Ziyang Zhang <ZiyangZhang@...ux.alibaba.com> wrote: >>>>>> On 2022/10/5 12:18, Ming Lei wrote: >>>>>>> On Tue, Oct 04, 2022 at 09:53:32AM -0400, Stefan Hajnoczi wrote: >>>>>>>> On Tue, 4 Oct 2022 at 05:44, Ming Lei <tom.leiming@...il.com> wrote: >>>>>>>>> On Mon, Oct 03, 2022 at 03:53:41PM -0400, Stefan Hajnoczi wrote: >>>>>>>>>> On Fri, Sep 30, 2022 at 05:24:11PM +0800, Ming Lei wrote: >>>>>>>>>>> ublk-qcow2 is available now. >>>>>>>>>> Cool, thanks for sharing! >>>>>>>>>> >>>>>>>>>>> So far it provides basic read/write function, and compression and snapshot >>>>>>>>>>> aren't supported yet. The target/backend implementation is completely >>>>>>>>>>> based on io_uring, and share the same io_uring with ublk IO command >>>>>>>>>>> handler, just like what ublk-loop does. >>>>>>>>>>> >>>>>>>>>>> Follows the main motivations of ublk-qcow2: >>>>>>>>>>> >>>>>>>>>>> - building one complicated target from scratch helps libublksrv APIs/functions >>>>>>>>>>> become mature/stable more quickly, since qcow2 is complicated and needs more >>>>>>>>>>> requirement from libublksrv compared with other simple ones(loop, null) >>>>>>>>>>> >>>>>>>>>>> - there are several attempts of implementing qcow2 driver in kernel, such as >>>>>>>>>>> ``qloop`` [2], ``dm-qcow2`` [3] and ``in kernel qcow2(ro)`` [4], so ublk-qcow2 >>>>>>>>>>> might useful be for covering requirement in this field >>>>>>>>>>> >>>>>>>>>>> - performance comparison with qemu-nbd, and it was my 1st thought to evaluate >>>>>>>>>>> performance of ublk/io_uring backend by writing one ublk-qcow2 since ublksrv >>>>>>>>>>> is started >>>>>>>>>>> >>>>>>>>>>> - help to abstract common building block or design pattern for writing new ublk >>>>>>>>>>> target/backend >>>>>>>>>>> >>>>>>>>>>> So far it basically passes xfstest(XFS) test by using ublk-qcow2 block >>>>>>>>>>> device as TEST_DEV, and kernel building workload is verified too. Also >>>>>>>>>>> soft update approach is applied in meta flushing, and meta data >>>>>>>>>>> integrity is guaranteed, 'make test T=qcow2/040' covers this kind of >>>>>>>>>>> test, and only cluster leak is reported during this test. >>>>>>>>>>> >>>>>>>>>>> The performance data looks much better compared with qemu-nbd, see >>>>>>>>>>> details in commit log[1], README[5] and STATUS[6]. And the test covers both >>>>>>>>>>> empty image and pre-allocated image, for example of pre-allocated qcow2 >>>>>>>>>>> image(8GB): >>>>>>>>>>> >>>>>>>>>>> - qemu-nbd (make test T=qcow2/002) >>>>>>>>>> Single queue? >>>>>>>>> Yeah. >>>>>>>>> >>>>>>>>>>> randwrite(4k): jobs 1, iops 24605 >>>>>>>>>>> randread(4k): jobs 1, iops 30938 >>>>>>>>>>> randrw(4k): jobs 1, iops read 13981 write 14001 >>>>>>>>>>> rw(512k): jobs 1, iops read 724 write 728 >>>>>>>>>> Please try qemu-storage-daemon's VDUSE export type as well. The >>>>>>>>>> command-line should be similar to this: >>>>>>>>>> >>>>>>>>>> # modprobe virtio_vdpa # attaches vDPA devices to host kernel >>>>>>>>> Not found virtio_vdpa module even though I enabled all the following >>>>>>>>> options: >>>>>>>>> >>>>>>>>> --- vDPA drivers >>>>>>>>> <M> vDPA device simulator core >>>>>>>>> <M> vDPA simulator for networking device >>>>>>>>> <M> vDPA simulator for block device >>>>>>>>> <M> VDUSE (vDPA Device in Userspace) support >>>>>>>>> <M> Intel IFC VF vDPA driver >>>>>>>>> <M> Virtio PCI bridge vDPA driver >>>>>>>>> <M> vDPA driver for Alibaba ENI >>>>>>>>> >>>>>>>>> BTW, my test environment is VM and the shared data is done in VM too, and >>>>>>>>> can virtio_vdpa be used inside VM? >>>>>>>> I hope Xie Yongji can help explain how to benchmark VDUSE. >>>>>>>> >>>>>>>> virtio_vdpa is available inside guests too. Please check that >>>>>>>> VIRTIO_VDPA ("vDPA driver for virtio devices") is enabled in "Virtio >>>>>>>> drivers" menu. >>>>>>>> >>>>>>>>>> # modprobe vduse >>>>>>>>>> # qemu-storage-daemon \ >>>>>>>>>> --blockdev file,filename=test.qcow2,cache.direct=of|off,aio=native,node-name=file \ >>>>>>>>>> --blockdev qcow2,file=file,node-name=qcow2 \ >>>>>>>>>> --object iothread,id=iothread0 \ >>>>>>>>>> --export vduse-blk,id=vduse0,name=vduse0,num-queues=$(nproc),node-name=qcow2,writable=on,iothread=iothread0 >>>>>>>>>> # vdpa dev add name vduse0 mgmtdev vduse >>>>>>>>>> >>>>>>>>>> A virtio-blk device should appear and xfstests can be run on it >>>>>>>>>> (typically /dev/vda unless you already have other virtio-blk devices). >>>>>>>>>> >>>>>>>>>> Afterwards you can destroy the device using: >>>>>>>>>> >>>>>>>>>> # vdpa dev del vduse0 >>>>>>>>>> >>>>>>>>>>> - ublk-qcow2 (make test T=qcow2/022) >>>>>>>>>> There are a lot of other factors not directly related to NBD vs ublk. In >>>>>>>>>> order to get an apples-to-apples comparison with qemu-* a ublk export >>>>>>>>>> type is needed in qemu-storage-daemon. That way only the difference is >>>>>>>>>> the ublk interface and the rest of the code path is identical, making it >>>>>>>>>> possible to compare NBD, VDUSE, ublk, etc more precisely. >>>>>>>>> Maybe not true. >>>>>>>>> >>>>>>>>> ublk-qcow2 uses io_uring to handle all backend IO(include meta IO) completely, >>>>>>>>> and so far single io_uring/pthread is for handling all qcow2 IOs and IO >>>>>>>>> command. >>>>>>>> qemu-nbd doesn't use io_uring to handle the backend IO, so we don't >>>>>>> I tried to use it via --aio=io_uring for setting up qemu-nbd, but not succeed. >>>>>>> >>>>>>>> know whether the benchmark demonstrates that ublk is faster than NBD, >>>>>>>> that the ublk-qcow2 implementation is faster than qemu-nbd's qcow2, >>>>>>>> whether there are miscellaneous implementation differences between >>>>>>>> ublk-qcow2 and qemu-nbd (like using the same io_uring context for both >>>>>>>> ublk and backend IO), or something else. >>>>>>> The theory shouldn't be too complicated: >>>>>>> >>>>>>> 1) io uring passthough(pt) communication is fast than socket, and io command >>>>>>> is carried over io_uring pt commands, and should be fast than virio >>>>>>> communication too. >>>>>>> >>>>>>> 2) io uring io handling is fast than libaio which is taken in the >>>>>>> test on qemu-nbd, and all qcow2 backend io(include meta io) is handled >>>>>>> by io_uring. >>>>>>> >>>>>>> https://github.com/ming1/ubdsrv/blob/master/tests/common/qcow2_common >>>>>>> >>>>>>> 3) ublk uses one single io_uring to handle all io commands and qcow2 >>>>>>> backend IOs, so batching handling is common, and it is easy to see >>>>>>> dozens of IOs/io commands handled in single syscall, or even more. >>>>>>> >>>>>>>> I'm suggesting measuring changes to just 1 variable at a time. >>>>>>>> Otherwise it's hard to reach a conclusion about the root cause of the >>>>>>>> performance difference. Let's learn why ublk-qcow2 performs well. >>>>>>> Turns out the latest Fedora 37-beta doesn't support vdpa yet, so I built >>>>>>> qemu from the latest github tree, and finally it starts to work. And test kernel >>>>>>> is v6.0 release. >>>>>>> >>>>>>> Follows the test result, and all three devices are setup as single >>>>>>> queue, and all tests are run in single job, still done in one VM, and >>>>>>> the test images are stored on XFS/virito-scsi backed SSD. >>>>>>> >>>>>>> The 1st group tests all three block device which is backed by empty >>>>>>> qcow2 image. >>>>>>> >>>>>>> The 2nd group tests all the three block devices backed by pre-allocated >>>>>>> qcow2 image. >>>>>>> >>>>>>> Except for big sequential IO(512K), there is still not small gap between >>>>>>> vdpa-virtio-blk and ublk. >>>>>>> >>>>>>> 1. run fio on block device over empty qcow2 image >>>>>>> 1) qemu-nbd >>>>>>> running qcow2/001 >>>>>>> run perf test on empty qcow2 image via nbd >>>>>>> fio (nbd(/mnt/data/ublk_null_8G_nYbgF.qcow2), libaio, bs 4k, dio, hw queues:1)... >>>>>>> randwrite: jobs 1, iops 8549 >>>>>>> randread: jobs 1, iops 34829 >>>>>>> randrw: jobs 1, iops read 11363 write 11333 >>>>>>> rw(512k): jobs 1, iops read 590 write 597 >>>>>>> >>>>>>> >>>>>>> 2) ublk-qcow2 >>>>>>> running qcow2/021 >>>>>>> run perf test on empty qcow2 image via ublk >>>>>>> fio (ublk/qcow2( -f /mnt/data/ublk_null_8G_s761j.qcow2), libaio, bs 4k, dio, hw queues:1, uring_comp: 0, get_data: 0). >>>>>>> randwrite: jobs 1, iops 16086 >>>>>>> randread: jobs 1, iops 172720 >>>>>>> randrw: jobs 1, iops read 35760 write 35702 >>>>>>> rw(512k): jobs 1, iops read 1140 write 1149 >>>>>>> >>>>>>> 3) vdpa-virtio-blk >>>>>>> running debug/test_dev >>>>>>> run io test on specified device >>>>>>> fio (vdpa(/dev/vdc), libaio, bs 4k, dio, hw queues:1)... >>>>>>> randwrite: jobs 1, iops 8626 >>>>>>> randread: jobs 1, iops 126118 >>>>>>> randrw: jobs 1, iops read 17698 write 17665 >>>>>>> rw(512k): jobs 1, iops read 1023 write 1031 >>>>>>> >>>>>>> >>>>>>> 2. run fio on block device over pre-allocated qcow2 image >>>>>>> 1) qemu-nbd >>>>>>> running qcow2/002 >>>>>>> run perf test on pre-allocated qcow2 image via nbd >>>>>>> fio (nbd(/mnt/data/ublk_data_8G_sc0SB.qcow2), libaio, bs 4k, dio, hw queues:1)... >>>>>>> randwrite: jobs 1, iops 21439 >>>>>>> randread: jobs 1, iops 30336 >>>>>>> randrw: jobs 1, iops read 11476 write 11449 >>>>>>> rw(512k): jobs 1, iops read 718 write 722 >>>>>>> >>>>>>> 2) ublk-qcow2 >>>>>>> running qcow2/022 >>>>>>> run perf test on pre-allocated qcow2 image via ublk >>>>>>> fio (ublk/qcow2( -f /mnt/data/ublk_data_8G_yZiaJ.qcow2), libaio, bs 4k, dio, hw queues:1, uring_comp: 0, get_data: 0). >>>>>>> randwrite: jobs 1, iops 98757 >>>>>>> randread: jobs 1, iops 110246 >>>>>>> randrw: jobs 1, iops read 47229 write 47161 >>>>>>> rw(512k): jobs 1, iops read 1416 write 1427 >>>>>>> >>>>>>> 3) vdpa-virtio-blk >>>>>>> running debug/test_dev >>>>>>> run io test on specified device >>>>>>> fio (vdpa(/dev/vdc), libaio, bs 4k, dio, hw queues:1)... >>>>>>> randwrite: jobs 1, iops 47317 >>>>>>> randread: jobs 1, iops 74092 >>>>>>> randrw: jobs 1, iops read 27196 write 27234 >>>>>>> rw(512k): jobs 1, iops read 1447 write 1458 >>>>>>> >>>>>>> >>>>>> Hi All, >>>>>> >>>>>> We are interested in VDUSE vs UBLK, too. And I have tested them with nullblk backend. >>>>>> Let me share some results here. >>>>>> >>>>>> I setup UBLK with: >>>>>> ublk add -t loop -f /dev/nullb0 -d QUEUE_DEPTH -q NR_QUEUE >>>>>> >>>>>> I setup VDUSE with: >>>>>> qemu-storage-daemon \ >>>>>> --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server=on,wait=off \ >>>>>> --monitor chardev=charmonitor \ >>>>>> --blockdev driver=host_device,cache.direct=on,filename=/dev/nullb0,node-name=disk0 \ >>>>>> --export vduse-blk,id=test,node-name=disk0,name=vduse_test,writable=on,num-queues=NR_QUEUE,queue-size=QUEUE_DEPTH >>>>>> >>>>>> Here QUEUE_DEPTH is 1, 32 or 128 and NR_QUEUE is 1 or 4. >>>>>> >>>>>> Note: >>>>>> (1) VDUSE requires QUEUE_DEPTH >= 2. I cannot setup QUEUE_DEPTH to 1. >>>>>> (2) I use qemu 7.1.0-rc3. It supports vduse-blk. >>>>>> (3) I do not use ublk null target so that the test is fair. >>>>>> (4) I setup fio with direct=1, bs=4k. >>>>>> >>>>>> ------------------------------ >>>>>> 1 job 1 iodepth, lat(usec) >>>>>> vduse ublk >>>>>> seq-read 22.55 11.15 >>>>>> rand-read 22.49 11.17 >>>>>> seq-write 25.67 10.25 >>>>>> rand-write 24.13 10.16 >>>>> Thanks for sharing. Any idea what the bottlenecks are for vduse and ublk? >>>>> >>>> I think one reason for the latency gap of sync I/O is that vduse uses >>>> workqueue in the I/O completion path but ublk doesn't. >>>> >>>> And one bottleneck for the async I/O in vduse is that vduse will do >>>> memcpy inside the critical section of virtqueue's spinlock in the >>>> virtio-blk driver. That will hurt the performance heavily when >>>> virtio_queue_rq() and virtblk_done() run concurrently. And it can be >>>> mitigated by the advance DMA mapping feature [1] or irq binding >>>> support [2]. >>> Hi Yongji, >>> >>> Yeah, that is the cost you paid for virtio. Wrt. userspace block device >>> or other sort of userspace devices, cmd completion is driven by >>> userspace, not sure if one such 'irq' is needed. >> I'm not sure, it can be an optional feature in the future if needed. >> >>> Even not sure if virtio >>> ring is one good choice for such use case, given io_uring has been proved >>> as very efficient(should be better than virtio ring, IMO). >>> >> Since vduse is aimed at creating a generic userspace device framework, >> virtio should be the right way IMO. > OK, it is the right way, but may not be the effective one. > >> And with the vdpa framework, the >> userspace device can serve both virtual machines and containers. > virtio is good for VM, but not sure it is good enough for other > cases. Well, virtio is not yet limited to virt and has been widely used in bare metal, containers, automotive and even edge in production environment for years. A lot of vendors has shipped their software or hardware virtio/vDPA products. > >> Regarding the performance issue, actually I can't measure how much of >> the performance loss is due to the difference between virtio ring and >> iouring. But I think it should be very small. The main costs come from >> the two bottlenecks I mentioned before which could be mitigated in the >> future. > Per my understanding, at least there are two places where virtio ring is > less efficient than io_uring: > > 1) io_uring uses standalone submission queue(SQ) and completion queue(CQ), > so no contention exists between submission and completion; but virtio queue > requires per-vq lock in both submission and completion. Virtio is not limited in its layout of the queue. I've used to proposed SQ/CQ model in the spec in the past but vendors complains a third format immediate after the second. Maybe it's time to revisit that, but it needs to be fully benchmarked and proved at first. > > 2) io_uring can use single system call of io_uring_enter() for both > submitting and completing, so one context switch is enough, together > with natural batch processing for both submission and completion, and > it is observed that dozens or more than one hundred of IOs can be > covered in single syscall; virtio requires one notification for submission and > another one for completion, You can queue several buffers before a kick to the virtqueue, with a polling device and driver, you don't even need any kick/notification. I don't see much difference here. > looks at least two context switch are required > for handling one IO(s). For virtio, the queue layout or ring design should not be bottleneck for the block device at least. I can give your some numbers measured by PPS (since network traffic is more queue layout sensitive than block): 1) vDPA vendor can achieve 30Mpps or even higher 2) software userspace virtio backends like vhost-user can do almost the same or even higher This is a strong hint that virtio ring should be sufficient for block. For NFV/wire-speed like 100G we do need more work on optimization on the queue/descriptor format. Thanks > >>> ublk uses io_uring pt cmd for handling both io submission and completion, >>> turns out the extra latency can be pretty small. >>> >>> BTW, one un-related topic, I saw the following words in >>> Documentation/userspace-api/vduse.rst: >>> >>> ``` >>> Note that only virtio block device is supported by VDUSE framework now, >>> which can reduce security risks when the userspace process that implements >>> the data path is run by an unprivileged user. >>> ``` >>> >>> But when I tried to start qemu-storage-daemon for creating vdpa-virtio >>> block by nor unprivileged user, 'Permission denied' is still returned, >>> can you explain a bit how to start such process by unprivileged user? >>> Or maybe I misunderstood the above words, please let me know. >>> >> Currently vduse should only allow privileged users by default. But >> sysadmin can change the permission of the vduse char device or pass >> the device fd to an unprivileged process IIUC. > I appreciate if you may provide a bit detailed steps for the above? > > BTW, I changed privilege of /dev/vduse/control to normal user, but > qemu-storage-daemon still returns 'Permission denied'. And if the > char dev is /dev/vduse/vduse0N, which is created by qemu-storage-daemon, > so how to change user of qemu-storage-daemon to unprivileged after > /dev/vduse/vduse0N is created? > > > > Thanks, > Ming >
Powered by blists - more mailing lists