Message-ID: <fd926012-6845-05e4-077b-6c8cfbf3d3cc@linux.alibaba.com>
Date:   Mon, 27 Jun 2022 16:20:55 +0800
From:   Ziyang Zhang <ZiyangZhang@...ux.alibaba.com>
To:     Ming Lei <ming.lei@...hat.com>
Cc:     linux-block@...r.kernel.org, io-uring@...r.kernel.org,
        linux-kernel@...r.kernel.org,
        Gabriel Krisman Bertazi <krisman@...labora.com>,
        Xiaoguang Wang <xiaoguang.wang@...ux.alibaba.com>,
        joseph.qi@...ux.alibaba.com,
        Ziyang Zhang <ZiyangZhang@...ux.alibaba.com>
Subject: [RFC] libubd: library for ubd(userspace block driver based on
 io_uring passthrough)

Hi Ming,

We are learning your ubd code and developing a library: libubd for ubd.
This article explains why we need libubd and how we design it.

Related threads:
(1) https://lore.kernel.org/all/Yk%2Fn7UtGK1vVGFX0@T590/
(2) https://lore.kernel.org/all/YnDhorlKgOKiWkiz@T590/
(3) https://lore.kernel.org/all/20220509092312.254354-1-ming.lei@redhat.com/
(4) https://lore.kernel.org/all/20220517055358.3164431-1-ming.lei@redhat.com/


Userspace block driver(ubd)[1], based on io_uring passthrough,
allows users to define their own backend storage in userspace
and provides block devices such as /dev/ubdbX.
Ming Lei has provided kernel driver code: ubd_drv.c[2]
and userspace code: ubdsrv[3].

ubd_drv.c simply passes all blk-mq IO requests
to ubdsrv through io_uring sqes/cqes. We think the kernel code
is pretty well-designed.

ubdsrv is implemented by a single daemon
and target(backend) IO handling(null_tgt and loop_tgt) 
is embedded in the daemon. 
While trying ubdsrv, we find ubdsrv is hard to be used 
by our backend.
First is description of our backend:

(1) a distributing system sends/receives IO requests 
    through network.

(2) The system use RPC calls among hundreds of
     storage servers and RPC calls are associated with data buffers
     allocated from a memory pool.

(3) On each server for each device(/dev/vdX), our backend runs
     many threads to handle IO requests and manage the device. 

Second are reasons why ubdsrv is hard to use for us:

(1) ubdsrv requires the target(backend) issues IO requests
    to the io_uring provided by ubdsrv but our backend 
    uses something like RPC and does not support io_uring.

(2) ubdsrv forks a daemon and it takes over everything.
    Users should type "list/stop/del" ctrl-commands to interact with
    the daemon. It is inconvenient for our backend
    because it has threads(from a C++ thread library) running inside.

(3) ubdsrv PRE-allocates internal data buffers for each ubd device.
    The data flow is:
    bio vectors <-1-> ubdsrv data buffer <-2-> backend buffer(our RPC buffer).
    Since ubdsrv does not export its internal data buffer to backend,
    the second copy is unavoidable. 
    PRE-allocating data buffer may not be a good idea for wasting memory
    if there are hundreds of ubd devices(/dev/ubdbX).

To better use ubd in more complicated scenarios, we have developed libubd.
It does not assume implementation of backend and can be embedded into it.
We refer to the code structure of tcmu-runner[4], 
which includes a library(libtcmu) for users 
to embed tcmu-runner inside backend's code. 
It:

(1) Does not fork/pthread_create but embedded in backend's threads

(2) Provides libubd APIs for backend to add/delete ubd devices 
    and fetch/commit IO requests

(3) simply passes backend-provided data buffers to ubd_drv.c in kernel,
    since the backend actually has no knowledge 
    on incoming data size until it gets an IO descriptor.

Note: 

(1) libubd is just a POC demo and is not stick to the principles of
    designing a library and we are still developing it now...

(2) The repo[5] including some useful examples using libubd. 

(3) We modify the kernel part: ubd_drv.c and 
    it[6] is against Ming Lei's newest branch[2]
    because we forked our branch from his early branch
    (v5.17-ubd-dev).

Thanks,
Zhang

[1]https://lore.kernel.org/all/Yk%2Fn7UtGK1vVGFX0@T590/
[2]https://github.com/ming1/linux/tree/my_for-5.19-ubd-devel_v3
[3]https://github.com/ming1/ubdsrv
[4]https://github.com/open-iscsi/tcmu-runner
[5]https://github.com/old-memories/libubd
[6]https://github.com/old-memories/linux/tree/v5.17-ubd-dev-mq-ubuf

Powered by blists - more mailing lists