Message-ID: <d23f8a90-34e1-4c3a-8eaa-2b095e1bea32@tiscali.it>
Date: Wed, 21 Aug 2024 11:27:07 +0200
From: Fabio Fantoni <fantonifabio@...cali.it>
To: Sergei Shtepa <sergei.shtepa@...ux.dev>, axboe@...nel.dk,
 hch@...radead.org, corbet@....net, snitzer@...nel.org
Cc: linux-block@...r.kernel.org, linux-doc@...r.kernel.org,
 linux-kernel@...r.kernel.org, linux-fsdevel@...r.kernel.org
Subject: Re: [PATCH v7 0/8] filtering and snapshots of a block devices

Il 09/02/2024 17:01, Sergei Shtepa ha scritto:
> Hi all.
>
> I am happy to offer an improved version of the block device filtering
> mechanism (blkfilter) and module for creating snapshots of block devices
> (blksnap).
>
> The filtering block device mechanism is implemented in the block layer.
> This allows to attach and detach block device filters. Filters extend the
> functionality of the block layer. See more in
> Documentation/block/blkfilter.rst.
>
> The main purpose of snapshots of block devices is to provide backups of
> them. See more in Documentation/block/blksnap.rst. The tool, library and
> tests for working with blksnap can be found on github.
> Link: https://github.com/veeam/blksnap/tree/stable-v2.0
> There is also documentation from which you can learn how to manage the
> module using the library and the console tool.
>
> Based on LK v6.8-rc3 with Christoph's patchset "clean up blk_mq_submit_bio".
> Link: https://lore.kernel.org/linux-block/50fbe76b-d77d-4a7e-bda4-3a3b754fbd7e@kernel.org/T/#t
>
> I express my appreciation and gratitude to Christoph. Thanks to his
> attention to the project, it was possible to raise the quality of the code.
> I probably wouldn't have made version 7 if it wasn't for his help.
> I am sure that the blksnap module will improve the quality of backup tools
> for Linux.
>
> v7 changes:
> - The location of the filtering of I/O units has been changed. This made it
>    possible to remove the additional call bio_queue_enter().
> - Remove configs BLKSNAP_DIFF_BLKDEV and BLKSNAP_CHUNK_DIFF_BIO_SYNC.
> - To process the ioctl, the switch statement is used instead of a table
>    with functions.
> - Instead of a file descriptor, the module gets a path on the file system.
>    This allows the kernel module to correctly open a file or block device
>    with exclusive access rights.
> - Fixed a bio leaking bugs.

Hi, looking 
https://lore.kernel.org/all/20240209160204.1471421-1-sergei.shtepa@linux.dev/ 
I don't see any response to this latest version of blksnap even though a 
lot of time has passed, and this seems strange to me.

 From what I see recipients were less to avoid being blocked by antispam 
and double shipments like previous versions, and there was an error in 
subjects from patch 3/8 to patch 8/8, where I think it should be 
"blksnap:" or "block/blksnap:" instead of "block:" at the beginning of 
the subjects.

However, the error in the subjects does not seem serious to me to avoid 
possible replies, or am I wrong?

Isn't there anyone who wants to comment or review?

I want to thank Sergei Shtepa for all his work and others people that 
helped with previous versions review and advices, especially Christoph 
Hellwig who helped so much, and I hope the project can proceed.

>
> v6 changes:
> - The difference storage has been changed.
>    In the previous version, the file was created only to reserve sector
>    ranges on a block device. The data was stored directly to the block
>    device in these sector ranges. Now saving and reading data is done using
>    'VFS' using vfs_iter_write() and vfs_iter_read() functions. This allows
>    not to depend on the filesystem and use, for example, tmpfs. Using an
>    unnamed temporary file allows hiding it from other processes and
>    automatically release it when the snapshot is closed.
>    However, now the module does not allow adding a block device to the
>    snapshot on which the difference storage is located. There is no way to
>    ensure the immutability of file metadata when writing data to a file.
>    This means that the metadata of the filesystem may change, which may
>    cause damage to the snapshot.
> - _IOW and _IOR were mixed up - fixed.
> - Protection against the use of the snapshots for block devices with
>    hardware inline encryption and data integrity was implemented.
>    Compatibility with them was not planned and has not been tested at the
>    moment.
>
> v5 changes:
> - Rebase for "kernel/git/axboe/linux-block.git" branch "for-6.5/block".
>    Link: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-6.5/block
>
> v4 changes:
> - Structures for describing the state of chunks are allocated dynamically.
>    This reduces memory consumption, since the struct chunk is allocated only
>    for those blocks for which the snapshot image state differs from the
>    original block device.
> - The algorithm for calculating the chunk size depending on the size of the
>    block device has been changed. For large block devices, it is now
>    possible to allocate a larger number of chunks, and their size is smaller.
> - For block devices, a 'filter' file has been added to /sys/block/<device>.
>    It displays the name of the filter that is attached to the block device.
> - Fixed a problem with the lack of protection against re-adding a block
>    device to a snapshot.
> - Fixed a bug in the algorithm of allocating the next bio for a chunk.
>    This problem was occurred on large disks, for which a chunk consists of
>    at least two bio.
> - The ownership mechanism of the diff_area structure has been changed.
>    This fixed the error of prematurely releasing the diff_area structure
>    when destroying the snapshot.
> - Documentation corrected.
> - The Sparse analyzer is passed.
> - Use __u64 type instead pointers in UAPI.
>
> v3 changes:
> - New block device I/O controls BLKFILTER_ATTACH and BLKFILTER_DETACH allow
>    to attach and detach filters.
> - New block device I/O control BLKFILTER_CTL allow sending command to
>    attached block device filter.
> - The copy-on-write algorithm for processing I/O units has been optimized
>    and has become asynchronous.
> - The snapshot image reading algorithm has been optimized and has become
>    asynchronous.
> - Optimized the finite state machine for processing chunks.
> - Fixed a tracking block size calculation bug.
>
> v2 changes:
> - Added documentation for Block Device Filtering Mechanism.
> - Added documentation for Block Devices Snapshots Module (blksnap).
> - The MAINTAINERS file has been updated.
> - Optimized queue code for snapshot images.
> - Fixed comments, log messages and code for better readability.
>
> v1 changes:
> - Forgotten "static" declarations have been added.
> - The text of the comments has been corrected.
> - It is possible to connect only one filter, since there are no others in
>    upstream.
> - Do not have additional locks for attach/detach filter.
> - blksnap.h moved to include/uapi/.
> - #pragma once and commented code removed.
> - uuid_t removed from user API.
> - Removed default values for module parameters from the configuration file.
> - The debugging code for tracking memory leaks has been removed.
> - Simplified Makefile.
> - Optimized work with large memory buffers, CBT tables are now in virtual
>    memory.
> - The allocation code of minor numbers has been optimized.
> - The implementation of the snapshot image block device has been
>    simplified, now it is a bio-based block device.
> - Removed initialization of global variables with null values.
> - only one bio is used to copy one chunk.
> - Checked on ppc64le.
>
> Sergei Shtepa (8):
>    documentation: filtering and snapshots of a block devices
>    block: filtering of a block devices
>    block: header file of the blksnap module interface
>    block: module management interface functions
>    block: handling and tracking I/O units
>    block: difference storage implementation
>    block: snapshot and snapshot image block device
>    block: Kconfig, Makefile and MAINTAINERS files
>
>   Documentation/block/blkfilter.rst             |  66 ++
>   Documentation/block/blksnap.rst               | 351 ++++++++++
>   Documentation/block/index.rst                 |   2 +
>   .../userspace-api/ioctl/ioctl-number.rst      |   1 +
>   MAINTAINERS                                   |  17 +
>   block/Makefile                                |   3 +-
>   block/bdev.c                                  |   2 +
>   block/blk-core.c                              |  26 +-
>   block/blk-filter.c                            | 257 +++++++
>   block/blk-mq.c                                |   7 +-
>   block/blk-mq.h                                |   2 +-
>   block/blk.h                                   |  11 +
>   block/genhd.c                                 |  10 +
>   block/ioctl.c                                 |   7 +
>   block/partitions/core.c                       |   9 +
>   drivers/block/Kconfig                         |   2 +
>   drivers/block/Makefile                        |   2 +
>   drivers/block/blksnap/Kconfig                 |  12 +
>   drivers/block/blksnap/Makefile                |  15 +
>   drivers/block/blksnap/cbt_map.c               | 225 +++++++
>   drivers/block/blksnap/cbt_map.h               |  90 +++
>   drivers/block/blksnap/chunk.c                 | 631 ++++++++++++++++++
>   drivers/block/blksnap/chunk.h                 | 134 ++++
>   drivers/block/blksnap/diff_area.c             | 577 ++++++++++++++++
>   drivers/block/blksnap/diff_area.h             | 175 +++++
>   drivers/block/blksnap/diff_buffer.c           | 114 ++++
>   drivers/block/blksnap/diff_buffer.h           |  37 +
>   drivers/block/blksnap/diff_storage.c          | 290 ++++++++
>   drivers/block/blksnap/diff_storage.h          | 103 +++
>   drivers/block/blksnap/event_queue.c           |  81 +++
>   drivers/block/blksnap/event_queue.h           |  64 ++
>   drivers/block/blksnap/main.c                  | 481 +++++++++++++
>   drivers/block/blksnap/params.h                |  16 +
>   drivers/block/blksnap/snapimage.c             | 135 ++++
>   drivers/block/blksnap/snapimage.h             |  10 +
>   drivers/block/blksnap/snapshot.c              | 462 +++++++++++++
>   drivers/block/blksnap/snapshot.h              |  65 ++
>   drivers/block/blksnap/tracker.c               | 369 ++++++++++
>   drivers/block/blksnap/tracker.h               |  78 +++
>   include/linux/blk-filter.h                    |  72 ++
>   include/linux/blk_types.h                     |   1 +
>   include/linux/sched.h                         |   1 +
>   include/uapi/linux/blk-filter.h               |  35 +
>   include/uapi/linux/blksnap.h                  | 384 +++++++++++
>   include/uapi/linux/fs.h                       |   3 +
>   45 files changed, 5430 insertions(+), 5 deletions(-)
>   create mode 100644 Documentation/block/blkfilter.rst
>   create mode 100644 Documentation/block/blksnap.rst
>   create mode 100644 block/blk-filter.c
>   create mode 100644 drivers/block/blksnap/Kconfig
>   create mode 100644 drivers/block/blksnap/Makefile
>   create mode 100644 drivers/block/blksnap/cbt_map.c
>   create mode 100644 drivers/block/blksnap/cbt_map.h
>   create mode 100644 drivers/block/blksnap/chunk.c
>   create mode 100644 drivers/block/blksnap/chunk.h
>   create mode 100644 drivers/block/blksnap/diff_area.c
>   create mode 100644 drivers/block/blksnap/diff_area.h
>   create mode 100644 drivers/block/blksnap/diff_buffer.c
>   create mode 100644 drivers/block/blksnap/diff_buffer.h
>   create mode 100644 drivers/block/blksnap/diff_storage.c
>   create mode 100644 drivers/block/blksnap/diff_storage.h
>   create mode 100644 drivers/block/blksnap/event_queue.c
>   create mode 100644 drivers/block/blksnap/event_queue.h
>   create mode 100644 drivers/block/blksnap/main.c
>   create mode 100644 drivers/block/blksnap/params.h
>   create mode 100644 drivers/block/blksnap/snapimage.c
>   create mode 100644 drivers/block/blksnap/snapimage.h
>   create mode 100644 drivers/block/blksnap/snapshot.c
>   create mode 100644 drivers/block/blksnap/snapshot.h
>   create mode 100644 drivers/block/blksnap/tracker.c
>   create mode 100644 drivers/block/blksnap/tracker.h
>   create mode 100644 include/linux/blk-filter.h
>   create mode 100644 include/uapi/linux/blk-filter.h
>   create mode 100644 include/uapi/linux/blksnap.h
>

Powered by blists - more mailing lists