Message-Id: <20250328060853.4124527-1-yukuai1@huaweicloud.com>
Date: Fri, 28 Mar 2025 14:08:39 +0800
From: Yu Kuai <yukuai1@...weicloud.com>
To: hch@....de,
	xni@...hat.com,
	colyli@...nel.org,
	axboe@...nel.dk,
	agk@...hat.com,
	snitzer@...nel.org,
	mpatocka@...hat.com,
	song@...nel.org,
	yukuai3@...wei.com
Cc: linux-block@...r.kernel.org,
	linux-kernel@...r.kernel.org,
	dm-devel@...ts.linux.dev,
	linux-raid@...r.kernel.org,
	yukuai1@...weicloud.com,
	yi.zhang@...wei.com,
	yangerkun@...wei.com
Subject: [PATCH RFC v2 00/14] md: introduce a new lockless bitmap

From: Yu Kuai <yukuai3@...wei.com>

#### Background

Redundant data is used to enhance data fault tolerance, and the storage
method for redundant data vary depending on the RAID levels. And it's
important to maintain the consistency of redundant data.

Bitmap is used to record which data blocks have been synchronized and which
ones need to be resynchronized or recovered. Each bit in the bitmap
represents a segment of data in the array. When a bit is set, it indicates
that the multiple redundant copies of that data segment may not be
consistent. Data synchronization can be performed based on the bitmap after
power failure or readding a disk. If there is no bitmap, a full disk
synchronization is required.

#### Key Concept

##### State Machine

Each bit is one byte, contain 6 difference state, see llbitmap_state. And
there are total 8 differenct actions, see llbitmap_action, can change state:

llbitmap state machine: transitions between states

|           | Startwrite | Startsync | Endsync | Abortsync| Reload   | Daemon | Discard   | Stale     |
| --------- | ---------- | --------- | ------- | -------  | -------- | ------ | --------- | --------- |
| Unwritten | Dirty      | x         | x       | x        | x        | x      | x         | x         |
| Clean     | Dirty      | x         | x       | x        | x        | x      | Unwritten | NeedSync  |
| Dirty     | x          | x         | x       | x        | NeedSync | Clean  | Unwritten | NeedSync  |
| NeedSync  | x          | Syncing   | x       | x        | x        | x      | Unwritten | x         |
| Syncing   | x          | Syncing   | Dirty   | NeedSync | NeedSync | x      | Unwritten | NeedSync  |

special illustration:
- Unwritten is special state, which means user never write data, hence there
  is no need to resync/recover data. This is safe if user create filesystems
  for the array, filesystem will make sure user will get zero data for
  unwritten blocks.
- After resync is done, change state from Syncing to Dirty first, in case
  Startwrite happen before the state is Clean.

##### Bitmap IO

A hidden disk, named mdxxx_bitmap, is created for bitmap, see details in
llbitmap_add_disk(). And a file is created as well to manage bitmap IO for
this disk, see details in llbitmap_open_disk(). Read/write bitmap is
converted to buffer IO to this file.

IO fast path will set bits to dirty, and those dirty bits will be cleared
by daemon after IO is done. llbitmap_barrier is used to syncronize between
IO path and daemon;

Test result: to be added

Noted:
1) user must apply the following mdadm patch, and then llbitmap can be
enabled by --bitmap=lockless
https://lore.kernel.org/all/20250327134853.1069356-1-yukuai1@huaweicloud.com/
2) this set is cooked on the top of my other set:
https://lore.kernel.org/all/20250219083456.941760-1-yukuai1@huaweicloud.com/

Yu Kuai (14):
  block: factor out a helper bdev_file_alloc()
  md/md-bitmap: pass discard information to bitmap_{start, end}write
  md/md-bitmap: remove parameter slot from bitmap_create()
  md: add a new sysfs api bitmap_version
  md: delay registeration of bitmap_ops until creating bitmap
  md/md-llbitmap: implement bit state machine
  md/md-llbitmap: implement hidden disk to manage bitmap IO
  md/md-llbitmap: implement APIs for page level dirty bits
    synchronization
  md/md-llbitmap: implement APIs to mange bitmap lifetime
  md/md-llbitmap: implement APIs to dirty bits and clear bits
  md/md-llbitmap: implement APIs for sync_thread
  md/md-llbitmap: implement all bitmap operations
  md/md-llbitmap: implement sysfs APIs
  md/md-llbitmap: add Kconfig

 block/bdev.c             |   21 +-
 drivers/md/Kconfig       |   12 +
 drivers/md/Makefile      |    1 +
 drivers/md/md-bitmap.c   |   10 +-
 drivers/md/md-bitmap.h   |   21 +-
 drivers/md/md-llbitmap.c | 1410 ++++++++++++++++++++++++++++++++++++++
 drivers/md/md.c          |  180 ++++-
 drivers/md/md.h          |    3 +
 include/linux/blkdev.h   |    1 +
 9 files changed, 1614 insertions(+), 45 deletions(-)
 create mode 100644 drivers/md/md-llbitmap.c

-- 
2.39.2

Powered by blists - more mailing lists