linux-kernel - [PATCH 00/10] erofs: 48-bit layout support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250310095459.2620647-1-hsiangkao@linux.alibaba.com>
Date: Mon, 10 Mar 2025 17:54:50 +0800
From: Gao Xiang <hsiangkao@...ux.alibaba.com>
To: linux-erofs@...ts.ozlabs.org
Cc: LKML <linux-kernel@...r.kernel.org>,
	Gao Xiang <hsiangkao@...ux.alibaba.com>
Subject: [PATCH 00/10] erofs: 48-bit layout support

Hi folks,

The current 32-bit block addressing limits EROFS to a 16TiB maximum
volume size with 4KiB blocks.  However, several new use cases now
require larger capacity support:
 - Massive datasets for model training to boost random sampling
   performance for each epoch;
 - Object storage clients using EROFS direct passthrough.

This extends core on-disk structures to support 48-bit block addressing,
such as inodes, device slots, and inode chunks.

In addition, it introduces an mtime field to 32-byte compact inodes for
basic timestamp support, as well as expands the superblock root NID to
an 8-byte rootnid_8b for out-of-place update incremental builds.

In order to support larger images beyond 32-bit block addressing and
efficient indexing of large compression units for compressed data, and
to better support popular compression algorithms (mainly Zstd) that
still lack native fixed-sized output compression support, introduce
byte-oriented encoded extents, so that these compressors are allowed
to retain their current methods.

Therefore, it speeds up Zstd image building a lot by using:
Processor: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz * 96
Dataset: enwik9
Build time Size Type Command Line
3m52.339s 266653696 FO -C524288 -zzstd,22
3m48.549s 266174464 FO -E48bit -C524288 -zzstd,22
0m12.821s 272134144 FI -E48bit -C1048576 --max-extent-bytes=1048576 -zzstd,22

It has been stress-tested on my local setup for a while without any
strange happens.

Thanks,
Gao Xiang

Gao Xiang (10):
  erofs: get rid of erofs_map_blocks_flatmode()
  erofs: simplify erofs_{read,fill}_inode()
  erofs: add 48-bit block addressing on-disk support
  erofs: implement 48-bit block addressing for unencoded inodes
  erofs: support dot-omitted directories
  erofs: initialize decompression early
  erofs: add encoded extent on-disk definition
  erofs: implement encoded extent metadata
  erofs: support unaligned encoded data
  erofs: enable 48-bit layout support

 fs/erofs/Kconfig             |  14 +--
 fs/erofs/data.c              | 133 +++++++++++-------------
 fs/erofs/decompressor.c      |   2 +-
 fs/erofs/dir.c               |   7 +-
 fs/erofs/erofs_fs.h          | 191 ++++++++++++++++-------------------
 fs/erofs/inode.c             | 126 +++++++++++------------
 fs/erofs/internal.h          |  30 +++---
 fs/erofs/super.c             |  49 ++++-----
 fs/erofs/sysfs.c             |   2 +
 fs/erofs/zdata.c             |  96 +++++++++---------
 fs/erofs/zmap.c              | 166 +++++++++++++++++++++++++-----
 include/trace/events/erofs.h |   2 +-
 12 files changed, 455 insertions(+), 363 deletions(-)

-- 
2.43.5