linux-kernel - Re: [PATCH RFC 0/4] erofs-utils: Add --meta_fix and --meta

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <406b4b4e-1a1e-4618-87b8-7b104838770f@huawei.com>
Date: Tue, 22 Apr 2025 22:37:19 +0800
From: Hongbo Li <lihongbo22@...wei.com>
To: Gao Xiang <hsiangkao@...ux.alibaba.com>, <xiang@...nel.org>,
	<chao@...nel.org>, <huyue2@...lpad.com>, <jefflexu@...ux.alibaba.com>
CC: <linux-erofs@...ts.ozlabs.org>, <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH RFC 0/4] erofs-utils: Add --meta_fix and --meta_only
 format options



On 2025/4/22 21:50, Gao Xiang wrote:
> Hi Hongbo,
> 
> On 2025/4/22 20:36, Hongbo Li wrote:
>> In this patchset, we have added two formatting options --meta_fix and
>> --meta_only to extend the ability of EROFS. In the case of using OBS,
>> we can convert the directory tree structure from OBS into the erofs
>> image and implement on-demand loading logic based on this. Since OBS
>> objects are often large, we need to separate the metadata area from
>> the data area, which is the reason we introduce the --meta_fix option.
>> To accelerate the formatting process, we can skip the formatting of
>> the raw data by adding --meta_only option.
> 
> Thanks for the patches!
> 
> I wonder if it's possible to reuse blobchunk.c codebase for
> such usage in the short term.
> 
Our initial plan was to reuse the blobchunk.c logic, but we found that 
the chunk-based layout has some minor issues when handling contiguous 
blocks—it would result in multiple elements in the chunk index array 
(whereas blobraw expects only oversized chunks).

Thanks,
Hongbo

> Also I hope there could be some better option knobs for these
> new features.
> 
>>
>> A simple usage example is as follows:
>>    1. Build one xattr with OBS key in s3fs.
>>    2. mkfs.erofs --meta_fix --meta_only data.img /mnt/s3fs to format
>>    3. Implement the loading logic in kernel or userspace.
>>
>> Based on the above logic, we can easily expose the directory tree
>> from OBS to users in the form of the EROFS file system and implement
>> on-demand data loading for large OBS objects.
> 
> BTW, It's possible to upstream OBS implementation to erofs-utils
> if any chance?
> 
> Thanks,
> Gao Xiang