linux-kernel - Re: [PATCH] erofs-utils: avoid redundant memcpy and sha256() for dedupe

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250815095449.4163442-1-wangzijie1@honor.com>
Date: Fri, 15 Aug 2025 17:54:49 +0800
From: wangzijie <wangzijie1@...or.com>
To: <hsiangkao@...ux.alibaba.com>
CC: <bintian.wang@...or.com>, <feng.han@...or.com>,
	<linux-erofs@...ts.ozlabs.org>, <linux-kernel@...r.kernel.org>,
	<stopire@...il.com>, <wangzijie1@...or.com>, <xiang@...nel.org>,
	<zhaoyifan28@...wei.com>
Subject: Re: [PATCH] erofs-utils: avoid redundant memcpy and sha256() for dedupe


> Hi Zijie,
> 
> On 2025/8/15 16:44, wangzijie wrote:
> > We have already use xxh64() for filtering first for dedupe, when we
> > need to skip the same xxh64 hash, no need to do memcpy and sha256(),
> > relocate the code to avoid it.
> > 
> > Signed-off-by: wangzijie <wangzijie1@...or.com>
> 
> Thanks for the patch, it makes sense to me since we only keep one
> record according to xxh64 (instead of sha256) for now:
> 
> Reviewed-by: Gao Xiang <hsiangkao@...ux.alibaba.com>
> 
> Although I think multi-threaded deduplication is more useful, see:
> https://github.com/erofs/erofs-utils/issues/25
> but I'm not sure if you're interested in it... ;-)

Hi Xiang,
Thank you for providing this information, I want to optimize mkfs time with
dedupe option and send this patch. I will find time to research Yifan's demo
of multi-threaded deduplication and try to provide some help.

> Thanks,
> Gao Xiang