linux-kernel - Re: [BUG?] bcachefs performance: read is way too slow when a file has no overwrite.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ka3sjrka6dugdaab2bvewfbonc3ksixumue3hs2juhajhjm37w@bnxvz5mozpgr>
Date: Fri, 6 Sep 2024 13:38:11 -0400
From: Kent Overstreet <kent.overstreet@...ux.dev>
To: David Wang <00107082@....com>
Cc: linux-bcachefs@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [BUG?] bcachefs performance: read is way too slow when a file
 has no overwrite.

On Fri, Sep 06, 2024 at 11:43:54PM GMT, David Wang wrote:
> 
> Hi,
> 
> I notice a very strange performance issue:
> When run `fio direct randread` test on a fresh new bcachefs, the performance is very bad:
> 	fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test  --bs=4k --iodepth=64 --size=1G --readwrite=randread  --runtime=600 --numjobs=8 --time_based=1
> 	...
> 	Run status group 0 (all jobs):
> 	   READ: bw=87.0MiB/s (91.2MB/s), 239B/s-14.2MiB/s (239B/s-14.9MB/s), io=1485MiB (1557MB), run=15593-17073msec
> 
> But if the files already exist and have alreay been thoroughly overwritten, the read performance is about 850MB+/s,
> almost 10-times better!
> 
> This means, if I copy some file from somewhere else, and make read access only afterwards, I would get really bad performance.
> (I copy files from other filesystem, and run fio read test on those files, the performance is indeed bad.)
> Copy some prepared files, and make readonly usage afterwards, this usage scenario is quite normal for lots of apps, I think.

That's because checksums are at extent granularity, not block: if you're
doing O_DIRECT reads that are smaller than the writes the data was
written with, performance will be bad because we have to read the entire
extent to verify the checksum.

block granular checksums will come at some point, as an optional feature
(most of the time you don't want them, and you'd prefer more compact
metadata)