linux-kernel - Re: [BUG?] bcachefs: keep writing to device when there is no high-level I/O activity.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <xzz5l6mvyyoqcjx7t5tq4hznqyhmyec3fqfcqoxyqvbecmribc@4vw3gnt5b46d>
Date: Thu, 5 Sep 2024 00:26:11 -0400
From: Kent Overstreet <kent.overstreet@...ux.dev>
To: David Wang <00107082@....com>
Cc: linux-bcachefs@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [BUG?] bcachefs: keep writing to device when there is no
 high-level I/O activity.

On Tue, Aug 27, 2024 at 05:49:33PM GMT, David Wang wrote:
> Hi,
> 
> I was using two partitions on same nvme device to compare filesystem performance,
> and I consistantly observed a strange behavior:
> 
> After 10 minutes fio test with bcachefs on one partition, performance degrade
> significantly for other filesystems on other partition (same device).
> 
> 	ext4  150M/s --> 143M/s
> 	xfs   150M/s --> 134M/s
> 	btrfs 127M/s --> 108M/s
> 
> Several round tests show the same pattern that bcachefs seems occupy some device resource
> even when there is no high-level I/O.
> 
> I monitor /proc/diskstats, and it confirmed that bcachefs do keep writing the device.
> Following is the time serial samples for "writes_completed" on my bcachefs partition:
> 
> writes_completed @timestamp
> 	       0 @1724748233.712
> 	       4 @1724748248.712    <--- mkfs
> 	       4 @1724748263.712
> 	      65 @1724748278.712
> 	   25350 @1724748293.712
> 	   63839 @1724748308.712    <--- fio started
>   	  352228 @1724748323.712
> 	  621350 @1724748338.712
> 	  903487 @1724748353.712
>         ...
> 	12790311 @1724748863.712
> 	13100041 @1724748878.712
> 	13419642 @1724748893.712
> 	13701685 @1724748908.712    <--- fio done (10minutes)
> 	13701769 @1724748923.712    <--- from here, average 5~7writes/second for 2000 seconds
> 	13701852 @1724748938.712
> 	13701953 @1724748953.712
> 	13702032 @1724748968.712
> 	13702133 @1724748983.712
> 	13702213 @1724748998.712
> 	13702265 @1724749013.712
> 	13702357 @1724749028.712
>         ...
> 	13712984 @1724750858.712
> 	13713076 @1724750873.712
> 	13713196 @1724750888.712
> 	13713299 @1724750903.712
> 	13713386 @1724750918.712
> 	13713463 @1724750933.712
> 	13713501 @1724750948.712   <--- writes stopped here
> 	13713501 @1724750963.712
> 	13713501 @1724750978.712
> 	...
> 
> Is this behavior expected? 

We've got tracepoints and counters (in sysfs) for checking what's doing
the writing.

If it's a moderate amount of IO, it's probably journal reclaim; it
trickles out writes if the journal has room (it's trying to spread out
bursty work). If it's a lot of IO, it might be a bug.

Changing journal reclaim behaviour (and other background tasks) is on
the todo list, the current behaviour is not great for machines where we
want to "race to idle" to save power.