Message-ID: <20250905103553.utwmp6rpi5rfmvsd@BitWizard.nl>
Date: Fri, 5 Sep 2025 12:35:53 +0200
From: Rogier Wolff <R.E.Wolff@...Wizard.nl>
To: linux-kernel@...r.kernel.org
Subject: Deleting a bunch of files takes long.

Hi all, 

I have a little linux-server machine that has two 8T harddrives. I
noticed that they were becoming a bit "full", so I decided to remove a
bunch of files. OK. Maybe a lot of files. About 121 million of them.

How long do you estimate that this would take? Ten minutes? An hour? 

I'm using EXT4. 

Now think about how much metadata needs to be read from the disk to
know what inodes to wipe, and directories to clear... 

So a file takes a directory entry of say 30 bytes? (depends on the
filename). How about we round that to 50. Then an inode. That used to
be 128 bytes, but nowadays 256 right? Addded together we're talking
about about 300 bytes, right? So the metadata is 36Gb. 

Let's round the disk speed to 100Mb/sec. So the OS will get to "touch"
all of the necessary metadata in about 360 seconds. Add in some
seeking and we're talking about 10 minutes. to an hour. Right?

Weeeeelllll.... its been running for over a week now. 

I would think that improving this, also improves things like
find . -name zasdflasdkjf 

I'm hoping that someone with the right skills agrees with me: "this is
absurd" and has the time to do something about it. 

To reproduce: I backup my homedir of about 700k files every day using
rsync. Then I cp -lr the whole thing to a new directory with the
timestamp as the name. I have about 120-200 copies when the disk fills
up and then I batch delete a bunch.

Why do I think that it should be technically possible to achieve
"reasonable" performance? 

I think that the directory contents needs to be clustered better. And
then when the system needs to read a directory it should prefetch more
directories, say as if there is a "blocksize" of 128k or even 1M. Even
if the "hit ratio" of the "extra 990k" is only 10% you're already
gaining a lot.

To get better clustering the system may need to keep "stats" about the
filesystem in the superblock. What percentage of the disk is dedicated
to directory contents? If, say (as I suspect in my case) that's 30%,
of each block group, reserve the first 30% for directories
only. i.e. allocate directory blocks from there, allocate data-blocks
from not-there.

Or grow the two different kinds of data from different ends. 

Maybe "readahead" is better served if inodeds are allocated from the
end of the inode area. All this would not require a significant
rewrite of the filesystem, just the place in the superblock where the 
current stats are stored on each unmount. 

Another strategy might be to allocate the directory info in the current
block group top-down from the previous block-group. 


	Roger. 

-- 
** R.E.Wolff@...Wizard.nl ** https://www.BitWizard.nl/ ** +31-15-2049110 **
** Verl. Spiegelmakerstraat 37 2645 LZ  Delfgauw, The Netherlands.  
** KVK: 27239233    **
f equals m times a. When your f is steady, and your m is going down
your a** is going up.  -- Chris Hadfield about flying up the space shuttle.
**  'a' for accelleration.

Powered by blists - more mailing lists