linux-ext4 - Ext4 speedup by storing metadata and data on separate devices

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <50AB63B4.1050506@icdsoft.com>
Date:	Tue, 20 Nov 2012 13:04:20 +0200
From:	Ivan Zahariev <famzah@...soft.com>
To:	linux-ext4@...r.kernel.org
Subject: Ext4 speedup by storing metadata and data on separate devices

Hello all,

This suggestion is not about storing the journal on a separate device.

Many of the tasks on an Ext4 file-system require a full or massive scan 
of the metadata. A few examples:
- backup: you need to get a list with all "mtime" or "size" changed 
files since last backup
- reporting: you need to get a list with all files of a particular 
"group" owner ID
- delete: deleting the "/home/$user" of someone with lots of data and files

I know many efforts have been made to make the (meta)data operations 
"local" -- this speeds up spinning disks operations a lot, also SSD 
ones. However, having the whole metadata on an SSD disk (or a RAID1 of 
two such disks) could speed up many common tasks a lot. And the hardware 
price for such a benefit is really affordable now.

I see two possible implementations:

1. Re-work the Ext4 metadata operations (that work with inodes, etc) to 
read/write on a separate block device.

or

2. Add an option to the "data locality" algorithm to force it to store 
all metadata only at the beginning of a device (we can pre-allocate 
enough space). We can then transparently map in the DM those blocks to a 
separate faster block device, thus making the changes to Ext4 minimal.

Does all this make sense, or I'm missing something obvious?

Thank you for your time.

-- 
Best regards.
Ivan Zahariev | System Administrator | ICDSoft Ltd.

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html