linux-ext4 - resize2fs memory footprint

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <4B78FE19.5040208@miray.de>
Date:	Mon, 15 Feb 2010 08:56:09 +0100
From:	Ulrich Bauer <u.bauer@...ay.de>
To:	linux-ext4@...r.kernel.org
Subject: resize2fs memory footprint

We develop a deployment solution that is able to resize up to 16 drives 
at a time. While testing, we observed that resize2fs' memory usage 
scales with the volume size. Even in case of relatively small file 
systems of about 1.5TB, resize2fs would need about 135MB of application 
memory. Wit 16 large drives, one would need some GB of RAM just to 
resize the drives. Our guess is that this is related to the super block 
group bitmaps of ext4 that are held in memory to manipulate/zero their 
on-disc counterparts. A quick observation of resize2fs' source revealed 
that the latest git tree already has a bitmap interface that allows for 
different implementations of the bitmap manipulation algorithms. To 
solve the memory usage problem, two solutions seem to be feasible:

- For the huge bitmaps that already exist on the volume, we could create 
a disk-backed bitmap implementation that would only cache a small 
working set of the entire bitmap. My guess is that we can implement this 
efficiently enough to not loose too much performance.
- For the all-zero bitmaps, we could implement a dummy bitmap that 
forces all bits to zero and spawns a real bitmap as soon as any bits are 
set to one. An alternative would be a tree-based aproach that works 
especially well when the bitmap is just sparsely set.

I'd be glad if one of the developers involved in the ext4 development 
could tell me if these thoughts make sense and if yes, are there any 
plans to incorporate these approaches into the ext library anytime soon 
or does it make sense if I would have a deeper look into these issues 
and implement them? Thanks in advance for any thoughts about this.

Sincerely
U. Bauer

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html