linux-ext4 - Re: [PATCH] Add better example about how to compress e2image raw image

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 26 Sep 2011 14:22:28 +0200
From:	Bernd Schubert <bernd.schubert@...m.fraunhofer.de>
To:	"Ted Ts'o" <tytso@....edu>
CC:	Carlos Maiolino <cmaiolino@...hat.com>,
	Andreas Dilger <adilger@...ger.ca>, linux-ext4@...r.kernel.org
Subject: Re: [PATCH] Add better example about how to compress e2image raw
 image

On 09/24/2011 06:31 PM, Ted Ts'o wrote:
> On Fri, Sep 23, 2011 at 05:51:24PM -0300, Carlos Maiolino wrote:
>> On Fri, Sep 23, 2011 at 02:24:23PM -0600, Andreas Dilger wrote:
>>> On 2011-09-23, at 12:47 PM, Carlos Maiolino wrote:
>>>> The current example in the man page uses bzip2 to compress
>>>> the raw image file created by the e2image, but, bzip2 does
>>>> not honors sparse files, which causes the image to have the
>>>> same size of the filesystem.
>>>> Using tar together with bzip2 will make the compressed file
>>>> to honor the sparsed file, which makes it more transportable
>>>> than the current one if the filesystem is large.
>
> The problem with using tar is that it requires extra disk space by the
> user --- somewhere a bit more than double the extra disk space
> (because you need to have space for the hda1.e2i file before it gets
> compressed).  For very large file systems, this can be quite
> significant.  My general philosophy has been to make things easy as
> possible for the users as being more important for the developers.
>
> For the developers, we do have contrib/make-sparse.c.  All we have to do is:
>
>      bunzip2<  hda1.e2i.bz2 | make-sparse hda1.e2i
>
> ... and this creates a sparse file in hda1.e2i.

The problem is that the bzip2 run will take a huge amount of time to 
compress all the zeros. In 2009 (with a recent CPU of that time) I 
aborted such a run for a 8TiB file system after a couple of days, then 
stored the e2image directly on disk and compressed it with tar and 
sparse support, which finished after only 12 hours...
I don't think more modern CPUs are much faster for single threaded runs 
as bzip2 does it.
So IMHO the man page should at least warn about that issue and suggest 
to use a similar tar command.


Cheers,
Bernd
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html