linux-ext4 - Re: [PATCH] Add better example about how to compress e2image raw image

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4E80A73F.2070205@redhat.com>
Date:	Mon, 26 Sep 2011 11:24:31 -0500
From:	Eric Sandeen <sandeen@...hat.com>
To:	"Ted Ts'o" <tytso@....edu>
CC:	Carlos Maiolino <cmaiolino@...hat.com>,
	Andreas Dilger <adilger@...ger.ca>, linux-ext4@...r.kernel.org
Subject: Re: [PATCH] Add better example about how to compress e2image raw
 image

On 9/24/11 11:31 AM, Ted Ts'o wrote:
> On Fri, Sep 23, 2011 at 05:51:24PM -0300, Carlos Maiolino wrote:
>> On Fri, Sep 23, 2011 at 02:24:23PM -0600, Andreas Dilger wrote:
>>> On 2011-09-23, at 12:47 PM, Carlos Maiolino wrote:
>>>> The current example in the man page uses bzip2 to compress
>>>> the raw image file created by the e2image, but, bzip2 does
>>>> not honors sparse files, which causes the image to have the
>>>> same size of the filesystem.
>>>> Using tar together with bzip2 will make the compressed file
>>>> to honor the sparsed file, which makes it more transportable
>>>> than the current one if the filesystem is large.
> 
> The problem with using tar is that it requires extra disk space by the
> user --- somewhere a bit more than double the extra disk space
> (because you need to have space for the hda1.e2i file before it gets
> compressed).  For very large file systems, this can be quite
> significant.  My general philosophy has been to make things easy as
> possible for the users as being more important for the developers.
> 
> For the developers, we do have contrib/make-sparse.c.  All we have to do is:
> 
>     bunzip2 < hda1.e2i.bz2 | make-sparse hda1.e2i
> 
> ... and this creates a sparse file in hda1.e2i.

or | cp --sparse=always /dev/stdin sparse.img works too.

But have you ever tried this with a multi-terabyte image?

It takes -forever- to process all those 0s, with cpus pegged.

The tar command seems to actually annotate the sparseness efficiently.

Ted, your concern about space - it doesn't take the full fs size worth
of space, right, just the metadata space?  So in general it should not
be THAT much ...

-Eric

>>> Even better would be the use of the QCOW2 format that Lukas added,
>>> if it could also be operated on directly by the e2fsprogs utils (I
>>> don't know if that is possible or not).
>>>
>> The QCOW2 format can only be operated with qcow2 capable tools like
>>> qemu-img to use directly with e2fsprogs tools, we still need to
>>> use raw images.
> 
> Yeah, it would be nice if we had an io_manager implementation that
> understood qcow2, which could then be used by dumpe2fs and debugfs.
> Hopefully at some point someone will implement it.
> 
> 						- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html