linux-ext4 - Re: [PATCH] Add better example about how to compress e2image raw image

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAOQ4uxiiJk79nz-ftVPNpCY_VYxLAH8QGR8+p9aAa3fheGs54w@mail.gmail.com>
Date:	Mon, 26 Sep 2011 23:01:37 +0300
From:	Amir Goldstein <amir73il@...il.com>
To:	"Ted Ts'o" <tytso@....edu>
Cc:	Eric Sandeen <sandeen@...hat.com>,
	Carlos Maiolino <cmaiolino@...hat.com>,
	Andreas Dilger <adilger@...ger.ca>, linux-ext4@...r.kernel.org,
	Shardul Mangade <shardul27@...il.com>
Subject: Re: [PATCH] Add better example about how to compress e2image raw image

On Mon, Sep 26, 2011 at 10:23 PM, Ted Ts'o <tytso@....edu> wrote:
> On Mon, Sep 26, 2011 at 11:24:31AM -0500, Eric Sandeen wrote:
>> >
>> >     bunzip2 < hda1.e2i.bz2 | make-sparse hda1.e2i
>> >
>> > ... and this creates a sparse file in hda1.e2i.
>>
>> or | cp --sparse=always /dev/stdin sparse.img works too.
>>
>> But have you ever tried this with a multi-terabyte image?
>>
>> It takes -forever- to process all those 0s, with cpus pegged.
>
> Yeah, I didn't realize until I read another message on this thread
> that bzip2's CPU problems were causing problems.  Is gzip sufficiently
> better, I wonder, or is it still problematic?
>
>> Ted, your concern about space - it doesn't take the full fs size worth
>> of space, right, just the metadata space?  So in general it should not
>> be THAT much ...
>
> Yes, it's just the metadata space that I was worried about.  So it's
> not *that* much, but it still adds up on large systems.  But then
> again, on large systems we precisely have the problem of bzip2 taking
> forever.
>
> If we decide that we're OK with not compressing qcow2, we could use
> qcow2.  But note that the qcow2 format is still very compressible ---
> it looks like it could do a better job removing zero blocks.  (I had a
> 256meg qcow2 e2image file compress down to 9 megs.)  Unfortunately we
> can't do stream compression with qcow2.

I wasn't sure if I should bring this up, but what the hack...
Shardul, one of our GSoC students, have implemented e4send/e4receive
for streaming of ext4
snapshot image (with data) to a remote machine. The code can be found
in his github repo:
https://github.com/shardulmangade/e2fsprogs-snapshots

His code mostly reuses e2image code and uses "LVM snapshot store"
format for streaming
block numbers + blocks content to e4receive, which writes to a sparse
file or block device.
LVM snapshot store format was chosen simply because we needed
something quick and the code was already implemented by another GSoC student
for his own project (revert to ext4 snapshot using LVM merge).

So without trying to promote upstream inclusion of this implementation,
just so you know:
1. the code works, although not configured for exporting metadata at the moment
2. it's simple
3. no intermediate files needed
4. output can be streamed compressed
5. Shardul would be happy to help with further questions

Amir.

>
> Long run I think we should make the qcow2 support better (by dropping
> all-zero blocks, and adding support for qcow2 to
> debugfs/dumpe2fs/e2fsck, and perhaps adding support for native
> compression).  Anyone looking for a project?  :-)
>
>                                                - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html