linux-kernel - Re: Reiser4 und LZO compression

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 29 Aug 2006 04:09:24 -0700
From:	"Ray Lee" <madrabbit@...il.com>
To:	"Nigel Cunningham" <ncunningham@...uxmail.org>
Cc:	"David Masover" <ninja@...phack.com>,
	"Jan Engelhardt" <jengelh@...ux01.gwdg.de>,
	"Edward Shishkin" <edward@...esys.com>,
	"Stefan Traby" <stefan@...lo-penguin.com>,
	"Hans Reiser" <reiser@...esys.com>,
	"Alexey Dobriyan" <adobriyan@...il.com>, reiserfs-list@...esys.com,
	linux-kernel@...r.kernel.org, "Andrew Morton" <akpm@...l.org>
Subject: Re: Reiser4 und LZO compression

On 8/29/06, Nigel Cunningham <ncunningham@...uxmail.org> wrote:
> Hi.
> On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote:
> > Nigel Cunningham wrote:
> > > We used gzip when we first implemented compression support, and found it
> > > to be far too slow. Even with the fastest compression options, we were
> > > only getting a few megabytes per second. Perhaps I did something wrong
> > > in configuring it, but there's not that many things to get wrong!
> >
> > All that comes to mind is the speed/quality setting -- the number from 1
> > to 9.  Recently, I backed up someone's hard drive using -1, and I
> > believe I was still able to saturate... the _network_.  Definitely try
> > again if you haven't changed this, but I can't imagine I'm the first
> > persson to think of it.
> >
> >  From what I remember, gzip -1 wasn't faster than the disk.  But at
> > least for (very) repetitive data, I was wrong:
> >
> > eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync'
> > 10+0 records in
> > 10+0 records out
> > 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec)
> >
> > real    0m3.746s
> > user    0m0.005s
> > sys     0m0.627s
> > eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 >
> > test; sync'
> > 10+0 records in
> > 10+0 records out
> > 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec)
> >   99.5%
> >
> > real    0m2.558s
> > user    0m1.554s
> > sys     0m0.680s
> > eve:~ sanity$
> >
> >
> >
> > This was on OS X, but I think it's still valid -- this is a slightly
> > older Powerbook, with a 5400 RPM drive, 1.6 ghz G4.
> >
> > -1 is still worlds better than nothing.  The backup was over 15 gigs,
> > down to about 6 -- loads of repetitive data, I'm sure, but that's where
> > you win with compression anyway.
>
> Wow. That's a lot better; I guess I did get something wrong in trying to
> tune deflate. That was pre-cryptoapi though; looking at
> cryptoapi/deflate.c, I don't see any way of controlling the compression
> level. Am I missing anything?

Compressing /dev/zero isn't a great test. The timings are really data-dependant:

ray@...enix:~$ time bash -c 'sudo dd if=/dev/zero bs=8M count=64 |
gzip -v1 >/dev/null'
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 7.60817 seconds, 70.6 MB/s
 99.6%

real    0m7.652s
user    0m6.581s
sys     0m0.701s
ray@...enix:~$ time bash -c 'sudo dd if=/dev/mem bs=8M count=64 | gzip
-v1 >/dev/null'
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 21.5863 seconds, 24.9 MB/s
 70.4%

real    0m21.626s
user    0m18.763s
sys     0m1.762s

This is on an AMD64 laptop.

Ray
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/