linux-kernel - Re: Reiser4 und LZO compression

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <1156830102.3790.15.camel@nigel.suspend2.net>
Date:	Tue, 29 Aug 2006 15:41:42 +1000
From:	Nigel Cunningham <ncunningham@...uxmail.org>
To:	Jan Engelhardt <jengelh@...ux01.gwdg.de>
Cc:	Edward Shishkin <edward@...esys.com>,
	Stefan Traby <stefan@...lo-penguin.com>,
	Hans Reiser <reiser@...esys.com>,
	Alexey Dobriyan <adobriyan@...il.com>,
	reiserfs-list@...esys.com, linux-kernel@...r.kernel.org,
	Andrew Morton <akpm@...l.org>
Subject: Re: Reiser4 und LZO compression

Hi.

On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote:
> >> >>Hmm.  LZO is the best compression algorithm for the task as measured by
> >> >>the objectives of good compression effectiveness while still having very
> >> >>low CPU usage (the best of those written and GPL'd, there is a slightly
> >> >>better one which is proprietary and uses more CPU, LZRW if I remember
> >> >>right.  The gzip code base uses too much CPU, though I think Edward made
> >> > 
> >> > I don't think that LZO beats LZF in both speed and compression ratio.
> >> > 
> >> > LZF is also available under GPL (dual-licensed BSD) and was choosen in favor
> >> > of LZO for the next generation suspend-to-disk code of the Linux kernel.
> >> > 
> >> > see: http://www.goof.com/pcg/marc/liblzf.html
> >> 
> >> thanks for the info, we will compare them
> >
> >For Suspend2, we ended up converting the LZF support to a cryptoapi
> >plugin. Is there any chance that you could use cryptoapi modules? We
> >could then have a hope of sharing the support.
> 
> I am throwing in gzip: would it be meaningful to use that instead? The 
> decoder (inflate.c) is already there.
> 
> 06:04 shanghai:~/liblzf-1.6 > l configure*
> -rwxr-xr-x  1 jengelh users 154894 Mar  3  2005 configure
> -rwxr-xr-x  1 jengelh users  26810 Mar  3  2005 configure.bz2
> -rw-r--r--  1 jengelh users  30611 Aug 28 20:32 configure.gz-z9
> -rw-r--r--  1 jengelh users  30693 Aug 28 20:32 configure.gz-z6
> -rw-r--r--  1 jengelh users  53077 Aug 28 20:32 configure.lzf

We used gzip when we first implemented compression support, and found it
to be far too slow. Even with the fastest compression options, we were
only getting a few megabytes per second. Perhaps I did something wrong
in configuring it, but there's not that many things to get wrong!

In contrast, with LZF, we get very high throughput. My current laptop is
an 1.8MHz Turion with a 7200 RPM (PATA) drive. Without LZF compression,
my throughput in writing an image is the maximum the drive & interface
can manage - 38MB/s. With LZF, I get roughly that divided by compression
ratio achieved, so if the compression ratio is ~50%, as it generally is,
I'm reading and writing the image at 75-80MB/s. During this time, all
the computer is doing is compressing pages using LZF and submitting
bios, with the odd message being send to the userspace interface app via
netlink. I realise this is very different to the workload you'll be
doing, but hopefully the numbers are somewhat useful:

nigel@...el:~$ cat /sys/power/suspend2/debug_info
Suspend2 debugging info:
- SUSPEND core   : 2.2.7.4
- Kernel Version : 2.6.18-rc4
- Compiler vers. : 4.1
- Attempt number : 1
- Parameters     : 0 32785 0 0 0 0
- Overall expected compression percentage: 0.
- Compressor is 'lzf'.
  Compressed 820006912 bytes into 430426371 (47 percent compression).
- Swapwriter active.
  Swap available for image: 487964 pages.
- Filewriter inactive.
- I/O speed: Write 74 MB/s, Read 70 MB/s.
- Extra pages    : 1913 used/2100.
nigel@...el:~$

(Modify hibernate.conf to disable compression, suspend again...)

nigel@...el:~$ cat /sys/power/suspend2/debug_info
Suspend2 debugging info:
- SUSPEND core   : 2.2.7.4
- Kernel Version : 2.6.18-rc4
- Compiler vers. : 4.1
- Attempt number : 2
- Parameters     : 0 32785 0 0 0 0
- Overall expected compression percentage: 0.
- Swapwriter active.
  Swap available for image: 487964 pages.
- Filewriter inactive.
- I/O speed: Write 38 MB/s, Read 39 MB/s.
- Extra pages    : 1907 used/2100.
nigel@...el:~$

Oh, I also have a debugging mode where I can get Suspend2 to just
compress the pages but not actually write anything. If I do that, it
says it can do 80MB/s on my kernel image, so the disk is still the
bottleneck, it seems.

Hope this all helps (and isn't information overload!)

Nigel

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/