linux-kernel - Re: [PATCH 0/2] Add support for ZSTD-compressed kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20171011000805.7b2nteb5hjbrfe7a@angband.pl>
Date:   Wed, 11 Oct 2017 02:08:05 +0200
From:   Adam Borowski <kilobyte@...band.pl>
To:     Nick Terrell <terrelln@...com>
Cc:     "hpa@...or.com" <hpa@...or.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "x86@...nel.org" <x86@...nel.org>,
        Kernel Team <Kernel-team@...com>, Chris Mason <clm@...com>,
        Yann Collet <cyan@...com>, Rene Rebe <rene@...ctcode.com>
Subject: Re: [PATCH 0/2] Add support for ZSTD-compressed kernel

On Tue, Oct 10, 2017 at 10:40:13PM +0000, Nick Terrell wrote:
> On 10/10/17, 2:56 PM, "hpa@...or.com" <hpa@...or.com> wrote:
> >On October 10, 2017 2:22:42 PM PDT, Nick Terrell <terrelln@...com> wrote:
> >>This patch set adds support for a ZSTD-compressed kernel and ramdisk
> >>images in the kernel boot process. It only integrates the support with
> >>x86, though the first patch is generic to all architectures.
> >>
> >>Zstandard requires slightly more memory during the kernel decompression
> >>on x86 (192 KB vs 64 KB), and the memory usage is independent of the
> >>window size.
> >>
> >>Zstandard requires memory proprortional to the window size used during
> >>compression for decompressing the ramdisk image, since streaming mode
> >>is
> >>used. Newer versions of zstd (1.3.2+) list the window size of a file
> >>with `zstd -lv <file>'. The absolute maximum amount of memory required
> >>is just over 8 MB.

> > And, pray tell, what are the actual results?  What is the trade-off of
> > kernel size versus decompression performance versus the other algorithms
> > that we already support?  Adding algorithms for their own sake is a bad
> > thing not a good thing.
> 
> Sorry I neglected to include the benchmarks I've run so far. I've included
> them below, and will add them to the next version's cover letter.
> 
> Comparing the command line tools on a kernel image that is 68970616 B large:
> 
> | Algorithm | Compression Ratio | Decompression MB/s |
> |-----------|-------------------|--------------------|
> | zstd      |              4.42 |              436.5 |
> | gzip      |              3.72 |              134.1 |
> | xz        |              4.83 |               53.1 |
> | lz4       |              3.18 |             1682.2 |
> | lzo       |              3.36 |              389.6 |
> | bzip2     |              4.03 |               33.3 |

Perhaps it'd be a good idea to cull some of bad algorithms?  I don't know
the memory used by those, but envelope of the table you just shown suggests
using bzip2 and lzo is pointless.  So is gzip, but it's widespread as the
default for initramfs producers, thus it'd be unsafe to kill it.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased
⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts.
⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got
⠈⠳⣄⠀⠀⠀⠀ agriculture, towns then cities.     -- whitroth on /.