linux-kernel - Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <8117EBED-C57E-40AA-8E29-F4DFAB97E059@fb.com>
Date:   Tue, 28 Aug 2018 02:36:30 +0000
From:   Nick Terrell <terrelln@...com>
To:     Adam Borowski <kilobyte@...band.pl>
CC:     Andi Kleen <ak@...ux.intel.com>,
        René Rebe <rene@...ctcode.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>,
        "H. Peter Anvin" <hpa@...or.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "x86@...nel.org" <x86@...nel.org>,
        Kernel Team <Kernel-team@...com>, "Chris Mason" <clm@...com>,
        Julian Andres Klode <julian.klode@...onical.com>
Subject: Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

On Aug 17, 2018, at 1:07 PM, Adam Borowski <kilobyte@...band.pl> wrote:
> On Fri, Aug 17, 2018 at 12:22:44PM -0700, Andi Kleen wrote:
>> On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote:
>>>> The "favourite compressor" seems to roughly change every year, so if
>>>> we keep adding new ones things will get more and more convoluted.
>>> 
>>> The above patchset drops just bzip2.  It is the only one that's strictly
>>> beaten in every way (ratio, time, memory usage), there are also no other
>> 
>> Does time include build time? I've been reverting back to gzip recently
>> because I care very much about that.
> 
> Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while
> ago), here's copypasta of a random 16824672 byte executable, in userspace,
> with default level setting:
> 
> 	comp	decomp	size
> xz	8.038s	0.356s	4320292
> bz2	2.265s	0.730s	5234516
> zst	0.274s	0.102s	5657626
> gz	0.880s	0.152s	6515505
> Z	0.499s	0.133s	8932459
> lzo	0.100s	0.095s	9198874
> 
> As you can see, zstd's compression time is drastically better than gzip,
> while ratio is better.  The default level is very low (-3 on -1..-22 scale)
> but you can crank it up for stronger compression.
> 
> The defaults fit your use case.
> 
>>> uses of bzip2 anywhere in the kernel so we'd get to drop its code
>>> completely: 900 lines of Linus' happiness.
>> 
>> Great!
>> 
>>> Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
>>> but those are used elsewhere thus there's hardly any gain.  If you want them
>>> gone, please say so -- I'll include their droppage.
>> 
>> Yes would be good to remove the kernel image support for those too
>> just to simplify the config process, even if it doesn't save much code.
> 
> There's one caveat: fast choices are quite new:
> * lz4 userspace tools are not even in current Debian stable (just unstable)
> * uncompressed kernel got in only this merge window
> * zstd has userspace tools in Debian stable but is not merged into the
>  kernel yet
> (other dists are probably similar)
> 
> Thus, it might be a good idea to keep lzo for a while longer.
> 
> Bare lzma can probably go -- xz filters are nice for binaries of archs it
> knows (disabled otherwise), and lack of header requires hacks to find out
> the payload's size.

Personally, I'd be very happy to see LZMA go. It is a custom implementation
that doesn't use the lib/xz/ library. When I was implementing decompress_zstd.c
I fuzzed all of the kernel decompressors, and unlzma() will crash on invalid input.
There is no reason, other than not breaking compatibility, to use LZMA over XZ.

> So it's up to you guys: do you want me to drop lzo and/or lzma?
> We can also drop them just for vmlinuz but not initrd.
> 
> 
> Meow!
> -- 
> ⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
> ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
> ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
> ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]