lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Tue, 28 Aug 2018 02:36:30 +0000 From: Nick Terrell <terrelln@...com> To: Adam Borowski <kilobyte@...band.pl> CC: Andi Kleen <ak@...ux.intel.com>, René Rebe <rene@...ctcode.com>, Thomas Gleixner <tglx@...utronix.de>, Ingo Molnar <mingo@...hat.com>, "H. Peter Anvin" <hpa@...or.com>, "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>, "x86@...nel.org" <x86@...nel.org>, Kernel Team <Kernel-team@...com>, "Chris Mason" <clm@...com>, Julian Andres Klode <julian.klode@...onical.com> Subject: Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel On Aug 17, 2018, at 1:07 PM, Adam Borowski <kilobyte@...band.pl> wrote: > On Fri, Aug 17, 2018 at 12:22:44PM -0700, Andi Kleen wrote: >> On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote: >>>> The "favourite compressor" seems to roughly change every year, so if >>>> we keep adding new ones things will get more and more convoluted. >>> >>> The above patchset drops just bzip2. It is the only one that's strictly >>> beaten in every way (ratio, time, memory usage), there are also no other >> >> Does time include build time? I've been reverting back to gzip recently >> because I care very much about that. > > Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while > ago), here's copypasta of a random 16824672 byte executable, in userspace, > with default level setting: > > comp decomp size > xz 8.038s 0.356s 4320292 > bz2 2.265s 0.730s 5234516 > zst 0.274s 0.102s 5657626 > gz 0.880s 0.152s 6515505 > Z 0.499s 0.133s 8932459 > lzo 0.100s 0.095s 9198874 > > As you can see, zstd's compression time is drastically better than gzip, > while ratio is better. The default level is very low (-3 on -1..-22 scale) > but you can crank it up for stronger compression. > > The defaults fit your use case. > >>> uses of bzip2 anywhere in the kernel so we'd get to drop its code >>> completely: 900 lines of Linus' happiness. >> >> Great! >> >>> Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), >>> but those are used elsewhere thus there's hardly any gain. If you want them >>> gone, please say so -- I'll include their droppage. >> >> Yes would be good to remove the kernel image support for those too >> just to simplify the config process, even if it doesn't save much code. > > There's one caveat: fast choices are quite new: > * lz4 userspace tools are not even in current Debian stable (just unstable) > * uncompressed kernel got in only this merge window > * zstd has userspace tools in Debian stable but is not merged into the > kernel yet > (other dists are probably similar) > > Thus, it might be a good idea to keep lzo for a while longer. > > Bare lzma can probably go -- xz filters are nice for binaries of archs it > knows (disabled otherwise), and lack of header requires hacks to find out > the payload's size. Personally, I'd be very happy to see LZMA go. It is a custom implementation that doesn't use the lib/xz/ library. When I was implementing decompress_zstd.c I fuzzed all of the kernel decompressors, and unlzma() will crash on invalid input. There is no reason, other than not breaking compatibility, to use LZMA over XZ. > So it's up to you guys: do you want me to drop lzo and/or lzma? > We can also drop them just for vmlinuz but not initrd. > > > Meow! > -- > ⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition: > ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17] > ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37] > ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]
Powered by blists - more mailing lists