| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Fri, 31 Jul 2020 11:47:57 +0200
From: Sedat Dilek <sedat.dilek@...il.com>
To: Nick Terrell <nickrterrell@...il.com>
Cc: Borislav Petkov <bp@...en8.de>,
Thomas Gleixner <tglx@...utronix.de>,
linux-kernel@...r.kernel.org, Chris Mason <clm@...com>,
linux-kbuild@...r.kernel.org, x86@...nel.org,
gregkh@...uxfoundation.org, Petr Malat <oss@...at.biz>,
Kees Cook <keescook@...omium.org>,
Kernel Team <Kernel-team@...com>,
Adam Borowski <kilobyte@...band.pl>,
Patrick Williams <patrickw3@...com>, rmikey@...com,
mingo@...nel.org, Patrick Williams <patrick@...cx.xyz>,
Norbert Lange <nolange79@...il.com>,
Andrew Morton <akpm@...ux-foundation.org>,
Alex Xu <alex_y_xu@...oo.ca>,
Masahiro Yamada <masahiroy@...nel.org>,
Nick Terrell <terrelln@...com>
Subject: Re: [GIT PULL][PATCH v10 0/8] Add support for ZSTD-compressed kernel
and initramfs
On Thu, Jul 30, 2020 at 9:11 PM Nick Terrell <nickrterrell@...il.com> wrote:
>
> From: Nick Terrell <terrelln@...com>
>
> Please pull from
>
> git@...hub.com:terrelln/linux.git tags/v10-zstd
>
> to get these changes. Alternatively the patchset is included.
>
> Hi all,
>
> This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
> initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
> are supported on all architectures. The ZSTD-compressed kernel is only
> hooked up to x86 in this patch set.
>
> Zstandard requires slightly more memory during the kernel decompression
> on x86 (192 KB vs 64 KB), and the memory usage is independent of the
> window size.
>
> Zstandard requires memory proprortional to the window size used during
> compression for decompressing the ramdisk image, since streaming mode is
> used. Newer versions of zstd (1.3.2+) list the window size of a file
> with `zstd -lv <file>'. The absolute maximum amount of memory required
> is just over 8 MB, but it can be controlled at compression time.
>
> This patch set has been boot tested with buildroot and QEMU based off
> of linux-5.6-rc6.
>
> On i386 and x86_64 I have tested the following configurations:
> * zstd compressed kernel and a separate zstd compressed initramfs
> * zstd compressed kernel and a built-in zstd compressed initramfs
> * gzip compressed kernel and a separate gzip compressed initramfs
> * gzip compressed kernel and a built-in gzip compressed initramfs
>
> On arm and aarch64 I tested the same configurations, except that the kernel is
> always gzip compressed.
>
> Facebook has been using v1 of these patches on x86_64 devices for more than 6
> months. When we switched from a xz compressed initramfs to a zstd compressed
> initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
> switched from a xz compressed kernel to a zstd compressed kernel we saved 2
> seconds of boot time.
>
> Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
> When we switched from an lzma compressed initramfs to a zstd compressed initramfs
> decompression time shrunk from 27 seconds to 8 seconds.
>
> The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> than the xz or lzma compressed kernels, and it decompresses faster than
> everything except lz4. See the table below for the measurement of an x86_64
> kernel ordered by compressed size:
>
> algo size
> xz 6,509,792
> lzma 6,856,576
> zstd 7,399,157
> gzip 8,522,527
> bzip 8,629,603
> lzo 9,808,035
> lz4 10,705,570
> none 32,565,672
>
> Alex Xu ran benchmarks in https://lkml.org/lkml/2020/7/1/722.
>
> v1 -> v2:
> - Rebase
> - usr/Makefile and init/Kconfig were changed so the patches were updated
> - No functional changes except to rebase
> - Split the patches up into smaller chunks
>
> v2 -> v3:
> - Add *.zst to the .gitignore in patch 8
> - Style nits in patch 3
> - Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
> 1 through 3
>
> v3 -> v4:
> - Increase the ZSTD_IOBUF_SIZE from 4KB to 128KB to improve performance.
> With this change I switch from malloc() to large_malloc() for the
> buffers.
> - Increase the maximum allowed window size from 8 MB to 128 MB, which is
> the max that zstd in the kernel supports.
>
> v4 -> v5:
> - Update commit message for patch 6 in response to comments
> - Rebase onto next-20200408
>
> v5 -> v6:
> - Rebase onto v5.8-rc4
>
> v6 -> v7:
> - (1/7) Don't define or use 'ZSTD_PREBOOT' to hide exports
> - (2/8) Drop 'lib: prepare xxhash for preboot environment'
> - (2/7) Use '__DISABLE_EXPORTS' in unzstd to hide exports
> - (3/7) Update zstd compression cmd to follow other compressors
> - (3/7) Add zstd22 cmd
> - (6/7) Use zstd -22 --ultra (zstd22) for x86 kernel compression
>
> v7 -> v8:
> - (2/7) Don't define '__DISABLE_EXPORTS'
> - (6/7) Define '__DISABLE_EXPORTS' in misc.c
>
> v8 -> v9:
> - Rebase onto v5.8-rc7
> - (2/7) Fix nits about comment style & typos
> - (3/7) Fix typo in init/Kconfig description
> - (6/7) Explain BOOT_HEAP_SIZE increase and define __DISABLE_EXPORTS in
> Makefile KBUILD_CFLAGS and remove definitions from kaslr.c and misc.c
>
> v9 -> v10:
> - (6/8) Fix commit message regarding __DISABLE_EXPORTS
> - (8/8) Add .zst files to Documentation/dontdiff
>
Hi Nick,
Hope with version 10 of your patchset the review process has an end
and it will picked up into tip Git.
Again: Tested-by: Sedat Dilek <sedat.dilek@...il.com>
Build-environments (more details see [1]):
#0: Debian/unstable AMD64
#1: LLVM v11.0.0-rc1 (clang-11, ld.lld-11 and LLVM "bin"utils)
#2: GCC v10.2 and GNU/ld.bfd v2.35 and GNU/binutils
With the best wishes,
- Sedat -
[1] https://github.com/ClangBuiltLinux/linux/issues/1086#issuecomment-667036028
> Best,
> Nick Terrell
>
> Adam Borowski (1):
> .gitignore: add ZSTD-compressed files
>
> Nick Terrell (7):
> lib: prepare zstd for preboot environment
> lib: add zstd support to decompress
> init: add support for zstd compressed kernel
> usr: add support for zstd compressed initramfs
> x86: bump ZO_z_extra_bytes margin for zstd
> x86: Add support for ZSTD compressed kernel
> Documentation: dontdiff: Add zstd compressed files
>
> .gitignore | 1 +
> Documentation/dontdiff | 1 +
> Documentation/x86/boot.rst | 6 +-
> Makefile | 3 +-
> arch/x86/Kconfig | 1 +
> arch/x86/boot/compressed/Makefile | 6 +-
> arch/x86/boot/compressed/kaslr.c | 7 -
> arch/x86/boot/compressed/misc.c | 4 +
> arch/x86/boot/header.S | 8 +-
> arch/x86/include/asm/boot.h | 11 +-
> include/linux/decompress/unzstd.h | 11 +
> init/Kconfig | 15 +-
> lib/Kconfig | 4 +
> lib/Makefile | 1 +
> lib/decompress.c | 5 +
> lib/decompress_unzstd.c | 345 ++++++++++++++++++++++++++++++
> lib/zstd/fse_decompress.c | 9 +-
> lib/zstd/zstd_internal.h | 14 +-
> scripts/Makefile.lib | 22 ++
> usr/Kconfig | 20 ++
> usr/Makefile | 1 +
> 21 files changed, 469 insertions(+), 26 deletions(-)
> create mode 100644 include/linux/decompress/unzstd.h
> create mode 100644 lib/decompress_unzstd.c
>
> --
> 2.27.0
>
Powered by blists - more mailing lists