linux-kernel - [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <CA+icZUXCn2an9aNDrm+-eneSAOyGibz0W1xYhwkA5k3B3U-5vQ@mail.gmail.com>
Date:   Sun, 29 Mar 2020 17:45:05 +0200
From:   Sedat Dilek <sedat.dilek@...il.com>
To:     Nick Terrell <nickrterrell@...il.com>,
        Nick Terrell <terrelln@...com>
Cc:     linux-kernel@...r.kernel.org, Chris Mason <clm@...com>,
        linux-kbuild@...r.kernel.org, x86@...nel.org,
        gregkh@...uxfoundation.org, Petr Malat <oss@...at.biz>,
        Kees Cook <keescook@...omium.org>,
        Kernel Team <Kernel-team@...com>,
        Adam Borowski <kilobyte@...band.pl>,
        Patrick Williams <patrickw3@...com>, rmikey@...com,
        mingo@...nel.org, Patrick Williams <patrick@...cx.xyz>,
        Nick Desaulniers <ndesaulniers@...gle.com>
Subject: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[ CC: Nick Desaulniers ]

Hi Nick,

I have tested your patchset v3 on top of Linux v5.6-rc7+
compiled/linked with Clang/LLD v9.0.1.

My Linux system is Debian/testing (will be "bullseye" aka version 11) AMD64.
I used zstd package version 1.4.4+dfsg-3.

I have no big numbers for you expect a comparison of initrd.img files
compressed with GZIP (default by Debian) and ZSTD:

root# du -k /boot/initrd.img-$(uname -r).{gz,zst}
29272   /boot/initrd.img-5.6.0-rc7-1-amd64-clang.gz
23540   /boot/initrd.img-5.6.0-rc7-1-amd64-clang.zst

I needed to do some modifications to stuff from
initramfs-tools/initramfs-tools-core (see end-titles).

Do you plan to push this for Linux v5.7?

Feel free to add credits for the whole series:

   Tested-by: Sedat Dilek <sedat.dilek@...il.com>

I have attached my kernel-config and dmesg-output.

Thanks.

Regards,
- Sedat -

P.S.: For posterity / ensuing ages - README_zstd-v3.txt (file is
attached seperately)

[ PREREQS ]

LINK: https://lore.kernel.org/lkml/20200325195849.407900-1-nickrterrell@gmail.com/
      ("[PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs")
LINK: https://github.com/terrelln/linux/commits/zstd-v3

zstd compression support for initramfs requires additional patches for
the Linux kernel (see above Link).

NOTE-1: Tested zstd-v3 patchset with Linux v5.6-rc7 on x86 64-bit (dileks)
NOTE-2: My Linux OS is Debian/testing AMD64.
NOTE-3: Install zstd Debian package (here: version 1.4.4+dfsg-3).


[ EDITS ]

[ EDIT: /etc/initramfs-tools/initramfs.conf ]

-# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz ]
+# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]

-COMPRESS=gzip
+COMPRESS=zstd


[ EDIT: /usr/sbin/mkinitramfs ]

case "${compress}" in

+zstd)  compress="zstd -19 -v" ;;
 bzip2|lzma|lzop)

esac

NOTE: In my kernel build-log I have seen 'zstd -19' to compress the
vmlinux file ('-v' for verbose output).

$ grep zstd build-log_5.6.0-rc7-1-amd64-clang.txt | grep vmlinux
  (cat arch/x86/boot/compressed/vmlinux.bin
arch/x86/boot/compressed/vmlinux.relocs | zstd -19 && printf
\220\016\300\001) > arch/x86/boot/compressed/vmlinux.bin.zst || (rm -f
arch/x86/boot/compressed/vmlinux.bin.zst ; false)


[ EDIT: /usr/bin/unmkinitramfs ]

# Extract a compressed cpio archive
xcpio()
{
        if gzip -t "$archive" >/dev/null 2>&1 ; then
                gzip -c -d "$archive"
+       elif zstdcat -t "$archive" >/dev/null 2>&1 ; then
+               zstdcat "$archive"
        elif xzcat -t "$archive" >/dev/null 2>&1 ; then

}


[ UPDATE-INITRAMFS ]

root# update-initramfs -c -k $(uname -r) -v 2>&1 | tee log_update-initramfs.txt

...
Building cpio /boot/initrd.img-5.6.0-rc7-1-amd64-clang.new initramfs
*** zstd command line interface 64-bits v1.4.4, by Yann Collet ***
/*stdin*\            : 23.58%   (89630208 => 21134850 bytes, /*stdout*\)


[ INITRD-IMAGE SIZES ]

root# du -k /boot/initrd.img-$(uname -r).{gz,zst}
29272   /boot/initrd.img-5.6.0-rc7-1-amd64-clang.gz
23540   /boot/initrd.img-5.6.0-rc7-1-amd64-clang.zst


[ INSPECT AND EXTRACT INITRD-IMAGE ]

root# lsinitramfs -l /boot/initrd.img-5.6.0-rc7-1-amd64-clang

root# unmkinitramfs -v /boot/initrd.img-5.6.0-rc7-1-amd64-clang
/path/to/unpacked_dir


[ CHECK VMLINUX ]

user$ file vmlinux
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
statically linked, BuildID[xxHash]=e04120f126f447e9, with debug_info,
not stripped

user$ file arch/x86/boot/compressed/vmlinux{,.bin*}
arch/x86/boot/compressed/vmlinux:         ELF 64-bit LSB executable,
x86-64, version 1 (SYSV), statically linked, not stripped
arch/x86/boot/compressed/vmlinux.bin:     ELF 64-bit LSB executable,
x86-64, version 1 (SYSV), statically linked,
BuildID[xxHash]=e04120f126f447e9, stripped
arch/x86/boot/compressed/vmlinux.bin.zst: Zstandard compressed data
(v0.8+), Dictionary ID: None

user$ du -k arch/x86/boot/compressed/vmlinux{,.bin*}
6536    arch/x86/boot/compressed/vmlinux
21448   arch/x86/boot/compressed/vmlinux.bin
6432    arch/x86/boot/compressed/vmlinux.bin.zst


-dileks // 29-MAR-2020: Add more Links; Add new "CHECK VMLINUX"
section; Add note about zstd package requirements
-dileks // 28-MAR-2020: Initial release

- EOT -

View attachment "dmesg-T_5.6.0-rc7-1-amd64-clang_zstd-initramfs.txt" of type "text/plain" (70127 bytes)

Download attachment "config-5.6.0-rc7-1-amd64-clang" of type "application/octet-stream" (226488 bytes)