linux-kernel - Re: [PATCH v2] kbuild: Add CONFIG_LZ4_LEVEL for choosing compression level

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <a8edf0c1-1fd1-4101-8c42-073a314dcb51@mailbox.org>
Date: Wed, 22 Jan 2025 19:34:23 +0100
From: Tor Vic <torvic9@...lbox.org>
To: Jann Horn <jannh@...gle.com>, Masahiro Yamada <masahiroy@...nel.org>,
 Nathan Chancellor <nathan@...nel.org>, Nicolas Schier <nicolas@...sle.eu>
Cc: Andrew Morton <akpm@...ux-foundation.org>, linux-kbuild@...r.kernel.org,
 linux-kernel@...r.kernel.org, Ard Biesheuvel <ardb@...nel.org>
Subject: Re: [PATCH v2] kbuild: Add CONFIG_LZ4_LEVEL for choosing compression
 level



On 1/22/25 16:08, Jann Horn wrote:
> Allow choosing the LZ4 compression level using the new config flag
> CONFIG_LZ4_LEVEL.
> 
> This will allow developers who want to quickly iterate with one-off kernel
> builds to speed up kernel compression at the cost of creating a larger
> kernel image.
> 
> LZ4 at level 9 (which the kernel uses by default) takes ~2.8
> seconds to compress a 110M large vmlinux.bin on my machine; LZ4 at level 1
> just needs ~220ms wall clock time.
> 
> Signed-off-by: Jann Horn <jannh@...gle.com>
> ---
> My patch provides kconfig for choosing compression level of LZ4; Tor Vic
> mentioned using kconfig for choosing the compression level of ZSTD.
> 
> Does my current approach of having a kconfig option specifically for
> LZ4 compression level look good (and we can add the same for ZSTD later
> if necessary); or should there be a single kconfig option that sets a
> compression level that is used across all compression algorithms?
> I guess doing it separately for different algorithms makes more sense
> because different compression levels might mean different things for
> different algorithms?

Hi Jann,

Yes, different algos have different ways of specifying compression 
levels. You cannot have one single option for all of the available 
compressors.

> 
> Do I need to somehow make it explicit that cmd_lz4 and cmd_lz4_with_size
> rely on CONFIG_LZ4_LEVEL, or does the build system figure that out on
> its own somehow? Experimentally it seems like the build system triggers
> a rebuild when the config option changes, but I'm not sure how that
> actually works.

Here is my take for zstd which I've been using for a while now.
(zstd levels above 19 need the '--ultra' switch, hence the honkytonk below).

--
 From 2e9a0e058e10a428d2ee079b969627e475cc0f2e Mon Sep 17 00:00:00 2001
From: Tor Vic <torvic9@...lbox.org>
Date: Tue, 6 Aug 2024 11:24:32 +0200
Subject: [PATCH] Kbuild, zstd: Allow users to choose a compression level for
  the kernel image

Signed-off-by: Tor Vic <torvic9@...lbox.org>
---
  init/Kconfig         | 13 +++++++++++++
  scripts/Makefile.lib |  9 ++++++++-
  2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/init/Kconfig b/init/Kconfig
index 105cdd05db759..9c39bc17964de 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -380,6 +380,19 @@ config KERNEL_UNCOMPRESSED

  endchoice

+menu "ZSTD compression options"
+	depends on KERNEL_ZSTD
+
+config ZSTD_COMPRESSION_LVL
+	int "Compression level (1-22)"
+	range 1 22
+	default "22"
+	help
+	  Choose a compression level for zstd kernel compression.
+	  Default is 22, which is the maximum.
+
+endmenu
+
  config DEFAULT_INIT
  	string "Default init path"
  	default ""
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 7395200538da8..58884363ebc62 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -516,6 +516,13 @@ quiet_cmd_xzmisc = XZMISC  $@
  # decompression is used, like initramfs decompression, zstd22 should 
likely not
  # be used because it would require zstd to allocate a 128 MB buffer.

+ifdef CONFIG_ZSTD_COMPRESSION_LVL
+zstd_comp_lvl	:= $(CONFIG_ZSTD_COMPRESSION_LVL)
+ifeq ($(call test-gt, $(zstd_comp_lvl), 19),y)
+zstd_comp_lvl	+= --ultra
+endif
+endif
+
  quiet_cmd_zstd = ZSTD    $@
        cmd_zstd = cat $(real-prereqs) | $(ZSTD) -19 > $@

@@ -523,7 +530,7 @@ quiet_cmd_zstd22 = ZSTD22  $@
        cmd_zstd22 = cat $(real-prereqs) | $(ZSTD) -22 --ultra > $@

  quiet_cmd_zstd22_with_size = ZSTD22  $@
-      cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 
--ultra; $(size_append); } > $@
+      cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -T0 
-$(zstd_comp_lvl); $(size_append); } > $@

  # ASM offsets
  # 
---------------------------------------------------------------------------
--