linux-kernel - Re: [PATCH RFC] x86: Add CONFIG_KERNEL

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAG48ez3ddjXA0HiRyS-wT6Fs_fkX-5cf9XRD061YhJu+NZ8dGw@mail.gmail.com>
Date: Wed, 22 Jan 2025 15:30:49 +0100
From: Jann Horn <jannh@...gle.com>
To: Tor Vic <torvic9@...lbox.org>
Cc: Ard Biesheuvel <ardb@...nel.org>, Thomas Gleixner <tglx@...utronix.de>, Ingo Molnar <mingo@...hat.com>, 
	Borislav Petkov <bp@...en8.de>, Dave Hansen <dave.hansen@...ux.intel.com>, x86@...nel.org, 
	"H. Peter Anvin" <hpa@...or.com>, Masahiro Yamada <masahiroy@...nel.org>, 
	Nathan Chancellor <nathan@...nel.org>, Nicolas Schier <nicolas@...sle.eu>, linux-efi@...r.kernel.org, 
	Andrew Morton <akpm@...ux-foundation.org>, linux-kbuild@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH RFC] x86: Add CONFIG_KERNEL_UNCOMPRESSED support

On Wed, Jan 22, 2025 at 3:19 PM Tor Vic <torvic9@...lbox.org> wrote:
> On 1/22/25 14:54, Jann Horn wrote:
> > On Wed, Jan 22, 2025 at 2:31 PM Ard Biesheuvel <ardb@...nel.org> wrote:
> >> Hi Jann,
> >>
> >> On Tue, 21 Jan 2025 at 23:16, Jann Horn <jannh@...gle.com> wrote:
> >>>
> >>> Support storing the kernel uncompressed for developers who want to quickly
> >>> iterate with one-off kernel builds.
> >>> Store it in the usual format with a 4-byte length suffix and keep this new
> >>> codepath as close as possible to the normal path where decompression
> >>> happens.
> >>>
> >>> The other compression methods offered by the kernel take some time;
> >>> even LZ4 (which the kernel uses at compression level 9) takes ~2.8
> >>> seconds to compress a 110M large vmlinux.bin on my machine.
> >>>
> >>> An alternate approach to this would be to offer customization of the LZ4
> >>> compression level through a kconfig variable; and yet another approach
> >>> would be to abuse the existing gzip decompression logic by storing the
> >>> kernel as "non-compressed" DEFLATE blocks, so that the decompression code
> >>> will essentially end up just doing a bunch of memcpy() calls.
> >>>
> >>
> >> This all seems pretty complicated, and adding yet another
> >> (pseudo-)compression method is not great in terms of maintenance
> >> burden, especially because there are other consumers of the compressed
> >> images (both for bzImage and EFI zboot)
> >>
> >> Did you try running gzip with -1 instead of -9? On my build machine,
> >> this reduces the compression time of a defconfig bzImage build from
> >> 4.3 seconds to 0.9 seconds.
> >
> > I tried lz4 with -1; that is very fast (240ms wall clock time on my
> > machine, and just 120ms user time):
> >
> > $ ls -lh arch/x86/boot/compressed/vmlinux.bin
> > -rwxr-x--- 1 [...] 110M Jan 22 00:01 arch/x86/boot/compressed/vmlinux.bin
> > $ cat arch/x86/boot/compressed/vmlinux.bin | time lz4 -l -9 - - | wc -c
> > 2.86user 0.04system 0:02.96elapsed 97%CPU (0avgtext+0avgdata 15756maxresident)k
> > 0inputs+0outputs (0major+220minor)pagefaults 0swaps
> > 46309676
> > $ cat arch/x86/boot/compressed/vmlinux.bin | time lz4 -l -1 - - | wc -c
> > 0.12user 0.06system 0:00.24elapsed 75%CPU (0avgtext+0avgdata 15524maxresident)k
> > 0inputs+0outputs (0major+94minor)pagefaults 0swaps
> > 56029608
> >
> > But I wasn't sure how to wire that up in a nice way. I guess the
> > nicest option would be to create a separate kconfig variable for the
> > compression level to use for any cmd_lz4/cmd_lz4_with_size invocations
> > in the build process; and then maybe only make this option visible if
> > LZ4 is selected as kernel compression method?
> >
> > Another option would be to create a new option in the "Kernel
> > compression mode" choice menu with a name like "LZ4 (fast)", turn
> > CONFIG_KERNEL_LZ4 into an internal flag that is selected by both LZ4
> > variants shown in the choice menu, and duplicate some of the make
> > rules, but that seems overly complicated.
> >
>
> Hello,
>
> In my opinion 'lz4 -9' doesn't make much sense.
> It's terribly slow and the compression ratio is also not exactly good.
>
> Instead, zstd seems to be a much better choice. Not quite as ultra fast
> as lz4 levels 1 to 3, but much better compression.

I think you're describing a slightly different usecase.

My goal here is something I can use for when I build a kernel, boot it
in QEMU, test something, and then immediately throw the kernel away -
I don't care that much how much disk space the kernel image uses, and
the goal I'm optimizing for is pretty much just the time needed for
one build followed by one boot.