linux-kernel - Re: [PATCH 2/2] kbuild: pass --stream-size --no-content-size to zstd

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <221F448E-58B4-42F7-95DC-58AD9CA3D226@fb.com>
Date:   Mon, 22 Nov 2021 19:20:03 +0000
From:   Nick Terrell <terrelln@...com>
To:     "Alex Xu (Hello71)" <alex_y_xu@...oo.ca>
CC:     Michael Forney <forney@...gle.com>,
        Masahiro Yamada <yamada.masahiro@...ionext.com>,
        Michal Marek <michal.lkml@...kovi.net>,
        "Nick Desaulniers" <ndesaulniers@...gle.com>,
        Ingo Molnar <mingo@...nel.org>,
        "Sedat Dilek" <sedat.dilek@...il.com>,
        Kees Cook <keescook@...omium.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-kbuild@...r.kernel.org" <linux-kbuild@...r.kernel.org>
Subject: Re: [PATCH 2/2] kbuild: pass --stream-size --no-content-size to zstd



> On Nov 20, 2021, at 8:53 AM, Alex Xu (Hello71) <alex_y_xu@...oo.ca> wrote:
> 
> Otherwise, it allocates 2 GB of memory at once. Even though the majority
> of this memory is never touched, the default heuristic overcommit
> refuses this request if less than 2 GB of RAM+swap is currently
> available. This results in "zstd: error 11 : Allocation error : not
> enough memory" and the kernel failing to build.
> 
> When the size is specified, zstd will reduce the memory request
> appropriately. For typical kernel sizes of ~32 MB, the largest mmap
> request will be reduced to 512 MB, which will succeed on all but the
> smallest devices.
> 
> For inputs around this size, --stream-size --no-content-size may
> slightly decrease the compressed size, or slightly increase it:
> https://github.com/facebook/zstd/issues/2848.

I like the change in general. However, I have a question of portability.
The --stream-size option was added in zstd-1.4.4 released November
2019 [0]. That means the build will fail for people with older zstd
versions. As a reference, Ubuntu Bionic (18.04) includes zstd-1.3.3,
and Ubuntu Focal (20.04) has zstd-1.4.4. So Bionic users will have to
install zstd manually to build the kernel.

I’d prefer to err on the side of caution for portability. I’d say once
Bionic hits end of life, it’d be safe to require zstd-1.4.4 or newer.
However, I see the need to reduce memory usage, as this issue has
come up before.

Could we detect the zstd version with `zstd --version` and pass
--stream-size if we detect the version is >= v1.4.4? That add some
complexity, but I think it would be worth it. Just be sure to document
why we are detecting support for --stream-size, and why it is important
to pass to zstd.

Best,
Nick Terrell

[0] https://github.com/facebook/zstd/releases/tag/v1.4.4

> Signed-off-by: Alex Xu (Hello71) <alex_y_xu@...oo.ca>
> ---
> scripts/Makefile.lib | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index ca901814986a..13d756fbcdc7 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -468,10 +468,10 @@ quiet_cmd_xzmisc = XZMISC  $@
> # be used because it would require zstd to allocate a 128 MB buffer.
> 
> quiet_cmd_zstd = ZSTD    $@
> -      cmd_zstd = { cat $(real-prereqs) | $(ZSTD) -19; $(size_append); } > $@
> +      cmd_zstd = { cat $(real-prereqs) | $(ZSTD) --stream-size=$(total_size) --no-content-size -19; $(size_append); } > $@
> 
> quiet_cmd_zstd22 = ZSTD22  $@
> -      cmd_zstd22 = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
> +      cmd_zstd22 = { cat $(real-prereqs) | $(ZSTD) --stream-size=$(total_size) --no-content-size -22 --ultra; $(size_append); } > $@
> 
> # ASM offsets
> # ---------------------------------------------------------------------------
> -- 
> 2.34.0
>