linux-kernel - Re: [PATCH] kbuild: modinst: Enable multithread xz compression

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Fri, 24 Feb 2023 14:38:45 +0900
From:   Masahiro Yamada <masahiroy@...nel.org>
To:     André Almeida <andrealmeid@...lia.com>
Cc:     linux-kernel@...r.kernel.org, linux-kbuild@...r.kernel.org,
        kernel-dev@...lia.com, Nathan Chancellor <nathan@...nel.org>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        Nicolas Schier <nicolas@...sle.eu>
Subject: Re: [PATCH] kbuild: modinst: Enable multithread xz compression

On Thu, Feb 23, 2023 at 9:17 AM André Almeida <andrealmeid@...lia.com> wrote:
>
> As it's done for zstd compression, enable multithread compression for
> xz to speed up module installation.
>
> Signed-off-by: André Almeida <andrealmeid@...lia.com>
> ---
>
> On my setup xz is a bottleneck during module installation. Here are the
> numbers to install it in a local directory, before and after this patch:
>
> $ time make INSTALL_MOD_PATH=/home/tonyk/codes/.kernel_deploy/ modules_install -j16
> Executed in  100.08 secs
>
> $ time make INSTALL_MOD_PATH=/home/tonyk/codes/.kernel_deploy/ modules_install -j16
> Executed in   28.60 secs


Heh, this is an interesting benchmark.

Without this patch, you ran 16 processes of 'xz' in parallel
since you gave -j16.

You created multi-threads in each xz process, then you got 3x faster.
What made it happen? How many threads can your system run?



I did not get such an improvement in my testing.
In my machine $(nproc) is 24.


[Without this patch]

$ time make INSTALL_MOD_PATH=/tmp/inst1  modules_install -j$(nproc)

real 0m33.965s
user 10m6.118s
sys 0m37.231s

[With this patch]

$ time make INSTALL_MOD_PATH=/tmp/inst1  modules_install -j$(nproc)

real 0m32.568s
user 10m4.472s
sys 0m39.132s



Given that GNU Make provides the parallel execution environment,
you can control the number of processes of 'xz'.

There is no point in forcing multi-threading, which the user
did not ask or ever want.













> ---
>  scripts/Makefile.modinst | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> index 4815a8e32227..28dcc523d2ee 100644
> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -99,7 +99,7 @@ endif
>  quiet_cmd_gzip = GZIP    $@
>        cmd_gzip = $(KGZIP) -n -f $<
>  quiet_cmd_xz = XZ      $@
> -      cmd_xz = $(XZ) --lzma2=dict=2MiB -f $<
> +      cmd_xz = $(XZ) --lzma2=dict=2MiB -f -T0 $<
>  quiet_cmd_zstd = ZSTD    $@
>        cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
>
> --
> 2.39.2
>


-- 
Best Regards
Masahiro Yamada