linux-kernel - Re: [RESEND v2] Kbuild: Update config_data.gz only if KCONFIG

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAK7LNASzP-pMu7Yd6nkoV_mxOUeouYNLW_xZUy94E_WCdjJ5PA@mail.gmail.com>
Date:   Sun, 25 Apr 2021 15:50:49 +0900
From:   Masahiro Yamada <masahiroy@...nel.org>
To:     Elliot Berman <eberman@...eaurora.org>
Cc:     Michal Marek <michal.lkml@...kovi.net>,
        Linux Kbuild mailing list <linux-kbuild@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Matthias Maennich <maennich@...gle.com>,
        Trilok Soni <tsoni@...eaurora.org>
Subject: Re: [RESEND v2] Kbuild: Update config_data.gz only if KCONFIG_CONFIG
 materially changed

On Sat, Apr 24, 2021 at 2:02 AM Elliot Berman <eberman@...eaurora.org> wrote:


Sorry for the delay.

This patch is over-engineering.

I will apply this.
https://patchwork.kernel.org/project/linux-kbuild/patch/20210425062407.1183801-5-masahiroy@kernel.org/

The 'cmp' command is not expensive.
md5sum is unneeded.





> If you update the timestamp of KCONFIG_CONFIG without actually changing
> anything, config_data.gz is re-generated and causes vmlinux to re-link.
> When Link Time Optimization is enabled, unnecessary re-linking of
> vmlinux is highly desirable since it adds several minutes to build time.
>
> Avoid touching config_data.gz by using a script to compare the existing
> config_data.gz, KCONFIG_CONFIG, or script itself to update only if any
> is mismatched.  The script follows gen_kheaders.sh approach for
> determing in update is needed. The script intentionally avoids
> re-compressing KCONFIG_CONFIG.
>
> The .config can be touched, for instance, by a build script which
> installs the default defconfig and then applies a defconfig fragment on
> top.
>
> For a simple example on my x86 machine, I modified x86 default defconfig to set
> CONFIG_IKCONFIG=y and run:
>   make -j50 defconfig tiny.config vmlinux
>   make -j50 defconfig tiny.config vmlinux
> With this patch, vmlinux is not re-built as a result of config_data.gz
> change.
>
> Changes in v2:
>  - Use md5 checksum to compare .config instead of gzip'ing again
>
> Signed-off-by: Elliot Berman <eberman@...eaurora.org>
> ---
>  kernel/.gitignore         |  1 +
>  kernel/Makefile           |  4 +++-
>  kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++
>  3 files changed, 35 insertions(+), 1 deletion(-)
>  create mode 100755 kernel/gen_config_data.sh
>
> diff --git a/kernel/.gitignore b/kernel/.gitignore
> index 78701ea..a191136 100644
> --- a/kernel/.gitignore
> +++ b/kernel/.gitignore
> @@ -1,4 +1,5 @@
>  # SPDX-License-Identifier: GPL-2.0-only
> +config_data.gz.md5
>  kheaders.md5
>  timeconst.h
>  hz.bc
> diff --git a/kernel/Makefile b/kernel/Makefile
> index 320f1f3..0784bf3d 100644
> --- a/kernel/Makefile
> +++ b/kernel/Makefile
> @@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o
>  $(obj)/configs.o: $(obj)/config_data.gz
>
>  targets += config_data.gz
> +quiet_cmd_genicfg = CHK     $(obj)/config_data.gz
> +      cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $<
>  $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
> -       $(call if_changed,gzip)
> +       $(call cmd,genicfg)
>
>  $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
>
> diff --git a/kernel/gen_config_data.sh b/kernel/gen_config_data.sh
> new file mode 100755
> index 00000000..e9ff193
> --- /dev/null
> +++ b/kernel/gen_config_data.sh
> @@ -0,0 +1,31 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0
> +
> +# This script generates a compressed version of .config, if its checksum has changed
> +set -e
> +
> +this_file="$(readlink -f "$0")"
> +outfile=$1
> +infile=$2
> +
> +config_md5="$(md5sum $infile | cut -d ' ' -f1)"
> +# Any changes to this script will also cause a rebuild of config_data.
> +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)"
> +if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi
> +
> +if [ -f $outfile.md5 ] &&
> +       [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] &&
> +       [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] &&
> +       [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then
> +               exit
> +fi
> +
> +if [ "${quiet}" != "silent_" ]; then
> +       echo "  GEN     $outfile"
> +fi
> +
> +${KGZIP} -c -n -f -9 $infile > $outfile
> +
> +echo "$config_md5" > $outfile.md5
> +echo "$this_file_md5" >> $outfile.md5
> +echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>


-- 
Best Regards
Masahiro Yamada