linux-kernel - Re: [PATCH] kbuild: disable pahole multithreading for reproducible builds

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <49fa2bd1-bd6e-4404-a1c7-2170d37f0194@oracle.com>
Date: Fri, 22 Mar 2024 10:07:00 +0000
From: Alan Maguire <alan.maguire@...cle.com>
To: Thomas Weißschuh <linux@...ssschuh.net>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Andrii Nakryiko <andrii@...nel.org>,
        Martin KaFai Lau
 <martin.lau@...ux.dev>,
        Eduard Zingerman <eddyz87@...il.com>, Song Liu <song@...nel.org>,
        Yonghong Song <yonghong.song@...ux.dev>,
        John Fastabend <john.fastabend@...il.com>,
        KP Singh <kpsingh@...nel.org>, Stanislav Fomichev <sdf@...gle.com>,
        Hao Luo <haoluo@...gle.com>, Jiri Olsa <jolsa@...nel.org>,
        Masahiro Yamada <masahiroy@...nel.org>,
        Nathan Chancellor <nathan@...nel.org>,
        Nicolas Schier <nicolas@...sle.eu>
Cc: bpf@...r.kernel.org, linux-kbuild@...r.kernel.org,
        linux-kernel@...r.kernel.org
Subject: Re: [PATCH] kbuild: disable pahole multithreading for reproducible
 builds

On 22/03/2024 08:31, Thomas Weißschuh wrote:
> A BTF type_id is a numeric identifier allocated by pahole through
> libbpfd. Ids are incremented for each allocation.
> Running pahole multithreaded makes the sequence of allocations
> non-deterministic which also makes the type_id itself non-deterministic.
> As the type_id end up in the binary this breaks reproducibility.
> 
> Therefore, if the kernel is built reproducibly as indicated by
> KBUILD_BUILD_TIMESTAMP, disable threading in pahole.
> 
> Fixes: b4f72786429c ("scripts/pahole-flags.sh: Parse DWARF and generate BTF with multithreading.")
> Fixes: 72d091846de9 ("kbuild: avoid too many execution of scripts/pahole-flags.sh")
> Signed-off-by: Thomas Weißschuh <linux@...ssschuh.net>

Yep, we've talked about sorting final BTF in various ways but until that
is implemented, non-parallel execution of pahole is the best way to get
reliabily-reproducing BTF ids. We should probably update the pahole
manpage to warn about parallel non-deterministic id allocation in the
meantime.

Reviewed-by: Alan Maguire <alan.maguire@...cle.com>

> ---
>  scripts/Makefile.btf | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> index 82377e470aed..2356929e62e4 100644
> --- a/scripts/Makefile.btf
> +++ b/scripts/Makefile.btf
> @@ -10,7 +10,10 @@ endif
>  
>  pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
>  
> +# threaded execution randomizes BTF type_id allocation, breaking reproducibility
> +ifeq ($(KBUILD_BUILD_TIMESTAMP),)
>  pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
> +endif
>  
>  pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)		+= --lang_exclude=rust
>  
> 
> ---
> base-commit: 8e938e39866920ddc266898e6ae1fffc5c8f51aa
> change-id: 20240322-pahole-reprodicible-02e904e45686
> 
> Best regards,