Message-ID: <CAEf4Bzaa+X4K3_NApFYHxWP1P7stnAvZH4to65D1600fie6H3w@mail.gmail.com>
Date: Thu, 12 Dec 2024 14:54:36 -0800
From: Andrii Nakryiko <andrii.nakryiko@...il.com>
To: Thomas Weißschuh <linux@...ssschuh.net>
Cc: Ihor Solodrai <ihor.solodrai@...me>, Alexei Starovoitov <ast@...nel.org>, 
	Daniel Borkmann <daniel@...earbox.net>, Andrii Nakryiko <andrii@...nel.org>, 
	Martin KaFai Lau <martin.lau@...ux.dev>, Eduard Zingerman <eddyz87@...il.com>, Song Liu <song@...nel.org>, 
	Yonghong Song <yonghong.song@...ux.dev>, John Fastabend <john.fastabend@...il.com>, 
	KP Singh <kpsingh@...nel.org>, Stanislav Fomichev <sdf@...ichev.me>, Hao Luo <haoluo@...gle.com>, 
	Jiri Olsa <jolsa@...nel.org>, Masahiro Yamada <masahiroy@...nel.org>, 
	Nathan Chancellor <nathan@...nel.org>, Nicolas Schier <nicolas@...sle.eu>, Kui-Feng Lee <kuifeng@...com>, 
	Alan Maguire <alan.maguire@...cle.com>, Martin Rodriguez Reboredo <yakoyoku@...il.com>, 
	Miguel Ojeda <ojeda@...nel.org>, bpf@...r.kernel.org, linux-kbuild@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH bpf-next] kbuild, bpf: Enable reproducible BTF generation

On Thu, Dec 12, 2024 at 1:07 PM Thomas Weißschuh <linux@...ssschuh.net> wrote:
>
> Hi Andrii,
>
> On 2024-12-12 11:23:03-0800, Andrii Nakryiko wrote:
> > On Tue, Dec 10, 2024 at 10:24 PM Thomas Weißschuh <linux@...ssschuh.net> wrote:
> > > On 2024-12-11 00:17:02+0000, Ihor Solodrai wrote:
> > > > On Tuesday, December 10th, 2024 at 3:23 PM, Thomas Weißschuh <linux@...ssschuh.net> wrote:
> > > >
> > > > >
> > > > >
> > > > > Pahole v1.27 added a new BTF generation feature to support
> > > > > reproducibility in the face of multithreading.
> > > > > Enable it if supported and reproducible builds are requested.
> > > > >
> > > > > As unknown --btf_features are ignored, avoid the test for the pahole
> > > > > version to keep the line readable.
> > > > >
> > > > > Fixes: b4f72786429c ("scripts/pahole-flags.sh: Parse DWARF and generate BTF with multithreading.")
> > > > > Fixes: 72d091846de9 ("kbuild: avoid too many execution of scripts/pahole-flags.sh")
> > > > > Link: https://lore.kernel.org/lkml/4154d202-5c72-493e-bf3f-bce882a296c6@gentoo.org/
> > > > > Link: https://lore.kernel.org/lkml/20240322-pahole-reprodicible-v1-1-3eaafb1842da@weissschuh.net/
> > > > > Signed-off-by: Thomas Weißschuh linux@...ssschuh.net
> > > > >
> > > > > ---
> > > > > scripts/Makefile.btf | 1 +
> > > > > 1 file changed, 1 insertion(+)
> > > > >
> > > > > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> > > > > index c3cbeb13de503555adcf00029a0b328e74381f13..da23265bc8b3cf43c0a1c89fbc4f53815a290e13 100644
> > > > > --- a/scripts/Makefile.btf
> > > > > +++ b/scripts/Makefile.btf
> > > > > @@ -22,6 +22,7 @@ else
> > > > >
> > > > > # Switch to using --btf_features for v1.26 and later.
> > > > > pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j$(JOBS) --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
> > > > > +pahole-flags-$(if $(KBUILD_BUILD_TIMESTAMP),y) += --btf_features=reproducible_build
> > > >
> > > > Hi Thomas,
> > > >
> > > > There are a couple of issues with reproducible_build flag which I
> > > > think are worth mentioning here. I don't know all the reasons behind
> > > > adding this now, and it's optional too, so feel free to discard my
> > > > comments.
> > > >
> > > > Currently with this flag, the BTF output is deterministic for a given
> > > > order of DWARF compilation units. So the BTF will be the same for the
> > > > same vmlinux binary. However, if the vmlinux is rebuilt due to an
> > > > incremental change in a source code, my understanding is that there is
> > > > no guarantee that DWARF CUs will be in the same order in the binary.
> > >
> > > The goal behind reproducible builds is to produce bit-by-bit idential
> > > binaries. If the CUs are in a different order then that requirement
> > > would have been broken there already.
> >
> > I'm curious, how do we guarantee that we get bit-by-bit identical
> > DWARF? Do we enforce the order of linking of .o files into the final
> > vmlinux? Is this described anywhere?
>
> The CU order has to be fixed, otherwise the non-debugging parts of the
> binary would not be reproducible either.
> For docs is Documentation/kbuild/reproducible-builds.rst, the linked
> reproducible-builds.org project has much more information.
>
> Also besides reproducible builds, lots of kernel components rely
> (accidentally or intentionally) on a stable initialization order, which
> is also defined by linking order.
>
> From Documentation/kbuild/makefiles.rst:
>
>         Link order is significant, because certain functions
>         (module_init() / __initcall) will be called during boot in the
>         order they appear. So keep in mind that changing the link
>         order may e.g. change the order in which your SCSI
>         controllers are detected, and thus your disks are renumbered.
>
> > > For an incremental build a full relink with *all* CUs is done, not only
> > > the changed once, so the order should always be the same.
> >
> > The concern here is whether linker guarantees that CUs' DWARF data
> > will be appended in exactly the same order in such case?
>
> Otherwise it wouldn't be reproducible in general.
> The pahole developers specifically implemented
> --btf_features=reproducible_build for use in the kernel; after I sent
> a precursor patch to this one (also linked in the patch):
>
> https://lore.kernel.org/lkml/20240322-pahole-reprodicible-v1-1-3eaafb1842da@weissschuh.net/
>
> In general the kernel already supports reproducible builds.

Great, thanks for all the info!

I do agree with Ihor that KBUILD_BUILD_TIMESTAMP is a non-obvious and
surprising way to enable this behavior, but if that's what's used for
other aspects of kernel build I guess it's fine by me.

Ihor's work on making BTF generation more deterministic w.r.t. CU
order would automatically benefit --btf_features=reproducible_build in
the end and might make it unnecessary, but there is no need to block
on a completion of that work.

>
> For my personal kernel builds only two incompatibilities/rough edges remain:
>
> * (parallel) BTF generation, which is fixed with this patch
> * CONFIG_MODULE_SIG with non-precreated keys (which I am working on)
>
> > > > At the same time, reproducible_build slows down BTF generation by
> > > > 30-50%, maybe more depending on the kernel config.
> > >
> > > If a user explicitly requests reproducibility then they should get it,
> > > even if it is slower.
> > >
> > > > Hopefully these problems will be solved in upcoming pahole releases.
> > >
> > > I don't see it as big problem. This is used for release builds, not
> > > during development.
> > >
> > > > Question: why KBUILD_BUILD_TIMESTAMP flag? Isn't it more appropriate
> > > > to use a separate flag for this particular feature?
> > >
> > > Adding an additional variable would need to be documented and would
> > > makes the feature harder to use. KBUILD_BUILD_TIMESTAMP already needs to
> > > be set by the user if they are building for reproducibility.
> > >
> > > > > ifneq ($(KBUILD_EXTMOD),)
> > > > > module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base
> > > > >
> > > > > ---
> > > > > base-commit: 7cb1b466315004af98f6ba6c2546bb713ca3c237
> > > > > change-id: 20241124-pahole-reproducible-2b879ac8bdab
> > > > >
> > > > > Best regards,
> > > > > --
> > > > > Thomas Weißschuh linux@...ssschuh.net

Powered by blists - more mailing lists