linux-kernel - Re: [PATCH v2] kbuild: distributed build support for Clang ThinLTO

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAF1bQ=Sh+N1ifCHK-15zeqt1tyzFtr-+nEJmSjEJOgfXgK9ufg@mail.gmail.com>
Date: Wed, 21 May 2025 09:40:53 -0700
From: Rong Xu <xur@...gle.com>
To: Eric Naim <dnaim@...hyos.org>
Cc: Masahiro Yamada <masahiroy@...nel.org>, Nathan Chancellor <nathan@...nel.org>, 
	Nicolas Schier <nicolas.schier@...ux.dev>, 
	Nick Desaulniers <nick.desaulniers+lkml@...il.com>, Bill Wendling <morbo@...gle.com>, 
	Justin Stitt <justinstitt@...gle.com>, Miguel Ojeda <ojeda@...nel.org>, 
	Thomas Gleixner <tglx@...utronix.de>, Alice Ryhl <aliceryhl@...gle.com>, 
	Sami Tolvanen <samitolvanen@...gle.com>, "Mike Rapoport (Microsoft)" <rppt@...nel.org>, 
	Rafael Aquini <aquini@...hat.com>, Michael Ellerman <mpe@...erman.id.au>, 
	Stafford Horne <shorne@...il.com>, Christophe Leroy <christophe.leroy@...roup.eu>, 
	Teresa Johnson <tejohnson@...gle.com>, linux-kernel@...r.kernel.org, 
	linux-kbuild@...r.kernel.org, llvm@...ts.linux.dev
Subject: Re: [PATCH v2] kbuild: distributed build support for Clang ThinLTO

On Mon, May 19, 2025 at 10:22 PM Eric Naim <dnaim@...hyos.org> wrote:
>
> Hi Rong Xu,
>
> On 5/8/25 04:55, xur@...gle.com wrote:
> > From: Rong Xu <xur@...gle.com>
> >
> > Add distributed ThinLTO build support for the Linux kernel.
> > This new mode offers several advantages: (1) Increased
> > flexibility in handling user-specified build options.
> > (2) Improved user-friendliness for developers. (3) Greater
> > convenience for integrating with objtool and livepatch.
> >
> > Note that "distributed" in this context refers to a term
> > that differentiates in-process ThinLTO builds by invoking
> > backend compilation through the linker, not necessarily
> > building in distributed environments.
> >
> > Distributed ThinLTO is enabled via the
> > `CONFIG_LTO_CLANG_THIN_DIST` Kconfig option. For example:
> >  > make LLVM=1 defconfig
> >  > scripts/config -e LTO_CLANG_THIN_DIST
> >  > make LLVM=1 oldconfig
> >  > make LLVM=1 vmlinux -j <..>
> >
> > The implementation changes the top-level Makefile with a
> > macro for generating `vmlinux.o` for distributed ThinLTO
> > builds. It uses the existing Kbuild infrastructure to
> > perform two recursive passes through the subdirectories.
> > The first pass generates LLVM IR object files, similar to
> > in-process ThinLTO. Following the thin-link stage, a second
> > pass compiles these IR files into the final native object
> > files. The build rules and actions for this two-pass process
> > are primarily implemented in `scripts/Makefile.build`.
> >
> > Currently, this patch focuses on building the main kernel
> > image (`vmlinux`) only. Support for building kernel modules
> > using this method is planned for a subsequent patch.
> >
> > Tested on the following arch: x86, arm64, loongarch, and
> > riscv.
> >
> > Some implementation details can be found here:
> > https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934
> >
> > Signed-off-by: Rong Xu <xur@...gle.com>
> > ---
> > Changelog since v1:
> > - Updated the description in arch/Kconfig based on feedback
> >   from Nathan Chancellor
> > - Revised file suffixes: .final_o -> .o.thinlto.native, and
> >   .final_a -> .a.thinlto.native
> > - Updated list of ignored files in .gitignore
> > ---
> >  .gitignore                        |  2 ++
> >  MAINTAINERS                       |  5 +++
> >  Makefile                          | 40 ++++++++++++++++++++---
> >  arch/Kconfig                      | 19 +++++++++++
> >  scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
> >  scripts/Makefile.lib              |  7 +++-
> >  scripts/Makefile.vmlinux_o        | 16 +++++++---
> >  scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
> >  scripts/head-object-list.txt      |  1 +
> >  9 files changed, 180 insertions(+), 15 deletions(-)
> >  create mode 100644 scripts/Makefile.vmlinux_thinlink
> >
>
> I noticed that both Makefile.autofdo and Makefile.propeller add extra linker flags when building with ThinLTO. Did you miss updating that or is the omission there intentional?

Thanks for catching this! One good aspect of distributed build mode is
that we no longer need the extra linker flags -- most of them are just
to pass the options to the BE compilation.
So this patch does not need these linker options.  But for the
Propeller build, we still need to pass one of the two profiles to the
final link, and I'll be sure to incorporate that into the patch.

However, I do need to make a change regarding file suffixes. The
is_kernel_obj macro in the Makefile.build uses the basename command.
The issue is that basename extracts everything before the last period
in a filename. So, for a file named "foo.o.thinlto.native", basename
returns "foo.o.thinlto", but kbuild expects it to return "foo".

To fix this, I'll adjust the suffixes to ".a_thinlto_native" and
".o_thinlto_native". I'll send the patch v3 shortly.

Thanks,

-Rong
>
> --
> Regards,
>   Eric