Message-ID: <CAF1bQ=Rb=0Etk2GLaWR+dXtMG00NhVYcHwx-E0=2FYu4mE1B_A@mail.gmail.com>
Date: Mon, 2 Jun 2025 15:20:42 -0700
From: Rong Xu <xur@...gle.com>
To: Masahiro Yamada <masahiroy@...nel.org>
Cc: Nathan Chancellor <nathan@...nel.org>, Eric Naim <dnaim@...hyos.org>, 
	Nicolas Schier <nicolas.schier@...ux.dev>, 
	Nick Desaulniers <nick.desaulniers+lkml@...il.com>, Bill Wendling <morbo@...gle.com>, 
	Justin Stitt <justinstitt@...gle.com>, Miguel Ojeda <ojeda@...nel.org>, 
	Thomas Gleixner <tglx@...utronix.de>, Alice Ryhl <aliceryhl@...gle.com>, 
	Sami Tolvanen <samitolvanen@...gle.com>, "Mike Rapoport (Microsoft)" <rppt@...nel.org>, 
	Rafael Aquini <aquini@...hat.com>, Michael Ellerman <mpe@...erman.id.au>, 
	Stafford Horne <shorne@...il.com>, Christophe Leroy <christophe.leroy@...roup.eu>, 
	Teresa Johnson <tejohnson@...gle.com>, linux-kernel@...r.kernel.org, 
	linux-kbuild@...r.kernel.org, llvm@...ts.linux.dev
Subject: Re: [PATCH v3] kbuild: distributed build support for Clang ThinLTO

On Fri, May 30, 2025 at 11:48 PM Masahiro Yamada <masahiroy@...nel.org> wrote:
>
> On Wed, May 28, 2025 at 6:05 AM Rong Xu <xur@...gle.com> wrote:
> >
> > On Mon, May 26, 2025 at 6:12 AM Masahiro Yamada <masahiroy@...nel.org> wrote:
> > >
> > > On Thu, May 22, 2025 at 6:35 AM <xur@...gle.com> wrote:
> > > >
> > > > From: Rong Xu <xur@...gle.com>
> > > >
> > > > Add distributed ThinLTO build support for the Linux kernel.
> > > > This new mode offers several advantages: (1) Increased
> > > > flexibility in handling user-specified build options.
> > > > (2) Improved user-friendliness for developers. (3) Greater
> > > > convenience for integrating with objtool and livepatch.
> > > >
> > > > Note that "distributed" in this context refers to a term
> > > > that differentiates in-process ThinLTO builds by invoking
> > > > backend compilation through the linker, not necessarily
> > > > building in distributed environments.
> > > >
> > > > Distributed ThinLTO is enabled via the
> > > > `CONFIG_LTO_CLANG_THIN_DIST` Kconfig option. For example:
> > > >  > make LLVM=1 defconfig
> > > >  > scripts/config -e LTO_CLANG_THIN_DIST
> > > >  > make LLVM=1 oldconfig
> > > >  > make LLVM=1 vmlinux -j <..>
> > > >
> > > > The implementation changes the top-level Makefile with a
> > > > macro for generating `vmlinux.o` for distributed ThinLTO
> > > > builds. It uses the existing Kbuild infrastructure to
> > > > perform two recursive passes through the subdirectories.
> > > > The first pass generates LLVM IR object files, similar to
> > > > in-process ThinLTO. Following the thin-link stage, a second
> > > > pass compiles these IR files into the final native object
> > > > files. The build rules and actions for this two-pass process
> > > > are primarily implemented in `scripts/Makefile.build`.
> > > >
> > > > Currently, this patch focuses on building the main kernel
> > > > image (`vmlinux`) only. Support for building kernel modules
> > > > using this method is planned for a subsequent patch.
> > > >
> > > > Tested on the following arch: x86, arm64, loongarch, and
> > > > riscv.
> > > >
> > > > Some implementation details can be found here:
> > > > https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934
> > > >
> > > > Signed-off-by: Rong Xu <xur@...gle.com>
> > > > ---
> > > > Changelog since v1:
> > > > - Updated the description in arch/Kconfig based on feedback
> > > >   from Nathan Chancellor
> > > > - Revised file suffixes: .final_o -> .o.thinlto.native, and
> > > >   .final_a -> .a.thinlto.native
> > > > - Updated list of ignored files in .gitignore
> > > >
> > > > Changelog since v2:
> > > > - Changed file suffixes: .o.thinlto.native -> .o_thinlto_native,
> > > >   and .a.thinlto.native -> .a_thinlto_native so that basename
> > > >   works as intended.
> > > > - Tested the patch with AutoFDO and Propeller.
> > > > ---
> > > >  .gitignore                        |  3 ++
> > > >  MAINTAINERS                       |  5 +++
> > > >  Makefile                          | 40 ++++++++++++++++++++---
> > > >  arch/Kconfig                      | 19 +++++++++++
> > > >  scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
> > > >  scripts/Makefile.lib              |  7 +++-
> > > >  scripts/Makefile.vmlinux_o        | 16 +++++++---
> > > >  scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
> > > >  scripts/head-object-list.txt      |  1 +
> > > >  9 files changed, 181 insertions(+), 15 deletions(-)
> > > >  create mode 100644 scripts/Makefile.vmlinux_thinlink
> > >
> > > I re-implemented the Makefiles to avoid
> > > the second recursion and hacky ifdefs.
> > > Attached.
> > >
> > > The topic branch is available in
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
> > > thinlto-dist-refactor
> > >
> > > I only compile and boot tested on x86 QEMU.
> > This implementation does look cleaner. But with one issue: it has a
> > unified cflags for
> > all the BE compilations. This means per-file flags, such as
> > CFLAGS_fork.o = -fabc,
> > are lost during the BE compilation for fork.thinlto_native.o.
> >
> > This exact issue was what I aimed to prevent with the two-recursion approach.
> > If we must avoid two recursions, perhaps we could leverage the saved per-file
> > pre-link commands (i.e., .*.o.cmd)?
>
> How important is this?
It is one of the main reasons for using distributed mode. For example,
for some source, the user chooses to use -O1 (or disable some loop
transformations), there is no way to do this with one-unified options.

>
> I do not know which compiler flags are consumed in the
> distributed thin lto stage.
>
> Are only compiler flags starting "-f" relevant?
All the flags that have impact on BE compilation need to pass to BE.
Yes, usually most -f flags are relevant, all the internal options,
like -mllvm -foo need
to be passed to BE. -D, -i, -include are OK to filter.

>
> In your implementation, you filter-out many compiler flags.
These are flags for the compiler front-end (clang). They don't have impact on
BE compilation.

>
>
> --
> Best Regards
> Masahiro Yamada

Powered by blists - more mailing lists