Message-ID: <20240919192408.e4phivt2h742e7mo@AALNPWDAGOMEZ1.aal.scsc.local>
Date: Thu, 19 Sep 2024 21:24:08 +0200
From: Daniel Gomez <da.gomez@...sung.com>
To: Sam James <sam@...too.org>
CC: Masahiro Yamada <masahiroy@...nel.org>, Kris Van Hees
	<kris.van.hees@...cle.com>, <linux-kernel@...r.kernel.org>,
	<linux-kbuild@...r.kernel.org>, <linux-modules@...r.kernel.org>,
	<linux-trace-kernel@...r.kernel.org>, Nick Alcock <nick.alcock@...cle.com>,
	Alan Maguire <alan.maguire@...cle.com>, Steven Rostedt
	<rostedt@...dmis.org>, Luis Chamberlain <mcgrof@...nel.org>, Masami
	Hiramatsu <mhiramat@...nel.org>, Nick Desaulniers <ndesaulniers@...gle.com>,
	Jiri Olsa <olsajiri@...il.com>, Elena Zannoni <elena.zannoni@...cle.com>
Subject: Re: [PATCH v10 2/4] kbuild: generate offset range data for builtin
 modules

On Thu, Sep 19, 2024 at 07:08:42PM +0100, Sam James wrote:
> Masahiro Yamada <masahiroy@...nel.org> writes:
> 
> > On Fri, Sep 20, 2024 at 2:07 AM Daniel Gomez <da.gomez@...sung.com> wrote:
> >>
> >> On Fri, Sep 06, 2024 at 10:45:03AM -0400, Kris Van Hees wrote:
> >> > Create file module.builtin.ranges that can be used to find where
> >> > built-in modules are located by their addresses. This will be useful for
> >> > tracing tools to find what functions are for various built-in modules.
> >> >
> >> > The offset range data for builtin modules is generated using:
> >> >  - modules.builtin: associates object files with module names
> >> >  - vmlinux.map: provides load order of sections and offset of first member
> >> >     per section
> >> >  - vmlinux.o.map: provides offset of object file content per section
> >> >  - .*.cmd: build cmd file with KBUILD_MODFILE
> >> >
> >> > The generated data will look like:
> >> >
> >> > .text 00000000-00000000 = _text
> >> > .text 0000baf0-0000cb10 amd_uncore
> >> > .text 0009bd10-0009c8e0 iosf_mbi
> >> > ...
> >> > .text 00b9f080-00ba011a intel_skl_int3472_discrete
> >> > .text 00ba0120-00ba03c0 intel_skl_int3472_discrete intel_skl_int3472_tps68470
> >> > .text 00ba03c0-00ba08d6 intel_skl_int3472_tps68470
> >> > ...
> >> > .data 00000000-00000000 = _sdata
> >> > .data 0000f020-0000f680 amd_uncore
> >> >
> >> > For each ELF section, it lists the offset of the first symbol.  This can
> >> > be used to determine the base address of the section at runtime.
> >> >
> >> > Next, it lists (in strict ascending order) offset ranges in that section
> >> > that cover the symbols of one or more builtin modules.  Multiple ranges
> >> > can apply to a single module, and ranges can be shared between modules.
> >> >
> >> > The CONFIG_BUILTIN_MODULE_RANGES option controls whether offset range data
> >> > is generated for kernel modules that are built into the kernel image.
> >> >
> >> > How it works:
> >> >
> >> >  1. The modules.builtin file is parsed to obtain a list of built-in
> >> >     module names and their associated object names (the .ko file that
> >> >     the module would be in if it were a loadable module, hereafter
> >> >     referred to as <kmodfile>).  This object name can be used to
> >> >     identify objects in the kernel compile because any C or assembler
> >> >     code that ends up into a built-in module will have the option
> >> >     -DKBUILD_MODFILE=<kmodfile> present in its build command, and those
> >> >     can be found in the .<obj>.cmd file in the kernel build tree.
> >> >
> >> >     If an object is part of multiple modules, they will all be listed
> >> >     in the KBUILD_MODFILE option argument.
> >> >
> >> >     This allows us to conclusively determine whether an object in the
> >> >     kernel build belong to any modules, and which.
> >> >
> >> >  2. The vmlinux.map is parsed next to determine the base address of each
> >> >     top level section so that all addresses into the section can be
> >> >     turned into offsets.  This makes it possible to handle sections
> >> >     getting loaded at different addresses at system boot.
> >> >
> >> >     We also determine an 'anchor' symbol at the beginning of each
> >> >     section to make it possible to calculate the true base address of
> >> >     a section at runtime (i.e. symbol address - symbol offset).
> >> >
> >> >     We collect start addresses of sections that are included in the top
> >> >     level section.  This is used when vmlinux is linked using vmlinux.o,
> >> >     because in that case, we need to look at the vmlinux.o linker map to
> >> >     know what object a symbol is found in.
> >> >
> >> >     And finally, we process each symbol that is listed in vmlinux.map
> >> >     (or vmlinux.o.map) based on the following structure:
> >> >
> >> >     vmlinux linked from vmlinux.a:
> >> >
> >> >       vmlinux.map:
> >> >         <top level section>
> >> >           <included section>  -- might be same as top level section)
> >> >             <object>          -- built-in association known
> >> >               <symbol>        -- belongs to module(s) object belongs to
> >> >               ...
> >> >
> >> >     vmlinux linked from vmlinux.o:
> >> >
> >> >       vmlinux.map:
> >> >         <top level section>
> >> >           <included section>  -- might be same as top level section)
> >> >             vmlinux.o         -- need to use vmlinux.o.map
> >> >               <symbol>        -- ignored
> >> >               ...
> >> >
> >> >       vmlinux.o.map:
> >> >         <section>
> >> >             <object>          -- built-in association known
> >> >               <symbol>        -- belongs to module(s) object belongs to
> >> >               ...
> >> >
> >> >  3. As sections, objects, and symbols are processed, offset ranges are
> >> >     constructed in a straight-forward way:
> >> >
> >> >       - If the symbol belongs to one or more built-in modules:
> >> >           - If we were working on the same module(s), extend the range
> >> >             to include this object
> >> >           - If we were working on another module(s), close that range,
> >> >             and start the new one
> >> >       - If the symbol does not belong to any built-in modules:
> >> >           - If we were working on a module(s) range, close that range
> >> >
> >> > Signed-off-by: Kris Van Hees <kris.van.hees@...cle.com>
> >> > Reviewed-by: Nick Alcock <nick.alcock@...cle.com>
> >> > Reviewed-by: Alan Maguire <alan.maguire@...cle.com>
> >> > Reviewed-by: Steven Rostedt (Google) <rostedt@...dmis.org>
> >> > Tested-by: Sam James <sam@...too.org>
> >> > ---
> >> >
> >> > Notes:
> >> >     Changes since v9:
> >> >      - Reverted support for build directory as optional 4th argument.
> >> >      - Added modules.builtin.ranges and vmlinux.o.map to CLEAN_FILES.
> >> >      - Fixed support for sparc64.
> >> >
> >> >     Changes since v8:
> >> >      - Added support for built-in Rust modules.
> >> >      - Added optional 4th argument to specify kernel build directory.
> >> >
> >> >     Changes since v7:
> >> >      - Removed extra close(fn).
> >> >      - Make CONFIG_BUILTIN_MODULE_RANGES depend on !lTO.
> >> >
> >> >     Changes since v6:
> >> >      - Applied Masahiro Yamada's suggestions (Kconfig, makefile, script).
> >> >
> >> >     Changes since v5:
> >> >      - Removed unnecessary compatibility info from option description.
> >> >
> >> >     Changes since v4:
> >> >      - Improved commit description to explain the why and how.
> >> >      - Documented dependency on GNU AWK for CONFIG_BUILTIN_MODULE_RANGES.
> >> >      - Improved comments in generate_builtin_ranges.awk
> >> >      - Improved logic in generate_builtin_ranges.awk to handle incorrect
> >> >        object size information in linker maps
> >> >
> >> >     Changes since v3:
> >> >      - Consolidated patches 2 through 5 into a single patch
> >> >      - Move CONFIG_BUILTIN_MODULE_RANGES to Kconfig.debug
> >> >      - Make CONFIG_BUILTIN_MODULE_RANGES select CONFIG_VMLINUX_MAP
> >> >      - Disable CONFIG_BUILTIN_MODULE_RANGES if CONFIG_LTO_CLANG_(FULL|THIN)=y
> >> >      - Support LLVM (lld) compiles in generate_builtin_ranges.awk
> >> >      - Support CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y
> >> >
> >> >     Changes since v2:
> >> >      - Add explicit dependency on FTRACE for CONFIG_BUILTIN_MODULE_RANGES
> >> >      - 1st arg to generate_builtin_ranges.awk is now modules.builtin.modinfo
> >> >      - Switched from using modules.builtin.objs to parsing .*.cmd files
> >> >      - Parse data from .*.cmd in generate_builtin_ranges.awk
> >> >      - Use $(real-prereqs) rather than $(filter-out ...)
> >> >     ---
> >> >
> >> >  Documentation/process/changes.rst   |   7 +
> >> >  Makefile                            |   1 +
> >> >  lib/Kconfig.debug                   |  15 +
> >> >  scripts/Makefile.vmlinux            |  18 +
> >> >  scripts/Makefile.vmlinux_o          |   3 +
> >> >  scripts/generate_builtin_ranges.awk | 508 ++++++++++++++++++++++++++++
> >> >  6 files changed, 552 insertions(+)
> >> >  create mode 100755 scripts/generate_builtin_ranges.awk
> >> >
> >> > diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
> >> > index 3fc63f27c226..00f1ed7c59c3 100644
> >> > --- a/Documentation/process/changes.rst
> >> > +++ b/Documentation/process/changes.rst
> >> > @@ -64,6 +64,7 @@ GNU tar                1.28             tar --version
> >> >  gtags (optional)       6.6.5            gtags --version
> >> >  mkimage (optional)     2017.01          mkimage --version
> >> >  Python (optional)      3.5.x            python3 --version
> >> > +GNU AWK (optional)     5.1.0            gawk --version
> >> >  ====================== ===============  ========================================
> >> >
> >> >  .. [#f1] Sphinx is needed only to build the Kernel documentation
> >> > @@ -192,6 +193,12 @@ platforms. The tool is available via the ``u-boot-tools`` package or can be
> >> >  built from the U-Boot source code. See the instructions at
> >> >  https://protect2.fireeye.com/v1/url?k=6b601b01-34fc322b-6b61904e-000babe598f7-59f65dfa7ee29fbf&q=1&e=8abd7076-5118-4660-a833-f762c2c71d32&u=https%3A%2F%2Fdocs.u-boot.org%2Fen%2Flatest%2Fbuild%2Ftools.html%23building-tools-for-linux
> >> >
> >> > +GNU AWK
> >> > +-------
> >> > +
> >> > +GNU AWK is needed if you want kernel builds to generate address range data for
> >> > +builtin modules (CONFIG_BUILTIN_MODULE_RANGES).
> >> > +
> >> >  System utilities
> >> >  ****************
> >> >
> >> > diff --git a/Makefile b/Makefile
> >> > index d57cfc6896b8..ec98a1e5b257 100644
> >> > --- a/Makefile
> >> > +++ b/Makefile
> >> > @@ -1482,6 +1482,7 @@ endif # CONFIG_MODULES
> >> >  # Directories & files removed with 'make clean'
> >> >  CLEAN_FILES += vmlinux.symvers modules-only.symvers \
> >> >              modules.builtin modules.builtin.modinfo modules.nsdeps \
> >> > +            modules.builtin.ranges vmlinux.o.map \
> >> >              compile_commands.json rust/test \
> >> >              rust-project.json .vmlinux.objs .vmlinux.export.c
> >> >
> >> > diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> >> > index a30c03a66172..5e2f30921cb2 100644
> >> > --- a/lib/Kconfig.debug
> >> > +++ b/lib/Kconfig.debug
> >> > @@ -571,6 +571,21 @@ config VMLINUX_MAP
> >> >         pieces of code get eliminated with
> >> >         CONFIG_LD_DEAD_CODE_DATA_ELIMINATION.
> >> >
> >> > +config BUILTIN_MODULE_RANGES
> >> > +     bool "Generate address range information for builtin modules"
> >> > +     depends on !LTO
> >> > +     depends on VMLINUX_MAP
> >> > +     help
> >> > +      When modules are built into the kernel, there will be no module name
> >> > +      associated with its symbols in /proc/kallsyms.  Tracers may want to
> >> > +      identify symbols by module name and symbol name regardless of whether
> >> > +      the module is configured as loadable or not.
> >> > +
> >> > +      This option generates modules.builtin.ranges in the build tree with
> >> > +      offset ranges (per ELF section) for the module(s) they belong to.
> >> > +      It also records an anchor symbol to determine the load address of the
> >> > +      section.
> >> > +
> >> >  config DEBUG_FORCE_WEAK_PER_CPU
> >> >       bool "Force weak per-cpu definitions"
> >> >       depends on DEBUG_KERNEL
> >> > diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux
> >> > index 5ceecbed31eb..dfb408aa19c6 100644
> >> > --- a/scripts/Makefile.vmlinux
> >> > +++ b/scripts/Makefile.vmlinux
> >> > @@ -33,6 +33,24 @@ targets += vmlinux
> >> >  vmlinux: scripts/link-vmlinux.sh vmlinux.o $(KBUILD_LDS) FORCE
> >> >       +$(call if_changed_dep,link_vmlinux)
> >> >
> >> > +# module.builtin.ranges
> >> > +# ---------------------------------------------------------------------------
> >> > +ifdef CONFIG_BUILTIN_MODULE_RANGES
> >> > +__default: modules.builtin.ranges
> >> > +
> >> > +quiet_cmd_modules_builtin_ranges = GEN     $@
> >> > +      cmd_modules_builtin_ranges = $(real-prereqs) > $@
> >> > +
> >> > +targets += modules.builtin.ranges
> >> > +modules.builtin.ranges: $(srctree)/scripts/generate_builtin_ranges.awk \
> >> > +                     modules.builtin vmlinux.map vmlinux.o.map FORCE
> >> > +     $(call if_changed,modules_builtin_ranges)
> >> > +
> >> > +vmlinux.map: vmlinux
> >> > +     @:
> >> > +
> >> > +endif
> >> > +
> >> >  # Add FORCE to the prerequisites of a target to force it to be always rebuilt.
> >> >  # ---------------------------------------------------------------------------
> >> >
> >> > diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
> >> > index d64070b6b4bc..0b6e2ebf60dc 100644
> >> > --- a/scripts/Makefile.vmlinux_o
> >> > +++ b/scripts/Makefile.vmlinux_o
> >> > @@ -45,9 +45,12 @@ objtool-args = $(vmlinux-objtool-args-y) --link
> >> >  # Link of vmlinux.o used for section mismatch analysis
> >> >  # ---------------------------------------------------------------------------
> >> >
> >> > +vmlinux-o-ld-args-$(CONFIG_BUILTIN_MODULE_RANGES)    += -Map=$@...p
> >> > +
> >> >  quiet_cmd_ld_vmlinux.o = LD      $@
> >> >        cmd_ld_vmlinux.o = \
> >> >       $(LD) ${KBUILD_LDFLAGS} -r -o $@ \
> >> > +     $(vmlinux-o-ld-args-y) \
> >> >       $(addprefix -T , $(initcalls-lds)) \
> >> >       --whole-archive vmlinux.a --no-whole-archive \
> >> >       --start-group $(KBUILD_VMLINUX_LIBS) --end-group \
> >> > diff --git a/scripts/generate_builtin_ranges.awk b/scripts/generate_builtin_ranges.awk
> >> > new file mode 100755
> >> > index 000000000000..b9ec761b3bef
> >> > --- /dev/null
> >> > +++ b/scripts/generate_builtin_ranges.awk
> >> > @@ -0,0 +1,508 @@
> >> > +#!/usr/bin/gawk -f
> >>
> >> This forces the gawk to be found always in /usr/bin. For systems where gawk can
> >> be located in other places, can we change the Shebang to:
> >>
> >> diff --git a/scripts/generate_builtin_ranges.awk b/scripts/generate_builtin_ranges.awk
> >> index b9ec761b3bef..886251c8d3f7 100755
> >> --- a/scripts/generate_builtin_ranges.awk
> >> +++ b/scripts/generate_builtin_ranges.awk
> >> @@ -1,4 +1,4 @@
> >> -#!/usr/bin/gawk -f
> >> +#!/usr/bin/env gawk -f
> >>  # SPDX-License-Identifier: GPL-2.0
> >>  # generate_builtin_ranges.awk: Generate address range data for builtin modules
> >>  # Written by Kris Van Hees <kris.van.hees@...cle.com>
> >
> >
> > No. We cannot fix it this way.

May I ask why if a distro installs gawk somewhere else, the "/usr/bin/env"
approach will not work either? I just want to understand that case.

> >
> >
> > I already pointed out this shebang issue.
> >
> > https://lore.kernel.org/lkml/CAK7LNASLc=ik9QdX4K_XuN=cg+1VcUBk-y5EnQEtOG+qOWaY=Q@mail.gmail.com/

To clarify, I've mentioned this because the patch landed in the linux-next
without the fix below. And I did not see there was a build error reporting it.

Thanks for sending the link!

> >
> >
> >
> > I thought Kris would send a fix up, but
> > perhaps people tend to be busy with LPC this week.
> >
> >
> 
> He did, see https://lore.kernel.org/all/20240912171646.1523528-1-kris.van.hees@oracle.com/.

Thanks for the link. That worked for me too.

> 
> >
> >> Not sure if it's too late? in that case I can send a patch to change this.
> >
> >
> > I can locally fix it up.
> >
> > Kris agreed with this fix.
> >
> >
> > diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux
> > index dfb408aa19c6..1284f05555b9 100644
> > --- a/scripts/Makefile.vmlinux
> > +++ b/scripts/Makefile.vmlinux
> > @@ -39,7 +39,7 @@ ifdef CONFIG_BUILTIN_MODULE_RANGES
> >  __default: modules.builtin.ranges
> >
> >  quiet_cmd_modules_builtin_ranges = GEN     $@
> > -      cmd_modules_builtin_ranges = $(real-prereqs) > $@
> > +      cmd_modules_builtin_ranges = gawk -f $(real-prereqs) > $@
> >
> >  targets += modules.builtin.ranges
> >  modules.builtin.ranges: $(srctree)/scripts/generate_builtin_ranges.awk \

Powered by blists - more mailing lists