Message-ID: <CA+icZUUdevE_n4+PgwisFdpxz=7XwaMciVKn+XnDHo-=UqRZ7A@mail.gmail.com>
Date: Thu, 15 Aug 2024 22:13:53 +0200
From: Sedat Dilek <sedat.dilek@...il.com>
To: Sami Tolvanen <samitolvanen@...gle.com>
Cc: Masahiro Yamada <masahiroy@...nel.org>, Luis Chamberlain <mcgrof@...nel.org>, 
	Miguel Ojeda <ojeda@...nel.org>, Greg Kroah-Hartman <gregkh@...uxfoundation.org>, 
	Matthew Maurer <mmaurer@...gle.com>, Alex Gaynor <alex.gaynor@...il.com>, 
	Wedson Almeida Filho <wedsonaf@...il.com>, Gary Guo <gary@...yguo.net>, Petr Pavlu <petr.pavlu@...e.com>, 
	Neal Gompa <neal@...pa.dev>, Hector Martin <marcan@...can.st>, Janne Grunau <j@...nau.net>, 
	Asahi Linux <asahi@...ts.linux.dev>, linux-kbuild@...r.kernel.org, 
	linux-kernel@...r.kernel.org, linux-modules@...r.kernel.org, 
	rust-for-linux@...r.kernel.org
Subject: Re: [PATCH v2 00/19] Implement DWARF modversions

On Thu, Aug 15, 2024 at 7:39 PM Sami Tolvanen <samitolvanen@...gle.com> wrote:
>
> Hi,
>
> Here's v2 of the DWARF modversions series [1]. The main motivation
> remains modversions support for Rust, which is important for
> distributions like Android that are eager to ship Rust kernel
> modules. However, per Luis' request [2], v2 drops all Rust specific
> bits from the series and instead adds the feature as an option
> for the entire kernel. Matt is addressing Rust modversion_info
> compatibility issues in a separate series [3], and we'll follow up
> with a patch to actually allow CONFIG_MODVERSIONS with Rust once
> these have been sorted out.
>
> A short background recap: Unlike C, Rust source code doesn't have
> sufficient information about the final ABI, as the compiler has
> considerable freedom in adjusting structure layout for improved
> performance [4], for example, which makes using a source code
> parser like genksyms a non-starter. Based on Matt's suggestion and
> previous feedback from maintainers, this series uses DWARF debugging
> information for computing versions. DWARF is an established and
> a relatively stable format, which includes all the necessary ABI
> details, and adding a CONFIG_DEBUG_INFO dependency for Rust symbol
> versioning seems like a reasonable trade-off.
>
> The first 16 patches of this series add a small tool for computing
> symbol versions from DWARF, called gendwarfksyms. When passed a
> list of exported symbols and an object file, the tool generates
> an expanded type string for each symbol, and computes symbol CRCs
> similarly to genksyms. gendwarfksyms is written in C and uses libdw
> to process DWARF, mainly because of the existing support for C host
> tools that use elfutils (e.g., objtool). The next two patches ensure
> that debugging information is present where we need it and fix a
> compilation issue with x86 asm-prototypes.h. The last patch adds
> gendwarfksyms as an alternative to genksyms.
>
> A quick note about performance: On my development system, building
> x86_64 defconfig with MODVERSIONS takes about 59.4s with gcc 13
> (avg. of ten runs). Adding DEBUG_INFO_DWARF5 increases the build
> time by ~23% to 73.3s. Switching from GENKSYMS to GENDWARFKSYMS
> reduces the build time by 6% to 68.9s, which is still ~16% slower
> than genksyms without debugging information. Therefore, if you
> already build kernels with debugging information, gendwarfksyms
> should be slightly faster. YMMV, of course.
>
> Things would change with LTO, because we won't have full DWARF
> until we have an ELF binary, which means we'd have to process
> vmlinux.o. This version of gendwarfksyms is still single-threaded
> as it seems we can't rely on libdw to be thread-safe. Processing
> a ThinLTO x86_64 defconfig vmlinux.o on my system takes ~2m16s,
> and would have to happen even on incremental builds, just like
> LTO linking itself. As cross-language LTO presumably isn't wildly
> popular yet, gendwarfksyms intentionally depends in !LTO in this
> version.
>
> Looking forward to hearing your thoughts!
>

Hi Sami,

so this work is on top of Linux v6.11-rc3 - can you tag it as gendwarfksyms-v2?

Thanks.

Best regards,
-Sedat-

https://github.com/samitolvanen/linux/tree/gendwarfksyms
https://github.com/samitolvanen/linux/tags

> Sami
>
> [1] https://lore.kernel.org/lkml/20240617175818.58219-17-samitolvanen@google.com/
> [2] https://lore.kernel.org/lkml/ZnIZEtkkQWEIGf9n@bombadil.infradead.org/
> [3] https://lore.kernel.org/lkml/20240806212106.617164-1-mmaurer@google.com/
> [4] https://lore.kernel.org/rust-for-linux/CAGSQo005hRiUZdeppCifDqG9zFDJRwahpBLE4x7-MyfJscn7tQ@mail.gmail.com/
>
> ---
>
> Changes in v2:
> - Per Luis' request, dropped Rust-specific patches and added
>   gendwarfksyms as an alternative to genksyms for the entire
>   kernel.
>
> - Added support for missing DWARF features needed to handle
>   also non-Rust code.
>
> - Changed symbol address matching to use the symbol table
>   information instead of relying on addresses in DWARF.
>
> - Added __gendwarfksyms_ptr patches to ensure the compiler emits
>   the necessary type information in DWARF even for symbols that
>   are defined in other TUs.
>
> - Refactored debugging output and moved the more verbose output
>   behind --dump* flags.
>
> - Added a --symtypes flag for generating a genksyms-style
>   symtypes output based on Petr's feedback, and refactored
>   symbol version calculations to be based on symtypes instead
>   of raw --dump-dies output.
>
> - Based on feedback from Greg and Petr, added --stable flag and
>   support for reserved data structure fields and declaration-onl
>   structures. Also added examples for using these features.
>
> - Added a GENDWARFKSYMS option and hooked up kbuild support
>   for both C and assembly code. Note that with gendwarfksyms,
>   we have to actually build a temporary .o file for calculating
>   assembly modversions.
>
> ---
>
> Sami Tolvanen (19):
>   tools: Add gendwarfksyms
>   gendwarfksyms: Add symbol list handling
>   gendwarfksyms: Add address matching
>   gendwarfksyms: Add support for type pointers
>   gendwarfksyms: Expand base_type
>   gendwarfksyms: Add a cache for processed DIEs
>   gendwarfksyms: Expand type modifiers and typedefs
>   gendwarfksyms: Expand subroutine_type
>   gendwarfksyms: Expand array_type
>   gendwarfksyms: Expand structure types
>   gendwarfksyms: Limit structure expansion
>   gendwarfksyms: Add die_map debugging
>   gendwarfksyms: Add symtypes output
>   gendwarfksyms: Add symbol versioning
>   gendwarfksyms: Add support for declaration-only data structures
>   gendwarfksyms: Add support for reserved structure fields
>   export: Add __gendwarfksyms_ptr_ references to exported symbols
>   x86/asm-prototypes: Include <asm/ptrace.h>
>   kbuild: Add gendwarfksyms as an alternative to genksyms
>
>  arch/x86/include/asm/asm-prototypes.h     |   1 +
>  include/linux/export.h                    |  15 +
>  kernel/module/Kconfig                     |  31 +
>  scripts/Makefile                          |   3 +-
>  scripts/Makefile.build                    |  34 +-
>  scripts/gendwarfksyms/.gitignore          |   2 +
>  scripts/gendwarfksyms/Makefile            |  12 +
>  scripts/gendwarfksyms/cache.c             |  51 ++
>  scripts/gendwarfksyms/crc32.c             |  69 ++
>  scripts/gendwarfksyms/crc32.h             |  34 +
>  scripts/gendwarfksyms/die.c               | 196 +++++
>  scripts/gendwarfksyms/dwarf.c             | 973 ++++++++++++++++++++++
>  scripts/gendwarfksyms/examples/declonly.c |  31 +
>  scripts/gendwarfksyms/examples/reserved.c |  66 ++
>  scripts/gendwarfksyms/gendwarfksyms.c     | 201 +++++
>  scripts/gendwarfksyms/gendwarfksyms.h     | 275 ++++++
>  scripts/gendwarfksyms/symbols.c           | 392 +++++++++
>  scripts/gendwarfksyms/types.c             | 557 +++++++++++++
>  18 files changed, 2936 insertions(+), 7 deletions(-)
>  create mode 100644 scripts/gendwarfksyms/.gitignore
>  create mode 100644 scripts/gendwarfksyms/Makefile
>  create mode 100644 scripts/gendwarfksyms/cache.c
>  create mode 100644 scripts/gendwarfksyms/crc32.c
>  create mode 100644 scripts/gendwarfksyms/crc32.h
>  create mode 100644 scripts/gendwarfksyms/die.c
>  create mode 100644 scripts/gendwarfksyms/dwarf.c
>  create mode 100644 scripts/gendwarfksyms/examples/declonly.c
>  create mode 100644 scripts/gendwarfksyms/examples/reserved.c
>  create mode 100644 scripts/gendwarfksyms/gendwarfksyms.c
>  create mode 100644 scripts/gendwarfksyms/gendwarfksyms.h
>  create mode 100644 scripts/gendwarfksyms/symbols.c
>  create mode 100644 scripts/gendwarfksyms/types.c
>
>
> base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
> --
> 2.46.0.184.g6999bdac58-goog
>
>

Powered by blists - more mailing lists