linux-kernel - Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <a45nq7wustxrztjxmkqzevv3mkki5oizfik7b24gqiyldhlkhv@4rpy4tzwi52l>
Date: Thu, 22 Aug 2024 11:55:05 +0800
From: Shung-Hsi Yu <shung-hsi.yu@...e.com>
To: Arnaldo Carvalho de Melo <acme@...nel.org>, dwarves@...r.kernel.org
Cc: Jiri Slaby <jirislaby@...nel.org>, Jiri Olsa <olsajiri@...il.com>, 
	masahiroy@...nel.org, linux-kernel@...r.kernel.org, 
	Nathan Chancellor <nathan@...nel.org>, Nicolas Schier <nicolas@...sle.eu>, 
	Alexei Starovoitov <ast@...nel.org>, Daniel Borkmann <daniel@...earbox.net>, 
	Andrii Nakryiko <andrii@...nel.org>, Martin KaFai Lau <martin.lau@...ux.dev>, 
	Eduard Zingerman <eddyz87@...il.com>, Song Liu <song@...nel.org>, 
	Yonghong Song <yonghong.song@...ux.dev>, John Fastabend <john.fastabend@...il.com>, 
	KP Singh <kpsingh@...nel.org>, Stanislav Fomichev <sdf@...ichev.me>, 
	Hao Luo <haoluo@...gle.com>, linux-kbuild@...r.kernel.org, bpf@...r.kernel.org, 
	msuchanek@...e.com
Subject: Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

(Add pahole maintainer and mailing list)

Hi Arnaldo,

We're running into kernel build failure on 32-bit (both full 32-bit and
32-bit userspace on 64-bit kernel) because pahole crashed due to virtual
memory exhaustion[1]. As a workaround we currently limit pahole's
parallel job count to 1 on such system[2]:

On Tue, 20 Aug 2024 10:59:50AM +0200, Jiri Slaby wrote:
[...]
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> index b75f09f3f424..f7de8e922bce 100644
> --- a/scripts/Makefile.btf
> +++ b/scripts/Makefile.btf
> @@ -12,7 +12,9 @@ endif
>  
>  pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
>  
> +ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
>  pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
> +endif
>  
>  pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
>  
> diff --git a/scripts/pahole-class.sh b/scripts/pahole-class.sh
> new file mode 100644
> index 000000000000..d15a92077f76
> --- /dev/null
> +++ b/scripts/pahole-class.sh
> @@ -0,0 +1,21 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Usage: $ ./pahole-class.sh pahole
> +#
> +# Prints pahole's ELF class, such as ELF64
> +
> +if [ ! -x "$(command -v "$@")" ]; then
> +	echo 0
> +	exit 1
> +fi
> +
> +PAHOLE="$(which "$@")"
> +CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
> +
> +# Scripts like scripts/dummy-tools/pahole
> +if [ -n "$CLASS" ]; then
> +	echo "$CLASS"
> +else
> +	echo ELF64
> +fi
> -- 

This helped lowered the memory usage enough so pahole no longer crash:

On Wed, Aug 21, 2024 at 09:29:57AM GMT, Jiri Slaby wrote:
> On 21. 08. 24, 8:40, Jiri Slaby wrote:
> >  From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
> > Run on 64bit:
> > pahole -j32 -> 4.102 GB
> > pahole -j16 -> 3.895 GB
> > pahole -j1 -> 3.706 GB
> > 
> > On 32bit (the same vmlinux):
> > pahole -j32 -> 2.870 GB (crash)
> > pahole -j16 -> 2.810 GB
> > pahole -j1 -> 2.444 GB

Jiri (Slaby) in the meanwhile has also proposed structure packing to
further reduce memory usage. (Note: I think the numbers below are from a
64-bit machine)

> From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
> (In reply to Jiri Slaby from comment #20)
> > | |   |   ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun
> (dwarf_loader.c:959)
> 
> So given this struct class_member is the largest consumer, running pahole on
> pahole. The below results in 4.102 GB -> 3.585 GB savings.
> 
> --- a/dwarves.h
> +++ b/dwarves.h
> @@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,
>   */
>  struct tag {
>         struct list_head node;
> +       const char       *attribute;
> +       void             *priv;
>         type_id_t        type;
>         uint16_t         tag;
> +       uint16_t         recursivity_level;
>         bool             visited;
>         bool             top_level;
>         bool             has_btf_type_tag;
> -       uint16_t         recursivity_level;
> -       const char       *attribute;
> -       void             *priv;
>  };
> 
>  // To use with things like type->type_enum ==
> perf_event_type+perf_user_event_type
> @@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct
> function *func)
>  struct class_member {
>         struct tag       tag;
>         const char       *name;
> +       uint64_t         const_value;
>         uint32_t         bit_offset;
>         uint32_t         bit_size;
>         uint32_t         byte_offset;
>         int              hole;
>         size_t           byte_size;
> +       uint32_t         alignment;
>         int8_t           bitfield_offset;
>         uint8_t          bitfield_size;
>         uint8_t          bit_hole;
>         uint8_t          bitfield_end:1;
> -       uint64_t         const_value;
> -       uint32_t         alignment;
>         uint8_t          visited:1;
>         uint8_t          is_static:1;
>         uint8_t          has_bit_offset:1;
>--

What do you think?

IIUC pahole's memory usage is largely tied to the number of entries in
vmlinux/kmodule DWARF, and there probably isn't much we could do about
that.

Shung-Hsi

1: https://bugzilla.suse.com/show_bug.cgi?id=1229450
2: https://lore.kernel.org/all/20240820085950.200358-1-jirislaby@kernel.org/