Message-ID: <CAEf4BzbyjuM8wXChRejxkYFSm0o=+udr7uGink-5Wo1Ayb6M_A@mail.gmail.com>
Date: Wed, 26 Feb 2025 08:57:38 -0800
From: Andrii Nakryiko <andrii.nakryiko@...il.com>
To: Alan Maguire <alan.maguire@...cle.com>
Cc: Alexei Starovoitov <alexei.starovoitov@...il.com>, 
	Stephen Brennan <stephen.s.brennan@...cle.com>, Masahiro Yamada <masahiroy@...nel.org>, 
	Andrii Nakryiko <andrii@...nel.org>, Nicolas Schier <nicolas@...sle.eu>, Kees Cook <kees@...nel.org>, 
	KP Singh <kpsingh@...nel.org>, Martin KaFai Lau <martin.lau@...ux.dev>, 
	Sami Tolvanen <samitolvanen@...gle.com>, Eduard Zingerman <eddyz87@...il.com>, 
	linux-arch <linux-arch@...r.kernel.org>, Stanislav Fomichev <sdf@...ichev.me>, 
	Kent Overstreet <kent.overstreet@...ux.dev>, Pasha Tatashin <pasha.tatashin@...een.com>, 
	Jiri Olsa <jolsa@...nel.org>, John Fastabend <john.fastabend@...il.com>, 
	Jann Horn <jannh@...gle.com>, Ard Biesheuvel <ardb@...nel.org>, 
	Yonghong Song <yonghong.song@...ux.dev>, Hao Luo <haoluo@...gle.com>, 
	Andrew Morton <akpm@...ux-foundation.org>, 
	Linux Kbuild mailing list <linux-kbuild@...r.kernel.org>, Daniel Borkmann <daniel@...earbox.net>, 
	Arnd Bergmann <arnd@...db.de>, Nathan Chancellor <nathan@...nel.org>, linux-debuggers@...r.kernel.org, 
	Alexei Starovoitov <ast@...nel.org>, Song Liu <song@...nel.org>, LKML <linux-kernel@...r.kernel.org>, 
	bpf <bpf@...r.kernel.org>
Subject: Re: [PATCH 2/2] btf: Add the option to include global variable types

On Wed, Feb 26, 2025 at 6:20 AM Alan Maguire <alan.maguire@...cle.com> wrote:
>
> On 25/02/2025 21:52, Andrii Nakryiko wrote:
> > On Tue, Feb 25, 2025 at 2:02 AM Alan Maguire <alan.maguire@...cle.com> wrote:
> >>
> >> On 07/02/2025 23:50, Alexei Starovoitov wrote:
> >>> On Thu, Feb 6, 2025 at 5:21 PM Stephen Brennan
> >>> <stephen.s.brennan@...cle.com> wrote:
> >>>> When the feature was implemented in pahole, my measurements indicated
> >>>> that vmlinux BTF size increased by about 25.8%, and module BTF size
> >>>> increased by 53.2%. Due to these increases, the feature is implemented
> >>>> behind a new config option, allowing users sensitive to increased memory
> >>>> usage to disable it.
> >>>>
> >>>
> >>> ...
> >>>> +config DEBUG_INFO_BTF_GLOBAL_VARS
> >>>> +       bool "Generate BTF type information for all global variables"
> >>>> +       default y
> >>>> +       depends on DEBUG_INFO_BTF && PAHOLE_VERSION >= 128
> >>>> +       help
> >>>> +         Include type information for all global variables in the BTF. This
> >>>> +         increases the size of the BTF information, which increases memory
> >>>> +         usage at runtime. With global variable types available, runtime
> >>>> +         debugging and tracers may be able to provide more detail.
> >>>
> >>> This is not a solution.
> >>> Even if it's changed to 'default n' distros will enable it
> >>> like they enable everything and will suffer a regression.
> >>>
> >>> We need to add a new module like vmlinux_btf.ko that will contain
> >>> this additional BTF data. For global vars and everything else we might need.
> >>>
> >>
> >> In this area, I've been exploring adding support for
> >> CONFIG_DEBUG_INFO_BTF=m , so that the BTF info for vmlinux is delivered
> >> via a module. From the consumer side, everything looks identical
> >> (/sys/kernel/btf/vmlinux is there etc), it is just that the .BTF section
> >> is delivered via btf_vmlinux.ko instead. The original need for this was
> >> that embedded folks noted that because in the current situation BTF data
> >> is in vmlinux, they cannot enable BTF because such small-footprint
> >> systems do not support a large vmlinux binary. However they could
> >> potentially use kernel BTF if it was delivered via a module. The other
> >> nice thing about module delivery in the general case is we can make use
> >> of module compression. In experiments I see a 5.8Mb vmlinux BTF reduce
> >> to a 1.8Mb btf_vmlinux.ko.gz module on-disk.
> >>
> >> The challenge in delivering vmlinux BTF in a module is that on module
> >> load during boot other modules expect vmlinux BTF to be there when
> >> adding their own BTF to /sys/kernel/btf. And kfunc registration from
> >> kernel and modules expects this also. So support for deferred BTF module
> >> load/kfunc registration is required too. I've implemented the former and
> >> now am working on the latter. Hope to have some RFC patches ready soon,
> >> but it looks feasible at this point.
> >
> > Lazy btf_vmlinux.ko loading when BTF is actually needed (i.e., when
> > user reads /sys/kernel/btf/vmlinux for the first time; or when BPF
> > program is validated and needs kernel BTF) would be great. Curious too
> > see how all that fits together!
> >
> >>
> >> Assuming such an option was available to small-footprint systems, should
> >> we consider adding global variables to core vmlinux BTF along with
> >> per-cpu variables? Then vmlinux BTF extras could be used for some of the
> >> additional optional representations like function site-specific data
> >> (inlines etc)? Or are there other factors other than on-disk footprint
> >> that we need to consider? Thanks!
> >
> > I'd keep BTF for variables separate from "core" vmlinux BTF. We can
> > have /sys/kernel/btf/vmlinux.vars, which would depend on
> > /sys/kernel/btf/vmlinux as a base BTF. Separately, we could eventually
> > have /sys/kernel/btf/vmlinux.inlines which would also have
> > /sys/kernel/btf/vmlinux as base BTF. If no one needs vmlinux.vars on
> > the system, we won't need to waste memory on it. Seems more modular
> > and extensible.
> >
>
> Sounds good. So thinking about how this fits with
> CONFIG_DEBUG_INFO_BTF=m, perhaps the approach would be to use
> btf_vmlinux.ko for all such extensible /sys/kernel/btf/vmlinux.vars,
> vmlinux.inlines etc. Each of these is derived from .BTF.vars ,
> .BTF.inlines sections in btf_vmlinux.ko. These are optionally included
> via CONFIG_DEBUG_INFO_BTF_EXTRAS list. If CONFIG_DEBUG_INFO_BTF=y the
> core vmlinux section stays in vmlinux itself and the extras are
> delivered via btf_vmlinux.ko, but if CONFIG_DEBUG_INFO_BTF=m, the
> vmlinux .BTF section is delivered in btf_vmlinux.ko too.
>
> If this makes sense, I'll try and put together the
> CONFIG_DEBUG_INFO_BTF=m support first, and that will give us a
> btf_vmlinux.ko to work with for delivery of extras. Thanks!

I'd keep our options open as to whether btf_vmlinux.ko contains all
vmlinux BTFs (core BTF, inlines, variables) or we have a separate
module for some subsets. E.g., variables, while a useful thing,
probably won't be used all that frequently (i.e., only while debugging
with drgn), so co-locating it with vmlinux BTF itself might be a waste
in most cases.

But other than that makes sense.

>
> Alan
>
> >>
> >> Alan
> >>
> >>> pw-bot: cr
> >>>
> >>
>

Powered by blists - more mailing lists