Message-ID: <6c89777a-65dc-44da-83cd-005cb6c82430@oracle.com>
Date: Wed, 26 Feb 2025 14:20:10 +0000
From: Alan Maguire <alan.maguire@...cle.com>
To: Andrii Nakryiko <andrii.nakryiko@...il.com>
Cc: Alexei Starovoitov <alexei.starovoitov@...il.com>,
        Stephen Brennan <stephen.s.brennan@...cle.com>,
        Masahiro Yamada <masahiroy@...nel.org>,
        Andrii Nakryiko <andrii@...nel.org>,
        Nicolas Schier <nicolas@...sle.eu>, Kees Cook <kees@...nel.org>,
        KP Singh <kpsingh@...nel.org>, Martin KaFai Lau <martin.lau@...ux.dev>,
        Sami Tolvanen <samitolvanen@...gle.com>,
        Eduard Zingerman
 <eddyz87@...il.com>,
        linux-arch <linux-arch@...r.kernel.org>,
        Stanislav Fomichev <sdf@...ichev.me>,
        Kent Overstreet <kent.overstreet@...ux.dev>,
        Pasha Tatashin <pasha.tatashin@...een.com>,
        Jiri Olsa <jolsa@...nel.org>,
        John Fastabend <john.fastabend@...il.com>,
        Jann Horn <jannh@...gle.com>, Ard Biesheuvel <ardb@...nel.org>,
        Yonghong Song <yonghong.song@...ux.dev>, Hao Luo <haoluo@...gle.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Linux Kbuild mailing list <linux-kbuild@...r.kernel.org>,
        Daniel Borkmann <daniel@...earbox.net>, Arnd Bergmann <arnd@...db.de>,
        Nathan Chancellor <nathan@...nel.org>, linux-debuggers@...r.kernel.org,
        Alexei Starovoitov <ast@...nel.org>, Song Liu <song@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>, bpf <bpf@...r.kernel.org>
Subject: Re: [PATCH 2/2] btf: Add the option to include global variable types

On 25/02/2025 21:52, Andrii Nakryiko wrote:
> On Tue, Feb 25, 2025 at 2:02 AM Alan Maguire <alan.maguire@...cle.com> wrote:
>>
>> On 07/02/2025 23:50, Alexei Starovoitov wrote:
>>> On Thu, Feb 6, 2025 at 5:21 PM Stephen Brennan
>>> <stephen.s.brennan@...cle.com> wrote:
>>>> When the feature was implemented in pahole, my measurements indicated
>>>> that vmlinux BTF size increased by about 25.8%, and module BTF size
>>>> increased by 53.2%. Due to these increases, the feature is implemented
>>>> behind a new config option, allowing users sensitive to increased memory
>>>> usage to disable it.
>>>>
>>>
>>> ...
>>>> +config DEBUG_INFO_BTF_GLOBAL_VARS
>>>> +       bool "Generate BTF type information for all global variables"
>>>> +       default y
>>>> +       depends on DEBUG_INFO_BTF && PAHOLE_VERSION >= 128
>>>> +       help
>>>> +         Include type information for all global variables in the BTF. This
>>>> +         increases the size of the BTF information, which increases memory
>>>> +         usage at runtime. With global variable types available, runtime
>>>> +         debugging and tracers may be able to provide more detail.
>>>
>>> This is not a solution.
>>> Even if it's changed to 'default n' distros will enable it
>>> like they enable everything and will suffer a regression.
>>>
>>> We need to add a new module like vmlinux_btf.ko that will contain
>>> this additional BTF data. For global vars and everything else we might need.
>>>
>>
>> In this area, I've been exploring adding support for
>> CONFIG_DEBUG_INFO_BTF=m , so that the BTF info for vmlinux is delivered
>> via a module. From the consumer side, everything looks identical
>> (/sys/kernel/btf/vmlinux is there etc), it is just that the .BTF section
>> is delivered via btf_vmlinux.ko instead. The original need for this was
>> that embedded folks noted that because in the current situation BTF data
>> is in vmlinux, they cannot enable BTF because such small-footprint
>> systems do not support a large vmlinux binary. However they could
>> potentially use kernel BTF if it was delivered via a module. The other
>> nice thing about module delivery in the general case is we can make use
>> of module compression. In experiments I see a 5.8Mb vmlinux BTF reduce
>> to a 1.8Mb btf_vmlinux.ko.gz module on-disk.
>>
>> The challenge in delivering vmlinux BTF in a module is that on module
>> load during boot other modules expect vmlinux BTF to be there when
>> adding their own BTF to /sys/kernel/btf. And kfunc registration from
>> kernel and modules expects this also. So support for deferred BTF module
>> load/kfunc registration is required too. I've implemented the former and
>> now am working on the latter. Hope to have some RFC patches ready soon,
>> but it looks feasible at this point.
> 
> Lazy btf_vmlinux.ko loading when BTF is actually needed (i.e., when
> user reads /sys/kernel/btf/vmlinux for the first time; or when BPF
> program is validated and needs kernel BTF) would be great. Curious too
> see how all that fits together!
> 
>>
>> Assuming such an option was available to small-footprint systems, should
>> we consider adding global variables to core vmlinux BTF along with
>> per-cpu variables? Then vmlinux BTF extras could be used for some of the
>> additional optional representations like function site-specific data
>> (inlines etc)? Or are there other factors other than on-disk footprint
>> that we need to consider? Thanks!
> 
> I'd keep BTF for variables separate from "core" vmlinux BTF. We can
> have /sys/kernel/btf/vmlinux.vars, which would depend on
> /sys/kernel/btf/vmlinux as a base BTF. Separately, we could eventually
> have /sys/kernel/btf/vmlinux.inlines which would also have
> /sys/kernel/btf/vmlinux as base BTF. If no one needs vmlinux.vars on
> the system, we won't need to waste memory on it. Seems more modular
> and extensible.
>

Sounds good. So thinking about how this fits with
CONFIG_DEBUG_INFO_BTF=m, perhaps the approach would be to use
btf_vmlinux.ko for all such extensible /sys/kernel/btf/vmlinux.vars,
vmlinux.inlines etc. Each of these is derived from .BTF.vars ,
.BTF.inlines sections in btf_vmlinux.ko. These are optionally included
via CONFIG_DEBUG_INFO_BTF_EXTRAS list. If CONFIG_DEBUG_INFO_BTF=y the
core vmlinux section stays in vmlinux itself and the extras are
delivered via btf_vmlinux.ko, but if CONFIG_DEBUG_INFO_BTF=m, the
vmlinux .BTF section is delivered in btf_vmlinux.ko too.

If this makes sense, I'll try and put together the
CONFIG_DEBUG_INFO_BTF=m support first, and that will give us a
btf_vmlinux.ko to work with for delivery of extras. Thanks!

Alan

>>
>> Alan
>>
>>> pw-bot: cr
>>>
>>

Powered by blists - more mailing lists