Date:   Thu, 28 Oct 2021 07:14:28 +0530
From:   Kumar Kartikeya Dwivedi <memxor@...il.com>
To:     Andrii Nakryiko <andrii.nakryiko@...il.com>
Cc:     Jiri Olsa <jolsa@...hat.com>, Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Andrii Nakryiko <andrii@...nel.org>,
        Networking <netdev@...r.kernel.org>, bpf <bpf@...r.kernel.org>,
        Martin KaFai Lau <kafai@...com>,
        Song Liu <songliubraving@...com>, Yonghong Song <yhs@...com>,
        John Fastabend <john.fastabend@...il.com>,
        KP Singh <kpsingh@...omium.org>
Subject: Re: [RFC bpf-next 0/2] bpf: Fix BTF data for modules

On Wed, Oct 27, 2021 at 11:23:55PM IST, Andrii Nakryiko wrote:
> On Wed, Oct 27, 2021 at 1:53 AM Jiri Olsa <jolsa@...hat.com> wrote:
> >
> > On Tue, Oct 26, 2021 at 09:12:31PM -0700, Andrii Nakryiko wrote:
> > > On Tue, Oct 26, 2021 at 5:03 AM Jiri Olsa <jolsa@...hat.com> wrote:
> > > >
> > > > On Mon, Oct 25, 2021 at 09:54:48PM -0700, Andrii Nakryiko wrote:
> > > > > On Sat, Oct 23, 2021 at 5:05 AM Jiri Olsa <jolsa@...hat.com> wrote:
> > > > > >
> > > > > > hi,
> > > > > > I'm trying to enable BTF for kernel module in fedora,
> > > > > > and I'm getting big increase on modules sizes on s390x arch.
> > > > > >
> > > > > > Size of modules in total - kernel dir under /lib/modules/VER/
> > > > > > from kernel-core and kernel-module packages:
> > > > > >
> > > > > >                current   new
> > > > > >       aarch64      60M   76M
> > > > > >       ppc64le      53M   66M
> > > > > >       s390x        21M   41M
> > > > > >       x86_64       64M   79M
> > > > > >
> > > > > > The reason for higher increase on s390x was that dedup algorithm
> > > > > > did not detect some of the big kernel structs like 'struct module',
> > > > > > so they are duplicated in the kernel module BTF data. The s390x
> > > > > > has many small modules that increased significantly in size because
> > > > > > of that even after compression.
> > > > > >
> > > > > > First issues was that the '--btf_gen_floats' option is not passed
> > > > > > to pahole for kernel module BTF generation.
> > > > > >
> > > > > > The other problem is more tricky and is the reason why this patchset
> > > > > > is RFC ;-)
> > > > > >
> > > > > > The s390x compiler generates multiple definitions of the same struct
> > > > > > and dedup algorithm does not seem to handle this at the moment.
> > > > > >
> > > > > > I put the debuginfo and btf dump of the s390x pnet.ko module in here:
> > > > > >   http://people.redhat.com/~jolsa/kmodbtf/
> > > > > >
> > > > > > Please let me know if you'd like to see other info/files.
> > > > > >
> > > > >
> > > > > Hard to tell what's going on without vmlinux itself. Can you upload a
> > > > > corresponding kernel image with BTF in it?
> > > >
> > > > sure, uploaded
> > > >
> > >
> > > vmlinux.btfdump:
> > >
> > > [174] FLOAT 'float' size=4
> > > [175] FLOAT 'double' size=8
> > >
> > > VS
> > >
> > > pnet.btfdump:
> > >
> > > [89318] INT 'float' size=4 bits_offset=0 nr_bits=32 encoding=(none)
> > > [89319] INT 'double' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> >
> > ugh, that's with no fix applied, sry
> >
> > I applied the first patch and uploaded new files
> >
> > now when I compare the 'module' struct from vmlinux:
> >
> >         [885] STRUCT 'module' size=1280 vlen=70
> >
> > and same one from pnet.ko:
> >
> >         [89323] STRUCT 'module' size=1280 vlen=70
> >
> > they seem to completely match, all the fields
> > and yet it still appears in the kmod's BTF
> >
>
> Ok, now struct module is identical down to the types referenced from
> the fields, which means it should have been deduplicated completely.
> This will require a more time-consuming debugging, though, so I'll put
> it on my TODO list for now. If you get to this earlier, see where the
> equivalence check fails in btf_dedup (sprinkle debug outputs around to
> see what's going on).
>

Hello Andrii,

I think I'm seeing something similar when working on the conntrack patches [0],
I was looking to match whether the type in a PTR_TO_BTF_ID register is same as
struct nf_conn, but it seems that there can be two BTF IDs for the same struct
type.

When doing bpftool dump, I see:

 ; bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep nf_conn
 ...
[89224] STRUCT 'nf_conn' size=256 vlen=15
 ...

 ; bpftool btf dump file /sys/kernel/btf/nf_conntrack format raw | grep nf_conn
 ...
[103077] STRUCT 'nf_conn' size=256 vlen=15
[104988] STRUCT 'nf_conn' size=256 vlen=15
[106490] STRUCT 'nf_conn' size=256 vlen=15
[108187] STRUCT 'nf_conn' size=256 vlen=15
 ...

Inside the kernel, when trying to match both, register PTR_TO_BTF_ID refers to
the nf_conntrack BTF ID, while the BTF_ID_LIST resolves to the one in vmlinux,
this ends up making the job of matching the two struct types a bit difficult
(for now, I am thinking of going with btf_struct_ids_match). My original plan
was to compare the result of btf_types_by_id.

[0]: https://github.com/kkdwivedi/linux/commits/conntrack-bpf

> > thanks,
> > jirka
> >
> > >
> > >
> > > > jirka
> > > >
> > > > >
> > > > > > I found code in dedup that seems to handle such situation for arrays,
> > > > > > and added 'some' fix for structs. With that change I can no longer
> > > > > > see vmlinux's structs in kernel module BTF data, but I have no idea
> > > > > > if that breaks anything else.
> > > > > >
> > > > > > thoughts? thanks,
> > > > > > jirka
> > > > > >
> > > > > >
> > > > > > ---
> > > > > > Jiri Olsa (2):
> > > > > >       kbuild: Unify options for BTF generation for vmlinux and modules
> > > > > >       bpf: Add support to detect and dedup instances of same structs
> > > > > >
> > > > > >  Makefile                  |  3 +++
> > > > > >  scripts/Makefile.modfinal |  2 +-
> > > > > >  scripts/link-vmlinux.sh   | 11 +----------
> > > > > >  scripts/pahole-flags.sh   | 20 ++++++++++++++++++++
> > > > > >  tools/lib/bpf/btf.c       | 12 ++++++++++--
> > > > > >  5 files changed, 35 insertions(+), 13 deletions(-)
> > > > > >  create mode 100755 scripts/pahole-flags.sh
> > > > > >
> > > > >
> > > >
> > >
> >

--
Kartikeya

Powered by blists - more mailing lists