Message-ID: <CAEf4BzYJCHB-oYqFqJTfHU4D795ewgkndQtR1Po5H521fH0oMg@mail.gmail.com>
Date:   Fri, 11 Feb 2022 14:20:27 -0800
From:   Andrii Nakryiko <andrii.nakryiko@...il.com>
To:     Toke Høiland-Jørgensen <toke@...hat.com>
Cc:     Michal Suchánek <msuchanek@...e.de>,
        Yonghong Song <yhs@...com>,
        Shung-Hsi Yu <shung-hsi.yu@...e.com>,
        bpf <bpf@...r.kernel.org>, Networking <netdev@...r.kernel.org>,
        Andrii Nakryiko <andrii@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Alexei Starovoitov <ast@...nel.org>
Subject: Re: BTF compatibility issue across builds

On Fri, Feb 11, 2022 at 9:20 AM Toke Høiland-Jørgensen <toke@...hat.com> wrote:
>
> Andrii Nakryiko <andrii.nakryiko@...il.com> writes:
>
> > On Thu, Feb 10, 2022 at 2:01 AM Michal Suchánek <msuchanek@...e.de> wrote:
> >>
> >> Hello,
> >>
> >> On Mon, Jan 31, 2022 at 09:36:44AM -0800, Yonghong Song wrote:
> >> >
> >> >
> >> > On 1/27/22 7:10 AM, Shung-Hsi Yu wrote:
> >> > > Hi,
> >> > >
> >> > > We recently run into module load failure related to split BTF on openSUSE
> >> > > Tumbleweed[1], which I believe is something that may also happen on other
> >> > > rolling distros.
> >> > >
> >> > > The error looks like the follow (though failure is not limited to ipheth)
> >> > >
> >> > >      BPF:[103111] STRUCT BPF:size=152 vlen=2 BPF: BPF:Invalid name BPF:
> >> > >
> >> > >      failed to validate module [ipheth] BTF: -22
> >> > >
> >> > > The error comes down to trying to load BTF of *kernel modules from a
> >> > > different build* than the runtime kernel (but the source is the same), where
> >> > > the base BTF of the two build is different.
> >> > >
> >> > > While it may be too far stretched to call this a bug, solving this might
> >> > > make BTF adoption easier. I'd natively think that we could further split
> >> > > base BTF into two part to avoid this issue, where .BTF only contain exported
> >> > > types, and the other (still residing in vmlinux) holds the unexported types.
> >> >
> >> > What is the exported types? The types used by export symbols?
> >> > This for sure will increase btf handling complexity.
> >>
> >> And it will not actually help.
> >>
> >> We have modversion ABI which checks the checksum of the symbols that the
> >> module imports and fails the load if the checksum for these symbols does
> >> not match. It's not concerned with symbols not exported, it's not
> >> concerned with symbols not used by the module. This is something that is
> >> sustainable across kernel rebuilds with minor fixes/features and what
> >> distributions watch for.
> >>
> >> Now with BTF the situation is vastly different. There are at least three
> >> bugs:
> >>
> >>  - The BTF check is global for all symbols, not for the symbols the
> >>    module uses. This is not sustainable. Given the BTF is supposed to
> >>    allow linking BPF programs that were built in completely different
> >>    environment with the kernel it is completely within the scope of BTF
> >>    to solve this problem, it's just neglected.
> >
> > You refer to BTF use in CO-RE with the latter. It's just one
> > application of BTF and it doesn't follow that you can do the same with
> > module BTF. It's not a neglect, it's a very big technical difficulty.
> >
> > Each module's BTFs are designed as logical extensions of vmlinux BTF.
> > And each module BTF is independent and isolated from other modules
> > extension of the same vmlinux BTF. The way that BTF format is
> > designed, any tiny difference in vmlinux BTF effectively invalidates
> > all modules' BTFs and they have to be rebuilt.
> >
> > Imagine that only one BTF type is added to vmlinux BTF. Last BTF type
> > ID in vmlinux BTF is shifted from, say, 1000 to 1001. While previously
> > every module's BTF type ID started with 1001, now they all have to
> > start with 1002 and be shifted by 1.
> >
> > Now let's say that the order of two BTF types in vmlinux BTF is
> > changed, say type 10 becomes type 20 and type 20 becomes type 10 (just
> > because of slight difference in DWARF, for instance). Any type
> > reference to 10 or 20 in any module BTF has to be renumbered now.
> >
> > Another one, let's say we add a new string to vmlinux BTF string
> > section somewhere at the beginning, say "abc" at offset 100. Any
> > string offset after 100 now has to be shifted *both* in vmlinux BTF
> > and all module BTFs. And also any string reference in module BTFs have
> > to be adjusted as well because now each module's BTF's logical string
> > offset is starting at 4 logical bytes higher (due to "abc\0" being
> > added and shifting everything right).
> >
> > As you can see, any tiny change in vmlinux BTF, no matter where,
> > beginning, middle, or end, causes massive changes in type IDs and
> > offsets everywhere. It's impractical to do any local adjustments, it's
> > much simpler and more reliable to completely regenerate BTF
> > completely.
>
> This seems incredibly brittle, though? IIUC this means that if you want
> BTF in your modules you *must* have not only the kernel headers of the
> kernel it's going to run on, but the full BTF information for the exact

>From BTF perspective, only vmlinux BTF. Having exact kernel headers
would minimize type information duplication.

> kernel image you're going to load that module on? How is that supposed
> to work for any kind of environment where everything is not built
> together? Third-party modules for distribution kernels is the obvious
> example that comes to mind here, but as this thread shows, they don't
> necessarily even have to be third party...
>
> How would you go about "completely regenerating BTF" in practice for a
> third-party module, say?

Great questions. I was kind of hoping you'll have some suggestions as
well, though. Not just complaints.

>
> -Toke
>

Powered by blists - more mailing lists