Message-ID: <Ygdjt0Qbki0tHG4k@syu-laptop.lan>
Date:   Sat, 12 Feb 2022 15:37:27 +0800
From:   Shung-Hsi Yu <shung-hsi.yu@...e.com>
To:     Toke Høiland-Jørgensen <toke@...hat.com>
CC:     Andrii Nakryiko <andrii.nakryiko@...il.com>,
        Michal Suchánek <msuchanek@...e.de>,
        Yonghong Song <yhs@...com>, bpf <bpf@...r.kernel.org>,
        Networking <netdev@...r.kernel.org>,
        Andrii Nakryiko <andrii@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Alexei Starovoitov <ast@...nel.org>
Subject: Re: BTF compatibility issue across builds

On Sat, Feb 12, 2022 at 12:58:51AM +0100, Toke Høiland-Jørgensen wrote:
> Andrii Nakryiko <andrii.nakryiko@...il.com> writes:
> 
> > On Fri, Feb 11, 2022 at 9:20 AM Toke Høiland-Jørgensen <toke@...hat.com> wrote:
> >>
> >> Andrii Nakryiko <andrii.nakryiko@...il.com> writes:
> >>
> >> > On Thu, Feb 10, 2022 at 2:01 AM Michal Suchánek <msuchanek@...e.de> wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >> On Mon, Jan 31, 2022 at 09:36:44AM -0800, Yonghong Song wrote:
> >> >> >
> >> >> >
> >> >> > On 1/27/22 7:10 AM, Shung-Hsi Yu wrote:
> >> >> > > Hi,
> >> >> > >
> >> >> > > We recently run into module load failure related to split BTF on openSUSE
> >> >> > > Tumbleweed[1], which I believe is something that may also happen on other
> >> >> > > rolling distros.
> >> >> > >
> >> >> > > The error looks like the follow (though failure is not limited to ipheth)
> >> >> > >
> >> >> > >      BPF:[103111] STRUCT BPF:size=152 vlen=2 BPF: BPF:Invalid name BPF:
> >> >> > >
> >> >> > >      failed to validate module [ipheth] BTF: -22
> >> >> > >
> >> >> > > The error comes down to trying to load BTF of *kernel modules from a
> >> >> > > different build* than the runtime kernel (but the source is the same), where
> >> >> > > the base BTF of the two build is different.
> >> >> > >
> >> >> > > While it may be too far stretched to call this a bug, solving this might
> >> >> > > make BTF adoption easier. I'd natively think that we could further split
> >> >> > > base BTF into two part to avoid this issue, where .BTF only contain exported
> >> >> > > types, and the other (still residing in vmlinux) holds the unexported types.
> >> >> >
> >> >> > What is the exported types? The types used by export symbols?
> >> >> > This for sure will increase btf handling complexity.
> >> >>
> >> >> And it will not actually help.
> >> >>
> >> >> We have modversion ABI which checks the checksum of the symbols that the
> >> >> module imports and fails the load if the checksum for these symbols does
> >> >> not match. It's not concerned with symbols not exported, it's not
> >> >> concerned with symbols not used by the module. This is something that is
> >> >> sustainable across kernel rebuilds with minor fixes/features and what
> >> >> distributions watch for.
> >> >>
> >> >> Now with BTF the situation is vastly different. There are at least three
> >> >> bugs:
> >> >>
> >> >>  - The BTF check is global for all symbols, not for the symbols the
> >> >>    module uses. This is not sustainable. Given the BTF is supposed to
> >> >>    allow linking BPF programs that were built in completely different
> >> >>    environment with the kernel it is completely within the scope of BTF
> >> >>    to solve this problem, it's just neglected.
> >> >
> >> > You refer to BTF use in CO-RE with the latter. It's just one
> >> > application of BTF and it doesn't follow that you can do the same with
> >> > module BTF. It's not a neglect, it's a very big technical difficulty.
> >> >
> >> > Each module's BTFs are designed as logical extensions of vmlinux BTF.
> >> > And each module BTF is independent and isolated from other modules
> >> > extension of the same vmlinux BTF. The way that BTF format is
> >> > designed, any tiny difference in vmlinux BTF effectively invalidates
> >> > all modules' BTFs and they have to be rebuilt.
> >> >
> >> > Imagine that only one BTF type is added to vmlinux BTF. Last BTF type
> >> > ID in vmlinux BTF is shifted from, say, 1000 to 1001. While previously
> >> > every module's BTF type ID started with 1001, now they all have to
> >> > start with 1002 and be shifted by 1.
> >> >
> >> > Now let's say that the order of two BTF types in vmlinux BTF is
> >> > changed, say type 10 becomes type 20 and type 20 becomes type 10 (just
> >> > because of slight difference in DWARF, for instance). Any type
> >> > reference to 10 or 20 in any module BTF has to be renumbered now.
> >> >
> >> > Another one, let's say we add a new string to vmlinux BTF string
> >> > section somewhere at the beginning, say "abc" at offset 100. Any
> >> > string offset after 100 now has to be shifted *both* in vmlinux BTF
> >> > and all module BTFs. And also any string reference in module BTFs have
> >> > to be adjusted as well because now each module's BTF's logical string
> >> > offset is starting at 4 logical bytes higher (due to "abc\0" being
> >> > added and shifting everything right).
> >> >
> >> > As you can see, any tiny change in vmlinux BTF, no matter where,
> >> > beginning, middle, or end, causes massive changes in type IDs and
> >> > offsets everywhere. It's impractical to do any local adjustments, it's
> >> > much simpler and more reliable to completely regenerate BTF
> >> > completely.
> >>
> >> This seems incredibly brittle, though? IIUC this means that if you want
> >> BTF in your modules you *must* have not only the kernel headers of the
> >> kernel it's going to run on, but the full BTF information for the exact
> >
> > From BTF perspective, only vmlinux BTF. Having exact kernel headers
> > would minimize type information duplication.
> 
> Right, I meant you'd need the kernel headers to compile the module, and
> the vmlinux BTF to build the module BTF info.
> 
> >> kernel image you're going to load that module on? How is that supposed
> >> to work for any kind of environment where everything is not built
> >> together? Third-party modules for distribution kernels is the obvious
> >> example that comes to mind here, but as this thread shows, they don't
> >> necessarily even have to be third party...
> >>
> >> How would you go about "completely regenerating BTF" in practice for a
> >> third-party module, say?
> >
> > Great questions. I was kind of hoping you'll have some suggestions as
> > well, though. Not just complaints.
> 
> Well, I kinda took your "not really a bug either" comment to mean you
> weren't really open to changing the current behaviour. But if that was a
> misunderstanding on my part, I do have one thought:
> 
> The "partial BTF" thing in the modules is done to save space, right?
> I.e., in principle there would be nothing preventing a module from
> including a full (self-contained) set of BTF in its .ko when it is
> compiled? Because if so, we could allow that as an optional mode that
> can be enabled if you don't mind taking the size hit (any idea how large
> that usually is, BTW?).

This seems quite nice IMO as no change need to be made on the generation
side of existing BTF tooling. I test it out on openSUSE Tumbleweed 5.16.5
kernel modules, and for the sake of completeness, includes both the case
where BTF is stripped and using a pre-trained zstd dictionary as well.

Uncompressed, no BTF                             362MiB -27%
Uncompressed, parital BTF                        499MiB +0%
Uncompressed, self-contained BTF                1026MiB +105%

Zstd compressed, no BTF                           95MiB -35%
Zstd compressed, partial BTF                     147MiB +0%
Zstd compressed, self-contained BTF              361MiB +145%
Zstd compressed (trained), self-contained BTF    299MiB +103%

So we'd expect quite a bit of hit as the size of kernel module would double.

For servers and workstation environment an additional ~200MiB of disk space
seems like tolerable trade-off if it can get third-party kernel module to
work. But I cannot speak for other kind of use cases.

> And then we could teach 'modprobe' to do a fresh deduplication of this
> full BTF set against the vmlinux BTF before loading such a module into the
> kernel.
> 
> Or am I missing some reason why that wouldn't work?

One minor problem would be this is essentially introducing a new kernel
module BTF format that uses exactly the same header.

Ever since the introduction of split BTF, we're reusing btf_header but
acting as if there's an extra hidden flag indicating whether the BTF is
self-contained or partial. So far we could implicitly guess the value of the
flag since BTF in vmlinux is always self-contained and BTF in kernel module
is always partial; but if self-contained BTF on kernel module is introduced
this will no longer be the case.

Not sure if it'd be a issue in practice though, as we could go through the
type info and see whether there's any type ID that is too large and cannot
be found.

Powered by blists - more mailing lists