Date:   Wed, 4 Dec 2019 20:26:38 -0800
From:   Jakub Kicinski <jakub.kicinski@...ronome.com>
To:     Alexei Starovoitov <alexei.starovoitov@...il.com>
Cc:     Andrii Nakryiko <andrii.nakryiko@...il.com>,
        Toke Høiland-Jørgensen 
        <toke@...hat.com>, Jiri Olsa <jolsa@...nel.org>,
        Arnaldo Carvalho de Melo <acme@...nel.org>,
        lkml <linux-kernel@...r.kernel.org>,
        Networking <netdev@...r.kernel.org>, bpf <bpf@...r.kernel.org>,
        Ingo Molnar <mingo@...nel.org>,
        Namhyung Kim <namhyung@...nel.org>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Peter Zijlstra <a.p.zijlstra@...llo.nl>,
        Michael Petlan <mpetlan@...hat.com>,
        Jesper Dangaard Brouer <brouer@...hat.com>,
        Daniel Borkmann <daniel@...earbox.net>,
        Martin KaFai Lau <kafai@...com>,
        Song Liu <songliubraving@...com>, Yonghong Song <yhs@...com>,
        Andrii Nakryiko <andriin@...com>,
        Quentin Monnet <quentin.monnet@...ronome.com>
Subject: Re: [PATCHv4 0/6] perf/bpftool: Allow to link libbpf dynamically

On Wed, 4 Dec 2019 19:17:20 -0800, Alexei Starovoitov wrote:
> On Wed, Dec 04, 2019 at 06:10:28PM -0800, Jakub Kicinski wrote:
> > On Wed, 4 Dec 2019 17:09:32 -0800, Alexei Starovoitov wrote:  
> > > On Wed, Dec 04, 2019 at 04:23:48PM -0800, Jakub Kicinski wrote:  
> > > > On Wed, 4 Dec 2019 15:39:49 -0800, Alexei Starovoitov wrote:    
> > > > > > Agreed. Having libbpf on GH is definitely useful today, but one can hope
> > > > > > a day will come when distroes will get up to speed on packaging libbpf,
> > > > > > and perhaps we can retire it? Maybe 2, 3 years from now? Putting
> > > > > > bpftool in the same boat is just more baggage.      
> > > > > 
> > > > > Distros should be packaging libbpf and bpftool from single repo on github.
> > > > > Kernel tree is for packaging kernel.    
> > > > 
> > > > Okay, single repo on GitHub:
> > > > 
> > > > https://github.com/torvalds/linux    
> > > 
> > > and how will you git submodule only libbpf part of kernel github into bcc
> > > and other projects?  
> > 
> > Why does bcc have to submodule libbpf? Is it in a "special
> > relationship" with libbpf as well? 
> > 
> > dnf/apt install libbpf
> > 
> > Or rather:
> > 
> > dnf/apt install bcc
> > 
> > since BCC's user doesn't care about dependencies. The day distroes
> > started packaging libbpf and bpftool the game has changed.  
> 
> have you ever built bcc ? or bpftrace?
> I'm not sure how to answer such 'suggestion'.

Perhaps someone else has more patience to explain it - why bcc can't
just use binary libbpf distribution (static lib + headers) and link
against it like it links against other libraries?

> > Please accept iproute2 as an example of a user space toolset closely
> > related to the kernel. If kernel release model and process made no
> > sense in user space, why do iproute2s developers continue to follow it
> > for years?   
> 
> imo iproute2 is an example how things should not be run.
> But that's a very different topic.

Please explain, the topic is how to maintain user space closely related
to the kernel.

Share with us what you dislike about iproute2 so we can fix it. Instead
of adding parts of it to bpftool and then pretending that the API added
to libbpf to facilitate that duplication is some internal bpftool-only
magic which then prevents us from dynamic linking..... 😠

> > > Packaging is different.  
> > 
> > There are mostly disadvantages, but the process should be well known.
> > perf has been packaged for years.  
> 
> perf was initially seen as something that should match kernel one to one.
> yet it diverged over years. I think it's a counter example.
> 
> > What do you mean? I've sure as hell sent patches to net with Fixes tags  
> 
> which was complete waste of time for people who were sending these
> patches, for maintainers who applied them and for all stables folks
> who carried them into kernel stable releases.
> Not a single libbpf build was made out of those sources.

Because libbpf just now entered the distroes, and you suggested the
distroes use the GH repo, so sure now it's wasted work.

IIRC there were bpftool crash fixes which landed in Fedora via stable.

> > > Even coding style is different.  
> > 
> > Is it? You mean the damn underscores people are making fun of? :/  
> 
> Are you trolling? Do you understand why __ is there?

Not the point. Tell me how the coding style is different. The
underscores is the only thing I could think of that's not common 
in the kernel.

> > libbpf doesn't have a roadmap either,   
> 
> I think you're contrasting that with kernel and saying
> that kernel has a roadmap ? What is kernel roadmap?

Kernel road map is the same as libbpf's road map.

> > it's not really a full-on project
> > on its own. What's 0.1.0 gonna be?  
> 
> whenever this bpf community decides to call it 0.1.0.
>
> > Besides stuff lands in libbpf before it hits a major kernel release.
> > So how are you gonna make libbpf releases independently from kernel
> > ones? What if a feature gets a last minute revert in the kernel and it's
> > in libbpf's ABI?  
> 
> You mean when kernel gets new feature, then libbpf gets new feature, then
> libbpf is released, but then kernel feature is reverted? Obviously we should
> avoid making a libbpf release that relies on kernel features that didn't reach
> the mainline. Yet there could be plenty of reasons why making libbpf release in
> the middle of kernel development cycle makes perfect sense.

But master of libbpf must have all features to test the kernel with,
right? So how do we branch of a release in the middle? That's only
possible if kernel cycle happens to not have had any features that
required libbpf yet?

Or are you thinking 3 tier branching where we'd branch off libbpf
release, say 2.6.0 that corresponds to kernel X, but it wouldn't be a
stable-only release, and we can still backport features added in kernel
X + 1 cycle, features which don't require kernel support, and release
libbpf 2.7.0?

Could work but it'd get tricky, cause if we want to break ABI we'd
actually need 4 tiers. ABI compat, kernel version, feature version,
stable version.

> Also reaching Linus's tree in rc1 is also not a guarantee of non-revert. Yet we
> release libbpf around rc1 because everyone expects bug-fixes after rc1. 

I consider current process to be broken. Hopefully we can improve it.

> So it's an exception that solidifies the rule.
>
> > > libbpf has to run on all kernels. Newer and older. How do you support
> > > that if libbpf is tied with the kernel?  
> > 
> > Say I have built N kernels UM or for a VM, and we have some test
> > suite: I pull libbpf, build it, run its tests. The only difference
> > between in tree and out of tree is that "pull libbpf" means pulling
> > smaller or larger repo. Doesn't matter that match, it's a low --depth
> > local clone.  
> 
> The expected CI is:
> 1. pull-req proposed.
> 2. CI picks it up, builds, run tests.
> 3. humans see results and land or reject pull-req.
> Now try to think through how CI on top of full kernel tree will
> be able to pick just the right commits to start build/test cycle.
> Is it going to cherry-pick from patchworks? That would be awesome.
> Yet intel 0bot results show that it's easier said than done.
> I'm not saying it's not possible. Just complex.
> If you have cycles to integrate *-next into kernelci.org, please go ahead.

Yes, it is very complex, I know. I've been hacking on something along
those lines for the last few weeks. Hopefully I'll have results at some
point..

First stab is just doing build testing, checkpatch, verify tags etc.
Uploading to patchwork, and sending an email if there were failures.

Even that's not easy as a weekend/evening task :( And it requires a lot
of manual inspection upfront before it's unleashed on the ML, because it
will catch a lot of stupid little stuff and a lot of people will get
grumpy.

We need to modernize the process across the board. I don't think having
zombie read-only repos on GitHub will give contributors confidence so
it's not a step in right direction. We should start from the hard
problem, that is the CI itself.

The problem of correlating user space and kernel patches will have to 
be solved for netdev, because netdev tests depend on iproute2.

Powered by blists - more mailing lists