Date:   Tue, 12 Mar 2019 12:53:35 -0400
From:   Kris Van Hees <kris.van.hees@...cle.com>
To:     Brendan Gregg <bgregg@...flix.com>
Cc:     Kris Van Hees <kris.van.hees@...cle.com>,
        Alexei Starovoitov <alexei.starovoitov@...il.com>,
        netdev@...r.kernel.org, bpf@...r.kernel.org,
        Daniel Borkmann <daniel@...earbox.net>
Subject: Re: [PATCH 0/2] bpf: context casting for tail call and gtrace prog
 type

On Mon, Mar 11, 2019 at 11:03:10PM -0700, Brendan Gregg wrote:
> On Mon, Mar 11, 2019 at 8:24 PM Kris Van Hees <kris.van.hees@...cle.com> wrote:
> >
> > On Mon, Mar 11, 2019 at 06:29:55PM -0700, Brendan Gregg wrote:
> > > On Mon, Mar 11, 2019 at 7:21 AM Kris Van Hees <kris.van.hees@...cle.com> wrote:
> > > >
> > > > On Thu, Mar 07, 2019 at 01:30:37PM -0800, Alexei Starovoitov wrote:
> > > > > On Tue, Mar 05, 2019 at 09:03:57PM -0500, Kris Van Hees wrote:
> > > [...]
> > > > > > But being able to do things like this without
> > > > > > needing to touch the context of any other BPF program type is a great benefit
> > > > > > to offer tracing tools, as far as I see it.
> > > > >
> > > > > I still don't understand what you're referring to by 'things like this'
> > > > > that somehow will be possible in the future, but not possible today.
> > > > > Could you please give concrete example?
> > > >
> > > > My apologies for not being clear.  I am referring to the features of the
> > > > gtrace context in terms of containing task information, and output buffers
> > > > to be used in BPF programs triggered from various probe sources (kprobe,
> > > > tracepoints, ...)  I would not want to suggest making changes to all the
> > > > different program contexts in order to support tracing needs because that
> > > > would be wrong.  Doing it in a central place makes it a lot easier to maintain
> > > > without impacting other program types, etc.
> > > >
> > > > Of course, yes, bpf_probe_read() and bpf_perf_event_output() can be used
> > > > to implement a lot of what existing tracing tools like DTrace can do, if you
> > > > write them based on that.  One limitations I am obviously working with is
> > > > that DTrace already exists and has existed for a long time.  And while it is
> > > > 100% available as open source, it involves a pretty involved set of patches to
> > > > be applied to the kernel to be able to use it which is just not ideal.  Hence
> > > > the goal to make it available by re-using as much of the existing features in
> > > > Linux as possible, while still maintaining the same level of functionality in
> > > > DTrace.  That means we need to fill the gaps - and from where I am sitting,
> > > > the ways to do that might as well be of use to others (if they want to).
> > > >
> > > > If phrasing things in the context of DTrace would make the conversation easier
> > > > I certainly don;t mind doing that, but I really don't want to limit my patches
> > > > to supporting just DTrace (even if right now it might be the only tracer using
> > > > it).
> > >
> > > As a concrete example, can you point to one of my own published DTrace
> > > tools that BPF can't do? These were created to solve many real
> > > production issues, and make good use cases. I've been porting them
> > > over to BPF (bcc and bpftrace) without too much problem, and I can't
> > > think of a single one that I couldn't port over today.
> >
> > I am unclear how pointing at one of your published DTrace tools would
> > contribute to this discussion.  Surely the scope of use cases is not limited
> > to the DTrace scripts you published?
> >
> > Either way, one of the features that I make use of is speculative tracing.
> 
> The subject was a concrete example. I don't think I used speculative
> tracing in any of my scripts. Can you pick a better example of
> something BPF can't do?

Well, then speculative tracing is a good example of something that cannot be
done right now.  The specopen.d script that is part of the DTrace documentation
and is also featured in the test suite makes for a concrete example.  That
script has been used as a kind of template script in situations where we had
to analyze code paths associated with a specific subset of conditions that
could not be known beforehand.

Are there other ways to accomplish the same thing?  Sure.  But I don't think
that is really the point.  There are often multiple tools that can do the same
thing (or close to the same thing), and people have the option to choose one
or the other.  DTrace is one of those tools, as is systemtap, bpftrace, perf,
and other tools.

> > And yes, even that could be handled with some ugly workarounds but my intent
> > is to implement things in a more clean way rather than depending on a bunch
> > of workarounds to make it somewhat work.
> >
> > > There's a few minor things that I'm currently doing workarounds for,
> > > like ppid, but that should be satisfied with a few more helpers. And
> > > if it's really niche, then BTF sounds like a good solution.
> >
> > Of course, we can always add more helpers to get to information that is
> > needed, but that is hardly a practical solution in the long run, and at
> > Plumbers 2019 it was already indicated that just adding helpers to get to
> > more information about tasks is not the route people want to take.
> >
> > > If your ultimate goal is to have a command called "dtrace" that runs D
> > > programs, to support your existing users, then I'd add a lex/yacc pair
> > > to bpftrace and have it emit a dtrace binary.
> >
> > My goal is not to have a command called dtrace that somehow simply provides
> > some form of support for dtrace scripts in some legacy support model.  My
> > goal is to make DTrace available on Linux based on existing kernel features
> > (and contributing extra features where needed, in a collaborative manner).
> 
> If bpftrace builds a dtrace binary that runs D code, then you just did
> make DTrace available on Linux.
> 
> And without changing the kernel.

If bpftrace could do everything that DTrace does, in a way that is 100%
transparent to the user, and without requiring extra dependencies like e.g.
software development packages (llvm, etc) to be installed on the system where
it will be used in production, I think we wouldn't be having this conversation.

Anyway, I feel we're getting off track here...  the discussion is not about
whether bpftrace can do what dtrace can do, or any other tool for that matter.
There is a need for DTrace on Linux and I am working on making that possible
in a way where DTrace is one among multiple tracing tools, and by leveraging
existing features in the kernel to the extent possible rather than putting it
in as an almost self-contained monolith.  We already made those patches 
available a while ago - but that isn't the right way to go about this in the
long run and it isn't a benefit to the overall community because there isn't
any good way other tools can make use of it.  As part of doing the work to
contribute DTrace as a tool within the Linux tracing framework I identify
areas where there are gaps in terms of what we need, and I contribute patches
that fill those gaps in a way that makes it possible for others to make use of
those features as well.

	Kris

Powered by blists - more mailing lists