Message-ID: <20190316014935.GP11634@oracle.com>
Date:   Fri, 15 Mar 2019 21:49:35 -0400
From:   Kris Van Hees <kris.van.hees@...cle.com>
To:     Brendan Gregg <bgregg@...flix.com>
Cc:     Kris Van Hees <kris.van.hees@...cle.com>,
        Alexei Starovoitov <alexei.starovoitov@...il.com>,
        netdev@...r.kernel.org, bpf@...r.kernel.org,
        Daniel Borkmann <daniel@...earbox.net>
Subject: Re: [PATCH 0/2] bpf: context casting for tail call and gtrace prog
 type

On Wed, Mar 13, 2019 at 09:30:02AM -0700, Brendan Gregg wrote:
> On Tue, Mar 12, 2019 at 9:54 AM Kris Van Hees <kris.van.hees@...cle.com> wrote:
> >
> > On Mon, Mar 11, 2019 at 11:03:10PM -0700, Brendan Gregg wrote:
> > > On Mon, Mar 11, 2019 at 8:24 PM Kris Van Hees <kris.van.hees@...cle.com> wrote:
> > > >
> > > > On Mon, Mar 11, 2019 at 06:29:55PM -0700, Brendan Gregg wrote:
> > > > > On Mon, Mar 11, 2019 at 7:21 AM Kris Van Hees <kris.van.hees@...cle.com> wrote:
> > > > > >
> > > > > > On Thu, Mar 07, 2019 at 01:30:37PM -0800, Alexei Starovoitov wrote:
> > > > > > > On Tue, Mar 05, 2019 at 09:03:57PM -0500, Kris Van Hees wrote:
> > > > > [...]
> > > > > > > > But being able to do things like this without
> > > > > > > > needing to touch the context of any other BPF program type is a great benefit
> > > > > > > > to offer tracing tools, as far as I see it.
> > > > > > >
> > > > > > > I still don't understand what you're referring to by 'things like this'
> > > > > > > that somehow will be possible in the future, but not possible today.
> > > > > > > Could you please give concrete example?
> > > > > >
> > > > > > My apologies for not being clear.  I am referring to the features of the
> > > > > > gtrace context in terms of containing task information, and output buffers
> > > > > > to be used in BPF programs triggered from various probe sources (kprobe,
> > > > > > tracepoints, ...)  I would not want to suggest making changes to all the
> > > > > > different program contexts in order to support tracing needs because that
> > > > > > would be wrong.  Doing it in a central place makes it a lot easier to maintain
> > > > > > without impacting other program types, etc.
> > > > > >
> > > > > > Of course, yes, bpf_probe_read() and bpf_perf_event_output() can be used
> > > > > > to implement a lot of what existing tracing tools like DTrace can do, if you
> > > > > > write them based on that.  One limitations I am obviously working with is
> > > > > > that DTrace already exists and has existed for a long time.  And while it is
> > > > > > 100% available as open source, it involves a pretty involved set of patches to
> > > > > > be applied to the kernel to be able to use it which is just not ideal.  Hence
> > > > > > the goal to make it available by re-using as much of the existing features in
> > > > > > Linux as possible, while still maintaining the same level of functionality in
> > > > > > DTrace.  That means we need to fill the gaps - and from where I am sitting,
> > > > > > the ways to do that might as well be of use to others (if they want to).
> > > > > >
> > > > > > If phrasing things in the context of DTrace would make the conversation easier
> > > > > > I certainly don;t mind doing that, but I really don't want to limit my patches
> > > > > > to supporting just DTrace (even if right now it might be the only tracer using
> > > > > > it).
> > > > >
> > > > > As a concrete example, can you point to one of my own published DTrace
> > > > > tools that BPF can't do? These were created to solve many real
> > > > > production issues, and make good use cases. I've been porting them
> > > > > over to BPF (bcc and bpftrace) without too much problem, and I can't
> > > > > think of a single one that I couldn't port over today.
> > > >
> > > > I am unclear how pointing at one of your published DTrace tools would
> > > > contribute to this discussion.  Surely the scope of use cases is not limited
> > > > to the DTrace scripts you published?
> > > >
> > > > Either way, one of the features that I make use of is speculative tracing.
> > >
> > > The subject was a concrete example. I don't think I used speculative
> > > tracing in any of my scripts. Can you pick a better example of
> > > something BPF can't do?
> >
> > Well, then speculative tracing is a good example of something that cannot be
> > done right now.  The specopen.d script that is part of the DTrace documentation
> > and is also featured in the test suite makes for a concrete example.  That
> > script has been used as a kind of template script in situations where we had
> > to analyze code paths associated with a specific subset of conditions that
> > could not be known beforehand.
> 
> I used spec tracing zero times in any of my published scripts from
> fifteen years of using DTrace, and zero times in my 1152-page DTrace
> book. Might I find a use one day? Maybe, just as you have. But I
> wouldn't call it a great example of necessary functionality!

Perhaps not for you, but it is for me.  There are other much simpler things
that are currently missing, such as being able to obtain euid/egid, ppid (as
you mentioned already), ...  There are workaround possible or helpers can be
added, but I would think that making them available by means of a mechanism
that is very much native to BPF (obtaining data from context, as is done for
networking packets) makes a lot more sense.

> Besides, I bet we could implement spec tracing using existing BPF maps
> if we wanted -- I just haven't had that need.
> 
> >
> > Are there other ways to accomplish the same thing?  Sure.  But I don't think
> > that is really the point.  There are often multiple tools that can do the same
> > thing (or close to the same thing), and people have the option to choose one
> > or the other.  DTrace is one of those tools, as is systemtap, bpftrace, perf,
> > and other tools.
> >
> > > > And yes, even that could be handled with some ugly workarounds but my intent
> > > > is to implement things in a more clean way rather than depending on a bunch
> > > > of workarounds to make it somewhat work.
> > > >
> > > > > There's a few minor things that I'm currently doing workarounds for,
> > > > > like ppid, but that should be satisfied with a few more helpers. And
> > > > > if it's really niche, then BTF sounds like a good solution.
> > > >
> > > > Of course, we can always add more helpers to get to information that is
> > > > needed, but that is hardly a practical solution in the long run, and at
> > > > Plumbers 2019 it was already indicated that just adding helpers to get to
> > > > more information about tasks is not the route people want to take.
> > > >
> > > > > If your ultimate goal is to have a command called "dtrace" that runs D
> > > > > programs, to support your existing users, then I'd add a lex/yacc pair
> > > > > to bpftrace and have it emit a dtrace binary.
> > > >
> > > > My goal is not to have a command called dtrace that somehow simply provides
> > > > some form of support for dtrace scripts in some legacy support model.  My
> > > > goal is to make DTrace available on Linux based on existing kernel features
> > > > (and contributing extra features where needed, in a collaborative manner).
> > >
> > > If bpftrace builds a dtrace binary that runs D code, then you just did
> > > make DTrace available on Linux.
> > >
> > > And without changing the kernel.
> >
> > If bpftrace could do everything that DTrace does, in a way that is 100%
> > transparent to the user, and without requiring extra dependencies like e.g.
> > software development packages (llvm, etc) to be installed on the system where
> > it will be used in production, I think we wouldn't be having this conversation.
> 
> bpftrace is solving that need, and it was built from the ground up to
> work with eBPF. It's going to provide 100% of useful functionality.
> Getting it available out of the box is both a user-space and distro
> problem.
> 
> >
> > Anyway, I feel we're getting off track here...  the discussion is not about
> > whether bpftrace can do what dtrace can do, or any other tool for that matter.
> > There is a need for DTrace on Linux and I am working on making that possible
> 
> I wanted DTrace on Linux five years ago. Now I don't. We have eBPF,
> which does more.

I respect your personal feeling on this, but there are people who use DTrace
on Linux, and who want DTrace to keep improving and be more based on existing
core kernel features.  You say that eBPF does more, but clearly it does not
since there are features DTrace has that eBPF currently does not provide (as
basic as ppid of a task).

But really, where does that even matter?  If eBPF ends up being able to do
more than the current DTrace can, then a DTrace that is making use of eBPF
will obviously be able to do more as well.  That's a great benefit in my
opinion.

> > in a way where DTrace is one among multiple tracing tools, and by leveraging
> 
> Multiple tracing tools? This will fracture the user and engineering
> communities. People have complained enough about Ftrace and perf and
> BPF -- we don't need a fourth tracing framework that doesn't solve any
> new problems.

People already use multiple tools and those tools continue being developed.
And I am not working on a tracing framework alongside whatever already exists,
but rather on integrating DTrace with the existing tracing features in the
kernel.  Even if you feel you don't need another tracing tool, fact is that
DTrace already exists on Linux even though the patches are not in the upstream
kernel.  People use it and like it.  Working to make it available without the
rather invasive kernel changes and instead building on the existing kernel
features means we also can contribute new features that expand the capabilities
of eBPF (within its excellent safety mechanisms) and hopefully other tracing
features.  What is wrong with that?

There will always be new features added to tools, there will always be new
tools that emerge.  That's the nature of software development.  And people
have a tendency to work with tools they like.  Should everyone stop writing
C programs because C++ exists?

> We can add a legacy interface to bpftrace for running D programs, to
> help people transition. That can be done as a user-space parser.

Perhaps - but there clearly are features you are missing.  And your statement
makes the rather big assumption that people *want* to transition.  Why would
you force people to transition, or live with some kind of legacy interface?

But the point of the discussion here is not whether bpftrace can be made to
behave as if it is dtrace - I submitted patches that implement new
functionality.  Are you suggesting that those patches should be rejected
off-hand based on the fact that you feel they are unnecessary because people
should use bpftrace instead?

	Kris

Powered by blists - more mailing lists