Date:	Thu, 6 Aug 2009 09:35:01 -0400
From:	Mathieu Desnoyers <mathieu.desnoyers@...ymtl.ca>
To:	Peter Zijlstra <a.p.zijlstra@...llo.nl>
Cc:	Pekka Enberg <penberg@...helsinki.fi>, Ingo Molnar <mingo@...e.hu>,
	acme <acme@...hat.com>, Steven Rostedt <rostedt@...dmis.org>,
	Frédéric Weisbecker <fweisbec@...il.com>,
	Eduard-Gabriel Munteanu <eduard.munteanu@...ux360.ro>,
	roland <roland@...hat.com>,
	Christoph Hellwig <hch@...radead.org>,
	Masami Hiramatsu <mhiramat@...hat.com>, fche <fche@...hat.com>,
	pierre-marc.fournier@...ymtl.ca, michel.dagenais@...ymtl.ca,
	linux-kernel@...r.kernel.org
Subject: Re: malloc() tracing in perf?

* Peter Zijlstra (a.p.zijlstra@...llo.nl) wrote:
> On Thu, 2009-08-06 at 10:48 +0300, Pekka Enberg wrote:
> > Hi,
> > 
> > It's me again :-).
> > 
> > I have a little user-space application that is pretty memory hungry and 
> > I want to understand why. I started to google around for a memory 
> > profiler or a malloc() tracer but didn't seem to find anything really 
> > useful.
> > 
> > But then it hit me, why can't I have kmemtrace + perf but for 
> > user-space? Something like the "Malloc Trace" shown here:
> > 
> > http://developer.apple.com/documentation/developertools/conceptual/SharkUserGuide/OtherProfilingandTracingTechniques/OtherProfilingandTracingTechniques.html#//apple_ref/doc/uid/TP40005233-CH6-SW17
> > 
> > Does this sound like something that could/should be part of "perf"? How 
> > would all this work anyway? Can we intercept malloc() and free() 
> > somehow? Where would the data be pushed? Am I just going perf-crazy and 
> > trying to turn it into a swiss army knife because it's so easy to use?-)
> 
> OK, you just trod into a wasp's nest :-)
> 
> 
> That sounds like uprobes, the equivalent of kprobes but for userspace.
> 
> I seem to have heard people are working on such a thing, but I can't
> seem to find a single LKML post with 'uprobe' in the subject in the past
> two years or something (except for MTUprobe) -- so I guess its not
> really going anywhere any fast.
> 
> Now doing probes on userspace is hard because you need to know more
> about the userspace bits than a kernel really ought to be interested in.
> 
> Then again, the only way to extract bits from userspace is to stop it --
> now one could do that using [pu]trace and have some monitoring app prod
> at it like any other debugger would, and I think this is the approach
> suggested by some (hch iirc).
> 
> Others seem to think we ought to stuff all this into the kernel, I can
> only imagine the pain that will cause, since you need to teach the
> kernel about these instrumentation sites' context, so I can only imagine
> it'd be through a kernel module interface much like system-tap does
> (they would be doing the in-kernel bit).
> 
> Then there are the tracer folks who also want to collect userspace
> traces. Some have proposed a sys_trace() call, others want to play silly
> games with mmap() and then there is the uprobe idea. Others (tglx and
> me) proposed letting userspace log itself and post-merge all the various
> trace buffers to get a complete picture.
> 

Hi !

Since we are comparing solutions, here is the path we are taking in
LTTng. We choose to let userspace log itself and merge the
kernel/userspace trace streams at post-processing. Past winter,
Pierre-Marc has been working on a port of the LTTng kernel tracer to
userspace. We named it "ust" library (for userspace tracing). Although
it still requires some polishing, it should be a generally usable state
(currently only on x86 with synchronized TSCs).

The plan is to gradually instrument userspace libraries and applications
with tracepoints, just like we are currently doing in the kernel.

The main difference between our approach and Dtrace is that Dtrace
requires a trap when userspace instrumentation is hit (yeah, even for
static instrumentation, isn't it a shame ?), adding a huge performance
overhead (especially on x86). Using tracepoints and staying in userspace
to log data helps keeping the performance overhead low.

For more info on lib ust :

Combined Tracing of the Kernel and Applications with LTTng at Linux
Symposium 2009 (presentation)
http://lttng.org/sites/default/files/presentation.pdf

Access to the git tree:

http://git.dorsal.polymtl.ca/?p=ust.git
(please note that we are about 99.9% done in asking permissions or
rewriting code to distribute under LGPL. I think we only wait for 1-2
bits of code to be rewritten. Pierre-Marc could tell more about it.
Until then, it's temporarily distributed under GPLv2.)

Eventually, parts of this ust library could be integrated to the libc,
so its usage should become more transparent that it currently is.

Eventually, the only dependency ust would have on the kernel is to
require a userspace trace clock mapped in a vDSO, synchronized with the
kernel trace clock.

Feedback is welcome,

Thanks,

Mathieu

> 
> Anyway, like you say, it has uses (potentially very powerful ones),
> Sun/Apple do it with Dtrace, Linux wants it but I don't think we quite
> agreed on how to do it :-)
> 
> 
> And here I see LKML isn't on the CC list, perhaps we should?
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists