Message-ID: <20090306101142.GA26659@elte.hu>
Date:	Fri, 6 Mar 2009 11:11:42 +0100
From:	Ingo Molnar <mingo@...e.hu>
To:	Mathieu Desnoyers <mathieu.desnoyers@...ymtl.ca>
Cc:	Linus Torvalds <torvalds@...ux-foundation.org>,
	linux-kernel@...r.kernel.org,
	Andrew Morton <akpm@...ux-foundation.org>,
	Steven Rostedt <rostedt@...dmis.org>,
	ltt-dev@...ts.casi.polymtl.ca,
	Peter Zijlstra <peterz@...radead.org>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Arjan van de Ven <arjan@...radead.org>,
	Pekka Paalanen <pq@....fi>,
	Arnaldo Carvalho de Melo <acme@...hat.com>,
	"H. Peter Anvin" <hpa@...or.com>, Martin Bligh <mbligh@...gle.com>,
	"Frank Ch. Eigler" <fche@...hat.com>,
	Tom Zanussi <tzanussi@...il.com>,
	Masami Hiramatsu <mhiramat@...hat.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	Jason Baron <jbaron@...hat.com>,
	Christoph Hellwig <hch@...radead.org>,
	Jiaying Zhang <jiayingz@...gle.com>,
	Eduard - Gabriel Munteanu <eduard.munteanu@...ux360.ro>,
	mrubin@...gle.com, md@...gle.com
Subject: Re: [RFC patch 00/41] LTTng 0.105 core for Linux 2.6.27-rc9


* Mathieu Desnoyers <mathieu.desnoyers@...ymtl.ca> wrote:

> Hi,
> 
> I spent the last 4-5 months working with the Fujitsu team at 
> implementing the tracer elements identified as goals at Kernel 
> Summit 2008 and at the following Plumber Conference. My idea 
> was to incremententally adapt the LTTng tracer, currently used 
> in the industry and well tested, to those requirements.
> 
> I spent the last days rearranging/folding/inspecting the LTTng 
> patchset to prepare it for an LKML post. The version 0.105 in 
> the LTTng git tree corresponds to the patchset I am posting 
> here. The said patchset will only include the core features of 
> LTTng, excluding the timestamping infrastructure (trace clock) 
> and excluding the instrumentation.

I'd like to merge the good bits into the tracing tree. Looking 
at the patches you submitted there's a lot of avoidable overlap 
with existing tracing features either present upstream already 
or queued up for v2.6.30 - and we need to work more on 
eliminating that overlap.

I dont think there's much fundamental disagreement just 
different implementations - so we should evaluate each of those 
details one by one, iteratively.

The first step would be to split the patches up into three 
logical buckets:

 - Unique features not present in the tracing infracture, in the 
   event tracer or other tracing plugins - those should be 
   structured as feature additions.

 - Features that you consider superior to existing tracing
   features of the kernel. For those, please iterate the
   existing code with your enhancements - instead of a parallel 
   implementation.

 - Items which offer nothing new and are not superior to 
   existing features, those should be dropped probably. This too 
   is a case by case thing.

Would you be interested in working with us on that? I know that 
both Steve and me would be very much interested in that. If you 
have time/interest to work on that then we can go through each 
patch one by one and categorize them and map out the way to go.

Let me give you a few examples of existing areas of overlap:

> The corresponding git tree contains also the trace clock 
> patches and the lttng instrumentation. The trace clock is 
> required to use the tracer, but it can be used without the 
> instrumentation : there is already a kprobes and userspace 
> event support included in this patchset.

The latest tracing tree includes kernel/tracing/trace_clock.c 
which offers three trace clock variants, with different 
performance/precision tradeoffs:

 trace_clock_local()   [ for pure CPU-local tracers with no idle 
                         events. This is the fastest but least 
                         coherent tracing clock. ]

 trace_clock()         [ intermediate, scalable clock with
                         usable but imprecise global coherency. ]

 trace_clock_global()  [ globally serialized, coherent clock. 
                         It is the slowest but most accurate variant. ]

Tracing plugins can pick their choice. (This is relatively new 
code but you get the idea.)

> This tracer exports binary data through buffers using 
> splice(). The resulting binary files can be parsed from 
> userspace because the format string metadata is exported in 
> the files. The event set can be enhanced by adding tracepoints 
> to the kernel code and by creating probe modules, which 
> connects callbacks to the tracepoints and contain the format 
> string metainformation. Those callbacks are responsible for 
> writing the data in the trace buffers. This separation between 
> the trace buffer format string and the tracepoints is done on 
> purpose so the core kernel instrumentation (tracepoints) is 
> not exported to userspace, which will make maintainance much 
> easier.

A tracepoint format specification mechanism plus working (and 
fast!) zero-copy splice() support of the ring-buffer exists in 
the latest tracing tree already - as you are probably aware of 
because you commented on those patches a few days ago.

There are 3 good ways to go from here regarding the trace 
buffering and splice code:

  1- we end up switching to the lttng version in essence
  2- we end up keeping the tracing tree version
  3- we end up somewhere inbetween

Which point in the above spectrum we will settle down on depends 
on the technical details.

Note, whichever path we choose a gradual, iterative workflow is 
still needed, so that we improve the existing upstream code with
lttng enhancements gradually.

This approach works for all your other patches as well. A 
direct, constructive comparison and active work on unifying them 
is required.

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists