Date:	Thu, 29 Oct 2009 18:17:33 -0400
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Masami Hiramatsu <mhiramat@...hat.com>
Cc:	linux-kernel@...r.kernel.org, Ingo Molnar <mingo@...e.hu>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Arnaldo Carvalho de Melo <acme@...hat.com>,
	"H. Peter Anvin" <hpa@...or.com>, Li Zefan <lizf@...fujitsu.com>,
	Lai Jiangshan <laijs@...fujitsu.com>,
	"David S. Miller" <davem@...emloft.net>,
	Stephen Hemminger <shemminger@...ux-foundation.org>
Subject: Re: [PATCH 0/3][RFC] tracing/kprobes: prevent jprobes from
 crashing function graph tracer

On Thu, 2009-10-29 at 18:02 -0400, Masami Hiramatsu wrote:
> > 
> > Jprobes and the function graph tracer use the same mechanism to trace
> > the exit of a function. Unfortunately, only one can be done at a time.
> > The function graph tracer replaces the return address with its own handler,
> > but so does jprobes. The two are not compatible.
> 
> AFAIK, Jprobe doesn't trace the exit of a function. I assume that
> jprobe's user handler causes the problem, since the handler never
> returns normal way.
> Instead of that, it just calls jprobe_return() which causes
> int3 to be trapped by kprobe's break handler. And the break handler
> fixup regs->ip to back to traced function.

Ah, yes, my documenting this is wrong. It's the skipped jprobe that
messed it up.

> 
> Actually, this will cause a problem with function graph tracer.
> The f-g-tracer push the return address into the special stack and replaces
> it with fixup function (This is similar (not same) mechanism of kretprobe.)
> And then the traced function returns, it returns to the fixup function and
> it pops the return address up and back to the real caller.
> 
> So, if the f-g-tracer traces jprobe user handler, the pop operation
> will be skipped because the the handler never returns.

Exactly!

> 
> > The solution I am proposing with this patch set is to add a call in
> > ftrace that lets other code in the kernel permanently disable functions from
> > being traced by the function and function graph tracer. As a probe function
> > is registered with jprobes, it calls this new function and that entry
> > will be removed from being traced.
> >
> > I tested this with this patch series and it does solve the problem.
> > 
> > Some issues though:
> > 
> > 1) this only works when DYNAMIC_FTRACE is enabled. We can prevent
> > function graph tracing with jprobes when DYNAMIC_FTRACE is not
> > enabled through Kconfig dependencies. Or have the registering of
> > a jprobe permanently disable function graph tracing.
> 
> IMHO, those *probe handler should be tagged as __kprobes and notrace.

Yeah, I agree. But how do you guarantee that it does. If one forgets,
than we still have the issue. We can perhaps test to make sure the
function is in the kprobes section. But that does not mean they will not
be notraced. The __kprobes and notrace are no longer in the same set.

> 
> > 2) This also prevents the function tracer from being able to trace a
> > function probe, even though the function tracer is not at issue
> > with this bug.
> 
> I think we can skip those user handlers, because those are irregular
> functions and user can control (enable/disable) it.

True, but it may be nice to still trace them.

> 
> BTW, in this specific case, I assume that it can use tracepoint
> instead of jprobe and move tcp_probe to a part of ftrace :-), isn't it?
> (Or, if it is just for a debugging, Systemtap can help it.)

That's a question for the networking guys.

-- Steve


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists