Message-ID: <4A8DF197.2080107@redhat.com>
Date:	Thu, 20 Aug 2009 21:00:07 -0400
From:	Masami Hiramatsu <mhiramat@...hat.com>
To:	Frederic Weisbecker <fweisbec@...il.com>
CC:	Ingo Molnar <mingo@...e.hu>, Steven Rostedt <rostedt@...dmis.org>,
	lkml <linux-kernel@...r.kernel.org>,
	Ananth N Mavinakayanahalli <ananth@...ibm.com>,
	Avi Kivity <avi@...hat.com>, Andi Kleen <ak@...ux.intel.com>,
	Christoph Hellwig <hch@...radead.org>,
	"Frank Ch. Eigler" <fche@...hat.com>,
	"H. Peter Anvin" <hpa@...or.com>, Jason Baron <jbaron@...hat.com>,
	Jim Keniston <jkenisto@...ibm.com>,
	"K.Prasad" <prasad@...ux.vnet.ibm.com>,
	Lai Jiangshan <laijs@...fujitsu.com>,
	Li Zefan <lizf@...fujitsu.com>,
	PrzemysławPawełczyk <przemyslaw@...elczyk.it>,
	Roland McGrath <roland@...hat.com>,
	Sam Ravnborg <sam@...nborg.org>,
	Srikar Dronamraju <srikar@...ux.vnet.ibm.com>,
	Tom Zanussi <tzanussi@...il.com>,
	Vegard Nossum <vegard.nossum@...il.com>,
	systemtap <systemtap@...rces.redhat.com>,
	kvm <kvm@...r.kernel.org>,
	DLE <dle-develop@...ts.sourceforge.net>
Subject: Re: [TOOL] kprobestest : Kprobe stress test tool

Frederic Weisbecker wrote:
>> Most of them can be fixed just by adding __kprobes.
>> Some of them which are already in the another section, kprobes
>> should check the symbols are in the section.
>
>
> You mean the blacklist?
>
> I also fear that putting bad kprobed functions into the kprobe
> section or into the blacklist may hide some kprobe internal bugs.
>
> Doing so is indeed mandatory for functions that trigger tracing
> recursion of things like that, but what if kprobe has an internal
> bug that only triggers while probe a certain class of function.
>
> Ie: it would be nice to identify the reason of the crash for
> each culprit in these lists.
 >
 > That may even help to find the others in advance.

Indeed, actually I've found some bugs while making jump-optimization
patches by using this stress test.
But some of them are obviously what we just forget to add __kprobes,
since those will be called from kprobes int3 handling functions.

And also, many lock-related code has been changed. I think
kprobes should use raw_*_lock, or prohibit to probe lock monitoring
functions like lockdep, because it will cause recursive call.

>
> Also kprobes seems to be a very fragile feature (that's what
> this selftest unearthes at least for me).
> And it really needs a recursion detection that stops every kprobing
> while reaching a given threshold of recursion. Something
> that would dump the stack and the falling kprobe structure.

Hmm, kprobes already has recursion detection(kp->nmiss), so
maybe, we can check it.

>
> That would avoid such hard lockups and also help to identify
> the dangerous symbols to probe.
>
>
>
>>> The problem is that I don't have any serial line in this
>>> box then I can't catch any crash log.
>>> My K7 testbox also died in my arms this afternoon.
>>>
>>> But I still have two other testboxes (one P2 and one P3),
>>> hopefully I could reproduce the problem in these boxes
>>> in which I can connect a serial line.
>>
>> Thank you for helping me to find it!
>>
>>> I've pushed your patches in the following git tree:
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/fgrederic/random-tracing.git \
>>> 	tracing/kprobes
>>>
>>> So you can send patches on top of this one.
>>
>> Great! I've found another trivial bugs, so I'll fix those on it.
>
> Cool :)
>
> Btw, here is the result of your stress test in a PIII (attaching the log
> and the config).

Thanks, I'll check that.

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@...hat.com

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists