linux-kernel - Re: [PATCH 2/4] tracing: add event trace infrastructure

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.DEB.2.00.0902250923360.12697@gandalf.stny.rr.com>
Date:	Wed, 25 Feb 2009 09:41:32 -0500 (EST)
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Andrew Morton <akpm@...ux-foundation.org>
cc:	Pekka Enberg <penberg@...helsinki.fi>, Ingo Molnar <mingo@...e.hu>,
	LKML <linux-kernel@...r.kernel.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Peter Zijlstra <peterz@...radead.org>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Theodore Tso <tytso@....edu>,
	Arjan van de Ven <arjan@...radead.org>,
	Pekka Paalanen <pq@....fi>,
	Arnaldo Carvalho de Melo <acme@...hat.com>,
	Jason Baron <jbaron@...hat.com>,
	Martin Bligh <mbligh@...gle.com>,
	Mathieu Desnoyers <compudj@...stal.dyndns.org>,
	"Frank Ch. Eigler" <fche@...hat.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	Jens Axboe <jens.axboe@...cle.com>,
	Masami Hiramatsu <mhiramat@...hat.com>,
	Steven Rostedt <srostedt@...hat.com>
Subject: Re: [PATCH 2/4] tracing: add event trace infrastructure

On Wed, 25 Feb 2009, Andrew Morton wrote:

> On Wed, 25 Feb 2009 11:00:53 +0200 Pekka Enberg <penberg@...helsinki.fi> wrote:
> Parse this:
> 
> irqsoff latency trace v1.1.5 on 2.6.26-rc8
> --------------------------------------------------------------------
>  latency: 97 us, #3/3, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:2)
>     -----------------
>     | task: swapper-0 (uid:0 nice:0 policy:0 rt_prio:0)
>     -----------------
>  => started at: apic_timer_interrupt
>  => ended at:   do_softirq
> 
> #                _------=> CPU#
> #               / _-----=> irqs-off
> #              | / _----=> need-resched
> #              || / _---=> hardirq/softirq
> #              ||| / _--=> preempt-depth
> #              |||| /
> #              |||||     delay
> #  cmd     pid ||||| time  |   caller
> #     \   /    |||||   \   |   /
>   <idle>-0     0d..1    0us+: trace_hardirqs_off_thunk (apic_timer_interrupt)
>   <idle>-0     0d.s.   97us : __do_softirq (do_softirq)
>   <idle>-0     0d.s1   98us : trace_hardirqs_on (do_softirq)
> 
> your time starts now.

Note, that is a legacy format, that works great for the latency tracers. 
Those include (the example you used) irqsoff, the preempt off, wake up 
latencies. This gives us a nice listing of why we are hitting a latency.
All other plugins generally use the trace file, although they could also 
use that file too.

As for your English comment. I showed the header to my wife, and she had 
no idea what irqs-off means, nor need-resched, nor preempt-depth ;-)   
They are not quite English as they are technical terms. Most of the names 
are hard coded into the kernel too.

Yeah, maybe that file (latency_trace) is a bit too much. I for one love 
it. It is great to load a kernel on some remote box, and run the irqs off 
latency tracer to see where the interrupts are disabled for the longest 
time. This format is really nice because it shows me when we are in an 
interrupt, or interrupts are disabled, and when the task should have been 
rescheduled.

This has help find places that we miss a preemption check too.

If I had to also include a user space program, I would need to tell the 
customer how to install it, and run it. Just telling someone to mount a 
directory, run your test, cat a file to some temp file and send me the 
temp file is much easier. Because most admins know how to do such things. 
No learning curve, except what to mount and what to cat.

I've given a few talks on ftrace. And when I'm approached afterward by 
people (usually the ones not paying attention to the speech ;-) asking 
what utils they need to install. You should see the smile on their faces 
when I tell them, "All you need is what is in busybox".  Then they say, 
"No, really? What do I need to use ftrace?" I again tell them, a kernel 
with ftrace and a minimum requirement of what is in busybox: cat, grep, 
echo. That is what makes ftrace powerful and robust.

Once I break that "busybox" rule. ftrace will be used by only a quarter of
those that use it now (and probably less).

-- Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/