linux-kernel - Re: [ath9k-devel] ath9k: massive unexplained latency in 2.6.27 (rc5, rc6, probably others)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20080919083208.GA15730@elte.hu>
Date:	Fri, 19 Sep 2008 10:32:08 +0200
From:	Ingo Molnar <mingo@...e.hu>
To:	Steven Noonan <steven@...inklabs.net>
Cc:	"Luis R. Rodriguez" <lrodriguez@...eros.com>,
	Luis Rodriguez <Luis.Rodriguez@...eros.com>,
	"ath9k-devel@...ts.ath9k.org" <ath9k-devel@...ts.ath9k.org>,
	linux-wireless <linux-wireless@...r.kernel.org>,
	LKML <linux-kernel@...r.kernel.org>,
	Steven Rostedt <rostedt@...dmis.org>
Subject: Re: [ath9k-devel] ath9k: massive unexplained latency in 2.6.27
	(rc5, rc6, probably others)


* Steven Noonan <steven@...inklabs.net> wrote:

> On Fri, Sep 19, 2008 at 1:17 AM, Ingo Molnar <mingo@...e.hu> wrote:
> > in tip/master there's an ftrace_printk() facility. You can just replace
> > the verbose printk()s with ftrace_printk() and see the result in
> > /debug/tracing/trace.
> >
> > This sort of tracing has very low overhead and can be used in an easy
> > ad-hoc manner with no extra infrastructure. Here's a few quick-start
> > links about how to enable the scheduler tracer:
> >
> >  http://redhat.com/~mingo/sched-devel.git/readme-tracer.txt
> >  http://redhat.com/~mingo/sched-devel.git/howto-trace-latencies.txt
> >
> > any ftrace_printk() you add in the kernel will show up in that trace.
> >
> > (If the scheduling events are uninteresting and clutter the output then
> > you might want to remove the scheduler tracing entries from kernel/*.c
> > by removing the trace_sched_*() calls or use a less noisy tracer.)
> >
> > to get enough of a trace history you might want to increase the number
> > of trace entries in /debug/tracing/trace_entries from 16K to 128K or so.
> >
> 
> Thanks for the suggestion, I'll do that. :)

one more suggestion: you should first check how large the 'scope' of 
your trace is - i.e. how long of a time span it can cover, in the 
workload that you are going to trace.

if it's too short - say just a few hundred milliseconds then you need a 
larger trace buffer or fewer trace entries.

another thing: you can auto-stop tracing on the failure condition. That 
way you dont have to worry about whether you notice the badness. 
Unfortunately there's no easy callback to just anonymously stop all 
tracing that goes on. There's ftrace_kill_atomic() but i'm not sure it 
properly stops the scheduler tracer.

What is needed is to set kernel/trace/trace.c's tracing_disabled 
variable to 0. I've Cc:-ed Steve Rostedt - Steve, why is there no easy 
ad-hoc way to trigger a stop-tracing event without elaborate changes?

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/