linux-kernel - Re: [PATCH 0/5] [GIT PULL] updates for tip/tracing/ftrace

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20090321200955.GE5956@nowhere>
Date:	Sat, 21 Mar 2009 21:09:56 +0100
From:	Frederic Weisbecker <fweisbec@...il.com>
To:	Ingo Molnar <mingo@...e.hu>
Cc:	Steven Rostedt <rostedt@...dmis.org>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	LKML <linux-kernel@...r.kernel.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Peter Zijlstra <peterz@...radead.org>
Subject: Re: [PATCH 0/5] [GIT PULL] updates for tip/tracing/ftrace

On Sat, Mar 21, 2009 at 07:18:58PM +0100, Ingo Molnar wrote:
> 
> * Frederic Weisbecker <fweisbec@...il.com> wrote:
> 
> > >  [<ffffffff8020c79d>] return_to_handler+0x0/0x73
> > >  [<ffffffff8029ea13>] rcu_pending+0x2c/0x5e
> > >  [<ffffffff8020c79d>] return_to_handler+0x0/0x73
> > >  [<ffffffff8026abef>] update_process_times+0x3c/0x77
> > >  [<ffffffff8020c79d>] return_to_handler+0x0/0x73
> > >  [<ffffffff802875dd>] tick_periodic+0x6e/0x70
> > 
> > 
> > Still hanging in the timer interrupt.
> > I guess it makes the timer interrupt servicing too slow and then
> > once it is serviced, another one is raised.
> > 
> > But the cause is perhaps more complex
> > 
> > I think you have had too much hanging of this type. I'm preparing 
> > a fix that checks periodically if the function graph tracer is 
> > spending too much time in an interrupt.
> > 
> > I guess I could count the number of function executed between the 
> > irq entry and its exit.
> > 
> > That's the best: if we are hanging in an interrupt, it could be 
> > whatever interrupt and the jiffies could not be progressing so I 
> > can't rely on time but only on number of functions executed.
> > 
> > May be 10000 calls is a good threshold before killing the function 
> > graph inside an interrupt?
> 
> i think the problem isnt even the IRQ handler - but the fact that 
> the (timer) irq handler gets re-triggered - so all we do is 
> processing timer IRQs.
> 
> Your patch would detect a timer IRQ hanging - but it would not 
> detect the 'system makes no progress because there's always anoter 
> pending timer IRQ to execute' situation.


Ah, you're right.

 
> So i think we need a "function trace watchdog" - which kills the 
> tracer if we do more than 100,000,000 entries since we started the 
> self-test, or so.
> 
> 	Ingo


The problem is that it can happen also on other contexts than selftests.
For example with ftrace=function_graph or by simply enabling the tracer
later.

Sometimes it can happen during the selftests, sometimes it's only
revealed by manually enabling it. I just remind another hang that
you reported earlier and which I half-solved by fixing a pointless
softirq call...

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/