linux-kernel - Re: [Patch -tip 0/4] Creation of the initcall tracer

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20080924085848.GA22251@elte.hu>
Date:	Wed, 24 Sep 2008 10:58:48 +0200
From:	Ingo Molnar <mingo@...e.hu>
To:	Frédéric Weisbecker <fweisbec@...il.com>
Cc:	linux-kernel@...r.kernel.org, Steven Rostedt <rostedt@...dmis.org>,
	Steven Noonan <steven@...inklabs.net>,
	Arjan van de Ven <arjan@...radead.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	"H. Peter Anvin" <hpa@...or.com>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>
Subject: Re: [Patch -tip 0/4] Creation of the initcall tracer


* Frédéric Weisbecker <fweisbec@...il.com> wrote:

> Ingo Molnar a écrit :
> > * Ingo Molnar <mingo@...e.hu> wrote:
> > 
> >> i've integrated tip/tracing/fastboot into tip/master and have started 
> >> testing it. It passed the basic tests already so i've just pushed out 
> >> the new tip/master. Please double-check that i have not messed up the 
> >> rename or the integration somewhere.
> > 
> > FYI, -tip testing found a boot hang with CONFIG_BOOT_TRACER=y, on a 
> > 32-bit T60 laptop. I've attached the config that provokes the hang. (you 
> > need initcall_debug on the boot commandline)
> > 
> > The hang always happens after a specific initcall, shortly after tracing 
> > is enabled. It just hangs indefinitely there, the last line displayed 
> > was:
> > 
> >  [   0.004017] initcall ehci_hcd_init+0x0/0x80 returned 0 after 228 msecs
> > 
> > it should have printed this next:
> > 
> > [    0.808024] calling  ohci_hcd_mod_init+0x0/0x80 @ 10
> > 
> > but that line is never printed.
> > 
> > i've bisected the hang down to this specific commit:
> > 
> >  | 35df98dfe2b2503436a8cfda301657810df50a3a is first bad commit
> >  | commit 35df98dfe2b2503436a8cfda301657810df50a3a
> >  | Author: Frédéric Weisbecker <fweisbec@...il.com>
> >  | Date:   Tue Sep 23 11:38:18 2008 +0100
> >  |
> >  |     tracing/ftrace: launch boot tracing after pre-smp initcalls
> > 
> > An idea: maybe the boot tracer is interacting with the ftrace self-tests 
> > somehow? CONFIG_FTRACE_SELFTEST=y is enabled.
> > 
> > another thing i noticed is that the sysprof tracer was initialized and 
> > self-tested shortly before the hang occured. Might be unrelated.
> > 
> > the bisection log is:
> > 
> >  # bad:  [6522eab5] Merge branch 'x86/uv'
> >  # good: [699b81f5] Merge branch 'x86/iommu'
> >  # bad:  [8007aae9] timers: fix build error in !oneshot case
> >  # good: [7f29b23d] Merge branch 'core/rcu'
> >  # bad:  [a54f17e3] manual merge of tracing/fastboot
> >  # good: [bcbbe946] tracing/ftrace: make tracing suitable to run the b
> >  # good: [d7095e5a] Merge branch 'fastboot'
> >  # bad:  [35df98df] tracing/ftrace: launch boot tracing after pre-smp
> >  # good: [42aaa6be] tracing/ftrace: give an entry on the config for bo
> > 
> > 	Ingo
> > 
> Ingo Molnar a écrit :
> > * Ingo Molnar <mingo@...e.hu> wrote:
> > 
> >> i've integrated tip/tracing/fastboot into tip/master and have started 
> >> testing it. It passed the basic tests already so i've just pushed out 
> >> the new tip/master. Please double-check that i have not messed up the 
> >> rename or the integration somewhere.
> > 
> > FYI, -tip testing found a boot hang with CONFIG_BOOT_TRACER=y, on a 
> > 32-bit T60 laptop. I've attached the config that provokes the hang. (you 
> > need initcall_debug on the boot commandline)
> > 
> > The hang always happens after a specific initcall, shortly after tracing 
> > is enabled. It just hangs indefinitely there, the last line displayed 
> > was:
> > 
> >  [   0.004017] initcall ehci_hcd_init+0x0/0x80 returned 0 after 228 msecs
> > 
> > it should have printed this next:
> > 
> > [    0.808024] calling  ohci_hcd_mod_init+0x0/0x80 @ 10
> > 
> > but that line is never printed.
> > 
> > i've bisected the hang down to this specific commit:
> > 
> >  | 35df98dfe2b2503436a8cfda301657810df50a3a is first bad commit
> >  | commit 35df98dfe2b2503436a8cfda301657810df50a3a
> >  | Author: Frédéric Weisbecker <fweisbec@...il.com>
> >  | Date:   Tue Sep 23 11:38:18 2008 +0100
> >  |
> >  |     tracing/ftrace: launch boot tracing after pre-smp initcalls
> > 
> > An idea: maybe the boot tracer is interacting with the ftrace self-tests 
> > somehow? CONFIG_FTRACE_SELFTEST=y is enabled.
> > 
> > another thing i noticed is that the sysprof tracer was initialized and 
> > self-tested shortly before the hang occured. Might be unrelated.
> > 
> > the bisection log is:
> > 
> >  # bad:  [6522eab5] Merge branch 'x86/uv'
> >  # good: [699b81f5] Merge branch 'x86/iommu'
> >  # bad:  [8007aae9] timers: fix build error in !oneshot case
> >  # good: [7f29b23d] Merge branch 'core/rcu'
> >  # bad:  [a54f17e3] manual merge of tracing/fastboot
> >  # good: [bcbbe946] tracing/ftrace: make tracing suitable to run the b
> >  # good: [d7095e5a] Merge branch 'fastboot'
> >  # bad:  [35df98df] tracing/ftrace: launch boot tracing after pre-smp
> >  # good: [42aaa6be] tracing/ftrace: give an entry on the config for bo
> > 
> > 	Ingo
> > 
> 
> Hehe! I was preparing a patch because I saw last night that the self-tests were reseting and touching the ring-buffer during the initcall tracing.
> And now I can see the results of this problem....
> 
> Following is a patch that should fix it.

applied to tip/tracing/fastboot, thanks Frédéric!

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/