linux-kernel - Re: 2.6.28-rc2 hates my e1000e

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.DEB.1.10.0811011326390.9409@gandalf.stny.rr.com>
Date:	Sat, 1 Nov 2008 13:35:36 -0400 (EDT)
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
cc:	Jonathan Corbet <corbet@....net>, Yinghai Lu <yinghai@...nel.org>,
	Ingo Molnar <mingo@...e.hu>, Robert Hancock <hancockr@...w.ca>,
	e1000-devel@...ts.sourceforge.net,
	LKML <linux-kernel@...r.kernel.org>
Subject: Re: 2.6.28-rc2 hates my e1000e


[ Note, lots of family activities this weekend, so my response may be 
  slow ]

On Sat, 1 Nov 2008, Linus Torvalds wrote:
> On Sat, 1 Nov 2008, Jonathan Corbet wrote:
> 
> > Networking is fine in the absence of NFS.  I retried things and
> > stress-tested it in a few ways with no trouble.  I think your last patch
> > fixes the network card just fine.
> > 
> > Then I tried NFS again, watching more closely this time around.
> > Everything locks up.  In fact, the soft lockup watchdog starts to
> > scream:
> 
> Interesting. I wonder why it happens for NFS, but not apparently for all 
> your other modules.
> 
> It does look very much like a ftrace issue, though, not NFS or 
> network-related. Steven? Is this something that you are aware of already, 
> with what looks like a lockup in ftrace_record_ip()?
> 

No, I have not seen this before. The code is now pretty straight forward.

Jon, could you do a gdb vmlinux and a li *ftrace_record_ip+0xcb to find 
the exact line that is?

Showing the call path of this we have:

in module.c:

  	/* sechdrs[0].sh_size is always zero */
	mseg = section_objs(hdr, sechdrs, secstrings, "__mcount_loc",
			    sizeof(*mseg), &num_mcount);
	ftrace_init_module(mseg, mseg + num_mcount);

Where we pass a table of mcount callers to the ftrace_init_module.

void ftrace_init_module(unsigned long *start, unsigned long *end)
{
	if (ftrace_disabled || start == end)
		return;
	ftrace_convert_nops(start, end);
}

I wonder if I should test to make sure start is < end :-/

ftrace_convert_nops does the following under a mutex.

	while (p < end) {
		addr = ftrace_call_adjust(*p++);
		ftrace_record_ip(addr);
	}


And ftrace_record_ip does:

ftrace_record_ip(unsigned long ip)
{
	struct dyn_ftrace *rec;
	if (!ftrace_enabled || ftrace_disabled)
		return NULL;
	rec = ftrace_alloc_dyn_node(ip);
	if (!rec)
		return NULL;
	rec->ip = ip;
	list_add(&rec->list, &ftrace_new_addrs);
	return rec;
}


The ftrace_alloc_dyn_node does allocate a page if we are running low, but 
there's no other loops or locks that I can see us deadlocking on.

-- Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/