linux-kernel - Re: frequent softlockups with 3.10rc6.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1372091079.18733.168.camel@gandalf.local.home>
Date:	Mon, 24 Jun 2013 12:24:39 -0400
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Dave Jones <davej@...hat.com>
Cc:	Oleg Nesterov <oleg@...hat.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	"Eric W. Biederman" <ebiederm@...ssion.com>,
	Andrey Vagin <avagin@...nvz.org>
Subject: Re: frequent softlockups with 3.10rc6.

On Mon, 2013-06-24 at 12:00 -0400, Dave Jones wrote:
> On Mon, Jun 24, 2013 at 10:52:29AM -0400, Steven Rostedt wrote:
>  
>  > > >  check_list_nodes corruption. next->prev should be prev (ffff88023b8a1a08), but was 00ffff88023b8a1a. (next=ffff880243288001).
>  > > 
>  > > Can't find "check_list_nodes" in lib/list_debug.c or elsewhere...
>  > > 
>  > > >   [<ffffffff816e467d>] dump_stack+0x19/0x1b
>  > > >   [<ffffffff8104a0c1>] warn_slowpath_common+0x61/0x80
>  > > >   [<ffffffff8104a12c>] warn_slowpath_fmt+0x4c/0x50
>  > > >   [<ffffffff81112c61>] rb_head_page_deactivate.isra.39+0x61/0x80
>  > > 
>  > > How? rb_list_head_clear() just modifies list->next directly.
>  > > 
>  > > > hopefully despite that it'll actually function as intended.
>  > > 
>  > > Yes ;)
>  > 
>  > I'm curious to what happened.
> 
> Ah, this is the first victim of my new 'check sanity of nodes during list walks' patch.
> It's doing the same prev->next next->prev checking as list_add and friends.
> I'm looking at getting it into shape for a 3.12 merge after some other preparatory patches
> go into 3.11

OK, and you may need to make an exception for the ring buffer. To do a
lockless swap out of the reader page for one of the pages in the buffer,
it uses the 2 LSB as flags. Notice the "next=ffff880243288001", that "1"
is a flag that states the next page is the "header" page (next to be
read). We use cmpxchg to update the pages to handle races between the
reader and writer.

In fact, the above code is done on reset of the ring buffer, where we
clear those bits:

	list_for_each(hd, cpu_buffer->pages)
		rb_list_head_clear(hd);

The function that was flagged, was the one going through and clearing
the bits.

It seems that your new tool is failing due to this trick.

-- Steve


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/