linux-kernel - Re: btrfs bio linked list corruption.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 12 Oct 2016 10:42:46 -0400
From:   Chris Mason <clm@...com>
To:     Dave Jones <davej@...emonkey.org.uk>,
        Al Viro <viro@...IV.linux.org.uk>, Josef Bacik <jbacik@...com>,
        David Sterba <dsterba@...e.com>, <linux-btrfs@...r.kernel.org>,
        Linux Kernel <linux-kernel@...r.kernel.org>
Subject: Re: btrfs bio linked list corruption.

On 10/12/2016 10:40 AM, Dave Jones wrote:
> On Wed, Oct 12, 2016 at 09:47:17AM -0400, Dave Jones wrote:
>  > On Tue, Oct 11, 2016 at 11:54:09AM -0400, Chris Mason wrote:
>  >  >
>  >  >
>  >  > On 10/11/2016 10:45 AM, Dave Jones wrote:
>  >  > > This is from Linus' current tree, with Al's iovec fixups on top.
>  >  > >
>  >  > > ------------[ cut here ]------------
>  >  > > WARNING: CPU: 1 PID: 3673 at lib/list_debug.c:33 __list_add+0x89/0xb0
>  >  > > list_add corruption. prev->next should be next (ffffe8ffff806648), but was ffffc9000067fcd8. (prev=ffff880503878b80).
>  >  > > CPU: 1 PID: 3673 Comm: trinity-c0 Not tainted 4.8.0-think+ #13
>  >  > >  ffffc90000d87458 ffffffff8d32007c ffffc90000d874a8 0000000000000000
>  >  > >  ffffc90000d87498 ffffffff8d07a6c1 0000002100000246 ffff88050388e880
>  >
>  > I hit this again overnight, it's the same trace, the only difference
>  > being slightly different addresses in the list pointers:
>  >
>  > [42572.777196] list_add corruption. prev->next should be next (ffffe8ffff806648), but was ffffc90000647cd8. (prev=ffff880503a0ba00).
>  >
>  > I'm actually a little surprised that ->next was the same across two
>  > reboots on two different kernel builds.  That might be a sign this is
>  > more repeatable than I'd thought, even if it does take hours of runtime
>  > right now to trigger it.  I'll try and narrow the scope of what trinity
>  > is doing to see if I can make it happen faster.
>
> .. and of course the first thing that happens is a completely different
> btrfs trace..
>
>
> WARNING: CPU: 1 PID: 21706 at fs/btrfs/transaction.c:489 start_transaction+0x40a/0x440 [btrfs]
> CPU: 1 PID: 21706 Comm: trinity-c16 Not tainted 4.8.0-think+ #14
>  ffffc900019076a8 ffffffffb731ff3c 0000000000000000 0000000000000000
>  ffffc900019076e8 ffffffffb707a6c1 000001e9f5806ce0 ffff8804f74c4d98
>  0000000000000801 ffff880501cfa2a8 000000000000008a 000000000000008a

This isn't even IO.  Uuughhhh.  We're going to need a fast enough test 
that we can bisect.

-chris