linux-kernel - Re: bio linked list corruption.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20161020065216.GB29032@gmail.com>
Date:   Thu, 20 Oct 2016 08:52:17 +0200
From:   Ingo Molnar <mingo@...nel.org>
To:     Linus Torvalds <torvalds@...ux-foundation.org>
Cc:     Philipp Hahn <pmhahn@...ahn.de>,
        Thomas Gleixner <tglx@...utronix.de>, Chris Mason <clm@...com>,
        Jens Axboe <axboe@...com>,
        Dave Jones <davej@...emonkey.org.uk>,
        Al Viro <viro@...iv.linux.org.uk>, Josef Bacik <jbacik@...com>,
        David Sterba <dsterba@...e.com>,
        linux-btrfs <linux-btrfs@...r.kernel.org>,
        Linux Kernel <linux-kernel@...r.kernel.org>
Subject: Re: bio linked list corruption.


* Linus Torvalds <torvalds@...ux-foundation.org> wrote:

> On Wed, Oct 19, 2016 at 10:09 AM, Philipp Hahn <pmhahn@...ahn.de> wrote:
> >
> > Nearly a month ago I reported also a "list_add corruption", but with 4.1.6:
> > <http://marc.info/?l=linux-kernel&m=147508265316854&w=2>
> >
> > That server rungs Samba4, which also is a heavy user of xattr.
> 
> That one looks very different. In fact, the list that got corrupted
> for you has since been changed to a hlist (which is *similar* to our
> doubly-linked list, but has a smaller head and does not allow adding
> to the end of the list).
> 
> Also, the "should be" and "was" values are very close, and switched:
> 
> should be ffffffff81ab3ca8, but was ffffffff81ab3cc8
> should be ffffffff81ab3cc8, but was ffffffff81ab3ca8
> 
> so it actually looks like it was the same data structure. In
> particular, it looks like enqueue_timer() ended up racing on adding an
> entry to one index in the "base->vectors[]" array, while hitting an
> entry that was pointing to another index near-by.
> 
> So I don't think it's related. Yours looks like some subtle timer base
> race. It smells like a locking problem with timers. I'm not seeing
> what it might be, but it *might* have been fixed by doing the
> TIMER_MIGRATING bit right in add_timer_on() (commit 22b886dd1018).
> 
> Adding some timer people just in case, but I don't think your 4.1
> report is related.

Side note: in case timer callback related corruption is suspected, a very 
efficient debugging method is to enable debugobjects tracking+checking:

CONFIG_DEBUG_OBJECTS=y
CONFIG_DEBUG_OBJECTS_SELFTEST=y
CONFIG_DEBUG_OBJECTS_FREE=y
CONFIG_DEBUG_OBJECTS_TIMERS=y
CONFIG_DEBUG_OBJECTS_WORK=y
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
CONFIG_DEBUG_OBJECTS_PERCPU_COUNTER=y
CONFIG_DEBUG_KOBJECT_RELEASE=y

( Appending these to the .config and running 'make oldconfig' should enable all of 
  these. )

If the problem is in any of these areas then a debug warning could trigger at a 
more convenient place than some later 'some unrelated bits got corrupted' warning.

Thanks,

	Ingo