Message-ID: <4FA345DA4F4AE44899BD2B03EEEC2FA908FF0BA2@SACEXCMBX04-PRD.hq.netapp.com>
Date:	Thu, 27 Sep 2012 21:21:59 +0000
From:	"Myklebust, Trond" <Trond.Myklebust@...app.com>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
CC:	Joerg Roedel <joro@...tes.org>,
	Joerg Roedel <joerg.roedel@....com>,
	"linux-nfs@...r.kernel.org" <linux-nfs@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: kernel BUG at /data/lemmy/linux.trees.git/fs/nfs/idmap.c:681!

On Thu, 2012-09-27 at 09:59 -0700, Linus Torvalds wrote:
> On Thu, Sep 27, 2012 at 9:16 AM, Myklebust, Trond
> <Trond.Myklebust@...app.com> wrote:
> >
> > I cannot see how that BUG_ON can be triggered in the current code, given
> > that the only place where idmap->idmap_key_cons is set to a non-NULL
> > value is covered by a mutex, and that it is always cleared before we
> > release said mutex.
> 
> Quite frankly, the "I cannot see" thing is *never* an excuse for a BUG_ON().
> 
> We don't do kernel-killing asserts in Linux. Never.
> 
> The only excuse for a BUG_ON() is "I cannot possibly continue, I don't
> even have an error path I can take".
> 
> If it's a fundamentally impossible situation, the BUG_ON() should
> never have been there in the first place!
> 
> And if it's a "I don't see how it could happen", then it should have
> been something like
> 
>     if (WARN_ON_ONCE(condition))
>         goto cleanup;
> 
> rather than a BUG_ON().
> 
> We have too many f*cking BUG_ON's in the kernel, and the fact that one
> triggers and it has taken a month and a half without it even being
> resolved is a problem.
> 
> Get rid of the thing, already, dammit. If you cannot figure out how it
> can happen, then the *last* thing you want to do is then kill the
> machine so that it's impossible to debug it sanely.
> 
> Besides, as far as I can tell, idmap_key_cons locking is suspect
> anyway. Stuff like this:
> 
>                 cons = ACCESS_ONCE(idmap->idmap_key_cons);
>                 idmap->idmap_key_cons = NULL;
> 
> is an almost certain example of "the code is racy, and we did it
> wrong". The above is basically *never* correct.
> 
> If the access is properly locked, then the ACCESS_ONCE() is a bug.
> 
> And if the access *isn't* properly locked, then setting things to NULL
> afterwards is in no way safe.
> 
> IOW, either way, it's broken. And there's at least two of those
> clearly buggy code-sequences involving that field.
> 
> So get rid of the BUG_ON() (possibly replacing it with the
> WARN_ON_ONCE), and please look at those ACCESS_ONCE() sequences and
> fix them. Either they happen under a lock, or they don't. None of this
> crazy racy crap, please.

The ACCESS_ONCE is a bug. It isn't needed, since the whole upcall is
covered by the idmap_mutex. I'll remove it and the BUG_ON() come the
merge window (or I can send a patch sooner if you care).

The downcall is ordered w.r.t. idmap_pipe_destroy_msg() (which clears
idmap->idmap_key_cons only on upcall failure). The downcall is also
protected against collisions with idmap_release_pipe() by means of the
inode->i_mutex. Finally, the call to request_key() (during which the
idmap_mutex is held) will always do an uninterruptible wait for one of
those 3 functions to complete.
IOW: all the functions that manipulate idmap->idmap_key_cons are
theoretically ordered w.r.t. each other. That's what I mean when I say
that I really don't understand why this is happening.

So the first thing to do is to try a _vanilla_ 3.6-rc7, and see if the
problem is reproducible without Joerg's extra patches.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@...app.com
www.netapp.com

Powered by blists - more mailing lists