linux-kernel - Re: timer code oops when calling mod_delayed

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20151031175404.3c57a17a@tlielax.poochiereds.net>
Date:	Sat, 31 Oct 2015 17:54:04 -0400
From:	Jeff Layton <jlayton@...chiereds.net>
To:	Tejun Heo <tj@...nel.org>
Cc:	linux-kernel@...r.kernel.org, bfields@...ldses.org,
	Michael Skralivetsky <michael.skralivetsky@...marydata.com>,
	Chris Worley <chris.worley@...marydata.com>,
	Trond Myklebust <trond.myklebust@...marydata.com>,
	Lai Jiangshan <laijs@...fujitsu.com>
Subject: Re: timer code oops when calling mod_delayed_work

On Sat, 31 Oct 2015 17:31:07 -0400
Tejun Heo <tj@...nel.org> wrote:

> Hello, Jeff.
> 
> On Sat, Oct 31, 2015 at 07:34:00AM -0400, Jeff Layton wrote:
> > > Heh, this one is tricky.  Yeah, try_to_grab_pending() missing PENDING
> > > would explain the failure but I can't see how it'd leak at the moment.
> > 
> > Thanks Tejun. Yeah, I realized that after sending the response above.
> > 
> > If you successfully delete the timer the timer then the PENDING bit
> > should already be set. Might be worth throwing in something like this,
> > just before the return 1:
> > 
> >     WARN_ON(!test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)))
> >
> > ...but I doubt it would fire. I think it's likely that the bug is
> > elsewhere.
> 
> I doubt that'd trigger either but it'd still be a good idea to throw
> that in just in case.
> 
> > The other thing is that we've had this code in place for a couple of
> > years now, and this is the first time I've seen an oops like this. I
> > suspect that this may be a recent regression, but I don't know that for
> > sure.
> 
> I can't think of any recent changes which could affect this.  This
> part of code has remained unchanged for years at least from the
> workqueue side.
> 
> > I have asked Chris and Michael to see if they can bisect it down, but
> > it may be a bit before they can get that done. Any insight you might
> > have in the meantime would helpful.
> 
> Yeah, I'd love to find out how reproducible the issue is.  If the
> problem is rarely reproducible, it might make sense to try
> instrumentation before trying bisection as it *could* be a latent bug
> which has been there all along and bisecting to the commit introducing
> the code wouldn't help us too much.
> 

It seems fairly reproducible, at least on v4.3-rc7 kernels:

This came about when I asked them to perf test some nfsd patches that I
have queued up. I patched a Fedora 4.3-rc7 kernel and wanted to see
what the perf delta was (with NFSv3, fwiw):

    Patched kernels here: http://koji.fedoraproject.org/koji/taskinfo?taskID=11598089

    Unpatched kernels here: http://koji.fedoraproject.org/koji/buildinfo?buildID=694377

Michael was using the SPEC SFS VDI workload to test, and was able to
get the same panic on both kernels. So it does seem to be reproducible.
It might even be possible to tune the VM to make the shrinker fire more
often, which may help tickle this more.

In any case, I've asked them to try something v4.2-ish and see if it's
reproducible there, and then try v4.1 if it is. I figure anything
earlier is probably not worth testing if it still fails on v4.1. If it
turns out not to be reproducible on those earlier kernels then we can
bisect from there to track it down.

Thanks again!
-- 
Jeff Layton <jlayton@...chiereds.net>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/