linux-kernel - Re: FW: Re: [dm-devel] [BUG 3.12.rc4] Oops: unable to handle kernel paging request during shutdown

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 30 Oct 2013 10:18:44 -0700
From:	Greg KH <greg@...ah.com>
To:	Mikulas Patocka <mpatocka@...hat.com>
Cc:	Alasdair G Kergon <agk@...hat.com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Mike Snitzer <snitzer@...hat.com>, Neil Brown <neilb@...e.de>,
	Frï¿½dï¿½ric Weisbecker <fweisbec@...il.com>,
	Knut Petersen <Knut_Petersen@...nline.de>,
	linux-kernel <linux-kernel@...r.kernel.org>,
	"dm-devel@...hat.com" <dm-devel@...hat.com>,
	Paul McKenney <paulmck@...ux.vnet.ibm.com>,
	Ingo Molnar <mingo@...nel.org>
Subject: Re: FW: Re: [dm-devel] [BUG 3.12.rc4] Oops: unable to handle kernel
 paging request during shutdown

On Wed, Oct 30, 2013 at 12:37:04PM -0400, Mikulas Patocka wrote:
> 
> 
> On Mon, 28 Oct 2013, Alasdair G Kergon wrote:
> 
> > On Fri, Oct 25, 2013 at 11:48 AM, Linus Torvalds
> > <torvalds@...ux-foundation.org> wrote:
> > >
> > > Yes, but nobody has actually been able to trigger it with those. It's
> > > pretty rare, and the debug options are so expensive that they aren't
> > > reasonable to enable generally...
> > >
> > > So we need to try to figure out how to trigger it, or narrow things
> > > down some way..
> > 
> > Ok, still trying to figure this out, and I do have another bug as a
> > result. I don't think this one is really the fundamental one either
> > that caused my crash during "yum upgrade", nor necessarily Knut's
> > problem during shutdown, but I'll keep looking.
> > 
> > And who knows.. Maybe this *does* explain Knut's issue.
> > 
> > Appended is a warning I get with DEBUG_TIMER_OBJECTS. Seems to be a
> > device-mapper issue. Alasdair, Neil, comments? It looks like
> > dm_destroy() is freeing an delayed_work entry that is still active...
> > 
> > I don't know exactly which field in the 'struct mapped_device' has
> > that delayed-work thing, but I assume it's the kobject.. Somebody who
> > knows this code better, please take a look!
> >
> >                    Linus
> 
> No field in device mapper has 'struct delayed_work' in it --- except 
> struct kobject:
> #ifdef CONFIG_DEBUG_KOBJECT_RELEASE
>         struct delayed_work     release;
> #endif
> 
> - so this is kobject manipulation bug. Dm calls dm_sysfs_exit, which calls 
> kobject_put. Dm then frees the structure that contained the kobject with 
> kfree and it triggers this warning.
> 
> Documentation/kobject.txt says that kobjects shouldn't be used this way - 
> that the structure should be freed from the release method. However, using 
> the API the correct way is impossible.
> 
> Unloading of a device driver is supposed to work like this:
> 1) you call the unload routine
> 2) it calls kobject_put (but the kobject may still be referenced by other 
> kernel code release method will be called when the references are dropped)
> 3) you don't free the driver structure
> 4) you exit the unload routing
> 
> ...
> 
> 5) the other references to kobject are dropped
> 6) release method is called, it calls kfree on the driver routine
> 7) the release method exits
> 
> Now, the problem is that between steps 4) and 5), someone may unload the 
> module and trigger the crash. It is impossible to protect against it. 
> 
> Another problem is that between steps 4) and 5), the driver is essentially 
> dead, but it must still respond to attr_show and attr_store methods on the 
> kobect - it is possible to handle it correctly, but it is not easy to test 
> - there is a possibility of a lot of bugs in drivers.
> 
> 
> I suggest that you implement a function kobject_put_free, that decrements 
> the kobject reference count and waits until others stop using the kobject 
> and the reference count drops to zero. Then, you change drivers to use 
> kobject_put_free instead of kobject_put in their unload routine - that 
> will fix this sort of module unload races.

The "module unload" issue is rare, thankfully, but yes, this type of
function will be showing up in 3.13-rc1 through the btrfs tree as it
needs that functionality, so feel free to use it to resolve this issue
if you need it.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/