Message-ID: <20140809164119.GI21930@wotan.suse.de>
Date:	Sat, 9 Aug 2014 18:41:19 +0200
From:	"Luis R. Rodriguez" <mcgrof@...e.com>
To:	David Miller <davem@...emloft.net>, gregkh@...uxfoundation.org
Cc:	mcgrof@...not-panic.com, tiwai@...e.de,
	linux-kernel@...r.kernel.org, penguin-kernel@...ove.SAKURA.ne.jp,
	joseph.salisbury@...onical.com, kay@...y.org,
	gnomes@...rguk.ukuu.org.uk, tim.gardner@...onical.com,
	pierre-fersing@...rref.org, akpm@...ux-foundation.org,
	oleg@...hat.com, bpoirier@...e.de,
	nagalakshmi.nandigama@...gotech.com,
	praveen.krishnamoorthy@...gotech.com,
	sreekanth.reddy@...gotech.com, abhijit.mahajan@...gotech.com,
	hariprasad@...lsio.com, santosh@...lsio.com,
	MPT-FusionLinux.pdl@...gotech.com, linux-scsi@...r.kernel.org,
	netdev@...r.kernel.org
Subject: Re: [PATCH v2 2/4] driver core: enable drivers to use deferred
	probe from init

On Wed, Jul 30, 2014 at 03:11:07PM -0700, David Miller wrote:
> From: "Luis R. Rodriguez" <mcgrof@...not-panic.com>
> Date: Mon, 28 Jul 2014 11:28:28 -0700
> 
> > Tetsuo bisected and found that commit 786235ee "kthread: make
> > kthread_create() killable" modified kthread_create() to bail as
> > soon as SIGKILL is received. This is causing some issues with
> > some drivers and at times boot. Joseph then found that failures
> > occur as the systemd-udevd process sends SIGKILL to modprobe if
> > probe on a driver takes over 30 seconds. When this happens probe
> > will fail on any driver, its why booting on some system will fail
> > if the driver happens to be a storage related driver. Some folks
> > have suggested fixing this by modifying kthread_create() to not
> > leave upon SIGKILL [3], upon review Oleg rejected this change and
> > the discussion was punted out to systemd to see if the default
> > timeout could be increased from 30 seconds to 120. The opinion of
> > the systemd maintainers is that the driver's behavior should
> > be fixed [4]. Linus seems to agree [5], however more recently even
> > networking drivers have been reported to fail on probe since just
> > writing the firmware to a device and kicking it can take easy over
> > 60 seconds [6]. Benjamim was able to trace the issues recently
> > reported on cxgb4 down to the same systemd-udevd 30 second timeout [6].
> > 
> > This is an alternative solution which enables drivers that are
> > known to take long to use deferred probe workqueue. This avoids
> > the 30 second timeout and lets us annotate drivers with long
> > init sequences.
> > 
> > As drivers determine a component is not yet available and needs
> > to defer probe you'll be notified this happen upon init for each
> > device but now with a message such as:
> > 
> > pci 0000:03:00.0: Driver cxgb4 requests probe deferral on init
> > 
> > You should see one of these per struct device probed.
> 
> It seems we're still discussing this.
> 
> I think I understand all of the underlying issues, and what I'll say
> is that perhaps we should use what Greg KH requested but via a helper
> that is easy to grep for.
> 
> I don't care if it's something like "module_long_probe_init()" and
> "module_long_probe_exit()", but it just needs to be some properly
> named interface which does the whole kthread or whatever bit.

I've tested the alternative kthread_run() proposal but unfortunately it
does not help resolve the issue, the timeout is still hit and a SIGKILL
still kills the driver probe. Please let me know how you'd all like us
to proceed, these defer probe patches do help cure the issue though.

I should also note that these work around patches can only be done once we
already know a driver fails to go over the timeout, root causing and
associating driver issues to the timeout has been very difficult with a few
drivers already, for this reason I've submitted a change for systemd to issue a
warning instead of killing kmod usage on udev after a timeout, that would make
this regression non-fatal, and let us more easily then hunt drivers that need
fixing much easily [0] [1]. As noted we'd still want to have drivers easily
annotated which require fixing, this orignal series would allow us to do that
by hunting for delay_probe. If there alternative and preferred strategies
please let me know.

[0] http://lists.freedesktop.org/archives/systemd-devel/2014-August/021812.html
[1] http://lists.freedesktop.org/archives/systemd-devel/2014-August/021821.html

  Luis
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists