linux-kernel - Re: [RFC PATCH 0/6] CPU hotplug: Reverse invocation of notifiers during CPU hotplug

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20120725164331.GB2378@linux.vnet.ibm.com>
Date:	Wed, 25 Jul 2012 09:43:32 -0700
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Alan Stern <stern@...land.harvard.edu>
Cc:	"Srivatsa S. Bhat" <srivatsa.bhat@...ux.vnet.ibm.com>,
	tglx@...utronix.de, mingo@...nel.org, peterz@...radead.org,
	rusty@...tcorp.com.au, namhyung@...nel.org, tj@...nel.org,
	rjw@...k.pl, nikunj@...ux.vnet.ibm.com, linux-pm@...r.kernel.org,
	linux-kernel@...r.kernel.org
Subject: Re: [RFC PATCH 0/6] CPU hotplug: Reverse invocation of notifiers
 during CPU hotplug

On Wed, Jul 25, 2012 at 12:10:41PM -0400, Alan Stern wrote:
> On Wed, 25 Jul 2012, Srivatsa S. Bhat wrote:
> 
> > On 07/25/2012 08:27 PM, Alan Stern wrote:
> > > On Wed, 25 Jul 2012, Srivatsa S. Bhat wrote:
> > > 
> > >> Hi,
> > >>
> > >> This patchset implements the approach of invoking the CPU hotplug callbacks
> > >> (notifiers) in one order during CPU online and in the reverse order during CPU
> > >> offline. The rationale behind this is that services for a CPU are started in a
> > >> particular order (perhaps, with implicit dependencies between them) while
> > >> bringing up the CPU, and hence, it makes sense to tear down the services in
> > >> the opposite order, thereby honoring most of the dependencies automatically
> > >> (and also correctly). This is explained in more detail in Patch 6.
> > > 
> > > This strongly suggests that a notifier chain may be the wrong mechanism
> > > to use here.  Notifiers provide only limited guarantees about ordering,
> > > and it's hard to say much about the services a particular chain will
> > > provide since callbacks can be added from anywhere.
> > > 
> > 
> > True, the ability to register any random callback from anywhere is still a
> > problem that we are fighting... The zillions of callbacks that we have today
> > makes the hotplug process quite entangled.. we can't even roll-back from a
> > failure easily!
> > 
> > > Instead of adding all this complication to the notifier mechanism, how 
> > > about using something else for CPU hotplug?
> > > 
> > 
> > The problem is that today, many different subsystems need to know about CPUs coming
> > up or going down.. And CPU hotplug is not atomic, it happens in stages, and the
> > coordination between those subsystems is what actually drives CPU hotplug, in a way.
> 
> All this reinforces the idea that notifiers are the wrong mechanism for 
> CPU hotplug.
> 
> > At present, I think that the best we can do is to redesign the hotplug code such that
> > the number of callbacks that are needed can be reduced to a minimum amount and then
> > have good control over what those callbacks do. For example, Thomas Gleixner posted
> > the park/unpark patchset[1], which not only speeds-up CPU hotplug by avoiding destruction
> > and creation of per-cpu kthreads on every hotplug operation, but also gets rid of quite
> > a few notifiers by providing a framework to manage those per-cpu kthreads...
> 
> I think the best you can do is stop using notifiers and use something 
> else instead.  For example, a simple set of function calls (assuming 
> you know beforehand what callbacks need to be invoked).

Unfortunately, we don't know beforehand.  There are loadable kernel
modules that need to be informed of CPU hotplug events.  So we do need
some runtime way to register to be notified CPU-hotplug events.

If it weren't for the modules, we might be able to use another data
section and get a contiguous list of functions that need to be called.
I suppose we could have such sections for modules and link/unlink them
at module load/unload time, but it is not at all clear to me that this
would be simpler than using notifiers.

But always happy to learn -- any specific suggestions?

> > One of the other ideas to improve the hotplug notifier stuff that came up during some
> > of the discussions was to implement explicit dependency tracking between the notifiers
> > and perhaps get rid of the priority numbers that are currently being used to provide
> > some sort of ordering between the callbacks. Links to some of the related discussions
> > are provided below.
> 
> This seems like misplaced over-engineering.

In fact, we have deferred dependency tracking.  If I remember correctly,
the main motivation was to execute notifiers in parallel, but the fraction
of time that hotplug spends executing notifiers proved to be quite small.
At this point, I would be surprised if we ever do much with dependency
tracking.

Again, any specific suggestions for improvement?

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/