linux-kernel - Re: [RFC] cpu/hotplug: Modify lock status before making cpu hotplug callbacks

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <alpine.DEB.2.20.1706121653140.1911@nanos>
Date:   Mon, 12 Jun 2017 16:59:13 +0200 (CEST)
From:   Thomas Gleixner <tglx@...utronix.de>
To:     John Allen <jallen@...ux.vnet.ibm.com>
cc:     LKML <linux-kernel@...r.kernel.org>,
        Nathan Fontenot <nfont@...ux.vnet.ibm.com>,
        Michael Bringmann <mwb@...ux.vnet.ibm.com>,
        Peter Zijlstra <peterz@...radead.org>
Subject: Re: [RFC] cpu/hotplug: Modify lock status before making cpu hotplug
 callbacks

On Wed, 7 Jun 2017, John Allen wrote:
> A deadlock has been observed during a cpu hot add on powerpc machines.
> 
> The situation is as follows:
> First, in _cpu_up, we take the cpu_hotplug lock and towards the end of _cpu_up,
> we make the cpu hotplug callbacks, one of which is arch_update_cpu_topology.
> For most other architectures, making this call while the parent thread has the
> cpu_hotplug lock seems harmless as most implementations of
> arch_update_cpu_topology appear to be quite simple. However, the powerpc
> implementation is significantly more complex and requires us to make a call to
> stop_machine which in turn attempts to take the cpu_hotplug lock in
> get_online_cpus. We then deadlock as the parent thread is waiting on the child
> to complete and the child is waiting on the parent to release the lock.
> 
> This solution attempts to resolve the issue by incrementing the cpu_hotplug
> refcount and releasing the cpu_hotplug mutex in _cpu_up so that the subsequent
> call to get_online_cpus can take the mutex while other invocations of _cpu_up
> are still prevented from executing as the refcount is non-zero. From my testing,
> this seems to resolve the deadlock, but I'm not sure if there is a reason that
> get_online_cpus *should* be locked out at this point.

This is horrible, really.

We have code pending in -next which reworks the hotplug lock to a per cpu
reader writer semaphore, so this hack is not going to work anymore.

See git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git smp/hotplug

This addresses the issue of calling stomp_machine() from inside a hotplug
locked region, by introducing stop_machine_cpuslocked(). That should solve
your problem nicely.

Thanks,

	tglx