linux-kernel - Re: [patch 10/29] lockup_detector/perf: Prevent cpu hotplug deadlock

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20170905145100.fudvvm3udlcwqioh@redhat.com>
Date:   Tue, 5 Sep 2017 10:51:00 -0400
From:   Don Zickus <dzickus@...hat.com>
To:     Thomas Gleixner <tglx@...utronix.de>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...nel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Borislav Petkov <bp@...en8.de>,
        Sebastian Siewior <bigeasy@...utronix.de>,
        Nicholas Piggin <npiggin@...il.com>,
        Chris Metcalf <cmetcalf@...lanox.com>,
        Ulrich Obergfell <uobergfe@...hat.com>
Subject: Re: [patch 10/29] lockup_detector/perf: Prevent cpu hotplug deadlock

On Fri, Sep 01, 2017 at 09:29:07PM +0200, Thomas Gleixner wrote:
> On Fri, 1 Sep 2017, Don Zickus wrote:
> > On Thu, Aug 31, 2017 at 09:16:08AM +0200, Thomas Gleixner wrote:
> > > The following deadlock is possible in the watchdog hotplug code:
> > > 
> > >   cpus_write_lock()
> > >     ...
> > >       takedown_cpu()
> > >         smpboot_park_threads()
> > >           smpboot_park_thread()
> > >             kthread_park()
> > >               ->park() := watchdog_disable()
> > >                 watchdog_nmi_disable()
> > >                   perf_event_release_kernel();
> > >                     put_event()
> > >                       _free_event()
> > >                         ->destroy() := hw_perf_event_destroy()
> > >                           x86_release_hardware()
> > >                             release_ds_buffers()
> > >                               get_online_cpus()
> > > 
> > > when a per cpu watchdog perf event is destroyed which drops the last
> > > reference to the PMU hardware. The cleanup code there invokes
> > > get_online_cpus() which instantly deadlocks because the hotplug percpu
> > > rwsem is write locked.
> > 
> > The main reason perf_event_release_kernel is in this path is because the
> > oprofile folks complained they couldn't use the perf counters when the
> > nmi_watchdog was disabled on the command line.
> 
> If the nmi watchdog is disabled on the command line then there are no
> counters claimed at all.

Ah, I see it now.  When you park all the threads, you clear the cpumask
which then calls the lockup detector_cleanup to release all perf_counters
that are no longer used by the cpumask.

Further reading this code, I see how the code covers various race conditions
we tried solving in the past.

Thanks!

Cheers,
Don