linux-kernel - Re: Crashes in perf_event_ctx_lock

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20171101181126.j727fqjmdthjz4xk@redhat.com>
Date:   Wed, 1 Nov 2017 14:11:27 -0400
From:   Don Zickus <dzickus@...hat.com>
To:     Guenter Roeck <linux@...ck-us.net>
Cc:     Thomas Gleixner <tglx@...utronix.de>,
        Peter Zijlstra <peterz@...radead.org>,
        linux-kernel@...r.kernel.org, Ingo Molnar <mingo@...nel.org>
Subject: Re: Crashes in perf_event_ctx_lock_nested

On Tue, Oct 31, 2017 at 03:11:07PM -0700, Guenter Roeck wrote:
> On Tue, Oct 31, 2017 at 10:32:00PM +0100, Thomas Gleixner wrote:
> 
> [ ...] 
> 
> > So we have to revert
> > 
> > a33d44843d45 ("watchdog/hardlockup/perf: Simplify deferred event destroy")
> > 
> > Patch attached.
> > 
> 
> Tested-by: Guenter Roeck <linux@...ck-us.net>
> 
> There is still a problem. When running
> 
> echo 6 > /proc/sys/kernel/watchdog_thresh
> echo 5 > /proc/sys/kernel/watchdog_thresh
> 
> repeatedly, the message
> 
> NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
> 
> stops after a while (after ~10-30 iterations, with fluctuations).
> After adding trace messages into hardlockup_detector_perf_disable()
> and hardlockup_detector_perf_enable(), I see:
> 
> hardlockup_detector_perf_disable: disable(0): Number of CPUs: 3
> hardlockup_detector_perf_disable: disable(1): Number of CPUs: 2
> hardlockup_detector_perf_disable: disable(2): Number of CPUs: 1
> hardlockup_detector_perf_disable: disable(3): Number of CPUs: 0
> ...
> hardlockup_detector_perf_disable: disable(0): Number of CPUs: 2
> hardlockup_detector_perf_disable: disable(1): Number of CPUs: 1
> hardlockup_detector_perf_disable: disable(2): Number of CPUs: 0
> hardlockup_detector_perf_disable: disable(3): Number of CPUs: -1
> ...
> hardlockup_detector_perf_enable: enable(1): Number of CPUs: -6
> hardlockup_detector_perf_enable: enable(3): Number of CPUs: -5
> hardlockup_detector_perf_enable: enable(2): Number of CPUs: -4
> hardlockup_detector_perf_enable: enable(0): Number of CPUs: -3
> 
> Maybe watchdog_cpus needs to be atomic ?

I switched it to atomic and it solves that problem.  The functionality isn't
broken currently, just the informational message.

Patch attached to try.

Cheers,
Don

---8<--------------

diff --git a/kernel/watchdog_hld.c b/kernel/watchdog_hld.c
index a7f137c1933a..8ee4da223b35 100644
--- a/kernel/watchdog_hld.c
+++ b/kernel/watchdog_hld.c
@@ -14,6 +14,7 @@
 #include <linux/nmi.h>
 #include <linux/module.h>
 #include <linux/sched/debug.h>
+#include <linux/atomic.h>
 
 #include <asm/irq_regs.h>
 #include <linux/perf_event.h>
@@ -25,7 +26,7 @@ static DEFINE_PER_CPU(struct perf_event *, dead_event);
 static struct cpumask dead_events_mask;
 
 static unsigned long hardlockup_allcpu_dumped;
-static unsigned int watchdog_cpus;
+static atomic_t watchdog_cpus = ATOMIC_INIT(0);
 
 void arch_touch_nmi_watchdog(void)
 {
@@ -189,7 +190,8 @@ void hardlockup_detector_perf_enable(void)
 	if (hardlockup_detector_event_create())
 		return;
 
-	if (!watchdog_cpus++)
+	/* use original value for check */
+	if (!atomic_fetch_inc(&watchdog_cpus))
 		pr_info("Enabled. Permanently consumes one hw-PMU counter.\n");
 
 	perf_event_enable(this_cpu_read(watchdog_ev));
@@ -207,7 +209,7 @@ void hardlockup_detector_perf_disable(void)
 		this_cpu_write(watchdog_ev, NULL);
 		this_cpu_write(dead_event, event);
 		cpumask_set_cpu(smp_processor_id(), &dead_events_mask);
-		watchdog_cpus--;
+		atomic_dec(&watchdog_cpus);
 	}
 }