Message-ID: <20160120041613.GA25878@localhost.localdomain>
Date:	Tue, 19 Jan 2016 21:16:14 -0700
From:	Jeff Merkey <linux.mdb@...il.com>
To:	linux-kernel@...r.kernel.org
Cc:	John Stultz <john.stultz@...aro.org>,
	Thomas Gleixner <tglx@...utronix.de>
Subject: [PATCH v2] Fix Hard Lockup in ktime_get_ts64 with debuggers

This patch fixes a hard system lockup that occurs when a system has been
held by a kernel debugger such as kgdb, kdb, or mdb on a multiprocessor
system with all processors suspended then released after 20-30 minutes
of debugging.

SUMMARY

An SMP multiprocessor capable kernel debugger must stop all processors
on a target system and trigger them to be held inside a NMI polling
routine while tracing and debugging active events on a target system.
This is typically accomplished by triggering an NMI interrupt via
an IPI call to each processor then directing the processors to enter
a suspend code loop from the NMI handlers.  While the processors
are held suspended, the system hardware timer will continue
to run as well as other hardware components.  Upon exiting
the debugger console, the processors are released and the system
is expected to resume normal operations.  In the case that causes this
bug, the Linux timekeeper subsystem sees a very large delta value
for the time in nanoseconds elapsed then attempts to process
this large number.  Since the function macro timespec_add_ns calls
an inline macro function that manually subtracts the values longhand
rather than use the processor to perform the divide operation, this
results in the hard lockup detector causing a panic because the function
is looping and hung for about 4 minutes on average calling this
longhand divide macro __iter_div_u64_rem.

CONDITIONS THAT RESULT IN THIS LOCKUP

Holding the system processors suspended in the NMI handlers for
20 minutes to several hours while examining and debugging a target
system will trigger this bug upon exit from the debugger console
on either kgdb, kdb, or mdb.  It appears to be caused by the delta
shift of the counters in the hardware clock in ns being calculated
and passed to the function to update the timespec.  The function being
replaced is little more than a cycle hog on the system and needs to be
replaced with something that does not trigger the hard lockup detector
if it sees a really big number for nanoseconds.

Signed-off-by: Jeff Merkey <linux.mdb@...il.com>
---
 include/linux/math64.h | 17 +++++++++++++++++
 include/linux/time.h   |  2 +-
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/include/linux/math64.h b/include/linux/math64.h
index 6e8b5b2..6769a3d 100644
--- a/include/linux/math64.h
+++ b/include/linux/math64.h
@@ -22,6 +22,15 @@ static inline u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder)
 }
 
 /**
+ * div_s64_rem64 - unsigned 64bit divide with 32bit divisor with 64bit remainder
+ */
+static inline u64 div_u64_rem64(u64 dividend, u32 divisor, u64 *remainder)
+{
+	*remainder = dividend % divisor;
+	return dividend / divisor;
+}
+
+/**
  * div_s64_rem - signed 64bit divide with 32bit divisor with remainder
  */
 static inline s64 div_s64_rem(s64 dividend, s32 divisor, s32 *remainder)
@@ -68,6 +77,14 @@ static inline u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder)
 }
 #endif
 
+#ifndef div_u64_rem64
+static inline u64 div_u64_rem64(u64 dividend, u32 divisor, u64 *remainder)
+{
+	*remainder = do_div(dividend, divisor);
+	return dividend;
+}
+#endif
+
 #ifndef div_s64_rem
 extern s64 div_s64_rem(s64 dividend, s32 divisor, s32 *remainder);
 #endif
diff --git a/include/linux/time.h b/include/linux/time.h
index 297f09f..e9e1fbc 100644
--- a/include/linux/time.h
+++ b/include/linux/time.h
@@ -258,7 +258,7 @@ extern struct timeval ns_to_timeval(const s64 nsec);
  */
 static __always_inline void timespec_add_ns(struct timespec *a, u64 ns)
 {
-	a->tv_sec += __iter_div_u64_rem(a->tv_nsec + ns, NSEC_PER_SEC, &ns);
+	a->tv_sec += div_u64_rem64(a->tv_nsec + ns, NSEC_PER_SEC, &ns);
 	a->tv_nsec = ns;
 }
 
-- 
1.8.3.1

Powered by blists - more mailing lists