linux-kernel - [PATCH 3/6] dataplane nohz: run softirqs synchronously on user entry

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1431107927-13998-4-git-send-email-cmetcalf@ezchip.com>
Date:	Fri, 8 May 2015 13:58:44 -0400
From:	Chris Metcalf <cmetcalf@...hip.com>
To:	Gilad Ben Yossef <giladb@...hip.com>,
	Steven Rostedt <rostedt@...dmis.org>,
	Ingo Molnar <mingo@...hat.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	"Rik van Riel" <riel@...hat.com>, Tejun Heo <tj@...nel.org>,
	Frederic Weisbecker <fweisbec@...il.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Christoph Lameter <cl@...ux.com>,
	"Srivatsa S. Bhat" <srivatsa.bhat@...ux.vnet.ibm.com>,
	<linux-kernel@...r.kernel.org>
CC:	Chris Metcalf <cmetcalf@...hip.com>
Subject: [PATCH 3/6] dataplane nohz: run softirqs synchronously on user entry

For tasks which have elected dataplane functionality, we run
any pending softirqs for the core before returning to userspace,
rather than ever scheduling ksoftirqd to run.  The problem we
fix is that by allowing another task to run on the core, we
guarantee more interrupts in the future to the dataplane task,
which is exactly what dataplane mode is required to prevent.

This may be an alternate approach to what Mike Galbraith
recently proposed in e.g.:

  https://lkml.org/lkml/2015/3/13/11

Signed-off-by: Chris Metcalf <cmetcalf@...hip.com>
---
 kernel/softirq.c         | 14 +++++++++++++-
 kernel/time/tick-sched.c | 26 +++++++++++++++++++++++++-
 2 files changed, 38 insertions(+), 2 deletions(-)

diff --git a/kernel/softirq.c b/kernel/softirq.c
index 479e4436f787..bc9406337f82 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -291,6 +291,15 @@ restart:
 		    --max_restart)
 			goto restart;
 
+		/*
+		 * For dataplane tasks, waking ksoftirqd because the
+		 * softirqs are slow is a bad idea; we would rather
+		 * synchronously finish whatever is interrupting us,
+		 * and then be able to cleanly enter dataplane mode.
+		 */
+		if (tick_nohz_is_dataplane())
+			goto restart;
+
 		wakeup_softirqd();
 	}
 
@@ -410,8 +419,11 @@ inline void raise_softirq_irqoff(unsigned int nr)
 	 *
 	 * Otherwise we wake up ksoftirqd to make sure we
 	 * schedule the softirq soon.
+	 *
+	 * For dataplane tasks, we will handle the softirq
+	 * synchronously on return to userspace.
 	 */
-	if (!in_interrupt())
+	if (!in_interrupt() && !tick_nohz_is_dataplane())
 		wakeup_softirqd();
 }
 
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 25fdd6bdd1eb..fd0e6e5c931c 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -398,8 +398,26 @@ void __init tick_nohz_init(void)
  */
 void tick_nohz_dataplane_enter(void)
 {
+	/*
+	 * Check for softirqs as close as possible to our return to
+	 * userspace, and run any that are waiting.  We need to ensure
+	 * that we can safely avoid running softirqd, which will cause
+	 * interrupts for nohz_full tasks.  Note that interrupts may
+	 * be enabled internally by do_softirq().
+	 */
+	do_softirq();
+
 	/* Drain the pagevecs to avoid unnecessary IPI flushes later. */
 	lru_add_drain();
+
+	/*
+	 * Disable interrupts again since other code running in this
+	 * function may have enabled them, and the caller expects
+	 * interrupts to be disabled on return.  Enabling them during
+	 * this call is safe since the caller is not assuming any
+	 * state that might have been altered by an interrupt.
+	 */
+	local_irq_disable();
 }
 
 #endif
@@ -771,7 +789,13 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 	if (need_resched())
 		return false;
 
-	if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
+	/*
+	 * If we are running dataplane for this process, don't worry
+	 * about pending softirqs; we will force them to run
+	 * synchronously before returning to userspace.
+	 */
+	if (unlikely(local_softirq_pending() && cpu_online(cpu) &&
+		     !tick_nohz_is_dataplane())) {
 		static int ratelimit;
 
 		if (ratelimit < 10 &&
-- 
2.1.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/