Subject: softirq: Allow early termination of softirq processing From: Peter Zijlstra Date: Fri Sep 11 17:50:17 CEST 2020 From: Peter Zijlstra Invidiual soft interrupts can run longer than the timeout, but the loop termination conditions (timeout or need_resched()) are only evaluated after processing all pending bits. Check the timeout condition after each processed pending bit and allow to terminate the loop early. The remaining unprocessed bits are or'ed back into local_softirq_pending() need_resched() is not checked after each bit because it's likely to be set after one of the first bits and way before the timeout condition happens. While checking need_resched() after each bit is reducing latency for the particular softirq vector which ended up setting TIF_NEEDRESCHED, it's counterproductive in terms of batch processing. The main objective of this change is to keep fairness intact. Adding need_resched() to the inner termination check needs to be carefully evaluated and done in a seperate change if at all. Caveat: Folding back the not yet processed bits without further handling is bound to cause starvation of the higher numbered softirq vectors when the lower ones which triggered the timeout are raised again before the remaining bits have been processed in the next run. This will be handled in a follow up change. [ tglx: Adopt to previous changes, use only the timeout condition in the loop and prevent wakeup of ksoftirqd in case there are no new bits pending ] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner --- kernel/softirq.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -304,20 +304,26 @@ asmlinkage __visible void __softirq_entr prev_count, preempt_count()); preempt_count_set(prev_count); } + if (__softirq_timeout(tbreak)) + break; } if (__this_cpu_read(ksoftirqd) == current) rcu_softirq_qs(); local_irq_disable(); - pending = local_softirq_pending(); if (pending) { + or_softirq_pending(pending); + } else { + pending = local_softirq_pending(); + if (!pending) + goto out; if (!__softirq_needs_break(tbreak) && --max_restart) goto restart; - - wakeup_softirqd(); } + wakeup_softirqd(); +out: lockdep_softirq_end(in_hardirq); account_irq_exit_time(current); __local_bh_enable(SOFTIRQ_OFFSET);