Message-ID: <20150313163849.GF5592@linutronix.de>
Date:	Fri, 13 Mar 2015 17:38:49 +0100
From:	Sebastian Andrzej Siewior <bigeasy@...utronix.de>
To:	Brian Silverman <brian@...oton-tech.com>
Cc:	linux-kernel@...r.kernel.org, linux-rt-users@...r.kernel.org,
	Austin Schuh <austin@...oton-tech.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...e.hu>
Subject: Re: CONFIG_PREEMPT_RT local_softirq_pending warning when ISR blocks

* Brian Silverman | 2015-03-09 20:36:27 [-0400]:

>> It looks like your softirq for net_rx is getting a packet and then after
>> raising NET_RX (again?) it blocks on a lock. In order to get this lock
>> it boosts and schedules bash. It gets runable but on the other CPU. On
>> CPU1 there is nothig going is nothing going and the only runable task is
>> the idle thread. And this is probably where the warning is written
>> because we go to idle while we should process a softirq instead.
>
>That sounds like the issue. Doing the softirq instead of going idle in
>this situation seems like it means calling thread_do_softirq() from
>__schedule, but I don't know where the right place is. Can anybody
>give me some help on where exactly to check for softirqs from?

I was slightly wrong.

>     irq/18-can1-7228  [001] d.....2  6854.629276: _raw_spin_lock <-enqueue_to_backlog
>     irq/18-can1-7228  [001] d...1.2  6854.629276: __raise_softirq_irqoff <-enqueue_to_backlog
>     irq/18-can1-7228  [001] d...1.2  6854.629276: do_raise_softirq_irqoff <-__raise_softirq_irqoff
>     irq/18-can1-7228  [001] d...2.2  6854.629276: softirq_raise: vec=3 [action=NET_RX]
>... # continues handling the can1 interrupt
>     irq/18-can1-7228  [001] ......6  6854.629286: rt_spin_lock <-get_page_from_freelist
>     irq/18-can1-7228  [001] ......6  6854.629287: rt_spin_lock_slowlock <-get_page_from_freelist
>     irq/18-can1-7228  [001] ......6  6854.629287: _raw_spin_lock <-rt_spin_lock_slowlock
>     irq/18-can1-7228  [001] ....1.6  6854.629287: __try_to_take_rt_mutex <-rt_spin_lock_slowlock
>     irq/18-can1-7228  [001] ....1.6  6854.629287: _raw_spin_lock_irq <-rt_spin_lock_slowlock
>     irq/18-can1-7228  [001] d...2.6  6854.629288: _raw_spin_unlock_irq <-rt_spin_lock_slowlock
>     irq/18-can1-7228  [001] ....1.6  6854.629288: task_blocks_on_rt_mutex <-rt_spin_lock_slowlock
You raised the softirq but you did not wakeup softirqd. It is expected
that you process softirq(s) in irq-thread context once you leave the
interrupt thread (that is on local_bh_enable() because the thread is run
with BH off).
But this did not happen yet. While you are in your interrupt thread you
got blocked on a lock. And since your CPU is idle otherwise, the
scheduler puts the idle task on.
softirq_check_pending_idle() has a check for this kind of things but
only if the ksoftirqd itself got blocked. In your case it is a process
with BH switched off.
You wouldn't see the warning if you start a task in userland that just
loops and keeps the CPU busy :)

So. One thing I noticed from looking at the code, is that if a thread is
marked IRQF_NO_SOFTIRQ_CALL() then it won't raise process softirqs at
all. This is a bug but since nobody uses IRQF_NO_SOFTIRQ_CALL nobody
noticed it so far (or nobody uses it because it is not working).

And for you. I'm not sure yet what is the best thing to do here. We
could
- teach softirq_check_pending_idle() to ignore these things because
  once the irq thread is unblocked, it will process the softirq.
- utilize the otherwise idle CPU and schedule ksoftirq with the proper
  mask.

Both isn't that easy, I think…

>Thanks,
>Brian

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists