linux-kernel - Re: [BUGREPORT] Tasklet scheduled issue in Linux 3.4.x-rt

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20140328163753.GB14842@linutronix.de>
Date:	Fri, 28 Mar 2014 17:37:53 +0100
From:	Sebastian Andrzej Siewior <bigeasy@...utronix.de>
To:	Yijing Wang <wangyijing@...wei.com>
Cc:	Ingo Molnar <mingo@...nel.org>,
	Peter Zijlstra <peterz@...radead.org>,
	LKML <linux-kernel@...r.kernel.org>,
	linux-rt-users@...r.kernel.org, Li Zefan <lizefan@...wei.com>,
	Zhangwei <jovi.zhangwei@...wei.com>
Subject: Re: [BUGREPORT] Tasklet scheduled issue in Linux 3.4.x-rt

* Yijing Wang | 2014-03-03 17:24:39 [+0800]:

>[2012-03-26 18:55:43][  929.252312] WARNING: at kernel/softirq.c:773 __tasklet_action+0x51/0x1a0()
>[2012-03-27 03:41:06][ 3647.886005] WARNING: at kernel/softirq.c:773 __tasklet_action+0x51/0x1a0()
>[2012-03-27 03:42:04][ 3705.434418] WARNING: at kernel/softirq.c:799 __tasklet_action+0xae/0x1a0()

>FC card hardware  -------> FC driver interrupt handler  --------->tasklet_schedule(fc driver tasklet) ------->tasklet running, call function process FC IO data.
>                        here will disable FC card interrupt                                                     here will enable FC card interrupt again

This looks okay.

>We found the tasklet state is 0x1(mean state is TASKLET_STATE_SCHED),count is 0, before we call tasklet_schedule().
>So the new tasklet can not add to CPU list.
>
>And I also add some dynamic debug in __tasklet_action(); after the issue occur, I open the dynamic debug.
>After we force the hardware reset to interrupt OS, we never found the FC driver tasklet running in dmesg(I identify the tasklet by its data).
>I guess the FC tasklet is not in CPU global tasklet list.
You guess correct.

>I hope somebody can help to look at it. If I missing something, let me know.

The tasklet is always added to the local cpu, never cross. That list is
always accessed with interrupts off.
With TASKLET_STATE_SCHED set, the next step is to add the task let to
the CPU's tasklet list. This isn't done if TASKLET_STATE_RUN is already
set which means __tasklet_action() is already busy serving the tasklet.
In that case it clears TASKLET_STATE_SCHED and invokes the tasklet
again.
After looking at it for a while I must say I have no idea how you
managed to keep TASKLET_STATE_SCHED set. Further, each time
TASKLET_STATE_RUN is cleared it is always with a cmpxchg() down to zero
which means TASKLET_STATE_SCHED is removed earlier.
That said, triggerring the warning at 773 is the first thing that went
wrong. After it has been added to the list, the TASKLET_STATE_RUN is
cleared again. I have no idea how it managed to remain still on except
that __tasklet_common_schedule() is invoked which is protected by the
SCHED bit…

>Thanks!
>Yijing.

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/