linux-kernel - Re: Intermittent early panic in try_to_wake

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 05 Nov 2009 19:44:36 -0400
From:	Kevin Winchester <kjwinchester@...il.com>
To:	Mike Galbraith <efault@....de>
CC:	Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	LKML <linux-kernel@...r.kernel.org>,
	"Rafael J. Wysocki" <rjw@...k.pl>,
	Steven Rostedt <rostedt@...dmis.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Yinghai Lu <yinghai@...nel.org>
Subject: Re: Intermittent early panic in try_to_wake_up

Mike Galbraith wrote:
> On Thu, 2009-10-22 at 20:33 -0300, Kevin Winchester wrote:
>> Hi,
>>
>> A week or two ago I saw a panic on boot in try_to_wake_up, but it was not
>> reproducible and I had not written down any trace information.  This
>> evening I saw it twice more, but then on the third boot things worked fine.
>> This time I copied down the stack trace:
>>
>> try_to_wake_up+0x2e/0x102
>> wake_up_process+0x10/0x12
>> kthread_create+0x88/0x12c
>> ?ksoftirqd+0x00/0xb7
>> cpu_callback+0x42/0x8f
>> ?spawn_ksoftirqd+0x0/0x39
>> spawn_ksoftirqd+0x17/0x39
>> do_one_initcall+0x58/0x147
>>
>> The first time it happened, I remember checking the git logs and it was
>> shortly after:
>>
>> commit f5dc37530ba8a35aae0f7f4f13781d1904f71e94
>> Author: Mike Galbraith <efault@....de>
>> Date:   Fri Oct 9 08:35:03 2009 +0200
>>
>>     sched: Update the clock of runqueue select_task_rq() selected
>>     
>>     In try_to_wake_up(), we update the runqueue clock, but
>>     select_task_rq() may select a different runqueue than the one we
>>     updated, leaving the new runqueue's clock stale for a bit.
>>     
>>     This patch cures occasional huge latencies reported by latencytop
>>     when coming out of idle on a mostly idle NO_HZ box.
>>     
>>     Signed-off-by: Mike Galbraith <efault@....de>
>>     Signed-off-by: Peter Zijlstra <a.p.zijlstra@...llo.nl>
>>     LKML-Reference: <1255070103.7639.30.camel@...ge.simson.net>
>>     Signed-off-by: Ingo Molnar <mingo@...e.hu>
>>
>>
>> ...so perhaps that has something to do with it.
> 
> I don't think that's very likely.  Box did explode near my grubby
> fingerprints though.
> 
>> Config below.  Any help would be appreciated.
> 
> Building with your config, try_to_wake_up+0x2e is around..
> 
> (gdb) list *try_to_wake_up+0x2e
> 0xffffffff81029107 is in try_to_wake_up (kernel/sched.c:2324).
> 2319            this_cpu = get_cpu();
> 2320
> 2321            smp_wmb();
> 2322            rq = orig_rq = task_rq_lock(p, &flags);
> 2323            update_rq_clock(rq);
> 2324            if (!(p->state & state))
> 2325                    goto out;
> 2326
> 2327            if (p->se.on_rq)
> 2328                    goto out_running;
> 
> I don't see how any of that can explode without something very bad
> having happened to ksoftirqd before we tried to wake it.
> 

I thought this problem had solved itself, but I've hit it again three times
in the last few days.

I've expanded the CC list a little (based on get_maintainer for
kernel/softirq.c, since that seems to be involved somehow, and Rafael since
this definitely seems to be a regression), to see if anyone else has any
ideas.

-- 
Kevin Winchester

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/