linux-kernel - Re: [bisected] Clocksource tsc unstable git

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20101109125842.GA8636@osiris.boeblingen.de.ibm.com>
Date:	Tue, 9 Nov 2010 13:58:42 +0100
From:	Heiko Carstens <heiko.carstens@...ibm.com>
To:	Peter Zijlstra <a.p.zijlstra@...llo.nl>
Cc:	markus@...ppelsdorf.de, john stultz <johnstul@...ibm.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Borislav Petkov <bp@...64.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"hpa@...ux.intel.com" <hpa@...ux.intel.com>,
	Ingo Molnar <mingo@...e.hu>,
	Andreas Herrmann <andreas.herrmann3@....com>, avi@...hat.com,
	mtosatti@...hat.com
Subject: Re: [bisected] Clocksource tsc unstable git

On Wed, Oct 27, 2010 at 09:36:55PM +0200, Peter Zijlstra wrote:
> On Wed, 2010-10-27 at 20:26 +0200, markus@...ppelsdorf.de wrote:
> > 
> > 34f971f6f7988be4d014eec3e3526bee6d007ffa is the first bad commit
> > commit 34f971f6f7988be4d014eec3e3526bee6d007ffa
> > Author: Peter Zijlstra <a.p.zijlstra@...llo.nl>
> > Date:   Wed Sep 22 13:53:15 2010 +0200
> > 
> >     sched: Create special class for stop/migrate work
> > 
> >     In order to separate the stop/migrate work thread from the SCHED_FIFO
> >     implementation, create a special class for it that is of higher priority than
> >     SCHED_FIFO itself.
> > 
> >     This currently solves a problem where cpu-hotplug consumes so much cpu-time
> >     that the SCHED_FIFO class gets throttled, but has the bandwidth replenishment
> >     timer pending on the now dead cpu.
> > 
> >     It is also required for when we add the planned deadline scheduling class above
> >     SCHED_FIFO, as the stop/migrate thread still needs to transcent those tasks.
> > 
> >     Tested-by: Heiko Carstens <heiko.carstens@...ibm.com>
> >     Signed-off-by: Peter Zijlstra <a.p.zijlstra@...llo.nl>
> >     LKML-Reference: <1285165776.2275.1022.camel@...top>
> >     Signed-off-by: Ingo Molnar <mingo@...e.hu>
> > 
> > Reverting the commit solves the kvm hang issue.
> > (If this issue is related to my original tsc problem is of course open for
> > debate, but I have a strong hunch it is.) 
> 
> Too weird,.. what does the hang look like?
> 
> Can you generate a sysrq-t dump? The thing I'm looking for is the
> migration/# thread being runnable but not being current.
> 
> How can I reproduce this?

I think there is a bug in pick_next_task_stop() in sched_stopclass.c:

If a stop-task scheduling class task (well... the migration thread ;) sets
its state to TASK_INTERRUPTIBLE and then gets preempted it will never
scheduled again, because pick_next_task_stop() ignores all tasks with a
state != TASK_RUNNING:

static struct task_struct *pick_next_task_stop(struct rq *rq)
{
	struct task_struct *stop = rq->stop;

	if (stop && stop->state == TASK_RUNNING)
		return stop;

	return NULL;
}

At least I have two dumps of machines where all cpus but one are running
the migration thread. Only the last one runs usual user space processes but
has a preempted migration thread:

>> trace 0x7e50cd40
================================================================
STACK TRACE FOR TASK: 0x7e50cd40 (migration/9)

 STACK:
 0 schedule+1204 [0x55f754]
 1 preempt_schedule+102 [0x56031e]
 2 _raw_spin_unlock_irq+118 [0x563fa6]
 3 cpu_stopper_thread+144 [0x1ad854]
 4 kthread+166 [0x1685aa]
 5 kernel_thread_starter+6 [0x106bea]
================================================================

struct task_struct {
        state = 0x1  /* TASK_INTERRUPTIBLE */
...

I would guess something like the below would probably fix it.
Does that make any sense or did I miss something obvious?

diff --git a/kernel/sched_stoptask.c b/kernel/sched_stoptask.c
index 45bddc0..d5cf344 100644
--- a/kernel/sched_stoptask.c
+++ b/kernel/sched_stoptask.c
@@ -26,7 +26,8 @@ static struct task_struct *pick_next_task_stop(struct rq *rq)
 {
 	struct task_struct *stop = rq->stop;
 
-	if (stop && stop->state == TASK_RUNNING)
+	/* got preempted if on_rq == 1 -- ignore */
+	if (stop && ((stop->state == TASK_RUNNING) || stop->se.on_rq))
 		return stop;
 
 	return NULL;
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/