linux-kernel - Re: Rescheduling interrupts

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20080122105741.GG21149@elte.hu>
Date:	Tue, 22 Jan 2008 11:57:41 +0100
From:	Ingo Molnar <mingo@...e.hu>
To:	S.Çağlar Onur <caglar@...dus.org.tr>
Cc:	LKML <linux-kernel@...r.kernel.org>,
	Arjan van de Ven <arjan@...ux.intel.com>
Subject: Re: Rescheduling interrupts

* S.Çağlar Onur <caglar@...dus.org.tr> wrote:

> Top causes for wakeups:
>   59,9% (238,4)      <kernel IPI> : Rescheduling interrupts
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   14,7% ( 58,6)         amarokapp : schedule_timeout (process_timeout)

hm, would be nice to figure out what causes these IPIs. Could you stick 
something like this into arch/x86/kernel/smp_32.c's 
smp_send_reschedule() function [this is the function that generates the 
IPI]:

static void native_smp_send_reschedule(int cpu)
{
        WARN_ON(cpu_is_offline(cpu));
        send_IPI_mask(cpumask_of_cpu(cpu), RESCHEDULE_VECTOR);
	if (panic_timeout > 0) {
		panic_timeout--;
		printk("IPI from task %s:%d on CPU#%d:\n", 
			current->comm, current->pid, cpu);
		dump_stack();
	}
}

NOTE: if you run an SMP kernel then first remove these two lines from 
kernel/printk.c:

        if (!oops_in_progress && waitqueue_active(&log_wait))
                wake_up_interruptible(&log_wait);

otherwise you'll get lockups. (the IPI is sent while holding the 
runqueue lock, so the printks will lock up)

then wait for the bad condition to occur on your system and generate a 
stream of ~10 backtraces, via:

	echo 10 > /proc/sys/kernel/panic

you should be getting 10 immediate backtraces - please send them to us. 
The backtraces should show the place that generates the wakeups. [turn 
on CONFIG_FRAME_POINTERS=y to get high quality backtraces.]

If you do _not_ get 10 immediate backtraces, then something in the 
system is generating such IPIs outside of the scheduler's control. That 
would suggest some other sort of borkage.

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/