linux-kernel - Re: Rescheduling interrupts

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <200801221455.16814.caglar@pardus.org.tr>
Date:	Tue, 22 Jan 2008 14:55:14 +0200
From:	"S.Çağlar Onur" <caglar@...dus.org.tr>
To:	Ingo Molnar <mingo@...e.hu>
Cc:	LKML <linux-kernel@...r.kernel.org>,
	Arjan van de Ven <arjan@...ux.intel.com>
Subject: Re: Rescheduling interrupts

Hi;

22 Oca 2008 Sal tarihinde, Ingo Molnar şunları yazmıştı: 
> * S.Çağlar Onur <caglar@...dus.org.tr> wrote:
> 
> > Top causes for wakeups:
> >   59,9% (238,4)      <kernel IPI> : Rescheduling interrupts
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >   14,7% ( 58,6)         amarokapp : schedule_timeout (process_timeout)
> 
> hm, would be nice to figure out what causes these IPIs. Could you stick 
> something like this into arch/x86/kernel/smp_32.c's 
> smp_send_reschedule() function [this is the function that generates the 
> IPI]:
> 
> static void native_smp_send_reschedule(int cpu)
> {
>         WARN_ON(cpu_is_offline(cpu));
>         send_IPI_mask(cpumask_of_cpu(cpu), RESCHEDULE_VECTOR);
> 	if (panic_timeout > 0) {
> 		panic_timeout--;
> 		printk("IPI from task %s:%d on CPU#%d:\n", 
> 			current->comm, current->pid, cpu);
> 		dump_stack();
> 	}
> }
> 
> NOTE: if you run an SMP kernel then first remove these two lines from 
> kernel/printk.c:
> 
>         if (!oops_in_progress && waitqueue_active(&log_wait))
>                 wake_up_interruptible(&log_wait);
> 
> otherwise you'll get lockups. (the IPI is sent while holding the 
> runqueue lock, so the printks will lock up)
> 
> then wait for the bad condition to occur on your system and generate a 
> stream of ~10 backtraces, via:
> 
> 	echo 10 > /proc/sys/kernel/panic
> 
> you should be getting 10 immediate backtraces - please send them to us. 
> The backtraces should show the place that generates the wakeups. [turn 
> on CONFIG_FRAME_POINTERS=y to get high quality backtraces.]
>  
> If you do _not_ get 10 immediate backtraces, then something in the 
> system is generating such IPIs outside of the scheduler's control. That 
> would suggest some other sort of borkage.
> 
> 	Ingo

I grabbed the logs two times to make sure to catch needed info. 1st [1] one is generated while "Rescheduling interrupts" wakeups ~200 times and 2nd one generated for ~350 wakeups.

[1] http://cekirdek.pardus.org.tr/~caglar/dmesg.1st 
[2] http://cekirdek.pardus.org.tr/~caglar/dmesg.2nd

Cheers
-- 
S.Çağlar Onur <caglar@...dus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/