linux-kernel - Re: [BUG 2.6.25-rc3] scheduler/hotplug: some processes are dealocked when cpu is set to offline

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20080307155002.GA22664@in.ibm.com>
Date:	Fri, 7 Mar 2008 21:20:02 +0530
From:	Gautham R Shenoy <ego@...ibm.com>
To:	Dmitry Adamushko <dmitry.adamushko@...il.com>
Cc:	Ingo Molnar <mingo@...e.hu>, Oleg Nesterov <oleg@...sign.ru>,
	Yi Yang <yi.y.yang@...el.com>, akpm@...ux-foundation.org,
	linux-kernel@...r.kernel.org, "Rafael J. Wysocki" <rjw@...k.pl>,
	Thomas Gleixner <tglx@...utronix.de>
Subject: Re: [BUG 2.6.25-rc3] scheduler/hotplug: some processes are
	dealocked when cpu is set to offline

On Fri, Mar 07, 2008 at 07:25:37PM +0530, Gautham R Shenoy wrote:
> On Fri, Mar 07, 2008 at 02:02:20PM +0100, Dmitry Adamushko wrote:
> > Hi,
> > 
> > 'watchdog' is of SCHED_FIFO class. The standard load-balancer doesn't
> > move RT tasks between cpus anymore and there is a special mechanism in
> > scher_rt.c instead (I think, it's .25 material).
> > 
> > So I wonder, whether __migrate_task() is still capable of properly
> > moving a RT task to another CPU (e.g. for the case when it's in
> > TASK_RUNNING state) without breaking something in the rt migration
> > mechanism (or whatever else) that would leave us with a runqueue in
> > the 'inconsistent' state...
> > (I've taken a quick look at the relevant code so can't confirm it yet)
> > 
> > maybe it'd be faster if somebody could do a quick test now with the
> > following line commented out in kernel/softlockup.c :: watchdog()
> > 
> > -         sched_setscheduler(current, SCHED_FIFO, &param);
> 
> Commenting out that like seems to work. Passed 500 iterations of
> cpu-hotplug without any problems.

This seems to unearth another problem. After some 850 successful
cpu-hotplug iterations I got this message.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
------------[ cut here ]------------
BUG: spinlock recursion on CPU#2, kstopmachine/32521
BUG: spinlock lockup on CPU#1, kstopmachine/32517, cc43db80
Pid: 32517, comm: kstopmachine Not tainted 2.6.25-rc3 #44
 [<c0284ebb>] <0>BUG: spinlock lockup on CPU#0, kstopmachine/32520, cc43db80
_raw_spin_lock+0xd5/0xf9
 [<c04e1a33>] <0>BUG: spinlock lockup on CPU#3, kstopmachine/32522, cc43db80
_spin_lock+0x20/0x28
Pid: 32522, comm: kstopmachine Not tainted 2.6.25-rc3 #44
 [<c04dfa41>] ?  [<c0284ebb>] schedule+0xb0/0x5ab
 [<c04dfa41>] schedule+0xb0/0x5ab
_raw_spin_lock+0xd5/0xf9
 [<c04e1d58>] ?  [<c04e1a33>] _spin_unlock_irqrestore+0x36/0x3c
 [<c0119fdc>] ? _spin_lock+0x20/0x28
 [<c0119806>] ? complete+0x34/0x3e
 [<c0143646>] double_lock_balance+0x3a/0x57
 [<c0119806>] do_stop+0xd4/0xfe
double_lock_balance+0x3a/0x57
 [<c0143572>] ?  [<c0119af9>] do_stop+0x0/0xfe
pull_rt_task+0x79/0x164
 [<c012f77f>]  [<c011ca9d>] kthread+0x3b/0x64
pre_schedule_rt+0x18/0x1a
 [<c012f744>] ?  [<c04dfaa4>] kthread+0x0/0x64
schedule+0x113/0x5ab
 [<c01056eb>]  [<c04e1923>] ? kernel_thread_helper+0x7/0x10
_write_unlock_irq+0x22/0x26
 =======================
 [<c0139dec>] ? trace_hardirqs_on+0xe9/0x111
 [<c012262a>] do_exit+0x5a6/0x5aa
 [<c013007b>] ? sample_to_timespec+0x16/0x35
 [<c0143670>] ? stopmachine+0x0/0x98
 [<c01056f1>] kernel_thread_helper+0xd/0x10
 =======================
BUG: spinlock lockup on CPU#2, kstopmachine/32521, c065eb00
Pid: 32521, comm: kstopmachine Not tainted 2.6.25-rc3 #44
 [<c0284ebb>] _raw_spin_lock+0xd5/0xf9
 [<c04e1cb5>] _spin_lock_irqsave+0x29/0x32
 [<c011a03f>] ? __wake_up+0x15/0x3b
 [<c011a03f>] __wake_up+0x15/0x3b
 [<c011fa68>] wake_up_klogd+0x2e/0x31
 [<c011fc1e>] release_console_sem+0x1b3/0x1bb
 [<c012002c>] vprintk+0x2a8/0x301
 [<c013ac74>] ? __lock_acquire+0xaae/0xaf6
 [<c012009a>] printk+0x15/0x17
 [<c0284c6f>] spin_bug+0x47/0xbd
 [<c0284e1b>] _raw_spin_lock+0x35/0xf9
 [<c04e1a33>] _spin_lock+0x20/0x28
 [<c011a09b>] ? task_rq_lock+0x36/0x5d
 [<c011a09b>] task_rq_lock+0x36/0x5d
 [<c011a338>] try_to_wake_up+0x19/0xd1
 [<c011a3fb>] default_wake_function+0xb/0xd
 [<c012f850>] autoremove_wake_function+0xf/0x33
 [<c0117ec9>] __wake_up_common+0x2f/0x5a
 [<c011a052>] __wake_up+0x28/0x3b
 [<c011fa68>] wake_up_klogd+0x2e/0x31
 [<c011fc1e>] release_console_sem+0x1b3/0x1bb
 [<c012002c>] vprintk+0x2a8/0x301
 [<c04e196c>] ? _read_unlock+0x1d/0x20
 [<c0441a43>] ? sock_def_readable+0x61/0x69
 [<c048dde8>] ? unix_dgram_sendmsg+0x399/0x410
 [<c012009a>] printk+0x15/0x17
 [<c011f40d>] warn_on_slowpath+0x2a/0x51
 [<c0130066>] ? sample_to_timespec+0x1/0x35
 [<c013ac74>] ? __lock_acquire+0xaae/0xaf6
 [<c011a446>] ? resched_cpu+0x2c/0x6f
 [<c0111691>] native_smp_send_reschedule+0x22/0x3f
 [<c01186c7>] __resched_task+0x5f/0x63
 [<c011a479>] resched_cpu+0x5f/0x6f
 [<c011bb6d>] scheduler_tick+0x214/0x28f
 [<c012737b>] update_process_times+0x3d/0x49
 [<c013751d>] tick_sched_timer+0x6e/0xa6
 [<c01374af>] ? tick_sched_timer+0x0/0xa6
 [<c0131e8c>] __run_hrtimer+0x39/0x70
 [<c0132668>] hrtimer_interrupt+0xeb/0x154
 [<c01127bc>] smp_apic_timer_interrupt+0x6c/0x80
 [<c0143670>] ? stopmachine+0x0/0x98
 [<c0105553>] apic_timer_interrupt+0x33/0x38
 [<c0143670>] ? stopmachine+0x0/0x98
 [<c013007b>] ? sample_to_timespec+0x16/0x35
 [<c0143701>] ? stopmachine+0x91/0x98
 [<c01056eb>] kernel_thread_helper+0x7/0x10
 =======================

BUG: spinlock lockup on CPU#0, kstopmachine/32520, c065eb00

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.config attached.


> 
> > 
> > 
> > -- 
> > Best regards,
> > Dmitry Adamushko
> 
> -- 
> Thanks and Regards
> gautham

-- 
Thanks and Regards
gautham

Download attachment "config.gz" of type "application/octet-stream" (9691 bytes)