Message-ID: <20091014153834.GA27433@goodmis.org>
Date:	Wed, 14 Oct 2009 11:38:34 -0400
From:	Steven Rostedt <rostedt@...dmis.org>
To:	dtslinux@...mail.com
Cc:	linux-kernel@...r.kernel.org
Subject: Re: [PATCH -rt] Preemption problem in kernel RT Patch

Note, LKML receives 600 emails a day. If you want to make sure your email is seen,
it is best to Cc those that post the -rt patches. Otherwise, your email is
likely to get lost in the noise.

On Wed, Oct 14, 2009 at 08:55:28AM -0400, dtslinux@...mail.com wrote:
> Hello All,
> 
> I am having an issue in kernel 2.6.24.7 with RT-27 patch. I am using a block device driver that is doing I/O operations on a virtual device. The driver is using separate kernel threads to perform read and write operations. The driver is working fine in the normal kernels, it is also working fine in RT-27 patch with 2.6.24.7 kernel, but some times I am getting following bug when performing write test with xdd benchmark (in RT-27 patch with 2.6.24.7 kernel)  :
> 
> WARNING: at kernel/rtmutex.c:979 rt_spin_lock()

This shows that a non "rtmutex" was used in the rtmutex code.

> Pid: 12634, comm: pdflush Tainted: GF       2.6.24.7-rt27 #9
>  [<c04046b8>] show_trace_log_lvl+0x1f/0x34
>  [<c0404f67>] show_trace+0x17/0x19
>  [<c04052e2>] dump_stack+0x6f/0x75
>  [<c063ac74>] rt_spin_lock+0x4a/0xa2
>  [<c04f33a4>] cfq_exit_io_context+0x30/0x56

The q->queue_lock used in cfq_exit_single_io_context is not an rtmutex.
Yes, this will crash the kernel.

-- Steve

>  [<c04ed88f>] exit_io_context+0x68/0x72
>  [<c04206c1>] do_exit+0x6c2/0x739
>  [<c040432d>] kernel_thread_helper+0xd/0x10
>  =======================
> 
> <1>BUG: unable to handle kernel NULL pointer dereference at virtual address 0000003d
> printing eip: c043d18a *pdpt = 00000000349dd001 *pde = 0000000000000000
> 
> <0>Oops: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
> Modules linked in: sysfs_driver(F) regularcache(F) dts nls_utf8 hfsplus ramdisk_driver bridge autofs4 hidp rfcomm l2cap bluetooth sunrpc ib_iser libiscsi scsi_transport_iscsi ib_srp scsi_transport_srp ib_ipoib ipv6 rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa ib_mad ib_core dm_mirror dm_multipath dm_mod sbs sbshc battery ac lp floppy sg serio_raw parport_pc parport snd_intel8x0 snd_ac97_codec 8250_pnp ac97_bus snd_seq_dummy 8250 serial_core snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss button snd_pcm e1000 snd_timer snd soundcore i2c_i801 snd_page_alloc i2c_core pcspkr ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
> 
> Pid: 12634, comm: pdflush Tainted: GF       (2.6.24.7-rt27 #9)
> EIP: 0060:[<c043d18a>] EFLAGS: 00010016 CPU: 0
> EIP is at task_blocks_on_rt_mutex+0xf8/0x240
> EAX: ef42406c EBX: 0000001a ECX: ef424044 EDX: ef424044
> ESI: ef424044 EDI: 00000009 EBP: e7ba7eec ESP: e7ba7ebc
>  DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 preempt:00000003
> Process pdflush (pid: 12634, ti=e7ba7000 task=e5268060 task.ti=e7ba7000)
> Stack: 00000001 ef424054 ef424044 00000296 00000000 e7ba7f04 ef424044 00000009
>        e7ba7f1c 00000296 ef424044 00000296 e7ba7f58 c063a57e 00000296 00000046
>        00000000 ffffffff 00000078 e7ba7f08 e7ba7f08 e7ba7f10 e7ba7f10 00000000
> Call Trace:
>  [<c04046b8>] show_trace_log_lvl+0x1f/0x34
>  [<c0404772>] show_stack_log_lvl+0xa5/0xb9
>  [<c040483a>] show_registers+0xb4/0x1b8
>  [<c0404a5e>] die+0x120/0x21b
>  [<c063ddfc>] do_page_fault+0x845/0xa07
>  [<c063c29a>] error_code+0x6a/0x70
>  [<c063a57e>] rt_spin_lock_slowlock+0xc5/0x1b7
>  [<c063a9d4>] __rt_spin_lock+0x48/0x4b
>  [<c063acbe>] rt_spin_lock+0x94/0xa2
>  [<c04f33a4>] cfq_exit_io_context+0x30/0x56
>  [<c04ed88f>] exit_io_context+0x68/0x72
>  [<c04206c1>] do_exit+0x6c2/0x739
>  [<c040432d>] kernel_thread_helper+0xd/0x10
>  =======================
> INFO: lockdep is turned off.
> Code: 24 08 4c 00 00 00 c7 44 24 04 83 42 6e c0 c7 04 24 b6 78 6d c0 e8 c0 0d fe ff e8 f5 80 fc ff 8b 7f 08 8b 4d e8 83 ef 0c 89 7d ec <39> 4f 34 74 04 0f 0b eb fe 8b 7d e8 8b 45 e4 83 c7 20 89 fa e8
> 
> <0>EIP: [<c043d18a>] task_blocks_on_rt_mutex+0xf8/0x240 SS:ESP 0068:e7ba7ebc
> ---[ end trace 432e3e53cc0cfa18 ]---
> Fixing recursive fault but reboot is needed!
> BUG: scheduling with irqs disabled: pdflush/0x00000002/12634
> caller is do_exit+0xcc/0x739
> Pid: 12634, comm: pdflush Tainted: GF     D 2.6.24.7-rt27 #9
>  [<c04046b8>] show_trace_log_lvl+0x1f/0x34
>  [<c0404f67>] show_trace+0x17/0x19
>  [<c04052e2>] dump_stack+0x6f/0x75
>  [<c0638e41>] schedule+0x8a/0x105
>  [<c04200cb>] do_exit+0xcc/0x739
>  [<c0404b51>] die+0x213/0x21b
>  [<c063ddfc>] do_page_fault+0x845/0xa07
>  [<c063c29a>] error_code+0x6a/0x70
>  [<c063a57e>] rt_spin_lock_slowlock+0xc5/0x1b7
>  [<c063a9d4>] __rt_spin_lock+0x48/0x4b
>  [<c063acbe>] rt_spin_lock+0x94/0xa2
>  [<c04f33a4>] cfq_exit_io_context+0x30/0x56
>  [<c04ed88f>] exit_io_context+0x68/0x72
>  [<c04206c1>] do_exit+0x6c2/0x739
>  [<c040432d>] kernel_thread_helper+0xd/0x10
>  =======================
> 
> I am confuse whether my driver is causing this problem or not as in the trace above I cannot find any function of my driver. All the functions are of kernel and pdflush is causing this problem.  In the above trace "Modules linked in: sysfs_driver(F) regularcache(F) dts "     are my modules.
> I am using CentOS 5 with 1GB RAM on  "Intel(R) Pentium(R) 4 CPU 2.80GHz".  Please guide me if I am doing any mistake
> 
> Thanks!
> Furahm
> 
> --
> This message was sent on behalf of dtslinux@...mail.com at openSubscriber.com
> http://www.opensubscriber.com/message/linux-kernel@vger.kernel.org/6978704.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists