Feb 26 14:17:52 shakti kernel: [90667.792936] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768] Feb 26 14:17:52 shakti kernel: [90667.793261] Feb 26 14:17:52 shakti kernel: [90667.793362] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1) Feb 26 14:17:52 shakti kernel: [90667.793495] EIP: 0060:[] EFLAGS: 00000293 CPU: 1 Feb 26 14:17:52 shakti kernel: [90667.794055] EIP is at tcp_packet+0x5e/0xbb0 [nf_conntrack] Feb 26 14:17:52 shakti kernel: [90667.794197] EAX: d8502634 EBX: f9114050 ECX: 00000014 EDX: e65d5480 Feb 26 14:17:52 shakti kernel: [90667.794316] ESI: d54c2a50 EDI: d54c2a50 EBP: 00000000 ESP: df9cf930 Feb 26 14:17:52 shakti kernel: [90667.794433] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Feb 26 14:17:52 shakti kernel: [90667.794554] CR0: 80050033 CR2: b7bec04d CR3: 00494000 CR4: 000006b0 Feb 26 14:17:52 shakti kernel: [90667.794658] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Feb 26 14:17:52 shakti kernel: [90667.794759] DR6: ffff0ff0 DR7: 00000400 Feb 26 14:17:52 shakti kernel: [90667.794880] [] svc_sock_enqueue+0x1e1/0x310 [sunrpc] Feb 26 14:17:52 shakti kernel: [90667.796041] [] svc_tcp_data_ready+0x2d/0x90 [sunrpc] Feb 26 14:17:52 shakti kernel: [90667.797031] [] tcp_current_mss+0x77/0x110 Feb 26 14:17:52 shakti kernel: [90667.797248] [] tcp_rcv_established+0x6dd/0x950 Feb 26 14:17:52 shakti kernel: [90667.797497] [] nf_ip_checksum+0x0/0x120 Feb 26 14:17:52 shakti kernel: [90667.797778] [] __nf_conntrack_find+0x1c/0x110 [nf_conntrack] Feb 26 14:17:52 shakti kernel: [90667.798423] [] tcp_packet+0x0/0xbb0 [nf_conntrack] Feb 26 14:17:52 shakti kernel: [90667.799063] [] nf_conntrack_in+0x201/0x4d0 [nf_conntrack] Feb 26 14:17:52 shakti kernel: [90667.799917] [] ipv4_conntrack_in+0x0/0x10 [nf_conntrack_ipv4] Feb 26 14:17:52 shakti kernel: [90667.800245] [] nf_iterate+0x57/0x80 Feb 26 14:17:52 shakti kernel: [90667.800434] [] ip_rcv_finish+0x0/0x370 Feb 26 14:17:52 shakti kernel: [90667.800664] [] nf_hook_slow+0x98/0xd0 Feb 26 14:17:52 shakti kernel: [90667.800870] [] ip_rcv_finish+0x0/0x370 Feb 26 14:17:52 shakti kernel: [90667.801095] [] ip_rcv+0x1ca/0x290 Feb 26 14:17:52 shakti kernel: [90667.801289] [] ip_rcv_finish+0x0/0x370 Feb 26 14:17:52 shakti kernel: [90667.801458] [] ip_rcv+0x0/0x290 Feb 26 14:17:52 shakti kernel: [90667.801656] [] netif_receive_skb+0x379/0x440 Feb 26 14:17:52 shakti kernel: [90667.801916] [] e1000_receive_skb+0x47/0x1a0 [e1000e] Feb 26 14:17:52 shakti kernel: [90667.802481] [] e1000_clean_rx_irq+0x124/0x300 [e1000e] Feb 26 14:17:52 shakti kernel: [90667.803021] [] e1000_clean_rx_irq+0x79/0x300 [e1000e] Feb 26 14:17:52 shakti kernel: [90667.803668] [] e1000_clean+0x54/0x220 [e1000e] Feb 26 14:17:52 shakti kernel: [90667.804220] [] net_rx_action+0x12d/0x210 Feb 26 14:17:52 shakti kernel: [90667.804480] [] __do_softirq+0x82/0x110 Feb 26 14:17:52 shakti kernel: [90667.804709] [] do_softirq+0x55/0x60 Feb 26 14:17:52 shakti kernel: [90667.804894] [] irq_exit+0x6d/0x80 Feb 26 14:17:52 shakti kernel: [90667.805049] [] smp_apic_timer_interrupt+0x55/0x80 Feb 26 14:17:52 shakti kernel: [90667.805254] [] apic_timer_interrupt+0x28/0x30 Feb 26 14:17:52 shakti kernel: [90667.805515] [] xor_sse_4+0x19e/0x360 [xor] Feb 26 14:17:52 shakti kernel: [90667.805918] [] xor_blocks+0x75/0xa0 [xor] Feb 26 14:17:52 shakti kernel: [90667.806205] [] async_xor+0x2fb/0x350 [async_xor] Feb 26 14:17:52 shakti kernel: [90667.806527] [] ops_complete_write+0x0/0x60 [raid456] Feb 26 14:17:52 shakti kernel: [90667.806897] [] ops_run_postxor+0xd1/0x160 [raid456] Feb 26 14:17:52 shakti kernel: [90667.807262] [] ops_complete_write+0x0/0x60 [raid456] Feb 26 14:17:52 shakti kernel: [90667.807698] [] handle_stripe5+0x1040/0x1590 [raid456] Feb 26 14:17:52 shakti kernel: [90667.808267] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:17:52 shakti kernel: [90667.808755] [] handle_stripe+0x28/0xf00 [raid456] Feb 26 14:17:52 shakti kernel: [90667.809094] [] enqueue_hrtimer+0x72/0x100 Feb 26 14:17:52 shakti kernel: [90667.809337] [] __do_softirq+0x82/0x110 Feb 26 14:17:52 shakti kernel: [90667.809574] [] __wake_up+0x3e/0x60 Feb 26 14:17:52 shakti kernel: [90667.809785] [] __release_stripe+0x10e/0x150 [raid456] Feb 26 14:17:52 shakti kernel: [90667.810131] [] release_stripe+0x2f/0x50 [raid456] Feb 26 14:17:52 shakti kernel: [90667.810464] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:17:52 shakti kernel: [90667.810938] [] raid5d+0x3d5/0x460 [raid456] Feb 26 14:17:52 shakti kernel: [90667.811313] [] common_interrupt+0x23/0x28 Feb 26 14:17:52 shakti kernel: [90667.811523] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:17:52 shakti kernel: [90667.812036] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:17:52 shakti kernel: [90667.812523] [] md_thread+0x23/0xe0 [md_mod] Feb 26 14:17:52 shakti kernel: [90667.813014] [] autoremove_wake_function+0x0/0x40 Feb 26 14:17:52 shakti kernel: [90667.813238] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:17:52 shakti kernel: [90667.813722] [] kthread+0x42/0x70 Feb 26 14:17:52 shakti kernel: [90667.813875] [] kthread+0x0/0x70 Feb 26 14:17:52 shakti kernel: [90667.814039] [] kernel_thread_helper+0x7/0x10 Feb 26 14:17:52 shakti kernel: [90667.814252] ======================= Feb 26 14:19:13 shakti kernel: [90748.930999] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768] Feb 26 14:19:13 shakti kernel: [90748.931321] Feb 26 14:19:13 shakti kernel: [90748.931421] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1) Feb 26 14:19:13 shakti kernel: [90748.931551] EIP: 0060:[] EFLAGS: 00000212 CPU: 1 Feb 26 14:19:13 shakti kernel: [90748.931672] EIP is at local_bh_enable+0x20/0xa0 Feb 26 14:19:13 shakti kernel: [90748.931796] EAX: df9ce000 EBX: 00000008 ECX: c040d120 EDX: e11b9e40 Feb 26 14:19:13 shakti kernel: [90748.931943] ESI: d90fb300 EDI: e11b9e40 EBP: 00000000 ESP: df9cfa68 Feb 26 14:19:13 shakti kernel: [90748.932059] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Feb 26 14:19:13 shakti kernel: [90748.932184] CR0: 8005003b CR2: b7a53000 CR3: 00494000 CR4: 000006b0 Feb 26 14:19:13 shakti kernel: [90748.932289] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Feb 26 14:19:13 shakti kernel: [90748.932388] DR6: ffff0ff0 DR7: 00000400 Feb 26 14:19:13 shakti kernel: [90748.932514] [] security_sock_rcv_skb+0xc/0x10 Feb 26 14:19:13 shakti kernel: [90748.932691] [] tcp_v4_rcv+0x7f6/0x9a0 Feb 26 14:19:13 shakti kernel: [90748.933001] [] ip_local_deliver_finish+0xf9/0x210 Feb 26 14:19:13 shakti kernel: [90748.933243] [] ip_rcv_finish+0xff/0x370 Feb 26 14:19:13 shakti kernel: [90748.933459] [] ip_rcv_finish+0x0/0x370 Feb 26 14:19:13 shakti kernel: [90748.933699] [] ip_rcv+0x1ca/0x290 Feb 26 14:19:13 shakti kernel: [90748.933896] [] ip_rcv_finish+0x0/0x370 Feb 26 14:19:13 shakti kernel: [90748.934090] [] ip_rcv+0x0/0x290 Feb 26 14:19:13 shakti kernel: [90748.934294] [] netif_receive_skb+0x379/0x440 Feb 26 14:19:13 shakti kernel: [90748.934572] [] e1000_receive_skb+0x47/0x1a0 [e1000e] Feb 26 14:19:13 shakti kernel: [90748.935143] [] e1000_clean_rx_irq+0x124/0x300 [e1000e] Feb 26 14:19:13 shakti kernel: [90748.935684] [] e1000_clean_rx_irq+0x79/0x300 [e1000e] Feb 26 14:19:13 shakti kernel: [90748.936372] [] e1000_clean+0x54/0x220 [e1000e] Feb 26 14:19:13 shakti kernel: [90748.936929] [] net_rx_action+0x12d/0x210 Feb 26 14:19:13 shakti kernel: [90748.937210] [] __do_softirq+0x82/0x110 Feb 26 14:19:13 shakti kernel: [90748.937451] [] do_softirq+0x55/0x60 Feb 26 14:19:13 shakti kernel: [90748.937637] [] irq_exit+0x6d/0x80 Feb 26 14:19:13 shakti kernel: [90748.937816] [] smp_apic_timer_interrupt+0x55/0x80 Feb 26 14:19:13 shakti kernel: [90748.937995] [] deadline_dispatch_requests+0x44/0xd0 Feb 26 14:19:13 shakti kernel: [90748.938209] [] apic_timer_interrupt+0x28/0x30 Feb 26 14:19:13 shakti kernel: [90748.938494] [] scsi_request_fn+0x1d7/0x380 [scsi_mod] Feb 26 14:19:13 shakti kernel: [90748.939343] [] elv_insert+0xd1/0x230 Feb 26 14:19:13 shakti kernel: [90748.939592] [] __generic_unplug_device+0x25/0x30 Feb 26 14:19:13 shakti kernel: [90748.939794] [] __make_request+0xba/0x5b0 Feb 26 14:19:13 shakti kernel: [90748.940122] [] generic_make_request+0x225/0x4c0 Feb 26 14:19:13 shakti kernel: [90748.940319] [] try_to_wake_up+0x4e/0x350 Feb 26 14:19:13 shakti kernel: [90748.940651] [] sched_clock+0x13/0x30 Feb 26 14:19:13 shakti kernel: [90748.940862] [] update_curr+0x103/0x110 Feb 26 14:19:13 shakti kernel: [90748.941109] [] handle_stripe5+0x8f0/0x1590 [raid456] Feb 26 14:19:13 shakti kernel: [90748.941450] [] lapic_next_event+0xc/0x10 Feb 26 14:19:13 shakti kernel: [90748.941756] [] getnstimeofday+0x34/0xf0 Feb 26 14:19:13 shakti kernel: [90748.941940] [] lapic_next_event+0xc/0x10 Feb 26 14:19:13 shakti kernel: [90748.942107] [] clockevents_program_event+0xa3/0x150 Feb 26 14:19:13 shakti kernel: [90748.942438] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:13 shakti kernel: [90748.942924] [] handle_stripe+0x28/0xf00 [raid456] Feb 26 14:19:13 shakti kernel: [90748.943355] [] __do_softirq+0x82/0x110 Feb 26 14:19:13 shakti kernel: [90748.943574] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:13 shakti kernel: [90748.944067] [] irq_exit+0x51/0x80 Feb 26 14:19:13 shakti kernel: [90748.944244] [] smp_apic_timer_interrupt+0x55/0x80 Feb 26 14:19:13 shakti kernel: [90748.944449] [] apic_timer_interrupt+0x28/0x30 Feb 26 14:19:13 shakti kernel: [90748.944675] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:13 shakti kernel: [90748.945178] [] release_stripe+0x2f/0x50 [raid456] Feb 26 14:19:13 shakti kernel: [90748.945524] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:13 shakti kernel: [90748.945989] [] raid5d+0x3d5/0x460 [raid456] Feb 26 14:19:13 shakti kernel: [90748.946379] [] common_interrupt+0x23/0x28 Feb 26 14:19:13 shakti kernel: [90748.946594] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:13 shakti kernel: [90748.947076] [] sys_setsid+0x2b/0xf0 Feb 26 14:19:13 shakti kernel: [90748.947262] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:13 shakti kernel: [90748.947744] [] md_thread+0x23/0xe0 [md_mod] Feb 26 14:19:13 shakti kernel: [90748.948230] [] autoremove_wake_function+0x0/0x40 Feb 26 14:19:13 shakti kernel: [90748.948440] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:13 shakti kernel: [90748.948916] [] kthread+0x42/0x70 Feb 26 14:19:13 shakti kernel: [90748.949071] [] kthread+0x0/0x70 Feb 26 14:19:13 shakti kernel: [90748.949239] [] kernel_thread_helper+0x7/0x10 Feb 26 14:19:13 shakti kernel: [90748.949459] ======================= Feb 26 14:19:25 shakti kernel: [90760.730510] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768] Feb 26 14:19:25 shakti kernel: [90760.730841] Feb 26 14:19:25 shakti kernel: [90760.730942] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1) Feb 26 14:19:25 shakti kernel: [90760.731070] EIP: 0060:[] EFLAGS: 00000296 CPU: 1 Feb 26 14:19:25 shakti kernel: [90760.731253] EIP is at nf_nat_in+0x1e/0xb0 [iptable_nat] Feb 26 14:19:25 shakti kernel: [90760.731391] EAX: 00000000 EBX: 0101a8c0 ECX: f6550000 EDX: c02d5af0 Feb 26 14:19:25 shakti kernel: [90760.731521] ESI: df9cfadc EDI: d4020d80 EBP: 80000000 ESP: df9cfa84 Feb 26 14:19:25 shakti kernel: [90760.731641] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Feb 26 14:19:25 shakti kernel: [90760.731765] CR0: 8005003b CR2: b7a53000 CR3: 00494000 CR4: 000006b0 Feb 26 14:19:25 shakti kernel: [90760.731868] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Feb 26 14:19:25 shakti kernel: [90760.731964] DR6: ffff0ff0 DR7: 00000400 Feb 26 14:19:25 shakti kernel: [90760.732087] [] ipt_route_hook+0x0/0x20 [iptable_mangle] Feb 26 14:19:25 shakti kernel: [90760.732300] [] nf_nat_in+0x0/0xb0 [iptable_nat] Feb 26 14:19:25 shakti kernel: [90760.732535] [] nf_iterate+0x57/0x80 Feb 26 14:19:25 shakti kernel: [90760.732726] [] ip_rcv_finish+0x0/0x370 Feb 26 14:19:25 shakti kernel: [90760.732966] [] nf_hook_slow+0x98/0xd0 Feb 26 14:19:25 shakti kernel: [90760.733176] [] ip_rcv_finish+0x0/0x370 Feb 26 14:19:25 shakti kernel: [90760.733414] [] ip_rcv+0x1ca/0x290 Feb 26 14:19:25 shakti kernel: [90760.733610] [] ip_rcv_finish+0x0/0x370 Feb 26 14:19:25 shakti kernel: [90760.733803] [] ip_rcv+0x0/0x290 Feb 26 14:19:25 shakti kernel: [90760.734008] [] netif_receive_skb+0x379/0x440 Feb 26 14:19:25 shakti kernel: [90760.734281] [] e1000_receive_skb+0x47/0x1a0 [e1000e] Feb 26 14:19:25 shakti kernel: [90760.734851] [] e1000_clean_rx_irq+0x124/0x300 [e1000e] Feb 26 14:19:25 shakti kernel: [90760.735389] [] e1000_clean_rx_irq+0x79/0x300 [e1000e] Feb 26 14:19:25 shakti kernel: [90760.736077] [] e1000_clean+0x54/0x220 [e1000e] Feb 26 14:19:25 shakti kernel: [90760.736633] [] net_rx_action+0x12d/0x210 Feb 26 14:19:25 shakti kernel: [90760.736904] [] __do_softirq+0x82/0x110 Feb 26 14:19:25 shakti kernel: [90760.737141] [] do_softirq+0x55/0x60 Feb 26 14:19:25 shakti kernel: [90760.737326] [] irq_exit+0x6d/0x80 Feb 26 14:19:25 shakti kernel: [90760.737502] [] do_IRQ+0x40/0x70 Feb 26 14:19:25 shakti kernel: [90760.737653] [] elv_dispatch_add_tail+0x1c/0x60 Feb 26 14:19:25 shakti kernel: [90760.737872] [] common_interrupt+0x23/0x28 Feb 26 14:19:25 shakti kernel: [90760.738139] [] scsi_request_fn+0x1d7/0x380 [scsi_mod] Feb 26 14:19:25 shakti kernel: [90760.738997] [] elv_insert+0xd1/0x230 Feb 26 14:19:25 shakti kernel: [90760.739223] [] __generic_unplug_device+0x25/0x30 Feb 26 14:19:25 shakti kernel: [90760.739427] [] __make_request+0xba/0x5b0 Feb 26 14:19:25 shakti kernel: [90760.739622] [] irq_exit+0x51/0x80 Feb 26 14:19:25 shakti kernel: [90760.739776] [] do_IRQ+0x40/0x70 Feb 26 14:19:25 shakti kernel: [90760.739945] [] e100_xmit_frame+0x30/0x110 [e100] Feb 26 14:19:25 shakti kernel: [90760.740325] [] common_interrupt+0x23/0x28 Feb 26 14:19:25 shakti kernel: [90760.740585] [] generic_make_request+0x225/0x4c0 Feb 26 14:19:25 shakti kernel: [90760.740779] [] mempool_free+0x88/0xa0 Feb 26 14:19:25 shakti kernel: [90760.740972] [] mempool_free+0x88/0xa0 Feb 26 14:19:25 shakti kernel: [90760.741154] [] mempool_free+0x88/0xa0 Feb 26 14:19:25 shakti kernel: [90760.741324] [] mempool_free+0x88/0xa0 Feb 26 14:19:25 shakti kernel: [90760.741585] [] __wake_up+0x3e/0x60 Feb 26 14:19:25 shakti kernel: [90760.741814] [] handle_stripe5+0x8f0/0x1590 [raid456] Feb 26 14:19:25 shakti kernel: [90760.742152] [] __kfree_skb+0x8/0x80 Feb 26 14:19:25 shakti kernel: [90760.742550] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:25 shakti kernel: [90760.743025] [] handle_stripe+0x28/0xf00 [raid456] Feb 26 14:19:25 shakti kernel: [90760.743437] [] __do_softirq+0x82/0x110 Feb 26 14:19:25 shakti kernel: [90760.743646] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:25 shakti kernel: [90760.744130] [] irq_exit+0x51/0x80 Feb 26 14:19:25 shakti kernel: [90760.744305] [] smp_apic_timer_interrupt+0x55/0x80 Feb 26 14:19:25 shakti kernel: [90760.744508] [] apic_timer_interrupt+0x28/0x30 Feb 26 14:19:25 shakti kernel: [90760.744728] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:25 shakti kernel: [90760.745239] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:25 shakti kernel: [90760.745717] [] raid5d+0x3d5/0x460 [raid456] Feb 26 14:19:25 shakti kernel: [90760.746099] [] common_interrupt+0x23/0x28 Feb 26 14:19:25 shakti kernel: [90760.746311] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:25 shakti kernel: [90760.746793] [] sys_setsid+0x2b/0xf0 Feb 26 14:19:25 shakti kernel: [90760.746992] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:25 shakti kernel: [90760.747471] [] md_thread+0x23/0xe0 [md_mod] Feb 26 14:19:25 shakti kernel: [90760.747954] [] autoremove_wake_function+0x0/0x40 Feb 26 14:19:25 shakti kernel: [90760.748173] [] md_thread+0x0/0xe0 [md_mod] Feb 26 14:19:25 shakti kernel: [90760.748646] [] kthread+0x42/0x70 Feb 26 14:19:25 shakti kernel: [90760.748798] [] kthread+0x0/0x70 Feb 26 14:19:25 shakti kernel: [90760.748963] [] kernel_thread_helper+0x7/0x10 Feb 26 14:19:25 shakti kernel: [90760.749177] =======================