Mar 5 09:45:54 shakti kernel: [764912.385931] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768] Mar 5 09:45:54 shakti kernel: [764912.386329] Mar 5 09:45:54 shakti kernel: [764912.386425] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1) Mar 5 09:45:54 shakti kernel: [764912.386558] EIP: 0060:[] EFLAGS: 00000296 CPU: 1 Mar 5 09:45:54 shakti kernel: [764912.386693] EIP is at kmem_cache_alloc+0x19/0xc0 Mar 5 09:45:54 shakti kernel: [764912.386820] EAX: c042d020 EBX: 00000020 ECX: 00000000 EDX: c02ae614 Mar 5 09:45:54 shakti kernel: [764912.386959] ESI: 00000100 EDI: 00000020 EBP: c042d020 ESP: df9cfc44 Mar 5 09:45:54 shakti kernel: [764912.387093] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Mar 5 09:45:54 shakti kernel: [764912.387233] CR0: 8005003b CR2: 084f0d4c CR3: 0506b000 CR4: 000006b0 Mar 5 09:45:54 shakti kernel: [764912.387353] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Mar 5 09:45:54 shakti kernel: [764912.387465] DR6: ffff0ff0 DR7: 00000400 Mar 5 09:45:54 shakti kernel: [764912.387586] [] netif_receive_skb+0x379/0x440 Mar 5 09:45:54 shakti kernel: [764912.387816] [] __alloc_skb+0x34/0x120 Mar 5 09:45:54 shakti kernel: [764912.388032] [] __netdev_alloc_skb+0x22/0x50 Mar 5 09:45:54 shakti kernel: [764912.388248] [] e1000_clean_rx_irq+0x24e/0x300 [e1000e] Mar 5 09:45:54 shakti kernel: [764912.388810] [] e1000_clean_rx_irq+0x79/0x300 [e1000e] Mar 5 09:45:54 shakti kernel: [764912.389536] [] e1000_clean+0x54/0x220 [e1000e] Mar 5 09:45:54 shakti kernel: [764912.390117] [] net_rx_action+0x12d/0x210 Mar 5 09:45:54 shakti kernel: [764912.390396] [] __do_softirq+0x82/0x110 Mar 5 09:45:54 shakti kernel: [764912.390627] [] do_softirq+0x55/0x60 Mar 5 09:45:54 shakti kernel: [764912.390797] [] irq_exit+0x6d/0x80 Mar 5 09:45:54 shakti kernel: [764912.390952] [] smp_apic_timer_interrupt+0x55/0x80 Mar 5 09:45:54 shakti kernel: [764912.391174] [] apic_timer_interrupt+0x28/0x30 Mar 5 09:45:54 shakti kernel: [764912.391450] [] generic_make_request+0x3c/0x4c0 Mar 5 09:45:54 shakti kernel: [764912.391686] [] handle_stripe5+0x8f0/0x1590 [raid456] Mar 5 09:45:54 shakti kernel: [764912.392448] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:45:54 shakti kernel: [764912.392957] [] handle_stripe+0x28/0xf00 [raid456] Mar 5 09:45:54 shakti kernel: [764912.393440] [] __do_softirq+0x82/0x110 Mar 5 09:45:54 shakti kernel: [764912.393660] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:45:54 shakti kernel: [764912.394218] [] __wake_up+0x3e/0x60 Mar 5 09:45:54 shakti kernel: [764912.394457] [] release_stripe+0x2f/0x50 [raid456] Mar 5 09:45:54 shakti kernel: [764912.394796] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:45:54 shakti kernel: [764912.395285] [] raid5d+0x3d5/0x460 [raid456] Mar 5 09:45:54 shakti kernel: [764912.395722] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:45:54 shakti kernel: [764912.396215] [] sys_setsid+0x2b/0xf0 Mar 5 09:45:54 shakti kernel: [764912.396407] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:45:54 shakti kernel: [764912.396903] [] md_thread+0x23/0xe0 [md_mod] Mar 5 09:45:54 shakti kernel: [764912.397406] [] autoremove_wake_function+0x0/0x40 Mar 5 09:45:54 shakti kernel: [764912.397627] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:45:54 shakti kernel: [764912.398117] [] kthread+0x42/0x70 Mar 5 09:45:54 shakti kernel: [764912.398271] [] kthread+0x0/0x70 Mar 5 09:45:54 shakti kernel: [764912.398444] [] kernel_thread_helper+0x7/0x10 Mar 5 09:45:54 shakti kernel: [764912.398659] ======================= Mar 5 09:49:20 shakti kernel: [765117.901534] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768] Mar 5 09:49:20 shakti kernel: [765117.901867] Mar 5 09:49:20 shakti kernel: [765117.901955] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1) Mar 5 09:49:20 shakti kernel: [765117.902077] EIP: 0060:[] EFLAGS: 00000246 CPU: 1 Mar 5 09:49:20 shakti kernel: [765117.902298] EIP is at ctnetlink_conntrack_event+0xa6/0x930 [nf_conntrack_netlink] Mar 5 09:49:20 shakti kernel: [765117.902429] EAX: 00000000 EBX: f9177ab8 ECX: eec59a50 EDX: 00000040 Mar 5 09:49:20 shakti kernel: [765117.902551] ESI: ffffffff EDI: 00000000 EBP: 00000000 ESP: df9cf900 Mar 5 09:49:20 shakti kernel: [765117.902672] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Mar 5 09:49:20 shakti kernel: [765117.902798] CR0: 8005003b CR2: b7e66000 CR3: 00494000 CR4: 000006b0 Mar 5 09:49:20 shakti kernel: [765117.902905] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Mar 5 09:49:20 shakti kernel: [765117.903006] DR6: ffff0ff0 DR7: 00000400 Mar 5 09:49:20 shakti kernel: [765117.903116] [] ipt_do_table+0x20b/0x4b0 [ip_tables] Mar 5 09:49:20 shakti kernel: [765117.903391] [] ipt_do_table+0x20b/0x4b0 [ip_tables] Mar 5 09:49:20 shakti kernel: [765117.903721] [] notifier_call_chain+0x30/0x60 Mar 5 09:49:20 shakti kernel: [765117.903931] [] atomic_notifier_call_chain+0x17/0x20 Mar 5 09:49:20 shakti kernel: [765117.904118] [] nf_ct_deliver_cached_events+0x8d/0xc0 [nf_conntrack] Mar 5 09:49:20 shakti kernel: [765117.904729] [] ipv4_confirm+0x29/0x50 [nf_conntrack_ipv4] Mar 5 09:49:20 shakti kernel: [765117.904994] [] ipv4_confirm+0x0/0x50 [nf_conntrack_ipv4] Mar 5 09:49:20 shakti kernel: [765117.905261] [] nf_iterate+0x57/0x80 Mar 5 09:49:20 shakti kernel: [765117.905415] [] ip_finish_output+0x0/0x2c0 Mar 5 09:49:20 shakti kernel: [765117.905622] [] nf_hook_slow+0x98/0xd0 Mar 5 09:49:20 shakti kernel: [765117.905795] [] ip_finish_output+0x0/0x2c0 Mar 5 09:49:20 shakti kernel: [765117.906001] [] ip_output+0x8c/0xa0 Mar 5 09:49:20 shakti kernel: [765117.906163] [] ip_finish_output+0x0/0x2c0 Mar 5 09:49:20 shakti kernel: [765117.906335] [] ip_queue_xmit+0x27f/0x430 Mar 5 09:49:20 shakti kernel: [765117.906504] [] dst_output+0x0/0x10 Mar 5 09:49:20 shakti kernel: [765117.906716] [] sched_clock+0x13/0x30 Mar 5 09:49:20 shakti kernel: [765117.906915] [] __wake_up_common+0x4b/0x80 Mar 5 09:49:20 shakti kernel: [765117.907095] [] _write_lock_bh+0x8/0x20 Mar 5 09:49:20 shakti kernel: [765117.907262] [] tcp_v4_send_check+0x44/0xf0 Mar 5 09:49:20 shakti kernel: [765117.907470] [] tcp_transmit_skb+0x497/0x820 Mar 5 09:49:20 shakti kernel: [765117.907702] [] tcp_send_ack+0x2b/0x100 Mar 5 09:49:20 shakti kernel: [765117.907892] [] __alloc_skb+0x55/0x120 Mar 5 09:49:20 shakti kernel: [765117.908082] [] tcp_send_ack+0xb6/0x100 Mar 5 09:49:20 shakti kernel: [765117.908256] [] tcp_rcv_established+0x5cc/0x950 Mar 5 09:49:20 shakti kernel: [765117.908491] [] __xfrm_policy_check+0x3e0/0x570 Mar 5 09:49:20 shakti kernel: [765117.908650] [] xfrm_policy_lookup+0x0/0x220 Mar 5 09:49:20 shakti kernel: [765117.908809] [] ipt_do_table+0x20b/0x4b0 [ip_tables] Mar 5 09:49:20 shakti kernel: [765117.909085] [] tcp_v4_do_rcv+0x340/0x6e0 Mar 5 09:49:20 shakti kernel: [765117.909255] [] ip_route_input+0x39/0xdf0 Mar 5 09:49:20 shakti kernel: [765117.909411] [] tcp_packet+0x0/0xbb0 [nf_conntrack] Mar 5 09:49:20 shakti kernel: [765117.910123] [] nf_ct_deliver_cached_events+0x8d/0xc0 [nf_conntrack] Mar 5 09:49:20 shakti kernel: [765117.910738] [] ipv4_confirm+0x29/0x50 [nf_conntrack_ipv4] Mar 5 09:49:20 shakti kernel: [765117.911007] [] ipv4_confirm+0x0/0x50 [nf_conntrack_ipv4] Mar 5 09:49:20 shakti kernel: [765117.911311] [] tcp_v4_rcv+0x7bd/0x9a0 Mar 5 09:49:20 shakti kernel: [765117.911593] [] ip_local_deliver_finish+0xf9/0x210 Mar 5 09:49:20 shakti kernel: [765117.911792] [] ip_rcv_finish+0xff/0x370 Mar 5 09:49:20 shakti kernel: [765117.911969] [] ip_rcv_finish+0x0/0x370 Mar 5 09:49:20 shakti kernel: [765117.912169] [] ip_rcv+0x1ca/0x290 Mar 5 09:49:20 shakti kernel: [765117.912329] [] ip_rcv_finish+0x0/0x370 Mar 5 09:49:20 shakti kernel: [765117.912487] [] ip_rcv+0x0/0x290 Mar 5 09:49:20 shakti kernel: [765117.912653] [] netif_receive_skb+0x379/0x440 Mar 5 09:49:20 shakti kernel: [765117.912892] [] e1000_receive_skb+0x47/0x1a0 [e1000e] Mar 5 09:49:20 shakti kernel: [765117.913425] [] e1000_clean_rx_irq+0x124/0x300 [e1000e] Mar 5 09:49:20 shakti kernel: [765117.913931] [] e1000_clean_rx_irq+0x79/0x300 [e1000e] Mar 5 09:49:20 shakti kernel: [765117.914525] [] tick_program_event+0x38/0x60 Mar 5 09:49:20 shakti kernel: [765117.914738] [] e1000_clean+0x54/0x220 [e1000e] Mar 5 09:49:20 shakti kernel: [765117.915303] [] net_rx_action+0x12d/0x210 Mar 5 09:49:20 shakti kernel: [765117.915549] [] __do_softirq+0x82/0x110 Mar 5 09:49:20 shakti kernel: [765117.915753] [] do_softirq+0x55/0x60 Mar 5 09:49:20 shakti kernel: [765117.915908] [] irq_exit+0x6d/0x80 Mar 5 09:49:20 shakti kernel: [765117.916049] [] smp_apic_timer_interrupt+0x55/0x80 Mar 5 09:49:20 shakti kernel: [765117.916249] [] apic_timer_interrupt+0x28/0x30 Mar 5 09:49:20 shakti kernel: [765117.916501] [] handle_stripe5+0x34/0x1590 [raid456] Mar 5 09:49:20 shakti kernel: [765117.917132] [] enqueue_task_fair+0x27/0x30 Mar 5 09:49:20 shakti kernel: [765117.917323] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:49:20 shakti kernel: [765117.917794] [] handle_stripe+0x28/0xf00 [raid456] Mar 5 09:49:20 shakti kernel: [765117.918110] [] try_to_wake_up+0x4e/0x350 Mar 5 09:49:20 shakti kernel: [765117.918391] [] __wake_up_common+0x4b/0x80 Mar 5 09:49:20 shakti kernel: [765117.918621] [] __wake_up+0x3e/0x60 Mar 5 09:49:20 shakti kernel: [765117.918850] [] __release_stripe+0x10e/0x150 [raid456] Mar 5 09:49:20 shakti kernel: [765117.919212] [] release_stripe+0x2f/0x50 [raid456] Mar 5 09:49:20 shakti kernel: [765117.919555] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:49:20 shakti kernel: [765117.920041] [] raid5d+0x3d5/0x460 [raid456] Mar 5 09:49:20 shakti kernel: [765117.920456] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:49:20 shakti kernel: [765117.920947] [] md_do_sync+0x17b/0xb90 [md_mod] Mar 5 09:49:20 shakti kernel: [765117.921458] [] finish_wait+0xc/0x70 Mar 5 09:49:20 shakti kernel: [765117.921670] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:49:20 shakti kernel: [765117.922157] [] md_thread+0x23/0xe0 [md_mod] Mar 5 09:49:20 shakti kernel: [765117.922648] [] autoremove_wake_function+0x0/0x40 Mar 5 09:49:20 shakti kernel: [765117.922882] [] md_thread+0x0/0xe0 [md_mod] Mar 5 09:49:20 shakti kernel: [765117.923362] [] kthread+0x42/0x70 Mar 5 09:49:20 shakti kernel: [765117.923520] [] kthread+0x0/0x70 Mar 5 09:49:20 shakti kernel: [765117.923693] [] kernel_thread_helper+0x7/0x10 Mar 5 09:49:20 shakti kernel: [765117.923898] =======================